6

Qlik Data Integration

Reference Architectures
for Business Acceleration
The Value of Effective
Data Integration
In need of a
Take a minute to think about one of your organization’s new business initiatives.
better approach?
Whether it’s a customer experience, a new service, or a cost-saving effort, the Consider these important questions:
likelihood that it’s dependent on data is high. And not just data from one source, but • How will you improve the speed
all types of data – historical and real time – from many sources at once are at which you deliver data?
now required to make every digital business initiative a success.
• Can you increase the volume
of data you deliver with your
All the data your organization collects should contribute to delivering real growth, current model?
innovation, and competitive edge to your business. Yet as more data floods into your
• How will you boost access
environment faster from more sources than ever, people-intensive integration tools
to data you provide to your people
are getting in the way. They’re bottlenecking the process of delivering and teams?
analytics-ready data to all your business initiatives the moment they’re needed which
is creating significant challenges when it comes to assessing data’s value • Can you grow the efficiency
of your data integration
and identifying valuable resources. That’s no way to get ahead.
process with existing tools?

Qlik Data Integration Reference Architectures | 2

The Need is Clear
Organizational leaders aren’t shy about expressing what they Strong investment in data
management & analytics yields
want from you, data engineers, and data integration solutions to
significant benefits:
accelerate top business initiatives:
• Operational efficiency
Integrated access to all data – Leaders want to bring together increasingly improvement (76%)
high volumes of data from a growing array of sources and replicate it to data
management and analytics platforms – without production app disruption • Revenue increase (75%)

Governance – Business leaders expect IT to track, maintain, and protect data • Profit increase (74%)
at every stage of the lifecycle Source: IDC, InfoBrief sponsored by Qlik, “Data as the
New Water: The Importance of Investing in Data and
Analytics Pipelines,” June 2020.
Agility – Leaders hope for the automation of the design, creation, and
continuous updating of data warehouses and data lakes on any cloud platform
to speed decision making

Qlik Data Integration meets these requirements and more. It’s the most effective data
integration for delivering all types of data from any source to the right people as quickly
and safely as possible. Qlik Data Integration automates the creation of data streams
and efficiently moves them to applications, data warehouses, and data lakes, delivering
business-ready data to Qlik Sense or other analytics solutions.

Qlik Data Integration Reference Architectures | 3

 6 Data Integration Use Cases
Effectively balance rising demands for data – at speed – against security and performance risks with modern, secure, efficient data-to-analytics
pipelines through DataOps for analytics from Qlik. Discover the value of Qlik Data Integration for bringing together all data from many sources and
making it more valuable to your business in these detailed reference architectures for six popular use cases.

1 2 3 4 5 6

Data Warehouse Data Lake Data Lakehouse Event-Driven Event-Driven Data Mesh
Power agile data warehousing Continuously provide accurate, Architect for a single source of Data – Lambda Data – Kappa Form a decentralized and distributed
with automation to quickly design, timely, and trusted transactional truth for all analytic initiatives – Reliably update a data lake and Handle real-time data processing data fabric foundation where domain
build, deploy, manage, and catalog data sets for business analytics. artificial intelligence (AI), business efficiently train ML models using and continuous data reprocessing data product owners use common
purpose-built data warehouses Automate the entire data pipeline intelligence (BI), machine learning three layers. Understand the using a single stream processing data infrastructure via self-service to
(especially cloud-based) faster (from real-time data ingestion to (ML), streaming analytics, data batch layer, operating on all data, engine. develop pipelines that share data in a
than traditional solutions. the creation and provisioning of science, and more. produces the most accurate governed, open approach. Adhere to
Data engineers: Invest in less
analytics-ready datasets) without Data engineers: Unify both results but at high latency; the core principles: a.) domain-oriented,
expensive hardware and solve
Data engineers: Meet or exceed manual scripting. data lake and data warehouse speed layer shows real-time views decentralized data ownership and
multi-layered, Lambda architecture
the demands for analytics- Data engineers: Realize faster automation in one user interface in a low-latency, near real-time architecture for scale, b.) data-
redundancy by replaying data
ready data marts that enable data- return on data lake investments to plan and execute either model; and the serving layer as-a-product as an architectural
instead of maintaining two
driven insights at the speed of while confidently meeting growing with ease. supports queries from batch and unit (built, deployed, maintained),
code bases (batch and speed
change. demands for analytics-ready speed layer results. c.) self-service data infrastructure
layers) to process unique events
datasets in real time. for the autonomous creation and
Data engineers: Predict upcoming continuously in real time
consumption of data products,
events accurately. while meeting standard
and d.) federated governance
quality of service.
and interoperability standards to
aggregate and correlate independent
data products.
Data engineers: Derive value from
analytical data at scale while the
data landscape, use cases, and
responses constantly change.
 REFERENCE ARCHITECTURE

1 Data Warehouse
Speed and simplify your data warehouse lifecycle – design, build, deploy, manage and catalog purpose-built
data warehouses – with automation for faster time to insights.

1 Real-Time Data Ingestion

6 5 2 7
Change data capture for real-time data
Data Ingestion Data Transformation Data Machine Catalog Qlik
Sources replication without impairing production
Profile Learning & Lineage Analytics
4 3 system performance
Customs Data Warehouse
Transformations Automation
SAP
2 Catalog & Lineage
Discover, govern, and protect data
REAL-TIME (CDC) 1 by leveraging a layer of common
PROFILE AND
MANIPULATE ENRICH DATA enterprise metadata
USER-DEFINED MODEL-DRIVEN DATA
DATABASE Other
3 Data Warehouse Automation
Initiatives
INCREMENTAL
LOAD
Automate the entire data warehouse
PUSHDOWN lifecycle to accelerate the availability
<SQL>
of analytics-ready data
SaaS

BUSINESS
BATCH LOAD Cloud Data Warehouse INTELLIGENCE 4 Custom Transformation
Create flexible, fit-for-purpose data
pipelines to transform raw data into data
MAINFRAME Application that is ready for analytics
Automation

DATA SCIENCE
5 Machine Learning to Enrich Data
AutoML to enrich data with prediction,
APPLICATIONS
scoring, classification and more
REVERSE ETL 8
RAW DATA CONFORMED DATA CONSUMABLE DATA 6 Data Profiling
MODERNIZATION Assess the quality and structure of data
FILES
CATALOG SYNC
sources to fix data quality issues and
promote good data governance

7 Qlik Analytics
Foundational Empower all your users to explore freely
Services at the speed of thought with hyperfast
SaaS AND ASSOCIATIVE AI – INSIGHT
MONITOR ORCHESTRATE GOVERN API HYBRID CLOUD ENGINE ADVISOR AI – AUTOML calculations, always in context, at scale

8 Reverse ETL
Replicating enriched data from the
warehouse back to the operational systems
Qlik Capabilities
of record
Others

Qlik Data Integration Reference Architectures | 5

 REFERENCE ARCHITECTURE

2 Data Lake
Automate your data lake pipeline – from real-time ingestion to processing and refining raw data and making
it accessible to consumers – without writing code for greater speed and agility.

1 Real-Time Data Ingestion

6 5 2 7 Change data capture for real-time data
Data Ingestion Data Transformation Data Machine Catalog Qlik replication without impairing production
Sources
Profile Learning & Lineage Analytics system performance
4 3
Customs Data Lake
Transformations Automation Catalog & Lineage
SAP
2
Discover, govern, and protect data
by leveraging a layer of common
REAL-TIME (CDC) 1
PROFILE AND enterprise metadata
MANIPULATE ENRICH DATA
USER-DEFINED MULTI-ZONED DATA
DATABASE
Other Data Lake Automation
3
INCREMENTAL Initiatives Automate the process of providing
LOAD
PUSHDOWN continuously updated, accurate, and trusted
<SQL>
datasets for business analytics
SaaS

BATCH LOAD Cloud Data Lake BUSINESS

4 Custom Transformation
INTELLIGENCE
Create flexible, fit-for-purpose data
pipelines to transform raw data into data
MAINFRAME Application that is ready for analytics
Automation
5 Machine Learning to Enrich Data
DATA SCIENCE
AutoML to enrich data with prediction,
APPLICATIONS scoring, classification, and more.
REVERSE ETL 8
RAW DATA LIVE VIEWS PERFORMANCE CONSUMABLE DATA 6 Data Profiling
VIEWS
MODERNIZATION
Assess the quality and structure of data
FILES sources to fix data quality issues and
CATALOG SYNC
promote good data governance

Qlik Analytics
7
Foundational
Empower all your users to explore freely
Services
at the speed of thought with hyperfast
MONITOR ORCHESTRATE GOVERN API
SaaS AND ASSOCIATIVE AI – INSIGHT
AI – AUTOML calculations, always in context, at scale
HYBRID CLOUD ENGINE ADVISOR

8 Reverse ETL
Replicating enriched data from the
warehouse back to the operational systems
Qlik Capabilities of record

Others

Qlik Data Integration Reference Architectures | 6

 REFERENCE ARCHITECTURE

3 Data Lakehouse
Combine the low-cost, broad data access and structured management of data lakes and data warehouses to
apply the full, current data set toward business intelligence, data analytics and machine learning for flexibility,
simplicity, and efficiency. Catalog & Lineage
1
5 4 1 6 Discover, govern, and protect data
Data Ingestion Data Transformation Data Machine Catalog Qlik by leveraging a layer of common
Sources
Profile Learning & Lineage Analytics enterprise metadata
3 2 2
Customs Data Lake Data Warehouse
Transformations Automation Automation Data Lake & Warehouse Automation
2
Automate the process of providing
SAP
continuously updated, accurate, and
REAL-TIME (CDC)
PROFILE AND trusted data sets for business analytics
MANIPULATE ENRICH DATA
USER-DEFINED MULTI-ZONED MODEL-DRIVEN DATA
Other Custom Transformation
3
INCREMENTAL Initiatives Create flexible, fit-for-purpose data pipelines
DATABASE
LOAD to transform raw data into data that is ready
PUSHDOWN
<SQL> for analytics

Machine Learning to Enrich Data

BATCH LOAD Cloud Data Lake BUSINESS
4
SaaS
INTELLIGENCE
AutoML to enrich data with prediction,
scoring, classification and more
Application
Automation Data Profiling
Assess the quality and structure of data
5
MAINFRAME
DATA SCIENCE sources to fix data quality issues and
promote good data governance
RAW DATA LIVE VIEWS PERFORMANCE CONSUMABLE DATA
VIEWS
Qlik Analytics
6
REVERSE ETL Discovery, interpretation, and communication
Cloud Data Warehouse
APPLICATIONS 7 MODERNIZATION
of meaningful patterns in data to apply
towards effective decision making

Reverse ETL
7 Replicating enriched data from the warehouse
CATALOG SYNC
FILES back to the operational systems of record

RAW DATA CONFORMED DATA CONSUMABLE DATA

Foundational
Qlik Capabilities Services
SaaS AND ASSOCIATIVE AI – INSIGHT
MONITOR ORCHESTRATE GOVERN API HYBRID CLOUD ENGINE ADVISOR AI – AUTOML
Others

Qlik Data Integration Reference Architectures | 7

 REFERENCE ARCHITECTURE

4 Event Driven Data Architecture - Lambda

Reliably update your data lake and efficiently train machine learning models – using batch, speed/
stream and serving layers – for accurate predicting of upcoming events.

1 Real-Time Data Ingestion

3
Change data capture for real-time data
Sources Data Data Streaming Platform Catalog Qlik replication without impairing production
Publishing & Lineage Analytics system performance

2 Batch / Incremental Load

SAP For historical data with the fault-tolerant,
REAL-TIME (CDC) 1 distributed storage, ensuring a low
possibility of errors even if the system
crashes
Other
INCREMENTAL KAFKA KINESIS EVENT HUBS PUB/SUB CONFLUENT Initiatives Qlik Analytics
DATABASE LOAD 3
2 Empower all your users to explore freely
at the speed of thought with hyperfast
calculations, always in context, at scale
BATCH LOAD

MAINFRAME 4 Monitor
BUSINESS
INTELLIGENCE Monitor data ingestion tasks from a single
pane of glass view
Cloud Data Lake Cloud Data Warehouse

5 Orchestrate
FILES Orchestrate data ingestion tasks from a
single pane of glass view

DATA SCIENCE 6 API

APIs to automate and integrate with
LOG DATA
other applications for monitoring and
orchestration

APPLICATION MODERNIZATION
RAW DATA PERFORMANCE CONFORMED DATA CONFORMED DATA CONSUMABLE DATA
DATA
& LIVE VIEWS

Foundational
Qlik Capabilities Services
SaaS AND ASSOCIATIVE AI – INSIGHT
MONITOR ORCHESTRATE GOVERN API HYBRID CLOUD ENGINE ADVISOR AI – AUTOML
Others
4 5 6 Qlik Data Integration Reference Architectures | 8
 REFERENCE ARCHITECTURE

5 Event Driven Data Architecture - Kappa

Handle real-time data processing and continuous data reprocessing with a single stream engine and
low-cost hardware for accurate processing of unique events happening continuously.

1 Real-Time Data Ingestion

2
Change data capture for real-time data
Data Data Streaming Platform Catalog Qlik
Sources replication without impairing production
Publishing & Lineage Analytics system performance

2 Qlik Analytics
SAP Empower all your users to explore freely
1 at the speed of thought with hyperfast
calculations, always in context, at scale
REAL-TIME (CDC)
Other Monitor
3
KAFKA KINESIS EVENT HUBS PUB/SUB CONFLUENT Initiatives Monitor data ingestion tasks from a single
DATABASE
pane of glass view

4 Orchestrate
Orchestrate data ingestion tasks from a
Cloud Data Lake Cloud Data Warehouse BUSINESS single pane of glass view
MAINFRAME INTELLIGENCE

5 API
APIs to automate and integrate with
other applications for monitoring and
FILES orchestration

DATA SCIENCE

LOG DATA

RAW DATA PERFORMANCE CONFORMED DATA CONFORMED DATA CONSUMABLE DATA MODERNIZATION
& LIVE VIEWS

APPLICATION
DATA

Foundational
Services
SaaS AND ASSOCIATIVE AI – INSIGHT
MONITOR ORCHESTRATE GOVERN API HYBRID CLOUD ENGINE ADVISOR AI – AUTOML

Qlik Capabilities 3 4 5

Others

Qlik Data Integration Reference Architectures | 9

 REFERENCE ARCHITECTURE

6 Data Mesh
Create a foundation for deriving value from analytical data at scale as your data landscape,
use cases, and responses continually change.

1 Catalog & Lineage

5 4 1 6
Discover, govern, and protect data
Data Ingestion Data Transformation Data Machine Catalog Qlik
Sources by leveraging a layer of common
Profile Learning & Lineage Analytics enterprise metadata
3 2 2
Customs Data Lake Data Warehouse
Transformations Automation Automation Data Lake & Warehouse Automation
2
SAP Automate the process of providing
REAL-TIME (CDC) continuously updated, accurate, and trusted
PROFILE AND
MANIPULATE ENRICH DATA
data sets for business analytics
USER-DEFINED MULTI-ZONED MODEL-DRIVEN DATA
Other Custom Transformation
3
DATABASE INCREMENTAL Initiatives Create flexible, fit-for-purpose data
LOAD
PUSHDOWN pipelines to transform raw data into data
<SQL>
that is ready for analytics

BATCH LOAD
BUSINESS
INTELLIGENCE 4 Machine Learning to Enrich Data
SaaS Data AutoML to enrich data with prediction,
Domain 1 scoring, classification and more
Application RAW LIVE PERFORMANCE CONSUMABLE CONFORMED
DATA VIEWS VIEWS DATA
Automation DATA Data Profiling
5
DATA SCIENCE
Assess the quality and structure of data
MAINFRAME sources to fix data quality issues and
promote good data governance
Data
Domain 2 6 Qlik Analytics
RAW CONFORMED CONSUMABLE
REVERSE ETL
DATA Empower all your users to explore freely
APPLICATIONS 7 DATA DATA MODERNIZATION
at the speed of thought with hyperfast
calculations, always in context, at scale

7 Reverse ETL
FILES CATALOG SYNC Data Replicating enriched data from the
Domain N warehouse back to the operational systems
of record

Foundational
Qlik Capabilities Services
SaaS AND ASSOCIATIVE AI – INSIGHT
MONITOR ORCHESTRATE GOVERN API HYBRID CLOUD ENGINE ADVISOR AI – AUTOML
Others

Qlik Data Integration Reference Architectures | 10

The Advantages of
Qlik Data Integration
With reference architectures such as Qlik Data Warehouse,
Qlik Data Lake, Qlik Data Lakehouse, Qlik Event-driven Data,
and Qlik Data Mesh, you and your organization gain:

Real-Time Insights – Our solutions uncover actionable

insights at the speed of business from critical transaction
systems and other enterprise sources without sacrificing
accuracy or quality so leaders can better respond to
intensifying competition.

Data Speed and Scale – Our modern, cloud-based

infrastructures together with real-time streaming and
automation help leaders quickly process and monetize
fast-growing data from all sources and formats.

Flexibility and Agility – Since data and analytics

infrastructures constantly change and grow, our solutions
allow technology (rather than scarce expertly skilled
resources) to scale and adapt to shifting needs to speed
outcomes and democratize access to quality data.

Efficiency – Our solutions lower costs, boost productivity,

and speed time-to-market.

Qlik Data Integration Reference Architectures | 11

Accelerate Your Business with Qlik
Qlik Data Integration automates the creation of data streams from core
transactional and enterprise systems, efficiently moving data to applications,
data warehouses, and data lakes in the cloud or on-premises, and then
cataloging and delivering analytics-ready data to Qlik Sense or other analytics
solutions including those provided by the major cloud providers.

By quickly delivering data to the user without typical business friction, Qlik Data
Integration powers the agility necessary to drive needed business value out of
scattered and disparate data.

Visit Qlik Data Integration to learn more »

Qlik Data Integration Reference Architectures | 12

About Qlik
Qlik transforms complex data landscapes into actionable insights, driving strategic business outcomes. Serving over 40,000 global customers, our portfolio
leverages advanced, enterprise-grade AI/ML and pervasive data quality. We excel in data integration and governance, offering comprehensive solutions that work
with diverse data sources. Intuitive and real-time analytics from Qlik uncover hidden patterns, empowering teams to address complex challenges and seize new
opportunities. Our AI/ML tools, both practical and scalable, lead to better decisions, faster. As strategic partners, our platform-agnostic technology and expertise
make our customers more competitive.

qlik.com

© 2024 QlikTech International AB. All company signs, names, logos, product names, and/or trade names referenced herein, whether or not appearing with the symbols ® or ™, are trademarks of QlikTech Inc or its affiliates. All other products, services, and company names mentioned herein may be trademarks of their
respective owners and are acknowledged as such. For a list of Qlik trademarks please visit: https://www.qlik.com/us/legal/trademarks 1-DM-04