Data integration

Challenges and opportunities

White paper
November 2022

Data governance

Data warehousing
and business intelligence

Machine learning

Open and Governed Data

Lakehouse Foundation

Data lake storage

Data integration

Hybrid and multicloud

data source
 Data integration 2

Contents

Introduction

Data integration challenges

The Microsoft vision for data integration

Comprehensive and cost-effective data integration

Microsoft services for data integration

Case study 1: Data integration drives digital transformation

Case study 2: Unifying fragmented data sources with integration architecture

Getting started
 Data integration 3

Zettabytes of data call for

major data integration

Managing the heavy volume and variety of data is a major Data integration challenges
struggle for organizations today. This struggle is intensified
by other variables—like data speed, location, and structure.
It’s challenging for organizations to collect, process, and
On top of that, to stay competitive, organizations need to
analyze this ever-expanding volume of data. Adding to
rapidly clean and process this data to ready it for business
that, the data is highly distributed (i.e., in different physical
intelligence (BI), artificial intelligence (AI), and machine
locations), diverse, and often constantly changing. Also,
learning (ML). The need to unify and prepare this data for
the technologies and skills needed to extract value from
analytics has driven many organizations to invest in data
data are varied. This gets in the way of making data usable
integration platforms and resources.
for business analysts, data scientists, and data engineers.

The need for data integration is in response to the shift

For central data teams, the main challenge is to break
toward the predictive era and a massive explosion of
down data silos across departments and geographies.
data. Over the last decade, data creation has increased
tremendously. As consumers and businesses accelerate
how much they create, consume, and share data, the
potential for growth is exponential.

Figure 1: Data integration challenges

 Data integration 4

They also need to develop a trusted, unified source The Microsoft Intelligent Data Platform helps you spend
such as a data lake or warehouse to store everything. less time managing data and more time creating value
From there, data engineers can extract, transform, and for your organization. It enables you to adapt rapidly, add
load (ETL) data to get it ready for analytics. This ETL layers of intelligence to applications, generate predictive
process is complex, especially when it captures the latest insights, and manage all your data—wherever it lives.
data changes. It also adds a time delay, so organizations
miss out on real-time analytics. It’s easy to generate insights you can act on when data
sources are unified. Microsoft data integration tooling
Business analysts have a different challenge. They need to brings it all together, whether it’s transactional data
find available datasets and prepare them for analytics. in your Customer Relationship Management (CRM)
With data siloed across an organization, it can be difficult system, operational data in your SQL database, or even
to uncover the right data. Often, different departments collaboration data from your Microsoft 365 applications.
may produce the same report, but with different versions
of the numbers. The organization as a whole lacks a single Critical data integration services lay the foundation for
source of truth for BI. your data and analytics ecosystem. These services work
together to consolidate and prepare data for analysis in
a central, secure environment. A governance layer maps
The Microsoft vision for your data landscape and tracks data integration pipelines,
giving you better end-to-end visibility. AI and machine
data integration learning (ML) services unlock new insights that flow
back into operational systems and enable intelligent
Through the Microsoft Intelligent Data Platform, applications. Finally, intuitive visualizations illustrate these
organizations have a way to integrate and unify their data, insights, bringing your data to life and empowering your
complete their end-to-end analytics strategies, and drive leaders to make impactful, business-transforming decisions.
better business outcomes.

Figure 2: Data integration is foundational for your analytics solution

 Data integration 5

Democratized data access: Help your entire team—

“Azure Synapse migration paved new avenues from data engineers to business analysts—find and use
of innovation by delivering unparalleled the data they need. Microsoft services support a range of
analytics on a unified platform. The integration abilities and roles, from technical experts to citizen end
of the end-to-end analytic framework with users—you can write your own queries or build code-free
Power BI and an enterprise data lake further pipelines. Low- and no-code data integrations for analytics,
accelerated time to insight.” AI, and ML speed up the process for mapping data flows
and bringing in new data sources.
—Anirudh Kala, CEO and Cofounder,
Celebal Technologies | full story Open and governed: Ensure your data stays valuable,
accessible, and trustworthy with unified governance that
makes it possible to map your data landscape. An open
Comprehensive and cost-effective and governed system empowers you to keep track of your
data integration integration pipelines and understand and easily trace
your data lineage.
Recently, Microsoft was named a Leader in the 2022
Gartner® Magic Quadrant™ for Data Integration Tools1. Alignment in a cloud ecosystem: Take advantage of
We believe this recognition shows our continued enterprise services in the Microsoft Cloud, like the Synapse
growth and commitment to delivering end-to-end Analytics workspace, AI, ML, and more. Integration with
and cost-effective data integration solutions. key Azure and Dynamics 365 services lets you scale your
data integration capabilities and simplify compliance.
Uniting your data through Microsoft services provides Tools like Microsoft Cost Management give you
the following benefits: the ability to monitor, allocate, and optimize cloud costs
with transparency, accuracy, and efficiency.
Connectivity: Bring together all your data sources—
whether they’re in the cloud or on-premises, stored in Support from ISV partners: Tap into the varied set of
Microsoft or non-Microsoft systems. Then use data independent software vendor (ISV) partners that focus
integration tooling to simplify data preparation for analysis. on data, part of the Microsoft Intelligent Data Platform’s
open and governed foundation. Several data integration
Reduced time to insights: Avoid stale data with a system partners have deeply integrated their solutions to deliver
that runs reliable data pipelines and is always up to date. a seamless Azure experience.
This way, you can be sure you’re working with the latest
insights, and your entire organization works off the most
current version.

Figure 3: Benefits of data integration with Microsoft

 Data integration 6

Trusted and secure services: Protect your data with Azure Data Factory is a managed cloud service that’s
multiple layers of security in Azure. Data integration built for large-scale data ingestion, transformation, and
with Microsoft means your data stores, pipelines, and orchestration. Use Azure Data Factory for:
warehouse are in a safe environment. You also get
built-in controls and services across identity, data, • Data engineering: Build, manage, and
networking, and apps. operationalize data ingestion and
transformation pipelines.

Microsoft services for • Data and cloud migration: Migrate data from
data integration on-premises or another cloud.

• Operational data integration: Support critical

An effective data integration foundation starts with tooling and/or continuous business processes with
that delivers comprehensive capabilities. Microsoft services ongoing data integration and synchronization.
help you to lay the groundwork with reliable data pipelines.
You can choose from more than 200 connectors to easily Azure Data Factory Studio provides your data engineers
move data and skip lengthy ETL processes. With a central, with a familiar environment to create their data integration
up-to-date data estate, you can bring the added layers of pipelines and data flows. This gives your teams more
intelligence into your applications to enable direct control and orchestration capabilities and enables
real-time model learning. This means that key insights flow heavy-duty data integration tasks for large amounts of
smoothly back into your operational systems to help you data. Hundreds of connectors enable source-specific
improve your rhythm of work. connectivity to support your end-to-end data estate,
no matter where your data lives.
Analytics
Azure Data Factory also integrates with Azure DevOps to
Services like Azure Synapse Analytics, Azure Data Factory, enable source control, release management, and CI/CD.
and Azure Synapse Link work together to combine With source control, your developers can collaborate,
your data and prepare it for analysis in a central, secure track, and save their changes to a branch of their choice.
environment. Simplified data migration, low- or no-
code ETL, enterprise business workflows, and metadata
management unlock use cases throughout your
organization.

Azure Synapse Analytics is your unified analytics service

that brings together data integration, enterprise data
warehousing, and big data analytics.

It gives you the freedom to query data on your terms, using

either serverless or dedicated options—at scale. Azure
Synapse provides a single experience to ingest, explore,
prepare, transform, manage, and serve data for immediate
BI and ML needs.

FIGURE 4A: AZURE DATA FACTORY CONNECTS YOUR DATA TO ANALYTICS

AND BUSINESS INTELLIGENCE CAPABILITIES—INCLUDING VISUALIZATIONS
 Data integration 7

Azure Synapse Link is a service that removes barriers

between Microsoft data stores and Azure Synapse
Analytics. It automatically moves data from operational
databases and business applications without long ETL
processes. This gives you near real-time analytics and a
M365 Data complete view of your business, enabling business analysts
+ to gain more value from your data. Synapse Link stands out
Azure Synapse because it does not impact the operational/transactional
system, replicates data from operational stores into
Synapse in near real time, and delivers automatic ETL.

Already connected to many Microsoft data stores, like

FIGURE 4B: SCENARIOS ENABLED BY INTEGRATING Azure Cosmos DB and Azure SQL Database, Azure Synapse
DATA FROM M365 INTO AZURE SYNAPSE Link will connect to more in the future. These connections
have all reached general availability (GA):

Microsoft Graph Data Connect uses the Microsoft

365 data you already have—with user IDs obfuscated • Dataverse
to protect your employees’ privacy—to show how your
• Azure Cosmos DB
teams communicate and collaborate. Microsoft Graph
Data Connect is a secure, scalable data service that quickly • SQL Server 2022
copies relevant Microsoft 365 datasets into your Azure
tenant for advanced analytics and insights. • Azure SQL Database

With Data Connect, you can save time, reduce costs, and
make important decisions with Microsoft 365 data. It can
help you unlock insights from customer relationships,
business processes, people productivity, and security
and compliance.

FIGURE 4C: AZURE SYNAPSE LINK ENABLES NEAR REAL-TIME INSIGHTS FROM MULTIPLE DATA SOURCES
 Data integration 8

Power BI Dataflows are a self-service, cloud-based,

Microsoft Purview provides a unified data data preparation technology. Dataflows can be used
governance solution to help manage and govern to easily ingest, cleanse, transform, integrate, enrich,
your on-premises, multicloud, and software as a service and schematize data from a large array of transactional
(SaaS) data. Easily create a complete, up-to-date map and observational sources, encompassing all data
preparation logic.
of your data landscape with automated data discovery,
sensitive data classification, and end-to-end data
lineage. When governing data integration pipelines, Case study 1: Data integration
Microsoft Purview delivers insights on data lineage to drives digital transformation
improve root cause analysis and traceability.
The true benefit of data integration is when it results
in better business outcomes. For Asahi Kasei, data
integration with Microsoft tooling unlocked new use
Business intelligence cases beyond the IT department and united 1,200 data
systems. Let’s take a deeper look.
All your data teams—whether deeply technical or not—can
discover and visualize data from across your organization. Asahi Kasei used new data management
Data democratization means your data stewards can curate infrastructure built on Azure Data Factory and
and share the data your teams need, and those teams can Microsoft Purview
easily uncover trends through AI/ML modeling, present the
insights, and then act on those insights. The Asahi Kasei Group makes a wide array of products,
from battery separators and biodegradable textiles to
Power BI is a unified, scalable platform for self-service engineering plastics and sound solutions. Data integration
and enterprise BI that lets you connect to and visualize is essential for different business groups to share vital
any data. information across the company.

Power Query is a data transformation and data As the digital transformation team worked to build their
preparation engine with a simplified user experience. data management infrastructure service, they faced three
It has connectors to hundreds of data sources, letting major challenges.
you manage your own data. It also gives you
enterprise-ready, hybrid, and on-premises access. • The first was information silos. Data was scattered
across different systems. This meant only individual
Power Query enables business analysts to manage data departments or teams could use their data.
preparation tasks on their own for workloads across
Power Platform, Dynamics 365, and Microsoft Exchange. • The second was cost and lead time. The process
This brings simple data integration experiences to of linking data across different systems occurred
familiar tools like Excel, Power BI, Power Apps, and independently. This delayed the use of data and
Power Automate so your teams can do more with larger investments.
data where they’re already using it.
• The third challenge was operational. As the
linkage between systems became more complex,
the company had to deal with a bigger load of
operations, maintenance, and cost.
 Data integration 9

An open and governed data platform Case study 2: Unifying

fragmented data sources with
Junko Okubo, Group Manager of Strategy & Planning
Group, noted the benefits of knowing where all their integration architecture
data was located on an open and governed platform.
“By using Azure Data Factory and Microsoft Purview, To improve insights, you sometimes have to start at
you can integrate the scattered data on the data hub the beginning. That’s exactly what Atea Sweden did.
and reduce the lead time for data integration. It also They connected data sources to build a robust product
enables you to promote data utilization across the recommendation model.
company while ensuring effective data governance.
When you integrate data at a specific location, you will With Customer Insights and Power BI dashboards,
be required to follow the most stringent standard to Atea can see the number of total customers, their
manage data, and this is what we wanted to avoid. roles within the buying organization, top product
We wanted to realize a setup where each employee categories, and the total transactions.
‘knows where the data is located’ while managing data
at where it should be. The data can be located by using Atea is the leading provider of IT infrastructure and
the catalog and retrieved as necessary.” related services to organizations within the Nordic and
Baltic regions. Atea’s competence and leading market
position in IT infrastructure have helped the company
“Corporate Strategy can now grasp grow faster than the market.
management data using the management
dashboard, which was built as part of Atea wanted to transform the buying experience in its
the utilization projects. Also, aside from business-to-business (B2B) eShop store. Rather than a
management dashboard, you can go to the broad-based engagement, it wanted to use timely and
Microsoft Purview screen for details when you targeted messaging to build customer loyalty.
need to check the data source on the Data
Factory from which the data originated. Atea worked on unifying data sources. It created an
integration architecture that used Power Query to
This ability to trace specific information all the help stitch data siloes together—such as integrating
way back to its data source is astonishing in transactional data with Dynamics 365 Customer Insights
a way. This setup is something data maniacs for its order transactions. It also used Power Query to
would find irresistible. In fact, it has earned take data from its ML product recommendation model
a reputation as being easy to read and easy and connect it with Customer Insights.
to use, especially among DX promotion
management people.”

—Junko Okubo, Group Manager of Strategy &

Planning Group, Corporate IT Management,
Digital Value Co-Creation,
Asahi Kasei Corporation | full story
 Data integration 10

Connecting data for a holistic view of the customer

“The unification of multiple data sources
Now, for the first time, Atea has a complete view of spread out across different locations
its eShop customers. With Customer Insights and internally is very simple and efficient
Power BI dashboards, Atea can see the number of total and enables Atea’s analytics team to
customers, their roles within the buying organization, construct segments and trigger real-time,
top product categories, and the total transactions. Atea’s personalized communications.”
ecommerce communication strategy now rests on a base
—Robert Chung, Head of Analytics,
of transparent and reliable data that can easily be sorted.
eCommerce, Atea Sweden | full story
“Bringing Customer Insights to our email marketing
helps us build better relationships with customers by
delivering content that acknowledges their interests and
preferences. With a unified view of customer data, we can
deliver refined communications that drive the next best
action in a customer’s journey,” says Chung.

Get started with Microsoft data integration today

Connect with an Azure sales specialist to learn more about Microsoft data integration and for details on pricing, analytics
best practices, setting up a proof of concept, and more. Or, try one of the following:

• Build in the cloud with an Azure free account

• Download a free Azure Synapse Analytics eBook

• Create your first Azure SQL Database

1
Gartner, Magic Quadrant for Data Integration Tools, August 17, 2022, Ehtisham Zaidi, Robert Thanaraj, Sharat Menon, and Nina Showell.
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire
document. The Gartner document is available upon request from Microsoft. GARTNER and Magic Quadrant are registered trademarks
and service mark of Gartner, Inc. and its affiliates in the United States and internationally and are used herein with permission. All rights
reserved. Gartner does not endorse any vendor, product, or service depicted in its research publications and does not advise technology
users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions
of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or
implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

© 2022 Microsoft Corporation. All rights reserved.

This document is provided “as is.” Information and views expressed in this document, including URL and other internet website
references, may change without notice. You bear the risk of using it.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use
this document for your internal, reference purposes.