EBook

Introduction to
Intelligent Data Platforms
 Introduction to Intelligent Data Platforms 2

Introduction to Intelligent Data

Platforms
Introduction........................................................................................................................................ 3

Defining an intelligent data platform.......................................................................................... 5

Exploring an intelligent data platform........................................................................................ 6

Operational databases..........................................................................................................................................7

Analytics.....................................................................................................................................................................7
Data ingestion.....................................................................................................................................................................8
Analytical data storage....................................................................................................................................................8
Data visualization..............................................................................................................................................................9

AI and machine learning....................................................................................................................................10

Data governance....................................................................................................................................................11
Cataloging data and building the business glossary...........................................................................................11
Capturing data lineage.................................................................................................................................................. 12

Hybrid capabilities................................................................................................................................................12

Increasing security with an intelligent data platform...............................................................................13

Network security.............................................................................................................................................................. 13
Access management....................................................................................................................................................... 14
Threat protection............................................................................................................................................................. 14
Information protection and encryption.................................................................................................................. 14
Key vault............................................................................................................................................................................. 14

Intelligent data platform for business growth..........................................................................15

Conclusion..........................................................................................................................................17
 Introduction to Intelligent Data Platforms 3

Introduction
Data is essential to businesses today. With the explosion of digital information and the
increasing availability of powerful analytic tools, companies can leverage data to gain
deep insights into their operations, customers and markets. By harnessing the power of
data, businesses can optimise their processes, make informed decisions and identify new
opportunities for growth and innovation. For example, a 2021 survey of IT and senior business
decision makers conducted by Vanson Bourne found that the average profit growth for
companies with an enterprise data strategy in place for longer than a year was 5.97%, compared
to just 2.50% for ones with no enterprise data strategy1.

Many organisations have stitched together a collection of point solutions to meet their
evolving data needs. For example, one supplier for the operational database, another for the
analytical platform and a separate tool for data integration. Each solution may have its own
tools, processes and interfaces, requiring different management and maintenance efforts.
Integrating data across silos can be expensive and time-consuming for teams, requiring
complex engineering to keep data synchronised. Implementing data governance to gain a
comprehensive understanding of the storage and utilisation of critical data in reporting can also
present significant challenges. Different solutions with varying security features and protocols
can make it difficult to keep data secure. Obtaining timely and accurate data needed for AI
analysis may not be possible. Ultimately, the lack of integration and standardisation can affect an
organisation’s ability to manage and analyse its data efficiently, resulting in missed opportunities
and suboptimal decision-making.

An intelligent data platform designed for modern data can easily overcome these obstacles.
It encompasses data governance, data management, data analysis and the ability to power AI
workloads. By taking a holistic approach to data, businesses can realise the full potential of data
assets and achieve success in today’s digital economy. Whether the data is on-premises, in the
public or private cloud, structured or unstructured, gigabytes of data or petabytes or anywhere
in between, an intelligent data platform can support the data strategy of any organisation.

1 Cloudera Enterprise Data Maturity Research Report

 Introduction to Intelligent Data Platforms 4

Data governance

AI

Analytics

Operational databases

Microsoft Intelligent Data Platform²

Figure 1: An example of an intelligent data platform.

In my experience, most companies have a lot of resources, they

have the technology and very smart people and they have tons and
tons of data. But success isn’t about data collection, it’s about data
management and insight.

Prashanth Southekal, Business Analytics Author, Professor, Head of the Data

for Business Performance Institute

2 The Microsoft Intelligent Data Platform build as of May 2023

 Introduction to Intelligent Data Platforms 5

Defining an intelligent data platform

An intelligent data platform supports an organisation’s data needs across the entire lifecycle,
from data collection to analysis and visualisation, while providing a foundation of governance
and security. It also has built-in capabilities to let an organisation’s developers harness the power
of their data without the need to spend a lot of time on repetitive tasks such as copying data
from one database to another.

An effective data platform considers multiple components, such as transactional databases,

unstructured data, IoT sensors, AI and machine learning, high-performance analytical data
stores, data analytics and visualisation. An orchestration layer manages the integration of
disparate tools and data stores. The governance layer helps manage the data across the
organisation.

The core building blocks of a data platform are shown in Figure 2.

Governance

Stream
Analytic
processing

Big data
Real-time Data
Data processing
message transformation
sources ingestion and
and modelling
management

On-premises IoT Hub

Data Factory
data feed
Data Flow

Orchestration

Figure 2: The core building blocks of a data platform

 Introduction to Intelligent Data Platforms 6

Exploring an intelligent data

platform
An intelligent data platform provides the following features:

● Operational databases: Stores, manages and tracks business information such as sales,
products and customer and employee information.

● Analytics: The tools for turning raw data into insights to make better decisions. It may
include copying data from operational databases, transforming its format and visualising
the data.

● AI and machine learning: Using algorithms to identify patterns in data to predict future
outcomes.

● Data governance: Processes, policies, roles and standards to ensure effective and efficient
use of information.

These features need to be supported by a foundation of security. Using them together allows an
organisation to get the most out of its data. This enables an organisation to work efficiently in
terms of the effort required and the computing capacity consumed.

The only constant for any organisation is change. Whether growing a business to meet customer
demands better, acquiring new businesses, meeting new regulatory requirements or responding
to an executive’s request for data to help make an important decision, a data platform will
continue to evolve. An intelligent data platform helps businesses respond faster to stay ahead of
the competition.

Time and money are your scarcest resources. You want to make sure
you’re allocating them in the highest-impact areas. Data reveals impact,
and with data, you can bring more science to your decisions.

Matthew Trififoro, Chief Marketing Officer, Vapor IO

 Introduction to Intelligent Data Platforms 7

Operational databases
An operational database is designed to manage and store data for day-to-day business
operations. These databases are optimised for fast and efficient data retrieval and updates,
making them ideal for transactional applications such as online retail or banking.

One of the difficulties organisations may face with operational databases is managing their data
estate. An organisation may use dozens or even hundreds of operational databases. Knowing
what data is stored in each can be critical for compliance with regulations such as General
Data Protection (GDPR), California Consumer Privacy Act (CCPA) and similar rules throughout
the world.

Another obstacle can be harnessing the data in an operational database for analysis and
visualisation. Analysis may require copying some of the operational data so it can be
restructured or combined with additional operational or external data. For example, an
organisation might combine sales data with weather data to help forecast the inventory based
on the predicted weather. However, keeping the data synchronised between operational and
analytical data stores can be a complex process.

An intelligent data platform addresses these complexities by providing governance to manage

this data. It also helps integrate data from operational databases with the other parts of the
intelligent data platform. It makes it easy to copy data from operational databases to analytical
data stores and keep the analytical data stores in sync as data changes. Capabilities that simplify
the integration of operational data with analytical platforms enable an organisation to take
advantage of its data to make better decisions.

Analytics
Analytics allows businesses to track progress, identify opportunities and make informed
decisions. By analysing data, businesses can uncover trends and patterns that may point to new
opportunities for growth. Analytics can also help companies improve their operational efficiency
and increase their profitability.

Three primary areas of analytics make up the key components of an intelligent data platform:

● Data ingestion

● Analytical data storage

● Data visualisation

These components must work together seamlessly to implement an effective analytics solution.
 Introduction to Intelligent Data Platforms 8

Integration of analytics with the other components of the data platform, such as operational
databases, data governance and security, is also critical. The term ‘integration tax’ refers to
the cost of integrating different data sources and systems into a single platform. Without an
intelligent data platform, the integration tax can be significant and often prevents organisations
from fully taking advantage of their data.

Data ingestion
Data ingestion is the process of obtaining and importing data from various sources to a storage
medium where it can be accessed, used and analysed by an organisation. Data can be streamed
in real time or ingested in batches.

Data is typically transformed and goes through three phases:

● Raw: Data that is simply a copy of the data from the source system.

● Refined: Data that is cleaned up. For example, if the source data from the cash registers in
a retail store, removing voided transactions would refine the data.

● Curated: Data that is summarised for reporting and analysis purposes. For example, if sales
data is to be analysed at the store level, aggregated data from all of the cash registers in a
store would count as curated data.

An intelligent data platform makes data ingestion and data integration easier by simplifying
the process of moving data throughout the platform. For example, tables can be automatically
copied from an operational database to analytical storage, avoiding the need to write code for
each table.

Analytical data storage

Operational databases are typically optimised to find and update a single record or a small
number of records. For example, a query to find the customer information for a particular
customer, or to retrieve all orders the customer has made, will run well on a properly designed
operational database. However, an analytics system may need to show total sales for the year
and then allow the user to drill down to a product category, product, region and store. If there
are only a few thousand sales records, the operational system’s performance may be acceptable.
But, if there are millions or billions of sales records, using analytical data storage separate from
the operational database can provide several benefits:

● Data storage and indexing that is optimised for analytical queries.

 Introduction to Intelligent Data Platforms 9

● Isolation from the operational database to ensure that analytical processing does not
impact the operations of the system by competing for resources or blocking updates.

● The ability to combine data from multiple data sources. For example, a utility bill may also
show temperature data along with usage.

The analytical data storage could be a data warehouse that uses a relational database model,
a data lake or a data lakehouse, which combines attributes of both a data warehouse and
a data lake.

An intelligent data platform supports all of these models, so an organisation can use any of
them or a combination that makes the most sense. For example, suppose an organisation is
trying to measure greenhouse gas emissions from an industrial operation. Sensor data may
initially be stored in a data lake. Once the data is summarised, the aggregated data may be
stored in a relational data warehouse for reporting and visualisation.

Data visualisation
A good data visualisation tool can help tell a story with data. This critical part of any intelligent
data platform allows for translating the volumes of data collected, curated and enriched into a
visual representation that business users understand.

A data visualisation tool should also support data democratisation, the principle and process
of making company data easily accessible to all members of an organisation. The primary goal
of data democratisation is to empower any user, technical or non-technical, to uncover insights
in the data to make better-informed business decisions and gain a competitive advantage.
Allowing people across the organisation to access the data they need and act on it can be critical
to a company’s success.
Data visualisation tools allow the creation of executive dashboards showing overall data and
then allow the user to manipulate it in various ways, such as looking at sales by date, geographic
region, department or product category. Some data visualisation tools also incorporate
advanced analytics features such as natural language processing to let users ask a question
about data and instantly get an answer. Another example of an AI feature that may be included
in data visualisation tools is the ability to detect anomalies or outliers in the data.

Data visualisation tools may store a copy of the data to allow high-performance interactions.
Others issue queries directly against the source system. Some allow for both options.

There are a tremendous number of options to consider within the analytics area. It is important
to consider each component not only individually, but also how they will work together with the
other parts of the analytics solution and overall data platform. For example, seamless integration
between the analytics data storage and the data visualisation tool will allow developers to spend
more time creating the visualisations the organisation needs and less time massaging data.
 Introduction to Intelligent Data Platforms 10

AI and machine learning

Data is the fuel that powers AI systems, and peak performance requires clean, integrated and
accessible data and cloud solutions that are purpose-built to run large AI models at scale.
Innovating with AI requires operational databases, analytics and governance solutions that
work together – by design – to enable organisations to spend less time on data integration and
more time on value creation. AI and machine learning are often used interchangeably, but AI is
any technique that lets computers generate tasks that are close to, or exceed, human parity, for
example, speech transcription. Machine learning is a subset of AI that allows machines to use
their experience to become better at carrying out tasks. A machine learning algorithm requires a
large amount of data (that includes both attributes and an outcome into a model) so that it can
learn from the data and drive the desired outcome.

One example of machine learning is a mobile phone company that is interested in predicting
which customers are likely to switch providers. There will likely be several attributes that
could affect the outcome, including customer demographics, the number of times they called
customer service, the number of lines on their plan and how recently they upgraded their
phone. The machine learning algorithm learns from this data what attributes increase or
decrease the likelihood of an outcome. Then new data that does not yet have an outcome can
be provided to the algorithm, and the algorithm will predict what the outcome is likely to be.

A natural language processing (NLP) model is another type of AI. It is a deep learning algorithm
that can recognise, summarise, translate, predict and generate text and other content using
knowledge gained from massive datasets. There are many applications for NLP models including
summarising news and email, translating text, customer service chatbots and coding.

For example, a wealth management organisation could use NLP to allow their financial advisers
to ask questions and receive easily digestible answers based exclusively on the firm’s internal
content. In addition, AI and machine learning can be used to support day-to-day operations,
for example, by detecting potentially fraudulent credit card transactions or enriching data for
analytics by evaluating whether the sentiment of a comment is positive, negative or neutral.
 Introduction to Intelligent Data Platforms 11

Data governance
Data governance refers to the collection of processes, policies, roles, metrics and standards
necessary to manage data effectively. By implementing sound data governance practices,
organisations can establish robust data management solutions to ensure their data is secure,
private, accurate and usable throughout the data life cycle.

A robust data governance solution provides all decision-makers within an organisation with a
single source of truth, i.e., the same data sets, terminology and view, giving more opportunities
for internal flexibility. It also ensures clean data management throughout an organisation’s
governance process, meaning procedures correctly generate, handle and protect the data to
keep it in compliance.

An intelligent data platform provides a unified data governance solution to help manage and
govern on-premises, multicloud and Software-as-a-Service (SaaS) data. It lets organisations
create a holistic, up-to-date map of their data landscape with automated data discovery,
sensitive data classification and end-to-end data lineage.

By 2026, 20% of high-performing organisations will use connected

governance to scale and execute on their digital ambitions.

Gartner, Top Trends in Data and Analytics, 2022

Cataloguing data and building the business glossary

A well-governed data policy has defined plans for how data should be stored and used. These
plans then need to be communicated to the organisation. A business glossary can be an
effective tool for communicating a data governance plan. In the broadest terms, a business
glossary is a collection of definitions that a business can use to make sure that the employees of
the business communicate clearly.

When it comes to data, the governance of that data is also catalogued in the business glossary.
When an employee looks up ‘customer sales data’, they should be able to see exactly what
that data is, who is responsible for that data, whether or not they can access it and the kinds of
reports it’s used for. A business glossary helps improve communication. When all employees
have an agreed-upon meaning of a term, they can use it confidently to communicate and
collaborate across all parts of the organisation.
 Introduction to Intelligent Data Platforms 12

88% of organisations surveyed report metadata as a significant

challenge.

• 11% rank metadata as their top priority

• 65% are not using metadata tools

• 71% have four or fewer data stewards and 26% have none

IDC, Chief Data Officers: The New Business Leaders, August 2020

Capturing data lineage

Data lineage is a set of answers to the questions we might have about data:

● What is the data about? ● How was the data collected?

● Why did we collect the data? ● Where was the data collected?

● When was the data collected? ● Who did we collect the data from?

Data lineage tools, such as those found in Microsoft Purview, a part of the Microsoft Intelligent
Data Platform, can visually represent this lineage. The goal of a data catalogue is for all the
data systems and data assets an organisation controls to connect and report their lineage to
the catalogue. Once this lineage metadata is available to the data catalogue, the catalogue can
present it to employees to help them build informed data governance use cases.

Hybrid capabilities
To accelerate time to insight and operational efficiency, organisations are looking for a platform,
typically a cloud solution, that can help streamline data processes. However, many customers
need to keep some of their applications and data stores on-premises. They may have regulatory
and compliance reasons, or in certain industries, such as manufacturing, they may need data
at the edge to be ingested, stored and analysed right away. In fact, most enterprise customers
looking to realise value with data and AI will have a hybrid and/or multi-cloud environment.
These organisations are also looking for centralised governance of their data estate, applications
and databases as well as a platform that allows them to discover, catalogue, ingest, analyse
and govern all their data – no matter where the data lives and what type of data they have. It is
important for an intelligent data platform to have the governance and security capabilities to
support a hybrid model.
 Introduction to Intelligent Data Platforms 13

Let’s consider a hospital that has a database supporting an application that monitors patients’
vital signs. The operational database needs to be in the hospital to mitigate the risk of a wide
area network outage. However, the scalability and tools available in the public cloud make it
a better option for analysing the data, looking for trends and ensuring the hospital’s quality
protocols that ensure patient health are followed. An intelligent data platform makes it easy to
work with data, wherever it may be, in a governed and secure fashion.

Increasing security with an intelligent data platform

Data security is at the core of a successful, intelligent data platform. Building a successful data
strategy requires data that can be trusted. An intelligent data platform allows organisations
to identify new threats and respond quickly. A layered defence-in-depth security approach is
employed to protect an organisation’s data assets. This is shown in Figure 3:

Network security

Access management

Threat protection

Information protection

Customer data

Figure 3: Defence-in-depth approach to protecting data

Network security
The network is the first line of defence to protect an organisation’s data. It ensures that data
access is only allowed from specified network locations. For example, access to a database that
supports a company’s website may be restricted to the web servers and a small number of
other addresses required for database administration. Outbound traffic can also be restricted to
prevent data exfiltration. Protecting data assets using network access greatly reduces the risk of
brute-force attacks.
 Introduction to Intelligent Data Platforms 14

Access management
Access management has two components:

● Authentication: This is the process of proving the user is who they claim to be. A directory
mechanism is the most common method for authentication. It provides capabilities such
as multi-factor authentication (MFA) and conditional access. It also provides the benefit of
centralised administration.

● Authorisation: Authorisation is the process of assigning specific privileges to a user or

group of users. The principle of least privilege should be used. A data governance tool
can be used to scan data assets and identify sensitive data to help implement an effective
authorisation policy.

Threat protection
Threat protection can be used to track database activities and analyse logs to detect unusual
behaviour and potential exploits. Alerts can be created for suspicious activities such as SQL
injection, potential data exfiltration and brute-force attacks. Details of suspicious activities and
recommendations can be viewed through cloud security posture management (CSPM) or a
cloud workload protection platform (CWPP).

Information protection and encryption

Transportation layer security provides encryption in transit. Transparent Data Encryption (TDE)
provides encryption at rest. This prevents accessing data if disks containing database data or
backups are stolen or disposed of in a non-secure manner.
Encryption in use can protect database columns that contain sensitive information, such
as credit card numbers and national identification numbers, from being accessed, even by
privileged users such as database administrators.

Key vault
A key vault is a cloud service for securely storing and accessing secrets. A secret is anything that
requires tightly controlled access, such as API keys, passwords, certificates or cryptographic keys.
Using a key vault helps avoid storing passwords or other secrets in code or configuration files
that can be exposed to unauthorised use. Instead, an administrator stores the secrets in the key
vault and authorises users or applications to access specific secrets. The secrets are encrypted in
transit between the key vault and the client, and access to the key vault is logged.
 Introduction to Intelligent Data Platforms 15

Intelligent data platform for business

growth
An intelligent data platform makes it easy to manage operational data, integrate data from
various sources to support analytics, apply AI techniques to generate insights and visualise the
data to make it accessible to decision makers while providing data governance and security.
These capabilities let companies use data to transform their business and drive growth.

Some of the business benefits of an intelligent data platform are:

● Faster and better decision-making based on data and AI

● More opportunities for innovation and differentiation

● Improved data quality and governance

● Reduced costs and complexity of data management

Data innovation leaders, those with the most mature practices around
using data to improve products, services, customer experience, etc.,
are finding bottom-line success that eludes less mature organisations.

After surveying 2,000 organisations worldwide, we saw that the ones

that achieved excellence in six key data maturity measures report
a 9.5% increase in gross profits, compared to organisations still at
the beginner level.

Splunk, The Economic Impact of Data Innovation 2023

 Introduction to Intelligent Data Platforms 16

The technology that allows an intelligent data platform to deliver these benefits includes:

● The virtually unlimited scale of the cloud provides computational power and storage
power that can help process even the largest volumes of data.

● Data governance capabilities make it easy to understand where an organisation’s data is

and identify any data quality issues.

● A streamlined process to integrate data from operational databases to analytical systems

with less development required.

● The ability to store the data in a data warehouse, data lake or data lakehouse (which
combines features of both a warehouse and a lake) makes it easy to consume the data,
whether it is for reporting and visualisation or for AI and machine learning.

● Data visualisation allows decision-makers to easily view the data they need to make the
best decisions for the organisation.

● Integrated AI and machine learning capabilities to help organisations unlock insights and
discover intelligent connections.

The platform we’ve built with Azure resources is helping us empower

our pharmacists and technicians. It offers them the tools they need to
effectively engage with patients, and that creates major benefits for our
customer experiences.

Luigi Guadagno: Vice President, Pharmacy and HealthCare Platform

Technology Walgreens

No matter how an organisation uses data, an intelligent data platform allows an organisation
to focus more on delivering results and less on tedious data management and data movement
activities.
 Introduction to Intelligent Data Platforms 17

Conclusion
The value of data to a business is clear. An intelligent data platform harnesses data to help serve
customers better and make more timely decisions, whether that data is structured or streaming,
on-premises or in the cloud or gigabytes or petabytes in size.

An intelligent data platform empowers an organisation to meet customer demands for

a high‑performing database to manage their customer’s transactions. It allows organisations
to govern an entire data estate from one place to understand where data is stored and how it
is used.

The analytics capabilities of an intelligent data platform support any use case, whether it is
analysing what has already happened or using AI to predict what is most likely to occur in the
future based on the data.

Data visualisation capabilities provide easy to understand reports and dashboards to users
throughout the organisation, whether they need summarised information or detailed data to
make the best decisions.

Finally, security is implemented throughout the intelligent data platform with multiple layers to
ensure that data is protected.

Get more value from your data – and reduce costs

Learn how the Microsoft Intelligent Data Platform offers significant savings for your data needs.

Save
up to 90% Save
up to 59%
Azure SQL Managed Instance Azure Synapse Analytics Microsoft Intelligent
costs up to 90% less than costs up to 59% less than Data Platform
Amazon RDS for SQL Server. Snowflake. Customer Stories
Learn more Learn more Explore