Engineering Data Mesh in Azure Cloud: Implement data mesh using Microsoft Azure's Cloud Adoption Framework

Engineering Data Mesh in Azure Cloud

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Niranjan Naikwadi

Publishing Product Manager: Yasir Ali Khan

Book Project Manager: Kirti Pisat

Senior Editor: Tazeen Shaikh

Technical Editor: Seemanjay Ameriya

Copy Editor: Safis Editing

Proofreader: Safis Editing

Indexer: Subalakshmi Govindhan

Production Designer: Joshua Misquitta

DevRel Marketing Coordinator: Vinishka Kalra

First published: March 2024

Production reference: 1150324

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-80512-078-0

www.packtpub.com

To my dear father, Ashok, and my cherished late mother, Asha, who have been guiding lights on my life's journey. To my beloved wife, Reshma, whose unwavering support and encouragement have been my constant source of strength.

Contributors

About the author

Aniruddha Deswandikar has more than three decades of industry experience working with start-ups, enterprises, and software companies. He has been an architect and a technology leader at Microsoft for almost two decades, helping Microsoft customers build scalable applications and analytical solutions. He has spent the past three years helping customers adopt and implement the data mesh architecture. He is one of the subject matter experts on data mesh and cloud-scale analytics at Microsoft Europe, helping both customers and internal teams to understand and deploy data mesh on Azure.

About the reviewer

Vinod Kumar is a customer success leader for Microsoft Global Accounts with over 25 years of industry experience delivering end-to-end cloud solutions to customers. Based in Singapore, he leads the Asia team to help customers build resilient cloud architectures, embrace digital innovation and transformation, secure their use of the cloud, and make informed decisions using AI and data solutions. He is a mechanical engineer from the College of Engineering, Guindy. He is a passionate technology leader who inspires people, embraces tech, and champions inclusion. He is an author of multiple books on SQL Server and an avid community speaker.

I'd like to thank my daughter, Saranya, and my whole family for giving me the space I needed to contribute to this book.

Table of Contents

Preface

Part 1: Rolling Out the Data Mesh in the Azure Cloud

1

Introducing Data Meshes

Exploring the evolution of modern data analytics

Discovering the challenges of modern-day enterprises

DaaP

Data domains

The data mesh solution

Summary

2

Building a Data Mesh Strategy

Is a data mesh for everybody?

Aligning your analytics strategy with your business strategy

Understanding data maturity models

Stage 1

Stage 2

Stage 3

Stage 4

Building the technology stack

The analytics team

Data governance

Approaches to building your data mesh

Summary

3

Deploying a Data Mesh Using the Azure Cloud-Scale Analytics Framework

Introduction to Azure CSA

Understanding landing zones

Organizing resources

Designing a cloud management structure

Hierarchical policies

Diving deeper into landing zones in CSA

Data management landing zone

Data landing zone

Automating landing zone deployment

IaC

Organizing resources in a landing zone

Networking topologies

Security and access control

Streamlining deployment through DevOps

Summary

4

Building a Data Mesh Governance Framework Using Microsoft Azure Services

Data mesh governance requirements

Data catalog

Collecting and managing metadata

Step 1 – ensure accuracy and completeness

Step 2 – verify data classification

Step 3 – add a business glossary

Step 4 – add lineage information

Monitoring and managing data quality

Implementing data observability

Summary

5

Security Architecture for Data Meshes

Understanding the security requirements of data mesh architecture

Understanding authentication and authorization in Azure

Managing data access

SQL Database

Data lakes

Data lake structure

Managing data privacy

Data masking

Data retention

Summary

6

Automating Deployment through Azure Resource Manager and Azure DevOps

Azure Resource Manager templates for landing zones

Understanding the ARM template structure

Source code control for ARM templates

Azure DevOps pipelines for deploying infrastructure

Base data product templates

T-shirt sizing

Landing zone requests

Landing zone approval

Landing zone deployment

Self-service portal

Customized templates

Summary

7

Building a Self-Service Portal for Common Data Mesh Operations

Why do we need a self-service portal?

Gathering requirements for the self-service portal

Requesting a data product zone

Browse and reuse pipeline

Data discovery

Access management

Requesting landing zones or data products

Data catalog

Hosting common data pipeline templates

Azure Data Factory

Azure Data Factory instance

Integration runtime

Creating linked services

Create a sequence of activities

Parameterize the pipeline

Continuous integration/continuous development

Data mesh portal integration

Other common features of a self-service portal

Architecting the self-service portal

Active Directory and Domain Name System (DNS)

Application Gateway

Azure App Service

Azure Cosmos DB

Git Repo and Azure DevOps pipelines

Network and security

Azure Cache for Redis (optional)

Azure SQL DB (optional)

Summary

Part 2: Practical Challenges of Implementing a Data Mesh

8

How to Design, Build, and Manage Data Contracts

What are data contracts?

What are the contents of a data contract?

Who creates and owns a data contract?

Who consumes the data contract?

How do we store data and access contracts?

How do we link data contracts to data consumption or pipelines?

Catalog and contract document design

Set up Cosmos DB

Write the integration code

Searching contracts and data assets

Put the pieces together

Summary

9

Data Quality Management

Why is data quality important?

How is data quality defined?

How to manage data quality

Accuracy

Completeness

Consistency

Timeliness

Validity

Uniqueness

Reliability

Data quality management systems

Completely decentralized

Completely centralized

The hybrid approach

Build versus buy

Popular data quality frameworks and tools

Summary

10

Master Data Management

Single source of truth

What causes discrepancies in master data?

MDM design patterns

MDM architecture for a data mesh

Build versus buy

Popular MDM tools

Summary

11

Monitoring and Data Observability

Piecing it all together – the importance of data mesh monitoring and data observability

How data mesh monitoring differs

Baking diagnostic logging into the landing zone templates

Azure Platform Metrics

Azure platform logs

Enabling diagnostic settings in an ARM template

Designing a data mesh operations center

Step 1 – collection

Step 2 – rank the critical metrics and events

Step 3 – build a threshold logic for each service in a data product

Step 4 – build a monitoring view for each resource

Step 5 – build a threshold logic for each data product

Step 6 – build a threshold logic for each data landing zone

Step 7 – set up alerts for critical metrics

Step 8 – host the dashboards in one location

Tooling for the DMOC

Azure Monitor

Log Analytics

Azure Data Explorer

Grafana

Power BI

Data observability

Setting up alerts

Piecing it all together

Summary

12

Monitoring Data Mesh Costs and Building a Cross-Charging Model

Components of data mesh costs

Cost models in a data mesh

Overview of cost management in Azure

Allocating costs to different data product groups and domains

How to determine the cost of shared resources

Summary

13

Understanding Data-Sharing Topologies in a Data Mesh

What is in-place sharing?

Understanding data-sharing challenges in a data mesh

Latency

Security and access control

Data formats and protocols

Exploring different methods available for sharing data

In-place access

Data pipelines

Data APIs

Data Share

Picking the right data-sharing topologies

In-place sharing

Data pipelines

Data APIs

Data sharing

Summary

Part 3: Popular Data Product Architectures

14

Advanced Analytics Using Azure Machine Learning, Databricks, and the Lakehouse Architecture

Requirements

Architecture

Components

Source data

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Databricks

Azure Machine Learning

Azure Kubernetes Service (AKS)

Power BI

Azure Data Share

Data flow

Scenarios

Summary

15

Big Data Analytics Using Azure Synapse Analytics

Requirements

Architecture

Components

Source data

Azure Synapse pipelines

Azure Data Lake Storage Gen2

Azure Synapse

Azure Cosmos DB

Azure AI Search

Power BI

Azure Data Share

Data flow

Scenarios

Summary

16

Event-Driven Analytics Using Azure Event Hubs, Azure Stream Analytics, and Azure Machine Learning

Requirements

Architecture

Components

Source data

Azure Event Hubs

Azure IoT Hub

Azure Stream Analytics

Azure Data Explorer

Azure Machine Learning

Azure Cosmos DB

Power BI

Data flow

Combining architectures for real-time and big data analytics

Scenarios

Summary

17

AI Using Azure Cognitive Services and Azure OpenAI

Requirements

Architecture

Components

Source data

Azure Data Factory

Azure Translator

Azure AI Document Intelligence

Azure OpenAI embedding models

Azure Redis Cache

Azure App Service

Semantic Kernel

Azure OpenAI

Bing search

Content filtering and security

Data flow/interactions

Scenarios

Summary

Index

Other Books You May Enjoy

Preface

In 2019, Zhamak Dehghani published her whitepaper on data mesh during her time at Thoughtworks. While it caught the attention of many large corporations, adopting data mesh was not easy. Most large companies have a strong legacy of analytical systems, and migrating them to a mesh architecture can be a daunting task. At the same time, the theoretical concepts of data mesh can be confusing when you map them to an actual analytical system.

In 2021, I started working with a large Microsoft customer that was struggling with their centralized data analytics platform. The platform was based on a central data lake and a single technology stack. It was rigid and was hard for all the stakeholders to adopt. As a result, many projects were creating their own siloed infrastructure, producing islands of data, technology, and expertise. We observed the dilemma the central analytics team was facing and proposed the data mesh architecture. It seemed that data mesh would solve most of their challenges around agility and adoption, as well as opening the doors to some other challenges, such as federated governance.

In the next year, we helped onboard this customer to data mesh. It was a long journey of multiple workshops followed by a consulting engagement where we built data mesh artifacts for them. Since then, I have been engaged with multiple customers on data mesh projects. As a member of a team of subject-matter experts on data mesh at Microsoft Europe, I have also guided other Microsoft team members on how to engage, design, and manage a data mesh project.

Along the way, I have realized that translating the theory of data mesh into a practical, production-ready system can be a challenge. A lot of terms get thrown around that actually can represent large projects in themselves.

This book consolidates information on all the challenges (and their solutions) involved in implementing data mesh on Microsoft Azure, going from understanding data mesh terminology and mapping it to Microsoft Azure artifacts to all those unknown things that only get mentioned as topics for you to look up for yourself in other data mesh resources. Some of these topics, such as master data management, data quality, and monitoring, can be large, complex systems in themselves.

The driving motivation behind writing this book is to help you understand the concepts of data mesh and to dive into their practical implementation. With this book, you will focus more on the benefits of a decentralized architecture and apply them to your own analytical landscape, rather than getting caught up in all the data mesh terminology.

Who this book is for

This book is for individuals who manage centralized analytical systems built on Microsoft Azure for medium-sized or large corporations and are looking to offer more agility and flexibility to their stakeholders.

This book is also ideal for small companies that currently do not have a well-designed analytical system and want to explore the idea of building a distributed analytical system to handle future growth and agility requirements.

What this book covers

Chapter 1

, Introducing Data Meshes, briefly covers the concepts from Zhamak Dehghani's original whitepaper and book on data mesh.

Chapter 2

, Building a Data Mesh Strategy, guides you in evaluating your company’s current maturity level where analytics is concerned, aligning the company’s strategy with the business strategy, and how data mesh architecture could play a role in that.

Chapter 3

, Deploying Data Mesh Using the Azure Cloud-Scale Analytics Framework, covers Microsoft’s own cloud-scale analytics framework for implementing data mesh.

Chapter 4

, Building a Data Mesh Governance Framework Using Microsoft Azure Services, talks about how the key to a successful data mesh implementation is managing federated governance. This chapter will cover all the aspects of data mesh governance and align it with Microsoft Azure services that can be used to implement it.

Chapter 5

, Security Architecture for Data Meshes, covers how with distributed data comes security challenges. Chapter 4

discusses network security. In this chapter, we will discuss various aspects of data security, such as access control and retention.

Chapter 6

, Automating Deployment through Azure Resource Manager and Azure DevOps, looks at how with distributed data and analytics comes distributed environments and products. The key to efficiently managing your environment is automation. This chapter walks you through all the aspects of automating the deployment and management of data mesh.

Chapter 7

, Building a Self-Service Portal for Common Data Mesh Operations, explores how data mesh promotes agility and innovation by democratizing data and analytical technologies. One of the ways to empower data mesh users is to give them tools to discover data and deployment environments. A common practice is to build a self-service data mesh portal. This chapter provides guidance on how to design and build a self-service portal.

Chapter 8

, How to Design, Build, and Manage Data Contracts, looks at how data mesh federates data ownership. Each team is responsible for the quality and reliability of their own data. In such a scenario, how do you build trust? This chapter discusses the formal method and process of maintaining data contracts and SLAs that help build trust and increase the reliability of data mesh.

Chapter 9

, Data Quality Management, explores how, as data mesh grows, data products become dependent on each other for their outcomes. Some of these products deliver key analytics that is critical to business operations. The bad data quality of one data product could impact multiple products. This chapter showcases how to build/buy an enterprise-class data quality management system.

Chapter 10

, Master Data Management, looks at Master Data Management (MDM), which provides a unified, consistent view of critical data entities across the organization; this is essential for data mesh’s principle of domain-oriented decentralized data ownership and architecture. In this chapter, we will look at buy-and-build options for MDM for data mesh.

Chapter 11

, Monitoring and Data Observability, covers monitoring and data observability, which are crucial for data mesh as they enable real-time insights into the health, performance, and reliability of data across decentralized domains. It is also one of the most challenging features to implement. It involves monitoring data products and data. In this chapter, we will design a Data Mesh Operations Center (DMOC) to consolidate all the monitoring aspects into one pane of glass.

Chapter 12

, Monitoring Data Mesh Costs and Building a Cross-Charging Model, covers how analytical systems are typically cost centers. They are investments, and there are many ways to manage and distribute costs. This chapter looks at various cost models, systems of monitoring costs, and ways of distributing the costs of shared and individual components.

Chapter 13

, Understanding Data-Sharing Topologies in a Data Mesh, looks at how one of the features of data mesh is to minimize the movement of data across the enterprise. It introduces the concept of in-place sharing. However, in-place sharing has its limitations and challenges. This chapter discusses various data-sharing topologies and describes the different scenarios for using each topology.

Chapter 14

, Advanced Analytics Using Azure Machine Learning, Databricks, and the Lakehouse Architecture, is a reference chapter that describes one of the most commonly used architectures for advanced analytics: the lakehouse architecture. The lakehouse architecture combines the scalable storage capabilities of a data lake with the data management and ACID transaction features of a data warehouse, enabling both analytical and transactional workloads on the same platform.

Chapter 15

, Big Data Analytics Using Azure Synapse Analytics, covers how big data processing is a common scenario in most companies today. This reference chapter discusses a possible architecture with Azure Synapse Analytics.

Chapter 16

, Event-Driven Analytics Using Azure Event Hubs, Azure Stream Analytics, and Azure Machine Learning, looks at how certain areas, such as social media data analysis, logistics, and supply chain, require the real-time or near-real-time analysis of data. This kind of data processing needs different kinds of services and storage. This chapter discusses these event processing components and how to lay them out in a real-time analytics architecture.

Chapter 17

, AI Using Azure Cognitive Services and Azure OpenAI, looks at how AI and machine learning have very different needs when it comes to data processing. They need quick cycles of training and re-training as data and models drift with time. Large language models bring in concepts such as prompt engineering and chaining. This chapter describes modern architectures for how to build Azure Cognitive Services- and Azure OpenAI-based models for natural-language-based interactions with your corporate data.

To get the most out of this book

While having read the original data mesh materials by Zhamak Dehghani would definitely be an advantage, it’s not a must. This book provides documentation references for all the Microsoft Azure services mentioned, but some working knowledge of Microsoft Azure will help you save time reading the docs.

For installation and setup of the preceding tools and platforms, please see the following references:

Installing PowerShell: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.4

Installing Azure Command Line Interface: https://learn.microsoft.com/en-us/cli/azure/install-azure-cl

Installing SQL Server Management Studio: https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver16

Setting up SQL Server Management Studio to query Azure SQL Database: https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-ssms?view=azuresql

Installing Python: https://www.python.org/downloads/release/python-3110/

Profisee SaaS Enterprise Data Management: "https://profisee.com/#

Note that the format of Chapters 14, 15, 16, and 17 is different from those of the previous chapters. That is because these chapters are architectural references. The aim of these chapters is to provide guidance on how to set up analytics for a given workload. You might also observe portions of text being repeated across those chapters. This is also by design. At a later point, you might want to refer to a specific reference chapter directly. In order to make sure you have everything you need in those four chapters, we repeat some of the text in them. Each reference chapter is designed to be a quick read and lets you explore all the components of the architecture using the reference links provided.

If you are using the digital version of this book, we advise you to type the code yourself. Doing so will help you avoid any potential errors related to the copying and pasting of code.

Access the GitHub repository

The GitHub repository functions as a valuable resource for future reference, enabling you to report any issues. Furthermore, the author can upload extra updates and examples to the repository, providing ongoing support. Access the repository on GitHub at https://github.com/PacktPublishing/Engineering-Data-Mesh-in-Azure-Cloud

.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/

. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: It sets a mask on Column01 as a default number mask, from the sixth digit to the fourteenth digit.

A block of code is set as follows:

1# Grant access to individual user at a Subscription Level

2  function GrantAccessAtSubscription ($userID, $roleDef, $subScope) {

3      New-AzRoleAssignment -SignInName $userID `

4      -RoleDefinitionName $roleDef `

5      -Scope $subScope

6}

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: Each Azure service comes with its own built-in roles. Azure Data Lake comes with three built-in roles: Reader, Contributor, and Owner.

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected]

and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata

and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected]

with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com

.

Share your thoughts

Once you’ve read Engineering Data Mesh in Azure Cloud, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page

for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781805120780

2. Submit your proof of purchase

3. That’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Rolling Out the Data Mesh in the Azure Cloud

Part 1 starts with the theory of the data mesh architecture as described by Zhamak Dehghani in her original whitepaper (https://www.thoughtworks.com/insights/whitepapers/the-data-mesh-shift

) and maps it to Microsoft Azure’s Well-Architected Framework, Cloud Adoption Framework, and cloud-scale analytics framework. Crossing this chasm is difficult for companies. This section will make it easier to understand the theory and apply it to your Microsoft Azure-based analytical systems. Whether you already have an existing central analytical system that you wish to migrate to a data mesh architecture or you are building an analytical system from the ground up, this part of the book will help you pave the way forward to adopt the data mesh architecture on Microsoft Azure.

This part has the following chapters:

Chapter 1

, Introducing Data Meshes

Chapter 2

, Building a Data Mesh Strategy

Chapter 3

, Deploying a Data Mesh Using the Azure Cloud-Scale Analytics Framework

Chapter 4

, Building a Data Mesh Governance Framework Using Microsoft Azure Services

Chapter 5

, Security Architecture for Data Meshes

Chapter 6

, Automating Deployment through Azure Resource Manager and Azure DevOps

Chapter 7

, Building a Self-Service Portal for Common Data Mesh Operations

1

Introducing Data Meshes

Before we start designing and implementing a data mesh architecture, it is important to understand Why consider a data mesh? This chapter briefly walks through the history of business intelligence (BI) and analytics. We will go through the events and transitions of how analytics has evolved over the last few decades and the current challenges that make a data mesh architecture an alternative to traditional centralized analytical systems.

In this chapter, we’re going to cover the following main topics:

Exploring the evolution of modern data analytics

Discovering the challenges of modern-day enterprises

Data as a product (DaaP)

Data domains

The data mesh solution

Exploring the evolution of