Data Architectures

in Azure
for Analytics & Big Data
October 20, 2018

Melissa Coates

Solution Architect, BlueGranite Blog: www.sqlchick.com

Microsoft Data Platform MVP Twitter: @sqlchick
Data Architecture
A set of
rules, policies, standards, & models
that govern and define
the type of data collected
& how it is
used, stored, managed, & integrated
within an organization
& its database systems
Source: Technopedia
Data Architecture Components

Source: https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/
Azure
Source: http://azureplatform.azurewebsites.net/

A public cloud
computing
platform and
infrastructure for
building,
deploying, and
managing MSFT-
specific and third
party software and
services through a
global network of
Microsoft-
managed
datacenters
Azure Technologies
Data Storage: Relational Databases
IaaS PaaS
Infrastructure as a Service Platform as a Service
SMP MPP
Symmetric Multi-Processing Massively Parallel Processing

Managed Database

Managed Instance
Relational database of your Azure SQL
Azure SQL Database
choice in a virtual machine Data Warehouse

Azure Azure Azure

MPP
Massively Parallel Processing Database Database Database
for for for
MySQL PostgreSQL MariaDB
Data Storage: Big Data
PaaS
Platform as a Service

Object Storage (Flat) Hierarchical Storage

Azure Data Lake

Azure Blob
Store (Gen1)
Storage

Multi-Modal

Azure Data Lake

Storage (Gen2)
(PREVIEW)
Data Storage: NoSQL
PaaS
Platform as a Service

Multi-Model

HDInsight
HBase Azure
Cluster Table Storage
Azure
CosmosDB

Key Column JSON Graph

Value Family Documents
Data Storage: Analytical & OLAP
IaaS PaaS
Infrastructure as a Service Platform as a Service

SQL Server Azure Power BI

Analysis Services Analysis Services
Compute: Big Data
IaaS PaaS SaaS
Infrastructure as a Service Platform as a Service Software as a Service

HDInsight HDInsight Azure

in a VM Spark Cluster Databricks Azure Data Lake
Analytics

HDInsight HDInsight
Interactive Hadoop Cluster
Query Cluster
(Hive LLAP)
Compute: Streaming & Event Processing
PaaS
Platform as a Service

HDInsight HDInsight HDInsight

Kafka Cluster Storm Cluster Spark Streaming
Cluster

Azure Azure Azure

IoT Hub Event Hub Stream
Analytics
Compute: Data Integration
IaaS PaaS
Serverless
Infrastructure as a Service Platform as a Service

Tool of your choice Azure Data Azure Azure

in a virtual machine Factory Databricks Functions

HDInsight: Azure
Spark, Hive, Pig, Automation
Scoop, Oozie
 Reference Architectures

Following are examples only!

There are many variations & opportunities to
exchange one Azure service for another.
Small/Medium Data Warehousing
Source Data Multi-Structured Data

Azure Blob
Storage
Reporting &
Analysis Tools

DW: Structured Data Semantic Layer

Power BI
Azure
Azure Analysis Services
SQL
Database Excel
Enterprise Data Warehousing and BI
Multi-Structured Data
Data Mart(s)
Azure Data
Lake Storage

Azure
DW: Structured Data SQL
Azure SQL Database
Data Warehouse
Power BI
Semantic Layer

Azure Excel
Analysis Services
Data Science and Artificial Intelligence
Multi-Structured Data Data Science and AI
Azure Data
Lake Storage
Azure Azure Machine Azure Cognitive
Databricks Learning HDInsight Services
DW: Structured Data
Azure SQL
Data Warehouse Data Mart(s)

Power BI

Azure
SQL
Database Excel
Unified Data Science & Data Engineering
Data Lake: Multi-Structured Data Structured Data
Azure Data Azure SQL
Scheduled
Lake Storage Notebook Database
Job

Raw Data
Operationalized
Curated Analytics
Data
Azure
Databricks
Exploratory
Data Science Analytics
Sandbox
Big Data Interactive Querying (SQL on Hadoop)
Hive Metastore
Data Lake
Azure Data Azure SQL
Lake Storage Database

Hive Data Warehouse

HDInsight
Interactive
HiveQL
Query Cluster
(Hive LLAP)
Big Data Batch Processing Big Data Job Processing
Data Lake: Multi-Structured Data
Azure Data Lake Analytics
Azure Data U-SQL
U-SQL Job Processing
Lake Storage Extensions
Job 1 Job 2

Python

ADLA Catalog
Database

Tables Procedures
External Cognitive
Views Functions Data Services
Sources
Schemas Assemblies
Azure
SQL DB
SQL Server Azure
in Azure VM SQL DW
IoT + Batch Data (Lambda Architecture)
Speed Layer Serving Layer
Streaming
Dashboard

Azure Azure Stream Power BI

Event Hub Analytics

Batch Layer
Azure
Analysis Services
Azure
Data Lake
Storage
Azure
SQL Data
Power BI Excel
Warehouse
Operational BI (Embedded BI)
Published Reports Embedded Visuals

Power BI Custom
Service Application

Premium
Source Data Data Model + Reports Capacity
REST
Azure SQL Power BI App API
Database Desktop Workspace calls
Web Application

Web Page

Azure SQL Web Cache

Database App

Diagnostics Backups

App Service
Plan
Storage Storage
Account Account
Wrap-Up
More Info
Azure Solution Architectures:
https://azure.microsoft.com/en-us/solutions/architecture/
More Info
Azure Data Architecture Guide:
https://docs.microsoft.com/en-us/azure/architecture/data-guide/
Thanks!

Download latest version of slides:

SQLChick.com > Presentations & Downloads page

Creative Commons Attribute to me as original No usage of this No derivatives or

License 3.0 author if you material for changes to this material
share this material commercial purposes