Data in Azure

Memi Lavi
www.memilavi.com
Data in Azure

• Azure provides many data solutions as cloud services

• Relational databases, NoSQL databases, object stores

• Fully managed services

• Can be part of Azure app or completely independent

• Various pricing models

• Always better than unmanaged solutions

Major Database Features

• What to look for when selecting a database?

Security Network isolation, Encryption

Backup Backup types, Retention Period

Availability SLA, Replication, DR

Database on VM

• Azure VM can be setup with database software

• There are ready-made VMs in the marketplace

Database on VM

• Pros of Database on VM:

• Full flexibility

• Full control
Database on VM

• Cons of Database on VM:

• You have to take care of everything:
• SLA
• Updates
• Availability
• Security
• Backups
• And more…
Database on VM

• In this section we’ll discuss only managed databases

Azure SQL

• Managed SQL Server on Azure

• Works like any other SQL Server using the same tools

• Great compatibility with on-prem SQL Server

• Depends on the exact Azure SQL Flavor

• Offers built-in security, backups, availability and more

• Flexible pricing models

Azure SQL Flavors

Azure SQL Database

Elastic Pool

Managed Instance
Azure SQL Database

• Managed SQL Server on Azure

• Single database on a single server

• Automatic backups, updates, scaling

• Good compatibility with on-prem SQL Server

• Not all features are supported

Azure SQL Database

• Security:

• IP firewall rules

• Service Endpoints

• SQL & Azure AD Authentication

• Secure communication (TLS)

• Data encrypted by default (TDE)

Azure SQL Database

• Backup:

Full Every week

Differential Every 12-24 hrs

Transaction Log Every 5-10 mins

Azure SQL Database

• Retention Period:

• Regular backup: 7-35 days (default is 7)

• Long term backup: up to 10 years

Azure SQL Database

• Availability:

• Backup is stored in a geo-redundant storage

• Active geo-replication

• SLA: 99.9% - 99.995%, depends on tier and redundancy

Azure SQL Database

• Compute Tiers:

Provisioned Serverless

• Pay for allocated resources • Pay for actual use – vCore + RAM /
second
regardless of actual use
• Automatically paused when inactive (pay
• Can be reserved just for storage)
• Slight delay when warming up
• Can’t be reserved
Elastic Pool

• Based on Azure SQL

• Allows storing multiple databases on single server

• Great for databases with low average utilization and infrequent

spikes
Elastic Pool
Elastic Pool
Elastic Pool

• Cost effective

• Purchase the compute resources you need, not the database

Managed Instance

• Closer to the on-prem SQL Server

• Near 100% compatible with on-prem SQL

• Can be deployed to VNet

• Business model close to on-prem one

Managed Instance

• Main differences:

• No active geo-replication • No availability zone

• SLA: 99.99% • No serverless tier

• Supports built-in functions • No Hyperscale

• Runs CLR code

• No auto scaling & tuning

Azure SQL Pricing
Quite complicated…
 Instance Instance
Flavor Service Tier
Type (vCores)

Instance
Pools
General 8…80
Purpose
Single
Managed
Instance
Instance Business
Critical
Azure SQL
 Purchase Instance
Flavor Type Model
Service Tier Compute Tier (vCores / DTUs)

Basic
vCores /
Standard DTUs
Managed DTU* Premium
Instance Single Serverless
vCores General Purpose
Database
Azure SQL Business Critical
DTU* Provisioned
Elastic Pool
Hyperscale
vCores

1 vCore =~ 100 DTU

 Purchase Instance
Flavor Type Model
Service Tier (vCores / DTUs)

Managed DTU*
Instance Basic
Single
vCores
Database Standard
Azure SQL
vCores /
DTU* Premium
Elastic Pool DTUs
vCores General Purpose

Business Critical

1 vCore =~ 100 DTU

Highly
inaccurate
Which Azure SQL to Choose?

Are you migrating

Managed Instance
an on-prem SQL?

Do you need multiple,

mostly low-utilization DBs? Elastic Pool

All other cases Azure SQL

Cosmos DB

• Fully managed NoSQL database

• Amazing performance - <10ms for 99% of operations

• Globally distributed

• Fully automatic management – updates, scaling, fixes etc.

• Multiple APIS:

• SQL, Mongo, Gremlin, Azure Table, Cassandra (per account)

Cosmos DB

• Hierarchical:

JSON
Document
Cosmos DB Availability

• Can be distributed across many regions (configurable)

• API automatically picks the closest one

• When using write replication SLA is 99.999% (!)

• Managed automatically, no code changes required

Cosmos DB Backup

• Full backup every 1-24 hours (default is 4)

• Retention period 20-30 days (default is 30)

Cosmos DB Security

• IP firewall rules

• Service Endpoints

• Private Endpoints

• Azure AD Authentication

• Secure communication (TLS)

• Data encrypted by default

Cosmos DB Partitions

• Data items are divided to partitions

• Logical group of items based on a specific property

• Example: In a cars database, the Model can be a partition property

Partition #1 Partition #2 Partition #3 Partition #n

Model: Model: Model: Model:

Mercedes Chevrolet Alfa Romeo Toyota
. . . . . . . .
Cosmos DB Partitions

• Partitions are the basic scale unit in Cosmos DB

• Distribution and scale are per partitions

• Make sure items are divided as evenly as possible

• It’s extremely important to select the right partition property

• Cannot be modified
Recording of L7S5 from the

Architecture course
Cosmos DB Consistency Levels

• Traditionally:

• Relational DB – Strong consistency: Call returns only after

successful commit in all replicas (High availability)

High Availability ≠ Low Latency
• NoSQL DB – Eventual consistency: Call returns immediately,

commit in replicas happens later (Low latency)

Cosmos DB Consistency Levels

• Cosmos DB offer five consistency levels:

• Strong (<= As in regular relational DB)

• Bounded Staleness

• Session

• Consistent Prefix

• Eventual (<= As in regular NoSQL DB)

Cosmos DB Consistency Levels

• The basic question with consistency is:

• If region X updates an item, and region Y reads this item, which

version will it get?

Strong

• Region Y will get the last version of the item updated in region X

• Used for mission critical data

Region X
Writes

Region Y
Reads

Timeline
Bounded Staleness
• Region Y will lag behind region X by K versions or T time
• Keeps the order of the versions
• Used for low write latency and when order is important

Region X
Writes

K=2

Region Y
Reads

Timeline
Session
• In a client session – Strong consistency
• Other clients – Consistent Prefix (sometimes Eventual)
Consistent Prefix
• Keeps the order of the versions
• No guarantee of the lag size (as opposed to Bounded Context)
• Used for low write latency and when reads are infrequent

Region X
Writes

Region Y
Reads

Timeline
Eventual
• No order guarantee
• No guarantee of the lag size (as opposed to Bounded Context)
• Used for count of Retweets, Likes, etc.

Region X
Writes

Region Y
Reads

Timeline
Cosmos DB Consistency Levels

• Configured at the account level

• Can be relaxed on the request level

Cosmos Pricing

• Based on RU/s

• Request Unit per Second

• 1 RU = Read item of size 1KB

• Read = Get the item by its ID, not by query

Cosmos Pricing

• 1 RU/s = Read 1 item of 1KB in 1 sec

• 400 RU/s = Read 400 items of 1KB in 1 sec

• Update, delete, insert, query – more than 1 RU

• You can see the actual RU consumed in the response header of the

results
Cosmos Pricing

• Pricing based on:

• Operations type: Provisioned, Auto Scale, Serverless

• Write Regions

• No. of provisioned RU/s

Cosmos Pricing

• Database Operations:

• Provisioned – Predefined number of RU/s, can be changed

manually later. Offers reserved capacity up to 65% discount

• Auto Scale – Set the maximum RU/s, Cosmos scales up to this

number. Good for unpredictable loads

• Serverless – Pay for what you use. In Preview, no SLA

Azure MySQL

• Managed MySQL on Azure

• Works like any other MySQL database using the same tools

• Great compatibility with on-prem MySQL database

• Offers built-in security, backups, availability and more

Azure MySQL

• Security:
• IP firewall rules
• Service Endpoints
• Private Endpoints
• Regular & Azure AD Authentication
• Secure communication (TLS)
• Data encrypted by default
Azure MySQL

• Backup:
• Depends on Service Tier:
• Basic – Full backup: daily
• General Purpose up to 4GB:
• Full backup: once a week
• Differential backup: twice a day
• Transaction log backup: every 5 minutes
Azure MySQL

• Backup:
• Depends on Service Tier:
• General Purpose up to 16GB:
• Full backup: Once created
• Differential backup: once a day
• Transaction log backup: every 5 minutes
Azure MySQL

• Retention Period:

• 7-35 days (default is 7)

• No native long term retention support

Azure MySQL

• Availability:

• Backup is stored in a geo-redundant storage

• In General Purpose and Memory Optimized tiers

• SLA: 99.99%
Azure MySQL Pricing

• Pricing based on:

• Tier:

• Basic – Require light compute and I/O performance (ie. dev)

• General Purpose – Most business workloads

• Memory Optimized – Require in-memory performance

• Compute (no. of vCores)

Azure MySQL Pricing

• Flexible Server deployment – currently in preview, not

recommended

• Reservations exist
Azure PostgreSQL

• Managed PostgreSQL on Azure

• Works like any other PostgreSQL database using the same tools

• Great compatibility with on-prem PostgreSQL database

• Includes Hyperscale deployment

• Offers built-in security, backups, availability and more

Azure PostgreSQL

• Security:
• IP firewall rules
• Service Endpoints
• Private Endpoints
• Regular & Azure AD Authentication
• Secure communication (TLS)
• Data encrypted by default
Azure PostgreSQL

• Backup:
• Depends on storage size:
• up to 4GB:
• Full backup: once a week
• Differential backup: twice a day
• Transaction log backup: every 5 minutes
Azure PostgreSQL

• Backup:
• Depends on storage size:
• up to 16GB:
• Full backup: Once created
• Differential backup: thrice a day
• Transaction log backup: every 5 minutes
Azure PostgreSQL

• Retention Period:

• 7-35 days (default is 7)

• No native long term retention support

Azure PostgreSQL

• Availability:

• Backup is stored in a geo-redundant storage

• In General Purpose and Memory Optimized tiers

• SLA: 99.99%
Azure Storage

• Object store

• Massively scalable

• Accessible via HTTP or HTTPS

• Client libraries for almost every language

• Durable and highly available

Azure Storage Types

Blobs Object Store This one we like ☺

File shares for cloud and Related more to

Files compute, quite low
on-prem deployments level

We’ll discuss this in the

Queues Queues (duh…) Messaging section

Similar to CosmosDB, but less

Tables NoSQL data store optimized for performance and
availability (cheaper, though…)

Disks Storage volumes for Managed by the VM, nothing

to deal with here
Azure VMs
Azure Blobs Storage
• Object store (Blob = Binary Large Object)
• Great for files, videos, documents, large texts etc.
• Up to 4.77TB per file (!), 190TB in preview (!!)
• Extremely cost effective
• Massively scalable
• Great availability options
• Extremely easy to use
• Usually used in conjunction with SQL / NoSQL database
Azure Blobs Storage

• Security:
• IP firewall rules
• Service Endpoints
• Private Endpoints
• Shared Access Signatures
• Access Keys & Azure AD Authentication
• Secure communication (TLS)
• Data encrypted by default
Azure Blobs Storage

• Structure:
Azure Blobs Storage Redundancy

• 6 options:
LRS ZRS GRS
Locally Redundant Storage Zone Redundant Storage Geo Redundant Storage
Data is synchronously copied 3 Data is synchronously copied to Data is synchronously copied 3 times
times within the same zone 3 zones in the Region within the same zone, and then copied
asynchronously to paired Region.
Data in the secondary Region is
accessible only after Failover process

GZRS RA-GRS RA-GZRS

Geo-Zone Redundant Storage Read Access-Geo Redundant Storage Read Access-Geo-Zone Redundant
Data is synchronously copied to 3 Data is synchronously copied 3 times within Storage
zones in the Region, and then the same zone, and then copied Data is synchronously copied to 3 zones
copied asynchronously to paired asynchronously to paired Region. within the same Region, and then copied
Region. There’s a read-access to the data in the asynchronously to paired Region.
Data in the secondary Region is secondary Region There’s a read-access to the data in the
accessible only after Failover secondary Region
process
Azure Blobs Storage Failover

• Normal circumstances:
Azure Blobs Storage Failover

• Primary Region fails:

Azure Blobs Storage Failover

• Failover to the secondary Region:

Azure Blobs Storage Failover

• Can be initiated via the:

• Portal

• Azure CLI

• PowerShell
Azure Blobs Storage Tiers

• Blobs are uploaded to one of three tiers:

Hot Cool Archive
- Data that’s accessed frequently - Data that’s accessed infrequently - Data for archival
- Best SLA (99.9%) - Slightly lower SLA (99%) - Stored offline, no SLA
- Highest storage costs - Lower storage costs - Can take hours to retrieve
- Lowest access costs - Higher access costs - Lowest storage costs
- Examples: - Must be stored for at least 30 - Highest access costs
- Photos to display days (or early deletion fees - Must be stored for at least 180
- Documents to show applied) days (or early deletion fees
- Examples: applied)
- Short term backup
- Data for future processing
Azure Blobs Storage Tiers

Hot Cool Archive

Azure Blobs Storage Tiers

• Retrieval time is the same in Hot and Cool tiers

• Archive tier does not support ZRS, GRS and RA-GRS redundancy

• Using RA-G(Z)RS, SLA improves to 99.99% (Hot) and 99.9% (Cool)

• Tier is set at account level, can be modified per blob

• Moving between tiers can be automated by lifecycle rules

Azure Blobs Storage Pricing

• Based on:

• Redundancy option

• Access tier

• Capacity
Azure Redis

• Managed Redis on Azure

• Provides lightning-fast in-memory, distributed cache

• Great for short-lived, frequently accessed data

• ie. shopping cart, stock quotes

• Fully compatible with OSS Redis (community edition) and

Enterprise Redis – depends on service tiers

Azure Redis

• Security:
• IP firewall rules
• Service Endpoints
• Private Endpoints
• Secure communication (TLS)
Azure Redis Service Tiers

Basic Based on a single VM, no SLA, no distribution.

Good for dev/test

Standard Based on two VMs, replicated.

SLA – Up to 99.9%

Premium High-performance, better throughput, lower latency.

SLA – Up to 99.95%

Enterprise Based on Redis Enterprise, offers additional features

(RediSearch, RedisBloom and more). SLA – Up to 99.99%

Enterprise Flash Uses non-volatile memory. Reduces storage cost.

SLA – Up to 99.99%
Azure Redis Pricing

• Based on:

• Tier

• Memory
 Cloud Architecture
Service Peering
Endpoint
App Gateway

NSG NSG

Function App AKS App Service VM Peering VM

Order Processing Cart App Inventory App Weather API Catalog App

NSG

ACR Redis
Cosmos DB Cart Docker
Azure SQL
Are we done?

• Nope ☺
• The Orders Function is publicly available and synchronous
• The inventory page is open for everyone
• We don’t really know how the app is functioning
• The website is not redundant – what happens if the whole region
goes down?
Selecting Data Store Solution

Data Type Used For… Examples Options in Azure

Relational Structured data Items in store, Azure SQL, MySQL,
demographic data PostgreSQL
NoSQL Semi-structured Reviews, Log Cosmos DB (with SQL,
data records, when Mongo, Azure Table
flexibility is API)
required
Graph Data representing Family tree Cosmos DB (with
relationships Gremlin API)
Blob Files, videos, docs Items’ photos Azure Blob Storage