Azure Storage building blocks

Azure storage building blocks

Internet

Azure Infrastructure Virtual Network

Content Delivery Data Lake
Network Storage
Storage account

Disk storage On-premise

data centre

Azure
Azure Virtual OS Disk Data Disk Storsimple
machine
Backup Restore Azure Queue Azure Blob

Azure Site
Queues & Containers
Messages & Blobs OS Images recovery
Azure Backup Managed disks
Azure Virtual
Monitoring & Diagnostics
Azure Table Archive Azure File machines Azure Data
Box

Tables & Entities Containers Files & Folders

& Files
Log
Monitor Alerts
analytics
Azure Storage services – Storage account
Azure storage account

• An Azure storage account is a secure account that gives you access to services in Azure Storage.
Your storage account provides the unique namespace for your storage resources.

• A storage account name should be unique across all existing storage account names in Azure
Types of Storage accounts

Azure Storage provides three types of storage accounts. Each type supports different features and
has its own pricing model.

Storage Supported Supported

account Supported performance access Replication Deployment
type services tiers tiers options model Encryption

General- Blob, File, Standard, Hot, Cool, LRS, ZRS, Resource Encrypted
purpose V2 Queue, Table, Premium Archive GRS, RA- Manager
and Disk GRS

General- Blob, File, Standard, N/A LRS, GRS, Resource Encrypted

purpose V1 Queue, Table, Premium RA-GRS Manager,
and Disk Classic

Blob Blob (block Standard Hot, Cool, LRS, GRS, Resource Encrypted
storage blobs and Archive RA-GRS Manager
append blobs
only)
Types of performance tiers

• Standard performance tiers are backed by magnetic drives and provides low cost per GB. They
are best for applications that are best for bulk storage or infrequently accessed data

• Premium storage performance are backed by solid state drives and offers consistency and low
latency performance. They can only be used with Azure virtual machine disks, and are best for I/O
intensive workload such as databases.
Storage account access tiers

Azure storage offers different storage tiers which allow you to store Blob object data in the most cost-
effective manner

Premium storage (preview) provides high performance hardware for data that is accessed
frequently.

Hot storage: is optimized for storing data that is accessed frequently.

Cool storage is optimized for storing data that is infrequently accessed and stored for at least 30
days.

Archive storage is optimized for storing data that is rarely accessed and stored for at least 180 days
with flexible latency requirements (on the order of hours).
Azure storage replication
Azure Storage replication copies your data so that it is protected from planned and unplanned events
ranging from transient hardware failures, network or power outages, massive natural disasters, and so
on.
Scenario LRS ZRS GRS RA-GRS

Node unavailability within a data Yes Yes Yes Yes

center

An entire data center (zonal or No Yes Yes Yes

non-zonal) becomes unavailable

A region-wide outage No No Yes Yes

Read access to your data (in a No No No Yes

remote, geo-replicated region) in
the event of region-wide
unavailability

Designed to provide __ durability at least 99.999999999% (11 9's) at least 99.9999999999% (12 9's) at least 99.99999999999999% (16 at least 99.99999999999999% (16
of objects over a given year 9's) 9's)

Supported storage account types GPv2, GPv1, Blob GPv2 GPv2, GPv1, Blob GPv2, GPv1, Blob

Availability SLA for read requests At least 99.9% (99% for cool access At least 99.9% (99% for cool access At least 99.9% (99% for cool access At least 99.99% (99.9% for Cool
tier) tier) tier) Access Tier)

Availability SLA for write requests At least 99.9% (99% for cool access At least 99.9% (99% for cool access At least 99.9% (99% for cool access At least 99.9% (99% for cool access
tier) tier) tier) tier)
Storage account endpoints

Every object that you store in Azure Storage has an address that includes your unique account name.
The combination of the account name and the Azure Storage service endpoint forms the endpoints for
your storage account.

For example, if your general-purpose storage account is named mystorageaccount, then the default
endpoints for that account are:

• Blob storage: http://mystorageaccount.blob.core.windows.net

• Table storage: http://mystorageaccount.table.core.windows.net
• Queue storage: http://mystorageaccount.queue.core.windows.net
• Azure Files: http://mystorageaccount.file.core.windows.net
Azure Storage services – Blob storage
Azure blob storage overview

Azure Blob storage is Microsoft's object storage solution for the cloud. Blob storage is optimized for
storing massive amounts of unstructured data, such as text or binary data.

Blob storage is ideal for:

• Serving images or documents directly to a browser.

• Storing files for distributed access.
• Streaming video and audio.
• Writing to log files.
• Storing data for backup and restore, disaster recovery, and archiving.
• Storing data for analysis by an on-premises or Azure-hosted service
Blob storage concepts

Storage Container Blob

Account
videos video.mp4
rudraapps
images Image1.jpg
Image1.jpg

• Every blob is placed in a container. Security policies can be assigned to containers which will be
cascaded to all objects under that container.

• A storage account can contain unlimited number of containers and each container can contain
unlimited number of blobs up to the limit of storage account size limit ( 2PB for US and Europe
and 500TB for all other regions including UK)

• You can refer to an object stored in the blob with the below format
• http://mystorageaccount.blob.core.windows.net/mycontainer/myblob

• The Blob service is based on a flat storage scheme, not a hierarchical scheme. However, you
may specify a character or string delimiter within a blob name to create a virtual hierarchy.
Blob types

Azure Storage offers three types of blobs—block blobs, append blobs, and page blobs (used for VHD
files).

• Block blobs store text and binary data, up to about 4.7 TB. Block blobs are made up of blocks of
data that can be managed individually.

• Append blobs are made up of blocks like block blobs, but are optimized for append operations.
Append blobs are ideal for scenarios such as logging data from virtual machines.

• Page blobs store random access files up to 8 TB in size. Page blobs store the VHD files that back
VMs.
Naming and Referencing
Container Names

A container name must be a valid DNS name, conforming to the following naming rules:

• Container names must start with a letter or number, and can contain only letters, numbers, and
the dash (-) character.
• All letters in a container name must be lowercase.
• Container names must be from 3 through 63 characters long.

Blob Names

• A blob name can contain any combination of characters.

• A blob name must be at least one character long and cannot be more than 1,024 characters long,
for blobs in Azure Storage.
• The Azure Storage emulator supports blob names up to 256 characters long. Blob names are
case-sensitive.
• Reserved URL characters must be properly escaped.
Metadata & Snapshots

Metadata Names

Metadata for a container or blob resource is stored as name-value pairs associated with the resource.
Metadata names must adhere to the naming rules for C# identifiers.

Blob Snapshots

A snapshot is a read-only version of a blob stored as it was at the time the snapshot was created. You
can use snapshots to create a backup or checkpoint of a blob. A snapshot blob name includes the
base blob URI plus a date-time value that indicates when the snapshot was created.

https://myaccount.blob.core.windows.net/mycontainer/myblob?snapshot=<DateTime>
Storage account & Blob service configuration
Storage firewall and Virtual networks
Internet

Azure Infrastructure

Storage account

Storage
firewall

Allowed Virtual
networks
Virtual Network

Allowed IP Ranges

Internet

Exceptions

Azure services
Custom domains

• You can configure a custom domain for accessing blob data in your Azure storage account. The
default endpoint for Azure Blob storage is <storage-account-name>.blob.core.windows.net.

• Limitations

• Azure Storage does not yet natively support HTTPS with custom domains. You can
currently use Azure CDN to access blobs by using custom domains over HTTPS.
• Storage accounts currently support only one custom domain name per account.

Resource type Default URL Custom domain URL

Storage account http://mystorageaccount.blob.core.windows.net http://www.rudra9.com

Blob http://mystorageaccount.blob.core.windows.net/myco http://www.rudra9.com/mycontainer

ntainer/myblob /myblob
Content delivery network
• The Azure Content delivery Network (CDN) caches static content at strategically placed locations
to provide maximum throughput for delivering content to users.

CDN location

North American
user

CDN location

North American
user
Blob in
Australian region
Other configuration areas
Azure Storage services – Storage security
Azure storage security overview

Securing your storage account Manage plane security

Securing access to your data Data Plane security

Encryption in transit Encryption in transit

Encryption at rest Encryption at rest

Cross Origin Resource Sharing CORS

Management plane security

• Management plane refers to the operations that effect the storage account itself.

• Role Based Access Control

• Each Azure subscription has an Azure Active Directory. Users, groups and applications from
that directory can be granted access to manage resources in the Azure subscription. This is
referred as Role Based Access Control.
• Access is granted by assigning the appropriate RBAC role to users, groups, and applications
at the right level. This level can be subscription, resources group and resources.

• Key points to remember

• When you assign a role, You can control access to operations used to manage the storage
account but not data objects in the account.
• However, you can give access to data objects by giving permission to read storage account
keys.
• Each role has a list of Actions & Not Actions.
• There are some standard roles available. For e.g. Owner, Reader, Contributor etc.
• You can define new custom role by selecting a set of actions from the list of available actions.
Data plane security

• Data plane security refers to the methods used to secure data objects (blobs, queues, tables and
files) within the storage account.

• Three methods for controlling access to data objects

• Using Azure AD to authorize access to containers and queues (Preview). Azure AD provides
advantages over other approaches to authorization, including removing the need to store
secrets in your code.
• Storage account keys
• Shared Access Signatures

• You can allow public access to your blobs by setting the access level for the container that holds
the blob accordingly.
Encryption in Transit

• Transport level Encryption using HTTPS

• Always use HTTPS when using REST APIs or accessing objects in storage.
• If you are using SAS, you can specify that only HTTPS should be used

• Using encryption in transit for Azure file shares

• SMB2.1 do not support encryption so connections are only allowed within the same region.
• SMB3.0 supports encryption and cross region access is allowed

• Client side encryption

• Encrypt the data before being transferred to Azure storage
• When retrieving the data from Azure, data is decrypted after it is received on the client side.
Encryption at rest
• Client side encryption
• Encrypt the data before being transferred to Azure storage
• When retrieving the data from Azure, data is decrypted after it is received on the client side

• Storage Service Encryption (SSE)

• SSE is enabled for all storage accounts and cannot be disabled
• SSE automatically encrypts data in all performance tiers (Standard and Premium), all
deployment models (Azure Resource Manager and Classic), and all of the Azure Storage
services (Blob, Queue, Table, and File).
• You can use either Microsoft-managed keys or your own custom keys.

• Azure Disk Encryption

• Encrypt the OS & data disks used by IaaS Virtual Machine
• You can enable encryption on existing IaaS VMs
• You can use customer provided encryption keys
Important note: Refer to the link in the resource section of this lecture for comparison between above three encryption types
Cross Origin Resource Sharing

• When a web browser running in one domain makes an HTTP request for a resource from a
different domain, this is called a cross-origin HTTP request

• Azure Storage allows you to enable CORS. For each storage account, you can specify domains
that can access the resources in that storage account. For example, enable CORS on the
mystorage.blob.core.windows.net storage account and configure it to allow access to
mywebsite.com

• CORS allows access but does not provide authentication which means you still need to use SAS
to access non-public storage resources.

• By default, CORS is disabled on all services.

Azure Storage services – File storage
Azure file storage

Account Share Directories Files

Documents Doc1.docx
Azure Storage Account

Azure File Storage

File Share 1
log1.txt
Logs

mov1.mp4
File Share 2 Media
Azure file storage concepts
• Storage Account: All access to Azure Storage is done through a storage account.
See Scalability and Performance Targets for details about storage account capacity.

• Share: A File Storage share is an SMB file share in Azure. All directories and files must be
created in a parent share. An account can contain an unlimited number of shares, and a share
can store an unlimited number of files, up to the 5 TiB total capacity of the file share.

• Directory: An optional hierarchy of directories.

• File: A file in the share. A file may be up to 1 TiB in size.

• URL format: For requests to an Azure file share made with the File REST protocol, files are
addressable using the following URL format:

https://<storage account>.file.core.windows.net/<share>/<directory>/directories>/<file>
File storage data access methods

• Azure Files offers two, built-in, convenient data access methods that you can use separately, or in
combination with each other, to access your data:

• Direct cloud access: Any Azure file share can be mounted by Windows, macOS,
and/or Linux with the industry standard Server Message Block (SMB) protocol or via the File REST
API. With SMB, reads and writes to files on the share are made directly on the file share in Azure.
To mount by a VM in Azure, the SMB client in the OS must support at least SMB 2.1. To mount on-
premises, such as on a user's workstation, the SMB client supported by the workstation must
support at least SMB 3.0 (with encryption)

• Azure File Sync: With Azure File Sync, shares can be replicated to Windows Servers on-premises
or in Azure. Your users would access the file share through the Windows Server, such as through
an SMB or NFS share. This is useful for scenarios in which data will be accessed and modified far
away from an Azure datacenter, such as in a branch office scenario.
Data transfer methods

• Azure File Sync: As part of a first sync between an Azure file share (a "Cloud Endpoint") and a
Windows directory namespace (a "Server Endpoint"), Azure File Sync will replicate all data from
the existing file share to Azure Files.

• Azure Import/Export: The Azure Import/Export service allows you to securely transfer large
amounts of data into an Azure file share by shipping hard disk drives to an Azure datacenter.

• Robocopy: Robocopy is a well known copy tool that ships with Windows and Windows Server.
Robocopy may be used to transfer data into Azure Files by mounting the file share locally, and then
using the mounted location as the destination in the Robocopy command.

• AzCopy: AzCopy is a command-line utility designed for copying data to and from Azure Files, as
well as Azure Blob storage, using simple commands with optimal performance. AzCopy is available
for Windows and Linux.
Azure Storage services – Table & Queue Storage
Azure Table storage Overview
Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore
which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for
storing structured, non-relational data.

Common uses of Table storage include:

• Storing TBs of structured data capable of serving web scale applications

• Storing datasets that don't require complex joins, foreign keys, or stored procedures and can be
denormalized for fast access

• Quickly querying data using a clustered index

• Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET
Libraries
Azure Table structure

Account Table Entity

Name = …

Azure Storage Account

Email = …
Customers
Name = …
Email = …

Street = …
Addresses
Postal = …
Table Storage concepts

• Accounts: All access to Azure Storage is done through a storage account. All access to Azure
Cosmos DB is done through a Table API account.

• Table: A table is a collection of entities. Tables don't enforce a schema on entities, which means a
single table can contain entities that have different sets of properties.

• Entity: An entity is a set of properties, similar to a database row. An entity in Azure Storage can
be up to 1MB in size. An entity in Azure Cosmos DB can be up to 2MB in size.

• Properties: A property is a name-value pair. Each entity can include up to 252 properties to store
data. Each entity also has three system properties that specify a partition key, a row key, and a
timestamp. Entities with the same partition key can be queried more quickly, and inserted/updated
in atomic operations. An entity's row key is its unique identifier within a partition.
Table Storage access points

• Azure Table Storage accounts use this format:

http://<storage account>.table.core.windows.net/<table>

• Azure Cosmos DB Table API accounts use this format: http://<storage

account>.table.cosmosdb.azure.com/<table>
Table Storage queries
• Queries retrieve data from tables.

• Because a table only has one index, query performance is usually related to the PartitionKey and
RowKey properties

http://<account>.windows.core.net/registrations(PartitionKey=”2011 New York City

Marathon__Full”,RowKey=”1234__John__M__55”)
Azure Queue storage Overview

• Azure Queue storage is a service for storing large number of messages in the cloud that can be
accessed from anywhere in the world using HTTP and HTTPS.

• A queue contains a set of messages. Queue name must be all lowercase.

• A single queue message can be up to 64KB in size. A message can remain in the queue for
maximum time of 7 days

• URL format : http://<storage account>.queue.core.windows.net/<queue>

• When message retrieved from the queue, it stays invisible for 30 seconds. Message needs to be
explicitly deleted from the queue to avoid getting picked up by another application.
Azure Storage services – Disk Storage
Azure Disk storage Overview
• Virtual machines in Azure use disks as a place to store an operating system, applications, and
data. All Azure virtual machines have at least two disks – a Windows operating system disk and a
temporary disk.

• Both the operating system disk and the image are virtual hard disks (VHDs) stored in an Azure
storage account. The VHDs used in Azure are .vhd files stored as page blobs in a standard or
premium storage account in Azure

• Virtual machines also can have one or more data disks, that are also stored as VHDs.
Storage Account
Disk storage

OS Disk Data Disk

Azure Virtual Unmanaged /

machine Managed disks
Types of Disks

• Standard HDD disks are backed by HDDs, and deliver cost-effective storage. Standard HDD
storage can be replicated locally in one datacenter, or be geo-redundant with primary and
secondary data centers

• Standard SSD disks are designed to address the same kind of workloads as Standard HDD
disks, but offer more consistent performance and reliability than HDD. Best suited for applications
like web servers that do not need high IOPS on disks. Where available, Standard SSD disks are
the recommended deployment option for most workloads.

• Premium SSD disks are backed by SSDs, and delivers high-performance, low-latency disk
support for VMs running I/O-intensive workloads.

A link to comparison of types of disks has been provided in the resources section of this
lecture. It is important that you go through the same.
Unmanaged and Managed disks

• Unmanaged disks are the traditional type of disks that have been used by VMs. With these disks,
you create your own storage account and specify that storage account when you create the disk.
Make sure you don't put too many disks in the same storage account, because you could exceed
the scalability targets of the storage account (20,000 IOPS, for example), resulting in the VMs
being throttled.

• Managed Disks handles the storage account creation/management in the background for you,
and ensures that you do not have to worry about the scalability limits of the storage account. You
simply specify the disk size and the performance tier (Standard/Premium), and Azure creates and
manages the disk for you.

Microsoft recommends that you use managed disks for all new VMs and convert your
previous unmanaged to managed disks.
Disks backup

Azure Backup
Service
Backup
extension

OS Disk Data Disk 1 Data Disk n

Data
Transfer

Snapshot Recovery
Service Vault
Azure Storage services – Monitoring
Storage monitoring overview

Diagnostics & troubleshooting

Client Applications Azure Storage Analytics,
client-side logging in the
Continuous Monitoring

Azure Storage Client Library,

Rich clients Mobile clients Web clients
Client and other third-party tools
logging
can be used to identify,

on demand
diagnose, and troubleshoot
Azure Storage related
Network issues.

Network
logging

Azure Storage services

Server
Blobs Queues File Table logging
Storage Analytics
• Azure Storage Analytics performs logging and provides metrics data for a storage account. You
can use this data to trace requests, analyze usage trends, and diagnose issues with your storage
account.

• By default, Metrics is enabled when you create a storage account. You can enable logging it
using Azure portal, Rest APIs or Client library. Use the Get Blob Service Properties, Get Queue
Service Properties, Get Table Service Properties, and Get File Service Properties operations to
enable Storage Analytics for each service.

• The aggregated data is stored in a well-known blob (for logging) and in well-known tables (for
metrics), which may be accessed using respective service APIs.

• Storage Analytics has a 20 TB limit on the amount of stored data that is independent of the total
limit for your storage account
Storage analytics logging

• Storage Analytics logs detailed information about successful and failed requests to a storage
service. This information can be used to monitor individual requests and to diagnose issues with a
storage service

• Both authenticated and anonymous requests will be logged but at different levels

• All logs are stored in block blobs in a container named $logs, which is automatically created when
Storage Analytics is enabled for a storage account. The $logs container is located in the blob
namespace of the storage account

• Each log will be written in the following format

• <service-name>/YYYY/MM/DD/hhmm/<counter>.log
Storage analytics metrics

• Storage Analytics can store metrics that include aggregated transaction statistics and capacity
data about requests to a storage service

• Transaction metrics

• Transaction aggregated data recorded at hourly or minute

• Data is recorded at service level and API operation level

• Capacity metrics

• Capacity data is recorded daily for a storage account's Blob service which includes Capacity,
ContainerCount, ObjectCount

• All metrics data for each of the storage services is stored in three tables reserved for that service.
Azure Storage - Tools
Storage explorer
• Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage
data on Windows, macOS, and Linux.

• Storage Explorer provides several ways to connect to storage accounts. For example, you can:

• Connect to storage accounts associated with your Azure subscriptions.

• Connect to storage accounts and services that are shared from other Azure subscriptions.

• Connect to and manage local storage by using the Azure Storage Emulator.

• Other services you can connect to

• Cosmos DB

• Data Lake store

Storage emulator

• The Microsoft Azure storage emulator provides a local environment that emulates the Azure Blob,
Queue, and Table services for development purposes.

• Using the storage emulator, you can test your application against the storage services locally,
without creating an Azure subscription or incurring any costs

• The storage emulator is available as part of the Microsoft Azure SDK. You can also install the
storage emulator by using the standalone installer (direct download).

• The storage emulator uses a local Microsoft SQL Server instance and the local file system to
emulate Azure storage services. By default, the storage emulator uses a database in Microsoft SQL
Server 2012 Express LocalDB.
Visual studio cloud & server explorer
• Server explorer

• The Azure Storage node in Server Explorer shows data in your local storage emulator account
and your other Azure storage accounts.

• To see the storage emulator account’s resources, expand the Development node

• To view resources in a storage account, expand the storage account’s node in Server Explorer
where you see Blobs, Queues, and Tables nodes.

• Cloud explorer

• Cloud Explorer enables you to view your Azure resources and resource groups, inspect their
properties, and perform key developer diagnostics actions from within Visual Studio.
Azure storage client library
• Client library

• Use connection strings to connect to an Azure Storage account, then use the client libraries'
classes and methods to work with blob, table, file, or queue storage.

• Install the NuGet package WIndowsAzure.Storage

• Management API’s

• Create and manage Azure Storage accounts and connection keys with the management API.

• Install the NuGet package Microsoft.Azure.Management.Storage.Fluent