Data Sync
Data Sync
SQL DB Documentation
SQL Database Service
About SQL DB
Choose a version of Azure SQL
Quickstarts
Create
Single database
Managed Instance
Configure
Logical server firewall rule
Client VM connection to a Managed Instance
Point-to-site connection to a Managed Instance
Load data
Import BACPAC into a single database
Restore a database backup to a Managed Instance
Query
SSMS
SQL Operations Studio
Azure portal
VS Code
.NET with Visual Studio
.NET core
.NET with Active Directory MFA
Go
Java
Node.js
PHP
Python
Ruby
Tutorials
1 - Design DB using SSMS
2 - Design DB using .NET
3 - Migrate to SQL Server DB using DMA
4 - Secure your SQL DB
5 - Implement a geo-distributed database
6 - Migrate to a Managed Instance using DMS
7 - Set up Azure SQL Data Sync
Samples
Code samples
Azure CLI
Azure PowerShell
Concepts
Purchasing models
vCore vs. DTU
vCore model
DTU model
Resource limits
Prepay for reserved capacity
Logical servers
Create and manage
Connectivity architecture
T-SQL differences
Single databases
Create and manage
vCore-based resource limits
DTU-based resource limits
Scale resources
Elastic pools
Elastic pools
Create and manage
vCore-based resource limits
DTU-based resource limits
Scale resources
Managed instances
Managed instances
VNet configuration
Custom DNS
Sync network configuration
Connectivity architecture
Connect applications
T-SQL differences
Transactional replication
Business continuity
Business continuity
High availability
Database backups
Long-term backup retention
Long-term backup retention
Configure using Azure blob storage
Configure using Azure vault (deprecated)
Failover groups and geo-replication
Failover groups and geo-replication
Configure geo-replication - Portal
Configure security for replicas
Database recovery
Database recovery
Recover from outage
Perform recovery drill
App design
Design for disaster recovery
Design for elastic pools
Design for app upgrades
Security
Security
Advanced Threat Protection
Advanced Threat Protection
Data discovery and classification
Vulnerability assessment
Threat detection - SQL Database
Threat detection - Managed Instance
Firewall rules
Create and manage
Virtual network endpoints
Virtual network endpoints - PowerShell
Access control
Access control
Logins and users
Azure AD authentication
Azure AD authentication
Configure Azure AD auth
Conditional access
Multi-factor auth
Configure multi-factor auth
Auditing
Auditing - Servers, pools, and databases
Auditing - Managed Instance
Table auditing (deprecated)
Dynamic data masking
Dynamic data masking
Configure masking
Always encrypted
Use the certificate store
Use the Azure key vault
TDE with Azure SQL
TDE with Azure SQL
TDE with Bring Your Own Key
Configure TDE with BYOK
Rotate TDE BYOK keys
Remove TDE protector
Elastic scalability
Resource scalability
Read Scale-Out
Database sharding
Database sharding
Elastic client library - consolidate
Upgrade elastic database client library
Shard maps
Query routing
Manage credentials
Shard querying
Elastic tools
Move sharded data
Elastic client library - Dapper
Elastic tools FAQ
Create sharded app
Query horizontally-sharded data
Move sharded data
Security configuration
Add a shard
Fix shard map problems
Migrate sharded DB
Create counters
Use entity framework
SQL features comparison
Monitor and tune
Monitor and tune
Perf monitoring
Perf recommendations
Intelligent query processing
Intelligent query processing
Adaptive query processing
Automatic tuning
Automatic tuning
Enable automatic tuning
Enable E-mail notifications
Intelligent Insights
Intelligent Insights
Diagnostic logging
Use diagnostics log
Performance troubleshooting
Performance recommendations
Use Query Performance Insight
Apply performance recommendations
Manual tuning
Manual tuning
Use DMVs to monitor perf
Operate query store
Monitor in-memory
Extended events
Extended events
Extended events - event file
Extended events - ring buffer
Create alerts
Troubleshoot connectivity
Azure Automation
Cross-database operations
Elastic Database jobs
Elastic Database jobs
Create and manage (PowerShell)
Create and manage (T-SQL)
Migrate (from old Elastic jobs)
Elastic jobs (deprecated)
Elastic jobs (deprecated)
Install elastic job (deprecated)
Create job - Portal (deprecated)
Create job - Azure PowerShell (deprecated)
Create job - Sample app (deprecated)
Uninstall elastic job (deprecated)
Elastic queries
Elastic queries
Query vertically partitioned data
Report across scaled-out data tier
Query across tables with different schemas
Elastic transactions
Multi-tenant SaaS
SaaS design patterns
SaaS video indexer
SaaS app security
Develop databases
Develop databases
JSON data
In-memory
Temporal tables
Retention policies
Configure In-Memory
How-to guides
Migrate to SQL Database
Migrate to SQL Database
Migrate to Managed Instance
Migrate TDE cert to Managed Instance
Manage SQL Database after migration
Load and move data
Copy a DB within Azure
Import a DB from a BACPAC
Export a DB to BACPAC
Data sync
SQL Data Sync
Replicate schema changes
Monitor with OMS
Best practices for Data Sync
Troubleshoot Data Sync
Load data with BCP
Load data with ADF
Manage file space
Develop apps
Connectivity
Use Spark Connector
Authenticate app
Error messages
Batching for perf
Connectivity guidance
DNS aliases
DNS alias PowerShell
Ports - ADO.NET
C and C ++
Excel
Multi-tenant SaaS sample application
Wingtip Tickets sample
General guidance
Standalone application
Deploy example app
Provision tenants
Database per tenant
Tutorial intro
Deploy example app
Provision tenants
Monitor database performance
Monitor with log analytics
Restore one tenant
Manage tenant schema
Cross-tenant reporting
Tenant analytics
With SQL DB
With SQL DW
Disaster recovery using geo-restore
Disaster recovery using database geo-replication
Multi-tenant database
Deploy example app
Provision tenants
Monitor database performance
Run ad-hoc queries
Manage tenant schema
ETL for analytics
Reference
Azure PowerShell
Azure CLI 2.0
.NET
Java
Node.js
Python
Ruby
PHP
T-SQL
REST
SQL Server tools
SQL Server Management Studio (SSMS)
SQL Server Data Tools (SSDT)
BCP
SQLCMD
SqlPackage
SQL Database Management Library package
SQL Server drivers
ADO.NET
JDBC
ODBC
Resources
Azure Roadmap
FAQ
Public data sets
Troubleshoot connectivity
Pricing
MSDN forum
Stack Overflow
Videos
Service updates
Architecture center
Customer stories
What is the Azure SQL Database service?
9/6/2018 • 14 minutes to read • Edit Online
SQL Database is a general-purpose relational database managed service in Microsoft Azure that supports
structures such as relational data, JSON, spatial, and XML. SQL Database delivers dynamically scalable
performance within two different purchasing models: a vCore-based purchasing model and a DTU -based
purchasing model. SQL Database also provides options such as columnstore indexes for extreme analytic
analysis and reporting, and in-memory OLTP for extreme transactional processing. Microsoft handles all patching
and updating of the SQL code base seamlessly and abstracts away all management of the underlying
infrastructure.
Azure SQL Database provides the following deployment options for an Azure SQL database:
As a single database with its own set of resources managed via a logical server
As a pooled database in an elastic pool with a shared set of resources managed via a logical server
As a part of a collection of databases known as a managed instance (in public preview ) that contains system
and user databases and sharing a set of resources
The following illustration shows these deployment options:
SQL Database shares its code base with the Microsoft SQL Server database engine. With Microsoft's cloud-first
strategy, the newest capabilities of SQL Server are released first to SQL Database, and then to SQL Server itself.
This approach provides you with the newest SQL Server capabilities with no overhead for patching or upgrading
- and with these new features tested across millions of databases. For information about new capabilities as they
are announced, see:
Azure Roadmap for SQL Database: A place to find out what’s new and what’s coming next.
Azure SQL Database blog: A place where SQL Server product team members blog about SQL Database
news and features.
IMPORTANT
To understand the feature differences between SQL Database and SQL Server, see SQL features.
SQL Database delivers predictable performance at multiple service levels that provides dynamic scalability with
no downtime, built-in intelligent optimization, global scalability and availability, and advanced security options —
all with near-zero administration. These capabilities allow you to focus on rapid app development and
accelerating your time to market, rather than allocating precious time and resources to managing virtual
machines and infrastructure. The SQL Database service is currently in 38 data centers around the world, with
more data centers coming online regularly, which enables you to run your database in a data center near you.
NOTE
SQL Database Managed Instance is currently in preview and is only available at a single service level. For more information,
see SQL Database Managed Instance.
IMPORTANT
SQL Database Managed Instance does not support elastic pools.
Built-in intelligence
With SQL Database, you get built-in intelligence that helps you dramatically reduce the costs of running and
managing databases and maximizes both performance and security of your application. Running millions of
customer workloads around-the-clock, SQL Database collects and processes a massive amount of telemetry data,
while also fully respecting customer privacy behind the scenes. Various algorithms are continuously evaluating
the telemetry data so that the service can learn and adapt with your application. Based on this analysis, the
service comes up with performance improving recommendations tailored to your specific workload.
Automatic performance monitoring and tuning
SQL Database provides detailed insight into the queries that you need to monitor. SQL Database's learns about
your database patterns and enables you to adapt your database schema to your workload. SQL Database
provides performance tuning recommendations, where you can review tuning actions and apply them.
However, constantly monitoring database is a hard and tedious task, especially when dealing with many
databases. Intelligent Insights does this job for you by automatically monitoring SQL Database performance at
scale and it informs you of performance degradation issues, it identifies the root cause of the issue and provides
performance improvement recommendations when possible.
Managing a huge number of databases might be impossible to do efficiently even with all available tools and
reports that SQL Database and Azure portal provide. Instead of monitoring and tuning your database manually,
you might consider delegating some of the monitoring and tuning actions to SQL Database using automatic
tuning. SQL Database automatically apply recommendations, tests, and verifies each of its tuning actions to
ensure the performance keeps improving. This way, SQL Database automatically adapts to your workload in
controlled and safe way. Automatic tuning means that the performance of your database is carefully monitored
and compared before and after every tuning action, and if the performance doesn’t improve, the tuning action is
reverted.
Today, many of our partners running SaaS multi-tenant apps on top of SQL Database are relying on automatic
performance tuning to make sure their applications always have stable and predictable performance. For them,
this feature tremendously reduces the risk of having a performance incident in the middle of the night. In
addition, since part of their customer base also uses SQL Server, they are using the same indexing
recommendations provided by SQL Database to help their SQL Server customers.
There are two automatic tuning aspects that are available in SQL Database:
Automatic index management: Identifies indexes that should be added in your database, and indexes that
should be removed.
Automatic plan correction: Identifies problematic plans and fixes SQL plan performance problems (coming
soon, already available in SQL Server 2017).
Adaptive query processing
We are also adding the adaptive query processing family of features to SQL Database, including interleaved
execution for multi-statement table-valued functions, batch mode memory grant feedback, and batch mode
adaptive joins. Each of these adaptive query processing features applies similar “learn and adapt” techniques,
helping further address performance issues related to historically intractable query optimization problems.
Easy-to-use tools
SQL Database makes building and maintaining applications easier and more productive. SQL Database allows
you to focus on what you do best: building great apps. You can manage and develop in SQL Database using tools
and skills you already have.
The Azure portal: A web-based application for managing all Azure services
SQL Server Management Studio: A free, downloadable client application for managing any SQL
infrastructure, from SQL Server to SQL Database
SQL Server Data Tools in Visual Studio: A free, downloadable client application for developing SQL Server
relational databases, Azure SQL databases, Integration Services packages, Analysis Services data models, and
Reporting Services reports.
Visual Studio Code: a free, downloadable, open-source, code editor for Windows, macOS, and Linux that
supports extensions, including the mssql extension for querying Microsoft SQL Server, Azure SQL Database,
and SQL Data Warehouse.
SQL Database supports building applications with Python, Java, Node.js, PHP, Ruby, and .NET on the MacOS,
Linux, and Windows. SQL Database supports the same connection libraries as SQL Server.
Next steps
See the pricing page for single database and elastic pools cost comparisons and calculators.
See these quickstarts to get you started:
Create a SQL database in the Azure portal
Create a SQL database with the Azure CLI
Create a SQL database using PowerShell
For a set of Azure CLI and PowerShell samples, see:
Azure CLI samples for SQL Database
Azure PowerShell samples for SQL Database
Choose a cloud SQL Server option: Azure SQL
(PaaS) Database or SQL Server on Azure VMs (IaaS)
9/13/2018 • 15 minutes to read • Edit Online
In Azure, you can have your SQL Server workloads running in a hosted infrastructure (IaaS ) or running as a
hosted service (PaaS ):
Azure SQL Database: A SQL database engine, based on the Enterprise Edition of SQL Server, that is
optimized for modern application development. Azure SQL Database offers several deployment options:
You can deploy a single database to a logical server.
You can deploy into an elastic pool on a logical server to share resources and reduce costs.
NOTE
An Azure SQL Database containing single and pooled databases offers most of database-scoped features of
SQL Server.
NOTE
With both versions, Azure SQL Database adds additional features that are not available in SQL Server, such as
built-in intelligence and management. With the first version, With Azure SQL Database Managed Instance,
Azure SQL Database offers shared resources for databases and additional instance-scoped features. Azure
SQL Database Managed Instance supports database migration with minimal to no database change.
SQL Server on Azure Virtual Machines: SQL Server installed and hosted in the cloud on Windows Server or
Linux virtual machines (VMs) running on Azure, also known as an infrastructure as a service (IaaS ). SQL Server
on Azure virtual machines is a good option for migrating on-premises SQL Server databases and applications
without any database change. All recent versions and editions of SQL Server are available for installation in an
IaaS virtual machine. The most significant difference from SQL Database is that SQL Server VMs allow full
control over the database engine. You can choose when maintenance/patching will start, to change the recovery
model to simple or bulk logged to enable faster load less log, to pause or start engine when needed, and you
can fully customize the SQL Server database engine. With this additional control comes with added
responsibility to manage the virtual machines.
Learn how each deployment option fits into the Microsoft data platform and get help matching the right option to
your business requirements. Whether you prioritize cost savings or minimal administration ahead of everything
else, this article can help you decide which approach delivers against the business requirements you care about
most.
As seen in the diagram, each offering can be characterized by the level of administration you have over the
infrastructure (on the X axis), and by the degree of cost efficiency achieved by database level consolidation and
automation (on the Y axis).
When designing an application, four basic options are available for hosting the SQL Server part of the application:
SQL Server on non-virtualized physical machines
SQL Server in on-premises virtualized machines (private cloud)
SQL Server in Azure Virtual Machine (Microsoft public cloud)
Azure SQL Database (Microsoft public cloud)
In the following sections, you learn about SQL Server in the Microsoft public cloud: Azure SQL Database and SQL
Server on Azure VMs. In addition, you explore common business motivators for determining which option works
best for your application.
A closer look at Azure SQL Database and SQL Server on Azure VMs
Azure SQL Database is a relational database-as-a-service (DBaaS ) hosted in the Azure cloud that falls into the
industry category of Platform -as-a -Service (PaaS ). SQL database is built on standardized hardware and software
that is owned, hosted, and maintained by Microsoft. With SQL Database, you can use built-in features and
functionality that require extensive configuration in SQL Server. When using SQL Database, you pay-as-you-go
with options to scale up or out for greater power with no interruption. Azure SQL Database is an ideal
environment for developing new applications in the cloud. And, with Azure SQL Database Managed Instance, you
can bring your own license. Additionally, this option provides all of the PaaS benefits of Azure SQL Database but
adds capabilities that were previously only available in SQL VMs. This includes a native virtual network (VNet) and
near 100% compatibility with on-premises SQL Server. Managed Instance is ideal for on-premises database
migrations to Azure with minimal changes required.
SQL Server on Azure Virtual Machines (VMs) falls into the industry category Infrastructure-as-a -Service
(IaaS ) and allows you to run SQL Server inside a virtual machine in the cloud. SQL Server virtual machines also
run on standardized hardware that is owned, hosted, and maintained by Microsoft. When using SQL Server on a
VM, you can either pay-as you-go for a SQL Server license already included in a SQL Server image or easily use
an existing license. You can also stop or resume the VM as needed.
In general, these two SQL options are optimized for different purposes:
Azure SQL Database is optimized to reduce overall management costs to the minimum for provisioning and
managing many databases. It reduces ongoing administration costs because you do not have to manage any
virtual machines, operating system or database software. You do not have to manage upgrades, high
availability, or backups. In general, Azure SQL Database can dramatically increase the number of databases
managed by a single IT or development resource. Elastic pools also support SaaS multi-tenant application
architectures with features including tenant isolation and the ability to scale to reduce costs by sharing
resources across databases. Azure SQL Database Managed Instance provides support for instance-scoped
features enabling easy migration of existing applications, as well as sharing resources amongst databases.
SQL Server running on Azure VMs is optimized for migrating existing applications to Azure or extending
existing on-premises applications to the cloud in hybrid deployments. In addition, you can use SQL Server in a
virtual machine to develop and test traditional SQL Server applications. With SQL Server on Azure VMs, you
have the full administrative rights over a dedicated SQL Server instance and a cloud-based VM. It is a perfect
choice when an organization already has IT resources available to maintain the virtual machines. These
capabilities allow you to build a highly customized system to address your application’s specific performance
and availability requirements.
The following table summarizes the main characteristics of SQL Database and SQL Server on Azure VMs:
Best for: New cloud-designed New applications or existing Existing applications that
applications that want to use on-premises applications require fast migration to the
the latest stable SQL Server that want to use the latest cloud with minimal changes
features andhave time stable SQL Server features or no changes. Rapid
constraints in development and that are migrated to the development and test
and marketing. cloud with minimal changes. scenarios when you do not
want to buy on-premises
non-production SQL Server
hardware.
Teams that need built-in Same as SQL Database. Teams that can configure,
high availability, disaster fine tune, customize, and
recovery, and upgrade for manage high availability,
the database. disaster recovery, and
patching for SQL Server.
Some provided automated
features dramatically simplify
this.
Teams that do not want to Same as SQL Database. You need a customized
manage the underlying environment with full
operating system and administrative rights.
configuration settings.
AZURE SQL DATABASE
LOGICAL SERVERS, ELASTIC
POOLS, AND SINGLE AZURE SQL DATABASE AZURE VIRTUAL MACHINE
DATABASES MANAGED INSTANCE SQL SERVER
Compatibility Supports most on-premises Supports almost all on- Supports all on-premises
database-level capabilities. premises instance-level and capabilities.
database-level capabilities.
Resources: You do not want to employ Same as SQL Database. You have some IT resources
IT resources for for configuration and
configuration and management. Some
management of the provided automated features
underlying infrastructure, dramatically simplify this.
but want to focus on the
application layer.
Total cost of ownership: Eliminates hardware costs Same as SQL Database. Eliminates hardware costs.
and reduces administrative
costs.
Business continuity: In addition to built-in fault Same as SQL Database, plus SQL Server on Azure VMs
tolerance infrastructure user-initiated, copy-only lets you set up a high
capabilities, Azure SQL backups are available. availability and disaster
Database provides features, recovery solution for your
such as automated backups, database’s specific needs.
Point-In-Time Restore, geo- Therefore, you can have a
restore, and failover groups system that is highly
and active geo-replication to optimized for your
increase business continuity. application. You can test and
For more information, see run failovers by yourself
SQL Database business when needed. For more
continuity overview. information, see High
Availability and Disaster
Recovery for SQL Server on
Azure Virtual Machines.
Hybrid cloud: Your on-premises Native virtual network With SQL Server on Azure
application can access data implementation and VMs, you can have
in Azure SQL Database. connectivity to your on- applications that run partly
premises environment using in the cloud and partly on-
Azure Express Route or VPN premises. For example, you
Gateway. can extend your on-premises
network and Active Directory
Domain to the cloud via
Azure Virtual Network. In
addition, you can store on-
premises data files in Azure
Storage using SQL Server
Data Files in Azure. For more
information, see Introduction
to SQL Server 2014 Hybrid
Cloud.
AZURE SQL DATABASE
LOGICAL SERVERS, ELASTIC
POOLS, AND SINGLE AZURE SQL DATABASE AZURE VIRTUAL MACHINE
DATABASES MANAGED INSTANCE SQL SERVER
Supports SQL Server Replication is not supported Fully supports SQL Server
transactional replication as a for Azure SQL Database transactional replication,
subscriber to replicate data. Managed Instance. Always On Availability
Groups, Integration Services,
and Log Shipping to
replicate data. Also,
traditional SQL Server
backups are fully supported
Summary
This article explored SQL Database and SQL Server on Azure Virtual Machines (VMs) and discussed common
business motivators that might affect your decision. The following is a summary of suggestions for you to
consider:
Choose Azure SQL Database if:
You are building new cloud-based applications to take advantage of the cost savings and performance
optimization that cloud services provide. This approach provides the benefits of a fully managed cloud service,
helps lower initial time-to-market, and can provide long-term cost optimization.
You want to have Microsoft perform common management operations on your databases and require stronger
availability SLAs for databases.
You want to migrate an existing application as-is to Azure SQL Database Managed Instance and take advantage
of additional parity with SQL Server and/or advanced security and networking. Managed Instance is a good
choice for both new and existing applications.
Choose SQL Server on Azure VMs if:
You have existing on-premises applications that you want to migrate or extend to the cloud, or if you want to
build enterprise applications larger than 4 TB. This approach provides the benefit of using the SQL Server
version and edition of your choice, large database capacity, full control over SQL Server and Windows/Linux,
and secure tunneling to on-premises. This approach minimizes costs for development and modifications of
existing applications.
You have existing IT resources and can ultimately own patching, backups, and database high availability. Notice
that some automated features dramatically simplify these operations.
Next steps
See Your first Azure SQL Database to get started with SQL Database.
See SQL Database pricing.
See Provision a SQL Server virtual machine in Azure to get started with SQL Server on Azure VMs.
Create an Azure SQL database in the Azure portal
9/6/2018 • 4 minutes to read • Edit Online
This quickstart walks through how to create a SQL database in Azure using the DTU -based purchasing model.
Azure SQL Database is a “Database-as-a-Service” offering that enables you to run and scale highly available
SQL Server databases in the cloud. This quickstart shows you how to get started by creating and then
querying a SQL database using the Azure portal.
If you don't have an Azure subscription, create a free account before you begin.
NOTE
This tutorial uses the DTU-based purchasing model but the vCore-based purchasing model is also available.
IMPORTANT
You must select the sample database on this form because it is used in the remainder of this quickstart.
4. Under Server, click Configure required settings and fill out the SQL server (logical server) form with
the following information, as shown on the following image:
Server name Any globally unique name For valid server names, see Naming
rules and restrictions.
Server admin login Any valid name For valid login names, see Database
Identifiers.
IMPORTANT
The server admin login and password that you specify here are required to log in to the server and its databases
later in this quickstart. Remember or record this information for later use.
5. When you have completed the form, click Select.
6. Click Pricing tier to specify the service tier, the number of DTUs, and the amount of storage. Explore the
options for the amount of DTUs and storage that is available to you for each service tier.
IMPORTANT
More than 1 TB of storage in the Premium tier is currently available in all regions except the following: UK North,
West Central US, UK South2, China East, USDoDCentral, Germany Central, USDoDEast, US Gov Southwest, US
Gov South Central, Germany Northeast, China North, US Gov East. In other regions, the storage max in the
Premium tier is limited to 1 TB. See P11-P15 Current Limitations.
7. For this quickstart, select the Standard service tier and then use the slider to select 10 DTUs (S0) and 1
GB of storage.
8. Accept the preview terms to use the Add-on Storage option.
IMPORTANT
More than 1 TB of storage in the Premium tier is currently available in all regions except the following: West
Central US, China East, USDoDCentral, USGov Iowa, Germany Central, USDoDEast, US Gov Southwest, Germany
Northeast, China North. In other regions, the storage max in the Premium tier is limited to 1 TB. See P11-P15
Current Limitations.
9. After selecting the server tier, the number of DTUs, and the amount of storage, click Apply.
10. Now that you have completed the SQL Database form, click Create to provision the database.
Provisioning takes a few minutes.
11. On the toolbar, click Notifications to monitor the deployment process.
2. Select SQL server authentication, provide the required login information, and then click OK to log in.
3. After you are authenticated as ServerAdmin, type the following query in the query editor pane.
4. Click Run and then review the query results in the Results pane.
5. Close the Query editor page, click OK to discard your unsaved edits.
Clean up resources
Save these resources if you want to go to Next steps and learn how to connect and query your database using
a number of different methods. If, however, you wish to delete the resources that you created in this quickstart,
use the following steps.
1. From the left-hand menu in the Azure portal, click Resource groups and then click myResourceGroup.
2. On your resource group page, click Delete, type myResourceGroup in the text box, and then click Delete.
Next steps
Now that you have a database, you need to create a server-level firewall rule to connect to it from your on-
premises tools. See Create server-level firewall rule
If creating a server-level firewall rule, you can connect and query using one of your favorite tools or
languages, including
Connect and query using SQL Server Management Studio
Connect and query using SQL Server Operations Studio
To create databases using Azure CLI, see Azure CLI samples
To create databases using Azure PowerShell, see Azure PowerShell samples
Create an Azure SQL Database Managed Instance
9/14/2018 • 3 minutes to read • Edit Online
This quickstart walks through how to create an Azure SQL Database Managed Instance in the Azure portal.
If you don't have an Azure subscription, create a free account before you begin.
4. Fill out the Managed Instance form with the requested information, using the information in the following
table:
Managed instance name Any valid name For valid names, see Naming rules
and restrictions.
Managed instance admin login Any valid user name For valid names, see Naming rules
and restrictions. Do not use
"serveradmin" as that is a reserved
server-level role.
Resource Group A new or existing resource group For valid resource group names, see
Naming rules and restrictions.
Location The location in which you want to For information about regions, see
create the Managed Instance Azure Regions.
Virtual network Select either Create new virtual To configure a virtual network for a
network or a virtual network that Managed Instance with custom
you previously created in the settings, see Configure SQL
resource group that you previously Managed Instance virtual network
provided in this form environment template in Github. For
information regarding the
requirements for configuring the
network environment for a
Managed Instance, see Configure a
VNet for Azure SQL Database
Managed Instance
5. Click Pricing tier to size compute and storage resources as well as review the pricing tier options. The
General Purpose pricing tier with 32 GB of memory and 16 vCores is the default value.
6. Use the sliders or text boxes to specify the amount of storage and the number of virtual cores.
7. When complete, click Apply to save your selection.
8. Click Create to deploy the Managed Instance.
9. Click the Notifications icon to view the status of deployment.
10. Click Deployment in progress to open the Managed Instance window to further monitor the
deployment progress.
IMPORTANT
For the first instance in a subnet, deployment time is typically much longer than in case of the subsequent instances. Do
not cancel deployment operation because it lasts longer than you expected. Creating the second Managed Instance in the
subnet only takes a couple of minutes.
Next steps
To learn about connecting to a Managed Instance, see:
For an overview of the connection options for applications, see Connect your applications to Managed
Instance.
For a quickstart showing how to connect to a Managed Instance from an Azure virtual machine, see
Configure an Azure virtual machine connection.
For a quickstart showing how to connect to a Managed Instance from an on-premises client computer
using a point-to-site connection, see Configure a point-to-site connection.
To restore an existing SQL Server database from on-premises to a Managed instance, you can use the Azure
Database Migration Service (DMS ) for migration to restore from a database backup file or the T-SQL
RESTORE command to restore from a database backup file.
Create a server-level firewall rule for your SQL
database using the Azure portal
9/6/2018 • 2 minutes to read • Edit Online
This quickstart walks through how to create a server-level firewall rule for an Azure SQL database to enable you
to connect to it from an on-premises resource.
Prerequisites
This quickstart uses as its starting point the resources created in this quickstart: Create an Azure SQL database
in the Azure portal
NOTE
SQL Database communicates over port 1433. If you are trying to connect from within a corporate network, outbound
traffic over port 1433 may not be allowed by your network's firewall. If so, you cannot connect to your Azure SQL
Database server unless your IT department opens port 1433.
1. After the deployment completes, click SQL databases from the left-hand menu and then click
mySampleDatabase on the SQL databases page. The overview page for your database opens, showing
you the fully qualified server name (such as mynewserver-20170824.database.windows.net) and
provides options for further configuration.
2. Copy this fully qualified server name for use to connect to your server and its databases in subsequent
quickstarts.
3. Click Set server firewall on the toolbar as shown in the previous image. The Firewall settings page for
the SQL Database server opens.
4. Click Add client IP on the toolbar to add your current IP address to a new firewall rule. A firewall rule
can open port 1433 for a single IP address or a range of IP addresses.
5. Click Save. A server-level firewall rule is created for your current IP address opening port 1433 on the
logical server.
6. Click OK and then close the Firewall settings page.
You can now connect to the SQL Database server and its databases using SQL Server Management Studio or
another tool of your choice from this IP address using the server admin account created previously.
IMPORTANT
By default, access through the SQL Database firewall is enabled for all Azure services. Click OFF on this page to disable for
all Azure services.
Clean up resources
Save these resources if you want to go to Next steps and learn how to connect and query your database using a
number of different methods. If, however, you wish to delete the resources that you created in this quickstart, use
the following steps.
1. From the left-hand menu in the Azure portal, click Resource groups and then click myResourceGroup.
2. On your resource group page, click Delete, type myResourceGroup in the text box, and then click Delete.
Next steps
Now that you have a database, you can connect and query using one of your favorite tools or languages,
including
Connect and query using SQL Server Management Studio
Connect and query using SQL Server Operations Studio
To learn how to design your first database, create tables, and insert data, see one of these tutorials:
Design your first Azure SQL database using SSMS
Design an Azure SQL database and connect with C# and ADO.NET
Configure Azure VM to connect to an Azure SQL
Database Managed Instance
9/14/2018 • 5 minutes to read • Edit Online
This quickstarts demonstrates how to configure an Azure virtual machine to connect to an Azure SQL Database
Managed Instance using SQL Server Management Studio (SSMS ). For a quickstart showing how to connect from
an on-premises client computer using a point-to-site connection, see Configure a point-to-site connection
Prerequisites
This quickstart uses as its starting point the resources created in this quickstart: Create a Managed Instance.
Name Any valid name For valid names, see Naming rules
and restrictions.
Address range (CIDR block) A valid range The default value is good for this
quickstart.
Network security group None The default value is good for this
quickstart.
2. Fill out the form with the requested information, using the information in the following table:
SETTING SUGGESTED VALUE DESCRIPTION
Resource Group The resource group that you This must be the resource group in
specified in the Create Managed which the VNet exists.
Instance quickstart.
Location The location for the resource group This value is populated based on the
resource group selected
Virtual machine name Any valid name For valid names, see Naming rules
and restrictions.
Admin Username Any valid user name For valid names, see Naming rules
and restrictions. Do not use
"serveradmin" as that is a reserved
server-level role.
Virtual Machine Size Any valid size The default in this template of
**Standard_B2s is sufficient for this
quickstart.
Virtual Network Name The location that you previously For information about regions, see
selected Azure Regions.
Subnet name The name of the subnet that you Do not choose the subnet in which
created in the previous procedure you created the Managed Instance
artifacts Location Sas token leave blank Do not change this value
If you used the suggested VNet name and the default subnet in creating your Managed Instance, you don't
need to change last two parameters. Otherwise you should change these values to the values that you
entered when you set up the network environment.
3. Select the I agree to terms and conditions stated above checkbox.
4. Click Purchase to deploy the Azure VM in your network.
5. Click the Notifications icon to view the status of deployment.
Do not continue until the Azure virtual machine is created.
2. Click Connect.
A Remote Desktop Protocol file (.rdp file) form appears with the public IP address and port number for the
virtual machine.
After you connect, you can view your system and user databases in the Databases node, and various objects in the
Security, Server Objects, Replication, Management, SQL Server Agent, and XEvent Profiler nodes.
Next steps
For a quickstart showing how to connect from an on-premises client computer using a point-to-site connection,
see Configure a point-to-site connection.
For an overview of the connection options for applications, see Connect your applications to Managed Instance.
To restore an existing SQL Server database from on-premises to a Managed instance, you can use the Azure
Database Migration Service (DMS ) for migration to restore from a database backup file or the T-SQL
RESTORE command to restore from a database backup file.
Configure a point-to-site connection to connect to
an Azure SQL Database Managed Instance from on-
premises computer
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to connect to an Azure SQL Database Managed Instance using SQL Server
Management Studio (SSMS ) from an on-premises client computer over a point-to-site connection. For
information about point-to-site connections, see About Point-to-Site VPN
Prerequisites
This quickstart:
Uses as its starting point the resources created in this quickstart: Create a Managed Instance.
Requires PowerShell 5.1 and Azure PowerShell 5.4.2 or higher your on-premises client computer.
Requires the newest version of SQL Server Management Studio (SSMS ) on your on-premises client computer
$scriptUrlBase = 'https://raw.githubusercontent.com/Microsoft/sql-server-
samples/master/samples/manage/azure-sql-db-managed-instance/attach-vpn-gateway'
$parameters = @{
subscriptionId = '<subscriptionId>'
resourceGroupName = '<resourceGroupName>'
virtualNetworkName = '<virtualNetworkName>'
certificateNamePrefix = '<certificateNamePrefix>'
}
3. Provide the requested parameters in the PowerShell script. The values for <subscriptionId> ,
<resourceGroup> and <virtualNetworkName> should match the ones that are used in Create Managed
Instance quickstart. The value for <certificateNamePrefix> can be a string of your choice.
4. Execute the PowerShell script.
4. Extract the files from the zip file and then open the extracted folder.
5. Havigate to the WindowsAmd64 folder and open the VpnClientSetupAmd64.exe file.
6. If you recieve a Windows protected your PC message, click More info and then click Run anyway.
\
7. Click Yes in the User Account Control dialog box to proceed.
8. In the MyNewVNet dialog box, click Yes to install a Vpn Client for MyNewVNet.
2. Click Connect.
3. In the MyNewVNet dialog box, click Connect.
4. When prompted that Connection Manager needs elevated privilege to update your route table, click Continue.
5. Click Yes in the User Account Control dialog box to proceed.
Next steps
For a quickstart showing how to connect from an Azure virtual machine, see Configure a point-to-site
connection
For an overview of the connection options for applications, see Connect your applications to Managed
Instance.
To restore an existing SQL Server database from on-premises to a Managed instance, you can use the Azure
Database Migration Service (DMS ) for migration to restore from a database backup file or the T-SQL
RESTORE command to restore from a database backup file.
Import a BACPAC file to a new Azure SQL Database
9/14/2018 • 4 minutes to read • Edit Online
When you need to import a database from an archive or when migrating from another platform, you can import
the database schema and data from a BACPAC file. A BACPAC file is a ZIP file with an extension of BACPAC
containing the metadata and data from a SQL Server database. A BACPAC file can be imported from Azure blob
storage (standard storage only) or from local storage in an on-premises location. To maximize the import speed, we
recommend that you specify a higher service tier and performance level, such as a P6, and then scale to down as
appropriate after the import is successful. Also, the database compatibility level after the import is based on the
compatibility level of the source database.
IMPORTANT
After you migrate your database to Azure SQL Database, you can choose to operate the database at its current compatibility
level (level 100 for the AdventureWorks2008R2 database) or at a higher level. For more information on the implications and
options for operating a database at a specific compatibility level, see ALTER DATABASE Compatibility Level. See also ALTER
DATABASE SCOPED CONFIGURATION for information about additional database-level settings related to compatibility levels.
>
To monitor the progress of the import operation, open the page for the logical server containing the database being
imported. Scroll down to Operations and then click Import/Export history.
NOTE
Azure SQL Database Managed Instance supported importing from a BACPAC file using the other methods in this article, but
does not currently support migrating using the Azure portal.
IMPORTANT
An Azure SQL Database logical server listens on port 1433. If you are attempting to connect to an Azure SQL Database
logical server from within a corporate firewall, this port must be open in the corporate firewall for you to successfully connect.
This example shows how to import a database using SqlPackage.exe with Active Directory Universal
Authentication:
SqlPackage.exe /a:Import /sf:testExport.bacpac /tdn:NewDacFX /tsn:apptestserver.database.windows.net /ua:True
/tid:"apptest.onmicrosoft.com"
To check the status of the import request, use the Get-AzureRmSqlDatabaseImportExportStatus cmdlet. Running
this immediately after the request usually returns Status: InProgress. When you see Status: Succeeded the
import is complete.
TIP
For another script example, see Import a database from a BACPAC file.
Limitations
Import to a database in elastic pool is not supported. You can import data into a singleton database and then
move the database to a pool.
Next steps
To learn how to connect to and query an imported SQL Database, see Connect to SQL Database with SQL
Server Management Studio and perform a sample T-SQL query.
For a SQL Server Customer Advisory Team blog about migrating using BACPAC files, see Migrating from SQL
Server to Azure SQL Database using BACPAC Files.
For a discussion of the entire SQL Server database migration process, including performance
recommendations, see Migrate a SQL Server database to Azure SQL Database.
To learn how to manage and share storage keys and shared access signitures securely, see Azure Storage
Security Guide.
Restore a database backup to an Azure SQL
Database Managed Instance
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to restore a backup of a database stored in Azure blob storage into the
Managed Instance using the Wide World Importers - Standard backup file. This method requires some downtime.
For a tutorial using the Azure Database Migration Service (DMS ) for migration, see Managed Instance migration
using DMS. For a discussion of the varous migration methods, see SQL Server instance migration to Azure SQL
Database Managed Instance.
Prerequisites
This quickstart:
Uses as its starting point the resources created in this quickstart: Create a Managed Instance.
Requires the newest version of SQL Server Management Studio on your on-premises client computer
Requires connectivity to your Managed Instance using SQL Server Management Studio. See these quickstarts
for connectivity options:
Connect to an Azure SQL Database Managed Instance from an Azure VM
Connect to an Azure SQL Database Managed Instance from on-premises using a Point-to-Site
connection.
Uses a preconfigured Azure blob storage account containing the the Wide World Importers - Standard backup
file (downloaded from https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-
importers-v1.0/WideWorldImporters-Standard.bak).
NOTE
For more information about backing up and rsstoring a SQL Server database using Azure blob storage and a Shared Access
Signature (SAS), see SQL Server Backup to URL.
4. Use the following script to create check the SAS credential and backup validity - providing the URL for the
container with the backup file:
5. Use the following script to restore the Wide World Importers database from a backup file - providing the
URL for the container with the backup file:
6. To track the status of your restore, run the following query in a new query session:
SELECT session_id as SPID, command, a.text AS Query, start_time, percent_complete
, dateadd(second,estimated_completion_time/1000, getdate()) as estimated_completion_time
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) a
WHERE r.command in ('BACKUP DATABASE','RESTORE DATABASE')`
Next steps
For troubleshooting backup to URL, see SQL Server Backup to URL Best Practices and Troubleshooting.
For an overview of the connection options for applications, see Connect your applications to Managed
Instance.
To query using one of your favorite tools or languages, see connect and query.
Azure SQL Database Connect and Query Quickstarts
4/25/2018 • 2 minutes to read • Edit Online
The following document includes links to Azure examples showing how to connect and query an Azure SQL
database. It also provides some recommendations for Transport Level Security.
Quickstarts
SQL Server Management Studio This quickstart demonstrates how to use SSMS to connect to
an Azure SQL database, and then use Transact-SQL
statements to query, insert, update, and delete data in the
database.
SQL Operations Studio This quickstart demonstrates how to use SQL Operations
Studio (preview) to connect to an Azure SQL database, and
then use Transact-SQL (T-SQL) statements to create the
TutorialDB used in SQL Operations Studio (preview) tutorials.
Azure portal This quickstart demonstrates how to use the Query editor to
connect to a SQL database, and then use Transact-SQL
statements to query, insert, update, and delete data in the
database.
Visual Studio Code This quickstart demonstrates how to use Visual Studio Code
to connect to an Azure SQL database, and then use Transact-
SQL statements to query, insert, update, and delete data in
the database.
.NET with Visual Studio This quickstart demonstrates how to use the .NET framework
to create a C# program with Visual Studio to connect to an
Azure SQL database and use Transact-SQL statements to
query data.
Next steps
For connectivity architecture information, see Azure SQL Database Connectivity Architecture.
Azure SQL Database: Use SQL Server Management
Studio to connect and query data
9/6/2018 • 4 minutes to read • Edit Online
SQL Server Management Studio (SSMS ) is an integrated environment for managing any SQL infrastructure,
from SQL Server to SQL Database for Microsoft Windows. This quickstart demonstrates how to use SSMS to
connect to an Azure SQL database, and then use Transact-SQL statements to query, insert, update, and delete
data in the database.
Prerequisites
This quickstart uses as its starting point the resources created in one of these quickstarts:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
This quickstart also requires that you configure a server-level firewall rule. For a quickstart showing how to do
this, see Create server-level firewall rule.
Install the latest SSMS
Before you start, make sure you have installed the newest version of SSMS.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
Connect to your database
Use SQL Server Management Studio to establish a connection to your Azure SQL Database server.
IMPORTANT
An Azure SQL Database logical server listens on port 1433. If you are attempting to connect to an Azure SQL Database
logical server from within a corporate firewall, this port must be open in the corporate firewall for you to successfully
connect.
Server name The fully qualified server name The name should be something like
this:
mynewserver20170313.database.
windows.net.
Login The server admin account This is the account that you specified
when you created the server.
Password The password for your server admin This is the password that you
account specified when you created the
server.
3. Click Options in the Connect to server dialog box. In the Connect to database section, enter
mySampleDatabase to connect to this database.
Query data
Use the following code to query for the top 20 products by category using the SELECT Transact-SQL statement.
1. In Object Explorer, right-click mySampleDatabase and click New Query. A blank query window opens that
is connected to your database.
2. In the query window, enter the following query:
3. On the toolbar, click Execute to retrieve data from the Product and ProductCategory tables.
Insert data
Use the following code to insert a new product into the SalesLT.Product table using the INSERT Transact-SQL
statement.
1. In the query window, replace the previous query with the following query:
2. On the toolbar, click Execute to insert a new row in the Product table.
Update data
Use the following code to update the new product that you previously added using the UPDATE Transact-SQL
statement.
1. In the query window, replace the previous query with the following query:
UPDATE [SalesLT].[Product]
SET [ListPrice] = 125
WHERE Name = 'myNewProduct';
2. On the toolbar, click Execute to update the specified row in the Product table.
Delete data
Use the following code to delete the new product that you previously added using the DELETE Transact-SQL
statement.
1. In the query window, replace the previous query with the following query:
DELETE FROM [SalesLT].[Product]
WHERE Name = 'myNewProduct';
2. On the toolbar, click Execute to delete the specified row in the Product table.
Next steps
For information about SSMS, see Use SQL Server Management Studio.
To connect and query using the Azure portal, see Connect and query with the Azure portal SQL Query editor .
To connect and query using Visual Studio Code, see Connect and query with Visual Studio Code.
To connect and query using .NET, see Connect and query with .NET.
To connect and query using PHP, see Connect and query with PHP.
To connect and query using Node.js, see Connect and query with Node.js.
To connect and query using Java, see Connect and query with Java.
To connect and query using Python, see Connect and query with Python.
To connect and query using Ruby, see Connect and query with Ruby.
Azure portal: Use the SQL Query editor to connect
and query data
9/6/2018 • 4 minutes to read • Edit Online
The SQL Query editor is a browser query tool that provides an efficient and lightweight way to execute SQL
queries on your Azure SQL Database or Azure SQL Data Warehouse without leaving the Azure portal. This
quickstart demonstrates how to use the Query editor to connect to a SQL database, and then use Transact-SQL
statements to query, insert, update, and delete data in the database.
Prerequisites
This quickstart uses as its starting point the resources created in one of these quickstarts:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
NOTE
Make sure that the "Allow access to Azure Services" option is set to "ON" in your SQL Server firewall settings. This option
gives the SQL Query editor access to your databases and data warehouses.
4. Click OK to login.
NOTE
Email accounts (for example outlook.com, hotmail.com, live.com, gmail.com, yahoo.com) are not yet supported as Active
Directory administrators. Make sure to choose a user that was either created natively in the Azure Active Directory, or
federated into the Azure Active directory.
1. Select SQL Servers from the left-hand menu, and select your SQL Server from the server list.
2. Select the Active Directory Admin setting from the settings menu of your SQL Server.
3. In the Active Directory admin blade, click the Set admin command, and select the user or group that will
be the Active Directory administrator.
4. At the top of the Active Directory admin blade, click the Save command to set your Active Directory
administrator.
Navigate to the SQL database you would like to query, click Data explorer (preview) from the left-hand menu.
The Data explorer page opens and automatically connects you to the database.
Click Run and then review the query results in the Results pane.
Insert data using Query Editor
Use the following code to insert a new product into the SalesLT.Product table using the INSERT Transact-SQL
statement.
1. In the query window, replace the previous query with the following query:
2. On the toolbar, click Run to insert a new row in the Product table.
2. On the toolbar, click Run to update the specified row in the Product table.
2. On the toolbar, click Run to delete the specified row in the Product table.
Next steps
To learn about the Transact-SQL supported in Azure SQL databases, see Transact-SQL differences in SQL
database.
Azure SQL Database: Use Visual Studio Code to
connect and query data
7/3/2018 • 4 minutes to read • Edit Online
Visual Studio Code is a graphical code editor for Linux, macOS, and Windows that supports extensions, including
the mssql extension for querying Microsoft SQL Server, Azure SQL Database, and SQL Data Warehouse. This
quickstart demonstrates how to use Visual Studio Code to connect to an Azure SQL database, and then use
Transact-SQL statements to query, insert, update, and delete data in the database.
Prerequisites
This quickstart uses as its starting point the resources created in one of these quickstarts:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
Install VS Code
Before you start, make sure you have installed the newest version of Visual Studio Code and loaded the mssql
extension. For installation guidance for the mssql extension, see Install VS Code and see mssql for Visual Studio
Code.
Configure VS Code
Mac OS
For macOS, you need to install OpenSSL which is a prerequiste for .Net Core that mssql extention uses. Open
your terminal and enter the following commands to install brew and OpenSSL.
Linux (Ubuntu)
No special configuration needed.
Windows
No special configuration needed.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
IMPORTANT
Before continuing, make sure that you have your server, database, and login information ready. Once you begin entering
the connection profile information, if you change your focus from Visual Studio Code, you have to restart creating the
connection profile.
1. In VS Code, press CTRL+SHIFT+P (or F1) to open the Command Palette.
2. Type sqlcon and press ENTER.
3. Press ENTER to select Create Connection Profile. This creates a connection profile for your SQL Server
instance.
4. Follow the prompts to specify the connection properties for the new connection profile. After specifying
each value, press ENTER to continue.
**Server name The fully qualified server name The name should be something like
this:
mynewserver20170313.database.
windows.net.
User name The server admin account This is the account that you specified
when you created the server.
Password (SQL Login) The password for your server admin This is the password that you
account specified when you created the
server.
Enter a name for this profile A profile name, such as A saved profile name speeds your
mySampleDatabase connection on subsequent logins.
5. Press the ESC key to close the info message that informs you that the profile is created and connected.
6. Verify your connection in the status bar.
Query data
Use the following code to query for the top 20 products by category using the SELECT Transact-SQL statement.
1. In the Editor window, enter the following query in the empty query window:
2. Press CTRL+SHIFT+E to retrieve data from the Product and ProductCategory tables.
Insert data
Use the following code to insert a new product into the SalesLT.Product table using the INSERT Transact-SQL
statement.
1. In the Editor window, delete the previous query and enter the following query:
Update data
Use the following code to update the new product that you previously added using the UPDATE Transact-SQL
statement.
1. In the Editor window, delete the previous query and enter the following query:
UPDATE [SalesLT].[Product]
SET [ListPrice] = 125
WHERE Name = 'myNewProduct';
Delete data
Use the following code to delete the new product that you previously added using the DELETE Transact-SQL
statement.
1. In the Editor window, delete the previous query and enter the following query:
Next steps
To connect and query using SQL Server Management Studio, see Connect and query with SSMS .
To connect and query using the Azure portal, see Connect and query with the Azure portal SQL query editor .
For an MSDN magazine article on using Visual Studio Code, see Create a database IDE with MSSQL
extension blog post.
Use .NET (C#) with Visual Studio to connect and
query an Azure SQL database
9/6/2018 • 3 minutes to read • Edit Online
This quickstart demonstrates how to use the .NET framework to create a C# program with Visual Studio to
connect to an Azure SQL database and use Transact-SQL statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
An installation of Visual Studio Community 2017, Visual Studio Professional 2017, or Visual Studio
Enterprise 2017.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
For ADO.NET
1. Continue by clicking Show database connection strings.
2. Review the complete ADO.NET connection string.
IMPORTANT
You must have a firewall rule in place for the public IP address of the computer on which you perform this tutorial. If you are
on a different computer or have a different public IP address, create a server-level firewall rule using the Azure portal.
namespace sqltest
{
class Program
{
static void Main(string[] args)
{
try
{
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = "your_server.database.windows.net";
builder.UserID = "your_user";
builder.Password = "your_password";
builder.InitialCatalog = "your_database";
connection.Open();
StringBuilder sb = new StringBuilder();
sb.Append("SELECT TOP 20 pc.Name as CategoryName, p.name as ProductName ");
sb.Append("FROM [SalesLT].[ProductCategory] pc ");
sb.Append("JOIN [SalesLT].[Product] p ");
sb.Append("ON pc.productcategoryid = p.productcategoryid;");
String sql = sb.ToString();
Next steps
Learn how to connect and query an Azure SQL database using .NET core on Windows/Linux/macOS.
Learn about Getting started with .NET Core on Windows/Linux/macOS using the command line.
Learn how to Design your first Azure SQL database using SSMS or Design your first Azure SQL database
using .NET.
For more information about .NET, see .NET documentation.
Retry logic example: Connect resiliently to SQL with ADO.NET
Use .NET Core (C#) to query an Azure SQL database
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to use .NET Core on Windows/Linux/macOS to create a C# program to connect
to an Azure SQL database and use Transact-SQL statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed .NET Core for your operating system.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
For ADO.NET
1. Continue by clicking Show database connection strings.
2. Review the complete ADO.NET connection string.
IMPORTANT
You must have a firewall rule in place for the public IP address of the computer on which you perform this tutorial. If you are
on a different computer or have a different public IP address, create a server-level firewall rule using the Azure portal.
2. Open sqltest.csproj with your favorite text editor and add System.Data.SqlClient as a dependency using the
following code:
<ItemGroup>
<PackageReference Include="System.Data.SqlClient" Version="4.4.0" />
</ItemGroup>
namespace sqltest
{
class Program
{
static void Main(string[] args)
{
try
{
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = "your_server.database.windows.net";
builder.UserID = "your_user";
builder.Password = "your_password";
builder.InitialCatalog = "your_database";
connection.Open();
StringBuilder sb = new StringBuilder();
sb.Append("SELECT TOP 20 pc.Name as CategoryName, p.name as ProductName ");
sb.Append("FROM [SalesLT].[ProductCategory] pc ");
sb.Append("JOIN [SalesLT].[Product] p ");
sb.Append("ON pc.productcategoryid = p.productcategoryid;");
String sql = sb.ToString();
dotnet restore
dotnet run
2. Verify that the top 20 rows are returned and then close the application window.
Next steps
Getting started with .NET Core on Windows/Linux/macOS using the command line.
Learn how to connect and query an Azure SQL database using the .NET framework and Visual Studio.
Learn how to Design your first Azure SQL database using SSMS or Design your first Azure SQL database
using .NET.
For more information about .NET, see .NET documentation.
Use ActiveDirectoryInteractive mode to connect to
Azure SQL Database
7/24/2018 • 8 minutes to read • Edit Online
This article provides a runnable C# code example that connects to your Microsoft Azure SQL Database. The C#
program uses the interactive mode of authentication, which supports Azure AD multi-factor authentication (MFA).
For instance, a connection attempt can include a verification code being sent to your mobile phone.
For more information about MFA support for SQL tools, see Azure Active Directory support in SQL Server Data
Tools (SSDT).
TIP
Our general search page for all kinds of .NET Framework APIs is available at the following link to our handy .NET API
Browser tool:
https://docs.microsoft.com/dotnet/api/
By adding the type name to the optional appended ?term= parameter, the search page can have our result ready and
waiting for us:
https://docs.microsoft.com/dotnet/api/?term=SqlAuthenticationMethod
SqlAuthenticationMethod enum
One namespaces that the C# example relies on is System.Data.SqlClient. Of special interest is the enum
SqlAuthenticationMethod. This enum has the following values:
SqlAuthenticationMethod.ActiveDirectory*Interactive*: Use this with an Azure AD user name, to
achieve multi-factor authentication MFA.
This value is the focus of the present article. It produces an interactive experience by displaying dialogs
for the user password, and then for MFA validation if MFA is imposed on this user.
This value is available starting with .NET Framework version 4.7.2.
SqlAuthenticationMethod.ActiveDirectory*Integrated: Use this for a *federated account. For a
federated account, the user name is known to the Windows domain. This method does not support MFA.
SqlAuthenticationMethod.ActiveDirectory*Password*: Use this for authentication that requires an
Azure AD user and the user's password. Azure SQL Database performs the authentication. This method
does not support MFA.
Depending on your particular scenario, you might not need values all the parameters in the preceding table.
C# code example
To compile this C# example, you must add a reference to the DLL assembly named
Microsoft.IdentityModel.Clients.ActiveDirectory.
Reference documentation
System.Data.SqlClient namespace:
Search: https://docs.microsoft.com/dotnet/api/?term=System.Data.SqlClient
Direct: System.Data.Client
Microsoft.IdentityModel.Clients.ActiveDirectory namespace:
Search: https://docs.microsoft.com/dotnet/api/?term=Microsoft.IdentityModel.Clients.ActiveDirectory
Direct: Microsoft.IdentityModel.Clients.ActiveDirectory
C# source code, in two parts
using System; // C# , part 1 of 2.
using DA = System.Data;
using SC = System.Data.SqlClient;
using AD = Microsoft.IdentityModel.Clients.ActiveDirectory;
using TX = System.Text;
using TT = System.Threading.Tasks;
namespace ADInteractive5
{
class Program
{
// ASSIGN YOUR VALUES TO THESE STATIC FIELDS !!
static public string Az_SQLDB_svrName = "<YOUR VALUE HERE>";
static public string AzureAD_UserID = "<YOUR VALUE HERE>";
static public string Initial_DatabaseName = "master";
// Some scenarios do not need values for the following two fields:
static public readonly string ClientApplicationID = "<YOUR VALUE HERE>";
static public readonly Uri RedirectUri = new Uri("<YOUR VALUE HERE>");
SC.SqlAuthenticationProvider.SetProvider(
SC.SqlAuthenticationMethod.ActiveDirectoryInteractive,
//SC.SqlAuthenticationMethod.ActiveDirectoryIntegrated, // Alternatives.
//SC.SqlAuthenticationMethod.ActiveDirectoryPassword,
provider);
Program.Connection();
}
try
{
sqlConnection.Open();
if (sqlConnection.State == DA.ConnectionState.Open)
{
var rdr = cmd.ExecuteReader();
var msg = new TX.StringBuilder();
while (rdr.Read())
{
{
msg.AppendLine(rdr.GetString(0));
}
Console.WriteLine(msg.ToString());
Console.WriteLine(":Success");
}
else
{
Console.WriteLine(":Failed");
}
sqlConnection.Close();
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("Connection failed with the following exception...");
Console.WriteLine(ex.ToString());
Console.ResetColor();
}
}
} // EOClass Program .
/// <summary>
/// SqlAuthenticationProvider - Is a public class that defines 3 different Azure AD
/// authentication methods. The methods are supported in the new .NET 4.7.2 .
/// .
/// 1. Interactive, 2. Integrated, 3. Password
/// .
/// All 3 authentication methods are based on the Azure
/// Active Directory Authentication Library (ADAL) managed library.
/// </summary>
public class ActiveDirectoryAuthProvider : SC.SqlAuthenticationProvider
{
// Program._ more static values that you set!
private readonly string _clientId = Program.ClientApplicationID;
private readonly Uri _redirectUri = Program.RedirectUri;
switch (parameters.AuthenticationMethod)
{
case SC.SqlAuthenticationMethod.ActiveDirectoryInteractive:
Console.WriteLine("In method 'AcquireTokenAsync', case_0 ==
'.ActiveDirectoryInteractive'.");
case SC.SqlAuthenticationMethod.ActiveDirectoryIntegrated:
Console.WriteLine("In method 'AcquireTokenAsync', case_1 ==
'.ActiveDirectoryIntegrated'.");
case SC.SqlAuthenticationMethod.ActiveDirectoryPassword:
Console.WriteLine("In method 'AcquireTokenAsync', case_2 == '.ActiveDirectoryPassword'.");
[C:\Test\VSProj\ADInteractive5\ADInteractive5\bin\Debug\]
>> ADInteractive5.exe
In method 'AcquireTokenAsync', case_0 == '.ActiveDirectoryInteractive'.
******** MY QUERY RAN SUCCESSFULLY!! ********
:Success
[C:\Test\VSProj\ADInteractive5\ADInteractive5\bin\Debug\]
>>
Next steps
Get-AzureRmSqlServerActiveDirectoryAdministrator
Use Go to query an Azure SQL database
9/7/2018 • 5 minutes to read • Edit Online
This quickstart demonstrates how to use Go to connect to an Azure SQL database. Transact-SQL statements to
query and modify data are also demonstrated.
Prerequisites
To complete this quickstart, make sure you have the following prerequisites:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed Go and related software for your operating system:
MacOS: Install Homebrew and GoLang. See Step 1.2.
Ubuntu: Install GoLang. See Step 1.2.
Windows: Install GoLang. See Step 1.2.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
2. Change directory to SqlServerSample and get and install the SQL Server driver for Go:
cd SqlServerSample
go get github.com/denisenkom/go-mssqldb
go install github.com/denisenkom/go-mssqldb
2. Connect to the database using sqlcmd and run the SQL script to create the schema, table, and insert some
rows. Replace the appropriate values for your server, database, username, and password.
package main
import (
_ "github.com/denisenkom/go-mssqldb"
"database/sql"
"context"
"log"
"fmt"
"errors"
)
var db *sql.DB
func main() {
// Build connection string
connString := fmt.Sprintf("server=%s;user id=%s;password=%s;port=%d;database=%s;",
server, user, password, port, database)
// Create employee
createID, err := CreateEmployee("Jake", "United States")
if err != nil {
log.Fatal("Error creating Employee: ", err.Error())
}
fmt.Printf("Inserted ID: %d successfully.\n", createID)
// Read employees
count, err := ReadEmployees()
if err != nil {
log.Fatal("Error reading Employees: ", err.Error())
}
fmt.Printf("Read %d row(s) successfully.\n", count)
if db == nil {
err = errors.New("CreateEmployee: db is null")
return -1, err
}
tsql := "INSERT INTO TestSchema.Employees (Name, Location) VALUES (@Name, @Location); select
convert(bigint, SCOPE_IDENTITY());"
row := stmt.QueryRowContext(
ctx,
sql.Named("Name", name),
sql.Named("Location", location))
var newID int64
err = row.Scan(&newID)
if err != nil {
return -1, err
}
// Execute query
rows, err := db.QueryContext(ctx, tsql)
if err != nil {
return -1, err
}
defer rows.Close()
return result.RowsAffected()
}
return result.RowsAffected()
}
go run sample.go
Connected!
Inserted ID: 4 successfully.
ID: 1, Name: Jared, Location: Australia
ID: 2, Name: Nikita, Location: India
ID: 3, Name: Tom, Location: Germany
ID: 4, Name: Jake, Location: United States
Read 4 row(s) successfully.
Updated 1 row(s) successfully.
Deleted 1 row(s) successfully.
Next steps
Design your first Azure SQL database
Go Driver for Microsoft SQL Server
Report issues or ask questions
Use Java to query an Azure SQL database
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to use Java to connect to an Azure SQL database and then use Transact-SQL
statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following prerequisites:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed Java and related software for your operating system:
MacOS: Install Homebrew and Java, and then install Maven. See Step 1.2 and 1.3.
Ubuntu: Install the Java Development Kit, and install Maven. See Step 1.2, 1.3, and 1.4.
Windows: Install the Java Development Kit, and Maven. See Step 1.2 and 1.3.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.4.0.jre8</version>
</dependency>
4. Also in pom.xml, add the following properties to your project. If you don't have a properties section, you
can add it after the dependencies.
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
import java.sql.Connection;
import java.sql.Statement;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.DriverManager;
// Connect to database
String hostName = "your_server.database.windows.net";
String dbName = "your_database";
String user = "your_username";
String password = "your_password";
String url =
String.format("jdbc:sqlserver://%s:1433;database=%s;user=%s;password=%s;encrypt=true;hostNameInCertific
ate=*.database.windows.net;loginTimeout=30;", hostName, dbName, user, password);
Connection connection = null;
try {
connection = DriverManager.getConnection(url);
String schema = connection.getSchema();
System.out.println("Successful connection - Schema: " + schema);
mvn package
mvn -q exec:java "-Dexec.mainClass=com.sqldbsamples.App"
2. Verify that the top 20 rows are returned and then close the application window.
Next steps
Design your first Azure SQL database
Microsoft JDBC Driver for SQL Server
Report issues/ask questions
Use Node.js to query an Azure SQL database
9/6/2018 • 3 minutes to read • Edit Online
This quickstart demonstrates how to use Node.js to create a program to connect to an Azure SQL database and
use Transact-SQL statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed Node.js and related software for your operating system:
MacOS: Install Homebrew and Node.js, and then install the ODBC driver and SQLCMD. See Step 1.2
and 1.3.
Ubuntu: Install Node.js, and then install the ODBC driver and SQLCMD. See Step 1.2 and 1.3 .
Windows: Install Chocolatey and Node.js, and then install the ODBC driver and SQL CMD. See Step
1.2 and 1.3.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
IMPORTANT
You must have a firewall rule in place for the public IP address of the computer on which you perform this tutorial. If you are
on a different computer or have a different public IP address, create a server-level firewall rule using the Azure portal.
npm init -y
npm install tedious
npm install async
function queryDatabase()
{ console.log('Reading rows from the Table...');
request.on('row', function(columns) {
columns.forEach(function(column) {
console.log("%s\t%s", column.metadata.colName, column.value);
});
});
connection.execSql(request);
}
node sqltest.js
2. Verify that the top 20 rows are returned and then close the application window.
Next steps
Learn about the Microsoft Node.js Driver for SQL Server
Learn how to connect and query an Azure SQL database using .NET core on Windows/Linux/macOS.
Learn about Getting started with .NET Core on Windows/Linux/macOS using the command line.
Learn how to Design your first Azure SQL database using SSMS or Design your first Azure SQL database
using .NET.
Learn how to Connect and query with SSMS
Learn how to Connect and query with Visual Studio Code.
Use PHP to query an Azure SQL database
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to use PHP to create a program to connect to an Azure SQL database and use
Transact-SQL statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed PHP and related software for your operating system:
MacOS: Install Homebrew and PHP, install the ODBC driver and SQLCMD, and then install the PHP
Driver for SQL Server. See Steps 1.2, 1.3, and 2.1.
Ubuntu: Install PHP and other required packages, and then install the PHP Driver for SQL Server. See
Steps 1.2 and 2.1.
Windows: Install the newest version of PHP for IIS Express, the newest version of Microsoft Drivers for
SQL Server in IIS Express, Chocolatey, the ODBC driver, and SQLCMD. See Steps 1.2 and 1.3.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
Insert code to query SQL database
1. In your favorite text editor, create a new file, sqltest.php.
2. Replace the contents with the following code and add the appropriate values for your server, database, user,
and password.
<?php
$serverName = "your_server.database.windows.net";
$connectionOptions = array(
"Database" => "your_database",
"Uid" => "your_username",
"PWD" => "your_password"
);
//Establishes the connection
$conn = sqlsrv_connect($serverName, $connectionOptions);
$tsql= "SELECT TOP 20 pc.Name as CategoryName, p.name as ProductName
FROM [SalesLT].[ProductCategory] pc
JOIN [SalesLT].[Product] p
ON pc.productcategoryid = p.productcategoryid";
$getResults= sqlsrv_query($conn, $tsql);
echo ("Reading data from table" . PHP_EOL);
if ($getResults == FALSE)
echo (sqlsrv_errors());
while ($row = sqlsrv_fetch_array($getResults, SQLSRV_FETCH_ASSOC)) {
echo ($row['CategoryName'] . " " . $row['ProductName'] . PHP_EOL);
}
sqlsrv_free_stmt($getResults);
?>
php sqltest.php
2. Verify that the top 20 rows are returned and then close the application window.
Next steps
Design your first Azure SQL database
Microsoft PHP Drivers for SQL Server
Report issues or ask questions
Retry logic example: Connect resiliently to SQL with PHP
Use Python to query an Azure SQL database
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to use Python to connect to an Azure SQL database and use Transact-SQL
statements to query data. For further sdk details, checkout our reference documentation, a pyodbc sample, and the
pyodbc GitHub repository.
Prerequisites
To complete this quickstart, make sure you have the following:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed Python and related software for your operating system:
MacOS: Install Homebrew and Python, install the ODBC driver and SQLCMD, and then install the
Python Driver for SQL Server. See Steps 1.2, 1.3, and 2.1.
Ubuntu: Install Python and other required packages, and then install the Python Driver for SQL Server.
See Steps 1.2, 1.3, and 2.1.
Windows: Install the newest version of Python (environment variable is now configured for you), install
the ODBC driver and SQLCMD, and then install the Python Driver for SQL Server. See Step 1.2, 1.3,
and 2.1.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
import pyodbc
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
driver= '{ODBC Driver 13 for SQL Server}'
cnxn =
pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+
password)
cursor = cnxn.cursor()
cursor.execute("SELECT TOP 20 pc.Name as CategoryName, p.name as ProductName FROM [SalesLT].[ProductCategory]
pc JOIN [SalesLT].[Product] p ON pc.productcategoryid = p.productcategoryid")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
python sqltest.py
2. Verify that the top 20 rows are returned and then close the application window.
Next steps
Design your first Azure SQL database
Microsoft Python Drivers for SQL Server
Python Developer Center
Use Ruby to query an Azure SQL database
9/6/2018 • 2 minutes to read • Edit Online
This quickstart demonstrates how to use Ruby to create a program to connect to an Azure SQL database and use
Transact-SQL statements to query data.
Prerequisites
To complete this quickstart, make sure you have the following prerequisites:
An Azure SQL database. You can use one of these techniques to create a database:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
A server-level firewall rule for the public IP address of the computer you use for this quickstart.
You have installed Ruby and related software for your operating system:
MacOS: Install Homebrew, install rbenv and ruby-build, install Ruby, and then install FreeTDS. See Step
1.2, 1.3, 1.4, and 1.5.
Ubuntu: Install prerequisites for Ruby, install rbenv and ruby-build, install Ruby, and then install
FreeTDS. See Step 1.2, 1.3, 1.4, and 1.5.
4. If you forget your server login information, navigate to the SQL Database server page to view the server
admin name. If necessary, reset the password.
IMPORTANT
You must have a firewall rule in place for the public IP address of the computer on which you perform this tutorial. If you are
on a different computer or have a different public IP address, create a server-level firewall rule using the Azure portal.
require 'tiny_tds'
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
client = TinyTds::Client.new username: username, password: password,
host: server, port: 1433, database: database, azure: true
ruby sqltest.rb
2. Verify that the top 20 rows are returned and then close the application window.
Next Steps
Design your first Azure SQL database
GitHub repository for TinyTDS
Report issues or ask questions about TinyTDS
Ruby Drivers for SQL Server
Tutorial: Design your first Azure SQL database using
SSMS
9/6/2018 • 9 minutes to read • Edit Online
Azure SQL Database is a relational database-as-a service (DBaaS ) in the Microsoft Cloud (Azure). In this tutorial,
you learn how to use the Azure portal and SQL Server Management Studio (SSMS ) to:
Create a database in the Azure portal*
Set up a server-level firewall rule in the Azure portal
Connect to the database with SSMS
Create tables with SSMS
Bulk load data with BCP
Query that data with SSMS
If you don't have an Azure subscription, create a free account before you begin.
NOTE
For the purpose of this tutorial, we are using the DTU-based purchasing model, but you do have the option of choosing
the vCore-based purchasing model.
Prerequisites
To complete this tutorial, make sure you have installed:
The newest version of SQL Server Management Studio (SSMS ).
The newest version of BCP and SQLCMD.
4. Click Server to create and configure a new server for your new database. Fill out the New server form
with the following information:
Server name Any globally unique name For valid server names, see Naming
rules and restrictions.
Server admin login Any valid name For valid login names, see Database
Identifiers.
5. Click Select.
6. Click Pricing tier to specify the service tier, the number of DTUs or vCores, and the amount of storage.
Explore the options for the number of DTUs/vCores and storage that is available to you for each service
tier. For the purpose of this tutorial, we are using the DTU -based purchasing model, but you do have the
option of choosing the vCore-based purchasing model.
7. For this tutorial, select the Standard service tier and then use the slider to select 100 DTUs (S3) and 400
GB of storage.
9. After selecting the server tier, the number of DTUs, and the amount of storage, click Apply.
10. Select a collation for the blank database (for this tutorial, use the default value). For more information
about collations, see Collations
11. Now that you have completed the SQL Database form, click Create to provision the database.
Provisioning takes a few minutes.
12. On the toolbar, click Notifications to monitor the deployment process.
NOTE
SQL Database communicates over port 1433. If you are trying to connect from within a corporate network, outbound
traffic over port 1433 may not be allowed by your network's firewall. If so, you cannot connect to your Azure SQL
Database server unless your IT department opens port 1433.
1. After the deployment completes, click SQL databases from the left-hand menu and then click
mySampleDatabase on the SQL databases page. The overview page for your database opens, showing
you the fully qualified server name (such as mynewserver-20170824.database.windows.net) and
provides options for further configuration.
2. Copy this fully qualified server name for use to connect to your server and its databases in subsequent
tutorials and quickstarts.
3. Click Set server firewall on the toolbar. The Firewall settings page for the SQL Database server opens.
4. Click Add client IP on the toolbar to add your current IP address to a new firewall rule. A firewall rule
can open port 1433 for a single IP address or a range of IP addresses.
5. Click Save. A server-level firewall rule is created for your current IP address opening port 1433 on the
logical server.
6. Click OK and then close the Firewall settings page.
You can now connect to the SQL Database server and its databases using SQL Server Management Studio or
another tool of your choice from this IP address using the server admin account created previously.
IMPORTANT
By default, access through the SQL Database firewall is enabled for all Azure services. Click OFF on this page to disable for
all Azure services.
Server name The fully qualified server name The name should be something like
this:
mynewserver20170824.database.
windows.net.
Login The server admin account This is the account that you specified
when you created the server.
Password The password for your server admin This is the password that you
account specified when you created the
server.
3. Click Options in the Connect to server dialog box. In the Connect to database section, enter
mySampleDatabase to connect to this database.
NOTE
You can also use the table designer in SQL Server Management Studio to create and design your tables.
1. In Object Explorer, right-click mySampleDatabase and click New Query. A blank query window opens
that is connected to your database.
2. In the query window, execute the following query to create four tables in your database:
-- Create Person table
You have now loaded sample data into the tables you created earlier.
Query data
Execute the following queries to retrieve information from the database tables. See Writing SQL Queries to
learn more about writing SQL queries. The first query joins all four tables to find all the students taught by
'Dominick Pope' who have a grade higher than 75% in his class. The second query joins all four tables and finds
all courses in which 'Noe Coleman' has ever enrolled.
1. In a SQL Server Management Studio query window, execute the following query:
-- Find the students taught by Dominick Pope who have a grade higher than 75%
SELECT person.FirstName,
person.LastName,
course.Name,
credit.Grade
FROM Person AS person
INNER JOIN Student AS student ON person.PersonId = student.PersonId
INNER JOIN Credit AS credit ON student.StudentId = credit.StudentId
INNER JOIN Course AS course ON credit.CourseId = course.courseId
WHERE course.Teacher = 'Dominick Pope'
AND Grade > 75
-- Find all the courses in which Noe Coleman has ever enrolled
SELECT course.Name,
course.Teacher,
credit.Grade
FROM Course AS course
INNER JOIN Credit AS credit ON credit.CourseId = course.CourseId
INNER JOIN Student AS student ON student.StudentId = credit.StudentId
INNER JOIN Person AS person ON person.PersonId = student.PersonId
WHERE person.FirstName = 'Noe'
AND person.LastName = 'Coleman'
Next steps
In this tutorial, you learned basic database tasks such as create a database and tables, load and query data, and
restore the database to a previous point in time. You learned how to:
Create a database
Set up a firewall rule
Connect to the database with SQL Server Management Studio (SSMS )
Create tables
Bulk load data
Query that data
Advance to the next tutorial to learn about designing a database using Visual Studio and C#.
Design an Azure SQL database and connect with C# and ADO.NET
Design an Azure SQL database and connect with C#
and ADO.NET
6/7/2018 • 10 minutes to read • Edit Online
Azure SQL Database is a relational database-as-a service (DBaaS ) in the Microsoft Cloud (Azure). In this tutorial,
you learn how to use the Azure portal and ADO.NET with Visual Studio to:
Create a database in the Azure portal
Set up a server-level firewall rule in the Azure portal
Connect to the database with ADO.NET and Visual Studio
Create tables with ADO.NET
Insert, update, and delete data with ADO.NET
Query data ADO.NET
If you don't have an Azure subscription, create a free account before you begin.
Prerequisites
An installation of Visual Studio Community 2017, Visual Studio Professional 2017, or Visual Studio Enterprise
2017.
4. Click Server to create and configure a new server for your new database. Fill out the New server form
with the following information:
Server name Any globally unique name For valid server names, see Naming
rules and restrictions.
Server admin login Any valid name For valid login names, see Database
Identifiers.
5. Click Select.
6. Click Pricing tier to specify the service tier, the number of DTUs, and the amount of storage. Explore the
options for the amount of DTUs and storage that is available to you for each service tier.
7. For this tutorial, select the Standard service tier and then use the slider to select 100 DTUs (S3) and 400
GB of storage.
9. After selecting the server tier, the number of DTUs, and the amount of storage, click Apply.
10. Select a collation for the blank database (for this tutorial, use the default value). For more information
about collations, see Collations
11. Click Create to provision the database. Provisioning takes about a minute and a half to complete.
12. On the toolbar, click Notifications to monitor the deployment process.
NOTE
SQL Database communicates over port 1433. If you are trying to connect from within a corporate network, outbound
traffic over port 1433 may not be allowed by your network's firewall. If so, you cannot connect to your Azure SQL Database
server unless your IT department opens port 1433.
1. After the deployment completes, click SQL databases from the left-hand menu and then click
mySampleDatabase on the SQL databases page. The overview page for your database opens, showing
you the fully qualified server name (such as mynewserver20170824.database.windows.net) and
provides options for further configuration.
2. Copy this fully qualified server name for use to connect to your server and its databases in subsequent
quick starts.
3. Click Set server firewall on the toolbar. The Firewall settings page for the SQL Database server opens.
4. Click Add client IP on the toolbar to add your current IP address to a new firewall rule. A firewall rule can
open port 1433 for a single IP address or a range of IP addresses.
5. Click Save. A server-level firewall rule is created for your current IP address opening port 1433 on the
logical server.
6. Click OK and then close the Firewall settings page.
You can now connect to the SQL Database server and its databases using SQL Server Management Studio or
another tool of your choice from this IP address using the server admin account created previously.
IMPORTANT
By default, access through the SQL Database firewall is enabled for all Azure services. Click OFF on this page to disable for
all Azure services.
using System;
using System.Data.SqlClient; // System.Data.dll
//using System.Data; // For: SqlDbType , ParameterDirection
namespace csharp_db_test
{
class Program
{
static void Main(string[] args)
{
try
{
var cb = new SqlConnectionStringBuilder();
cb.DataSource = "your_server.database.windows.net";
cb.UserID = "your_user";
cb.Password = "your_password";
cb.InitialCatalog = "your_database";
Submit_6_Tsql_SelectEmployees(connection);
}
}
catch (SqlException e)
{
Console.WriteLine(e.ToString());
}
Console.WriteLine("View the report output here, then press any key to end the program...");
Console.ReadKey();
}
C# block 7: ExecuteNonQuery
Previous / Next
This method is called for operations that modify the data content of tables without returning any data rows.
=================================
Now, CreateTables (10)...
=================================
Now, Inserts (20)...
=================================
Now, UpdateJoin (30)...
2 rows affected, by UpdateJoin.
=================================
Now, DeleteJoin (40)...
=================================
Now, SelectEmployees (50)...
0199be49-a2ed-4e35-94b7-e936acf1cd75 , Alison , 20 , acct , Accounting
f0d3d147-64cf-4420-b9f9-76e6e0a32567 , Barbara , 17 , hres , Human Resources
cf4caede-e237-42d2-b61d-72114c7e3afa , Carol , 22 , acct , Accounting
cdde7727-bcfd-4f72-a665-87199c415f8b , Elle , 15 , NULL , NULL
[C:\csharp_db_test\csharp_db_test\bin\Debug\]
>>
Next steps
In this tutorial, you learned basic database tasks such as create a database and tables, load and query data, and
restore the database to a previous point in time. You learned how to:
Create a database
Set up a firewall rule
Connect to the database with Visual Studio and C#
Create tables
Insert, update, and delete data
Query data
Advance to the next tutorial to learn about migrating your data.
Migrate your SQL Server database to Azure SQL Database
Migrate your SQL Server database to Azure SQL
Database using DMA
9/6/2018 • 11 minutes to read • Edit Online
Moving your SQL Server database to an Azure SQL Database single database is as simple as creating an empty
SQL database in Azure and then using the Data Migration Assistant (DMA) to import the database into Azure. For
additional migration options, see Migrate your database to Azure SQL Database.
IMPORTANT
To migrate to Azure SQL Database Managed Instance, see Migrate from SQL Server to a Managed Instance
Prerequisites
To complete this tutorial, make sure the following prerequisites are completed:
Installed the newest version of SQL Server Management Studio (SSMS ).
Installed the newest version of the Data Migration Assistant (DMA).
You have identified and have access to a database to migrate. This tutorial uses the SQL Server 2008R2
AdventureWorks OLTP database on an instance of SQL Server 2008R2 or newer, but you can use any database
of your choice. To fix compatibility issues, use SQL Server Data Tools
4. Click Server to create and configure a new server for your new database. Fill out the New server form
with the following information:
Server name Any globally unique name For valid server names, see Naming
rules and restrictions.
Server admin login Any valid name For valid login names, see Database
Identifiers.
5. Click Select.
6. Click Pricing tier to specify the service tier, the number of DTUs, and the amount of storage. Explore the
options for the number of DTUs and storage that is available to you for each service tier.
7. For this tutorial, select the Standard service tier and then use the slider to select 100 DTUs (S3) and 400
GB of storage.
9. After selecting the server tier, the number of DTUs, and the amount of storage, click Apply.
10. Select a collation for the blank database (for this tutorial, use the default value). For more information
about collations, see Collations
11. Now that you have completed the SQL Database form, click Create to provision the database. Provisioning
takes a few minutes.
12. On the toolbar, click Notifications to monitor the deployment process.
NOTE
SQL Database communicates over port 1433. If you are trying to connect from within a corporate network, outbound traffic
over port 1433 may not be allowed by your network's firewall. If so, you cannot connect to your Azure SQL Database server
unless your IT department opens port 1433.
1. After the deployment completes, click SQL databases from the left-hand menu and then click
mySampleDatabase on the SQL databases page. The overview page for your database opens, showing
you the fully qualified server name (such as mynewserver-20170824.database.windows.net) and
provides options for further configuration.
2. Copy this fully qualified server name for use to connect to your server and its databases in subsequent
quickstarts.
3. Click Set server firewall on the toolbar. The Firewall settings page for the SQL Database server opens.
4. Click Add client IP on the toolbar to add your current IP address to a new firewall rule. A firewall rule can
open port 1433 for a single IP address or a range of IP addresses.
5. Click Save. A server-level firewall rule is created for your current IP address opening port 1433 on the
logical server.
6. Click OK and then close the Firewall settings page.
You can now connect to the SQL Database server and its databases using SQL Server Management Studio, Data
Migration Assistant, or another tool of your choice from this IP address using the server admin account created in
the previous procedure.
IMPORTANT
By default, access through the SQL Database firewall is enabled for all Azure services. Click OFF on this page to disable for all
Azure services.
2. In the left-hand menu, click + New to create an Assessment project. Fill in the requested values and then
click Create:
Source server type SQL Server This is the only source currently
supported
SETTING SUGGESTED VALUE DESCRIPTION
Target server type Azure SQL Database Choices include: Azure SQL Database,
SQL Server, SQL Server on Azure
virtual machines
Migration Scope Schema and data Choices include: Schema and data,
schema only, data only
3. On the Select source page, fill in the requested values and then click Connect:
Server name Your server name or IP address Your server name or IP address
Connection properties Select Encrypt connection and Choose the properties appropriate
Trust server certificate as for the connect to your server
appropriate for your environment.
4. Select a single database from your source server to migrate to Azure SQL Database and then click Next.
For this tutorial, there is only a single database.
5. On the Select target page, fill in the requested values and then click Connect:
Server name Your fully qualified Azure Database Your fully qualified Azure Database
server name server name from the previous
procedure
Authentication type SQL Server Authentication SQL Server authentication is the only
option as this tutorial is written, but
Active Directory Integrated
Authentication and Active Directory
Password Authentication are also
supported by Azure SQL Database
Connection properties Select Encrypt connection and Choose the properties appropriate
Trust server certificate as for the connect to your server
appropriate for your environment.
6. Select the database from the target server that you created in a previous procedure and then click Next to
start the source database schema assessment process. For this tutorial, there is only a single database.
Notice that the compatibility level for this database is set to 140, which is the default compatibility level for
all new databases in Azure SQL Database.
IMPORTANT
After you migrate your database to Azure SQL Database, you can choose to operate the database at a specified
compatibility level for backward compatibility purposes. For more information on the implications and options for
operating a database at a specific compatibility level, see ALTER DATABASE Compatibility Level. See also ALTER
DATABASE SCOPED CONFIGURATION for information about additional database-level settings related to
compatibility levels.
7. On the Select objects page, after the source database schema assessment process completes, review the
objects selected for migration and review the objects containing issues. For example, review the
dbo.uspSearchCandidateResumes object for SERVERPROPERTY ('LCID') behavior changes and the
HumanResourcesJobCandidate object for Full-Text Search changes.
IMPORTANT
Depending upon the database's design and your application's design, when you migrate your source database, you
may need to modify either or both your database or your application after migration (and, in some cases, before
migration). For information about Transact-SQL differences that may affect your migration, see Resolving Transact-
SQL differences during migration to SQL Database.
8. Click Generate SQL script to script the schema objects in the source database.
9. Review the generated script and then click Next issue as needed to review the identified assessment issues
and recommendations. For example, for Full-Text Search, the recommendation when you upgrade is to test
your applications leveraging the Full-Text features. You can save or copy the script if you wish.
10. Click Deploy schema and watch the schema migration process.
11. When the schema migration completes, review the results for errors and then, assuming there are none,
click Migrate data.
12. On the Select tables page, review the tables selected for migration and then click Start data migration.
13. Watch the migration process.
Server name The fully qualified server name The name should be something like
this:
mynewserver20170824.database.
windows.net.
Login The server admin account This is the account that you specified
when you created the server.
Password The password for your server admin This is the password that you
account specified when you created the
server.
3. Click Options in the Connect to server dialog box. In the Connect to database section, enter
mySampleDatabase to connect to this database.
4. Click Connect. The Object Explorer window opens in SSMS.
5. In Object Explorer, expand Databases and then expand mySampleDatabase to view the objects in the
sample database.
Next steps
In this tutorial you learned to:
Prerequisites
To complete this tutorial, make sure you have the following:
Installed the newest version of SQL Server Management Studio (SSMS ).
Installed Microsoft Excel
Created an Azure SQL server and database - See Create an Azure SQL database in the Azure portal, Create a
single Azure SQL database using the Azure CLI, and Create a single Azure SQL database using PowerShell.
1. Click SQL databases from the left-hand menu and click the database you would like to configure the
firewall rule for on the SQL databases page. The overview page for your database opens, showing you the
fully qualified server name (such as mynewserver-20170313.database.windows.net) and provides
options for further configuration.
2. Click Set server firewall on the toolbar as shown in the previous image. The Firewall settings page for
the SQL Database server opens.
3. Click Add client IP on the toolbar to add the public IP address of the computer connected to the portal
with or enter the firewall rule manually and then click Save.
4. Click OK and then click the X to close the Firewall settings page.
You can now connect to any database in the server with the specified IP address or IP address range.
NOTE
SQL Database communicates over port 1433. If you are trying to connect from within a corporate network, outbound traffic
over port 1433 may not be allowed by your network's firewall. If so, you will not be able to connect to your Azure SQL
Database server unless your IT department opens port 1433.
Create a database-level firewall rule using SSMS
Database-level firewall rules enable you to create different firewall settings for different databases within the same
logical server and to create firewall rules that are portable - meaning that they follow the database during a
failover rather than being stored on the SQL server. Database-level firewall rules can only be configured by using
Transact-SQL statements and only after you have configured the first server-level firewall rule. For more
information, see Azure SQL Database server-level and database-level firewall rules.
Follows these steps to create a database-specific firewall rule.
1. Connect to your database, for example using SQL Server Management Studio.
2. In Object Explorer, right-click on the database you want to add a firewall rule for and click New Query. A
blank query window opens that is connected to your database.
3. In the query window, modify the IP address to your public IP address and then execute the following query:
It is best practice to create these non-administrator accounts at the database level to connect to your database
unless you need to execute administrator tasks like creating new users. Please review the Azure Active Directory
tutorial on how to authenticate using Azure Active Directory.
3. If you prefer to enable an Audit type (or location?) different from the one specified at the server level, turn
ON Auditing, and choose the Blob Auditing Type. If server Blob auditing is enabled, the database-
configured audit will exist side-by-side with the server Blob audit.
4. Select Storage Details to open the Audit Logs Storage Blade. Select the Azure storage account where logs
will be saved, and the retention period, after which the old logs will be deleted, then click OK at the bottom.
TIP
Use the same storage account for all audited databases to get the most out of the auditing reports templates.
5. Click Save.
IMPORTANT
If you want to customize the audited events, you can do this via PowerShell or REST API - see SQL database auditing for
more information.
If anomalous database activities are detected, you will receive an email notification upon detection of
anomalous database activities. The email will provide information on the suspicious security event including
the nature of the anomalous activities, database name, server name and the event time. In addition, it will
provide information on possible causes and recommended actions to investigate and mitigate the potential
threat to the database. The next steps walk you through what to do should you receive such an email:
6. In the email, click on the Azure SQL Auditing Log link, which will launch the Azure portal and show the
relevant auditing records around the time of the suspicious event.
7. Click on the audit records to view more information on the suspicious database activities such as SQL
statement, failure reason and client IP.
8. In the Auditing Records blade, click Open in Excel to open a pre-configured excel template to import and
run deeper analysis of the audit log around the time of the suspicious event.
NOTE
In Excel 2010 or later, Power Query and the Fast Combine setting is required.
9. To configure the Fast Combine setting - In the POWER QUERY ribbon tab, select Options to display the
Options dialog. Select the Privacy section and choose the second option - 'Ignore the Privacy Levels and
potentially improve performance':
10. To load SQL audit logs, ensure that the parameters in the settings tab are set correctly and then select the
'Data' ribbon and click the 'Refresh All' button.
11. The results appear in the SQL Audit Logs sheet which enables you to run deeper analysis of the
anomalous activities that were detected, and mitigate the impact of the security event in your application.
Next steps
In this tutorial, you learned to improve the protection of your database against malicious users or unauthorized
access with just a few simple steps. You learned how to:
Set up firewall rules for your server and or database
Connect to your database using a secure connection string
Manage user access
Protect your data with encryption
Enable SQL Database auditing
Enable SQL Database threat detection
Advance to the next tutorial to learn how to implement a geo-distributed database.
Implement a geo-distributed database
Implement a geo-distributed database
9/14/2018 • 9 minutes to read • Edit Online
In this tutorial, you configure an Azure SQL database and application for failover to a remote region, and then test
your failover plan. You learn how to:
Create database users and grant them permissions
Set up a database-level firewall rule
Create a geo-replication failover group
Create and compile a Java application to query an Azure SQL database
Perform a disaster recovery drill
If you don't have an Azure subscription, create a free account before you begin.
Prerequisites
To complete this tutorial, make sure the following prerequisites are completed:
Installed the latest Azure PowerShell.
Installed an Azure SQL database. This tutorial uses the AdventureWorksLT sample database with a name of
mySampleDatabase from one of these quick starts:
Create DB - Portal
Create DB - CLI
Create DB - PowerShell
Have identified a method to execute SQL scripts against your database, you can use one of the following
query tools:
The query editor in the Azure portal. For more information on using the query editor in the Azure portal,
see Connect and query using Query Editor.
The newest version of SQL Server Management Studio, which is an integrated environment for
managing any SQL infrastructure, from SQL Server to SQL Database for Microsoft Windows.
The newest version of Visual Studio Code, which is a graphical code editor for Linux, macOS, and
Windows that supports extensions, including the mssql extension for querying Microsoft SQL Server,
Azure SQL Database, and SQL Data Warehouse. For more information on using this tool with Azure
SQL Database, see Connect and query with VS Code.
IMPORTANT
These cmdlets require Azure PowerShell 4.0. This sample requires the Azure PowerShell module version 5.1.1 or later. Run
Get-Module -ListAvailable AzureRM to find the version. If you need to install or upgrade, see Install Azure PowerShell
module. Run Connect-AzureRmAccount to create a connection with Azure.
1. Populate variables for your PowerShell scripts using the values for your existing server and sample
database, and provide a globally unique value for failover group name.
$adminlogin = "ServerAdmin"
$password = "ChangeYourAdminPassword1"
$myresourcegroupname = "<your resource group name>"
$mylocation = "<your resource group location>"
$myservername = "<your existing server name>"
$mydatabasename = "mySampleDatabase"
$mydrlocation = "<your disaster recovery location>"
$mydrservername = "<your disaster recovery server name>"
$myfailovergroupname = "<your unique failover group name>"
$myfailovergroup = New-AzureRMSqlDatabaseFailoverGroup `
–ResourceGroupName $myresourcegroupname `
-ServerName $myservername `
-PartnerServerName $mydrservername `
–FailoverGroupName $myfailovergroupname `
–FailoverPolicy Automatic `
-GracePeriodWithDataLossHours 2
$myfailovergroup
$myfailovergroup = Get-AzureRmSqlDatabase `
-ResourceGroupName $myresourcegroupname `
-ServerName $myservername `
-DatabaseName $mydatabasename | `
Add-AzureRmSqlDatabaseToFailoverGroup `
-ResourceGroupName $myresourcegroupname ` `
-ServerName $myservername `
-FailoverGroupName $myfailovergroupname
$myfailovergroup
For detailed guidance on installing and configuring Java and Maven environment, go the Build an app using SQL
Server, select Java, select MacOS, and then follow the detailed instructions for configuring Java and Maven in
step 1.2 and 1.3.
Linux (Ubuntu)
Open your terminal and navigate to a directory where you plan on creating your Java project. Install Maven by
entering the following commands:
For detailed guidance on installing and configuring Java and Maven environment, go the Build an app using SQL
Server, select Java, select Ubuntu, and then follow the detailed instructions for configuring Java and Maven in
step 1.2, 1.3, and 1.4.
Windows
Install Maven using the official installer. Use Maven to help manage dependencies, build, test, and run your Java
project. For detailed guidance on installing and configuring Java and Maven environment, go the Build an app
using SQL Server, select Java, select Windows, and then follow the detailed instructions for configuring Java and
Maven in step 1.2 and 1.3.
cd SqlDbSamples
4. Using your favorite editor, open the pom.xml file in your project folder.
5. Add the Microsoft JDBC Driver for SQL Server dependency to your Maven project by opening your
favorite text editor and copying and pasting the following lines into your pom.xml file. Do not overwrite the
existing values prepopulated in the file. The JDBC dependency must be pasted within the larger
“dependencies” section ( ).
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.1.0.jre8</version>
</dependency>
6. Specify the version of Java to compile the project against by adding the following “properties” section into
the pom.xml file after the "dependencies" section.
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
7. Add the following "build" section into the pom.xml file after the "properties" section to support manifest
files in jars.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<archive>
<manifest>
<mainClass>com.sqldbsamples.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
package com.sqldbsamples;
import java.sql.Connection;
import java.sql.Statement;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Timestamp;
import java.sql.DriverManager;
import java.util.Date;
import java.util.concurrent.TimeUnit;
try {
for(int i = 1; i < 1000; i++) {
// loop will run for about 1 hour
System.out.print(i + ": insert on primary " + (insertData((highWaterMark +
i))?"successful":"failed"));
TimeUnit.SECONDS.sleep(1);
TimeUnit.SECONDS.sleep(1);
System.out.print(", read from secondary " + (selectData((highWaterMark +
i))?"successful":"failed") + "\n");
TimeUnit.SECONDS.sleep(3);
}
} catch(Exception e) {
e.printStackTrace();
}
}
mvn package
2. When finished, execute the following command to run the application (it runs for about 1 hour unless you
stop it manually):
#######################################
## GEO DISTRIBUTED DATABASE TUTORIAL ##
#######################################
Switch-AzureRMSqlDatabaseFailoverGroup `
-ResourceGroupName $myresourcegroupname `
-ServerName $mydrservername `
-FailoverGroupName $myfailovergroupname
2. Observe the application results during failover. Some inserts fail while the DNS cache refreshes.
3. Find out which role your disaster recovery server is performing.
$mydrserver.ReplicationRole
4. Failback.
Switch-AzureRMSqlDatabaseFailoverGroup `
-ResourceGroupName $myresourcegroupname `
-ServerName $myservername `
-FailoverGroupName $myfailovergroupname
5. Observe the application results during failback. Some inserts fail while the DNS cache refreshes.
6. Find out which role your disaster recovery server is performing.
$fileovergroup = Get-AzureRMSqlDatabaseFailoverGroup `
-FailoverGroupName $myfailovergroupname `
-ResourceGroupName $myresourcegroupname `
-ServerName $mydrservername
$fileovergroup.ReplicationRole
Next steps
In this tutorial, you learned to configure an Azure SQL database and application for failover to a remote region,
and then test your failover plan. You learned how to:
Create database users and grant them permissions
Set up a database-level firewall rule
Create a geo-replication failover group
Create and compile a Java application to query an Azure SQL database
Perform a disaster recovery drill
Advance to the next tutorial to migrate SQL Server to Azure SQL Database Managed Instance using DMS.
Migrate SQL Server to Azure SQL Database Managed Instance using DMS
Migrate SQL Server to Azure SQL Database
Managed Instance offline using DMS
8/24/2018 • 8 minutes to read • Edit Online
You can use the Azure Database Migration Service to migrate the databases from an on-premises SQL Server
instance to an Azure SQL Database Managed Instance. For additional methods that may require some manual
effort, see the article SQL Server instance migration to Azure SQL Database Managed Instance.
IMPORTANT
Migration projects from SQL Server to Azure SQL Database Managed Instance are in preview and subject to the
Supplemental Terms of Use for Microsoft Azure Previews.
In this tutorial, you migrate the Adventureworks2012 database from an on-premises instance of SQL Server to
an Azure SQL Database Managed Instance by using the Azure Database Migration Service.
In this tutorial, you learn how to:
Create an instance of the Azure Database Migration Service.
Create a migration project by using the Azure Database Migration Service.
Run the migration.
Monitor the migration.
Download a migration report.
Prerequisites
To complete this tutorial, you need to:
Create a VNET for the Azure Database Migration Service by using the Azure Resource Manager deployment
model, which provides site-to-site connectivity to your on-premises source servers by using either
ExpressRoute or VPN. Learn network topologies for Azure SQL DB Managed Instance migrations using the
Azure Database Migration Service.
Ensure that your Azure Virtual Network (VNET) Network Security Group rules don't block the following
communication ports 443, 53, 9354, 445, 12000. For more detail on Azure VNET NSG traffic filtering, see the
article Filter network traffic with network security groups.
Configure your Windows Firewall for source database engine access.
Open your Windows Firewall to allow the Azure Database Migration Service to access the source SQL Server,
which by default is TCP port 1433.
If you're running multiple named SQL Server instances using dynamic ports, you may wish to enable the SQL
Browser Service and allow access to UDP port 1434 through your firewalls so that the Azure Database
Migration Service can connect to a named instance on your source server.
If you're using a firewall appliance in front of your source databases, you may need to add firewall rules to
allow the Azure Database Migration Service to access the source database(s) for migration, as well as files via
SMB port 445.
Create an Azure SQL Database Managed Instance by following the detail in the article Create an Azure SQL
Database Managed Instance in the Azure portal.
Ensure that the logins used to connect the source SQL Server and target Managed Instance are members of
the sysadmin server role.
Create a network share that the Azure Database Migration Service can use to back up the source database.
Ensure that the service account running the source SQL Server instance has write privileges on the network
share that you created and that the computer account for the source server has read/write access to the same
share.
Make a note of a Windows user (and password) that has full control privilege on the network share that you
previously created. The Azure Database Migration Service impersonates the user credential to upload the
backup files to Azure storage container for restore operation.
Create a blob container and retrieve its SAS URI by using the steps in the article Manage Azure Blob Storage
resources with Storage Explorer, be sure to select all permissions (Read, Write, Delete, List) on the policy
window while creating the SAS URI. This detail provides the Azure Database Migration Service with access to
your storage account container for uploading the backup files used for migrating databases to Azure SQL
Database Managed Instance.
2. Select the subscription in which you want to create the instance of the Azure Database Migration Service,
and then select Resource providers.
3. Search for migration, and then to the right of Microsoft.DataMigration, select Register.
3. On the Create Migration Service screen, specify a name for the service, the subscription, and a new or
existing resource group.
4. Select the location in which you want to create the instance of DMS.
5. Select an existing virtual network (VNET) or create one.
The VNET provides the Azure Database Migration Service with access to the source SQL Server and
target Azure SQL Database Managed Instance.
For more information on how to create a VNET in Azure portal, see the article Create a virtual network
using the Azure portal.
For additional detail, see the article Network topologies for Azure SQL DB Managed Instance migrations
using the Azure Database Migration Service.
6. Select a pricing tier.
For more information on costs and pricing tiers, see the pricing page.
2. On the Azure Database Migration Service screen, search for the name of the instance that you created,
and then select the instance.
3. Select + New Migration Project.
4. On the New migration project screen, specify a name for the project, in the Source server type text
box, select SQL Server, in the Target server type text box, select Azure SQL Database Managed
Instance, and then for Choose type of activity, select Offline data migration.
5. Select Create to create the project.
SSL connections that are encrypted using a self-signed certificate does not provide strong security. They
are susceptible to man-in-the-middle attacks. You should not rely on SSL using self-signed certificates in a
production environment or on servers that are connected to the internet.
3. Select Save.
4. On the Select source databases screen, select the Adventureworks2012 database for migration.
5. Select Save.
Select logins
1. On the Select logins screen, select the logins that you want to migrate.
NOTE
This release only supports migrating the SQL logins.
2. Select Save.
Choose source backup option Choose the option I will provide latest backup files
when you already have a full backup files available for
DMS to use for database migration. Choose the option I
will let Azure Database Migration Service create
backup files when you want DMS to take the source
database full backup at first and use it for migration.
Network location share The local SMB network share that the Azure Database
Migration Service can take the source database backups
to. The service account running source SQL Server
instance must have write privileges on this network share.
Provide an FQDN or IP addresses of the server in the
network share, for example,
'\\servername.domainname.com\backupfolder' or '\\IP
address\backupfolder'.
User name Make sure that the Windows user has full control
privilege on the network share that you provided above.
The Azure Database Migration Service will impersonate
the user credential to upload the backup files to Azure
storage container for restore operation. If TDE-enabled
databases are selected for migration, the above windows
user must be the built-in administrator account and User
Account Control must be disabled for Azure Database
Migration Service to upload and delete the certificates
files.)
2. Select Save.
4. Select Save.
2. After the migration completes, select Download report to get a report listing the details associated with
the migration process.
3. Verify that the target database on the target Azure SQL Database Managed Instance environment.
Next steps
For a tutorial showing you how to migrate a database to a Managed Instance using the T-SQL RESTORE
command, see Restore a backup to a Managed Instance using the restore command.
For information about Managed Instance, see What is a Managed Instance.
For information about connecting apps to a Managed Instance, see Connect applications.
Set up SQL Data Sync
9/7/2018 • 12 minutes to read • Edit Online
In this tutorial, you learn how to set up Azure SQL Data Sync by creating a hybrid sync group that contains both
Azure SQL Database and SQL Server instances. The new sync group is fully configured and synchronizes on the
schedule you set.
This tutorial assumes that you have at least some prior experience with SQL Database and with SQL Server.
For an overview of SQL Data Sync, see Sync data across multiple cloud and on-premises databases with Azure
SQL Data Sync.
For complete PowerShell examples that show how to configure SQL Data Sync, see the following articles:
Use PowerShell to sync between multiple Azure SQL databases
Use PowerShell to sync between an Azure SQL Database and a SQL Server on-premises database
3. On the SQL databases page, select the existing SQL database that you want to use as the hub database for
Data Sync. The SQL database page opens.
The hub database is the central endpoint of the sync topology, in which a sync group has multiple database
endpoints. All other database endpoints in the same sync group - that is, all member databases - sync with
the hub database.
4. On the SQL database page for the selected database, select Sync to other databases. The Data Sync page
opens.
Create a new Sync Group
1. On the Data Sync page, select New Sync Group. The New sync group page opens with Step 1, Create
sync group, highlighted. The Create Data Sync Group page also opens.
2. On the Create Data Sync Group page, do the following things:
a. In the Sync Group Name field, enter a name for the new sync group.
b. In the Sync Metadata Database section, choose whether to create a new database (recommended)
or to use an existing database.
NOTE
Microsoft recommends that you create a new, empty database to use as the Sync Metadata Database. Data
Sync creates tables in this database and runs a frequent workload. This database is automatically shared as
the Sync Metadata Database for all of your Sync groups in the selected region. You can't change the Sync
Metadata Database or its name without dropping it.
If you chose New database, select Create new database. The SQL Database page opens. On the
SQL Database page, name and configure the new database. Then select OK.
If you chose Use existing database, select the database from the list.
c. In the Automatic Sync section, first select On or Off.
If you chose On, in the Sync Frequency section, enter a number and select Seconds, Minutes,
Hours, or Days.
d. In the Conflict Resolution section, select "Hub wins" or "Member wins."
"Hub wins" means that, when a conflict occurs, the data in the hub database overwrites the
conflicting data in the member database. "Member wins" means that, when a conflict occurs, the data
in the member database overwrites the conflicting data in the hub database.
e. Select OK and wait for the new sync group to be created and deployed.
6. In the Username and Password fields, enter the existing credentials for the SQL Database server on which
the member database is located. Don't enter new credentials in this section.
7. Select OK and wait for the new sync member to be created and deployed.
Add an on-premises SQL Server database
In the Member Database section, optionally add an on-premises SQL Server to the sync group by selecting Add
an On-Premises Database. The Configure On-Premises page opens.
On the Configure On-Premises page, do the following things:
1. Select Choose the Sync Agent Gateway. The Select Sync Agent page opens.
2. On the Choose the Sync Agent Gateway page, choose whether to use an existing agent or create a new
agent.
If you chose Existing agents, select the existing agent from the list.
If you chose Create a new agent, do the following things:
a. Download the client sync agent software from the link provided and install it on the computer where
the SQL Server is located.
IMPORTANT
You have to open outbound TCP port 1433 in the firewall to let the client agent communicate with the
server.
NOTE
If you get a firewall error at this point, you have to create a firewall rule on Azure to allow incoming traffic
from the SQL Server computer. You can create the rule manually in the portal, but you may find it easier to
create it in SQL Server Management Studio (SSMS). In SSMS, try to connect to the hub database on Azure.
Enter its name as <hub_database_name>.database.windows.net. To configure the Azure firewall rule, follow
the steps in the dialog box. Then return to the Client Sync Agent app.
i. In the Client Sync Agent app, click Register to register a SQL Server database with the agent. The
SQL Server Configuration dialog box opens.
j. In the SQL Server Configuration dialog box, choose whether to connect by using SQL Server
authentication or Windows authentication. If you chose SQL Server authentication, enter the existing
credentials. Provide the SQL Server name and the name of the database that you want to sync.
Select Test connection to test your settings. Then select Save. The registered database appears in
the list.
k. You can now close the Client Sync Agent app.
l. In the portal, on the Configure On-Premises page, select Select the Database. The Select
Database page opens.
m. On the Select Database page, in the Sync Member Name field, provide a name for the new sync
member. This name is distinct from the name of the database itself. Select the database from the list.
In the Sync Directions field, select Bi-directional Sync, To the Hub, or From the Hub.
n. Select OK to close the Select Database page. Then select OK to close the Configure On-Premises
page and wait for the new sync member to be created and deployed. Finally, click OK to close the
Select sync members page.
3. To connect to SQL Data Sync and the local agent, add your user name to the role DataSync_Executor . Data
Sync creates this role on the SQL Server instance.
Next steps
Congratulations. You have created a sync group that includes both a SQL Database instance and a SQL Server
database.
For more info about SQL Data Sync, see:
Sync data across multiple cloud and on-premises databases with Azure SQL Data Sync
Best practices for Azure SQL Data Sync
Monitor Azure SQL Data Sync with Log Analytics
Troubleshoot issues with Azure SQL Data Sync
Complete PowerShell examples that show how to configure SQL Data Sync:
Use PowerShell to sync between multiple Azure SQL databases
Use PowerShell to sync between an Azure SQL Database and a SQL Server on-premises database
Download the SQL Data Sync REST API documentation
For more info about SQL Database, see:
SQL Database Overview
Database Lifecycle Management
Azure CLI samples for Azure SQL Database
9/6/2018 • 2 minutes to read • Edit Online
The following table includes links to Azure CLI script examples for Azure SQL Database.
Create a single database and configure a firewall rule This CLI script example creates a single Azure SQL database
and configures a server-level firewall rule.
Create elastic pools and move pooled databases This CLI script example creates SQL elastic pools, and moves
pooled Azure SQL databases, and changes performance
levels.
Scale a single database This CLI script example scales a single Azure SQL database to
a different performance level after querying the size
information for the database.
Scale an elastic pool This CLI script example scales a SQL elastic pool to a different
performance level.
Create and manage a Managed Instance These CLI scripts show you have to create and manage a
Managed Instance using the Azure CLI
Azure PowerShell samples for Azure SQL Database
9/6/2018 • 2 minutes to read • Edit Online
The following table includes links to sample Azure PowerShell scripts for Azure SQL Database.
Create a single database and configure a firewall rule This PowerShell script creates a single Azure SQL database
and configures a server-level firewall rule.
Create elastic pools and move pooled databases This PowerShell script creates Azure SQL Database elastic
pools, and moves pooled databases, and changes
performance levels.
Create and manage a Managed Instance These CLI scripts show you have to create and manage a
Managed Instance using the Azure PowerShell
Configure and failover a single database using active geo- This PowerShell script configures active geo-replication for a
replication single Azure SQL database and fails it over to the secondary
replica.
Configure and failover a pooled database using active geo- This PowerShell script configures active geo-replication for an
replication Azure SQL database in a SQL elastic pool, and fails it over to
the secondary replica.
Configure and failover a failover group for a single database This PowerShell script configures a failover group for an Azure
SQL Database server instance, adds a database to the failover
group, and fails it over to the secondary server
Scale a single database This PowerShell script monitors the performance metrics of an
Azure SQL database, scales it to a higher performance level
and creates an alert rule on one of the performance metrics.
Scale an elastic pool This PowerShell script monitors the performance metrics of an
Azure SQL Database elastic pool, scales it to a higher
performance level, and creates an alert rule on one of the
performance metrics.
Configure auditing and threat-detection This PowerShell script configures auditing and threat
detection policies for an Azure SQL database.
Copy a database to new server This PowerShell script creates a copy of an existing Azure SQL
database in a new Azure SQL server.
Import a database from a bacpac file This PowerShell script imports a database to an Azure SQL
server from a bacpac file.
Sync data between SQL databases This PowerShell script configures Data Sync to sync between
multiple Azure SQL databases.
Sync data between SQL Database and SQL Server on- This PowerShell script configures Data Sync to sync between
premises an Azure SQL database and a SQL Server on-premises
database.
Update the SQL Data Sync sync schema This PowerShell script adds or removes items from the Data
Sync sync schema.
Azure SQL Database purchasing models and
resources
9/12/2018 • 7 minutes to read • Edit Online
Azure SQL Database enables you to easily purchase fully managed PaaS database engine that fits your
performance and cost needs. Depending on the deployment model of Azure SQL Database, you can select the
purchasing model that fits your needs:
Logical servers in Azure SQL Database offers two purchasing models for compute, storage, and IO resources: a
DTU -based purchasing model and a vCore-based purchasing model.
Managed Instances in Azure SQL Database only offer the vCore-based purchasing model.
The following table and chart compare and contrast these two purchasing models.
DTU-based model This model is based on a bundled Best for customers who want simple,
measure of compute, storage, and IO pre-configured resource options.
resources. Performance levels are
expressed in terms of Database
Transaction Units (DTUs) for single
databases and elastic Database
Transaction Units (eDTUs) for elastic
pools. For more on DTUs and eDTUs,
see What are DTUs and eDTUs?
vCore-based model This model allows you to independently Best for customers who value flexibility,
choose compute and storage resources. control, and transparency.
It also allows you to use Azure Hybrid
Benefit for SQL Server to gain cost
savings.
IMPORTANT
Compute, IOs, data and log storage are charged per database or elastic pool. Backups storage is charged per each database.
For details of Managed Instance charges, refer to Azure SQL Database Managed Instance. Region limitations: The vCore-
based purchasing model is not yet available in the following regions: West Europe, France Central, UK South, UK West and
Australia Southeast.
If your database or elastic pool consumes more than 300 DTU conversion to vCore may reduce your cost. You can
convert using your API of choice or using the Azure portal, with no downtime. However, conversion is not
required. If the DTU purchasing model meets your performance and business requirements, you should continue
using it. If you decide to convert from the DTU -model to vCore-model, you should select the performance level
using the following rule of thumb: each 100 DTU in Standard tier requires at least 1 vCore in General Purpose tier;
each 125 DTU in Premium tier requires at least 1 vCore in Business Critical tier.
A pool is given a set number of eDTUs for a set price. Within the elastic pool, individual databases are given the
flexibility to auto-scale within the configured boundaries. A database under heavier load will consume more eDTUs
to meet demand. Databases under lighter loads will consume less eDTUs. Databases with no load will consume no
eDTUs. By provisioning resources for the entire pool, rather than per database, management tasks are simplified,
providing a predictable budget for the pool.
Additional eDTUs can be added to an existing pool with no database downtime and with no impact on the
databases in the pool. Similarly, if extra eDTUs are no longer needed, they can be removed from an existing pool at
any point in time. You can add or subtract databases to the pool or limit the amount of eDTUs a database can use
under heavy load to reserve eDTUs for other databases. If a database is predictably under-utilizing resources, you
can move it out of the pool and configure it as a single database with a predictable amount of required resources.
How can I determine the number of DTUs needed by my workload?
If you are looking to migrate an existing on-premises or SQL Server virtual machine workload to Azure SQL
Database, you can use the DTU Calculator to approximate the number of DTUs needed. For an existing Azure SQL
Database workload, you can use SQL Database Query Performance Insight to understand your database resource
consumption (DTUs) to gain deeper insight for optimizing your workload. You can also use the sys.dm_db_
resource_stats DMV to view resource consumption for the last hour. Alternatively, the catalog view
sys.resource_stats displays resource consumption for the last 14 days, but at a lower fidelity of five-minute
averages.
How do I know if I could benefit from an elastic pool of resources?
Pools are suited for a large number of databases with specific utilization patterns. For a given database, this pattern
is characterized by a low utilization average with relatively infrequent utilization spikes. SQL Database
automatically evaluates the historical resource usage of databases in an existing SQL Database server and
recommends the appropriate pool configuration in the Azure portal. For more information, see when should an
elastic pool be used?
Next steps
For vCore-based purchasing model, see vCore-based purchasing model
For the DTU -based purchasing model, see DTU -based purchasing model.
Choosing a vCore service tier, compute, memory,
storage, and IO resources
8/30/2018 • 7 minutes to read • Edit Online
The vCore-based purchasing model enables you to independently scale compute and storage resources,
match on-premises performance, and optimize price. It also enables you to choose generation of hardware:
Gen 4 - Up to 24 logical CPUs based on Intel E5-2673 v3 (Haswell) 2.4 GHz processors, vCore = 1 PP
(physical core), 7 GB per core, attached SSD
Gen 5 - Up to 80 logical CPUs based on Intel E5-2673 v4 (Broadwell) 2.3 GHz processors, vCore=1 LP
(hyper-thread), 5.1. GB per core, fast eNVM SSD
vCore model also allows you to use Azure Hybrid Use Benefit for SQL Server to gain cost savings.
Best for Most business workloads. Offers Business applications with high IO
budget oriented balanced and requirements. Offers highest
scalable compute and storage resilience to failures using several
options. isolated replicas.
IO throughput (approximate) Single Database: 500 IOPS per vCore 5000 IOPS per core with 200000
with 7000 maximum IOPS maximum IOPS
Managed Instance: Depends on size
of file
For more information, see vCore resource limits in Singelton Database and vCore resource limits in
Managed Instance.
IMPORTANT
If you need less than one vCore of compute capacity, use the DTU-based purchasing model.
Storage considerations
Consider the following:
The allocated storage is used by data files (MDF ) and log files (LDF ) files.
Each single database performance level supports a maximum database size, with a default max size of 32
GB.
When you configure the required single database size (size of MDF ), 30% of additional storage is
automatically added to support LDF
Storage size in Managed Instance must be specified in multiples of 32 GB.
You can select any singleton database size between 10 GB and the supported maximum
For Standard storage, increase or decrease size in 10-GB increments
For Premium storage, increase or decrease size in 250-GB increments
In the General Purpose service tier, tempdb uses an attached SSD and this storage cost is included in the
vCore price.
In the Business Critical service tier, tempdb shares the attached SSD with the MDF and LDF files and the
tempDB storage cost is included in the vCore price.
IMPORTANT
You are charged for the total storage allocated for MDF and LDF.
To monitor the current total size of MDF and LDF, use sp_spaceused. To monitor the current size of the
individual MDF and LDF files, use sys.database_files.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
CURRENT SERVICE TIER TARGET SERVICE TIER MIGRATION TYPE USER ACTIONS
* Each 100 DTU in Standard tier requires at least 1 vCore and each 125 DTU in Premium tier requires at
least 1 vCore
Migration of failover groups
Migration of failover groups with multiple databases requires individual migration of the primary and
secondary databases. During that process, the same considerations and sequencing rules apply. After the
databases are converted to the vCore-based model, the failover group will remain in effect with the same
policy settings.
Creation of a geo -replication secondary
You can only create a geo-secondary using the same service tier as the primary. For database with high log
generation rate, it is advised that the secondary is created with the same performance level as the primary. If
you are creating a geo-secondary in the elastic pool for a single primary database, it is advised that the pool
has the maxVCore setting that matches the primary database performance level. If you are creating a geo-
secondary in the elastic pool for a primary in another elastic pool, it is advised that the pools have the same
maxVCore settings
Using database copy to convert a DTU -based database to a vCore -based database.
You can copy any database with a DTU -based performance level to a database with a vCore-based
performance level without restrictions or special sequencing as long as the target performance level
supports the maximum database size of the source database. This is because the database copy creates a
snapshot of data as of the starting time of the copy operation and does not perform data synchronization
between the source and the target.
Next steps
For details on specific performance levels and storage size choices available for single database, see SQL
Database vCore-based resource limits for single databases
For details on specific performance levels and storage size choices available for elastic pools see SQL
Database vCore-based resource limits for elastic pools.
Choosing a DTU-based service tier, performance
level, and storage resources
8/20/2018 • 7 minutes to read • Edit Online
Service tiers are differentiated by a range of performance levels with a fixed amount of included storage,
fixed retention period for backups, and fixed price. All service tiers provide flexibility of changing
performance levels without downtime. Single databases and elastic pools are billed hourly based on service
tier and performance level.
IMPORTANT
SQL Database Managed Instance, currently in public preview does not support a DTU-based purchasing model. For
more information, see Azure SQL Database Managed Instance.
IO throughput 2.5 IOPS per DTU 2.5 IOPS per DTU 48 IOPS per DTU
(approximate)
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
IMPORTANT
More than 1 TB of storage in the Premium tier is currently available in all regions except the following: West Central
US, China East, USDoDCentral, Germany Central, USDoDEast, US Gov Southwest, USGov Iowa, Germany Northeast,
China North. In other regions, the storage max in the Premium tier is limited to 1 TB. See P11-P15 Current
Limitations.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
DTU Benchmark
Physical characteristics (CPU, memory, IO ) associated to each DTU measure are calibrated using a
benchmark that simulates real-world database workload.
Correlating benchmark results to real world database performance
It is important to understand that all benchmarks are representative and indicative only. The transaction
rates achieved with the benchmark application will not be the same as those that might be achieved with
other applications. The benchmark comprises a collection of different transaction types run against a
schema containing a range of tables and data types. While the benchmark exercises the same basic
operations that are common to all OLTP workloads, it does not represent any specific class of database or
application. The goal of the benchmark is to provide a reasonable guide to the relative performance of a
database that might be expected when scaling up or down between performance levels. In reality,
databases are of different sizes and complexity, encounter different mixes of workloads, and will respond in
different ways. For example, an IO -intensive application may hit IO thresholds sooner, or a CPU -intensive
application may hit CPU limits sooner. There is no guarantee that any particular database will scale in the
same way as the benchmark under increasing load.
The benchmark and its methodology are described in more detail below.
Benchmark summary
ASDB measures the performance of a mix of basic database operations that occur most frequently in
online transaction processing (OLTP ) workloads. Although the benchmark is designed with cloud
computing in mind, the database schema, data population, and transactions have been designed to be
broadly representative of the basic elements most commonly used in OLTP workloads.
Schema
The schema is designed to have enough variety and complexity to support a broad range of operations.
The benchmark runs against a database comprised of six tables. The tables fall into three categories: fixed-
size, scaling, and growing. There are two fixed-size tables; three scaling tables; and one growing table.
Fixed-size tables have a constant number of rows. Scaling tables have a cardinality that is proportional to
database performance, but doesn’t change during the benchmark. The growing table is sized like a scaling
table on initial load, but then the cardinality changes in the course of running the benchmark as rows are
inserted and deleted.
The schema includes a mix of data types, including integer, numeric, character, and date/time. The schema
includes primary and secondary keys, but not any foreign keys - that is, there are no referential integrity
constraints between tables.
A data generation program generates the data for the initial database. Integer and numeric data is
generated with various strategies. In some cases, values are distributed randomly over a range. In other
cases, a set of values is randomly permuted to ensure that a specific distribution is maintained. Text fields
are generated from a weighted list of words to produce realistic looking data.
The database is sized based on a “scale factor.” The scale factor (abbreviated as SF ) determines the
cardinality of the scaling and growing tables. As described below in the section Users and Pacing, the
database size, number of users, and maximum performance all scale in proportion to each other.
Transactions
The workload consists of nine transaction types, as shown in the table below. Each transaction is designed
to highlight a particular set of system characteristics in the database engine and system hardware, with
high contrast from the other transactions. This approach makes it easier to assess the impact of different
components to overall performance. For example, the transaction “Read Heavy” produces a significant
number of read operations from disk.
Workload mix
Transactions are selected at random from a weighted distribution with the following overall mix. The overall
mix has a read/write ratio of approximately 2:1.
Read Lite 35
Read Medium 20
Read Heavy 5
Update Lite 20
Update Heavy 3
Insert Lite 3
Insert Heavy 2
Delete 2
CPU Heavy 10
Next steps
For details on specific performance levels and storage size choices available for single databases, see
SQL Database DTU -based resource limits for single databases.
For details on specific performance levels and storage size choices available for elastic pools, see SQL
Database DTU -based resource limits.
Overview Azure SQL Database resource limits
8/20/2018 • 3 minutes to read • Edit Online
This article provides an overview of the Azure SQL Database resource limits and provides information regarding
what happens when those resource limits are hit or exceeded.
NOTE
To obtain more DTU /eDTU quota, vCore quota, or more servers than the default amount, a new support request can be
submitted in the Azure portal for the subscription with issue type “Quota”. The DTU / eDTU quota and database limit per
server constrains the number of elastic pools per server.
IMPORTANT
As the number of databases approaches the limit per server, the following can occur:
Increasing latency in running queries against the master database. This includes views of resource utilization statistics
such as sys.resource_stats.
Increasing latency in management operations and rendering portal viewpoints that involve enumerating databases in the
server.
Next steps
See SQL Database FAQ for answers to frequently asked questions.
For information about general Azure limits, see Azure subscription and service limits, quotas, and constraints.
For information about DTUs and eDTUs, see DTUs and eDTUs.
For information about tempdb size limits, see https://docs.microsoft.com/sql/relational-
databases/databases/tempdb-database#tempdb-database-in-sql-database.
Prepay for SQL Database compute resources with
Azure SQL Database reserved capacity
9/6/2018 • 5 minutes to read • Edit Online
Save money with Azure SQL Database by prepaying for Azure SQL Database compute resources compared to
pay-as-you-go prices. With Azure SQL Database reserved capacity, you make an upfront commitment on SQL
Database for a period of one or three years to get a significant discount on the compute costs. To purchase SQL
Database reserved capacity, you need to specify the Azure region, deployment type, performance tier, and term.
You do not need to assign the reservation to SQL Database instances. Matching SQL Database instances, that are
already running or ones that are newly deployed, will automatically get the benefit. By purchasing a reservation,
you are pre-paying for the compute costs for the SQL Database instances for a period of one or three years. As
soon as you buy a reservation, the SQL Database compute charges that match the reservation attributes are no
longer charged at the pay-as-you go rates. A reservation does not cover software, networking, or storage charges
associated with the SQL Database instance. At the end of the reservation term, the billing benefit expires and the
SQL Databases are billed at the pay-as-you go price. Reservations do not auto-renew. For pricing information, see
the SQL Database reserved capacity offering.
You can buy Azure SQL Database reserved capacity in the Azure portal. To buy SQL Database reserved capacity:
You must be in the Owner role for at least one Enterprise or Pay-As-You-Go subscription.
For Enterprise subscriptions, Azure reservation purchases must be enabled in the EA portal.
For Cloud Solution Provider (CSP ) program, only the admin agents or sales agents can purchase SQL Database
reserved capacity.
The details on how enterprise customers and Pay-As-You-Go customers are charged for reservation purchases is
covered in Frequently asked questions.
FIELD DESCRIPTION
Deployment Type The SQL deployment type that you want to buy the
reservation for.
Performance Tier The performance tier for the SQL Database instances.
5. Review the cost of the SQL Database reserved capacity reservation in the Costs section.
6. Select Purchase.
7. Select View this Reservation to see the status of your purchase.
Next steps
The vCore reservation discount is applied automatically to the number of SQL Database instances that match the
SQL Database reserved capacity reservation scope and attributes. You can update the scope of the SQL Database
reserved capacity reservation through Azure portal, PowerShell, CLI or through the API.
To learn how to manage the SQL Database reserved capacity reservation, see Manage SQL Database reserved
capacity.
To learn more about Azure Reservations, see the following articles:
What are Azure Reservations?
Manage Azure Reservations
Understand Azure Reservations discount
Understand reservation usage for your Pay-As-You-Go subscription
Understand reservation usage for your Enterprise enrollment
Azure Reservations in Partner Center Cloud Solution Provider (CSP ) program
Manage Azure SQL servers, databases, and firewalls using the Azure
portal
You can create the Azure SQL database's resource group ahead of time or while creating the server itself. There
are multiple methods for getting to a new SQL server form, either by creating a new SQL server or as part of
creating a new database.
Create a blank SQL server (logical server)
To create an Azure SQL Database server (without a database) using the Azure portal, navigate to a blank SQL
server (logical server) form.
Create a blank or sample SQL database
To create an Azure SQL database using the Azure portal, navigate to a blank SQL Database form and provide the
requested information. You can create the Azure SQL database's resource group and logical server ahead of time
or while creating the database itself. You can create a blank database or create a sample database based on
Adventure Works LT.
IMPORTANT
For information on selecting the pricing tier for your database, see DTU-based purchasing model and vCore-based
purchasing model.
IMPORTANT
To configure performance properties for a database, see DTU-based purchasing model and vCore-based purchasing model.
TIP
For an Azure portal quickstart, see Create an Azure SQL database in the Azure portal.
CMDLET DESCRIPTION
TIP
For a PowerShell quickstart, see Create a single Azure SQL database using PowerShell. For PowerShell example scripts, see
Use PowerShell to create a single Azure SQL database and configure a firewall rule and Monitor and scale a single SQL
database using PowerShell.
Manage Azure SQL servers, databases, and firewalls using the Azure
CLI
To create and manage Azure SQL server, databases, and firewalls with the Azure CLI, use the following Azure CLI
SQL Database commands. Use the Cloud Shell to run the CLI in your browser, or install it on macOS, Linux, or
Windows. For creating and managing elastic pools, see Elastic pools.
CMDLET DESCRIPTION
az sql db list Lists all databases and data warehouses in a server, or all
databases in an elastic pool
TIP
For an Azure CLI quickstart, see Create a single Azure SQL database using the Azure CLI. For Azure CLI example scripts, see
Use CLI to create a single Azure SQL database and configure a firewall rule and Use CLI to monitor and scale a single SQL
database.
IMPORTANT
You cannot create or delete a server using Transact-SQL.
COMMAND DESCRIPTION
CREATE DATABASE (Azure SQL Database) Creates a new database. You must be connected to the
master database to create a new database.
ALTER DATABASE (Azure SQL Data Warehouse) Modifies an Azure SQL Data Warehouse.
sys.database_service_objectives (Azure SQL Database) Returns the edition (service tier), service objective (pricing
tier), and elastic pool name, if any, for an Azure SQL database
or an Azure SQL Data Warehouse. If logged on to the master
database in an Azure SQL Database server, returns
information on all databases. For Azure SQL Data Warehouse,
you must be connected to the master database.
sys.dm_db_resource_stats (Azure SQL Database) Returns CPU, IO, and memory consumption for an Azure SQL
Database database. One row exists for every 15 seconds,
even if there is no activity in the database.
sys.resource_stats (Azure SQL Database) Returns CPU usage and storage data for an Azure SQL
Database. The data is collected and aggregated within five-
minute intervals.
sys.database_connection_stats (Azure SQL Database) Contains statistics for SQL Database database connectivity
events, providing an overview of database connection
successes and failures.
sys.event_log (Azure SQL Database) Returns successful Azure SQL Database database connections,
connection failures, and deadlocks. You can use this
information to track or troubleshoot your database activity
with SQL Database.
sp_set_firewall_rule (Azure SQL Database) Creates or updates the server-level firewall settings for your
SQL Database server. This stored procedure is only available
in the master database to the server-level principal login. A
server-level firewall rule can only be created using Transact-
SQL after the first server-level firewall rule has been created
by a user with Azure-level permissions
sys.firewall_rules (Azure SQL Database) Returns information about the server-level firewall settings
associated with your Microsoft Azure SQL Database.
sp_delete_firewall_rule (Azure SQL Database) Removes server-level firewall settings from your SQL
Database server. This stored procedure is only available in the
master database to the server-level principal login.
sp_set_database_firewall_rule (Azure SQL Database) Creates or updates the database-level firewall rules for your
Azure SQL Database or SQL Data Warehouse. Database
firewall rules can be configured for the master database, and
for user databases on SQL Database. Database firewall rules
are useful when using contained database users.
sys.database_firewall_rules (Azure SQL Database) Returns information about the database-level firewall settings
associated with your Microsoft Azure SQL Database.
sp_delete_database_firewall_rule (Azure SQL Database) Removes database-level firewall setting from your Azure SQL
Database or SQL Data Warehouse.
TIP
For a quickstart using SQL Server Management Studio on Microsoft Windows, see Azure SQL Database: Use SQL Server
Management Studio to connect and query data. For a quickstart using Visual Studio Code on the macOS, Linux, or
Windows, see Azure SQL Database: Use Visual Studio Code to connect and query data.
Manage Azure SQL servers, databases, and firewalls using the REST
API
To create and manage Azure SQL server, databases, and firewalls, use these REST API requests.
COMMAND DESCRIPTION
Next steps
To learn about migrating a SQL Server database to Azure, see Migrate to Azure SQL Database.
For information about supported features, see Features.
Azure SQL Database Connectivity Architecture
9/14/2018 • 5 minutes to read • Edit Online
This article explains the Azure SQL Database connectivity architecture and explains how the different components
function to direct traffic to your instance of Azure SQL Database. These Azure SQL Database connectivity
components function to direct network traffic to the Azure database with clients connecting from within Azure and
with clients connecting from outside of Azure. This article also provides script samples to change how connectivity
occurs, and the considerations related to changing the default connectivity settings.
Connectivity architecture
The following diagram provides a high-level overview of the Azure SQL Database connectivity architecture.
The following steps describe how a connection is established to an Azure SQL database through the Azure SQL
Database software load-balancer (SLB ) and the Azure SQL Database gateway.
Clients within Azure or outside of Azure connect to the SLB, which has a public IP address and listens on port
1433.
The SLB directs traffic to the Azure SQL Database gateway.
The gateway redirects the traffic to the correct proxy middleware.
The proxy middleware redirects the traffic to the appropriate Azure SQL database.
IMPORTANT
Each of these components has distributed denial of service (DDoS) protection built-in at the network and the app layer.
IMPORTANT
When using service endpoints with Azure SQL Database your policy is Proxy by default. To enable connectivity from inside
your Vnet, allow outbound connections to the Azure SQL Database Gateway IP addresses specified in the list below. When
using service endpoints we highly recommend changing your connection policy to Redirect to enable better performance. If
you change your connection policy to Redirect it will not be sufficient to allow outbound on your NSG to Azure SQLDB
gateway IPs listed below, you must allow outbound to all Azure SQLDB IPs. This can be accomplished with the help of NSG
(Network Security Groups) Service Tags. For more information, see Service Tags.
UK North 13.87.97.210
UK South 1 51.140.184.11
UK South 2 13.87.34.7
UK West 51.141.8.11
West US 2 13.66.226.202
The following PowerShell script shows how to change the connection policy.
Connect-AzureRmAccount
Select-AzureRmSubscription -SubscriptionName <Subscription Name>
# Subscription ID
$subscriptionId = "<Subscription GUID>"
# Create an App Registration in Azure Active Directory. Ensure the application type is set to NATIVE
# Under Required Permissions, add the API: Windows Azure Service Management API
$authHeader = @{
'Content-Type'='application\json; '
'Authorization'=$result.CreateAuthorizationHeader()
}
# Apply Changes
Invoke-RestMethod -Uri
"https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Micros
oft.Sql/servers/$serverName/connectionPolicies/Default?api-version=2014-04-01-preview" -Method PUT -Headers
$authHeader -Body $body -ContentType "application/json"
The following CLI script shows how to change the connection policy.
# Get SQL Server ID
sqlserverid=$(az sql server show -n sql-server-name -g sql-server-group --query 'id' -o tsv)
# Set URI
id="$sqlserverid/connectionPolicies/Default"
Next steps
For information on how to change the Azure SQL Database connection policy for an Azure SQL Database
server, see conn-policy.
For information about Azure SQL Database connection behavior for clients that use ADO.NET 4.5 or a later
version, see Ports beyond 1433 for ADO.NET 4.5.
For general application development overview information, see SQL Database Application Development
Overview.
Resolving Transact-SQL differences during migration
to SQL Database
5/23/2018 • 5 minutes to read • Edit Online
When migrating your database from SQL Server to Azure SQL Server, you may discover that your database
requires some re-engineering before the SQL Server can be migrated. This article provides guidance to assist you
in both performing this re-engineering and understanding the underlying reasons why the re-engineering is
necessary. To detect incompatibilities, use the Data Migration Assistant (DMA).
Overview
Most Transact-SQL features that applications use are fully supported in both Microsoft SQL Server and Azure
SQL Database. For example, the core SQL components such as data types, operators, string, arithmetic, logical,
and cursor functions, work identically in SQL Server and SQL Database. There are, however, a few T-SQL
differences in DDL (data-definition language) and DML (data manipulation language) elements resulting in T-
SQL statements and queries that are only partially supported (which we discuss later in this article).
In addition, there are some features and syntax that is not supported at all because Azure SQL Database is
designed to isolate features from dependencies on the master database and the operating system. As such, most
server-level activities are inappropriate for SQL Database. T-SQL statements and options are not available if they
configure server-level options, operating system components, or specify file system configuration. When such
capabilities are required, an appropriate alternative is often available in some other way from SQL Database or
from another Azure feature or service.
For example, high availability is built into Azure, so configuring Always On is not necessary (although you may
want to configure active geo-replication for faster recovery in the event of a disaster). So, T-SQL statements
related to availability groups are not supported by SQL Database, and the dynamic management views related to
Always On are also not supported.
For a list of the features that are supported and unsupported by SQL Database, see Azure SQL Database feature
comparison. The list on this page supplements that guidelines and features article, and focuses on Transact-SQL
statements.
Next steps
For a list of the features that are supported and unsupported by SQL Database, see Azure SQL Database feature
comparison. The list on this page supplements that guidelines and features article, and focuses on Transact-SQL
statements.
Create and manage logical servers and single
databases in Azure SQL Database
9/12/2018 • 7 minutes to read • Edit Online
You can create and manage Azure SQL database logical servers and single databases using the Azure portal,
PowerShell, Azure CLI, REST API, and Transact-SQL.
IMPORTANT
To configure performance properties for a database, see DTU-based purchasing model and vCore-based purchasing model.
TIP
For an Azure portal quickstart, see Create an Azure SQL database in the Azure portal.
TIP
For a PowerShell quickstart, see Create a single Azure SQL database using PowerShell. For PowerShell example scripts, see
Use PowerShell to create a single Azure SQL database and configure a firewall rule and Monitor and scale a single SQL
database using PowerShell.
CMDLET DESCRIPTION
TIP
For an Azure CLI quickstart, see Create a single Azure SQL database using the Azure CLI. For Azure CLI example scripts, see
Use CLI to create a single Azure SQL database and configure a firewall rule and Use CLI to monitor and scale a single SQL
database.
CMDLET DESCRIPTION
az sql db list Lists all databases and data warehouses in a server, or all
databases in an elastic pool
TIP
For a quickstart using SQL Server Management Studio on Microsoft Windows, see Azure SQL Database: Use SQL Server
Management Studio to connect and query data. For a quickstart using Visual Studio Code on the macOS, Linux, or Windows,
see Azure SQL Database: Use Visual Studio Code to connect and query data.
IMPORTANT
You cannot create or delete a server using Transact-SQL.
COMMAND DESCRIPTION
CREATE DATABASE (Azure SQL Database) Creates a new database. You must be connected to the master
database to create a new database.
ALTER DATABASE (Azure SQL Data Warehouse) Modifies an Azure SQL Data Warehouse.
sys.database_service_objectives (Azure SQL Database) Returns the edition (service tier), service objective (pricing tier),
and elastic pool name, if any, for an Azure SQL database or an
Azure SQL Data Warehouse. If logged on to the master
database in an Azure SQL Database server, returns
information on all databases. For Azure SQL Data Warehouse,
you must be connected to the master database.
sys.dm_db_resource_stats (Azure SQL Database) Returns CPU, IO, and memory consumption for an Azure SQL
Database database. One row exists for every 15 seconds, even
if there is no activity in the database.
sys.resource_stats (Azure SQL Database) Returns CPU usage and storage data for an Azure SQL
Database. The data is collected and aggregated within five-
minute intervals.
sys.database_connection_stats (Azure SQL Database) Contains statistics for SQL Database database connectivity
events, providing an overview of database connection
successes and failures.
sys.event_log (Azure SQL Database) Returns successful Azure SQL Database database connections,
connection failures, and deadlocks. You can use this
information to track or troubleshoot your database activity
with SQL Database.
sp_set_firewall_rule (Azure SQL Database) Creates or updates the server-level firewall settings for your
SQL Database server. This stored procedure is only available in
the master database to the server-level principal login. A
server-level firewall rule can only be created using Transact-
SQL after the first server-level firewall rule has been created by
a user with Azure-level permissions
sys.firewall_rules (Azure SQL Database) Returns information about the server-level firewall settings
associated with your Microsoft Azure SQL Database.
sp_delete_firewall_rule (Azure SQL Database) Removes server-level firewall settings from your SQL Database
server. This stored procedure is only available in the master
database to the server-level principal login.
sp_set_database_firewall_rule (Azure SQL Database) Creates or updates the database-level firewall rules for your
Azure SQL Database or SQL Data Warehouse. Database
firewall rules can be configured for the master database, and
for user databases on SQL Database. Database firewall rules
are useful when using contained database users.
sys.database_firewall_rules (Azure SQL Database) Returns information about the database-level firewall settings
associated with your Microsoft Azure SQL Database.
COMMAND DESCRIPTION
sp_delete_database_firewall_rule (Azure SQL Database) Removes database-level firewall setting from your Azure SQL
Database or SQL Data Warehouse.
COMMAND DESCRIPTION
Next steps
To learn about migrating a SQL Server database to Azure, see Migrate to Azure SQL Database.
For information about supported features, see Features.
Azure SQL Database vCore-based purchasing model
limits for a single database
9/6/2018 • 5 minutes to read • Edit Online
This article provides the detailed resource limits for Azure SQL Database single databases using the vCore-based
purchasing model.
For DTU -based purchasing model limits, see SQL Database DTU -based resource limits.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see Manage
file space in Azure SQL Database.
PERFORMANCE
LEVEL GP_GEN4_1 GP_GEN4_2 GP_GEN4_4 GP_GEN4_8 GP_GEN4_16 GP_GEN4_24
H/W 4 4 4 4 4 4
generation
vCores 1 2 4 8 16 24
IO latency 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write)
(approximate) 5-10 ms 5-10 ms 5-10 ms 5-10 ms 5-10 ms 5-10 ms
(read) (read) (read) (read) (read) (read)
Number of 1 1 1 1 1 1
replicas
PERFORM
ANCE GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_
LEVEL 2 4 8 16 24 32 40 80
H/W 5 5 5 5 5 5 5
generatio
n
vCores 2 4 8 16 24 32 40 80
Max data 1024 1024 1536 3072 4096 4096 4096 4096
size (GB)
Max log 307 307 461 614 1229 1229 1229 1229
size (GB)
Number 1 1 1 1 1 1 1 1
of replicas
Included 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB
backup size size size size size size size size
storage
PERFORMANCE
LEVEL BC_GEN4_1 BC_GEN4_2 BC_GEN4_4 BC_GEN4_8 BC_GEN4_16 BC_GEN4_24
H/W 4 4 4 4 4 4
generation
PERFORMANCE
LEVEL BC_GEN4_1 BC_GEN4_2 BC_GEN4_4 BC_GEN4_8 BC_GEN4_16 BC_GEN4_24
vCores 1 2 4 8 16 24
In-memory 1 2 4 8 20 36
OLTP storage
(GB)
Storage type Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD
IO latency 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write)
(approximate) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read)
Number of 3 3 3 3 3 3
replicas
H/W 5 5 5 5 5 5 5 5
generatio
n
vCores 2 4 8 16 24 32 40 80
Storage Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD
type
Max data 1024 1024 1024 1024 2048 4096 4096 4096
size (GB)
Max log 307 307 307 307 614 1229 1229 1229
size (GB)
Number 3 3 3 3 3 3 3 3
of replicas
Included 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB
backup size size size size size size size size
storage
Next steps
See SQL Database FAQ for answers to frequently asked questions.
See Overview Azure SQL Database resource limits for information about limits at the server and subscription
levels.
For information about general Azure limits, see Azure subscription and service limits, quotas, and constraints.
Resource limits for single databases using the DTU-
based purchasing model
8/15/2018 • 9 minutes to read • Edit Online
This article provides the detailed resource limits for Azure SQL Database single databases using the DTU -based
purchasing model.
For DTU -based purchasing model resource limits for elastic pools, see DTU -based resource limits - elastic pools.
For vCore-based resource limits, see vCore-based resource limits - single databases and vCore-based resource
limits - elastic pools.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see Manage
file space in Azure SQL Database.
Max DTUs 5
Max storage choices 250 250 250 250, 500, 750, 1024
(GB)
PERFORMANCE LEVEL S0 S1 S2 S3
Max storage 250, 500, 750, 250, 500, 750, 250, 500, 750, 250, 500, 750, 250, 500, 750,
choices (GB) 1024 1024 1024 1024 1024
Max storage 500, 750, 500, 750, 500, 750, 500, 750, 4096 4096
choices (GB) 1024 1024 1024 1024
Max in- 1 2 4 8 14 32
memory
OLTP storage
(GB)
PERFORMANCE
LEVEL P1 P2 P4 P6 P11 P15
IMPORTANT
More than 1 TB of storage in the Premium tier is currently available in all regions except the following: West Central US,
China East, USDoDCentral, Germany Central, USDoDEast, US Gov Southwest, Germany Northeast, USGovIowa, China North.
In other regions, the storage max in the Premium tier is limited to 1 TB. See P11-P15 Current Limitations.
Changing the service tier and/or performance level of a database creates a replica of the original database at the
new performance level, and then switches connections over to the replica. No data is lost during this process but
during the brief moment when we switch over to the replica, connections to the database are disabled, so some
transactions in flight may be rolled back. The length of time for the switch-over varies, but is less than 30 seconds
99% of the time. If there are large numbers of transactions in flight at the moment connections are disabled, the
length of time for the switch-over may be longer.
The duration of the entire scale-up process depends on both the size and service tier of the database before and
after the change. For example, a 250-GB database that is changing to, from, or within a Standard service tier,
should complete within six hours. For a database the same size that is changing performance levels within the
Premium service tier, the scale-up should complete within three hours.
TIP
To monitor in-progess operations, see: Manage operations using the SQL REST API, Manage operations using CLI, Monitor
operations using T-SQL and these two PowerShell commands: Get-AzureRmSqlDatabaseActivity and Stop-
AzureRmSqlDatabaseActivity.
If you are upgrading to a higher service tier or performance level, the database max size does not increase
unless you explicitly specify a larger size (maxsize).
To downgrade a database, the database used space must be smaller than the maximum allowed size of the
target service tier and performance level.
When downgrading from Premium to the Standard tier, an extra storage cost applies if both (1) the max size
of the database is supported in the target performance level, and (2) the max size exceeds the included storage
amount of the target performance level. For example, if a P1 database with a max size of 500 GB is downsized
to S3, then an extra storage cost applies since S3 supports a max size of 500 GB and its included storage
amount is only 250 GB. So, the extra storage amount is 500 GB – 250 GB = 250 GB. For pricing of extra
storage, see SQL Database pricing. If the actual amount of space used is less than the included storage amount,
then this extra cost can be avoided by reducing the database max size to the included amount.
When upgrading a database with geo-replication enabled, upgrade its secondary databases to the desired
performance tier before upgrading the primary database (general guidance for best performance). When
upgrading to a different, upgrading the secondary database first is required.
When downgrading a database with geo-replication enabled, downgrade its primary databases to the desired
performance tier before downgrading the secondary database (general guidance for best performance). When
downgrading to a different edition, downgrading the primary database first is required.
The restore service offerings are different for the various service tiers. If you are downgrading to the Basic tier,
there is a lower backup retention period - see Azure SQL Database Backups.
The new properties for the database are not applied until the changes are complete.
Single database: Limitations of P11 and P15 when the maximum size
greater than 1 TB
A maximum size greater than 1 TB for P11 and P15 database is supported in the following regions: Australia East,
Australia Southeast, Brazil South, Canada Central, Canada East, Central US, France Central, Germany Central,
Japan East, Japan West, Korea Central, North Central US, North Europe, South Central US, South East Asia, UK
South, UK West, US East2, West US, US Gov Virginia, and West Europe. The following considerations and
limitations apply to P11 and P15 databases with a maximum size greater than 1 TB:
If you choose a maximum size greater than 1 TB when creating a database (using a value of 4 TB or 4096 GB ),
the create command fails with an error if the database is provisioned in an unsupported region.
For existing P11 and P15 databases located in one of the supported regions, you can increase the maximum
storage to beyond 1 TB in increments of 256 GB up to 4 TB. To see if a larger size is supported in your region,
use the DATABASEPROPERTYEX function or inspect the database size in the Azure portal. Upgrading an
existing P11 or P15 database can only be performed by a server-level principal login or by members of the
dbmanager database role.
If an upgrade operation is executed in a supported region the configuration is updated immediately. The
database remains online during the upgrade process. However, you cannot utilize the full amount of storage
beyond 1 TB of storage until the actual database files have been upgraded to the new maximum size. The
length of time required depends upon on the size of the database being upgraded.
When creating or updating a P11 or P15 database, you can only choose between 1-TB and 4-TB maximum size
in increments of 256 GB. When creating a P11/P15, the default storage option of 1 TB is pre-selected. For
databases located in one of the supported regions, you can increase the storage maximum to up to a maximum
of 4 TB for a new or existing single database. For all other regions, the maximum size cannot be increased
above 1 TB. The price does not change when you select 4 TB of included storage.
If the maximum size of a database is set to greater than 1 TB, then it cannot be changed to 1 TB even if the
actual storage used is below 1 TB. Thus, you cannot downgrade a P11 or P15 with a maximum size larger than
1 TB to a 1 TB P11 or 1 TB P15 or lower performance tier, such as P1-P6). This restriction also applies to the
restore and copy scenarios including point-in-time, geo-restore, long-term-backup-retention, and database
copy. Once a database is configured with a maximum size greater than 1 TB, all restore operations of this
database must be run into a P11/P15 with a maximum size greater than 1 TB.
For active geo-replication scenarios:
Setting up a geo-replication relationship: If the primary database is P11 or P15, the secondary(ies) must
also be P11 or P15; lower performance tiers are rejected as secondaries since they are not capable of
supporting more than 1 TB.
Upgrading the primary database in a geo-replication relationship: Changing the maximum size to more
than 1 TB on a primary database triggers the same change on the secondary database. Both upgrades
must be successful for the change on the primary to take effect. Region limitations for the more than 1-
TB option apply. If the secondary is in a region that does not support more than 1 TB, the primary is not
upgraded.
Using the Import/Export service for loading P11/P15 databases with more than 1 TB is not supported. Use
SqlPackage.exe to import and export data.
Next steps
See SQL Database FAQ for answers to frequently asked questions.
See Overview Azure SQL Database resource limits for information about limits at the server and subscription
levels.
For information about general Azure limits, see Azure subscription and service limits, quotas, and constraints.
For information about DTUs and eDTUs, see DTUs and eDTUs.
For information about tempdb size limits, see https://docs.microsoft.com/sql/relational-
databases/databases/tempdb-database#tempdb-database-in-sql-database.
Scale single database resources in Azure SQL
Database
8/6/2018 • 9 minutes to read • Edit Online
This article describes how to scale the compute and storage resources available for a single database in Azure
SQL Database.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
TIP
To monitor in-progess operations, see: Manage operations using the SQL REST API, Manage operations using CLI, Monitor
operations using T-SQL and these two PowerShell commands: Get-AzureRmSqlDatabaseActivity and Stop-
AzureRmSqlDatabaseActivity.
If you are upgrading to a higher service tier or performance level, the database max size does not increase
unless you explicitly specify a larger size (maxsize).
To downgrade a database, the database used space must be smaller than the maximum allowed size of the
target service tier and performance level.
When upgrading a database with geo-replication enabled, upgrade its secondary databases to the desired
performance tier before upgrading the primary database (general guidance for best performance). When
upgrading to a different, upgrading the secondary database first is required.
When downgrading a database with geo-replication enabled, downgrade its primary databases to the desired
performance tier before downgrading the secondary database (general guidance for best performance).
When downgrading to a different edition, downgrading the primary database first is required.
The new properties for the database are not applied until the changes are complete.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
Changing the service tier and/or performance level of a database creates a replica of the original database at the
new performance level, and then switches connections over to the replica. No data is lost during this process but
during the brief moment when we switch over to the replica, connections to the database are disabled, so some
transactions in flight may be rolled back. The length of time for the switch-over varies, but is less than 30
seconds 99% of the time. If there are large numbers of transactions in flight at the moment connections are
disabled, the length of time for the switch-over may be longer.
The duration of the entire scale-up process depends on both the size and service tier of the database before and
after the change. For example, a 250-GB database that is changing to, from, or within a Standard service tier,
should complete within six hours. For a database the same size that is changing performance levels within the
Premium service tier, the scale-up should complete within three hours.
TIP
To monitor in-progess operations, see: Manage operations using the SQL REST API, Manage operations using CLI, Monitor
operations using T-SQL and these two PowerShell commands: Get-AzureRmSqlDatabaseActivity and Stop-
AzureRmSqlDatabaseActivity.
If you are upgrading to a higher service tier or performance level, the database max size does not increase
unless you explicitly specify a larger size (maxsize).
To downgrade a database, the database used space must be smaller than the maximum allowed size of the
target service tier and performance level.
When downgrading from Premium to the Standard tier, an extra storage cost applies if both (1) the max size
of the database is supported in the target performance level, and (2) the max size exceeds the included
storage amount of the target performance level. For example, if a P1 database with a max size of 500 GB is
downsized to S3, then an extra storage cost applies since S3 supports a max size of 500 GB and its included
storage amount is only 250 GB. So, the extra storage amount is 500 GB – 250 GB = 250 GB. For pricing of
extra storage, see SQL Database pricing. If the actual amount of space used is less than the included storage
amount, then this extra cost can be avoided by reducing the database max size to the included amount.
When upgrading a database with geo-replication enabled, upgrade its secondary databases to the desired
performance tier before upgrading the primary database (general guidance for best performance). When
upgrading to a different, upgrading the secondary database first is required.
When downgrading a database with geo-replication enabled, downgrade its primary databases to the desired
performance tier before downgrading the secondary database (general guidance for best performance).
When downgrading to a different edition, downgrading the primary database first is required.
The restore service offerings are different for the various service tiers. If you are downgrading to the Basic
tier, there is a lower backup retention period - see Azure SQL Database Backups.
The new properties for the database are not applied until the changes are complete.
Next steps
For overall resource limits, see SQL Database vCore-based resource limits - single databases and SQL Database
DTU -based resource limits - elastic pools.
Elastic pools help you manage and scale multiple
Azure SQL databases
8/9/2018 • 12 minutes to read • Edit Online
SQL Database elastic pools are a simple, cost-effective solution for managing and scaling multiple
databases that have varying and unpredictable usage demands. The databases in an elastic pool are on a
single Azure SQL Database server and share a set number of resources at a set price. Elastic pools in Azure
SQL Database enable SaaS developers to optimize the price performance for a group of databases within a
prescribed budget while delivering performance elasticity for each database.
Elastic pools enable the developer to purchase resources for a pool shared by multiple databases to
accommodate unpredictable periods of usage by individual databases. You can configure resources for the
pool based either on the DTU -based purchasing model or the vCore-based purchasing model. The resource
requirement for a pool is determined by the aggregate utilization of its databases. The amount of resources
available to the pool is controlled by the developer budget. The developer simply adds databases to the pool,
sets the minimum and maximum resources for the databases (either minimum and maximum DTUs or
minimum or maximum vCores depending on your choice of resourcing model), and then sets the resources
of the pool based on their budget. A developer can use pools to seamlessly grow their service from a lean
startup to a mature business at ever-increasing scale.
Within the pool, individual databases are given the flexibility to auto-scale within set parameters. Under
heavy load, a database can consume more resources to meet demand. Databases under light loads consume
less, and databases under no load consume no resources. Provisioning resources for the entire pool rather
than for single databases simplifies your management tasks. Plus, you have a predictable budget for the
pool. Additional resources can be added to an existing pool with no database downtime, except that the
databases may need to be moved to provide the additional compute resources for the new eDTU
reservation. Similarly, if extra resources are no longer needed they can be removed from an existing pool at
any point in time. And you can add or subtract databases to the pool. If a database is predictably under-
utilizing resources, move it out.
NOTE
When moving databases into or out of an elastic pool, there is no downtime except for a brief period of time (on the
order of seconds) at the end of the operation when database connections are dropped.
For the five-minute period illustrated, DB1 peaks up to 90 DTUs, but its overall average usage is less than
five DTUs. An S3 performance level is required to run this workload in a single database, but this leaves
most of the resources unused during periods of low activity.
A pool allows these unused DTUs to be shared across multiple databases, and so reduces the DTUs needed
and overall cost.
Building on the previous example, suppose there are additional databases with similar utilization patterns as
DB1. In the next two figures below, the utilization of four databases and 20 databases are layered onto the
same graph to illustrate the non-overlapping nature of their utilization over time using the DTU -based
purchasing model:
The aggregate DTU utilization across all 20 databases is illustrated by the black line in the preceding figure.
This shows that the aggregate DTU utilization never exceeds 100 DTUs, and indicates that the 20 databases
can share 100 eDTUs over this time period. This results in a 20x reduction in DTUs and a 13x price reduction
compared to placing each of the databases in S3 performance levels for single databases.
This example is ideal for the following reasons:
There are large differences between peak utilization and average utilization per database.
The peak utilization for each database occurs at different points in time.
eDTUs are shared between many databases.
The price of a pool is a function of the pool eDTUs. While the eDTU unit price for a pool is 1.5x greater than
the DTU unit price for a single database, pool eDTUs can be shared by many databases and fewer total
eDTUs are needed. These distinctions in pricing and eDTU sharing are the basis of the price savings
potential that pools can provide.
The following rules of thumb related to database count and database utilization help to ensure that a pool
delivers reduced cost compared to using performance levels for single databases.
Minimum number of databases
If the aggregate amount of resources for single databases is more than 1.5x the resources needed for the
pool, then an elastic pool is more cost effective.
DTU -based purchasing model example
At least two S3 databases or at least 15 S0 databases are needed for a 100 eDTU pool to be more cost-
effective than using performance levels for single databases.
Maximum number of concurrently peaking databases
By sharing resources, not all databases in a pool can simultaneously use resources up to the limit available
for single databases. The fewer databases that concurrently peak, the lower the pool resources can be set
and the more cost-effective the pool becomes. In general, not more than 2/3 (or 67%) of the databases in
the pool should simultaneously peak to their resources limit.
DTU -based purchasing model example
To reduce costs for three S3 databases in a 200 eDTU pool, at most two of these databases can
simultaneously peak in their utilization. Otherwise, if more than two of these four S3 databases
simultaneously peak, the pool would have to be sized to more than 200 eDTUs. If the pool is resized to more
than 200 eDTUs, more S3 databases would need to be added to the pool to keep costs lower than
performance levels for single databases.
Note this example does not consider utilization of other databases in the pool. If all databases have some
utilization at any given point in time, then less than 2/3 (or 67%) of the databases can peak simultaneously.
Resource utilization per database
A large difference between the peak and average utilization of a database indicates prolonged periods of low
utilization and short periods of high utilization. This utilization pattern is ideal for sharing resources across
databases. A database should be considered for a pool when its peak utilization is about 1.5 times greater
than its average utilization.
DTU -based purchasing model example
An S3 database that peaks to 100 DTUs and on average uses 67 DTUs or less is a good candidate for
sharing eDTUs in a pool. Alternatively, an S1 database that peaks to 20 DTUs and on average uses 13 DTUs
or less is a good candidate for a pool.
Creating a new SQL Database elastic pool using the Azure portal
There are two ways you can create an elastic pool in the Azure portal.
1. You can create an elastic pool by searching SQL elastic pool in the Marketplace or clicking +Add on
the SQL elastic pools browse blade. You are able to specify a new or existing server through this pool
provisioning workflow.
2. Or you can create an elastic pool by navigating to an existing SQL server and clicking Create pool to
create a pool directly into that server. The only difference here is you skip the step where you specify the
server during the pool provisioning workflow.
NOTE
You can create multiple pools on a server, but you can't add databases from different servers into the same pool.
The pool's service tier determines the features available to the elastics in the pool, and the maximum
amount of resources available to each database. For details, see Resource limits for elastic pools in the DTU
model. For vCore-based resource limits for elastic pools, see vCore-based resource limits - elastic pools.
To configure the resources and pricing of the pool, click Configure pool. Then select a service tier, add
databases to the pool, and configure the resource limits for the pool and its databases.
When you have completed configuring the pool, you can click 'Apply', name the pool, and click 'OK' to
create the pool.
If you want more information about the pool you can click on any of the available information in this
overview. Clicking on the Resource utilization chart will take you to the Azure Monitoring view where you
can customize the metrics and time window shown in the chart. Clicking on any available notifications will
take you to a blade that shows the full details of that alert or recommendation.
If you would like to monitor the databases inside your pool, you can click on Database resource
utilization in the Monitoring section of the resource menu on the left.
To customize the chart display
You can edit the chart and the metric page to display other metrics such as CPU percentage, data IO
percentage, and log IO percentage used.
On the Edit Chart form, you can select a fixed time range or click custom to select any 24-hour window in
the last two weeks, and then select the resources to monitor.
To select databases to monitor
By default, the chart in the Database Resource Utilization blade will show the top 5 databases by DTU or
CPU (depending on your service tier). You can switch up the databases in this chart by selecting and
unselecting databases from the list below the chart via the checkboxes on the left.
You can also select more metrics to view side by side in this database table to get a more complete view of
your databases performance.
For more information, see create SQL Database alerts in Azure portal .
Next steps
To scale elastic pools, see Scaling elastic pools and Scale an elastic pool - sample code
For a video, see Microsoft Virtual Academy video course on Azure SQL Database elastic capabilities
To learn more about design patterns for SaaS applications using elastic pools, see Design Patterns for
Multi-tenant SaaS Applications with Azure SQL Database.
For a SaaS tutorial using elastic pools, see Introduction to the Wingtip SaaS application.
Create and manage elastic pools in Azure SQL
Database
9/12/2018 • 5 minutes to read • Edit Online
With an elastic pool, you determine the amount of resources that the elastic pool requires to handle the workload
of its databases, and the amount of resources for each pooled database.
CMDLET DESCRIPTION
Get-AzureRmSqlElasticPool Gets elastic pools and their property values on a logical SQL
server.
TIP
Creation of many databases in an elastic pool can take time when done using the portal or PowerShell cmdlets that create
only a single database at a time. To automate creation into an elastic pool, see CreateOrUpdateElasticPoolAndPopulate.
TIP
For Azure CLI example scripts, see Use CLI to move an Azure SQL database in a SQL elastic pool and Use Azure CLI to scale a
SQL elastic pool in Azure SQL Database.
CMDLET DESCRIPTION
az sql elastic-pool list-editions Also includes available pool DTU settings, storage limits, and
per database settings. In order to reduce verbosity, additional
storage limits and per database settings are hidden by default.
IMPORTANT
You cannot create, update, or delete an Azure SQL Database elastic pool using Transact-SQL. You can add or remove
databases from an elastic pool, and you can use DMVs to return information about existing elastic pools.
COMMAND DESCRIPTION
CREATE DATABASE (Azure SQL Database) Creates a new database in an existing pool or as a single
database. You must be connected to the master database to
create a new database.
ALTER DATABASE (Azure SQL Database) Move a database into, out of, or between elastic pools.
sys.elastic_pool_resource_stats (Azure SQL Database) Returns resource usage statistics for all the elastic database
pools in a logical server. For each elastic database pool, there is
one row for each 15 second reporting window (four rows per
minute). This includes CPU, IO, Log, storage consumption and
concurrent request/session utilization by all databases in the
pool.
sys.database_service_objectives (Azure SQL Database) Returns the edition (service tier), service objective (pricing tier),
and elastic pool name, if any, for an Azure SQL database or an
Azure SQL Data Warehouse. If logged on to the master
database in an Azure SQL Database server, returns
information on all databases. For Azure SQL Data Warehouse,
you must be connected to the master database.
Elastic pools - Create Or Update Creates a new elastic pool or updates an existing elastic pool.
Elastic pool Database Activities Returns activity on databases inside of an elastic pool.
Next steps
For a video, see Microsoft Virtual Academy video course on Azure SQL Database elastic capabilities
To learn more about design patterns for SaaS applications using elastic pools, see Design Patterns for Multi-
tenant SaaS Applications with Azure SQL Database.
For a SaaS tutorial using elastic pools, see Introduction to the Wingtip SaaS application.
Azure SQL Database vCore-based purchasing model
limits for elastic pools
9/14/2018 • 8 minutes to read • Edit Online
This article provides the detailed resource limits for Azure SQL Database elastic pools and pooled databases
using the vCore-based purchasing model.
For DTU -based purchasing model limits, see SQL Database DTU -based resource limits - elastic pools.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see
Manage file space in Azure SQL Database.
NOTE
The resource limits of individual databases in elastic pools are generally the same as for single databases outside of pools
that has the same performance level. For example, the max concurrent workers for an GP_Gen4_1 database is 200 workers.
So, the max concurrent workers for a database in a GP_Gen4_1 pool is also 200 workers. Note, the total number of
concurrent workers in GP_Gen4_1 pool is 210.
PERFORMANCE
LEVEL GP_GEN4_1 GP_GEN4_2 GP_GEN4_4 GP_GEN4_8 GP_GEN4_16 GP_GEN4_24
H/W 4 4 4 4 4 4
generation
vCores 1 2 4 8 16 24
IO latency 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write) 5-7 ms (write)
(approximate) 5-10 ms 5-10 ms 5-10 ms 5-10 ms 5-10 ms 5-10 ms
(read) (read) (read) (read) (read) (read)
Min/max 0, 0.25, 0.5, 1 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1,
elastic pool 2 2, 4 2, 4, 8 2, 4, 8, 16 2, 4, 8, 16, 24
vcore choices
per database
Number of 1 1 1 1 1 1
replicas
PERFORM
ANCE GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_
LEVEL 2 4 8 16 24 32 40 80
H/W 5 5 5 5 5 5 5 5
generatio
n
PERFORM
ANCE GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_ GP_GEN5_
LEVEL 2 4 8 16 24 32 40 80
vCores 2 4 8 16 24 32 40 80
Max data 512 756 1536 2048 3072 4096 4096 4096
size (GB)
Max log 154 227 461 614 922 1229 1229 1229
size
Number 1 1 1 1 1 1 1 1
of replicas
Included 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB
backup size size size size size size size size
storage
PERFORMANCE
LEVEL BC_GEN4_1 BC_GEN4_2 BC_GEN4_4 BC_GEN4_8 BC_GEN4_16 BC_GEN4_24
H/W 4 4 4 4 4 4
generation
vCores 1 2 4 8 16 24
In-memory 1 2 4 8 20 36
OLTP storage
(GB)
Storage type Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD
IO latency 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write) 1-2 ms (write)
(approximate) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read) 1-2 ms (read)
Min/max N/A 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1, 0, 0.25, 0.5, 1,
elastic pool 2 2, 4 2, 4, 8 2, 4, 8, 16 2, 4, 8, 16, 24
vcore choices
per database
Number of 3 3 3 3 3 3
replicas
PERFORM
ANCE BC_GEN5_ BC_GEN5_ BC_GEN5_ BC_GEN5_ BC_GEN5_ BC_GEN5_ BC_GEN5_ BC_GEN5_
LEVEL 2 4 8 16 24 32 40 80
H/W 5 5 5 5 5 5 5 5
generatio
n
vCores 2 4 8 16 24 32 40 80
Storage Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD Local SSD
type
Max data 1024 1024 1024 1024 2048 4096 4096 4096
size (GB)
Max log 307 307 307 307 614 1229 1229 1229
size
Number 3 3 3 3 3 3 3 3
of replicas
Included 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB 1X DB
backup size size size size size size size size
storage
If all vCores of an elastic pool are busy, then each database in the pool receives an equal amount of compute
resources to process queries. The SQL Database service provides resource sharing fairness between databases
by ensuring equal slices of compute time. Elastic pool resource sharing fairness is in addition to any amount of
resource otherwise guaranteed to each database when the vCore min per database is set to a non-zero value.
Database properties for pooled databases
The following table describes the properties for pooled databases.
PROPERTY DESCRIPTION
Max vCores per database The maximum number of vCores that any database in the
pool may use, if available based on utilization by other
databases in the pool. Max vCores per database is not a
resource guarantee for a database. This setting is a global
setting that applies to all databases in the pool. Set max
vCores per database high enough to handle peaks in
database utilization. Some degree of overcommitting is
expected since the pool generally assumes hot and cold
usage patterns for databases where all databases are not
simultaneously peaking.
Min vCores per database The minimum number of vCores that any database in the
pool is guaranteed. This setting is a global setting that applies
to all databases in the pool. The min vCores per database
may be set to 0, and is also the default value. This property is
set to anywhere between 0 and the average vCores
utilization per database. The product of the number of
databases in the pool and the min vCores per database
cannot exceed the vCores per pool.
Max storage per database The maximum database size set by the user for a database in
a pool. Pooled databases share allocated pool storage, so the
size a database can reach is limited to the smaller of
remaining pool storage and database size. Max database size
refers to the maximum size of the data files and does not
include the space used by log files.
Next steps
See SQL Database FAQ for answers to frequently asked questions.
See Overview Azure SQL Database resource limits for information about limits at the server and subscription
levels.
For information about general Azure limits, see Azure subscription and service limits, quotas, and constraints.
Resources limits for elastic pools using the DTU-
based purchasing model
8/15/2018 • 8 minutes to read • Edit Online
This article provides the detailed resource limits for Azure SQL Database elastic pools and pooled databases using
the DTU -based purchasing model.
For DTU -based purchasing model resource limits for single databases, see DTU -based resource limits - single
databases. For vCore-based resource limits, see vCore-based resource limits - single databases and vCore-based
resource limits - elastic pools.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see Manage
file space in Azure SQL Database.
NOTE
The resource limits of individual databases in elastic pools are generally the same as for single databases outside of pools
based on DTUs and the service tier. For example, the max concurrent workers for an S2 database is 120 workers. So, the max
concurrent workers for a database in a Standard pool is also 120 workers if the max DTU per database in the pool is 50
DTUs (which is equivalent to S2).
Max In- N/A N/A N/A N/A N/A N/A N/A N/A
Memory
OLTP
storage
per pool
(GB)
EDTUS PER
POOL 50 100 200 300 400 800 1200 1600
Min 0, 5 0, 5 0, 5 0, 5 0, 5 0, 5 0, 5 0, 5
eDTUs
choices
per
database
Max 5 5 5 5 5 5 5 5
eDTUs
choices
per
database
Max 2 2 2 2 2 2 2 2
storage
per
database
(GB)
Max storage 50, 250, 500 100, 250, 200, 250, 500, 300, 500, 750, 400, 500, 750, 800, 1024,
choices per 500, 750 750, 1024 1024, 1280 1024, 1280, 1280, 1536,
pool (GB) 1536 1792, 2048
Min eDTUs 0, 10, 20, 50 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50,
choices per 100 100, 200 100, 200, 300 100, 200, 300, 100, 200, 300,
database 400 400, 800
Max eDTUs 10, 20, 50 10, 20, 50, 10, 20, 50, 10, 20, 50, 10, 20, 50, 10, 20, 50,
choices per 100 100, 200 100, 200, 300 100, 200, 300, 100, 200, 300,
database 400 400, 800
Max storage 1200, 1280, 1600, 1792, 2000, 2048, 2500, 2560, 3000, 3072,
choices per pool 1536, 1792, 2048, 2304, 2304, 2560, 2816, 3072, 3328, 3584,
(GB) 2048, 2304, 2560, 2816, 2816, 3072, 3328, 3584, 3840, 4096
2560 3072 3328, 3584 3840, 4096
Min eDTUs 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50, 0, 10, 20, 50,
choices per 100, 200, 300, 100, 200, 300, 100, 200, 300, 100, 200, 300, 100, 200, 300,
database 400, 800, 1200 400, 800, 1200, 400, 800, 1200, 400, 800, 1200, 400, 800, 1200,
1600 1600, 2000 1600, 2000, 1600, 2000,
2500 2500, 3000
Max eDTUs 10, 20, 50, 100, 10, 20, 50, 100, 10, 20, 50, 100, 10, 20, 50, 100, 10, 20, 50, 100,
choices per 200, 300, 400, 200, 300, 400, 200, 300, 400, 200, 300, 400, 200, 300, 400,
database 800, 1200 800, 1200, 1600 800, 1200, 1600, 800, 1200, 1600, 800, 1200, 1600,
2000 2000, 2500 2000, 2500,
3000
Max storage 250, 500, 750, 500, 750, 1024 750, 1024 1024 1536
choices per pool 1024
(GB)
Max In-Memory 1 2 4 10 12
OLTP storage per
pool (GB)
Min eDTUs per 0, 25, 50, 75, 125 0, 25, 50, 75, 0, 25, 50, 75, 0, 25, 50, 75, 0, 25, 50, 75,
database 125, 250 125, 250, 500 125, 250, 500, 125, 250, 500,
1000 1000, 1500
Max eDTUs per 25, 50, 75, 125 25, 50, 75, 125, 25, 50, 75, 125, 25, 50, 75, 125, 25, 50, 75, 125,
database 250 250, 500 250, 500, 1000 250, 500, 1000,
1500
Max In-Memory 16 20 24 28 32
OLTP storage per
pool (GB)
Min eDTUs 0, 25, 50, 75, 0, 25, 50, 75, 0, 25, 50, 75, 0, 25, 50, 75, 0, 25, 50, 75,
choices per 125, 250, 500, 125, 250, 500, 125, 250, 500, 125, 250, 500, 125, 250, 500,
database 1000, 1750 1000, 1750 1000, 1750 1000, 1750 1000, 1750,
4000
Max eDTUs 25, 50, 75, 125, 25, 50, 75, 125, 25, 50, 75, 125, 25, 50, 75, 125, 25, 50, 75, 125,
choices per 250, 500, 1000, 250, 500, 1000, 250, 500, 1000, 250, 500, 1000, 250, 500, 1000,
database 1750 1750 1750 1750 1750, 4000
IMPORTANT
More than 1 TB of storage in the Premium tier is currently available in all regions except the following: West Central US,
China East, USDoDCentral, Germany Central, USDoDEast, US Gov SouthWest, Germany NorthEast, USGov Iowa, China
North. In other regions, the storage max in the Premium tier is limited to 1 TB. See P11-P15 Current Limitations.
If all DTUs of an elastic pool are used, then each database in the pool receives an equal amount of resources to
process queries. The SQL Database service provides resource sharing fairness between databases by ensuring
equal slices of compute time. Elastic pool resource sharing fairness is in addition to any amount of resource
otherwise guaranteed to each database when the DTU min per database is set to a non-zero value.
Database properties for pooled databases
The following table describes the properties for pooled databases.
PROPERTY DESCRIPTION
PROPERTY DESCRIPTION
Max eDTUs per database The maximum number of eDTUs that any database in the
pool may use, if available based on utilization by other
databases in the pool. Max eDTU per database is not a
resource guarantee for a database. This setting is a global
setting that applies to all databases in the pool. Set max
eDTUs per database high enough to handle peaks in database
utilization. Some degree of overcommitting is expected since
the pool generally assumes hot and cold usage patterns for
databases where all databases are not simultaneously
peaking. For example, suppose the peak utilization per
database is 20 eDTUs and only 20% of the 100 databases in
the pool are peak at the same time. If the eDTU max per
database is set to 20 eDTUs, then it is reasonable to
overcommit the pool by 5 times, and set the eDTUs per pool
to 400.
Min eDTUs per database The minimum number of eDTUs that any database in the pool
is guaranteed. This setting is a global setting that applies to all
databases in the pool. The min eDTU per database may be set
to 0, and is also the default value. This property is set to
anywhere between 0 and the average eDTU utilization per
database. The product of the number of databases in the pool
and the min eDTUs per database cannot exceed the eDTUs
per pool. For example, if a pool has 20 databases and the
eDTU min per database set to 10 eDTUs, then the eDTUs per
pool must be at least as large as 200 eDTUs.
Max storage per database The maximum database size set by the user for a database in
a pool. However, pooled databases share allocated pool
storage. Even if the total max storage per database is set to
be greater than the total available storage space of the pool,
the total space actually used by all of the databases will not
be able to exceed the available pool limit. Max database size
refers to the maximum size of the data files and does not
include the space used by log files.
Next steps
See SQL Database FAQ for answers to frequently asked questions.
See Overview Azure SQL Database resource limits for information about limits at the server and subscription
levels.
For information about general Azure limits, see Azure subscription and service limits, quotas, and constraints.
For information about DTUs and eDTUs, see DTUs and eDTUs.
For information about tempdb size limits, see https://docs.microsoft.com/sql/relational-
databases/databases/tempdb-database#tempdb-database-in-sql-database.
Scale elastic pool resources in Azure SQL Database
8/1/2018 • 4 minutes to read • Edit Online
This article describes how to scale the compute and storage resources available for elastic pools and pooled
databases in Azure SQL Database.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see Manage
file space in Azure SQL Database.
IMPORTANT
Under some circumstances, you may need to shrink a database to reclaim unused space. For more information, see Manage
file space in Azure SQL Database.
Next steps
For overall resource limits, see SQL Database vCore-based resource limits - elastic pools and SQL Database DTU -
based resource limits - elastic pools.
What is a Managed Instance (preview)?
9/11/2018 • 15 minutes to read • Edit Online
Azure SQL Database Managed Instance (preview ) is a new deployment model of Azure SQL Database,
providing near 100% compatibility with the latest SQL Server on-premises (Enterprise Edition) Database
Engine, providing a native virtual network (VNet) implementation that addresses common security concerns,
and a business model favorable for on-premises SQL Server customers. Managed Instance allows existing
SQL Server customers to lift and shift their on-premises applications to the cloud with minimal application
and database changes. At the same time, Managed Instance preserves all PaaS capabilities (automatic
patching and version updates, automated backups, high-availability ), that drastically reduces management
overhead and TCO.
IMPORTANT
For a list of regions in which Managed Instance is currently available, see Migrate your databases to a fully managed
service with Azure SQL Database Managed Instance.
Azure SQL Database Managed Instance is designed for customers looking to migrate a large number of
apps from on-premises or IaaS, self-built, or ISV provided environment to fully managed PaaS cloud
environment, with as low migration effort as possible. Using the fully automated Data Migration Service
(DMS ) in Azure, customers can lift and shift their on-premises SQL Server to a Managed Instance that offers
compatibility with SQL Server on-premises and complete isolation of customer instances with native VNET
support. With Software Assurance, you can exchange their existing licenses for discounted rates on a SQL
Database Managed Instance using the Azure Hybrid Use Benefit for SQL Server. SQL Database Managed
Instance is the best migration destination in the cloud for SQL Server instances that require high security and
a rich programmability surface.
By General Availability, Managed Instance aims to deliver close to 100% surface area compatibility with the
latest on-premises SQL Server version through a staged release plan.
To decide between Azure SQL Database Single Database, Azure SQL Database Managed Instance, and SQL
Server IaaS hosted in Virtual Machine see how to choose the right version of SQL Server in Azure cloud.
IMPORTANT
A Managed Instance runs with all of the features of the most recent version of SQL Server, including online operations,
automatic plan corrections, and other enterprise performance enhancements.
Isolated environment (VNet integration, single tenant Azure Resource Manager API for automating service
service, dedicated compute and storage) provisioning and scaling
Transparent data encryption (TDE) Azure portal functionality for manual service provisioning
Azure AD authentication, single sign-on support and scaling
Adheres to compliance standards same as Azure SQL Data Migration Service
database
SQL auditing
Threat Detection
GEN 4 GEN 5
Hardware Intel E5-2673 v3 (Haswell) 2.4 GHz Intel E5-2673 v4 (Broadwell) 2.3 GHz
processors, attached SSD vCore = 1 processors, fast eNVM SSD, vCore=1
PP (physical core) LP (hyper-thread)
GEN 4 GEN 5
Performance levels 8, 16, 24 vCores 8, 16, 24, 32, 40, 64, 80 vCores
IMPORTANT
Changing your service tier from General Purpose to Business Critical or vice versa is not supported in Public Preview. If
you want to migrate your databases to an instance in different service tier, you can create new instance, restore
databases with point in time restore from the original instance and then drop original instance if it is not needed
anymore.
FEATURE DESCRIPTION
SQL Server version / build SQL Server Database Engine (latest stable)
Max storage per database Determined by the max storage size per instance
Expected storage IOPS 500-7500 IOPS per data file (depends on data file). See
Premium Storage
* A virtual core represents the logical CPU offered with an option to choose between generations of
hardware. Gen 4 Logical CPUs are based on Intel E5-2673 v3 (Haswell) 2.4 GHz processors and Gen 5
Logical CPUs are based on Intel E5-2673 v4 (Broadwell) 2.3 GHz processors.
For more information see Standard/General Purpose availability and architecture in Azure SQL Database.
Business Critical service tier
Business Critical service tier is built for applications with high IO requirements. It offers highest resilience to
failures using several isolated Always On replicas.
The following list outlines the key characteristics of the Business Critical service tier:
Designed for business applications with highest performance and HA requirements
Comes with super-fast SSD storage (up to 1 TB on Gen 4 and up to 4 TB on Gen 5)
Supports up to 100 databases per instance
Built-in additional read-only instance that can be used for reporting and other read-only workloads
In-Memory OLTP that can be used for workload with high-prefrmance requirements
FEATURE DESCRIPTION
Max storage per database Determined by the max storage size per instance
For more information see Premium/Business Critical availability and architecture in Azure SQL Database.
IMPORTANT
Place multiple managed instance in the same subnet, wherever that is allowed by your security requirements, as that
will bring you additional benefits. Collocating instances in the same subnet will significantly simplify networking
infrastructure maintenance and reduce instance provisioning time, since long provisioning duration is associated with
the cost of deploying first managed instance in a subnet.
Database migration
Managed Instance targets user scenarios with mass database migration from on-premises or IaaS database
implementations. Managed Instance supports several database migration options:
Backup and restore
The migration approach leverages SQL backups to Azure blob storage. Backups stored in Azure storage blob
can be directly restored into Managed Instance using the T-SQL RESTORE command.
For a quickstart showing how to restore the Wide World Importers - Standard database backup file, see
Restore a backup file to a Managed Instance. This quickstart shows you have to upload a backup file to
Azure blog storage and secure it using a Shared access signature (SAS ) key.
For information about restore from URL, see Native RESTORE from URL.
Data Migration Service
The Azure Database Migration Service is a fully managed service designed to enable seamless migrations
from multiple database sources to Azure Data platforms with minimal downtime. This service streamlines
the tasks required to move existing third party and SQL Server databases to Azure. Deployment options
include Azure SQL Database, Managed Instance, and SQL Server in Azure VM at Public Preview. See How
to migrate your on-premises database to Managed Instance using DMS.
IMPORTANT
For a list of supported, partially supported, and unsupported features, see SQL Database features. For a list of T-SQL
differences in Managed Instances versus SQL Server, see Managed Instance T-SQL Differences from SQL Server
Next steps
To learn how to create your first Managed Instance, see Quick-start guide.
For a features and comparison list, see SQL common features.
For more information about VNet configuration, see Managed Instance VNet Configuration.
For a quickstart that creates a Managed Instance and restores a database from a backup file, see Create a
Managed Instance.
For a tutorial using the Azure Database Migration Service (DMS ) for migration, see Managed Instance
migration using DMS.
For pricing information, see SQL Database Managed Instance pricing.
Configure a VNet for Azure SQL Database Managed
Instance
9/13/2018 • 6 minutes to read • Edit Online
Azure SQL Database Managed Instance (preview ) must be deployed within an Azure virtual network (VNet) .
This deployment enables the following scenarios:
Connecting to a Managed Instance directly from an on-premises network
Connecting a Managed Instance to linked server or another on-premises data store
Connecting a Managed Instance to Azure resources
Plan
Plan how you deploy a Managed Instance in virtual network using your answers to the following questions:
Do you plan to deploy single or multiple Managed Instances?
The number of Managed Instances determines the minimum size of the subnet to allocate for your
Managed Instances. For more information, see Determine the size of subnet for Managed Instance.
Do you need to deploy your Managed Instance into an existing virtual network or you are creating a new
network?
If you plan to use an existing virtual network, you need to modify that network configuration to
accommodate your Managed Instance. For more information, see Modify existing virtual network for
Managed Instance.
If you plan to create new virtual network, see Create new virtual network for Managed Instance.
Requirements
To create a Managed Instance, create a dedicated subnet (the Managed Instance subnet) inside the virtual
network that conforms to the following requirements:
Dedicated subnet: The Managed Instance subnet must not contain any other cloud service associated with
it, and it must not be a Gateway subnet. You won’t be able to create a Managed Instance in a subnet that
contains resources other than Managed Instance, and you can not later add other resources in the subnet.
Compatible Network Security Group (NSG): An NSG associated with a Managed Instance subnet must
contain rules shown in the following tables (Mandatory inbound security rules and Mandatory outbound
security rules) in front of any other rules. You can use an NSG to fully control access to the Managed Instance
data endpoint by filtering traffic on port 1433.
Compatible user-defined route table (UDR): The Managed Instance subnet must have a user route table
with 0.0.0.0/0 Next Hop Internet as the mandatory UDR assigned to it. In addition, you can add a UDR that
routes traffic that has on-premises private IP ranges as a destination through virtual network gateway or
virtual network appliance (NVA).
Optional custom DNS: If a custom DNS is specified on thevirtual netword, Azure's recursive resolver IP
address (such as 168.63.129.16) must be added to the list. For more information, see Configuring Custom
DNS. The custom DNS server must be able to resolve host names in the following domains and their
subdomains: microsoft.com, windows.net, windows.com, msocsp.com, digicert.com, live.com,
microsoftonline.com, and microsoftonline-p.com.
No service endpoints: The Managed Instance subnet must not have a service endpoint associated to it.
Make sure that service endpoints option is disabled when creating the virtual network.
Sufficient IP addresses: The Managed Instance subnet must have the bare minimum of 16 IP addresses
(recommended minimum is 32 IP addresses). For more information, see Determine the size of subnet for
Managed Instances
IMPORTANT
You won’t be able to deploy a new Managed Instance if the destination subnet is not compatible with all of these
requirements. When a Managed Instance is created, a Network Intent Policy is applied on the subnet to prevent non-
compliant changes to networking configuration. After the last instance is removed from the subnet, the Network Intent
Policy is removed as well
IMPORTANT
Subnet size with 16 IP addresses is the bare minimum with limited potential for the further Managed Instance scale out.
Choosing subnet with the prefix /27 or below is highly recommended.
If you plan to deploy multiple Managed Instances inside the subnet and need to optimize on subnet size, use
these parameters to form a calculation:
Azure uses five IP addresses in the subnet for its own needs
Each General Purpose instance needs two addresses
Each Business Critical instance needs four addresses
Example: You plan to have three General Purpose and two Business Critical Managed Instances. That means
you need 5 + 3 * 2 + 2 * 4 = 19 IP addresses. As IP ranges are defined in power of 2, you need the IP range of 32
(2^5) IP addresses. Therefore, you need to reserve the subnet with subnet mask of /27.
IMPORTANT
Calculation displayed above will become obsolete with further improvements.
This button will open a form that you can use to configure network environment where you can deploy
Managed Instance.
NOTE
This Azure Resource Manager template will deploy virtual network with two subnets. One subnet called
ManagedInstances is reserved for Managed Instances and has pre-configured route table, while the other subnet
called Default is used for other resources that should access Managed Instance (for example, Azure Virtual
Machines). You can remove Default subnet if you don't need it.
3. Configure network environment. On the following form you can configure parameters of your network
environment:
You might change the names of VNet and subnets and adjust IP ranges associated to your networking resources.
Once you press "Purchase" button, this form will create and configure your environment. If you don't need two
subnets you can delete the default one.
$scriptUrlBase = 'https://raw.githubusercontent.com/Microsoft/sql-server-samples/master/samples/manage/azure-
sql-db-managed-instance/prepare-subnet'
$parameters = @{
subscriptionId = '<subscriptionId>'
resourceGroupName = '<resourceGroupName>'
virtualNetworkName = '<virtualNetworkName>'
subnetName = '<subnetName>'
}
Next steps
For an overview, see What is a Managed Instance
For a tutorial showing how to create a VNet, create a Managed Instance, and restore a database from a
database backup, see Creating an Azure SQL Database Managed Instance.
For DNS issues, see Configuring a Custom DNS
Configuring a Custom DNS for Azure SQL Database
Managed Instance
9/6/2018 • 2 minutes to read • Edit Online
An Azure SQL Database Managed Instance (preview ) must be deployed within an Azure virtual network (VNet) .
There are a few scenarios (i.e. linked servers to other SQL instances in your cloud or hybrid environment) that
require private host names to be resolved from the Managed Instance. In this case, you need to configure a
custom DNS inside Azure. Since Managed Instance uses the same DNS for its inner workings, the virtual network
DNS configuration needs to be compatible with Managed Instance.
To make a custom DNS configuration is compatible with the Managed Instance, you need to:
Configure custom DNS server so it is able to resolve public domain names
Put Azure Recursive Resolver DNS IP address 168.63.129.16 at the end of the virtual network DNS list
2. Switch to Custom and enter your custom DNS server IP address as well as Azure's recursive resolvers IP
address 168.63.129.16.
IMPORTANT
Not setting Azure’s recursive resolver in DNS list causes the Managed Instance to enter faulty state. Recovering from
that state may require you to create new instance in a VNet with the compliant networking policies, create instance
level data, and restore your databases. See VNet Configuration.
Next steps
For an overview, see What is a Managed Instance
For a tutorial showing you how to create a new Managed Instance, see Creating a Managed Instance.
For information about configuring a VNet for a Managed Instance, see VNet configuration for Managed
Instances
Sync networking configuration for Azure App Service
hosting plan
7/25/2018 • 2 minutes to read • Edit Online
It might happen that although you integrated your app with an Azure Virtual Network, you can't establish
connection to Managed Instance. One thing you can try is to refresh networking configuration for your service
plan.
Next steps
For information about configuring your VNet for Managed Instance, see Managed Instance VNet configuration.
Azure SQL Database Managed Instance Connectivity
Architecture
9/6/2018 • 3 minutes to read • Edit Online
This article provides the Azure SQL Database Managed Instance communication overview and explains
connectivity architecture as well as how the different components function to direct traffic to the Managed Instance.
Communication overview
The following diagram shows entities that connect to Managed Instance as well as resources that Managed
Instance has to reach out in order to function properly.
Communication that is depicted on the bottom of the diagram represents customer applications and tools
connecting to Managed Instance as data source.
As Managed Instance is platform-as-a-services (PaaS ) offering, Microsoft manages this service using automated
agents (management, deployment, and maintenance) based on telemetry data streams. As Managed Instance
management is solely Microsoft responsibility, customers are not able to access Managed Instance virtual cluster
machines through RDP.
Some SQL Server operations initiated by the end users or applications may require Managed Instance to interact
with the platform. One case is the creation of a Managed Instance database - a resource that is exposed through
the portal, PowerShell, and Azure CLI.
Managed Instance depends on other Azure Services for its proper functioning (such as Azure Storage for backups,
Azure Service Bus for telemetry, Azure AD for authentication, Azure Key Vault for TDE, and so forth) and initiates
connections to them accordingly.
All communications, stated above, are encrypted and signed using certificates. To make sure that communicating
parties are trusted, Managed Instance constantly verifies these certificates by contacting Certificate Authority. If the
certificates are revoked or Managed Instance could not verify them, it closes the connections to protect the data.
High-level connectivity architecture
At a high level, Managed Instance is a set of service components, hosted on a dedicated set of isolated virtual
machines that run inside the customer virtual network subnet and form a virtual cluster.
Multiple Managed Instances could be hosted in single virtual cluster. The cluster is automatically expanded or
contracted if needed when the customer changes the number of provisioned instances in the subnet.
Customer applications could connect to Managed Instance, query and update databases only if they run inside the
virtual network or peered virtual network or VPN / Express Route connected network using endpoint with private
IP address.
Microsoft management and deployment services run outside of the virtual network so connection between
Managed Instance and Microsoft services goes over the endpoints with public IP addresses. When Managed
Instance creates outbound connection, on receiving end it looks like it’s coming from this public IP due to Network
Address Translation (NAT).
Management traffic flows through the customer virtual network. That means that elements of virtual network
infrastructure affect and could potentially harm management traffic causing instance to enter faulty state and
become unavailable.
IMPORTANT
To improve customer expirience and service availability, Microsoft applies Network Intent Policy on Azure virtual network
infrastructure elements that could affect Managed Instance functioning. This is a platform mechanism to communicate
transparently networking requirements to end users, with main goal to prevent network misconfiguration and ensure normal
Managed Instance operations. Upon Managed Instance deletion Network Intent Policy is removed as well.
Next steps
For an overview, seeWhat is a Managed Instance
For more information about VNet configuration, seeManaged Instance VNet Configuration.
For a quickstart see how to create Managed Instance:
From the Azure portal
Using PowerShell
using Azure Resource Manager template
using Azure Resource Manager template (jumpbox with SSMS included)
Connect your application to Azure SQL Database
Managed Instance
8/23/2018 • 3 minutes to read • Edit Online
Today you have multiple choices when deciding how and where you host your application.
You may choose to host application in the cloud either by using Azure App Service or some of Azure's virtual
network (VNet) integrated options like Azure App Service Environment, Virtual Machine, Virtual Machine Scale
Set. You could also take hybrid cloud approach and keep your applications on-premises.
Whatever choice you made, you can connect it to a Managed Instance (preview ).
IMPORTANT
VNet peering scenario for Managed Instance is limited to the networks in the same region due to constraints of the Global
Virtual Network peering.
DRIVER/TOOL VERSION
Next steps
For information about Managed Instance, see What is a Managed Instance.
For a tutorial showing you how to create a new Managed Instance, see Create a Managed Instance.
Azure SQL Database Managed Instance T-SQL
differences from SQL Server
9/14/2018 • 13 minutes to read • Edit Online
Azure SQL Database Managed Instance (preview ) provides high compatibility with on-premises SQL Server
Database Engine. Most of the SQL Server Database Engine features are supported in Managed Instance. Since
there are still some differences in syntax and behavior, this article summarizes and explains these differences.
T-SQL differences and unsupported features
Features that have different behavior in Managed Instance
Temporary limitations and known issues
Limitations:
Managed Instance can back up a database to a backup with up to 32 stripes, which is enough for the
databases up to 4 TB if backup compression is used.
Max backup stripe size is 195 GB (maximum blob size). Increase the number of stripes in the backup
command to reduce individual stripe size and stay within this limit.
TIP
To work around this limitation on-premises, backup to DISK instead of backup to URL , upload backup file to blob, then
restore. Restore supports bigger files because a different blob type is used.
CREATE CERTIFICATE
FROM BINARY = asn_encoded_certificate
WITH PRIVATE KEY ( <private_key_options> )
CLR
Managed Instance cannot access file shares and Windows folders, so the following constraints apply:
Only CREATE ASSEMBLY FROM BINARY is supported. See CREATE ASSEMBLY FROM BINARY.
CREATE ASSEMBLY FROM FILE is not supported. See CREATE ASSEMBLY FROM FILE.
ALTER ASSEMBLY cannot reference files. See ALTER ASSEMBLY.
Compatibility levels
Supported compatibility levels are: 100, 110, 120, 130, 140
Compatibility levels below 100 are not supported.
The default compatibility level for new databases is 140. For restored databases, compatibility level will
remain unchanged if it was 100 and above.
See ALTER DATABASE Compatibility Level.
Credential
Only Azure Key Vault and SHARED ACCESS SIGNATURE identities are supported. Windows users are not supported.
See CREATE CREDENTIAL and ALTER CREDENTIAL.
Cryptographic providers
Managed Instance cannot access files so cryptographic providers cannot be created:
CREATE CRYPTOGRAPHIC PROVIDER is not supported. See CREATE CRYPTOGRAPHIC PROVIDER.
ALTER CRYPTOGRAPHIC PROVIDER is not supported. See ALTER CRYPTOGRAPHIC PROVIDER.
Collation
Server collation is SQL_Latin1_General_CP1_CI_AS and cannot be changed. See Collations.
Database options
Multiple log files are not supported.
In-memory objects are not supported in the General Purpose service tier.
There is a limit of 280 files per instance implying max 280 files per database. Both data and log files are
counted toward this limit.
Database cannot contain filegroups containing filestream data. Restore will fail if .bak contains FILESTREAM
data.
Every file is placed in Azure Premium storage. IO and throughput per file depend on the size of each
individual file, in the same way as they do for Azure Premium Storage disks. See Azure Premium disk
performance
CREATE DATABASE statement
The following are CREATE DATABASE limitations:
Files and filegroups cannot be defined.
CONTAINMENT option is not supported.
WITH options are not supported.
TIP
As workaround, use ALTER DATABASE after CREATE DATABASE to set database options to add files or to set
containment.
For more information, see ALTER DATABASE SET PARTNER and SET WITNESS and CREATE ENDPOINT …
FOR DATABASE_MIRRORING.
DBCC
Undocumented DBCC statements that are enabled in SQL Server are not supported in Managed Instance.
Trace Flags are not supported. See Trace Flags.
DBCC TRACEOFF is not supported. See DBCC TRACEOFF.
DBCC TRACEON is not supported. See DBCC TRACEON.
Distributed transactions
Neither MSDTC nor Elastic Transactions are currently supported in Managed Instance.
Extended Events
Some Windows-specific targets for XEvents are not supported:
etw_classic_sync target is not supported. Store .xel files on Azure blob storage. See etw_classic_sync
target.
event_file target is not supported. Store .xel files on Azure blob storage. See event_file target.
External libraries
In-database R and Python external libraries are not yet supported. See SQL Server Machine Learning Services.
Filestream and Filetable
filestream data is not supported.
Database cannot contain filegroups with FILESTREAM data
FILETABLE is not supported
Tables cannot have FILESTREAM types
The following functions are not supported:
GetPathLocator()
GET_FILESTREAM_TRANSACTION_CONTEXT()
PathName()
GetFileNamespacePath()
FileTableRootPath()
Polybase
External tables referencing the files in HDFS or Azure blob storage are not supported. For information about
Polybase, see Polybase.
Replication
Replication is available for public preview on Managed Instance. For information about Replication, see SQL
Server Replication.
RESTORE statement
Supported syntax
RESTORE DATABASE
RESTORE FILELISTONLY ONLY
RESTORE HEADER ONLY
RESTORE LABELONLY ONLY
RESTORE VERIFYONLY ONLY
Unsupported syntax
RESTORE LOG ONLY
RESTORE REWINDONLY ONLY
Source
FROM URL (Azure blob storage) is only supported option.
FROM DISK / TAPE /backup device is not supported.
Backup sets are not supported.
WITH options are not supported ( No DIFFERENTIAL , STATS , etc.)
ASYNC RESTORE - Restore continues even if client connection breaks. If your connection is dropped, you can
check sys.dm_operation_status view for the status of a restore operation (as well as for CREATE and DROP
database). See sys.dm_operation_status.
The following database options are set/overridden and cannot be changed later:
NEW_BROKER (if broker is not enabled in .bak file)
ENABLE_BROKER (if broker is not enabled in .bak file)
AUTO_CLOSE=OFF (if a database in .bak file has AUTO_CLOSE=ON )
RECOVERY FULL (if a database in .bak file has SIMPLE or BULK_LOGGED recovery mode)
Memory optimized filegroup is added and called XTP if it was not in the source .bak file
Any existing memory optimized filegroup is renamed to XTP
SINGLE_USER and RESTRICTED_USER options are converted to MULTI_USER
Limitations:
.BAK files containing multiple backup sets cannot be restored.
.BAK files containing multiple log files cannot be restored.
Restore will fail if .bak contains FILESTREAM data.
Backups containing databases that have active In-memory objects cannot currently be restored.
Backups containing databases where at some point In-Memory objects existed cannot currently be restored.
Backups containing databases in read-only mode cannot currently be restored. This limitation will be removed
soon.
For information about Restore statements, see RESTORE Statements.
Service broker
Cross-instance service broker is not supported
sys.routes - Prerequisite: select address from sys.routes. Address must be LOCAL on every route. See
sys.routes.
CREATE ROUTE - you cannot CREATE ROUTE with ADDRESS other than LOCAL . See CREATE ROUTE.
ALTER ROUTE cannot ALTER ROUTE with ADDRESS other than LOCAL . See ALTER ROUTE.
For information about creating and altering tables, see CREATE TABLE and ALTER TABLE.
Behavior changes
The following variables, functions, and views return different results:
SERVERPROPERTY('EngineEdition') returns value 8. This property uniquely identifies Managed Instance. See
SERVERPROPERTY.
SERVERPROPERTY('InstanceName') returns NULL, because the concept of instance as it exists for SQL Server
does not apply to Managed Instance. See SERVERPROPERTY ('InstanceName').
@@SERVERNAME returns full DNS 'connectable' name, for example, my-managed-
instance.wcus17662feb9ce98.database.windows.net. See @@SERVERNAME.
SYS.SERVERS - returns full DNS 'connectable' name, such as myinstance.domain.database.windows.net for
properties 'name' and 'data_source'. See SYS.SERVERS.
@@SERVICENAME returns NULL, because the concept of service as it exists for SQL Server does not apply to
Managed Instance. See @@SERVICENAME.
SUSER_ID is supported. Returns NULL if AAD login is not in sys.syslogins. See SUSER_ID.
SUSER_SID is not supported. Returns wrong data (temporary known issue). See SUSER_SID.
GETDATE() and other built-in date/time functions always returns time in UTC time zone. See GETDATE.
Next steps
For details about Managed Instance, see What is a Managed Instance?
For a features and comparison list, see SQL common features.
For a tutorial showing you how to create a new Managed Instance, see Creating a Managed Instance.
Overview of business continuity with Azure SQL
Database
9/12/2018 • 11 minutes to read • Edit Online
Azure SQL Database is an implementation of the latest stable SQL Server Database Engine configured and
optimized for Azure cloud environment that provides high availability and resiliency to the errors that might
affect your business process. Business continuity in Azure SQL Database refers to the mechanisms, policies,
and procedures that enable a business to continue operating in the face of disruption, particularly to its
computing infrastructure. In the most of the cases, Azure SQL Database will handle the disruptive events that
might happen in the cloud environment and keep your business processes running. However, there are some
disruptive events that cannot be handled by SQL Database such as:
User accidentally deleted or updated a row in a table.
Malicious attacker succeeded to delete data or drop a database.
Earthquake caused a power outage and temporary disabled data-center.
These cases cannot be controlled by Azure SQL Database, so you would need to do use the business
continuity features in SQL Database that enables you to recover your data and keep your applications running.
This overview describes the capabilities that Azure SQL Database provides for business continuity and
disaster recovery. Learn about options, recommendations, and tutorials for recovering from disruptive events
that could cause data loss or cause your database and application to become unavailable. Learn what to do
when a user or application error affects data integrity, an Azure region has an outage, or your application
requires maintenance.
GENERAL BUSINESS
CAPABILITY BASIC STANDARD PREMIUM PURPOSE CRITICAL
Point in Time Any restore Any restore Any restore Any restore Any restore
Restore from point within point within 35 point within 35 point within point within
backup seven days days days configured configured
period (up to 35 period (up to 35
days) days)
Geo-restore ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO
from geo- <1h <1h <1h <1h <1h
replicated
backups
Restore from ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO ERT < 12 h, RPO
SQL long-term < 1 wk < 1 wk < 1 wk < 1 wk < 1 wk
retention
Active geo- ERT < 30s, RPO ERT < 30 s, RPO ERT < 30 s, RPO ERT < 30 s, RPO ERT < 30 s, RPO
replication < 5s <5s <5s <5s <5s
IMPORTANT
To use active geo-replication and auto-failover groups, you must either be the subscription owner or have
administrative permissions in SQL Server. You can configure and fail over using the Azure portal, PowerShell, or the REST
API using Azure subscription permissions or using Transact-SQL with SQL Server permissions.
This feature is used to protect against business disruption if a data center outage occurs or during an
application upgrade. To enable automated and transparent failover you should organize your geo-replicated
databases into groups using the auto-failover group feature of SQL Database. Use active geo-replication and
auto-failover groups if your application meets any of these criteria:
Is mission critical.
Has a service level agreement (SLA) that does not allow for 24 hours or more of downtime.
Downtime may result in financial liability.
Has a high rate of data change is high and losing an hour of data is not acceptable.
The additional cost of active geo-replication is lower than the potential financial liability and associated loss
of business.
When you take action, how long it takes you to recover, and how much data loss you incur depends upon how
you decide to use these business continuity features in your application. Indeed, you may choose to use a
combination of database backups and active geo-replication depending upon your application requirements.
For a discussion of application design considerations for stand-alone databases and for elastic pools using
these business continuity features, see Design an application for cloud disaster recovery and Elastic pool
disaster recovery strategies.
The following sections provide an overview of the steps to recover using either database backups or active
geo-replication. For detailed steps including planning requirements, post recovery steps, and information
about how to simulate an outage to perform a disaster recovery drill, see Recover a SQL Database from an
outage.
Prepare for an outage
Regardless of the business continuity feature you use, you must:
Identify and prepare the target server, including server-level firewall rules, logins, and master database level
permissions.
Determine how to redirect clients and client applications to the new server
Document other dependencies, such as auditing settings and alerts
If you do not prepare properly, bringing your applications online after a failover or a database recovery takes
additional time and likely also require troubleshooting at a time of stress - a bad combination.
Fail over to a geo -replicated secondary database
If you are using active geo-replication and auto-failover groups as your recovery mechanism, you can
configure an automatic failover policy or use manual failover. Once initiated, the failover causes the secondary
to become the new primary and ready to record new transactions and respond to queries - with minimal data
loss for the data not yet replicated. For information on designing the failover process, see Design an
application for cloud disaster recovery.
NOTE
When the data center comes back online the old primaries automatically reconnect to the new primary and become
secondary databases. If you need to relocate the primary back to the original region, you can initiate a planned failover
manually (failback).
NOTE
If the data center comes back online before you switch your application over to the recovered database, you can cancel
the recovery.
Perform post failover / recovery tasks
After recovery from either recovery mechanism, you must perform the following additional tasks before your
users and applications are back up and running:
Redirect clients and client applications to the new server and restored database
Ensure appropriate server-level firewall rules are in place for users to connect (or use database-level
firewalls)
Ensure appropriate logins and master database level permissions are in place (or use contained users)
Configure auditing, as appropriate
Configure alerts, as appropriate
Next steps
For a discussion of application design considerations for stand-alone databases and for elastic pools, see
Design an application for cloud disaster recovery and Elastic pool disaster recovery strategies.
High-availability and Azure SQL Database
9/10/2018 • 8 minutes to read • Edit Online
Azure SQL Database is highly available database Platform as a Service that guarantees that your database is
up and running 99.99% of time, without worrying about maintenance and downtimes. This is a fully managed
SQL Server Database Engine process hosted in the Azure cloud that ensures that your SQL Server database is
always upgraded/patched without affecting your workload. When an instance is patched or fails over, the
downtime is generally not noticable if you employ retry logic in your app. If the time to complete a failover is
longer than 60 seconds, you should open a support case. Azure SQL Database can quickly recover even in the
most critical circumstances ensuring that your data is always available.
Azure platform fully manages every Azure SQL Database and guarantees no data loss and a high percentage
of data availability. Azure automatically handles patching, backups, replication, failure detection, underlying
potential hardware, software or network failures, deploying bug fixes, failovers, database upgrades, and other
maintenance tasks. SQL Server engineers have implemented the best-known practices, ensuring that all the
maintenance operations are completed in less than 0.01% time of your database life. This architecture is
designed to ensure that committed data is never lost and that maintenance operations are performed without
affecting workload. There are no maintenance windows or downtimes that should require you to stop the
workload while the database is upgraded or maintained. Built-in high availability in Azure SQL Database
guarantees that database will never be single point of failure in your software architecture.
Azure SQL Database is based on SQL Server Database Engine architecture that is adjusted for the cloud
environment in order to ensure 99.99% availability even in the cases of infrastructure failures. There are two
high-availability architectural models that are used in Azure SQL Database (both of them ensuring 99.99%
availability):
Standard/general purpose model that is based on a separation of compute and storage. This architectural
model relies on high availability and reliability of storage tier, but it might have some potential performance
degradation during maintenance activities.
Premium/business critical model that is based on a cluster of database engine processes. This architectural
model relies on a fact that there is always a quorum of available database engine nodes and has minimal
performance impact on your workload even during maintenance activities.
Azure upgrades and patches underlying operating system, drivers, and SQL Server Database Engine
transparently with the minimal down-time for end users. Azure SQL Database runs on the latest stable version
of SQL Server Database Engine and Windows OS, and most of the users would not notice that the upgrades
are performed continuously.
IMPORTANT
Zone redundant databases and elastic pools are currently only supported in the Premium service tier. During public
preview, backups and audit records are stored in RA-GRS storage and therefore may not be automatically available in
case of a zone-wide outage.
The zone redundant version of the high availability architecture is illustrated by the following diagram:
Read scale-out
As described, Premium and Business Critical service tiers leverage quorum-sets and Always On technology for
High Availability both in single zone and zone redundant configurations. One of the benefits of AlwaysOn is
that the replicas are always in the transactionally consistent state. Because the replicas have the same
performance level as the primary, the application can take advantage of that extra capacity for servicing the
read-only workloads at no extra cost (read scale-out). This way the read-only queries will be isolated from the
main read-write workload and will not affect its performance. Read scale-out feature is intended for the
applications that include logically separated read-only workloads such as analytics, and therefore could
leverage this additional capacity without connecting to the primary.
To use the Read Scale-Out feature with a particular database, you must explicitly activate it when creating the
database or afterwards by altering its configuration using PowerShell by invoking the Set-
AzureRmSqlDatabase or the New -AzureRmSqlDatabase cmdlets or through the Azure Resource Manager
REST API using the Databases - Create or Update method.
After Read Scale-Out is enabled for a database, applications connecting to that database will be directed to
either the read-write replica or to a read-only replica of that database according to the ApplicationIntent
property configured in the application’s connection string. For information on the ApplicationIntent property,
see Specifying Application Intent.
If Read Scale-Out is disabled or you set the ReadScale property in an unsupported service tier, all connections
are directed to the read-write replica, independent of the ApplicationIntent property.
Conclusion
Azure SQL Database is deeply integrated with the Azure platform and is highly dependent on Service Fabric
for failure detection and recovery, on Azure Storage Blobs for data protection and Availability Zones for higher
fault tolerance. At the same time, Azure SQL database fully leverages the Always On Availability Group
technology from SQL Server box product for replication and failover. The combination of these technologies
enables the applications to fully realize the benefits of a mixed storage model and support the most demanding
SLAs.
Next steps
Learn about Azure Availability Zones
Learn about Service Fabric
Learn about Azure Traffic Manager
Learn about automatic SQL Database backups
8/29/2018 • 7 minutes to read • Edit Online
SQL Database automatically creates database backups and uses Azure read-access geo-redundant storage
(RA-GRS ) to provide geo-redundancy. These backups are created automatically and at no additional charge.
You don't need to do anything to make them happen. Database backups are an essential part of any business
continuity and disaster recovery strategy because they protect your data from accidental corruption or
deletion. If your security rules require that your backups are available for an extended period of time, you can
configure a long-term backup retention policy. For more information, see Long-term retention.
NOTE
This article provides steps for how to delete personal data from the device or service and can be used to support your
obligations under the GDPR. If you’re looking for general info about GDPR, see the GDPR section of the Service Trust
portal.
NOTE
In Azure storage, the term replication refers to copying files from one location to another. SQL's database replication
refers to keeping multiple secondary databases synchronized with a primary database.
IMPORTANT
If you delete the Azure SQL server that hosts SQL databases, all elastic pools and databases that belong to the server
are also deleted and cannot be recovered. You cannot restore a deleted server. But if you configured long-term
retention, the backups for the databases with LTR will not be deleted and these databases can be restored.
NOTE
This article provides steps for how to delete personal data from the device or service and can be used to support your
obligations under the GDPR. If you’re looking for general info about GDPR, see the GDPR section of the Service Trust
portal.
NOTE
Thes APIs will only impact the PITR retention period. If you configured LTR for your database, it will not be impacted.
See Long-term backup retention for details of how to change the LTR retention period(s).
IMPORTANT
This API is included in AzureRM.Sql PowerShell Module starting from version 4.7.0-preview.
Request Body
{
"properties":{
"retentionDays":28
}
}
Sample Response
Status code: 200
{
"id": "/subscriptions/00000000-1111-2222-3333-
444444444444/providers/Microsoft.Sql/resourceGroups/resourceGroup/servers/testserver/databases/testDatabas
e/backupShortTermRetentionPolicies/default",
"name": "default",
"type": "Microsoft.Sql/resourceGroups/servers/databases/backupShortTermRetentionPolicies",
"properties": {
"retentionDays": 28
}
}
Next steps
Database backups are an essential part of any business continuity and disaster recovery strategy because
they protect your data from accidental corruption or deletion. To learn about the other Azure SQL
Database business continuity solutions, see Business continuity overview.
To restore to a point in time using the Azure portal, see restore database to a point in time using the Azure
portal.
To restore to a point in time using PowerShell, see restore database to a point in time using PowerShell.
To configure, manage, and restore from long-term retention of automated backups in Azure blob storage
using the Azure portal, see Manage long-term backup retention using the Azure portal.
To configure, manage, and restore from long-term retention of automated backups in Azure blog storage
using PowerShell, see Manage long-term backup retention using PowerShell.
Store Azure SQL Database backups for up to 10
years
7/16/2018 • 3 minutes to read • Edit Online
Many applications have regulatory, compliance, or other business purposes that require you to retain database
backups beyond the 7-35 days provided by Azure SQL Database automatic backups. By using the long-term
retention (LTR ) feature, you can store specified SQL database full backups in RA-GRS blob storage for up to
10 years. You can then restore any backup as a new database.
NOTE
1. The LTR copies are created by Azure storage service so the copy process has no performance impact on the existing
database.
2. The policy applies to the future backups. E.g. if the specified WeekOfYear is in the past when the policy is configured,
the first LTR backup will be created next year.
3. To restore a database from the LTR storage, you can select a specific backup based on its timestamp. The database
can be restored to any existing server under the same subscription as the original database.
NOTE
When the original primary database recovers from the outage that cause it to failover, it will become a new secondary.
Therefore, the backup creation will not resume and the existing LTR policy will not take effect until it becomes the
primary again.
Next steps
Because database backups protect data from accidental corruption or deletion, they're an essential part of any
business continuity and disaster recovery strategy. To learn about the other SQL Database business-continuity
solutions, see Business continuity overview.
Manage Azure SQL Database long-term backup
retention
9/7/2018 • 4 minutes to read • Edit Online
You can configure Azure SQL database with a long-term backup retention policy (LTR ) to automatically retain
backups in Azure blob storage for up to 10 years. You can then recover a database using these backups using the
Azure portal or PowerShell.
2. In the Configure policies pane, select if want to retain weekly, monthly or yearly backups and specify the
retention period for each.
3. When complete, click Apply.
View backups and restore from a backup using Azure portal
View the backups that are retained for a specific database with a LTR policy, and restore from those backups.
1. In the Azure portal, select your SQL server and then click Manage Backups. On the Available backups
tab, select the database for which you want to see available backups.
4. Click OK to restore your database from the backup in Azure SQL storage to the new database.
5. On the toolbar, click the notification icon to view the status of the restore job.
6. When the restore job is completed, open the SQL databases page to view the newly restored database.
NOTE
From here, you can connect to the restored database using SQL Server Management Studio to perform needed tasks, such
as to extract a bit of data from the restored database to copy into the existing database or to delete the existing database
and rename the restored database to the existing database name.
Use PowerShell to configure long-term retention policies and restore
backups
The following sections show you how to use PowerShell to configure the long-term backup retention, view
backups in Azure SQL storage, and restore from a backup in Azure SQL storage.
IMPORTANT
LTR V2 API is supported in the following PowerShell versions:
AzureRM.Sql-4.5.0 or newer
AzureRM-6.1.0 or newer
Connect-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId $subId
# create LTR policy with WeeklyRetention = 12 weeks. MonthlyRetention and YearlyRetention = 0 by default.
Set-AzureRmSqlDatabaseBackupLongTermRetentionPolicy -ServerName $serverName -DatabaseName $dbName -
ResourceGroupName $resourceGroup -WeeklyRetention P12W
# create LTR policy with WeeklyRetention = 12 weeks, YearlyRetetion = 5 years and WeekOfYear = 16 (week of
April 15). MonthlyRetention = 0 by default.
Set-AzureRmSqlDatabaseBackupLongTermRetentionPolicy -ServerName $serverName -DatabaseName $dbName -
ResourceGroupName $resourceGroup -WeeklyRetention P12W -YearlyRetention P5Y -WeekOfYear 16
# Get the list of LTR backups from the Azure region under
# the named server.
$ltrBackups = Get-AzureRmSqlDatabaseLongTermRetentionBackup -Location $server.Location -ServerName $serverName
# Get the LTR backups for a specific database from the Azure region under the named server
$ltrBackups = Get-AzureRmSqlDatabaseLongTermRetentionBackup -Location $server.Location -ServerName $serverName
-DatabaseName $dbName
# List LTR backups only from live databases (you have option to choose All/Live/Deleted)
$ltrBackups = Get-AzureRmSqlDatabaseLongTermRetentionBackup -Location $server.Location -DatabaseState Live
NOTE
From here, you can connect to the restored database using SQL Server Management Studio to perform needed tasks, such
as to extract a bit of data from the restored database to copy into the existing database or to delete the existing database
and rename the restored database to the existing database name. See point in time restore.
Next steps
To learn about service-generated automatic backups, see automatic backups
To learn about long-term backup retention, see long-term backup retention
Configure and restore from Azure SQL Database
long-term backup retention using Azure Recovery
Services Vault
9/12/2018 • 8 minutes to read • Edit Online
You can configure the Azure Recovery Services vault to store Azure SQL database backups and then recover a
database using backups retained in the vault using the Azure portal or PowerShell.
NOTE
As part of the initial release of the preview of long-term backup retention in October 2016, backups were stored in the Azure
Services Recovery Service vault. This update removes this dependency, but for backward compatibility the original API is
supported until May 31, 2018. If you need to interact with backups in the Azure Services Recovery vault, see Long-term
backup retention using Azure Services Recovery Service vault.
Azure portal
The following sections show you how to use the Azure portal to configure the Azure Recovery Services vault, view
backups in the vault, and restore from the vault.
Configure the vault, register the server, and select databases
You configure an Azure Recovery Services vault to retain automated backups for a period longer than the retention
period for your service tier.
1. Open the SQL Server page for your server.
2. Click Long-term backup retention.
3. On the Long-term backup retention page for your server, review and accept the preview terms (unless
you have already done so - or this feature is no longer in preview ).
4. To configure long-term backup retention, select that database in the grid and then click Configure on the
toolbar.
5. On the Configure page, click Configure required settings under Recovery service vault.
6. On the Recovery services vault page, select an existing vault, if any. Otherwise, if no recovery services
vault found for your subscription, click to exit the flow and create a recovery services vault.
9. Select your subscription and resource group, and then select the location for the vault. When done, click
Create.
IMPORTANT
The vault must be located in the same region as the Azure SQL logical server, and must use the same resource group
as the logical server.
10. After the new vault is created, execute the necessary steps to return to the Recovery services vault page.
11. On the Recovery services vault page, click the vault and then click Select.
12. On the Configure page, provide a valid name for the new retention policy, modify the default retention
policy as appropriate, and then click OK.
NOTE
Retention policy names don't allow some characters, including spaces.
13. On the Long-term backup retention page for your database, click Save and then click OK to apply the
long-term backup retention policy to all selected databases.
14. Click Save to enable long-term backup retention using this new policy to the Azure Recovery Services vault
that you configured.
IMPORTANT
Once configured, backups show up in the vault within next seven days. Do not continue this tutorial until backups show up in
the vault.
View backups in long-term retention using Azure portal
View information about your database backups in long-term backup retention.
1. In the Azure portal, open your Azure Recovery Services vault for your database backups (go to All
resources and select it from the list of resources for your subscription) to view the amount of storage used
by your database backups in the vault.
3. Click OK to restore your database from the backup in the vault to the new database.
4. On the toolbar, click the notification icon to view the status of the restore job.
5. When the restore job is completed, open the SQL databases page to view the newly restored database.
NOTE
From here, you can connect to the restored database using SQL Server Management Studio to perform needed tasks, such
as to extract a bit of data from the restored database to copy into the existing database or to delete the existing database
and rename the restored database to the existing database name.
PowerShell
The following sections show you how to use PowerShell to configure the Azure Recovery Services vault, view
backups in the vault, and restore from the vault.
Create a recovery services vault
Use the New -AzureRmRecoveryServicesVault to create a recovery services vault.
IMPORTANT
The vault must be located in the same region as the Azure SQL logical server, and must use the same resource group as the
logical server.
#$resourceGroupName = "{resource-group-name}"
#$serverName = "{server-name}"
$serverLocation = (Get-AzureRmSqlServer -ServerName $serverName -ResourceGroupName $resourceGroupName).Location
$recoveryServiceVaultName = "{new-vault-name}"
Set your server to use the recovery vault for its long-term retention backups
Use the Set-AzureRmSqlServerBackupLongTermRetentionVault cmdlet to associate a previously created recovery
services vault with a specific Azure SQL server.
# Set your server to use the vault to for long-term backup retention
NOTE
Some cmdlets require that you set the vault context before running (Set-AzureRmRecoveryServicesVaultContext) so you see
this cmdlet in a few related snippets. You set the context because the policy is part of the vault. You can create multiple
retention policies for each vault and then apply the desired policy to specific databases.
# Retrieve the default retention policy for the AzureSQLDatabase workload type
$retentionPolicy = Get-AzureRmRecoveryServicesBackupRetentionPolicyObject -WorkloadType AzureSQLDatabase
# Set the retention value to two years (you can set to any time between 1 week and 10 years)
$retentionPolicy.RetentionDurationType = "Years"
$retentionPolicy.RetentionCount = 2
$retentionPolicyName = "my2YearRetentionPolicy"
# Set the vault context to the vault you are creating the policy for
Set-AzureRmRecoveryServicesVaultContext -Vault $vault
# the following commands find the container associated with the server 'myserver' under resource group
'myresourcegroup'
$container = Get-AzureRmRecoveryServicesBackupContainer -ContainerType AzureSQL -FriendlyName $vault.Name
NOTE
From here, you can connect to the restored database using SQL Server Management Studio to perform needed tasks, such
as to extract a bit of data from the restored database to copy into the existing database or to delete the existing database
and rename the restored database to the existing database name. See point in time restore.
Login-AzureRmAccount
$vaults = Get-AzureRmRecoveryServicesVault
$vault = $vaults | where { $_.Name -eq $VaultName }
Next steps
To learn about service-generated automatic backups, see automatic backups
To learn about long-term backup retention, see long-term backup retention
To learn about restoring from backups, see restore from backup
Overview: Active geo-replication and auto-failover
groups
9/7/2018 • 23 minutes to read • Edit Online
Active geo-replication is Azure SQL Database feature that allows you to create readable replicas of your
database in the same or different data center (region).
Active geo-replication is designed as a business continuity solution that allows the application to perform
quick disaster recovery in case of a data center scale outage. If geo-replication is enabled, the application can
initiate failover to a secondary database in a different Azure region. Up to four secondaries are supported in
the same or different regions, and the secondaries can also be used for read-only access queries. The failover
must be initiated manually by the application or the user. After failover, the new primary has a different
connection end point.
NOTE
Active geo-replication is available for all databases in all service tiers in all regions. Active Geo-replication is not
available in Managed Instance.
If you are using active geo-replication and for any reason your primary database fails, or simply needs to be
taken offline, you can initiate failover to any of your secondary databases. When failover is activated to one of
the secondary databases, all other secondaries are automatically linked to the new primary. If you are using
auto-failover groups to manage database recovery and any outage that impacts one or several of the
databases in the group results in automatic failover. You can configure the auto-failover policy that best meets
your application needs, or you can opt out and use manual activation. In addition, auto-failover groups
provide read-write and read-only listener end-points that remain unchanged during failovers. Whether you
use manual or automatic failover activation, failover switches all secondary databases in the group to
primary. After the database failover is completed, the DNS record is automatically updated to redirect the
end-points to the new region.
You can manage replication and failover of an individual database or a set of databases on a server or in an
elastic pool using active geo-replication. You can do that using
The Azure portal
PowerShell: Single database
PowerShell: Elastic pool
PowerShell: Failover Group
Transact-SQL: Single database or elastic pool
REST API: Single database
REST API: Failover group.
After failover, ensure the authentication requirements for your server and database are configured on the
new primary. For details, see SQL Database security after disaster recovery. This applies both to active geo-
replication and auto-failover groups.
Active geo-replication leverages the Always On technology of SQL Server to asynchronously replicate
committed transactions on the primary database to a secondary database using snapshot isolation. Auto-
failover groups provide the group semantics on top of active geo-replication but the same asynchronous
replication mechanism is used. While at any given point, the secondary database might be slightly behind the
primary database, the secondary data is guaranteed to never have partial transactions. Cross-region
redundancy enables applications to quickly recover from a permanent loss of an entire datacenter or parts of
a datacenter caused by natural disasters, catastrophic human errors, or malicious acts. The specific RPO data
can be found at Overview of Business Continuity.
The following figure shows an example of active geo-replication configured with a primary in the North
Central US region and secondary in the South Central US region.
Because the secondary databases are readable, they can be used to offload read-only workloads such as
reporting jobs. If you are using active geo-replication, it is possible to create the secondary database in the
same region with the primary, but it does not increase the application's resilience to catastrophic failures. If
you are using auto-failover groups, your secondary database is always created in a different region.
In addition to disaster recovery active geo-replication can be used in the following scenarios:
Database migration: You can use active geo-replication to migrate a database from one server to
another online with minimum downtime.
Application upgrades: You can create an extra secondary as a fail back copy during application
upgrades.
To achieve real business continuity, adding database redundancy between datacenters is only part of the
solution. Recovering an application (service) end-to-end after a catastrophic failure requires recovery of all
components that constitute the service and any dependent services. Examples of these components include
the client software (for example, a browser with a custom JavaScript), web front ends, storage, and DNS. It is
critical that all components are resilient to the same failures and become available within the recovery time
objective (RTO ) of your application. Therefore, you need to identify all dependent services and understand the
guarantees and capabilities they provide. Then, you must take adequate steps to ensure that your service
functions during the failover of the services on which it depends. For more information about designing
solutions for disaster recovery, see Designing Cloud Solutions for Disaster Recovery Using active geo-
replication.
NOTE
The log replay is delayed on the secondary database if there are schema updates on the Primary. The latter requires a
schema lock on the secondary database.
Multiple readable secondaries: Two or more secondary databases increase redundancy and level of
protection for the primary database and application. If multiple secondary databases exist, the application
remains protected even if one of the secondary databases fails. If there is only one secondary database,
and it fails, the application is exposed to higher risk until a new secondary database is created.
NOTE
If you are using active geo-replication to build a globally distributed application and need to provide read-only access
to data in more than four regions, you can create secondary of a secondary (a process known as chaining). This way
you can achieve virtually unlimited scale of database replication. In addition, chaining reduces the overhead of
replication from the primary database. The trade-off is the increased replication lag on the leaf-most secondary
databases.
Support of elastic pool databases: Each replica can separately participate in an Elastic Pool or not be in
any elastic pool at all. The pool choice for each replica is separate and does not depend upon the
configuration of any other replica (whether Primary or Secondary). Each Elastic Pool is contained within a
single region, therefore multiple replicas in the same topology can never share an Elastic Pool.
Configurable performance level of the secondary database: Both primary and secondary databases
are required to have the same service tier. It is also strongly recommended that secondary database is
created with the same performance level (DTUs or vCores) as the primary. A secondary with lower
performance level is at risk of an increased replication lag, potential unavailability of the secondary, and
consequently at risk of substantial data loss after a failover. As a result, the published RPO = 5 sec cannot
be guaranteed. The other risk is that after failover the application’s performance will be impacted due to
insufficient compute capacity of the new primary until it is upgraded to a higher performance level. The
time of the upgrade depends on the database size. If you decide to create the secondary with lower
performance level, the log IO percentage chart on Azure portal provides a good way to estimate the
minimal performance level of the secondary that is required to sustain the replication load. For example, if
your Primary database is P6 (1000 DTU ) and its log IO percent is 50% the secondary needs to be at least
P4 (500 DTU ). You can also retrieve the log IO data using sys.resource_stats or sys.dm_db_resource_stats
database views. For more information on the SQL Database performance levels, see What are SQL
Database Service Tiers.
User-controlled failover and failback: A secondary database can explicitly be switched to the primary
role at any time by the application or the user. During a real outage the “unplanned” option should be
used, which immediately promotes a secondary to be the primary. When the failed primary recovers and
is available again, the system automatically marks the recovered primary as a secondary and bring it up-
to-date with the new primary. Due to the asynchronous nature of replication, a small amount of data can
be lost during unplanned failovers if a primary fails before it replicates the most recent changes to the
secondary. When a primary with multiple secondaries fails over, the system automatically reconfigures the
replication relationships and links the remaining secondaries to the newly promoted primary without
requiring any user intervention. After the outage that caused the failover is mitigated, it may be desirable
to return the application to the primary region. To do that, the failover command should be invoked with
the “planned” option.
Keeping credentials and firewall rules in sync: We recommend using database firewall rules for geo-
replicated databases so these rules can be replicated with the database to ensure all secondary databases
have the same firewall rules as the primary. This approach eliminates the need for customers to manually
configure and maintain firewall rules on servers hosting both the primary and secondary databases.
Similarly, using contained database users for data access ensures both primary and secondary databases
always have the same user credentials so during a failover, there is no disruptions due to mismatches with
logins and passwords. With the addition of Azure Active Directory, customers can manage user access to
both primary and secondary databases and eliminating the need for managing credentials in databases
altogether.
NOTE
When adding a database that already has a secondary database in a server that is not part of the failover group, a new
secondary is created in the secondary server.
NOTE
When system detects that the databases in the group are still online (for example, the outage only impacted the
service control plane), it immediately activates the failover with full data synchronization (friendly failover) regardless of
the value set by GracePeriodWithDataLossHours. This behavior ensures that there is no data loss during the
recovery. The grace period takes effect only when a friendly failover is not possible. If the outage is mitigated before the
grace period expires, the failover is not activated.
Multiple failover groups: You can configure multiple failover groups for the same pair of servers to
control the scale of failovers. Each group fails over independently. If your multi-tenant application uses
elastic pools, you can use this capability to mix primary and secondary databases in each pool. This way
you can reduce the impact of an outage to only half of the tenants.
IMPORTANT
Elastic pools with 800 or less DTUs and more than 250 databases using geo-replication may encounter issues including
longer planned failovers and degraded performance. These issues are more likely to occur for write intensive workloads,
when geo-replication endpoints are widely separated by geography, or when multiple secondary endpoints are used
for each database. Symptoms of these issues are indicated when the geo-replication lag increases over time. This lag
can be monitored using sys.dm_geo_replication_link_status. If these issues occur, then mitigations include increasing
the number of pool DTUs, or reducing the number of geo-replicated databases in the same pool.
NOTE
If you are using the read-only listener to load-balance a read-only workload, make sure that this workload is
executed in a VM or other resorce in the secondary region so it can connect to the secondary database.
Using failover groups and SQL database firewall rules
If your business continuity plan requires failover using groups with automatic failover, you can restrict access
to your SQL database using the traditional firewall rules. To support automatic failover, follow these steps:
1. Create a public IP
2. Create a public load balancer and assign the public IP to it.
3. Create a virtual network and the virtual machines for your front-end components
4. Create network security group and configure inbound connections.
5. Ensure that the outbound connections are open to Azure SQL database by using ‘Sql’ service tag.
6. Create a SQL database firewall rule to allow inbound traffic from the public IP address you create in step
1.
For more information about on how to configure outbound access and what IP to use in the firewall rules, see
Load balancer outbound connections.
The above configuration will ensure that the automatic failover will not block connections from the front-end
components and assumes that the application can tolerate the longer latency between the front end and the
data tier.
IMPORTANT
To guarantee business continuity for regional outages you must ensure geographic redundancy for both front-end
components and the databases.
NOTE
If you created secondary database as part of the failover group configuration it is not recommended to downgrade the
secondary database. This is to ensure your data tier has sufficient capacity to process your regular workload after
failover is activated.
ALTER DATABASE (Azure SQL Database) Use ADD SECONDARY ON SERVER argument to create a
secondary database for an existing database and starts
data replication
ALTER DATABASE (Azure SQL Database) Use REMOVE SECONDARY ON SERVER to terminate a data
replication between a SQL Database and the specified
secondary database.
sys.geo_replication_links (Azure SQL Database) Returns information about all existing replication links for
each database on the Azure SQL Database logical server.
sys.dm_geo_replication_link_status (Azure SQL Database) Gets the last replication time, last replication lag, and other
information about the replication link for a given SQL
database.
sys.dm_operation_status (Azure SQL Database) Shows the status for all database operations including the
status of the replication links.
sp_wait_for_database_copy_sync (Azure SQL Database) causes the application to wait until all committed
transactions are replicated and acknowledged by the active
secondary database.
Remove-AzureRmSqlDatabaseFailoverGroup Removes the failover group from the server and deletes all
secondary databases included the group
IMPORTANT
For sample scripts, see Configure and failover a single database using active geo-replication, Configure and failover a
pooled database using active geo-replication, and Configure and failover a failover group for a single database.
Get Create or Update Database Status Returns the status during a create operation.
Set Secondary Database as Primary (Planned Failover) Sets which replica database is primary by failing over from
the current primary replica database.
Set Secondary Database as Primary (Unplanned Failover) Sets which replica database is primary by failing over from
the current primary replica database. This operation might
result in data loss.
API DESCRIPTION
Get Replication Link Gets a specific replication link for a given SQL database in a
geo-replication partnership. It retrieves the information
visible in the sys.geo_replication_links catalog view.
Replication Links - List By Database Gets all replication links for a given SQL database in a geo-
replication partnership. It retrieves the information visible
in the sys.geo_replication_links catalog view.
Delete Replication Link Deletes a database replication link. Cannot be done during
failover.
Delete Failover Group Removes the failover group from the server
Failover (Planned) Fails over from the current primary server to this server.
Force Failover Allow Data Loss ails over from the current primary server to this server. This
operation might result in data loss.
Next steps
For sample scripts, see:
Configure and failover a single database using active geo-replication
Configure and failover a pooled database using active geo-replication
Configure and failover a failover group for a single database
For a business continuity overview and scenarios, see Business continuity overview
To learn about Azure SQL Database automated backups, see SQL Database automated backups.
To learn about using automated backups for recovery, see Restore a database from the service-initiated
backups.
To learn about authentication requirements for a new primary server and database, see SQL Database
security after disaster recovery.
Configure active geo-replication for Azure SQL
Database in the Azure portal and initiate failover
9/14/2018 • 3 minutes to read • Edit Online
This article shows you how to configure active geo-replication for SQL Database in the Azure portal and to
initiate failover.
To initiate failover with the Azure portal, see Initiate a planned or unplanned failover for Azure SQL Database
with the Azure portal.
To configure active geo-replication by using the Azure portal, you need the following resource:
An Azure SQL database: The primary database that you want to replicate to a different geographical region.
NOTE
Active geo-replication must be between databases in the same subscription.
NOTE
If the partner database already exists (for example, as a result of terminating a previous geo-replication relationship) the
command fails.
1. In the Azure portal, browse to the database that you want to set up for geo-replication.
2. On the SQL database page, select geo-replication, and then select the region to create the secondary
database. You can select any region other than the region hosting the primary database, but we
recommend the paired region.
3. Select or configure the server and pricing tier for the secondary database.
4. Optionally, you can add a secondary database to an elastic pool. To create the secondary database in a pool,
click elastic pool and select a pool on the target server. A pool must already exist on the target server. This
workflow does not create a pool.
5. Click Create to add the secondary.
6. The secondary database is created and the seeding process begins.
7. When the seeding process is complete, the secondary database displays its status.
Initiate a failover
The secondary database can be switched to become the primary.
1. In the Azure portal, browse to the primary database in the geo-replication partnership.
2. On the SQL Database blade, select All settings > geo-replication.
3. In the SECONDARIES list, select the database you want to become the new primary and click Failover.
4. Click Yes to begin the failover.
The command immediately switches the secondary database into the primary role.
There is a short period during which both databases are unavailable (on the order of 0 to 25 seconds) while the
roles are switched. If the primary database has multiple secondary databases, the command automatically
reconfigures the other secondaries to connect to the new primary. The entire operation should take less than a
minute to complete under normal circumstances.
NOTE
This command is designed for quick recovery of the database in case of an outage. It triggers failover without data
synchronization (forced failover). If the primary is online and committing transactions when the command is issued some
data loss may occur.
5. A confirmation window opens. Click Yes to remove the database from the geo-replication partnership. (Set it
to a read-write database not part of any replication.)
Next steps
To learn more about active geo-replication, see active geo-replication.
For a business continuity overview and scenarios, see Business continuity overview.
Configure and manage Azure SQL Database security
for geo-restore or failover
5/23/2018 • 4 minutes to read • Edit Online
This topic describes the authentication requirements to configure and control active geo-replication and the steps
required to set up user access to the secondary database. It also describes how to enable access to the recovered
database after using geo-restore. For more information on recovery options, see Business Continuity Overview.
NOTE
Active geo-replication is now available for all databases in all service tiers.
NOTE
If you fail over or geo-restore to a server that does not have properly configured logins, access to it will be limited to the
server admin account.
Setting up logins on the target server involves three steps outlined below:
1. Determine logins with access to the primary database:
The first step of the process is to determine which logins must be duplicated on the target server. This is
accomplished with a pair of SELECT statements, one in the logical master database on the source server and one
in the primary database itself.
Only the server admin or a member of the LoginManager server role can determine the logins on the source
server with the following SELECT statement.
Only a member of the db_owner database role, the dbo user, or server admin, can determine all of the database
user principals in the primary database.
NOTE
The INFORMATION_SCHEMA and sys users have NULL SIDs, and the guest SID is 0x00. The dbo SID may start with
0x01060000000001648000000000048454, if the database creator was the server admin instead of a member of
DbManager.
Next steps
For more information on managing database access and logins, see SQL Database security: Manage database
access and login security.
For more information on contained database users, see Contained Database Users - Making Your Database
Portable.
For information about using and configuring active geo-replication, see active geo-replication
For information about using geo-restore, see geo-restore
Recover an Azure SQL database using automated
database backups
8/20/2018 • 7 minutes to read • Edit Online
SQL Database provides these options for database recovery using automated database backups and backups in
long-term retention. You can restore from a database backup to:
A new database on the same logical server recovered to a specified point in time within the retention period.
A database on the same logical server recovered to the deletion time for a deleted database.
A new database on any logical server in any region recovered to the point of the most recent daily backups in
geo-replicated blob storage (RA-GRS ).
IMPORTANT
You cannot overwrite an existing database during restore.
A restored database incurs an extra storage cost under the following conditions:
Restore of P11–P15 to S4-S12 or P1–P6 if the database max size is greater than 500 GB.
Restore of P1–P6 to S4-S12 if the database max size is greater than 250 GB.
The extra cost is because the max size of the restored database is greater than the amount of storage included
for the performance level, and any extra storage provisioned above the included amount is charged extra. For
pricing details of extra storage, see the SQL Database pricing page. If the actual amount of space used is less
than the amount of storage included, then this extra cost can be avoided by reducing the database max size to
the included amount.
NOTE
Automated database backups are used when you create a database copy.
Recovery time
The recovery time to restore a database using automated database backups is impacted by several factors:
The size of the database
The performance level of the database
The number of transaction logs involved
The amount of activity that needs to be replayed to recover to the restore point
The network bandwidth if the restore is to a different region
The number of concurrent restore requests being processed in the target region.
For a very large and/or active database, the restore may take several hours. If there is prolonged outage
in a region, it is possible that there are large numbers of geo-restore requests being processed by other
regions. When there are many requests, the recovery time may increase for databases in that region.
Most database restores complete within 12 hours.
For a single subscription, there are some limitations on number of concurrent restore requests (including point
in time restore, geo restore and restore from long-term retention backup) being submitted and proceeded:
MAX # OF CONCURRENT REQUESTS BEING MAX # OF CONCURRENT REQUESTS BEING
PROCESSED SUBMITTED
There is no built-in functionality to do bulk restore. The Azure SQL Database: Full Server Recovery script is an
example of one way of accomplishing this task.
IMPORTANT
To recover using automated backups, you must be a member of the SQL Server Contributor role in the subscription or be
the subscription owner - see RBAC: Built-in roles. You can recover using the Azure portal, PowerShell, or the REST API. You
cannot use Transact-SQL.
Point-in-time restore
You can restore an existing database to an earlier point in time as a new database on the same logical server
using the Azure portal, PowerShell, or the REST API.
TIP
For a sample PowerShell script showing how to perform a point-in-time restore of a database, see Restore a SQL database
using PowerShell.
The database can be restored to any service tier or performance level, and as a single database or into an elastic
pool. Ensure you have sufficient resources on the logical server or in the elastic pool to which you are restoring
the database. Once complete, the restored database is a normal, fully accessible, online database. The restored
database is charged at normal rates based on its service tier and performance level. You do not incur charges
until the database restore is complete.
You generally restore a database to an earlier point for recovery purposes. When doing so, you can treat the
restored database as a replacement for the original database or use it to retrieve data from and then update the
original database.
Database replacement: If the restored database is intended as a replacement for the original database, you
should verify the performance level and/or service tier are appropriate and scale the database if necessary.
You can rename the original database and then give the restored database the original name using the
ALTER DATABASE command in T-SQL.
Data recovery: If you plan to retrieve data from the restored database to recover from a user or application
error, you need to write and execute the necessary data recovery scripts to extract data from the restored
database to the original database. Although the restore operation may take a long time to complete, the
restoring database is visible in the database list throughout the restore process. If you delete the database
during the restore, the restore operation is canceled and you are not charged for the database that did not
complete the restore.
Azure portal
To recover to a point in time using the Azure portal, open the page for your database and click Restore on the
toolbar.
Deleted database restore
You can restore a deleted database to the deletion time for a deleted database on the same logical server using
the Azure portal, PowerShell, or the REST (createMode=Restore). You can restore a deleted database to an
earlier point in time during the retention using PowerShell.
TIP
For a sample PowerShell script showing how to restore a deleted database, see Restore a SQL database using PowerShell.
IMPORTANT
If you delete an Azure SQL Database server instance, all its databases are also deleted and cannot be recovered. There is
currently no support for restoring a deleted server.
Azure portal
To recover a deleted database during its DTU -based model retention period or vCore-based model retention
period using the Azure portal, open the page for your server and in the Operations area, click Deleted
databases.
Geo-restore
You can restore a SQL database on any server in any Azure region from the most recent geo-replicated full and
differential backups. Geo-restore uses a geo-redundant backup as its source and can be used to recover a
database even if the database or datacenter is inaccessible due to an outage.
Geo-restore is the default recovery option when your database is unavailable because of an incident in the
region where the database is hosted. If a large-scale incident in a region results in unavailability of your
database application, you can restore a database from the geo-replicated backups to a server in any other
region. There is a delay between when a differential backup is taken and when it is geo-replicated to an Azure
blob in a different region. This delay can be up to an hour, so, if a disaster occurs, there can be up to one hour
data loss. The following illustration shows restore of the database from the last available backup in another
region.
TIP
For a sample PowerShell script showing how to perform a geo-restore, see Restore a SQL database using PowerShell.
Point-in-time restore on a geo-secondary is not currently supported. Point-in-time restore can be done only on
a primary database. For detailed information about using geo-restore to recover from an outage, see Recover
from an outage.
IMPORTANT
Recovery from backups is the most basic of the disaster recovery solutions available in SQL Database with the longest
Recovery Point Objective (RPO) and Estimate Recovery Time (ERT). For solutions using small size databases (e.g. Basic
service tier or small size tenant databases in elastic pools), geo-restore is frequently a reasonable DR solution with an ERT
of 12 hours. For solutions using large databases and require shorter recovery times, you should consider using Failover
groups and active geo-replication. Active geo-replication offers a much lower RPO and ERT as it only requires you initiate
a failover to a continuously replicated secondary. For more information on business continuity choices, see Overview of
business continuity.
Azure portal
To geo-restore a database during its DTU -based model retention period or vCore-based model retention period
using the Azure portal, open the SQL Databases page and then click Add. In the Select source text box, select
Backup. Specify the backup from which to perform the recovery in the region and on the server of your choice.
REST API
API DESCRIPTION
Get Create or Update Database Status Returns the status during a restore operation
Summary
Automatic backups protect your databases from user and application errors, accidental database deletion, and
prolonged outages. This built-in capability is available for all service tiers and performance levels.
Next steps
For a business continuity overview and scenarios, see Business continuity overview.
To learn about Azure SQL Database automated backups, see SQL Database automated backups.
To learn about long-term retention, see Long-term retention.
To learn about faster recovery options, see Failover groups and active geo-replication.
Restore an Azure SQL Database or failover to a
secondary
7/16/2018 • 5 minutes to read • Edit Online
Azure SQL Database offers the following capabilities for recovering from an outage:
Active geo-replication and failover groups
Geo-restore
Zone-redundant databases
To learn about business continuity scenarios and the features supporting these scenarios, see Business continuity.
NOTE
If you are using zone-redundant Premium or Business Critical databases or pools, the recovery process is automated and
the rest of this material does not apply.
NOTE
If you are using failover groups and chose automatic failover, the recovery process is automated and transparent to the
application.
Depending on your application tolerance to downtime and possible business liability you can consider the
following recovery options.
Use the Get Recoverable Database (LastAvailableBackupDate) to get the latest Geo-replicated restore point.
NOTE
You should configure and test your server firewall rules and logins (and their permissions) during a disaster recovery drill.
These server-level objects and their configuration may not be available during the outage.
Next steps
To learn about Azure SQL Database automated backups, see SQL Database automated backups
To learn about business continuity design and recovery scenarios, see Continuity scenarios
To learn about using automated backups for recovery, see restore a database from the service-initiated
backups
Performing Disaster Recovery Drill
5/23/2018 • 2 minutes to read • Edit Online
It is recommended that validation of application readiness for recovery workflow is performed periodically.
Verifying the application behavior and implications of data loss and/or the disruption that failover involves is a
good engineering practice. It is also a requirement by most industry standards as part of business continuity
certification.
Performing a disaster recovery drill consists of:
Simulating data tier outage
Recovering
Validate application integrity post recovery
Depending on how you designed your application for business continuity, the workflow to execute the drill can
vary. This article describes the best practices for conducting a disaster recovery drill in the context of Azure SQL
Database.
Geo-restore
To prevent the potential data loss when conducting a disaster recovery drill, perform the drill using a test
environment by creating a copy of the production environment and using it to verify the application’s failover
workflow.
Outage simulation
To simulate the outage, you can rename the source database. This causes application connectivity failures.
Recovery
Perform the geo-restore of the database into a different server as described here.
Change the application configuration to connect to the recovered database and follow the Configure a database
after recovery guide to complete the recovery.
Validation
Complete the drill by verifying the application integrity post recovery (including connection strings, logins, basic
functionality testing, or other validations part of standard application signoffs procedures).
Failover groups
For a database that is protected using failover groups, the drill exercise involves planned failover to the secondary
server. The planned failover ensures that the primary and the secondary databases in the failover group remain in
sync when the roles are switched. Unlike the unplanned failover, this operation does not result in data loss, so the
drill can be performed in the production environment.
Outage simulation
To simulate the outage, you can disable the web application or virtual machine connected to the database. This
results in the connectivity failures for the web clients.
Recovery
Make sure the application configuration in the DR region points to the former secondary, which becomes the
fully accessible new primary.
Initiate planned failover of the failover group from the secondary server.
Follow the Configure a database after recovery guide to complete the recovery.
Validation
Complete the drill by verifying the application integrity post recovery (including connectivity, basic functionality
testing, or other validations required for the drill signoffs).
Next steps
To learn about business continuity scenarios, see Continuity scenarios.
To learn about Azure SQL Database automated backups, see SQL Database automated backups
To learn about using automated backups for recovery, see restore a database from the service-initiated backups.
To learn about faster recovery options, see active geo-replication and failover groups.
Designing globally available services using Azure
SQL Database
9/12/2018 • 11 minutes to read • Edit Online
When building and deploying cloud services with Azure SQL Database, you use failover groups and active geo-
replication to provide resilience to regional outages and catastrophic failures. The same feature allows you to
create globally distributed applications optimized for local access to the data. This article discusses common
application patterns, including the benefits and trade-offs of each option.
NOTE
If you are using Premium or Business Critical databases and elastic pools, you can make them resilient to regional outages
by converting them to zone redundant deployment configuration. See Zone-redundant databases.
NOTE
Azure traffic manager is used throughout this article for illustration purposes only. You can use any load-balancing solution
that supports priority routing method.
NOTE
All transactions committed after the failover are lost during the reconnection. After the failover is completed, the application
in region B is able to reconnect and restart processing the user requests. Both the web application and the primary
database are now in region B and remain co-located. n>
If an outage happens in region B, the replication process between the primary and the secondary database gets
suspended but the link between the two remains intact (1). Traffic managed detects that connectivity to Region B
is broken and marks the endpoint web app 2 as Degraded (2). The application's performance is not impacted in
this case, but the database becomes exposed and therefore at higher risk of data loss in case region A fails in
succession.
NOTE
For disaster recovery, we recommend the configuration with application deployment limited to two regions. This is because
most of the Azure geographies have only two regions. This configuration does not protect your application from a
simultaneous catastrophic failure of both regions. In an unlikely event of such a failure, you can recover your databases in a
third region using geo-restore operation.
Once the outage is mitigated, the secondary database automatically resynchronizes with the primary. During
synchronization, performance of the primary can be impacted. The specific impact depends on the amount of data
the new primary acquired since the failover. The following diagram illustrates an outage in the secondary region:
The key advantages of this design pattern are:
The same web application is deployed to both regions without any region-specific configuration and doesn’t
require additional logic to manage failover.
Application performance is not impacted by failover as the web application and the database are always co-
located.
The main tradeoff is that the application resources in Region B are underutilized most of the time.
NOTE
If the outage in the primary region is mitigated within the grace period, traffic manager detects the restoration of
connectivity in the primary region and switches user traffic back to the application instance in region A. That application
instance resumes and operates in read-write mode using the primary database in region A as illustrated by the previous
diagram.
If an outage happens in region B, the traffic manager detects the failure of the end point web-app-2 in region B
and marks it degraded (1). In the meantime, the failover group switches the read-only listener to region A (2). This
outage does not impact the end user experience but the primary database is exposed during the outage. The
following diagram illustrates a failure in the secondary region:
Once the outage is mitigated, the secondary database is immediately synchronized with the primary and the read-
only listener is switched back to the secondary database in region B. During synchronization performance of the
primary could be slightly impacted depending on the amount of data that needs to be synchronized.
This design pattern has several advantages:
It avoids data loss during the temporary outages.
Downtime depends only on how quickly traffic manager detects the connectivity failure, which is configurable.
The tradeoff is that the application must be able to operate in read-only mode.
NOTE
The failover group configuration defines which region is used for failover. Because the new primary is in a different
geography the failover results in longer latency for both OLTP and read-only workloads until the impacted region is back
online.
At the end of the day (for example at 11PM local time) the active databases should be switched to the next region
(North Europe). This task can be fully automated by using Azure scheduling service. The task involves the
following steps:
Switch primary server in the failover group to North Europe using friendly failover (1)
Remove the failover group between East US and North Europe
Create a new failover group with the same name but between North Europe and East Asia (2).
Add the primary in North Europe and secondary in East Asia to this failover group (3).
The following diagram illustrates the new configuration after the planned failover:
If an outage happens in North Europe for example, the automatic database failover is initiated by the failover
group, which effectively results in moving the application to the next region ahead of schedule (1). In that case the
US East is the only remaining secondary region until North Europe is back online. The remaining two regions
serve the customers in all three geographies by switching roles. Azure scheduler has to be adjusted accordingly.
Because the remaining regions get additional user traffic from Europe, the application's performance is impacted
not only by additional latency but also by an increased number of end user connections. Once the outage is
mitigated in North Europe, the secondary database there is immediately synchronized with the current primary.
The following diagram illustrates an outage in North Europe:
NOTE
You can reduce the time when the end user’s experience in Europe is degraded by the long latency. To do that you should
proactively deploy an application copy and create the secondary database(s) in another local region (West Europe) as a
replacement of the offline application instance in North Europe. When the latter is back online you can decide whether to
continue using West Europe or to remove the copy of the application there and switch back to using North Europe,
Active-passive deployment for disaster Read-write access < 5 sec Failure detection time + DNS TTL
recovery with co-located database
access
Active-active deployment for Read-write access < 5 sec Failure detection time + DNS TTL
application load balancing
Active-passive deployment for data Read-only access < 5 sec Read-only access = 0
preservation
Next steps
For a business continuity overview and scenarios, see Business continuity overview
To learn about geo-replication and failover groups, see active geo-replication
For information about active geo-replication with elastic pools, see Elastic pool disaster recovery strategies.
Disaster recovery strategies for applications using
SQL Database elastic pools
8/28/2018 • 13 minutes to read • Edit Online
Over the years we have learned that cloud services are not foolproof and catastrophic incidents happen. SQL
Database provides several capabilities to provide for the business continuity of your application when these
incidents occur. Elastic pools and single databases support the same kind of disaster recovery (DR ) capabilities.
This article describes several DR strategies for elastic pools that leverage these SQL Database business continuity
features.
This article uses the following canonical SaaS ISV application pattern:
A modern cloud-based web application provisions one SQL database for each end user. The ISV has many
customers and therefore uses many databases, known as tenant databases. Because the tenant databases typically
have unpredictable activity patterns, the ISV uses an elastic pool to make the database cost very predictable over
extended periods of time. The elastic pool also simplifies the performance management when the user activity
spikes. In addition to the tenant databases the application also uses several databases to manage user profiles,
security, collect usage patterns etc. Availability of the individual tenants does not impact the application’s
availability as whole. However, the availability and performance of management databases is critical for the
application’s function and if the management databases are offline the entire application is offline.
This article discusses DR strategies covering a range of scenarios from cost sensitive startup applications to ones
with stringent availability requirements.
NOTE
If you are using Premium or Business Critical databases and elastic pools, you can make them resilient to regional outages
by converting them to zone redundant deployment configuration. See Zone-redundant databases.
If the outage was temporary, it is possible that the primary region is recovered by Azure before all the database
restores are complete in the DR region. In this case, orchestrate moving the application back to the primary
region. The process takes the steps illustrated on the next diagram.
Cancel all outstanding geo-restore requests.
Fail over the management databases to the primary region (5). After the region’s recovery, the old primaries
have automatically become secondaries. Now they switch roles again.
Change the application's connection string to point back to the primary region. Now all new accounts and
tenant databases are created in the primary region. Some existing customers see their data temporarily
unavailable.
Set all databases in the DR pool to read-only to ensure they cannot be modified in the DR region (6).
For each database in the DR pool that has changed since the recovery, rename or delete the corresponding
databases in the primary pool (7).
Copy the updated databases from the DR pool to the primary pool (8).
Delete the DR pool (9)
At this point your application is online in the primary region with all tenant databases available in the primary
pool.
The key benefit of this strategy is low ongoing cost for data tier redundancy. Backups are taken automatically by
the SQL Database service with no application rewrite and at no additional cost. The cost is incurred only when the
elastic databases are restored. The trade-off is that the complete recovery of all tenant databases takes significant
time. The length of time depends on the total number of restores you initiate in the DR region and overall size of
the tenant databases. Even if you prioritize some tenants' restores over others, you are competing with all the
other restores that are initiated in the same region as the service arbitrates and throttles to minimize the overall
impact on the existing customers' databases. In addition, the recovery of the tenant databases cannot start until
the new elastic pool in the DR region is created.
As in the first scenario, the management databases are quite active so you use a single geo-replicated database for
it (1). This ensures the predictable performance for new customer subscriptions, profile updates, and other
management operations. The region in which the primaries of the management databases reside is the primary
region and the region in which the secondaries of the management databases reside is the DR region.
The paying customers’ tenant databases have active databases in the “paid” pool provisioned in the primary
region. Provision a secondary pool with the same name in the DR region. Each tenant is geo-replicated to the
secondary pool (2). This enables quick recovery of all tenant databases using failover.
If an outage occurs in the primary region, the recovery steps to bring your application online are illustrated in the
next diagram:
The key benefit of this strategy is that it provides the highest SLA for the paying customers. It also guarantees
that the new trials are unblocked as soon as the trial DR pool is created. The trade-off is that this setup increases
the total cost of the tenant databases by the cost of the secondary DR pool for paid customers. In addition, if the
secondary pool has a different size, the paying customers experience lower performance after failover until the
pool upgrade in the DR region is completed.
As in the previous scenarios, the management databases are quite active so configure them as single geo-
replicated databases (1). This ensures the predictable performance of the new customer subscriptions, profile
updates and other management operations. Region A is the primary region for the management databases and
the region B is used for recovery of the management databases.
The paying customers’ tenant databases are also geo-replicated but with primaries and secondaries split between
region A and region B (2). This way, the tenant primary databases impacted by the outage can fail over to the
other region and become available. The other half of the tenant databases are not be impacted at all.
The next diagram illustrates the recovery steps to take if an outage occurs in region A.
NOTE
The failover operation is asynchronous. To minimize the recovery time, it is important that you execute the tenant
databases' failover command in batches of at least 20 databases.
At this point your application is back online in region B. All paying customers have access to their data while the
trial customers experience delay when accessing their data.
When region A is recovered you need to decide if you want to use region B for trial customers or failback to using
the trial customers pool in region A. One criteria could be the % of trial tenant databases modified since the
recovery. Regardless of that decision, you need to re-balance the paid tenants between two pools. the next
diagram illustrates the process when the trial tenant databases fail back to region A.
Cancel all outstanding geo-restore requests to trial DR pool.
Fail over the management database (8). After the region’s recovery, the old primary automatically became the
secondary. Now it becomes the primary again.
Select which paid tenant databases fail back to pool 1 and initiate failover to their secondaries (9). After the
region’s recovery, all databases in pool 1 automatically became secondaries. Now 50% of them become
primaries again.
Reduce the size of pool 2 to the original eDTU (10) or number of vCores.
Set all restored trial databases in the region B to read-only (11).
For each database in the trial DR pool that has changed since the recovery, rename or delete the corresponding
database in the trial primary pool (12).
Copy the updated databases from the DR pool to the primary pool (13).
Delete the DR pool (14)
The key benefits of this strategy are:
It supports the most aggressive SLA for the paying customers because it ensures that an outage cannot impact
more than 50% of the tenant databases.
It guarantees that the new trials are unblocked as soon as the trail DR pool is created during the recovery.
It allows more efficient use of the pool capacity as 50% of secondary databases in pool 1 and pool 2 are
guaranteed to be less active than the primary databases.
The main trade-offs are:
The CRUD operations against the management databases have lower latency for the end users connected to
region A than for the end users connected to region B as they are executed against the primary of the
management databases.
It requires more complex design of the management database. For example, each tenant record has a location
tag that needs to be changed during failover and failback.
The paying customers may experience lower performance than usual until the pool upgrade in region B is
completed.
Summary
This article focuses on the disaster recovery strategies for the database tier used by a SaaS ISV multi-tenant
application. The strategy you choose is based on the needs of the application, such as the business model, the SLA
you want to offer to your customers, budget constraint etc. Each described strategy outlines the benefits and
trade-off so you could make an informed decision. Also, your specific application likely includes other Azure
components. So you review their business continuity guidance and orchestrate the recovery of the database tier
with them. To learn more about managing recovery of database applications in Azure, refer to Designing cloud
solutions for disaster recovery.
Next steps
To learn about Azure SQL Database automated backups, see SQL Database automated backups.
For a business continuity overview and scenarios, see Business continuity overview.
To learn about using automated backups for recovery, see restore a database from the service-initiated
backups.
To learn about faster recovery options, see active geo-replication.
To learn about using automated backups for archiving, see database copy.
Managing rolling upgrades of cloud applications
using SQL Database active geo-replication
8/23/2018 • 8 minutes to read • Edit Online
NOTE
Active geo-replication is now available for all databases in all tiers.
Learn how to use geo-replication in SQL Database to enable rolling upgrades of your cloud application. Because
upgrade is a disruptive operation, it should be part of your business continuity planning and design. In this article
we look at two different methods of orchestrating the upgrade process, and discuss the benefits and trade-offs of
each option. For the purposes of this article we will use a simple application that consists of a web site connected
to a single database as its data tier. Our goal is to upgrade version 1 of the application to version 2 without any
significant impact on the end-user experience.
When evaluating the upgrade options you should consider the following factors:
Impact on application availability during upgrades. How long the application function may be limited or
degraded.
Ability to roll back in case of an upgrade failure.
Vulnerability of the application if an unrelated catastrophic failure occurs during the upgrade.
Total dollar cost. This includes additional redundancy and incremental costs of the temporary components used
by the upgrade process.
NOTE
Note the preparation steps will not impact the application in the production slot and it can function in full access mode.
Once the preparation steps are completed the application is ready for the actual upgrade. The following diagram
illustrates the steps involved in the upgrade process.
1. Set the primary database in the production slot to read-only mode (3). This will guarantee that the production
instance of the application (V1) will remain read-only during the upgrade thus preventing the data divergence
between the V1 and V2 database instances.
2. Disconnect the secondary database using the planned termination mode (4). It will create a fully synchronized
independent copy of the primary database. This database will be upgraded.
3. Turn the primary database to read-write mode and run the upgrade script in the stage slot (5).
If the upgrade completed successfully you are now ready to switch the end users to the staged copy the
application. It will now become the production slot of the application. This involves a few more steps as illustrated
on the following diagram.
1. Switch the online endpoint in the ATM profile to contoso-2.azurewebsites.net, which points to the V2 version of
the web site (6). It now becomes the production slot with the V2 application and the end-user traffic is directed
to it.
2. If you no longer need the V1 application components so you can safely remove them (7).
If the upgrade process is unsuccessful, for example due to an error in the upgrade script, the stage slot should be
considered compromised. To roll back the application to the pre-upgrade state you simply revert the application in
the production slot to full access. The steps involved are shown on the next diagram.
1. Set the database copy to read-write mode (8). This will restore the full V1 functionally in the production slot.
2. Perform the root cause analysis and remove the compromised components in the stage slot (9).
At this point the application is fully functional and the upgrade steps can be repeated.
NOTE
The rollback does not require changes in ATM profile as it already points to contoso-1.azurewebsites.net as the active
endpoint.
The key advantage of this option is that you can upgrade an application in a single region using a set of simple
steps. The dollar cost of the upgrade is relatively low. The main tradeoff is that if a catastrophic failure occurs
during the upgrade the recovery to the pre-upgrade state will involve re-deployment of the application in a
different region and restoring the database from backup using geo-restore. This process will result in significant
downtime.
NOTE
Note the preparation steps will not impact the application in the production slot and it can function in full access mode.
Once the preparation steps are completed, the stage slot is ready for the upgrade. The following diagram illustrates
the upgrade steps.
1. Set the primary database in the production slot to read-only mode (6). This will guarantee that the production
instance of the application (V1) will remain read-only during the upgrade thus preventing the data divergence
between the V1 and V2 database instances.
2. Disconnect the secondary database in the same region using the planned termination mode (7). It will create a
fully synchronized independent copy of the primary database, which will automatically become a primary after
the termination. This database will be upgraded.
3. Turn the primary database in the stage slot to read-write mode and run the upgrade script (8).
If the upgrade completed successfully you are now ready to switch the end users to the V2 version of the
application. The following diagram illustrates the steps involved.
1. Switch the active endpoint in the ATM profile to contoso-2.azurewebsites.net, which now points to the V2
version of the web site (9). It now becomes a production slot with the V2 application and end-user traffic is
directed to it.
2. If you no longer need the V1 application so you can safely remove it (10 and 11).
If the upgrade process is unsuccessful, for example due to an error in the upgrade script, the stage slot should be
considered compromised. To roll back the application to the pre-upgrade state you simply revert to using the
application in the production slot with full access. The steps involved are shown on the next diagram.
1. Set the primary database copy in the production slot to read-write mode (12). This will restore the full V1
functionally in the production slot.
2. Perform the root cause analysis and remove the compromised components in the stage slot (13 and 14).
At this point the application is fully functional and the upgrade steps can be repeated.
NOTE
The rollback does not require changes in ATM profile as it already points to contoso-1.azurewebsites.net as the active
endpoint.
The key advantage of this option is that you can upgrade both the application and its geo-redundant copy in
parallel without compromising your business continuity during the upgrade. The main tradeoff is that it requires
double redundancy of each application component and therefore incurs higher dollar cost. It also involves a more
complicated workflow.
Summary
The two upgrade methods described in the article differ in complexity and the dollar cost but they both focus on
minimizing the time when the end user is limited to read-only operations. That time is directly defined by the
duration of the upgrade script. It does not depend on the database size, the service tier you chose, the web site
configuration and other factors that you cannot easily control. This is because all the preparation steps are
decoupled from the upgrade steps and can be done without impacting the production application. The efficiency of
the upgrade script is the key factor that determines the end-user experience during upgrades. So the best way you
can improve it is by focusing your efforts on making the upgrade script as efficient as possible.
Next steps
For a business continuity overview and scenarios, see Business continuity overview.
To learn about Azure SQL Database automated backups, see SQL Database automated backups.
To learn about using automated backups for recovery, see restore a database from automated backups.
To learn about faster recovery options, see active geo-replication.
Securing your SQL Database
9/12/2018 • 6 minutes to read • Edit Online
This article walks through the basics of securing the data tier of an application using Azure SQL Database. In
particular, this article gets you started with resources for protecting data, controlling access, and proactive
monitoring.
For a complete overview of security features available on all flavors of SQL, see the Security Center for SQL
Server Database Engine and Azure SQL Database. Additional information is also available in the Security and
Azure SQL Database technical white paper (PDF ).
Protect data
Encryption
SQL Database secures your data by providing encryption for data in motion with Transport Layer Security, for
data at rest with Transparent Data Encryption, and for data in use with Always Encrypted.
IMPORTANT
All connections to Azure SQL Database require encryption (SSL/TLS) at all times while data is "in transit" to and from the
database. In your application's connection string, you must specify parameters to encrypt the connection and not to trust
the server certificate (this is done for you if you copy your connection string out of the Azure portal), otherwise the
connection does not verify the identity of the server and is susceptible to "man-in-the-middle" attacks. For the ADO.NET
driver, for instance, these connection string parameters are Encrypt=True and TrustServerCertificate=False. For
information about TLS and connectivity, see TLS considerations
Control access
SQL Database secures your data by limiting access to your database using firewall rules, authentication
mechanisms requiring users to prove their identity, and authorization to data through role-based memberships
and permissions, as well as through row -level security and dynamic data masking. For a discussion of the use of
access control features in SQL Database, see Control access.
IMPORTANT
Managing databases and logical servers within Azure is controlled by your portal user account's role assignments. For more
information on this article, see Role-based access control in Azure portal.
Proactive monitoring
SQL Database secures your data by providing auditing and threat detection capabilities.
Auditing
SQL Database Auditing tracks database activities and helps you to maintain regulatory compliance, by recording
database events to an audit log in your Azure Storage account. Auditing enables you to understand ongoing
database activities, as well as analyze and investigate historical activity to identify potential threats or suspected
abuse and security violations. For additional information, see Get started with SQL Database Auditing.
Threat detection
Threat Detection complements auditing, by providing an additional layer of security intelligence built into the
Azure SQL Database service that detects unusual and potentially harmful attempts to access or exploit databases.
You are alerted about suspicious activities, potential vulnerabilities and SQL injection attacks, as well as
anomalous database access patterns. Threat Detection alerts can be viewed from Azure Security Center and
provide details of suspicious activity and recommend action on how to investigate and mitigate the threat. Threat
Detection costs $15/server/month. It is free for the first 60 days. For more information, see Get started with SQL
Database Threat Detection.
Compliance
In addition to the above features and functionality that can help your application meet various security
requirements, Azure SQL Database also participates in regular audits and has been certified against a number of
compliance standards. For more information, see the Microsoft Azure Trust Center, where you can find the most
current list of SQL Database compliance certifications.
Security management
SQL Database helps you manage your data security by providing database scans and a centralized security
dashboard using SQL Vulnerability Assessment.
Vulnerability Assessment: SQL Vulnerability Assessment (currently in preview ) is an easy to configure tool built
into Azure SQL Database that can help you discover, track, and remediate potential database vulnerabilities. The
assessment executes a vulnerability scan on your database, and generates a report that gives you visibility into
your security state, including actionable steps to resolve security issues and enhance your database security. The
assessment report can be customized for your environment, by setting an acceptable baseline for permission
configurations, feature configurations, and database settings. This can help you to:
Meet compliance requirements that require database scan reports.
Meet data privacy standards.
Monitor a dynamic database environment where changes are difficult to track.
For more information, see SQL Vulnerability Assessment.
Next steps
For a discussion of the use of access control features in SQL Database, see Control access.
For a discussion of database auditing, see SQL Database auditing.
For a discussion of threat detection, see SQL Database threat detection.
Advanced Threat Protection for Azure SQL Database
9/12/2018 • 3 minutes to read • Edit Online
SQL Advanced Threat Protection is a unified package for advanced SQL security capabilities. It includes
functionality for discovering and classifying sensitive data, surfacing and mitigating potential database
vulnerabilities, and detecting anomalous activities that could indicate a threat to your database. It provides a single
go-to location for enabling and managing these capabilities.
Overview
SQL Advanced Threat Protection (ATP ) provides a set of advanced SQL security capabilities, including Data
Discovery & Classification, Vulnerability Assessment, and Threat Detection.
Data Discovery & Classification (currently in preview ) provides capabilities built into Azure SQL Database for
discovering, classifying, labeling & protecting the sensitive data in your databases. It can be used to provide
visibility into your database classification state, and to track the access to sensitive data within the database and
beyond its borders.
Vulnerability Assessment is an easy to configure service that can discover, track, and help you remediate
potential database vulnerabilities. It provides visibility into your security state, and includes actionable steps to
resolve security issues, and enhance your database fortifications.
Threat Detection detects anomalous activities indicating unusual and potentially harmful attempts to access or
exploit your database. It continuously monitors your database for suspicious activities, and provides immediate
security alerts on potential vulnerabilities, SQL injection attacks, and anomalous database access patterns.
Threat Detection alerts provide details of the suspicious activity and recommend action on how to investigate
and mitigate the threat.
Enable SQL ATP once to enable all of these included features. With one click, you can enable ATP on your entire
database server, applying to all databases on the server.
ATP pricing aligns with Azure Security Center standard tier at $15/node/month, where each protected SQL
Database server is counted as one node. The first 60 days after enablement are considered a free trial period and
are not charged. For more information, see the Azure Security Center pricing page.
1. Enable ATP
Enable ATP by navigating to Advanced Threat Protection under the Security heading in your Azure SQL
Database pane. To enable ATP for all databases on the server, click Enable Advanced Threat Protection on the
server.
NOTE
The cost of ATP is $15/node/month, where a node is the entire SQL logical server. You are thus paying only once for
protecting all databases on the server with ATP. The first 60 days are considered a free trial.
Select or create a storage account for saving scan results. You can also turn on periodic recurring scans to
configure Vulnerability Assessment to run automatic scans once per week. A scan result summary is sent to the
email address(es) you provide.
3. Start classifying data, tracking vulnerabilities, and investigating threat
alerts
Click the Data Discovery and Classification card to see recommended sensitive columns to classify and to
classify your data with persistent sensitivity labels. Click the Vulnerability Assessment card to view and manage
vulnerability scans and reports, and to track your security stature. If security alerts have been received, click the
Threat Detection card to view details of the alerts and to see a consolidated report on all alerts in your Azure
subscription via the Azure Security Center security alerts page.
Advanced Threat Protection settings for your server can also be reached from the ATP database pane. Click
Settings in the main ATP pane, and then click View Advanced Threat Protection server settings.
Next steps
Learn more about Data Discovery & Classification
Learn more about Vulnerability Assessment
Learn more about Threat Detection
Learn more about Azure Security Center
Azure SQL Database Data Discovery and
Classification
9/13/2018 • 5 minutes to read • Edit Online
Data Discovery & Classification (currently in preview ) provides advanced capabilities built into Azure SQL
Database for discovering, classifying, labeling & protecting the sensitive data in your databases. Discovering
and classifying your most sensitive data (business, financial, healthcare, personally identifiable data (PII), and so
on.) can play a pivotal role in your organizational information protection stature. It can serve as infrastructure for:
Helping meet data privacy standards and regulatory compliance requirements.
Various security scenarios, such as monitoring (auditing) and alerting on anomalous access to sensitive data.
Controlling access to and hardening the security of databases containing highly sensitive data.
Data Discovery & Classification is part of the SQL Advanced Threat Protection (ATP ) offering, which is a unified
package for advanced SQL security capabilities. Data Discovery & Classification can be accessed and managed
via the central SQL ATP portal.
NOTE
This document relates to Azure SQL Database only. For SQL Server (on-prem), see SQL Data Discovery and Classification.
3. The Overview tab includes a summary of the current classification state of the database, including a
detailed list of all classified columns, which you can also filter to view only specific schema parts,
information types and labels. If you haven’t yet classified any columns, skip to step 5.
4. To download a report in Excel format, click on the Export option in the top menu of the window.
5. To begin classifying your data, click on the Classification tab at the top of the window.
6. The classification engine scans your database for columns containing potentially sensitive data and
provides a list of recommended column classifications. To view and apply classification
recommendations:
To view the list of recommended column classifications, click on the recommendations panel at the
bottom of the window:
Review the list of recommendations – to accept a recommendation for a specific column, check the
checkbox in the left column of the relevant row. You can also mark all recommendations as accepted
by checking the checkbox in the recommendations table header.
To apply the selected recommendations, click on the blue Accept selected recommendations
button.
7. You can also manually classify columns as an alternative, or in addition, to the recommendation-based
classification:
Click on Add classification in the top menu of the window.
In the context window that opens, select the schema > table > column that you want to classify, and
the information type and sensitivity label. Then click on the blue Add classification button at the
bottom of the context window.
8. To complete your classification and persistently label (tag) the database columns with the new classification
metadata, click on Save in the top menu of the window.
Automated/Programmatic classification
You can use T-SQL to add/remove column classifications, as well as retrieve all classifications for the entire
database.
NOTE
When using T-SQL to manage labels, there is no validation that labels added to a column exist in the organizational
information protection policy (the set of labels that appear in the portal recommendations). It is therefor up to you to
validate this.
Next steps
Learn more about SQL Advanced Threat Protection.
Consider configuring Azure SQL Database Auditing for monitoring and auditing access to your classified
sensitive data.
SQL Vulnerability Assessment
9/12/2018 • 5 minutes to read • Edit Online
SQL Vulnerability Assessment is an easy to configure service that can discover, track, and help you remediate
potential database vulnerabilities. Use it to proactively improve your database security.
Vulnerability Assessment is part of the SQL Advanced Threat Protection (ATP ) offering, which is a unified
package for advanced SQL security capabilities. Vulnerability Assessment can be accessed and managed via the
central SQL ATP portal.
Implementing VA
The following steps implement VA on SQL Database.
1. Run a scan
Get started with VA by navigating to Advanced Threat Protection under the Security heading in your Azure
SQL Database pane. Click to enable Advanced Threat Protection, and then click on the Vulnerability
Assessment card, which automatically opens the Vulnerability Assessment settings card.
Start by configuring a storage account where your scan results will be stored. For information about storage
accounts, see About Azure storage accounts. Once storage is configured, click Scan to scan your database for
vulnerabilities.
NOTE
The scan is lightweight and safe. It takes a few seconds to run, and is entirely read-only. It does not make any changes to
your database.
VA can now be used to monitor that your database maintains a high level of security at all times, and that your
organizational policies are met. If compliance reports are required, VA reports can be helpful to facilitate the
compliance process.
Next steps
Learn more about SQL Advanced Threat Protection
Learn more about Data Discovery & Classification
Azure SQL Database Threat Detection
9/12/2018 • 5 minutes to read • Edit Online
Azure SQL Database Threat Detection detects anomalous activities indicating unusual and potentially harmful
attempts to access or exploit databases.
Threat Detection is part of the SQL Advanced Threat Protection (ATP ) offering, which is a unified package for
advanced SQL security capabilities. Threat Detection can be accessed and managed via the central SQL ATP
portal.
2. Click a specific alert to get additional details and actions for investigating this threat and remediating
future threats.
For example, SQL injection is one of the most common Web application security issues on the Internet
that is used to attack data-driven applications. Attackers take advantage of application vulnerabilities to
inject malicious SQL statements into application entry fields, breaching or modifying data in the database.
For SQL Injection alerts, the alert’s details include the vulnerable SQL statement that was exploited.
Explore threat detection alerts for your database in the Azure portal
SQL Database Threat Detection integrates its alerts with Azure Security Center. A live SQL threat detection tiles
within the database and SQL ATP blades in the Azure portal tracks the status of active threats.
Click Threat detection alert to launch the Azure Security Center alerts page and get an overview of active SQL
threats detected on the database.
Azure SQL Database Threat Detection alerts
Threat Detection for Azure SQL Database detects anomalous activities indicating unusual and potentially
harmful attempts to access or exploit databases and it can trigger the following alerts:
Vulnerability to SQL Injection: This alert is triggered when an application generates a faulty SQL
statement in the database. This may indicate a possible vulnerability to SQL injection attacks. There are two
possible reasons for the generation of a faulty statement:
A defect in application code that constructs the faulty SQL statement
Application code or stored procedures don't sanitize user input when constructing the faulty SQL
statement, which may be exploited for SQL Injection
Potential SQL injection: This alert is triggered when an active exploit happens against an identified
application vulnerability to SQL injection. This means the attacker is trying to inject malicious SQL statements
using the vulnerable application code or stored procedures.
Access from unusual location: This alert is triggered when there is a change in the access pattern to SQL
server, where someone has logged on to the SQL server from an unusual geographical location. In some
cases, the alert detects a legitimate action (a new application or developer maintenance). In other cases, the
alert detects a malicious action (former employee, external attacker).
Access from unusual Azure data center: This alert is triggered when there is a change in the access pattern
to SQL server, where someone has logged on to the SQL server from an unusual Azure data center that was
seen on this server during the recent period. In some cases, the alert detects a legitimate action (your new
application in Azure, Power BI, Azure SQL Query Editor). In other cases, the alert detects a malicious action
from an Azure resource/service (former employee, external attacker).
Access from unfamiliar principal: This alert is triggered when there is a change in the access pattern to
SQL server, where someone has logged on to the SQL server using an unusual principal (SQL user). In some
cases, the alert detects a legitimate action (new application, developer maintenance). In other cases, the alert
detects a malicious action (former employee, external attacker).
Access from a potentially harmful application: This alert is triggered when a potentially harmful
application is used to access the database. In some cases, the alert detects penetration testing in action. In
other cases, the alert detects an attack using common attack tools.
Brute force SQL credentials: This alert is triggered when there is an abnormal high number of failed logins
with different credentials. In some cases, the alert detects penetration testing in action. In other cases, the alert
detects brute force attack.
Next steps
Learn more about SQL Advanced Threat Protection.
Learn more about Azure SQL Database Auditing
Learn more about Azure Security Center
For more information on pricing, see the SQL Database Pricing page
Azure SQL Database Managed Instance Threat
Detection
9/12/2018 • 4 minutes to read • Edit Online
SQL Threat Detection detects anomalous activities indicating unusual and potentially harmful attempts to access
or exploit databases in an Azure SQL Database Managed Instance (preview ).
Overview
Threat Detection detects anomalous database activities indicating potential security threats to Managed Instance.
Threat Detection is now in preview for Managed Instance.
Threat Detection provides a new layer of security, which enables customers to detect and respond to potential
threats as they occur by providing security alerts on anomalous database activities. Threat Detection makes it
simple to address potential threats to the Managed Instance without the need to be a security expert or manage
advanced security monitoring systems. For a full investigation experience, it is recommended to enable Azure
Managed Instance Auditing, which writes database events to an audit log in your Azure storage account.
SQL Threat Detection integrates alerts with Azure Security Center, and, each protected Managed Instance is billed
at the same price as Azure Security Center Standard tier, at $15/node/month, where each protected Managed
Instance is counted as one node.
2. Click the View recent SQL alerts link in the email to launch the Azure portal and show the Azure Security
Center alerts page, which provides an overview of active SQL threats detected on the Managed Instance’s
database.
3. Click a specific alert to get additional details and actions for investigating this threat and remediating future
threats.
For example, SQL injection is one of the common Web application security issues on the Internet. SQL
injection is used to attack data-driven applications. Attackers take advantage of application vulnerabilities to
inject malicious SQL statements into application entry fields, breaching or modifying data in the database.
For SQL Injection alerts, the alert’s details include the vulnerable SQL statement that was exploited.
Managed Instance Threat Detection alerts
Threat Detection for Managed Instance detects anomalous activities indicating unusual and potentially harmful
attempts to access or exploit databases and it can trigger the following alerts:
Vulnerability to SQL Injection: This alert is triggered when an application generates a faulty SQL statement
in the database. This may indicate a possible vulnerability to SQL injection attacks. There are two possible
reasons for the generation of a faulty statement:
A defect in application code that constructs the faulty SQL statement
Application code or stored procedures don't sanitize user input when constructing the faulty SQL
statement, which may be exploited for SQL Injection
Potential SQL injection: This alert is triggered when an active exploit happens against an identified
application vulnerability to SQL injection. It means that the attacker is trying to inject malicious SQL
statements using the vulnerable application code or stored procedures.
Access from unusual location: This alert is triggered when there is a change in the access pattern to a
Managed Instance, where someone has logged on to the Managed Instance from an unusual geographical
location. In some cases, the alert detects a legitimate action (a new application or developer’s maintenance
operation). In other cases, the alert detects a malicious action (former employee, external attacker, and so on).
Access from unusual Azure data center: This alert is triggered when there is a change in the access pattern
to Managed Instance, where someone has logged on to the Managed Instance from an Azure Data Center that
was not seen accessing this Managed Instance during the recent period. In some cases, the alert detects a
legitimate action (your new application in Azure, Power BI, Azure SQL Query Editor, and so on). In other cases,
the alert detects a malicious action from an Azure resource/service (former employee, external attacker).
Access from unfamiliar principal: This alert is triggered when there is a change in the access pattern to
Managed Instance server, where someone has logged on to the Managed Instance using an unusual principal
(SQL user). In some cases, the alert detects a legitimate action (new application developer’s maintenance
operation). In other cases, the alert detects a malicious action (former employee, external attacker).
Access from a potentially harmful application: This alert is triggered when a potentially harmful
application is used to access the database. In some cases, the alert detects penetration testing in action. In other
cases, the alert detects an attack using common attack tools.
Brute force SQL credentials: This alert is triggered when there is an abnormal high number of failed logins
with different credentials. In some cases, the alert detects penetration testing in action. In other cases, the alert
detects a brute force attack.
Next steps
Learn about Managed Instance, see What is a Managed Instance
Learn more about Managed Instance Auditing
Learn more about Azure Security Center
Azure SQL Database and SQL Data Warehouse
firewall rules
9/12/2018 • 11 minutes to read • Edit Online
Microsoft Azure SQL Database and SQL Data Warehouse provide a relational database service for Azure and
other Internet-based applications. To help protect your data, firewalls prevent all access to your database
server until you specify which computers have permission. The firewall grants access to databases based on
the originating IP address of each request.
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created
on the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data
Warehouse.
Overview
Initially, all Transact-SQL access to your Azure SQL server is blocked by the firewall. To begin using your
Azure SQL server, you must specify one or more server-level firewall rules that enable access to your Azure
SQL server. Use the firewall rules to specify which IP address ranges from the Internet are allowed, and
whether Azure applications can attempt to connect to your Azure SQL server.
To selectively grant access to just one of the databases in your Azure SQL server, you must create a database-
level rule for the required database. Specify an IP address range for the database firewall rule that is beyond
the IP address range specified in the server-level firewall rule, and ensure that the IP address of the client falls
in the range specified in the database-level rule.
IMPORTANT
SQL Data Warehouse only supports server-level firewall rules and does not support database-level firewall rules.
Connection attempts from the Internet and Azure must first pass through the firewall before they can reach
your Azure SQL server or SQL Database, as shown in the following diagram:
Server-level firewall rules: These rules enable clients to access your entire Azure SQL server, that is, all
the databases within the same logical server. These rules are stored in the master database. Server-level
firewall rules can be configured by using the portal or by using Transact-SQL statements. To create server-
level firewall rules using the Azure portal or PowerShell, you must be the subscription owner or a
subscription contributor. To create a server-level firewall rule using Transact-SQL, you must connect to the
SQL Database instance as the server-level principal login or the Azure Active Directory administrator
(which means that a server-level firewall rule must first be created by a user with Azure-level permissions).
Database-level firewall rules: These rules enable clients to access certain (secure) databases within the
same logical server. You can create these rules for each database (including the master database) and they
are stored in the individual databases. Database-level firewall rules for master and user databases can only
be created and managed by using Transact-SQL statements and only after you have configured the first
server-level firewall. If you specify an IP address range in the database-level firewall rule that is outside the
range specified in the server-level firewall rule, only those clients that have IP addresses in the database-
level range can access the database. You can have a maximum of 128 database-level firewall rules for a
database. For more information on configuring database-level firewall rules, see the example later in this
article and see sp_set_database_firewall_rule (Azure SQL Database).
Recommendation: Microsoft recommends using database-level firewall rules whenever possible to enhance
security and to make your database more portable. Use server-level firewall rules for administrators and when
you have many databases that have the same access requirements and you don't want to spend time
configuring each database individually.
IMPORTANT
Windows Azure SQL Database supports a maximum of 128 firewall rules.
NOTE
For information about portable databases in the context of business continuity, see Authentication requirements for
disaster recovery.
NOTE
To access Azure SQL Database from your local computer, ensure the firewall on your network and local computer allows
outgoing communication on TCP port 1433.
IMPORTANT
Database-level firewall rules can only be created and managed using Transact-SQL.
To improve performance, server-level firewall rules are temporarily cached at the database level. To refresh the
cache, see DBCC FLUSHAUTHCACHE.
TIP
You can use SQL Database Auditing to audit server-level and database-level firewall changes.
TIP
For a tutorial, see Create a DB using the Azure portal.
The following examples review the existing rules, enable a range of IP addresses on the server Contoso, and
deletes a firewall rule:
To delete a server-level firewall rule, execute the sp_delete_firewall_rule stored procedure. The following
example deletes the rule named ContosoFirewallRule:
az sql server firewall-rule list Server Lists the firewall rules on a server
az sql server firewall-rule show Server Shows the detail of a firewall rule
The following example sets a server-level firewall rule using the Azure CLI:
TIP
For an Azure CLI example in the context of a quick start, see Create DB - Azure CLI and Create a single database and
configure a firewall rule using the Azure CLI
Q. Is the person or team configuring or auditing the firewall rules, centrally managing firewall rules for many
(perhaps 100s) of databases?
This selection depends upon your needs and environment. Server-level firewall rules might be easier to
configure, but scripting can configure rules at the database-level. And even if you use server-level firewall
rules, you might need to audit the database-firewall rules, to see if users with CONTROL permission on the
database have created database-level firewall rules.
Q. Can I use a mix of both server-level and database-level firewall rules?
Yes. Some users, such as administrators might need server-level firewall rules. Other users, such as users of a
database application, might need database-level firewall rules.
Next steps
For a quick start on creating a database and a server-level firewall rule, see Create an Azure SQL database.
For help in connecting to an Azure SQL database from open source or third-party applications, see Client
quick-start code samples to SQL Database.
For information on additional ports that you may need to open, see the SQL Database: Outside vs inside
section of Ports beyond 1433 for ADO.NET 4.5 and SQL Database
For an overview of Azure SQL Database security, see Securing your database
Use Virtual Network service endpoints and rules for
Azure SQL Database and SQL Data Warehouse
9/12/2018 • 13 minutes to read • Edit Online
Virtual network rules are one firewall security feature that controls whether your Azure SQL Database or SQL
Data Warehouse server accepts communications that are sent from particular subnets in virtual networks. This
article explains why the virtual network rule feature is sometimes your best option for securely allowing
communication to your Azure SQL Database.
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created on
the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
To create a virtual network rule, there must first be a virtual network service endpoint for the rule to reference.
How to create a virtual network rule
If you only create a virtual network rule, you can skip ahead to the steps and explanation later in this article.
Limitations
For Azure SQL Database, the virtual network rules feature has the following limitations:
A Web App can be mapped to a private IP in a VNet/subnet. Even if service endpoints are turned ON
from the given VNet/subnet, connections from the Web App to the server will have an Azure public IP
source, not a VNet/subnet source. To enable connectivity from a Web App to a server that has VNet
firewall rules, you must Allow all Azure services on the server.
In the firewall for your SQL Database, each virtual network rule references a subnet. All these referenced
subnets must be hosted in the same geographic region that hosts the SQL Database.
Each Azure SQL Database server can have up to 128 ACL entries for any given virtual network.
Virtual network rules apply only to Azure Resource Manager virtual networks; and not to classic
deployment model networks.
Turning ON virtual network service endpoints to Azure SQL Database also enables the endpoints for the
MySQL and PostgreSQL Azure services. However, with endpoints ON, attempts to connect from the
endpoints to your MySQL or PostgreSQL instances will fail.
The underlying reason is that MySQL and PostgreSQL do not presently support ACLing.
On the firewall, IP address ranges do apply to the following networking items, but virtual network rules do
not:
Site-to-Site (S2S ) virtual private network (VPN )
On-premises via ExpressRoute
Considerations when using Service Endpoints
When using service endpoints for Azure SQL Database, review the following considerations:
Outbound to Azure SQL Database Public IPs is required: Network Security Groups (NSGs) must be
opened to Azure SQL Database IPs to allow connectivity. You can do this by using NSG Service Tags for
Azure SQL Database.
ExpressRoute
If your network is connected to the Azure network through use of ExpressRoute, each circuit is configured with
two public IP addresses at the Microsoft Edge. The two IP addresses are used to connect to Microsoft Services,
such as to Azure Storage, by using Azure Public Peering.
To allow communication from your circuit to Azure SQL Database, you must create IP network rules for the
public IP addresses of your circuits. In order to find the public IP addresses of your ExpressRoute circuit, open a
support ticket with ExpressRoute by using the Azure portal.
NOTE
If you intend to add a service endpoint to the VNet firewall rules of your Azure SQL Database server, first ensure that
service endpoints are turned On for the subnet.
If service endpoints are not turned on for the subnet, the portal asks you to enable them. Click the Enable button on the
same blade on which you add the rule.
PowerShell alternative
A PowerShell script can also create virtual network rules. The crucial cmdlet New-
AzureRmSqlServerVirtualNetworkRule. If interested, see PowerShell to create a Virtual Network service
endpoint and rule for Azure SQL Database.
REST API alternative
Internally, the PowerShell cmdlets for SQL VNet actions call REST APIs. You can call the REST APIs directly.
Virtual Network Rules: Operations
Prerequisites
You must already have a subnet that is tagged with the particular Virtual Network service endpoint type name
relevant to Azure SQL Database.
The relevant endpoint type name is Microsoft.Sql.
If your subnet might not be tagged with the type name, see Verify your subnet is an endpoint.
Azure portal steps
1. Sign in to the Azure portal.
2. Then navigate the portal to SQL servers > Firewall / Virtual Networks.
3. Set the Allow access to Azure services control to OFF.
IMPORTANT
If you leave the control set to ON, your Azure SQL Database server accepts communication from any subnet.
Leaving the control set to ON might be excessive access from a security point of view. The Microsoft Azure Virtual
Network service endpoint feature, in coordination with the virtual network rule feature of SQL Database, together
can reduce your security surface area.
5. In the new Create/Update pane, fill in the controls with the names of your Azure resources.
TIP
You must include the correct Address prefix for your subnet. You can find the value in the portal. Navigate All
resources > All types > Virtual networks. The filter displays your virtual networks. Click your virtual network,
and then click Subnets. The ADDRESS RANGE column has the Address prefix you need.
6. Click the OK button near the bottom of the pane.
7. See the resulting virtual network rule on the firewall pane.
NOTE
The following statuses or states apply to the rules:
Ready: Indicates that the operation that you initiated has Succeeded.
Failed: Indicates that the operation that you initiated has Failed.
Deleted: Only applies to the Delete operation, and indicates that the rule has been deleted and no longer applies.
InProgress: Indicates that the operation is in progress. The old rule applies while the operation is in this state.
Related articles
Azure virtual network service endpoints
Azure SQL Database server-level and database-level firewall rules
The virtual network rule feature for Azure SQL Database became available in late September 2017.
Next steps
Use PowerShell to create a virtual network service endpoint, and then a virtual network rule for Azure SQL
Database.
Virtual Network Rules: Operations with REST APIs
Use PowerShell to create a Virtual Service endpoint
and rule for Azure SQL Database and SQL Data
Warehouse
9/12/2018 • 11 minutes to read • Edit Online
Both Azure SQL Database and SQL Data Warehouse support Virtual Service endpoints.
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created on
the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
This article provides and explains a PowerShell script that takes the following actions:
1. Creates a Microsoft Azure Virtual Service endpoint on your subnet.
2. Adds the endpoint to the firewall of your Azure SQL Database server, to create a virtual network rule.
Your motivations for creating a rule are explained in: Virtual Service endpoints for Azure SQL Database.
TIP
If all you need is to assess or add the Virtual Service endpoint type name for SQL Database to your subnet, you can skip
ahead to our more direct PowerShell script.
Major cmdlets
This article emphasizes cmdlet named New-AzureRmSqlServerVirtualNetworkRule, which adds the subnet
endpoint to the access control list (ACL ) of your Azure SQL Database server, thereby creating a rule.
The following list shows the sequence of other major cmdlets that you must run to prepare for your call to New-
AzureRmSqlServerVirtualNetworkRule. In this article, these calls occur in script 3 "Virtual network rule":
1. New -AzureRmVirtualNetworkSubnetConfig: Creates a subnet object.
2. New -AzureRmVirtualNetwork: Creates your virtual network, giving it the subnet.
3. Set-AzureRmVirtualNetworkSubnetConfig: Assigns a Virtual Service endpoint to your subnet.
4. Set-AzureRmVirtualNetwork: Persists updates made to your virtual network.
5. New -AzureRmSqlServerVirtualNetworkRule: After your subnet is an endpoint, adds your subnet as a
virtual network rule, into the ACL of your Azure SQL Database server.
Offers the parameter -IgnoreMissingVnetServiceEndpoint, starting in Azure RM PowerShell
Module version 5.1.1.
Prerequisites for running PowerShell
You can already log in to Azure, such as through the Azure portal.
You can already run PowerShell scripts.
NOTE
Please ensure that service endpoints are turned on for the Vnet/Subnet that you want to add to your Server otherwise
creation of the Vnet Firewall Rule will fail.
Script 1: Variables
This first PowerShell script assigns values to variables. The subsequent scripts depend on these variables.
IMPORTANT
Before you run this script, you can edit the values, if you like. For example, if you already have a resource group, you might
want to edit your resource group name as the assigned value.
Your subscription name should be edited into the script.
$yesno = Read-Host 'Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]';
if ('yes' -eq $yesno) { Connect-AzureRmAccount; }
###########################################################
## Assignments to variables used by the later scripts. ##
###########################################################
$SubscriptionName = 'yourSubscriptionName';
Select-AzureRmSubscription -SubscriptionName $SubscriptionName;
$ResourceGroupName = 'RG-YourNameHere';
$Region = 'westcentralus';
$VNetName = 'myVNet';
$SubnetName = 'mySubnet';
$VNetAddressPrefix = '10.1.0.0/16';
$SubnetAddressPrefix = '10.1.1.0/24';
$VNetRuleName = 'myFirstVNetRule-ForAcl';
$SqlDbServerName = 'mysqldbserver-forvnet';
$SqlDbAdminLoginName = 'ServerAdmin';
$SqlDbAdminLoginPassword = 'ChangeYourAdminPassword1';
Script 2: Prerequisites
This script prepares for the next script, where the endpoint action is. This script creates for you the following listed
items, but only if they do not already exist. You can skip script 2 if you are sure these items already exist:
Azure resource group
Azure SQL Database server
PowerShell script 2 source code
$gottenResourceGroup = $null;
$gottenResourceGroup = Get-AzureRmResourceGroup `
-Name $ResourceGroupName `
-ErrorAction SilentlyContinue;
$gottenResourceGroup = New-AzureRmResourceGroup `
-Name $ResourceGroupName `
-Location $Region;
$gottenResourceGroup;
}
else { Write-Host "Good, your Resource Group already exists - $ResourceGroupName."; }
$gottenResourceGroup = $null;
###########################################################
## Ensure your Azure SQL Database server already exists. ##
###########################################################
Write-Host "Check whether your Azure SQL Database server already exists.";
$sqlDbServer = $null;
$sqlDbServer = Get-AzureRmSqlServer `
-ResourceGroupName $ResourceGroupName `
-ServerName $SqlDbServerName `
-ErrorAction SilentlyContinue;
Write-Host "Gather the credentials necessary to next create an Azure SQL Database server.";
$sqlAdministratorCredentials = New-Object `
-TypeName System.Management.Automation.PSCredential `
-ArgumentList `
$SqlDbAdminLoginName, `
$(ConvertTo-SecureString `
-String $SqlDbAdminLoginPassword `
-AsPlainText `
-Force `
);
$sqlDbServer = New-AzureRmSqlServer `
-ResourceGroupName $ResourceGroupName `
-ServerName $SqlDbServerName `
-Location $Region `
-SqlAdministratorCredentials $sqlAdministratorCredentials;
$sqlDbServer;
}
else { Write-Host "Good, your Azure SQL Database server already exists - $SqlDbServerName."; }
$sqlAdministratorCredentials = $null;
$sqlDbServer = $null;
$subnet = New-AzureRmVirtualNetworkSubnetConfig `
-Name $SubnetName `
-AddressPrefix $SubnetAddressPrefix `
-ServiceEndpoint $ServiceEndpointTypeName_SqlDb;
$vnet = New-AzureRmVirtualNetwork `
-Name $VNetName `
-AddressPrefix $VNetAddressPrefix `
-Subnet $subnet `
-ResourceGroupName $ResourceGroupName `
-Location $Region;
###########################################################
## Create a Virtual Service endpoint on the subnet. ##
###########################################################
$vnet = Set-AzureRmVirtualNetworkSubnetConfig `
-Name $SubnetName `
-AddressPrefix $SubnetAddressPrefix `
-VirtualNetwork $vnet `
-ServiceEndpoint $ServiceEndpointTypeName_SqlDb;
Write-Host "Persist the updates made to the virtual network > subnet.";
$vnet = Set-AzureRmVirtualNetwork `
-VirtualNetwork $vnet;
###########################################################
###########################################################
## Add the Virtual Service endpoint Id as a rule, ##
## into SQL Database ACLs. ##
###########################################################
$vnet = Get-AzureRmVirtualNetwork `
-ResourceGroupName $ResourceGroupName `
-Name $VNetName;
$subnet = Get-AzureRmVirtualNetworkSubnetConfig `
-Name $SubnetName `
-VirtualNetwork $vnet;
Write-Host "Add the subnet .Id as a rule, into the ACLs for your Azure SQL Database server.";
$vnetRuleObject1 = New-AzureRmSqlServerVirtualNetworkRule `
-ResourceGroupName $ResourceGroupName `
-ServerName $SqlDbServerName `
-VirtualNetworkRuleName $VNetRuleName `
-VirtualNetworkSubnetId $subnet.Id;
$vnetRuleObject1;
$vnetRuleObject2 = Get-AzureRmSqlServerVirtualNetworkRule `
-ResourceGroupName $ResourceGroupName `
-ServerName $SqlDbServerName `
-VirtualNetworkRuleName $VNetRuleName;
$vnetRuleObject2;
Script 4: Clean-up
This final script deletes the resources that the previous scripts created for the demonstration. However, the script
asks for confirmation before it deletes the following:
Azure SQL Database server
Azure Resource Group
You can run script 4 any time after script 1 completes.
PowerShell script 4 source code
######### Script 4 ########################################
## Clean-up phase A: Unconditional deletes. ##
## ##
## 1. The test rule is deleted from SQL DB ACL. ##
## 2. The test endpoint is deleted from the subnet. ##
## 3. The test virtual network is deleted. ##
###########################################################
Remove-AzureRmSqlServerVirtualNetworkRule `
-ResourceGroupName $ResourceGroupName `
-ServerName $SqlDbServerName `
-VirtualNetworkRuleName $VNetRuleName `
-ErrorAction SilentlyContinue;
$vnet = Get-AzureRmVirtualNetwork `
-ResourceGroupName $ResourceGroupName `
-Name $VNetName;
Remove-AzureRmVirtualNetworkSubnetConfig `
-Name $SubnetName `
-VirtualNetwork $vnet;
Write-Host "Delete the virtual network (thus also deletes the subnet).";
Remove-AzureRmVirtualNetwork `
-Name $VNetName `
-ResourceGroupName $ResourceGroupName `
-ErrorAction SilentlyContinue;
###########################################################
## Clean-up phase B: Conditional deletes. ##
## ##
## These might have already existed, so user might ##
## want to keep. ##
## ##
## 1. Azure SQL Database server ##
## 2. Azure resource group ##
###########################################################
$yesno = Read-Host 'CAUTION !: Do you want to DELETE your Azure SQL Database server AND your Resource Group?
[yes/no]';
if ('yes' -eq $yesno)
{
Write-Host "Remove the Azure SQL DB server.";
Remove-AzureRmSqlServer `
-ServerName $SqlDbServerName `
-ResourceGroupName $ResourceGroupName `
-ErrorAction SilentlyContinue;
Remove-AzureRmResourceGroup `
-Name $ResourceGroupName `
-ErrorAction SilentlyContinue;
}
else
{
Write-Host "Skipped over the DELETE of SQL Database and resource group.";
}
[C:\WINDOWS\system32\]
0 >> C:\Demo\PowerShell\sql-database-vnet-service-endpoint-powershell-s1-variables.ps1
Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]: yes
Environment : AzureCloud
Account : [email protected]
TenantId : 11111111-1111-1111-1111-111111111111
SubscriptionId : 22222222-2222-2222-2222-222222222222
SubscriptionName : MySubscriptionName
CurrentStorageAccount :
[C:\WINDOWS\system32\]
0 >> C:\Demo\PowerShell\sql-database-vnet-service-endpoint-powershell-s2-prerequisites.ps1
Check whether your Resource Group already exists.
Creating your missing Resource Group - RG-YourNameHere.
ResourceGroupName : RG-YourNameHere
Location : westcentralus
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/22222222-2222-2222-2222-222222222222/resourceGroups/RG-YourNameHere
ResourceGroupName : RG-YourNameHere
ServerName : mysqldbserver-forvnet
Location : westcentralus
SqlAdministratorLogin : ServerAdmin
SqlAdministratorPassword :
ServerVersion : 12.0
Tags :
Identity :
[C:\WINDOWS\system32\]
0 >> C:\Demo\PowerShell\sql-database-vnet-service-endpoint-powershell-s3-vnet-rule.ps1
Define a subnet 'mySubnet', to be given soon to a virtual network.
Create a virtual network 'myVNet'. Give the subnet to the virtual network that we created.
WARNING: The output object type of this cmdlet will be modified in a future release.
Assign a Virtual Service endpoint 'Microsoft.Sql' to the subnet.
Persist the updates made to the virtual network > subnet.
ResourceGroupName : RG-YourNameHere
ServerName : mysqldbserver-forvnet
Location : westcentralus
SqlAdministratorLogin : ServerAdmin
SqlAdministratorPassword :
ServerVersion : 12.0
Tags :
Identity :
IMPORTANT
Before you run this script, you must edit the values assigned to the $-variables, near the top of the script.
### 1. LOG into to your Azure account, needed only once per PS session. Assign variables.
$yesno = Read-Host 'Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]';
$yesno = Read-Host 'Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]';
if ('yes' -eq $yesno) { Connect-AzureRmAccount; }
$SubscriptionName = 'yourSubscriptionName';
Select-AzureRmSubscription -SubscriptionName "$SubscriptionName";
$ResourceGroupName = 'yourRGName';
$VNetName = 'yourVNetName';
$SubnetName = 'yourSubnetName';
$SubnetAddressPrefix = 'Obtain this value from the Azure portal.'; # Looks roughly like: '10.0.0.0/24'
### 2. Search for your virtual network, and then for your subnet.
$subnet = $null;
for ($nn=0; $nn -lt $vnet.Subnets.Count; $nn++)
{
$subnet = $vnet.Subnets[$nn];
if ($subnet.Name -eq $SubnetName)
{ break; }
$subnet = $null;
}
$endpointMsSql = $null;
for ($nn=0; $nn -lt $subnet.ServiceEndpoints.Count; $nn++)
{
$endpointMsSql = $subnet.ServiceEndpoints[$nn];
if ($endpointMsSql.Service -eq $ServiceEndpointTypeName_SqlDb)
{
$endpointMsSql;
break;
}
$endpointMsSql = $null;
}
### 4. Add a Virtual Service endpoint of type name 'Microsoft.Sql', on your subnet.
$vnet = Set-AzureRmVirtualNetworkSubnetConfig `
-Name $SubnetName `
-AddressPrefix $SubnetAddressPrefix `
-VirtualNetwork $vnet `
-ServiceEndpoint $ServiceEndpointTypeName_SqlDb;
Actual output
The following block displays our actual feedback (with cosmetic edits).
<# Our output example (with cosmetic edits), when the subnet was already tagged:
Do you need to log into Azure (only one time per powershell.exe session)? [yes/no]: no
Environment : AzureCloud
Account : [email protected]
TenantId : 11111111-1111-1111-1111-111111111111
SubscriptionId : 22222222-2222-2222-2222-222222222222
SubscriptionName : MySubscriptionName
CurrentStorageAccount :
ProvisioningState : Succeeded
Service : Microsoft.Sql
Locations : {westcentralus}
To provide security, Azure SQL Database and SQL Data Warehouse control access with firewall rules limiting
connectivity by IP address, authentication mechanisms requiring users to prove their identity, and authorization
mechanisms limiting users to specific actions and data.
IMPORTANT
For an overview of the SQL Database security features, see SQL security overview. For a tutorial, see Secure your Azure SQL
Database. For an overview of SQL Data Warehouse security features, see SQL Data Warehouse security overview
Authentication
SQL Database supports two types of authentication:
SQL Authentication, which uses a username and password. When you created the logical server for your
database, you specified a "server admin" login with a username and password. Using these credentials, you can
authenticate to any database on that server as the database owner, or "dbo."
Azure Active Directory Authentication, which uses identities managed by Azure Active Directory and is
supported for managed and integrated domains. Use Active Directory authentication (integrated security)
whenever possible. If you want to use Azure Active Directory Authentication, you must create another server
admin called the "Azure AD admin," which is allowed to administer Azure AD users and groups. This admin
can also perform all operations that a regular server admin can. See Connecting to SQL Database By Using
Azure Active Directory Authentication for a walkthrough of how to create an Azure AD admin to enable Azure
Active Directory Authentication.
The Database Engine closes connections that remain idle for more than 30 minutes. The connection must login
again before it can be used. Continuously active connections to SQL Database require reauthorization (performed
by the database engine) at least every 10 hours. The database engine attempts reauthorization using the originally
submitted password and no user input is required. For performance reasons, when a password is reset in SQL
Database, the connection is not reauthenticated, even if the connection is reset due to connection pooling. This is
different from the behavior of on-premises SQL Server. If the password has been changed since the connection
was initially authorized, the connection must be terminated and a new connection made using the new password.
A user with the KILL DATABASE CONNECTION permission can explicitly terminate a connection to SQL Database by
using the KILL command.
User accounts can be created in the master database and can be granted permissions in all databases on the
server, or they can be created in the database itself (called contained users). For information on creating and
managing logins, see Manage logins. Use contained databases to enhance portability and scalability. For more
information on contained users, see Contained Database Users - Making Your Database Portable, CREATE USER
(Transact-SQL ), and Contained Databases.
As a best practice your application should use a dedicated account to authenticate -- this way you can limit the
permissions granted to the application and reduce the risks of malicious activity in case your application code is
vulnerable to a SQL injection attack. The recommended approach is to create a contained database user, which
allows your app to authenticate directly to the database.
Authorization
Authorization refers to what a user can do within an Azure SQL Database, and this is controlled by your user
account's database role memberships and object-level permissions. As a best practice, you should grant users the
least privileges necessary. The server admin account you are connecting with is a member of db_owner, which has
authority to do anything within the database. Save this account for deploying schema upgrades and other
management operations. Use the "ApplicationUser" account with more limited permissions to connect from your
application to the database with the least privileges needed by your application. For more information, see
Manage logins.
Typically, only administrators need access to the master database. Routine access to each user database should be
through non-administrator contained database users created in each database. When you use contained database
users, you do not need to create logins in the master database. For more information, see Contained Database
Users - Making Your Database Portable.
You should familiarize yourself with the following features that can be used to limit or elevate permissions:
Impersonation and module-signing can be used to securely elevate permissions temporarily.
Row -Level Security can be used limit which rows a user can access.
Data Masking can be used to limit exposure of sensitive data.
Stored procedures can be used to limit the actions that can be taken on the database.
Next steps
For an overview of the SQL Database security features, see SQL security overview.
To learn more about firewall rules, see Firewall rules.
To learn about users and logins, see Manage logins.
For a discussion of proactive monitoring, see Database Auditing and SQL Database Threat Detection.
For a tutorial, see Secure your Azure SQL Database.
Controlling and granting database access to SQL
Database and SQL Data Warehouse
9/12/2018 • 11 minutes to read • Edit Online
After firewall rules configuration, you can connect to Azure SQL Database and SQL Data Warehouse as one of
the administrator accounts, as the database owner, or as a database user in the database.
NOTE
This topic applies to Azure SQL server, and to SQL Database and SQL Data Warehouse databases created on the Azure
SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
TIP
For a tutorial, see Secure your Azure SQL Database.
Server admin
When you create an Azure SQL server, you must designate a Server admin login. SQL server creates that
account as a login in the master database. This account connects using SQL Server authentication (user
name and password). Only one of these accounts can exist.
Azure Active Directory admin
One Azure Active Directory account, either an individual or security group account, can also be configured
as an administrator. It is optional to configure an Azure AD administrator, but an Azure AD administrator
must be configured if you want to use Azure AD accounts to connect to SQL Database. For more
information about configuring Azure Active Directory access, see Connecting to SQL Database or SQL
Data Warehouse By Using Azure Active Directory Authentication and SSMS support for Azure AD MFA
with SQL Database and SQL Data Warehouse.
The Server admin and Azure AD admin accounts has the following characteristics:
Are the only accounts that can automatically connect to any SQL Database on the server. (To connect to a
user database, other accounts must either be the owner of the database, or have a user account in the user
database.)
These accounts enter user databases as the dbo user and they have all the permissions in the user
databases. (The owner of a user database also enters the database as the dbo user.)
Do not enter the master database as the dbo user, and have limited permissions in master.
Are not members of the standard SQL Server sysadmin fixed server role, which is not available in SQL
database.
Can create, alter, and drop databases, logins, users in master, and server-level firewall rules.
Can add and remove members to the dbmanager and loginmanager roles.
Can view the sys.sql_logins system table.
Configuring the firewall
When the server-level firewall is configured for an individual IP address or range, the SQL server admin and
the Azure Active Directory admin can connect to the master database and all the user databases. The initial
server-level firewall can be configured through the Azure portal, using PowerShell or using the REST API.
Once a connection is made, additional server-level firewall rules can also be configured by using Transact-SQL.
Administrator access path
When the server-level firewall is properly configured, the SQL server admin and the Azure Active
Directory admin can connect using client tools such as SQL Server Management Studio or SQL Server Data
Tools. Only the latest tools provide all the features and capabilities. The following diagram shows a typical
configuration for the two administrator accounts.
When using an open port in the server-level firewall, administrators can connect to any SQL Database.
Connecting to a database by using SQL Server Management Studio
For a walk-through of creating a server, a database, server-level firewall rules, and using SQL Server
Management Studio to query a database, see Get started with Azure SQL Database servers, databases, and
firewall rules by using the Azure portal and SQL Server Management Studio.
IMPORTANT
It is recommended that you always use the latest version of Management Studio to remain synchronized with updates
to Microsoft Azure and SQL Database. Update SQL Server Management Studio.
Additional server-level administrative roles
In addition to the server-level administrative roles discussed previously, SQL Database provides two restricted
administrative roles in the master database to which user accounts can be added that grant permissions to
either create databases or manage logins.
Database creators
One of these administrative roles is the dbmanager role. Members of this role can create new databases. To
use this role, you create a user in the master database and then add the user to the dbmanager database role.
To create a database, the user must be a user based on a SQL Server login in the master database or contained
database user based on an Azure Active Directory user.
1. Using an administrator account, connect to the master database.
2. Optional step: Create a SQL Server authentication login, using the CREATE LOGIN statement. Sample
statement:
NOTE
Use a strong password when creating a login or contained database user. For more information, see Strong
Passwords.
To improve performance, logins (server-level principals) are temporarily cached at the database level. To
refresh the authentication cache, see DBCC FLUSHAUTHCACHE.
3. In the master database, create a user by using the CREATE USER statement. The user can be an Azure
Active Directory authentication contained database user (if you have configured your environment for
Azure AD authentication), or a SQL Server authentication contained database user, or a SQL Server
authentication user based on a SQL Server authentication login (created in the previous step.) Sample
statements:
CREATE USER [[email protected]] FROM EXTERNAL PROVIDER; -- To create a user with Azure Active
Directory
CREATE USER Ann WITH PASSWORD = '<strong_password>'; -- To create a SQL Database contained database
user
CREATE USER Mary FROM LOGIN Mary; -- To create a SQL Server user based on a SQL Server
authentication login
4. Add the new user, to the dbmanager database role by using the ALTER ROLE statement. Sample
statements:
NOTE
The dbmanager is a database role in master database so you can only add a database user to the dbmanager
role. You cannot add a server-level login to database-level role.
5. If necessary, configure a firewall rule to allow the new user to connect. (The new user might be covered
by an existing firewall rule.)
Now the user can connect to the master database and can create new databases. The account creating the
database becomes the owner of the database.
Login managers
The other administrative role is the login manager role. Members of this role can create new logins in the
master database. If you wish, you can complete the same steps (create a login and user, and add a user to the
loginmanager role) to enable a user to create new logins in the master. Usually logins are not necessary as
Microsoft recommends using contained database users, which authenticate at the database-level instead of
using users based on logins. For more information, see Contained Database Users - Making Your Database
Portable.
Non-administrator users
Generally, non-administrator accounts do not need access to the master database. Create contained database
users at the database level using the CREATE USER (Transact-SQL ) statement. The user can be an Azure
Active Directory authentication contained database user (if you have configured your environment for Azure
AD authentication), or a SQL Server authentication contained database user, or a SQL Server authentication
user based on a SQL Server authentication login (created in the previous step.) For more information, see
Contained Database Users - Making Your Database Portable.
To create users, connect to the database, and execute statements similar to the following examples:
Initially, only one of the administrators or the owner of the database can create users. To authorize additional
users to create new users, grant that selected user the ALTER ANY USER permission, by using a statement such
as:
To give additional users full control of the database, make them a member of the db_owner fixed database
role using the ALTER ROLE statement.
NOTE
One common reason to create a database user based on a logical server login is for users that need access to multiple
databases. Since contained database users are individual entities, each database maintains its own user and its own
password. This can cause overhead as the user must then remember each password for each database, and it can
become untenable when having to change multiple passwords for many databases. However, when using SQL Server
Logins and high availability (active geo-replication and failover groups), the SQL Server logins must be set manually at
each server. Otherwise, the database user will no longer be mapped to the server login after a failover occurs, and will
not be able to access the database post failover. For more information on configuring logins for geo-replication, please
see Configure and manage Azure SQL Database security for geo-restore or failover.
Permissions
There are over 100 permissions that can be individually granted or denied in SQL Database. Many of these
permissions are nested. For example, the UPDATE permission on a schema includes the UPDATE permission on
each table within that schema. As in most permission systems, the denial of a permission overrides a grant.
Because of the nested nature and the number of permissions, it can take careful study to design an appropriate
permission system to properly protect your database. Start with the list of permissions at Permissions
(Database Engine) and review the poster size graphic of the permissions.
Considerations and restrictions
When managing logins and users in SQL Database, consider the following:
You must be connected to the master database when executing the CREATE/ALTER/DROP DATABASE
statements.
The database user corresponding to the Server admin login cannot be altered or dropped.
US -English is the default language of the Server admin login.
Only the administrators (Server admin login or Azure AD administrator) and the members of the
dbmanager database role in the master database have permission to execute the CREATE DATABASE and
DROP DATABASE statements.
You must be connected to the master database when executing the CREATE/ALTER/DROP LOGIN statements.
However using logins is discouraged. Use contained database users instead.
To connect to a user database, you must provide the name of the database in the connection string.
Only the server-level principal login and the members of the loginmanager database role in the master
database have permission to execute the CREATE LOGIN , ALTER LOGIN , and DROP LOGIN statements.
When executing the CREATE/ALTER/DROP LOGIN and CREATE/ALTER/DROP DATABASE statements in an ADO.NET
application, using parameterized commands is not allowed. For more information, see Commands and
Parameters.
When executing the CREATE/ALTER/DROP DATABASE and CREATE/ALTER/DROP LOGIN statements, each of
these statements must be the only statement in a Transact-SQL batch. Otherwise, an error occurs. For
example, the following Transact-SQL checks whether the database exists. If it exists, a DROP DATABASE
statement is called to remove the database. Because the DROP DATABASE statement is not the only
statement in the batch, executing the following Transact-SQL statement results in an error.
When executing the CREATE USER statement with the FOR/FROM LOGIN option, it must be the only
statement in a Transact-SQL batch.
When executing the ALTER USER statement with the WITH LOGIN option, it must be the only statement in a
Transact-SQL batch.
To CREATE/ALTER/DROP a user requires the ALTER ANY USER permission on the database.
When the owner of a database role tries to add or remove another database user to or from that database
role, the following error may occur: User or role 'Name' does not exist in this database. This error
occurs because the user is not visible to the owner. To resolve this issue, grant the role owner the
VIEW DEFINITION permission on the user.
Next steps
To learn more about firewall rules, see Azure SQL Database Firewall.
For an overview of all the SQL Database security features, see SQL security overview.
For a tutorial, see Secure your Azure SQL Database.
For information about views and stored procedures, see Creating views and stored procedures
For information about granting access to a database object, see Granting Access to a Database Object
Use Azure Active Directory Authentication for
authentication with SQL Database, Managed
Instance, or SQL Data Warehouse
9/12/2018 • 8 minutes to read • Edit Online
Azure Active Directory authentication is a mechanism of connecting to Azure SQL Database and SQL Data
Warehouse by using identities in Azure Active Directory (Azure AD ).
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created
on the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data
Warehouse.
With Azure AD authentication, you can centrally manage the identities of database users and other Microsoft
services in one central location. Central ID management provides a single place to manage database users and
simplifies permission management. Benefits include the following:
It provides an alternative to SQL Server authentication.
Helps stop the proliferation of user identities across database servers.
Allows password rotation in a single place.
Customers can manage database permissions using external (Azure AD ) groups.
It can eliminate storing passwords by enabling integrated Windows authentication and other forms of
authentication supported by Azure Active Directory.
Azure AD authentication uses contained database users to authenticate identities at the database level.
Azure AD supports token-based authentication for applications connecting to SQL Database.
Azure AD authentication supports ADFS (domain federation) or native user/password authentication for a
local Azure Active Directory without domain synchronization.
Azure AD supports connections from SQL Server Management Studio that use Active Directory Universal
Authentication, which includes Multi-Factor Authentication (MFA). MFA includes strong authentication with
a range of easy verification options — phone call, text message, smart cards with pin, or mobile app
notification. For more information, see SSMS support for Azure AD MFA with SQL Database and SQL Data
Warehouse.
NOTE
Connecting to SQL Server running on an Azure VM is not supported using an Azure Active Directory account. Use a
domain Active Directory account instead.
The configuration steps include the following procedures to configure and use Azure Active Directory
authentication.
1. Create and populate Azure AD.
2. Optional: Associate or change the active directory that is currently associated with your Azure Subscription.
3. Create an Azure Active Directory administrator for the Azure SQL Database server, the Managed Instance,
or the Azure SQL Data Warehouse.
4. Configure your client computers.
5. Create contained database users in your database mapped to Azure AD identities.
6. Connect to your database by using Azure AD identities.
NOTE
To learn how to create and populate Azure AD, and then configure Azure AD with Azure SQL Database, Managed
Instance, and SQL Data Warehouse, see Configure Azure AD with Azure SQL Database.
Trust architecture
The following high-level diagram summarizes the solution architecture of using Azure AD authentication with
Azure SQL Database. The same concepts apply to SQL Data Warehouse. To support Azure AD native user
password, only the Cloud portion and Azure AD/Azure SQL Database is considered. To support Federated
authentication (or user/password for Windows credentials), the communication with ADFS block is required.
The arrows indicate communication pathways.
The following diagram indicates the federation, trust, and hosting relationships that allow a client to connect to
a database by submitting a token. The token is authenticated by an Azure AD, and is trusted by the database.
Customer 1 can represent an Azure Active Directory with native users or an Azure AD with federated users.
Customer 2 represents a possible solution including imported users; in this example coming from a federated
Azure Active Directory with ADFS being synchronized with Azure Active Directory. It's important to
understand that access to a database using Azure AD authentication requires that the hosting subscription is
associated to the Azure AD. The same subscription must be used to create the SQL Server hosting the Azure
SQL Database or SQL Data Warehouse.
Administrator structure
When using Azure AD authentication, there are two Administrator accounts for the SQL Database server and
Managed Instance; the original SQL Server administrator and the Azure AD administrator. The same concepts
apply to SQL Data Warehouse. Only the administrator based on an Azure AD account can create the first Azure
AD contained database user in a user database. The Azure AD administrator login can be an Azure AD user or
an Azure AD group. When the administrator is a group account, it can be used by any group member, enabling
multiple Azure AD administrators for the SQL Server instance. Using group account as an administrator
enhances manageability by allowing you to centrally add and remove group members in Azure AD without
changing the users or permissions in SQL Database. Only one Azure AD administrator (a user or group) can be
configured at any time.
Permissions
To create new users, you must have the ALTER ANY USER permission in the database. The ALTER ANY USER
permission can be granted to any database user. The ALTER ANY USER permission is also held by the server
administrator accounts, and database users with the CONTROL ON DATABASE or ALTER ON DATABASE permission for
that database, and by members of the db_owner database role.
To create a contained database user in Azure SQL Database, Managed Instance, or SQL Data Warehouse, you
must connect to the database or instance using an Azure AD identity. To create the first contained database user,
you must connect to the database by using an Azure AD administrator (who is the owner of the database). This
is demonstrated in Configure and manage Azure Active Directory authentication with SQL Database or SQL
Data Warehouse. Any Azure AD authentication is only possible if the Azure AD admin was created for Azure
SQL Database or SQL Data Warehouse server. If the Azure Active Directory admin was removed from the
server, existing Azure Active Directory users created previously inside SQL Server can no longer connect to the
database using their Azure Active Directory credentials.
Next steps
To learn how to create and populate Azure AD, and then configure Azure AD with Azure SQL Database or
Azure SQL Data Warehouse, see Configure and manage Azure Active Directory authentication with SQL
Database, Managed Instance, or SQL Data Warehouse.
For an overview of access and control in SQL Database, see SQL Database access and control.
For an overview of logins, users, and database roles in SQL Database, see Logins, users, and database roles.
For more information about database principals, see Principals.
For more information about database roles, see Database roles.
For more information about firewall rules in SQL Database, see SQL Database firewall rules.
Configure and manage Azure Active Directory
authentication with SQL Database, Managed
Instance, or SQL Data Warehouse
9/12/2018 • 19 minutes to read • Edit Online
This article shows you how to create and populate Azure AD, and then use Azure AD with Azure SQL Database
and SQL Data Warehouse. For an overview, see Azure Active Directory Authentication.
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created on
the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
IMPORTANT
Connecting to SQL Server running on an Azure VM is not supported using an Azure Active Directory account. Use a
domain Active Directory account instead.
NOTE
Users that are not based on an Azure AD account (including the Azure SQL server administrator account), cannot create
Azure AD-based users, because they do not have permission to validate proposed database users with the Azure AD.
Your Managed Instance needs permissions to read Azure AD to successfully accomplish tasks such as
authentication of users through security group membership or creation of new users. For this to work, you need
to grant permissions to Managed Instance to read Azure AD. There are two ways to do it: from Portal and
PowerShell. The following steps both methods.
1. In the Azure portal, in the upper-right corner, select your connection to drop down a list of possible Active
Directories.
2. Choose the correct Active Directory as the default Azure AD.
This step links the subscription associated with Active Directory with Managed Instance making sure that
the same subscription is used for both Azure AD and the Managed Instance.
3. Navigate to Managed Instance and select one that you want to use for Azure AD integration.
4. Select on banner on top of Active Directory admin page. If you are logged in as Global/Company
administrator in Azure AD, you can do it from Azure portal or using PowerShell.
If you are logged in as Global/Company administrator in Azure AD, you can do it from the Azure portal or
execute a PowerShell script.
5. After operation is successfully completed, following notification will show up in top right corner:
6. Now you can choose your Azure AD admin for your Managed Instance. For that, on the Active Directory
admin page, select Set admin command.
7. In the Add admin page, search for a user, select the user or group to be an administrator, and then select
Select.
The Active Directory admin page shows all members and groups of your Active Directory. Users or
groups that are grayed out cannot be selected because they are not supported as Azure AD
administrators. See the list of supported admins in Azure AD Features and Limitations. Role-based access
control (RBAC ) applies only to the Azure portal and is not propagated to SQL Server.
IMPORTANT
When setting up the Azure AD admin, the new admin name (user or group) cannot already be present in the virtual master
database as a SQL Server authentication user. If present, the Azure AD admin setup will fail and rolling back its creation,
indicating that such an admin (name) already exists. Since such a SQL Server authentication user is not part of the Azure
AD, any effort to connect to the server using Azure AD authentication fails.
TIP
To later remove an Admin, at the top of the Active Directory admin page, select Remove admin, and then select Save.
The following two procedures show you how to provision an Azure Active Directory administrator for your
Azure SQL server in the Azure portal and by using PowerShell.
Azure portal
1. In the Azure portal, in the upper-right corner, select your connection to drop down a list of possible Active
Directories. Choose the correct Active Directory as the default Azure AD. This step links the subscription-
associated Active Directory with Azure SQL server making sure that the same subscription is used for
both Azure AD and SQL Server. (The Azure SQL server can be hosting either Azure SQL Database or
Azure SQL Data Warehouse.)
2. In the left banner select All services, and in the filter type in SQL server. Select Sql Servers.
NOTE
On this page, before you select SQL servers, you can select the star next to the name to favorite the category and
add SQL servers to the left navigation bar.
The process of changing the administrator may take several minutes. Then the new administrator appears in the
Active Directory admin box.
NOTE
When setting up the Azure AD admin, the new admin name (user or group) cannot already be present in the virtual master
database as a SQL Server authentication user. If present, the Azure AD admin setup will fail; rolling back its creation and
indicating that such an admin (name) already exists. Since such a SQL Server authentication user is not part of the Azure
AD, any effort to connect to the server using Azure AD authentication fails.
To later remove an Admin, at the top of the Active Directory admin page, select Remove admin, and then
select Save.
PowerShell
To run PowerShell cmdlets, you need to have Azure PowerShell installed and running. For detailed information,
see How to install and configure Azure PowerShell.
To provision an Azure AD admin, execute the following Azure PowerShell commands:
Connect-AzureRmAccount
Select-AzureRmSubscription
Cmdlets used to provision and manage Azure AD admin:
Use PowerShell command get-help to see more information for each of these commands, for example
get-help Set-AzureRmSqlServerActiveDirectoryAdministrator .
The following script provisions an Azure AD administrator group named DBA_Group (object id
40b79501-b343-44ed-9ce7-da4c8cc7353f ) for the demo_server server in a resource group named Group-23:
The DisplayName input parameter accepts either the Azure AD display name or the User Principal Name. For
example, DisplayName="John Smith" and DisplayName="[email protected]" . For Azure AD groups only the Azure
AD display name is supported.
NOTE
The Azure PowerShell command Set-AzureRmSqlServerActiveDirectoryAdministrator does not prevent you from
provisioning Azure AD admins for unsupported users. An unsupported user can be provisioned, but can not connect to a
database.
NOTE
The Azure AD ObjectID is required when the DisplayName is not unique. To retrieve the ObjectID and DisplayName
values, use the Active Directory section of Azure Classic Portal, and view the properties of a user or group.
The following example returns information about the current Azure AD admin for Azure SQL server:
You can also provision an Azure Active Directory Administrator by using the REST APIs. For more information,
see Service Management REST API Reference and Operations for Azure SQL Database Operations for Azure
SQL Database
CLI
You can also provision an Azure AD admin by calling the following CLI commands:
COMMAND DESCRIPTION
az sql server ad-admin create Provisions an Azure Active Directory administrator for Azure
SQL server or Azure SQL Data Warehouse. (Must be from
the current subscription.)
az sql server ad-admin delete Removes an Azure Active Directory administrator for Azure
SQL server or Azure SQL Data Warehouse.
az sql server ad-admin list Returns information about an Azure Active Directory
administrator currently configured for the Azure SQL server
or Azure SQL Data Warehouse.
az sql server ad-admin update Updates the Active Directory administrator for an Azure SQL
server or Azure SQL Data Warehouse.
NOTE
Database users (with the exception of administrators) cannot be created using the Azure portal. RBAC roles are not
propagated to SQL Server, SQL Database, or SQL Data Warehouse. Azure RBAC roles are used for managing Azure
Resources, and do not apply to database permissions. For example, the SQL Server Contributor role does not grant
access to connect to the SQL Database or SQL Data Warehouse. The access permission must be granted directly in the
database using Transact-SQL statements.
To create an Azure AD -based contained database user (other than the server administrator that owns the
database), connect to the database with an Azure AD identity, as a user with at least the ALTER ANY USER
permission. Then use the following Transact-SQL syntax:
Azure_AD_principal_name can be the user principal name of an Azure AD user or the display name for an Azure
AD group.
Examples: To create a contained database user representing an Azure AD federated or managed domain user:
To create a contained database user representing an Azure AD or federated domain group, provide the display
name of a security group:
To create a contained database user representing an application that connects using an Azure AD token:
For more information about creating contained database users based on Azure Active Directory identities, see
CREATE USER (Transact-SQL ).
NOTE
Removing the Azure Active Directory administrator for Azure SQL server prevents any Azure AD authentication user from
connecting to the server. If necessary, unusable Azure AD users can be dropped manually by a SQL Database administrator.
NOTE
If you receive a Connection Timeout Expired, you may need to set the TransparentNetworkIPResolution parameter of
the connection string to false. For more information, see Connection timeout issue with .NET Framework 4.6.1 -
TransparentNetworkIPResolution.
When you create a database user, that user receives the CONNECT permission and can connect to that database
as a member of the PUBLIC role. Initially the only permissions available to the user are any permissions granted
to the PUBLIC role, or any permissions granted to any Azure AD groups that they are a member of. Once you
provision an Azure AD -based contained database user, you can grant the user additional permissions, the same
way as you grant permission to any other type of user. Typically grant permissions to database roles, and add
users to roles. For more information, see Database Engine Permission Basics. For more information about special
SQL Database roles, see Managing Databases and Logins in Azure SQL Database. A federated domain user
account that is imported into a managed domain as an external user, must use the managed domain identity.
NOTE
Azure AD users are marked in the database metadata with type E (EXTERNAL_USER) and for groups with type X
(EXTERNAL_GROUPS). For more information, see sys.database_principals.
IMPORTANT
Support for Azure Active Directory authentication is available with SQL Server 2016 Management Studio and SQL Server
Data Tools in Visual Studio 2015. The August 2016 release of SSMS also includes support for Active Directory Universal
Authentication, which allows administrators to require Multi-Factor Authentication using a phone call, text message, smart
cards with pin, or mobile app notification.
Using an Azure AD identity to connect using SSMS or SSDT
The following procedures show you how to connect to a SQL database with an Azure AD identity using SQL
Server Management Studio or SQL Server Database Tools.
Active Directory integrated authentication
Use this method if you are logged in to Windows using your Azure Active Directory credentials from a federated
domain.
1. Start Management Studio or Data Tools and in the Connect to Server (or Connect to Database
Engine) dialog box, in the Authentication box, select Active Directory - Integrated. No password is
needed or can be entered because your existing credentials will be presented for the connection.
2. Select the Options button, and on the Connection Properties page, in the Connect to database box,
type the name of the user database you want to connect to. (The AD domain name or tenant ID” option
is only supported for Universal with MFA connection options, otherwise it is greyed out.)
Active Directory password authentication
Use this method when connecting with an Azure AD principal name using the Azure AD managed domain. You
can also use it for federated accounts without access to the domain, for example when working remotely.
Use this method to authenticate to SQL DB/DW with Azure AD for native of federated Azure AD users. A native
user is one explicitly created in Azure AD and being authenticated using user name and password, while a
federated user is a Windows user whose domain is federated with Azure AD. The latter method (using user &
password) can be used when a user wants to use his windows credential, but his local machine is not joined with
the domain ( i.e. using a remote access). In this case a Windows user can indicate his domain account and
password and can authenticate to SQL DB/DW using federated credentials.
1. Start Management Studio or Data Tools and in the Connect to Server (or Connect to Database Engine)
dialog box, in the Authentication box, select Active Directory - Password.
2. In the User name box, type your Azure Active Directory user name in the format [email protected].
This must be an account from the Azure Active Directory or an account from a domain federate with the
Azure Active Directory.
3. In the Password box, type your user password for the Azure Active Directory account or federated
domain account.
4. Select the Options button, and on the Connection Properties page, in the Connect to database box, type
the name of the user database you want to connect to. (See the graphic in the previous option.)
string ConnectionString =
@"Data Source=n9lxnyuzhv.database.windows.net; Authentication=Active Directory Integrated; Initial
Catalog=testdb;";
SqlConnection conn = new SqlConnection(ConnectionString);
conn.Open();
The connection string keyword Integrated Security=True is not supported for connecting to Azure SQL
Database. When making an ODBC connection, you will need to remove spaces and set Authentication to
'ActiveDirectoryIntegrated'.
Active Directory password authentication
To connect to a database using integrated authentication and an Azure AD identity, the Authentication keyword
must be set to Active Directory Password. The connection string must contain User ID/UID and Password/PWD
keywords and values. The following C# code sample uses ADO .NET.
string ConnectionString =
@"Data Source=n9lxnyuzhv.database.windows.net; Authentication=Active Directory Password; Initial
Catalog=testdb; [email protected]; PWD=MyPassWord!";
SqlConnection conn = new SqlConnection(ConnectionString);
conn.Open();
Learn more about Azure AD authentication methods using the demo code samples available at Azure AD
Authentication GitHub Demo.
Azure AD token
This authentication method allows middle-tier services to connect to Azure SQL Database or Azure SQL Data
Warehouse by obtaining a token from Azure Active Directory (AAD ). It enables sophisticated scenarios including
certificate-based authentication. You must complete four basic steps to use Azure AD token authentication:
1. Register your application with Azure Active Directory and get the client id for your code.
2. Create a database user representing the application. (Completed earlier in step 6.)
3. Create a certificate on the client computer runs the application.
4. Add the certificate as a key for your application.
Sample connection string:
For more information, see SQL Server Security Blog. For information about adding a certificate, see Get started
with certificate-based authentication in Azure Active Directory.
sqlcmd
The following statements, connect using version 13.1 of sqlcmd, which is available from the Download Center.
sqlcmd -S Target_DB_or_DW.testsrv.database.windows.net -G
sqlcmd -S Target_DB_or_DW.testsrv.database.windows.net -U [email protected] -P MyAADPassword -G -l 30
Next steps
For an overview of access and control in SQL Database, see SQL Database access and control.
For an overview of logins, users, and database roles in SQL Database, see Logins, users, and database roles.
For more information about database principals, see Principals.
For more information about database roles, see Database roles.
For more information about firewall rules in SQL Database, see SQL Database firewall rules.
Conditional Access (MFA) with Azure SQL Database
and Data Warehouse
9/12/2018 • 2 minutes to read • Edit Online
Both Azure SQL Database and SQL Data Warehouse support Microsoft Conditional Access.
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created on
the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
The following steps show how to configure SQL Database to enforce a Conditional Access policy.
Prerequisites
You must configure your SQL Database or SQL Data Warehouse to support Azure Active Directory
authentication. For specific steps, see Configure and manage Azure Active Directory authentication with SQL
Database or SQL Data Warehouse.
When multi-factor authentication is enabled, you must connect with at supported tool, such as the latest SSMS.
For more information, see Configure Azure SQL Database multi-factor authentication for SQL Server
Management Studio.
Next steps
For a tutorial, see Secure your Azure SQL Database.
Universal Authentication with SQL Database and
SQL Data Warehouse (SSMS support for MFA)
9/13/2018 • 5 minutes to read • Edit Online
Azure SQL Database and Azure SQL Data Warehouse support connections from SQL Server Management
Studio (SSMS ) using Active Directory Universal Authentication. Download the latest SSMS - On the client
computer, download the latest version of SSMS, from Download SQL Server Management Studio (SSMS ). For
all the features in this article, use at least July 2017, version 17.2. The most recent connection dialog box, looks
like this:
Next steps
For configuration steps, see Configure Azure SQL Database multi-factor authentication for SQL Server
Management Studio.
Grant others access to your database: SQL Database Authentication and Authorization: Granting Access
Make sure others can connect through the firewall: Configure an Azure SQL Database server-level firewall
rule using the Azure portal
Configure and manage Azure Active Directory authentication with SQL Database or SQL Data Warehouse
Microsoft SQL Server Data-Tier Application Framework (17.0.0 GA)
SQLPackage.exe
Import a BACPAC file to a new Azure SQL Database
Export an Azure SQL database to a BACPAC file
C# interface IUniversalAuthProvider Interface
When using Active Directory- Universal with MFA authentication, ADAL tracing is available beginning
with SSMS 17.3. Off by default, you can turn on ADAL tracing by using the Tools, Options menu, under
Azure Services, Azure Cloud, ADAL Output Window Trace Level, followed by enabling Output in the
View menu. The traces are available in the output window when selecting Azure Active Directory option.
Configure multi-factor authentication for SQL Server
Management Studio and Azure AD
9/12/2018 • 3 minutes to read • Edit Online
This topic shows you how to use Azure Active Directory multi-factor authentication (MFA) with SQL Server
Management Studio. Azure AD MFA can be used when connecting SSMS or SqlPackage.exe to Azure SQL
Database and SQL Data Warehouse. For an overview of Azure SQL Database multi-factor authentication, see
Universal Authentication with SQL Database and SQL Data Warehouse (SSMS support for MFA) .
NOTE
This topic applies to Azure SQL server, and to both SQL Database and SQL Data Warehouse databases that are created on
the Azure SQL server. For simplicity, SQL Database is used when referring to both SQL Database and SQL Data Warehouse.
Configuration steps
1. Configure an Azure Active Directory - For more information, see Administering your Azure AD directory,
Integrating your on-premises identities with Azure Active Directory, Add your own domain name to Azure AD,
Microsoft Azure now supports federation with Windows Server Active Directory, and Manage Azure AD using
Windows PowerShell.
2. Configure MFA - For step-by-step instructions, see What is Azure Multi-Factor Authentication?, Conditional
Access (MFA) with Azure SQL Database and Data Warehouse. (Full conditional access requires a Premium
Azure Active Directory (Azure AD ). Limited MFA is available with a standard Azure AD.)
3. Configure SQL Database or SQL Data Warehouse for Azure AD Authentication - For step-by-step
instructions, see Connecting to SQL Database or SQL Data Warehouse By Using Azure Active Directory
Authentication.
4. Download SSMS - On the client computer, download the latest SSMS, from Download SQL Server
Management Studio (SSMS ). For all the features in this topic, use at least July 2017, version 17.2.
3. If you are connecting as a guest user, you must click Options, and on the Connection Property dialog box,
complete the AD domain name or tenant ID box. For more information, see Universal Authentication with
SQL Database and SQL Data Warehouse (SSMS support for MFA).
4. As usual for SQL Database and SQL Data Warehouse, you must click Options and specify the database on the
Options dialog box. (If the connected user is a guest user ( i.e. [email protected]), you must check the box and
add the current AD domain name or tenant ID as part of Options. See Universal Authentication with SQL
Database and SQL Data Warehouse (SSMS support for MFA). Then click Connect.
5. When the Sign in to your account dialog box appears, provide the account and password of your Azure
Active Directory identity. No password is required if a user is part of a domain federated with Azure AD.
NOTE
For Universal Authentication with an account that does not require MFA, you connect at this point. For users
requiring MFA, continue with the following steps:
6. Two MFA setup dialog boxes might appear. This one time operation depends on the MFA administrator
setting, and therefore may be optional. For an MFA enabled domain this step is sometimes pre-defined (for
example, the domain requires users to use a smartcard and pin).
7. The second possible one time dialog box allows you to select the details of your authentication method. The
possible options are configured by your administrator.
8. The Azure Active Directory sends the confirming information to you. When you receive the verification code,
enter it into the Enter verification code box, and click Sign in.
When verification is complete, SSMS connects normally presuming valid credentials and firewall access.
Next steps
For an overview of Azure SQL Database multi-factor authentication, see Universal Authentication with SQL
Database and SQL Data Warehouse (SSMS support for MFA).
Grant others access to your database: SQL Database Authentication and Authorization: Granting Access
Make sure others can connect through the firewall: Configure an Azure SQL Database server-level firewall rule
using the Azure portal
When using Active Directory- Universal with MFA authentication, ADAL tracing is available beginning with
SSMS 17.3. Off by default, you can turn on ADAL tracing by using the Tools, Options menu, under Azure
Services, Azure Cloud, ADAL Output Window Trace Level, followed by enabling Output in the View
menu. The traces are available in the output window when selecting Azure Active Directory option.
Get started with SQL database auditing
9/13/2018 • 9 minutes to read • Edit Online
Azure SQL database auditing tracks database events and writes them to an audit log in your Azure storage
account. Auditing also:
Helps you maintain regulatory compliance, understand database activity, and gain insight into
discrepancies and anomalies that could indicate business concerns or suspected security violations.
Enables and facilitates adherence to compliance standards, although it doesn't guarantee compliance. For
more information about Azure programs that support standards compliance, see the Azure Trust Center.
IMPORTANT
Audit logs are written to Append Blobs in an Azure Blob storage on your Azure subscription.
Premium Storage is currently not supported by Append Blobs.
Storage in VNet is currently not supported.
3. If you prefer to set up a server auditing policy, you can select the View server settings link on the
database auditing page. You can then view or modify the server auditing settings. Server auditing
policies apply to all existing and newly created databases on this server.
4. If you prefer to enable auditing on the database level, switch Auditing to ON.
If server auditing is enabled, the database-configured audit will exist side-by-side with the server audit.
5. New - You now have multiple options for configuring where audit logs will be written. You can write logs
to an Azure storage account, to an OMS workspace for consumption by Log Analytics, or to event hub
for consumption using event hub. You can configure any combination of these options, and audit logs
will be written to each.
6. To configure writing audit logs to a storage account, select Storage and open Storage details. Select the
Azure storage account where logs will be saved, and then select the retention period. The old logs will be
deleted. Then click OK.
7. To configure writing audit logs to an OMS workspace, select Log Analytics (Preview) and open Log
Analytics details. Select or create the OMS workspace where logs will be written and then click OK.
8. To configure writing audit logs to an event hub, select Event Hub (Preview) and open Event Hub
details. Select the event hub where logs will be written and then click OK. Be sure that the event hub is
in the same region as your database and server.
9. Click Save.
10. If you want to customize the audited events, you can do this via PowerShell cmdlets or the REST API.
11. After you've configured your auditing settings, you can turn on the new threat detection feature and
configure emails to receive security alerts. When you use threat detection, you receive proactive alerts on
anomalous database activities that can indicate potential security threats. For more information, see Getting
started with threat detection.
Audit records opens, from which you'll be able to view the logs.
You can view specific dates by clicking Filter at the top of the Audit records page.
You can switch between audit records that were created by the server audit policy and the database
audit policy by toggling Audit Source.
You can view only SQL injection related audit records by checking Show only audit records for
SQL injections checkbox.
Use the system function sys.fn_get_audit_file (T-SQL ) to return the audit log data in tabular format.
For more information on using this function, see sys.fn_get_audit_file.
Use Merge Audit Files in SQL Server Management Studio (starting with SSMS 17):
1. From the SSMS menu, select File > Open > Merge Audit Files.
2. The Add Audit Files dialog box opens. Select one of the Add options to choose whether to
merge audit files from a local disk or import them from Azure Storage. You are required to
provide your Azure Storage details and account key.
3. After all files to merge have been added, click OK to complete the merge operation.
4. The merged file opens in SSMS, where you can view and analyze it, as well as export it to an XEL
or CSV file, or to a table.
Use Power BI. You can view and analyze audit log data in Power BI. For more information and to access a
downloadable template, see Analyzie audit log data in Power BI.
Download log files from your Azure Storage blob container via the portal or by using a tool such as
Azure Storage Explorer.
After you have downloaded a log file locally, double-click the file to open, view, and analyze the logs in
SSMS.
You can also download multiple files simultaneously via Azure Storage Explorer. To do so, right-click a
specific subfolder and select Save as to save in a local folder.
Additional methods:
After downloading several files or a subfolder that contains log files, you can merge them locally as
described in the SSMS Merge Audit Files instructions described previously.
View blob auditing logs programmatically:
Use the Extended Events Reader C# library.
Query Extended Events Files by using PowerShell.
If you chose to write audit logs to Log Analytics:
To view audit logs in Log Analytics, open your Log Analytics workspace and under Search and analyze
logs, click view logs. In the Log Search view, you can start by clicking All collected data.
From here, you can use Operations Management Suite (OMS ) Log Analytics to run advanced searches
on your audit log data. Log Analytics gives you real-time operational insights using integrated search
and custom dashboards to readily analyze millions of records across all of your workloads and servers.
For additional useful information about OMS Log Analytics search language and commands, see Log
Analytics search reference.
If you chose to write audit logs to Event Hub:
To consume audit logs data from Event Hub, you will need to set up a stream to consume events and write
them to a target. For more information, see Azure Event Hubs Documentation.
Production practices
With geo-replicated databases, when you enable auditing on the primary database the secondary database will
Audi ti ng geo-repl i cated databases
have an identical auditing policy. It is also possible to set up auditing on the secondary database by enabling
auditing on the secondary server, independently from the primary database.
Server-level (recommended): Turn on auditing on both the primary server as well as the secondary
server - the primary and secondary databases will each be audited independently based on their
respective server-level policy.
Database-level: Database-level auditing for secondary databases can only be configured from Primary
database auditing settings.
Auditing must be enabled on the primary database itself, not the server.
After auditing is enabled on the primary database, it will also become enabled on the secondary
database.
IMPORTANT
With database-level auditing, the storage settings for the secondary database will be identical to those of
the primary database, causing cross-regional traffic. We recommend that you enable only server-level
auditing, and leave the database-level auditing disabled for all databases.
In production, you are likely to refresh your storage keys periodically. When writing audit logs to Azure storage,
Storage key regenerati on
you need to resave your auditing policy when refreshing your keys. The process is as follows:
1. Open Storage Details. In the Storage Access Key box, select Secondary, and click OK. Then click
Save at the top of the auditing configuration page.
2. Go to the storage configuration page and regenerate the primary access key.
3. Go back to the auditing configuration page, switch the storage access key from secondary to primary, and
then click OK. Then click Save at the top of the auditing configuration page.
4. Go back to the storage configuration page and regenerate the secondary access key (in preparation for the
next key's refresh cycle).
Additional Information
For details about the log format, hierarchy of the storage folder and naming conventions, see the Blob
Audit Log Format Reference.
IMPORTANT
Azure SQL Database Audit stores 4000 characters of data for character fields in an audit record. When the
statement or the data_sensitivity_information values returned from an auditable action contain more than
4000 characters, any data beyond the first 4000 characters will be truncated and not audited.
Audit logs are written to Append Blobs in an Azure Blob storage on your Azure subscription:
Premium Storage is currently not supported by Append Blobs.
Storage in VNet is currently not supported.
The default auditing policy includes all actions and the following set of action groups, which will audit all
the queries and stored procedures executed against the database, as well as successful and failed logins:
BATCH_COMPLETED_GROUP
SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP
FAILED_DATABASE_AUTHENTICATION_GROUP
You can configure auditing for different types of actions and action groups using PowerShell, as
described in the Manage SQL database auditing using Azure PowerShell section.
Azure SQL Database Managed Instance Auditing tracks database events and writes them to an audit log in your
Azure storage account. Auditing also:
Helps you maintain regulatory compliance, understand database activity, and gain insight into discrepancies
and anomalies that could indicate business concerns or suspected security violations.
Enables and facilitates adherence to compliance standards, although it doesn't guarantee compliance. For more
information about Azure programs that support standards compliance, see the Azure Trust Center.
IMPORTANT
Use a storage account in the same region as the Managed Instance server to avoid cross-region
reads/writes.
Provide a container Name, set Public access level to Private, and then click OK.
In the containers list, click the newly created container and then click Container properties.
Copy the container URL by clicking the copy icon and save the URL (for example, in Notepad) for
future use. The container URL format should be
https://<StorageName>.blob.core.windows.net/<ContainerName>
3. The following steps generate an Azure Storage SAS Token used to grant Managed Instance Auditing
access rights to the storage account.
Navigate to the Azure Storage account where you created the container in the previous step.
Click on Shared access signature in the Storage Settings menu.
NOTE
Renew the token upon expiry to avoid audit failures.
IMPORTANT
Remove the question mark (“?”) character from the beginning of the token.
4. Connect to your Managed Instance via SQL Server Management Studio (SSMS ).
5. Execute the following T-SQL statement to create a new Credential using the Container URL and SAS
Token that you created in the previous steps:
6. Execute the following T-SQL statement to create a new Server Audit (choose your own audit name, use the
Container URL that you created in the previous steps):
IMPORTANT
The method for viewing audit records from the Azure portal (‘Audit records’ pane) is currently unavailable for Managed
Instance.
IMPORTANT
This document applies only to Table Auditing, which is now deprecated.
Please use the new Blob Auditing method, which does not require downlevel client connection string modifications.
Additional info on Blob Auditing can be found in Get started with SQL database auditing.
Database Auditing works automatically with SQL clients that support TDS redirection. Note that redirection does
not apply when using the Blob Auditing method.
SQL Database dynamic data masking limits sensitive data exposure by masking it to non-privileged users.
Dynamic data masking helps prevent unauthorized access to sensitive data by enabling customers to designate
how much of the sensitive data to reveal with minimal impact on the application layer. It’s a policy-based security
feature that hides the sensitive data in the result set of a query over designated database fields, while the data in
the database is not changed.
For example, a service representative at a call center may identify callers by several digits of their credit card
number, but those data items should not be fully exposed to the service representative. A masking rule can be
defined that masks all but the last four digits of any credit card number in the result set of any query. As another
example, an appropriate data mask can be defined to protect personally identifiable information (PII) data, so that
a developer can query production environments for troubleshooting purposes without violating compliance
regulations.
Credit card Masking method, which exposes the last four digits of
the designated fields and adds a constant string as a prefix
in the form of a credit card.
XXXX-XXXX-XXXX-1234
Custom text Masking method, which exposes the first and last
characters and adds a custom padding string in the middle.
If the original string is shorter than the exposed prefix and
suffix, only the padding string is used.
prefix[padding]suffix
Set up dynamic data masking for your database using REST API
See Operations for Azure SQL Database.
Get started with SQL Database dynamic data
masking with the Azure portal
9/13/2018 • 2 minutes to read • Edit Online
This article shows you how to implement dynamic data masking with the Azure portal. You can also implement
dynamic data masking using Azure SQL Database cmdlets or the REST API.
Set up dynamic data masking for your database using the Azure portal
1. Launch the Azure portal at https://portal.azure.com.
2. Navigate to the settings page of the database that includes the sensitive data you want to mask.
3. Click the Dynamic Data Masking tile that launches the Dynamic Data Masking configuration page.
Alternatively, you can scroll down to the Operations section and click Dynamic Data Masking.
4. In the Dynamic Data Masking configuration page, you may see some database columns that the
recommendations engine has flagged for masking. In order to accept the recommendations, just click Add
Mask for one or more columns and a mask is created based on the default type for this column. You can
change the masking function by clicking on the masking rule and editing the masking field format to a
different format of your choice. Be sure to click Save to save your settings.
5. To add a mask for any column in your database, at the top of the Dynamic Data Masking configuration
page, click Add Mask to open the Add Masking Rule configuration page.
6. Select the Schema, Table and Column to define the designated field for masking.
7. Choose a Masking Field Format from the list of sensitive data masking categories.
8. Click Save in the data masking rule page to update the set of masking rules in the dynamic data masking policy.
9. Type the SQL users or AAD identities that should be excluded from masking, and have access to the
unmasked sensitive data. This should be a semicolon-separated list of users. Users with administrator
privileges always have access to the original unmasked data.
TIP
To make it so the application layer can display sensitive data for application privileged users, add the SQL user or AAD
identity the application uses to query the database. It is highly recommended that this list contain a minimal number
of privileged users to minimize exposure of the sensitive data.
10. Click Save in the data masking configuration page to save the new or updated masking policy.
Next steps
For an overview of dynamic data masking, see dynamic data masking.
You can also implement dynamic data masking using Azure SQL Database cmdlets or the REST API.
Always Encrypted: Protect sensitive data in SQL
Database and store your encryption keys in the
Windows certificate store
9/13/2018 • 12 minutes to read • Edit Online
This article shows you how to secure sensitive data in a SQL database with database encryption by using the
Always Encrypted Wizard in SQL Server Management Studio (SSMS ). It also shows you how to store your
encryption keys in the Windows certificate store.
Always Encrypted is a new data encryption technology in Azure SQL Database and SQL Server that helps protect
sensitive data at rest on the server, during movement between client and server, and while the data is in use,
ensuring that sensitive data never appears as plaintext inside the database system. After you encrypt data, only
client applications or app servers that have access to the keys can access plaintext data. For detailed information,
see Always Encrypted (Database Engine).
After configuring the database to use Always Encrypted, you will create a client application in C# with Visual
Studio to work with the encrypted data.
Follow the steps in this article to learn how to set up Always Encrypted for an Azure SQL database. In this article,
you will learn how to perform the following tasks:
Use the Always Encrypted wizard in SSMS to create Always Encrypted Keys.
Create a Column Master Key (CMK).
Create a Column Encryption Key (CEK).
Create a database table and encrypt columns.
Create an application that inserts, selects, and displays data from the encrypted columns.
Prerequisites
For this tutorial, you'll need:
An Azure account and subscription. If you don't have one, sign up for a free trial.
SQL Server Management Studio version 13.0.700.242 or later.
.NET Framework 4.6 or later (on the client computer).
Visual Studio.
If the New Firewall Rule window opens, sign in to Azure and let SSMS create a new firewall rule for you.
Create a table
In this section, you will create a table to hold patient data. This will be a normal table initially--you will configure
encryption in the next section.
1. Expand Databases.
2. Right-click the Clinic database and click New Query.
3. Paste the following Transact-SQL (T-SQL ) into the new query window and Execute it.
IMPORTANT
Your application must use SqlParameter objects when passing plaintext data to the server with Always Encrypted columns.
Passing literal values without using SqlParameter objects will result in an exception.
1. Open Visual Studio and create a new C# console application. Make sure your project is set to .NET
Framework 4.6 or later.
2. Name the project AlwaysEncryptedConsoleApp and click OK.
NOTE
This is the only change required in a client application specific to Always Encrypted. If you have an existing application that
stores its connection string externally (that is, in a config file), you might be able to enable Always Encrypted without
changing any code.
// Instantiate a SqlConnectionStringBuilder.
SqlConnectionStringBuilder connStringBuilder =
new SqlConnectionStringBuilder("replace with your connection string");
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data;
using System.Data.SqlClient;
namespace AlwaysEncryptedConsoleApp
{
class Program
class Program
{
// Update this line with your Clinic database connection string from the Azure portal.
static string connectionString = @"Replace with your connection string";
// Create a SqlConnectionStringBuilder.
SqlConnectionStringBuilder connStringBuilder =
new SqlConnectionStringBuilder(connectionString);
InsertPatient(new Patient() {
SSN = "999-99-0001", FirstName = "Orlando", LastName = "Gee", BirthDate =
DateTime.Parse("01/04/1964") });
InsertPatient(new Patient() {
SSN = "999-99-0002", FirstName = "Keith", LastName = "Harris", BirthDate =
DateTime.Parse("06/20/1977") });
InsertPatient(new Patient() {
SSN = "999-99-0003", FirstName = "Donna", LastName = "Carreras", BirthDate =
DateTime.Parse("02/09/1973") });
InsertPatient(new Patient() {
SSN = "999-99-0004", FirstName = "Janet", LastName = "Gates", BirthDate =
DateTime.Parse("08/31/1985") });
InsertPatient(new Patient() {
SSN = "999-99-0005", FirstName = "Lucy", LastName = "Harrington", BirthDate =
DateTime.Parse("05/06/1993") });
string ssn;
// This very simple validation only checks that the user entered 11 characters.
// In production be sure to check all user input and use the best validation for your specific
application.
do
{
Console.WriteLine("Please enter a valid SSN (ex. 123-45-6789):");
ssn = Console.ReadLine();
} while (ssn.Length != 11);
// The example allows duplicate SSN entries so we will return all records
// that match the provided value and store the results in selectedPatients.
Patient selectedPatient = SelectPatientBySSN(ssn);
// Check if any records were returned and display our query results.
if (selectedPatient != null)
{
Console.WriteLine("Patient found with SSN = " + ssn);
Console.WriteLine(selectedPatient.FirstName + " " + selectedPatient.LastName + "\tSSN: "
+ selectedPatient.SSN + "\tBirthdate: " + selectedPatient.BirthDate);
}
else
{
Console.WriteLine("No patients found with SSN = " + ssn);
}
sqlCmd.Parameters.Add(paramSSN);
sqlCmd.Parameters.Add(paramFirstName);
sqlCmd.Parameters.Add(paramLastName);
sqlCmd.Parameters.Add(paramBirthDate);
if (reader.HasRows)
{
while (reader.Read())
{
patients.Add(new Patient()
{
SSN = reader[0].ToString(),
FirstName = reader[1].ToString(),
LastName = reader["LastName"].ToString(),
BirthDate = (DateTime)reader["BirthDate"]
});
}
}
}
catch (Exception ex)
{
throw;
}
}
return patients;
}
sqlCmd.Parameters.Add(paramSSN);
if (reader.HasRows)
{
while (reader.Read())
{
patient = new Patient()
{
SSN = reader[0].ToString(),
FirstName = reader[1].ToString(),
LastName = reader["LastName"].ToString(),
BirthDate = (DateTime)reader["BirthDate"]
};
}
}
else
{
patient = null;
}
}
catch (Exception ex)
{
throw;
}
}
return patient;
}
// This method simply deletes all records in the Patients table to reset our demo.
static int ResetPatientsTable()
{
int returnValue = 0;
}
catch (Exception ex)
{
returnValue = 1;
}
}
return returnValue;
}
}
class Patient
{
public string SSN { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime BirthDate { get; set; }
}
}
}
You can see that the encrypted columns do not contain any plaintext data.
To use SSMS to access the plaintext data, you can add the Column Encryption Setting=enabled parameter to
the connection.
1. In SSMS, right-click your server in Object Explorer, and then click Disconnect.
2. Click Connect > Database Engine to open the Connect to Server window, and then click Options.
3. Click Additional Connection Parameters and type Column Encryption Setting=enabled.
You can now see the plaintext data in the encrypted columns.
NOTE
If you connect with SSMS (or any client) from a different computer, it will not have access to the encryption keys and will not
be able to decrypt the data.
Next steps
After you create a database that uses Always Encrypted, you may want to do the following:
Run this sample from a different computer. It won't have access to the encryption keys, so it will not have access
to the plaintext data and will not run successfully.
Rotate and clean up your keys.
Migrate data that is already encrypted with Always Encrypted.
Deploy Always Encrypted certificates to other client machines (see the "Making Certificates Available to
Applications and Users" section).
Related information
Always Encrypted (client development)
Transparent Data Encryption
SQL Server Encryption
Always Encrypted Wizard
Always Encrypted Blog
Always Encrypted: Protect sensitive data in SQL
Database and store your encryption keys in Azure
Key Vault
9/14/2018 • 13 minutes to read • Edit Online
This article shows you how to secure sensitive data in a SQL database with data encryption using the Always
Encrypted Wizard in SQL Server Management Studio (SSMS ). It also includes instructions that will show you how
to store each encryption key in Azure Key Vault.
Always Encrypted is a new data encryption technology in Azure SQL Database and SQL Server that helps protect
sensitive data at rest on the server, during movement between client and server, and while the data is in use.
Always Encrypted ensures that sensitive data never appears as plaintext inside the database system. After you
configure data encryption, only client applications or app servers that have access to the keys can access plaintext
data. For detailed information, see Always Encrypted (Database Engine).
After you configure the database to use Always Encrypted, you will create a client application in C# with Visual
Studio to work with the encrypted data.
Follow the steps in this article and learn how to set up Always Encrypted for an Azure SQL database. In this article
you will learn how to perform the following tasks:
Use the Always Encrypted wizard in SSMS to create Always Encrypted keys.
Create a column master key (CMK).
Create a column encryption key (CEK) .
Create a database table and encrypt columns.
Create an application that inserts, selects, and displays data from the encrypted columns.
Prerequisites
For this tutorial, you'll need:
An Azure account and subscription. If you don't have one, sign up for a free trial.
SQL Server Management Studio version 13.0.700.242 or later.
.NET Framework 4.6 or later (on the client computer).
Visual Studio.
Azure PowerShell, version 1.0 or later. Type (Get-Module azure -ListAvailable).Version to see what version
of PowerShell you are running.
Connect-AzureRmAccount
$subscriptionId = (Get-AzureRmSubscription -SubscriptionName $subscriptionName).Id
Set-AzureRmContext -SubscriptionId $subscriptionId
If the New Firewall Rule window opens, sign in to Azure and let SSMS create a new firewall rule for you.
Create a table
In this section, you will create a table to hold patient data. It's not initially encrypted--you will configure encryption
in the next section.
1. Expand Databases.
2. Right-click the Clinic database and click New Query.
3. Paste the following Transact-SQL (T-SQL ) into the new query window and Execute it.
IMPORTANT
Your application must use SqlParameter objects when passing plaintext data to the server with Always Encrypted columns.
Passing literal values without using SqlParameter objects will result in an exception.
1. Open Visual Studio and create a new C# Console Application (Visual Studio 2015 and earlier) or Console
App (.NET Framework) (Visual Studio 2017 and later). Make sure your project is set to .NET Framework 4.6
or later.
2. Name the project AlwaysEncryptedConsoleAKVApp and click OK.
3. Install the following NuGet packages by going to Tools > NuGet Package Manager > Package Manager
Console.
Run these two lines of code in the Package Manager Console.
Install-Package Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider
Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory
// Instantiate a SqlConnectionStringBuilder.
SqlConnectionStringBuilder connStringBuilder =
new SqlConnectionStringBuilder("replace with your connection string");
SqlColumnEncryptionAzureKeyVaultProvider azureKeyVaultProvider =
new SqlColumnEncryptionAzureKeyVaultProvider(GetToken);
providers.Add(SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, azureKeyVaultProvider);
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providers);
}
Always Encrypted sample console application
This sample demonstrates how to:
Modify your connection string to enable Always Encrypted.
Register Azure Key Vault as the application's key store provider.
Insert data into the encrypted columns.
Select a record by filtering for a specific value in an encrypted column.
Replace the contents of Program.cs with the following code. Replace the connection string for the global
connectionString variable in the line that directly precedes the Main method with your valid connection string
from the Azure portal. This is the only change you need to make to this code.
Run the app to see Always Encrypted in action.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data;
using System.Data.SqlClient;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider;
namespace AlwaysEncryptedConsoleAKVApp
{
class Program
{
// Update this line with your Clinic database connection string from the Azure portal.
static string connectionString = @"<connection string from the portal>";
static string applicationId = @"<application ID from your AAD application>";
static string clientKey = "<key from your AAD application>";
// Create a SqlConnectionStringBuilder.
SqlConnectionStringBuilder connStringBuilder =
new SqlConnectionStringBuilder(connectionString);
InsertPatient(new Patient()
{
SSN = "999-99-0001",
FirstName = "Orlando",
LastName = "Gee",
BirthDate = DateTime.Parse("01/04/1964")
});
InsertPatient(new Patient()
{
SSN = "999-99-0002",
FirstName = "Keith",
LastName = "Harris",
BirthDate = DateTime.Parse("06/20/1977")
});
InsertPatient(new Patient()
{
SSN = "999-99-0003",
FirstName = "Donna",
LastName = "Carreras",
BirthDate = DateTime.Parse("02/09/1973")
});
InsertPatient(new Patient()
{
SSN = "999-99-0004",
FirstName = "Janet",
LastName = "Gates",
BirthDate = DateTime.Parse("08/31/1985")
});
InsertPatient(new Patient()
{
SSN = "999-99-0005",
FirstName = "Lucy",
LastName = "Harrington",
BirthDate = DateTime.Parse("05/06/1993")
});
string ssn;
// This very simple validation only checks that the user entered 11 characters.
// In production be sure to check all user input and use the best validation for your specific
application.
do
{
Console.WriteLine("Please enter a valid SSN (ex. 999-99-0003):");
ssn = Console.ReadLine();
} while (ssn.Length != 11);
// The example allows duplicate SSN entries so we will return all records
// that match the provided value and store the results in selectedPatients.
// that match the provided value and store the results in selectedPatients.
Patient selectedPatient = SelectPatientBySSN(ssn);
// Check if any records were returned and display our query results.
if (selectedPatient != null)
{
Console.WriteLine("Patient found with SSN = " + ssn);
Console.WriteLine(selectedPatient.FirstName + " " + selectedPatient.LastName + "\tSSN: "
+ selectedPatient.SSN + "\tBirthdate: " + selectedPatient.BirthDate);
}
else
{
Console.WriteLine("No patients found with SSN = " + ssn);
}
SqlColumnEncryptionAzureKeyVaultProvider azureKeyVaultProvider =
new SqlColumnEncryptionAzureKeyVaultProvider(GetToken);
providers.Add(SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, azureKeyVaultProvider);
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providers);
}
public async static Task<string> GetToken(string authority, string resource, string scope)
{
var authContext = new AuthenticationContext(authority);
AuthenticationResult result = await authContext.AcquireTokenAsync(resource, _clientCredential);
if (result == null)
throw new InvalidOperationException("Failed to obtain the access token");
return result.AccessToken;
}
sqlCmd.Parameters.Add(paramSSN);
sqlCmd.Parameters.Add(paramFirstName);
sqlCmd.Parameters.Add(paramLastName);
sqlCmd.Parameters.Add(paramBirthDate);
if (reader.HasRows)
{
while (reader.Read())
{
patients.Add(new Patient()
{
SSN = reader[0].ToString(),
FirstName = reader[1].ToString(),
LastName = reader["LastName"].ToString(),
BirthDate = (DateTime)reader["BirthDate"]
});
}
}
}
catch (Exception ex)
{
throw;
}
}
return patients;
}
sqlCmd.Parameters.Add(paramSSN);
if (reader.HasRows)
{
while (reader.Read())
{
patient = new Patient()
{
SSN = reader[0].ToString(),
FirstName = reader[1].ToString(),
LastName = reader["LastName"].ToString(),
BirthDate = (DateTime)reader["BirthDate"]
};
}
}
else
{
patient = null;
}
}
catch (Exception ex)
{
throw;
}
}
return patient;
}
// This method simply deletes all records in the Patients table to reset our demo.
static int ResetPatientsTable()
{
int returnValue = 0;
}
catch (Exception ex)
catch (Exception ex)
{
returnValue = 1;
}
}
return returnValue;
}
}
class Patient
{
public string SSN { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime BirthDate { get; set; }
}
}
You can see that the encrypted columns do not contain any plaintext data.
To use SSMS to access the plaintext data, you first need to ensure that the user has proper permissions to the
Azure Key Vault: get, unwrapKey, and verify. For detailed information, see Create and Store Column Master Keys
(Always Encrypted).
Then add the Column Encryption Setting=enabled parameter during your connection.
1. In SSMS, right-click your server in Object Explorer and choose Disconnect.
2. Click Connect > Database Engine to open the Connect to Server window and click Options.
3. Click Additional Connection Parameters and type Column Encryption Setting=enabled.
4. Run the following query on the Clinic database.
You can now see the plaintext data in the encrypted columns.
Next steps
After you create a database that uses Always Encrypted, you may want to do the following:
Rotate and clean up your keys.
Migrate data that is already encrypted with Always Encrypted.
Related information
Always Encrypted (client development)
Transparent data encryption
SQL Server encryption
Always Encrypted wizard
Always Encrypted blog
Transparent data encryption for SQL Database and
Data Warehouse
9/13/2018 • 7 minutes to read • Edit Online
Transparent data encryption (TDE ) helps protect Azure SQL Database and Azure Data Warehouse against the
threat of malicious activity. It performs real-time encryption and decryption of the database, associated backups,
and transaction log files at rest without requiring changes to the application. By default, TDE is enabled for all
newly deployed Azure SQL databases. TDE cannot be used to encrypt the logical master database in SQL
Database. The master database contains objects that are needed to perform the TDE operations on the user
databases.
TDE will need to be manually enabled for older databases or Azure SQL Data Warehouse.
Transparent data encryption encrypts the storage of an entire database by using a symmetric key called the
database encryption key. This database encryption key is protected by the transparent data encryption protector.
The protector is either a service-managed certificate (service-managed transparent data encryption) or an
asymmetric key stored in Azure Key Vault (Bring Your Own Key). You set the transparent data encryption protector
at the server level.
On database startup, the encrypted database encryption key is decrypted and then used for decryption and re-
encryption of the database files in the SQL Server Database Engine process. Transparent data encryption performs
real-time I/O encryption and decryption of the data at the page level. Each page is decrypted when it's read into
memory and then encrypted before being written to disk. For a general description of transparent data encryption,
see Transparent data encryption.
SQL Server running on an Azure virtual machine also can use an asymmetric key from Key Vault. The
configuration steps are different from using an asymmetric key in SQL Database. For more information, see
Extensible key management by using Azure Key Vault (SQL Server).
IMPORTANT
All newly created SQL databases are encrypted by default by using service-managed transparent data encryption. Existing
databases before May 2017 and databases created through restore, geo-replication, and database copy aren't encrypted by
default.
Get-AzureRmSqlServerKeyVaultKey Gets the Key Vault keys for a SQL server instance
COMMAND DESCRIPTION
ALTER DATABASE (Azure SQL Database) SET ENCRYPTION ON/OFF encrypts or decrypts a database
You can't switch the transparent data encryption protector to a key from Key Vault by using Transact-SQL. Use
PowerShell or the Azure portal.
COMMAND DESCRIPTION
Create Or Update Server Adds an Azure Active Directory identity to a SQL Server
instance (used to grant access to Key Vault)
Create Or Update Server Key Adds a Key Vault key to a SQL Server instance
Delete Server Key Removes a Key Vault key from a SQL Server instance
COMMAND DESCRIPTION
Get Server Keys Gets a specific Key Vault key from a SQL Server instance
List Server Keys By Server Gets the Key Vault keys for a SQL Server instance
Create Or Update Encryption Protector Sets the transparent data encryption protector for a SQL
Server instance
Get Encryption Protector Gets the transparent data encryption protector for a SQL
Server instance
List Encryption Protectors By Server Gets the transparent data encryption protectors for a SQL
Server instance
Create Or Update Transparent Data Encryption Configuration Enables or disables transparent data encryption for a database
Get Transparent Data Encryption Configuration Gets the transparent data encryption configuration for a
database
List Transparent Data Encryption Configuration Results Gets the encryption result for a database
Next steps
For a general description of transparent data encryption, see Transparent data encryption.
To learn more about transparent data encryption with Bring Your Own Key support for SQL Database and Data
Warehouse, see Transparent data encryption with Bring Your Own Key support.
To start using transparent data encryption with Bring Your Own Key support, see the how -to guide Turn on
transparent data encryption by using your own key from Key Vault by using PowerShell.
For more information about Key Vault, see the Key Vault documentation page.
Transparent Data Encryption with Bring Your Own
Key support for Azure SQL Database and Data
Warehouse
9/13/2018 • 13 minutes to read • Edit Online
Bring Your Own Key (BYOK) support for Transparent Data Encryption (TDE ) allows you to encrypt the Database
Encryption Key (DEK) with an asymmetric key called TDE Protector. The TDE Protector is stored under your
control in Azure Key Vault, Azure’s cloud-based external key management system. Azure Key Vault is the first key
management service with which TDE has integrated support for BYOK. The TDE DEK, which is stored on the boot
page of a database is encrypted and decrypted by the TDE protector. The TDE Protector is stored in Azure Key
Vault and never leaves the key vault. If the server's access to the key vault is revoked, a database cannot be
decrypted and read into memory. The TDE protector is set at the logical server level and is inherited by all
databases associated with that server.
With BYOK support, users can now control key management tasks including key rotations, key vault permissions,
deleting keys, and enable auditing/reporting on all TDE protectors using Azure Key Vault functionality. Key Vault
provides central key management, leverages tightly monitored hardware security modules (HSMs), and enables
separation of duties between management of keys and data to help meet regulatory compliance.
TDE with BYOK provides the following benefits:
Increased transparency and granular control with the ability to self-manage the TDE protector
Central management of TDE protectors (along with other keys and secrets used in other Azure services) by
hosting them in Key Vault
Separation of key and data management responsibilities within the organization, to support separation of
duties
Greater trust from your own clients, since Key Vault is designed so that Microsoft does not see or extract any
encryption keys.
Support for key rotation
IMPORTANT
For those using service-managed TDE who would like to start using Key Vault, TDE remains enabled during the process of
switching over to a TDE protector in Key Vault. There is no downtime nor re-encryption of the database files. Switching from
a service-managed key to a Key Vault key only requires re-encryption of the database encryption key (DEK), which is a fast
and online operation.
IMPORTANT
It is important to note that once a TDE Protector is stored in Azure Key Vault, it never leaves Azure Key Vault. The
logical server can only send key operation requests to the TDE protector key material within Key Vault, and never accesses
or caches the TDE protector. The Key Vault administrator has the right to revoke Key Vault permissions of the server at
any point, in which case all connections to the server are cut off.
NOTE
If the Azure AD Identity is accidentally deleted or the server’s permissions are revoked using the key vault’s
access policy, the server loses access to the key vault, and TDE encrypted databases are dropped within 24 hours.
Configure Azure Key Vault without a VNET or firewall. If SQL loses access to the key vault, TDE encrypted
databases are dropped within 24 hours.
Enable auditing and reporting on all encryption keys: Key Vault provides logs that are easy to inject into other
security information and event management (SIEM ) tools. Operations Management Suite (OMS ) Log
Analytics is one example of a service that is already integrated.
To ensure high-availability of encrypted databases, configure each logical server with two Azure Key Vaults that
reside in different regions.
Guidelines for configuring the TDE Protector (asymmetric key) stored in Azure Key Vault
Create your encryption key locally on a local HSM device. Ensure this is an asymmetric, RSA 2048 key so it is
storable in Azure Key Vault.
Escrow the key in a key escrow system.
Import the encryption key file (.pfx, .byok, or .backup) to Azure Key Vault.
NOTE
For testing purposes, it is possible to create a key with Azure Key Vault, however this key cannot be escrowed, because the
private key can never leave the key vault. Always back up and escrow keys used to encrypt production data, as the loss of
the key (accidental deletion in key vault, expiration etc.) results in permanent data loss.
Use a key without an expiration date – and never set an expiration date on a key already in use: once the key
expires, the encrypted databases lose access to their TDE Protector and are dropped within 24 hours.
Ensure the key is enabled and has permissions to perform get, wrap key, and unwrap key operations.
Create an Azure Key Vault key backup before using the key in Azure Key Vault for the first time. Learn more
about the Backup-AzureKeyVaultKey command.
Create a new backup whenever any changes are made to the key (for example, add ACLs, add tags, add key
attributes).
Keep previous versions of the key in the key vault when rotating keys, so older database backups can be
restored. When the TDE Protector is changed for a database, old backups of the database are not updated to
use the latest TDE Protector. Each backup needs the TDE Protector it was created with at restore time. Key
rotations can be performed following the instructions at Rotate the Transparent Data Encryption Protector
Using PowerShell.
Keep all previously used keys in Azure Key Vault after changing back to service-managed keys. This ensures
database backups can be restored with the TDE protectors stored in Azure Key Vault. TDE protectors created
with Azure Key Vault have to be maintained until all stored backups have been created with service-managed
keys.
Make recoverable backup copies of these keys using Backup-AzureKeyVaultKey.
To remove a potentially compromised key during a security incident without the risk of data loss, follow the
steps at Remove a potentially compromised key.
NOTE
When assigning the key vault to the server, it is important to start with the secondary server. In the second step assign the
key vault to the primary server and update the TDE Protector, the Geo-DR link will continue to work because at this point
the TDE Protector used by the replicated database is available to both servers.
Before enabling TDE with customer managed keys in Azure Key Vault for a SQL Database Geo-DR scenario, it is
important to create and maintain two Azure Key Vaults with identical contents in the same regions that will be
used for SQL Database geo-replication. “Identical contents” specifically means that both key vaults must contain
copies of the same TDE Protector(s) so that both servers have access to the TDE Protectors use by all databases.
Going forward, it is required to keep both key vaults in sync, which means they must contain the same copies of
TDE Protectors after key rotation, maintain old versions of keys used for log files or backups, TDE Protectors must
maintain the same key properties and the key vaults must maintain the same access permissions for SQL.
Follow the steps in Active geo-replication overview to test and trigger a failover, which should be done on a
regular basis to confirm the access permissions for SQL to both key vaults have been maintained.
Backup and Restore
Once a database is encrypted with TDE using a key from Key Vault, any generated backups are also encrypted
with the same TDE Protector.
To restore a backup encrypted with a TDE Protector from Key Vault, make sure that the key material is still in the
original vault under the original key name. When the TDE Protector is changed for a database, old backups of the
database are not updated to use the latest TDE Protector. Therefore, we recommend that you keep all old
versions of the TDE Protector in Key Vault, so database backups can be restored.
If a key that might be needed for restoring a backup is no longer in its original key vault, the following error
message is returned: "Target server does not have access to all AKV Uris created between <Timestamp #1> and
<Timestamp #2>. Please retry operation after restoring all AKV Uris."
To mitigate this, run the Get-AzureRmSqlServerKeyVaultKey cmdlet to return the list of keys from Key Vault that
were added to the server (unless they were deleted by a user). To ensure all backups can be restored, make sure
the target server for the backup has access to all of these keys.
Get-AzureRmSqlServerKeyVaultKey `
-ServerName <LogicalServerName> `
-ResourceGroup <SQLDatabaseResourceGroupName>
To learn more about backup recovery for SQL Database, see Recover an Azure SQL database. To learn more
about backup recovery for SQL Data Warehouse, see Recover an Azure SQL Data Warehouse.
Additional consideration for backed up log files: Backed up log files remain encrypted with the original TDE
Encryptor, even if the TDE Protector was rotated and the database is now using a new TDE Protector. At restore
time, both keys will be needed to restore the database. If the log file is using a TDE Protector stored in Azure Key
Vault, this key will be needed at restore time, even if the database has been changed to use service-managed TDE
in the meantime.
PowerShell and CLI: Enable Transparent Data
Encryption using your own key from Azure Key Vault
9/13/2018 • 5 minutes to read • Edit Online
This article walks through how to use a key from Azure Key Vault for Transparent Data Encryption (TDE ) on a SQL
Database or Data Warehouse. To learn more about the TDE with Bring Your Own Key (BYOK) Support, visit TDE
Bring Your Own Key to Azure SQL.
$server = Set-AzureRmSqlServer `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-AssignIdentity
If you are creating a server, use the New -AzureRmSqlServer cmdlet with the tag -Identity to add an Azure AD
identity during server creation:
$server = New-AzureRmSqlServer `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-Location <RegionName> `
-ServerName <LogicalServerName> `
-ServerVersion "12.0" `
-SqlAdministratorCredentials <PSCredential> `
-AssignIdentity
Set-AzureRmKeyVaultAccessPolicy `
-VaultName <KeyVaultName> `
-ObjectId $server.Identity.PrincipalId `
-PermissionsToKeys get, wrapKey, unwrapKey
Step 3. Add the Key Vault key to the server and set the TDE Protector
Use the Add-AzureRmSqlServerKeyVaultKey cmdlet to add the key from the Key Vault to the server.
Use the Set-AzureRmSqlServerTransparentDataEncryptionProtector cmdlet to set the key as the TDE
protector for all server resources.
Use the Get-AzureRmSqlServerTransparentDataEncryptionProtector cmdlet to confirm that the TDE protector
was configured as intended.
NOTE
The combined length for the key vault name and key name cannot exceed 94 characters.
TIP
An example KeyId from Key Vault: https://contosokeyvault.vault.azure.net/keys/Key1/1a1a2b2b3c3c4d4d5e5e6f6f7g7g8h8h
<# Add the key from Key Vault to the server #>
Add-AzureRmSqlServerKeyVaultKey `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-KeyId <KeyVaultKeyId>
<# Set the key as the TDE protector for all resources under the server #>
Set-AzureRmSqlServerTransparentDataEncryptionProtector `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-Type AzureKeyVault `
-KeyId <KeyVaultKeyId>
<# To confirm that the TDE protector was configured as intended: #>
Get-AzureRmSqlServerTransparentDataEncryptionProtector `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName>
Set-AzureRMSqlDatabaseTransparentDataEncryption `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-DatabaseName <DatabaseName> `
-State "Enabled"
Now the database or data warehouse has TDE enabled with an encryption key in Key Vault.
<# Check the encryption progress for a database or data warehouse #>
Get-AzureRMSqlDatabaseTransparentDataEncryptionActivity `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-DatabaseName <DatabaseName>
Set-AzureRMSqlDatabaseTransparentDataEncryption `
-ServerName <LogicalServerName> `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-DatabaseName <DatabaseName> `
-State "Disabled”
Use the Get-AzureRmSqlServerKeyVaultKey cmdlet to return the list of Key Vault keys added to the server.
Use the Remove-AzureRmSqlServerKeyVaultKey to remove a Key Vault key from the server.
<# The key set as the TDE Protector cannot be removed. #>
Remove-AzureRmSqlServerKeyVaultKey `
-KeyId <KeyVaultKeyId> `
-ServerName <LogicalServerName> `
-ResourceGroupName <SQLDatabaseResourceGroupName>
Troubleshooting
Check the following if an issue occurs:
If the key vault cannot be found, make sure you're in the right subscription using the Get-
AzureRmSubscription cmdlet.
Get-AzureRmSubscription `
-SubscriptionId <SubscriptionId>
If the new key cannot be added to the server, or the new key cannot be updated as the TDE Protector, check
the following:
The key should not have an expiration date
The key must have the get, wrap key, and unwrap key operations enabled.
Next steps
Learn how to rotate the TDE Protector of a server to comply with security requirements: Rotate the
Transparent Data Encryption protector Using PowerShell.
In case of a security risk, learn how to remove a potentially compromised TDE Protector: Remove a potentially
compromised key.
Step 3. Add the Key Vault key to the server and set the TDE Protector
cli
# add server key and update encryption protector
az sql server key create -g "ResourceGroupName" -s "ServerName" -t "AzureKeyVault" -u "FullVersionedKeyUri
az sql server tde-key update -g "ResourceGroupName" -s "ServerName" -t AzureKeyVault -u
"FullVersionedKeyUri"
NOTE
The combined length for the key vault name and key name cannot exceed 94 characters.
TIP
An example KeyId from Key Vault: https://contosokeyvault.vault.azure.net/keys/Key1/1a1a2b2b3c3c4d4d5e5e6f6f7g7g8h8h
Now the database or data warehouse has TDE enabled with an encryption key in Key Vault.
This article describes key rotation for an Azure SQL server using a TDE protector from Azure Key Vault. Rotating
an Azure SQL server’s TDE protector means switching to a new asymmetric key that protects the databases on
the server. Key rotation is an online operation and should only take a few seconds to complete, because this only
decrypts and re-encrypts the database’s data encryption key, not the entire database.
This guide discusses two options to rotate the TDE protector on the server.
NOTE
A paused SQL Data Warehouse must be resumed before key rotations.
IMPORTANT
Do Not Delete previous versions of the key after a rollover. When keys are rolled over, some data is still encrypted with the
previous keys, such as older database backups.
Prerequisites
This how -to guide assumes that you are already using a key from Azure Key Vault as the TDE protector for an
Azure SQL Database or Data Warehouse. See Transparent Data Encryption with BYOK Support.
You must have Azure PowerShell version 3.7.0 or newer installed and running.
[Recommended but optional] Create the key material for the TDE protector in a hardware security module
(HSM ) or local key store first, and import the key material to Azure Key Vault. Follow the instructions for using
a hardware security module (HSM ) and Key Vault to learn more.
Add-AzureKeyVaultKey `
-VaultName <KeyVaultName> `
-Name <KeyVaultKeyName> `
-Destination <HardwareOrSoftware>
<# Set the key as the TDE protector for all resources
under the server #>
Set-AzureRmSqlServerTransparentDataEncryptionProtector `
-Type AzureKeyVault `
-KeyId <KeyVaultKeyId> `
-ServerName <LogicalServerName> `
-ResourceGroup <SQLDatabaseResourceGroupName>
Set-AzureRmSqlServerTransparentDataEncryptionProtector `
-Type AzureKeyVault `
-KeyId <KeyVaultKeyId> `
-ServerName <LogicalServerName> `
-ResourceGroup <SQLDatabaseResourceGroupName>
To switch the TDE protector from BYOK mode to Microsoft-managed, use the Set-
AzureRmSqlServerTransparentDataEncryptionProtector cmdlet.
Set-AzureRmSqlServerTransparentDataEncryptionProtector `
-Type ServiceManaged `
-ServerName <LogicalServerName> `
-ResourceGroup <SQLDatabaseResourceGroupName>
Next steps
In case of a security risk, learn how to remove a potentially compromised TDE protector: Remove a
potentially compromised key
Get started with Bring Your Own Key support for TDE: Turn on TDE using your own key from Key Vault
using PowerShell
Remove a Transparent Data Encryption (TDE)
protector using PowerShell
9/13/2018 • 3 minutes to read • Edit Online
Prerequisites
You must have an Azure subscription and be an administrator on that subscription
You must have Azure PowerShell version 4.2.0 or newer installed and running.
This how -to guide assumes that you are already using a key from Azure Key Vault as the TDE protector for an
Azure SQL Database or Data Warehouse. See Transparent Data Encryption with BYOK Support to learn more.
Overview
This how -to guide describes how to respond to a potentially compromised TDE protector for an Azure SQL
Database or Data Warehouse that is using TDE with Bring Your Own Key (BYOK) support. To learn more about
BYOK support for TDE, see the overview page.
The following procedures should only be done in extreme cases or in test environments. Review the how -to guide
carefully, as deleting actively used TDE protectors from Azure Key Vault can result in data loss.
If a key is ever suspected to be compromised, such that a service or user had unauthorized access to the key, it’s
best to delete the key.
Keep in mind that once the TDE protector is deleted in Key Vault, all connections to the encrypted databases
under the server are blocked, and these databases go offline and get dropped within 24 hours. Old
backups encrypted with the compromised key are no longer accessible.
This how -to guide goes over two approaches depending on the desired result after the incident response:
To keep the Azure SQL databases / Data Warehouses accessible
To make the Azure SQL databases / Data Warehouses inaccessible
# Set the key as the TDE protector for all resources under the server
Set-AzureRmSqlServerTransparentDataEncryptionProtector `
-ResourceGroupName <SQLDatabaseResourceGroupName> `
-ServerName <LogicalServerName> `
-Type AzureKeyVault -KeyId <KeyVaultKeyId>
3. Make sure the server and any replicas have updated to the new TDE protector using the Get-
AzureRmSqlServerTransparentDataEncryptionProtector cmdlet.
NOTE
It may take a few minutes for the new TDE protector to propagate to all databases and secondary databases under
the server.
Get-AzureRmSqlServerTransparentDataEncryptionProtector `
-ServerName <LogicalServerName> `
-ResourceGroupName <SQLDatabaseResourceGroupName>
5. Delete the compromised key from Key Vault using the Remove-AzureKeyVaultKey cmdlet.
Remove-AzureKeyVaultKey `
-VaultName <KeyVaultName> `
-Name <KeyVaultKeyName>
6. To restore a key to Key Vault in the future using the Restore-AzureKeyVaultKey cmdlet:
Restore-AzureKeyVaultKey `
-VaultName <KeyVaultName> `
-InputFile <BackupFilePath>
Next steps
Learn how to rotate the TDE protector of a server to comply with security requirements: Rotate the
Transparent Data Encryption protector Using PowerShell
Get started with Bring Your Own Key support for TDE: Turn on TDE using your own key from Key Vault
using PowerShell
Scale database resources
8/20/2018 • 3 minutes to read • Edit Online
Azure SQL Database enables you to dynamically add more resources to your database with minimal downtime.
Overview
When demand for your app grows from a handful of devices and customers to millions, Azure SQL Database
scales on the fly with minimal downtime. Scalability is one of the most important characteristics of PaaS that
enables you to dynamically add more resources to your service when needed. Azure SQL Database enables you to
easily change resources (CPU power, memory, IO throughput, and storage) allocated to your databases.
You can mitigate performance issues due to increased usage of your application that cannot be fixed using indexing
or query rewrite methods. Adding more resources enables you to quickly react when your database hits the
current resource limits and needs more power to handle the incoming workload. Azure SQL Database also enables
you to scale-down the resources when they are not needed to lower the cost. You don’t need to worry about
purchasing hardware and changing underlying infrastructure. Scaling database can be easily done via Azure portal
using a slider.
Azure SQL Database offers a DTU -based purchasing model or the vCore-based purchasing model.
The DTU -based purchasing model offers a blend of compute, memory, and IO resources in three service tiers to
support lightweight to heavyweight database workloads: Basic, Standard, and Premium. Performance levels
within each tier provide a different mix of these resources, to which you can add additional storage resources.
The vCore-based purchasing model lets you choose the number of vCores, the amount or memory, and the
amount and speed of storage. You can build your first app on a small, single database at a low cost per month
and then change its service tier manually or programmatically at any time to meet the needs of your solution.
You can adjust performance without downtime to your app or to your customers. Dynamic scalability enables
your database to transparently respond to rapidly changing resource requirements and enables you to only pay
for the resources that you need when you need them.
NOTE
Dynamic scalability is different from autoscale. Autoscale is when a service scales automatically based on criteria, whereas
dynamic scalability allows for manual scaling without downtime.
Single Azure SQL Database supports manual dynamic scalability, but not autoscale. For a more automatic
experience, consider using elastic pools, which allow databases to share resources in a pool based on individual
database needs. However, there are scripts that can help automate scalability for a single Azure SQL Database. For
an example, see Use PowerShell to monitor and scale a single SQL Database.
You can change DTU service tiers or vCore characteristics at any time with minimal downtime to your application
(generally averaging under four seconds). For many businesses and apps, being able to create databases and dial
performance up or down on demand is enough, especially if usage patterns are relatively predictable. But if you
have unpredictable usage patterns, it can make it hard to manage costs and your business model. For this scenario,
you use an elastic pool with a certain number of eDTUs that are shared among multiple databases in the pool.
All three flavors of Azure SQL Database offer some ability to dynamically scale your databases:
In Azure SQL Single Database, you can use either DTU or vCore models to define maximum amount of
resources that will be assigned to each database.
Azure SQL Managed Instance uses vCores mode and enables you to define maximum CPU cores and
maximum of storage allocated to your instance. All databases within the instance will share the resources
allocated to the instance.
Azure SQL Elastic Pools enable you to define maximum resource limit per group of databases in the pool.
Next steps
For information about improving database performance by changing database code, see Find and apply
performance recommendations.
For information about letting built-in database intelligence optimize your database, see Automatic tuning.
For information about Read Scale-out in the Azure SQL Database service, see how to use read-only replicas to
load balance read-only query workloads.
For information about a Database sharding, see Scaling out with Azure SQL Database.
Use read-only replicas to load balance read-only
query workloads (preview)
8/28/2018 • 5 minutes to read • Edit Online
Read Scale-Out allows you to load balance Azure SQL Database read-only workloads using the capacity of read-
only replicas.
These replicas are provisioned with the same performance level as the read-write replica used by the regular
database connections. The Read Scale-Out feature allows you to load balance SQL Database read-only
workloads using the capacity of one of the read-only replicas instead of sharing the read-write replica. This way
the read-only workload will be isolated from the main read-write workload and will not affect its performance.
The feature is intended for the applications that include logically separated read-only workloads, such as analytics,
and therefore could gain performance benefits using this additional capacity at no extra cost.
To use the Read Scale-Out feature with a particular database, you must explicitly enable it when creating the
database or afterwards by altering its configuration using PowerShell by invoking the Set-AzureRmSqlDatabase
or the New -AzureRmSqlDatabase cmdlets or through the Azure Resource Manager REST API using the
Databases - Create or Update method.
After Read Scale-Out is enabled for a database, applications connecting to that database will be directed to either
the read-write replica or to a read-only replica of that database according to the ApplicationIntent property
configured in the application’s connection string. For information on the ApplicationIntent property, see
Specifying Application Intent.
If Read Scale-Out is disabled or you set the ReadScale property in an unsupported service tier, all connections are
directed to the read-write replica, independent of the ApplicationIntent property.
NOTE
During preview, Query Data Store and Extended Events are not supported on the read-only replicas.
Data consistency
One of the benefits of Always ON is that the replicas are always in the transactionally consistent state, but at
different points in time there may be some small latency between different replicas. Read Scale-Out supports
session-level consistency. It means, if the read-only session reconnects after a connection error caused by replica
unavailability, it can be redirected to a replica that is not 100% up-to-date with the read-write replica. Likewise, if
an application writes data using a read-write session and immediately reads it using a read-only session, it is
possible that the latest updates are not immediately visible. This is because the transaction log redo to the replicas
is asynchronous.
NOTE
Replication latencies within the region are low and this situation is rare.
Server=tcp:<server>.database.windows.net;Database=<mydatabase>;ApplicationIntent=ReadOnly;User ID=
<myLogin>;Password=<myPassword>;Trusted_Connection=False; Encrypt=True;
Either of the following connection strings connects the client to a read-write replica (replacing the items in the
angle brackets with the correct values for your environment and dropping the angle brackets):
Server=tcp:<server>.database.windows.net;Database=<mydatabase>;ApplicationIntent=ReadWrite;User ID=
<myLogin>;Password=<myPassword>;Trusted_Connection=False; Encrypt=True;
Server=tcp:<server>.database.windows.net;Database=<mydatabase>;User ID=<myLogin>;Password=
<myPassword>;Trusted_Connection=False; Encrypt=True;
You can verify whether you are connected to a read-only replica by running the following query. It will return
READ_ONLY when connected to a read-only replica.
NOTE
At any given time only one of the AlwaysON replicas is accessible by the ReadOnly sessions.
To disable read scale-out for an existing database (replacing the items in the angle brackets with the correct values
for your environment and dropping the angle brackets):
To create a new database with read scale-out enabled (replacing the items in the angle brackets with the correct
values for your environment and dropping the angle brackets):
Enabling and disabling Read Scale-Out using the Azure SQL Database
REST API
To create a database with read scale-out enabled, or to enable or disable read scale-out for an existing database,
create, or update the corresponding database entity with the readScale property set to Enabled or Disabled as
in the below sample request.
Method: PUT
URL:
https://management.azure.com/subscriptions/{SubscriptionId}/resourceGroups/{GroupName}/providers/Microsoft.Sql
/servers/{ServerName}/databases/{DatabaseName}?api-version= 2014-04-01-preview
Body:
{
"properties":
{
"readScale":"Enabled"
}
}
NOTE
During preview, we will not perform round-robin or any other load balanced routing between the local replicas of the
secondary database.
Next steps
For information about using PowerShell to set read scale-out, see the Set-AzureRmSqlDatabase or the New -
AzureRmSqlDatabase cmdlets.
For information about using the REST API to set read scale-out, see Databases - Create or Update.
Scaling out with Azure SQL Database
5/23/2018 • 6 minutes to read • Edit Online
You can easily scale out Azure SQL databases using the Elastic Database tools. These tools and features let you
use the database resources of Azure SQL Database to create solutions for transactional workloads, and
especially Software as a Service (SaaS ) applications. Elastic Database features are composed of the:
Elastic Database client library: The client library is a feature that allows you to create and maintain sharded
databases. See Get started with Elastic Database tools.
Elastic Database split-merge tool: moves data between sharded databases. This is useful for moving data
from a multi-tenant database to a single-tenant database (or vice-versa). See Elastic database Split-Merge
tool tutorial.
Elastic Database jobs (preview ): Use jobs to manage large numbers of Azure SQL databases. Easily perform
administrative operations such as schema changes, credentials management, reference data updates,
performance data collection, or tenant (customer) telemetry collection using jobs.
Elastic Database query (preview ): Enables you to run a Transact-SQL query that spans multiple databases.
This enables connection to reporting tools such as Excel, PowerBI, Tableau, etc.
Elastic transactions: This feature allows you to run transactions that span several databases in Azure SQL
Database. Elastic database transactions are available for .NET applications using ADO .NET and integrate
with the familiar programming experience using the System.Transaction classes.
The following graphic shows an architecture that includes the Elastic Database features in relation to a
collection of databases.
In this graphic, colors of the database represent schemas. Databases with the same color share the same schema.
1. A set of Azure SQL databases are hosted on Azure using sharding architecture.
2. The Elastic Database client library is used to manage a shard set.
3. A subset of the databases are put into an elastic pool. (See What is a pool?).
4. An Elastic Database job runs scheduled or ad hoc T-SQL scripts against all databases.
5. The split-merge tool is used to move data from one shard to another.
6. The Elastic Database query allows you to write a query that spans all databases in the shard set.
7. Elastic transactions allow you to run transactions that span several databases.
Why use the tools?
Achieving elasticity and scale for cloud applications has been straightforward for VMs and blob storage - simply
add or subtract units, or increase power. But it has remained a challenge for stateful data processing in relational
databases. Challenges emerged in these scenarios:
Growing and shrinking capacity for the relational database part of your workload.
Managing hotspots that may arise affecting a specific subset of data - such as a busy end-customer (tenant).
Traditionally, scenarios like these have been addressed by investing in larger-scale database servers to support
the application. However, this option is limited in the cloud where all processing happens on predefined
commodity hardware. Instead, distributing data and processing across many identically structured databases (a
scale-out pattern known as "sharding") provides an alternative to traditional scale-up approaches both in terms
of cost and elasticity.
Sharding
Sharding is a technique to distribute large amounts of identically structured data across a number of
independent databases. It is especially popular with cloud developers creating Software as a Service (SAAS )
offerings for end customers or businesses. These end customers are often referred to as “tenants”. Sharding may
be required for any number of reasons:
The total amount of data is too large to fit within the constraints of a single database
The transaction throughput of the overall workload exceeds the capabilities of a single database
Tenants may require physical isolation from each other, so separate databases are needed for each tenant
Different sections of a database may need to reside in different geographies for compliance, performance, or
geopolitical reasons.
In other scenarios, such as ingestion of data from distributed devices, sharding can be used to fill a set of
databases that are organized temporally. For example, a separate database can be dedicated to each day or week.
In that case, the sharding key can be an integer representing the date (present in all rows of the sharded tables)
and queries retrieving information for a date range must be routed by the application to the subset of databases
covering the range in question.
Sharding works best when every transaction in an application can be restricted to a single value of a sharding
key. That ensures that all transactions are local to a specific database.
Multi-tenant and single-tenant
Some applications use the simplest approach of creating a separate database for each tenant. This is the single
tenant sharding pattern that provides isolation, backup/restore ability, and resource scaling at the granularity
of the tenant. With single tenant sharding, each database is associated with a specific tenant ID value (or
customer key value), but that key need not always be present in the data itself. It is the application’s responsibility
to route each request to the appropriate database - and the client library can simplify this.
Others scenarios pack multiple tenants together into databases, rather than isolating them into separate
databases. This is a typical multi-tenant sharding pattern - and it may be driven by the fact that an application
manages large numbers of small tenants. In multi-tenant sharding, the rows in the database tables are all
designed to carry a key identifying the tenant ID or sharding key. Again, the application tier is responsible for
routing a tenant’s request to the appropriate database, and this can be supported by the elastic database client
library. In addition, row -level security can be used to filter which rows each tenant can access - for details, see
Multi-tenant applications with elastic database tools and row -level security. Redistributing data among
databases may be needed with the multi-tenant sharding pattern, and this is facilitated by the elastic database
split-merge tool. To learn more about design patterns for SaaS applications using elastic pools, see Design
Patterns for Multi-tenant SaaS Applications with Azure SQL Database.
Move data from multiple to single -tenancy databases
When creating a SaaS application, it is typical to offer prospective customers a trial version of the software. In
this case, it is cost-effective to use a multi-tenant database for the data. However, when a prospect becomes a
customer, a single-tenant database is better since it provides better performance. If the customer had created
data during the trial period, use the split-merge tool to move the data from the multi-tenant to the new single-
tenant database.
Next steps
For a sample app that demonstrates the client library, see Get started with Elastic Database tools.
To convert existing databases to use the tools, see Migrate existing databases to scale out.
To see the specifics of the elastic pool, see Price and performance considerations for an elastic pool, or create a
new pool with elastic pools.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Building scalable cloud databases
5/23/2018 • 3 minutes to read • Edit Online
Scaling out databases can be easily accomplished using scalable tools and features for Azure SQL Database. In
particular, you can use the Elastic Database client library to create and manage scaled-out databases. This
feature lets you easily develop sharded applications using hundreds—or even thousands—of Azure SQL
databases. Elastic jobs can then be used to help ease management of these databases.
To download:
The Java version of the library, see Maven Central Repository.
The .NET version of the library, see NuGet.
Documentation
1. Get started with Elastic Database tools
2. Elastic Database features
3. Shard map management
4. Migrate existing databases to scale out
5. Data dependent routing
6. Multi-shard queries
7. Adding a shard using Elastic Database tools
8. Multi-tenant applications with elastic database tools and row -level security
9. Upgrade client library apps
10. Elastic queries overview
11. Elastic database tools glossary
12. Elastic Database client library with Entity Framework
13. Elastic database client library with Dapper
14. Split-merge tool
15. Performance counters for shard map manager
16. FAQ for Elastic database tools
Client capabilities
Scaling out applications using sharding presents challenges for both the developer as well as the administrator.
The client library simplifies the management tasks by providing tools that let both developers and
administrators manage scaled-out databases. In a typical example, there are many databases, known as
"shards," to manage. Customers are co-located in the same database, and there is one database per customer (a
single-tenant scheme). The client library includes these features:
Shard Map Management: A special database called the "shard map manager" is created. Shard map
management is the ability for an application to manage metadata about its shards. Developers can use
this functionality to register databases as shards, describe mappings of individual sharding keys or key
ranges to those databases, and maintain this metadata as the number and composition of databases
evolves to reflect capacity changes. Without the elastic database client library, you would need to spend a
lot of time writing the management code when implementing sharding. For details, see Shard map
management.
Data dependent routing: Imagine a request coming into the application. Based on the sharding key
value of the request, the application needs to determine the correct database based on the key value. It
then opens a connection to the database to process the request. Data dependent routing provides the
ability to open connections with a single easy call into the shard map of the application. Data dependent
routing was another area of infrastructure code that is now covered by functionality in the elastic
database client library. For details, see Data dependent routing.
Multi-shard queries (MSQ ): Multi-shard querying works when a request involves several (or all) shards. A
multi-shard query executes the same T-SQL code on all shards or a set of shards. The results from the
participating shards are merged into an overall result set using UNION ALL semantics. The functionality as
exposed through the client library handles many tasks, including: connection management, thread
management, fault handling, and intermediate results processing. MSQ can query up to hundreds of shards.
For details, see Multi-shard querying.
In general, customers using elastic database tools can expect to get full T-SQL functionality when submitting
shard-local operations as opposed to cross-shard operations that have their own semantics.
Next steps
Elastic Database Client Library (Java, .NET) - to download the library.
Get started with elastic database tools - to try the sample app that demonstrates client functions.
GitHub (Java, .NET) - to make contributions to the code.
Azure SQL Database elastic query overview - to use elastic queries.
Moving data between scaled-out cloud databases - for instructions on using the split-merge tool.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Upgrade an app to use the latest elastic database
client library
9/12/2018 • 3 minutes to read • Edit Online
New versions of the Elastic Database client library are available through NuGetand the NuGetPackage Manager
interface in Visual Studio. Upgrades contain bug fixes and support for new capabilities of the client library.
For the latest version: Go to Microsoft.Azure.SqlDatabase.ElasticScale.Client.
Rebuild your application with the new library, as well as change your existing Shard Map Manager metadata
stored in your Azure SQL databases to support new features.
Performing these steps in order ensures that old versions of the client library are no longer present in your
environment when metadata objects are updated, which means that old-version metadata objects won’t be created
after upgrade.
Upgrade steps
1. Upgrade your applications. In Visual Studio, download and reference the latest client library version into all
of your development projects that use the library; then rebuild and deploy.
In your Visual Studio solution, select Tools --> NuGet Package Manager --> Manage NuGet Packages for
Solution.
(Visual Studio 2013) In the left panel, select Updates, and then select the Update button on the package
Azure SQL Database Elastic Scale Client Library that appears in the window.
(Visual Studio 2015) Set the Filter box to Upgrade available. Select the package to update, and click the
Update button.
(Visual Studio 2017) At the top of the dialog, select Updates. Select the package to update, and click the
Update button.
Build and Deploy.
2. Upgrade your scripts. If you are using PowerShell scripts to manage shards, download the new library
version and copy it into the directory from which you execute scripts.
3. Upgrade your split-merge service. If you use the elastic database split-merge tool to reorganize sharded
data, download and deploy the latest version of the tool. Detailed upgrade steps for the Service can be found here.
4. Upgrade your Shard Map Manager databases. Upgrade the metadata supporting your Shard Maps in
Azure SQL Database. There are two ways you can accomplish this, using PowerShell or C#. Both options are
shown below.
Option 1: Upgrade metadata using PowerShell
1. Download the latest command-line utility for NuGet from here and save to a folder.
2. Open a Command Prompt, navigate to the same folder, and issue the command:
nuget install Microsoft.Azure.SqlDatabase.ElasticScale.Client
3. Navigate to the subfolder containing the new client DLL version you have just downloaded, for example:
cd .\Microsoft.Azure.SqlDatabase.ElasticScale.Client.1.0.0\lib\net45
4. Download the elastic database client upgrade scriptlet from the Script Center, and save it into the same folder
containing the DLL.
5. From that folder, run “PowerShell .\upgrade.ps1” from the command prompt and follow the prompts.
Option 2: Upgrade metadata using C#
Alternatively, create a Visual Studio application that opens your ShardMapManager, iterates over all shards, and
performs the metadata upgrade by calling the methods UpgradeLocalStore and UpgradeGlobalStore as in this
example:
ShardMapManager smm =
ShardMapManagerFactory.GetSqlShardMapManager
(connStr, ShardMapManagerLoadPolicy.Lazy);
smm.UpgradeGlobalStore();
These techniques for metadata upgrades can be applied multiple times without harm. For example, if an older
client version inadvertently creates a shard after you have already updated, you can run upgrade again across all
shards to ensure that the latest metadata version is present throughout your infrastructure.
Note: New versions of the client library published to-date continue to work with prior versions of the Shard Map
Manager metadata on Azure SQL DB, and vice-versa. However to take advantage of some of the new features in
the latest client, metadata needs to be upgraded. Note that metadata upgrades will not affect any user-data or
application-specific data, only objects created and used by the Shard Map Manager. And applications continue to
operate through the upgrade sequence described above.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Scale out databases with the shard map manager
5/23/2018 • 12 minutes to read • Edit Online
To easily scale out databases on SQL Azure, use a shard map manager. The shard map manager is a special
database that maintains global mapping information about all shards (databases) in a shard set. The metadata
allows an application to connect to the correct database based upon the value of the sharding key. In addition,
every shard in the set contains maps that track the local shard data (known as shardlets).
Understanding how these maps are constructed is essential to shard map management. This is done using the
ShardMapManager class (Java, .NET, found in the Elastic Database client library to manage shard maps.
Or you can implement a multi-tenant database model using a list mapping to assign multiple tenants to a single
database. For example, DB1 is used to store information about tenant ID 1 and 5, and DB2 stores data for
tenant 7 and tenant 10.
integer integer
long long
guid uuid
byte[] byte[]
datetime timestamp
timespan duration
datetimeoffset offsetdatetime
1 Database_A
3 Database_B
4 Database_C
6 Database_B
... ...
[1,50) Database_A
[50,100) Database_B
KEY SHARD LOCATION
[100,200) Database_C
[400,600) Database_C
... ...
Each of the tables shown above is a conceptual example of a ShardMap object. Each row is a simplified
example of an individual PointMapping (for the list shard map) or RangeMapping (for the range shard map)
object.
Constructing a ShardMapManager
A ShardMapManager object is constructed using a factory ( Java, .NET) pattern. The
ShardMapManagerFactory.GetSqlShardMapManager (Java, .NET) method takes credentials (including the
server name and database name holding the GSM ) in the form of a ConnectionString and returns an instance
of a ShardMapManager.
Please Note: The ShardMapManager should be instantiated only once per app domain, within the
initialization code for an application. Creation of additional instances of ShardMapManager in the same app
domain results in increased memory and CPU utilization of the application. A ShardMapManager can contain
any number of shard maps. While a single shard map may be sufficient for many applications, there are times
when different sets of databases are used for different schema or for unique purposes; in those cases multiple
shard maps may be preferable.
In this code, an application tries to open an existing ShardMapManager with the TryGetSqlShardMapManager
(Java, .NET method. If objects representing a Global ShardMapManager (GSM ) do not yet exist inside the
database, the client library creates them there using the CreateSqlShardMapManager (Java, .NET) method.
// Try to get a reference to the Shard Map Manager in the shardMapManager database.
// If it doesn't already exist, then create it.
ShardMapManager shardMapManager = null;
boolean shardMapManagerExists =
ShardMapManagerFactory.tryGetSqlShardMapManager(shardMapManagerConnectionString,ShardMapManagerLoadPolicy.La
zy, refShardMapManager);
shardMapManager = refShardMapManager.argValue;
if (shardMapManagerExists) {
ConsoleUtils.writeInfo("Shard Map %s already exists", shardMapManager);
}
else {
// The Shard Map Manager does not exist, so create it
shardMapManager = ShardMapManagerFactory.createSqlShardMapManager(shardMapManagerConnectionString);
ConsoleUtils.writeInfo("Created Shard Map %s", shardMapManager);
}
// Try to get a reference to the Shard Map Manager via the Shard Map Manager database.
// If it doesn't already exist, then create it.
ShardMapManager shardMapManager;
bool shardMapManagerExists = ShardMapManagerFactory.TryGetSqlShardMapManager(
connectionString,
ShardMapManagerLoadPolicy.Lazy,
out shardMapManager);
if (shardMapManagerExists)
{
Console.WriteLine("Shard Map Manager already exists");
}
else
{
// Create the Shard Map Manager.
ShardMapManagerFactory.CreateSqlShardMapManager(connectionString);
Console.WriteLine("Created SqlShardMapManager");
shardMapManager = ShardMapManagerFactory.GetSqlShardMapManager(
connectionString,
ShardMapManagerLoadPolicy.Lazy);
// The connectionString contains server name, database name, and admin credentials for privileges on both
the GSM and the shards themselves.
}
For the .NET version, you can use PowerShell to create a new Shard Map Manager. An example is available
here.
return shardMap;
}
// Creates a new Range Shard Map with the specified name, or gets the Range Shard Map if it already exists.
public static RangeShardMap<T> CreateOrGetRangeShardMap<T>(ShardMapManager shardMapManager, string
shardMapName)
{
// Try to get a reference to the Shard Map.
RangeShardMap<T> shardMap;
bool shardMapExists = shardMapManager.TryGetRangeShardMap(shardMapName, out shardMap);
if (shardMapExists)
{
ConsoleUtils.WriteInfo("Shard Map {0} already exists", shardMap.Name);
}
else
{
// The Shard Map does not exist, so create it
shardMap = shardMapManager.CreateRangeShardMap<T>(shardMapName);
ConsoleUtils.WriteInfo("Created Shard Map {0}", shardMap.Name);
}
return shardMap;
}
sm.DeleteMapping(sm.MarkMappingOffline(sm.GetMappingForKey(25)));
Adding a shard
Applications often need to add new shards to handle data that is expected from new keys or key ranges, for a
shard map that already exists. For example, an application sharded by Tenant ID may need to provision a new
shard for a new tenant, or data sharded monthly may need a new shard provisioned before the start of each
new month.
If the new range of key values is not already part of an existing mapping and no data movement is necessary, it
is simple to add the new shard and associate the new key or range to that shard. For details on adding new
shards, see Adding a new shard.
For scenarios that require data movement, however, the split-merge tool is needed to orchestrate the data
movement between shards in combination with the necessary shard map updates. For details on using the split-
merge tool, see Overview of split-merge
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Data dependent routing
5/23/2018 • 6 minutes to read • Edit Online
Data dependent routing is the ability to use the data in a query to route the request to an appropriate
database. This is a fundamental pattern when working with sharded databases. The request context may also be
used to route the request, especially if the sharding key is not part of the query. Each specific query or
transaction in an application using data dependent routing is restricted to accessing a single database per
request. For the Azure SQL Database Elastic tools, this routing is accomplished with the ShardMapManager
(Java, .NET) class.
The application does not need to track various connection strings or DB locations associated with different slices
of data in the sharded environment. Instead, the Shard Map Manager opens connections to the correct
databases when needed, based on the data in the shard map and the value of the sharding key that is the target
of the application’s request. The key is typically the customer_id, tenant_id, date_key, or some other specific
identifier that is a fundamental parameter of the database request.
For more information, see Scaling Out SQL Server with Data Dependent Routing.
Use lowest privilege credentials possible for getting the shard map
If an application is not manipulating the shard map itself, the credentials used in the factory method should have
just read-only permissions on the Global Shard Map database. These credentials are typically different from
credentials used to open connections to the shard map manager. See also Credentials used to access the Elastic
Database client library.
// Syntax:
public Connection openConnectionForKey(Object key, String connectionString, ConnectionOptions options)
// Syntax:
public SqlConnection OpenConnectionForKey<TKey>(TKey key, string connectionString, ConnectionOptions
options)
The key parameter is used as a lookup key into the shard map to determine the appropriate database for the
request.
The connectionString is used to pass only the user credentials for the desired connection. No database
name or server name is included in this connectionString since the method determines the database and
server using the ShardMap.
The connectionOptions (Java, .NET) should be set to ConnectionOptions.Validate if an environment
where shard maps may change and rows may move to other databases as a result of split or merge
operations. This involves a brief query to the local shard map on the target database (not to the global shard
map) before the connection is delivered to the application.
If the validation against the local shard map fails (indicating that the cache is incorrect), the Shard Map Manager
queries the global shard map to obtain the new correct value for the lookup, update the cache, and obtain and
return the appropriate database connection.
Use ConnectionOptions.None only when shard mapping changes are not expected while an application is
online. In that case, the cached values can be assumed to always be correct, and the extra round-trip validation
call to the target database can be safely skipped. That reduces database traffic. The connectionOptions may
also be set via a value in a configuration file to indicate whether sharding changes are expected or not during a
period of time.
This example uses the value of an integer key CustomerID, using a ShardMap object named
customerShardMap.
ps.setInt(1, productId);
ps.setInt(2, customerId);
ps.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
int customerId = 12345;
int newPersonId = 4321;
// Connect to the shard for that customer ID. No need to call a SqlConnection
// constructor followed by the Open method.
using (SqlConnection conn = customerShardMap.OpenConnectionForKey(customerId,
Configuration.GetCredentialsConnectionString(), ConnectionOptions.Validate))
{
// Execute a simple command.
SqlCommand cmd = conn.CreateCommand();
cmd.CommandText = @"UPDATE Sales.Customer
SET PersonID = @newPersonID WHERE CustomerID = @customerID";
cmd.Parameters.AddWithValue("@customerID", customerId);cmd.Parameters.AddWithValue("@newPersonID",
newPersonId);
cmd.ExecuteNonQuery();
}
The OpenConnectionForKey method returns a new already-open connection to the correct database.
Connections utilized in this way still take full advantage of connection pooling.
The OpenConnectionForKeyAsync method (Java, .NET) is also available if your application makes use
asynchronous programming.
ps.setInt(1, productId);
ps.setInt(2, customerId);
ps.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
});
} catch (Exception e) {
throw new StoreException(e.getMessage(), e);
}
int customerId = 12345;
int newPersonId = 4321;
Configuration.SqlRetryPolicy.ExecuteAction(() =>
{
// Connect to the shard for a customer ID.
using (SqlConnection conn = customerShardMap.OpenConnectionForKey(customerId,
Configuration.GetCredentialsConnectionString(), ConnectionOptions.Validate))
{
// Execute a simple command
SqlCommand cmd = conn.CreateCommand();
cmd.Parameters.AddWithValue("@customerID", customerId);
cmd.Parameters.AddWithValue("@newPersonID", newPersonId);
cmd.ExecuteNonQuery();
Console.WriteLine("Update completed");
}
});
Packages necessary to implement transient fault handling are downloaded automatically when you build the
elastic database sample application.
Transactional consistency
Transactional properties are guaranteed for all operations local to a shard. For example, transactions submitted
through data-dependent routing execute within the scope of the target shard for the connection. At this time,
there are no capabilities provided for enlisting multiple connections into a transaction, and therefore there are no
transactional guarantees for operations performed across shards.
Next steps
To detach a shard, or to reattach a shard, see Using the RecoveryManager class to fix shard map problems
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Credentials used to access the Elastic Database client
library
5/23/2018 • 3 minutes to read • Edit Online
The Elastic Database client library uses three different kinds of credentials to access the shard map manager.
Depending on the need, use the credential with the lowest level of access possible.
Management credentials: for creating or manipulating a shard map manager. (See the glossary.)
Access credentials: to access an existing shard map manager to obtain information about shards.
Connection credentials: to connect to shards.
See also Managing databases and logins in Azure SQL Database.
The variable smmAdminConnectionString is a connection string that contains the management credentials. The
user ID and password provide read/write access to both shard map database and individual shards. The
management connection string also includes the server name and database name to identify the global shard map
database. Here is a typical connection string for that purpose:
"Server=<yourserver>.database.windows.net;Database=<yourdatabase>;User ID=<yourmgmtusername>;Password=
<yourmgmtpassword>;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;”
Do not use values in the form of "username@server"—instead just use the "username" value. This is because
credentials must work against both the shard map manager database and individual shards, which may be on
different servers.
Access credentials
When creating a shard map manager in an application that does not administer shard maps, use credentials that
have read-only permissions on the global shard map. The information retrieved from the global shard map under
these credentials is used for data-dependent routing and to populate the shard map cache on the client. The
credentials are provided through the same call pattern to GetSqlShardMapManager:
// Obtain shard map manager.
ShardMapManager shardMapManager = ShardMapManagerFactory.GetSqlShardMapManager(smmReadOnlyConnectionString,
ShardMapManagerLoadPolicy.Lazy);
Note the use of the smmReadOnlyConnectionString to reflect the use of different credentials for this access on
behalf of non-admin users: these credentials should not provide write permissions on the global shard map.
Connection credentials
Additional credentials are needed when using the OpenConnectionForKey (Java, .NET) method to access a
shard associated with a sharding key. These credentials need to provide permissions for read-only access to the
local shard map tables residing on the shard. This is needed to perform connection validation for data-dependent
routing on the shard. This code snippet allows data access in the context of data-dependent routing:
In this example, smmUserConnectionString holds the connection string for the user credentials. For Azure SQL
DB, here is a typical connection string for user credentials:
As with the admin credentials, do not use values in the form of "username@server". Instead, just use "username".
Also note that the connection string does not contain a server name and database name. That is because the
OpenConnectionForKey call automatically directs the connection to the correct shard based on the key. Hence,
the database name and server name are not provided.
See also
Managing databases and logins in Azure SQL Database
Securing your SQL Database
Getting started with Elastic Database jobs
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Multi-shard querying
9/12/2018 • 2 minutes to read • Edit Online
Overview
With the Elastic Database tools, you can create sharded database solutions. Multi-shard querying is used for
tasks such as data collection/reporting that require running a query that stretches across several shards. (Contrast
this to data-dependent routing, which performs all work on a single shard.)
1. Get a RangeShardMap (Java, .NET) or ListShardMap (Java, .NET) using the TryGetRangeShardMap (Java,
.NET), the TryGetListShardMap (Java, .NET), or the GetShardMap (Java, .NET) method. See Constructing a
ShardMapManager and Get a RangeShardMap or ListShardMap.
2. Create a MultiShardConnection (Java, .NET) object.
3. Create a MultiShardStatement or MultiShardCommand (Java, .NET).
4. Set the CommandText property (Java, .NET) to a T-SQL command.
5. Execute the command by calling the ExecuteQueryAsync or ExecuteReader (Java, .NET) method.
6. View the results using the MultiShardResultSet or MultiShardDataReader (Java, .NET) class.
Example
The following code illustrates the usage of multi-shard querying using a given ShardMap named myShardMap.
A key difference is the construction of multi-shard connections. Where SqlConnection operates on a single
database, the MultiShardConnection takes a collection of shards as its input. Populate the collection of shards
from a shard map. The query is then executed on the collection of shards using UNION ALL semantics to
assemble a single overall result. Optionally, the name of the shard where the row originates from can be added to
the output using the ExecutionOptions property on command.
Note the call to myShardMap.GetShards(). This method retrieves all shards from the shard map and provides
an easy way to run a query across all relevant databases. The collection of shards for a multi-shard query can be
refined further by performing a LINQ query over the collection returned from the call to
myShardMap.GetShards(). In combination with the partial results policy, the current capability in multi-shard
querying has been designed to work well for tens up to hundreds of shards.
A limitation with multi-shard querying is currently the lack of validation for shards and shardlets that are queried.
While data-dependent routing verifies that a given shard is part of the shard map at the time of querying, multi-
shard queries do not perform this check. This can lead to multi-shard queries running on databases that have
been removed from the shard map.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Elastic Database tools glossary
5/23/2018 • 2 minutes to read • Edit Online
The following terms are defined for the Elastic Database tools, a feature of Azure SQL Database. The tools are
used to manage shard maps, and include the client library, the split-merge tool, elastic pools, and queries.
These terms are used in Adding a shard using Elastic Database tools and Using the RecoveryManager class to fix
shard map problems.
Range shard map: A shard map in which the shard distribution strategy is based on multiple ranges of
contiguous values.
Reference tables: Tables that are not sharded but are replicated across shards. For example, zip codes can be
stored in a reference table.
Shard: An Azure SQL database that stores data from a sharded data set.
Shard elasticity: The ability to perform both horizontal scaling and vertical scaling.
Sharded tables: Tables that are sharded, i.e., whose data is distributed across shards based on their sharding key
values.
Sharding key: A column value that determines how data is distributed across shards. The value type can be one
of the following: int, bigint, varbinary, or uniqueidentifier.
Shard set: The collection of shards that are attributed to the same shard map in the shard map manager.
Shardlet: All of the data associated with a single value of a sharding key on a shard. A shardlet is the smallest unit
of data movement possible when redistributing sharded tables.
Shard map: The set of mappings between sharding keys and their respective shards.
Shard map manager: A management object and data store that contains the shard map(s), shard locations, and
mappings for one or more shard sets.
Verbs
Horizontal scaling: The act of scaling out (or in) a collection of shards by adding or removing shards to a shard
map, as shown below.
Merge: The act of moving shardlets from two shards to one shard and updating the shard map accordingly.
Shardlet move: The act of moving a single shardlet to a different shard.
Shard: The act of horizontally partitioning identically structured data across multiple databases based on a
sharding key.
Split: The act of moving several shardlets from one shard to another (typically new ) shard. A sharding key is
provided by the user as the split point.
Vertical Scaling: The act of scaling up (or down) the performance level of an individual shard. For example,
changing a shard from Standard to Premium (which results in more computing resources).
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Moving data between scaled-out cloud databases
5/23/2018 • 18 minutes to read • Edit Online
If you are a Software as a Service developer, and suddenly your app undergoes tremendous demand, you need
to accommodate the growth. So you add more databases (shards). How do you redistribute the data to the new
databases without disrupting the data integrity? Use the split-merge tool to move data from constrained
databases to the new databases.
The split-merge tool runs as an Azure web service. An administrator or developer uses the tool to move
shardlets (data from a shard) between different databases (shards). The tool uses shard map management to
maintain the service metadata database, and ensure consistent mappings.
Download
Microsoft.Azure.SqlDatabase.ElasticScale.Service.SplitMerge
Documentation
1. Elastic database Split-Merge tool tutorial
2. Split-Merge security configuration
3. Split-merge security considerations
4. Shard map management
5. Migrate existing databases to scale-out
6. Elastic database tools
7. Elastic Database tools glossary
// Reference tables
schemaInfo.Add(new ReferenceTableInfo("dbo", "region"));
schemaInfo.Add(new ReferenceTableInfo("dbo", "nation"));
// Sharded tables
schemaInfo.Add(new ShardedTableInfo("dbo", "customer", "C_CUSTKEY"));
schemaInfo.Add(new ShardedTableInfo("dbo", "orders", "O_CUSTKEY"));
// Publish
smm.GetSchemaInfoCollection().Add(Configuration.ShardMapName, schemaInfo);
The tables ‘region’ and ‘nation’ are defined as reference tables and will be copied with split/merge/move
operations. ‘customer’ and ‘orders’ in turn are defined as sharded tables. C_CUSTKEY and O_CUSTKEY serve
as the sharding key.
Referential Integrity
The split-merge service analyzes dependencies between tables and uses foreign key-primary key relationships
to stage the operations for moving reference tables and shardlets. In general, reference tables are copied first in
dependency order, then shardlets are copied in order of their dependencies within each batch. This is necessary
so that FK-PK constraints on the target shard are honored as the new data arrives.
Shard Map Consistency and Eventual Completion
In the presence of failures, the split-merge service resumes operations after any outage and aims to complete
any in progress requests. However, there may be unrecoverable situations, e.g., when the target shard is lost or
compromised beyond repair. Under those circumstances, some shardlets that were supposed to be moved may
continue to reside on the source shard. The service ensures that shardlet mappings are only updated after the
necessary data has been successfully copied to the target. Shardlets are only deleted on the source once all their
data has been copied to the target and the corresponding mappings have been updated successfully. The
deletion operation happens in the background while the range is already online on the target shard. The split-
merge service always ensures correctness of the mappings stored in the shard map.
Billing
The split-merge service runs as a cloud service in your Microsoft Azure subscription. Therefore charges for
cloud services apply to your instance of the service. Unless you frequently perform split/merge/move
operations, we recommend you delete your split-merge cloud service. That saves costs for running or deployed
cloud service instances. You can re-deploy and start your readily runnable configuration whenever you need to
perform split or merge operations.
Monitoring
Status tables
The split-merge Service provides the RequestStatus table in the metadata store database for monitoring of
completed and ongoing requests. The table lists a row for each split-merge request that has been submitted to
this instance of the split-merge service. It gives the following information for each request:
Timestamp: The time and date when the request was started.
OperationId: A GUID that uniquely identifies the request. This request can also be used to cancel the
operation while it is still ongoing.
Status: The current state of the request. For ongoing requests, it also lists the current phase in which the
request is.
CancelRequest: A flag that indicates whether the request has been cancelled.
Progress: A percentage estimate of completion for the operation. A value of 50 indicates that the operation
is approximately 50% complete.
Details: An XML value that provides a more detailed progress report. The progress report is periodically
updated as sets of rows are copied from source to target. In case of failures or exceptions, this column also
includes more detailed information about the failure.
Azure Diagnostics
The split-merge service uses Azure Diagnostics based on Azure SDK 2.5 for monitoring and diagnostics. You
control the diagnostics configuration as explained here: Enabling Diagnostics in Azure Cloud Services and
Virtual Machines. The download package includes two diagnostics configurations - one for the web role and one
for the worker role. These diagnostics configurations for the service follow the guidance from Cloud Service
Fundamentals in Microsoft Azure. It includes the definitions to log Performance Counters, IIS logs, Windows
Event Logs, and split-merge application event logs.
Deploy Diagnostics
To enable monitoring and diagnostics using the diagnostic configuration for the web and worker roles provided
by the NuGet package, run the following commands using Azure PowerShell:
$storage_name = "<YourAzureStorageAccount>"
$key = "<YourAzureStorageAccountKey"
$config_path = "<YourFilePath>\SplitMergeWebContent.diagnostics.xml"
$service_name = "<YourCloudServiceName>"
$config_path = "<YourFilePath>\SplitMergeWorkerContent.diagnostics.xml"
$service_name = "<YourCloudServiceName>"
You can find more information on how to configure and deploy diagnostics settings here: Enabling Diagnostics
in Azure Cloud Services and Virtual Machines.
Retrieve diagnostics
You can easily access your diagnostics from the Visual Studio Server Explorer in the Azure part of the Server
Explorer tree. Open a Visual Studio instance, and in the menu bar click View, and Server Explorer. Click the
Azure icon to connect to your Azure subscription. Then navigate to Azure -> Storage -> -> Tables ->
WADLogsTable. For more information, see Browsing Storage Resources with Server Explorer.
The WADLogsTable highlighted in the figure above contains the detailed events from the split-merge service’s
application log. Note that the default configuration of the downloaded package is geared towards a production
deployment. Therefore the interval at which logs and counters are pulled from the service instances is large (5
minutes). For test and development, lower the interval by adjusting the diagnostics settings of the web or the
worker role to your needs. Right-click on the role in the Visual Studio Server Explorer (see above) and then
adjust the Transfer Period in the dialog for the Diagnostics configuration settings:
Performance
In general, better performance is to be expected from the higher, more performant service tiers in Azure SQL
Database. Higher IO, CPU and memory allocations for the higher service tiers benefit the bulk copy and delete
operations that the split-merge service uses. For that reason, increase the service tier just for those databases
for a defined, limited period of time.
The service also performs validation queries as part of its normal operations. These validation queries check for
unexpected presence of data in the target range and ensure that any split/merge/move operation starts from a
consistent state. These queries all work over sharding key ranges defined by the scope of the operation and the
batch size provided as part of the request definition. These queries perform best when an index is present that
has the sharding key as the leading column.
In addition, a uniqueness property with the sharding key as the leading column will allow the service to use an
optimized approach that limits resource consumption in terms of log space and memory. This uniqueness
property is required to move large data sizes (typically above 1GB ).
How to upgrade
1. Follow the steps in Deploy a split-merge service.
2. Change your cloud service configuration file for your split-merge deployment to reflect the new
configuration parameters. A new required parameter is the information about the certificate used for
encryption. An easy way to do this is to compare the new configuration template file from the download
against your existing configuration. Make sure you add the settings for
“DataEncryptionPrimaryCertificateThumbprint” and “DataEncryptionPrimary” for both the web and the
worker role.
3. Before deploying the update to Azure, ensure that all currently running split-merge operations have finished.
You can easily do this by querying the RequestStatus and PendingWorkflows tables in the split-merge
metadata database for ongoing requests.
4. Update your existing cloud service deployment for split-merge in your Azure subscription with the new
package and your updated service configuration file.
You do not need to provision a new metadata database for split-merge to upgrade. The new version will
automatically upgrade your existing metadata database to the new version.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Using elastic database client library with Dapper
5/23/2018 • 8 minutes to read • Edit Online
This document is for developers that rely on Dapper to build applications, but also want to embrace elastic
database tooling to create applications that implement sharding to scale out their data tier. This document
illustrates the changes in Dapper-based applications that are necessary to integrate with elastic database tools. Our
focus is on composing the elastic database shard management and data-dependent routing with Dapper.
Sample Code: Elastic database tools for Azure SQL Database - Dapper integration.
Integrating Dapper and DapperExtensions with the elastic database client library for Azure SQL Database is
easy. Your applications can use data-dependent routing by changing the creation and opening of new
SqlConnection objects to use the OpenConnectionForKey call from the client library. This limits changes in your
application to only where new connections are created and opened.
Dapper overview
Dapper is an object-relational mapper. It maps .NET objects from your application to a relational database (and
vice versa). The first part of the sample code illustrates how you can integrate the elastic database client library
with Dapper-based applications. The second part of the sample code illustrates how to integrate when using both
Dapper and DapperExtensions.
The mapper functionality in Dapper provides extension methods on database connections that simplify submitting
T-SQL statements for execution or querying the database. For instance, Dapper makes it easy to map between
your .NET objects and the parameters of SQL statements for Execute calls, or to consume the results of your SQL
queries into .NET objects using Query calls from Dapper.
When using DapperExtensions, you no longer need to provide the SQL statements. Extensions methods such as
GetList or Insert over the database connection create the SQL statements behind the scenes.
Another benefit of Dapper and also DapperExtensions is that the application controls the creation of the database
connection. This helps interact with the elastic database client library which brokers database connections based on
the mapping of shardlets to databases.
To get the Dapper assemblies, see Dapper dot net. For the Dapper extensions, see DapperExtensions.
Technical Guidance
Data-dependent routing with Dapper
With Dapper, the application is typically responsible for creating and opening the connections to the underlying
database. Given a type T by the application, Dapper returns query results as .NET collections of type T. Dapper
performs the mapping from the T-SQL result rows to the objects of type T. Similarly, Dapper maps .NET objects
into SQL values or parameters for data manipulation language (DML ) statements. Dapper offers this functionality
via extension methods on the regular SqlConnection object from the ADO .NET SQL Client libraries. The SQL
connection returned by the Elastic Scale APIs for DDR are also regular SqlConnection objects. This allows us to
directly use Dapper extensions over the type returned by the client library’s DDR API, as it is also a simple SQL
Client connection.
These observations make it straightforward to use connections brokered by the elastic database client library for
Dapper.
This code example (from the accompanying sample) illustrates the approach where the sharding key is provided by
the application to the library to broker the connection to the right shard.
The call to the OpenConnectionForKey API replaces the default creation and opening of a SQL Client connection.
The OpenConnectionForKey call takes the arguments that are required for data-dependent routing:
The shard map to access the data-dependent routing interfaces
The sharding key to identify the shardlet
The credentials (user name and password) to connect to the shard
The shard map object creates a connection to the shard that holds the shardlet for the given sharding key. The
elastic database client APIs also tag the connection to implement its consistency guarantees. Since the call to
OpenConnectionForKey returns a regular SQL Client connection object, the subsequent call to the Execute
extension method from Dapper follows the standard Dapper practice.
Queries work very much the same way - you first open the connection using OpenConnectionForKey from the
client API. Then you use the regular Dapper extension methods to map the results of your SQL query into .NET
objects:
Note that the using block with the DDR connection scopes all database operations within the block to the one
shard where tenantId1 is kept. The query only returns blogs stored on the current shard, but not the ones stored
on any other shards.
SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() =>
{
using (SqlConnection sqlconn =
shardingLayer.ShardMap.OpenConnectionForKey(tenantId2, connStrBldr.ConnectionString,
ConnectionOptions.Validate))
{
var blog = new Blog { Name = name2 };
sqlconn.Insert(blog);
}
});
Limitations
The approaches outlined in this document entail a couple of limitations:
The sample code for this document does not demonstrate how to manage schema across shards.
Given a request, we assume that all its database processing is contained within a single shard as identified by
the sharding key provided by the request. However, this assumption does not always hold, for example, when it
is not possible to make a sharding key available. To address this, the elastic database client library includes the
MultiShardQuery class. The class implements a connection abstraction for querying over several shards. Using
MultiShardQuery in combination with Dapper is beyond the scope of this document.
Conclusion
Applications using Dapper and DapperExtensions can easily benefit from elastic database tools for Azure SQL
Database. Through the steps outlined in this document, those applications can use the tool's capability for data-
dependent routing by changing the creation and opening of new SqlConnection objects to use the
OpenConnectionForKey call of the elastic database client library. This limits the application changes required to
those places where new connections are created and opened.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Elastic database tools FAQ
5/23/2018 • 2 minutes to read • Edit Online
If I have a single-tenant per shard and no sharding key, how do I populate the sharding key for the schema info?
The schema info object is only used to split merge scenarios. If an application is inherently single-tenant, then it
does not require the Split Merge tool and thus there is no need to populate the schema info object.
I’ve provisioned a database and I already have a Shard Map Manager, how do I register this new database as a shard?
Please see Adding a shard to an application using the elastic database client library.
How much do elastic database tools cost?
Using the elastic database client library does not incur any costs. Costs accrue only for the Azure SQL databases
that you use for shards and the Shard Map Manager, as well as the web/worker roles you provision for the Split
Merge tool.
Why are my credentials not working when I add a shard from a different server?
Do not use credentials in the form of “User ID=username@servername”, instead simply use “User ID =
username”. Also, be sure that the “username” login has permissions on the shard.
Do I need to create a Shard Map Manager and populate shards every time I start my applications?
No—the creation of the Shard Map Manager (for example,
ShardMapManagerFactory.CreateSqlShardMapManager) is a one-time operation. Your application should
use the call ShardMapManagerFactory.TryGetSqlShardMapManager() at application start-up time. There
should only one such call per application domain.
I have questions about using elastic database tools, how do I get them answered?
Please reach out to us on the Azure SQL Database forum.
When I get a database connection using a sharding key, I can still query data for other sharding keys on the same shard. Is this by
design?
The Elastic Scale APIs give you a connection to the correct database for your sharding key, but do not provide
sharding key filtering. Add WHERE clauses to your query to restrict the scope to the provided sharding key, if
necessary.
Can I use a different Azure Database edition for each shard in my shard set?
Yes, a shard is an individual database, and thus one shard could be a Premium edition while another be a Standard
edition. Further, the edition of a shard can scale up or down multiple times during the lifetime of the shard.
Does the Split Merge tool provision (or delete) a database during a split or merge operation?
No. For split operations, the target database must exist with the appropriate schema and be registered with the
Shard Map Manager. For merge operations, you must delete the shard from the shard map manager and then
delete the database.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Get started with Elastic Database Tools
8/28/2018 • 4 minutes to read • Edit Online
This document introduces you to the developer experience for the elastic database client library by helping
you run a sample app. The sample app creates a simple sharded application and explores key capabilities of
the Elastic Database Tools feature of Azure SQL Database. It focuses on use cases for shard map
management, data-dependent routing, and multi-shard querying. The client library is available for .NET as
well as Java.
mvn install
4. To start the sample project, in the ./sample directory, run the following command:
5. To learn more about the client library capabilities, experiment with the various options. Feel free to
explore the code to learn about the sample app implementation.
Congratulations! You have successfully built and run your first sharded application by using Elastic Database
Tools on Azure SQL Database. Use Visual Studio or SQL Server Management Studio to connect to your
SQL database and take a quick look at the shards that the sample created. You will notice new sample shard
databases and a shard map manager database that the sample has created.
To add the client library to your own Maven project, add the following dependency in your POM file:
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>elastic-db-tools</artifactId>
<version>1.0.0</version>
</dependency>
Congratulations! You have successfully built and run your first sharded application by using Elastic Database
Tools on SQL Database. Use Visual Studio or SQL Server Management Studio to connect to your SQL
database and take a quick look at the shards that the sample created. You will notice new sample shard
databases and a shard map manager database that the sample has created.
IMPORTANT
We recommend that you always use the latest version of Management Studio so that you stay synchronized with
updates to Azure and SQL Database. Update SQL Server Management Studio.
Next steps
For more information about Elastic Database Tools, see the following articles:
Code samples:
Elastic Database Tools (.NET, Java)
Elastic Database Tools for Azure SQL - Entity Framework Integration
Shard Elasticity on Script Center
Blog: Elastic Scale announcement
Channel 9: Elastic Scale overview video
Discussion forum: Azure SQL Database forum
To measure performance: Performance counters for shard map manager
Report across scaled-out cloud databases (preview)
5/23/2018 • 4 minutes to read • Edit Online
You can create reports from multiple Azure SQL databases from a single connection point using an elastic query.
The databases must be horizontally partitioned (also known as "sharded").
If you have an existing database, see Migrating existing databases to scaled-out databases.
To understand the SQL objects needed to query, see Query across horizontally partitioned databases.
Prerequisites
Download and run the Getting started with Elastic Database tools sample.
2. In the command window, type "1" and press Enter. This creates the shard map manager, and adds two shards
to the server. Then type "3" and press Enter; repeat the action four times. This inserts sample data rows in your
shards.
3. The Azure portal should show three new databases in your server:
At this point, cross-database queries are supported through the Elastic Database client libraries. For
example, use option 4 in the command window. The results from a multi-shard query are always a UNION
ALL of the results from all shards.
In the next section, we create a sample database endpoint that supports richer querying of the data across
shards.
"username" and "password" should be the same as login information used in step 6 of Download and run
the sample app in Getting started with elastic database tools.
External data sources
To create an external data source, execute the following command on the ElasticDBQuery database:
"CustomerIDShardMap" is the name of the shard map, if you created the shard map and shard map manager
using the elastic database tools sample. However, if you used your custom setup for this sample, then it should be
the shard map name you chose in your application.
External tables
Create an external table that matches the Customers table on the shards by executing the following command on
ElasticDBQuery database:
You will notice that the query aggregates results from all the shards and gives the following output:
4. In the Data Connection Wizard type the server name and login credentials. Then click Next.
5. In the dialog box Select the database that contains the data you want, select the ElasticDBQuery
database.
6. Select the Customers table in the list view and click Next. Then click Finish.
7. In the Import Data form, under Select how you want to view this data in your workbook, select Table
and click OK.
All the rows from Customers table, stored in different shards populate the Excel sheet.
You can now use Excel’s powerful data visualization functions. You can use the connection string with your server
name, database name and credentials to connect your BI and data integration tools to the elastic query database.
Make sure that SQL Server is supported as a data source for your tool. You can refer to the elastic query database
and external tables just like any other SQL Server database and SQL Server tables that you would connect to with
your tool.
Cost
There is no additional charge for using the Elastic Database Query feature.
For pricing information see SQL Database Pricing Details.
Next steps
For an overview of elastic query, see Elastic query overview .
For a vertical partitioning tutorial, see Getting started with cross-database query (vertical partitioning) .
For syntax and sample queries for vertically partitioned data, see Querying vertically partitioned data)
For syntax and sample queries for horizontally partitioned data, see Querying horizontally partitioned data)
See sp_execute _remote for a stored procedure that executes a Transact-SQL statement on a single remote
Azure SQL Database or set of databases serving as shards in a horizontal partitioning scheme.
Deploy a split-merge service
9/11/2018 • 12 minutes to read • Edit Online
The split-merge tool lets you move data between sharded databases. See Moving data between scaled-out cloud
databases
Prerequisites
1. Create an Azure SQL DB database that will be used as the split-merge status database. Go to the Azure portal.
Create a new SQL Database. Give the database a name and create a new administrator and password. Be
sure to record the name and password for later use.
2. Ensure that your Azure SQL DB server allows Azure Services to connect to it. In the portal, in the Firewall
Settings, ensure the Allow access to Azure Services setting is set to On. Click the "save" icon.
3. Create an Azure Storage account for diagnostics output.
4. Create an Azure Cloud Service for your Split-Merge service.
IMPORTANT
At this time, the status database must use the Latin collation (SQL_Latin1_General_CP1_CI_AS). For more
information, see Windows Collation Name (Transact-SQL).
With Azure SQL DB, the connection string typically is of the form:
Server=myservername.database.windows.net; Database=mydatabasename;User ID=myuserID;
Password=mypassword; Encrypt=True; Connection Timeout=30
4. Enter this connection string in the cscfg file in both the SplitMergeWeb and SplitMergeWorker role
sections in the ElasticScaleMetadata setting.
5. For the SplitMergeWorker role, enter a valid connection string to Azure storage for the
WorkerRoleSynchronizationStorageAccountConnectionString setting.
Configure security
For detailed instructions to configure the security of the service, refer to the Split-Merge security configuration.
For the purposes of a simple test deployment for this tutorial, a minimal set of configuration steps will be
performed to get the service up and running. These steps enable only the one machine/account executing them to
communicate with the service.
Create a self-signed certificate
Create a new directory and from this directory execute the following command using a Developer Command
Prompt for Visual Studio window:
makecert ^
-n "CN=*.cloudapp.net" ^
-r -cy end -sky exchange -eku "1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2" ^
-a sha1 -len 2048 ^
-sr currentuser -ss root ^
-sv MyCert.pvk MyCert.cer
You are asked for a password to protect the private key. Enter a strong password and confirm it. You are then
prompted for the password to be used once more after that. Click Yes at the end to import it to the Trusted
Certification Authorities Root store.
Create a PFX file
Execute the following command from the same window where makecert was executed; use the same password
that you used to create the certificate:
Please note that for production deployments separate certificates should be used for the CA, for encryption, the
Server certificate and client certificates. For detailed instructions on this, see Security Configuration.
Ensure that the server name does not begin with https://.
Ensure that your Azure SQL DB server allows Azure Services to connect to it. To do this, open your database in
the portal and ensure that the Allow access to Azure Services setting is set to On**.
2. Polls the web frontend for the split request status and
waits until the request completes.
A SQL authentication login with read/write access to the DBs will be needed for the Split-Merge service to
move data and update the shard map. Since the Split-Merge Service runs in the cloud, it does not currently
support Integrated Authentication.
Make sure the Azure SQL server is configured to allow access from the IP address of the machine running
these scripts. You can find this setting under the Azure SQL server / configuration / allowed IP addresses.
3. Execute the SetupSampleSplitMergeEnvironment.ps1 script to create the sample environment.
Running this script will wipe out any existing shard map management data structures on the shard map
manager database and the shards. It may be useful to rerun the script if you wish to re-initialize the shard
map or shards.
Sample command line:
.\SetupSampleSplitMergeEnvironment.ps1
-UserName 'mysqluser'
-Password 'MySqlPassw0rd'
-ShardMapManagerServerName 'abcdefghij.database.windows.net'
4. Execute the Getmappings.ps1 script to view the mappings that currently exist in the sample environment.
.\GetMappings.ps1
-UserName 'mysqluser'
-Password 'MySqlPassw0rd'
-ShardMapManagerServerName 'abcdefghij.database.windows.net'
5. Execute the ExecuteSampleSplitMerge.ps1 script to execute a split operation (moving half the data on the
first shard to the second shard) and then a merge operation (moving the data back onto the first shard). If
you configured SSL and left the http endpoint disabled, ensure that you use the https:// endpoint instead.
Sample command line:
.\ExecuteSampleSplitMerge.ps1
-UserName 'mysqluser'
-Password 'MySqlPassw0rd'
-ShardMapManagerServerName 'abcdefghij.database.windows.net'
-SplitMergeServiceEndpoint 'https://mysplitmergeservice.cloudapp.net'
-CertificateThumbprint '0123456789abcdef0123456789abcdef01234567'
If you receive the below error, it is most likely a problem with your Web endpoint’s certificate. Try
connecting to the Web endpoint with your favorite Web browser and check if there is a certificate error.
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for
the SSL/TLSsecure channel.
6. Experiment with other data types! All of these scripts take an optional -ShardKeyType parameter that allows
you to specify the key type. The default is Int32, but you can also specify Int64, Guid, or Binary.
Create requests
The service can be used either by using the web UI or by importing and using the SplitMerge.psm1 PowerShell
module which will submit your requests through the web role.
The service can move data in both sharded tables and reference tables. A sharded table has a sharding key
column and has different row data on each shard. A reference table is not sharded so it contains the same row
data on every shard. Reference tables are useful for data that does not change often and is used to JOIN with
sharded tables in queries.
In order to perform a split-merge operation, you must declare the sharded tables and reference tables that you
want to have moved. This is accomplished with the SchemaInfo API. This API is in the
Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.Schema namespace.
1. For each sharded table, create a ShardedTableInfo object describing the table’s parent schema name
(optional, defaults to “dbo”), the table name, and the column name in that table that contains the sharding key.
2. For each reference table, create a ReferenceTableInfo object describing the table’s parent schema name
(optional, defaults to “dbo”) and the table name.
3. Add the above TableInfo objects to a new SchemaInfo object.
4. Get a reference to a ShardMapManager object, and call GetSchemaInfoCollection.
5. Add the SchemaInfo to the SchemaInfoCollection, providing the shard map name.
An example of this can be seen in the SetupSampleSplitMergeEnvironment.ps1 script.
The Split-Merge service does not create the target database (or schema for any tables in the database) for you.
They must be pre-created before sending a request to the service.
Troubleshooting
You may see the below message when running the sample powershell scripts:
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the
SSL/TLS secure channel.
This error means that your SSL certificate is not configured correctly. Please follow the instructions in section
'Connecting with a web browser'.
If you cannot submit requests you may see this:
In this case, check your configuration file, in particular the setting for
WorkerRoleSynchronizationStorageAccountConnectionString. This error typically indicates that the worker
role could not successfully initialize the metadata database on first use.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Split-merge security configuration
5/23/2018 • 12 minutes to read • Edit Online
To use the Split/Merge service, you must correctly configure security. The service is part of the Elastic Scale
feature of Microsoft Azure SQL Database. For more information, see Elastic Scale Split and Merge Service
Tutorial.
Configuring certificates
Certificates are configured in two ways.
1. To Configure the SSL Certificate
2. To Configure Client Certificates
To obtain certificates
Certificates can be obtained from public Certificate Authorities (CAs) or from the Windows Certificate Service.
These are the preferred methods to obtain certificates.
If those options are not available, you can generate self-signed certificates.
%ProgramFiles(x86)%\Windows Kits\x.y\bin\x86
Get the WDK from Windows 8.1: Download kits and tools
Allowed IP addresses
Access to the service endpoints can be restricted to specific ranges of IP addresses.
<EndpointAcls>
<EndpointAcl role="SplitMergeWeb" endPoint="HttpIn" accessControl="DenyAll" />
<EndpointAcl role="SplitMergeWeb" endPoint="HttpsIn" accessControl="AllowAll" />
</EndpointAcls>
The rules in an access control group are configured in a section of the service configuration file.
The format is explained in Network Access Control Lists documentation. For example, to allow only IPs in the
range 100.100.0.0 to 100.100.255.255 to access the HTTPS endpoint, the rules would look like this:
<AccessControl name="Retricted">
<Rule action="permit" description="Some" order="1" remoteSubnet="100.100.0.0/16"/>
<Rule action="deny" description="None" order="2" remoteSubnet="0.0.0.0/0" />
</AccessControl>
<EndpointAcls>
<EndpointAcl role="SplitMergeWeb" endPoint="HttpsIn" accessControl="Restricted" />
Refer to the documentation for Dynamic IP Security in IIS for other supported values.
makecert ^
-n "CN=myservice.cloudapp.net" ^
-e MM/DD/YYYY ^
-r -cy end -sky exchange -eku "1.3.6.1.5.5.7.3.1" ^
-a sha1 -len 2048 ^
-sv MySSL.pvk MySSL.cer
To customize:
-n with the service URL. Wildcards ("CN=*.cloudapp.net") and alternative names
("CN=myservice1.cloudapp.net, CN=myservice2.cloudapp.net") are supported.
-e with the certificate expiration date Create a strong password and specify it when prompted.
Then, copy the same thumbprint as the SSL certificate in the CA certificate setting:
makecert ^
-n "CN=MyCA" ^
-e MM/DD/YYYY ^
-r -cy authority -h 1 ^
-a sha1 -len 2048 ^
-sr localmachine -ss my ^
MyCA.cer
To customize it
-e with the certification expiration date
Update the value of the following setting with the same thumbprint:
Customizing:
-n with an ID for to the client that will be authenticated with this certificate
-e with the certificate expiration date
MyID.pvk and MyID.cer with unique filenames for this client certificate
This command will prompt for a password to be created and then used once. Use a strong password.
Customizing:
MyID.pvk and MyID.cer with the filename for the client certificate
Customizing:
MyID.pvk and MyID.cer with the filename for the encryption certificate
Find certificate
Follow these steps:
1. Run mmc.exe.
2. File -> Add/Remove Snap-in…
3. Select Certificates.
4. Click Add.
5. Choose the certificate store location.
6. Click Finish.
7. Click OK.
8. Expand Certificates.
9. Expand the certificate store node.
10. Expand the Certificate child node.
11. Select a certificate in the list.
Export certificate
In the Certificate Export Wizard:
1. Click Next.
2. Select Yes, then Export the private key.
3. Click Next.
4. Select the desired output file format.
5. Check the desired options.
6. Check Password.
7. Enter a strong password and confirm it.
8. Click Next.
9. Type or browse a filename where to store the certificate (use a .PFX extension).
10. Click Next.
11. Click Finish.
12. Click OK.
Import certificate
In the Certificate Import Wizard:
1. Select the store location.
Select Current User if only processes running under current user will access the service
Select Local Machine if other processes in this computer will access the service
2. Click Next.
3. If importing from a file, confirm the file path.
4. If importing a .PFX file:
a. Enter the password protecting the private key
b. Select import options
5. Select "Place" certificates in the following store
6. Click Browse.
7. Select the desired store.
8. Click Finish.
If the Trusted Root Certification Authority store was chosen, click Yes.
9. Click OK on all dialog windows.
Upload certificate
In the Azure portal
1. Select Cloud Services.
2. Select the cloud service.
3. On the top menu, click Certificates.
4. On the bottom bar, click Upload.
5. Select the certificate file.
6. If it is a .PFX file, enter the password for the private key.
7. Once completed, copy the certificate thumbprint from the new entry in the list.
Credentials stored in this database are encrypted. However, as a best practice, ensure that both web and worker
roles of your service deployments are kept up to date and secure as they both have access to the metadata
database and the certificate used for encryption and decryption of stored credentials.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Adding a shard using Elastic Database tools
5/23/2018 • 2 minutes to read • Edit Online
// sm is a RangeShardMap object.
// Add a new shard to hold the range being added.
Shard shard2 = null;
For the .NET version, you can also use PowerShell as an alternative to create a new Shard Map Manager. An
example is available here.
Important: Use this technique only if you are certain that the range for the updated mapping is empty. The
preceding methods do not check data for the range being moved, so it is best to include checks in your code. If
rows exist in the range being moved, the actual data distribution will not match the updated shard map. Use the
split-merge tool to perform the operation instead in these cases.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Using the RecoveryManager class to fix shard map
problems
9/12/2018 • 8 minutes to read • Edit Online
The RecoveryManager class provides ADO.Net applications the ability to easily detect and correct any
inconsistencies between the global shard map (GSM ) and the local shard map (LSM ) in a sharded database
environment.
The GSM and LSM track the mapping of each database in a sharded environment. Occasionally, a break occurs
between the GSM and the LSM. In that case, use the RecoveryManager class to detect and repair the break.
The RecoveryManager class is part of the Elastic Database client library.
For term definitions, see Elastic Database tools glossary. To understand how the ShardMapManager is used to
manage data in a sharded solution, see Shard map management.
In this example, the RecoveryManager is initialized from the ShardMapManager. The ShardMapManager
containing a ShardMap is also already initialized.
Since this application code manipulates the shard map itself, the credentials used in the factory method (in the
preceding example, smmConnectionString) should be credentials that have read-write permissions on the GSM
database referenced by the connection string. These credentials are typically different from credentials used to
open connections for data-dependent routing. For more information, see Using credentials in the elastic database
client.
rm.DetachShard(s.Location, customerMap);
The shard map reflects the shard location in the GSM before the deletion of the shard. Because the shard was
deleted, it is assumed this was intentional, and the sharding key range is no longer in use. If not, you can execute
point-in time restore. to recover the shard from an earlier point-in-time. (In that case, review the following section
to detect shard inconsistencies.) To recover, see Point in time recovery.
Since it is assumed the database deletion was intentional, the final administrative cleanup action is to delete the
entry to the shard in the shard map manager. This prevents the application from inadvertently writing information
to a range that is not expected.
rm.DetectMappingDifferences(location, shardMapName);
The RecoveryToken parameter enumerates the differences in the mappings between the GSM and the LSM for
the specific shard.
The MappingDifferenceResolution enumeration is used to indicate the method for resolving the difference
between the shard mappings.
MappingDifferenceResolution.KeepShardMapping is recommended that when the LSM contains the
accurate mapping and therefore the mapping in the shard should be used. This is typically the case if there is a
failover: the shard now resides on a new server. Since the shard must first be removed from the GSM (using
the RecoveryManager.DetachShard method), a mapping no longer exists on the GSM. Therefore, the LSM
must be used to re-establish the shard mapping.
rm.AttachShard(location, shardMapName)
The location parameter is the server name and database name, of the shard being attached.
The shardMapName parameter is the shard map name. This is only required when multiple shard maps are
managed by the same shard map manager. Optional.
This example adds a shard to the shard map that has been recently restored from an earlier point-in time. Since
the shard (namely the mapping for the shard in the LSM ) has been restored, it is potentially inconsistent with the
shard entry in the GSM. Outside of this example code, the shard was restored and renamed to the original name
of the database. Since it was restored, it is assumed the mapping in the LSM is the trusted mapping.
rm.AttachShard(s.Location, customerMap);
var gs = rm.DetectMappingDifferences(s.Location);
foreach (RecoveryToken g in gs)
{
rm.ResolveMappingDifferences(g, MappingDifferenceResolution.KeepShardMapping);
}
Best practices
Geo-failover and recovery are operations typically managed by a cloud administrator of the application
intentionally utilizing one of Azure SQL databases business continuity features. Business continuity planning
requires processes, procedures, and measures to ensure that business operations can continue without
interruption. The methods available as part of the RecoveryManager class should be used within this work flow to
ensure the GSM and LSM are kept up-to-date based on the recovery action taken. There are five basic steps to
properly ensuring the GSM and LSM reflect the accurate information after a failover event. The application code
to execute these steps can be integrated into existing tools and workflow.
1. Retrieve the RecoveryManager from the ShardMapManager.
2. Detach the old shard from the shard map.
3. Attach the new shard to the shard map, including the new shard location.
4. Detect inconsistencies in the mapping between the GSM and LSM.
5. Resolve differences between the GSM and the LSM, trusting the LSM.
This example performs the following steps:
1. Removes shards from the Shard Map that reflect shard locations before the failover event.
2. Attaches shards to the Shard Map reflecting the new shard locations (the parameter
"Configuration.SecondaryServer" is the new server name but the same database name).
3. Retrieves the recovery tokens by detecting mapping differences between the GSM and the LSM for each shard.
4. Resolves the inconsistencies by trusting the mapping from the LSM of each shard.
{
ShardLocation slNew = new ShardLocation(Configuration.SecondaryServer, s.Location.Database);
rm.DetachShard(s.Location);
rm.AttachShard(slNew);
var gs = rm.DetectMappingDifferences(slNew);
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Migrate existing databases to scale out
5/23/2018 • 4 minutes to read • Edit Online
Easily manage your existing scaled-out sharded databases using Azure SQL Database database tools (such as the
Elastic Database client library ). First convert an existing set of databases to use the shard map manager.
Overview
To migrate an existing sharded database:
1. Prepare the shard map manager database.
2. Create the shard map.
3. Prepare the individual shards.
4. Add mappings to the shard map.
These techniques can be implemented using either the .NET Framework client library, or the PowerShell scripts
found at Azure SQL DB - Elastic Database tools scripts. The examples here use the PowerShell scripts.
For more information about the ShardMapManager, see Shard map management. For an overview of the elastic
database tools, see Elastic Database features overview.
The multi-tenant model assigns several tenants to a single database (and you can distribute groups of tenants
across multiple databases). Use this model when you expect each tenant to have small data needs. In this model,
assign a range of tenants to a database using range mapping.
Or you can implement a multi-tenant database model using a list mapping to assign multiple tenants to a single
database. For example, DB1 is used to store information about tenant ID 1 and 5, and DB2 stores data for tenant
7 and tenant 10.
Based on your choice, choose one of these options:
Option 1: create a shard map for a list mapping
Create a shard map using the ShardMapManager object.
Add-Shard
-ShardMap $ShardMap
-SqlServerName '<shard_server_name>'
-SqlDatabaseName '<shard_database_name>'
# The $ShardMap is the shard map created in step 2.
Step 4 option 3: map the data for multiple tenants on a single database
For each tenant, run the Add-ListMapping (option 1).
Summary
Once you have completed the setup, you can begin to use the Elastic Database client library. You can also use
data-dependent routing and multi-shard query.
Next steps
Get the PowerShell scripts from Azure SQL DB -Elastic Database tools scripts.
The tools are also on GitHub: Azure/elastic-db-tools.
Use the split-merge tool to move data to or from a multi-tenant model to a single tenant model. See Split merge
tool.
Additional resources
For information on common data architecture patterns of multi-tenant software-as-a-service (SaaS ) database
applications, see Design Patterns for Multi-tenant SaaS Applications with Azure SQL Database.
Questions and Feature Requests
For questions, use the SQL Database forum and for feature requests, add them to the SQL Database feedback
forum.
Performance counters for shard map manager
5/23/2018 • 2 minutes to read • Edit Online
You can capture the performance of a shard map manager, especially when using data dependent routing.
Counters are created with methods of the Microsoft.Azure.SqlDatabase.ElasticScale.Client class.
Counters are used to track the performance of data dependent routing operations. These counters are accessible in
the Performance Monitor, under the "Elastic Database: Shard Management" category.
For the latest version: Go to Microsoft.Azure.SqlDatabase.ElasticScale.Client. See also Upgrade an app to use the
latest elastic database client library.
Prerequisites
To create the performance category and counters, the user must be a part of the local Administrators group on
the machine hosting the application.
To create a performance counter instance and update the counters, the user must be a member of either the
Administrators or Performance Monitor Users group.
ShardMapManagerFactory.CreatePerformanceCategoryAndCounters()
You can also use this PowerShell script to execute the method. The method creates the following performance
counters:
Cached mappings: Number of mappings cached for the shard map.
DDR operations/sec: Rate of data dependent routing operations for the shard map. This counter is updated
when a call to OpenConnectionForKey() results in a successful connection to the destination shard.
Mapping lookup cache hits/sec: Rate of successful cache lookup operations for mappings in the shard map.
Mapping lookup cache misses/sec: Rate of failed cache lookup operations for mappings in the shard map.
Mappings added or updated in cache/sec: Rate at which mappings are being added or updated in cache for
the shard map.
Mappings removed from cache/sec: Rate at which mappings are being removed from cache for the shard
map.
Performance counters are created for each cached shard map per process.
Notes
The following events trigger the creation of the performance counters:
Initialization of the ShardMapManager with eager loading, if the ShardMapManager contains any shard maps.
These include the GetSqlShardMapManager and the TryGetSqlShardMapManager methods.
Successful lookup of a shard map (using GetShardMap(), GetListShardMap() or GetRangeShardMap()).
Successful creation of shard map using CreateShardMap().
The performance counters will be updated by all cache operations performed on the shard map and mappings.
Successful removal of the shard map using DeleteShardMap()reults in deletion of the performance counters
instance.
Best practices
Creation of the performance category and counters should be performed only once before the creation of
ShardMapManager object. Every execution of the command CreatePerformanceCategoryAndCounters() clears
the previous counters (losing data reported by all instances) and creates new ones.
Performance counter instances are created per process. Any application crash or removal of a shard map from
the cache will result in deletion of the performance counters instances.
See also
Elastic Database features overview
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Elastic Database client library with Entity Framework
5/23/2018 • 16 minutes to read • Edit Online
This document shows the changes in an Entity Framework application that are needed to integrate with the Elastic
Database tools. The focus is on composing shard map management and data-dependent routing with the Entity
Framework Code First approach. The Code First - New Database tutorial for EF serves as the running example
throughout this document. The sample code accompanying this document is part of elastic database tools' set of
samples in the Visual Studio Code Samples.
Requirements
When working with both the elastic database client library and Entity Framework APIs, you want to retain the
following properties:
Scale-out: To add or remove databases from the data tier of the sharded application as necessary for the
capacity demands of the application. This means control over the creation and deletion of databases and using
the elastic database shard map manager APIs to manage databases, and mappings of shardlets.
Consistency: The application employs sharding, and uses the data-dependent routing capabilities of the client
library. To avoid corruption or wrong query results, connections are brokered through the shard map manager.
This also retains validation and consistency.
Code First: To retain the convenience of EF’s code first paradigm. In Code First, classes in the application are
mapped transparently to the underlying database structures. The application code interacts with DbSets that
mask most aspects involved in the underlying database processing.
Schema: Entity Framework handles initial database schema creation and subsequent schema evolution
through migrations. By retaining these capabilities, adapting your app is easy as the data evolves.
The following guidance instructs how to satisfy these requirements for Code First applications using elastic
database tools.
// Only static methods are allowed in calls into base class c'tors.
private static DbConnection CreateDDRConnection(
ShardMap shardMap,
T shardingKey,
string connectionStr)
{
// No initialization
Database.SetInitializer<ElasticScaleContext<T>>(null);
// Ask shard map to broker a validated connection for the given key
SqlConnection conn = shardMap.OpenConnectionForKey<T>
(shardingKey, connectionStr, ConnectionOptions.Validate);
return conn;
}
Main points
A new constructor replaces the default constructor in the DbContext subclass
The new constructor takes the arguments that are required for data-dependent routing through elastic
database client library:
the shard map to access the data-dependent routing interfaces,
the sharding key to identify the shardlet,
a connection string with the credentials for the data-dependent routing connection to the shard.
The call to the base class constructor takes a detour into a static method that performs all the steps
necessary for data-dependent routing.
It uses the OpenConnectionForKey call of the elastic database client interfaces on the shard map to
establish an open connection.
The shard map creates the open connection to the shard that holds the shardlet for the given sharding
key.
This open connection is passed back to the base class constructor of DbContext to indicate that this
connection is to be used by EF instead of letting EF create a new connection automatically. This way the
connection has been tagged by the elastic database client API so that it can guarantee consistency under
shard map management operations.
Use the new constructor for your DbContext subclass instead of the default constructor in your code. Here is an
example:
The new constructor opens the connection to the shard that holds the data for the shardlet identified by the value
of tenantid1. The code in the using block stays unchanged to access the DbSet for blogs using EF on the shard
for tenantid1. This changes semantics for the code in the using block such that all database operations are now
scoped to the one shard where tenantid1 is kept. For instance, a LINQ query over the blogs DbSet would only
return blogs stored on the current shard, but not the ones stored on other shards.
Transient faults handling
The Microsoft Patterns & Practices team published the The Transient Fault Handling Application Block. The library
is used with elastic scale client library in combination with EF. However, ensure that any transient exception
returns to a place where you can ensure that the new constructor is being used after a transient fault so that any
new connection attempt is made using the constructors you tweaked. Otherwise, a connection to the correct shard
is not guaranteed, and there are no assurances the connection is maintained as changes to the shard map occur.
The following code sample illustrates how a SQL retry policy can be used around the new DbContext subclass
constructors:
SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() =>
{
using (var db = new ElasticScaleContext<int>(
sharding.ShardMap,
tenantId1,
connStrBldr.ConnectionString))
{
var blog = new Blog { Name = name };
db.Blogs.Add(blog);
db.SaveChanges();
…
}
});
REWRITTEN CONSTRUCTOR
CURRENT CONSTRUCTOR FOR DATA BASE CONSTRUCTOR NOTES
// Enter a new shard - i.e. an empty database - to the shard map, allocate a first tenant to it
// and kick off EF intialization of the database to deploy schema
public void RegisterNewShard(string server, string database, string connStr, int key)
{
// Go into a DbContext to trigger migrations and schema deployment for the new shard.
// This requires an un-opened connection.
using (var db = new ElasticScaleContext<int>(connStrBldr.ConnectionString))
{
// Run a query to engage EF migrations
(from b in db.Blogs
select b).Count();
}
// Register the mapping of the tenant to the shard in the shard map.
// After this step, data-dependent routing on the shard map can be used
this.ShardMap.CreatePointMapping(key, shard);
}
This sample shows the method RegisterNewShard that registers the shard in the shard map, deploys the schema
through EF migrations, and stores a mapping of a sharding key to the shard. It relies on a constructor of the
DbContext subclass (ElasticScaleContext in the sample) that takes a SQL connection string as input. The code
of this constructor is straight-forward, as the following example shows:
// C'tor to deploy schema and migrations to a new shard
protected internal ElasticScaleContext(string connectionString)
: base(SetInitializerForConnection(connectionString))
{
}
// Only static methods are allowed in calls into base class c'tors
private static string SetInitializerForConnection(string connnectionString)
{
// You want existence checks so that the schema can get deployed
Database.SetInitializer<ElasticScaleContext<T>>(
new CreateDatabaseIfNotExists<ElasticScaleContext<T>>());
return connnectionString;
}
One might have used the version of the constructor inherited from the base class. But the code needs to ensure
that the default initializer for EF is used when connecting. Hence the short detour into the static method before
calling into the base class constructor with the connection string. Note that the registration of shards should run in
a different app domain or process to ensure that the initializer settings for EF do not conflict.
Limitations
The approaches outlined in this document entail a couple of limitations:
EF applications that use LocalDb first need to migrate to a regular SQL Server database before using elastic
database client library. Scaling out an application through sharding with Elastic Scale is not possible with
LocalDb. Note that development can still use LocalDb.
Any changes to the application that imply database schema changes need to go through EF migrations on all
shards. The sample code for this document does not demonstrate how to do this. Consider using Update-
Database with a ConnectionString parameter to iterate over all shards; or extract the T-SQL script for the
pending migration using Update-Database with the -Script option and apply the T-SQL script to your shards.
Given a request, it is assumed that all of its database processing is contained within a single shard as identified
by the sharding key provided by the request. However, this assumption does not always hold true. For example,
when it is not possible to make a sharding key available. To address this, the client library provides the
MultiShardQuery class that implements a connection abstraction for querying over several shards. Learning
to use the MultiShardQuery in combination with EF is beyond the scope of this document
Conclusion
Through the steps outlined in this document, EF applications can use the elastic database client library's capability
for data-dependent routing by refactoring constructors of the DbContext subclasses used in the EF application.
This limits the changes required to those places where DbContext classes already exist. In addition, EF
applications can continue to benefit from automatic schema deployment by combining the steps that invoke the
necessary EF migrations with the registration of new shards and mappings in the shard map.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us on
the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Feature comparison: Azure SQL Database versus
SQL Server
9/14/2018 • 6 minutes to read • Edit Online
Azure SQL Database shares a common code base with SQL Server. The features of SQL Server supported by
Azure SQL Database depend on the type of Azure SQL database that you create. With Azure SQL Database,
you can either create a database as part of a managed instance (currently in public preview ) or you can create a
database that is part of Logical server and optionally placed in an Elastic pool.
Microsoft continues to add features to Azure SQL Database. Visit the Service Updates webpage for Azure for
the newest updates using these filters:
Filtered to the SQL Database service.
Filtered to General Availability (GA) announcements for SQL Database features.
Always Encrypted Yes - see Cert store and Key vault Yes - see Cert store and Key vault
Always On Availability Groups High availability is included with every High availability is included with every
database. Disaster recovery is database. Disaster recovery is
discussed in Overview of business discussed in Overview of business
continuity with Azure SQL Database continuity with Azure SQL Database
Attach a database No No
Built-in functions Most - see individual functions Yes - see Stored procedures, functions,
triggers differences
Database mirroring No No
Database snapshots No No
DBCC statements Most - see individual statements Yes - see DBCC differences
DDL statements Most - see individual statements Yes - see T-SQL differences
Extended events Some - see Extended events in SQL Yes - see Extended events differences
Database
Filestream No No
Full-text search Third-party word breakers are not Third-party word breakers are not
supported supported
Functions Most - see individual functions Yes - see Stored procedures, functions,
triggers differences
Geo-replication Yes No
Language elements Most - see individual elements Yes - see T-SQL differences
SUPPORTED IN AZURE SQL
SUPPORTED IN AZURE SQL DATABASE/MANAGED INSTANCE
SQL FEATURE DATABASE/LOGICAL SERVER (PREVIEW)
Linked servers No - see Elastic query Only to SQL Server and SQL Database
Log shipping High availability is included with every High availability is included with every
database. Disaster recovery is database. Disaster recovery is
discussed in Overview of business discussed in Overview of business
continuity with Azure SQL Database continuity with Azure SQL Database
Point in time database restore Yes - see SQL Database recovery Yes - see SQL Database recovery
Polybase No No
Policy-based management No No
Restore database from backup From automated backups only - see From automated backups - see SQL
SQL Database recovery Database recovery and from full
backups - see Backup differences
Semantic search No No
Set statements Most - see individual statements Yes - see T-SQL differences
SQL Server Agent No - see Elastic jobs Yes - see SQL Server Agent differences
SQL Server Analysis Services (SSAS) No -see Azure Analysis Services No - see Azure Analysis Services
SQL Server Auditing No - see SQL Database auditing Yes - see Auditing differences
SQL Server Integration Services (SSIS) Yes, with a managed SSIS in Azure Yes, with a managed SSIS in Azure
Data Factory (ADF) environment, Data Factory (ADF) environment,
where packages are stored in SSISDB where packages are stored in SSISDB
hosted by Azure SQL Database and hosted by Managed Instance and
executed on Azure SSIS Integration executed on Azure SSIS Integration
Runtime (IR), see Create Azure-SSIS IR Runtime (IR), see Create Azure-SSIS IR
in ADF. in ADF.
To compare the SSIS features in SQL To compare the SSIS features in SQL
Database and Managed Instance, see Database and Managed Instance, see
Compare SQL Database and Managed Compare SQL Database and Managed
Instance (Preview). Instance (Preview).
SQL Server Replication Transactional and snapshot replication Yes (Public preview) - Replication with
subscriber only SQL Database Managed Instance
(Public preview)
System stored functions Most - see individual functions Yes - see Stored procedures, functions,
triggers differences
System stored procedures Some - see individual stored Yes - see Stored procedures, functions,
procedures triggers differences
System tables Some - see individual tables Yes - see T-SQL differences
System catalog views Some - see individual views Yes - see T-SQL differences
Temporary tables Local and database-scoped global Local and instance-scoped global
temporary tables temporary tables
Trace flags No No
VNet Partial - see VNET Endpoints Yes, Resource Manager model only
Windows Server Failover Clustering High availability is included with every High availability is included with every
database. Disaster recovery is database. Disaster recovery is
discussed in Overview of business discussed in Overview of business
continuity with Azure SQL Database continuity with Azure SQL Database
Next steps
For information about the Azure SQL Database service, see What is SQL Database?
For information about a Managed Instance, see What is a Managed Instance?.
Monitoring and performance tuning
8/6/2018 • 3 minutes to read • Edit Online
Azure SQL Database is automatically managed and flexible data service where you can easily monitor usage, add
or remove resources (CPU, memory, io), find recommendations that can improve performance of your database,
or let database adapt to your workload and automatically optimize performance.
This article provides overview of monitoring and performance tuning options that are available in Azure SQL
Database.
If your Azure issue is not addressed in this article, visit the Azure forums on MSDN and Stack Overflow. You can
post your issue in these forums, or post to @AzureSupport on Twitter. You also can submit an Azure support
request. To submit a support request, on the Azure support page, select Get support.
IMPORTANT
It is recommended that you always use the latest version of Management Studio to remain synchronized with updates to
Microsoft Azure and SQL Database. Update SQL Server Management Studio.
Next steps
To enable automatic tuning in Azure SQL Database and let automatic tuning feature fully manage your
workload, see Enable automatic tuning.
To use manual tuning, you can review Tuning recommendations in Azure portal and manually apply the ones
that improve performance of your queries.
Change resources that are available in your database by changing Azure SQL Database service tiers
Monitoring database performance in Azure SQL
Database
8/6/2018 • 12 minutes to read • Edit Online
Monitoring the performance of a SQL database in Azure starts with monitoring the resource utilization relative to
the level of database performance you choose. Monitoring helps you determine whether your database has excess
capacity or is having trouble because resources are maxed out, and then decide whether it's time to adjust the
performance level and service tiers of your database in the DTU -based purchasing model or vCore-based
purchasing model. You can monitor your database using graphical tools in the Azure portal or using SQL dynamic
management views.
TIP
Use Azure SQL Intelligent Insights for automatic monitoring of your database performance. Once a performance issue is
detected, a diagnostic log is generated with details and Root Cause Analysis (RCA) of the issue. Performance improvement
recommendation is provided when possible.
NOTE
sys.dm_db_resource_stats returns an empty result set when used in Web and Business edition databases, which are retired.
SELECT
AVG(avg_cpu_percent) AS 'Average CPU use in percent',
MAX(avg_cpu_percent) AS 'Maximum CPU use in percent',
AVG(avg_data_io_percent) AS 'Average data IO in percent',
MAX(avg_data_io_percent) AS 'Maximum data IO in percent',
AVG(avg_log_write_percent) AS 'Average log write use in percent',
MAX(avg_log_write_percent) AS 'Maximum log write use in percent',
AVG(avg_memory_usage_percent) AS 'Average memory use in percent',
MAX(avg_memory_usage_percent) AS 'Maximum memory use in percent'
FROM sys.dm_db_resource_stats;
From the data, this database currently has a peak CPU load of just over 50 percent CPU use relative to the P2
performance level (midday on Tuesday). If CPU is the dominant factor in the application’s resource profile, then
you might decide that P2 is the right performance level to guarantee that the workload always fits. If you expect an
application to grow over time, it's a good idea to have an extra resource buffer so that the application doesn't ever
reach the performance-level limit. If you increase the performance level, you can help avoid customer-visible errors
that might occur when a database doesn't have enough power to process requests effectively, especially in latency-
sensitive environments. An example is a database that supports an application that paints webpages based on the
results of database calls.
Other application types might interpret the same graph differently. For example, if an application tries to process
payroll data each day and has the same chart, this kind of "batch job" model might do fine at a P1 performance
level. The P1 performance level has 100 DTUs compared to 200 DTUs at the P2 performance level. The P1
performance level provides half the performance of the P2 performance level. So, 50 percent of CPU use in P2
equals 100 percent CPU use in P1. If the application does not have timeouts, it might not matter if a job takes 2
hours or 2.5 hours to finish, if it gets done today. An application in this category probably can use a P1
performance level. You can take advantage of the fact that there are periods of time during the day when resource
use is lower, so that any "big peak" might spill over into one of the troughs later in the day. The P1 performance
level might be good for that kind of application (and save money), as long as the jobs can finish on time each day.
Azure SQL Database exposes consumed resource information for each active database in the sys.resource_stats
view of the master database in each server. The data in the table is aggregated for 5-minute intervals. With the
Basic, Standard, and Premium service tiers, the data can take more than 5 minutes to appear in the table, so this
data is more useful for historical analysis rather than near-real-time analysis. Query the sys.resource_stats view to
see the recent history of a database and to validate whether the reservation you chose delivered the performance
you want when needed.
NOTE
You must be connected to the master database of your logical SQL database server to query sys.resource_stats in the
following examples.
This example shows you how the data in this view is exposed:
SELECT TOP 10 *
FROM sys.resource_stats
WHERE database_name = 'resource1'
ORDER BY start_time DESC
The next example shows you different ways that you can use the sys.resource_stats catalog view to get
information about how your SQL database uses resources:
1. To look at the past week’s resource use for the database userdb1, you can run this query:
SELECT *
FROM sys.resource_stats
WHERE database_name = 'userdb1' AND
start_time > DATEADD(day, -7, GETDATE())
ORDER BY start_time DESC;
2. To evaluate how well your workload fits the performance level, you need to drill down into each aspect of
the resource metrics: CPU, reads, writes, number of workers, and number of sessions. Here's a revised
query using sys.resource_stats to report the average and maximum values of these resource metrics:
SELECT
avg(avg_cpu_percent) AS 'Average CPU use in percent',
max(avg_cpu_percent) AS 'Maximum CPU use in percent',
avg(avg_data_io_percent) AS 'Average physical data IO use in percent',
max(avg_data_io_percent) AS 'Maximum physical data IO use in percent',
avg(avg_log_write_percent) AS 'Average log write use in percent',
max(avg_log_write_percent) AS 'Maximum log write use in percent',
avg(max_session_percent) AS 'Average % of sessions',
max(max_session_percent) AS 'Maximum % of sessions',
avg(max_worker_percent) AS 'Average % of workers',
max(max_worker_percent) AS 'Maximum % of workers'
FROM sys.resource_stats
WHERE database_name = 'userdb1' AND start_time > DATEADD(day, -7, GETDATE());
3. With this information about the average and maximum values of each resource metric, you can assess how
well your workload fits into the performance level you chose. Usually, average values from
sys.resource_stats give you a good baseline to use against the target size. It should be your primary
measurement stick. For an example, you might be using the Standard service tier with S2 performance
level. The average use percentages for CPU and IO reads and writes are below 40 percent, the average
number of workers is below 50, and the average number of sessions is below 200. Your workload might fit
into the S1 performance level. It's easy to see whether your database fits in the worker and session limits. To
see whether a database fits into a lower performance level with regards to CPU, reads, and writes, divide the
DTU number of the lower performance level by the DTU number of your current performance level, and
then multiply the result by 100:
S1 DTU / S2 DTU * 100 = 20 / 50 * 100 = 40
The result is the relative performance difference between the two performance levels in percentage. If your
resource use doesn't exceed this amount, your workload might fit into the lower performance level.
However, you need to look at all ranges of resource use values, and determine, by percentage, how often
your database workload would fit into the lower performance level. The following query outputs the fit
percentage per resource dimension, based on the threshold of 40 percent that we calculated in this example:
SELECT
(COUNT(database_name) - SUM(CASE WHEN avg_cpu_percent >= 40 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'CPU Fit Percent'
,(COUNT(database_name) - SUM(CASE WHEN avg_log_write_percent >= 40 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'Log Write Fit Percent'
,(COUNT(database_name) - SUM(CASE WHEN avg_data_io_percent >= 40 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'Physical Data IO Fit Percent'
FROM sys.resource_stats
WHERE database_name = 'userdb1' AND start_time > DATEADD(day, -7, GETDATE());
Based on your database service level objective (SLO ), you can decide whether your workload fits into the
lower performance level. If your database workload SLO is 99.9 percent and the preceding query returns
values greater than 99.9 percent for all three resource dimensions, your workload likely fits into the lower
performance level.
Looking at the fit percentage also gives you insight into whether you should move to the next higher
performance level to meet your SLO. For example, userdb1 shows the following CPU use for the past week:
24.5 100.00
The average CPU is about a quarter of the limit of the performance level, which would fit well into the
performance level of the database. But, the maximum value shows that the database reaches the limit of the
performance level. Do you need to move to the next higher performance level? Look at how many times
your workload reaches 100 percent, and then compare it to your database workload SLO.
SELECT
(COUNT(database_name) - SUM(CASE WHEN avg_cpu_percent >= 100 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'CPU fit percent'
,(COUNT(database_name) - SUM(CASE WHEN avg_log_write_percent >= 100 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'Log write fit percent'
,(COUNT(database_name) - SUM(CASE WHEN avg_data_io_percent >= 100 THEN 1 ELSE 0 END) * 1.0) /
COUNT(database_name) AS 'Physical data IO fit percent'
FROM sys.resource_stats
WHERE database_name = 'userdb1' AND start_time > DATEADD(day, -7, GETDATE());
If this query returns a value less than 99.9 percent for any of the three resource dimensions, consider either
moving to the next higher performance level or use application-tuning techniques to reduce the load on the
SQL database.
4. This exercise also considers your projected workload increase in the future.
For elastic pools, you can monitor individual databases in the pool with the techniques described in this section.
But you can also monitor the pool as a whole. For information, see Monitor and manage an elastic pool.
Maximum concurrent requests
To see the number of concurrent requests, run this Transact-SQL query on your SQL database:
To analyze the workload of an on-premises SQL Server database, modify this query to filter on the specific
database you want to analyze. For example, if you have an on-premises database named MyDatabase, this
Transact-SQL query returns the count of concurrent requests in that database:
This is just a snapshot at a single point in time. To get a better understanding of your workload and concurrent
request requirements, you'll need to collect many samples over time.
Maximum concurrent logins
You can analyze your user and application patterns to get an idea of the frequency of logins. You also can run real-
world loads in a test environment to make sure that you're not hitting this or other limits we discuss in this article.
There isn’t a single query or dynamic management view (DMV ) that can show you concurrent login counts or
history.
If multiple clients use the same connection string, the service authenticates each login. If 10 users simultaneously
connect to a database by using the same username and password, there would be 10 concurrent logins. This limit
applies only to the duration of the login and authentication. If the same 10 users connect to the database
sequentially, the number of concurrent logins would never be greater than 1.
NOTE
Currently, this limit does not apply to databases in elastic pools.
Maximum sessions
To see the number of current active sessions, run this Transact-SQL query on your SQL database:
If you're analyzing an on-premises SQL Server workload, modify the query to focus on a specific database. This
query helps you determine possible session needs for the database if you are considering moving it to Azure SQL
Database.
Again, these queries return a point-in-time count. If you collect multiple samples over time, you’ll have the best
understanding of your session use.
For SQL Database analysis, you can get historical statistics on sessions by querying the sys.resource_stats view
and reviewing the active_session_count column.
Next steps
Automatically tune database indexes and query execution plans using Azure SQL Database automatic tuning.
Monitor database performance automatically using Azure SQL Intelligent Insights. This feature provides
diagnostics information and root cause analysis of performance issues.
Performance recommendations for SQL Database
8/6/2018 • 5 minutes to read • Edit Online
Azure SQL Database learns and adapts with your application. It provides customized recommendations that
enable you to maximize the performance of your SQL databases. SQL Database continuously assesses and
analyzes the usage history of your SQL databases. The recommendations that are provided are based on
database-unique workload patterns and help improve performance.
TIP
Automatic tuning is the recommended method for performance tuning. Intelligent Insights is the recommended method
for monitoring performance.
Fix schema issues recommendations appear when the SQL Database service notices an anomaly in the
number of schema-related SQL errors that are happening on your SQL database. This recommendation typically
appears when your database encounters multiple schema-related errors (invalid column name, invalid object
name, and so on) within an hour.
“Schema issues” are a class of syntax errors in SQL Server. They occur when the definition of the SQL query and
the definition of the database schema aren't aligned. For example, one of the columns that's expected by the
query might be missing in the target table or vice-versa.
The “Fix schema issue” recommendation appears when the Azure SQL Database service notices an anomaly in
the number of schema-related SQL errors that are happening on your SQL database. The following table shows
the errors that are related to schema issues:
201 Procedure or function '' expects parameter '', which was not
supplied.
Next steps
Monitor your recommendations and continue to apply them to refine performance. Database workloads are
dynamic and change continuously. SQL Database Advisor continues to monitor and provide recommendations
that can potentially improve your database's performance.
For more information about automatic tuning of database indexes and query execution plans, see Azure SQL
Database automatic tuning.
For more information about automatically monitoring database performance with automated diagnostics and
root cause analysis of performance issues, see Azure SQL Intelligent Insights.
For more information about how to use performance recommendations in the Azure portal, see Performance
recommendations in the Azure portal.
See Query Performance Insights to learn about and view the performance impact of your top queries.
Automatic tuning in Azure SQL Database
9/12/2018 • 4 minutes to read • Edit Online
Azure SQL Database Automatic tuning provides peak performance and stable workloads through continuous
performance tuning based on AI and machine learning.
Automatic tuning is a fully managed intelligent performance service that uses built-in intelligence to
continuously monitor queries executed on a database, and it automatically improves their performance. This is
achieved through dynamically adapting database to the changing workloads and applying tuning
recommendations. Automatic tuning learns horizontally from all databases on Azure through AI and it
dynamically improves its tuning actions. The longer an Azure SQL Database runs with automatic tuning on, the
better it performs.
Azure SQL Database Automatic tuning might be one of the most important features that you can enable to
provide stable and peak performing database workloads.
Next steps
To enable automatic tuning in Azure SQL Database to manage your workload, see Enable automatic tuning.
To manually review and apply Automatic tuning recommendations, see Find and apply performance
recommendations.
To learn how to use T-SQL to apply and view Automatic tuning recommendations, see Manage automatic
tuning via T-SQL.
To learn about building email notifications for Automatic tuning recommendations, see Email notifications
for automatic tuning.
To learn about built-in intelligence used in Automatic tuning, see Artificial Intelligence tunes Azure SQL
databases.
To learn about how Automatic tuning works in Azure SQL Database and SQL server 2017, see SQL Server
automatic tuning.
Enable automatic tuning
9/13/2018 • 3 minutes to read • Edit Online
Azure SQL Database is an automatically managed data service that constantly monitors your queries and
identifies the action that you can perform to improve performance of your workload. You can review
recommendations and manually apply them, or let Azure SQL Database automatically apply corrective actions -
this is known as automatic tuning mode.
Automatic tuning can be enabled at the server or the database level through the Azure portal, REST API calls and
T-SQL commands.
NOTE
Please note that DROP_INDEX option at this time is not compatible with applications using partition switching and index
hints and should not be enabled in these cases.
Select the automatic tuning options you want to enable and select Apply.
Automatic tuning options on a server are applied to all databases on this server. By default, all databases inherit
configuration from their parent server, but this can be overridden and specified for each database individually.
REST API
Find out more about using REST API to enable Automatic tuning on a server, see SQL Server Automatic tuning
UPDATE and GET HTTP methods.
NOTE
The general recommendation is to manage the automatic tuning configuration at server level so the same configuration
settings can be applied on every database automatically. Configure automatic tuning on an individual database only if you
need that database to have different settings than others inheriting settings from the same server.
Azure portal
To enable automatic tuning on a single database, navigate to the database in Azure portal and select Automatic
tuning.
Individual automatic tuning settings can be separately configured for each database. You can manually configure
an individual automatic tuning option, or specify that an option inherits its settings from the server.
Please note that DROP_INDEX option at this time is not compatible with applications using partition switching
and index hints and should not be enabled in these cases.
Once you have selected your desired configuration, click Apply.
Rest API
Find out more about using REST API to enable Automatic tuning on a single database, see SQL Database
Automatic tuning UPDATE and GET HTTP methods.
T -SQL
To enable automatic tuning on a single database via T-SQL, connect to the database and execute the following
query:
ALTER DATABASE current SET AUTOMATIC_TUNING = AUTO | INHERIT | CUSTOM
Setting automatic tuning to AUTO will apply Azure Defaults. Setting it to INHERIT, automatic tuning
configuration will be inherited from the parent server. Choosing CUSTOM, you will need to manually configure
automatic tuning.
To configure individual automatic tuning options via T-SQL, connect to the database and execute the query such
as this one:
ALTER DATABASE current SET AUTOMATIC_TUNING (FORCE_LAST_GOOD_PLAN = ON, CREATE_INDEX = DEFAULT, DROP_INDEX =
OFF)
Setting the individual tuning option to ON, will override any setting that database inherited and enable the tuning
option. Setting it to OFF, will also override any setting that database inherited and disable the tuning option.
Automatic tuning option, for which DEFAULT is specified, will inherit the configuration from the database level
automatic tuning setting.
Find our more abut T-SQL options to configure Automatic tuning, see ALTER DATABASE SET Options (Transact-
SQL ) for SQL Database logical server.
Next steps
Read the Automatic tuning article to learn more about automatic tuning and how it can help you improve your
performance.
See Performance recommendations for an overview of Azure SQL Database performance recommendations.
See Query Performance Insights to learn about viewing the performance impact of your top queries.
Email notifications for automatic tuning
5/23/2018 • 10 minutes to read • Edit Online
SQL Database tuning recommendations are generated by Azure SQL Database Automatic tuning. This solution
continuously monitors and analyzes workloads of SQL Databases providing customized tuning recommendations
for each individual database related to index creation, index deletion, and optimization of query execution plans.
SQL Database Automatic tuning recommendations can be viewed in the Azure portal, retrieved with REST API
calls, or by using T-SQL and PowerShell commands. This article is based on using a PowerShell script to retrieve
automatic tuning recommendations.
If you have several Azure subscriptions for which you would like to build the same automation, you need to repeat
this process for your other subscriptions.
Required version of AzureRM.Resources and AzureRM.Sql modules needs to be version 4 and above.
Follow these steps to load a PowerShell script inside the runbook created:
Inside the “Edit PowerShell Runbook” pane, select “RUNBOOKS” on the menu tree and expand the view
until you see the name of your runbook (in this example “AutomaticTuningEmailAutomation”). Select this
runbook.
On the first line of the “Edit PowerShell Runbook” (starting with the number 1), copy-paste the following
PowerShell script code. This PowerShell script is provided as-is to get you started. Modify the script to suite
your needs.
In the header of the provided PowerShell script, you need to replace <SUBSCRIPTION_ID_WITH_DATABASES> with your
Azure subscription ID. To learn how to retrieve your Azure subscription ID, see Getting your Azure Subscription
GUID.
In case of several subscriptions, you can add them as comma-delimited to the “$subscriptions” property in the
header of the script.
# Get credentials
$Conn = Get-AutomationConnection -Name AzureRunAsConnection
Connect-AzureRmAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -
CertificateThumbprint $Conn.CertificateThumbprint
# Skip if master
if ($DatabaseName -eq "master") {
continue
}
Ensure to adjust the content by customizing the PowerShell script to your needs.
With the above steps, the PowerShell script to retrieve Automatic tuning recommendations is loaded in Azure
Automation. The next step is to automate and schedule the email delivery job.
TIP
To send automated emails to different recipients, create separate flows. In these additional flows, change the recipient email
address in the “To” field, and the email subject line in the “Subject” field. Creating new runbooks in Azure Automation with
customized PowerShell scripts (such as with change of Azure subscription ID) enables further customization of automated
scenarios, such is for example emailing separate recipients on Automated tuning recommendations for separate
subscriptions.
The above concludes steps required to configure the email delivery job workflow. The entire flow consisting of
three actions built is shown in the following image.
To test the flow, click on “Run Now” in the upper right corner inside the flow pane.
Statistics of running the automated jobs, showing success of email notifications sent out, can be seen from the
Flow analytics pane.
The Flow analytics is helpful for monitoring the success of job executions, and if required for troubleshooting. In
the case of troubleshooting, you also might want to examine the PowerShell script execution log accessible
through Azure Automation app.
The final output of the automated email looks similar to the following email received after building and running
this solution:
By adjusting the PowerShell script, you can adjust the output and formatting of the automated email to your
needs.
You might further customize the solution to build email notifications based on a specific tuning event, and to
multiple recipients, for multiple subscriptions or databases, depending on your custom scenarios.
Next steps
Learn more on how automatic tuning can help you improve database performance, see Automatic tuning in
Azure SQL Database.
To enable automatic tuning in Azure SQL Database to manage your workload, see Enable automatic tuning.
To manually review and apply Automatic tuning recommendations, see Find and apply performance
recommendations.
Intelligent Insights
5/23/2018 • 9 minutes to read • Edit Online
Azure SQL Database Intelligent Insights lets you know what is happening with your database performance.
Intelligent Insights uses built-in intelligence to continuously monitor database usage through artificial
intelligence and detect disruptive events that cause poor performance. Once detected, a detailed analysis is
performed that generates a diagnostics log with an intelligent assessment of the issue. This assessment consists
of a root cause analysis of the database performance issue and, where possible, recommendations for
performance improvements.
PROPERTY DETAILS
Observed time range Start and end time for the period of the detected insight.
Impacted queries and error codes Query hash or error code. These can be used to easily
correlate to affected queries. Metrics that consist of either
query duration increase, waiting time, timeout counts, or
error codes are provided.
Root cause analysis Root cause analysis of the issue identified in a human-
readable format. Some insights might contain a performance
improvement recommendation where possible.
Performance issues that are recorded in the diagnostics log are flagged with one of the three states of an issue
lifecycle: "Active", "Verifying", and "Complete". After a performance issue is detected, and as long it's deemed as
present by SQL Database built-in intelligence, the issue is flagged as "Active". When the issue is considered
mitigated, it's verified and the issue status is changed to "Verifying". After SQL Database built-in intelligence
considers the issue resolved, the issue status is flagged as "Complete".
Intelligent Insights shines in discovering and troubleshooting SQL Database performance issues. In order to use
Intelligent Insights to troubleshoot SQL Database performance, see Troubleshoot Azure SQL Database
performance issues with Intelligent Insights.
After the Intelligent Insights diagnostics log is configured to stream data to SQL Analytics, you can monitor the
SQL database by using SQL Analytics.
Detection metrics
Metrics used for detection models that generate Intelligent Insights are based on monitoring:
Query duration
Timeout requests
Excessive wait time
Errored out requests
Query duration and timeout requests are used as primary models in detecting issues with database workload
performance. They're used because they directly measure what is happening with the workload. To detect all
possible cases of workload performance degradation, excessive wait time and errored-out requests are used as
additional models to indicate issues that affect the workload performance.
The system automatically considers changes to the workload and changes in the number of query requests
made to the database to dynamically determine normal and out-of-the-ordinary database performance
thresholds.
All of the metrics are considered together in various relationships through a scientifically derived data model that
categorizes each performance issue detected. Information provided through an intelligent insight includes:
Details of the performance issue detected.
A root cause analysis of the issue detected.
Recommendations on how to improve the performance of the monitored SQL database, where possible.
Query duration
The query duration degradation model analyzes individual queries and detects the increase in the time it takes to
compile and execute a query compared to the performance baseline.
If SQL Database built-in intelligence detects a significant increase in query compile or query execution time that
affects workload performance, these queries are flagged as query duration performance degradation issues.
The Intelligent Insights diagnostics log outputs the query hash of the query degraded in performance. The query
hash indicates whether the performance degradation was related to query compile or execution time increase,
which increased query duration time.
Timeout requests
The timeout requests degradation model analyzes individual queries and detects any increase in timeouts at the
query execution level and the overall request timeouts at the database level compared to the performance
baseline period.
Some of the queries might time out even before they reach the execution stage. Through the means of aborted
workers vs. requests made, SQL Database built-in intelligence measures and analyzes all queries that reached
the database whether they got to the execution stage or not.
After the number of timeouts for executed queries or the number of aborted request workers crosses the
system-managed threshold, a diagnostics log is populated with intelligent insights.
The insights generated contain the number of timed-out requests and the number of timed-out queries.
Indication of the performance degradation is related to timeout increase at the execution stage, or the overall
database level is provided. When the increase in timeouts is deemed significant to database performance, these
queries are flagged as timeout performance degradation issues.
Errored requests
The errored requests degradation model monitors individual queries and detects an increase in the number of
queries that errored out compared to the baseline period. This model also monitors critical exceptions that
crossed absolute thresholds managed by SQL Database built-in intelligence. The system automatically considers
the number of query requests made to the database and accounts for any workload changes in the monitored
period.
When the measured increase in errored requests relative to the overall number of requests made is deemed
significant to workload performance, affected queries are flagged as errored requests performance degradation
issues.
The Intelligent Insights log outputs the count of errored requests. It indicates whether the performance
degradation was related to an increase in errored requests or to crossing a monitored critical exception threshold
and measured time of the performance degradation.
If any of the monitored critical exceptions cross the absolute thresholds managed by the system, an intelligent
insight is generated with critical exception details.
Next steps
Learn how to troubleshoot SQL Database performance issues with Intelligent Insights.
Use the Intelligent Insights SQL Database performance diagnostics log.
Learn how to monitor SQL Database by using SQL Analytics.
Learn how to collect and consume log data from your Azure resources.
Azure SQL Database metrics and diagnostics logging
9/13/2018 • 16 minutes to read • Edit Online
Azure SQL Database can emit metrics and diagnostics logs for easier monitoring. You can configure SQL
Database to store resource usage, workers and sessions, and connectivity into one of these Azure resources:
Azure Storage: Used for archiving vast amounts of telemetry for a small price.
Azure Event Hubs: Used for integrating SQL Database telemetry with your custom monitoring solution or
hot pipelines.
Azure Log Analytics: Used for an out-of-the-box monitoring solution with reporting, alerting, and
mitigating capabilities. This is a feature of the Operations Management Suite (OMS )
Enable logging
Metrics and diagnostics logging is not enabled by default. You can enable and manage metrics and diagnostics
logging by using one of the following methods:
Azure portal
PowerShell
Azure CLI
Azure Monitor REST API
Azure Resource Manager template
When you enable metrics and diagnostics logging, you need to specify the Azure resource where selected data is
collected. Options available include:
Log Analytics
Event Hubs
Storage
You can provision a new Azure resource or select an existing resource. After selecting the storage resource, you
need to specify which data to collect. Options available include:
All metrics: Contains DTU percentage, DTU limit, CPU percentage, physical data read percentage, log write
percentage, Successful/Failed/Blocked by firewall connections, sessions percentage, workers percentage,
storage, storage percentage, and XTP storage percentage.
QueryStoreRuntimeStatistics: Contains information about the query runtime statistics, such as CPU usage and
query duration.
QueryStoreWaitStatistics: Contains information about the query wait statistics, which tells you what your
queries waited on, such as CPU, LOG, and LOCKING.
Errors: Contains information about SQL errors that happened on this database.
DatabaseWaitStatistics: Contains information about how much time a database spent waiting on different wait
types.
Timeouts: Contains information about timeouts that happened on a database.
Blocks: Contains information about blocking events that happened on a database.
SQLInsights: Contains Intelligent Insights. Learn more about Intelligent Insights.
Audit / SQLSecurityAuditEvents: Currently unavailable.
If you select Event Hubs or a storage account, you can specify a retention policy. This policy deletes data that is
older than a selected time period. If you specify Log Analytics, the retention policy depends on the selected pricing
tier. For more information, see Log Analytics pricing.
To learn how to enable logging and understand the metrics and log categories that are supported by the various
Azure services, we recommend that you read:
Overview of metrics in Microsoft Azure
Overview of Azure diagnostics logs
Azure portal
1. To enable metrics and diagnostics logs collection in the portal, go to your SQL Database or elastic pool
page, and then select Diagnostics settings.
2. Create new or edit existing diagnostics settings by selecting the target and the telemetry.
PowerShell
To enable metrics and diagnostics logging by using PowerShell, use the following commands:
To enable storage of diagnostics logs in a storage account, use this command:
Set-AzureRmDiagnosticSetting -ResourceId [your resource id] -StorageAccountId [your storage account id]
-Enabled $true
The storage account ID is the resource ID for the storage account where you want to send the logs.
To enable streaming of diagnostics logs to an event hub, use this command:
Set-AzureRmDiagnosticSetting -ResourceId [your resource id] -ServiceBusRuleId [your service bus rule
id] -Enabled $true
To enable sending diagnostics logs to a Log Analytics workspace, use this command:
You can obtain the resource ID of your Log Analytics workspace by using the following command:
(Get-AzureRmOperationalInsightsWorkspace).ResourceId
PS C:\> $WSID =
"/subscriptions/<subID>/resourcegroups/<RG_NAME>/providers/microsoft.operationalinsights/workspaces/<WS
_NAME>"
PS C:\> .\Enable-AzureRMDiagnostics.ps1 -WSID $WSID
Azure CLI
To enable metrics and diagnostics logging by using the Azure CLI, use the following commands:
To enable storage of diagnostics logs in a storage account, use this command:
azure insights diagnostic set --resourceId <resourceId> --storageId <storageAccountId> --enabled true
The storage account ID is the resource ID for the storage account where you want to send the logs.
To enable streaming of diagnostics logs to an event hub, use this command:
To enable sending diagnostics logs to a Log Analytics workspace, use this command:
azure insights diagnostic set --resourceId <resourceId> --workspaceId <resource id of the log analytics
workspace> --enabled true
2. On your home page, the Azure SQL Analytics tile appears. Select this tile to open the SQL Analytics
dashboard.
Use the SQL Analytics solution
SQL Analytics is a hierarchical dashboard that allows you to move through the hierarchy of SQL Database
resources. To learn how to use the SQL Analytics solution, see Monitor SQL Database by using the SQL Analytics
solution.
insights-metrics-minute/resourceId=/SUBSCRIPTIONS/s1id1234-5679-0123-4567-
890123456789/RESOURCEGROUPS/TESTRESOURCEGROUP/PROVIDERS/MICROSOFT.SQL/
servers/Server1/databases/database1/y=2016/m=08/d=22/h=18/m=00/PT1H.json
If you want to record the data from the elastic pool, the blob name is a bit different:
Elastic pool eDTU percentage, eDTU used, eDTU limit, CPU percentage,
physical data read percentage, log write percentage, sessions
percentage, workers percentage, storage, storage percentage,
storage limit, XTP storage percentage
Logs
Query Store runtime statistics
PROPERTY DESCRIPTION
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
total_query_wait_time_ms_d Total wait time of the query on the specific wait category.
query_param_type_d 0
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
Blockings dataset
PROPERTY DESCRIPTION
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
Deadlocks dataset
PROPERTY DESCRIPTION
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
ResourceGroup Name of the resource group that the database belongs to.
ElasticPoolName_s Name of the elastic pool that the database belongs to, if any.
Next steps
To learn how to enable logging and understand the metrics and log categories supported by the various Azure
services, read:
Overview of metrics in Microsoft Azure
Overview of Azure diagnostics logs
To learn about Event Hubs, read:
What is Azure Event Hubs?
Get started with Event Hubs
To learn more about Storage, see how to download metrics and diagnostics logs from Storage.
Use the Intelligent Insights Azure SQL Database
performance diagnostics log
9/12/2018 • 6 minutes to read • Edit Online
This page provides information on how to use the Azure SQL Database performance diagnostics log generated by
Intelligent Insights, its format, and the data it contains for your custom development needs. You can send this
diagnostics log to Azure Log Analytics, Azure Event Hubs, Azure Storage, or a third-party solution for custom
DevOps alerting and reporting capabilities.
Log header
The diagnostics log uses JSON standard format to output Intelligent Insights findings. The exact category
property for accessing an Intelligent Insights log is the fixed value "SQLInsights".
The header of the log is common and consists of the time stamp (TimeGenerated) that shows when an entry was
created. It also includes a resource ID (ResourceId) that refers to the particular SQL Database the entry relates to.
The category (Category), level (Level), and operation name (OperationName) are fixed properties whose values
do not change. They indicate that the log entry is informational and that it comes from Intelligent Insights
(SQLInsights).
Detected issues
The next section of the Intelligent Insights performance log contains performance issues that were detected
through built-in artificial intelligence. Detections are disclosed in properties within the JSON diagnostics log.
These detections consist of the category of an issue, the impact of the issue, the queries affected, and the metrics.
The detections properties might contain multiple performance issues that were detected.
Detected performance issues are reported with the following detections property structure:
"detections_s" : [{
"impact" : 1 to 3, // impact of the issue detected, possible values 1-3 (1 low, 2 moderate, 3 high impact)
"category" : "Detectable performance pattern", // performance issue detected, see the table
"details": <Details outputted> // details of an issue (see the table)
}]
Detectable performance patterns and the details that are outputted to the diagnostics log are provided in the
following table.
Detection category
The category (category) property describes the category of detectable performance patterns. See the following
table for all possible categories of detectable performance patterns. For more information, see Troubleshoot
database performance issues with Intelligent Insights.
Depending on the performance issue detected, the details outputted in the diagnostics log file differ accordingly.
Impact
The impact (impact) property describes how much a detected behavior contributed to the problem that a database
is having. Impacts range from 1 to 3, with 3 as the highest contribution, 2 as moderate, and 1 as the lowest
contribution. The impact value might be used as an input for custom alerting automation, depending on your
specific needs. The property queries impacted (QueryHashes) provide a list of the query hashes that were affected
by a particular detection.
Impacted queries
The next section of the Intelligent Insights log provides information about particular queries that were affected by
the detected performance issues. This information is disclosed as an array of objects embedded in the impact_s
property. The impact property consists of entities and metrics. Entities refer to a particular query (Type: Query).
The unique query hash is disclosed under the value (Value) property. In addition, each of the queries disclosed is
followed by a metric and a value, which indicate a detected performance issue.
In the following log example, the query with the hash 0x9102EXZ4 was detected to have an increased duration of
execution (Metric: DurationIncreaseSeconds). The value of 110 seconds indicates that this particular query took
110 seconds longer to execute. Because multiple queries can be detected, this particular log section might include
multiple query entries.
"impact" : [{
"entity" : {
"Type" : "Query", // type of entity - query
"Value" : "query hash value", // for example "0x9102EXZ4" query hash value },
"Metric" : "DurationIncreaseSeconds", // measured metric and the measurement unit (in this case seconds)
"Value" : 110 // value of the measured metric (in this case seconds)
}]
Metrics
The unit of measurement for each metric reported is provided under the metric (metric) property with the
possible values of seconds, number, and percentage. The value of a measured metric is reported in the value
(value) property.
The DurationIncreaseSeconds property provides the unit of measurement in seconds. The CriticalErrorCount unit
of measurement is a number that represents an error count.
// example of reported root cause analysis of the detected performance issue, in a human-readable format
"rootCauseAnalysis_s" : "High data IO caused performance to degrade. It seems that this database is missing
some indexes that could help."
You can use the Intelligent Insights performance log with Azure Log Analytics or a third-party solution for custom
DevOps alerting and reporting capabilities.
Next steps
Learn about Intelligent Insights concepts.
Learn how to troubleshoot Azure SQL Database performance issues with Intelligent Insights.
Learn how to monitor Azure SQL Database by using Azure SQL Analytics.
Learn how to collect and consume log data from your Azure resources.
Troubleshoot Azure SQL Database performance
issues with Intelligent Insights
9/12/2018 • 23 minutes to read • Edit Online
This page provides information on Azure SQL Database performance issues that are detected through the
Intelligent Insights database performance diagnostics log. This diagnostics log can be sent to Azure Log Analytics,
Azure Event Hubs, Azure Storage, or a third-party solution for custom DevOps alerting and reporting capabilities.
NOTE
For a quick SQL Database performance troubleshooting guide through Intelligent Insights, see the Recommended
troubleshooting flow flowchart in this document.
Memory Pressure Workers that requested memory grants have to wait for
memory allocations for statistically significant amounts of
time. Or an increased accumulation of workers that requested
memory grants exists, which affects SQL Database
performance.
Missing Index A missing index issue was detected, which affects SQL
Database performance.
DETECTABLE PERFORMANCE PATTERNS DETAILS OUTPUTTED
New Query A new query was detected, which affects overall SQL
Database performance.
Unusual Wait Statistic Unusual database wait times were detected, which affects SQL
Database performance.
TempDB Contention Multiple threads try to access the same tempDB resources,
which causes a bottleneck that affects SQL Database
performance.
Elastic pool DTU Shortage A shortage of available eDTUs in the elastic pool affects SQL
Database performance.
Database-Scoped Configuration Value Change A configuration change on the database affects SQL Database
performance.
Pricing Tier Downgrade A pricing tier downgrade action decreased available resources,
which affects SQL Database performance.
TIP
For continuous performance optimization of SQL Database, enable Azure SQL Database automatic tuning. This unique
feature of SQL Database built-in intelligence continuously monitors your SQL database, automatically tunes indexes, and
applies query execution plan corrections.
The following section describes the previously listed detectable performance patterns in more detail.
Workload Increase
What is happening
This performance pattern identifies issues caused by a workload increase or, in its more severe form, a workload
pile-up.
This detection is made through a combination of several metrics. The basic metric measured is detecting an
increase in workload compared with the past workload baseline. The other form of detection is based on
measuring a large increase in active worker threads that is large enough to affect the query performance.
In its more severe form, the workload might continuously pile up due to the inability of the SQL database to
handle the workload. The result is a continuously growing workload size, which is the workload pile-up condition.
Due to this condition, the time that the workload waits for execution grows. This condition represents one of the
most severe database performance issues. This issue is detected through monitoring the increase in the number
of aborted worker threads.
Troubleshooting
The diagnostics log outputs the number of queries whose execution has increased and the query hash of the
query with the largest contribution to the workload increase. You can use this information as a starting point for
optimizing the workload. The query identified as the largest contributor to the workload increase is especially
useful as your starting point.
You might consider distributing the workloads more evenly to the database. Consider optimizing the query that is
affecting the performance by adding indexes. You also might distribute your workload among multiple databases.
If these solutions aren't possible, consider increasing the pricing tier of your SQL database subscription to
increase the amount of resources available.
Memory Pressure
What is happening
This performance pattern indicates degradation in the current database performance caused by memory
pressure, or in its more severe form a memory pile-up condition, compared to the past seven-day performance
baseline.
Memory pressure denotes a performance condition in which there is a large number of worker threads
requesting memory grants in the SQL database. The high volume causes a high memory utilization condition in
which the SQL database is unable to efficiently allocate memory to all workers that request it. One of the most
common reasons for this issue is related to the amount of memory available to the SQL database on one hand.
On the other hand, an increase in workload causes the increase in worker threads and the memory pressure.
The more severe form of memory pressure is the memory pile-up condition. This condition indicates that a
higher number of worker threads are requesting memory grants than there are queries releasing the memory.
This number of worker threads requesting memory grants also might be continuously increasing (piling up)
because the SQL database engine is unable to allocate memory efficiently enough to meet the demand. The
memory pile-up condition represents one of the most severe database performance issues.
Troubleshooting
The diagnostics log outputs the memory object store details with the clerk (that is, worker thread) marked as the
highest reason for high memory usage and relevant time stamps. You can use this information as the basis for
troubleshooting.
You can optimize or remove queries related to the clerks with the highest memory usage. You also can make sure
that you aren't querying data that you don't plan to use. Good practice is to always use a WHERE clause in your
queries. In addition, we recommend that you create nonclustered indexes to seek the data rather than scan it.
You also can reduce the workload by optimizing or distributing it over multiple databases. Or you can distribute
your workload among multiple databases. If these solutions aren't possible, consider increasing the pricing tier of
your SQL database subscription to increase the amount of memory resources available to the database.
For additional troubleshooting suggestions, see Memory grants meditation: The mysterious SQL Server memory
consumer with many names.
Locking
What is happening
This performance pattern indicates degradation in the current database performance in which excessive database
locking is detected compared to the past seven-day performance baseline.
In modern RDBMS, locking is essential for implementing multithreaded systems in which performance is
maximized by running multiple simultaneous workers and parallel database transactions where possible. Locking
in this context refers to the built-in access mechanism in which only a single transaction can exclusively access the
rows, pages, tables, and files that are required and not compete with another transaction for resources. When the
transaction that locked the resources for use is done with them, the lock on those resources is released, which
allows other transactions to access required resources. For more information on locking, see Lock in the database
engine.
If transactions executed by the SQL engine are waiting for prolonged periods of time to access resources locked
for use, this wait time causes the slowdown of the workload execution performance.
Troubleshooting
The diagnostics log outputs locking details that you can use as the basis for troubleshooting. You can analyze the
reported blocking queries, that is, the queries that introduce the locking performance degradation, and remove
them. In some cases, you might be successful in optimizing the blocking queries.
The simplest and safest way to mitigate the issue is to keep transactions short and to reduce the lock footprint of
the most expensive queries. You can break up a large batch of operations into smaller operations. Good practice is
to reduce the query lock footprint by making the query as efficient as possible. Reduce large scans because they
increase chances of deadlocks and adversely affect overall database performance. For identified queries that cause
locking, you can create new indexes or add columns to the existing index to avoid the table scans.
For more suggestions, see How to resolve blocking problems that are caused by lock escalation in SQL Server.
Increased MAXDOP
What is happening
This detectable performance pattern indicates a condition in which a chosen query execution plan was parallelized
more than it should have been. The SQL Database query optimizer can enhance the workload performance by
executing queries in parallel to speed up things where possible. In some cases, parallel workers processing a
query spend more time waiting on each other to synchronize and merge results compared to executing the same
query with fewer parallel workers, or even in some cases compared to a single worker thread.
The expert system analyzes the current database performance compared to the baseline period. It determines if a
previously running query is running slower than before because the query execution plan is more parallelized
than it should be.
The MAXDOP server configuration option on SQL Database is used to control how many CPU cores can be used
to execute the same query in parallel.
Troubleshooting
The diagnostics log outputs query hashes related to queries for which the duration of execution increased
because they were parallelized more than they should have been. The log also outputs CXP wait times. This time
represents the time a single organizer/coordinator thread (thread 0) is waiting for all other threads to finish
before merging the results and moving ahead. In addition, the diagnostics log outputs the wait times that the
poor-performing queries were waiting in execution overall. You can use this information as the basis for
troubleshooting.
First, optimize or simplify complex queries. Good practice is to break up long batch jobs into smaller ones. In
addition, ensure that you created indexes to support your queries. You can also manually enforce the maximum
degree of parallelism (MAXDOP ) for a query that was flagged as poor performing. To configure this operation by
using T-SQL, see Configure the MAXDOP server configuration option.
Setting the MAXDOP server configuration option to zero (0) as a default value denotes that SQL Database can
use all available logical CPU cores to parallelize threads for executing a single query. Setting MAXDOP to one (1)
denotes that only one core can be used for a single query execution. In practical terms, this means that parallelism
is turned off. Depending on the case-per-case basis, available cores to the database, and diagnostics log
information, you can tune the MAXDOP option to the number of cores used for parallel query execution that
might resolve the issue in your case.
Pagelatch Contention
What is happening
This performance pattern indicates the current database workload performance degradation due to pagelatch
contention compared to the past seven-day workload baseline.
Latches are lightweight synchronization mechanisms used by SQL Database to enable multithreading. They
guarantee consistency of in-memory structures that include indices, data pages, and other internal structures.
There are many types of latches available on the SQL database. For simplicity purposes, buffer latches are used to
protect in-memory pages in the buffer pool. IO latches are used to protect pages not yet loaded into the buffer
pool. Whenever data is written to or read from a page in the buffer pool, a worker thread needs to acquire a
buffer latch for the page first. Whenever a worker thread attempts to access a page that isn't already available in
the in-memory buffer pool, an IO request is made to load the required information from the storage. This
sequence of events indicates a more severe form of performance degradation.
Contention on the page latches occurs when multiple threads concurrently attempt to acquire latches on the same
in-memory structure, which introduces an increased wait time to query execution. In the case of pagelatch IO
contention, when data needs to be accessed from storage, this wait time is even larger. It can affect workload
performance considerably. Pagelatch contention is the most common scenario of threads waiting on each other
and competing for resources on multiple CPU systems.
Troubleshooting
The diagnostics log outputs pagelatch contention details. You can use this information as the basis for
troubleshooting.
Because a pagelatch is an internal control mechanism of SQL Database, it automatically determines when to use
them. Application decisions, including schema design, can affect pagelatch behavior due to the deterministic
behavior of latches.
One method for handling latch contention is to replace a sequential index key with a nonsequential key to evenly
distribute inserts across an index range. Typically, a leading column in the index distributes the workload
proportionally. Another method to consider is table partitioning. Creating a hash partitioning scheme with a
computed column on a partitioned table is a common approach for mitigating excessive latch contention. In the
case of pagelatch IO contention, introducing indexes helps to mitigate this performance issue.
For more information, see Diagnose and resolve latch contention on SQL Server (PDF download).
Missing Index
What is happening
This performance pattern indicates the current database workload performance degradation compared to the past
seven-day baseline due to a missing index.
An index is used to speed up the performance of queries. It provides quick access to table data by reducing the
number of dataset pages that need to be visited or scanned.
Specific queries that caused performance degradation are identified through this detection for which creating
indexes would be beneficial to the performance.
Troubleshooting
The diagnostics log outputs query hashes for the queries that were identified to affect the workload performance.
You can build indexes for these queries. You also can optimize or remove these queries if they aren't required. A
good performance practice is to avoid querying data that you don't use.
TIP
Did you know that SQL Database built-in intelligence can automatically manage the best-performing indexes for your
databases?
For continuous performance optimization of SQL Database, we recommend that you enable SQL Database automatic
tuning. This unique feature of SQL Database built-in intelligence continuously monitors your SQL database and
automatically tunes and creates indexes for your databases.
New Query
What is happening
This performance pattern indicates that a new query is detected that is performing poorly and affecting the
workload performance compared to the seven-day performance baseline.
Writing a good-performing query sometimes can be a challenging task. For more information on writing queries,
see Writing SQL queries. To optimize existing query performance, see Query tuning.
Troubleshooting
The diagnostics log outputs information up to two new most CPU -consuming queries, including their query
hashes. Because the detected query affects the workload performance, you can optimize your query. Good
practice is to retrieve only data you need to use. We also recommend using queries with a WHERE clause. We
also recommend that you simplify complex queries and break them up into smaller queries. Another good
practice is to break down large batch queries into smaller batch queries. Introducing indexes for new queries is
typically a good practice to mitigate this performance issue.
Consider using Azure SQL Database Query Performance Insight.
TempDB Contention
What is happening
This detectable performance pattern indicates a database performance condition in which a bottleneck of threads
trying to access tempDB resources exists. (This condition isn't IO related.) The typical scenario for this
performance issue is hundreds of concurrent queries that all create, use, and then drop small tempDB tables. The
system detected that the number of concurrent queries using the same tempDB tables increased with sufficient
statistical significance to affect database performance compared to the past seven-day performance baseline.
Troubleshooting
The diagnostics log outputs tempDB contention details. You can use the information as the starting point for
troubleshooting. There are two things you can pursue to alleviate this kind of contention and increase the
throughput of the overall workload: You can stop using the temporary tables. You also can use memory-optimized
tables.
For more information, see Introduction to memory-optimized tables.
Plan Regression
What is happening
This detectable performance pattern denotes a condition in which SQL Database utilizes a suboptimal query
execution plan. The suboptimal plan typically causes increased query execution, which leads to longer wait times
for the current and other queries.
The SQL database determines the query execution plan with the least cost to a query execution. As the type of
queries and workloads change, sometimes the existing plans are no longer efficient, or perhaps SQL Database
didn't make a good assessment. As a matter of correction, query execution plans can be manually forced.
This detectable performance pattern combines three different cases of plan regression: new plan regression, old
plan regression, and existing plans changed workload. The particular type of plan regression that occurred is
provided in the details property in the diagnostics log.
The new plan regression condition refers to a state in which SQL Database starts executing a new query
execution plan that isn't as efficient as the old plan. The old plan regression condition refers to the state when SQL
Database switches from using a new, more efficient plan to the old plan, which isn't as efficient as the new plan.
The existing plans changed workload regression refers to the state in which the old and the new plans
continuously alternate, with the balance going more toward the poor-performing plan.
For more information on plan regressions, see What is plan regression in SQL Server?.
Troubleshooting
The diagnostics log outputs the query hashes, good plan ID, bad plan ID, and query IDs. You can use this
information as the basis for troubleshooting.
You can analyze which plan is better performing for your specific queries that you can identify with the query
hashes provided. After you determine which plan works better for your queries, you can manually force it.
For more information, see Learn how SQL Server prevents plan regressions.
TIP
Did you know that SQL Database built-in intelligence can automatically manage the best-performing query execution plans
for your databases?
For continuous performance optimization of SQL Database, we recommend that you enable SQL Database automatic
tuning. This unique feature of SQL Database built-in intelligence continuously monitors your SQL database and
automatically tunes and creates best-performing query execution plans for your databases.
Database-Scoped Configuration Value Change
What is happening
This detectable performance pattern indicates a condition in which a change in the database-scoped configuration
causes performance regression that is detected compared to the past seven-day database workload behavior. This
pattern denotes that a recent change made to the database-scoped configuration doesn't seem to be beneficial to
your database performance.
Database-scoped configuration changes can be set for each individual database. This configuration is used on a
case-by-case basis to optimize the individual performance of your database. The following options can be
configured for each individual database: MAXDOP, LEGACY_CARDINALITY_ESTIMATION,
PARAMETER_SNIFFING, QUERY_OPTIMIZER_HOTFIXES, and CLEAR PROCEDURE_CACHE.
Troubleshooting
The diagnostics log outputs database-scoped configuration changes that were made recently that caused
performance degradation compared to the previous seven-day workload behavior. You can revert the
configuration changes to the previous values. You also can tune value by value until the desired performance level
is reached. You can copy database-scope configuration values from a similar database with satisfactory
performance. If you're unable to troubleshoot the performance, revert to the default SQL Database default values
and attempt to fine-tune starting from this baseline.
For more information on optimizing database-scoped configuration and T-SQL syntax on changing the
configuration, see Alter database-scoped configuration (Transact-SQL ).
Slow Client
What is happening
This detectable performance pattern indicates a condition in which the client using the SQL database can't
consume the output from the database as fast as the database sends the results. Because SQL Database isn't
storing results of the executed queries in a buffer, it slows down and waits for the client to consume the
transmitted query outputs before proceeding. This condition also might be related to a network that isn't
sufficiently fast enough to transmit outputs from the SQL database to the consuming client.
This condition is generated only if a performance regression is detected compared to the past seven-day database
workload behavior. This performance issue is detected only if a statistically significant performance degradation
occurs compared to previous performance behavior.
Troubleshooting
This detectable performance pattern indicates a client-side condition. Troubleshooting is required at the client-side
application or client-side network. The diagnostics log outputs the query hashes and wait times that seem to be
waiting the most for the client to consume them within the past two hours. You can use this information as the
basis for troubleshooting.
You can optimize performance of your application for consumption of these queries. You also can consider
possible network latency issues. Because the performance degradation issue was based on change in the last
seven-day performance baseline, you can investigate whether recent application or network condition changes
caused this performance regression event.
Intelligent Insights usually needs one hour of time to perform the root cause analysis of the performance issue. If
you can't locate your issue in Intelligent Insights and it's critical to you, use the Query Store to manually identify
the root cause of the performance issue. (Typically, these issues are less than one hour old.) For more information,
see Monitor performance by using the Query Store.
Next steps
Learn Intelligent Insights concepts.
Use the Intelligent Insights Azure SQL Database performance diagnostics log.
Monitor Azure SQL Database by using Azure SQL Analytics.
Learn to collect and consume log data from your Azure resources.
Monitor and improve performance
9/12/2018 • 2 minutes to read • Edit Online
Azure SQL Database identifies potential problems in your database and recommends actions that can improve
performance of your workload by providing intelligent tuning actions and recommendations.
To review your database performance, use the Performance tile on the Overview page, or navigate down to
"Support + troubleshooting" section:
In the "Support + troubleshooting" section, you can use the following pages:
1. Performance overview to monitor performance of your database.
2. Performance recommendations to find performance recommendations that can improve performance of your
workload.
3. Query Performance Insight to find top resource consuming queries.
4. Automatic tuning to let Azure SQL Database automatically optimize your database.
Performance Overview
This view provides a summary of your database performance, and helps you with performance tuning and
troubleshooting.
The Recommendations tile provides a breakdown of tuning recommendations for your database (top three
recommendations are shown if there are more). Clicking this tile takes you to Performance
recommendations.
The Tuning activity tile provides a summary of the ongoing and completed tuning actions for your database,
giving you a quick view into the history of tuning activity. Clicking this tile takes you to the full tuning history
view for your database.
The Auto-tuning tile shows the auto-tuning configuration for your database (tuning options that are
automatically applied to your database). Clicking this tile opens the automation configuration dialog.
The Database queries tile shows the summary of the query performance for your database (overall DTU
usage and top resource consuming queries). Clicking this tile takes you to Query Performance Insight.
Performance recommendations
This page provides intelligent tuning recommendations that can improve your database's performance. The
following types of recommendations are shown on this page:
Recommendations on which indexes to create or drop.
Recommendations when schema issues are identified in the database.
Recommendations when queries can benefit from parameterized queries.
You can also find complete history of tuning actions that were applied in the past.
Learn how to find an apply performance recommendations in Find and apply performance recommendations
article.
Automatic tuning
Azure SQL databases can automatically tune database performance by applying performance recommendations.
To learn more, read Automatic tuning article. To enable it, read how to enable automatic tuning.
Additional resources
Azure SQL Database performance guidance for single databases
When should an elastic pool be used?
Azure SQL Database Query Performance Insight
8/6/2018 • 7 minutes to read • Edit Online
Managing and tuning the performance of relational databases is a challenging task that requires significant
expertise and time investment. Query Performance Insight allows you to spend less time troubleshooting
database performance by providing the following:
Deeper insight into your databases resource (DTU ) consumption.
The top queries by CPU/Duration/Execution count, which can potentially be tuned for improved
performance.
The ability to drill down into the details of a query, view its text and history of resource utilization.
Performance tuning annotations that show actions performed by SQL Azure Database Advisor
Prerequisites
Query Performance Insight requires that Query Store is active on your database. If Query Store is not
running, the portal prompts you to turn it on.
Permissions
The following role-based access control permissions are required to use Query Performance Insight:
Reader, Owner, Contributor, SQL DB Contributor, or SQL Server Contributor permissions are required
to view the top resource consuming queries and charts.
Owner, Contributor, SQL DB Contributor, or SQL Server Contributor permissions are required to view
query text.
The top queries view opens and the top CPU consuming queries are listed.
2. Click around the chart for details.
The top line shows overall DTU% for the database, while the bars show CPU% consumed by the selected
queries during the selected interval (for example, if Past week is selected each bar represents one day).
The bottom grid represents aggregated information for the visible queries.
Query ID - unique identifier of query inside database.
CPU per query during observable interval (depends on aggregation function).
Duration per query (depends on aggregation function).
Total number of executions for a particular query.
Select or clear individual queries to include or exclude them from the chart using checkboxes.
3. If your data becomes stale, click the Refresh button.
4. You can use sliders and zoom buttons to change observation interval and investigate spikes:
5. Optionally, if you want a different view, you can select Custom tab and set:
Metric (CPU, duration, execution count)
Time interval (Last 24 hours, Past week, Past month).
Number of queries.
Aggregation function.
Viewing individual query details
To view query details:
1. Click any query in the list of top queries.
2. The details view opens and the queries CPU consumption/Duration/Execution count is broken down over
time.
3. Click around the chart for details.
Top chart shows line with overall database DTU%, and the bars are CPU% consumed by the selected
query.
Second chart shows total duration by the selected query.
Bottom chart shows total number of executions by the selected query.
4. Optionally, use sliders, zoom buttons or click Settings to customize how query data is displayed, or to pick a
different time period.
If you want to know more or apply advisor recommendation, click the icon. It will open details of action. If it’s an
active recommendation you can apply it straight away using command.
Multiple annotations.
It’s possible, that because of zoom level, annotations that are close to each other will get collapsed into one. This
will be represented by special icon, clicking it will open new blade where list of grouped annotations will be
shown. Correlating queries and performance tuning actions can help to better understand your workload.
Second case happens when Query Store is Off or parameters aren’t set optimally.
You can change the Retention and Capture policy and enable Query Store by executing commands below or
directly from portal:
Recommended retention and capture policy
There are two types of retention policies:
Size based - if set to AUTO it will clean data automatically when near max size is reached.
Time based - by default we will set it to 30 days, which means, if Query Store will run out of space, it will
delete query information older than 30 days
Capture policy could be set to:
All - Captures all queries.
Auto - Infrequent queries and queries with insignificant compile and execution duration are ignored.
Thresholds for execution count, compile and runtime duration are internally determined. This is the default
option.
None - Query Store stops capturing new queries, however runtime stats for already captured queries are
still collected.
We recommend setting all policies to AUTO and clean policy to 30 days:
Increase size of Query Store. This could be performed by connecting to a database and issuing following query:
Applying these settings will eventually make Query Store collecting new queries, however if you don’t want to
wait you can clear Query Store.
NOTE
Executing following query will delete all current information in the Query Store.
Summary
Query Performance Insight helps you understand the impact of your query workload and how it relates to
database resource consumption. With this feature, you will learn about the top consuming queries, and easily
identify the ones to fix before they become a problem.
Next steps
For additional recommendations about improving the performance of your SQL database, click
Recommendations on the Query Performance Insight blade.
Find and apply performance recommendations
9/4/2018 • 5 minutes to read • Edit Online
You can use the Azure portal to find performance recommendations that can optimize performance of your
Azure SQL Database or to correct some issue identified in your workload. Performance recommendation
page in Azure portal enables you to find the top recommendations based on their potential impact.
Viewing recommendations
To view and apply performance recommendations, you need the correct role-based access control permissions in
Azure. Reader, SQL DB Contributor permissions are required to view recommendations, and Owner, SQL DB
Contributor permissions are required to execute any actions; create or drop indexes and cancel index creation.
Use the following steps to find performance recommendations on Azure portal:
1. Sign in to the Azure portal.
2. Go to All services > SQL databases, and select your database.
3. Navigate to Performance recommendation to view available recommendations for the selected database.
Performance recommendations are shown in the table similar to the one shown on the following figure:
Recommendations are sorted by their potential impact on performance into the following categories:
IMPACT DESCRIPTION
You can also view the status of the historical operations. Select a recommendation or status to see more
information.
Here is an example of "Create index" recommendation in the Azure portal.
Applying recommendations
Azure SQL Database gives you full control over how recommendations are enabled using any of the following
three options:
Apply individual recommendations one at a time.
Enable the Automatic tuning to automatically apply recommendations.
To implement a recommendation manually, run the recommended T-SQL script against your database.
Select any recommendation to view its details and then click View script to review the exact details of how the
recommendation is created.
The database remains online while the recommendation is applied -- using performance recommendation or
automatic tuning never takes a database offline.
Apply an individual recommendation
You can review and accept recommendations one at a time.
1. On the Recommendations page, select a recommendation.
2. On the Details page, click Apply button.
NOTE
Please note that if SQL Database Automatic tuning is enabled, and if you have manually discarded a recommendation from
the list, such recommendation will never be applied automatically. Discarding a recommendation is a handy way for users
to have Automatic tuning enabled in cases when requiring that a specific recommendation shouldn’t be applied. You can
revert this behavior by adding discarded recommendations back to the Recommendations list by selecting the Undo
Discard option.
NOTE
Please note that DROP_INDEX option at this time is not compatible with applications using partition switching and index
hints and should not be enabled in these cases.
Monitoring operations
Applying a recommendation might not happen instantaneously. The portal provides details regarding the status
of recommendation. The following are possible states that an index can be in:
STATUS DESCRIPTION
STATUS DESCRIPTION
Reverting a recommendation
If you used the performance recommendations to apply the recommendation (meaning you did not manually
run the T-SQL script), it automatically reverts the change if it finds the performance impact to be negative. If for
any reason you simply want to revert a recommendation, you can do the following:
1. Select a successfully applied recommendation in the Tuning history area.
2. Click Revert on the recommendation details page.
Next steps
Monitor your recommendations and continue to apply them to refine performance. Database workloads are
dynamic and change continuously. Azure SQL Database continues to monitor and provide recommendations
that can potentially improve your database's performance.
See Automatic tuning to learn more about the automatic tuning in Azure SQL Database.
See Performance recommendations for an overview of Azure SQL Database performance recommendations.
See Query Performance Insights to learn about viewing the performance impact of your top queries.
Additional resources
Query Store
CREATE INDEX
Role-based access control
Tuning performance in Azure SQL Database
8/23/2018 • 21 minutes to read • Edit Online
Azure SQL Database provides recommendations that you can use to improve performance of your database, or
you can let Azure SQL Database automatically adapt to your application and apply changes that will improve
performance of your workload.
In you don't have any applicable recommendations, and you still have performance issues, you might use the
following methods to improve performances:
Increase the service tiers in your DTU -based purchasing model or your vCore-based purchasing model to
provide more resources to your database.
Tune your application and apply some best practices that can improve performance.
Tune the database by changing indexes and queries to more efficiently work with data.
These are manual methods because you need to decide the amount of resources meet your needs. Otherwise, you
would need to rewrite the application or database code and deploy the changes.
NOTE
This article focuses on performance guidance for single databases in Azure SQL Database. For performance guidance related
to elastic pools, see Price and performance considerations for elastic pools. Note, though, that you can apply many of the
tuning recommendations in this article to databases in an elastic pool, and get similar performance benefits.
Basic: The Basic service tier offers good performance predictability for each database, hour over hour. In a
Basic database, sufficient resources support good performance in a small database that doesn't have multiple
concurrent requests. Typical use cases when you would use Basic service tier are:
You're just getting started with Azure SQL Database. Applications that are in development often
don't need high-performance levels. Basic databases are an ideal environment for database
development or testing, at a low price point.
You have a database with a single user. Applications that associate a single user with a database
typically don’t have high concurrency and performance requirements. These applications are candidates
for the Basic service tier.
Standard: The Standard service tier offers improved performance predictability and provides good
performance for databases that have multiple concurrent requests, like workgroup and web applications.
When you choose a Standard service tier database, you can size your database application based on
predictable performance, minute over minute.
Your database has multiple concurrent requests. Applications that service more than one user at a
time usually need higher performance levels. For example, workgroup or web applications that have low
to medium IO traffic requirements supporting multiple concurrent queries are good candidates for the
Standard service tier.
Premium: The Premium service tier provides predictable performance, second over second, for each Premium
or Business Critical database. When you choose the Premium service tier, you can size your database
application based on the peak load for that database. The plan removes cases in which performance variance
can cause small queries to take longer than expected in latency-sensitive operations. This model can greatly
simplify the development and product validation cycles for applications that need to make strong statements
about peak resource needs, performance variance, or query latency. Most Premium service tier use cases have
one or more of these characteristics:
High peak load. An application that requires substantial CPU, memory, or input/output (IO ) to
complete its operations requires a dedicated, high-performance level. For example, a database operation
known to consume several CPU cores for an extended time is a candidate for the Premium service tier.
Many concurrent requests. Some database applications service many concurrent requests, for
example, when serving a website that has a high traffic volume. Basic and Standard service tiers limit
the number of concurrent requests per database. Applications that require more connections would
need to choose an appropriate reservation size to handle the maximum number of needed requests.
Low latency. Some applications need to guarantee a response from the database in minimal time. If a
specific stored procedure is called as part of a broader customer operation, you might have a
requirement to have a return from that call in no more than 20 milliseconds, 99 percent of the time. This
type of application benefits from the Premium service tier, to make sure that the required computing
power is available.
The service level that you need for your SQL database depends on the peak load requirements for each resource
dimension. Some applications use a trivial amount of a single resource, but have significant requirements for
other resources.
Service tier capabilities and limits
At each service tier, you set the performance level, so you have the flexibility to pay only for the capacity you need.
You can adjust capacity, up or down, as workload changes. For example, if your database workload is high during
the back-to-school shopping season, you might increase the performance level for the database for a set time,
July through September. You can reduce it when your peak season ends. You can minimize what you pay by
optimizing your cloud environment to the seasonality of your business. This model also works well for software
product release cycles. A test team might allocate capacity while it does test runs, and then release that capacity
when they finish testing. In a capacity request model, you pay for capacity as you need it, and avoid spending on
dedicated resources that you might rarely use.
Why service tiers?
Although each database workload can differ, the purpose of service tiers is to provide performance predictability
at various performance levels. Customers with large-scale database resource requirements can work in a more
dedicated computing environment.
Azure SQL Database can help you find and fix common missing index conditions. DMVs that are built into Azure
SQL Database look at query compilations in which an index would significantly reduce the estimated cost to run a
query. During query execution, SQL Database tracks how often each query plan is executed, and tracks the
estimated gap between the executing query plan and the imagined one where that index existed. You can use
these DMVs to quickly guess which changes to your physical database design might improve overall workload
cost for a database and its real workload.
You can use this query to evaluate potential missing indexes:
After it's created, that same SELECT statement picks a different plan, which uses a seek instead of a scan, and then
executes the plan more efficiently:
The key insight is that the IO capacity of a shared, commodity system is more limited than that of a dedicated
server machine. There's a premium on minimizing unnecessary IO to take maximum advantage of the system in
the DTU of each performance level of the Azure SQL Database service tiers. Appropriate physical database design
choices can significantly improve the latency for individual queries, improve the throughput of concurrent
requests handled per scale unit, and minimize the costs required to satisfy the query. For more information about
the missing index DMVs, see sys.dm_db_missing_index_details.
Query tuning and hinting
The query optimizer in Azure SQL Database is similar to the traditional SQL Server query optimizer. Most of the
best practices for tuning queries and understanding the reasoning model limitations for the query optimizer also
apply to Azure SQL Database. If you tune queries in Azure SQL Database, you might get the additional benefit of
reducing aggregate resource demands. Your application might be able to run at a lower cost than an untuned
equivalent because it can run at a lower performance level.
An example that is common in SQL Server and which also applies to Azure SQL Database is how the query
optimizer "sniffs" parameters. During compilation, the query optimizer evaluates the current value of a parameter
to determine whether it can generate a more optimal query plan. Although this strategy often can lead to a query
plan that is significantly faster than a plan compiled without known parameter values, currently it works
imperfectly both in SQL Server and in Azure SQL Database. Sometimes the parameter is not sniffed, and
sometimes the parameter is sniffed but the generated plan is suboptimal for the full set of parameter values in a
workload. Microsoft includes query hints (directives) so that you can specify intent more deliberately and override
the default behavior of parameter sniffing. Often, if you use hints, you can fix cases in which the default SQL
Server or Azure SQL Database behavior is imperfect for a specific customer workload.
The next example demonstrates how the query processor can generate a plan that is suboptimal both for
performance and resource requirements. This example also shows that if you use a query hint, you can reduce
query run time and resource requirements for your SQL database:
DROP TABLE psptest1;
CREATE TABLE psptest1(col1 int primary key identity, col2 int, col3 binary(200));
DECLARE @a int = 0;
SET NOCOUNT ON;
BEGIN TRANSACTION
WHILE @a < 20000
BEGIN
INSERT INTO psptest1(col2) values (1);
INSERT INTO psptest1(col2) values (@a);
SET @a += 1;
END
COMMIT TRANSACTION
CREATE INDEX i1 on psptest1(col2);
GO
CREATE TABLE t1 (col1 int primary key, col2 int, col3 binary(200));
GO
The setup code creates a table that has skewed data distribution. The optimal query plan differs based on which
parameter is selected. Unfortunately, the plan caching behavior doesn't always recompile the query based on the
most common parameter value. So, it's possible for a suboptimal plan to be cached and used for many values,
even when a different plan might be a better plan choice on average. Then the query plan creates two stored
procedures that are identical, except that one has a special query hint.
Example, part 1
Example, part 2
(We recommend that you wait at least 10 minutes before you begin part 2 of the example, so that the results are
distinct in the resulting telemetry data.)
EXEC psp2 @param2=1;
TRUNCATE TABLE t1;
DECLARE @i int = 0;
WHILE @i < 1000
BEGIN
EXEC psp2 @param2=2;
TRUNCATE TABLE t1;
SET @i += 1;
END
Each part of this example attempts to run a parameterized insert statement 1,000 times (to generate a sufficient
load to use as a test data set). When it executes stored procedures, the query processor examines the parameter
value that is passed to the procedure during its first compilation (parameter "sniffing"). The processor caches the
resulting plan and uses it for later invocations, even if the parameter value is different. The optimal plan might not
be used in all cases. Sometimes you need to guide the optimizer to pick a plan that is better for the average case
rather than the specific case from when the query was first compiled. In this example, the initial plan generates a
"scan" plan that reads all rows to find each value that matches the parameter:
Because we executed the procedure by using the value 1, the resulting plan was optimal for the value 1 but was
suboptimal for all other values in the table. The result likely isn't what you would want if you were to pick each
plan randomly, because the plan performs more slowly and uses more resources.
If you run the test with SET STATISTICS IO set to ON , the logical scan work in this example is done behind the
scenes. You can see that there are 1,148 reads done by the plan (which is inefficient, if the average case is to return
just one row ):
The second part of the example uses a query hint to tell the optimizer to use a specific value during the
compilation process. In this case, it forces the query processor to ignore the value that is passed as the parameter,
and instead to assume UNKNOWN . This refers to a value that has the average frequency in the table (ignoring skew ).
The resulting plan is a seek-based plan that is faster and uses fewer resources, on average, than the plan in part 1
of this example:
You can see the effect in the sys.resource_stats table (there is a delay from the time that you execute the test and
when the data populates the table). For this example, part 1 executed during the 22:25:00 time window, and part 2
executed at 22:35:00. The earlier time window used more resources in that time window than the later one
(because of plan efficiency improvements).
SELECT TOP 1000 *
FROM sys.resource_stats
WHERE database_name = 'resource1'
ORDER BY start_time DESC
NOTE
Although the volume in this example is intentionally small, the effect of suboptimal parameters can be substantial, especially
on larger databases. The difference, in extreme cases, can be between seconds for fast cases and hours for slow cases.
You can examine sys.resource_stats to determine whether the resource for a test uses more or fewer resources
than another test. When you compare data, separate the timing of tests so that they are not in the same 5-minute
window in the sys.resource_stats view. The goal of the exercise is to minimize the total amount of resources
used, and not to minimize the peak resources. Generally, optimizing a piece of code for latency also reduces
resource consumption. Make sure that the changes you make to an application are necessary, and that the
changes don't negatively affect the customer experience for someone who might be using query hints in the
application.
If a workload has a set of repeating queries, often it makes sense to capture and validate the optimality of your
plan choices because it drives the minimum resource size unit required to host the database. After you validate it,
occasionally reexamine the plans to help you make sure that they have not degraded. You can learn more about
query hints (Transact-SQL ) .
Cross-database sharding
Because Azure SQL Database runs on commodity hardware, the capacity limits for a single database are lower
than for a traditional on-premises SQL Server installation. Some customers use sharding techniques to spread
database operations over multiple databases when the operations don't fit inside the limits of a single database in
Azure SQL Database. Most customers who use sharding techniques in Azure SQL Database split their data on a
single dimension across multiple databases. For this approach, you need to understand that OLTP applications
often perform transactions that apply to only one row or to a small group of rows in the schema.
NOTE
SQL Database now provides a library to assist with sharding. For more information, see Elastic Database client library
overview.
For example, if a database has customer name, order, and order details (like the traditional example Northwind
database that ships with SQL Server), you could split this data into multiple databases by grouping a customer
with the related order and order detail information. You can guarantee that the customer's data stays in a single
database. The application would split different customers across databases, effectively spreading the load across
multiple databases. With sharding, customers not only can avoid the maximum database size limit, but Azure SQL
Database also can process workloads that are significantly larger than the limits of the different performance
levels, as long as each individual database fits into its DTU.
Although database sharding doesn't reduce the aggregate resource capacity for a solution, it's highly effective at
supporting very large solutions that are spread over multiple databases. Each database can run at a different
performance level to support very large, "effective" databases with high resource requirements.
Functional partitioning
SQL Server users often combine many functions in a single database. For example, if an application has logic to
manage inventory for a store, that database might have logic associated with inventory, tracking purchase orders,
stored procedures, and indexed or materialized views that manage end-of-month reporting. This technique makes
it easier to administer the database for operations like backup, but it also requires you to size the hardware to
handle the peak load across all functions of an application.
If you use a scale-out architecture in Azure SQL Database, it's a good idea to split different functions of an
application into different databases. By using this technique, each application scales independently. As an
application becomes busier (and the load on the database increases), the administrator can choose independent
performance levels for each function in the application. At the limit, with this architecture, an application can be
larger than a single commodity machine can handle because the load is spread across multiple machines.
Batch queries
For applications that access data by using high-volume, frequent, ad hoc querying, a substantial amount of
response time is spent on network communication between the application tier and the Azure SQL Database tier.
Even when both the application and Azure SQL Database are in the same data center, the network latency
between the two might be magnified by a large number of data access operations. To reduce the network round
trips for the data access operations, consider using the option to either batch the ad hoc queries, or to compile
them as stored procedures. If you batch the ad hoc queries, you can send multiple queries as one large batch in a
single trip to Azure SQL Database. If you compile ad hoc queries in a stored procedure, you could achieve the
same result as if you batch them. Using a stored procedure also gives you the benefit of increasing the chances of
caching the query plans in Azure SQL Database so you can use the stored procedure again.
Some applications are write-intensive. Sometimes you can reduce the total IO load on a database by considering
how to batch writes together. Often, this is as simple as using explicit transactions instead of auto-commit
transactions in stored procedures and ad hoc batches. For an evaluation of different techniques you can use, see
Batching techniques for SQL Database applications in Azure. Experiment with your own workload to find the right
model for batching. Be sure to understand that a model might have slightly different transactional consistency
guarantees. Finding the right workload that minimizes resource use requires finding the right combination of
consistency and performance trade-offs.
Application-tier caching
Some database applications have read-heavy workloads. Caching layers might reduce the load on the database
and might potentially reduce the performance level required to support a database by using Azure SQL Database.
With Azure Redis Cache, if you have a read-heavy workload, you can read the data once (or perhaps once per
application-tier machine, depending on how it is configured), and then store that data outside your SQL database.
This is a way to reduce database load (CPU and read IO ), but there is an effect on transactional consistency
because the data being read from the cache might be out of sync with the data in the database. Although in many
applications some level of inconsistency is acceptable, that's not true for all workloads. You should fully
understand any application requirements before you implement an application-tier caching strategy.
Next steps
For more information about DTU -based service tiers, see DTU -based purchasing model.
For more information about vCore-based service tiers, see vCore-based purchasing model.
For more information about elastic pools, see What is an Azure elastic pool?
For information about performance and elastic pools, see When to consider an elastic pool
Monitoring Azure SQL Database using dynamic
management views
8/8/2018 • 3 minutes to read • Edit Online
Microsoft Azure SQL Database enables a subset of dynamic management views to diagnose performance
problems, which might be caused by blocked or long-running queries, resource bottlenecks, poor query plans, and
so on. This topic provides information on how to detect common performance problems by using dynamic
management views.
SQL Database partially supports three categories of dynamic management views:
Database-related dynamic management views.
Execution-related dynamic management views.
Transaction-related dynamic management views.
For detailed information on dynamic management views, see Dynamic Management Views and Functions
(Transact-SQL ) in SQL Server Books Online.
Permissions
In SQL Database, querying a dynamic management view requires VIEW DATABASE STATE permissions. The
VIEW DATABASE STATE permission returns information about all objects within the current database. To grant
the VIEW DATABASE STATE permission to a specific database user, run the following query:
GRANT VIEW DATABASE STATE TO database_user;
In an instance of on-premises SQL Server, dynamic management views return server state information. In SQL
Database, they return information regarding your current logical database only.
The following query returns the size of individual objects (in megabytes) in your database:
Monitoring connections
You can use the sys.dm_exec_connections view to retrieve information about the connections established to a
specific Azure SQL Database server and the details of each connection. In addition, the sys.dm_exec_sessions view
is helpful when retrieving information about all active user connections and internal tasks. The following query
retrieves information on the current connection:
SELECT
c.session_id, c.net_transport, c.encrypt_option,
c.auth_scheme, s.host_name, s.program_name,
s.client_interface_name, s.login_name, s.nt_domain,
s.nt_user_name, s.original_login_name, c.connect_time,
s.login_time
FROM sys.dm_exec_connections AS c
JOIN sys.dm_exec_sessions AS s
ON c.session_id = s.session_id
WHERE c.session_id = @@SPID;
NOTE
When executing the sys.dm_exec_requests and sys.dm_exec_sessions views, if you have VIEW DATABASE STATE
permission on the database, you see all executing sessions on the database; otherwise, you see only the current session.
See also
Introduction to SQL Database
Operating the Query Store in Azure SQL Database
7/6/2018 • 2 minutes to read • Edit Online
Query Store in Azure is a fully managed database feature that continuously collects and presents detailed historic
information about all queries. You can think about Query Store as similar to an airplane's flight data recorder that
significantly simplifies query performance troubleshooting both for cloud and on-premises customers. This article
explains specific aspects of operating Query Store in Azure. Using this pre-collected query data, you can quickly
diagnose and resolve performance problems and thus spend more time focusing on their business.
Query Store has been globally available in Azure SQL Database since November, 2015. Query Store is the
foundation for performance analysis and tuning features, such as SQL Database Advisor and Performance
Dashboard. At the moment of publishing this article, Query Store is running in more than 200,000 user databases
in Azure, collecting query-related information for several months, without interruption.
IMPORTANT
Microsoft is in the process of activating Query Store for all Azure SQL databases (existing and new).
MAX_STORAGE_SIZE_MB Specifies the limit for the 100 Enforced for new databases
data space that Query Store
can take inside the customer
database
QUERY_CAPTURE_MODE Specifies whether all queries AUTO Enforced for all databases
or only a subset of queries
are tracked
IMPORTANT
These defaults are automatically applied in the final stage of Query Store activation in all Azure SQL databases (see preceding
important note). After this light up, Azure SQL Database won’t be changing configuration values set by customers, unless
they negatively impact primary workload or reliable operations of the Query Store.
If you want to stay with your custom settings, use ALTER DATABASE with Query Store options to revert
configuration to the previous state. Check out Best Practices with the Query Store in order to learn how top chose
optimal configuration parameters.
Next steps
SQL Database Performance Insight
Additional resources
For more information check out the following articles:
A flight data recorder for your database
Monitoring Performance By Using the Query Store
Query Store Usage Scenarios
Monitor In-Memory OLTP storage
6/22/2018 • 2 minutes to read • Edit Online
When using In-Memory OLTP, data in memory-optimized tables and table variables resides in In-Memory OLTP
storage. Each Premium and Business Critical service tier has a maximum In-Memory OLTP storage size. See DTU -
based resource limits - single database, DTU -based resource limits - elastic pools,vCore-based resource limits -
single databases and vCore-based resource limits - elastic pools.
Once this limit is exceeded, insert and update operations may start failing with error 41823 for standalone
databases and error 41840 for elastic pools. At that point you need to either delete data to reclaim memory, or
upgrade the performance tier of your database.
Determine whether data fits within the In-Memory OLTP storage cap
Determine the storage caps of the different service tiers. See DTU -based resource limits - single database, DTU -
based resource limits - elastic pools,vCore-based resource limits - single databases and vCore-based resource
limits - elastic pools.
Estimating memory requirements for a memory-optimized table works the same way for SQL Server as it does in
Azure SQL Database. Take a few minutes to review that article on MSDN.
Table and table variable rows, as well as indexes, count toward the max user data size. In addition, ALTER TABLE
needs enough room to create a new version of the entire table and its indexes.
NOTE
In rare cases, errors 41823 and 41840 can be transient, meaning there is enough available In-Memory OLTP storage, and
retrying the operation succeeds. We therefore recommend to both monitor the overall available In-Memory OLTP storage
and to retry when first encountering error 41823 or 41840. For more information about retry logic, see Conflict Detection
and Retry Logic with In-Memory OLTP.
Next steps
For monitoring guidance, see Monitoring Azure SQL Database using dynamic management views.
Extended events in SQL Database
6/27/2018 • 5 minutes to read • Edit Online
This topic explains how the implementation of extended events in Azure SQL Database is slightly different
compared to extended events in Microsoft SQL Server.
SQL Database V12 gained the extended events feature in the second half of calendar 2015.
SQL Server has had extended events since 2008.
The feature set of extended events on SQL Database is a robust subset of the features on SQL Server.
XEvents is an informal nickname that is sometimes used for 'extended events' in blogs and other informal
locations.
Additional information about extended events, for Azure SQL Database and Microsoft SQL Server, is available at:
Quick Start: Extended events in SQL Server
Extended Events
Prerequisites
This topic assumes you already have some knowledge of:
Azure SQL Database service.
Extended events in Microsoft SQL Server.
The bulk of our documentation about extended events applies to both SQL Server and SQL Database.
Prior exposure to the following items is helpful when choosing the Event File as the target:
Azure Storage service
PowerShell
Using Azure PowerShell with Azure Storage - Provides comprehensive information about PowerShell
and the Azure Storage service.
Code samples
Related topics provide two code samples:
Ring Buffer target code for extended events in SQL Database
Short simple Transact-SQL script.
We emphasize in the code sample topic that, when you are done with a Ring Buffer target, you should
release its resources by executing an alter-drop ALTER EVENT SESSION ... ON DATABASE DROP TARGET ...;
statement. Later you can add another instance of Ring Buffer by
ALTER EVENT SESSION ... ON DATABASE ADD TARGET ... .
Event File target code for extended events in SQL Database
Phase 1 is PowerShell to create an Azure Storage container.
Phase 2 is Transact-SQL that uses the Azure Storage container.
Transact-SQL differences
When you execute the CREATE EVENT SESSION command on SQL Server, you use the ON SERVER
clause. But on SQL Database you use the ON DATABASE clause instead.
The ON DATABASE clause also applies to the ALTER EVENT SESSION and DROP EVENT SESSION
Transact-SQL commands.
A best practice is to include the event session option of STARTUP_STATE = ON in your CREATE EVENT
SESSION or ALTER EVENT SESSION statements.
The = ON value supports an automatic restart after a reconfiguration of the logical database due to a
failover.
NAME OF
CATALOG VIEW DESCRIPTION
sys.database_event_session_targets Returns a row for each event target for an event session.
sys.database_event_sessions Returns a row for each event session in the SQL Database
database.
In Microsoft SQL Server, similar catalog views have names that include .server_ instead of .database_. The name
pattern is like sys.server_event_%.
sys.dm_xe_database_session_object_columns Shows the configuration values for objects that are bound to
a session.
sys.dm_xe_database_sessions Returns a row for each event session that is scoped to the
current database.
In Microsoft SQL Server, similar catalog views are named without the _database portion of the name, such as:
sys.dm_xe_sessions, instead of name
sys.dm_xe_database_sessions.
DMVs common to both
For extended events there are additional DMVs that are common to both Azure SQL Database and Microsoft
SQL Server:
sys.dm_xe_map_values
sys.dm_xe_object_columns
sys.dm_xe_objects
sys.dm_xe_packages
SELECT
o.object_type,
p.name AS [package_name],
o.name AS [db_object_name],
o.description AS [db_obj_description]
FROM
sys.dm_xe_objects AS o
INNER JOIN sys.dm_xe_packages AS p ON p.guid = o.package_guid
WHERE
o.object_type in
(
'action', 'event', 'target'
)
ORDER BY
o.object_type,
p.name,
o.name;
Restrictions
There are a couple of security-related differences befitting the cloud environment of SQL Database:
Extended events are founded on the single-tenant isolation model. An event session in one database cannot
access data or events from another database.
You cannot issue a CREATE EVENT SESSION statement in the context of the master database.
Permission model
You must have Control permission on the database to issue a CREATE EVENT SESSION statement. The
database owner (dbo) has Control permission.
Storage container authorizations
The SAS token you generate for your Azure Storage container must specify rwl for the permissions. The rwl
value provides the following permissions:
Read
Write
List
Performance considerations
There are scenarios where intensive use of extended events can accumulate more active memory than is healthy
for the overall system. Therefore the Azure SQL Database system dynamically sets and adjusts limits on the
amount of active memory that can be accumulated by an event session. Many factors go into the dynamic
calculation.
If you receive an error message that says a memory maximum was enforced, some corrective actions you can take
are:
Run fewer concurrent event sessions.
Through your CREATE and ALTER statements for event sessions, reduce the amount of memory you specify
on the MAX_MEMORY clause.
Network latency
The Event File target might experience network latency or failures while persisting data to Azure Storage blobs.
Other events in SQL Database might be delayed while they wait for the network communication to complete. This
delay can slow your workload.
To mitigate this performance risk, avoid setting the EVENT_RETENTION_MODE option to
NO_EVENT_LOSS in your event session definitions.
Related links
Using Azure PowerShell with Azure Storage.
Azure Storage Cmdlets
Using Azure PowerShell with Azure Storage - Provides comprehensive information about PowerShell and the
Azure Storage service.
How to use Blob storage from .NET
CREATE CREDENTIAL (Transact-SQL )
CREATE EVENT SESSION (Transact-SQL )
Jonathan Kehayias' blog posts about extended events in Microsoft SQL Server
The Azure Service Updates webpage, narrowed by parameter to Azure SQL Database:
https://azure.microsoft.com/updates/?service=sql-database
Other code sample topics for extended events are available at the following links. However, you must routinely
check any sample to see whether the sample targets Microsoft SQL Server versus Azure SQL Database. Then you
can decide whether minor changes are needed to run the sample.
Event File target code for extended events in SQL
Database
5/23/2018 • 9 minutes to read • Edit Online
You want a complete code sample for a robust way to capture and report information for an extended event.
In Microsoft SQL Server, the Event File target is used to store event outputs into a local hard drive file. But such
files are not available to Azure SQL Database. Instead we use the Azure Storage service to support the Event File
target.
This topic presents a two-phase code sample:
PowerShell, to create an Azure Storage container in the cloud.
Transact-SQL:
To assign the Azure Storage container to an Event File target.
To create and start the event session, and so on.
Prerequisites
An Azure account and subscription. You can sign up for a free trial.
Any database you can create a table in.
Optionally you can create an AdventureWorksLT demonstration database in minutes.
SQL Server Management Studio (ssms.exe), ideally its latest monthly update version. You can download
the latest ssms.exe from:
Topic titled Download SQL Server Management Studio.
A direct link to the download.
You must have the Azure PowerShell modules installed.
The modules provide commands such as - New-AzureStorageAccount.
## TODO: Before running, find all 'TODO' and make each edit!!
cls;
#--------------- 1 -----------------------
'Script assumes you have already logged your PowerShell session into Azure.
But if not, run Connect-AzureRmAccount (or Connect-AzureRmAccount), just one time.';
#Connect-AzureRmAccount; # Same as Connect-AzureRmAccount.
#-------------- 2 ------------------------
'
TODO: Edit the values assigned to these variables, especially the first few!
';
$subscriptionName = 'YOUR_SUBSCRIPTION_NAME';
$resourceGroupName = 'YOUR_RESOURCE-GROUP-NAME';
$policySasExpiryTime = '2018-08-28T23:44:56Z';
$policySasStartTime = '2017-10-01';
#--------------- 3 -----------------------
#-------------- 4 ------------------------
'
Clean up the old Azure Storage Account after any previous run,
before continuing this new run.';
If ($storageAccountName)
{
Remove-AzureRmStorageAccount `
-Name $storageAccountName `
-ResourceGroupName $resourceGroupName;
}
#--------------- 5 -----------------------
[System.DateTime]::Now.ToString();
'
Create a storage account.
This might take several minutes, will beep when ready.
...PLEASE WAIT...';
New-AzureRmStorageAccount `
-Name $storageAccountName `
-Location $storageAccountLocation `
-ResourceGroupName $resourceGroupName `
-SkuName 'Standard_LRS';
[System.DateTime]::Now.ToString();
[System.Media.SystemSounds]::Beep.Play();
'
Get the access key for your storage account.
';
$accessKey_ForStorageAccount = `
(Get-AzureRmStorageAccountKey `
-Name $storageAccountName `
-ResourceGroupName $resourceGroupName
).Value[0];
"`$accessKey_ForStorageAccount = $accessKey_ForStorageAccount";
#--------------- 6 -----------------------
# The context will be needed to create a container within the storage account.
'Create a context object from the storage account and its primary access key.
';
$context = New-AzureStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $accessKey_ForStorageAccount;
$containerObjectInStorageAccount = New-AzureStorageContainer `
-Name $containerName `
-Context $context;
New-AzureStorageContainerStoredAccessPolicy `
-Container $containerName `
-Context $context `
-Policy $policySasToken `
-Permission $policySasPermission `
-ExpiryTime $policySasExpiryTime `
-StartTime $policySasStartTime;
'
Generate a SAS token for the container.
';
Try
{
$sasTokenWithPolicy = New-AzureStorageContainerSASToken `
-Name $containerName `
-Context $context `
-Policy $policySasToken;
}
Catch
{
$Error[0].Exception.ToString();
}
#-------------- 7 ------------------------
'Display the values that YOU must edit into the Transact-SQL script next!:
';
"storageAccountName: $storageAccountName";
"containerName: $containerName";
"sasTokenWithPolicy: $sasTokenWithPolicy";
'
REMINDER: sasTokenWithPolicy here might start with "?" character, which you must exclude from Transact-SQL.
';
'
(Later, return here to delete your Azure Storage account. See the preceding Remove-AzureRmStorageAccount -
Name $storageAccountName)';
'
Now shift to the Transact-SQL portion of the two-part code sample!';
# EOFile
Take note of the few named values that the PowerShell script prints when it ends. You must edit those values into
the Transact-SQL script that follows as phase 2.
WARNING
The SAS key value generated by the preceding PowerShell script might begin with a '?' (question mark). When you use the
SAS key in the following T-SQL script, you must remove the leading '?'. Otherwise your efforts might be blocked by security.
Transact-SQL code
---- TODO: First, run the earlier PowerShell portion of this two-part code sample.
---- TODO: Second, find every 'TODO' in this Transact-SQL file, and edit each.
---- Transact-SQL code for Event File target on Azure SQL Database.
IF EXISTS
(SELECT * FROM sys.objects
WHERE type = 'U' and name = 'gmTabEmployee')
BEGIN
DROP TABLE gmTabEmployee;
END
GO
IF NOT EXISTS
(SELECT * FROM sys.symmetric_keys
WHERE symmetric_key_id = 101)
BEGIN
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '0C34C960-6621-4682-A123-C7EA08E3FC46' -- Or any newid().
END
GO
IF EXISTS
(SELECT * FROM sys.database_scoped_credentials
-- TODO: Assign AzureStorageAccount name, and the associated Container name.
-- TODO: Assign AzureStorageAccount name, and the associated Container name.
WHERE name = 'https://gmstorageaccountxevent.blob.core.windows.net/gmcontainerxevent')
BEGIN
DROP DATABASE SCOPED CREDENTIAL
-- TODO: Assign AzureStorageAccount name, and the associated Container name.
[https://gmstorageaccountxevent.blob.core.windows.net/gmcontainerxevent] ;
END
GO
CREATE
DATABASE SCOPED
CREDENTIAL
-- use '.blob.', and not '.queue.' or '.table.' etc.
-- TODO: Assign AzureStorageAccount name, and the associated Container name.
[https://gmstorageaccountxevent.blob.core.windows.net/gmcontainerxevent]
WITH
IDENTITY = 'SHARED ACCESS SIGNATURE', -- "SAS" token.
-- TODO: Paste in the long SasToken string here for Secret, but exclude any leading '?'.
SECRET = 'sv=2014-02-14&sr=c&si=gmpolicysastoken&sig=EjAqjo6Nu5xMLEZEkMkLbeF7TD9v1J8DNB2t8gOKTts%3D'
;
GO
IF EXISTS
(SELECT * from sys.database_event_sessions
WHERE name = 'gmeventsessionname240b')
BEGIN
DROP
EVENT SESSION
gmeventsessionname240b
ON DATABASE;
END
GO
CREATE
EVENT SESSION
gmeventsessionname240b
ON DATABASE
ADD EVENT
sqlserver.sql_statement_starting
(
ACTION (sqlserver.sql_text)
WHERE statement LIKE 'UPDATE gmTabEmployee%'
)
ADD TARGET
package0.event_file
(
-- TODO: Assign AzureStorageAccount name, and the associated Container name.
-- Also, tweak the .xel file name at end, if you like.
SET filename =
'https://gmstorageaccountxevent.blob.core.windows.net/gmcontainerxevent/anyfilenamexel242b.xel'
)
WITH
(MAX_MEMORY = 10 MB,
MAX_DISPATCH_LATENCY = 3 SECONDS)
;
GO
ALTER
EVENT SESSION
gmeventsessionname240b
ON DATABASE
STATE = START;
GO
UPDATE gmTabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 2
WHERE EmployeeDescr = 'Jane Doe';
UPDATE gmTabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 13
WHERE EmployeeDescr = 'Jane Doe';
ALTER
EVENT SESSION
gmeventsessionname240b
ON DATABASE
STATE = STOP;
GO
SELECT
*, 'CLICK_NEXT_CELL_TO_BROWSE_ITS_RESULTS!' as [CLICK_NEXT_CELL_TO_BROWSE_ITS_RESULTS],
CAST(event_data AS XML) AS [event_data_XML] -- TODO: In ssms.exe results grid, double-click this
cell!
FROM
sys.fn_xe_file_target_read_file
(
-- TODO: Fill in Storage Account name, and the associated Container name.
'https://gmstorageaccountxevent.blob.core.windows.net/gmcontainerxevent/anyfilenamexel242b',
null, null, null
);
GO
DROP
EVENT SESSION
gmeventsessionname240b
ON DATABASE;
GO
If the target fails to attach when you run, you must stop and restart the event session:
Output
When the Transact-SQL script completes, click a cell under the event_data_XML column header. One element is
displayed which shows one UPDATE statement.
Here is one element that was generated during testing:
UPDATE gmTabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 2
WHERE EmployeeDescr = 'Jane Doe';
UPDATE gmTabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 13
WHERE EmployeeDescr = 'Jane Doe';
The preceding Transact-SQL script used the following system function to read the event_file:
sys.fn_xe_file_target_read_file (Transact-SQL )
An explanation of advanced options for the viewing of data from extended events is available at:
Advanced Viewing of Target Data from Extended Events
Converting the code sample to run on SQL Server
Suppose you wanted to run the preceding Transact-SQL sample on Microsoft SQL Server.
For simplicity, you would want to completely replace use of the Azure Storage container with a simple file such
as C:\myeventdata.xel. The file would be written to the local hard drive of the computer that hosts SQL
Server.
You would not need any kind of Transact-SQL statements for CREATE MASTER KEY and CREATE
CREDENTIAL.
In the CREATE EVENT SESSION statement, in its ADD TARGET clause, you would replace the Http
value assigned made to filename= with a full path string like C:\myfile.xel.
No Azure Storage account need be involved.
More information
For more info about accounts and containers in the Azure Storage service, see:
How to use Blob storage from .NET
Naming and Referencing Containers, Blobs, and Metadata
Working with the Root Container
Lesson 1: Create a stored access policy and a shared access signature on an Azure container
Lesson 2: Create a SQL Server credential using a shared access signature
Extended Events for Microsoft SQL Server
Ring Buffer target code for extended events in SQL
Database
5/23/2018 • 5 minutes to read • Edit Online
You want a complete code sample for the easiest quick way to capture and report information for an extended
event during a test. The easiest target for extended event data is the Ring Buffer target.
This topic presents a Transact-SQL code sample that:
1. Creates a table with data to demonstrate with.
2. Creates a session for an existing extended event, namely sqlserver.sql_statement_starting.
The event is limited to SQL statements that contain a particular Update string: statement LIKE
'%UPDATE tabEmployee%'.
Chooses to send the output of the event to a target of type Ring Buffer, namely package0.ring_buffer.
3. Starts the event session.
4. Issues a couple of simple SQL UPDATE statements.
5. Issues a SQL SELECT statement to retrieve event output from the Ring Buffer.
sys.dm_xe_database_session_targets and other dynamic management views (DMVs) are joined.
6. Stops the event session.
7. Drops the Ring Buffer target, to release its resources.
8. Drops the event session and the demo table.
Prerequisites
An Azure account and subscription. You can sign up for a free trial.
Any database you can create a table in.
Optionally you can create an AdventureWorksLT demonstration database in minutes.
SQL Server Management Studio (ssms.exe), ideally its latest monthly update version. You can download
the latest ssms.exe from:
Topic titled Download SQL Server Management Studio.
A direct link to the download.
Code sample
With very minor modification, the following Ring Buffer code sample can be run on either Azure SQL Database or
Microsoft SQL Server. The difference is the presence of the node '_database' in the name of some dynamic
management views (DMVs), used in the FROM clause in Step 5. For example:
sys.dm_xe_database_session_targets
sys.dm_xe_session_targets
GO
---- Transact-SQL.
---- Step set 1.
SET NOCOUNT ON;
GO
IF EXISTS
(SELECT * FROM sys.objects
WHERE type = 'U' and name = 'tabEmployee')
BEGIN
DROP TABLE tabEmployee;
END
GO
IF EXISTS
(SELECT * from sys.database_event_sessions
WHERE name = 'eventsession_gm_azuresqldb51')
BEGIN
DROP EVENT SESSION eventsession_gm_azuresqldb51
ON DATABASE;
END
GO
CREATE
EVENT SESSION eventsession_gm_azuresqldb51
ON DATABASE
ADD EVENT
sqlserver.sql_statement_starting
(
ACTION (sqlserver.sql_text)
WHERE statement LIKE '%UPDATE tabEmployee%'
)
ADD TARGET
package0.ring_buffer
(SET
max_memory = 500 -- Units of KB.
);
GO
UPDATE tabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 1015;
SELECT
se.name AS [session-name],
ev.event_name,
ac.action_name,
st.target_name,
se.session_source,
st.target_data,
CAST(st.target_data AS XML) AS [target_data_XML]
FROM
sys.dm_xe_database_session_event_actions AS ac
UPDATE tabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 102;
UPDATE tabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 1015;
UPDATE tabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 102;
UPDATE tabEmployee
SET EmployeeKudosCount = EmployeeKudosCount + 1015;
The definition of your event session is updated, but not dropped. Later you can add another instance of the Ring
Buffer to your event session:
Overview
This article shows you how to set up Azure SQL Database and Data Warehouse alerts using the Azure portal. This
article also provides best practices for setting alert periods.
You can receive an alert based on monitoring metrics for, or events on, your Azure services.
Metric values - The alert triggers when the value of a specified metric crosses a threshold you assign in either
direction. That is, it triggers both when the condition is first met and then afterwards when that condition is no
longer being met.
Activity log events - An alert can trigger on every event, or, only when a certain number of events occur.
You can configure an alert to do the following when it triggers:
send email notifications to the service administrator and co-administrators
send email to additional emails that you specify.
call a webhook
You can configure and get information about alert rules using
Azure portal
PowerShell
command-line interface (CLI)
Azure Monitor REST API
SQL DW ONLY: Select Monitoring under the COMMON TASKS section. Click the DWU Usage
graph.
3. Select the Add alert command and fill in the fields.
4. Name your alert rule, and choose a Description, which also shows in notification emails.
5. Select the Metric you want to monitor, then choose a Condition and Threshold value for the metric. Also
choose the Period of time that the metric rule must be satisfied before the alert triggers. So for example, if you
use the period "PT5M" and your alert looks for CPU above 80%, the alert triggers when the average CPU has
been above 80% for 5 minutes. Once the first trigger occurs, it again triggers when the average CPU is below
80% over 5 minutes. The CPU measurement occurs every 1 minute. Consult the table below for supported
time windows and the aggregation type that each alert uses- not all alerts use the average value.
6. Check Email owners... if you want administrators and co-administrators to be emailed when the alert fires.
7. If you want additional emails to receive a notification when the alert fires, add them in the Additional
Administrator email(s) field. Separate multiple emails with semi-colons -
[email protected];[email protected]
8. Put in a valid URI in the Webhook field if you want it called when the alert fires.
9. Select OK when done to create the alert.
Within a few minutes, the alert is active and triggers as previously described.
Next steps
Get an overview of Azure monitoring including the types of information you can collect and monitor.
Learn more about configuring webhooks in alerts.
Get an overview of diagnostic logs and collect detailed high-frequency metrics on your service.
Get an overview of metrics collection to make sure your service is available and responsive.
Troubleshoot connection issues to Azure SQL
Database
5/23/2018 • 4 minutes to read • Edit Online
When the connection to Azure SQL Database fails, you receive error messages. This article is a centralized topic
that helps you troubleshoot Azure SQL Database connectivity issues. It introduces the common causes of
connection issues, recommends a troubleshooting tool that helps you identity the problem, and provides
troubleshooting steps to solve transient errors and persistent or non-transient errors.
If you encounter the connection issues, try the troubleshoot steps that are described in this article.
If your Azure issue is not addressed in this article, visit the Azure forums on MSDN and Stack Overflow. You can
post your issue in these forums, or post to @AzureSupport on Twitter. You also can submit an Azure support
request. To submit a support request, on the Azure support page, select Get support.
Cause
Connection problems may be caused by any of the following:
Failure to apply best practices and design guidelines during the application design process. See SQL Database
Development Overview to get started.
Azure SQL Database reconfiguration
Firewall settings
Connection time-out
Incorrect login information
Maximum limit reached on some Azure SQL Database resources
Generally, connection issues to Azure SQL Database can be classified as follows:
Transient errors (short-lived or intermittent)
Persistent or non-transient errors (errors that regularly recur)
Error code 40613: "Database <x> on server <y> is not currently available. Please retry the connection later. If
the problem persists, contact customer support, and provide them the session tracing ID of <z>"
NOTE
This error message is typically transient (short-lived).
This error occurs when the Azure database is being moved (or reconfigured) and your application loses its
connection to the SQL database. SQL database reconfiguration events occur because of a planned event (for
example, a software upgrade) or an unplanned event (for example, a process crash, or load balancing). Most
reconfiguration events are generally short-lived and should be completed in less than 60 seconds at most.
However, these events can occasionally take longer to finish, such as when a large transaction causes a long-
running recovery.
Steps to resolve transient connectivity issues
1. Check the Microsoft Azure Service Dashboard for any known outages that occurred during the time during
which the errors were reported by the application.
2. Applications that connect to a cloud service such as Azure SQL Database should expect periodic
reconfiguration events and implement retry logic to handle these errors instead of surfacing these as
application errors to users. Review the Transient errors section and the best practices and design guidelines at
SQL Database Development Overview for more information and general retry strategies. Then, see code
samples at Connection Libraries for SQL Database and SQL Server for specifics.
3. As a database approaches its resource limits, it can seem to be a transient connectivity issue. See
Troubleshooting Performance Issues.
4. If connectivity problems continue, or if the duration for which your application encounters the error exceeds 60
seconds or if you see multiple occurrences of the error in a given day, file an Azure support request by selecting
Get Support on the Azure Support site.
Next steps
Troubleshoot Azure SQL Database performance issues
Search the documentation on Microsoft Azure
View the latest updates to the Azure SQL Database service
Additional resources
SQL Database Development Overview
General transient fault-handling guidance
Connection libraries for SQL Database and SQL Server
Managing Azure SQL databases using Azure
Automation
9/13/2018 • 2 minutes to read • Edit Online
This guide will introduce you to the Azure Automation service, and how it can be used to simplify management of
your Azure SQL databases.
Next steps
Now that you've learned the basics of Azure Automation and how it can be used to manage Azure SQL databases,
follow these links to learn more about Azure Automation.
Azure Automation Overview
My first runbook
Azure Automation learning map
Azure Automation: Your SQL Agent in the Cloud
Manage groups of databases with Elastic Database
Jobs
9/12/2018 • 11 minutes to read • Edit Online
Elastic Database Jobs provide the ability to run one or more T-SQL scripts in parallel, across a large number of
databases, on a schedule or on-demand.
Run jobs against any combination of databases: one or more individual databases, all databases on a server,
all databases in an elastic pool, or shardmap, with the added flexibility to include or exclude any specific database.
Jobs can run across multiple servers, multiple pools, and can even run against databases in different
subscriptions. Servers and pools are dynamically enumerated at runtime, so jobs run against all databases that
exist in the target group at the time of execution.
The following image shows a job agent executing jobs across the different types of target groups:
Elastic Job agent The Azure resource you create to run and manage Jobs.
Job database An Azure SQL database the job agent uses to store job
related data, job definitions, etc.
Target group The set of servers, pools, databases, and shard maps to run a
job against.
Target group
A target group defines the set of databases a job step will execute on. A target group can contain any number and
combination of the following:
Azure SQL server - if a server is specified, all databases that exist in the server at the time of the job execution
are part of the group. The master database credential must be provided so that the group can be enumerated
and updated prior to job execution.
Elastic pool - if an elastic pool is specified, all databases that are in the elastic pool at the time of the job
execution are part of the group. As for a server, the master database credential must be provided so that the
group can be updated prior to the job execution.
Single database - specify one or more individual databases to be part of the group.
Shardmap - databases of a shardmap.
TIP
At the moment of job execution, dynamic enumeration re-evaluates the set of databases in target groups that include
servers or pools. Dynamic enumeration ensures that jobs run across all databases that exist in the server or pool at
the time of job execution. Re-evaluating the list of databases at runtime is specifically useful for scenarios where pool or
server membership changes frequently.
Pools and single databases can be specified as included or excluded from the group. This enables creating a target
group with any combination of databases. For example, you can add a server to a target group, but exclude
specific databases in an elastic pool (or exclude an entire pool).
A target group can include databases in multiple subscriptions, and across multiple regions. Note that cross-
region executions have higher latency than executions within the same region.
The following examples show how different target group definitions are dynamically enumerated at the moment
of job execution to determine which databases the job will run:
Example 1 shows a target group that consists of a list of individual databases. When a job step is executed using
this target group, the job step's action will be executed in each of those databases.
Example 2 shows a target group that contains an Azure SQL Server as a target. When a job step is executed
using this target group, the server is dynamically enumerated to determine the list of databases that are currently
in the server. The job step's action will be executed in each of those databases.
Example 3 shows a similar target group as Example 2, but an individual database is specifically excluded. The job
step's action will not be executed in the excluded database.
Example 4 shows a target group that contains an elastic pool as a target. Similar to Example 2, the pool will be
dynamically enumerated at job run time to determine the list of databases in the pool.
Example 5 and Example 6 show advanced scenarios where Azure SQL Servers, elastic pools, and databases, can
be combined using include and exclude rules.
Example 7 shows that the shards in a shard map can also be evaluated at job run time.
Job
A job is a unit of work that is executed on a schedule or as a one-time job. A job consists of one or more job steps.
Job step
Each job step specifies a T-SQL script to execute, one or more target groups to run the T-SQL script against, and
the credentials the job agent needs to connect to the target database. Each job step has customizable timeout and
retry policies, and can optionally specify output parameters.
Job output
The outcome of a job's steps on each target database are recorded in detail, and script output can be captured to a
specified table. You can specify a database to save any data returned from a job.
Job history
Job execution history is stored in the Job database. A system cleanup job purges execution history that is older
than 45 days. To remove history less than 45 days old, call the sp_purge_history stored procedure in the Job
database.
Scope Any number of Azure SQL databases Any single database in the same SQL
and/or data warehouses in the same Server instance as the SQL agent.
Azure cloud as the job agent. Targets
can be in different logical servers,
subscriptions, and/or regions.
Supported APIs and Tools Portal, PowerShell, T-SQL, Azure T-SQL, SQL Server Management Studio
Resource Manager (SSMS)
Similarly, a script must be able to execute successfully by logically testing for and countering any conditions it
finds.
Next steps
Create and manage Elastic Jobs using PowerShell
Create and manage Elastic Jobs using Transact-SQL (T-SQL )
Create an Elastic Job agent using PowerShell
7/13/2018 • 8 minutes to read • Edit Online
Elastic jobs enable the running of one or more Transact-SQL (T-SQL ) scripts in parallel across many databases.
In this tutorial you learn the steps required to run a query across multiple databases:
Create an Elastic Job agent
Create job credentials so that jobs can execute scripts on its targets
Define the targets (servers, elastic pools, databases, shard maps) you want to run the job against
Create database scoped credentials in the target databases so the agent connect and execute jobs
Create a job
Add job steps to a job
Start execution of a job
Monitor a job
Prerequisites
If you don't have already have an Azure subscription, create a free account before you begin.
Install the AzureRM.Sql 4.8.1-preview module to get the latest Elastic Job cmdlets. Run the following commands
in PowerShell with administrative access.
# Installs the latest PackageManagement powershell package which PowershellGet v1.6.5 is dependent on
Find-Package PackageManagement -RequiredVersion 1.1.7.2 | Install-Package -Force
# Installs the latest PowershellGet module which adds the -AllowPrerelease flag to Install-Module
Find-Package PowerShellGet -RequiredVersion 1.6.5 | Install-Package -Force
# Places AzureRM.Sql preview cmdlets side by side with existing AzureRM.Sql version
Install-Module -Name AzureRM.Sql -AllowPrerelease -RequiredVersion 4.8.1-preview -Force
# Confirm if module successfully imported - if the imported version is 4.8.1, then continue
Get-Module AzureRM.Sql
# Create a server
Write-Output "Creating a server..."
$AgentServerName = Read-Host "Please enter an agent server name"
$AgentServerName = $AgentServerName + "-" + [guid]::NewGuid()
$AdminLogin = Read-Host "Please enter the server admin name"
$AdminPassword = Read-Host "Please enter the server admin password"
$AdminPasswordSecure = ConvertTo-SecureString -String $AdminPassword -AsPlainText -Force
$AdminCred = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList $AdminLogin,
$AdminPasswordSecure
$AgentServer = New-AzureRmSqlServer -ResourceGroupName $ResourceGroupName -Location $Location -ServerName
$AgentServerName -ServerVersion "12.0" -SqlAdministratorCredentials ($AdminCred)
# Create a target server and some sample databases - uses the same admin credential as the agent server just
for simplicity
Write-Output "Creating target server..."
$TargetServerName = Read-Host "Please enter a target server name"
$TargetServerName = $TargetServerName + "-" + [guid]::NewGuid()
$TargetServer = New-AzureRmSqlServer -ResourceGroupName $ResourceGroupName -Location $Location -ServerName
$TargetServerName -ServerVersion "12.0" -SqlAdministratorCredentials ($AdminCred)
Create job credentials so that jobs can execute scripts on its targets
Jobs use database scoped credentials to connect to the target databases specified by the target group upon
execution. These database scoped credentials are also used to connect to the master database to enumerate all the
databases in a server or an elastic pool, when either of these are used as the target group member type.
The database scoped credentials must be created in the job database.
All target databases must have a login with sufficient permissions for the job to complete successfully.
In addition to the credentials in the image, note the addition of the GRANT commands in the following script.
These permissions are required for the script we chose for this example job. Because the example creates a new
table in the targeted databases, each target db needs the proper permissions to successfully run.
To create the required job credentials (in the job database), run the following script:
# In the master database (target server)
# - Create the master user login
# - Create the master user from master user login
# - Create the job user login
$Params = @{
'Database' = 'master'
'ServerInstance' = $TargetServer.ServerName + '.database.windows.net'
'Username' = $AdminLogin
'Password' = $AdminPassword
'OutputSqlErrors' = $true
'Query' = "CREATE LOGIN masteruser WITH PASSWORD='password!123'"
}
Invoke-SqlCmd @Params
$Params.Query = "CREATE USER masteruser FROM LOGIN masteruser"
Invoke-SqlCmd @Params
$Params.Query = "CREATE LOGIN jobuser WITH PASSWORD='password!123'"
Invoke-SqlCmd @Params
$TargetDatabases | % {
$Params.Database = $_
$Params.Query = $CreateJobUserScript
Invoke-SqlCmd @Params
$Params.Query = $GrantAlterSchemaScript
Invoke-SqlCmd @Params
$Params.Query = $GrantCreateScript
Invoke-SqlCmd @Params
}
Define the target databases you want to run the job against
A target group defines the set of one or more databases a job step will execute on.
The following snippet creates two target groups: ServerGroup, and ServerGroupExcludingDb2. ServerGroup
targets all databases that exist on the server at the time of execution, and ServerGroupExcludingDb2 targets all
databases on the server, except TargetDb2:
Write-Output "Creating test target groups..."
# Create ServerGroup target group
$ServerGroup = $JobAgent | New-AzureRmSqlElasticJobTargetGroup -Name 'ServerGroup'
$ServerGroup | Add-AzureRmSqlElasticJobTarget -ServerName $TargetServerName -RefreshCredentialName
$MasterCred.CredentialName
Create a job
Write-Output "Creating a new job"
$JobName = "Job1"
$Job = $JobAgent | New-AzureRmSqlElasticJob -Name $JobName -RunOnce
$Job
After successful completion you should see two new tables in TargetDb1, and only one new table in TargetDb2:
Monitor status of job executions
The following snippets get job execution details:
Clean up resources
Delete the Azure resources created in this tutorial by deleting the resource group.
TIP
If you plan to continue to work with these jobs, do not clean up the resources created in this article. If you do not plan to
continue, use the following steps to delete all resources created in this article.
Next steps
In this tutorial, you ran a Transact-SQL script against a set of databases. You learned how to do the following tasks:
Create an Elastic Job agent
Create job credentials so that jobs can execute scripts on its targets
Define the targets (servers, elastic pools, databases, shard maps) you want to run the job against
Create database scoped credentials in the target databases so the agent connect and execute jobs
Create a job
Add a job step to the job
Start an execution of the job
Monitor the job
Manage Elastic Jobs using Transact-SQL
Use Transact-SQL (T-SQL) to create and manage
Elastic Database Jobs
9/6/2018 • 39 minutes to read • Edit Online
This article provides many example scenarios to get started working with Elastic Jobs using T-SQL.
The examples use the stored procedures and views available in the job database.
Transact-SQL (T-SQL ) is used to create, configure, execute, and manage jobs. Creating the Elastic Job agent is not
supported in T-SQL, so you must first create an Elastic Job agent using the portal, or PowerShell.
--Connect to the job database specified when creating the job agent
-- Create a db master key if one does not already exist, using your own password.
CREATE MASTER KEY ENCRYPTION BY PASSWORD='<EnterStrongPasswordHere>';
--View the recently created target group and target group members
SELECT * FROM jobs.target_groups WHERE target_group_name='ServerGroup1';
SELECT * FROM jobs.target_group_members WHERE target_group_name='ServerGroup1';
--Connect to the job database specified when creating the job agent
--View the recently created target group and target group members
SELECT * FROM [jobs].target_groups WHERE target_group_name = N'ServerGroup';
SELECT * FROM [jobs].target_group_members WHERE target_group_name = N'ServerGroup';
-- View the recently created target group and target group members
SELECT * FROM jobs.target_groups WHERE target_group_name = N'PoolGroup';
SELECT * FROM jobs.target_group_members WHERE target_group_name = N'PoolGroup';
--Connect to the job database specified when creating the job agent
--Connect to the job database specified when creating the job agent
--Connect to the job database specified when creating the job agent
--Connect to the job database specified when creating the job agent
EXEC jobs.sp_update_job
@job_name='ResultsJob',
@enabled=1,
@schedule_interval_type='Minutes',
@schedule_interval_count=15
Cancel a job
The following example shows how to cancel a job.
Connect to the job database and run the following command:
--Connect to the job database specified when creating the job agent
--Connect to the job database specified when creating the job agent
-- Delete history of a specific job’s executions older than the specified date
EXEC jobs.sp_purge_jobhistory @job_name='ResultPoolsJob', @oldest_date='2016-07-01 00:00:00'
sp_add_job
Adds a new job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job. The name must be unique and cannot contain the percent (%) character. job_name is
nvarchar(128), with no default.
[ @description = ] 'description'
The description of the job. description is nvarchar(512), with a default of NULL. If description is omitted, an empty
string is used.
[ @enabled = ] enabled
Whether the job’s schedule is enabled. Enabled is bit, with a default of 0 (disabled). If 0, the job is not enabled and
does not run according to its schedule; however, it can be run manually. If 1, the job will run according to its
schedule, and can also be run manually.
[ @schedule_interval_type =] schedule_interval_type
Value indicates when the job is to be executed. schedule_interval_type is nvarchar(50), with a default of Once, and
can be one of the following values:
'Once',
'Minutes',
'Hours',
'Days',
'Weeks',
'Months'
[ @schedule_interval_count = ] schedule_interval_count
Number of schedule_interval_count periods to occur between each execution of the job. schedule_interval_count is
int, with a default of 1. The value must be greater than or equal to 1.
[ @schedule_start_time = ] schedule_start_time
Date on which job execution can begin. schedule_start_time is DATETIME2, with the default of 0001-01-01
00:00:00.0000000.
[ @schedule_end_time = ] schedule_end_time
Date on which job execution can stop. schedule_end_time is DATETIME2, with the default of 9999-12-31
11:59:59.0000000.
[ @job_id = ] job_id OUTPUT
The job identification number assigned to the job if created successfully. job_id is an output variable of type
uniqueidentifier.
Return Code Values
0 (success) or 1 (failure)
Remarks
sp_add_job must be run from the job agent database specified when creating the job agent. After sp_add_job has
been executed to add a job, sp_add_jobstep can be used to add steps that perform the activities for the job. The
job’s initial version number is 0, which will be incremented to 1 when the first step is added.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_update_job
Updates an existing job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job to be updated. job_name is nvarchar(128).
[ @new_name = ] 'new_name'
The new name of the job. new_name is nvarchar(128).
[ @description = ] 'description'
The description of the job. description is nvarchar(512).
[ @enabled = ] enabled
Specifies whether the job’s schedule is enabled (1) or not enabled (0). Enabled is bit.
[ @schedule_interval_type= ] schedule_interval_type
Value indicates when the job is to be executed. schedule_interval_type is nvarchar(50) and can be one of the
following values:
'Once',
'Minutes',
'Hours',
'Days',
'Weeks',
'Months'
[ @schedule_interval_count= ] schedule_interval_count
Number of schedule_interval_count periods to occur between each execution of the job. schedule_interval_count is
int, with a default of 1. The value must be greater than or equal to 1.
[ @schedule_start_time= ] schedule_start_time
Date on which job execution can begin. schedule_start_time is DATETIME2, with the default of 0001-01-01
00:00:00.0000000.
[ @schedule_end_time= ] schedule_end_time
Date on which job execution can stop. schedule_end_time is DATETIME2, with the default of 9999-12-31
11:59:59.0000000.
Return Code Values
0 (success) or 1 (failure)
Remarks
After sp_add_job has been executed to add a job, sp_add_jobstep can be used to add steps that perform the
activities for the job. The job’s initial version number is 0, which will be incremented to 1 when the first step is
added.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_delete_job
Deletes an existing job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job to be deleted. job_name is nvarchar(128).
[ @force = ] force
Specifies whether to delete if the job has any executions in progress and cancel all in-progress executions (1) or
fail if any job executions are in progress (0). force is bit.
Return Code Values
0 (success) or 1 (failure)
Remarks
Job history is automatically deleted when a job is deleted.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_add_jobstep
Adds a step to a job.
Syntax
[jobs].sp_add_jobstep [ @job_name = ] 'job_name'
[ , [ @step_id = ] step_id ]
[ , [ @step_name = ] step_name ]
[ , [ @command_type = ] 'command_type' ]
[ , [ @command_source = ] 'command_source' ]
, [ @command = ] 'command'
, [ @credential_name = ] 'credential_name'
, [ @target_group_name = ] 'target_group_name'
[ , [ @initial_retry_interval_seconds = ] initial_retry_interval_seconds ]
[ , [ @maximum_retry_interval_seconds = ] maximum_retry_interval_seconds ]
[ , [ @retry_interval_backoff_multiplier = ] retry_interval_backoff_multiplier ]
[ , [ @retry_attempts = ] retry_attempts ]
[ , [ @step_timeout_seconds = ] step_timeout_seconds ]
[ , [ @output_type = ] 'output_type' ]
[ , [ @output_credential_name = ] 'output_credential_name' ]
[ , [ @output_subscription_id = ] 'output_subscription_id' ]
[ , [ @output_resource_group_name = ] 'output_resource_group_name' ]
[ , [ @output_server_name = ] 'output_server_name' ]
[ , [ @output_database_name = ] 'output_database_name' ]
[ , [ @output_schema_name = ] 'output_schema_name' ]
[ , [ @output_table_name = ] 'output_table_name' ]
[ , [ @job_version = ] job_version OUTPUT ]
[ , [ @max_parallelism = ] max_parallelism ]
Arguments
[ @job_name = ] 'job_name'
The name of the job to which to add the step. job_name is nvarchar(128).
[ @step_id = ] step_id
The sequence identification number for the job step. Step identification numbers start at 1 and increment without
gaps. If an existing step already has this id, then that step and all following steps will have their id's incremented so
that this new step can be inserted into the sequence. If not specified, the step_id will be automatically assigned to
the last in the sequence of steps. step_id is an int.
[ @step_name = ] step_name
The name of the step. Must be specified, except for the first step of a job which (for convenience) has a default
name of 'JobStep'. step_name is nvarchar(128).
[ @command_type = ] 'command_type'
The type of command that is executed by this jobstep. command_type is nvarchar(50), with a default value of TSql,
meaning that the value of the @command_type parameter is a T-SQL script.
If specified, the value must be TSql.
[ @command_source = ] 'command_source'
The type of location where the command is stored. command_source is nvarchar(50), with a default value of Inline,
meaning that the value of the @command_source parameter is the literal text of the command.
If specified, the value must be Inline.
[ @command = ] 'command'
The command must be valid T-SQL script and is then executed by this job step. command is nvarchar(max), with a
default of NULL.
[ @credential_name = ] ‘credential_name’
The name of the database scoped credential stored in this job control database that is used to connect to each of
the target databases within the target group when this step is executed. credential_name is nvarchar(128).
[ @target_group_name = ] ‘target-group_name'
The name of the target group that contains the target databases that the job step will be executed on.
target_group_name is nvarchar(128).
[ @initial_retry_interval_seconds = ] initial_retry_interval_seconds
The delay before the first retry attempt, if the job step fails on the initial execution attempt.
initial_retry_interval_seconds is int, with default value of 1.
[ @maximum_retry_interval_seconds = ] maximum_retry_interval_seconds
The maximum delay between retry attempts. If the delay between retries would grow larger than this value, it is
capped to this value instead. maximum_retry_interval_seconds is int, with default value of 120.
[ @retry_interval_backoff_multiplier = ] retry_interval_backoff_multiplier
The multiplier to apply to the retry delay if multiple job step execution attempts fail. For example, if the first retry
had a delay of 5 second and the backoff multiplier is 2.0, then the second retry will have a delay of 10 seconds and
the third retry will have a delay of 20 seconds. retry_interval_backoff_multiplier is real, with default value of 2.0.
[ @retry_attempts = ] retry_attempts
The number of times to retry execution if the initial attempt fails. For example, if the retry_attempts value is 10,
then there will be 1 initial attempt and 10 retry attempts, giving a total of 11 attempts. If the final retry attempt
fails, then the job execution will terminate with a lifecycle of Failed. retry_attempts is int, with default value of 10.
[ @step_timeout_seconds = ] step_timeout_seconds
The maximum amount of time allowed for the step to execute. If this time is exceeded, then the job execution will
terminate with a lifecycle of TimedOut. step_timeout_seconds is int, with default value of 43,200 seconds (12
hours).
[ @output_type = ] 'output_type'
If not null, the type of destination that the command’s first result set is written to. output_type is nvarchar(50), with
a default of NULL.
If specified, the value must be SqlDatabase.
[ @output_credential_name = ] 'output_credential_name'
If not null, the name of the database scoped credential that is used to connect to the output destination database.
Must be specified if output_type equals SqlDatabase. output_credential_name is nvarchar(128), with a default
value of NULL.
[ @output_subscription_id = ] 'output_subscription_id'
Needs description.
[ @output_resource_group_name = ] 'output_resource_group_name'
Needs description.
[ @output_server_name = ] 'output_server_name'
If not null, the fully qualified DNS name of the server that contains the output destination database. Must be
specified if output_type equals SqlDatabase. output_server_name is nvarchar(256), with a default of NULL.
[ @output_database_name = ] 'output_database_name'
If not null, the name of the database that contains the output destination table. Must be specified if output_type
equals SqlDatabase. output_database_name is nvarchar(128), with a default of NULL.
[ @output_schema_name = ] 'output_schema_name'
If not null, the name of the SQL schema that contains the output destination table. If output_type equals
SqlDatabase, the default value is dbo. output_schema_name is nvarchar(128).
[ @output_table_name = ] 'output_table_name'
If not null, the name of the table that the command’s first result set will be written to. If the table doesn't already
exist, it will be created based on the schema of the returning result-set. Must be specified if output_type equals
SqlDatabase. output_table_name is nvarchar(128), with a default value of NULL.
[ @job_version = ] job_version OUTPUT
Output parameter that will be assigned the new job version number. job_version is int.
[ @max_parallelism = ] max_parallelism OUTPUT
The maximum level of parallelism per elastic pool. If set, then the job step will be restricted to only run on a
maximum of that many databases per elastic pool. This applies to each elastic pool that is either directly included
in the target group or is inside a server that is included in the target group. max_parallelism is int.
Return Code Values
0 (success) or 1 (failure)
Remarks
When sp_add_jobstep succeeds, the job’s current version number is incremented. The next time the job is
executed, the new version will be used. If the job is currently executing, that execution will not contain the new
step.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_update_jobstep
Updates a job step.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job to which the step belongs. job_name is nvarchar(128).
[ @step_id = ] step_id
The identification number for the job step to be modified. Either step_id or step_name must be specified. step_id is
an int.
[ @step_name = ] 'step_name'
The name of the step to be modified. Either step_id or step_name must be specified. step_name is nvarchar(128).
[ @new_id = ] new_id
The new sequence identification number for the job step. Step identification numbers start at 1 and increment
without gaps. If a step is reordered, then other steps will be automatically renumbered.
[ @new_name = ] 'new_name'
The new name of the step. new_name is nvarchar(128).
[ @command_type = ] 'command_type'
The type of command that is executed by this jobstep. command_type is nvarchar(50), with a default value of TSql,
meaning that the value of the @command_type parameter is a T-SQL script.
If specified, the value must be TSql.
[ @command_source = ] 'command_source'
The type of location where the command is stored. command_source is nvarchar(50), with a default value of Inline,
meaning that the value of the @command_source parameter is the literal text of the command.
If specified, the value must be Inline.
[ @command = ] 'command'
The command(s) must be valid T-SQL script and is then executed by this job step. command is nvarchar(max),
with a default of NULL.
[ @credential_name = ] ‘credential_name’
The name of the database scoped credential stored in this job control database that is used to connect to each of
the target databases within the target group when this step is executed. credential_name is nvarchar(128).
[ @target_group_name = ] ‘target-group_name'
The name of the target group that contains the target databases that the job step will be executed on.
target_group_name is nvarchar(128).
[ @initial_retry_interval_seconds = ] initial_retry_interval_seconds
The delay before the first retry attempt, if the job step fails on the initial execution attempt.
initial_retry_interval_seconds is int, with default value of 1.
[ @maximum_retry_interval_seconds = ] maximum_retry_interval_seconds
The maximum delay between retry attempts. If the delay between retries would grow larger than this value, it is
capped to this value instead. maximum_retry_interval_seconds is int, with default value of 120.
[ @retry_interval_backoff_multiplier = ] retry_interval_backoff_multiplier
The multiplier to apply to the retry delay if multiple job step execution attempts fail. For example, if the first retry
had a delay of 5 second and the backoff multiplier is 2.0, then the second retry will have a delay of 10 seconds and
the third retry will have a delay of 20 seconds. retry_interval_backoff_multiplier is real, with default value of 2.0.
[ @retry_attempts = ] retry_attempts
The number of times to retry execution if the initial attempt fails. For example, if the retry_attempts value is 10,
then there will be 1 initial attempt and 10 retry attempts, giving a total of 11 attempts. If the final retry attempt
fails, then the job execution will terminate with a lifecycle of Failed. retry_attempts is int, with default value of 10.
[ @step_timeout_seconds = ] step_timeout_seconds
The maximum amount of time allowed for the step to execute. If this time is exceeded, then the job execution will
terminate with a lifecycle of TimedOut. step_timeout_seconds is int, with default value of 43,200 seconds (12
hours).
[ @output_type = ] 'output_type'
If not null, the type of destination that the command’s first result set is written to. To reset the value of output_type
back to NULL, set this parameter's value to '' (empty string). output_type is nvarchar(50), with a default of NULL.
If specified, the value must be SqlDatabase.
[ @output_credential_name = ] 'output_credential_name'
If not null, the name of the database scoped credential that is used to connect to the output destination database.
Must be specified if output_type equals SqlDatabase. To reset the value of output_credential_name back to NULL,
set this parameter's value to '' (empty string). output_credential_name is nvarchar(128), with a default value of
NULL.
[ @output_server_name = ] 'output_server_name'
If not null, the fully qualified DNS name of the server that contains the output destination database. Must be
specified if output_type equals SqlDatabase. To reset the value of output_server_name back to NULL, set this
parameter's value to '' (empty string). output_server_name is nvarchar(256), with a default of NULL.
[ @output_database_name = ] 'output_database_name'
If not null, the name of the database that contains the output destination table. Must be specified if output_type
equals SqlDatabase. To reset the value of output_database_name back to NULL, set this parameter's value to ''
(empty string). output_database_name is nvarchar(128), with a default of NULL.
[ @output_schema_name = ] 'output_schema_name'
If not null, the name of the SQL schema that contains the output destination table. If output_type equals
SqlDatabase, the default value is dbo. To reset the value of output_schema_name back to NULL, set this
parameter's value to '' (empty string). output_schema_name is nvarchar(128).
[ @output_table_name = ] 'output_table_name'
If not null, the name of the table that the command’s first result set will be written to. If the table doesn't already
exist, it will be created based on the schema of the returning result-set. Must be specified if output_type equals
SqlDatabase. To reset the value of output_server_name back to NULL, set this parameter's value to '' (empty
string). output_table_name is nvarchar(128), with a default value of NULL.
[ @job_version = ] job_version OUTPUT
Output parameter that will be assigned the new job version number. job_version is int.
[ @max_parallelism = ] max_parallelism OUTPUT
The maximum level of parallelism per elastic pool. If set, then the job step will be restricted to only run on a
maximum of that many databases per elastic pool. This applies to each elastic pool that is either directly included
in the target group or is inside a server that is included in the target group. To reset the value of max_parallelism
back to null, set this parameter's value to -1. max_parallelism is int.
Return Code Values
0 (success) or 1 (failure)
Remarks
Any in-progress executions of the job will not be affected. When sp_update_jobstep succeeds, the job’s version
number is incremented. The next time the job is executed, the new version will be used.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users
sp_delete_jobstep
Removes a job step from a job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job from which the step will be removed. job_name is nvarchar(128), with no default.
[ @step_id = ] step_id
The identification number for the job step to be deleted. Either step_id or step_name must be specified. step_id is
an int.
[ @step_name = ] 'step_name'
The name of the step to be deleted. Either step_id or step_name must be specified. step_name is nvarchar(128).
[ @job_version = ] job_version OUTPUT
Output parameter that will be assigned the new job version number. job_version is int.
Return Code Values
0 (success) or 1 (failure)
Remarks
Any in-progress executions of the job will not be affected. When sp_update_jobstep succeeds, the job’s version
number is incremented. The next time the job is executed, the new version will be used.
The other job steps will be automatically renumbered to fill the gap left by the deleted job step.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_start_job
Starts executing a job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job from which the step will be removed. job_name is nvarchar(128), with no default.
[ @job_execution_id = ] job_execution_id OUTPUT
Output parameter that will be assigned the job execution's id. job_version is uniqueidentifier.
Return Code Values
0 (success) or 1 (failure)
Remarks
None.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_stop_job
Stops a job execution.
Syntax
Arguments
[ @job_execution_id = ] job_execution_id
The identification number of the job execution to stop. job_execution_id is uniqueidentifier, with default of NULL.
Return Code Values
0 (success) or 1 (failure)
Remarks
None.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_add_target_group
Adds a target group.
Syntax
Arguments
[ @target_group_name = ] 'target_group_name'
The name of the target group to create. target_group_name is nvarchar(128), with no default.
[ @target_group_id = ] target_group_id OUTPUT The target group identification number assigned to the job if
created successfully. target_group_id is an output variable of type uniqueidentifier, with a default of NULL.
Return Code Values
0 (success) or 1 (failure)
Remarks
Target groups provide an easy way to target a job at a collection of databases.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_delete_target_group
Deletes a target group.
Syntax
Arguments
[ @target_group_name = ] 'target_group_name'
The name of the target group to delete. target_group_name is nvarchar(128), with no default.
Return Code Values
0 (success) or 1 (failure)
Remarks
None.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
sp_add_target_group_member
Adds a database or group of databases to a target group.
Syntax
Arguments
[ @target_group_name = ] 'target_group_name'
The name of the target group to which the member will be added. target_group_name is nvarchar(128), with no
default.
[ @membership_type = ] 'membership_type'
Specifies if the target group member will be included or excluded. target_group_name is nvarchar(128), with
default of ‘Include’. Valid values for target_group_name are ‘Include’ or ‘Exclude’.
[ @target_type = ] 'target_type'
The type of target database or collection of databases including all databases in a server, all databases in an Elastic
pool, all databases in a shard map, or an individual database. target_type is nvarchar(128), with no default. Valid
values for target_type are ‘SqlServer’, ‘SqlElasticPool’, ‘SqlDatabase’, or ‘SqlShardMap’.
[ @refresh_credential_name = ] 'refresh_credential_name'
The name of the logical server. refresh_credential_name is nvarchar(128), with no default.
[ @server_name = ] 'server_name'
The name of the logical server that should be added to the specified target group. server_name should be
specified when target_type is ‘SqlServer’. server_name is nvarchar(128), with no default.
[ @database_name = ] 'database_name'
The name of the database that should be added to the specified target group. database_name should be specified
when target_type is ‘SqlDatabase’. database_name is nvarchar(128), with no default.
[ @elastic_pool_name = ] ‘elastic_pool_name'
The name of the Elastic pool that should be added to the specified target group. elastic_pool_name should be
specified when target_type is ‘SqlElasticPool’. elastic_pool_name is nvarchar(128), with no default.
[ @shard_map_name = ] ‘shard_map_name'
The name of the shard map pool that should be added to the specified target group. elastic_pool_name should be
specified when target_type is ‘SqlSqlShardMap’. shard_map_name is nvarchar(128), with no default.
[ @target_id = ] target_group_id OUTPUT
The target identification number assigned to the target group member if created added to the target group.
target_id is an output variable of type uniqueidentifier, with a default of NULL. Return Code Values 0 (success) or 1
(failure)
Remarks
A job executes on all databases within a server or Elastic pool at time of execution, when a logical server or Elastic
pool is included in the target group.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
Examples
The following example adds all the databases in the London and NewYork servers to the group Servers
Maintaining Customer Information. You must connect to the jobs database specified when creating the job agent,
in this case ElasticJobs.
--Connect to the jobs database specified when creating the job agent
USE ElasticJobs ;
GO
sp_delete_target_group_member
Removes a target group member from a target group.
Syntax
--Connect to the jobs database specified when creating the job agent
USE ElasticJobs ;
GO
sp_purge_jobhistory
Removes the history records for a job.
Syntax
Arguments
[ @job_name = ] 'job_name'
The name of the job for which to delete the history records. job_name is nvarchar(128), with a default of NULL.
Either job_id or job_name must be specified, but both cannot be specified.
[ @job_id = ] job_id
The job identification number of the job for the records to be deleted. job_id is uniqueidentifier, with a default of
NULL. Either job_id or job_name must be specified, but both cannot be specified.
[ @oldest_date = ] oldest_date
The oldest record to retain in the history. oldest_date is DATETIME2, with a default of NULL. When oldest_date is
specified, sp_purge_jobhistory only removes records that are older than the value specified.
Return Code Values
0 (success) or 1 (failure) Remarks Target groups provide an easy way to target a job at a collection of databases.
Permissions
By default, members of the sysadmin fixed server role can execute this stored procedure. They restrict a user to
just be able to monitor jobs, you can grant the user to be part of the following database role in the job agent
database specified when creating the job agent:
jobs_reader
For details about the permissions of these roles, see the Permission section in this document. Only members of
sysadmin can use this stored procedure to edit the attributes of jobs that are owned by other users.
Examples
The following example adds all the databases in the London and NewYork servers to the group Servers
Maintaining Customer Information. You must connect to the jobs database specified when creating the job agent,
in this case ElasticJobs.
--Connect to the jobs database specified when creating the job agent
EXEC sp_delete_target_group_member
@target_group_name = N'Servers Maintaining Customer Information',
@server_name = N'London.database.windows.net';
GO
Job views
The following views are available in the jobs database.
VIEW DESCRIPTION
jobs_executions view
[jobs].[jobs_executions]
Shows job execution history.
jobs view
[jobs].[jobs]
Shows all jobs.
schedule_start_time datetime2(7) Date and time the job was last started
execution.
job_versions view
[jobs].[job_verions]
Shows all job versions.
jobsteps view
[jobs].[jobsteps]
Shows all steps in the current version of each job.
step_name nvarchar(128) Unique (for this job) name for the step.
jobstep_versions view
[jobs].[jobstep_versions]
Shows all steps in all versions of each job. The schema is identical to jobsteps.
target_groups view
[jobs].[target_groups]
Lists all target groups.
target_groups_members view
[jobs].[target_groups_members]
Shows all members of all target groups.
COLUMN NAME DATA TYPE DESCRIPTION
Resources
Transact-SQL Syntax Conventions
Next steps
Create and manage Elastic Jobs using PowerShell
Authorization and Permissions SQL Server
Migrate to the new Elastic Database jobs
7/13/2018 • 12 minutes to read • Edit Online
Prerequisites
The upgraded version of Elastic Database jobs has a new set of PowerShell cmdlets for use during migration.
These new cmdlets transfer all of your existing job credentials, targets (including databases, servers, custom
collections), job triggers, job schedules, job contents, and jobs over to a new Elastic Job agent.
Install the latest Elastic Jobs cmdlets
If you don't have already have an Azure subscription, create a free account before you begin.
Install the AzureRM.Sql 4.8.1-preview module to get the latest Elastic Job cmdlets. Run the following commands
in PowerShell with administrative access.
# Installs the latest PackageManagement powershell package which PowershellGet v1.6.5 is dependent on
Find-Package PackageManagement -RequiredVersion 1.1.7.2 | Install-Package -Force
# Installs the latest PowershellGet module which adds the -AllowPrerelease flag to Install-Module
Find-Package PowerShellGet -RequiredVersion 1.6.5 | Install-Package -Force
# Places AzureRM.Sql preview cmdlets side by side with existing AzureRM.Sql version
Install-Module -Name AzureRM.Sql -AllowPrerelease -RequiredVersion 4.8.1-preview -Force
# Confirm if module successfully imported - if the imported version is 4.8.1, then continue
Get-Module AzureRM.Sql
# Register your subscription for the for the Elastic Jobs public preview feature
Register-AzureRmProviderFeature -FeatureName sqldb-JobAccounts -ProviderNamespace Microsoft.Sql
# Get an existing database to use as the job database - or create a new one if necessary
$db = Get-AzureRmSqlDatabase -ResourceGroupName <resourceGroupName> -ServerName <serverName> -DatabaseName
<databaseName>
# Create a new elastic job agent
$agent = $db | New-AzureRmSqlElasticJobAgent -Name <agentName>
Migration
Now that both the old and new Elastic Jobs cmdlets are initialized, migrate your job credentials, targets, and jobs
to the new job database.
Setup
$ErrorActionPreference = "Stop";
Migrate credentials
$oldCreds = Get-AzureSqlJobCredential
$oldCreds | % {
$oldCredName = $_.CredentialName
$oldUserName = $_.UserName
Write-Output ("Credential " + $oldCredName)
$oldCredential = Get-Credential -UserName $oldUserName `
-Message ("Please enter in the password that was used for your credential " +
$oldCredName)
try
{
$cred = New-AzureRmSqlElasticJobCredential -ParentObject $agent -Name $oldCredName -Credential
$oldCredential
}
catch [System.Management.Automation.PSArgumentException]
{
$cred = Get-AzureRmSqlElasticJobCredential -ParentObject $agent -Name $oldCredName
$cred = Set-AzureRmSqlElasticJobCredential -InputObject $cred -Credential $oldCredential
}
To migrate your credentials, execute the following command by passing in the $agent PowerShell object from
earlier.
Migrate-Credentials $agent
Sample output
Migrate targets
# Flatten list
for ($i=$targetGroups.Count - 1; $i -ge 0; $i--)
{
# Fetch target group's initial list of targets unexpanded
$targets = $targetGroups[$i]
$expandedTargets = $targetGroups[$target.TargetDescription.CustomCollectionName]
# Migrate server target from old jobs to new job's target group
function Add-ServerTarget ($target, $tg) {
$jobTarget = Get-AzureSqlJobTarget -TargetId $target.TargetId
$serverName = $jobTarget.ServerName
$credName = $jobTarget.MasterDatabaseCredentialName
$t = Add-AzureRmSqlElasticJobTarget -ParentObject $tg -ServerName $serverName -RefreshCredentialName
$credName
}
# Migrate database target from old jobs to new job's target group
function Add-DatabaseTarget ($target, $tg) {
$jobTarget = Get-AzureSqlJobTarget -TargetId $target.TargetId
$serverName = $jobTarget.ServerName
$databaseName = $jobTarget.DatabaseName
$exclude = $target.Membership
# Migrate shard map target from old jobs to new job's target group
function Add-ShardMapTarget ($target, $tg) {
$jobTarget = Get-AzureSqlJobTarget -TargetId $target.TargetId
$smName = $jobTarget.ShardMapName
$serverName = $jobTarget.ShardMapManagerServerName
$databaseName = $jobTarget.ShardMapManagerDatabaseName
$credName = $jobTarget.ShardMapManagerCredentialName
$exclude = $target.Membership
return $tgName
}
return $tgName
}
To migrate your targets (servers, databases, and custom collections) to your new job database, execute the
Migrate-TargetGroups cmdlet to perform the following:
Root level targets that are servers and databases will be migrated to a new target group named "
(<serverName>, <databaseName>)" containing only the root level target.
A custom collection will migrate to a new target group containing all child targets.
Migrate-TargetGroups $agent
Sample output:
Migrate jobs
$oldJobs = Get-AzureSqlJob
$newJobs = [System.Collections.ArrayList] @()
# Schedule
$oldJobTriggers = Get-AzureSqlJobTrigger -JobName $oldJob.JobName
if ($oldJobTriggers.Count -ge 1)
{
foreach ($trigger in $oldJobTriggers)
{
# Migrates jobs
function Setup-Job ($job, $agent) {
$jobName = $newJob.JobName
$jobDescription = $newJob.Description
try {
$job = New-AzureRmSqlElasticJob -ParentObject $agent -Name $jobName `
-Description $jobDescription -IntervalType $intervalType -IntervalCount $intervalCount `
-StartTime $startTime -EndTime $endTime
return $job
}
catch [System.Management.Automation.PSArgumentException] {
$job = Get-AzureRmSqlElasticJob -ParentObject $agent -Name $jobName
$job = $job | Set-AzureRmSqlElasticJob -Description $jobDescription -IntervalType $intervalType -
IntervalCount $intervalCount `
-StartTime $startTime -EndTime $endTime
return $job
}
}
# Create or update a job that runs once
else {
try {
$job = New-AzureRmSqlElasticJob -ParentObject $agent -Name $jobName `
-Description $jobDescription -RunOnce
return $job
}
catch [System.Management.Automation.PSArgumentException] {
$job = Get-AzureRmSqlElasticJob -ParentObject $agent -Name $jobName
$job = $job | Set-AzureRmSqlElasticJob -Description $jobDescription -RunOnce
return $job
}
}
}
# Migrates job steps
function Setup-JobStep ($newJob, $job) {
$defaultJobStepName = 'JobStep'
$contentName = $newJob.Description
$commandText = (Get-AzureSqlJobContentDefinition -ContentName $contentName).CommandText
$targetGroupName = $newJob.TargetGroupName
$credentialName = $newJob.CredentialName
$output = $newJob.Output
try {
$jobStep = $job | Add-AzureRmSqlElasticJobStep -Name $defaultJobStepName `
-TargetGroupName $targetGroupName -CredentialName $credentialName -CommandText $commandText `
-OutputDatabaseObject $outputDatabase `
-OutputSchemaName $outputSchemaName -OutputTableName $outputTableName `
-OutputCredentialName $outputCredentialName
}
catch [System.Management.Automation.PSArgumentException] {
$jobStep = $job | Get-AzureRmSqlElasticJobStep -Name $defaultJobStepName
$jobStep = $jobStep | Set-AzureRmSqlElasticJobStep -TargetGroupName $targetGroupName `
-CredentialName $credentialName -CommandText $commandText `
-OutputDatabaseObject $outputDatabase `
-OutputSchemaName $outputSchemaName -OutputTableName $outputTableName `
-OutputCredentialName $outputCredentialName
}
}
else {
try {
$jobStep = $job | Add-AzureRmSqlElasticJobStep -Name $defaultJobStepName -TargetGroupName
$targetGroupName -CredentialName $credentialName -CommandText $commandText
}
catch [System.Management.Automation.PSArgumentException] {
$jobStep = $job | Get-AzureRmSqlElasticJobStep -Name $defaultJobStepName
$jobStep = $jobStep | Set-AzureRmSqlElasticJobStep -TargetGroupName $targetGroupName -CredentialName
$credentialName -CommandText $commandText
}
}
Log-ChildOutput ("Added step " + $jobStep.StepName + " using target group " + $jobStep.TargetGroupName + "
using credential " + $jobStep.CredentialName)
Log-ChildOutput("Command text script taken from content name " + $contentName)
To migrate your jobs, job content, job triggers, and job schedules over to your new Elastic Job agent's database,
execute the Migrate-Jobs cmdlet passing in your agent.
Jobs with multiple triggers with different schedules are separated into multiple jobs with naming scheme: "
<jobName> (<scheduleName>)".
Job contents are migrated to a job by adding a default job step named JobStep with associated command text.
Jobs are disabled by default so that you can validate them before enabling them.
Migrate-Jobs $agent
Sample output:
Migration Complete
The job database should now have all of the job credentials, targets, job triggers, job schedules, job contents, and
jobs migrated over.
To confirm that everything migrated correctly, use the following scripts:
$jobs | Start-AzureRmSqlElasticJob
For any jobs that were running on a schedule, remember to enable them so that they can run in the background:
Next steps
Create and manage Elastic Jobs using PowerShell
Create and manage Elastic Jobs using Transact-SQL (T-SQL )
Managing scaled-out cloud databases
6/14/2018 • 7 minutes to read • Edit Online
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you
currently use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration
scripts to quickly upgrade to the latest version.
Elastic Database jobs is a customer-hosted Azure Cloud Service that enables the execution of ad-hoc and
scheduled administrative tasks, which are called jobs. With jobs, you can easily and reliably manage large
groups of Azure SQL Databases by running Transact-SQL scripts to perform administrative operations.
To manage scaled-out sharded databases, the Elastic Database jobs feature (preview ) enables you to reliably
execute a Transact-SQL (T-SQL ) script across a group of databases, including:
a custom-defined collection of databases (explained below )
all databases in an elastic pool
a shard set (created using Elastic Database client library ).
Documentation
Install the Elastic Database job components.
Get started with Elastic Database jobs.
Create and manage jobs using PowerShell.
Create and manage scaled out Azure SQL Databases
NOTE
In the Azure portal, only a reduced set of functions limited to SQL Azure elastic pools is available. Use the PowerShell APIs
to access the full set of current functionality.
Applications
Perform administrative tasks, such as deploying a new schema.
Update reference data-product information common across all databases. Or schedules automatic updates
every weekday, after hours.
Rebuild indexes to improve query performance. The rebuilding can be configured to execute across a
collection of databases on a recurring basis, such as during off-peak hours.
Collect query results from a set of databases into a central table on an on-going basis. Performance queries
can be continually executed and configured to trigger additional tasks to be executed.
Execute longer running data processing queries across a large set of databases, for example the collection of
customer telemetry. Results are collected into a single destination table for further analysis.
Idempotent scripts
The scripts must be idempotent. In simple terms, "idempotent" means that if the script succeeds, and it is run
again, the same result occurs. A script may fail due to transient network issues. In that case, the job will
automatically retry running the script a preset number of times before desisting. An idempotent script has the
same result even if has been successfully run twice.
A simple tactic is to test for the existence of an object before creating it.
Similarly, a script must be able to execute successfully by logically testing for and countering any conditions it
finds.
Next steps
Install the components, then create and add a log in to each database in the group of databases. To further
understand job creation and management, see creating and managing elastic database jobs. See also Getting
started with Elastic Database jobs.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Installing Elastic Database jobs overview
6/14/2018 • 6 minutes to read • Edit Online
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you currently
use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration scripts to
quickly upgrade to the latest version.
Elastic Database jobs can be installed via PowerShell or through the Azure portal.You can gain access to create
and manage jobs using the PowerShell API only if you install the PowerShell package. Additionally, the
PowerShell APIs provide significantly more functionality than the portal at this point in time.
If you have already installed Elastic Database jobs through the Portal from an existing elastic pool, the latest
Powershell preview includes scripts to upgrade your existing installation. It is highly recommended to upgrade
your installation to the latest Elastic Database jobs components in order to take advantage of new functionality
exposed via the PowerShell APIs.
Prerequisites
An Azure subscription. For a free trial, see Free trial.
Azure PowerShell. Install the latest version using the Web Platform Installer. For detailed information, see How
to install and configure Azure PowerShell.
NuGet Command-line Utility is used to install the Elastic Database jobs package. For more information, see
http://docs.nuget.org/docs/start-here/installing-nuget.
The Elastic Database jobs files are placed in the local directory in a folder named
Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x where x.x.xxxx.x reflects the version number. The
PowerShell cmdlets (including required client .dlls) are located in the tools\ElasticDatabaseJobs sub-
directory and the PowerShell scripts to install, upgrade and uninstall also reside in the tools sub-directory.
3. Navigate to the tools sub-directory under the Microsoft.Azure.SqlDatabase.Jobs.x.x.xxx.x folder by typing
cd tools, for example:
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*>cd tools
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*\tools>Unblock-File
.\InstallElasticDatabaseJobsCmdlets.ps1
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*\tools>.\InstallElasticDatabaseJobsCmdlets.ps1
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*>cd tools
2. Execute the .\InstallElasticDatabaseJobs.ps1 PowerShell script and supply values for its requested variables.
This script will create the components described in Elastic Database jobs components and pricing along
with configuring the Azure Cloud Service to appropriately use the dependent components.
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*\tools>Unblock-File
.\InstallElasticDatabaseJobs.ps1
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.x.x.xxxx.x*\tools>.\InstallElasticDatabaseJobs.ps1
When you run this command a window opens asking for a user name and password. This is not your Azure
credentials, enter the user name and password that will be the administrator credentials you want to create for the
new server.
The parameters provided on this sample invocation can be modified for your desired settings. The following
provides more information on the behavior of each parameter:
PARAMETER DESCRIPTION
ResourceGroupLocation Provides the Azure location to be used for the newly created
Azure components. This parameter defaults to the Central US
location.
ServiceVmSize Provides the VM size for usage within the Cloud Service. This
parameter defaults to A0. Parameters values of A0/A1/A2/A3
are accepted which cause the worker role to use an
ExtraSmall/Small/Medium/Large size, respectively. Fo more
information on worker role sizes, see Elastic Database jobs
components and pricing.
SqlServerDatabaseSlo Provides the service level objective for a Standard edition. This
parameter defaults to S0. Parameter values of
S0/S1/S2/S3/S4/S6/S9/S12 are accepted which cause the
Azure SQL Database to use the respective SLO. For more
information on SQL Database SLOs, see Elastic Database jobs
components and pricing.
SqlServerAdministratorUserName Provides the admin user name for the newly created Azure
SQL Database server. When not specified, a PowerShell
credentials window will open to prompt for the credentials.
SqlServerAdministratorPassword Provides the admin password for the newly created Azure
SQL Database server. When not provided, a PowerShell
credentials window will open to prompt for the credentials.
For systems that target having large numbers of jobs running in parallel against a large number of databases, it is
recommended to specify parameters such as: -ServiceWorkerCount 2 -ServiceVmSize A2 -SqlServerDatabaseSlo
S2.
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.dll.x.x.xxx.x*\tools>Unblock-File .\InstallElasticDatabaseJobs.ps1
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.dll.x.x.xxx.x*\tools>.\InstallElasticDatabaseJobs.ps1 -
ServiceWorkerCount 2 -ServiceVmSize A2 -SqlServerDatabaseSlo S2
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.dll.x.x.xxx.x*\tools>Unblock-File .\UpdateElasticDatabaseJobs.ps1
PS C:\*Microsoft.Azure.SqlDatabase.Jobs.dll.x.x.xxx.x*\tools>.\UpdateElasticDatabaseJobs.ps1 -ServiceVmSize A1
-ServiceWorkerCount 2
PARAMETER DESCRIPTION
ResourceGroupName Identifies the Azure resource group name used when the
Elastic Database job components were initially installed. This
parameter defaults to “__ElasticDatabaseJob”. Since it is not
recommended to change this value, you shouldn't have to
specify this parameter.
5. Type a user name and password for a database admin. As part of the installation, a new Azure SQL
Database server is created. Within this new server, a new database, known as the control database, is
created and used to contain the meta data for Elastic Database jobs. The user name and password created
here are used for the purpose of logging in to the control database. A separate credential is used for script
execution against the databases within the pool.
6. Click the OK button. The components are created for you in a few minutes in a new Resource group. The
new resource group is pinned to the start board, as shown below. Once created, elastic database jobs
(Cloud Service, SQL Database, Service Bus, and Storage) are all created in the group.
7. If you attempt to create or manage a job while elastic database jobs is installing, when providing
Credentials you will see the following message.
If uninstallation is required, delete the resource group. See How to uninstall the Elastic Database job components.
Next steps
Ensure a credential with the appropriate rights for script execution is created on each database in the group, for
more information see Securing your SQL Database. See Creating and managing an Elastic Database jobs to get
started.
Create and manage scaled out Azure SQL databases
using elastic jobs (preview)
9/12/2018 • 3 minutes to read • Edit Online
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you currently
use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration scripts to
quickly upgrade to the latest version.
Elastic Database jobs simplify management of groups of databases by executing administrative operations such
as schema changes, credentials management, reference data updates, performance data collection or tenant
(customer) telemetry collection. Elastic Database jobs is currently available through the Azure portal and
PowerShell cmdlets. However, the Azure portal surfaces reduced functionality limited to execution across all
databases in an elastic pool. To access additional features and execution of scripts across a group of databases
including a custom-defined collection or a shard set (created using Elastic Database client library ), see Creating
and managing jobs using PowerShell. For more information about jobs, see Elastic Database jobs overview.
Prerequisites
An Azure subscription. For a free trial, see Free trial.
An elastic pool. See About elastic pools.
Installation of elastic database job service components. See Installing the elastic database job service.
Creating jobs
1. Using the Azure portal, from an existing elastic database job pool, click Create job.
2. Type in the username and password of the database administrator (created at installation of Jobs) for the
jobs control database (metadata storage for jobs).
3. In the Create Job blade, type a name for the job.
4. Type the user name and password to connect to the target databases with sufficient permissions for script
execution to succeed.
5. Paste or type in the T-SQL script.
6. Click Save and then click Run.
Run idempotent jobs
When you run a script against a set of databases, you must be sure that the script is idempotent. That is, the script
must be able to run multiple times, even if it has failed before in an incomplete state. For example, when a script
fails, the job will be automatically retried until it succeeds (within limits, as the retry logic will eventually cease the
retrying). The way to do this is to use the an "IF EXISTS" clause and delete any found instance before creating a
new object. An example is shown here:
Alternatively, use an "IF NOT EXISTS" clause before creating a new instance:
IF NOT EXISTS (SELECT columns.name FROM sys.columns INNER JOIN sys.tables on columns.object_id =
tables.object_id WHERE tables.name = 'TestTable' AND columns.name = 'AdditionalInformation')
BEGIN
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you currently
use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration scripts to
quickly upgrade to the latest version.
The PowerShell APIs for Elastic Database jobs (in preview ), let you define a group of databases against which
scripts will execute. This article shows how to create and manage Elastic Database jobs using PowerShell
cmdlets. See Elastic jobs overview.
Prerequisites
An Azure subscription. For a free trial, see Free one-month trial.
A set of databases created with the Elastic Database tools. See Get started with Elastic Database tools.
Azure PowerShell. For detailed information, see How to install and configure Azure PowerShell.
Elastic Database jobs PowerShell package: See Installing Elastic Database jobs
Select your Azure subscription
To select the subscription you need your subscription Id (-SubscriptionId) or subscription name (-
SubscriptionName). If you have multiple subscriptions you can run the Get-AzureRmSubscription cmdlet and
copy the desired subscription information from the result set. Once you have your subscription information, run
the following commandlet to set this subscription as the default, namely the target for creating and managing
jobs:
The PowerShell ISE is recommended for usage to develop and execute PowerShell scripts against the Elastic
Database jobs.
Custom Collection Child Target Database target that is referenced from Add-AzureSqlJobChildTarget
a custom collection.
Remove-AzureSqlJobChildTarget
Use-AzureSqlJobConnection -CurrentAzureSubscription
To update credentials
When passwords change, use the Set-AzureSqlJobCredential cmdlet and set the CredentialName parameter.
$scriptCommandText = "
IF NOT EXISTS (SELECT name FROM sys.tables WHERE name = 'TestTable')
BEGIN
CREATE TABLE TestTable(
TestTableId INT PRIMARY KEY IDENTITY,
InsertionTime DATETIME2
);
END
GO
INSERT INTO TestTable(InsertionTime) VALUES (sysutcdatetime());
GO"
IF NOT EXISTS (SELECT columns.name FROM sys.columns INNER JOIN sys.tables on columns.object_id =
tables.object_id WHERE tables.name = 'TestTable' AND columns.name = 'AdditionalInformation')
BEGIN
ALTER TABLE TestTable
ADD AdditionalInformation NVARCHAR(400);
END
GO
To execute a job
This PowerShell script executes an existing job:
Update the following variable to reflect the desired job name to have executed:
Use the same Get-AzureSqlJobExecution cmdlet with the IncludeChildren parameter to view the state of
child job executions, namely the specific state for each job execution against each database targeted by the job.
Get-AzureSqlJobExecution
Retrieve all top level job executions, including inactive job executions:
Get-AzureSqlJobExecution -IncludeInactive
Retrieve all child job executions of a provided job execution ID, including inactive job executions:
Retrieve all job executions created using a schedule / job combination, including inactive jobs:
Retrieve all jobs targeting a specified shard map, including inactive jobs:
Retrieve all jobs targeting a specified custom collection, including inactive jobs:
$customCollectionName = "{Custom Collection Name}"
$target = Get-AzureSqlJobTarget -CustomCollectionName $customCollectionName
Get-AzureSqlJobExecution -TargetId $target.TargetId -IncludeInactive
Retrieve the list of job task executions within a specific job execution:
Cancel a job
Elastic Database Jobs supports cancellation requests of jobs. If Elastic Database Jobs detects a cancellation
request for a job currently being executed, it will attempt to stop the job.
There are two different ways that Elastic Database Jobs can perform a cancellation:
1. Cancel currently executing tasks: If a cancellation is detected while a task is currently running, a cancellation
will be attempted within the currently executing aspect of the task. For example: If there is a long running
query currently being performed when a cancellation is attempted, there will be an attempt to cancel the query.
2. Canceling task retries: If a cancellation is detected by the control thread before a task is launched for execution,
the control thread will avoid launching the task and declare the request as canceled.
If a job cancellation is requested for a parent job, the cancellation request will be honored for the parent job and
for all of its child jobs.
To submit a cancellation request, use the Stop-AzureSqlJobExecution cmdlet and set the JobExecutionId
parameter.
$dacpacUri = "{Uri}"
$dacpacName = "{Dacpac Name}"
$dacpac = New-AzureSqlJobContent -DacpacUri $dacpacUri -ContentName $dacpacName
Write-Output $dacpac
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you
currently use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration
scripts to quickly upgrade to the latest version.
Elastic Database jobs (preview ) for Azure SQL Database allows you to reliably execute T-SQL scripts that span
multiple databases while automatically retrying and providing eventual completion guarantees. For more
information about the Elastic Database job feature, see Elastic jobs.
This article extends the sample found in Getting started with Elastic Database tools. When completed, you learn
how to create and manage jobs that manage a group of related databases. It is not required to use the Elastic
Scale tools in order to take advantage of the benefits of Elastic jobs.
Prerequisites
Download and run the Getting started with Elastic Database tools sample.
2. In the command window, type "1" and press Enter. This creates the shard map manager, and adds two
shards to the server. Then type "3" and press Enter; repeat this action four times. This inserts sample data
rows in your shards.
3. The Azure portal should show three new databases:
At this point, we create a custom database collection that reflects all the databases in the shard map. This
allows us to create and execute a job that adds a new table across shards.
Here we would usually create a shard map target, using the New-AzureSqlJobTarget cmdlet. The shard map
manager database must be set as a database target and then the specific shard map is specified as a target.
Instead, we are going to enumerate all the databases in the server and add the databases to the new custom
collection with the exception of master database.
Creates a custom collection and add all databases in the server to the
custom collection target with the exception of master.
$customCollectionName = "dbs_in_server"
New-AzureSqlJobTarget -CustomCollectionName $customCollectionName
$ResourceGroupName = "ddove_samples"
$ServerName = "samples"
$dbsinserver = Get-AzureRMSqlDatabase -ResourceGroupName $ResourceGroupName -ServerName $ServerName
$dbsinserver | %{
$currentdb = $_.DatabaseName
$ErrorActionPreference = "Stop"
Write-Output ""
Try
{
New-AzureSqlJobTarget -ServerName $ServerName -DatabaseName $currentdb | Write-Output
}
Catch
{
$ErrorMessage = $_.Exception.Message
$ErrorCategory = $_.CategoryInfo.Reason
else
{
throw $_
}
Try
{
if ($currentdb -eq "master")
{
Write-Host $currentdb "will not be added custom collection target" $CustomCollectionName "."
}
else
{
Add-AzureSqlJobChildTarget -CustomCollectionName $CustomCollectionName -ServerName $ServerName -
DatabaseName $currentdb
Write-Host $currentdb "was added to" $CustomCollectionName "."
}
}
Catch
{
$ErrorMessage = $_.Exception.Message
$ErrorCategory = $_.CategoryInfo.Reason
else
{
throw $_
}
}
$ErrorActionPreference = "Continue"
}
Get-AzureSqlJobExecution -IncludeInactive
Retrieve all child job executions of a provided job execution ID, including inactive job executions:
Retrieve all job executions created using a schedule / job combination, including inactive jobs:
Retrieve all jobs targeting a specified shard map, including inactive jobs:
Retrieve all jobs targeting a specified custom collection, including inactive jobs:
Retrieve the list of job task executions within a specific job execution:
Cancel a job
Elastic Database Jobs supports jobs cancellation requests. If Elastic Database Jobs detects a cancellation request
for a job currently being executed, it attempts to stop the job.
There are two different ways that Elastic Database Jobs can perform a cancellation:
1. Canceling Currently Executing Tasks: If a cancellation is detected while a task is currently running, a
cancellation is attempted within the currently executing aspect of the task. For example: If there is a long
running query currently being performed when a cancellation is attempted, there is an attempt to cancel the
query.
2. Canceling Task Retries: If a cancellation is detected by the control thread before a task is launched for
execution, the control thread avoids launching the task and declare the request as canceled.
If a job cancellation is requested for a parent job, the cancellation request is honored for the parent job and for all
of its child jobs.
To submit a cancellation request, use the Stop-AzureSqlJobExecution cmdlet and set the JobExecutionId
parameter.
4. In the Data Connection Wizard type the server name and login credentials. Then click Next.
5. In the dialog box Select the database that contains the data you want, select the ElasticDBQuery
database.
6. Select the Customers table in the list view and click Next. Then click Finish.
7. In the Import Data form, under Select how you want to view this data in your workbook, select Table
and click OK.
All the rows from Customers table, stored in different shards populate the Excel sheet.
Next steps
You can now use Excel’s data functions. Use the connection string with your server name, database name and
credentials to connect your BI and data integration tools to the elastic query database. Make sure that SQL
Server is supported as a data source for your tool. Refer to the elastic query database and external tables just like
any other SQL Server database and SQL Server tables that you would connect to with your tool.
Cost
There is no additional charge for using the Elastic Database query feature. However, at this time this feature is
available only on Premium and Business Critical databases and elastic pools as an end point, but the shards can
be of any service tier.
For pricing information see SQL Database Pricing Details.
Additional resources
Not using elastic database tools yet? Check out our Getting Started Guide. For questions, please reach out to us
on the SQL Database forum and for feature requests, please add them to the SQL Database feedback forum.
Uninstall Elastic Database jobs components
6/14/2018 • 2 minutes to read • Edit Online
IMPORTANT
This article is for the customer-hosted version of Elastic Database jobs. Elastic Database jobs are being deprecated and
replaced with new Azure-hosted Elastic Database jobs. For new jobs, use the latest Elastic Database jobs. If you currently
use the older customer-hosted jobs, see Migrate to the new Elastic Database jobs for directions and migration scripts to
quickly upgrade to the latest version.
Elastic Database jobs components can be uninstalled using either the Azure portal or PowerShell.
$ResourceGroupName = "__ElasticDatabaseJob"
Switch-AzureMode AzureResourceManager
Write-Host "Removing the Azure Resource Group: $ResourceGroupName. This may take a few minutes.”
Remove-AzureResourceGroup -Name $ResourceGroupName -Force
Write-Host "Completed removing the Azure Resource Group: $ResourceGroupName. Elastic database job
compoennts are now uninstalled."
Next steps
To re-install Elastic Database jobs, see Installing the Elastic Database job service
For an overview of Elastic Database jobs, see Elastic Database jobs overview.
Azure SQL Database elastic query overview
(preview)
7/3/2018 • 9 minutes to read • Edit Online
The elastic query feature (in preview ) enables you to run a Transact-SQL query that spans multiple databases in
Azure SQL Database. It allows you to perform cross-database queries to access remote tables, and to connect
Microsoft and third-party tools (Excel, Power BI, Tableau, etc.) to query across data tiers with multiple
databases. Using this feature, you can scale out queries to large data tiers in SQL Database and visualize the
results in business intelligence (BI) reports.
NOTE
Elastic query works best for reporting scenarios where most of the processing (filtering, aggregation) can be performed
on the external source side. It is not suitable for ETL operations where large amount of data is being transferred from
remote database(s). For heavy reporting workloads or data warehousing scenarios with more complex queries, also
consider using Azure SQL Data Warehouse.
IMPORTANT
You must possess ALTER ANY EXTERNAL DATA SOURCE permission. This permission is included with the ALTER
DATABASE permission. ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the underlying data
source.
Reference data: The topology is used for reference data management. In the figure below, two tables (T1 and
T2) with reference data are kept on a dedicated database. Using an elastic query, you can now access tables T1
and T2 remotely from other databases, as shown in the figure. Use topology 1 if reference tables are small or
remote queries into reference table have selective predicates.
Figure 2 Vertical partitioning - Using elastic query to query reference data
Cross-database querying: Elastic queries enable use cases that require querying across several SQL
databases. Figure 3 shows four different databases: CRM, Inventory, HR, and Products. Queries performed in
one of the databases also need access to one or all the other databases. Using an elastic query, you can
configure your database for this case by running a few simple DDL statements on each of the four databases.
After this one-time configuration, access to a remote table is as simple as referring to a local table from your T-
SQL queries or from your BI tools. This approach is recommended if the remote queries do not return large
results.
Figure 3 Vertical partitioning - Using elastic query to query across various databases
The following steps configure elastic database queries for vertical partitioning scenarios that require access to a
table located on remote SQL databases with the same schema:
CREATE MASTER KEY mymasterkey
CREATE DATABASE SCOPED CREDENTIAL mycredential
CREATE/DROP EXTERNAL DATA SOURCE mydatasource of type RDBMS
CREATE/DROP EXTERNAL TABLE mytable
After running the DDL statements, you can access the remote table “mytable” as though it were a local table.
Azure SQL Database automatically opens a connection to the remote database, processes your request on the
remote database, and returns the results.
The following steps configure elastic database queries for horizontal partitioning scenarios that require access
to a set of tables located on (typically) several remote SQL databases:
CREATE MASTER KEY mymasterkey
CREATE DATABASE SCOPED CREDENTIAL mycredential
Create a shard map representing your data tier using the elastic database client library.
CREATE/DROP EXTERNAL DATA SOURCE mydatasource of type SHARD_MAP_MANAGER
CREATE/DROP EXTERNAL TABLE mytable
Once you have performed these steps, you can access the horizontally partitioned table “mytable” as though it
were a local table. Azure SQL Database automatically opens multiple parallel connections to the remote
databases where the tables are physically stored, processes the requests on the remote databases, and returns
the results. More information on the steps required for the horizontal partitioning scenario can be found in
elastic query for horizontal partitioning.
To begin coding, see Getting started with elastic query for horizontal partitioning (sharding) .
T-SQL querying
Once you have defined your external data sources and your external tables, you can use regular SQL Server
connection strings to connect to the databases where you defined your external tables. You can then run T-SQL
statements over your external tables on that connection with the limitations outlined below. You can find more
information and examples of T-SQL queries in the documentation topics for horizontal partitioning and vertical
partitioning.
Cost
Elastic query is included into the cost of Azure SQL Database databases. Note that topologies where your
remote databases are in a different data center than the elastic query endpoint are supported, but data egress
from remote databases is charged regularly Azure rates.
Preview limitations
Running your first elastic query can take up to a few minutes on the Standard performance tier. This time is
necessary to load the elastic query functionality; loading performance improves with higher performance
tiers.
Scripting of external data sources or external tables from SSMS or SSDT is not yet supported.
Import/Export for SQL DB does not yet support external data sources and external tables. If you need to use
Import/Export, drop these objects before exporting and then re-create them after importing.
Elastic query currently only supports read-only access to external tables. You can, however, use full T-SQL
functionality on the database where the external table is defined. This can be useful to, e.g., persist
temporary results using, for example, SELECT <column_list> INTO <local_table>, or to define stored
procedures on the elastic query database that refer to external tables.
Except for nvarchar(max), LOB types are not supported in external table definitions. As a workaround, you
can create a view on the remote database that casts the LOB type into nvarchar(max), define your external
table over the view instead of the base table and then cast it back into the original LOB type in your queries.
Columns of nvarchar(max) data type in result set disable advanced batching technics used in Elastic Query
implementation and may affect performance of query for an order of magnitude, or even two orders of
magnitude in non-canonical use cases where large amount of non-aggregated data is being transferred as a
result of query.
Column statistics over external tables are currently not supported. Table statistics are supported, but need to
be created manually.
Feedback
Share feedback on your experience with elastic queries with us below, on the MSDN forums, or on
Stackoverflow. We are interested in all kinds of feedback about the service (defects, rough edges, feature gaps).
Next steps
For a vertical partitioning tutorial, see Getting started with cross-database query (vertical partitioning) .
For syntax and sample queries for vertically partitioned data, see Querying vertically partitioned data)
For a horizontal partitioning (sharding) tutorial, see Getting started with elastic query for horizontal
partitioning (sharding).
For syntax and sample queries for horizontally partitioned data, see Querying horizontally partitioned data)
See sp_execute _remote for a stored procedure that executes a Transact-SQL statement on a single remote
Azure SQL Database or set of databases serving as shards in a horizontal partitioning scheme.
Get started with cross-database queries (vertical
partitioning) (preview)
7/3/2018 • 2 minutes to read • Edit Online
Elastic database query (preview ) for Azure SQL Database allows you to run T-SQL queries that span multiple
databases using a single connection point. This article applies to vertically partitioned databases.
When completed, you will: learn how to configure and use an Azure SQL Database to perform queries that span
multiple related databases.
For more information about the elastic database query feature, see Azure SQL Database elastic database query
overview.
Prerequisites
ALTER ANY EXTERNAL DATA SOURCE permission is required. This permission is included with the ALTER
DATABASE permission. ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the
underlying data source.
Now, execute following query on the Customers database to create the CustomerInformation table and input
the sample data.
The "username" and "password" should be the username and password used to log in into the Customers
database. Authentication using Azure Active Directory with elastic queries is not currently supported.
External data sources
To create an external data source, execute the following command on the Orders database:
External tables
Create an external table on the Orders database, which matches the definition of the CustomerInformation table:
Cost
Currently, the elastic database query feature is included into the cost of your Azure SQL Database.
For pricing information, see SQL Database Pricing.
Next steps
For an overview of elastic query, see Elastic query overview .
For syntax and sample queries for vertically partitioned data, see Querying vertically partitioned data)
For a horizontal partitioning (sharding) tutorial, see Getting started with elastic query for horizontal
partitioning (sharding).
For syntax and sample queries for horizontally partitioned data, see Querying horizontally partitioned data)
See sp_execute _remote for a stored procedure that executes a Transact-SQL statement on a single remote
Azure SQL Database or set of databases serving as shards in a horizontal partitioning scheme.
Reporting across scaled-out cloud databases
(preview)
6/13/2018 • 7 minutes to read • Edit Online
Sharded databases distribute rows across a scaled out data tier. The schema is identical on all participating
databases, also known as horizontal partitioning. Using an elastic query, you can create reports that span all
databases in a sharded database.
For a quick start, see Reporting across scaled-out cloud databases.
For non-sharded databases, see Query across cloud databases with different schemas.
Prerequisites
Create a shard map using the elastic database client library. see Shard map management. Or use the sample
app in Get started with elastic database tools.
Alternatively, see Migrate existing databases to scaled-out databases.
The user must possess ALTER ANY EXTERNAL DATA SOURCE permission. This permission is included with
the ALTER DATABASE permission.
ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the underlying data source.
Overview
These statements create the metadata representation of your sharded data tier in the elastic query database.
1. CREATE MASTER KEY
2. CREATE DATABASE SCOPED CREDENTIAL
3. CREATE EXTERNAL DATA SOURCE
4. CREATE EXTERNAL TABLE
NOTE
Make sure that the "<username>" does not include any "@servername" suffix.
<External_Data_Source> ::=
CREATE EXTERNAL DATA SOURCE <data_source_name> WITH
(TYPE = SHARD_MAP_MANAGER,
LOCATION = '<fully_qualified_server_name>',
DATABASE_NAME = ‘<shardmap_database_name>',
CREDENTIAL = <credential_name>,
SHARD_MAP_NAME = ‘<shardmapname>’
) [;]
Example
The external data source references your shard map. An elastic query then uses the external data source and the
underlying shard map to enumerate the databases that participate in the data tier. The same credentials are used
to read the shard map and to access the data on the shards during the processing of an elastic query.
<sharded_external_table_options> ::=
DATA_SOURCE = <External_Data_Source>,
[ SCHEMA_NAME = N'nonescaped_schema_name',]
[ OBJECT_NAME = N'nonescaped_object_name',]
DISTRIBUTION = SHARDED(<sharding_column_name>) | REPLICATED |ROUND_ROBIN
Example
WITH
(
DATA_SOURCE = MyExtSrc,
SCHEMA_NAME = 'orders',
OBJECT_NAME = 'order_details',
DISTRIBUTION=SHARDED(ol_w_id)
);
Remarks
The DATA_SOURCE clause defines the external data source (a shard map) that is used for the external table.
The SCHEMA_NAME and OBJECT_NAME clauses map the external table definition to a table in a different
schema. If omitted, the schema of the remote object is assumed to be “dbo” and its name is assumed to be
identical to the external table name being defined. This is useful if the name of your remote table is already taken
in the database where you want to create the external table. For example, you want to define an external table to
get an aggregate view of catalog views or DMVs on your scaled out data tier. Since catalog views and DMVs
already exist locally, you cannot use their names for the external table definition. Instead, use a different name
and use the catalog view’s or the DMV’s name in the SCHEMA_NAME and/or OBJECT_NAME clauses. (See the
example below.)
The DISTRIBUTION clause specifies the data distribution used for this table. The query processor utilizes the
information provided in the DISTRIBUTION clause to build the most efficient query plans.
1. SHARDED means data is horizontally partitioned across the databases. The partitioning key for the data
distribution is the <sharding_column_name> parameter.
2. REPLICATED means that identical copies of the table are present on each database. It is your responsibility
to ensure that the replicas are identical across the databases.
3. ROUND_ROBIN means that the table is horizontally partitioned using an application-dependent distribution
method.
Data tier reference: The external table DDL refers to an external data source. The external data source specifies
a shard map which provides the external table with the information necessary to locate all the databases in your
data tier.
Security considerations
Users with access to the external table automatically gain access to the underlying remote tables under the
credential given in the external data source definition. Avoid undesired elevation of privileges through the
credential of the external data source. Use GRANT or REVOKE for an external table just as though it were a
regular table.
Once you have defined your external data source and your external tables, you can now use full T-SQL over your
external tables.
select
w_id as warehouse,
o_c_id as customer,
count(*) as cnt_orderline,
max(ol_quantity) as max_quantity,
avg(ol_amount) as avg_amount,
min(ol_delivery_d) as min_deliv_date
from warehouse
join orders
on w_id = o_w_id
join order_line
on o_id = ol_o_id and o_w_id = ol_w_id
where w_id > 100 and w_id < 200
group by w_id, o_c_id
EXEC sp_execute_remote
N'MyExtSrc',
N'select count(w_id) as foo from warehouse'
Best practices
Ensure that the elastic query endpoint database has been given access to the shardmap database and all
shards through the SQL DB firewalls.
Validate or enforce the data distribution defined by the external table. If your actual data distribution is
different from the distribution specified in your table definition, your queries may yield unexpected results.
Elastic query currently does not perform shard elimination when predicates over the sharding key would
allow it to safely exclude certain shards from processing.
Elastic query works best for queries where most of the computation can be done on the shards. You typically
get the best query performance with selective filter predicates that can be evaluated on the shards or joins
over the partitioning keys that can be performed in a partition-aligned way on all shards. Other query
patterns may need to load large amounts of data from the shards to the head node and may perform poorly
Next steps
For an overview of elastic query, see Elastic query overview .
For a vertical partitioning tutorial, see Getting started with cross-database query (vertical partitioning) .
For syntax and sample queries for vertically partitioned data, see Querying vertically partitioned data)
For a horizontal partitioning (sharding) tutorial, see Getting started with elastic query for horizontal
partitioning (sharding).
See sp_execute _remote for a stored procedure that executes a Transact-SQL statement on a single remote
Azure SQL Database or set of databases serving as shards in a horizontal partitioning scheme.
Query across cloud databases with different schemas
(preview)
6/13/2018 • 6 minutes to read • Edit Online
Vertically-partitioned databases use different sets of tables on different databases. That means that the schema is
different on different databases. For instance, all tables for inventory are on one database while all accounting-
related tables are on a second database.
Prerequisites
The user must possess ALTER ANY EXTERNAL DATA SOURCE permission. This permission is included with
the ALTER DATABASE permission.
ALTER ANY EXTERNAL DATA SOURCE permissions are needed to refer to the underlying data source.
Overview
NOTE
Unlike with horizontal partitioning, these DDL statements do not depend on defining a data tier with a shard map through
the elastic database client library.
NOTE
Ensure that the <username> does not include any "@servername" suffix.
<External_Data_Source> ::=
CREATE EXTERNAL DATA SOURCE <data_source_name> WITH
(TYPE = RDBMS,
LOCATION = ’<fully_qualified_server_name>’,
DATABASE_NAME = ‘<remote_database_name>’,
CREDENTIAL = <credential_name>
) [;]
IMPORTANT
The TYPE parameter must be set to RDBMS.
Example
The following example illustrates the use of the CREATE statement for external data sources.
External Tables
Syntax:
CREATE EXTERNAL TABLE [ database_name . [ schema_name ] . | schema_name . ] table_name
( { <column_definition> } [ ,...n ])
{ WITH ( <rdbms_external_table_options> ) }
)[;]
<rdbms_external_table_options> ::=
DATA_SOURCE = <External_Data_Source>,
[ SCHEMA_NAME = N'nonescaped_schema_name',]
[ OBJECT_NAME = N'nonescaped_object_name',]
Example
The following example shows how to retrieve the list of external tables from the current database:
Remarks
Elastic query extends the existing external table syntax to define external tables that use external data sources of
type RDBMS. An external table definition for vertical partitioning covers the following aspects:
Schema: The external table DDL defines a schema that your queries can use. The schema provided in your
external table definition needs to match the schema of the tables in the remote database where the actual data
is stored.
Remote database reference: The external table DDL refers to an external data source. The external data
source specifies the logical server name and database name of the remote database where the actual table
data is stored.
Using an external data source as outlined in the previous section, the syntax to create external tables is as follows:
The DATA_SOURCE clause defines the external data source (i.e. the remote database in case of vertical
partitioning) that is used for the external table.
The SCHEMA_NAME and OBJECT_NAME clauses provide the ability to map the external table definition to a
table in a different schema on the remote database, or to a table with a different name, respectively. This is useful
if you want to define an external table to a catalog view or DMV on your remote database - or any other situation
where the remote table name is already taken locally.
The following DDL statement drops an existing external table definition from the local catalog. It does not impact
the remote database.
Permissions for CREATE/DROP EXTERNAL TABLE: ALTER ANY EXTERNAL DATA SOURCE permissions are
needed for external table DDL which is also needed to refer to the underlying data source.
Security considerations
Users with access to the external table automatically gain access to the underlying remote tables under the
credential given in the external data source definition. You should carefully manage access to the external table in
order to avoid undesired elevation of privileges through the credential of the external data source. Regular SQL
permissions can be used to GRANT or REVOKE access to an external table just as though it were a regular table.
SELECT
c_id as customer,
c_lastname as customer_name,
count(*) as cnt_orderline,
max(ol_quantity) as max_quantity,
avg(ol_amount) as avg_amount,
min(ol_delivery_d) as min_deliv_date
FROM customer
JOIN orders
ON c_id = o_c_id
JOIN order_line
ON o_id = ol_o_id and o_c_id = ol_c_id
WHERE c_id = 100
EXEC sp_execute_remote
N'MyExtSrc',
N'select count(w_id) as foo from warehouse'
Best practices
Ensure that the elastic query endpoint database has been given access to the remote database by enabling
access for Azure Services in its SQL DB firewall configuration. Also ensure that the credential provided in the
external data source definition can successfully log into the remote database and has the permissions to
access the remote table.
Elastic query works best for queries where most of the computation can be done on the remote databases.
You typically get the best query performance with selective filter predicates that can be evaluated on the
remote databases or joins that can be performed completely on the remote database. Other query patterns
may need to load large amounts of data from the remote database and may perform poorly.
Next steps
For an overview of elastic query, see Elastic query overview .
For a vertical partitioning tutorial, see Getting started with cross-database query (vertical partitioning) .
For a horizontal partitioning (sharding) tutorial, see Getting started with elastic query for horizontal
partitioning (sharding).
For syntax and sample queries for horizontally partitioned data, see Querying horizontally partitioned data)
See sp_execute _remote for a stored procedure that executes a Transact-SQL statement on a single remote
Azure SQL Database or set of databases serving as shards in a horizontal partitioning scheme.
Distributed transactions across cloud databases
9/12/2018 • 7 minutes to read • Edit Online
Elastic database transactions for Azure SQL Database (SQL DB ) allow you to run transactions that span several
databases in SQL DB. Elastic database transactions for SQL DB are available for .NET applications using ADO
.NET and integrate with the familiar programming experience using the System.Transaction classes. To get the
library, see .NET Framework 4.6.1 (Web Installer).
On premises, such a scenario usually required running Microsoft Distributed Transaction Coordinator (MSDTC ).
Since MSDTC is not available for Platform-as-a-Service application in Azure, the ability to coordinate distributed
transactions has now been directly integrated into SQL DB. Applications can connect to any SQL Database to
launch distributed transactions, and one of the databases will transparently coordinate the distributed transaction,
as shown in the following figure.
Common scenarios
Elastic database transactions for SQL DB enable applications to make atomic changes to data stored in several
different SQL Databases. The preview focuses on client-side development experiences in C# and .NET. A server-
side experience using T-SQL is planned for a later time.
Elastic database transactions targets the following scenarios:
Multi-database applications in Azure: With this scenario, data is vertically partitioned across several databases
in SQL DB such that different kinds of data reside on different databases. Some operations require changes to
data which is kept in two or more databases. The application uses elastic database transactions to coordinate
the changes across databases and ensure atomicity.
Sharded database applications in Azure: With this scenario, the data tier uses the Elastic Database client library
or self-sharding to horizontally partition the data across many databases in SQL DB. One prominent use case
is the need to perform atomic changes for a sharded multi-tenant application when changes span tenants.
Think for instance of a transfer from one tenant to another, both residing on different databases. A second case
is fine-grained sharding to accommodate capacity needs for a large tenant which in turn typically implies that
some atomic operations needs to stretch across several databases used for the same tenant. A third case is
atomic updates to reference data that are replicated across databases. Atomic, transacted, operations along
these lines can now be coordinated across several databases using the preview. Elastic database transactions
use two-phase commit to ensure transaction atomicity across databases. It is a good fit for transactions that
involve less than 100 databases at a time within a single transaction. These limits are not enforced, but one
should expect performance and success rates for elastic database transactions to suffer when exceeding these
limits.
Development experience
Multi-database applications
The following sample code uses the familiar programming experience with .NET System.Transactions. The
TransactionScope class establishes an ambient transaction in .NET. (An “ambient transaction” is one that lives in
the current thread.) All connections opened within the TransactionScope participate in the transaction. If different
databases participate, the transaction is automatically elevated to a distributed transaction. The outcome of the
transaction is controlled by setting the scope to complete to indicate a commit.
scope.Complete();
}
Sharded database applications
Elastic database transactions for SQL DB also support coordinating distributed transactions where you use the
OpenConnectionForKey method of the elastic database client library to open connections for a scaled out data tier.
Consider cases where you need to guarantee transactional consistency for changes across several different
sharding key values. Connections to the shards hosting the different sharding key values are brokered using
OpenConnectionForKey. In the general case, the connections can be to different shards such that ensuring
transactional guarantees requires a distributed transaction. The following code sample illustrates this approach. It
assumes that a variable called shardmap is used to represent a shard map from the elastic database client library:
scope.Complete();
}
Limitations
The following limitations currently apply to elastic database transactions in SQL DB:
Only transactions across databases in SQL DB are supported. Other X/Open XA resource providers and
databases outside of SQL DB cannot participate in elastic database transactions. That means that elastic
database transactions cannot stretch across on premises SQL Server and Azure SQL Database. For distributed
transactions on premises, continue to use MSDTC.
Only client-coordinated transactions from a .NET application are supported. Server-side support for T-SQL
such as BEGIN DISTRIBUTED TRANSACTION is planned, but not yet available.
Transactions across WCF services are not supported. For example, you have a WCF service method that
executes a transaction. Enclosing the call within a transaction scope will fail as a
System.ServiceModel.ProtocolException.
Next steps
For questions, please reach out to us on the SQL Database forum and for feature requests, please add them to the
SQL Database feedback forum.
Multi-tenant SaaS database tenancy patterns
6/4/2018 • 12 minutes to read • Edit Online
When designing a multi-tenant SaaS application, you must carefully choose the tenancy model that best fits the
needs of your application. A tenancy model determines how each tenant’s data is mapped to storage. Your choice
of tenancy model impacts application design and management. Switching to a different model later is sometimes
costly.
This article describes alternative tenancy models.
Each app instance is installed in a separate Azure resource group. The resource group can belong to a
subscription that is owned by either the software vendor or the tenant. In either case, the vendor can manage the
software for the tenant. Each application instance is configured to connect to its corresponding database.
Each tenant database is deployed as a standalone database. This model provides the greatest database isolation.
But the isolation requires that sufficient resources be allocated to each database to handle its peak loads. Here it
matters that elastic pools cannot be used for databases deployed in different resource groups or to different
subscriptions. This limitation makes this standalone single-tenant app model the most expensive solution from an
overall database cost perspective.
Vendor management
The vendor can access all the databases in all the standalone app instances, even if the app instances are installed
in different tenant subscriptions. The access is achieved via SQL connections. This cross-instance access can
enable the vendor to centralize schema management and cross-database query for reporting or analytics
purposes. If this kind of centralized management is desired, a catalog must be deployed that maps tenant
identifiers to database URIs. Azure SQL Database provides a sharding library that is used together with a SQL
database to provide a catalog. The sharding library is formally named the Elastic Database Client Library.
Database cost per tenant High; is sized for peaks. Low; pools used. Lowest, for small tenants in
MT DBs.
Next steps
Deploy and explore a multi-tenant Wingtip application that uses the database-per-tenant SaaS model -
Azure SQL Database
Welcome to the Wingtip Tickets sample SaaS Azure SQL Database tenancy app
Video indexed and annotated for multi-tenant SaaS
app using Azure SQL Database
5/23/2018 • 4 minutes to read • Edit Online
This article is an annotated index into the time locations of an 81 minute video about SaaS tenancy models or
patterns. This article enables you to skip backward or forward in the video to which portion interests you. The
video explains the major design options for a multi-tenant database application on Azure SQL Database. The video
includes demos, walkthroughs of management code, and at times more detail informed by experience than might
be in our written documentation.
The video amplifies information in our written documentation, found at:
Conceptual: Multi-tenant SaaS database tenancy patterns
Tutorials: The Wingtip Tickets SaaS application
The video and the articles describe the many phases of creating a multi-tenant application on Azure SQL Database
in the cloud. Special features of Azure SQL Database make it easier to develop and implement multi-tenant apps
that are both easier to manage and reliably performant.
We routinely update our written documentation. The video is not edited or updated, so eventually more of its detail
may become outdated.
3. Agenda, 0:04:09
4. Multi-tenant web app, 0:05:00
Next steps
First tutorial article
Multi-tenant applications with elastic database tools
and row-level security
9/12/2018 • 11 minutes to read • Edit Online
Elastic database tools and row -level security (RLS ) cooperate to enable scaling the data tier of a multi-tenant
application with Azure SQL Database. Together these technologies help you build an application that has a highly
scalable data tier. The data tier supports multi-tenant shards, and uses ADO.NET SqlClient or Entity
Framework. For more information, see Design Patterns for Multi-tenant SaaS Applications with Azure SQL
Database.
Elastic database tools enable developers to scale out the data tier with standard sharding practices, by using
.NET libraries and Azure service templates. Managing shards by using the Elastic Database Client Library helps
automate and streamline many of the infrastructural tasks typically associated with sharding.
Row-level security enables developers to safely store data for multiple tenants in the same database. RLS
security policies filter out rows that do not belong to the tenant executing a query. Centralizing the filter logic
inside the database simplifies maintenance and reduces the risk of a security error. The alternative of relying on
all client code to enfore security is risky.
By using these features together, an application can store data for multiple tenants in the same shard database. It
costs less per tenant when the tenants share a database. Yet the same application can also offer its premium
tenants the option of paying for their own dedicated single-tenant shard. One benefit of single-tenant isolation is
firmer performance guarantees. In a single-tenant database, there is no other tenant competing for resources.
The goal is to use the elastic database client library data-dependent routing APIs to automatically connect each
given tenant to the correct shard database. Only one shard contains particular TenantId value for the given tenant.
The TenantId is the sharding key. After the connection is established, an RLS security policy within the database
ensures that the given tenant can access only those data rows that contain its TenantId.
NOTE
The tenant identifier might consist of more than one column. For convenience is this discussion, we informally assume a
single-column TenantId.
// Ask shard map to broker a validated connection for the given key.
SqlConnection conn = null;
try
{
conn = shardMap.OpenConnectionForKey(
shardingKey,
connectionStr,
ConnectionOptions.Validate);
return conn;
}
catch (Exception)
{
if (conn != null)
{
conn.Dispose();
}
throw;
}
}
// ...
Now the SESSION_CONTEXT is automatically set with the specified TenantId whenever ElasticScaleContext is
invoked:
// Program.cs
SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() =>
{
using (var db = new ElasticScaleContext<int>(
sharding.ShardMap, tenantId, connStrBldr.ConnectionString))
{
var query = from b in db.Blogs
orderby b.Name
select b;
ADO.NET SqlClient
For applications using ADO.NET SqlClient, create a wrapper function around method
ShardMap.OpenConnectionForKey. Have the wrapper automatically set TenantId in the SESSION_CONTEXT to
the current TenantId before returning a connection. To ensure that SESSION_CONTEXT is always set, you should
only open connections using this wrapper function.
// Program.cs
// Wrapper function for ShardMap.OpenConnectionForKey() that
// automatically sets SESSION_CONTEXT with the correct
// tenantId before returning a connection.
// As a best practice, you should only open connections using this method
// to ensure that SESSION_CONTEXT is always set before executing a query.
// ...
public static SqlConnection OpenConnectionForTenant(
ShardMap shardMap, int tenantId, string connectionStr)
{
SqlConnection conn = null;
try
{
// Ask shard map to broker a validated connection for the given key.
conn = shardMap.OpenConnectionForKey(
tenantId, connectionStr, ConnectionOptions.Validate);
return conn;
}
catch (Exception)
{
if (conn != null)
{
conn.Dispose();
}
throw;
}
}
// ...
Console.WriteLine(@"--
All blogs for TenantId {0} (using ADO.NET SqlClient):", tenantId4);
TIP
In a complex project you might need to add the predicate on hundreds of tables, which could be tedious. There is a helper
stored procedure that automatically generates a security policy, and adds a predicate on all tables in a schema. For more
information, see the blog post at Apply Row-Level Security to all tables - helper script (blog).
Now if you run the sample application again, tenants see only rows that belong to them. In addition, the application
cannot insert rows that belong to tenants other than the one currently connected to the shard database. Also, the
app cannot update the TenantId in any rows it can see. If the app attempts to do either, a DbUpdateException is
raised.
If you add a new table later, ALTER the security policy to add FILTER and BLOCK predicates on the new table.
Now the application does not need to specify a TenantId when inserting rows:
SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() =>
{
using (var db = new ElasticScaleContext<int>(
sharding.ShardMap, tenantId, connStrBldr.ConnectionString))
{
// The default constraint sets TenantId automatically!
var blog = new Blog { Name = name };
db.Blogs.Add(blog);
db.SaveChanges();
}
});
NOTE
If you use default constraints for an Entity Framework project, it is recommended that you NOT include the TenantId column
in your EF data model. This recommendation is because Entity Framework queries automatically supply default values that
override the default constraints created in T-SQL that use SESSION_CONTEXT. To use default constraints in the sample
project, for instance, you should remove TenantId from DataClasses.cs (and run Add-Migration in the Package Manager
Console) and use T-SQL to ensure that the field only exists in the database tables. This way, EF does automatically supply
incorrect default values when inserting data.
Maintenance
Adding new shards: Execute the T-SQL script to enable RLS on any new shards, otherwise queries on these
shards are not be filtered.
Adding new tables: Add a FILTER and BLOCK predicate to the security policy on all shards whenever a new
table is created. Otherwise queries on the new table are not be filtered. This addition can be automated by using
a DDL trigger, as described in Apply Row -Level Security automatically to newly created tables (blog).
Summary
Elastic database tools and row -level security can be used together to scale out an application’s data tier with
support for both multi-tenant and single-tenant shards. Multi-tenant shards can be used to store data more
efficiently. This efficiency is pronounced where a large number of tenants have only a few rows of data. Single-
tenant shards can support premium tenants which have stricter performance and isolation requirements. For more
information, see Row -Level Security reference.
Additional resources
What is an Azure elastic pool?
Scaling out with Azure SQL Database
Design Patterns for Multi-tenant SaaS Applications with Azure SQL Database
Authentication in multitenant apps, using Azure AD and OpenID Connect
Tailspin Surveys application
This article walks through the basic considerations that a developer should be aware of when writing code to
connect to Azure SQL Database.
TIP
For a tutorial showing you how to create a server, create a server-based firewall, view server properties, connect using SQL
Server Management Studio, query the master database, create a sample database and a blank database, query database
properties, connect using SQL Server Management Studio, and query the sample database, see Get Started Tutorial.
Tools
You can leverage open-source tools like cheetah, sql-cli, VS Code. Additionally, Azure SQL Database works with
Microsoft tools like Visual Studio and SQL Server Management Studio. You can also use the Azure Management
Portal, PowerShell, and REST APIs help you gain additional productivity.
Resource limitations
Azure SQL Database manages the resources available to a database using two different mechanisms: Resources
governance and enforcement of limits. For more information, see:
DTU -based resource model limits - Single Database
DTU -based resource model limits - Elastic Pools
vCore-based resource limits - Single Databases
vCore-based resource limits - Elastic Pools
Security
Azure SQL Database provides resources for limiting access, protecting data, and monitoring activities on a SQL
Database.
More information: Securing your SQL Database.
Authentication
Azure SQL Database supports both SQL Server authentication users and logins, as well as Azure Active
Directory authentication users and logins.
You need to specify a particular database, instead of defaulting to the master database.
You cannot use the Transact-SQL USE myDatabaseName; statement on SQL Database to switch to another
database.
More information: SQL Database security: Manage database access and login security.
Resiliency
When a transient error occurs while connecting to SQL Database, your code should retry the call. We recommend
that retry logic use backoff logic, so that it does not overwhelm the SQL Database with multiple clients retrying
simultaneously.
Code samples: For code samples that illustrate retry logic, see samples for the language of your choice at:
Connection libraries for SQL Database and SQL Server.
More information: Error messages for SQL Database client programs.
Managing connections
In your client connection logic, override the default timeout to be 30 seconds. The default of 15 seconds is too
short for connections that depend on the internet.
If you are using a connection pool, be sure to close the connection the instant your program is not actively
using it, and is not preparing to reuse it.
Network considerations
On the computer that hosts your client program, ensure the firewall allows outgoing TCP communication on
port 1433. More information: Configure an Azure SQL Database firewall.
If your client program connects to SQL Database while your client runs on an Azure virtual machine (VM ), you
must open certain port ranges on the VM. More information: Ports beyond 1433 for ADO.NET 4.5 and SQL
Database.
Client connections to Azure SQL Database sometimes bypass the proxy and interact directly with the
database. Ports other than 1433 become important. For more information, Azure SQL Database connectivity
architecture and Ports beyond 1433 for ADO.NET 4.5 and SQL Database.
Next steps
Explore all the capabilities of SQL Database.
Getting started with JSON features in Azure SQL
Database
5/23/2018 • 6 minutes to read • Edit Online
Azure SQL Database lets you parse and query data represented in JavaScript Object Notation (JSON ) format, and
export your relational data as JSON text.
JSON is a popular data format used for exchanging data in modern web and mobile applications. JSON is also
used for storing semi-structured data in log files or in NoSQL databases like Azure Cosmos DB. Many REST web
services return results formatted as JSON text or accept data formatted as JSON. Most Azure services such as
Azure Search, Azure Storage, and Azure Cosmos DB have REST endpoints that return or consume JSON.
Azure SQL Database lets you work with JSON data easily and integrate your database with modern services.
Overview
Azure SQL Database provides the following functions for working with JSON data:
If you have JSON text, you can extract data from JSON or verify that JSON is properly formatted by using the
built-in functions JSON_VALUE, JSON_QUERY, and ISJSON. The JSON_MODIFY function lets you update value
inside JSON text. For more advanced querying and analysis, OPENJSON function can transform an array of
JSON objects into a set of rows. Any SQL query can be executed on the returned result set. Finally, there is a FOR
JSON clause that lets you format data stored in your relational tables as JSON text.
The FOR JSON PATH clause formats the results of the query as JSON text. Column names are used as keys, while
the cell values are generated as JSON values:
[
{"CustomerName":"Eric Torres","PhoneNumber":"(307) 555-0100","FaxNumber":"(307) 555-0101"},
{"CustomerName":"Cosmina Vlad","PhoneNumber":"(505) 555-0100","FaxNumber":"(505) 555-0101"},
{"CustomerName":"Bala Dixit","PhoneNumber":"(209) 555-0100","FaxNumber":"(209) 555-0101"}
]
The result set is formatted as a JSON array where each row is formatted as a separate JSON object.
PATH indicates that you can customize the output format of your JSON result by using dot notation in column
aliases. The following query changes the name of the "CustomerName" key in the output JSON format, and puts
phone and fax numbers in the "Contact" sub-object:
{
"Name":"Nada Jovanovic",
"Contact":{
"Phone":"(215) 555-0100",
"Fax":"(215) 555-0101"
}
}
In this example we returned a single JSON object instead of an array by specifying the
WITHOUT_ARRAY_WRAPPER option. You can use this option if you know that you are returning a single object
as a result of query.
The main value of the FOR JSON clause is that it lets you return complex hierarchical data from your database
formatted as nested JSON objects or arrays. The following example shows how to include Orders that belong to
the Customer as a nested array of Orders:
Instead of sending separate queries to get Customer data and then to fetch a list of related Orders, you can get all
the necessary data with a single query, as shown in the following sample output:
{
"Name":"Nada Jovanovic",
"Phone":"(215) 555-0100",
"Fax":"(215) 555-0101",
"Orders":[
{"OrderID":382,"OrderDate":"2013-01-07","ExpectedDeliveryDate":"2013-01-08"},
{"OrderID":395,"OrderDate":"2013-01-07","ExpectedDeliveryDate":"2013-01-08"},
{"OrderID":1657,"OrderDate":"2013-01-31","ExpectedDeliveryDate":"2013-02-01"}
]
}
The JSON data used in this example is represented by using the NVARCHAR (MAX) type. JSON can be inserted
into this table or provided as an argument of the stored procedure using standard Transact-SQL syntax as shown
in the following example:
Any client-side language or library that works with string data in Azure SQL Database will also work with JSON
data. JSON can be stored in any table that supports the NVARCHAR type, such as a Memory-optimized table or a
System-versioned table. JSON does not introduce any constraint either in the client-side code or in the database
layer.
update Products
set Data = JSON_MODIFY(Data, '$.Price', 60)
where Id = 1
The JSON_VALUE function extracts a value from JSON text stored in the Data column. This function uses a
JavaScript-like path to reference a value in JSON text to extract. The extracted value can be used in any part of
SQL query.
The JSON_QUERY function is similar to JSON_VALUE. Unlike JSON_VALUE, this function extracts complex sub-
object such as arrays or objects that are placed in JSON text.
The JSON_MODIFY function lets you specify the path of the value in the JSON text that should be updated, as
well as a new value that will overwrite the old one. This way you can easily update JSON text without reparsing the
entire structure.
Since JSON is stored in a standard text, there are no guarantees that the values stored in text columns are properly
formatted. You can verify that text stored in JSON column is properly formatted by using standard Azure SQL
Database check constraints and the ISJSON function:
If the input text is properly formatted JSON, the ISJSON function returns the value 1. On every insert or update of
JSON column, this constraint will verify that new text value is not malformed JSON.
In the example above, we can specify where to locate the JSON array that should be opened (in the $.Orders path),
what columns should be returned as result, and where to find the JSON values that will be returned as cells.
We can transform a JSON array in the @orders variable into a set of rows, analyze this result set, or insert rows
into a standard table:
END
The collection of orders formatted as a JSON array and provided as a parameter to the stored procedure can be
parsed and inserted into the Orders table.
Next steps
To learn how to integrate JSON into your application, check out these resources:
TechNet Blog
MSDN documentation
Channel 9 video
To learn about various scenarios for integrating JSON into your application, see the demos in this Channel 9 video
or find a scenario that matches your use case in JSON Blog posts.
Optimize performance by using In-Memory
technologies in SQL Database
7/16/2018 • 17 minutes to read • Edit Online
By using In-Memory technologies in Azure SQL Database, you can achieve performance improvements with
various workloads: transactional (online transactional processing (OLTP )), analytics (online analytical processing
(OLAP )), and mixed (hybrid transaction/analytical processing (HTAP )). Because of the more efficient query and
transaction processing, In-Memory technologies also help you to reduce cost. You typically don't need to upgrade
the pricing tier of the database to achieve performance gains. In some cases, you might even be able reduce the
pricing tier, while still seeing performance improvements with In-Memory technologies.
Here are two examples of how In-Memory OLTP helped to significantly improve performance:
By using In-Memory OLTP, Quorum Business Solutions was able to double their workload while improving
DTUs by 70%.
DTU means database transaction unit, and it includes a measurement of resource consumption.
The following video demonstrates significant improvement in resource consumption with a sample workload:
In-Memory OLTP in Azure SQL Database Video.
For more information, see the blog post: In-Memory OLTP in Azure SQL Database Blog Post
In-Memory technologies are available in all databases in the Premium tier, including databases in Premium
elastic pools.
The following video explains potential performance gains with In-Memory technologies in Azure SQL Database.
Remember that the performance gain that you see always depends on many factors, including the nature of the
workload and data, access pattern of the database, and so on.
Downgrading to a lower Premium tier: Data in memory-optimized tables must fit within the In-Memory OLTP
storage that is associated with the pricing tier of the database or is available in the elastic pool. If you try to lower
the pricing tier or move the database into a pool that doesn't have enough available In-Memory OLTP storage,
the operation fails.
Columnstore indexes
Downgrading to Basic or Standard: Columnstore indexes are supported only on the Premium pricing tier and on
the Standard tier, S3 and above, and not on the Basic tier. When you downgrade your database to an
unsupported tier or level, your columnstore index becomes unavailable. The system maintains your columnstore
index, but it never leverages the index. If you later upgrade back to a supported tier or level, your columnstore
index is immediately ready to be leveraged again.
If you have a clustered columnstore index, the whole table becomes unavailable after the downgrade. Therefore
we recommend that you drop all clustered columnstore indexes before you downgrade your database to an
unsupported tier or level.
Downgrading to a lower supported tier or level: This downgrade succeeds if the whole database fits within the
maximum database size for the target pricing tier, or within the available storage in the elastic pool. There is no
specific impact from the columnstore indexes.
Error 40536
If you get error 40536 when you run the T-SQL script, run the following T-SQL script to verify whether the
database supports In-Memory:
A result of 0 means that In-Memory isn't supported, and 1 means that it is supported. To diagnose the problem,
ensure that the database is at the Premium service tier.
About the created memory-optimized items
Tables: The sample contains the following memory-optimized tables:
SalesLT.Product_inmem
SalesLT.SalesOrderHeader_inmem
SalesLT.SalesOrderDetail_inmem
Demo.DemoSalesOrderHeaderSeed
Demo.DemoSalesOrderDetailSeed
You can inspect memory-optimized tables through the Object Explorer in SSMS. Right-click Tables > Filter >
Filter Settings > Is Memory Optimized. The value equals 1.
Or you can query the catalog views, such as:
Natively compiled stored procedure: You can inspect SalesLT.usp_InsertSalesOrder_inmem through a catalog
view query:
DECLARE
@i int = 0,
@od SalesLT.SalesOrderDetailType_inmem,
@SalesOrderID int,
@DueDate datetime2 = sysdatetime(),
@CustomerID int = rand() * 8000,
@BillToAddressID int = rand() * 10000,
@ShipToAddressID int = rand() * 10000;
To make the _ondisk version of the preceding T-SQL script for ostress.exe, you would replace both occurrences of
the _inmem substring with _ondisk. These replacements affect the names of tables and stored procedures.
Install RML utilities and ostress
Ideally, you would plan to run ostress.exe on an Azure virtual machine (VM ). You would create an Azure VM in
the same Azure geographic region where your AdventureWorksLT database resides. But you can run ostress.exe
on your laptop instead.
On the VM, or on whatever host you choose, install the Replay Markup Language (RML ) utilities. The utilities
include ostress.exe.
For more information, see:
The ostress.exe discussion in Sample Database for In-Memory OLTP.
Sample Database for In-Memory OLTP.
The blog for installing ostress.exe.
Run the _inmem stress workload first
You can use an RML Cmd Prompt window to run our ostress.exe command line. The command-line parameters
direct ostress to:
Run 100 connections concurrently (-n100).
Have each connection run the T-SQL script 50 times (-r50).
EXECUTE Demo.usp_DemoReset;
2. Copy the text of the preceding ostress.exe command line to your clipboard.
3. Replace the <placeholders> for the parameters -S -U -P -d with the correct real values.
4. Run your edited command line in an RML Cmd window.
Result is a duration
When ostress.exe finishes, it writes the run duration as its final line of output in the RML Cmd window. For
example, a shorter test run lasted about 1.5 minutes:
11/12/15 00:35:00.873 [0x000030A8] OSTRESS exiting normally, elapsed time: 00:01:31.867
EXECUTE Demo.usp_DemoReset;
2. Edit the ostress.exe command line to replace all _inmem with _ondisk.
3. Rerun ostress.exe for the second time, and capture the duration result.
4. Again, reset the database (for responsibly deleting what can be a large amount of test data).
Expected comparison results
Our In-Memory tests have shown that performance improved by nine times for this simplistic workload, with
ostress running on an Azure VM in the same Azure region as the database.
Level 130 is not directly related to In-Memory features. But level 130 generally provides faster query
performance than 120.
Key tables and columnstore indexes
dbo.FactResellerSalesXL_CCI is a table that has a clustered columnstore index, which has advanced
compression at the data level.
dbo.FactResellerSalesXL_PageCompressed is a table that has an equivalent regular clustered index, which
is compressed only at the page level.
Key queries to compare the columnstore index
There are several T-SQL query types that you can run to see performance improvements. In step 2 in the T-SQL
script, pay attention to this pair of queries. They differ only on one line:
FROM FactResellerSalesXL_PageCompressed a
FROM FactResellerSalesXL_CCI a
-- Execute a typical query that joins the Fact Table with dimension tables
-- Note this query will run on the Page Compressed table, Note down the time
SET STATISTICS IO ON
SET STATISTICS TIME ON
GO
SELECT c.Year
,e.ProductCategoryKey
,FirstName + ' ' + LastName AS FullName
,count(SalesOrderNumber) AS NumSales
,sum(SalesAmount) AS TotalSalesAmt
,Avg(SalesAmount) AS AvgSalesAmt
,count(DISTINCT SalesOrderNumber) AS NumOrders
,count(DISTINCT a.CustomerKey) AS CountCustomers
FROM FactResellerSalesXL_PageCompressed a
INNER JOIN DimProduct b ON b.ProductKey = a.ProductKey
INNER JOIN DimCustomer d ON d.CustomerKey = a.CustomerKey
Inner JOIN DimProductSubCategory e on e.ProductSubcategoryKey = b.ProductSubcategoryKey
INNER JOIN DimDate c ON c.DateKey = a.OrderDateKey
GROUP BY e.ProductCategoryKey,c.Year,d.CustomerKey,d.FirstName,d.LastName
GO
SET STATISTICS IO OFF
SET STATISTICS TIME OFF
GO
-- This is the same Prior query on a table with a clustered columnstore index CCI
-- The comparison numbers are even more dramatic the larger the table is (this is an 11 million row table
only)
SET STATISTICS IO ON
SET STATISTICS TIME ON
GO
SELECT c.Year
,e.ProductCategoryKey
,FirstName + ' ' + LastName AS FullName
,count(SalesOrderNumber) AS NumSales
,sum(SalesAmount) AS TotalSalesAmt
,Avg(SalesAmount) AS AvgSalesAmt
,count(DISTINCT SalesOrderNumber) AS NumOrders
,count(DISTINCT a.CustomerKey) AS CountCustomers
FROM FactResellerSalesXL_CCI a
INNER JOIN DimProduct b ON b.ProductKey = a.ProductKey
INNER JOIN DimCustomer d ON d.CustomerKey = a.CustomerKey
Inner JOIN DimProductSubCategory e on e.ProductSubcategoryKey = b.ProductSubcategoryKey
INNER JOIN DimDate c ON c.DateKey = a.OrderDateKey
GROUP BY e.ProductCategoryKey,c.Year,d.CustomerKey,d.FirstName,d.LastName
GO
In a database with the P2 pricing tier, you can expect about nine times the performance gain for this query by
using the clustered columnstore index compared with the traditional index. With P15, you can expect about 57
times the performance gain by using the columnstore index.
Next steps
Quick Start 1: In-Memory OLTP Technologies for Faster T-SQL Performance
Use In-Memory OLTP in an existing Azure SQL application
Monitor In-Memory OLTP storage for In-Memory OLTP
Additional resources
Deeper information
Learn how Quorum doubles key database’s workload while lowering DTU by 70% with In-Memory OLTP
in SQL Database
In-Memory OLTP in Azure SQL Database Blog Post
Learn about In-Memory OLTP
Learn about columnstore indexes
Learn about real-time operational analytics
See Common Workload Patterns and Migration Considerations (which describes workload patterns where
In-Memory OLTP commonly provides significant performance gains)
Application design
In-Memory OLTP (In-Memory Optimization)
Use In-Memory OLTP in an existing Azure SQL application
Tools
Azure portal
SQL Server Management Studio (SSMS )
SQL Server Data Tools (SSDT)
Getting Started with Temporal Tables in Azure SQL
Database
5/23/2018 • 7 minutes to read • Edit Online
Temporal Tables are a new programmability feature of Azure SQL Database that allows you to track and analyze
the full history of changes in your data, without the need for custom coding. Temporal Tables keep data closely
related to time context so that stored facts can be interpreted as valid only within the specific period. This property
of Temporal Tables allows for efficient time-based analysis and getting insights from data evolution.
Temporal Scenario
This article illustrates the steps to utilize Temporal Tables in an application scenario. Suppose that you want to
track user activity on a new website that is being developed from scratch or on an existing website that you want to
extend with user activity analytics. In this simplified example, we assume that the number of visited web pages
during a period of time is an indicator that needs to be captured and monitored in the website database that is
hosted on Azure SQL Database. The goal of the historical analysis of user activity is to get inputs to redesign
website and provide better experience for the visitors.
The database model for this scenario is very simple - user activity metric is represented with a single integer field,
PageVisited, and is captured along with basic information on the user profile. Additionally, for time based
analysis, you would keep a series of rows for each user, where every row represents the number of pages a
particular user visited within a specific period of time.
Fortunately, you do not need to put any effort in your app to maintain this activity information. With Temporal
Tables, this process is automated - giving you full flexibility during website design and more time to focus on the
data analysis itself. The only thing you have to do is to ensure that WebSiteInfo table is configured as temporal
system-versioned. The exact steps to utilize Temporal Tables in this scenario are described below.
In SSDT, chose “Temporal Table (System-Versioned)” template when adding new items to the database project.
That will open table designer and enable you to easily specify the table layout:
You can also a create temporal table by specifying the Transact-SQL statements directly, as shown in the example
below. Note that the mandatory elements of every temporal table are the PERIOD definition and the
SYSTEM_VERSIONING clause with a reference to another user table that will store historical row versions:
When you create system-versioned temporal table, the accompanying history table with the default configuration
is automatically created. The default history table contains a clustered B -tree index on the period columns (end,
start) with page compression enabled. This configuration is optimal for the majority of scenarios in which temporal
tables are used, especially for data auditing.
In this particular case, we aim to perform time-based trend analysis over a longer data history and with bigger
data sets, so the storage choice for the history table is a clustered columnstore index. A clustered columnstore
provides very good compression and performance for analytical queries. Temporal Tables give you the flexibility to
configure indexes on the current and temporal tables completely independently.
NOTE
Columnstore indexes are available in the Premium tier and in the Standard tier, S3 and above.
The following script shows how default index on history table can be changed to the clustered columnstore:
CREATE CLUSTERED COLUMNSTORE INDEX IX_WebsiteUserInfoHistory
ON dbo.WebsiteUserInfoHistory
WITH (DROP_EXISTING = ON);
Temporal Tables are represented in the Object Explorer with the specific icon for easier identification, while its
history table is displayed as a child node.
It is important to notice that the update query doesn’t need to know the exact time when the actual operation
occurred nor how historical data will be preserved for future analysis. Both aspects are automatically handled by
the Azure SQL Database. The following diagram illustrates how history data is being generated on every update.
You can easily modify this query to analyze the site visits as of a day ago, a month ago or at any point in the past
you wish.
To perform basic statistical analysis for the previous day, use the following example:
To search for activities of a specific user, within a period of time, use the CONTAINED IN clause:
DECLARE @hourAgo datetime2 = DATEADD(HOUR, -1, SYSUTCDATETIME());
DECLARE @twoHoursAgo datetime2 = DATEADD(HOUR, -2, SYSUTCDATETIME());
SELECT * FROM dbo.WebsiteUserInfo
FOR SYSTEM_TIME CONTAINED IN (@twoHoursAgo, @hourAgo)
WHERE [UserID] = 1;
Graphic visualization is especially convenient for temporal queries as you can show trends and usage patterns in
an intuitive way very easily:
Similarly, you can change column definition while your workload is active:
Finally, you can remove a column that you do not need anymore.
Alternatively, use latest SSDT to change temporal table schema while you are connected to the database (online
mode) or as part of the database project (offline mode).
Next steps
For detailed information on Temporal Tables, check out MSDN documentation. Visit Channel 9 to hear a real
customer temporal implemenation success story and watch a live temporal demonstration.
Manage historical data in Temporal Tables with
retention policy
5/23/2018 • 7 minutes to read • Edit Online
Temporal Tables may increase database size more than regular tables, especially if you retain historical data for a
longer period of time. Hence, retention policy for historical data is an important aspect of planning and managing
the lifecycle of every temporal table. Temporal Tables in Azure SQL Database come with easy-to-use retention
mechanism that helps you accomplish this task.
Temporal history retention can be configured at the individual table level, which allows users to create flexible
aging polices. Applying temporal retention is simple: it requires only one parameter to be set during table creation
or schema change.
After you define retention policy, Azure SQL Database starts checking regularly if there are historical rows that are
eligible for automatic data cleanup. Identification of matching rows and their removal from the history table occur
transparently, in the background task that is scheduled and run by the system. Age condition for the history table
rows is checked based on the column representing end of SYSTEM_TIME period. If retention period, for example,
is set to six months, table rows eligible for cleanup satisfy the following condition:
In the preceding example, we assumed that ValidTo column corresponds to the end of SYSTEM_TIME period.
Database flag is_temporal_history_retention_enabled is set to ON by default, but users can change it with
ALTER DATABASE statement. It is also automatically set to OFF after point in time restore operation. To enable
temporal history retention cleanup for your database, execute the following statement:
IMPORTANT
You can configure retention for temporal tables even if is_temporal_history_retention_enabled is OFF, but automatic
cleanup for aged rows is not triggered in that case.
Retention policy is configured during table creation by specifying value for the HISTORY_RETENTION_PERIOD
parameter:
CREATE TABLE dbo.WebsiteUserInfo
(
[UserID] int NOT NULL PRIMARY KEY CLUSTERED
, [UserName] nvarchar(100) NOT NULL
, [PagesVisited] int NOT NULL
, [ValidFrom] datetime2 (0) GENERATED ALWAYS AS ROW START
, [ValidTo] datetime2 (0) GENERATED ALWAYS AS ROW END
, PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)
)
WITH
(
SYSTEM_VERSIONING = ON
(
HISTORY_TABLE = dbo.WebsiteUserInfoHistory,
HISTORY_RETENTION_PERIOD = 6 MONTHS
)
);
Azure SQL Database allows you to specify retention period by using different time units: DAYS, WEEKS,
MONTHS, and YEARS. If HISTORY_RETENTION_PERIOD is omitted, INFINITE retention is assumed. You can
also use INFINITE keyword explicitly.
In some scenarios, you may want to configure retention after table creation, or to change previously configured
value. In that case use ALTER TABLE statement:
IMPORTANT
Setting SYSTEM_VERSIONING to OFF does not preserve retention period value. Setting SYSTEM_VERSIONING to ON without
HISTORY_RETENTION_PERIOD specified explicitly results in the INFINITE retention period.
To review current state of the retention policy, use the following query that joins temporal retention enablement
flag at the database level with retention periods for individual tables:
SELECT DB.is_temporal_history_retention_enabled,
SCHEMA_NAME(T1.schema_id) AS TemporalTableSchema,
T1.name as TemporalTableName, SCHEMA_NAME(T2.schema_id) AS HistoryTableSchema,
T2.name as HistoryTableName,T1.history_retention_period,
T1.history_retention_period_unit_desc
FROM sys.tables T1
OUTER APPLY (select is_temporal_history_retention_enabled from sys.databases
where name = DB_NAME()) AS DB
LEFT JOIN sys.tables T2
ON T1.history_table_id = T2.object_id WHERE T1.temporal_type = 2
Excellent data compression and efficient retention cleanup makes clustered columnstore index a perfect choice for
scenarios when your workload rapidly generates high amount of historical data. That pattern is typical for intensive
transactional processing workloads that use temporal tables for change tracking and auditing, trend analysis, or IoT
data ingestion.
Index considerations
The cleanup task for tables with rowstore clustered index requires index to start with the column corresponding the
end of SYSTEM_TIME period. If such index doesn't exist, you cannot configure a finite retention period:
Msg 13765, Level 16, State 1
Setting finite retention period failed on system -versioned temporal table
'temporalstagetestdb.dbo.WebsiteUserInfo' because the history table
'temporalstagetestdb.dbo.WebsiteUserInfoHistory' does not contain required clustered index. Consider creating a
clustered columnstore or B -tree index starting with the column that matches end of SYSTEM_TIME period, on the
history table.
It is important to notice that the default history table created by Azure SQL Database already has clustered index,
which is compliant for retention policy. If you try to remove that index on a table with finite retention period,
operation fails with the following error:
Msg 13766, Level 16, State 1
Cannot drop the clustered index 'WebsiteUserInfoHistory.IX_WebsiteUserInfoHistory' because it is being used for
automatic cleanup of aged data. Consider setting HISTORY_RETENTION_PERIOD to INFINITE on the
corresponding system -versioned temporal table if you need to drop this index.
Cleanup on the clustered columnstore index works optimally if historical rows are inserted in the ascending order
(ordered by the end of period column), which is always the case when the history table is populated exclusively by
the SYSTEM_VERSIONIOING mechanism. If rows in the history table are not ordered by end of period column
(which may be the case if you migrated existing historical data), you should re-create clustered columnstore index
on top of B -tree rowstore index that is properly ordered, to achieve optimal performance.
Avoid rebuilding clustered columnstore index on the history table with the finite retention period, because it may
change ordering in the row groups naturally imposed by the system-versioning operation. If you need to rebuild
clustered columnstore index on the history table, do that by re-creating it on top of compliant B -tree index,
preserving ordering in the rowgroups necessary for regular data cleanup. The same approach should be taken if
you create temporal table with existing history table that has clustered column index without guaranteed data
order:
/*Create B-tree ordered by the end of period column*/
CREATE CLUSTERED INDEX IX_WebsiteUserInfoHistory ON WebsiteUserInfoHistory (ValidTo)
WITH (DROP_EXISTING = ON);
GO
/*Re-create clustered columnstore index*/
CREATE CLUSTERED COLUMNSTORE INDEX IX_WebsiteUserInfoHistory ON WebsiteUserInfoHistory
WITH (DROP_EXISTING = ON);
When finite retention period is configured for the history table with the clustered columnstore index, you cannot
create additional non-clustered B -tree indexes on that table:
The query plan includes additional filter applied to end of period column (ValidTo) in the Clustered Index Scan
operator on the history table (highlighted). This example assumes that one MONTH retention period was set on
WebsiteUserInfo table.
However, if you query history table directly, you may see rows that are older than specified retention period, but
without any guarantee for repeatable query results. The following picture shows query execution plan for the
query on the history table without additional filters applied:
Do not rely your business logic on reading history table beyond retention period as you may get inconsistent or
unexpected results. We recommend that you use temporal queries with FOR SYSTEM_TIME clause for analyzing
data in temporal tables.
In-Memory OLTP can be used to improve the performance of transaction processing, data ingestion, and transient
data scenarios, in Premium and Business Critical tier databases without increasing the pricing tier.
NOTE
Learn how Quorum doubles key database’s workload while lowering DTU by 70% with SQL Database
Step 1: Ensure you are using a Premium and Business Critical tier
database
In-Memory OLTP is supported only in Premium and Business Critical tier databases. In-Memory is supported if
the returned result is 1 (not 0):
For the TRANSACTION_ISOLATION_LEVEL, SNAPSHOT is the most common value for the natively
compiled stored procedure. However, a subset of the other values are also supported:
REPEATABLE READ
SERIALIZABLE
The LANGUAGE value must be present in the sys.languages view.
How to migrate a stored procedure
The migration steps are:
1. Obtain the CREATE PROCEDURE script to the regular interpreted stored procedure.
2. Rewrite its header to match the previous template.
3. Ascertain whether the stored procedure T-SQL code uses any features that are not supported for natively
compiled stored procedures. Implement workarounds if necessary.
For details see Migration Issues for Natively Compiled Stored Procedures.
4. Rename the old stored procedure by using SP_RENAME. Or simply DROP it.
5. Run your edited CREATE PROCEDURE T-SQL script.
Related links
In-Memory OLTP (In-Memory Optimization)
Introduction to Natively Compiled Stored Procedures
Memory Optimization Advisor
SQL Server database migration to Azure SQL
Database
6/22/2018 • 6 minutes to read • Edit Online
In this article, you learn about the primary methods for migrating a SQL Server 2005 or later database to a
single or pooled database in Azure SQL Database. For information on migrating to a Managed Instance, see
Migrate to SQL Server instance to Azure SQL Database Managed Instance (preview ).
NOTE
To migrate a non-SQL Server database, including Microsoft Access, Sybase, MySQL Oracle, and DB2 to Azure SQL
Database, see SQL Server Migration Assistant.
NOTE
Rather than using DMA, you can also use a BACPAC file. See Import a BACPAC file to a new Azure SQL Database.
IMPORTANT
Use the latest version of SQL Server Management Studio to remain synchronized with updates to Microsoft Azure and SQL
Database. Older versions of SQL Server Management Studio cannot set up SQL Database as a subscriber. Update SQL
Server Management Studio.
1. Set up Distribution
Using SQL Server Management Studio (SSMS )
Using Transact-SQL
2. Create Publication
Using SQL Server Management Studio (SSMS )
Using Transact-SQL
3. Create Subscription
Using SQL Server Management Studio (SSMS )
Using Transact-SQL
Some tips and differences for migrating to SQL Database
Use a local distributor
Doing so causes a performance impact on the server.
If the performance impact is unacceptable, you can use another server but it adds complexity in
management and administration.
When selecting a snapshot folder, make sure the folder you select is large enough to hold a BCP of every
table you want to replicate.
Snapshot creation locks the associated tables until it is complete, so schedule your snapshot appropriately.
Only push subscriptions are supported in Azure SQL Database. You can only add subscribers from the source
database.
Resolving database migration compatibility issues
There are a wide variety of compatibility issues that you might encounter, depending both on the version of SQL
Server in the source database and the complexity of the database you are migrating. Older versions of SQL
Server have more compatibility issues. Use the following resources, in addition to a targeted Internet search
using your search engine of choices:
SQL Server database features not supported in Azure SQL Database
Discontinued Database Engine Functionality in SQL Server 2016
Discontinued Database Engine Functionality in SQL Server 2014
Discontinued Database Engine Functionality in SQL Server 2012
Discontinued Database Engine Functionality in SQL Server 2008 R2
Discontinued Database Engine Functionality in SQL Server 2005
In addition to searching the Internet and using these resources, use the MSDN SQL Server community forums
or StackOverflow.
IMPORTANT
SQL Database Managed Instance enables you to migrate an existing SQL Server instance and its databases with minimal to
no compatibility issues. See What is an Managed Instance.
Next steps
Use the script on the Azure SQL EMEA Engineers blog to Monitor tempdb usage during migration.
Use the script on the Azure SQL EMEA Engineers blog to Monitor the transaction log space of your database
while migration is occurring.
For a SQL Server Customer Advisory Team blog about migrating using BACPAC files, see Migrating from
SQL Server to Azure SQL Database using BACPAC Files.
For information about working with UTC time after migration, see Modifying the default time zone for your
local time zone.
For information about changing the default language of a database after migration, see How to change the
default language of Azure SQL Database.
SQL Server instance migration to Azure SQL
Database Managed Instance
9/6/2018 • 7 minutes to read • Edit Online
In this article, you learn about the methods for migrating a SQL Server 2005 or later version instance to Azure
SQL Database Managed Instance (preview ).
At a high level, the database migration process looks like:
NOTE
To migrate a single database into either a single database or elastic pool, see Migrate a SQL Server database to Azure SQL
Database.
To learn how to create the VNet infrastructure and a Managed Instance, see Create a Managed Instance.
IMPORTANT
It is important to keep your destination VNet and subnet always in accordance with Managed Instance VNET requirements.
Any incompatibility can prevent you from creating new instances or using those that you already created.
The following table provides more information regarding the methods you can use depending on source SQL
Server version you are running:
Put backup to Azure Storage Prior SQL 2012 SP1 CU2 Upload .bak file directly to Azure
storage
For a quickstart showing how to restore a database backup to a Managed Instance using a SAS credential, see
Restore from backup to a Managed Instance.
Monitor applications
Track application behavior and performance after migration. In Managed Instance, some changes are only
enabled once the database compatibility level has been changed. Database migration to Azure SQL Database
keeps its original compatibility level in majority of cases. If the compatibility level of a user database was 100 or
higher before the migration, it remains the same after migration. If the compatibility level of a user database was
90 before migration, in the upgraded database, the compatibility level is set to 100, which is the lowest supported
compatibility level in Managed Instance. Compatibility level of system databases is 140.
To reduce migration risks, change the database compatibility level only after performance monitoring. Use Query
Store as optimal tool for getting information about workload performance before and after database compatibility
level change, as explained in Keep performance stability during the upgrade to newer SQL Server version.
Once you are on a fully managed platform, take advantages that are provided automatically as part of the SQL
Database service. For instance, you don’t have to create backups on Managed Instance - the service performs
backups for you automatically. You no longer must worry about scheduling, taking, and managing backups.
Managed Instance provides you the ability to restore to any point in time within this retention period using Point
in Time Recovery (PITR ). During public preview, the retention period is fixed to seven days. Additionally, you do
not need to worry about setting up high availability as high availability is built in.
To strengthen security, consider using some of the features that are available:
Azure Active Directory Authentication at the database level
Use advanced security features such as Auditing, Threat Detection, Row -Level Security, and Dynamic Data
Masking ) to secure your instance.
Next steps
For information about Managed Instances, see What is a Managed Instance?.
For a tutorial that includes a restore from backup, see Create a Managed Instance.
For tutorial showing migration using DMS, see Migrate your on-premises database to Managed Instance
using DMS.
Migrate certificate of TDE protected database to
Azure SQL Database Managed Instance
9/12/2018 • 3 minutes to read • Edit Online
When migrating a database protected by Transparent Data Encryption to Azure SQL Database Managed Instance
using native restore option, the corresponding certificate from the on-premises or IaaS SQL Server needs to be
migrated before database restore. This article walks you through the process of manual migration of the certificate
to Azure SQL Database Managed Instance:
Export certificate to a Personal Information Exchange (.pfx) file
Extract certificate from file to base-64 string
Upload it using PowerShell cmdlet
For an alternative option using fully managed service for seamless migration of both TDE protected database and
corresponding certificate, see How to migrate your on-premises database to Managed Instance using Azure
Database Migration Service.
IMPORTANT
Transparent Data Encryption for Azure SQL Database Managed Instance works in service-managed mode. Migrated
certificate is used for restore of the TDE protected database only. Soon after restore is done, the migrated certificate gets
replaced by a different, system-managed certificate.
Prerequisites
To complete the steps in this article, you need the following prerequisites:
Pvk2Pfx command-line tool installed on the on-premises server or other computer with access to the certificate
exported as a file. Pvk2Pfx tool is part of the Enterprise Windows Driver Kit, a standalone self-contained
command-line environment.
Windows PowerShell version 5.0 or higher installed.
AzureRM PowerShell module installed and updated.
AzureRM.Sql module version 4.10.0 or higher. Run the following commands in PowerShell to install/update
the PowerShell module:
USE master
GO
SELECT db.name as [database_name], cer.name as [certificate_name]
FROM sys.dm_database_encryption_keys dek
LEFT JOIN sys.certificates cer
ON dek.encryptor_thumbprint = cer.thumbprint
INNER JOIN sys.databases db
ON dek.database_id = db.database_id
WHERE dek.encryption_state = 3
3. Execute the following script to export the certificate to a pair of files (.cer and .pvk), keeping the public and
private key information:
USE master
GO
BACKUP CERTIFICATE TDE_Cert
TO FILE = 'c:\full_path\TDE_Cert.cer'
WITH PRIVATE KEY (
FILE = 'c:\full_path\TDE_Cert.pvk',
ENCRYPTION BY PASSWORD = '<SomeStrongPassword>'
)
4. Use PowerShell console to copy certificate information from a pair of newly created files to a Personal
Information Exchange (.pfx) file, using Pvk2Pfx tool:
.\pvk2pfx -pvk c:/full_path/TDE_Cert.pvk -pi "<SomeStrongPassword>" -spc c:/full_path/TDE_Cert.cer -pfx
c:/full_path/TDE_Cert.pfx
certlm
2. In the Certificates MMC snap-in expand the path Personal -> Certificates to see the list of certificates
3. Right click certificate and click Export…
4. Follow the wizard to export certificate and private key to a Personal Information Exchange format
2. Once all preparation steps are done, run the following commands to upload base-64 encoded certificate to
the target Managed Instance:
The certificate is now available to the specified Managed Instance and backup of corresponding TDE protected
database can be restored successfully.
Next steps
In this article, you learned how to migrate certificate protecting encryption key of database with Transparent Data
Encryption, from the on-premises or IaaS SQL Server to Azure SQL Database Managed Instance.
See Restore a database backup to an Azure SQL Database Managed Instance to learn how to restore a database
backup to an Azure SQL Database Managed Instance.
New DBA in the cloud – Managing your database in
Azure SQL Database
9/12/2018 • 26 minutes to read • Edit Online
Moving from the traditional self-managed, self-controlled environment to a PaaS environment can seem a bit
overwhelming at first. As an app developer or a DBA, you would want to know the core capabilities of the platform
that would help you keep your application available, performant, secure and resilient - always. This article aims to
do exactly that. The article succinctly organizes resources and gives you some guidance on how to best use the key
capabilities of SQL Database to manage and keep your application running efficiently and achieve optimal results
in the cloud. Typical audience for this article would be those who:
Are evaluating migration of their application(s) to Azure SQL DB – Modernizing your application(s).
Are In the process of migrating their application(s) – On-going migration scenario.
Have recently completed the migration to Azure SQL DB – New DBA in the cloud.
This article discusses some of the core characteristics of Azure SQL DB as a platform that you can readily leverage.
They are the following: -
Business continuity and disaster recovery (BCDR )
Security and compliance
Intelligent database monitoring and maintenance
Data movement
Basic 7
Standard 35
Premium 35
In addition, the Long-Term Retention (LTR ) feature allows you to hold onto your backup files for a much longer
period specifically, for up to 10 years, and restore data from these backups at any point within that period.
Furthermore, the database backups are kept on a geo-replicated storage to ensure resilience from regional
catastrophe. You can also restore these backups in any Azure region at any point of time within the retention
period. See Business continuity overview.
How do I ensure business continuity in the event of a datacenter-level disaster or regional catastrophe?
Because your database backups are stored on a geo-replicated storage subsystem to ensure that in case of a
regional disaster, you can restore the backup to another Azure region. This is called geo-restore. The RPO
(Recovery Point Objective) for this is generally < 1 hour and the ERT (Estimated Recovery Time – few minutes to
hours).
For mission-critical databases, Azure SQL DB offers, active geo-replication. What this essentially does is that it
creates a geo-replicated secondary copy of your original database in another region. For example, if your database
is initially hosted in Azure West US region and you want regional disaster resilience. You’d create an active geo
replica of the database in West US to say East US. When the calamity strikes on West US, you can failover to the
East US region. Configuring them in an Auto-Failover Group is even better because this ensures that the database
automatically fails over to the secondary in East US in case of a disaster. The RPO for this is < 5 seconds and the
ERT < 30 seconds.
If an auto-failover group is not configured, then your application needs to actively monitor for a disaster and
initiate a failover to the secondary. You can create up to 4 such active geo-replicas in different Azure regions. It gets
even better. You can also access these secondary active geo-replicas for read-only access. This comes in very handy
to reduce latency for a geo-distributed application scenario.
How does my disaster recovery plan change from on-premises to SQL Database?
In summary, the traditional on-premises SQL Server setup required you to actively manage your Availability by
using features such as Failover Clustering, Database Mirroring, Transaction Replication, Log Shipping etc. and
maintain and manage backups to ensure Business Continuity. With SQL Database, the platform manages these for
you, so you can focus on developing and optimizing your database application and not worry about disaster
management as much. You can have backup and disaster recovery plans configured and working with just a few
clicks on the Azure portal (or a few commands using the PowerShell APIs).
To learn more about Disaster recovery, see: Azure SQL Db Disaster Recovery 101
Prefer not to use Azure Active Directory (AD) in Azure Use SQL authentication
Used AD on SQL Server on-premises Federate AD with Azure AD, and use Azure AD authentication.
With this, you can use Single Sign-On.
Need to enforce multi-factor authentication (MFA) Require MFA as a policy through Microsoft Conditional Access,
and use Azure AD Universal authentication with MFA support.
Have guest accounts from Microsoft accounts (live.com, Use Azure AD Universal authentication in SQL Database/Data
outlook.com) or other domains (gmail.com) Warehouse, which leverages Azure AD B2B Collaboration.
Are logged in to Windows using your Azure AD credentials Use Azure AD integrated authentication.
from a federated domain
Are logged in to Windows using credentials from a domain Use Azure AD integrated authentication.
not federated with Azure
Have middle-tier services which need to connect to SQL Use Azure AD integrated authentication.
Database or SQL Data Warehouse
Reserved IPs
Another option is to provision reserved IPs for your VMs, and whitelist those specific VM IP addresses in the
server firewall settings. By assigning reserved IPs, you save the trouble of having to update the firewall rules with
changing IP addresses.
What port do I connect to SQL Database on?
Port 1433. SQL Database communicates over this port. To connect from within a corporate network, you have to
add an outbound rule in the firewall settings of your organization. As a guideline, avoid exposing port 1433 outside
the Azure boundary. You can run SSMS in Azure using Azure RemoteApp. This does not require you to open
outgoing connections to port 1433, the IP is static, so the DB can be open to only the RemoteApp and it supports
Multi Factor Authentication (MFA).
How can I monitor and regulate activity on my server and database in SQL Database?
SQL Database Auditing
With SQL Database, you can turn ON Auditing to track database events. SQL Database Auditing records database
events and writes them into an audit log file in your Azure Storage Account. Auditing is especially useful if you
intend to gain insight into potential security and policy violations, maintain regulatory compliance etc. It allows you
to define and configure certain categories of events that you think need auditing and based on that you can get
preconfigured reports and a dashboard to get an overview of events occurring on your database. You can apply
these auditing policies either at the database level or at the server level. A guide on how to turn on auditing for
your server/database, see: Enable SQL Database Auditing.
Threat Detection
With threat detection, you get the ability to act upon security or policy violations discovered by Auditing very easily.
You don’t need to be a security expert to address potential threats or violations in your system. Threat detection
also has some built-in capabilities like SQL Injection detection. SQL Injection is an attempt to alter or compromise
the data and a quite common way of attacking a database application in general. SQL Database Threat Detection
runs multiple sets of algorithms which detect potential vulnerabilities and SQL injection attacks, as well as
anomalous database access patterns (such as access from an unusual location or by an unfamiliar principal).
Security officers or other designated administrators receive an email notification if a threat is detected on the
database. Each notification provides details of the suspicious activity and recommendations on how to further
investigate and mitigate the threat. To learn how to turn on Threat detection, see: Enable SQL Database Threat
Detection.
How do I protect my data in general on SQL Database?
Encryption provides a strong mechanism to protect and secure your sensitive data from intruders. Your encrypted
data is of no use to the intruder without the decryption key. Thus, it adds an extra layer of protection on top of the
existing layers of security built in SQL Database. There are two aspects to protecting your data in SQL Database:
Your data that is at-rest in the data and log files
Your data that is in-flight.
In SQL Database, by default, your data at rest in the data and log files on the storage subsystem is completely and
always encrypted via Transparent Data Encryption [TDE ]. Your backups are also encrypted. With TDE there are no
changes required on your application side that is accessing this data. The encryption and decryption happen
transparently; hence the name. For protecting your sensitive data in-flight and at rest, SQL Database provides a
feature called Always Encrypted (AE ). AE is a form of client-side encryption which encrypts sensitive columns in
your database (so they are in ciphertext to database administrators and unauthorized users). The server receives
the encrypted data to begin with. The key for Always Encrypted is also stored on the client side, so only authorized
clients can decrypt the sensitive columns. The server and data administrators cannot see the sensitive data since
the encryption keys are stored on the client. AE encrypts sensitive columns in the table end to end, from
unauthorized clients to the physical disk. AE supports equality comparisons today, so DBAs can continue to query
encrypted columns as part of their SQL commands. Always Encrypted can be used with a variety of key store
options, such as Azure Key Vault, Windows certificate store, and local hardware security modules.
Database server can access sensitive No Yes, since encryption is for the data at
data rest
Allowed T-SQL operations Equality comparison All T-SQL surface area is available
You can also view this analysis under the “Advisor” section:
How do I monitor the performance and resource utilization in SQL Database?
In SQL Database you can leverage the intelligent insights of the platform to monitor the performance and tune
accordingly. You can monitor performance and resource utilization in SQL Database using the following methods:
Azure portal: The Azure portal shows a single database’s utilization by selecting the database and clicking
the chart in the Overview pane. You can modify the chart to show multiple metrics, including CPU
percentage, DTU percentage, Data IO percentage, Sessions percentage, and Database size percentage.
From this chart, you can also configure alerts by resource. These alerts allow you to respond to resource conditions
with an email, write to an HTTPS/HTTP endpoint or perform an action. See the Monitoring database performance
in SQL Database for detailed instructions.
Dynamic Management Views: You can query the sys.dm_db_resource_stats dynamic management view to
return resource consumption statistics history from the last hour and the sys.resource_stats system catalog view
to return history for the last 14 days.
Query Performance Insight: Query Performance Insight allows you to see a history of the top resource-
consuming queries and long-running queries for a specific database. You can quickly identify TOP queries
by resource utilization, duration, and frequency of execution. You can track queries and detect regression.
This feature requires Query Store to be enabled and active for the database.
Azure SQL Analytics (Preview) in Log Analytics: Azure Log Analytics allows you to collect and visualize
key Azure SQL Azure performance metrics, supporting up to 150,000 SQL Databases and 5,000 SQL
Elastic pools per workspace. You can use it to monitor and receive notifications. You can monitor SQL
Database and elastic pool metrics across multiple Azure subscriptions and elastic pools and can be used to
identify issues at each layer of an application stack.
I am noticing performance issues: How does my SQL Database troubleshooting methodology differ from SQL
Server?
A major portion of the troubleshooting techniques you would use for diagnosing query and database performance
issues remain the same. After all the same SQL Server engine powers the cloud. However, the platform - Azure
SQL DB has built in ‘intelligence’. It can help you troubleshoot and diagnose performance issues even more easily.
It can also perform some of these corrective actions on your behalf and in some cases, proactively fix them -
automatically.
Your approach towards troubleshooting performance issues can significantly benefit by using intelligent features
such as Query Performance Insight(QPI) and Database Advisor in conjunction and so the difference in
methodology differs in that respect – you no longer need to do the manual work of grinding out the essential
details that might help you troubleshoot the issue at hand. The platform does the hard work for you. One example
of that is QPI. With QPI, you can drill all the way down to the query level and look at the historical trends and
figure out when exactly the query regressed. The Database Advisor gives you recommendations on things that
might help you improve your overall performance in general like - missing indexes, dropping indexes,
parameterizing your queries etc.
With performance troubleshooting, it is important to identify whether it is just the application or the database
backing it, that’s impacting your application performance. Often the performance problem lies in the application
layer. It could be the architecture or the data access pattern. For example, consider you have a chatty application
that is sensitive to network latency. In this case, your application suffers because there would be many short
requests going back and forth ("chatty") between the application and the server and on a congested network, these
roundtrips add up fast. To improve the performance in this case, you can use Batch Queries. Using batches helps
you tremendously because now your requests get processed in a batch; thus, helping you cut down on the
roundtrip latency and improve your application performance.
Additionally, if you notice a degradation in the overall performance of your database, you can monitor the
sys.dm_db_resource_stats and sys.resource_stats dynamic management views in order to understand CPU, IO, and
memory consumption. Your performance maybe impacted because your database is starved of resources. It could
be that you may need to change the performance level and/or service tier based on the growing and shrinking
workload demands.
For a comprehensive set of recommendations for tuning performance issues, see: Tune your database.
How do I ensure I am using the appropriate service tier and performance level?
SQL Database offers various service tiers Basic, Standard, and Premium. Each service tier you get a guaranteed
predictable performance tied to that service level. Depending on your workload, you may have bursts of activity
where your resource utilization might hit the ceiling of the current performance level that you are in. In such cases,
it is useful to first start by evaluating whether any tuning can help (for example, adding or altering an index etc.). If
you still encounter limit issues, consider moving to a higher performance level or service level.
For making sure you’re on the right performance level, you can monitor your query and database resource
consumption through one of the above-mentioned ways in “How do I monitor the performance and resource
utilization in SQL Database”. Should you find that your queries/databases are consistently running hot on
CPU/Memory etc. you can consider scaling up to a higher performance level. Similarly, if you note that even during
your peak hours, you don’t seem to use the resources as much; consider scaling down from the current
performance level.
If you have a SaaS app pattern or a database consolidation scenario, consider using an Elastic pool for cost
optimization. Elastic pool is a great way to achieve database consolidation and cost-optimization. To read more
about managing multiple databases using Elastic Pool, see: Manage pools and databases.
How often do I need to run database integrity checks for my database?
SQL Database uses some smart techniques that allow it to handle certain classes of data corruption automatically
and without any data loss. These techniques are built in to the service and are leveraged by the service when need
arises. On a regular basis, your database backups across the service are tested by restoring them and running
DBCC CHECKDB on it. If there are issues, SQL Database proactively addresses them. Automatic page repair is
leveraged for fixing pages that are corrupt or have data integrity issues. The database pages are always verified
with the default CHECKSUM setting that verifies the integrity of the page. SQL Database proactively monitors and
reviews the data integrity of your database and, if issues arise, addresses them with the highest priority. In addition
to these, you may choose to optionally run your own integrity checks at your will. For more information, see Data
Integrity in SQL Database
Data movement after migration
How do I export and import data as BACPAC files from SQL Database?
Export: You can export your Azure SQL database as a BACPAC file from the Azure portal
Import: You can also import data as a BACPAC file into the database using the Azure portal.
Next steps
Learn about SQL Database.
Copy an Azure SQL database
6/14/2018 • 5 minutes to read • Edit Online
Azure SQL Database provides several methods for creating a transactionally consistent copy of an existing Azure
SQL database on either the same server or a different server. You can copy a SQL database by using the Azure
portal, PowerShell, or T-SQL.
Overview
A database copy is a snapshot of the source database as of the time of the copy request. You can select the same
server or a different server, its service tier and performance level, or a different performance level within the same
service tier (edition). After the copy is complete, it becomes a fully functional, independent database. At this point,
you can upgrade or downgrade it to any edition. The logins, users, and permissions can be managed
independently.
NOTE
If you decide to cancel the copying while it is in progress, execute the DROP DATABASE statement on the new database.
Alternatively, executing the DROP DATABASE statement on the source database also cancels the copying process.
Resolve logins
After the new database is online on the destination server, use the ALTER USER statement to remap the users
from the new database to logins on the destination server. To resolve orphaned users, see Troubleshoot Orphaned
Users. See also How to manage Azure SQL database security after disaster recovery.
All users in the new database retain the permissions that they had in the source database. The user who initiated
the database copy becomes the database owner of the new database and is assigned a new security identifier
(SID ). After the copying succeeds and before other users are remapped, only the login that initiated the copying,
the database owner, can log in to the new database.
To learn about managing users and logins when you copy a database to a different logical server, see How to
manage Azure SQL database security after disaster recovery.
Next steps
For information about logins, see Manage logins and How to manage Azure SQL database security after
disaster recovery.
To export a database, see Export the database to a BACPAC.
Import a BACPAC file to a new Azure SQL Database
9/14/2018 • 4 minutes to read • Edit Online
When you need to import a database from an archive or when migrating from another platform, you can import
the database schema and data from a BACPAC file. A BACPAC file is a ZIP file with an extension of BACPAC
containing the metadata and data from a SQL Server database. A BACPAC file can be imported from Azure blob
storage (standard storage only) or from local storage in an on-premises location. To maximize the import speed,
we recommend that you specify a higher service tier and performance level, such as a P6, and then scale to down
as appropriate after the import is successful. Also, the database compatibility level after the import is based on the
compatibility level of the source database.
IMPORTANT
After you migrate your database to Azure SQL Database, you can choose to operate the database at its current
compatibility level (level 100 for the AdventureWorks2008R2 database) or at a higher level. For more information on the
implications and options for operating a database at a specific compatibility level, see ALTER DATABASE Compatibility Level.
See also ALTER DATABASE SCOPED CONFIGURATION for information about additional database-level settings related to
compatibility levels. >
To monitor the progress of the import operation, open the page for the logical server containing the database
being imported. Scroll down to Operations and then click Import/Export history.
NOTE
Azure SQL Database Managed Instance supported importing from a BACPAC file using the other methods in this article,
but does not currently support migrating using the Azure portal.
IMPORTANT
An Azure SQL Database logical server listens on port 1433. If you are attempting to connect to an Azure SQL Database
logical server from within a corporate firewall, this port must be open in the corporate firewall for you to successfully
connect.
This example shows how to import a database using SqlPackage.exe with Active Directory Universal
Authentication:
SqlPackage.exe /a:Import /sf:testExport.bacpac /tdn:NewDacFX /tsn:apptestserver.database.windows.net /ua:True
/tid:"apptest.onmicrosoft.com"
To check the status of the import request, use the Get-AzureRmSqlDatabaseImportExportStatus cmdlet. Running
this immediately after the request usually returns Status: InProgress. When you see Status: Succeeded the
import is complete.
TIP
For another script example, see Import a database from a BACPAC file.
Limitations
Import to a database in elastic pool is not supported. You can import data into a singleton database and then
move the database to a pool.
Next steps
To learn how to connect to and query an imported SQL Database, see Connect to SQL Database with SQL
Server Management Studio and perform a sample T-SQL query.
For a SQL Server Customer Advisory Team blog about migrating using BACPAC files, see Migrating from
SQL Server to Azure SQL Database using BACPAC Files.
For a discussion of the entire SQL Server database migration process, including performance
recommendations, see Migrate a SQL Server database to Azure SQL Database.
To learn how to manage and share storage keys and shared access signitures securely, see Azure Storage
Security Guide.
Export an Azure SQL database to a BACPAC file
5/23/2018 • 4 minutes to read • Edit Online
When you need to export a database for archiving or for moving to another platform, you can export the database
schema and data to a BACPAC file. A BACPAC file is a ZIP file with an extension of BACPAC containing the
metadata and data from a SQL Server database. A BACPAC file can be stored in Azure blob storage or in local
storage in an on-premises location and later imported back into Azure SQL Database or into a SQL Server on-
premises installation.
IMPORTANT
Azure SQL Database Automated Export was retired on March 1, 2017. You can use long-term backup retention or Azure
Automation to periodically archive SQL databases using PowerShell according to a schedule of your choice. For a sample,
download the sample PowerShell script from Github.
NOTE
BACPACs are not intended to be used for backup and restore operations. Azure SQL Database automatically creates
backups for every user database. For details, see Business Continuity Overview and SQL Database backups.
To check the status of the export request, use the Get-AzureRmSqlDatabaseImportExportStatus cmdlet. Running
this immediately after the request usually returns Status: InProgress. When you see Status: Succeeded the
export is complete.
Next steps
To learn about long-term backup retention of an Azure SQL database backup as an alternative to exported a
database for archive purposes, see Long-term backup retention.
For a SQL Server Customer Advisory Team blog about migrating using BACPAC files, see Migrating from SQL
Server to Azure SQL Database using BACPAC Files.
To learn about importing a BACPAC to a SQL Server database, see Import a BACPCAC to a SQL Server
database.
To learn about exporting a BACPAC from a SQL Server database, see Export a Data-tier Application and
Migrate your first database.
If you are exporting from SQL Server as a prelude to migration to Azure SQL Database, see Migrate a SQL
Server database to Azure SQL Database.
To learn how to manage and share storage keys and shared access signitures securely, see Azure Storage
Security Guide.
Sync data across multiple cloud and on-premises
databases with SQL Data Sync
9/12/2018 • 9 minutes to read • Edit Online
SQL Data Sync is a service built on Azure SQL Database that lets you synchronize the data you select bi-
directionally across multiple SQL databases and SQL Server instances.
NOTE
If you're using an on premises database as a member database, you have to install and configure a local sync agent.
ETL (OLTP to OLAP) Azure Data Factory or SQL Server Integration Services
Migration from on-premises SQL Server to Azure SQL Azure Database Migration Service
Database
Next steps
Update the schema of a synced database
Do you have to update the schema of a database in a sync group? Schema changes are not automatically
replicated. For some solutions, see the following articles:
Automate the replication of schema changes in Azure SQL Data Sync
Use PowerShell to update the sync schema in an existing sync group
Monitor and troubleshoot
Is SQL Data Sync performing as expected? To monitor activity and troubleshoot issues, see the following articles:
Monitor Azure SQL Data Sync with Log Analytics
Troubleshoot issues with Azure SQL Data Sync
Learn more about Azure SQL Database
For more info about SQL Database, see the following articles:
SQL Database Overview
Database Lifecycle Management
Automate the replication of schema changes in Azure
SQL Data Sync
9/12/2018 • 7 minutes to read • Edit Online
SQL Data Sync lets users synchronize data between Azure SQL databases and on-premises SQL Server in one
direction or in both directions. One of the current limitations of SQL Data Sync is a lack of support for the
replication of schema changes. Every time you change the table schema, you need to apply the changes manually
on all endpoints, including the hub and all members, and then update the sync schema.
This article introduces a solution to automatically replicate schema changes to all SQL Data Sync endpoints.
1. This solution uses a DDL trigger to track schema changes.
2. The trigger inserts the schema change commands in a tracking table.
3. This tracking table is synced to all endpoints using the Data Sync service.
4. DML triggers after insertion are used to apply the schema changes on the other endpoints.
This article uses ALTER TABLE as an example of a schema change, but this solution also works for other types of
schema changes.
IMPORTANT
We recommend that you read this article carefully, especially the sections about Troubleshooting and Other considerations,
before you start to implement automated schema change replication in your sync environment. We also recommend that
you read Sync data across multiple cloud and on-premises databases with SQL Data Sync. Some database operations may
break the solution described in this article. Additional domain knowledge of SQL Server and Transact-SQL may be required to
troubleshoot those issues.
This table has an identity column to track the order of schema changes. You can add more fields to log more
information if needed.
Create a table to track the history of schema changes
On all endpoints, create a table to track the ID of the most recently applied schema change command.
Create an ALTER TABLE DDL trigger in the database where schema changes are made
Create a DDL trigger for ALTER TABLE operations. You only need to create this trigger in the database where
schema changes are made. To avoid conflicts, only allow schema changes in one database in a sync group.
-- You can add your own logic to filter ALTER TABLE commands instead of replicating all of them.
The trigger inserts a record in the schema change tracking table for each ALTER TABLE command. This example
adds a filter to avoid replicating schema changes made under schema DataSync, because these are most likely
made by the Data Sync service. Add more filters if you only want to replicate certain types of schema changes.
You can also add more triggers to replicate other types of schema changes. For example, create
CREATE_PROCEDURE, ALTER_PROCEDURE and DROP_PROCEDURE triggers to replicate changes to stored
procedures.
Create a trigger on other endpoints to apply schema changes during insertion
This trigger executes the schema change command when it is synced to other endpoints. You need to create this
trigger on all the endpoints, except the one where schema changes are made (that is, in the database where the
DDL trigger AlterTableDDLTrigger is created in the previous step).
CREATE TRIGGER SchemaChangesTrigger
ON SchemaChanges
AFTER INSERT
AS
DECLARE \@lastAppliedId bigint
DECLARE \@id bigint
DECLARE \@sqlStmt nvarchar(max)
SELECT TOP 1 \@lastAppliedId=LastAppliedId FROM SchemaChangeHistory
SELECT TOP 1 \@id = id, \@SqlStmt = SqlStmt FROM SchemaChanges WHERE id \> \@lastAppliedId ORDER BY id
IF (\@id = \@lastAppliedId + 1)
BEGIN
EXEC sp_executesql \@SqlStmt
UPDATE SchemaChangeHistory SET LastAppliedId = \@id
WHILE (1 = 1)
BEGIN
SET \@id = \@id + 1
IF exists (SELECT id FROM SchemaChanges WHERE ID = \@id)
BEGIN
SELECT \@sqlStmt = SqlStmt FROM SchemaChanges WHERE ID = \@id
EXEC sp_executesql \@SqlStmt
UPDATE SchemaChangeHistory SET LastAppliedId = \@id
END
ELSE
BREAK;
END
END
This trigger runs after the insertion and checks whether the current command should run next. The code logic
ensures that no schema change statement is skipped, and all changes are applied even if the insertion is out of
order.
Sync the schema change tracking table to all endpoints
You can sync the schema change tracking table to all endpoints using the existing sync group or a new sync group.
Make sure the changes in the tracking table can be synced to all endpoints, especially when you're using one-
direction sync.
Don't sync the schema change history table, since that table maintains different state on different endpoints.
Apply the schema changes in a sync group
Only schema changes made in the database where the DDL trigger is created are replicated. Schema changes
made in other databases are not replicated.
After the schema changes are replicated to all endpoints, you also need to take extra steps to update the sync
schema to start or stop syncing the new columns.
Add new columns
1. Make the schema change.
2. Avoid any data change where the new columns are involved until you've completed the step that creates the
trigger.
3. Wait until the schema changes are applied to all endpoints.
4. Refresh the database schema and add the new column to the sync schema.
5. Data in the new column is synced during next sync operation.
Remove columns
1. Remove the columns from the sync schema. Data Sync stops syncing data in these columns.
2. Make the schema change.
3. Refresh the database schema.
Update data types
1. Make the schema change.
2. Wait until the schema changes are applied to all endpoints.
3. Refresh the database schema.
4. If the new and old data types are not fully compatible - for example, if you change from int to bigint -
sync may fail before the steps that create the triggers are completed. Sync succeeds after a retry.
Rename columns or tables
Renaming columns or tables makes Data Sync stop working. Create a new table or column, backfill the data, and
then delete the old table or column instead of renaming.
Other types of schema changes
For other types of schema changes - for example, creating stored procedures or dropping an index- updating the
sync schema is not required.
Other Considerations
Database users who configure the hub and member databases need to have enough permission to execute
the schema change commands.
You can add more filters in the DDL trigger to only replicate schema change in selected tables or operations.
You can only make schema changes in the database where the DDL trigger is created.
If you are making a change in an on-premises SQL Server database, make sure the schema change is
supported in Azure SQL Database.
If schema changes are made in databases other than the database where the DDL trigger is created, the
changes are not replicated. To avoid this issue, you can create DDL triggers to block changes on other
endpoints.
If you need to change the schema of the schema change tracking table, disable the DDL trigger before you
make the change, and then manually apply the change to all endpoints. Updating the schema in an AFTER
INSERT trigger on the same table does not work.
Don't reseed the identity column by using DBCC CHECKIDENT.
Don't use TRUNCATE to clean up data in the schema change tracking table.
Monitor SQL Data Sync with Log Analytics
8/7/2018 • 6 minutes to read • Edit Online
To check the SQL Data Sync activity log and detect errors and warnings, you previously had to check SQL Data
Sync manually in the Azure portal, or use PowerShell or the REST API. Follow the steps in this article to configure
a custom solution that improves the Data Sync monitoring experience. You can customize this solution to fit your
scenario.
For an overview of SQL Data Sync, see Sync data across multiple cloud and on-premises databases with Azure
SQL Data Sync.
3. After running the query, select the bell that says Alert.
4. Under Generate alert based on, select Metric Measurement.
a. Set the Aggregate Value to Greater than.
b. After Greater than, enter the threshold to elapse before you receive notifications. Transient errors
are expected in Data Sync. To reduce noise, set the threshold to 5.
5. Under Actions, set Email notification to "Yes." Enter the desired email recipients.
6. Click Save. The specified recipients now receive email notifications when errors occur.
Code samples
Download the code samples described in this article from the following locations:
Data Sync Log PowerShell Runbook
Data Sync Log Analytics View
Next steps
For more info about SQL Data Sync, see:
Sync data across multiple cloud and on-premises databases with Azure SQL Data Sync
Set up Azure SQL Data Sync
Best practices for Azure SQL Data Sync
Troubleshoot issues with Azure SQL Data Sync
Troubleshoot issues with Azure SQL Data Sync
Complete PowerShell examples that show how to configure SQL Data Sync:
Use PowerShell to sync between multiple Azure SQL databases
Use PowerShell to sync between an Azure SQL Database and a SQL Server on-premises database
For more info about SQL Database, see:
SQL Database Overview
Database Lifecycle Management
Best practices for SQL Data Sync
8/20/2018 • 8 minutes to read • Edit Online
This article describes best practices for Azure SQL Data Sync.
For an overview of SQL Data Sync, see Sync data across multiple cloud and on-premises databases with Azure
SQL Data Sync.
Setup
Database considerations and constraints
SQL Database instance size
When you create a new SQL Database instance, set the maximum size so that it's always larger than the database
you deploy. If you don't set the maximum size to larger than the deployed database, sync fails. Although SQL Data
Sync doesn't offer automatic growth, you can run the ALTER DATABASE command to increase the size of the
database after it has been created. Ensure that you stay within the SQL Database instance size limits.
IMPORTANT
SQL Data Sync stores additional metadata with each database. Ensure that you account for this metadata when you
calculate space needed. The amount of added overhead is related to the width of the tables (for example, narrow tables
require more overhead) and the amount of traffic.
Table considerations and constraints
Selecting tables
You don't have to include all the tables that are in a database in a sync group. The tables that you include in a sync
group affect efficiency and costs. Include tables, and the tables they are dependent on, in a sync group only if
business needs require it.
Primary keys
Each table in a sync group must have a primary key. The SQL Data Sync service can't sync a table that doesn't
have a primary key.
Before using SQL Data Sync in production, test initial and ongoing sync performance.
Provisioning destination databases
SQL Data Sync provides basic database autoprovisioning.
This section discusses the limitations of provisioning in SQL Data Sync.
Autoprovisioning limitations
SQL Data Sync has the following limitations for autoprovisioning:
Select only the columns that are created in the destination table. Any columns that aren't part of the sync group
aren't provisioned in the destination tables.
Indexes are created only for selected columns. If the source table index has columns that aren't part of the sync
group, those indexes aren't provisioned in the destination tables.
Indexes on XML type columns aren't provisioned.
CHECK constraints aren't provisioned.
Existing triggers on the source tables aren't provisioned.
Views and stored procedures aren't created on the destination database.
ON UPDATE CASCADE and ON DELETE CASCADE actions on foreign key constraints aren't recreated in the
destination tables.
If you have decimal or numeric columns with a precision greater than 28, SQL Data Sync may encounter a
conversion overflow issue during sync. We recommend that you limit the precision of decimal or numeric
columns to 28 or less.
Recommendations
Use the SQL Data Sync autoprovisioning capability only when you are trying out the service.
For production, provision the database schema.
Where to locate the hub database
Enterprise-to-cloud scenario
To minimize latency, keep the hub database close to the greatest concentration of the sync group's database traffic.
Cloud-to-cloud scenario
When all the databases in a sync group are in one datacenter, the hub should be located in the same datacenter.
This configuration reduces latency and the cost of data transfer between datacenters.
When the databases in a sync group are in multiple datacenters, the hub should be located in the same
datacenter as the majority of the databases and database traffic.
Mixed scenarios
Apply the preceding guidelines to complex sync group configurations, such as those that are a mix of enterprise-
to-cloud and cloud-to-cloud scenarios.
Sync
Avoid slow and costly initial sync
In this section, we discuss the initial sync of a sync group. Learn how to help prevent an initial sync from taking
longer and being more costly than necessary.
How initial sync works
When you create a sync group, start with data in only one database. If you have data in multiple databases, SQL
Data Sync treats each row as a conflict that needs to be resolved. This conflict resolution causes the initial sync to
go slowly. If you have data in multiple databases, initial sync might take between several days and several months,
depending on the database size.
If the databases are in different datacenters, each row must travel between the different datacenters. This increases
the cost of an initial sync.
Recommendation
If possible, start with data in only one of the sync group's databases.
Design to avoid sync loops
A sync loop occurs when there are circular references within a sync group. In that scenario, each change in one
database is endlessly and circularly replicated through the databases in the sync group.
Ensure that you avoid sync loops, because they cause performance degradation and might significantly increase
costs.
Changes that fail to propagate
Reasons that changes fail to propagate
Changes might fail to propagate for one of the following reasons:
Schema/datatype incompatibility.
Inserting null in non-nullable columns.
Violating foreign key constraints.
What happens when changes fail to propagate?
Sync group shows that it's in a Warning state.
Details are listed in the portal UI log viewer.
If the issue is not resolved for 45 days, the database becomes out of date.
NOTE
These changes never propagate. The only way to recover in this scenario is to re-create the sync group.
Recommendation
Monitor the sync group and database health regularly through the portal and log interface.
Maintenance
Avoid out-of-date databases and sync groups
A sync group or a database in a sync group can become out of date. When a sync group's status is Out-of-date, it
stops functioning. When a database's status is Out-of-date, data might be lost. It's best to avoid this scenario
instead of trying to recover from it.
Avoid out-of-date databases
A database's status is set to Out-of-date when it has been offline for 45 days or more. To avoid an Out-of-date
status on a database, ensure that none of the databases are offline for 45 days or more.
Avoid out-of-date sync groups
A sync group's status is set to Out-of-date when any change in the sync group fails to propagate to the rest of the
sync group for 45 days or more. To avoid an Out-of-date status on a sync group, regularly check the sync group's
history log. Ensure that all conflicts are resolved, and that changes are successfully propagated throughout the
sync group databases.
A sync group might fail to apply a change for one of these reasons:
Schema incompatibility between tables.
Data incompatibility between tables.
Inserting a row with a null value in a column that doesn't allow null values.
Updating a row with a value that violates a foreign key constraint.
To prevent out-of-date sync groups:
Update the schema to allow the values that are contained in the failed rows.
Update the foreign key values to include the values that are contained in the failed rows.
Update the data values in the failed row so they are compatible with the schema or foreign keys in the target
database.
Avoid deprovisioning issues
In some circumstances, unregistering a database with a client agent might cause sync to fail.
Scenario
1. Sync group A was created by using a SQL Database instance and an on-premises SQL Server database, which
is associated with local agent 1.
2. The same on-premises database is registered with local agent 2 (this agent is not associated with any sync
group).
3. Unregistering the on-premises database from local agent 2 removes the tracking and meta tables for sync
group A for the on-premises database.
4. Sync group A operations fail, with this error: "The current operation could not be completed because the
database is not provisioned for sync or you do not have permissions to the sync configuration tables."
Solution
To avoid this scenario, don't register a database with more than one agent.
To recover from this scenario:
1. Remove the database from each sync group that it belongs to.
2. Add the database back into each sync group that you removed it from.
3. Deploy each affected sync group (this action provisions the database).
Modifying a sync group
Don't attempt to remove a database from a sync group and then edit the sync group without first deploying one
of the changes.
Instead, first remove a database from a sync group. Then, deploy the change and wait for deprovisioning to finish.
When deprovisioning is finished, you can edit the sync group and deploy the changes.
If you attempt to remove a database and then edit a sync group without first deploying one of the changes, one or
the other operation fails. The portal interface might become inconsistent. If this happens, refresh the page to
restore the correct state.
Next steps
For more information about SQL Data Sync, see:
Sync data across multiple cloud and on-premises databases with Azure SQL Data Sync
Set up Azure SQL Data Sync
Monitor Azure SQL Data Sync with Log Analytics
Troubleshoot issues with Azure SQL Data Sync
Complete PowerShell examples that show how to configure SQL Data Sync:
Use PowerShell to sync between multiple Azure SQL databases
Use PowerShell to sync between an Azure SQL Database and a SQL Server on-premises database
For more information about SQL Database, see:
SQL Database overview
Database lifecycle management
Troubleshoot issues with SQL Data Sync
8/7/2018 • 14 minutes to read • Edit Online
This article describes how to troubleshoot known issues with Azure SQL Data Sync. If there is a resolution for an
issue, it's provided here.
For an overview of SQL Data Sync, see Sync data across multiple cloud and on-premises databases with Azure
SQL Data Sync.
Sync issues
Sync fails in the portal UI for on-premises databases that are associated with the client agent
My sync group is stuck in the processing state
I see erroneous data in my tables
I see inconsistent primary key data after a successful sync
I see a significant degradation in performance
I see this message: "Cannot insert the value NULL into the column . Column does not allow nulls." What
does this mean, and how can I fix it?
How does Data Sync handle circular references? That is, when the same data is synced in multiple sync
groups, and keeps changing as a result?
Sync fails in the portal UI for on-premises databases that are associated with the client agent
Sync fails in the SQL Data Sync portal UI for on-premises databases that are associated with the client agent. On
the local computer that's running the agent, you see System.IO.IOException errors in the Event Log. The errors
say that the disk has insufficient space.
Cause. The drive has insufficient space.
Resolution. Create more space on the drive on which the %TEMP% directory is located.
My sync group is stuck in the processing state
A sync group in SQL Data Sync has been in the processing state for a long time. It doesn't respond to the stop
command, and the logs show no new entries.
Any of the following conditions might result in a sync group being stuck in the processing state:
Cause. The client agent is offline
Resolution. Be sure that the client agent is online and then try again.
Cause. The client agent is uninstalled or missing.
Resolution. If the client agent is uninstalled or otherwise missing:
1. Remove the agent XML file from the SQL Data Sync installation folder, if the file exists.
2. Install the agent on an on-premises computer (it can be the same or a different computer). Then, submit
the agent key that's generated in the portal for the agent that's showing as offline.
Cause. The SQL Data Sync service is stopped.
Resolution. Restart the SQL Data Sync service.
1. In the Start menu, search for Services.
2. In the search results, select Services.
3. Find the SQL Data Sync service.
4. If the service status is Stopped, right-click the service name, and then select Start.
NOTE
If the preceding information doesn't move your sync group out of the processing state, Microsoft Support can reset the
status of your sync group. To have your sync group status reset, in the Azure SQL Database forum, create a post. In the
post, include your subscription ID and the sync group ID for the group that needs to be reset. A Microsoft Support
engineer will respond to your post, and will let you know when the status has been reset.
Cause. A likely cause of this error is that the password on the local server has changed since you created
the agent and agent password.
Resolution. Update the agent's password to your current server password:
1. Locate the SQL Data Sync client agent service.
a. Select Start.
b. In the search box, enter services.msc.
c. In the search results, select Services.
d. In the Services window, scroll to the entry for SQL Data Sync Agent.
2. Right-click SQL Data Sync Agent, and then select Stop.
3. Right-click SQL Data Sync Agent, and then select Properties.
4. On SQL Data Sync Agent Properties, select the Log in tab.
5. In the Password box, enter your password.
6. In the Confirm Password box, reenter your password.
7. Select Apply, and then select OK.
8. In the Services window, right-click the SQL Data Sync Agent service, and then click Start.
9. Close the Services window.
I can't submit the agent key
After you create or re-create a key for an agent, you try to submit the key through the SqlAzureDataSyncAgent
application. The submission fails to complete.
NOTE
If sync metadata tables remain after a "force delete", use deprovisioningutil.exe to clean them up.
Local Sync Agent app can't connect to the local sync service
Resolution. Try the following steps:
1. Exit the app.
2. Open the Component Services Panel.
a. In the search box on the taskbar, enter services.msc.
b. In the search results, double-click Services.
3. Stop the SQL Data Sync service.
4. Restart the SQL Data Sync service.
5. Reopen the app.
IMPORTANT
Don't delete any files while sync is in progress.
WARNING
You lose all changes made to this database while it was offline.
Next steps
For more information about SQL Data Sync, see:
Sync data across multiple cloud and on-premises databases with Azure SQL Data Sync
Set up Azure SQL Data Sync
Best practices for Azure SQL Data Sync
Monitor Azure SQL Data Sync with Log Analytics
Complete PowerShell examples that show how to configure SQL Data Sync:
Use PowerShell to sync between multiple Azure SQL databases
Use PowerShell to sync between an Azure SQL Database and a SQL Server on-premises database
For more information about SQL Database, see:
SQL Database Overview
Database Lifecycle Management
Load data from CSV into Azure SQL Database (flat
files)
5/23/2018 • 2 minutes to read • Edit Online
You can use the bcp command-line utility to import data from a CSV file into Azure SQL Database.
(Optional) To export your own data from a SQL Server database, open a command prompt and run the following
command. Replace TableName, ServerName, DatabaseName, Username, and Password with your own
information.
sqlcmd.exe -S <server name> -d <database name> -U <username> -P <password> -I -Q "SELECT * FROM DimDate2 ORDER
BY 1;"
20150101 1 3
20150201 1 3
20150301 1 3
20150401 2 4
20150501 2 4
20150601 2 4
20150701 3 1
20150801 3 1
20150801 3 1
DATEID CALENDARQUARTER FISCALQUARTER
20151001 4 2
20151101 4 2
20151201 4 2
Next steps
To migrate a SQL Server database, see SQL Server database migration.
Copy data to or from Azure SQL Database by using
Azure Data Factory
9/13/2018 • 14 minutes to read • Edit Online
This article explains how to use Copy Activity in Azure Data Factory to copy data from or to Azure SQL Database.
It builds on the Copy Activity overview article, which presents a general overview of Copy Activity.
Supported capabilities
You can copy data from or to Azure SQL Database to any supported sink data store. And you can copy data from
any supported source data store to Azure SQL Database. For a list of data stores that are supported as sources or
sinks by Copy Activity, see the Supported data stores and formats table.
Specifically, this Azure SQL Database connector supports these functions:
Copy data by using SQL authentication and Azure Active Directory (Azure AD ) Application token
authentication with a service principal or Managed Service Identity (MSI).
As a source, retrieve data by using a SQL query or stored procedure.
As a sink, append data to a destination table or invoke a stored procedure with custom logic during the copy.
IMPORTANT
If you copy data by using Azure Data Factory Integration Runtime, configure an Azure SQL server firewall so that Azure
Services can access the server. If you copy data by using a self-hosted integration runtime, configure the Azure SQL server
firewall to allow the appropriate IP range. This range includes the machine's IP that is used to connect to Azure SQL Database.
Get started
You can create a pipeline with the copy activity by using one of the following tools or SDKs. Select a link to go to a
tutorial with step-by-step instructions to create a pipeline with a copy activity.
Copy Data tool
Azure portal
.NET SDK
Python SDK
Azure PowerShell
REST API
Azure Resource Manager template.
The following sections provide details about properties that are used to define Data Factory entities specific to an
Azure SQL Database connector.
servicePrincipalId Specify the application's client ID. Yes, when you use Azure AD
authentication with a service principal.
servicePrincipalKey Specify the application's key. Mark this Yes, when you use Azure AD
field as a SecureString to store it authentication with a service principal.
securely in Data Factory, or reference a
secret stored in Azure Key Vault.
tenant Specify the tenant information (domain Yes, when you use Azure AD
name or tenant ID) under which your authentication with a service principal.
application resides. Retrieve it by
hovering the mouse in the top-right
corner of the Azure portal.
For different authentication types, refer to the following sections on prerequisites and JSON samples, respectively:
SQL authentication
Azure AD application token authentication: Service principal
Azure AD application token authentication: Managed Service Identity
TIP
If you hit error with error code as "UserErrorFailedToConnectToSqlServer" and message like "The session limit for the database
is XXX and has been reached.", add Pooling=false to your connection string and try again.
SQL authentication
Linked service example that uses SQL authentication
{
"name": "AzureSqlDbLinkedService",
"properties": {
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value": "Server=tcp:<servername>.database.windows.net,1433;Database=<databasename>;User ID=
<username>@<servername>;Password=<password>;Trusted_Connection=False;Encrypt=True;Connection Timeout=30"
}
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
4. Grant the service principal needed permissions as you normally do for SQL users or others. Run the
following code:
2. Provision an Azure Active Directory administrator for your Azure SQL server on the Azure portal if
you haven't already done so. The Azure AD administrator can be an Azure AD user or Azure AD group. If
you grant the group with MSI an admin role, skip steps 3 and 4. The administrator will have full access to
the database.
3. Create contained database users for the Azure AD group. Connect to the database from or to which you
want to copy data by using tools like SSMS, with an Azure AD identity that has at least ALTER ANY USER
permission. Run the following T-SQL:
4. Grant the Azure AD group needed permissions as you normally do for SQL users and others. For
example, run the following code:
EXEC sp_addrolemember [role name], [your AAD group name];
{
"name": "AzureSqlDbLinkedService",
"properties": {
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value": "Server=tcp:<servername>.database.windows.net,1433;Database=<databasename>;Connection
Timeout=30"
}
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
Dataset properties
For a full list of sections and properties available for defining datasets, see the Datasets article. This section
provides a list of properties supported by the Azure SQL Database dataset.
To copy data from or to Azure SQL Database, set the type property of the dataset to AzureSqlTable. The
following properties are supported:
{
"name": "AzureSQLDbDataset",
"properties":
{
"type": "AzureSqlTable",
"linkedServiceName": {
"referenceName": "<Azure SQL Database linked service name>",
"type": "LinkedServiceReference"
},
"typeProperties": {
"tableName": "MyTable"
}
}
}
Copy Activity properties
For a full list of sections and properties available for defining activities, see the Pipelines article. This section
provides a list of properties supported by the Azure SQL Database source and sink.
Azure SQL Database as the source
To copy data from Azure SQL Database, set the type property in the Copy Activity source to SqlSource. The
following properties are supported in the Copy Activity source section:
Points to note
If the sqlReaderQuery is specified for the SqlSource, Copy Activity runs this query against the Azure SQL
Database source to get the data. Or you can specify a stored procedure. Specify
sqlReaderStoredProcedureName and storedProcedureParameters if the stored procedure takes
parameters.
If you don't specify either sqlReaderQuery or sqlReaderStoredProcedureName, the columns defined in the
structure section of the dataset JSON are used to construct a query. select column1, column2 from mytable
runs against Azure SQL Database. If the dataset definition doesn't have the structure, all columns are selected
from the table.
When you use sqlReaderStoredProcedureName, you still need to specify a dummy tableName property in
the dataset JSON.
SQL query example
"activities":[
{
"name": "CopyFromAzureSQLDatabase",
"type": "Copy",
"inputs": [
{
"referenceName": "<Azure SQL Database input dataset name>",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "<output dataset name>",
"type": "DatasetReference"
}
],
"typeProperties": {
"source": {
"type": "SqlSource",
"sqlReaderQuery": "SELECT * FROM MyTable"
},
"sink": {
"type": "<sink type>"
}
}
}
]
"activities":[
{
"name": "CopyFromAzureSQLDatabase",
"type": "Copy",
"inputs": [
{
"referenceName": "<Azure SQL Database input dataset name>",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "<output dataset name>",
"type": "DatasetReference"
}
],
"typeProperties": {
"source": {
"type": "SqlSource",
"sqlReaderStoredProcedureName": "CopyTestSrcStoredProcedureWithParameters",
"storedProcedureParameters": {
"stringData": { "value": "str3" },
"identifier": { "value": "$$Text.Format('{0:yyyy}', <datetime parameter>)", "type": "Int"}
}
},
"sink": {
"type": "<sink type>"
}
}
}
]
writeBatchSize Inserts data into the SQL table when No. The default is 10000.
the buffer size reaches writeBatchSize.
The allowed value is integer (number of
rows).
TIP
When you copy data to Azure SQL Database, Copy Activity appends data to the sink table by default. To do an upsert or
additional business logic, use the stored procedure in SqlSink. Learn more details from Invoking stored procedure from SQL
Sink.
"activities":[
{
"name": "CopyToAzureSQLDatabase",
"type": "Copy",
"inputs": [
{
"referenceName": "<input dataset name>",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "<Azure SQL Database output dataset name>",
"type": "DatasetReference"
}
],
"typeProperties": {
"source": {
"type": "<source type>"
},
"sink": {
"type": "SqlSink",
"writeBatchSize": 100000
}
}
}
]
Destination table
NOTE
The target table has an identity column.
{
"name": "SampleTarget",
"properties": {
"structure": [
{ "name": "name" },
{ "name": "age" }
],
"type": "AzureSqlTable",
"linkedServiceName": {
"referenceName": "TestIdentitySQL",
"type": "LinkedServiceReference"
},
"typeProperties": {
"tableName": "TargetTbl"
}
}
}
NOTE
Your source and target table have different schema.
The target has an additional column with an identity. In this scenario, you must specify the structure property in
the target dataset definition, which doesn’t include the identity column.
"sink": {
"type": "SqlSink",
"SqlWriterTableType": "MarketingType",
"SqlWriterStoredProcedureName": "spOverwriteMarketing",
"storedProcedureParameters": {
"category": {
"value": "ProductA"
}
}
}
In your database, define the stored procedure with the same name as the SqlWriterStoredProcedureName. It
handles input data from your specified source and merges into the output table. The parameter name of the table
type in the stored procedure should be the same as the tableName defined in the dataset.
In your database, define the table type with the same name as the sqlWriterTableType. The schema of the table
type should be same as the schema returned by your input data.
AZURE SQL DATABASE DATA TYPE DATA FACTORY INTERIM DATA TYPE
bigint Int64
binary Byte[]
bit Boolean
date DateTime
Datetime DateTime
datetime2 DateTime
Datetimeoffset DateTimeOffset
Decimal Decimal
Float Double
image Byte[]
int Int32
money Decimal
numeric Decimal
real Single
rowversion Byte[]
AZURE SQL DATABASE DATA TYPE DATA FACTORY INTERIM DATA TYPE
smalldatetime DateTime
smallint Int16
smallmoney Decimal
sql_variant Object *
time TimeSpan
timestamp Byte[]
tinyint Byte
uniqueidentifier Guid
varbinary Byte[]
xml Xml
Next steps
For a list of data stores supported as sources and sinks by Copy Activity in Azure Data Factory, see Supported data
stores and formats.
Manage file space in Azure SQL Database
8/16/2018 • 6 minutes to read • Edit Online
This article describes different types of storage space in Azure SQL Database, and steps that can be taken when
the file space allocated for databases and elastic pools needs to be explicitly managed.
Overview
In Azure SQL Database, most storage space metrics displayed in the Azure portal and the following APIs
measure the number of used data pages for databases and elastic pools:
Azure Resource Manager based metrics APIs including PowerShell get-metrics
T-SQL: sys.dm_db_resource_stats
T-SQL: sys.resource_stats
T-SQL: sys.elastic_pool_resource_stats
There are workload patterns where the allocation of underlying data files for databases can become larger than
the amount of used data pages. This can occur when space used increases and data is subsequently deleted.
This is because file space allocated is not automatically reclaimed when data is deleted. In such scenarios, the
allocated space for a database or pool may exceed supported limits and prevent data growth or prevent
performance tier changes, and require shrinking data files to mitigate.
The SQL DB service does not automatically shrink data files to reclaim unused allocated space due to the
potential impact to database performance. However, customers may shrink data files via self-service at a time of
their choosing by following the steps described in Reclaim unused allocated space.
NOTE
Unlike data files, the SQL Database service automatically shrinks log files since that operation does not impact database
performance.
Data space used The amount of space used to store Generally, space used increases
database data in 8 KB pages. (decreases) on inserts (deletes). In
some cases, the space used does not
change on inserts or deletes
depending on the amount and pattern
of data involved in the operation and
any fragmentation. For example,
deleting one row from every data page
does not necessarily decrease the
space used.
DATABASE QUANTITY DEFINITION COMMENTS
Data space allocated The amount of formatted file space The amount of space allocated grows
made available for storing database automatically, but never decreases
data. after deletes. This behavior ensures
that future inserts are faster since
space does not need to be
reformatted.
Data space allocated but unused The difference between the amount of This quantity represents the maximum
data space allocated and data space amount of free space that can be
used. reclaimed by shrinking database data
files.
Data max size The maximum amount of space that The amount of data space allocated
can be used for storing database data. cannot grow beyond the data max
size.
The following diagram illustrates the relationship between the different types of storage space for a database.
-- Connect to master
-- Database data space used in MB
SELECT TOP 1 storage_in_megabytes AS DatabaseDataSpaceUsedInMB
FROM sys.resource_stats
WHERE database_name = 'db1'
ORDER BY end_time DESC
-- Connect to database
-- Database data max size in bytes
SELECT DATABASEPROPERTYEX('db1', 'MaxSizeInBytes') AS DatabaseDataMaxSizeInBytes
Data space allocated but unused The difference between the amount of This quantity represents the maximum
data space allocated and data space amount of space allocated for the
used by all databases in the elastic elastic pool that can be reclaimed by
pool. shrinking database data files.
Data max size The maximum amount of data space The space allocated for the elastic pool
that can be used by the elastic pool for should not exceed the elastic pool max
all of its databases. size. If this occurs, then space allocated
that is unused can be reclaimed by
shrinking database data files.
-- Connect to master
-- Elastic pool data space used in MB
SELECT TOP 1 avg_storage_percent * elastic_pool_storage_limit_mb AS ElasticPoolDataSpaceUsedInMB
FROM sys.elastic_pool_resource_stats
WHERE elastic_pool_name = 'ep1'
ORDER BY end_time DESC
Elastic pool data space allocated and unused allocated space
Modify the following PowerShell script to return a table listing the space allocated and unused allocated space
for each database in an elastic pool. The table orders databases from those with the greatest amount of unused
allocated space to the least amount of unused allocated space. Units of the query result are in MB.
The query results for determining the space allocated for each database in the pool can be added together to
determine the total space allocated for the elastic pool. The elastic pool space allocated should not exceed the
elastic pool max size.
The PowerShell script requires SQL Server PowerShell module – see Download PowerShell module to install.
-- Connect to master
-- Elastic pools max size in MB
SELECT TOP 1 elastic_pool_storage_limit_mb AS ElasticPoolMaxSizeInMB
FROM sys.elastic_pool_resource_stats
WHERE elastic_pool_name = 'ep1'
ORDER BY end_time DESC
IMPORTANT
Consider rebuilding database indexes After database data files are shrunk, indexes may become fragmented and lose their
performance optimization effectiveness. If this occurs, then the indexes should be rebuilt. For more information on
fragmentation and rebuilding indexes, see Reorganize and Rebuild Indexes.
Next steps
For information about database max sizes, see:
Azure SQL Database vCore-based purchasing model limits for a single database
Resource limits for single databases using the DTU -based purchasing model
Azure SQL Database vCore-based purchasing model limits for elastic pools
Resources limits for elastic pools using the DTU -based purchasing model
For more information about the SHRINKDATABASE command, see SHRINKDATABASE.
For more information on fragmentation and rebuilding indexes, see Reorganize and Rebuild Indexes.
Connectivity libraries and frameworks for SQL
Server
5/23/2018 • 2 minutes to read • Edit Online
Check out our Get started tutorials to quickly get started with programming languages such as C#, Java, Node.js,
PHP, and Python. Then build an app by using SQL Server on Linux or Windows or Docker on macOS.
The following table lists connectivity libraries or drivers that client applications can use from a variety of
languages to connect to and use SQL Server running on-premises or in the cloud. You can use them on Linux,
Windows, or Docker and use them to connect to Azure SQL Database and Azure SQL Data Warehouse.
ADDITIONAL
LANGUAGE PLATFORM RESOURCES DOWNLOAD GET STARTED
PHP Windows, Linux, PHP SQL driver for Operating system: Get started
macOS SQL Server * Windows
* Linux
* macOS
Node.js Windows, Linux, Node.js driver for SQL Install Get started
macOS Server
Python Windows, Linux, Python SQL driver Install choices: Get started
macOS * pymssql
* pyodbc
Ruby Windows, Linux, Ruby driver for SQL Install Get started
macOS Server
The following table lists examples of object-relational mapping (ORM ) frameworks and web frameworks that
client applications can use with SQL Server running on-premises or in the cloud. You can use the frameworks on
Linux, Windows, or Docker and use them to connect to SQL Database and SQL Data Warehouse.
Related links
SQL Server drivers that are used to connect from client applications
Connect to SQL Database:
Connect to SQL Database by using .NET (C#)
Connect to SQL Database by using PHP
Connect to SQL Database by using Node.js
Connect to SQL Database by using Java
Connect to SQL Database by using Python
Connect to SQL Database by using Ruby
Retry logic code examples:
Connect resiliently to SQL with ADO.NET
Connect resiliently to SQL with PHP
Accelerate real-time big data analytics with Spark
connector for Azure SQL Database and SQL Server
5/23/2018 • 4 minutes to read • Edit Online
The Spark connector for Azure SQL Database and SQL Server enables SQL databases, including Azure SQL
Database and SQL Server, to act as input data source or output data sink for Spark jobs. It allows you to utilize
real-time transactional data in big data analytics and persist results for adhoc queries or reporting. Compared to
the built-in JDBC connector, this connector provides the ability to bulk insert data into SQL databases. It can
outperform row by row insertion with 10x to 20x faster performance. The Spark connector for Azure SQL
Database and SQL Server also supports AAD authentication. It allows you securely connecting to your Azure SQL
database from Azure Databricks using your AAD account. It provides similar interfaces with the built-in JDBC
connector. It is easy to migrate your existing Spark jobs to use this new connector.
Download
To get started, download the Spark to SQL DB connector from the azure-sqldb-spark repository on GitHub.
The Spark connector for Azure SQL Database and SQL Server utilizes the Microsoft JDBC Driver for SQL Server
to move data between Spark worker nodes and SQL databases:
The dataflow is as follows:
1. The Spark master node connects to SQL Server or Azure SQL Database and loads data from a specific table or
using a specific SQL query
2. The Spark master node distributes data to worker nodes for transformation.
3. The Worker node connects to SQL Server or Azure SQL Database and writes data to the database. User can
choose to use row -by-row insertion or bulk insert.
The following diagram illustrates the data flow.
Build the Spark to SQL DB connector
Currently, the connector project uses maven. To build the connector without dependencies, you can run:
mvn clean package
Download the latest versions of the JAR from the release folder
Include the SQL DB Spark JAR
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
Reading data from Azure SQL Database or SQL Server with specified SQL query
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
import org.apache.spark.sql.SaveMode
collection.write.mode(SaveMode.Append).sqlDB(config)
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.query._
val query = """
|UPDATE Customers
|SET ContactName = 'Alfred Schmidt', City= 'Frankfurt'
|WHERE CustomerID = 1;
""".stripMargin
sqlContext.SqlDBQuery(config)
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
Write data to Azure SQL database or SQL Server using Bulk Insert
The traditional jdbc connector writes data into Azure SQL database or SQL Server using row -by-row insertion.
You can use Spark to SQL DB connector to write data to SQL database using bulk insert. It significantly improves
the write performance when loading large data sets or loading data into tables where a column store index is used.
import com.microsoft.azure.sqldb.spark.bulkcopy.BulkCopyMetadata
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
/**
Add column Metadata.
If not specified, metadata is automatically added
from the destination table, which may suffer performance.
*/
var bulkCopyMetadata = new BulkCopyMetadata
bulkCopyMetadata.addColumnMetadata(1, "Title", java.sql.Types.NVARCHAR, 128, 0)
bulkCopyMetadata.addColumnMetadata(2, "FirstName", java.sql.Types.NVARCHAR, 50, 0)
bulkCopyMetadata.addColumnMetadata(3, "LastName", java.sql.Types.NVARCHAR, 50, 0)
df.bulkCopyToSqlDB(bulkCopyConfig, bulkCopyMetadata)
//df.bulkCopyToSqlDB(bulkCopyConfig) if no metadata is specified.
Next steps
If you haven't already, download the Spark connector for Azure SQL Database and SQL Server from azure-sqldb-
spark GitHub repository and explore the additional resources in the repo:
Sample Azure Databricks notebooks
Sample scripts (Scala)
You might also want to review the Apache Spark SQL, DataFrames, and Datasets Guide and the Azure Databricks
documentation.
Get the required values for authenticating an
application to access SQL Database from code
5/23/2018 • 2 minutes to read • Edit Online
To create and manage SQL Database from code you must register your app in the Azure Active Directory (AAD )
domain in the subscription where your Azure resources have been created.
# Sign in to Azure.
Connect-AzureRmAccount
# If you have multiple subscriptions, uncomment and set to the subscription you want to work with.
#$subscriptionId = "{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}"
#Set-AzureRmContext -SubscriptionId $subscriptionId
$appName = "{app-name}"
$uri = "http://{app-name}"
$secret = "{app-password}"
# If you still get a PrincipalNotFound error, then rerun the following until successful.
$roleassignment = New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName
$azureAdApplication.ApplicationId.Guid
This article lists SQL error codes for SQL Database client applications, including database connection errors,
transient errors (also called transient faults), resource governance errors, database copy issues, elastic pool, and
other errors. Most categories are particular to Azure SQL Database, and do not apply to Microsoft SQL Server.
See also system error messages.
Related topics:
More detailed information is available here: Azure SQL Database resource limits.
ERROR CODE SEVERITY DESCRIPTION
ERRORCORRECTIV
ERRORNUMBER ERRORSEVERITY ERRORFORMAT ERRORINSERTS ERRORCAUSE EACTION
1132 EX_RESOURCE The elastic pool Elastic pool space Attempting to Consider
has reached its limit in MBs. write data to a increasing the
storage limit. The database when DTUs of and/or
storage usage for the storage limit adding storage
the elastic pool of the elastic pool to the elastic
cannot exceed has been pool if possible in
(%d) MBs. reached. order to increase
its storage limit,
reduce the
storage used by
individual
databases within
the elastic pool,
or remove
databases from
the elastic pool.
ERRORCORRECTIV
ERRORNUMBER ERRORSEVERITY ERRORFORMAT ERRORINSERTS ERRORCAUSE EACTION
10929 EX_USER The %s minimum DTU / vCore min The total number Consider
guarantee is %d, per database; of concurrent increasing the
maximum limit is DTU / vCore max workers DTUs or vCores
%d, and the per database (requests) across of the elastic pool
current usage for all databases in if possible in
the database is the elastic pool order to increase
%d. However, the attempted to its worker limit,
server is exceed the pool or remove
currently too limit. databases from
busy to support the elastic pool.
requests greater
than %d for this
database. See
http://go.microso
ft.com/fwlink/?
LinkId=267637
for assistance.
Otherwise, please
try again later.
40857 EX_USER Elastic pool not name of server; Specified elastic Provide a valid
found for server: elastic pool name pool does not elastic pool
'%ls', elastic pool exist in the name.
name: '%ls'. specified server.
40858 EX_USER Elastic pool '%ls' elastic pool Specified elastic Provide new
already exists in name, server pool already elastic pool
server: '%ls' name exists in the name.
specified logical
server.
40859 EX_USER Elastic pool does elastic pool Specified service Provide the
not support service tier tier is not correct edition or
service tier '%ls'. supported for leave service tier
elastic pool blank to use the
provisioning. default service
tier.
40860 EX_USER Elastic pool '%ls' elastic pool Elastic pool and Specify correct
and service name; service service objective combination of
objective '%ls' level objective can be specified elastic pool and
combination is name together only if service objective.
invalid. service objective
is specified as
'ElasticPool'.
ERRORCORRECTIV
ERRORNUMBER ERRORSEVERITY ERRORFORMAT ERRORINSERTS ERRORCAUSE EACTION
40861 EX_USER The database database edition, The database Do not specify a
edition '%.ls' elastic pool edition is database edition
cannot be service tier different than the which is different
different than elastic pool than the elastic
the elastic pool service tier. pool service tier.
service tier which Note that the
is '%.ls'. database edition
does not need to
be specified.
40862 EX_USER Elastic pool name None Elastic pool Specify the elastic
must be specified service objective pool name if
if the elastic pool does not using the elastic
service objective uniquely identify pool service
is specified. an elastic pool. objective.
40864 EX_USER The DTUs for the DTUs for elastic Attempting to Retry setting the
elastic pool must pool; elastic pool set the DTUs for DTUs for the
be at least (%d) service tier. the elastic pool elastic pool to at
DTUs for service below the least the
tier '%.*ls'. minimum limit. minimum limit.
40865 EX_USER The DTUs for the DTUs for elastic Attempting to Retry setting the
elastic pool pool; elastic pool set the DTUs for DTUs for the
cannot exceed service tier. the elastic pool elastic pool to no
(%d) DTUs for above the greater than the
service tier '%.*ls'. maximum limit. maximum limit.
40867 EX_USER The DTU max per DTU max per Attempting to Consider using
database must database; elastic set the DTU max the elastic pool
be at least (%d) pool service tier per database service tier that
for service tier below the supports the
'%.*ls'. supported limit. desired setting.
40868 EX_USER The DTU max per DTU max per Attempting to Consider using
database cannot database; elastic set the DTU max the elastic pool
exceed (%d) for pool service tier. per database service tier that
service tier '%.*ls'. beyond the supports the
supported limit. desired setting.
40870 EX_USER The DTU min per DTU min per Attempting to Consider using
database cannot database; elastic set the DTU min the elastic pool
exceed (%d) for pool service tier. per database service tier that
service tier '%.*ls'. beyond the supports the
supported limit. desired setting.
40891 EX_USER The DTU min per DTU min per Attempting to Ensure the DTU
database (%d) database; DTU set the DTU min min per
cannot exceed max per per database databases does
the DTU max per database. higher than the not exceed the
database (%d). DTU max per DTU max per
database. database.
TBD EX_USER The storage size elastic pool The max size for Set the max size
for an individual service tier the database of the database
database in an exceeds the max within the limits
elastic pool size allowed by of the max size
cannot exceed the elastic pool allowed by the
the max size service tier. elastic pool
allowed by '%.*ls' service tier.
service tier elastic
pool.
Related topics:
Create an elastic pool (C#)
Manage an elastic pool (C#).
Create an elastic pool (PowerShell)
Monitor and manage an elastic pool (PowerShell).
General errors
The following errors do not fall into any previous categories.
Next steps
Read about Azure SQL Database features.
Read about DTU -based purchasing model.
Read about vCore-based purchasing model.
How to use batching to improve SQL Database
application performance
5/23/2018 • 25 minutes to read • Edit Online
Batching operations to Azure SQL Database significantly improves the performance and scalability of your
applications. In order to understand the benefits, the first part of this article covers some sample test results that
compare sequential and batched requests to a SQL Database. The remainder of the article shows the techniques,
scenarios, and considerations to help you to use batching successfully in your Azure applications.
Batching strategies
Note about timing results in this article
NOTE
Results are not benchmarks but are meant to show relative performance. Timings are based on an average of at least 10
test runs. Operations are inserts into an empty table. These tests were measured pre-V12, and they do not necessarily
correspond to throughput that you might experience in a V12 database using the new DTU service tiers or vCore service
tiers. The relative benefit of the batching technique should be similar.
Transactions
It seems strange to begin a review of batching by discussing transactions. But the use of client-side transactions
has a subtle server-side batching effect that improves performance. And transactions can be added with only a few
lines of code, so they provide a fast way to improve performance of sequential operations.
Consider the following C# code that contains a sequence of insert and update operations on a simple table.
The best way to optimize this code is to implement some form of client-side batching of these calls. But there is a
simple way to increase the performance of this code by simply wrapping the sequence of calls in a transaction.
Here is the same code that uses a transaction.
transaction.Commit();
}
Transactions are actually being used in both of these examples. In the first example, each individual call is an
implicit transaction. In the second example, an explicit transaction wraps all of the calls. Per the documentation for
the write-ahead transaction log, log records are flushed to the disk when the transaction commits. So by including
more calls in a transaction, the write to the transaction log can delay until the transaction is committed. In effect,
you are enabling batching for the writes to the server’s transaction log.
The following table shows some ad-hoc testing results. The tests performed the same sequential inserts with and
without transactions. For more perspective, the first set of tests ran remotely from a laptop to the database in
Microsoft Azure. The second set of tests ran from a cloud service and database that both resided within the same
Microsoft Azure datacenter (West US ). The following table shows the duration in milliseconds of sequential inserts
with and without transactions.
On-Premises to Azure:
1 130 402
10 1208 1226
1 21 26
10 220 56
NOTE
Results are not benchmarks. See the note about timing results in this topic.
Based on the previous test results, wrapping a single operation in a transaction actually decreases performance. But
as you increase the number of operations within a single transaction, the performance improvement becomes
more marked. The performance difference is also more noticeable when all operations occur within the Microsoft
Azure datacenter. The increased latency of using SQL Database from outside the Microsoft Azure datacenter
overshadows the performance gain of using transactions.
Although the use of transactions can increase performance, continue to observe best practices for transactions and
connections. Keep the transaction as short as possible, and close the database connection after the work completes.
The using statement in the previous example assures that the connection is closed when the subsequent code block
completes.
The previous example demonstrates that you can add a local transaction to any ADO.NET code with two lines.
Transactions offer a quick way to improve the performance of code that makes sequential insert, update, and delete
operations. However, for the fastest performance, consider changing the code further to take advantage of client-
side batching, such as table-valued parameters.
For more information about transactions in ADO.NET, see Local Transactions in ADO.NET.
Table -valued parameters
Table-valued parameters support user-defined table types as parameters in Transact-SQL statements, stored
procedures, and functions. This client-side batching technique allows you to send multiple rows of data within the
table-valued parameter. To use table-valued parameters, first define a table type. The following Transact-SQL
statement creates a table type named MyTableType.
CREATE TYPE MyTableType AS TABLE
( mytext TEXT,
num INT );
In code, you create a DataTable with the exact same names and types of the table type. Pass this DataTable in a
parameter in a text query or stored procedure call. The following example shows this technique:
cmd.Parameters.Add(
new SqlParameter()
{
ParameterName = "@TestTvp",
SqlDbType = SqlDbType.Structured,
TypeName = "MyTableType",
Value = table,
});
cmd.ExecuteNonQuery();
}
In the previous example, the SqlCommand object inserts rows from a table-valued parameter, @TestTvp. The
previously created DataTable object is assigned to this parameter with the SqlCommand.Parameters.Add
method. Batching the inserts in one call significantly increases the performance over sequential inserts.
To improve the previous example further, use a stored procedure instead of a text-based command. The following
Transact-SQL command creates a stored procedure that takes the SimpleTestTableType table-valued parameter.
Then change the SqlCommand object declaration in the previous code example to the following.
In most cases, table-valued parameters have equivalent or better performance than other batching techniques.
Table-valued parameters are often preferable, because they are more flexible than other options. For example,
other techniques, such as SQL bulk copy, only permit the insertion of new rows. But with table-valued parameters,
you can use logic in the stored procedure to determine which rows are updates and which are inserts. The table
type can also be modified to contain an “Operation” column that indicates whether the specified row should be
inserted, updated, or deleted.
The following table shows ad-hoc test results for the use of table-valued parameters in milliseconds.
1 124 32
10 131 25
100 338 51
NOTE
Results are not benchmarks. See the note about timing results in this topic.
The performance gain from batching is immediately apparent. In the previous sequential test, 1000 operations took
129 seconds outside the datacenter and 21 seconds from within the datacenter. But with table-valued parameters,
1000 operations take only 2.6 seconds outside the datacenter and 0.4 seconds within the datacenter.
For more information on table-valued parameters, see Table-Valued Parameters.
SQL bulk copy
SQL bulk copy is another way to insert large amounts of data into a target database. .NET applications can use the
SqlBulkCopy class to perform bulk insert operations. SqlBulkCopy is similar in function to the command-line
tool, Bcp.exe, or the Transact-SQL statement, BULK INSERT. The following code example shows how to bulk
copy the rows in the source DataTable, table, to the destination table in SQL Server, MyTable.
There are some cases where bulk copy is preferred over table-valued parameters. See the comparison table of
Table-Valued parameters versus BULK INSERT operations in the article Table-Valued Parameters.
The following ad-hoc test results show the performance of batching with SqlBulkCopy in milliseconds.
OPERATIONS ON-PREMISES TO AZURE (MS) AZURE SAME DATACENTER (MS)
1 433 57
10 441 32
100 636 53
NOTE
Results are not benchmarks. See the note about timing results in this topic.
In smaller batch sizes, the use table-valued parameters outperformed the SqlBulkCopy class. However,
SqlBulkCopy performed 12-31% faster than table-valued parameters for the tests of 1,000 and 10,000 rows. Like
table-valued parameters, SqlBulkCopy is a good option for batched inserts, especially when compared to the
performance of non-batched operations.
For more information on bulk copy in ADO.NET, see Bulk Copy Operations in SQL Server.
Multiple -row Parameterized INSERT statements
One alternative for small batches is to construct a large parameterized INSERT statement that inserts multiple
rows. The following code example demonstrates this technique.
cmd.ExecuteNonQuery();
}
This example is meant to show the basic concept. A more realistic scenario would loop through the required
entities to construct the query string and the command parameters simultaneously. You are limited to a total of
2100 query parameters, so this limits the total number of rows that can be processed in this manner.
The following ad-hoc test results show the performance of this type of insert statement in milliseconds.
1 32 20
OPERATIONS TABLE-VALUED PARAMETERS (MS) SINGLE-STATEMENT INSERT (MS)
10 30 25
100 33 51
NOTE
Results are not benchmarks. See the note about timing results in this topic.
This approach can be slightly faster for batches that are less than 100 rows. Although the improvement is small,
this technique is another option that might work well in your specific application scenario.
DataAdapter
The DataAdapter class allows you to modify a DataSet object and then submit the changes as INSERT, UPDATE,
and DELETE operations. If you are using the DataAdapter in this manner, it is important to note that separate calls
are made for each distinct operation. To improve performance, use the UpdateBatchSize property to the number
of operations that should be batched at a time. For more information, see Performing Batch Operations Using
DataAdapters.
Entity framework
Entity Framework does not currently support batching. Different developers in the community have attempted to
demonstrate workarounds, such as override the SaveChanges method. But the solutions are typically complex
and customized to the application and data model. The Entity Framework codeplex project currently has a
discussion page on this feature request. To view this discussion, see Design Meeting Notes - August 2, 2012.
XML
For completeness, we feel that it is important to talk about XML as a batching strategy. However, the use of XML
has no advantages over other methods and several disadvantages. The approach is similar to table-valued
parameters, but an XML file or string is passed to a stored procedure instead of a user-defined table. The stored
procedure parses the commands in the stored procedure.
There are several disadvantages to this approach:
Working with XML can be cumbersome and error prone.
Parsing the XML on the database can be CPU -intensive.
In most cases, this method is slower than table-valued parameters.
For these reasons, the use of XML for batch queries is not recommended.
Batching considerations
The following sections provide more guidance for the use of batching in SQL Database applications.
Tradeoffs
Depending on your architecture, batching can involve a tradeoff between performance and resiliency. For example,
consider the scenario where your role unexpectedly goes down. If you lose one row of data, the impact is smaller
than the impact of losing a large batch of unsubmitted rows. There is a greater risk when you buffer rows before
sending them to the database in a specified time window.
Because of this tradeoff, evaluate the type of operations that you batch. Batch more aggressively (larger batches
and longer time windows) with data that is less critical.
Batch size
In our tests, there was typically no advantage to breaking large batches into smaller chunks. In fact, this subdivision
often resulted in slower performance than submitting a single large batch. For example, consider a scenario where
you want to insert 1000 rows. The following table shows how long it takes to use table-valued parameters to insert
1000 rows when divided into smaller batches.
1000 1 347
500 2 355
100 10 465
50 20 630
NOTE
Results are not benchmarks. See the note about timing results in this topic.
You can see that the best performance for 1000 rows is to submit them all at once. In other tests (not shown here),
there was a small performance gain to break a 10000 row batch into two batches of 5000. But the table schema for
these tests is relatively simple, so you should perform tests on your specific data and batch sizes to verify these
findings.
Another factor to consider is that if the total batch becomes too large, SQL Database might throttle and refuse to
commit the batch. For the best results, test your specific scenario to determine if there is an ideal batch size. Make
the batch size configurable at runtime to enable quick adjustments based on performance or errors.
Finally, balance the size of the batch with the risks associated with batching. If there are transient errors or the role
fails, consider the consequences of retrying the operation or of losing the data in the batch.
Parallel processing
What if you took the approach of reducing the batch size but used multiple threads to execute the work? Again, our
tests showed that several smaller multithreaded batches typically performed worse than a single larger batch. The
following test attempts to insert 1000 rows in one or more parallel batches. This test shows how more
simultaneous batches actually decreased performance.
BATCH SIZE [ITERATIONS] TWO THREADS (MS) FOUR THREADS (MS) SIX THREADS (MS)
NOTE
Results are not benchmarks. See the note about timing results in this topic.
There are several potential reasons for the degradation in performance due to parallelism:
There are multiple simultaneous network calls instead of one.
Multiple operations against a single table can result in contention and blocking.
There are overheads associated with multithreading.
The expense of opening multiple connections outweighs the benefit of parallel processing.
If you target different tables or databases, it is possible to see some performance gain with this strategy. Database
sharding or federations would be a scenario for this approach. Sharding uses multiple databases and routes
different data to each database. If each small batch is going to a different database, then performing the operations
in parallel can be more efficient. However, the performance gain is not significant enough to use as the basis for a
decision to use database sharding in your solution.
In some designs, parallel execution of smaller batches can result in improved throughput of requests in a system
under load. In this case, even though it is quicker to process a single larger batch, processing multiple batches in
parallel might be more efficient.
If you do use parallel execution, consider controlling the maximum number of worker threads. A smaller number
might result in less contention and a faster execution time. Also, consider the additional load that this places on the
target database both in connections and transactions.
Related performance factors
Typical guidance on database performance also affects batching. For example, insert performance is reduced for
tables that have a large primary key or many nonclustered indexes.
If table-valued parameters use a stored procedure, you can use the command SET NOCOUNT ON at the
beginning of the procedure. This statement suppresses the return of the count of the affected rows in the
procedure. However, in our tests, the use of SET NOCOUNT ON either had no effect or decreased performance.
The test stored procedure was simple with a single INSERT command from the table-valued parameter. It is
possible that more complex stored procedures would benefit from this statement. But don’t assume that adding
SET NOCOUNT ON to your stored procedure automatically improves performance. To understand the effect, test
your stored procedure with and without the SET NOCOUNT ON statement.
Batching scenarios
The following sections describe how to use table-valued parameters in three application scenarios. The first
scenario shows how buffering and batching can work together. The second scenario improves performance by
performing master-detail operations in a single stored procedure call. The final scenario shows how to use table-
valued parameters in an “UPSERT” operation.
Buffering
Although there are some scenarios that are obvious candidate for batching, there are many scenarios that could
take advantage of batching by delayed processing. However, delayed processing also carries a greater risk that the
data is lost in the event of an unexpected failure. It is important to understand this risk and consider the
consequences.
For example, consider a web application that tracks the navigation history of each user. On each page request, the
application could make a database call to record the user’s page view. But higher performance and scalability can
be achieved by buffering the users’ navigation activities and then sending this data to the database in batches. You
can trigger the database update by elapsed time and/or buffer size. For example, a rule could specify that the batch
should be processed after 20 seconds or when the buffer reaches 1000 items.
The following code example uses Reactive Extensions - Rx to process buffered events raised by a monitoring class.
When the buffer fills or a timeout is reached, the batch of user data is sent to the database with a table-valued
parameter.
The following NavHistoryData class models the user navigation details. It contains basic information such as the
user identifier, the URL accessed, and the access time.
The NavHistoryDataMonitor class is responsible for buffering the user navigation data to the database. It contains
a method, RecordUserNavigationEntry, which responds by raising an OnAdded event. The following code shows
the constructor logic that uses Rx to create an observable collection based on the event. It then subscribes to this
observable collection with the Buffer method. The overload specifies that the buffer should be sent every 20
seconds or 1000 entries.
public NavHistoryDataMonitor()
{
var observableData =
Observable.FromEventPattern<NavHistoryDataEventArgs>(this, "OnAdded");
observableData.Buffer(TimeSpan.FromSeconds(20), 1000).Subscribe(Handler);
}
The handler converts all of the buffered items into a table-valued type and then passes this type to a stored
procedure that processes the batch. The following code shows the complete definition for both the
NavHistoryDataEventArgs and the NavHistoryDataMonitor classes.
public class NavHistoryDataEventArgs : System.EventArgs
{
public NavHistoryDataEventArgs(NavHistoryData data) { Data = data; }
public NavHistoryData Data { get; set; }
}
public NavHistoryDataMonitor()
{
var observableData =
Observable.FromEventPattern<NavHistoryDataEventArgs>(this, "OnAdded");
observableData.Buffer(TimeSpan.FromSeconds(20), 1000).Subscribe(Handler);
}
cmd.Parameters.Add(
new SqlParameter()
{
ParameterName = "@NavHistoryBatch",
SqlDbType = SqlDbType.Structured,
TypeName = "NavigationHistoryTableType",
Value = navHistoryBatch,
});
cmd.ExecuteNonQuery();
}
}
}
To use this buffering class, the application creates a static NavHistoryDataMonitor object. Each time a user accesses
a page, the application calls the NavHistoryDataMonitor.RecordUserNavigationEntry method. The buffering logic
proceeds to take care of sending these entries to the database in batches.
Master detail
Table-valued parameters are useful for simple INSERT scenarios. However, it can be more challenging to batch
inserts that involve more than one table. The “master/detail” scenario is a good example. The master table
identifies the primary entity. One or more detail tables store more data about the entity. In this scenario, foreign
key relationships enforce the relationship of details to a unique master entity. Consider a simplified version of a
PurchaseOrder table and its associated OrderDetail table. The following Transact-SQL creates the PurchaseOrder
table with four columns: OrderID, OrderDate, CustomerID, and Status.
Each order contains one or more product purchases. This information is captured in the PurchaseOrderDetail table.
The following Transact-SQL creates the PurchaseOrderDetail table with five columns: OrderID, OrderDetailID,
ProductID, UnitPrice, and OrderQty.
The OrderID column in the PurchaseOrderDetail table must reference an order from the PurchaseOrder table. The
following definition of a foreign key enforces this constraint.
In order to use table-valued parameters, you must have one user-defined table type for each target table.
Then define a stored procedure that accepts tables of these types. This procedure allows an application to locally
batch a set of orders and order details in a single call. The following Transact-SQL provides the complete stored
procedure declaration for this purchase order example.
CREATE PROCEDURE sp_InsertOrdersBatch (
@orders as PurchaseOrderTableType READONLY,
@details as PurchaseOrderDetailTableType READONLY )
AS
SET NOCOUNT ON;
In this example, the locally defined @IdentityLink table stores the actual OrderID values from the newly inserted
rows. These order identifiers are different from the temporary OrderID values in the @orders and @details table-
valued parameters. For this reason, the @IdentityLink table then connects the OrderID values from the @orders
parameter to the real OrderID values for the new rows in the PurchaseOrder table. After this step, the
@IdentityLink table can facilitate inserting the order details with the actual OrderID that satisfies the foreign key
constraint.
This stored procedure can be used from code or from other Transact-SQL calls. See the table-valued parameters
section of this paper for a code example. The following Transact-SQL shows how to call the sp_InsertOrdersBatch.
declare @orders as PurchaseOrderTableType
declare @details as PurchaseOrderDetailTableType
INSERT @orders
([OrderID], [OrderDate], [CustomerID], [Status])
VALUES(1, '1/1/2013', 1125, 'Complete'),
(2, '1/13/2013', 348, 'Processing'),
(3, '1/12/2013', 2504, 'Shipped')
INSERT @details
([OrderID], [ProductID], [UnitPrice], [OrderQty])
VALUES(1, 10, $11.50, 1),
(1, 12, $1.58, 1),
(2, 23, $2.57, 2),
(3, 4, $10.00, 1)
This solution allows each batch to use a set of OrderID values that begin at 1. These temporary OrderID values
describe the relationships in the batch, but the actual OrderID values are determined at the time of the insert
operation. You can run the same statements in the previous example repeatedly and generate unique orders in the
database. For this reason, consider adding more code or database logic that prevents duplicate orders when using
this batching technique.
This example demonstrates that even more complex database operations, such as master-detail operations, can be
batched using table-valued parameters.
UPSERT
Another batching scenario involves simultaneously updating existing rows and inserting new rows. This operation
is sometimes referred to as an “UPSERT” (update + insert) operation. Rather than making separate calls to INSERT
and UPDATE, the MERGE statement is best suited to this task. The MERGE statement can perform both insert and
update operations in a single call.
Table-valued parameters can be used with the MERGE statement to perform updates and inserts. For example,
consider a simplified Employee table that contains the following columns: EmployeeID, FirstName, LastName,
SocialSecurityNumber:
In this example, you can use the fact that the SocialSecurityNumber is unique to perform a MERGE of multiple
employees. First, create the user-defined table type:
Next, create a stored procedure or write code that uses the MERGE statement to perform the update and insert.
The following example uses the MERGE statement on a table-valued parameter, @employees, of type
EmployeeTableType. The contents of the @employees table are not shown here.
MERGE Employee AS target
USING (SELECT [FirstName], [LastName], [SocialSecurityNumber] FROM @employees)
AS source ([FirstName], [LastName], [SocialSecurityNumber])
ON (target.[SocialSecurityNumber] = source.[SocialSecurityNumber])
WHEN MATCHED THEN
UPDATE SET
target.FirstName = source.FirstName,
target.LastName = source.LastName
WHEN NOT MATCHED THEN
INSERT ([FirstName], [LastName], [SocialSecurityNumber])
VALUES (source.[FirstName], source.[LastName], source.[SocialSecurityNumber]);
For more information, see the documentation and examples for the MERGE statement. Although the same work
could be performed in a multiple-step stored procedure call with separate INSERT and UPDATE operations, the
MERGE statement is more efficient. Database code can also construct Transact-SQL calls that use the MERGE
statement directly without requiring two database calls for INSERT and UPDATE.
Recommendation summary
The following list provides a summary of the batching recommendations discussed in this article:
Use buffering and batching to increase the performance and scalability of SQL Database applications.
Understand the tradeoffs between batching/buffering and resiliency. During a role failure, the risk of losing an
unprocessed batch of business-critical data might outweigh the performance benefit of batching.
Attempt to keep all calls to the database within a single datacenter to reduce latency.
If you choose a single batching technique, table-valued parameters offer the best performance and flexibility.
For the fastest insert performance, follow these general guidelines but test your scenario:
For < 100 rows, use a single parameterized INSERT command.
For < 1000 rows, use table-valued parameters.
For >= 1000 rows, use SqlBulkCopy.
For update and delete operations, use table-valued parameters with stored procedure logic that determines the
correct operation on each row in the table parameter.
Batch size guidelines:
Use the largest batch sizes that make sense for your application and business requirements.
Balance the performance gain of large batches with the risks of temporary or catastrophic failures. What
is the consequence of retries or loss of the data in the batch?
Test the largest batch size to verify that SQL Database does not reject it.
Create configuration settings that control batching, such as the batch size or the buffering time window.
These settings provide flexibility. You can change the batching behavior in production without
redeploying the cloud service.
Avoid parallel execution of batches that operate on a single table in one database. If you do choose to divide a
single batch across multiple worker threads, run tests to determine the ideal number of threads. After an
unspecified threshold, more threads will decrease performance rather than increase it.
Consider buffering on size and time as a way of implementing batching for more scenarios.
Next steps
This article focused on how database design and coding techniques related to batching can improve your
application performance and scalability. But this is just one factor in your overall strategy. For more ways to
improve performance and scalability, see Azure SQL Database performance guidance for single databases and
Price and performance considerations for an elastic pool.
Troubleshoot, diagnose, and prevent SQL connection
errors and transient errors for SQL Database
8/1/2018 • 15 minutes to read • Edit Online
This article describes how to prevent, troubleshoot, diagnose, and mitigate connection errors and transient errors
that your client application encounters when it interacts with Azure SQL Database. Learn how to configure retry
logic, build the connection string, and adjust other connection settings.
7. Edit the connection string as needed. i.e. Insert your password into the connection string, or remove
"@<servername>" from the username if the username or server name are too long.
8. In one format or another, paste the connection string information into your client program code.
For more information, see Connection strings and configuration files.
Connection: IP address
You must configure the SQL Database server to accept communication from the IP address of the computer that
hosts your client program. To set up this configuration, edit the firewall settings through the Azure portal.
If you forget to configure the IP address, your program fails with a handy error message that states the necessary
IP address.
1. Sign in to the Azure portal.
2. In the list on the left, select All services.
3. Scroll and select SQL servers.
4. In the filter text box, start typing the name of your server. Your row is displayed.
5. Select the row for your server. A blade for your server is displayed.
6. On your server blade, select Settings.
7. Select Firewall.
8. Select Add Client IP. Type a name for your new rule in the first text box.
9. Type in the low and high IP address values for the range you want to enable.
It can be handy to have the low value end with .0 and the high value end with .255.
10. Select Save.
For more information, see Configure firewall settings on SQL Database.
Connection: Ports
Typically, you need to ensure that only port 1433 is open for outbound communication on the computer that hosts
your client program.
For example, when your client program is hosted on a Windows computer, you can use Windows Firewall on the
host to open port 1433.
1. Open Control Panel.
2. Select All Control Panel Items > Windows Firewall > Advanced Settings > Outbound Rules >
Actions > New Rule.
If your client program is hosted on an Azure virtual machine (VM ), read Ports beyond 1433 for ADO.NET 4.5 and
SQL Database.
For background information about configuration of ports and IP addresses, see Azure SQL Database firewall.
Connection: ADO.NET 4.6.2 or later
If your program uses ADO.NET classes like System.Data.SqlClient.SqlConnection to connect to SQL
Database, we recommend that you use .NET Framework version 4.6.2 or later.
Starting with ADO.NET 4.6.2:
The connection open attempt to be retried immediately for Azure SQL databases, thereby improving the
performance of cloud-enabled apps.
Starting with ADO.NET 4.6.1:
For SQL Database, reliability is improved when you open a connection by using the SqlConnection.Open
method. The Open method now incorporates best-effort retry mechanisms in response to transient faults for
certain errors within the connection timeout period.
Connection pooling is supported, which includes an efficient verification that the connection object it gives your
program is functioning.
When you use a connection object from a connection pool, we recommend that your program temporarily closes
the connection when it's not immediately in use. It's not expensive to reopen a connection, but it is to create a new
connection.
If you use ADO.NET 4.0 or earlier, we recommend that you upgrade to the latest ADO.NET. As of August 2018,
you can download ADO.NET 4.6.2.
Diagnostics
Diagnostics: Test whether utilities can connect
If your program fails to connect to SQL Database, one diagnostic option is to try to connect with a utility program.
Ideally, the utility connects by using the same library that your program uses.
On any Windows computer, you can try these utilities:
SQL Server Management Studio (ssms.exe), which connects by using ADO.NET
sqlcmd.exe, which connects by using ODBC
After your program is connected, test whether a short SQL SELECT query works.
Diagnostics: Check the open ports
If you suspect that connection attempts fail due to port issues, you can run a utility on your computer that reports
on the port configurations.
On Linux, the following utilities might be helpful:
netstat -nap
nmap -sS -O 127.0.0.1
Change the example value to be your IP address.
On Windows, the PortQry.exe utility might be helpful. Here's an example execution that queried the port situation
on a SQL Database server and that was run on a laptop computer:
[C:\Users\johndoe\]
>> portqry.exe -n johndoesvr9.database.windows.net -p tcp -e 1433
querying...
TCP port 1433 (ms-sql-s service): LISTENING
[C:\Users\johndoe\]
>>
SELECT
object_name
,CAST(f.event_data as XML).value
('(/event/@timestamp)[1]', 'datetime2') AS [timestamp]
,CAST(f.event_data as XML).value
('(/event/data[@name="error"]/value)[1]', 'int') AS [error]
,CAST(f.event_data as XML).value
('(/event/data[@name="state"]/value)[1]', 'int') AS [state]
,CAST(f.event_data as XML).value
('(/event/data[@name="is_success"]/value)[1]', 'bit') AS [is_success]
,CAST(f.event_data as XML).value
('(/event/data[@name="database_name"]/value)[1]', 'sysname') AS [database_name]
FROM
sys.fn_xe_telemetry_blob_target_read_file('el', null, null, null) AS f
WHERE
object_name != 'login_event' -- Login events are numerous.
and
'2015-06-21' < CAST(f.event_data as XML).value
('(/event/@timestamp)[1]', 'datetime2')
ORDER BY
[timestamp] DESC
;
Enterprise Library 6
Enterprise Library 6 (EntLib60) is a framework of .NET classes that helps you implement robust clients of cloud
services, one of which is the SQL Database service. To locate topics dedicated to each area in which EntLib60 can
assist, see Enterprise Library 6 - April 2013.
Retry logic for handling transient errors is one area in which EntLib60 can assist. For more information, see 4 -
Perseverance, secret of all triumphs: Use the Transient Fault Handling Application Block.
NOTE
The source code for EntLib60 is available for public download from the Download Center. Microsoft has no plans to make
further feature updates or maintenance updates to EntLib.
return true;
case 10928:
case 10929:
case 10053:
case 10054:
case 10060:
case 40197:
case 40540:
case 40613:
case 40143:
case 233:
case 64:
// DBNETLIB Error Code: 20
// The instance of SQL Server you attempted to connect to
// does not support encryption.
case (int)ProcessNetLibErrorCode.EncryptionNotSupported:
return true;
}
}
}
else if (ex is TimeoutException)
{
return true;
}
else
{
EntityException entityException;
if ((entityException = ex as EntityException) != null)
{
return this.IsTransient(entityException.InnerException);
}
}
}
return false;
}
Next steps
For more information on troubleshooting other common SQL Database connection issues, see Troubleshoot
connection issues to Azure SQL Database.
Connection libraries for SQL Database and SQL Server
SQL Server connection pooling (ADO.NET)
Retrying is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of
adding retry behavior to just about anything.
DNS alias for Azure SQL Database
5/23/2018 • 4 minutes to read • Edit Online
Azure SQL Database has a Domain Name System (DNS ) server. PowerShell and REST APIs accept calls to create
and manage DNS aliases for your SQL Database server name.
A DNS alias can be used in place of the Azure SQL Database server name. Client programs can use the alias in
their connection strings. The DNS alias provides a translation layer that can redirect your client programs to
different servers. This layer spares you the difficulties of having to find and edit all the clients and their connection
strings.
Common uses for a DNS alias include the following cases:
Create an easy to remember name for an Azure SQL Server.
During initial development, your alias can refer to a test SQL Database server. When the application goes live,
you can modify the alias to refer to the production server. The transition from test to production does not
require any modification to the configurations several clients that connect to the database server.
Suppose the only database in your application is moved to another SQL Database server. Here you can modify
the alias without having to modify the configurations of several clients.
Domain Name System (DNS) of the Internet
The Internet relies on the DNS. The DNS translates your friendly names into the name of your Azure SQL
Database server.
Related resources
Overview of business continuity with Azure SQL Database, including disaster recovery.
Next steps
PowerShell for DNS Alias to Azure SQL Database
PowerShell for DNS Alias to Azure SQL Database
5/23/2018 • 3 minutes to read • Edit Online
This article provides a PowerShell script that demonstrates how you can manage a DNS alias for Azure SQL
Database. The script runs the following cmdlets which takes the following actions:
The cmdlets used in the code example are the following:
New -AzureRMSqlServerDNSAlias: Creates a new DNS alias in the Azure SQL Database service system. The
alias refers to Azure SQL Database server 1.
Get-AzureRMSqlServerDNSAlias: Get and list all the DNS aliases that are assigned to SQL DB server 1.
Set-AzureRMSqlServerDNSAlias: Modifies the server name that the alias is configured to refer to, from server
1 to SQL DB server 2.
Remove-AzureRMSqlServerDNSAlias: Remove the DNS alias from SQL DB server 2, by using the name of
the alias.
The preceding PowerShell cmdlets were added to the AzureRm.Sql module starting with version 5.1.1.
DNS alias in connection string
To connect a particular Azure SQL Database server, a client such as SQL Server Management Studio (SSMS ) can
provide the DNS alias name instead of the true server name. In the following example server string, the alias any-
unique-alias-name replaces the first dot-delimited node in the four node server string:
Example server string: any-unique-alias-name.database.windows.net .
Prerequisites
If you want to run the demo PowerShell script given in this article, the following prerequisites apply:
An Azure subscription and account. For a free trial, click https://azure.microsoft.com/free/.
Azure PowerShell module, with cmdlet New-AzureRMSqlServerDNSAlias.
To install or upgrade, see Install Azure PowerShell module.
Run Get-Module -ListAvailable AzureRM; in powershell_ise.exe, to find the version.
Two Azure SQL Database servers.
Code example
The following PowerShell code example starts by assign literal values to several variables. To run the code, you
must first edit all the placeholder values to match real values in your system. Or you can just study the code. And
the console output of the code is also provided.
################################################################
### Assign prerequisites. ###
################################################################
cls;
$SubscriptionName = '<EDIT-your-subscription-name>';
[string]$SubscriptionGuid_Get = '?'; # The script assigns this value, not you.
$SqlServerDnsAliasName = '<EDIT-any-unique-alias-name>';
$2ResourceGroupName = '<EDIT-rg-2>';
$2SqlServerName = '<EDIT-sql-2>';
$SubscriptionGuid_Get = Get-AzureRmSubscription `
-SubscriptionName $SubscriptionName;
################################################################
### Working with DNS aliasing for Azure SQL DB server. ###
################################################################
Write-Host '[2] Get and list all the DNS aliases that are assigned to SQL DB server 1.';
Get-AzureRMSqlServerDNSAlias `
–ResourceGroupName $1ResourceGroupName `
-ServerName $1SqlServerName;
Write-Host '[3] Move the DNS alias from 1 to SQL DB server 2.';
Set-AzureRMSqlServerDNSAlias `
–ResourceGroupName $2ResourceGroupName `
-NewServerName $2SqlServerName `
-ServerDNSAliasName $SqlServerDnsAliasName `
-OldServerResourceGroup $1ResourceGroupName `
-OldServerName $1SqlServerName `
-OldServerSubscriptionId $SubscriptionGuid_Get;
Write-Host '[4] Remove the DNS alias from SQL DB server 2.';
Remove-AzureRMSqlServerDNSAlias `
–ResourceGroupName $2ResourceGroupName `
-ServerName $2SqlServerName `
-ServerDNSAliasName $SqlServerDnsAliasName;
Environment : AzureCloud
Account : [email protected]
TenantId : 72f988bf-1111-1111-1111-111111111111
SubscriptionId : 45651c69-2222-2222-2222-222222222222
SubscriptionName : mysubscriptionname
CurrentStorageAccount :
[C:\windows\system32\]
>>
Next steps
For a full explanation of the DNS Alias feature for SQL Database, see DNS alias for Azure SQL Database.
Ports beyond 1433 for ADO.NET 4.5
9/14/2018 • 2 minutes to read • Edit Online
This topic describes the Azure SQL Database connection behavior for clients that use ADO.NET 4.5 or a later
version.
IMPORTANT
For information about connectivity architecture, see Azure SQL Database connectivity architecture.
Outside vs inside
For connections to Azure SQL Database, we must first ask whether your client program runs outside or inside the
Azure cloud boundary. The subsections discuss two common scenarios.
Outside: Client runs on your desktop computer
Port 1433 is the only port that must be open on your desktop computer that hosts your SQL Database client
application.
Inside: Client runs on Azure
When your client runs inside the Azure cloud boundary, it uses what we can call a direct route to interact with the
SQL Database server. After a connection is established, further interactions between the client and database
involve no Azure SQL Database Gateway.
The sequence is as follows:
1. ADO.NET 4.5 (or later) initiates a brief interaction with the Azure cloud, and receives a dynamically
identified port number.
The dynamically identified port number is in the range of 11000-11999 or 14000-14999.
2. ADO.NET then connects to the SQL Database server directly, with no middleware in between.
3. Queries are sent directly to the database, and results are returned directly to the client.
Ensure that the port ranges of 11000-11999 and 14000-14999 on your Azure client machine are left available for
ADO.NET 4.5 client interactions with SQL Database.
In particular, ports in the range must be free of any other outbound blockers.
On your Azure VM, the Windows Firewall with Advanced Security controls the port settings.
You can use the firewall's user interface to add a rule for which you specify the TCP protocol along with
a port range with the syntax like 11000-11999.
Version clarifications
This section clarifies the monikers that refer to product versions. It also lists some pairings of versions between
products.
ADO.NET
ADO.NET 4.0 supports the TDS 7.3 protocol, but not 7.4.
ADO.NET 4.5 and later supports the TDS 7.4 protocol.
ODBC
Microsoft SQL Server ODBC 11 or above
JDBC
Microsoft SQL Server JDBC 4.2 or above (JDBC 4.0 actually supports TDS 7.4 but does not implement
“redirection”)
Related links
ADO.NET 4.6 was released on July 20, 2015. A blog announcement from the .NET team is available here.
ADO.NET 4.5 was released on August 15, 2012. A blog announcement from the .NET team is available
here.
A blog post about ADO.NET 4.5.1 is available here.
Microsoft® ODBC Driver 17 for SQL Server® - Windows, Linux, & macOS
https://www.microsoft.com/en-us/download/details.aspx?id=56567
Connect to Azure SQL Database V12 via Redirection
https://blogs.msdn.microsoft.com/sqlcat/2016/09/08/connect-to-azure-sql-database-v12-via-redirection/
TDS protocol version list
SQL Database Development Overview
Azure SQL Database firewall
How to: Configure firewall settings on SQL Database
Connect to SQL Database using C and C++
5/23/2018 • 5 minutes to read • Edit Online
This post is aimed at C and C++ developers trying to connect to Azure SQL DB. It is broken down into sections so
you can jump to the section that best captures your interest.
At this point, you have configured your Azure SQL DB and are ready to connect from your C++ code.
Alternatively, you could create a DSN file using the wizard that is launched when no command arguments are
provided. We recommend that you try this option as well. You can use this DSN file for automation and protecting
your authentication settings:
Congratulations! You have now successfully connected to Azure SQL using C++ and ODBC on Windows. You can
continue reading to do the same for Linux platform as well.
sudo su
sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/mssql-ubuntu-test/ xenial main" >
/etc/apt/sources.list.d/mssqlpreview.list'
sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893
apt-get update
apt-get install msodbcsql
apt-get install unixodbc-dev-utf16 #this step is optional but recommended*
Launch Visual Studio. Under Tools -> Options -> Cross Platform -> Connection Manager, add a connection to
your Linux box:
After connection over SSH is established, create an Empty project (Linux) template:
You can then add a new C source file and replace it with this content. Using the ODBC APIs SQLAllocHandle,
SQLSetConnectAttr, and SQLDriverConnect, you should be able to initialize and establish a connection to your
database. Like with the Windows ODBC sample, you need to replace the SQLDriverConnect call with the details
from your database connection string parameters copied from the Azure portal previously.
retcode = SQLDriverConnect(
hdbc, NULL, "Driver=ODBC Driver 13 for SQL"
"Server;Server=<yourserver>;Uid=<yourusername>;Pwd=<"
"yourpassword>;database=<yourdatabase>",
SQL_NTS, outstr, sizeof(outstr), &outstrlen, SQL_DRIVER_NOPROMPT);
To launch your application, bring up the Linux Console from the Debug menu:
If your connection was successful, you should now see the current database name printed in the Linux Console:
Congratulations! You have successfully completed the tutorial and can now connect to your Azure SQL DB from
C++ on Windows and Linux platforms.
Next steps
Review the SQL Database Development Overview
More information on the ODBC API Reference
Additional resources
Design Patterns for Multi-tenant SaaS Applications with Azure SQL Database
Explore all the capabilities of SQL Database
Connect Excel to an Azure SQL database and create
a report
5/23/2018 • 4 minutes to read • Edit Online
Connect Excel to a SQL database in the cloud and import data and create tables and charts based on values in the
database. In this tutorial you will set up the connection between Excel and a database table, save the file that stores
data and the connection information for Excel, and then create a pivot chart from the database values.
You'll need a SQL database in Azure before you get started. If you don't have one, see Create your first SQL
database to get a database with sample data up and running in a few minutes. In this article, you'll import sample
data into Excel from that article, but you can follow similar steps with your own data.
You'll also need a copy of Excel. This article uses Microsoft Excel 2016.
4. In the SQL Server Database dialog box, select Database on the left side, and then enter in your User
Name and Password for the SQL database server you want to connect to. Select Connect to open the
Navigator.
TIP
Depending on your network environment, you may not be able to connect or you may lose the connection if the SQL
Database server doesn't allow traffic from your client IP address. Go to the Azure portal, click SQL servers, click your
server, click firewall under settings and add your client IP address. See How to configure firewall settings for details.
5. In the Navigator, select the database you want to work with from the list, select the tables or views you
want to work with (we chose vGetAllCategories), and then select Load to move the data from your SQL
Azure database to your excel spreadsheet.
Import the data into Excel and create a pivot chart
Now that you've established the connection, you have several different options with how to load the data. For
example, the following steps create a pivot chart based on the data found in your SQL Database.
1. Follow the steps in the previous section, but this time, instead of selecting Load, select Load to from the Load
drop-down.
2. Next, select how you want to view this data in your workbook. We chose PivotChart. You can also choose
to create a New worksheet or to Add this data to a Data Model. For more information on Data Models,
see Create a data model in Excel.
The worksheet now has an empty pivot table and chart.
3. Under PivotTable Fields, select all the check-boxes for the fields you want to view.
TIP
If you want to connect other Excel workbooks and worksheets to the database, select the Data tab, and select Recent
Sources to launch the Recent Sources dialog box. From there, choose the connection you created from the list, and then
click Open.
2. In the Data Connection Wizard, type in your server name and your SQL Database credentials. Select
Next.
a. Select the database that contains your data from the drop-down.
b. Select the table or view you're interested in. We chose vGetAllCategories.
c. Select Next.
3. Select the location of your file, the File Name, and the Friendly Name in the next screen of the Data
Connection Wizard. You can also choose to save the password in the file, though this can potentially expose
your data to unwanted access. Select Finish when ready.
4. Select how you want to import your data. We chose to do a PivotTable. You can also modify the properties
of the connection by select Properties. Select OK when ready. If you did not choose to save the password
with the file, then you will be prompted to enter your credentials.
5. Verify that your new connection has been saved by expanding the Data tab, and selecting Existing
Connections.
Next steps
Learn how to Connect to SQL Database with SQL Server Management Studio for advanced querying and
analysis.
Learn about the benefits of elastic pools.
Learn how to create a web application that connects to SQL Database on the back-end.
The Wingtip Tickets SaaS application
5/23/2018 • 3 minutes to read • Edit Online
The same Wingtip Tickets SaaS application is implemented in each of three samples. The app is a simple event
listing and ticketing SaaS app targeting small venues - theaters, clubs, etc. Each venue is a tenant of the app, and
has its own data: venue details, lists of events, customers, ticket orders, etc. The app, together with the
management scripts and tutorials, showcases an end-to-end SaaS scenario. This includes provisioning tenants,
monitoring and managing performance, schema management, and cross-tenant reporting and analytics.
Each sample includes the application code, plus management scripts and tutorials that explore a range of design
and management patterns. Each sample deploys in less that five minutes. All three can be deployed side-by-side
so you can compare the differences in design and management.
Next steps
Conceptual descriptions
A more detailed explanation of the application tenancy patterns is available at Multi-tenant SaaS database
tenancy patterns
Tutorials and code
Standalone app per tenant:
Tutorials for standalone app .
Code for standalone app, on GitHub.
Database per tenant:
Tutorials for database per tenant.
Code for database per tenant, on GitHub.
Sharded multi-tenant:
Tutorials for sharded multi-tenant.
Code for sharded multi-tenant, on GitHub.
General guidance for working with Wingtip Tickets
sample SaaS apps
5/23/2018 • 3 minutes to read • Edit Online
This article contains general guidance for running the Wingtip Tickets sample SaaS applications that use Azure
SQL Database.
Next steps
Deploy the Wingtip Tickets SaaS Standalone Application
Deploy the Wingtip Tickets SaaS Database per Tenant application
Deploy the Wingtip Tickets SaaS Multi-tenant Database application
Deploy and explore a standalone single-tenant
application that uses Azure SQL Database
5/23/2018 • 4 minutes to read • Edit Online
In this tutorial, you deploy and explore the Wingtip Tickets SaaS sample application developed using the
standalone application, or app-per-tenant, pattern. The application is designed to showcase features of Azure SQL
Database that simplify enabling multi-tenant SaaS scenarios.
The standalone application or app-per-tenant pattern deploys an application instance for each tenant. Each
application is configured for a specific tenant and deployed in a separate Azure resource group. Multiple instances
of the application are provisioned to provide a multi-tenant solution. This pattern is best suited to smaller numbers,
of tenants where tenant isolation is a top priority. Azure has partner programs that allow resources to be deployed
into a tenant’s subscription and managed by a service provider on the tenant’s behalf.
In this tutorial, you will deploy three standalone applications for three tenants into your Azure subscription. You
have full access to explore and work with the individual application components.
The application source code and management scripts are available in the WingtipTicketsSaaS -StandaloneApp
GitHub repo.
In this tutorial you learn:
How to deploy the Wingtip Tickets SaaS Standalone Application.
Where to get the application source code, and management scripts.
About the servers and databases that make up the app.
Additional tutorials will be released. They will allow you to explore a range of management scenarios based on this
application pattern.
Dogwood Dojo
It is best to use only lowercase letters, numbers, and hyphens in your resource names.
For Resource group, select Create new, and then provide a lowercase Name for the resource group.
wingtip-sa-<venueName>-<user> is the recommended pattern. For <venueName>, substitute the
venue name with no spaces. For <user>, substitute the user value from below. With this pattern,
resource group names might be wingtip -sa -contosoconcerthall-af1, wingtip -sa -dogwooddojo -af1,
wingtip -sa -fabrikamjazzclub -af1.
Select a Location from the drop-down list.
For User - We recommend a short user value, such as your initials plus a digit: for example, af1.
3. Deploy the application.
Click to agree to the terms and conditions.
Click Purchase.
4. Monitor the status of all three deployments by clicking Notifications (the bell icon to the right of the
search box). Deploying the apps takes around five minutes.
Additional resources
To learn about multi-tenant SaaS applications, see Design patterns for multi-tenant SaaS applications.
This article covers the provisioning and cataloging of new tenants using the standalone app per tenant SaaS
pattern. This article has two major parts:
Conceptual discussion of provisioning and cataloging new tenants
A tutorial that highlights sample PowerShell code that accomplishes the provisioning and cataloging
The tutorial uses the Wingtip Tickets sample SaaS application, adapted to the standalone app per tenant
pattern.
When deploying an application for a tenant, the app and database are provisioned in a new resource group created
for the tenant. Using separate resource groups isolates each tenant's application resources and allows them to be
managed independently. Within each resource group, each application instance is configured to access its
corresponding database directly. This connection model contrasts with other patterns that use a catalog to broker
connections between the app and the database. And as there is no resource sharing, each tenant database must be
provisioned with sufficient resources to handle its peak load. This pattern tends to be used for SaaS applications
with fewer tenants, where there is a strong emphasis on tenant isolation and less emphasis on resource costs.
The tenant catalog holds a mapping between a tenant identifier and a tenant database, allowing an identifier to be
resolved to a server and database name. In the Wingtip SaaS app, the tenant identifier is computed as a hash of
the tenant name, although other schemes could be used. While standalone applications don't need the catalog to
manage connections, the catalog can be used to scope other actions to a set of tenant databases. For example,
Elastic Query can use the catalog to determine the set of databases across which queries are distributed for cross-
tenant reporting.
IMPORTANT
Do not edit the data in the catalog database or the local shard map in the tenant databases directly. Direct updates are not
supported due to the high risk of data corruption. Instead, edit the mapping data by using EDCL APIs only.
Tenant provisioning
Each tenant requires a new Azure resource group, which must be created before resources can be provisioned
within it. Once the resource group exists, an Azure Resource Management template can be used to deploy the
application components and the database, and then configure the database connection. To initialize the database
schema, the template can import a bacpac file. Alternatively, the database can be created as a copy of a ‘template’
database. The database is then further updated with initial venue data and registered in the catalog.
Tutorial
In this tutorial you learn how to:
Provision a catalog
Register the sample tenant databases that you deployed earlier in the catalog
Provision an additional tenant and register it in the catalog
An Azure Resource Manager template is used to deploy and configure the application, create the tenant database,
and then import a bacpac file to initialize it. The import request may be queued for several minutes before it is
actioned.
At the end of this tutorial, you have a set of standalone tenant applications, with each database registered in the
catalog.
Prerequisites
To complete this tutorial, make sure the following prerequisites are completed:
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
The three sample tenant apps are deployed. To deploy these apps in less than five minutes, see Deploy and
explore the Wingtip Tickets SaaS Standalone Application pattern.
As an alternative to using the Data Explorer you can connect to the database from SQL Server Management
Studio. To do this, connect to the server wingtip-
Note that you should not edit data directly in the catalog - always use the shard management APIs.
You can then inspect the new resources created in the Azure portal.
To stop billing, delete resource groups
When you have finished exploring the sample, delete all the resource groups you created to stop the associated
billing.
Additional resources
To learn more about multi-tenant SaaS database applications, see Design patterns for multi-tenant SaaS
applications.
Next steps
In this tutorial you learned:
How to deploy the Wingtip Tickets SaaS Standalone Application.
About the servers and databases that make up the app.
How to delete sample resources to stop related billing.
You can explore how the catalog is used to support various cross-tenant scenarios using the database-per-tenant
version of the Wingtip Tickets SaaS application.
Introduction to a multitenant SaaS app that uses the
database-per-tenant pattern with SQL Database
5/23/2018 • 2 minutes to read • Edit Online
The Wingtip SaaS application is a sample multitenant app. The app uses the database-per-tenant SaaS application
pattern to service multiple tenants. The app showcases features of Azure SQL Database that enable SaaS
scenarios by using several SaaS design and management patterns. To quickly get up and running, the Wingtip
SaaS app deploys in less than five minutes.
Application source code and management scripts are available in the WingtipTicketsSaaS -DbPerTenant GitHub
repo. Before you start, see the general guidance for steps to download and unblock the Wingtip Tickets
management scripts.
Application architecture
The Wingtip SaaS app uses the database-per-tenant model. It uses SQL elastic pools to maximize efficiency. For
provisioning and mapping tenants to their data, a catalog database is used. The core Wingtip SaaS application uses
a pool with three sample tenants, plus the catalog database. The catalog and tenant servers have been provisioned
with DNS aliases. These aliases are used to maintain a reference to the active resources used by the Wingtip
application. These aliases are updated to point to recovery resources in the disaster recovery tutorials. Completing
many of the Wingtip SaaS tutorials results in add-ons to the initial deployment. Add-ons such as analytic
databases and cross-database schema management are introduced.
As you go through the tutorials and work with the app, focus on the SaaS patterns as they relate to the data tier. In
other words, focus on the data tier, and don't overanalyze the app itself. Understanding the implementation of
these SaaS patterns is key to implementing these patterns in your applications. Also consider any necessary
modifications for your specific business requirements.
TUTORIAL DESCRIPTION
Guidance and tips for the SQL Database multitenant SaaS app Download and run PowerShell scripts to prepare parts of the
example application.
Deploy and explore the Wingtip SaaS application Deploy and explore the Wingtip SaaS application with your
Azure subscription.
Provision and catalog tenants Learn how the application connects to tenants by using a
catalog database, and how the catalog maps tenants to their
data.
Monitor and manage performance Learn how to use monitoring features of SQL Database and
set alerts when performance thresholds are exceeded.
Monitor with Azure Log Analytics Learn how to use Log Analytics to monitor large amounts of
resources across multiple pools.
Restore a single tenant Learn how to restore a tenant database to a prior point in
time. Also learn how to restore to a parallel database, which
leaves the existing tenant database online.
Manage tenant database schema Learn how to update schema and update reference data
across all tenant databases.
Run cross-tenant distributed queries Create an ad-hoc analytics database, and run real-time
distributed queries across all tenants.
Run analytics on extracted tenant data Extract tenant data into an analytics database or data
warehouse for offline analytics queries.
Next steps
General guidance and tips when you deploy and use the Wingtip Tickets SaaS app example
Deploy the Wingtip SaaS application
Deploy and explore a multitenant SaaS app that
uses the database-per-tenant pattern with SQL
Database
8/2/2018 • 10 minutes to read • Edit Online
In this tutorial, you deploy and explore the Wingtip Tickets SaaS database-per-tenant application (Wingtip). The
app uses a database-per-tenant pattern to store the data of multiple tenants. The app is designed to showcase
features of Azure SQL Database that simplify how to enable SaaS scenarios.
Five minutes after you select Deploy to Azure, you have a multitenant SaaS application. The app includes a
SQL database that runs in the cloud. The app is deployed with three sample tenants, each with its own database.
All the databases are deployed into a SQL elastic pool. The app is deployed to your Azure subscription. You have
full access to explore and work with the individual components of the app. The application C# source code and
the management scripts are available in the WingtipTicketsSaaS -DbPerTenant GitHub repo.
In this tutorial you learn:
How to deploy the Wingtip SaaS application.
Where to get the application source code and management scripts.
About the servers, pools, and databases that make up the app.
How tenants are mapped to their data with the catalog.
How to provision a new tenant.
How to monitor tenant activity in the app.
A series of related tutorials offers to explore various SaaS design and management patterns. The tutorials build
beyond this initial deployment. When you use the tutorials, you can examine the provided scripts to see how the
different SaaS patterns are implemented. The scripts demonstrate how features of SQL Database simplify the
development of SaaS applications.
Prerequisites
To complete this tutorial, make sure Azure PowerShell is installed. For more information, see Get started with
Azure PowerShell.
IMPORTANT
Some authentication and server firewalls are intentionally unsecured for demonstration purposes. We recommend
that you create a new resource group. Don't use existing resource groups, servers, or pools. Don't use this
application, scripts, or any deployed resources for production. Delete this resource group when you're finished with
the application to stop related billing.
Resource group: Select Create new, and provide the unique name you chose earlier for the resource
group.
Location: Select a location from the drop-down list.
User: Use the user name value you chose earlier.
3. Deploy the application.
a. Select to agree to the terms and conditions.
b. Select Purchase.
4. To monitor deployment status, select Notifications (the bell icon to the right of the search box).
Deploying the Wingtip Tickets SaaS app takes approximately five minutes.
IMPORTANT
Executable contents (scripts and DLLs) might be blocked by Windows when .zip files are downloaded from an external
source and extracted. Follow the steps to unblock the .zip file before you extract the scripts. Unblocking makes sure the
scripts are allowed to run.
The tenant name is parsed from the URL by the events app.
The tenant name is used to create a key.
The key is used to access the catalog to obtain the location of the tenant's database.
The catalog is implemented by using shard map management.
The Events Hub uses extended metadata in the catalog to construct the list-of-events page URLs for each
tenant.
In a production environment, typically you create a CNAME DNS record to point a company internet domain to
the Traffic Manager DNS name.
Demo-LoadGenerator.ps1 actions
Demo -LoadGenerator.ps1 mimics an active workload of customer transactions. The following steps describe the
sequence of actions that Demo -LoadGenerator.ps1 initiates:
1. Demo -LoadGenerator.ps1 starts LoadGenerator.ps1 in the foreground.
Both .ps1 files are stored under the folders Learning Modules\Utilities\.
2. LoadGenerator.ps1 loops through all tenant databases in the catalog.
3. LoadGenerator.ps1 starts a background PowerShell job for each tenant database:
By default, the background jobs run for 120 minutes.
Each job causes a CPU -based load on one tenant database by executing sp_CpuLoadGenerator. The
intensity and duration of the load varies depending on $DemoScenario .
sp_CpuLoadGenerator loops around a SQL SELECT statement that causes a high CPU load. The time
interval between issues of the SELECT varies according to parameter values to create a controllable
CPU load. Load levels and intervals are randomized to simulate more realistic loads.
This .sql file is stored under WingtipTenantDB\dbo\StoredProcedures\.
4. If $OneTime = $false , the load generator starts the background jobs and then continues to run. Every 10
seconds, it monitors for any new tenants that are provisioned. If you set $OneTime = $true , the
LoadGenerator starts the background jobs and then stops running in the foreground. For this tutorial,
leave $OneTime = $false .
Use Ctrl-C or Stop Operation Ctrl-Break if you want to stop or restart the load generator.
If you leave the load generator running in the foreground, use another PowerShell ISE instance to run
other PowerShell scripts.
Before you continue with the next section, leave the load generator running in the job-invoking state.
NOTE
Many Wingtip SaaS scripts use $PSScriptRoot to browse folders to call functions in other scripts. This variable is
evaluated only when the full script is executed by pressing F5. Highlighting and running a selection with F8 can
result in errors. To run the scripts, press F5.
Additional resources
For more information, see additional tutorials that build on the Wingtip Tickets SaaS database-per-tenant
application.
To learn about elastic pools, see What is an Azure SQL elastic pool?.
To learn about elastic jobs, see Manage scaled-out cloud databases.
To learn about multitenant SaaS applications, see Design patterns for multitenant SaaS applications.
Next steps
In this tutorial you learned:
How to deploy the Wingtip Tickets SaaS application.
About the servers, pools, and databases that make up the app.
How tenants are mapped to their data with the catalog.
How to provision new tenants.
How to view pool utilization to monitor tenant activity.
How to delete sample resources to stop related billing.
Next, try the Provision and catalog tutorial.
Learn how to provision new tenants and register
them in the catalog
5/23/2018 • 9 minutes to read • Edit Online
In this tutorial, you learn how to provision and catalog SaaS patterns. You also learn how they're implemented in
the Wingtip Tickets SaaS database-per-tenant application. You create and initialize new tenant databases and
register them in the application's tenant catalog. The catalog is a database that maintains the mapping between
the SaaS application's many tenants and their data. The catalog plays an important role in directing application
and management requests to the correct database.
In this tutorial, you learn how to:
Provision a single new tenant.
Provision a batch of additional tenants.
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS database-per-tenant app is deployed. To deploy it in less than five minutes, see
Deploy and explore the Wingtip Tickets SaaS database-per-tenant application.
Azure PowerShell is installed. For more information, see Get started with Azure PowerShell.
IMPORTANT
The mapping data is accessible in the catalog database, but don't edit it. Edit mapping data by using Elastic Database Client
Library APIs only. Directly manipulating the mapping data risks corrupting the catalog and isn't supported.
Trace the script's execution by using the Debug menu options. Press F10 and F11 to step over or into the called
functions. For more information about debugging PowerShell scripts, see Tips on working with and debugging
PowerShell scripts.
You don't need to explicitly follow this workflow. It explains how to debug the script.
Import the CatalogAndDatabaseManagement.psm1 module. It provides a catalog and tenant-level
abstraction over the Shard Management functions. This module encapsulates much of the catalog pattern and
is worth exploring.
Import the SubscriptionManagement.psm1 module. It contains functions for signing in to Azure and
selecting the Azure subscription you want to work with.
Get configuration details. Step into Get-Configuration by using F11, and see how the app config is specified.
Resource names and other app-specific values are defined here. Don't change these values until you are
familiar with the scripts.
Get the catalog object. Step into Get-Catalog, which composes and returns a catalog object that's used in
the higher-level script. This function uses Shard Management functions that are imported from
AzureShardManagement.psm1. The catalog object is composed of the following elements:
$catalogServerFullyQualifiedName is constructed by using the standard stem plus your user name:
catalog -<user>.database.windows .net.
$catalogDatabaseName is retrieved from the config: tenantcatalog.
$shardMapManager object is initialized from the catalog database.
$shardMap object is initialized from the tenantcatalog shard map in the catalog database. A catalog
object is composed and returned. It's used in the higher-level script.
Calculate the new tenant key. A hash function is used to create the tenant key from the tenant name.
Check if the tenant key already exists. The catalog is checked to make sure the key is available.
The tenant database is provisioned with New-TenantDatabase. Use F11 to step into how the
database is provisioned by using an Azure Resource Manager template.
The database name is constructed from the tenant name to make it clear which shard belongs to which
tenant. You also can use other database naming conventions. A Resource Manager template creates a
tenant database by copying a template database (baseTenantDB ) on the catalog server. As an alternative,
you can create a database and initialize it by importing a bacpac. Or you can execute an initialization script
from a well-known location.
The Resource Manager template is in the …\Learning Modules\Common\ folder:
tenantdatabasecopytemplate.json
The tenant database is further initialized. The venue (tenant) name and the venue type are added. You
also can do other initialization here.
The tenant database is registered in the catalog. It's registered with Add -TenantDatabaseToCatalog
by using the tenant key. Use F11 to step into the details:
The catalog database is added to the shard map (the list of known databases).
The mapping that links the key value to the shard is created.
Additional metadata about the tenant (the venue's name) is added to the Tenants table in the catalog.
The Tenants table isn't part of the Shard Management schema, and it isn't installed by the EDCL. This
table illustrates how the catalog database can be extended to support additional application-specific
data.
After provisioning completes, execution returns to the original Demo -ProvisionAndCatalog script. The Events
page opens for the new tenant in the browser.
Provision a batch of tenants
This exercise provisions a batch of 17 tenants. We recommend that you provision this batch of tenants before
starting other Wingtip Tickets SaaS database-per-tenant tutorials. There are more than just a few databases to
work with.
1. In the PowerShell ISE, open ...\Learning Modules\ProvisionAndCatalog\Demo -ProvisionAndCatalog.ps1.
Change the $DemoScenario parameter to 3:
$DemoScenario = 3, Provision a batch of tenants.
2. To run the script, press F5.
The script deploys a batch of additional tenants. It uses an Azure Resource Manager template that controls the
batch and delegates provisioning of each database to a linked template. Using templates in this way allows Azure
Resource Manager to broker the provisioning process for your script. The templates provision databases in
parallel and handle retries, if needed. The script is idempotent, so if it fails or stops for any reason, run it again.
Verify the batch of tenants that successfully deployed
In the Azure portal, browse to your list of servers and open the tenants1 server. Select SQL databases, and
verify that the batch of 17 additional databases is now in the list.
Other provisioning patterns
Other provisioning patterns not included in this tutorial:
Pre-provisioning databases: The pre-provisioning pattern exploits the fact that databases in an elastic pool
don't add extra cost. Billing is for the elastic pool, not the databases. Idle databases consume no resources. By pre-
provisioning databases in a pool and allocating them when needed, you can reduce the time to add tenants. The
number of databases pre-provisioned can be adjusted as needed to keep a buffer suitable for the anticipated
provisioning rate.
Auto-provisioning: In the auto-provisioning pattern, a provisioning service provisions servers, pools, and
databases automatically, as needed. If you want, you can include pre-provisioning databases in elastic pools. If
databases are decommissioned and deleted, gaps in elastic pools can be filled by the provisioning service. Such a
service can be simple or complex, such as handling provisioning across multiple geographies and setting up geo-
replication for disaster recovery.
With the auto-provisioning pattern, a client application or script submits a provisioning request to a queue to be
processed by the provisioning service. It then polls the service to determine completion. If pre-provisioning is
used, requests are handled quickly. The service provisions a replacement database in the background.
Next steps
In this tutorial you learned how to:
Provision a single new tenant.
Provision a batch of additional tenants.
Step into the details of provisioning tenants and registering them into the catalog.
Try the Performance monitoring tutorial.
Additional resources
Additional tutorials that build on the Wingtip Tickets SaaS database-per-tenant application
Elastic database client library
Debug scripts in the Windows PowerShell ISE
Monitor and manage performance of Azure SQL
databases and pools in a multi-tenant SaaS app
6/25/2018 • 15 minutes to read • Edit Online
In this tutorial, several key performance management scenarios used in SaaS applications are explored. Using a
load generator to simulate activity across all tenant databases, the built-in monitoring and alerting features of
SQL Database and elastic pools are demonstrated.
The Wingtip Tickets SaaS Database Per Tenant app uses a single-tenant data model, where each venue (tenant)
has their own database. Like many SaaS applications, the anticipated tenant workload pattern is unpredictable and
sporadic. In other words, ticket sales may occur at any time. To take advantage of this typical database usage
pattern, tenant databases are deployed into elastic database pools. Elastic pools optimize the cost of a solution by
sharing resources across many databases. With this type of pattern, it's important to monitor database and pool
resource usage to ensure that loads are reasonably balanced across pools. You also need to ensure that individual
databases have adequate resources, and that pools are not hitting their eDTU limits. This tutorial explores ways to
monitor and manage databases and pools, and how to take corrective action in response to variations in workload.
In this tutorial you learn how to:
Simulate usage on the tenant databases by running a provided load generator
Monitor the tenant databases as they respond to the increase in load
Scale up the Elastic pool in response to the increased database load
Provision a second Elastic pool to load balance database activity
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS Database Per Tenant app is deployed. To deploy in less than five minutes, see Deploy
and explore the Wingtip Tickets SaaS Database Per Tenant application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
Get the Wingtip Tickets SaaS Database Per Tenant application scripts
The Wingtip Tickets SaaS Multi-tenant Database scripts and application source code are available in the
WingtipTicketsSaaS -DbPerTenant GitHub repo. Check out the general guidance for steps to download and
unblock the Wingtip Tickets SaaS scripts.
DEMO SCENARIO
The load generator applies a synthetic CPU -only load to every tenant database. The generator starts a job for each
tenant database, which calls a stored procedure periodically that generates the load. The load levels (in eDTUs),
duration, and intervals are varied across all databases, simulating unpredictable tenant activity.
1. In the PowerShell ISE, open …\Learning Modules\Performance Monitoring and Management\Demo -
PerformanceMonitoringAndManagement.ps1. Keep this script open as you'll run several scenarios during this
tutorial.
2. Set $DemoScenario = 2, Generate normal intensity load.
3. Press F5 to apply a load to all your tenant databases.
Wingtip Tickets SaaS Database Per Tenant is a SaaS app, and the real-world load on a SaaS app is typically
sporadic and unpredictable. To simulate this, the load generator produces a randomized load distributed across all
tenants. Several minutes are needed for the load pattern to emerge, so run the load generator for 3-5 minutes
before attempting to monitor the load in the following sections.
IMPORTANT
The load generator is running as a series of jobs in your local PowerShell session. Keep the Demo-
PerformanceMonitoringAndManagement.ps1 tab open! If you close the tab, or suspend your machine, the load generator
stops. The load generator remains in a job-invoking state where it generates load on any new tenants that are provisioned
after the generator is started. Use Ctrl-C to stop invoking new jobs and exit the script. The load generator will continue to
run, but only on existing tenants.
Because there are additional databases in the pool beyond the top five, the pool utilization shows activity that is
not reflected in the top five databases chart. For additional details, click Database Resource Utilization:
Set performance alerts on the pool
Set an alert on the pool that triggers on >75% utilization as follows:
1. Open Pool1 (on the tenants1 -dpt-<user> server) in the Azure portal.
2. Click Alert Rules, and then click + Add alert:
3. Provide a name, such as High DTU,
4. Set the following values:
Metric = eDTU percentage
Condition = greater than
Threshold = 75
Period = Over the last 30 minutes
5. Add an email address to the Additional administrator email(s) box and click OK.
Scale up a busy pool
If the aggregate load level increases on a pool to the point that it maxes out the pool and reaches 100% eDTU
usage, then individual database performance is affected, potentially slowing query response times for all
databases in the pool.
Short-term, consider scaling up the pool to provide additional resources, or removing databases from the pool
(moving them to other pools, or out of the pool to a stand-alone service tier).
Longer term, consider optimizing queries or index usage to improve database performance. Depending on the
application's sensitivity to performance issues its best practice to scale a pool up before it reaches 100% eDTU
usage. Use an alert to warn you in advance.
You can simulate a busy pool by increasing the load produced by the generator. Causing the databases to burst
more frequently, and for longer, increasing the aggregate load on the pool without changing the requirements of
the individual databases. Scaling up the pool is easily done in the portal or from PowerShell. This exercise uses the
portal.
1. Set $DemoScenario = 3, Generate load with longer and more frequent bursts per database to increase the
intensity of the aggregate load on the pool without changing the peak load required by each database.
2. Press F5 to apply a load to all your tenant databases.
3. Go to Pool1 in the Azure portal.
Monitor the increased pool eDTU usage on the upper chart. It takes a few minutes for the new higher load to kick
in, but you should quickly see the pool start to hit max utilization, and as the load steadies into the new pattern, it
rapidly overloads the pool.
1. To scale up the pool, click Configure pool at the top of the Pool1 page.
2. Adjust the Pool eDTU setting to 100. Changing the pool eDTU does not change the per-database settings
(which is still 50 eDTU max per database). You can see the per-database settings on the right side of the
Configure pool page.
3. Click Save to submit the request to scale the pool.
Go back to Pool1 > Overview to view the monitoring charts. Monitor the effect of providing the pool with more
resources (although with few databases and a randomized load it’s not always easy to see conclusively until you
run for some time). While you are looking at the charts bear in mind that 100% on the upper chart now
represents 100 eDTUs, while on the lower chart 100% is still 50 eDTUs as the per-database max is still 50 eDTUs.
Databases remain online and fully available throughout the process. At the last moment as each database is ready
to be enabled with the new pool eDTU, any active connections are broken. Application code should always be
written to retry dropped connections, and so will reconnect to the database in the scaled-up pool.
Next steps
In this tutorial you learn how to:
Simulate usage on the tenant databases by running a provided load generator
Monitor the tenant databases as they respond to the increase in load
Scale up the Elastic pool in response to the increased database load
Provision a second Elastic pool to load balance the database activity
Restore a single tenant tutorial
Additional resources
Additional tutorials that build upon the Wingtip Tickets SaaS Database Per Tenant application deployment
SQL Elastic pools
Azure automation
Log Analytics - Setting up and using Log Analytics tutorial
Set up and use Log Analytics with a multitenant SQL
Database SaaS app
8/2/2018 • 5 minutes to read • Edit Online
In this tutorial, you set up and use Azure Log Analytics to monitor elastic pools and databases. This tutorial builds
on the Performance monitoring and management tutorial. It shows how to use Log Analytics to augment the
monitoring and alerting provided in the Azure portal. Log Analytics supports monitoring thousands of elastic
pools and hundreds of thousands of databases. Log Analytics provides a single monitoring solution, which can
integrate monitoring of different applications and Azure services across multiple Azure subscriptions.
In this tutorial you learn how to:
Install and configure Log Analytics.
Use Log Analytics to monitor pools and databases.
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS database-per-tenant app is deployed. To deploy in less than five minutes, see Deploy
and explore the Wingtip Tickets SaaS database-per-tenant application.
Azure PowerShell is installed. For more information, see Get started with Azure PowerShell.
See the Performance monitoring and management tutorial for a discussion of SaaS scenarios and patterns and
how they affect the requirements on a monitoring solution.
Monitor and manage database and elastic pool performance with Log
Analytics
For Azure SQL Database, monitoring and alerting is available on databases and pools in the Azure portal. This
built-in monitoring and alerting is convenient, but it's also resource-specific. That means it's less well suited to
monitor large installations or provide a unified view across resources and subscriptions.
For high-volume scenarios, you can use Log Analytics for monitoring and alerting. Log Analytics is a separate
Azure service that enables analytics over diagnostic logs and telemetry that's gathered in a workspace from
potentially many services. Log Analytics provides a built-in query language and data visualization tools that allow
operational data analytics. The SQL Analytics solution provides several predefined elastic pool and database
monitoring and alerting views and queries. Log Analytics also provides a custom view designer.
Log Analytics workspaces and analytics solutions open in the Azure portal and in Operations Management Suite.
The Azure portal is the newer access point, but it might be behind the Operations Management Suite portal in
some areas.
Create performance diagnostic data by simulating a workload on your tenants
1. In the PowerShell ISE, open ..\WingtipTicketsSaaS -MultiTenantDb -master\Learning Modules\Performance
Monitoring and Management\Demo -PerformanceMonitoringAndManagement.ps1. Keep this script open
because you might want to run several of the load generation scenarios during this tutorial.
2. If you haven't done so already, provision a batch of tenants to make the monitoring context more
interesting. This process takes a few minutes.
a. Set $DemoScenario = 1, Provision a batch of tenants.
b. To run the script and deploy an additional 17 tenants, press F5.
3. Now start the load generator to run a simulated load on all the tenants.
a. Set $DemoScenario = 2, Generate normal intensity load (approx. 30 DTU ).
b. To run the script, press F5.
Install and configure Log Analytics and the Azure SQL Analytics
solution
Log Analytics is a separate service that must be configured. Log Analytics collects log data, telemetry, and metrics
in a Log Analytics workspace. Just like other resources in Azure, a Log Analytics workspace must be created. The
workspace doesn't need to be created in the same resource group as the applications it monitors. Doing so often
makes the most sense though. For the Wingtip Tickets app, use a single resource group to make sure the
workspace is deleted with the application.
1. In the PowerShell ISE, open ..\WingtipTicketsSaaS -MultiTenantDb -master\Learning Modules\Performance
Monitoring and Management\Log Analytics\Demo -LogAnalytics.ps1.
2. To run the script, press F5.
Now you can open Log Analytics in the Azure portal or the Operations Management Suite portal. It takes a few
minutes to collect telemetry in the Log Analytics workspace and to make it visible. The longer you leave the
system gathering diagnostic data, the more interesting the experience is.
Use Log Analytics and the SQL Analytics solution to monitor pools and
databases
In this exercise, open Log Analytics and the Operations Management Suite portal to look at the telemetry
gathered for the databases and pools.
1. Browse to the Azure portal. Select All services to open Log Analytics. Then search for Log Analytics.
7. Change the filter setting to modify the time range. For this tutorial, select Last 1 hour.
8. Select a single database to explore the query usage and metrics for that database.
A page opens that shows the pools and databases on the server.
11. Select a pool. On the pool page that opens, scroll to the right to see the pool metrics.
12. Back in the Log Analytics workspace, select OMS Portal to open the workspace there.
In the Operations Management Suite portal, you can explore the log and metric data in the workspace further.
Monitoring and alerting in Log Analytics are based on queries over the data in the workspace, unlike the alerting
defined on each resource in the Azure portal. By basing alerts on queries, you can define a single alert that looks
over all databases, rather than defining one per database. Queries are limited only by the data available in the
workspace.
For more information on how to use Log Analytics to query and set alerts, see Work with alert rules in Log
Analytics.
Log Analytics for SQL Database charges based on the data volume in the workspace. In this tutorial, you created a
free workspace, which is limited to 500 MB per day. After that limit is reached, data is no longer added to the
workspace.
Next steps
In this tutorial you learned how to:
Install and configure Log Analytics.
Use Log Analytics to monitor pools and databases.
Try the Tenant analytics tutorial.
Additional resources
Additional tutorials that build on the initial Wingtip Tickets SaaS database-per-tenant application deployment
Azure Log Analytics
Restore a single tenant with a database-per-tenant
SaaS application
5/23/2018 • 6 minutes to read • Edit Online
The database-per-tenant model makes it easy to restore a single tenant to a prior point in time without affecting
other tenants.
In this tutorial, you learn two data recovery patterns:
Restore a database into a parallel database (side by side).
Restore a database in place, replacing the existing database.
Restore into a parallel database This pattern can be used for tasks such as review, auditing,
and compliance to allow a tenant to inspect their data from
an earlier point. The tenant's current database remains online
and unchanged.
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip SaaS app is deployed. To deploy in less than five minutes, see Deploy and explore the Wingtip
SaaS application.
Azure PowerShell is installed. For details, see Get started with Azure PowerShell.
2. Scroll the list of events, and make a note of the last event in the list.
"Accidentally" delete the last event
1. In the PowerShell ISE, open ...\Learning Modules\Business Continuity and Disaster
Recovery\RestoreTenant\Demo -RestoreTenant.ps1, and set the following value:
$DemoScenario = 1, Delete last event (with no ticket sales).
2. Press F5 to run the script and delete the last event. The following confirmation message appears:
3. The Contoso events page opens. Scroll down and verify that the event is gone. If the event is still in the list,
select Refresh and verify that it's gone.
Restore a tenant database in parallel with the production database
This exercise restores the Contoso Concert Hall database to a point in time before the event was deleted. This
scenario assumes that you want to review the deleted data in a parallel database.
The Restore-TenantInParallel.ps1 script creates a parallel tenant database named ContosoConcertHall_old, with a
parallel catalog entry. This pattern of restore is best suited for recovering from a minor data loss. You also can use
this pattern if you need to review data for compliance or auditing purposes. It's the recommended approach when
you use geo-replication.
1. Complete the Simulate a tenant accidentally deleting data section.
2. In the PowerShell ISE, open ...\Learning Modules\Business Continuity and Disaster
Recovery\RestoreTenant\Demo -RestoreTenant.ps1.
3. Set $DemoScenario = 2, Restore tenant in parallel.
4. To run the script, press F5.
The script restores the tenant database to a point in time before you deleted the event. The database is restored to
a new database named ContosoConcertHall_old. The catalog metadata that exists in this restored database is
deleted, and then the database is added to the catalog by using a key constructed from the
ContosoConcertHall_old name.
The demo script opens the events page for this new tenant database in your browser. Note from the URL
http://events.wingtip-dpt.<user>.trafficmanager.net/contosoconcerthall_old that this page shows data from
the restored database where _old is added to the name.
Scroll the events listed in the browser to confirm that the event deleted in the previous section was restored.
Exposing the restored tenant as an additional tenant, with its own Events app, is unlikely to be how you provide a
tenant access to restored data. It serves to illustrate the restore pattern. Typically, you give read-only access to the
old data and retain the restored database for a defined period. In the sample, you can delete the restored tenant
entry after you're finished by running the Remove restored tenant scenario.
1. Set $DemoScenario = 4, Remove restored tenant.
2. To run the script, press F5.
3. The ContosoConcertHall_old entry is now deleted from the catalog. Close the events page for this tenant in
your browser.
Next steps
In this tutorial, you learned how to:
Restore a database into a parallel database (side by side).
Restore a database in place.
Try the Manage tenant database schema tutorial.
Additional resources
Additional tutorials that build on the Wingtip SaaS application
Overview of business continuity with Azure SQL Database
Learn about SQL Database backups
Manage schema in a SaaS application using the
database-per-tenant pattern with Azure SQL
Database
5/23/2018 • 6 minutes to read • Edit Online
As a database application evolves, changes inevitably need to be made to the database schema or reference data.
Database maintenance tasks are also needed periodically. Managing an application that uses the database per
tenant pattern requires that you apply these changes or maintenance tasks across a fleet of tenant databases.
This tutorial explores two scenarios - deploying reference data updates for all tenants, and rebuilding an index on
the table containing the reference data. The Elastic jobs feature is used to execute these actions on all tenant
databases, and on the template database used to create new tenant databases.
In this tutorial you learn how to:
Create a job agent
Cause T-SQL jobs to be run on all tenant databases
Update reference data in all tenant databases
Create an index on a table in all tenant databases
To complete this tutorial, make sure the following prerequisites are met:
The Wingtip Tickets SaaS Database Per Tenant app is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip Tickets SaaS database per tenant application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
The latest version of SQL Server Management Studio (SSMS ) is installed. Download and Install SSMS
NOTE
This tutorial uses features of the SQL Database service that are in a limited preview (Elastic Database jobs). If you wish to do
this tutorial, provide your subscription ID to [email protected] with subject=Elastic Jobs Preview. After you
receive confirmation that your subscription has been enabled, download and install the latest pre-release jobs cmdlets. This
preview is limited, so contact [email protected] for related questions or support.
NOTE
This tutorial uses features of the SQL Database service that are in a limited preview (Elastic Database jobs). If you wish to do
this tutorial, provide your subscription ID to [email protected] with subject=Elastic Jobs Preview. After you
receive confirmation that your subscription has been enabled, download and install the latest pre-release jobs cmdlets. This
preview is limited, so contact [email protected] for related questions or support.
Get the Wingtip Tickets SaaS database per tenant application scripts
The application source code and management scripts are available in the WingtipTicketsSaaS -DbPerTenant
GitHub repo. Check out the general guidance for steps to download and unblock the Wingtip Tickets SaaS scripts.
Next steps
In this tutorial you learned how to:
Create a job agent to run across T-SQL jobs multiple databases
Update reference data in all tenant databases
Create an index on a table in all tenant databases
Next, try the Ad-hoc reporting tutorial to explore running distributed queries across tenant databases.
Additional resources
Additional tutorials that build upon the Wingtip Tickets SaaS Database Per Tenant application deployment
Managing scaled-out cloud databases
Create and manage scaled-out cloud databases
Cross-tenant reporting using distributed queries
9/11/2018 • 8 minutes to read • Edit Online
In this tutorial, you run distributed queries across the entire set of tenant databases for reporting. These queries
can extract insights buried in the day-to-day operational data of the Wingtip Tickets SaaS tenants. To do this, you
deploy an additional reporting database to the catalog server and use Elastic Query to enable distributed queries.
In this tutorial you learn:
How to deploy an reporting database
How to run distributed queries across all tenant databases
How global views in each database can enable efficient querying across tenants
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS Database Per Tenant app is deployed. To deploy in less than five minutes, see Deploy
and explore the Wingtip Tickets SaaS Database Per Tenant application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
SQL Server Management Studio (SSMS ) is installed. To download and install SSMS, see Download SQL
Server Management Studio (SSMS ).
One opportunity with SaaS applications is to use the vast amount of tenant data stored in the cloud to gain
insights into the operation and usage of your application. These insights can guide feature development, usability
improvements, and other investments in your apps and services.
Accessing this data in a single multi-tenant database is easy, but not so easy when distributed at scale across
potentially thousands of databases. One approach is to use Elastic Query, which enables querying across a
distributed set of databases with common schema. These databases can be distributed across different resource
groups and subscriptions, but need to share a common login. Elastic Query uses a single head database in which
external tables are defined that mirror tables or views in the distributed (tenant) databases. Queries submitted to
this head database are compiled to produce a distributed query plan, with portions of the query pushed down to
the tenant databases as needed. Elastic Query uses the shard map in the catalog database to determine the
location of all tenant databases. Setup and query of the head database are straightforward using standard
Transact-SQL, and support querying from tools like Power BI and Excel.
By distributing queries across the tenant databases, Elastic Query provides immediate insight into live production
data. As Elastic Query pulls data from potentially many databases, query latency can be higher than equivalent
queries submitted to a single multi-tenant database. Design queries to minimize the data that is returned to the
head database. Elastic Query is often best suited for querying small amounts of real-time data, as opposed to
building frequently used or complex analytics queries or reports. If queries don't perform well, look at the
execution plan to see what part of the query is pushed down to the remote database and how much data is being
returned. Queries that require complex aggregation or analytical processing may be better handles by extracting
tenant data into a database or data warehouse optimized for analytics queries. This pattern is explained in the
tenant analytics tutorial.
Get the Wingtip Tickets SaaS Database Per Tenant application scripts
The Wingtip Tickets SaaS Multi-tenant Database scripts and application source code are available in the
WingtipTicketsSaaS -DbPerTenant GitHub repo. Check out the general guidance for steps to download and
unblock the Wingtip Tickets SaaS scripts.
-- Notice the plural name 'Venues'. This view projects a VenueId column.
SELECT * FROM Venues
-- This view projects the VenueId retrieved from the Venues table.
SELECT * FROM VenueEvents
In these views, the VenueId is computed as a hash of the Venue name, but any approach could be used to
introduce a unique value. This approach is similar to the way the tenant key is computed for use in the catalog.
To examine the definition of the Venues view:
1. In Object Explorer, expand contosoconcerthall > Views:
2. Right-click dbo.Venues.
3. Select Script View as > CREATE To > New Query Editor Window
Script any of the other Venue views to see how they add the VenueId.
With the catalog database as the external data source, queries are distributed to all databases registered in
the catalog at the time the query runs. As server names are different for each deployment, this script gets
the location of the catalog database from the current server (@@servername) where the script is executed.
The external tables that reference the global views described in the previous section, and defined with
DISTRIBUTION = SHARDED (VenueId). Because each VenueId maps to a single database, this
improves performance for many scenarios as shown in the next section.
The local table VenueTypes that is created and populated. This reference data table is common in all tenant
databases, so it can be represented here as a local table and populated with the common data. For some
queries, having this table defined in the head database can reduce the amount of data that needs to be
moved to the head database.
If you include reference tables in this manner, be sure to update the table schema and data whenever you
update the tenant databases.
4. Press F5 to run the script and initialize the adhocreporting database.
Now you can run distributed queries, and gather insights across all tenants!
Next steps
In this tutorial you learned how to:
Run distributed queries across all tenant databases
Deploy a reporting database and define the schema required to run distributed queries.
Now try the Tenant Analytics tutorial to explore extracting data to a separate analytics database for more complex
analytics processing.
Additional resources
Additional tutorials that build upon the Wingtip Tickets SaaS Database Per Tenant application
Elastic Query
Cross-tenant analytics using extracted data
5/23/2018 • 13 minutes to read • Edit Online
In this tutorial, you walk through a complete analytics scenario. The scenario demonstrates how analytics can
enable businesses to make smart decisions. Using data extracted from each tenant database, you use analytics to
gain insights into tenant behavior and application usage. This scenario involves three steps:
1. Extract data from each tenant database and Load into an analytics store.
2. Transform the extracted data for analytics processing.
3. Use business intelligence tools to draw out useful insights, which can guide decision making.
In this tutorial you learn how to:
Create the tenant analytics store to extract the data into.
Use elastic jobs to extract data from each tenant database into the analytics store.
Optimize the extracted data (reorganize into a star-schema).
Query the analytics database.
Use Power BI for data visualization to highlight trends in tenant data and make recommendation for
improvements.
Finally, the analytics store is queried using PowerBI to highlight insights into tenant behavior and their use of the
Wingtip Tickets application. You run queries that:
Show the relative popularity of each venue
Highlight patterns in ticket sales for different events
Show the relative success of different venues in selling out their event
Understanding how each tenant is using the service is used to explore options for monetizing the service and
improving the service to help tenants be more successful. This tutorial provides basic examples of the kinds of
insights that can be gleaned from tenant data.
Setup
Prerequisites
To complete this tutorial, make sure the following prerequisites are met:
The Wingtip Tickets SaaS Database Per Tenant application is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip SaaS application
The Wingtip Tickets SaaS Database Per Tenant scripts and application source code are downloaded from
GitHub. See download instructions. Be sure to unblock the zip file before extracting its contents. Check out the
general guidance for steps to download and unblock the Wingtip Tickets SaaS scripts.
Power BI Desktop is installed. Download Power BI Desktop
The batch of additional tenants has been provisioned, see the Provision tenants tutorial.
A job account and job account database have been created. See the appropriate steps in the Schema
management tutorial.
Create data for the demo
In this tutorial, analysis is performed on ticket sales data. In the current step, you generate ticket data for all the
tenants. Later this data is extracted for analysis. Ensure you have provisioned the batch of tenants as described
earlier, so that you have a meaningful amount of data. A sufficiently large amount of data can expose a range of
different ticket purchasing patterns.
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics\Demo -
TenantAnalytics.ps1, and set the following value:
$DemoScenario = 1 Purchase tickets for events at all venues
2. Press F5 to run the script and create ticket purchasing history for every event in each venue. The script runs for
several minutes to generate tens of thousands of tickets.
Deploy the analytics store
Often there are numerous transactional databases that together hold all tenant data. You must aggregate the
tenant data from the many transactional databases into one analytics store. The aggregation enables efficient
query of the data. In this tutorial, an Azure SQL Database database is used to store the aggregated data.
In the following steps, you deploy the analytics store, which is called tenantanalytics. You also deploy predefined
tables that are populated later in the tutorial:
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics\Demo -
TenantAnalytics.ps1
2. Set the $DemoScenario variable in the script to match your choice of analytics store:
To use SQL database without column store, set $DemoScenario = 2
To use SQL database with column store, set $DemoScenario = 3
3. Press F5 to run the demo script (that calls the Deploy-TenantAnalytics.ps1 script) which creates the tenant
analytics store.
Now that you have deployed the application and filled it with interesting tenant data, use SQL Server
Management Studio (SSMS ) to connect tenants1-dpt-<User> and catalog-dpt-<User> servers using Login =
developer, Password = P@ssword1. See the introductory tutorial for more guidance.
Data extraction
Create target groups
Before proceeding, ensure you have deployed the job account and jobaccount database. In the next set of steps,
Elastic Jobs is used to extract data from each tenant database, and to store the data in the analytics store. Then the
second job shreds the data and stores it into tables in the star-schema. These two jobs run against two different
target groups, namely TenantGroup and AnalyticsGroup. The extract job runs against the TenantGroup, which
contains all the tenant databases. The shredding job runs against the AnalyticsGroup, which contains just the
analytics store. Create the target groups by using the following steps:
1. In SSMS, connect to the jobaccount database in catalog-dpt-<User>.
2. In SSMS, open …\Learning Modules\Operational Analytics\Tenant Analytics\ TargetGroups.sql
3. Modify the @User variable at the top of the script, replacing with the user value used when you deployed the
Wingtip SaaS app.
4. Press F5 to run the script that creates the two target groups.
Extract raw data from all tenants
Extensive data modifications might occur more frequently for ticket and customer data than for event and venue
data. Therefore, consider extracting ticket and customer data separately and more frequently than you extract
event and venue data. In this section, you define and schedule two separate jobs:
Extract ticket and customer data.
Extract event and venue data.
Each job extracts its data, and posts it into the analytics store. There a separate job shreds the extracted data into
the analytics star-schema.
1. In SSMS, connect to the jobaccount database in catalog-dpt-<User> server.
2. In SSMS, open ...\Learning Modules\Operational Analytics\Tenant Analytics\ExtractTickets.sql.
3. Modify @User at the top of the script, and replace with the user name used when you deployed the Wingtip
SaaS app
4. Press F5 to run the script that creates and runs the job that extracts tickets and customers data from each
tenant database. The job saves the data into the analytics store.
5. Query the TicketsRawData table in the tenantanalytics database, to ensure that the table is populated with
tickets information from all tenants.
Repeat the preceding steps, except this time replace \ExtractTickets.sql with \ExtractVenuesEvents.sql in step
2.
Successfully running the job populates the EventsRawData table in the analytics store with new events and
venues information from all tenants.
Data reorganization
Shred extracted data to populate star-schema tables
The next step is to shred the extracted raw data into a set of tables that are optimized for analytics queries. A star-
schema is used. A central fact table holds individual ticket sales records. Other tables are populated with related
data about venues, events, and customers. And there are time dimension tables.
In this section of the tutorial, you define and run a job that merges the extracted raw data with the data in the star-
schema tables. After the merge job is finished, the raw data is deleted, leaving the tables ready to be populated by
the next tenant data extract job.
1. In SSMS, connect to the jobaccount database in catalog-dpt-<User>.
2. In SSMS, open …\Learning Modules\Operational Analytics\Tenant Analytics\ShredRawExtractedData.sql.
3. Press F5 to run the script to define a job that calls the sp_ShredRawExtractedData stored procedure in the
analytics store.
4. Allow enough time for the job to run successfully.
Check the Lifecycle column of jobs.jobs_execution table for the status of job. Ensure that the job
Succeeded before proceeding. A successful run displays data similar to the following chart:
Data exploration
Visualize tenant data
The data in the star-schema table provides all the ticket sales data needed for your analysis. To make it easier to
see trends in large data sets, you need to visualize it graphically. In this section, you learn how to use Power BI to
manipulate and visualize the tenant data you have extracted and organized.
Use the following steps to connect to Power BI, and to import the views you created earlier:
1. Launch Power BI desktop.
2. From the Home ribbon, select Get Data, and select More… from the menu.
3. In the Get Data window, select Azure SQL Database.
4. In the database login window, enter your server name (catalog-dpt-<User>.database.windows.net). Select
Import for Data Connectivity Mode, and then click OK.
5. Select Database in the left pane, then enter user name = developer, and enter password = P@ssword1.
Click Connect.
6. In the Navigator pane, under the analytics database, select the star-schema tables: fact_Tickets,
dim_Events, dim_Venues, dim_Customers and dim_Dates. Then select Load.
Congratulations! You have successfully loaded the data into Power BI. Now you can start exploring interesting
visualizations to help gain insights into your tenants. Next you walk through how analytics can enable you to
provide data-driven recommendations to the Wingtip Tickets business team. The recommendations can help to
optimize the business model and customer experience.
You start by analyzing ticket sales data to see the variation in usage across the venues. Select the following
options in Power BI to plot a bar chart of the total number of tickets sold by each venue. Due to random variation
in the ticket generator, your results may be different.
The preceding plot confirms that the number of tickets sold by each venue varies. Venues that sell more tickets are
using your service more heavily than venues that sell fewer tickets. There may be an opportunity here to tailor
resource allocation according to different tenant needs.
You can further analyze the data to see how ticket sales vary over time. Select the following options in Power BI to
plot the total number of tickets sold each day for a period of 60 days.
The preceding chart displays that ticket sales spike for some venues. These spikes reinforce the idea that some
venues might be consuming system resources disproportionately. So far there is no obvious pattern in when the
spikes occur.
Next you want to further investigate the significance of these peak sale days. When do these peaks occur after
tickets go on sale? To plot tickets sold per day, select the following options in Power BI.
The preceding plot shows that some venues sell a lot of tickets on the first day of sale. As soon as tickets go on
sale at these venues, there seems to be a mad rush. This burst of activity by a few venues might impact the service
for other tenants.
You can drill into the data again to see if this mad rush is true for all events hosted by these venues. In previous
plots, you observed that Contoso Concert Hall sells a lot of tickets, and that Contoso also has a spike in ticket sales
on certain days. Play around with Power BI options to plot cumulative ticket sales for Contoso Concert Hall,
focusing on sale trends for each of its events. Do all events follow the same sale pattern?
The preceding plot for Contoso Concert Hall shows that the mad rush does not happen for all events. Play around
with the filter options to see sale trends for other venues.
The insights into ticket selling patterns might lead Wingtip Tickets to optimize their business model. Instead of
charging all tenants equally, perhaps Wingtip should introduce service tiers with different performance levels.
Larger venues that need to sell more tickets per day could be offered a higher tier with a higher service level
agreement (SLA). Those venues could have their databases placed in pool with higher per-database resource
limits. Each service tier could have an hourly sales allocation, with additional fees charged for exceeding the
allocation. Larger venues that have periodic bursts of sales would benefit from the higher tiers, and Wingtip
Tickets can monetize their service more efficiently.
Meanwhile, some Wingtip Tickets customers complain that they struggle to sell enough tickets to justify the
service cost. Perhaps in these insights there is an opportunity to boost ticket sales for underperforming venues.
Higher sales would increase the perceived value of the service. Right click fact_Tickets and select New measure.
Enter the following expression for the new measure called AverageTicketsSold:
AverageTicketsSold = DIVIDE(DIVIDE(COUNTROWS(fact_Tickets),DISTINCT(dim_Venues[VenueCapacity]))*100,
COUNTROWS(dim_Events))
Select the following visualization options to plot the percentage tickets sold by each venue to determine their
relative success.
The preceding plot shows that even though most venues sell more than 80% of their tickets, some are struggling
to fill more than half the seats. Play around with the Values Well to select maximum or minimum percentage of
tickets sold for each venue.
Earlier you deepened your analysis to discover that ticket sales tend to follow predictable patterns. This discovery
might let Wingtip Tickets help underperforming venues boost ticket sales by recommending dynamic pricing. This
discover could reveal an opportunity to employ machine learning techniques to predict ticket sales for each event.
Predictions could also be made for the impact on revenue of offering discounts on ticket sales. Power BI
Embedded could be integrated into an event management application. The integration could help visualize
predicted sales and the effect of different discounts. The application could help devise an optimum discount to be
applied directly from the analytics display.
You have observed trends in tenant data from the WingTip application. You can contemplate other ways the app
can inform business decisions for SaaS application vendors. Vendors can better cater to the needs of their tenants.
Hopefully this tutorial has equipped you with tools necessary to perform analytics on tenant data to empower
your businesses to make data-driven decisions.
Next steps
In this tutorial, you learned how to:
Deployed a tenant analytics database with pre-defined star schema tables
Used elastic jobs to extract data from all the tenant database
Merge the extracted data into tables in a star-schema designed for analytics
Query an analytics database
Use Power BI for data visualization to observe trends in tenant data
Congratulations!
Additional resources
Additional tutorials that build upon the Wingtip SaaS application.
Elastic Jobs.
Explore SaaS analytics with Azure SQL Database,
SQL Data Warehouse, Data Factory, and Power BI
5/23/2018 • 16 minutes to read • Edit Online
In this tutorial, you walk through an end-to-end analytics scenario. The scenario demonstrates how analytics over
tenant data can empower software vendors to make smart decisions. Using data extracted from each tenant
database, you use analytics to gain insights into tenant behavior, including their use of the sample Wingtip Tickets
SaaS application. This scenario involves three steps:
1. Extract data from each tenant database into an analytics store, in this case, a SQL Data Warehouse.
2. Optimize the extracted data for analytics processing.
3. Use Business Intelligence tools to draw out useful insights, which can guide decision making.
In this tutorial you learn how to:
Create the tenant analytics store for loading.
Use Azure Data Factory (ADF ) to extract data from each tenant database into the analytics data warehouse.
Optimize the extracted data (reorganize into a star-schema).
Query the analytics data warehouse.
Use Power BI for data visualization to highlight trends in tenant data and make recommendation for
improvements.
Finally, the star-schema tables are queried. Query results are displayed visually using Power BI to highlight insights
into tenant behavior and their use of the application. With this star-schema, you run queries that expose:
Who is buying tickets and from which venue.
Patterns and trends in the sale of tickets.
The relative popularity of each venue.
This tutorial provides basic examples of insights that can be gleaned from the Wingtip Tickets data. Understanding
how each venue uses the service might cause the Wingtip Tickets vendor to think about different service plans
targeted at more or less active venues, for example.
Setup
Prerequisites
NOTE
This tutorial uses features of the Azure Data Factory that are currently in a limited preview (linked service parameterization). If
you wish to do this tutorial, provide your subscription ID here. We will send you a confirmation as soon as your subscription
has been enabled.
To complete this tutorial, make sure the following prerequisites are met:
The Wingtip Tickets SaaS Database Per Tenant application is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip SaaS application.
The Wingtip Tickets SaaS Database Per Tenant scripts and application source code are downloaded from
GitHub. See download instructions. Be sure to unblock the zip file before extracting its contents.
Power BI Desktop is installed. Download Power BI Desktop.
The batch of additional tenants has been provisioned, see the Provision tenants tutorial.
Create data for the demo
This tutorial explores analytics over ticket sales data. In this step, you generate ticket data for all the tenants. In a
later step, this data is extracted for analysis. Ensure you provisioned the batch of tenants (as described earlier) so
that you have enough data to expose a range of different ticket purchasing patterns.
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics DW\Demo -
TenantAnalyticsDW.ps1, and set the following value:
$DemoScenario = 1 Purchase tickets for events at all venues
2. Press F5 to run the script and create ticket purchasing history for all the venues. With 20 tenants, the script
generates tens of thousands of tickets and may take 10 minutes or more.
Deploy SQL Data Warehouse, Data Factory, and Blob Storage
In the Wingtip Tickets app, the tenants' transactional data is distributed over many databases. Azure Data Factory
(ADF ) is used to orchestrate the Extract, Load, and Transform (ELT) of this data into the data warehouse. To load
data into SQL Data Warehouse most efficiently, ADF extracts data into intermediate blob files and then uses
PolyBase to load the data into the data warehouse.
In this step, you deploy the additional resources used in the tutorial: a SQL Data Warehouse called tenantanalytics,
an Azure Data Factory called dbtodwload -<user>, and an Azure storage account called wingtipstaging<user>. The
storage account is used to temporarily hold extracted data files as blobs before they are loaded into the data
warehouse. This step also deploys the data warehouse schema and defines the ADF pipelines that orchestrate the
ELT process.
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics DW\Demo -
TenantAnalyticsDW.ps1 and set:
$DemoScenario = 2 Deploy tenant analytics data warehouse, blob storage, and data factory
2. Press F5 to run the demo script and deploy the Azure resources.
Now review the Azure resources you deployed:
Tenant databases and analytics store
Use SQL Server Management Studio (SSMS ) to connect to tenants1-dpt-<user> and catalog-dpt-<user>
servers. Replace <user> with the value used when you deployed the app. Use Login = developer and Password =
P@ssword1. See the introductory tutorial for more guidance.
In the Object Explorer:
1. Expand the tenants1 -dpt-<user> server.
2. Expand the Databases node, and see the list of tenant databases.
3. Expand the catalog -dpt-<user> server.
4. Verify that you see the analytics store containing the following objects:
a. Tables raw_Tickets, raw_Customers, raw_Events and raw_Venues hold raw extracted data from the
tenant databases.
b. The star-schema tables are fact_Tickets, dim_Customers, dim_Venues, dim_Events, and dim_Dates.
c. The stored procedure, sp_transformExtractedData is used to transform the data and load it into the
star-schema tables.
Blob storage
1. In the Azure Portal, navigate to the resource group that you used for deploying the application. Verify that a
storage account called wingtipstaging<user> has been added.
This section explores the data factory created. Follow the steps below to launch the data factory:
1. In the portal, click the data factory called dbtodwload-<user>.
2. Click Author & Monitor tile to launch the Data Factory designer in a separate tab.
In the overview page, switch to Author tab on the left panel and observe that there are three pipelines and three
datasets created.
The three nested pipelines are: SQLDBToDW, DBCopy, and TableCopy.
Pipeline 1 - SQLDBToDW looks up the names of the tenant databases stored in the Catalog database (table
name: [__ShardManagement].[ShardsGlobal]) and for each tenant database, executes the DBCopy pipeline. Upon
completion, the provided sp_TransformExtractedData stored procedure schema, is executed. This stored
procedure transforms the loaded data in the staging tables and populates the star-schema tables.
Pipeline 2 - DBCopy looks up the names of the source tables and columns from a configuration file stored in
blob storage. The TableCopy pipeline is then run for each of the four tables: TicketFacts, CustomerFacts,
EventFacts, and VenueFacts. The Foreach activity executes in parallel for all 20 databases. ADF allows a maximum
of 20 loop iterations to be run in parallel. Consider creating multiple pipelines for more databases.
Pipeline 3 - TableCopy uses row version numbers in SQL Database (rowversion) to identify rows that have been
changed or updated. This activity looks up the start and the end row version for extracting rows from the source
tables. The CopyTracker table stored in each tenant database tracks the last row extracted from each source table
in each run. New or changed rows are copied to the corresponding staging tables in the data warehouse:
raw_Tickets, raw_Customers, raw_Venues, and raw_Events. Finally the last row version is saved in the
CopyTracker table to be used as the initial row version for the next extraction.
There are also three parameterized linked services that link the data factory to the source SQL Databases, the
target SQL Data Warehouse, and the intermediate Blob storage. In the Author tab, click on Connections to
explore the linked services, as shown in the following image:
Corresponding to the three linked services, there are three datasets that refer to the data you use in the pipeline
activities as inputs or outputs. Explore each of the datasets to observe connections and parameters used. AzureBlob
points to the configuration file containing source and target tables and columns, as well as the tracker column in
each source.
Data warehouse pattern overview
SQL Data Warehouse is used as the analytics store to perform aggregation on the tenant data. In this sample,
PolyBase is used to load data into the SQL Data warehouse. Raw data is loaded into staging tables that have an
identity column to keep track of rows that have been transformed into the star-schema tables. The following image
shows the loading pattern:
Slowly Changing Dimension (SCD ) type 1 dimension tables are used in this example. Each dimension has a
surrogate key defined using an identity column. As a best practice, the date dimension table is pre-populated to
save time. For the other dimension tables, a CREATE TABLE AS SELECT... (CTAS ) statement is used to create a
temporary table containing the existing modified and non-modified rows, along with the surrogate keys. This is
done with IDENTITY_INSERT=ON. New rows are then inserted into the table with IDENTITY_INSERT=OFF. For
easy roll-back, the existing dimension table is renamed and the temporary table is renamed to become the new
dimension table. Before each run, the old dimension table is deleted.
Dimension tables are loaded before the fact table. This sequencing ensures that for each arriving fact, all referenced
dimensions already exist. As the facts are loaded, the business key for each corresponding dimension is matched
and the corresponding surrogate keys are added to each fact.
The final step of the transform deletes the staging data ready for the next execution of the pipeline.
Trigger the pipeline run
Follow the steps below to run the complete extract, load, and transform pipeline for all the tenant databases:
1. In the Author tab of the ADF user interface, select SQLDBToDW pipeline from the left pane.
2. Click Trigger and from the pulled down menu click Trigger Now. This action runs the pipeline immediately. In
a production scenario, you would define a timetable for running the pipeline to refresh the data on a schedule.
3. Connect to the data warehouse with SSMS and query the star-schema tables to verify that data was loaded in
these tables.
Once the pipeline has completed, the fact table holds ticket sales data for all venues and the dimension tables are
populated with the corresponding venues, events, and customers.
Data Exploration
Visualize tenant data
The data in the star-schema provides all the ticket sales data needed for your analysis. Visualizing data graphically
makes it easier to see trends in large data sets. In this section, you use Power BI to manipulate and visualize the
tenant data in the data warehouse.
Use the following steps to connect to Power BI, and to import the views you created earlier:
1. Launch Power BI desktop.
2. From the Home ribbon, select Get Data, and select More… from the menu.
3. In the Get Data window, select Azure SQL Database.
4. In the database login window, enter your server name (catalog-dpt-<User>.database.windows.net).
Select Import for Data Connectivity Mode, and then click OK.
5. Select Database in the left pane, then enter user name = developer, and enter password = P@ssword1.
Click Connect.
6. In the Navigator pane, under the analytics database, select the star-schema tables: fact_Tickets,
dim_Events, dim_Venues, dim_Customers and dim_Dates. Then select Load.
Congratulations! You successfully loaded the data into Power BI. Now explore interesting visualizations to gain
insights into your tenants. Let's walk through how analytics can provide some data-driven recommendations to the
Wingtip Tickets business team. The recommendations can help to optimize the business model and customer
experience.
Start by analyzing ticket sales data to see the variation in usage across the venues. Select the options shown in
Power BI to plot a bar chart of the total number of tickets sold by each venue. (Due to random variation in the
ticket generator, your results may be different.)
The preceding plot confirms that the number of tickets sold by each venue varies. Venues that sell more tickets are
using your service more heavily than venues that sell fewer tickets. There may be an opportunity here to tailor
resource allocation according to different tenant needs.
You can further analyze the data to see how ticket sales vary over time. Select the options shown in the following
image in Power BI to plot the total number of tickets sold each day for a period of 60 days.
The preceding chart shows that ticket sales spike for some venues. These spikes reinforce the idea that some
venues might be consuming system resources disproportionately. So far there is no obvious pattern in when the
spikes occur.
Next let's investigate the significance of these peak sale days. When do these peaks occur after tickets go on sale?
To plot tickets sold per day, select the options shown in the following image in Power BI.
This plot shows that some venues sell large numbers of tickets on the first day of sale. As soon as tickets go on sale
at these venues, there seems to be a mad rush. This burst of activity by a few venues might impact the service for
other tenants.
You can drill into the data again to see if this mad rush is true for all events hosted by these venues. In previous
plots, you saw that Contoso Concert Hall sells many tickets, and that Contoso also has a spike in ticket sales on
certain days. Play around with Power BI options to plot cumulative ticket sales for Contoso Concert Hall, focusing
on sale trends for each of its events. Do all events follow the same sale pattern? Try to produce a plot like the one
below.
This plot of cumulative ticket sales over time for Contoso Concert Hall for each event shows that the mad rush
does not happen for all events. Play around with the filter options to explore sale trends for other venues.
The insights into ticket selling patterns might lead Wingtip Tickets to optimize their business model. Instead of
charging all tenants equally, perhaps Wingtip should introduce service tiers with different performance levels.
Larger venues that need to sell more tickets per day could be offered a higher tier with a higher service level
agreement (SLA). Those venues could have their databases placed in pool with higher per-database resource limits.
Each service tier could have an hourly sales allocation, with additional fees charged for exceeding the allocation.
Larger venues that have periodic bursts of sales would benefit from the higher tiers, and Wingtip Tickets can
monetize their service more efficiently.
Meanwhile, some Wingtip Tickets customers complain that they struggle to sell enough tickets to justify the service
cost. Perhaps in these insights there is an opportunity to boost ticket sales for underperforming venues. Higher
sales would increase the perceived value of the service. Right click fact_Tickets and select New measure. Enter the
following expression for the new measure called AverageTicketsSold:
AverageTicketsSold = DIVIDE(DIVIDE(COUNTROWS(fact_Tickets),DISTINCT(dim_Venues[VenueCapacity]))*100,
COUNTROWS(dim_Events))
Select the following visualization options to plot the percentage tickets sold by each venue to determine their
relative success.
The plot above shows that even though most venues sell more than 80% of their tickets, some are struggling to fill
more than half their seats. Play around with the Values Well to select maximum or minimum percentage of tickets
sold for each venue.
Next steps
In this tutorial, you learned how to:
Deploy a SQL Data Warehouse populated with a star schema for tenant analytics.
Use Azure Data Factory to extract data from each tenant database into the analytics data warehouse.
Optimize the extracted data (reorganize into a star-schema).
Query the analytics data warehouse.
Use Power BI to visualize trends in data across all the tenants.
Congratulations!
Additional resources
Additional tutorials that build upon the Wingtip SaaS application.
Use geo-restore to recover a multitenant SaaS
application from database backups
5/23/2018 • 18 minutes to read • Edit Online
This tutorial explores a full disaster recovery scenario for a multitenant SaaS application implemented with the
database per tenant model. You use geo-restore to recover the catalog and tenant databases from automatically
maintained geo-redundant backups into an alternate recovery region. After the outage is resolved, you use geo-
replication to repatriate changed databases to their original region.
Geo-restore is the lowest-cost disaster recovery solution for Azure SQL Database. However, restoring from geo-
redundant backups can result in data loss of up to one hour. It can take considerable time, depending on the size of
each database.
NOTE
Recover applications with the lowest possible RPO and RTO by using geo-replication instead of geo-restore.
This tutorial explores both restore and repatriation workflows. You learn how to:
Sync database and elastic pool configuration info into the tenant catalog.
Set up a mirror image environment in a recovery region that includes application, servers, and pools.
Recover catalog and tenant databases by using geo-restore.
Use geo-replication to repatriate the tenant catalog and changed tenant databases after the outage is resolved.
Update the catalog as each database is restored (or repatriated) to track the current location of the active copy
of each tenant's database.
Ensure that the application and tenant database are always co-located in the same Azure region to reduce
latency.
Before you start this tutorial, complete the following prerequisites:
Deploy the Wingtip Tickets SaaS database per tenant app. To deploy in less than five minutes, see Deploy and
explore the Wingtip Tickets SaaS database per tenant application.
Install Azure PowerShell. For details, see Getting started with Azure PowerShell.
NOTE
The application is recovered into the paired region of the region in which the application is deployed. For more information,
see Azure paired regions.
This tutorial uses features of Azure SQL Database and the Azure platform to address these challenges:
Azure Resource Manager templates, to reserve all needed capacity as quickly as possible. Azure Resource
Manager templates are used to provision a mirror image of the original servers and elastic pools in the
recovery region. A separate server and pool are also created for provisioning new tenants.
Elastic Database Client Library (EDCL ), to create and maintain a tenant database catalog. The extended catalog
includes periodically refreshed pool and database configuration information.
Shard management recovery features of the EDCL, to maintain database location entries in the catalog during
recovery and repatriation.
Geo-restore, to recover the catalog and tenant databases from automatically maintained geo-redundant
backups.
Asynchronous restore operations, sent in tenant-priority order, are queued for each pool by the system and
processed in batches so the pool isn't overloaded. These operations can be canceled before or during execution
if necessary.
Geo-replication, to repatriate databases to the original region after the outage. There is no data loss and
minimal impact on the tenant when you use geo-replication.
SQL server DNS aliases, to allow the catalog sync process to connect to the active catalog regardless of its
location.
TIP
Hover the mouse over the location to enlarge the display.
2. Select the Contoso Concert Hall tenant and open its event page.
In the footer, notice the tenant's server name. The location is the same as the catalog server's location.
3. In the Azure portal, review and open the resource group in which you deployed the app.
Notice the resources and the region in which the app service components and SQL Database servers are
deployed.
IMPORTANT
For simplicity, the sync process and other long-running recovery and repatriation processes are implemented in these
samples as local PowerShell jobs or sessions that run under your client user login. The authentication tokens issued when you
log in expire after several hours, and the jobs will then fail. In a production scenario, long-running processes should be
implemented as reliable Azure services of some kind, running under a service principal. See Use Azure PowerShell to create a
service principal with a certificate.
1. In the PowerShell ISE, open the ...\Learning Modules\UserConfig.psm1 file. Replace <resourcegroup> and
<user> on lines 10 and 11 with the value used when you deployed the app. Save the file.
2. In the PowerShell ISE, open the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
RestoreFromBackup\Demo-RestoreFromBackup.ps1 script.
In this tutorial, you run each of the scenarios in this PowerShell script, so keep this file open.
3. Set the following:
$DemoScenario = 1: Start a background job that syncs tenant server and pool configuration info into the
catalog.
4. To run the sync script, select F5.
This information is used later to ensure that recovery creates a mirror image of the servers, pools, and
databases in the recovery region.
Leave the PowerShell window running in the background and continue with the rest of this tutorial.
NOTE
The sync process connects to the catalog via a DNS alias. The alias is modified during restore and repatriation to point to the
active catalog. The sync process keeps the catalog up to date with any database or pool configuration changes made in the
recovery region. During repatriation, these changes are applied to the equivalent resources in the original region.
Imagine there's an outage in the region in which the application is deployed, and run the recovery script:
1. In the PowerShell ISE, in the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
RestoreFromBackup\Demo-RestoreFromBackup.ps1 script, set the following value:
$DemoScenario = 2: Recover the app into a recovery region by restoring from geo-redundant backups.
2. To run the script, select F5.
The script opens in a new PowerShell window and then starts a set of PowerShell jobs that run in
parallel. These jobs restore servers, pools, and databases to the recovery region.
The recovery region is the paired region associated with the Azure region in which you deployed the
application. For more information, see Azure paired regions.
3. Monitor the status of the recovery process in the PowerShell window.
NOTE
To explore the code for the recovery jobs, review the PowerShell scripts in the ...\Learning Modules\Business Continuity and
Disaster Recovery\DR-RestoreFromBackup\RecoveryJobs folder.
NOTE
Other tutorials in the sample are not designed to run with the app in the recovery state. If you want to explore other
tutorials, be sure to repatriate the application first.
The restore process creates all the recovery resources in a recovery resource group. The cleanup process deletes
this resource group and removes all references to the resources from the catalog.
1. In the PowerShell ISE, in the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
RestoreFromBackup\Demo-RestoreFromBackup.ps1 script, set:
$DemoScenario = 6: Delete obsolete resources from the recovery region.
2. To run the script, select F5.
After cleaning up the scripts, the application is back where it started. At this point, you can run the script again or
try out other tutorials.
Designing the application to ensure that the app and the database are
co-located
The application is designed to always connect from an instance in the same region as the tenant's database. This
design reduces latency between the application and the database. This optimization assumes the app-to-database
interaction is chattier than the user-to-app interaction.
Tenant databases might be spread across recovery and original regions for some time during repatriation. For each
database, the app looks up the region in which the database is located by doing a DNS lookup on the tenant server
name. In SQL Database, the server name is an alias. The aliased server name contains the region name. If the
application isn't in the same region as the database, it redirects to the instance in the same region as the database
server. Redirecting to the instance in the same region as the database minimizes latency between the app and the
database.
Next steps
In this tutorial, you learned how to:
Use the tenant catalog to hold periodically refreshed configuration information, which allows a mirror image
recovery environment to be created in another region.
Recover Azure SQL databases into the recovery region by using geo-restore.
Update the tenant catalog to reflect restored tenant database locations.
Use a DNS alias to enable an application to connect to the tenant catalog throughout without reconfiguration.
Use geo-replication to repatriate recovered databases to their original region after an outage is resolved.
Try the Disaster recovery for a multitenant SaaS application using database geo-replication tutorial to learn how to
use geo-replication to dramatically reduce the time needed to recover a large-scale multitenant application.
Additional resources
Additional tutorials that build upon the Wingtip SaaS application
Disaster recovery for a multi-tenant SaaS application
using database geo-replication
5/23/2018 • 17 minutes to read • Edit Online
In this tutorial, you explore a full disaster recovery scenario for a multi-tenant SaaS application implemented using
the database-per-tenant model. To protect the app from an outage, you use geo -replication to create replicas for
the catalog and tenant databases in an alternate recovery region. If an outage occurs, you quickly fail over to these
replicas to resume normal business operations. On failover, the databases in the original region become secondary
replicas of the databases in the recovery region. Once these replicas come back online they automatically catch up
to the state of the databases in the recovery region. After the outage is resolved, you fail back to the databases in
the original production region.
This tutorial explores both the failover and failback workflows. You'll learn how to:
Sync database and elastic pool configuration info into the tenant catalog
Set up a recovery environment in an alternate region, comprising application, servers, and pools
Use geo -replication to replicate the catalog and tenant databases to the recovery region
Fail over the application and catalog and tenant databases to the recovery region
Later, fail over the application, catalog and tenant databases back to the original region after the outage is
resolved
Update the catalog as each tenant database is failed over to track the primary location of each tenant's database
Ensure the application and primary tenant database are always colocated in the same Azure region to reduce
latency
Before starting this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS database per tenant app is deployed. To deploy in less than five minutes, see Deploy
and explore the Wingtip Tickets SaaS database per tenant application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
Tutorial overview
In this tutorial, you first use geo-replication to create replicas of the Wingtip Tickets application and its databases
in a different region. Then, you fail over to this region to simulate recovering from an outage. When complete, the
application is fully functional in the recovery region.
Later, in a separate repatriation step, you fail over the catalog and tenant databases in the recovery region to the
original region. The application and databases stay available throughout repatriation. When complete, the
application is fully functional in the original region.
NOTE
The application is recovered into the paired region of the region in which the application is deployed. For more information,
see Azure paired regions.
2. Click on the Contoso Concert Hall tenant and open its event page.
In the footer, notice the tenant server name. The location will be the same as the catalog server's
location.
3. In the Azure portal, open the resource group in which the app is deployed
Notice the region in which the servers are deployed.
IMPORTANT
For simplicity, the sync process and other long running recovery and repatriation processes are implemented in these
tutorials as local Powershell jobs or sessions that run under your client user login. The authentication tokens issued when
you login will expire after several hours and the jobs will then fail. In a production scenario, long-running processes should be
implemented as reliable Azure services of some kind, running under a service principal. See Use Azure PowerShell to create a
service principal with a certificate.
1. In the PowerShell ISE, open the ...\Learning Modules\UserConfig.psm1 file. Replace <resourcegroup> and
<user> on lines 10 and 11 with the value used when you deployed the app. Save the file!
2. In the PowerShell ISE, open the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
FailoverToReplica\Demo-FailoverToReplica.ps1 script and set:
$DemoScenario = 1, Start a background job that syncs tenant server, and pool configuration info into
the catalog
3. Press F5 to run the sync script. A new PowerShell session is opened to sync the configuration of tenant
resources.
Leave the PowerShell window running in the background and continue with the rest of the tutorial.
NOTE
The sync process connects to the catalog via a DNS alias. This alias is modified during restore and repatriation to point to the
active catalog. The sync process keeps the catalog up-to-date with any database or pool configuration changes made in the
recovery region. During repatriation, these changes are applied to the equivalent resources in the original region.
NOTE
This tutorial adds geo-replication protection to the Wingtip Tickets sample application. In a production scenario for an
application that uses geo-replication, each tenant would be provisioned with a geo-replicated database from the outset. See
Designing highly available services using Azure SQL Database
1. In the PowerShell ISE, open the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
FailoverToReplica\Demo-FailoverToReplica.ps1 script and set the following values:
$DemoScenario = 2, Create mirror image recovery environment and replicate catalog and tenant
databases
2. Press F5 to run the script. A new PowerShell session is opened to create the replicas.
NOTE
In an outage scenario, the primary databases in the original region are offline. Force fail over on the secondary
breaks the connection to the primary without trying to apply any residual queued transactions. In a DR drill scenario
like this tutorial, if there is any update activity at the time of failover there could be some data loss. Later, during
repatriation, when you fail over databases in the recovery region back to the original region, a normal failover is used
to ensure there is no data loss.
8. Monitors the SQL database service to determine when databases have been failed over. Once a tenant
database is failed over, it updates the catalog to record the recovery state of the tenant database and mark
the tenant as online.
Tenant databases can be accessed by the application as soon as they're marked online in the catalog.
A sum of rowversion values in the tenant database is stored in the catalog. This value acts as a
fingerprint that allows the repatriation process to determine if the database has been updated in the
recovery region.
Run the script to fail over to the recovery region
Now imagine there is an outage in the region in which the application is deployed and run the recovery script:
1. In the PowerShell ISE, open the ...\Learning Modules\Business Continuity and Disaster Recovery\DR -
FailoverToReplica\Demo-FailoverToReplica.ps1 script and set the following values:
$DemoScenario = 3, Recover the app into a recovery region by failing over to replicas
2. Press F5 to run the script.
The script opens in a new PowerShell window and then starts a series of PowerShell jobs that run in
parallel. These jobs fail over tenant databases to the recovery region.
The recovery region is the paired region associated with the Azure region in which you deployed the
application. For more information, see Azure paired regions.
3. Monitor the status of the recovery process in the PowerShell window.
NOTE
To explore the code for the recovery jobs, review the PowerShell scripts in the ...\Learning Modules\Business Continuity and
Disaster Recovery\DR-FailoverToReplica\RecoveryJobs folder.
NOTE
With only a few databases to recover, you may not be able to refresh the browser before recovery has
completed, so you may not see the tenants while they are offline.
If you open an offline tenant's Events page directly, it displays a 'tenant offline' notification. For
example, if Contoso Concert Hall is offline, try to open http://events.wingtip-dpt.
<user>.trafficmanager.net/contosoconcerthall
4. In the browser, refresh the Wingtip Tickets Events Hub page to see Hawthorn Hall included.
If you provisioned Hawthorn Hall without waiting for the other tenants to restore, other tenants may still
be offline.
4. Open the tenants2 -dpt-<user>-recovery SQL server. Notice it contains the database hawthornhall and the
elastic pool, Pool1. The hawthornhall database is configured as an elastic database in Pool1 elastic pool.
5. Navigate back to the resource group and click on the Contoso Concert Hall database on the tenants1 -dpt-
<user>-recovery server. Click on Geo-Replication on the left side.
Next steps
In this tutorial you learned how to:
Sync database and elastic pool configuration info into the tenant catalog
Set up a recovery environment in an alternate region, comprising application, servers, and pools
Use geo -replication to replicate the catalog and tenant databases to the recovery region
Fail over the application and catalog and tenant databases to the recovery region
Fail back the application, catalog and tenant databases to the original region after the outage is resolved
You can learn more about the technologies Azure SQL database provides to enable business continuity in the
Business Continuity Overview documentation.
Additional resources
Additional tutorials that build upon the Wingtip SaaS application
Deploy and explore a sharded multi-tenant
application that uses Azure SQL Database
9/4/2018 • 11 minutes to read • Edit Online
In this tutorial, you deploy and explore a sample multi-tenant SaaS application that is named Wingtip Tickets. The
Wingtip Tickets app is designed to showcase features of Azure SQL Database that simplify the implementation of
SaaS scenarios.
This implementation of the Wingtip Tickets app uses a sharded multi-tenant database pattern. The sharding is by
tenant identifier. Tenant data is distributed to a particular database according to the tenant identifier values.
This database pattern allows you to store one or more tenants in each shard or database. You can optimize for
lowest cost by having each database be shared by multiple tenants. Or you can optimize for isolation by having
each database store only one tenant. Your optimization choice can be made independently for each specific tenant.
Your choice can be made when the tenant is first stored, or you can change your mind later. The application is
designed to work well either way.
App deploys quickly
The app runs in the Azure cloud and uses Azure SQL Database. The deployment section that follows provides the
blue Deploy to Azure button. When the button is pressed, the app is fully deployed to your Azure subscription
within five minutes. You have full access to work with the individual application components.
The application is deployed with data for three sample tenants. The tenants are stored together in one multi-
tenant database.
Anyone can download the C# and PowerShell source code for Wingtip Tickets from its GitHub repository.
Learn in this tutorial
How to deploy the Wingtip Tickets SaaS application.
Where to get the application source code, and management scripts.
About the servers and databases that make up the app.
How tenants are mapped to their data with the catalog.
How to provision a new tenant.
How to monitor tenant activity in the app.
A series of related tutorials is available that build upon this initial deployment. The tutorials explore a range of
SaaS design and management patterns. When you work through the tutorials, you are encouraged to step
through the provided scripts to see how the different SaaS patterns are implemented.
Prerequisites
To complete this tutorial, make sure the following prerequisites are completed:
The latest Azure PowerShell is installed. For details, see Getting started with Azure PowerShell.
IMPORTANT
For this demonstration, do not use any pre-existing resource groups, servers, or pools. Instead, choose Create a
new resource group. Delete this resource group when you are finished with the application to stop related billing.
Do not use this application, or any resources it creates, for production. Some aspects of authentication, and the
server firewall settings, are intentionally insecure in the app to facilitate the demonstration.
For Resource group - Select Create new, and then provide a Name for the resource group (case
sensitive).
Select a Location from the drop-down list.
For User - We recommend that you choose a short User value.
3. Deploy the application.
Click to agree to the terms and conditions.
Click Purchase.
4. Monitor deployment status by clicking Notifications, which is the bell icon to the right of the search box.
Deploying the Wingtip app takes approximately five minutes.
NOTE
Executable contents (scripts, DLLs) may be blocked by Windows when zip files are downloaded from an external source and
extracted. When extracting the scripts from a zip file, use the following steps to unblock the .zip file before extracting. By
unblocking the .zip file, you ensure the scripts are allowed to run.
NOTE
You must run the PowerShell scripts only by pressing the F5 key, not by pressing F8 to run a selected part of the
script. The problem with F8 is that the $PSScriptRoot variable is not evaluated. This variable is needed by many
scripts to navigate folders, invoke other scripts, or import modules.
The new Red Maple Racing tenant is added to the Tenants1 database and registered in the catalog. The new
tenant's ticket-selling Events site opens in your browser:
Refresh the Events Hub, and the new tenant now appears in the list.
3. Go back to the resource group and select the tenants1 -mt server that holds the tenant databases.
The tenants1 database is a multi-tenant database in which the original three tenants, plus the first tenant
you added, are stored. It is configured as a 50 DTU Standard database.
The salixsalsa database holds the Salix Salsa dance venue as its only tenant. It is configured as a
Standard edition database with 50 DTUs by default.
Monitor the performance of the database
If the load generator has been running for several minutes, enough telemetry is available to look at the database
monitoring capabilities built into the Azure portal.
1. Browse to the tenants1-mt<user> server, and click tenants1 to view resource utilization for the database
that has four tenants in it. Each tenant is subject to a sporadic heavy load from the load generator:
The DTU utilization chart nicely illustrates how a multi-tenant database can support an unpredictable
workload across many tenants. In this case, the load generator is applying a sporadic load of roughly 30
DTUs to each tenant. This load equates to 60% utilization of a 50 DTU database. Peaks that exceed 60% are
the result of load being applied to more than one tenant at the same time.
2. Browse to the tenants1-mt<user> server, and click the salixsalsa database. You can see the resource
utilization on this database that contains only one tenant.
The load generator is applying a similar load to each tenant, regardless of which database each tenant is in. With
only one tenant in the salixsalsa database, you can see that the database could sustain a much higher load than
the database with several tenants.
Resource allocations vary by workload
Sometimes a multi-tenant database requires more resources for good performance than does a single-tenant
database, but not always. The optimal allocation of resources depends on the particular workload characteristics
for the tenants in your system.
The workloads generated by the load generator script are for illustration purposes only.
Additional resources
To learn about multi-tenant SaaS applications, see Design patterns for multi-tenant SaaS applications.
To learn about elastic pools, see:
Elastic pools help you manage and scale multiple Azure SQL databases
Scaling out with Azure SQL Database
Next steps
In this tutorial you learned:
How to deploy the Wingtip Tickets SaaS Multi-tenant Database application.
About the servers, and databases that make up the app.
Tenants are mapped to their data with the catalog.
How to provision new tenants, into a multi-tenant database and single-tenant database.
How to view pool utilization to monitor tenant activity.
How to delete sample resources to stop related billing.
Now try the Provision and catalog tutorial.
Provision and catalog new tenants in a SaaS
application using a sharded multi-tenant Azure SQL
database
5/23/2018 • 12 minutes to read • Edit Online
This article covers the provisioning and cataloging of new tenants, in a multi-tenant sharded database model or
pattern.
This article has two major parts:
Conceptual discussion of the provisioning and cataloging of new tenants.
Tutorial that highlights the PowerShell script code that accomplishes the provisioning and cataloging.
The tutorial uses the Wingtip Tickets SaaS application, adapted to the multi-tenant sharded database
pattern.
Database pattern
This section, plus a few more that follow, discuss the concepts of the multi-tenant sharded database pattern.
In this multi-tenant sharded model, the table schemas inside each database include a tenant key in the primary key
of tables that store tenant data. The tenant key enables each individual database to store 0, 1, or many tenants. The
use of sharded databases makes it easy for the application system to support a very large number of tenants. All
the data for any one tenant is stored in one database. The large number of tenants are distributed across the many
sharded databases. A catalog database stores the mapping of each tenant to its database.
Isolation versus lower cost
A tenant that has a database all to itself enjoys the benefits of isolation. The tenant can have the database restored
to an earlier date without being restricted by the impact on other tenants. Database performance can be tuned to
optimize for the one tenant, again without having to compromise with other tenants. The problem is that isolation
costs more than it costs to share a database with other tenants.
When a new tenant is provisioned, it can share a database with other tenants, or it can be placed into its own new
database. Later you can change your mind and move the database to the other situation.
Databases with multiple tenants and single tenants are mixed in the same SaaS application, to optimize cost or
isolation for each tenant.
Tenant catalog pattern
When you have two or more databases that each contain at least one tenant, the application must have a way to
discover which database stores the tenant of current interest. A catalog database stores this mapping.
Tenant key
For each tenant, the Wingtip application can derive a unique key, which is the tenant key. The app extracts the
tenant name from the webpage URL. The app hashes the name to obtain the key. The app uses the key to access
the catalog. The catalog cross-references information about the database in which the tenant is stored. The app
uses the database info to connect. Other tenant key schemes can also be used.
Using a catalog allows the name or location of a tenant database to be changed after provisioning without
disrupting the application. In a multi-tenant database model, the catalog accommodates moving a tenant between
databases.
Tenant metadata beyond location
The catalog can also indicate whether a tenant is offline for maintenance or other actions. And the catalog can be
extended to store additional tenant or database metadata, such as the following items:
The service tier or edition of a database.
The version of the database schema.
The tenant name and its SLA (service level agreement).
Information to enable application management, customer support, or devops processes.
The catalog can also be used to enable cross-tenant reporting, schema management, and data extract for analytics
purposes.
Elastic Database Client Library
In Wingtip, the catalog is implemented in the tenantcatalog database. The tenantcatalog is created using the Shard
Management features of the Elastic Database Client Library (EDCL ). The library enables an application to create,
manage, and use a shard map that is stored in a database. A shard map cross-references the tenant key with its
shard, meaning its sharded database.
During tenant provisioning, EDCL functions can be used from applications or PowerShell scripts to create the
entries in the shard map. Later the EDCL functions can be used to connect to the correct database. The EDCL
caches connection information to minimize the traffic on the catalog database and speed up the process of
connecting.
IMPORTANT
Do not edit the data in the catalog database through direct access! Direct updates are not supported due to the high risk of
data corruption. Instead, edit the mapping data by using EDCL APIs only.
Tutorial begins
In this tutorial, you learn how to:
Provision a tenant into a multi-tenant database
Provision a tenant into a single-tenant database
Provision a batch of tenants into both multi-tenant and single-tenant databases
Register a database and tenant mapping in a catalog
Prerequisites
To complete this tutorial, make sure the following prerequisites are completed:
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
The Wingtip Tickets SaaS Multi-tenant Database app is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip Tickets SaaS Multi-tenant Database application
Get the Wingtip scripts and source code:
The Wingtip Tickets SaaS Multi-tenant Database scripts and application source code are available in the
WingtipTicketsSaaS -MultitenantDB GitHub repo.
See the general guidance for steps to download and unblock the Wingtip scripts.
While the Azure portal shows the tenant databases, it doesn't let you see the tenants inside the shared database.
The full list of tenants can be seen in the Events Hub webpage of Wingtip, and by browsing the catalog.
Using Wingtip Tickets events hub page
Open the Events Hub page in the browser (http:events.wingtip-mt.<USER>.trafficmanager.net)
Using catalog database
The full list of tenants and the corresponding database for each is available in the catalog. A SQL view is provided
that joins the tenant name to the database name. The view nicely demonstrates the value of extending the
metadata that is stored in the catalog.
The SQL view is available in the tenantcatalog database.
The tenant name is stored in the Tenants table.
The database name is stored in the Shard Management tables.
1. In SQL Server Management Studio (SSMS ), connect to the tenants server at catalog-mt.
<USER>.database.windows.net, with Login = developer, and Password = P@ssword1
2. In the SSMS Object Explorer, browse to the views in the tenantcatalog database.
3. Right click on the view TenantsExtended and choose Select Top 1000 Rows. Note the mapping between
tenant name and database for the different tenants.
Additional resources
Elastic database client library
How to Debug Scripts in Windows PowerShell ISE
Next steps
In this tutorial you learned how to:
Provision a single new tenant into a shared multi-tenant database and its own database
Provision a batch of additional tenants
Step through the details of provisioning tenants, and registering them into the catalog
Try the Performance monitoring tutorial.
Monitor and manage performance of sharded multi-
tenant Azure SQL database in a multi-tenant SaaS
app
6/25/2018 • 10 minutes to read • Edit Online
In this tutorial, several key performance management scenarios used in SaaS applications are explored. Using a
load generator to simulate activity across sharded multi-tenant databases, the built-in monitoring and alerting
features of SQL Database are demonstrated.
The Wingtip Tickets SaaS Multi-tenant Database app uses a sharded multi-tenant data model, where venue
(tenant) data is distributed by tenant ID across potentially multiple databases. Like many SaaS applications, the
anticipated tenant workload pattern is unpredictable and sporadic. In other words, ticket sales may occur at any
time. To take advantage of this typical database usage pattern, databases can be scaled up and down to optimize
the cost of a solution. With this type of pattern, it's important to monitor database resource usage to ensure that
loads are reasonably balanced across potentially multiple databases. You also need to ensure that individual
databases have adequate resources and are not hitting their DTU limits. This tutorial explores ways to monitor and
manage databases, and how to take corrective action in response to variations in workload.
In this tutorial you learn how to:
Simulate usage on a sharded multi-tenant database by running a provided load generator
Monitor the database as it responds to the increase in load
Scale up the database in response to the increased database load
Provision a tenant into a single-tenant database
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS Multi-tenant Database app is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip Tickets SaaS Multi-tenant Database application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
DEMO SCENARIO
IMPORTANT
The load generator is running as a series of jobs in a new PowerShell window. If you close the session, the load generator
stops. The load generator remains in a job-invoking state where it generates load on any new tenants that are provisioned
after the generator is started. Use Ctrl-C to stop invoking new jobs and exit the script. The load generator will continue to
run, but only on existing tenants.
Next steps
In this tutorial you learn how to:
Simulate usage on a sharded multi-tenant database by running a provided load generator
Monitor the database as it responds to the increase in load
Scale up the database in response to the increased database load
Provision a tenant into a single-tenant database
Additional resources
Azure automation
Run ad hoc analytics queries across multiple Azure
SQL databases
5/23/2018 • 7 minutes to read • Edit Online
In this tutorial, you run distributed queries across the entire set of tenant databases to enable ad hoc interactive
reporting. These queries can extract insights buried in the day-to-day operational data of the Wingtip Tickets SaaS
app. To do these extractions, you deploy an additional analytics database to the catalog server and use Elastic
Query to enable distributed queries.
In this tutorial you learn:
How to deploy an ad hoc reporting database
How to run distributed queries across all tenant databases
To complete this tutorial, make sure the following prerequisites are completed:
The Wingtip Tickets SaaS Multi-tenant Database app is deployed. To deploy in less than five minutes, see
Deploy and explore the Wingtip Tickets SaaS Multi-tenant Database application
Azure PowerShell is installed. For details, see Getting started with Azure PowerShell
SQL Server Management Studio (SSMS ) is installed. To download and install SSMS, see Download SQL
Server Management Studio (SSMS ).
SaaS applications can analyze the vast amount of tenant data that is stored centrally in the cloud. The analyses
reveal insights into the operation and usage of your application. These insights can guide feature development,
usability improvements, and other investments in your apps and services.
Accessing this data in a single multi-tenant database is easy, but not so easy when distributed at scale across
potentially thousands of databases. One approach is to use Elastic Query, which enables querying across a
distributed set of databases with common schema. These databases can be distributed across different resource
groups and subscriptions. Yet one common login must have access to extract data from all the databases. Elastic
Query uses a single head database in which external tables are defined that mirror tables or views in the
distributed (tenant) databases. Queries submitted to this head database are compiled to produce a distributed
query plan, with portions of the query pushed down to the tenant databases as needed. Elastic Query uses the
shard map in the catalog database to determine the location of all tenant databases. Setup and query are
straightforward using standard Transact-SQL, and support ad hoc querying from tools like Power BI and Excel.
By distributing queries across the tenant databases, Elastic Query provides immediate insight into live production
data. However, as Elastic Query pulls data from potentially many databases, query latency can sometimes be
higher than for equivalent queries submitted to a single multi-tenant database. Be sure to design queries to
minimize the data that is returned. Elastic Query is often best suited for querying small amounts of real-time data,
as opposed to building frequently used or complex analytics queries or reports. If queries do not perform well, look
at the execution plan to see what part of the query has been pushed down to the remote database. And assess how
much data is being returned. Queries that require complex analytical processing might be better served by saving
the extracted tenant data into a database that is optimized for analytics queries. SQL Database and SQL Data
Warehouse could host such the analytics database.
This pattern for analytics is explained in the tenant analytics tutorial.
By using the catalog database as the external data source, queries are distributed to all databases registered
in the catalog when the query is run. Because server names are different for each deployment, this
initialization script gets the location of the catalog database by retrieving the current server
(@@servername) where the script is executed.
The external tables that reference tenant tables are defined with DISTRIBUTION = SHARDED (VenueId).
This routes a query for a particular VenueId to the appropriate database and improves performance for
many scenarios as shown in the next section.
The local table VenueTypes that is created and populated. This reference data table is common in all tenant
databases, so it can be represented here as a local table and populated with the common data. For some
queries, this may reduce the amount of data moved between the tenant databases and the adhocreporting
database.
If you include reference tables in this manner, be sure to update the table schema and data whenever you
update the tenant databases.
4. Press F5 to run the script and initialize the adhocreporting database.
Now you can run distributed queries, and gather insights across all tenants!
Next steps
In this tutorial you learned how to:
Run distributed queries across all tenant databases
Deploy an ad hoc reporting database and add schema to it to run distributed queries.
Now try the Tenant Analytics tutorial to explore extracting data to a separate analytics database for more complex
analytics processing.
Additional resources
Elastic Query
Manage schema in a SaaS application that uses
sharded multi-tenant SQL databases
8/29/2018 • 7 minutes to read • Edit Online
This tutorial examines the challenges in maintaining a fleet of databases in a Software as a Service (SaaS )
application. Solutions are demonstrated for fanning out schema changes across the fleet of databases.
Like any application, the Wingtip Tickets SaaS app will evolve over time, and will require changes to the database.
Changes may impact schema or reference data, or apply database maintenance tasks. With a SaaS application
using a database per tenant pattern, changes must be coordinated across a potentially massive fleet of tenant
databases. In addition, you must incorporate these changes into the database provisioning process to ensure they
are included in new databases as they are created.
Two scenarios
This tutorial explores the following two scenarios:
Deploy reference data updates for all tenants.
Rebuild an index on the table that contains the reference data.
The Elastic Jobs feature of Azure SQL Database is used to execute these operations across tenant databases. The
jobs also operate on the 'template' tenant database. In the Wingtip Tickets sample app, this template database is
copied to provision a new tenant database.
In this tutorial you learn how to:
Create a job agent.
Execute a T-SQL query on multiple tenant databases.
Update reference data in all tenant databases.
Create an index on a table in all tenant databases.
Prerequisites
The Wingtip Tickets multi-tenant database app must already be deployed:
For instructions, see the first tutorial, which introduces the Wingtip Tickets SaaS multi-tenant database
app:
Deploy and explore a sharded multi-tenant application that uses Azure SQL Database.
The deploy process runs for less than five minutes.
You must have the sharded multi-tenant version of Wingtip installed. The versions for Standalone and
Database per tenant do not support this tutorial.
The latest version of SQL Server Management Studio (SSMS ) must be installed. Download and Install
SSMS.
Azure PowerShell must be installed. For details, see Getting started with Azure PowerShell.
NOTE
This tutorial uses features of the Azure SQL Database service that are in a limited preview ( Elastic Database jobs). If you wish
to do this tutorial, provide your subscription ID to [email protected] with subject=Elastic Jobs Preview. After you
receive confirmation that your subscription has been enabled, download and install the latest pre-release jobs cmdlets. This
preview is limited, so contact [email protected] for related questions or support.
NOTE
This tutorial uses features of the SQL Database service that are in a limited preview (Elastic Database jobs). If you wish to do
this tutorial, provide your subscription ID to [email protected] with subject=Elastic Jobs Preview. After you
receive confirmation that your subscription has been enabled, download and install the latest pre-release jobs cmdlets. This
preview is limited, so contact [email protected] for related questions or support.
Additional resources
Managing scaled-out cloud databases
Create and manage scaled-out cloud databases
Next steps
In this tutorial you learned how to:
Create a job agent to run T-SQL jobs across multiple databases
Update reference data in all tenant databases
Create an index on a table in all tenant databases
Next, try the Ad-hoc reporting tutorial to explore running distributed queries across tenant databases.
Cross-tenant analytics using extracted data
5/23/2018 • 13 minutes to read • Edit Online
In this tutorial, you walk through a complete analytics scenario. The scenario demonstrates how analytics can
enable businesses to make smart decisions. Using data extracted from sharded database, you use analytics to gain
insights into tenant behavior, including their use of the sample Wingtip Tickets SaaS application. This scenario
involves three steps:
1. Extract data from each tenant database into an analytics store.
2. Optimize the extracted data for analytics processing.
3. Use Business Intelligence tools to draw out useful insights, which can guide decision making.
In this tutorial you learn how to:
Create the tenant analytics store to extract the data into.
Use elastic jobs to extract data from each tenant database into the analytics store.
Optimize the extracted data (reorganize into a star-schema).
Query the analytics database.
Use Power BI for data visualization to highlight trends in tenant data and make recommendation for
improvements.
Finally, the star-schema tables are queried. The query results are displayed visually to highlight insights into tenant
behavior and their use of the application. With this star-schema, you can run queries that help discover items like
the following:
Who is buying tickets and from which venue.
Hidden patterns and trends in the following areas:
The sales of tickets.
The relative popularity of each venue.
Understanding how consistently each tenant is using the service provides an opportunity to create service plans to
cater to their needs. This tutorial provides basic examples of insights that can be gleaned from tenant data.
Setup
Prerequisites
To complete this tutorial, make sure the following prerequisites are met:
The Wingtip Tickets SaaS Multi-tenant Database application is deployed. To deploy in less than five minutes,
see Deploy and explore the Wingtip Tickets SaaS Multi-tenant Database application
The Wingtip SaaS scripts and application source code are downloaded from GitHub. Be sure to unblock the zip
file before extracting its contents. Check out the general guidance for steps to download and unblock the
Wingtip Tickets SaaS scripts.
Power BI Desktop is installed. Download Power BI Desktop
The batch of additional tenants has been provisioned, see the Provision tenants tutorial.
A job agent and job agent database have been created. See the appropriate steps in the Schema
management tutorial.
Create data for the demo
In this tutorial, analysis is performed on ticket sales data. In the current step, you generate ticket data for all the
tenants. Later this data is extracted for analysis. Ensure you have provisioned the batch of tenants as described
earlier, so that you have a meaningful amount of data. A sufficiently large amount of data can expose a range of
different ticket purchasing patterns.
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics\Demo -
TenantAnalytics.ps1, and set the following value:
$DemoScenario = 1 Purchase tickets for events at all venues
2. Press F5 to run the script and create ticket purchasing history for every event in each venue. The script runs for
several minutes to generate tens of thousands of tickets.
Deploy the analytics store
Often there are numerous transactional sharded databases that together hold all tenant data. You must aggregate
the tenant data from the sharded database into one analytics store. The aggregation enables efficient query of the
data. In this tutorial, an Azure SQL Database database is used to store the aggregated data.
In the following steps, you deploy the analytics store, which is called tenantanalytics. You also deploy predefined
tables that are populated later in the tutorial:
1. In PowerShell ISE, open …\Learning Modules\Operational Analytics\Tenant Analytics\Demo -
TenantAnalytics.ps1
2. Set the $DemoScenario variable in the script to match your choice of analytics store. For learning purposes,
SQL database without columnstore is recommended.
To use SQL database without columnstore, set $DemoScenario = 2
To use SQL database with columnstore, set $DemoScenario = 3
3. Press F5 to run the demo script (that calls the Deploy-TenantAnalytics.ps1 script) which creates the tenant
analytics store.
Now that you have deployed the application and filled it with interesting tenant data, use SQL Server
Management Studio (SSMS ) to connect tenants1-mt-<User> and catalog-mt-<User> servers using Login =
developer, Password = P@ssword1.
Data extraction
Create target groups
Before proceeding, ensure you have deployed the job account and jobaccount database. In the next set of steps,
Elastic Jobs is used to extract data from the sharded tenants database, and to store the data in the analytics store.
Then the second job shreds the data and stores it into tables in the star-schema. These two jobs run against two
different target groups, namely TenantGroup and AnalyticsGroup. The extract job runs against the
TenantGroup, which contains all the tenant databases. The shredding job runs against the AnalyticsGroup, which
contains just the analytics store. Create the target groups by using the following steps:
1. In SSMS, connect to the jobaccount database in catalog-mt-<User>.
2. In SSMS, open …\Learning Modules\Operational Analytics\Tenant Analytics\ TargetGroups.sql
3. Modify the @User variable at the top of the script, replacing with the user value used when you deployed the
Wingtip Tickets SaaS Multi-tenant Database application.
4. Press F5 to run the script that creates the two target groups.
Extract raw data from all tenants
Transactions might occur more frequently for ticket and customer data than for event and venue data. Therefore,
consider extracting ticket and customer data separately and more frequently than you extract event and venue
data. In this section, you define and schedule two separate jobs:
Extract ticket and customer data.
Extract event and venue data.
Each job extracts its data, and posts it into the analytics store. There a separate job shreds the extracted data into
the analytics star-schema.
1. In SSMS, connect to the jobaccount database in catalog-mt-<User> server.
2. In SSMS, open ...\Learning Modules\Operational Analytics\Tenant Analytics\ExtractTickets.sql.
3. Modify @User at the top of the script, and replace with the user name used when you deployed the Wingtip
Tickets SaaS Multi-tenant Database application.
4. Press F5 to run the script that creates and runs the job that extracts tickets and customers data from each
tenant database. The job saves the data into the analytics store.
5. Query the TicketsRawData table in the tenantanalytics database, to ensure that the table is populated with
tickets information from all tenants.
Repeat the preceding steps, except this time replace \ExtractTickets.sql with \ExtractVenuesEvents.sql in step
2.
Successfully running the job populates the EventsRawData table in the analytics store with new events and venues
information from all tenants.
Data reorganization
Shred extracted data to populate star-schema tables
The next step is to shred the extracted raw data into a set of tables that are optimized for analytics queries. A star-
schema is used. A central fact table holds individual ticket sales records. Dimension tables are populated with data
about venues, events, customers, and purchase dates.
In this section of the tutorial, you define and run a job that merges the extracted raw data with the data in the star-
schema tables. After the merge job is finished, the raw data is deleted, leaving the tables ready to be populated by
the next tenant data extract job.
1. In SSMS, connect to the jobaccount database in catalog-mt-<User>.
2. In SSMS, open …\Learning Modules\Operational Analytics\Tenant Analytics\ShredRawExtractedData.sql.
3. Press F5 to run the script to define a job that calls the sp_ShredRawExtractedData stored procedure in the
analytics store.
4. Allow enough time for the job to run successfully.
Check the Lifecycle column of jobs.jobs_execution table for the status of job. Ensure that the job
Succeeded before proceeding. A successful run displays data similar to the following chart:
Data exploration
Visualize tenant data
The data in the star-schema table provides all the ticket sales data needed for your analysis. To make it easier to
see trends in large data sets, you need to visualize it graphically. In this section, you learn how to use Power BI to
manipulate and visualize the tenant data you have extracted and organized.
Use the following steps to connect to Power BI, and to import the views you created earlier:
1. Launch Power BI desktop.
2. From the Home ribbon, select Get Data, and select More… from the menu.
3. In the Get Data window, select Azure SQL Database.
4. In the database login window, enter your server name (catalog-mt-<User>.database.windows.net). Select
Import for Data Connectivity Mode, and then click OK.
5. Select Database in the left pane, then enter user name = developer, and enter password = P@ssword1.
Click Connect.
6. In the Navigator pane, under the analytics database, select the star-schema tables: fact_Tickets,
dim_Events, dim_Venues, dim_Customers and dim_Dates. Then select Load.
Congratulations! You have successfully loaded the data into Power BI. Now you can start exploring interesting
visualizations to help gain insights into your tenants. Next you walk through how analytics can enable you to
provide data-driven recommendations to the Wingtip Tickets business team. The recommendations can help to
optimize the business model and customer experience.
You start by analyzing ticket sales data to see the variation in usage across the venues. Select the following options
in Power BI to plot a bar chart of the total number of tickets sold by each venue. Due to random variation in the
ticket generator, your results may be different.
The preceding plot confirms that the number of tickets sold by each venue varies. Venues that sell more tickets are
using your service more heavily than venues that sell fewer tickets. There may be an opportunity here to tailor
resource allocation according to different tenant needs.
You can further analyze the data to see how ticket sales vary over time. Select the following options in Power BI to
plot the total number of tickets sold each day for a period of 60 days.
The preceding chart displays that ticket sales spike for some venues. These spikes reinforce the idea that some
venues might be consuming system resources disproportionately. So far there is no obvious pattern in when the
spikes occur.
Next you want to further investigate the significance of these peak sale days. When do these peaks occur after
tickets go on sale? To plot tickets sold per day, select the following options in Power BI.
The preceding plot shows that some venues sell a lot of tickets on the first day of sale. As soon as tickets go on sale
at these venues, there seems to be a mad rush. This burst of activity by a few venues might impact the service for
other tenants.
You can drill into the data again to see if this mad rush is true for all events hosted by these venues. In previous
plots, you observed that Contoso Concert Hall sells a lot of tickets, and that Contoso also has a spike in ticket sales
on certain days. Play around with Power BI options to plot cumulative ticket sales for Contoso Concert Hall,
focusing on sale trends for each of its events. Do all events follow the same sale pattern?
The preceding plot for Contoso Concert Hall shows that the mad rush does not happen for all events. Play around
with the filter options to see sale trends for other venues.
The insights into ticket selling patterns might lead Wingtip Tickets to optimize their business model. Instead of
charging all tenants equally, perhaps Wingtip should introduce service tiers with different performance levels.
Larger venues that need to sell more tickets per day could be offered a higher tier with a higher service level
agreement (SLA). Those venues could have their databases placed in pool with higher per-database resource
limits. Each service tier could have an hourly sales allocation, with additional fees charged for exceeding the
allocation. Larger venues that have periodic bursts of sales would benefit from the higher tiers, and Wingtip
Tickets can monetize their service more efficiently.
Meanwhile, some Wingtip Tickets customers complain that they struggle to sell enough tickets to justify the
service cost. Perhaps in these insights there is an opportunity to boost ticket sales for under performing venues.
Higher sales would increase the perceived value of the service. Right click fact_Tickets and select New measure.
Enter the following expression for the new measure called AverageTicketsSold:
AverageTicketsSold = DIVIDE(DIVIDE(COUNTROWS(fact_Tickets),DISTINCT(dim_Venues[VenueCapacity]))*100,
COUNTROWS(dim_Events))
Select the following visualization options to plot the percentage tickets sold by each venue to determine their
relative success.
The preceding plot shows that even though most venues sell more than 80% of their tickets, some are struggling
to fill more than half the seats. Play around with the Values Well to select maximum or minimum percentage of
tickets sold for each venue.
Earlier you deepened your analysis to discover that ticket sales tend to follow predictable patterns. This discovery
might let Wingtip Tickets help underperforming venues boost ticket sales by recommending dynamic pricing. This
discovery could reveal an opportunity to employ machine learning techniques to predict ticket sales for each
event. Predictions could also be made for the impact on revenue of offering discounts on ticket sales. Power BI
Embedded could be integrated into an event management application. The integration could help visualize
predicted sales and the effect of different discounts. The application could help devise an optimum discount to be
applied directly from the analytics display.
You have observed trends in tenant data from the Wingtip Tickets SaaS Multi-tenant Database application. You
can contemplate other ways the app can inform business decisions for SaaS application vendors. Vendors can
better cater to the needs of their tenants. Hopefully this tutorial has equipped you with tools necessary to perform
analytics on tenant data to empower your businesses to make data-driven decisions.
Next steps
In this tutorial, you learned how to:
Deployed a tenant analytics database with pre-defined star schema tables
Used elastic jobs to extract data from all the tenant database
Merge the extracted data into tables in a star-schema designed for analytics
Query an analytics database
Use Power BI for data visualization to observe trends in tenant data
Congratulations!
Additional resources
Elastic Jobs.
SQL Database FAQ
9/14/2018 • 18 minutes to read • Edit Online
What is a vCore?
A virtual core represents the logical CPU offered with an option to choose between generations of hardware. Gen
4 Logical CPUs are based on Intel E5-2673 v3 (Haswell) 2.4-GHz processors and Gen 5 Logical CPUs are based
on Intel E5-2673 v4 (Broadwell) 2.3-GHz processors.
Are there dual-use rights with Azure Hybrid Benefit for SQL Server?
You have 180 days of dual use rights of the license to ensure migrations are running seamlessly. After that 180-
day period, the SQL Server license can only be used in the cloud in SQL Database, and does not have dual use
rights on-premises and in the cloud.
How does Azure Hybrid Benefit for SQL Server differ from license
mobility?
Today, we offer license mobility benefits to SQL Server customers with Software Assurance that allows re-
assignment of their licenses to third-party shared servers. This benefit can be used on Azure IaaS and AWS EC2.
Azure Hybrid Benefit for SQL Server differs from license mobility in two key areas:
It provides economic benefits for moving highly virtualized workloads to Azure. SQL EE customers can get 4
cores in Azure in the General Purpose SKU for every core they own on-premises for highly virtualized
applications. License mobility does not allow any special cost benefits for moving virtualized workloads to the
cloud.
It provides for a PaaS destination on Azure (SQL Database Managed Instance) that is highly compatible with
SQL Server on-premises
What are the specific rights of the Azure Hybrid Benefit for SQL
Server?
SQL Database customers will have the following rights associated with Azure Hybrid Benefit for SQL Server:
LICENSE FOOTPRINT WHAT DOES AZURE HYBRID BENEFIT FOR SQL SERVER GET YOU?
LICENSE FOOTPRINT WHAT DOES AZURE HYBRID BENEFIT FOR SQL SERVER GET YOU?
SQL Server Enterprise Edition core customers with SA Can pay Base Rate on either General Purpose or Business
Critical SKU
SQL Server Standard Edition core customers with SA Can pay Base Rate on General Purpose SKU only
Does the cost of compute and storage depend on the service tier that
I choose?
The compute cost reflects the total compute capacity that is provisioned for the application. In the Business
Critical service tier, we automatically allocate at least 3 Always ON replicas. To reflect this additional allocation of
compute resources, the vCore price is approximately 2.7x higher in Business Critical. For the same reason, the
higher storage price per GB in the Business Critical tier reflects the high IO and low latency of the SSD storage.
At the same time, the cost of backup storage is not different because in both cases we use a class of standard
storage.
Are there any database feature differences between the existing DTU-
based and new vCore-based service tiers?
The new service tiers support a superset of the features available with the current DTU -based offerings. The
additional features include a set of additional dynamic management views (DMVs) and additional resource
configuration options.
Will the new vCore-based tiers support point in time restore (PITR) for
35 days as today?
You can configure the backup retention for PITR between 7 and 35 days. The backups storage will be charged
separately based on the actual storage consumption if it exceeds the storage amount equal to the maximum data
size. In preview, by default the PITR retention period is set to 7 days. In many cases, the maximum data size is
sufficient for storing 7 days of backups.
How long does it take to change the service tier or performance level
of a single database or move a database in and out of an elastic pool?
Changing the service tier of a database and moving in and out of a pool requires the database to be copied on the
platform as a background operation. Changing the service tier can take from a few minutes to several hours
depending on the size of the databases. In both cases, the databases remain online and available during the move.
For details on changing single databases, see Change the service tier of a database.
How does elastic pool usage using the DTU-based purchasing model
show up on my bill?
Elastic pool charges show up on your bill as Elastic DTUs (eDTUs) or vCores plus storage in the increments
shown on the pricing page. There is no per-database charge for elastic pools. You are billed for each hour a pool
exists at the highest eDTU or vCores, regardless of usage or whether the pool was active for less than an hour.
DTU -based purchasing model examples:
If you create a Standard elastic pool with 200 eDTUs at 11:18 a.m., adding five databases to the pool, you are
charged for 200 eDTUs for the whole hour, beginning at 11 a.m. through the remainder of the day.
On Day 2, at 5:05 a.m., Database 1 begins consuming 50 eDTUs and holds steady through the day. Databases
2-5 fluctuate between 0 and 80 eDTUs. During the day, you add five other databases that consume varying
eDTUs throughout the day. Day 2 is a full day billed at 200 eDTU.
On Day 3, at 5 a.m. you add another 15 databases. Database usage increases throughout the day to the point
where you decide to increase eDTUs for the pool from 200 to 400 at 8:05 p.m. Charges at the 200 eDTU level
were in effect until 8 pm and increases to 400 eDTUs for the remaining four hours.
How are elastic pool billed for the DTU-based purchasing model?
Elastic pools are billed per the following characteristics:
An elastic pool is billed upon its creation, even when there are no databases in the pool.
An elastic pool is billed hourly. This is the same metering frequency as for performance levels of single
databases.
If an elastic pool is resized, then the pool is not billed according to the new amount of resources until the
resizing operation completes. This follows the same pattern as changing the performance level of single
databases.
The price of an elastic pool is based on the resources of the pool. The price of an elastic pool is independent of
the number and utilization of the elastic databases within it.
For details, see SQL Database pricing, DTU -based purchasing model, and vCore-based purchasing model.
NOTE
For a limited period, backup charges and IO charges are free of charge.
NOTE
You cannot change the name of the server admin account after it is created.
If there is a network failure between two regions, how does the retry
logic work when geo-replication is set up?
If there is a disconnect, we retry every 10 seconds to re-establish connections.
What tools are available to monitor the replication lag between the
primary database and geo-secondary?
We expose the real-time replication lag between the primary database and geo-secondary through a DMV. For
details, see sys.dm_geo_replication_link_status.
Browse this list of public data sets for data that you can use to prototype and test storage and analytics services
and solutions.
US Government data Over 190,000 data sets covering Files of various sizes in various formats
agriculture, climate, consumer, including HTML, XML, CSV, JSON, Excel,
ecosystems, education, energy, finance, and many others. You can filter available
health, local government, data sets by file format.
manufacturing, maritime, ocean, public
safety, and science and research in the
U.S.
US Census data Statistical data about the population of Data sets are in various formats.
the U.S.
Earth science data from NASA Over 32,000 data collections covering Data sets are in various formats.
agriculture, atmosphere, biosphere,
climate, cryosphere, human dimensions,
hydrosphere, land surface, oceans, sun-
earth interactions, and more.
Airline flight delays and other "The U.S. Department of Files are in CSV format.
transportation data Transportation's (DOT) Bureau of
Transportation Statistics (BTS) tracks the
on-time performance of domestic flights
operated by large air carriers. Summary
information on the number of on-time,
delayed, canceled, and diverted flights
appears ... in summary tables posted on
this website."
Traffic fatalities - US Fatality Analysis "FARS is a nationwide census providing "Create your own fatality data run
Reporting System (FARS) NHTSA, Congress, and the American online by using the FARS Query System.
public yearly data regarding fatal Or download all FARS data from 1975
injuries suffered in motor vehicle traffic to present from the FTP Site."
crashes."
Toxic chemical data - EPA Toxicity "EPA's most updated, publicly available Data sets are available in various
ForeCaster (ToxCast™) data high-throughput toxicity data on formats including spreadsheets, R
thousands of chemicals. This data is packages, and MySQL database files.
generated through the EPA's ToxCast
research effort."
DATA SOURCE ABOUT THE DATA ABOUT THE FILES
Toxic chemical data - NIH Tox21 Data "The 2014 Tox21 data challenge is Data sets are available in SMILES and
Challenge 2014 designed to help scientists understand SDF formats. The data provides "assay
the potential of the chemicals and activity data and chemical structures on
compounds being tested through the the Tox21 collection of ~10,000
Toxicology in the 21st Century initiative compounds (Tox21 10K)."
to disrupt biological pathways in ways
that may result in toxic effects."
Biotechnology and genome data from Multiple data sets covering genes, Data sets are in text, XML, BLAST, and
the NCBI genomes, and proteins. other formats. A BLAST app is available.
New York City taxi data "Taxi trip records include fields capturing Data sets are in CSV files by month.
pick-up and drop-off dates/times, pick-
up and drop-off locations, trip
distances, itemized fares, rate types,
payment types, and driver-reported
passenger counts."
Microsoft Research data sets - "Data Multiple data sets covering human- Data sets are in various formats, zipped
Science for Research" computer interaction, audio/video, data for download.
mining/information retrieval,
geospatial/location, natural language
processing, and robotics/computer
vision.
Public genome data "A diverse data set of whole human Data sets, after extraction, are in UNIX
genomes are freely available for public text format. Analysis tools are also
use to enhance any genomic study..." available.
The provider, Complete Genomics, is a
private for-profit corporation.
Open Science Data Cloud data "The Open Science Data Cloud provides Data sets are in various formats.
the scientific community with resources
for storing, sharing, and analyzing
terabyte and petabyte-scale scientific
datasets."
Global climate data - WorldcLIM "WorldClim is a set of global climate These files contain geospatial data. For
layers (gridded climate data) with a more info, see Data format.
spatial resolution of about 1 km2. These
data can be used for mapping and
spatial modeling."
Data about human society - The GDELT "The GDELT Project is the largest, most The raw data files are in CSV format.
Project comprehensive, and highest resolution
open database of human society ever
created."
Advertising click prediction data for "The largest ever publicly released ML
machine learning from Criteo dataset." For more info, see Criteo's 1
TB Click Prediction Dataset.
DATA SOURCE ABOUT THE DATA ABOUT THE FILES
ClueWeb09 text mining data set from "The ClueWeb09 dataset was created to See Dataset Information.
The Lemur Project support research on information
retrieval and related human language
technologies. It consists of about 1
billion web pages in 10 languages that
were collected in January and February
2009."
GitHub activity data from The "The GHTorrent project [is] an effort to MySQL database dumps are in CSV
GHTorrent project create a scalable, queriable, offline format.
mirror of data offered through the
GitHub REST API. GHTorrent monitors
the GitHub public event time line. For
each event, it retrieves its contents and
their dependencies, exhaustively."
Stack Overflow data dump "This is an anonymized dump of all "Each site [such as Stack Overflow] is
user-contributed content on the Stack formatted as a separate archive
Exchange network [including Stack consisting of XML files zipped via 7-zip
Overflow]." using bzip2 compression. Each site
archive includes Posts, Users, Votes,
Comments, PostHistory, and PostLinks."