Gianluca Hotz

#sqlsat

SQL Server Modernization

Who am I?
Gianluca Hotz | @glhotz | [email protected]
Independent Consultant, Founder and Mentor SolidQ
20+ years on SQL Server (from 4.21 in 1996)
Database modeling and development, sizing and administration, upgrade and migration,
performance tuning
Interests
Relational model, DBMS architecture, Security, High Availability and Disaster Recovery
Community
20 years Microsoft MVP SQL Server (from 1998)
Founder and President UGISS
User Group Italiano SQL Server (PASS Chapter)
Modernization?
Low Complexity
Version upgrades (e.g. 2008 R2 -> 2017)
Edition upgrades/downgrades (e.g. Standard -> Enterprise)
Medium Complexity
Virtualization
Consolidation
Adding HA/DR
High Complexity
Re-Platforming (e.g. Oracle -> SQL Server or Azure SQL Database)
Re-Architecting (e.g. SQL Server -> Azure SQL Datawarehouse or Data Lake)
Holistic Approach
Data Platform technologies
Project types
Hardware modernization and/or consolidation
Virtualization (Private and Public Cloud)
Software solution deployments
SharePoint, Navision, CRM, …

Infrastructure
hardware, virtualization platforms, storage solutions, HA/DR & backup solutions…
Application development
Licensing
SQL Server flavors
On-Premises
SQL Server (Database Engine, SSAS, SSRS, SSIS, DQS, MDS, MLS)
Analytics Platform System (APS was SQL Server PDW)
IaaS
SQL Server (Database Engine, SSAS, SSRS, SSIS, DQS, MDS, MLS)
PaaS
Azure SQL Database, Azure SQL Data Warehouse
Amazon RDS for SQL Server
Richer Data Platform offering (e.g. Data Lake, Data Factory, etc.)
Changing platform
Can be easy
Migrating databases to PaaS or containerized versions (but same RDBMS)
or…
Hard: converting databases to other RDBMSs
data must be physically migrated (different types, different representations,…)
database code must be converted (constraints, triggers, functions, procedures,…)
Application code can be
converted completely, to a new platform
just adapted to run against the new RDBMS
adapted to support multiple RDBMSs
Different kind of projects

Migration
one time conversion
solution switch-over at a certain point

Porting
side-by-side conversion & evolution
no switch-over, current solutions remain
Migration Projects
Usually done by final customers (i.e. not ISVs)
Driving factors
cost reduction
move away from vendor(s)
solution obsolescence
unreliability and/or supportability problems

Application
may be converted to a new platform
may be adapted to run with new RDBMS
Porting Projects
Usually done by ISVs
Driving factors
get into new markets
cost reduction for customers (TCO)
more choice for customers
become less dependent from vendor(s)
Application
may be ported to new platform
happens rarely, means re-write or 2 codebases
may be adapted to run with new RDBMS
Different kind of Databases to migrate/port
Simple schema
all the logic is inside the application
simple access methods (mostly by cursors)
very limited use of SQL dialects
database is used mostly for storage
Complex schema
much of the logic is inside the database
heavy usage of SQL dialects
database used as a rule and inference engine
Origin of problems
Sometimes porting issues arise from
misusage of functionalities
bad decoupling of application/database code
too much row oriented code in the database
low technical skills of coders/architects
Never judge
every application has it own story
a lot of decisions may be compromises
a lot of money was probably already invested
Application portability
Code in the RDBMS
has been a best practice for years
still a very good solution for both
integrity
scalability
however… portability is an issue!
different SQL dialects
different type systems
different engine features
Changing Architecture

Can be much more harder!

More complex
Low familiarity with non RBMS technologies
Low familiarity with unstructured dta
New implementation patterns
ML & AI will stop to be buzzwords…
Version Upgrades: motivations

Compliancy (e.g. end of support and/or security)

Better Performance

New functionalities

Winter is coming!
Version Upgrades: barriers
Licensing Costs
Regressions risk
Application changes costs
Complexity
Testing changes (application or certification processes)
Planning for low downtime
Lack of time/resources/competencies
Edition upgrades/downgrades
Upgrades
Motivations similar to version upgrades
Barriers similar to version upgrades
Complexity usually lower because setup covers the scenario

Downgrades
Main motivation to save money
Barriers similar to version upgrades
Needs reinstallation, less impact for non production environment
Hardest problems
Factors that require application changes
Breaking changes
Discontinued functionalities
Plan affecting changes may cause performance regressions
New Cardinality Estimator
Hotfixes
New QP strategies/behavior (e.g. Intelligent Query Processing)
Testing changes is hard!
Complete Workload coverage
Concurrent access
Database Compatibility Level
Version Native Level Supported Levels
Sets behavior compatible with SQL Server 2019 150 150, 140, 130, 120,
110, 100

specified version of SQL Server SQL Server 2017 140 140, 130, 120, 110,
100
Azure SQL Database 130 140, 130, 120, 110,
100
SQL Server 2016 130 130, 120, 110, 100

SQL Server 2014 120 120, 110, 100

SQL Server 2012 110 110, 100, 90

SQL Server 2008 R2 100 100, 90, 80

SQL Server 2008 100 100, 90, 80

SQL Server 2005 90 90, 80

SQL Server 2000 80 80

DB Compatibility Level based certification
Target for a database compatibility level
Stop certifying for any given platform (On-Premises, Cloud, PaaS/IaaS)
Stop certifying for a named SQL Server version
Microsoft Database Compatibility Level Protection
Full functional protection (once assessment tools runs with no errors)
Query Plan shape protection (on comparable hardware)
Maintaining backward compatibility very important for SQL Server
team…
DB Compatibility Level certification benefits
Simplified certification across platforms
E.g. SQL Server On-Premises and Azure SQL Database Managed Instances
Improved risk management
Decoupling application upgrade from database support upgrade cycles
Allow customers to upgrade/migrate to new/different versions
Benefits from new features Without changing functionalities
Not everything is gated by Database Compatibility levels e.g.
Adaptive Joins: gated
Auto-tuning, new Query memory gateways, leveraging NUMA architecture, Columnstore indexes:
non-gated
Functional Change Protection
Breaking Changes
Behavior changes resulting in different outcomes!
Not all protected under Database Compatibility
Protected changes examples
Improved accuracy in compatibility level 130+
Conversion behavior setting LANGUAGE context info
Conversion behavior parsing dates
Non protected changes examples
Column names changes in system objects
Functional Change Protection & Trace Flags
Sometimes functional changes are gated by Trace Flags

Example: truncation error 8152

Gated by trace flag 460
Will be on by default in 2019 with Compatibility Level 150
https://blogs.msdn.microsoft.com/sql_server_team/string-or-binary-data-
would-be-truncated-replacing-the-infamous-error-8152
Discontinued functionalities
Not protected by Compatibility Level!
Removed from database engine
Need to change queries using it
Examples
FASTFIRSTROW
replaced by OPTION (FAST n) in SQL Server 2012
sp_dboption
system stored procedure removed in SQL Server 2012
Query Plan shape protection
Plan-affecting changes
Query Optimizer fixes (Trace Flag 4199)
Changes to Cardinality Estimator
These will very likely cause regressions!
Especially Cardinality Estimator
Many queries better performance, other worse
Are you sure you understood how fixes works?
What about fixes for incorrect result causing regressions?
Query Optimizer Fixes with TF 4199
Compatibili TF QO QO
SQL Server 2016+ QO fixes from ty Level 4199 changes changes for
from current
previous versions enabled by default previous version
level post-RTM
under the latest level
100-120 Off Disabled Disabled
previously you had to enable them but
you enabled also current version, post- On Enabled Enabled
RTM, fixes
zero reported errors in support cases 130 Off Enabled Disabled
after changing the default
On Enabled Enabled
Fully Documented
140 Off Enabled Disabled
https://support.microsoft.com/en-
us/help/974006/sql-server-query- On Enabled Enabled
optimizer-hotfix-trace-flag-4199-
servicing-model
Query Optimizer fixes further control
SQL Server 2016+ database-level granularity
ALTER DATABASE SCOPED CONFIGURATION SET
QUERY_OPTIMIZER_HOTFIXES = ON|OFF

SQL Server 2016 SP1+ query-level granularity

OPTION(USE HINT('ENABLE_QUERY_OPTIMIZER_HOTFIXES’))
Cardinality Estimator problems
Essential for query plan generation

Was largely based on SQL Server 7.0

Redesigned in SQL Server 2014+

Ascending data estimation, correlation assumptions, join
containment, fixed estimation values (e.g. MTVF)
Will most probably cause regressions in performance!
Controlling Cardinality Estimation
SQL Server 2014
Bound to Database Compatibility Level 😬
Trace Flags (global, session and query levels)
Trace flag 2312 reverts to CE version 70
Trace flag 9481 use CE version 120

SQL Server 2016+ database-level configuration

ALTER DATABASE SCOPED CONFIGURATION SET
LEGACY_CARDINALITY_ESTIMATION = ON|OFF
SQL Server 2016 SP1+ query-level configuration
OPTION(USE HINT('FORCE_LEGACY_CARDINALITY_ESTIMATION'))
OPTION(USE HINT('FORCE_DEFAULT_CARDINALITY_ESTIMATION'))
Query Optimizer Compatibility Level
SQL Server 2017 CU10 introduces query-level granularity
OPTION(USE
HINT('QUERY_OPTIMIZER_COMPATIBILITY_LEVEL_n’))
Includes QO fixes enabled by default up to that level
TF 4199 enabled globally will still enable all fixes

https://blogs.msdn.microsoft.com/sql_server_team/developers-choice-hinting-query-execution-model
Query Optimizer Hints
USE HINT Trace Flag DB

SQL Server 2016 SP1+ DISABLE_OPTIMIZED_NESTED_LOOP 2340

Option

FORCE_LEGACY_CARDINALITY_ESTIMATION 9841 Yes

Control QO behavior ENABLE_QUERY_OPTIMIZER_HOTFIXES 4199 Yes

DISABLE_PARAMETER_SNIFFING 4136 Yes

Mainly to replace trace flags ASSUME_MIN_SELECTIVITY_FOR_FILTER_ESTIMAT

ES
4137 (Old CE)
9471 (New CE)

DISABLE_OPTIMIZER_ROWGOAL 4138

ENABLE_HIST_AMENDMENT_FOR_ASC_KEYS 4139

ASSUME_JOIN_PREDICATE_DEPENDS_ON_FILTERS
9476 (New CE)

FORCE_DEFAULT_CARDINALITY_ESTIMATION 2312

QUERY_OPTIMIZER_COMPATIBILITY_LEVEL_n

QUERY_PLAN_PROFILE

https://blogs.msdn.microsoft.com/sql_server_team/developers-choice-use-hint-query-hints
Microsoft recommended upgrade Plan

Upgrade to
latest SQL
Quickly fix
Server Move to
Create a regressions
version and Enable Query latest
baseline of by forcing
keeping Store compatibility
the workload last known
source level
good plan!
compatibility
level
Upgrade Strategies
Side-by-side In-Place
Allow OS Upgrade No additional Hardware
Easier testing No data migration
Easier rollback strategy
Less downtime
Side-by-side Upgrade Service Availability
Synchronization Strategy Notes
Log Shipping Cutover typically in minutes
Replication Cutover potentially in seconds
Backup/Restore From minutes to hours (depends on
strategy e.g. usage of differential and/or
log backups)
Filesystem/SAN Copy Dependent on technology
Database Mirroring/Availability Groups Cutover typically in seconds
Upgrade process to minimize risks

Functional &
Remediate
Performance
applications
tests
Data
Discover Assess Convert Cutover
Sync
Migrate
schema, Optimize
objects & data
Discover Phase
Which SQL Server versions do I have?

Which Editions am I running?

Which SQL Server components are installed?

How many cores are on each server?

How many databases are in each instance?

What are the sizes of all my databases?

What are the settings for each instance and database?

Assessment Tools
Microsoft Assessment and Planning (MAP) Toolkit
SQL PowerDoc
PowerShell Script gather data from WMI in XML
Windows and SQL Server rich inventory in Excel
Best Practices Reports in Excel
Consolidates multiple server/instances
Official https://github.com/kendalvandyke/sqlpowerdoc
Newer https://github.com/bryan5989/SQLPowerDoc
Quest Enterprise Reporter for SQL Server
Based on previously free Discovery Wizard for SQL Server
Part of a suite for compliancy covering several other products (e.g. AD)
Microsoft Upgrade/Migration Tools
Upgrade Advisor
Data Migration Assistant (DMA)
Data Experimentation Assistant (DEA)
Data Migration Service (DMS)
SQL Server Migration Assistant (SSMA)
SQL Server Data Tools for Visual Studio (SSDT)
SQL Server Management Studio (SSMS)
“Export Data Tier Application”
“Deploy Database to Microsoft Azure SQL Database” Wizard
Upgrade Advisor
Static Analysis
Database Engine
Analysis Services
Reporting Services
Integration Services
Dynamic Analysis
Trace Files
Obsolete!
Use only if source database non supported
You can still find it in SQL Server Feature Packs (2014 latest)
Data Migration Assistant
On-premises instances assessment
Blocking issues
Partially supported or unsupported features
Discover issues that may affect compatibility
Breaking Changes, Behavior Changes, Deprecated Features
Discover new features that may benefit after upgrading
Performance, security, Storage
Azure SQL Database SKU Recommendations
Performance counters analysis to suggest SKU Azure SQL Database
Performs the migration
Versions supported by DMA

Source Destination
SQL Server 2005 SQL Server 2012
SQL Server 2008 SQL Server 2014
SQL Server 2008 R2 SQL Server 2016
SQL Server 2012 SQL Server 2017
SQL Server 2014 on Windows and Linux

SQL Server 2016 Azure SQL Database

SQL Server 2017 on Windows Azure SQL Database

Managed Instance
Bulk Analysis

Repository instances/databases to assess in SQL/CSV

Automation with PowerShell

Import in reporting database

Reports with PowerBI

DMA Problems
Cannot save sessions!
Only Assessment data in JSON format
Azure SQL Database SKU Recommendations
Command Line Interface only
Currently not working…
Bulk Analysis
Cumbersome process
Errors
dmaDataCollector crashes with pubs, dmacmd.exe is a closed project
LoadWarehouse.sql needs to be fixed
Data Migration Assistant Resources
Docs
https://docs.microsoft.com/en-us/sql/dma/dma-overview
https://docs.microsoft.com/en-us/sql/dma/dma-
consolidatereports
Blog
https://blogs.msdn.microsoft.com/datamigration/tag/dma
https://blogs.msdn.microsoft.com/datamigration/tag/data-
migration-assistant
Database Experimentation Assistant
A/B testing solution for SQL Server
Workload comparison
e.g. upgrades, new indexes, etc.
Capture
SQL Trace
Extended Events (V2.6+)
Replay
DReplay integration
Analysis
Performance improvements and regressions
DEA Analysis Example
Versions supported by DEA
Source
SQL Server 2005+
Target
SQL Server 2005+
Analysis
SQL Server 2008+
Data Migration Service Resources
Docs
https://docs.microsoft.com/en-us/sql/dea/database-
experimentation-assistant-overview

Blog
https://blogs.msdn.microsoft.com/datamigration/tag/dea
Azure Database Migration Service

On-premises

SQL Server 2005 - 2017

My

Assessment Azure Database

SSMA & DMA Migration Service
DMS Migration Scenarios
SQL Server -> Azure SQL Database Offline

SQL Server -> Azure SQL Database Online

SQL Server -> Azure SQL Database Managed Instance Offline

SQL Server -> Azure SQL Database Managed Instance Online

MySQL -> Azure DB for MySQL Online

PostgreSQL -> Azure DB for PostgreSQL Online

Data Migration Service Resources
Docs
https://docs.microsoft.com/en-us/azure/dms/dms-overview
https://docs.microsoft.com/en-us/azure/dms/tutorial-sql-server-to-azure-
sql

Blog
https://blogs.msdn.microsoft.com/datamigration/tag/azure-database-
migration-service
https://blogs.msdn.microsoft.com/datamigration/tag/dms
SQL Server Migration Assistant
Migration effort assessment

Code conversion

Data migration

Testing

Deployment
Platforms supported by SSMA

Source Destination
Microsoft Access SQL Server 2008
DB2 SQL Server 2008 R2
MySQL SQL Server 2012
Oracle SQL Server 2014
SAP ASE (Sybase ASE) SQL Server 2016
SQL Server 2017 on Windows and Linux
SQL Server 2019 on Windows and Linux
Azure SQL Database
Azure SQL Database Managed Instance
Azure SQL Data Warehouse (Oracle only)
SSMA for Oracle: how it works (101)
Creates two “helper” databases
sysdb
ssmatesterdb
Stores metadata in the file system
data stored in xml compressed format
can rename to .gzip and uncompress it
shred xml into lot of files for caching
reports directory
can be safely deleted and re-generated
metadata reside in the original .mb files
SSMA for Oracle: sysdb helper database
Provides
code infrastructure to emulate features
system functions
schema features (e.g. sequences)
engine features (e.g. autonomous transactions)
session bound data storage
emulated metadata storage (e.g. packages)
Maintenance
protect metadata
performance related depends on workload
SSMA for Oracle: does it really work?
Yes, but don’t expect miracles!
Works well with simple schemas…
Usually won’t do all the work by itself
Very useful for
assessing efforts
assessing problems
doing prototypes
as a learning tool
to translate/emulate some functionalities
Testing
Nothing can replace good testing!
Hard to have tests with good coverage of the whole workload
Sometimes is more cost effective to run fewer tests and fix the
problems later…

SQL Server DReplay infrastructure

Often impossible to capture and replay even a fraction of the
workload
Workload Tools
Very good for side-by-side, real-time workload replay
Captures XE batch completed and replay commands on-the-fly

Captures execution telemetry (cpu, read, writes, duration)

Reports to find regressions

Developed by a well-known MVP ☺

https://github.com/spaghettidba/WorkloadTools
DBA Tools
Lot of PowerShell cmdlets to make the DBA life easier

Many cmdlets target specifically migration/upgrade scenarios

E.g copy logins with synch option

One stop cmdlet migrating an

All instance-level assets (e.g. credentials, linked servers, etc. )
Via backup/restore automation (just specify network share)

https://dbatools.io
Start-DbaMigration
https://youtu.be/hg8tovMRX2k
After the upgrade: remember the plan

Upgrade to
latest SQL
Quickly fix
Server Move to
Create a regressions
version and Enable Query latest
baseline of by forcing
keeping Store compatibility
the workload last known
source level
good plan!
compatibility
level
Automatic Tuning
Automatic plan correction
Automatically force last good plan when regression detected
Minimum 10 CPU seconds improvement
sys.dm_db_tuning_recommendations for manual corrections

SQL Server 2017 and Azure SQL Database

https://docs.microsoft.com/en-us/sql/relational-databases/automatic-tuning/automatic-tuning
Query Tuning Assistant
Well-defined workflow targeting patterns for query regressions
Independence vs. Correlation
Simple Containment vs. Base Containment
Multi-statement table-valued function (MSTVF) fixed cardinality guess

SQL Server Management Studio Wizard-Based Tool

Can be automated via PowerShell

Query Tuning Assistant plan

Upgrade to Collect Instead of

latest SQL Move to workload only forcing
Server Create a latest with new last known
version and Enable baseline of compatibilit level and good plan,
keeping Query Store the y level (or start experiment
source workload any target comparing with tuning
compatibilit level to test) with recommend
y level baseline ation actions

https://docs.microsoft.com/en-us/sql/relational-databases/performance/upgrade-dbcompat-using-qta
Modernization Stream Phases in a SQL
Project
Partner Collect Inspire

Assess Define Inventory Envision Scope

Modernize Detail Setup Test Execute

Support Tune Mentor

Assessment phase
• Finalize assessment’s scope
• Finalize questionnaires to be sent to project representatives
Define • Activities kick-off

• Detailed inventory of Windows Server & SQL Servers

• Static analysis of databases & SQL code
Inventory • Project representatives interviews (questionnaires)

• Consolidate & analyze inventory data

• Review configuration, performance, reliability and security best practices
Envision • Prepare modernization scenarios and macro action-plan

• Discuss deliverables and scenarios with customer

• Finalize modernization scenarios
Scope • Prepare next phases offering (Modernize & Support)
Assessment phase: reducing uncertainty
Better planning/flexibility
Size better understood results in reduced expectations and scope
Allows to run several Modernization phases at different times
Leaves the customer with something tangible
Detailed assets inventory
Performance/Security/Reliability automatic assessment
Static analysis of databases code
Possibility to rescope/change scenarios
Enough data to independently run the project
On-Premises to Azure PaaS/IaaS migrations
IaaS
SQL Server on Azure VMs
SQL Server on Azure Containers
SQL Server on Azure Kubernetes
PaaS
Azure SQL Database
Azure SQL Database Managed Instance
On-Premises to SQL Server PaaS migrations
 On-Premise to Managed Instance online
workflow
Azure Blob Storage

Cloud network
SQL DB
Managed Instance

Subnet used for MI Subnet used for DMS

SMB Network Share

On-Premise
Other on-premises to PaaS strategies
Method 1 Method 2 Method 3
SQL Server
SQL Server SQL Server
Visual
1. Import
Source
Studio
Source Source
DB Database
DB
DB Project
1. Export 4. Copy
1. Generate
2. Transform
5. Publish
SSMS OR 1. Deploy
SQL Azure Data
SSMS Copy *.sql Migratio
T-SQL
.bacpac
Migration DB n
Wizard Assistant
2. Import 2. Execute
6.
3. Edit, Build
Export/
& Test
Import
or Deploy 4. Publish (schema only)
Target Target
DB Target
DB
DB

Azure SQL Database Azure SQL Database

Azure SQL Database

Compatible database: manual Database almost compatible: Refactor with Visual Studio and Data
copy or deploy wizard in SSMS SQL Azure Migration Wizard Migration Assistant, final deploy with
DMA and SSMS SSMS
Migration with minimal downtime
Transactional Replication
Data Migration Resources
Blog
https://blogs.msdn.microsoft.com/datamigration
Upgrade SQL Server
https://docs.microsoft.com/sql/database-engine/install-windows/upgrade-sql-
server
Post-migration validation and optimization guide
https://docs.microsoft.com/sql/relational-databases/post-migration-validation-
and-optimization-guide
Azure Database Migration Guide
https://datamigration.microsoft.com
Q&A