Download Full SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory Andy Leonard PDF All Chapters
Download Full SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory Andy Leonard PDF All Chapters
com
https://textbookfull.com/product/sql-server-data-automation-
through-frameworks-building-metadata-driven-frameworks-with-
t-sql-ssis-and-azure-data-factory-andy-leonard/
OR CLICK BUTTON
DOWNLOAD NOW
https://textbookfull.com/product/sql-server-execution-plans-for-sql-
server-2008-through-to-2017-and-azure-sql-database-3rd-edition-grant-
fritchey/
textboxfull.com
https://textbookfull.com/product/biota-grow-2c-gather-2c-cook-loucas/
textboxfull.com
Quick Start Guide to Azure Data Factory Azure Data Lake
Server and Azure Data Warehouse 1st Edition Mark Beckner
https://textbookfull.com/product/quick-start-guide-to-azure-data-
factory-azure-data-lake-server-and-azure-data-warehouse-1st-edition-
mark-beckner/
textboxfull.com
https://textbookfull.com/product/data-integration-life-cycle-
management-with-ssis-a-short-introduction-by-example-1st-edition-andy-
leonard/
textboxfull.com
https://textbookfull.com/product/sql-server-big-data-clusters-data-
virtualization-data-lake-and-ai-platform-benjamin-weissman/
textboxfull.com
https://textbookfull.com/product/azure-sql-revealed-a-guide-to-the-
cloud-for-sql-server-professionals-bob-ward/
textboxfull.com
https://textbookfull.com/product/azure-sql-revealed-a-guide-to-the-
cloud-for-sql-server-professionals-1st-edition-bob-ward/
textboxfull.com
SQL Server Data
Automation
Through
Frameworks
Building Metadata-Driven Frameworks with
T-SQL, SSIS, and Azure Data Factory
—
Andy Leonard
Kent Bradshaw
SQL Server Data
Automation Through
Frameworks
Building Metadata-Driven
Frameworks with T-SQL, SSIS,
and Azure Data Factory
Andy Leonard
Kent Bradshaw
SQL Server Data Automation Through Frameworks: Building Metadata-Driven
Frameworks with T-SQL, SSIS, and Azure Data Factory
Andy Leonard Kent Bradshaw
Farmville, VA, USA Providence Forge, VA, USA
For Ann
—Kent
Table of Contents
About the Authors���������������������������������������������������������������������������������������������������� xi
Introduction�����������������������������������������������������������������������������������������������������������xvii
v
Table of Contents
vi
Table of Contents
vii
Table of Contents
viii
Table of Contents
Index��������������������������������������������������������������������������������������������������������������������� 385
ix
About the Authors
Andy Leonard is Chief Data Engineer at Enterprise Data
& Analytics, creator and Data Philosopher at DILM (Data
Integration Lifecycle Management) Suite, an Azure Data
Factory and SQL Server Integration Services trainer and
consultant, and a BimlHero. He is a SQL Server database and
data warehouse developer, community mentor, engineer,
and farmer. Andy is coauthor of SQL Server Integration
Services Design Patterns, Data Integration Life Cycle
Management with SSIS, and The Biml Book.
xi
About the Technical Reviewer
André van Meulebrouck has a keen interest in functional
programming, especially Haskell and F#.
He also likes data technologies from markup languages
to databases and F# type providers.
He lives in Southern California with his wife “Tweety”,
and is active in athletics: hiking, mountain biking, and
gravity/balance sports like freestyle skating (in-line and ice),
skateboarding, surfing, and sandboarding.
To keep his mind sharp, he does compositional origami, plays classical guitar, and
enjoys music notation software.
xiii
Acknowledgments
This book would not have been possible without the help of Kent Bradshaw, my
coauthor, friend, and brother. This book represents Kent’s first foray into authoring, and
he delivered outstanding work.
Thanks André van Meulebrouck for proofreading, working through the samples, and
providing feedback on the manuscript.
I owe my coworkers at Enterprise Data & Analytics (EntDNA.com) a debt of gratitude
for their encouragement and for covering for me when I stayed up too late trying to
figure out how to make the code work. We have an awesome team, especially my
brothers from other mothers – Nick Harris, Shannon Lowder, and Reeves Smith.
Donald Farmer inspires me every time we interact. As Principal Program Manager at
Microsoft, Donald worked extensively with SSIS and helped shape the product. Donald
continues to shape data software by providing vendors unique strategic guidance at
TreeHive Strategy (treehivestrategy.com).
I am certain there are many excellent editors in this business. Jonathan Gennick is
the best with whom I have had the privilege to work. Without Jill Balzano’s input and
project management, I would have been lost!
Finally, I thank my family for their understanding. My children Stevie Ray, Emma, and
Riley who live at home at the time of this writing, and Manda and Penny who have children
of their own. And Christy, to whom I dedicate this book, my wife, my love. Thank you.
—Andy
Writing this book never really entered my mind until my great friend and EDNA
colleague, Andy Leonard, convinced me to do it. I truly appreciate his confidence in me,
and it was a very interesting experience. I’m really glad that I could work together with
him on this project.
Many thanks to Jonathan Gennick, Jill Balzano, and André van Meulebrouck for all of
their help and suggestions. They made things so much easier.
Most important of all, I want to thank my wife, Ann. Much of where I am today is due
to her love, support, and encouragement. I couldn’t begin to do it without her. Thank you.
—Kent
xv
Introduction
Frameworks have existed for as long as people have been building things. When
people began developing solutions using computers, frameworks soon followed. Like
woodworking jigs, frameworks exist to simplify work and speed up the process of
developing a solution, whether the problem is to complete a woodworking project or
deliver an enterprise data integration solution.
1. Database developers
Database developers will learn about driving database object execution from stored
procedures and will see one example of a database framework.
Data engineers and data integration developers will learn how to use SQL Server
Integration Services (SSIS) to implement a metadata-driven data integration framework.
Biml developers will see one example of using BimlExpress and metadata to
rapidly produce several SSIS packages in selected design patterns and an example of
interrogating flat files to generate flat file formats and flat file connection managers via
Biml.
xvii
Introduction
C
hapter 1: Stored Procedures 101
Before we dive into database development, let’s cover some basics of stored procedures.
C
hapter 2: Automation with Stored Procedures
Once you have a solid grasp on stored procedures, we will use what we’ve learned about
stored procedures to automate execution using the controller pattern, which executes
one or more stored procedures.
C
hapter 3: Stored Procedure Orchestrators
In this chapter, we examine high-level execution management using the orchestrator
pattern, which executes one or more controller stored procedures.
C
hapter 4: A Stored Procedure–Based Metadata-Driven
Framework
Metadata-driven frameworks are a recurring theme in this book. In this chapter, we
store information about stored procedures, controllers, and orchestrators to execute
framework applications.
xviii
Introduction
C
hapter 5: A Simple, Custom, File-Based SSIS Framework
In this chapter, we define a data integration/engineering framework metadata database
that manages configuration and execution. We introduce a version of an SSIS framework
that manages these subject areas for SSIS executed on-premises.
C
hapter 6: Framework Execution Engine
In this chapter, we add instrumentation in the form of Information and Error events.
Built-in SSIS logging will surface these messages, which are useful when troubleshooting
SSIS framework application issues.
C
hapter 7: Framework Logging
In this chapter, we persist execution metadata to “Instance” tables. The
ApplicationInstance table stores one record for each instance of an SSIS framework
application execution. The ApplicationPackageInstance table stores one record for each
instance of an SSIS framework application package execution. Important execution
instance attributes – such as start and end times and execution status – are maintained
in the “Instance” tables.
C
hapter 8: Azure-SSIS Integration Runtime
In this chapter, we introduce Azure Data Factory (ADF) and the Azure-SSIS integration
runtime by walking through the provisioning processes.
C
hapter 9: Deploy a Simple, Custom, File-Based Azure-SSIS
Framework
In Chapters 5–7, we designed and constructed an SSIS framework aimed at on-premises
SSIS execution, configuration, and logging. In this chapter, we begin migrating the SSIS
framework designed in Chapters 5–7 by provisioning a new Azure SQL database and
then deploying the metadata database to our new Azure SQL database. We provision an
Azure File Share and migrate test SSIS packages to the Azure File Share. Finally, we begin
building the Azure Data Factory parent pipeline that serves as the execution engine for
the ADF version of the SSIS framework.
xix
Introduction
C
hapter 10: Framework Logging in ADF
In this chapter, we add logging functionality to the parent ADF pipeline, much like the
functionality we added to the on-premises version of the SSIS framework in Chapter 7.
C
hapter 11: Fault Tolerance in the ADF Framework
In this chapter, we complete ADF execution engine functionality by implementing fault
tolerance to programmatically stop (or not stop) pipeline execution based on SSISConfig
metadata configurations.
C
onclusion
This book is for people who want to dive deeper into SQL Server and SSIS automation.
We will be discussing and demonstrating database and Integration Services frameworks
and covering such topics as SQL Server, SSIS, Azure, and Azure Data Factory.
As you can see, we have a long journey filled with learning and new insight ahead of
us. Let’s get started!
xx
PART I
Stored Procedure-Based
Database Frameworks
CHAPTER 1
3
© Andy Leonard, Kent Bradshaw 2020
A. Leonard and K. Bradshaw, SQL Server Data Automation Through Frameworks,
https://doi.org/10.1007/978-1-4842-6213-9_1
Chapter 1 Stored Procedures 101
Now, consider that monolith broken up into multiple procedures, each of which
performs a unit of work. When a change is made and needs to be tested, that effort can
be isolated to just what is necessary to perform that unit of work, and the validation is
concentrated on the result of the procedure. And, with it now being multiple procedures,
modifications can usually be done simultaneously by multiple developers, and their
efforts can be mutually exclusive. Over time, that approach can prove to be much more
cost effective and efficient.
That is where a framework helps to organize and manage processes to provide
the most flexibility in development and can minimize the maintenance effort (which,
sometimes, is not considered until it becomes an obvious issue). A framework provides
a consistent methodology for assembling and executing processes. It also promotes
writing code in small units of work that can potentially be mixed, matched, and reused.
It adds complexity to the development and deployment processes but can reduce the
effort for production scheduling. The framework can also provide greater flexibility for
managing the execution of the process.
D
emonstration of a Framework
To begin the analysis of the framework concept, we need a process. Our example to
follow shows a framework built to run a daily process against an example schema. The
details of that process don’t matter to the example. Just consider that any production
system might have something that needs to be done each day, and what follows is a
framework by which to make those daily processes happen.
Also, part of the example is a monthly process. Just as a system might need certain
tasks to be done each day, it’s also common to have certain things that need to be done
once monthly. In designing such a system, one must take into account the order in
which daily and monthly processes execute when their schedules intersect on – in our
example – the first of each month.
For the purpose of this book, a simple process has been developed (NOTE: all of the
code described can be downloaded at entdna.com. You can also find a link to the code
from the book’s catalog page on Apress.com). Downloading the example code enables
you to follow along with the upcoming examples on your own machine.
4
Chapter 1 Stored Procedures 101
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
5
Chapter 1 Stored Procedures 101
Do you have a SQL Server instance that you can use for learning purposes? Connect
to that instance as an administrator, for example, as the sa user. Then, in SQL Server
Management Studio (SSMS), open a “New Query” window, copy the code from Listing 1-1,
and execute it to create the example schema used in this and subsequent chapters.
6
Chapter 1 Stored Procedures 101
SET NOCOUNT ON
/*********************************************/
/* Log the START of the procedure to the process log */
/*********************************************/
7
Chapter 1 Stored Procedures 101
SET @RetStat = 0
/******************************************/
/* Force an ERROR CONDITION for this procedure */
/******************************************/
/****************************************************/
/* Log the COMPLETION of the procedure to the process log */
/****************************************************/
IF @RetStat = 0
BEGIN
INSERT INTO FWDemo.ProcessLog (
ProcessLogMessage,
CreateDate
)
VALUES ('Procedure FWDemo.DailyProcess1 - COMPLETED',
GETDATE()
)
8
Chapter 1 Stored Procedures 101
END
ELSE
BEGIN
INSERT INTO FWDemo.ProcessLog (
ProcessLogMessage,
CreateDate
)
VALUES ('Procedure FWDemo.DailyProcess1 - ERROR',
GETDATE()
)
END
RETURN @RetStat
GO
9
Chapter 1 Stored Procedures 101
SET NOCOUNT ON
/*********************************************/
/* Log the START of the procedure to the process log */
/*********************************************/
SET @RetStat = 0
/******************************************/
/* Force an ERROR CONDITION for this procedure */
/******************************************/
10
Chapter 1 Stored Procedures 101
--SET @RetStat = 1
/****************************************************/
/* Log the COMPLETION of the procedure to the process log */
/****************************************************/
IF @RetStat = 0
BEGIN
INSERT INTO FWDemo.ProcessLog (
ProcessLogMessage,
CreateDate
)
VALUES ('Procedure FWDemo.DailyProcess2 - COMPLETED',
GETDATE()
)
END
ELSE
BEGIN
INSERT INTO FWDemo.ProcessLog (
ProcessLogMessage,
CreateDate
)
VALUES ('Procedure FWDemo.DailyProcess2 - ERROR',
GETDATE()
)
END
RETURN @RetStat
GO
In a “New Query” window in SSMS, execute the code from Listing 1-2 while
connected to the FWDemo schema. The code creates two stored procedures that
together make up a daily process. With those procedures in place, you can turn your
attention to the next problem, which is to schedule those procedures to actually run each
day.
11
Chapter 1 Stored Procedures 101
Listing 1-3. Daily Process execute statements and process log SELECT statement
EXECUTE FWDemo.DailyProcess1
EXECUTE FWDemo.DailyProcess2
SELECT ProcessLogID
,ProcessLogMessage
,CreateDate
FROM FWDemo.ProcessLog
ORDER BY ProcessLogID desc
12
Chapter 1 Stored Procedures 101
SET NOCOUNT ON
/*********************************************/
/* Log the START of the procedure to the process log */
/******************************** ************/
13
Chapter 1 Stored Procedures 101
/******************************************/
/* Force an ERROR CONDITION for this procedure */
/******************************************/
/****************************************************/
/* Log the COMPLETION of the procedure to the process log */
/****************************************************/
IF @RetStat = 0
BEGIN
INSERT INTO FWDemo.ProcessLog (
ProcessLogMessage,
CreateDate
)
VALUES ('Procedure FWDemo.MonthlyProcess1 - COMPLETED',
GETDATE()
)
END
14
Another Random Scribd Document
with Unrelated Content
»Niin, niin», vastasi hän tehden kiihkeän liikkeen. »Miksi ette
sanonut minulle tätä? Miksi ette tunnustanut minulle, vaikkapa
viimeisessä hetkessä? Minä… mutta ei sanaakaan enää!» jatkoi hän
valittavalla äänellä ja koetti hoputtaa hevostaan eteenpäin. »Minä
olen kuullut kylliksi. Te haavoitatte sydäntäni, herra de Berault.
Vastedes rukoilen Jumalalta voimaa antamaan teille anteeksi.»
»Mutta te? Te?» Hänen äänensä oli niin muuttunut, että tuskin
tunsin sitä. »Mitä te aiotte tehdä? Minä en käsitä teitä, hyvä herra.»
XIII luku
Martinpäivän aatto
»Kyllä on.»
Vakuutin hänelle vielä kerran, että hän oli hupsu. Mutta siitä
huolimatta ja kaikkien järkeilyjeni jälkeenkin tunsin mieltäni
ahdistavan. Hän oli suuri mies, hän jos kukaan, ja kaikki hylkäsivät
hänet; ja minä — no, minulla ei ollut mitään syytä rakastaa häntä.
Mutta minä olin ottanut vastaan rahaa häneltä, olin ottanut
suorittaakseni häneltä asian ja jättänyt sen silleen. Jos hän kaatui
ennen kuin minä parhaimmalla tahdollanikaan ehdin täyttää
sitoumukseni hänelle, no niin — sitä parempi minulle. Se oli minun
voittoni — sotaonnea, sattuman suosiollisuutta. Mutta jos sitävastoin
piilouduin, otin ajan liittolaisekseni ja pysyttelin täällä hänen vielä
ollessaan jaloillaan, vaikkakin horjuen, niin kuinka silloin puhdistui
kunniani? Mitä arvoa oli silloin niillä suurilla sanoilla, joita olin
lausunut mademoisellelle Agenissa? Olisin samanlainen kuin
ritariromaanin pelkuriraukka, joka makasi ojassa taistelun riehuessa
ja sitten tuli esille kaiken päätyttyä kerskumaan urhoollisuudestaan.
Sitten kun olin tästä päässyt selville, läksin aivan tyynesti levolle.
Mutta päivän valjetessa heräsin, ja minun oli mitä vaikeinta pysyä
hiljaa, kunnes kuulin Frisonin nousseen makuulta. Huusin silloin
häntä, saadakseni tietää, kuuluiko mitään uutta, ja makasin odotellen
ja kuunnellen, hänen mentyänsä kaupungille tiedustelemaan.
Minusta tuntui kuin olisi kokonainen iankaikkisuus kulunut ennen
kuin hän palasi.
XIV luku
Jälkikesä
Niin, suuren kardinaalin vastaanottohuoneessa olin minä yksin
odottamassa! Katselin ympäri huonetta, pitkää kapeaa salia, jossa
hänen oli joka aamu tapana kävellä sen jälkeen kun oli
vastaanottanut arvokkaimmat suojattinsa. Tuijotin hämmentyneenä
joka suunnalle. Lavitsat seinien vierillä olivat tyhjät, ikkunakomerot
samaten. Sinne tänne sovitetut maalatut ja veistetyt kardinaalihatut,
isot R-kirjaimet ja vaakunakilvet katselivat alas tyhjälle lattialle.
Pienellä tuolilla peräoven luona istui kuitenkin mustaan puettu mies,
joka aivan rauhallisena luki tai oli lukevinaan pientä kirjaa eikä
kertaakaan vilkaissut sivulle. Tuollainen hidasverinen, veltto luonne,
jollaisia versoo suurten miesten varjossa.
»Niin», vastasin.
»Niin.»
»Niin.»
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
textbookfull.com