100% found this document useful (3 votes)
50 views

SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks With T-SQL, SSIS, and Azure Data Factory Andy Leonard

Automation

Uploaded by

thorelroad
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (3 votes)
50 views

SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks With T-SQL, SSIS, and Azure Data Factory Andy Leonard

Automation

Uploaded by

thorelroad
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 62

Download the full version of the textbook now at textbookfull.

com

SQL Server Data Automation Through


Frameworks: Building Metadata-Driven
Frameworks with T-SQL, SSIS, and Azure Data
Factory Andy Leonard
https://textbookfull.com/product/sql-server-data-
automation-through-frameworks-building-metadata-
driven-frameworks-with-t-sql-ssis-and-azure-data-
factory-andy-leonard/

Explore and download more textbook at https://textbookfull.com


Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

SQL Server Execution Plans For SQL Server 2008 through to


2017 and Azure SQL Database 3rd Edition Grant Fritchey

https://textbookfull.com/product/sql-server-execution-plans-for-sql-
server-2008-through-to-2017-and-azure-sql-database-3rd-edition-grant-
fritchey/
textbookfull.com

Building Custom Tasks for SQL Server Integration Services:


The Power of .NET for ETL for SQL Server 2019 and Beyond
2nd Edition Andy Leonard
https://textbookfull.com/product/building-custom-tasks-for-sql-server-
integration-services-the-power-of-net-for-etl-for-sql-server-2019-and-
beyond-2nd-edition-andy-leonard/
textbookfull.com

Building Custom Tasks for SQL Server Integration Services


The Power of NET for ETL for SQL Server 2019 and Beyond
Second Edition Andy Leonard
https://textbookfull.com/product/building-custom-tasks-for-sql-server-
integration-services-the-power-of-net-for-etl-for-sql-server-2019-and-
beyond-second-edition-andy-leonard/
textbookfull.com

UGC Psychology Paper Unit 10 10 1st Edition Manah Kamalam

https://textbookfull.com/product/ugc-psychology-paper-unit-10-10-1st-
edition-manah-kamalam/

textbookfull.com
New Advances in Mechanisms, Mechanical Transmissions and
Robotics : MTM & Robotics 2020 Erwin-Christian Lovasz

https://textbookfull.com/product/new-advances-in-mechanisms-
mechanical-transmissions-and-robotics-mtm-robotics-2020-erwin-
christian-lovasz/
textbookfull.com

SAP Gateway and OData Third Edition SAP PRESS Carsten


Bönnen

https://textbookfull.com/product/sap-gateway-and-odata-third-edition-
sap-press-carsten-bonnen/

textbookfull.com

Labov A Guide for the Perplexed Matthew J. Gordon

https://textbookfull.com/product/labov-a-guide-for-the-perplexed-
matthew-j-gordon/

textbookfull.com

Improving Abiotic Stress Tolerance in Plants 1st Edition


M. Iqbal R. Khan (Editor)

https://textbookfull.com/product/improving-abiotic-stress-tolerance-
in-plants-1st-edition-m-iqbal-r-khan-editor/

textbookfull.com

Effective C An introduction to professional C programming


1st Edition Robert C. Seacord

https://textbookfull.com/product/effective-c-an-introduction-to-
professional-c-programming-1st-edition-robert-c-seacord/

textbookfull.com
Design and Analysis of Experiments with R Ninth Edition
John Lawson

https://textbookfull.com/product/design-and-analysis-of-experiments-
with-r-ninth-edition-john-lawson/

textbookfull.com
SQL Server Data
Automation
Through
Frameworks
Building Metadata-Driven Frameworks with
T-SQL, SSIS, and Azure Data Factory

Andy Leonard
Kent Bradshaw
SQL Server Data
Automation Through
Frameworks
Building Metadata-Driven
Frameworks with T-SQL, SSIS,
and Azure Data Factory

Andy Leonard
Kent Bradshaw
SQL Server Data Automation Through Frameworks: Building Metadata-Driven
Frameworks with T-SQL, SSIS, and Azure Data Factory
Andy Leonard Kent Bradshaw
Farmville, VA, USA Providence Forge, VA, USA

ISBN-13 (pbk): 978-1-4842-6212-2 ISBN-13 (electronic): 978-1-4842-6213-9


https://doi.org/10.1007/978-1-4842-6213-9

Copyright © 2020 by Andy Leonard, Kent Bradshaw


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Jonathan Gennick
Development Editor: Laura Berendson
Coordinating Editor: Jill Balzano
Cover image designed by Freepik (www.freepik.com)
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-
sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a
Delaware corporation.
For information on translations, please e-mail [email protected]; for reprint,
paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/9781484262122. For more
detailed information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
For Christy
—Andy

For Ann
—Kent
Table of Contents
About the Authors���������������������������������������������������������������������������������������������������� xi

About the Technical Reviewer������������������������������������������������������������������������������� xiii


Acknowledgments���������������������������������������������������������������������������������������������������xv

Introduction�����������������������������������������������������������������������������������������������������������xvii

Part I: Stored Procedure-Based Database Frameworks������������������������������������������� 1

Chapter 1: Stored Procedures 101��������������������������������������������������������������������������� 3


The Need for a Framework����������������������������������������������������������������������������������������������������������� 3
Demonstration of a Framework���������������������������������������������������������������������������������������������������� 4
An Example Schema���������������������������������������������������������������������������������������������������������������� 5
The Daily Process�������������������������������������������������������������������������������������������������������������������� 6
Executing the Daily Process�������������������������������������������������������������������������������������������������� 12
Including a Monthly Process������������������������������������������������������������������������������������������������� 12
Summary������������������������������������������������������������������������������������������������������������������������������������ 18

Chapter 2: Automation with Stored Procedures����������������������������������������������������� 19


A Daily Process Controller����������������������������������������������������������������������������������������������������������� 19
Monthly Process Controller��������������������������������������������������������������������������������������������������������� 23
Summary������������������������������������������������������������������������������������������������������������������������������������ 26

Chapter 3: Stored Procedure Orchestrators����������������������������������������������������������� 27


The Orchestrator������������������������������������������������������������������������������������������������������������������������� 27
A Quick Process Overview���������������������������������������������������������������������������������������������������������� 31
A Look at Troubleshooting Issues������������������������������������������������������������������������������������������������ 31
Summary������������������������������������������������������������������������������������������������������������������������������������ 32

v
Table of Contents

Chapter 4: A Stored Procedure–Based Metadata-Driven Framework�������������������� 33


Building the Metadata����������������������������������������������������������������������������������������������������������������� 33
Metadata-Ready Controllers������������������������������������������������������������������������������������������������������� 40
Metadata-Ready Orchestrator����������������������������������������������������������������������������������������������������� 47
Troubleshooting with Metadata�������������������������������������������������������������������������������������������������� 50
Summary������������������������������������������������������������������������������������������������������������������������������������ 56

Part II: SSIS Frameworks���������������������������������������������������������������������������������������� 57

Chapter 5: A Simple, Custom, File-­Based SSIS Framework������������������������������������ 59


An SSIS Framework, Defined and Designed������������������������������������������������������������������������������� 60
Functionality�������������������������������������������������������������������������������������������������������������������������� 60
Empathy�������������������������������������������������������������������������������������������������������������������������������� 60
Simplicity������������������������������������������������������������������������������������������������������������������������������� 60
Building a File-Based SSIS Framework�������������������������������������������������������������������������������������� 61
Obtain the Code��������������������������������������������������������������������������������������������������������������������� 61
Metadata-Driven Execution Management����������������������������������������������������������������������������� 62
The SSISConfig Database������������������������������������������������������������������������������������������������������ 62
Config Schema���������������������������������������������������������������������������������������������������������������������� 63
The Config.Applications Table������������������������������������������������������������������������������������������������ 65
The Config.Packages Table���������������������������������������������������������������������������������������������������� 67
A Sample SSIS Solution�������������������������������������������������������������������������������������������������������������� 75
Configure Visual Studio 2019 for SSIS Development������������������������������������������������������������� 76
Create the Sample SSIS Solution������������������������������������������������������������������������������������������ 78
SSIS Framework Metadata Management����������������������������������������������������������������������������������� 85
Add an SSIS Application�������������������������������������������������������������������������������������������������������� 85
Add SSIS Packages��������������������������������������������������������������������������������������������������������������� 88
Assign SSIS Application Packages���������������������������������������������������������������������������������������� 91
Conclusion���������������������������������������������������������������������������������������������������������������������������������� 96

vi
Table of Contents

Chapter 6: Framework Execution Engine��������������������������������������������������������������� 97


Create a Parent SSIS Package���������������������������������������������������������������������������������������������������� 97
Log Execution Values������������������������������������������������������������������������������������������������������������������ 98
Retrieve SSIS Application Packages from SSISConfig�������������������������������������������������������������� 104
Iterate Application Packages����������������������������������������������������������������������������������������������������� 120
Log Application Package Values����������������������������������������������������������������������������������������������� 130
Execute Application Packages�������������������������������������������������������������������������������������������������� 135
Log Execution Results��������������������������������������������������������������������������������������������������������������� 150
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 158

Chapter 7: Framework Logging���������������������������������������������������������������������������� 159


Create a Log Schema���������������������������������������������������������������������������������������������������������������� 159
Add Application Instance Logging to Parent.dtsx���������������������������������������������������������������������� 164
Add Application Package Instance Logging to Parent.dtsx������������������������������������������������������� 176
Viewing Execution Report Data������������������������������������������������������������������������������������������������� 191
Configure Fault Tolerance in SSIS Framework Application Metadata��������������������������������������� 197
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 8: Azure-SSIS Integration Runtime��������������������������������������������������������� 207


Getting Started with Azure�������������������������������������������������������������������������������������������������������� 207
Provisioning an Azure Data Factory������������������������������������������������������������������������������������������ 208
Provisioning Azure Storage������������������������������������������������������������������������������������������������������� 217
Provisioning an Azure-SSIS Integration Runtime for SSIS Package Files��������������������������������� 223
Stopping the Azure-SSIS Integration Runtime�������������������������������������������������������������������������� 236
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 240

Chapter 9: Deploy a Simple, Custom, File-Based Azure-SSIS Framework������������ 241


Provisioning the SSISConfig Database�������������������������������������������������������������������������������������� 241
Deploy the Simple, Custom, File-Based Azure-SSIS Framework���������������������������������������������� 249
Provision an Azure File Share��������������������������������������������������������������������������������������������������� 267
Upload SSIS Packages��������������������������������������������������������������������������������������������������������� 270
Update PackageLocation Values������������������������������������������������������������������������������������������ 272

vii
Table of Contents

Build the SSIS Framework ADF Execution Engine�������������������������������������������������������������������� 273


Retrieve a List of SSIS Packages����������������������������������������������������������������������������������������� 273
Execute the Retrieved SSIS Packages��������������������������������������������������������������������������������� 287
View Test Execution Logs���������������������������������������������������������������������������������������������������� 296
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 299

Chapter 10: Framework Logging in ADF��������������������������������������������������������������� 301


Add the ApplicationName Parameter���������������������������������������������������������������������������������������� 301
A Quick Review of SSIS Framework Applications and Packages���������������������������������������� 302
Add Application Instance Logging��������������������������������������������������������������������������������������������� 303
Modifying the Log.ApplicationInstance Table���������������������������������������������������������������������� 303
Adding the Log.InsertApplicationInstance Stored Procedure���������������������������������������������� 304
Logging Application Instance���������������������������������������������������������������������������������������������� 306
Adding the Log.UpdateApplicationInstanceStatus Stored Procedure���������������������������������� 311
Updating Application Instance��������������������������������������������������������������������������������������������� 312
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 316
Updating Application Instance on Success�������������������������������������������������������������������������� 317
Adding the ApplicationPackageId Field������������������������������������������������������������������������������� 320
Adding the ApplicationInstanceId Pipeline Variable������������������������������������������������������������� 321
Add the Child Pipeline��������������������������������������������������������������������������������������������������������� 325
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 333
Add Application Package Instance Logging������������������������������������������������������������������������������ 334
Modifying the Log.ApplicationPackageInstance Table��������������������������������������������������������� 335
Adding the Log.InsertApplicationPackageInstance Stored Procedure��������������������������������� 336
Logging Application Package Instance�������������������������������������������������������������������������������� 338
Adding the Log.UpdateApplicationPackageInstanceStatus Stored Procedure�������������������� 344
Updating Application Package Instance������������������������������������������������������������������������������ 345
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 348
Updating Application Package Instance on Success����������������������������������������������������������� 350
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 353

viii
Table of Contents

Chapter 11: Fault Tolerance in the ADF Framework��������������������������������������������� 355


A Brief Introduction to Fault Tolerance�������������������������������������������������������������������������������������� 355
Add ADF Managed Identity to Contributor Role������������������������������������������������������������������������� 356
Add Application Package Fault Tolerance��������������������������������������������������������������������������������� 359
About Fail Application on Package Failure�������������������������������������������������������������������������� 359
Implementing Application Package Fault Tolerance������������������������������������������������������������ 359
Checking the Logic�������������������������������������������������������������������������������������������������������������� 362
Implementing the Fault Tolerance Logic������������������������������������������������������������������������������ 363
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 378
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 383

Index��������������������������������������������������������������������������������������������������������������������� 385

ix
About the Authors
Andy Leonard is Chief Data Engineer at Enterprise Data
& Analytics, creator and Data Philosopher at DILM (Data
Integration Lifecycle Management) Suite, an Azure Data
Factory and SQL Server Integration Services trainer and
consultant, and a BimlHero. He is a SQL Server database and
data warehouse developer, community mentor, engineer,
and farmer. Andy is coauthor of SQL Server Integration
Services Design Patterns, Data Integration Life Cycle
Management with SSIS, and The Biml Book.

Kent Bradshaw is the founder or Tudor Data Solutions,


LLC. With over 40 years of IT experience, he is a SQL
Server database/ETL developer and database architect
with a background in Medicaid claims, public schools,
government, retail, and insurance systems. In 2011,
Kent founded Tudor Data Solutions, LLC, to pursue new
development opportunities which led to his association with
Andy Leonard and Enterprise Data & Analytics. In 2017, Kent
received the MPP certification for Data Science.

xi
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
About the Technical Reviewer
André van Meulebrouck has a keen interest in functional
programming, especially Haskell and F#.
He also likes data technologies from markup languages
to databases and F# type providers.
He lives in Southern California with his wife “Tweety”,
and is active in athletics: hiking, mountain biking, and
gravity/balance sports like freestyle skating (in-line and ice),
skateboarding, surfing, and sandboarding.
To keep his mind sharp, he does compositional origami, plays classical guitar, and
enjoys music notation software.

xiii
Acknowledgments
This book would not have been possible without the help of Kent Bradshaw, my
coauthor, friend, and brother. This book represents Kent’s first foray into authoring, and
he delivered outstanding work.
Thanks André van Meulebrouck for proofreading, working through the samples, and
providing feedback on the manuscript.
I owe my coworkers at Enterprise Data & Analytics (EntDNA.com) a debt of gratitude
for their encouragement and for covering for me when I stayed up too late trying to
figure out how to make the code work. We have an awesome team, especially my
brothers from other mothers – Nick Harris, Shannon Lowder, and Reeves Smith.
Donald Farmer inspires me every time we interact. As Principal Program Manager at
Microsoft, Donald worked extensively with SSIS and helped shape the product. Donald
continues to shape data software by providing vendors unique strategic guidance at
TreeHive Strategy (treehivestrategy.com).
I am certain there are many excellent editors in this business. Jonathan Gennick is
the best with whom I have had the privilege to work. Without Jill Balzano’s input and
project management, I would have been lost!
Finally, I thank my family for their understanding. My children Stevie Ray, Emma, and
Riley who live at home at the time of this writing, and Manda and Penny who have children
of their own. And Christy, to whom I dedicate this book, my wife, my love. Thank you.
—Andy

Writing this book never really entered my mind until my great friend and EDNA
colleague, Andy Leonard, convinced me to do it. I truly appreciate his confidence in me,
and it was a very interesting experience. I’m really glad that I could work together with
him on this project.
Many thanks to Jonathan Gennick, Jill Balzano, and André van Meulebrouck for all of
their help and suggestions. They made things so much easier.
Most important of all, I want to thank my wife, Ann. Much of where I am today is due
to her love, support, and encouragement. I couldn’t begin to do it without her. Thank you.

—Kent

xv
Introduction
Frameworks have existed for as long as people have been building things. When
people began developing solutions using computers, frameworks soon followed. Like
woodworking jigs, frameworks exist to simplify work and speed up the process of
developing a solution, whether the problem is to complete a woodworking project or
deliver an enterprise data integration solution.

Who Is This Book For?


Anyone who is interested in data-related automation or executing multiple chunks
of code with a single command will find value in these pages. Three audiences will
particularly benefit from this book:

1. Database developers

2. Data engineers and data integration developers

3. Business Intelligence Markup Language (Biml) developers

Database developers will learn about driving database object execution from stored
procedures and will see one example of a database framework.
Data engineers and data integration developers will learn how to use SQL Server
Integration Services (SSIS) to implement a metadata-driven data integration framework.
Biml developers will see one example of using BimlExpress and metadata to
rapidly produce several SSIS packages in selected design patterns and an example of
interrogating flat files to generate flat file formats and flat file connection managers via
Biml.

How This Book Is Structured


In the following sections, we will explore the chapter layout of this book. There are two
parts that guide you through the process of building a database framework using stored
procedures and an SSIS framework. Each of those parts is composed of several chapters.

xvii
Introduction

Part I: Stored Procedure–Based Database Frameworks


In the first part of this book, we will focus on building a database framework using
stored procedures. These chapters are targeted primarily at those who develop database
solutions. We do not endeavor to create the most comprehensive primer for every aspect
of database development; we cover the basics well.

C
 hapter 1: Stored Procedures 101
Before we dive into database development, let’s cover some basics of stored procedures.

C
 hapter 2: Automation with Stored Procedures
Once you have a solid grasp on stored procedures, we will use what we’ve learned about
stored procedures to automate execution using the controller pattern, which executes
one or more stored procedures.

C
 hapter 3: Stored Procedure Orchestrators
In this chapter, we examine high-level execution management using the orchestrator
pattern, which executes one or more controller stored procedures.

C
 hapter 4: A Stored Procedure–Based Metadata-Driven
Framework
Metadata-driven frameworks are a recurring theme in this book. In this chapter, we
store information about stored procedures, controllers, and orchestrators to execute
framework applications.

Part II: SSIS Frameworks


If your enterprise is executing a few SSIS packages, there is no need for an SSIS
framework. However, if your enterprise is executing several hundred (or several
thousand) SSIS packages, you definitely need an SSIS framework.

xviii
Introduction

C
 hapter 5: A Simple, Custom, File-Based SSIS Framework
In this chapter, we define a data integration/engineering framework metadata database
that manages configuration and execution. We introduce a version of an SSIS framework
that manages these subject areas for SSIS executed on-premises.

C
 hapter 6: Framework Execution Engine
In this chapter, we add instrumentation in the form of Information and Error events.
Built-in SSIS logging will surface these messages, which are useful when troubleshooting
SSIS framework application issues.

C
 hapter 7: Framework Logging
In this chapter, we persist execution metadata to “Instance” tables. The
ApplicationInstance table stores one record for each instance of an SSIS framework
application execution. The ApplicationPackageInstance table stores one record for each
instance of an SSIS framework application package execution. Important execution
instance attributes – such as start and end times and execution status – are maintained
in the “Instance” tables.

C
 hapter 8: Azure-SSIS Integration Runtime
In this chapter, we introduce Azure Data Factory (ADF) and the Azure-SSIS integration
runtime by walking through the provisioning processes.

C
 hapter 9: Deploy a Simple, Custom, File-Based Azure-SSIS
Framework
In Chapters 5–7, we designed and constructed an SSIS framework aimed at on-premises
SSIS execution, configuration, and logging. In this chapter, we begin migrating the SSIS
framework designed in Chapters 5–7 by provisioning a new Azure SQL database and
then deploying the metadata database to our new Azure SQL database. We provision an
Azure File Share and migrate test SSIS packages to the Azure File Share. Finally, we begin
building the Azure Data Factory parent pipeline that serves as the execution engine for
the ADF version of the SSIS framework.

xix
Introduction

C
 hapter 10: Framework Logging in ADF
In this chapter, we add logging functionality to the parent ADF pipeline, much like the
functionality we added to the on-premises version of the SSIS framework in Chapter 7.

C
 hapter 11: Fault Tolerance in the ADF Framework
In this chapter, we complete ADF execution engine functionality by implementing fault
tolerance to programmatically stop (or not stop) pipeline execution based on SSISConfig
metadata configurations.

C
 onclusion
This book is for people who want to dive deeper into SQL Server and SSIS automation.
We will be discussing and demonstrating database and Integration Services frameworks
and covering such topics as SQL Server, SSIS, Azure, and Azure Data Factory.
As you can see, we have a long journey filled with learning and new insight ahead of
us. Let’s get started!

xx
PART I

Stored Procedure-Based
Database Frameworks
CHAPTER 1

Stored Procedures 101


One of the most common issues continually facing IT organizations is finding the proper
balance between the effort to develop and deploy processes into production against the
efficiency and effectiveness of production control operators. The effort to develop and
deploy and the effectiveness of production control seem to be diametrically opposed.
Making it easier to develop and deploy processes usually means more work and manual
intervention for production control. The real question that needs to be considered is
where is it better to “feel the pain”? Pushing the effort toward the development side of
the equation can slow down the throughput but minimizes the liability of issues at the
production process level. This and the next few chapters are going to concentrate on
processes that are executed using stored procedures. In this chapter, you’ll get a basic
introduction to stored procedures that is the foundation for the chapters that follow.
You’ll see how to create a child stored procedure, and we’ll provide you a template that
you can use to create similar procedures in your own work.

The Need for a Framework


When just in development mode, it is awfully easy (and impressive) to construct
large, monolithic procedures that do everything from beginning to end. They are the
proverbial “black box” where something goes in, many gyrations take place, and then the
desired result occurs. Those are great until a problem occurs, or just a change in business
requirements means that modifications need to be made. When the procedure does
so much, what does it take to test it once the modifications are done? Even though the
change only impacts 10 percent of the procedure, the entire procedure has to be tested.
What does it take to accomplish that? What if there are several intricate modifications
that need to be made? How difficult does it become for more than one developer to work
on the changes and coordinate their efforts?

3
© Andy Leonard, Kent Bradshaw 2020
A. Leonard and K. Bradshaw, SQL Server Data Automation Through Frameworks,
https://doi.org/10.1007/978-1-4842-6213-9_1
Chapter 1 Stored Procedures 101

Now, consider that monolith broken up into multiple procedures, each of which
performs a unit of work. When a change is made and needs to be tested, that effort can
be isolated to just what is necessary to perform that unit of work, and the validation is
concentrated on the result of the procedure. And, with it now being multiple procedures,
modifications can usually be done simultaneously by multiple developers, and their
efforts can be mutually exclusive. Over time, that approach can prove to be much more
cost effective and efficient.
That is where a framework helps to organize and manage processes to provide
the most flexibility in development and can minimize the maintenance effort (which,
sometimes, is not considered until it becomes an obvious issue). A framework provides
a consistent methodology for assembling and executing processes. It also promotes
writing code in small units of work that can potentially be mixed, matched, and reused.
It adds complexity to the development and deployment processes but can reduce the
effort for production scheduling. The framework can also provide greater flexibility for
managing the execution of the process.

D
 emonstration of a Framework
To begin the analysis of the framework concept, we need a process. Our example to
follow shows a framework built to run a daily process against an example schema. The
details of that process don’t matter to the example. Just consider that any production
system might have something that needs to be done each day, and what follows is a
framework by which to make those daily processes happen.
Also, part of the example is a monthly process. Just as a system might need certain
tasks to be done each day, it’s also common to have certain things that need to be done
once monthly. In designing such a system, one must take into account the order in
which daily and monthly processes execute when their schedules intersect on – in our
example – the first of each month.
For the purpose of this book, a simple process has been developed (NOTE: all of the
code described can be downloaded at entdna.com. You can also find a link to the code
from the book’s catalog page on Apress.com). Downloading the example code enables
you to follow along with the upcoming examples on your own machine.

4
Chapter 1 Stored Procedures 101

An Example Schema


Listing 1-1 shows code to create a schema called FWDemo that will contain everything
needed for the demonstration. Also, there is code to create a table called FWDemo.
ProcessLog. Including a pattern for writing to this table throughout all of the procedures
certainly adds some complexity and overhead to the procedures, but what it provides in
monitoring and troubleshooting more than makes up for the upfront effort.

Listing 1-1. Schema and log table creation

print 'FWDemo Schema'


If Not Exists(Select name
              From sys.schemas
              Where name='FWDemo')
begin
  print ' - Creating FWDemo schema'
  declare @sql varchar(255) = 'Create Schema FWDemo'
  exec(@sql)
  print ' - FWDemo schema created'
end
Else
print ' - FWDemo schema already exists.'
print ''
GO

IF  EXISTS (SELECT * FROM sys.objects


            WHERE object_id = OBJECT_ID(N'FWDemo.ProcessLog')
              AND type in (N'U'))
    DROP TABLE FWDemo.ProcessLog
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO
5
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Chapter 1 Stored Procedures 101

CREATE TABLE [FWDemo].[ProcessLog](


      [ProcessLogID]      [int] IDENTITY(1,1) NOT NULL,
      [ProcessLogMessage] [nvarchar](255)     NOT NULL,
      [CreateDate]        [smalldatetime]     NOT NULL
)
GO

SET ANSI_PADDING OFF


GO

Do you have a SQL Server instance that you can use for learning purposes? Connect
to that instance as an administrator, for example, as the sa user. Then, in SQL Server
Management Studio (SSMS), open a “New Query” window, copy the code from Listing 1-1,
and execute it to create the example schema used in this and subsequent chapters.

The Daily Process


Listing 1-2 shows the code to create two stored procedures that will make up our
demonstration Daily Process. We are providing two procedures in our example because
it’s common to have more than one, and having two allows us to show how to make the
execution of subsequent procedures depend upon the success of earlier ones – because
the need to execute a series of procedures and halt or take other actions when an error
occurs is the real-life scenario that most of us face.
These procedures (as well as all others that we will be using) can be compiled and
executed for your own testing. You will notice that there is some code commented out
(lines that are preceded with ‘--’) in each procedure that can be invoked (remove the
‘--’, then recompile) to create an error condition. This ability to create an error condition
allows testing for successful and unsuccessful completions that will become more
important as we progress through demonstration iterations in later chapters.
For the sake of this exercise, we will declare a business rule for the Daily Process
stating that FWDemo.DailyProcess1 must complete successfully before FWDemo.
DailyProcess2 can be executed. FWDemo.DailyProcess2 must then complete
successfully before the Daily Process can be deemed successfully executed.

6
Chapter 1 Stored Procedures 101

Listing 1-2. Daily Process stored procedures

If Exists(Select s.name + '.' + p.name


          From sys.procedures p
          Join sys.schemas s
            On s.schema_id = p.schema_id
          Where s.name = 'FWDemo'
            And p.name = 'DailyProcess1')
begin
  print ' - Dropping FWDemo.DailyProcess1 stored procedure'
  Drop Procedure FWDemo.DailyProcess1
  print ' - FWDemo.DailyProcess1 stored procedure dropped'
end
GO

CREATE PROCEDURE FWDemo.DailyProcess1


AS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
--
-- Purpose: This procedure is part of the Stored Procedure Framework Demo.
--
-- NOTE: An Error situation can be created for testing/demo purposes by
--    un-commenting the Error code in the body of the procedure.  To return
--    to a procedure with a successful execution, re-comment the code or    
--    recompile the original.
--
---------------------------------------------------------------------------
---------------------------------------------------------------------------

SET NOCOUNT ON

/*********************************************/
/*  Log the START of the procedure to the process log  */
/*********************************************/

7
Chapter 1 Stored Procedures 101

INSERT INTO FWDemo.ProcessLog (


      ProcessLogMessage,
      CreateDate
)
Values ('Procedure FWDemo.DailyProcess1 - STARTING',
      GETDATE()
)

DECLARE @RetStat int

SET @RetStat = 0

/******************************************/
/*  Force an ERROR CONDITION for this procedure  */
/******************************************/

--INSERT INTO FWDemo.ProcessLog (


--    ProcessLogMessage,
--    CreateDate
--)
--VALUES ('Procedure FWDemo.DailyProcess1 - Problem Encountered',
--    GETDATE()
--)
--SET @RetStat = 1

/****************************************************/
/*  Log the COMPLETION of the procedure to the process log     */
/****************************************************/

IF @RetStat = 0
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess1 - COMPLETED',
            GETDATE()
      )

8
Chapter 1 Stored Procedures 101

   END
ELSE
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess1 - ERROR',
      GETDATE()
      )
   END

RETURN @RetStat
GO

If Exists(Select s.name + '.' + p.name


          From sys.procedures p
          Join sys.schemas s
            On s.schema_id = p.schema_id
          Where s.name = 'FWDemo'
            And p.name = 'DailyProcess2')
begin
  print ' - Dropping FWDemo.DailyProcess2 stored procedure'
  Drop Procedure FWDemo.DailyProcess2
  print ' - FWDemo.DailyProcess2 stored procedure dropped'
end
GO

CREATE PROCEDURE FWDemo.DailyProcess2


AS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
--
-- Purpose: This procedure is part of the Stored Procedure Framework Demo.
--

9
Chapter 1 Stored Procedures 101

-- NOTE: An Error situation can be created for testing/demo purposes by


--    un-commenting the Error code in the body of the procedure.  To return
--    to a procedure with a successful execution, re-comment the code or
--    recompile the original.
--
---------------------------------------------------------------------------
---------------------------------------------------------------------------

SET NOCOUNT ON

/*********************************************/
/*  Log the START of the procedure to the process log  */
/*********************************************/

INSERT INTO FWDemo.ProcessLog (


      ProcessLogMessage,
      CreateDate
)
Values ('Procedure FWDemo.DailyProcess2 - STARTING',
      GETDATE()
)

DECLARE @RetStat int

SET @RetStat = 0

/******************************************/
/*  Force an ERROR CONDITION for this procedure  */
/******************************************/

--INSERT INTO FWDemo.ProcessLog (


--    ProcessLogMessage,
--    CreateDate
--)
--VALUES ('Procedure FWDemo.DailyProcess2 - Problem Encountered',
--    GETDATE()
--)

10
Random documents with unrelated
content Scribd suggests to you:
salt provisions, and using no exercise, had sometimes occasion for
art to assist nature. But these briny meals of pork and beef, Colonel
Fourgeoud insisted were much more wholesome food in a tropical
country than fresh provisions; for, by a most curious theory, he
asserted that the latter corrupted in the stomach by the heat,
whereas the others underwent a proper digestion. Unfortunately for
us, there were but few on board either the Cerberus or the Charon,
whose stomachs were in a state to digest such food. I had also
some plaisters on board the barge, but these were soon expended
by the running ulcers, with which the whole crew was covered; and
this was easily accounted for, since in this climate, [160]where the air
is impregnated with myriads of invisible animalcula, the smallest
scratch immediately becomes a running sore. The best antidote and
cure for such complaints in this country is lemon or lime acid, but
this we had not. The next best mode of treatment is never to expose
an open wound, or even the smallest scratch, to the air; but the
instant they are received, to cover them with grey paper wetted with
spirits, or any kind of moisture, so that it may stick to the skin. For
my own part, no man could enjoy a better state of health than I
continued to enjoy, wearing nothing but my long trowsers, and
checked shirt loose at the collar and turned up in the sleeves. Nay,
even when the sun was not too hot, I stripped all together, and
twice every day continued to plunge into the water: by these means
I was always cool and clean; I also daily used a cheering glass of
wine, having first hung it a few fathoms under water, which
rendered it much more cool and pleasant.

During all these hardships, I must not forget the high gratification
which was afforded us one day by a few marcusas that we found in
this place, which had been left there standing ever since the estate,
many years ago, had been destroyed. There was indeed but one
single old tree, I should rather say a shrub, for the plant which bears
them falls more properly under that description. This delicious fruit is
of an oval form, and of an orange or golden colour; the blossom
resembling the passion-flower. They are commonly larger, but some
less than [161]a hen’s egg, and are broken open as one would break
an egg; they are then found to inclose an ash-coloured succulent
jelly, full of small seeds. This is sipped out of the shell, being sweet
mixed with acid, of an exquisite flavour, and so cool that it reminds
one of ice marmalade.

Here we remarked a variety of beautiful butterflies, particularly some


of an azure blue, which are exceedingly large, and between the
showers skimmed and hovered amongst the green boughs, to which
their ultramarine hue, brightened by the sun, bore the most
enchanting contrast: but as I could not catch one of them while I
staid here, I must defer the farther description to another part of the
work.

This evening we heard the sound of a drum, which we could


suppose to be no other than that of the rebels; nevertheless we
determined to continue dressing our victuals ashore, still keeping on
our guard, according to the advice of Mr. Kleynhans.

On the 9th Mr. Macdonald was much worse; however, on seeing me


receive a letter from Colonel Fourgeoud, he seemed to revive, as we
all did, expecting now to be relieved from our horrid situation. But
what was our mortification, at reading that we were still to continue
on this forlorn station! This letter was accompanied by a present of
fish-hooks and tackle, to make up for the deficiency of other
refreshments; and, indeed, of salt [162]provisions, which began daily
to get both worse and less—

Timeo Danæos et dona ferentes.

The receipt of such unwelcome intelligence made the whole crew


declare they were sacrificed for no manner of purpose; while the
negroes sighed, pronouncing the words, Ah! poty backera! Oh! poor
Europeans! By the distribution however of a few tamarinds, oranges,
lemons, and Madeira wine, which were by this opportunity sent me
by my best friend at Paramaribo, I found means to impart, not only
to my officers, but also to my drooping soldiers, some relief. But this
cheering sun-shine could not last long: and the day following we
were as much distressed as ever, when I had once more recourse to
the nimble inhabitants of the forest, and brought down two monkies
with my gun from the top of the mangroves, where they were
sporting in flocks consisting of some hundreds.

On the 11th, I sent two men sick to the hospital, and the same
evening we again heard the drums. On the following day, at noon,
we were disturbed by a hurricane; the Charon broke loose from her
anchors, and was driven ashore, her upper works being terribly
damaged by the stumps of trees, &c. that hung over the river, while
the water from the clouds broke in upon us like a torrent, and I
expected no less than a shipwreck. [163]

On the 15th the other officer, Lieutenant Baron Owen, came down
sick from the Cerberus, and at his request I ventured to send him
down to Paramaribo. I now received another letter from Colonel
Fourgeoud, with a little money for the men to purchase
refreshments, where nothing was to be met with: but not a word
intimating that we were to be relieved.

On the 20th I received a report, that the Cerberus, having only four
private men left, had retired to the post at La Rochelle; and, on the
21st, I sent two of my men to her assistance, and ordered her back
to her former station.

I now was myself at last attacked with a fever, and upon the whole
felt myself in a truly distressed condition; deprived by sickness of my
two only officers, and my serjeant. My men upon the three stations
(viz. the two barges and Devil’s Harwar together) melted down to
fifteen, from the number of forty-two, without a surgeon, or
refreshment, surrounded with a black forest, and exposed to the
mercy of relentless enemies; who must be formidable indeed, should
they be informed by any means of our defenceless situation. The
remaining few were, with truth, declaring they were doomed to
destruction; insomuch, that they could with difficulty be prevented
from mutiny, and from proceeding down the river Cottica with the
Charon against my orders.

For my own part, I was not altogether free from uneasiness. In fact,
a few troops from all quarters ought [164]to have marched against
the enemy, when they crossed the Patamaca Creek; that is, from La
Rochelle, Devil’s Harwar, and the river Pirica; when, being assailed at
once from three quarters, the rebels might have been, if not entirely
routed, at least severely punished for their presumption: not to
speak of the happy effect which such a check must have had in
saving the lives and property of those victims, who, after such
incursions, are generally devoted to their rage.

On the 23d I felt myself rather better; and between the fits of the
fever shot a couple of large black monkies to make some broth, to
supply the want of fresh provisions; but as the destruction of one of
these animals was attended with such circumstances, as for ever
after had almost deterred me from going a monkey-hunting, I must
beg leave to relate them as they happened:—Seeing me on the side
of the river in the canoe, the creature made a halt from skipping
after his companions, and being perched on a branch that hung over
the water, examined me with attention, and the strongest marks of
curiosity: no doubt, taking me for a giant of his own species; while
he chattered prodigiously, and kept dancing and shaking the bough
on which he rested with incredible strength and agility. At this time I
laid my piece to my shoulder, and brought him down from the tree
into the stream;—but may I never again be a witness to such a
scene! The miserable animal was not dead, but mortally wounded. I
seized him by the tail, and taking it in both [165]my hands to end his
torment, I swung him round, and hit his head against the side of the
canoe; but the poor creature still continued alive, and looking at me
in the most affecting manner that can be conceived, I knew no other
means to end this murder, than to hold him under water till he was
drowned, while my heart felt sick on his account: for his dying little
eyes still continued to follow me with seeming reproach, till their
light gradually forsook them, and the wretched animal expired. I felt
so much on this occasion, that I could neither taste of him nor his
companion, when they were dressed, though I saw that they
afforded to some others a delicious repast.

That monkies, especially when young, are no bad food, may easily
be accounted for, since they feed on nothing but fruits, nuts, eggs,
young birds, &c.; and indeed, in my opinion, all young quadrupeds
are eatable: but when one compares those which are killed in the
woods to those filthy and disgusting creatures that disgrace the
streets, no wonder that they should disgust the least delicate
stomach. As for the wild ones, I have eaten them boiled, roasted,
and stewed, and found their flesh white, juicy, and good: the only
thing that disgusted me was, their little hands and their heads,
which when dressed, being deprived of the skin, appeared like the
hands and the skull of a young infant. I have already observed that
there are in Guiana many different species, from the [166]large
ourang-outang to the very small sarcawinkee. The former, however, I
never have seen, nor heard described, while I was in this country; as
for the latter, I shall describe him on another occasion, and shall
only, for the present, give an account of those which I met with on
this cruise. That which I shot the second instant is what is called in
Surinam micoo: it is nearly the size of a fox, and of a reddish grey
colour, with a black head and very long tail. Those I killed on the
tenth were indeed exceedingly beautiful, and much more delicate
when dressed than the former: they are called the keesee-keesee by
the inhabitants, are about the size of a rabbit, and most
astonishingly nimble. The colour of their body is reddish, and the
tail, which is long, is black at the extremity; but the fore-feet are
orange colour. The head is very round, the face milk white, with a
round black patch in the middle, in which are the mouth and the
nostrils; and this disposition of the features give it the appearance of
a mask: the eyes are black, and remarkably lively. These monkies we
saw daily pass along the sides of the river, skipping from tree to
tree, but mostly about mid-day, and in very numerous bodies,
regularly following each other like a little army, with their young
ones on their backs, not unlike little knapsacks. Their manner of
travelling is thus: the foremost walks to the extremity of a bough,
from which it bounds to the extremity of one belonging to the next
tree, often at a [167]most astonishing distance, and with such
wonderful activity and precision, that it never once misses its aim:
the others one by one, and even the females, with their little ones
on their backs, which stick fast to the mother, follow their leader, and
perform the same leap with the greatest seeming facility and safety:
they also are remarkable for climbing up the nebees or natural
ropes, with which many parts of the forest are interwoven. These
nebees grow in such a manner as to afford the appearance at first
sight of a fleet at anchor.
Blake Sculpt.
The Mecoo & Kishee Kishee Monkeys.

London, Published Decr. 2nd. 1793, by J. Johnson, St. Pauls


Church Yard.
The monkies, I am told, have sometimes two young ones sucking,
like the human species. I have been a witness how these animals
towards sun-set ascend to the tops of the palm-trees, some of which
are above one hundred feet in height, where they sleep safe in the
large diverging branches. The keesee-keesee is such a beautiful and
delicate fond creature, that it is by many people kept as a favourite,
when it wears a silver chain. Besides its mimic drolleries, it is
remarkable for its good-nature and chirruping voice, which
pronounces peeteeco peeteeco without intermission. They are easily
tamed, and are taken by means of a strong glue made by the
Indians, which is something like our European bird-lime.

The other species, of the shooting of which I have just given the
horrid account, were called by my negroes monkee-monkee. The
only facts which I can relate farther of them is, that they are in size
between the two former described, and all over black. One
circumstance I ought [168]not to omit, which is very remarkable, viz.
that one morning I saw from my barge a monkey of this kind come
down to the water’s edge, rinsing his mouth, and appearing to clean
his teeth with one of his fingers: he was first discovered by one of
the slaves, who pointed him out to my great amusement.

Here I shall end the subject for the present, after adding that the
above animals are sociable, and that they are very tenacious of life,
as I have shewn. It is almost superfluous to mention, that the usual
distinction between what are called monkies and apes, consists in
this, that the first have all tails, of which the latter are divested; but
never having met with in Guiana any of the latter description, I
believe them more to be the inhabitants of Asia and Africa, than of
the part of the new world distinguished by the name of South
America, The monkies are often mischievous near the plantations,
where they commit depredations on the sugar-canes, &c. yet of this
I but one time have been a witness.
As I am speaking of the animals found in this part of the country, I
must not omit the otters here, called tavous, which in the
Cormoetibo Creek frequently attracted our attention by their
disagreeable noise: as they are amphibious, they live mostly on fish;
they are about three feet in length, grey-coloured, and all over
spotted with white; their legs are short, they are web-footed, and
armed with five claws; the head is round, the nose beset with
whiskers like a cat; the eyes are small, and [169]placed above the
ears; the tail is very short. This animal moves aukwardly upon land,
but in the rivers proceeds with great velocity. In Guiana it is said
there is another species of otters, which are much larger, but these I
never saw.

Notwithstanding the favourable appearances of the preceding day, I


was, on the 24th, exceedingly ill indeed, not being able to sit up in
my hammock, under which the black boy Quaco now lay, crying for
his master, and on the following day the poor lad himself fell sick; at
the same time I was also obliged to send three men in a fever to
Devil’s Harwar. As misfortunes often croud together, I received, at
this fatal period, the melancholy account that the officer, Mr. Ower,
was also dead, having expired on his passage downwards at the
Alida estate, where he was buried. My ensign, Mr. Cottenburgh, who
had since gone to Paramaribo, died next; and for myself no better
was now to be expected. In the height of a burning fever I now lay,
forsaken by all my officers and men, without a friend to comfort me,
and without assistance of any kind, except what the poor remaining
negro slaves could afford me, by boiling a little water to make some
tea. In this situation the reader may judge of the consolation which
was afforded me, the very evening when these accumulated
misfortunes seemed to threaten our extinction, by the receipt of an
order from the Colonel, to come down with both the barges to
Devil’s Harwar, where I was again to take post [170]on shore, and
relieve Captain Orzinga, of the Society service, who with his men
was to proceed to La Rochelle in Patamaca, to strengthen the troops
already there. Ill as I was, this had such a powerful effect on my
spirits, that I immediately ordered the Cerberus down to the mouth
of the Cormoetibo Creek, where she joined me that evening.

On the 26th, we bid farewel to this destructive place; and having


weighed anchor, rowed down as far as Barbacoeba, during which
time a circumstance happened, which will probably prove more
entertaining than the repeated accounts of sickness and deaths.

As I was resting in my hammock, between the paroxysms of my


fever, about half way between Cormoetibo and Barbacoeba, while
the Charon was floating down, the sentinel called to me that he had
seen and challenged something black and moving in the brushwood
on the beach, which gave no answer; but which, from its size, he
concluded must be a man. I immediately dropped anchor; and
having manned the canoe, ill as I was, I stepped into it, and rowed
up to the place mentioned by the sentinel. Here we all stepped
ashore to reconnoitre, as I suspected it to be no other than a rebel
spy, or a straggling party detached by the enemy; but one of my
slaves, of the name of David, declared it was no negro, but a large
amphibious snake, which could not be far from the beach, and I
might have an opportunity of shooting it if I pleased. To this,
however, I [171]had not the least inclination, from the uncommon size
of the creature, from my weakness, and the difficulty of getting
through the thicket, which seemed impenetrable to the water’s
edge; and therefore ordered all of them to return on board. The
negro then asked me liberty to step forward and shoot it himself,
assuring me it could not be at any great distance, and warranting
me against all danger. This declaration inspired me with so much
pride and emulation, that I determined to take his first advice, and
kill it myself; provided he would point it out to me, and be
responsible for the hazard, by standing at my side, from which I
swore that if he dared to move, I should level the piece at himself,
and blow out his own brains.

To this the negro cheerfully agreed; and having loaded my gun with
a ball-cartridge, we proceeded; David cutting a path with a bill-hook,
and a marine following, with three more loaded firelocks to keep in
readiness. We had not gone above twenty yards through mud and
water, the negro looking every way with an uncommon degree of
vivacity and attention; when starting behind me, he called out, “Me
see snakee!” and in effect there lay the animal, rolled up under the
fallen leaves and rubbish of the trees; and so well covered, that it
was some time before I distinctly perceived the head of this monster,
distant from me not above sixteen feet, moving its forked tongue,
while its eyes, from their uncommon brightness, appeared to emit
sparks of fire. I now, resting my piece [172]upon a branch, for the
purpose of taking a surer aim, fired; but missing the head, the ball
went through the body, when the animal struck round, and with
such astonishing force as to cut away all the underwood around him
with the facility of a scythe mowing grass; and by flouncing his tail,
caused the mud and dirt to fly over our heads to a considerable
distance. Of this proceeding however we were not torpid spectators,
but took to our heels, and crouded into the canoe. The negro now
intreated me to renew the charge, alluring me the snake would be
quiet in a few minutes, and at any rate persisting in the assertion
that he was neither able nor inclined to pursue us; which opinion he
supported by walking before me, till I should be ready to fire. And
thus I again undertook to make the trial, especially as he said that
his first starting backwards had only proceeded from a desire to
make room for me. I now found the snake a little removed from his
former station, but very quiet, with his head as before, lying out
among the fallen leaves, rotten bark, and old moss. I fired at it
immediately, but with no better success than the other time: and
now, being but slightly wounded, he sent up such a cloud of dust
and dirt, as I never saw but in a whirlwind, and made us once more
suddenly retreat to our canoe; where now, being heartily tired of the
exploit, I gave orders to row towards the barge: but David still
intreating me to permit him to kill the animal, I was, by his
persuasions, induced to make a third and last attempt, in company
with him. [173]Thus, having once more discovered the snake, we
discharged both our pieces at once, and with this good effect, that
he was now by one of us shot through the head. David, who was
made completely happy by this successful conclusion, ran leaping
with joy, and lost no time in bringing the boat-rope, in order to drag
him down to the canoe; but this again proved not a very easy
undertaking, since the creature, notwithstanding its being mortally
wounded, still continued to wreath and twist about, in such a
manner as rendered it dangerous for any person to approach him.
The negro, however, having made a running noose on the rope, and
after some fruitless attempts to make an approach, threw it over his
head with much dexterity; and now, all taking hold of the rope, we
dragged him to the beach, and tied him to the stern of the canoe, to
take him in tow. Being still alive, he kept swimming like an eel; and I
having no relish for such a ship-mate on board, whose length
(notwithstanding to my astonishment all the negroes declared it to
be but a young one come to about its half growth) I found upon
measuring it to be twenty-two feet and some inches; and its
thickness about that of my black boy Quaco, who might then be
about twelve years old, and round whose waist I since measured the
creature’s skin.

Being arrived along-side of the Charon, the next consideration was,


how to dispose of this immense animal; when it was at length
determined to bring him on shore at Barbacoeba, to have him
skinned, and take out the [174]oil, &c. In order to effect this purpose,
the negro David having climbed up a tree with the end of the rope,
let it down over a strong forked bough, and the other negroes
hoisted up the snake, and suspended him from the tree. This done,
David, with a sharp knife between his teeth, now left the tree, and
clung fast upon the monster, which was still twisting, and began his
operations by ripping it up, and stripping down the skin as he
descended. Though I perceived that the animal was no longer able
to do him any injury, I confess I could not without emotion see a
man stark naked, black and bloody, clinging with arms and legs
round the slimy and yet living monster. This labour, however, was not
without its use, since he not only dexterously finished the operation,
but provided me, besides the skin, with above four gallons of fine
clarified fat, or rather oil, though there was wasted perhaps as much
more. This I delivered to the surgeons at Devil’s Harwar, for the use
of the wounded men in the hospital, for which I received their hearty
thanks, it being considered, particularly for bruises, a very excellent
remedy. When I signified my surprize to see the snake still living,
after he was deprived of his intestines and skin, Caramaca, the old
negro, whether from experience or tradition, assured me he would
not die till after sun-set. The negroes now cut him in slices, in order
to dress and feast upon him, they all declaring that he was
exceedingly good and wholesome; but to their [175]great
mortification I refused to give my concurrence, and we rowed down
with the skin to Devil’s Harwar.
Blake Sculpt.
The skinning of the Aboma Snake, shot by Cap. Stedman.

London, Published Decr. 2d. 1793, by J. Johnson, St. Paul’s


Church Yard.
Of this species several skins are preserved in the British and Mr.
Parkinson’s Museums. It is called by Mr. Westley Lyboija, and Boa in
the British Encyclopœdia, to which publication I refer the reader for
a perfect account, and an excellent engraving, of this wonderful
creature, which in the colony of Surinam is called Aboma. Its length,
when full grown, is said to be sometimes forty feet, and more than
four feet in circumference; its colour is a greenish black on the back;
a fine brownish yellow on the sides, and a dirty white under the
belly: the back and sides being spotted with irregular black rings,
with a pure white in the middle. Its head is broad and flat, small in
proportion to the body, with a large mouth, and a double row of
teeth: it has two bright prominent eyes; is covered all over with
scales, some about the size of a shilling; and under the body, near
the tail, armed with two strong claws like cock-spurs, to help it in
seizing its prey. It is an amphibious animal, that is, it delights in low
and marshy places, where it lies coiled up like a rope, and concealed
under moss, rotten timber, and dried leaves, to seize its prey by
surprize, which from its immense bulk it is not active enough to
pursue. When hungry, it will devour any animal that comes within its
reach, and is indifferent whether it is a sloth, a wild boar, a stag, or
even a tiger; round which [176]having twisted itself by the help of its
claws, so that the creature cannot escape, it breaks, by its
irresistible force, every bone in the animal’s body, which it then
covers over with a kind of slime or slaver from its mouth, to make it
slide; and at last gradually sucks it in, till it disappears: after this, the
aboma cannot shift its situation, on account of the great knob or
knot which the swallowed prey occasions in that part of the body
where it rests till it is digested; for till then it would hinder the snake
from sliding along the ground. During that time the aboma wants no
other subsistence. I have been told of negroes being devoured by
this animal, and am disposed to credit the account; for should they
chance to come within its reach when hungry, it would as certainly
seize them as any other animal. I do not apprehend that its flesh,
which is very white, and looks like that of fish, is in any respect
pernicious to the stomach. I should have had no objection to the
negroes eating it till it was consumed, had I not observed a kind of
dissatisfaction among the remaining marines, who would not have
been pleased with my giving the negroes the use of the kettle to boil
it. The bite of this snake is said not to be venomous; nor do I believe
it bites at all from any other impulse than hunger.

I shall only add, that having nailed its skin on the bottom of the
canoe, and dried it in the sun, sprinkling it over with wood-ashes to
prevent it from corruption, I [177]sent it to a friend at Paramaribo,
whence it was since sent to Holland as a curiosity.

However extraordinary this account may appear to many readers, let


them peruse the narrative which is related by a gentleman in the
island of Ceylon, who saw a tiger killed there by a snake he calls the
anacunda, but in a quite different manner, and their wonder will
cease. I must add, however, that this gentleman’s relation is so very
marvellous, that, notwithstanding what I have experienced, I must
confess it very greatly staggered my faith 1.

This business being ended, I also made an end of the cruise, by


dropping down before the Society post Devil’s Harwar, in order the
next day to assume the command. [178]

Doctor Bancroft mentions the power of fascination in the aboma. This I am


1
obliged to contradict. Nor can I, without great difficulty, attribute this quality
even to the rattle-snake. ↑
[Contents]
CHAP. VIII.

Three Estates burnt, and the Inhabitants murdered by the Rebels—Real


Picture of Misery and Distress—Specimen of a March through the Woods
of Surinam—Colonel Fourgeoud and the remaining Troops leave
Paramaribo.

On the 27th of August I relieved Captain Orzinga with his men, and
took the command of Devil’s Harwar, having been on board the
Charon exactly fifty-six days, in the most wretched condition that
can be described: but I hoped now to get the better of my complaint
by the help of a few refreshments, such as milk, &c. which could not
be obtained in our former situation. The Society troops (above one
hundred in number) being to set off next day with my empty barges
to La Rochelle, in Patamaca, I reviewed my marines, when I found I
had left out of five officers but two, who were both sick, the three
others being dead; I had also only one serjeant, two corporals, and
fifteen privates, out of fifty-four healthy men, who embarked with
me on the 2d day of last July. This army was not more than
sufficient to defend the hospital, (which was crouded with sick) the
ammunition and victualling magazine, &c. on a spot where lately had
been kept three hundred soldiers, particularly [179]while the enemy
were certainly lurking not far off: in consideration of which, the
Society Captain reinforced me with twenty of his men. The next
evening he entertained me and my two subalterns with a supper of
fresh meat, both roast and boiled, to our great comfort and surprize;
but which, to my unspeakable mortification, proved to be the
individual poor cow with her calf, on whom we had built all our
hopes for a little relief. It appeared that one of his sentinels, as
concerted between them, had shot it by a wilful mistake. Thus did
Captain Orzinga, for the sake of a momentary gratification, deprive
us all of that lasting comfort on which we had so much depended,
and of which we had so much need, being altogether emaciated for
want of wholesome and nutritive food.

On the morning of the 28th the Society troops rowed to Patamaca,


when, examining the twenty soldiers they had left me, they proved
to be the refuse of the whole, part with agues, wounds, ruptures,
and rotten limbs, and most of them next day were obliged to enter
the hospital.

On the 29th, having bastonaded my late pilot for stealing from the
soldiers, I dispatched the information to Colonel Fourgeoud, that I
had taken post, and acquainting him with my weak situation,
requested a proper reinforcement. In the evening two of my men
died.

All things now being regulated and settled, I thanked Heaven in the
expectation of getting some rest, being [180]still extremely weak; and
with these cheering hopes retired at ten o’clock at night to my
hammock; but this tranquillity was again of short duration, for
having scarcely closed my eyes I was awaked by my serjeant, and
the following letter put into my hand, sent by an express from the
captain of the militia, or bargers in Cottica.

“Sir,

“This is to acquaint you, that the rebels have burnt three estates by your
side, Suyingheyd, Peru, and L’Esperance, the ruins of which are still
smoking; and that they have cut the throats of all the white inhabitants
that fell in their way. As on their retreat they must pass close by where
you are posted, be on your guard.—I am in haste.

“Your’s, &c.

(Signed) “Stoeleman.”
Conscious of my defenceless situation, I immediately started up; and
the express who brought the letter having spread the news the
moment of his landing, there was no necessity for beating to arms,
since not only the few soldiers who were well, but the whole hospital
burst out; and several of them, in spite of my opposition, crawling
on their hands and feet to their arms, dropped dead upon the spot.
—May I never behold such another scene [181]of misery and distress!
Lame, blind, sick, and wounded, in the hope of preserving a
wretched existence, rushed upon certain death! They could only, in a
word, be compared to the distressed army and navy at Carthagena,
commanded by the British Admiral Vernon, whom Thomson
describes—

—— —— —— “You, gallant Vernon, saw


The miserable scene, you pitying saw,
To infant weakness sunk, the warriour’s arm;
Saw the deep-racking pang, the ghastly form,
The lip pale quivering, and the beamless eye,
No more with ardour bright.”

For my own part, I was in a very weakly condition indeed; however,


we continued to lie all night on our arms, during which I pressed the
messenger to stay, in order to add one to our miserable number,
being determined to sell our lives as dearly as possible. But no
enemy appearing in the morning, we buried the dead in their
hammocks, not having a board to make a coffin on the whole post.
In this situation I lost all patience, and had the audacity to write to
my commander, that (besides what had happened) my last men
stood upon the brink of the grave, from hardships and for want of
being properly supported; the very waiters of the hospital having
deserted on the moment of my arrival here, and gone to
Paramaribo. Our whole number, indeed, was [182]now melted down
to twelve men, who were to protect twelve buildings, and that with
no more than two very small chests of ammunition, and no retreat
for the sick, as the barges were gone to Patamaca, and the last
canoe dispatched with my letter to Colonel Fourgeoud; for I had set
adrift that belonging to the express, who was a book-keeper of a
neighbouring plantation, in order to prevent him or any other from
making their escape. In this situation, I was now obliged to convert
the slaves into soldiers: these I armed with a hatchet, not daring to
trust them with a firelock. For this whole night we again watched
under arms, and in the morning found two more of our little party
dead on the ground.

I now began really to think we were all devoted to destruction, while


the men, regardless of all order (self-preservation excepted) threw
out the most bitter invectives against their persecutor, Fourgeoud,
which I could not prevent; nor can I help remarking the generalship
of the rebel negroes, who had kept lying quiet till the removal of the
Society troops from Devil’s Harwar, and seized the very first day of
their departure, convinced of its being guarded only by my sick and
emaciated soldiers, in order to commit their depredations on the
Cottica estates. They well knew that my force was not sufficient to
pursue them, nay, hardly to stand in my own defence: all this,
however, was but according to my expectation; while, on the
contrary, had my strength been sufficient, they could never have
escaped [183]at least from being cut off in their retreat, especially if
the troops in Rio Perica had acted conjointly with those in Cottica, by
patroling the path between the two rivers, across which the rebels
were twice unavoidably obliged to pass.

On the 1st of September we waited once more till morning, and then
buried another of my poor men; while I yet cannot conceive how
any one was able to survive such a series of toil, in such a
debilitated state, and in a tropical climate; yet some did, though few.
At length, being persuaded that the rebels must have past the
Cordon, without having thought proper to pay us a visit on their
retreat, I determined to let the remaining few watch no longer, but
permit them to die a natural death. At last, in the evening, when all
was too late, there came down by water from the post La Rochelle
to our assistance, one officer and ten men.—I having had but nine
left to do the duty at the time of their arrival.

On the 2d another man died; and I once more reviewed my forces,


which now amounted exactly to seven marines, the few scarecrows
of the Society excepted; however, the chance of being massacred by
the rebels was at this time over, thanks alone to their pusillanimity,
or rather their hurry!

I now received a letter from Colonel Fourgeoud, condoling with me


on the loss of so many good officers, acquainting me that I was to
be reinforced; and that on my recommendation my serjeant, Mr. de
Cabanus, was appointed [184]an ensign; which gave me pleasure,
and took place at a very suitable time, since this day my poor Ensign
Macdonald was sent down very sick to Paramaribo. I answered to all
this, that I was obliged to him; adding, that while I remained
without reinforcement, I could not be accountable for what
consequences might happen, in a place where I was left to defend a
whole river with none but sick people; and even these without
sufficient ammunition, and hourly expiring for want of proper
medicines, or a surgeon to attend them, there being none here but
one or two surgeon’s mates belonging to the troops of the Society,
who could do little more than occasionally draw blood, and cut off a
beard or a corn.

On the 4th we buried another of my marines, and on the following


day another died; and I had not one now remaining who was not ill,
or who was not rendered unserviceable, by his feet being swelled
with the insects called chigoes: these poor men were mostly
Germans, who had been accustomed to a healthy climate in their
own country. I began now to be reconciled to putting my last man
under ground, and almost wishing to leap into the grave after him
myself; when a barge arrived from Paramaribo with the proper
reinforcement, ammunition, provisions, medicines, a surgeon, and
an order from my chief to trace out the track of the rebels
immediately, on the former path of communication called the
Cordon, between Cottica and Perica, and to write him [185]the result
of my discoveries; he intimated also that he intended to keep his
magazines at Devil’s Harwar, and that I was not to make use of the
spot I had found out for that purpose at Barbacoeba Creek.

On the 6th I prepared to march myself, having recovered a little


strength, on the grand project of discovery, and then placed the
ammunition in the magazine.

As the manner of marching in this country is so very different from


that in Europe, I shall, before we set out, endeavour briefly to
describe the nature of these expeditions.

In the first place, in Surinam no such thing is practicable as three or


even two ranks; thus there is no marching by divisions or platoons;
—but the whole party being dressed in one rank, face to the right,
and every man follows his leader, the negro slaves interspersed
between the men, in order to guard their persons as well as what
they carry; and this manner of marching is called Indian file. With a
detachment of sixty men, consisting of one captain, two subalterns,
two serjeants, four corporals, one surgeon, and fifty privates, twenty
negro slaves at least ought to be employed, for the use of whom
their masters are paid at the rate of two shillings sterling a day by
the colony; and this is a much greater expence than waggons and
horses would be, which in this country cannot be employed for
military service. [186]
The manner of interspersing them amongst the troops is as follows:
The foremost are generally two negroes, with bill-hooks to cut a
way, so as to make a practicable path, with one corporal and two
men to reconnoitre the front, and, in case of necessity, to give the
alarm; and then one subaltern, six privates, and a corporal, form the
van. Then follows, at some distance, the corps in two divisions; in
the first, one captain, one corporal, twelve privates, one surgeon,
and two negroes to carry the powder; in the second, is one serjeant
and twelve privates; and then again follows, at some distance, the
rear guard, consisting of one subaltern, one serjeant, one corporal,
and eighteen privates, with sixteen negroes to carry the medicines,
beef, bread, spades, axes, rum, &c.; the sick also are carried. The
three last of all being one corporal and two men at a distance, to
give the alarm in case of an attack, as the others had orders to do in
the front, which ends the train.
T. Conder Sculpt.
Order of March thro’ the Woods of Surinam.

References to the above March.


1. Two Negroes with Bill hooks to open a
Path.
2. One Corporal & Two Privates, to cover
Van. A
the Van.
3. One Subaltern, Six Privates, & one
Corporal.
1. The Captain or commanding Officer.
2. The Surgeon.
3. Two Privates, to cover the Powder.
4. A Negro with a Box of Ball Cartridges.
5. Two Privates. Main
B
6. A Negro with a Box of Ball Cartridges. Body.
7. Eight Privates.
8. One Corporal.
9. Twelve Privates.
10. One Sergeant.
1. A Subaltern Officer.
2. Two Privates.
3. Three Negroes, with Medicines, Kettles,
Axes, Spades, &c.
4. Two Privates.
5. Three Negroes with Salt Beef, Salt Pork,
&c.
6. Two Privates.
7. Three Negroes, with Black Bread, or
Rusk Biscuit.
8. One Private. Rear
9. Two Negroes, with Kill-devil, or New Guard
Rum. or Corps C
10. One Private. de
11. One Negro, with the Captain’s Provisions. Reserve.
12. One Private.
13. One Negro, with Provisions for the two
Subaltern Officers.
14. One Private.
15. Three Negroes to carry the Sick &
Wounded.
16. Six Privates.
17. One Sergeant.
18. One Corporal & two Privates, to cover
the Rear.

Marks to be cut on the Trees on a March.

A B C
Fourgeoud’s 1st. Column, 2d. Ditto, 3d. Ditto,
Sub. A. Sub. B. Sub. C.
D E F
Society’s 1st. Column, 2d. Ditto, 3d. Ditto,
Sub. D. Sub. E. Sub. F.

London, Published Decr. 1st. 1791 by J. Johnson, St. Paul’s Church


Yard.

Every thing being ready, according to the above rules, for my small
party, which consisted of myself, an officer of the Society, Mr.
Hertsbergh, one surgeon’s mate, one guide, two serjeants, two
corporals, forty privates, and only eight negro slaves to cut open the
passage, and carry the baggage, we faced to the right at six o’clock
in the morning, and sallied forth into the woods, keeping our course
directly for the Perica river; and having marched till about eleven
o’clock on the Cordon, I discovered, as I had expected, [187]the track
of the rebels by the marks of their footsteps in the mud, by the
broken bottles, plantain-shells, &c. and found that by appearance it
bore towards Pinneburgh, already mentioned.

I had now indeed found the nest, but the birds were flown. We
continued our march till eight o’clock, when we arrived at the
Society post Soribo, in Perica, in a most shocking condition, having
waded through water and mire above our hips, climbed over heaps
of fallen trees, and crept underneath on our bellies. This, however,
was not the worst, for our flesh was terribly mangled and torn by
the thorns, and stung by the Patat lice, ants, and wassy-wassy or
wild bees. This last is an insect not larger than an English blue-bottle
fly, and is of a black colour, quite different from our bees; they are
never kept in hives, but swarm wild in the forest, where they build in
hollow trees or between the branches; their nests being sometimes
as large as an inflated cow’s bladder, to which they bear no bad
resemblance, both in colour and smoothness, except in being less
regularly oval; from these abodes (when the nest or the branches
are inadvertently touched) thousands of warriors sally forth; and this
little flying army is extremely formidable, pitching always by instinct
on the eyes, lips and hair, whence they cannot easily be dislodged;
their stings generally cause a fever, and swell the parts so very much
that they occasion blindness for several hours; their honey [188]is of
a dark-brown colour, and so is their wax, but gummy, being both of
little value.

The worst of our sufferings, however, was the fatigue of marching in


a burning sun, and the last two hours in total darkness, holding each
other by the hand; and having left ten men behind, some with
agues, some stung blind, and some with their feet full of chigoes.
Being in the most hospitable manner received at Soribo by the
commanding officer, I went to my hammock very ill of a fever.

On the following morning I felt myself better for my night’s rest; but
neither myself nor my men were able to march back, wherefore the
other captain sent a small party of his soldiers to pick up the poor
marines I had lost the day before, and of whom they brought with
them seven, carried in hammocks tied to poles, each by two
negroes, the other three having scrambled back to Devil’s Harwar.

During our stay here I wrote a letter to Colonel Fourgeoud, couched


in such terms as few people in their full senses would do to their
commanders, viz. that I had found the path; that if I had had
support in time I might have cut off the enemy’s retreat, instead of
finding their foot-steps only; but that now all was too late, and the
party all knocked up to no purpose. This letter, I have been since
told, incensed him, as it is easy to suppose, in the highest degree.
Being sufficiently refreshed to renew my march, we left Soribo on
the 9th, at four o’clock in [189]the morning, and at four o’clock P. M.
arrived, after indescribable sufferings, at Devil’s Harwar, covered
over with mud and blood, and our legs and thighs cut and torn by
the thorns and branches; most of the men being without shoes and
stockings of necessity, while I, who had gone this march in the same
condition from choice, had absolutely suffered the least of the whole
party, by having inured myself gradually to walk barefooted on the
barges.

At Devil’s Harwar, I now found Lieutenant Colonel Westerloo and a


quarter-master arrived to take the command, his troops not being
expected till the next day. I was by this circumstance, however,
made exceedingly happy, hoping at last to meet with some relief;
and having ceded him my written orders, the magazine, hospital, &c.
&c. I stripped and plunged into the river to wash myself and take a
swim, by which (being before much over-heated) I found myself
greatly refreshed, as well as by receiving a quantity of fine fruit,
Jamaica rum, wine and sugar, from Joanna;—but how did my blood
chill, when the quarter-master told me, as a secret, that my
serjeant, one Fowler, having first got drunk with my wine, offered
violence to this poor woman; and that he was to be at Devil’s
Harwar next day, when I should see the marks of her just
resentment on his face!

The reader will, I trust, excuse my violence, when I [190]tell him, that
I vowed immediate destruction to the villain: and having ordered a
negro to cut twelve bamboo canes, I retired like a person deprived
of his senses, determined to punish him according to his supposed
crime.

On the 10th there arrived two subalterns, with a second barge full of
men, ammunition, medicine, and provisions, which having marched
into quarters and stowed, I sent for the hapless Fowler, whose face
being in three places wounded, I locked him up in a room, and,

You might also like