0% found this document useful (0 votes)
346 views79 pages

Postgresql Query Optimization The Ultimate Guide To Building Efficient Queries 2nd Edition Henrietta Dombrovskaya Download

The document is a comprehensive guide on PostgreSQL query optimization, authored by Henrietta Dombrovskaya, Boris Novikov, and Anna Bailliekova. It covers various aspects of building efficient queries, including optimization goals, execution plans, and indexing strategies. The second edition includes updated content and practical techniques for improving database performance.

Uploaded by

smarinamzi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
346 views79 pages

Postgresql Query Optimization The Ultimate Guide To Building Efficient Queries 2nd Edition Henrietta Dombrovskaya Download

The document is a comprehensive guide on PostgreSQL query optimization, authored by Henrietta Dombrovskaya, Boris Novikov, and Anna Bailliekova. It covers various aspects of building efficient queries, including optimization goals, execution plans, and indexing strategies. The second edition includes updated content and practical techniques for improving database performance.

Uploaded by

smarinamzi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 79

Postgresql Query Optimization The Ultimate Guide

To Building Efficient Queries 2nd Edition


Henrietta Dombrovskaya download

https://ebookbell.com/product/postgresql-query-optimization-the-
ultimate-guide-to-building-efficient-queries-2nd-edition-
henrietta-dombrovskaya-54836104

Explore and download more ebooks at ebookbell.com


Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Postgresql Query Optimization The Ultimate Guide To Building Efficient


Queries 2 Converted Henrietta Dombrovskaya

https://ebookbell.com/product/postgresql-query-optimization-the-
ultimate-guide-to-building-efficient-queries-2-converted-henrietta-
dombrovskaya-55507510

Postgresql Query Optimization The Ultimate Guide To Building Efficient


Queries 1st Edition Henrietta Dombrovskaya

https://ebookbell.com/product/postgresql-query-optimization-the-
ultimate-guide-to-building-efficient-queries-1st-edition-henrietta-
dombrovskaya-24506964

Postgresql Query Optimization The Ultimate Guide To Building Efficient


Queries Henrietta Dombrovskaya Boris Novikov Anna Bailliekova

https://ebookbell.com/product/postgresql-query-optimization-the-
ultimate-guide-to-building-efficient-queries-henrietta-dombrovskaya-
boris-novikov-anna-bailliekova-24506962

Postgresql 10 High Performance Expert Techniques For Query


Optimization High Availability And Efficient Database Maintenance 1st
Edition Ibrar Ahmed

https://ebookbell.com/product/postgresql-10-high-performance-expert-
techniques-for-query-optimization-high-availability-and-efficient-
database-maintenance-1st-edition-ibrar-ahmed-7185404
Postgresql 10 High Performance Expert Techniques For Query
Optimization High Availability And Efficient Database Maintenance
Ibrar Ahmed

https://ebookbell.com/product/postgresql-10-high-performance-expert-
techniques-for-query-optimization-high-availability-and-efficient-
database-maintenance-ibrar-ahmed-11091988

Postgresql 10 High Performance Expert Techniques For Query


Optimization High Availability And Efficient Database Maintenance
Ibrar Ahmed

https://ebookbell.com/product/postgresql-10-high-performance-expert-
techniques-for-query-optimization-high-availability-and-efficient-
database-maintenance-ibrar-ahmed-55463690

Essential Postgresql Your Guide To Database Design Query Optimization


And Administration Saxena Swati

https://ebookbell.com/product/essential-postgresql-your-guide-to-
database-design-query-optimization-and-administration-saxena-
swati-219092474

Postgresql Configuration Best Practices For Performance And Security


1st Edition Baji Shaik

https://ebookbell.com/product/postgresql-configuration-best-practices-
for-performance-and-security-1st-edition-baji-shaik-46148980

Postgresql Configuration Best Practices For Performance And Security


1st Edition Baji Shaik

https://ebookbell.com/product/postgresql-configuration-best-practices-
for-performance-and-security-1st-edition-baji-shaik-46659076
PostgreSQL Query
Optimization
The Ultimate Guide to Building
Efficient Queries
Second Edition

Henrietta Dombrovskaya
Boris Novikov
Anna Bailliekova
PostgreSQL Query Optimization: The Ultimate Guide to Building Efficient Queries,
Second Edition
Henrietta Dombrovskaya Boris Novikov
DRW Holdings, Chicago, IL, USA Database expert, Helsinki, Finland
Anna Bailliekova
UrbanFootprint, Milwaukee, Wisconsin, USA

ISBN-13 (pbk): 979-8-8688-0068-9 ISBN-13 (electronic): 979-8-8688-0069-6


https://doi.org/10.1007/979-8-8688-0069-6

Copyright © 2024 by Henrietta Dombrovskaya, Boris Novikov, Anna Bailliekova


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Shaul Elson
Development Editor: Laura Berendson
Project Manager: Jessica Vakili
Cover designed by Straive
Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza,
Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@
springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole
member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc
is a Delaware corporation.
For information on translations, please e-mail [email protected]; for reprint,
paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub. For more detailed information, please visit https://www.apress.com/gp/services/
source-code.
Paper in this product is recyclable
Table of Contents
About the Authors�������������������������������������������������������������������������������������������������� xiii

About the Technical Reviewer���������������������������������������������������������������������������������xv


Acknowledgments�������������������������������������������������������������������������������������������������xvii

Introduction������������������������������������������������������������������������������������������������������������xix

Chapter 1: Why Optimize?���������������������������������������������������������������������������������������� 1


What Do We Mean by Optimization?��������������������������������������������������������������������������������������������� 1
Why It Is Difficult: Imperative and Declarative������������������������������������������������������������������������������ 2
Optimization Goals������������������������������������������������������������������������������������������������������������������������ 6
Optimizing Processes�������������������������������������������������������������������������������������������������������������������� 7
Optimizing OLTP and OLAP������������������������������������������������������������������������������������������������������ 8
Database Design and Performance����������������������������������������������������������������������������������������� 9
Application Development and Performance�������������������������������������������������������������������������� 11
Other Stages of the Lifecycle������������������������������������������������������������������������������������������������ 11
PostgreSQL Specifics������������������������������������������������������������������������������������������������������������������ 12
Summary������������������������������������������������������������������������������������������������������������������������������������ 13

Chapter 2: Theory: Yes, We Need It!������������������������������������������������������������������������ 15


Query Processing Overview�������������������������������������������������������������������������������������������������������� 15
Compilation��������������������������������������������������������������������������������������������������������������������������� 15
Optimization and Execution��������������������������������������������������������������������������������������������������� 16
Relational, Logical, and Physical Operations������������������������������������������������������������������������������ 17
Relational Operations������������������������������������������������������������������������������������������������������������ 17
Logical Operations����������������������������������������������������������������������������������������������������������������� 21
Queries as Expressions: Thinking in Sets������������������������������������������������������������������������������ 22
Operations and Algorithms���������������������������������������������������������������������������������������������������� 23
Summary������������������������������������������������������������������������������������������������������������������������������������ 23
iii
Table of Contents

Chapter 3: Even More Theory: Algorithms�������������������������������������������������������������� 25


Algorithm Cost Models���������������������������������������������������������������������������������������������������������������� 25
Data Access Algorithms�������������������������������������������������������������������������������������������������������������� 27
Storage Structures���������������������������������������������������������������������������������������������������������������� 27
Full Scan�������������������������������������������������������������������������������������������������������������������������������� 28
Index-Based Table Access����������������������������������������������������������������������������������������������������� 29
Index-Only Scan�������������������������������������������������������������������������������������������������������������������� 31
Comparing Data Access Algorithms�������������������������������������������������������������������������������������� 32
Index Structures�������������������������������������������������������������������������������������������������������������������������� 34
What Is an Index?������������������������������������������������������������������������������������������������������������������ 34
B-Tree Indexes����������������������������������������������������������������������������������������������������������������������� 36
Why Are B-Trees Used So Often?������������������������������������������������������������������������������������������� 38
Other Kinds of Indexes���������������������������������������������������������������������������������������������������������� 39
Combining Relations������������������������������������������������������������������������������������������������������������������� 39
Nested Loops������������������������������������������������������������������������������������������������������������������������� 40
Hash-Based Algorithms��������������������������������������������������������������������������������������������������������� 42
Sort-Merge Algorithm������������������������������������������������������������������������������������������������������������ 43
Comparing Algorithms����������������������������������������������������������������������������������������������������������� 45
Summary������������������������������������������������������������������������������������������������������������������������������������ 45

Chapter 4: Understanding Execution Plans������������������������������������������������������������ 47


Putting Everything Together: How an Optimizer Builds an Execution Plan��������������������������������� 47
Reading Execution Plans������������������������������������������������������������������������������������������������������������� 48
Understanding Execution Plans�������������������������������������������������������������������������������������������������� 53
What Is Going On During Optimization?��������������������������������������������������������������������������������� 54
Why Are There So Many Execution Plans to Choose From?�������������������������������������������������� 55
How Are Execution Costs Calculated?����������������������������������������������������������������������������������� 56
How Can the Optimizer Be Led Astray?��������������������������������������������������������������������������������� 59
Summary������������������������������������������������������������������������������������������������������������������������������������ 60

iv
Table of Contents

Chapter 5: Short Queries and Indexes�������������������������������������������������������������������� 61


What Makes a Query “Short”?���������������������������������������������������������������������������������������������������� 62
Choosing Selection Criteria��������������������������������������������������������������������������������������������������������� 64
Index Selectivity�������������������������������������������������������������������������������������������������������������������� 64
Unique Indexes and Constraints�������������������������������������������������������������������������������������������� 66
Indexes and Non-equal Conditions��������������������������������������������������������������������������������������������� 70
Indexes and Column Transformations����������������������������������������������������������������������������������� 70
Indexes and the like Operator����������������������������������������������������������������������������������������������������� 75
Using Multiple Indexes���������������������������������������������������������������������������������������������������������������� 77
Compound Indexes��������������������������������������������������������������������������������������������������������������������� 78
How Do Compound Indexes Work?���������������������������������������������������������������������������������������� 78
Lower Selectivity������������������������������������������������������������������������������������������������������������������� 80
Using Indexes for Data Retrieval������������������������������������������������������������������������������������������� 81
Covering Indexes������������������������������������������������������������������������������������������������������������������� 82
Excessive Selection Criteria�������������������������������������������������������������������������������������������������������� 83
Partial Indexes���������������������������������������������������������������������������������������������������������������������������� 89
Indexes and Join Order��������������������������������������������������������������������������������������������������������������� 90
When Are Indexes Not Used�������������������������������������������������������������������������������������������������������� 94
Avoiding Index Usage������������������������������������������������������������������������������������������������������������ 94
Why Does PostgreSQL Ignore My Index?������������������������������������������������������������������������������� 94
Let PostgreSQL Do Its Job!��������������������������������������������������������������������������������������������������������� 97
How to Build the Right Index(es)����������������������������������������������������������������������������������������������� 101
To Build or Not to Build�������������������������������������������������������������������������������������������������������� 101
Which Indexes Are Needed?������������������������������������������������������������������������������������������������ 102
Which Indexes Are Not Needed?����������������������������������������������������������������������������������������� 102
Indexes and Short Query Scalability����������������������������������������������������������������������������������������� 103
Summary���������������������������������������������������������������������������������������������������������������������������������� 103

v
Table of Contents

Chapter 6: Long Queries and Full Scans��������������������������������������������������������������� 105


Which Queries Are Considered Long?��������������������������������������������������������������������������������������� 105
Long Queries and Full Scans���������������������������������������������������������������������������������������������������� 107
Long Queries and Hash Joins��������������������������������������������������������������������������������������������������� 108
Long Queries and the Order of Joins���������������������������������������������������������������������������������������� 109
What Is a Semi-join?����������������������������������������������������������������������������������������������������������� 109
Semi-joins and Join Order��������������������������������������������������������������������������������������������������� 111
More on Join Order�������������������������������������������������������������������������������������������������������������� 113
What Is an Anti-join?����������������������������������������������������������������������������������������������������������� 116
Semi- and Anti-joins Using the JOIN Operator�������������������������������������������������������������������� 118
When Is It Necessary to Specify Join Order?���������������������������������������������������������������������� 120
Grouping: Filter First, Group Last���������������������������������������������������������������������������������������������� 123
Grouping: Group First, Select Last��������������������������������������������������������������������������������������������� 131
Using SET Operations���������������������������������������������������������������������������������������������������������������� 134
Avoiding Multiple Scans������������������������������������������������������������������������������������������������������������ 138
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 145

Chapter 7: Long Queries: Additional Techniques�������������������������������������������������� 147


Structuring Queries������������������������������������������������������������������������������������������������������������������� 147
Temporary Tables and CTEs������������������������������������������������������������������������������������������������������ 148
Temporary Tables���������������������������������������������������������������������������������������������������������������� 148
Common Table Expressions (CTEs)�������������������������������������������������������������������������������������� 150
Views: To Use or Not to Use������������������������������������������������������������������������������������������������������ 155
Why Use Views?������������������������������������������������������������������������������������������������������������������ 162
Materialized Views�������������������������������������������������������������������������������������������������������������������� 162
Creating and Using Materialized Views������������������������������������������������������������������������������� 162
Refreshing Materialized Views�������������������������������������������������������������������������������������������� 164
Should I Create a Materialized View?���������������������������������������������������������������������������������� 164
Do Materialized Views Need to Be Optimized?�������������������������������������������������������������������� 167
Dependencies���������������������������������������������������������������������������������������������������������������������� 167
Partitioning�������������������������������������������������������������������������������������������������������������������������������� 168

vi
Table of Contents

Does Partitioning Improve Performance?��������������������������������������������������������������������������������� 170


Why Create a Partitioned Table?����������������������������������������������������������������������������������������������� 173
Parallelism�������������������������������������������������������������������������������������������������������������������������������� 173
Summary���������������������������������������������������������������������������������������������������������������������������������� 174

Chapter 8: Optimizing Data Modification�������������������������������������������������������������� 175


What Is DML?���������������������������������������������������������������������������������������������������������������������������� 175
Two Ways to Optimize Data Modification���������������������������������������������������������������������������������� 175
How Does DML Work?��������������������������������������������������������������������������������������������������������������� 176
Low-Level Input/Output������������������������������������������������������������������������������������������������������� 176
The Impact of Concurrency Control������������������������������������������������������������������������������������� 177
Data Modification and Indexes�������������������������������������������������������������������������������������������������� 180
DML and Vacuum���������������������������������������������������������������������������������������������������������������������� 181
Mass UPDATE/DELETE��������������������������������������������������������������������������������������������������������� 181
Frequent Updates���������������������������������������������������������������������������������������������������������������� 182
Referential Integrity and Triggers���������������������������������������������������������������������������������������������� 182
Summary���������������������������������������������������������������������������������������������������������������������������������� 183

Chapter 9: Design Matters������������������������������������������������������������������������������������ 185


Design Matters�������������������������������������������������������������������������������������������������������������������������� 185
Why Use a Relational Model?���������������������������������������������������������������������������������������������������� 189
Types of Databases������������������������������������������������������������������������������������������������������������� 190
Entity-Attribute-Value Model����������������������������������������������������������������������������������������������� 190
Key-Value Model������������������������������������������������������������������������������������������������������������������ 191
Hierarchical Model�������������������������������������������������������������������������������������������������������������� 192
Combining the Best of Different Worlds������������������������������������������������������������������������������ 193
Flexibility vs. Efficiency and Correctness���������������������������������������������������������������������������������� 193
Must We Normalize?����������������������������������������������������������������������������������������������������������������� 195
Use and Misuse of Surrogate Keys������������������������������������������������������������������������������������������� 197
Summary���������������������������������������������������������������������������������������������������������������������������������� 203

vii
Table of Contents

Chapter 10: What About Configuration Parameters?�������������������������������������������� 205


PostgreSQL Configuration Parameters Overview���������������������������������������������������������������������� 205
Memory Allocation��������������������������������������������������������������������������������������������������������������� 206
Connections and Sessions��������������������������������������������������������������������������������������������������� 206
Tuning Parameters for Better Performance������������������������������������������������������������������������������ 208
Are There Better Ways?������������������������������������������������������������������������������������������������������������� 213
Other Limitations of Parameter Tuning������������������������������������������������������������������������������������� 216
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 216

Chapter 11: Application Development and Performance�������������������������������������� 217


Response Time Matters������������������������������������������������������������������������������������������������������������ 217
World Wide Wait������������������������������������������������������������������������������������������������������������������������ 218
Performance Metrics���������������������������������������������������������������������������������������������������������������� 219
Impedance Mismatch���������������������������������������������������������������������������������������������������������������� 219
A Road Paved with Good Intentions������������������������������������������������������������������������������������������ 220
Application Development Patterns�������������������������������������������������������������������������������������� 220
“Shopping List Problem”����������������������������������������������������������������������������������������������������� 222
Interfaces���������������������������������������������������������������������������������������������������������������������������� 224
Welcome to the World of ORM��������������������������������������������������������������������������������������������� 225
In Search of a Better Solution��������������������������������������������������������������������������������������������������� 227
Summary���������������������������������������������������������������������������������������������������������������������������������� 230

Chapter 12: Functions������������������������������������������������������������������������������������������ 231


Function Creation���������������������������������������������������������������������������������������������������������������������� 231
Internal Functions���������������������������������������������������������������������������������������������������������������� 232
User-Defined Functions������������������������������������������������������������������������������������������������������� 232
Introducing Procedural Language��������������������������������������������������������������������������������������� 233
Dollar Quoting���������������������������������������������������������������������������������������������������������������������� 234
Function Parameters and Function Output: Void Functions������������������������������������������������� 235
Function Overloading����������������������������������������������������������������������������������������������������������� 235
Function Execution�������������������������������������������������������������������������������������������������������������������� 237
Function Execution Internals����������������������������������������������������������������������������������������������������� 240

viii
Table of Contents

Functions and Performance������������������������������������������������������������������������������������������������������ 243


How Using Functions Can Worsen Performance����������������������������������������������������������������� 243
Any Chance Functions Can Improve Performance?������������������������������������������������������������ 245
Functions and User-Defined Types�������������������������������������������������������������������������������������������� 246
User-Defined Data Types����������������������������������������������������������������������������������������������������� 246
Functions Returning Composite Types�������������������������������������������������������������������������������� 247
Using Composite Types with Nested Structure������������������������������������������������������������������������� 250
Functions and Type Dependencies�������������������������������������������������������������������������������������������� 255
Data Manipulation with Functions�������������������������������������������������������������������������������������������� 255
Functions and Security������������������������������������������������������������������������������������������������������������� 257
What About Business Logic?����������������������������������������������������������������������������������������������������� 258
Functions in OLAP Systems������������������������������������������������������������������������������������������������������ 259
Parameterizing�������������������������������������������������������������������������������������������������������������������� 260
No Explicit Dependency on Tables and Views���������������������������������������������������������������������� 261
Ability to Execute Dynamic SQL������������������������������������������������������������������������������������������� 261
Stored Procedures�������������������������������������������������������������������������������������������������������������������� 261
Functions with No Results��������������������������������������������������������������������������������������������������� 261
Functions and Stored Procedures��������������������������������������������������������������������������������������� 262
Transaction Management���������������������������������������������������������������������������������������������������� 262
Exception Processing���������������������������������������������������������������������������������������������������������� 263
Summary���������������������������������������������������������������������������������������������������������������������������������� 264

Chapter 13: Dynamic SQL������������������������������������������������������������������������������������� 265


What Is Dynamic SQL���������������������������������������������������������������������������������������������������������������� 265
Why Dynamic SQL Works Better in Postgres����������������������������������������������������������������������� 265
What About SQL Injection?�������������������������������������������������������������������������������������������������� 266
How to Use Dynamic SQL for an Optimal Execution Plan��������������������������������������������������������� 266
How to Use Dynamic SQL in OLAP Systems������������������������������������������������������������������������������ 274
Using Dynamic SQL for Flexibility��������������������������������������������������������������������������������������������� 277
Using Dynamic SQL to Aid the Optimizer���������������������������������������������������������������������������������� 286
FDWs and Dynamic SQL������������������������������������������������������������������������������������������������������������ 289
Summary���������������������������������������������������������������������������������������������������������������������������������� 290
ix
Table of Contents

Chapter 14: Avoiding the Pitfalls of Object-Relational Mapping�������������������������� 291


Why Application Developers Like NORM����������������������������������������������������������������������������������� 292
ORM vs. NORM�������������������������������������������������������������������������������������������������������������������������� 292
NORM Explained����������������������������������������������������������������������������������������������������������������������� 294
NORM in the Application Perspective���������������������������������������������������������������������������������������� 301
NORM from a Database Perspective����������������������������������������������������������������������������������������� 304
Mapping JSON to the Database������������������������������������������������������������������������������������������� 305
Generating Database Code�������������������������������������������������������������������������������������������������� 306
Getting Data from the Database������������������������������������������������������������������������������������������ 308
Modifying Data in the Database������������������������������������������������������������������������������������������ 311
Why Not Store JSON?!�������������������������������������������������������������������������������������������������������������� 314
Performance Gains������������������������������������������������������������������������������������������������������������������� 314
Working Together with Application Developers������������������������������������������������������������������������� 315
Summary���������������������������������������������������������������������������������������������������������������������������������� 315

Chapter 15: More Complex Filtering and Search�������������������������������������������������� 317


Full Text Search������������������������������������������������������������������������������������������������������������������������� 317
Multidimensional and Spatial Search��������������������������������������������������������������������������������������� 319
Generalized Index Types in PostgreSQL������������������������������������������������������������������������������������ 320
GIST Indexes������������������������������������������������������������������������������������������������������������������������ 320
Indexes for Full Text Search������������������������������������������������������������������������������������������������ 320
Indexing Very Large Tables�������������������������������������������������������������������������������������������������� 322
Indexing JSON and JSONB�������������������������������������������������������������������������������������������������������� 323
Summary���������������������������������������������������������������������������������������������������������������������������������� 326

Chapter 16: Ultimate Optimization Algorithm������������������������������������������������������� 327


Major Steps������������������������������������������������������������������������������������������������������������������������������� 327
Step-by-Step Guide������������������������������������������������������������������������������������������������������������������� 328
Step 1: Short or Long?��������������������������������������������������������������������������������������������������������� 328
Step 2: Short������������������������������������������������������������������������������������������������������������������������ 328

x
Table of Contents

Step 3: Long������������������������������������������������������������������������������������������������������������������������� 330


Step 4: Incremental Updates����������������������������������������������������������������������������������������������� 330
Step 5: Non-incremental Long Query���������������������������������������������������������������������������������� 330
But Wait—There Is More!��������������������������������������������������������������������������������������������������������� 331
Summary���������������������������������������������������������������������������������������������������������������������������������� 332

Chapter 17: Conclusion����������������������������������������������������������������������������������������� 333

Index��������������������������������������������������������������������������������������������������������������������� 335

xi
About the Authors
Henrietta Dombrovskaya is a database researcher and
developer with over 40 years of academic and industrial
experience. She holds a PhD in computer science from
the University of Saint Petersburg, Russia. At present, she
is a database architect at DRW Holdings, Chicago, Illinois.
She is an active member of the PostgreSQL community, a
frequent speaker at the PostgreSQL conference, and the
local organizer of the Chicago PostgreSQL User Group. Her
research interests are tightly coupled with practice and
are focused on developing efficient interactions between
applications and databases.

Boris Novikov’s experience includes leading postgraduate


research groups and advising dozens of PhD students
while partnering and collaborating with industrial clients.
His research interests are in the broad area of information
management and include design, development, and tuning
of databases, applications, and database management
systems (DBMSs). He also has interests in distributed
scalable systems for stream processing and analytics.

xiii
About the Authors

Anna Bailliekova is Senior Data Engineer at


UrbanFootprint, where she works on data platform tools
and cloud ops for geospatial applications. Previously,
she worked in data engineering at Zendesk and held a
variety of DBA and BI roles at Epic. Her writing can be
found in the newsletter Better Streets MKE. She received
her undergraduate degree with college honors in political
science and computer science from Knox College in
Galesburg, Illinois.

xiv
About the Technical Reviewer
Tom Kincaid is Vice President of Database Development at EnterpriseDB. Tom
has been developing, deploying, and supporting database systems and enterprise
software for over 25 years. Prior to joining EnterpriseDB, Tom was General Manager of
2ndQuadrant in North America where he oversaw all aspects of 2ndQuadrant’s dynamic
and growing business for Postgres products, training, support, and professional services.
He worked directly with companies from all industries and of all sizes helping them
successfully make Postgres part of their mission-critical operations.
Tom has overseen the design and delivery of Postgres training solutions as well as
the deployment of PostgreSQL both at Fortune 500 financial institutions and at military
facilities all over the world. Teams Tom has managed have delivered major features that
have become part of the PostgreSQL open source database.
Tom is also the founder and one of the organizers of the Boston PostgreSQL
User Group.

xv
Acknowledgments
It takes many people to bring a book into the world, most of whose names do not appear
on the cover. Firstly, we want to thank Jonathan Gennick, who came up with the idea
of this book and navigated it through the first edition. Without his initiative, this book
wouldn’t exist. We’re also grateful to the entire team at Apress who have supported this
endeavor across two editions.
The contributions of Tom Kincaid as the technical reviewer cannot be overstated.
His careful, thorough, and thoughtful feedback improved the content, organization,
and usability of the text. This book is more precise, more understandable, and more
comprehensive, thanks to Tom. We’re grateful that he returned for the second edition,
graciously taking another close read. Any remaining issues are, of course, our own
responsibility.
After the first edition of this book was released, we heard from many people
regarding issues in the postgres_air database, topics readers wished were covered,
and passages that could have been more clear. This second edition incorporates many
of their suggestions. We are grateful to everyone who took the time to read the book
closely and share their comments and suggestions with us. In particular, Hannu Krosing
provided thorough, detailed, and specific feedback on postgres_air, and Egor Rogov
provided many helpful suggestions for making the book more understandable and clear.

—Henrietta Dombrovskaya, Boris Novikov, Anna Bailliekova

Thank you to Jeff Czaplewski, Alyssa Ritchie, and Greg Nelson, who spent hours,
days, and weeks making No-ORM (NORM) work with Java. My time at EDB was a chance
to work with and learn from the best of the Postgres best. My colleagues at DRW—both
application and database administrator (DBA) teams—have given me new opportunities
to push the limits of Postgres.
—Henrietta Dombrovskaya

xvii
Acknowledgments

I’d like to thank Andy Civettini for teaching me how to write and talk about technical
topics in an accessible way and for years of academic and professional encouragement.
My colleagues at UrbanFootprint challenge and inspire me every day. Finally, John,
Nadia, and Kira Bailliekova have each supported me and sacrificed for the sake of this
book; I am endlessly grateful to them.

—Anna Bailliekova

xviii
Introduction
“Optimization” is a broad enough term to encompass performance tuning, personal
improvement, and marketing via social engine and invariably evinces high hopes and
expectations from readers. As such, it is prudent to begin not by introducing what
is covered, but rather, why this book exists and what will not be covered, to avoid
disappointing readers who approach it with inappropriate expectations. Then, we
proceed with what this book is about, the target audience, what is covered, and how to
get the most use out of it.

Why We Wrote This Book


Like many authors, we wrote this book because we felt we could not not write it.
We are educators and practitioners; as such, we see both how and what computer
science students are taught in class and what knowledge they lack when they enter the
workforce. We do not like what we see and hope this book will help bridge this gap.
When learning about data management, most students never see a real production
database, and even more alarming, many of their professors never see one, either. While
lack of exposure to real-life systems affects all computer science students, the education
of future database developers and database administrators (DBAs) suffers the most.
Using a small training database, students can learn how to write syntactically correct
SQL and perhaps even write a SELECT statement that accurately retrieves desired data.
However, learning to write performant queries requires a production-sized dataset.
Moreover, it might not be evident that performance might present a problem if a student
is working with a dataset that can easily fit into the computer’s main memory and return
a result in milliseconds regardless of the complexity of the query.
In addition to lacking exposure to realistic datasets, students often don’t use DBMSs
that are widely used in industry. While the preceding statement is true in relation
to many DBMSs, in the case of PostgreSQL, it is even more frustrating. PostgreSQL
originated in an academic environment and is maintained as an open source project,
making it an ideal database for teaching relational theory and demonstrating database
internals. However, so far, few academic institutions have adopted PostgreSQL for their
educational needs.
xix
Introduction

As PostgreSQL is rapidly developing and becoming a more powerful tool, more


and more businesses are choosing it over proprietary DBMSs, in part to reduce
costs. More and more IT managers are looking for employees who are familiar with
PostgreSQL. More and more potential candidates learn to use PostgreSQL on their own
and miss opportunities to get the most out of it.
We hope that this book will help all interested parties: candidates, hiring managers,
database developers, and organizations that are switching to PostgreSQL for their
data needs.

What Won’t Be Covered


Often, when users start to complain that “everything is slow” or nightly backups finish
in broad daylight and everyone seems to be saying that “the database needs to be
optimized,” conversations are almost exclusively focused on database configuration
parameters and occasionally on underlying Linux parameters. It is often assumed that as
soon as the correct values for parameters are chosen and the database instance has been
restarted, all the world’s problems will be solved.
Countless times, we have worked with customers expecting magical incantations and
cheat codes. Countless times, these customers have expressed a deep disappointment
when the “wizards” to whom they have appealed suggest looking for the queries that
are executed most often or check for missing indexes. After listening politely to all our
suggestions, they kept asking: So can you suggest any other parameter changes?
Indeed, it is tempting to be able to solve every problem at once. That temptation
leads to the popular belief that there is a sort of dark magic and secret codes, a button
hidden under a desk somewhere that makes the database run faster.
Since we are aware of these misconceptions, we want to be transparent from the
very beginning. The following is the list of topics that are often discussed in books about
optimization but will not be covered in this book and why:

• Server optimization – With the mass migration to various cloud


environments and existing organizational structures, database
developers are unlikely to have a say in how servers are configured.

xx
Introduction

• PostgreSQL configuration parameters – In this second edition of the


book, we do cover this topic. However, it comprises a relatively small
portion of the book, as their impact on performance is overrated, as
we will demonstrate, and usually, database developers do not have
the necessary privileges to alter them (with a few exceptions).

• Distributed systems – We do not have enough industrial experience


with them.

• Transactions – Their impact on performance is very limited (although


we will cover some cases when they can have a major impact).

• New and cool features – These change with every new release, and
our goal is to cover the fundamentals.

• Magic, rituals, cheat codes, etc. – We are not proficient in these


approaches to optimization.

There are plenty of books available that cover all of the topics listed previously,
except the last, but this book is not one of them. Instead, we focus on everyday
challenges database developers face: when that one application page keeps timing out,
when a customer is kicked out of the application just before the “Contract Signed” page,
when the CEO dashboard is showing an hourglass instead of yesterday’s product KPI, or
when procuring more hardware is not an option.
Everything we are presenting in this book has been tested and implemented in an
industrial environment, and though it may look like magic, we will explain any query
performance improvement or lack thereof.

Target Audience
Most of the time, a book about optimization is viewed as a book for DBAs. Since our goal
is to prove that optimization is more than just building indexes, we hope that this book
will be beneficial for a broader audience.
This book is for IT professionals working in PostgreSQL who want to develop
performant and scalable applications. It is for anyone whose job title contains the
words “database developer” or “database administrator” or who is a backend developer
charged with programming database calls. It is also useful to system architects involved
in the overall design of application systems running against a PostgreSQL database.

xxi
Introduction

What about report writers and business intelligence specialists? Unfortunately, large
analytical reports are most often thought of as being slow by definition. However, if a
report is written without considering how it will perform, the execution time might end
up being not just minutes or hours, but years! For most analytical reports, execution time
can be significantly reduced by using simple techniques covered in this book.

What Readers Will Learn


In this book, the readers will learn how to

• Identify optimization goals in OLTP (Online Transaction Processing)


and OLAP (Online Analytical Processing) systems

• Read and understand PostgreSQL execution plans

• Identify indexes that will improve query performance

• Optimize full table scans

• Distinguish between long queries and short queries

• Choose the right optimization technique for each query type

• Avoid the pitfalls of ORM frameworks

At the end of the book, we present the Ultimate Optimization Algorithm,


which guides a database developer through the process of producing the most
performant query.

The Postgres Air Database


Throughout this book, examples are built on one of the databases of a virtual airline
company called Postgres Air. This company connects over 600 virtual destinations
worldwide, offers about 32,000 direct virtual flights weekly, and has over 100,000 virtual
members in its frequent flyer program and many more passengers every week. The
company fleet consists of virtual aircraft.
Please note that all data provided in this database is fictional and provided for
illustrative purposes only. Although some data appears very realistic (especially
descriptions of airports and aircraft), they cannot be used as sources of information about
real airports or aircraft. All phone numbers, email addresses, and names are generated.

xxii
Introduction

To install the training database on your local system, please refer to the GitHub repo:
github.com/Hettie-d/postgres_air.
The README.md file contains the link to the data directory and detailed installation
instructions.
In addition, after you restore the data, you will need to run the script in Listing 1 to
create several indexes.

Listing 1. Initial set of indexes

SET search_path TO postgres_air;


CREATE INDEX flight_departure_airport ON flight(departure_airport);
CREATE INDEX flight_scheduled_departure ON flight  (scheduled_departure);
CREATE INDEX flight_update_ts ON flight  (update_ts);
CREATE INDEX booking_leg_booking_id ON booking_leg  (booking_id);
CREATE INDEX booking_leg_update_ts ON booking_leg  (update_ts);
CREATE INDEX account_last_name ON account (last_name);

We will use this database schema to illustrate the concepts and methods that are
covered in this book. You can also use this schema to practice optimization techniques.
This schema contains data that might be stored in an airline booking system. We
assume that you have booked a flight online, at least once, so the data structure should
be easily understood. Of course, the structure of this database is much simpler than the
structure of any real database of this kind.
Anyone who books a flight needs to create an account, which stores login
information, first and last names, and contact information. We also store data about
frequent flyers, which might or might not be attached to an account. A person who
makes a booking can book for several passengers, who might or might not have their
accounts in the system. Each booking may include several flights (legs). Before the flight,
each traveler is issued a boarding pass with a seat number.
The Entity-Relationship (ER) diagram for this database is presented in Figure 1.

• airport stores information about airports and contains the airport’s


three-character (IATA) code, name, city, geographical location, and
time zone.

xxiii
Introduction

• flight stores information about flights between airports. For each


flight, the table stores a flight number, arrival and departure airports,
scheduled and actual arrival and departure times, aircraft code, and
flight status.

• account stores login credentials, the account holder’s first and


last names, and possibly a reference to a frequent flyer program
membership; each account may potentially have multiple phone
numbers, which are stored in the phone table.

• frequent_flyer stores information about membership in the frequent


flyer program.

• booking contains information about booked trips; each trip may have
several booking legs and several passengers.

• booking_leg stores individual legs of bookings.

• passenger stores information about passengers, linked to each


booking. Note that a passenger ID is unique to a single booking;
for any other booking, the same person will have a different
passenger ID.

• aircraft provides the aircraft’s description, and the seat table stores
seat maps for each of the aircraft types.

• Finally, the boarding_pass table stores information about issued


boarding passes.

xxiv
Introduction

Figure 1. ER diagram of the booking schema

xxv
CHAPTER

The Key to
Digital
Transformation
Success
In the age of digitization, businesses face a critical imperative: to adapt and
embrace innovation or risk being left behind in a rapidly evolving world.
The concept of creative destruction, which is the process where innovation
and technology advancements are reshaping industries and business models,
is the driving force behind digital transformations. In this chapter, you’ll
explore what digital transformation is and the state of digital transformation
programs where unfortunately there is a high failure rate. Learn about the
challenges of digital transformation and the key role strategic alignment plays
in its success. Discover how process serves as a common language, aligning
organizations vertically and horizontally, and explore the Process Inventory

© Michael Schank 2023


M. Schank, Digital Transformation Success,
https://doi.org/10.1007/978-1-4842-9816-9_1
4 Chapter 1 | The Key to Digital Transformation Success

framework’s role in identifying, managing, and aligning processes. By the end


of this chapter, you’ll be equipped with an understanding of how Process
Inventory not only enables digital transformation’s success but also has the
power to transform how the organization integrates technical capabilities,
leads to operational excellence, streamlines the management of change, and
enhances risk management capabilities.

Nokia’s Mobile Phone Business


Founded in 1865 in Finland, Nokia initially focused on various industries,
including pulp and paper, rubber, and electronics [1]. In the 1990s, Nokia
made a strategic decision to shift its focus toward mobile phones. The
company overtook Motorola in October 1998 to become the best-selling
mobile phone brand in the world. Their net operating profits went from $1
billion in 1995 to almost $4 billion by 1999. A major reason for their success
was catering to a youthful consumer market and fashion-oriented consumers
with colorful and replaceable back covers. They were also innovators,
introducing the first camera phone to the market in 2002 and pioneering
mobile gaming with a preloaded version of Snake.
In 2007, Apple introduced the iPhone. By Q4 2007, half of all smartphones
sold in the world were Nokias, while the iPhone market share was 5% [2].
The iPhone revolutionized the mobile industry with its touchscreen interface
and robust app ecosystem. Nokia underestimated the impact of smartphones
and the changing preferences of consumers. They continued to rely on their
traditional strengths, such as hardware design and the Symbian operating
system, while underestimating the importance of software and user
experience. The Symbian platform quickly became outdated and posed
difficulties for developers after the advent of iOS and Android. Nokia’s market
share dropped by 90%, and in 2013, their mobile phone business was acquired
by Microsoft.
A study named “Distributed Attention and Shared Emotions in the Innovation
Process: How Nokia Lost the Smartphone Battle” [3] was conducted,
interviewing 76 of Nokia’s top and middle managers, engineers, and external
experts to determine the root causes. The study found that the downfall can
be attributed to a lack of organizational alignment. It highlighted several
factors, including a culture of fear, intimidated middle managers, a lack of
technical competence among top managers, and a short-term focus on market
demands. The internal politics and shared fear created a climate of mistrust
that hindered innovation.
Digital Transformation Success 5

Creative Destruction in the Age of Digitization


The concept of creative destruction, introduced by Austrian American
economist Joseph Schumpeter in 1942 [4], has impacted various companies,
such as Nokia, Blockbuster Video, Kodak, Blackberry, and Toys ‘R’ Us to name
a few. Creative destruction describes the process whereby innovation and
technological advancements disrupt established industries and business models.
According to Schumpeter’s theory, it is inherent in human nature to be
creative and strive for an improved standard of living. Additionally, the
pursuit of profit incentives drives constant innovation and technological
advancements [5]. In a competitive environment, producers generate ideas
that advance technologies, redesign existing products and processes, and
introduce new products. This competition results in improved products at
lower costs, ultimately benefiting consumers.
This continuous process of innovation and economic competition contributes
to overall economic growth and enhances the standard of living for consumers.
However, it also poses a significant risk to incumbent organizations that fail to
adapt as they can be displaced or even eliminated by more innovative
competitors. This is driving the urgency in many organizations to invest in
their own digital transformation program.

Digital Transformation Defined


Digital transformation is the application of digital capabilities to processes,
products, and assets to improve efficiency, enhance customer value, manage
risk, and uncover new monetization opportunities [6]. There are many digital
capabilities driving this, such as artificial intelligence, cloud computing, Internet
of Things (IoT), big data analytics, blockchain, and more. This goes beyond
simply acquiring these new technologies and teaching your staff to use them.
Digital transformation must be framed in a business context, which is why the
preceding definition includes “uncover new monetization opportunities.”
Monetization opportunities may mean new product revenue streams,
deepening customer relationships, expanding to new ones, or reducing the
cost of doing business. The implications of maximizing these capabilities for an
organization can be far-reaching and may fundamentally change core
business models.
The essence of digital transformation lies in becoming a customer-centric,
data-driven, and agile organization. It entails ensuring that key decisions,
actions, and processes are guided by data-driven insights rather than just
human intuition [7].
6 Chapter 1 | The Key to Digital Transformation Success

The challenge, and opportunity, is that the capabilities of these technologies


are advancing at a rapid rate, making it hard to accurately predict how they
will impact businesses in the next five years or beyond. The external
environment must also be factored in, such as changes to the competitive
landscape, customer preferences, globalization, and regulatory environment.
This means that digital transformation must be viewed as a continuous
journey, requiring a strong digital strategy and adaptive execution to respond
to changes through iterations and continuous learning. A crucial factor in
being adaptive is achieving operational excellence, which involves optimizing
and maximizing efficiency across all resources leveraged to operate your
business. By embracing operational excellence, organizations ensure their
flexibility and responsiveness in the face of advancing digital capabilities and
evolving external environments.

State of Digital Transformation


According to a study, digital companies achieve 1.8 times higher [8] earnings
growth. Technology advancements mean that digital companies have greater
productivity improvements, better customer experiences, and stronger innovation.
The global digital transformation market is projected to grow from $695.5 billion
in 2023 to over $3.14 trillion by 2030, at a compound annual growth rate (CAGR)
of 24.1% during this period [9]. About 89% [10] of enterprises are planning to
adopt or have already adopted a digital business strategy.
Shockingly, 70% of such initiatives have been found to fail to reach their goals
[8]. This represents a massive investment with a very high failure rate. The
implication is more than just lost investment; it also means frustrated
stakeholders, lost ground to competitors, unsatisfied customers, and in some
cases, costs such as high regulatory penalties.

Why Digital Transformations Fail


According to McKinsey [11], the reasons transformations fail are as follows:
• Lack of Aspiration: The CEO fails to set a sufficiently
high aspiration. People throughout the organization do
not buy in, and they lack the motivation to invest extra
energy in making the change happen.
• Lack of a Shared Vision: During the early stages of the
transformation, the CEO does not build conviction
within the team about the importance of the change or
craft a change narrative that convinces people they need
to drive the transformation.
Digital Transformation Success 7

• Lack of Engagement: People throughout the


organization do not buy in, resulting in a lack of willingness
to invest the extra energy required to drive the
transformation.
• Low Investment in Capability Building: The
leadership team neglects to address skill gaps within the
organization or fails to free up key resources from their
other day-to-day responsibilities.
• Insufficient Structure of the Transformation
Team: The organization fails to establish the right change
management structure and cadence for leadership and
transformation management oversight.
For many medium and large organizations, these can be challenging hurdles to
overcome due to their complexity. With organizations that have tens or
hundreds of thousands of employees, it becomes an immense challenge to align
and coordinate all those resources, especially when the transformation goals
are ambitious. However, complexity alone cannot be solely blamed as there are
many digitally successful organizations that effectively manage complexity.
The underlying root cause of these challenges is a lack of strategic
alignment. Strategic alignment means that all elements of a business,
including market strategy and the way the company is organized, are arranged
in such a way as to best support the fulfillment of its long-term purpose [12].
This means aligning the purpose, vision, and mission statements, which
articulate why the organization exists and the future they are driving toward,
with strategy, which is how the business will achieve that purpose. Strategy
defines which products and services to offer, which markets to serve, and in
this context, how they will adopt digital technology to further support the
purpose of the organization. Strategy needs to be aligned with the capabilities
of the organization, which are all resources including people, processes,
technology, data, vendors, etc. that support the organization. An example is
an organization that is looking to achieve superior customer service must
translate that into a high level of excellence when frontline staff are engaging
with customers directly.
There are two important aspects of strategic alignment: vertical alignment
and horizontal alignment. Vertical alignment refers to aligning strategies, goals,
and performance metrics hierarchically from the CEO through middle
management down to the single contributors. Horizontal alignment means
that business units and functional areas such as technology, finance, and risk
collaborate strongly across these areas and avoid siloed operating models,
which bring a great number of inefficiencies. Achieving strategic alignment
facilitates unification across resources which not only enables an organization
to meet their digital transformation goals but enables organizational agility.
8 Chapter 1 | The Key to Digital Transformation Success

But how do organizations achieve strategic alignment? After all, this is not a
new concept, yet it exists in so few organizations. I believe the answer lies in
language. Everyone in these organizations is focused on their responsibilities,
and they develop their perspectives based on their work. This often results in
lower-level contributors not fully understanding the strategic messages
coming from the CEO. As a result, they may fail to invest their energies in
contributing to the transformational ambitions of leadership. When a person
from a business unit conveys requirements to a technology professional, they
may struggle to accurately communicate what they are looking for, leading to
poor requirements, or delivered software that misses the mark. This lack of
understanding and effective collaboration contributes to the formation of
redundancies within organizations, where different functional units create
processes that others already perform.
A common language should create a shared understanding vertically and
horizontally. It would align objectives, facilitate better communications, bridge
cultural differences, and enable conflict resolution. What are the requirements
of this common language? First, since it must support the purpose of the
organization, it has to be oriented in the language of the business. This
eliminates, for instance, technical terminology, you wouldn’t want to state as
a strategy that you’re going to implement artificial intelligence unless you
could convey how it supports your customers, generates revenues, or furthers
the mission. But it does have to be a bridge to cross-functional topics such as
technology, risk, data, and regulators. Lastly, it needs to connect a high-level
view that senior leadership would understand to a low-level view that a single
contributor would understand.

Process Is the Common Language


The only candidate that meets those requirements is process. I’ll illustrate this
in a story that highlights how organizations evolve with process at its core.
Let’s say that you are talking to a couple of friends about starting a business.
That conversation begins with the actions or processes that must occur. The
conversation would go something like this: “We can create great designs,
print them on t-shirts, and sell them at concerts.” The purpose of the company
will be to make money or get a message out, but in that statement, the
business processes have been identified:
• Create designs
• Print design on T-shirts
• Sell at concerts
Digital Transformation Success 9

Let’s say that this company has success. Then they realize that they will have
to file taxes, but none of them have the skill set or the time, so they hire
someone for that purpose and have now added the process of
• Calculate tax liabilities
• File taxes
• Pay tax liabilities
This means that they’ve now expanded the organization chart to support the
need to do taxes. Then they purchase tax software for them which is the
beginning of their technology portfolio. The business is doing so well that you
seek out investors, but investors will want financial reports that are based on
solid accounting standards, which leads to them hire an accountant to perform
the following processes:
• Close books
• Create financial report
This accountant will require accounting software. Now they’ve started a
finance functional unit which has tax and accounting processes, an organization
chart, and a technology portfolio.
I can go on, but what you can see is that the needs of the organization to
support its purpose drive the need for processes. Process is core to
organizational capabilities as people are hired, technology is purchased or
built, and risk is incurred, all to support the processes of the business. Plus, a
hierarchy of processes evolves as processes get more complex and lower-
level resources are hired to perform more detailed tasks.

The Process Inventory Framework


The Process Inventory is a framework for capturing every process within an
organization and a methodology for managing this information sustainably
over time. One of the guiding principles for Process Inventory is that it must
be a complete representation of processes for the organization in scope. To
accomplish this, interviews must be conducted following the organizational
chart since that represents a complete inventory of people in the organization.
These interviews start at the top of the organizational chart, working down
from business units or functional levels to the team or individual contributor
level. Stakeholders are requested to provide formal attestations to ensure
completeness and accuracy in the inventory, including any process inventories
collected by their teams.
10 Chapter 1 | The Key to Digital Transformation Success

Starting from the top of the organizational chart enables vertical alignment,
allowing for a direct trace of process ownership from the CEO to individual
contributors at all levels.
Organizations often have multiple sources of information that describe
various aspects of their operations, such as application repositories, risk
repositories, product, and channel repositories, and more. However, this data
is usually managed in silos and lacks concrete associations across topics,
hindering horizontal alignment.
The Process Inventory framework establishes a taxonomy that classifies all
processes executed to support the organization’s purpose. This taxonomy
serves as an index for aligning different aspects of the organization through a
single business-oriented language. A modeling team can extract information
from these repositories and create associations, at the process name level or
through creating process models, where operational information intersects
with processes.
This achieves horizontal alignment in three ways. Firstly, it enables functional
teams to precisely identify how their concepts intersect with processes. For
example, by examining the intersection of applications in an application
repository with processes, it becomes possible to determine which applications
support specific processes and even drill down to the level of identifying which
APIs within those applications support specific steps in a process. This is
valuable in identifying the scope and requirements for change management
initiatives. Secondly, it provides transparency to all stakeholders regarding the
activities of each business unit and functional team. This transparency helps
break down silos and facilitates better coordination. Lastly, it promotes a
culture of accountability by identifying points of ownership and fostering
strong collaboration across organizational units where necessary.
This framework is analogous to Google Maps, where you can view Earth from
space, but then you can zoom in to see details of your house and the signs on
your street. Google’s initiative called Ground Truth [13] marries numerous
data sets about our world, such as traffic patterns, restaurant ratings, and
street view images to physical locations. This has revolutionized not only
navigation but also how we interact in the world and how advertisers reach us.

The Digital Transformation Journey


In a digital transformation, this common language enables the leadership team
to articulate their aspirations for a digital future, and the transformation
teams can identify specific processes that need to change or processes to be
added. The shared vision can be cascaded throughout the organization, clearly
defining responsibilities and performance metrics to drive the transformation.
This culture of accountability will foster engagement as everyone will
Digital Transformation Success 11

understand how their work directly contributes to the bigger picture.


Essentially, this enables solutions to the typical failures encountered in digital
transformations.
In this book, I will extensively detail the constructs of this framework, including
the operating model and the methods to create and maintain the process
inventory and associated metadata sourced from various repositories.
I will cover the method of defining a digital strategy and how to set up the
transformation program for success. The core of this process involves defining
the digital transformation strategy in the context of how it supports the
organization’s purpose and aligns with operational capabilities. This
encompasses defining the business areas and use cases that need improvement,
along with the ways of working. I provide details on how the Process Inventory
framework can transform these use cases.
A significant aspect of adapting to the digital age is integrating new technology
capabilities. Technology is typically one of the largest investments an
organization makes and can be a source of pain, especially when the technical
architecture becomes rigid. I will detail how this framework can lead to much
better IT architecture documentation, creating alignment between IT assets
and the processes they serve. This visibility can help rationalize complexity by
eliminating redundant systems and increasing reuse, which is key to flexibility
and efficiency. Additionally, I will demonstrate how this directly supports the
design and implementation of digital technologies such as intelligent process
automation, the adoption of artificial intelligence and machine learning at
scale, and how to build a business-led modular architecture design, which is
crucial for an agile architecture.
Operational excellence is achieved with this level of transparency at your
fingertips. You can precisely identify all the resources that support a business
or process area, enabling the identification of inefficiencies or opportunities
to streamline operations. Continuous improvement will be promoted by a
framework that enables better monitoring of business processes and the
collection of key performance indicators.
Workforces will be transformed by having greater training and development
programs to equip staff members with skills and knowledge, not only in their
domain but also with a broader understanding. This will foster a culture of
innovation by unlocking knowledge that was held by so few and encouraging
everyone to identify better ways of doing things.
Change management is a significant challenge for many organizations,
particularly in large change efforts. This framework has the potential to greatly
transform the entire process. The alignment of strategic priorities can be
mapped to the impacted processes. This clarity brings more objectivity to the
change portfolio investment process, resulting in stronger effectiveness in the
allocation of discretionary funds to projects with the greatest strategic needs.
12 Chapter 1 | The Key to Digital Transformation Success

Project scope can be stated in terms of the business processes involved. This
approach ensures certainty of impacts across all resources leveraged by the
business process, including people, processes, controls, data, and technology.
The result is enhanced cost and timeline estimates and better resource
alignment.
The change process would undergo significant changes by anchoring it to this
framework. Framing the contents of SDLC, Agile, and organizational change
management deliverables, such as business requirements, agile user stories,
technology designs, control designs, testing scripts, and user training, using
the Process Inventory taxonomy as the structure will increase coordination
across teams and enhance traceability for those responsible for project
management in these efforts. This also has the benefit of generating better
change status reports by communicating the progress and readiness of
individual processes as the numerator for percent complete reporting.
The efficiencies provided by this framework will enable faster change
implementation with reduced investment needs, thereby freeing up resources
to concentrate on enhancing customer experience and driving product
innovations.
Risk management is strengthened by giving risk professionals a
comprehensive inventory of processes, which will enable them to perform
better risk assessments and create a more accurate inventory of risks across
all risk types. This allows them, in conjunction with their business partners, to
prioritize and design effective controls and ensure regulatory compliance. The
risk operating model will be empowered across all three lines of defense for
stronger effectiveness and coordination such as arming internal audit with
information needed to effectively test controls. This will give senior leaders
and boards confidence that they are getting accurate risk information to make
their strategic decisions.

Framework Implementation Considerations


This may seem overwhelming, but I assure you it’s not. I have dedicated 12
years of my career to developing and testing this framework in various
scenarios. I have successfully implemented it at multiple large financial
institutions, tailoring it to their specific needs. I have refined the methods to
be efficient with limited resources and to ensure the quality of the models,
which forms the basis for all the benefits I have presented.
The framework is designed to be incremental and scalable according to your
requirements as defined by your transformation strategy. It can be applied at
the enterprise level or selectively to specific business units or functions. You
have the flexibility to implement only the use cases that are relevant to your
organization.
Digital Transformation Success 13

Indeed, adopting this framework requires an investment of time and resources.


However, it’s important to recognize that organizations pay a price for
misalignment. You have a choice: invest upfront to avoid inefficiencies, costly
penalties, or losing market share to more digitally adept competitors, or bear
the consequences later.

Key Takeaways
• Creative destruction is driving the digital age through
innovation and technological advancements which are
reshaping business models and industries.
• Digital transformation is the application of digital
capabilities to processes, products, and assets to improve
efficiency, enhance customer value, manage risk, and
uncover new monetization opportunities.
• High failure rate as 70% of such initiatives fail to reach
their goals, which means lost investment, frustrated
stakeholders, lost ground to competitors, unsatisfied
customers, and potentially high regulatory penalties.
• Lack of strategic alignment is the underlying root
cause of failure which hinders collaboration and
organizational focus toward strategic goals.
• A common language is needed to facilitate alignment.
Process, as captured by the Process Inventory framework,
provides the common language.
• Process Inventory enables digital transformation
success and transforms how the organization operates
which leads to operational excellence.
In the next chapter, we will describe the Process Inventory framework, which
includes models, modeling characteristics, metadata, and conceptual
architecture.
CHAPTER

Overview of
the Process
Inventory
Framework
In this chapter, you will explore the components of the Process Inventory
framework. Delve into the significance of this framework in establishing an
ontology to organize organizational knowledge. Understand the methods for
constructing and maintaining a Process Inventory, and explore the conceptual
architecture used to integrate metadata from diverse authoritative sources.
Additionally, you’ll gain insights into related models, such as process models,
customer journeys, and value streams. The framework, with all its associated
components, is critical for structuring and utilizing knowledge effectively
within an organization to deliver transformational value.

© Michael Schank 2023


M. Schank, Digital Transformation Success,
https://doi.org/10.1007/978-1-4842-9816-9_2
16 Chapter 2 | Overview of the Process Inventory Framework

 he Origins of the Process


T
Inventory Framework
I started my career in 1996 at Andersen Consulting, which later became
Accenture. My college degree had a focus on information systems, so I began
in a technology practice. My first engagement was with a large bank in
Charlotte that was developing a direct banking platform, an emerging concept
at that time.
I initially joined a production support team where we handled issues with
middleware code. Middleware is an application pattern for distributing
messages to appropriate platforms. My role involved triaging issues, identifying
root causes within the application code, and developing and testing fixes.
During the first 15 years of my career, I primarily focused on technology. I
worked as a C++ and Java developer and a solution architect and led large
teams in software delivery.
I transitioned from retail banking to commercial banking without having a
strong understanding of these businesses. While I knew about retail banks
since I was a customer with a checking account, commercial banking was
completely foreign to me. My focus was on delivering quality software on time
and supporting production to minimize customer impact. Most of my
interactions were with other technologists; I was curious about what the
business did, but I didn’t have a clear path to explore it.
In 2011, I joined EY, which introduced me to a new world. At that time, EY
had limited software development engagement, and their clients were typically
senior leaders across various business areas such as risk, technology, and
more. The work involved helping these leaders develop solutions to enhance
their organizations’ effectiveness. It was a refreshing shift from my technology-
centric mindset.
One of my early engagements at EY was with a large retail bank. They had
surplus budgeted funds that needed to be spent; otherwise, the funds would
be lost the following year. One of their requirements was to build a business
capability model for their retail banking business. While not an exhaustive
exercise, we interviewed key individuals in their businesses and created a
model with three levels of depth. It was a moment of epiphany for me. The
entire business’s activities were encapsulated in a single model, exactly what I
had been seeking during my technology days to understand what the business
did. I was enthused about the potential of this concept to bring diverse
perspectives together and what this could mean for delivery on a wide range
of topics. From that point on, I positioned myself as a business architect,
eager to delve deeper into this concept, and using this branding helped me
connect with clients implementing similar approaches.
Digital Transformation Success 17

BUSINESS ARCHITECTURE

Business architecture emerged as a concept in the 1980s. British management


consultant Edwin E. Tozier was one of the first to mention the concept in an article
he wrote titled “Developing Strategies for Management Information Systems” [1].
In this article, he distinguished between information architecture and business
architecture, stating his theory that each business function may be supported by one
or more information systems and that information systems need to be adaptable to
organizational strategies.
Business architecture has evolved significantly since then. Several standard-setting
organizations such as TOGAF (The Open Group Architecture Framework), OMG (Object
Management Group), the Business Architecture Guild, and others have been advancing
the concepts and practices, although a common definition remains elusive.
The definition that has resonated the most with me is from OMG, which states
that business architecture is “a blueprint of the enterprise that provides a common
understanding of the organization and is used to align strategic objectives and tactical
demands” [1]. This definition always made intuitive sense to me as changes to a
physical building wouldn’t be made without such a blueprint, but many organizations
will spend significant amounts of money without any similar concept.
The core deliverable in business architecture is the business capability model. A
business capability model represents the required capabilities that an organization
must deliver for its purpose. They are largely independent of specific organizational
structures, reporting relationships, political agendas, cultural aspects of individual
business leaders, current initiatives, and projects [2]. Business capability models have
been used for various purposes, such as strategic planning, where a heatmap may
be created to highlight the capabilities in most need of improvement and investment.
They are also used to map assets, such as IT systems, which enables better
communication with teams involved in implementing strategies. Generally, however,
business architecture and business capability models do not model specific processes
as their aim is to be high level and agnostic to organizational structure.

Shortly after that, I was brought in to assist a client in the mortgage industry.
As a mitigation strategy following the 2008 financial crisis, they had decided
to outsource several critical business functions to a third-party provider.
However, the project was in deep trouble, with status reports showing red on
most measures despite a significant investment made in terms of dollars and
resources. The requirements team lacked structure and was not making
progress toward delivering a complete and high-quality product. Additionally,
the business architecture team they had in place was struggling to add value.
Recognizing the need for a strategic pause to recalibrate the approach, the
senior client stakeholders gave us a three-week timeframe.
18 Chapter 2 | Overview of the Process Inventory Framework

The business architecture team was implementing an approach I had not


encountered before. They utilized a sophisticated modeling platform and
connected their business capability model to detailed process models. During
the pause, I collaborated with the leadership of the business architecture team
to develop a comprehensive plan. This plan included a stakeholder engagement
model, a rigorous approach to work product creation, developing standards, a
quality assurance process, and even an estimating model for more precise
delivery date projections. When the work resumed, we essentially created a
Process Inventory, although that was not the name used at that time. We
conducted interviews with functional areas to capture every process they
performed and then created process models to detail how those processes
were executed. These inventories and process models underwent a stringent
quality assurance review to ensure compliance with standards before the
identified process owner attested to their accuracy. This structure enabled the
requirements team to measure their progress toward completion, gain a
detailed understanding of the process operations, and obtain sign-off on final
requirements from named process owners. By establishing this factory-like
model with clear activity sequences and accountabilities, the project transformed
from red to bright green. This experience taught me valuable lessons, particularly
the anchoring power of this concept in large-scale transformation programs. I
also recognized the importance of connecting business processes, maintaining
high standards, and prioritizing quality as critical success factors.
From that point on, I sought to refine this methodology and test it in diverse
scenarios, including an insurance spin-off, a bank merger, strategic architecture
design for a wealth management firm, business continuity planning for a capital
markets firm, and risk management for a global financial utility.
My focus shifted to selling this concept within the financial services industry,
which put me in front of numerous clients with complex problems to solve.
While many of these clients acknowledged the significance of process within
their solutions, they struggled to articulate how it would play a role. For
others, I seamlessly integrated it into proposals, knowing it would be beneficial.
Regardless, I achieved success in selling the approach as its intuitive appeal
resonated with everyone.
This chapter delves into the framework’s details, encompassing various
models and their interconnectedness, providing a solid foundation for
transformation initiatives.

The Process Inventory Framework


In Chapter 1, I described how Process Inventory is a common language that
facilitates strategic alignment, which is critical for transforming an organization
into the digital age and operating with excellence. This is certainly true, but
I’d like to go a bit deeper to describe what it represents.
Digital Transformation Success 19

The Process Inventory framework represents an ontology for an organization.


The word “ontology” gained usage with ancient philosophers like Plato and
Aristotle. “Ontology” comes from the Greek words “ontos,” meaning “being,”
and “logos,” meaning “study” or “discourse” [3]. As they defined the term,
ontology was the study of what exists and how the things that exist relate to
each other, providing answers to fundamental questions about existence.
In modern times, ontologies are used in many fields such as information
sciences, Semantic Web, bioinformatics, healthcare, natural language
processing, artificial intelligence, and more. Today, the concept is defined as a
formal representation of knowledge that encompasses a set of concepts, their
properties, and the relationships between them within a specific domain. It
provides a shared understanding of the structure and semantics of a domain,
enabling effective knowledge organization, reasoning, and information
integration [4]. To put this in layman’s terms for an organization, it maps out
how all the resources and concepts intersect to support the mission of the
organization.

Purpose & Strategy


Vision Customers
Mission
Product & Channels Customer Persona
Goals
Customer Journeys
Objectives

Product Category Channels


Tactics Moments of Truth

Product Subcategory
Interactions
Product

Process Risk Management


Enterprise Architecture
External
Events
System Interfaces Systems Control
Functions Risk
Regulatory
Obligation Laws,
Platform / Service Regulations

Vendor Contract Data


Element
Vendor
Level N Data Enterprise Data
Entity
Vendor Management Data
Level 2
Subject
Area
Organizational Level 1
Structure

Figure 2-1. Organizational Ontology


20 Chapter 2 | Overview of the Process Inventory Framework

Figure 2-1 is a visual representation of this ontology. As you can see, and this
is not an exhaustive representation, there are numerous concepts that must
come together to support an organization. Organizing knowledge in a specific
domain is typically done through a taxonomy. Taxonomies are classification
systems in which entities in domains are placed in hierarchical categories that
are exclusive and exhaustive. Exhaustive means that every possible category,
especially at the top of the hierarchy, is represented, and exclusive means that
every entity belongs to only one category. To illustrate this, I’ll use the classic
example of the animal kingdom taxonomy. At the top level of the hierarchy,
each animal is classified as vertebrate or invertebrate. A level below vertebrates
are reptiles, birds, fish, mammals, and amphibians. The levels go deeper, but
this taxonomy is exhaustive as there are no other high-level categories. It’s
exclusive in that any animal can exist in one and only one category.
As represented in Figure 2-1, each of the concepts that support an organization
is its own taxonomy, which makes the ontology for an organization a
polycategorical ontology, meaning knowledge is to be understood in how
multiple taxonomies intersect. In polycategorical ontologies, entities are
distinguished by how fundamental they are to the ontology, meaning how
dependent their existence is relative to other entities. I discussed in Chapter 1
that the organization’s purpose and strategy are the most fundamental to the
existence of everything, but after that, the process is fundamental to all
capabilities of the organization. For instance, an accounting system wouldn’t be
needed without the need for accounting processes.
The earlier ontology definition includes understanding the relationships
between concepts within a specific domain; in this case, the domain is the
organization. So, a consistent approach for mapping the concepts between
these taxonomies is needed to have a coherent representation of knowledge.
Since process is the most fundamental to the capabilities of the organization,
it must be the top-level concept, meaning the entities in other taxonomies
need to be mapped to how they support processes in the Process Inventory
taxonomy. That is how process becomes the common language of the
organization within this ontology.
Since organizational units are constructed to support the need of processes,
this ontology explicitly maps disparate concepts to the Process Inventory
taxonomy, providing horizontal alignment of knowledge and information.
For the remainder of this book, I will refer to the integration of these concepts
as process metadata, which is information about the process. There are
two broad categories of metadata. The first is operational information
metadata, which provides information on the process’s existence. This
includes the products it supports, the legal entity it belongs to, or who owns
the process, or the business capabilities it represents. The second is resource
metadata, which details the resources leveraged in the execution of the
Digital Transformation Success 21

process. This includes systems, data, channels, organizational units, third-


party vendors, etc. The implication for modeling is that operational information
metadata can be designated at the taxonomy or process definition level, while
resource metadata can be further detailed through process models.
Figure 2-2 shows the same model but framed hierarchically. Since Process
Inventory is constructed following the organizational structure, then it
provides views that are relevant from the CEO through a low-level individual
contributor, which is key to vertical alignment.

Why
Purpose – Vision, Mission
Strategies - Goals, Objectives, & Tactics

Motivates Delivers

The Process Inventory Framework

Customer
Personas

Engages Engages
What
Customerr Journeys / Value Stream
lue/offerings to customers
What value/offerings

Leverages Enables Who


Process Inventory
at the business does
What

Defines Implements

Processes How
How
w processes executes

Requires Supports

Process Metadata With What


Actors, systems,
ems, data, risk/controls, metrics

Figure 2-2. The Process Inventory Framework

This framework clearly delineates aspects of the 5W (and 1 H) model, which


are crucial for understanding the motivational system of the organization.
This provides a tool that helps organizations connect the “Why” in terms of
22 Chapter 2 | Overview of the Process Inventory Framework

their purpose and strategies to the “What” processes they execute to achieve
their goals, the “Who” responsible for the results, and the “How” those
processes are performed, as well as “With What” resources are leveraged.
The framing in Figure 2-2 will be leveraged in the rest of this chapter to
describe various parts of this framework.

Process Inventory
Definition
Merriam-Webster defines a process as “a series of actions or operations
conducing to an end.” I point out this definition to highlight that this framework
takes a generous definition, meaning that the goal of Process Inventory is to
identify every action that is performed by anyone in the organization, any
customer-facing technology systems, and any outsourced third-party
providers.
The result of capturing all processes is an end-to-end process framework.
I define this through the characteristics of (1) capturing everything an
organization does and (2) being able to model processes from start to finish
as they traverse different parts of the organization.
Here are some characteristics of Process Inventory:
• Comprehensive Coverage: The goal of Process
Inventory is to capture everything that a business does
within an identified scope. This is achieved through an
interview process that aligns with the organizational
hierarchy as highlighted in Figure 2-3.
• Accuracy Through Attestation: Since Process
Inventory is constructed through the organizational
hierarchy, this provides ownership to review the accuracy
of the model after its initial creation and in each
subsequent maintenance review. Asking stakeholders for
formal attestation encourages them to pay attention to
the relevant details and ensure accuracy.
• Alignment with Strategy: The transparency of
processes enables a clear line of sight in how each process
contributes to the achievement of the organization’s
goals. It can be assessed in terms of its strategic
importance and performance metrics.
Digital Transformation Success 23

• Top-Down Decomposition: High-level activities as


seen by the CEO or senior leadership are made up of
subprocesses, and those subprocesses can be made up of
other subprocesses. This model provides the ability to
drill down to the appropriate level of detail given the
encountered problem.
• Models Current State and Future State: Capturing
Process Inventory is typically done on the current
inventory of processes, but it can be useful to show
future state processes when executing a transformation
program.
• Process Levels: Since this inventory is constructed by
leveraging an organizational hierarchy, there is no set
level for documenting process models, such as Level 3, as
different organizational hierarchies can be built at
different levels of depth.

Organizational Hierarchy

Head of Widget
Business

Pat

Process Inventory
Product Customer Widget Business
Sales Lead
Management Service Lead
Lead
Morgan Manage Sales Manage Products Manage Customer Service
Riley Hunter

Manage Sales Operations Conduct Product Research Manage Call Centers

Sales Operations Product Create Sales Report Generate Metrics Report Address Customer Questions
Call Center
Research
John Kyle Onboard Salesforce Conduct Customer Interviews Support Transaction Processing
Amy
Create Sales Material Summarize Opportunities Triage Customer Issues
Business Product Design
Development Complaints Support Sales Software
Alex Kim
Jane
Business Development Product Design Complaints Handling

Identify Prospects Manage Improvement Backlog Perform Initial Complaint Triage

Conduct Sales Call Create product design Research Complaint

Negotiate Contract Terms Plan enhancements Create and Propose Resolution

Close Contract Test and Validate Test and Validate

Perform Post Launch Review Conduct close out discussion

Figure 2-3. The Process Inventory Identification Process

Quality Is Paramount
Since Process Inventory serves as the common language for organizing
organizational knowledge, ensuring quality is crucial. These models contribute
to a repository of information interconnected with other process metadata.
When stakeholders begin utilizing this information in their roles, encountering
significant quality errors can undermine their confidence in the effort.
Overcoming negative perceptions of poor quality can be challenging.
24 Chapter 2 | Overview of the Process Inventory Framework

For each model, I will outline an approach to construction that includes


multiple quality review points and stakeholder sign-offs. Adopting these
quality checkpoints is of utmost importance to prevent costly or fatal
mistakes. It is considered a best practice to conduct internal standards
alignment reviews before seeking stakeholder attestation, thereby avoiding
the need for them to re-attest if quality issues are discovered.

Process Naming
One principle that is important when constructing the inventory of processes
is to provide clarity regarding the actions taking place. A good test for that is
whether someone not familiar with the business unit or functional area can
intuitively understand what the process accomplishes just by reading its name.
One effective way to achieve this is by adopting a verb + noun naming
convention. Since processes represent actions, the verb specifies the action,
and the noun represents the entity on which the action is performed. Examples
of good process names include “Open Account,” “Submit Payment,” and
“Document Dispute Resolution.” Additionally, when naming processes, it is
advisable to minimize the use of generic verbs, such as “Manage,” as they lack
precision in describing the action being taken.
Furthermore, it is crucial to use the language of the business when engaging
stakeholders as they are being asked to take ownership and be accountable
for performance. The process names should align with their existing vernacular.
For instance, if they refer to a process as “Onboard Client,” it is important
not to force them to call it “Open Account” just to match the terminology
used by other business units.
In my experience, process owners value consistency and readability that
standards bring while still allowing flexibility to align with their preferred
vernacular.

Maintenance
When embarking on this work, one of the initial concerns raised is how to
maintain the model’s accuracy. This is a significant question, considering the
constant changes that occur within many organizations, including the
deployment of large-scale programs and organizational restructuring.
Addressing this concern involves two key components.
Firstly, each process area must undergo periodic attestation, such as yearly,
wherein the process owner identifies and highlights any changes and
subsequently attests to the revised model. This ensures that updates are
captured and acknowledged. Periodic attestations are not a new concept as
they are used in several topics such as access controls, financial reports, and
compliance reporting just to name a few.
Digital Transformation Success 25

The second aspect involves integrating this model into the organization’s
business management practices. Part 2 of this book will delve into this topic,
encompassing the management of efficient operations, change management,
risk management, and technology environment management. By instilling a
sense of shared ownership among stakeholders, with everyone actively
invested in maintaining an up-to-date model, integrating and attesting changes
on a near real-time basis becomes more streamlined.

Process Ownership
A significant advantage of this framework lies not only in identifying the
business processes within an organization but also in providing details
regarding accountability for each process and process area. Process owners
hold responsibility for executing processes and, in many cases, for identifying
and driving changes to those processes. Since Process Inventory is built upon
the organizational hierarchy, none of these accountabilities are new. However,
adopting this framework enhances transparency by clarifying ownership and
may lead to more defined performance metrics for both processes and their
respective owners.

Process Inventory

Business Name Leader 1

Widget Business
Widget Business
Leader

Manage Sales Sales Operaons


Operaons Leader

Idenfy Prospects Process Owner

Figure 2-4. Process Ownership Model

As depicted in Figure 2-4, every point in the Process Inventory taxonomy will
have an identified owner. This implies that the entire reporting structure will
bear some level of accountability for process performance.
26 Chapter 2 | Overview of the Process Inventory Framework

This structure facilitates easier maintenance of the Process Inventory when


ownership transitions occur. In such cases, contacting the leader higher up in
the chain allows for the identification of new owners.
Furthermore, this structure not only brings transparency to process
accountability but also enables the cascading of goals throughout the
organization. It ensures that lower-level teams have a clear understanding of
how their efforts contribute to the organization’s overarching objectives.

Omnichannel Processes
An exception to identifying processes by following the organizational
hierarchy is with omnichannel processes. Many products that organizations
offer allow for multiple customer contact points via various channels. These
channels can include in-person interactions through physical stores or phone
calls, as well as digital interactions through websites, mobile phones, or other
electronic interfaces. Omnichannel aims to provide a seamless and effortless,
high-quality customer experience that occurs within and between contact
channels [7]. This can be achieved by providing a consistent experience to
customers across all channels. However, it can become much more
sophisticated by offering customers a hyper-personalized experience that
leverages insights gained from the knowledge of the customer, including
tracking their behavior as they interact across channels. This may also involve
providing customers with the ability to start a transaction in one channel,
such as the Web, and seamlessly continue it in another channel, such as
physically going to a store.
A critical success factor in implementing the organization’s omnichannel
strategy is having a clear inventory of interactions, and by which channel,
customers can interact with for a product.
A starting point for this is the product manager for each product. They can
provide the list of transactions or processes associated with that product and
indicate the channels in which those processes can be executed. This list
should then be included in the Process Inventory for each channel. Having
consistent process names across channels will enable teams to harmonize
behaviors and collaborate with technology experts to develop the necessary
back-end platforms that facilitate the omnichannel strategy.

Outsourced Processes
Many organizations outsource business processes to vendors for various
reasons, such as not considering them part of their core value-add or lacking
the necessary skills and resources. It is crucial to note that while the execution
of certain processes may be delegated to a third-party vendor, the accountability
Digital Transformation Success 27

for their performance remains with the organization. This means that if a
vendor experiences a high-profile data breach or performance issue that
impacts operations or attracts regulatory attention, the originating
organization is held responsible. To address this, it is important to include the
outsourced vendor processes in the Process Inventory.
Depending on the contractual arrangement, the vendor may not provide
specific details on how the processes are executed. However, having clarity on
the list of outsourced processes enables better monitoring of service level
agreements (SLAs) and facilitates stronger management of these vendors. By
monitoring performance against agreed-upon SLAs, organizations can
proactively address any issues, ensure compliance, and maintain effective
oversight of their outsourced processes.
Incorporating these considerations into the Process Inventory helps
organizations maintain transparency, mitigate risks, and uphold their
accountability even when processes are outsourced to external vendors.

Integration with Business Capability Model


and Standard Process Models
In the definition I provided on business architecture and business capability
models earlier, I mentioned that they are largely independent of specific
organizational structures. There are also standard process taxonomy models
for various industries, such as the APQC Process Classification Framework,
which have adopted the same principle of being agnostic of specific
organizational structures. This means that a capability or a process will be
represented once and only once in those models regardless of how many
business units or functional areas execute those processes.
However, this poses a challenge when trying to rationalize overlapping
processes, standardize how certain processes are executed, or identify risks
and implement effective controls. Leadership and practitioner teams can lose
sight of these idiosyncrasies.
That doesn’t mean these models lack value. They are extremely valuable in
highlighting similar processes across disparate business units or functional
areas. Since the Process Inventory leverages the specific vernacular of the
business, it may not be immediately apparent that processes across different
groups are performing the same tasks.
The ideal solution is to leverage both models and map them together, providing
each stakeholder with the information they are seeking. For example, a risk
team that needs to understand all the processes executed by an organizational
unit will see all the processes in that area and how their execution may pose
risks. On the other hand, a design team aiming to rationalize processes to
28 Chapter 2 | Overview of the Process Inventory Framework

create a shared services organization will gain insights into the process
owners, their unique requirements, and how those must be implemented in
the new organization.

Table 2-1. Highlights the Main Differences Between Process Inventory and a Business
Capability or Standard Process Model

Process Inventory Business Capability or


Standard Process Model
• Constructed via the organizational hierarchy • Agnostic of organization structure
• Represents a complete inventory of all the • Represents a complete inventory of
processes that a business unit or functional all that a business can do
area performs
• Changes with organizational or pro- • Stable and only changes with
cess changes significant organizational changes
• Requires interviewing the business • Can be obtained and leveraged
immediately

Figure 2-5 shows the APQC Process Classification Framework. This diagram
is their industry-agnostic model, but they also have tailored ones for different
industries. The standardization is clear from the top-level view. This model is
agnostic to organizational structures.

Process Classification Framework Levels

Level 1 – Category
• 3.0 Market and Sell Product

Level 2 – Process Group


• 3.5 Develop and manage sales plans

Level 3 – Process
• 3.5.2 Manage Customers and Accounts

Level 4 – Activity
• 3.5.2.5 Manage Customer Master Data

Level 5 – Tasks
• 3.5.2.5.1 Collect and merge internal and third-
party customer information

Figure 2-5. APQC Process Classification Framework


Digital Transformation Success 29

Figure 2-6 illustrates a real-world situation which is relatively simplistic, in


which two different business units of the organization have sales teams. In
many cases, especially with organizations with multiple business units or
geographies, it can be hard to understand the overlap. Also, as you get deeper
within an organization, it’s possible that you’ll find processes that are
performed in an organization that are outside of the original mission of that
group. An example is when a business unit creates a shadow IT [5] organization
because they perceive or experience limitations in the central IT organization.
Deciphering the details and performing alignment across the enterprise is
made possible when both models are leveraged together.

Organizational Hierarchy

Chief Executive
Officer

Head of Widget Head of Professional


Business Services Business

Process Inventory Process Inventory


Sales Lead Sales Lead
Manage Sales Manage Sales

Manage Sales Operations Business Development

Create Sales Report Identify Prospects


Business
Onboard Salesforce Sales Operations Conduct Sales Call
Development
Create Sales Material Offer Perspectives
John
Support Sales Software
Contract
Market Research
Negotiations
Contract Development
Market Research
Identify Scope
Identify Customer Segments
Estimate Pricing
Conduct Sales Call
Negotiate Contract Terms
Negotiate Contract Terms
Close Contract
Close Contract

Figure 2-6. The Process Inventory Identification Process

The solution is illustrated in Figure 2-7. The Process Inventory is created then
each process identified is mapped to the lowest level available and appropriate
in the Standard Model.
Random documents with unrelated
content Scribd suggests to you:
shall ever sleep in this house.'—Thereupon the individual, who had
so expressed himself, descended the stairs with a tremendous
stamping of his feet, as if he were very indignant at the treatment
he had complained of; and Miss Matilda—one of the young ladies
who had visited in my room on the preceding evening—returned into
her apartment, banging the door violently behind her. This incident
opened my eyes to the dread truth:—I was in a brothel!

"I threw myself on a chair and burst into a flood of tears. Merciful
heavens! for what fate was I reserved? Had I indeed fallen so low
that my only home was a loathsome den of iniquity like that? For
some minutes after the occurrence of the incident just related, I felt
as if my senses were leaving me. Suddenly the door opened, and
Mrs. Harpy made her appearance. She seemed astonished at the
condition in which she found me, and was about to make some
remark, when I threw myself at her feet, exclaiming, 'I conjure you,
madam—if you have any pity for a poor friendless orphan—let me
leave your house this moment!'—'And where will you go, my dear
child?' she said.—'To the workhouse, ma'am: anywhere, rather than
remain here!' I answered.—'This is a pretty recompense for my
kindness towards you,' she observed. 'If it had not been for me, you
would have died in the streets.'—'Far better for me were it, had I so
perished!' I exclaimed.—'Now, Miss,' cried Mrs. Harpy, growing
angry, 'what is the meaning of all this nonsense?'—'Can you ask
me?' I demanded. 'Oh! that the feelings which prompted you to
assist me, should have been any other save the disinterested
benevolence for which I so sincerely thanked you!'—'Then you know
where you are, Miss, I suppose?' she said, with a leer; and, before I
had time to give any reply, she added, 'I meant you to find it out in a
day or two; and it's as well now as a few hours later. Here you are,
and here you will stay. You shall be treated just in proportion as you
behave; and this evening, I shall introduce some fine nobleman or
gentleman to you.'—'Never!' I cried: then moving towards the door, I
said, 'Detain me at your peril!'—'So I shall,' answered Mrs. Harpy,
coolly. 'I've got your I. O. U. for twenty pounds; and if you go any
where, it will be to Whitecross Street prison, before you're many
hours older. Remember, it's for necessaries; and so no plea of
minority or any other gammon of that kind, will avail you.'—I
remembered the slip of paper which I had signed; and my heart
sank within me, as I saw how completely I was in the power of that
vile woman.—'So now you understand how you are situated,' she
continued, softening in her tone and manner. 'This is what all young
girls like you must come to, sooner or later; and you'll be very happy
here, I can assure you. This evening a nobleman who patronizes my
house, will call upon you; and if you have any of your nonsense with
him, I'll send you straight to Whitecross Street to-morrow
morning.'—With these words she left the room, locking the door
behind her.
"I cannot attempt to explain the nature of my feelings during the
remainder of that day. A good dinner was sent up to me; but I could
not eat a mouthful. The servant asked if I should like to see any of
the 'young ladies;' and I answered in a manner which convinced her
how I recoiled from the detestable proposal. She smiled—as I
thought, significantly,—as much as to say, 'You will talk differently in
a very short time.'—At about nine o'clock Mrs. Harpy sent up word
that I was to dress myself in my best attire—a command with which
I positively refused to comply for I was determined that, happen
what might, I would not assist in the sacrifice of myself!
"At ten o'clock the servant brought up waxlights, and a tray
containing a bottle of champagne, glasses, and several plates of
fruits and cakes. I watched these preparations in a state of dumb
despair, bordering on stupefaction. Another half hour passed; and
steps once more ascended the stairs. My heart palpitated violently!
The door was thrown open;—a man elegantly dressed entered the
room;—I cast one glance towards him, and, uttering a faint cry, sank
insensible on the carpet. It was Lord Dunstable!
"When I awoke, I found that nobleman hanging over me, bathing
my temples. He compelled me to drink a glass of wine; and I soon
recovered full consciousness of the miseries of my condition. Starting
from the half-embrace in which Lord Dunstable had clasped me, I
surveyed him with horror. 'Do I frighten you, Lydia?' he exclaimed. 'I
must confess that our meeting is a strange one. The old woman sent
to tell me that she had a prize; but I little expected to find you
here.'—'My presence in this house of infamy, my lord,' I answered,
'is one of the links in that chain of degradation of which you forged
the first link. To you I owe all the disgrace and all the sorrow that I
have endured. Not contented with my ruin, you deprived me of my
brother.'—'Come, Lydia, this is absurd,' he cried. 'In the first place, a
young female who meets a gentleman and walks with him in Parks
or elsewhere, must not expect to escape the usual consequences.
Secondly, your brother challenged me, like a rash and headstrong
young fellow as he was: I sent him due warning by my second that I
was certain to shoot him; but he would not take good advice, and I
did shoot him.'—'And had you no regard for me at that moment?' I
asked.—'Egad!' he replied, 'I only thought of myself. I fancied that if
I did not shoot him, he might perform that good office for me; and
so I was resolved not to give him a second chance.'—'Surely you
cannot be in your senses, my lord,' I exclaimed, 'to talk of so serious
a matter in such a flippant style?'—'Come, let us understand each
other, Lydia,' he said. 'I did not come to such a house as this to
receive a lesson in morals. Do you wish me to remain here with you
until to-morrow?'—'No: a thousand times no,' I replied. 'Your hand is
red with the blood of my poor brother.'—'Very well, Lydia,' he
answered coolly; 'then I will take myself off as quietly as I came. But
for old acquaintance' sake I must do the thing handsomely.'—I heard
his observation, the flippant tone of which made me avert my head
from him in disgust; and I did not therefore see why he lingered for
a few moments. At length he left the room, saying, 'Bye, bye, Liddy;'
and when the door closed behind him, he began to hum an opera-
tune, as he descended the stairs.
"Scarcely could he have had time to gain the street door, when
Mrs. Harpy bounded into my room, exclaiming, 'Well, my dear, you
have behaved very well, for his lordship went away in an excellent
humour. What did he give you?'—'Give me!' I repeated, surveying
that horrible woman with mingled indignation and terror.—'By Jove,
he's a lord in name and nature both!' ejaculated Mrs. Harpy, as her
eyes caught sight of a bank-note which lay upon the table. 'Twenty
pounds, as I'm a living woman!' and she clutched the object of her
delighted avarice.—'Hold, madam!' I exclaimed. 'Not one farthing of
that money will I retain! The man who gave it killed my brother!'—'I
don't care who he's killed, or who he means to kill,' answered the
old woman, 'But here's his money; and that I intend to keep.'—'You
keep it!' I cried.—'Yes; who else? What an ungrateful hussy you
must be, after I took you out of the street! This room and your
board will cost you a guinea a-day. Then your clothes, washing, and
other things are all extra. So I'll keep nineteen pound fifteen shillings
on account; and you shall have a crown for pocket money. If that is
not generous, I don't know what is; but I like to do the thing what's
right.'—With these words she threw five shillings on the table, and
walked off with the twenty pound note.
"This unexpected interview with Lord Dunstable and its result
stamped my degradation, and made me reckless. He had seen me in
a brothel; and in the excitement of our meeting, I had not explained
to him how I became an inmate of that house. Then he left behind
him a sum of money; and, as I was unable to restore it to him with
an indignant refusal of any succour at his hands, he would naturally
conceive that I availed myself of his bounty. My pride was wounded
to such an irreparable degree, that I felt, if you can understand me,
a total unwillingness to endeavour to maintain it any longer. I was
spirit-crushed. I fancied that it was no use to contend any more
against my fate. I considered myself to be now so lost and degraded
in the estimation of that one man whom I had loved, that I had
nothing else in the world to induce me to study character,
reputation, or pride. I accordingly abandoned myself to what I firmly
believed to be my destiny; and, seating myself at the table, I poured
out a glass of champagne. For a moment I sighed as I remembered
that it was champagne that had led to my ruin in the first instance:—
then I laughed at what I called 'my folly,' and emptied the glass. The
wine cheered me, but, at the same time, confirmed me in that
recklessness which had succeeded the first feeling of utter and
irredeemable degradation. I drank another glass: the last spark of
virtuous aspiration was then extinguished in my bosom. The other
young ladies suddenly made their appearance: I received them with
open arms;—we sate down to drink and chat;—I was put to bed in a
disgusting state of intoxication; and on the following morning I
awoke—reconciled to a life of infamy!
"Pardon me, if I dwell for a few minutes upon the characteristics
of those houses of abomination, in one of which I was now located.
Mrs. Harpy was an admirable type of her profession. She was mean
and griping in the extreme when wringing an extra shilling, or even
an extra penny, from her boarders, as we were called; and yet she
was profuse and liberal in supplying us with costly wine. If we
complained of having to eat cold meat two days running, she would
storm, and declare that we lived too well as it was;—but she would
think nothing of giving us a bottle of champagne, which could not
have cost her less than eight or ten shillings, after dinner. She took
from us every farthing that we received, and invariably made us out
her debtors, although she never showed us any accounts. To give
you an idea of her way of managing, I will relate a little anecdote.
One Saturday afternoon, Matilda (whom I have before mentioned)
asked her for a sovereign; adding, 'You know I have given you
altogether thirteen guineas this week.'—'Thirteen guineas!' screamed
the old woman: 'I'll take my Bible oath it was only twelve.'—'Well,
call it even twelve, if you like,' said the young female: 'you can well
spare me a sovereign.'—'Lord bless the girl!' cried Mrs. Harpy. 'Why,
there's seven guineas for your board and lodging; two guineas for
your washing; that's ten; a guinea for pocket money; and a guinea
for letters and needles and thread; that makes up the twelve, or else
I never went to school to learn compound addition.'—'And
multiplication too,' said Matilda. 'Why, I had but one letter all the
week, and that was paid.'—'Well, my dear,' answered Mrs. Harpy,
'we will ask the postman. Come! I'll stand another bottle of
champagne now, and you shall have an extra sovereign for yourself
next Saturday, if you're lucky in the meantime.'
"We were complete slaves to this Mrs. Harpy. She had got a note-
of-hand for twenty pounds from each of us; and if any one even so
much as hinted at leaving her, she immediately threatened to wreak
her vengeance by means of the sheriffs' officer. She seldom allowed
us to go out to take any exercise, for fear we should decamp
altogether; but every now and then we would all go together to
Gravesend or Richmond by the steam-boats, or else to Copenhagen
House, in the summer time, and to some minor theatre in the winter.
Oh! the misery of that existence! We were slaves to an old wretch
who was enriching herself at our expense, whilst we had not an
opportunity of hoarding a single guinea against any sudden
necessity or misfortune. Then, what atrocious proceedings were
frequently enacted in that house! Hard by lived three or four idle
fellows, who dressed flashily, spent a great deal of money, and yet
had no visible employment or resources. Those ruffians were the
blinks, or bullies, belonging to Mrs. Harpy's establishment. Their
tricks were manifold. For instance, they would pick up, at a tavern,
coach-office, the theatre, or other public place, some country
gentleman, or even a clergyman, whom they would ply with liquor,
and then induce to accompany them to 'their aunt's,' where they
would meet 'some delightful girls.' Of course this was Mrs. Harpy's
establishment. The respectable country gentleman, or clergyman,
was plied with more liquor; and, if he would not drink fast enough,
his wine was drugged for him. When he awoke in the morning, he
would find himself in bed with one of the 'delightful girls.' Presently,
one of the bullies would rush into the room, declare that the
gentleman had debauched 'his cousin,' and threaten an exposure.
Then the poor victim was glad to compromise the business by
paying a considerable sum, in order to hush up the matter at once.
"Sometimes, the bullies would attempt a similar scheme of
extortion in reference to individuals who came voluntarily to the
house; and if the latter resisted the exorbitant demands made upon
them, they were not unfrequently maltreated in a most shameful
manner. It often happened that a gentleman would become a
regular visitor to the house, if he took a fancy to one particular
boarder: in such a case he probably adopted a false name, and took
every precaution to avoid discovery as to who he was. The girl
whom he visited, was then directed to pump him; and if she failed to
elicit the desired particulars, one of the bullies was instructed to
watch and dog him when he left the house. By these means, his real
name, residence, position, and circumstances, were speedily
ascertained. If he were moving in a very respectable sphere, was
married, or had any particular motives to induce him to keep his
intrigue secret, he was the very kind of person who suited Mrs.
Harpy and her bullies. The next time he visited the house, he would
be surrounded by those ruffians, menaced with exposure, and forced
to pay a considerable sum of money to purchase silence. But the evil
did not terminate there. From that time forth, the unfortunate
gentleman would be periodically beset by his persecutors; and fresh
extortions would be effected to renew the pledge of secrecy on their
part. Married men, moving in respectable spheres, have been driven
to suicide by this atrocious system! Many a time have I read, in the
newspapers, instances of self-destruction on the part of gentlemen
whose pecuniary, social, or domestic circumstances afforded not the
least appearance of any possible motive for such a deed;—and then
I have thought within myself that those poor victims had been
hunted to death by extortioners of the class which I have described!
The man who has a character to lose, or who has the peace of his
family to consider, knows not how fearfully both are compromised,
both endangered, when he so far forgets himself as to set foot in a
house of infamy. He may imagine that his secret never can transpire
—that neither his family nor friends can, by any possible means,
ever discover that he has thus erred;—but, if he be an individual,
who, by his wealth and social position, appears worth the trouble of
looking after, he will most assuredly find himself a prey to the vilest
of extortioners. His happiness will be undermined and destroyed; he
will live in constant dread of exposure: and deeply—deeply will he
rue the day that he ever set foot in a brothel!
"The most bare-faced robberies are practised in even what are
called 'the respectable dress-houses.' A gentleman, wearing a
handsome watch and chain, is pretty certain to have it stolen from
him; and when he remonstrates, he is perhaps met with a counter-
accusation of having given a bad sovereign in payment for
champagne, on the preceding evening. On one occasion, a young
gentleman who was so plundered, and so accused, carried the
business to the Marlborough Street Police-Office. Mrs. Harpy
attended, denied the robbery in the most indignant manner, and
persisted in the accusation relative to the base sovereign. The
proceedings took such a turn that the young gentleman was
searched; and in his pockets were found other counterfeit
sovereigns, exactly resembling the one produced by Mrs. Harpy.
Then Mrs. Harpy sent for her wine-merchant, her butcher, and her
baker, who were all her near neighbours: and those tradesmen
declared that Mrs. Harpy kept a most respectable boarding-house,
and that she was a lady of good connexions and undoubted
integrity. The magistrate then appealed to the policeman within
whose beat Gerrard Street was included; and as he received five
guineas a year from Mrs. Harpy for shutting his eyes, it was not
likely that he would open them on that occasion. He fully
corroborated the evidence of the wine-merchant, butcher, and
baker; and the young gentleman was committed for trial for passing
base money. Mrs. Harpy's story was that he had presented himself
on the preceding evening at her house, and arranged to become a
boarder in her establishment; that he obtained from her the change
for the bad sovereign; and that, when accused of the act, he had
turned round with a counter-charge relative to his watch. The
magistrate declared that there was no doubt of Mrs. Harpy's perfect
respectability, and commented severely on the 'infamous behaviour
of the prisoner, in trumping up so vile an accusation, as a means of
releasing himself from the odium of the charge laid against him.'
This young man belonged to a highly respectable family; and he had
given a fictitious name in answer to the magistrate's question, for he
had only been married six months, and was naturally anxious to
conceal his visit to a brothel from the knowledge of his friends. But
when he was committed for trial, he was forced to send for them,
confess his indiscretion, and implore them to save him from the
ignominy of exposure in a court of justice. A compromise with Mrs.
Harpy was accordingly effected: she paid fifty pounds in forfeit of
her recognizances to prosecute: and she received two hundred to
abstain from farther proceedings! I need scarcely say that the young
gentleman really had been plundered of his watch, and that the
entire business of the counterfeit money had been arranged to ruin
him. Again I declare that no one knows the woeful risks he incurs
when he sets foot in a house of ill-fame. That one false step may
embitter the remainder of his days!
"Some weeks elapsed ere I was completely aware of the infamies
which were perpetrated in Mrs. Harpy's den; and then I resolved to
leave the place, whatever might subsequently become of me. At
length an opportunity served; and one evening, with only a small
parcel of necessaries under my arm, and a few shillings in my purse,
I slipped out of that scene of iniquities. I cannot enter into further
details; suffice it to say, from that moment commenced an existence
of fearful vicissitudes,—starvation one day, luxury the next,—the
most abrupt descents into the lowest abyss of destitution, and the
most sudden elevations to comfort, though still a career of infamy,—
wanderings for many, many nights together, without knowing where
to lay my head, and then a lodging and a good bed! Oh! it was
horrible, that precariousness of life to which I was doomed!
"How often did I reflect upon the times of my innocence! Now and
then I saw well-known names mentioned in the newspapers. The
consecutive and rapid promotions of Lord Dunstable and
Cholmondeley were not unnoticed by me. The presentation of the
Honourable Adeline Enfield to court was an incident which affected
me deeply; for it naturally led me to compare her elevated position
with my degraded and wretched state. But one event, which was
recorded in the newspapers, gave me, I must confess, some
satisfaction: this was the bankruptcy of Mrs. Lambkin and her
committal to Newgate for having fraudulently disposed of her
property. I afterwards learnt that she died miserably in that gaol.
"But my own vicissitudes continued! Oh! let those who are prone
to turn away from the unfortunate woman with disgust and
abhorrence, rather exercise a feeling of sympathy in her behalf. She
does not drag her weary frame nightly along the pavement, through
choice, but from necessity. In all weathers must she ply her
miserable trade—or starve. Then to what indignities is she
subjected! Every drunken ruffian considers himself justified in ill-
using her: every brutal fellow jostles against her, and addresses her
in terms of insult. Do they think that, because she is compelled to
ply her hideous trade, she has no feelings? But it is chiefly from the
young men who rove about the streets at night, smoking cigars,
wearing pea-coats, and carrying sticks, that the unfortunate woman
is doomed to receive the deepest indignity:—yes, from those who
ought to have more chivalry in their dispositions! There is one base
extortion to which the unfortunate woman is subjected, and which I
must mention. I allude to the necessity of feeing the policeman
belonging to that beat where the unhappy creature walks. The
miserable wretch who deviated from this practice, either through
inability or unwillingness, would never have a moment's peace. The
moment she was accosted in the street by a gentleman, the officer
would come up and order her brutally to move on; and perhaps he
would add violence to harsh words. Then, on the slightest pretence
—and often without any at all—the miserable woman is dragged off
to the station-house, charged with creating a disturbance, and taken
next morning before the magistrate. In vain may she protest her
innocence of the offence charged against her: in vain may she
denounce the vindictive motives of the officer. The word of one
policeman is deemed worth the oaths of ten thousand degraded
females; and the accused is sentenced to Bridewell accordingly. No
one can conceive the amount of the wrongs inflicted by the police
upon the most miserable class of women!
"I could enter into details respecting the lives of unfortunate
females, which would inspire you with horror—and yet with deep
compassion. But I have already dwelt too long on a subject which
should never be mentioned without caution to the pure-minded
woman. In reference to myself, I need only add that having passed
through all the terrible phases of a career of infamy,—each day
beholding me more degraded, and sinking lower and lower amongst
the low,—I was reduced to a condition when beggary appeared the
only resource left From this appalling condition your goodness has
relieved me; and God alone must reward you—I never can!"
CHAPTER CLXXVIII.

THE TAVERN AT FRIULI.

Through the broad meadows, the waving woods, and the delicious
valleys which lie on the northern side of the Ferretti, in the State of
Castelcicala, two foot-travellers pursued their way.
Lovely flowed the river amidst the meads that were clothed in the
country's everlasting green.
Busy hamlets, neat farm-houses, and the chateaux of nobles or
wealthy gentlemen, varied the appearance of the magnificent
landscape.
Although it was the middle of November, the climate was as mild
and genial as that of September in the British Islands: the vines had
not been entirely stripped of their luscious fruit; and the citrons, so
plentiful that they were but little prized by the inhabitants, grew wild
by the road-side.
Here groups of mighty chesnut-trees afforded a delicious shade to
the way-worn traveller: there the tapering spire of a village church,
or the white walls and slated roof of some lordly country-seat,
appeared above the verdant mulberry-groves.
Nevertheless, the woodlands of Castelcicala were not
characterised by that gloominess of foliage which invests the English
and German forests with such awful solemnity; for the leaves were
of a brighter green, and the density of their shade was relieved by
the luxuriousness of the botany that spread its rich and varied
colours over the surface of the land.
The banks of the Ferretti yielded an immense profusion of
aromatic herbs, which imparted a delicious perfume and, at the
same time, a freshness to the air.
Much as those two travellers had been accustomed to admire the
loveliness of their own native England, they could not avoid
exclamations of joy and surprise as they pursued their way amidst
the fertile plains of Castelcicala.
We need scarcely inform our readers that those travellers were
Richard Markham and his faithful Morcar.
Our hero, dressed in a neat but modest garb, and carrying a
portfolio of drawing materials under his arm, journeyed along a little
in advance of his attendant, who bore a small valise of necessaries.
In his pocket-book Richard had secured the two passports, for
himself and follower, which the interest of Mario Bazzano had
obtained, and which were made out in fictitious names.
Fastened to a riband round his neck, and carefully concealed
beneath his raiment, was a small morocco leather case, containing
the sealed letter left him, with such mysterious instructions, by
Thomas Armstrong.
The well-filled purse which the generosity of the Grand Duchess
had supplied, and a map of the Duchy, completed the stock of
materials with which the travellers had deemed it fit to furnish
themselves.
Their way now lay, according to the advice which Richard had
received from the Grand Duchess, towards Friuli: thence it was his
intention to strike off abruptly in a longitudinal direction, and,
passing between Dandolo and Lipari, proceed straight toward the
Neapolitan frontier.
On the fourth evening the two travellers arrived at Friuli, having
walked upon an average thirty miles each day, and slept at night in
some cottage or farm-house.
They did not, however, penetrate into the fine and spacious town
which they had now reached; but stopped at a small tavern in the
suburbs. There they ordered supper, which was served up to them in
the public room, as Richard did not think it prudent to excite notice
by having a private apartment.
Several other persons were sitting in the public room, busily
engaged in imbibing the various liquors suited to their respective
palates, and discussing, with great solemnity, the political aspect of
the State.
By their conversation Markham judged that they must be the small
tradesmen of the suburbs of the town, as they all seemed well
acquainted with each other, and spoke as if they were in the habit of
meeting at that tavern every evening after the bustle and cares of
the day's business.
"Are you certain, neighbour," said one worthy burgher, addressing
himself to another, "that the proclamation will be made to-morrow
morning?"
"I believe, gentlemen," answered the individual thus appealed to,
"you are all aware that my wife's father is Adjunct to the Mayor of
Friuli; and the title of Adjunct is pretty nearly synonymous with that
of Deputy. Well, then, gentlemen, my father-in-law being, you
perceive, as good as Deputy-Mayor," continued the speaker, thinking
that his prosiness would add to his importance, "he cannot fail to be
in the mayor's secrets. That once granted, gentlemen, you can easily
estimate the value of my authority for the tidings I reported to you
just now. You may therefore rely on it, that the proclamation placing
the entire province of Montecuculi under martial law, will be read in
Friuli, as well as in all the other towns, villages, and hamlets of the
aforesaid province, to-morrow morning, at nine o'clock."
"Then I suppose the whole Duchy will be placed under martial
law?" observed another member of the party.
"No doubt of it," said the second speaker. "The worshipful mayor
hinted as much to the not less worshipful adjunct, or deputy, this
afternoon."
"The province of Abrantani has been for some time in an
exceptional state, you know," said the individual who had first
spoken; "and by all accounts, we had much better be under the yoke
of the Austrians at once—just like the northern provinces of Italy. I
tell you what," added the individual who was now addressing his
companions,—"I tell you what," he repeated, sinking his voice
almost to a whisper, "there is not a man in Castelcicala who will not
be ready to draw his sword against this most odious tyranny."
"Hush! hush!" exclaimed the relative of the civic authority, as he
glanced towards Richard Markham and Morcar; "we do not know
who may overhear us, as the adjunct often observes to me."
"The gentleman is an artist, and looks like a foreigner, too," said
the individual whose freedom of speech had provoked this
remonstrance: "he is not likely to meddle with our political business."
"Gentlemen," said Richard, "it is true that I understand your
language, although I speak it imperfectly; but if you apprehend that
I should make any improper use of the remarks which fall from you,
I will at once retire to a private room."
"Well spoken!" ejaculated one of the company. "No, sir—you shall
not leave the room on our account. If I mistake not, you must be an
Englishman or a Frenchman; and I like both those nations—for they
know what true freedom is, while we are slaves,—abject slaves."
"Yes,—and I admire the English, too," cried the person who had
before spoken with so little reserve. "Have they not given an asylum
to that excellent Prince who is only exiled because he was the
people's friend—because he wished to obtain for us a Constitution
that would give us Houses of Parliament or Chambers, to be the
bulwark of our liberties? Is not our Grand Duchess an
Englishwoman? and has she not exerted herself to the utmost to
mitigate the severity of Angelo III? That is no secret. And, when I
think of it, I remember hearing at Ossore (where I was, you know, a
few days ago,) that it was a young Englishman who rallied the
Constitutionalists when they were flying, after the fall of General
Grachia."
"What became of him?" asked one of the company.
"It is known that he was taken prisoner," was the reply; "but as he
disappeared almost immediately afterwards, it is supposed that he
was hurried off without delay to one of the fortresses in the interior
—Pinalla or Estella, for instance. Poor young fellow—I wish he had
had better luck! But, as I was saying, you see we have good reason
to admire the English—God bless them!"
"Amen!" exclaimed several voices.
The emotions of our hero, while this discourse was progressing,
may be more readily imagined than explained: but prudence on his
own account, and obedience to the advice of the Grand Duchess,
sealed his lips.
Morcar continued to eat and drink without excitement, because
the conversation passing around was totally unintelligible to him.
The relative of the mayor's adjunct was dilating pompously on the
duties of a sovereign, when a post-chaise drove furiously up to the
door of the tavern.
All was immediately bustle and confusion.
"Horses! four horses wanted!" shouted a voice in the passage.
Then commenced the rattling of harness,—the running hither and
thither of ostlers,—and the usual calling and bawling which
characterise such occasions.
All the inmates of the coffee-room, with the exception of Markham
and the gipsy, rushed out to stare at the equipage.
Scarcely was the room thus left comparatively empty, when a tall
man, wrapped in an ample travelling cloak, entered hastily, followed
by the landlord.
"Here—we have not a moment to lose—give me change for this
bank-note," cried the traveller.
"Yes, sir," said the host, and hurried from the room.
"Signor Bazzano," whispered our hero, who had started from his
seat at the sound of the traveller's voice.
"What! Signor Markham!" said the young aide-de-camp, shaking
him kindly by the hand. "This is indeed most fortunate! But I have
not a moment to spare. Listen! terrible events have taken place at
Montoni: you are in danger. You must separate from your attendant,
and each gain the Neapolitan frontier by a separate route. Follow my
advice, my dear Markham,—as you value your life!"
At that moment the host re-appeared with the gold and silver in
change for the note; and Bazzano, having hastily consigned the
money to his pocket, hurried from the room,—but not before he had
darted a significant glance upon our hero.
In a few moments the post-chaise drove rapidly away.
Richard returned to his seat in a cruel state of uncertainty, doubt,
and suspense.
What could that precipitate journey mean? was Bazzano the sole
occupant of the carriage? what terrible events could have occurred
at Montoni? and what was that fearful peril which would oblige him
to adopt so painful a precaution as to separate from his companion?
Richard was at a total loss how to solve these queries which
naturally suggested themselves to his mind.
While he was yet pondering on the singularity of the incidents
which had occurred, all within the space of three or four minutes,
the company poured back again to the coffee-room.
"Something mysterious there," said one.
"Yes—a post-chaise with the blinds drawn down," observed
another.
"Four horses—and travelling like wild-fire," exclaimed a third. "The
tall man in the cloak, who rode outside, came into this room. What
did he want, sir?" demanded the speaker, turning abruptly towards
Markham; "for I believe you did not leave the room."
"He obtained change from the landlord for a bank-note, sir,"
answered our hero laconically.
"Oh! that was all—eh? Well—the thing still looks odd—particularly
in such troubled times as these. Did anybody hear the orders given
to the postilions?"
"The tall man in the cloak said in a loud voice, 'The road towards
Dandolo, my boys!'" observed another of the company.
Richard smiled imperceptibly; for he thought within himself, "Then
it is precisely because Bazzano said in a loud tone, 'Towards
Dandolo,' that the travellers are going in another direction."
The company continued to debate, as all gossips will, upon the
incident which had just occurred; and Richard determined to lose no
more time ere he explained to Morcar, who had of course recognised
the young aide-de-camp, the nature of the warning he had received
from this individual.
He according bade the assembled guests "Good night," and left
the room, followed by Morcar.
At his request, the landlord conducted them to a double-bedded
room; and the moment the host had retired, Richard communicated
to the gipsy all that Bazzano had said to him.
"There is but one course to pursue, sir," exclaimed Morcar.
"Which is that?" asked Richard.
"To follow the Castelcicalan officer's advice," returned Morcar. "He
saved your life—he restored me to your service—and he is incapable
of deceiving us. He is your friend, sir—and you must obey him."
"But, my poor Morcar," said Richard, "I cannot part with you. I
have lured you away from your family and native land, to lead you
into these difficulties; and I would sooner die than abandon you in a
strange country, with even the language of which you are
unacquainted."
"My dear, good master," exclaimed the gipsy, his eyes dimmed
with tears, "it will go to my heart to leave you; but if your life is in
danger, I shall not hesitate a moment. Besides, the same peril that
would overtake one, would crush both, were we together when it
came; and it is folly for either of us to run idle risks in such a strait.
No—let us follow the advice of your friend."
"Again, I say, Morcar, that I cannot part with you. Were any thing
fatal to happen to you, I should never forgive myself. No," continued
Richard, "you shall remain with me. If danger come, it is only I who
will suffer—for it seems that it is only my life which is in danger. And
this is probable enough."
"Ah! sir—I am not afraid of myself," exclaimed Morcar: "I would
lay down my life to serve you! But I am convinced that you will only
attract unpleasant attention to yourself, if you travel with a follower:
one person can slip unperceived through so many perilous places,
where two together would be suspected. Besides, sir, I shall not be
quite so badly off in this strange country, as you suppose."
"How so, Morcar?" demanded Richard, surveying him with
astonishment.
"There are Zingarees in this land as well as elsewhere," replied
Morcar; "and amongst them I shall be safe."
"On that consideration alone," exclaimed Richard, struck by the
truth of the observation, and well-pleased at the idea that his faithful
dependant would indeed derive no small benefit, under
circumstances, from the aid of that extensive and mysterious
freemasonry to which he belonged,—"on that consideration alone I
will consent to this separation. At day-break we will rise, and each
take a different route. I will give you the map of Castelcicala, as its
geography has been so well studied by me that I am fully
acquainted with the direction of all the principal towns and cities.
But let us fix a place where we can meet again. Our grand object
must be to gain the city of Naples. On your arrival there, proceed to
the abode of the English Consul, and leave with him the name of the
inn where you put up: if I have reached Naples before you, that
functionary will be enabled to tell you where I am to be found."
"I will strictly follow your instructions, sir," said Morcar.
"And now, my good friend," continued our hero, "I must speak to
you as if I were making my last will and testament; for heaven alone
knows whether I shall ever quit this country alive. You remember the
secret of my affection for a noble lady, which I communicated to you
the night before we landed on the Castelcicalan coast?"
"Not a syllable of what you told me, sir, has been effaced from my
memory," replied Morcar. "You enjoined me that, if any thing fatal
should occur to yourself, and Providence should enable me to return
to England, I was to seek the Princess Isabella, and break to her the
tidings and manner of your death, with the assurance that your last
thoughts were given to her!"
"Such was my request, Morcar," said Richard. "I need now observe
little more than repeat it. Let the one who reaches Naples first wait
for the other fifteen days; and, if he come not by the expiration of
that period, then let him——"
"Surmise the worst," added Morcar, seeing that our hero
hesitated. "Your message to the Princess shall be delivered—if God
ordain that so sad a result ensues. And, on your part, sir—if I come
not to the place of appointment, and you succeed in reaching it——"
"Say no more, my dear friend," interrupted Markham, pressing the
gipsy's hand; "we understand each other!"
And they each dashed away the tears from their eyes.
Richard then divided the contents of his purse into two equal
portions, and presented one to Morcar. The gipsy positively refused
to accept any thing beyond a few pieces of gold; but Markham was
more positive still, and compelled him to assent to the equitable
partition of the large sum which Eliza's bounty had supplied.
They then retired to rest.
At day-break Markham started up; but he looked in vain for
Morcar.
On the table stood a pile of gold: it was the one which our hero
had forced upon the gipsy;—and only two of the pieces had been
taken from the heap.
"Generous man;" cried Markham: "God grant that I may one day
be enabled to reward him for his fidelity and devotion to me!"
Having hastily dressed himself, our hero concealed about his
person the few necessaries that were indispensable, and left the
remainder in his valise.
He then descended to the coffee-room, hurried over a slight
refreshment, and, having settled the account, took his departure,
telling the landlord to keep the valise for him until his return.
But now how lonely, forlorn, and friendless did he feel, as he
hurried away from the inn where he had parted with his faithful
dependant!
CHAPTER CLXXIX.

THE JOURNEY.

Richard Markham struck into the fields, and pursued his way in a
southerly direction.
He avoided even the small hamlets, and kept as much as possible
in the open country.
Being unaware of the precise nature of the danger which menaced
his life,—although of course connecting it with the part which he had
recently played in the invasion,—he feared lest printed descriptions
of his person, with rewards for his apprehension, might be
circulated; and this source of terror induced him to choose the most
secluded paths.
It was long after sunset when he stopped at a small country
public-house, where he determined to rest for the night.
To his great joy the coffee-room was unoccupied by other
travellers; and the landlord appeared a simple, honest kind of half-
farmer, half-publican, who never troubled himself about any one's
business save his own.
A good supper and a bottle of very excellent wine tended to raise
our hero's spirits: and when the meal was concluded, he fell into a
train of meditation on the events of the preceding evening.
A thousand times did he ask himself who could be the occupant of
that chaise which was journeying in such haste? for that there was
some person inside the vehicle, who had urgent reasons for the
utmost circumspection, the fact of the drawn blinds would not
permit him to doubt. Moreover, the young aide-de-camp was
evidently riding outside for the purpose of answering any questions
that might be put, paying the bills, directing the postillions, and in all
respects acting with a view to save the person or persons inside
from the necessity of giving their own orders.
The words—"Terrible events have occurred at Montoni"—were also
fraught with a most menacing and mysterious importance. What
could they mean? whom had these events endangered? Was it
possible that the kindness of the Grand Duchess towards himself had
been detected? And if so, what results could such a discovery have
produced?
While he was thus lost in the most painful conjectures, a
horseman suddenly galloped up to the door of the inn; and in a few
moments the traveller himself entered the coffee-room.
He was a slightly-built, middle-aged man, with a good-humoured
expression of countenance. He was attired in a kind of undress
cavalry uniform, consisting of a foraging-cap with a broad gold band,
a laced jacket, trousers with a red stripe down each leg, and a very
small black leathern knapsack at his back.
"Now, landlord," he exclaimed, as he entered the room, followed
by the individual whom he thus addressed, "some supper at once—
not a moment's unnecessary delay—and see that a fresh horse is
ready in twenty minutes. That is all the rest I can allow myself here."
The landlord bustled about to serve up the best his house could
afford in such haste; and in the meantime the new-comer addressed
himself to our hero.
"Rather chilly this evening, sir," he said.
"And yet you can scarcely feel the cold, considering the pace at
which you appear to ride," returned Richard with a smile.
"Egad! I do not ride so for pleasure, I can assure you," observed
the man. "But I presume that you are travelling in this country for
your amusement," he added: "for I perceive by your accent that you
are not a Castelcicalan, and I can judge your avocation by that
portfolio lying near you."
"You have guessed correctly," answered Richard. "Have you
travelled far to-day?"
"A considerable distance. I am, as perhaps you may know by my
dress, a government courier: and I am the bearer of dispatches from
Montoni to the Captain-General of Montecuculi."
"Any thing new in the capital?" asked Richard, scarcely able to
conceal the anxiety with which he waited for a reply.
"Great news," was the answer. "The Grand Duchess has fled."
"Fled!" ejaculated Markham.
"Yes—left the capital—gone no one knows where, and no one
knows why," continued the courier. "Montoni is in a dreadful
ferment. Martial law was proclaimed there the day before yesterday;
and a tremendous crowd collected in the Palace-square in the
evening. The military were called out, but refused to fire upon the
people. Numerous conflicting reports are in circulation: some say
that the Grand Duke has sent to demand the aid of an Austrian
force. The people attacked the mansion of the Prime Minister; and
the firmness of the Political Prefect alone prevented serious mischief.
In fact, sir," added the courier, sinking his voice to a whisper, "we are
on the eve of great events; and for my part—although I am in the
government employment—I don't think it's treason to say that I
would as soon serve Alberto as Angelo."
At that moment the landlord entered with a tray containing the
courier's supper; and the conversation ceased. Nor had our hero an
opportunity of reviving it; for the courier was too busily engaged
with his knife and fork to utter a word during his meal; and the
moment it was terminated, he wished Markham good night and took
his departure.
Still our hero had gleaned enough to afford him some clue to the
mystery of the post-chaise. The Grand Duchess had fled: the reason
of her flight was not publicly known. Was it not probable that she
was an occupant of the post-chaise which journeyed so swiftly? did
not this idea receive confirmation from the fact that Mario Bazzano
accompanied the vehicle?
Then again occurred the question, had the Grand Duchess
involved herself in difficulty by her generosity towards him? The bare
supposition of such an occurrence was the source of the most
poignant anguish in the breast of Richard Markham.
He retired to rest; but his sleep was uneasy; and he awoke at an
early hour, little refreshed. He was however compelled to pursue his
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge


connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and


personal growth every day!

ebookbell.com

You might also like