Instant Ebook Access, One Click Away – Begin at ebooknice.

com

(Ebook) Pro SQL Server 2022 Wait Statistics: A

Practical Guide to Analyzing Performance in SQL
Server and Azure SQL Database, 3rd Edition by
Thomas LaRock, Enrico van de Laar ISBN
9781484287712, 1484287711
https://ebooknice.com/product/pro-sql-server-2022-wait-
statistics-a-practical-guide-to-analyzing-performance-in-
sql-server-and-azure-sql-database-3rd-edition-47559896

OR CLICK BUTTON

DOWLOAD EBOOK

Get Instant Ebook Downloads – Browse at https://ebooknice.com

 Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

(Ebook) Pro SQL Server 2022 Wait Statistics: A Practical Guide to Analyzing
Performance in SQL Server and Azure SQL Database, 3rd Edition by Thomas LaRock,
Enrico van de Laar ISBN 9781484287705, 1484287703

https://ebooknice.com/product/pro-sql-server-2022-wait-statistics-a-practical-
guide-to-analyzing-performance-in-sql-server-and-azure-sql-database-3rd-
edition-47559882

ebooknice.com

(Ebook) Pro SQL Server 2019 Wait Statistics: A Practical Guide to Analyzing
Performance in SQL Server - Second Edition by Enrico van de Laar ISBN 9781484249161,
148424916X

https://ebooknice.com/product/pro-sql-server-2019-wait-statistics-a-practical-
guide-to-analyzing-performance-in-sql-server-second-edition-52952162

ebooknice.com

(Ebook) SQL Server Execution Plans. For SQL Server 2008 through to 2017 and Azure
SQL Database by Grant Fritchey ISBN 9781910035221, 191003522X

https://ebooknice.com/product/sql-server-execution-plans-for-sql-
server-2008-through-to-2017-and-azure-sql-database-11063662

ebooknice.com

(Ebook) Expert Performance Indexing in Azure SQL and SQL Server 2022 by Edward
Pollack, Jason Strate ISBN 9781484292150, 1484292154

https://ebooknice.com/product/expert-performance-indexing-in-azure-sql-and-sql-
server-2022-48211984

ebooknice.com
(Ebook) Practical Database Auditing for Microsoft SQL Server and Azure SQL:
Troubleshooting, Regulatory Compliance, and Governance by Josephine Bush ISBN
9781484286333, 1484286332

https://ebooknice.com/product/practical-database-auditing-for-microsoft-sql-
server-and-azure-sql-troubleshooting-regulatory-compliance-and-
governance-46188488

ebooknice.com

(Ebook) SQL Server Big Data Clusters : Data Virtualization, Data Lake, and AI
Platform by Benjamin Weissman, Enrico van de Laar ISBN 9781484259849, 9781484259856,
148425984X, 1484259858

https://ebooknice.com/product/sql-server-big-data-clusters-data-virtualization-
data-lake-and-ai-platform-22474702

ebooknice.com

(Ebook) Azure SQL Revealed: A Guide to the Cloud for SQL Server Professionals by Bob
Ward ISBN 9781484259313, 1484259319

https://ebooknice.com/product/azure-sql-revealed-a-guide-to-the-cloud-for-sql-
server-professionals-12286544

ebooknice.com

(Ebook) Azure SQL Revealed: A Guide to the Cloud for SQL Server Professionals by Bob
Ward ISBN 9781484259306, 1484259300

https://ebooknice.com/product/azure-sql-revealed-a-guide-to-the-cloud-for-sql-
server-professionals-12286546

ebooknice.com

(Ebook) Healthy SQL: A Comprehensive Guide to Healthy SQL Server Performance by

Robert Pearl ISBN 9781430267737, 1430267739

https://ebooknice.com/product/healthy-sql-a-comprehensive-guide-to-healthy-sql-
server-performance-5108072

ebooknice.com
Thomas LaRock and Enrico van de Laar

Pro SQL Server 2022 Wait Statistics

A Practical Guide to Analyzing Performance in SQL
Server and Azure SQL Database
3rd ed.
Thomas LaRock
East Longmeadow, MA, USA

Enrico van de Laar

Drachten, The Netherlands

ISBN 978-1-4842-8770-5 e-ISBN 978-1-4842-8771-2

https://doi.org/10.1007/978-1-4842-8771-2

© Thomas LaRock, Enrico van de Laar 2015, 2019, 2023

This work is subject to copyright. All rights are solely and exclusively
licensed by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in
any other physical way, and transmission or information storage and
retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks,

service marks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general
use.

The publisher, the authors, and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the
material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Apress imprint is published by the registered company APress
Media, LLC, part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY
10004, U.S.A.
This book is dedicated to all the accidental database administrators,
developers, and anyone who has written a database query and wondered
“what the hell is taking so long?”
Also, for Roy, Moss, and anyone else who decided to turn it off and back on
again.
Introduction
“Write the book you wish someone else would have written and handed
to you when you were starting as a DBA.”
Those words were spoken to me by my friend and mentor, Kevin
Kline, roughly 13 years ago. At the time I was writing my first book,
DBA Survivor, and I asked Kevin for advice on how to approach the
project. His answer gave me clarity, and I’ve used the same approach for
this book you are now reading.
At the time I started as a junior DBA, Tom Davidson’s well-known
SQL Server 2005 Waits and Queues whitepaper was years away from
publication. What I knew about waits I would find using DBCC
statements against (the artist formally known as) Sybase ASE and SQL
Server 2000 instances.
In other words, I didn’t know much.
With the release of SQL Server 2005 and the publication of the
Davidson whitepaper, wait statistics became a viable tuning
methodology. Administrators and developers could now use waits and
queues to understand exactly why a query was running longer than
expected. Overnight our team transitioned from reacting to query
performance issues to being proactive in understanding which
resources the overall database workload needed most.
Every request sent to a database server has the same constraints:
memory, processing, disk, network, and locking/blocking. It doesn’t
matter if you want to rely solely on execution plans for query tuning;
the physical and logical constraints for the query remain the same: they
are just presented differently in an execution plan. The waits and
queue tuning methodology reduces the complexity and time
necessary for query performance tuning by an order of magnitude.
Once you understand how the database engine processes requests,
how waits happen, and how to track them, you are well on your way to
being an expert in query performance tuning.
And that’s the goal of this book. When you are done reading, I want
you to have all the skills necessary to be an expert in query
performance tuning. That’s the book I wish someone would have
written and handed to me when I was first starting as a DBA.
 To reach the goal, this book has been split into two unequal parts.
Part I, “Foundations of Wait Statistics Analysis,” provides details on how
the database engine processes a query (officially called a request, which
is sent by a session, after a connection to the instance is established)
followed by information on how to query wait statistics information
through various SQL Server dynamic management views (DMVs). Part I
finishes with an overview of the Query Store feature and guidance on
how to create and gather metrics to build your own baselines.
Part II, “Wait Types,” dives into specific waits, the causes, some
examples, and possible resolutions. The chapters are divided by wait
categories, which is a bit tricky as some waits (such as PAGEIOLATCH)
have overlap between more than one possible constraint (memory and
disk). Therefore, the chapters break down specific waits into categories
by CPU, IO, backups, locks, latches, high-availability and disaster-
recovery, preemptive, background and miscellaneous, and In-Memory
OLTP.
Yes, waits for background and miscellaneous are included, despite
their being benign for query performance. It’s important for you to
know why (and when) these waits happen and when they are safe to
ignore (they usually are, but not always).
One thing to note, the examples in this book use a database named
GalacticWorks. This is a modified version of AdventureWorks I use for a
variety of demos when teaching my classes. The examples in the book
will work with AdventureWorks, so don’t panic about not having
GalacticWorks; you’ll be fine with most versions of AdventureWorks.
When you finish this book, I want you to have the confidence to
tackle any query performance tuning problem. You’ll have the details,
information, and knowledge necessary to be an expert. And maybe
soon enough, you’ll be teaching others and maybe someday write your
own book, too.
Any source code or other supplementary material referenced by the
author in this book is available to readers on GitHub
(https://github.com/Apress). For more detailed information, please
visit http://www.apress.com/source-code.
Acknowledgments
There are many people to acknowledge and thank for helping me with
this book. I’ll do my best to include as many as I can, but please don’t be
offended if I forget you; it’s not on purpose.
I’ll start with my wife, Suzanne, for her patience as I spent many off-
hours completing this book. Oh, and for all the time I’ve spent away
from home for the past 15 years.
Thanks to Bob Ward of Microsoft for helping uncover new SQL 2022
features, as well as your willingness to share your knowledge on SQL
Server for the past 25 years. And thanks for your time in Barcelona in
2006 when you inspired me to want to learn more about SQL Server.
Thanks to my partner in #TeamData, Karen Lopez, for your help,
support, and friendship for many years. I became a better technical
writer and presenter by learning from you.
Thanks to Kevin Kline for your guidance and support as I left my
career as a production DBA to become a Technical Advocate and for
providing an example of what proper community leadership looks like.
To Buck Woody, for your support in helping me understand my
strengths and your guidance in career opportunities through the years.
To Craig and Vinny, for giving me the opportunity to fail as a DBA,
and to Frank and Lori for not letting it happen as much as it should
have.
To Rie, Betsy, Rochelle, and everyone on the Microsoft Community
team for awarding me the Microsoft MVP status all these years.
Finally, to Jonathan, for thinking of me when it came time to update
this book for SQL 2022. Thank you for the opportunity to write another
book for you, 12 years later.
Table of Contents
Part I: Foundations of Wait Statistics Analysis
Chapter 1:​Wait Statistics Internals
A Brief History of Wait Statistics
The SQLOS
Schedulers, Tasks, and Worker Threads
Sessions
Requests
Tasks
Worker Threads
Schedulers
Putting It All Together
Wait Statistics
Summary
Chapter 2:​Querying SQL Server Wait Statistics
sys.​dm_​os_​wait_​stats
sys.​dm_​os_​waiting_​tasks
Understanding sys.​dm_​os_​waiting_​tasks
Querying sys.​dm_​os_​waiting_​tasks
sys.​dm_​exec_​requests
Understanding sys.​dm_​exec_​requests
Querying sys.​dm_​exec_​requests
sys.​dm_​exec_​session_​wait_​stats
Combining DMVs to Detect Waits Happening Now
Viewing Wait Statistics Using Perfmon
Capturing Wait Statistics Using Extended Events
 Capture Wait Statistics Information for a Specific Query
Analyzing Wait Statistics on a Per-Query Basis Using Execution
Plans
Summary
Chapter 3:​The Query Store
What Is the Query Store?​
Enabling the Query Store
Enable the Query Store Using SSMS
Enable the Query Store Using T-SQL
Query Store Architecture
How Wait Statistics Are Processed in the Query Store
Accessing Wait Statistics Through Query Store Reports
Accessing Wait Statistics Through Query Store DMVs
Summary
Chapter 4:​Building a Solid Baseline
What Are Baselines?​
Visualizing Your Baselines
Baseline Types and Statistics
Baseline Pitfalls
Too Much Information
Know Your Metrics
Find the Big Measurement Changes
Use Fixed Intervals
Building a Baseline for Wait Statistics Analysis
Reset Capture Method
Delta Capture Method
 Using SQL Server Agent to Schedule Measurements
Wait Statistics Baseline Analysis
Summary
Part II: Wait Types
Chapter 5:​CPU-Related Wait Types
CXPACKET
What Is the CXPACKET Wait Type?​
Lowering CXPACKET Wait Time by Tuning the Parallelism
Configuration Options
Lowering CXPACKET Wait Time by Resolving Skewed
Workloads
Introduction of the CXCONSUMER Wait Type
CXPACKET Summary
SOS_​SCHEDULER_​YIELD
What Is the SOS_​SCHEDULER_​YIELD Wait Type?​
Lowering SOS_​SCHEDULER_​YIELD Waits
SOS_​SCHEDULER_​YIELD Summary
THREADPOOL
What Is the THREADPOOL Wait Type?​
THREADPOOL Example
Gaining Access to Our SQL Server During THREADPOOL
Waits
Lowering THREADPOOL Waits Caused by Parallelism
Lowering THREADPOOL Waits Caused by User Connections
THREADPOOL Summary
Chapter 6:​IO-Related Wait Types
ASYNC_​IO_​COMPLETION
 What Is the ASYNC_​IO_​COMPLETION Wait Type?​
ASYNC_​IO_​COMPLETION Example
Lowering ASYNC_​IO_​COMPLETION Waits
ASYNC_​IO_​COMPLETION Summary
ASYNC_​NETWORK_​IO
What Is the ASYNC_​NETWORK_​IO Wait Type?​
ASYNC_​NETWORK_​IO Example
Lowering ASYNC_​NETWORK_​IO Waits
ASYNC_​NETWORK_​IO Summary
CMEMTHREAD
What Is the CMEMTHREAD Wait Type?​
Lowering CMEMTHREAD Waits
CMEMTHREAD Summary
IO_​COMPLETION
What Is the IO_​COMPLETION Wait Type?​
IO_​COMPLETION Example
Lowering IO_​COMPLETION Waits
IO_​COMPLETION Summary
LOGBUFFER and WRITELOG
What Are the LOGBUFFER and WRITELOG Wait Types?​
LOGBUFFER and WRITELOG Example
Lowering LOGBUFFER and WRITELOG Waits
LOGBUFFER and WRITELOG Summary
RESOURCE_​SEMAPHORE
What Is the RESOURCE_​SEMAPHORE Wait Type?​
RESOURCE_​SEMAPHORE Example
 Lowering RESOURCE_​SEMAPHORE Waits
RESOURCE_​SEMAPHORE Summary
RESOURCE_​SEMAPHORE_​QUERY_​COMPILE
What Is the RESOURCE_​SEMAPHORE_​QUERY_​COMPILE Wait
Type?​
RESOURCE_​SEMAPHORE_​QUERY_​COMPILE Example
Lowering RESOURCE_​SEMAPHORE_​QUERY_​COMPILE Waits
RESOURCE_​SEMAPHORE_​QUERY_​COMPILE Summary
SLEEP_​BPOOL_​FLUSH
What Is the SLEEP_​BPOOL_​FLUSH Wait Type?​
SLEEP_​BPOOL_​FLUSH Example
Lowering SLEEP_​BPOOL_​FLUSH Waits
SLEEP_​BPOOL_​FLUSH Summary
WRITE_​COMPLETION
What Is the WRITE_​COMPLETION Wait Type?​
WRITE_​COMPLETION Example
Lowering WRITE_​COMPLETION Waits
WRITE_​COMPLETION Summary
Chapter 7:​Backup-Related Wait Types
BACKUPBUFFER
What Is the BACKUPBUFFER Wait Type?​
BACKUPBUFFER Example
Lowering BACKUPBUFFER Waits
BACKUPBUFFER Summary
BACKUPIO
What Is the BACKUPIO Wait Type?​
 BACKUPIO Example
Lowering BACKUPIO Waits
BACKUPIO Summary
BACKUPTHREAD
What Is the BACKUPTHREAD Wait Type?​
BACKUPTHREAD Example
Lowering BACKUPTHREAD Waits
BACKUPTHREAD Summary
Chapter 8:​Lock-Related Wait Types
Introduction to Locking and Blocking
Lock Modes and Compatibility
Locking Hierarchy
Isolation Levels
Querying Lock Information
LCK_​M_​S
What Is the LCK_​M_​S Wait Type?​
LCK_​M_​S Example
Lowering LCK_​M_​S Waits
LCK_​M_​S Summary
LCK_​M_​U
What Is the LCK_​M_​U Wait Type?​
LCK_​M_​U Example
Lowering LCK_​M_​U Waits
LCK_​M_​U Summary
LCK_​M_​X
What Is the LCK_​M_​X Wait Type?​
 LCK_​M_​X Example
Lowering LCK_​M_​X Waits
LCK_​M_​X Summary
LCK_​M_​I[xx]
What Is the LCK_​M_​I[xx] Wait Type?​
LCK_​M_​I[xx] Example
Lowering LCK_​M_​I[xx] Waits
LCK_​M_​I[xx] Summary
LCK_​M_​SCH_​S and LCK_​M_​SCH_​M
What Are the LCK_​M_​SCH_​S and LCK_​M_​SCH_​M Wait Types?​
LCK_​M_​SCH_​S and LCK_​M_​SCH_​M Example
Lowering LCK_​M_​SCH_​S and LCK_​M_​SCH_​M Waits
LCK_​M_​SCH_​S and LCK_​M_​SCH_​M Summary
Chapter 9:​Latch-Related Wait Types
Introduction to Latches
Latch Modes
Latch Waits
sys.​dm_​os_​latch_​stats
Page-Latch Contention
PAGELATCH_​[xx]
What Is the PAGELATCH_​[xx] Wait Type?​
PAGELATCH_​[xx] Example
Lowering PAGELATCH_​[xx] Waits
PAGELATCH_​[xx] Summary
LATCH_​[xx]
What Is the LATCH_​[xx] Wait Type?​
 LATCH_​[xx] Example
Lowering LATCH_​[xx] Waits
LATCH_​[xx] Summary
PAGEIOLATCH_​[xx]
What Is the PAGEIOLATCH_​[xx] Wait Type?​
PAGEIOLATCH_​[xx] Example
Lowering PAGEIOLATCH_​[xx] Waits
PAGEIOLATCH_​[xx] Summary
Chapter 10:​High-Availability and Disaster-Recovery Wait Types
DBMIRROR_​SEND
What Is the DBMIRROR_​SEND Wait Type?​
DBMIRROR_​SEND Example
Lowering DBMIRROR_​SEND Waits
DBMIRROR_​SEND Summary
HADR_​LOGCAPTURE_​WAIT and HADR_​WORK_​QUEUE
What Are the HADR_​LOGCAPTURE_​WAIT and HADR_​WORK_​
QUEUE Wait Types?​
HADR_​LOGCAPTURE_​WAIT and HADR_​WORK_​QUEUE
Summary
HADR_​SYNC_​COMMIT
What Is the HADR_​SYNC_​COMMIT Wait Type?​
HADR_​SYNC_​COMMIT Example
Lowering HADR_​SYNC_​COMMIT Waits
HADR_​SYNC_​COMMIT Summary
REDO_​THREAD_​PENDING_​WORK
What Is the REDO_​THREAD_​PENDING_​WORK Wait Type?​
REDO_​THREAD_​PENDING_​WORK Summary
Chapter 11:​Preemptive Wait Types
SQL Server on Linux
PREEMPTIVE_​OS_​ENCRYPTMESSAGE and PREEMPTIVE_​OS_​
DECRYPTMESSAGE
What Are the PREEMPTIVE_​OS_​ENCRYPTMESSAGE and
PREEMPTIVE_​OS_​DECRYPTMESSAGE Wait Types?​
PREEMPTIVE_​OS_​ENCRYPTMESSAGE and PREEMPTIVE_​OS_​
DECRYPTMESSAGE Example
Lowering PREEMPTIVE_​OS_​ENCRYPTMESSAGE and
PREEMPTIVE_​OS_​DECRYPTMESSAGE Waits
PREEMPTIVE_​OS_​ENCRYPTMESSAGE and PREEMPTIVE_​OS_​
DECRYPTMESSAGE Summary
PREEMPTIVE_​OS_​WRITEFILEGATHER
What Is the PREEMPTIVE_​OS_​WRITEFILEGATHER Wait
Type?​
PREEMPTIVE_​OS_​WRITEFILEGATHER Example
Lowering PREEMPTIVE_​OS_​WRITEFILEGATHER Waits
PREEMPTIVE_​OS_​WRITEFILEGATHER Summary
PREEMPTIVE_​OS_​AUTHENTICATIONOP​S
What Is the PREEMPTIVE_​OS_​AUTHENTICATIONOP​S Wait
Type?​
PREEMPTIVE_​OS_​AUTHENTICATIONOP​S Example
Lowering PREEMPTIVE_​OS_​AUTHENTICATIONOP​S Waits
PREEMPTIVE_​OS_​AUTHENTICATIONOP​S Summary
PREEMPTIVE_​OS_​GETPROCADDRESS
What Is the PREEMPTIVE_​OS_​GETPROCADDRESS Wait Type?​
PREEMPTIVE_​OS_​GETPROCADDRESS Example
Lowering PREEMPTIVE_​OS_​GETPROCADDRESS Waits
 PREEMPTIVE_​OS_​GETPROCADDRESS Summary
Chapter 12:​Background and Miscellaneous Wait Types
CHECKPOINT_​QUEUE
What Is the CHECKPOINT_​QUEUE Wait Type?​
CHECKPOINT_​QUEUE Summary
DIRTY_​PAGE_​POLL
What Is the DIRTY_​PAGE_​POLL Wait Type?​
DIRTY_​PAGE_​POLL Summary
LAZYWRITER_​SLEEP
What Is the LAZYWRITER_​SLEEP Wait Type?​
LAZYWRITER_​SLEEP Summary
MSQL_​XP
What Is the MSQL_​XP Wait Type?​
MSQL_​XP Example
Lowering MSQL_​XP Waits
MSQL_​XP Summary
OLEDB
What Is the OLEDB Wait Type?​
OLEDB Example
Lowering OLEDB Waits
OLEDB Summary
TRACEWRITE
What Is the TRACEWRITE Wait Type?​
TRACEWRITE Example
Lowering TRACEWRITE Waits
TRACEWRITE Summary
 WAITFOR
What Is the WAITFOR Wait Type?​
WAITFOR Example
WAITFOR Summary
Chapter 13:​In-Memory OLTP–Related Wait Types
Introduction to In-Memory OLTP
Checkpoint File Pairs (CFPs)
Isolation
Transaction Log Changes
WAIT_​XTP_​HOST_​WAIT
What Is the WAIT_​XTP_​HOST_​WAIT Wait Type?​
WAIT_​XTP_​HOST_​WAIT Summary
WAIT_​XTP_​CKPT_​CLOSE
What Is the WAIT_​XTP_​CKPT_​CLOSE Wait Type?​
WAIT_​XTP_​CKPT_​CLOSE Summary
WAIT_​XTP_​OFFLINE_​CKPT_​NEW_​LOG
What Is the WAIT_​XTP_​OFFLINE_​CKPT_​NEW_​LOG Wait Type?​
WAIT_​XTP_​OFFLINE_​CKPT_​NEW_​LOG Summary
Appendix I:​Example SQL Server Machine Configurations
Appendix II:​Spinlocks
Appendix III:​Latch Classes
Appendix IV:​Waits and DMVs
Index
About the Authors
Thomas LaRock
has over 20 years of IT experience holding administrator roles. He is a
Microsoft Certified Master in SQL Server and a Microsoft Data Platform
MVP since 2009. LaRock has spent much of his career working with
data and databases, which led to his selection as a Technical Advocate
for Confio Software in 2010 for the software now known as SolarWinds
Database Performance Analyzer (DPA).
Currently, he serves as a Head Geek for SolarWinds, a company
specializing in software for enterprise infrastructure monitoring. This
role allows for LaRock to work with a variety of customers, helping to
solve questions regarding network, application, and database
performance tuning and virtualization. You can reach Thomas through
his blog (thomaslarock.com/blog) and find him on Twitter
(@SQLRockstar).

Enrico van de Laar

has been working with data in various formats and sizes for over 15
years. He is a data and advanced analytics consultant for DataHeroes
where he helps organizations optimize their data platform
environment and helps them with their first steps in the world of
advanced analytics. He is a Data Platform MVP since 2014 and a
frequent speaker on various data-related events throughout the world.
He frequently blogs about technologies such as Microsoft SQL Server
and Azure Machine Learning on his blog at enricovandelaar.com. You
can contact Enrico on Twitter at @evdlaar.
About the Technical Reviewers
Denny Cherry
is the owner and principal consultant for
Denny Cherry & Associates Consulting
and has over two decades of experience
working with platforms such as Azure,
Microsoft SQL Server, Hyper-V, vSphere,
and Enterprise Storage solutions.
Denny’s areas of technical expertise
include system architecture,
performance tuning, security,
replication, and troubleshooting. Denny
currently holds several of the Microsoft
certifications related to SQL Server for
versions 2000 through 2017 including
the Microsoft Certified Master as well as
being a Microsoft MVP for over a decade.
Denny has written several books and dozens of technical articles on
SQL Server management and how SQL Server integrates with various
other technologies.

Joseph D’Antoni
is a Principal Consultant at Denny Cherry
& Associates Consulting. He is
recognized as a VMWare vExpert and a
Microsoft Data Platform MVP and has
over 20 years of experience working in
both Fortune 500 and smaller firms. He
has worked extensively on database
platforms and cloud technologies and
has specific expertise in performance
tuning, infrastructure, and disaster
recovery.
Part I
Foundations of Wait Statistics Analysis
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2023
T. LaRock, E. van de Laar, Pro SQL Server 2022 Wait Statistics
https://doi.org/10.1007/978-1-4842-8771-2_1

1. Wait Statistics Internals

Thomas LaRock1 and Enrico van de Laar2

(1) East Longmeadow, MA, USA

(2) Drachten, The Netherlands

SQL Server wait statistics are an important tool for analyzing

performance-related problems and optimizing SQL Server
performance. However, they are not always known to many database
administrators or developers. I believe this has to do with their
relatively complex nature, the sheer volume of different types available,
and the lack of documentation for many types of wait statistics. Wait
statistics are also directly tied to the specific SQL Server on which they
occur, meaning it is impossible to compare wait statistics of Server A to
wait statistics of Server B, even with identical hardware and database
configuration. Every configuration option, from the hardware firmware
level to the configuration of the SQL Server Native Client on the client
computers, will impact the wait statistics!
Therefore, we will start this book with a review of SQL Server
internals pertaining to wait statistics. We will discuss how wait events
are generated, how to view wait events, and how to analyze them as
part of a proper performance tuning methodology. Then in Part II of
this book, we will explore and examine specific wait events.
This chapter will look at the history of wait statistics through the
various versions of SQL Server. Next, we will review the SQL Operating
System, or SQLOS. The architecture of the SQLOS is closely tied to wait
statistics and to performance troubleshooting in general. The rest of
this chapter is dedicated to one of the most important aspects of wait
statistics: thread scheduling.
Before we begin, I want to mention a few things related to the
terminology used when discussing wait statistics. In the introduction of
this book and the preceding paragraphs, I mentioned the term wait
statistics. The sentence “compare wait statistics of Server A to wait
statistics of Server B” is technically wrong, since we compare the wait
time (the total time we have been waiting on a resource) of a specific
wait type (the specific wait type related to the resource we are waiting
on). From this point on, when I use the term wait statistics, I mean the
concept of wait statistics, and I will use the correct terms wait time and
wait type where appropriate.

A Brief History of Wait Statistics

SQL Server has been around for quite some time now. The first release
of SQL Server dates back to 1989 and was released for the OS/2
platform. Until SQL Server 6.0, released in 1995, Microsoft worked
together with Sybase to develop SQL Server. In 1995, however,
Microsoft and Sybase went their separate ways. Both Microsoft and
Sybase stayed active in the database world (SAP acquired Sybase in
2010). In 2022, Microsoft will release SQL Server 2022, while SAP
released SAP Sybase ASE 16 in 2014 (but is still maintained today),
both relational enterprise-level database systems.
Between SQL Server 6.0 and SQL Server 2022, many things have
changed, and it is difficult to compare any two versions alongside one
another. One thing which has not changed through the years is the
concept of wait statistics. Each version of SQL Server stores information
about its internal processes. Even though the way we access such
information has changed over the years, wait statistics remain an
important part of the internal logging process.
In early versions of SQL Server, access to wait statistics required the
use of undocumented commands. Figure 1-1 shows how to query wait
statistics information in SQL Server 6.5 using the DBCC command.
Figure 1-1 SQL Server wait statistics for SQL Server 6.5
One of the big changes introduced with SQL Server 2005 was the
conversion of many internal functions and commands into Dynamic
Management Views (DMVs) and Dynamic Management Functions
(DMFs). This made it easier to query and analyze the information
returned by functions and commands, including wait statistics
information. A new way of performance analysis was born with the
release of the SQL Server 2005 Microsoft whitepaper SQL Server 2005
Waits and Queues by Tom Davidson.
In the subsequent releases of SQL Server, the volume of new and
different wait types grew exponentially whenever new features or
configuration options were introduced. If you look at Figure 1-1, you
will notice that 21 different wait types were returned. Figure 1-2 shows
the amount of wait types, as the number of rows returned, available in
SQL Server 2022.
Figure 1-2 SQL Server wait statistics in SQL Server 2022
With the release of SQL Server 2022 RC0, the number of wait types
increased to more than 1,300 different wait types! The number of wait
types will likely continue to grow in future SQL Server releases, as new
features are introduced or existing features are changed. Thankfully
there is a lot more information available about wait statistics now than
there was in SQL Server 6.5!

The SQLOS
The world of computer hardware changes constantly. Every year, or in
some cases every month, we stuff more cores inside of processors,
increase the memory capacity of mainboards, or introduce entirely new
hardware concepts like PCI-based persistent flash storage. Database
Management Systems (or DBMSs) are eager to take advantage of new
hardware trends. Because of the fast-changing nature of hardware and
the need to utilize new hardware options as soon as they become
available, the SQL Server team decided to change the SQL Server
platform layer starting with SQL Server 2005.
Before SQL Server 2005, the platform layer of SQL Server was
restricted, with many operations performed by the operating system
itself. This meant it was difficult for SQL Server to keep up with the fast-
changing world of server hardware, as changing a complete operating
system in order to utilize faster hardware or new hardware features is a
time-consuming and complex operation.
Figure 1-3 shows the (simplified) architecture of SQL Server before
the introduction of the SQLOS in SQL Server 2005.

Figure 1-3 SQL Server architecture before the introduction of SQLOS

SQL Server 2005 introduced one of the biggest changes to the SQL
Server engine seen to this day, the SQLOS. This is a completely new
platform layer functioning as a user-level operating system. The SQLOS
has made it possible to fully utilize current and future hardware and
has enabled features like advanced parallelism. The SQLOS is highly
configurable and adjusts itself to the hardware it is running on, thus
making it perfectly scalable for high-end or low-end systems alike.
Figure 1-4 shows the (simplified) architecture of SQL Server 2005,
including the SQLOS layer.

Figure 1-4 SQL Server 2005 architecture

The SQLOS changed the way SQL Server accesses processor

resources by introducing schedulers, tasks, and worker threads. This
gives the SQLOS greater control of how work should be completed by
the processors. The Windows operating system uses a preemptive
scheduling approach. This means that Windows will give every process
that needs processor time a priority and fixed slice of time, or a
quantum. This process priority is calculated from a number of variables
like resource usage, expected runtime, current activity, and so forth. By
using preemptive scheduling, the Windows operating system can
choose to interrupt a process when a process with a higher priority
needs processor time. This way of scheduling can have a negative
impact on processes generated by SQL Server, since those processes
could easily be interrupted by higher priority ones, including those of
other applications. For this reason, the SQLOS uses its own
(cooperative) non-preemptive scheduling mechanism, making sure that
Windows processes cannot interrupt SQLOS processes.

Nerd Note SQL Server 7 and SQL Server 2000 also used non-
preemptive scheduling using User Mode Scheduling (UMS). SQLOS
brought many more system components closer together, thus
enabling better performance and scalability.

There are some exceptions when the SQLOS cannot use non-preemptive
scheduling, for instance, when the SQLOS needs to access a resource
through the Windows operating system. We will discuss these
exceptions later in this book in Chapter 11, “Preemptive Wait Types.”

Schedulers, Tasks, and Worker Threads

Because SQLOS uses a different method to execute requests than the
Windows operating system uses, SQL Server introduced a different way
to schedule processor time using schedulers, tasks, and worker threads.
Figure 1-5 shows the different parts of SQL Server scheduling and how
they relate to each other.
Figure 1-5 SQL Server scheduling

Sessions
A session is the connection a client has to SQL Server after it has been
successfully authenticated. We can access session information by
querying the sys.dm_exec_sessions DMV using the following query:

SELECT *
FROM sys.dm_exec_sessions;
 In older versions of SQL Server, user sessions had a session_id
greater than 50; everything lower was reserved for internal SQL Server
processes. However, on very busy servers, there was a possibility SQL
Server needed to use a session_id greater than 50. If you are only
interested in information about user-initiated sessions, it is better to
filter the results of the sys.dm_exec_sessions DMV using the
is_user_process column instead of filtering on a session_id greater than
50. The following query will only return user sessions and will filter out
the internal system sessions:

SELECT *
FROM sys.dm_exec_sessions
WHERE is_user_process = 1;

Figure 1-6 shows a small part of the results of this query.

Figure 1-6 sys.dm_exec_sessions results

There are many columns returned by the sys.dm_exec_sessions

DMV that provide information about each session. Some of the more
interesting columns include the following:
host_process_id – The process ID or (PID) of the client program
connected to SQL Server will show NULL for internal sessions.
cpu_time – The amount of processor time (in milliseconds) the
session has used since it was first established.
memory_usage – The amount of memory (8k pages) used by the
session since it was first established.
reads – Number of reads performed by this session since it was first
established.
writes – Number of writes performed by this session since it was
first established.
 row_count – Number of rows returned for this session up to this
point.
status – Possible values include “running,” “sleeping,” “dormant,” and
“preconnect.”
The most common values of the status column are “running,” which
indicates that one or more requests are currently being processed from
this session, and “sleeping,” which means no requests are currently
being processed from this session.

Requests
A request is the SQL Server execution engine’s representation of a
query submitted by a session. Again, we will use a DMV to query
information about a request; in this case, we query the
sys.dm_exec_requests DMV:

SELECT *
FROM sys.dm_exec_requests;

Figure 1-7 shows a portion of the results of this query.

Figure 1-7 sys.dm_exec_requests results

 The sys.dm_exec_requests DMV is an incredibly powerful tool for
troubleshooting performance-related issues. This DMV has a lot of
information about current queries and helps detect performance
bottlenecks relatively quickly. Because the sys.dm_exec_requests DMV
also displays wait statistics–related information, we will take a
thorough look at it in Chapter 2, “Querying SQL Server Wait Statistics.”

Tasks
Tasks represent the work to be performed by the SQLOS, but tasks do
not perform any work themselves. When a request is received by SQL
Server, one or more tasks are created to fulfill the request. The number
of tasks generated for a request depends on if the query request is
being performed using parallelism or if it’s being run serially.
We will use the sys.dm_os_tasks DMV to query the task
information:

SELECT *
FROM sys.dm_os_tasks;

Figure 1-8 shows a part of the results of the query.

Figure 1-8 sys.dm_os_tasks results

When you query the sys.dm_os_tasks DMV, you will discover it

returns many results, even on servers with little user activity. This is
because SQL Server uses tasks for its own processes as well.
 There are some interesting columns in this DMV worth exploring to
see the relations between different DMVs:
session_id – The ID of the session which requested this task.
worker_address – The memory address of the worker thread
associated with this task.
task_address – The memory address of this task.
task_state – Possible values include “pending,” “runnable,” “running,”
“suspended,” “done,” and “spinloop.”
The most common values of the task_state column are “running,”
“runnable,” and “suspended.”

Worker Threads
Every task created has a worker thread assigned, and the worker
thread then performs the actions requested by the task.

Nerd Note A worker thread does not perform the work itself;
instead it requests a thread from the Windows operating system to
perform the work. For the sake of simplicity, and the fact the
Windows thread runs outside the SQLOS, I have left this step out of
Figure 1-5. You can access information about the Windows operating
system threads by querying sys.dm_os_threads if you are
interested.

When a task requests a worker thread, SQL Server will first look for an
idle worker thread to assign to the task. In the case when no idle
worker thread is located and the maximum number of worker threads
has been reached, the request is queued until a worker thread finishes
its current work and becomes available.
There is a limit to the number of worker threads SQL Server has
available for processing requests. This number is automatically
calculated and configured by SQL Server during startup. We calculate
the maximum number of worker threads ourselves using these
formulas:

32-bit system with <= 4 logical 256 worker threads

processors
32-bit system with > 4 logical processors 256 + ((# of logical processors – 4) * 8)
64-bit system with <= 4 logical
512 worker threads
processors
512 + ((# of logical processors – 4) *
64-bit system with > 4 logical processors
16)
Example: For a 64-bit system with 16 logical processors, we
calculate the maximum number of worker threads using the preceding
formula:

512 + ((# of logical processors – 4) * 16) =

512 + ((16 – 4) * 16) =
512 + (12 * 16) =
512 + (192) = 704

which would assign a maximum of 704 worker threads.

The number of worker threads has a default of 0, allowing SQL
Server to set the number of max worker threads using the preceding
formulas when it starts. We alter the default by changing the max
worker threads options in SQL Server’s properties, as illustrated by
Figure 1-9.
Figure 1-9 Processors page in the Server Properties
Generally speaking, you should not need to change the max worker
threads option, and my advice is to leave the setting alone, as it should
only be changed in very specific cases (I will discuss one of those
potential cases in Chapter 5, “CPU-Related Wait Types,” when we talk
about THREADPOOL waits).
One thing to keep in mind is worker threads require memory to
work. For 32-bit systems, this is 512 KB for every worker thread; 64-bit
systems will need 2048 KB for every worker thread. Thus, changing the
number of worker threads can potentially impact the memory
requirements of SQL Server. This does not mean you need a massive
amount of memory just for your worker threads – SQL Server will
automatically destroy idle worker threads after 15 minutes or if SQL
Server is under heavy memory pressure.
SQL Server supplies us with a DMV to query information about the
worker threads, sys.dm_os_workers. Figure 1-10 shows some of the
results of this query:

SELECT *
FROM sys.dm_os_workers;

Figure 1-10 Results of querying sys.dm_os_workers

The sys.dm_os_workers DMV is large and complex with many

columns marked as “Internal use only” by Microsoft. However, the
columns task_address and scheduler_address are available to link
together the different DMVs we have discussed so far.
Worker threads iterate through different phases while exposed to
the processor, which we observe when we examine the state column in
the sys.dm_os_workers DMV:
​INIT – The worker thread is initialized by the SQLOS.
RUNNING – The worker thread is currently performing work on a
processor.
RUNNABLE – The worker thread is ready to run on a processor.
SUSPENDED – The worker thread is waiting for a resource.
 The states the worker threads go through while performing work is
the main focus of this book. Every time a worker thread is not in the
“RUNNING” state, it has to wait, and the SQLOS records this information
giving us valuable insight into what the worker thread has been waiting
on and how long it has been waiting.

Schedulers
The scheduler component’s main task is to – surprise – schedule work
in the form of tasks on a physical processor(s). When a task requests
processor time, it is the scheduler that assigns worker threads to the
task, so the request is processed. The scheduler is also responsible for
making sure worker threads cooperate with each other and yield the
processor when their slice of time, or quantum, has expired. We call this
cooperative, or non-preemptive, scheduling. The need for worker
threads to yield when their processor time has expired comes from the
fact that a scheduler allows only one worker thread to run on a
processor at a time. If the worker threads didn’t need to yield, a worker
thread might stay on the processor for an infinite amount of time,
blocking all usage of that processor.
There is a one-on-one relation between processors and schedulers.
If your system has two processors, each with four cores, there will be
eight schedulers that the SQLOS can use to process user requests, each
of them mapped to one of the logical processors.
We can access information about the schedulers by running a query
against the sys.dm_os_schedulers DMV:

SELECT *
FROM sys.dm_os_schedulers;

The results of the query are shown in Figure 1-11.

Figure 1-11 sys.dm_os_schedulers query results
The SQL Server on which I ran this query has two processors with
two cores, which means there should be four schedulers to process
user requests. If we look at Figure 1-11, however, we notice there are
more than four schedulers returned by the query. SQL Server uses its
own schedulers to perform internal tasks, and those schedulers are
returned by the DMV and marked “HIDDEN ONLINE” in the status
column. The schedulers available for user requests are marked as
“VISIBLE ONLINE.” There is also a special type of scheduler with the
status “VISIBLE ONLINE (DAC).” This is a scheduler dedicated for use
with the Dedicated Administrator Connection (DAC). This scheduler
makes it possible to connect to SQL Server in situations where it is
unresponsive, for instance, when there are no free worker threads
available on the schedulers processing user requests.
We view the number of worker threads a scheduler has associated
with it by looking at the current_workers_count column. This number
also includes worker threads not performing any work. The
active_workers_count shows us the worker threads that are active on
the specific scheduler. This doesn’t mean they are running on the
processor, as worker threads with states of “RUNNING,” “RUNNABLE,”
and “SUSPENDED” also count toward this number. The
work_queue_count is also an interesting column as it provides insight
into how many tasks are waiting for a free worker thread. If you see
high numbers in this column, it might mean you are experiencing CPU
pressure.

Putting It All Together

All SQL Server scheduling parts discussed above are connected to each
other, as every connection, session, and query request passes through
these same components. The following text is an example of how a
query request would get processed:
A user connects to the SQL Server through an application.
↓
SQL Server creates a session after the login process is completed
successfully.
↓
When the user sends a query to the SQL Server, a task and a request
are created to represent the unit of work that needs to be done.
↓
The scheduler will assign worker threads to the task so it can be
completed.
To see this information in SQL Server, we join some of the DMVs
discussed. The query in Listing 1-1 will show you an example of how to
combine different DMVs to get scheduling information about a specific
session (in this case a session with an ID of 55).

SELECT
r.session_id AS 'Session ID',
r.command AS 'Type of Request',
qt.[text] AS 'Query Text',
t.task_address AS 'Task Address',
t.task_state AS 'Task State',
w.worker_address AS 'Worker Address',
w.[state] AS 'Worker State',
s.scheduler_address AS 'Scheduler Address',
s.[status] AS 'Scheduler State'
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) qt
INNER JOIN sys.dm_os_tasks t
ON r.task_address = t.task_address
INNER JOIN sys.dm_os_workers w
ON t.worker_address = w.worker_address
INNER JOIN sys.dm_os_schedulers s
 ON w.scheduler_address = s.scheduler_address
WHERE r.session_id = 70;
Listing 1-1 Join the different DMVs together to query scheduling information
Figure 1-12 shows the information that the query returned on my
test SQL Server. To keep the results readable, I only selected columns
from the DMVs to show the relation between them.

Figure 1-12 Results of the query from Listing 1-1

In the results, we see Session ID 70 made a SELECT query request.

In the query from Listing 1-1, we obtain the text of the query request
using a cross apply with the sys.dm_exec_sql_text Dynamic
Management Function. The request was then mapped to a task, and the
task began running. The task was then mapped to a worker thread, and
the task was now in a running state. This meant the query began
processing on a processor. The Scheduler Address column shows on
which specific scheduler our worker thread was run.

Wait Statistics
So far, we have gone deep into the different components which perform
scheduling for SQL Server and how they are interconnected. Now we
turn our attention to the main topic of this book: wait statistics.
In the section on worker threads earlier in this chapter, I described
the states a worker thread is in while performing work on a scheduler.
When a worker thread is performing its work, it goes through three
different phases (or queues) in the scheduler process. Depending on
the phase (or queue) a worker thread is in, it will enter the “RUNNING,”
“RUNNABLE,” or “SUSPENDED” state. Figure 1-13 shows an abstract
view of a scheduler with the different phases.
Figure 1-13 Scheduler and its phases and queues
When a worker thread obtains access to a scheduler, it will
generally start in the Waiter List and be assigned the “SUSPENDED”
state. The Waiter List (shown as Waiting in Figure 1-13) is an
unordered list of worker threads with the “SUSPENDED” state and is
waiting for resources to become available. Here, resources are anything
on the system, such as data pages or maybe a lock request. While a
worker thread is in the Waiter List, the SQLOS records the type of
resource it needs to continue its work (this is the wait type) and the
time it spends waiting before that specific resource becomes available
(this is the resource wait time).
When a worker thread receives access to the resources it needs, it
will move to the Runnable Queue, a first-in-first-out list of all worker
threads with access to their resources and ready to be run on a
processor. The time a worker thread spends in the Runnable Queue is
recorded as the signal wait time.
 The top worker thread in the Runnable Queue will move to the
“RUNNING” phase, where it will receive processor time to perform its
work. The time it spends on the processor is recorded as CPU time. In
the meantime, the other worker threads in the Runnable Queue move
up the list, and worker threads that have received their requested
resources move from the Waiter List into the Runnable Queue.

Nerd Note The Runnable Queue is a first-in-first-out list (FIFO). If

this was not true, a task could theoretically never move from
runnable to running on a busy system. The Waiter List, however, is
not FIFO.

While a worker thread is in the “RUNNING” phase, there are three

scenarios that can happen:
1. The worker thread needs additional resources; in this case, it
moves from the “RUNNING” phase to the Waiter List (what we see
as SUSPENDED).

2. The worker thread spends its quantum (fixed value of 4

milliseconds) and therefore must yield; the worker thread is moved
to the bottom of the Runnable Queue.

3. The worker thread is done with its work and will leave the
scheduler.

Worker threads move through the three different phases all the
time. It is common for one worker thread to rotate through multiple
times until all work is done.
Knowing the different lengths of time a request spends in one of the
three different phases makes it possible to calculate total request
execution time and also the total time a request waits for either
processor time or resource time.
Expressed as a mathematical formula, the total request execution
time would look like this:
Random documents with unrelated
content Scribd suggests to you:
"The man Zara loves. And that you may know the worst, let me tell
you that she is a dancer at a Bethnal Green music-hall."

"Hum!" said Mrs. Perage, smiling grimly. "And by mentioning her

profession and position you think that I will have a bad opinion of
her. Fudge! I have met with dancers much better as regards morals
than many a woman received at Court. Don't be a fool and think you
are talking to an inexperienced girl."

"Well, I did talk to an inexperienced girl," said Hench rather bitterly,

"and she has turned on me."

"Why not? You gave her no explanation."

"How could I, when she ran away while I was speaking? I couldn't
follow quickly enough, as my foot is yet weak."

"Your ankle, you mean--be careful in your speech." Mrs. Perage

rubbed her nose again and her eyes grew calmer. "I'll have a cup of
tea if you will have the decency to give me one."

Owain rang for a fresh cup and saucer. "I thought you wouldn't
condescend to eat and drink with a pariah."

"Fudge!" said Mrs. Perage again, and very sharply. "Who said you
were a pariah, you silly fellow? That's merely hurt vanity on your
part."

"How can I help being hurt, when I am so misjudged?"

"Look here." Mrs. Perage bent forward and shook his shoulder. "Are
you a man or a twopenny-halfpenny school-girl?"

"I'm an ass," confessed Owain, ashamed of his petty outbreak. "But

I have an attack of nerves, I think, owing to my dreadful position."
"Hum!" Mrs. Perage rubbed her nose, received a cup and saucer
from Mrs. Bell, who had just entered the room, and sent that fragile
person out again. "Jim hinted at trouble. It seems he was right."

"Jim knows all about it."

"Well, then, I don't. Wait till I fill my cup and then you can tell me."

"Tell you what?"

"Drat the man, you know. It's more than this trouble with Gwen you
have to tell me about."

"I think that I had better tell you about the trouble with Gwen first."

"What's the use of beginning at the wrong end? Relate the story
from start to finish and then I'll understand more about this
interview in the churchyard with this ridiculous old woman."

"Madame Alpenny."

"Hum! The name fits her. Go on."

"I have already told you most of my life---"

"And have left out the most interesting part, apparently. See here,
Hench, or rather, I should say, Owain." Mrs. Perage drank some of
her tea and continued slowly. "I am an old woman with a romantic
heart. I love Gwen and I have taken a fancy to you. Both you and
Gwen come of a bad stock, as old Mynydd Evans was a miser, Owain
Evans was a profligate, and Madoc Evans was a scoundrel, fit for any
deed of wickedness. You two children are the best of the bunch, and
I expect get your decent morals from your mothers. I want to see
you happy and married. Now, don't disappoint me."

"I certainly won't, if Gwen won't," said Owain promptly.

"Hum! Gwen is a more difficult person to manage. However, if you
leave it to me, I think in some way things will be put right."

"Oh, I shall leave everything to you, with pleasure," said Hench

eagerly. "And I thank you for the trouble you are taking. Your
advice----"

"Cannot be given further until I am in possession of facts,"

interrupted Mrs. Perage, and finishing his sentence in a different
way. "I know that you are Owain's son and inherit the property. I
know that you love Gwen, and that it is possible, in spite of existing
circumstances, that you will marry her. Also I am aware that Madoc
was murdered--by that tramp, I presume."

"No!" said Hench sharply, and ready to make a clean breast. "I am
the tramp."

"Ha!" exclaimed the old lady in a tone of surprise. "You are the
tramp? Well, I withdraw my accusation, as I am sure you are
innocent enough. But what I was coming to when you interrupted
me was that I wish to know more. Jim says you are in trouble."

"In very great trouble. And if you will help me---"

"Bless the man, what I came here for was to help. But I can't do
that on half-confidences. You must speak plainly. Now, no more talk.
Begin." Hench did as he was ordered, and in a very short time Mrs.
Perage was in possession of all facts connected with the
advertisement; with the keeping of the appointment and the
discovery of the body; and with the schemes of Madame Alpenny.
Her strong old face did not betray much emotion, although she was
inwardly astonished at the revelations, but she kept her eyes on
Owain until he ceased speaking, and then rubbed her nose, as was
her custom when perplexed or annoyed. As she made no remark,
Hench did so. "What do you think?"
"Hum!" said Mrs. Perage, starting from the brown study in which she
was involved. "You've brought your pigs to a pretty market, young
man. Well, well, we must see what is best to be done."

"You don't believe me to be guilty?"

"Would I be still sitting here if I did? Don't be a fool. Not that I

blame the person who got Madoc out of the way very much. He was
such a disagreeable person, that I often thought I'd be hanged for
killing him myself."

"Mrs. Perage!"

"It sounds dreadful, doesn't it?" she said good-humouredly. "But

then you see I am a dreadful person in the eyes of many milk-and-
water people, because I have my own decided opinions and go my
own way. I suppose it's wrong to say a word against the dead,
although I don't see why we should talk of nothing but virtues they
never possessed while alive. Well, let the man rest; he did a lot of
harm when he was alive, and wherever he has gone to, he's making
mischief. You didn't murder him, anyhow?"

"I certainly did not," answered Hench, smiling. "But the question is,
who did?"

"Ah"--Mrs. Perage kilted up her dress and folded her hands on her
knees--"a very difficult question to answer. But Madame Alpenny
didn't, although you seem to have some idea that she is the guilty
person."

"She knew my uncle and all about the disposal of the property
through the confidence made to her by my father twenty years ago."

"That doesn't prove that she murdered Madoc. She wanted you to
marry her daughter undoubtedly after she laid hold of the clue which
led her to learn that you were likely to inherit ten thousand a year.
But why should she put her neck in a noose?"
"She might have wished me to get possession of the property at
once, and have murdered my uncle in the hope that I would go to
the spot and then run the risk of being arrested. I believe myself
that it was all a plot to get me under her thumb. I did go to the
rendezvous and I _am implicated. Well?"

Mrs. Perage rubbed her nose again. "The devil's in it for trouble,"
she muttered. "Perhaps I am premature in assuming that this
woman is innocent, but it seems incredible that she should run such
a risk. I shall have to see her first before I make up my mind. She's
clever."

"In a foxy sort of way."

"Hum! The fox doesn't do things on a big scale in the way of killing."

Hench answered flippantly, as the conversation was getting on his

nerves. "What about hen-roost massacres?"

Mrs. Perage rose, and was about to rebuke him when she saw, as
Gwen had seen earlier, the white pinched look on his face. "You're
over-wrought, my friend. I want you to promise me two things."

"Yes. What are they?" asked the young man wearily.

"In the first place do not make any move in these matters until I
give you leave. I have a plan in my head."

"What is it?"

"I shan't tell it until it is carried out. In the second place do not come
to my house until to-morrow afternoon."

"But Gwen will believe more than ever that I am----"

"What she thinks you are in a moment of rage on her part," finished
Mrs. Perage. "That's just it. If you see her now you will spoil all. Wait
until I tell you that it is safe to come."

"Very well. But I can't let you take my burden on your shoulders and
stay here doing nothing. It's not cricket."

"You'll get all the cricket you require, I promise you," said Mrs.
Perage as she took her departure. "I don't mind telling you," she
added, glancing back, "that it interests me to have something
exciting of this sort to do. Life is rather dull hereabouts."

"I only hope it will not prove too exciting."

The old lady laughed and stepped briskly out of the cottage, while
Owain remained where he was kicking against the pricks. He wished
to see Gwen, but as he had promised to wait for instructions he
could not do so. Like the lady who had just left, he found life in
Cookley intolerably dull at the moment. But then, as Gwen was not
beside him, he would have found it equally dull had he been alone in
Paris or London. It was Gwen who made up his existence, and
nothing else mattered particularly. To such lengths does the passion
of love lead ordinarily sensible human beings.

Meanwhile, Mrs. Perage walked home briskly, turning over certain

plans in her very capable mind. She did not seek out Gwen, who was
weeping in the retirement of her bedroom, since all explanations at
the present moment were futile. But Mrs. Perage decided that when
the girl grew calmer a very positive explanation, which could not be
mistaken, should be made to her by the right person. To bring about
this necessary event she looked up her nephew, whom she found
dawdling in the garden with a cigarette and a French novel. Vane lay
on the grass under a shady tree clothed in white flannels, and
looked rather alarmed when his aunt appeared. The day was hot,
and Mrs. Perage was so uncommonly active that she was scarcely a
desirable companion for a lazy man. His anxiety was therefore
natural.
"Sit up and listen," said Mrs. Perage, getting to work at once. "I've
seen our young friend, and I now know as much as you do."

Jim sat up cross-legged, resigned to the worst, and Mrs. Perage

seated herself on the rustic bench under the tree with the air of a
judge trying a particularly vicious criminal. "Need we discuss matters
just now?" he asked in a bored tone. "I'm so comfortable. Peter is
bringing me some tea, I have a book and a case of cigarettes, so on
the whole----"

"Don't be an ass, Jim. You can be busy enough if you like."

"That's just it, Aunt Emma," remonstrated the barrister, clutching his
ankles. "I don't like. There's nothing to be done at present. I'll see
Owain this evening and hear how he settled with that old woman."

"He has settled nothing. But he managed to get her to leave him
alone for seven days. In that time much can be done."

"Very probably. I'm sure I wish to do all I can. And Gwen?"

"She's crying in her bedroom. She will continue to cry until she is
assured that Owain really loves her and not this other girl. You know
what I mean?"

"Well, as you related what took place in the churchyard and as Gwen
repeated the story to me, I must admit that I do know. I say, Aunt
Emma, you don't think Miss Evans minds me calling her Gwen, as I--
--"

"Oh, don't talk rubbish," interrupted Mrs. Perage quickly. "We have
more important things to speak about. This evening you must go to
town by the seven train,"--she glanced at her watch. "That will give
you time to have dinner comfortably, as you needn't dress."

"But, I say,"--Vane looked rather disgusted,--"I don't want to go to

town."
"You must," said his aunt impressively. "Go to Bethnal Green, and
bring down with you to-morrow Mademoiselle Zara."

"What for?"

"Bless the man, can't you understand? Only this Zara creature can
set Gwen's mind at rest. She can explain that Hench never really
loved her and only offered himself to her to gain a home and a
companion."

"Can't Owain tell Gwen that?"

"He might tell it to her fifty times and she would not believe him,"
said Mrs. Perage shrewdly. "But when this girl speaks everything will
be put right straight away. Then we can consider what is best to be
done about the other and more serious business. But you must see,
Jim, that it is first necessary to adjust matters between Gwen and
Hench."

"Well, Aunt Emma, you understand your own sex better than I do,
so I suppose it is best for me to bring Zara Alpenny down."

"I am quite positive it is."

"Good! I'll enjoy my dinner and then go to town by the train you
mention. I can bring Mademoiselle Zara to your house about two
o'clock to-morrow. Now that's all right." Vane yawned and rose. "Ah,
here comes Peter with the tea."

Mrs. Perage looked rather grimly on the freckled page who carried
on a tray the beverage which Mr. Vane desired. Hench had told her
how Madame Alpenny had learned his whereabouts through Simon,
alias_ Bottles, and the same could have only acquired the knowledge
through Peter.

"Here!" she said sharply. "Do you write to your brother in town and
tell him all the gossip of the village?"
"Me, mum? No, mum," said Peter, rather alarmed by her peremptory
tone.

"Don't tell lies, boy," said his mistress sternly. "You told your brother
that Mr. Hench was staying at Mrs. Bell's cottage."

"I know I did, mum." Peter began to whimper. "But I hope I didn't
do no harm, mum. Simon, he thinks no end of Mr. Hench, so I
thought as I'd tell him. But it's all right, mum. Simon knows what
he's about."

"What do you mean by that?" questioned Vane quickly, for the page
spoke in a very significant tone. Peter shuffled and wriggled
uncomfortably. "Simon will tell you, sir, when the time comes," he
replied evasively.

"Tell what?"

"What Simon knows, sir."

"And what does Simon know?"

"I can't tell you, sir. Simon's clever. He knows a thing or two."

"And so do I," said Mrs. Perage sternly. "And one is that you are not
to write gossiping letters from my house."

"No, mum, I won't!" And Peter went away as quickly as he could lest
he should be questioned further. "Now what does that mean?" asked
Mrs. Perage shrewdly. "Is this brat and his brother mixed up in this
dangerous business?"

"It seems like it," replied Jim, stirring his tea meditatively. "But Peter
may have written in all innocence, knowing how Bottles adores
Owain."
"Bottles, as you call him, didn't tell Madame Alpenny in all
innocence," she snapped.

"Hum!" said Vane, quite in his aunt's style, "we'll look into the
matter." And he did so on the morrow when he went to Bethnal
Green.

CHAPTER XVI

EXPLANATIONS

Gwen was thoroughly miserable. On returning from the churchyard

she had shut herself up in her bedroom, after a sobbing description
to Mrs. Perage and Vane of what had taken place. In this seclusion
she remained, speaking little, eating less, and only sleeping
occasionally when exhausted Nature insisted upon having her own
sensible way. The trouble Gwen was now undergoing seemed ever
so much worse than that which she had already undergone. The
death of her father had been dreadful, but he had been such a
tyrant that--to speak plainly--his loss had not broken her heart. But
now she felt certain that her heart was really and truly broken, as
the idea of losing Owain was like a nightmare. The girl by this time
fully recognized that she loved her cousin dearly, even though that
love had grown as rapidly and unexpectedly as Jonah's gourd.
Perhaps, like the same, it would perish as quickly. Gwen attempted
to assure herself of this, but could not self-hypnotise herself into
such a belief. Her passion was too genuine, too strong, too
overwhelming, to be got rid of so easily.
Yet--she asked herself this question frequently--how could she
believe that Owain loved her, when she had heard from his own lips
that he had proposed to another girl? Gwen considered that she had
been very generous in forgiving his masquerading, although she
admitted that under the circumstances the assumption of a false
name had been pardonable. But that he should have loved some one
else, and should have proposed to that some one, seemed to her to
be monstrous. It was impossible for her to forget or forgive such a
thing. She assured herself that self-respect demanded the adoption
of this merciless attitude, but the cause of it--which she would not
admit--was really jealousy. But whatever it was the feeling made her
wretched, and for long hours the poor child tossed and turned and
shivered and wept, as she wondered what her future was likely to
be. She had youth, she had beauty, she had money, but all these
desirable things were as dust and ashes, lacking the companionship
of the man she loved. And as he had condemned himself out of his
own mouth she could not see how the position of things was to be
altered.

In her bluff way, Mrs. Perage was very sorry for the girl, as she saw
how truly genuine was her suffering. The old lady strongly
sympathized with that despairing feeling of youth which believes
that the world has come to an end because things do not turn out as
expected. Not that she believed Gwen's world had ended, but
understood easily enough how the girl thought so. To put matters
right, Mrs. Perage set herself to work in the hope of proving that the
sun was merely obscured for the moment. For a day and a night she
left the sufferer alone, so that she might get over the first stage of
misery and anger. Then the old dame entered the bedroom and
proceeded to develop her scheme, which she hoped would put the
crooked straight.

"Well, my dear," she said in a brisk and heartless manner, as she

seated herself on the bed, "have you overcome your fit of self-pity?"
"Oh, how unkind you are," wailed Gwen, who did not expect such a
speech. "My heart is broken."

"No, my dear, your vanity is hurt."

"Vanity? I have no vanity."

"Well, well, we will call it pride, self-respect, dignity, or any other

pretty name which appeals to you," said Mrs. Perage complacently.
"Anyhow, you can't lie here amongst the ruins of your life. Have
some breakfast and get up."

"I can't eat and I can't drink. How can you expect me to?" cried
Gwen, who was intensely exasperated by this matter-of-fact speech.
"You will make me angry, Mrs. Perage."

"I want to, since anger will make you see things in a more sensible
light. You can't live on air, you know, my dear, or on love either,
especially as this last is nonexistent."

The spirit of contradiction, begotten by anger, made the invalid

resent this last remark. "Love isn't nonexistent," she declared
crossly. "I love Owain still, although he doesn't deserve my affection
in the least. I call it a shame for him to come here and save my life
and make me love him, when all the time he is engaged to another
girl."

"Who told you that he was?" inquired Mrs. Perage dryly, and very
well satisfied with the result her conversation was producing.

"He told me so himself, and I told you how he was," said Gwen
incoherently. "He admitted that he had proposed to the nasty
daughter of that horrid woman."

"Well," said Mrs. Perage coolly, "a young man must gain experience
somehow."
"Owain shan't gain any at my expense," retorted Gwen viciously.
"After all, I don't think that he is worth troubling about."

"Of course he isn't," said Mrs. Perage, wishing to emphasize this

opinion. "So lie down and go to sleep and forget all about him. You
can't eat, you know."

"Yes, I can." Gwen rose in the bed angrily. "I shall have my breakfast
and get up and go about things just as if nothing had happened."

Mrs. Perage shook her old head wisely. "You have not the strength."

"I have--I have. Ring the bell and order some tea and toast."

"Peter is bringing up some sort of a meal, my dear. Ah, there is his

knock. I will take the tray," and Mrs. Perage went to the door to do
so, chuckling at the way in which she was dealing with the situation.
"Give it to me, Peter; now you can go. By the way, Gwen, shall I
send him for the doctor?"

"No. I'm quite well," said the girl indignantly. So Peter was dismissed
and the tray was placed on the bed. "Leave me to eat, Mrs. Perage,
and you can come back after I have dressed."

"Foolish! Foolish!" said the old dame, leaving the room. "You are
attempting too much." And she departed, still chuckling to think how
easily this somewhat difficult young lady had fallen into the trap.

Gwen, quite ignorant that she was acting exactly as Mrs. Perage
desired, sipped the tea and nibbled at the toast. Pride speedily came
to her aid, and when the meal was finished she felt much better.
Self-pity was now merged in a sense of anger that Owain had dared
to treat her so shamefully, therefore she dressed herself in her
prettiest frock with the intention of proving to him that she felt his
treachery less than he might have expected. When she walked into
the drawing-room, Mrs. Perage looked up to see a smartly dressed
young lady with sparkling eyes and a fine colour, in place of the
white-faced invalid she had left. So far the result of the experiment
was distinctly good.

"And of course," suggested the old lady artfully, "you have quite
decided to throw Owain overboard."

"What else would you have me do?" demanded Gwen revengefully.

"Hum!" said Mrs. Perage in a meditative manner. "I think I should

ask for an explanation."

"There can be no explanation likely to satisfy me."

"That entirely depends upon my common-sense way of looking at

things," said Mrs. Perage dryly. "Or on your common-sense, if you
come to that. By the way, that girl is coming down here this
afternoon--she will arrive in an hour."

"What girl?"

"Hum!" Mrs. Perage skirted round the subject and did not give an
entirely direct reply. "Your breakfast has been your luncheon, for it is
now two o'clock, so such a queer exchange of meals must have
upset you. Perhaps you had better not be present."

"What girl are you talking about?" asked Gwen, her colour coming
and going, although she knew perfectly well what was meant. "And I
am in quite enough good health to see any girl. How dare she come
here?"

"Ah!"--Mrs. Perage chuckled,--"you guess what I mean, I see. Well,

my dear Jim was rather put out about your quarrel with Hench, so
he suggested at my desire that it would be as well for him to go to
town and bring Mademoiselle Zara with him down here to explain
matters."
"I don't require any explanation," said Gwen, holding her head very
high.

"Bless the girl, did I say so? This Zara woman is coming to explain to
me. I may as well be plain, Gwen. It was I who told Jim to go to
town and fetch her, since it is necessary that I should learn what a
rascal Hench is."

"He's not a rascal; I'm sure he's not a rascal." Gwen stamped her
foot and grew very red.

"Oh yes, he is, my dear. To propose to one girl and to make love to
another is not right. I must inquire into his character, you know, so
as to see if he is a decent man to know. Now Mademoiselle Zara can
tell us the truth. But I don't want you to be present."

"But I shall!" cried Miss Evans, with another stamp. "It is my right to
be present. The explanation concerns me more than any one else."

"Oh, well, if you insist upon being present, I have no more to say."
Mrs. Perage shrugged her shoulders, and making a wilful mistake.
"Did you say 'present' or 'pleasant'?"

"Pleasant. You must be pleasant to Mademoiselle Zara, as, after all,

you do not care anything for your cousin."

"I do. All the same I am angry with him. I shall be present and be
pleasant just as I please. And now I shall take a walk in the park so
as to calm my nerves. I'm sure Owain has upset them enough." And
Gwen hastily departed, while Mrs. Perage chuckled more than ever.

"Fiery little Welsh temper she has," murmured the old lady. "I don't
envy Hench when he makes her his wife. Hum! So that's settled. Let
us hope good will come of the interview." She rubbed her nose.
"Gwen's a handful to manage, but by contradiction I fancy that I
have secured my own way."
Of course this was quite true, although Miss Evans, walking in the
park, was perfectly sure that she was acting contrary to Mrs.
Perage's wishes. By this time the girl was in a fine temper, ready to
quarrel with any one about anything. In fact she felt very much
inclined to fight for what she considered were her rights, so far as
concerned her cousin. In some queer way, Gwen arrived at the
conclusion that by saving her life Hench had given her some sort of
claim over him. Of course, she would never marry him; nothing
would ever induce her to marry such a faithless person. But she
intended to hint at her fantastic claim by ordering him to make Zara
his wife. Then, on further reflection, she did not like him to marry
the dancer, as she loved him herself. Still, as he was unworthy of her
love, perhaps it would be as well to allow him to carry out his
proposal to Madame Alpenny's daughter. He would certainly be
miserable, which would serve him right, as Zara was bound to be a
minx and a cat and several other disagreeable things. In this
incoherent way Miss Evans thought, while working off her anger as
best she could by walking at top speed up one path and down
another. She did not know whether to laugh or to cry, to rage or to
fret; all she did know was that everything seemed to be wrong, and
that the bottom had fallen out of creation.

When Gwen again ventured into the house, she found the drawing-
room tenanted by Mrs. Perage, her nephew, and two visitors. One of
these was a handsome, untidily dressed young fellow, who wore his
hair rather long after the manner of musicians; the other was a tall
girl, gaunt, striking-looking, with something of the gipsy in her
appearance. She wore a red velvet hat and a long red velvet mantle,
the violent hues of which harmonized well with her somewhat sallow
complexion and bold dark eyes. When Gwen entered, this girl was
laughing and showed a row of very white teeth, which added to her
handsome looks.

"Mademoiselle Zara, this is Miss Evans," said Mrs. Perage, rising to

make a rapid introduction. "Gwen, this is Madame Alpenny's
daughter, and Mr. Bracken, to whom she is engaged."
"Engaged?" Gwen started back and gasped. "But I don't
understand."

"Mademoiselle Zara will explain," said Mrs. Perage swiftly, and

collecting the two men with her eyes. "Mr. Bracken, I must show you
my garden, as I am sure you take an interest in flowers. Come with
me. You also, Jim, as you must go to Mrs. Bell's and bring Hench
here."
"I don't wish to see him," called out Gwen hurriedly, but Mrs. Perage
took no notice of the speech, as she had already conducted the two
men out of the room, leaving the two girls alone.

Gwen eyed Zara and Zara eyed Gwen with great curiosity, and used
their intuitions with so much skill that in two minutes each girl knew
all about the nature of the other girl. Miss Evans could not deny but
what the dancer was handsome enough to attract any one, even the
most fastidious, while Zara thought that Gwen was one of the most
charming young ladies she had ever seen.

"I'm sure he will be very happy with you," she said abruptly.

"Who?" asked Gwen, sitting down and getting ready to fence.

Zara laughed meaningly. "My dear, there is only one 'he' in the world
for you."

"So I thought, until I found him out," retorted Miss Evans sharply.

"Oh, I understand all about your finding him out. Mr. Vane gave me
a full description of my mother's meddling. But if you had waited to
hear what took place after your departure from the churchyard there
would have been no need for me to come down."

"I did not ask you to come down," said Gwen pointedly.

"You did not. Mrs. Perage did, however, as she was anxious for your
mistake to be corrected. I am anxious, also, else I would not have
troubled to take this long journey."

"Why did you undertake it, then?"

"Because I have the greatest respect for Mr. Hench."

"The greatest love, you mean."

"Indeed, I mean nothing of the sort," said Zara candidly. "I have no
more love for Mr. Hench than I have for that table. Didn't you hear
Mrs. Perage say that I was engaged to Mr. Bracken?"

"Yes! I suppose you are," admitted Gwen reluctantly. "But there is

always one who loves and one who is loved, you know."

"Heine, the German poet, said that, Miss Evans. I congratulate you
on the wide range of your reading. It shows that you are not narrow,
and not being narrow, I trust that you will do Mr. Hench justice."

"He proposed to you. I heard him say so myself."

"My dear," said the dancer, after the lenient fashion of an elder sister,
"Mr. Hench at that time would have proposed to any woman of
decent character and decent looks. Your Heine quotation implied
that although I did not love him, he loved me. There you are entirely
wrong. He admired me, certainly, but----"

"But he proposed to you," interrupted Miss Evans doggedly.

Zara's cheeks grew crimson and her voice became sharper. "We are
two women talking together," she said decisively. "Therefore, it is
useless for us to skirt about the bush as we would do with men. Mr.
Hench never loved me; he had no conception of love when he
proposed, and I told him so. Can't you understand how a lonely man
must wish for a home and a comrade, so that he may have some
centre in life? I used those very words to him. Mr. Bracken gives me
that true love which is more than admiration, which was all Mr.
Hench had to offer. He could not give me his heart because he did
not know that he possessed one. Since coming here he has made
the discovery that he has a heart and he has given it to you."

"Have you seen him; did he tell you so?"

It took Zara a moment or so to quell her rising anger, and she felt
inclined to shake this silly little girl who was not to be convinced by
common-sense explanations. "I have not seen Mr. Hench, nor if you
wish it will I see him."

"Oh, it's nothing to me," said Gwen with an air of finality.

"Then it ought to be. Mr. Vane told me what Mr. Hench told him."

"What is that?"

"You know quite well," retorted Zara tartly. "It is that Mr. Hench
loves you better than you deserve."

"How can you tell what I deserve?"

"I am only going by what I see of you now," said the dancer
patiently. "You really love Mr. Hench, and you are fighting against
your feelings, because you believe that he loves me, which is not the
case. As you can see that I am speaking the truth, it is unworthy of
you to speak as you do. Therefore, I say that Mr. Hench loves you
better than you deserve. I don't know," cried Zara, becoming
exasperated, "why you force me to make so unnecessary an
explanation, as you are quite aware of what I mean."

Gwen was so impressed by the dancer's earnest speech that she

became much more reasonable. "I am a pig, I know," she murmured
rather inelegantly. "But it isn't pleasant to love a man and then to
hear from his own lips that he proposed to another woman."

"Pooh! You are making a mountain out of a molehill," said Zara

contemptuously. "If Mr. Hench had proposed to me after he met you,
then there might be some sense in your attitude. But I tell you he
did not know the meaning of love when he proposed to me, and
would have proposed to any other woman just as readily. His first
acquaintance with love was when he saved your life. He is heart and
soul devoted to you. My dear"--Zara rose, and bending over Gwen,
took her hand--"don't be foolish and throw away a love which will
make you the happiest woman in the world."
"Can you swear that Owain loves me?" asked Gwen, more and more
impressed.

"Personally, I cannot. But from what Mr. Vane has told me I certainly
can declare that Mr. Hench adores you."

"Yes." Miss Evans stared hard at nothing. "I believe he does."

"Then why are you making all this trouble?"

"You are a woman and ask me that?"

Zara laughed. "It is absurd, I know. But I am anxious to put things

right. My mother made trouble and I came down to make peace.
Don't send me away with my errand unaccomplished."

Gwen jumped up and kissed the dancer. "No, I won't. I am quite

satisfied with your explanation. I have been very silly and have
made myself quite ill in worrying over things. And if Owain comes----
"

"Owain is coming," interrupted Zara quickly, as she glanced out of

the open French window of the room. "Yonder he is with Mr. Vane,
who was sent to bring him by Mrs. Perage. My dear"--she kissed
Gwen's cheek--"I will slip out to join Mrs. Perage and Ned in the
garden. You stay here and make it up with Mr. Hench. No half-
measures, mind. Be generous and loyal." And with a smiling nod the
dancer flitted through the window just as the footsteps of Owain
were heard in the hall.

"Oh!" said Gwen, drawing a long breath, "how nearly I have lost
him."

Vane had sense enough not to enter along with his friend, as he
thoroughly understood the saying about two being company and
three none. In a most loyal fashion he obliterated himself, and
Owain walked into the room by himself. The young man looked worn
and ill, so that Gwen's heart was touched, and she felt ashamed of
her conduct, which was responsible for his wilted appearance.
Almost without thought she flew into his arms.

"I'm a horrid creature," she murmured. "Do forgive me and I'll be

good."

"Oh!"--Owain's pale face flushed suddenly and his brown eyes

sparkled--"then you don't believe----"

"I believe that you love me. Mademoiselle Zara has explained
everything."

"Thank God for that. Where is she?"

"Do you wish to see her?" asked Miss Evans jealously.

"Only to thank her. But that can come later. Meantime"--he bent and
kissed her three or four times--"oh, Gwen, how could you think that
I loved any one in the world but you--you--you?"

"I was silly and wicked and--and----"

"No! No! There was some cause for your anger, as Madame Alpenny
told so skilful a lie. It wasn't all a lie, of course, as I did propose to
Zara."

"I know you did, and I know why you did. But you will be much
happier with me than with her," said the girl naïvely.

"Than with any one, Gwen," cried the young man fervently. "Oh, my
dear, to think how nearly I have lost you."

"I said that to myself about you, just before you entered," whispered
Gwen in a penitent tone. "Do forgive me."

"On condition that you forgive me," pleaded Owain fondly.

"Dear, there is nothing to forgive," said the girl, abasing herself. "It is
all my fault--all my fault. I'm a nasty little jealous animal."

"Just the kind of animal I like." Owain pressed her hard in his arms.
"I'll never, never let you go again, and now that we are together and
you are on my side, I am prepared to face the worst."

"Face what?"

"Ah, I forgot; you don't understand. I have a long explanation to

give." Hench paused and looked nervous, as he drew Gwen to a
chair and sat down to take her on his knee. "You won't hate me, or
doubt me?"

"Never! Never!" Gwen positively. "I'll never doubt you again. What is
the matter?"

"Murder is the matter!"

"What?" She started back and stared at his perturbed face. "The
murder of----"

"Yes! The murder of your father. You know that tramp you suspect?"

"The one who asked the way to the Gipsy Stile? Yes."

"I am that tramp."

"It's impossible."

"It is quite true. I have explained matters to Vane and to Mrs.

Perage. Now I must explain them to you. Having admitted that I am
the tramp you suspect----"

Gwen stopped him by laying her hand over his mouth. "I don't
suspect the tramp, now that you are he," she said vehemently. "You
are innocent, I am sure."
"How can you be sure?" asked Hench sharply. "Because you saved
my life," replied Gwen in a truly feminine fashion. "No one who
saved a person's life would commit a murder."

"Well, I can scarcely admit the logic of that reasoning," said Hench,
unable to refrain from a smile, in spite of the desperate situation.
"But I am glad that you so far trust me."

"I trust you to the death."

"Darling!"--he kissed her--"that gives me the courage to tell you all!"

And he did tell her all then and there, from the time of the
conversation with Madame Alpenny down to the moment when she
accused him in the churchyard. "So you see, Gwen," he concluded in
a melancholy tone, "that although perfectly innocent, this woman
has the power to have me arrested."

"You shall not be arrested," said Gwen, with sparkling eyes and red
cheeks.

"Then you don't believe me to be guilty?"

"What a silly question to ask." This time it was Gwen who kissed. "Is
it likely that I would still be sitting on your knee if I thought you
killed my father? Of course, the whole thing is difficult and
mysterious, but I am on your side, Owain, and we will fight it out
together."

"Yes! Yes!" Hench rose and swung her off her feet right into his
arms. "I am not afraid now. Your love will give me strength to
conquer my enemies. But it will be an ordeal for you."

"An ordeal which will prove the depth of my love, dear. And I
deserve such an ordeal. I doubted you once; but I'll never, never,
never, never doubt you again. Owain, darling, everything will come
right. There is Mr. Vane and Mrs. Perage and myself and you.
Against us is only that horrid old woman."
"She holds a strong hand in the game, though," murmured the
young man doubtfully. "We hold a stronger. Right will always prevail
against might."

"Gwen! Gwen! You are a tower of strength. You put new life into me.
Yes, we will fight; we will fight, fight to the end."

"And win!" cried Gwen. "Oh, never doubt, Owain. We must win!"

CHAPTER XVII

BLACKMAIL

After the reconciliation between the lovers nothing remained but to

go into the garden and announce that Mademoiselle Zara's errand
had been wholly successful. Gwen was now quite amiably disposed
towards her rival, and was indeed very thankful to her for the
peacemaking explanation. Along with Hench she went into the hot
sunshine, and as they walked across the lawns towards the glade
where they were likely to find the others, Owain warned Gwen that
Zara was wholly ignorant of her mother's schemes. "Only you and I,
Mrs. Perage and Jim Vane, know about her accusation," said the
young man seriously. "So don't hint a word of the business to Zara."

"Of course I won't," agreed Gwen readily. "But what steps are you
going to take, Owain, in order to counterplot her?"

"Madame Alpenny? Well, I haven't any idea in my head just now,

and, at all events, she has given me a week to think over things. Let
us leave matters as they are until to-morrow, and then we can call a
council of war and see what is best to be done. There's no doubt
that Madame Alpenny has me in a tight place."

"She has," said Gwen cheerfully. "But we may be able to turn the
tables on her."

"In what way?"

"I don't know," mused the girl. "It seems to me that this woman
knows more about the death of my father than she will admit. She
may be guilty herself."

Hench shook his head. "I have some such idea myself, and yet it
seems impossible. What had she to gain?"

"A fortune through you," said Gwen promptly. "By means of that
advertisement which brought you to the Gipsy Stile, she implicated
you in the murder, which she may have executed before you arrived.
Once under her thumb, she hoped to compel you to marry Zara, and
so would have gained control of the money."

"I am not under her thumb yet," said Hench grimly. "And what is
more, I don't intend to be, strong as is her position. Whether she is
guilty or innocent I can't say, as I am ignorant of her doings on the
night of the first of July. But I should like to know, Gwen, why your
father put that advertisement into the papers, and why he appointed
the Gipsy Stile as the place of meeting?"

"I can't explain," she answered doubtfully. "My father never said a
word to me about the advertisement, or, indeed, about Madame
Alpenny's visit. I asked him who she was and he told me to mind my
own business."

"Well, Madame Alpenny can explain, as I believe she suggested the

advertisement dodge herself." Owain reflected for a moment.
"There's something queer behind all this, Gwen, and when we learn
what that something is, I daresay we will find out who murdered
your father. And then----"

"Hush," said Gwen suddenly, as they turned round the corner of a

green alley which ran between high box hedges. "Here they are."

As a matter of fact the lovers stumbled right into the centre of a

group consisting of Mrs. Perage and her guests. They all appeared to
be smiling, and the smiles grew very broad when the reconciled
couple came towards them. Mrs. Perage caught Gwen by the
shoulders and looked into her tell-tale blue eyes.

"Is it all right, you nuisance?" she demanded gruffly.

"All right!" assented Gwen, giving her a kiss. "Thanks to----"

"To me," cried the dancer gaily. "I am the goddess of Peace."

Hench took her hand and kissed it. "I can never thank you
sufficiently."

"I don't require thanks, Mr. Hench. But did I not tell you that when
you really fell in love you would understand how wholly different it
was to your feeling for me?"

"You did, and I have learned the difference. Admiration is moonlight,

and love is the most glowing of sunshine."

"How poetical," said Vane with a shrug.

"And how true. Jim, I have to thank you for bringing Mademoiselle
Zara with the olive branch. Bless you, as a friend in need."

"Bless Aunt Emma, rather, old son. She suggested the idea."

"It seemed the only way of convincing a stupid man," said Mrs.
Perage lightly. "However, all's well that ends well, so let us go in and
have some tea. Our visitors have to leave in an hour."
All this time Bracken, silent according to custom, was smiling
amiably at the man he had at one time considered his rival. Now he
advanced and shook him by the hand, much to the approval of Zara,
for Bracken had given her considerable trouble over Hench's
attentions. Mrs. Perage, still holding on tightly to Gwen, was walking
in front, together with Vane, so Owain had the pleasant task of
escorting Zara and her lover to the house. He was glad of this, as he
wished to say something and repay the dancer for her kindness.

"When are you two going to be married?" he asked abruptly.

Zara sighed. "I don't know," she confessed sadly. "Ned expected to
get some money from his mother, but she died without leaving any.
Neither I nor Ned make enough money to keep ourselves and my
mother, so we can't think of marrying for a long time."

"Madame Alpenny seems to be the stumbling block," mused Hench

thoughtfully.

"She is," declared Bracken in a gruff, rough way. "Zara and I could
manage by ourselves on what we earn, if it wasn't for that cattish
old woman."

"Ned! Ned! Don't call names. After all, my mother is my mother."

"She is very selfish, and makes you miserable to please herself," said
Bracken crossly. "I shall never make much money as I am not a
genius as you are, Zara. If you could only get the engagement you
deserve you would make sufficient to settle your mother, and then
we could get married."

"Allow me to see to that," said Owain quickly. "See here, Bracken,

and you, Zara, you may not know it but I am a rich man."

"I am very glad," said the dancer honestly. "You have made money,
then?"
"I have inherited money--a large income. I owe you much, as but for
you things would not have been squared."

"It was the least I could do, Mr. Hench."

"It was a very great deal to do, as the task was a delicate one.
However, what I mean is this, that as you have been my friend you
must allow me to be yours. Therefore"--Owain spoke slowly and
deliberately--"I wish you, with Bracken's approval, of course, to
accept one thousand pounds."

"Oh!" gasped Zara, flushing as red as her cloak. "I couldn't think of
it."

"Nor can I," said Bracken resentfully. "I can keep my own wife."

"My dear people,"--Owain being between them took an arm of

each,--"if you like you can pay me back on some future occasion.
Zara, your mother will bother me to marry you until some barrier is
raised which will prevent your being my possible wife. At present, as
you have stated, you are not able to marry for want of money. Now
if I give you this thousand pounds, which I can very easily spare, I
want you to get married quietly. When your mother learns that you
are Mrs. Bracken she will leave me alone. Then you can give her a
sum of money to live on in the meantime and will be able to rest on
your oars and look about for a better engagement. You see?"

"Yes," said Zara gratefully. "I see, and I am very much obliged. If I
can give my mother half the money she will go to her people in
Buda Pesth and amuse herself with gambling. Then with five
hundred pounds Ned and and I can manage to get to the West End.
Money always brings money, and I am sure that I could get an
engagement."

"Didn't your mother go in search of one for you?" asked Hench,

nodding.
Zara's lip curled and she looked more disdainful than ever. "My
mother said that she went, but she never did."

Hench started. "She was absent for a few days, I remember."

"Yes. On business, she told me. But what her business was I never
knew. It had nothing to do with an engagement, however, or I
should have known."

Of course Owain knew very well on what business Madame Alpenny

had been engaged, but he was wise enough to make no remark.
Also at the moment his attention was distracted by Bracken, who
had been thinking in his heavy way.

"If you will allow Zara and me to pay you back the money with
interest at five per cent," he observed, reflectively, "we don't mind--
eh, Zara?"

"No," she rejoined promptly. "I shall take the money with pleasure
then, as it will certainly help us to get married in spite of my
mother's opposition. I am very grateful for your kind help, Mr.
Hench."

"I am only doing what I ought to do," said Owain frankly. "You have
done me a good turn, so it is only right that I should do you and
Bracken one. I shall see my lawyers next week and arrange for the
money to be paid to you by cheque, or in notes, or gold, whichever
you prefer."

"Say a cheque, Hench," remarked Bracken, with a sigh of relief. "I

have a banking account. It's a very small one--still, it is a banking
account."

"Good. I will call at The Home of the Muses some day next week
with the cheque, and meantime you can see about getting married."

"Oh, Ned!" cried Zara.

"Oh, Zara!" cried Ned, and they embraced, even though they were in
sight of the drawing-room windows.

"Well," said Hench philosophically, "I have made two people happy,
anyhow."

"We will be happier if you are happy yourself, you generous man,"
said Zara.

"Oh, that's all right," replied Hench hurriedly, for he did not wish to
be thanked or praised. "Come and have some tea. We'll keep this
little arrangement to ourselves."

The visitors were very pleased at the result of their visit, which they
had been far from expecting, and the tea was unusually gay. Gwen
could not show enough attention to Zara, and Mrs. Perage, who had
taken a fancy to the honest dullness of Ned, looked after him in her
brusque way. Owain and his beloved were silent from sheer
happiness, in spite of the thunder-clouds which still obscured the
sun, so it was left to Jim Vane to brighten the party with chatter and
gaiety. He was entirely successful, and the visitors left with a sense
of great enjoyment. Zara looked younger, less fatigued and
unapproachable than usual, while Bracken's stolid good-looking face
was wreathed in smiles. And Hench saw them off at the station with
a sense of thankfulness that he had been able to help them. He was
so happy himself in having gained Gwen's love that he wished every
one else to be happy, and moreover was delighted that he had been
able to repay Zara for her good work. He returned to his lodgings to
dress, and then went to dine at Mrs. Perage's hospitable board.

Gwen wished to hold the council of war after dinner, but Hench
refused. He considered that the day had been quite sufficiently filled
with events, and did not wish to start a discussion which was likely
to be prolonged into the small hours. Gwen looked tired after all the
excitement she had undergone, and Hench himself felt rather weary.
The true fact was that a sense of anxiety lay beneath their surface
gaiety, and they were feeling the suspense more than they thought.
Mrs. Perage and her nephew were also rather silent; so in spite of
the reconciliation of the lovers the evening was rather a failure. With
her usual prompt way of dealing with things, Mrs. Perage sent Hench
away at half-past nine o'clock.

"We are all worn out with bother," she said briskly. "So it is best for
all of us to have a good night's rest and then we can deal with other
and more serious matters to-morrow."

"One serious matter has been put right, thanks to you," said Hench,
looking fondly at Gwen. "It was just as well to take the bull by the
horns," said Mrs. Perage candidly. "And I am glad that Zara proved
to be so sensible a creature. And when you tell Gwen what--what----
" she hesitated, not knowing if it was wise to speak.

"What peril I am in," finished Hench. "Oh, I've done that this
afternoon."

"The deuce you have!" cried Vane, turning from his friend to Gwen.
"And what do you think of the matter, Miss Evans?"

"I don't know what to think," said Gwen promptly. "Save that I
believe Owain to be innocent, and I will stand by him to the end,
whatever it may be."

"Good. And the accusation of Madame----"

"Jim," commanded his aunt sharply, "do hold your tongue. This is
not the time to begin a discussion. To-morrow, when our wits are
clearer, we can talk. Owain, go home to bed. Jim and I will turn our
backs while you take leave of Gwen."

This was not necessary, as Gwen accompanied her lover to the door
and kisses were exchanged in the twilight of the summer night. But
the two were so long in parting that Mrs. Perage had to come on the
scene and fairly shut the door in the face of this lingering lover.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebooknice.com