How To Use SQL Servers Extended Events and Notifications White Paper 24902
How To Use SQL Servers Extended Events and Notifications White Paper 24902
MONITORING YOUR SQL SERVER ENVIRONMENT notifications and extended events—that can help you achieve
SQL Server comes with a wide array of tools for monitoring your these goals. With these two features, we’ll look at the error log
and deadlocks, and demonstrate how you can get relevant
environment. There are logs and traces that provide information
information delivered as it occurs. We’ll also look at ways that run-
when errors occur, but these are often used passively to react to
time errors can be captured and used to help reduce the amount
events that have already occurred. As DBAs, we need to monitor
of time required to investigate issues.
our environments proactively and create solutions as issues arise.
In this white paper, we will look at a couple technologies—event
SQL Server’s event
notifications and
extended events provide
a method to garner THE ROLE OF THE DBA ways to accomplish the monitoring that
is required. Many of these ways, such as
timely and detailed One of the core duties for all DBAs is to
monitor the environments for which they SQL Profiler and Performance Monitor,
information on issues. are responsible. This includes tasks that can provide a lot of general information
when problems arise, but they are
range from designing and deploying
often unable to provide the timely and
databases to managing database backups.
detailed information that is necessary to
We must also monitor performance and
immediately begin troubleshooting issues.
troubleshoot problems as they arise.
In this white paper, we will look at two
These job functions require us to obtain
technologies that are already available
timely information on issues that we deal
within SQL Server 2005 and SQL Server
with, and that information must be detailed
2008: event notifications and extended
enough so it can be acted upon to resolve
events. Each of these provides a method
issues effectively and efficiently.
to garner timely and detailed information
Throughout the last few years, the growth on issues. We will also look at some
in the number of databases that SQL common monitoring the DBAs should be
Server DBAs must manage has outpaced doing, and how we can automate and
the increase in DBAs to manage those delve deeper into that monitoring. These
databases. We must begin leveraging solutions will provide a better method
what we have available tomanage the of leveraging SQL Server to accomplish
hundreds and thousands of databases what’s needed to help us stay on top of
we have now, instead of the dozens of our SQL Server environments.
databases that we worked with a few
years back. To succeed, we must learn
to do more with less and improve our
capacity to manage and monitor our SQL
Server environments. There are countless
2
USING SQL SERVER FEATURES data available to monitor. With event
notifications, the data is available as part of
Two SQL Server features will be discussed
the feature. The DBA can concentrate on
in this white paper: event notifications and
how to handle the information, instead of
extended events. Both of these features
how to make it available.
are already a part of SQL Server and can
be used in your environment today. With Using event notifications to process and
just a little configuration, you can raise the handle information from the DDL trigger
bar on your monitoring and delve deeper and SQL Trace events that are exposed
into the problems that you face. allows these events to be monitored in
ways that improve the ability to scale the
AN INTRODUCTION TO EVENT
DBA across the SQL Server environment.
NOTIFICATIONS
Event notifications is an event-triggering AN INTRODUCTION TO
platform that was introduced with SQL EXTENDED EVENTS
Server 2005. It utilizes events based on Extended events are a lightweight
data-definition data (DDL) statements and event-handling platform introduced
SQL Trace events to feed event data into in SQL Server 2008. This platform
Service Broker tables. Through the use captures events, such as wait events,
of Service Broker, the data from these user errors, and deadlocks, as they occur.
Event notifications allow
events can be processed in whatever
manner is needed based on the event.
After the events have been triggered, events to be monitored
additional information that may be
For instance, the event data can be useful in troubleshooting the event can in ways that improve a
inserted into monitoring tables or sent to
DBAs via email.
be appended to it before it is sent to a
destination, or target, that can be easily
DBA’s ability to scale
Event notifications differ from DDL triggers queried. Extended events are very similar across the SQL
to what is available through SQL Profiler
in that they execute asynchronously. Due
or SQL Trace. In fact, it is intended as a Server environment.
to this difference, event notifications
replacement for both technologies. With
can be triggered without affecting the
the next release of SQL Server, code-
transaction that generated the event. This
named “Denali,” SQL Profiler will be
benefit is realized in two ways: First, this
deprecated for the SQL Server engine.
removes the risk that DDL-centric event
notifications will cause the triggering A key benefit to extended events is its
transactions to roll back. Second, any performance when compared to other
performance or resource costs associated monitoring options. With SQL Trace,
with the DDL trigger will be offloaded from there is often the concern that SQL Trace
the triggering transaction. sessions can add to a performance issue.
Extended events were designed with
When looking at SQL Trace events, made
performance in mind, and tests have
available through event notifications, the
shown that the impact of extended events
platform allows the DBA to perform actions
on transactions is about two microseconds
on SQL Trace events and process them,
per triggered event. Additionally, while
instead of just storing the information in a
events are triggered synchronously,
file or table. With SQL Trace and profiler,
many destinations for extended events
these events could be collected, but
are populated asynchronously, further
there wasn’t a method to process the
reducing the likelihood that the monitoring
data. The bulk of the effort was spent
will have an impact on performance.
building a process to make the SQL Trace
3
In order to be as flexible as possible, DEADLOCKS
extended events were designed so that a Deadlocks are an unfortunate reality in
triggering event can be sent to any of the many environments. As DBAs, we are not
available destinations. This means that necessarily responsible for the occurrence
any destination, or target, can receive any of the deadlocks, but we are responsible for
single event or multiple events without detecting and attempting to prevent them
needing special configuration to handle from reoccurring. What do we do as DBAs to
the various events. Also, any group monitor for and prevent deadlocks?
of events can be bundled together in
extended event sessions. If the properties The common scenario that we face goes
of an event change over time, the session a little something like this. You, the DBA,
information is as valid with the first version are sitting at your desk minding your
of the event collected as it is with the own business. In walks the developer or
future versions. business user, asking about their failed
transaction. Unless you are actively looking
Extended events provide a depth of at transactions on the server, you may not
information about events that far exceeds be able to help the developer or business
what is available through SQL Trace, user at all. This time, let’s pretend that they
SQL Profiler, Performance Monitor, or
A key benefit of many of the DMVs within SQL Server.
brought the error message and it mentions
that there was a deadlock. If you have the
extended events is These other tools can provide high-level
information on page splits, latches, and
deadlock trace flags turned on, you could
open the error log and look for information
performance, when wait stats. Only extended events can look on the deadlock. While it is useful to enable
at when these events occur and provide
compared to other information such as the plan handle,
the trace flags, which places the deadlocks
graph in the error log, it isn’t very useful if
monitoring options. T-SQL stack, and much more, as they are you aren’t actively looking for deadlocks.
triggered within the database. Also, reading the deadlock graph through
the error log is a bit of a pain.
Extended events provide a high-
performance doorway to look into SQL In a variation on this scenario, suppose
Server to monitor and provide valuable you’re a proactive DBA who actively
troubleshooting information. Most of this monitors performance counters. You
information is available in other tools and may be monitoring the perfmon counter
features, but not in a manner that performs deadlocks/sec. so that an alert will be
as well as extended events. received whenever a deadlock occurs.
This is a valid method for monitoring
STARTER SOLUTIONS
deadlocks because it tells you how many
In this paper, we’ll focus on a few deadlocks are occurring and the scope
real-world scenarios for using event of the problem. But it does nothing about
notifications and extended events. For what transactions have deadlocked. Again,
each of these scenarios, we’ll walk you would need to have the trace flags in
through a method for addressing each of place to push deadlocks into the error log;
them using both event notifications and which brings you back to the issues with
extended events. The scenarios we will the first scenario.
cover include:
Both of these scenarios for monitoring and
• Deadlocks resolving deadlocks lack one key element:
• Blocking These methods do not provide timely
information on the deadlocks. When the
• Error messages deadlocks occur, you aren’t prompted to
act or don’t have the information on hand
For each of these scenarios, we’ll go
necessary to act, and must look to other
through ways that you can apply event
means to provide this.
notifications and extended events to the
monitoring that you need to accomplish.
4
RESOLVING DEADLOCKS WITH any case, you will need to process the
EVENT NOTIFICATIONS items in the service broker queue.
Fortunately, event notifications provide rich This is where the power of service broker
and timely information on deadlocks. The comes into play. With performance
deadlock event can occur, and a process counter alerts or trace flags, there isn’t any
can be built to automatically research method to transform the information as
and provide information on the deadlock it is collected into something timely and
shortly after it occurs. useful. Through service broker, a stored
procedure can be written that responds
There are a few steps involved in
to deadlock events. Event notifications
configuring deadlock monitoring through
allow deadlock graphs to be transformed,
event notifications:
stored, and sent wherever they
1. Enable service broker on the database. need to go.
2. Create a service broker queue to receive To illustrate some of the potential, let’s say
the event notification messages. we want to do the following whenever a
3. Create a service broker service to deliver deadlock is encountered:
event notification messages.
1. Store the deadlock graph in a table. Event notifications
4. Create a service broker route to route the
event notification message to the service
2. Retrieve the cached plans associated with
the deadlock in another table.
provide rich and
broker queue.
3. Email the deadlock graph to the
timely information
5. Create event notification on deadlock
event to create messages and send them
DBA team. on deadlocks.
to the service broker service. All of these can be easily accomplished
These steps are implemented through the through a stored procedure on the event
following script: that was created. To start, we’ll need
a couple tables to store the deadlock
Once the event notification is created, graphs and the cached plans.
deadlock event notifications will start
to populate the service broker queue. Once the tables are created to store the
Depending on your system, these event notification messages, the stored
messages could be rare or frequent. In procedure can be created. Since we are
working with service broker, the stored
5
Event notifications allow
deadlock graphs to be
transformed, stored,
and sent wherever they
need to go.
procedure will be a bit cumbersome See the code below for accomplishing
since it is based on a stylized template these steps. One item to note: at the end
for retrieving messages from service of the stored procedure is the addition of
broker queues. a signature to the stored procedure. This
signature is required to allow SQL Server
The stored procedure is going to go to have the proper permissions to use
through a few steps for each of the sp_send_dbmail in the stored procedure.
deadlock notification messages. These More about signing and the certificate
steps are: can be found in the appendix of this
white paper.
1. Retrieve deadlock message from
DeadlockNotificationQueue queue.
6
Every time the stored
procedure is changed,
the stored procedure
will need to be signed.
7
One piece to mention in the stored At this point, the event notification
procedure above is the inclusion of queues and the stored procedure for
a signature on the stored procedure. processing the deadlock messages
This provides the stored procedure havebeen created. The last step is to
permission to execute sp_send_mail assign the stored procedure to the service
without enabling the TRUSTWORTHY broker queue for the deadlock events.
database option. Every time the stored This is completed by altering the queue
procedure is changed, the stored to assign the stored procedure to the
procedure will need to be signed. queue. By doing this, when deadlock
Additional information on signing stored event messages are received, the stored
procedures is found in the Appendix. procedure will get called to process them
until there are no more messages.
In situations where
you haven’t set up
deadlock monitoring,
information necessary
to troubleshoot recent
deadlocks is already Enabling the capture of deadlocks with and started. One of the events that this
being stored. event notifications takes a bit more to set session captures is the deadlock graph.
up than enabling a trace flag. But in the
end, doing this will allow you to be alerted Without any setup or configuration,
to deadlock events as they happen, deadlocks in the system_health session
providing the opportunity to troubleshoot can be accessed through the use of the
issues immediately, rather than when end query below.
users start to notice them.
Running this query returns the deadlock
RESOLVING DEADLOCKS WITH graph and the time in which the deadlock
EXTENDED EVENTS occurred. These can be used to start
digging into the causes of the deadlock.
Another way to monitor deadlocks is In situations where you haven’t set
through extended events. In SQL Server up deadlock monitoring, information
2008, an extended event session named necessary to troubleshoot recent
system_health is automatically created deadlocks is already being stored.
8
Figure 1. Extended events results from system health
If your SQL Server environment contains This solution can be found in Jonathan
SQL Server with versions lower than SQL Kehayias’ article Retrieving Deadlock
Server 2008 Service Pack 1 Cumulative Graphs with SQL Server 2008 Extended
Update Package 6 or SQL Server 2008 Events. Also, more information on parsing
R2 Cumulative Update Package 1, you will the deadlock graph is available from
encounter a bug converting the deadlock Michael Zilberstein in he post Parsing
graphs above into XML. The issue is Extended Events xml_deadlock_report.
detailed in the Knowledge Base article The key is to determine
978629. To resolve the issue, use the
query below instead: what is necessary
for your applications
and environment to
make certain that
blocking isn’t leading to
performance issues and
application timeouts.
9
To reconfigure the blocked process threshold, run the following:
As with deadlock monitoring, once the threshold within SQL Server to enable the
service broker queue is created, blocked blocked process report.
process information will begin to be
collected. There are a few more steps For the processing of blocked process
that need to happen before we will be reports, the procedure will be similar to
able to start collecting blocked process the deadlock monitoring process. We’ll
reports. Namely, we need to build a stored want a table blocked process reports, for
procedure to process the blocked process historical purposes.
reports and configure the blocked process
10
Next, we want to build a stored procedure The stored procedure will have a similar
that can do the following actions to each layout to the one for the deadlock
blocked process report message that is monitoring. This is mainly because stored
received through event notifications: procedures for service broker follow a
stylized format.
1. Store the blocked process report in
a table.
11
With this created, the procedure can be added to the blocked process report service
broker queue. Adding this can be done with the following SQL statement:
At this point, the only thing left to seconds a blocked process will wait
configure is the threshold for creating before creating a report. Configuring the
blocked process reports. This is done as blocked process report to be collected
a configuration value at the SQL Server after five seconds is done with the
instance level. The value provided to statements below:
the configuration determines how many
12
When blocking occurs, the information can be extracted from the session with the
following query:
Another advantage to
using extended events
is the opportunity to
use multiple targets and
collect information
over time.
An example output from this would look like the results shown below.
In the sample output above, a single lock capped. If the actions included are not
occurred. The event had approximately enough, more details can be added, such
18K milliseconds of waiting. The locking as session id, host name, application
was on an IU or intent to update. Added to name, etc.
that are the actions for sql_text and tsql_
stack. The sql_text provides information Another advantage to using extended
on the SQL that was submitted in the events is the opportunity to use multiple
transaction. Then the tsql_stack provides targets and collect information over time.
the stack of calls leading to the statement For example, instead of the ring_buffer
that encountered the blocking. target, the sessions could be configured
to use the asynchronous_bucketizer or
This information could probably be synchronous_bucketizer target. With
collected and reported on in other ways. those targets, the transactions causing
What makes this different from using blocking can be grouped together and
event notifications is that it provides a lot blocking patterns on specific statements
of control over the details returned. The can be seen. This flexibility allows
extended event session can be filtered for deep and meaningful blocking
to look at only certain databases, servers, troubleshooting with minimal effort on the
users, etc. After a specified number DBA’s part.
of events, the event collection can be
13
MONITORING ERRORS On top of both of these challenges is
the amount of clutter that shows up in
A key challenge we have as DBAs is
the error log. We get notices of backups
finding the time to adequately monitor
completed, DBCC messages, user
the SQL Server errors and error logs in
errors, applications errors, etc. Some
our environments. In an ideal world, a
have high-severity urgency and need
DBA would be on top of all errors in an
to be addressed right away. Others are
environment by reviewing error logs
informational and should be “seen but not
on a daily basis, making certain that
heard.” As we look through the error log,
nothing unexpected appears within them.
we need to ignore some of the statistics
Unfortunately, monitoring error logs doesn’t
to find the messages of real value—
scale well. As a DBA becomes responsible
otherwise we risk missing messages that
for more and more instances, the number
are important or even critical to address.
of logs to review increases. It doesn’t
take many instances before the DBA has MONITORING ERRORS WITH
to choose between reading error logs or EVENT NOTIFICATIONS
doing any other work.
Fortunately, event notifications offer a
Another part of this challenge is that solution to the dilemma of reading through
As we look through reading the error logs the day after issues
appeared is the wrong time to address
the error log. One of the events provided
in event notifications is the ERRORLOG
the error log, we need the issue. Whether the information in event. With this event notification,
the error log is about corruption or an messages sent to the error log can also
to ignore some of the account lockout, DBAs need to know be sent to a service broker queue for
statistics to find the about the issues that are happening further processing.
in real time. Along these same lines, if
messages of real value. there isn’t time to check all of the error The error log event notification can be set
logs on a daily basis, there certainly isn’t up through the following script:
time to continuously monitor error logs
throughout the day looking for new events.
14
As with the other event notifications, we’ll all error messages that are logged for
need to build a stored procedure to add error 825.
to the service broker queue to process
the error log messages. One downside to The error log store procedure will have the
this event notification is that all messages following steps:
sent to the error log will also go to the
1. Filter error log messages for those
error log service-broker queue. All of the
of value.
messages won’t be needed; otherwise,
the error log event notification would lose 2. Email the error log message to the
its usefulness as an alternative to picking DBA team.
through the error log.
The stored procedure will look like the
For simplicity, the stored procedure following:
will retain all error log messages with a
severity of 19 and higher. It will also retain
With an ERRORLOG
event notification,
messages sent to
the error log can also
be sent to a service
broker queue for
further processing.
15
The extended events
platform provides the
flexibility and depth
With this created, the procedure can be added to the error log service-broker queue.
needed to gather Adding this can be done with the following SQL statement:
information on errors in
SQL Server.
At this point, SQL Server will begin to log, such as the error message “could not
send emails when these errors occur find store procedure”:
in your environment, freeing you from Msg 2812, Level 16, State 62, Line 2
having to regularly review the error logs to Could not find stored procedure ‘asdfasdf’
detect errors. The email will include basic
information on the error to get you started While this and similar errors are part of
reviewing the issue. the applications that use SQL Server,
some applications lack the error handling
MONITORING ERRORS WITH necessary to properly communicate
EXTENDED EVENTS the issue to the end user or developer.
Sometimes, when the errors bubble up,
The errors that appear in the error
the context of the error message is lost
log aren’t the only ones you will care
along with important details regarding
about as a DBA. Although we aren’t
the error.
always the ones writing applications
in our environments, we do bear Extended events can provide a
the responsibility of knowing when lightweight tracing platform to collect
applications fail and why. For these errors as they occur, providing the DBA
responsibilities, the extended events with a chance to review recent error
platform provides the flexibility and depth messages and assist developers with
needed to gather information on errors in troubleshooting issues. It’s also a way to
SQL Server. In the context of errors, we keep a pulse on issues that could become
are talking about errors that users will larger in the future.
often see but that don’t end up in the error
16
For this example, we are going to create • sqlserver.session_id
an extended event session that captures
• sqlserver.sql_text
three error messages. The errors are:
208, 2812, and 4121. These translate to: • sqlserver.client_app_name
Invalid object name, Could not find stored
• sqlserver.client_hostname
procedure, and Cannot find either column
or the user-defined function or aggregate, • sqlserver.database_id
respectively. They can be captured
through the event sqlserver.error_ • sqlserver.username
reported (this is the event triggered any These details will provide the information
time an error is encountered). Along with needed to take an error and track it down
this event, the extended event session will to an application to fix the issues that are
capture the following details: being encountered. The extended event
session for this can be created with the
following script:
When errors have occurred that the event session is capturing, they can be read through
a query similar to this:
17
From here, a DBA can easily begin • Error messages: In the last scenario, we
tracking down issues and informing reviewed how event notifications can
developers of where errors are occurring, free the DBA from monitoring error logs
the frequency in which they are occurring, and receiving information on issues
as they happen, rather than when the
and what T-SQL statements are causing
DBA happens to review the error log.
them to occur. This example only As for extended events, the example
includes a few error messages, but it can demonstrated how extended events can
easily be expanded to cover many more be used to monitor for user errors that
messages, helping to resolve other error DBAs don’t often deal with.
messages you may encounter such as
truncation errors.
18
APPENDIX: DATABASE OLAP solutions, as well as assessment
MAIL CERTIFICATE and implementation of SQL Server
environments for best practices,
In this white paper, I referenced a
performance, and high-availability
certificate that’s needed for signing
solutions. Jason is a SQL Server MCITP
stored procedures. This certificate
who participated in the development
provides those stored procedures
of Microsoft Certification exams for
with the permissions necessary to use
SQL Server 2008.
sp_send_mail. The certificate used in
the demonstrations was created using He is actively involved with the local PASS
the script above. chapter (SQL Server User Group) and
currently serves as Director of Program
More information on using signed
Development. Over the
certifications can be found on
past year, Jason worked with the
MSDN in the article Tutorial: Signing
current board to organize the PASSMN
Stored Procedures with a Certificate.
SQL Summit 2009 in September for
Additionally, Jonathan Kehayias
the local community.
discusses using signed certificates to
execute sp_send_dbmail in the article Jason enjoys helping others in the
Using a Certificate Signed Stored SQL Server community, and does so
Procedure to Execute sp_send_dbmail. by speaking at technical conferences
ABOUT THE AUTHOR and user group meetings. Most recently,
Jason presented at the SSWUG Virtual
Jason Strate, Digineer Inc., is a database Conferences, TechFuse, numerous SQL
architect and administrator with more than Saturdays, and PASSMN user group
15 years of experience. He has been a meetings. A contributing author for the
recipient of the Microsoft Most Valuable Microsoft white paper “Empowering
Professional (MVP) for SQL Server since Enterprise Solutions with SQL Server
July 2009. 2008 Enterprise Edition,” Jason is also an
active blogger with a focus on SQL Server
His experience includes design and and related technologies.
implementation of both OLTP and
19
ABOUT QUEST
Quest helps our customers reduce tedious administration tasks so they can focus on the innovation necessary for their businesses to
grow. Quest® solutions are scalable, affordable and simple-to-use, and they deliver unmatched efficiency and productivity. Combined
with Quest’s invitation to the global community to be a part of its innovation, as well as our firm commitment to ensuring customer
satisfaction, Quest will continue to accelerate the delivery of the most comprehensive solutions for Azure cloud management, SaaS,
security, workforce mobility and data-driven insight.
This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software
license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable
agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including
photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest
Software Inc.
The information in this document is provided in connection with Quest Software products. No license, express or implied, by estoppel
or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest Software products.
EXCEPT AS SET FORTH IN THE TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT,
QUEST SOFTWARE ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY
RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST SOFTWARE BE LIABLE FOR ANY DIRECT,
INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES
FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE
THIS DOCUMENT, EVEN IF QUEST SOFTWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest Software
makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves
the right to make changes to specifications and product descriptions at any time without notice. Quest Software does not make any
commitment to update the information contained in this document.
Patents
Quest Software is proud of our advanced technology. Patents and pending patents may apply to this product. For the most current
information about applicable patents for this product, please visit our website at www.quest.com/legal.
Trademarks
Quest and the Quest logo are trademarks and registered trademarks of Quest Software Inc. in the U.S.A. and other countries. For
a complete list of Quest Software trademarks, please visit our website at www.quest.com/legal. All other trademarks, servicemarks,
registered trademarks, and registered servicemarks are the property of their respective owners.
20
Whitepaper-SQL-ExtEvtAndNotif-US-EC-25291