MICROSOFT® SQL SERVER™

TPC-E BENCHMARK KIT REFERENCE

Version 1.14.0-1041
© Copyright Microsoft, 2008 - 2018

Jamie Reding ([email protected])

SQL Server Performance Team
Microsoft Corporation
END-USER LICENSE AGREEMENT FOR MICROSOFT TPC-E BENCHMARK KIT
IMPORTANT READ CAREFULLY: This Microsoft End-User License Agreement (EULA) is a legal agreement between you (either an
individual or a single entity) and Microsoft Corporation for the Microsoft software product identified above, which includes
computer software and may include associated media, printed materials, and online or electronic documentation (SOFTWARE
PRODUCT). By installing, copying, or otherwise using the SOFTWARE PRODUCT, you agree to be bound by the terms of this
EULA. If you do not agree to the terms of this Agreement, you are not authorized to use the SOFTWARE PRODUCT.
The SOFTWARE PRODUCT is protected by copyright laws and international copyright treaties, as well as other intellectual
property laws and treaties. The SOFTWARE PRODUCT is licensed, not sold.
1. GRANT OF LICENSE. This EULA grants you the following rights:
Use. Microsoft grants to you the right to install and use copies of the SOFTWARE PRODUCT only in conjunction with validly
licensed copies of Microsoft SQL Server and/or Microsoft Windows NT Server software. You may also make copies of the
SOFTWARE PRODUCT for backup and archival purposes.
2. RESTRICTIONS.
--You must maintain all copyright notices on all copies of the SOFTWARE PRODUCT.
--You may not distribute copies of the SOFTWARE PRODUCT to third parties.
--You may not rent, lease or lend the SOFTWARE PRODUCT.
--You may not use the SOFTWARE PRODUCT or any derivative works thereof to internally test database management system
software other than Microsoft SQL Server and/or operating system software other than Microsoft Windows NT.
-- You may not disclose the results of any benchmark tests using the SOFTWARE PRODUCT to any third party without
Microsoft’s prior written approval.
-- You may not disclose or provide the SOFTWARE PRODUCT or any derivative works thereof, or any information relating to
the SOFTWARE PRODUCT (including the existence of the SOFTWARE PRODUCT or the results of use and testing or benchmark
testing), to any third party without Microsoft’s written permission.
3. TERMINATION. Without prejudice to any other rights, Microsoft may terminate this EULA if you fail to comply with the terms
and conditions of this EULA. In such event, you must destroy all copies of the SOFTWARE PRODUCT.
4. COPYRIGHT. All title and copyrights in and to the SOFTWARE PRODUCT and any copies thereof are owned by Microsoft or its
suppliers. All title and intellectual property rights in and to the content which may be accessed through use of the SOFTWARE
PRODUCT is the property of the respective content owner and may be protected by applicable copyright or other intellectual
property laws and treaties. This EULA grants you no rights to use such content.
5. UPGRADES. If the SOFTWARE PRODUCT is labeled as an upgrade, you must be properly licensed to use a product identified
by Microsoft as being eligible for the upgrade in order to use the SOFTWARE PRODUCT. A SOFTWARE PRODUCT labeled as an
upgrade replaces and/or supplements the product that formed the basis for your eligibility for the upgrade. You may use the
resulting upgraded product only in accordance with the terms of this EULA.
6. U.S. GOVERNMENT RESTRICTED RIGHTS.
The SOFTWARE PRODUCT is provided with RESTRICTED RIGHTS. Use, duplication, or disclosure by the Government is subject to
restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS
252.227-7013 or subparagraphs (c)(1) and (2) of the Commercial Computer Software Restricted Rights at 48 CFR 52.227-19, as
applicable. Manufacturer is Microsoft Corporation/One Microsoft Way/Redmond, WA 98052-6399.
7. EXPORT RESTRICTIONS.
You agree that you will not export or re-export the SOFTWARE PRODUCT to any country, person, entity or end user subject to
U.S.A. export restrictions. Restricted countries currently include, but are not necessarily limited to Cuba, Iran, Iraq, Libya, North
Korea, Syria, and the Federal Republic of Yugoslavia (Serbia and Montenegro, U.N. Protected Areas and areas of Republic of
Bosnia and Herzegovina under the control of Bosnian Serb forces). You warrant and represent that neither the U.S.A. Bureau of
Export Administration nor any other federal agency has suspended, revoked or denied your export privileges.
8. NO WARRANTY. ANY USE OF THE SOFTWARE PRODUCT IS AT YOUR OWN RISK. THE SOFTWARE PRODUCT IS PROVIDED FOR
USE ONLY WITH MICROSOFT SQL SERVER AND/OR MICROSOFT WINDOWS NT SERVER SOFTWARE. TO THE MAXIMUM EXTENT
PERMITTED BY APPLICABLE LAW, MICROSOFT AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES AND CONDITIONS, EITHER

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 2

EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE, AND NONINFRINGEMENT.
9. NO LIABILITY FOR CONSEQUENTIAL DAMAGES.
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL MICROSOFT OR ITS SUPPLIERS BE LIABLE FOR
ANY SPECIAL, INCIDENTAL, INDIRECT, OR CONSEQUENTIAL DAMAGES WHATSOEVER (INCLUDING, WITHOUT LIMITATION,
DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION, OR ANY OTHER
PECUNIARY LOSS) ARISING OUT OF THE USE OF OR INABILITY TO USE THE SOFTWARE PRODUCT, EVEN IF MICROSOFT HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. BECAUSE SOME STATES AND JURISDICTIONS DO NOT ALLOW THE
EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT
APPLY TO YOU.
10. LIMITATION OF LIABILITY. MICROSOFT'S ENTIRE LIABILITY AND YOUR EXCLUSIVE REMEDY UNDER THIS EULA SHALL NOT
EXCEED FIVE DOLLARS (US$5.00).
11. MISCELLANEOUS
This EULA is governed by the laws of the State of Washington, U.S.A.
Should you have any questions concerning this EULA, or if you desire to contact Microsoft for any reason, please contact the
Microsoft subsidiary serving your country, or write:
Microsoft Sales Information Center/One Microsoft Way/Redmond, WA 98052-6399.
Si vous avez acquis votre produit Microsoft au CANADA, la garantie limitée suivante vous concerne:
EXCLUSION DE GARANTIES. Microsoft renonce entièrement à toute garantie pour le LOGICIEL. Le LOGICIEL et toute autre
documentation s'y rapportant sont fournis « comme tels » sans aucune garantie quelle qu'elle soit, expresse ou implicite, y
compris, mais ne se limitant pas aux garanties implicites de la qualité marchande ou un usage particulier. Le risque total
découlant de l'utilisation ou de la performance du LOGICIEL est entre vos mains.
RESPONSABILITÉ LIMITÉE. La seule obligation de Microsoft et votre recours exclusif concernant ce contrat n'excèderont pas
cinq dollare (US$5.00).
ABSENCE DE RESPONSABILITÉ POUR LES DOMMAGES INDIRECTS.
Microsoft ou ses fournisseurs ne pourront être tenus responsables en aucune circonstance de tout dommage quel qu'il soit (y
compris mais non de façon limitative les dommages directs ou indirects causés par la perte de bénéfices commerciaux,
l'interruption des affaires, la perte d'information commerciale ou toute autre perte pécuniaire) résultant de l'utilisation ou de
l'impossibilité d'utilisation de ce produit, et ce, même si la société Microsoft a été avisée de l' éventualité de tels dommages.
Certains états/juridictions ne permettent pas l'exclusion ou la limitation de responsabilité relative aux dommages indirects ou
consécutifs, et la limitation ci-dessus peut ne pas s'appliquer à votre égard. La présente Convention est régie par les lois de la
province d'Ontario, Canada. Chacune des parties à la présente reconnaît irrévocablement la compétence des tribunaux de la
province d'Ontario et consent à instituer tout litige qui pourrait découler de la présente auprès des tribunaux situés dans le
district judiciaire de York, province d'Ontario. Au cas où vous auriez des questions concernant cette licence ou que vous
désiriez vous mettre en rapport avec Microsoft pour quelque raison que ce soit, veuillez contacter la succursale Microsoft
desservant votre pays, dont l'adresse est fournie dans ce produit, ou écrire à: Microsoft Customer Sales and Service, One
Microsoft Way, Redmond, Washington 98052 6399.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 3

CONTENTS

MICROSOFT® SQL SERVER™ TPC-E BENCHMARK KIT REFERENCE ................................................................... 1

END-USER LICENSE AGREEMENT FOR MICROSOFT TPC-E BENCHMARK KIT .........................................................2

INTRODUCTION ................................................................................................................................................ 6

IMPORTANT PUBLICATION NOTE ............................................................................................................... 6

SOFTWARE REQUIREMENTS ........................................................................................................................ 7

MICROSOFT TPC-E KIT USAGE AND RESTRICTIONS .............................................................................. 8

SQL SERVER TRACE FLAGS......................................................................................................................... 10

TPC-E BENCHMARK OVERVIEW ............................................................................................................................ 11

Application Components ......................................................................................................................................12

Transaction Summary ..........................................................................................................................................12

Model Description ................................................................................................................................................14

BENCHMARK KIT COMPONENTS ............................................................................................................... 17

TPC-E BENCHMARK DATABASE SETUP .................................................................................................... 18

Server Configuration ............................................................................................................................................18

Database Setup and Configuration: .....................................................................................................................18

Database Setup with Scripting .............................................................................................................................24

SUT TIER-A SETUP.......................................................................................................................................... 24

SUT_CE_Server .....................................................................................................................................................25

SUT_MEE_Server .................................................................................................................................................25

Starting the SUT Servers ......................................................................................................................................26

Stopping the SUT Servers .....................................................................................................................................26

SUT Server Errors .................................................................................................................................................26

SUT TIME SYNCHRONIZATION ............................................................................................................................. 27

SUT.Config ...........................................................................................................................................................27

SyncTime ..............................................................................................................................................................27

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 4

 VerifyTime ............................................................................................................................................................28

BENCHCRAFT ....................................................................................................................................................... 28

BenchCraft Installation ........................................................................................................................................29

Potential BenchCraft Startup/Runtime Errors .....................................................................................................29

BenchCraft Profiles ..............................................................................................................................................30

POST-RUN PROCESSING ....................................................................................................................................... 34

PERFORMANCE CONSIDERATIONS AND TUNING ................................................................................................. 35

SQL Server Tuning ................................................................................................................................................35

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 5

INTRODUCTION

The Microsoft TPC-E Benchmark Kit provides a complete implementation of the TPC-E benchmark. The aim of this
benchmark kit is twofold:

1. to provide a tool to evaluate SQL Server’s performance on specific hardware platforms

2. to aid in publishing audited TPC-E benchmarks.

This document begins with a general overview of the TPC-E benchmark and its transactions. It then describes the
individual components of the Microsoft SQL Server TPC-E Benchmark Kit and demonstrates how these tools can be
used.

This document also offers some suggestions and useful information concerning SQL Server performance tuning as
it applies to the TPC-E benchmark. The information presented here may also prove to be useful in other
performance tuning scenarios.

IMPORTANT PUBLICATION NOTE

Prior to publishing any TPC-E result with Microsoft SQL Server you must logon to the Microsoft SQL Server TPC
Benchmarking Support SharePoint site
(https://microsoft.sharepoint.com/teams/Microsoft_SQL_Server_TPC_Benchmarking_Support) and do the
following:

• Generate a price quote for all Microsoft products used in your result
o The price quote is automatically generated based on you input and will be sent via e-mail to you
• Request permission to publish your result
o Permission to publish a benchmark result on SQL Server is required by the End-User License
Agreement (EULA). A copy of the EULA is found at the beginning of this document as well as on
the web site when you downloaded this benchmark kit
o It generally takes 5 business days to go through the approval process so be sure to allow enough
time prior to your publication date

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 6

SOFTWARE REQUIREMENTS

This version of the TPC-E benchmark kit requires the following components for building the test database, running
the benchmark, and building test drivers:

• Microsoft SQL Server 2014

• Microsoft SQL Server 2014 Standard Edition
• Microsoft SQL Server 2014 Enterprise Edition
• Microsoft SQL Server 2012 (MAY NOT BE USED FOR PUBLICATIONS)
• Microsoft SQL Server 2012 Standard Edition
• Microsoft SQL Server 2012 Enterprise Edition
• Microsoft SQL Server 2008 R2 (MAY NOT BE USED FOR PUBLICATIONS)
• Microsoft SQL Server 2008 R2 Standard Edition
• Microsoft SQL Server 2008 R2 Enterprise Edition
• Microsoft SQL Server 2008 R2 Developer Edition
• Microsoft Visual Studio 2008 Express Edition or later (only necessary if you need to rebuild the EGen
Components)
• Microsoft ODBC SDK (only necessary if there is a need to rebuild the EGen Components)
• Microsoft Windows Server 2012 and 2012 R2
• Windows Server 2012 R2 Standard Edition
• Windows Server 2012 R2 Enterprise Edition
• Windows Server 2012 R2 Datacenter Edition
• Windows Server 2012 Standard Edition
• Windows Server 2012 Enterprise Edition
• Windows Server 2012 Datacenter Edition
• You may use Microsoft Windows Server 2008 for the SUT Server systems

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 7

MICROSOFT TPC-E KIT USAGE AND RESTRICTIONS

Initially, we are asking that you do not modify certain parts of our kit. There are several reasons for this. First,
customization of the kit will make it much more difficult for us to support you. With a single version of the kit
there will be fewer problems and any problems that do occur will be fixed faster. Finally, we want to establish a
solid base of TPC-E results across many platforms. Having everyone using the same kit will improve the
comparability of results.

The parts of the kit which cannot be changed are:

1. Source code for the frame implementations.

2. The stored procedures which implement the transaction.
3. The database schemas, i.e., the scripts for create tables, etc.
4. Database options such as startup parameters, trace flags, and isolation settings.
1. For trace flags, a few trace flags are documented in the kit. These will be allowed and can be set
or not, at the sponsor’s discretion. No other trace flags can be used.
2. You may use only the following startup options for SQL Server:
1. -c
2. -x
3. -E

What can be changed? You are free to customize the following things:

1. Database files layout and allocation (e.g., the script Create_Database.sql in the kit).
2. All sp_configure options, e.g. affinity mask, fibers, max worker threads, max memory.
3. Disk configuration, e.g., RAID, controllers, hardware settings, etc.
4. Hardware configuration:

a. CPU, memory, network, etc.

b. NUMA tuning

5. Benchmark configuration and tuning:

c. # of customers
d. DB size
e. Configuration of tier A/B hardware
f. etc.

6. BenchCraft configuration and tuning:

g. Number of CE and MEE drivers

h. Number of SUT servers
i. Pacing
j. # of emulated users
k. etc.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 8

As with any benchmark publication using SQL Server, you must get written permission from Microsoft prior to
publication. Unapproved modifications to the kit will most likely result in permission to publish being denied.
Please work with us well in advance if you want to make any changes.

Please send requests to publish results to me. For TPC-E results, allow at least one week for a response. We will
need to have the following documentation provided in softcopy at the time of the request:

1. Complete “Supporting Files” as required by the spec that will be submitted to the TPC.
2. Benchcraft txn report for the reported run (the XLT file, not a screenshot or printout).
3. SQL Server error log for the reported run (complete log from server startup).
4. All output in the directory under .\MSTPCE1.13.0-1027\SetupLogs corresponding to the reported run.

Include all files in a single ZIP archive.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 9

SQL SERVER TRACE FLAGS

Frequently, trace flags are used to alter the behavior of SQL Server. The following trace flag(s) are currently
supported for publication. If you are instructed to use a trace flag that is not on this list, please contact Jamie
Reding ([email protected]) or Charles Levine ([email protected]) prior to any publication
efforts.

The currently supported trace flags are:

• T652 – Disable page pre-fetching scans

o Trace flag 652 causes SQL Server to no longer bring database pages into the buffer pool before
these database pages are consumed by scans.

• T661 – Disable the ghost record removal process

o Ghost records are the result of delete operations and are later purged by the ghost record
removal process. Trace flag 661 disables the removal process.

• T834 – Use Windows large-page allocations for the buffer pool

o Trace flag 834 causes SQL Server to use the Windows large-page allocations for memory
allocated to the buffer pool. The page size varies by platform, but will be from 2MB to 16MB.

• T3502 – Display Checkpoint messages in the SQL Server errorlog

o Trace flag 3502 causes SQL Server to log timestamp information for the checkpoint process in the
SQL Server errorlog.

• T8744 – Disallow pre-fetch for ranges

o This trace flag disables pre-fetching for all Nested Loop operators.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 10

TPC-E BENCHMARK OVERVIEW

Business and Application Environment

TPC Benchmark™ E is composed of a set of transactional operations designed to exercise system functionalities in a
manner representative of complex OLTP application environments. These transactional operations have been
given a life-like context, portraying the activity of a brokerage firm, to help users relate intuitively to the
components of the benchmark. The workload is centered on the activity of processing brokerage trades and uses
a schema, which is logically divided in four sets of tables.

TPC-E models the activity of brokerage firm that must manage customer accounts, execute customer trade orders,
and be responsible for the interactions of customers with financial markets. TPC-E does not attempt to be a model
of how to build an actual application. The following diagram illustrates the transaction flow of the business model
portrayed in the benchmark:

Customer Market

Customer Market
Initiated Triggered
Transactions Transactions

Brokerage

Business Model Transaction Flow

The purpose of a benchmark is to reduce the diversity of operations found in a production application, while
retaining the application's essential performance characteristics so that the workload can be representative of a
production system. A large number of functions have to be performed to manage a production brokerage system.
Many of these functions are not of primary interest for performance analysis, since they are proportionally small in
terms of system resource utilization or in terms of frequency of execution. Although these functions are vital for a
production system, they merely create excessive diversity in the context of a standard benchmark and have been
omitted in TPC-E.

The Company portrayed by the benchmark is a brokerage firm with customers who generate transactions related
to trades, account inquiries, and market research. The brokerage firm in turn interacts with financial markets to
execute orders on behalf of the customers and updates relevant account information.

The number of customers defined for the brokerage firm can be varied to represent the workloads of different size
businesses.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 11

This benchmark is composed of a set of transactions that are executed against three sets of database tables that
represent market data, customer data, and broker data. A fourth set of tables contains generic dimension data
such as zip codes. The following diagram illustrates the key components of the environment:

Customers Brokers Market

Invoke the following transactions …

READ-WRITE READ-ONLY
•Market-Feed •Broker-Volume •Security-Detail
•Trade-Order •Customer-Position •Trade-Lookup
•Trade-Result •Market-Watch •Trade-Status
•Trade-Update

… against the following data

Customer Data Brokerage Data Market Data

APPLICATION COMPONENTS

The benchmark has been reduced to simplified form of the application environment. Error! Reference source not
found.

TRANSACTION SUMMARY

BROKER-VOLUME

The Broker-Volume Transaction is designed to emulate a brokerage house’s “up-to-the-minute” internal

business processing. An example of a Broker-Volume Transaction would be a manager generating a report on
the current performance potential of various brokers.

CUSTOMER-POSITION

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 12

The Customer-Position Transaction is designed to emulate the process of retrieving the customer’s profile and
summarizing their overall standing based on current market values for all assets. This is representative of the work
performed when a customer asks the question “What am I worth today?”

MARKET-FEED

The Market-Feed Transaction is designed to emulate the process of tracking the current market activity. This is
representative of the brokerage house processing the “ticker-tape” from the market exchange.

MARKET-WATCH

The Market-Watch Transaction is designed to emulate the process of monitoring the overall performance of the
market by allowing a customer to track the current daily trend (up or down) of a collection of securities. The
collection of securities being monitored may be based upon a customer’s current holdings, a customer’s watch list
of prospective securities, or a particular industry.

SECURITY-DETAIL

The Security-Detail Transaction is designed to emulate the process of accessing detailed information on a
particular security. This is representative of a customer doing research on a security prior to making a decision
about whether or not to execute a trade.

TRADE-LOOKUP

The Trade-Lookup Transaction is designed to emulate information retrieval by either a customer or a broker to
satisfy their questions regarding a set of trades. The various sets of trades are chosen such that the work is
representative of:

• performing general market analysis

• reviewing trades for a period of time prior to the most recent account statement
• analyzing past performance of a particular security
• analyzing the history of a particular customer holding

TRADE-ORDER

The Trade Order Transaction is designed to emulate the process of buying or selling a security by a Customer,
Broker, or authorized third-party. If the person executing the trade order is not the account owner, the
Transaction will verify that the person has the appropriate authorization to perform the trade order. The
Transaction allows the person trading to execute buys at the current market price, sells at the current market
price, or limit buys and sells at a requested price. The Transaction also provides an estimate of the financial
impact of the proposed trade by providing profit/loss data, tax implications, and anticipated commission fees. This
allows the trader to evaluate the desirability of the proposed security trade before either submitting or canceling
the trade.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 13

TRADE-RESULT

The Trade-Result Transaction is designed to emulate the process of completing a stock market trade. This is
representative of a brokerage house receiving from the market exchange the final confirmation and price for the
trade. The customer’s holdings are updated to reflect that the trade has completed. Estimates generated when the
trade was ordered for the broker commission and other similar quantities are replaced with the actual numbers
and historical information about the trade is recorded for later reference.

TRADE-STATUS

The Trade-Status Transaction is designed to emulate the process of providing an update on the status of a
particular set of trades. It is representative of a customer reviewing a summary of the recent trading activity for
one of their accounts.

TRADE-UPDATE

The Trade-Update Transaction is designed to emulate the process of making minor corrections or updates to a set
of trades. This is analogous to a customer or broker reviewing a set of trades, and discovering that some minor
editorial corrections are required. The various sets of trades are chosen such that the work is representative of:

• reviewing general market trends

• reviewing trades for a period of time prior to the most recent account statement
• reviewing past performance of a particular security

DATA-MAINTENANCE

The Data-Maintenance Transaction is designed to emulate the periodic modifications to data that is mainly static
and used for reference. This is analogous to updating a customer’s email address or other data that seldom
changes.

TRADE-CLEANUP

The Trade-Cleanup Transaction is used to cancel any pending or submitted trades from the database.

MODEL DESCRIPTION

ENTITY RELATIONSHIPS

Trading in TPC-E is done by Accounts. Accounts belong to Customers. Customers are serviced by Brokers. Accounts
trade Securities that are issued by Companies.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 14

The total set of Securities that can be traded and the total set of Companies that issue Securities scales along with
the number of Customers. For each unit of 1,000 Customers, there are 685 Securities and 500 Companies (with
Companies issuing 1 to 5 Securities, mostly common shares, but some preferred as well).

All Companies belong to one of the 102 Industries. Each Industry belongs to one of the 12 market Sectors.

Each Account picks its average ten Securities to trade from across the entire range of Securities.

Securities to be traded can be identified by the security symbol or by the company name and security issue.

DIFFERENCES BETWEEN CUSTOMER TIERS

The basic scaling unit of a TPC-E database is a set of 1,000 Customers. 20% of each 1,000 Customers belong to Tier
1, 60% to Tier 2, and 20% to Tier 3. Tier 2 Customers trade twice as often as Tier 1 Customers. Tier 3 Customers
trade three times as often as Tier 1 Customers. In general, customer trading is non-uniform by tier within each set
of 1,000 Customers.

Tier 1 Customers have 1 to 4 Accounts (average 2.5). Tier 2 Customers have 2 to 8 Accounts (average 5.0). Tier 3
Customers have 5 to 10 Accounts (average 7.5). Overall, there is an average of five Accounts per Customer.

The minimum and maximum number of Securities that are traded by each Account varies by Customer Tier and by
the number of Accounts for each Customer. The average number of Securities traded per Account is ten (so the
average number of Securities traded per Customer is fifty). For each Account, the same set of Securities is traded
for both the initial database population and for any Test Run.

CUSTOMER PARTITIONING

TPC-E scales with Customers. It is conceivable that Customer information could be partitioned into groups of
related Customers. This is called Customer Partitioning. The advantage of Customer Partitioning is that it increases
locality of reference within each sub-group of Customers. Transactions relating to a particular set of Customers
would be directed to that set of Customers rather than to all Customers.

TRADE TYPES

Trade requests come in two basic flavors: Buy (50%) and Sell (50%). Those are further broken down into Trade
Types, depending on whether the request was a Market Order (60%) or a Limit Order (40%).

For Market Orders, the two trade types are Market-Buy (30%) and Market-Sell (30%). For Limit Orders, the three
trade types are Limit-Buy (20%), Limit-Sell (10%) and Stop-Loss (10%).

Market-Buy and Market-Sell are trade requests to buy and sell immediately at the current market price, whatever
price that may be. Limit-Buy is a request to buy only when the market price is at or below the specified limit price.
Limit-Sell is a request to sell only when the market price is at or above the specified limit price. Stop-Loss is a
request to sell only when (or if) the market price drops to or below the specified limit price.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 15

If the specified limit price has not been reached when the Limit Order is requested, it is considered an Out-of-the-
Money request and remains “Pending” until the specified limit price is reached. Reaching the limit price is
guaranteed to occur within 15 minutes based on EGenDriverMEE implementation details. The act of noticing that
a “Pending” limit request has reached or exceeded its specified limit price and submitting it to the market
exchange to be traded is known as triggering of the pending limit order.

EFFECTS OF TRADING ON HOLDINGS

For a given account and security, holdings will be either all long (positive quantities) or all short (negative
quantities).

Long positions represent shares of the security that were bought (purchased and paid for) by the customer for the
account. The customer owns the shares of the security and may sell them at a later time (hopefully, for a higher
price).

Short positions represent shares of the security that were borrowed from the broker (or Brokerage) and were sold
by the customer for the account. In the short sale case, the customer has received the funds from that sell, but
still has to cover the sell by later purchasing an equal number of shares (hopefully at a lower price) from the
market and returning those shares to the broker.

Before EGenLoader runs, there are no trades and no positions in any security for any account. EGenLoader
simulates running the benchmark for three hundred Business Days of initial trading, so that the initial database
will be ready for benchmark execution.

If the first trade for a security in an account is a buy, a long position will be established (positive quantity in
HOLDING row). Subsequent buys in the same account for the same security will add holding rows with positive
quantities. Subsequent sells will reduce holding quantities or delete holding rows to satisfy the sell trade. All
holdings may be eliminated, in which case the position becomes empty. If the sell quantity still is not satisfied, the
position changes from long to short (see below).

If the first trade for a security in an account is a sell, a short position will be established (negative quantity in
HOLDING row). Subsequent sells in the same account for the same security will add holding rows with negative
quantities. Subsequent buys will reduce holding quantities (toward zero) or delete holding rows to satisfy the buy
trade. All holdings may be eliminated, in which case the position becomes empty. If the buy quantity still is not
satisfied, the position changes from short to long.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 16

BENCHMARK KIT COMPONENTS

The following section describes the benchmark kit itself and how to use the components. This kit includes all of
the files necessary to build the database, populate tables, and measure the TPC-E workload. It also includes source
code and the executable for the EGen package as distributed by the TPC.

This version of the benchmark kit also contains a Driver (called BenchCraft) that will allow you to run a full end-to-
end benchmark test as required by the TPC-E benchmark specification.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 17

TPC-E BENCHMARK DATABASE SETUP

SERVER CONFIGURATION

Generally speaking, the target server should be suitable for the test. That is, it should have sufficient memory,
processing power, and storage to support both Windows Server and SQL Server. In order to understand the
potential throughput and required hardware for a particular SUT, it may be valuable to consult the FDR (Full
Disclosure Report) of a previously published TPC-E result that was implemented with a similar hardware platform.
See the performance considerations and tuning section further on in this document for a more detailed discussion
of appropriate server configurations and other performance-related issues.

SQL SERVER INSTALLATION:

Once you have selected the appropriate hardware for the server, your next step is to install SQL Server and build a
TPC-E database. Use the following steps as reference.

When installing SQL Server you should install with the Latin1_General_Binary collation option. The default sort
order in the SQL Server setup is Dictionary Case Insensitive, so you should change this option during the SQL Server
installation. While the benchmark will run properly on a server installed with other sort orders, the binary sort
order will generally offer the best overall performance. If you do not specify the correct collation, you may specify
it in the create database process. For more information, see Create Database in SQL Server Books Online.

Once SQL Server has been installed, copy the benchmark kit in its entirety to a directory on the server. This
provides a central location for the benchmark kit’s files and for the database build process. In addition, database
builds are generally faster when initiated locally.

DATABASE SETUP AND CONFIGURATION:

The TPC-E kit contains all files necessary to build a TPC-E database in the \Scripts directory and its associated sub-
directories. This includes scripts that build the devices and database for a given number of Customers, scripts
necessary to create the database objects, and a loader utility for populating the tables. The files in the #.Cust
directories (where # corresponds to the number of Customers) are specific to the scale of the database you intend
to build. This includes scripts for creating the database (Database\Create_Database.sql), creating backup devices
(Database\Backup_Devices.sql), and backup/restore the database after the build (Database\Backup.sql and
Database\Restore.sql.

FLAT FILE LOAD CUSTOMIZATION STEPS

You will need to customize the database creation, backup, and restore scripts to reflect your environment. In
addition, you may wish to create new #.Cust directories to further refine the scale of the database you wish to
build.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 18

FLAT FILE CONFIGURATION FILE

If you are using the flat file loading process, you may optionally pass a flat file location configuration file to the
setup process. If you choose to not use a configuration file, you will be prompted for each location individually. If
you do use the configuration file, you will be prompted to enter the full path and file name for the configuration
file. The file must contain a full path name to the locations you want to place the flat files on. Each location should
be on a line by itself. There must be a location for each EGenLoader instance you are using. If there are more or
less flat file locations specified compared to the desired EGenLoader instances, the setup process will display an
error message and quit. For example, the following lines would be in a configuration file for a database build using
4 EGenLoader instances:

E:\Flat_Out
F:\Flat_Out
G:\Flat_Out
H:\Flat_Out

TPCE_SETUP OPTIONS

Also, if you are new to TPC-E benchmarking it is strongly recommended that you start by building a fairly small
database (1000 Customers) to get familiar with the build process and tools. TPC-E databases of less than 5000
Customers are not valid for publication, but their small size makes them ideal for learning the mechanics.

After you have customized the files in the desired #.Cust\Database, you may start the process of creating your TPC-
E database.

This version of the kit now allows you to create the TPC-E database either with the traditional interactive method
or with the new XML automation scripting. Please see the Database Setup With Scripting section for additional
information.

TPC-E databases in the Microsoft TPC-E benchmark kit are created interactively using a Windows Command file
TPCE_Setup.cmd. This command file calls a Windows Scripting Host Visual Basic script that performs the necessary
steps to create the TPC-E database. It will prompt you for the parameters required. It will fill-in the default values
where necessary.

The following information must be passed to the setup process during the prompting phase:

• BUILD OPTION
There are 11 options that you may specify. They are:
o CREATEDATABASEFULL
▪ Remove any existing TPC-E database
▪ Create a TPC-E database
▪ Create the TPC-E data types
▪ Create the TPC-E tables
▪ Install the TPC-E stored procedures
▪ Set the pre-load database options
▪ Load the data via either ODBC or Flat Files
▪ Convert the NI_ITEM data type for flat file loads only

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 19

 ▪ Optionally delete the flat files
▪ Create the clustered indexes
▪ Create the non-clustered indexes
▪ Verify/Create the TPC-E check constraints
▪ Create the TPC-E foreign key constraints
▪ Create the TID_RANGES table
▪ Create the TPCE_INFO table
▪ Set the post-load database options
▪ Load the TPCE_INFO table
▪ Optionally backup the TPC-E database
▪ Optionally calculate the space used
▪ Optionally execute DB_Check
▪ Optionally execute DB_Pimary_Key_Check
▪ Optionally execute DB_Tables
▪ Optionally execute Insert Duplicates check
▪ Optionally execute Referential Integrity check

o CREATEDATABASE
▪ Remove any existing TPC-E database
▪ Create a TPC-E database
▪ Create the TPC-E data types

o CREATETABLESFULL
▪ Create the TPC-E tables
▪ Install the TPC-E stored procedures
▪ Set the pre-load database options
▪ Load the data via either ODBC or Flat Files
▪ Convert the NI_ITEM data type for flat file loads only
▪ Optionally delete the flat files
▪ Create the clustered indexes
▪ Create the non-clustered indexes
▪ Verify/Create the TPC-E check constraints
▪ Create the TPC-E foreign key constraints
▪ Create the TID_RANGES table
▪ Create the TPCE_INFO table
▪ Set the post-load database options
▪ Load the TPCE_INFO table
▪ Optionally backup the TPC-E database
▪ Optionally calculate the space used
▪ Optionally execute DB_Check
▪ Optionally execute DB_Pimary_Key_Check
▪ Optionally execute DB_Tables
▪ Optionally execute Insert Duplicates check
▪ Optionally execute Referential Integrity check

o CREATETABLES
▪ Create the TPC-E tables
▪ Install the TPC-E stored procedures

o LOADDATAFULL

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 20

 ▪ Set the pre-load database options
▪ Load the data via either ODBC or Flat Files
▪ Convert the NI_ITEM data type for flat file loads only
▪ Optionally delete the flat files
▪ Create the clustered indexes
▪ Create the non-clustered indexes
▪ Verify/Create the TPC-E check constraints
▪ Create the TPC-E foreign key constraints
▪ Create the TID_RANGES table
▪ Create the TPCE_INFO table
▪ Set the post-load database options
▪ Load the TPCE_INFO table
▪ Optionally backup the TPC-E database
▪ Optionally calculate the space used
▪ Optionally execute DB_Check
▪ Optionally execute DB_Pimary_Key_Check
▪ Optionally execute DB_Tables
▪ Optionally execute Insert Duplicates check
▪ Optionally execute Referential Integrity check

o LOADDATA
▪ Set the pre-load database options
▪ Load the data via either ODBC or Flat Files
▪ Convert the NI_ITEM data type for flat file loads only
▪ Optionally delete the flat files

o BUILDFLATFILESONLY
▪ Generate the flat files

o POSTLOAD
▪ Set the pre-load database options
▪ Convert the NI_ITEM data type for flat file loads only
▪ Optionally delete the flat files
▪ Create the clustered indexes
▪ Create the non-clustered indexes
▪ Verify/Create the TPC-E check constraints
▪ Create the TPC-E foreign key constraints
▪ Create the TID_RANGES table
▪ Create the TPCE_INFO table
▪ Set the post-load database options
▪ Load the TPCE_INFO table
▪ Optionally backup the TPC-E database

o BACKUPDATABASE
▪ Backs up the TPC-E database

o SPACEUSAGE
▪ Calculate the space used

o VERIFYLOAD

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 21

 ▪ Execute DB_Check
▪ Execute DB_Pimary_Key_Check
▪ Execute DB_Tables
▪ Check the indexes, foreign key constraints, and check constraints
▪ Execute Insert Duplicates check
▪ Execute Referential Integrity check

• SERVER NAME
The default for this prompt is the name of the server that TPCE_Setup.cmd is executing on. It is highly
recommended that you run the setup process on the database server.

• SQL SERVER LOGIN METHOD

Selecting “Yes” to this prompt will use Windows Authentication for the SQL Server connection. Selecting
“No” will require you to enter a SQL Server user id and password.

• SQL SERVER USER ID

This prompt will only be displayed if you select No to the SQL Server Login Method prompt. The default
value is “sa”. It is highly recommended that you use “sa” for the user id.

• SQL SERVER PASSWORD

This prompt will only be displayed if you select No to the SQL Server Login Method prompt. Enter the
password for the account you specified for the SQL Server User ID above.

• NUMBER OF CUSTOMERS
Enter the number of customers to populate. Note, there must be an associated XXX.CUST directory
created prior to building. If the directory does not exist, the build will fail.
If you specify a value of less than 5,000 customers, you will be given a warning that configurations of less
than 5,000 customers are not valid for publication. Selecting Yes will allow you to continue for testing
purposes. Selecting No will allow you to re-enter the value.

• NUMBER OF EGENLOADER INSTANCES

This specifies the number of instances of the EGenLoader that will be executed simultaneously. The
default value is equal to the number of processors on the system that the setup is running on. Specifying
a value greater than the number of processors may lead to undesirable results.

• BUILD OR REBUILD THE FLAT FILES

Allows you to reuse flat files from a previous data file build. The default value is Yes.

• DELETE FLAT FILES AFTER LOADING

Instructs the setup process to delete all flat files after the data load is complete. The default value is No.

• USE A FLAT FILE CONFIGURATION FILE

Allows you to provide the setup process a text file with the flat file out locations rather than keying them
in individually. The format of this file is documented in the Flat File Configuration File section above. The
default value is Yes.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 22

 • FLAT FILE LOCATIONS
if you do not provide a flat file configuration file then you will be prompted for an output location for each
EGenLoader instances previously selected.

• SCALE FACTOR - OPTIONAL

Sets the scale factor for the EGenLoader instance(s). The default is 500. Changing this value may lead to
unusable TPC-E Database builds.

• NUMBER OF WORK DAYS FOR INITIAL TRADES - OPTIONAL

Sets the number of days of initial trades that the EGenLoader instance(s) will generate. The default is 300.
Changing this value may lead to unusable TPC-E Database builds.

• BACKUP THE DATABASE

Allows the sponsor to back up the TPC-E database after completing the build. The default value it No.

• EXECUTE SPACE USAGE SCRIPTS

Allows the sponsor to run the space usage scripts. The default value it No.

• EXECUTE AUDIT SCRIPTS

Allows the sponsor to bypass the execution of the TPC-E auditor scripts. These scripts include; DB Check,
DB Tables, Space Usage, Referential Integrity tests, Insertion of Duplicates tests, etc… The default value it
Yes.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 23

DATABASE SETUP WITH SCRIPTING

You can now semi-automate the database setup process by passing in an XML-like script to the setup process. The
script can include any of the build options above and will check the input file to ensure all the proper information
has been included. If required information is missing, the setup process will abort and display an error message
indicating the error. The scripting feature only supports the use of a flat file configuration file if you are creating or
loading the flat files. There are several sample scripts in the Automation_Scripts directory.

To invoke the setup process with an automation script, you do the following:

TPCE_Setup.cmd –s <automation script path and name>

For example, if you are using the sample Create Database Full script for 5,000 customers, you would run this:

TPCE_Setup.cmd –s Automation_Scripts\CreateDatabaseFull_5000_Customers.xml

The internal format of the script is as follows. Note, the tag and elements are not case sensitive, except for the
SQL User Id and SQL Password (if used). The tags can appear in any order.

<TPCE_SETUP>
<NumberOfCustomers>5000</NumberOfCustomers>
<BuildOption>CreateDatabaseFull</BuildOption>
<ServerName>TPCE_Test_System</ServerName>
<Authentication>Windows</Authentication>
<SQLUserID></SQLUserID>
<SQLPwd></SQLPwd>
<NumberOfEGenLoaders>5</NumberOfEGenLoaders>
<BuildFlatFiles>Yes</BuildFlatFiles>
<DeleteFlatFiles>No</DeleteFlatFiles>
<FlatFileConfigFileName>C:\FF_Config_5.txt</FlatFileConfigFileName>
<IndexBuildParallelism>16</IndexBuildParallelism>
<ScaleFactor>500</ScaleFactor>
<InitialTradeDays>300</InitialTradeDays>
<RunAuditScripts>No</RunAuditScripts>
<RunSpaceUsage>Yes</RunSpaceUsage>
<BackupDatabase>Yes</ BackupDatabase >
</TPCE_SETUP>

SUT TIER-A SETUP

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 24

The SUT Tier-A drivers are combinations of TPC provided code and Microsoft platform specific code. There are 2
different SUT components you will need to run. They are SUT_CE_Server.exe and SUT_MEE_Server.exe. You may
run them on the database server or other system(s) in the SUT. Each driver may optionally take arguments.

NOTE: YOU MUST RUN THE VISUAL STUDIO PACKAGE ON EACH TIER-A SYSTEM TO ENSURE THAT THE PROPER
VISUAL STUDIO DLLS ARE PRESENT. THE PACKAGE IS LOCATED IN \VS_MODULES. RUN VCREDIST_X86.EXE TO
INSTALL THE DLLS. THIS PACKAGE IS VALID FOR X86, X64, AND IA64 ARCHITECTURES.

SUT_CE_SERVER

The SUT_CE_Server.exe provides the customer emulator functions for the TPC-E benchmark. BenchCraft
communicates to SUT_CE_Server via the driver, discussed below. The SUT_CE_Server.exe is located in the
MSTPCE.1.14.0-1041\EGen\SUT_CE_Server\Release directory. The SUT_CE_Server is executed from a command
line and it takes the following parameters:

• -s <Database Server Name>

This option specifies the DNS name or IP Address of the database server. This value defaults to the local
system.

• -d <Database Name>
This option specifies the database name to connect to. The default value is tpce.

• -p <Port To Listen On>

This option allows you to specify a port for this instance of the SUT_CE_Server to listen on. If you are
running multiple SUT_CE_Servers on a single system then you will need to specify a port for each
instance. The default value is 39099.

• -c <MDAC/SNAC>
This specifies the network protocol for the SUT_CE_Server. The default is MDAC.

• -m <MEE Name>
This is the name, i.e. Driver1, from the BenchCraft profile. It is only required if you have more than one
instance of SUT_CE_Server launched.

• -v
Directs the SUT_CE_Server to display verbose output.

• -?
Displays the usage information.

SUT_MEE_SERVER

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 25

The SUT_MEE_Server.exe provides the market emulator functions for the TPC-E benchmark. BenchCraft
communicates to SUT_MEE_Server via the driver, discussed below. The SUT_MEE_Server.exe is located in the
MSTPCE.1.14.0-1041\EGen\SUT_MEE_Server\Release directory. The SUT_MEE_Server is executed from a
command line and it takes the following parameters:

• -s <Database Server Name>

This option specifies the DNS name or IP Address of the database server. This value defaults to the local
system.

• -d <Database Name>
This option specifies the database name to connect to. The default value is tpce.

• -p <Port To Listen On>

This option allows you to specify a port for this instance of the SUT_MEE_Server to listen on. If you are
running multiple SUT_MEE_Servers on a single system then you will need to specify a port for each
instance. The default value is 38122.

• -c <MDAC/SNAC>
This specifies the network protocol for the SUT_MEE_Server. The default is MDAC.

• -m <MEE Name>
This is the name, i.e. Driver1, from the BenchCraft profile. It is only required if you have more than one
instance of SUT_MEE_Server launched.

• -v
Directs the SUT_MEE_Server to display verbose output.

• -?
Displays the usage information.

STARTING THE SUT SERVERS

Execution of the SUT servers is very straightforward. For each one, simply run the executable. At a minimum you
must specify the Database Server Name of the target TPC-E server. The SUT component will run in the window
where it was started.

STOPPING THE SUT SERVERS

The normal method for shutting down the SUT Servers is by pressing ‘Q’. This causes the SUT Server in question to
perform a graceful shutdown. You may encounter a condition, when errors were output in the SUT Server
window, where the SUT Server will not shut down with the ‘Q’. In this case, you may use ‘CTRL-C’ to terminate the
SUT Server.

SUT SERVER ERRORS

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 26

Should any one of the SUT Servers encounter an error during processing, all activity will cease in all SUT Servers.
This is by design to facilitate debugging of the error.

SUT TIME SYNCHRONIZATION

The Microsoft TPC-E Kit contains utilities to synchronize the system clocks as required in the TPC-E specification.
Prior to running the workload, you must synchronize all system clocks. The process uses a configuration file
(SUT.Config) which contains the machine names for the individual machines in the SUT. All system clocks are
synchronized to the DBMS server.

SUT.CONFIG

The SUT.Config file is used as input to the clock synchronization process. It has the following format:

<DBMS Server>
SQLPERFTPCE1
<SUT Servers>
SQLPERF-OLTP2
SQLPERF-OLTP3
SQLPERF-OLTP4
<DRIVER Systems>
SQLPERF-OLTP1
<END>

Enter each system name below the appropriate header. The machine name used under the DBMS Server name
will be used as the time server.

SYNCTIME

The SyncTime.cmd, which calls SyncTime.vbs, reads the SUT.Config file to get the SUT configuration. It will check
the status of the Windows Time Server service on the DBMS Server system specified. If the service is not started, it
will start it. Then it will process the rest of the SUT.Config file. The vbs script will execute w32tm to synchronize
the clocks.

Sample output of SyncTIme using the SUT.Config above:

****************************************************************************
* *
* Microsoft TPC-E Benchmark Kit Ver. 1.13.0-1027 *
* *
* Time Synchronization Processing *
* *
****************************************************************************

Synchronizing time based on the settings in

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 27

 D:\MSTPCE.1.14.0-1041\Tools\SyncTime\SUT.Config.

Synchronizing time on SUT Server:SQLPERF-OLTP2 to SQLPERFTPCE1.

Time synchronized on SUT Server:SQLPERF-OLTP2.

Synchronizing time on SUT Server:SQLPERF-OLTP3 to SQLPERFTPCE1.

Time synchronized on SUT Server:SQLPERF-OLTP3.

Synchronizing time on SUT Server:SQLPERF-OLTP4 to SQLPERFTPCE1.

Time synchronized on SUT Server:SQLPERF-OLTP4.

Synchronizing time on Driver:SQLPERF-OLTP1 to SQLPERFTPCE1.

Time synchronized time on Driver:SQLPERF-OLTP1.

****************************************************************************
* *
* Microsoft TPC-E Benchmark Kit Ver. 1.13.0-1027 *
* *
* Time synchronization complete. *
* *
****************************************************************************

VERIFYTIME

The VerifyTime.cmd file MUST be run on each component of the SUT. It requires a single argument of the time
server system name. Executing this command file will verify the system time between the local machine and the
time server. The difference will be displayed as the output. For example, running VerifyTime.cmd on one of the
systems above:

C:\>verifytime SQLPERFTPCE1

Verifying local system time to SQLPERFTPCE1

Tracking SQLPERFTPCE1 [10.193.30.28].

Collecting 1 samples.
The current time is 6/5/2007 3:18:41 PM (local time).
15:18:41, +01.8222628s

BENCHCRAFT

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 28

The Microsoft TPC-E implementation uses BenchCraft as the driver. The look and feel is exactly the same as that of
TPC-C BenchCraft. The main caveat for BenchCraft is that you must uninstall any and all previous versions of
BenchCraft from the driver system. This version is NOT compatible with the previous versions at this time.

BENCHCRAFT INSTALLATION

You install BenchCraft in the same manner as with previous versions. The setup executable is in MSTPCE.1.14.0-
1041\BenchCraft. The executable is BenchcraftSetup2420E.msi. Simply follow the prompts to install BenchCraft.

POTENTIAL BENCHCRAFT STARTUP/RUNTIME ERRORS

SIDE-BY-SIDE CONFIGURATION ON INITIAL BENCHCRAFT STARTUP

This message indicates that you do not have the proper Visual Studio 2008 Redistributable code installed on the
BenchCraft machine.

To resolve this problem, you must run vc2008SP1redist_x86_with_ATL_Fix.exe. This is found in MSTPCE.1.14.0-
1041\VS_Modules.

DRIVER ENGINE LAUNCH ERRORS

These messages indicate that the BenchCraft Master and/or Slave components were installed to run as a Windows
Network Service.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 29

To resolve this, you must either change the account to have Administrative privileges or you must re-install the
Master and/or Slave components under the local Administrator account.

BENCHCRAFT PROFILES

BenchCraft uses a profile to define the drivers, market emulators, parameter sets, and users for the TPC-E
benchmark. Once the BenchCraft installation is complete, you can start the BenchCraft executable and create a
profile.

USER EMULATOR PROFILE

Right click on the BenchCraft window and select New UE. This will display the window to help you create a new
User Emulator. You will need to specify the following for a User Emulator:

• NAME
This is the name of the User Emulator. The default is fine here.

• PARAMETER SET
As with previous versions of BenchCraft, you can specify different runtime parameters. You can simply
use the default parameter set or specify one you define. That process is described below.

• DESCRIPTION
Optional text string for some unique description of the User Emulator

• MACHINE NAME
This is the machine name where the User Emulator driver will run.

• LOG FILE NAME

This is the fully qualified path and file name for the UE log.

• SIZE OF TARGET DB
This is where you plug in the number of customer in the target tpce database.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 30

 • SCALE DOWN
You may configure BenchCraft to run with fewer active customers than those configured. Check the box
and select the number of customers to access. NOTE: This is NOT a legal option for an audited run.

• RUN DATA MAINTENANCE

Check this box to instruct BenchCraft to run the Data Maintenance transaction. This will add one
connection to your user count. This connection is for the Data Maintenance transaction only. You only
need to check this box for one driver engine. The execution period for the transaction (default of 1
minute) is set in the Parameter Set specified for the driver engine.

• RUN CHECKPOINT
CURRENTLY; PLEASE DO NOT RUN THE CHECKPOINT FROM BENCHCRAFT. WE ARE
INVESTIGATING AN ISSUE WHERE YOU CAN HAVE OVERLAPPING CHECKPOINTS. YOU WILL
FIND A SQL SCRIPT IN THE SCRIPTS\UTILITY DIRECTORY (CHECKPOINT_TPCE_DATABASE.SQL)
WHICH YOU MAY USE TO RUN THE CHECKPOINT DURING YOUR TESTING AND
MEASUREMENTS.
Check this box to instruct BenchCraft to generate a database checkpoint. This will add 1 connection to
your user count. This connection is used for the checkpoint loop only. You only need to check this box
for one of your driver engines. The checkpoint interval, default value of 7.5 minutes (450 seconds), is set
in the Parameter Set specified for the driver engine.

• THROTTLE - CONNECT RATE

The number of users per minute that BenchCraft will connect to the database. These users will only
establish the connection; they will not start doing transactions.

• THROTTLE - START RATE

The number of users per minute that BenchCraft will instruct to start doing transactions.

• CPU AFFINITY
Allows you to lock the UE driver to a particular CPU. This is useful when running multiple UEs or MEEs on
a multi-processor system.

• ADDITIONAL OPTIONS: (SEPARATE WITH A “;”)

Reserved for Future Use

MARKET EMULATOR PROFILE

Once you have specified and created the User Emulator(s) you can create the Market Emulator(s). You define the
MEE by right-clicking on the BenchCraft windows and select New MEE. You will need to specify the following for a
Market Emulator:

• NAME
This is the name of the Market Emulator. The default is fine here.

• DESCRIPTION
Optional text string for some unique description of the Market Emulator

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 31

 • MACHINE NAME
This is the machine name where the Market Emulator driver will run.

• LOG FILE NAME

This is the fully qualified path and file name for the MEE log.

• SUT SERVER TO CONNECT

This option specifies the system name where an instance of SUT_MEE_Server is running. If you are
running multiple instances of SUT_MEE_Server on a single machine, you must also specify the port
number that the particular SUT_MEE_Server is listening on. This port is specified when you start the
SUT_MEE_Server.exe. See above.

• SIZE OF TARGET DB
This is where you plug in the number of customer in the target tpce database.

• SCALE DOWN
You may configure BenchCraft to run with fewer active customers than those configured. Check the box
and select the number of customers to access. NOTE: This is NOT a legal option for an audited run.

• CPU AFFINITY
Allows you to lock the MEE driver to a particular CPU. This is useful when running multiple UEs or MEEs
on a multi-processor system.

• ADDITIONAL OPTIONS: (SEPARATE WITH A “;”)

Reserved for Future Use

USER GROUPS

Once you have specified and created the User Emulator(s) and the Market Emulator(s) you can create the Users.
Simply click on the Users button on the toolbar and then right-click on the BenchCraft window and select New.
You will need to specify the following for a User:

• USER EMULATOR ENGINE NAME

This is the name of a User Emulator defined when creating User Emulator drivers above.

• NUMBER OF USERS
Enter the number of desired user threads for this User Emulator driver.

• ENABLE PARTITIONING
Checking this box will allow you to partition the user group across a subset of the total customer
population. If you check this box you will need to specify the Start from Customer, the Customer Count
from Starting, and the Percent Inside the Partition.

• SUT FRONT-END SERVER

This option specifies the system name where an instance of SUT_CE_Server is running. If you are running
multiple instances of SUT_CE_Server on a single machine, you must also specify the port number that the

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 32

 particular SUT_CE_Server is listening on. This port is specified when you start the SUT_CE_Server.exe.
See above.

BENCHCRAFT PARAMETER SETS

BenchCraft Parameter sets allow you to specify different benchmark related parameter settings. This is very
similar to TPC-C. To create a new parameter set click on the Parms icon and then right-click on the BenchCraft
windows and select New.

You must check the Checkpoint Interval and make sure it is set for 450 seconds. This matches the TPC-E 1.4.0
specification requirements. See Clause 6.6.5.3.

You can dial in any values appropriate to your testing. BenchCraft will do some basic checking to make sure that
the percentages are 100%, etc. After you have built a parameter set, you can modify the Driver for the User
Engine and specify the new parameter set.

PACING AND USERS

The “pacing” of transaction generation and execution is controlled by the Pacing value in the parameter set. The
value set here directly relates to the number of users you must define. The higher you set the pacing value, the
fewer users will be required to achieve the desired performance. Conversely, if you set the pacing value lower,
you will need to define more users. A value of 399999 effectively disables the pacing algorithm.

The number of users, along with the per-user transaction rate, controls how many total transactions are executed
against the database. The total number of users multiplied by the transaction rate per user divided by 10 equals
the approximate tpsE rate.

BENCHCRAFT RUNTIME

The process for running the workload is as follows:

• Start BenchCraft and either create a new profile or load a previously created profile.
• Click the Launch button to launch the UE(s) and the MEE(s).
• Click the Start button to start the workload.
o Initially, the users will connect to the SUT Servers based on the Connect Rate throttle specified in
the User Emulator.
o If the Start Rate throttle is greater than 0, then the users will start executing TPC-E transactions
as well.
o If you have set the Start Rate throttle to 0, you may override it by clicking on the Rates icon and
specifying a new value.
• Once the workload has started, you may click on the Stats button will show you the runtime stats.
• Once you have finished the run you must first pause the User Emulators and Market Emulators by clicking
on the Pause icon in the menu bar. This allows the workload to come to a graceful halt.
• After the system has entered a paused state, you may click on the Stop icon to stop all the drivers.
• When the workload is completely stopped, you can use the Tools menu selections to Sort/Merge the logs

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 33

 and produce the reports as you have done with BenchCraft for TPC-C.

POST-RUN PROCESSING

Once you have completed the performance run, there are several steps that need to be done. They are:

• TRADE CLEANUP
After every performance run, you must run the Trade_Cleanup.sql script. This script cancels any pending
or submitted trades that are “leftover” from the performance run (see Clause 3.3.12). Failure to run
Trade Cleanup will result in very unpredictable behavior on subsequent performance runs. The presence
of the pending trades will cause the Market Feed transaction to be flooded with requests which can
make the subsequent run very unstable and unusable.

You may run this script from the SQL Server Management Studio or via the SQLCMD command prompt.

• BENCHCRAFT – SORT/MERGE
When you have stopped the Customer Emulator(s) and Market Emulator(s) on BenchCraft you must run a
sort and merge of the BenchCraft log files. This process can be found on the Tools menu of the
BenchCraft user interface. This process will combine all the BenchCraft logs into a single log file for later
processing.

• BENCHCRAFT – TRANSACTION REPORT

The Transaction Report, found on the Tools menu of the BenchCraft user interface, will process the
unified log file created with the Sort/Merge function. You may optionally enter a time range, in 24 hour
format, to specify a particular portion of the performance run. The report generated can be
copied/imported into the 3rd sheet of TxnReportE.xls file for further processing and metric calculation..

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 34

PERFORMANCE CONSIDERATIONS AND TUNING

SQL SERVER TUNING

Tuning SQL Server for the test run is crucial to achieve optimum performance. The following is a description of SQL
Server tuning parameters that can be adjusted for a given system. Note that the SQL Server documentation
provides additional information on these tuning parameters. SQL Server handles a great deal of the tuning
internally. It monitors the use of the system and database(s) and dynamically configures itself accordingly. The
following SQL Server parameters may be modified:

SQL CONFIGURATION PARAMETER: SHOW ADVANCED OPTIONS

Many of the parameters listed below are not initially shown with sp_configure. Setting Show Advanced Options to
1 will allow you to view and modify these parameters.

SQL CONFIGURATION PARAMETER: AFFINITY MASK

Specifies a bit mask that SQL Server will use to associate its threads to the processors on your system. For
example, On a 4 CPU SMP machine you should set this value to a decimal 15 (0xF)

SQL CONFIGURATION PARAMETER: LIGHTWEIGHT POOLING

Lightweight pooling allows SQL Server to manage scheduling within the normal Windows Server thread structures.
Enabling lightweight pooling will reduce excessive context switching thereby reducing processor overhead. Setting
this option to 1 will cause SQL Server to switch to fiber mode scheduling. The default value is 0.

SQL CONFIGURATION PARAMETER: MIN SERVER MEMORY, MAX SERVER MEMORY

SQL Server will attempt to maximize memory usage if you let these values default. If you wish more control over
these settings, you may override the default settings. The Min Server Memory value specifies the amount of
memory that is guaranteed allocated to SQL Server. The Max Server Memory value specifies upper limit of
memory that can be allocated to SQL Server. The dynamic behavior of SQL Server allows you to leave these at
their defaults. In that condition, as the TPC-E system ramps-up, SQL Server will consume as much memory as the
operating system will allow. If you are running anything else on the system, like PerfMon, etc… it will prohibit SQL
Server from gaining access to additional memory. During the ramp-up phase, it is not unusual for SQL Server to
generate page faults. This condition will occur until SQL Server has finished bringing pages into the data cache.

SQL CONFIGURATION PARAMETER: RECOVERY INTERVAL

Specifies how often SQL Server will automatically issue checkpoints. In the case of TPC-E where checkpoints are
issued through the BenchCraft this automatic check-pointing should be avoided.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 35

SQL CONFIGURATION PARAMETER: USER CONNECTIONS

Specifies the number of user connections allowed to connect to SQL Server. Generally, it is best to leave this at its
default value. This will allow SQL Server to dynamically allocate user connections as they are needed. The cost per
user connection is about 50K.

Microsoft SQL Server TPC-E Benchmark Kit Reference Page 36