Download the full version of the ebook at

https://ebookultra.com

SQL Performance Explained Markus Winand

https://ebookultra.com/download/sql-performance-
explained-markus-winand/

Explore and download more ebook at https://ebookultra.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Building Performance Dashboards and Balanced Scorecards

with SQL Server Reporting Services 1st Edition Knight

https://ebookultra.com/download/building-performance-dashboards-and-
balanced-scorecards-with-sql-server-reporting-services-1st-edition-
knight/
ebookultra.com

Modern Fortran in Practice Markus

https://ebookultra.com/download/modern-fortran-in-practice-markus/

ebookultra.com

SQL Server T SQL Recipes 4th Edition Jason Brimhall

https://ebookultra.com/download/sql-server-t-sql-recipes-4th-edition-
jason-brimhall/

ebookultra.com

Microsoft SQL Server 2012 T SQL 1st Edition Tom Coffing

https://ebookultra.com/download/microsoft-sql-server-2012-t-sql-1st-
edition-tom-coffing/

ebookultra.com
European Democracies 9th Edition Markus M.L. Crepaz

https://ebookultra.com/download/european-democracies-9th-edition-
markus-m-l-crepaz/

ebookultra.com

Modernizing Enterprise Java 1st Edition Markus Eisele

https://ebookultra.com/download/modernizing-enterprise-java-1st-
edition-markus-eisele/

ebookultra.com

Essential SQL on SQL Server 2008 1st Edition Dr. Sikha

Bagui

https://ebookultra.com/download/essential-sql-on-sql-server-2008-1st-
edition-dr-sikha-bagui/

ebookultra.com

Myths Legends Explained Neil Philip

https://ebookultra.com/download/myths-legends-explained-neil-philip/

ebookultra.com

Joe Celko s SQL for Smarties Fourth Edition Advanced SQL

Programming Joe Celko

https://ebookultra.com/download/joe-celko-s-sql-for-smarties-fourth-
edition-advanced-sql-programming-joe-celko/

ebookultra.com
SQL Performance Explained Markus Winand Digital
Instant Download
Author(s): Markus Winand
ISBN(s): 9783950307825, 3950307826
Edition: Paperback
File Details: PDF, 1.17 MB
Year: 2012
Language: english
 MA CO
JOR VER
SQ S A
L D LL
ATA
BA

SQL SES

PERFORMANCE
EXPLAINED
ENGLISH EDITION

EVERYTHING DEVELOPERS NEED TO KNOW ABOUT SQL PERFORMANCE

MARKUS WINAND
License Agreement
This ebook is licensed for your personal enjoyment only. This ebook may
not be re-sold or given away to other people. If you would like to share
this book with another person, please purchase an additional copy for each
person. If you’re reading this book and did not purchase it, or it was not
purchased for your use only, then please return to
http://SQL-Performance-Explained.com/
and purchase your own copy. Thank you for respecting the hard work of
the author.
Publisher:
Markus Winand

Maderspergerstasse 1-3/9/11
1160 Wien
AUSTRIA
<[email protected]>

Copyright © 2012 Markus Winand

All rights reserved. No part of this publication may be reproduced, stored,

or transmitted in any form or by any means —electronic, mechanical,
photocopying, recording, or otherwise — without the prior consent of the
publisher.

Many of the names used by manufacturers and sellers to distinguish their

products are trademarked. Wherever such designations appear in this book,
and we were aware of a trademark claim, the names have been printed in
all caps or initial caps.

While every precaution has been taken in the preparation of this book, the
publisher and author assume no responsibility for errors and omissions, or
for damages resulting from the use of the information contained herein.

The book solely reflects the author’s views. The database vendors men-
tioned have neither supported the work financially nor verified the content.

DGS - Druck- u. Graphikservice GmbH — Wien — Austria

Cover design:
tomasio.design — Mag. Thomas Weninger — Wien — Austria

Cover photo:
Brian Arnold — Turriff — UK

Copy editor:
Nathan Ingvalson — Graz — Austria

2014-08-26
SQL Performance Explained

Everything developers need to

know about SQL performance

Markus Winand
Vienna, Austria
Contents
Preface ............................................................................................ vi

1. Anatomy of an Index ...................................................................... 1

The Index Leaf Nodes .................................................................. 2
The Search Tree (B-Tree) .............................................................. 4
Slow Indexes, Part I .................................................................... 6

2. The Where Clause ......................................................................... 9

The Equality Operator .................................................................. 9
Primary Keys ....................................................................... 10
Concatenated Indexes .......................................................... 12
Slow Indexes, Part II ............................................................ 18
Functions .................................................................................. 24
Case-Insensitive Search Using UPPER or LOWER .......................... 24
User-Defined Functions ........................................................ 29
Over-Indexing ...................................................................... 31
Parameterized Queries ............................................................... 32
Searching for Ranges ................................................................. 39
Greater, Less and BETWEEN ..................................................... 39
Indexing LIKE Filters ............................................................. 45
Index Merge ........................................................................ 49
Partial Indexes ........................................................................... 51
NULL in the Oracle Database ....................................................... 53
Indexing NULL ....................................................................... 54
NOT NULL Constraints ............................................................ 56
Emulating Partial Indexes ..................................................... 60
Obfuscated Conditions ............................................................... 62
Date Types .......................................................................... 62
Numeric Strings .................................................................. 68
Combining Columns ............................................................ 70
Smart Logic ......................................................................... 72
Math .................................................................................. 77

iv
 SQL Performance Explained

3. Performance and Scalability ......................................................... 79

Performance Impacts of Data Volume ......................................... 80
Performance Impacts of System Load .......................................... 85
Response Time and Throughput ................................................. 87

4. The Join Operation ....................................................................... 91

Nested Loops ............................................................................ 92
Hash Join ................................................................................. 101
Sort Merge .............................................................................. 109

5. Clustering Data ........................................................................... 111

Index Filter Predicates Used Intentionally ................................... 112
Index-Only Scan ........................................................................ 116
Index-Organized Tables ............................................................. 122

6. Sorting and Grouping ................................................................. 129

Indexing Order By .................................................................... 130
Indexing ASC, DESC and NULLS FIRST/LAST ...................................... 134
Indexing Group By .................................................................... 139

7. Partial Results ............................................................................ 143

Querying Top-N Rows ............................................................... 143
Paging Through Results ............................................................ 147
Using Window Functions for Pagination .................................... 156

8. Modifying Data .......................................................................... 159

Insert ...................................................................................... 159
Delete ...................................................................................... 162
Update .................................................................................... 163

A. Execution Plans .......................................................................... 165

Oracle Database ....................................................................... 166
PostgreSQL ............................................................................... 172
SQL Server ............................................................................... 180
MySQL ..................................................................................... 188

Index ............................................................................................. 193

v
Preface

Developers Need to Index

SQL performance problems are as old as SQL itself— some might even say
that SQL is inherently slow. Although this might have been true in the early
days of SQL, it is definitely not true anymore. Nevertheless SQL performance
problems are still commonplace. How does this happen?

The SQL language is perhaps the most successful fourth-generation

programming language (4GL). Its main benefit is the capability to separate
“what” and “how”. An SQL statement is a straight description what is needed
without instructions as to how to get it done. Consider the following
example:

SELECT date_of_birth
FROM employees
WHERE last_name = 'WINAND'

The SQL query reads like an English sentence that explains the requested
data. Writing SQL statements generally does not require any knowledge
about inner workings of the database or the storage system (such as disks,
files, etc.). There is no need to tell the database which files to open or how
to find the requested rows. Many developers have years of SQL experience
yet they know very little about the processing that happens in the database.

The separation of concerns — what is needed versus how to get it — works

remarkably well in SQL, but it is still not perfect. The abstraction reaches
its limits when it comes to performance: the author of an SQL statement
by definition does not care how the database executes the statement.
Consequently, the author is not responsible for slow execution. However,
experience proves the opposite; i.e., the author must know a little bit about
the database to prevent performance problems.

It turns out that the only thing developers need to learn is how to index.
Database indexing is, in fact, a development task. That is because the
most important information for proper indexing is not the storage system
configuration or the hardware setup. The most important information for
indexing is how the application queries the data. This knowledge —about

vi
 Preface: Developers Need to Index

the access path— is not very accessible to database administrators (DBAs) or

external consultants. Quite some time is needed to gather this information
through reverse engineering of the application: development, on the other
hand, has that information anyway.

This book covers everything developers need to know about indexes — and
nothing more. To be more precise, the book covers the most important
index type only: the B-tree index.

The B-tree index works almost identically in many databases. The book only
uses the terminology of the Oracle® database, but the principles apply to
other databases as well. Side notes provide relevant information for MySQL,
PostgreSQL and SQL Server®.

The structure of the book is tailor-made for developers; most chapters

correspond to a particular part of an SQL statement.

CHAPTER 1 - Anatomy of an Index

The first chapter is the only one that doesn’t cover SQL specifically; it
is about the fundamental structure of an index. An understanding of
the index structure is essential to following the later chapters — don’t
skip this!

Although the chapter is rather short —only about eight pages —

after working through the chapter you will already understand the
phenomenon of slow indexes.

CHAPTER 2 - The Where Clause

This is where we pull out all the stops. This chapter explains all aspects
of the where clause, from very simple single column lookups to complex
clauses for ranges and special cases such as LIKE.

This chapter makes up the main body of the book. Once you learn to
use these techniques, you will write much faster SQL.

CHAPTER 3 - Performance and Scalability

This chapter is a little digression about performance measurements
and database scalability. See why adding hardware is not the best
solution to slow queries.

CHAPTER 4 - The Join Operation

Back to SQL: here you will find an explanation of how to use indexes
to perform a fast table join.

vii
 Preface: Developers Need to Index

CHAPTER 5 - Clustering Data

Have you ever wondered if there is any difference between selecting a
single column or all columns? Here is the answer —along with a trick
to get even better performance.

CHAPTER 6 - Sorting and Grouping

Even order by and group by can use indexes.

CHAPTER 7 - Partial Results

This chapter explains how to benefit from a “pipelined” execution if
you don’t need the full result set.

CHAPTER 8 - Insert, Delete and Update

How do indexes affect write performance? Indexes don’t come for
free — use them wisely!

APPENDIX A - Execution Plans

Asking the database how it executes a statement.

viii
Chapter 1

Anatomy of an Index

“An index makes the query fast” is the most basic explanation of an index I
have ever seen. Although it describes the most important aspect of an index
very well, it is —unfortunately—not sufficient for this book. This chapter
describes the index structure in a less superficial way but doesn’t dive too
deeply into details. It provides just enough insight for one to understand
the SQL performance aspects discussed throughout the book.

An index is a distinct structure in the database that is built using the

create index statement. It requires its own disk space and holds a copy
of the indexed table data. That means that an index is pure redundancy.
Creating an index does not change the table data; it just creates a new data
structure that refers to the table. A database index is, after all, very much
like the index at the end of a book: it occupies its own space, it is highly
redundant, and it refers to the actual information stored in a different
place.

Clustered Indexes
SQL Server and MySQL (using InnoDB) take a broader view of what
“index” means. They refer to tables that consist of the index structure
only as clustered indexes. These tables are called Index-Organized
Tables (IOT) in the Oracle database.
Chapter 5, “Clustering Data”, describes them in more detail and
explains their advantages and disadvantages.

Searching in a database index is like searching in a printed telephone

directory. The key concept is that all entries are arranged in a well-defined
order. Finding data in an ordered data set is fast and easy because the sort
order determines each entries position.

1
 Chapter 1: Anatomy of an Index

A database index is, however, more complex than a printed directory

because it undergoes constant change. Updating a printed directory for
every change is impossible for the simple reason that there is no space
between existing entries to add new ones. A printed directory bypasses this
problem by only handling the accumulated updates with the next printing.
An SQL database cannot wait that long. It must process insert, delete and
update statements immediately, keeping the index order without moving
large amounts of data.

The database combines two data structures to meet the challenge: a doubly
linked list and a search tree. These two structures explain most of the
database’s performance characteristics.

The Index Leaf Nodes

The primary purpose of an index is to provide an ordered representation of
the indexed data. It is, however, not possible to store the data sequentially
because an insert statement would need to move the following entries to
make room for the new one. Moving large amounts of data is very time-
consuming so the insert statement would be very slow. The solution to
the problem is to establish a logical order that is independent of physical
order in memory.

The logical order is established via a doubly linked list. Every node has links
to two neighboring entries, very much like a chain. New nodes are inserted
between two existing nodes by updating their links to refer to the new
node. The physical location of the new node doesn’t matter because the
doubly linked list maintains the logical order.

The data structure is called a doubly linked list because each node refers
to the preceding and the following node. It enables the database to read
the index forwards or backwards as needed. It is thus possible to insert
new entries without moving large amounts of data—it just needs to change
some pointers.

Doubly linked lists are also used for collections (containers) in many
programming languages.

2
 The Index Leaf Nodes

Programming Language Name

Java java.util.LinkedList
.NET Framework System.Collections.Generic.LinkedList
C++ std::list

Databases use doubly linked lists to connect the so-called index leaf nodes.
Each leaf node is stored in a database block or page; that is, the database’s
smallest storage unit. All index blocks are of the same size —typically a few
kilobytes. The database uses the space in each block to the extent possible
and stores as many index entries as possible in each block. That means
that the index order is maintained on two different levels: the index entries
within each leaf node, and the leaf nodes among each other using a doubly
linked list.

Figure 1.1. Index Leaf Nodes and Corresponding Table Data

Index Leaf Nodes Table
(sort ed) (not sort ed)
2

lu 1
lu 2
lu 3

4
mn

co mn
co mn
co mn
mn
D
lu

WI

lu
co

RO

co

11 3C AF A 34 1 2
13 F3 91 A 27 5 9
18 6F B2
A 39 2 5
X 21 7 2
21 2C 50
27 0F 1B A 11 1 6
27 52 55
A 35 8 3
X 27 3 2
34 0D 1E
35 44 53 A 18 3 6
39 24 5D A 13 7 4

Figure 1.1 illustrates the index leaf nodes and their connection to the table
data. Each index entry consists of the indexed columns (the key, column 2)
and refers to the corresponding table row (via ROWID or RID). Unlike the
index, the table data is stored in a heap structure and is not sorted at all.
There is neither a relationship between the rows stored in the same table
block nor is there any connection between the blocks.

3
 Chapter 1: Anatomy of an Index

The Search Tree (B-Tree)

The index leaf nodes are stored in an arbitrary order —the position on the
disk does not correspond to the logical position according to the index
order. It is like a telephone directory with shuffled pages. If you search
for “Smith” but first open the directory at “Robinson”, it is by no means
granted that Smith follows Robinson. A database needs a second structure
to find the entry among the shuffled pages quickly: a balanced search tree—
in short: the B-tree.

Figure 1.2. B-tree Structure

es
Branch Node Leaf Nodes

od

es
e
N
od

od
ch
N

N
40 4A 1B

an
t

af
o
Ro

Le
Br
43 9F 71
46 A2 D2 11 3C AF
13 F3 91
18 6F B2

21 2C 50
18 27 0F 1B
27 27 52 55
39

46 8B 1C 34 0D 1E
35 44 53
39 24 5D

53 A0 A1 40 4A 1B
43 9F 71
53 0D 79 46 A2 D2

46 46 8B 1C
53 A0 A1
53 0D 79

53 39
46
53
57 55 9C F6

57 55 9C F6 83
98
83 57 B1 C1
57 50 29

83 57 B1 C1 67 C4 6B
83 FF 9D
83 AF E9

57 50 29 84 80 64
86 4C 2F
88 06 5B

89 6A 3E
88 90 7D 9A
94 94 36 D4
67 C4 6B 98

95 EA 37
98 5E B2
83 FF 9D 98 D8 4F

83 AF E9

Figure 1.2 shows an example index with 30 entries. The doubly linked list
establishes the logical order between the leaf nodes. The root and branch
nodes support quick searching among the leaf nodes.

The figure highlights a branch node and the leaf nodes it refers to. Each
branch node entry corresponds to the biggest value in the respective leaf
node. That is, 46 in the first leaf node so that the first branch node entry
is also 46. The same is true for the other leaf nodes so that in the end the

4
 The Search Tree (B-Tree)

branch node has the values 46, 53, 57 and 83. According to this scheme, a
branch layer is built up until all the leaf nodes are covered by a branch node.

The next layer is built similarly, but on top of the first branch node level.
The procedure repeats until all keys fit into a single node, the root node.
The structure is a balanced search tree because the tree depth is equal at
every position; the distance between root node and leaf nodes is the same
everywhere.

Note
A B-tree is a balanced tree—not a binary tree.

Once created, the database maintains the index automatically. It applies

every insert, delete and update to the index and keeps the tree in balance,
thus causing maintenance overhead for write operations. Chapter 8,
“Modifying Data”, explains this in more detail.

Figure 1.3. B-Tree Traversal

46 8B 1C
53 A0 A1
46 53 0D 79
39
53
83
57
98 55 9C F6
83
57 B1 C1
57 50 29

Figure 1.3 shows an index fragment to illustrate a search for the key “57”.
The tree traversal starts at the root node on the left-hand side. Each entry
is processed in ascending order until a value is greater than or equal to (>=)
the search term (57). In the figure it is the entry 83. The database follows
the reference to the corresponding branch node and repeats the procedure
until the tree traversal reaches a leaf node.

Important
The B-tree enables the database to find a leaf node quickly.

5
 Chapter 1: Anatomy of an Index

The tree traversal is a very efficient operation—so efficient that I refer to it

as the first power of indexing. It works almost instantly—even on a huge data
set. That is primarily because of the tree balance, which allows accessing
all elements with the same number of steps, and secondly because of the
logarithmic growth of the tree depth. That means that the tree depth grows
very slowly compared to the number of leaf nodes. Real world indexes with
millions of records have a tree depth of four or five. A tree depth of six is
hardly ever seen. The box “Logarithmic Scalability” describes this in more
detail.

Slow Indexes, Part I

Despite the efficiency of the tree traversal, there are still cases where an
index lookup doesn’t work as fast as expected. This contradiction has fueled
the myth of the “degenerated index” for a long time. The myth proclaims
an index rebuild as the miracle solution. The real reason trivial statements
can be slow —even when using an index —can be explained on the basis of
the previous sections.

The first ingredient for a slow index lookup is the leaf node chain. Consider
the search for “57” in Figure 1.3 again. There are obviously two matching
entries in the index. At least two entries are the same, to be more precise:
the next leaf node could have further entries for “57”. The database must
read the next leaf node to see if there are any more matching entries. That
means that an index lookup not only needs to perform the tree traversal,
it also needs to follow the leaf node chain.

The second ingredient for a slow index lookup is accessing the table.
Even a single leaf node might contain many hits — often hundreds. The
corresponding table data is usually scattered across many table blocks (see
Figure 1.1, “Index Leaf Nodes and Corresponding Table Data”). That means
that there is an additional table access for each hit.

An index lookup requires three steps: (1) the tree traversal; (2) following the
leaf node chain; (3) fetching the table data. The tree traversal is the only
step that has an upper bound for the number of accessed blocks—the index
depth. The other two steps might need to access many blocks—they cause
a slow index lookup.

6
 Slow Indexes, Part I

Logarithmic Scalability
In mathematics, the logarithm of a number to a given base is the
power or exponent to which the base must be raised in order to
1
produce the number [Wikipedia ].
In a search tree the base corresponds to the number of entries per
branch node and the exponent to the tree depth. The example index
in Figure 1.2 holds up to four entries per node and has a tree depth
3
of three. That means that the index can hold up to 64 (4 ) entries. If
4
it grows by one level, it can already hold 256 entries (4 ). Each time
a level is added, the maximum number of index entries quadruples.
The logarithm reverses this function. The tree depth is therefore
log4(number-of-index-entries).
The logarithmic growth enables
Tree Depth Index Entries
the example index to search a
million records with ten tree 3 64
levels, but a real world index is 4 256
even more efficient. The main 5 1,024
factor that affects the tree depth,
6 4,096
and therefore the lookup perfor-
mance, is the number of entries 7 16,384
in each tree node. This number 8 65,536
corresponds to— mathematically
9 262,144
speaking — the basis of the loga-
rithm. The higher the basis, the 10 1,048,576
shallower the tree, the faster the
traversal.
Databases exploit this concept to a maximum extent and put as many
entries as possible into each node— often hundreds. That means that
every new index level supports a hundred times more entries.

1
http://en.wikipedia.org/wiki/Logarithm

7
 Chapter 1: Anatomy of an Index

The origin of the “slow indexes” myth is the misbelief that an index lookup
just traverses the tree, hence the idea that a slow index must be caused by a
“broken” or “unbalanced” tree. The truth is that you can actually ask most
databases how they use an index. The Oracle database is rather verbose in
this respect and has three distinct operations that describe a basic index
lookup:

INDEX UNIQUE SCAN

The INDEX UNIQUE SCAN performs the tree traversal only. The Oracle
database uses this operation if a unique constraint ensures that the
search criteria will match no more than one entry.

INDEX RANGE SCAN

The INDEX RANGE SCAN performs the tree traversal and follows the leaf
node chain to find all matching entries. This is the fallback operation
if multiple entries could possibly match the search criteria.

TABLE ACCESS BY INDEX ROWID

The TABLE ACCESS BY INDEX ROWID operation retrieves the row from
the table. This operation is (often) performed for every matched record
from a preceding index scan operation.

The important point is that an INDEX RANGE SCAN can potentially read a large
part of an index. If there is one more table access for each row, the query
can become slow even when using an index.

8
Chapter 2

The Where Clause

The previous chapter described the structure of indexes and explained the
cause of poor index performance. In the next step we learn how to spot
and avoid these problems in SQL statements. We start by looking at the
where clause.

The where clause defines the search condition of an SQL statement, and it
thus falls into the core functional domain of an index: finding data quickly.
Although the where clause has a huge impact on performance, it is often
phrased carelessly so that the database has to scan a large part of the index.
The result: a poorly written where clause is the first ingredient of a slow
query.

This chapter explains how different operators affect index usage and how
to make sure that an index is usable for as many queries as possible. The
last section shows common anti-patterns and presents alternatives that
deliver better performance.

The Equality Operator

The equality operator is both the most trivial and the most frequently
used SQL operator. Indexing mistakes that affect performance are still
very common and where clauses that combine multiple conditions are
particularly vulnerable.

This section shows how to verify index usage and explains how
concatenated indexes can optimize combined conditions. To aid
understanding, we will analyze a slow query to see the real world impact
of the causes explained in Chapter 1.

9
 Chapter 2: The Where Clause

Primary Keys
We start with the simplest yet most common where clause: the primary key
lookup. For the examples throughout this chapter we use the EMPLOYEES
table defined as follows:

CREATE TABLE employees (

employee_id NUMBER NOT NULL,
first_name VARCHAR2(1000) NOT NULL,
last_name VARCHAR2(1000) NOT NULL,
date_of_birth DATE NOT NULL,
phone_number VARCHAR2(1000) NOT NULL,
CONSTRAINT employees_pk PRIMARY KEY (employee_id)
);

The database automatically creates an index for the primary key. That
means there is an index on the EMPLOYEE_ID column, even though there is
no create index statement.

The following query uses the primary key to retrieve an employee’s name:

SELECT first_name, last_name

FROM employees
WHERE employee_id = 123

The where clause cannot match multiple rows because the primary key
constraint ensures uniqueness of the EMPLOYEE_ID values. The database does
not need to follow the index leaf nodes —it is enough to traverse the index
tree. We can use the so-called execution plan for verification:

---------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
---------------------------------------------------------------
| 0 |SELECT STATEMENT | | 1 | 2 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 2 |
|*2 | INDEX UNIQUE SCAN | EMPLOYEES_PK | 1 | 1 |
---------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
2 - access("EMPLOYEE_ID"=123)

10
 Primary Keys

The Oracle execution plan shows an INDEX UNIQUE SCAN — the operation that
only traverses the index tree. It fully utilizes the logarithmic scalability of
the index to find the entry very quickly —almost independent of the table
size.

Tip
The execution plan (sometimes explain plan or query plan) shows the
steps the database takes to execute an SQL statement. Appendix A on
page 165 explains how to retrieve and read execution plans with
other databases.

After accessing the index, the database must do one more step to
fetch the queried data (FIRST_NAME, LAST_NAME) from the table storage:
the TABLE ACCESS BY INDEX ROWID operation. This operation can become a
performance bottleneck —as explained in “Slow Indexes, Part I”— but there
is no such risk in connection with an INDEX UNIQUE SCAN. This operation
cannot deliver more than one entry so it cannot trigger more than one table
access. That means that the ingredients of a slow query are not present
with an INDEX UNIQUE SCAN.

Primary Keys without Unique Index

A primary key does not necessarily need a unique index — you can
use a non-unique index as well. In that case the Oracle database
does not use an INDEX UNIQUE SCAN but instead the INDEX RANGE SCAN
operation. Nonetheless, the constraint still maintains the uniqueness
of keys so that the index lookup delivers at most one entry.
One of the reasons for using non-unique indexes for a primary keys
are deferrable constraints. As opposed to regular constraints, which
are validated during statement execution, the database postpones
the validation of deferrable constraints until the transaction is
committed. Deferred constraints are required for inserting data into
tables with circular dependencies.

11
 Chapter 2: The Where Clause

Concatenated Indexes
Even though the database creates the index for the primary key
automatically, there is still room for manual refinements if the key consists
of multiple columns. In that case the database creates an index on all
primary key columns — a so-called concatenated index (also known as multi-
column, composite or combined index). Note that the column order of a
concatenated index has great impact on its usability so it must be chosen
carefully.

For the sake of demonstration, let’s assume there is a company merger.

The employees of the other company are added to our EMPLOYEES table so it
becomes ten times as large. There is only one problem: the EMPLOYEE_ID is
not unique across both companies. We need to extend the primary key by
an extra identifier — e.g., a subsidiary ID. Thus the new primary key has two
columns: the EMPLOYEE_ID as before and the SUBSIDIARY_ID to reestablish
uniqueness.

The index for the new primary key is therefore defined in the following way:

CREATE UNIQUE INDEX employee_pk

ON employees (employee_id, subsidiary_id);

A query for a particular employee has to take the full primary key into
account— that is, the SUBSIDIARY_ID column also has to be used:

SELECT first_name, last_name

FROM employees
WHERE employee_id = 123
AND subsidiary_id = 30;
---------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
---------------------------------------------------------------
| 0 |SELECT STATEMENT | | 1 | 2 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 2 |
|*2 | INDEX UNIQUE SCAN | EMPLOYEES_PK | 1 | 1 |
---------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("EMPLOYEE_ID"=123 AND "SUBSIDIARY_ID"=30)

12
 Concatenated Indexes

Whenever a query uses the complete primary key, the database can use
an INDEX UNIQUE SCAN — no matter how many columns the index has. But
what happens when using only one of the key columns, for example, when
searching all employees of a subsidiary?

SELECT first_name, last_name

FROM employees
WHERE subsidiary_id = 20;

----------------------------------------------------
| Id | Operation | Name | Rows | Cost |
----------------------------------------------------
| 0 | SELECT STATEMENT | | 106 | 478 |
|* 1 | TABLE ACCESS FULL| EMPLOYEES | 106 | 478 |
----------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
1 - filter("SUBSIDIARY_ID"=20)

The execution plan reveals that the database does not use the index. Instead
it performs a FULL TABLE SCAN. As a result the database reads the entire table
and evaluates every row against the where clause. The execution time grows
with the table size: if the table grows tenfold, the FULL TABLE SCAN takes ten
times as long. The danger of this operation is that it is often fast enough
in a small development environment, but it causes serious performance
problems in production.

Full Table Scan

The operation TABLE ACCESS FULL, also known as full table scan, can
be the most efficient operation in some cases anyway, in particular
when retrieving a large part of the table.
This is partly due to the overhead for the index lookup itself, which
does not happen for a TABLE ACCESS FULL operation. This is mostly
because an index lookup reads one block after the other as the
database does not know which block to read next until the current
block has been processed. A FULL TABLE SCAN must get the entire table
anyway so that the database can read larger chunks at a time (multi
block read). Although the database reads more data, it might need to
execute fewer read operations.

13
 Chapter 2: The Where Clause

The database does not use the index because it cannot use single columns
from a concatenated index arbitrarily. A closer look at the index structure
makes this clear.

A concatenated index is just a B-tree index like any other that keeps the
indexed data in a sorted list. The database considers each column according
to its position in the index definition to sort the index entries. The first
column is the primary sort criterion and the second column determines the
order only if two entries have the same value in the first column and so on.

Important
A concatenated index is one index across multiple columns.

The ordering of a two-column index is therefore like the ordering of a

telephone directory: it is first sorted by surname, then by first name. That
means that a two-column index does not support searching on the second
column alone; that would be like searching a telephone directory by first
name.

Figure 2.1. Concatenated Index D

_I

Index-Tree
D
_I

RY
EE

IA
D

OY

ID
_I
D

PL

BS
_I

_I

RY
D

EM

SU
_I

RY

EE

IA
EE

IA

OY

ID

123 20 ROWID
OY

ID

PL

BS
PL

BS

EM

SU

123 21 ROWID
EM

SU

123 18 123 27 ROWID

121 25
123 27
126 30
125 30
131 11 124 10 ROWID
126 30
124 20 ROWID
125 30 ROWID

The index excerpt in Figure 2.1 shows that the entries for subsidiary 20 are
not stored next to each other. It is also apparent that there are no entries
with SUBSIDIARY_ID = 20 in the tree, although they exist in the leaf nodes.
The tree is therefore useless for this query.

14
 Concatenated Indexes

Tip
Visualizing an index helps in understanding what queries the index
supports. You can query the database to retrieve the entries in index
order (SQL:2008 syntax, see page 144 for proprietary solutions
using LIMIT, TOP or ROWNUM):

SELECT <INDEX COLUMN LIST>

FROM <TABLE>
ORDER BY <INDEX COLUMN LIST>
FETCH FIRST 100 ROWS ONLY;

If you put the index definition and table name into the query, you
will get a sample from the index. Ask yourself if the requested rows
are clustered in a central place. If not, the index tree cannot help find
that place.

We could, of course, add another index on SUBSIDIARY_ID to improve query

speed. There is however a better solution — at least if we assume that
searching on EMPLOYEE_ID alone does not make sense.

We can take advantage of the fact that the first index column is always
usable for searching. Again, it is like a telephone directory: you don’t need
to know the first name to search by last name. The trick is to reverse the
index column order so that the SUBSIDIARY_ID is in the first position:

CREATE UNIQUE INDEX EMPLOYEES_PK

ON EMPLOYEES (SUBSIDIARY_ID, EMPLOYEE_ID);

Both columns together are still unique so queries with the full primary
key can still use an INDEX UNIQUE SCAN but the sequence of index entries is
entirely different. The SUBSIDIARY_ID has become the primary sort criterion.
That means that all entries for a subsidiary are in the index consecutively
so the database can use the B-tree to find their location.

15
 Chapter 2: The Where Clause

Important
The most important consideration when defining a concatenated
index is how to choose the column order so it can be used as often
as possible.

The execution plan confirms that the database uses the “reversed” index.
The SUBSIDIARY_ID alone is not unique anymore so the database must
follow the leaf nodes in order to find all matching entries: it is therefore
using the INDEX RANGE SCAN operation.

--------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
--------------------------------------------------------------
| 0 |SELECT STATEMENT | | 106 | 75 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 106 | 75 |
|*2 | INDEX RANGE SCAN | EMPLOYEE_PK | 106 | 2 |
--------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
2 - access("SUBSIDIARY_ID"=20)

In general, a database can use a concatenated index when searching with

the leading (leftmost) columns. An index with three columns can be used
when searching for the first column, when searching with the first two
columns together, and when searching using all columns.

Even though the two-index solution delivers very good select performance
as well, the single-index solution is preferable. It not only saves storage
space, but also the maintenance overhead for the second index. The fewer
indexes a table has, the better the insert, delete and update performance.

To define an optimal index you must understand more than just how
indexes work — you must also know how the application queries the data.
This means you have to know the column combinations that appear in the
where clause.

Defining an optimal index is therefore very difficult for external consultants

because they don’t have an overview of the application’s access paths.
Consultants can usually consider one query only. They do not exploit
the extra benefit the index could bring for other queries. Database
administrators are in a similar position as they might know the database
schema but do not have deep insight into the access paths.

16
 Concatenated Indexes

The only place where the technical database knowledge meets the
functional knowledge of the business domain is the development
department. Developers have a feeling for the data and know the access
path. They can properly index to get the best benefit for the overall
application without much effort.

17
 Chapter 2: The Where Clause

Slow Indexes, Part II

The previous section explained how to gain additional benefits from an
existing index by changing its column order, but the example considered
only two SQL statements. Changing an index, however, may affect all
queries on the indexed table. This section explains the way databases pick
an index and demonstrates the possible side effects when changing existing
indexes.

The adopted EMPLOYEE_PK index improves the performance of all queries that
search by subsidiary only. It is however usable for all queries that search
by SUBSIDIARY_ID — regardless of whether there are any additional search
criteria. That means the index becomes usable for queries that used to use
another index with another part of the where clause. In that case, if there
are multiple access paths available it is the optimizer’s job to choose the
best one.

The Query Optimizer

The query optimizer, or query planner, is the database component
that transforms an SQL statement into an execution plan. This
process is also called compiling or parsing. There are two distinct
optimizer types.
Cost-based optimizers (CBO) generate many execution plan variations
and calculate a cost value for each plan. The cost calculation is based
on the operations in use and the estimated row numbers. In the
end the cost value serves as the benchmark for picking the “best”
execution plan.
Rule-based optimizers (RBO) generate the execution plan using a hard-
coded rule set. Rule based optimizers are less flexible and are seldom
used today.

18
 Slow Indexes, Part II

Changing an index might have unpleasant side effects as well. In our

example, it is the internal telephone directory application that has become
very slow since the merger. The first analysis identified the following query
as the cause for the slowdown:

SELECT first_name, last_name, subsidiary_id, phone_number

FROM employees
WHERE last_name = 'WINAND'
AND subsidiary_id = 30;

The execution plan is:

Example 2.1. Execution Plan with Revised Primary Key Index

---------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
---------------------------------------------------------------
| 0 |SELECT STATEMENT | | 1 | 30 |
|*1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 30 |
|*2 | INDEX RANGE SCAN | EMPLOYEES_PK | 40 | 2 |
---------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("LAST_NAME"='WINAND')
2 - access("SUBSIDIARY_ID"=30)

The execution plan uses an index and has an overall cost value of 30.
So far, so good. It is however suspicious that it uses the index we just
changed— that is enough reason to suspect that our index change caused
the performance problem, especially when bearing the old index definition
in mind— it started with the EMPLOYEE_ID column which is not part of the
where clause at all. The query could not use that index before.

For further analysis, it would be nice to compare the execution plan before
and after the change. To get the original execution plan, we could just
deploy the old index definition again, however most databases offer a
simpler method to prevent using an index for a specific query. The following
example uses an Oracle optimizer hint for that purpose.

SELECT /*+ NO_INDEX(EMPLOYEES EMPLOYEE_PK) */

first_name, last_name, subsidiary_id, phone_number
FROM employees
WHERE last_name = 'WINAND'
AND subsidiary_id = 30;

19
 Chapter 2: The Where Clause

The execution plan that was presumably used before the index change did
not use an index at all:

----------------------------------------------------
| Id | Operation | Name | Rows | Cost |
----------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 477 |
|* 1 | TABLE ACCESS FULL| EMPLOYEES | 1 | 477 |
----------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
1 - filter("LAST_NAME"='WINAND' AND "SUBSIDIARY_ID"=30)

Even though the TABLE ACCESS FULL must read and process the entire table,
it seems to be faster than using the index in this case. That is particularly
unusual because the query matches one row only. Using an index to find
a single row should be much faster than a full table scan, but in this case
it is not. The index seems to be slow.

In such cases it is best to go through each step of the troublesome execution

plan. The first step is the INDEX RANGE SCAN on the EMPLOYEES_PK index.
That index does not cover the LAST_NAME column—the INDEX RANGE SCAN can
consider the SUBSIDIARY_ID filter only; the Oracle database shows this in
the “Predicate Information” area— entry “2” of the execution plan. There
you can see the conditions that are applied for each operation.

Tip
Appendix A, “Execution Plans”, explains how to find the “Predicate
Information” for other databases.

The INDEX RANGE SCAN with operation ID 2 (Example 2.1 on page 19)
applies only the SUBSIDIARY_ID=30 filter. That means that it traverses the
index tree to find the first entry for SUBSIDIARY_ID 30. Next it follows the
leaf node chain to find all other entries for that subsidiary. The result of the
INDEX RANGE SCAN is a list of ROWIDs that fulfill the SUBSIDIARY_ID condition:
depending on the subsidiary size, there might be just a few ones or there
could be many hundreds.

The next step is the TABLE ACCESS BY INDEX ROWID operation. It uses the
ROWIDs from the previous step to fetch the rows —all columns— from the
table. Once the LAST_NAME column is available, the database can evaluate
the remaining part of the where clause. That means the database has to
fetch all rows for SUBSIDIARY_ID=30 before it can apply the LAST_NAME filter.

20
 Slow Indexes, Part II

The statement’s response time does not depend on the result set size
but on the number of employees in the particular subsidiary. If the
subsidiary has just a few members, the INDEX RANGE SCAN provides better
performance. Nonetheless a TABLE ACCESS FULL can be faster for a huge
subsidiary because it can read large parts from the table in one shot (see
“Full Table Scan” on page 13).

The query is slow because the index lookup returns many ROWIDs — one for
each employee of the original company— and the database must fetch them
individually. It is the perfect combination of the two ingredients that make
an index slow: the database reads a wide index range and has to fetch many
rows individually.

Choosing the best execution plan depends on the table’s data distribution
as well so the optimizer uses statistics about the contents of the database.
In our example, a histogram containing the distribution of employees over
subsidiaries is used. This allows the optimizer to estimate the number
of rows returned from the index lookup —the result is used for the cost
calculation.

Statistics
A cost-based optimizer uses statistics about tables, columns, and
indexes. Most statistics are collected on the column level: the number
of distinct values, the smallest and largest values (data range),
the number of NULL occurrences and the column histogram (data
distribution). The most important statistical value for a table is its
size (in rows and blocks).
The most important index statistics are the tree depth, the number
of leaf nodes, the number of distinct keys and the clustering factor
(see Chapter 5, “Clustering Data”).
The optimizer uses these values to estimate the selectivity of the
where clause predicates.

21
 Chapter 2: The Where Clause

If there are no statistics available— for example because they were deleted—
the optimizer uses default values. The default statistics of the Oracle
database suggest a small index with medium selectivity. They lead to the
estimate that the INDEX RANGE SCAN will return 40 rows. The execution plan
shows this estimation in the Rows column (again, see Example 2.1 on page
19). Obviously this is a gross underestimate, as there are 1000 employees
working for this subsidiary.

If we provide correct statistics, the optimizer does a better job. The

following execution plan shows the new estimation: 1000 rows for the
INDEX RANGE SCAN. Consequently it calculated a higher cost value for the
subsequent table access.

---------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
---------------------------------------------------------------
| 0 |SELECT STATEMENT | | 1 | 680 |
|*1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 680 |
|*2 | INDEX RANGE SCAN | EMPLOYEES_PK | 1000 | 4 |
---------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("LAST_NAME"='WINAND')
2 - access("SUBSIDIARY_ID"=30)

The cost value of 680 is even higher than the cost value for the execution
plan using the FULL TABLE SCAN (477, see page 20). The optimizer will
therefore automatically prefer the FULL TABLE SCAN.

This example of a slow index should not hide the fact that proper indexing
is the best solution. Of course searching on last name is best supported by
an index on LAST_NAME:

CREATE INDEX emp_name ON employees (last_name);

22
 Slow Indexes, Part II

Using the new index, the optimizer calculates a cost value of 3:

Example 2.2. Execution Plan with Dedicated Index

--------------------------------------------------------------
| Id | Operation | Name | Rows | Cost |
--------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 3 |
|* 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 3 |
|* 2 | INDEX RANGE SCAN | EMP_NAME | 1 | 1 |
--------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
1 - filter("SUBSIDIARY_ID"=30)
2 - access("LAST_NAME"='WINAND')

The index access delivers — according to the optimizer’s estimation— one

row only. The database thus has to fetch only that row from the table: this
is definitely faster than a FULL TABLE SCAN. A properly defined index is still
better than the original full table scan.

The two execution plans from Example 2.1 (page 19) and Example 2.2
are almost identical. The database performs the same operations and
the optimizer calculated similar cost values, nevertheless the second plan
performs much better. The efficiency of an INDEX RANGE SCAN may vary
over a wide range —especially when followed by a table access. Using an
index does not automatically mean a statement is executed in the best way
possible.

23
 Chapter 2: The Where Clause

Functions
The index on LAST_NAME has improved the performance considerably, but
it requires you to search using the same case (upper/lower) as is stored in
the database. This section explains how to lift this restriction without a
decrease in performance.

Note
MySQL 5.6 does not support function-based indexing as described
below. As an alternative, virtual columns were planned for MySQL
6.0 but were introduced in MariaDB 5.2 only.

Case-Insensitive Search Using UPPER or LOWER

Ignoring the case in a where clause is very simple. You can, for example,
convert both sides of the comparison to all caps notation:

SELECT first_name, last_name, phone_number

FROM employees
WHERE UPPER(last_name) = UPPER('winand');

Regardless of the capitalization used for the search term or the LAST_NAME
column, the UPPER function makes them match as desired.

Note
Another way for case-insensitive matching is to use a different
“collation”. The default collations used by SQL Server and MySQL do
not distinguish between upper and lower case letters— they are case-
insensitive by default.

24
 Case-Insensitive Search Using UPPER or LOWER

The logic of this query is perfectly reasonable but the execution plan is not:

----------------------------------------------------
| Id | Operation | Name | Rows | Cost |
----------------------------------------------------
| 0 | SELECT STATEMENT | | 10 | 477 |
|* 1 | TABLE ACCESS FULL| EMPLOYEES | 10 | 477 |
----------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
1 - filter(UPPER("LAST_NAME")='WINAND')

It is a return of our old friend the full table scan. Although there is an index
on LAST_NAME, it is unusable —because the search is not on LAST_NAME but
on UPPER(LAST_NAME). From the database’s perspective, that’s something
entirely different.

This is a trap we all might fall into. We recognize the relation between
LAST_NAME and UPPER(LAST_NAME) instantly and expect the database to “see”
it as well. In reality the optimizer’s view is more like this:

SELECT first_name, last_name, phone_number

FROM employees
WHERE BLACKBOX(...) = 'WINAND';

The UPPER function is just a black box. The parameters to the function
are not relevant because there is no general relationship between the
function’s parameters and the result.

Tip
Replace the function name with BLACKBOX to understand the opti-
mizer’s point of view.

Compile Time Evaluation

The optimizer can evaluate the expression on the right-hand side
during “compile time” because it has all the input parameters. The
Oracle execution plan (“Predicate Information” section) therefore
only shows the upper case notation of the search term. This behavior
is very similar to a compiler that evaluates constant expressions at
compile time.

25
 Chapter 2: The Where Clause

To support that query, we need an index that covers the actual search term.
That means we do not need an index on LAST_NAME but on UPPER(LAST_NAME):

CREATE INDEX emp_up_name

ON employees (UPPER(last_name));

An index whose definition contains functions or expressions is a so-called

function-based index (FBI). Instead of copying the column data directly into
the index, a function-based index applies the function first and puts the
result into the index. As a result, the index stores the names in all caps
notation.

The database can use a function-based index if the exact expression of the
index definition appears in an SQL statement —like in the example above.
The execution plan confirms this:

--------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
--------------------------------------------------------------
| 0 |SELECT STATEMENT | | 100 | 41 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 100 | 41 |
|*2 | INDEX RANGE SCAN | EMP_UP_NAME | 40 | 1 |
--------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
2 - access(UPPER("LAST_NAME")='WINAND')

It is a regular INDEX RANGE SCAN as described in Chapter 1. The database

traverses the B-tree and follows the leaf node chain. There are no dedicated
operations or keywords for function-based indexes.

Warning
Sometimes ORM tools use UPPER and LOWER without the developer’s
knowledge. Hibernate, for example, injects an implicit LOWER for case-
insensitive searches.

The execution plan is not yet the same as it was in the previous section
without UPPER; the row count estimate is too high. It is particularly strange
that the optimizer expects to fetch more rows from the table than the
INDEX RANGE SCAN delivers in the first place. How can it fetch 100 rows from
the table if the preceding index scan returned only 40 rows? The answer is
that it can not. Contradicting estimates like this often indicate problems
with the statistics. In this particular case it is because the Oracle database

26
 Case-Insensitive Search Using UPPER or LOWER

does not update the table statistics when creating a new index (see also
“Oracle Statistics for Function-Based Indexes” on page 28).

After updating the statistics, the optimizer calculates more accurate

estimates:

--------------------------------------------------------------
|Id |Operation | Name | Rows | Cost |
--------------------------------------------------------------
| 0 |SELECT STATEMENT | | 1 | 3 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 3 |
|*2 | INDEX RANGE SCAN | EMP_UP_NAME | 1 | 1 |
--------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------
2 - access(UPPER("LAST_NAME")='WINAND')

Note
The so-called “extended statistics” on expressions and column groups
were introduced with Oracle release 11g.

Although the updated statistics do not improve execution performance in

this case— the index was properly used anyway—it is always a good idea to
check the optimizer’s estimates. The number of rows processed for each
operation (cardinality estimate) is a particularly important figure that is
also shown in SQL Server and PostgreSQL execution plans.

Tip
Appendix A, “Execution Plans”, describes the row count estimates in
SQL Server and PostgreSQL execution plans.

SQL Server does not support function-based indexes as described but it does
offer computed columns that can be used instead. To make use of this,
you have to first add a computed column to the table that can be indexed
afterwards:

ALTER TABLE employees ADD last_name_up AS UPPER(last_name);

CREATE INDEX emp_up_name ON employees (last_name_up);

SQL Server is able to use this index whenever the indexed expression
appears in the statement. You do not need to rewrite your query to use the
computed column.

27
 Chapter 2: The Where Clause

Oracle Statistics for Function-Based Indexes

The Oracle database maintains the information about the number of
distinct column values as part of the table statistics. These figures
are reused if a column is part of multiple indexes.
Statistics for a function-based index (FBI) are also kept on table level
as virtual columns. Although the Oracle database collects the index
statistics for new indexes automatically (since release 10g), it does not
update the table statistics. For this reason, the Oracle documentation
recommends updating the table statistics after creating a function-
based index:

After creating a function-based index, collect statistics

on both the index and its base table using the DBMS_STATS
package. Such statistics will enable Oracle Database to
correctly decide when to use the index.
—Oracle Database SQL Language Reference

My personal recommendation goes even further: after every index

change, update the statistics for the base table and all its indexes.
That might, however, also lead to unwanted side effects. Coordinate
this activity with the database administrators (DBAs) and make a
backup of the original statistics.

28
 User-Defined Functions

User-Defined Functions

Function-based indexing is a very generic approach. Besides functions like

UPPER you can also index expressions like A + B and even use user-defined
functions in the index definition.

There is one important exception. It is, for example, not possible to refer
to the current time in an index definition, neither directly nor indirectly,
as in the following example.

CREATE FUNCTION get_age(date_of_birth DATE)

RETURN NUMBER
AS
BEGIN
RETURN
TRUNC(MONTHS_BETWEEN(SYSDATE, date_of_birth)/12);
END;
/

The function GET_AGE uses the current date (SYSDATE) to calculate the age
based on the supplied date of birth. You can use this function in all parts
of an SQL query, for example in select and the where clauses:

SELECT first_name, last_name, get_age(date_of_birth)

FROM employees
WHERE get_age(date_of_birth) = 42;

The query lists all 42-year-old employees. Using a function-based index is

an obvious idea for optimizing this query, but you cannot use the function
GET_AGE in an index definition because it is not deterministic. That means
the result of the function call is not fully determined by its parameters. Only
functions that always return the same result for the same parameters —
functions that are deterministic— can be indexed.

The reason behind this limitation is simple. When inserting a new row, the
database calls the function and stores the result in the index and there it
stays, unchanged. There is no periodic process that updates the index. The
database updates the indexed age only when the date of birth is changed
by an update statement. After the next birthday, the age that is stored in
the index will be wrong.

29
 Chapter 2: The Where Clause

Besides being deterministic, PostgreSQL and the Oracle database require

functions to be declared to be deterministic when used in an index so you
have to use the keyword DETERMINISTIC (Oracle) or IMMUTABLE (PostgreSQL).

Caution
PostgreSQL and the Oracle database trust the DETERMINISTIC or
IMMUTABLE declarations— that means they trust the developer.
You can declare the GET_AGE function to be deterministic and use it in
an index definition. Regardless of the declaration, it will not work as
intended because the age stored in the index will not increase as the
years pass; the employees will not get older—at least not in the index.

Other examples for functions that cannot be “indexed” are random number
generators and functions that depend on environment variables.

Think about it
How can you still use an index to optimize a query for all 42-year-
old employees?

30
 Over-Indexing

Over-Indexing

If the concept of function-based indexing is new to you, you might be

tempted to just index everything, but this is in fact the very last thing you
should do. The reason is that every index causes ongoing maintenance.
Function-based indexes are particularly troublesome because they make it
very easy to create redundant indexes.

The case-insensitive search from above could be implemented with the

LOWER function as well:

SELECT first_name, last_name, phone_number

FROM employees
WHERE LOWER(last_name) = LOWER('winand');

A single index cannot support both methods of ignoring the case. We could,
of course, create a second index on LOWER(last_name) for this query, but
that would mean the database has to maintain two indexes for each insert,
update, and delete statement (see also Chapter 8, “Modifying Data”). To
make one index suffice, you should consistently use the same function
throughout your application.

Tip
Unify the access path so that one index can be used by several
queries.

Tip
Always aim to index the original data as that is often the most useful
information you can put into an index.

31
 Chapter 2: The Where Clause

Parameterized Queries

This section covers a topic that is skipped in most SQL textbooks:

parameterized queries and bind parameters.

Bind parameters— also called dynamic parameters or bind variables— are an

alternative way to pass data to the database. Instead of putting the values
directly into the SQL statement, you just use a placeholder like ?, :name or
@name and provide the actual values using a separate API call.

There is nothing bad about writing values directly into ad-hoc statements;
there are, however, two good reasons to use bind parameters in programs:

Security
1
Bind variables are the best way to prevent SQL injection .

Performance
Databases with an execution plan cache like SQL Server and the
Oracle database can reuse an execution plan when executing the same
statement multiple times. It saves effort in rebuilding the execution
plan but works only if the SQL statement is exactly the same. If you put
different values into the SQL statement, the database handles it like a
different statement and recreates the execution plan.

When using bind parameters you do not write the actual values but
instead insert placeholders into the SQL statement. That way the
statements do not change when executing them with different values.

1
http://en.wikipedia.org/wiki/SQL_injection

32
 Parameterized Queries

Naturally there are exceptions, for example if the affected data volume
depends on the actual values:

99 rows selected.
SELECT first_name, last_name
FROM employees
WHERE subsidiary_id = 20;

---------------------------------------------------------------
|Id | Operation | Name | Rows | Cost |
---------------------------------------------------------------
| 0 | SELECT STATEMENT | | 99 | 70 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 99 | 70 |
|*2 | INDEX RANGE SCAN | EMPLOYEE_PK | 99 | 2 |
---------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------

2 - access("SUBSIDIARY_ID"=20)

An index lookup delivers the best performance for small subsidiaries, but a
TABLE ACCESS FULL can outperform the index for large subsidiaries:

1000 rows selected.

SELECT first_name, last_name

FROM employees
WHERE subsidiary_id = 30;
----------------------------------------------------
| Id | Operation | Name | Rows | Cost |
----------------------------------------------------
| 0 | SELECT STATEMENT | | 1000 | 478 |
|* 1 | TABLE ACCESS FULL| EMPLOYEES | 1000 | 478 |
----------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("SUBSIDIARY_ID"=30)

In this case, the histogram on SUBSIDIARY_ID fulfills its purpose. The

optimizer uses it to determine the frequency of the subsidiary ID mentioned
in the SQL query. Consequently it gets two different row count estimates
for both queries.

33
 Chapter 2: The Where Clause

The subsequent cost calculation will therefore result in two different cost
values. When the optimizer finally selects an execution plan it takes the
plan with the lowest cost value. For the smaller subsidiary, it is the one
using the index.

The cost of the TABLE ACCESS BY INDEX ROWID operation is highly sensitive
to the row count estimate. Selecting ten times as many rows will elevate
the cost value by that factor. The overall cost using the index is then even
higher than a full table scan. The optimizer will therefore select the other
execution plan for the bigger subsidiary.

When using bind parameters, the optimizer has no concrete values

available to determine their frequency. It then just assumes an equal
distribution and always gets the same row count estimates and cost values.
In the end, it will always select the same execution plan.

Tip
Column histograms are most useful if the values are not uniformly
distributed.
For columns with uniform distribution, it is often sufficient to divide
the number of distinct values by the number of rows in the table.
This method also works when using bind parameters.

If we compare the optimizer to a compiler, bind variables are like program

variables, but if you write the values directly into the statement they
are more like constants. The database can use the values from the SQL
statement during optimization just like a compiler can evaluate constant
expressions during compilation. Bind parameters are, put simply, not
visible to the optimizer just as the runtime values of variables are not
known to the compiler.

From this perspective, it is a little bit paradoxical that bind parameters can
improve performance if not using bind parameters enables the optimizer
to always opt for the best execution plan. But the question is at what price?
Generating and evaluating all execution plan variants is a huge effort that
does not pay off if you get the same result in the end anyway.

34
 Parameterized Queries

Tip
Not using bind parameters is like recompiling a program every time.

Deciding to build a specialized or generic execution plan presents a

dilemma for the database. Either effort is taken to evaluate all possible
plan variants for each execution in order to always get the best execution
plan or the optimization overhead is saved and a cached execution plan
is used whenever possible — accepting the risk of using a suboptimal
execution plan. The quandary is that the database does not know if the
full optimization cycle delivers a different execution plan without actually
doing the full optimization. Database vendors try to solve this dilemma
with heuristic methods — but with very limited success.

As the developer, you can use bind parameters deliberately to help resolve
this dilemma. That is, you should always use bind parameters except for
values that shall influence the execution plan.

Unevenly distributed status codes like “todo” and “done” are a good
example. The number of “done” entries often exceeds the “todo” records by
an order of magnitude. Using an index only makes sense when searching
for “todo” entries in that case. Partitioning is another example — that is, if
you split tables and indexes across several storage areas. The actual values
can then influence which partitions have to be scanned. The performance
of LIKE queries can suffer from bind parameters as well as we will see in
the next section.

Tip
In all reality, there are only a few cases in which the actual values
affect the execution plan. You should therefore use bind parameters
if in doubt — just to prevent SQL injections.

The following code snippets show how to use bind parameters in various
programming languages.

35
Exploring the Variety of Random
Documents with Different Content
his lips as a steed dashed around the bend, bearing upon his back—
a woman!
Yes, a woman, or, rather a young girl, for she was none other than
Ruth Ramsey, who, quickly discovering an unlooked-for obstacle in
her path, attempted to draw rein. But she was too late; her steed
was a willful animal, not easily checked, and before she could come
to a halt the outlaw leader spurred alongside of her, and his left
hand grasped her bridle rein.
“Leo Randolph! You here!” she demanded.
It was all she could say, and across her face swept a deathly
pallor.
“Yes, sweet Ruth, your lover of lang syne is delighted to behold
you once more,” said the chief, with irony in his voice.
“It was proven you were an outlaw,” she said, “the leader of a wild
and desperate band; men called you Kansas King because you ruled
the border and none dare face you. Yes, all these things were
proven, and—and—I found I had loved unworthily.”
Ruth spoke half aloud, her eyes downcast, as though musing with
the past.
“Ruth, all these things were told against me; what was proven
was that I had been brought up by a fond mother who idolized her
boy, yet upon whose life a stain rested, and hence the curse fell
upon the son. That mother died, Ruth, and then came the news to
her son that a brand rested upon his life.
“Was it any wonder, then, that he threw away the advantages
bestowed upon him by his loving mother, and became a wild and
reckless outcast? Oh, Ruth, you cannot know how I have suffered,
and what a curse, a misery, my life has been. If you knew you would
pity me—and pity begets love—’tis said. You did love me once,
Ruth.”
The outlaw chief laid his hand softly upon the gloved hand of the
girl, who, quietly withdrawing the hand, replied kindly:
 “I thought I loved you once, Leo; but I did not know my heart;
and yet, had your life been different, and not a blot upon the earth,
we might have been more to each other than lovers; but you have
not forgotten that when my father exiled you from our home, and I
told you I did not love you, you basely endeavored to carry me off.”
“No, Ruth, I have not forgotten. I loved you, and that must be my
excuse. I longed to have you with me, to have you my bride, and—
forgive me, Ruth—I was mad enough to think that I might persuade
you to become my wife.”
“My consent never could have been won by force, Leo Randolph;
but, this is idle, to thus stand and talk with you. Believe me, I feel
for you in the evil career you have chosen. But I must hasten, for
the night is coming on and I was foolish to venture thus far from the
fort.”
Ruth attempted to ride on, but the outlaw chief still kept his hand
firmly upon her rein while he asked:
“How is it you are thus far from your camp, and alone?”
“I came out with my father and brother for a ride. They discovered
traces of Indians near the fort, and rode on to investigate, telling me
to return, for I was not half a mile away. I lost my road, and only
just now discovered that my way back lay through this gulch.”
Again she urged her horse forward, yet the chief held him firmly in
his strong grasp.
“Mr. Randolph, will you release my bridle rein?” said Ruth, in a
firm voice.
“Miss Ramsey, I will not—hold! Hear me, and heed—you are in my
power, and I am a desperate man. Go with me willingly; become my
wife, and I will relinquish my evil life and live for you alone; refuse,
and——”
“You plead in vain, Mr. Randolph; your evil life has already put out
every spark of regard I ever felt for you. Again I ask you to release
my rein.”
 “And again I say I will not. More—if you will not be a willing bride,
you shall be an unwilling one.”
“God have mercy upon me!” groaned poor Ruth as she reeled as if
about to fall from her saddle.
 C H A P T E R X X X V.
THE ANSWERED C R Y.

The moonlight that fell weirdly upon the Haunted Valley, and
lighted up the sad scene enacted there, also cast its silvery radiance
upon the mountain hut of the hermit chief. Pacing to and fro in the
moonlight, with quick, nervous tread, was Gray Chief, his brow dark,
and his lips set stern and hard.
A few moments before White Slayer and his chiefs had left a
council which had determined a deadly extermination of every
paleface in the Black Hills. Gray Chief had been pleased with the
decision of White Slayer, for to him all white men were enemies, and
he desired that not only should the miners perish, but also the
outlaws.
In that council it had been decided that they should seem to agree
to Kansas King’s arrangement for an alliance, and by so doing disarm
suspicion, and get him and his men in their power. After that the
Sioux warriors were to fall upon them and not a man should escape
—no, not one, swore the hermit chief.
Having thus disposed of their would-be allies, it was believed that
the Indians could arm themselves with the weapons taken from the
outlaws, and then make war upon the two camps of the invaders.
The old hermit chuckled gleefully as he thought over his plans, and
saw how eagerly the Indians had agreed to them.
Yet, had he known, within the cabin window stood one who had
heard every arrangement made, and after learning all she could,
arose from her crouching attitude and stole away. If the hermit had
known this, he would not have walked the ledge in the moonlight,
gloating over his diabolical invention to rid the Black Hills of every
paleface who had invaded their unknown fastnesses.
 After parting with Buffalo Bill, Pearl had returned home and
learned from Valleolo, the Indian woman, that the chiefs were to
assemble at once. Instantly she secreted herself in her room, and
from her ambush learned their plans, after which she hurried away
through the cavern, descended the hills to the Indian village, and
quickly mounted a splendid horse which White Slayer had captured
in battle and presented to her.
Like the wind she then rode through the valleys and over the hills,
directing her course toward the Ramsey settlement, as she dared
not take the lower cañon leading to the fort of the miners. At length
she drew near the spot where she had been told the palefaces were
encamped, and was just turning into the narrow gulch leading to the
stockade fort, when she heard a loud cry for help.
“Help, help! Oh, Heaven, save me!” again rang the cry, and in a
woman’s voice.
With the impulsiveness of her nature, Pearl was about to dash at
once to the rescue, when there came the sound of coming hoofs.
The next instant, riding up the gulch, she beheld two horses bearing
a man and a girl, the man holding the girl firmly in her saddle, and
at the same time grasping with his other hand the bridle rein of her
horse.
They were Kansas King and Ruth Ramsey. Infuriated by her
refusal of his love, the outlaw chief was bearing the girl by force to
his camp, in spite of her heart-rending cries for help.
“Hold!”
The voice was that of a woman, yet it had in it a stern and
determined ring that brought the robber chief and his captive to a
sudden halt. Before them, seated upon her horse, with her rifle
leveled at the broad breast of Kansas King, was Pearl, the Maid of
the Hills. At the command Kansas King drew rein.
“Well, girl, what do you want?” he asked.
“That you ride on and leave that girl alone,” firmly replied Pearl.
 “Ha! a stern command from such sweet lips; but what if I refuse?”
“I will kill you.”
“Harsher still, my mountain beauty; but your aim may not be true,
and——”
“One wave of my hand, Kansas King, and you might find out how
true is my aim. Do you think I am a fool, to come this far from my
home unprotected?”
Pearl spoke as though there were a hundred warriors at her back.
The outlaw chief glanced somewhat nervously around, and,
doubtless believing that the rocks and trees did conceal innumerable
redskins, he said:
“You hold the winning card, fair Pearl of the Hills. I yield to the
command of sweet lips, which yet I may punish for their unkind
words with a kiss. Ruth Ramsey, we will meet again. Fair maids, I
bid you good evening.”
Then, with a muttered curse, Kansas King drove his spurs deep
into the flanks of his horse, and dashed away up the gulch at a mad
speed. Before the rattle of his horse’s hoofs died away, there
resounded through the cañon the heavy tramp of many feet. In
dismay, Ruth cried:
“Come; oh, come, for the Indians are coming!”
Pearl listened an instant, and then said:
“No, those are not Indians, for I hear the iron ring against the
rocks of white men’s shod horses; they are your friends.”
Before more could be said a long line of horsemen filed around a
bend in the cañon. Whether friendly or hostile, it was then too late
to fly.
 CHAPTER XXXVI.
UNCLE SAM’S BOYS.

The column of horsemen that was filing at a slow trot through the
cañon were, as Pearl had said, not Indians, but palefaces, and with a
half cry of joy, Ruth saw that they were troopers, dressed in the
uniform of United States cavalry. It was a squadron of less than a
score. At their head rode a young and dashing officer of perhaps
twenty-five years of age.
At a glance, womanlike, both the girls took in his superb form,
splendid seat in the saddle, stylish uniform and broad shoulders,
with the straps of a captain thereon. Then they saw his handsome,
daring face, with its dark, earnest eyes, and firm mouth, shaded by
a dark mustache.
Certainly he was an elegant-looking young officer, and into his
frank, noble face the two girls, the daughter of the prairie, and the
child of the hills, gazed with admiration and trust.
With surprise upon his features, a pleased surprise he did not
attempt to conceal, the young officer drew rein before the two girls,
whose horses stood side by side across the cañon, and, respectfully
raising his plumed hat, said pleasantly:
“This is an unlooked-for pleasure—meeting ladies in these wild
hills.”
“And a particular pleasure, sir, to us, at least to me, for there is
certainly need for you and your troopers here,” replied Ruth.
Pearl remained silent, and the young captain again said:
“My instructions were to come into these hills and protect all white
settlers. I expected to find here a band of rude miners—certainly not
any ladies.”
 “I, sir, am the daughter of Captain Ramsey,” said Ruth. “He is the
leader of a small party of settlers who came here to establish homes
and also dig for gold; this girl I never met until ten minutes ago,
when she saved me from a terrible fate—a fate to which death was
preferable.”
Ruth Ramsey spoke with exceeding earnestness.
“Indeed!” exclaimed the young officer. “This young lady, then,
does not belong to your settlement. Can there be another band of
settlers in these hills?”
He asked the question with surprise, gazing with admiration upon
Pearl’s lovely face. Pearl flushed slightly, to find herself the object of
such ardent notice, and replied:
“I was on my way to warn the palefaces of danger, when I came
suddenly upon this lady and Kansas King, the outlaw, who was
forcing her to accompany him.”
“Warn the palefaces of danger? Are you not a paleface?” asked
the astonished soldier.
“I am a paleface, yes. But I cannot say more than that I was
going to tell the settlers that White Slayer and his band are to move
to-morrow night upon their forts, and that there is no hope for them
unless they at once leave these hills.”
“And you! Are you not in danger?” said Ruth Ramsey earnestly.
“No, I am not in danger; but you must escape from the red devils,
who will soon be on the warpath against every paleface who has
lately come into the hills.”
“You bring bad news, miss,” said the officer, “and yet I fear true
tidings, as I know the bitterness of the Indians to those who would
settle here. To-morrow night, you say, they will commence the
attack?”
“Yes, sir.”
 “And Major Wells will not be up before day after to-morrow,
hasten as he may, and I have but fourteen men with me,” was the
thoughtful statement.
“You have other troops coming, then, sir?” asked Ruth anxiously.
“Yes, over a hundred troopers; I was merely an advance guard;
here, Wentworth, hasten back with all dispatch and ask Major Wells
to ride his horses down but that he reaches here to-morrow night.”
The captain turned to a horseman who was half scout, half soldier,
and a bold-looking fellow, who promptly replied:
“I’ll fetch him, Captain Archer, if hoofs can make it!”
“Do so, Wentworth, and bring him to this point, do you hear?”
“Aye, aye, sir!” and away dashed the courier at full speed.
“Now, young ladies,” said the officer, “there is but one thing for me
to do, and that is to go secretly into camp near here and await the
attack upon the fort, and then endeavor to make the redskins
believe a large force of cavalry has come to the assistance of the
settlers. Were the Indians to know that I had but my present force
they would not fear me, so I beg that you keep my presence in the
hills a secret, and in the time of need I will be on hand. My orders,
Miss Ramsey, are to protect the lives of the settlers.”
“I will guide you to a safe place, sir, where you could conceal a
hundred men,” Pearl volunteered.
Then she considerately added:
“We should first see this lady home.”
“True. Miss Ramsey, we will ride with you to within a short
distance of your camp,” replied the young officer.
The cavalcade at once moved off, Pearl guiding, and as they rode
along the two girls and the young officer chatted pleasantly together.
At length the stockade was visible, and the party halted, while Ruth,
after bidding adieu to the captain, kissed her new-found friend and
rode on alone.
Then away dashed Pearl, side by side with the captain, and behind
came the troopers riding in Indian file. A gallop of two miles brought
them to one of those gorges so common in the Black Hills, and into
this Pearl led the way until they came to a small glen, fertile and well
watered.
“Here you can rest secure, sir. If there is any change in the plans
of the Indians, I will come and let you know,” said she.
Then she made known to the officer all that had transpired, with
which the reader is already acquainted. In surprise and
astonishment, the young man listened: and then said kindly, taking
her hand:
“The settlers have much to thank you for, miss, I assure you, and
it is noble of you to thus warn them of danger, at the risk of your
life, for I feel that you are an inmate of the village of the Sioux to
thus know their plans. This, I hope, will not be our last meeting, and
in full sincerity I say, if in any way I can befriend you, command me.
My name is Edwin Archer, and I am a captain of cavalry, now on the
prairie border.”
Pearl made no reply, waved her hand pleasantly, and away
bounded her steed on the return to the Indian village.
 CHAPTER XXXVII.
T H E FA I R Y G L E N .

When Ruth Ramsey returned to the stockade she found the whole
settlement about to turn out in search of her. Her friends were
delighted at her return, for they had believed her lost, or captured
by the Indians, as her father and brother had returned some time
before, and reported that she had started home.
Ruth made known her startling adventure with Kansas King, her
rescue by a strange white girl; but the coming of the cavalry she
kept to herself, as the officer had requested her to do. The settlers
were all in a state of fermentation at the hostile position assumed by
the Sioux, and the coming into the hills of Kansas King and his band.
Buffalo Bill had made known the enmity of the Indians and
advised that the settlers should move over to the miners’ fort until
after the battle they knew must come with the Indians.
There were some who declared against the move, unwilling to
leave off their gold digging, and thus a war of words was
progressing, when suddenly Buffalo Bill again appeared in their
midst, and at once his report settled the matter.
Two hours after, the stockade was deserted by one and all, and
the men at once set off for the miners’ camp, excepting those
designated to go with the women and children into the Haunted
Valley. A mile from the stockade the party divided, with many tears,
kind wishes, and tender farewells, and Buffalo Bill led his precious
charge by the nearest route to the valley where Red Hand awaited
them.
After an hour’s tramp, they entered a narrow gorge, the western
inlet to the valley. Ahead of them Buffalo Bill suddenly descried a
tall, upright form coming toward them.
 It was Red Hand. He bowed pleasantly to the party, pressed lightly
the hand Ruth extended to him, and said simply:
“Come.”
Leading the way through the beautiful yet strangely wild glen, Red
Hand turned, after a walk of a third of a mile, into a thick piece of
timber, through which ran an indistinct trail. A still farther walk
through the woods of two hundred yards, and before them arose the
precipitous and lofty sides of the mountain, pierced by several
narrow gorges, that appeared like lanes through the massive hills.
Into one of these chasms, for they were hardly anything more,
Red Hand walked, and soon it widened into a perfect bowl, with
towering walls upon every side. It was a fairy spot, where one would
love to dwell and dream away a lifetime, far away from the cares of
the world.
And there, sheltered against the base of the lofty hills, was a neat
little cabin home—a hermitage in the hills. It was a humble abode,
built of stout logs, and yet around it was an air of comfort, while the
interior, consisting of two rooms, certainly looked cozy and most
comfortable, for the furniture, though of rude manufacture, was
useful, and around the walls were many articles of use and
enjoyment, from rifles, knives, and pistols, cooking utensils, and a
very fair selection of books.
“This was her home,” he said simply and meaningly, speaking to
Buffalo Bill. “From here to his grave is but a short distance, and her
going there has marked a distinct trail. And, friend Cody, last night I
made strange discoveries.”
Turning to Captain Ramsey, Red Hand requested him to keep his
party in the gorge. Promising to bring the anxious mothers, wives,
sisters, and daughters good news, Buffalo Bill set out with Red Hand
for the fort, which they knew, before many hours, would be the
scene of a terrible border battle.
The scout even had his doubts as to a result in favor of the
whites.
 “Cody, if it comes to the worst, you can wait in the gorge until the
Indians believe you escaped before the fight, and then make for the
settlement with all haste.”
“I will try to take care of myself,” was the cheerful answer.
“Never mind me, old fellow; but, if we do go under, why, redskins’
scalps will be a drug in the market,” and a sad smile played upon
Red Hand’s face.
 CHAPTER XXXVIII.
T H E W A R C R Y.

Night, serenely beautiful, with its silver moon lighting up the bold
scenery upon every hand, came again to the Black Hills, and the
shadow of the mountains fell upon the miners’ fort, where all
seemed lost in deep repose. But the silence resting there was a
treacherous one, for within those stockade walls were half a hundred
brave men resting upon their arms and awaiting the coming of their
foes, who, all knew, were to hurl themselves against them that
night.
Since the day before, when he had left the valley retreat with Red
Hand, Buffalo Bill had been constantly on the move, scouting about
the hills, and his reconnoissance had discovered the plan of attack
decided upon by the Indians.
According to promise, Pearl had met him in the gorge, and told
him that from the ledge she had witnessed the coming of Kansas
King, and heard all that had passed between him and her father,
who had told the outlaw chief that the night following he would
come to his camp with five hundred warriors, and that they would
together move on the miners’ stronghold.
Kansas King had agreed to Gray Chief’s plans, and then took his
departure, apparently satisfied with the good faith of his allies. As
for the old hermit, he laughed in his sleeve at the way he had fooled
the outlaw, for it was his intention that very night to hurl his whole
force upon the robber camp, and, after a general massacre, to divide
his warriors into two parties and at once attack the two paleface
encampments.
As soon as he learned the plans of the Indians, and also heard
from Pearl about the arrival of the cavalry in the Black Hills, Buffalo
Bill at once set out on his return to the stronghold.
Whether Kansas King suspected the hermit chief of bad faith, or
determined to strike a blow himself against the settlements, is not
known; but certain it is, that, as soon as darkness set in, he moved
his men at once toward the Ramsey stockade, and after a gallant
charge up to the walls, discovered that the occupants had deserted
the place.
Chagrined at this discovery, the outlaw chief rode with all dispatch
toward the stronghold of the miners, and arrived there about the
time that Gray Chief and his red warriors reached the camping
ground of the robbers, to find that they had fled.
With rage at the move of Kansas King, the Indians at once set out
for the Ramsey settlement, gloating over their anticipated revel in
blood. Again were they doomed to disappointment, and in fear that
their enemies had escaped them they rode rapidly for the stronghold
of the miners.
Before they arrived, however, they heard the rattle of firearms.
Then it flashed across the hermit chief that Kansas King had
outwitted him and was determined to alone take the plunder from
the miners and reduce their stronghold to ashes.
The firing grew louder, and then the fort came in sight, the flashes
of the rifles lighting up the dark mountainside. As the band of
warriors pressed on, Kansas King suddenly confronted the hermit
chief, and, with coolness, said:
“Well, old man, you procrastinated too much, so I have begun the
fight!”
Both men felt that the other was playing some deep game; yet
they were anxious to receive aid, the one from the other. The
outlaws had already suffered severely, and at a glance the hermit
chief and White Slayer felt that the stronghold would not be easily
taken.
 So the outlaws and the Sioux concluded to fight together against
the miners. The Indians were thrown into position, and the battle at
once raged in all its fierceness. In vain the outlaws, under their
reckless young leader, hurled themselves against the stockade walls;
in vain the warriors resorted to every cunning artifice known to
them.
The brave little garrison poured in constantly a galling fire upon
their enemies, and many an outlaw and Indian bit the dust.
“Come, this will never do. We must charge in column with our
whole force and throw ourselves over the walls. I will lead,” cried
Kansas King, almost wild with fury at the stubborn resistance of the
gallant defenders.
“It is the only chance, I see. Here, White Slayer, form your men
for a bold rush,” replied the stern old hermit chief.
Then, with demoniacal yells, the mad column of outlaws and
redskins started upon the charge. Like hail the leaden bullets fell in
their midst, and terrible was the havoc; but on they pressed—Kansas
King, the hermit chief, and White Slayer at their head.
On, still on, until the dark column reached the stockade. Springing
upon the shoulders of the braves, the daring White Slayer was the
next instant upon the top of the wall, his wild war whoop echoing
defiance and triumph.
But suddenly behind the Indians came a ringing order in trumpet
tones:
“Troopers to the rescue—charge!”
Then was heard the hearty cheer of regular soldiers, a rattling of
sabers, a heavy tramping of many hoofs, and upon the rear of the
attacking force rushed a squadron of cavalry, half a hundred strong,
and at their head rode Captain Edwin Archer.
The sight that followed was a scene of terrible carnage, for in wild
dismay the Indians and outlaws fled, the battle lost to them at the
moment they believed victory their own. As the stampede became
general, two men mounted their horses and dashed rapidly away up
the gorge.
But upon their tracks rode two other men who had dashed out of
the stronghold in hot pursuit. The two who were flying in advance
for their lives were the hermit chief and Kansas King, both bitterly
cursing their misfortune.
The two men who had ridden from the stronghold in pursuit were
Red Hand and Buffalo Bill. On flew the two chiefs up the dark gorge,
and like bloodhounds on the trail rode Red Hand and the famous
scout.
Up the valley, over the ridges, through the cañon, up to the base
of the hill, whereon stood the hermit’s cabin, rushed the riders. Here
the two fugitives sprang from their horses and darted up the steep
ascent.
But close behind them was Red Hand and Buffalo Bill. At last the
ledge was reached, and upon it the hermit turned at bay, for he saw
that Red Hand was close behind him. Like an enraged beast, the
hermit chief cried:
“Tracked to my lair at last—at last; but, Vincent Vernon, you shall
die!”
With gleaming knife, the old hermit sprang forward, but Red
Hand, with a cry of rage, as though he recognized the man before
him, and had some bitter injury of the past to avenge, met him with
a terrible earnestness—met him to hurl him back from him with a
strength that was marvelous, and with one plunge of his blade sent
its keen point deep into the broad bosom of his foe.
One stifled cry, and the hermit chief fell back his full length upon
the hard rock, just as Kansas King, who had found the door of the
cabin barred against him, turned also at bay, to be met by a blow
from the pistol butt of Buffalo Bill, which felled him, stunned, to the
earth.
 CHAPTER XXXIX.
THE MYSTERY S O LV E D.

Upon the rocky ledge, in front of the cabin, the moonlight

streamed with almost noonday brilliancy, and lighted up a strange
scene. Lying upon the rock was the hermit chief, his long gray beard
and hair shining like silver in the moonlight, and his broad chest
heaving with every hard-drawn breath—for the hermit had received
his death wound.
Standing near was Kansas King, a bloodstain upon his forehead,
from a wound made by the butt of the scout’s pistol.
The face of the hermit was pallid with pain and some inward
emotion of bitterness. The face of the man whose deeds had won
him the name of Kansas King was still unmoved and reckless.
In front of these men stood Buffalo Bill and Red Hand. Red Hand
was slightly in advance, and he was speaking, while his deep voice
was stern and almost cruel in tone. He was saying:
“Carter Bainbridge, you have but a short time to live. Before your
soul takes its flight, I would have you speak, if the story I am now
about to relate is not true in every word.”
After a moment, the hermit replied:
“Hell has certainly aided you, Vincent Vernon, in letting your hand
take my life; tell all you wish to, for I care not now—no, not now—
ha! here comes Pearl.”
At that moment the girl rushed from the cabin, and, beholding the
strange scene and the hermit lying wounded upon the rock, cried:
“Father, my father! Are you dying?”
Quickly Red Hand stepped forward, and, restraining her, said:
 “My dear girl, this man is not your father—waste not your pity on
him.”
“Not my father! Oh, surely you are——”
“He tells the truth, Pearl; I am not your father. Listen and he will
tell you all.”
The hermit spoke with difficulty.
“Yes, I tell the truth, as you shall all hear,” said Red Hand. “Many
years ago, in a New England State, I was living with my widowed
mother; my father, a naval officer, having died when I was a mere
lad. My mother had wealth, and, being youthful and handsome, had
many admirers.
“When I was fifteen years of age I first saw this man—Carter
Bainbridge—known to you all as the Hermit of the Black Hills. This
man became, as I believed, the husband of my mother. She loved
him dearly, and so did I; but his was a black heart, for already he
had a wife living in a Southern State—the mother of a son whom
this man brought to our house after his marriage with my mother,
and passed off as his nephew.
“From the day of that son’s arrival, there began a plot for my
mother’s and my wealth, for the pretended nephew was as bad as
his professed uncle. At length I entered the navy as a midshipman,
and after an absence of three years returned to find my mother
dead.
“Even then I suspected no evil, but long afterward an investigation
proved that this man had cruelly taken my mother’s life. Again I
went to sea, and I left this man and his son at my house, as I
believed, but the son, as a common seaman, shipped on my vessel,
and as I was pacing the deck one night in a hard blow, I was thrown
overboard by a sailor who approached me unawares.
“The vessel went on, for none had seen the act, and I would have
been lost had not a schooner picked me up not twenty minutes after
I was hurled into the sea. Returning home again, I found the father
and son there. Their fright at my appearance I took for surprise and
joy, for all believed me lost, and the man who had thrown me into
the sea had left the vessel at the first port and returned to report his
success.
“Dwelling in the same town where was my home was a physician
and his daughter, an only child. That girl I loved with my whole
heart, and before I again went to sea she became my wife.
“With perfect trust, I left her at home with my supposed
stepfather and his son, while her father, the doctor, accompanied me
to sea as my guest, for his health was in a precarious condition, and
he believed a sea voyage would benefit him.
“When in Spain, a year after my marriage, word came from my
wife of the birth of a little daughter. Then my father-in-law, who was
still with me, urged that I should resign and return home. I followed
his advice, and together we were to sail for London. The night
before we sailed from Spain, when my father-in-law and I were
returning to the hotel late in the evening, an assassin sprang from a
dark corner and struck him to the heart with a knife.
“Strange to say, I was arrested as his murderer, and sent to
America for trial, for he was a man of vast wealth, and my wife was
his only heir. For nearly two years I lay in prison, and then was
acquitted, for no proof could be found against me.
“And yet, in all that time my wife did not come near me, nor did
my stepfather or his son. At last I left my cell, and returned to my
home, to find I had no home, no wife, no child. This man, Carter
Bainbridge, had sold all my property that he could lay hands on, and
my wife had gone off with the son, whose name was Ben Talbot.
“My child, I was told, was dead; and I believed it, especially when
I received a letter from my misguided wife, bidding me farewell, and
telling me that she intended to die by her own hand. Considerable
property, left me by an aunt, I still had, and, with money at my
disposal, I started to hunt down Carter Bainbridge and Ben Talbot.
 “It was long and tedious work, but I tracked this old man, step by
step, for a long time, and discovered much of his evil life—aye, I
discovered that he had deceived another woman, who believed she
became his wife, and was then cast off by him, after he had robbed
her of her wealth, and left her and her boy to starve.
“That woman was the mother of the man known as Kansas King.”
With breathless suspense, all had listened to the story of Red
Hand, and yet none were prepared for the sudden and startling
assertion he made regarding the parentage of the outlaw chief.
As for Kansas King, he stood amazed and silent—for a moment—
and then said bitterly:
“Red Hand, I feel that you speak the truth; tell me, old man, am I
your son?”
“Is your right name Leo Randolph?” faintly asked the hermit.
“So men call me; but if my parentage was dishonorable I hold no
claim to any name.”
“You are, then, my son.”
“Good God! Well, if I am hung by Captain Archer here, my fate will
be the proper thing, I suppose, and yet I prefer hanging to
acknowledging you as my father.”
The outlaw spoke with terrible bitterness. Then Red Hand
continued, in the same deep tones:
“At length, I tracked this man to his home, and I believed I killed
him, for I drove my knife deep into his side. It was the first time my
hand was stained with blood, though from my birth I have borne this
mark which has given me my name upon the frontier.”
Red Hand held up his hand so that the moonlight revealed its
crimson hue. Again he went on:
“But I was only half avenged, for Ben Talbot still lived. What
destiny ever led my footsteps into these hills, God only knows; but
here, five years ago, I met Ben Talbot—and killed him.”
“Tell me, Vincent Vernon, tell me—is the grave in the Haunted
Valley that of my son?” said the old hermit eagerly.
“It is; I killed him, and, for the sake of the happy days we had
passed together in boyhood, I buried him, and carved his name
upon a tree at the head of his grave.”
“I knew of the grave, but never saw it—never knew that my son
lay buried there, for I thought he had gone East,” muttered the old
hermit.
“Tell me, Carter Bainbridge,” continued Red Hand, “did Ben Talbot
come here with you?”
“Yes; I fled here in fear of my life, for I have been a great sinner,
and Ben and Grace came with me; but we had a quarrel, and they
left, as I believed, to go East and——”
“And they settled in the Haunted Valley, and there they lived, until
I killed Ben Talbot. Then poor Grace still remained, alone, to watch
his grave, until last night she fell by her own hand, as this scout
knows. Aye, fell by her own hand, and we two buried her there in
the valley.
“Then I sought the cabin where they lived, and the papers I found
there told me all; yes, that Ben Talbot had slain the father of my
wife, and then placed the crime at my door to have me hung, and
that, believing the story told her, Grace had fled, a guilty thing, from
my love. But I have forgiven her all. Aye, more did I learn, and that
is that this girl here, who has heard every word of my story, is my
own daughter. Pearl, will you come to your father’s heart?”
Words cannot portray the tenderness with which Red Hand spoke,
and, comprehending the whole plot of crime against him, and feeling
that he was indeed her father, the girl sprang forward and nestled
close in the arms of the man whose life had known so much of
misery.
 Not a word, not a motion, marred the silent joy of that moment
for those two, father and daughter, so cruelly divided through life.
Finally Red Hand turned once more to the old hermit, and said:
“Carter Bainbridge, I can now, in my joy, even forgive you.”
No word of reply came, the eyes gazed straight at the moon with
a fixed stare, and the voice of Buffalo Bill said quietly:
“He’s gone to another trapping ground, comrade.”
It was indeed true; and Red Hand turned and led poor Pearl into
the cabin, to prepare for the return to the stronghold of the miners.
 CHAPTER XL.
TWO WEDDINGS.

In the shadow of the hill that sheltered his cabin, Carter

Bainbridge, the Hermit of the Black Hills, found his last earthly
hermitage—the grave. Standing by, watching the burial of the
hermit, was Pearl, leaning upon the arm of her father, and so intent
were Tom Sun, Lone Dick, and Buffalo Bill in digging the grave, and
Edwin Archer in gazing upon the beautiful face and form of Pearl
Vernon, that no one noticed the prisoner, Kansas King, quietly steal
away, until all was over.
Search and pursuit were then useless, and, mounting their steeds,
awaiting them in the gorge, the party started for the miners’
stronghold, where they arrived just at sunrise, and were greeted
with wild hurrahs from all.
Buffalo Bill then accompanied Red Hand and his daughter to the
Haunted Valley, and while he went on to tell the glad tidings of
victory to the anxious party in the secret retreat, the husband and
the daughter halted at the grave of poor Grace, and, guilty though
she was, they sorrowed for her most deeply.
During the day the whole party of miners and settlers were
gathered together at the stronghold. Most warmly was Pearl
welcomed by Ruth Ramsey and all, when they heard the strange
story of her eventful life, and hearty congratulations were bestowed
upon Red Hand in honor of his new-found happiness.
Toward evening Major Wells arrived with his squadron. Though the
settlers and gold seekers had nothing to fear while the soldiers were
there to protect them, the danger from hostile Indians was still so
great that the scout and the officers urged the settlers not to remain
in the hills.
 The greater part of the two bands were most willing to acquiesce,
and the following day the entire company, accompanied by the
cavalry, left the inhospitable but beautiful land, and took up their
march for the boundary of civilization.
During the march, Edwin Archer and Pearl Vernon were often
together, and so also were Red Hand, now known as Vincent Vernon,
and Ruth Ramsey.
The result of this intimacy was that, shortly after their arrival at
North Platte, there was an engagement entered into between each
couple, to be consummated one year from that date.
Then were the two bands scattered to the four winds of heaven,
some remaining upon the frontier, among whom was Lone Dick, who
returned to trapping, and Tom Sun, who entered the army under
Major Wells.
As for Captain Ramsey, he went East with his family, and
purchased a home in Maryland, while Captain Edwin Archer started
for New York to take possession of a fortune left him by a maiden
aunt. Tired of a wild life on the border, and rejoiced to have found a
beautiful daughter, Red Hand also left for New York, where he
placed Pearl at school for one year.
She became the bride of Edwin Archer the same day that beheld
Ruth Ramsey married to Vincent Vernon, and well I know that every
reader of these lines will wish them happiness as they journey
through life together.
The great scout, after solving the mystery of Red Hand, departed
for military duty at Fort Hays. Notwithstanding the fact that Buffalo
Bill was attached to the fort, his duties made it necessary for him to
roam over the vast expanse of prairies and to aid travelers whenever
he found them in distress.
 CHAPTER XLI.
THE BRANDED BROTHERHOOD.

Picture to yourself a bivouac of outlaws, a wild-looking but

picturesque camp scene far out in the “land of the setting sun.” A
“prairie sea” is upon every hand, here and there dotted with a
timber island, a cool and refreshing covert from the heat of the
plain.
Miles and miles of land, unfurrowed by the plowshare, untilled by
human hands, stretch away in boundless expanse as far as mortal
vision can sweep. Winding its silvery length along, like a huge
serpent crawling across the rolling prairies, is a clear and lazy river,
its waters cold and inviting, coming from the icy fountains in the
hills, and its banks flower-spangled and many-hued, while here and
there a motte, or growth of timber, casts fantastic shadows across
the stream.
In the deep recesses and shady retreats of one of the larger of
these mottes is this bivouac of bandits. The day is far spent, the sun
is near its setting, and its last rays cause the tall trees to stretch
their shadows far out over the waving grass, which, under the
influence of a light wind, resembles the restless waves of the ocean.
Into this encampment of the outlaws I would have the reader
accompany me, in imagination, for there he will behold a scene
never to be met with amid the boundaries of civilization. These men
formed a wild and striking assemblage of horsemen, dismounted and
gathered in groups, either preparing their evening meal around the
blazing camp fires, or else indifferently lounging around, awaiting
the completion of the culinary arrangements.
A strange set of human beings they were, of many tongues and
costumes, but with the buckskin leggings, flannel shirt, and slouch
hat predominating. They were men outlawed from the homes of
civilization; men upon whose brows rested the curse of Cain, and
who were branded, far and wide, as a brotherhood of bandits.
Many of them were dashing, daring, and gallant fighters, but
turned the gifts God had given them to prey upon the lives and
fortunes of their fellow men. Amid that motley group might be seen
the deserter from the army of the United States, the lively
Frenchman, the florid Englishman, the beer-loving German, the
swarthy Spaniard, the half-breed, the full-blooded Indian, and the
American.
Truly they were a bold and reckless set, held in check by one man,
who, half reclining before a bright fire, watched the movements of
his negro cook, and ever and anon addressed some words to the
three or four of his comrades around him.
Once that elegant but powerful form had been clad in the uniform
of an honored cavalry officer of his country’s service, and the dark
and lustrous eyes had, amid the brilliant saloons of the distant cities,

Looked love to eyes

That spoke again.

But that was long ago, and time had brought many changes, and
branded his once proud name with infamy. Fully six feet in height,
and of supple, graceful form, the chief of the Branded Brotherhood
wore buckskin, with trousers elaborately worked with beads, and
fringed down the outer seams.
Instead of moccasins, his feet were incased in high-top cavalry
boots, armed with huge spurs; and a blue silk shirt and Mexican
jacket, profusely adorned with silver buttons, completed his
costume, excepting a gray slouch hat, with exceedingly broad brim,
which was turned up on one side.
The hands and face of the outlaw were burned as brown as the
sun and exposure could make them; a heavy brown beard, of a like
shade, with his long, curling hair, completely hid the lower features
of his face; but his nose was straight and firm, his forehead broad
and intellectual, his eyes strangely fiery and savage, while within
their inmost depths was an expression hard to fathom, for at times it
looked like fear, again was expressive of sadness, and at others of
hatred and mischief.
His men knew him only as “the chief.” Along the frontier he was
called “Captain Ricardo, the Bandit,” but what his real name was
none knew.
Nor did any one know whence he came, only it was surmised that
he had once been a distinguished cavalry officer, who, having been
dismissed from the service for a crime committed, had taken to the
plains as a highway robber, until, in a few years he had organized
the band of which he was chief, and which had spread terror far and
wide along the border.
The chief’s horse, a splendid-looking iron-gray, fed near by, and,
serving as a resting place for his arm was a Mexican saddle, with a
belt, containing two revolvers and a bowie knife, which Captain
Ricardo kept near at hand.
The persons immediately surrounding the chief consisted of the
negro cook, a cunning-faced, wiry fellow, black as a coal, who never,
sleeping or waking, went without his revolver and knife, which he
kept in a large leather belt around his waist.
It was said the negro, whom his master called Buttermilk—as a
contrast to his color—knew more of the chief’s life than did any one
else; but, if so, he was never known to betray that knowledge.
Then there was an Indian scout, a powerful and evil-looking Sioux,
who had betrayed his own people and then sought refuge in the
outlaw band, and, thoroughly knowing the whole country, Captain
Ricardo found him an able ally.
There were also two others, both white men; one a square-
framed, brutal-faced man of forty-five, whom the chief had made his
second in command, and the other a renegade trapper and hunter,
who, having robbed his comrades, a few years before, had sought
the band for protection.
Turning to his officer, who was impatiently watching the rather
lazy preparations of the negro, Buttermilk, Captain Ricardo
remarked, in a voice strangely soft and pleasant for one who led his
wild life:
“I see no reason why the train should not fall easily into our
hands, for they must cross the river at a point near here.”
“Yes, chief; but if we wait for them to come up here the troop will
have rejoined them, and now, you know, the Injun here says
Captain la Clyde and his troopers are off on a scout and the train has
only its own men to guard it.”
This was the answer of the lieutenant, who answered to the name
of Red Roark, both on account of his red hair and beard and his
bloody deeds, for at heart he was a perfect brute.
“The chief’s right,” said the renegade trapper. “You hear me talk,
Red Roark. If we waits for them fellers here they’ll come
onsuspectinglike, right onto our trap; but ef we goes out on the
prairie to fight ’em, then we’ll get some hard knocks and no pay. You
see, I’s been in thar train, as I told the chief, and I knows what I’s
talkin’ about.”
The trapper was squatted down on the ground near the chief, who
replied:
“You really went into their train, Long Dave?”
“You bet! I just tole ’em I was a hunter as was going to the forts,
and I tell you they has just got a ticklish-lookin’ set of fellers to
tackle. They axed me ’bout you, chief, and ef I thought they’d run
across you, and, of course, I tole ’em no, and they said ef they did
you’d have to git up early to catch them napping.”
“How many fighting men are there, Long Dave?”
 “Some forty or more, big boys included; and then there’s the
twenty troopers under Captain la Clyde, who you might count on, for
he just goes scouting around, you see, and has taken a shine to one
of the gals in the train, and he’s going to be on hand when it comes
to a row, you bet.”
“Which way did the cavalry go when they left the train last night?”
“That’s jist what I was going to find out when I seed that devil of
a fellow they call Buffalo Bill a-coming across the prairie, and I jest
lit out for these diggin’s, you bet, chief, kase I knows that fellow, and
don’t want him near me.”
“You refer to Buffalo Bill, the army scout?”
“Yes, the fellow is getting mighty bold of late.”
“He is, indeed, and I would be willing to pay a round sum to take
him, for he has thwarted my plans more than once. Well, we’ll lie in
wait for the train here, and to-night, Long Dave, you and Black Wolf
must start out and bring me the exact whereabouts of both the train
and the troopers, for this rich harvest must not be lost for want of
reaping. Now let us have supper, Buttermilk, you lazy dog.”
“You be lazy, too, if you have to cook tough ole buffalo bull a
t’ousand year ole,” grumbled the negro, who always had a way of
answering back when addressed, and which his master appeared not
to notice, but would severely punish in any one else.
Just as night set in the chief and his three comrades fell to and
were soon enjoying the really delicious meal which Buttermilk had
prepared. An hour or more passed away and the bandit camp was as
silent as a “city of the dead,” for the men had rolled themselves in
their blankets and sought their rest, excepting the half a dozen
sentinels who had been set to keep watch and ward.
Now and then the howl of a hungry wolf out on the prairie broke
the stillness of the night, or the startled snort of a horse was heard.
Then again all was quiet, until suddenly there rang forth the sharp
crack of a rifle, followed by a death shriek. Instantly every man in