Database Systems

File Organization and Indexes

This Lecture…
 File Organization

 Indexes
Files of Records

 Page or block is OK when doing I/O, but higher

levels of DBMS operate on records, and files of
records.
 FILE: A collection of pages, each containing a
collection of records. Must support:
 insert/delete/modify record

 read a particular record (specified using record

id)
 scan all records (possibly with some conditions

on the records to be retrieved)

File Organization

 Three types
 Heap File Organization
 Sequential File Organization
 Hashing File Organization
 Alternative File Organizations

Many alternatives exist, each ideal for some situation , and not so
good in others:
 Heap files: Suitable when typical access is a file scan
retrieving all records.
 Search (Equality/Range) needs to scan the file

 Insert: At the end of file

 Delete: Search for record and delete record

 Sorted Files: Best if records must be retrieved in some order,

or only a `range’ of records is needed.
 Search (Equality/Range): Efficient

 Insert: Finding the position, inserting & move records

 Delete Search for record, delete & move records

Alternative File Organizations…
(contd.)
 Hashed Files: Good for equality selections.
 File is a collection of buckets. Bucket = primary page plus

zero or more overflow pages.

 Hashing function h: h(r) = bucket in which record r

belongs. h looks at only some of the fields of r, called the

search fields.
Alternative File Organizations…
(contd.)
 Hashed Files:
 Search (Equality): good for equality (if based
on search key). Otherwise scan table
 Search (Range): needs to scan the file
 Insert: search for primary bucket (hash) and
insert
 Delete: search for primary bucket (hash) if
available, else scan file & delete record
Example
ExampleLibrary
LibraryCatalog
Catalog/ /Book
BookIndex
Index

Indexes

 An index on a file speeds up selections

on the search key fields for the index.

 Any subset of the fields of a relation can be

the search key for an index on the relation.

 Search key is not the same as key (minimal

set of fields that uniquely identify a record in
a relation).
Characteristics
 Indexes provide fast access

 Indexes takes space

 Need to be careful in creating only useful indexes

 May slow-down certain inserts/updates/

deletes (maintain indexes)
Explain
Explainon
onboard
board

Alternatives for Data Entry k* in

Index
 An index contains a collection of data entries,
and supports efficient retrieval of all data
entries k* with a given key value k.

 Three alternatives:
1. Data record with key value k (Alt. 1)
2. <k, rid of data record with search key value
k> (Alt. 2)
3. <k, list of rids of data records with search
key k> (Alt. 3)
Terminology
 File of records containing index entries
= index file

 There are several organization

techniques for building index files =
access methods
 Properties of Indexes…

 Clustered vs. Unclustered Index

Index entries
CLUSTERED direct search for UNCLUSTERED
data entries

Data entries Data entries

(Index File)
(Data file)

Data Records Data Records

 Can have at most one clustered index per table

 Cost of retrieving data records through index varies greatly based on
whether index is clustered or not!
 Properties… (contd.)

 Dense vs. Sparse: If there is Ashby, 25, 3000

at least one data entry per Basu, 33, 4003

22

search key value (in some

25
Bristow, 30, 2007
30

data record), then dense. Ashby

Cass, 50, 5004

33
Cass
 Alt. 1 always leads to Smith Daniels, 22, 6003
40
dense index. Jones, 40, 6003
44

 Every sparse index is Smith, 44, 3000

44

clustered!
50
Tracy, 44, 5004

 Sparse indexes are Sparse Index Dense Index

on on
smaller; however, some Name Data File Age
useful optimizations are
based on dense indexes.
Properties… (contd.)
 Primary vs. secondary: If search key
contains primary key, then called
primary index.
 Unique index: Search key contains a
candidate key.
 Properties… (contd.)
Examples of composite key
indexes using lexicographic order.
 Composite Search Keys: Search on a
combination of fields. 11,80 11
 Equality query: Every field value is 12,10 12
equal to a constant value. E.g. wrt 12,20 name age sal 12
<sal,age> index: 13,75 bob 12 10 13
 age=20 and sal =75
<age, sal> cal 11 80 <age>
 Range query: Some field value is joe 12 20
not a constant. E.g.: 10,12 sue 13 75 10
 age =20; or age=20 and sal > 20
20,12 Data records
10 75,13 sorted by name 75
 Data entries in index sorted by search 80,11 80
key to support range queries. <sal, age> <sal>
Data entries in index Data entries
sorted by <sal,age> sorted by <sal>
Indexes in SQL…
 Index is not a part of SQL-92

 However, all major DBMSs provide facilities

for index creation
 CREATE INDEX…
 DROP INDEX…

 SQL Server support indexes (clustered and

non-clustered indexes)
 Range Searches
 ``Find all students with gpa > 3.0’’
 If data is in sorted file, do binary search to find first such

student, then scan to find others.

 Cost of binary search can be quite high.

 Simple idea: Create an `index’ file.

k1 k2 kN Index File

Page 1 Page 2 Page 3 Page N Data File

 Can do binary search on (smaller) index file!

 B+ Tree: The Most Widely Used
Index
 Insert/delete at log F N cost; keep tree height-balanced.
(F = fanout, N = # leaf pages)
 Minimum 50% occupancy (except for root). Each node
(except root) contains d <= m <= 2d entries. The
parameter d is called the order of the tree.
 Supports equality and range-searches efficiently.

Index Entries
(Direct search)

Data Entries
("Sequence set")
B+ Trees in Practice
 Typical order: 100. Typical fill-factor: 67%.
 average fanout = 133

 Typical capacities:
 Height 4: 1334 = 312,900,700 records

 Height 3: 1333 = 2,352,637 records

 Can often hold top levels in buffer pool:

 Level 1 = 1 page = 8 Kbytes
 Level 2 = 133 pages = 1 Mbyte
 Level 3 = 17,689 pages = 133 MBytes
B+ Tree…
 Search begins at root, and key comparisons
direct it to a leaf
 Each Node has search keys (Ki) and pointers
(Pi).
 Pi points to a sub-tree in which all key values
K are such that Ki ≤ K < Ki+1
Search
func tree_search (nodepointer, search key value K) returns
nodepointer
/ / Searches tree for entry
if *nodepointer is a leaf, return nodepointer;
else,
if K < K1 then return tree_search(Po, K);
else,
if K ≥ Km then return tree_search(Pm, K) // m = # entries
else,
find i such that Ki ≤K < Ki+1;
return tree_search(Pi, K)
end if
end if
 Example B+ Tree…

 Search for 5*, 15*, all data entries >=

24* ... Root
13 17 24 30

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39*

 Based on the search for 15*, we know it is not in the tree!

Inserting a Data Entry into a
B+ Tree
Find correct leaf L.
Put data entry onto L.
If L has enough space, done!
Else, must split L (into L and a new node L2)
Redistribute entries evenly, copy up middle key.
Insert index entry pointing to L2 into parent of L.
This can happen recursively
To split index node, redistribute entries evenly, but push up
middle key. (Contrast with leaf splits.)
Splits “grow” tree; root split increases height.
Tree growth: gets wider or one level taller at top.
 Inserting 8* into Example B+
Tree
Entry to be inserted in parent node.
 Observe how 5 (Note that 5 is
s copied up and
minimum continues to appear in the leaf.)

occupancy is
guaranteed in 2* 3* 5* 7* 8*

both leaf and

index pg splits.
 Note difference
between copy-up Entry to be inserted in parent node.
(Note that 17 is pushed up and only
17
and push-up; be appears once in the index. Contrast
this with a leaf split.)
sure you
understand the 5 13 24 30
reasons for this.
 Example B+ Tree After
Inserting 8*
Root
17

5 13 24 30

2* 3* 5* 7* 8* 14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39*

 Notice that root was split, leading to increase in height.

 In this example, we can avoid split by re-distributing
entries; however, this is usually not done in practice.
 Deleting a Data Entry from a
B+ Tree
 Start at root, find leaf L where entry belongs.
 Remove the entry.
 If L is at least half-full, done!

 If L has only d-1 entries,

 Try to re-distribute, borrowing from sibling (adjacent

node with same parent as L).

 If re-distribution fails, merge L and sibling.

 If merge occurred, must delete entry (pointing to L or sibling)

from parent of L.

Merge could propagate to root, decreasing height .
 Example Tree After (Inserting
8*, Then) Deleting 19* and
20* ...
Root

17

5 13 27 30

2* 3* 5* 7* 8* 14* 16* 22* 24* 27* 29* 33* 34* 38* 39*

 Deleting 19* is easy.

 Deleting 20* is done with re-distribution.
Notice how middle key is copied up.
 ... And Then Deleting
24*
 Must merge. 30

 Observe `toss’ of
index entry (on right), 22* 27* 29* 33* 34* 38* 39*

and `pull down’ of

index entry (below).
Root
5 13 17 30

2* 3* 5* 7* 8* 14* 16* 22* 27* 29* 33* 34* 38* 39*

Duplicates in B+ Trees…
 We have ignored duplicates so far…

 Alternatives…
 Overflow leaf pages

 Duplicate values in the leaf pages

 Make unique key values (by adding rowid’s)

 Preferred approach by many DBMSs
 Hashing

 Hash-based indexes are best for equality

selections.

 Cannot support range searches.

 Static and dynamic hashing techniques

exists
 Static Hashing
 # primary pages fixed, allocated sequentially,
never de-allocated; overflow pages if needed.
 h(k) mod N = bucket to which data entry with
key k belongs. (N = # of buckets)
0
h(key) mod N
2
key
h

N-1
Primary bucket pages Overflow pages
 Static Hashing… (contd.)

 Buckets contain data entries.

 Hash fn works on search key field of record
r. Must distribute values over range 0...N-1.
 h(key) = (a * key + b) usually works well.
 a and b are constants; lots known about how
to tune h.
Static Hashing… (contd.)
Problems…
 Insertion can create long overflow
chains can develop and degrade
performance.
 Deletion may waste space
 Extendible and Linear Hashing:
Dynamic techniques to fix this problem.
 Extendible Hashing

 Situation: Bucket (primary page) becomes full. Why not

re-organize file by doubling # of buckets?
 Reading and writing all pages is expensive!

 Idea: Use directory of pointers to buckets , double # of

buckets by doubling the directory, splitting just the

bucket that overflowed!
 Directory much smaller than file, so doubling it is much

cheaper. Only one page of data entries is split. No

overflow page!
 Trick lies in how hash function is adjusted!
 LOCAL DEPTH 2
Bucket A
4* 12* 32*16*

Example
GLOBAL DEPTH

2 2
Bucket B
00 1* 5* 21* 13*
 Directory is array of size 4.
01
 To find bucket for r, take last 10 2
`global depth’ # bits of h(r); 10*
Bucket C
11
we denote r by h(r).
 If h(r) = 5 = binary 101, it
DIRECTORY 2
is in bucket pointed to by 15* 7* 19*
Bucket D
01.
DATA PAGES

 Insert: If bucket is full, split it (allocate new page, re-distribute).

 If necessary, double the directory. (As we will see, splitting a
bucket does not always require doubling; we can tell by
comparing global depth with local depth for the split bucket.)
 Insert h(r)=20 (Causes
Doubling)
LOCAL DEPTH 2 3
Bucket A LOCAL DEPTH
GLOBAL DEPTH 32*16* 32*16*Bucket A
GLOBAL DEPTH

2 2
3 2
00 1* 5* 21*13*Bucket B 000 1* 5* 21*13*Bucket B
01 001
10 2 2
010
10* Bucket C
11 10*
011 Bucket C
100
2
DIRECTORY 101 2
Bucket D
15*7* 19*
110 15*7* 19* Bucket D
111
2
3
4* 12*20* Bucket A2
DIRECTORY 4* 12*20* Bucket A2
(`split image'
of Bucket A) (`split image'
of Bucket A)
 Points to Note

 20 = binary 10100. Last 2 bits (00) tell us r belongs in A or

A2. Last 3 bits needed to tell which.
 Global depth of directory : Max # of bits needed to tell

which bucket an entry belongs to.

 Local depth of a bucket: # of bits used to determine if an

entry belongs to this bucket.

 Not all splits double the directory size

 Example: Insert 9*
Points to Note (contd.)
 When does bucket split cause directory
doubling?
 Before insert, local depth of bucket = global
depth. Insert causes local depth to become >
global depth; directory is doubled by copying
it over and `fixing’ pointer to split image page.
(Use of least significant bits enables efficient
doubling via copying of directory!)
 Comments on Extendible
Hashing
 If directory fits in memory, equality search answered with
one disk access; else two.
 100MB file, 100 bytes/rec, 4K pages contains 1,000,000

records (as data entries) and 25,000 directory elements;

chances are high that directory will fit in memory.
 Directory grows in spurts, and, if the distribution of hash

values is skewed, directory can grow large.

 Multiple entries with same hash value cause problems!
Comments on Extendible Hashing
(contd.)
 Delete: If removal of data entry
makes bucket empty, can be merged
with `split image’. If each directory
element points to same bucket as its
split image, can halve directory.
Summary
 File Organizations
 Indexes
 B+ Tree
 Hashing (Extendible)