File Structures PDF
Uploaded by
Roberta CostaFile Structures PDF
Uploaded by
Roberta CostaMichael J.
Folk
Bill
Zoellick
File
Structures
Second Edition
Digitized by the Internet Archive
in
2011
http://www.archive.org/details/filestructuresOOfolk
File
SECOND
EDITION
Structures
MICHAEL
J.
FOLK
University of Illinois
BILL ZOELLICK
Avalanche Development Company
TV Addison-Wesley
Publishing
Company,
Inc.
Menlo Park, California
New York
Don Mills, Ontario Wokingham, England Amsterdam
Bonn Sydney Singapore Tokyo Madrid
Reading, Massachusetts
San Juan
Milan
Paris
Sponsoring Editor Peter Shcpard
Production Administrator Juliet Silveri
Copyeditor Patricia Daly
Text Designer Melinda Grosser for silk
Cover Designer Peter Blaiwas
Technical Art Consultant Dick Morton
Scot Graphics
Illustrator
Roy Logan
Manufacturing Supervisor
Photographs on pages 126 and 187 courtesy of
S.
Sukumar. Figure 10.7 on page 470
courtesy of International Business Machines Corporation.
Library of Congress Cataloging-in-Publication Data
Folk, Michael
J.
File structures
Michael
J.
Folk, Bill Zoellick.
2nd
ed.
cm.
p.
Includes bibliographical references and index.
ISBN 0-201-55713-4
1.
II.
File
organization (Computer science)"
I.
Zoellick, Bill.
Title.
QA76.9.F5F65
1992
005.74 dc20
91-16314
CIP
book have been included for their
They have been tested with care but are not guaranteed for any
purpose. The publisher does not offer any warranties or representations, nor does
The programs and
applications presented in this
instructional value.
particular
it
accept any liabilities with respect to the
Many
of the designations used
claimed
as
was aware
trademarks.
o\\\
Where
programs or
by manufacturers and
applications.
sellers to distinguish their
products are
those designations appear in this book, and Addison- Wesley
trademark claim, the designations have been printed
in initial caps or
all
caps.
Copyright
<
1992 by Addison-Wesley Publishing
All rights reserved.
No
system, or transmitted,
Inc.
may be reproduced, stored in
any form or by any means, electronic, mechanical,
part of this publication
Company.
a retrieval
photocopying, recording, or otherwise, without the prior written permission of the
publisher. Printed in the United States of America.
12
3 4 5 6 7 8 9 10-DO-9594939291
To
Pauline and Rachel
and
To Karen, Joshua, and
Peter
Preface
We wrote the first edition to promote file structure literacy.
familiarity with the tools used to organize
story of
how
files. It
also
Knowing
the different tools have evolved.
Literacy implies
means knowing the
the story
is
the
basis for using the tools appropriately.
The first edition told the story of file structures up to about 1980. This
second edition continues the story, examining developments such as
extendible hashing and optical disc storage that have moved from being a
research topic at the start of the
last
decade to
mature technology by
its
end.
While the history of file structures provides the key organizing principle
for
much of this
text,
we
also find ourselves compelled, particularly in this
second edition, to attend to developments in computing hardware and
system software. In the last twenty years computers have evolved from
being expensive monoliths, maintained by a priesthood of specialists, to
being appliances as ubiquitous as toasters. No longer do we need to
confront a corps of analysts to get information in and out of a computer. We
do it ourselves. Today, more often than yesterday, programmers design
and build their own file structures.
This text shows you how to design and build efficient file structures.
you need is a good programming language, a good operating system,
and the conceptual tools that enable you to think through alternative file
structure designs that apply to the task at hand. The first six chapters of this
book give you the basic tools to design simple file structures from the
ground up. We provide examples of program code and, if you are a UNIX
user, we show you, whenever possible, how to use this operating system to
help with much of the work. Building on the first six chapters of foundation
work, the last five chapters introduce you to the most important high-level
+
trees, and
file structure designs, including sequential access, B-trees and B
All
hashing and extendible hashing.
VI
PREFACE
The
last
ten years of development in software design are reason
we have
for this second edition, but
enough
also used this edition to discuss the
decreased cost and increased availability of computer storage hardware. For
computer configurations over
available RAM on
computers of all sizes. In 1986, when we completed the first edition of this
book, it was rare that a personal computer had more than 640 Kbytes of
RAM. Now, even for many mundane applications, four Mbytes is
common, and sometimes even mandatory. A decade ago, a sophisticated
mainframe system that was used extensively for sorting large files typically
had two to four Mbytes of primary memory; now 32 to 64 Mbytes is
common on workstations, and there are some computers with several
one of the most dramatic changes
instance,
the
decade
past
gigabytes of
RAM
that,
on
assuming
files
RAM
when
sorting
available
is
For example, most
sorting of large
is
the
in
amount of
RAM.
When more
differently.
increase in
the
is
disk.
available, sorting
second edition
is
we
can approach
it is
that
on disk
is
reflects this
structures problems
with the
One reason for this
much more viable than
always done on tape.
scarce, sorting
Now
file
earlier file structure texts deal
RAM
is
on tape
much
is
cheaper and more readily
not only viable, it is usually preferable. This
change and others that arise from changes in
computer hardware.
Using the Book as a College Text
The
first
edition has been used extensively as a text for
many
many
different kinds
of universities. Because the book is quite
readable, students typically are expected to read the entire book over the
course of a semester. The text covers the basics; class lectures can expand
and supplement the material presented in the text. The lecturer is free to
explore more complex topics and applications, relying on the text to supply
of students in
different kinds
the fundamentals.
A word
issues
of caution:
presented in the
material.
The
It is
much time on
Move quickly
easy to spend too
first
relatively large
six
chapters.
number of pages devoted
the low-level
through
this
to these matters
is
of the percentage of the course that should be spent on them.
The intent, instead, is to provide thorough coverage in the text so that the
instructor can simply assign these chapters as background reading, saving
precious lecture time for more important topics.
It is important to get students involved in writing file processing
programs early in the semester. Consider starting with a file reading and
not
a reflection
PREFACE
VII
due after the first week of class. The inclusion in
the text of sample programs in both C and Pascal makes it easier to work
in this hands-on style. We recommend that, by the time the students
encounter the B-tree chapter, they should have already written programs
that access a data set through a simple index structure. Since the students
then already have first-hand experience with the fundamental organizational
issues, it is possible for lectures to focus on the conceptual issues involved
writing assignment that
is
in B-tree design.
we
approximation of
the sequence of topics used in the book, especially through the first six
chapters. We have already stressed that we wrote the book so that it can be
read from cover to cover. It is not a reference work. Instead, we develop
Finally,
ideas as
makes
A Book
we
it
suggest that instructors adhere to
a close
proceed from chapter to chapter. Skipping around
difficult for students to
in the
book
follow this development.
Computing Professionals
for
Both authors used
We
to teach, but
we now
design and write programs for a
book with our colleagues in mind. The
style is conversational; the intent is to provide a book that you can read over
a number of evenings, coming away with a good sense of how to approach
file structure design problems. If you are already familiar with basic file
living.
wrote and revised
this
structure design concepts, skim through the
first six
chapters and begin
reading about cosequential access methods in Chapter 7. Subsequent
+
chapters introduce you to B-trees, B trees, hashing, and extendible
hashing. These are key design tools for any practicing
building
file
structures.
We have
tried to present
them
programmer who
in a
way
that
is
is
both
thorough and readable.
If
you
are not already a serious
UNIX
seven chapters will give you
environment in which to work with
user, the
a feel for
first
files.
UNIX
why UNIX
Similarly, the
material in the
is
several of the chapters provide an introduction to the use of
Also, if you need to build and access
text,
you may be
you can adapt
Finally,
to
able to use these
file
powerful
programs
with
in
files.
structures similar to the ones in the
programs
as a
source code toolkit that
your needs.
we know that an increasing number of computing
professionals
CD-ROM.
Appendix
with the need
in
introduced
principles
design
A not only provides an example of how the
this text are applied to this important medium, but it also gives you a good
to understand
are confronted
introduction to the
medium
itself.
and use
VIII
PREFACE
Acknowledgements
There are a number of people we would like to thank for help in preparing
this second edition. Peter Shepard, our editor at Addison- Wesley, initiated
the idea of a new edition, kept after us to get it done, and saw the
production through to completion. We thank our reviewers; James
Canning, Jan Carroll, Suzanne Dietrich, Terry Johnson, Theodore Norman, Gregory Riccardi, and Cliff Shaffer. We also thank Deebak Khanna
for comments and suggestions for improving the code.
Since the publication of the first edition, we have received a great deal
of feedback from readers. Their suggestions and contributions have had a
major effect on this second edition, and in fact are largely responsible for
our completely rewriting several of the chapters.
Colleagues with whom we work have also contributed to the second
edition, many without knowing they were doing so. We are grateful to
them for information, explanations, and ideas that have improved our own
understanding of many of the topics covered in the book. These colleagues
include Chin Chau Low, Tim Krauskopf, Joseph Hardin, Quincey Koziol,
Carlos Donohue, S. Sukumar, Mike Page, and Lee Fife.
Thanks
are
still
outstanding to people
who
contributed to the
initial
Marilyn Aiken, Art Crotzer, Mark Dalton, Don Fisher, Huey Liu,
Gail Meinert, and Jim Van Doren.
We thank J. S. Bach; whose magnificent contribution of music to work
edition:
this work possible.
Most important of all, we thank
by makes
Pauline, Rachel, Karen, Joshua and
Peter for putting
up with
write, are tired
day, and stay up too late
all
fathers
and husbands
at
who
night to
up too early to
write some more. It's
get
the price of fame.
Boulder, Colorado
B.Z.
Urbana,
M.F.
Illinois
Contents
Introduction to File Structures
1.1
The Heart of
1.2
A
A
1.3
File Structure
Short History of
Key Terms
Fundamental
File Processing
Physical Files and Logical Files
2.2
Opening
Operations
Files
2.3
Closing
2.4
Reading and Writing
13
Files
14
2.4.1 Read and Write Functions
2.4.2 A Program
to Display the
2.4.3 Detecting End-of-File
Seeking
File Structure Literacy
2.1
2.5
Design
File Structure
Conceptual Toolkit:
Summary
Design
14
Contents of a
File
15
18
18
2.5.1 Seeking
in
2.5.2 Seeking
in
Pascal
19
20
2.6
Special Characters in Files
2.7
The
2.8
Physical and Logical Files in
UNIX
21
Directory Structure
22
UNIX
2.8.1 Physical Devices as UNIX Files
23
23
IX
CONTENTS
2.8.2 The Console, the Keyboard, and Standard Error
2.8.3
2.9
File-related
2.10
UNIX
Summary
and Pipes
I/O Redirection
Header
27
Key Terms
Further Readings
26
Files
Commands
Filesystem
24
25
29
26
Exercises
31
33
Secondary Storage and System Software
3.1
Disks
37
3.1.1 The Organization of Disks
37
3.1.2 Estimating Capacities and Space Needs
3.1.3 Organizing Tracks by Sector
3.1.4 Organizing Tracks by Block
38
41
45
47
3.1.5 Nondata Overhead
3.1.6 The Cost of a Disk Access
49
3.1.7 Effect of Block. Size on Performance: A UNIX Example
Magnetic Tape
56
3.2.1 Organization of Data on Tapes
56
3.2.2 Estimating Tape Length Requirements
3.2.3 Estimating Data Transmission Times
3.2.4 Tape Applications
3.3
Disk Versus Tape
3.4
Storage as
3.5
A Journey
61
Byte
3.5.1 The
File
3.5.2 The
I/O Buffer
62
63
64
Manager
64
3.5.3 The Byte Leaves RAM: The
3.6
Buffer
Management
3.6.2 Buffering Strategies
I/O
in
UNIX
I/O
Processor and Disk Controller
68
3.6.1 Buffer Bottlenecks
3.7
57
59
60
Hierarchy
of
53
54
3.1.8 Disk as Bottleneck
3.2
35
69
69
72
3.7.1 The Kernel
72
3.7.2 Linking
File
Names
3.7.3 Normal
Files,
to Files
76
Special Files, and Sockets
78
66
CONTENTS
3.7.4 Block
78
I/O
3.7.5 Device Drivers
79
3.7.6 The Kernel and Filesystems
3.7.7 Magnetic Tape and UNIX
Summary
80
Key Terms
Further Readings
Field
87
Exercises
91
Fundamental
4.1
82
79
80
File Structure
and Record Organization
4.1.1 A Stream
Concepts
94
94
File
4.1.2 Field Structures
96
4.1.3 Reading a Stream
of Fields
4.1.4 Record Structures
99
101
4.1.5 A Record Structure That Uses a Length Indicator
4.1.6 Mixing Numbers and Characters: Use of a
4.2
Record Access
File
109
4.2.2 A Sequential Search
4.2.3 UNIX Tools
111
for Sequential
Processing
4.2.4 Direct Access
115
More about Record
Structures
114
117
4.3.1 Choosing a Record Structure and Record Length
4.3.2 Header Records
4.4
4.5
File
Access and
Beyond Record
File
120
Organization
Structures
122
123
124
4.5.1 Abstract Data Models
4.5.2 Headers and Self-Describing Files
4.5.3 Metadata
125
125
4.5.4 Color Raster Images
4.5.5 Mixing Object Types
4.5.6 Object-oriented
4.5.7 Extensibility
4.6
103
Dump
109
4.2.1 Record Keys
4.3
93
File
128
in
One
Access
File
132
133
Portability and Standardization
134
4.6.1 Factors Affecting Portability
134
4.6.2 Achieving Portability
136
129
117
107
XI
XJI
CONTENTS
Summary
142
Further Readings
Programs
Pascal
152
Programs
167
Files for
Data Compression
Performance
183
185
185
5.1.1 Using a Different Notation
5.1.2 Suppressing Repeating Sequences
5.1.3 Assigning Variable-length Codes
186
188
5.1.4 Irreversible Compression Techniques
5.2
146
Exercises
144
153
Organizing
5.1
Key Terms
5.1.5 Compression
in
UNIX
189
Reclaiming Space
in Files
190
189
5.2.1 Record Deletion- and Storage Compaction
190
5.2.2 Deleting Fixed-length Records for Reclaiming Space Dynamically
5.2.3 Deleting Variable-length Records
198
5.2.4 Storage Fragmentation
5.2.5 Placement Strategies
5.3
201
An
Finding Things Quickly:
Searching 203
5.3.1 Finding Things
in
Introduction to Internal Sorting and Binary
Simple Field and Record
5.3.2 Search by Guessing: Binary Search
RAM
File in
Files
203
204
5.3.3 Binary Search versus Sequential Search
5.3.4 Sorting a Disk
204
206
5.3.5 The Limitations of Binary Searching and Internal Sorting
5.4
Keysorting
209
5.4.2 Limitations of the Keysort Method
5.4.3 Another Solution:
Why
5.4.4 Pinned Records
213
Further Readings
207
208
5.4.1 Description of the Method
Summary 214
Key Terms
223
192
196
211
Bother to Write the File Back?
217
Exercises
219
212
CONTENTS
Indexing
225
6.1
What
6.2
6.3
Basic Operations on an Indexed, Entry-Sequenced File
6.4
Indexes That Are
6.5
Indexing to Provide Access by Multiple Keys
6.6
Retrieval
6.7
Improving the Secondary Index Structure: Inverted
an Index?
Is
226
Simple Index with an Entry-Sequenced
6.7.1 A
Too Large
to
Hold
227
Memory
First
Attempt
6.8
Selective Indexes
6.9
Binding
230
234
235
Using Combinations of Secondary Keys
239
242
Lists
242
at a Solution
6.7.2 A Better Solution: Linking the
Summary
in
File
244
References
List of
248
249
Key Terms
250
Further Readings
252
253
Exercises
256
Cosequential Processing and the Sorting
of Large
7.1
Files
A Model
for
257
Implementing Cosequential Processes
7.1.1 Matching
Names
in
7.1.2 Merging Two Lists
7.1.3
7.2
Summary
Lists
Application of the
Model
7.2.1 The Problem
268
of the
to a General
Model
Extension of the Model
to the
7.3.2 A Selection Tree
Second Look
at
for
Model
266
Ledger Program
Ledger Program
276
276
Merging Large Numbers
Sorting in
268
271
to Include Multiway Merging
7.3.1 A K-way Merge Algorithm
7.4
259
263
of the Cosequential Processing
7.2.2 Application
7.3
Two
259
RAM
7.4.1 Overlapping Processing and
Heapsort
I/O:
7.4.2 Building the Heap while Reading
7.4.3 Sorting while Writing out to the
of Lists
279
in
File
the File
283
280
281
278
XIII
CONTENTS
XIV
7.5
Merging
7.5.1
Way
as a
of Sorting Large
How Much Time Does
7.5.2 Sorting a
File
That
on Disk
Merge Sort Take?
Number
of
292
File Size
7.5.4 Hardware-based Improvements
285
287
290
Ten Times Larger
Is
7.5.3 The Cost of Increasing the
7.5.5 Decreasing the
Files
293
Seeks Using Multiple-step Merges
7.5.6 Increasing Run Lengths Using Replacement Selection
304
7.5.7 Replacement Selection Plus Multistep Merging
7.5.8 Using Two Disk Drives with Replacement Selection
7.5.9 More Drives? More Processors?
7.5.10 Effects
of
Multiprogramming
7.5.11 A Conceptual Toolkit
7.6
Sorting Files on Tape
312
314
7.6.4 Tapes versus Disks
for External Sorting
Sort-Merge Packages
318
Sorting and Cosequential Processing in
7.8.1 Sorting and Merging
in
UNIX
7.8.2 Cosequential Processing
Summary
Key Terms
322
Further Readings
310
315
7.6.3 Multiphase Merges
310
for External Sorting
7.6.2 The K-way Balanced Merge
7.8
307
309
311
7.6.1 The Balanced Merge
7.7
295
298
UNIX
318
318
Utilities in
325
317
UNIX
Exercises
320
328
331
B-Trees and Other Tree-structured
File Organizations 333
The Invention of the B-Tree
8.1
Introduction:
8.2
Statement of the Problem
8.3
Binary Search Trees
8.4
AVL
8.5
Paged Binary Trees
8.6
The Problem with
8.7
B-Trees: Working up from the
8.8
Splitting
Trees
as a
334
336
Solution
337
340
343
the
Top-down Construction of Paged
and Promoting
347
Bottom
347
Trees
345
XV
CONTENTS
8.9
Algorithms for B-Trcc Searching and Insertion
8.10
B-Tree Nomenclature
8.11
Formal Definition of B-Trce Properties
8.12
Worst-case Search Depth
8.13
Deletion, Redistribution, and Concatenation
364
364
Redistribution during Insertion:
A Way
to
Improve Storage
371
Utilization
8.15
B* Trees
8.16
Buffering of Pages: Virtual B-Trees
372
8.16.1 LRU Replacement
373
375
376
8.16.2 Replacement Based on Page Height
8.16.3 Importance
of Virtual B-Trees
377
8.17
Placement of Information Associated with the Key
8.18
Variable-length Records and Keys
Summary
Key Terms
380
Further Readings
Programs
Pascal
366
370
8.13.1 Redistribution
8.14
352
362
379
383
Exercises
387
to Insert
Programs
382
377
Keys
into a
Keys
to Insert
B-Tree
into a
389
B-Tree
397
The B + Tree Family and Indexed
Sequential
File Access 405
9.1
9.2
406
Indexed Sequential Access
Maintaining
Sequence Set
407
9.2.1 The Use of Blocks
9.2.2 Choice
of
Block Size
9.3
Adding
9.4
The Content of the
407
410
Simple Index to the Sequence Set
9.5
The Simple
9.6
Simple Prefix
Prefix
B+
41
Index: Separators Instead of
Tree
Keys
413
416
Tree Maintenance
417
9.6.1 Changes Localized to Single Blocks
9.6.2 Changes Involving Multiple Blocks
in
in
9.7
Index Set Block Size
9.8
Internal Structure of Index Set Blocks:
the Sequence Set
the Sequence Set
417
418
421
Variable-order B-Tree
422
CONTENTS
XVI
9.9
Loading
+
9.10
9.11
B-Trees,
B+
10.1
425
Trees, and Simple Prefix
Key Terms
436
B+
Trees in Perspective
437
Exercises
443
Further Readings
Hashing
Tree
429
Trees
Summary 434
B+
Simple Prefix
445
446
Introduction
10.1.1 What
is
Hashing?
447
448
10.1.2 Collisions
10.2
10.3
Hashing Functions and Record Distributions
Simple Hashing Algorithm
among Addresses
10.3.1 Distributing Records
10.3.2
Some
450
455
Other Hashing Methods
456
10.3.3 Predicting the Distribution of Records
10.3.4 Predicting Collisions
10.4
How Much
Extra
for a Full File
Memory
10.4.2 Predicting Collisions
Collision Resolution
10.5.1
How
Storing
for Different
of
Packing Densities
per Address: Buckets
10.8
of
476
Tombstones
of Deletions
480
for Insertions
481
and Additions on Performance
Other Collision Resolution Techniques
10.8.1 Double Hashing
483
483
10.8.2 Chained Progressive Overflow
484
10.8.3 Chaining with a Separate Overflow Area
10.8.4 Scatter Tables: Indexing Revisited
10.9
471
472
479
10.7.1 Tombstones for Handling Deletions
10.7.3 Effects
466
468
10.6.2 Implementation Issues
10.7.2 Implications
463
467
Buckets on Performance
Making Deletions
462
by Progressive Overflow
More Than One Record
10.6.1 Effects
10.7
Should Be Used?
Progressive Overflow Works
10.5.2 Search Length
10.6
461
462
10.4.1 Packing Density
10.5
453
454
Patterns of Record Access
488
487
486
482
431
CONTENTS
Summary 489
Key Terms
Further Readings
492
Extendible Hashing
11.1
Introduction
11.2
How
503
504
Extendible Hashing Works
505
505
11.2.1 Tries
11.2.2 Turning the Trie
11.2.3 Splitting
11.3
495
Exercises
501
Implementation
507
into a Directory
Handle Overflow
to
508
510
510
11.3.1 Creating the Addresses
11.3.2 Implementing the Top-level Operations
11.3.3 Bucket and Directory Operations
11.3.4 Implementation Summary
11.4
513
514
519
520
Deletion
11.4.1 Overview
of the Deletion
11.4.2 A Procedure
520
Process
Buddy Buckets
for Finding
522
11.4.4 Implementing the Deletion Operations
11.4.5 Summary
11.5
Extendible Hashing Performance
526
11.5.1 Space
526
11.5.2 Space Utilization
11.6
11.6.1 Dynamic Hashing
11.6.3 Approaches
Further Readings
Appendix A:
Appendix
Using
A. 2.
Exercises
File Structures
Introduction to
535
533
537
539
A.2
A. 2.1
528
to Controlling Splitting
Key Terms
A.l
this
527
528
530
11.6.2 Linear Hashing
534
Buckets
for the Directory
Alternative Approaches
Summary
526
of the Deletion Operation
Utilization for
520
522
11.4.3 Collapsing the Directory
CD-ROM
as a File
CD-ROM
542
CD-ROM
Short History of
on
543
CD-ROM
543
Structure Problem
545
541
XVII
XVIII
CONTENTS
A.3
Physical Organization of
A. 3.1 Reading
A. 3. 2
CLV
Pits
CAV
Instead of
547
549
A. 3. 4 Structure of a Sector
A.4
CD-ROM
Strengths and Weaknesses
A.4. 2 Data Transfer Rate
552
552
A. 4. 3 Storage Capacity
A. 4. 4
553
Read-Only Access
553
A.4. 5 Asymmetric Writing and Reading
A.5
Tree Structures on
A.
Loading Procedures and Other Considerations
5 Trees as Secondary Indexes on
Hashed
A.
Files
CD-ROM
on
Bucket Size
A. 6. 3
How
The
CD-ROM
System
Helps
558
Design Exercise
560
A. 7. 3
A Hybrid Design
562
559
563
Appendix B: ASCII Table
566
Appendix C: String Functions
in Pascal: tools. pre
Functions and Procedures Used to Operate on strng
Appendix D: Comparing Disk Drives
Bibliography
Index
559
559
A. 7. 2
Summary
556
CD-ROM's Read-Only Status
File
The Problem
CD-ROM
557
558
the Size of
CD-ROM
A. 7.1
555
556
557
6.1 Design Exercises
A. 6. 2
A.6. 4 Advantages of
A.7
553
5.4 Virtual Trees and Buffering Blocks
A. 5.
A.6
553
554
5.2 Block Size
A. 5.3 Special
A.
CD-ROM
Design Exercises
A. 5.1
552
552
Seek Performance
A. 4.1
546
546
549
3 Addressing
A. 3.
CD-ROM
and Lands
581
575
567
572
567
Introduction to
File Structures
CHAPTER OBJECTIVES
Introduce the primary design issues that characterize
file
structure design.
Survey the history of
file
ing the developments in
much about how
structure design, since trac-
to design
Introduce the notions of
structures teaches us
file
our
file
own
file
structures.
structure literacy and of
a conceptual toolkit for file structure design.
CHAPTER OUTLINE
1.1
1.2
The Heart of File Structure Design
1.3
mm
Conceptual Toolkit: File
Structure Literacy
Short History of File Structure
Design
1.1
The Heart
Design
of File Structure
Disks are slow. They are also technological marvels, packing hundreds of
megabytes on disks that can fit into a notebook computer. Only a few years
ago, disks with that kind of capacity looked like small washing machines.
However, relative to the other parts of a computer, disks are slow.
How slow? The time it takes to get information back from even
relatively slow electronic random access memory (RAM) is about 120
nanoseconds, or 120 billionths of a second. Getting the same information
from a typical disk might take 30 milliseconds, or 30 thousandths of a
second. To understand the size of this difference, we need an analogy.
Assume that RAM access is like finding something in the index of this
book. Let's say that this local, book-in-hand access takes 20 seconds.
Assume that disk access is like sending to a library for the information you
cannot find here in this book. Given that our "RAM access" takes 20
seconds,
how
long does the "disk access" to the library take, keeping the
ratio the
same
as that
is
of a
real
RAM access and disk access? The disk access
RAM access. This means that
quarter of a million times longer than the
getting information back
from the
library takes 5,000,000 seconds,
almost 58 days. Disks are very slow compared to
than
or
RAM.
On the other hand, disks provide enormous capacity at much less cost
RAM. They also keep the information stored on them when they are
turned off. The tension between
enormous, nonvolatile capacity
design.
Good
file
a disk's relatively
is
slow access time and
the driving force behind
structure design will give us access to
all
file
the capacity
without making our applications spend a lot of time waiting for the
how to develop such file designs.
This book shows you
its
structure
disk.
A SHORT HISTORY OF FILE STRUCTURE DESIGN
1.2
A Short
History of File Structure Design
Put another way, our goal is to show you how to think creatively about file
structure design problems. Part of our approach to doing this is based on
history: After introducing basic principles of design in the first part of this
book, we devote the last part to studying some of the key developments in
file design over the last 30 years. The problems that researchers struggle
same issues that you confront in addressing any substantial
Working through the approaches used to address major
design issues shows you a lot about how to approach new design
with
file
file
reflect the
design problem.
problems.
The general goals of research and development
drawn directly from our library analogy:
we would
be
in file structures can
we need with one acwe do not want to issue a
series of 58-day requests before we get what we want.
If it is impossible to get what we need in one access, we want strucIdeally,
like to get the
cess to the disk. In terms
information
of our analogy,
tures that allow us to find the target information with as
as possible.
For example, you
may remember from your
few accesses
studies of
data structures that a binary search allows us to find a particular
record
among
50,000 other records with no more than 16 compari-
to look 16 places on a disk before finding what we
want takes too much time. We need file structures that allow us to
find what we need with only two or three trips to the disk.
We want our file structures to group information so we are likely to
But having
sons.
we need with only one trip to the disk. If we need
name, address, phone number, and account balance, we
get everything
client's
would
to
prefer to get
all
that information at once, rather than having
look in several places for
It is
relatively easy to
these goals
when we have
it.
come up with
files
file
structure designs that
that never change.
that maintain these qualities as
files
Designing
file
meet
structures
change, growing and shrinking as
added and deleted, is much more difficult.
files presumed that files were on tape, since most files
were. Access was sequential, and the cost of access grew in direct
proportion to the size of the file. As files grew intolerably large for unaided
sequential access and as storage devices like disk drives became available,
indexes were added to files. The indexes made it possible to keep a list of
keys and pointers in a smaller file that could be searched more quickly;
given the key and pointer, the user had direct access to the large, primary
information
Early
file.
is
work with
INTRODUCTION TO
STRUCTURES
FILE
some of the same, sequential flavor
grew they too became
Unfortunately, simple indexes had
as
the data
themselves, and as the indexes
files
manage, especially for dynamic files in which the set of keys
changes. Then, m the early 1960s, the idea of applying tree structures
emerged as a potential solution. Unfortunately, trees can grow very
unevenly as records are added and deleted, resulting in long searches
difficult to
requiring
many
disk accesses to find a record.
1963 researchers developed the
In
AVL
tree,
an elegant, self-adjusting
RAM.
Other researchers began to look for
or something like them, to files. The problem
binary tree structure for data in
ways to apply AVL trees,
was that even with a balanced binary
tree,
dozens of accesses are required
even moderate-sized files. A way was needed to keep a
tree balanced when each node of the tree was not a single record, as in a
binary tree, but a file block containing dozens, perhaps even hundreds,
to find a record in
of records.
It took nearly 10 more years of design work before a solution emerged
in the form of the B-tree. Part of the reason that finding a solution took so
long was that the approach required for file structures was very different
from the approach that worked in RAM. Whereas AVL trees grow from
down
the top
as
records are added, B-trees
grow from
the
B-trees provided excellent access performance, but there
longer could
a file
bottom up.
was a cost: No
be accessed sequentially with efficiency. Fortunately,
this
problem was solved almost immediately by adding a linked list structure at
the bottom level of the B-tree. The combination of a B-tree and a sequential
linked
called a
list is
Over
B + tree.
the following 10 years B-trees and
many commercial
file
entries
practical terms, this
file
entry
among
trees
became
the basis for
systems, since they provide access times that
proportion to log^N, where
number of
B+
is
indexed in
means
the
number of entries
a single
in the file
grow
and k
is
in
the
block of the B-tree structure. In
that B-trees can guarantee that
you can
find
one
millions of others with only three or four trips to the disk.
Further, B-trees guarantee that as
you add and
delete entries,
performance
stays about the same.
Being able to get information back with just three or four accesses is
But how about our goal of being able to get what we want
with a single request? An approach called hashing is a good way to do that
with files that do not change size greatly over time. From early on, hashed
indexes were used to provide fast access to files. However, until recently,
pretty good.
hashing did not work well with
volatile, dynamic files that changed greatly
development of B-trees, researchers turned to work on
extendible, dynamic hashing that could retrieve information
in size. After the
svstems tor
A CONCEPTUAL TOOLKIT:
with one
We
most, two disk accesses no matter how big the file becomes.
book with a careful look at this work, which took place from
close this
A Conceptual
part of the 1980s.
first
Toolkit: File Structure Literacy
As we move through
decades, watching
first
STRUCTURE LITERACY
or, at
the late 1970s through the
1.3
FILE
the developments in
file
file
structures over the last three
structure design evolve as
it
addresses
dynamic
files
sequentially, then through tree structures, and finally through direct
we
same design problems and design tools keep
number of disk accesses by collecting data into
buffers, blocks, or buckets; we manage the growth of these collections by
splitting them, which requires that we find a way to increase our address or
index space, and so on. Progress takes the form of finding new ways to
combine these basic tools of file design.
We think of these tools as conceptual tools. They are ways of framing
access,
emerging.
see
We
that
the
decrease the
a design problem. Our own work in file structures has
by understanding the tools thoroughly, and by studying how
have been combined to produce such diverse approaches as B-trees
and addressing
shown
us that
the tools
and extendible hashing, we develop mastery and flexibility in our own use
of the tools. In other words, we acquire literacy with regard to file
This text
structures.
is
designed to help readers acquire
file
structure
through 6 introduce the basic tools; Chapters 7 through
11 introduce readers to the highlights of the past several decades of file
structure design, showing how the basic tools are used to handle efficient
+
trees, hashed indexes, and extendible,
sequential access, B-trees, B
literacy.
Chapters
dynamic hashed
files.
SUMMARY
The key design problem that shapes
large amount of time that is required
structure designs focus
file
structure design
is
the relatively
from disk. All file
and maximizing the
to get information
on minimizing disk
likelihood that the information the user will
accesses
want
is
already in
RAM.
This text begins by introducing the basic concepts and issues associated
with file structures. The last half of the book tracks the development of file
structure design as
it
30 years. The key problem
finding ways to minimize disk
has evolved over the
addressed throughout
this
evolution
is
last
INTRODUCTION TO
FILE
STRUCTURES
files that keep changing in content and size. Tracking these
developments takes us first through work on sequential file access, then
through developments in tree-structured access, and finally to relatively
recent work on direct access to information in files.
Our experience has been that the study of the principal research and
design contributions to file structures, focusing on how the design work
uses the same tools in new ways, provides a solid foundation for thinking
creatively about new problems in file structure design.
accesses for
KEY TERMS
AVL
tree.
self-adjusting binary tree structure that can guarantee
access times for data in
B-tree.
good
RAM.
tree structure that provides fast access to data stored in files.
Unlike binary trees, in which the branching factor from a node of
is two, the descendents from a node of a B-tree can be a
much larger number. We introduce B-trees in Chapter 8.
the tree
B+
tree.
on
variation
the B-tree structure that provides sequential ac+
discuss B
trees at
cess to the data as well as fast-indexed access.
length in Chapter
We
9.
Extendible hashing. An approach to hashing that works well with files
that undergo substantial changes in size over time.
File structures. The organization of data on secondary storage devices
such
as disks.
Hashing. An
access
mechanism
that transforms the search
key into
storage address, thereby providing very fast access to stored data.
Sequential access. Access that takes records
first, then the next, and so on.
in order, looking at the
Fundamental
File
Processing Operations
CHAPTER OBJECTIVES
Describe the process of linking a logical file within
to an actual physical file or device.
program
Describe the procedures used to create, open, and
close
files.
Describe the procedures used for reading from and
writing to files.
Introduce the concept of position within a file and describe procedures for seeking different positions.
Provide an introduction to the organization of the
UNIX file system.
Present the
file
UNIX
operations and
view of
a file,
commands
and describe
based on
this
UNIX
view.
CHAPTER OUTLINE
2.1
Physical Files and Logical Files
2.6
Special Characters in Files
2.2
Opening
2.7
The
2.3
Closing Files
2.8
Physical and Logical Files in
2.4
Reading and Writing
Files
Directory Structure
UNIX
UNIX
2.8.1
Read and Write Functions
2.4.2 A Program to Display the
Physical Devices as
2.8.2
The Console,
Contents of a File
2.4.3 Detecting End-of-File
2.8.3 I/O Redirection and Pipes
2.4.1
2.5
UNIX
and Standard Error
Seeking
Seeking in C
2.5.2 Seeking in Pascal
2.5.1
2.1
Files
the Keyboard,
2.9
File-related
2.10
UNIX
File
Header
Files
System Commands
Physical Files and Logical Files
When we
talk
about
on
a file
collection of bytes stored there.
physically exists.
these physical
From
disk or tape,
we
when
word
file,
the
refer to a particular
is
used in
this sense,
disk drive might contain hundreds, even thousands, of
files.
the standpoint of an application program, the notion of a
To
file is
somewhat like a telephone line connected
to a telephone network. The program can receive bytes through this phone
line, or send bytes down it, but knows nothing about where these bytes
actually come from or where they go. The program knows only about its
own end of the phone line. Moreover, even though there may be thousands
of physical files on a disk, a single program is usually limited to the use of
different.
the program, a
only about 20
The
file is
files.
application
program
relies
on the operating system
to take care
the details of the telephone switching system, as illustrated in Fig. 2.1.
could be that bytes coming
an actual physical
file,
down
the line into the
or they might
come from
program
It
originate
from
some
other
the keyboard or
input device. Similarly, the bytes that the program sends
of
down
the line
might end up in a file, or they could appear on the terminal screen.
Although the program often doesn't know where bytes are coming from or
where they are going, it does know which line it is using. This line is usually
OPENING FILES
referred to as the logical file to distinguish this view
from the
physical
files
on
the disk or tape.
Before the program can open
making
receive instructions about
phone
such
line)
and some physical
IBM's OS/MVS,
as
file
a file for use, the
a
operating system must
hookup between
or device.
When
a logical file (e.g., a
using operating systems
these instructions are provided through job
On
minicomputers and microcomputers, more
as UNIX, MS-DOS, and VMS provide the
instructions within the program. For example, in Turbo Pascal^ the
association between a logical file called inp_jile and a physical file called
myfile.dat is made with the following statement:
control language (JCL).
modern operating systems such
assignCi np_f ile, 'myfile.dat
'
file named
make the hookup by assigning a logical file (phone
The number identifying the particular phone line that is assigned
This statement asks the operating system to find the physical
and then
myfile.dat
line) to
is
it.
to
returned through the FILE variable inp_file, which
name. This logical
name
is
what we use
is
the
to refer to the
file's
file
logical
inside the
program. Again, the telephone analogy applies: My office phone is
connected to six telephone lines. When I receive a call I get an intercom
message such as, "You have a call on line three." The receptionist does not
say, "You have a call from 918-123-4567." I need to have the call identified
logically,
2.2
not physically
Opening
Files
Once we have
hooked up to a physical file or device,
what we intend to do with the file. In general, we have
two options: (1) open an existing file or (2) create a new file, deleting any
existing contents in the physical file. Opening a file makes it ready for use
by the program. We are positioned at the beginning of the file and are ready
to start reading or writing. The file contents are not disturbed by the open
we need
a logical file identifier
to declare
statement. Creating a
file
also
opens the
file in
use after creation. Since a newly created
initially the
file
the sense that
is
compilers vary widely with regard to I/O procedures, since standard Pasway of I/O definition. Throughout this book we use the term Pasdiscussing features common to most Pascal implementations. When we refer to
cal contains little in the
when
ready for
only use that makes sense.
+ Different Pascal
cal
it is
has no contents, writing
the features of a specific implementation, such as
Turbo
Pascal,
we
say so.
10
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
Logical
files
Program
Limit of approximately
20 phone lines
Physical
files
Printer
FIGURE 2.1 The program
files
and physical
files
relies
on the operating system to make connections between
In Pascal the reset(
rewrite (
in
Turbo
logical
and devices.
statement
Pascal
is
statement
used to create
we might
ass ign( inp_f
reset ( i np_f
le
le)
use
is
used to open existing files and the
ones. For example, to open a file
new
sequence of statements such
myfile.dat*);
as:
OPENING FILES
CLEAN UP
YOUR MESS
11
SAFETY
FIRST
Operating system switchboard
Can make connections to thousands
of
Note
that
statement.
we use the
To create a
files
logical
file
in
or I/O devices
file
name, not the physical one, in the reset()
Pascal, the statements might read:
Turbo
'myfile.dat');
i 1 e
rewrite (inp_file)
a55 ign( out_f
FUNDAMENTAL
PROCESSING OPERATIONS
FILE
We can open an existing file or create a new one in C through the
UNIX system function open( This function takes two required argu).
ments and
a third
argument
that
fd = open(f ilename,
The
is
optional:
flags,
[pmode]);
return value fd and the arguments filename, flags, and pmode have the
following meanings:
fd
The fde
descriptor.
Using our
earlier analogy, this
is
the
phone
used to refer to the file within the
an integer. If there is an error in the attempt to
line (logical file identifier)
program. It is
open the file, this value
filename
flags
is
negative.
character string containing the physical
file
name. (Later
we
discuss pathnames that include directory information about
the
file's
The
argument can be
location. This
argument
pathname.)
an integer that controls the operation of
the open function, determining whether it opens an existing
file for reading or writing. It can also be used to indicate that
you want to create a new file, or open an existing file but delete its contents. The value of flags is set by performing a bitwise
of the following values, among others. 1
flags
is
"
OR
D_APPEND
Append every
the
0_CREAT
write operation to the end of
file.
Create and open
no
a file for writing.
This has
effect if the file already exists.
CUE X C L
Return an error if 0_C R E A T E
and the file exists.
CURDONLY
CURDWR
D_TRUNC
Open
a file for
reading only.
Open
a file for
reading and writing.
If the file exists, truncate
zero, destroying
CUHRONLY
Open
its
it
is
specified
to a length
of
contents.
a file for writing only.
Some of these flags cannot be used in combination with one
another. Consult your documentation for details, as well as
for other options.
pmode
If
0_C RE AT
ment
pmode
is
specified,
pmode
specifies the protection
is
a three-digit octal
is
required. This integer argufile. In UNIX, the
mode for the
number
that indicates
how
the
file
can be used by the owner (first digit), by members of the
owner's group (second digit), and by everyone else (third
\
+ These
piler.
values are defined in an "include"
The name of
system.
the include
file is
file
packaged with your
often fcntl.h or file. h, but
it
UNIX
system or
C com-
can vary from system to
CLOSING FILES
digit).
The
sion, the
first bit of each octal digit indicates read permissecond write permission, and the third execute per-
mission. So,
owner has
if pmode is
the octal
number
0751, the
file's
and execute permission for the file; the
owner's group would have read and execute permission; and
everyone else has only execute permission:
read, write,
rwe rwe rwe
111
10
owner
group
PMODE = 0751=
world
Given this description of the open( ) function, we can develop some
examples to show how it can be used to open and create files in C. The
following function call opens an existing file for reading and writing, or
creates a new one if necessary. If the file exists it is opened without change;
reading or writing
would
start at the file's first byte.
fd = openCf ilename,
The following
already
fd
a file
new
is
new
file
only
0_TRUNC, 0751);
if there is
this
not already
name
exists,
it is
negative value to indicate an error.
0_CREAT
If there is
contents are truncated.
with
in filename. If a file
openCf ilename, D_RDNR
and writing.
its
0_CREAT
not opened and the function returns
fd
0_CREAT, 0751);
in filename,
a call that will create a
with the name specified
for reading
file
with the name specified
openCf ilename, 0_RDWR
Finally, here
file
call creates a
0_RDWR
D_EXCL,
0751);
is tied more to the host operating system than to a
For example, implementations of Pascal running on
systems that support file protection, such as VAX/VMS, often include
extensions to standard Pascal that let you associate a protection status with
File protection
specific language.
a file
2.3
when you
create
it.
Closing Files
file is like hanging up the
is available for taking
phone
line
phone,
the
the
hang
up
phone. When you
logical
file name or file
the
file,
close
a
or placing another call; when you
In terms of our telephone line analogy, closing a
descriptor
is
available for use with another
file.
Closing
a file that
has been
used for output also ensures that everything has been written to the file. As
you will learn in a later chapter, it is more efficient to move data to and from
secondary storage in blocks than
it
is
to
move
data one byte at a time.
14
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
Consequently, the operating system does not immediately send off the
we write, but saves them up in a buffer for transfer as a block of data.
Closing a file makes sure that the buffer for that file has been flushed of data
bytes
and that everything
we have
written has actually been sent to the
file.
automatically by the operating system when a
program terminates normally. Consequently, the explicit use of a CLOSE
Files are usually closed
statement within
program
is
needed only
as protection against data loss in
the event of program interruption and to free
up
logical filenames for reuse.
Some
languages, including Standard Pascal, do not even provide a CLOSE
statement. However, explicit file closing is possible in the C language,
Pascal, PL/I, and most other languages used for serious file processing
VAX
work.
Now
2.4
how to connect and disconnect programs to and
how to open the files, you are ready to start sending
you know
that
from physical
files
and receiving
data.
and
Reading and Writing
Reading and writing are fundamental to
that
make
file
file
processing; they are the actions
processing an input/output (I/O) operation.
The
actual
form of
the read and write statements used in different languages varies.
Some
languages provide very high-level access to reading and writing and
automatically take care of details for the programmer. Other languages
provide access
explore
at a
much lower
some of these
level.
Our
use of Pascal and
allows us to
differences.^
2.4.1 Read and Write Functions
We begin here with
reading and writing at a relatively low level. It is useful
have a kind of systems-level understanding of what happens when we
send and receive information to and from a file.
A low-level read call requires three pieces of information, expressed
here as arguments to a generic READ( ) function.
to
RE ADC Sour ce_f ile,Destinati on_addr Size)
Source_f i le
The RE AD(
We
to read.
name (phone
'To accentuate these differences and provide
systems level, we use the read( ) and write(
functions such as fgetc( ), jgets( ), and so on.
a
)
call
must know from where it
by logical file
through which data is re-
is
specify the source
line)
look at I/O operations at something closer to
system calls in C rather than higher-level
15
READING AND WRITING
(Remember, before we do any reading
already opened the file, so the
connection between a logical file and a specific
ceived.
we must have
physical
file
or device already exists.)
READ( must know
Dest inat ion_addr
where to place the inforfrom the input file. In this gefunction we specify the destination by
)
mation
neric
it
reads
giving the
address of the
first
where we want
Size
Finally,
READ(
memory
block
to store the data.
)
must know how much inin from the file. Here the
formation to bring
argument
A WRITE statement is
is
supplied as a byte count.
similar; the only difference
that the data
is
moves
in
the other direction:
NRITECDestinat ion_f i
Destinat i on_f
i 1
le
The
Source_addr
logical
name we
file
ze)
use for sending the
data.
Source_addr
WRITE(
must know where
mation that
it
ification as the first
Size
to find the infor-
We
provide this specaddress of the memory
will send.
block where the data
is
The number of bytes
to be written
stored.
must be
supplied.
2.4.2 A Program to Display the Contents of a
File
do some reading and writing to see how these functions are used. This
file processing program, which we call LIST, opens a file for
input and reads it, character by character, sending each character to the
screen after it is read from the file. LIST includes the following steps:
Let's
simple
first
Display a prompt for the
2.
Read the
user's response
name of the
input
from the keyboard
file.
into a variable called file-
name.
3.
Open
4.
While there
5.
the
file
for input.
are
still
characters to be read
from the input
a.
read a character from the
b.
write the character to the terminal screen.
Close the input
file
file.
Figures 2.2 and 2.3 are, respectively,
tions
of
this
file,
and
program.
these implementations.
It is
and Pascal language implementabetween
instructive to look at the differences
FUNDAMENTAL
/*
**
*/
1 i
5 t
c
.
PROCESSING OPERATIONS
FILE
-program to read characters from
to the terminal screen
file and write them
^include <stdio.h>
^include <fcntl.h>
ma i n(
char
fd;
/ *
file descriptor */
char filename[20];
int
printf ("Enter the name of the file:
get s( f i 1 ename)
fd =open(f i lename, D_RD0NLY);
while (readCfd, &c
1)
wr iteCSTDOUT, &c
"
!=
0)
1);
close(fd);
FIGURE 2.2 The LIST program
/*
/*
/*
Step
Step
Step
/*
/*
Step 4a */
Step 4b */
/*
Step
*/
*/
*/
*/
in C.
Steps 1 and 2 of the program involve writing and reading, but in each
of the implementations' this is accomplished through the usual functions for
handling the screen and keyboard. Step 4a, where we read from the input
the first instance of actual file I/O. Note that the read( ) call in the C
language parallels the low-level, generic READ( ) statement we described
file, is
system call in C as the model for our
argument gives the file descriptor
(C's version of a logical file name) as the source for the input, the second
argument gives the address of a character variable used as the destination for
the data, and the third argument specifies that only one byte will be read.
earlier; in truth,
low-level
we
READ(
).
The arguments
information
name
file
at a
used the read(
The
for
function's
first
the Pascal
read(
call
communicate the same
Once again, the first argument is the logical
source. The second argument gives the name of a
higher
level.
for the input
character variable used as a destination; given the name, Pascal can find the
address. Because of Pascal's strong emphasis
argument of the generic
that since
we
READ(
function
is
on variable
types, the third
not required. Pascal assumes
are reading data into a variable of type char,
we must want
to
read only one byte.
After a character
is
read,
again the differences between
to
I/O used
in
we
write
it
out to the screen in Step 4b.
Once
and Pascal indicate the range of approaches
different languages. Everything must be stated explicitly in
17
READING AND WRITING
the
write (
Using the
) call.
special, assigned file descriptor
of
STDOUT
to identify the terminal screen as the destination for our writing,
wnte( STDOUT,
&c
means: "Write to the screen the contents from memory starting at the
address Sec. Write only one byte." Beginning C programmers should pay
special attention to the use
particular
call,
as a
of the
&
symbol
very low-level
provide the starting address
in
RAM
call,
in the write(
call here; this
requires that the
programmer
of the bytes to be transferred.
STDOUT,
which stands for "standard output," is an integer value
defined in the file stdio.h, which has been included at the top of the program. The actual value of STDOUT that is set in stdio.h is, by convention,
always
The concept of standard output and
1.
counterpart "standard
its
input" are covered later in the section "Physical and Logical
Files
in
UNIX."
FIGURE 2.3 The LIST program
Pascal.
in
PROGRAM list (INPUT, OUTPUT);
{
reads input from
file and writes
it
to
screen
the terminal
>
VAR
c
char;
file of char;
<
logical file name
packed array [1..201 of char;
physical file name
infile
filename
BEGIN {main>
wnteCEnter
the name of the file:
readln(f i lename)
filename);
reset ( mf l le
while not ( eof ( i nf l 1 e ) ) DO
;
');
{
<
<
Step
Step
Step
>
>
>
BEGIN
readdnf ile,c)
write(c)
<
Step 4a
Step 4b
<
Step
<
>
>
END;
closednf lie)
END.
>
>
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
Again the Pascal code operates
name
is
specified in a writef
higher
at a
When no logical file
we are writing
char, Pascal assumes we
level.'*'
statement, Pascal assumes that
to the terminal screen. Since the variable
c is
of type
The statement becomes simply
are writing a single byte.
write(c)
As
in the read(
bytes; the
statement, Pascal takes care of finding the address of the
programmer need
specify only the
name of the
variable
that
is
associated with that address.
2.4.3 Detecting End-of-File
in Figs. 2.2 and 2.3 have to know when to end the while loop
and stop reading characters. Pascal and C signal the end-of-file condition
differently, illustrating two of the most commonly used approaches to
The programs
end-of-file detection.
Pascal supplies
end-of-file.
Boolean function,
As we read from
eof(
),
which can be used
to test for
operating system keeps track of our
a file, the
with a read/write pointer. This is necessary so when the
system knows where to get it. The eof( ) function
queries the system to see whether the read/ write pointer has moved past the
last element in the file. If it has, eof( ) returns true; otherwise it returns false.
location in the
next byte
As
file
read, the
is
Fig. 2.3 illustrates,
byte. For an
empty
we
file,
use the eof(
eof(
) call
before trying to read the next
immediately returns
true
and no bytes are
read.
In the
read(
2.5
language, the read(
) call
returns the
returns a value of zero, then the
file.
So, rather than using an eof(
run
as
long
function,
as the read( ) call finds
number of bytes
program has reached
we
something
read. If
the end of the
construct the while loop to
to read.
Seeking
In the preceding
sample programs
reading one byte after another until
a
byte
we
is
read, the operating
we read through the
we reach the end of the
system moves
file sequentially,
file.
Every time
the read/write pointer ahead, and
are ready to read the next byte.
C does not have similar high-level functions. In fact, the standard C
panoply of higher-level I/O functions, including putc( ), which functions
for characters exactly like the Pascal write( ) shown here. We have chosen to emphasize the
use of the lower-level C functions mainly for pedagogical reasons. They provide opportunities for us to understand more fully the way file I/O works.
""This is
not to say that
library provides a
19
SEEKING
Sometimes we want to read or write without taking the time to go
through every byte sequentially. Perhaps we know that the next piece of
information we need is 10,000 bytes away, and so we want to jump there
to begin reading. Or perhaps we need to jump to the end of the file so we
can add
new information there. To satisfy these needs we must
movement of the read/write pointer.
be able to
control the
The
action of
called seeking.
moving
directly to a certain position in a
seek requires at least
two
often
file is
pieces of information, expressed
SEEK(
here as arguments to the generic pseudocode function
):
SEEK ( Sou rce_f ile,Df f set)
Source_f
Offset
The
le
logical
file
The number of
moved from
Now,
name
in
which the seek
positions in the
the start of the
file
will occur.
the pointer
is
to be
file.
we want to move directly from the origin to the 373rd position in
we don't have to move sequentially through the first 372
positions first. Instead, we can say
if
a file called data,
SEEK( data,
2.5.1 Seeking
in
373
UNIX
been incorporated into many
implementations of the C language is the ability to view a file as a
potentially very large array of bytes that just happens to be kept on secondary
storage. In an array of bytes in RAM, we can move to any particular byte
One of
the features of
that has
through the use of a subscript. The
provides
any byte in
The
C language seek function,
a similar capability for files.
where the
called lseek(
function has the following form:
lseekCfd, byte_offset, origin)
j^^citz
K~
variables have the following meanings:
pos
A long integer value returned by lseek( ) equal to the
position (in bytes) of the read/write pointer after it has
been moved.
fd
The
file
descriptor of the
file
to
which the
lseek(
) is
to
be applied.
byte_offset
),
us set the read/ write pointer to
a file.
lseek(
pos =
It lets
The number of bytes to move from some origin in the
file. The byte offset must be specified as a long integer,
hence the name Iseek for long seek. When appropriate,
the byte_offset can be negative.
20
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
origin
value that specifies the starting position from which the
is to be taken. The origin can have the value 0, 1,
byte_offset
or
2-f
heek(
from
lseek(
from the current
2lseek(
from the end of the
the beginning of the
position;
file.
^CzAL^
file;
The following program fragment shows how you could
move
to a position that
long pos
i
nt
f d
seek
is
( i
373 bytes into
fd,
nt
2.5.2 Seeking
in
373L,
a file is a
byte-by-byte
long offset,
to
origin);
int
in Pascal differs
acter or integer, or
a file in
from the
view
sequence of bytes, so addressing within the
basis.
When we
particular type.
within
Pascal
seek to a position,
dress in terms of bytes. In Pascal a
some
use lseek(
a file.
0);
The view of a file as presented
two important respects:
B.AJk^
pos=lseek(fd
In
& Ll 9^-
^C^? r* -
S^C^-
3fc?
it
Pascal
may
is
file is a
record can be
in
be
we
in at least
on
file is
express the ad-
sequence of "records" of
simple scalar such as a char-
more complex
structure.
Addressing
terms of these records. For example,
if a
made up of 100-byte records, and we want to refer to the
fourth record, we would do so in Pascal simply by referencing
record number 4. In C, where the view is solely and always in terms
of bytes, we would have to address the fourth record as byte address
file is
400.
Standard Pascal actually does not provide for seeking. The model for
I/O for standard Pascal
tially.
is
magnetic
tape,
which must be read sequen-
In standard Pascal, adding data to the
reading the entire
file
from beginning
end of
a file
requires
to end, writing out the data
from the input file to a second, output file, and then adding the new
data to the end of the output file. However, many implementations
of Pascal such as VAX Pascal and Turbo Pascal have extended the
standard and do support seeking.
1, and 2 are almost always used here, they are not guaranteed to
implementations. Consult your documentation.
"'"Although the values 0,
work
tor
all
SPECIAL CHARACTERS
There
ANSI/IEEE
an extension to Pascal proposed by the Joint
is
Standards Committee (1984) that
the future.
It
may
21
IN FILES
Pascal
be included in the Pascal standard
in
includes the following procedures and functions that permit
seeking:
SeekWrite(f,n) A procedure that positions the file /on the element
with index n and places the file in write mode, so the selected
and following elements may be modified.
SeekRead(f,n) A procedure that positions the file /on the element
with index n and places the file in read mode, so the selected and
following elements
position
beyond
the end of the
Position(f)
may
file,
If
SeekRead(
then the
attempts to
positioned
file is
at
file.
function that returns the index value representing the
position of the current
EndPosition(f)
file
element.
function that returns the index value representing
the position of the last
Many
be examined.
the end of the
file
element.
Pascal implementations, recognizing the need to provide seeking
had already implemented seeking functions before these
Consequently, the mechanisms for handling
seeking vary widely among implementations.
capabilities,
proposals were set forth.
2.6
Special Characters
in Files
As you create the file structures described in this
some difficulty with extra, unexpected characters
you may encounter
up in your files,
text,
that turn
with characters that disappear, and with numeric counts that are inserted
your files. Here are some examples of the kinds of things you might
into
encounter:
On many
small computers you
value of 26)
is
appended
at
may
the end of your
use this to indicate end-of-file even
This
is
most
likely to
find that a
happen on
if
Control-Z (ASCII
Some
files.
applications
you have not placed
MS-DOS
it
there.
systems.
Some systems adopt
file
1"
value of 13)
""When
we
characters
a convention of indicating end-of-line in a text
of characters consisting of a carriage return (CR: ASCII
and a line feed (LF: ASCII value of 10). Sometimes I/O
as a pair
use the term text file in this text,
a specific standard character
from
wise specified, the ASCII character
describes the
ASCII
character
set.
set
we
are referring to a
file
such as ASCII or
will be assumed. Appendix
set,
consisting entirely of
EBCDIC.
Unless other-
contains a table that
>
t
^~Q? (^
22
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
procedures written for such systems automatically expand single
characters or
LF
characters into
CR-LF
pairs.
tion of characters can cause a great deal of difficulty. Again,
most
likely to
encounter
phenomenon on
this
Users of larger systems, such
the opposite problem. Certain
riage return characters
them with
as a line
a count
of
file
of the characters
in
you
are
systems.
find that they have just
formats under
file
from your
MS-DOS
VMS
remove car-
without asking you, replacing
what the system has perceived
text.
These are just
that record
VMS, may
as
CR
This unrequested addi-
few examples of the kinds of uninvited modifications
or I/O support packages might make to
management systems
your files. You will find that they are usually associated with the concepts
of a line of text or end of a file. In general, these modifications to your files
are an attempt to make your life easier by doing things for you
automatically. This might, in fact, work out for users who want to do
nothing more than store some text in a file. Unfortunately, however,
programmers building sophisticated file structures must sometimes spend a
lot of time finding ways to disable this automatic assistance so they can have
complete control over what they are building. Forewarned is forearmed;
readers who encounter these kinds of difficulties as they build the file
structures described in this text can take some comfort from the knowledge
that the experience they gain in disabling automatic assistance will serve
them
2.7
well, over
and over,
in the future.
The UNIX Directory Structure
No matter what computer system you have,
even if it is a small PC, chances
hundreds or even thousands of files you have access to. To
provide convenient access to such large numbers of files, your computer has
are there are
some method for organizing its
The UNIX filesystem is a
with the
root
of the
including the root,
programs and
data,
UNIX
this is called the filesy stem.
by the character 7\ All
can contain two kinds of
and directories
references to devices, as
stored in a
In
tree-structured organization of
tree signified
drives are also treated like
name
files.
UNIX
files
shown
in
(Fig. 2.4).
UNIX,
files:
regular
directories,
directories,
files
with
Since devices such as tape
directories can also contain
in the dev directory in Fig. 2.4.
directory corresponds to
what we
call its
The
file
physical
name.
Since every file in a UNIX system is part of the filesystem that begins
with root, any file can be uniquely identified by giving its absolute pathname.
For instance, the true, unambiguous name of the file "addr" in Fig. 2.4 is
23
THE UNIX DIRECTORY STRUCTURE
(root)
adb
cc
console kbd
yacc
libc.a
TAPE
libm.a
FIGURE 2.4 Sample UNIX directory structure.
7 is used both to indicate the root directory
and to separate directory names from the file name.)
When you issue commands to a UNIX system, you do so within some
directory, which is called your current directory. A pathname for a file that
does not begin with a 7 describes the location of a file relative to the
/usr6/mydir/addr. (Note that the
'
'
current directory. Hence, if your current directory in Fig. 2.4
uniquely identifies the
The
file
2.8
mydir, addr
special filename "." stands for the current directory,
stands for the parent of the current directory.
directory
is
/usr6/ mydir /addr.
is
/usr6/mydir/DF, "../addr" refers to the
Physical and Logical Files
in
Hence,
file
if
and ".."
your current
/usr6/mydir /addr.
UNIX
2.8.1 Physical Devices as UNIX Files
One of the most powerful ideas in UNIX is reflected in its notion of what
a file is. In UNIX, a file is a sequence of bytes, without any implication of
how or where the bytes are stored or where they originate. This simple
24
FUNDAMENTAL
PROCESSING OPERATIONS
FILE
conceptual view of
a file
makes
it
operations what might require
possible in
many
different operating system. For example,
disk as the source of a
on
things
also files
But
disks.
in
because
file,
UNIX,
in Fig. 2.4, /dev/kbd
pressed;
the
console
accepts
corresponding symbols on
allows so
very few
easy to think of a magnetic
are used to the idea of storing such
devices like the keyboard and the console are
and /dev / console respectively. The keyboard
,
sequence of bytes
when keys are
and displays their
a screen.
we say that the UNIX concept of a file is simple when it
many different physical things to be called files? Doesn't this make
more complicated, not
matter what physical representation
UNIX
operations on a
can
the situation
file is
logically
it is
do with
to
many
sequence of bytes that are sent to the computer
produces
How
we
UNIX
times as
the same. In
by an integer
its
the
simpler?
a file
The
may
simplest form, a
file
UNIX
trick in
take, the logical
UNIX
file is
descriptor. This integer
is that no
view of a
represented
an index to an
is
more complete information about the file. A keyboard, a disk file,
magnetic tape are all represented by integers. Once the integer that
array of
and
describes a
logical
file is
name of
whether the
file
program can access that file. If it knows the
program can access that file without knowing
comes from a disk, a tape, or a telephone.
identified, a
a file,
2.8.2 The Console, the Keyboard, and Standard Error
We
see an
program
example of the duality between devices and
/*
/*
/*
Step
Step
Step
1)
while (readCfd, &c
writeCSTDOUT, &c
1);
/*
/*
Step 4a */
Step 4b */
>
0)
We
LIST
the
printf ("Enter the name of the file: ");
getsCf ilename)
fd =open(f ilename, 0_RD0NLY);
;
The
files in
in Fig. 2.2:
logical
file
is
some
STDOUT,
small integer value returned by the open(
assign this integer to the variable^ in Step
integer
defined as
earlier in the
3.
In Step 4b,
program,
we
*/
*/
*/
) call.
use the
to identify the
file to be written to.
There are two other file descriptors that are special in UNIX: The
keyboard is called STDIN (standard input) and the error file is called
STDERR (standard error). Hence, STDIN is the keyboard on your terminal.
console as the
The statement
readCSTDIN,
k,
1);
PHYSICAL AND LOGICAL FILES
reads a single character
like
STDOUT,
is
from your terminal.
usually just your console.
STDERR
25
UNIX
an error
is
When your
IN
file
which,
compiler detects an
error, it generally writes the error message to this file, which means
normally that the error message turns up on your screen. As with STDIN,
the values STDIX and STDERR are usually defined in stdio.h.
Steps 1 and 2 of the LIST program also involve reading and writing
from
STDIX
STDOUT. Since an enormous amount of I/O involves
most programming languages have special functions to
or
these devices,
perform console input and output
in LIST, the C functions print/ and gets
are used. Ultimately, however, print/ and gets send their output through
STDOUT and STDIN, respectively. But these statements hide important
elements of the I/O process. For our purposes, the second set of read and
write statements is more interesting and instructive.
2.8.3
I/O Redirection
Suppose you would
to a regular
file,
the output of
and Pipes
change the LIST program so it writes its output
STDOUT. Or suppose you wanted to use
input to another program. Because it is common to
like to
rather than to
LIST
as
do both of these, UNIX provides convenient shortcuts
switching between standard I/O (STDIX and STDOUT) and regular
I/O. These shortcuts are called I/O redirection and pipes/
I/O redirection lets you specify at xecuj|op time alternate files
input or output. The notations for input and output redirection are
want
to
<
>
For example,
if
output from
STDOUT to
>
What
if,
you wanted
UNIX
pipes
for
(redirect STDIN to "file")
(redirect STDDUT to "file")
file
file
list
for
file
the executable
myf
i 1
LIST program
a file called
let
programl
it
immediately
you do
I
called "list,"
we
redirect the
instead of storing the output
to use
is
"myfile" by entering the line
this.
in
from the
list
program
in a
file,
another program to sort the results?
The notation
for a
UNIX
pipe
is '|\
Hence,
program2
Stnctly speaking, I/O redirection and pipes are part of a UNIX shell, which is the cominterpreter that sits on top of the core UNIX operating system, the kernel. For the
purpose of this discussion, this distinction is not important.
mand
26
FUNDAMENTAL
FILE
PROCESSING OPERATIONS
means take any S TDO UT output from program 1 and use it in place of any
STDIN input to program2. Since UNIX has a special program called sort,
which takes its input from STDIN, you can sort the output from the list
program, without using an intermediate file, by entering
list
Since
sort
writes
sort
its
output to
STDOUT,
the sorted listing appears
on your
terminal screen unless you use additional pipes or redirection to send
it
elsewhere.
2.9
File-related
UNIX,
Header
Files
file
operations. For example,
return a special value indicating end-of-file
beyond
names and values that you
some C functions
(EOF) when you try to read
like all operating systems, has special
must use when performing
the end of a
file.
you use in an open( ) call to indicate whether you
want read-only, write-only, or read/write access. Unless we know just
where to look, it is often not easy to find where these values are defined.
UNIX handles the problem by putting such definitions in special header
files such as /usr/include, which can be found in special directories.
Recall the flags that
Three header files relevant to the material in this chapter are stdio.h,
and file.h. EOF, for instance, is defined on many UNIX systems in
/usr/include /stdio.h, as are the file pointers STDIN, STDOUT, and
STDERR. And the flags 0_RDONLY, 0_WRONLY, and O.RDWR
can usually be found in /usr/ include /sys/file.h or possibly one of the files that
fcntl.h,
it
includes.
It
would be
instructive for
you
to
browse through these
files,
as well as
others that pique your curiosity.
2.10
UNIX Filesystem Commands
UNIX provides many commands for
manipulating
are relevant to the material in this chapter.
files.
We
list a
few
options, but the simplest uses of most should be obvious. Consult a
manual
for
cat
tail
more information on how
filenames
filename
that
Most of them have many
to use them.
Print the contents of the
named
Print the last 10 lines of
the text
text
file.
files,
UNIX
SUMMARY
cp
filel ftle2
mv
filel file2
rm
filenames
chmod
Copy filel to file2.
Move (rename) filel to file2.
Remove (delete) the named files.
Change the protection mode on the named
mode filename
files.
Is
List the contents
mkdir
rmdir
name
Create
name
Remove
of the directory.
a directory
the
with the given name.
named
directory.
SUMMARY
This
chapter
OPEN(
introduces
CREATE(
),
fundamental operations of
the
CLOSE(
),
),
READ(
WRITE(
),
),
systems:
file
and SEEK( ).
link between a
Each of these operations involves the creation or use of a
on a secondary device and a logical file that represents a
program's more abstract view of the same file. When the program describes
physical file stored
an operation using the
gets
performed on the corresponding physical
The
six operations
Sometimes they are
functions, and sometimes they
),
for instance,
Before
we
can use
many
different
commands, sometimes they
built-in
are direct calls to an operating system.
languages provide the user with
SEEK(
file.
appear in programming languages in
forms.
all
name, the equivalent physical operation
logical file
all
six
are
Not
The operation
operations.
not available in standard Pascal.
is
a physical file,
we must
link
it
to a logical
file.
In
some programming environments we do this with a statement (e.g., assign
Turbo Pascal) or with instructions outside of the program (e.g., job
control language QCL] instructions). In other languages the link between
the physical file and a logical file is made with OPEN(
or CREATE( ).
The operations CREATE( and OPEN( make files ready for reading
in
or writing.
operates
CREATE(
on an already
causes a
pointer to the beginning of the
between
a logical file
that the file buffer
to the
is
new
physical
existing physical
and
its
file.
file,
file
to be created.
OPEN(
usually setting the read/write
The CLOSE(
operation breaks the link
corresponding physical
file. It
also
flushed so everything that was written
is
makes sure
actually sent
file.
The I/O operations READ(
systems
level, require three
The
logical
and
WRITE(
),
when viewed
at a
items of information:
name of the
file
to be read
from or written
to;
low,
27
28
FUNDAMENTAL
An
address
of
computer"
An
PROCESSING OPERATIONS
FILE
memory
area to be used for the "inside of the
part of the exchange;
how much
indication of
data
is
and
to be read or written.
These three fundamental elements of the exchange are illustrated in Fig. 2.5.
READ( ) and WRITE( ) are sufficient for moving sequentially through
a file to any desired position, but this form of access is often very inefficient.
Some languages provide
a certain
position in a
operation.
The
giving us
lseek(
seek operations that
file.
)
operation
great deal of
let a
program move
directly to
provides direct access by means of the lseek(
lets
freedom
us view a
file as a
in deciding
Standard Pascal does not support direct
how
to organize a
access, but
file
kind of large array,
many
file.
dialects
of
Pascal do.
One
other useful
file
operation involves
has been reached. End-of-file detection
is
knowing when
handled
the end of a
in different
file
ways by
different languages.
Much
effort goes into shielding
the physical characteristics of
programmers from having
files,
about the physical organization of
When we
but inevitably there are
files
that
to deal
little
with
details
programmers must know.
have our program operate on files at a very low level (as we
do a great deal in this text), we must be on the lookout for little surprises
inserted in our file by the operating system or applications.
The
try to
UNIX
file
system, called the filesystem, organizes
files in a tree
and subdirectories expressable by their pathnames. It
is possible to navigate around the filesystem as you work with UNIX files.
UNIX views both physical devices and traditional disk files as files, so,
for example, a keyboard (STDIN), a console (STDOUT), and a tape drive
all are considered files.
This simple conceptual view of files makes it
possible in UNIX to do with a very few operations what might require
many times the operations on a different operating system.
I/O redirection and pipes are convenient shortcuts provided in UNIX for
transferring file data between files and standard I/O. Header files in UNIX,
such at stdio.h, contain special names and values that you must use when
structure,
with
all files
FIGURE 2.5 The exchange between memory and external device.
identified
Amount of data
to transfer
KEY TERMS
performing file operations. It is important to be aware of the most common
of these in use on your system.
The following section lists a sampling of UNIX commands for
manipulating files.
KEY TERMS
Access mode. Type of
access allowed.
file
The
modes
variety of access
permitted varies from operating system to operating system.
When
Buffering.
input or output
destination immediately,
we
we
is
saved up rather than sent off to
say that
it is
its
buffered. In later chapters,
we
can dramatically improve the performance of proand write data if we buffer the I/O.
Byte offset. The distance, measured in bytes, from the beginning of the
file. The very first byte in the file has an offset of 0, the second byte
has an offset of 1, and so on.
CLOSE( ). A function or system call that breaks the link between a logical file name and the corresponding physical file name.
CREATE( ). A function or system call that causes a file to be created
on secondary storage and may also bind a logical name to the file's
find that
grams
that read
physical
name
see
OPEN(
).
CREATE(
call to
the generation of information used
by
the system to
also results in
manage
the
file,
such as time of creation, physical location, and access privileges for
anticipated users of the
End-of-file (EOF).
An
file.
indicator within a
has occurred, a function that
countered
tells if
(e.g., eof( ) in Pascal),
file
that the
the end of a
file
end of the file
has been en-
or a system-specific value that
returned by file-processing functions indicating that the end of
is
a file
has been encountered in the process of carrying out the function
(e.g.,
EOF
UNIX
UNIX).
in
File descriptor.
open(
A
)
small, non-negative integer value returned
or creat(
) call
that
is
used
as a logical
name
by
for the
UNIX
file
system calls.
Filesystem. The name used in UNIX to describe a collection of files
and directories organized into a tree-structured hierarchy.
Header file. A file in a UNIX environment that contains definitions and
declarations commonly shared among many other files and applications. In C, header files are included in other files by means of the
in later
"#include" statement
(see Fig. 2.2).
The header
files stdio.h, file.h,
29
30
FUNDAMENTAL
PROCESSING OPERATIONS
FILE
and fcntl.h described
and definitions used
in this chapter contain
important declarations
in file processing.
I/O redirection. The redirection of a stream of input or output from its
normal place. For instance, the operator '>' can be used to redirect
to a file output that would normally be sent to the console.
Logical file. The file as seen by the program. The use of logical files
allows a program to describe operations to be performed on a file
without knowing what actual physical file will be used. The program may then be used to process any one of a number of different
files that share the same structure.
OPEN( ). A function or system call that makes a file ready for use. It
may
bind
also
file
include information
Pathname.
name to a physical file. Its arguments inname and the physical file name and may also
a logical file
clude the logical
on how
the
file is
expected to be accessed.
character string that describes the location of a
rectory. If the
pathname
starts
with
a 7',
then
it
file
or di-
gives the absolute
the complete path from the root directory to the file.
Otherwise it gives the relative pathname the path relative to the current working directory.
Physical file. A file that actually exists on secondary storage. It is the
file as known by the computer operating system and that appears in
pathname
its file
directory.
A UNIX
operator specified by the symbol '|' that carries data
from one process to another. The originating process specifies that
the data is to go to STDOUT, and the receiving process expects the
data from STDIN. For example, to send the standard output from a
program makedata to the standard input of a program called usedata,
use the command "makedata usedata".
Protection mode. An indication of how a file can be accessed by vari-
Pipe.
ous classes of users. In
octal
number
UNIX, the protection mode is a three-digit
how the file can be read, written to, and
that indicates
executed by the owner, by members of the owner's group, and by
everyone
READ(
).
device.
(1) a
else.
function or system
When viewed
at
Source_file logical
call
used to obtain input from a file or
it requires three arguments:
the lowest level,
name corresponding
to an
open
file; (2)
Destination_address for the bytes that are to be read; and
or
amount of data
SEEK(
).
to be read.
function or system
specified position in the
tions allow
(3)
the
the Size
programs
file.
call that sets
Languages
the read/write pointer to a
that provide seeking func-
to access specific elements of a
rather than having to read through a
file
from
file directly,
the beginning (sequen-
EXERCISES
each time a specific item
is desired. In C, the lseek( ) system
provides this capability. Standard Pascal does not have a seeking
capability, but many nonstandard dialects of Pascal do.
tially)
call
Standard I/O. The source and destination conventionally used
and output.
By
ways
for input
there are three types of standard I/O: standard
(STDOUT),
standard output
default
STDERR
STDIN is
and
the keyboard, and
STDERR
(standard
STDOUT and
I/O redirection and pipes provide
are the console screen.
to override these defaults.
WRITE(
(1) a
UNIX,
(STDIN),
input
error).
ties.
In
).
function or system
When viewed
Destination_file
call
lowest
at the
used to provide output capabili-
level,
it
requires three arguments:
name corresponding
to an
open
file; (2)
Source_address of the bytes that are to be written; and
amount of the data to be written.
(3)
the
the Size or
EXERCISES
1. Look up operations equivalent to OPEN( ), CEOSE( ), CREATE( ),
READ( ), WRITE( ), and SEEK in other high-level languages, such as
PL/I, COBOL, and Fortran. Compare them with the C or Pascal versions.
2.
If
a)
you use C:
Make
of the different ways to perform the file operations
), CLOSE( ), READ( ), and WRITE( ). Why
there more than one way to do each operation?
a list
CREATE(
is
b)
How
OPEN(
),
would you use
Show how
lseek(
to find the current position in a file?
change the permissions on a file myfile so the owner
has read and write permissions, group members have execute permission, and others have no permission.
d) What is the difference between pmode and 0_RDWR? What
pmodes and 0_RDWR are available on your system?
e) In some typical C environments, such as UNIX and MS-DOS, all
of the following represent ways to move data from one place to anc)
to
other:
scanfC
fscanf(
getc( )
)
)
Describe as
useful.
fgetcC
gets(
fgetsC
<
)
to the
cat (or type)
main (argc argv)
7
many of these
Which belong
system?
read(
as
you
can,
and indicate
how
they might be
language, and which belong to the operating
31
32
3.
FUNDAMENTAL
PROCESSING OPERATIONS
FILE
you use Pascal:
What ways are provided
If
a)
file
operations
WRITE(
tell
)? If
why.
If
CREATE(
there
in
your version of Pascal
OPEN(
),
),
CLOSE(
more than one way
is
an operation
is
missing,
how
to
do
are
its
),
to
perform the
), and
READ(
a certain
operation,
functions carried
out?
Implement
b)
SEEK(
function in your Pascal,
if
it
does not
al-
ready have one.
4.
couple of years ago
One
new compiler
compiler.
that the
difference
a company we know of bought a new COBOL
between the new compiler and the old one was
did not automatically close
program terminated, whereas the old compiler
did this cause
when execution of a
What sorts of problems
files
did.
when some of the old software was
new compiler?
executed after having
been recompiled with the
5.
Look
the
at
two LIST programs
Pascal, the sequence
test,
Why
write.
of steps
in the
in the text.
loop
is test,
Each has
a while loop. In
read, write. In C,
it is
loop construction used for
read,
What would happen in Pascal if we used
C? What would happen in C if we used
the difference?
the
the
Pascal loop construction?
6.
In Fig. 2.4:
Give the full pathname for a file in directory DF.
Suppose your current directory is bin. Show how to copy the file
libdf.a to the directory DF without changing your current directory.
a.
b.
7.
What
is
direct error
8.
the difference
Look up
the
9.
WC
Find
STDOUT and STDERR? Find how to
compilation on your system to
UNIX command
environment, and explain
1
between
messages from
why
it
STDERR.
we. Execute the following in a
gives the
number of files
UNIX
in the directory.
on your system, and find what value is used to indicate
Also examine fde.h or jend. h and describe in general what its
stdio.h
end-of-file.
contents are for.
Programming Exercises
10. Make the LIST program we provide in
compiler on your operating system.
11.
Write
program
to
program
open the
file and store
and read the string.
to create a
file
this
chapter
a string in
work with your
it.
Write another
FURTHER READINGS
12.
Try
file
with an access
setting the protection
mode on
a file to read-only,
mode of read/write. What
13. Implement the UNIX command
from the end of the file to be copied
tail -n,
to
then opening the
happens?
where
is
the
number of lines
STDOUT.
Change
the program LIST so it reads from the STDIN, rather than a
and writes to a file, rather than the STDOUT. Show how to execute
the new version of the program in a UNIX environment, given that the
input is actually in a file called instuff. (You can also do this in most
14.
file,
MS-DOS
environments.)
a program to read a series of names, one per line, from standard
and write out those names spelled in reverse order to standard
output. Use I/O redirection and pipes to do the following:
a. Input a series of names that are typed in from the keyboard and
15.
Write
input,
them out, reversed, to a file called filel.
Read the names in from filel; then write them
write
b.
out, re-reversed, to
a file called file2.
Read the names
c.
the resulting
list
from jile2, reverse them
in
of reversed words using
again,
and then sort
sort.
FURTHER READINGS
Introductory textbooks on
only briefly,
if at all.
C and Pascal tend to treat the fundamental file operations
This
is
particularly true with regard to C, since there are
higher-level standard I/O functions in C, such as the read operations fgets(
) and
do provide treatment of the
fundamental file operations are Bourne (1984), Kernighan and Pike (1984), and
Kernighan and Ritchie (1978, 1988). These books also provide discussions of
higher-level I/O functions that we omitted from our text.
As for UNIX specifically, as of this writing there are two dominant flavors of
UNIX: UNIX System V from AT&T, the originators of UNIX, and 4.3BSD
(Berkeley Software Distribution) UNIX from the University of California at
Berkeley. The two versions are close enough that learning about either will give you
fgetc(
).
Some books on C and/or
UNIX
that
good understanding of UNIX generally. However, as you begin to use UNIX,
you will need reference material on the specific version that you are using. There are
many accessible texts on both versions, including Morgan and McGilton (1987) on
System V, and Wang (1988) on 4.3BSD. Less readable but absolutely essential to a
serious UNIX user is the 4.3BSD UNIX Programmers Reference Manual (U.C.
a
Berkeley, 1986) or the System
V Interface
Definition
(AT&T,
1986).
For Pascal, these operations vary so greatly from one implementation to
another that it is probably best to consult user's manuals and literature relating to
33
34
FUNDAMENTAL
your
as
specific
some
FILE
PROCESSING OPERATIONS
implementation. Cooper (1983) covers the ISO standard Pascal,
extensions. Jensen and Wirth (1974)
others arc based. Wirth (1975) discusses
file
operations in the section,
Problems: Files."
is
some
as well
on which all
with standard Pascal and
the definition of Pascal
difficulties
"An Important Concept and
a Persistent
Source of
Secondary Storage and
System Software
CHAPTER OBJECTIVES
Describe the organization of typical disk drives, including basic units of organization and their relationships.
and describe the factors affecting disk access
methods for estimating access
times and space requirements.
Identify
time, and describe
Describe magnetic tapes, identify some tape applications, and investigate the implications of block size
on space requirements and transmission speeds.
Identify fundamental differences
criteria that
between media and
can be used to match the right
medium
to an application.
Describe in general terms the events that occur when
data is transmitted between a program and a secondary storage device.
Introduce concepts and techniques of buffer management.
Illustrate
many of the
concepts introduced in the
chapter, especially system software concepts, in the
context of
UNIX.
35
CHAPTER OUTLINE
3.1
Disks
A Journey
3.5
3.1.1 The Organization of Disks
3.1.2 Estimating Capacities and Space
3.5.1
3.5.2
Needs
3.5.3
by Sector
Organizing Tracks by Block
Nondata Overhead
The Cost of a Disk Access
Effect of Block Size on
Performance: A UNIX Example
The File Manager
The I/O Buffer
The Byte Leaves RAM: The
3.1.3 Organizing Tracks
I/O Processor and Disk
3.1.4
Controller
3.1.5
3.1.6
3.1.7
Buffer
3.6
I/O in
3.7
Magnetic Tape
3.2.1
Organization of Data on Tapes
Tape Length
3.2.2 Estimating
The Kernel
3.7.
Linking
File
3.7.
Normal
Files, Special Files,
3.7.
3.7.
3.7.
3.4
Storage as a Hierarchy
Good
design
is
Names
always responsive to the constraints of the
the environment. This
is
to Files
and
Block I/O
Device Drivers
The Kernel and File Systems
Magnetic Tape and UNIX
3.7.
Times
3.2.4 Tape Applications
Disk versus Tape
UNIX
3.7.
Sockets
Requirements
3.2.3 Estimating Data Transmission
3.3
Management
Buffer Bottlenecks
3.6.2 Buffering Strategies
3.6.1
3.1.8 Disk as Bottleneck
3.2
of a Byte
as true for file structure
design as
medium and
it is
to
for designs
wood and
stone. Given the ability to create, open, and close files, and to
and write, we can perform the fundamental operations of file
construction. Now we need to look at the nature and limitations of the
devices and systems used to store and retrieve files, preparing ourselves for
in
seek, read,
file
design.
If files
were stored just
called file structures.
the tools
we would
The
in
RAM,
need to build
devices are very different
from
file
RAM. An
is
applications.
RAM. One
that accesses to secondary storage take
impact,
would be no
there
separate discipline
general study of data structures
would give
us
all
But secondary storage
difference, as already noted,
much more time
is
than do accesses to
even more important difference, measured in terms of design
that not
all
accesses are equal.
Good
knowledge of disk and tape performance
minimize access
costs.
file
structure design uses
to arrange data in
ways
that
37
DISKS
we examine the
on the constraints
In this chapter
devices, focusing
We
of secondary storage
that shape our design work in the
characteristics
a look at the major media used in the
magnetic disks, and tapes. We follow this
with an overview of the range of other devices and media used for
secondary storage. Next, by following the journey of a byte, we take a brief
chapters that follow.
begin with
storage and processing of
look
at
when
look
3.1
at
many
files,
hardware and software that become involved
by a program to a file on a disk. Finally, we take a closer
one of the most important aspects of file management
buffering.
the
byte
is
pieces of
sent
Disks
Compared to the time it takes to access an item in RAM, disk accesses are
always expensive. However, not all disk accesses are equally expensive. The
reason for this has to do with the way a disk drive works. Disk drives"
"
belong to
a class
because they
with
of devices
make
serial devices,
it
known
as direct access storage devices
possible to access data
directly.
DASDs
(DASDs)
are contrasted
the other major class of secondary storage devices. Serial
devices use media such as magnetic tape that permit only serial access
particular data item cannot be read or written until
it
on the tape have been read or written
Magnetic disks come
in
many
all
of the data preceding
in order.
forms. So-called hard disks offer high
low cost per bit. Hard disks are the most common disk used
everyday file processing. Floppy disks are inexpensive, but they are slow
and hold relatively little data. Floppies are good for backing up individual
and for transporting small amounts of data.
files or other floppies
Removable disk packs are hard disks that can be mounted on the same drive
at different times, providing a convenient form of backup storage that also
capacity and
in
makes it possible to access data
Nonmagnetic disk media,
directly.
becoming
Appendix A for a
especially optical discs, are
creasingly important for secondary storage. (See
treatment of optical disc storage and
its
infull
applications.)
3.1.1 The Organization of Disks
The information
stored on a disk
is
stored on the surface of one or
more
platters (Fig. 3.1). The arrangement is such that the information is stored in
successive tracks on the surface of the disk (Fig. 3.2). Each track is often
""When
we
use the terms disks or disk
drives,
we
are referring to magnetic disk media.
38
SECONDARY STORAGE AND SYSTEM SOFTWARE
2>
2>
2>
2>
FIGURE 3.1 Schematic
illustration of disk drive.
divided into
of a
disk.
file,
the
Boom
Read/write heads
Spindle
Platters
number of sectors.
When
READ(
sector
statement
computer operating system
is
the smallest addressable portion
byte from
disk
finds the correct surface, track,
and
and
calls for a particular
sector, reads the entire sector into a special area in
RAM called a buffer,
then finds the requested byte within that buffer.
number of platters, it may be called a disk pack. The
above and below one another form a cylinder (Fig.
3.3). The significance of the cylinder is that all of the information on a single
cylinder can be accessed without moving the arm that holds the read/write
If a disk drive uses a
tracks that are directly
heads.
Moving
arm
called seeking. This
arm movement
is
usually the
Disks range in width from 2 to about 14 inches. They range
in storage
this
is
slowest part of reading information from
a disk.
3.1.2 Estimating Capacities and Space Needs
capacity
from
less
pack, the top and
than 400,000 bytes to billions of bytes. In
bottom
platter each contribute
one
surface,
a typical disk
and
all
other
Tracks
Sectors
Gaps
FIGURE 3.2 Surface of disk showing tracks and sectors.
FIGURE 3.3 Schematic illustration of disk drive viewed as a set
of
seven cylinders.
Seven
cylinders
39
40
SECONDARY STORAGE AND SYSTEM SOFTWARE
platters contribute
cylinder
is
two
surfaces to the pack, so the
The amount of data
number of
tracks per
number of platters.
function of the
that can be held
on
track depends
on how densely
on the quality
can be stored on
of the recording medium and the size of the read/ write heads.) An
inexpensive, low-density disk can hold about 4 kilobytes on a track, and 35
the disk surface. (This in turn depends
bits
on a surface. A top-of-the-linc disk can hold about 50 kilobytes on a
and more than 1,000 tracks on a surface. Table D.l in Appendix D
shows how a variety of disk drives compare in terms of capacity, performance, and cost.
Since a cylinder consists of a group of tracks, a track consists of a group
of sectors, and a sector consists of a group of bytes, it is easy to compute
track, cylinder, and drive capacities:
tracks
track,
Track capacity = number of sectors per track X bytes per sector
Cylinder capacity
= number of tracks
per cylinder X track capacity
Drive capacity = number of cylinders x cylinder capacity.
If
to
we know
compute
the number of bytes in a file, we can use these relationships
amount of disk space the file is likely to require. Suppose,
that we want to store a file with 20,000 fixed-length data
the
for instance,
records
on
"typical"
300-megabyte small computer disk with the
following characteristics:
How many
of bytes per sector
Number
o{ sectors per track
Number
of tracks per cylinder
Number
of cylinders = 1,331.
cylinders does the
file
20,000
- =
40
=11
require if each data record requires 256
two
bytes? Since each sector can hold
One
= 512
Number
records, the
1AAnn
10,000
sectors.
= 440
sectors
file
requires
cylinder can hold
40 x
so the
number of cylinders
1 1
required
10,000
is
approximately
22.7 cylinders.
440
Of course,
it
may
does not have
be that
a disk drive
with 22.7 cylinders of available space
22.7 physically contiguous cylinders available. In this likely
41
DISKS
case, the file
might
in fact
have to be spread out over dozens, perhaps even
hundreds, of cylinders.
3.1.3 Organizing Tracks by Sector
There are two basic ways to organize data on a disk: by sector and by
user-defined block. So far, we have only mentioned sector organizations. In
this section we examine sector organizations more closely. In the following
section
we
look
at
block organizations.
The Physical Placement of
Sectors
There are several views
can have of the organization of sectors on
a track.
The
that
one
simplest view, one
most users most of the time, is that sectors are adjacent,
segments
of a track that happen to hold a file (Fig. 3.4a). This is
fixed-sized
often a perfectly adequate way to view a file logically, but it may not be a
that suffices for
good way
to store sectors physically.
When you want
one right
to read a series of sectors that are
after the other,
you often cannot read
all
FIGURE 3.4 Two views of the organization of sectors on a 32-sector track.
15\ 16 117/18X19
72
32 \ 31
(a)
23 \ 4
27/14/
17/30/11
i20\7
(b)
in the
same
adjacent sectors.
track,
That
is
42
SECONDARY STORAGE AND SYSTEM SOFTWARE
because, after reading the data,
it
takes the disk controller a certain
of time to process the received information before
So, if logically adjacent sectors
it is
amount
ready to accept more.
were placed on the disk so they were
also
we would miss the start of the following sector while we
the one we had just read in. Consequently, we would be
physically adjacent,
were processing
one sector per revolution of the disk.
I/O system designers usually approach this problem by interleaving the
sectors, leaving an interval of several physical sectors between logically
adjacent sectors. Suppose our disk had an interleaving factor of 5. The
assignment of logical sector content to the 32 physical sectors in a track is
illustrated in Fig. 3.4(b). If you study this figure, you can see that it takes
five revolutions to read the entire 32 sectors of a track. That is a big
able to read only
improvement over 32 revolutions.
Over the last year or two,
high-performance disks can
now
controller
offer
speeds
successive sectors actually are physically adjacent,
an entire track in
Clusters
performance,
the
is
sectors.
cluster has
cluster
map
It
is
the logical parts of the
does
this
a fixed
been found on
by viewing the
file
manager
that
possible to read
file
to their corresponding
file as a series
number of contiguous
sectors.
a disk, all sectors in that cluster
without requiring an additional seek.
To view a file as a series of clusters and
the
it
view of sector organization, also designed to improve
view maintained by that part of a computer's operating
the file manager. When a program accesses a file, it is the
system that we call
file manager's job to
physical locations.
making
means
revolution of the disk.
a single
third
have improved so
interleaving. This
ties logical
still
"*"
of
clusters
Once
of
given
can be accessed
maintain the sectored view,
sectors to the physical clusters that they belong
(FAT). The FAT contains a list of all the
ordered according to the logical order of the sectors they
contain. With each cluster entry in the FAT is an entry giving the physical
to
by using
clusters in a
a file allocation table
file,
location of the cluster (Fig. 3.5).
On many
systems, the system administrator can decide
how many
sectors there should be in a cluster. For instance, in the standard physical
disk structure used
by
cluster size to be used
VAX
on
systems, the system administrator sets the
a disk
when
the disk
is
initialized.
The
default
is three 512-byte sectors per cluster, but the cluster size may be set to
any value between 1 and 65,535 sectors. Since clusters represent physically
contiguous groups of sectors, larger clusters guarantee the ability to read
value
""It
is
not always physically contiguous; the degree of physical contiguity
the interleaving factor.
is
determined by
43
DISKS
File allocation table
(FAT)
The
part of the
FAT
pertaining
to
our
Cluster
Cluster
number
location
file
^^^\
r^
^r
-~~\
~t^"
^^?
FIGURE 3.5 The file manager determines which cluster
is to be accessed.
in
the
file
has the sector
that
more
sectors without seeking,
performance gains
substantial
so the use of large clusters can lead to
when
being processed sequentially.
a file is
Extents Our final view of sector organization represents a further attempt
to emphasize physical contiguity of sectors in a file, hence minimizing
seeking even more. (If you are getting the idea that the avoidance of seeking
is an important part of file design, you are right.) If there is a lot of free
room on a disk, it may be possible to make a file consist entirely of
contiguous clusters.
extent: All
When this is
of its sectors,
contiguous whole
tracks,
(Fig. 3.6a).
the case,
and
This
(if it is
is a
we say
of one
large enough) cylinders form one
good
that the
file
consists
situation, especially if the file
be processed sequentially, because it means that the whole
accessed with a minimum amount of seeking.
is
to
file,
can be
If there is
not enough contiguous space available to contain an entire
the
divided into
file is
an extent.
When new
make them
is
file
two or more noncontiguous
clusters are
added to
Each part is
manager tries to
parts.
a file, the file
physically contiguous to the previous end of the
unavailable for
this, it
must add one or more extents
file,
but
if space
(Fig. 3.6b).
The
44
SECONDARY STORAGE AND SYSTEM SOFTWARE
(a)
(b)
FIGURE 3.6
single
File extents
(shaded area represents space on disk used by a
file).
most important thing
extents in a
the
file
to understand about extents
increases, the
file
amount of seeking required
file is
records and sectors. There are
that as the
number of
disk,
and
to process the file increases.
Fragmentation Generally, all sectors on
same number of bytes. If, for example, the
the size of all records in a
is
becomes more spread out on the
given drive must contain the
size
300 bytes, there
two ways
of a sector
is
to deal
is
512 bytes and
no convenient
with
fit
between
this situation: Store
only one record per sector, or allow records to span sectors, so the
beginning of
another (Fig.
The
first
record might be found in one sector and the end of
it
in
3.7).
option has the advantage that any record can be retrieved by
it has the disadvantage that it might leave an
retrieving just one sector, but
enormous amount of unused space within each sector. This loss of space
within a sector is called internal fragmentation. The second option has the
45
DISKS
advantage that
it
loses
no space from
disadvantage that some records
internal fragmentation, but
may
it
has the
be retrieved only by accessing two
sectors.
Another potential source of internal fragmentation
of
clusters. Recall that a cluster
allocated for a
file.
When
the
is
results
from the use
the smallest unit of space that can be
number of
bytes in a
file
is
not an exact
multiple of the cluster
size, there will be internal fragmentation in the last
For instance, if a cluster consists of three 512-byte sectors,
a file containing one byte would use up 1,536 bytes on the disk; 1,535 bytes
would be wasted due to internal fragmentation.
Clearly, there are important trade-offs in the use of large cluster sizes.
extent of the
file.
A disk that is expected to have mainly large files that will often be processed
would usually be given a large cluster size, since internal
fragmentation would not be a big problem and the performance gains
sequentially
might be great. A disk holding smaller files or files that are usually accessed
only randomly would normally be set up with small clusters.
3.1.4 Organizing Tracks by Block
Sometimes disk tracks
numbers of user-defined
are
not
blocks
divided into sectors,
whose
size
but into integral
can vary. (Note:
The word
FIGURE 3.7 Alternate record organization within sectors (shaded areas represent
data records, and unshaded areas represent unused space).
(a)
block
46
SECONDARY STORAGE AND SYSTEM SOFTWARE
Sector
Sector 3
Sector 2
Sector 4
1111111111 1111111111 1111222222
2 2
3 3 3
Sector 5
Sector 6
444 4444444444
4 4 15 5
5.
(a)
11111111111.
.111111111222.
.22',333;444444.
.4444 441555
(b)
FIGURE 3.8 Sector organization versus block organization.
meaning
of the
UNIX
I/O system. See section
3.7 for details.) When the data on a track is organized by block, this usually
means that the amount of data transferred in a single I/O operation can vary
depending on the needs of the software designer, not the hardware. Blocks
can normally be either fixed or variable in length, depending on the
requirements of the file designer. As with sectors, blocks are often referred
to as physical records. (Sometimes the word block is used as a synonym for
a sector or group of sectors. To avoid confusion, we do not use it in that
way here.) Figure 3.8 illustrates the difference between one view of data on
a sectored track and that of a blocked track.
A block organization does not present the sector-spanning and fragmentation problems of sectors because blocks can vary in size to fit the logical
organization of the data. A block is usually organized to hold an integral
number of logical records. The term blocking factor is used to indicate the
number of records that are to be stored in each block in a file. Hence, if we
had a file with 300-byte records, a block-addressing scheme would let us
define a block to be some convenient multiple of 300 bytes, depending on
the needs of the program. No space would be lost to internal fragmentation,
and there would be no need to load two blocks to retrieve one record.
has a different
in the context
Generally speaking, blocks are superior to sectors
when
it is
desirable to
have the physical allocation of space for records correspond to their logical
organization. (There are disk drives that allow both sector-addressing and
block-addressing, but
we do
In block-addressing
by one or more
Typically there
number of
not describe them here. See Bohl, 1981.)
schemes, each block of data
is
usually accompanied
subblocks containing extra information about the data block.
is
a count subblock that
contains
(among other
things) the
bytes in the accompanying data block (Fig. 3.9a). There
also be a key subblock containing the
key
may
for the last record in the data block
DISKS
(Fig. 3.9b).
When
47
key subblocks are used, a track can be searched by the
disk controller for a block or record identified by a given key. This
that a
program can ask
its
disk drive to search
among
all
track for a block with a desired key. This approach can result in
searches
than
means
on a
much more
the blocks
normally
possible with sector-addressable
schemes, in which keys cannot generally be interpreted without first
loading them into primary memory.
efficient
are
3.1.5 Nondata Overhead
Both blocks and sectors require that a certain amount of space be taken up
on the disk in the form of nondata overhead. Some of the overhead consists
of information that is stored on the disk during preformatting, which is done
before the disk can be used.
On sector-addressable disks, preformatting involves storing, at the
beginning of each sector, such information as sector address, track address,
and condition (whether the sector is usable or defective). Preformatting also
placing gaps and synchronization marks between fields of
information to help the read/write mechanism distinguish between them.
This nondata overhead usually is of no concern to the programmer. When
the sector size is given for a certain drive, the programmer can assume that
this is the amount of actual data that can be stored in a sector.
On a block-organized disk, some of the nondata overhead is invisible to
the programmer, but some of it must be accounted for by the programmer.
Since subblocks and interblock gaps have to be provided with every block,
involves
FIGURE 3.9 Block addressing requires that each physical data block be accompanied by one
more subblocks containing information about its contents.
or
III
I
i
(a)
(b)
mSM
48
SECONDARY STORAGE AND SYSTEM SOFTWARE
there
generally
is
more nondata information provided with blocks than
with sectors. Also, since the number and
application to another, the relative
when block
can vary
addressing
is
sizes
of blocks can vary from one
amount of space taken up by overhead
used. This
is
illustrated in the following
example.
Suppose we have a block-addressable disk drive with 20,000 bytes per
and the amount of space taken up by subblocks and interblock gaps
is equivalent to 300 bytes per block. We want to store a file containing
100-byte records on the disk. How many records can be stored per track if
track,
the blocking factor
1.
If there are 10
is
10,
or
if
it is
60?
100-byte records per block, each block holds 1,000
+ 1,000, or 1,300, bytes of track space
bytes of data and uses 300
when overhead
can
on
fit
is
taken into account.
The number of blocks which
20,000-byte track can be expressed
20,000
1,300 J
L15.38J
as
15.
So 15 blocks, or 150 records, can be stored per track. (Note that we
have to take the floor of the result because a block cannot span two
tracks.)
If there are
60 100-byte records per block, each block holds 6,000
The number of
bytes of data and uses 6,300 bytes of track space.
blocks per track can be expressed
20,000
as
3.
6,300
So 3 blocks, or 180 records, can be stored per
track.
Clearly, the larger blocking factor can lead to
storage.
there
is
When
less
more
efficient use
blocks are larger, fewer blocks are required to hold
space
consumed by
the 300 bytes of overhead that
a file,
of
so
accompany
each block.
Can we conclude from this example that larger blocking factors always
more efficient storage utilization? Not necessarily. Since we can put
only an integral number of blocks on a track, and since tracks are fixed in
length, we almost always lose some space at the end of a track. Here we
lead to
have the internal fragmentation problem again, but this time it applies to
fragmentation within a track. The greater the block size, the greater
potential amount of internal track fragmentation. What would have
happened if we had chosen a blocking factor of 98 in the preceding example?
What about 97?
The flexibility introduced by
result
in
the use of blocks, rather than sectors, can
savings in time and efficiency,
since
it
lets
the
programmer
49
DISKS
determine to
disk.
On
a large
extent
how
the negative side,
data are to be organized physically on
blocking schemes
the
require
programmer
and/or operating system to do the extra work of determining the data
organization. Also, the very flexibility introduced by the use of blocking
schemes precludes the synchronization of I/O operations with the physical
movement of the disk, which sectoring permits. This means that strategies
such as sector interleaving cannot be used to improve performance.
3.1.6 The Cost
of a Disk
Access
To give you a feel for the factors contributing to the total amount of time
needed to access a file on a fixed disk, we calculate some access times. A disk
access can be divided into three distinct physical operations, each with its
own cost: seek rime, rotational delay, and transfer tiiiie.
Seek
Time
Seek time
is
the time required to
move
the access
The amount of time spent seeking during a
depends, of course, on how far the arm has to move. If we are
correct cylinder.
sequentially and the
arm
to the
disk access
accessing
packed into several consecutive cylinders,
seeking needs to be done only after all of the tracks on a cylinder have been
processed, and even then the read/write head needs to move the width of
only one track. At the other extreme, if we are alternately accessing sectors
file
from two
files
file is
that are stored at opposite extremes
innermost cylinder,
one
on
outermost cylinder),
the
at
disk (one at the
seeking
is
very
expensive.
Seeking
is
likely to be
more
costly in a multiuser environment,
where
several processes are contending for use of the disk at one time, than in
single-user environment,
where disk usage
is
dedicated to one process.
Since seeking can be very costly, system designers often go to great
extremes to minimize seeking. In an application that merges three files, for
example, it is not unusual to see the three input files stored on three different
drives and the output file stored on a fourth drive, so no seeking need be
done
as
I/O operations jump from
Since
it is
traversed in every seek,
required for
file
usually impossible to
particular
for each access are
third of the total
we
file
random,
to
know
file.
exactly
how many
operation. If the starting and ending positions
it
turns out that the average seek traverses one
number of cylinders
that the read/write
Manufacturers' specifications for disk drives often
"Derivations of
tracks will be
usually try to determine the average seek time
this result, as well as
more
list
head ranges over. 7
this
figure as the
detailed and refined models, can be found in
Wiederhold (1983), Knuth (1973b). Teory and Fry
(1982),
and Salzberg (1988).
50
SECONDARY STORAGE AND SYSTEM SOFTWARE
"
FIGURE 3.10 When a single file can span several tracks on a cylinder, we
can stagger the beginnings of the tracks to avoid rotational delay when
moving from track to track during sequential access.
average seek time for the drives. Most hard disks available today (1991)
have average seek times of less than 40 milliseconds (msec), and highperformance disks have average seek times as low as 10 msec.
Rotational Delay
Rotational delay refers to the time
we want
it
takes for the disk
under the read/write head. Hard disks
is one revolution per 16.7 msec.
On average, the rotational delay is half a revolution, or about 8.3 msec. On
floppy disks, which often rotate at only 360 rpm, average rotational delay
to rotate so the sector
is
usually rotate at about 3,600 rpm,
is
a sluggish
which
83.3 msec.
As in the case of seeking, these averages apply only when the read/ write
head moves from some random place on the disk surface to the target track.
In
many
circumstances, rotational delay can be
For example, suppose that you have
much
less
than the average.
two or more tracks,
on one cylinder, and that you write
a file that requires
of available tracks
the file to disk sequentially, with one write call. When the first track is
filled, the disk can immediately begin writing to the second track, without
any rotational delay. The "beginning" of the second track is effectively
staggered by just the amount of time it takes to switch from the read/write
head on the first track to the read/write head on the second. Rotational
delay, as it were, is virtually nonexistent. Furthermore, when you read the
file back, the position of data on the second track ensures that there is no
that there are plenty
rotational delay in switching
illustrates this
from one
staggered arrangement.
track to
another.
Figure 3.10
51
DISKS
Time
Once the data we want is under the read/write
can be transferred. The transfer time is given by the formula
Transfer
Transfer time
number of bytes transferred
X
number of bytes on a track
head,
it
rotation time.
time for one sector depends on the number
For example, if there are 32 sectors per track, the time
required to transfer one sector would be l/32nd of a revolution, or 0.5
msec.
If a drive is sectored, the transfer
of sectors on
a track.
Some Timing Computations
show how
situations that
times.
time
it
We
compare the time
will
takes to access
we
Let's look at
different types
all
it
of
two
file
different
takes to access a
of the records
file
processing
access can affect access
file in
sequence with the
in the file randomly. In the
former
much of the file as we can whenever we access it. In the
random-access case, we are able to use only one record on each access.
case,
The
use as
basis for
our calculations
is
"typical" 300-megabyte fixed disk
described in Table 3.1. This particular disk
is
typical
of one that might be
used with a workstation in 1991. Although it is typical only of a certain class
of fixed disk, the observations we draw as we perform these calculations are
quite general.
The
disks used with larger,
bigger and faster than
more expensive computers
factors contributing to total access times are essentially the same.
TABLE
Minimum
3.1
Specifications of disk drive used
(track-to--track) seek
time
examples
in
in text
6 msec
msec
Average seek time
18
Rotational delay
8.3
Maximum
16.7 msec/track, or 1,229 bytes/msec
transfer rate
are
but the nature and relative costs of the
this disk,
Bytes per sector
512
Sectors per track
40
Tracks per cylinder
11
msec
Tracks per surface
1,331
Interleave factor
Cluster size
8 sectors
5* cjtu/*ti/**
Smallest extent size
5 clusters
p^-^ "tJsoce^k-
52
SECONDARY STORAGE AND SYSTEM SOFTWARE
Since our drive uses
smallest extent
is
of 8 sectors (4,096 bytes) and the
a cluster size
5 clusters, space
allocated for storing
is
files in
with an interleave factor of
units. Sectors are interleaved
given track can be transferred
1,
one-track
so data on a
at the stated transfer rate.
suppose that we wish to know how long it will take, using this
drive, to read a 2,048-K-byte file that is divided into 8,000 256-byte records.
First we need to know how the file is distributed on the disk. Since the
4,096-byte cluster holds 16 records, the file will be stored as a sequence of
500 4,096-byte clusters. Since the smallest extent size is 5 clusters, the 500
clusters are stored as 100 extents, occupying 100 tracks.
This means that the disk needs 100 tracks to hold the entire 2,048 K
bytes that we want to read. We assume a situation in which the 100 tracks
are randomly dispersed over the surface of the disk. (This is an extreme
situation chosen to dramatize the point we want to make. Still, it is not so
extreme that it could not easily occur on a typical overloaded disk that has
a large number of small files.)
Now we are ready to calculate the time it would take to read the
2,048-K-byte file from the disk. We first estimate the time it takes to read
the file sector by sector in sequence. This process involves the following
Let's
operations for each track:
Average seek
msec
msec
16.7 msec
18
Rotational delay
Read one
8.3
track
Total
We
want
to find
and read 100
Total time
Now
let's
msec.
43
tracks, so the
100 x 43 msec
calculate the time
it
4,300 msec
would
4.3 seconds.
take to read in the
same 8,000
records using random access rather than sequential access. In other words,
one sector right
rather than being able to read
we have
to access the records in
track to track every time
we
after another,
some order
read a
new
that requires
we assume
that
jumping from
sector. This process involves the
following operations for each record:
Average seek
Read one
cluster 11
Total
Total time
msec
msec
3.3 msec
18
Rotational delay
8.3
16.7)
msec
29.6
8,000 x 29.6 msec
236,800 msec
236.8 seconds.
This difference in performance between sequential access and random
access is very important. If we can get to the right location on the disk and
53
DISKS
we are clearly much better off than
we are if we have to jump around, seeking every time we need a new record.
Remember that seek time is very expensive; when we are performing disk
operations we should try to minimize seeking.
read a lot of information sequentially,
3.1.7 Effect
In deciding
of
how
Block Size on Performance: A UNIX Example
best to organize disk storage allocation for several versions
of BSD UNIX, the Computer Systems Research Group (CSRG) in
Berkeley investigated the trade-offs between block size and performance in
a UNIX environment (Leffler et al., 1989). The results of their research
provide an interesting case study involving trade-offs between block size,
fragmentation, and access time.
The
standard
CSRG
at
research indicated that
the time
on
UNIX
minimum
systems,
w as
T
block
size
of 512 bytes,
not very efficient
in a typical
UNIX
environment. Files that were several blocks long often were
scattered over many cylinders, resulting in frequent seeks and thereby
significantly decreasing throughput. The researchers found that doubling
the block size to 1,024 bytes improved performance by more than a factor
of 2. But even with 1,024-byte blocks, they found that throughput was only
about 4% of the theoretical maximum. Eventually, they found that
4,096-byte blocks provided the fastest throughput, but this led to large
amounts of wasted space due to internal fragmentation. These results are
summarized in Table 3.2.
TABLE 3.2
The amount
of
Space
Percent
Used
Waste
wasted space as
a function of block size
Organization
(Mbyte)
775.2
0.0
Data only, no separation between
807.8
4.2
Data only, each
828.7
6.9
Data + inodes, 512-byte block
866.5
11.8
Data + inodes, 1,024-byte
948.5
22.4
1,128.3
45.6
+ modes,
Data + inodes,
Data
The Design and Implementation of the 4.3BSD
UNIX
file starts
files
on 512-byte boundary
2,048-byte
4,096-byte
UNIX file system
block UNIX file system
block UNIX file system
block UNIX file system
Operating System, Loftier et
al.,
p.
198.
54
SECONDARY STORAGE AND SYSTEM SOFTWARE
To
gain the advantages of both the 4,096-byte and the 512-byte
systems, the Berkeley group implemented
a variation of the cluster concept
implementation, they allocate 4,096-byte
that are big enough to need them; but for smaller files, they
(see section 3.1.3).
blocks for
files
In the
new
allow the large blocks to be divided into one or more fragments. With a
fragment size of 512 bytes, as many as eight small files can be stored in one
block, greatly reducing internal fragmentation.
With the 4,096/512 system,
wasted space was found to decline to about 12%.
3.1.8 Disk as Bottleneck
Disk performance is increasing steadily, even dramatically, but disk speeds
lag far behind local network speeds. A high-performance disk drive
with 50 K bytes per track can transmit at a peak rate of about 3 megabytes
per second, and only a fraction of that under normal conditions.
High-performance networks, in contrast, can transmit at rates of as much as
100 megabytes per second. The result can often mean that a process is disk
bound
the network and the CPU have to wait inordinate lengths of time
still
for the disk to transmit data.
A number of techniques are used to solve this problem. One is
multiprogramming, in which the CPU works on other jobs while waiting
for the data to arrive. But if multiprogramming is not available, or if the
process simply cannot, afford to lose so
ways must be found
is
much
time waiting for the disk,
up disk I/O.
One technique that is now offered on many high-performance systems
called striping. Disk striping involves splitting the parts of a file on several
to speed
different drives, then letting the separate drives deliver parts of the
file
to the
network simultaneously.
For example, suppose we have a 10-megabyte file spread across 20
high-performance (3 megabytes per second) drives that hold 50 K per track.
The first drive has the first 50 K of the file, the second drive has the second
50 K, etc., through the twentieth drive. The first drive also holds the
twenty-first 50 K, and so forth until 10 megabytes are stored. Collectively,
the 20 drives can deliver to the network 250 K per revolution, a combined
rate of 60 megabytes per second.
Disk striping exemplifies an important concept that we see more and
more in system configurations parallelism. Whenever there is a bottleneck
at
some point
in the system, consider duplicating the thing that
is
the source
of the bottleneck, and configure the system so several of them operate
in
parallel.
Another approach
the disk at
all.
As
to solving the disk bottleneck
the cost of
RAM steadily decreases,
is
to avoid accessing
more and more
users
55
DISKS
RAM to hold data that a few years ago had to be kept on a disk.
Two effective ways in which RAM can be used to replace secondary storage
are RAM disks and disk caches.
A RAM disk a large part of RAM configured to simulate the behavior
are using
is
of a mechanical disk in every respect except speed and volatility. Since data
can be located in
without a seek or rotational delay,
disks can
provide much faster access than mechanical disks. Since RAM is normally
volatile, the contents of a
disk are lost when the computer is turned
off.
disks are often used in place of floppy disks because they are
much faster than floppies and because relatively little
is needed to
RAM
RAM
RAM
RAM
RAM
simulate a typical floppy disk.
A disk cache^ is a large block of RAM configured to contain pages of data
a disk. A typical disk-caching scheme might use a 256-K cache with
from
When
a disk.
data
is
requested from secondary
looks into the disk cache to see
first
data. If
if it
memory,
the
file
manager
contains the page with the requested
does, the data can be processed immediately. Otherwise, the
it
manager reads the page containing
the data
from
disk, replacing
file
some page
already in the disk cache.
Cache memory can provide
especially
locality.
when
substantial
improvements
program's data access patterns exhibit
Locality exists in a
when
file
in
a
performance,
high degree of
blocks that are accessed in close
temporal sequence are stored close to one another on the disk. When a
disk cache is used, blocks that are close to one another on the disk are much
more likely to belong to the page or pages that are read in with a single
read, diminishing the likelihood that extra reads are
needed for extra ac-
cesses.
RAM
disks and cache memory are examples of
important and frequently used family of I/O techniques.
look
at
buffering,
We
very
take a closer
buffering in section 3.6.
we
examples of the need to
and disk caches, there
make trade-offs in file
is tension between the cost/capacity advantages of disk over RAM, on the
on the other. Striping provides
one hand, and the speed of
opportunities to increase throughput enormously, but at the cost of a more
In these three techniques
see once again
processing. With
RAM disks
RAM
complex and sophisticated disk management system. Good
file
design
balances these tensions and costs creatively.
f The
term
spect to
opposed to disk cache) generally refers to a very high-speed block of
performs the same types of performance-enhancing operations with
that a disk cache does with respect to secondary memory.
cache (as
mary memory
RAM
that
prire-
56
SECONDARY STORAGE AND SYSTEM SOFTWARE
Magnetic Tape
3.2
Magnetic tape units belong
of devices that provide no direct
to a class
accessing facility but that can provide very rapid sequential access to data.
Tapes are compact, stand up well under different environmental conditions,
are easy to store and transport, and are less expensive than disks.
3.2.1 Organization of Data on Tapes
Since tapes are accessed sequentially, there
identify the locations of data
byte within a
start
of the
file
file.
parallel tracks,
on
On
a tape.
corresponds directly to
We may
is
no need
is
sequence of
(see Fig. 3.11), the nine bits that are at
a typical tape as a set
as a
one-bit-wide
of
there are nine tracks
bits. If
corresponding positions in the nine
respective tracks are taken to constitute one byte, plus
can be thought of
of
physical position relative to the
its
envision the surface of
each of which
for addresses to
a tape, the logical position
slice
a.
parity
of tape. Such
bit.
So
byte
a slice is called a
frame.
The
parity bit
is
not part of the data but
is
used to check the validity of
make the number of 1 bits
frame odd. Even parity works similarly but is rarely used with tapes.
Frames (bytes) are grouped into data blocks whose size can vary from
a few bytes to many kilobytes, depending on the needs of the user. Since
tapes are often read one block at a time, and since tapes cannot stop or start
instantaneously, blocks are separated by interblock gaps, which contain no
the data. If odd parity
is
in effect, this bit
is
set to
in the
FIGURE 3.11
Nine-track tape.
Track
Frame
J.
I
I
111
1
Gap
JU
Data block
JU
Gap
57
MAGNETIC TAPE
information and are long enough to permit stopping and starting.
odd
tapes use
parity,
of consecutive
Tape
no
valid frame can contain
frames
is
used to
all
bits,
so
a large
When
number
the interrecord gap.
fill
come in many shapes, sizes, and speeds. Performance
among drives can usually be measured m terms of three
drives
differences
quantities:
commonly 800,
or 6,250
per inch
much
30,000
Tape speed commonly 30
200 inches per second
and
of interblock gap commonly between 0.3 inch and 0.75
Tape density
1,600,
track, but recently as
(bpi) per
bits
bpi;
as
to
(ips);
Size
Note
inch.
that a 6,250-bpi nine-track tape contains 6,250 bits per inch per track,
and 6,250 bytes per inch when the full nine tracks are taken together. Thus,
in the computations that follow, 6,250 bpi is usually taken to mean 6,250
bytes of data per inch.
3.2.2 Estimating Tape Length Requirements
we want
backup copy of a large mailing list file with one
million 100-byte records. If we want to store the file on a 6,250-bpi tape that
has an interblock gap of 0.3 inches, how much tape is needed?
To answer this question we first need to determine what takes up space
on the tape. There are two primary contributors: interblock gaps and data
blocks. For every data block there is an interblock gap. If we let
Suppose
to store a
the physical length of a data block,
g =
n
the length of an interblock gap, and
the
number of data
then the space requirement
for storing the
s
We know
b is
we
blocks,
(b
file is
g).
are. In fact,
not know what b and
depends on our choice of b. Suppose
choose each data block to contain one 100-byte record. Then b, the
that
is
0.3 inch, but
whatever we want
length of each block,
block
=
it
is
to be,
we do
and
;/
//
given by
size (bytes per block)
jr
rr
tape density (bytes per inch)
:
- r~^- =
0.016 inch.
6,250
number of blocks, is one million (one per record).
The number of records stored in a physical block is called the blocking
factor. It has the same meaning that it had when it was applied to the use ol
and
n,
the
58
SECONDARY STORAGE AND SYSTEM SOFTWARE
The blocking factor we have chosen here is 1
because each block has only one record. Hence, the space requirement for
blocks for disk storage.
the
file is
1,000,000 x (0.016
1,000,000 x 0.316 inch
316,000 inches
26,333
0.3) inch
feet.
Magnetic tapes range in length from 300 feet to 3,600 feet, with 2,400
being the most common length. Clearly, we need quite a few
2,400-foot tapes to store the file. Or do we? You may have noticed that our
choice of block size was not a very smart one from the standpoint of space
feet
utilization.
The
up about 19
interblock gaps in the physical representation of the
times as
much
snapshot of our tape,
Gap
Data
Clearly,
Gap
space on the tape
we
we
is
we were
file
take
to take a
like this:
Gap
Data
Data
not used!
should consider increasing the relative amount of space
used for actual data
If
would look something
Data
Most of the
tape.
it
space as the data blocks do. If
if
we want
to try to squeeze the
increase the blocking factor,
we
file
onto one 2,400-foot
can decrease the
number of
which decreases the number of interblock gaps, which in turn
decreases the amount of space consumed by interblock gaps. For example,
if we increase the blocking factor from 1 to 50, the number of blocks
becomes
blocks,
^Q =
20,000,
and the space requirement for interblock gaps decreases from 300,000 inches
to 6,000 inches. The space requirement for the data is of course the same as
it was previously. What has changed is the relative amount of space occupied
by the gaps, as compared to the data. Now a snapshot of the tape would
look much different:
Data
Gap
Data
Gap
Data
Gap
Data
Gap
Data
59
MAGNETIC TAPE
We leave it to
when
you
show
to
that the
blocking factor of 50
When we compute
numbers
file
can
easily
fit
on one 2,400-foot tape
used.
is
the space requirements for our
that are quite specific to our
A more
file.
we produce
file,
general measure of the
of choosing different block sizes is effective recording density. The
is supposed to reflect the amount of actual data
that can be stored per inch of tape. Since this depends exclusively on the
relative sizes of the interblock gap and the data block, it can be defined as
effect
effective recording density
number of bytes
number of inches
When
block
is
per block
required to store
is used in our example, the number of bytes per
and the number of inches required to store a block is 0.316.
100,
100 bytes
is
= 316 4bpi
'
0.316 inches
is
a far cry
at
it,
space utilization
of data blocks and interblock gaps. Let us
amount of
time
it
'
from the nominal recording density of 6,250
way you look
Either
sizes
block'
blocking factor of 1
Hence, the effective recording density
which
bpi.
sensitive to the relative
is
now
see
how
they affect the
takes to transmit tape data.
3.2.3 Estimating Data Transmission Times
If
you understand
the role of interblock gaps and data block sizes in
determining effective recording density, you can probably see immediately
of data transmission. Two other
of data transmission to or from tape are the
nominal recording density and the speed with which the tape passes the
that these
two
factors also affect the rate
factors that affect the rate
read/write head. If we
know
these
two
values,
we
can compute the nominal
data transmission rate:
Nominal
rate
tape density (bpi)
Hence, our 6,250-bpi, 200-ips tape has
6,250 x 200
x tape speed
(ips).
nominal transmission
1,250,000 bytes/sec
1,250 kilobytes/sec.
is competitive with most disk drives.
But what about those interblock gaps? Once our data
rate
of
This rate
interblock gaps, the
example, that
we
gets dispersed
effective transmission rate certainly suffers.
use our blocking factor of
with the same
by
Suppose, for
file
and tape
60
SECONDARY STORAGE AND SYSTEM SOFTWARE
discussed in
the
preceding section (1,000,000 100-byte records,
0.3-inch gap).
We
saw
that the effective recording density for this tape
organization
316.4 bpi.
drive
is
If the tape
effective transmission rate
is
moving
at a rate
of 200
ips,
then
its
is
316.4 x 200
63,280 bytes/sec
63.3 kilobytes/sec,
about one twentieth of the nominal rate!
should be clear that a blocking factor larger than
a rate that is
It
and that
result,
substantially larger blocking
factor
improves on this
improves on it
substantially.
Although there
size
are other factors that can influence performance, block
generally considered to be the one variable with the greatest influence
is
on space
and data transmission rate. The other factors we have
gap size, tape speed, and recording density
are often beyond
the control of the user. Another factor that can sometimes be important is
the time it takes to start and stop the tape. We consider start/stop time in the
exercises at the end of this chapter.
included
utilization
3.2.4 Tape Applications
Magnetic tape
tions
if
the
an appropriate
is
files
medium
for sequential processing applica-
being processed are not likely also to be used
that require direct access. For example, consider the
in applications
problem of updating
monthly periodical. Is it essential that the list be kept
absolutely current, or is a monthly update of the list sufficient?
If information must be up-to-the-minute, then the medium must
permit direct access so individual updates can be made immediately. But if
mailing
for a
list
list needs to be current only when mailing labels are printed, all
of the changes that occur during the course of a month can be collected in
one batch and put into a transaction file that is sorted in the same way that
the mailing list is sorted. Then a program that reads through the two files
simultaneously can be executed, making all the required changes in one pass
the mailing
through the
data.
Since tape
data offline.
megabytes
is
relatively inexpensive,
At current
prices,
costs about 30 times as
it is
an excellent
medium
removable disk pack
much
blocked, can hold the same amount. Tape
as a reel
is
of tape
for storing
that holds
that,
good medium
150
properly
for archival
storage and for transporting data, as long as the data does not have to be
available
on short notice
for direct processing.
of tape drive, a streaming tape drive, is used widely lor
nonstop, high-speed dumping of data to and from disks. Generally less
special kind
61
DISK VERSUS TAPE
expensive than general-purpose tape drives, it is also
processing that involves much starting and stopping.
3.3
less
suited
for
Disk versus Tape
magnetic tape and magnetic disk accounted for the lion's share
of all secondary storage applications. Disk was excellent for random access
and storage of files for which immediate access was desired; tape was ideal
In the past,
for processing data sequentially
and for long-term storage of
somewhat in favor of disk.
tape was preferable to disk
files.
Over
time, these roles have changed
The major reason
processing
that
for sequential
one process, while disk generally
serves several processes. This means that between accesses a disk read/ write
head tends to move away from the location where the next sequential access
that tapes are dedicated to
is
resulting in an expensive seek; while the tape drive, being
will occur,
dedicated to one process, pays no such price in seek time.
This problem of excessive seeking has gradually diminished, and disk
much of the secondary storage niche previously occupied by
has taken over
change is largely due to the continued dramatic decreases in the
and
storage. To fully understand this change, we need to
understand the role of
buffer space in performing I/O."*" Briefly, it is
that performance depends largely on how big a chunk of a file we can
transmit at any time; as more
space becomes available for I/O buffers,
the number of accesses decreases correspondingly, which means that the
number of seeks required goes down as well. Most systems now available,
even small systems, have enough
available to decrease the number of
tape. This
RAM
cost of disk
RAM
RAM
RAM
accesses required to process
most
files
to a level that
makes disk
quite
competitive with tape for sequential processing. This change, added to the
superior versatility and decreasing costs of disks, has resulted in use of disk
for
most
of tape.
This
sequential processing,
is
which
in the past
was primarily the domain
not to say that tapes should not be used for sequential
file is kept on tape, and there are enough drives available to
processing. If a
use
file
them
for sequential processing,
it
may
directly from tape than to stream
be more efficient to process the
it
to
disk and then process
it
sequentially.
Although
tions, tape
Tape
is still
it
has lost ground to disk in sequential processing applica-
remains important as a medium for long-term archival storage.
far less expensive than magnetic disk, and it is very easy and fast
t Techniques for
RAM
buffering are covered in section 3.6.
62
SECONDARY STORAGE AND SYSTEM SOFTWARE
of files between tape and disk. In this context,
one of our most important media (along with
to stream large files or sets
tape has
emerged
CD-ROM)
as
for tertiary storage.
Storage as a Hierarchy
3.4
Although the best mixture of devices
the needs of the system's users,
we
for a
computing system depends on
can imagine any computing system
as a
hierarchy of storage devices of different speed, capacity, and cost. Figure
3.12 summarizes the different types of storage found
at different levels in
FIGURE 3.12 Approximate comparisons of types of storage, circa 1991
Types of
Devices and
Access times
Capacities
Cost
memory
media
(sec)
(bytes)
(cents/bit)
- Primary
Core and
Registers
10
10
10-10 9
10- 1(T 3
RAM
RAM
disk
and
disk cache
Secondary
-_.
Direct-access
Magnetic disks
Serial
Tape and
10" 3 -10
-10 9
10- 2 -l(T o
10-10 n
10" 5 -10" 7
10
mass storage
_- Offline _,
Archival
and
backup
Removable
magnetic
disks,
optical discs,
and tapes
10- 10'
10
-10 12
5
1(T -10
A JOURNEY OF A BYTE
63
Operating system's
User's program:
WRITE
M
(
file
i/o system:
text M ,c,l)
Get one byte from variable c
in user
Write
it
program 's data
area.
to current location
in text file.
User's data area:
c:
FIGURE 3.13 The WRITE( ) statement tells the operating system to send one
character to disk and gives the operating system the location of the character. The operating system takes over the job of doing the actual writing and
then returns control to the calling program.
such hierarchies and shows approximately
and
access time, capacity,
3.5
A Journey
how
they compare in terms of
cost.
of a Byte
a program writes a byte to a file on a disk? We know
what the program does (it says WRITE(.
and we now know
.)),
something about how the byte is stored on a disk, but we haven't looked at
what happens between the program and the disk. The whole story of what
happens to data between program and disk is not one we can tell here, but
we can give you an idea of the many different pieces of hardware and
software involved and the many jobs that have to be done by looking at one
example of a journey of one byte.
Suppose we want to append a byte representing the character 'P' stored
What happens when
in a character variable
somewhere on
a disk.
that the byte will take
WRITECTEXT,
but the journey
is
The WRITE(
to a file
named
in the variable
TEXT
c,
1)
longer than
this
statement results in
simple statement suggests.
a call to the
computer's operating
system, which has the task of seeing that the rest of the journey
successfully (Fig.
stored
the program's point of view, the entire journey
might be represented by the statement
much
)
From
3.13).
is completed
Often our program can provide the operating
64
SECONDARY STORAGE AND SYSTEM SOFTWARE
system with information that helps it carry out this task more effectively,
but once the operating system has taken over, the job of overseeing the rest
of the journey is largely beyond our program's control.
3.5.1 The
An
Manager
File
operating system
is
not
program, but
a single
a collection
of programs,
each one designed to manage a different part of the computer's resources.
Among
these
devices.
We call
programs
are ones that deal with file-related matters
and I/O
of programs the operating system's file manager.
The file manager may be thought of as several layers of procedures (Fig.
3.14), with the upper layers dealing mostly with symbolic, or logical,
aspects of file management, and the lower layers dealing more with the
physical aspects. Each layer calls the one below it, until, at the lowest level,
the byte
The
is
this subset
actually written to the disk.
manager begins by finding out whether the logical characterare consistent with what we are asking it to do with the file.
It may look up the requested file in a table, where it finds out such things
as whether the file has been opened, what type of file the byte is being sent
to (a binary file, a text file, some other organization), who the file's owner
is, and whether WRITE( ) access is allowed for this particular user of the
file
of the
istics
file
file.
The
file
manager must
also
to be deposited. Since the 'P'
needs to
know where
sector in the
file.
is
determine where in the
to be
the end of the
This information
appended
file is
is
to the
file
TEXT the
the
file,
file
'P' is
manager
the physical location of the
obtained from the
file
last
allocation table
(FAT) described earlier. From the FAT, the file manager locates the
and sector where the byte is to be stored.
drive,
cylinder, track,
3.5.2 The
Next, the
I/O Buffer
file
manager determines whether the
already in
'P' is
RAM
sector that
or needs to be loaded into
RAM.
is
to contain the
If the sector
needs
manager must find an available system I/O buffer space
from the disk. Once it has the sector in a buffer in RAM,
to be loaded, the file
for
the
it,
then read
it
manager can deposit the 'P' into its proper position in the buffer
3.15). The system I/O buffer allows the file manager to read and write
file
(Fig.
data in sector-sized or block-sized units. In other words,
manager
to ensure that the organization
of data
in
RAM
it
enables the
file
conforms to the
have on the disk.
Instead of sending the sector immediately to the disk, the file manager
usually waits to see if it can accumulate more bytes going to the same sector
organization
it
will
65
A JOURNEY OF A BYTE
Logical
1.
The program
asks the operating system to write the contents of the
variable c to the next available position in
2.
The operating system passes
3.
The
file
about
it,
manager looks up
TEXT
such as whether the
types of access are allowed,
name TEXT corresponds
4.
The
file
The
file
to the file
manager.
in a table containing information
file is
if
on
open and available for
any, and what physical
file
use,
what
the logical
to.
manager searches a
location of the sector that
5.
the job
TEXT.
allocation table for the physical
file
is to
manager makes sure
contain the byte.
that the last sector in the
stored in a system I/O buffer in
RAM,
has been
file
then deposits the
into
its
proper position in the buffer.
6.
The
file
the byte
7.
manager gives instructions to the I/O processor about where
is stored in RAM and where it needs to be sent on the disk.
The I/O processor
finds a time
when
the drive
is
available to receive
the data and puts the data in proper format for the disk.
buffer the data to send
it
It
may
also
out in chunks of the proper size for the
disk.
8.
The I/O processor sends
9.
The
the data to the disk controller.
controller instructs the drive to move the read/write head to the
proper track, waits for the desired sector to come under the
read/write head, then sends the byte to the drive to be deposited, bitby-bit, on the surface of the disk.
Physical
FIGURE 3.14 Layers of procedures involved in transmitting a byte from a program's data area to a file called TEXT on disk.
before actually transmitting anything. Even though the statement WRITE(TEXT,c,l) seems to imply that our character is being sent immediately to
the disk,
it
(There are
buffer
may in fact be
many situations
is filled
would have
TEXT
kept in
in
before transmitting
to flush
so the data
all
RAM
it.
for
some time before
it
is
sent.
manager cannot wait until
For instance, if TEXT were closed,
which the
file
a
it
output buffers holding data waiting to be written to
would not be
lost.)
66
SECONDARY STORAGE AND SYSTEM SOFTWARE
File i/o system:
User's program:
WRITE ("text",c,
If necessary,
sector
1)
2.
load
last
from "TEXT"
into
system output buffer
Move *P into system
output buffer
User's data area:
c:
GJ
I/O
*"
system's
output buffer
P'
FIGURE 3.15 The file manager moves P from the program's data area to a system
output buffer, where it may join other bytes headed for the same place on the
disk. If necessary, the file manager may have to load the corresponding sector
from the disk into the system output buffer.
3.5.3 The Byte Leaves RAM: The
and Disk Controller
I/O
Processor
So far, all of our byte's activities have occurred within the computer's
primary memory and have probably been carried out by the computer's
central processing unit (CPU). The byte has travelled along data paths that
are designed to be very fast and that are relatively expensive. Now it is time
for the byte to travel along a data path that is likely to be slower and
narrower than the one in primary memory. (A typical computer might have
an internal data-path width of four bytes, whereas the width of the path
leading to the disk might be only two bytes.)
Because of bottlenecks created by these differences in speed and
data-path widths, our byte and its companions might have to wait for an
become available. This also means that the CPU has
on its hands as it deals out information in small enough chunks
and at slow enough speeds that the world outside can handle them. In fact,
the differences between the internal and external speeds for transmitting
external data path to
extra time
67
A JOURNEY OF A BYTE
data are often so great that the
CPU
can transmit to several external devices
simultaneously.
The processes of disassembling and assembling groups of bytes for
transmission to and from external devices are so specialized that it is
unreasonable to ask an expensive, general-purpose CPU to spend its
when a simpler device could do the job as well,
do the work that it is most suited for. Such a
special-purpose device is called an I/O processor.
An I/O processor may be anything from a simple chip capable of taking
a byte and, on cue, just passing it on; to a powerful, small computer capable
of executing very sophisticated programs and communicating with many
valuable time doing I/O
freeing
CPU
the
to
devices simultaneously.
The I/O processor
operating system, but once
it
takes
its
begins processing I/O,
from
instructions
it
the
runs independently,
CPU) of the task of communicating
with secondary storage devices. This allows I/O processes and internal
relieving the operating system (and the
"
computing
In
to overlap."
typical
computer, the
processor that there
is
file
manager might now
data in the buffer that
is
tell
the
I/O
to be transmitted to the disk,
how much
data there is, and where it is to go on the disk. This information
might come in the form of a little program that the operating system
constructs and the I/O processor executes (Fig. 3.16).
The job of actually controlling the operation of the disk is done by a
device called a disk controller. The I/O processor asks the disk controller if
the disk drive is available for writing. If there is much I/O processing, there
is a good chance that the drive will not be available and that our byte will
have to wait in its buffer until the drive becomes available.
What happens next often makes the time spent so far seem insignificant
in
comparison: The disk drive
instructed to
is
move
on the drive where our byte and
be stored. For the
first
mechanical!
The
time, a device
is
its
read/write head to
companions are to
being asked to do something
the track and sector
its
read/write head must seek to the proper track (unless
it is
already there), and then wait until the disk has spun around so the desired
sector
is
under the head. Once the track and sector are located, the I/O
processor (or perhaps the controller) can send out bytes, one
the drive.
drive,
Our
where
it
byte waits until
probably
is
its
stored in
at a
time, to
turn comes, then travels, alone, to the
a little
one-byte buffer while
it
waits to
be deposited on the disk.
On many systems the I/O processor can take data directly from RAM, without further
involvement from the CPU. This process is called direct memory access (DMA). On other
systems, the CPU must place the data in special I/O registers before the I/O processor can
have access to it.
t
68
SECONDARY STORAGE AND SYSTEM SOFTWARE
File
Manager
Invoke I/O processor
User's program:
I/O
processor
User's data area:
program
^
!i
L___i>
I
,-a
System
buffer
I/O processor
FIGURE 3.16 The file manager sends the I/O processor instructions in the form of
an I/O processor program. The I/O processor gets the data from the system
buffer, prepares it for storing on the disk, and then sends it to the disk controller, which deposits it on the surface of the disk.
under the read/ write head, the eight bits of our
byte are deposited, one at a time, on the surface of the disk (Fig. 3.16).
There the 'P' remains, at the end of its journey, spinning about at a leisurely
Finally, as the disk spins
50 to 100 miles per hour.
3.6
Buffer
Any
Management
from some knowledge of what happens to data
program's data area and secondary storage. One aspect
of this process that is particularly important is the use of buffers. Buffering
involves working with large chunks of data in
so the number of
accesses to secondary storage can be reduced. We concentrate on the
operation of system I/O buffers, but be aware that the use of buffers within
programs can also substantially affect performance.
user of files can benefit
travelling
between
RAM
69
BUFFER MANAGEMENT
3.6.1 Buffer Bottlenecks
We know
that a
hold incoming
file
manager
allocates
I/O buffers
that are big
enough
we have said nothing so far about how many
common for file managers to allocate several
data, but
are used. In fact,
it is
to
buffers
buffers
performing I/O.
To understand the need for several system buffers, consider what
happens if a program is performing both input and output on one character
at a time, and only one I/O buffer is available. When the program asks for
its first character, the I/O buffer is loaded with the sector containing the
character, and the character is transmitted to the program. If the program
then decides to output a character, the I/O buffer is filled with the sector
into which the output character needs to go, destroying its original
for
contents.
Then when
the next input character
is
needed, the buffer contents
have to be written to disk to make room for the (original) sector containing
the second input character, and so on.
Fortunately, there is a simple and generally effective solution to this
ridiculous state of affairs, and that is to use more than one system buffer.
For this reason, I/O systems almost always use at least two buffers
one
for input and one for output.
Even if a program transmits data in only one direction, the use of a
single system I/O buffer can slow it down considerably. We know, for
instance, that the operation of reading a sector from a disk is extremely slow
compared to the amount of time it takes to move data in RAM, so we can
guess that a program that reads many sectors from a file might have to
spend much of its time waiting for the I/O system to fill its buffer every
time a read operation is performed before it can begin processing. When this
happens, the program that is running is said to be I/O bound the CPU
spends much of its time just waiting for I/O to be performed. The solution
to this problem is to use more than one buffer and to have the I/O system
filling the
next sector or block of data while the
CPU
is
processing the
current one.
3.6.2 Buffering Strategies
Multiple Buffering Suppose that a program is only writing to a disk and
that it is I/O bound. The CPU wants to be filling a buffer at the same tune
that I/O is being performed. If two buffers arc used and I/O-CPU
overlapping is permitted, the CPU can be filling one buffer while the
contents of the other are being transmitted to disk.
When
both tasks are
70
SECONDARY STORAGE AND SYSTEM SOFTWARE
the roles of the buffers can be exchanged.
finished,
swapping the
called
roles
of two buffers
doti ble buffering.
after
This technique of
each output (or input) operation
Double buffering allows
operating on one buffer while the other buffer
is
the operating system to be
is
being loaded or emptied
(Fig. 3.17).
The
idea of
swapping system buffers
to allow processing
and I/O
to
overlap need not be restricted to two buffers. In theory, any
buffers can be used, and they can be organized in a variety
number of
of ways. The
management of system buffers is usually done by the operating
system and can rarely be controlled by programmers who do not work at
the systems level. It is common, however, for users to be able to control the
number of system buffers assigned to jobs.
Some file systems use a buffering scheme called buffer pooli ng: When a
system buffer is needed, it is taken from a pool of available buffers and used.
When the system receives a request to read a certain sector or block, it looks
to see if one of its buffers already contains that sector or block. If no buffer
contains it, then the system finds from its pool of buffers one that is not
actual
currently in use and loads the sector or block into
it.
FIGURE 3.17 Double buffering: (a) The contents of system I/O buffer 1 are sent to
is being filled; and (b) the contents of buffer 2 are sent to
disk while I/O buffer 1 is being filled.
disk while I/O buffer 2
To
disk
To
disk
(a)
ogram data area
(b)
71
BUFFER MANAGEMENT
Several different schemes are used to decide which buffer to take from
a
buffer pool.
buffer that
is
One
generally effective strategy
le ast recently
use d.
least-recently-used queue, so
less-recently-used
(LRU)
buffers
When
it is
to take
is
buffer
is
allowed to retain
new
its
The
have been accessed.
strategy for replacing old data with
from the pool
accessed,
data has
is
it
data until
tha'j
put on
all
other
least-recently-used
many
applications
computing. It is based on the assumption that a block of data that
has been used recently is more likely to be needed in the near future than
one that has been used less recently. (We encounter LRU again in later
in
chapters.)
It is
ceases
difficult to predict the point at
to
which the addition of extra buffers
As the cost of RAM
contribute to improved performance.
continues to decrease, so does the cost of using
the other hand, the
more
buffers there are, the
system to manage them. When
different numbers of buffers.
Move Mode and
Locate
in
Mode
more and bigger buffers. On
more time it takes for the file
doubt, consider experimenting with
Sometimes
it
is
not necessary
must always be copied from a system buffer
versa), the amount of time taken to perform
This
way of
handling buffered data
moving chunks of data from one
is
called
place in
to a
the
to
When
data
program buffer ( or
vice
distinguish between a program's data area and system buffers.
move
can be substantial.
move mode, since
it
involves
RAM to another before they can
be accessed.
There
are
wo way s
that
move mode
can be avoided. If the
file
manager
can perform I/O directly between secondary storage and the program's data
no extra move
manager could use
system buffers to handle all I/O, but provide the program with the locations,
through the use of pointer variables, of the system buffers. Both techniques
are examples of a general approach to buffering called locate mod e. When
locate mode is used, a program is able to operate directly on data in the I/O
buffer, eliminating the need to transfer data between an I/O buffer and a
program buffer.
area,
is
necessary. Alter nalively^ the
file
Suppose you are reading in a file with many blocks,
where each block consists of a header followed by data. You would like to
put the headers in one buffer and the data in a different buffer so the data can
be processed as a single entity. The obvious way to do this is to read the
whole block into a single big buffer, and then move the different parts to
their own buffers. Sometimes we can avoid this two-step process using a
technique called sc^lfx^in^uL With scatter input, a single READ call
Scatter/Gather I/O
72
SECONDARY STORAGE AND SYSTEM SOFTWARE
identifies
block
is
not one, but
a collection
of buffers into which data from
a single
to be scattered.
The converse of
scatter input
gather oulgut,.
is
With gather output,
and written with a single WRITE call,
avoiding the need to copy them to a single output buffer. When the cost of
copying several buffers into a single output buffer is high, scatter/gather can
have a significant effect on the running time of a program.
It is not always obvious when features like scatter/gather, locate mode,
and buffer pooling are available in an operating system. You often have to
go looking for them. Sometimes you can invoke them by communicating
with your operating system, and sometimes you can cause them to be
invoked by organizing your program in ways that are compatible with the
way the operating system does I/O. Throughout this text we return many
times to the issue of how to enhance performance by thinking about how
buffers work and adapting programs and file structures accordingly.
several buffers can be gathered
3.7
I/O in
UNIX
We see in the journey of a byte that we can view I/O as proceeding through
several layers. UNIX provides a good example of how these layers occur in
operating system, so we conclude this chapter with a look at UNIX.
of course beyond the scope of this text to describe the UNIX I/O layers
in detail. Rather, our objective here is just to pick a few features of UNIX
that illustrate points made in the text. A secondary objective is to familiarize
you with some of the important terminology used in describing UNIX
systems. For a comprehensive, detailed look at how UNIX works, plus a
thorough discussion of the design decisions involved in creating and
improving UNIX, see Leffler et al. (1989).
a real
It is
3.7.1 The Kernel
In Fig. 3. 14
we
see
how
the process of transmitting data
an external device can be described
The topmost
name,
file a
file.
body of text, an image, an
The
view
program
to
a series
of layers.
We
store in a
array of numbers, or
some other
that an application has of
what goes
layers that follow collectively carry out the task of turning
the logical object into a collection of bits
Likewise, the topmost I/O layer in
logical terms. This layer in
logical
from
proceeding through
layer deals with data in logical, structural terms.
logical entity. This reflects the
into a
as
views on
files.
UNIX
on
a physical device.
UNIX
deals with data primarily in
consists of processes that
impose
certain
Processes are associated with solving some problem,
I/O IN
PROCESSES
user programs
shell
73
UNIX
commands
libraries
system
call
interface
KERNEL
I/O
block
character
network
system
(normal
I/O system
files)
printers, etc.)
(terminals,
block device drivers
network interface drivers
TIT TTT
consoles
disk...
(sockets)
character device drivers
TT
disk
I/O system
printers...
..networks...
HARDWARE
FIGURE 3.18 Kernel
such
as
I/O structure.
counting the words in the
Processes include
and
programs
library
files,
once
The
we
scanf(
numbers,
this layer is the
UNIX
kernel views
all
I/O
as
pass control to the kernel
a file are
gone.
The
or searching for somebody's address.
and tail, user programs that operate on
and fread( ) that are called from
etc.
kernel,
The components of the
the layers.^
3.18.
routines like
to read strings,
Below
file
shell routines like cat
which incorporates
all
the rest of
kernel that do I/O are illustrated in Fig.
operating on
all
sequence of bytes, so
assumptions about the logical view of
decision to design
UNIX
in this
way
to
make
all
operations below the top layer independent of an application's logical view
of
a file
UNIX
""It
is
as a
is
unusual.
beyond the scope of
tion of the
It
is
also
one of the main attractions in choosing
UNIX lets us make all of the decisions
focus for this text, for
UNIX
this text to describe the
UNIX
kernel in detail. For
kernel, including the I/O system, see Leffler et
al.
(1989).
a full
descrip-
74
SECONDARY STORAGE AND SYSTEM SOFTWARE
about the logical structure of
think about the
file
a file,
beyond the
imposing no
it must be
fact that
on
restrictions
built
from
how we
sequence of
bytes.
Let's illustrate the journey
in this chapter
by tracing the
of a byte through the kernel, as we did earlier
of an I/O statement. We assume in this
character to disk. This corresponds to the left
results
example that we are writing a
branch of the I/O system in Fig. 3.18.
When your program executes a system
call
such
as
write (fd, &c, 1);
invoked immediately. * The routines that let processes
communicate directly with the kernel make up the syst em call interface. In
this case, the system call instructs the kernel to writelfcharacter to a file.
The kernel I/O system begins by connecting the file descriptor (fd) in
your program to some file or device in the filesystem. It does this by
proceeding through a series of four tables that enable the kernel to find its
the
kernel
way from
is
process to the places on the disk that will hold the
The
refer to.
file
that they
four tables are
descriptor table;
a file
an open
with information about open files;
which is part of a structure called an index
file table,
a file allocation table,
node; and
a table
of index nodes, with one entry for each
Although these
a sense,
tables are
"owned" by
managed by
file
in use.
the kernel's I/O system, they are, in
different parts of the system:
The file descriptor table is owned by the process (your program).
The open file table and index node tables are owned by the kernel.
The index node itself is part of the filesystem.
The
four tables are invoked in turn by the kernel to get the information
needs to write to your
file
on
disk. Let's see
how
this
works by looking
it
at
the functions of the tables.
The
of the
open
file
file
descriptors used
file table.
entries for
and
descriptor table (Fig. 3.19a)
by
simple table that associates each
process with an entry in another
Every process has
all files it
is
its
own
descriptor table,
has opened, including the "files"
table, the
which includes
STDIN, STDOUT,
STDERR.
^This should not be confused with a library call, such as fprintf( ), which invokes the standard library to perform some additional operations on the data, such as converting it to an
ASCII format, and then makes a corresponding system call.
I/O IN
75
UNIX
(a) descriptor table
table
file
file
descriptor
entry
*-
to open file
table
(b) open file table
Number of
inode
processes
Offset
of next
ptr to
R/W
write
table
mode
using
access
routine
entry
it
to inode
^^^
write
table
100
""
write() routine
for this type
of
FIGURE 3.19 Descriptor table and open
The
file
file
table.
open file.
added to the open file
entries are called file structures, and they contain important
information about how the corresponding file is to be used, such as the
read/write mode used when it was opened, the number of processes
open
Every time a
table. These
file
file is
able (Fig. 3.19b) contains entries for every
opened or
created, a
new
entry
is
and the offset within the file to be used for the next read
or write. The open file table also contains an array of pointers to generic
currently using
it,
76
SECONDARY STORAGE AND SYSTEM SOFTWARE
functions that can be used to operate on the
depending on the type of
file.
These functions
will differ
file.
same open file
one process could read part of a file, another process could
read the next part, and so forth, with each process taking over where the
previous one stopped. On the other hand, if the same file is opened by two
separate open ( ) statements, two separate entries are made in the table, and
the two processes operate on the file quite independently.^
The information in the open file table is transitory. It tells the kernel
what it can do with a file that has been opened in a certain way and provides
information on how it can operate on the file. The kernel still needs more
information about the file itself, such as where the file is stored on disk, how
big the file is, and who owns it. This information is found in an ndex node,
possible for several different processes to refer to the
It is
table entry, so
more commonly
An
inode
structure.
inode exists
inode
is
file
as
more permanent
long
as its
a file
is
opened,
open file table's file
open for access, but an
structure than an
structure exists only while a
corresponding
kept on disk with the
is
When
referred to as an 'mode (Fig. 3.20).
file is
For
file exists.
this reason, a file's
(though not physically adjacent to the file).
copy of its inode is usually loaded into
where
file
RAM
added to the aforementioned inode table for rapid access.
For the purposes of our discussion, the most important component of
the inode is a list (index) of the disk blocks that make up the file. This list
it is
is
the
UNIX
counterpart to the
in this chapter.*
knows
all
that
Once
it
fi le
needs to
processor program that
is
know
about the
is
program is called a device driver.
The device driver sees that your data
proper place on disk. Before we look at the
instructive to look at
kinds of
file
data that
3.7.2 Linking
It is
File
it
how
must
Names
instructive to look a
cess at the
that
meaning of these may be
*This might not be
table often has a
you
to be written.
is
role
more
file.
closely at
All
UNIX,
its
buffer to
of device drivers in
how
references
It"
you
are independently reading
difficult to
moved from
In
among
this
its
UNIX,
the different
to Files
a file
to
name
files
is
actually
begin with
file with one prowith another, the
are writing to a
from the
file
determine.
simple linear array.
dynamic,
it
then invokes an I/O
It
the kernel distinguishes
there are risks in letting this happen.
same time
file.
deal with.
little
linked to the corresponding
*Of course,
described earlier
appropriate for the type of data, the type of
operation, and the type of device that
it is
we
allocation table that
the kernel's I/O system has the inode information,
To accommodate both
tree-like structure.
large and small
files,
this
I/O IN
77
UNIX
device
permissions
owner's userid
file size
block count
w^
tile
allocation
table
FIGURE 3.20 An inode. The inode is the data structure used by UNIX to describe
file. It includes the device containing the file, permissions, owner and group
the
IDs,
and
allocation table,
file
directory, for
is
it is
just a small
pointer to the
inode of a
name
link
is
to
other things.
in directories that file
file's
It
are kept. In fact, a directory
a file
name
together with
other information about the
file.
RAM
When
and to
a directory to the
provides a direct reference from the
used to bring the inode into
It is
file,
inode on disk.^ This pointer from
called a hard link.
entry in the open
names
that contains, for each
file
file is
all
among
a file is
opened,
file
hard
this
up the corresponding
set
file table.
possible for several
file
names
can have several different names.
to point to the
field in the
same inode, so one
inode
tells
how many
file
hard
means that if a file name is deleted and
same file, the file itself is not deleted; its
decremented by one.
links there are to the inode. This
there are other
file
names
inode's hard-link count
is
for the
just
There is another kind of link, called a soft link, or syrnbolkjliik A
symbolic link links a file name to another file name, rather than to an actual
file. Instead of being a pointer to an inode, a soft link is a pathname of some
.
^The
actual structure of a directory
sential parts. See Leffler, et
al.
is
a little
more complex than
(1989) for details.
this,
but these arc the
es-
78
SECONDARY STORAGE AND SYSTEM SOFTWARE
symbolic link does not point to an actual file, it can refer to a
file in a different file system. Symbolic links are not
supported on all UNIX systems. UNIX System 4.3BSD supports symbolic
links, but System V does not.
Since
file.
directory or even to a
3.7.3 Normal
Files,
Special Files, and Sockets
The "everything is a file" concept in UNIX works only when we recognize
that some files are quite a bit different from others. We see in Fig. 3.18 that
the kernel distinguishes among three different types of files. Normal files are
files that this text is about. Special files almost always represent a stream
of characters and control signals that drive some device, such as a line
the
printer or a graphics device.
table (Fig.
The
3.19a) are special
first
files.
three
file
descriptors in the descriptor
Sockets are abstractions that serve as
endpoints for interprocess communication.
At
a certain
are very similar,
conceptual level, these three different types of
and many of the same routines can be used
UNIX
files
any
of them. For instance, you can establish access to all three types by opening
them, and you can write to them with the write( ) system call.
3.7.4 Block
I/O
In Fig.
we
3.18,
to access
see that the three different types of files access their
respective devices via three different I/O systems, the block I/O system, the
I/O system, and the network I/O system. Henceforth we ignore the
second and third categories, since it is normal file I/O that we are most
character
concerned with in this text.*
The block I/O system is the
the journey of a byte.
It
viewed by the user
data,
device like
UNIX
concerns
as a
itself
counterpart of the
with
how
sequence of bytes, onto
disk or tape. Given a byte to store
on
file
to transmit
a
manager in
normal file
block-oriented
a disk, for
example,
it
arranges to read in the sector containing the byte to be replaced, to replace
the byte, and to write the sector back to the disk.
UNIX view of a block device most closely resembles that of a disk.
randomly addressable array of fixed blocks. Originally all blocks were
512 bytes, which was the common sector size on most disks. No other
organization (such as clusters) was imposed on the placement of files on
The
It is
not entirely true. Sockets, for example, can be used to move normal
network systems bypass the normal
favor of sockets to squeeze every bit of performance out of the network.
iThis
is
place to place. In fact, high-performance
files
file
from
system
in
I/O IN
disk. (In section 3.1.7
with
we saw how
UNIX
the design of later
79
UNIX
systems dealt
convention.)
this
3.7.5 Device Drivers
For each peripheral device there
is a separate set of routines, called a device
performs the actual I/O between the I/O buffer and the device.
A device driver is roughly equivalent to the I/O processor program
described in the journey of a byte.
Since the block I/O system views a peripheral device as an array of
driver, that
physical blocks, addressed as block
driver's
job
is
physical blocks, and see that
it
0,
from
to take a block
block
etc.,
1,
a buffer,
block I/O device
destined for one of these
gets deposited in the proper physical place
the device. This saves the block I/O part of the kernel
from having
to
on
know
it is writing to, other than its identity and
thorough discussion of device drivers for block,
character, and network I/O can be found in Leffler et al. (1989).
anything about the specific device
that
it is
block device.
3.7.6 The Kernel and Filesystems
In Chapter 2
we
filesystem
is
the actual
files in
UNIX concept of a filesystem. A UNIX
of files, together with secondary information about
described the
a collection
the system.
filesystem includes the directory structure,
and the inodes that describe the files.
In our discussions we talk about the filesystem as if it is part of the
kernel's I/O system, which it is, but it is also in a sense separate from it. All
where the kernel
parts of a filesystem reside on disk, rather than in
the
kernel
as needed.
by
does its work. These parts are brought into
the directories, ordinary
files,
RAM
RAM
This separation of the filesystem from the kernel has many advantages. One
important advantage is that we can tune a filesystem to a particular device
or usage pattern independently of how the kernel views files. The
discussions in section 3.1.7 of
4.3BSD block
organization are file-system
concerns, for example, and need not have any effect on
how
the kernel
works.
Another advantage of keeping the filesystem and I/O system distinct is
that we can have separate filesystems that are organized differently, perhaps
on different devices, but are accessible by the same kernel. In Appendix A,
for instance, we describe the design of a filesystem on CDROM that is
organized quite differently from a typical disk-based file system yet looks
just like any other filesystem to the user and to the I/O system.
80
SECONDARY STORAGE AND SYSTEM SOFTWARE
3.7.7 Magnetic Tape and UNIX
it is to computing, magnetic tape is somewhat of an orphan in
view of I/O. A magnetic tape unit has characteristics similar to
both block I/O devices (being block oriented) and character devices (being
Important
the
as
UNIX
primarily used for sequential access), but does not
fit
nicely into either
category. Character devices read and write streams of data, not blocks, and
block devices in general access blocks randomly, not sequentially.
I/O is generally the least inappropriate of the two
inappropriate paradigms for tape, a tape device is normally considered in
UNIX to be a block I/O device and hence is accessed through the block I/O
interface. But because the block I/O interface is most often used to write to
random-access devices, disks, it does not require blocks to be written in
sequence, as they must be written to a tape. This problem is solved by
allowing only one write request at a time per tape drive. When highperformance I/O is required, the character device interface can be used in a
raw mode to stream data to tapes, bypassing the stage that requires the data
Since
block
to be collected into relativelv small blocks before or after transmission.
SUMMARY
In this chapter
we look
at
the software environment in
which
file
processing
programs must operate and at some of the hardware devices on which files
are commonly stored, hoping to understand how they influence the ways
we design and process files. We begin by looking at the two most common
storage media: magnetic disks and tapes.
A disk drive consists of a set of read/write heads that are interspersed
among one or more platters. Each platter contributes one or two surfaces,
each surface contains a set of concentric tracks, and each track is divided into
sectors or blocks.
read/write heads
is
The
set
of tracks that can be read without moving the
called a cylinder.
by sector and by
block. Used in this context, the term block refers to a group of records that
are stored together on a disk and treated as a unit for I/O purposes. When
There are two basic ways
blocks are used, the user
is
to address data
better able to
make
on
disks:
the physical organization of
and hence can sometimes
improve performance. Block-organized drives also sometimes make it
possible for the disk drive to search among blocks on a track for a record
with a certain key without first having to transmit the unwanted blocks into
data correspond to
its
logical
organization,
RAM.
Three possible disadvantages of block-organized devices are the danger
of internal track fragmentation, the burden of dealing with the extra
SUMMARY
complexity that the user has to bear, and the
the kinds of synchronization (such
some of
loss
of opportunities to do
as sector interleaving)
that
sector-addressing devices provide.
The
cost of a disk access can be
for seeking, rotational delay,
used,
it is
physically
measured in terms of the time it takes
and transfer time. If sector interleaving is
possible to access logically adjacent sectors
by one or more
sectors.
Although
by separating them
takes
it
much
less
time to
access a single record directly than sequentially, the extra seek time required
for
doing direct accesses makes it much slower than sequential access when
of records is to be accessed.
Despite increasing disk performance, network speeds have improved to
a series
the point that disk access
system.
is
often a significant bottleneck in an overall I/O
number of techniques
including striping, the use of
are available for addressing this problem,
RAM
disks,
can have a major
from 512 bytes
especially
effect
and disk caching.
BSD UNIX
Research done in connection with
shows
that block size
on performance. By increasing the default block size
throughput was improved enormously,
to 4,096 bytes,
for large
files,
because eight times
transferred in a single access.
as
A negative consequence
much
data
could be
of this reorganization
wasted storage increased from 6.9% for 512-byte blocks to 45.6%
It turned out that this problem of wasted space could
treating
the 4,096-byte blocks as clusters of 512-byte
be dealt with by
blocks, w^hich could be allocated to different files.
Though not as important as disks, magnetic tape has an important niche
was
that
for 4,096-byte blocks.
in file processing.
Tapes are inexpensive, reasonably
fast for sequential
processing, compact, robust, and easy to store and transport. Data are
usually organized
on
tapes in one-bit-wide parallel tracks, with a bit-wide
cross-section of tracks interpreted as one or
processing speed and
space utilization,
it is
more
bytes.
When
estimating
important to recognize the role
played by the interblock gap. Effective recording density and effective
transmission rate are useful measurements of the performance one can
expect to achieve for a given physical
file
organization.
secondary storage media, we see that
disks are replacing tape in more and more cases. This is largely because
is becoming less expensive, relative to secondary storage, which
means that one of the earlier advantages of tape over disk, the ability to do
In
comparing disk and tape
as
RAM
sequential access without seeking, has diminished significantly.
RAM
to disk.
journey of a byte as it is sent from
and
programs
The journey involves the participation of many different
This chapter follows
devices, including
a user's
tem;
program, which makes the
initial call to
the operating sys-
81
82
SECONDARY STORAGE AND SYSTEM SOFTWARE
file manager, which maintains tables of information that it uses to translate between the program's logical view of
the file and the physical file where the byte is to be stored;
an I/O processor and its software, which transmit the byte, synchronizing the transmission of the byte between an I/O buffer in
and the disk;
the disk controller and its software, which instruct the drive about
how to find the proper track and sector, then send the byte; and
tne disk drive, which accepts the byte and deposits it on the disk sur-
the operating system's
RAM
face.
Next,
for
we
on techniques
improve performance. Some techniques include
take a closer look at buffering, focusing mainly
managing
buffers to
double buffering, buffer pooling, locate-mode buffering, and scatter/gather
buffering.
We
second look at I/O layers, this time concentrating
I/O system call begins with a call to the UNIX
kernel, which knows nothing about the logical structure of a file, treating all
data essentially the same
as a sequence of bytes to be transmitted to some
on
conclude with
UNIX. We
see that every
external device. In doing
its
work
tables: a file descriptor table,
access table in the
to use
and
how
file's
inode.
to access
it,
it
the I/O system in the kernel invokes four
an open
Once
calls
an inode table, and a
file table,
the kernel has determined
on
file
which device
device driver to carry out the actual
accessing.
Although it treats every file as a sequence of bytes, the kernel I/O
system deals differently with three different types of I/O: block I/O,
character I/O, and network I/O. In this text we concentrate on block I/O.
We look briefly at the special role of the file system within the kernel,
how it uses links to connect file names in directories to their
corresponding inodes. Finally, we remark on the reasons that magnetic tape
does not fit well into the UNIX paradigm for I/O.
describing
KEY TERMS
bpi. Bits per inch per track.
tracks.
On
a tape,
On
a disk, data
is
recorded serially on
data are recorded in parallel
on
several tracks, so a
6,250-bpi nine-track tape contains 6,250 bytes per inch,
when
all
nine
tracks are taken into account (one track being used for parity).
Block. Unit of data organization corresponding
to the
amount of data
transferred in a single access. Block often refers to a collection of
KEY TERMS
records, but
it
may
be
a collection
of sectors
has no correspondence to the organization of the data.
sometimes
called a physical record; a sector
whose
(see cluster)
is
sometimes
block
size
is
called a
block.
Block device.
in
In
UNIX,
device such as
disk drive that
is
organized
blocks and accessed accordingly.
Block I/O. I/O between a computer and a block device.
Block organization. Disk drive organization that allows
the user to
define the size and organization of blocks, and then access a block by
giving
its
block address or the key of one of
its
records. (See sector
organization.)
Blocking factor. The number of records
Character device. In
tape drive
UNIX,
when stream I/O
stored in one block.
device such as
is
keyboard or printer
(or
used) that sends or receives data in the
form of a stream of characters.
Character I/O. I/O between a computer and a character device.
Cluster. Minimum unit of space allocation on a sectored disk, consisting
of one or more contiguous sectors. The use of large clusters can improve sequential access times by guaranteeing the ability to read
longer spans of data without seeking. Small clusters tend to decrease
internal fragmentation.
Controller. Device that directly controls the operation of one or more
secondary storage devices, such as disk drives and magnetic tape
units.
Count subblock. On block-organized
drives, a small block that pre-
cedes each data block and contains information about the data block,
such as its byte count and its address.
Cylinder. The set of tracks on a disk that are directly above and below
each other. All of the tracks in a given cylinder can be accessed without having to move the access arm; that is, they can be accessed
without the expense of seek time.
Descriptor table. In UNIX, a table associated with a single process that
links all of the file descriptors generated by that process to corresponding entries in an open file table.
Device driver. In UNIX, an I/O processor program invoked by the
kernel that performs I/O for a particular device.
Direct access storage device (DASD). Disk or other secondary storage device that permits access to a specific sector or block of data
without first requiring the reading of the blocks that precede it.
Direct memory access (DMA). Transfer of data directly between
and peripheral devices, without significant involvement by the
RAM
CPU.
83
84
SECONDARY STORAGE AND SYSTEM SOFTWARE
Disk cache. A segment of RAM configured to contain pages of data
from a disk. Disk caches can lead to substantial improvements in access time
when
access requests exhibit a high degree of locality.
Disk pack. An assemblage of magnetic
tical shaft.
pack of disks
number of cylinders
If disk
is
disks
mounted on
the
same ver-
treated as a single unit consisting of a
equivalent to the
number of tracks
per surface.
packs are removable, different packs can be mounted on the
same drive
at different times,
providing
convenient form of offline
storage for data that can be accessed directly.
Effective recording density. Recording density
after taking into ac-
count the space used by interblock gaps, nondata subblocks, and
other space-consuming items that
accompany
Effective transmission rate. Transmission rate
data.
after taking into ac-
count the time used to locate and transmit the block of data in which
a
desired record occurs.
One or more adjacent clusters allocated as part (or all) of a file.
The number of extents in a file reflects how dispersed the file is over
the disk. The more dispersed a file, the more seeking must be done
in moving from one part of the file to another.
File allocation table (FAT). A table that contains mappings to the
Extent.
physical locations of
File
all
the clusters in
all files
on disk
storage.
manager. The part of an operating system that is responsible for
managing files, including a collection of programs whose responsibilities range from keeping track of files to invoking I/O processes that
transmit information between primary and secondary storage.
File structure. In connection with the
open
file
table in a
UNIX
kernel,
the term file structure refers to a structure that holds information the
kernel needs about an open
such things
rently using
as the file's
it,
and the
file.
File structure
read/write mode,
information includes
number of processes
offset within the file to be
cur-
used for the next
read or write.
Filesystem. In
UNIX,
a hierarchical collection
of
single secondary device, such as a hard disk or
files,
usually kept
on
CD-ROM.
Fixed disk. A disk drive with platters that may not be removed.
Formatting. The process of preparing a disk for data storage, involving
such things as laying out sectors, setting up the disk's file allocation
table, and checking for damage to the recording medium.
Fragmentation. Space that goes unused within a cluster, block, track,
or other unit of physical storage. For instance, track fragmentation
when space on a track goes unused because there
enough space left to accommodate a complete block.
Frame. A one-bit-wide slice of tape, usually representing a
occurs
is
not
single byte.
KEY TERMS
Hard
link. In
to the
UNIX,
links to a single
deleted until
Index node.
its
all
directory that connects
hence
file;
a file
name
There can be several hard
can have several names. A file is not
a file
hard links to the
file.
file
are deleted.
UNIX, a data structure associated with a file that deAn index node includes such information as a file's
In
scribes the
type,
an entry in
inode of the corresponding
file.
owner and group IDs, and a list of the disk blocks
file. A more common name for index node is
comprise the
that
inode.
Inode. See index node.
Interblock gap. An interval of blank space that separates sectors,
blocks, or subblocks on tape or disk. In the case of tape, the gap
provides sufficient space for the tape to accelerate or decelerate
starting or stopping.
read/write heads to
when
On
tell
both tapes and disks the gaps enable the
accurately when one sector (or block or sub-
block) ends and another begins.
Interleaving factor. Since it is often not possible to read physically adjacent sectors of a disk, logically adjacent sectors are sometimes arranged so they are not physically adjacent. This is called interleaving.
The interleaving factor refers to the number of physical sectors the
next logically adjacent sector
is
located
from the current
sector being
read or written.
I/O processor. A device that
work on non-I/O tasks.
carries out
Kernel. The central part of the
Key subblock. On
UNIX
I/O
tasks,
allowing the
to
operating system.
block-addressable drives,
block that contains the
key of the last record in the data block that follows
it,
allowing the
among the blocks on
without having to load the blocks into primary
a track for a block containing a
drive to search
certain key,
CPU
mem-
ory.
Mass storage system. General term
applied to storage units with large
capacity. Also applied to very high-capacity secondary storage systems that are capable of transmitting data between a disk and any of
several thousand tape cartridges within a
few seconds.
density. Recording density on a disk track or
magnetic tape without taking into account the effects of gaps or non-
Nominal recording
data subblocks.
Nominal transmission
rate. Transmission rate of a disk or tape unit
without taking into account the effects of such extra operations as
seek time for disks and interblock gap traversal time for tapes.
Open file table. In UNIX, a table owned by the kernel with an entry,
called a file structure, for each
open
file.
See
file structure.
85
86
SECONDARY STORAGE AND SYSTEM SOFTWARE
An
Parity.
error-checking technique in which an extra parity
panies each byte and
is
set in
such
way
that the total
bit
accom-
number of
even (even parity) or odd (odd parity).
Platter. One disk in the stack of disks on a disk drive.
Process. An executing program. In UNIX, several instances of the same
program can be executing at the same time, as separate processes.
The kernel keeps a separate file descriptor table for each process.
configured to simulate a disk.
disk. Block of
Rotational delay. The time it takes for the disk to rotate so the desired
bits is
RAM
RAM
sector is under the read/write head.
Scatter/gather I/O. Buffering techniques that involve, on input, scattering incoming data into more than one buffer, and, on output,
gathering data from several buffers to be output as
chunk of
a single
data.
Sector.
The
fixed-sized data blocks that together
make up
on
the tracks
certain disk drives. Sectors are the smallest addressable unit
on
a disk
whose tracks are made up of sectors.
Sector organization. Disk drive organization that uses sectors.
Seek time. The time required to move the access arm to the correct cylinder on a disk drive.
Sequential access device.
card reader, in which the
the beginning.
device, such as a magnetic tape unit or
medium
Sometimes
(e.g., tape)
must be accessed from
called a serial device.
Socket. In UNIX, a socket is an abstraction that serves as an endpoint
of communication within some domain. For example, a socket can
be used to provide direct communication between two computers.
Although
in
some ways
the kernel treats sockets like
files,
we do
not
deal with sockets in this text.
Soft link. See symbolic
link.
UNIX,
ters
the term special file refers to a stream of characand control signals that drive some device, such as a line printer
or
graphics device.
Special
a
file.
In
Streaming tape drive. A tape drive whose primary purpose is dumping large amounts of data from disk to tape or from tape to disk.
Subblock. When blocking is used, there are often separate groupings of
information concerned with each individual block. For example,
count subblock,
key subblock, and
data subblock might
all
be
present.
Symbolic link. In UNIX, an entry in a directory that gives the pathname of a file. Since a symbolic link is an indirect pointer to a file,
not
with the file
can point to directories, or even to files
is
as closely associated
as a
hard
link.
Symbolic
in other filesystems.
it
links
EXERCISES
Track. The set of bytes on a single surface of a disk that can be accessed
without seeking (without moving the access arm). The surface of a
disk can be thought of as a series of concentric circles, with each circle corresponding to a particular position of the access arm and read/
write heads. Each of these circles is a track.
Transfer time. Once the data we want is under the read/write head, we
have to wait for it to pass under the head as we read it. The amount
of time required for this motion and reading is the transfer time.
EXERCISES
Determine as well as you can what the journey of a byte would be like
on your system. You may have to consult technical reference manuals that
describe your computer's file management system, operating system, and
peripheral devices. You may also want to talk to local gurus who have
experience using your system.
1.
Suppose you are writing
2.
a list
of names to
write statement.
Why is it not a good idea to
and then reopen
it
Find out what
3.
for
utility routines are available
computing system, there are
of users, depending on what
When you
every write,
create or
open
Compared
on your computer system
you have a large
utilization. If
different routines available for different kinds
privileges and responsibilities they have.
a file in
information to your computer's
properly.
one name per
file after
before the next write?
monitoring I/O performance and disk
4.
a text file,
close the
file
or Pascal, you must provide certain
manager so it can handle your file
to certain languages, such as PL/I or
COBOL,
the
amount of information you must provide in C or Pascal is very small. Find
a text or manual on PL/I or COBOL and look up the ENVIRONMENT
file description attribute, which can be used to tell the file manager a great
deal about how you expect a file to be organized and used. Compare PL/I
or
COBOL
with
available to the
5.
Much is
way
file
or Pascal in terms of the types of
said in section 3.
to store files.
every
Assume
that
must occupy
stored on
a file is
problems does
file
specifications
programmer.
it
create?
about
how
disk space
is
organized physically
no such complex organization
a single
tape.
contiguous piece of
How
does
this
is
a disk,
used and that
somewhat the
What
simplify disk storage?
87
88
6.
SECONDARY STORAGE AND SYSTEM SOFTWARE
program requests that a 128-byte
manager may have to read a sector from
can write the record. Why? What could you do to decrease
disk drive uses 512-byte sectors. If a
record be written to disk, the
the disk before
the
7.
it
number of times such an
file
extra read
is
likely to occur?
We
have seen that some disk operating systems allocate storage space
in clusters and/or extents, rather than sectors, so the size of any file
a multiple of a cluster or extent.
a. What are some advantages and potential disadvantages of this
method of allocating disk space?
b. How appropriate would the use of large extents be for an application that mostly involves sequential access of very large files?
c. How appropriate would large extents be for a computing system
that serves a large number of C programmers? (C programs tend to
be small, so there are likely to be many small files that contain C
programs.)
d. The VAX record management system uses a default cluster size
of three 512-byte sectors but lets a user reformat a drive with any
cluster size from 1 to 65,535 sectors. When might a cluster size larger
than three sectors be desirable? When might a smaller cluster size be
on disks
must be
desirable?
8.
In early
disk,
UNIX
systems, inodes were kept together on one part of
while the corresponding data was scattered elsewhere on the disk.
Later editions divided disk drives into groups of adjacent cylinders called
cylinder groups, in
corresponding data.
which each cylinder group contains inodes and their
How does this new organization improve perfor-
mance?
UNIX
systems, the minimum block size was 512 bytes, with
of one. The block size was increased to 1,024 bytes in 4.0BSD,
more than doubling its throughput. Explain how this could occur.
9.
In early
a cluster size
10.
Draw
the
numbers
11.
The IBM 3350
pictures that illustrate the role of fragmentation in determining
in
Table
3.2, section 3.1.7.
disk drive uses block addressing.
The two subblock
organizations described in the text are available:
Count-data, where the extra space used by count subblock and interis equivalent to 185 bytes; and
Count-key-data, where the extra space used by the count and key
subblocks and accompanying gaps is equivalent to 267 bytes, plus
block gaps
the key size.
An IBM
cylinder,
3350 has 19,069 usable bytes available per track, 30 tracks per
and 555 cylinders per drive. Suppose you have a file with 350,000
EXERCISES
80-byte records that you want to store on
a 3350 drive. Answer the
following questions. Unless otherwise directed, assume that the blocking
factor is 10 and that the count-data subblock organization is used.
a.
How many
How many
blocks can be stored on one track?
records?
b.
How many
blocks can be stored on one track
data subblock organization
Make
is
used and key
size
the count-key-
if
13 bytes?
is
graph that shows the effect of block size on storage utilization, assuming count-data subblocks. Use the graph to help predict
the best and worst possible blocking factor in terms of storage utilic.
zation.
Assuming that access to the file is always sequential, use the
graph from the preceding question to predict the best and worst
blocking factor. Justify your answer in terms of efficiency of storage
utilization and processing time.
d.
e.
How many
cylinders are required to hold the
10 and count-data format)?
How much
file
(blocking factor
space will go unused due to
internal track fragmentation?
If the file were stored on contiguous cylinders and if there were
no interference from other processes using the disk drive, the average
seek time for a random access of the file would be about 12 msec.
Use this rate to compute the average time needed to access one
f.
record randomly.
g.
how
Explain
fected
retrieval
time for random accesses of records
by increasing block
size.
af-
is
Discuss trade-offs between storage
and retrieval when different block sizes are used. Make a
with different block sizes to illustrate your explanations.
Suppose the file is to be sorted and a shell sort is to be used to
efficiency
table
h.
sort the
Since the
file.
sorted in place,
on the
that this requires
sents the total
random
It is
memory,
estimated (Knuth, 1973b,
number of records
access. If
provide
disk.
too large to read into
about 15N1.25 moves of records, where
all
to sort the file? (As
We
file is
much
in the
of the preceding
you
will see, this
will be
it
p.
380)
N repre-
Each move requires
file.
is
true,
how
is
not
very good solution.
better ones in Chapter
7,
which
long does
it
take
deals with cose-
quential processing.)
12.
there
sectored disk drive differs from one with
is
less
of
correspondence between
block organization
the
logical
and
in that
physical
organization of data records or blocks.
For example, consider the Digital
RM05
disk drive,
which uses sector
has 32 512-byte sectors per track, 19 tracks per cylinder, and
823 cylinders per drive. From the drive's (and drive controller's) point ot
addressing.
It
89
90
SECONDARY STORAGE AND SYSTEM SOFTWARE
view, a
file is
just a vector of bytes divided into 512-byte sectors. Since the
knows nothing about where one record ends and another begins, a
record can span two or more sectors, tracks, or cylinders.
One common way that records are formatted on the RM05 is to place
a two-byte field at the beginning of each block, giving the number of bytes
drive
of
followed by the data
store a
a.
There
no extra gap and no other
you want to
file with 350,000 80-byte records, answer the following questions:
How many records can be stored on one track if one record is
data,
overhead. Assuming that
this
itself.
organization
is
is
used, and that
stored per block?
b.
c.
How many
How might
cylinders are required to hold the
you block records
sults in 10 actual records
file?
so each physical record access re-
being accessed?
What
are the benefits
of do-
ing this?
13. Suppose you have a collection of 500 large images stored in files, one
image per file, and you wish to "animate" these images by displaying them
in sequence on a workstation at a rate of at least 15 images per second over
a high-speed network. Your secondary storage consists of a disk farm with
30 disk drives, and your disk manager permits striping over as many as 30
drives, if you request it. Your drives are guaranteed to perform I/O at a
steady rate of 2 megabytes per second. Each image is 3 megabytes in size.
Network transmission speeds
a.
mation
b.
are.
not a problem.
Describe in broad terms the steps involved in doing such an aniin real
time from disk.
Describe the performance issues that you have to consider in im-
plementing the animation. Use numbers.
c. How might you configure your I/O system to achieve the desired
performance?
Consider the 1,000,000-record mailing list file discussed in the text. The
to be backed up on 2,400-foot reels of 6,250-bpi tape with 0.3-inch
interblock gaps. Tape speed is 200 inches per second.
a. Show that only one tape would be required to back up the file if a
blocking factor of 50 is used.
b. If a blocking factor of 50 is used, how many extra records could
be accommodated on a 2,400-foot tape?
c. What is the effective recording density when a blocking factor of
14.
file is
50
is
d.
How
used?
large does the blocking factor have to be to achieve the
maximum
effective recording density?
What
negative results can re-
from increasing the blocking factor? (Note: An I/O buffer
enough to hold a block must be allocated.)
sult
large
FURTHER READINGS
the minimum blocking factor required to fit the
onto the tape?
f.
If a blocking factor of 50 is used, how long would it take to read
one block, including the gap? What would the effective transmission
rate be? How long would it take to read the entire file?
g. How long would it take to perform a binary search for one
What would be
e.
file
file, assuming that it is not possible to read backwards
on the tape? (Assume that it takes 60 seconds to rewind the tape.)
Compare this with the expected average time it would take for a sequential search for one record.
h. We implicitly assume in our discussions of tape performance that
the tape drive is always reading or writing at full speed, so no time is
lost by starting and stopping. This is not necessarily the case. For ex-
record in the
ample,
some
drives automatically stop after writing each block.
Suppose that the extra time it takes to start before reading a block
and to stop after reading the block totals 1 msec, and that the drive
must start before and stop after reading each block. How much will
the effective transmission rate be decreased due to starting and stop-
ping
if the
blocking factor
is
1?
What
if
it is
50?
15.
Why are there interblock gaps
just
jam
16.
The use of large blocks can lead to severe internal fragmentation of
on disks. Does this occur when tapes are used? Explain.
all
on
tapes? In other words,
why do we not
records into one block?
tracks
FURTHER READINGS
Many
textbooks contain more detailed information on the material covered
chapter. In the area of operating systems and
file
in this
management systems, we have
found the operating system texts by Deitel (1984), Peterson and Silberschatz (1985),
and Madnick and Donovan (1974) useful. Hanson (1982) has a great deal of material
on blocking and buffering, secondary storage devices, and performance. Flores's
book
(1973)
on peripheral devices may be
bit dated,
but
it
contains a
com-
prehensive treatment of the subject.
Bohl (1981) provides
DASDs. Chaney and Johnson
thorough treatment of mainframe-oriented IBM
good article on maximizing hard disk
(1984) wrote a
performance on small computers. Ritchie and Thompson (1974), Kermghan and
Ritchie (1978), Deitel (1984), and McKusick et al. (1984) provide information on
I/O is handled in the UNIX operating system. The latter provides a good
of ways in which a filesystem can be altered to provide substantially taster
throughput for certain applications. A comprehensive coverage of UNIX I/O from
the design perspective can be found in Leffler et al. (1989).
how
file
case study
91
92
SECONDARY STORAGE AND SYSTEM SOFTWARE
Information on specific systems and devices can often be found in manuals and
documentation published by manufacturers. (Unfortunately, information about
how software actually works is often proprietary and therefore not available.) If you
use a VAX, we recommend the manuals Introduction to the VAX Record Management
Services (Digital, 1978), VAX Software Handbook (Digital, 1982), and Peripherals
Handbook
(Digital,
Laboratories'
IBM PCs
useful.
1981).
UNIX users will find
useful to look at
UNIX I/O System by Dennis Ritchie (1979).
it
monograph The
will find the Disk Operating System (Microsoft,
1983 or
later)
the Bell
Users ot
manual
Fundamental File
Structure Concepts
CHAPTER OBJECTIVES
Introduce
file
structure concepts dealing with
Stream files;
and record boundaries;
Fixed-length and variable-length
Field
fields
and
records;
Search keys and canonical forms;
Sequential search;
Direct access; and
File access
and
file
organization.
Examine other kinds of file
structures in terms of
Abstract data models;
Metadata;
Object-oriented file access; and
Extensibility.
Examine
issues
of portability and standardization.
93
CHAPTER OUTLINE
4.1
Field
and Record Organization
4.4
4.1.1
4.5
Stream
File
File Access
Beyond Record
4.1.2 Field Structures
4.1.3 Reading a Stream of Fields
4.1.4 Record Structures
Record Structure That Uses a
4.1.5
Length Indicator
4.1.6 Mixing Numbers and
Characters: Use of a File Dump
Structures
Abstract Data Models
4.5.2
More Complex Headers
4.5.4 Color Raster Images
Mixing Object Types
in
One
File
4.5.6 Object-oriented File Access
4.5.7 Extensibility
4.2.1
Record Keys
4.2.2
A Sequential Search
UNIX Tools for Sequential
4.6
More about Record
Portability and Standardization
4.6.1
Factors Affecting Portability
4.6.2 Achieving Portability
Processing
4.2.4 Direct Access
4.3
Organization
4.5.1
4.5.5
Record Access
4.2.3
File
4.5.3 Metadata
4.2
and
Structures
Choosing a Record Structure
and Record Length
4.3.2 Header Records
4.3.1
4.1
Field
and Record Organization
When we build file structures we are imposing order on data. In this chapter
we investigate the many forms that this ordering can take. We begin by
looking
at the
base case:
4.1.1 A Stream
Suppose the
program
out as
file
organized
as a
are building contains
name and
names and addresses from
stream of consecutive bytes to
OUTPUT,
is
stream of bytes.
File
we
to accept
a file
described in the pseudocode
address information.
the keyboard, writing
file
shown
them
with the logical name
in Fig. 4.1.
Implementations of this program in both C and Pascal, called writstrm.c
and writstrm.pas, are provided in the C and Pascal Programs sections at the
end of this chapter. You should type in this program, working in either
C or Pascal, compile it, and run it. We use it as the basis for a number
of experiments, and you can get a better feel for the differences between
AND RECORD ORGANIZATION
FIELD
95
PROGRAM: writstrm
get output file name and open it with the logical name OUTPUT
get LAST name as input
while
LAST name has a length > 0)
get FIRST name, ADDRESS, CITY, STATE and ZIP as input
(
write
write
write
write
write
write
LAST
FIRST
ADDRESS
CITY
STATE
ZIP
to
to
to
to
to
to
the
the
the
the
the
the
file
file
file
file
file
file
OUTPUT
OUTPUT
OUTPUT
OUTPUT
OUTPUT
OUTPUT
get LAST name as input
endwhile
close OUTPUT
end PROGRAM
FIGURE 4.1
Program
the
file
to write out a
structures
we
name and address
are discussing if
file
as a stream of bytes.
you perform
the experiments your-
self
The following names and
John Ames
123 Maple
Stillwater, OK 74075
When we
AmesJohnl
list
Map 1 eS
23
The program
a
the output
i 1 1
file
addresses are used as input to the program:
Alan Mason
Eastgate
Ada, OK 74820
90
on our terminal
screen, here
wa t er 0K74 75MasonA lan90 Ea 5 t gat
writes the information out to the
file
specifications, the
there
is
We
no way
we
program
put
to get
all
it
creates a kind
what we
e Ada
see:
OK 7482
precisely as specified: as
stream of bytes containing no added information. But
problem. Once
is
of "reverse
in
meeting our
Humpty-Dumpty"
that information together as a single byte stream,
apart again.
of the fundamental organizational units of
our input data; these fundamental units are not the individual characters, but
meaningful aggregates of characters, such as "John Ames" or "123 Maple."
have
lost the integrity
96
FUNDAMENTAL
When we
fields.
STRUCTURE CONCEPTS
FILE
working with
arc
files,
we
these fundamental aggregates
call
field is the smallest logically meaningful unit
field
logical notion;
is
it
is
necessarily exist in any physical sense,
When we
structure.
write out our
of information
yet
name and
it
is
in a file.^
does not
important to the file's
conceptual tool.
field
address information as a stream
of undifferentiated bytes, we lose track of the fields that make the
information meaningful. We need to organize the file in some way that lets
us keep the information divided into fields.
4.1.2 Field Structures
There are many ways of adding structure to files
fields. Four of the most common methods are
Force the
of
fields into a predictable length.
Begin each
Place
to maintain the identity
field
with
a delimiter at
length indicator.
the end of each field to separate
from the next
it
field.
Use
"keyword = value" expression
to identify each field
and
its
contents.
Method
Fix the Length of Fields
1:
in their length. If
pull
we
them back out of the
field.
We
can define
Using
this
looks like that
file
shown
our sample
file
way
to the
or a record in Pascal to
vary
we
can
end of the
hold these
in Fig. 4.2.
kind of fixed-field length structure changes our output so
shown
in Fig. 4.3(a).
Simple arithmetic
recover the data in terms of the original
One
fields in
simply by counting our
a structure in
fixed-length fields, as
The
force the fields into predictable lengths, then
obvious disadvantage of
this
is
it
sufficient to let us
fields.
approach
is
that
adding
all
the
padding required to bring the fields up to a fixed length makes the file much
Rather than using 4 bytes to store the last name Ames, we use 10. We
can also encounter problems with data that is too long to fit into the
allocated amount of space. We could solve this second problem by fixing all
larger.
enough to cover all cases, but this would
problem of wasted space in the file even worse.
the fields at lengths that are large
just
make
"'"Readers
the
first
should not confuse the term field and
some programming
record
with the meanings given to them by
languages, including Pascal. In Pascal,
record
is
an aggregate data
where each member is referred to as
a field. As we shall see, there is often a direct correspondence between these definitions of
the terms and the fields and records that are used in files. However, the terms field and
record as we use them have much more general meanings than they do in Pascal.
structure that can contain
members of different
types,
FIELD
InC:
In Pascal:
struct {
char lastCIO];
char firstClO]
char addressCl5]
char city[15];
char stateC2]
char zipC9]
} set_of__fields;
TYPE
set _of_field; s = RECORD
last
packed array [1
first
packed array [1
address
city
state
zip
97
AND RECORD ORGANIZATION
packed
packed
packed
packed
array
array
array
array
[1
[1
[1
CI
of
of
of
of
of
of
10]
10]
15]
15]
2]
9]
char
char
char
char
char
char
END;
FIGURE 4.2 Fixed-length
fields.
Because of these
is
difficulties, the fixed-field
approach to structuring data
often inappropriate for data that inherently contain
of fields, such
variability in the length
kinds of data for which fixed-length
field
is
already fixed in length, or
lengths, using a
file
names and
fields are
if
there
is
a large
addresses.
amount of
But there
highly appropriate.
very
little
If
are
every
variation in field
structure consisting of a continuous stream of bytes
organized into fixed-length
Method 2: Begin Each
make
as
fields is often a
Field with a
very good solution.
Length Indicator
Another way
to
possible to count to the end of a field involves storing the field
it
length just ahead of the
too long (length
less
field, as illustrated in Fig. 4.3(b). If the fields are
than 256 bytes),
it is
not
possible to store the length in
single byte at the start of each field.
Separate the Fields with Delimiters We can also preserve
by separating them with delimiters. All we need to do
is choose some special character or sequence of characters that will not
appear within a field and then insert that delimiter into the file after writing
Method
3:
the identity of fields
each
field.
The choice of a
delimiter character can be very important since
it
must
way of processing. In many instances
white-space characters (blank, new line, tab) make excellent delimiters because
they provide a clean separation between fields when we list them on the
be
a character that
does not get in the
most programming languages include I/O statements
assume that fields are separated by white space.
Unfortunately, white space would be a poor choice for our file
console. Also,
by
that,
default,
blanks
often
occur
as
legitimate
characters
within
an
address
since
field.
98
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
Ames
John
123 Maple
Stillwater
OK74075377-1808
Mason
Alan
90 Eastgate
Ada
OK74820
(a) Field
lengths fixed. Place blanks in the spaces where the
phone number would
Ames John 123 Maple Stillwater OK 74075 377-1808
i
MasonlAlanl 90 Eastqate Ada OKI 74820
I
go.
(b) Delimiters are used to indicate the end of a field. Place the delimiter for the "empty" field
immediately after the delimiter for the previous field.
is
...
S-illwater|OK| 74075 377-1808 #Mason
I
90Eastgate Ada OK 74820
...
(c) Place the field for business phone at the end of the record.
encountered, assume that the field is missing.
SURNAME=Ames FIRSTNAME=John STREET=123 Maple
I
(d) Use a keyword to identify each
assumed to be missing.
field. If the
keyword
is
If the
...
end-of-record mark
is
ZIP = 74075 PHONE=3377-1 ono
missing, the corresponding field
is
FIGURE 4.3 Four methods for organizing fields within records to account for possible missing
the examples, the second record is missing the phone number.
fields. In
Therefore, instead of white space
delimiter, so our
file
we
use the vertical bar character as our
appears as in Fig. 4.3(c). Readers should jnodify the
st ream-of-bytes programs, writstrm.c and writstrm.pas (found in the
and Pascal Programs sections at the end of this chapter), changing them
so they place a delimiter after each field. We use this delimited field format
original
in the
next few sample programs.
Method
4:
Use
"Keyword = Value" Expression
to Identify Fields
This option, illustrated in Fig. 4.2(d), has an advantage that the others do
not: It is the first structure in which a field provides information about itself.
Such
self-describing structures
can be very useful tools for organizing
files
FIELD
in
AND RECORD ORGANIZATION
many applications. It is easy to tell what
if we don't know ahead of time what
even
contain.
It is
also a
fields are
contained in
fields the file is
99
a file,
supposed to
good format for dealing with missing fields. If a field is
makes it obvious, because the keyword is simply not
missing, this format
there.
You may have noticed in Fig. 4.3(d) that this format is used in
combination with another format, a delimiter to separate fields. While this
may not always be necessary, in this case it is helpful because it shows the
division between each value and the keyword for the following field.
Unfortunately, for the address file this format also wastes a lot of space.
Fifty percent or more of the file's space could be taken up by the keywords.
But there are applications in which this format does not demand so much
overhead. We discuss some of these applications in section 4.5.
4.1.3 Reading a Stream
of Fields
Given modified versions of writstrm.c and writstrm.pas that use delimiters to
we can write a program called readstrm that reads the stream
of bytes back in, breaking the stream into fields. It is convenient to conceive
of the program on two levels, as shown in the pseudocode description
provided in Fig. 4.4. The outer level of the program opens the file and then
calls the function readfield( ) until readfield( ) returns a field length of zero,
indicating that there are no more fields to read. The readfield( ) function, in
turn, works through the file, character by character, collecting characters
into a field until the function encounters a delimiter or the end of the file.
The function returns a count of the characters that are found in the field.
Implementations of readstrm in both C and Pascal are included with the
programs at the end of this chapter.
When this program is run using our delimited-field version of the file
containing data for John Ames and Alan Mason, the output looks like this:
separate fields,
Field
Field
Field
Field
Field
Field
Field
Field
Field
Field
Field
Field
lwat er
St
OK
74075
Mason
Alan
90 Eastgate
8
9
i 1
Ada
1
1
Ames
John
123 Maple
12
QK
74820
00
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
Define Constant: DELIMITER
'!'
readstrm
PROGRAM:
get input file name and open as INPUT
initialize FIELD_C0UNT
FIELD_LENGTH := readfield (INPUT, FIELD_C0NTENT
FIELD_LENGTH >
while
)
increment the FIELD_C0UNT
write FIELD.COUNT and FIELD_CONTENT to the screen
FIELD_LENGTH := readfield (INPUT, FIELD_C0NTENT
endwhile
close INPUT
end PROGRAM
FUNCTION:
readfield (INPUT, FIELD_C0NTENT
initialize I
initialize CH
while (not EOF (INPUT) and CH does not equal DELIMITER
read a character from INPUT into CH
increment I
FIELD_C0NTENT [I]
= CH
:
endwhile
return (length of field that was read)
end FUNCTION
FIGURE 4.4 Program to read fields from a
Clearly,
these data.
as a
we now
file
preserve the notion of
But something
stream of fields. In
is still
fact,
six fields are a set associated
are a set
records.
of
and display them on the screen.
fields associated
missing.
a field as
we
store
and retrieve
We do not really think of this
the fields need to be grouped into sets.
The
file
first
with someone named John Ames. The next six
with Alan Mason. We call these sets of fields
FIELD
AND RECORD ORGANIZATION
4.1.4 Record Structures
record
in
terms of a higher level of organization. Like the notion of a field, a record
can be defined as a
another conceptual tool.
set
It is
offields that belong together when the
another level of organization that
file is
viewed
is
we impose
on the data
in
to preserve meaning. Records do not necessarily exist in the file
any physical sense, yet they are an important logical notion included in
the
structure.
file's
Here
are
some of the most
often used methods for organizing a
file
into
records:
Require that the records be
Require that the records be
a predictable
a
predictable
number of bytes
number of fields
in length.
in length.
Begin each record with a length indicator consisting of a count of the
number of bytes that the record contains.
Use a second file to keep track of the beginning byte address for
each record.
Place a delimiter at the end of each record to separate
it
from the
next record.
1: Make Records a Predictable Number of Bytes (Fixedlength Records) A fixed-length record file is one in which each record
contains the same number of bytes. This method of recognizing records is
analogous to the first method we discussed for making fields recognizable.
Method
'
As we
will see in the chapters that follow, fixed-length record structures are
among the most commonly used methods for organizing files.
The C structure set_of_fields (or the Pascal RECORD of the same name)
that we define in our discussion of fixed-length fields is actually an example
of a fixed-length record as well as an example of fixed-length fields. We have
a fixed number of fields, each with a predetermined length, which combine
to make a fixed-length record. This kind of field and record structure is
illustrated in Fig. 4.5(a).
however, that fixing the number of bytes in
a record does not imply that the sizes or number of fields in the record must
be fixed. Fixed-length records are frequently used as containers "to hold
It is
important to
realize,
numbers of variable length fields. It is also possible to mix fixedand variable-length fields within a record. Figure 4.5(b) illustrates how
variable-length fields might be placed in a fixed-length record.
variable
Method
2:
Make Records
a Predictable
than specifying that each record in a
bytes,
we
good way
can specify that
it
file
Number
will contain a fixed
to organize the records in the
of Fields Rather
some fixed number of
number of fields. This is a
contain
name and
address
file
we have
been
102
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
Ames
John
123 Maple
Stillwater
0K74075
Mason
Alan
90 Eastgate
Ada
0K74820
(a)
Ames John
;
123 Maple
Stillwater OK 74075
Unused space
(b)
Ames John 123 Maple Stillwater OK 74075 Mason Alan 90 Eastgate Ada OK
|
(c)
FIGURE 4.5 Three ways of making the lengths of records constant and predictable, (a) Counting
bytes: fixed-length records with fixed-length fields, (b) Counting bytes: fixed-length records with
variable-length fields, (c) Counting fields: six fields per record.
looking
The
at.
writstrm
program
asks for six pieces of information for
every person, so there are six contiguous
(Fig. 4.5c).
fields in the file for
each record
We could modify readstrm to recognize fields simply by counting
the fields modulo six, outputting record
every time the count
boundary information
to the screen
starts over.
3: Begin Each Record with a Length Indicator
We can
communicate the length of records by beginning each record with a field
Method
containing an integer that indicates
the record (Fig.
This
4.6a).
variable-length records.
Method
4:
We
is
look
how many bytes there are in the rest of
commonly used method for handling
at
it
more
closely in the next section.
We
Use an Index to Keep Track of Addresses
index to keep a byte offset for each record in the original
offsets
us
the length of each record.
We
index and then seek to the record
illustrates this two-file
Method
at a
can use an
The byte
allow us to find the beginning of each successive record and also
compute
in the
file.
5:
is
let
record
in the data file.
Figure 4.6(b)
End of Each Record
This option,
mechanism.
Place a Delimiter at the
record level,
look up the position of
exactly analogous to the solution
we
used to keep the
FIELD
103
AND RECORD ORGANIZATION
sample program we developed. As with fields, the
must not get in the way of processing. Because we often
fields distinct in the
delimiter character
want
our console,
to read files directly at
delimiter for
that contain readable text
files
(carriage return/new-line pair or,
acter
common
'\n'). In Fig 4.6(c)
we
UNIX
on
choice of a record
the end-of-line character
is
systems, just a new-line char-
use a '#' character as the record delimiter.
4.1.5 A Record Structure That Uses a Length Indicator
Not one of
these approaches to preserving the idea of a record in a
file is
situations. Selection of a method for record organization
depends on the nature of the data and on what you need to do with it. We
begin by looking at a record structure that uses a record-length field at the
beginning of the record. This approach lets us preserve the variability in the
length of records that is inherent in our initial stream file.
appropriate for
all
Writing the Variable-length Records to the File We call the program
that builds this new, variable-length record structure writrec. The set of
programs at the end of this chapter contains versions of this program in C
and Pascal. Implementing this program is partially a matter of building on
FIGURE 4.6 Record structures for variable-length records, (a) Beginning each record with a length
indicator, (b) Using an index file to keep track of record addresses, (c) Placing the delimiter '#' at
the end of each record.
40kmes John 123 Maple Stillwater OK 74075 J6Mason Alan 90 Eastgate
\
(a)
Data
Ames John 123 Maple Stillwater OK 74075 Mason Alan
file:
T
Index
file:
00
40
(b)
Ames John 123 Maple Stillwater 0K 74075 #Mason Alan 90 Eastgate Ada 0K
;
(c)
04
FUNDAMENTAL
the writstrm
addressing
If
FILE
STRUCTURE CONCEPTS
program that we created
some new problems:
we want
earlier in this chapter,
to put a length indicator at the beginning
(before any other fields),
the fields in each record
but also involves
of every record
we must know the sum of the lengths
before we can begin writing the record
of
to
We
need to accumulate the entire contents of a record in a
bu ffcr before writing it out
In what form should we write the record-length field to the file? As
a binary integer? As a series of ASCII characters?
the
file.
The concept of buffering
with
files.
into
which we place the
is
one
we
run into again and again
In the case ofwritrec, the buffer can
fields
and
simply be
field delimiters as
as
we work
character array
we
collect
them.
Resetting the buffer length to zero and adding information to the buffer can
be handled using the loop logic provided in Fig.
Representing the Record Length
record length
is
a little
length in the form of
a natural
more
4.7.
The question of how to represent
One option would be to write
difficult.
much
two-byte binary integer before each record. This is
it does not require us to go to the trouble of
bigger numbers with an integer than
number of ASCII
by.tes
(e.g.,
32,767 versus 99).
FIGURE 4.7 Main program logic for writrec.
get LAST name as input
while
LAST name has a length >
set length of string in BUFFER to zero
concatenate: BUFFER + LAST name + DELIMITER
)
while
input fields exist for record
get the FIELD
(
concatenate: BUFFER
endwhile
FIELD
DELIMITER
write length of string in BUFFER to the file
write the string in BUFFER to the file
get LAST name as input
endwhile
the
solution in C, since
converting the record length into character form. Furthermore,
represent
the
we
It
is
we
can
can with the same
also conceptually
FIELD
AND RECORD ORGANIZATION
since it illustrates the use of a fixed-length, binary
combination with variable-length character fields.
interesting,
Although we could use
we might
between
choose,
and
instead,
this
same solution
to account for
field
05
in
for a Pascal implementation,
some important
differences
Pascal:
Unlike C, Pascal automatically converts binary integers into characof those integers if we are writing to a text file.
Consequently, it is no trouble at all to convert the record length into
a character form: It happens automatically.
In Pascal, a file is defined as a sequence of elements of a single type.
Since we have a file of variable-length strings of characters, the natural type for the file is that of a character.
ter representations
do in C is to store the integers in the file
two-byte fields containing integers. In Pascal it is easier to
make use of the automatic conversion of integers into characters for text
files. File structure design is always an exercise in flexibility. Neither of
these approaches is correct; good design consists of choosing the approach
that is most appropriate for a given language and computing environment. In
the programs included at the end of this chapter, we have implemented our
record structure both ways, using integer-length fields in C and character
representations in Pascal. The output from the Pascal implementation is
shown in Fig. 4.8. Each record now has a record-length field preceding the
data fields. This field is delimited by a blank. For example, the first record
(for John Ames) contains 40 characters, counting from the first 'A' in
"Ames" to the final delimiter after "74075," so the characters '4' and '0' are
placed before the record, followed by a blank.
Since the C version of writrec uses binary integers for the record length,
we cannot simply print it to a console screen. We need a way to interpret the
noncharacter portion of the file. For this, we introduce in the next section
the file dump, a valuable tool for viewing the contents of files. But first,
let's look at a program that will read in any file that is written by writrec.
In short, the easiest thing to
as fixed-length,
Reading the Variable-length Records from the
structure of variable-length records preceded
FIGURE 4.8 Records preceded by record-length fields
in
File
Given our file
by record-length fields, it is
character form.
123 Maple Stillwater OK 74075 36 Mason Alan 90
Eastgate Ada OK 74820
40 Ames John
\
06
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
readrec
PROGRAM:
open input file as INP_FILE
initialize SCAN_P0S to
RECORD_LENGTH := get_rec INP_FILE, BUFFER)
while (RECORD_LENGTH > 0)
SCAN_P0S := get_fld(FIELD, BUFFER, SCAN_POS,RECORD_LENGTH)
while (SCAN_P0S > 0)
print FIELD on the SCREEN
SCAN_P0S := get_fld( FIELD, BUFFER, SCAN_POS,RECORD_LENGTH)
endwhile
(
RECORD_LENGTH
endwhile
end PROGRAM
FUNCTION:
:=
get_rec INP_FILE, BUFFER)
(
get_rec INP_FILE, BUFFER)
(
if EOF (INP_FILE)
then return
read the RECORD_LENGTH
read the record contents into the BUFFER
return the RECORD_LENGTH
end FUNCTION
FUNCTION
ge t_f Id FIELD BUFFER, SCAN_P0S RECORD_LENGTH
(
if SCAN_P0S == RECORD.LENGTH then return
get a character CH at the SCAN_P0S in the BUFFER
while (SCAN_P0S < RECORD_LENGTH and CH is not a DELIMITER)
place CH into the FIELD
increment the SCAN_P0S
get a character CH at the SCAN_P0S in the BUFFER
endwhile
return the SCAN_P0S
end FUNCTION
FIGURE 4.9 Main program logic
for readrec,
along with functions get_rec(
and get_fld(
).
FIELD
easy to write
program
that reads
through the
file,
record by record,
The program
displaying the fields from each of the records on the screen.
logic
shown
is
in Fig. 4.9.
The main program
calls
the function get_rec(
that reads records into a buffer; this call continues until get_rec(
Once get_rec(
value of 0.
is
(SCAN_POS)
get_Jld(
or
the
in the
reads characters
returns a
places a record's contents into a buffer, the buffer
passed to the function get_fld(
position
107
AND RECORD ORGANIZATION
from
end of the record
).
The
call to get_jld( )
argument
list.
includes
scanning
SCAN_POS,
Starting at the
the buffer into a field until either a delimiter
is
reached.
Function get_Jld(
returns
the
SCAN_POS for use on the next call. Implementations ofwritrec and readrec
in both C and Pascal are included along with the other programs at the end
of
this chapter.
4.1.6 Mixing Numbers and Characters: Use
File
dumps
give us the ability to look inside
of a File
Dump
a file at the actual
bytes that are
stored there. Consider, for instance, the record-length information in the
program output that we were examining a moment ago. The length
Ames record, which is the first one in the file, is 40 characters,
including delimiters. In the Pascal version of writrec, where we store the
ASCII character representation of this decimal number, the actual bytes
Pascal
of the
stored
in
the file
implementation,
look
like the representation in Fig.
where we choose
to
4.10(a).
In the
represent the length field
as
C
a
two-byte integer, the bytes look like the representation in Fig. 4.10(b).
As you can see the number 40 is not the same as the set of characters '4'
and '0'. The hex value of the binary integer 40 is 0x28; the hex values of the
x 30. (We are using the C language
characters '4' and '0' are
x 34 and
convention of identifying hexadecimal numbers through the use of the
prefix Ox.) So, when we are storing a number in ASCII form, it is the hex
FIGURE 4.10 The number 40, stored as ASCII characters and as a short integer.
Decimal value
of number
Hex value
stored
ASCII
character form
in bytes
(a)
40 stored as ASCII chars:
40
34
30
'4'
'0'
(b)
40 stored as a 2-byte integer:
40
00
28
'\0'
"('
08
FUNDAMENTAL
ASCII
values of the
number
STRUCTURE CONCEPTS
FILE
characters that
go into the
file,
not the hex value of the
itself.
Figure
4.
shows
10(b)
an integer (this
is
the byte representation of the
called storing the
number
number 40
stored as
form, even though
in binary
Now
usually view the output as a hexadecimal number).
we
the hexadecimal
file is that of the number itself. The ASCII characters that
happen to be associated with the number's actual hexadecimal value have no
obvious relationship to the number. Here is what the version of the file that
uses binary integers for record lengths looks like if we simply print it on a
value stored in the
terminal screen:
(Ames
tt_
^0x28
John
is
Blank, since
123 Maple
code for
ascii
'\0' is
Stillwater
OK
74075
$Mason Alan
tf_
^ 0x28
'('
unprintable.
...
'*'
ascii
is
Blank:
'\0' is
code for
unprintable.
The ASCII representations of characters and numbers in the actual record
come out nicely enough, but the binary representations of the length fields
are displayed cryptically. Let's take a different look at the
using the
od
UNIX dump
-xc
< f
i 1
this
file,
time
UNIX command
Entering the
utility od.
ename>
produces the following:
Values
Offset
0000000
\0
3037
^ASCII
^Hex
\0
0024
7374
4561
3020
7
736f
6761
2
I
6461
4d61
7c39
6e
6572
357c
61
7c41
6174
6c77
696c
5374
3734
a
416c
7465
3320
3132
6e7c
657c
6f68
7c4a
S
4b7c
6e7c
0000100
7c4f
0000060
6573
706c
4d61
0000040
416d
0028
0000020
7c4f
4b7c
3734
3832
307c
As you can see, the display is divided into three different kinds of data. The
column on the left labeled Offset gives the offset of the first byte of the row
that is being displayed. The byte offsets are given in octal form; since each
line contains 16 (decimal) bytes, moving from one line to the next adds 020
to the range. Every pair of lines in the printout contains interpretations of
the bytes in the file in hexadecimal and ASCII. These representations were
requested on the command line with the -xc flag (x = "hex;" c =
"character").
Let's look at the first
row of ASCII
values.
As you would
'('
expect, the
RECORD ACCESS
09
ASCII form appears in this row in a readable way.
which there is no printable ASCII
x 00. But there
representation. The only such value appearing in this file is
could be many others. For example, the hexadecimal value of the number
500,000,000 is 0xlDCD6500. If you write this value out to a file, an od of
data placed in the
But there
the
file
file in
are hexadecimal values for
with the option -xc looks
0000000 \035\315
like this:
\0
1dcd 6500
The only
handles
printable byte in this
of the others by
all
file is
the one with the value 0x65 (V).
listing their equivalent octal values in the
Od
ASCII
representation.
The hex dump of this output from
this
file
we have
version of writrec shows
an interesting mix of
represents
structure
organizational tools
the
encountered. In
a single
number of
record
binary and ASCII data. Each record consists of a fixed-length
count) and several delimited, variable-length
different data types
file
fields.
and organizational methods
is
how
the
we have both
field (the
byte
This kind of mixing of
common
in real-world
structures.
A Note about Byte Order
or a computer from
DEC,
computer you are using is an IBM PC
VAX, your octal dump for this file will
one we see here. These machines store the
If the
such
as a
probably be different from the
values of numbers in reverse order from the
example,
if this
dump were
executed on an
way we
IBM PC,
think of them. For
the hex representation
of the first two-byte value in the file would be 0x2800, rather than 0x0028.
This reverse order also applies to long, four-byte integers on these
machines. This is an aspect of files that you need to be aware of if you expect
to make sense out of dumps like this one. A more serious consequence of
the byte-order differences among machines occurs when we move files
from a machine with one type of byte ordering to one with a different byte
ordering. We discuss this problem and ways to deal with it in section 4.6,
"Portability and Standardization."
4.2
Record Access
4.2.1 Record Keys
Since our
new
file
structure so clearly focuses
the quantity of information that
is
on the notion of a record
it makes sense
being read or written,
as
to
think in terms of retrieving just one specific record rather than having to
read
all
the
way through
the
file,
displaying everything.
When
looking for
1 1
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
an individual record,
convenient to identify the record with
it is
a key
based
on the record's contents. For example, in our name and address file we
might want to access the "Ames record" or the "Mason record" rather than
thinking in terms of the "first record" or "second record." (Can you
remember which record comes first?) This notion of a ke_y is another
fundamental conceptual tool. We need to develop a more exact idea of what
a key is.
When we
want
are looking for a record containing the last
name Ames, we
form "AMES",
"ames", or "Ames". To do this, we must define a standard form for keys,
along with associated rules and procedures for converting keys into this
standard form. A standard form of this kind is often called a canonical form
for the key. One meaning of the word canon is rule, and the word canonical
means conforming to the rule. A canonical form for a search key is the single
representation for that key that conforms to the rule.
As a simple example, we could state that the canonical form for a key
requires that the key consist solely of uppercase letters and have no extra
to recognize
blanks
it
even
end. So, if a user enters
at the
to the canonical
form
"AMES"
"Ames", we would convert
before searching for
,
record. If there
a single record,
is
not
one-to-one relationship between the key and
then the program has to provide additional mechanisms to
allow the user to resolve the confusion that can result
record
fits a
when more than one
we are looking for
particular "key. Suppose, for example, that
John Ames's
address. If there are several records in the
different people
named John Ames, how should
finds.
The
simplest solution
is
file.
When
for several
it
provide
way of
The prevention
takes
the user enters a
new
to prevent such confusion.
new records are added to the
we form a unique canonical key
file
program respond?
first John Ames that it
the
it should not just give the address of the
Should it give all the addresses at once? Should
scrolling through the records?
Certainly
place as
the key
it.
often desirable to have ^isthutke^s or keys that uniquely identify
It is
a single
the user enters the key in the
if
and then search the
file for that key. This concern about uniqueness applies only to primary keys.
A primary key is, by definition, the key that is used to identify a record
record,
for that record
uniquely.
An
It is also possible, as we see later, to search on s econdary keys
example of a secondary key might be the city field in our name and address
file. If we wanted to find all the records in the file for people who live in
towns named Stillwater, we would use some canonical form of "Stillwater"
as a secondary key. Typically, secondary keys do not uniquely identify a
.
record.
Although a person's name might at first seem to be a good choice for
primary key, a person's name runs a high risk of failing the test for
RECORD ACCESS
uniqueness.
A name is
a perfectly fine
important secondary key
likelihood that
The reason
in a retrieval
two names
a
name
is
in the
a risky
a real data value. In general,
think
we
are choosing a
secondary key, and
same
file
choice for
if
it
is
often an
too great
is
will be identical.
a
primary key
primary keys should be
unique key,
in fact
system, but there
1 1 1
is
dataless.
contains data there
that
it
contains
Even when we
is
danger that
unforeseen identical values could occur. Sweet (1985) cites an example of a
file system that used a person's Social Security number as a primary key for
turned out
personnel records.
It
represented in the
file,
a large
that, in the particular
population that was
number of people who were not U.S.
citizens
were included, and in a different part of the organization all of these people
had been assigned the Social Security number 999-99-9999!
Another reason, other than uniqueness, that a primary key should be
dataless is that a primary key should be unchanguw. If information that
a certain record changes, and that information is contained
primary key, what do you do about the primary key? You probably
cannot change the primary key itself, in most cases, because there are likely
to be reports, memos, indexes, or other sources of information that refer to
the record by its primary key. As soon as you change the key, those
corresponds to
in a
become useless.
A good rule of thumb is to avoid trying to put data into primary keys.
If we want to access records according to data content, we should assign this
content to secondary keys. We give a more detailed look at record access by
references
primary and secondary keys in Chapter 6. For the rest of this chapter, we
suspend our concern about whether a key is primary or secondary and
concentrate simply on finding things by key.
4.2.2 A Sequential Search
Now that you know about keys,
reads through the
file,
you should be
able to write a
record by record, looking for
program
that
record with
Such sequential searching is just a simple extension of our
readrec program, adding a comparison operation to the main loop to see if
the key for the record matches the key we are seeking. We leave the actual
program as an exercise.
particular key.
Evaluating Performance of Sequential Search In the chapters that
follow, we find ways to search for records that are faster than the sequential
search mechanism. We can use sequential searching as a kind of baseline
against which to measure the improvements that we make. It is important,
therefore, to find some way of expressing the amount of time and work
expended in a sequential search.
1 1
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
performance measure requires that we decide on a unit of
on the performance of the
whole process. When we describe the performance of searches that take
place in electronic RAM, where comparison operations are more expensive
Developing
work
that usefully represents the constraints
memory, we
than fetch operations to bring data in from
usually use the
number of comparisons required for the search as the measure of work. But,
given that the cost of a comparison in
is so small compared to the cost
of a disk access, comparisons do not fairly represent the performance
RAM
constraints for a search through a
count low-level
READ(
file
storage. Instead,
calls.
we
seek and that any one
from the discussions of matters such
a
on secondary
We assume that each READ( call requires
READ( call is as costly as any other. We know
as
system buffering
in
Chapter 3 that
these assumptions are not strictly accurate. But, in a multiuser environment
where many processes
enough
are using the disk at once, they are close
to
correct to be useful.
Suppose
we have
file
with 1,000 records and
sequential search to find Al Smith's record.
required? If Al Smith's record
read in only
makes 1,000
is
a single record. If
READ(
calls
the
it is
first
we want
How many READ(
one
in the
file,
to use a
)
calls are
program has to
file, the program
the
the last record in the
before concluding the search. For an average
search, 500 calls are needed.
we
If
double the number of records
average and the
maximum number
of
in a
we
file,
READ(
sequential search to find Al Smith's record in a
requires,
on the average, 1,000
required for a sequential search
records in the
calls.
is
double both the
also
required.
calls
Using
of 2,000 records,
In other words, the amount of work
file
directly proportional to the
number of
file.
In general, the
with n records
is
work
required to search sequentially for
proportional to n;
it
takes at
most
record in
a file
comparisons; on
//
it takes approximately nil comparisons. A sequential search is said
of the order O(n) because the time it takes is proportional to w.*
average
to be
Improving Sequential Search Performance with Record Blocking
It is interesting and useful to apply some of the information from Chapter
3 about disk performance to the problem of improving sequential search
We learned in
Chapter 3 that the major cost associated with a
on the
disk. Once data transfer begins, it is relatively fast, although still much
slower than a data transfer within RAM. Consequently, the cost of seeking
performance.
disk access
""If
is a
is
the time required to perform a seek to the right location
you are not familiar with
good source.
this
"big-oh" notation, you should look
it
up.
Knuth
(1973a)
RECORD ACCESS
and reading
record and then seeking and reading another record
than the cost of seeking just once and then reading
(Once
we
again,
are
assuming
required for each separate
to
two
call.) It
follows that
we
all
at
greater
seek
is
should be able
improve the performance of sequential searching by reading
several records
is
successive records.
multiuser environment in which
READ(
1 1
in a block
of
once and then processing that block of records
in
RAM.
We
began this chapter with a stream of bytes. We grouped the bytes
and then grouped the fields into records. Now we arc
considering a yet higher level of organization, grouping records into blocks.
This new level of grouping, however, differs from the others. Whereas
fields and records are ways of maintaining the logical organization within
the file, blocking is done strictly as a performance measure. As such, the
into
fields,
block
size
usually related
is
more
to the physical properties
of the disk drive
than to the content of the data. For instance, on sector-oriented disks the
block size
is
almost always some multiple of the sector
Suppose we have
size.
of 4,000 records and that the average length of
record
is
512
If
our
operating
system uses sector-sized buffers of 512
a
bytes.
bytes, then an unblocked sequential search requires, on the average, 2,000
READ( ) calls before it can retrieve a particular record. By blocking the
records in groups of 16 per block, so each READ( ) call brings in 8 kilobytes
worth of records, the number of reads required for an average search comes
down
a file
Each READ( requires slightly more time, since more data is
from the disk, but this is a cost that is usually well worth paying
to 125.
transferred
for such a large reduction in the
There
number of reads.
are several things to note
from
this analysis
and discussion of
record blocking:
Although blocking can
ments,
tion.
it
The
result in substantial
performance improve-
does not change the order of the sequential search operacost of searching
is still
tion to increases in the size of the
O(n), increasing in direct proporfile.
RAM
Blocking clearly reflects the differences between
access speed
and the cost of accessing secondary storage.
Blocking does not change the number of comparisons that must be
RAM, and it probably increases the amount of data transbetween disk and RAM. (We always read a whole block, even
if the record we are seeking is the first one in the block.)
Blocking saves time because it decreases the amount of seeking. We
find, again and again, that this differential between the cost of seeking and the cost of other operations, such as data transfer or RAM
done
in
ferred
access,
is
the force that drives
file
structure design.
FUNDAMENTAL
1 1
FILE
STRUCTURE CONCEPTS
When
Sequential Searching Is Good Much of the remainder of this text
devoted to identifying better ways to access individual records; sequential
searching is just too expensive for most serious retrieval situations. This is
is
unfortunate, because sequential access has
over other types of access:
the simplest of
file
It is
two major
practical advantages
extremely easy to program, and
it
requires
structures.
Whether sequential search is advisable depends largely on how the file
be used, how fast the computer system is that is performing the search,
and structural aspects of the file. There are many situations in which a
sequential search is often reasonable. Here are some examples:
is
to
ASCII
files in
which you
are searching for
some
pattern (see grep in
the next section);
Files
with few records
Files that
(e.g.,
10 records);
hardly ever need to be searched
(e.g.,
tape
files
usually
used for other kinds of processing); and
which you want all records with a certain secondary key
where a large number of matches is expected.
Files in
value,
Fortunately, these sorts of applications do occur often in day-to-day
computing
for
so
often, in fact, that operating systems provide
performing sequential processing.
this, as
we
UNIX is
many
utilities
one of the best examples of
see in the next section.
4.2.3 UNIX Tools
for
Sequential Processing
Recognizing the importance of having a standard file structure that is simple
and easy to program, the most common file structure that occurs in UNIX
is an ASCII file with the new-line character as the record delimiter and, when
-r-A%
possible, white space as the field delimiter. Practically all files that
using
we
create
UNIX editors use this structure. And since most of the built-in C and
Pascal functions that perform I/O write to this kind of file,
it is
common
to
numbers or words separated by blanks
or tabs, and records separated by new-line characters. Such files are simple
and easy to process. We can, for instance, generate an ASCII file with a
simple program, and then use an editor to browse through it or alter it.
UNIX provides a rich array of tools for working with files in this form.
see data files that consist of fields of
Since this kind of file structure
is
inherently sequential (records are variable
we have to pass from record to record to find any particular
field or record), many of these tools process files sequentially.
Suppose, for instance, that we choose the white-space/new-line
in length, so
file, ending every field with a tab and ending every
While this causes some problems in distinguishing
white space, but it doesn't separate a field), and in that sense
structure for our address
record with
fields (a
blank
new
is
line.
RECORD ACCESS
is
UNIX
For example,
such
utilities,
cat
buys us something very valuable: the full use of
around the white-space/new-line structure.
can print the file on our console using any of a number of
not an ideal structure,
those
1 1
it
tools that are built
we
as
>cat myf i 1
Stillwater DK 74075
Ames
John 123 Maple
Alan 90 Eastgate Ada
DK 74820
Mason
Or we
can use tools like
wc and
grep for processing the
files.
wc The command wc
("word count") reads through an ASCII file
number of lines (delimited by new lines), words
(delimited by white space), and characters in a file:
sequentially and counts the
>wc myf
i 1
grep
It is
common
character string in
sequentially,
(and
it.
want to find out
For ASCII files
recognize. In
a pattern,
its
if a text file
has a certain
word
or
that can reasonably be searched
provides an excellent
variants egrep and fgrep).
its
file
to
UNIX
regular expression,"
the
76
14
filter
The word
for
doing
this called grep
grep stands for "generalized
which describes the type of pattern
that grep
is
simplest form, grep searches sequentially through a
and then returns to standard output
(the console)
all
able to
file
for
the lines in
that contain the pattern.
>grep Ada myf
Mason
Alan
i 1
90 Eastgate Ada
We can also combine tools to create,
on the
OK
>grep Ada
some very powerful file
number of words in all
fly,
processing software. For example, to find the
records containing the
74820
word Ada:
wc
36
As we move through the text we will encounter
powerful UNIX commands that sequentially process
white-space/new-line structure.
number of
files
other
with the basic
4.2.4 Direct Access
The most
record
is
through a file for a
We have direct access
can seek directly to the beginning of the record and
radical alternative to searching sequentially
a retrieval
to a record
mechanism known
when we
as direct accesj.
1 1
FUNDAMENTAL
read
is
STRUCTURE CONCEPTS
Whereas sequential searching is an O(n) operation, direct access
no matter how large the file is, we can still get to the record we want
in.
it
O(l);
with
FILE
a single seek.
Direct access
required record
predicated on
is
is.
Sometimes
carried in a separate index
not have an index.
We
file.
knowing where
the beginning of the
information about record location is
this
But, for the
moment, we assume
assume, instead, that
we know
we do
that
the relative record
(RRN ) of the record that we want. The idea of an RRN is an
important concept that emerges from viewing a file as a collection of
records rather than a collection of bytes. If a file is a sequence of records,
then the RRN of a record gives its position relative to the beginning of the
number
file.
The
first
record in
a file
has
RRN 0,
the next has
"
RRN
1,
and so
forth."
file, we might tie a record to its RRN by
membership numbers that are related to the order in which we
enter the records in the file. The person with the first record might have a
membership number of 1001, the second a number of 1002, and so on.
Given a membership number, we can subtract 1001 to get the RRN of the
In
our name and address
assigning
record.
What can we do with
have been using so
tells
far,
this
which
RRN? Not much,
us the relative position of the record
records, but
records as
we
we
still
given the
structures
file
The
consist of variable-length records.
we want
we want. An exercise
go, to get to the record
sequence of
in the
have to read sequentially through the
at
we
RRN
file,
counting
the end of this
a method of moving through the file called skip sequential
which can improve performance somewhat, but looking for a
particular RRN is still an O(n) process.
To support direct access by RRN, we need to work with records of
fixed, known length. If the records are all the same length, then we can use
chapter explores
processing,
a record's
RRN
to the start
an
RRN
record,
to calculate the byte offset
of the
file.
For instance,
of 546 and our
we
file
if
we
of the
as follows:
= 546 x
128
In general, given a fixed-length record
file
byte offset of
offset
record with an
RRN
Byte
offset
of
//
69,888.
where the record
t In
this
byte offset calculation
keeping with the conventions of
:av-lhised count. In
some
file
is
size
n X
v,
the
r.
differ
with regard to
done and even with regard
and Turbo
is
is
Programming languages and operating systems
where
of the record relative
has a fixed-length record size of 128 bytes per
can calculate the byte offset
Byte
start
are interested in the record with
Pascal,
systems, the count starts
at
we assume
1
that the
rather than
0.
to
whether
RRN
is
MORE ABOUT RECORD STRUCTURES
byte offsets are used for addressing within
MS-DOS
operating systems), where
bytes, the application
command
a file
to
in
is
jump
files.
In
the calculation and uses the lseek(
to the byte that begins the record. All
terms of bytes. This
responsibility for translating an
and
sequence of
a file is treated as just a
program does
UNIX
(and the
1 1
very low-level view of
is
RRN
movement
the
files;
wholly
into a byte offset belongs
within
to
the application program.
The PL/I language and the operating environments in which PL/I is
(OS/MVS, VMS) are examples of a much different, higher-level
view of files. The notion of a sequence of bytes is simply not present when
you are working with record-oriented files in this environment. Instead,
files are viewed as collections of records that are accessed by keys. The
often used
operating system takes care of the translation between a key and
a record's
key is, in fact, just the record's RRN, but
of actual location within the file is still not the
location. In the simplest case, the
the
determination
programmer's concern.
If
we
at all in
of
no seeking
limit ourselves to the use of standard Pascal, the question
seeking by bytes or seeking by records
we
standard Pascal. But, as
is
not an issue: There
said earlier,
is
many implementations of
Pascal extend the standard definition of the language to allow direct access
to
different
locations in a
file.
The nature of
these extensions
varies
according to the differences in the host operating systems around which the
extensions were developed. All the same, one feature that
across implementations
is
that a
in Pascal
file
is
consistent
always consists of elements of
A file is a sequence of integers, characters, arrays, or records,
and so on. Addressing is always in terms of this fundamental element size.
For example, we might have a file of datarec, where datarec is defined as
a single type.
TYPE datarec = packed array [0..64] of char;
Seeking within
datarec,
which
datarec
number
into the
4.3
is
this file
is
in
terms of multiples of the elementary unit
to say in multiples
3 (zero-based count),
of
I
65-byte entity.
am jumping
If
195 bytes
ask to
(3
jump
X 65 =
to
195)
file.
More about Record Structures
4.3.1 Choosing a Record Structure and Record Length
Once we
decide to fix the length of our records so
give us direct access to
Clearly, this decision
is
a record,
we
we
can use the
RRN
to
have to decide on a record length.
related to the size
of the
fields
we want
to store in
1 1
STRUCTURE CONCEPTS
FUNDAMENTAL
FILE
the record.
Sometimes the decision
of
is
easy.
Suppose we
are building a
file
transactions that contain the following information about each
sales
transaction:
six-digit account
number of the
purchaser;
Six digits for the date field;
A
A
A
five-character stock
number
for item purchased;
three-digit field for quantity; and
10-position field for total cost.
These are all fixed-length fields; the sum of the field lengths is 30 bytes.
Normally, we would simply stick with this record size, but if performance
is so important that we need to squeeze every bit of speed out of our
retrieval system, we might try to fit the record size to the block
organization of our disk. For instance, if we intend to store the records on
a typical sectored disk (see Chapter 3) with a sector size of 512 bytes or some
other power of 2, we might decide to pad the record out to 32 bytes so we
can place an integral number of records in a sector. That way, records will
never span sectors.
The choice of a record length is more complicated when the lengths of
the fields can vary, as in our name and address file. If we choose a record
length that is the sum of our estimates of the largest possible values for all
the fields, we can be reasonably sure that we have enough space for
everything, but we also waste a lot of space. If, on the other hand, we are
conservative in our use of space and fix the lengths of fields at smaller
values, we may have to leave information out of a field. Fortunately, we can
avoid this problem to some degree through appropriate design of the field
structure within a record.
In our earlier discussion
of record structures, we saw that there are two
take toward organizing fields within a
approaches
we
can
fixed-length record.
The
first,
general
illustrated in Fig. 4.11(a), uses fixed-length
fields inside the fixed-length record.
sales transaction file
in Fig.
This
previously described.
is
the approach
we
took for the
The second approach,
illustrated
4.11(b), uses the fixed-length record as a kind of standard-sized
container for holding something that looks like
The
first
approach has the virtue of simplicity:
out" the fixed-length
fields
from within
variable-length record.
It is
very easy to "break
fixed-length record.
The second
approach lets us take advantage of an averaging-out effect that usually
occurs: The longest names are not likely to appear in the same record as the
longest address
field.
By
letting the field boundaries vary,
we
make
the two
can
more efficient use of a fixed amount of space. Also, note that
approaches are not mutually exclusive. Given a record that contains a
number of truly fixed-length fields and some fields that have variable-
119
MORE ABOUT RECORD STRUCTURES
Ames
John
123 Maple
Stillwater
0K74075
Mason
Alan
90 Eastgate
Ada
0K74820
(a)
Ames John 123 Maple Stillwater OK 74075
j
Mason Alan 90 Eastgate Ada OK 74820
;
Unused
space-
Unused space
(b)
FIGURE 4.1
Two fundamental approaches
to field structure within a fixed-
length record, (a) Fixed-length records with fixed-length fields, (b) Fixed-length
records with variable-length fields.
length information,
two approaches.
The programs
programs
at
and
which
update. pas,
are included in the set
change
it,
and then write
it
of
user to
back. These programs create
structure that uses variable-length fields within fixed-length records.
Given the
this
update. c
design a record structure that combines these
the end of this chapter, use direct access to allow
retrieve a record,
a file
we might
is
variability in the length
of the
fields in
our name and address
file,
an appropriate choice.
One of the
must be resolved in the design of
of distinguishing the real-data portion of the
record from the unused-space portion. The range of possible solutions
parallels that of the solutions for recognizing variable-length records in any
other context: We can place a record-length count at the beginning of the
record, we can use a special delimiter at the end of the record, we can count
fields, and so on. Because both update. c and update. pas use a character string
buffer to collect the fields, and because we are handling character strings
differently in C than in Pascal (strings are null-terminated in C; we keep a
byte count of the string length at the beginning of the Pascal strings), it is
this
interesting questions that
kind of structure
convenient to use
tations. In the
is
that
a slightly different file structure for
version
we
fill
null characters. In the Pascal version
we
(an integer) at the start of the record to
record are valid. As usual, there
structure; instead
and
situation.
we
the
two implemen-
out the unused portion of the record with
is
no
actually place a fixed-length field
tell
how many
is
in the
way to implement this file
most appropriate for our needs
single right
seek the solution that
of the bytes
20
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
dump
output from each of these programs.
number of
other ideas, such as the use of header
Figure 4.12 shows the hex
The output
records,
introduces
which we discuss
For now, however, just look
in the next section.
of the data records. We have italicized the length fields at the
of the records in the output from the Pascal program. Although we
filled out the records created by the Pascal program with blanks to make the
output more readable, this blank fill is unnecessary. The length field at the
at the structure
start
of the record guarantees that
start
we do
not read past the end of the data in
the record.
4.3.2 Header Records
often necessary or useful to keep track of
It is
about
a file to assist in future
at the
beginning of the
file
use of the
file.
some
general information
header record
is
often placed
to hold this kind of information. For example,
is no easy way to jump to the end of a file,
even though the implementation supports direct access. One simple
solution to this problem is to keep a count of the number of records in the
file and to store that count somewhere. We might also find it useful to
include information such as the length of the data records, the date and
time of the file's most recent update, and so on. Header records can help
in
some
make
versions of Pascal there
a file a self-describing object, freeing the
software that accesses the
file
from having to know a priori everything about its structure, and hence
making the file-access software able to deal with more variation in file
structures.
The header record usually has a different structure than the data records
file. The output from update. c, for instance, uses a 32-byte header
in the
record, whereas the data records each contain 64 bytes. Furthermore, the
by
data records created
update. c contain only character data,
header record contains an integer that
tells
how many
whereas the
data records are in the
file.
Implementing a header record presents more of a challenge for the
programmer. Recall that the Standard Pascal view of a file is one of
a repeated collection of components, all of which are the same component
type. Since a header record is fundamentally a different kind of record than
Pascal
the other records in a
In
some
file,
Pascal does not naturally support header records.
cases, Pascal lets us get
variant
record in Pascal
around this problem by using variant records.
one that can have different meanings,
is
depending on context. Unfortunately,
so
its
same
use as
header record
is
size as all other records in the
When
a variant
record cannot vary in
constrained by the fact that
it
size,
must be the
file.
faced with a language like Standard Pascal that strictly proscribes
the types of records
we
can use in
a file,
we
often find ourselves resorting
i_
+-
o
u
c
3
a
o
3
QJ
i-
QJ
l_
TJ
O
u
QJ
"H
o
U
TJ
>
i_
-C
._L
+J
to
4-
-M
$2
^A ^
0)
djDlz:
cz
CJ
i_
-^
>>
cn xi
QJ
--"
QJ
i-
in
X)
L.
ID
O
u
OJ
+->
in
QJ
i.
-t->
OJ
(=
CK
4-
=>
C=
in
QJ
l.
CJ
QJ
QJ
JZZ
CO
O
3
QJ
-t->
-t-J
+->
>>
X
+J
X
ID
_i
ro
o ^
CD O
(=
ID
ID
"D
<L
C
O
in
ID
*ID
TD
^-
4^r
CJ
r^
c\j
OJ
r^
co
oo
CD
"d"
en
r^
T
CD
U r^
r*
rs
QJ
CD
CD
CO
CD
<+-
CD
CD
m o
CVJ
T
cn
CD
o o
O O
o o
o o
o o
CD
CD
CD
r,
ID
cn
-h*
ro
r*
tn
cvj
r*
xi
-q-
4^a-
r^
1^
u ,_
CD
,
CD
CD
sr
CD
u ,_
CD
T
\T
00
^T
O
r^
\r
r^
i_n
^r
oo
i>.
CD
r^
00
OJ
^r
r^
LD
00
CD
r^
00
iv
uo
CD
U
1^
LD
CD
u u u
CD
CD
4CD
ro
<r
00
rs
cn
r^
r-.
o
t r^
o o o o
"srcDooj
o O t- rO CD O CD
O O CD CD
o o o o
O CD CD CD
sr
ID
lu
in
111
i.
<*
*
^
QJ
cn
+-1
^
0
m *
ID
TJ
(vj
10
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CVJ
00
00
CJ
03
->
QJ
CD
<
CVJ
CVJ
CD
CVJ
CD
sr
cD
CVJ
o o
o o
o o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
,_
CD
[V
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
O
CD
CD
CD
CD
CD
CD
CD
ro
CD
CD
CD
O
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
O
O
CD
CD
CD
\r
CVJ
CVJ
co
1^
00
cn
00
00
CD
CD
CD
CD
CD
CD
CD
O O O
CD
CD
CD
^-(Docvj
,-
O
CD
CD
CD
CD
CD
O
CD
CD
CD
CD
O
O O O O
CD
CD
CD
CD
CD
CD
,_
XI
TT
CD
CD
CD
CD
CD
CD
CD
CD
O
CD
CD
O
CD CD CD O
O O O O
CD
CD
CD
CJ
s*
CD
CD
O O
CD
CD
CD
CD
CD
CD
CD
CD
O
CD
CD
CD
CD
^r
CD
CD
CD
CD
CD
CD
CD
CD
O
O
O
O
O
O
CD
CD
CD
CD
CD
o
CD o
CD rCD CD
O O O O O
CD CD CD CD CD
CD CD O CD CD
cvj
CD
CD
CD
CD
r--
i*v
r^
CD
o o
o o
o
CVJ
u ,_
cvj
CO
T
^3-
CVJ
^r
t>
UO
CO
OJ
CD
CD
00
CD
4CD
CJ
CD
CD
CD
CD
r>
CVJ
r-
L0
00
QJ
\r
CVJ
CD
r^
r^
^r
1^
00
CVJ
CD
00
r^
00
uo
oo
CVJ
ro
0-
r^
uo
CD
uo
CD
X)
CD
T
\r
O
O
CD
CJ
cj
o
o o
CVJ
,_
co
\r
iv
00
ro
0J
^r
r^
CD
^r
r^
CVJ
CVJ
o X o
^d-
CO
,_
4-
cvj
CD
XI
^T
CVJ
00
00
CO
r^
CO
* o
cn
r\ o
o
o
o
4- ,_
CD
r^
O
[N.
OsJ
o
o
CVJ
o CD
o o
CVJ
-Q
^.
CD
Ul
M-
cvj
Cvj
CVJ
CO
CJ
CO
CVJ
I*"".
CVJ
00
CVJ
Cvj
00
CO
oo
^r
00
r^
00
00
CJ
oo
CD
OJ
o o
o o
CVJ
CVJ
CVJ
CVJ
ro
^r
cvi
cvj
o x o o
o o
(J)
4-
Cvj
ro
*-
CD
oj
CD
r^
r^
cvj
(D
CD
CD
oj
CD
oj
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
CD
Cvj
ST
=5
c5
03
CD
_Q
<D
o E
si
CD
OJ
OJ
CD
TT
OJ
CD
CD
CD
CD
-,,
c/)
J-;
Cj
>^
_Q
T3
CJ
CO
0J
oj
fC
o
c
'"5
CD
^ x:
ro
j2)
^:
"o
O 4-
<D
03
^^
TJ
03
CD
CJ
x:
djo
OJ
CJ
CD
h-
C3
CN
O c
X
i_
O O O O O
CJ
CVJ
CVJ
CJ
QJ
OJ
CVJ
CD
T3
-6
o1
o ci
Cvj
cvj
CVJ
ro
CJ
X X 3
to
u
X 6 TJ
^-
cvj
CD
OS)
+->
O O
O
CD O
oj oo
OJ
O O O CD O
CD CD O CD O
CD CD O CD O
CD
JC
4'
JC 0)
r^
CD
-3
CVJ
cvj
CD
*Q
00
>^
TJ JQ
CJ
CJ
i
CVJ
CVJ
03
X
o
> O
o o
CD o
o
u o
CD o
CVJ
(V\
CO
CJ
Ih.
o o
o CD
CVJ
>^
*- jd
CD
cc
CVJ
CO
CD
Q.
t
CJ
CVJ
rv
in
CD
O
^
CVJ
"3
CVJ
c:
..
nr
~ CO
<r
o o
o o
cu
T O CD
CM
o O
1^
T
CD
CD
-*=
CJ
cu
O O
>
= TD
-^ ?!
^fe
CJ
*-
in
CO
X3 "O
fO
<
CD
CD
CD
CD
<D
JV
ro
OJ
CO
o
o 2
O
o
DJD
CO
GO
ro
l/l
^
ID
QJ
in
CD
cn
o
o
o
o
o
o
o
o
o
-.
E17
-t->
*-
uo
\r
<v
x:
<,
^"^
<_
EI
cx>
~3
2 ffl
O '^
CJJ
<=
s
d.1
CUD
CJ
CJ
DJO
g?
-
QJ
CD
E
C
CO
c
x:
-trL
CU3
-t-^
*f 00 >,
JQ CD
LU TJ
II
O "O
CD
CJ x:
CD CJ 03 cd
X
u_ cn c= x=
121
22
FUNDAMENTAL
FILE
We
to tricks.
STRUCTURE CONCEPTS
use such
We just
a trick in update. pas:
use the
initial
integer
purpose in the header record. In the data
records this field holds a count of the bytes of valid data within the record;
in the header record it holds a count of the data records in the file.
Header records are a widely used, important file design tool. For
field in the
record for
when we
example,
a different
reach the point where
we
are discussing the construction
of tree-structured indexes for files, we see that header records are often
placed at the beginning of the index to keep track of matters such as the
RRN of the record that is the root of the index. We investigate some more
elaborate uses of header records later in this chapter and also in subsequent
chapters.
4.4
Access and
File
In the course
File
Organization
of our discussions in
we have
this chapter,
looked
at
Variable-length records;
Fixed-length records;
Sequential access; and
Direct access.
The
two of these
first
has to do with file
access
with
a useful one;
is
relate to aspects
access.
The
of file organization. The second pair
interaction
we need
to look at
between
it
more
organization and
file
file
closely before continuing
this chapter.
Most of what we have considered
so far
falls
into the category of
file
organization:
Can
Is
the
file
be divided into
fields?
there a higher level of organization to the
file
that
combines the
fields into records?
Do
the records have the same number of bytes or fields?
do we distinguish one record from another?
do we organize the internal structure of a fixed-length record
so we can distinguish between data and extra space?
all
How
How
We
have seen that there are many possible answers to these questions and
of a particular file organization depends on many things,
including the file-handling facilities of the language you are using and the
that the choice
use you
want
Using
to
make of the file.
file
implies access.
We
looked
first
ultimately developing a sequential search. So long as
individual records began, sequential access
at
we
sequential
did not
access,
know where
was the only option open
to us.
BEYOND RECORD STRUCTURES
23
When we wanted
direct access, we fixed the length of our records, and this
allowed us to calculate precisely where each record began and to seek
directly to
it.
In other
words, our desire for direct
fixed-length record
file
organization.
Does
access
this
caused us to choose
mean
that
we
can equate
fixed-length records with direct access? Definitely not. There
is nothing
about our having fixed the length of the records in a file that precludes
sequential access; we certainly could write a program that reads sequentially
through
fixed-length record
Not only
can
sequentially, but
we
we
elect
file.
to
read through the fixed-length records
can also provide direct access to variable-length records
simply by keeping a list of the byte offsets from the start of the file for the
placement of each record. We chose a fixed-length record structure in
update. c and update. pas because it is simple and adequate for the data that we
want to store. Although the lengths of our names and addresses vary, the
variation is not so great that we cannot accommodate it in a fixed-length
record.
Consider, however, the effects of using
fixed-length record organi-
zation to provide direct access to records that are documents ranging in
length from a few hundred bytes to over
hundred kilobytes. Fixed-length
of space, so some form of
variable-length record structure would have to be found. Developing file
structures to handle such situations requires that you clearly distinguish
between the matter of access and your options regarding organization.
The restrictions imposed by the language and file system used to
develop your applications do impose limits on your ability to take
advantage of this distinction between access method and organization. For
records
would be
disastrously
wasteful
C language provides the programmer with the ability to
implement direct access to variable-length records, since it allows access to
any byte in the file. On the other hand, Pascal, even when seeking is
supported, imposes limitations related to Pascal's definition of a file as a
collection of elements that are all of the same type and, consequently, size.
Since the elements must all be of the same size, direct access to
example, the
variable-length records
4.5
is
difficult, at best, in Pascal.
Beyond Record Structures
Now
on the concepts of organization and access, we
look at some interesting new file organizations and more complex ways of
accessing files. We want to extend the notion of a file beyond the simple
that
we
have
a grip
idea of records and fields.
124
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
We begin
with the idea of abstract data models. Our purpose here is to
put some distance between the physical and the logical organization of files,
to allow us to focus more on the information content of files and less on
physical format.
4.5.1 Abstract Data Models
The
history of
file
structures and
file
processing parallels the history of
computer hardware and software. When file processing first became
common on computers, magnetic tape and punched cards were the primary
means used to store files, RAM space was dear, and programming
languages were primitive. Programmers as well as users were compelled to
view file data exactly as it might appear on a tape or cards
as a sequence
of fields and records. Even after data was loaded into RAM, the tools for
manipulating and viewing the data were unsophisticated and reflected the
magnetic tape metaphor. Data processing meant processing fields and
records in the traditional sense.
Gradually, computer users began to recognize that computers could
process
more than just
fields
and records. Computers could,
for instance,
process and transmit sound, and they could process and display images and
These kinds of applications deal with information
metaphor of data stored as sequences of records
that does not nicely
that are divided into fields, even if, ultimately, the data might be stored
physically in the form of fields and records. It is easier, in the mind's eye,
to envision data objects such as documents, images, and sound as objects
documents
(Fig. 4.13).
fit
that
we
the
manipulate in ways that are specific to the objects themselves, rather
than simply as fields and records on
The notion
medium
is
that
we need
a disk.
not view data only
captured in the phrase
as
FIGURE 4.13 Data such as sound, images, and documents do not
of data stored
it
appears on
abstract data model, a
as sequences of records that are divided into
fit
fields.
term
a particular
that encourages
the traditional metaphor
BEYOND RECORD STRUCTURES
25
an application-oriented view of data, rather than
The organization and
in
terms of
how
an application views the data, rather than
might physically be
One way
file is
to
a medium-oriented view.
methods of abstract data models are described
access
that
how
the data
stored.
we
know
save a user from having to
keep information
about objects
in a
in the file that file-access software can use to
"understand" those objects. A good example of how
file structure information in a header.
might be done
this
is
to put
4.5.2 Headers and Self-Describing
We have
seen
how
Files
header record can be used to keep track of how
many
our programming language permits it, we can
put much more elaborate information about a file's structure in the header.
When a file's header contains this sort of information, we say the file is
records there are in a
file.
If
we
Suppose, for instance, that
self-describing.
store in a
file
the following
information:
A name
for each field;
The width of each field; and
The number of fields per record.
We
can now write a program that can read and print a meaningful display
of files with any number of fields per record and any variety of fixed-length
widths. In general, the
field
file's
header,
As
usual, there
is
files in
file
structure information
our software needs to
the less
structure of an individual
structures of
more
put into
about the specific
file.
a trade-off: If
the
know
we
programs
we do
field and record
and write them, the programs
not hard-code the
that read
themselves must be more sophisticated. They must be flexible enough to
interpret the self-descriptions that they find in the
file
headers.
4.5.3 Metadata
Suppose you
are an
astronomer interested
telescopes that scan the sky, and
digital representations
you want
of these images
in
studying images generated by
to design a
(Fig. 4.14).
You
file
structure for the
expect to have
many
images, perhaps thousands, that you want to study, and you want to store
one image per file. While you are primarily interested in studying the
images themselves, you will certainly need information about each image:
where in the sky the image is from, when it was made, what telescope was
used, references to related images, and so forth.
data that describes the
This kind of information is called metadata
primary data in a file. Metadata can be incorporated into any file whose
126
FUNDAMENTAL
STRUCTURE CONCEPTS
FILE
FIGURE 4.14 To make sense
two-Mbyte image, an
astronomer needs such metadata as the kind of image it
of this
is, the part of the sky it is
from, and the telescope that
was used to view it. Astronomical metadata is often
stored
data
in
the
itself.
same
file
(This image
as the
shows
polarized radio emission from
the southern spiral galaxy
NGC 5236 [M83] as observed
with the Very Large Array radio telescope in New Mexico.)
primary data requires supporting information. If a file is going to be shared
users, some of whom might not otherwise have easy access to its
by many
metadata,
it
A common
may
be most convenient to store the metadata in the
place to store metadata in a
Typically, a
community of users of a
file is
particular kind of data agrees
standard format for holding metadata. For example,
called
FITS
(Flexible
file itself.
the header record.
on
standard format
Image Transport System) has been developed by the
International Astronomers'
Union
for storing the kind of astronomical data
FITS header is a collection of 2,880-byte
which each record contains a single
piece of metadata. Figure 4.15 shows part of a FITS header. In a FITS file,
the header is followed by the actual numbers that describe the image, one
just described in a
blocks of 80-byte
file's
header.^
ASCII
records, in
number per observed point of the image.
Note that the designers of the FITS format chose to use ASCII in the
header, but binary values for the image. ASCII headers are easy to read and
process and, since they occur only once, take up relatively little extra space.
Since the numbers that make a FITS image are rarely read by humans, but
binary
rather are first processed into a picture and then displayed, binary format
the preferred choice for them.
t For
more
Readings."
details
on FITS,
see the references listed at the
end of
this
chapter
"Further
is
BEYOND RECORD STRUCTURES
=
=
=
=
=
=
=
-
27
CONFORMS TO BASIC FORMAT
BITS PER PIXEL
2 / NUMBER OF AXES
256 / RA AXIS DIMENSION
256 / DEC AXIS DIMENSION
F / T MEANS STANDARD EXTENSIONS EXIST
0.000100000 / TRUE = TAPE*BSCALE +BZERO
0.000000000 / OFFSET TO TRUE PIXEL VALUES
MAP_TYPE= REL_EXPOSURE'/ INTENSITY OR RELATIVE EXPOSURE MAP
=
/ DIMENSIONLESS PEAK EXPOSURE FRACTION
BUNIT
CRVAL1
0.625 / RA
REF POINT VALUE (DEGREES)
=
CRPIX1
128.500 / RA
REF POINT PIXEL LOCATION
=
-0.006666700 / RA
CDELT1
INCREMENT ALONG AXIS (DEGREES)
= 'RA
/
TAN'
RA
TYPE
CTYPE1
=
CROTA1
0.000 / RA
ROTATION
71.967 / DEC REF POINT VALUE (DEGREES)
CRVAL2 =
CRPIX2 =
128.500 / DEC REF POINT PIXEL LOCATION
0.006666700 / DEC
CDELT2 =
INCREMENT ALONG AXIS (DEGREES)
/ DEC
CTYPE2 = 'DEC--TAN'
TYPE
CR0TA2 =
ROTATION
0.000 / DEC
=
EPOCH
1950.0 / EPOCH OF COORDINATE SYSTEM
ARR_TYPE=
=DP
4 /
3 = FP, 4=1
DATAMAX 1.000 / PEAK INTENSITY (TRUE)
DATAMIN =
0.000 / MINIMUM INTENSITY (TRUE)
-22.450 / ROLL ANGLE (DEGREES)
ROLL_ANG=
BAD_ASP =
/
0=good, 1=bad(Do not use roll angle)
TIME_LIV=
5649.6 / LIVE TIME (SECONDS)
OBJECT = 'REM6791
/
SEQUENCE NUMBER
AVGOFFY =
1.899 / AVG Y OFFSET IN PIXELS, 8 ARCSEC/PIXEL
AVGOFFZ =
2.578 / AVG Z OFFSET IN PIXELS, 8 ARCSEC/PIXEL
RMSOFFY =
0.083 / ASPECT SOLN RMS Y PIXELS, 8 ARCSC/PIX
RMSOFFZ =
0.204 / ASPECT SOLN RMS Z PIXELS, 8 ARCSC/PIX
TELESCOP= 'EINSTEIN
/
TELESCOPE
INSTRUME= 'IPC
/
FOCAL PLANE DETECTOR
OBSERVER= 2
/
OBSERVER #: 0=CFA; 1=CAL; 2=MIT; 3=GSFC
=
GALL
119.370 / GALACTIC LONGITUDE OF FIELD CENTER
=
GALB
9.690 / GALACTIC LATITUDE OF FIELD CENTER
DATE_OBS= '80/238
/
YEAR & DAY NUMBER FOR OBSERVATION START
DATE_STP= '80/238
/
YEAR & DAY NUMBER FOR OBSERVATION STOP
= "SNR SURVEY: CTA1
TITLE
ORIGIN = 'HARVARD-SMITHSONIAN CENTER FOR ASTROPHYSICS
= '22/09/1989
DATE
/
DATE FILE WRITTEN
= '05:26:53
TIME
TIME FILE WRITTEN
/
END
SIMPLE
BITPIX
NAXIS
NAXIS1
NAXIS2
EXTEND
BSCALE
BZERO
16
'
'
'
FIGURE 4.15 Sample FITS header. On each line, the data to the left of the 7* is the actual
metadata (data about the raw data that follows in the file). For example, the second line
("BITPIX = 16") indicates that the raw data in the file will be stored in 16-bit integer format. Everything to the right of a V is a comment, describing for the reader the meaning of
the metadata that precedes it. Even a person uninformed about the FITS format can learn a
great deal about this file just by reading through the header.
28
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
is a good example of an abstract data model. The data
meaningless without the interpretive information contained in the
header, and FITS-specific methods must be employed to convert FITS data
FITS image
itself is
into an understandable image.
we
look
Another example
is
the raster image,
which
at next.
4.5.4 Color Raster Images
From
a user's
device as
it is
point of view, a
a data processor.
spreadsheets, or numbers,
we
modern computer is as much a graphical
Whether we are working with documents,
viewing and storing pictures
in addition to whatever other information we work with. Let's examine one
type of image, the color raster image, as a means to filling in our conceptual
understanding of data objects.
color raster image
on
is
are likely to be
a rectangular array
of colored dots, or pixels,^
FITS image is a raster image in the sense
that the numbers that make up a FITS image can be converted to colors and
then displayed on a screen. There are many different kinds of metadata that
can go with a raster image, including
that are displayed
a screen.
The dimensions of the image: the number or pixels per row and the
number of rows.
The number of bits used to describe each pixel. This determines how
many
colors can be associated with each pixel.
display only
two
colors, usually black
display four colors (2
and so
2
),
image can
image can
1-bit
2-bit
an 8-bit image can display 256 colors
(2
),
forth.
color lookup table,
or palette, indicating which color
to each pixel value in the image.
table
and white.
2-bit
with 4 colors, an 8-bit image uses
image uses
a table
is
to be assigned
color lookup
with 256 colors, and
so forth.
we think of an image as an abstract data
we might associate with images? There
If
that
with getting things in and out of
store_\mage routine.
Then
type,
what
are
some methods
are the usual ones associated
computer:
read_image routine and
there are those that deal with images as special
objects; for example,
Display an image in
window on
console screen;
color lookup table;
Overlay one image onto another to produce a composite image; and
Display several images in succession, producing an animation.
Associate an image with
'Pixel stands for "picture
element."
a particular
BEYOND RECORD STRUCTURES
The
color raster image
an example of
is
129
type of data object that
more than the traditional field/record file structure. This is
particularly true when more than one image might be stored in a single file,
or when we want to store a document or other complex object together
with images in a file. Let's look at some ways to mix object types in one file.
requires
4.5.5 Mixing Object Types
Keywords
The FITS header
described earlier,
nique,
be contained
in
One
File
4.15) illustrates an important tech-
(Fig.
and records: the use of
do not know what fields are going
for identifying
keywords. In the case of FITS
to
in
we
headers,
any given header, so
fields
we
identify each field using a
"keyword = value" format.
Why does this format work for FITS
files,
for our address file? For the address
we saw
demanded
the
file.
In
file
whereas
it
was inappropriate
keywords
that the use of
high price in terms of space, possibly even doubling the
FITS
files
When
size
amount of overhead introduced by keywords
the
of
is
FITS file in the example
quite small.
contains approximately 2 megabytes. The keywords in the header occupy a
total of about 400 bytes, or about 0.02% of the total file space.
Tags
the
image
With the addition
we
via
included, the
is
keywords of
file
structure information and
more than just a collection of
repeated fields and records. Can we extend this notion beyond the header to
other, more elaborate objects? For example, suppose an astronomer would
metadata to
header,
like to store several
see that a
can be
file
FITS images of different
sizes in a file,
usual metadata, plus perhaps lab notes describing
from the image
(Fig. 4.16).
Now we
can think of our
may
be very different in content
structures do not handle well. Maybe we need
objects that
together with the
what the
view
new
scientist learned
mixture ot
our previous file
kind of file structure.
file as a
that
There are many ways to address this new file design problem. One
would be simply to put each type of object into a variable-length record and
FIGURE 4.16 Information that an astronomer wants to include
----.
yj-.y.i
...
in a file.
MMIS1
KAXIS2
600
max: si
".-
MMUS4
kax:i<
yjvx
ssoale - :.::
BZERO - 1S8E<
BZES
U5x=rO
500
>
t::
l
1S8E14
U5y=rA
30
FUNDAMENTAL
write our
The
like:
FILE
processing programs so they
file
first
STRUCTURE CONCEPTS
record
is
image; the third record
header for the
is
first
know what
each record looks
image; the second record
document; the fourth
the
is
header for the second
is a
image; and so forth.
This solution
drawbacks:
workable and simple, but
is
it
has
some
familiar
Objects must be accessed sequentially, making access to individual
images
The
in large files
file
time consuming.
must contain exactly the
actly the order indicated.
We
objects that are described, in ex-
could not, for instance, leave out the
notebook for some of the images (or in some cases leave out the
notebook altogether) without rewriting all programs that access the
file
to reflect the changes in the
solution to these problems
is
file's
structure.
hinted
at in
the
FITS header: Each
line
keyword that identifies the metadata field that follows in
not use keywords to identify all objects in the file
not just
begins with a
Why
line.
fields in the headers,
but the headers themselves,
the
the
as well as the
images and
any other objects we might need to store? Unfortunately, the "keyword =
it is short and fits easily in an
data" format makes sense in a FITS header
80-byte line
but it doesn't work at all for objects that vary enormously in
size and content. Fortunately, we can generalize the keyword idea to
address these problems by making two changes:
Lift the restriction that
enough
each record be 80 bytes, and
to hold the object that
Place the
keywords
is
let it
be big
referenced by the keyword.
in an index table, together
with the byte
offset
of
the actual metadata (or data) and a length indicator that indicates
how many
The term
this
In
type of
it,
we
bytes the metadata (or data) occupies in the
tag is
file
file.
commonly used in place of keyword in connection with
The resulting structure is illustrated in Fig. 4.17.
structure.
encounter two important conceptual tools for
file
design:
(1)
the
use of an index table to hold descriptive information about the primary data,
and
(2)
the use of tags to distinguish different types of objects. These tools
allow us to store in one
one another
Tag
structures are
For example,
mixture of objects
and content.
file a
in structure
common among
a structure called
popular tagged
file
objects
standard
file
TIFF (Tagged Image
format used for storing images.
that can vary
from
formats in use today.
File
HDF
Format)
is
very
(Hierarchical Data
Format) is a standard tagged structure used for storing many different kinds
of scientific data, including images. In the world of document storage and
retrieval, SGML (Standard General Markup Language) is a language for
131
BEYOND RECORD STRUCTURES
Index table
with tags:
header
notes
header
image
rrx
SIMPLE - -7HAX1S - 4
HAXIS1 - :.:
KAXIS2 - *::
hax:s3 - l
HAXIS4 > i
8SCALE - 0.015
BZERO - 158EU
=:v?le
MAXIS
MAX! SI
MAX Is;
500
600
"_i
MAX1S4
BSCALE
BZEfcO
0.015
i53E-:4
U5x=ro
FIGURE 4.17
Same
as Fig. 4.16, except with tags identifying the objects.
describing
document
structures and for defining tags used to
mark up
structure. Like FITS, each of these provides an interesting study in
that
file
design and standardization. References to further information on each are
provided
at
the end of this chapter, in "Further Readings."
Accessing Files with Mixtures of Data Objects
files
to contain widely varying objects
is
The
idea of allowing
compelling, especially for appli-
amounts of metadata or unpredictable mixes of
of data, for it frees us of the requirement that all records be
fundamentally the same. As usual, we must ask what this freedom costs us.
To gain some insight into the costs, imagine that you want to write a
program to access objects in such a file. You now have to read and write
tags as well as data, and the structure and format for different data types are
likely to be different. Here are some questions you will have to answer
almost immediately:
cations that require large
different kinds
When we want
to read an object
of
a particular type,
how
do
we
search for the object?
When we want
how
and where do we
and where exactly do we put the object?
Given that different objects will have very different appearances
within a file, how do we determine the correct method for storing or
store
its
to store an object in the
file,
tag,
retrieving the object?
The
first
tags
and pointers to
two questions have
do with accessing the table
the objects. Solutions to this problem
to
that contains the
are dealt
with in
32
FUNDAMENTAL
detail
question,
STRUCTURE CONCEPTS
FILE
Chapter
how
6,
so
we
defer their discussion until then.
to determine the correct
implications that
we
on
briefly touch
4.5.6 Object-oriented
File
We
abstract data
have used the term
methods
The
third
for accessing objects, has
here.
Access
application has of a data object. This
model to describe the view that an
is
essentially an
in-RAM,
application-
oriented view of an object, one that ignores the physical format of objects
as
they are stored in
files.
Taking
this
view of objects buys our software two
things:
delegates to separate modules the responsibility of translating to
and from the physical format of the object, letting the application
modules concentrate on the task at hand. (For example, an image
processing program that can operate in
on 8-bit images should
not have to worry about the fact that a particular image comes from
a file that uses the 32-bit FITS format.)
It opens up the possibility of working with objects that at some level
fit the same abstract data model, even though they are stored in different formats. The in-RAM representations of the images could be
identical, even though they come from files with quite different forIt
RAM
mats.)
File access
access,
that exploits these possibilities could be called object-oriented
emphasizing the
oriented
between
parallels
it
and the well-known object-
programming paradigm.
As an example
that illustrates
both points, suppose you have an image
processing application program (we'll
call
nfind_star) that operates in
RAM
and you need to process a collection of images. Some are
FITS format and some in TIFF files in a different
format. An object-oriented approach (Fig. 4.18) would provide the
application program with a routine (let's call it read_jmage( )) for reading
images into
in the expected 8-bit form, letting the application
concentrate on the image processing task. For its part, the routine
read_\mage( ), given a file to get an image from, determines the format ot
the image within the file, invokes the proper procedure to read the image in
format
that format, and converts it from that format into the 8-bit
on
8-bit images,
stored in FITS
files in a
RAM
RAM
that the application needs.
Tagged
file
file
formats are one
organization
accompanied by
and
a
file
way
access.
specification
to
implement
The
this
specification
conceptual view of
ot
tag
can
be
of methods for reading, writing, and
133
BEYOND RECORD STRUCTURES
program find_star
read_image ("starl"
process image
image)
image
end find star
RAM
(FITS
file)
Disk
FIGURE 4.18 Example of object-oriented access. The program find_star knows nothing about
file format of the image that it wants to read. The routine readjmage has methods to
convert the image from whatever format it is stored in on disk into the 8-bit in-RAM format
required by find_star.
the
otherwise manipulating the corresponding data object according to the
needs
of an application.
Indeed,
definition of the abstract data
format lends
itself to the
any specification that separates the
that of the corresponding file
model from
object-oriented approach.
4.5.7 Extensibility
One of the
advantages of using tags to identify objects within
do not have
software
to
may
know
a priori
what
all
files is
of the objects will look
eventually have to deal with.
We
that
like that
have just seen that
if
we
our
our
34
STRUCTURE CONCEPTS
FUNDAMENTAL
FILE
program
methods
to be able to access a mixture of objects in a
file, it must have
and writing each object. Once we build into our
software a mechanism for choosing the appropriate methods for a given
type of object, it is easy to imagine extending, at some future time, the
types of objects that our software can support. Every time we encounter a
new type of object that we would like to accommodate in our files, we can
implement methods for reading and writing that object and add those
methods to the repertoire of methods available to our file processing
is
for reading
software.
4.6
Portability
and Standardization
recurring theme in several of the examples that
we have just
seen
is
the
want to share files. Sharing files means making sure
accessible on all of the different computers that they might turn
idea that people often
that they are
up on, and that they are somehow compatible with all of the different
programs that will access them. In this final section, we look at two
complementary topics that affect the sharability of files: portability and
standardization.
4.6.1 Factors Affecting Portability
Imagine that you work for a company that wishes to share simple data files
as our address file with some other business. You get together with the
other business to agree on a common field and record format, and you
discover that your business does all of its programming and computing in
C on a Sun computer and the other business uses Turbo Pascal on an IBM
PC. What sorts of issues would you expect to arise?
such
among Operating Systems
"Unexpected Characters in Files," we saw
Differences
linefeed character every time
whereas on most other
every time our address
file
file
it
In
that
encounters
systems
this is
Chapter 2
in the section
MS-DOS
adds an extra
carriage return character,
not the case. This means that
has a byte with hex value OxOd, whether or not
that byte is meant to be a carriage return, the file is extended by an extra
0x0a byte.
This example illustrates the fact that the ultimate physical format of the
same logical file can vary depending on differences among operating systems.
Differences
among Languages
header records,
forced to
we
make our
chose to
Earlier in this chapter,
make our C header 32
Pascal header 64 bytes.
when
discussing
bytes, but
allows us to
we were
mix and match
35
fixed record lengths according to our needs, but Pascal requires that
all
PORTABILITY AND STANDARDIZATION
records in
This
nontext
file
illustrates a
be the same
size.
second factor impeding portability
physical layout offiles produced with different languages
way
the languages let
you define
structures within a
may
among
files:
The
be constrained by the
file.
Differences in Machine Architectures Consider again the header
record that we produce in the C version of our address file. The hex dump
of the file (Fig. 4.13), which was generated using C on a Sun 3 computer,
shows
this
header record in the
0000000
The
,
or
0020 0000 0000 0000 0000 0000 0000 0000
two bytes contain the number of records in the file, in this case
If the same C program is compiled and executed on an IBM
first
20 16 or 32 10
PC
first line:
VAX,
the hex
0000000
Why
dump
of the header record will look
like this:
2000 0000 0000 0000 0000 0000 0000 0000
program? The answer
both cases the numbers were written to the file exactly as they
appeared in RAM, and the two different machines represent two-byte
integers differently
the Sun stores the high-order byte, followed by the
low-order byte; the IBM PC and VAX store the low-order byte, followed
is
are the bytes reversed in this version of the
that in
by the high-order
byte.
This reverse order also applies to four-byte integers on these machines.
For example, in our discussion of
IBM
dumps we saw
that the
hexadecimal
is ldcd6500 16 If you write this value out to a file on
PC, or some other reverse-order machine, a hex dump of the file
value of 500,000,000 10
an
file
created looks like
0000000
this:
0065 cdld
The problem of data representation is not restricted only to binary
numbers. The way structures, such as C structs or Pascal records, are laid
out in
can vary from machine to machine and compiler to compiler.
For example, suppose you have a C program containing the following lines
of code:
RAM
struct
{
i
char
>
i t
cost;
i den t
em
write (fd,
&item,
sizeof (item));
and you want to write files using this code on two different machines, a
Cray 2 and a Sun 3. Because it likes to operate on 64-bit words, Cray's C
36
FUNDAMENTAL
STRUCTURE CONCEPTS
FILE
compiler allocates
it
allocates
minimum
of eight bytes for any element
16 bytes for the struct item.
Cray writes 16 bytes
statement, then, the
When
it
in a struct, so
executes the write(
The same program
you probably would expect,
to the
file.
compiled on a Sun 3 writes only eight bytes, as
and on most IBM PCs it writes six bytes: same exact program; same
language; three different results.
Text
is
encoded differently on
also
differences are primarily restricted to
different platforms. In this case the
two
different types of systems: those
and those that use ASCII. EBCDIC is a standard created
by IBM, so machines that need to maintain compatibility with IBM must
support EBCDIC. Most others support ASCII. A few support both.
Hence, text written to a file from an EBCDIC-based machine may well not
be readable by an ASCII-based machine.
Equally serious, when we go beyond simple English text, is the
problem of representing different character sets from different national
languages. This is an enormous problem for developers of text databases.
that use
EBCDIC^
4.6.2 Achieving Portability
among
Differences
languages, operating systems, and machine architec-
major problems when we need to generate portable
Achieving portability means determining how to deal with these
differences. And the differences are often not just differences between two
platforms, for many different platforms could be involved.
The most important requirement for achieving portability is to
recognize that it is not a trivial matter and to take steps ahead of time to
tures represent three
files.
insure
it.
Here
are
some
guidelines.
Standard Physical Record Format and Stay with It A
is one that is represented the same physically, no matter
what language, machine, or operating system is used. FITS is a good
example of a physical standard, for it specifies exactly the physical format of
each header record, the keywords that are allowed, the order in which
keywords may appear, and the bit pattern that must be used to represent the
binary numbers that describe the image.
Agree on
physical standard
Unfortunately, once
"improve" on
a
it
standard
by changing
standard. If the standard
is
sometimes be avoided. FITS,
over
its
it
in
is
established,
it
is
very tempting to
some way, thereby rendering
it
no longer
sufficiently extensible, this temptation can
for example, has been extended a
lifetime to support data objects that
few times
were not anticipated
""EBCDIC stands for Extended Binary Coded Decimal Interchange Code.
in its
PORTABILITY AND STANDARDIZATION
original design, yet
all
37
additions have remained compatible with the original
format.
One way
to
make
simple enough that
sure that a standard has staying
files
power
is
to
can be written in the standard format from
make
a
it
wide
range of machines, languages, and operating systems. FITS again exemplifies
such
standard.
FITS headers
are
ASCII 80-byte records
in blocks
of 36
records each, and FITS images are stored as one contiguous block of
numbers, both very simple structures that are easy
modern operating systems and languages.
Agree on
most
to read
and write
Standard Binary Encoding for Data Elements
in
most
The two
common types
of basic data elements are text and numbers. In the case
of text, ASCII and EBCDIC represent the most common encoding
schemes, with ASCII standard on virtually all machines except IBM
mainframes. Depending on the anticipated environment, one of these
should be used to represent all text."*"
The situation for binary numbers is a little cloudier. Although the
number of different encoding schemes is not large, the likelihood of having
to share data among machines that use different binary encodings can be
quite high, especially when the same data is processed both on large
mainframes and on smaller computers. Two standards efforts have helped
diminish the problem, however: IEEE Standard formats, and External Data
Representation (XDR).
IEEE has established standard format specifications for 32-bit, 64-bit,
and 128-bit floating point numbers, and for 8-bit, 16-bit, and 32-bit
integers. With a few notable exceptions (e.g., IBM mainframes, Cray, and
Digital) most computer manufacturers have followed these guidelines in
designing their machines. This effort goes a long way toward providing
portable
number encoding schemes.
is an effort to go the rest of the way. XDR specifies not only a set
of standard encodings for all files (the IEEE encodings), but provides for a
set of routines for each machine for converting from its binary encoding
when writing to a file, and vice versa (Fig. 4.19). Hence, when we want to
store numbers in XDR, we can read or write them by replacing read and
XDR
write routines in our
program with
XDR routines.
The
XDR routines take
care of the conversions.*
""Actually, there are different
applications,
character
versions of both
and for the purposes of
this text,
ASCII and EBCDIC. However,
it is
for
most
sufficient to consider each as a single
set.
used for more than just number conversions. It allows a C programmer to deoriginated as a Sun
scribe arbitrary data structures in a machine-independent fashion.
protocol for transmitting data that is accessed by more than one type of machine. For further information, see Sun (1986 or later).
*XDR
is
XDR
138
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
x:
XDR float
(&xdrs,
&x)
234.5
RAM
XDR specifies a standard external data representation for numbers stored
XDR routines are provided for converting to and from the XDR representation
FIGURE 4.19
in
file.
encoding scheme used on the host machine. Here a routine called
) translates a 32-bit floating point number from its XDR representaon disk to that of the host machine.
to the
XDR_f loatC
tion
Once again, FITS provides us with an excellent example: The binary
numbers that constitute a FITS image must conform to the IEEE Standard.
Any program written on a machine with XDR support can thus read and
write portable FITS files.
Number and Text Conversion
encodings
IBM
is
not feasible.
Sometimes the use of standard data
For example, suppose you are working primarily
mainframes with software that deals with floating point numbers
and text. If you choose to store your data in IEEE Standard formats, every
time your program reads or writes a number or character it must translate
the number from the IBM format to the corresponding IEEE format. This
is not only time-consuming but can result in loss of accuracy. It is probably
better in this case to store your data in native IBM format in your files.
What happens, then, when you want to move your files back and forth
between your IBM and a VAX, which uses a different native format for
numbers and generally uses ASCII for text? You need a way to convert
from the IBM format to the VAX format and back. One solution is to write
(or borrow) a program that translates IBM numbers and text to their VAX
on
PORTABILITY AND STANDARDIZATION
and vice versa. This simple solution
equivalents,
is
illustrated
in
39
Fig.
4.20(a).
But what if, in addition to IBM and VAX computers, you find that
your data is likely to be shared among many different platforms that use
different numeric encodings? One way to solve this problem is to write a
program to convert from each of the representations to every other
representation. This solution, illustrated in Fig. 4.20(b), can get rather
complicated. In general,
need n(n
messy. Not
for each
to
1)
you have
if
of where the
know which
many
file
they are to be exported to
1)
If n
is
large, this can
be very
you need to keep track,
came from and/or where it is going in order
In this case, a better solution
n(n
encoding schemes, you will
translators, but
translator to use.
intermediate format, such as
Fig. 4.20(c).
(Why?)
different translators.
only do you need
file,
n different
it
and
it
down
cut
to agree
translate files into
the
number of
on
standard
XDR whenever
platform. This solution
a different
Not only does
to 2n, but
would probably be
XDR,
is
illustrated in
translators
from
should be easy to find translators to convert from most
platforms to and from
XDR. One
requires two conversions to
negative aspect of this solution
go from any one platform
is
that
it
to another, a cost that
has to be weighed against the complexity of providing n(n
1)
translators.
Conversion
Suppose you are a doctor and you have
organ taken periodically over several
minutes. You want to look at a certain image in the collection using a
program that lets you zoom in and out and detect special features in the
image. You have another program that lets you animate the collection of
images, showing how it changes over several minutes. Finally, you want to
annotate the images and store them in a special X-ray archive, and you have
another program for doing that. What do you do if each of these three
programs requires that your image be in a different format?
The conversion problems that apply to atomic data encodings also
apply to file structures for more complex objects, like images, but at a
different level. Whereas character and number encodings are tied closely to
specific platforms, more complex objects and their representations just as
File Structure
X-ray
raster
images of
a particular
often are tied to specific applications.
For example, there are many software packages that deal with images,
and very little agreement about a file format for storing them. When we
look at this software, we find different solutions to this problem:
Require that the user supply images in a format that is compatible
with the one used by the package. This places the responsibility on
the user to convert from one format to another. For such situations,
140
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
From:
To:
(a)
Converting between
IBM and Vax
native format
requires two conversion routines.
From:
To:
(b)
Converting directly between five different native formats
requires 20 conversion routines.
To
8c
From:
From:
To:
XDR
(c) Converting between five different native formats via an
intermediate standard format requires 10 conversion routines.
FIGURE 4.20 Direct conversion between n native machines formats requires n (n - 1) conversion routines, as illustrated in (a)
and (b). Conversion via an intermediate standard format requires
2n conversion routines, as illustrated in (c).
PORTABILITY AND STANDARDIZATION
may be preferable to provide utility programs that translate from
one format to another and that are invoked whenever translating.
Process only images that adhere to some predefined standard format.
This places the responsibility on a community of users and software
developers for agreeing on and enforcing a standard. FITS is a good
example of this approach.
Include different sets of I/O methods capable of converting an image
from several different formats into a standard RAM structure that
the package can work with. This places the burden on the software
developer to develop I/O methods for file object types that may be
stored differently but for the purposes of an application are conceptually the same. You may recognize this approach as a variation on the
concept of object-oriented access that we discussed earlier.
it
System Differences
File
to
another,
Finally, if
organized physically. For example,
512-byte blocks, but
such
as
you move
2,880-bytes
non-UNIX
UNIX
files
system
files
are
to tapes in
systems often use different block
When
you may need
problem.
to deal
systems write
file
way
sizes,
where the FITS
between systems,
thirty-six 80-byte records. (Guess
blocking format comes from?)
UNIX
from one
files
chances are you will find differences in the
with
and Portability
problem just described,
this
transferring
files
Recognizing problems such
UNIX
provides a utility called
intended primarily for copying tape data to and from
as the block-size
dd.
Although dd
UNIX systems,
be used to convert data from any physical source. The dd
following options, among others:
utility
it
is
can
provides the
Convert from one block size to another;
Convert fixed-length records to variable length, or vice versa;
Convert ASCII to EBCDIC, or vice versa;
Convert all characters to lowercase (or to uppercase); and
Swap every pair of bytes.
Of
course, the greatest contribution
discussed here
is
UNIX
itself.
By
its
UNIX
makes
to the
problems
simplicity and ubiquity,
UNIX
encourages the use of the same operating system, the same file system, the
same views of devices, and the same general views of file organization, no
matter what particular hardware platform you happen to be using.
For example, one of the authors works in an
nationwide constituency that operates many different
two Crays, a Connection Machine, and many Sun,
Graphics, and Digital workstations. Because each
organization with
computers, including
Apple, IBM, Silicon
runs
some
flavor of
42
FUNDAMENTAL
UNIX,
STRUCTURE CONCEPTS
FILE
they
incorporate precisely the same view of
all
all
external storage
ASCII, and they all provide the same basic
programming environment and file management utilities. Files are not
perfectly portable within this environment, for reasons that we have
covered in this chapter, but the availability of UNIX goes a long way
toward facilitating the rapid and easy transfer of files among the applications, programming environments, and hardware systems that the organithey
devices,
all
use
zation supports.
SUMMARY
The lowest
we
normally impose on a file is a
file merely as a stream of
bytes, we lose the ability to distinguish among the fundamental informational units of our data. We call these fundamental pieces of information
fields. Fields are grouped together to form records. Recognizing fields and
of organization
level
stream of bytes. Unfortunately,
recognizing records requires that
There are many ways
from the next:
that
by storing
data in a
we impose structure on
to separate
one
field
the data in the
file.
from the next and one record
Fix the length of each field or record.
Begin each
it
field
number of bytes
or record with a count of the
that
contains.
Use
delimiters to
In the case
of
mark
the divisions
between
another useful technique
fields,
value" form to identify
fields.
entities.
is
to use a
In the case of records,
"keyword =
another useful
where each record begins.
which
records are grouped into
higher level of organization, in
blocks, is also often imposed on files. This level is imposed to improve I/O
performance rather than our logical view of the file.
technique
is
to use a second, index
that tells
file
One
In this chapter
at
we
use the record structure that uses
simple
file
individuals.
before
We use buffering to
we know
its
length indicator
it
to the
complete record
length field of each record as a binary
In the
former
contents of our
file.
case,
it
is
for writing
accumulate the data in an individual record
length to write
allowing us to read in
digits.
programs
and reading
of variable-length records containing names and addresses of
the beginning of each record to develop
file.
at
Buffers are also useful in
one time.
number
We
represent the
or as a sequence of
useful to use a file
dump
to
ASCII
examine the
SUMMARY
Sometimes we identify individual records by their relative record numbers
(RRNs) in a file. It is also common, however, to identify a record by a key
whose value is based on some of the record's content. Key values must
occur in, or be converted to, some predetermined canonical form if they are
to be recognized accurately and unambiguously by programs. If every
record's key value is distinct from all others, the key can be used to identify
and locate the unique record in the file. Keys that are used in this way are
called primary keys.
we
In this chapter
through
a file
look
looking for
at
can perform poorly for long
searching
time for
process
the technique of searching sequentially
record with
files,
a particular key. Sequential search
but there are times
reasonable. Record blocking can be used to
is
a sequential search substantially.
sequentially are
files
wc and
Two
mechanism
the beginning of a record.
accessing the record
The
directly,
for looking
This,
sequential
UNIX
utilities that
clear that
some of the
useful
grep.
In our discussion of ways to separate records,
methods provide
when
improve the I/O
up or
in turn,
by RRN,
it is
calculating the byte
offset
of
opens up the possibility of
rather than sequentially.
RRN
simplest record formats for permitting direct access by
involve the use of fixed-length records.
When
the data itself actually
in fixed-size quantities (e.g., zip codes), fixed-length records
comes
can provide
good performance and good space utilization. If there is a lot of variation in
amount and size of data in records, however, the use of fixed-length
the
records can result in expensive waste of space. In such cases the designer
should look carefully
Sometimes
such as the
it is
at
number of
beginning of the
the possibility of using variable-length records.
helpful to keep track of general information about
file it
records they contain.
pertains to,
is
files,
header record, stored at the
a useful tool for storing this
kind of
information.
important to be aware of the difference between file access and file
We try to organize files in such a way that they give us the
types of access we need for a particular application. For example, one of the
It is
organization.
advantages of a fixed-length record organization
is
that
it
allows
access that is
either sequential or direct.
view of a file as a more or less regular
and records, we present a more purely logical view of the
contents of files in terms of abstract data models, a view that lets applications
ignore the physical structure of files altogether.
This view is often more appropriate to data objects such as sound,
images, and documents. We call files self-describing when they do not require
In addition to the traditional
collection of fields
an application to reveal their structure,
but provide that information
143
44
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
themselves. Another concept that deviates from the traditional view
is
which the file contains data that describe the primary data in
FITS files, used for storing astronomical images, contain extensive
metadata, in
the
file.
headers with metadata.
The
makes
file.
use of abstract data models, self-describing
it
When
models
mix
possible to
a variety
files, and metadata
of different types of data objects in one
this is the case, file access is
also facilitate extensible files
more object oriented. Abstract data
whose structures can be extended
files
accommodate new kinds of objects.
Portability becomes increasingly important as files are used in more
heterogeneous computing environments. Differences among operating
to
systems, languages, and machine architectures
portability.
One
important
way
all
lead to the need for
to foster portability
which means agreeing on physical formats, encodings
and
file
structures.
If a
standard does not exist and
it
is
standardization,
for data elements,
becomes necessary
to convert
from
one format to another, it is still often much simpler to have one standard
format that all converters convert into and out of. UNIX provides a utility
called dd that facilitates data conversion. The UNIX environment itself
supports portability simply by being commonly available on a large
number of platforms.
KEY TERMS
A collection of records stored as a physically contiguous unit on
secondary storage. In this chapter, we use record blocking to improve I/O performance during sequential searching.
Block.
Byte count
field.
that gives the
byte count
field at the
beginning of
number of bytes used
field
allows
program
a variable-length
to store the record.
The
record
use of
to transmit (or skip over) a vari-
able-length record without having to deal with the record's internal
structure.
Canonical form. A standard form for a key that can be derived, by the
application of well-defined rules, from the particular, nonstandard
form of the data found in a record's key field(s) or provided in a
search request supplied by a user.
Delimiter.
One
or
more
characters used to separate fields and records in
a file.
Direct access.
location of
A
a
file
accessing
mode
that involves
jumping
to the exact
record. Direct access to a fixed-length record
is
usually
KEY TERMS
accomplished by using
its relative
record
byte offset, and then seeking to the
Extensibility.
characteristic
number (RRN), computing
byte of the record.
its
first
of some
file
organizations that makes
possible to extend the types of objects that the format can
it
accommo-
date without having to redesign the format. For example, tagged
file
formats lend themselves to extensibility, for they allow the addition
of
new
tags for
new
new methods
data objects and associated
for
accessing the objects.
Field.
The
smallest logically meaningful unit of information in a
record in
usually
made up of several
file.
fields.
method. The approach used to locate information in a
two alternatives are sequential access and direct
File-access
file.
a file is
In general, the
access.
method. The combination of conceptual and physiused to distinguish one record from another and one
File organization
cal structures
field
from another.
An
example of
kind of
fixed-length records containing variable
delimited
same
organization
is
fields.
Fixed-length record. A
same length. Records
ters so
file
numbers of variable-length
file
organization in which
are
padded with blanks,
they extend to the fixed length. Since
length,
it is
all
records have the
nulls, or other charac-
all
the records have the
possible to calculate the beginning position of any
record, making direct access possible.
Header record. A record placed at the beginning of a
store information about the
file
contents and the
file
file
that
is
used to
organization.
Key. An expression derived from one or more of the fields within a
record that can be used to locate that record. The fields used to build
the key are sometimes called the key fields. Keyed access provides a
way of performing
retrieval
content-based retrieval of records, rather than
based merely on
a record's position.
Metadata. Data in a file that is not the primary data, but describes the
primary data in a file. Metadata can be incorporated into any file
whose primary data requires supporting information. If a file is going to be shared by many users, some of whom might not otherwise
have easy access to its metadata, it may be most convenient to store
the metadata in the
file is
file itself.
A common
place to store metadata in a
the header record.
Object-oriented
file access.
form of file
access in
access data objects in terms of the applications'
which applications
in-RAM view of the
methods associated with the objects are responsible
and from the physical format of the object, letting
the application concentrate on the task at hand.
objects. Separate
for translating to
145
46
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
files that describes how amenable
of different machines, via a variety of
different operating systems, languages, and applications.
Primary key. A key that uniquely identifies each record and that is
used as the primary method of accessing the records.
Record. A collection of related fields. For example, the name, address,
etc. of an individual in a mailing list file would probably make up
Portability. That characteristic of
they are to access on
one record.
Relative record
a variety
number (RRN). An
record relative to the beginning of
records, the
RRN
index giving the position of
its file.
If a file has fixed-length
can be used to calculate the byte
offset
of
record
so the record can be accessed directly.
Self-describing
files.
Files that contain
information such
as the
number
and formal descriptions of the file's record
structure, which can be used by software in determining how to
of records in the
cess the
file.
file
file's
header
is
good
ac-
place for this information.
Sequential access. Sequential access to a file means reading the file from
the beginning and continuing until you have read in everything that
you need. The
alternative
is
direct access.
Sequential search. A method of searching a file by reading the file
from the beginning and continuing until the desired record has been
found.
Stream of bytes. Term
describing the lowest-level view of a
begin with the basic stream-of-bytes view of
a file,
we
file.
If
we
can then impose
own higher levels of order on the file, including field, record,
and block structures.
Variable-length record. A file organization in which the records have
no predetermined length. They are just as long as they need to be,
hence making better use of space than fixed-length records do. Unfortunately, we cannot calculate the byte offset of a variable-length
record by knowing only its relative record number.
our
EXERCISES
1.
Find situations for which each of the four
the text
might be appropriate.
field structures
described in
Do the same for each of the record structures
described.
2.
Discuss the appropriateness of using the following characters to delimit
fields or records: carriage return, linefeed, space,
comma,
period, colon,
EXERCISES
Can you
escape.
think of situations in which you might want to use
different delimiters for different fields?
Suppose you want to change the programs in section 4.1 to include
phone number field in each record. What changes need to be made?
3.
4. Suppose you need to keep a file in which every record has both fixedand variable-length fields. For example, suppose you want to create a file of
employee records, using fixed-length fields for each employee's ID
(primary key), sex, birthdate, and department, and using variable-length
fields for each name and address. What advantages might there be to using
such a structure? Should we put the variable-length portion first or last?
Either approach is possible; how can each be implemented?
5.
One
record structure not described in
labeled record structure each field that
describing
ZP
and
its
is
this
chapter
represented
is
is
called labeled. In a
preceded by
a label
LN, FN, AD, CT, ST,
fixed-length fields for a name and
contents. For example, if the labels
are used to describe the six
address record,
it
might appear
as follows:
LNAmesbbbbbbFNJohnbbbbbbAD123 Map 1 ebbbbbbCTSt
i 1 1
Under what conditions might
even desirable, record
this
be
a reasonable,
water STDKZP74075bbbb
structure?
6.
Define the terms stream of bytes, stream offields, and stream of records.
7. Find out what basic file structures are
programming language that you are currently
available
to
you
in
the
using. For example, does
your language recognize a sequence-of-bytes structure? Does it recognize
of text? Record blocking? For those types of structures that your
language does not recognize, describe how you might implement them
using structures that your language does recognize.
lines
Report on the basic
and record structures available
in
PL/I or
9. Compare the use of ASCII characters to represent everything
with the use of binary and ASCII data mixed together.
in a file
8.
field
COBOL.
10. If
you
list
the contents of a
file
containing both binary and ASCII
on your terminal screen, what results can you expect? What
happens when you list a completely binary file on your screen? (Warning: If
you actually try this, do so with a very small file. You could lock up or
reconfigure your terminal, or even log yourself off!)
characters
11. If a
key
in a record
is
already in canonical form and the key
is
the
first
147
48
FUNDAMENTAL
of the record,
field
it is
FILE
STRUCTURE CONCEPTS
possible to search for a record
separating out the key field
from the
rest
of the
by key without ever
fields.
Explain.
1985) that primary keys should be
unchanging, unambiguous, and unique." These concepts are
interrelated since, for example, a key that contains data runs a greater risk
12.
has been suggested (Sweet,
It
"dataless,
of changing than a dataless key. Discuss the importance of each of these
concepts, and show by example how their absence can cause problems. The
primary key used in our example file violates at least one of the criteria.
How might you redesign the file (and possibly its corresponding information content) so primary keys satisfy these criteria?
13.
How many comparisons
would be required on average
using sequential search in a 10,000-record disk
file,
how many
comparisons are required?
are stored per block,
if
only one record
14. In
is
how many
If the file
is
to find a record
record
is
not in the
blocked so 20 records
disk accesses are required on average?
What
stored per block?
we assume that
do the assumptions change on a
magnetic disk? How do these changed
our evaluation of performance for sequential search,
every read results in
assumptions
seek.
affect the analysis
Look up
How
to a
of sequential searching?
UNIX commands grep,
What motivates the differences?
the differences between the
Why
and fgrep.
machine with access
single-user
15.
file? If the
are they different?
egrep,
16. Give a formula for finding the byte offset of a fixed-length record
which the RRN of the first record is 1 rather than 0.
17.
Why
is
in
variable-length record structure unworkable for the update
program? Does
it
help if
we have
an index that points to the beginning of
each variable-length record?
18.
The
How
update
must the
deletion if
program
file
we do
lets
the user change records, but not delete records.
structure and access procedures be modified to allow for
not care about reusing the space from deleted records?
How do the file structures and procedures
change
if
we do want
to reuse the
space?
19. In
our discussion of the uses of relative record numbers (RRNs), we
file in which there is a direct correspondence
suggest that you can create a
as membership number, and RRN, so we can
by knowing just the name or membership number.
What kinds of difficulties can you envision with this simple correspondence
between membership number and RRN? What happens if we want to delete
between
primary key, such
find a person's record
EXERCISES
name? What happens
variable-length record
20.
The following
file
filled in.
How
long
dump
type produced by the
is
we change the information
and the new record is longer?
if
file
describes the
first
in a record in a
few bytes from
a file
version ofwritrec, but the right-hand column
the
record?
first
What
are
its
of the
is
not
contents?
0000000 00264475 6D707C46 7265647C 38323120
0000020 4B6C7567 657C4861 636B6572 7C50417C
0000040 36353533 357C2E2E 48657861 64656369
21.
Assume
we have
that
variable-length record
(greater than 1,000 bytes each,
with
for a record
file
on the average). Assume
a particular
RRN.
with long records
that
we
are looking
Describe the benefits of using the
a byte count field to skip sequentially from record to record to
one we want. This is called skip sequential processing. Use your
knowledge of system buffering to describe why this is useful only for long
records. If the records are sorted in order by key and blocked, what
information do you have to place at the start of each block to permit even
contents of
find the
faster skip sequential processing?
Suppose you have a fixed-length record with fixed-length fields, and
the sum of the field lengths is 30 bytes. A record with a length of 30 bytes
would hold them all. If we intend to store the records on a sectored disk
with 512-byte sectors (see Chapter 3), we might decide to pad the record
out to 32 bytes so we can place an integral number of records in a sector.
Why would we want to do this?
22.
23.
Why
is
it
important to distinguish between
file
access
and
file
organization?
What is an abstract data model? Why did the early file processing
programs not deal with abstract data models? What are the advantages of
using abstract data models in applications? In what way does the UNIX
concept of standard input and standard output conform to the notion of an
abstract data model? (See "Physical and Logical Files in UNIX" in Chap24.
ter 2.)
25.
What
is
26. In the
about the
scientific
metadata?
some metadata provides information
and some provides information about the
which the corresponding image was recorded. Give
FITS header
files's
in Fig. 4.15,
structure,
context in
three examples of each.
149
50
FUNDAMENTAL
FITS header
27. In the
program
to
determine
STRUCTURE CONCEPTS
FILE
in Fig.
how
the block containing the header
large
is
the
file?
4.15, there
must be
What proportion of the
enough information for a
Assuming that the size of
is
to read the entire
file.
of 2,880 bytes,
a multiple
file
we
28. In the discussion of field organization,
list
"keyword = value"
the
How
construct as one possible type of field organization.
applied in tagged
object-oriented
file
file
structures?
access?
How
How
does
do tagged
how
contains header information?
tagged
file
file
is
this
notion
structure support
formats support extensibil-
ity?
29. List three factors that affect portability in
ways
30. List three
31.
What
is
that portability can be achieved in
XDR? XDR
is
described in this chapter. If
"Further Readings"
ways
that
it
files.
at
actually
much more
you have
the end of this
files.
extensive than what
we
XDR
documentation (see
look
chapter),
up XDR and list the
access to
supports portability.
we see two possible record structures for our address file,
one based on C and one based on Pascal. Discuss portability problems that
might arise from using these record structures in a heterogeneous
computing environment. (Hint: Some compilers allocate space for character
fields starting on word boundaries, and others do not.)
32. In Fig. 4.2,
Programming Exercises
33. Rewrite writstrm so
the
new
it
uses delimiters as field separators.
The output of
version of writstrm should be readable by readstrm.c or readstrm .pas
34. Create versions
of
writrec
and
readrec that use the
following fixed-field
lengths rather than delimiters.
Last name:
15 characters
name:
15 characters
First
Address:
30 characters
City:
20 characters
State:
2 characters
Zip:
5 characters
35. Write the
Make
36.
it
program described
in the preceding
store five records per block.
Implement the program find.
problem so
it
uses blocks.
EXERCISES
37.
Rewrite the program find so
file. For example,
position in the
can find
it
if
record on the basis of
requested to find the 547
th
its
record in
would read through the first 546 records, then print the contents of
th
record. Use skip sequential search (see exercise 21) to avoid
the 547
reading the contents of unwanted records.
file, it
program
38. Write a
similar to find, but with the following differences.
from the keyboard, the program reads them
Instead of getting record keys
from
a separate transaction
file
that contains only the keys of the records to
be extracted. Instead of printing the records on the screen, it writes them
out to a separate output file. First, assume that the records are in no
Then assume
by key. In the latter
than^m^?
both the main
particular order.
that
are sorted
case,
efficient
39.
Make any
a.
or
all
how
can you
file
and the transaction
file
make your program more
of the following alterations to
update. pas or update. c.
Let the user identify the record to be changed by name, rather
than
RRN.
Let the user change individual fields without having to change an
b.
entire record.
c.
40.
Let the user choose to view the entire
Modify
file.
update. c or update. pas to signal the user
when
record exceeds
The modification should allow the user to bring the
and input it again. What are some other
would make the program more robust?
the fixed-record length.
record
down
to an acceptable size
modifications that
Change update. c or update. pas to a batch program that reads a transaction
file in which each transaction record contains an RRN of a record that is to
be updated, followed by the new contents of the record, and then makes the
41.
changes in
batch run. Although not necessary,
the transaction
file
it
might be desirable
to sort
by RRN. Why?
42. Write a
program
dump. The
file
and outputs the file contents as a file
format similar to the one used in the
examples in this chapter. The program should accept the name of the input
file on the command line. Output should be to standard output (terminal
dump
that reads a
file
should have
screen).
43.
Develop
a set
of rules for translating the dates August
7,
1949,
1949, 8-7-49, 08-07-49, 8/7/49, and other, similar variations into
Aug.
7,
common
canonical form. Write a function that accepts a string containing a date in
one of these forms and returns the canonical form, according to your rules.
Be sure to document the limitations of your rules and function.
151
52
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
program to read in a FITS file and print
The size of the image (e.g., 256 by 256)
The title of the image
The telescope used to make the image
The date the image file was created
The average pixel value in the image (use BSCALE and BZERO)
44. Write a
a.
b.
c.
d.
e.
FURTHER READINGS
Many
textbooks cover basic material on field and record structure design, but only
few go into the options and design considerations in much detail. Teorey and Fry
(1982) and Wiederhold (1983) are two possible sources. Hanson's (1982) chapter,
"Choice of File Organization," is excellent but is more meaningful after you read
the material in the later chapters of this text. You can learn a lot about alternative
types of file organization and access by studying descriptions of options available in
certain languages and file management systems. PL/I offers a particularly rich set of
alternatives, and Pollack and Sterling (1980) describe them thoroughly.
Sweet (1985) is a short but stimulating article on key field design. A number of
interesting algorithms for improving performance in sequential searches are
described in Gonnet (1984) and, of course, Knuth (1973b). Lapin (1987) provides a
detailed coverage of portability in UNIX and C programming. For our coverage o{
XDR, we used the documentation in Sun (1986).
Our primary source of information on FITS is not formally printed text, but
online materials. A good paper defining the original FITS format is Wells (1981).
The FITS image and FITS header shown in this chapter, as well as the
documentation of how FITS works, can (at the time of writing, at least) be found
on an anonymous ftp server at the INTERNET address 128.183.10.4.
a
PROGRAMS:
153
FILEIO.H
C Programs
The
programs
listed in the
discussed in the text.
are contained in the following
name and
Writes out
writstrm.c
following pages correspond to the programs
The programs
address information as
files.
stream of con-
secutive bytes.
input and prints
readstrm.c
Reads
writrec.c
Writes a variable-length record file that uses a byte count
the beginning of each record to give its length.
readrec.c
Reads through a file, record by record, displaying the
from each of the records on the screen.
stream
file as
it
to the screen.
at
fields
Contains support functions for reading individual records or
These functions are needed by programs in readrec.c
and find, c
getrf.c
fields.
Searches sequentially through a
find.c
file
for a record with a partic-
ular key.
Combines
makekey.c
first
and
last
names and converts them
canonical form. Calls strtrimf
and
ucase(
),
to a
found
key
in
in strjuncs.c.
strfuncs.c
Contains two string support functions: strtrimf ) trims the
blanks from the ends of strings; ucase( ) converts alphabetic
characters to uppercase.
update.
Allows new records to be added
to a file or old records to
be
changed.
Fileio.h
All of the
programs include
useful definitions.
were
/*
fileio.h
to be run
on
Some of
a
UNIX
header
file
called fileio.h
which contains some
programs
these are system dependent. If the
system
fileio.h
might look
like this:
header file containing file I/O definitions
*/
(continued)
54
FUNDAMENTAL
ude <stdio.h>
<fcntl .h>
nc
nc lude
STRUCTURE CONCEPTS
FILE
'define PMDDE
0755
#define DELIM_STR
'define DELIM_CHR
#define out_s t
"I"
j
r ( f d
wr
5 )
e(
fd )
( s )
5 t r
write((fd),DELIM_STR,1
'define
d_t o_recbuf f ( rb
treat ( rb
en(
5 ) )
Id)
s t r
ca
t (
rb DEL IM_STR)
,
'define MAX_REC_SIZE 512
fw ritstrm.c
writstrm.c
creates name and address file that is strictly a stream of
bytes (no delimiters, counts, or other information to
distinguish fields and records).
/*
simple modification, to the out_str macro:
\
wr i t e( ( f d) ( 5 ) s t r len( s ) )
#define ou t_s t r ( f d 5 )
write((fd),DELIM_STR,1 );
changes the program so that it creates delimited fields.
*/
'include "fileio.h"
#def ine
ma
ou t_s
)
t r (
fd
write((fd),(s),strlen(s))
s )
char firstC303, last[30], address[30], city[20];
char stateM5], z i p 9
char f i lename [15];
[
nt
fd
pr intf ("Enter the name of
get s( f i lename)
if
creaUf
((fd =
pr int
("f
exitd);
>
i 1
the file you wish to create:
PMODE )
i 1 ename
opening error
,
<
0)
program
opped\n" )
");
printf("\n\nType
get s( last
>
in a last name
PROGRAMS: READSTRM.C
155
(surname), or <CR> toexit\n>>>");
while (strlen(last)
0)
>
printf ("\nFirst Name:");
gets(first);
printf ("
Address:")
gets (addres s )
City:")
printf ("
gets(city)
printf ("
State:")
gets(state)
printfC"
Zip:")
;
get s(
ip)
/output the strings to the buffer and then to the file*/
out_5 1 r(fd,last);
out _5 1 r (f d first)
out _5 t r ( f d address)
out_str(fd,city)
out _str(fd, state)
out_5 t r ( f d z ip)
;
/* prepare for next entry */
printf("\n\nType in a last name (surname), or <CR> toexit\n>>>");
get s( las
t )
>
/* close the file before leaving
close(fd)
*/
Hieadstrm.c
/ *
reads t rm
reads
stream of delimited fields
*/
^include "fileio.h"
int readf ield( int fd, char sM);
mai n(
int
f d
30
char
char
int
lename
1 d_count
f
f
[ 1
5
(continued)
156
FUNDAMENTAL
STRUCTURE CONCEPTS
FILE
printf (" Enter name of file to read: ");
ge t 5 ( f i 1 ename )
if ((fd = openCf i 1 ename Q_RD0NLY ) ) < 0) {
printf ("fi le opening error
program
,
opped\ n"
exitd);
main program loop -- calls readfieldC ) for as long
as the function succeeds
f ld_count =
while ((n = readf ield(fd,s)) > 0)
printf ("\tfield # %3d:
%s \n" + + f 1 d_count s )
/*
close(fd)
>
int
eadf
d(
char
fd,
nt
sM)
int
char
-
0;
while
s
sti
read(fd,&c
(
[
++
'
\0
,1 )
>
&&
!=
DELIM_CHR)
/*
'
append null to end string
*/
return (i);
Pvv
Writrec.c
/ *
writrec.c
creates name and address file using fixed length (2-byte)
record length field ahead of each record
*/
#include "fileio.h"
char recbuf f MAX_REC_S ZE +11;
= {
char *prompt
'Enter Last Name -- or <CR> to exit
First name
[
'"
>;
Address
City
State
Zip
/* null string to terminate the prompt loop */
*/
ma
C
[
57
f d, i
shor
pr
char response 50
char f i 1 ename
5
i
PROGRAMS: WRITREC.C
rec_lgt h
nt
"Ent er the name of the file you wish to create:
"
gets(filename);
if
((fd - creat(f
l 1
ename PMODE
,
) )
<
printfCfile opening error
exitd);
0)
program
opped \ n"
>
prmtf ("\n\n
get
7.
s", prompt
>
response )
while
( s t r 1
en( response
0)
\0
recbuf f [03 =
1 d_t o_r ecbuf f (recbuff
response)
for (i=1; *prompt[i] != '\0'
i + +)
'
prmtf ("7.s", prompt
get
response)
d_t o_r ecbuf f (recbuff response)
;
/* write out the record length and buffer contents
rec_lgth = s t r 1 en( recbuf f )
write(fd &rec__lgth sizeof(rec_lgth))
write(fd, recbuff ,rec_lgth)
;
/* prepare for next entry */
printf ("\ n\n %s" prompt [01 )
gets(response)
;
>
close the file before leaving
close(fd)
/*
*/
>
/*
question:
How does the termination condition work in the for loop:
for (i-1; *promptEi] != \0
i++)
;
What does the
*/
"
"
refer to?
Why do we need the "*"?
*/
58
/ *
FUNDAMENTAL
STRUCTURE CONCEPTS
FILE
Readrec.c
readrec
...
reads through a file, record by record, displaying the
fields from each of the records on the screen.
/
^include "fileio.h"
main(
rec_count
fld_count;
5can_po5
short rec_lgth;
char f i 1 ename
5
char recbuff MAX_REC_S IZE +11;
char f ield[MAX_REC_SIZE +1];
fd,
int
int
pr
get
if
"En t er name of file to read:
ename)
((fd = open(f
s ( f
pr
"
i 1
nt
exitd
"f
i 1
ename 0_RD0NLY )
opening error
i 1
<
0)
program
s t
opped\n")
);
ec_count =
5can_po5 =
while
((rec_lgth = get_rec ( fd recbuff
r
) )
>
0)
printf ("Record %d\n" ++rec_count )
f 1 d_coun t =
while ((scan_po5 = get_f 1 d( f i e 1 d recbuff scan_pos
rec_lgth))
printf ("\tField %d: %s \n" + + f 1 d_count f i e 1 d )
,
>
>
/*
lose(f d
question -- why can
to scan_pos just once, outside
assign
*/
of the while loop for records?
I
C PROGRAMS: GETRF.C
59
rsetrf.c
/*
get rf
...
Two functions used by programs
get_rec(
get_fld(
in
readrec.c and find.c:
reads a variable length record from file fd
into the character array recbuff.
moves a field from recbuff into the character
array field, inserting a '\0' to make it a
string.
*/
'include "fileio.h"
get_rec(int fd, char recbuffM)
{
short
rec__lgt h
(readCfd, &rec_lgth, 2) == 0) /* get record length */
/* return
return(O);
if EOF */
/* read record */
rec_lgth = readCfd, recbuff, rec_lgth);
return(rec_lgth)
if
get_fld(char field [
char recbuffM
short scan_pos
short rec_lgth)
short fpos = 0;
if
/*
position in "field" array
*/
(scan_po5 == rec_lgth) /*if no more fields to read,*/
return(O);
/*return scan_pos of 0.*/
/* scanning loop */
while ( 5can_po5 < rec_lgth &&
(f ieldCfpos++] = recbuff [scan_pos++]
!=
DELIM_CHR)
ifCfieldtfpos-1] = =DELIM_CHR)/*if last character
f
ieldC --f po5
'XO
is a field*/
/*delimiter, replace with null*/
else
fieldCfpos] =
ret urn( 5can_po5
/*otherwise, just ensure that
the field is nu 1 1 - ermi na t ed*
/*return position of start of next field*/
'\0';
60
FUNDAMENTAL
STRUCTURE CONCEPTS
FILE
Find.c
/*
find.c ...
searches sequentially through
particular key.
file for
record with
*/
^include "fileio.h"
#define TRUE
#define FALSE
1
main(
fd, scan_pos;
short r ec_l gt h
int mat ched
char search_key[30]
k ey_f ound
30
char f i 1 ename
5
char recbuff MAX_REC_S ZE +13;
char f ield[MAX_REC_SIZE +11;
int
[ 1
pr i nt
get s (
if
pr
lastC30], firstC303;
"En t er name of file to search:
ename )
((fd = open(f
f
");
i 1
nt
exitd
"f
i 1
ename D_RD0NLY )
opening error
i 1
<
0)
program
s t
opped\n")
);
>
("\n\nEnter last name: ");
gets(last);
pr i nt f ("\nEnt er first name: " )
gets(first);
makekey(last, first, search_key);
pr int
/*
get
search key */
matched = FALSE;
while
(Imatched && (rec_lgth = get_r ec ( fd recbuff
) )
>
5can_po5 =
recbuff, scan_pos, rec_lgth);
5can_po5 = get_f ld( las t
recbuff, scan_pos, rec_lgth)
5can_po5 = get_f ld(f i r s t
first, key_found);
mak ekey( las t
if (strcmp (key_found, search_key) == 0)
matched = TRUE;
;
>
/*
if
{
if
record found, print the fields
(ma t ched
*/
161
PROGRAMS: MAKEKEY.C
pnntf("\n\nRecord found :\n\n");
5can_po5 =
/*
wh
break out the fields */
e( ( 5can_po5 = get_f ld(field,recbuff, scan_pos ,rec_lgth))>0)
printf ("\t%s\n'\ field)
i 1
else
printf ("\n\nRecord not found. \n");
>
>
ques tions:
/ *
-why does scan_pos get set to zero inside the while loop here?
-what would happen if we wrote the loop that reads records
((rec_lgth = ge t_r ec ( f d ecbuf f ) ) >
&&
like this: while
ma t ched )
,
*/
Makekey.c
/*
mak ek ey( las
r s t
s )
...
function to make a key from the first and last names passed
Returns the key in
through the functions arguments.
canonical form through the address passed through the
argument s.
Calling routine is responsible for ensuring
that s is large enough to hold the return string.
Value returned through the function name
the string returned through s.
is
the
length of
*/
makekey(char lasttl, char
irst
[ ]
,char
sM)
{
i
nt
en
=
enf
/* trim the last name */
im( las t )
/* place it in the return string
las t )
/* append a blank at the end */
';
s[lenl++] =
=
s[ lenl
\0
/* trim the first name */
lenf = s t r t r im( f i r s t )
/* append it to the string */
s
rcat ( s f i r s t )
lenl
s t
s t r t r
cpy (
'
'
/* convert everything to uppercase
ucase(s,s);
returnClenl + lenf);
*/
*/
62
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
Strfuncs.c
/ *
51
rf uncs . c
module containing the following functions:
strtrim(s) trims blanks from the end of the ( nu 1 1 - ermi nat ed)
string referenced by the string address s.
Nhen
done, the parameter s points to the trimmed string
The function returns the length of the trimmed
string.
ucase(si,so) converts all lowercase alphabetic characters in
the string at address si into uppercase characters
returning the converted string through the address
so
*/
strtrimC char s[])
{
i
nt
for
(i
strlen(s)-1;
/* now that
to form a
==
'
i--)
the blanks are trimmed, reaffix null on the end
string */
= \0
s[++i
return(i);
]
i>=0 &&
ucase (char siM, char
soM)
while (*5o++ = (*5i >=
5i + +
'a'
&&
*si
<=
'z')
*si
&
0x5f
*si)
>
fupdate
/ *
update c ...
program to open or create a fixed length record file for
Records to be
updating.
Records may be added or changed.
changed must be accessed by relative record number
.
*/
^include "fileio.h"
#define REC_LGTH 64
static char *promptM = {"
Last Name:
"
First name
Address:
City:
State:
163
PROGRAMS: UPDATE.C
"
Zip:
mi
static int fd;
static struct {
short
rec_count
char
fill[30];
>
head
static
static
static
static
static
menu(
recbuffM);
read_and_show(
change( )
ma i n(
int
int
as k_i nf o( char
as k_r r n( )
);
menu_choice
rn
by t e_po5
char f i lename
5
long 1 seek ( )
char recbuf f MAX_REC_S ZE + 1];/*buffer to hold
[ 1
record*/
printf ("Enter the name of the file: ");
g e t 5 ( f i lename);
if (( fd = openCf ilename, 0_RDWR)) < 0) /*if OPEN fails*/
{
fd = creatCf ilename, PMODE);
head rec_count = 0;
wr i te(fd Ahead s i zeof (head) )
.
/*then CREAT*/
/*initialize header */
/*write header rec*/
>
/* existing file opened -- read in header */
read(fd &head,sizeof(head));
/* main program loop -- call menu and then jump to options */
whi le( (menu_choice = menu( )) < 3)
else
swi
ch(menu_cho ice)
case
printfC Input
/* add a new record */
the informationfor the new record --\n\n M
ask_info(recbuff)
by t e_po5 = head.rec_count * REC_LGTH
1 seek ( f d
(
ong ) byte_pos,0);
);
sizeof(head);
(continued)
64
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
write(fd,recbuff REC_LGTH);
head r ec_count + +
,
break
case
/*
2:
rrn = as k_r rn(
/*
if
update existing record */
if rrn is too big, print error message
(rrn >= head rec_coun t ) {
pr i nt f "Record Number is too large");
printfC... returning to menu ...");
break
...
*/
>
/* otherwise, seek to the record ... */
byte_po5 = rrn * REC_LGTH + s i zeof (head )
1 seek ( fd ,( long)
byte_pos,0);
display it and ask for changes
read_and_show( );
(change( )
i f
/*
...
*/
pr int f ("\n\n nput the revised Va lues \n\n")
ask_info(recbuff)
lseek(fd,(long) byte_pos,0);
write(fd,recbuff ,REC_LGTH);
I
>
break
/* end swi tch */
>
/* end while */
;
>
rewrite correct record count to header before leaving
lseek(fd,OL,0);
write(fd,&head,sizeof(head))
close(fd);
/*
>
/*
menu( ) ...
local function to ask user for next operation.
Returns numeric value of user response
*/
static menu( ) {
int choice;
char response [10
1;
printf ("\n\n\n\n
FILE UPDATING PROGRAMNn" )
pr intf ("\n\nYou May Choose to:\n\n");
printf("\t1.
Add a record to the end of the file\n");
pr intf ("\ t2
Retrieve a record for Upda t i ng\n")
pr int f ( "\ t 3
Leave the Pr ogram\n\n" )
.
*/
C PROGRAMS: UPDATE.C
pr i nt f ("Enter the number of your
get s( response)
choice = atoi(response);
ret ur n( cho ice)
65
");
choice:
>
/ *
as k_i nf o(
...
function to accept input of name and address fields,
writing them to the buffer passed as a parameter
local
*/
static as k_i nf o( char recbuffM)
{
fie ld_count
i nt
char response 50
,
i
]
for
clear the record buffer */
(i = 0; i < REC_LGTH; r ecbuf f
/*
get
/*
for
the fields
(i=0;
++
'\0)
*/
*prompt[i]
!=
'NO
i++)
printf ( %s" prompt[i]);
ge t s response )
f 1 d_t o_r ecbuf f (recbuff response)
,,
>
>
/*
as k_r rn(
...
function to ask for the relative record number of the
record that is to be updated.
local
*/
static as k_r r n(
i
nt
r r
char response
[ 1
M \n\n nput
i nt f
the Relative Record Number of the Record
that\n");
");
pr i nt f ("\ tyou want to update:
get s( response)
rrn = atoi(response);
return(rrn)
pr
/*
read_and_show( ) ...
Note that this
local function to read and display a record.
function does not include a seek -- reading starts at the
current position in the file
*/
static
ead_and_show(
(continued)
66
FUNDAMENTAL
FILE
STRUCTURE CONCEPTS
char recbuff [MAX_REC_S ZE +
I
int
MAX_REC_S ZE +13;
I
scan.pos, data_lgth;
5can_po5 =
readCfd, recbuff ,REC_LGTH);
;
M \n\n\n\nExi 5 1 i
i nt f
ng Record Content 5 \n" )
/* ensure that record ends with
\
recbuff CREC_LGTH] =
pr
'
null
*/
data_lgth = strlen(recbuff);
while ((scan_po5 = get_f 1 d( f eld recbuff scan_pos
data_lgth))
,
printf
>
0)
\t%s\n", field)
>
/ *
changeC
...
local function to ask user whether or not he wants to change
otherwise
if the answer is yes,
the record.
Returns
1
*/
{
static change( )
char response [
(
"\n\nDo you want to change this record?\n");
i nt f
Answer Y or N, followed by <CR> ==>");
printfC"
gets(response)
ucaseC response, response);
returnCCresponseCO] '== Y') ?
0);
pr
167
PASCAL PROGRAMS
Pascal Programs
The
Pascal programs listed in the following pages correspond to the
programs discussed in the text. Each program is organized into one or more
files, as
follows.
writstrm.pas
Writes out
name and
address information as a stream of con-
secutive bytes.
readstrm.pas
Reads
writrec.pas
Writes a variable length record file that uses a byte count
the beginning of each record to give its length.
readrec.pas
Reads through a file, record by record, displaying the
from each of the records on the screen.
stream
file as
input and prints
it
to the screen.
Supports functions for reading individual records or
get. pre
These functions are needed by the program
Searches sequentially through a
find. pas
file
at
fields
fields.
in readrec.pas.
for a record
with
a partic-
ular key.
Allows new records
update. pas
to be
added to
file,
or old records to be
changed.
Support function for update. pas, which converts
type strng to a variable of type datarec.
stod.prc
In addition to these
the tools for operating
contained in Appendix
files,
on
there
is
a file called tools. pre,
variables of type strng.
at
a variable
of
which contains
listing
of
tools. pre is
the end of the textbook.
We have added line numbers to some of these Pascal listings to assist the
reader in finding specific
The
files
program statements.
do not contain
and stod.prc.
that contain Pascal functions or procedures but
main programs
are given the extension
.pre, as in get. pre
168
PASCAL PROGRAMS
Writstrm.pas
Some
things to note about writstrm.pas:
The comment {SB-} on
piler, instructing
Without
it
line 6
to handle
this directive
a directive to the
is
keyboard input
we would
function properly in the
WHILE
as a
Turbo
Pascal
com-
standard Pascal
file.
not be able to handle the len_str()
loop on line 36.
The comment {$1 tools. pre} on line 24 is also a directive to the
Turbo Pascal compiler, instructing it to include the file tools. pre in
the compilation. The procedures read_str, len_str, and fwrite_str are
the
in
file tools. pre.
we choose not
conforming to standard Passtrng type, which is a packed array
The length of the strng is stored in
Although Turbo Pascal supports
come
to use that type here to
a special string type,
closer to
cal. Instead, we create our own
{O..MAX_REC_SIZE} of char.
the zeroth byte of the array as a character value. If
value in the zeroth byte of the array, then
ORD(X)
X
is
is
the character
the length of
the string.
The assign statement on line 31 is one that is nonstandard. It is
Turbo Pascal procedure, which, in this case, assigns filename to
file,
so
further operation
all
PROGRAM writstrm
on
outfile will
operate on the disk
out-
file.
NPUT OUTPUT)
,
writes out name and address information as
consecutive bytes >
stream of
{$B->
Directive to the Turbo Pascal compiler, instructing
handle keyboard input as a standard Pascal file >
8
9
1
CONST
DELIM_CHR
MAX_REC_SIZE
11
'I';
=
255;
12
13
14
TYPE
strng
inp_list
filetype
15
16
17
18
19
20
21
22
23
=
=
MAX_REC_S ZE of char;
packed array
first address ,c ity, state zip)
( last
packed arrayC1..40l of char;
[
VAR
response
resp_type
filename
outfile
array [inp
inp_list;
filetype;
text;
list]
of
strng
it
to
PASCAL PROGRAMS: READSTRM.PAS
{$1
24
25
26
27
28
tools. pre)
Another directive,
too Is. pre
30
31
instructing the compiler to include the file
);
32
33
writeC'Type in a last name, or press <CR>
read 5 trCresponset last )
while ( 1 en_5 t r response las t ) >
)
DO
BEGIN
{
get all the input for one person }
write(
First Name: ');
r ead_5 tr(response[first
)
write(
Addr es s
)
read s trCresponsetaddress] )
34
35
36
37
38
39
40
to
exit:
);
'
41
'
42
43
44
45
46
47
48
49
writeC
City: );
ead s trCresponsetcity] )
writeC'
State: ');
r ead_5 trCresponset state]
writeC'
Zip: ');
r
)-,
read_5 trCresponselzip]);
write the responses to the file }
resp_type := last TO zip DO
f wr i t e_5 trCoutfile, response! resp_type
50
51
for
52
53
54
start the next round of
input
writeC'Type in a last name, or press <CR>
read_5 t rC response [last])
55
56
57
58
59
69
BEGIN {main}
writeC Enter the name of the file:
readlnCfi lename)
assignCoutfi le f i lename)
rewriteCoutfi le)
29
to
exit:
');
END;
closeCoutfile)
END.
Readstrm.pas
PROGRAM readstrm
{
NPUT OUTPUT)
,
program that reads a stream file Cfields separated by
delimiters) as input and prints it to the screen >
A
CONST
DELIM_CHR
MAX_REC_SIZE
!
=
255;
(continued)
170
PASCAL PROGRAMS
TYPE
5 1
rng
letype
packed array
MAX_REC_S ZE
packed array M..40] of char;
lename
nf
of
char;
VAR
f
f
letype
str
integer
i nt eger
5 1 rng
ld_count
ld_len
{$1
text
le
tools. prc>
FUNCTION readfield (VAR infile
{
text; VAR str
strng):
int eger
Function readfield reads characters from file infile until it
reaches end of file of a "I".
Readfield puts the characters in
str and returns the length of str >
VAR
i
ch
integer;
char
BEGIN
i
:=
0;
ch
'
while (not EOF(infile)) and (ch <> DELIM_CHR) DO
BEGIN
read (infile, ch);
i
s t r
=
[
ch
END;
i
:=
strCOJ := CHR(i);
readf ield
=
i
END;
:
BEGIN {MAIN}
write ('Enter the name of the file that you wish to open:
readln (filename);
assign(infile,filename);
reset (infile);
f
ld_coun t
');
fld_len := readf ie 1 d( i nf i le 5 t r )
while (fld_len > 0) DO
BEGIN
fld_count := fld_count +
2)
writeC field #
f 1 d_count
{
write_str()
wr i te_st r(st r )
,
'
'
is
in
tools. pre
PASCAL PROGRAMS: WRITREC.PAS
fld_len
:=
END;
ose
( i
nf
readfield(infile str)
7
le)
END.
Writrec.pas
Note about
writrec.pas: After writing the rec_\gth to outfile
write a space to the
PROGRAM wntrec
This
is
NPUT OUTPUT)
,
on
line 69,
we
because in Pascal values to be read into
must be separated by
integer variables
1
file.
spaces, tabs, or end-of-line markers.
{$B->
CONST
DELIM_CHR =
MAX_REC_SIZE
8
9
TYPE
5;
255;
5trng = packed array
filetype = packed array
[
1
1
'I'
12
13
14
15
16
17
18
19
20
MAX_REC_S ZE
C 1
.40
of
of
char;
char;
VAR
filename
outfile
response
buffer
rec_lgth
{$1
filetype;
text;
5 t r
ng
strng;
integer;
too Is. prc>
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
PROCEDURE
C
ld_to_buf er(VAR buff:
This procedure concatenates
buff >
strng;
and
s:
strng);
delimiter to end of
VAR
strng;
d_5 t r
BEGIN
cat_str(buff,s);
d_str[0] := CHRC1 );
:= DELIM_CHR;
d_str[1
cat_str(buff d_s t r
:
END;
(continued)
172
37
38
39
40
41
PASCAL PROGRAMS
BEGIN {main}
write( 'Enter the name of the file you wish to create
readln(filename)
assignCoutfile,filename)
rewriteCoutfile);
');
42
43
44
45
46
47
48
49
50
writeC'Enter Last Name -- or <CR> to exit: );
read_5 t r( response)
while ( 1 en_st r response) > 0) DO
BEGIN
=
bufferCO]
CHR(O);
{Set length of string
in buffer to
>
f 1 d_t o_buf f er (buffer response)
;
51
writeC
52
53
54
55
56
57
58
59
60
ead_5 t r( response)
f 1 d__t o_b ufferCbuffer
response)
First name
writeC
Address:');
rea d_5 t r( response)
f 1 d_t o_buf f er (buffer response)
;
writeC
read_5 tr(response)
f 1 d_t o_buf f er (buffer response)
writeC'
r ead_5 trCresponse)
f 1 d_t o_buf f er Cbuffer response)
writeC'
read_5 t rC response)
f 1 d_t o_buf f er Cbuffer response)
City:
'
61
State:
);
62
63
64
65
66
67
68
69
Zip:');
write out the record length and buffer contents
{
rec_lgth := 1 en_s t r C buf f er )
writeCoutfile,rec_lgth)
wr i t eCout f i le
)
f wr i t e_5 1 rCoutfile, buffer);
>
70
71
'
'
72
73
74
75
76
77
78
{
prepare for next entry >
writeC'Enter Last Name -- or <CR> to exit:
read_5 t rC response)
END;
closeCoutfile)
END.
Readrec.pas
PROGRAM readrec
{
');
NPUT OUTPUT)
,
This program reads through a file, record by record, displaying
the fields from each of the records on the screen. >
PASCAL PROGRAMS: READREC.PAS
173
{$B->
CONST
=
i nput_5 i ze
255
DELIM_CHR = h
MAX_REC_SIZE = 255;
;
TYPE
npu t_5 i ze
of char;
strng = packed array
filetype = packed array M..40] of char;
[
VAR
f
i 1
ename
out f
le
ec_coun
5can_po5
{$1
{$1
ex t
nt eger
ec_l gt h
1
tools. pre
get. prc>
eger
i nt eger
i nt eger
strng;
strng;
i
d_coun
buffer
field
f
letype
BEGIN {main}
write( 'Enter name of file to read:
readln (filename);
as5ign(outfile,filename)
reset (outfile);
');
rec_count
=
5can_po5
=
rec_lgth := ge t_rec out f i le buff er )
while rec_lgth >
DO
BEGIN
,rec_count);
wr telnC Record
rec_count := rec_count +
f ld_count
=
5can_po5 := ge t_f 1 d( ield buf f er scan_pos r ec_l gt h )
while 5can_po5 >
DO
BEGIN
);
writeC
f 1 d_count
Field
writ e_5 tr(field);
fld_count := fld_count + 1;
5can_po5 := get_f 1 d( f i e 1 d buf f er scan_pos r ec_l g t h
:
'
END;
rec_lgth
:=
END;
close(outfile)
END.
get_r ec out f i le buf f er
74
PASCAL PROGRAMS
Get.prc
FUNCTION get_rec(VAR fd:
VAR buffer:
text;
strng);
integer;
{
A function that reads a record and its length from file fd.
The function returns the length of the record. If EOF is
encountered get_rec() returns
>
VAR
rec_lgth
integer;
space
char
BEGIN
if EOF(fd) then
get_r ec
=
else
BEGIN
readCfd ,rec_lgth)
r ead( f d
space)
f r ead_5 tr(fd, buffer, r ec_l gt h )
get_rec := rec_lgth
END
:
END;
FUNCTION get_fld(VAR
rng buff er
;
rec_lgth:
{
integer):
rng VAR scanpos: integer;
integer;
;
function that starts reading at scanpos and reads characters
from the buffer until it reaches a delimiter or the end of the
record. It returns scanpos for use on the next call. }
VAR
f
pos
nt eger
BEGIN
scanpos = rec_lgth then
get_fld :=
else
BEGIN
if
pos
scanpos := scanpos +
fieldCfposl := buf fer scanpos
while (f ieldtfpos] <> DELIM_CHR) and (scanpos
BEGIN
fpos := fpos
1;
scanpos := scanpos +1;
fieldCfposl := buf fer scanpos
1
<
rec_lgth)DO
END;
if
fieldCfpos]
f ieldCO]
:
DELIM_CHR then
CHRCfpos -
:=
=
,v~l-
PASCAL PROGRAMS: FIND.PAS
175
else
fieldCO] := CHR(fpos);
get_fld := scanpos
END
END;
Find. pas
PROGRAM find
i
NPUT OUTPUT)
,
This program reads through a file, record by record, looking
for a record with a particular key.
If a match occurs, when
all the fields in the record are displayed.
Otherwise a message
is displayed indicating that the record was not found. >
{$B->
CONST
MAX_REC_SIZE
DELIM_CHR =
255;
=
'I'
TYPE
strng = packed array
MAX_REC_S IZE of char;
filetype = packed array C1..403 of char;
[
VAR
f
i 1
ename
out f
last
le
first
search_key
length
matched
rec_l gt
buffer
5can_po5
key_f ound
field
$
{ $
tools, pre
get pre >
letype
ext
s t rng
s t rng
s t rng
t
nt eger
boolean
nt eger
strng
i nt eger
strng;
strng;
;
BEGIN {main}
writeC'Enter name of file to search:
readlnCf i lename)
);
as5ign(outfile,filename)
reset(outfile);
(continued)
76
PASCAL PROGRAMS
writeCEnter last name:
read_5 t
r (
las
');
t )
writeCEnter first name:
');
read_5 1 r(first);
makekeydast first sear ch_k ey )
,
matched := FALSE;
rec_lgth := ge t_rec out f i 1 e buff er )
while ((not matched) and (rec_lgth
Beg
>
0)) DO
=
5can_po5
5can_po5 := ge t_f 1 d( las t buff er scan_pos rec_l gth )
5can_po5 := get_f 1 d( f i r 5 t buf f er scan_pos rec_l gt h )
makekey(last first k ey_f ound )
if cmp_s t r ( key_f ound search_key ) =
then
matched := TRUE
else
rec_lgth := get_rec(out f i 1 e buf f er )
:
END;
lose( out
if
le)
record found, print the fields
mat ched t hen
BEGIN
wr i t e 1 n( Record found:');
if
wr
i t
>
5can_po5
{
break out the fields >
5can_po5 := ge t_f 1 d( f i e 1 d buf f er scan_pos r ec_l gt h )
while 5can_po5 >
DO
BEGIN
writ e_5 tr(field)
5can_po5 := get_f 1 d( f i e Id buf f er scan_pos rec_l gt h
,
END;
END
else
writeln(
'
Record not found.
);
END.
Update. pas
Some
things to note about update. pas:
name and address fields are read in as
and procedure Jld_to_buffer() writes the fields to strbuff (also of
type strng). Writing strbuff to outfile would result in a type mismatch,
since outfile is a file of type datarec. However, the procedure stod(),
In the procedure ask_info(), the
strngs,
177
PASCAL PROGRAMS: UPDATE.PAS
located in stod.prc, converts a variable of type strng to a variable of
type datarec to write the buffer to the
cated on
The
lines
seek() statements
on
dard; they are features of
PROGRAM update
file.
The
calls to stod() are lo-
210 and 237.
NPUT OUTPUT )
,
lines 212, 229, 239,
Turbo
and 250 are not stan-
Pascal.
{$B->
{
program to open or create a fixed length record file for
updating.
Records may be added or changed.
Records to be
changed must be accessed by relative record number }
CONST
MAX_REC_SIZE
REC_LGTH
DELIMCHR
255;
64;
'I';
TYPE
s t r
packed array
MAX_REC_S ZE of char;
packed array M..40] of char;
RECORD
len
integer;
data
packed array
REC_LGTH of cha
ng
filet ype
datarec
END;
VAR
f
i 1
ou
ename
f
i 1
response
menu cho ice
strbuff
by t
head
r r n
drecbuff
i
ec_count
{$1
{$1
tools. pr c
>
stod.prc
>
{ $
get
pre
PROCEDURE
{
d__t
s t r ng
integer
datarec
integer
datarec
integer
integer
pos
filetype;
file of datarec
char
i nt eger
;
>
ld_to_buf er(VAR buff: strng;
o_buf f er concatenates strng
end of buff
and
s:
strng);
delimiter to the
>
(continued)
178
PASCAL PROGRAMS
VAR
d_str
strng;
BEGIN
ca t_5 t r ( but f 5 )
d_str[0] := CHRC1);
d_strt
=
DELIM_CHR;
cat_str(buf f ,d_str)
:
END;
FUNCTION menu integer
{local function to ask user for next operation.
value of user response }
Returns numeric
VAR
choice
int eger
BEGIN
writeln;
writelnC
FILE UPDATING PROGRAM' )
writeln;
writelnC'You May Choose to: );
writeln;
writelnC'
Add a record to the end of the file');
writelnC'
2.
Retrieve a record for updating');
writelnC'
3.
Leave the program');
writeln;
writeC'Enter the number of your choice: ');
r eadl nCchoice)
writeln;
=
menu
choice
:
'
END;
PROCEDURE ask_infoCVAR strbuff: strng);
{local procedure to accept input of name and address fields
writing them to the buffer passed as a parameter >
VAR
response
strng;
BEGIN
{
clear the record buffer
clear_strCbuff)
:
get the fields }
{
writeC'
Last Name: ');
r ead_5 trCresponse)
f 1 d_t o_buf f er C strbuff response)
writeC'
First Name: ');
r ead_5 trCresponse)
buffer Cstrbuff response)
f 1 d
t o
;
179
PASCAL PROGRAMS: UPDATE.PAS
writeC
Address:
)
ead_5 tr(response);
f 1 d_t o_b ufferCstrbuff response)
94
95
96
97
98
99
100
1
01
02
City:
');
read_5 tr(response);
f 1 d_ to bufferCstrbuff, response);
writeC'
State: ');
r ead_5 tr( response);
f 1 d_t o_buf f er (strbuff
response);
writeC'
Zip:');
r ead_5 tr(response);
fid' to buff er C strbuff response);
,
03
04
07
writeC
105
106
1
'
'
wr
i t
END;
108
09
110
1
12
13
FUNCTION ask_rrn:
integer;
function to ask for the relative record number of the record
that is to be updated.
>
114
1
15
116
1
17
18
19
120
VAR
rrn
integer;
BEGIN
i t e 1 n(
npu t the relative record number of the record that');
writeC'
you want to update: ');
read ln(rrn)
wr
'
wr i t e 1 n
122
as k r un
=
rrn
123
END;
124
PROCEDURE read_and_show;
125
126 {procedure to read and display a record. This procedure does not
127
include a seek -- reading starts at the current file position
128
129 VAR
130
5can_po5
i n t eger
131
dr ecbuf f
da t ar ec
132
integer
133
data__l g t h
integer
134
field
5 t r ng
135
strbuff
5 t r ng
136
BEGIN
137
scan pos
138
readCoutfi le drecbuff)
139
140
<
convert drecbuff to type strng }
141
strbuffCO] := CHR( drecbuff 1 en )
142
for i :=
to drecbuff .len DO
121
>
(continued)
80
PASCAL PROGRAMS
143:
strbufffi] := dr ecbuf f da t a [ i
144:
145:
wr i t e 1 n( Ex i s t i ng Record Contents');
146:
writeln;
147:
148:
data_lgth := 1 en_s t r ( 5 rbuf f )
149:
5can_po5 := get_f 1 d( f i e 1 d 5 rbuf f scan_pos da ta_l g t h )
while scan_pos >
D)
150:
BEGIN
151:
write_5tr(f ield)
152:
153:
5can_po5 := ge t_f 1 d( f i e 1 d 5 rbuf f scan = pos,data:= lgth)
154:
END
155: END;
156:
157:
158: FUNCTION change: integer;
159:
160: { function to ask the user whether or not to change the
161:
record.
Returns
if the answer is yes,
otherwise.
162:
163: VAR
164:
char;
response
165: BEGIN
166:
writeln('Do you want to change this record?');
167:
wnteC
Answer Y or N, followed by <CR> = = >);
168:
readln(response);
1 69:
writeln;
y ) then
170:
if (response =
Y ) or (response =
=
171:
change
172:
else
73
change
=
174: END;
175: BEGIN {main}
176:
write( 'Enter the name of the file: ');
177:
read 1 n( f i 1 ename )
178:
ass i gn( out f i 1 e f i 1 ename )
179:
180:
write('Does this file already exist? (respond Y or N): ');
181:
read ln( response )
writeln
82:
183:
if (response = Y') OR (response = y') then
184:
BEGIN
185:
open outfile
>
r ese t ( ou t f i 1 e )
{
186:
get header
>
read(out f i le ,head)
{
187:
{
read in record count >
rec_count := head.len
188:
END
189:
else
190:
BEGIN
create outfile
}
191:
rewr te(outf i le)
(
initialize record count }
{
192:
rec_count := 0;
]
'
'
'
'
'
PASCAL PROGRAMS: UPDATE.PAS
193:
194:
195:
196:
197:
198:
199:
200:
201:
202:
203:
204:
205:
206:
20 7:
208:
209:
210:
211:
212:
213:
214:
215:
216:
217:
218:
219:
220:
221:
222:
223:
224:
225:
226:
227:
228:
229:
230:
231:
232:
233:
234:
235:
236:
237:
238:
239:
240:
241
rec_count;
to REC_LGTH DO
head.dataU] := CHR(O);
wr i te( out f i 1 e head )
head.len
for
:=
:=
place in header record
>
set header data to nulls)
write header rec
}
{
i
END;
main program loop -- call menu and then jump to options
{
menu_choice := menu;
while menu_choice < 3 DO
BEGIN
CASE menu_choice OF
1
>
add a new record >
i
BEGIN
writeln( Input the information for the new record --');
writeln;
writeln;
ask_info(strbuff );
{convert strbuff to type datarec}
stodCdrechbuf f st rbuf f )
rrn := rec_count + 1;
seek(outfile,rrn);
wr i t e( ou t f i 1 e dr ecbuf f )
rec count := rec count +
:
'
END;
2
update existing record
>
BEGIN
rrn
{
if
:=
ask_rrn;
rrn is too big, print error message ...
(rrn > rec_count) or (rrn < 1) then
BEGIN
wr i te( Record Number is out of range');
wr i teln( "... returning to menu...')
END
if
else
BEGIN
seek(outf
otherwise, seek to the record
le ,rrn)
...
>
display it and ask for changes
read_and_show
{
if
...
>
then
change =
BEGIN
writeln(' Input the revised Values: ');
ask_info(strbuff );
convert strbuff to type
{
stod(drecbuf f st rbuf f )
datarec }
seek(outf i le,rrn)
wr iteCoutf i le ,drecbuf f )
END
1
(continued)
82
PASCAL PROGRAMS
242
243
244
245
246
247
248
249
250
END;
menu_choice
END; { while
:=
menu
rewrite correct record count to header before leaving
{
head.len := rec
count;
seekCoutf i le 0)
write(outfile,head);
close(outfile)
,
251
252
253
END
END
CASE >
<
END
Stod.prc
PROCEDURE stod (VAR drecbuff: datarec; strbuff; strng);
{
A procedure that converts
variable of type datarec
variable of type strng to
>
VAR
i
i nt eger
BEGIN
drecbuff. len
:
for
:=
:=
mi n( REC_LGTH
to drecbuff. len DO
drecbuff. datati]
:=
:=
END;
:=
en_s
strbuffCi];
{
Clear the rest of the buffer
while i < REC_LGTH DO
BEGIN
drecbuff .datati]
END
'
'
r ( s t
rbuf f
) )
>
Organizing Files
for Performance
CHAPTER OBJECTIVES
Look
at several
Look
at storage
space in a
I
Develop
approaches to data compression.
compaction as a simple
way of reusing
file.
procedure for deleting fixed-length records
file space to be reused dynamically.
that allows vacated
I
Illustrate the use
avail
I
of
linked
lists
and
stacks to
manage an
list.
Consider several approaches to the problem of deleting
variable-length records.
Introduce the concepts associated with the terms
internal
fragmentation and external fragmentation
I
Outline some placement strategies associated with the reuse of space in a variable-length record file.
Provide an introduction to the idea underlying
a binary
search
I
Undertake an examination of the limitations of binary
searching.
Develop
a keysort
procedure for sorting larger
files;
in-
vestigate the costs associated with keysort.
I
Introduce the concept of
pinned
record.
183
CHAPTER OUTLINE
5.1
Data Compression
Using
Finding Things Quickly: An
Introduction to Internal Sorting
and Binary Searching
5.3
Notation
5.1.2 Suppressing Repeating
Sequences
5.1.3 Assigning Variable-length
5.1.1
a Different
Finding Things in Simple Field
and Record Files
5.3.2 Search by Guessing: Binary
5.3.1
Codes
5.1.4 Irreversible
Compression
Search
Techniques
5.1.5
5.2
Compression
5.3.3 Binary Search versus Sequential
UNIX
in
Reclaiming Space in
5.2.1
Search
5.3.4 Sorting a Disk File in
Files
5.3.5
RAM
The Limitations of Binary
Record Deletion and Storage
Searching and Internal Sorting
Compaction
Keysorting
5.4
5.2.2 Deleting Fixed-length Records
Reclaiming Space
Dynamically
for
Description of the
5.4.1
Method
5.2.3 Deleting Variable-length
Another Solution: Why Bother
to Write the File Back?
5.4.4 Pinned Records
Records
5.4.3
5.2.4 Storage Fragmentation
5.2.5 Placement Strategies
We
how
have already seen
consider
how
a file
to
is
and records and other
file
Method
5.4.2 Limitations of the Keysort
important
be accessed
file
when
some
for the
is
cases reorganize,
file
deciding on
structures. In this chapter
organization, but the motivation
organize, or in
it is
we
a little different.
files in
system designer to
how
to create fields
continue to focus on
We look at
direct response to the
ways
to
need to
improve performance.
In the first section
smaller.
we
look
Compression techniques
basic information in the
at
how we
let
us
make
organize
files
files
to
make them
smaller by encoding the
file.
Next we look at ways to reclaim unused space in files to improve
performance. Compaction is a batch process that we can use to purge holes
of unused space from a file that has undergone many deletions and updates.
Then we investigate dynamic ways to maintain performance by reclaiming
space made available by deletions and updates of records during the life of
a file.
In the third section
sorting
them
we examine
the
problem of reorganizing files by
Then, in an effort to find
to support simple binary searching.
DATA COMPRESSION
85
method, we begin a conceptual line of thought that will
continue throughout the rest of this text: We find a way to improve file
performance by creating an external structure through which we can access
a better sorting
the
5.1
file.
Data Compression
In this section
reasons for
Use
we
look
making
at
files
some ways
to
make
smaller. Smaller
smaller.
files
less storage, resulting in cost savings;
Can be
transmitted faster, decreasing access time or, alternatively,
lowing the same access time with
and
Can be processed
as to take
up
al-
lower and cheaper bandwidth;
faster sequentially.
Data compression involves encoding the information
way
There are many
files
less space.
Many
in a file in
such
different techniques are available for
Some are very general and some are designed only for
of data, such as speech, pictures, text, or instrument data. The
variety of data compression techniques is so large that we can only touch on
the topic here, with a few examples.
compressing
data.
specific kinds
5.1.1
Using a Different Notation
Remember our
fields,
fields
address
file
such
as these are
good candidates
"state" field in the address
many
from Chapter
4?
It
had several fixed-length
including "state," "zip code," and "phone number." Fixed-length
bits are really
could represent
all
file
needed for
for compression. For instance, the
required
this field?
two ASCII
bytes, 16 bits.
How
Since there are only 50 states,
possible states with only six bits. (Why?) Thus,
we
we
could
encode all state names in a single one-byte field, resulting in a space savings
of one byte, or 50%, per occurrence of the state field.
This type of compression technique, in which we decrease the number
of bits by finding a more compact notation/ is one of many compression
techniques classified as redundancy reduction. The 10 bits that we were able to
throw away were redundant in the sense that having 16 bits instead of 6
provided no extra information.
"'"Note that the original two-letter notation
tation tor the full state
name.
we
used for "state"
is
itself a
more compact no-
86
ORGANIZING FILES FOR PERFORMANCE
What
are the costs of this
compression scheme? In
this case, there are
many:
By
using a pure binary encoding,
we have made
the
file
unreadable
by humans.
We
whenever we add a new stateand a similar cost for decoding when we need
to get a readable version of state name from the file.
We must also now incorporate the encoding and/or decoding modules in all software that will process our address file, increasing the
complexity of the software.
incur
name
some
field to
cost in encoding time
our
file,
With so many costs, is this kind of compression worth it? We can
answer this only in the context of a particular application. If the file is
already fairly small, if the file is often accessed by many different pieces of
software, and if some of the software that will access the file cannot deal
with binary data (e.g., an editor), then this form of compression is a bad
idea. On the other hand, if the file contains several million records and is
generally processed by one program, compression is probably a very good
idea. Since the encoding and decoding algorithms for this kind of
compression are extremely simple, the savings in access time is likely to
exceed any processing time required for encoding or decoding.
5.1.2 Suppressing Repeating Sequences
Imagine an 8-bit image of the sky that has been processed so only objects
above a certain brightness are identified and all other regions of the image
are set to some background color represented by the pixel value 0. (See Fig.
5.1.)
Sparse arrays of this sort are very good candidates for compression of
which in this example works as follows.
unused byte value to indicate that a run-length
code follows. Then, the run-length encoding algorithm goes like this:
a sort called run-length encoding,
First,
we
choose one
special,
Read through the
values to the
file
pixels that
make up
in sequence, except
the image, copying the pixel
where the same
more than once in succession.
Where the same value occurs more than once
pixel value oc-
curs
in succession, substi-
tute the following three bytes, in order:
The special run-length code indicator;
The pixel value that is repeated; and
The number of times that the value is
repeated (up to 256 times).
DATA COMPRESSION
187
FIGURE 5.1 The empty space
in this astronomical image is
represented by repeated se-
quences of the same value
and is thus a good candidate
for compression. (This FITS
image shows a radio continuum structure around the spiral galaxy NGC 891 as observed with the Westerbork
Synthesis radio telescope in
The Netherlands.)
we wish to compress an image using run-length
we can omit the byte Oxff from the rep-
For example, suppose
encoding, and
we
find that
resentation of the image.
indicator.
How
We
would we
choose the byte Oxff as our run-length code
encode the following sequence of hexadecimal
byte values?
22 23 24 24 24 24 24 24 24 25 26 26 26 26 26 26 25 24
The
first three pixels are to be copied in sequence. The runs of 24 and 26 are
both run-length encoded. The remaining pixels are copied in sequence. The
resulting sequence is
22 23 ff
24
07 25 ff
26 06 25 24
Run-length encoding is another example of redundancy reduction.
(Why?) It can be applied to many kinds of data, including text, instrument
data, and sparse matrices. Like the compact notation approach, the
run-length encoding algorithm is a simple one whose associated costs rarely
affect performance appreciably.
Unlike compact notation, run-length encoding does not guarantee any
particular amount of space savings. A "busy" image with a lot of variation
will not benefit appreciably from run-length encoding. Indeed, under some
88
ORGANIZING FILES FOR PERFORMANCE
pressed" image that
prevent
"com(Why? Can you
the aforementioned algorithm could result in a
circumstances,
larger than the original image.
is
this?)
5.1.3 Assigning Variable-length Codes
Suppose you have two
symbols to use in an encoding scheme: a
have to assign combinations of dots and
dashes to letters of the alphabet. If you are very clever, you might determine
the most frequently occurring letters of the alphabet (e and t) and use a
single dot for one and a single dash for the other. Other letters of the
alphabet will be assigned two or more symbols, with the more frequently
occurring letters getting fewer symbols.
Sound familiar? You may recognize this scheme as the oldest and most
common of the variable-length codes, the Morse code. Variable-length codes,
in general, are based on the principle that some values occur more
dot ("") and
different
dash ("-").
You
frequently than others, so the codes for those values should take the least
amount of
space. Variable-length codes are another
form of redundancy
reduction.
A variation on the compact notation technique, the Morse code can be
implemented using a table lookup, where the table never changes. In
contrast, since many sets of data values do not exhibit a predictable
frequency distribution, more modern variable-length coding techniques
dynamically build the tables that describe the encoding scheme.
most
successful
of these
the
is
Huffman
code,
One of the
which determines the
probabilities of each value occurring in the data set, and then builds a binary
which the search path
tree in
value.
More
for each value represents the code for that
frequently occurring values are given shorter search paths in
then turned into a table, much like a Morse code table,
encode and decode the data.
For example, suppose we have a data set containing only the seven
letters shown in Fig. 5.2, and each letter occurs with the probability
the tree. This tree
is
that can be used to
indicated.
The
third
row
in the figure
shows
be assigned to the letters. Based on Fig.
encoded as "101000000001."
the
Huffman codes
5.2, the string
FIGURE 5.2 Example showing the Huffman encoding for a set of
seven letters, assuming certain probabilities. (From Lynch, 1985.
Let ter
Probability:
Code
abed
0.4
0.1
010
0.1
011
0.1
0.1
0.1
0.1
0000
0001
0010
0011
that
would
"abde" would be
DATA COMPRESSION
In the example, the letter a occurs
others, so
it
number of
bits
case as
many
much more
assigned the one-bit code
is
1.
as four bits are required.
This
is
minimum
letters is three, yet in this
necessary trade-off to insure
that the distinct codes can be stored together, without delimiters
them, and
89
often than any of the
Notice that the
needed to represent these seven
between
be recognized.
still
5.1.4 Irreversible Compression Techniques
The techniques we have
discussed so far preserve
all
information in the
original data. In effect, they take advantage of the fact that the data, in
its
removed and
Another type of compression, irreversible
based on the assumption that some information can be
original form, contains redundant information that can be
then reinserted
compression,
at a later time.
is
sacrificed.
An example of irreversible compression would be shrinking a raster
image from, say, 400-by-400 pixels to 100-by-100 pixels. The new image
contains one pixel for every 16 pixels in the original image, and there is no
way, in general, to determine what the original pixels were from the one
new
pixel.
compression
Irreversible
less
is
compression, but there are times
common
when
in data files than reversible
the information that
or no value. For example, speech compression
a
technique that transmits
be synthesized
at
is
in
is
lost
by
is
of little
voice coding,
paramaterized description of speech, which can
the receiving end with varying
5.1.5 Compression
often done
amounts of
distortion.
UNIX
Both Berkeley and System
V UNIX
provide compression routines that are
heavily used and quite effective. System
has routines called pack and
which use Huffman codes on a byte-by-byte basis. Typically, pack
achieves 25 to 40% reduction on text files, but appreciably less on binary
files that have a more uniform distribution of byte values. When pack
unpack,
compresses
it automatically appends a ".z" to the end of the packed
any future user that the file has been compressed using the
standard compression algorithm.
Berkeley UNIX has routines called compress and uncompress, which use
an effective dynamic method called Lempel-Ziv (Welch, 1984). Except for
using different compression schemes, compress and uncompress behave
file,
a file,
signalling to
"^Irreversible
compression
is
sometimes
average information (entropy)
is
called
reduced.
"entropy reduction" to emphasize that the
90
ORGANIZING FILES FOR PERFORMANCE
almost the same
of
files it
pack and unpack." Compress appends
has compressed.
1"
as
Since these routines are readily available on
effective general-purpose routines,
it is
".Z"
to the
end
UNIX systems and are very
wise to use them whenever there are
not compelling reasons to use other techniques.
5.2
Reclaiming Space
Suppose
that the
record in
new
record
the extra data?
from the
You
a
is
in Files
variable-length record
file is
could append
it
end of the
to the
a way
What do you do with
modified in such
longer than the original record.
file
and put
original record space to the extension of the record.
rewrite the
whole record
at
the end of the
(unless the
file
file
pointer
You
could
needs to be
of the record. Each solution
drawback: In the former case, the job of processing the record is more
awkward and slower than it was originally; in the latter case, the file
contains wasted space.
In this section we take a close look at the way file organization
deteriorates as a file is modified. In general, modifications can take any one
of three forms:
sorted), leaving a hole at the original location
has
Record addition;
Record updating; and
Record deletion.
If the
only kind of change to
deterioration of the kind
we
a file
is
cover in
variable-length records are updated, or
record addition, there
chapter.
this
when
It
is
only
is
no
when
either fixed- or variable-
length records are deleted, that maintenance issues
become complicated and
interesting. Since record updating can always be treated as a record deletion
followed by
When
record addition, our focus
record has been deleted,
is
we want
on the
effects
of record deletion.
to reuse the space.
5.2.1 Record Deletion and Storage Compaction
files smaller by looking for places in a file where
and then recovering this space. Since empty spaces
occur in files when we delete records, we begin our discussion of
compaction with a look at record deletion.
Storage compaction
there
is
no data
makes
at all,
"''Many implementations of System
Berkeley extensions.
V UNIX
also support
compress and uncompress
as
RECLAIMING SPACE
Any
record-deletion
mark
place a special
address
us
for
simple and usually workable approach
Chapter
in
field in a deleted record.
file
4,
we might
place an asterisk as the
Figures 5.3(a) and 5.3(b)
show
is
name and
first
address
similar to the one in Chapter 4 before and after the second record
marked
deleted.
as
(The dots
padding between the
Once we
how
last field
the ends of records
at
to
to
name and
For example, in the
in each deleted record.
developed
file
must provide some way
strategy
recognize records as deleted.
191
IN FILES
is
and 2 represent
and the end of each record.)
are able to recognize a record as deleted, the next question
is
from the record. Approaches to this problem that
rely on storage compaction do nothing at all to reuse the space for a while.
The records are simply marked as deleted and left in the file for a period of
time. Programs using the file must include logic that causes them to ignore
records that are marked as deleted. One nice side effect of this approach is
that
to reuse the space
it is
usually possible to allow the user to "undelete" a record with very
This
little effort.
is
rather than destroy
once. After deleted records have accumulated for
program
out
is
(Fig.
used to reconstruct the
5.3c).
If there
compaction
is
records.
also possible,
It is
mark
particularly easy if you keep the deleted
some of the original data,
The reclamation of the space from the deleted
field,
through
file
with
all
in a special
our example.
as in
records happens
some
time,
all at
special
the deleted records squeezed
enough space, the simplest way to do this
copy program that skips over the deleted
though more complicated and time-consuming,
is
a file
do the compaction in place. Either of these approaches can be used with
both fixed- and variable-length records.
to
FIGURE 5.3 Storage requirements of sample file using 64-byte fixed-length records, (a)
Before deleting the second record, (b) After deleting the second record, (c) After compaction
the second record is gone.
Ames! John! 123 Maple Stillwater OK 74075
Morrison!Sebastian!9035 South Hillcrest Forest Village OK 74820
Brown! Martha! 625 KimbarkiDes Moines IA 50311
!
(a)
Ames! John 1123 Maple Stillwater OK 174075
!
rrison!Sebastian!9035 South Hillcrest Forest Village OK 74820
Brown Martha 625 KimbarkiDes Moines IA 50311
*!
(b)
Ames John 123 Maple Stillwater OK 74075
Brown! Martha 1625 KimbarkiDes Moines IA 50311
!
(c)
92
ORGANIZING FILES FOR PERFORMANCE
The
decision about
how
can be based on either the
often to run the storage compaction
number of deleted
accounting
programs, for example,
compaction procedure on certain files
it
often
at the
program
records or on the calendar. In
makes sense
end of the
fiscal
run
to
year or
some
other point associated with closing the books.
5.2.2 Deleting Fixed-length Records
for
Reclaiming
Space Dynamically
most widely used of the storage
There are some applications, however,
that are too volatile and interactive for storage compaction to be useful. In
these situations we want to reuse the space from deleted records as soon as
possible. We begin our discussion of such dynamic storage reclamation
with a second look at fixed-length record deletion, since fixed-length
records make the reclamation problem much simpler.
In general, to provide a mechanism for record deletion with subsequent
reutilization of the freed space, we need to be able to guarantee two things:
Storage compaction
the simplest and
is
reclamation methods
we
discuss.
That deleted records are marked in some special way; and
That we can find the space that deleted records once occupied so we
can reuse that space when we add records.
We
a method of meeting the first requirement: We
by putting a field containing an asterisk at the
have already identified
mark records
as deleted
beginning of deleted records.
If you are working with fixed-length records and are willing to search
sequentially through a file before adding a record, you can always provide
you have provided the first. Space reutilization can
form of looking through the file, record by record, until a deleted
record is found. If the program reaches the end of the file without finding
a deleted record, then the new record can be appended at the end.
Unfortunately, this approach makes adding records an intolerably slow
process if the program is an interactive one and the user has to sit at the
the second guarantee if
take the
terminal and wait as the record addition takes place.
To make
record reuse
happen more quickly, we need
A way
A way
Linked
to
to
Lists
know immediately if there are empty slots in the
jump directly to one of those slots if they exist.
The use of
available records can
structure in
its
a linked
list
for stringing together
meet both of these needs.
linked
list
file;
and
all
of the
is
a data
which each element or node contains some kind of reference
successor in the
list.
(See Fig. 5.4.)
to
RECLAIMING SPACE
FIGURE 5.4 A linked
193
IN FILES
list.
If you have a head reference to the first node in the list, you can move
through the list by looking at each node, and then at the node's pointer
field, so you know where the next node is located. When you finally
encounter a pointer field with some special, predetermined end-of-list
value, you stop the traversal of the list. In Fig. 5.4 we use a 1 in the pointer
field to mark the end of the list.
When a list is made up of deleted records that have become available
space within the
file,
new
the
list is
usually called an avail
list.
When
inserting a
any one available record is just as
good as any other. There is no reason to prefer one open slot over another
since all the slots are the same size. It follows that there is no reason for
ordering the avail list in any particular way. (As we see later, this situation
changes for variable-length records.)
record into
The
Stacks
which
So, if
all
simplest
insertions
we have
way
file,
to handle a
list
managed
as a stack.
RRN
Head
pointer
Head
\
S
RRN 3,
stack
is
a list in
RRN
5
it
list.
record
looks like this before and
(3)
one end of the
RRN
RRN
pointer
at
as a stack that contains relative
and 2, and then add
the addition of the new node:
When
list is
and removals of nodes take place
an avail
numbers (RRN)
after
fixed-length record
RRN
added to the top or front of a stack, we say that it
next thing that happens is a request for some
available space, the request is filled by taking RRN 3 from the avail list.
is
new node
is
pushed onto the stack.
If the
94
ORGANIZING FILES FOR PERFORMANCE
This
called popping the stack.
is
only records 5 and
The
list
Linking and Stacking Deleted Records
Now we
it
We
know immediately if there are empty slots in the
jump directly to one of those slots if they exist.
to
to
contains
can meet the two
space from deleted records.
criteria for rapid access to reusable
A way
A way
which
returns to a state in
2.
need
file;
and
Placing the deleted records on
a stack meets both criteria. If the pointer
of the stack contains the end-of-list value, then we know that
there are not any empty slots and that we have to add new records by
appending them to the end of the file. If the pointer to the stack top contains
to the top
a valid
node
reference,
then
available, but also exactly
Where do we keep
or
a separate file,
we need
to
structures.
when
is it
we know
where
to find
the stack?
Is it a
not only that
reusable slot
is
it.
separate
somehow embedded
list,
perhaps maintained in
within the data
file?
Once
again,
be careful to distinguish between physical and conceptual
The
deleted, available records are not actually
moved anywhere
we need them,
they are pushed onto the stack. They stay right where
located in the
file.
The
stacking and linking
rearranging the links used to
make one
is
done by arranging and
available record slot point to the
we are working with fixed-length records in a disk file, rather
memory addresses, the pointing is not done with pointer variables
next. Since
than with
in the
formal sense, but through relative record numbers (RRNs).
Suppose we
are
working with
contained seven records
(RRNs
fixed-length record
file
that
once
0-6). Furthermore, suppose that records 3
have been deleted, in that order, and that deleted records are marked by
first field with an asterisk. We can then use the second field of
a deleted record to hold the link to the next record on the avail list. Leaving
out the details of the valid, in-use records, Fig. 5.5(a) shows how the file
might look.
Record 5 is the first record on the avail list (top of the stack) since it is
the record that is most recently deleted. Following the linked list, we see
and
replacing the
that record 5 points to record 3. Since the link field for record 3 contains -1,
which
is
our end-of-list marker,
we know
that record 3
is
the last slot
available for reuse.
Figure 5.5(b) shows the same
file after
record
1 is
also deleted.
Note
that
the contents of all the other records on the avail list remain unchanged.
Treating the list as a stack results in a minimal amount of list reorganization
when we push and pop records to and from the list.
If we now add a new name to the file, it is placed in record 1, since
RRN 1 is the first available record. The avail list would return to the
RECLAIMING SPACE
List
head
(first
available record) -* 5
2
Edwards
Bates
195
IN FILES
Wills
*-l
Masters
Masters
*3
Chavez
*3
Chavez
(a)
List
head
(first
available record)
*5
Edwards
Wills
*-l
(b)
List
head
(first
available record)
1st
new
Wills
rec
Edwards
3rd new rec
Masters
2nd new
Chavez
rec
(c)
FIGURE 5.5 Sample
file
showing linked
5, in that order, (b) After deletion of
new
lists of
deleted records,
records 3, 5, and
1,
in
(a)
After deletion of records 3
records.
shown
configuration
the avail
list,
we
the size of the
5.5c). If yet
avail list
at
is
in Fig. 5.5(a). Since there are
file.
After that, however, the avail
name
empty and
the end of the
still
could add two more names to the
another
is
added to the
that the
name
file,
file
two record
on
slots
without increasing
would be empty (Fig.
program knows that the
list
the
requires the addition of a
new
record
file.
Implementing Fixed-length Record Deletion
nisms that place deleted records on
list
Implementing mechaand that treat the avail
need a suitable place to keep
linked avail
as a stack is relatively straightforward.
We
list
RRN
of the first available record on the avail list. Since this is
information that is specific to the data file, it can be carried in a header
record at the start of the file.
When we delete a record we must be able to mark the record as deleted,
and then place it on the avail list. A simple way to do this is to place an *
the
and
that order, (c) After insertion of three
'
96
ORGANIZING FILES FOR PERFORMANCE
(or some other special mark) at the beginning of the record as a deletion
mark, followed by the RRN of the next record on the avail list.
Once we have a list of available records within a file, we can reuse the
space previously occupied by deleted records. For this
single function that returns either (1) the
(2)
the
RRN
RRN
appended
ot the next record to be
we would
of a reusable record
if
no reusable
write
slot,
or
slots are
available.
5.2.3 Deleting Variable-length Records
Now that we have a
mechanism for handling an avail list of available space
once records are deleted, let's apply this mechanism to the more complex
problem of reusing space from deleted variable-length records. We have
seen that to support record reuse through an avail list, we need
A way
to link the deleted records together into a
list (i.e.,
place to
put
a link field);
An
An
algorithm for adding newly deleted records to the avail list; and
algorithm for finding and removing records from the avail list
when we
are ready to use them.
An
Avail List of Variable-length Records What kind of file structure
do we need to support an avail list of variable-length records? Since we will
want to delete whole records and then place records on an avail list, we need
a structure in which the record is a clearly defined entity. The file structure
in which we define the length of each record by placing a byte count of the
record contents
at
the beginning of each record will serve us well in this
regard.
We
can handle the contents of
a deleted
we
did with fixed-length records. That
the
first field,
followed by
is,
binary link
variable-length record just as
we
field
can place
a single asterisk in
pointing to the next deleted
list. The avail list itself can be organized just as it was
with fixed-length records, but with one difference: We cannot use relative
record numbers (RRNs) for links. Since we cannot compute the byte offset
record on the avail
of variable-length records from
their
RRNs,
the links
must contain the byte
offsets themselves.
To
illustrate,
suppose
we
begin with
variable-length record
containing the three records for Ames, Morrison, and
earlier.
Figure 5.6(a) shows what the
file
Brown
file
introduced
looks like (minus the header)
and Fig. 5.6(b) shows what it looks like after the
before any
deletion of the second record. The periods in the deleted record signify
deletions,
discarded characters.
RECLAIMING SPACE
197
IN FILES
HEAD. FIRST_AVAIL: -1
40 Ames! John! 123 Maple Stillwater OK !74075 64 Morrison Sebastian
!9035 South Hillcrest Forest Village OK 74820 45 Brown !Martha 62
5 Kimbark!Des Moines IA 50311
!
(a)
HEAD. FIRST AVAIL: 43
40 Ames John 123 Maple Stillwater OK 74075 64
!
*
-1
45 Brown Martha 62
!
Kimbark!Des MoinesilA 50311!
(b)
FIGURE 5.6 A sample
sample
file
cluded), (b)
stored
in
Sample
file for illustrating
variable-length record deletion,
(a) Original
variable-length format with byte count (header record not
file
after deletion of the
in-
second record (periods show discarded
characters).
Adding and Removing Records
Let's address the questions
and removing records to and from the
list
of adding
together, since they are clearly
With fixed-length records we could access the avail list as a stack
because one member of the avail list is just as usable as any other. That is not
true when the record slots on the avail list differ in size, as they do in a
variable-length record file. We now have an extra condition that must be
met before we can reuse a record: The record must be the right size. For the
related.
moment we
define right size as "big enough." Later
we
find that
it
is
to be more particular about the meaning of right size.
even likely, that we need to search through the avail list for
a record slot that is the right size. We can't just pop the stack and expect the
first available record to be big enough. Finding a proper slot on the avail list
now means traversing the list until a record slot is found that is big enough
to hold the new record that is to be inserted.
For example, suppose the avail list contains the deleted record slots
shown in Fig. 5.7(a), and a record that requires 55 bytes is to be added. Since
the avail list is not empty, we traverse the records whose sizes are 47 (too
small), 38 (too small), and 72 (big enough). Having found a slot big enough
to hold our record, we remove it from the avail list by creating a new link
that jumps over the record (Fig. 5.7b). If we had reached the end of the avail
list before finding a record that was large enough, we would have appended
sometimes useful
It is
the
new
possible,
record
at
the end of the
file.
98
ORGANIZING FILES FOR PERFORMANCE
Size
Removed record
72
(b)
FIGURE 5.7 Removal of a record from an avail list with variablelength records, (a) Before removal, (b) After removal.
Since this procedure for finding a reusable record looks through the
entire avail
list
if
necessary,
we do
not need
putting newly deleted records onto the
somewhere on
it.
list,
a sophisticated
If a
list,
just as
method
for
record of the right size
is
our get-available-record procedure eventually finds
we can continue to push new members
we do with fixed-length records.
follows that
It
the
this
list.
onto the front of
Development of algorithms for adding and removing avail list records
to you as part of the exercises found at the end of this chapter.
is left
5.2.4 Storage Fragmentation
Let's look again at the fixed-length record version
(Fig. 5.8).
The
of our three-record
dots at the ends of the records represent characters
we
file
use as
padding between the last field and the end of the records. The padding is
wasted space; it is part of the cost of using fixed-length records. Wasted
space within
Clearly,
record
is
we want
called internal fragmentation.
minimize internal fragmentation.
to
working with fixed-length
FIGURE 5.8 Storage requirements of sample
we
records,
file
If
we
using 64-byte fixed-length records.
Ames John 123 Maple Stillwater OK 74075
Morrison! Sebastian 19035 South Hillcrest Forest Village OK 174820
Brown Martha 625 Kirabark'Des Moines IA 50311
I
are
attempt such minimization by
RECLAIMING SPACE
IN FILES
99
40 Ames! John! 123 Maple Stillwater OK 74075 64 Morrison Sebastian
!9035 South Hillcrest Forest Village OK 74820 45 Brown iMartha 62
5 KimbarkiDes Moines IA 50311
!
FIGURE 5.9 Storage requirements of sample
count field.
record.
record length that
But unless the
a certain
using variable-length records with a
file
is
what we need for each
we have to put up with
as close as possible to
actual data
is
fixed in length,
amount of internal fragmentation
One of the
choosing
in a fixed-length record
attractions of variable-length records
is
that they
file.
minimize
wasted space by doing away with internal fragmentation. The space set
aside for each record is exactly as long as it needs to be. Compare the
fixed-length example with the one in Fig. 5.9, which uses the variablelength record structure
a byte count followed by delimited data fields.
The only space (other than the delimiters) that is not used for holding data
in each record is the count field. If we assume that this field uses two bytes,
this amounts to only six bytes for the three-record file. The fixed-length
record file wastes 24 bytes in the very first record.
But before we start congratulating ourselves for solving the problem of
wasted space due to internal fragmentation, we should consider what
happens in a variable-length record file after a record is deleted and replaced
with a shorter record. If the shorter record takes less space than the original
record, internal fragmentation results. Figure 5.10 shows how the problem
FIGURE 5.10 Illustration of fragmentation with variable-length records, (a) After deletion of
the second record (unused characters in the deleted record are replaced by periods), (b) After
the subsequent addition of the record for Al Ham.
HEAD. FIRST AVAIL: 43
I
40 Ames
John 123 Maple Stillwater OK 74075 64
|
*
[
-1
45 Brown Martha 62
]
KimbarkjDes Moines IA 50311
(a)
HEAD. FIRST_AVAIL: -1
40 Ames John; 123 Maple Stillwater OK 74075
OK J70332;
5 KimbarkiDes Moines IA 50311
!
(b)
64 Ham;Al[28 Elm Ada
45 Brown [Martha 62
\
200
ORGANIZING FILES FOR PERFORMANCE
HEAD. FIRST_AV'AIL: 43
40 Ames John 123 Maple Stillwater OK 74075 35 *
-1
26 Ham Al 28 Elm Ada OK 70332 45 Brown Martha 6
25 Kimbark Des Moines IA 50311
;
FIGURE 5.1 1 Combatting internal fragmentation by putting the unused part of the deleted
back on the avail list.
slot
could occur with our sample
and the following record
is
when
file
the second record in the
file is
deleted
added:
Ham|Al|28 E lm|Ada|OK|70332|
appears that escaping internal fragmentation
It
vacated by the deleted record
we
not so easy. The slot
is
37 bytes larger than
is
is
needed for the new
as part of the new record, they are
and are therefore unusable. But instead of keeping the
64-byte record slot intact, suppose we break it into two parts: one part to
hold the new Ham record, and the other to be placed back on the avail list.
record. Since
not on the avail
treat the extra
37 bytes
list
we would take only as much space as necessary
would be no internal fragmentation.
Figure 5.11 shows what our file looks like if we
Since
for the
Ham
record,
there
use this approach to
Al Ham. We steal the space for the Ham record from the
end of the 64-byte slot and leave the first 35 bytes of the slot on the avail list.
(The available space is 35 rather than 37 bytes because we need two bytes to
insert the record for
form
new
The 35
size field for the
bytes
on the
still
Ham
avail
record.)
can be used to hold yet another record.
list
Figure 5.12 shows the effect of inserting the following 25-byte record:
Lee|Ed|Rt
2|Ada|OK|74820|
As we would expect, the new record is carved out of the 35-byte record that
is on the avail list. The data portion of the new record requires 25 bytes, and
FIGURE 5.12 Addition of the second record into the
slot originally
occupied by a single de-
leted record.
HEAD,
FIRST AVAIL: 43
1
-1 ... 25 Lee Ed
40 Ames John 123 Maple Stillwater OK 74075 8 *
Rt 2 Ada OK 74820 26 Ham Al 28 Elm Ada OK 70332 45 Brown Martha 6
25 Kimbarkl Des Moines IA 50311
;
RECLAIMING SPACE
then
we need two more
in the record
still
bytes for another size
on the
avail
201
IN FILES
This leaves eight bytes
field.
list.
What are the chances of finding a record that can make use of these eight
Our guess would be that the probability is close to zero. These eight
bytes?
bytes are not usable, even though they are not trapped inside any other
The
an example of external fragmentation.
record. This
is
the avail
rather than being locked inside some other record, but
list
space
is
actually
is
on
too
fragmented to be reused.
There are some interesting ways to combat external fragmentation.
One way, which we discussed at the beginning of this chapter, is storage
compaction. We could simply regenerate the file when external fragmentation becomes intolerable. Two other approaches are as follows:
If two record slots on the avail list are physically
them to make a single, larger record slot. This is
adjacent,
combine
called coalescing the
holes in the storage space.
Try
minimize fragmentation before it happens by adopting a
placement strategy that the program can use as it selects a record
from the avail list.
to
Coalescing holes presents some interesting problems. The avail
not kept in physical record order;
physically adjacent, there
adjacent to each other
provides
on the
discussion
developing
is
of
if there are
no reason
avail
this
to
two
list is
deleted records that are
presume
Exercise 15
list.
slot
at
that they are linked
the end of this chapter
problem along with
framework
for
a solution.
The development of
better placement strategies,
warrants
matter.
It
among
alternative strategies
is
a topic that
is
not
however,
is
a different
separate discussion, since the choice
as
obvious
as
it
might seem
at first
glance.
5.2.5 Placement Strategies
Earlier
we
discussed
ways
to
We
add and remove variable-length records from
add records by treating the avail list as a stack, putting
When we need to remove a record slot from the
avail list (to add a record to the file), we look through the list, starting at the
beginning, until we either find a record slot that is big enough or reach the
end of the list.
This is called afirst-fit placement strategy. The least possible amount of
work is expended when we place newly available space on the list, and we
are not very particular about the closeness of fit as we look for a record slot
to hold a new record. We accept the first available record slot that will do
an avail
list.
deleted records at the front.
202
ORGANIZING FILES FOR PERFORMANCE
the job, regardless of whether the slot
or whether
We
it is
a perfect
is
10 times bigger than
what
is
needed
fit.
more orderly approach for placing
on the avail list, keeping them in either ascending or descending
sequence by size. Rather than always putting the newly deleted records at
the front of the list, these approaches involve moving through the list,
could, of course, develop a
records
looking for the place to insert the record to maintain the desired sequence.
If we order the avail list in ascending order by size, what is the effect on
the closeness of
fit
of the records that are retrieved from the
list?
Since the
procedure searches sequentially through the avail list until it
encounters a record that is big enough to hold the new record, the first
record encountered is the smallest record that will do the job. The fit
retrieval
and the new record's needs would be as close as
we can make it. This is called a best-fit placement strategy.
A best-fit strategy is intuitively appealing. There is, of course, a price to
be paid for obtaining this fit. We end up having to search through at least
a part of the list not only when we get records from the list, but also when
we put newly deleted records on the list. In a real-time environment the
extra processing time could be significant.
A less obvious disadvantage of the best-fit strategy is related to the idea
of finding the best possible fit: The free area left over after inserting a new
record into a slot is as small as possible. Often this remaining space is too
small to be useful, resulting in external fragmentation. Furthermore, the
slots that are least likely to be useful are the ones that will be placed toward
the beginning of the list, making first-fit searches increasingly long as time
between the
available slot
goes on.
These problems suggest an alternative strategy: What if we arrange the
it is in descending order by size? Then the largest record slot on
the avail list would always be at the head of the list. Since the procedure that
retrieves records starts its search at the beginning of the avail list, it always
returns the largest available record slot if it returns any slot at all. This is
known as a worst- fit placement strategy. The amount of space in the record
slot beyond what is actually needed is as large as possible.
A worst-fit strategy does not, at least initially, sound very appealing. But
avail list so
consider the following:
The procedure
only
for
at the first
enough
removing records can be simplified
element of the avail
list.
If the first
so
it
looks
record slot
is
not
do the job, none of the others will be.
By extracting the space we need from the largest available slot, we
are assured that the unused portion of the slot is as large as possible,
large
to
decreasing the likelihood of external fragmentation.
203
FINDING THINGS QUICKLY: AN INTRODUCTION TO INTERNAL SORTING AND BINARY SEARCHING
What can you conclude from all of this? It should be clear that no one
placement strategy is superior for all circumstances. The best you can do is
formulate a series of general observations and then, given a particular design
seems most appropriate. Here are
have to be yours.
situation, try to select the strategy that
some
'
suggestions.
The judgment
will
Placement strategies make sense only with regard
able-length record
files.
With fixed-length
to volatile, vari-
records, placement
is
sim-
ply not an issue.
&x
is lost due to internal fragmentation, then the choice is between
and best fit. A worst-fit strategy truly makes internal fragmentation worse.
If the space is lost due to external fragmentation, then one should give
If space
first fit
careful consideration to a worst-fit strategy.
5.3
Finding Things Quickly: An Introduction to Internal
Sorting and Binary Searching
This text begins with
storage.
discussion of the cost of accessing secondary
You may remember
accessing
magnify
that the
magnitude of the difference between
RAM and seeking information on a fixed disk such that, if we
the time for a RAM access to 20 seconds, a similarly magnified
is
disk access
would
take 58 days.
So far we have not had to pay much attention to this cost. This section,
then, marks a kind of turning point. Once we move from fundamental
organizational issues to the matter of searching a file for a particular piece of
information, the cost of a seek becomes a major factor in determining our
And what is true for searching is all the more true for sorting. If
you have studied sorting algorithms, you know that even a good sort
involves making many comparisons. If each of these comparisons involves
approach.
a seek,
the sort
Our
is
agonizingly slow.
discussion of sorting and searching, then, goes
getting the job done.
We
beyond simply
develop approaches that minimize the number of
disk accesses and that therefore minimize the
amount of time expended.
This concern with minimizing the number of seeks continues to be
focus throughout the rest of this text. This
for
ways
to order
major
just the beginning of a quest
and find things quickly.
5.3.1 Finding Things
All of the
is
in
Simple Field and Record
programs we have written up
strengths they offer, share a major failing:
Files
to this point, despite
The only way
any other
to retrieve or find
204
ORGANIZING FILES FOR PERFORMANCE
record with any degree of rapidity
to look for
is
it
by
relative record
number (RRN). If the file has fixed-length records, knowing the RRN
us compute the record's byte offset and jump to it using direct access.
But what if we do not
want? How likely is it that
"What
RRN
the byte offset or
of the record
we
question about this file would take the form,
23?" Not very likely, of course. We are
the record stored in
a
RRN
is
much more
question
know
lets
is
know
likely to
more
the identity of a record
likely to take the form,
"What
is
by
its
key, and the
the record for Bill Kelly?"
Given the methods of organization developed so far, access by key
a sequential search. What if there is no record containing the
requested key? Then we would have to look through the entire file. What
if we suspect that there might be more than one record that contains the
key, and we want to find them all? Once again, we would be doomed to
looking at every record in the file. Clearly, we need to find a better way to
handle keyed access. Fortunately, there are many better ways.
implies
5.3.2 Search by Guessing: Binary Search
Suppose we
are looking for a record for Bill Kelly in a
fixed-length records, and suppose the
We
ascending order by key.
file is
of 1,000
KELLY BILL
by comparing
start
file
sorted so the records appear in
canonical form of the search key) with the middle key in the
file,
(the
which
is
whose RRN is 500. The result of the comparison tells us which half
of the file contains Bill Kelly's record. Next, we compare KELLY BILL with
the middle key among records in the selected half of the file to find out
the key
which quarter of the
file Bill
Kelly's record
until either Bill Kelly's record
is
found or
we
is in.
This process
is
repeated
have narrowed the number of
potential records to zero.
This kind of searching
binary searching
is
shown
comparisons to find
that
it is
not in the
is
called binary searching.
in Fig. 5.13.
Bill Kelly's record, if
file.
Compare
this
w ith
T
An
algorithm for
Binary searching takes
it is
in the
a sequential
at
most 10
or to determine
file,
search for the record.
1,000 records, then it takes at most 1,000 comparisons to find a
given record (or establish that it is not present); on the average, 500
If there are
comparisons are needed.
5.3.3 Binary Search versus Sequential Search
In general, a binary search
of
with n records takes
a file
at
most
"
|_log FtJ
comparisons 1
"In this text, log x refers to the logarithm function to the base
intended,
it
is
so indicated.
2.
When
any other base
is
FINDING THINGS QUICKLY: AN INTRODUCTION TO INTERNAL SORTING
/*
205
AND BINARY SEARCHING
function to perform a binary search in the file associated with the
logical name INPUT. Assumes that INPUT contains REC0RD_C0UNT records.
Searches for the key KEY_S0UGHT. Returns RRN of record containing
key if the key is found; otherwise returns -1
*/
FUNCTION:
n_sea r ch( NPUT
I
LOW :=0
HIGH := REC0RD_C0UNT
KEY_S0UGHT, REC0RD_C0UNT
initialize lower bound for searching
intialize high bound -- we subtract
from the count since RRNs start from
/*
-
/*
while (LOW <= HIGH)
GUESS := (LOW
HIGH)
find
/*
*/
1
*/
midpoint
*/
read record with RRN of GUESS
place canonical form of key from record GUESS into KEY_F0UND
(KEY_S0UGHT < KEY_F0UND)
HIGH := GUESS else if (KEY_S0UGHT > KEY_F0UND)
:= GUESS +
LOW
GUESS is too high
so reduce upper
bound
/* GUESS is too low
/* increase lower
bound
if
I*
/*
*/
*/
/
*/
else
return(GUESS)
endwh i
match -- return the RRN
/*
*/
return (-1)
/*
if
FIGURE 5.13 The bin_search(
loop completes,
)
function
in
then key was not
found
*/
pseudocode.
and on average approximately
|_log
A binary search is
wj
+-
comparisons.
therefore said to be
that a sequential search
of the same
O (log n).
file
In contrast,
requires at
most
you may
recall
n comparisons,
and
which is to say that a sequential search is O(n).
The difference between a binary search and a sequential search becomes
on average
Vi n,
even more dramatic as we increase the size of the file to be searched. If we
double the number of records in the file, we double the number of
comparisons required for sequential search; when binary search is used,
doubling the file size adds only one more guess to our worst case. This
makes sense, since we know that each guess eliminates half of the possible
choices. So, if we tried to find Bill Kelly's record in a file of 2,000 records,
it
would
take at
most
1
-I-
[Jog 2,000J
=11
comparisons,
206
ORGANIZING FILES FOR PERFORMANCE
whereas
sequential search
-n
would average
=
1,000 comparisons,
and could take up to 2,000 comparisons.
Binary searching is clearly a more attractive way to find things than is
sequential searching. But, as you might expect, there is a price to be paid
before we can use binary searching: Binary searching works only when the
list of records is ordered in terms of the key we are using in the search. So,
to make use of binary searching, we have to be able to sort a list on the basis
of a key.
Sorting is a very important part of file processing. Next, we look at
some simple approaches to sorting files in RAM, at the same time
introducing some important new concepts in file structure design. In
Chapter 7 we take a second look at sorting, when we deal with some tough
problems that occur when files are too large to sort in RAM.
5.3.4 Sorting a Disk
File in
RAM
Consider the operation of any internal sorting algorithm with which you
The algorithm requires multiple passes over the list that is to be
sorted, comparing and reorganizing the elements. Some of the items in the
list are moved a long distance from their original positions in the list. If such
an algorithm were applied directly to data stored on a disk, it is clear that
there would be a lot of jumping around, seeking, and rereading of data.
This would be a very slow operation
unthinkably slow.
are familiar.
If the entire
alternative
is
contents of the
file
disk, but this
from the disk
to read the entire file
the sorting there, using an internal
way we
can access
can be held in
sort.
it
RAM,
into
very attractive
memory, and then do
We still have to access the data on the
sequentially, sector after sector, without
having to incur the cost of a lot of seeking and the cost of multiple passes
over the disk.
This is one instance of a general class of solutions to the problem of
minimizing disk usage: Force your disk access into a sequential mode,
performing the more complex, direct accesses in RAM.
Unfortunately, it is often not possible to use this simple kind of
solution, but when you can, you should take advantage of it. In the case of
sorting, internal sorts are increasingly viable as the
increases.
which
in
good
sorts files in
Chapter
7.
illustration
RAM if
it
of an internal sort
is
amount of RAM space
the
can find enough space. This
UNIX
utility
sort utility,
is
described
207
FINDING THINGS QUICKLY: AN INTRODUCTION TO INTERNAL SORTING AND BINARY SEARCHING
5.3.5 The Limitations of Binary Searching and Internal Sorting
Let's look at three problems associated with our "sort, then binary search"
approach to finding things.
Problem
Two
Binary Searching Requires More than One or
1:
Accesses
In the average case,
binary search requires approximately
n] + Vi comparisons. If each comparison requires a disk access, a series
of binary searches on a list of 1,000 items requires, on the average, 9.5
accesses per request. If the list is expanded to 100,000 items, the average
search length extends to 16.5 accesses. Although this is a tremendous
improvement over the cost of a sequential search for the key, it is also true
that 16 accesses, or even 9 or 10 accesses, is not a negligible cost. The cost
of this seeking is particularly noticeable, and objectionable, if we are doing
a large enough number of repeated accesses by key.
|_log
When we access records by relative record number (RRN) rather than
by key, we are able to retrieve a record with a single access. That is an order
of magnitude of improvement over the 10 or more accesses that binary
searching requires with even a moderately large file. Ideally, we would like
to
RRN
approach
performance,
retrieval
while
maintaining
still
the
advantages of access by key. In the following chapter, on the use of index
structures,
Problem
we
2:
begin to look
Keeping
at
ways
to
a File Sorted Is
move toward
Very Expensive
use a binary search has a price attached to
order by key. Suppose
as
often as
we
we
are
this ideal.
it:
working with
We must keep
a file to
search for existing records. If
we
Our
the
file
ability to
in sorted
which we add records
leave the
in
file
unsorted
on the average each search
requires reading through half the file. Each record addition, however, is
very fast, since it involves nothing more than jumping to the end of the file
and writing a record.
order, doing sequential searches for records, then
If,
as
an alternative,
substantially
on the
But we encounter
all
we
keep the
file
in sorted order,
cost of searching, reducing
difficulty
when we add
the records in sorted order. Inserting a
it
a record, since
new
we
can cut
to a handful
we want
record into the
down
of accesses.
file
to
keep
requires,
we
not only read through half the records, but that we
open up the space required for the insertion. We are
doing more work than if we simply do sequential searches on an
on the average,
that
also shift the records to
actually
unsorted
The
file.
costs
of maintaining
a file that
can be accessed through binary
searching are not always as large as in this example involving frequent
record addition. For example,
it is
often the case that searching
is
required
208
ORGANIZING FILES FOR PERFORMANCE
much more
frequently than
record addition. In such
is
more than
benefits of faster retrieval can
sorted.
As another example,
the
This can be an
circumstance, the
of keeping the
offset the costs
many
file
which record
additions can be accumulated in a transaction file and made in a batch mode.
By sorting the list of new records before adding them to the main file, it is
possible to merge them with the existing records. As we see in Chapter 7,
such merging is a sequential process, passing only once over each record in
file.
So, despite
appears to be
searching also
its
there are
efficient, attractive
approach to maintaining the
us see
what
However, knowing
the costs of binary
the requirements will be for better solutions
problem of finding things by key. Better solutions
one of the following conditions:
to the
file.
problems, there are situations in which binary searching
useful strategy.
lets
applications in
have to meet
will
at least
They
flr
will not involve reordering of the records in the
y^iew record
is
file
They
will be associated with data structures that allow for substan-
tially
more
rapid, efficient reordering
In the chapters that follow
we
these categories. Solutions of the
indexes.
They can
of the
file.
develop approaches that
first
fall
Problem
also involve hashing. Solutions
3:
An
Internal Sort
to use binary searching
works only
sort
if
is
of the second type can
we
Works Only on Small
is
we
An
file
ability
internal
into the
cannot do
that,
we
need a different kind of sort.
In the following section we develop
then
so large that
in order.
Our
file.
can read the entire contents of
If the file
file
Files
limited by our ability to sort the
computer's electronic memory.
into each of
type can involve the use of simple
involve the use of tree structures, such as a B-tree, to keep the
a variation
called a keysort. Like internal sorting, keysort
large a
file it
can sort, but
its
limit
keysort begins to illuminate
is
larger.
is
on
internal sorting
limited in terms of
how
More importantly, our work on
new approach
to the
problem of finding
things that will allow us to avoid the sorting of records in a
5.4
when
added; and
file.
Keysorting
Keysort,
sort a
keys;
sometimes referred
to as tag sort,
is
based on the idea that
when we
RAM the only things that we really need to sort are the record
into RAM during the
therefore, we do not need to read the whole
file in
sorting process. Instead,
file
we
read the keys from the
file
into
RAM,
sort
KEYSORTING
them, and then rearrange the records
ordering of the keys.
in
the
according to the
file
Since keysort never reads the complete set of records into
can sort larger
files
than
regular internal sort, given the
209
new
memory,
it
same amount of
RAM.
5.4.1 Description of the Method
we assume that we are dealing with a fixed-length
of the kind developed in Chapter 4, with a count of the number
records stored in a header record. We begin by reading the keys into an
array of identically sized character fields, with each row of the array
containing a key. We call this array KEYNODES[], and we call the key
field KEYNODES[].KEY. Figure 5.14 illustrates the relationship between
the array KEYNODES[] and the actual file at the time that the keysort
procedure begins.
There must, of course, be some way of relating the keys back to the
records from which they have been extracted. Consequently, each node of
the array KEYNODES[] has a second field KEYNODES[].RRN that
contains the RRN of the record associated with the corresponding key.
The actual sorting process simply sorts the KEYNODES[] array
according to the KEY field. This produces an arrangement like that shown
To keep
record
things simple,
file
FIGURE 5.14 Conceptual view of KEYNODES array to be used
routine, and record array on secondary store.
RAM
in
by internal sort
**
KEYNODES
array
Records
RRN
KEY
HARRISON SUSAN
Harrison Susan
KELLOG BILL
KelloglBilll 17 Maple...
HARRIS MARGARET
Harris
BELL ROBERT
Bell! Robert
In
RAM
387 Eastern...
Margaret 4343 West...
1
8912
Hill...
On secondary store
210
ORGANIZING FILES FOR PERFORMANCE
%/
3 -^7*
.
KEYNODES array
Records
RRN
KEY
BELL ROBERT
Harrison Susan
HARRIS MARGARET
KelloglBUIl 17 Maple...
HARRISON SUSAN
Harris
KELLOG BILL
In
Bell
RAM
Margaret 4343 West.
I
Robert 8912
I
Hill...
On secondary store
FIGURE 5.15 Conceptual view of
in Fig. 5.15.
way
387 Eastern..
KEYNODES
array and
file
after sorting keys in
The elements of KEYNODES[]
that the first element has the
to the first position in the
file,
are
now
RAM
sequenced in such
RRN of the record that should be moved
the second element identifies the record that
should be second, and so forth.
Once KEYNODES[] is sorted, we are ready to reorganize the file
this new ordering. This process can be described as follows:
according to
for
i:
to
number of records
Seek in the input
file
to the record
whose
RRN
is
KEYNODES[i].RRN.
Read
this
record into
buffer in
RAM.
Write the contents of the buffer out to output
file.
Figure 5.16 outlines the keysort procedure in pseudocode. This pro-
cedure works
much
the
same way
that a
normal
internal sort
would work,
but with two important differences:
RAM
array, we simply read
Rather than read entire records into a
each record into a temporary buffer, extract the key, and then dis-
j card
it;
and
When we
read
them
are writing the records out in sorted order,
in a
second time, since they are not
all
we have
stored in
to
RAM.
211
KEYSORTING
PROGRAM: keysort
open input file as
N F LE
create output file as DUT_FILE
I
read header record from
N F LE and write a copy to 0UT_FILE
:= record count from header record
I
REC_CDUNT
/*
for
read in records;
1
:=
to
up KEYNODES array */
set
REC_C0UNT
N_F LE into BUFFER
read record from
extract canonical key and place it in KE YN0DE5
KEYNODEStil .RRN = i
I
KEY
KEY
thereby ordering RRNs correspondingly */
sort KEYNODESC
sort (KEYNODES, REC_C0UNT)
/*
/*
/*
for
read in records according to sorted order, and write them
in this order
out
i
:=
to
REC_C0UNT
N_F LE to record with RRN of KEYNODES
seek in
N_F LE
read the record into BUFFER from
I
close
end PROGRAM
*/
*/
].
RRN
write BUFFER contents to 0UT_FILE
N_F LE and 0UT_FILE
I
FIGURE 5.16 Pseudocode
for keysort.
fUUJ&
?/w>
5.4.2 Limitations of the Keysort Method
At first glance, keysorting appears to be an obvious improvement over sorts
performed entirely in RAM; it might even appear to be a case of getting
something for nothing. We know that sorting is an expensive operation and
that we want to do it in RAM. Keysorting allows us to achieve this
objective without having to hold the entire
file in
RAM
at
once.
But, while reading about the operation of writing the records out in
sorted order, even a casual reader probably senses a cloud on this apparently
bright horizon. In keysort
before
we
can write out the
we need to read in the records a second time
new sorted file. Doing something twice is never
desirable.
But the problem
Look
them out
to the
input
sequentially. Instead,
file
is
worse than
that.
carefully at the for loop that reads in the records before writing
from the sorted
new
file.
You
can see that
KEYNODES[]
to seek to each record
and read
we
are
to the
it
we
are not reading
RRNs
through the
in sorted order,
moving
of the records. Since
we have
working
in before writing
it
back out, creating the
21 2
ORGANIZING FILES FOR PERFORMANCE
sorted
file
requires as
many random
As we have noted
records.
seeks into the input
number of
between the time required
difference
to
times,
read
all
sequentially and the time required to read those
seek to each record separately.
What
is
worse,
we
file as
there
is
the records in a
same records
are
there are
an enormous
if
performing
accesses in alternation with write statements to the output
file.
file
we must
of these
all
So, even the
which would otherwise appear to be sequential,
most
in
cases involves seeking. The disk drive must move the head back and
forth between the two files as it reads and writes.
writing of the output
file,
The getting-something-for-nothing
aspect of keysort has suddenly
work of sorting in RAM,
from the map supplied
trivial matter when the only
evaporated. Even though keysort does the hard
it
turns out that creating a sorted version of the
by the
KEYNODES[]
array
not
is
at all a
file
copies of the records are kept on secondary store.
5.4.3 Another Solution:
Why
Bother to Write the
The fundamental
idea behind keysort
an entire record
when
is
File
an attractive one:
Back?
Why work
with
the only parts of interest, as far as sorting and
searching are concerned, are the fields used to form the key? There
is
compelling parsimony behind this idea, and it makes keysorting look
promising. The promise fades only when we run into the problem of
rearranging
It is
all
the records in the
interesting to ask
file
so they reflect the new, sorted order.
whether we can avoid
bothering with the task that
this
giving us trouble:
is
problem by simply not
What
if
we just
skip the
time-consuming business of writing out a sorted version of the file? What
if, instead, we simply write out a copy of the array of canonical key nodes?
If we do without writing the records back in sorted order, writing out the
contents of our KEYNODES[] array instead, we will have written a
program that outputs an index to the original file. The relationship between
the
two
This
files is illustrated in Fig.
is
5.17.
an instance of one of our favorite categories of solutions to
computer science problems:
If
some
part of a process begins to look like a
Can you do without it? Instead
of creating a new, sorted copy of the file to use for searching, we have
created a second kind of file, an index file, that is to be used in conjunction
with the original file. If we are looking for a particular record, we do our
binary search on the index file, then use the RRN stored in the index file
record to find the corresponding record in the original file.
There is much to say about the use of index files, enough to fill several
chapters. The next chapter is about the various ways we can use simple
indexes, which is the kind of index we illustrate here. In later chapters we
bottleneck, consider skipping
it
altogether.
KEYSORTING
Index
Original
file
file
BELL ROBERT
Harrison
HARRIS MARGARET
Kellogg
HARRISON SUSAN
Harris
KELLOGG BILL
Belli Robert 18912 Hill
FIGURE 5.17 Relationship between the index
file
Susan 387 Eastern
!
Bill
I
213
17
Maple.
Margaret 4343 West
;
and the data
file.
about different ways of organizing the index to provide more flexible
and easier maintenance.
talk
access
5.4.4 Pinned Records
we discussed the problem of updating and maintaining files.
of that discussion revolved around the problems of deleting records
and keeping track of the space vacated by deleted records so it can be reused.
An avail list of deleted record slots is created by linking all of the available
slots together. This linking is done by writing a link field into each deleted
record that points to the next deleted record. This link field gives very
specific information about the exact physical location of the next available
In section 5.2
Much
record.
When a
we
file
contains such references to the physical locations of records,
say that these records are pinned.
particular choice
these
files
of terminology
containing an avail
that cannot
You
can gain an appreciation for this
you consider
(such as an index
become what
file)
is
file
pinned record
or in
is
some other
one
file
contain references to the physical location of the
moved,
these references
no longer lead
to the record;
are called dangling pointers, pointers leading to incorrect,
meaningless locations in the
file.
Clearly, the use of pinned records in a
difficult
the effects of sorting one of
of deleted records.
be moved. Other records in the same
record. If the record
they
list
if
and sometimes impossible. But what
file
if
can
make
we want
sorting
more
to support rapid
21
ORGANIZING FILES FOR PERFORMANCE
access
by key, while
deletion?
One
solution
still
is
reusing the space
to use an index
records, while keeping the actual data
the
a
problem of finding things
file
to
file in its
made
available
Once again,
we need to take
original order.
leads to the suggestion that
close look at the use of indexes, which, in turn, leads us to the next
chapter.
SUMMARY
we look at ways to organize or reorganize files to improve
performance in some way.
Data compression methods are used to make files smaller by re-encoding
data that goes into a file. Smaller files use less storage, take less time to
transmit, and can often be processed faster sequentially.
The notation used for representing information can often be made more
compact. For instance, if a two-byte field in a record can take on only 50
values, the field can be encoded using only 6 bits instead of 16. Another
form of compression called run-length encoding encodes sequences of
repeating values, rather than writing all of the values in the file.
A third form of compression assigns variable-length codes to values
depending on how frequently the values occur. Values that occur often are
given shorter codes, so they take up less space. Huffman codes are an example
of variable-length codes.
In this chapter
Some compression
techniques are
tion in the encoding process.
The
irreversible in that
UNIX utilities
they lose informa-
compress, uncompress, pack,
and unpack provide good compression in UNIX.
A second way to save space in a file is to recover space in the file after
it has undergone changes. A volatile file, one that undergoes many changes,
can deteriorate very rapidly unless measures are taken to adjust the file
organization to the changes. One result of making changes to files is storage
fragmentation.
Internal fragmentation occurs
In
fixed-length
record
file,
when
there
internal
is
wasted space within
variable-length record
file
when one
a record.
fragmentation can result
variable-length records are stored in fixed slots.
record
is
It
when
can also occur in
replaced by another record of
smaller size. External fragmentation occurs when holes of unused space
between records are created, normally because of record deletions.
There are a number of ways to combat fragmentation. The simplest is
storage compaction, which squeezes out unused space caused by external
fragmentation by sliding all of the undeleted records together. Compaction
is generally done in a batch mode.
a
by record
keep the sorted order of the
SUMMARY
Fragmentation can be dealt with dynamically by reclaiming deleted space
records are added. The need to keep track of the space to be reused
makes this approach more complex than compaction.
We begin with the problem of deleting fixed-length records. Since
when
finding the
first field
of a fixed-length record
can be accomplished by placing
Since
all
records in
is
very easy, deleting
mark
a special
fixed-length record
record
in the first field.
are the
file
same
size,
the reuse
of deleted records need not be complicated. The solution we adopt consists
of collecting all the available record slots into an avail list. The avail list is
created
by stringing together
the deleted records to
all
form
a linked
list
of
deleted record spaces.
In a fixed-length record
other
maintain the linked avail
are
added to the
slots are
any one record
file,
slot
is
just as usable as any
they are interchangeable. Consequently, the simplest
slot;
avail
list is
list
removed from
to treat
it
as a stack.
Newly
way
to
available records
by pushing them onto the front of the list; record
list by popping them from the front of the
the avail
list.
Next,
we
form
still
consider the matter of deleting variable-length records.
linked
of available record
list
We
but with variable-length
slots,
we need to be sure that a record slot is the right size to hold the new
Our initial definition of right size is simply in terms of being big
enough. Consequently, we need a procedure that can search through the
avail list until it finds a record slot that is big enough to hold the new record.
records
record.
Given such
on the
deleted records
complementary function that places newly
list, we can implement a system that deletes and
and
a function,
avail
reuses variable-length records.
We then consider the amount and nature of fragmentation that develops
inside a
file
due to record deletion and
space
internally if the
develop
new
because
procedure that breaks
two or more
into
a
lost
is
it
reuse.
Fragmentation can happen
locked up inside
is
a record.
smaller ones, using exactly as
record, leaving the remainder
on the
much
avail
list.
space as
We
We
record slot
a single, large, variable-length
is
needed for
see that, although
could decrease the amount of wasted space, eventually the remaining
fragments are too small to be useful. When this happens, the space is lost to
this
external fragmentation
There are
fragmentation.
things that one can do to minimize external
(1)
compacting the
of fragmentation becomes excessive;
level
on the
a
number of
They include
avail
list
to
make
larger,
more
placement strategy to select slots
file
in a batch
mode when
generally useful slots; and
for
reuse in a
way
adopting
minimizes
(3)
that
fragmentation. Development of algorithms for coalescing holes
part
of the exercises
more
at
careful discussion.
the
record slots
(2) coalescing adjacent
is
left as
the end of this chapter. Placement strategies need
215
21 6
ORGANIZING FILES FOR PERFORMANCE
The placement
up
strategy used
record deletion and reuse procedures
simply, "If the record slot
in sorted order,
it
is
to this point
is a first-fit
big enough, use
it."
by the variable-length
strategy. This strategy
By
is
list
two other placement
easy to implement either of
is
keeping the avail
strategies:
which a new record is placed in the smallest slot that is
enough to hold it. This is an attractive strategy for variablelength record files in which the fragmentation is internal. It involves
more overhead than other placement strategies.
Worst fit, in which a new record is placed in the largest record slot
Best fit, in
still
big
available.
The
idea
is
to
have the left-over portion of the
slot
be
as
large as possible.
no firm rule for selecting a placement strategy; the best one can do
judgment based on a number of guidelines.
In the third major section of this chapter, we look at ways to find things
quickly in a file through the use of a key. In preceding chapters it was not
There
is
is
use informed
knowing its physical location or
we explore some of the problems and
possible to access a record rapidly without
relative
record number.
Now
opportunities associated with keyed direct access.
This
key
develops
chapter
only
one
method of
binary searching. Binary searching requires O
finding
records
by
comparisons to
(log n)
with n records, and hence is far superior to sequential
works only on a sorted file, a sorting
procedure is an absolute necessity. The problem of sorting is complicated
by the fact that we are sorting files on secondary storage rather than vectors
in RAM. We need to develop a sorting procedure that does not require
seeking back and forth over the file.
Three disadvantages are associated with sorting and binary searching as
developed up to this point:
find a record in a
file
searching. Since binary searching
Binary searching
searching, but
per record.
is
an enormous improvement over sequential
it still
The need
in applications
where
usually requires
more than one
for fewer disk accesses
a large
or
becomes
number of records
two
accesses
especially acute
are to be accessed
by
key.
The requirement
that the file be kept in sorted order can be expenFor active files to which records are added frequently, the cost
of keeping the file in sorted order can outweigh the benefits of binary searching.
sive.
A RAM
sort can be used only
the size of the
files
that
given our sorting tools.
we
on
relatively small
files.
This limits
could organize for binary searching,
KEY TERMS
The
problem can be solved
third
partially
by developing more powerful
sorting procedures, such as a keysort. This approach to sorting resembles a
RAM
sort in
Instead,
most
respects, but does not use
reads in only the keys
it
uses the sorted
from the
RAM
to hold the entire
file.
records, sorts the keys, and then
of keys to rearrange the records on secondary storage so
list
they are in sorted order.
The disadvantage to a keysort is that rearranging a file of n records
random seeks out to the original file, which can take much more
time than does a sequential reading of the same number of records. The
requires
//
inquiry into keysorting
to
not wasted, however. Keysorting naturally leads
is
the suggestion that
we merely w rite
T
the sorted
list
of keys off to
secondary storage, setting aside the expensive matter of rearranging the
This
list
file.
of keys, coupled with RRN tags pointing back to the original
an example of an index. We look at indexing more closely in
records,
is
Chapter
6.
This chapter closes with
cost of sorting
elsewhere
(in
a discussion of another, potentially hidden,
and searching. Pinned records are records that are referenced
same
the
position in the
file.
file
or in
some
other
file)
according to their physical
Sorting and binary searching cannot be applied to a
containing pinned records, since the sorting, by definition,
change the physical position of the record. Such
is
file
likely to
change causes other
record to become inaccurate, creating the problem of
references to this
dangling pointers.
KEY TERMS
Avail
list.
list
of the space, freed through record deletion, that
available for holding
chapter, this
list
new
is
records. In the examples considered in this
of space took the form of
linked
list
of deleted
records.
Best fit. A placement strategy for selecting the space on the avail list
used to hold a new record. Best-fit placement finds the available
record slot that is closest in size to what is needed to hold the new
record.
Binary search. A binary search algorithm locates a key in a sorted list
by repeatedly selecting the middle element of the list, dividing the
list in half, and forming a new, smaller list from the half that conThis process
tains the key.
the kev that
Coalescence.
If
is
is
continued until the selected element
is
sousht.
two
deleted, available records are physically adjacent,
they can be combined to form
a single, larger available
This process of combining smaller available spaces into
record space.
a larger
one
217
21
ORGANIZING FILES FOR PERFORMANCE
is
known
Coalescence
as coalescing holes.
is
way
to counteract the
problem of external fragmentation.
Compaction. A way of getting
all
of
rid
the records together so there
is
by sliding
between them.
external fragmentation
all
no space
Data compression. Encoding information
lost
in a file in
such
way
as to
take up less space.
External fragmentation. A form of fragmentation that occurs in a file
when there is unused space outside or between individual records.
First fit. A placement strategy for selecting a space from the avail list.
First-fit placement selects the first available record slot large enough
to hold the
new
record.
Fragmentation. The unused space within
locked within individual records
a file.
The space can be
(internal fragmentation) or outside or
between individual records (external fragmentation).
code. A variable-length code in which the lengths of the
codes are based on their probability of occurrence.
Internal fragmentation. A form of fragmentation that occurs when
space is wasted in a file because it is locked up, unused, inside of
Huffman
records. Fixed-length record structures often result in internal frag-
mentation.
Irreversible compression. Compression in which information
Keysort.
entire
method of sorting a file that does not require holding the
file in memory. Only the keys are held in memory, along
with pointers that
tie
these keys to the records in the
they are extracted.
The keys
used to construct
new
sorted order.
less
cess
Linked
cific
RAM
file
new
file
from which
list
of keys
is
that has the records in
The primary advantage of a keysort is
a RAM sort. The disadvantage
a
file
and the sorted
than does
of constructing
list.
are sorted,
version of the
that
is
it
requires
that the pro-
requires a lot of seeking for records.
collection of nodes that have been organized into a spe-
sequence by means of references placed in each node that point
to a single successor node.
The
logical
order of
different than the actual physical order of the
er's
is lost.
linked
nodes
list is
in the
often
comput-
memory.
Pinned record.
record
is
pinned
structures that refer to
it
by
its
when
there are other records or
physical location.
It is
file
pinned in the
we are not free to alter the physical location of the record:
doing so destroys the validity of the physical references to the
record. These references become useless dangling pointers.
Placement strategy. As used in this chapter, a placement strategy is a
mechanism for selecting the space on the avail list that is to be used
sense that
to hold a
new
record added to the
file.
EXERCISES
Redundancy reduction. Any form of compression
219
that does not lose
information.
Run-length encoding.
compression method
peated codes are replaced by a count of the
in
which runs of
number of
re-
repetitions
by the code that is repeated.
in which all additions and deletions take
of
the code, followed
Stack.
kind of
list
same end.
Variable-length encoding. Any encoding scheme
place at
the
are
of different lengths.
More
which the codes
in
frequently occurring codes are given
shorter lengths than are frequently occurring codes.
ing
Worst
Huffman encod-
an example of variable-length encoding.
is
fit.
placement strategy for selecting
space from the avail
Worst-fit placement selects the largest record
small the
new
record
is.
slot,
regardless of
list.
how
Insofar as this leaves the largest possible
record slot for reuse, worst
fit
can sometimes help minimize external
fragmentation.
EXERCISES
our discussion of compression, we show how we can compress the
name" field from 16 bits to 6 bits, yet we say that this gives us a space
savings of 50%, rather than 62.5%, as we would expect. Why is this so?
What other measures might we take to achieve the full 62.5% savings?
1.
In
"state
2. What is redundancy reduction?
of redundancy reduction?
Why is run-length encoding an example
3. What is the maximum run length that can be handled in the run-length
encoding described in the text? If much longer runs were common, how
might you handle them?
4.
Encode each of
results,
(a)
(b)
and indicate
01
01
01
01
how you might improve
the algorithm.
01 01 01 01 01 01 02 03 03 03 03 03 03 03 04 05 06 06 07
02 02 03 03 04 05 06 06 05 05 04 04
01
5.
From
Fig. 5.2,
6.
What
is
How
the following using run-length encoding. Discuss the
determine the Huffman code for the sequence "daeab".
the difference between internal and external fragmentation?
can compaction affect the amount of internal fragmentation in
What about
external fragmentation?
a file?
220
ORGANIZING FILES FOR PERFORMANCE
7.
In-placc compaction purges deleted records
separate
new
file.
What
compaction compared
from a file without creating
and disadvantages of in-place
which a separate compacted file is
are the advantages
compaction
to
in
created?
8.
loss
9.
Why
is a worst-fit placement strategy a bad choice
of space due to internal fragmentation?
if
there
is
significant
Conceive of an inexpensive way to keep
amount of fragmentation
in a
a continuous record of the
This fragmentation measure could be
file.
used to trigger the batch processes used to reduce fragmentation.
10. Suppose a file must remain sorted.
placement strategies available?
11.
How
does
this affect the
range of
Develop a pseudocode description of a procedure for performing
compaction in a variable-length record file that contains size fields
in-place
at
the start of each record.
12.
Consider the process of updating rather than deleting
variable-length
record. Outline a procedure for handling such updating, accounting for the
update possibly resulting in either
longer or shorter record.
we raised the question of where to keep the stack
of available records. Should it be a separate list, perhaps
maintained in a separate file, or should it be embedded within the data file?
We choose the latter organization for our implementation. What advantages and disadvantages are there to the second approach? What other kinds
of file structures can you think of to facilitate various kinds of record
13. In section 5.4,
containing the
list
deletion?
14. In
some
the record
is
files,
each record has
inactive rather than deleted.
record?
a delete bit that is set to
to indicate that
deleted. This bit can also be used to indicate that a record
Could
reactivation be
What
is
is
required to reactivate an inactive
done with the deletion procedures we have
used?
we outlined three general approaches to the problem of
minimizing storage fragmentation: (a) implementation of a placement
strategy; (b) coalescing of holes; and (c) compaction. Assuming an
interactive programming environment, which of these strategies would be
used "on the fly," as records are added and deleted? Which strategies would
be used as batch processes that could be run periodically?
15. In this chapter
16.
Why
record
do placement
files?
strategies
make
sense only with variable-length
EXERCISES
17.
Compare
the average case performance of binary search with sequential
search for records, assuming
That the records being sought are guaranteed to be in the file;
That half of the time the records being sought are not in the file; and
That half of the time the records being sought are not in the file and
that missing records must be inserted.
Make
showing your performance comparisons
a table
for files of 1,000,
2,000, 4,000, 8,000, and 16,000 records.
18. If the records in exercise 17 are
how
blocked with 20 records per block,
does this affect the performance of the binary and sequential searches?
19.
An
internal sort
works only with
files
small enough to
fit
in
RAM.
Some computing systems provide users with an almost unlimited amount
of RAM with a memory management technique called virtual storage.
Discuss the use of internal sorting to sort large
files
on systems
that use
virtual storage.
Our discussion of keysorting covers the considerable expense associated
with the process of actually creating the sorted output file, given the sorted
vector of pointers to the canonical key nodes. The expense revolves around
20.
two primary
areas of difficulty:
jump around
file, performing many seeks
new, sorted order; and
Writing the output file at the same time we are reading the input
jumping back and forth between the files can involve seeking.
Having
to
in the input
to
retrieve the records in their
Design an approach to
this
problem
that uses buffers to hold a
number of
records, therefore mitigating these difficulties. If your solution
viable,
obviously the buffers must use
place entirely within electronic
less
RAM
than would
file;
is
a sort
to be
taking
memory.
Programming Exercises
21.
Rewrite the program update. c or
records to
fixed-length record
file
update. pas so
it
can delete and add
using one of the replacement procedures
discussed in this chapter.
program similar to the one described
works with variable-length record files.
22. Write a
but that
23.
Develop
in the preceding exercise,
pseudocode description of a variable-length record deletion
if the newly deleted record is contiguous with
procedure that checks to see
221
222
ORGANIZING FILES FOR PERFORMANCE
any other deleted records.
make
If there
a single, larger available
is
record
contiguity, coalesce the records to
slot.
Some
things to consider as
you
address this problem are as follows:
a.
The
avail
list
does not keep records arranged in physical order; the
list is not necessarily the next deleted record
next record on the avail
in the physical file. Is
avail
list,
you do
b.
possible to
it
merge
two views of the
these
the physical order and the logical order, into a single
this,
what placement strategy
will
list? If
you use?
Physical adjacency can include records that precede as well as fol-
low
the
newly deleted
How
record.
will
you look
for a deleted
record that precedes the newly deleted record?
c.
as
Maintaining two views of the list of deleted records implies that
you discover physically adjacent records you have to rearrange
links to
tions
update the nonphysical avail
would we encounter
if
list.
What
additional complica-
we were combining
the coalescing of
holes with a best-fit or worst-fit strategy?
Implement the bin_search() function in either C or Pascal. Write a driver
program named search to test the function bin_search(). Assume that the files
are created with the update program developed in Chapter 4, and then
sorted. Include enough debug information in the search driver and
bin_search() function to watch the binary searching logic as it makes
successive guesses about where to place the new record.
24.
25.
Modify
the bin_search() function so if the key
the relative record
number
The function should
also
is
not in the
file, it
returns
key would occupy were it in the file.
continue to indicate whether the key was found or
that the
not.
26. Rewrite the search driver
from
exercise 24 so
it
uses the
function developed in exercise 25. If the sought-after key
new
is
bin_search()
in the
file,
the
program should display the record contents. If the key is not found, the
program should display a list of the keys that surround the position that the
key would have occupied. You should be able to move backward or
forward through this list at will. Given this modification, you do not have
to remember an entire key to retrieve it. If, for example, you know that you
are looking for someone named Smith, but cannot remember the person's
first name, this new program lets you jump to the area where all the Smith
records are stored.
you recognize
You
the right
can then scroll back and forth through the keys until
first
name.
27. Write an internal sort that can sort a variable-length record
kind produced by the writrec programs in Chapter
4.
file
of the
FURTHER READINGS
FURTHER READINGS
A
thorough treatment of data compression techniques can be found in Lynch (1985).
is described in Welch (1984). Huffman encoding is covered
The Lempel-Ziv method
in
many data structures texts, and also in Knuth (1973a).
Somewhat surprising, the literature concerning storage
reuse often does not consider these issues
fragmentation and
from the standpoint of secondary
storage.
Typically, storage fragmentation, placement strategies, coalescing of holes, and
garbage collection are considered in the context of reusing space within electronic
random
access
memory (RAM). As you
the concepts to secondary storage,
it is
read this literature with the idea of applying
necessary to evaluate each strategy in light of
the cost of accessing secondary storage.
used in electronic
RAM
Some
are too expensive
Discussions about space
management
strategies that are attractive
on secondary
in
RAM
when
storage.
are usually
heading "Dynamic Storage Allocation." Knuth (1973a) provides
found under the
a good, though
overview of the fundamental concerns associated with dynamic storage
placement strategies. Much of Knuth's discussion is reworked
and made more approachable by Tremblay and Sorenson (1984). Standish (1980)
provides a more complete overview of the entire subject, reviewing much of the
important literature on the subject.
This chapter only touches the surface of issues relating to searching and sorting
files. A large part of the remainder of this text is devoted to exploring the issues in
more detail, so one source for further reading is the present text. But there is much
more that has been written about even the relatively simple issues raised in this
chapter. The classic reference on sorting and searching is Knuth (1973b). Knuth
provides an excellent discussion of the limitations of keysort methods. He also
develops a very complete discussion of binary searching, clearly bringing out the
analogy between binary searching and the use of binary trees. Baase (1978) provides
a clear, understandable analysis of binary search performance.
technical,
allocation, including
223
Indexing
CHAPTER OBJECTIVES
Introduce concepts of indexing that have broad apfile systems.
plications in the design of
Introduce the use of a simple linear index to provide
rapid access to records in an entry-sequenced, variable-length record file.
Investigate the implications of the use of indexes
for
file
maintenance.
Describe the use of indexes to provide access to
records by more than one key.
Introduce the idea of an inverted
Boolean operations on lists.
Discuss the issue of when
address in the data file.
iS
to
list,
illustrating
bind an index
key
to an
Introduce and investigate the implications of selfindexing
files.
225
CHAPTER OUTLINE
6.1
What
6.2
6.3
Is
an Index?
6.6
Secondary Keys
Simple Index with an EntrySequenced File
6.7
Improving the Secondary Index
Structure: Inverted Lists
Basic Operations on an Indexed,
Entry-Sequenced
Retrieval Using Combinations of
6.7.1
File
6.7.2
6.4
Indexes That Are
Hold
6.5
in
Too Large
to
First Attempt at a Solution
Better Solution: Linking the
List of References
Memory
Indexing to Provide Access by
Multiple Keys
6.1
A
A
What
Is
6.8
Selective Indexes
6.9
Binding
an Index?
few pages of many books contain an index. Such an index is a table
a list of topics (keys) and numbers of pages where the topics can
be found (reference fields).
All indexes are based on the same basic concept
keys and reference
fields. The types of indexes we examine in this chapter are called simple
The
last
containing
indexes because they are represented using simple arrays
contain the keys and reference
fields. In later
chapters
of structures that
we
look
at
indexing
schemes that use more complex data structures, especially trees. In this
chapter, however, we want to emphasize that indexes can be very simple
and still provide powerful tools for file processing.
The index to a book provides a way to find a topic quickly. If you have
ever had to use a book without a good index, you already know that an
index is a desirable alternative to scanning through the book sequentially to
find a topic. In general, indexing is another way to handle the problem that
we explored in Chapter 5: An index is a way to find things.
Consider what would happen if we tried to apply the previous chapter's
methods, sorting and binary searching, to the problem of finding things in
a book. Rearranging all the words in the book so they were in alphabetical
order certainly would make finding any particular term easier but would
obviously have disastrous effects on the meaning of the book. In a sense, the
terms in the book are pinned records. This is an absurd example, but it
clearly underscores the power and importance of the index as a conceptual
tool. Since
it
works by
indirection, an index
without actually rearranging the
file.
lets
you impose order on a
file
This not only keeps us from disturbing
A SIMPLE INDEX WITH AN ENTRY-SEQUENCED
pinned records, but also makes matters such
expensive than they are with
Take,
as
as
227
FILE
much
record addition
less
a sorted file.
another example, the problem of finding books in
We want to be able to locate books by a specific author,
One way of achieving
by
a library.
their titles, or
by
have three copies of each book
and three separate library buildings. All of the books in one building would
be sorted by author's name, another building would contain books arranged
by title, and the third would have them ordered by subject. Again, this is an
absurd example, but one that underscores another important advantage of
subject areas.
this is to
indexing. Instead of using multiple arrangements, a library uses a card
catalog.
The
is actually a set of three indexes, each using a
and all of them using the same catalog number as a
Another use of indexing, then, is to provide multiple access
card catalog
different key field,
reference field.
paths to a
We
file.
also find that indexing gives us keyed access
to
variable-length record
our discussion of indexing by exploring this problem of
access to variable-length records and the simple solution that indexing
files.
Let's begin
provides.
6.2
A Simple Index
with an Entry-Sequenced File
Suppose we own an extensive collection of musical recordings and we want
keep track of the collection through the use of computer files. For each
recording, we keep the information shown in Fig. 6. 1. The data file records
are variable length. Figure 6.2 illustrates such a collection of data records.
to
We
refer to this data record file as Datafle.
There
are a
number of approaches
variable-length record
file
that could be used to create a
to hold these records; the record addresses used
in Fig. 6.2 suggest that each record
skip sequential access and easier
be preceded by
file
a size field that
maintenance. This
is
permits
the structure
we
use.
Suppose
initials for
identification
number
Title
Composer or composers
Artist or artists
Label (publisher)
we formed
the record
primary key for these records consisting of the
company
label
combined with
FIGURE 6.1 Contents of a data record.
the record
company's
228
Rec.
addr.
32t
77
INDEXING
ID
number
Label
LO\
RCA
2312
2626
Title
Composer(s)
Romeo and Juliet
C
Prokofiev
Maazel
Beethoven
Julliard
Corea
Corea
Beethoven
Giulini
Springsteen
Springsteen
Quartet in
Artist(s)
Sharp Minor
132
167
WAR
ANG
396
COL
DG
MER
COL
DG
442
FF
211
256
300
353
Touchstone
23699
Symphony No.
3795
38358
Nebraska
18807
Symphony No.
Coq d'or Suite
Symphony No.
75016
31809
139201
245
Beethoven
Karajan
Rimsky-Korsakov
Leinsdorf
Dvorak
Bernstein
Violin Concerto
Beethoven
Good News
Sweet Honev
the
tAssume there
is
Ferras
Sweet Honev in
in
Rock
the
Rock
a header recoi d that uses the first 32 bytes.
FIGURE 6.2 Sample contents of Datafile.
ID number. This
will
make
good primary key
since
it
should provide
file. We call this key the Label ID. The
ID consists of the uppercase form of the Label
followed immediately by the ASCII representation of the ID number.
unique key for each entry in the
form
canonical
field
for the Label
For example,
LDN2312
How
we
keyed access to
and then use binary searching?
Unfortunately, binary searching depends on being able to jump to the
middle record in the file. This is not possible in a variable-length record file
there is no
because direct access by relative record number is not possible
way to know where the middle record is in any group of records.
could
organize the
Could we
individual records?
file
sort the
to provide rapid
file
An
alternative to sorting
illustrates
such an index.
is
to construct an index for the
On the right is
the data
file
file.
Figure 6.3
containing information
about our collection of recordings, with one variable-length data record per
recording. Only four fields are shown (Label, ID number, Title, and
Composer), but
it is
easy to imagine the other information
filling
out each
record.
On the left is the index file,
key
data
(left justified,
file.
Each key
blank
is
filled)
each record of which contains
corresponding
12-character
to a certain Label
ID
in the
associated with a reference field giving the address of the
A SIMPLE INDEX WITH AN ENTRY-SEQUENCED
Indexfile
Key
229
FILE
Datafile
Reference
Address of
field
record
Actual data record
ANG3795
167
32
LON
COL31809
353
77
RCA
COL38358
211
132
WAR
DG139201
396
167
ANG
DG18807
256
211
COL
FF245
442
256
DG
32
300
MER
300
353
COL
77
396
DG
132
442
FF 245 Good News Sweet Honey In The
LON2312
MER75016
RCA2626
WAR23699
2312
2626 Quartet in
C Sharp Minor
38358 Nebraska Springsteen
ANG3795
The
full
31809 Symphony No. 9 Dvorak
\
Violin Concerto
Beethoven
file.
ANG3795, for example,
number 167, meaning that
containing the
field
Symphony No. 9 Beethoven
byte of the corresponding data record.
the record containing
75016 Coq d'or Suite Rimsky
corresponds to the reference
FIGURE 6.3 Sample index with corresponding data
first
3795 Symphony No. 9 Beethoven
139201
Prokofiev
23699 Touchstone Corea
18807
Romeo and Juliet
information on the recording with Label ID
can be found starting
structure of the index
byte
at
file is
number 167
very simple.
It is
in the record
a
file.
fixed-length record
which each record has two fixed-length fields: a key field and a
byte-offset field. There is one record in the index file for every record in the
file
in
data
file.
Note
also
that
the index
is
Consequently, although Label ID
it is
not necessarily the
first
sorted,
whereas the data
ANG3795
entry in the data
is
the
file.
first
file
is
not.
entry in the index;
In fact, the data
file is
entry
which means that the records occur in the order that they are
entered into the file. As we see soon, the use of an entry-sequenced file can
sequenced,
make record
with
addition and
a data file that is
file
maintenance
kept sorted by
some
much
key.
simpler than
is
the case
230
INDEXING
PROCEDURE retrieve_record(KEY)
find position of KEY in Indexfile /* Probably using binary search */
look up the BYTE_0FFSET of the corresponding record in Datafile
use SEEK() and the byte_offset to move to the data record
read the record from Datafile
end PROCEDURE
FIGURE 6.4 Retrieve _record():
procedure to retrieve a single record from Datafile through
Indexfile.
Using the index to provide access
The steps needed to retrieve
to the data
matter.
shown
Datafile are
in the
a single
file
procedure retrieve_record(
this retrieval strategy is relatively straightforward,
We
now
are
The index
because
with
it
By
dealing with
file is
two
files
contains
simple
KEY
in Fig. 6.4.
it
the index
from
Although
some
features
uses fixed-length records (which
is
likely
it is
and the data
file
work with
considerably easier to
binary search) and because
the data
is
comment:
that deserve
by Label ID
record with key
than the data
file.
file
why we can search it
to be much smaller than
file.
file have fixed-length records, we impose
on the sizes of our keys. In this example we assume that the
primary key field is long enough to retain every key's unique iden-
requiring that the index
a limit
tity.
The
problems
use of a small, fixed key field in the index could cause
if a
key's uniqueness
is
truncated
away
as
it is
placed in the
fixed index field.
In the example, the index carries
no information other than
and the reference fields, but this need not be the
example, keep the length of each Datafile record
6.3
case.
We
the keys
could, for
in Indexfile.
Basic Operations on an Indexed, Entry-Sequenced
We
have noted that the process of keeping
searching for records can be very expensive.
using
sorted to permit binary
One of the
simple index with an entry-sequenced data
addition can take place
long
files
as the
index
is
much more
small
enough
record length
is
consisting of
no more than
short, this
is
not
a
quickly than with
to be held entirely in
a difficult
File
great advantages of
file
is
that record
sorted data
memory.
file as
If the index
condition to meet for small
files
few thousand records. For the moment our
231
BASIC OPERATIONS ON AN INDEXED, ENTRY-SEQUENCED FILE
discussions assume that the condition
is
met and
that the index
secondary storage into an array of structures called
when
consider what should be done
the index
is
INDEX[
is
].
read from
Later
too large to
fit
we
into
memory.
Keeping the index in memory as the program runs also lets us find
records by key more quickly with an indexed file than with a sorted one
since the binary searching can be performed entirely in memory. Once the
byte offset for the data record
is
hand, requires
found, then
The use of
required to retrieve the record.
a single
sorted data
is
file
that
is
coupled with
simple index requires the development of procedures to handle
of different
all
on the other
file,
seek for each step of the binary search.
The support and maintenance of an entry-sequenced
a
seek
tasks. Besides the retrieve_record(
ously, other procedures used to find things
number
algorithm described previ-
by means of the index include
the following:
Create the original empty index and data
Load the index
Rewrite the index
Add
files;
memory before using it;
from memory after using it;
into
file
file
records to the data
file
and index;
file; and
Delete records from the data
Update records
in the data
file.
Creating the Files
empty
files,
quite easily
Both the index file and the data file are created as
with header records and nothing else. This can be accomplished
by creating the files and writing headers to both files.
Memory We assume that the index file is
Loading the Index into
small
to
memory, so we define an array INDEX[ ] to
hold the index records. Each array element has the structure of an index
enough
fit
into primary
record. Loading the index
file
into
memory,
then,
is
simply
matter of
reading in and saving the index header record and then reading the records
from the index
file
into the
INDEX[
array. Since this will be a sequential
and since the records are short, the procedure should be written so it
reads a large number of index records at once, rather than one record at a
read,
time.
File from Memory
When processing of an
indexed file is completed, it is necessary to rewrite INDEXf ] back into the
index file if the array has been changed in any way. In Fig. 6.5, the
Rewriting the Index
procedure rewrite_index(
describes the steps for doing this.
232
INDEXING
PROCEDURE rewrite_index(
check a status flag that tells whether the INDEX [] array
has been changed in any way.
if there were changes, then
open the index file as a new empty file
update the header record and rewrite the header
write the index out to the newly created file
close the index file
end PROCEDURE
FIGURE 6.5 The rewrite_index() procedure.
It is
important to consider what happens
rewriting of the index
if this
does not take place, or takes place incompletely. Programs do not always
run to completion.
failures, against the
program designer needs
guard against power
to
wrong
operator turning the machine off at the
other such disasters.
One of the
time, and
serious dangers associated with reading an
memory and then writing it out when the program is over is that
copy of the index on'disk will be out of date and incorrect if the program
interrupted. It is imperative that a program contain at least the following
index into
the
is
two safeguards
Q'
to protect against this kind
There should be
when
the index
of
error:
mechanism
that permits the
is
out of date.
One
program
to
know
possibility involves setting a sta-
copy of the index in memory is changed. This
be written into the header record of the index file
on disk as soon as the index is read into memory, and then subsequently cleared when the index is rewritten. All programs could
check the status flag before using an index. If the flag is found to be
tus flag as
soon
as the
status flag could
set,
then the program would
If a
program
know
detects that an index
that the index
is
out of date.
is
out of date, the program must
have access to a procedure that reconstructs the index from the data
file. This should happen automatically, taking place before any attempt is made to use the index.
Record Addition
Adding
new
record to the data
file
requires that
we
add a record to the index file. Adding to the data file itself is easy. The
exact procedure depends, of course, on the kind of variable-length file
also
233
BASIC OPERATIONS ON AN INDEXED, ENTRY-SEQUENCED FILE
organization being used. In any case,
when we add
know
file
data record
we
should
which we wrote the
record. This information, along with the canonical form of the record's
array.
key, must be placed in the INDEX[
array is kept in sorted order by key, insertion of
Since the INDEX[
the starting byte_offset of the
location at
new
index record probably requires some rearrangement of the index.
way, the situation is similar to the one we face as we add records to a
sorted data file. We have to shift or slide all the records that have keys that
come in order after the key of the record we are inserting. The shifting
opens up a space for the new record. The big difference between the work
we have to do on the index records and the work required for a sorted data
array is contained wholly in memory. All of the
file is that the INDEX[
index rearrangement can be done without any file access.
the
In a
Record Deletion
In
Chapter 5
we
describe a
deleting records in variable-length record
files
number of approaches
that allow for the reuse
to
of the
space occupied by these records. These approaches are completely viable for
our data file since, unlike a sorted data file, the records in this file need not
be moved around to maintain an ordering on the file. This is one of the great
advantages of an indexed file organization: We have rapid access to
individual records by key without disturbing pinned records. In fact, the
indexing
Of
itself pins all the records.
when we
from the data file we must also
from our index file. Since the index is
an array during program execution, deleting the index record
the other records to close up the space may not be an overly
course,
delete a record
delete the corresponding entry
contained in
and shifting
expensive operation. Alternatively,
as deleted, just as
we might mark
Record Updating
Q
we
could simply mark the index record
the corresponding data record.
Record updating
falls
into
two
categories:
The update changes
bring about
the value of the key field. This kind of update can
reordering of the index file as well as the data file.
Conceptually, the easiest
deletion followed
by an
implemented while
that he or she
The update
is
still
way
to think
of
this
kind of change
is
as a
addition. This delete/add approach can be
providing the program user with the view
merely changing
a record.
does not affect the key field. This
not require rearrangement of the index
second kind of update does
but may well involve re-
file,
file. If the record size is unchanged or decreased
by the update, the record can be written directly into its old space,
but if the record size is increased by the update, a new slot for the
ordering of the data
234
INDEXING
record will have to be found. In the
the rewritten record
field
6.4
latter case the starting
must replace the old address
address of
in the byte_offset
of the corresponding index record.
Indexes That Are Too Large to Hold
in
Memory
The methods we have been
discussing, and, unfortunately, many of the
advantages associated with them, are tied to the assumption that the index
file is small enough to be loaded into memory in its entirety. If the index is
large for this approach to be practical, then index access and
maintenance must be done on secondary storage. With simple indexes of
the kind we have been discussing, accessing the index on a disk has the
too
following disadvantages:
Binary searching of the index requires several seeks rather than takmemory speeds. Binary searching of an index
on secondary storage is not substantially faster than the binary
ing place at electronic
searching of a sorted
file.
Index rearrangement due to record addition or deletion requires shifting or sorting records on secondary storage. This is literally millions
of times more expensive than the cost of these same operations when
performed in electronic memory.
Although these problems are no worse than those associated with the
file that is sorted by key, they are severe enough to warrant the
use of any
consideration of alternatives.
in
memory, you should
A
A
Any
time
simple index
too large to hold
consider using
hashed organization if access speed
tree-structured index,
such
is
as a B-tree, if
top priority; or
you need
both keyed access and ordered, sequential
These alternative
is
file
the flexibility of
access.
organizations are discussed at length in the
chapters that follow. But, before writing off the use of simpje indexes on
secondary storage altogether, we should note that they provide some
important advantages over the use of a data file sorted by key even if the
index cannot be held in memory:
simple index makes
keyed access
it
possible to use a binary search to obtain
to a record in a variable-length record
file.
The index
provides the service of associating a fixed-length and therefore binary-searchable record with each variable-length data record.
235
INDEXING TO PROVIDE ACCESS BY MULTIPLE KEYS
If the
index records are substantially smaller than the data
file
records, sorting and maintaining the index can be less expensive than
would be
,
sorting and maintaining the data
cause there
is
less
information to
If there are pinned records in the data
rearrange the keys without
There
one
that
is
file.
move around
moving
file,
This
is
in the
simply be-
index
file.
the use of an index lets us
the data records.
another advantage associated with the use of simple indexes,
we have
not yet discussed.
It,
in itself, can
be reason enough to use
simple indexes even if they do not fit into memory. Remember the analogy
between an index and a library card catalog? The card catalog provides
multiple views or arrangements of the library's collection, even though
there is only one set of books arranged in a single order. Similarly, we can
use multiple indexes to provide multiple views of a data
6.5
file.
Indexing to Provide Access by Multiple Keys
One
question that might reasonably arise
business
using
is
pretty interesting, but
key such
as
at this
who would
DG18807? What
want
is
point
is,
"All this indexing
ever want to find a record
the
Symphony No.
9 record
by Beethoven."
our analogy between our index and a library card
Suppose we think of our primary key, the Label ID, as a kind of
catalog number. Like the catalog number assigned to a book, we have taken
care to make our Label ID unique. Now, in a library it is very unusual to
begin by looking for a book with a particular catalog number (e.g., "I am
looking for a book with a catalog number QA331T5 1959."). Instead, one
generally begins by looking for a book on a particular subject, with a
particular title, or by a particular author (e.g., "I am looking for a book on
functions," or "I am looking for The Theory of Functions by Titchmarsh.").
Given the subject, author, or title, one looks in the card catalog to find the
primary key, the catalog number.
Let's return to
catalog.
Similarly, we could build a catalog for our record collection consisting
of entries for album title, composer, and artist. These fields are secondary key
fields.
Just as the library catalog relates an author entry (secondary key) to a
card catalog
Composer
number (primary
Along with
this
key), so can
we build
an index
the similarities, there
is
that relates
an important difference between
kind of secondary key index and the card catalog
library,
file
to Label ID, as illustrated in Fig. 6.6.
in a library.
In a
once you have the catalog number you can usually go directly to the
236
INDEXING
Composer index
Secondary key
Primary key
BEETHOVEN
ANG3795
BEETHOVEN
DG139201
BEETHOVEN
DG18807
key
BEETHOVEN
RCA2626
^4 5i^i9^ r/^.
COREA
WAR23699
DVORAK
COL31809
PROKOFIEV
LON2312
RIMSKY-KORSAKOV
MER75016
SPRINGSTEEN
COL38358
SWEET HONEY IN THE R
FF245
book
**>
p^;TF6r^'^t'
ja>
FIGURE 6.6 Secondary key
index organized by composer.
books are arranged in order by catalog
books are sorted by primary key. The actual
data records in our file, on the other hand, are entry sequenced. Consequently, after consulting the composer index to find the Label ID, you must
consult one additional index, our primary key index, to find the actual byte
offset of the record that has this particular Label ID. The procedure is
summarized in Fig. 6.7.
Clearly it is possible to relate secondary key references (e.g.,
Beethoven) directly to a byte offset (211) rather than to a primary key
(DG18807). However, there are excellent reasons for postponing this
binding of a secondary key to a specific address for as long as possible.
These reasons become clear as we discuss the way that fundamental file
operations such as record deletion and updating are affected by the use of
stacks to find the
number.
since the
In other words, the
secondary indexes.
Record Addition
the
file
means adding
When
a
secondary index
is
present, adding a record to
record to the secondary index.
The
cost of doing this
237
INDEXING TO PROVIDE ACCESS BY MULTIPLE KEYS
PROCEDURE search_on_secondary KEY
(
search for KEY in the secondary index
once the correct secondary index record is found, set LABEL_ID
to the primary key value in the record's reference field
call retrieve _record(LABEL_ID) to get the data record
end PROCEDURE
FIGURE 6.7 Search _on_secondary: an algorithm to retrieve a single record from Datafile
through a secondary key index.
is
very similar to the cost of adding
record to the primary index: Either
must be shifted or a vector of pointers to structures needs to be
rearranged. As with primary indexes, the cost of doing this decreases
records
greatly if the secondary index can be read into electronic
changed
memory and
there.
Note that the key field in the secondary index file is stored in canonical
form (all of the composers' names are capitalized), since this is the form that
we want to use when we are consulting the secondary index. If we want to
print out the name in normal, mixed upper- and lowercase form, we can
pick up that form from the original data file. Also note that the secondary
keys are held to a fixed length, which means that sometimes they are
truncated. The definition of the canonical form should take this length
restriction into account if searching the index is to work properly.
One
important difference between
secondary index and
primary
sample
index illustrated in Fig. 6.6, there are four records with the key
BEETHOVEN. Duplicate keys are, of course, grouped together. Within
this group, they should be ordered according to the values of the reference
index
is
that a secondary index can contain duplicate keys. In the
fields. In this
example, that means placing them in order by Label ID. The
reasons for this second level of ordering
discuss retrieval based
Record Deletion
become
clear a little later, as
we
on combinations of two or more secondary keys.
Deleting
ences to that record in the
file
record usually implies removing
system. So, removing
would mean removing not only
all
refer-
record from the data
the corresponding record in the
in the secondary indexes that refer
of
the
records
primary index, but also all
to this primary index record. The problem with this is that secondary
indexes, like the primary index, are maintained in sorted order by key.
file
238
INDEXING
Consequently, deleting
would involve rearranging
record
records to close up the space
open by
left
the remaining
deletion.
This delete-all-references approach would indeed be advisable if the
secondary index referenced the data file directly. If we did not delete the
secondary key references, and if the secondary keys were associated with
actual byte offsets in the data file, it could be difficult to tell when these
references
were no longer
The
record problem.
would be pointing
and subsequent
be associated with different data records.
have carefully avoided referencing actual addresses in the
we
search to find the secondary key,
we do
time on primary key. Since the primary index does
changes due to record deletion, a search for the primary key of a
another search,
record
another instance of the pinned-
file,
secondary key index. After
reflect
is
to byte offsets that could, after deletion
space reuse in the data
But we
This
valid.
reference fields associated with the secondary keys
that
this
been deleted will fail, returning
sense, the updated primary key index
has
condition. In a
check, protecting us
from trying
to retrieve records that
Consequently, one option that
from the data
file is
to
open
is
to us
record-not-found
acts as a
kind of final
no longer
when we
exist.
delete a record
modify and rearrange only the primary key index.
We could safely leave intact the references
to the deleted record that exist in
the secondary key indexes. Searches starting
that lead to a deleted record are caught
from
when we
secondary key index
consult the primary key
index.
If there are a
from not having
number of secondary key
to rearrange
can be substantial. This
is
of these indexes
all
especially important
indexes are kept on secondary storage.
system, where the user
is
indexes, the savings that result
waiting
at a
It is
also
when a record is deleted
when the secondary key
important
in
an interactive
terminal for the deletion operation to
complete.
There
is,
of course,
a cost associated
with
this short cut:
Deleted records
With a file system that undergoes
few deletions, this is not usually a problem. With a somewhat more volatile
file structure, it is possible to address the problem by periodically removing
from the secondary index files all records that contain references that are no
take up space in the secondary index
files.
system is so volatile that even periodic
purging is not adequate, it is probably time to consider another index
structure, such as a B-tree, which allows for deletion without having to
rearrange a lot of records.
longer in the primary index. If a
file
In our discussion of record deletion, we find that the
primary key index serves as a kind of protective buffer, insulating the
Record Updating
239
RETRIEVAL USING COMBINATIONS OF SECONDARY KEYS
secondary indexes from changes
in the data
file.
This insulation extends to
record updating as well. If our secondary indexes contain references directly
to byte offsets in the data
file,
then updates to the data
file
that result in
changing a record's physical location in the file also require updating the
secondary indexes. But, since we are confining such detailed information to
the primary index, data file updates affect the secondary index only when
they change either the primary or the secondary key. There are three
possible situations:
Update changes the secondary key: If the secondary key
we may have
to rearrange the secondary
sorted order. This can be
is
key index so
changed, then
it
stays in
a relatively expensive operation.
Update changes the primary key: This kind of change has
a large
on the primary key index, but often requires only
we
that
affected reference field (Label_id in our example) in
all
impact
update the
the secondary
indexes. This involves searching the secondary indexes (on the un-
changed secondary keys) and rewriting the affected fixed-length
does not require reordering of the secondary indexes unless
the corresponding secondary key occurs more than once in the index. If a secondary key does occur more than once, there may be
some local reordering, since records having the same secondary key
are ordered by the reference field (primary key).
Update confined to other fields: All updates that do not affect either the
primary or secondary key fields do not affect the secondary key index, even if the update is substantial. Note that if there are several
secondary key indexes associated with a file, updates to records often
affect only a subset of the secondary indexes.
field. It
6.6
Retrieval Using Combinations of Secondary Keys
One of the most important applications of secondary keys involves using
two or more of them in combination to retrieve special subsets of records
from the data file. To provide an example of how this can be done, we will
extract another secondary key index from our file of recordings. This one
uses the recording's
title
respond to requests such
as the key, as illustrated in Fig. 6.8.
all
poser);
COL38358 (primary key access);
work (secondary key com-
the recordings of Beethoven's
and
can
as
Find the record with Label ID
Find
Now we
240
INDEXING
Title index
Primary key
Secondary key
COQ DOR SUITE
MER75016
GOOD NEWS
FF245
NEBRASKA
COL38358
QUARTET
IN C
SHARP M
RCA2626
ROMEO AND JULIET
LON2312
SYMPHONY NO.
ANG3795
SYMPHONY NO.
COL31809
SYMPHONY NO.
DG18807
TOUCHSTONE
WAR23699
VIOLIN CONCERTO
DG139201
FIGURE 6.8 Secondary key
index organized by recording
title.
Find
all
the recordings titled "Violin Concerto" (secondary key
title).
What
is
more
interesting,
however,
is
that
we
can also respond to
request that combines retrieval on the composer index with retrieval on the
title index, such as: Find all recordings of Beethoven's Symphony No. 9.
Without the use of secondary indexes, this kind of request requires a
sequential search through the entire file. Given a file containing thousands,
or even just hundreds, of records, this is a very expensive process. But, with
the aid of secondary indexes, responding to this request is simple and quick.
We begin by recognizing that this request can be rephrased as a Boolean
AND operation,
specifying the intersection of
two
Find all data records with:
composer = "BEETHOVEN" AND title
subsets of the data
"SYMPHONY NO.
file:
9'
241
RETRIEVAL USING COMBINATIONS OF SECONDARY KEYS
We begin
our response to this request by searching the composer index
of Label IDs that identify records with Beethoven as the
composer. (An exercise at the end of this chapter describes a binary search
procedure that can be used for this kind of retrieval.) This yields the
following list of Label IDs:
for the
list
ANG3795
DG139201
DG18807
RCA2626
Next we
that
search the
title
index for the Label IDs associated with records
SYMPHONY NO.
have
9 as the
key:
title
ANG3795
COL31809
DG18807
Now we
perform the Boolean
combining the
in the
output
lists
which
is
match operation,
that appear in both
lists
are placed
list.
Composers
ANG3795
DG139201
DG 18807
RCA2626
We
AND,
members
so only the
Titles
ANG3795
CDL31809
>DG 18807
>
Matched list
>ANG379S
>DG18807
'
give careful attention to algorithms for performing this kind of
Note that this kind of matching is much
combined are in sorted order. That is the
reason why, when we have more than one entry for a given secondary key,
the records are ordered by the primary key reference fields.
match operation
in
Chapter
7.
easier if the lists that are being
once
Finally,
we
file
we have
the
list
of primary keys occurring
in
both
lists.
can proceed to the primary key index to look up the addresses of the data
records.
ANG
DG
',
',
This
3795
18807
is
useful in a
Then we can
!
retrieve the records:
Symphony No.
Symphony No.
',
Beethoven
Beethoven
Guilini
Karajan
makes computer-indexed file systems
manual systems. We have
record, and yet. working through the
that far exceeds the capabilities of
secondary indexes,
in
the kind of operation that
way
only one copy of each data
them
9
9
order by
we have
title,
file
multiple views of these records:
by composer, or by any other
We can look at
field that interests us.
242
INDEXING
Using the computer's ability to combine sorted lists rapidly, we can even
combine different views, retrieving intersections (Beethoven AND Symphony No. 9) or unions (Beethoven OR Prokofiev OR Symphony No. 9) of
these views. And since our data file is entry sequenced, we can do all of this
without having to sort data file records, confining our sorting to the smaller
index records which can often be held in electronic memory.
Now
indexes,
we have
that
we
can look
of the design and uses of secondary
improve these indexes so they take less
a general idea
at
ways
to
space and require less sorting.
6.7
Improving the Secondary Index Structure:
Inverted Lists
The secondary index
structures that
we have
developed so
far result in
two
distinct difficulties:
We
have to rearrange the index file every time a new record is added
to the file, even if the new record is for an existing secondary key.
For example, if we add another recording of Beethoven's Symphony
No. 9 to our collection, both the composer and title indexes would
have to be rearranged, even though both indexes already contain entries for secondary keys (but not the Label IDs) that are being added.
If there are duplicate secondary keys, the secondary key field is repeated for each entry. This wastes space, making the files larger than
necessary. Larger index files are less likely to be able to fit in elec-
tronic
6.7.1 A
One
memory.
First
Attempt
at a Solution
simple response to these
structure so
example,
it
we might
use
is to change the secondary index
of references with each secondary key. For
difficulties
associates an array
record structure that allows us to associate up to
four Label ID reference fields with
BEETHOVEN
ANG3795
a single
DG139201
secondary key,
DG18807
as in
RCA2626
Figure 6.9 provides a schematic example of how such an index would look
if
used with our sample data
file.
243
IMPROVING THE SECONDARY INDEX STRUCTURE: INVERTED LISTS
Revised composer index
Set of primary key references
Secondary key
BEETHOVEN
ANG3795
COREA
WAR23699
DVORAK
COL31809
PROKOFIEV
LON2312
RIMSKY-KORSAKOV
MER75016
SPRINGSTEEN
COL38358
SWEET HONEY IN THE R
FF245
DG139201
FIGURE 6.9 Secondary key index containing space
RCA2626
DG18807
for multiple references for
each secondary
key.
The major
solution of our
contribution of this revised index structure
first difficulty:
is
toward the
the need to rearrange the secondary index
file
Looking at Fig. 6.9, we can
see that the addition of another recording of a work by Prokofiev does not
require the addition of another record to the index. For example, if we add
every time a
new
record
is
added
to the data
file.
the recording
ANG
36193
we need
to
Piano Concertos
and
Prokofiev
Francois
modify only the corresponding secondary index record by
inserting a second Label ID:
PROKOFIEV
ANG36193
LON2312
Since we are not adding another record to the secondary index, there is no
need to rearrange any records. All that is required is a rearrangement of the
fields in the existing
record for Prokofiev.
244
INDEXING
Although
this
secondary index
it
new
file
structure helps avoid the need to rearrange the
so often,
it
does have some problems. For one thing,
provides space for only four Label IDs to be associated with
a given key.
very likely case that more than four Label IDs will go with some key,
need a mechanism for keeping track of the extra Label IDs.
In the
we
second problem has to do with space usage. Although the structure
does help avoid the waste of space due to the repetition of identical keys,
this
space savings comes
at a potentially
high
cost.
By
length of each of the secondary index records to hold
we might
easily lose
by not repeating
more
extending the fixed
more
reference fields,
space to internal fragmentation than
we
gained
identical keys.
we don't want to waste any more space than we have to, we need
whether we can improve on this record structure. Ideally, what we
would like to do is develop a new design, a revision of our revision, that
Since
to ask
Retains the attractive feature of not requiring reorganization of the
secondary indexes for every
new
entry to the data
file;
Allows more than four Label IDs to be associated with each secondary key; and
Does away w ith the waste of space due to internal fragmentation.
T
6.7.2 A Better Solution: Linking the
List of
References
such as our secondary indexes, in which a secondary key leads to a set
of one or more primary keys, are called inverted lists. The sense in which the
list is inverted should be clear if you consider that we are working our way
backward from a secondary key to the primary key to the record itself.
The second word in the term inverted list also tells us something
Files
important: that
Our
we
are, in fact, dealing
with
list
of primary key references.
number of Label
revised secondary index, which
IDs for each secondary key, reflects this list aspect of the data more directly
than did our initial secondary index. Another way of conceiving of this list
aspect of our inverted list is illustrated in Fig. 6.10.
As Fig. 6. 10 shows, an ideal situation would be to have each secondary
key point to a different list of primary key references. Each of these lists
could grow to be just as long as it needs to be. If we add the new Prokofiev
record, the
PROKOFIEV
collects together a
list
of Prokofiev references becomes
ANG36193
LON2312
245
IMPROVING THE SECONDARY INDEX STRUCTURE: INVERTED LISTS
of primary
key references
Lists
Secondary key index
BEETHOVEN
ANG3795
COREA
DG139201
DVORAK
DG18807
PROKOFIEV
RCA2626
WAR23699
COL31809
LON2312
FIGURE 6.10 Conceptual view of the primary key reference fields as a series of
Similarly,
adding two
additional elements to the
new Beethoven
list
recordings
lists.
adds just two
of references associated with the Beethoven
key. Unlike our record structure that allocates
enough space
for four Label
IDs for each secondary key, the lists could contain hundreds of references,
if needed, while still requiring only one instance of a secondary key. On the
other hand,
if a list requires
internal fragmentation.
the
file
only one element, then no space
Most important of
all,
we need
can
we
set
if a new composer is added to the
up an unbounded number of different
varying length, without creating a large number of small
way
is
through the use of linked lists.
it consists of records with two
index so
a field containing the relative record
We
lost to
to rearrange only
of secondary keys
How
is
files?
file.
lists,
each of
The
simplest
could redefine our secondary
fields
number of
secondary key
the
first
field,
and
corresponding
246
INDEXING
primary key reference (Label ID) in the inverted list. The actual primary key
references associated with each secondary key would be stored in a separate,
entry-sequenced file.
Given the sample data we have been working with, this new design
would result in a secondary key file for composers and an associated Label
ID file that are organized as illustrated in Fig. 6.11. Following the links for
the list of references associated with Beethoven helps us see how the Label
ID List file is organized. We begin, of course, by searching the secondary
key index of composers for Beethoven. The record that we find points us
number (RRN) 3 in the Label ID List file. Since this is a
it is easy to jump to RRN 3 and read in its Label ID
to relative record
fixed-length
file,
(ANG3795). Associated with this Label ID is a link to a record with RRN
8. We read in the Label ID for that record, adding it to our list (ANG379
DG139201). We continue following links and collecting Label IDs until the
list
looks like
this:
ANG3795
The
link field in the last record read
of 1. As
a value
DG18807
DG139201
in
our
earlier
RCA2626
from the Label ID
programs,
List file contains
this indicates end-of-list, so
know that we now have all the Label ID
To illustrate how record addition affects the Secondary Index
ID List files, we add the Prokofiev recording mentioned earlier:
we
references for Beethoven records.
36193
ANG
You
last
one
record
Piano Concertos
and
Prokofiev
and Label
Francois
can see (Fig. 6.11) that the Label ID for this new recording is the
ID List file, since this file is entry sequenced. Before this
in the Label
is
added, there
is
only one Prokofiev recording.
It
has a Label
ID of
we want to keep the Label ID Lists in order by ASCII
character values, the new recording is inserted in the list for Prokofiev so it
logically precedes the LON2312 recording.
Associating the Secondary Index file with a new file containing linked
LON2312.
lists
Since
of references provides some advantages over any of the structures
considered up to this point:
The only time we need to rearrange the Secondary Index file is when
a new composer's name is added or an existing composer's name is
it was misspelled on input). Deleting or adding recomposer who is already in the index involves changing only the Label ID List file. Deleting all the recordings for a composer could be handled by modifying the Label ID List file, while
changed
(e.g.,
cordings for
leaving the entry in the Secondary Index
file
in place, using a value
247
IMPROVING THE SECONDARY INDEX STRUCTURE: INVERTED LISTS
Improved
revision of the
composer index
Secondary Index file
Label
BEETHOVEN
ID
LON2312
COREA
RCA2626
DVORAK
WAR23699
PROKOFIEV
ANG3795
RIMSKY-KORSAKOV
COL38358
SPRINGSTEEN
DG18807
SWEET HONEY IN THE R
MER75016
COL31809
DG139201
FF245
ANG36193
10
FIGURE 6.1
Secondary key index referencing linked
of
primary key references.
the task
is
list
of entries for
this
empty.
is
In the event that
is
lists of
in its reference field to indicate that the
composer
List file
we do need
now since
quicker
to rearrange the
Secondary Index
file,
there are fewer records and each record
smaller.
Since there
is
less
need for sorting,
it
follows that there
penalty associated with keeping the Secondary Index
ondary storage, leaving more room
in
RAM
less
is
files
off
of
on
sec-
for other data struc-
tures.
[j
The Label ID
J
^
needs to be sorted.
List file
is
entry sequenced. That means that
it
never
ID List file is a fixed-length record file, it would be
implement a mechanism for reusing the space from de-
Since the Label
very easy to
leted records, as described in
Chapter
5.
*^T
7&
248
INDEXING
There
also at least
is
one potentially
significant disadvantage to this kind
of file organization: The Label IDs associated with
given composer are no
longer guaranteed to be physically grouped together. The technical term for
such "togetherness"
such
as this,
it
is
is
with
locality;
a linked,
less likely that there will
entry-sequenced structure
be locality associated with the
groupings of reference fields for a given secondary key. Note, for
list of Label IDs for Prokofiev consists of the very last and
the very first records in the file. This lack of locality means that picking up
the references for a composer that has a long list of references could involve
a large amount of seeking back and forth on the disk. Note that this kind of
logical
example, that our
seeking would not be required for our original Secondary Index
file
structure.
One obvious
List file in
antidote to this seeking problem
memory.
is
to
keep the Label ID
This could be expensive and impractical, given
many
secondary indexes, except for the interesting possibility of using the same
Label ID List file to hold the lists for a number of Secondary Index files.
Even if the file of reference lists were too large to hold in memory, it might
be possible to obtain
the
file
in
memory
as
memory
performance improvement by holding only a part of
paging sections of the file in and out of
at a time,
they are needed.
Several exercises
at
more thoroughly. These
the end of the chapter explore these possibilities
are very important problems, since the notion
of
fundamental to the design of B-trees and
other methods for handling large indexes on secondary storage.
dividing the index into pages
6.8
is
Selective Indexes
Another interesting feature of secondary indexes
divide a
file
providing
into parts,
is
that they can be used to
selective view.
For example,
possible to build a selective index that contains only the
titles
of
it
is
classical
recordings in the record collection. If we have additional information about
file, such as the date the recording was released,
could build selective indexes such as "recordings released prior to 1970"
and "recordings since 1970." Such selective index information could be
the recordings in the data
we
combined
"List
all
into
Boolean
AND
operations to respond to requests such
1970." Selective indexes are sometimes useful
as,
Symphony released since
when the contents of a file fall
the recordings of Beethoven's Ninth
naturally and logically into several broad categories.
BINDING
6.9
249
Binding
recurrent and very important question that emerges in the design of
systems that
utilize
physical address of
its
indexes
is:
At what point
in
time
is
the hey
bound
file
to the
associated record?
system we are designing in the course of this chapter, the
primary keys to an address takes place at the time the files are
The secondary keys, on the other hand, are bound to an address
In the file
binding of our
constructed.
at the time that they are actually used.
Binding at the time of the file construction results in faster access. Once
you have found the right index record, you have in hand theT>yte offset of
the data record you are seeking. If we elected to bind our secondary keys to
their associated records at the time of file construction, so when we find the
record in the composer index we would know immediately that
the data record begins at byte 353 in the data file, secondary key retrieval
would be simpler and faster. The improvement in performance is
particularly noticeable if both the primary and secondary index files are used
on secondary storage rather than in memory. Given the arrangement we
designed, we would have to perform a binary search of the composer index
and then a binary search of the primary key index before being able to jump
to the data record. Binding early, at file construction time, does away
at*?\Mflfi|
entirely with the need to search on the primary key.
The disadvantage of binding directly in the tile, of binding tightly, is that
reorganizations of the data file must result in modifications to all bound
DVORAK
index
files. This reorganization cost can be very expensive, particularly
with simple index files in which modification would often mean shifting
records. By postponing binding until execution time, when the records are
actually being used, we are able to develop a secondary key system that
involves a minimal amount of reorganization when records are added or
deleted.
Another important advantage of postponing binding until a record is
actually retrieved is that this approach is safer. As we see in the system that
we set up, associating the secondary keys with reference fields consisting of
primary keys allows the primary key index to act as a kind of final check of
whether a record is really in the file. The secondary indexes can afford to be
wrong. This situation is very different if the secondary index keys are
bound, containing addresses. We would then be jumping directly
from the secondary key into the data file; the address would need to be
tightly
right.
250
INDEXING
This brings up a related safety aspect: It is always more desirable to have
make important changes in one place, rather than having to make them
in many places. With a bind-at-retrieval-time scheme such as we developed,
we need to remember to make a change in only one place, the primary key
index, if we move a data record. With a more tightly bound system, we
have to make many changes successfully to keep the system internally
consistent, braving power failures, user interruptions, and so on.
When designing a new file system, it is better to deal with this question
to
of binding intentionally and early
binding just happen. In general,
in the design process, rather than letting the
tight, in-the-data
binding
is
most
attractive
when
The
data
file is static
or nearly so, requiring
and updating of records; and
Rapid performance during actual retrieval
little
or no adding, de-
leting,
is
high priority.
For example, tight binding is desirable for file organization on a massproduced, read-only optical disk. The addresses will never change since no
new records can ever be added; consequently, there is no reason not to
obtain the extra performance associated with tight binding.
For
file
applications in
which record
addition, deletion, and updating
occur, however, binding at retrieval time
is
option. Postponing binding for as long as possible
operations simpler and safer. If the
and, in particular, if the indexes use
as B-trees, retrieval
additional
work
performance
is
file
structures are carefully designed,
more
sophisticated organizations such
usually quite acceptable, even given the
required by a bind-at-retrieval system.
SUMMARY
We
began
this
sorting as a
sorting,
chapter with the assertion that indexing
way of structuring
indexing
permits
variable-length record
addition, deletion,
and
files.
perform binary
to
If the
file
an alternative to
searches
for
keys
in
index can be held in memory, record
retrieval can
indexed, entry-sequenced
be done
much more
than with a sorted
quickly with an
file.
much more than merely improve on access time: They
new capabilities that are inconceivable with access
on sorted data records. The most exciting new capability
Indexes can do
can provide us with
methods based
us
is
so records can be found by key. Unlike
a file
do
more desirable
usually makes these
usually the
SUMMARY
involves the use of multiple secondary indexes. Just as
allows us to regard
a collection
subject order, so index
records in a data
file.
files
We
lists
in
a library
author order,
card catalog
title
order, or
allow us to maintain different views of the
find that
obtain different views of the
associated
of books
we
file,
cannot only use secondary indexes to
we
but that
can also combine the
of primary key references and thereby combine particular
views.
In this chapter
indexes of
two
The need
The need
added
we
address the problem of
how
to rid our secondary
liabilities:
to repeat duplicate secondary keys;
and
to rearrange the secondary indexes every time a record
to the data
is
file.
A first solution to these problems involves associating a fixed-size vector
with each secondary key. This solution results in an
internal fragmentation but serves to illustrate the
attractiveness of handling the reference fields associated with a particular
secondary key as a group, or list.
Our next iteration of solutions to our secondary index problems is
more successful and much more interesting. We can treat the primary key
of reference
fields
overly large
amount of
file, forming the necessary lists
through the use of link fields associated with each primary record entry. This
allows us to create a secondary index file that, in the case of the composer
index, needs rearrangement only when we add new composers to the data
file. The entry-sequenced file of linked reference lists never requires sorting.
We call this kind of secondary index structure an inverted list.
There are also, of course, disadvantages associated with our new
solution. The most serious disadvantage is that our file demonstrates less
references themselves as an entry-sequenced
of associated records are less likely to be physically adjacent.
A good antidote to this problem is to hold the file of linked lists in memory.
We note that this is made more plausible because a single file of primary
references can link the lists for a number of secondary indexes.
As indicated by the length and breadth of our consideration of
locality: Lists
secondary indexing, multiple keys, and inverted
among
most
the
lists,
interesting aspects of indexed access to
these topics are
files.
The concepts
of secondary indexes and inverted lists become even more powerful later, as
we develop index structures that are themselves more powerful than the
simple indexes that we consider here. But, even so, we already see that for
small files consisting of no more than a few thousand records, approaches
to inverted lists that rely merely on simple indexes can provide a user with
a great deal
of capability and
flexibility.
251
252
INDEXING
KEY TERMS
Binding. Binding takes place when
physical record in the data
key
is
associated with a particular
In general, binding can take place
file.
either during the preparation
program execution.
of the data file and indexes or during
former case, which is called tight binding,
In the
the indexes contain explicit references to the associated physical data
record. In the latter case, the connection
ular physical record
in the course
is
postponed
file.
in
file
that they are entered into the
An
index
a
is
key and
a partic-
actually retrieved
of program execution.
Entry-sequenced
Index.
between
until the record
is
a tool for
which the records occur
in the order
file.
finding records in
a file. It consists
of
key
on which the index is searched and a reference field that tells
where to find the data file record associated with a particular key.
Inverted list. The term inverted list refers to indexes in which a key may
be associated with a list of reference fields pointing to documents that
contain the key. The secondary indexes developed toward the end of
field
this
Key
chapter are examples of inverted
field.
The key
the canonical
field is the
lists.
portion of an index record that contains
form of the key
that
is
being sought.
.when records that will be accessed
given temporal sequence are found in physical proximity to each
Locality. Locality exists in
a file
in a
other on the disk. Increased locality usually results in better perfor-
mance, since records that are in the same physical area can often be
brought into memory with a single read request to the disk.
Reference field. The reference field is the portion of an index record
that contains information about
where
to find the data record con-
taining the information listed in the associated key field of the index.
Selective index.
selective index contains keys for only a portion
the records in the data
view of
file.
a specific subset
Simple index.
Such an index provides the user with
of the
file's
of
a
records.
All the index structures discussed in this chapter are sim-
ple indexes insofar as they are
all
built
around the idea of an ordered,
sequence of index records. All these simple indexes share a
common weakness: Adding records to the index is expensive. As we
see later, tree-structured indexes provide an alternate, more efficient
linear
solution to this problem.
EXERCISES
EXERCISES
1.
Until now,
it
was not possible
variable-length record
With
Does
file.
fixed-length record
mean
this
perform
to
binary search on
Why does indexing make binary search possible?
is possible to perform a binary search.
need not be used with fixed-length record
file it
that indexing
files?
2.
Why
is title
chapter? If
it
not used
were used
primary key in the data file described in this
secondary key, what problems would have to
as a
as a
be considered in deciding on
3.
What
is
canonical form for
titles?
the purpose of keeping an out-of-date-status flag in the header
record of an index? In a multiprogramming environment, this flag might be
found to be set by one program because another program is in the process
of reorganizing the index. How should the first program respond to this
situation?
4.
Explain
how
5.
When
record in
the use of an index pins the data records in a
secondary key indexes
is
updated, corresponding primary and
may
not have to be altered, depending on
a data file
may
or
file.
whether the file has fixed- or variable-length records, and depending on the
type of change made to the data record. Make a list of the different updating
situations that can occur, and explain how each affects the indexes.
when you add the following recording
assuming that the composer index shown in Fig. 6.9
is used. How might you solve the problem without substantially changing
the secondary key index structure?
6.
Discuss the problem that occurs
to the recordings
LON
1259
file,
Fidelio
Beethoven
and when
7.
What
is
8.
How
are the structures in Fig. 6.11
an inverted
list,
is it
Maazel
useful?
changed by the addition of the
recording
LON
1259
Fidelio
Beethoven
Maazel
Suppose you have the data file described in this chapter, greatly
a primary key index and secondary key indexes organized
by composer, artist, and title. Suppose that an inverted list structure is used
to organize the secondary key indexes. Give step-by-step descriptions of
how a program might answer the following queries:
a. List all recordings of Bach or Beethoven; and
b. List all recordings by Perleman of pieces by Mozart or Joplin.
9.
expanded, with
253
254
10.
INDEXING
One
inverted
problem of diminished locality when using
same Label ID List file to hold the lists for several
possible antidote to the
lists is
to use the
of the secondary index files. This increases the likelihood that the secondary
indexes can be kept in primary memory. Draw a diagram of a single Label
ID List file that can be used to hold references for both the secondary index
of composers and the secondary index of titles. How would you handle the
difficulties that this arrangement presents with regard to maintaining the
Label ID List
file?
11. Discuss the following structures as antidotes to the possible loss
locality in a
of
secondary key index:
Leave space for multiple references for each secondary key (Fig. 6.9).
Allocate variable-length records for each secondary key value, where
each record contains the secondary key value, followed by the Label
IDs, followed by free space for later addition of new Label IDs. The
amount of free space left could be fixed, or it could be a function of
the size of the original
12.
file
list
of Label IDs.
The method and timing of binding
system
speed and
flexibility.
two important
affect
attributes
of a
Discuss the relevance of these attributes,
and the effect of binding time on them, for a hospital patient information
system designed to provide information about current patients by patient
name, patient ID, location, medication, doctor or doctors, and illness.
Programming and Design Exercises
13.
Implement the
retrieve_record( )
14. In solving the preceding
deciding
how many
procedure outlined in
mechanism for
each record. At least
problem, you have to create
bytes to read from the Datafile for
Fig. 6.4.
a
four options are open to you:
a.
Jump
to the
byte_ofifiset,
read the size
field,
then use this informa-
tion to read the record.
b.
Build an index
file
that contains a record size field that reflects the
true size
of the data record, including the
Datafile.
Use
many
c.
size field carried in the
the size field carried in the index
file
to decide
how
bytes to read.
Follow much the same strategy
as in
option
(b),
except use a
Datafile that does not contain internal size fields.
d.
Jump
to the byte_offset
bytes (e.g., 512 bytes).
and read
Once
buffer, use the size field at the start
many
a fixed,
overly large
these bytes are read into a
number of
memory
how
of the buffer to decide
bytes to break out of the buffer.
EXERCISES
Evaluate each of these options, listing the advantages and disadvantages of
each.
Implement procedures
15.
to the
index
When
16.
some of
first
and to write back the
INDEXf
array
searching secondary indexes that contain multiple records for
we do
we want to
not want to find just any record for
the keys,
secondary key;
the
to read in
file.
given
find the first record containing that key. Finding
record allows us to read ahead, sequentially, extracting
all
of the
records for the given key. Write a variation of a binary search function that
returns the relative record
key.
The function should
17. If a Label
be held in
ID
List file
memory
in
the
first
record containing the given
return a negative value
key cannot be found.
as the
entirety,
if the
one shown in Fig. 6.11 is too large to
might still be possible to improve
number of blocks of the file in memory. These
such
its
performance by retaining
number of
it
blocks are called pages. Since the records in the Label ID List
file
are each 16
bytes long, a page might consist of 32 records (512 bytes). Write
function
would hold the most recently used eight pages in memory. Calls for a
specific record from the Label ID List file would be routed through this
function. It would check to see if the record exists in one of the pages that
is already in memory. If so, the function would return the values of the
record fields immediately. If not, the function would read in the page
containing the desired record, either writing out or dumping the page that
was used least recently. Clearly, if a page has been changed, it needs to be
written out rather than dumped. When the program is over, all pages still
in memory must be checked to see if they should be written out.
that
18. Assuming the use of a paged index
problem, and given that the Label ID List
any particular order of data entry (initial
as
described in the preceding
file is
file
entry sequenced,
is
there
loading) that tends to give
performance than other methods? How does the use of an
organization method such as that described in problem 10, which combines
the linked lists from several secondary indexes into a single file, affect your
answer about performance?
better
19.
The Label ID
schemes
is
List file
is
entry sequenced. Development of paging
simpler for entry-sequenced
files
than for
files
that are kept in
sorted order. List the additional difficulties involved in the design of a
paging system for
a sorted index,
such
the possibility that there will be a
full,
design such a paging system.
new key when
the page in
which
primary key index. Accepting
that are only partially
will you handle the insertion of a
as the
number of pages
How
it
belongs
is
full?
255
256
INDEXING
FURTHER READINGS
Wc
have
much more
to say about indexing in later chapters,
subjects of tree-structured indexes and of indexed sequential
where we take up
file
organizations.
the
The
topics developed in the current chapter, particularly those relating to secondary
indexes and inverted
texts.
The few
files,
texts that
are also covered
we
list
by many other
file
and data structure
here are of interest because they either develop
more detail or present the material from a different viewpoint.
Wiederhold (1983) provides a survey of many of the index structures we
discuss, along with a number of others. His treatment is more mathematical than
that provided in our text. Users interested in looking at indexed files in the context
of PL/I and of large IBM mainframes will want to see Bradley (1982). A brief,
readable overview of a number of different file organizations is provided in J. D.
certain topics in
Ullman (1980).
Tremblay and Sorenson
(1984) provide a comparison of inverted
provides
users.
a similar discussion,
M.
list
structures
Loomis (1983)
along with some examples oriented toward COBOL
with an alternative organization called
multilist
files.
Salton and McGill (1983) discuss inverted
application in information retrieval systems.
lists
in
E.
S.
the context of their
Cosequential Processing
and the Sorting of
Large Files
CHAPTER OBJECTIVES
Describe
tivities
a class
known
of frequently used processing ac-
as cosequential processes.
Provide a general model for implementing
eties of cosequential processes.
Illustrate the use
of the model to solve
all
vari-
number of
different kinds of cosequential processing prob-
lems, including problems other than simple merges
and matches.
I
Introduce heapsort as an approach to overlapping
I/O with sorting
in
Show how merging
very large
RAM.
provides the basis for sorting
files.
Examine the costs of K-way merges on
find ways to reduce those costs.
Introduce the notion of replacement
disk and
selection.
Examine some of the fundamental concerns
ated with sorting large
files
associ-
using tapes rather than
disks.
Introduce
UNIX
utilities for sorting,
merging, and
cosequential processing.
257
^^
CHAPTER OUTLINE
7.1
Model
for
Implementing
7.5.3
Cosequential Processes
Matching Names in Two Lists
7.1.2 Merging Two Lists
7.1.3 Summary of the Cosequential
7.5.4
Hardware-based Improvements
7.5.5
Decreasing the Number of
Seeks Using Multiple-step
Merges
Model
7.5.6
7.2
Application of the Model to a
General Ledger Program
7.2.1
7.5.7
Model
to the
7.5.8
Using Two Disk Drives with
Replacement Selection
7.5.9
More
Ledger Program
Extension of the Model to Include
7.5.10 Effects of
A K-way Merge
A Selective Tree
Algorithm
7.3.2
for Merging
Large Numbers of Lists
7.4
Second Look
7.4.1
at
Sorting in
7.5.11
Reading
More
Multiprogramming
Conceptual Toolkit for
External Sorting
7.6
RAM
Sorting Files on Tape
7.6.1
7.6.2
Overlapping Processing and
I/O: Heapsort
7.4.2 Building the
Drives?
Processors?
Multiway Merging
7.3.1
Run Lengths Using
Replacement Selection
Replacement Selection Plus
Increasing
Multistep Merging
The Problem
7.2.2 Application of the
7.3
the File
Size
7.1.1
Processing
The Cost of Increasing
The Balanced Merge
The K-way Balanced Merge
7.6.3 Multiphase
Merges
7.6.4 Tapes versus Disks for External
Heap while
Sorting
in the File
7.4.3 Sorting while Writing out to the
7.7
Sort-Merge Packages
7.8
Sorting and Cosequential
Processing in UNIX
File
7.5
Merging as a Way of Sorting Large
Files on Disk
7.5.1
How Much
Time Does
7.8.1
Merge
Sort Take?
7.5.2 Sorting a File That
Is
Sorting and Merging in
UNIX
7.8.2 Cosequential Processing
Utilities in
Ten
UNIX
Times Larger
Cosequential operations involve
the
coordinated processing of two or more
Sometimes the processing results
of the input lists; sometimes the goal is a matching, or
intersection, of the lists; and other times the operation is a combination of
matching and merging. These kinds of operations on sequential lists are the
basis of a great deal of file processing.
In the first half of this chapter we develop a general model for doing
cosequential operations, illustrate its use for simple matching and merging
sequential
lists to
produce a single output
in a merging, or union,
list.
259
A MODEL FOR IMPLEMENTING COSEQUENTIAL PROCESSES
it to the development of a more complex general
Next we apply the model to multiway merging, which is
component of external sort-merge operations. We conclude the
operations, and then apply
ledger program.
an essential
chapter with
trade-offs,
7.1
a discussion of external sort-merge procedures, strategies, and
paying special attention to performance considerations.
A Model
Implementing Cosequential Processes
for
Cosequential operations usually appear to be simple to construct; given the
information that
we
can be turned into
provide in
reality.
this chapter, this
However,
it
is
appearance of simplicity
also true that approaches to
cosequential processing are often confused, poorly organized, and incor-
These examples of bad practice are by no means limited to student
the problems also arise in commercial programs and in
textbooks. The difficulty with these incorrect programs is usually that they
are not organized around a single, clear model for cosequential processing.
Instead, they seem to deal with the various exception conditions and
problems in a cosequential process in an ad hoc rather than systematic way.
This section addresses such lack of organization head on. We present a
single, simple model that can be the basis for the construction of any kind
of cosequential process. By understanding and adhering to the design
principles inherent in the model, you will be able to write cosequential
procedures that are simple, short, and robust.
rect.
programs;
7.1.1 Matching
Names
Suppose
we want
Fig. 7.1.
This operation
We
a
in
Two
to output the
is
names common
and that the
lists
to the
two
lists
shown
in
usually called a match operation, or an intersection.
assume, for the moment, that
list,
Lists
we
will not allow duplicate keys within
are sorted in ascending order.
We begin by reading in the initial name from each list, and we find that
We output this first name as a member of the match set, or
intersection set. We then read in the next name from each list. This time the
they match.
name
lists
in List 2
is less
visually, as
we
than the
are
When we are processing these
we are trying to match the
and scan down List 2 until we either find it or
we eventually find a match for CARTER, so
name
in List
1.
now, we remember
that
name CARTER from List 1,
jump beyond it. In this case,
we output it, read in the next name from each list, and continue the process.
Eventually we come to the end of one of the lists. Since we are looking for
names common to both lists, we know we can stop at this point.
Although the match procedure appears
number of
well.
to be quite simple, there are a
matters that have to be dealt with to
make
it
work reasonably
260
List
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
List 2
ADAMS
ANDERSON
ANDREWS
BECH
BURNS
CARTER
DAVIS
DEMPSEY
GRAY
JAMES
JOHNSON
KATZ
PETERS
ROSEWALD
SCHMIDT
THAYER
WALKER
WILLIS
ADAMS
CARTER
CHIN
DAVIS
FOSTER
GARWICK
JAMES
JOHNSON
KARNS
LAMBERT
MILLER
PETERS
RESTON
ROSEWALD
TURNER
FIGURE 7.1 Sample input
lists for
cosequential operations.
Initializing: We need to arrange things in such
dure gets going properly.
Synchronizing:
list is
never so
that the proce-
have to make sure that the current name from one
far
ahead of the current name on the other
2,
end-of-file conditions:
When we
or
we need
program.
errors:
to halt the
When
list
that a
means reading the next name
sometimes from both lists.
this
Handling
Recognizing
way
We
match will be missed. Sometimes
from List 1, sometimes from List
file 2,
get to the end of either
an error occurs in the data
names or names out of sequence) we want
to detect
(e.g.,
it
file 1
duplicate
and take some
action.
Finally, we would like our algorithm to be reasonably efficient, simple,
and easy to alter to accommodate different kinds of data. The key to
accomplishing these objectives in the model we are about to present lies in
synchronization.
the way we deal with the second item in our list
At each step in the processing of the two lists, we can assume that we
have two names to compare: a current name from List 1 and a current name
from List 2. Let's call these two current names NAME_1 and NAME_2.
We can compare the two names to determine whether NAME_1 is less
than, equal to, or greater than
NAME_2:
261
A MODEL FOR IMPLEMENTING COSEQUENTIAL PROCESSES
If
NAME_1
List
If
NAME_1
List 2;
than
NAME_2, we
name from
read the next
is
greater than
NAME_2, we
read the next
we
name and
name from
and
names
names from
If the
It
is less
1;
are the same,
the
two
output the
read the next
lists.
turns out that this can be handled very cleanly with a single loop
one three-way conditional statement, as illustrated in the
7.2. The key feature of this algorithm is that control always
the head of the main loop after every step of the operation. This means
containing
algorithm in Fig.
returns to
that
1
no extra
gets ahead
logic
of
is
required within the loop to handle the case
List 2, or List 2 gets
ahead of List
1,
PROGRAM: match
call initialize!) procedure to:
- open input files LIST_1 and LIST_2
- create output file 0UT_FILE
- set MORE_NAMES_EXIST to TRUE
- initialize sequence checking variables
to get NAME_1 from LIST_1
to get NAME_2 from LIST_2
while (MORE_NAMES_EXIST)
if (NAME_1 < NAME_2)
call input () to get NAME_1 from LIST_1
else if (NAME_1 > NAME_2
call input () to get NAME_2 from LIST_2
/* match
names are the same */
write NAME_1 to 0UT_FILE
call input () to get NAME_1 from LIST_1
call input () to get NAME_2 from LIST_2
endif
endwhile
finish_up(
else
end PROGRAM
List
or the end-of-file
FIGURE 7.2 Cosequential match procedure based on a single loop.
call input ()
call input()
when
262
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
is reached on one list before it is on the other. Since each pass
through the main loop looks at the next pair of names, the fact that one
list may be longer than the other does not require any special logic. Nor
the while statement simply checks the
does the end-of-file condition
MORE_NAMES_EXIST flag on every cycle.
condition
The
logic inside
the loop
conditions can exist after reading
them. Since
when
the
Note
as
we
are
names
implementing
is
a
a
equally simple.
name; the if
match process
'
Only
else
three
possible
logic handles
all
of
here, output occurs only
are the same.
main program does not concern itself with such matters
sequence checking and end-of-file detection. Since their presence in the
that the
main loop would only obscure the main synchronization logic, they have
been relegated to subprocedures.
Since the end-of-file condition is detected during input, the setting of
the MORE_NAMES_EXIST flag is done in the inputf ) procedure. The
input( ) procedure can also be used to check the condition that the lists be in
strictly
ascending order (no duplicate entries within
Fig. 7.3 illustrates
one method of handling these
list).
tasks.
The algorithm
in
This "filling out"
FIGURE 7.3 Input routine for match procedure.
PROCEDURE:
input
()
/* input routine for MATCH procedure */
input arguments:
INP_FILE
PREVI0US_NAME
file descriptor for input file to be used
(could be LIST_1 OR LIST_2)
last name read from this list
arguments used to return values:
name to be returned from input procedure
NAME
flag used by main loop to halt processing
MORE_NAMES_EXIST
:
read next NAME from INP_FILE
/* check for end of file,
if (EOF)
M0RE_NAMES_EXIST
:=
duplicate names, names out of order
FALSE
else if (NAME <= PREVIOUS_NAME)
issue sequence check error
abort processing
endif
PREVIOUS_NAME
end PROCEDURE
:=
NAME
/* set flag to end processing
*/
*/
263
A MODEL FOR IMPLEMENTING COSEQUENTIAL PROCESSES
PROCEDURE:
initialize(
arguments used to return values:
PREV_1, PREV_2
previous name variables for the 2 lists
LIST_1, LIST_2
file descriptors for input files to be used
MORE NAMES EXIST
flag used by main loop to halt processing
/* set both the previous_name variables (one for each list) to
a value that is guaranteed to be less than any input value */
PREV_1
PREV_2
L0W_VALUE
L0W_VALUE
:=
open file for List
open file for List
as LIST_1
as LIST_2
if (both open statements succeed)
MORE_NAMES_EXIST
:=
TRUE
end PROCEDURE
FIGURE 7.4 Initialization procedure for cosequential processing.
of the
input()
would
use.
All
procedure also indicates the arguments that the procedure
we need now
procedure
initialize(
that
)
to
complete the logic
begins
procedure,
the
shown
is
a description
in Fig. 7.4,
performs three
1.
It
opens the input and output
2.
It
sets the
MORE_NAMES_EXIST
3.
It
sets the
previous_name variables (one for each
guaranteed to be
PREV_1
and
less
to
special
The
two
lists
tasks:
LOW_VALUE
is
to a value that
list)
is
of setting
that the procedure ineffect
first
two records
in
way.
Given these program fragments, you should be able
the
The
TRUE.
flag to
put() does not need to treat the reading of the
any
initialize(
files.
than any input value.
PREV_2
of the
main cosequential match procedure.
provided in
Fig.
7.1,
following
to
work through
the pseudocode,
and
demonstrate to yourself that these simple procedures can handle the various
resynchronization problems that these sample lists present.
7.1.2 Merging Two Lists
The three-way
test,
single-loop
model
for cosequential processing can
easily be modified to handle merging of
lists
as
well as matching,
as
264
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
PROGRAM: merge
call initialize!) procedure to:
- open input files LIST_1 and LIST_2
- create output file 0UT_FILE
- set MORE_NAMES_EXIST to TRUE
- initialize sequence checking variables
call input
call input
()
()
to get NAME_1 from LIST_1
to get NAME_2 from LIST_2
while (MORE_NAMES_EXIST)
< NAME_2)
write NAME_1 to 0UT_FILE
call input () to get NAME_1 from LIST_1
if (NAME_1
else if (NAME_1 > NAME_2
write NAME_2 to 0UT_FILE
call input () to get NAME_2 from LIST_2
/* match
names are the same */
write NAME_1 to 0UT_FILE
call input () to get NAME_1 from LIST_1
call input (J to get NAME_2 from LIST_2
endif
endwhile
finish_up(
else
end PROGRAM
FIGURE 7.5 Cosequential merge procedure based on a single loop.
illustrated in Fig. 7.5.
the if
An
else
Note
that
we now produce
list contents.
merge is
between matching and merging is that with
construction since
important difference
output for every case of
a union of the
merging we must read completely through each of the lists. This necessitates a change in our input( ) procedure, since the version used for matching sets the MORE_NAMES_EXIST flag to FALSE as soon as we
detect end-of-file for one of the lists. We need to keep this flag set to
TRUE as long as there are records in either list. At the same time, we must
recognize that one of the lists has been read completely, and we should
avoid trying to read from it again. Both of these goals can be achieved if
265
A MODEL FOR IMPLEMENTING COSEQUENTIAL PROCESSES
we simply
NAME
set the
variable for the completed
list
to
some
value
that
Cannot possibly occur as a legal input value; and
Has a higher collating sequence value than any possible legal input
value. In other words, this special value would come after all legal
put values in the
We
Fig. 7.6
files
file's
refer to this special value as
HIGH_VALUE
shows how
in-
ordered sequence.
are read to completion.
OTHER_LIST_NAME
to
whether the other input
list
HIGH_VALUE. The
pseudocode
in
can be used to ensure that both input
Note that we have
argument list so
the
has reached
its
to
add the argument
the
function
knows
end.
FIGURE 7.6 Input routine for merge procedure.
PROCEDURE:
input
/* input routine for MERGE procedure */
()
input arguments
INP_FILE
PREVIOUS_NAME
OTHER_LIST_NAME
file descriptor for input file to be used
(could be LIST_1 OR LIST_2)
last name read from this list
most recent name read from the other list
arguments used to return values:
name to be returned from input procedure
NAME
flag used by main loop to halt processing
MORE_NAMES_EXIST
:
read next NAME from INP_FILE
and OTHER_LIST_NAME == HIGH_VALUE)
/* end of both lists
MORE_NAMES_EXIST := FALSE
if (EOF)
else if (EOF)
NAME := HIGH_VALUE
else if (NAME <= PREVIOUS_NAME)
issue sequence check error
abort processing
endif
PREVIOUS_NAME
end PROCEDURE
NAME
*/
/* just this list ended */
/* sequence check
*/
266
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Once
again, you should use this logic to work, step by step, through
provided in Fig. 7.1 to see how the resynchronization is handled
and how the use of the HIGH_VALUE forces the procedure to finish both
the
lists
lists
before terminating. Note that the version of input(
HIGH_VALUE
incorporating the
logic can also be used for matching procedures, producing
The only disadvantage to doing so is that the matching
procedure would no longer terminate as soon as one list is completely
processed, but would go through the extra work of reading all the way
through the unmatched entries at the end of the other list.
correct results.
With these two examples, we have covered all of the
Now let us summarize the model before adapting
complex problem.
model.
Summary
7.1.3
of the Cosequential Processing
pieces of our
to a
it
more
Model
Generally speaking, the model can be applied to problems that involve the
performance of set operations (union, intersection, and more complex
on two or more sorted input files to produce one or more output
processes)
files.
In this
summary of the
we assume
cosequential processing model,
that
two input files and one output file. It is important to
understand that the model makes certain general assumptions about the
there are only
nature of the data and type of problem to be solved. Here
is
a list
of the
assumptions, together with clarifying comments.
Assumptions
Comments
Two
In
or more input files are to be
processed in a parallel fashion to produce one or more output files.
Each
fields,
file is
sorted
and
all files
on one or more key
It is
are ordered in the
the
same ways on the same
In
some
cases, there
key value
that
is
some cases an output file may be
same file as one of the input files.
the
not necessary that
same record
all files
have
structures.
fields.
must
exist a
high
greater than any le-
gitimate record key, and a low key
value that is less than any legitimate
record key.
The use of a high key value and a low
key value is not absolutely necessary,
but can help avoid the need to deal
with beginning-of-file and end-of-file
conditions as special cases, hence decreasing complexity.
Records are to be processed
sorted order.
in logical
The
physical ordering of records
is ir-
relevant to the model, but in practice
be very important to the way
is implemented. Physical
ordering can have a large impact on
it
may
the
model
processing efficiency.
267
A MODEL FOR IMPLEMENTING COSEQUENTIAL PROCESSES
Assumptions
Comments
For each file there is only one current
record. This is the record whose key
is accessible within the main synchro-
The model does not
prohibit looking
ahead or looking back at records, but
such operations should be restricted to
subprocedures and should not be allowed to affect the structure of the
main synchronization loop.
nization loop.
Records can be manipulated only
internal
in
memory.
program cannot alter a record
on secondary storage.
in
place
Given these assumptions, here
are the essential
components of the
model.
1.
cal
records in the respective
low
set to the
2.
One main
long
3.
Current records for
Initialization.
all files
are read
the
logi-
first
all files
are
synchronization loop
as relevant records
is
used, and the loop continues as
remain.
Within the body of the main synchronization loop is a selection
based on comparison of the record keys from respective input file
such
as
if
cur r ent_f
i 1
two input
e1_k ey
cur rent_f
else if
else
/*
end
the selection takes a
files,
cur r ent_f
>
e1_key
<
i 1
cur r ent_f
i 1
e2_k ey ) then
current keys equal
Input
read
form
e2_k ey ) then
files
and output
files
are sequence checked
the previous_key value with the current_key value
in.
After
a successful
by comparing
when
sequence check, previousjkey
current_key to prepare for the next input operation
sponding
5.
from
Previous_key values for
value.
records. If there are
4.
files.
record
is
is
set to
on the corre-
file.
when end-of-file ocwhen high values have
High
values are substituted for actual key values
curs.
The main processing loop
occurred for
all
relevant input
terminates
files.
The
use of high values eliminates
the need to add special code to deal with each end-of-file condition.
(This step is not needed in a pure match procedure, since a match
procedure halts when the first end-of-file condition is encountered.)
268
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
6.
All possible
I/O and error detection
activities are to
be relegated to
subprocesses, so the details of these activities do not obscure the
principal processing logic.
three-way test, single-loop model for creating cosequential
both simple and robust. You will find very few applications
requiring the coordinated sequential processing of two files that cannot be
handled neatly and efficiently with the model. We now look at a problem
This
processes
that
is
is
much more complex
than
simple match or merge, but that
nevertheless lends itself nicely to solution by means of the model.
7.2
Application of the Model to a General Ledger Program
7.2.1 The Problem
Suppose
we
are given the
problem of designing
general ledger
program
The system includes a journal file and a
the month-by-month summaries of the
part of an accounting system.
file.
The
ledger contains
associated with each of the bookkeeping accounts.
as
ledger
values
sample portion of the
FIGURE 7.7 Sample ledger fragment containing checking and expense accounts.
Acct.
no.
Account
title
101
102
Checking account #1
Checking account #2
505
510
515
520
525
530
535
540
545
550
555
560
565
Advertising expense
Auto expenses
Bank charges
Books and publications
Interest expense
Legal expense
Miscellaneous expense
Office expense
Postage and shipping
Rent
Supplies
Travel and entertainment
Utilities
Jan
Feb
Mar
1032.57
543 78
2114.56
3094.17
5219.23
1321.20
25.00
195.40
0.00
27.95
103.50
25.00
307.92
5.00
27.95
255.20
25.00
501.12
5.00
87.40
380 27
12.45
57.50
21.00
500.00
112.00
62.76
84.89
17.87
105.25
27.63
1000.00
167.50
198.12
190.60
23.87
138.37
57.45
1500.00
241.80
307.74
278 48
Apr
269
APPLICATION OF THE MODEL TO A GENERAL LEDGER PROGRAM
Debit/
Acct.
no.
Check no
Date
Description
101
510
101
550
101
505
102
540
101
510
1271
1271
1272
1272
1273
1273
670
670
1274
1274
04/02/86
04/02/86
04/02/86
04/02/86
04/04/86
04/04/86
04/07/86
04/07/86
04/09/86
04/09/86
Auto expense
Tune up and minor repair
Rent
Rent for April
Advertising
Newspaper ad re: new product
Office expense
Printer ribbons (6)
Auto expense
Oil change
credit
78 70
78.70
500.00
500.00
87.50
87.50
32.78
32.78
12.50
12.50
.
FIGURE 7.8 Sample journal entries.
ledger, containing only checking
and expense accounts,
is
illustrated in
Fig. 7.7.
The journal
file
monthly transactions that are ultimately to
Figure 7.8 shows what these journal transactions
contains the
be posted to the ledger file.
look like. Note that the entries in the journal
file
are paired. This
is
because
every check involves both subtracting an amount from the checking
account balance and adding an amount to
at least
one expense account. The
accounting program package needs procedures for creating
interactively,
probably outputting records to the
file as
this
journal
file
checks are keyed in
and then printed.
Once
file is complete for a given month, which means that
of the transactions for that month, the journal must be posted
to the ledger. Posting involves associating each transaction with its account
in the ledger. For example, the printed output produced for accounts 101,
102, 505, and 510 during the posting operation, given the journal entries in
Fig. 7.8, might look like the output illustrated in Fig. 7.9.
How is the posting process implemented? Clearly, it uses the account
it
the journal
contains
number
all
as a key to relate the
journal transactions to the ledger records.
possible solution involves building an index for the ledger, so
we
can
One
work
through the journal transactions, using the account number in each journal
entry to look up the correct ledger record. But this solution involves
seeking back and forth across the ledger file as we work through the
journal. Moreover, this solution does not really address the issue of creating
the output
list,
in
which
collected together. Before
all
we
journal entries for even the
the journal entries relating to an account are
could print out the ledger balances and collect
first
account, 101,
we would have
to
proceed
all
270
101
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Checking Account #1
1271 04/02/86 Auto expense
1272 04/02/86 Rent
1273 04/04/86 Advertising
1274 04/09/86 Auto expense
- 78.70
- 500.00
- 87.50
12.50
Prev.
Prev.
510
5219.23
bal
1321.20
New bal: 4540.53
Checking account #2
670 04/07/86 Office expense
102
505
bal
32.78
New bal: 1288.42
Advertising expense
1273 04/04/86 Newspaper ad re: new product
25.00
Prev. bal:
Auto expenses
1271 04/02/86
1274 04/09/86
87.50
New bal:
112.50
78.70
12.50
Tune up and minor repair
Oil change
501.12
Prev. bal:
New bal:
592.32
FIGURE 7.9 Sample ledger printout showing the effect of posting from the journal.
the
way through
account 101
as
the journal
we
collect
list. Where would we save the transactions
them during this complete pass through
for
the
journal?
A much
better
solution
is
to
begin by collecting
all
the journal
transactions that relate to a given account. This involves sorting the journal
transactions
FIGURE 7.10
by account number, producing
List of journal transactions sorted
a list
ordered
by account number.
Debit/
Acct.
no.
Check no
Date
101
101
101
101
102
1271
1272
1273
1274
670
1273
1271
1274
670
1272
04/02/86
04/02/86
04/04/86
04/09/86
04/07/86
04/04/86
04/02/86
04/09/86
04/07/86
04/02/86
505
510
510
540
550
as in Fig. 7.10.
Description
Auto expense
Rent
Advertising
Auto expense
Office expense
Newspaper ad re: new product
Tune up and minor repair
Oil change
Printer ribbons (6)
Rent for April
credit
- 78.70
- 500.00
- 87.50
- 12.50
- 32.78
87.50
78.70
12.50
32.78
500.00
27
APPLICATION OF THE MODEL TO A GENERAL LEDGER PROGRAM
Ledger
Journal
list
101
Checking account #1
102
505
510
Checking account #2
Advertising expense
Auto expenses
FIGURE 7.1
Conceptual view
101
101
101
101
102
505
510
510
of cosequential
list
Auto expense
Rent
Advertising
Auto expense
Office expense
Newspaper ad re: new product
Tune up and minor repair
Oil change
1271
1272
1273
1274
670
1273
1271
1274
matching
of the ledger
and journal
files.
Now we can create our output list by working through both the ledger
and the sorted journal cosequentially meaning that we process the two lists
sequentially and in parallel. This concept is illustrated in Fig. 7.11. As we
start working through the two lists, we note that we have an initial match
on account number. We know that multiple entries are possible in the
journal file, but not in the ledger, so we move ahead to the next entry in the
journal. The account numbers still match. We continue doing this until the
,
account numbers no longer match.
We
then resynchronize the cosequential
by moving ahead in the ledger list.
This matching process seems simple, as it in fact is, as long as every
account in one file also appears in another. But there will be ledger accounts
for which there is no journal entry, and there can be typographical errors
that create journal account numbers that do not actually exist in the ledger.
Such situations can make resynchronization more complicated and can
result in erroneous output or infinite loops if the programming is done in an
ad hoc way. By using the cosequential processing model, we can guard
against these problems. Let us now apply the model to our ledger problem.
action
7.2.2 Application of the Model to the Ledger Program
The ledger program must perform two
It
needs to update the ledger
file
tasks:
with the correct balance for each ac-
count for the current month.
It
must produce
a printed
version of the ledger that not only shows
the beginning and current balance for each account, but also
lists all
the journal transactions for the month.
We
focus on the second task since
it
is
the
most
difficult.
Let's
look
again at the form of the printed output, this time extending the output to
272
101
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Checking account #1
1271
04/02/86 Auto expense
1272 04/02/86 Rent
1273 04/04/86 Advertising
1274 04/09/86 Auto expense
Prev.
102
515
520
bal:
1321.20
Advertising expense
1273 04/04/86 Newspaper ad
Auto expenses
1271
04/02/86
1274 04/09/86
32.78
New bal: 1288.42
new product
25.00
87.50
New bal:
Tune up and minor repair
Oil change
Prev. bal:
501.12
78.70
12.50
New bal:
592.32
re:
Prev.
510
5219.23
Checking account #2
670 04/07/86 Office expense
Prev.
505
bal:
- 78.70
- 500.00
- 87.50
12.50
New bal 4540.53
bal:
112.50
Bank charges
Prev.
bal:
5.00
New Bal:
5.00
Prev.
bal:
87.40
New bal:
87.40
Books and publications
FIGURE 7.12 Sample ledger printout
include a few
for the first six accounts.
more accounts
as
shown
in Fig. 7.12.
As you can
see, the
printed output from the ledger program shows the balances of
all
accounts, whether or not there were transactions for the account.
From
point of view of the ledger accounts, the process
is
a merge,
ledger
the
since even
unmatched ledger accounts appear in the output.
What about unmatched journal accounts? The ledger accounts and
journal accounts are not equal in authority. The ledger file defines the set of
legal accounts; the journal file contains entries that are to be posted to the
accounts listed in the ledger.
not match
The
existence of a journal account that does
ledger account indicates an error.
From
the point of view of the
one of matching. Our
procedure needs to implement a kind of combined merging/matching
algorithm while simultaneously handling the chores of printing account
title lines, individual transactions, and summary balances.
Another difference between the ledger posting operation and the
journal accounts,
straightforward
the posting process
is
strictly
matching and merging algorithms
is
that
the
ledger
procedure must accept duplicate entries for account numbers in the journal
273
APPLICATION OF THE MODEL TO A GENERAL LEDGER PROGRAM
while
still
earlier
treating a duplicate entry in the ledger as an error. Recall that our
matching and merging routines accept keys only
order, rejecting
The
as
functions that
variables that
ascending
duplicates.
all
inherent simplicity of the three-way
our favor
in
in strict
test,
single-loop
we make these modifications. First,
we use for the ledger and journal
we need for use in the main loop.
let's
model works
look
files,
input
at the
identifying the
Figure 7.13 presents
We
have
treated individual variables within the ledger record as return values to
draw
pseudocode for the procedure
that accepts input
from the
attention to these variables; in practice the procedure
ledger.
would probably
return
the entire ledger record to the calling routine so that other procedures could
have access to things such as the account title as they print the ledger. We
are overlooking such matters here, focusing instead on the variables that are
FIGURE 7.13 Input routine for ledger
PROCEDURE:
ledger_input
input arguments:
L_FILE
J_ACCT
file.
file descriptor for ledger file
current value of journal account number
arguments used to return values:
L_ACCT
account number of new ledger record
balance for this ledger record
L_BAL
flag used by main loop to halt processing
MORE RECORDS EXIST
static,
local variable that retains its value between calls
last acct number read from ledger file
PREV_L_ACCT
read next record from L_FILE, assigning values to L_ACCT and L_BAL
if (EOF)
and (J_ACCT == HIGH_VALUE)
:= FALSE
MORE_RECORDS_EXIST
/* end of both files
*/
/* just ledger is done
*/
/* sequence check
/* (permit no duplicates)
*/
*/
else if (EOF)
L_ACCT
:=
HIGH_VALUE
else if (L_ACCT <= PREV_L_ACCT)
issue sequence check error
abort processing
endif
PREV_L_ACCT
end PROCEDURE
:=
L_ACCT
274
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
involved in the cosequential logic. Note that since
for use with ledger entries,
number
we
this
function
is
strictly
can keep track of the previous ledger account
locally within the procedure rather than pass this value in as an
argument.
Figure 7.14 outlines the logic for the procedure used to accept input
from the journal
though
file.
It is
including that
respects,
a
it
similar to the ledger_input() procedure in
most
returns values for individual variables,
even
working implementation would probably return
the entire journal
record. Note, however, that the sequence-checking logic
journal_input(). In this procedure
same account number
FIGURE 7.14 Input routine for journal
PROCEDURE:
ournal_input
as
we need
is
different in
to accept records that
have the
previous records. Given these input procedures,
file.
input arguments:
J_FILE
L_ACCT
file descriptor for journal file
current value of ledger account number
arguments used to return values:
account number of new journal record
J_ACCT
amount of this journal transaction
TRANS_AMT
flag used by main loop to halt processing
MORE RECORDS EXIST
local variable that retains its value between calls
last acct number read from journal file
PREV_J_ACCT
static,
read next record from J_FILE, assigning values to J_ACCT
and TRANS.AMT
if (EOF)
and (L_ACCT == HIGH_VALUE)
:= FALSE
MORE_RECORDS_EXIST
/* end of both files
*/
/* just ledger is done
*/
/* sequence check
/* (permit duplicates)
*/
*/
else if (EOF)
J_ACCT
HIGH_VALUE
:=
else if (J_ACCT < PREV_J_ACCT)
issue sequence check error
abort processing
endif
PREV_J_ACCT
end PROCEDURE
J_ACCT
275
APPLICATION OF THE MODEL TO A GENERAL LEDGER PROGRAM
PROGRAM:
ledger
call initialize() procedure to:
- open input files L_FILE and J_FILE
- set MORE_RECORDS_EXIST to TRUE
call ledger_input
PREV_L_BAL
L_BAL
call journal_input
/* set starting ledger balance
for this first ledger account
*/
/* we have read all the journal
entries for this account
*/
while (MORE_RECORDS_EXIST)
if (L_ACCT
< J_ACCT)
print PREV_L_BAL and L_BAL
call ledger_input
if (L_ACCT < HIGH_VALUE)
print account number and title for new ledger account
PREV_L_BAL = L_BAL
endif
(
else if (L.ACCT > J.ACCT)
print error message
call journal_input
/* bad journal account number
*/
/* match
add journal transaction amount
/* to ledger balance for this account
else
*/
*/
L_BAL := L_BAL + TRANS_AMT
output the transaction to the printed ledger
call journal_input
endif
endwhile
(
end PROGRAM
FIGURE 7.15 Cosequential procedure to process ledger and journal
files to
produce printed
ledger output.
we
can handle our cosequential processing and output as illustrated in
Fig. 7.15.
The reasoning behind
1.
If the ledger
account
the three-way test
is
less
is
as follows:
than the journal account, then there are
no more transactions to add to this ledger account (perhaps there
were none at all), so we print out the ledger account balances and
read in the next ledger account. If the account exists (value
<
276
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
HIGH_VALUE), we
PREV_BAL
date the
journal account
print the
title line
for the
new
account and up-
variable.
2.
If the
3.
matched journal account, perhaps due to an input error. We print an
error message and continue.
If the account numbers match, then we have a journal transaction
that
is
action
is
less
than the ledger account, then
to be posted to the current ledger account.
amount
We
it is
an un-
add the trans-
to the account balance, print the description of the
Note that unlike
merging algorithms, we do
transaction, and then read in the next journal entry.
the
match
case in either the matching or
not read in
new
entry from both accounts. This
our acceptance of more than one journal entry for
is a
reflection
a single
of
ledger
account.
The development of
cosequential processing
contributes to
its
this
model
adaptability.
ledger posting procedure from our basic
illustrates
We
entirely different direction, extending
it
the simplicity of the
model
to enable cosequential processing
more than two input files at once. To
model to include multiway merging.
7.3
how
can also generalize the model in an
illustrate this,
we now
of
extend the
Extension of the Model to Include Multiway Merging
The most common
than two input files
lists
to create a single,
as the order
of
more
which we want to merge K input
sequentially ordered output list. Kis often referred to
application of cosequential processes requiring
is
K-way
merge, in
K-way merge.
7.3.1 A /(-way Merge Algorithm
Recall the synchronizing loop
we
use to handle
two-way merge of two
lists of names (Fig. 7.5). This merging operation can be viewed as a process
of deciding which of two input names has the minimum value, outputting
name, and then moving ahead in the list from which that name is taken.
of duplicate input entries, we move ahead in each list.
Given a min() function that returns the name with the lowest collating
sequence value, there is no reason to restrict the number of input names to
two. The procedure could be extended to handle three (or more) input lists
that
In the event
as
shown
in Fig. 7.16.
Clearly, the expensive part of this procedure
in
which
lists
the
name
occurs and which
files
is
the series of tests to see
therefore need to be read.
277
EXTENSION OF THE MODEL TO INCLUDE MULTIWAY MERGING
while (MORE_NAMES_EXIST)
OUT_NAME = min(NAME_l, NAME_2
write OUT_NAME to OUT_FILE
NAME_3
...
NAME_K
if (NAME_1 == OUT_NAME)
call input () to get NAME_1 from LIST_1
if (NAME.2 == 0UT_NAME)
call input () to get NAME_2 from LIST_2
if (NAME_3 == 0UT_NAME)
call input () to get NAME_3 from LIST_3
if (NAME_K == 0UT_NAME)
call input () to get NAME_K from LIST_K
endwhile
FIGURE 7.16 /(-way merge loop, accounting
for duplicate
names.
Note that since the name can occur in several lists, every one of these //tests
must be executed on every cycle through the loop. However, it is often
possible to guarantee that a single name, or key, occurs in only one
procedure becomes much simpler and more
reference our lists through a vector of list names
this case, the
we
listen, list[2], list
C3 J
...
efficient.
list.
In
Suppose
listCK]
and suppose we reference the names (or keys) that are being used from these
lists at any given point in the cosequential process through another vector:
named], name[23, name [3],
Then
the procedure
shown
nameCK]
in Fig. 7.17 can be used,
that the input () procedure attends to the
This procedure clearly
...
differs in
assuming once again
MORE_NAMES_EXIST
many ways from our
initial
flag.
three-way
procedure that merges two lists. But, even so, the
single-loop parentage is still evident: There is no looping within a list. We
determine which list has the key with the lowest value, output that key,
test,
single-loop
move
as
it is
ahead one key in that
powerful.
list,
and loop again. The procedure
is
as
simple
278
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
7.3.2 A Selection Tree
for
The K-way merge described
8 or so.
When we
Merging Large Numbers
in Fig. 7.17
begin merging
works nicely if K
number of
larger
sequential comparisons to find the key with the
noticeably expensive.
merge more than
of Lists
no
larger than
lists,
the set of
is
minimum
We see later that for practical
reasons
value becomes
it is
rare to
want
one time, so the use of sequential
comparisons is normally a good strategy. If there is a need to merge
considerably more than eight lists, we could replace the loop of comparisons with a selection tree.
Use of a selection tree is an example of the classic time versus space
trade-off that we so often encounter in computer science. We reduce the
time required to find the key with the lowest value by using a data structure
to save information about the relative key values across cycles of the
procedure's main loop. The concept underlying a selection tree can be
readily communicated through a diagram such as that in Fig. 7.18. Here we
have used lists in which the keys are numbers rather than names.
The selection tree is a kind of tournament tree in which each higher-level
node represents the "winner" (in this case the minimum key value) of the
to
eight
files
at
FIGURE 7.17 K-way merge loop, assuming no duplicate names.
/* initialize the process by reading in a name from each list
for i := 1 to K
call input () to get nameti] from listCi]
next
/* now start the K-way merge */
while (MORE_NAMES_EXIST)
/* find subscript of name that has the lowest collating
sequence value among the names available on the K lists
LOWEST
for
to K
if (nameti]
i
LOWEST
next
< name [LOWEST]
:
write nameCLOWEST] to 0UT_FILE
/* now replace the name that was written out
call input () to get nameCLOWEST] from listCLOWEST]
endwhile
A SECOND LOOK AT SORTING
IN
279
RAM
7, 10,
17
List
9, 19,
23
List
11, 13,
32
List 2
18, 22,
24.
List 3
List 4
List 5
15. 20,
30.
List 6
8, 16,
29.
List 7
Input
12, 14, 21
5,
FIGURE 7.18 Use of a selection tree to assist
mum value in a /(-way merge.
in
6,
25
the selection of a key with mini-
comparison between the two descendent keys. The minimum value is
at the root node of the tree. If each key has an associated reference
to the list from which it came, it is a simple matter to take the key at the
root, read the next element from the associated list, and then run the
tournament again. Since the tournament tree is a binary tree, its depth is
always
rio g2
for a
merge of K
lists.
The number of comparisons
new tournament winner
being
7.4
a linear
In
Chapter 5
to
fit
of course, related to
is,
required to establish
this depth,
rather than
function of K.
A Second Look
enough
Ki
we
in
at Sorting in
RAM
considered the problem of sorting
RAM. The
operation
we
disk
file
that
is
small
described involves three separate
steps:
from disk
RAM.
1.
Read the
2.
Sort the records using a standard sorting procedure, such as Shell-
3.
Write the
entire
file
into
sort.
The
total
steps.
file
back to
disk.
time taken for sorting the
file is
the
sum of the
times for the three
We see that this procedure is much faster than sorting the file in place,
on the
disk, because
both reading and writing are sequential.
280
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Can we improve on the time that it takes for this RAM sort? If we
assume that we are reading and writing the file as efficiently as possible, and
we have chosen the best internal sorting routine available, it would seem
not. Fortunately, there is one way that we might speed up an algorithm that
has several parts, and that is to perform some of those parts in parallel.
Of the three operations involved in sorting a file that is small enough to
fit into RAM, is there any way to perform some of them in parallel? If we
have only one disk drive, clearly we cannot overlap the reading and writing
how about doing either
that we sort the file?
operations, but
the
same time
7.4.1 Overlapping Processing and
Most of the time when we use an
the
whole
file in
memory
the reading or writing (or both) at
I/O:
Heapsort
internal sort
before
we
can
we have to
wait until
start sorting. Is there
we have
an internal
and that can begin sorting numbers
immediately as they are read in, rather than waiting for the whole file to be
in memory? In fact, there is, and we have already seen part of it in this
chapter. It is called heapsort, and it is loosely based on the same principle as
sorting algorithm that
is
reasonably
fast
the selection tree.
Recall that the selection tree compares keys as
time
key
a
it
new key
arrives,
it is
compared
goes to the front of the
because
it
means
that
we
tree.
is,
This
is
encounters them. Each
file is
and
if
it is
the largest
very useful for our purposes
can begin sorting keys
rather than waiting until the entire
That
it
to the others,
as
they arrive in
loaded before
we
RAM,
start sorting.
sorting can occur in parallel with reading.
Unfortunately, in the case of the selection tree, each time a new largest
key is found it is output to the file. We cannot allow this to happen if we
want to sort the whole file because we cannot begin outputting records until
we know which one comes first, second, etc., and we won't know this until
we have seen all of the keys.
Heapsort solves this problem by keeping all of the keys in a structure
called a heap. A heap is a binary tree with these properties:
Each node has a single key, and that key is less than or equal to the
key at its parent node.
i\2j It is a complete binary tree, which means that all of its leaves are on
no more than two levels, and all of the keys on the lowest level are
'*~\.)
^_
in the leftmost position.
Because of properties 1 and 2, storage for the tree can be allocated
sequentially as an array in such a way that the indices of the left and
right children of node i are 2i and 2i + 1, respectively. Conversely,
the index of the parent of node j is |_j/2j.
A SECOND LOOK AT SORTING IN
23456789
IT
/\
/\
281
RAM
P
ew^>
Q-
/\
FIGURE 7.19 A heap
in
both
Figure
its
tree form
and as
it
would be stored
in
an array.
19 shows a heap in both its tree form and as it would be stored
Note that this is only one of many possible heaps for the given
7.
in an array.
of keys. In practice, each key has an associated record that is either stored
with the key or pointed to by a pointer stored with the key.
Property 3 is very useful for our purposes, because it means that a heap
is just an array of keys, where the positions of the keys in the array are
sufficient to impose an ordering on the entire set of keys. There is no need
for pointers or other dynamic data structuring overhead to create and
maintain the heap. (As we pointed out earlier, there may be pointers
associating each key with its corresponding record, but this has nothing to
do with maintaining the heap itself.)
set
in the array
7.4.2 Building the Heap while Reading
The algorithm
we
for heapsort has
two
parts. First
output the keys in sorted order. The
same time
that
we
The
shown
7.20.
Fig.
first
the File
we
build the heap, and then
stage can occur at virtually the
read in the data, so in terms of computer time
essentially free.
in
in
it
comes
basic steps in the algorithm for building the heap are
Figure 7.21 contains
sample application of
this
algorithm.
This describes
how we
build the heap, but
it
doesn't
the input overlap with the heap-building procedure.
To
tell
how
to
FIGURE 7.20 Procedure for building a heap.
For
:=
to
REC0RD_C0UNT
Read in the next record and append it to the end of the
array; call it5 key K
While K 15 less than the key of its parent:
Exchange the record with key K with its parent
next
make
solve that problem,
282
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
FDCGHIBEA
New key to
Heap, after insertion
be inserted
of the
Selected heaps
new key
in tree
form
12345678'
F
12
D F
12
C F D
^\
123456789
C F D G
123456789
C F D G H
123456789
C F D G H
123456789
B F C G H
123456789
BECFHIDG
123456789.
ABCEHIDGF
g'Nf
HI
FIGURE 7.21 Sample application of the heap-building algorithm. The keys
G, H, I, B, E, and A are added to the heap in the order shown.
we need
to look at
not going to do
a
block of records
F,
D, C,
how we perform the read operation. For starters, we are
we want a new record. Instead, we read
seek every time
at a
time into an input buffer, and then operate on
all
of
RAM
the input buffer for each new block of keys can be part of the RAM
the records in the block before going on to the next block. In terms of
storage,
is set up for the heap itself. Each time we read in a new block, we
append it to the end of the heap (i.e., the input buffer "moves" as the
heap gets larger). The first new record is then at the end of the heap array,
as required by the algorithm (Fig. 7.20). Once that record is absorbed into
the heap, the next new record is at the end of the heap array, ready to be
absorbed into the heap, and so forth.
Use of an input buffer avoids doing an excessive number of seeks, but
it still doesn't let input occur at the same time that we build the heap. We
area that
just
A SECOND LOOK AT SORTING
IN
283
RAM
saw in Chapter 3 that the way to make processing overlap with I/O is to use
more than one buffer. With multiple buffering, as we process the keys in
one block from the file, we can simultaneously be reading in later blocks
from the file. If we use multiple buffers, how many should we use, and
where should we put them? We already answered these questions when we
decided to put each
new
a new
new
block
at the
block, the array gets bigger
by
end of the array. Each time
we add
the size of that block, in effect creating
file. So the number of buffers is the
and they are located in sequence in the array
input buffer for each block in the
number of blocks
in the
file,
itself.
Figure 7.22 illustrates the technique that
we append
employing
on
we
we have just
described,
where
block of records to the end of the heap, thereby
RAM-sized
set
of input buffers.
Now we
read in
new
blocks
having to wait for processing before reading in a
block. On the other hand, processing (heap building) cannot occur
given block until the block to be processed is read in, so there may
as fast as
new
new
each
can, never
be some delay in processing
if
processing speeds are faster than reading
speeds.
7.4.3 Sorting while Writing out to the
The second and
Again,
it is
First, let's
final step
File
involves writing out the heap in sorted order.
possible to overlap I/O (in this case writing) with processing.
look
at the
Again, there
is
algorithm for outputting the sorted keys
nothing inherent in
this
algorithm that
(Fig. 7.23).
lets it
overlap
with I/O, but we can take advantage of certain features of the algorithm to
make overlapping happen. First, we see that we know immediately which
record will be written first in the sorted file; next, we know what will come
second; and so forth. So as soon as we have identified a block of records, we
can write out that block, and while we are writing out that block we can be
identifying the next block, and so forth.
Furthermore, each time we identify a block to write out, we make the
heap smaller by exactly the size of a block, freeing that space for a new
output buffer. So just as was the case when building the heap, we can have
as many output buffers as there are blocks in the file. Again, a little
coordination is required between processing and output, but the conditions
exist for the two to overlap almost completely.
A final point worth making about this algorithm is that all I/O that it
performs is essentially sequential. All records are read in in the order in
which they occur in the file to be sorted, and all records are written out in
sorted order. The technique could work equally well if the file were kept on
tape or disk.
More
can be done with
importantly, since
all
minimum amount
I/O
is
sequential,
of seeking.
we know
that
it
284
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Total
RAM area allocated for heap
First input buffer. First part
SZ
added
to the heap, then the
of heap is built here. The first record
second record is added, and so forth.
is
Second input buffer. This buffer is being
filled while heap is being built in first buffer.
Second part of heap is built here. The first record
added to the heap, then the second record, etc.
is
Third input buffer. This buffer is filled
is being built in second buffer.
while heap
Third part of heap
is
built here.
1
Fourth input buffer
heap
is
is
filled while
being built in third buffer.
FIGURE 7.22 Illustration of the technique described in the text for overlapping input with
heap building in RAM. First read in a block into the first part of RAM. The first record is the
first record in the heap. Then extend the heap to include the second record, and incorporate
that record into the heap, and so forth. While the first block is being processed, read in the
second block. When the first block is a heap, extend it to include the first record in the second block, incorporate that record into the heap, and go on to the next record. Continue until
all blocks are read in and the heap is completed.
FIGURE 7.23 Procedure
For
:=
to
for outputting the
contents of a heap
in
sorted order.
REC0RD_C0UNT
Output the record in the first position in the array (this
record has the smallest key).
Move the key in the last position in the array (call it K)
to the first position, and define the heap as having one
fewer member than it previously had.
While K is larger than both keys of its children:
Exchange K with the smaller of its two children's keys
next
MERGING AS A WAY OF SORTING LARGE
7.5
Way
Merging as a
In
Chapter 5
we
FILES
of Sorting Large Files
ran into problems
too large to be wholly contained in
when we needed
RAM. The
285
ON DISK
on Disk
to sort files that
chapter offered
were
a partial,
but
ultimately unsatisfactory, solution to this problem in the form of a key sort,
in
which we needed
to hold only the keys in
RAM,
along with pointers to
each key's corresponding record. Keysort had two shortcomings:
Once the keys were sorted, we then had to bear the substantial cost
of seeking to each record in sorted order, reading each record in and
then writing it out into the new, sorted file.
With keysorting, the size of the file that can be sorted is limited by
the number of key/pointer pairs that can be contained in RAM.
Consequently, we still cannot sort really large files.
RAM
As an example of the kind of file we cannot
sort with either a
sort
have a file with 800,000 records, each of which is
100 bytes long and contains a key field that is 10 bytes long. The total length
of this file is about 80 megabytes. Let us further suppose that we have one
megabyte of
available as a work area, not counting
used to
hold the program, operating system, I/O buffers, and so forth. Clearly, we
cannot sort the whole file in RAM. We cannot even sort all the keys in
or a keysort, suppose
we
RAM
RAM
RAM.
The multiway merge algorithm
discussed in section 7.3 provides the
beginning of an attractive solution to the problem of sorting large files such
as this one. Since
sorting algorithms such as heapsort can work in
RAM
place, using only a small
some temporary
amount of overhead
variables,
reading records into
we
can create
for maintaining pointers
a sorted subset
RAM until the RAM work area
work
area,
disk as a sorted subfile.
We
is
of our
almost
and
full file
full,
by
sorting
and then writing the sorted records back to
such a sorted subfile a run. Given the
memory constraints and record size in our example, a run could contain
approximately
the records in this
call
1,000,000 bytes of
RAM
-7
-r
100 bytes per record
Once we
again filling
example,
we
10,000 records.
we then read in a new set of records, once
and create another run of 10,000 records. In our
repeat this process until we have created 80 runs, with each run
create the first run,
RAM,
containing 10,000 sorted records.
Once we have the 80 runs in 80 separate files on disk, we can perform
an 80-way merge of these runs, using the multiway merge logic outlined in
286
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
800,000 unsorted records
iT
oo
<X~wOO
80 internal sorts
^^
80 runs, each containing 10,000 sorted records
iic^-i
800,000 records in sorted order
FIGURE 7.24 Sorting through the creation
merging of runs.
of runs (sorted subfiles)
section 7.3, to create a completely sorted
records.
schematic view of
this
file
and subsequent
containing
all
provided in Fig. 7.24.
This solution to our sorting problem has the following
It
can, in fact, sort large
files,
the original
run creation and merging process
and can be extended to
is
features:
files
of any
size.
Reading of the input file during the run creation step is sequential,
and hence is much faster than input that requires seeking for every
record individually
(as in a keysort).
Reading through each run during merging and writing out the sorted
records
is
also sequential.
Random
accesses are required only as
switch from run to run during the merge operation.
we
MERGING AS A WAY OF SORTING LARGE
If a
heapsort
RAM
used for the
is
in section 7.4,
we
in-RAM
FILES
287
ON DISK
part of the merge, as described
can overlap these operations with I/O, so the in-
part does not add appreciably to the total time for the merge.
Since I/O
largely sequential, tapes can be used if necessary for both
is
input and output operations.
How Much Time Does
7.5.1
Merge Sort Take?
This general approach to the problem of sorting large
To compare
takes.
long
We do this
it
files
approach to others, we now look at
by taking our 800,000-record example
this
do
takes to
merge
sort
looks promising.
how much
file
time
and seeing
it
how
on the hypothetical disk drive whose
specifications are listed in Table 3.2. (Please note that our intention here
mean anything
is
any environment other
Nor do we want to
overwhelm you with numbers or provide you with magic formulas for
determining how long a particular sort on a real system will really take.
Rather, our goal in this section is to derive some benchmarks that we can
use to compare several variations on the basic merge sort approach to
not to derive time estimates that
than the hypothetical environment
sorting external
We
the
we have
in
posited.
files.)
can simplify matters by making the following assumptions about
computing environment:
Entire
files
are always stored in contiguous areas
and
a single
seek
is
We
(extents),
cylinder-to-cylinder seek takes no time. Hence, only one
required for any single sequential access.
Extents that span
such
on disk
way
more than one
track are physically staggered in
that only one rotational delay
is
required per access.
see in Fig. 7.24 that there are four times
During the sort phase:
Reading all records
into
when I/O
RAM for sorting
is
performed:
and forming runs; and
Writing sorted runs out to disk.
During the merge phase:
Reading sorted runs into
Writing sorted
Let's look at each
of these
1:
Since
we
sort the
time from the
file
file.
in
In
RAM
for merging;
and
out to disk.
in order.
RAM
for Sorting and Forming Runs
one-megabyte chunks, we read in one megabyte at
a sense, RAM is a one-megabyte input buffer that
Reading Records into
Step
a
file
288
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
we
fill
up 80 times
to
form the 80
runs. In
computing the
total
time to input
we need to include the amount of time it takes to access each block
(seek time + rotational delay), plus the amount of time it takes to transfer
each block. We keep these two times separate because, as we see later in our
each run,
calculations, the role that each plays can vary significantly depending on the
approach used.
From Table 3.2 we see that seek and rotational delay times are 18 msec"
and 8.3 msec, respectively, so total time per seek is 26.3 msec* The
transmission rate is approximately 1,229 bytes per msec. Total input time
for the sort phase consists of the time required for 80 seeks, plus the time
required to transfer 80 megabytes:
"
80 seeks X 26.3 msec
Access:
80 megabytes
Transfer:
1,229 bytes/msec
=
=
67 seconds.
Total:
Step
2:
2 seconds
65 seconds
Writing Sorted Runs out to Disk In this case, writing is just the
the same number of seeks and the same amount of data
reverse of reading
to transfer.
So
it
takes another 67 seconds to write out the 80 sorted runs.
RAM
for Merging Since we have
Step 3: Reading Sorted Runs into
for storing runs, we divide one megabyte into 80
one megabyte of
parts for buffering the 80 runs. In a sense, we are reallocating our one
megabyte of
as 80 input buffers. Each of the 80 buffers then holds
l/80th of a run (12,500 bytes), so we have to access each run 80 times to read
all of it. Since there are 80 runs, to complete the merge operation (Fig. 7.25)
we end up making
RAM
RAM
80 runs x 80 seeks
Total seek and rotation time
80 megabytes
is still
is
6,400 seeks.
then 6,400 X 26.3 msec
transferred, transfer time
is still
=168
seconds. Since
65 seconds.
computing environment has many active users pulling the read/write head to
other parts of the disk, seek time is actually likely to be less than the average, since many
of the blocks that make up the file are probably going to be physically adjacent to one another on the disk. Many will be on the same cylinder, requiring no seeks at all. However,
for simplicity we assume the average seek time.
""Unless the
*For simplicity, we use the term seek even though we really mean seek and rotational delay.
Hence, the time we give for a seek is the time that it takes to perform an average seek followed by an average rotational delay.
MERGING AS A WAY OF SORTING LARGE
1st
ii
FILES
289
ON DISK
run = 80 buffers' worth (80 accesses)
mi mi
ii
ii ii
ii
ii
2nd run = 80 buffers' worth (80 accesses)
H Ml Ml
1
II
III
II
II
II
800,000
sorted records
80th run = 80 buffers' worth (80 accesses)
i
ii
ii
ii
FIGURE 7.25 Effect of buffering on the number of seeks required, where each run
large as the available work area in RAM.
Step
4:
Writing Sorted File out to Disk
writing out the
Unlike steps
buffer,
before
we
it is
are
file,
and
now
actually
2,
we need
to
know how
To compute
is
as
the time for
big our output buffers are.
RAM sorting space doubled as our I/O
RAM space for storing the data from the runs
where our big
using that
merged.
To keep
matters simple,
let
us assume that
can allocate two 20,000-byte output buffers." With 20,000-byte buffers,
1"
we
we
need to make
80,000,000 bytes
4,000 seeks.
20,000 bytes per seek
Total seek and rotation time
Transfer time
is still
is
then 4,000 X 26.3 msec
=105
seconds.
65 seconds.
The time estimates for the four steps are summarized in the first row in
7.1. The total time for this merge sort is 537 seconds, or 8 minutes,
57 seconds. The sort phase takes 134 seconds, and the merge phase takes 403
Table
seconds.
To gain an appreciation of the improvement that this merge sort
approach provides us, we need only look at how long it would take us to
do one part of a nonmerging method like the keysort method described in
We
is
use
two
buffers to allow double buffering;
approximately the
size
of
a track
we
use 20,000 bytes per buffer because that
on our hypothetical disk
drive.
290
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
TABLE
7.1
Time estimates
for
merge
phase (steps 1 and 2)
phase is 403 seconds.
is
80-megabyte
file, assuming use of
Table 3.2. The total time for the sort
134 seconds, and the total time for the merge
sort of
hypothetical disk drive described
in
Number
Amount
Seek + Rotation
Transfer
of
Transferred
(Megabytes)
Time
Time
(Seconds)
(Seconds)
Seeks
Total Time
(Seconds)
Sort: reading
80
80
65
Sort: writing
80
80
65
67
6,400
80
168
65
233
Merge: reading
Merge: writing
Totals
67
4,000
80
105
65
170
10,560
320
277
260
537
Chapter
5.
The
last
part of the keysort algorithm (Fig. 5.17) consists of this
for loop:
/*
/*
for
read in records according to sorted order, and write them
in this order
out
i
:=
to
*/
*/
REC_CDUNT
seek in IN_FILE to record with RRN of KEYN0DES
N_F LE
read the record into BUFFER from
I
RRN
write BUFFER contents to DUT_FILE
This for loop requires us to do
a separate seek for
every record in the
file.
That is 800,000 seeks. At 26.3 msec per seek, the total time required to
perform that one operation works out to 21,040 seconds, or 5 hours, 50
minutes, 40 seconds!
Clearly, for large files the merge sort approach in general is the best
option of any that we have seen. Does this mean that we have found the best
technique for sorting large files? If sorting is a relatively rare event and files
are not too large, the particular approach to merge sorting that we have just
looked at produces acceptable results. Let's see how those results stand up
as we change some of the parameters of our sorting example.
7.5.2 Sorting a
The
first
File
That
Is
Ten Times Larger
question that comes to
applicability of a
mind when we ask about the general
is, What happens when we make the
computing technique
problem bigger? In this instance, we need
up as we scale up the size of the file.
to ask
how
this
approach stands
MERGING AS A WAY OF SORTING LARGE
Before
we
look
at
how
bigger
file affects
291
ON DISK
FILES
the performance of our
merge sort, it will help to examine the kinds of I/O that are being done in
the two different phases, the sort phase and the merge phase. We will see
that for the purposes of finding ways to improve on our original approach,
we need pay attention only to one of the two phases.
A major difference between the sort phase and the merge phase is in the
amount of sequential (vs. random) access that each performs. By using
we
all I/O is,
minimal seeking, we
cannot algorithmically speed up I/O during the sort phase. No matter what
we do with the records in the file, we have to read them and write them all
at least once. Since we cannot improve on this phase by changing the way
we do the sort or merge, we ignore the sort phase in the analysis that
heapsort to create runs during the sort phase,
guarantee that
in a sense, sequential.^ Since sequential access implies
follows.
the
The merge phase is a different matter. In particular, the reading step of
merge phase is different. Since there is a RAM buffer for each run, and
these buffers get loaded and reloaded at unpredictable times, the read step of
the merge phase is to a large extent one in which random accesses are the
norm. Furthermore, the number and size of the RAM buffers that we read
the run data into determine the number of times we have to do random
accesses. If we can somehow reconfigure these buffers in ways that reduce
the number of random accesses, we can speed up I/O correspondingly. So,
if we are going to look for ways to improve performance in a merge sort
algorithm, our best hope is to look for ways to cut down on the number of random
accesses that occur while reading runs during the
What about
the write step of the
not influenced by differences in the
phase, this step
is
Improvements
in the
On
merge phase.
merge phase? Like
the other hand,
when we measure
the steps of the sort
way we
organize runs.
way we organize the merge sort do not affect this step.
we will see later that it is helpful to include this phase
the results of changes in the organization of the
merge
sort.
To sum up, since the merge phase is the only one in which we can
improve performance by improving the method, we concentrate on it from
now on. Now let's
get back to the question that
What happens when we make
the
we started
problem bigger? How,
time for the merge phase affected if our
file is
this section with:
for instance,
is
800,000?
"'"It
is
not sequential in the sense that in a multiuser
environment there
will be other users
pulling the read/write head to other parts of the disk between reads and writes, possibly
forcing the disk to do
seek each time
it
the
8,000,000 records rather than
reads or writes a block.
r^
292
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
TABLE 7.2
Time estimates
merge
phase
800-megabyte
file, assuming use of
Table 3.2. The total time for the merge
19,186 seconds, or 5 hours, 19 minutes, 22 seconds.
for
sort of
hypothetical disk drive described
is
in
Number
Amount
Seek + Rotation
Transfer
of
Transferred
(Megabytes)
Time
Time
(Seconds)
(Seconds)
Seeks
Total Time
(Seconds)
Merge: Reading
640,000
800
16,832
651
Merge: Writing
40,000
800
1,050
651
1,703
680,000
1,600
17,882
1,302
19,186
Totals
we increase
space, we
17,483
file by a factor of 10 without increasing the
need to create more runs. Instead of 80 initial
10,000-record runs, we now have 800 runs. This means we have to do an
800-way merge in our one megabyte of
space. This, in turn, means
that during the merge phase we must divide
into 800 buffers. Each of
the 800 buffers holds 1 /800th of a run, so we would end up making 800
seeks per run, and
If
RAM
the size of our
clearly
RAM
RAM
800 runs x 800 seeks/run
The times
for the
merge phase
are
640,000 seeks altogether.
summarized
in
Table
7.2.
Note
that
is over 5 hours and 19 minutes, almost 50 times greater than
80-megabyte file. By increasing the size of our file, we have gotten
ourselves back into the situation we had with keysort, where we can't do
the job we need to do without doing a huge amount of seeking. In this
instance, by increasing the order of the merge from 80 to 800, we made it
necessary to divide our one-megabyte RAM area into 800 tiny buffers for
doing I/O, and because the buffers are tiny each requires many seeks to
the total time
for the
process
its
corresponding run.
to improve performance, clearly we need to look for ways
improve
on
to
the amount of time spent getting to the data during the
merge phase. We will do this shortly, but first let us generalize what we
If
we want
have just observed.
7.5.3 The Cost of Increasing the
File Size
Obviously, the big difference between the time it took to merge the
8-megabyte file and the 800-megabyte file was due to the difference in total
seek and rotational delay times. You probably noticed that the number of
MERGING AS A WAY OF SORTING LARGE
seeks for the larger
100
is
file is
number of seeks
100 times the
for the first
the square of the difference in size between the
formalize this relationship as follows: In general, for a
runs where each run
for each
of the runs
so
K seeks
is
two
files.
file,
and
We
can
K-way merge of K
RAM space available,
as large as the
293
ON DISK
FILES
the buffer size
is
of
size
RAM
space
are required to read in
K runs
all
= I x
size
of each run,
of the records
in each individual run.
merge operation requires K2 seeks.
2
Hence, measured in terms of seeks, our sort merge is an 0(K ) operation.
Since K is directly proportional to N (if we increase the number of records
from 800,000 to 8,000,000, K increases from 80 to 800) it also follows that
2
our sort merge is an 0(N ) operation, measured in terms of seeks.
Since there are
This
we
brief,
altogether, the
formal look establishes the principle that
as files
grow
large,
can expect the time required for our merge sort to increase rapidly.
would be very
nice if
we
could find some ways to reduce
this
It
time.
Fortunately, there are several:
Allocate
more hardware, such
as disk drives,
RAM,
and I/O chan-
nels;
Perform the merge in more than one step, reducing the order of each
merge and increasing the buffer size for each run;
Algorithmically increase the lengths of the initial sorted runs; and
Find ways to overlap I/O operations.
In the following sections
with the
first:
Invest in
we
look
at
each of these in
detail,
beginning
more hardware.
7.5.4 Hardware-based Improvements
We
have seen that changes in our sorting algorithm can improve
performance. Likewise, there are changes that we can make in our hardware
that will also improve performance. In this section we look at three possible
changes to a system configuration that could lead to substantial decreases in
sort time:
Increasing the
Increasing the
Increasing the
amount of RAM;
number of disk drives; and
number of I/O channels.
RAM
Increasing the Amount of
It should be clear now that when we
have to divide limited buffer space into many small buffers, we increase
294
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
overwhelm all other sorting
number of seeks is
file size, given a fixed amount
seek and rotation times to the point where they
operations.
Roughly speaking,
the increase in the
proportional to the square of the increase in
of
total buffer space.
It
RAM space ought to have a
A larger RAM size means longer and
stands to reason, then, that increasing
substantial effect
on
total sorting time.
fewer initial runs during the sort phase, and it means fewer seeks per run
during the merge phase. The product of fewer runs and fewer seeks per run
means
a substantial
reduction in total seeks.
with our 8,000,000-record file, which took
about 5 hours, 20 minutes using one megabyte of RAM. Suppose we are
able to obtain 4 megabytes of
buffer space for our sort. Each of the
Let's test this conclusion
RAM
would
from 10,000 records to 40,000 records, resulting
in 200 40,000-record runs. For the merge phase, the internal buffer space
would be divided into 200 buffers, each capable of holding 1 /200th of a run,
meaning that there would be 200 X 200 = 40,000 seeks. Using the same
time estimates that we used for the previous two cases, the total time for
this merge is 56 minutes, 45 seconds, nearly a sixfold improvement.
initial
runs
increase
Number
of Dedicated Disk Drives If we could have a
no other users contending for use
of the same read/write heads, there would be no delay due to seek time after
the original runs are generated. The primary source of delay would now be
rotational delays and transfers, which would occur every time a new block
Increasing the
separate read/write head for every run and
had to be read in.
For example, if each run is on a separate, dedicated drive, our 800-way
merge calls for only 800 seeks (one seek per run), down from 640,000, and
cutting the total seek and rotation times from 11,500 seconds to 14 seconds.
Of course we can't configure 800 separate disk drives every time we want
to do a sort, but perhaps something short of this is possible. For instance,
if we had two disk drives to dedicate for the merge, we could assign one to
input and the other to output, so reading and writing could overlap
whenever they occurred simultaneously. (This approach takes some clever
buffer management, however. We discuss this later in this chapter.)
Increasing the
Number
of I/O Channels
two transmissions can occur
If there
is
only one I/O
same time, and the total
transmission time is the one we have computed. But if there is a separate
I/O channel for each disk drive, I/O can overlap completely.
For example, if for our 800-way merge there are 800 channels from 800
channel, then no
at
the
disk drives, then transmissions can overlap completely. Practically speaking,
it is
unlikely that 800 channels and 800 disk drives are available, and
MERGING AS A WAY OF SORTING LARGE
even
all
if
FILES
295
ON DISK
it is unlikely that all transmissions would overlap because
would not need to be refilled at one time. Nevertheless,
the number of I/O channels could improve transmission time
they were,
buffers
increasing
substantially.
So
we
see that there are
control over
how
ways
which external sorting occupies
have
are likely to
at least
improve performance
is
configured. In those environments in
a large
some such
we are not able to expand
might have. When this is
system
the case,
improve performance, and
this
is
we have some
to
our hardware
if
percentage of computing time,
control.
we
On the other hand, many times
specifically to
meet sorting needs
we need to look for
what we do now.
that
we
algorithmic ways to
7.5.5 Decreasing the Number of Seeks Using
Multiple-step Merges
One of the
hallmarks of
a solution to a file structure
problem,
as
opposed
of a mere data structure problem, is the attention given to the
between accessing information on disk and
accessing information in RAM. If our merging problem involved only
operations, the relevant measure of work, or expense, would be the
number of comparisons required to complete the merge. The merge pattern
that would minimize the number of comparisons for our sample problem,
in which we want to merge 800 runs, would be the 800-way merge
considered. Looked at from a point of view that ignores the cost of seeking,
to the solution
enormous
difference in cost
RAM
this
K-way merge
has the following desirable characteristics:
Each record
read only once.
is
If a selection tree
is
used for the comparisons performed in the mergnumber of com-
ing operation, as described in section 7.3, then the
parisons required for
tion of
Since
is
K-way merge of N
records
(total) is a
func-
log K.
directly proportional to
N,
this is
an
0(N
numbers of comparisons), which
reasonably efficient even as N grows large.
tion (measured in
is
log N) operato say that
it is
be very good news were we working exclusively in
sort procedure is to be able to sort
files that are too large to fit into RAM. Given the task at hand, the costs
associated with disk seeks are orders of magnitude greater than the costs of
operations in RAM. Consequently, if we can sacrifice the advantages of an
800-way merge, trading them for savings in access time, we may be able to
obtain a net gain in performance.
This would
RAM,
all
but the very purpose of this merge
296
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
We
have seen that one of the keys to reducing seeks is to reduce the
that we have to merge, thereby giving each run a bigger
number of runs
share of available buffer space. In the previous section
we accomplished
this
by adding more memory. Multiple-step merging provides a way for us to
apply the same principle without having to go out and buy more memory.
In multiple-step merging, we do not try to merge all runs at one time.
Instead, we break the original set of runs into small groups and merge the
runs in these groups separately.
On
each of these smaller merges, more
and hence, fewer seeks are required
of the smaller merges are completed, a second pass
merges the new set of merged runs.
It should be clear that this approach will lead to fewer seeks on the first
pass, but now there is a second pass. Not only are a number of seeks
required for reading and writing on the second pass, but extra transmission
time is used in reading and writing all records in the file. Do the advantages
of the two-pass approach outweigh these extra costs? Let's revisit the merge
step of our 8-million record sort to find out.
Recall that we began with 800 runs of 10,000 records each. Rather than
merging all 800 runs at once, we could merge them as, say, 25 sets of 32
runs each, followed by a 25-way merge of the intermediate runs. This
buffer space
per run.
scheme
is
is
available for each run,
When
all
illustrated in Fig. 7.26.
When compared
to our original
disadvantage of requiring that
FIGURE 7.26 Two-step merge of
runs.
32 runs
32 runs
VV V
800-way merge,
this
approach has the
read every record twice: once to form the
25 sets of 32 runs each
32 runs
800
we
MERGING AS A WAY OF SORTING LARGE
FILES
intermediate runs and then again to form the final sorted
each step of the merge
is
reading from 25 input
and avoid
to use larger buffers
files at a
number of
a large
297
ON DISK
But, since
file.
we are able
seeks. When we
time,
disk
analyzed the seeking required for the 800-way merge, disregarding seeking
for the output
we
file,
800-way merge involved 640,000
perform similar calculations for our
calculated that the
seeks between the input
files.
Let's
multistep merge.
First
Merge Step
For each of the 32-way merges of the
input buffer can hold V32 run, so
initial
runs, each
we end up making 32 X 32 = 1,024 seeks.
we make 25 x 1,024 = 25,600 seeks. Each
For all 25 of the 32-way merges,
of the resulting runs is 320,000 records, or 32 megabytes.
Second Merge Step
space
For each of the 25
final runs, Vis
of the
total buffer
400 records, or Vsoo run.
step there are 800 seeks per run, so we end up making 25 X
allocated, so each input buffer can hold
is
Hence, in this
800 = 20,000 seeks, and
The
So,
total
number of seeks
by accepting the
number of
for the
two
steps
25,600
cost of processing each record twice,
seeks for reading in
spent a penny for extra
But what about the
from 640,000
20,000
to 45,600,
we
45,600.
reduce the
we
and
haven't
RAM.
total
time for the merge?
inputting data, but there are costs.
We now
We save on access times for
have to transmit
all
of the
records four times instead of two, so transmission time increases by 651
seconds. Also,
we
write the records out twice, rather than once, requiring
an extra 40,000 seeks.
for the
merge
is
When we add
in these extra operations, the total
5,907 seconds, or about
5 hours, 20 minutes for the single-step merge. These results are
in
Table
Once more, note
over the data for
is
If
summarized
7.3.
that the essence of what
we have done is
to increase the available buffer space for each run.
trade
time
hour, 38 minutes, compared to
We
to find a
way
trade extra passes
dramatic decrease in random accesses. In
this case the
certainly a profitable one.
we
can achieve such an improvement with
do even
better with three steps? Perhaps, but
7.3 that
we have
reduced
total seek
it is
file,
we
where
three-step merge would
and rotation times
transmission times are about as expensive. Since
require yet another pass over the
two-step merge, can
important to note in Table
we may
to the point
have reached
point of
diminishing returns.
We also could have chosen to distribute our initial runs differently.
How would the merge perform if we did 400 two-way merges, followed by
298
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Time estimates
TABLE 7.3
for two-step
merge
hypothetical disk drive described
sort of
in
800-megabyte
Table 3.2. The
total
file,
time
assuming use
is 1 hour, 31
of
minutes.
Number
Amount
Seek + Rotation
Transfer
of
Transferred
(Megabytes)
Time
Time
(Seconds)
(Seconds)
Seeks
Total Time
(Seconds)
1st
Merge: Reading
25,600
800
673
651
1,324
1st
Merge: Writing
40,000
800
1,052
651
1,703
2nd Merge: Reading
20,000
800
526
651
1,177
2nd Merge: Writing
40,000
800
1,052
651
1,703
125,600
3,200
3,303
2,604
5,907
Totals
one 400-way merge, for instance? A rigorous analysis of the trade-offs
between seek and rotation time and transmission time, accounting for
different buffer sizes, is beyond the scope of our treatment of the subject.'*'
Our goal is simply to establish the importance of the interacting roles of the
major costs in performing merge sorts: seek and rotation time, transmission
time, buffer size, and number of runs. In the next section we focus on the
the number of runs.
pivotal role of the last of these
7.5.6 Increasing Run Lengths Using Replacement Selection
What would happen
if
we
could
somehow
increase the size of the initial
runs? Consider, for example, our earlier sort of 8,000,000 records in which
each record was 100 bytes.
Our
10,000 records because the
RAM work area was limited to
Suppose
we
are
somehow
initial
able
to
runs were limited to approximately
create
one megabyte.
runs of twice this length,
containing 20,000 records each. Then, rather than needing to perform an
800-way merge, we need
is
to
do only
400-way merge. The
divided into 400 buffers, each holding
the
number of seeks
required per run
is
available
RAM
/800th of a run. (Why?) Hence,
800, and the total
number of seeks
is
800 seeks/run x 400 runs
half the
""For
number
320,000 seeks,
required for the 800-way merge of 10,000-byte runs.
more rigorous and
end of
detailed analyses of these issues, consult the references cited at the
this chapter, especially
Knuth (1973b) and Salzberg
(1988, 1990).
MERGING AS A WAY OF SORTING LARGE
In general, if
we
can
somehow
FILES
299
ON DISK
increase the size of the initial runs,
we
amount of work required during the merge step of the sorting
process. A longer initial run means fewer total runs, which means a
lower-order merge, which means bigger buffers, which means fewer seeks.
But how, short of buying twice as much memory for the computer, can we
create initial runs that are twice as large as the number of records that we can
hold in RAM? The answer, once again, involves sacrificing some efficiency
in our in-RAM operations in return for decreasing the amount of work to
be done on disk. In particular, the answer involves the use of an algorithm
decrease the
known
as replacement selection.
Replacement selection
from memory
replacing
it
with
implemented
1.
is
based on the idea of always
key
and then
Replacement selection can be
selecting the
that has the lowest value, outputting that key,
a
new key from
the input
list.
as follows:
of records and sort them using heapsort. This
heap of sorted values. Call this heap the primary heap.
Instead of writing out the entire primary heap in sorted order (as we
do in a normal heapsort), write out only the record whose key has
Read
in a collection
creates a
2.
3.
4.
the lowest value.
Bring in a new record and compare the value of its key with that of
the key that has just been output.
a.
If the new key value is higher, insert the new record into its
proper place in the primary heap along with the other records
that are being selected for output. (This makes the new record
part of the run that is being created, which means that the run
being formed will actually be larger than the number of keys
that can be held in memory at one time.)
b.
If the new record's key value is lower, place the record in a secondary heap of records with key values lower than those already
written out. (It cannot be put into the primary heap, because it
cannot be included in the run that is being created.)
Repeat step 3 as long as there are records left in the primary heap
and there are records to be read in. When the primary heap is empty,
make the secondary heap into the primary heap and repeat steps 2
and 3.
To
see
how
this
works,
input
list
of only
keys.
As
Fig. 7.27 illustrates,
six
keys and
let's
begin with
simple example, using an
memory work area that can hold
we begin by reading into RAM the
a
there and use heapsort to sort them.
We
only three
three keys
key with the
minimum value, which happens to be 5 in this example, and output that
key. We now have room in the heap for another key, so we read one from
that
fit
select the
300
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Input:
67,
21,
12,
47,
5,
16
t_ Front of input string
Memory
Remaining input
21,
67,
21,
67
12
(P
47
16
12
47
16
21
67
47
16
67
47
21
47
-
67
67
Output run
3)
67
12,
16,
12,
21,
16,
12,
47
21,
16,
12,
47
21,
16,
12,
FIGURE 7.27 Example of the principle underlying replacement selection.
key, which has a value of 12, now becomes a
of keys to be sorted into the output run. In fact, since it
is smaller than the other keys in RAM, 12 is the next key that is output. A
new key is read into its place, and the process continues. When the process
is complete, it produces a sorted list of six keys while using only three
the input
list.
member of the
memory
in
set
locations.
In this
happens
The new
example the entire file is created using only one heap, but what
fourth key in the input list is 2 rather than 12? This key arrives
if the
memory
The
too
late to
be output into
its
proper position relative to the other
been written to the output list. Step 3b in the
algorithm handles this case by placing such values in a second heap, to be
included in the next run. Figure 7.28 illustrates how this process works.
keys:
During the
5 has already
first
run,
when
keys are brought in that are too small to be
we mark them with parentheses, indicating
have to be held for the second run.
It
is
interesting to use this example to compare the action of
replacement selection to the procedure we have been using up to this point,
namely that of reading keys into RAM, sorting them, and outputting a run
that is the size of the
space. In this example our input list contains 13
included in the primary heap,
that they
RAM
keys.
A series of successive RAM sorts,
results in five runs.
runs.
The replacement
given only three
Since the disk accesses during a multiway
expense,
replacement selection's
ability
to
fewer, runs can be an important advantage.
Two
questions emerge
memory
locations,
selection procedure results in only
at this point:
merge can be
create longer,
two
major
and therefore
MERGING AS A WAY OF SORTING LARGE
FILES
301
ON DISK
Input:
33,
18,
24,
58,
14,
17,
67,
21,
7,
12,
47,
5,
16
Front of input string
Memory
Remaining input
33,
18,
24,
58,
14,
17,
7,
21,
67,
33,
18,
24,
58,
14,
17,
7,
21,
67
33,
18,
24,
58,
14,
17,
7,
21
33,
18,
24,
58,
14,
17,
33,
18,
24,
58,
14,
17
33,
18,
24,
58,
14
33,
18,
24,
58
12
(P
47
16
12
47
16
67
47
16
67
47
21
67
47
7)
67
(1?)
7)
(14)
(17)
7)
tart building the
33,
18,
24,
33,
18,
24
33,
18
58
33
Output run
3)
67,
12,
16,
12,
21,
16,
12,
47,
21,
16,
12,
47,
21,
16,
12,
second
14
17
14
17
58
24
17
58
24
18
58
24
33
58
33
58
58
14,
58,
17,
14,
18,
17,
14,
17,
14,
24,
18,
33,
24,
18,
17,
14,
33,
24,
18,
17,
14,
FIGURE 7.28 Step-by-step operation of replacement selection working to form two sorted runs.
1.
2.
in memory, how long a run can
placement selection to produce, on the average?
What are the costs of using replacement selection?
Given P locations
Average Run Length for Replacement Selection
first
question
is
that,
on the average, we can expect
P memory locations. Knuth
intuitive argument for why
A
clever
way
to
discovered by E.
a circular track
situation
shown
we
expect re-
The answer
to the
run length of 2P, given
(1973b)^ provides an excellent description of an
this is so:
show
F.
that 2P is indeed the expected run length was
Moore, who compared the situation to a snowplow on
[U.S. Patent 2983904 (1961), Cols. 3-4]. Consider the
[below]; flakes of snow are falling uniformly on
a circular
From Donald Knuth, The Art of Computer Programming, 1973, Addison-Wesley, Reading,
Mass. Pages 254-55 and Figs. 64 and 65. Reprinted with permission.
+
302
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
road, and a lone
snow
snowplow
is
continually clearing the snow.
has been plowed off the road,
it
Once
the
disappears from the system. Points
< x < 1; a flake of
on the road may be designated by real numbers x,
snow falling at position x represents an input record whose key is x, and
the snowplow represents the output of replacement selection. The ground
speed of the snowplow
that
it
is
inversely proportional to the height of the
encounters, and the situation
is
snow
perfectly balanced so that the total
A new run is
amount of snow on the road at all times is exactly
output whenever the plow passes point 0.
P.
After this system has been in operation for awhile,
it is
formed
in the
it
will
approach
speed (because of the circular
snow
is
at
constant height
linearly in front
intuitively clear that
which the snowplow runs at constant
symmetry of the track). This means that the
a stable situation in
when it meets the plow, and the height drops off
as shown [below]. It follows that the volume
of the plow
of snow removed in one revolution (namely the run length)
amount
is
twice the
present at any one time (namely P).
lllHHHIHil
Falling
snow
Future snow
Existing
Total length of the road
snow/=fc(o^==!
|
^(Op~
MERGING AS A WAY OF SORTING LARGE
So,
given
random ordering of
we
keys,
FILES
303
ON DISK
can expect replacement
hold in
form runs that contain about twice as many records as we can
memory at one time. It follows that replacement selection creates
half as
many
selection to
assuming
the
runs as does
same amount of memory. (As we
selection does, in fact, have to
RAM
of
sorts
and the
see in a
make do with
RAM
memory
contents,
have access to
moment, the replacement
less
sort
memory
than does the
sort.)
It is
actually often possible to create runs that are substantially longer
than 2P. In
many
random; the keys
the order of the records
applications,
produce runs
(Consider what would happen
Replacement selection becomes an
ordered input
is
not
wholly
are often already partially in ascending order. In these
cases replacement selection can
2P.
RAM
of
a series
that the replacement selection
if
that,
the input
on the average, exceed
list
is
already sorted.)
especially valuable tool for such partially
files.
The Costs of Using Replacement
Selection
Unfortunately, the no-
free-lunch rule applies to replacement selection, as
it
does to so
many
other
of file structure design. In the worked-by-hand examples we have
looked at up to this point, we have been inputting records into memory one
at a time. We know, in fact, that the cost of seeking out to disk for every
areas
single input record
which means,
is
prohibitive. Instead,
in turn, that
we
operation of replacement selection.
output buffering. This
sorting,
To
is
cost,
we want
Some of it
and the
affect
it
to buffer the input,
of the memory for the
has to be used for input and
are not able to use
all
has on available space for
illustrated in Fig. 7.29.
need for buffering during the replacement
selection step, let's return to our example in which we sort 8 million
records, given a memory area that can hold 10,000 records.
see the effects
of
this
FIGURE 7.29 In-RAM sort versus replacement selection, in
terms of their use of available
heapsort area
(a)
In-RAM
sort: all available
i/o buffer
(b)
Replacement
space used for the
RAM
sort.
heapsort area
selection:
some of
available space
is
used for
i/o.
for sorting operation.
304
CQSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
RAM
For the
records into
sorting
memory
10,000 records
methods such
until
full,
is
it
we
as heapsort,
which simply read
can perform sequential reads of
800 runs have been created. This means that
at a time, until
the sort step requires 1,600 seeks: 800 for reading and 800 for writing.
For replacement selection
we might
use an input/output buffer that can
enough space to hold 7,500
records for the actual replacement selection process. If the I/O buffer holds
hold, for example, 2,500 records, leaving
2,500 records,
so
we
can perform sequential reads of 2,500 records
at a
time,
takes 8,000,000/2,500
3,200 seeks to access all records in the file.
This means that the sort step for replacement selection requires 6,400 seeks:
it
3,200 for reading and 3,200 for writing.
If the records occur in a random key sequence, the average run length
using replacement selection will be 2 X 7,500 = 15,000 records, and there
about 8,000,000/15,000
will be
we
step
534 such runs produced. For the merge
divide the one megabyte of
average of 18.73 records, so
RAM into 534 buffers,
we end up making
which hold an
15,000/18.73
801 seeks
per run, and
801 seeks per run x 534 runs
427,734 seeks altogether.
Table 7.4 compares the access times required to sort the 8 million
records using both a
sort and replacement selection. The table
RAM
800-way merge and two replacement selection examples. The second replacement selection example, which produces runs of
40,000 records while using only 7,500 record storage locations in memory,
assumes that there is already a good deal of sequential ordering within the
includes our
initial
input records.
It
is
clear that,
given randomly distributed input data, replacement
selection can substantially reduce the
number of runs formed. Even though
as many seeks to form the runs,
amount of seeking effort required to merge the runs
more than offsets the extra amount of seeking that is required to form the
runs. And when the original data is assumed to possess enough order to
make the runs 40,000 records long, replacement selection produces less than
replacement selection requires four times
the reduction in the
one third
as
many
seeks as
RAM
sorting.
7.5.7 Replacement Selection Plus Multistep Merging
While these comparisons highlight the advantages of replacement selection
RAM
we would
choose the one-step
merge patterns shown in Table 7.4. We have seen that two-step merges can
result in much better performance than one-step merges. Table 7.5 shows
how these same three sorting schemes compare when two-step merges are
over
sorting,
probably not in
reality
>s
to
c Z
09
Mo
cd
o B
C/5
"O
ed
to
ed
to
b*XH
^
<
OT
XV
CO
_C
CD
o E
.a o
M
GO
C CO
CO
3 =3
CO V|_
o
"O
i_
o CD
o -O
CD
E
c 3
o C
33
E
00
HZ
,i
(ft
0P
5 c
**
CD
fi
O 3
o
^m
03
Z3
cr
CD
o
D
CO
cd
[/)
-C
o
CO
o
u M
_*
o
u
i
C o
3
^_
CD
Z3
cr
CD
i_
CO
CD
E
'-
o
CD
s?
CD
S/3
d
o
o
o o
CD
CO
CO
CD
o c s
N 3 5Cfl
Un
03
CD
o CO
c c
o
CD
CO
E
03 CD
Q. O
E
o
03
(m
5-
a
5/3
Q.
CD
^t
r-
LU
E O
CN
<U
~-
_o
Uh
rs
O
-
t/3
i_
00
ri
<
hh
2^
Q,^o
c2
^
as
gJS
u 12
o C
^-n
305
>s
ed
73 .3
c
CD
E
CD
O
LO
LD
00
"*
CM
00
en
<*
"*
vC
2 =
C/3
TO
p<
A 03 ^
o o .5
PN -w
Q.
CD
Im
mC
HH
TO
o
CO
.>
<
ce
<~
HZ
^
CM
K
CM
ScJS
o
^
'
'
'
.c
*->
<+-
.Q
GO
r
C
C
CO
3
cz^
s S 2
co
O
O
SO O
lO O
CM CM
o
o
o
oo
q
^C CM
r- ^c
00 rH
cn in
CM
oo"
so
L.
o
o
CD
> *
* %
00
Cl
cd
las
X C
x c
8-3
-5
S
CM
LO
00
l<sl
*-*
TO
8
CO
CM
"5
o
2 CM
^
00 J,
Lj
4-1
-a
"^
fi
*5
CD
8|
o c
CD
*~
CD
>
8J
1
c
.3 J o
y. a u-
o
o
O
8
LO
o
LO
o
o
LO
cm'
cm'
TO
n_ -C
C
O
._2
&
TO
CD
fe.2
OO
CD
CD
CO
s o S
tf
c*o
t2
If)
P^
LU
1
<
1
306
J_,
4->
T3
r^
s-
u
ed
2
<
1m
&
Cm
<
o
,
00
'
-S
Cm
o
vi
es:
-MJ
*m
(/>
c/5
S o c
CJ
<~>
^^
-r
'->
^ C
8 OT!
-- u O rt o
Cm <y U
o c3 J
IU -c- S > Cm O
MERGING AS A WAY OF SORTING LARGE
From Table
used.
less in
7.5
every case than
we
it
see that the total
was
FILES
307
ON DISK
number of seeks
dramatically
is
method
for the one-step merges. Clearly, the
used to form runs is not nearly
than one-step, merges.
as
important
as the use
of multistep, rather
Furthermore, since the number of seeks required for the merge steps is
smaller in all cases, while the number of seeks required to form runs
much
have a bigger effect proportionally on the final
total, and the differences between the RAM-sort based method and
replacement selection are diminished.
The differences between the one-step and two-step merges are
exaggerated by the results in Table 7.5, because they don't take into account
the amount of time spent transmitting the data. The two-step merges
and disk two more times
require that we transfer the data between
than do the one-step merges. Table 7.6 shows the results after adding
transmission time to our results. The two-step merges are still better, and
remains the same, the
latter
RAM
replacement selection
still
wins, but the results are
less
dramatic.
7.5.8 Using Two Disk Drives with Replacement Selection
and fortunately, replacement selection offers an opportunity
sort
to save on both transmission and seek times in ways that
methods do not. As usual, this is at a cost, but if sorting time is expensive,
it could well be worth the cost.
Suppose that we have two disk drives available that we can assign the
separate dedicated tasks of reading and writing during replacement
selection. One drive, which contains the original file, does only input, and
Interestingly,
RAM
the other does only output. This has
two very
nice results: (1)
It
input and output can overlap, reducing transmission time by
50%; and
If
(2)
seeking
we have two
to take advantage
two
means
as
that
much
as
virtually eliminated.
is
disks at our disposal,
of them.
We
we should
memory
configure
also configure
as follows:
memory
We
allocate
buffers each for input and output, permitting double buffering, and
allocate the rest
of memory for forming the selection
tree.
This arrangement
might proceed
to take advantage
illustrated in Fig. 7.30.
is
Let's see
of
how
the
merge
sort process
this configuration.
First,
the sort phase.
the heap-sized part of
We
begin by reading in enough records to
the heap. Next, as we
memory, and form
records from the heap into one of the output buffers,
we
fill
up
move
replace those
records with records from one of the input buffers, adjusting the tree in the
usual manner. While
filling the
we empty one
input buffer into the
tree,
we
can be
other one from the input disk. This permits processing and input
c 5
co .5
.2
a C
co
OS
Hhhi
c 5 S c
.22 *S
0)
5 I &
ho* oft
tt
c
2
*
tN
00
J*
03
0)
LO
<T)
C
St
0:
co
CO
>N
03
s?
<N
r
^
4-
^
S
.,
04
c/5
lo
CN
o
LO
o
O
OC
CN G\
i
O
V
O U
'J
o
c
E
o
On
^H
||3
*-H
v.
<N
X o
C
i
CN
LT5
o
in
CN
CN
w
O C
CO
<si
s g
s -8
S ? S
o
h
.,
>*
O O
PC _Q
>-
T o
So.
ji
oilC
too
X R O
j~.
rt
J-c
? o
p5c OC
<u
<u
co
&
a
308
C
O
u
O
u
CO
ft
X>-
-a
>.
>,
CN
in
<N
>^
CO
X ,
v x U
ScSD
*-*
'E
CO
CO
Ctf
-Q .S
4-'
CO
CO
MERGING AS A WAY OF SORTING LARGE
FILES
ON DISK
309
input
buffers
output
buffers
FIGURE 7.30 Memory organization for replacement selection.
to overlap. Similarly, at the
buffers
from the
tree,
we
same time
we
that
are filling
one of the output
can be transmitting the contents of the other to the
way, run selection and output can overlap.
During the merge phase, the output disk becomes the input disk, and
vice versa. Since the runs are all on the same disk, seeking will occur on the
input disk. But output is still sequential, since it goes to a dedicated drive.
Because of the overlapping of so many parts of this procedure, it is
difficult to estimate the amount of time the procedure is likely to take. But
it should be clear that by substantially reducing seeking and transmission
time, we are attacking those parts of the sort merge that are the most costly.
output disk. In
this
7.5.9 More Drives? More Processors?
If
two
Isn't
it
drives can
improve performance, why not
true that the
phase, the faster
course the
we
more
drives
we have
can perform I/O?
number and speed of I/O
to
Up
more?
hold runs during the merge
three, or four, or
to a point this
is
true,
but of
processors must be sufficient to keep
up with the data streaming in and out. And there will also be a point at
which I/O becomes so fast that processing can't keep up with it.
But who is to say that we can use only one processor? A decade ago, it
would have been far-fetched to imagine doing sorting with more than one
processor, but it is very common now to be able to dedicate more than one
processor to a single job. Possibilities include the following:
Mainframe computers, many of which spend a great deal of their
time sorting, commonly come with two or more processors that can
simultaneously work on different parts of the same problem.
Vector and array processors can be programmed to execute certain
kinds of algorithms orders of magnitude faster than scalar processors.
310
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Massively
machines provide thousands, even millions, of
at the same time communicate in complex ways with one another.
Very fast local area networks and communication software make it
relatively easy to parcel out different parts of the same process to
parallel
processors that can operate independently and
several different machines.
It is
these
not appropriate, in
newer
cover in detail the implications of
this text, to
architectures for external sorting.
But just
the past decade in the availability and performance of
have altered the way
many more
we
look
at
as the
changes over
RAM and disk storage
we can expect it to change
new architectures becomes
external sorting,
times as the current generation of
commonplace.
7.5.10 Effects
of
Multiprogramming
of external sorting on disk we are, of course, making tacit
assumptions about the computing environment in which this merging is
In our discussions
We
are assuming, for example, that the merge job is running
environment (no multiprogramming). If, in fact, the
operating system is multiprogrammed, as it normally is, the total time for
the I/O might be longer, as our job waits for other jobs to perform their
taking place.
in
dedicated
I/O.
On
the other hand, one of the reasons for
multiprogramming
is
to
allow the operating system to find ways to increase the efficiency of the
overall system
by overlapping processing and I/O among
the system could be performing I/O for our job while
it
different jobs.
was doing
So
CPU
processing on others, and vice versa, diminishing any delays caused by
overlap of I/O and
CPU
processing within our job.
Effects such as these are very hard to predict, even
when you have
much
information about your system. Only experimentation can determine
what
real
performance will be
like
7.5.11 A Conceptual Toolkit
We can now list many tools
on
busy, multiuser system.
for External Sorting
that can
improve external sorting performance.
should be our goal to add these various tools to our conceptual toolkit for
designing external sorts and to pull them out and use them whenever they
It
are appropriate.
following:
full listing
of our
new
set
of tools would include the
311
SORTING FILES ON TAPE
For
in-RAM
lap input
Use
as
time
With
RAM
as possible.
more
much
It
makes the runs longer and promerge phase.
buffers during the
number of initial runs
is
it
and output with internal processing.
much
vides bigger and/or
If the
forming the original list of
and double buffering, we can over-
sorting, use heapsort for
sorted elements in a run.
is
so large that total seek and rotation
greater than total transmission time, use a multistep
It increases the amount of transmission time but
number of seeks enormously.
merge.
the
Consider using replacement selection for
cially if there is a possibility that the
Use more than one
initial
can decrease
run formation, espe-
runs will be partially ordered.
disk drive and I/O channel so reading and writ-
ing can overlap. This
is
especially true if there are not other users
on
the system.
Keep
mind
the fundamental elements of external sorting and their
and look for ways to take advantage of new architectures and systems, such as parallel processing and high-speed local
area networks.
in
relative costs,
7.6
Sorting Files on Tape
There was a time when it was usually
on tape than on disk, but this is much
is still used in external sorting, and
faster to
perform large external
we would
sorts
now. Nevertheless, tape
less the case
be remiss
if
we
did not
consider sort merge algorithms designed for tape.
There are a large number of approaches to sorting files on tape. After
approximately 100 pages of closely reasoned discussion of different
alternatives for tape sorting, Knuth (1973b) summarizes his analysis in the
following way:
Theorem A.
It is
difficult to decide
which merge pattern
is
best in a
given situation.
Because of the complexity and number of alternative approaches and
way that these alternatives depend so closely on the specific
characteristics of the hardware at a particular computer installation, our
because of the
merely to communicate some of the fundamental issues
For a more comprehensive
discussion of specific alternatives we recommend Knuth's (1973b) work as
objective here
is
associated with tape sorting and merging.
a starting point.
312
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Viewed from
general perspective, the steps involved in sorting on
tape resemble those that
we
1.
Distribute the unsorted
2.
Merge
discussed with regard to sorting on disk:
file
into sorted runs;
the runs into a single sorted
Replacement selection
is
and
file.
almost always
good choice
as a
method
for
You will remember that the
when we are working on disk is that
creating the initial runs during a tape sort.
problem with replacement
the
amount of seeking
selection
required during run creation
more than
offsets the
advantage of creating longer runs. This seeking problem disappears when
the input is from tape. So, for a tape-to-tape sort, it is almost always
advisable to take advantage of the longer runs created by replacement
selection.
7.6.1 The Balanced Merge
Given
that the question
of
how
create the initial runs has
to
such
merging process that we
encounter all of the choices and complexities implied by Knuth's tonguein-cheek theorem. These choices begin with the question of how to distribute
the initial runs on tape and extend into questions about the process of
merging from this initial distribution. Let's look at some examples to show
what we mean.
Suppose we have a file that, after the sort phase, has been divided into
10 runs. We look at a number of different methods for merging these runs
on tape, assuming that our computer system has four tape drives. Since the
initial, unsorted file is read from one of the drives, we have the choice of
initially distributing the 10 runs on two or three of the other drives. We
begin with a method called two-way balanced merging, which requires that the
initial distribution be on two drives, and that at each step of the merge,
except the last, the output be distributed on two drives. Balanced merging
is the simplest tape merging algorithm that we look at; it is also, as you will
straightforward answer,
it is
clear that
it is
in the
see, the slowest.
The balanced merge proceeds according
to the pattern illustrated in Fig.
7.31.
This balanced merge process
is
The numbers
expressed in an alternate,
form in Fig. 7.32.
measured in terms of the number of
run. For example, in step
By
all
initial
runs included in each merged
the input runs consist of a single initial run.
At the start of
Tl contains one run consisting of four initial runs
run consisting of two initial runs. This method of illustration
step 2 the input runs each consist of a pair of initial runs.
step 3,
tape drive
followed by
more compact
inside the table are the run lengths
SORTING FILES ON TAPE
Tape
Step
313
Contains runs
Tl
Rl
R3
R5
R7
T2
T3
T4
R2
R4
R6
R8
R1-R2
R3-R4
R5-R6
R7-R8
R9-R10
Tl
T2
R1-R4
R5-R8
R9-R10
T3
T4
R9
RIO
Tl
Step 2
Step 3
T2
T3
T4
Tl
Step 4
Step 5
T2
T3
T4
R1-R8
R9-R10
R1-R10
Tl
T2
T3
T4
FIGURE 7.31
Balanced four-tape merge
more
grow
shows
of
10
runs.
way some of
combine and
one run that is copied
again and again stays at length 2 until the end. The form used in this
illustration is used throughout the following discussions on tape merging.
Since there is no seeking, the cost associated with balanced merging on
tape is measured in terms of how much time is spent transmitting the data.
In the example, we passed over all of the data four times during the merge
phase. In general, given some number of initial runs, how many passes over
the data will a two-way balanced merge take? That is, if we start with
runs, how many passes are required to reduce the number of runs to 1?
clearly
the
the intermediate runs
into runs of lengths 2, 4, and 8, whereas the
314
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
XI
T2
T3
T4
11111
11111
Step 2
Step 3
4 2
Step 4
Step 5
Step
Merge ten runs
22222
Merge ten runs
4
10
Merge ten runs
Merge ten runs
FIGURE 7.32 Balanced four-tape merge of 10 runs expressed
in
more compact
table notation.
Since each step combines
the
number
two
runs, the
for the previous step. If p
number of runs after each step is half
the number of passes, then we can
is
express this relationship as follows:
(Vif
from which
it
can be
shown
N=
N<
1,
that
P =
In our simple example,
Hog, N~l
10, so four passes
over the data were required.
Recall that for our partially sorted 800-megabyte
flog 2 200 "| = 8 passes are required for
file
there
were 200 runs, so
balanced merge. If reading and
writing overlap perfectly, each pass takes about 11 minutes," so the total
1"
time
is 1
hour, 28 minutes. This time
merges, even
when
a single
outweigh the savings
is
disk drive
not competitive with our disk-based
is
used.
The transmission times
far
in seek times.
7.6.2 The /(-way Balanced Merge
If
we want
to
tells
us that
improve on
this
approach,
it is
clear that
we must
number of passes over the data. A quick look at
we can reduce the number of passes by increasing
to reduce the
find
ways
the formula
the order of
assumes the 6,250 bpi tape used in the examples in Chapter 3. If the transports speed
200 inches per second, the transmission rate is 1,250 Kbytes per second, assuming no
blocking. At this rate an 800-megabyte file takes 640 seconds, or 10.67 minutes to read.
""This
is
SORTING FILES ON TAPE
each merge. Suppose, for instance, that
we have
315
20 tape drives, 10 for input
and 10 for output, at each step. Since each step combines 10 runs, the
number of runs
after
each step
is
one tenth the number for the previous
step.
we have
Hence,
(Vxof
N<
and
p = Rogio
In general,
at
~l
A k-way balanced merge is one in which the order of the merge
each step (except possibly the
last)
is
k.
required for a k-way balanced merge with
Hence, the number of passes
N initial
runs
is
v = r~iog N~i.
fe
10-way balanced merge of our 800-megabyte
file with 200 runs,
200 1 = 3, so three passes are required. The best estimated time now
is reduced to a more respectable 42 minutes. Of course, the cost is quite
high: We must keep 20 working tape drives on hand for the merge.
For
|~logio
7.6.3 Multiphase Merges
The balanced merging algorithm has the advantage of being very simple; it
easy to write a program to perform this algorithm. Unfortunately, one
is
reason
it
is
simple
is
that
it
is
"dumb" and cannot take advantage of
how we can improve on it.
when we merge the extra run with empty
opportunities to save work. Let's see
We
can begin by noting that
runs in steps 3 and
4,
we
don't really accomplish anything. Figure 7.33
shows how we can dramatically reduce the amount of work that has to be
done by simply not copying the extra run during step 3. Instead of merging
this run with a dummy run, we simply stop tape T3 where it is. Tapes Tl
and T2 now each contains a single run made up of four of the initial runs.
We rewind all the tapes but T3 and then perform a three-way merge of the
runs on tapes Tl, T2, and T3, writing the final result on T4. Adding this
intelligence to the merging procedure reduces the number of initial runs that
must be read and written from 40 down to 28.
The example in Fig. 7.33 clearly indicates that there are ways to
improve on the performance of balanced merging. It is important to be able
to state, in general terms, what it is about this second merging pattern that
saves work:
We
use a higher-order merge. In place of
use one three-way merge.
two two-way merges, we
316
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
T2
Tl
Step
11111
Step 2
Step 3
T3
T4
2 2 2
2 2
Merge
ten runs
Merge
eight runs
Merge ten runs
Step 4
10
FIGURE 7.33 Modification of balanced four-tape merge that does not rewind
between steps 2 and 3 to avoid copying runs.
We
extend the merging of runs from one tape over several
we merge some
Specifically,
in step 4.
We
could say that
of the runs from T3
we merge
the runs
steps.
and some
in step 3
from T3
in
two
phases.
These
ideas, the use
of higher-order merge patterns and the merging of
runs from a tape in phases, are the basis for two well-known approaches to
merging called polyphase merging and cascade
merges share the following characteristics:
The
a
initial
distribution of runs
J 1-way merge, where J
The
is
is
the
such that
merging.
at least
Figure 7.34 illustrates
the initial
number of available
distribution of the runs across the tapes
ten contain different
In general,
is
merge
initial
such that the tapes of-
numbers of runs.
how
polyphase merge can be used to merge 10
runs that must be read and written from 40 (for
two-way merge)
to 25.
is
tape drives.
runs distributed on four tape drives. This merge pattern reduces the
of
these
easy to see that this reduction
It is
is
number
balanced
consequence
of the use of several three-way merges in place of two-way merges. It
should also be clear that the ability to do these operations as three-way
merges is related to the uneven nature of the initial distribution. Consider,
for example,
what happens
than 5-3-2.
We
T3, but
this also clears all the
Tl. Obviously,
second
if
the initial distribution of runs
is
4-3-3
rather
can perform three three-way merges to open up space on
we
runs off of T2 and leaves only a single run on
are not able to
perform another three-way merge
as a
step.
Several questions arise at this point:
1.
How
does one choose an
efficient
merge
pattern?
initial
distribution that leads readily to an
SORTING FILES ON TAPE
T2
Tl
1
11111
Step 2
..111
Step 3
...
Step 4
....
Step
T4
33
10
Step 5
T3
Merge
six
Merge
five
317
runs
runs
Merge four runs
Merge ten runs
FIGURE 7.34 Polyphase four-tape merge of 10 runs.
2.
Are there algorithmic descriptions of the merge
patterns, given an
initial distribution?
3.
N runs
and J tape drives, is there some way to compute the
merging performance so we have a yardstick against which
compare the performance of any specific algorithm?
Given
optimal
to
beyond the scope of this text; in
answer to question 3 requires a more mathematical approach
to the problem than the one we have taken here. Readers wanting more
than an intuitive understanding of how to set up initial distributions should
Precise answers to these questions are
particular, the
consult
Knuth
(1973b).
7.6.4 Tapes versus Disks
for External Sorting
RAM
was considered a substantial amount of
of
any single job, and extra disk drives were very
costly. This meant that many of the disk sorting techniques to decrease seeking that we have seen were not available to us or were very
decade ago 100
memory
to allocate to
limited.
we want
our 800-megabyte file, and
available, instead of one megabyte. The
there is only 100 K of
approach that we used for allocating memory for replacement selection
would provide 25 K for buffering, and 75 K for our selection tree. From this
Suppose, for instance, that
to sort
RAM
we
can expect 5,334 runs of 1,500 records each, versus 534 when there is a
RAM. For a one-step merge, this 10-fold increase in the
megabyte of
number of runs
results in a 100-fold increase in the
took three hours with one megabyte of memory
for the seeks!
No wonder
no seeking, were
tapes,
preferred.
which
number of seeks. What
now
takes 300 hours, just
are basically sequential
and require
31
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
now
RAM
much more readily available. Runs can be longer and
much less of a problem. Transmission time is now
more important. The best way to decrease transmission time is to reduce
the number of passes over the data, and we can do this by increasing the
But
is
fewer, and seeks are
order of the merge. Since disks are random-access devices, very large order
merges can be performed, even if there is only one drive. Tapes, however,
are not random-access devices; we need an extra tape drive for every extra
run we want to merge. Unless a large number of drives is available, we can
only perform low-order merges, and that means large numbers of passes
over the data. Disks are better.
7.7
Sort-Merge Packages
Many
utility programs are available for users who need to sort
Often the programs have enough intelligence to choose from one
of several strategies, depending on the nature of the data to be sorted and the
available system configuration. They also often allow users to exert some
control (if they want it) over the organization of data and strategies used.
Consequently, even if you are using a commercial sort package rather than
designing your own sorting procedure, it is useful to be familiar with the
variety of different ways to design merge sorts. It is especially important to
have a good general understanding of the most important factors and
large
very good
files.
trade-offs influencing performance.
7.8
Sorting and Cosequential Processing
UNIX
has a
number of utilities
also has sorting routines, but
for
in
UNIX
performing cosequential processing. It
at the level of sophistication that you
nothing
find in production sort-merge packages. In the following discussion
introduce
some of
these
utilities.
For
full
details,
consult the
we
UNIX
documentation.
7.8.1 Sorting and Merging
Because
UNIX
sorting of large
is
in
UNIX
not an environment in which one expects to do frequent
files
of the type
we
discuss in this chapter, sophisticated
sort-merge packages are not generally available on UNIX systems. Still, the
sort routines you find in UNIX are quick and flexible and quite adequate for
the types of applications that are common in a UNIX environment. We can
SORTING AND COSEQUENTIAL PROCESSING
divide
UNIX
two
sorting into
IN
319
UNIX
command, and
categories: (1) the sort
(2)
callable sorting routines.
UNIX
The
Command
sort
options, but the simplest one
(A
lexical order.
character
'\n'.)
command
sorted
one
is
line
line
By
is
sort
command
to sort the lines in an
default the sort utility takes
named on
has
ASCII
many
file in
different
ascending
any sequence of characters ending with the new-line
and writes the sorted
too large to
file is
is
The
fit
in
RAM,
file
sort
its
input
file
name from
to standard output. If the
performs
merge
sort. If
the input line, sort sorts and merges the
file
the
to be
more than
files.
As a simple example, suppose we have an ASCII file called team with
names of members of a basketball team, together with their classes and their
scoring averages:
Jean Smith Senior 7.8
Chris Mason Junior 9.6
Pat Jones Junior 3.2
Leslie Brown Sophomore 18.2
Pat Jones Freshman 11.4
To
sort the
file,
enter
$
sort team
Chris Mason Junior 9.6
Jean Smith Senior 7.8
Leslie Brown Sophomore 18.2
Pat Jones Freshman 11.4
Pat Jones Junior 3.2
Notice that by default sort considers an entire line as the sort key.
Hence, of the two players named "Pat Jones," the freshman occurs first in
"Freshman" is lexically smaller than "Junior." The
assumption that the key is an entire line can be overridden by sorting on
specified key fields. For sort a key field is assumed to be any sequence of
characters delimited by spaces or tabs. You can indicate which key fields to
use for sorting by giving their positions:
the output because
+po5
where posl
tells which
of the
$
-pos2
tells
how many
sort
file
+1
If pos2
is
before starting the key, and pos2
omitted, the key extends to the end
-2 team
team to be sorted according to the
form of posl and pos2
to start a
fields to skip
end with.
Hence, entering
field to
line.
causes the
a
key with.)
that allows
you
last
names. (There
is
also
to specify the character within a field
320
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
The following options, among
ASCII ordering used by sort:
-d
Use "dictionary"
ordering:
others, allow
Only
you
to override the default
and blanks are
letters, digits,
signifi-
cant in comparisons.
-f
"Fold" lowercase letters into uppercase. (This
defined in Chapter 4.)
the canonical
is
form
that
we
-r
"Reverse" the sense of comparison: Sort
Notice that
and within
in
descending ASCII order.
compares groups of
Chapter 4, records
are lines, and fields are groups of characters delimited by white space. This
is consistent with the most common UNIX view of fields and records
sort sorts lines,
characters delimited
within
The
UNIX
text
by white
it
files.
Library Routine
qsort
lines
space. In the language of
UNIX
The
library routine qsort (
general sorting routine. Given a table of data, qsort(
the table in place.
table could be the contents
where the elements of the
table are
its
nel,
int
of a
is
sorts the elements in
file,
loaded into
records. In C, qsort ()
RAM,
defined as
is
follows:
qsortCchar *ba5e,
The argument
base
is
int
a pointer to the
elements in the table; and width
argument, compar(
),
is
the
is
name of
width,
int
base of the data, nel
*compar (
is
the
) )
number of
The last
the size of each element.
a
user-supplied comparison function
) must have two parameters,
which are pointers to elements that are to be compared. When qsort( ) needs
to compare two elements, it passes to comparf ) pointers to these elements,
and compar( ) compares them, returning an integer that is less than, equal to,
or greater than zero, depending on whether the first argument is considered
that qsort(
uses to
compare keys. Compar(
equal to, or greater than the second argument. A full explanation
of how to use qsort( ) is beyond the scope of this text. Consult the UNIX
documentation for details.
less than,
7.8.2 Cosequential Processing
UNIX
utility,
UNIX
number of utilities for cosequential
when used to merge files, is one example.
provides a
introduce three others:
cmp
Utilities in
difj]
The
sort
In this section
we
processing.
cmp, and cotnm.
Suppose you find in your computer that you have two team files,
one called team and the other called my team. You think that the two files are
the same, but you are not sure. You can use the command cmp to find out.
SORTING AND COSEQUENTIAL PROCESSING
IN
321
UNIX
cmp compares two files. If they differ, it prints the byte and line number
where they differ; otherwise it does nothing. If all of one file is identical to
the first part of another, it reports that end-of-file was reached on the
shorter file before any differences were found.
For example, suppose the
team and myteam have the following
file
contents:
team
myteam
Jean Smith Senior 7.8
Chris Mason Junior 9.6
Pat Jones Junior 3.2
Leslie Brown Sophomore 18.2
Pat Jones Freshman 11.4
Jean Smith Senior 7.8
Stacy Fox Senior 1.6
Chris Mason Junior 9.6
Pat Jones Junior 5.2
Leslie Brown Sophomore 18.2
Pat Jones Freshman 11.4
cmp
tells
you where they
differ:
cmp team myteam
team myteam differ:
char 23 line
files on a byte-by-byte basis until it finds a
makes no assumptions about fields or records. It works with
Since cmp simply compares
difference,
it
both text and nontext
files.
useful if you just want to know if two files are different, but it
you much about how they differ. The command diff gives fuller
information, diff telh what lines must be changed in two files to bring them
cmp
diff
doesn't
is
tell
into agreement. For example:
team myteam
diff
1a2
>
Stacy Fox Senior
.6
3c4
<
Pat Jones Junior 3.2
1
Jones Junior 5.2
Pat
>
The "la2"
indicates that after line
in the first
file,
we need
to add line 2
make them agree. This is followed by the line from
the second file that would need to be added. The "3c4" indicates that we
need to change line 3 in the first file to make it look like line 4 in the second
from
file.
the second
This
leading
is
"<"
indicates that
file
to
followed by
a listing
of the two differing
indicates that the line
it is
from the second
is
from the
first
lines,
file,
where the
and the
">"
file.
One other indicator that
a line in the first file
means
could appear in d(ffoutput is "d", meaning that
has been deleted in the second file. For example, "12dl5"
that line 12 in the first
file
appears to have been deleted from being
322
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
right after line 15 in the second
work with
lines
of text.
file.
Notice that
diff,
like sort,
is
designed to
would not work well with non-ASCII
It
text
files.
comm
Whereas diff tells what is different about two files, comm compares
which must be ordered in ASCII collating sequence, to see what
they have in common. The syntax for comm is the following:
two
files,
comm [-123] filel
file2
comm produces three columns of output. Column 1 lists
filel only; column 2 lists lines in file2 only, and column
in
both
files.
the lines that are in
lists lines
that are
For example,
sort team >ts
sort myteam >ms
comm ts ms
Chris Mason Junior 9.6
Jean Smith Senior 7.8
$
$
Leslie Brown Sophomore 18.2
Pat Jones Freshman 11.4
Pat
Jones Junior 3.2
Pat Jones Junior 5.2
Stacy Fox Senior
.6
1
Selecting any of the flags
you
1, 2,
or 3 allows you to print only those columns
are interested in.
The
sort, diff,
representative of
comm, and emp commands (and the
what
is
available in
UNIX
qsort() function) are
for sorting
and cosequential
As we have said, they have many useful options
cover and that you will be interested in reading about.
processing.
that
SUMMARY
In the first half
and apply
merge
it
of this chapter,
to
sorting.
we
develop
two common problems
cosequential processing
updating
In the second half of the chapter
we
model
general ledger and
identify the
most
important factors influencing performance in merge-sorting operations and
suggest some strategies for achieving good performance.
The cosequential processing model can be applied to problems that
involve operations such as matching and merging (and combinations of
these)
on two or more sorted input
files.
We
begin the chapter by
we
don't
SUMMARY
illustrating the use
common
to
to
two
of the model to perform
and
lists,
merge of two
simple match of the elements
lists.
perform these two operations embody
The procedures we develop
all
the basic elements of the
model.
most complete form, the model depends on
certain assumptions
enumerate these assumptions in our
formal description of the model. Given these assumptions, we can describe
the processing components of the model.
The real value of the cosequential model is that it can be adapted to
more substantial problems than simple matches or merges without too
In
its
about the data
much
in the input files.
alteration.
We
We
illustrate this
by using
the
model
to design a general
ledger accounting program.
model involve only two
multiway merge to show how the
All of our early sample applications of the
input
We next adapt the
files.
model
to a
model might be extended to deal with more than two input lists. The
problem of finding the minimum key value during each pass through the
main loop becomes more complex as the number of input files increases. Its
solution involves replacing the three-way selection statement with either a
multiway
selection or a procedure that keeps current keys in a
that can be processed
We
more
see that the application
well for small values of
it is
more
k,
list
structure
conveniently.
of the model to fe-way merging performs
but that for values of k greater than eight or
efficient to find the
minimum key
so,
value by means of a selection
tree.
After discussing multiway merging,
we shift our attention
to a
problem
encountered in a previous chapter
how to sort large files. We
begin with files that are small enough to fit into
and introduce an
efficient sorting algorithm, heapsort, which makes it possible to overlap I/O
with the sorting process.
we
that
RAM
The generally accepted
is some form of merge
sorts
1.
Break the
file
nal sorting
2.
Merge
into
solution
sort.
when
merge
two or more
a file
is
too large for
sort involves
two
in-RAM
steps:
sorted subfiles, or runs, using inter-
methods; and
the runs.
to keep every run in a separate file so we can perform
one pass through the runs. Unfortunately, practical
considerations sometimes make it difficult to do this effectively.
The critical elements when merging many files on disk are seek and
rotational delay times and transmission times. These times depend largely
on two interrelated factors: the number of different runs being merged and
Ideally,
the
we would like
merge
step with
323
324
the
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
amount of internal
buffer space available to hold parts of the runs.
We
can reduce seek and rotational delay times in two ways:
By performing the merge in more than one step;
By increasing the sizes of the initial sorted runs.
In
and/or
both cases, the order of each merge step can be reduced, increasing the
of the internal buffers and allowing more data to be processed per seek.
sizes
Looking
at the first alternative,
means
that
total data
we need
we
see
how
number of seeks
several steps can decrease the
performing the merge
dramatically, though
it
in
also
through the data more than once (increasing
to read
transmission time).
The second
through use of an algorithm called
Replacement selection, which can be implemented
using the selection tree mentioned earlier, involves selecting the key from
memory that has the lowest value, outputting that key, and replacing it
with a new key from the input list.
With randomly organized files, replacement selection can be expected
to produce runs twice as long as the number of internal storage locations
available for performing the algorithms. Although this represents a major
step toward decreasing the number of runs needing to be merged, it carries
with it an additional cost. The need for a large buffer for performing the
replacement selection operation leaves relatively little space for the I/O
buffer, which means that many more seeks are involved in forming the runs
than are needed when the sort step uses an in-RAM sort. If we compare the
alternative
realized
is
replacement selection.
total
number of seeks
required by the
two
different approaches,
replacement selection can actually require more seeks;
tially better
only
when
Next we turn our
there
is a
it
we find
great deal of order in the initial
on
that
performs substanfile.
I/O with
tapes does not involve seeking, the problems and solutions associated with
tape sorting can differ from those associated with disk sorting, although the
fundamental goal of working with fewer, longer runs remains. With tape
sorting, the primary measure of performance is the number of times each
record must be transmitted. (Other factors, such as tape rewind time, can
also be important, but
attention to
we do
file
sorting
tapes. Since file
not consider them here.)
Since tapes do not require seeking, replacement selection
always
good choice
for creating initial runs. Since the
available to hold run files
the
files
on the
tapes. In
is
limited, the next question
most
cases,
each of several tapes, reserving one or
it is
is
how
drives
to distribute
necessary to put several runs on
more other
tapes for the results. This
generally leads to merges of several steps, with the total
being decreased after each merge step.
almost
is
number of
Two
number of runs
approaches to doing
this are
KEY TERMS
balanced merges
and multiphase merges. In
tapes contain approximately the
number of output
a fe-way
balanced merge,
same number of
all
input
runs, there are the
same
tapes as there are input tapes, and the input tapes are read
through entirely during each
of k after each step.
step.
The number of runs
decreased by a
is
factor
multiphase merge (such
as
polyphase merge or
requires that the runs initially be distributed unevenly
the available tapes. This increases the order of the
cascade merge)
among
merge and
but one of
all
as a result
can
number of times each record has to be read. It turns out that the
distribution of runs among the first set of input tapes has a major
on the number of times each record has to be read.
decrease the
initial
effect
Next,
available
we
discuss briefly the existence of sort-merge
on most large systems and can be very
conclude the chapter with
a listing
of
UNIX
utilities,
flexible
utilities
and
which
effective.
are
We
used for sorting and
cosequential processing.
KEY TERMS
Balanced merge. A multistep merging technique that uses the same
number of input devices as output devices. A two-way balanced
merge uses two input tapes, each with approximately the same number of runs on it, and produces two output tapes, each with approximately half as many runs as the input tapes. A balanced merge is
suitable for merge sorting with tapes, though it is not generally the
best method (see multiphase merging).
cmp. A UNIX utility for determining whether two files are identical.
Given two files, it reports the first byte where the two files differ, if
they
comtn.
differ.
A UNIX
utility for
common. Given two
determining what
files, it
the lines that are in the
lines
two
files
have
reports the lines they have in
first file
and not
in the second,
in
common,
and the
lines
second file and not in the first.
Cosequential operations. Operations applied to problems that involve
the performance of union, intersection, and more complex set operations on two or more sorted- input files to produce one or more output files built from some combination of the elements of the input
files. Cosequential operations commonly occur in matching, merging, and file-updating problems.
that are in the
325
326
diff.
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
A UNIX
determining
utility for
two files. It
make it like
all
the lines that differ
between
reports the lines that need to be added to the
the second, the lines that need to be deleted
first file to
from the
file to make it like the first, and the lines that need to be
changed in the first file to make it like the second.
heapsort. A sorting algorithm especially well suited for sorting large
second
files
that
fit
in
RAM
because
variation of heapsort
is
its
execution can overlap with I/O.
used to obtain longer runs in the replacement
selection algorithm.
HIGH_VALUE. A
value used in the cosequential model that
than any possible key value.
By
assigning
HIGH_VALUE
is
greater
as the
current key value for files for which an end-of-file condition has
been encountered, extra logic for dealing with end-of-file conditions
can be simplified.
fc-way merge. A merge in which k input files are merged to produce
one output
file.
LOW_VALUE. A
value used in the cosequential model that is less than
any possible key value. By assigning LOW_VALUE as the previous
key value during initialization, the need for certain other special
start-up code is eliminated.
Match. The process of forming a sorted output file consisting of all the
elements common to two or more sorted input files.
Merge. The process of forming a sorted output file that consists
of the union of the elements from two or more sorted input
files.
Multiphase merge.
merge
which the initial distrimerge is a J 1 -way
merge (J is the number of available tape drives), and in which the
distribution of runs across the tapes is such that the merge performs
bution of runs
efficiently at
is
every
Multistep merge.
multistep tape
such that
at least
the
in
initial
step. (See polyphase merge.)
merge
in
which not
all
runs are merged in one
of runs are merged separately, each set producing one long run consisting of the records from all of its runs.
These new, longer sets are then merged, either all together or in several sets. After each step, the number of runs is decreased and the
step. Rather, several sets
length of the runs
is
increased.
run consisting of the entire
file.
The output of the
is
is
a single
(Be careful not to confuse our use of
the term multistep merge with multiphase merge.)
merge
final step
Although
multistep
more time-consuming than is a single-step
can involve much less seeking when performed on a disk,
theoretically
merge, it
and it may be the only reasonable way
the number of tape drives is limited.
to
perform
merge on tape
if
KEY TERMS
Order of a merge. The number of different
or runs, being
files,
merged. For example, the 100 is the order of a 100-way merge.
Polyphase merge. A multiphase merge in which, ideally, the merge
order
qsort.
is
maximized
ploys
every step.
at
general-purpose
UNIX
library routine for sorting files that
em-
user-defined comparison function.
Replacement
selection.
method of creating initial runs based on the
from memory whose key has the
idea of always selecting the record
lowest value, outputting that record, and then replacing
with
new
record from the input
list.
When new
it
memory
in
records are
brought in whose keys are greater than those of the most recently
output records, they eventually become part of the run being created. When new records have keys that are less than those of the
most recently output records, they are held over for the next run.
Replacement selection generally produces runs that are substantially
longer than runs that can be created by in-RAM sorts, and hence can
help improve performance in merge sorting. When using replacement selection with merge sorts on disk, however, one must be careful that the extra seeking required for replacement selection does not
outweigh the benefits of having longer runs to merge.
Run. A sorted subset of a file resulting from the sort step of a sort
merge or one of the steps of a multistep merge.
Selection tree. A binary tree in which each higher-level node represents
the winner of the comparison between the two descendent keys. The
minimum (or maximum) value in a selection tree is always at the
root node,
making
ing several
lists.
the selection tree a
It is
also a
good
key structure
in
data structure for
replacement selection
algorithms, which can be used for producing long runs for
sorts.
(Tournament
sort,
an internal
merg-
sort, is also
merge
based on the use of
selection tree.)
Sequence checking. Checking that records in
order. It is recommended that all files used
a file are in the
expected
in a cosequential opera-
tion be sequence checked.
A UNIX utility for sorting and merging files.
Synchronization loop. The main loop in the cosequential processing
model. A primary feature of the model is to do all synchronization
sort.
within
a single
loop, rather than in multiple nested loops.
keep the main synchronization loop
objective
is
to
This
is
done by
ble.
as
second
simple as possi-
restricting the operations that occur within the
loop to those that involve current keys, and by relegating
special logic as possible (such as error checking
ing) to subprocedures.
as
much
and end-of-file check-
327
328
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
Theorem
(Knuth).
It is
difficult to decide
which merge pattern
is
best in a given situation.
EXERCISES
1.
Write an output procedure to go with the procedures described in
7. 1 for doing cosequential matching. As a defensive measure, it is a
section
good idea to have the output procedure do sequence checking
manner as the input procedure does.
2.
Consider the cosequential
in the
initialization routine in Fig. 7.4. If
same
PREV_1
LOW_VALUE in this routine, how would
input() have to be changed? How would this affect the adaptability of input ()
PREV_2 were
and
not
set to
for use in other cosequential processing algorithms?
Consider the cosequential merge procedures described in section 7.1.
Comment on how they handle the following situations. If they do not
correctly handle a situation, indicate how they might be altered to do so.
a. List 1 empty and List 2 not empty
b. List 1 not empty and List 2 empty
c. List 1 empty and List 2 empty
3.
4.
it
In the ledger procedure
example
also updates the ledger file
5.
Use
the /e-way
in section 7.2,
with the
merge example
new
modify the procedure so
account balances for the month.
as the basis for a
procedure that
is
/e-way match.
6.
are
Figure 7. 17 shows a loop for doing a /e-way merge, assuming that there
no duplicate names. If duplicate names are allowed, one could add to the
procedure a facility for keeping
names. Alter the procedure to do
7.
In section 7.3,
keys
at
two methods
list
of subscripts of duplicate lowest
are presented for choosing the lowest of/?
each step in a /e-way merge:
Compare
this.
a linear
search and use of a selection tree.
in terms of numbers of
comparisons for k = 2, 4, 8, 16, 32, and 100. Why do you think the linear
approach is recommended for values of k less than 8?
8.
the performances of the
two approaches
Suppose you have 8 megabytes of
800,000-record
file
How long does it take to sort the
rithm described in section 7.5.1?
a.
RAM
available for sorting the
described in section 7.5.1.
file
using the merge sort algo-
EXERCISES
b.
How
long does
take to sort the
it
file
using the keysort algorithm
described in Chapter 5?
c.
Why
work
will keysort not
if
there
is
one megabyte of
RAM
available for the sorting phase?
9.
How much
seek time
is
the one described in section
the
amount of available
10.
Performance
7.5 if the
why
is
50 msec and
500 K? 100 K?
is
often measured in terms of the
is
the
number of comparisons
measuring performance in sorting large
11. In
one-step merge such as
time for an average seek
internal buffer space
in sorting
comparisons. Explain
required to perform
is
number of
not adequate for
files.
our computations involving the merge
sorts,
we made
the simpli-
fying assumption that only one seek and one rotational delay are required
for
any single sequential
access. If this
were not the
case, a great deal
more
time would be required to perform I/O. For example, for the 80-megabyte
file
used in the example in section 7.5.1, for the input step of the sort phase
all records into
for sorting and forming runs"), each
RAM
("reading
individual run could require
many
accesses.
extent size for our hypothetical drive
track),
and that
all files
separately (one seek
a.
b.
c.
Now
let's
assume
that the
20,000 bytes (approximately one
are stored in track-sized blocks that
must be accessed
and one rotational delay per block).
How many seeks does step now require?
How long do steps 1, 2, 3, and 4 now take?
How does increasing the file size by a factor
total
12.
is
of 10
now
affect the
time required for the merge sort?
Derive two formulas for the number of seeks required to perform the
step of a one-step /e-way sort merge of a file with r records divided
merge
into k runs,
If
where the amount of available
an internal sort
of each run is M,
the length of each run
Assume
RAM
is
equivalent to
M records.
used for the sort phase, you can assume that the length
but if replacement selection is used, you can assume that
is
is
about 2M.
Why?
system with four separately addressable disk drives,
hundred megabytes. Assume that the
80-megabyte file described in section 7.5 is already on one of the drives.
Design a sorting procedure for this sample file that uses the separate drives
to minimize the amount of seeking required. Assume that the final sorted
file is written off to tape and that buffering for this tape output is handled
invisibly by the operating system. Is there any advantage to be gained by
using replacement selection?
13.
a quiet
each of which
is
able to hold several
329
330
COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES
14. Use replacement selection to
assuming P = 4.
a. 23 29 5 17 9 55 41 3 51
b. 3 5 9 11 17 18 23 24 29
c. 55 51 47 41 33 29 24 23
Suppose you have
15.
so 10 cylinders
produce runs from the following
33 18 24 11 47
33 41 47 51 55
18 17 11 9 5 3
a disk drive that has 10
may be
files,
read/write heads per surface,
accessed at any one time without having to
you could control
on disk, how might you be able to
a sort merge?
actuator arm. If
move
the
the physical organization of runs stored
exploit this arrangement in performing
Assume we need
16.
patterns starting
c.
8-4-2
7-4-3
6-5-3
d.
5-5-4.
a.
b.
to merge 14 runs on four tape drives. Develop merge
from each of these initial distributions:
A four-tape polyphase merge is to be performed to
17.
25 16 45 29 38 23 50 22 19 43 30
runs are of length
following runs
(a
1.
After
1 1
27 47. The original
initial sorting,
1:
24/36/13/25
Tape
2:
16
45
29
38
23
50
Tape
3:
22
19
43
30
11
27
b.
1,
24 36 13
list is
2,
on tape
4. Initial
and 3 contain the
slash separates runs):
Tape
a.
tapes
list
sort the
47
Show the contents of tape 4 after one merge phase.
Show the contents of all four tapes after the second and
fourth
phases.
c.
Comment on
the appropriateness of the original
4-6-7
distribu-
tion for performing a polyphase merge.
18. Obtain a copy of the manual for one or more commercially available
sort-merge packages. Identify the different kinds of choices available to
users of the packages. Relate the options to the performance issues discussed
in this chapter.
Programming Exercises
19.
in
20.
in
Implement the cosequential match procedures described
Implement the cosequential merge procedures described
in section 7.1
or Pascal.
or Pascal.
in section 7.
FURTHER READINGS
21.
Implement
complete program corresponding to the solution to the
general ledger problem presented in section 7.2.
22.
Design and implement
a.
Examine
program to do the following:
two sorted files Ml and M2.
the contents of
COMMON
containing a copy of records
Produce a third file
from the original two files that are identical.
c. Produce a fourth file DIFF that contains all records from the two
b.
files
that are not identical.
FURTHER READINGS
The
two separate topics: the
model for cosequential processing, and discussion of external
merging procedures on tape and disk. Although most file processing texts discuss
cosequential processing, they usually do it in the context of specific applications,
rather than presenting a general model that can be adapted to a variety of
applications. We found this useful and flexible model through Dr. James VanDoren,
who developed this form of the model himself for presentation in the file structures
course that he teaches. We are not aware of any discussion of the cosequential model
subject matter treated in this chapter can be divided into
presentation of a
elsewhere in the literature.
of work has been done toward developing simple and effective
file updating, which is an important instance of
cosequential processing. The results deal with some of the same problems the
cosequential model deals with, and some of the solutions are similar. See Levy
(1982) and Dwyer (1981) for more.
Unlike cosequential processing, external sorting is a topic that is covered
widely in the literature. The most complete discussion of the subject, by far, is in
Quite
a bit
algorithms to do sequential
Knuth
some
Knuth
book in
(1973b). Students interested in the topic of external sorting must, at
point, familiarize themselves with Knuth's definitive
also describes replacement selection, as evidenced
summary of the
subject.
by our quoting from
his
this chapter.
Salzberg (1987) provides an excellent analytical treatment of external sorting,
and Salzberg (1990) describes an approach that takes advantage of replacement
selection, parallelism, distributed computing, and large amounts of memory. Lorin
(1975) spends several chapters on sort-merge techniques. Bradley (1982) provides a
good treatment of replacement selection and multiphase merging, including some
interesting comparisons of processing time on different devices. Tremblay and
Sorenson (1984) and Loomis (1983) also have chapters on external sorting.
Since the sorting of large files accounts for a large percentage of data processing
time, most systems have sorting utilities available. IBM's DFSORT (described in
IBM, 1985) is a flexible package for handling sorting and merging applications. A
VAX
sort utility
is
described in Digital (1984).
331
B-Trees and Other
Tree-structured File
Organizations
CHAPTER OBJECTIVES
Place the development of B-trees in the historical
context of the problems they were designed to
solve.
I
Look
might be
paged AVL
briefly at other tree structures that
used on secondary storage, such
as
trees.
I
Provide an understanding of the important properties possessed by B-trees, and show how these
properties are especially well suited to secondary
storage applications.
Describe fundamental operations on B-trees.
I
Introduce the notion of page buffering and virtual
B-trees.
Describe variations of the fundamental B-tree algorithms, such as those used to build B * trees and
B-trees with variable-length records.
333
CHAPTER OUTLINE
8.1
Introduction:
The Invention of the
8.13 Deletion, Redistribution,
B-Tree
Concatenation
8.2
Statement of the Problem
8.13.1
8.3
Binary Search Trees
8.4
AVL
8.5
Paged Binary Trees
8.6
The Problem with
8.7
as a
Solution
and
Redistribution
8.14 Redistribution during Insertion:
Way to Improve Storage
Trees
Utilization
8.15
the
Top-Down
B*
Trees
8.16 Buffering of Pages: Virtual
Construction of Paged Trees
B-Trees
B-Trees: Working up from the
8.16.1
Bottom
8.16.2 Replacement Based on Page
LRU
Replacement
Height
and Promoting
8.8
Splitting
8.9
Algorithms for B-Tree Searching
and Insertion
8.10
B-Tree Nomenclature
8.18 Variable-length
Records and Keys
8.11
Formal Definition of B-Tree
C Program
Keys into
Properties
B-Tree
8.12 Worst-case Search
8.16.3 Importance of Virtual B-Trees
Depth
8.17
Placement of Information
Associated with the Key
Pascal
to Insert
Program
to Insert
Keys into
B-Tree
8.1
Introduction:
Computer
science
that at the start
The Invention
is a
young
of 1970,
of the B-Tree
discipline.
As evidence of this youth, consider
had twice travelled to the moon,
after astronauts
B-trees did not yet exist. Today, only 15 years
a
major, general-purpose
file
system that
is
later, it is
not built around
hard to think of
a
B-tree design.
Douglas Comer, in his excellent survey article, "The Ubiquitous
B-Tree" [1979], recounts the competition among computer manufacturers
and independent research groups that developed in the late 1960s. The goal
was the discovery of a general method for storing and retrieving data in
large file systems that would provide rapid access to the data with minimal
overhead cost. Among the competitors were R. Bayer and E. McCreight,
who were working for Boeing Corporation at that time. In 1972 they
published an article, "Organization and Maintenance of Large Ordered
335
INTRODUCTION: THE INVENTION OF THE B-TREE
By
Indexes," which announced B-trees to the world.
published his survey
Comer was
able
B-trees had already
article,
to
that
state
organization for indexes in
We
become
so widely used that
de facto,
is,
when Comer
standard
the
database system."
have reprinted the
"the B-tree
1979,
first
few paragraphs of the 1972 Bayer and
McCreight article^ because it so concisely describes the facets of the
problem that B-trees were designed to solve: how to access and maintain
efficiently an index that is too large to hold in memory. You will remember
that this is the same problem that is left unresolved in Chapter 6, on simple
index structures. It will be clear as you read Bayer and McCreight's
introduction that their
back
in the
work goes
straight to the heart
of the issues
we
raise
indexing chapter.
In this paper
index for
we
consider the problem of organizing and maintaining an
dynamically changing random access
a collection
of index elements which are pairs
adjacent data items,
The key x
information
random
namely
identifies
file.
(x, a)
By
an index
of fixed
we mean
size physically
key x and some associated information
unique element
in
the
index,
the
a.
associated
typically a pointer to a record or a collection of records in a
is
access
file.
For
this
paper the associated information
is
of no
further interest.
We assume that the index itself is so voluminous that only rather small
parts
of it can be kept in main store
must be kept on some backup
store.
are pseudo random access devices
time
as
opposed
The
class
which have
random
to a true
one time. Thus the bulk of the index
at
of backup stores considered
rather long access or wait
access device like core store
and
rather high data rate once the transmission of physically sequential data has
been
Typical pseudo random access devices
initiated.
moving head
disks,
Since the data
the index and
keys
to
drums, and data
file itself
changes,
it
elements,
retrieve
are:
fixed and
cells.
must be possible not only
to search
but also to delete and to insert
more accurately index elements economically.
The index orga-
nization described in this paper allows retrieval, insertion, and deletion ot
keys in time proportional to log^
size
or better, where
/ is
the size of the
dependent natural number which describes the page
such that the performance of the maintenance and retrieval scheme
index, and k
is
a device
becomes near optimal.
Exercises 17, 18, and 19 at the end of Chapter 6 introduced
of a paged index. Bayer and McCreight's statement that
developed a scheme with retrieval time proportional to log^ /,
related to the page size, is very significant. As we will see, the use
"From
Ada-Informatica, 1:173-189,
permission.
1972, Springer Verlag,
New
the notion
they have
where k
is
of a B-tree
York. Reprinted with
336
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
of 64 to index a file with a million records results in being
key for any record in no more than four seeks to the disk.
A binary search on the same file can require as many as 20 seeks. Moreover,
we are talking about getting this kind of performance from a system that
requires only minimal overhead as keys are inserted and deleted.
Before looking in detail at Bayer and McCreight's solution, let's first
return to a more careful look at the problem, picking up where we left off
in Chapter 6. We will also look at some of the data and file structures that
were routinely used to attack the problem before the invention of B-trees.
Given this background, it will be easier to appreciate the contribution made
by Bayer and McCreight's work.
with
page
size
able to find the
One
provides
matter before
last
we
begin:
Why
the
name
B-tree?
Comer
(1979)
this footnote:
'
lie origin
of 'B-tree
As we
Creight].
'
'
shall see,
has never been explained by [Bayer and
Mc-
"balanced," "broad," or "bushy" might apply.
Others suggest that the "B" stands for Boeing. Because of his contribuhowever, it seems appropriate to think of B-trees as "Bayer"-trees.
tions,
8.2
Statement
of the
Problem
The fundamental problem with keeping an index on secondary
of course, that accessing secondary storage
problem can be broken down into two more
Biliary searching requires too
many
seeks.
is
slow.
specific
storage
is,
This fundamental
problems:
Searching for
key on
disk
often involves seeking to different disk tracks. Since seeks are expensive, a search that has to
look in more than three or four locations
more time than is desirable. If
before finding the key often requires
we
are using a binary search, four seeks
is
only enough to differenti-
between 15 items. An average of about 9.5 seeks is required to
find a key in an index of 1,000 items using a binary search. We need
to find a way to home in on a key using fewer seeks.
It can be very expensive to keep the index in sorted order so we can perform
a binary search. As we saw in Chapter 6, if inserting a key involves
ate
number of the other keys in the index, index maintevery nearly impractical on secondary storage for indexes
consisting of only a few hundred keys, much less thousands of keys.
We need to find a way to make insertions and deletions that have
moving
nance
a large
is
only local effects in the index, rather than requiring massive reorganization.
337
BINARY SEARCH TREES AS A SOLUTION
AX
CL
DE
FIGURE 8.1 Sorted
list
HN
FT
FB
NR
KF
JD
PA
RF
WS
TK
SD
YJ
of keys.
These were the two
problems that confronted Bayer and McCreight
in 1970. They serve as guideposts for steering our discussion of the use of
tree structures for secondary storage retrieval.
8.3
critical
Binary Search Trees as a Solution
Let's begin
by addressing the second of these two problems, looking at the
list in sorted order so we can perform binary searches.
cost of keeping a
Given the sorted
list
in Fig. 8.1,
shown
as a binary search tree, as
we
can express
Using elementary data structure techniques,
create nodes that contain right
can be constructed as
representation of the
8.2. In each
and
left link fields
two
left
list
it
is
simple matter to
so the binary search tree
linked structure. Figure 8.3 illustrates
first
node, the
binary search of this
in Fig. 8.2.
levels
and right
of the binary search
tree
links point to the left
shown
and right
linked
in Fig.
children
of the node.
If
each node
is
treated as a fixed-length record in
contain relative record numbers
(RRNs) pointing
which the
link fields
to other nodes, then
it is
possible to place such a tree structure on secondary storage. Figure 8.4
illustrates the contents
of the 15 records that would be required to form the
binary tree depicted in Fig. 8.2.
of the link fields in the file are empty because they
with no children. l^j^xMXl^^fjlod^jaeedio contain some
special character, suchas f, to indicat e t hat the search through the tree h as
Note
that over half
are leaf nodes,
FIGURE 8.2 Binary search tree representation of the
KF.
list
of keys.
338
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
FIGURE 8.3 Linked representation of part of a binary search
tree.
reached the leaf level and that there are no more nodes on the search
We
leave the fields blank in this figure
tomake them more
illustrating the potentially substantial cost in
incurred by this kind of linked representation of
But
to focus
important
new
have to sort the
records in the
on the
costs
a tree.
and not the advantages
file
to be able to
perform
binary search.
illustrated in Fig. 8.4 appear in
random
to
is
capability that this tree structure gives us:
file
noticeable,
terms of space utilization
miss the
We
no longer
Note that the
rather than sorted
FIGURE 8.4 Record contents
[ill
for a linked representation of
the binary tree
Key
Left
Right
child
child
Left
Key
FB
HN
JD
KF
RF
10
CL
SD
11
NR
AX
12
DE
YJ
13
WS
PA
14
TK
FT
Right
child child
in Fig.
8.2.
339
BINARY SEARCH TREES AS A SOLUTION
FIGURE 8.5 Binary search tree with LV added.
The sequence of the
order.
structure of the tree;
in the link fields.
that if
we add
records in the
new key
has no necessary relation to the
positive consequence that follows
to the
such
file,
to create a tree that
we would get with
file
the information about the logical structure
The very
appropriate leaf node
as
all
as
L V, we
need only link
provides search performance that
a binary search on a sorted
list.
The
tree
with
carried
is
from
is
LV
this is
it
to the
as
good
added
is
illustrated in Fig. 8.5.
Search performance on
balanced state.
By
a leaf
does not differ
level.
For the tree in Fig.
to complete balance,
this tree
we mean
is
still
good because
the tree
is
in a
of the shortest path to
from the height of the longest path by more than one
balanced
where
that the height
8.5, this difference
all
the paths
of one
from root
is
we
as close as
can get
to leaf are exactly the
same
length.
Consider what happens if we go on to enter the following eight keys to
the tree in the sequence in which they appear:
NP MB TM LA UF ND TS NK
Just searching
down through
the tree and adding each key at
position in the search tree results in the tree
The
tree
is
now
shown
its
correct
in Fig. 8.6.
out of balance. This is a typical result for trees built by
treeasTiey occur without rearrangement. The
placing keys ""into Hie
resulting disparity
between the length of various search paths
is
undesirable
any binary search tree, but is especially troublesome if the nodes of the
tree are being kept on secondary storage. There are now keys that require
seven, eight, or nine seeks for retrieval. A binary search on a sorted list of
these 24 keys requires only five seeks in the worst case. Although the use of
a tree lets us avoid sorting, we are paying for this convenience in terms of
extra seeks at retrieval time. For trees with hundreds of keys, in which an
in
out-of-balance search path might extend to 30, 40, or
is
too high.
more
seeks, this price
340
B-TREES AND OTHER TREE-STRUCTURED
HN
CL
/ \DE
AX
FILE
ORGANIZATIONS
PA
/ \JD
FT
/
NR
/
LV
WS
RF
/X
TK
YT
\TM
'\
NP
UF
/
MB
\ND
TS
NK
FIGURE 8.6 Binary search tree showing the effect
8.4
of
added
keys.
AVL Trees
we
is no necessary relationship between the order in
and the structure of the tree. We stress the word
necessary because it is clear that order of entry is, in fact, important in
determining the structure of the sample tree illustrated in Fig. 8.6. The
reason for this sensitivity to the order of entry is that, so far, we have just
been linking the newest nodes at the leaf levels of the tree. This approach
Earlier
said that there
which keys
are entered
can result in
some very
undesirable tree organizations.
example, that our keys consist of the
letters
A-G,
and that
keys in alphabetical order. Linking the nodes together
\
\D
\G
FIGURE 8.7 A degenerate
tree.
as
Suppose,
for
we receive these
we receive them
341
AVL TREES
FIGURE 8.8 AVL trees.
produces
degenerate tree that
is,
in fact,
nothing more than
linked
list,
as illustrated in Fig. 8.7.
The
tree as
solution to this problem
we
receive
elegant
method
known
as
new
A VL trees,
in
a height-balanarftree^
of difference that
a
common
AVL
tree
is
honor of the
that
which
M.
Landis
who
first
This means that there
is
An AVL
defined them.
a limit
placed on the
M^
tree
amount
AVL tree the maximum allowable difference is
HB(1)
general class of height-balanced trees
AVL trees.
not in balance
By
mance
is
tree.
known
as
1.
It
An
is
HB(k)
In each of these trees, the root of the subtree that
marked with an X.
make AVL
feat ures that
setting a
subtrees,
maximum
AVL
trees
important are
as follows:
allowable difference in the height of any
trees guarantee a certain
in searching;
Maintaining
minimum
level
AVL
form as new nodes are inserted involves
of four possible rotations. Each of the rota-
FIGURE 8.9 Trees that are not AVL trees.
two
of perfor-
and
a tree in
the use of one of a set
trees
The trees illustrated in Fig. 8.8 have the AVL, or HB(1) property. Note
no two subtrees of any root differ by more than one level. The trees in
The two
/\
One
of
are permitted to be k levels out of balance.
Fig. 8.9 are not
is
nodes of the
pair of Russian mathematicians, G.
therefore called a height-balanced 1-tree or
member of a more
trees,
to reorganize the
allowed between the heights of any two subtrees sharing
root. In an
is
somehow
for handling such reorganization results in a class
Add'son-Vel'skii and E.
is
is
keys, maintaining a near optimal tree structure.
/\
X
/
m m
342
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
FIGURE 8.10 A completely balanced search
tions
confined to
is
a single, local area
of the rotations requires only
of the
tree.
tree.
The most complex
five pointer reassignments.
AVL
trees are an important class of data structure. The operations used
and maintain AVL trees are described in Knuth (1973b), Standish
(1980), and elsewhere. AVL trees are not themselves directly applicable to
most file structure problems because, like all strictly binary trees, they have
they are too deep. However, in the context of our general
too many levels
discussion of the problem of accessing and maintaining indexes that are too
to build
large to
fit
in
memory,
AVL
trees are interesting because they suggest that
possible to define procedures that maintain height balance.
it is
The
fact that an
AVL
tree
height-balanced guarantees that search
is
performance approximates that of a completely balanced tree^For example,
the completely balancedTorm oTTTreelnade up from the input keys
BCGEFDA
is
illustrated in Fig. 8.10,
keys, arriving in the
For
given
and the
AVL
same sequence,
completely balanced
N possible keys,
looks
of the
tree.
For an
So,
illustrated in Fig. 8.11.
is
tree, the
AVL
(N +
tree, the
1.44 log 2
levels.
from the same input
worst-case search to find
key,
at
log 2
levels
tree resulting
given 1,000,000 keys,
1)
worst-case search could look
(N+
at
2)
completely balanced tree requires
some of the keys, but never to 21 levels. If the tree
maximum number of levels increases to only 28. This
seeking to 20 levels for
is
an
AVL
tree, the
FIGURE 8.1
A search tree conAVL procedures.
structed using
343
PAGED BINARY TREES
is
very interesting
AVL
given that the
result,
no more than
single reorganization requires
procedures guarantee that
five pointer reassignments.
Empirical studies by VanDoren and Gray (197 4V
shown
among
others,
have
that such local reorganizations are required for approximately every
other insertion into the tree and for approximately every fourth deletion. So
height balancing using
AVL
methods guarantees
that
we
will obtain a
reasonable approximation to optimal binary tree performance
is
at a cost that
acceptable in most applications using primary, random-access
When we
memory.
more
are using secondary storage, a procedure that requires
than five or six seeks to find
unacceptable. Returning to the
key
is less
than desirable; 20 or 28 seeks
two problems
that
we
is
identified earlier in
this chapter:
Binary searching requires too
Keeping an index
we
is
seeks;
and
expensive,
can see that height-balanced trees provide an acceptable solution to the
second problem.
8.5
many
in sorted order
Now we
need to turn our attention to the
first
problem.
Paged Binary Trees
Once
again
we
are confronting
secondary storage devices:
It
is
perhaps the most
critical feature
of
takes a relatively long time to seek to a specific
location, but once the read head
a
what
is
positioned and ready, reading or writing
stream of contiguous bytes proceeds rapidly. This combination of slow
seek and fast data transfer leads naturally to the notion of paging. In
system, you do not incur the cost of
a paged
few bytes.
of the disk, you read
disk seek just to get a
once you have taken the time to seek to an area
file. This page might consist of a great many
individual records. If the next bit of information you need from the disk is
in the page that was just read in, you have saved the cost of a disk access.
Paging, then, is a potential solution to our searching problem. By
dividing a binary tree into pages and then storing each page in a block of
contiguous locations on disk, we should be able to reduce the number of
seeks associated with any search. Figure 8.12 illustrates such a paged tree. In
this tree we are able to locate any one of the 63 nodes in the tree with no
more than two disk accesses. Note that every page holds seven nodes and
can branch to eight new pages. If we extend the tree to one additional level
of paging, we add 64 new pages; we can then find any one of 511 nodes in
only three seeks. Adding yet another level of paging lets us find any one of
4,095 nodes in only four seeks. A binary search of a list of 4,095 items can
Instead,
in
an entire page from the
take as
many
as 12 seeks.
344
B-TREES AND OTHER TREE-STRUCTURED
ORGANIZATIONS
FILE
A A
mi
nnnn nnnn
A A
a
/\
/\
9 9 *
-3,
n n n
FIGURE 8.12 Paged binary
A ^A
S
l\ l\ l\ l\
/\ /\
3 #
/\
/\
nn
ft
A A
A A
/wwwi
S:
/\ /\ l\ l\
/i
/in
tree.
Clearly, breaking the tree into pages has the potential to result in faster
searching on secondary storage, providing us with
we
than any other form of keyed access that
much
faster retrieval
have considered up to
this
Moreover, our use of a page size of seven in Fig. 8. 12 is dictated more
by the constraints of the printed page than by anything having to do with
secondary storage devices. A more typical example of a page size might be
8 kilobytes capable of holding 511 key /reference field pairs. Given this page
size, and assuming that each page contains a completely balanced, full tree,
and that the pages themselves are organized as a completely balanced, full
tree, it is then possible to find any one of 134,217,727 keys with only three
seeks. That is the kind of performance we are looking for. Note that, while
the number of seeks required for a worst-case search of a completely full,
point.
balanced binary tree
is
(N +
log 2
where
is
the
number of keys
in the tree, the
the paged versions of a completely
full,
log fe+1
1)
number of seeks
balanced tree
(N +
required for
is
1)
once again, the number of keys. The new variable, k, is the
held in a single page. The second formula is actually a
generalization of the first, since the number of keys in a page of a purely
where
is,
number of keys
=!
345
THE PROBLEM WITH THE TOP-DOWN CONSTRUCTION OF PAGED TREES
binary tree
is
It is
makes
the logarithmic effect of the page size that
the
impact of paging so dramatic:
log 2 (134,217,727
log 511 +
(134,217,727
1)
27 seeks
1)
3 seeks.
come free. Every access to a page
amount of data, most of which is not
used. This extra transmission time is well worth the cost, however, because
it saves so many seeks, which are far more time-consuming than the extra
transmissions. A much more serious problem, which we look at next, has
to do with keeping the paged tree organized.
The use of
large pages does not
requires the transmission of a large
8.6
The Problem with the Top-down Construction
of Paged Trees
Breaking
pages is a strategy that is well suited to the physical
of
secondary
storage devices such as disks. The problem,
characteristics
once we decide to implement a paged tree, is how to build it. If we have the
entire set of keys in hand before the tree is built, the solution to the problem
is relatively straightforward: We can sort the list of keys and build the tree
a tree into
from this sorted list. Most importantly, if we plan to start building the tree
from the root, we know that the middle key in the sorted list of keys should
be the root key within the root page of the tree. In short, we know where to
begin and are assured that
a
this
beginning point will divide the
set
of keys
in
balanced manner.
Unfortunately,
receiving keys in
the
problem
is
random order and
them. Assume that
we must
much more
inserting
complicated
them
build a paged tree as
we
as
soon
as
we
if
we
are
receive
receive the following
sequence of single-letter keys:
CSDTAMPIBWNGURKEHOLJYQZFXV
We
will build a
keys per page. As
paged binary
we
tree that contains a
insert the keys,
we
necessary to keep each page as balanced as
illustrated in Fig. 8.13.
in pages), this tree
as
is
does not turn out too badly. (Consider, for example,
arrive in alphabetical order.)
is
not dramatically misshapen,
it
clearly illustrates
from the top down.
from the root, the initial keys must, of necessity, go into the
example at least two of these keys, C and D, are not keys that
the difficulties inherent in building a paged binary tree
start
root. In this
of three
Evaluated in terms of the depth of the tree (measured
what happens if the keys
Even though this tree
When you
maximum
them within a page
possible. The resulting tree
rotate
346
B-TREES AND OTHER TREE-STRUCTURED
ORGANIZATIONS
FILE
A A
H
\O
A
Y
FIGURE 8.13 Paged tree constructed from keys arriving
we want
there.
beginning of the
They
in
random input sequence.
adjacent in sequence and tend toward the
of keys. Consequently, they force the tree out of
are
total set
balance.
Once the wrong keys are placed in the root of the tree (or in the root
of any subtree further down the tree), what can you do about it?
Unfortunately, there is no easy answer to this. We cannot simply rotate
entire pages of the tree in the same way that we would rotate individual
keys in an unpaged tree. If we rotate the tree so the initial root page moves
down to the left, moving the C and D keys into a better position, then the
S key is out of place. So we must break up the pages. This opens up a whole
world of possibilities and difficulties. Breaking up the pages implies
rearranging them to create new pages that are both internally balanced and
well arranged relative to other pages. Try creating a page rearrangement
algorithm for the simple, three-keys-per-page tree from Fig. 8.13. You will
find it very difficult to create an algorithm that has only local effects,
rearranging just a few pages. The tendency is for rearrangements and
adjustments to spread out through a large part of the tree. This situation
grows even more complex with larger page sizes.
So, although we have determined that the idea of collecting keys into
pages is a very good one from the standpoint of reducing seeks to the disk,
347
SPLITTING AND PROMOTING
we have
not yet found
confronting
CK
at least
way
to collect the right keys.
two unresolved
How
do we ensure
good
separator keys, dividing
We
are
still
questions:
that the keys in the root
up the
set
page turn out to be
of other keys more or
less
evenly?
B^
How
do we avoid grouping keys, such
ple, that
There
is,
should not share
size
C, D, and 5 in our exam-
page?
in addition, a third question that
because of the small page
*&
as
we have
of our sample
not yet had to confront
tree:
How
can we guarantee that each of the pages contains at least some
minimum number of keys? If we are working with a larger page
size, such as 8,191 keys per page, we want to avoid situations in
which
a large
number of pages each
Bayer and McCreight's 1972 B-tree
precisely tow ard these questions.
contains only a few dozen keys.
article
provides a solution directed
8.7
B-Trees: Working up from the Bottom
A number
in computer science have
problem from a different viewpoint. B-trees are
an example of this viewpoint-shift phenomenon.
The key insight required to make the leap from the kinds of trees we
have been considering to a new solution, B-trees, is that we can choose to
build trees upward from the bottom instead of downward from the top. So far, we
have assumed the necessity of starting construction from the root as a given.
Then, as we found that we had the wrong keys in the root, we tried to find
ways to repair the problem with rearrangement algorithms. Bayer and
McCreight recognized that the decision to work down from the root was,
of itself, the problem. Rather than finding ways to undo a bad situation,
they decided to avoid the difficulty altogether. With B-trees, you allow the
root to emerge, rather than set it up and then find ways to change it.
of the elegant, powerful ideas used
grown out of looking
8.8
Splitting
at a
and Promoting
of an ordered sequence of keys and a
with the paged
trees shown previously; there is just an ordered list of keys and some
pointers. The number of pointers always exceeds the number of keys by
In a B-tree, a page, or node, consists
set
of pointers. There
is
no
explicit tree within a node, as
348
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
FIGURE 8.14
Initial ieaf of
B-tree with a page size of
seven.
one.
The maximum number of pointers
that can be stored in a
we have
the order of the B-tree. For example, suppose
Each page can hold
most seven keys and
at
node
is
called
an order-eight B-tree.
eight pointers.
Our
initial leaf of
the tree might have a structure like that illustrated in Fig. 8.14 after the
insertion of the letters
B C G E
The
DA
starred (*) fields are the pointer fields. In this leaf, as in any other
leaf node, the value
of
all
definition, a leaf node has
do not lead
pages
the pointers
no children
to other pages in the tree.
usually
contain
an
is
set to indicate end-of-list.
We assume that the pointers in the leaf
pointer
invalid
By
in the tree; consequently, the pointers
such
value,
as
1.
Note,
incidentally, that this leaf is also our root.
In a real-life application there
also usually
is
some other information
stored with the key, such as a reference to a record containing data that are
associated with the key. Consequently, additional pointer fields in each
page might actually lead to some associated data records that are stored
But,
elsewhere.
paraphrasing Bayer and McCreight,
purposes, "the associated information
Building the
a single
page
first
is
our present
As we insert new keys, we use
memory and, working in memory,
Since we are working in electronic
easy enough.
disk access to read the page into
insert the
key into
memory,
this insertion is relatively
its
for
of no further interest."
is
place in the page.
inexpensive compared to the cost of
additional disk accesses.
But what happens
the kev
full.
we
come
keys
in?
Suppose
try to insert the J
we
leaf to
choose between the leaves
accommodate the new
when
shown
a
to
add
is
in Fig. 8.15
searching. In short,
higher level in the
J key.
we want
find that our Jeaf
distributing the keys as evenly as
can between the old leaf node and the new one, as
Si nre wgjiow have two lea ves, we need to create
FIGURE 8.15 Splitting the
we
We then split the le af intoTwo leaves,
tree to enable us to
as additional
to the B-tree._W hen
we
349
SPLITTING AND PROMOTING
FIGURE 8.16 Promotion of the E key into a root node.
need to create
leaves. In this
new root. We do this by promoting a key that separates th e
case, we promote the E from the first position in the second
a
leaf, as illustrated in Fig.
In this
in
two
8.16.
example we describe the
steps to
make
and promotion are handled
Let's see
how
paged binary
splitting
and the promotion operations
the procedure as clear as possible; in practice, splitting
in a single operation.
B-tree grows given the key sequence that produces the
tree illustrated in Fig. 8.13.
The sequence
is
CSDTAMPIBWNGURKEHOLJYQZFXV
We use an order-four B -tree
(four poi nter fields and three key fields pe r
page), since this corresponds to the page size of the paged binary tree.
such
more
Using
small page size has the additional advantage of causing pages to
frequently,
promotion.
We
split
providing us with more examples of splitting and
omit
explicit indication
of the pointer
fields so
we
can
fit
on the printed page.
8. 17 illustrates the growth of the tree up to the point at which the
root node is about to split. Figure 8.18 shows the tree after the splitting of
the root node. The figure also shows how the tree continues to grow as the
larger tree
Figure
remaining keys in the sequence are added. We number each of the tree's
pages (upper left corner of each node) so you can distinguish the newly
added pages from the ones already in the tree.
Note
that the tree
is
always perfectly balanced
path from the root to any leaf is the same
other
leaf.
as the
w ith regar d to height; the
path from the root to any
Also note that the keys that are promoted upward into the tree
of keys we want in a root: keys that are good
are necessarily the kind
separators.
pages
fill
By working up from the leaf level, splitting and promoting as
we overcome the problems that plague our earlier paged
up,
binary tree efforts.
Insertion of C,
S,
and
D
c D
into the initial page:
Insertion of
forces the split
and the promotion of
5:
c D
A added without
incident:
A C D
Insertion of
split
\<^r
M forces another
and the promotion of D:
'3
A C
P,
I,
B,
and
^cx/vCtX*^
W inserted
into existing pages:
D N
A B C
N causes another
followed by the promotion of N. G, U, and R are
Insertion of
split,
added
to existing pages:
A B C
FIGURE 8.17 Growth of a B-tree, part
is imminent.
ting of the root
350
I.
The
tree grows to a point at
which
split-
m^ o<? 33
Insertion of K causes a split at leaf level,
followed by the promotion of K. This
causes a split of the root. N is promoted
to
become the new
root.
is
added
to a leaf:
D K
r
E G
A B C
Insertion of
H causes
a leaf to split.
/
M
T U
P R
H is
promoted. O, L, and/ are added:
A B C
T U
Y and Q force two more leaf
and promotions. Remaining letters
Insertion of
splits
are added:
B C
FIGURE 8.18 Growth of a B-tree, part
II.
M
The
root splits to
T L V
add
new
level;
X Y
remaining keys are
inserted.
351
352
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
8.9
Algorithms for B-Tree Searching and Insertion
Now
that
we
have had
look
a brief
at
how
work on paper,
make them work
B-trees
outline the structures and algorithms required to
let's
in a
computer. Most of the code that follows is pseudocode. C and Pascal
implementations of the algorithms can be found at the end of this chapter.
Page Structure
We
used by
As you
a B-tree.
begin by defining one possible form for the page
see later in this chapter
and
in the following
ways to construct the page of a B-tree. We
start with a simple one in which each key is a single character. If the
maximum number of keys and children allowed on a page is MAXKEYS
chapter, there are
many
different
MAXCHILDREN, respectively, then the following
C and Pascal describe a page called PAGE.
and
structures ex-
pressed in
In C:
struct BTPAGE {
short
KEYCDUNT;
char
KEYCMAXKEYS]
CHILDCMAXKEYS+1
short
>
PAGE;
;
/* number of keys stored
/* the actual keys
/RRNs of children
&
in
PAGE */
*/
3SS^S
* /
In Pascal:
TYPE
BTPAGE
RECORD
KEYCDUNT
KEY
CHILD
nt eger
ar ray [
ar ray [
1
1
MAXKEYS] of char;
MAXCHILDREN] of integer
END;
VAR
PAGE
BTPAGE;
Given this page structure, the file containing the B-tree consists of a set
of fixed-length records. Each record contains one page of the tree. Since the
keys in the tree are single letters, this structure uses an array of characters to
hold the keys. More typically, the key array is a vector of strings rather than
The variable PAGE.KEYCOUNT is useful
when the algorithms must determine whether a page is full or not. The
PAGE.CHILD[] array contains the RRNs of PAGE's children, if there are
any. When there is no descendent, the corresponding element of PAGE.
CHILD[] is set to a nonaddress value, which we call NIL. Figure 8.19
just a vector of characters.
shows two pages
in a B-tree
of order four.
ALGORITHMS FOR B-TREE SEARCHING AND INSERTION
353
Part of a B-tree:
2
H K
/
A B C
'NT
(a)
Contents of
PAGE
-
for pages 2
and
KEYCOUNT f ^
{
KEY
3:
array
CHILD
p*
array
*>*
Page 2
Page 3
NIL
NIL
NIL
(b)
FIGURE 8.19 A B-tree of order four, (a) An internal node and some leaf
nodes, (b) Nodes 2 and 3, as we might envision them in the structure
PAGE.
Searching
The
first
procedure. Searching
yet
-^n
^-~*n
still
is
B-tree algorithms
a
good
we examine
place to begin because
illustrates the characteristic aspects
They are recursive; and
They work in two stages, operating
are a tree-searching
it is
relatively simple
of most B-tree algorithms:
alternatively
on
entire pages
and
then within pages.
The searching procedure
calls itself recursively,
seeking to a page and
at successively lower
of the tree until it either finds the key or finds that it cannot descend
further, having reached beyond the leaf level. Figure 8.20 contains a
description of the searching procedure in pseudocode.
then searching through the page, looking for the key
levels
354
B-TREES AND OTHER TREE-STRUCTURED
FUNCTION:
search (RRN, KEY, FOUND_RRN
if RRN == NIL then
FILE
ORGANIZATIONS
FOUND_POS)
/* stopping condition for the recursion */
return NOT FOUND
else
read page RRN into PAGE
look through PAGE for KEY, setting POS equal to the
position where KEY occurs or should occur,
if KEY was found then
F0UND_RRN = RRN
/* current RRN contains the key */
F0UND_P0S = POS
return FOUND
else /* follow CHILD reference to next level down */
return(search(PAGE.CHILDCPOS], KEY, F0UND_RRN, F0UND_P0S
endif
endif
:
end FUNCTION
FIGURE 8.20 Function search (RRN, KEY,
F0UND_RRN, FOUND_POS)
searches
re-
cursively through the B-tree to find KEY. Each invocation searches the page refer-
enced by RRN. The arguments FOUND_RRN and FOUND_POS identify the page
and position of the key, if it is found. If searchO finds the key, it returns FOUND.
it goes beyond the leaf level without finding the key, it returns NOT FOUND.
Let's
work through
the function
by hand, searching
for the
If
key Kin the
RRN
RRN
not
NIL,
so
the
(2).
function reads the root into PAGE, then searches for K among the elements
of PAGE.KEY[]. The K
not found. Since K should go between D and
tree illustrated in Fig. 8.21.
argument equal
We begin by
RRN
to the
calling the function
of the root
This
with the
is
is
N, POS identifies position
in the root as the position of the pointer to
where the search should proceed. So search () calls itself, this time using the
1"*"
RRN
stored in
On
PAGE.CHILD[1]. The
the next
M. Once again the
PAGE.KEY[]. Again,
indicates
is
searches
not
found.
where the search should proceed.
RRN
RRN
is 3.
for
This
and
among the keys in
time PAGE.CHILD[2]
Searchf) calls itself again, this
through the various
stored in
levels
of
return
statements until the program that
originally calls search() receives the information that the key
f
I,
PAGE.CHILD[2].
Since this call is from a leaf node, PAGE.CHILD[2] is NIL, so the call
search() fails immediately. The value NOT FOUND is passed back
time using the
to
function
value of this
reads the page containing the keys G,
call, search()
We
will use zero origin indexing in these examples, so the leftmost
PAGE.KEY[0], and
the
RRN
of the leftmost child
is
key
PAGE.CHILDfO].
is
in a
not found.
page
is
355
ALGORITHMS FOR B-TREE SEARCHING AND INSERTION
D N
/
G
A B C
/.XT
M
P R
T U
FIGURE 8.21
B-tree used for
the search example.
Now let's use search()
same downward path
position 2 of page
FOUND_POS,
2 of page
3,
3.
to look for
that
It
it
M, which
is
in the tree.
did for K, but this time
and 2
stores the values 3
respectively, indicating that
and returns the value
in
it
It
follows the
finds the
M in
FOUND_RRN
and
M can be found in the position
FOUND.
and Promotion There ar e two important obsermake abo ut the insertion, splitting, and promotion proc ess.
Insertion, Splitting,
vations
*a
It
we
begins with a search that proceeds
level;
&r
can
all
the
way down
to the leaf
and
at the leaf level, the work of inand promotion proceeds upward from the bottom.
After finding the insertion location
sertion, splitting,
Consequently,
we
can conceive of our recursive procedure
as
having three
phases:
1.
search-page step
the recursive
2.
The
recursive call
the tree as
3.
it
itself,
before
which moves the operation down through
searches for either the key or the place to insert
Insertion, splitting,
recursive
that, as in the search() function, takes place
call;
call,
and promotion logic that are executed
the action taking place
it;
and
after the
on the upward return path
fol-
lowing the recursive descent.
We
need an example of an insertion so we can watch the insertion
procedure work through these phases. Let's insert the $ character into the
tree shown in the top half of Fig. 8.22, which contains all of the letters of
the alphabet. Since the ASCII character sequence places the $ character
ahead of the character A, the insertion is into the page with an RRN of 0.
This page and its parent are both already full, so the insertion causes
splitting and promotion that result in the tree shown in the bottom half of
Fig. 8.22.
356
B-TREES AND OTHER TREE-STRUCTURED
Before inserting
FILE
ORGANIZATIONS
$:
After inserting
$:
H N
FIGURE 8.22 The effect of adding $ to the tree constructed
in Fig.
8.18.
Now let's see how the insert!) function performs this splitting and
promotion. Since the function operates recursively, it is important to
understand
insert()
how
the function arguments are used
function that
we
CURRENT_RRN
on successive
calls.
The
are about to describe uses four arguments:
The
RRN
use.
As
o{ the B-tree page that
is
currently in
the function recursively descends and as-
cends the
tree, all the
RRNs
on the search and
in-
sertion path are used.
KEY
The key
PROMO_KEY
Argument used only
that
is
to be inserted.
to carry
back the return
and the pro-
value. If the insertion results in a split
motion of a key, PROMO_KEY contains the
promoted key on the ascent back up the tree.
PROMO_R_CHILD
another return value argument. If there
higher levels of the calling sequence
must not only insert the promoted key value,
This
is
is
a split,
357
ALGORITHMS FOR B-TREE SEARCHING AND INSERTION
RRN of the new page created in
PROMO_KEY inserted,
PROMO_R_CHILD the right child pointer
but also the
the
When
split.
is
is
inserted with
it.
In addition to the values returned via the
and
PROMO_R_CHILD,
makes
NO PROMOTION
PROMO_KEY
PROMOTION if
arguments
insertQ returns the value
it
done and
ERROR if the insertion cannot be made.
Figure 8.23 illustrates the way the values of these arguments change as
the insert() function is called and calls itself to perform the insertion of the
$ character. The figure makes a number of important points:
promotion,
nothing is promoted, and
a
During the search
changes
as the
fected
by
an insertion
step part of the insertion, only
function
path of successive
The
if
calls
CURRENT_RRN
descending the
calls itself,
is
This search
tree.
includes every page of the tree that can be af-
and promotion on the return path.
when CURRENT_RRN is NIL. There are no
splitting
search step ends
further levels to search.
As each
recursive
call returns,
we
execute the insertion and splitting
PROOtherwise, we
logic at that level. If the lower-level function returns the value
MOTION,
then we have a key to insert at this level.
have no work to do and can just return. For example,
insert
at the highest (root) level of the tree without
therefore return
that the
NO PROMOTION
PROMO_KEY
and
from
we
are able to
splitting,
and
That means
from this level
this level.
PROMO_R_CHILD
have no meaning.
Given
this
introduction to the
insert()
to look at an algorithm for the function
function's operation,
shown in
Fig. 8.24.
we
are ready
We have already
described insertO's arguments. There are several important local variables as
well:
PAGE
The page
NEWPAGE
New
POS
The position in
or would occur
P_B_RRN
The
that insert()
page created
currently examining.
if a split occurs.
PAGE
where the key occurs
(if it is
present)
(if inserted).
relative record
level. If a split
is
number promoted
occurs
at the
from below up
next lower
level,
to this
P_B_RRN
number of the new page created
during the split. P_B_RRN is the right child that is inserted
with P_B_KEY into PAGE.
contains the relative record
P_B_KEY
The key promoted from below up
along with
P_B_RRN,
is
to this level. This key,
inserted into
PAGE.
358
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
KEY =
CURRENT RRN =
NO PROMOTION
PROMO_KEY = <undefined>
PROMO_R_CHILD = <undefined>
Return value:
fc
Search
step
Recursive
call
Insertion and
KEY =
splitting logic
mm
CURRENT RRN
Return value: PROMOTION
PROMO_KEY = H
PROMO R CHILD = 12
Search
step
Recursive
call
Insertion and
KEY =
splitting logic
CURRENT RRN
PROMOTION
PROMO_KEY = B
PROMO R CHILD = 11
Return value:
Search
fet
step
Recursive
Insertion
KEY =
call
and
4n
splitting logic
CURRENT RRN = NIL
PROMOTION
PROMO_KEY = $
PROMO R CHILD = NIL
Return value:
Search
step
Recursive
Insertion
call
and
splitting logic
FIGURE 8.23 Pattern of recursive calls to insert $ into the B-tree as illustrated
in Fig.
8.22.
ALGORITHMS FOR B-TREE SEARCHING AND INSERTION
insert
FUNCTION:
CURRENT_RRN, KEY PROMO_R_CHILD, PROMO_KEY)
if CURRENT_RRN
/* past bottom of tree */
NIL then
PROMO_KEY := KEY
PR0M0_R_CHILD := NIL
return PROMOTION
/* promote original key and NIL */
else
read page at CURRENT_RRN into PAGE
search for KEY in PAGE.
let POS := the position where KEY occurs or should occur.
if KEY found then
issue error message indicating duplicate key
return ERROR
RETURN_VALUE
insert PAGE CHILD [ POS]
(
KEY,
P_B_RRN, P_B_KEY)
if RETURN_VALUE
= = NO PROMOTION or ERROR then
return RETURN_VALUE
elseif there is space in PAGE for P_B_KEY then
insert P_B_KEY and P_B_RRN (promoted from below) in PAGE
return NO PROMOTION
else
split L ( P_B_KEY, P_B_RRN, PAGE, PR0M0_KEY, PR0M0_R_CHILD, NEWPAGE
write PAGE to file at CURRENT_RRN
write NEWPAGE to file at rrn PR0M0_R_CHILD
/* promoting PR0M0_KEY and PR0M0_R_CHILD */
return PROMOTION
endif
end FUNCTION
FIGURE 8.24 Function insert
inserts a
KEY
in a
B-tree.
number CURRENT_RRN.
sively until
it
finds
KEY
(CURRENT_RRN,
The
If
in a
KEY,
PROMO_R_CHILD, PROMO_KEY)
insertion attempt starts at the page with relative record
page is not a
page or reaches a
this
leaf page, the function calls itself recurleaf.
If
it
finds KEY,
it
issues an error
ERROR. If there is space for KEY in PAGE, KEY is inserted. Otherwise, PAGE is split. A split assigns the value of the middle key to
PROMO_KEY and the relative record number of the newly created page to PROMO_R_CHILD so insertion can continue on the recursive ascent back up the tree. If a promotion does occur, insertO indicates this by returning PROMOTION. Otherwise, it returns
NO PROMOTION.
message and
quits, returning
359
360
B-TREES AND OTHER TREE-STRUCTURED
PROCEDURE:
split (I_KEY,
I_RRN,
FILE
ORGANIZATIONS
PROMO_KEY, PROMO_R_CHILD, NEWPAGE
PAGE,
copy all keys and pointers from PAGE into a working page that
can hold one extra key and child.
insert I_KEY and I_RRN into their proper places in the working page.
allocate and initialize a new page in the B-tree file to hold NEWPAGE.
set PR0M0_KEY to value of middle key, which will be promoted after
the split.
set PR0M0_R_CHILD to RRN of NEWPAGE.
copy keys and child pointers preceding PR0M0_KEY from the working
page to PAGE.
copy keys and child pointers following PR0M0_KEY from the working
page to NEWPAGE.
end PROCEDURE
FIGURE 8.25 Split (l_KEY, l_RRN, PAGE, PROMO_KEY, PROMO_R_CHILD, NEWPAGE), a
procedure that inserts l_KEY and l_RRN, causing overflow, creates a new page called
NEWPAGE, distributes the keys between the original PAGE and NEWPAGE, and determines
which key and RRN to promote. The promoted key and RRN are returned via the arguments
PROMO_KEY
and
PROMO_R_CHILD.
When
functions.
coded
in a real language,
The most obvious one
distributes the keys
between the
determines which key and
description of a simple
insertf)
uses a
number of support
which creates a new page,
page and the new page, and
split(),
original
RRN
split()
is
to
promote. Figure 8.25 contains a
is also encoded in C and
procedure, which
Pascal at the end of this chapter.
You
should pay careful attention to
only the key
is
promoted from
the
how
split()
working page
moves
all
of the
data.
Note
that
CHILD RRNs
back to PAGE and NEWPAGE. The RRS that is promoted
of NEWPAGE, since NEWPAGE is the right descendent from
are transferred
is
the
RRN
promoted key. Figure 8.26 illustrates the working page activity among
the working page, and the function arguments.
The version of splitf) described here is less efficient than might
sometimes be desirable, since it moves more data than it needs to. In
Exercise 17 you are asked to implement a more efficient version of split().
the
PAGE, NEWPAGE,
361
ALGORITHMS FOR B-TREE SEARCHING AND INSERTION
The Top Level
We
need
routine to
tie
together our
insert!
and
split(
procedures and to do some things that are not done by the lower-level
Our
routines.
Open
Read
driver
or create the B-tree
Create
It is
new
PAGE
root node
that the
FIGURE 8.26 The movement
Contents of
file,
do the following:
and identify or create the root page.
call \nsert(
to put the
tree.
routine driver
assumed
able to
keys to be stored in the B-tree. and
keys in the
The
must be
when
shown
RRN
insert(
splits the
in Fig. 8.27 carries
of the root node
is
current root page.
out these top-level tasks.
stored in the B-tree
file itself.
of data in splitO.
are copied to the working page.
PAGE
7D
K
Working page
^
w
I_KEY
into
(B)
and
I_RRN
(11) are inserted
working page.
^f
Contents of working page are divided
between PAGE and NEWPAGE, except
for the middle key (H). H promoted,
along with the
RRN
(12) of
t
PROMO RRN
NEWPAGE.
NEWPAGE
PAGE
PROMO KEY
7*-
362
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
MAIN PROCEDURE: driver
if the B-tree file exists, then
open B-tree file
else
create a B-tree file and place the first key in the root
get RRN of root page from file and store it in ROOT
get a key and store it in KEY
while keys exist
if (insert (ROOT, KEY, PR0M0_R_CHILD, PROMO.KEY) == PROMOTION) then
create a new root page with key := PR0M0_KEY, left
child := ROOT, and right child := PR0M0_R_CHILD
set ROOT to RRN of new root page
get next key and store it in KEY
endwhile
write RRN stored in ROOT back to B-tree file
close B-tree file
end MAIN PROCEDURE
FIGURE 8.27 Driver for building a B-tree.
if the file exists. If the file
does
exist, driver
opens
it
root node. If it does not exist, driver must create the
and gets the RRN of the
file and build an original
root page. Since a root must contain
at least one key, this involves getting
and placing it in the root. Next, driver
reads in the keys to be inserted, one at a time, and calls insertf) to insert the
keys into the B-tree file. If insertf) splits the root node, it promotes a key and
right child in PROMO_KEY and PROMO_R_CHILD, and driver uses
the
first
key
to be inserted in the tree
these to create a
8.10
new
root.
B-Tree Nomenclature
Before moving on to discuss B-tree performance and variations on the basic
B-tree algorithms, we need to formalize our B-tree terminology. Providing
careful definitions
of terms such
the properties that
must be present
as order
and
leaf enables us to state precisely
for a data structure to qualify as a B-tree.
This definition of B-tree properties, in turn, informs our discussion of
matters such as the procedure for deleting keys from a B-tree.
Unfortunately, the literature on B-t rge_s
terms relatin g to
Barges.
Reading
is
that literature
not uniform in
its
us e_ojl
and keeping up with new
363
B-TREE NOMENCLATURE
developments therefore require some flexibility and some background: The
reader needs to be aware of the different usages of some of the fundamental
terms.
For example, Bayer and McCreight (1972),
Comer
others refer to the order of a B-tree as the minimum
(1979),
and
number of keys
few
that can
be in a page of a tree. So, our initial sample B-tree (Fig. 8.16), which can
hold a maximum of seven keys per page, has an order of three, using Bayer
and McCreight's terminology. The problem with this definition of order is
that it becomes clumsy when you try to account for pages that hold a
maximum number of keys that is odd. For example, consider the following
question: Within the Bayer and McCreight framework, is the page of an
order three B-tree
full
when
it
contains six keys or
when
it
contains seven
keys?
Knuth (1973b) and
others have addressed the odd/even confusion
by
maximum number of descendents that
a page can have. This is the definition of order that we use in this text. Note
that this definition differs from Bayer and McCreight's in two ways: It
defining the order of a B-tree to be the
references a
maximum, not
minimum, and
it
counts descendents rather than
keys.
Use of Knuth's
number of keys in a
definition
must be coupled with the
B-tree page
descendents from the page.
is
always one
Consequently,
less
fact
than the
that the
number of
B-tree of order 8 has
maximum of seven keys per page. In general, given a B-tree
maximum number of keys per page is m 1.
When you split the page of a B-tree, the descendents
of order m, the
are divided as
between the new page and the old page. Conseq uently,
every page except the root and the leaves has at east ml 2 descendents
Expressed in terms of a ceiling function, we can say that the minimum
It follows that the minimum numnumber of descendents is \ m/2
ber of keys per page is
ro/2
1, so our initial sample B-tree has an
order of eight, which means that it ca n hold no m ore than seven kev s
per page and that all of the pages except the root contain at least thr ee
evenly
as possible
.__
1.
\J&pL
(The
other term that is used differently by different authors is leaf. Bayer
and McCreight refer to the lowest level of keys in a B-tree as the leaf level.
This is consistent with the nomenclature we have used in this text. Other
authors, including Knuth, consider the leaves of a B-tree to be one level
below the lowest level of keys. In other words, they consider the leaves to
be the actual data records that might be pointed to by the lowest level of
keys in the tree. We do not use this definition, sticking instead with the
notion of leaf
as the
lowest level of keys in the B-tree.
364
8.1
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
Formal Definition of B-Tree Properties
Given these definitions of order and leaf, we can formulate
statement of the properties of a B-tree of order m:
2.
Every page has a maximum of m descendents.
Every page, except for the root and the leaves, has
3.
The
4.
All the leaves
1.
at least [
precise
m/2~\ de-
scendents.
6.
8.12
two descendents (unless
appear on the same level.
at least
it is
a leaf).
nonleaf page with k descendents contains k 1 keys.
leaf page contains at least [ m/2~] - 1 keys and no more than
A
A
5.
root has
m -
keys.
Worst-case Search Depth
important to have a quantitative understanding of the relationship
between the page size of a B-tree, the number of keys to be stored in the
tree, and the number of levels that the tree can extend. For example, you
It
is
might
know
that
you need
to store 1,000,000 keys
and
nature of your storage hardware and the size of your keys,
to consider using a B-tree
of order 512
(maximum of 511
that,
given the
it is
reasonable
keys per page).
Given these two facts, you need to be able to answer the question, "In the
worst case, what will be the maximum number of disk accesses required to
locate a key in the tree?" This is the same as asking how deep the tree
will be.
We can answer this question by beginning with the observation that the
number of descendents from any level of a B-tree is one greater than the
number of keys contained at that level and all the levels above it. Figure 8.28
illustrates this relation for the tree
we
constructed earlier in this chapter.
T his tree contains 27 kevs fall the letters of the alphabet and S). If you co unt
me number of potential descendents trailing from the leaf level, you see that
there are 28 of them.
Next we need
to observe that
properties to calculate the
from any
of
level
we
can use the formal definition of B-tree
minimum number of descendents
B-tree of
some given
order. This
is
that can extend
of interest because
we are interested in the worst-case depth of the tree. The worst case occurs
when every page of the tree has only the minimum number of descendents.
In such a case
iimal breadth.
it
he keys are spread over
maximal height for the
tree
and
WORST-CASE SEARCH DEPTH
365
H N
ddd
ddddddd ddd ddd
dd
FIGURE 8.28 A B-tree with
Fo r
root page
keys can have
(A/
1)
T U V
X Y
dddddddd
dd
descendents from the
leaf level.
minimum number of descendents from the
two, so the second level of the tree contains only two pages.
B-tree o f order m, the
is
Ea cRof these pages,
in turn has at least
,
mil 1 de scendents.
he third
level,
then, co ntains
2
X [ ml2~\
pages. Since each of these pages, once again, has a
minimum of [ m/2~\
descendents, the general pattern of the relation between depth and the
minimum number of descendents
Minimum number
Level
1
0?
of descendents
(root)
x \ mll\
x fm/21 x [~m/2~|or 2 x
3
2 X \ml2~}
tf
takes the following form:
\
'
mil'}
4?
d-\
2 x fro/21
So.
in
general,
for
any
descendents extending fro
level
d of a B-tree,
thaf level
2
\^
x fm/2l d-\
the
minimum numbe r of
366
B-TREES AND OTHER TREE-STRUCTURED
Wejcnow
Let
level.
that
s call
ORGANIZATIONS
N keys
free with.
descendents from
N+
we know
than the
at
HesrenHe ntS from
that the
number
We
d.
descendents and the
of height d
a tree
its
leaf
can express the
minimum number
of
as
N+1>2X
since
has
the depth of the tree at the leaf level
between the
relationship
FILE
[ ml2~]
d-
number of descendents from any
cannot be
tree
for a worst-case tree of that depth. Solving for
d,
we
less
arrive
the following expression:
d
+ logrw2l
((N
l)/2).
This expression gives us an upper bound for the depth of
a B-tree
A/keys. Let's find the upper bound for the hypothetical tree that
at the start
of
this section: a tree
Substituting these specific
of order 512 that contains 1,000,000 keys.
numbers
d
<
with
we describe
into the expression,
we
find that
log 256 500000.5,
or
d
<
3.37.
So we can say that given 1,000,000 keys,
of no more than three levels.
8.13
B-tree of order 512 has a depth
Deletion, Redistribution, and Concatenation
Indexing 1,000,000 keys in no more than three levels of a tree is precisely
the kind of performance we are looking for. As we have just seen, this
performance
is
predicated on the B-tree properties
Every page except
is
coupled t o the r ules that
for the root
describe earlier; in
broad and shallow rathe r
particular, the ab:Qity_to_guarantee that B-trees are
than na rrow and dee p
we
state the
and the leaves has
follow ing:
at least
\~
m/2~\ de-
scendents;
A
A
nonleaf page with k descendents contains k
leaf
We
page contains
at
least [ ml2~\
keys; and
keys and no more than
keys.
have already seen that the process of page
these properties are maintained
when new keys
We
splitting guarantees that
are inserted into the tree.
need to develop some kind of equally reliable guarantee that
proper ties are maintained when keys are deleted from the tree.
jJiese
-Working through some simple deletion situations by hand helps us
a kev can result in several different
demonstrate that the deletion of
367
DELETION, REDISTRIBUTION, AND CONCATENATION
of these situations and the associated
situations. Figure 8.29 illustrates each
response in the course of several deletions from an order six B-tree.
The
simplest situation
is
illustrated in case
cause the contents of page 5 to drop below the
Deleting the key J does not
minimum number of keys.
Consequently, deletion involves nothing more than removing the key from
the page and rearranging the keys within the page to close up the space.
M (case
2) is more complicated. If we simply remove the
becomes very difficult to reorganize the tree to maintain
its B-tree structure. Since this problem can occur whenever we delete a key
from a nonleaf page, we always delete keys only from leaf pages. If a key
Deleting the
M from the root,
to be deleted
is
swap
its
it
not in a
lea f, there
is
an easy
wav
to get
it
into a leaf:
We
immediate successor, which is guaranteed to be in a lea f,
then delete it immediately from the leaf. In our example, we can swap the
with tne iV in page b, then delete 'the
from page 6. This simple
operation does not put the
out of order, since all keys in the subtree of
which Nis a part must be greater than N. (Can you see why this is the case?)
In case 3 we delete R from page 7. If we simply remove R and do
nothing more, the page that it is in has only one key. The minimum
number of keys for the leaf page of an order six tree is
it
wit h
r6/2~|Therefore,
we
2.
have to take some kind of action to correct
this
condition. Since the neighboring page 8 (called a sibling since
parent) has
more than
the
minimum number
it
underflow
has the
same
of keys, the corrective action
between the pages. Redistribution
ust
change in the key that is in the parent page so it continues to
act as a separator between the lower-level pages. In the example, we move
the U and Finto page 7, and move I^into the separator position in page 2.
The deletion of A in case 4 results in a situation that cannot be resolved
by redistribution. Addressing the underflow in page 3 by moving keys
from page 4 only transfers the underflow condition. There are not enough
keys to share between two pages. The solution to this is concatenation,
combining the two pages and the key from the parent page to make a single
consists of redistributing the keys
also result in a
full
page.
Concatenation
is
essentially the reverse
of splitting. Like
splitting,
it
can
propagate upward through the B-tree. Just as splitting promotes a key,
concatenation must involve demotion of keys, and this can in turn cause
underflow
in the parent page. This
is
just
what happens
in
our example.
Our concatenation of pages 3 and 4 pulls the key D from the parent page
down to the leaf level, leading to case 5: The loss of the D from the parent
page causes it, in turn, to underflow. Once again, redistribution does not
solve the problem, so concatenation
must be used.
1: No action.
Delete J from page 5. Since page 5 has more
than the minimum number of keys,
Case
be removed without reorganization.
J can
Case
2:
Swap with immediate
Swap
X Y
X Y
successor.
M (page 0) with N (page
and then delete M from page
Delete
6),
6.
Case
3:
Redistribution.
Delete R. Underflow occurs. Redistribute keys
among pages
2, 7,
and
8 to restore balance
between leaves.
Promote
move U and V
into page
7.
u V
FIGURE 8.29 Six situations that can occur during deletions.
368
,i
Case
4:
Concatenation.
Delete A. Underflow occurs, but
it cannot be
addressed by redistribution. Concatenate the
keys from pages 3 and 4, plus the D from
page 1 into one page.
Underflow
u V
X Y
X Y
New
page
3:
Underflow propagates upward.
1 has underflow. Again, we cannot
redistribute, so we concatenate.
Case
Now
5:
page
Underflow
moves up to
here
C D E F
Case
6:
Height of tree decreased.
Since the root contains only one key,
it is
absorbed into the new root.
c D E F
U V
X Y
369
370
B-TREES AND OTHER TREE-STRUCTURED
Note
ORGANIZATIONS
propagation of the underflow condition does not
the
that
FILE
necessarily imply the propagation of concatenation. If page 2
(Q and W) had
contained another key, then redistribution, not concatenation, would be
used to resolve the underflow condition at the second level of the tree.
Case 6 shows what happens when concatenation propagates all the way
The concatenation of pages 1 and 2 absorbs the only key in the
root page, decreasing the height of the tree by one level.
The steps involve d in deleting keys from a B-tree can be su mmarized as
to the root.
follow
If the
key
successor,
to be deleted
which
Q1
Delete the key.
fuA
If the leaf
>~v
(Aj
now
further action
If the leaf
swap
a leaf,
contains at least the
it
with
its
immediate
minimum number of keys, no
required.
is
now
not in
is
in a leaf.
is
contains one too few keys, look at the
left
and right
siblings.
a.
If a sibling
has
more than
the
minimum number
of keys, redis-
tribute.
b.
If neither sibling has
two
leaves and the
more than
If leaves are concatenated,
If the last
the
minimum,
median key from the parent
apply steps
key from the root
is
3-6
concatenate the
into
one
leaf.
to the parent.
removed, then the height of the
tree
decreases.
8.13.1 Redistribution
Unlike concatenation, which is a kind of reverse split, redistribution is a
idea. Our insertion algorithm does not involve operations analogous to
new
redistribution.
Redistribution differs from both splitting and concatenation in that
does not propagat e.
It is
guaranteed to have
strictly local effects.
the term sibling implies that the pages have the
Note
it
that
same parent page. If there are
adjacent but do not have the
two nodes at the leaf level that are logically
same parent (for example, IJK and NOP in the
tree at the top
of Fig. 8.29),
these nodes are not siblings. Redistribution algorithms are generally written
moving keys between nodes that are not siblings,
even when they are logically adjacent. Can you see the reasoning behind
so they do not consider
this restriction?
Another difference between redistribution on the one hand a nd
is that there is no necessary, iixe3
concatenation and splitting on the other
prescription for
how
the keys should be rearranged.
single deletion in a
REDISTRIBUTION DURING INSERTION: A
WAY
37
TO IMPROVE STORAGE UTILIZATION
pr operly formed B-tree cannot cause an underflow of
more than one key
moving only
.
Therefore, redistribution can restore the B-tree properties by
one key from a sibling into the page that has underflowed, even if the
distribution of the keys between the pages is very uneven. Suppose, for
example, that we are managing a B-tree of order 101. The minimum
number of keys that can be in a page is 50, the maximum is 100. Suppose
we have one page that contains the minimum and a sibling that contains the
maximum. If a key is deleted from the page containing 50 keys, an
underflow condition occurs. We can correct the condition through
redistribution by moving one key, 50 keys, or any number of keys that falls
between 1 and 50. The usual strategy is to divide the keys as evenly as
possible between the pages. In this instance that means moving 25 keys.
8.14
Redistribution during Insertion: A
Storage Utilization
As you may
recall,
to redistribution; splitting
as
to
Improve
B-tree insertion does not require an operation analogous
is
able to account for
This does not mean, however, that
during insertion
Way
it is
not
an option, particularly since
algorithms must already include
all
instances of overflow.
desirable to
a set
use redistribution
of B-tree maintenance
redistribution procedure to support
Given that a redistribution procedure is already present, what
advantage might we gain by using it as an alternative to node splitting?
deletion.
Redistribution during insertion
is
way of
avoiding,
postponing, the creation of new pages. Rather than splitting
creating
two approximately
or
a full
at
least
page and
half-full pages, redistribution lets us place
some
of the overflowing keys into another page. The use of redistribution in place
of splitting should therefore tend to make a B-tree more efficient in terms
of its utilization of space.
It is possible to quantify this efficiency of space utilization by viewing
the amount of space used to store information as a percentage of the total
amount of space required to hold the B-tree. After a node splits, each of the
two resulting pages is about half full. So, in the worst case, space utilization
in a B-tree using two-way splitting is around 50%. Of course, the actual
degree of space utilization
lias
shown
approaches
is
better than this worst- case ligure.
Yao
(1978)
of relatively large order, space Trtilizatioli
theoretical average of about 69% if insertion is handled
that, for large trees
a
through two-wav splittin gs
The idea of using redistribution as an alternative to splitting when
possible, splitting a page only when both of its siblings are full, is
introduced in Bayer and McCreight's original paper (1972). The paper
372
B-TREES AND OTHER TREE-STRUCTURED
includes
some experimental
of
in a space utilization
insertions.
possible,
testing
When
ORGANIZATIONS
results that
67%
show
that
two-way
splitting results
for a tree of order 121 after 5,000
random
was repeated, using redistribution when
increased to over 86%. Subsequent empirical
the experiment
space utilization
by Davis
FILE
(1974) (B-tree of order 49) and Crotzer (1975) (B-tree of
order 303) also resulted in space utilization exceeding
85% when
redistri-
bution was used.
These findings and others suggest that any serious
application of B-trees to even moderately large files should implement
insertion procedures that handle overflow through redistribution when
possible.
8.15
B* Trees
and amplification of work on B-trees
In his review
Knuth (1973b)
in 1973,
new
extends the notion of redistribution during insertion to include
for splitting.
He calls
afi* tree.
Consider
the resulting variation
system
in
which we
rules
on the fundamental B-tree form
are postponing
splitting
we
redistribution, as outlined in the preceding section. If
through
are considering
any page other than the root, we know that when it finally is time to split,
the page has at least one sibling that is also full. This opens up the possibility
of a two-to-three split rather than the usual one-to-two or two-way split.
Figure 8.30 illustrates such a
The important
split.
aspect of this two-to-three split
is
that
that are each about two-thirds full rather than just half
possible to define a
new
kind of B-tree, called
B*
it
results in pages
full.
tree,
This makes it
which has the
following properties:
maximum
1.
Every page has
2.
Every page except for
the root
of m descendents.
and the leaves has
at least
(2m
l)/3 de-
scendents.
3.
The
4.
All the leaves appear
5.
6.
A
A
root has
at least
two descendents (unless
on the same level.
it is
a leaf).
nonleaf page with k descendents contains k 1 keys.
leaf page contains at least \_(2m 1)/3J keys and no more than
keys.
of properties and the set we define
for a conventional B-tree are in rules 2 and 6: a B* tree has pages__tiiat
contain a minimum of (2m - l)/3j keys. This new property, o f course,
The
critical
changes between
this se t
affects
procedures for deletion and redistribution.
BUFFERING OF PAGES: VIRTUAL B-TREES
373
Original tree:
A C D F
H K
T V X
Two-to-three-split:
H K M
After the insertion of the
key B.
C D
FIGURE 8.30 A two-to-three
T V X
split.
To impleme nt B * tree proced ures, o ne must also deal with the question
of sj>h tting. the root, which, bv definition, never has a sibling. If there is no
sibling, no two-to-three split is possible. Knu th suggests allowing the roo t
grow to a size larger than the other pages so. when it does split, it can
producetwo pages that are each about tw o-thirds full. This sugges tion has
to
the advantage of ensuring that
Tree charact eristic's.
However,
all
it
pa,g e s
below the root
level
adhere to
B*
has the disadvantage of requiring that the
page that is larger than all the others. Another
solution is to handle the splitting of the root as a conventional one-to-two
split. This second solution avoids any special page-handling logic. On the
other hand, it complicates deletion, redistribution, and other procedures
procedures be able to handle
must be sensitive to the minimum number of keys allowed in a page.
Such procedures would have to be able to recognize that pages descending
from the root might legally be only half full.
that
8.16
Buffering of Pages: Virtual B-Trees
We
have seen
very
that,
given some additional refinements, the B-tree can be
efficient, flexible
ties after
storage structure that maintains
its
balanced proper-
repeated deletions and insertions and that provides access to any
374
B-TREES AND OTHER TREE-STRUCTURED
key with just
aspects, as
few disk
we have
so
using this structure to
a
FILE
accesses.
far,
full
However, focusing on just
the structural
can cause us inadvertently to overlook ways of
advantage. For example, the fact that
depth of three levels does not
accesses to retrieve keys
ORGANIZATIONS
at all
from pages
mean
that
we
B-tree has
need to do three disk
We can do much better
at the leaf level.
than that.
Obtaining better performance from B-trees involves looking in a
way at our original problem. We needed to find a way to make
efficient use of indexes that are too large to be held entirely in RAM. Up to
this point we have approached this problem in an all-or-nothing way: An
index has been either held entirely in RAM, organized as a list or binary
tree, or has been accessed entirely on secondary store, using a B-tree
precise
structure. But, stating that
not imply that
we
For example, assume
records and that
we
cannot hold
cannot hold some of
we
we have
it
ALL
of an index in
RAM does
there.
an index that contains
cannot reasonably use more than 256
megabyte of
RAM
of
for
any given time. Given a page size of 4 K, holding around
64 keys per page, our B-tree can be contained in three levels. We can reach
any one of our keys in no more than three disk accesses. That is certainly
acceptable, but why should we settle for this kind of performance? Why not
try to find a way to bring the average number of disk accesses per search
down to one disk access or less?
Thinking of the problem strictly in terms of physical storage structures,
retrieval averaging one disk access or less sounds impossible. But,
remember, our objective was to find a way to manage our megabyte of
index within 256 K of RAM, not within the 4 K required to hold a single
page of our tree.
We know that every search through the tree requires access to the root
page. Rather than accessing the root page again and again at the start of
and just keep it there.
every search, we could read the root page into
requirement from 4 K to 8 K, since we
This strategy increases our
need 4 K for the root and 4 K for whatever other page we read in, but this
is still much less than the 256 K that are available. This very simple strategy
reduces our worst-case search to two disk accesses, and the average search
index storage
at
RAM
RAM
to
under two accesses (keys in the root require no disk
require one access).
access; keys at the
first level
This simple,
keep-the-root strategy suggests
an important,
general approach: Ra ther than jnsr holding the root page in
create a page buffer
ho 1rl
cnt1 ir
number of B-tree
more
RAM, we
pages, perhaps
can
5, 1CL
or
more. As we read pages in from the disk in response to user requests, we fill
if we
"up trie buffer. Then, when a page is requested, we access it from
read
we
then
in
RAM,
can, thereby avoiding a disk access. If the page is not
RAM
375
BUFFERING OF PAGES: VIRTUAL B -TREES
into the buffer from secondary storage, replacing one of the pages that
was previously there. A B-tree that uses a RAM buffer in this way is
sometimes referred to as a virtual B-tree.
it
8.16.1 LRU Replacement
Clearly, such a buffering
request a page that
is
scheme works only if we are more likely to
one that is not. The process of
in the buffer than
accessing the disk to bring in a page that
a
is
page fault. There are two causes of page
1.
We
2.
It
not already in the buffer
is
called
faults:
have never used the page.
was once
in the buffer
but has since been replaced with
new
page.
The first cause of page faults is unavoidable: If we have not yet read in
and used a page, there is no way it can already be in the buffer. But the
second cause is one we can try to minimize through buffer management.
The critical management decision arises when we need to read a new page
into a buffer that is already full: Which page do we decide to replace?
One common approach is to replace the page that was least recently
used; this
is
called
LRU
replacing the page that
page
is
was
always read in
replacing the root,
replacement. Note that this
first,
which
different
is
from
read into the buffer least recently. Since the root
is
simply replacing the oldest page results in
an undesirable outcome. Instead, the
LRU
method keeps track of the actual requests for pages. Since the root is
requested on every search, it seldom, if ever, is selected for replacement.
The page to be replaced is the one that has gone the longest time without a
request for use.
Some
research
number of pages
by Webster
that
(1980)
shows the
replacement strategy. Table 8.1 summarizes
TABLE
8.1
Effect of using
more buffers with
a simple
of increasing the
Average Accesses per Search
3.00
1.71
LRU
small but representative
LRU replacement
Buffer Count
Number of keys = 2,400
Total pages = 140
Tree height = 3 levels
effect
can be held in the buffer area under an
strategy
10
1.42
20
0.97
376
B-TREES AND OTHER TREE-STRUCTURED
portion of Webster's results.
search given different
LRU
using a simple
FILE
It lists
ORGANIZATIONS
the average
numbers of page
number of disk
accesses per
These results are obtained
replacement strategy without accounting for page
buffers.
height.
B+
Webster's study was conducted using
rather than simple
+
B-trees. In the next chapter, where we look closely at B trees, you see that
+
the nature of B
trees accounts for the fact that, given one buffer, the
+
average search length is 3.00. With B trees, all searches must go all the way
+
to the leaf level every time. The fact that Webster used B
trees, however,
trees
does not detract from the usefulness of his results
as
an illustration of the
positive impact of page buffering. Keeping less than
15% of
the tree in
RAM (20 pages out of the total 140) reduces the average number of accesses
per search to less than one.
simple B-tree, since not
Note
that
all
The
results are
decision to use
the
we
even more dramatic with
searches have to proceed to the leaf level.
LRU
replacement
is
based on the
more likely to need a page that we have used
need a page that we have never used or one that we
used some time ago. If this assumption is not valid, then there is absolutely
no reason to preferentially retain pages that were used recently. The term
for this kind of assumption is temporal locality. We are assuming that there
is a kind of clustering of the use of certain pages over time. The hierarchical
nature of a B-tree makes this kind of assumption reasonable.
For example, during redistribution after overflow or underflow, we
access a page and then access its sibling. Because B-trees are hierarchical,
accessing a set of sibling pages involves repeated access to the parent page
in rapid succession. This is an instance of temporal locality; it is easy to see
assumption that
recently than
how
it is
we
are
are to
related to the tree's hierarchy.
8.16.2 Replacement Based on Page Height
There
is
another,
more
direct
way
to use the hierarchical nature
of the
B-tree to guide decisions about page replacement in the buffers.
simple, keep-the-root strategy exemplifies this alternative:
the pages that occur at the highest levels of the tree. Given a
Our
Always retain
larger amount
of buffer space, it might be possible to retain not only the root, but also all
of the pages at the second level of a tree.
Let's explore this notion by returning to a previous example in which
and a 1-megabyte index. Since our page
we have access to 256 K of
size is 4 K, we could build a buffer area that holds 64 pages within the
area. Assume that our 1 megabyte worth of index requires around 1.2
megabytes of storage on disk (storage utilization = 83%). Given the 4 K
page size, this 1.2 megabytes requires slightly more than 300 pages. We
RAM
RAM
377
PLACEMENT OF INFORMATION ASSOCIATED WITH THE KEY
assume
It
that,
on the average, each of our pages has around 30 descendents.
follows that our three-level tree has, of course,
followed by 9 or 10 pages
level,
Using
at
page
a single
the second level, with
all
at the
root
the remaining
page replacement strategy that always retains
the higher-level pages, it is clear that our 64-page buffer eventually contains
the root page and all the pages at the second level. The approximately 50
pages
at the leaf level.
remaining buffer slots are used to hold leaf-level pages. Decisions about
which of these pages to replace can be handled through an LRU strategy.
For many searches, all of the pages required are already in the buffer; the
search requires no disk accesses.
possible to bring the average
to a
number
that
is less
easy to see how, given a sizable buffer,
number of disk accesses per search down
It is
it is
than one.
Webster's research (1980) also investigates the effect of taking page
height into account, giving preference to pages that are higher in the tree
when
it
comes time
Augmenting the LRU
to
decide which pages
to
keep
page height reduces the average number of accesses, given
from 1.42
accesses per search
8.16.3 Importance
It is
difficult to
scheme
down
the
buffers.
10-page buffer,
to 1.12 accesses per search.
of Virtual B-Trees
overemphasize the importance of including
page buffering
into any implementation of a B-tree index structure. Because the
B-tree structure itself is so interesting and powerful,
trap of thinking that the B-tree organization
the
in
strategy with a weighting factor that accounts for
problem of accessing large indexes
we have
secondary storage. As
required to handle large indexes.
amount of memory
is itself a
way
We
easy to
fall
into the
sufficient solution to
must be maintained on
that
emphasized, to
sight of the original problem: to find a
it is
fall
into that trap
to reduce the
is
to lose
amount of memory
did not, however, need to reduce the
amount required for a single index page. It is
enough memory to hold a number of pages. Doing
to the
usually possible to find
so can dramatically increase system performance.
8.17
Placement
of Information
Early in this chapter
we
Associated with the Key
focused on the B-tree index
itself,
setting aside
consideration of the actual information associated with the keys.
any
We
paraphrased Bayer and McCreight and stated that "the associated information
is
of no further interest."
But, of course, in any actual application the associated information
in fact, the true object
of interest. Rarely do
we ever want
is,
to index keys just
378
B-TREES AND OTHER TREE-STRUCTURED
to be able to find the keys
themselves.
we
associated with the key that
ORGANIZATIONS
FILE
It
want
really
usually the information
is
to find. So, before closing our
discussion of B-tree indexes,
it is important to turn to the question of where
and how to store the information indexed by the keys in the tree.
Fundamentally, we have two choices. We can
Store the information in the B-tree along with the key; or
Place the information in a separate
the key with a relative record
within the index
file:
number
we
references the location of the information in that separate
The
advantage that the
couple
or byte address pointer that
file.
approach has over the second is that
The information
is right there with the key.
However, if the amount of information
associated with each key is relatively large, then storing the information
with the key reduces the number of keys that can be placed in a page of the
B-tree. As the number of keys per page is reduced, the order of the tree is
distinct
once the key
reduced,
is
found, no
and the
tree
more
first
disk accesses are required.
tends
become
to
since
taller
there
fewer
are
descendents from each page. So, the advantage of the second method
given associated information that has
that,
is
long length relative to the
length of a key, placing the associated information elsewhere allows us to
build a higher-order and therefore possibly shallower tree.
we need
For example, assume
1,000 keys and associated
to index
information records. Suppose that the length required to store
associated information
is
128 bytes. Furthermore, suppose that
the associated information elsewhere,
key and
if
we can store just the key and
to the associated information in only 16 bytes.
Given
a B-tree
we
a
its
store
pointer
page that had
512 bytes available for keys and associated information, the two fundamental storage alternatives translate into the following orders of B-trees:
Information stored with key: four keys per page
order
Pointer stored with key: 32 keys per page
order 33
Using the formula
worst-case
developed
for
finding
w/key)
d(info elsewhere)
we
and
tree.
depth
of B-trees
earlier:
d (in fo
So, if
the
five tree;
** +
500.5
6.66
k)g 17 500. 5
3.
log3
19
store the information with the keys, the tree has a worst-case
levels. If we store the information elsewhere, we end up
reducing the height of the worst-case tree to three. Even though the
depth of six
method costs us one
number of accesses to
additional indirection associated with the second
second method
record in the worst case.
access, the
a
still
reduces the total
disk
find
379
VARIABLE-LENGTH RECORDS AND KEYS
general,
In
then,
the decision about
where
to store the associated
information should be guided by some calculations that compare the depths
of the trees that result. The critical factor that influences these calculations
is the ratio of overall record length to the length of just a key and pointer.
If
you can put many key/pointer
key/record
tion
8.18
from
pair,
it is
pairs in the area required for a single, full
probably advisable to remove the associated informa-
the B-tree and put
it
in a separate
file.
Variable-length Records and Keys
In
many
applications the information associated with a key varies in length.
Secondary indexes referencing inverted
One way
this.
information in
to
a
handle
separate,
this
lists
variability
is
to
variable-length record
contain a reference to the information in this other
is
to allow a variable
Up
number of keys and
example of
are an excellent
place
associated
the
would
file;
the B-tree
file.
Another approach
records in a B-tree page.
we have regarded B-trees as being of some order m.
fixed maximum and minimum number of keys that it can
to this point
Each page has
The notion of a variable-length record, and, therefore, a
variable number of keys per page, is a significant departure from the point
of view we have developed so far. A B-tree with a variable number of keys
per page clearly has no single, fixed order.
The variability in length can also extend to the keys themselves as well
as to entire records. For example, in a file in which people's names are the
legally hold.
keys,
we might
choose to use only
as
much
space as required for a name,
rather than allocate a fixed-size field for each key.
chapters,
to put
implementing
a structure
many more names
in a
internal fragmentation. If
we
fields
given amount of space since
can put
larger
number of descendents from
fewer
levels.
Accommodating
As we saw
with variable-length
more keys
it
in earlier
can allow us
does
in a page, then
page and, very probably,
away with
we have
a tree
with
this variability in length means using a different kind
of page structure. We look at page structures appropriate for use with
+
variable-length keys in detail in the next chapter, where we discuss B
trees. We also need a different criterion for deciding when a page is full and
when it is in an underflow condition. Rather than use a maximum and
minimum number of keys per page, we need to use a maximum and
minimum number of bytes.
Once the fundamental mechanisms for handling variable-length keys or
records are in place, interesting new possibilities emerge. For example, we
might consider the notion of biasing the key promotion mechanism so the
380
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
shortest variable-length keys (or key/record pairs) are
preference to longer keys.
The
idea
is
that
we want
promoted upward
to
in
have pages with the
numbers of descendents up high in the tree, rather than at the leaf
Branching out as broadly as possible as high as possible in the tree
tends to reduce the overall height of the tree. McCreight (1977) explores
this notion in the article, "Pagination of B* Trees with Variable-Length
Records."
The principal point we want to make with these examples of variations
on B-tree structures is that this chapter introduces only the most basic forms
of this very useful, flexible file structure. Actual implementations of B-trees
do not slavishly follow the textbook form of B-trees. Instead, they use
many of the other organizational techniques we study in this book, such as
variable-length record structures, in combination with the fundamental
B-tree organization to make new, special-purpose file structures uniquely
suited to the problems at hand.
largest
level.
SUMMARY
We begin this chapter by picking up the problem we left unsolved at the end
of Chapter
Simple, linear indexes
6:
RAM memory,
that
work
well
if
they are held in electronic
but are expensive to maintain and search
secondary storage
is
most evident
in
two
if
they are so big
The expense of using
they must be held on secondary storage.
areas:
Sorting of the index; and
Searching, since even binary searching required
more than just two
or three disk accesses.
We
first
address the question of structuring an index so
it
can be kept
in order
without sorting.
We use tree structures
we need
a balanced tree to
ensure that the tree does not become overly deep
after repeated
random
insertions.
We
see that
to
do
AVL
this,
trees
discovering that
provide
way of
amount of overhead.
Next we turn to the problem of reducing the number of disk accesses
required to search a tree. The solution to this problem involves dividing the
balancing a binary tree with only a small
tree into pages, so a substantial portion
of the
tree can
be retrieved with
Paged indexes let us search through very large numbers
of keys with only a few disk accesses.
Unfortunately, we find that it is difficult to combine the idea of paging
of tree structures with the balancing of these trees by AVL methods. The
most obvious evidence of this difficulty is associated with the problem of
selecting the members of the root page of a tree or subtree when the tree is
built in the conventional top-down manner. This sets the stage for
single disk access.
SUMMARY
work on
introducing Bayer and McCreight's
B-trees,
paging and balancing dilemma by starting from the leaf
which solves the
level, promoting
keys upward as the tree grows.
Our
discussion of B-trees begins with examples of searching, insertion,
how B-trees grow while maintaining
Next we formalize our description of B-trees.
and promotion to show
splitting,
balance in a paged structure.
This formal definition permits us to develop
worst-case B-tree depth.
The formal
on developing deletion procedures
keys are removed from a tree.
Once
we
formula for estimating
work
when
description also motivates our
that maintain the B-tree properties
the fundamental structure and procedures for B-trees are in place,
refining and improving on these ideas. The first set of
improvements involves increasing the storage utilization within B-trees. Of
begin
course, increasing storage utilization can also result in a decrease in the
height of the tree, and therefore in improvements in performance.
We
find
by sometimes redistributing keys during insertion, rather than splitting
pages, we can improve storage utilization in B-trees so it averages around
85%. Carrying our search for increased storage efficiency even farther, we
find that we can combine redistribution during insertion with a different
that
kind of splitting to ensure that the pages are about two-thirds
than only one-half
full
after the split.
Trees using
redistribution and two-to-three splitting are called
Next we turn
to the matter
this
B*
combination of
trees.
of buffering pages, creating
a virtual B-tree.
We note that the use of memory is not an all-or-nothing choice:
are too large to
fit
secondary storage.
then
We
we
memory do
If we hold pages
not have to be accessed
into
rather
full
Indexes that
entirely
that are likely to be reused in
can save the expense of reading these pages in from the disk again.
develop two methods of guessing which pages are to be reused.
method
from
RAM,
uses the height of the page in the tree to decide
which pages
One
to keep.
Keeping the root has the highest priority, the root's descendents have the
The second method for selecting pages to keep in
RAM is based on recentness of use: We always replace the least-recentlyused (LRU) page, retaining the pages used most recently. We see that it is
possible to combine these methods, and that doing so can result in the
ability to find keys while using an average of less than one disk access per
next priority, and so on.
search.
We
then turn to the question of where to place the information
associated with a key in the B-tree index. Storing
it
with the key
is
attractive
same as finding the information;
required. However, if the associated
because, in that case, finding the key
is
no additional disk accesses are
information takes up a lot of space,
the
it
can reduce the order of the
thereby increasing the tree's height. In such cases
store the associated information in a separate
it is
file.
tree,
often advantageous to
381
382
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
We
close the chapter with a brief look at the use of variable-length
records within the pages of a B-tree, noting that significant savings in space
and consequent reduction
in the height
variable-length records.
The modification of
many
variations
from the use of
tree can result
the basic textbook B-tree
of variable-length records is just one example
that are used in real-world implementa-
definition to include the use
of the
of the
on B-trees
tions.
KEY TERMS
AVL
tree. A height-balanced (HB(1)) binary tree in which insertions
and deletions can be performed with minimal accesses to local nodes.
AVL trees are interesting because they keep branches from getting
overly long after many random insertions.
B-tree of order m. A multiway search tree with these properties:
1.
2.
Every node has a maximum of m descendents.
Every node except the root and the leaves has
at
least
|~w/2l
descendents.
3.
The
4.
All of the leaves appear
5.
6.
A
A
root has
two descendents (unless
on the same level.
at least
it is
nonleaf page with k descendents contains k
leaf
page contains
at least
[ ml2~\
a leaf).
keys.
keys and no more than
keys.
B-trees are built
pages always
upward from
the leaf level, so creation of
new
starts at the leaf level.
The power of B-trees
lies in
the facts that they are balanced (no
overly long branches); they are shallow (requiring few seeks); they
accommodate random
deletions and insertions at a relatively
while remaining in balance; and they guarantee
at least
50%
low
cost
storage
utilization.
B*
which each node is at least two-thirds full.
provide better storage utilization than do B-trees.
Concatenation. When a B-tree node underflows (becomes less than
50% full), it sometimes becomes necessary to combine the node with
tree.
B*
special B-tree in
trees generally
an adjacent node, thus decreasing the total number of nodes in the
tree. Since concatenation involves a change in the number of nodes
in the tree, its effects can require reorganization at many levels of the
tree.
Height-balanced tree.
each node there is a
tree structure
limit to the
with
a special
property: For
amount of difference
that
is
allowed
EXERCISES
among the heights of any of the node's subtrees. An HB(k) tree allows subtrees to be k levels out of balance. (See AVL tree.)
Leaf of a B-tree. A page at the lowest level in a B-tree. All leaves in a
B-tree occur at the same level.
Order of a B-tree. The maximum number of descendents that a node
in the B-tree
can have.
Paged index. An index that is divided into blocks, or pages, each of
which can hold many keys. The use of paged indexes allows us to
search through very large
numbers of keys with only
few disk
ac-
cesses.
Promotion of a key. The movement of a key from one node
into a
node (creating the higher-level node, if necessary) when
the original node becomes overfull and must be split.
Redistribution. When a B-tree node underflows (becomes less than
50% full), it may be possible to move keys into the node from an
adjacent node with the same parent. This helps ensure that the 50%full property is maintained. When keys are redistributed, it becomes
higher-level
necessary to alter the contents of the parent as well. Redistribution,
as opposed to concatenation, does not involve creation or deletion of
nodes
its effects are entirely local. Redistribution can also often be
used as an alternative to splitting.
Splitting. Creation of two nodes out of one because the original node
becomes overfull. Splitting results in the need to promote a key to a
higher-level node to provide an index separating the two new nodes.
Virtual B-tree. A B-tree index in which several pages are kept in
in anticipation of the possibility that one or more of them will be
needed by a later access. Many different strategies can be applied to
replacing pages in
when virtual B-trees are used, including the
least-recently-used strategy and height-weighted strategies.
RAM
RAM
EXERCISES
1.
Balanced binary trees can be effective index structures for
part or
all
when
RAM-based
become so large that
of them must be kept on secondary storage. The following
indexing, but they have several drawbacks
they
questions should help bring these drawbacks into focus, and thus reinforce
the need for an alternative structure such as the B-tree.
a.
There are two major problems with using binary search to search
simple sorted index on secondary storage: The number of disk ac-
cesses
is
larger than
index sorted
is
we would
substantial.
search tree alleviate?
like;
and the time
it
takes to keep the
Which of the problems does
binary
383
384
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
b.
Why
c.
In
is it
important to keep search trees balanced?
is an AVL tree better than a simple binary search
what way
tree?
d.
Suppose you have
a file
with 1,000,000 keys stored on disk
completely
full,
what
maximum number
the
is
the tree
is
balanced binary search
paged
in the
manner
tree. If the tree is
in a
not paged,
of accesses required to find
illustrated in Fig. 8.12, but
key?
If
with each
page able to hold 15 keys and to branch to 16 new pages, what is the
maximum number of accesses required to find a key? If the page size
is increased to hold 511 keys with branches to 512 nodes, how does
the maximum number of accesses change?
e. Consider the problem of balancing the three-key-per-page tree in
Fig. 8.13
by rearranging the pages.
Why
is it
difficult to create a
tree-balancing algorithm that has only local effects?
size increases to a
more
become difficult to guarantee that each of the pages
some minimum number of keys?
g.
Although B-trees
downward from
search trees for external searching, binary trees are
3.
Why
Describe the necessary parts of
differ
from an
is
a leaf
still
commonly
this so?
node of a B-tree.
How
does
a leaf
internal node?
Since leaf nodes never have children,
pointer fields in a leaf node
need for pointer
the top.
are generally considered superior to binary
used for internal searching.
2.
contains at least
Explain the following statement: B-trees are built upward from
f.
the bottom, whereas binary trees are built
node
When the page
why does it
512 keys),
likely size (such as
it
might be possible
to use the
to point to data records. This could eliminate the
fields to data records in the internal nodes.
Why? What
are
the implications of doing this in terms of storage utilization and retrieval
time?
4.
sets
Show
the B-trees of order four that result
from loading
the following
of keys in order.
a.
b.
c.
d.
C GJ X
CGJXNSUOAEBHI
CGJXNSUOAEBHIF
CGJXNSUOAEBHIFKLQRTVUWZ
Figure 8.23 shows the pattern of recursive calls involved in inserting a
$ into the B-tree in Fig. 8.22. Suppose that subsequent to this insertion, the
character [ is inserted after the Z. (The ASCII code for [ is greater than the
5.
ASCII code
for Z.)
Draw
a figure similar to Fig.
8.23 which shows the
pattern of recursive calls required to perform this insertion.
385
EXERCISES
6.
Given
B-tree of order 256
What is
What is
a.
b.
the
the
maximum number
minimum number
of descendents from
of descendents from
page?
page (ex-
cluding the root and leaves)?
d.
What
What
e.
How many
c.
is
the
is
the
minimum number of descendents from
minimum number of descendents from
keys are there on
the root?
a leaf?
nonleaf page with 200 descen-
dents?
f.
What
the
is
maximum
depth of the tree
if
it
contains 100,000
keys?
Using
method
to that used to derive the formula for
formula for best case, or minimum depth, for an
order m B-tree with
keys. What is the minimum depth of the tree
described in the preceding question?
7.
similar
worst-case depth, derive
Suppose you have a B-tree index for an unsorted file containing
data
where each key has stored with it the RRN of the corresponding
record. The depth of the B-tree is d. What are the maximum and minimum
numbers of disk accesses required to
8.
records,
a.
Retrieve
b.
Add
c.
Delete
d.
Retrieve
Assume
arrived at
9.
Show
deleted
record;
a record;
a
record; and
all
records from the
page buffering
your answer.
that
is
file
in sorted order.
not used. In each case, indicate
how you
the trees that result after each of the keys A, B, Q, and
from the following B-tree of order
is
five.
D H
A B C
r
F
K L
N O
\v
386
10.
B-TREES AND OTHER TREE-STRUCTURED
A common
unless
11.
belief about B-trees
100%
it is
full.
Discuss
FILE
ORGANIZATIONS
that a B-tree cannot
is
grow deeper
this.
Suppose you want to delete
key from
node
in a B-tree.
You
look
at
the right sibling and find that redistribution does not work; concatenation
would be
12.
What
Do you
it
introduce?
compare with
13.
What
is
look to the left and see that redistribution
choose to concatenate or redistribute?
difference
the
is
improvement does
does
You
necessary.
option here.
that
B*
between
tree offer
over
B*
does the minimum
of an order m B-tree?
B-tree?
How
can
it
an
and a B-tree? What
and what complications
depth of an order m B * tree
tree
a B-tree,
How
a virtual
is
be possible to average fewer than
one access per key when
from a three-level virtual B-tree?
Write a pseudocode description for an LRU replacement scheme for a
10-page buffer used in implementing a virtual B-tree.
retrieving keys
between storing the information indexed by the
14. Discuss the trade-offs
keys in
15.
We
B-tree with the key and storing the information in
noted
that,
given variable-length keys,
it is
a separate file.
possible to optimize a
by building in a bias toward promoting shorter keys. With fixed-order
trees we promote the middle key. In a variable-order, variable-length key
tree, what is the meaning of "middle key"? What are the trade-offs
associated with building in a bias toward shorter keys in this selection of a
key for promotion? Outline an implementation for this selection and
promotion process.
tree
Programming Exercises
16.
Implement the programs
procedure that performs
created
at the
end of this chapter and add
by the program. As an example, here
traversal
of the
tree
shown
parenthesized symmetric traversal of
is
recursive
a
B-tree
the result of a parenthesized
in Fig. 8.18:
(((A,B,C)D(E,F,G)H(I,J)K(L,M))N((0,P)Q(R)S(T,U,V)W(X,Y,Z)))
The split() routine in the B-tree programs
to make it more efficient.
17.
it
program
key
is
18.
Write
19.
Write an interactive program that allows
delete kevs
from
that searches for a
a B-tree.
not very
efficient.
Rewrite
in a B-tree.
a
user to find, insert, and
FURTHER READINGS
20. Write a B-tree
program
that uses keys that are strings, rather than single
characters.
program that builds a B-tree index for
records contain more information than just a key.
21.
Write
a data file in
which
FURTHER READINGS
Currently available textbooks on
file and data structures contain surprisingly brief
These discussions do not, in general, add substantially to the
information presented in this chapter and the following chapter. Consequently,
readers interested in more information about B-trees must turn to the articles that
have appeared in journals over the past 15 years.
The article that introduced B-trees to the world is Bayer and McCreight's
"Organization and Maintenance of Large Ordered Indexes" (1972). It describes the
theoretical properties of B-trees and includes empirical results concerning, among
other things, the effect of using redistribution in addition to splitting during
insertion. Readers should be aware that the notation and terminology used in this
article differ from that used in this text in a number of important respects.
Comer's (1979) survey article, "The Ubiquitous B-tree," provides an excellent
overview of some important variations on the basic B-tree form. Knuth's (1973b)
discussion of B-trees, although brief, is an important resource, in part because many
of the variant forms such as B* trees were first collected together in Knuth's
discussion. McCreight (1977) looks specifically at operations on trees that use
variable-length records and that are therefore of variable order. Although this article
speaks specifically about B* trees, the consideration of variable-length records can
be applied to many other B-tree forms. In "Time and Space Optimality on B-trees,"
Rosenberg and Snyder (1981) analyze the effects of initializing B-trees with the
minimum number of nodes. In "Analysis of Design Alternatives for Virtual
Memory Indexes," Murayama and Smith (1977) look at three factors that affect the
cost of retrieval: choice of search strategy, whether or not pages in the index are
structured, and whether or not keys are compressed. Zoellick (1986) discusses the
use of B-tree- like structures on optical discs.
Since B-trees in various forms have become a standard file organization for
databases, a good deal of interesting material on applications of B-trees can be found
in the database literature. Ullman (1986) Held and Stonebraker (1978), and Snyder
(1978) discuss the use of B-trees in database systems generally. Ullman (1986) covers
the problem of dealing with applications in which several programs have access to
the same database concurrently and identifies literature concerned with concurrent
discussions
on
B-trees.
access to B-tree.
Uses of B-trees for secondary key access are covered in many of the previously
cited references. There is also a growing literature on multidimensional dynamic
indexes, including a B-tree- like structure called a k-d B-tree. K-d B-trees are
387
388
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
described in papers by Ouskel and Scheuermann (1981) and Robinson (1981). Other
tries and grid files. Tries are
and data structures, including Knuth (1973b) and
Loomis (1983). Grid files are covered thoroughly in Nievergelt et al. (1984).
An interesting early paper on the use of dynamic tree structures for processing
files is "The Use of Tree Structures for Processing Files," by Sussenguth (1963).
Wagner (1973) and Keehn and Lacy (1974) examine the index design considerations
that led to the development of VSAM. VSAM uses an index structure very similar
to a B-tree, but appears to have been developed independently of Bayer and
McCreight's work. Readers interested in learning more about AVL trees will find a
good, approachable discussion of the algorithms associated with these trees in
approaches to secondary indexing include the use of
covered
in
many
Standish (1980).
tree operations
texts
on
files
Knuth (1973b)
and properties.
takes a
more
rigorous, mathematical look at
AVL
C Programs
PROGRAMS TO INSERT KEYS INTO A B-TREE
to Insert
Keys
389
into a B-Tree
The C program that follows implements the insert program described in the
text. The only difference between this program and the one in the text is
that this program builds a B-tree of order five, whereas the one in the text
builds a B-tree of order four. Input characters are taken from standard I/O,
with
q indicating
end of
The program
data.
from
requires the use of functions
driver, c
Contains the main program, which
described in the text very closely.
insert, c
Contains
several
files:
parallels the driver
program
the recursive function that finds the proper place
insertf),
for a key, inserts
it,
and supervises
splitting
and promotions.
Contains all support functions that directly perform I/O. The
header files fdeio.h and stdio.h must be available for inclusion in
btio.c
btio.c.
Contains the
btutil.c
split ()
All the
/*
of the support functions, including the function
rest
described in the text.
programs include the header
file
called bt.h.
bt.h.
header file for btree programs
.
*/
#def
#def
#def
#def
#def
#def
MAXKEYS
MINKEYS
ne
ine
i ne
i ne
ine
ine
i
NIL
NDKEY
MAXKEYS/2
(-1
/@/
NO
YES
typedef s t rue t {
shor t keycoun t;
char
keyCMAXKEYS]
shor t childEMAXKEYS+1
>
BTPAGE
;
/*
/*
/*
number of keys in page
*/
* /
the ac t ua 1 keys
ptrs to rrns of descendants*/
(continued)
390
B-TREES AND OTHER TREE-STRUCTURED
#define PAGESIZE
FILE
ORGANIZATIONS
zeof (BTPAGE
/* rrn of root page */
extern short
root;
/* file descriptor of btree file */
extern int btfd;
/* file descriptor of input file */
extern int infd;
/ *
pr o t otypes *
btcloseC);
btopenC )
btread(short rrn, BTPAGE *page_ptr);
btwr i t e( shor t r r n
BTPAGE *page_ptr);
creat e_root (char key
short left
shor t right)
short creat e_t ree( )
shor t get page( )
;
shor t get root ()
insert (short rrn, char key, short *pr omo_r_ch i 1 d char
ins_i n_page(char key, short r_child, BTPAGE *p_page);
;
key);
pageinit (BTPAGE *p_page);
put root ( shor t root);
search_node( char key, BTPAGE *p_page, short *pos);
split(char key, short r_child, BTPAGE *p_oldpage, char *P
k ey
short *promo_r_ch i Id BTPAGE *p_newpage);
,
Driver.c
/*
driver.c...
Driver for btree tests:
Opens or creates b-tree file.
Gets next key and calls insert to insert
If necessary, creates a new root.
key in tr
*/
#include <stdio.h>
^include "bt .h"
ma i n(
tells if a promotion from below */
#/
page
root
of
rrn
short root,
*/
rrn promoted from below
promo_rrn;
/
below
from
promo_key,/* key promoted
char
*/
/* next key to insert in tree
key;
promoted; /* boolean:
int
/*
/*
if
e
(btopenO)
root
get root ()
root
c r
5e
ea t e_t
/*
try to open btree.dat and get
/*
if
ee(
btree.dat not there, create
root
*/
it
*/
PROGRAMS TO INSERT KEYS INTO A
391
B-TREE: INSERT.C
while ((key = getcharO) != 'q') {
promoted = insert(root, key, &promo_rrn, &pr omo_k ey )
if ( promo t ed
root, promo_rrn);
root = c rea t e_root ( promo_k ey
,
>
btclose();
>
Insert.
/ *
insert .c.
Contains insertO function to insert a key into a btree.
Calls itself recursively until bottom of tree is reached.
Then inserts key in node.
If node is out of room,
calls splitO to split node
promotes middle key and rrn of new node
*/
^include "bt .h"
/* inser t (
Argument s
...
rrn:
*pr omo_r_ch
key:
i 1
*promo_key:
rrn of page to make insertion in
child promoted up from here to next level
key to be inserted here or lower
key promoted up from here to next level
*/
insert(short rrn, char key, short *pr omo_r_ch i
char *promo_key)
/* current page
BTPAGE page,
/* new page created
newpage;
int found, promoted; /* boolean values
short
pos
char
p_b_rrn;
p_b_key;
if
*/
if
split occurs
*/
*/
rrn promoted from below
key promoted from below
/*
/*
*/
*/
(rrn == NIL)
/* past bottom of tree... "promote"*/
i
/* original key so that it will be */
*promo_key = key;
*/
*promo_r_chi Id = NIL;/* inserted at leaf level
return (YES);
>
btread(rrn, &page);
found = search_node( k ey
&page, &pos);
if (found) {
printfC Error: attempt to insert duplicate key: %c \n\007", key);
return (0);
,
>
(continued)
392
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
promoted = i nser t (page ch i 1 d po5
key, &p_b_rrn, &p_b_key);
if ( promot ed)
return (NO);
/* no promotion */
if (page. keycount < MAXKEYS) {
ins_in_page(p_b_key p_b_rrn Apage) /* OK to insert key and*/
btwrite(rrn, &page);
/* pointer in this page. */
return (NO);
/* no promotion */
.
>
else
i t(p_b_key
p_b_rrn &page ,promo_key promo_r_chi
btwrite(rrn, &page);
btwrite( *promo_r_chi Id, &newpage);
return (YES);
/* promotion */
spl
Id
Anewpage);
>
Btio.c
/*
btio.c...
Contains btree functions that directly involve file i/o:
btopen() -- open file "btree.dat" to hold the btree.
btcloseO -- close "btree.dat"
getrootO -- get rrn of root node from first two bytes of btree.dat
putrootO -- put rrn of root node in first two bytes of btree.dat
cr eat e_t ree( ) -- create "btree.dat" and root node
getpageO -- get next available block in "btree.dat" for a new page
btread() -- read page number rrn from "btree.dat"
btwriteO -- write page number rrn to "btree.dat"
*/
^include "stdio.h"
#include "bt .h"
^include "fileio.h"
int
btfd;
bt open(
global file descriptor for "btree.dat"
/*
btfd = openC'btree.dat", 0_RDWR);
return(btfd > 0);
btclose(
close(btfd)
>
short get root
i
( )
*/
short root
long lseekC
PROGRAMS TO INSERT KEYS INTO A
B-TREE: BTIO.C
393
lseekCbtfd, OL, 0);
if CreadCbtfd, &root, 2) == 0) {
Unable to get root \007\n")
printf ("Error
.
exitd
return (root);
>
put root
shor
root)
lseekCbtfd,
teCbtf d
wr
OL,
0);
&root
2)
short cr eat e_t ree(
{
char key
creatC"btree.dat M ,PMODE);
/* Have to close and reopen to insure
closeCbtfd);
/* read/write access on many systems.
btopenC);
*/
/* Get first key.
key = getcharC);
return ( c r ea t e_r oo t ( k ey NIL, NIL));
btfd
short get page(
long lseekO, addr;
addr = lseekCbtfd, OL, 2) - 2L;
return CCshort) addr / PAGESIZE);
}
btreadCshort rrn, BTPAGE *page_ptr)
{
long
lseekC), addr;
addr = Clong)rrn * C ong )P AGES ZE + 2L
lseekCbtfd, addr, 0);
return C readCblfd, page_ptr, PAGESIZE)
I
>
btwr
i t
eC shor
rrn,
BTPAGE *page_ptr)
long lseekC), addr
addr = Clong) rrn * Clong) PAGESIZE + 2L
lseekCbtfd, addr, 0);
return CwriteCbtfd, page_ptr, PAGESIZE));
;
>
);
*/
*/
394
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
Btutil.c
btut ll.c...
/*
Contains utility functions for btree program:
e_r oo t
--
initialize root node and insert one key
NDKEY in all "key" slots and NIL in "child" slots
search_node( ) -- return YES if key in node, else NO. In either case,
put key's correct position in pos.
ins in_page() -- insert key and right child in page
splitC) -- split node by creating new node and moving half of keys to
new node. Promote middle key and rrn of new node.
c r
ea
pageinitO
( )
--
get and
put
*/
'include "bt.h"
c r
ea
e_r oo t char key,
short
left,
short
right)
BTPAGE page;
short rrn;
rrn = getpageC);
page i n i t ( &page )
page.key[01 = key;
page.childtO] = left;
=
right;
page ch i 1 d
page.keycount = 1;
btwriteCrrn Apage)
putroot(rrn)
return(rrn)
;
[ 1
page i
i t
(BTPAGE *p_page)
/*
p_page: pointer to
page
*/
int
for
))
<
MAXKEYS;
(j = 0;
= NDKEY;
p_page->key[
=
NIL;
p_page- >ch i Id
j
>
p_page->childtMAXKEYS]
NIL;
search_node(char key, BTPAGE *p_page, short *pos)
/* pos: position where key is or should be inserted
{
int
for
(i
0;
*po5
<
p_page- > k eycoun t
&&
key
>
p_page- > k ey
i 3
if
PROGRAMS TO INSERT KEYS INTO A
395
B-TREE: BTUTIL.C
*po5 < p_page - > eycoun t
&&
key == p_page/* key is in page */
return (YES);
> k
ey *pos
[
else
return (NO);
key
/ *
not
is
page */
in
ns_i n_page( cha r
r_child, BTPAGE *p_page)
short
key,
{
i
for
=
p_page- > k eycoun t
key < p_page=
p_page- > k ey i
p_page - > k ey i =
p_page- >ch i 1 d i +
p_page- >c h i 1 d i
(1
>
ey
&&
>
0;
i--)
>
p_page- >keycount++
=
p_page- > k ey i
key;
=
r_child;
p_page- >ch i 1 d i +
;
/* split ()
Argument s
inserted
promoted up from here
to be inserted
promoted up from here
pointer to old page structure
pointer to new page structure
key to be
key to be
child rrn
rrn to be
key:
promo key:
r_child:
promo r child:
p_oldpage:
p_newpage:
*/
splitCchar key, short r_child, BTPAGE *p_oldpage, char *promo_key,
BTPAGE *p_newpage)
short *pr omo_r_c h i 1 d
,
{
i
short mid;
char
wor k k eys [MAXKEYS+
short wor k c h MAX KE YS+ 2
t
*/
tells where split is to occur
*/
temporarily holds keys, before split
/* temporarily holds children, before split*/
/*
1
/ *
(i=0; l < MAXKEYS; i++) {
workkeystil = p_o 1 dpage - > k ey
workchti] = p_o 1 dpage -> ch i 1 d
/*
move keys and children from
old page into work arrays
&&
for
/*
[
i ]
*/
*/
>
workchti] = p_o 1 dpage- > ch i 1 d
for (i=MAXKEYS; key < wor k k ey s
workkeystil = wor k k eys i workchti+11 = workchti]
[
l
[
>
0;
- - ) { / *
insert new key*/
>
workkeystil
workchti+1]
key;
r_child;
*promo_r_chi Id = getpageC);
i n i t
p_newpage )
(
page
/*
/*
create new page for split,
and promote rrn of new page
*/
*/
(continued)
396
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
/* move first half of keys and
1
<
MINKEYS; i + + ) {
(1 =
=
/* children to old page, second
p_o 1 dpage - > k ey
wor keyst i ]
=
/*
workch[ i
half to new page
p_o 1 dpage- >ch i 1 d i
=
p_newpage- > k ey i
wor k k ey s i + + M NKEYS
=
p_newpage- >ch i 1 d i
wor kch[ i +1 +MI NKEYS ]
/* mark second half of old
p_oldpage->key[ l+MINKEYS] = NOKEY;
/* page as empty
p_oldpage->child[ i+1 +MINKEYS] = NIL;
for
*/
*/
*/
p_oldpage->child[MINKEYS] = wor k ch M NKE YS
p_newpage->child[MINKEYS] = wor k ch i
+M NKEYS
p_newpage->keycount = MAXKEYS - MINKEYS;
p_o 1 dpage- > eycount = MINKEYS;
/* promote middle key
*promo_key = wor k k eys M NKEYS
[
*/
*/
*/
PASCAL PROGRAMS TO INSERT KEYS INTO A B-TREE: DRIVER.PAS
397
Pascal Programs to Insert Keys into a B-Tree
The
Pascal
program
that follows
in the text.
The only
difference
implements the insert program described
between this program and the one in the text
is that this program builds a B-tree of order five, whereas the one in the text
builds a B-tree of order four. Input characters are taken from standard I/O,
with q indicating end of data.
The main program includes three nonstandard compiler directives:
{$B->
{$1
{$1
btutil.prc}
insert .pre}
The $B
as a
instructs the
standard Pascal
The
Turbo
Pascal compiler to handle keyboard input
file.
$1 directives instruct the
compiler to include the
files btutil.prc
and
main program. These two files contain functions needed by
the main program. So the B-tree program requires the use of functions
from three files:
insert. pre
in the
Contains the main program, which closely
driver. pas
driver
program described
parallels the
in the text.
insert. pre
Contains insertf), the recursive function that finds the
proper place for a key, inserts it, and supervises splitting
and promotions.
btutil.prc
Contains
all
other support functions, including the func-
tion split() described in the text.
Driver.pas
PROGRAM btree
NPUT OUTPUT)
,
Driver for B-tree tests
(continued)
398
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
Opens or creates btree file
Gets next key and calls insert to insert key in tree
If necessary, creates a new root.
{$B->
CONST
MAXKEYS = 4;
MAXCHLD = 5;
MAXWKEYS = 5;
MAXWCHLD = 6;
NOKEY = '@'
NO = FALSE;
YES = TRUE;
NULL = -1
{maximum number of keys
P a 9 e>
{maximum number of children in page}
{maximum number of keys in working space}
{maximum number of children in working space}
{symbol to indicate no key}
TYPE
BTPAGE = RECORD
keycount
integer;
{number of keys in page
}
key
array [1.. MAXKEYS] of char;
{the actual keys
}
child
array
MAXCHLD of integer; {ptrs to RRNs of descendents}
:
END;
VAR
promoted
boo 1 ean
oo t
pr omo_r
integer
promo_k ey
btfd
char
file of BTPAGE
MINKEYS
PAGESIZE
integer
i nt eger
{$1
{$1
ey
{tells if a promotion from below
{RRN of root
{RRN promoted from below
{key promoted from below
{next key to insert in tree
{global file descriptor for
{"btree.dat"
{min. number of keys in a page
{size of a page
btutil.prc}
insert. pre}
BEGIN {main}
MINKEYS
PAGESIZE
if
MAXKEYS DIV 2;
sizeof (BTPAGE)
{try to open btree.dat and get root}
btopen then
root
root
:=
get root
{if btree.dat
else
ead( k ey )
create_tree
NHILE (key <>
BEGIN
q'
DO
not
there,
create it}
399
PASCAL PROGRAMS TO INSERT KEYS INTO A B-TREE: INSERT.PRC
promoted
if
i nser t ( roo t
k ey
promo_r rn promo_k ey )
then
c r eat e_root ( promo_k ey root promo_r rn
:=
pr omo ted
root
ead( key)
END;
:=
btclose
END
Insert.prc
FUNCTION insert (rrn: integer;key: char;VAR pr omo_r_ch i
VAR promo_key: char): boolean;
Function to insert
integer;
key into a B-tree:
Calls itself recursively until the bottom of the tree is reached.
Then inserts the key in the node.
If node is out of room, then it calls split() to split the node and
promotes the middle key and RRN of new node.
VAR
{current page
{new page created if split occurs
{tells if key is already in B-tree
{tells if key is promoted
{position that key is to go in
{RRN promoted from below
{key promoted from below
page,
newpage
found
promoted
BTPAGE;
pos
b_rrn
p_b_key
boolean
i n t eger
char
GIN
{past bottom of tree... "promote"
(rrn = NULL) th
{original key so that it will be
BEGIN
{inserted at leaf level
promo key := key;
promo r child := NULL;
insert := YES
END
else
BEGIN
btreadCrrn ,page)
found := search n ode ( k ey page pos )
if (found ) then
BEGIN
key);
attempt to insert duplicate key:
wr telnC Error
insert := NO
END
if
'
}
>
(continued)
400
B-TREES AND OTHER TREE-STRUCTURED FILE ORGANIZATIONS
else
BEGIN
promoted := 1 nser t ( page ch 1 1 d pos
k ey
p_b_r rn p_b_k ey )
if (NOT promoted) then
insert := ND
{no promotion}
else
BEGIN
if (page, keycount < MAXKEYS) then
BEGIN
{OK to insert key
p_b_rrn page ) {and pointer in this
i ns_i n_page( p_b_key
btwrite(rrn,page);
{page.
insert := NO
{no promotion}
END
else
BEGIN
spl i t ( p_b_k ey p_b_r r n page promo_k ey
promo_r_ch ild,newpage);
btwrite(rrn,page)
btwrite( promo_r ch ild,newpage);
insert := YES
{promotion}
END
END
END
]
END
END:
Btutil.prc
FUNCTION btopen
BOOLEAN;
{Function to open "btree.dat"
it returns false}
:
if
it
already exists. Otherwise
VAR
response
char;
BEGIN
assign(btfd, 'btree.dat );
write('Does btree.dat already exist? (respond
readln(response)
writeln;
if (response = 'Y') OR (response = 'y') then
BEGIN
reset(btfd)
btopen := TRUE
END
else
btopen := FALSE
:
'
END;
or
N):
*);
}
}
401
PASCAL PROGRAMS TO INSERT KEYS INTO A B-TREE: BTUTILPRC
PROCEDURE btclose;
{Procedure to close "btree.dat"}
BEGIN
close (btfd);
END;
FUNCTION getroot
integer;
{Function to get the RRN of the root node from first record of btree.dat)
:
VAR
root
BTPAGE;
BEGIN
seek(btfd,0);
if (not EOF) then
BEGIN
r ead( btfd, root);
getroot := r oo t
eycoun t
END
else
wr i t e 1 n( Er r or
Unable to get root.')
:
'
END;
FUNCTION getpage
integer;
{Function that gets the next available block in "btree.dat" for
BEGIN
getpage := f i 1 es i ze( b t f d )
:
new page)
END;
PROCEDURE pageinit (VAR p_page
BTPAGE);
{puts NOKEY in all "key" slots and NULL in "child" slots}
:
VAR
j
eger
BEGIN
for
:=
to MAXKEYS
DO
BEGIN
:= NOKEY;
p_page. key[
:= NULL;
p_page ch i 1 d
1
END;
p_page.child[MAXKEYS+1
:=
NULL
END;
PROCEDURE putroot (root: integer);
{Puts RRN of root node in the keycount of the first record of btree.dat
>
VAR
rootrrn
BTPAGE;
BEGIN
seek(btf d,0)
rootrrn. keycount
:
:=
root;
(continued)
402
B-TREES AND OTHER TREE-STRUCTURED
FILE
ORGANIZATIONS
pageinit (rootrrn);
write(btfd,rootrrn)
END;
PROCEDURE btread (rrn
integer; VAR page_ptr
{reads page number RRN from btree.dat}
BEGIN
seek (btfd,rrn);
read(btfd, page_pt r )
BTPAGE)
END;
PROCEDURE btwnte (rrn
integer; page_ptr
{writes page number RRN to btree.dat}
BEGIN
seekCbtf d,rrn)
write(btfd, page p t r )
BTPAGE)
END;
FUNC TION create_root (key :char; left, right
integer):
{get and initialize root node and insert one key}
VAR
page
BTPAGE;
rrn
integer
BEGIN
rrn
=
get page
page i n i t ( page)
key
page k ey
=
left
page ch i 1 d
right
page ch i 1 d 2
page k eycount
=
:
btwrite(rrn,page)
putroot(rrn)
create_root := rrn
;
END;
integer;
FUNCTION create_tree
{creates "btree.dat" and the root node}
VAR
rootrrn
int eger
BEGIN
:
rewnteCbtf d)
r
ead( key);
rootrrn := get page;
putroot(rootrrn)
create_tree := c r ea t e_r oo t
;
END;
( k
ey NULL NULL
,
integer;
403
PASCAL PROGRAMS TO INSERT KEYS INTO A B-TREE: BTUTIL.PRC
FUNCTION search_node( key char
p_page
{returns YES if key in node, else NO.
position in pos>
:
BTPAGE VAR pos integer): boolean
either case, put key's correct
;
In
VAR
1
eger
BEGIN
i
:=
while ((i
l
pos
<=
page
eye oun t
AND (key
>
p_page
ey
l ] )
DO
((pos <= p_page
eycount
search__node := YES
else
sear ch_node
NO
=
if
AND (key
p_page
ey pos
[
) )
then
END;
PROCEDURE ins_in_page (key: char r_chi
{insert key and right child in page}
;
integer; VAR p_page:
Id:
BTPAGE);
VAR
i
integer;
BEGIN
:= p_page
i
eycount
while ((key < p_page
BEGIN
:=
p_page k ey i
p_page ch i 1 d i +
1;
ey
l
[
page
:=
ey
AND
i
[
p_page ch i
.
>
1)) DO
l ]
END;
p
page
eye oun t
p_page k ey
p_page ch i 1
[
p_page
:=
eye oun
key;
:=
r_child
:=
END;
PROCEDURE split (key: char; r_child: integer; VAR p_oldpage: BTPAGE;
integer;
VAR promo_key: char; VAR pr omo_r_ch l 1 d
VAR p_newpage: BTPAGE);
:
{split node by creating new node and moving half of keys to new node.
Promote middle key and RRN of new node.)
VAR
i
integer;
{temporarily holds keys,}
of char;
workkeys
array
MA XNKE YS
{
before split}
of integer; {temporarily holds children
workch
MA XNCHLD
array
before split }
{
:
(continued)
404
B-TREES AND OTHER TREE-STRUCTURED
ORGANIZATIONS
FILE
BEGIN
:=
to MAXKEYS
DO
1
BEGIN
workkeyslil := p_o 1 dpage k ey
workchEi] := p_o dpage ch i 1 d
for
{move keys and children from
{old page into work arrays
}
}
END;
workchCMAXKEYS+1
:=
p_o 1 dpage ch
.
i 1
i
MAXKEYS
=
while ((key < wor k k ey s i )
AND
BEGIN
workkeystil := wor k k ey 5 i workchEi+1] := workchtil;
:
>
:=
1)) DO
}
key;
r_child;
{create new page for split
{and promote RRN of new page
{move first half of keys and
{children to old page,
{second half to new page.
i 1
page
{insert new key
omo_r_ch d := getpage;
i n i t
p_newpage )
(
for i
TO MINKEYS DO
=
pr
workchti+11
END;
wor k k ey s
MAXKEYS-
BEGIN
:= workkeystil;
p_o 1 dpage k ey[ i
:= workchtil;
p_o 1 dpage ch i 1 d i
:= wor k k ey s i + + M NKEYS
p_newpage k ey i
:= wor k ch i + + M NKEYS
p_newpage. child! i
p_oldpage. key[ i+MINKEYSl
{mark second half of old
=
NOKEY;
p_oldpage.child[ i+1 +MINKEYS] := NULL
<page as empty
]
>
>
>
>
>
>
>
END;
p_oldpage.child[MINKEYS+1
if
wor k ch M NKEYS+
:=
odd(MAXKEYS)
t
hen beg i
:= wor k k ey s MAXNKEYS
p_newpage. k ey M NKEYS+
p_newpage.child[MINKEYS+2] := wor k ch MAXNCHLD
:= wor k ch MA XWCHLD=
p_newpage.chi ld[MINKEYS+1
[
end
else
:= wor k ch MA XNCHLD
p_newpage.chi 1 d M NKEYS+
p_newpage. keycount := MAXKEYS - MINKEYS;
p_oldpage. keycount := MINKEYS;
{promote middle key
promo_key := wor k k eys M NKE Y S+1]
1
END;
>
4
The B Tree Family
and Indexed Sequential
File
Access
CHAPTER OBJECTIVES
Introduce indexed sequential
files.
Describe operations on a sequence set of blocks that
maintains records in order by key.
Show how
sequence
set can be built on top of the
produce an indexed sequential file
an index
set to
structure.
Introduce the use of a B-tree to maintain the index
+
set, thereby introducing B
trees and simple prefix
B+
trees.
Illustrate
fix
B+
how
the B-tree index set in a simple pre-
tree can
variable
be of variable order, holding
number of separators.
Compare
the strengths and weaknesses of
+
trees, simple prefix B
trees, and B-trees.
B+
405
CHAPTER OUTLINE
9.1
Indexed Sequential Access
9.2
Maintaining a Sequence Set
9.2.1
9.6.2
Blocks
The Use of Blocks
9.2.2 Choice of Block Size
9.3
Adding
9.5
The Simple
9.6
Simple Prefix
Maintenance
9.6.1
9.8
Internal Structure of Index Set
9.9
9.10
Separators Instead of Keys
B+
Prefix
B+
Sequence Set
Index Set Block Size
Blocks:
The Content of the Index:
in the
9.7
Simple Index to the
Sequence Set
9.4
Changes Involving Multiple
Loading
Simple Prefix B" Tree
Trees
9.11 B-Trees,
Tree
Variable-order B-Tree
Prefix
B"
B+
Trees, and Simple
Trees in Perspective
Tree
Changes Localized
to Single
Blocks in the Sequence Set
9.1
Indexed Sequential Access
Indexed sequential
views of a file:
The
Indexed:
file
file
structures provide a choice
can be seen
as a set
between two
of records that
is
alternative
indexed
by key;
or
Sequential:
The
ous records
The
file
can be accessed sequentially (physically contigu-
no seeking),
returning records in order by key.
idea of having a single organizational
method
that provides both of
have had to choose between
them. As a somewhat extreme, though instructive, example of the potential
divergence of these two choices, suppose that we have developed a file
these views
is
new
one.
Up
to this point
we
structure consisting of a set of entry-sequenced records indexed
by
separate B-tree. This structure can provide excellent indexed access to any
individual record by key, even as records are added and deleted.
suppose that
we
also
want
cosequential processing
to use this
we want
Since the actual records in this
file as
to retrieve
file
Now let's
part of a cosequential merge. In
all
system are
the records in order
by key.
entry sequenced, rather than
by key, the only way to retrieve them in order by key is
pointers from
through the index. For a file of
records, following the
essentially random seeks
the index into the entry sequenced set requires
into the record file. This is a much less efficient process than the sequential
physically sorted
407
MAINTAINING A SEQUENCE SET
so much so
reading of physically adjacent records
for
any situation
On
in
which cosequential processing
is
that
a
the other hand, our discussions of indexing
it is
unacceptable
frequent occurrence.
show
us that a
file
consisting of a set of records sorted by key, though ideal for cosequential
processing,
an unacceptable structure
is
by key
delete records
What
in
random
when we want
to access, insert,
and
order.
an application involves both interactive random access and
if
cosequential batch processing? There are
many examples of such dual-mode
applications. Student record systems at universities, for example, require
keyed access
to individual records while also requiring a large
batch processing,
when
grades are posted or
when
amount of
during
both batch processing of
charge slips and interactive checks of account status. Indexed sequential
access methods were developed in response to these kinds of needs.
as
fees are paid
registration. Similarly, credit card systems require
9.2
Maintaining a Sequence Set
We set aside,
for the
moment,
the indexed part of indexed sequential access,
focusing on the problem of keeping
of records in physical order by key
ordered set of records as
sequence set. We will assume that once we have a good way of maintaining
sequence set, we will find some way to index it as well.
as
a
a set
We
records are added and deleted.
refer to this
9.2.1 The Use of Blocks
We
can immediately rule out the idea of sorting and resorting the entire
sequence
entire
set as
the changes.
deletion
records are added and deleted, since
One of the
to just
part
best
ways
When we
We
the buffers
4:
we know
instead to find a
to restrict the effects
of the sequence
encountered in chapters 3 and
output.
We need
an expensive process.
file is
We
set
that sorting an
way
of an insertion or
involves
tool
can collect the records into
we
first
blocks.
block records, the block becomes the basic unit of input and
at once. Consequently, the size of
read and write entire blocks
we
use in
After reading in
program
block,
all
is
such that they can hold an entire block.
the records in
block are in
RAM,
can work on them or rearrange them much more rapidly.
An example helps illustrate how the use of blocks can help
sequence set in order. Suppose we have records that are keyed on
and collected together so there are four records
link fields in
to localize
in a block.
We
where we
us keep a
last
name
also include
each block that point to the preceding block and the following
408
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
block.
We need
these fields because, as
FILE
ACCESS
you
will see, consecutive blocks are
not necessarily physically adjacent.
As with B-trees, the insertion of new records into a block can cause the
block to overflow. The overflow condition can be handled by a blocksplitting process that is analogous to, but not the same as, the blocksplitting process used in a B-tree. For example, Fig. 9.1(a) shows what our
blocked sequence set looks like before any insertions or deletions take place.
We show only the forward links. In Fig. 9.1(b) we have inserted a new
record with the key CARTER. This insertion causes block 2 to split. The
second half of what was originally block 2
Note
we
is
found
in
block 4 after the
split.
that this block-splitting process operates differently than the splitting
encountered in B-trees. In
a B-tree a split results in the promotion
of
Here things are simpler: We just divide the records between two
blocks and rearrange the links so we can still move through the file in order
by key, block after block.
Deletion of records can cause a block to be less than half full and
therefore to underflow. Once again, this problem and its solutions are
analogous to what we encounter when working with B-trees. Underflow in
record.
B-tree can lead to either of
If a
neighboring node
is
two
solutions:
also half full,
we
can concatenate the two
nodes, freeing one up for reuse.
neighboring nodes are more than half
If the
records between the nodes to
make
full,
we
the distribution
can
redistribute
more
nearly
even.
the
Underflow within a block of our sequence set can be handled through
same kinds of processes. As with insertion, the process for the sequence
set is
simpler than the process for B-trees since the sequence
set is not a tree
and there are therefore no keys and records in a parent node. In Fig. 9.1(c)
we show the effects of deleting the record for DAVIS. Block 4 underflows
and is then concatenated with its successor in logical sequence, which is
block 3. The concatenation process frees up block 3 for reuse. We do not
show an example in which underflow leads to redistribution, rather than
concatenation, since it is easy to see how the redistribution process works.
Records are simply moved between logically adjacent blocks.
Given the separation of records into blocks, along with these fundamental block-splitting, concatenation, and redistribution operations, we
can keep a sequence set in order by key without ever having to sort the
entire set
of records. As always, nothing comes
free;
consequently, there are
costs associated with this avoidance of sorting:
Once
made, our file takes up more space than an unof sorted records because of internal fragmentation
insertions are
blocked
file
MAINTAINING A SEQUENCE SET
Block
ADAMS
Block 2
w BYNUM
Block 3
W DENVER
BAIRD
fc
ELLIS
BIXBY
CARSON
COLE
BOONE
DAVIS
(a)
Block
ADAMS
Block 2
fc
w
BYNUM
DENVER
Block 3
Block 4
fe
w
COLE
BAIRD
ELLIS
DAVIS
BIXBY
CARSON
BOONE
CARTER
(b)
Block
ADAMS
fe
Block 2
BYNUM
BAIRD
BIXBY
CARSON
CARTER
BOONE
Block 3
Available
for reuse
Block 4
w COLE
DENVER
ELLIS
(c)
Block splitting and concatenation due to insertions and deletions in
(a) Initial blocked sequence set. (b) Sequence set after insertion of CARTER record block 2 splits, and the contents are divided between
block 4 is less
blocks 2 and 4. (c) Sequence set after deletion of DAVIS record
than half full, so it is concatenated with block 3.
FIGURE 9.1
the sequence set.
409
41
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
within a block. However, we can apply the same kinds of strategies
used to increase space utilization in a B-tree (e.g., the use of redistribution in place of splitting during insertion, two-to-three splitting,
and so on). Once again, the implementation of any of these strategies
must account for the fact that the sequence set is not a tree and that
there is therefore no promotion of records.
The order of the records is not necessarily physically sequential
throughout the file. The maximum guaranteed extent of physical sequentially
This
last
is
within
a block.
point leads us to the important question of selecting
block
size.
9.2.2 Choice of Block Size
As we work with our sequence
set,
block
is
the basic unit for our I/O
When we read data from the disk, we never read less than a
block; when we write data, we always write at least one block. A block is
also, as we have said, the maximum guaranteed extent of physical
sequentiality. It follows that we should think in terms of large blocks, with
operations.
each block holding
many
one of identifying the
big
on block
size:
Why
not
make
we can fit the entire file in a single block?
One answer to this is the same as the reason why we
RAM
our
records. So the question of block size
limits
first
becomes
the block size so
cannot always use
enough RAM available. So
consideration regarding an upper bound for block size is as follows:
sort
on
a file:
Consideration
We
usually do not have
size should be such that we can hold several
blocks in
at once. For example, in performing a
block split or concatenation, we want to be able to
at a time. If we are
hold at least two blocks in
implementing two-to-three splitting to conserve disk
at
space, we need to hold at least three blocks in
The block
1:
RAM
RAM
RAM
a time.
Although we
sequence
are
presently
set sequentially
randomly accessing
in an entire
we
a single
block to get
focusing on the ability to access our
eventually want to consider the problem of
set. We have to read
any one record within that block. We can
record from our sequence
at
therefore state a second consideration:
Consideration
2:
block should not take
an unlimited amount of
RAM, we would want to place an upper limit on the
block size so we would not end up reading in the en-
Reading
in or writing out a
very long. Even
tire file just to
if
we had
get at a single record.
41
ADDING A SIMPLE INDEX TO THE SEQUENCE SET
This second consideration
We
very long?
knowledge of
not
interested in a
so
mandatory limitation, but
block because
And where is
When we discussed
introduced the term
it
is
it
a sensible
is
still
that
it
We
are
at
which we can guarantee such
that?
sector formatted disks back in Chapter 3,
cluster.
uses
cluster
up eight
guarantees a
As we move from
one:
contains records that are physically adjacent,
the
is
minimum number of
allocated at a time. If a cluster consists of eight sectors, then a
only one byte
is
long
size should be such that we can access a
block without having to bear the cost of a disk seek
within the block read or block write operation.
adjacency.
clustering
How
The block
not extend blocks beyond the point
let's
imprecise:
the performance characteristics of disk drives:
(redefined):
is
a little
can refine this consideration by factoring in some of our
Consideration 2
This
more than
is
sectors
on the
minimum amount
cluster to cluster in reading a
disk.
file
we
sectors
containing
The reason
for
of physical sequentiality.
file,
we may
incur a disk
without seeking.
block size, then, is to make
seek, but within a cluster the data can be accessed
One reasonable
suggestion for deciding on
Often the cluster size on a disk
system has already been determined by the system administrator. But what
if you are configuring a disk system for a particular application and can
therefore choose your own cluster size? Then you need to consider the
issues relating to cluster size raised in Chapter 3, along with the constraints
available and the number of blocks you
imposed by the amount of
each block equal to the size of
want
to hold in
a cluster.
RAM
RAM at once. As
is
so often the case, the final decision will
probably be a compromise between a number of divergent considerations.
The important thing is that the compromise be a truly informed decision,
based on knowledge of how I/O devices and file structures work, rather
than just
If
a guess.
you
that allows
point
is
to
revise this
working with a disk system that is not sector oriented, but
you to choose the block size for a particular file, a good starting
think of a block as an entire track of the disk. You may want to
downward, to half a track, for instance, depending on memory
are
constraints, record size,
9.3
and other
factors.
Sequence Set
Adding a Simple Index
to the
We
for maintaining a set of records so
have created
access
them
mechanism
sequentially in order
by key.
It is
we
can
based on the idea of grouping
the records into blocks and then maintaining the blocks, as records are
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
412
12
ADAMS-BERNEE
BOLEN-CAGE
CAMP-DUTTON
ACCESS
FILE
\
/
EMBRY-EVANS
FABER-FOLK Ni FOLKS-GADDIS
FIGURE 9.2 Sequence of blocks showing the range of keys
each block.
in
added and deleted, through splitting, concatenation, and redistribution.
Now let's see whether we can find an efficient way to locate some specific
block containing
We
illustrated in Fig. 9.2. This
actually read
it is
given the record's key.
a particular record,
can view each of our blocks as containing
is
a range
of records,
as
an outside view of the blocks (we have not
any blocks and so do not
know
exactly what they contain), but
choose which block might have
for example, that if we are looking
sufficiently informative to allow us to
we
the record
are seeking.
for a record with the
We
key
can
see,
BURNS, we
want
to retrieve
and inspect the
second block.
It is
these
easy to see
blocks.
We
how we
could construct
might choose,
for
a simple, single-level
example,
fixed-length records that contain the key for the
as
shown in Fig. 9.3.
The combination of this kind of index with
to
last
build
index for
an index of
record in each block,
the sequence set of blocks
provides complete indexed sequential access. If we need to retrieve a specific
record,
we
consult the index and then retrieve the correct block; if we need
we start at the first block and read through the linked list
we have read them all. As simple as this approach is, it is in
sequential access
of blocks until
fact a very workable one
as
long
RAM
as the entire
memory. The requirement
important for two reasons:
Since this
we
is
for the
sequence
Key
Block number
BERNE
CAGE
DUTTON
EVANS
FOLK
GADDIS
4
5
by means of
set illustrated
index can be held in electronic
the index be held in
simple index of the kind
find specific records
FIGURE 9.3 Simple index
that
in
we
RAM
discussed in Chapter
6,
binary search of the index.
Fig. 9.2.
is
41 3
THE CONTENT OF THE INDEX: SEPARATORS INSTEAD OF KEYS
Binary searching works well
but, as
many
As
we saw
in the
seeks if the
if
file is
on
RAM,
the searching takes place in
previous chapter on B-trees,
it
requires too
secondary storage device.
the blocks in the sequence set are changed through splitting, con-
Updating
catenation, and redistribution, the index has to be updated.
simple, fixed-length record index of this kind
dex
and contained
RAM.
works well
if
the in-
however, the updating requires seeking to individual index records on disk, the prorelatively small
is
cess can
in
become very expensive. Once again,
more completely in earlier chapters.
If,
this
is
we
point
discussed
What do we
do, then, if the
index does not conveniently
found
we
blocks
in
that
we
contains so
into
RAM?
many
blocks that the block
In the preceding chapter
could divide the index structure into pages,
much
we
like the
are discussing here, handling several pages, or blocks, of the index
RAM at a time.
file
file
fit
More
specifically,
we found
that B-trees are an excellent
structure for handling indexes that are too large to
This suggests that
we might
fit
entirely in
RAM.
organize the index to our sequence set as a
B-tree.
The use of
B-tree index for our sequence set of blocks
very powerful notion. The resulting hybrid structure
which
is
appropriate since
is,
known
it is
we need
to
The purpose of
the index
we
The index
Keys
is to assist us when we
The index must guide us to
are building
searching for a record with a specific key.
all.
tree,
keep in the index.
of the Index: Separators Instead of
set at
B+
The Content
block in the sequence
in fact, a
as a
B-tree index plus a sequence set that holds
+
can fully develop the notion of a B tree, we
it is
the actual records. Before we
need to think more carefully about what
9.4
is
set that contains the record, if it exists in the
serves as a kind of roadmap for the sequence
interested in the content of the index only insofar as
it
can
assist
are
the
sequence
set.
We are
us in getting
to the correct block in the sequence set; the index set does not itself contain
answers,
it
contains only information about where to go to get answers.
Given this view of the index set as a roadmap, we can take the very
important step of recognizing that we do not need to have actual keys in the
index set. Our real need is for separators. Figure 9.4 shows one possible set of
separators for the sequence set in Fig. 9.2.
Note
that there are
many
potential separators capable of distinguishing
between two blocks. For example, all of the strings shown between blocks
3 and 4 in Fig. 9.5 are capable of guiding us in our choice between the blocks
as we search for a particular key. If a string comparison between the key and
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
414
ACCESS
CAM
BO
Separators:
FILE
12
ADAMS-BERNE
FOLKS
"7
CAMP-DUTTON
EMBRV-EVANSSj FABER-FOLK Si FOLKS-GADDIS
FIGURE 9.4 Separators between blocks
3
in
the sequence set.
any of these separators shows that the key precedes the separator, we look
for the key in block 3. If the key follows the separator, we look in block 4.
If
we
are willing to treat the separators as variable-length entities within
our index structure (we
by placing the
as the separator to
about
how
to
we
can save space
4.
we
Note
use
that
is
not always
are separators that are
function
Note
shown
in Fig. 9.6
blocks 5 and
6,
follows that, as
must decide
one that
is
by using
in the Pascal
the logic
procedure
embodied
in the
listed in Fig. 9.7.
is the same as the
by the separator between
which is the same as the first key contained in block 6. It
we use the separators as a roadmap to the sequence set, we
is
produce
a separator that
illustrated in Fig. 9.4
is to the right of the separator or the
of the separator according to the following rule:
to retrieve the block that
to the left
Relation of Search
Key <
Key =
separator
Key >
separator
list
and
that these functions can
second key. This situation
FIGURE 9.5 A
this later),
guide our choice between blocks 3 and
the other separators contained in Fig. 9.4
do
index structure. Consequently,
unique shortest separator. For example, BK, BN, and
all the same length and that are equally effective
separators between blocks 1 and 2 in Fig. 9.4. We choose BO and all of
there
BO
as
talk
shortest separator in the
Key and
Separator
separator
Decision
Go
Go
Go
left
right
right
of potential separators.
DUTU
CAMP-DUTTON
DVXGHESJF
DZ
E
EMBRY-EVANS
EBQX
ELEEMOSYNARY
^-n
41 5
THE CONTENT OF THE INDEX: SEPARATORS INSTEAD OF KEYS
/* find_sep(keyl, key2, sep)
...
finds shortest string that serves as a separator between keyl and
Returns this separator through the address provided by
key2
the "sep" parameter
.
the function assumes that key2 follows keyl in collating sequence
V
f ind_sep
keyl key2 sep
char keyl[], key2[],
(
sep[];
while
(*sep++
*sep='\0';
*key2++)
== *keyl++)
/* ensure that separator string is null terminated */
FIGURE 9.6 C function to find a shortest separator.
FIGURE 9.7 Pascal procedure to find a shortest separator.
PROCEDURE find_sep (keyl, key2
strng
VAR sep
strng)
{
finds the shortest string that serves as a separator between keyl and
key2
Returns the separator through the variable sep.
Strings are
handled as character arrays in which the length of the string is stored
in the Oth position of the array.
The type "strng" is used for strings.
:
Assumes that key2 follows keyl in collating sequence.
Uses two functions defined in the Appendix:
len_str(s)
returns the length of the string s.
min(i,j)
compares i and j and returns the smallest value
>
VAR
i, minlgth
integer;
BEGIN
minlgth := min( len_str (keyl
:
:=
len_str(key2)
1;
while (keylCi] = key2Ci]) and
BEGIN
sepCi]
= key2Ci]
(i
<= minlgth) DO
:=
END;
sepCi]
sepCO]
END;
:=
:=
key2Ci];
CHR(i)
set length indicator in separator array
41
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
E
I
Index
FIGURE 9.8 A B-tree index set
9.5
The Simple
for the
Prefix B
Figure 9.8 shows
a
sequence
how we
set,
forming a simple prefix B
tree.
Tree
can form the separators identified in Fig. 9.4 into
The B-tree index is called the
Taken together with the sequence set, it forms a file structure
+
simple prefix B
tree. The modifier simple prefix indicates that the
B-tree index of the sequence set blocks.
index
set.
called a
index
set contains shortest separators,
copies of the actual keys.
Our
or prefixes of the keys rather than
separators are simple because they are,
They are actually just the initial letters within the keys.
More complicated (not simple) methods of creating separators from key
simply, prefixes:
remove unnecessary characters from the front of the separator as
from the rear. (See Bayer and Unterauer, 1977, for a more complete
prefixes
well as
discussion of prefix
Note
B+
"
1
trees.)"
that since the index set
branches to
N+
EMBRY, we
separator E.
retrieving the
children. If
start at the
is
we
a B-tree, a
node containing
N separators
are searching for the record with the
root of the index
set,
comparing
EMBRY
key
to the
Since EMBRY comes after E, we branch to the right,
node containing the separators F and FOLKS. Since EMBRY
on B trees and simple prefix B trees is remarkably inconsistent in the nomenclature used for these structures. B + trees are sometimes called B* trees; simple prefix
B + trees are sometimes called simple prefix B-trees. Comer's important article in Computing
Surveys in 1979 has reduced some of the confusion by providing a consistent, standard nomenclature which we use here.
""The literature
SIMPLE PREFIX B
TREE MAINTENANCE
417
the first of these separators, we follow the branch that is
of the F separator, which leads us to block 4, the correct block in
the sequence set.
comes before even
to the left
9.6
Simple Prefix B
Tree Maintenance
9.6.1 Changes Localized to Single Blocks
in
the Sequence Set
suppose that we want to delete the records for EMBRY and FOLKS,
suppose that neither of these deletions results in any concatenation
or redistribution within the sequence set. Since there is no concatenation or
redistribution, the effect of these deletions on the sequence set is limited to
changes within blocks 4 and 6. The record that was formerly the second
record in block 4 (let's say that its key is ERVIN) is now the first record.
Similarly, the former second record in block 6 (we assume it has a key of
FROST) now starts that block. These changes can be seen in Fig. 9.9.
Let's
and
let's
The more
interesting question
is
what
effect,
if
any, these deletions
have on the index set. The answer is that since the number of sequence set
blocks is unchanged, and since no records are moved between blocks, the
index set can also remain unchanged. This is easy to see in the case of the
EMBRY deletion: E is still a perfectly good separator for sequence set
blocks 3 and 4, so there is no reason to change it in the index set. The case
E
!
FIGURE 9.9 The deletion of the
index set unchanged.
EMBRY
and FOLKS records from the sequence set leaves the
41 8
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FOLKS
of the
deletion
appears both as
To
index
set.
these
two
key
more confusing
a little
is
ACCESS
FILE
in the deleted record
and
since the string
avoid confusion, remember to distinguish clearly between
FOLKS: FOLKS can continue to serve as a
uses of the string
FOLKS
separator between blocks 5 and 6 even though the
(One could argue
we
separator,
a shorter
FOLKS
within the
as a separator
that although
we do
record
is
deleted.
FOLKS
not need to replace the
should do so anyway because
it is
now
possible to construct
However, the cost of making such a change in the index
outweighs the benefits associated with saving a few bytes of
separator.
set usually
space.)
The
effect
of inserting into the sequence
cause block splitting
is
much
the
same
not result in concatenation: The index
example, that
we
by the separators
insert a record for
in the
index
into block 4 of the sequence
room
in
4,
we
set
We
records that do not
of these deletions that do
remains unchanged. Suppose, for
set
EATON.
we
new
find that
Following the path indicated
we will insert
The new record becomes
but no change in the index
the
new
record
assume, for the moment, that there
set is necessary.
This
the
is
is
record
first
not surprising
decided to insert the record into block 4 on the basis of the existing
information in the index
index
set.
for the record in the block.
block
since
set,
set
as the effect
is
set. It
follows that the existing information in the
sufficient to allow us to find the record again.
9.6.2 Changes Involving Multiple Blocks
in
the Sequence Set
What happens when
the addition and deletion of records to and from the
change the number of blocks in the sequence set? Clearly,
if we have more blocks, we need additional separators in the index set, and
if we have fewer blocks, we need fewer separators. Changing the number
of separators certainly has an effect on the index set, where the separators
sequence
set does
are stored.
Since the index set for a simple prefix
B+
tree
is
actually just a
normal
B-tree, the changes to the index set are handled according to the familiar
and deletion. 1 In the following examples, we
assume that the index set is a B-tree of order three, which means that the
maximum number of separators we can store in a node is two. We use this
small node size for the index set to illustrate node splitting and concatena"
rules for B-tree insertion
separators. As you will see later, actual
+
implementations of simple prefix B trees place a much larger number of
few
tion while using only a
separators in a
""As
we
you study
node of the index
you may find it helpful
much more detail.
the material here,
discuss B-tree operations in
set.
to refer
back to Chapter
8,
where
SIMPLE PREFIX B
We
assume
that there
is
block
set
A new block
hold the second half of what was originally the
is
shown
an insertion into the
that this insertion causes the block to split.
419
TREE MAINTENANCE
begin with an insertion into the sequence
Specifically, let's
in to
'
set,
block, and
brought
block. This new
(block 7)
first
linked into the correct position in the sequence
in Fig. 9.9.
first
is
following block
and preceding block 2 (these are the physical block numbers). These
changes to the sequence set are illustrated in Fig. 9.10.
Note that the separator that formerly distinguished between blocks 1
and 2, the string BO, is now the separator for blocks 7 and 2. We need a
new separator, with a value of AY, to distinguish between blocks 1 and 7.
As we go to place this separator into the index set, we find that the node into
which we want to insert it, containing BO and CAM, is already full.
Consequently, insertion of the new separator causes a split and promotion,
1
according to the usual rules for B-trees. The promoted separator,
placed in the root of the index
BO,
is
set.
Now let's
suppose we delete a record from block 2 of the sequence set
underflow condition and consequent concatenation of blocks
2 and 3. Once the concatenation is complete, block 3 is no longer needed in
the sequence set, and the separator that once distinguished between blocks
2 and 3 must be removed from the index set. Removing this separator,
CAM, causes an underflow in an index set node. Consequently, there is
that causes an
FIGURE 9.10 An insertion into block 1 causes a split and the consequent addition of block 7.
of a block in the sequence set requires a new separator in the index set. Insertion
of the AY separator into the node containing BO and CAM causes a node split in the index set
B-tree and consequent promotion of BO to the root.
The addition
420
THE
TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
FIGURE 9.1 1 A deletion from block 2 causes underflow and the consequent concatenation of
blocks 2 and 3. After the concatenation, block 3 is no longer needed and can be placed on
an avail list. Consequently, the separator CAM is no longer needed. Removing CAM from its
node in the index set forces a concatenation of index set nodes, bringing BO back down from
the root.
another concatenation,
this
time in the index
that
set,
results
in
the
demotion of the BO separator from the root, bringing it back down into a
node with the AY separator. Once these changes are complete, the simple
prefix
B+
tree has the structure illustrated in Fig. 9.11.
Although in these examples
node split in the index set, and
in a concatenation in the
index
block
concatenation in the sequence
set,
split in the
there
is
sequence
not always
this
set results in
set results
correspondence
of action. Insertions and deletions in the index set are handled as standard
B-tree operations; whether there is splitting or a simple insertion,
concatenation or a simple deletion, depends entirely on how full the index
set node is.
Writing procedures to handle these kinds of operations is a straightfor-
ward
task if you remember that the changes take place from the bottom up.
Record insertion and deletion always take place in the sequence set, since
that is where the records are. If splitting, concatenation, or redistribution is
necessary, perform the operation just as you would if there were no index set
at all. Then, after the record operations in the sequence set are complete,
make changes
If
as
blocks are
necessary in the index
split in the
serted into the index
sequence
set:
set, a
new
separator
must be
in-
set;
421
INDEX SET BLOCK SIZE
If
blocks are concatenated in the sequence
moved from
set, a
separator
must be
and
If records are redistributed between blocks in the sequence
value of a separator in the index set must be changed.
Index
set
the index
re-
set;
set,
the
operations are performed according to the rules for B-trees.
This means that node splitting and concatenation propagate up through the
set. We see this in our examples as the BO
and out of the root. Note that the operations on the
sequence set do not involve this kind of propagation. That is because the
sequence set is a linear, linked list, whereas the index set is a tree. It is easy
to lose sight of this distinction and think of an insertion or deletion in terms
+
of a single operation on the entire simple prefix B tree. This is a good way
to become confused. Remember: Insertions and deletions happen in the
sequence set since that is where the records are. Changes to the index set are
secondary; they are a byproduct of the fundamental operations on the
sequence set.
higher levels of the index
separator
9.7
moves
in
Index Set Block Size
Up
we
have ignored the important issues of the size and
Our examples have used extremely small
index set nodes and have treated them as fixed-order B-tree nodes, even
though the separators are variable in length. We need to develop more
realistic, useful ideas about the size and structure of index set nodes.
The physical size of a node for the index set is usually the same as the
physical size of a block in the sequence set. When this is the case, we speak
of index set blocks, rather than nodes, just as we speak of sequence set blocks.
There are a number of reasons for using a common block size for the index
and sequence sets:
to this point
structure of the index set nodes.
The block size for the sequence set is usually chosen because there is
a good fit between this block size, the characteristics of the disk
drive, and the amount of memory available. The choice of an index
set
block
fore, the
size
block
the index
size that
is
best for the sequence set
is
factors; there-
usually best for
set.
A common
scheme
governed by consideration of the same
is
block
size
makes
it
easier to
to create a virtual simple prefix
implement
B+
buffering
tree, similar to the virtual
B-trees discussed in the preceding chapter.
The index
set
blocks and sequence
within the same
file
blocks are often mingled
between two separate files
set
to avoid seeking
422
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
while accessing the simple prefix B tree. Use of one file for both
kinds of blocks is simpler if the block sizes are the same.
9.8
Internal Structure of Index Set Blocks:
A Variable-order B-Tree
Given
a large,
fixed-size block for the index
separators within
how do we
set,
store the
In the examples considered so far, the block structure
it?
such that it can contain only a fixed number of separators. The entire
motivation behind the use of shortest separators is the possibility of packing
more of them into a node. This motivation disappears completely if the
is
index
set uses a
fixed-order B-tree in which there
number of
a fixed
is
separators per node.
We
want each index
variable-length separators.
set
How
block
hold
to
should
we go
variable
number of
about searching through
these separators? Since the blocks are probably large, any single block can
hold
a large
we want
number of separators. Once we
to be able to
separators.
We
do
read a block into
RAM for use,
binary rather than sequential search on
therefore need to structure the block so
it
its list
can support
of
a
binary search, despite the fact that the separators are of variable length.
Chapter
In
6,
which covers indexing, we see that the use of a separate
a means of performing binary searches on a list of
index can provide
variable-length entities. If the index itself consists of fixed-length refer-
we
ences,
can use binary searching on the index, retrieving the actual
variable-length records or fields through indirection. For example, suppose
we
are
going to place the following
of separators into an index block:
set
As, Ba, Bro, C, Ch, Cra, Dele, Edi, Err, Fa,
(We
are using lowercase letters, rather than
find the separators
more
easily
all
when we
Fie.
uppercase
letters,
concatenate them.)
concatenate these separators and build an index for them,
as
so
you can
We
shown
could
in Fig.
9.12.
If
we
are using this block of the index set as a
the record in the sequence set for "Beck",
the index to the separators, retrieving
which
starts
separator
Our
in position
by looking
10.
Note
at the starting
binary search eventually
first
that
roadmap
we perform
to help us find
binary search on
the middle separator, "Cra",
we
can find the length of
this
position of the separator that follows.
tells
us
that
"Beck"
"Ba" and "Bro". Then what do we do?
The purpose of the index set roadmap is to guide
falls
between the
separators
the levels of the simple prefix
B+
tree,
us
downward through
leading us to the sequence set block
423
INTERNAL STRUCTURE OF INDEX SET BLOCKS: A VARIABLE-ORDER B-TREE
AsBaBroCChCraDeleEdiErrFaFle
00 02 04 07 08 10 13 17 20 23 25
Index
Concatenated
to separators
separators
FIGURE 9.12 Variable-length separators and corresponding index.
we want
to retrieve. Consequently, the index set block needs
store references to
lower
level
a relative
of the
its
children, to the blocks descending
We
tree.
assume
that the references are
block number (RBN), which
number except
If there are
that
it
from
is
analogous to
some way
it
made
to
in the next
in
terms of
a relative
record
references a fixed-length block rather than a record.
N separators within a block, the block has N + children, and
N + RBNs in addition to the separators and
1
therefore needs space to store
the index to the separators.
There are many ways to combine the list of separators, index to
and list of RBNs into a single index set block. One possible
approach is illustrated in Fig. 9.13. In addition to the vector of separators,
the index to these separators, and the list of associated block numbers, this
separators,
block structure includes:
Separator count:
We
need
this to help us find the
the index to the separators so
we
middle element
in
can begin our binary search.
Total length of separators: The list of concatenated separators varies in
length from block to block. Since the index to the separators begins
at the
list is
end of
so
we
this variable-length
we need
list,
to
know how
long the
can find the beginning of our index.
Let's suppose,
once again, that
we
are looking for a record with the
key
"Beck" and that the search has brought us to the index set block pictured in
Fig. 9.13. The total length of the separators and the separator count allows
Separator count
r
r
2H
Total length of separators
AsBaBroCChCraDeleEdiErrFaFle
Separators
00 02 04 07 08 10 13 17 20 23 25
*U Index
to separators
FIGURE 9.13 Structure of an index set block.
BOO B01 B02 B03 B04 B05 B06 B07 BOS B09 B10 Bll
*U
Relative block
numbers
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
424
Separator
subscript:
BOO
FILE
ACCESS
01234 56789
As
B01
Ba
B02
Bro B03
Ch
B04
B05 Cra B06 Dele B07
Edi
B08
Err
B09
Fa
10
BIO
Bll
Fie
FIGURE 9.14 Conceptual relationship of separators and relative block numbers.
us to find the beginning, the end, and consequently the middle of the index
to the separators.
As
in the preceding
of the separators through
example,
this index, finally
we perform
binary search
concluding that the key "Beck"
falls between the separators "Ba" and "Bro". Conceptually, the relation
between the keys and the RBNs is as illustrated in Fig. 9.14. (Why isn't this
good physical arrangement?)
As Fig. 9.14 makes clear, discovering that the key falls between "Ba"
and "Bro" allows us to decide that the next block we need to retrieve has the
RBN stored in the B02 position of the RBN vector. This next block could
be another index
set block,
could be the sequence
set
and thus another block of the roadmap, or
block that
we
it
are looking for. In either case, the
quantity and arrangement of information in the current index set block
is
conduct our binary search within the index block and then
+
proceed to the next block in the simple prefix B tree.
There are many alternate ways to arrange the fundamental components
of this index block. (For example, would it be easier to build the block if the
vector of keys were placed at the end of the block? How would you handle
sufficient to let us
the fact that the block consists of both character and integer entities with
no
constant, fixed dividing point between them?) For our purposes here, the
implementation details for this particular index block structure are
not nearly as important as the block's conceptual structure. This kind of index
block structure illustrates two important points.
The first point is that a block is not just an arbitrary chunk cut out of
a homogeneous file; it can be more than just a set of records. A block can
specific
have
a sophisticated internal structure all its
own, including
its
own internal
index, a collection of variable-length records, separate sets of fixed-length
records,
and so
forth.
This idea of building more sophisticated data
becomes increasingly attractive as the block
structures inside of each block
it becomes imperative that we have an
of the data within a block once it has been
+
read into RAM. This point applies not only to simple prefix B trees, but
to any file structure using a large block size.
size increases.
efficient
With very
large blocks
way of processing
all
LOADING A SIMPLE PREFIX B
The second
425
TREE
is that a node within the B-trce index set of our simple
of variable order, since each index set block contains a
variable number of separators. This variability has interesting implications:
prefix
B+
tree
point
is
The number of separators in a block is directly limited by block size
rather than by some predetermined order (as in an order M B-tree).
The index set will have the maximum order, and therefore the mini-
mum
depth, that
form
the separators.
possible given the degree of compression used to
is
of
Since the tree
is
when
is full,
block
comparing
mum.
or half
full,
are
separator count against
Decisions about
come more
The
variable order, operations
when
such
no longer
some
as
simple matter of
fixed
determining
maximum
or mini-
to split, concatenate, or redistribute be-
complicated.
exercises at the end of this chapter provide opportunities for
exploring variable-order trees more thoroughly.
9.9
Loading a Simple Prefix B
In the previous description
Tree
of the simple prefix
B*
tree,
we
on
something
focus
first
a sequence set, and subsequently present the index set as
added or built on top of the sequence set. It is not only possible to
conceive of simple prefix B^ trees this way, as a sequence set with an added
index, but one can also build them this way.
One way of building a simple prefix B~ tree, of course, is through a
series of successive insertions. We would use the procedures outlined in
+
section 9.6, where we discuss the maintenance of simple prefix B trees, to
split or redistribute blocks in the sequence set and in the index set as we
added blocks to the sequence set. The difficulty with this approach is that
splitting and redistribution are relatively expensive. They involve searching
down through the tree for each insertion and then reorganizing the tree as
necessary on the way back up. These operations are tine for tree maintenance
as the tree is updated, but when we are loading the tree we do not have to
contend with a random-order insertion and therefore do not need procedures
that are so powerful, flexible, and expensive. Instead, we can begin by
building
that
is
sorting the records that are to be loaded.
next record
we
encounter
Working from
blocks, one
fills
up.
by one,
sorted
file,
starting a
As we make
Then we can guarantee
the next record
is
we
new
the transition
we need
that the
to load.
can place the records into sequence set
when the one we are working with
between two sequence set blocks, we can
block
426
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
Next separator:
CAT
Next
CATCH-CHECK
sequence
block
set
FIGURE 9.15 Formation of the
first
index set block as the sequence set
is
loaded.
determine the shortest separator for the blocks. We can
separators into an index set block that we build and hold in
collect
these
RAM until
it is
full.
develop an example of how this works, let's assume that we have
of records associated with terms that are being compiled for a book
index. The records might consist of a list of the occurrences of each term.
In Fig. 9.15 we show four sequence set blocks that have been written out to
To
sets
the disk and one index set block that has been built in
shortest separators derived
from the sequence
set
RAM
from the
block keys. As you can
FIGURE 9.16 Simultaneous building of two index set levels as the sequence set continues to
grow.
CAT
00 -1
-1
1
Index block
containing no
separators
LOADING A SIMPLE PREFIX B + TREE
see,
427
the next sequence set block consists of
CATCH
through
CHECK,
a set of terms ranging from
and therefore the next separator is CAT. Let's
suppose that the index set block is now full. We write it out to disk. Now
what do we do with the separator CAT?
Clearly, we need to start a new index block. But we cannot place CAT
into another index block at the same level as the one containing the
separators ALW, ASP, and BET since we cannot have two blocks at the
same level without having a parent block. Instead, we promote the CAT
separator to a higher-level block.
point directly to the sequence
blocks. This
means
that
RAM
we
build
in
as
we
However, the higher-level block cannot
must point to the lower-level index
set; it
will
now
be building two levels of the index
sequence
the
set.
Figure
9.16
illustrates
set
this
working-on-two-levels phenomenon: The addition of the CAT separator
requires us to start a new, root-level index block as well as a lower-level
index block. (Actually, we are working on three levels at once since we are
also constructing the sequence set blocks in
the index looks like after even
RAM.)
more sequence
set
Figure 9.17 shows what
blocks are added. As you
see, the lower-level index block that contained no separators when we
added CAT to the root has now filled up. To establish that the tree works,
do a search for the term CATCH. Then search for the two terms CASUAL
and CATALOG. How can you tell that these terms are not in the sequence
can
set?
It
is
instructive to ask
CHECK,
what would happen
if
the last record
so the construction of the sequence sets and index sets
were
would stop
+
with the configuration shown in Fig. 9.16. The resulting simple prefix B
tree would contain an index set node that holds no separators. This is not an
isolated, one-time possibility. If we use this sequential loading method to
build the tree, there will be many points during the loading process at which
there is an empty or nearly empty index set node. If the index set grows to
more than two levels, this empty node problem can occur at even higher
levels of the tree, creating a potentially severe out-of-balance problem.
Clearly, these empty node and nearly empty node conditions violate the
B-tree rules that apply to the index set. However, once a tree is loaded and
goes into regular use, the very fact that a node is violating B-tree conditions
can be used to guarantee that the node will be corrected through the action
of normal B-tree maintenance operations. It is easy to write the procedures
for insertion and deletion so a redistribution procedure is invoked when an
underfull node
is
encountered.
The advantages of loading
sequential
operation
following
simple prefix
a
sort
B+
tree in this
of the records,
outweigh the disadvantages associated with the
way,
as a
almost always
possibility
of creating
428
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
ALHASPBET
ACCESS-ALSO
FIGURE 9.17 Continued growth
of index set built
up from the sequence
set.
blocks that contain too few records or too few separators.
advantage
is
that the loading process goes
more quickly
The output can be written sequentially;
We make only one pass over the data, rather than
associated with random order insertions; and
No
blocks need to be reorganized
as
we
proceed.
The
principal
since
the
many
passes
429
TREES
There are two additional advantages to using a separate loading process
we have described. These advantages are related to
performance after the tree is loaded rather than performance during loading:
such as the one
Random insertion produces blocks that are, on the average, between
67% and 80% full. In the preceding chapter, as we discussed B-trees,
we increased this storage utilization by mechanisms such as using redistribution during insertion rather than using just block splitting.
still, we never had the option of filling the blocks completely so
we had 100% utilization. The sequential loading process changes
this. If we want, we can load the tree so it starts out with 100% utilization. This is an attractive option if we do not expect to add very
many records to the tree. On the other hand, if we do anticipate
many insertions, sequential loading allows us to select any other degree of utilization that we want. Sequential loading gives us much
But,
more
control over the
newly loaded
amount and placement of empty space
in the
tree.
example presented
In the loading
in Fig. 9.16,
we
write out the
first
four sequence set blocks, then write out the index set block containing the separators for these sequence set blocks. If
file
we
use the same
for both sequence set and index set blocks, this process guaran-
tees that
quence
an index
set
set
block
blocks that are
starts
out in physical proximity to the se-
descendents. In other words, our se-
its
is creating a degree of spatial locality within
our file. This locality can minimize seeking as we search down
through the tree.
quential loading process
9.10
Trees
+
have focused primarily on simple prefix B
trees. These structures are actually a variant of an approach to file
+
organization known simply as a B Tree. The difference between a simple
Our
discussions
prefix
B+
tree
up
and
to this point
a plain
B+
tree
is
that the latter structure does not involve
the use of prefixes as separators. Instead, the separators in the index set are
simply copies of the actual keys. Contrast the index
9.18,
which
block that
B+
is
illustrates the initial
loading steps for a
illustrated in Fig. 9.15,
where we
set
shown
block
tree,
in Fig.
with the index
are building a simple prefix
tree.
The
operations performed on
discussed for simple prefix
trees consist
of
a set
B+
trees.
trees are essentially the
Both B
of records arranged
in
same
as
those
and simple prefix
key order in a sequence
trees
B+
set,
430
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
FILE
ACCESS
Next separator:
CATCH
Next
CATCH-CHECK
sequence
set
FIGURE 9.18 Formation of the
first
index set block
in
block
tree without the use of shortest
separators.
coupled with an index set that provides rapid access to the block containing
any particular key/record combination. The only difference is that in the
simple prefix B tree we build an index set of shortest separators formed
from key prefixes.
One of the reasons behind our decision to focus first on simple prefix
B trees, rather than on the more general notion of a B + tree, is that we
want to distinguish between the role of the separators in the index set and
keys in the sequence
set. It is
much more
difficult to
make
this distinction
when
the separators are exact copies of the keys. By beginning with simple
+
prefix B
trees, we have the pedagogical advantage of working with
separators that are clearly different than the keys in the sequence set.
+
But another reason for starting with simple prefix B trees revolves
around the
plain
B+
implies that
we
can.
they are quite often a more desirable alternative than the
want the index set to be as shallow as possible, which
we want to place as many separators into an index set block as
fact that
tree.
We
Why use anything longer than the simple prefix in the index set?
general, the
answer
to this question
is
anything longer than a simple prefix
prefix
B+
trees are often a
factors that
keys
might argue
good
that
not, in fact,
as a separator;
solution.
in favor
we do
of using
There
a
B+
are,
want
In
to use
consequently, simple
however,
at least
two
tree that uses full copies
of
as separators:
The reason
for using shortest separators
an index set block. As
we
have already
is
to
pack more of them into
said, this implies, ineluctably,
the use of variable-length fields within the index set blocks. For
some
applications the cost of the extra overhead required to maintain
and use
this variable-length structure
outweighs the benefits of
shorter separators. In these cases one might choose to build a
B-TREES, B
straightforward
TREES, AND SIMPLE PREFIX B
B+
TREES
IN
431
PERSPECTIVE
tree using fixed-length copies
of the keys from
the sequence set as separators.
Some key
fix
sets
method
is
do not show much compression when
the simple pre-
used to produce separators. For example, suppose the
keys consist of large, consecutive alphanumeric sequences such
34C18K756, 34C18K757, 34C18K758, and so
as
on. In this case, to en-
joy appreciable compression, we need to use compression techniques
that remove redundancy from the front of the key. Bayer and Unterauer (1977) describe such compression methods. Unfortunately,
they are more expensive and complicated than simple prefix compression. If we calculate that tree height remains acceptable with the
use of full copies of the keys as separators, we might elect to use the
no-compression option.
9.1
B-Trees, B Trees, and Simple Prefix B
Trees in Perspective
and the preceding chapter we have looked at a number of
+
file structures. These tools
B-trees, B trees, and
simple prefix B
trees
have similar-sounding names and a number of
In this chapter
"tools" used in building
common
features.
We
need
way
to differentiate these tools so
we
can
choose the most appropriate one for a given file structure job.
Before addressing this problem of differentiation, however, we should
point out that these are not the only tools in the toolbox. Because B-trees,
reliably
B+
trees,
and
their relatives are such powerful, flexible file structures,
it is
them as the answer to all problems.
Simple index structures of the kind discussed in
Chapter 6, which are maintained wholly in RAM, are a much simpler,
neater solution when they suffice for the job at hand. As we saw at the
indexes are not limited to direct
beginning of this chapter, simple
access situations. This kind of index can be coupled with a sequence set of
blocks to provide effective indexed sequential access as well. It is only when
that
the index grows so large that we cannot economically hold it in
+
we need to turn to paged index structures such as B-trees and B trees.
In the chapter that follows we encounter yet another tool, known as
hashing. Like simple RAM-based indexes, hashing is an important alterna+
tive to B-trees, B
trees, and so on. In many situations, hashing can provide
faster access to a very large number of records than can the use of a member
of the B-tree family.
into the trap of regarding
easy to
fall
This
a serious mistake.
is
RAM
RAM
432
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
B+
FILE
ACCESS
and simple prefix B trees are not a panacea.
However, they do have broad applicability, particularly for situations that
require the ability to access a large file both sequentially, in order by key,
and through an index. All three of these different tools share the following
So, B-trees,
trees,
characteristics:
They
tire
are
all
paged index structures, which means
blocks of information into
RAM
at
once.
many
possible to choose between a great
As
that they bring ena
consequence,
it is
alternatives (e.g., the keys
hundreds of thousands of records) with just a few seeks out to
The shape of these trees tends to be broad and shallow.
All three approaches maintain height-balanced trees. The trees do not
grow in an uneven way, which would result in some potentially
long searches for certain keys.
for
disk storage.
In
grow from
cases the trees
all
the
bottom up. Balance
is
main-
tained through block splitting, concatenation, and redistribution.
With
all
three structures
it is
possible to obtain greater storage effi-
ciency through the use of two-to-three splitting and of redistribution
in place
of block
splitting
scribed in Chapter
when
possible.
These techniques are de-
8.
All three approaches can be implemented as virtual tree structures in
which the most recently used blocks are held in RAM. The advantages of virtual trees were described in Chapter 8.
Any
of these approaches can be adapted for use with variable-length
a block similar to those outlined in
records using structures inside
this chapter.
For
of
all
this similarity, there are
some important
differences.
These
differences are brought into focus through a review of the strengths and
unique characteristics of each of these three
B-Trees
B-trees contain information that
member of
each pair
file
is
the key; the other
is
structures.
grouped
member
as a set
is
of pairs.
One
the associated infor-
These pairs are distributed over all the nodes of the B-tree. Consequently, we might find the information we are seeking at any level of the
+
+
B-tree. This differs from B trees and simple prefix B trees, which require
mation.
all
searches to proceed
all
the
way down
to the lowest,
sequence
set level
of
the tree. Because the B-tree itself contains the actual keys and associated
information, and there
is
therefore
Given
a large
no need
for additional storage to hold
B^
up
less
space than does
enough block
size
and an implementation
separators, a B-tree can take
tree as a virtual B-tree,
it is
tree.
that treats the
possible to use a B-tree for ordered sequential
access as well as for indexed access.
The ordered
sequential access
is
B-TREES, B
TREES, AND SIMPLE PREFIX B + TREES
obtained through an in-order traversal of the
virtual tree
tree.
433
PERSPECTIVE
IN
The implementation
as a
necessary so this traversal does not involve seeking as
is
it
returns to the next highest level of the tree. This use of a B-tree for indexed
sequential access
works only when
the record information
is
actually stored
within the B-tree. If the B-tree merely contains pointers to records that are
sequence off in some other
in entry
workable because of
all
then indexed sequential access
file,
is
not
the seeking required to retrieve the actual record
information.
B-trees are most attractive
each record stored in the
record,
when
the key itself comprises a large part of
When
the key
possible to build a broader,
is
it
tree.
is
only
small part of the
+
shallower tree using
tree
methods.
B+
that in the
set
The primary
Trees
tree
all
known
of blocks
difference
between the B
the key and record information
as the sequence set.
not in the upper-level, tree-like portion
sequence
set
provided through
is
tree
and the B-tree
contained in a linked
is
The key and record information
B+
of the
tree.
is
Indexed access to
is
this
conceptually (though not necessarily
B+
tree the index set
of copies of the keys that represent the boundaries between
sequence set blocks. These copies of keys are called separators since they
separate a sequence set block from its predecessor.
+
There are two significant advantages that the B tree structure provides
over the B-tree:
physically) separate structure called the index
set.
In a
consists
The sequence
set
can be processed in
a truly linear, sequential
way,
providing efficient access to records in order by key; and
The
ten
use of separators, rather than entire records, in the index set of-
means
index
set
that the
block in
number of separators
a
B+
that can be placed in a single
tree substantially exceeds the
could be placed in an equal-sized block in
records that
number of
a B-tree.
Sepa-
rators (copies of keys) are simply smaller than the key/record pairs
you can put more of them in a block of
given size, it follows that the number of other blocks descending
+
from that block can be greater. As a consequence, a B tree approach can often result in a shallower tree than would a B-tree apstored in a B-tree. Since
proach.
In
practice,
the latter of these
important one. The impact of the
it
is
first
two advantages
advantage
is
is
often the
more
lessened by the fact that
often possible to obtain acceptable performance during an in-order
traversal
B-tree.
of
B-tree through the page buffering mechanism of
a virtual
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
434
Simple Prefix
using
aB
B+
ACCESS
We just indicated that the primary advantage of
+
is that aB
tree sometimes allows us to
Trees
tree instead
FILE
of a B-tree
we can obtain a higher branching factor out
of the upper-level blocks of the tree. The simple prefix B + tree builds on
this advantage by making the separators in the index set smaller than the
keys in the sequence set, rather than just using copies of these keys. If the
build a shallower tree because
we can fit more of them into a block to obtain
an even higher branching factor out of the block. In a sense, the simple
+
+
prefix B
tree takes one of the strongest features of the B
tree one step
separators are smaller, then
farther.
The price we have to pay to obtain this separator compression and
consequent increase in branching factor is that we must use an index set
block structure that supports variable-length fields. The question of
whether
this price is
worth the gain
is
one that has
to be considered
on
case-by-case basis.
SUMMARY
We
begin
this
chapter by presenting a
new problem.
In previous chapters
we
provided either indexed access or sequential access in order by key, but
without finding an efficient way to provide both of these kinds of access.
This chapter explores one class of solutions to this problem, a class based on
the use of a blocked sequence set and an associated index set.
The sequence
Since
all
set
holds
all
of the
tions to the sequence set,
we
start
structures with an examination of a
changes.
file's
insertion or deletion operations
The fundamental
on
data records in order
the
file
by key.
begin with modifica-
our study of indexed sequential
method for managing sequence
tools used to insert
and delete records while
keeping everything in order within the sequence
set
are ones that
file
set
still
we
block concatenation, and
8: block splitting,
between blocks. The critical difference between
the use made of these tools for B-trees and the use made here is that there
is no promotion of records or keys during block splitting in a sequence set.
A sequence set is just a linked list of blocks, not a tree; therefore there is no
place to promote anything to. So, when a block splits, all the records are
divided between blocks at the same level; when blocks are concatenated
there is no need to bring anything down from a parent node.
encountered in
Chapter
redistribution of records
In this chapter,
sequence
set
we
also discuss the question of
blocks. There
is
no
precise
answer
we
how
large to
can give to
this
make
question
SUMMARY
between applications and environments.
since conditions vary
block should be large, but not so large that
in
RAM
or cannot read in
of
a single
Once we
a seek. In
of a cluster (on sector-formatted
disks) or
disk track.
are able to build
and maintain
sequence
set,
matter of building an index for the blocks in the sequence
enough
In general a
cannot hold several blocks
block without incurring the cost of
practice, blocks are often the size
the size
we
we
turn to the
set. If
the index
RAM,
one very satisfactory solution is to use a
simple index that might contain, for example, the key for the last record in
every block of the sequence set.
is
small
If the
index
to
fit
in
set turns
out to be too large to
fit
in
RAM, we recommend
same strategy we developed in the preceding chapter when
simple index outgrows the available RAM space: We turn the index into
the use of the
a
a
combination of a sequence set with a B-tree index set is our
+
first encounter with the structure known as a B
tree.
+
Before looking at B trees as complete entities, we take a closer look at
the makeup of the index set. The index set does not hold any information
that we would ever seek for its own sake. Instead, an index set is used only
as a roadmap to guide searches into the sequence set. The index set consists
of separators that allow us to choose between sequence set blocks. There are
many possible separators for any two sequence set blocks, so we might as
well choose the shortest separator. The scheme we use to find this shortest
separator consists of finding the common prefix of the two keys on either
side of a block boundary in the sequence set, and then going one letter
+
beyond this common prefix to define a true separator.
tree with an
index set made up of separators formed in this way is called a simple prefix
B-tree. This
AB
B+
tree.
We study the mechanism used to maintain the index set as insertions
+
and deletions are made in the sequence set of a B tree. The principal
observation we make about all of these operations is that the primary action
is where the records are. Changes to the
of the fundamental operations
byproduct
set are secondary; they are a
on the sequence set. We add a new separator to the index set only if we form
a new block in the sequence set; we delete a separator from the index set
only if we remove a block from the sequence set through concatenation.
Block overflow and underflow in the index set differ from the operations on
the sequence set in that the index set is potentially a multilevel structure and
is
within the sequence
set,
since that
index
is
therefore handled as a B-tree.
The
size
of blocks in the index
for the sequence set.
variable-length
To
separators
set is usually the
same
as the size
create blocks containing variable
while
at
the
chosen
numbers of
same time supporting binary
435
436
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
searching,
we
block header
FILE
ACCESS
develop an internal structure for the block that consists of
count and total separator length), the
fields (for the separator
variable-length separators themselves, an index to these separators, and a
vector of relative block numbers
(RBNs)
for the blocks descending
from
the index set block. This illustrates an important general principle about
large blocks within
homogeneous
set
structure of their
We
file
structures:
They
are
more than just
of records; blocks often have
own,
apart
from the
out of a
a slice
a sophisticated internal
larger structure of the file.
+
a B
tree.
find that if
We
turn next to the problem of loading
we
of records sorted by key, we can use a single-pass, sequential
process to place these records into the sequence set. As we move from block
to block in building the sequence set, we can extract separators and build the
blocks of the index set. Compared to a series of successive insertions that
work down from the top of the tree, this sequential loading process is much
more efficient. Sequential loading also lets us choose the percentage of space
utilized, right up to a goal of 100%.
The chapter closes with a comparison of B-trees, B + trees, and simple
+
+
prefix B
trees offer over B-trees
trees. The primary advantages that B
with
start
a set
are:
They support
The index set
records, so
than
We
true indexed sequential access; and
contains only separators, rather than full keys and
+
often possible to create a B tree that is shallower
it is
a B-tree.
suggest that the second of these advantages
important
indexed sequential access in
many
circumstances.
often the
The simple
second advantage and carries it farther,
separators and potentially producing an even shallower
+
tree is that
this extra compression in a simple prefix B
variable-length fields and a variable-order tree.
takes
is
more
one, since treating a B-tree as a virtual tree provides acceptable
this
prefix
B+
tree
compressing the
tree.
The
we must
price for
deal with
KEY TERMS
A B+
of records that are ordered
set that provides indexed
access to the records. All of the records are stored in the sequence
set. Insertions and deletions of records are handled by splitting, concatenating, and redistributing blocks in the sequence set. The index
set, which is used only as a finding aid to the blocks in the sequence
tree.
sequentially
set, is
tree consists
of
a sequence set
by key, along with an
managed
as a B-tree.
index
EXERCISES
Index
set.
The index
set consists
of separators that provide information
+
about the boundaries between the blocks in the sequence set of a B
tree. The index set can locate the block in the sequence set that contains the record
corresponding to
Indexed sequential access. Indexed
a certain
key.
sequential access
is
not actually
method, but rather a term used to describe situations in
which a user wants both sequential access to records, ordered by
+
key, and indexed access to those same records. B trees are just one
method for providing indexed sequential access.
Separator. Separators are derived from the keys of the records on either
side of a block boundary in the sequence set. If a given key is in one
of the two blocks on either side of a separator, the separator reliably
tells the user which of the two blocks holds the key.
Sequence set. The sequence set is the base level of an indexed sequential
+
file structure, such as B
tree. It contains all of the records in the
single-access
file.
When
read in logical order, block after block, the sequence set
lists all of the records in order by key.
Shortest separator. Many possible separators can be used to distinguish
between any two blocks in the sequence set. The class of shortest
separators consists of those separators that take the least space, given
a particular
We looked carefully at a compresremoving as many letters as possible
of the separators, forming the shortest simple prefix
compression strategy.
sion strategy that consists of
from the
that can
rear
still
Simple prefix
serve as a separator.
tree.
B + tree in which the index set
B+
shortest separators that are
simple prefixes,
as
is
made up of
described in the defini-
tion for shortest separator.
of variable order when the number of direct
is variable. This occurs
when the B-tree nodes contain a variable number of keys or separators. This form is most often used when there is variability in the
+
trees always make
lengths of the keys or separators. Simple prefix B
use of a variable-order B-tree as an index set so it is possible to take
advantage of the compression of separators and place more of them
Variable order.
B-tree
is
descendents from any given node of the tree
in a block.
m
EXERCISES
1.
Describe
access:
(a)
file
structures that permit each of the following types of
sequential
sequential access.
access
only;
(b)
direct
access
only;
(c)
indexed
437
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
438
2.
A B+
tree
structure
sequential access. Since
whenever
B+
is
FILE
ACCESS
generally superior to a B-tree for indexed
why not use a B + tree
trees incorporate B-trees,
hierarchical indexed structure
is
called for?
Consider the sequence set shown in Fig. 9.1(b). Show the sequence set
keys DOVER and EARNEST are added; then show the sequence
set after the key DAVIS is deleted. Did you use concatenation or
redistribution for handling the underflow?
3.
after the
4.
What
considerations affect your choice of a block size for constructing
you know something about expected patterns of access
random versus an even division
between the two), how might this affect your choice of block size? On a
sector-oriented drive, how might sector size and cluster size affect your
a
sequence
set? If
(primarily sequential versus primarily
choice of
5.
It
block size?
possible to construct an indexed sequential
is
tree-structured index.
could be used.
index?
6.
Under what
Under what
(such as an
AVL
The index
of
discussed in Chapter
a
8,
without using
conditions might one consider using such an
conditions might
tree) rather
set
file
simple index like the one developed in Chapter 6
B+
than
tree
is
it
be reasonable to use
binary tree
B-tree for the index?
just a B-tree, but unlike the B-trees
the separators do not have to be keys.
Why
the
difference?
7.
How
differ
does block splitting in the sequence
from block
splitting in the
index
set
of a simple prefix
B+
tree
set?
key BOLEN in the simple prefix B tree in Fig. 9.8 is deleted
from the sequence set node, how is the separator BO in the parent node
8.
If the
affected?
+
Consider the simple prefix B tree shown in Fig. 9.8. Suppose a key
added to block 5 results in a split of block 5 and the consequent addition of
block 8, so blocks 5 and 8 appear as follows:
9.
FABER-FINGER
FINNEY-FOLK
3
a.
b.
What does
Suppose
the tree look like after the insertion?
that,
subsequent to the insertion,
a deletion causes
under-
EXERCISES
flow and the consequent concatenation of blocks 4 and
5.
What does
the tree look like after the deletion?
c.
Describe
a case in
which
10.
Why
often a
is it
good
a deletion results in redistribution, rather
show
than concatenation, and
the effect
it
has
on the
tree.
idea to use the same block size for the index set
+
and the sequence set in a simple prefix B tree? Why should the index
nodes and the sequence set nodes usually be kept in the same file?
11.
Show
conceptual view of an index
illustrated in Fig. 9.12, that
Ab Arch
Also show
Astron
more
is
set block,
similar to the
set
one
loaded with the separators
B Bea
detailed
view of the index block,
as illustrated in Fig.
9.13.
of records is sorted by key, the process of loading a B
tree can be handled by using a single-pass sequential process, instead of
randomly inserting new records into the tree. What are the advantages of
this approach?
12. If the initial set
13.
Show how
the simple prefix
B+
tree in Fig. 9.17
changes after the
addition of the node
ITEMIZE-JAR
Assume
that the index set
does not have
14.
a
Use
room
for the
node containing the separators EF, H, and IG
new
separator but that there
the data stored in the simple prefix
room
in the root.
B+
tree in Fig. 9.17 to construct
+
tree is of order four.
that the index set of the B
+
+
the resulting B
tree.
tree with the simple prefix B
tree.
Compare
is
Assume
15. The use of variable-length separators and/or key compression changes
some of the rules about how we define and use a B-tree and how we
measure B-tree performance.
a. How does it affect our definition of the order of
b.
Suggest
criteria for
deciding
when
B-tree?
splitting, concatenation,
and
redistribution should be performed.
c.
What
difficulties arise in estimating
maximum number
16.
in
Make
a table
terms of the
simple prefix
B+
tree height,
of accesses, and space?
comparing
criteria listed
B+
and simple prefix B trees
below. Assume that the B-tree nodes do not
B-trees,
trees,
439
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
440
ACCESS
FILE
RRNs of data
answers based on a
contain data records, but only keys and corresponding
some
records. In
tree's
will
you will be able to give
number of keys in the tree.
cases
height or the
depend on unknown
factors,
such
specific
In other cases, the answers
of access or average
as patterns
separator length.
The number of accesses required to retrieve a record from
of height h (average, best case, and worst case).
a.
The number of accesses
b.
worst
c.
a tree
required to insert a record (best and
cases).
The number of accesses
worst
required to delete
record (best and
cases).
The number of accesses required
to process a file of n keys seassuming that each node can hold a maximum of k keys
and a minimum of k/2 keys (best and worst cases).
e. The number of accesses required to process a file of n keys sequentially, assuming that there are h + 1 node-sized buffers availd.
quentially,
able.
17. Some commercially available indexed sequential file organizations are
based on block interval splitting approaches very similar to those used with
B+
trees.
which
is
VSAM
called
key-sequenced access
+
much
organized
how
IBM's
like a
offers the user several file access
tree.
and which
Look up
results
a description
modes, one of
in
file
being
of VSAM and report
key-sequenced organization relates to a B tree, and also how it
offers the user file handling capabilities well beyond those of a straightforward B tree implementation. (See the Further Readings section of this
chapter for articles and books on VSAM.)
on
its
Although
methods now
18.
B+
trees
provide the basis for most indexed sequential access
in use, this was not always the case. A method called ISAM
Readings for this chapter) was once very common, especially
on large computers. ISAM uses a rigid tree-structured index consisting of at
least two and at most three levels. Indexes at these levels are tailored to the
specific disk drive being used. Data records are organized by track, so the
lowest level of an ISAM index is called the track index. Since the track index
points to the track on which a data record can be found, there is one track
(see Further
index for each cylinder.
overflow, the track
is
When
not
the addition of data records causes a track to
split.
Instead, the extra records are put into a
separate overflow area and chained together in logical order. Hence, every
entry in a track index
addition to
The
its
may
pointer to the
essential
difference
tree- like organizations
is
contain
home
pointer to the overflow area, in
track.
between the
in the
ISAM
way overflow
organization
and
B+
records are handled. In the
EXERCISES
of ISAM, overflow records are simply added to a chain of overflow
+
the index structure is not altered. In the B
tree case, overflow
records are not tolerated. When overflow occurs, a block is split and the
case
records
index structure
Can you
accommodate
altered to
the extra data block.
think of any advantages of using the
more
index
Why do you
separate overflow areas to handle overflow
+
think B tree- like approaches are replacing those that
use overflow chains to hold overflow records? Consider the
in
rigid
ISAM, with
structure of
records?
is
terms of both sequential and direct access,
two approaches
as well as addition
and deletion
of records.
Programming Exercises
We
chapter by discussing operations on
a sequence set, which is
of blocks containing records. Only later do we add the
concept of an index set to provide faster access to the blocks in the sequence
set. The following programming problems echo this approach, requiring
begin
this
just a linked
you
list
to write a
first
program
that builds a sequence set,
and
to the sequence
functions that maintain the sequence
functions to add an index set
programs can be implemented
19.
Write
program
set,
in either
that accepts a
file
then to write
finally to write
creating a
set,
programs and
tree. These
B+
or Pascal.
of strings
as input.
The input
file
should be sorted so the strings are in ascending order. Your program should
use this input
The
file
strings are stored in 15-byte records;
sequence
Sequence
The
to build a sequence set with the following characteristics:
first
set
set
block
is
128 bytes long;
blocks are doubly linked;
block in the output
other things,
header block containing,
file is a
a reference to the
RRN
of the
first
among
block in the se-
quence set;
Sequence set blocks are loaded so they are as full as possible; and
Sequence set blocks contain other fields (other than the actual records
containing the strings) as needed.
Write an update program that accepts strings input from the keyboard,
along with an instruction either to search, add, or delete the string from the
20.
sequence
set.
The program should have
the following characteristics:
Strings in the sequence set must, of course, be kept in order;
Response to the search instruction should be
either
found or not
found;
string should not be
added
if
it is
already in the sequence
set;
441
THE B + TREE FAMILY AND INDEXED SEQUENTIAL
442
Blocks in the sequence
half
set
FILE
ACCESS
should never be allowed to be
less
than
and
full;
Splitting, redistribution,
and concatenation operations should be
written as separate procedures so they can be used in subsequent pro-
gram development.
21. Write a
assume
program
file
form of
a B-tree.
B+
two
You may
levels.
The
should have the following characteristics:
The index
Do
set in the
that the B-tree index will never be deeper than
resulting
that traverses the sequence set created in the preceding
and that builds an index
exercises
set
and the sequence
set,
taken together, should constitute
tree;
not compress the keys
as
you form
the separators for the index
set;
Index
set blocks, like
and
Index
set
blocks.
sequence
set blocks,
should be 128 bytes long;
blocks should be kept in the same
The header block should
file as
the sequence set
contain a reference to the root of
the index set as well as the already existing reference to the begin-
ning of the sequence
22. Write a
new
set.
version of the update program that acts on the entire
B+
you created
in the preceding exercise. Search, add, and delete
should be supported, as they are in the earlier update program.
B-tree characteristics should be maintained in the index set; the sequence set
tree that
capabilities
should, as before, be maintained so blocks are always at least half
full.
Consider the block structure illustrated in Fig. 9.13, in which an index
is used to permit binary searching for a key in an index page.
Each index set block contains three variable length sets of items: a set of
separators, an index to the separators, and a set of relative block numbers.
Develop code in Pascal or C for storing these items in an index block and
for searching the block for a separator. You need to answer such questions
23.
to separators
as:
Where should
the three sets be placed relative to one another?
Given the data types permitted by the language you are using, how
can you handle the fact that the block consists of both character and
integer data with no fixed dividing point between them?
As items are added to a block, how do you decide when a block is
too
full to insert
another separator?
FURTHER READINGS
FURTHER READINGS
The
suggestion for the
initial
IP
tree structure appears to
have come from Knuth
(1973b), although he did not name or develop the approach. Most of the literature
+
that discusses B
trees in detail (as opposed to describing specific implementations
VSAM)
such as
provides what
is
form of
in the
articles rather
than textbooks.
perhaps the best brief overview of
B^
Comer
(1979)
Bayer and Untcrauer
(1977) offer a definitive article describing techniques for compressing separators.
The article includes consideration of simple prefix B^ trees as well as a more general
+
approach called a prefix B tree. McCreight (1977) describes an algorithm for taking
advantage of the variation in the lengths of separators in the index set of a B" tree.
McCreight's algorithm attempts to ensure that short separators, rather than longer
promoted
ones, are
up
is
in the tree
shallower
have
as
trees.
split. The intent is to shape the tree so blocks higher
number of immediate descendents, thereby creating a
blocks
a greater
tree.
Rosenberg and Snyder (1981) study the effects of initializing a compact B-tree
on later insertions and deletions. The use of batch insertions and deletions to B-trees,
+
rather than individual updates, is proposed and analyzed in Lang et al. (1985). B
trees are
compared with more
rigid indexed sequential
file
organizations (such as
Batory (1981) and in IBM's VSAM Planning Guide.
There are many commercial products that use methods related to the
ISAM)
in
B+
operations described in this chapter, but detailed descriptions of their underlying
structures are scarce.
An
exception to
(VSAM), one of
the
sequential access.
Wagner
insights into the early
this
is
tree
file
IBM's Virtual Storage Access Method
most widely used commercial products providing indexed
(1973) and Keehn and Lacy (1974) provide interesting
thinking behind VSAM. They also include considerations of
key maintenance, key compression, secondary indexes, and indexes to multiple data
sets. Good descriptions of VSAM can be found in several sources, and from a
variety of perspectives, in
IBM's
(VSAM
in a
B+
VSAM Planning Guide,
Bohl
(1981),
Comer
(1979)
Bradley (1982) (emphasis on implementation
PL/I environment), and Loomis (1983) (with examples from COBOL).
as
an example of
tree),
VAX-11 Record Management Services (RMS). Digital's file and record access
subsystem of the VAX/ VMS operating system, uses a B"" tree- like structure to
support indexed sequential access (Digital, 1979). Many microcomputer implementations
of B
trees can
(Borland, 1984).
be found, including dBase
III
and Borland's Turbo Toolbox
443
Hashing
10
CHAPTER OBJECTIVES
Introduce the concept of hashing.
Examine
the
problem of choosing
one
algorithm, present a reasonable
describe
some
good
hashing
in detail,
and
others.
Explore three approaches for reducing collisions: randomization of addresses, use of extra memory, and
storage of several records per address.
Develop and use mathematical tools for analyzing
performance differences resulting from the use of
different hashing techniques.
Examine problems associated with file
and discuss some solutions.
Examine
effects
of patterns of record
deterioration
access
on perfor-
mance.
445
CHAPTER OUTLINE
10.1
10.5.2 Search Length
Introduction
10.1.1
What
Hashing?
is
10.6 Storing
More Than One Record
10.1.2 Collisions
per Address: Buckets
10.2
10.6.1
10.3
Hashing Functions and Record
on
Performance
10.6.2 Implementation Issues
Simple Hashing Algorithm
Effects of Buckets
Distributions
10.3.1
10.3.2
Distributing Records
10.7
among
Making Deletions
Tombstones
10.7.1
Some Other Hashing Methods
Deletions
10.7.2 Implications of Tombstones
10.3.3 Predicting the Distribution of
for Insertions
Records
10.7.3 Effects of Deletions and
10.3.4 Predicting Collisions for a Full
Additions on Performance
File
10.4
Handling
for
Addresses
How Much
Memory
Be Used?
Other Collision Resolution
Techniques
10.4.1 Packing Density
10.8.1
10.4.2 Predicting Collisions for
10.8.2 Chained Progressive
Extra
Should
10.8
Double Hashing
Overflow
10.8.3 Chaining with a Separate
Different Packing Densities
Overflow Area
Resolution by
Progressive Overflow
10.5 Collision
10.5.1
How
10.8.4 Scatter Tables: Indexing
Revisited
Progressive Overflow
10.9 Patterns
Works
10.1
of Record Access
Introduction
O(l) access to
a
files
means
that
how
no matter
record always takes the same, small
sequential searching gives us
grows
O(N)
in proportion to the size
number of
access,
of the
big the
file.
file
grows, access to
seeks.
By
contrast,
wherein the number of seeks
As we saw
in the preceding
chapters, B-trees
improve on
N)
access; the
number of seeks
increases as the logarithm to the base k of the
number of
records,
where k
is
this greatly,
measure of the
providing 0(\og k
leaf size.
0(\og k N) access can provide
very good retrieval performance, even for very large
files,
but
it is still
not
O(l) access.
Holy Grail of file structure design.
what we want to achieve, but until
In a sense, O(l) access has been the
Everyone agrees
that O(l) access
is
~_i
447
INTRODUCTION
about 10 years ago
was not
it
one could develop
clear that
O(l) access strategies that would work on dynamic
that
files
general class of
change greatly
in size.
In this chapter
They provide
in size.
we
begin with
Static
hashing was the
following chapter
we show how
begun to
dynamic and
ways
has
find
hash Junction
is
art until
research and design
and O(l)
file
increases
about 1980.
work during
In
the
the 1980s
access, to files that are
Hashing?
like a black
is
The
box
retrieving records. In Fig.
be the home address of
is
like
produces an address every time you
that
it is
function h(K) that transforms
resulting address
10.1, the
hash function to the address
Hashing
of the
state
to extend hashing,
in a key. ISAore formally,
into an address.
to
description of static hashing techniques.
increase in size over time.
10.1.1 What
drop
us with O(l) access but are not extensible as the
4.
That
key
is.
used
key
as the basis for storing
LOWELL
is
//(LOWELL) =
and
transformed by the
4.
Address 4
is
said
LOWELL.
indexing in that
relative record address.
is
Hashing
it
differs
involves associating
from indexing
in
key with
two important
ways:
With hashing, the addresses generated appear to be random
there is
no immediately obvious connection between the key and the location
of the corresponding record, even though the key is used to determine the location of the record. For this reason, hashing is sometimes referred to
as randomizing.
With hashing, two different keys may be transformed to the same
address so two records may be sent to the same place in the file.
When this occurs, it is called a collision and some means must be
found
to deal
with
it.
Consider the following simple example. Suppose you want to store 75
file, where the key to each record is a person's name. Suppose
also that you set aside space for 1.000 records. The key can be hashed by
taking two numbers from the ASCII representations of the first two
records in a
of the name, multiplying these together, then using the
rightmost three digits of the result for the address. Table 10.1 shows how
three names would produce three addresses. Note that even though the
characters
names
are listed in alphabetical order, there
addresses.
They appear
to be in random order.
is
no apparent order
to the
448
HASHING
Address
Record
Key
K = LOWE
Address
FIGURE 10.1
LOWELL.
r:
Hashing the key
LOWELL
to
address
LOWELL'S
home address
4.
10.1.2 Collisions
Now
sample file with the name OLIVIER.
with the same two letters as the name
they produce the same address (004). There is a collision between
suppose there
Since the
is
key
name OLIVIER
LOWELL,
the record for
in the
starts
OLIVIER and the
hash to the same address
as
Collisions cause problems.
space, so
we must
record for
LOWELL. We refer to
keys that
synonyms.
We
cannot put two records in the same
resolve collisions.
We
do
this in
two ways: by choosing
hashing algorithms partly on the basis of how few collisions they are likely
to produce,
TABLE 10.1
Name
and by playing some
tricks
with the ways
we
store records.
A simple hashing scheme
ASCII Code
First
Two
for
Letters
Product
Home
Address
BALL
66
65
66 x 65
LOWELL
76
79
76 x 79
TREE
84
82
84 x 82
=
=
-
4,290
290
6,004
004
6,888
888
449
INTRODUCTION
The
ideal solution to collisions
that avoids collisions altogether.
turns out to be
is
to find a transformation algorithm
Such an algorithm
much more
is
called a perfect hashing
hashing
algorithm than one might expect, however. Suppose, for example, that you
algorithm.
want
It
to store 4,000 records
difficult to find a perfect
among
5,000 available addresses. It can be
of the huge number of possible hashing
12(,0,)l)
algorithms for doing this, only one out of l()
avoids collisions
altogether. Hence, it is usually not worth trying.^
shown (Hanson,
A more
1982)
that
practical solution
is
acceptable number. For example,
to reduce the
if
number of
collisions to an
only one out of 10 searches for
record
results in a collision, then the average
number of
retrieve a record remains quite low.
There are several different ways to
reduce the
number of collisions,
Spread out the
compete
records.
for the
including the following three:
Collisions occur
same
address. If
that distributes the records fairly
dresses, then
we would
disk accesses required to
we
when two
could find
or
more records
hashing algorithm
randomly among
the available ad-
not have large numbers of records clustering
around certain addresses. Our sample hash algorithm, which uses
only two letters from the key, is not good on this account because
certain combinations of two letters are quite common in starting
names, while others are uncommon (e.g., compare the number of
names that start with "JO" with the number that start with "XZ").
We need to find a hashing algorithm that distributes records moi
randomly.
Use extra memory.
It is
easier to find a hash
algorithm that avoids col-
we have only a few records to distribute among many adwe have about the same number of records as adOur sample hashing algorithm is very good on this account
lisions if
dresses than if
dresses.
since there are 1,000 possible addresses and only 75 addresses (corre-
sponding to the 75 records) will be generated. The obvious disadvantage to spreading out the records
is
that storage space
is
wasted.
(In
the example, 7.5% of the available record space is used, and the remaining 92.5% is wasted.) There is no simple answer to the question
of how much empty space should be tolerated to get the best hash-
ing performance, but
some techniques
are provided later in this
not unreasonable to try to generate perfect hashing functions for small (less than 500).
of keys, such as might be used to look up reserved words in a programming language. But files generally contain more than a few hundred keys, or they contain sets of
keys that change frequently, so they are not normally considered candidates for perfect
hashing functions. See Knuth (1973b), Sager (1985), Chang (1984), and Chichelli (1980) for
more on perfect hashing functions.
'''It
is
stable sets
450
HASHING
chapter for measuring the relative gains in performance for different
amounts of
free space.
Put more than one record
sumed
tacitly that
exactly one record, but there
create our
file
in
Up
at a single address.
to
now we
each physical record location in
such
way
is
usually
that every
no reason
file
a file
have ascould hold
why we
address
is
cannot
big enough to
hold several records.
If, for example, each record is 80 bytes long,
with 512-byte physical records, we can store up
to six records at each file address. Each address is able to tolerate five
synonyms. Addresses that can hold several records in this way are
sometimes called buckets.
and
we
create a
file
In the following sections
methods, and
as
we do
so
we
we
elaborate
present
on these collision-reducing
for managing hashed
some programs
files.
10.2
A Simple Hashing Algorithm
One
goal in choosing any hashing algorithm should be to spread out
records as uniformly as possible over the range of addresses available.
use of the term hash for this technique suggests what
Our
is
done
The
to achieve this.
to hash means "to chop into small
muddle or confuse." The algorithm used previously chops off
the first two letters and then uses the resulting ASCII codes to produce a
number that is in turn chopped to produce the address. It is not very good
at avoiding clusters of synonyms because so many names begin with the
same two letters.
One problem with the algorithm is that it does not really do very much
hashing. It uses only two letters of the key and it does not do much with the
two letters. Now let us look at a hash function that does much more
randomizing, primarily because it uses more of the key. It is a reasonably
good basic algorithm and is likely to give good results no matter what kinds
dictionary reminds us that the verb
pieces
of keys are used.
a specific
It is
also an algorithm that
is
instance of the algorithm does not
not too
work
difficult to alter in case
well.
This algorithm has three steps:
1.
Represent the key in numerical form.
2.
Fold and add.
3.
Divide by
prime number and use the remainder
Step 1. Represent the Key in Numerical Form
number, then this step is already accomplished. If it
as the address.
key is already a
string of characters,
If the
is
451
A SIMPLE HASHING ALGORITHM
we
ASCII code of each
take the
character and use
it
to
form
number. For
example,
76 79 87 69 76 76 32 32 32 32 32 32
nlir
=
LOWELL
L
D
W
L
L
L
Blanks
,
In this algorithm
letters.
By
among
differences
we
more
using
use the entire key, rather than just the
we
parts of a key,
first
two
increase the likelihood that
The
when
the keys cause differences in addresses produced.
extra processing time required to do this
compared
>t
<
to the potential
improvement
is
usually insignificant
in performance.
Step 2. Fold and Add Folding and adding means chopping off pieces of the
number and adding them together. In our algorithm we chop off pieces
with two ASCII numbers each:
76 79
These number
87 69
76 76
',
32 32
thought of
pairs can be
32 32
32 32
as integer variables
(rather than
which is how they started out) so we can do arithmetic
on them. If we can treat them as integer variables, then we can add them.
This is easy to do in C because C allows us to do arithmetic on characters.
character variables,
In Pascal,
we
can use the ord() function to obtain the integer position of
character within the computer's character
we add
Before
the fact that in
limited.
32,767
the numbers,
most
we
set.
have to mention
cases the sizes of
a problem caused by
numbers we can add together are
On some microcomputers,
(15 bits)
adding the
first
for example, integer values that exceed
overflow
errors
or become negative. For example,
cause
five of the foregoing numbers gives
7679 + 8769 + 7676 + 3232 + 3232 = 30588.
Adding
in the last 3,232
would, unfortunately, push the result over the
33,820), causing an overflow error.
32,767 (30,588 + 3,232 =
Consequently, we need to make sure
maximum
32,767.
We
can do this by
first
that each successive
sum
is
less
identifying the largest single value
we
than
will
ever add in our summation, and then making sure after each step that our
intermediate result differs from 32,767 by that amount.
In our case, let us
assume
that keys consist only
of blanks and uppercase
ZZ.
Suppose we choose 19,937 as our largest allowable intermediate result. This
differs from 32,767 by much more than 9,090, so we can be confident (in
this example) that no new addition will cause overflow. We can ensure in
our algorithm that no intermediate sum exceeds 19,937 by using the mod
alphabetic characters, so the largest addend
is
9,090, corresponding to
452
HASHING
which returns
operator,
the remainder
when one
integer
is
divided by
another:
+
+
4187 +
7419 +
10651 +
7679
16448
The number 13,883
Why
did
we
8769 -* 16448
7676 -> 24124
3232^
7419
7419
3232 - 10651
3232 -> 13883
10651
is
mod 19937 ->
mod 19937 ->
mod 19937-^
mod 19937 -
mod 19937 -^
16448
24124
13883
16448
4187
7419
10651
13883
the result of the fold-and-add operation.
bound
use 19,937 as our upper
rather than, say, 20,000?
Because the division and subtraction operations associated with the mod
operator are more than just a way of keeping the number small; they are
part of the transformation work of the hash function. As we see in the
number
more random distribution than does transformation by
number 19,937 is prime.
discussion for the next step, division by a prime
usually produces
nonprime. The
Divide by the Size of the Address Space The purpose of this
down to size the number produced in step 2 so it falls within
the range of addresses of records in the file. This can be done by dividing
that number by a number that is the address size of the file, and then taking
the remainder. The remainder will be the home address of the record.
Step
step
3.
is
to cut
We
can represent this operation symbolically as follows: If 5 represents
sum produced in step 2 (13,820 in the example), n represents the divisor
(the number of addresses in the file), and a represents the address we are
trying to produce, we apply the formula
the
The remainder produced by
and n
the
file.
mod
mod
n.
operator will be
number between
1.
Suppose, for example, that
our
we
decide to use the 100 addresses
0-99
for
In terms of the preceding formula,
a
= 13820 mod
100
20.
Since the number of addresses allocated for the file does not have to be
any specific size (as long as it is big enough to hold all of the actual records
to be stored in the file), we have a great deal of freedom in choosing the
divisor n. It is a good thing that we do, because the choice of n can have a
how
major
effect
on
prime
number
distribute
well the records are spread out.
is
nonprime can
usually used for the divisor because primes tend to
much more uniformly than do nonprimes. A
work well in many cases, however, especially if it has no
remainders
453
HASHING FUNCTIONS AND RECORD DISTRIBUTIONS
FUNCTION hash(KEY,MAXAD)
set
set
SUM to
to
while (J
<
12)
SUM to (SUM + 100*KEY[J]
ncremen t J by 2
set
i
endwh i
KEYCJ
1]) mod 19937
return (SUM mod MAXAD)
end FUNCTION
FIGURE 10.2 Function hash( KEY, MAXAD) uses folding and prime number division
compute a hash address.
to
prime divisors less than 20 (Hanson, 1982). Since the remainder is going to
be the address of a record, we choose a number as close as possible to the
desired size of the address space. This number actually determines the size
of the address space. For a file with 75 records, a good choice might be 101,
which would leave the
the
file
74.3%
If 101 is the size of the address
example becomes
a
Hence,
the record
whose key
is
=
=
full
space, the
13820
= 0.743).
home address of the
(74/101
mod
record in
101
84.
LOWELL
is
assigned
record
to
number 84
in the
file.
that
The procedure
described previously can be carried out with a function
we
described mostly in pseudocode in Fig. 10.2. Procedure
call hash(),
hash() takes
at least
two
inputs:
returned by hash()
10.3
KEY, which must
12 characters, and
is
MAXAD,
the address.
Hashing Functions and Record Distributions
Of the two hash functions we have so far examined, one spreads out records
pretty well, and one does not spread
look
at
ways
distributions
be an array of ASCII codes for
which has the address size. The value
them out well
at all. In this section
makes
it
easier
we
Understanding
to discuss other hashing methods.
to describe distributions of records in
files.
454
HASHING
10.3.1 Distributing Records among Addresses
Figure 10.3 illustrates three different distributions of seven records
10 addresses. Ideally, a hash function should distribute records in a
no
by distribution
among
file
so
Such a distribution
is called uniform because the records are spread out uniformly among the
addresses. We pointed out earlier that completely uniform distributions are
so hard to find that it is generally not considered worth trying to find them.
Distribution (b) illustrates the worst possible kind of distribution. All
records share the same home address, resulting in the maximum number of
there are
collisions.
will be a
collisions, as illustrated
The more
a distribution
(a).
looks like this one, the
more
collisions
problem.
Distribution
somewhat spread
(c)
illustrates
out, but
with
a
a
distribution in
few
collisions.
which the records
is the most likely
This
are
case
we have a function that distributes keys randomly. If a hash function is
random, then for a given key every address has the same likelihood of being
if
chosen as every other address. The fact that a certain address is chosen for
one key neither diminishes nor increases the likelihood that the same
address will be chosen for another key.
It should be clear that if a random hash function is used to generate a
large number of addresses from a large number of keys, then simply by
chance some addresses are going to be generated more often than others. If
you have, for example, a random hash function that generates addresses
between
and 99, and you give the function 100 keys, you would expect
FIGURE 10.3 Different distributions,
(worst case), (c) A few synonyms.
(a)
Worst
Best
Record
Address
(a)
UAfuI
No synonyms
Record
Address
(b)
(uniform), (b) All
synonyms
Acceptable
Record
Address
(c)
455
HASHING FUNCTIONS AND RECORD DISTRIBUTIONS
some of
ideal,
it is
random
among
distribution of records
an acceptable alternative, given that
to find a function that gives a
may
to be
at all.
Although
not
more than once and some
the 100 addresses to be chosen
chosen not
it is
available addresses
is
practically impossible
uniform distribution. Uniform distributions
be out of the question,
when we
but there are times
distributions that are better than
random
in the sense that,
can find
while they do
fair number of synonyms, they spread out records among
more uniformly than does a random distribution.
generate a
addresses
10.3.2
It
Some
would be
Other Hashing Methods
nice
better-than-random
if
were
there
distribution
by
in
hash
function
cases,
all
but
that
there
hashing function depends on the
guaranteed
is
not.
The
of keys that
are actually hashed. Therefore, the choice of a proper hashing function
should involve some intelligent consideration of the keys to be hashed, and
perhaps some experimentation. The approaches to choosing a reasonable
hashing function covered in this section are ones that have been found to
work well, given the right circumstances. Further details on these and other
methods can be found in Knuth (1973b), Maurer (1975), Hanson (1982),
and Sorenson et al. (1978).
Here are some methods that are potentially better than random:
distribution generated
Examine keys for
rally
a pattern.
Sometimes keys
spread themselves out. This
is
more
fall
set
in patterns that natu-
likely to
be true of numeric
a set of employee identifinumbers might be ordered according to when the employees
entered an organization. This might even lead to no synonyms. If
some part of a key shows a usable underlying pattern, a hash func-
keys than of alphabetic keys. For example,
cation
tion that extracts that part of the
key can
also be used.
Fold parts of the key. Folding is one stage in the method discussed earlier. It involves extracting digits from part of a key and adding the
extracted parts together. This
terns but in
method destroys
some circumstances may
the original key pat-
preserve the separation between
certain subsets of keys that naturally spread themselves out.
Divide the key by a number. Division by the address size and use of
the remainder usually
is
involved somewhere in
hash function since
produce an address within a certain
range. Division preserves consecutive key sequences, so you can take
advantage of sequences that effectively spread out keys. However, if
there are several consecutive key sequences, division by a number
the purpose of the function
is
to
456
HASHING
many small factors can result in many collisions. Research
shown that numbers with no divisors less than 19 generally
avoid this problem. Division by a prime is even more likely than dithat has
has
vision
by
nonprime
to generate different results
from
different con-
secutive sequences.
The preceding methods are designed to take advantage of natural
among the keys. The next two methods should be tried when, for
some reason, the better-than-random methods do not work. In these cases,
orderings
randomization
the goal.
is
method
Square the key and take the middle. This popular
the mid-square method) involves treating the key as
(often called
a single large
number, squaring the number, and extracting whatever number of
digits is needed from the middle of the result. For example, suppose
you want to generate addresses between and 99. If the key is the
number 453, its square is 205,209. Extracting the middle two digits
yields a number between
and 99, in this case 52. As long as the
keys do not contain many leading or trailing zeros, this method usually
produces
method
fairly
that
is
it
random
results.
One
unattractive feature of this
often requires multiple precision arithmetic.
Radix transformation. This method involves converting the key to
some number base other than the one you are working in, and then
taking the result modulo the maximum address as the hash address.
and
For example, suppose you want to generate addresses between
key
99. If the
382; 382
mod
is
number
the decimal
99
85, so 85
is
453,
its
base 11 equivalent
is
the hash address.
Radix transformation is generally more reliable than the mid-square
for approaching true randomization, though mid-square has been
found to give good results when applied to some sets of keys.
method
10.3.3 Predicting the Distribution
Given
that
it
records
among
predict
how
that a large
is
of
Records
nearly impossible to achieve a uniform distribution of
the available addresses in a
file, it is
records are likely to be distributed. If
number of addresses
to
them than they can
of
collisions.
is
hold, then
likely to
we know
have
important to be able to
we know,
far
more
that there are
for example,
records assigned
going to be
a lot
457
HASHING FUNCTIONS AND RECORD DISTRIBUTIONS
Although there
among
collisions
are
no nice mathematical
distributions
are
that
tools available for predicting
than random,
better
there
are
mathematical tools for understanding just this kind of behavior when
records are distributed randomly. If we assume a random distribution
(knowing
that very likely
random), we can use these
our hashing method is likely
will be better than
it
tools to obtain conservative estimates of
how
to behave.
The Poisson
We
a
We want to
Distribution"''
that are likely to occur in a
predict the
begin by concentrating on what happens to
hash function
is
When
questions.
applied to a key.
of the keys
all
We
would
number of collisions
one record
that can hold only
file
a single
like to
in a file are hashed,
at
an address.
given address
when
answer the following
what
is
the likelihood
that
None
will hash to the given address?
Exactly one key will hash to the address?
Exactly two keys will hash to the address (two synonyms)?
Exactly three, four (and so on) keys will hash to the address?
All keys in the
file
will hash to the
same given address?
Which of these outcomes would you expect to be fairly likely, and
which quite unlikely? Suppose there are N addresses in a file. When a single
key is hashed, there are two possible outcomes with respect to the given
address:
A The
B
address
is
not chosen; or
The address
is
chosen.
How do we express
bothp(A) and a stand for
p(B) and
the probability that the address
stand for the probability that the address
P(B) =
since the address has
"''This
uted
one chance
in
p(A) = a
addresses in a
file if a
is
is
If
we
let
not chosen, and
chosen, then
= ^>
N of being
N- =
-^1
section develops a formula for predicting the
among
two outcomes?
the probabilities of the
random hashing
ways
chosen, and
in
which records
function
is
used.
The
will be distrib-
discussion assumes
knowledge of some elementary concepts of probability and combinatorics. You may want
to skip the development and go straight to the formula, which is introduced in the next
section.
458
HASHING
since the address has
(N =
chances in
N of not being
chosen. If there are
of our address being chosen
0.1, and the probability of the address not being chosen is a
10 addresses
10), the probability
1/10 =
0.1 = 0.9.
is
Now suppose two keys are hashed. What is the probability that both
keys hash to our given address? Since the two applications of the hashing
function are independent of one another, the probability that both will
produce the given address
p(BB) =
a product:
is
--
tor
N=
10: b
0.1
0.1
0.01.
Of course, other outcomes are possible when two keys are hashed. For
example, the second key could hash to an address other than the given
address.
The
p(BA) =
probability of this
In general,
is
- -M
when we want
the product
for
to
know
N=
10: b
xbxbxa
how
2 3
a b
0.9
0.09.
the probability of a certain sequence
of outcomes, such as BABBA, we can replace each
respectively, and compute the indicated product:
p(BABBA) = bx
0.1
for
N=
B by
and
2 3
10: a b
and
h,
(0.9) (0.1)
of three Bs and two
shown. We want to know the
probability that there are a certain number of Bs and As, but without regard
to order. For example, suppose we are hashing four keys and we want to
This example shows
As, where the Bs and
know how likely it is
to find the probability
As occur
in the order
that exactly
This can occur in six ways,
all
two of the keys hash to our given address.
ways having the same probability:
six
Outcome
Probability
BBAA
BABA
BAAB
ABBA
ABAB
AABB
bbaa
bV
baba = bV
baab = bV
abba = bV
abab = bV
aabb = bV
=
For
N=
10
(0.1) (0.9)
2
(0.1) (0.9)
2
(0.1) (0.9)
2
(0.1) (0.9)
2
(0.1) (0.9)
2
(0.1) (0.9)
=
=
=
=
=
0.0036
0.0036
0.0036
0.0036
0.0036
0.0036
Since these six sequences are independent of one another, the probability of two Bs and two As is the sum of the probabilities of the individual
outcomes:
459
HASHING FUNCTIONS AND RECORD DISTRIBUTIONS
p(BBAA) + p(BABA) +
+ p(AABB) =
2 2
6b a
x 0.0036 = 0.0216.
The 6 in the expression 6b a~ represents the number of ways two s and two
As can be distributed among four places.
In general, the event "r trials result in r x As and x Bs" can happen
in as many ways as r x letters A can be distributed among r places. The
probability of each such
way
is
and the number of such ways
is
given by the formula
(r
This
is
the
items out of
well-known formula
of
a set
items.
It
x)\x
number of ways of selecting x
when r keys are hashed, the
for the
follows that
probability that an address will be chosen x times and not chosen
x times
can be expressed as
p(x)
Furthermore,
we know
if
= Ca'~ x b x
that there are
N addresses
precise about the individual probabilities of
available, we can be
and B, and the formula
becomes
p(x)
where
= C
has the definition given previously.
What does
this
mean?
It
means
that
for example,
if,
compute the probability that a given address will have
by the hashing function using the formula
x =
0,
we
can
records assigned to
it
*) =
If
1,
this
c (' h\-
(h)-
formula gives the probability that one record will be assigned
to a given address:
p(\)
CI
This expression has the disadvantage that
(Try
it
for 1,000 addresses and 1,000 records:
for large values
of
and
r,
there
is
it
is
N=
awkward
to
compute.
1,000.) Fortunately,
function that
is
very good
460
HASHING
approximation for p(x) and
much
is
easier to
compute.
It
is
called the
Poisson function.
The Poisson Function Applied to Hashing
which we also denote by p(x), is given by
x
where N,
(r/N) e-
P(*)
The Poisson
function,
(r/x >
Tj
and p(x) have exactly the same meaning they have
previous section. That is, if
N=
r
x,
r,
the
number of available
number of records
the
addresses;
to be stored;
x = the number of records assigned
then p(x) gives the probability
to it after the
*s
in the
and
to a given address,
that a given address will
hashing function has been applied
to all
have had x records assigned
n records.
Suppose, for example, that there are 1,000 addresses (N = 1,000) and
whose keys are to be hashed to the addresses (r = 1,000).
1,000 records
Since r/N
to
it
(x
=
0)
1,
the probability that a given address will have no keys hashed
becomes
=1jr
P(0)
The
0-368.
probabilities that a given address will have exactly one, two, or
three keys, respectively, hashed to
it
are
= ]~
0.368
p(2)=-^j- =
0.184
p(\)
[r
I'e
p(3)
If
we
0.061.
can use the Poisson function to estimate the probability that
given address will have
predict the
number of records, we can also use it to
that will have a certain number of records
a certain
number of addresses
assigned.
For example, suppose there are 1,000 addresses (N = 1,000) and 1,000
(r = 1,000). Multiplying 1,000 by the probability that a given
address will have x records assigned to it gives the expected total number of
records
addresses with x records assigned to them. That
number of addresses with x
is,
records assigned to them.
\,000p(x) gives the
461
HASHING FUNCTIONS AND RECORD DISTRIBUTIONS
N addresses,
In general, if there are
then the expected
them
addresses with x records assigned to
number of
is
Np(x).
way of thinking about p(x). Rather than thinking
measure of probability, we can think of p(x) as giving the
proportion of addresses having x logical records assigned by hashing.
Now that we have a tool for predicting the expected proportion of
addresses that will have zero, one, two, etc. records assigned to them by a
This suggests another
about p(x)
as a
random hashing
function,
we
can apply
this tool to predicting
numbers of
collisions.
10.3.4 Predicting Collisions
for a Full File
Suppose you have a hashing function that you believe will distribute records
randomly, and you want to store 10,000 records in 10,000 addresses. How
many addresses do you expect to have no records assigned to them?
= 10,000, r/N = 1. Hence the proportion of
Since r = 10,000 and
addresses with
records assigned should be
= LIV =
1
P(0)
ir
The number of addresses with no
0-3679.
records assigned
10,000 x p (0)
How many addresses should have one,
is
3,679.
two, and three records assigned,
respectively?
10,000 x ^(l)
0.3679 x 10,000
3,679
10,000 x p (2)
0.1839 x 10,000
1,839
10,000 x p (3)
0.0613 x 10,000
613.
Since the 3,679 addresses corresponding to x
have exactly one
record assigned to them, their records have no synonyms.
two records
The 1,839
however, represent potential trouble. If
each such address has space only for one record, and two records are
assigned to them, there is a collision. This means that 1,839 records will fit
into the addresses, but another 1,839 will not fit. There will be 1,839
addresses with
apiece,
overflow records.
Each of the 613 addresses with three records apiece has an even bigger
problem. If each address has space for only one record, there will be two
overflow records per address. Corresponding to these addresses will be a
462
HASHING
of 2 X 613 = 1,226 overflow records. This is a bad situation. We have
thousands of records that do not fit into the addresses assigned by the
hashing function. We need to develop a method for handling these overflow
total
records.
10.4
But
first, let's
How Much
We
Extra
try to reduce the number of
overflow records.
Memory Should Be Used?
have seen the importance of choosing a good hashing algorithm to
A second way to decrease the number of collisions (and
reduce collisions.
thereby decrease the average search length)
to use extra
is
memory. The
developed in the previous section can be used to help us determine the
effect of the use of extra memory on performance.
tools
10.4.1 Packing Density
The term
packing density refers to the ratio of the
stored
to the
(r)
For example,
number of available
spaces (N):^
Number of records _
Number of spaces
if there are
the packing density
r_
75 records (n
75)
75o/o.
number of records
to be
and 100 addresses (N
100),
is
1=
0.75
The packing
density gives a measure of the amount of space in a file that
and it is the only such value needed to assess performance
in a hashing environment, assuming that the hash method used gives a
reasonably random distribution of records. The raw size of a file and its
address space do not matter; what is important is the relative sizes of the
two, which are given by the packing density.
Think of packing density in terms of tin cans lined up on a 10-foot
length of fence. If there are 10 tin cans and you throw a rock, there is a
certain likelihood that you will hit a can. If there are 20 cans on the same
length of fence, the fence has a higher packing density and your rock is
is
actually used,
more
""We
likely to hit a can.
assume here
that only
essarily the case, as
we
So
it is
with records
one record can be stored
see later.
at
in a
file.
The more
each address. In
fact, that is
records
not nec-
HOW MUCH EXTRA MEMORY SHOULD
there are packed into a given
will occur
the
when
new
file
record
is
likely
it is
that a collision
added.
We need to decide how much space we are willing to waste
number of collisions. The answer depends in large measure on
circumstances.
example,
at
We
want
to
have
need
few
as
the expense of requiring the
10.4.2 Predicting Collisions
We
more
space, the
463
BE USED?
to reduce
particular
collisions as possible, but not, for
file
to use
for Different
of the
two
disks instead of one.
Packing Densities
of changing the packing
number of collisions
that are likely to occur for a given packing density. Fortunately, the Poisson
function provides us with just the tool to do this.
You may have noted already that the formula for packing density (r/N)
occurs twice in the Poisson formula
a quantitative description
density. In particular,
we need
effects
to be able to predict the
P(x)
(r/N) e~
r/N
numbers of records (r) and addresses (N) always occur together
They never occur independently. An obvious implication of
this is that the way records are distributed depends partly on the ratio of the
number of records to the number of available addresses, and not on the
absolute numbers of records or addresses. The same behavior is exhibited
by 500 records distributed among 1,000 addresses as by 500,000 records
Indeed, the
as the ratio r/N.
distributed
among
1,000,000 addresses.
Suppose that 1,000 addresses are allocated to hold 500 records in a
randomly hashed file, and that each address can hold one record. The
packing density for the
file is
r_
= 500 =
1,000
Let us answer the following questions about the distribution of records
among
the available addresses in the
How many
How many
file:
addresses should have no records assigned to them?
addresses should have exactly one record assigned (no
synonyms)?
How many
onyms?
Assuming
addresses should have one record plus one or
more syn-
one record can be assigned to each home adoverflow records can be expected?
What percentage of records should be overflow records?
dress,
that only
how many
464
HASHING
1.
How many
addresses should have no records assigned to them? Since p(0)
gives the proportion of addresses with no records assigned, the
ber of such addresses
Np(0)
2.
How many
num-
is
1,000 x
=
-
607.
5 )" g
(-
1,000 x 0.607
addresses should have exactly one record assigned (no syn-
onyms)?
Np(\)
3.
How many
1,000 x
=
=
303.
^2L
1,000 x 0.303
addresses should have one record plus one or
more synonyms?
The
values o p(2), p(3), p(4), and so on give the proportions of addresses with one, two, three, and so on synonyms assigned to them.
Hence
the
sum
p(2)
gives the proportion of
all
may
appear to require
p(3)
addresses with at least one
a great deal
since the values of p(x)
p(4)
grow
synonym. This
of computation, but
it
doesn't
quite small for x larger than 3. This
should make intuitive sense. Since the
file is
50%
only
loaded, one
would not expect very many keys to hash to any one address.
Therefore, the number of addresses with more than about three keys
hashed to them should be quite small. We need only compute the results up to p(5) before they become insignificantly small:
p(2)
p(2>)
p(4)
p(5)
=
=
N and this
Assuming
many
0.0002
or
more synonyms
is
just the
result:
N\p{2)
4.
0.0016
0.0902.
The number of addresses with one
product of
+ 0.0126 +
0.0758
p{3)
=
-
1,000
x 0.0902
90.
that only one record can be assigned to each
home
address,
how
overflow records could be expected? For each of the addresses rep-
resented by p(2), one record can be stored at the address and one
must be an overflow record. For each address represented by p{2>),
one record can be stored
at
the address, two are overflow records,
HOW MUCH EXTRA MEMORY SHOULD
465
BE USED?
and so on. Hence, the expected number of overflow records
is
given by
1
N
=
=
5.
+ 2 X
x p (2)
NX
[1
1,000 x
N X p(3)
[1
+ 3 x iV x p(4) + 4 X
X p (5)
x p (3) + 3 x p (4) + 4 x p(5)]
x 0.0758 + 2 x 0.0126 + 3x 0.0016 + 4 x 0.0002]
p(2)
107.
Wliat percentage of records should be overflow records? If there are 107
overflow records and 500 records
flow records is
jjgConclusion:
only one record,
0.214
in
all,
then the proportion of over-
= 21.4%.
If the
packing density
we
can expect about
50% and each address can hold
21% of all records to be stored
is
somewhere other than at their home addresses.
Table 10.2 shows the proportion of records
home
if
that are not stored in their
addresses for several different packing densities.
the packing density
is
10%, then about
5%
The
of the time
table
we
shows
that
try to access
another record there. If the density is 100%, then
of all records collide with other records at their home addresses.
The 4.8% collision rate that results when the packing density is 10% looks
a record, there is already
about
TABLE 10.2
37%
Effect of packing density on the proportion of records not stored at their
home addresses
Packing
Density (%)
Synonyms
as
of Records
10
4.8
20
9.4
30
13.6
40
17.6
50
21.4
60
24.8
70
28.1
80
31.2
90
34.1
100
36.8
466
HASHING
very good until you realize that for every record
in
your
file
there will be
nine unused spaces!
The 36.8%
that results from 100% usage looks good when viewed in
unused space. Unfortunately, 36.8% doesn't tell the whole
story. If 36.8% of the records are not at their home addresses, then they are
somewhere else, probably in many cases using addresses that are home
addresses for other records. The more homeless records there are, the more
contention there is for space with other homeless records. After a while,
clusters of overflow records can form, leading in some cases to extremely
long searches for some of the records. Clearly, the placement of records that
collide is an important matter. Let us now look at one simple approach to
placing overflow records.
terms of
10.5
0%
Collision Resolution by Progressive Overflow
Even
if a
hashing algorithm
is
occur. Therefore, any hashing
very good,
is
likely that collisions will
program must incorporate some method
dealing with records that cannot
number of techniques
it
fit
into their
home
for
addresses. There are a
for handling overflow records, and the search for
FIGURE 10.4 Collision resolution with progressive overflow.
Novak
Rosen
York's
home
address (busy)
Jaspei
2nd
Moreley
-3rd try (busy)
try (busy)
-4th try (open)
York's actual
address
467
COLLISION RESOLUTION BY PROGRESSIVE OVERFLOW
Key
Blue
"1
98
Hash
Address
routine
"U
99
99
Jello
Wrapping around
FIGURE 10.5 Searching
for
an address beyond the end of
ever-better techniques continues to be
several approaches, but
works
and
well.
we
The technique
a file.
a lively area
concentrate on
of research.
We examine
very simple one that often
has various names, including progressive overflow
linear probing.
How
10.5.1
An example
Progressive Overflow Works
occurs is shown in Fig. 10.4.
whose key is York in the file.
Unfortunately, the name York hashes to the same address as the name
Rosen, whose record is already stored there. Since York cannot fit in its
of a situation
In the example,
home
address,
we want
it is
in
which
a collision
to store the record
an overflow record.
If
progressive overflow
used, the
is
next several addresses are searched in sequence until an empty one
The
first free
address 9
is
is
the
Eventually
6,
found.
record found empty, so the record pertaining to
first
stored in address
hashes to
is
address becomes the address of the record. In the example,
York
9.
we need
to find
York's record in the
file.
the search for the record begins at address
York's record there, so
it
proceeds to look
where it finds York.
An interesting problem occurs when
at
Since
6. It
York
still
does not find
successive records until
it
gets
to address 9,
or for
record
at the
end of the
file.
This
there
is
is
search for an open space
illustrated in Fig. 10.5, in
which
468
HASHING
assumed that the file can hold 100 records in addresses 0-99. Blue is
hashed to record number 99, which is already occupied by Jello. Since the
file holds only 100 records, it is not possible to use 100 as the next address.
The way this is handled in progressive overflow is to wrap around the
address space of the file by choosing address
as the next address. Since, in
this case, address
is not occupied, Blue gets stored in address 0.
What happens if there is a search for a record but the record was never
it is
placed in the
file?
The
and then proceeds
home
search begins, as before, at the record's
look for
to
it
in successive locations.
Two
address,
things can
happen:
If
an open address
sume
this
If the file
is it
means
is full,
is
encountered, the searching routine might as-
that the record
clear that the record
when we approach
is
filling
is
not in the
comes back
the search
not in the
our
file,
to
file.
When
searching can
slow, whether or not the record being sought
The
cases,
it
greatest strength of progressive overflow
is
a perfectly
adequate method. There
or
file;
where
is
it
began. Only then
this occurs,
become
in the
or even
intolerably
file.
is its
simplicity. In
are,
however,
many
collision-
handling techniques that perform better than progressive overflow, and
examine some of them later in this chapter.
progressive overflow on performance.
we
Now let us look at the effect of
10.5.2 Search Length
The reason
to avoid
overflow
disk accesses) have to occur
If there are a lot
of course, that extra searches (hence, extra
a record is not found in its home address.
there are going to be a lot of overflow records
is,
when
of collisions,
taking up spaces where they ought not to be. Clusters of records can form,
resulting in the placement of records a long
way from home,
so
many
disk
accesses are required to retrieve them.
Consider the following set of keys and the corresponding addresses
produced by some hash function.
Key
Home
Address
Adams
20
Bates
21
Cole
21
Dean
22
Evans
20
469
COLLISION RESOLUTION BY PROGRESSIVE OVERFLOW
Number
of
Actual
Home
accesses needed
address
address
to retrieve
20
Adams
20
21
Bates
21
22
Cole
21
23
Dean
22
24
Evans
20
25
FIGURE 10.6 Illustration of the effects of clustering of records. As
keys are clustered, the number of accesses required to access
later keys can become large.
If these records are
is
used to resolve
loaded into an empty
collisions,
file,
and progressive overflow
only two of the records will be
at their
addresses. All the others require extra accesses to retrieve.
shows where each key
is
stored, together with information
accesses are required to retrieve
The term
number of
from secondary memory.
long
way from its home
accesses required to
In the context of hashing, the
search length for a record increases every time there
is
on how many
it.
search length refers to the
retrieve a record
home
Figure 10.6
is
address, the search length
a collision. If a
may
record
be unacceptable.
good measure of the extent of the overflow problem is average search
length. The average search length is just the average number of times you
can expect to have to access the disk to retrieve a record. A rough estimate
of average search length may be computed by finding the total search length
(the sum of the search lengths of the individual records) and dividing this by
the number of records:
Average search length
total search length
total
number of records'
470
HASHING
In the
example, the average search length for the
1
1+2
five records
is
= ?2
With no
is 1, since only one
any record. (We indicated earlier that an
algorithm that distributes records so evenly that no collisions occur is
appropriately called a perfect hashing algorithm, and that, unfortunately,
such an algorithm is almost impossible to construct.) On the other hand, if
a large number of the records in a file results in collisions, the average search
length becomes quite long. There are ways to estimate the expected average
search length, given various file specifications, and we discuss them in a
access
is
collisions at
needed
to
all,
the average search length
retrieve
later section.
It
turns out that, using progressive overflow, the average search length
goes up very rapidly as the packing density increases. The curve in Fig.
10.7,
adapted from Peterson (1957), illustrates the problem. If the packing
is kept as low as 60%, the average record takes fewer than two tries
density
to access, but for a
it
much more
desirable packing density of
80%
or more,
increases very rapidly.
Average search lengths of greater than 2.0
unacceptable, so
it
appears that
of your storage space
it is
are generally considered
to get tolerable performance. Fortunately,
FIGURE 10.7 Average search length versus packing density
one record can be stored per address, progressive overflow
sions, and the file has just been loaded.
hashed file in which
used to resolve colli-
in a
is
Average
search
length
20
40
40%
we can
usually necessary to use less than
60
Packing density
80
100
471
STORING MORE THAN ONE RECORD PER ADDRESS: BUCKETS
improve on
by making one small change
to
our
hashing program. The change involves putting more than one record
at a
this situation substantially
single address.
10.6
Storing
More Than One Record per Address: Buckets
when
computer receives information from a disk, it is just
I/O system to transfer several records as it is to transfer
a single record. Recall too that sometimes it might be advantageous to think
of records as being grouped together in blocks rather than stored individuRecall that
about
Therefore,
ally.
as easy for the
why
not extend the idea of
The word
address of a group of records?
a
block of records that
bucket
record address in
is
sometimes used
retrieved in one disk access, especially
is
records are seen as sharing the same address.
On
a file to
an
to describe
when
those
sector-addressing disks, a
bucket typically consists of one or more sectors; on block-addressing disks,
a bucket might be a block.
Consider the following
set
of keys, which
is
to be loaded into a hash
file.
Home
Key
Address
Green
30
Hall
30
Jenks
32
King
33
Land
33
Marx
33
Nutt
33
Figure 10.8 illustrates part of
are loaded.
Each address
a file into
which the records with these keys
in the file identifies a
records corresponding to three synonyms.
to
Nutt cannot be accommodated
in a
bucket capable of holding the
Only
home
the record corresponding
address.
When a record is to be stored or retrieved, its home bucket address is
determined by hashing. The entire bucket is loaded into primary memory.
An in-RAM search through successive records in the bucket can then be
used to find the desired record. When a bucket is filled, we still have to
worry about the record overflow problem (as in the case of Nutt), but this
occurs
much
less
often
hold only one record.
when
buckets are used than
when
each address can
472
HASHING
Bucket
address
30
Bucket contents
Green
Hall
31
32
Jenks
33
King
(Nutt
.
Land
Marks
...
is
an overflow
.
record)
FIGURE 10.8 An illustration of buckets. Each bucket can hold up to three
records. Only one synonym (Nutt) results in overflow.
10.6.1 Effects of Buckets on Performance
When
buckets are used, the formula used to compute packing density
changed
slightly since each bucket address can hold
To compute how
densely packed a
file is,
more than one
we need
to consider
is
record.
both the
number or records we can put at each
number of addresses and b is the number
of records that fit in a bucket, then bN is the number of available locations
for records. If r is still the number of records in the file, then
number of addresses
address (bucket
(buckets) and the
size). If
the
is
Packing density
Suppose
we
the following
We
have
a file in
bN
which 750 records
two ways we might organize
can store the 750 data records
location can hold one record.
the
among
1,000 locations, where each
The packing
750
are to be stored. Consider
file.
density in this case
is
75%.
1,000
We
can store the 750 records
tion has a bucket size of 2.
among 500
There are
still
locations,
store the 750 records, so the packing density
bN
0.75
= 75%
where each
loca-
1,000 places (2 x 500) to
is still
473
STORING MORE THAN ONE RECORD PER ADDRESS: BUCKETS
we might at first not expect
improve performance, but in fact it does
improve performance dramatically. The key to the improvement is that,
Since the packing density
the use of buckets in this
way
not changed,
is
to
although there are fewer addresses, each individual address has more
for variation in the number of records assigned to it.
room
performance for these two ways of
in the same amount of space. The
the fundamental description of each file
Let's calculate the difference in
storing the
same number of records
starting point for our calculations
is
structure.
with
Buckets
without
Buckets
File
File
Number of records
Number of addresses
Bucket
= 750
N=
size
= 750
N = 500
b = 2
r
1,000
1
Packing density
0.75
0.75
Ratio of records to addresses
r/N = 0.75
r/N =
1.5
To
determine the number of overflow records that are expected in the
case of each file, recall that when a random hashing function is used, the
Poisson function
,
p(x)
(r/N) e-"
xi
gives the expected proportion of addresses assigned x records. Evaluating
the function for the
two
different
file
organizations,
assigned to addresses according to the distributions
We
see
from
the table that
when
make
many
22.3% of
intuitive sense
the addresses have
since
in the
addresses to choose from,
it
find that records are
shown
in
Table 10.3.
buckets are not used, 42.3% of the
addresses have no records assigned, whereas
used, only
we
when two-record
buckets are
no records assigned. This should
two-record case there are only half
as
stands to reason that a greater proportion
of the addresses are chosen to contain at least one record.
Note that the bucket column in Table 10.3 is longer than the nonbucket
column. Does this mean that there are more synonyms in the bucket case
than in the nonbucket case? Indeed it does, but half of those synonyms do
not result in overflow records because each bucket can hold two records.
Let us examine this further by computing the exact number of overflow
records likely to occur in the two cases.
474
HASHING
TABLE 10.3
p(x)
Poisson distributions for two different
File
without
Buckets
File with
Buckets
(r/N = 0.75)
(r/N
0.472
0.223
p(l)
0.354
0.335
p(2)
0.133
0.251
p(3)
0.033
0.126
P(4)
0.006
0.047
P(5)
0.001
0.014
0.001
P(7)
organizations
1.5)
p(0)
P(6)
file
0.004
In the case of the
file
with bucket
size one,
any address that
exactly one record does not have any overflow.
Any
is
more
number of
than one record does have overflow. Recall that the expected
overflow records
which, for r/N
1,000 x
is
given by
+2x
[1
x p (2)
0.75 and
[1
assigned
address with
p (3) + 3 x p (4) + 4 x p(S) +
N=
1,000,
is
approximately
x 0.1328 + 2 x 0.0332 + 3 x 0.0062 + 4 x 0.0009 +
5 x 0.0001] = 222.
The 222 overflow
records represent
29.6% overflow.
of the bucket file, any address that is assigned either one or
two records does not have overflow. The value of p(\) (with r/N =1.5)
gives the proportion of addresses assigned exactly one record, and p(2)
(with r/N = 1.5) gives the proportion of addresses assigned exactly two
records. It is not until we get to p(3) that we encounter addresses for which
there are overflow records. For each address represented by p(3), two
records can be stored at the address, and one must be an overflow record.
Similarly, for each address represented by p(4), there are two overflow
records, and so forth. Hence, the expected number of overflow records in
In the case
the bucket
file is
[1
x p (3) + 2 x p (4) + 3 x
p(S)
4 x p(6)
+ ...],
475
STORING MORE THAN ONE RECORD PER ADDRESS: BUCKETS
which
for
r/N =1.5 and
500 x
[1
= 500
is
approximately
x 0.1255 + 2 x 0.0471 + 3 x 0.0141 + 4 x 0.0035 +
5 x 0.0008] = 140.
The 140 overflow records represent 18.7% overflow.
We have shown that with one record per address and a packing density
of 75%, the expected number of overflow records is 29.6%. When 500
buckets are used, each capable of holding two records, the packing density
remains 75%, but the expected number of overflow records drops to
18.7%. That is about a 37% decrease in the number of times the program
is going to have to look elsewhere for a record. As the bucket size gets
performance continues to improve.
Table 10.4 shows the proportions of collisions that occur for different
packing densities and for different bucket sizes. We see from the table, for
example, that if we keep the packing density at 75% and increase the bucket
size to 10, record accesses result in overflow only 4% of the time.
It should be clear that the use of buckets can improve hashing
performance substantially. One might ask, "How big should buckets be?"
Unfortunately, there is no simple answer to this question because it depends
very much on a number of different characteristics of the system, including
larger,
TABLE 10.4
Synonyms causing
densities
collisions as a percent of records
nd different bucket sizes
for different
packing
Bucket Size
Packing
Density
<%)
10
100
10
4.8
0.6
0.0
0.0
0.0
20
9.4
2.2
0.1
0.0
0.0
30
13.6
4.5
0.4
0.0
0.0
40
17.6
7.3
1.1
0.1
0.0
50
21.3
10.4
2.5
0.4
0.0
60
24.8
13.7
4.5
1.3
0.0
70
28.1
17.0
7.1
2.9
0.0
75
29.6
18.7
8.6
4.0
0.0
80
31.2
20.4
10.3
5.3
0.1
90
34.1
23.8
13.8
8.6
0.8
100
36.8
27.1
17.6
12.5
4.0
476
HASHING
the sizes of buffers the operating system can manage, sector and track
on
capacities
and access times of the hardware
disks,
(seek, rotation,
and
data transfer times).
As
good idea to use buckets larger than a
Even a track, however, can sometimes
when one considers the amount of time it takes to transmit an
as compared to the amount of time it takes to transmit a few
a rule, it is
probably not
track (unless records are very large).
be too large
entire track,
sectors. Since hashing
almost always involves retrieving only one record
per search, any extra transmission time resulting from the use of extra large
buckets
essentially wasted.
is
many
is the best bucket size. For example,
with 200-byte records is to be stored on a disk system that
uses 1,024-byte clusters. One could consider each cluster as a bucket, store
five records per cluster, and let the remaining 24 bytes go unused. Since it
is no more expensive, in terms of seek time, to access a five-record cluster
than it is to access a single record, the only losses from the use of buckets
are the extra transmission time and the 24 unused bytes.
The obvious question now is, "How do improvements in the number
of collisions affect the average search time?" The answer depends in large
measure on characteristics of the drive on which the file is loaded. If there
are a large number of tracks in each cylinder, there will be very little seek
time because overflow records will be unlikely to spill over from one
cylinder to another. If, on the other hand, there is only one track per
cylinder, seek time could be a major consumer of search time.
In
suppose that
A less
is
cases a single cluster
a file
exact measure of the
average search length, which
amount of time required
we
introduced
to retrieve a record
earlier. In the case
of buckets,
average search length represents the average number of buckets that must
Table 10.5 shows the expected average
with different packing densities and bucket sizes,
given that progressive overflow is used to handle collisions. Clearly, the use
of buckets seems to help a great deal in decreasing the average search length.
be accessed to retrieve
search lengths for
The bigger
a record.
files
the bucket, the shorter the search length.
10.6.2 Implementation Issues
of this text, we paid quite a bit of attention to issues
producing, using, and maintaining random-access files with
fixed-length records that are accessed by relative record number (RRN).
In the early chapters
involved
in
Since a hashed
RRN, you
Hashed
respects,
file is a
fixed-length record
file
whose records
are accessed
by
know much about implementing hashed files.
from the files we discussed earlier in two important
should already
files
differ
however:
477
STORING MORE THAN ONE RECORD PER ADDRESS: BUCKETS
TABLE 10.5
Average number of accesses required
in
a successful search by
progressive overflow
Bucket Sizes
Packing
Density
(%)
10
50
10
1.06
1.01
1.00
1.00
1.00
30
1.21
1.06
1.00
1.00
1.00
40
1.33
1.10
1.01
1.00
1.00
50
1.50
1.18
1.03
1.00
1.00
60
1.75
1.29
1.07
1.01
1.00
70
2.17
1.49
1.14
1.04
1.00
80
3.00
1.90
1.29
1.11
1.01
90
5.50
3.15
1.78
1.35
1.04
95
10.50
5.6
2.7
1.8
1.1
Adapted from Donald Knuth, The Art of Computer Programming, Vol.
Mass. Page 536. Reprinted with permission.
5 1973,
3,
Addison-Wesley, Read-
ing,
1.
Since a hash function depends on there being a fixed
available addresses, the logical size of a hashed
fore the
as
long
size to
file
file
can be populated with records, and
it
number of
must be fixed bemust remain fixed
same hash function is used. (We use the phrase logical
leave open the possibility that physical space be allocated as
as the
needed.)
2.
Since the
to
its
home
RRN
of
record in
hashed
file is
uniquely related
key, any procedures that add, delete, or change a record
do so without breaking the bond between
dress. If this
bond
is
broken, the record
is
record and
no longer
its
must
home
accessible
ad-
by
hashing.
We
must keep these
work with hashed
Bucket Structure
special needs in
mind when we write programs
to
files.
The only difference between a file with buckets and
which each address can hold only one key is that with a bucket file
each address has enough space to hold more than one logical record. All
records that are housed in the same bucket share the same address. Suppose,
one
in
478
HASHING
for example, that
Here
An empty
Two
we want
many
to store as
are three such buckets with different
bucket:
entries:
full bucket:
as five names in one bucket.
numbers of records.
JONES
ARNSWORTH
JONES
ARNSWORTH STOCKTON BRICE
Each bucket contains
has stored in
it.
a counter that
keeps track of how
Collisions can occur only
causes the counter to exceed the
The counter
tells
us
when
many records it
new record
the addition of a
number of records
how many
THROOP
bucket can hold.
data records are stored in a bucket, but
tell us which slots are used and which are not. We need a way
whether or not a record slot is empty. One simple way to do this is
to use a special marker to indicate an empty record, just as we did with
deleted records earlier.. We use the key value ///// to mark empty records in
does not
it
to
tell
the preceding illustration.
Initializing a File for
remain
fixed,
it
Hashing
makes sense
in
Since the
most
logical size
of a hashed
file
must
cases to allocate physical space for the
we
begin storing data records in it. This is generally done by
of empty spaces for all records, and then filling the slots as
they are needed with the data records. (It is not necessary to construct a file
of empty records before putting data in it, but doing so increases the
file
before
creating a
file
likelihood that records will be stored close to one another on the disk,
avoids the error that occurs
when
an attempt
is
made
to read a missing
makes it easy to process the file sequentially, without having
empty records in any special way.)
record, and
treat the
to
Loading a Hash File A program that loads a hash file is similar in many
ways to earlier programs we use for populating fixed-length record files,
with two differences. First, the program uses the function hash() to produce
a
home
address for each key. Second, the program looks for
the record
by
starting with the bucket stored at
its
home
a free
space for
address and then,
479
MAKING DELETIONS
home bucket is full, continuing to look at successive buckets until one
found that is not full. The new record is inserted in this bucket, which is
rewritten to the file at the location from which it is loaded.
If, as it searches for an empty bucket, a loading program passes the
maximum allowable address, it must wrap around to the beginning
address. A potential problem occurs in loading a hash file when so many
records have been loaded into the file that there are no empty spaces left.
A naive search for an open slot can easily result in an infinite loop. Obviously, we want to prevent this from occurring by having the program make
if
the
is
sure that there
space available for each
is
new
somewhere
record
in the
file.
Another problem
when an attempt
that often arises
made
when adding
records to
files
occurs
add a record that is already stored in the file. If
there is a danger of duplicate keys occurring, and duplicate keys are not
allowed in the file, some mechanism must be found for dealing with this
problem.
10.7
is
to
Making Deletions
Deleting a record from
record for
The
two
slot freed
searches;
It
hashed
file is
more complicated than adding
by the deletion must not be allowed
to hinder later
and
should be possible to reuse the freed
When
reasons:
progressive overflow
is
slot for later additions.
used, a search for a record terminates if
this, we do not want to leave
open addresses that break overflow searches improperly. The following
example illustrates the problem.
Adams, Jones, Morris, and Smith are stored in a hash file in which each
address can hold one record. Adams and Smith both are hashed to address
5, and Jones and Morris are hashed to address 6. If they are loaded in
an open address
is
encountered. Because of
alphabetical order using progressive overflow for collisions, they are stored
in the locations
shown
in Fig. 10.9.
Smith starts at address 5 (Smith's home address).
successively looks for Smith at addresses 6, 7, and 8, then finds Smith at 8.
Now suppose Morris is deleted, leaving an empty space, as illustrated in
Fig. 10.10. A search for Smith again starts at address 5, and then looks at
addresses 6 and 7. Since address 7 is now empty, it is reasonable for the
search
program
for
to conclude that Smith's record
is
not in the
file.
480
HASHING
Home
Actual
Record
address
address
Adams
Jones
Adams
Morris
Jones
Smith
Morris
Smith
FIGURE 10.9
File organization before deletions.
10.7.1 Tombstones
Chapter 5
In
One
we
for
Handling Deletions
discussed techniques for dealing with the deletion problem.
simple technique
we
use for identifying deleted records involves
replacing the deleted record (or just
its
key) with
marker indicating that a
is sometimes
record once lived there but no longer does. Such a marker
FIGURE 10.10 The same organization as in Fig. 10.9, with
Morris deleted.
Adams
Jones
Smith
481
MAKING DELETIONS
Adams
Jones
######
Smith
FIGURE 10.11 The same file as
Fig. 10.9 after the insertion of
tombstone
The
is
that
it
for Morris.
referred to as a tombstone (Wiederhold, 1983).
of tombstones
in
The
nice thing about the use
solves both of the problems described previously:
freed space does not break
sequence of searches for
a record;
and
The
freed space
is
obviously available and
may
be reclaimed for
later
additions.
Figure 10.11 illustrates how the sample file might look aftc the
tombstone ###### is inserted for the deleted record. Now a search for
Smith does not halt at the empty record number 7. Instead, it uses the
######
It is
as
an indication that
it
should continue the search.
not necessary to insert tombstones every time
a deletion occurs.
example, suppose in the preceding example that the record for Smith
following the Smith record
For
is
to
empty, nothing is
lost by marking Smith's slot as empty rather than inserting a tombstone.
Indeed, it is actually unwise to insert a tombstone where it is not needed. (If,
after putting an unnecessary tombstone in Smith's slot, a new record is
added at address 9, how would a subsequent unsuccessful search for Smith
be affected?)
be deleted. Since the
slot
10.7.2 Implications
of
Tombstones
is
for Insertions
With the introduction of the use of tombstones, the insertion of records
becomes slightly more difficult than our earlier discussions imply. Whereas
programs that perform initial loading simply search for the first occurrence
of an empty record slot (signified by the presence of the key /////), it is now
482
HASHING
permissible to insert
record where either ///// or
######
which
yields a shorter average
occurs as the
key.
This
new
feature,
search length, brings with
earlier
example
shown
in Fig. 10.
in
1 1
it
is
desirable because
a certain
which Morris
.
is
it
danger. Consider, for example, the
deleted,
giving the
file
organization
Now suppose you want a program to insert Smith into
the file. If the program simply searches until it encounters a ######, it
never notices that Smith is already in the file. We almost certainly don't
want to put a second Smith record into the file, since doing so means that
later searches would never find the older Smith record. To prevent this
from occurring, the program must examine the entire cluster of contiguous
keys and tombstones to ensure that no duplicate key exists, and then go
back and insert the record in the first available tombstone, if there is one.
10.7.3 Effects
of Deletions
The use of tombstones
enables our search algorithms to
storage recovery, but one can
after a
and Additions on Performance
still
number of deletions and
expect
some
work and
additions occur within a
Consider, for example, our
little
helps in
deterioration in performance
four-record
file
file.
of Adams, Jones,
one slot further from its
home address than it needs to be. If the tombstone is never to be used to
store another record, every retrieval of Smith requires one more access than
is absolutely necessary. More generally, after a large number of additions
and deletions, one can expect to find many tombstones occupying places
that could be occupied by records whose home records precede them but
that are stored after them. In effect, each tombstone represents an
unexploited opportunity to reduce by one the number of locations that
must be scanned while searching for these records.
Some experimental studies show that after a 50% to 150% turnover of
records, a hashed file reaches a point of equilibrium, so average search
length is as likely to get better as it is to get worse (Bradley, 1982; Peterson,
1957). By this time, however, search performance has deteriorated to the
point that the average record is three times as far (in terms of accesses) from
its home address as it would be after initial loading. This means, for
Smith, and Morris. After deleting Morris, Smith
example, that
if after original
is
loading the average search length
be about 1.6 after the point of equilibrium
is
is
1.2,
it
will
reached.
There are three types of solutions to the problem of deteriorating
One involves doing a bit of local reorganizing every
time a deletion occurs. For example, the deletion algorithm might examine
average search lengths.
483
OTHER COLLISION RESOLUTION TECHNIQUES
the records that follow a tombstone to see if the search length can be
shortened by moving the record backward toward
Another solution involves completely reorganizing the
search length reaches an unacceptable value.
its
file
home
address.
after the
average
third type of solution
involves using an altogether different collision resolution algorithm.
10.8
Other Collision Resolution Techniques
randomized hashing using progressive overflow with
reasonably sized buckets generally performs well. If it does not perform
wxll enough, however, there are a number of variations that may perform
even better. In this section we discuss some refinements that can often
improve hashing performance when using external storage.
Despite
its
simplicity,
10.8.1 Double Hashing
One of the problems
with progressive overflow is that if many records hash
vicinity, clusters of records can form. As the packing
density approaches one, this clustering tends to lead to extremely long
to buckets in the
searches for
same
some
records.
One method
for avoiding clustering
is
to store
overflow records a long way from their home addresses by double hashing.
With double hashing, when a collision occurs, a second hash function is
applied to the key to produce a number c that is relatively prime to the
number of addresses.^ The value c is added to the home address to produce
the overflow address. If the overflow address is already occupied, c is added
to it to produce another overflow address. This procedure continues until a
free overflow address is found.
Double hashing does tend to spread out the records in a file, but it
from a potential problem that is encountered in several improved
overflow methods: It violates locality by deliberately moving overflow
records some distance from their home addresses, increasing the likelihood
suffers
that the disk will
need extra time to get to the
new overflow
address. If the
more than one cylinder, this could require an expensive extra
head movement. Double hashing programs can solve this problem if they
are able to generate overflow addresses in such a way that overflow records
are kept on the same cylinder as home records.
file
""If
covers
is
divisors.
the
number of addresses,
then
and
N are relatively
prime
if
they have no
common
484
HASHING
10.8.2 Chained Progressive Overflow
Chained progressive overflow
problems caused by clustering.
is
another technique designed to avoid the
works
in the same manner as progressive
synonyms are linked together with pointers. That is,
each home address contains a number indicating the location of the next
record with the same home address. The next record in turn contains a
It
overflow', except that
home
pointer to the following record with the same
The
net effect of this
that for each set of
is
connecting their records, and
it is
address, and so forth.
synonyms
this list that
is
there
searched
is
linked
when
list
record
is
sought.
The advantage of chained progressive overflow over simple progressive
overflow is that only records with keys that are synonyms need to be
accessed in any given search. Suppose, for example, that the set of keys
shown in Fig. 10.12 is to be loaded in the order shown into a hash file with
bucket size one, and progressive overflow is used. A search for Cole
involves an access to Adams (a synonym) and Bates (not a synonym). Flint,
the worst case, requires six accesses, only two of which involve synonyms.
Since Adams, Cole, and Flint are synonyms, a chaining algorithm
forms a 'inked list connecting these three names, with Adams at the head of
the list. Since Bates and Dean are also synonyms, they form a second list.
This arrangement
from
decreases
is
10.13.
illustrated in Fig.
The average
search length
2.5 to
1
1+2
1+3
The use of chained
some details that are not
link field
must be added
progressive overflow requires that
we
attend to
required for simple progressive overflow.
to each record, requiring the use
of
First, a
a little
more
FIGURE 10.12 Hashing with progressive overflow.
Home
Search
Key
address
Actual
address
length
Adams
20
20
Bates
21
21
Cole
20
22
Dean
21
23
Evans
24
20
Average search le ngth =
Flint
(1
24
25
3+1
6)/6
==
2.5
485
OTHER COLLISION RESOLUTION TECHNIQUES
Home
Actual
address
address
20
20
Adams
21
21
Bates
20
22
Cole
21
23
Dean
24
24
Evans
20
25
Flint
Address of
Data
synonym
Search
length
next
FIGURE 10.13 Hashing with chained progressive overflow. Adams, Cole, and
Flint are synonyms; Bates and Dean are synonyms.
must guarantee
storage. Second, a chaining algorithm
that
it is
possible to
any synonym by starting at its home address. This second
requirement is not a trivial one, as the following example shows.
Suppose that in the example Dean's home address is 22 instead of 21.
Since, by the time Dean is loaded, address 22 is already occupied by Cole,
get
to
Dean
still
ends up
address 23.
at
Does
this
mean
that Cole's pointer should
point to 23 (Dean's actual address) or to 25 (the address of Cole's
Flint)? If the pointer
kept intact, but
is
Dean
The problem here
by
home
problem
for
some record
is
handled easily
are not
is
list
joining
If the pointer
is
synonym
Adams, Cole, and
23, Flint
Flint
is
is lost.
that a certain address (22) that should be occupied
is
occupied by
a different record.
One solution to
home address
to require that every address that qualifies as a
in the file actually
when
two-pass loading.
Two-pass loading,
passes.
is lost.
record (Dean)
the
two
25, the linked
file
as the
On the first pass,
is
hold
first
home
record.
loaded by using
The problem can be
a
technique called
name implies, involves loading a hash file in
only home records are loaded. All records that
home records are kept in a separate file. This guarantees that no
home addresses are occupied by overflow records. On the second
potential
overflow record is loaded and stored in one of the free addresses
according to whatever collision resolution technique is being used.
pass, each
486
HASHING
Two-pass loading guarantees
is
home
address, so
it
home
that every potential
problem
solves the
address actually
example.
in the
It
does not
and additions will not re-create the same
problem, however. As long as the file is used to store both home records
and overflow records, there remains the problem of overflow records
displacing new records that hash to an address occupied by an overflow
guarantee that
later deletions
record.
The methods used for handling these problems after initial loading are
somewhat complicated and can, in a very volatile file, require many extra
disk accesses. (For more information on techniques for maintaining
pointers, see Knuth, 1973b and Bradley, 1982.) It would be nice if we could
somehow altogether avoid this problem of overflow lists bumping into one
another, and that is what the next method does.
10.8.3 Chaining with a Separate Overflow Area
One way
to
keep overflow records from occupying
they should not be
to
is
move them
hashing schemes are variations of
addresses
is
all
this basic
and the
called the prime data area,
called the overflow area.
home
to a separate
addresses where
overflow
approach. The
set
area.
set
of
Many
home
of overflow addresses is
is that it keeps all
The advantage of this approach
unused but potential home addresses free for later additions.
In terms of the file we examined in the preceding section, the records
for Cole, Dean, and Flint could have been stored in a separate overflow area
rather than potential
home
addresses for later-arriving records (Fig. 10.14).
Now
no problem occurs when a new record is added. If its home address
has room, it is stored there. If not, it is moved to the overflow file, where
it is added to the linked list that starts at the home address.
If the bucket size for the primary file is large enough to prevent
excessive numbers of overflow records, the overflow file can be a simple
file with a bucket size of one. Space can be allocated for
overflow records only when it is needed.
The use of a separate overflow area simplifies processing somewhat and
would seem to improve performance, especially when many additions and
deletions occur. However, this is not always the case. If the separate
entry-sequenced
overflow area
is
on
a different
cylinder than
is
the
home
address, every
search for an overflow record will involve a very costly head
Studies
show
that actual access time
records are stored in
a separate
is
generally worse
movement.
when overflow
overflow area than when they are stored
in
the prime overflow area (Lum, 1971).
One
situation in
the packing density
which
is
a separate
overflow area
greater than one
there
are
is
required occurs
more
records than
when
home
487
OTHER COLLISION RESOLUTION TECHNIQUES
Home
Primary
address
data area
20
Adams
21
Bates
Overflow
area
Cole
Dean
22
Flint
23
24
-*
.
Evans
FIGURE 10.14 Chaining to a separate overflow area. Adams, Cole, and
synonyms; Bates and Dean are synonyms.
addresses.
If,
for example,
it is
anticipated that a
file
Flint are
will
grow beyond
capacity of the initial set of home addresses and that rehashing the
a larger
address space
is
not reasonable, then
separate overflow area
the
with
file
must
be used.
10.8.4 Scatter Tables: Indexing Revisited
Suppose you have
no records, only pointers to
records. The file is obviously just an index that is searched by hashing rather
than by some other method. The term scatter table (Severance, 1974) is often
a
hash
file
applied to this approach to
organization of
a file
using
that contains
file
organization. Figure 10.15 illustrates the
a scatter table.
The scatter table organization provides many of the same advantages
simple indexing generally provides, with the additional advantage that the
search of the index itself requires only one access. (Of course, that one
one more than other forms of hashing require, unless the scatter
table can be kept in primary memory.) The data file can be implemented in
many different ways. For example, it can be a set of linked lists of
access
is
synonyms
(as
shown
in Fig. 10.15), a sorted
file,
or an entry-sequenced
file.
Also, scatter table organizations conveniently support the use of variable-
length records. For
more information on
and Teorey and Fry (1982).
scatter tables, see Severance (1974)
488
HASHING
k.
20
21
1*
Adams
Bates
Cole
b Dean
Flint
??
23
24
Evans
25
FIGURE 10.15 Example of a scatter table structure. Because the hashed part
file may be organized in any way that is appropriate.
is
an index,
the data
10.9
Patterns of Record Access
Twenty
Twenty
L.
percent of the fishermen catch 80 percent of the
fish.
percent of the burglars steal 80 percent of the loot.
M. Boyd
The use of different
necessarily the best
collision resolution techniques
way of improving performance
is
in a
not the only nor
hashed
file.
If
we
know something
about the patterns of record access, for example, then it is
often possible to use simple progressive overflow techniques and still
achieve very good performance.
Suppose you have a grocery store with 10,000 different categories of
grocery items, and you have on your computer a hashed inventory file with
a
record for each of the 10,000 items that your
time an item
accessed. Since the
file is
hashed,
records are distributed randomly
up the
company
handles. Every
purchased, the record that corresponds to that item must be
is
file. Is it
it is
reasonable to assume that the 10,000
among
the available addresses that
make
equally reasonable to assume that the distribution of accesses
to the records in the inventory are
randomly
distributed? Probably not.
Milk, for example, wall be retrieved very frequently, brie seldom.
There
is
a principle
The Concept of
used by economists called the Pareto Principle, or
Few and the Trivial Many, which in file terms
the Vital
says that a small percentage of the records in a
percentage of the accesses.
80/20 Rule of
file
account for
a large
popular version of the Pareto Principle
Thumb: 80% of
the accesses are performed
on 20%
is
the
of the
SUMMARY
records. In our groceries
items, brie
among
file,
among
milk would be
the
20%
489
high-activity
the rest.
We cannot take advantage of the 80/20 principle in a file structure unless
we know something about the probable distribution of record accesses.
Once we have this information, we need to find a way to place the
high-activity items
possible.
way
If,
when
that the
loaded
at
where they can be found with
items are loaded into
20% (more or
home
or near their
less)
a file,
that are
as
few accesses
as
they can be loaded in such a
most likely to be accessed are
most of the transactions will
addresses, then
access records that have short search lengths, so the effective average search
length will be shorter than the nominal average search length that
defined
we
earlier.
For example, suppose our grocery store's file handling program keeps
number of times each item is accessed during a one-month
period. It might do this by storing with each record a counter that starts at
zero and is incremented every time the item is accessed. At the end of the
month the records for all the items in the inventory are dumped onto a file
that is sorted in descending order according to the number of times they
have been accessed. When the sorted file is rehashed and reloaded, the first
records to be loaded are the ones that, according to the previous month's
experience, are most likely to be accessed. Since they are the first ones
loaded, they are also the ones most likely to be loaded into their home
addresses. If reasonably sized buckets are used, there will be very few, if any,
high-activity items that are not in their home addresses and therefore
track of the
retrievable in
one
access.
SUMMARY
There are three major modes for accessing files: sequentially, which provides
O(N) performance, through tree structures, which can produce 0(\og k N)
performance, and directly. Direct access provides O(l) performance, which
means that the number of accesses required to retrieve a record is constant
and independent of the size of the file. Hashing is the primary form of
organization used to provide direct access.
Hashing can provide
we
faster access than
study, usually with very
little
most of the other organizations
storage overhead, and
most types of primary keys. Ideally, hashing makes
record with only one disk access, but this ideal
primary disadvantage of hashing
by key.
is
that
hashed
is
it
it is
adaptable to
possible to find any
rarely achieved.
files
may
The
not be sorted
490
HASHING
Hashing involves the application of a hash function h(K) to a record key
The address is taken to be the home address of the
record whose key is K, and it forms the basis for searching for the record.
The addresses produced by hash functions generally appear to be random.
When two or more keys hash to the same address, they are called
synonyms. If an address cannot accommodate all of its synonyms, collisions
result. When collisions occur, some of the synonyms cannot be stored in the
home address and must be stored elsewhere. Since searches for records
begin with home addresses, searches for records that are not stored at their
home addresses generally involve extra disk accesses. The term average
search length is used to describe the average number of disk accesses that are
required to retrieve a record. An average search length of 1 is ideal.
Much of the study of hashing deals with techniques for decreasing the
number and effects of collisions. In this chapter we look at three general
approaches to reducing the number of collisions:
K to produce an address.
Spreading out the records;
Using extra memory; and
Using buckets.
Spreading out the records involves choosing a hashing function that
distributes the records at least
randomly over the address
space.
distribution
spreads out records evenly, resulting in no collisions.
or nearly
random
distribution
is
much
easier to achieve
and
A uniform
A random
is
usually
considered acceptable.
In this chapter a simple hashing algorithm
is
developed to demonstrate
the kinds of operations that take place in a hashing algorithm.
The
three
steps in the algorithm are:
1.
Represent the key in numerical form;
2.
Fold and add; and
3.
Divide by the
When we examine
size
of the address space, producing
a valid address.
several different types of hashing algorithms,
sometimes algorithms can be found
we
see that
produce better-than-random distributions. Failing this, we suggest some algorithms that generally produce
distributions which are approximately random.
The Poisson distribution provides a mathematical tool for examining in
detail the effects of a random distribution. Poisson functions can be used to
predict the numbers of addresses likely to be assigned 0, 1, 2, and so on,
records, given the number of records to be hashed and the number of
available addresses. This allows us to predict the number of collisions likely
to occur when a file is hashed, the number of overflow records likely to
that
occur, and sometimes the average search length.
SUMMARY
Using extra memory is another way to avoid collisions. When a fixed
number of keys is hashed, the likelihood of synonyms occurring decreases
as the number of possible addresses increases. Hence, a file organization that
allocates many more addresses than are likely to be used has fewer
synonyms than one that allocates few extra addresses. The term packing
density describes the
proportion of available address space that actually holds
function is used to determine how differences in
The Poisson
records.
packing density influence the percentage of records that are likely to be
synonyms.
is the third method for avoiding collisions. File addresses
more records, depending on how the file is organized by the
The number of records that can be stored at a given address,
Using buckets
can hold one or
file
designer.
called bucket size, determines the point at
address will overflow.
effects
The Poisson
which records assigned
to the
function can be used to explore the
of variations in bucket sizes and packing densities. Large buckets,
a low packing density, can result in very small average
combined with
search lengths.
Although we can reduce the number of collisions, we need some means
with collisions when they do occur. We examined one simple
to deal
collision resolution technique in detail
store a
new
record results in
progressive overflow. If an attempt to
a collision,
progressive overflow involves
searching through the addresses that follow the record's
order until one
not found in
its
is
found to hold the new record.
home
If a
home
record
is
address in
sought and
is
address, successive addresses are searched until either
found or an empty address is encountered.
Progressive overflow is simple and sometimes works very well.
Progressive overflow creates long search lengths, however, when the
packing density is high and the bucket size is low. It also sometimes
produces clusters of records, creating very long search lengths for new
the record
records
is
whose home
addresses are in the clusters.
Three problems associated with record deletion
1.
The
possibility that
empty
slots created
by
in
hashed
files
are
deletions will hinder later
searches for overflow records;
2.
The need
leted;
3.
to
recover space
made
available
when
records are de-
and
The deterioration of average search lengths caused by empty spaces
keeping records further from home than they need be.
The
that are
first two problems can be solved by using tombstones to mark spaces
empty (and can be reused for new records) but should not halt a
search for a record. Solutions to the deterioration problem include local
491
492
HASHING
reorganization, complete
file
reorganization, and the choice of a collision-
resolving algorithm that does not cause deterioration to occur.
Because overflow records have a major influence on performance,
different overflow handling techniques have been proposed. Four
such techniques that are appropriate for file applications are discussed
many
briefly:
1.
Double hashing reduces local clustering but
records so far from
2.
home
may
place
some overflow
that they require extra seeks.
Chained progressive overflow reduces search lengths by requiring that
when a record is being sought. For
only synonyms be examined
chained overflow to work, every address that qualifies
record for
nisms for
3.
some record in the file must hold a home
making sure that this occurs are discussed.
home
Mecha-
Chaining with a separate overflow area simplifies chaining substantially
and has the advantage
that the
ways more appropriate
overflow area may be organized in
overflow records. A danger of
to handling
Scatter tables
is that it might lose locality.
combine indexing with hashing. This approach provides
much more
flexibility in
this
4.
as a
record.
approach
using scatter tables
is
organizing the data
that, unless the
requires
one extra disk access for every
Since in
many
accessed,
home
we
than
we
it is
disadvantage of
RAM,
it
more
frequently than
often worthwhile to take access patterns
can identify those records that are most likely to be
can take measures to
less
search.
cases certain records are accessed
others (the 80/20 rule of thumb),
into account. If
file.
index can be held in
make
sure that they are stored closer to
frequently accessed records, thus decreasing the
average search length.
One
such measure
is
to load the
effective
most frequently
accessed records before the others.
KEY TERMS
Average search length.
We
define average search length as the sum of
number of accesses required for each record in the file divided by the
number of records in the file. This definition does not take into account
the number of accesses required for unsuccessful searches, nor does it
the
account for the
fact that
some
records are likely to be accessed
more
often than others. See 80/20 rule of thumb.
Better-than-random. This term is applied to distributions in which the
records are spread out
more uniformly than they would be
if
the
KEY TERMS
hash function distributed them randomly. Normally, the distribution
produced by a hash function is a little bit better than random.
Bucket.
An
area of space
and
for storage
on the
retrieval
By
eral logical records.
file
that
is
treated as a physical record
purposes but that
is
capable of storing sev-
storing and retrieving logical records in buck-
than individually, access times can, in
ets rather
many
cases,
be im-
proved substantially.
Collision. Situation in which
room
not have sufficient
curs,
some means
Double hashing. A
c,
which
is
hashed to an address that does
is
collision resolution
is
ber of addresses) as
record
record
When
a collision
oc-
has to be found to resolve the collision.
handled by applying
number
to store the record.
scheme
in
which
collisions are
second hash function to the key to produce a
added to the original address (modulo the numa
many
located or an
times as necessary until either the desired
empty space
is
found. Double hashing helps
avoid some of the clustering that occurs with progressive overflow.
The
thumb. An assumption
80/20 rule of
80%) of the
accesses are
performed on
that a large percentage (e.g.,
small percentage (e.g.,
20%)
of the records in a file. When the 80/20 rule applies, the effective average search length is determined largely by the search lengths of the
more
active records, so attempts to
make
these search lengths short
can result in substantially improved performance.
Fold and add.
sized parts
added.
The
A
of
method of hashing
a
key are extracted
resulting
sum
in
which the encodings of fixedevery two bytes) and are
(e.g.,
can be used to produce an address.
Hashing. A technique for generating a unique home address for a given
key. Hashing is used when rapid access to a key (or its corresponding record)
is
required. In this chapter applications of hashing involve
direct access to records in a
cess items in arrays in
file,
RAM.
but hashing
is
also often used to ac-
In indexing, for example, an index
might be organized for hashing rather than for binary search if exfast searching of the index is desired.
Home address. The address generated by a hash function for a given
tremely
key. If a record
for the record
the record.
is
is
stored at
its
home
address, then the search length
one because only one access
record not
at its
home
is
required to retrieve
address requires
more than one
access to retrieve or store.
Indexed hash.
Instead of using the results of a hash to produce the ad-
dress of a record, the hash can be used to identify a location in an
index that in turn points to the address of the record. Although
approach requires one extra access for every search,
ble to organize the actual data records in a
way
it
makes
it
this
possi-
that facilitates other
types of processing, such as sequential processing.
493
494
HASHING
Mid-square method.
the key
is
hashing method in which
representation of
squared and some digits from the middle of the result are
used to produce the address.
Minimum
is
hashing. Hashing scheme in which the number of addresses
number of records. No storage space is
exactly equal to the
wasted.
Open
addressing. See progressive overflow.
Overflow. The
its
home
when
situation that occurs
record cannot be stored in
address.
Packing density. The proportion of allocated
space that actually
file
holds records. (Sometimes referred to as load factor.) If
full, its
are the
packing density
is
50%. The packing
two most important measures
ol a collision occurring
when
in
a file is half
density and bucket size
determining the likelihood
searching for
record in
a file.
Perfect hashing function. A hashing function that distributes records
uniformly, minimizing the number of collisions. Perfect hashing
functions are very desirable, but they are extremely difficult to find
for large sets of keys.
Poisson distribution. Distribution generated by the Poisson function,
approximate the distribution of records among
is random. A particular Poisson distribution depends on the ratio of the number of records to the number of
available addresses. A particular instance of the Poisson function,
p(x), gives the proportion of addresses that will have x keys assigned
to them. See better-than-random.
Prime division. Division of a number by a prime number and use of
the remainder as an address. If the address size is taken to be a prime
number p, a large number can be transformed into a valid address by
dividing it by p. In hashing, division by primes is often preferred to
division by nonprimes because primes tend to produce more random
which can be used
to
addresses if the distribution
remainders.
Progressive overflow. An overflow handling technique in which collisions are resolved by storing a record in the next available address
after its home address. Progressive overflow is not the most efficient
overflow handling technique, but it is one of the simplest and is adequate for many applications.
Randomize. To produce a number
(e.g.,
by hashing)
that appears to be
random.
Synonyms. Two
When
each
file
or
more
result in collisions. If buckets are used, several
are
same address.
synonyms always
records whose keys
different keys that hash to the
address can hold only one record,
synonyms may be
stored without collisions.
EXERCISES
Tombstone. A special marker placed in the key field of a record to
mark it as no longer valid. The use of tombstones solves two problems associated with the deletion of records: The freed space does
not break a sequential search for
ily
recognized
Uniform. Term
out evenly
as available
and
a record,
may
and the freed space
be reclaimed for
is
eas-
later additions.
applied to a distribution in which records are spread
among
addresses. Algorithms that produce uniform distri-
butions are better than randomizing algorithms in that they tend to
avoid the numbers of collisions that would occur by chance from
randomizing algorithm.
EXERCISES
1.
Use
the function
hash(KEY,
MAXAD)
described in the text to answer
the following questions.
a.
What
b.
Find two different words of more than four characters that are
is
the value of hash("Browns", 101)?
synonyms.
c. It is assumed
in the text that the function hash() does not
need to
generate an integer greater than 19,937. This could present a problem
we
if
ways
have
with addresses larger than 19,937. Suggest some
around this problem.
a file
to get
it is important to understand the relationof the available memory, the number of keys to be
hashed, the range of possible keys, and the nature of the keys. Let us give
2.
In understanding hashing,
ships
between the
names
size
to these quantities, as follows:
M=
the
number of memory
spaces available (each capable of hold-
ing one record);
r
=
=
the
number of records
the
number of unique home
to be stored in the
memory
spaces;
addresses produced by hashing the
record keys; and
key, which
may
be any combination of exactly five uppercase
characters.
Suppose h(K)
and
Ma.
is
hash function that generates addresses between
1.
How many
were one upperunique keys are possible? (Hint: If
would be 26 possible unique keys.)
case letter, rather than five, there
b.
How
are n
and
related?
495
496
HASHING
How
If the function h
would
3.
M related?
were minimum perfect hashing function,
and M be related?
c.
d.
are
n,
r,
and
The following
table
different hash functions
how
shows distributions of keys resulting from three
on a file with 6,000 records and 6,000 addresses.
Function
Function
Function
<J(0)
0.71
0.25
0.40
*(1)
0.05
0.50
0.36
d{2)
0.05
0.25
0.15
d(3)
0.05
0.00
0.05
d(4)
0.05
0.00
0.02
d(S)
0.04
0.00
0.01
d(6)
0.05
0.00
0.01
d(l)
0.00
0.00
0.00
a.
Which of the
records that
b.
c.
d.
4.
Which
Which
Which
There
is
is
three functions
(if
any) generates a distribution of
approximately random?
generates a distribution that
(if
is
nearest to uniform?
any) generates a distribution that
is
worse than random?
function should be chosen?
surprising mathematical result called the birthday paradox that
says that if there are
more than 23 people in a room, then there is a
two of them have the same birthday. How
than 50-50 chance that
better
is
the
birthday paradox illustrative of a major problem associated with hashing?
5. Suppose that 10,000 addresses are allocated to hold 8,000 records in a
randomly hashed file and that each address can hold one record. Compute
the following values:
The packing density for the file;
The expected number of addresses with no records
them by the hash function;
c. The expected number of addresses with one record
a.
b.
assigned to
assigned (no
synonyms);
The expected number of addresses with one record
more synonyms;
e. The expected number of overflow records; and
f.
The expected percentage of overflow records.
d.
plus
one or
EXERCISES
6.
Consider the
file
described in the preceding exercise.
number of overflow
expected
records
if the
What
the
is
10,000 locations are reorganized
as
a.
b.
7.
5,000 two-record buckets; and
1000 10-record buckets?
Make
1, 2, 5,
showing Poisson function values for r/N = 0.1, 0.5,
Examine the table and discuss any features and patterns
a table
and
10.
0.8,
that
provide useful information about hashing.
8.
There
is
an overflow handling technique called count-key progressive
works on block-addressable
number from a key,
overflow (Bradley, 1982) that
Instead of generating a relative record
disks as follows.
the hash function
generates an address consisting of three values: a cylinder, a track, and a
block number.
The corresponding
numbers
three
constitute the
home
address of the record.
Since block-organized drives (see Chapter
find a record with a given key, there
to find out
whether or not
it
is
no need
can often scan a track to
3)
to load a block into
contains a particular record.
The I/O processor
can direct the disk drive to search a track for the desired record.
empty record
direct the disk to search for an
its
home
a.
position, effectively
What
sive
about
is it
this
slot if a
memory
record
is
It
can even
not found in
implementing progressive overflow.
technique that makes
it
superior to progres-
overflow techniques that might be implemented on sector-orga-
nized drives.
b.
The main disadvantage of this technique
only with a bucket
size
of
Why
is
is
that
this the case,
it
can be used
and
why
is it
disadvantage?
9. In discussing
implementation
issues,
we
suggest initializing the data
file
marked empty before loading the file with
There are some good reasons for doing this. However, there
might be some reasons not to do it this way. For example, suppose you
want a hash file with a very low packing density and cannot afford to have
the unused space allocated. How might a file management system be
designed to work with a very large logical file, but allocate space only for
by
creating real records that are
actual data.
those blocks in the
file
that actually contain data?
10. This exercise (inspired
by an example
concerns the problem of deterioration.
are to be
made
to a
file.
Tombstones
in
Wiederhold, 1983, p. 136)
and deletions
A number of additions
are to be used
where necessary
preserve search paths to overflow records.
Show what the file looks like after
compute the average search length.
a.
the following operations, and
to
497
498
HASHING
Operation
Add
Add
Add
Add
Add
Alan
Home
Address
Bates
Cole
Dean
Evans
Del
Bates
Del
Cole
Add
Add
Finch
Del
Alan
Gates
Add Hart
How
has the use of tombstones caused the
What would be
file
to deteriorate?
the effect of reloading the remaining items in the
file
Dean, Evans, Finch, Gates, Hart?
b. What would be the effect of reloading the remaining items using
two-pass loading?
in the order
11. Suppose you have a file in which 20% of the records account for 80%
of the accesses, and that you want to store the file with a packing density of
and a bucket size of 5. When the file is loaded, you load the active 20% of
the records first. After the active 20% of the records are loaded, and before
what is the packing density of the partially
Using this packing density, compute the percentage of the active
20% which would be overflow records. Comment on the results.
the other records are loaded,
filled file?
our computations of average search lengths, we consider only the
takes for successful searches. If our hashed file were to be used in such
a way that searches were often made for items that are not in the file, it
would be useful to have statistics on average search length for an unsuccessful
search. If a large percentage of searches to a hashed file are unsuccessful,
how do you expect this to affect overall performance if overflow is han12. In
times
it
dled by
a.
Progressive overflow; or
b.
Chaining to
a separate
(See Knuth, 1973b, pp.
overflow area?
535-539
for a treatment of these differences.)
Although hashed files are not generally designed to support access to
in any sorted order, there may be times when batches of
transactions need to be performed on a hashed data file. If the data file is
13.
records
EXERCISES
sorted (rather than hashed), these transactions are normally carried out by
some
of cosequential process, which means that the transaction file also
file is hashed, the transaction file might also be
presorted, but on the basis of the home addresses of its records rather than
sort
has to be sorted. If the data
some more "natural" criterion.
Suppose you have a file whose records are usually accessed directly, but
that is periodically updated from a transaction file. List the factors you
would have to consider in deciding between using an indexed sequential
organization and hashing. (See Hanson, 1982, pp. 280-285, for a discussion
of these
issues.)
We
assume throughout this chapter that a hashing program should be
correctly whether a given key is located at a certain address. If
this were not so, there would be times when we would assume that a record
exists when in fact it does not, a seemingly disastrous result. But consider
what Doug Mcllroy did in 1978 when he was designing a spelling checker
program. He found that by letting his program allow one out of every 4,000
misspelled words to sneak by as valid (and using a few other tricks), he
could fit a 75,000-word spelling dictionary into 64 K of RAM, thereby
improving performance enormously.
Mcllroy was willing to tolerate one undetected misspelled word out of
14.
able to
tell
every 4,000 because he observed that drafts of papers rarely contained more
than 20 errors, so one could expect
program to
cases where
it
fail
it
at
to detect a misspelled
might be reasonable
most one out of every 200 runs of the
word. Can you think of some other
to report that a
key
exists
when
in fact
does not?
Jon Bentley (1985) provides an excellent account of Mcllroy 's program,
on the process of solving problems of this nature. D.
Dodds (1982) discusses this general approach to hashing, called
J.
check-hashing. Read Bentley's and Dodds's articles, and report on them to
your class. Perhaps they will inspire you to write a spelling checker.
plus several insights
Programming Exercises
15.
Implement and
16. Create a
key
this
version of the function hash().
with one record for every city in California. The
name of the corresponding city. (For the
exercise, there need be no fields other than the key field.)
hashed
in each record
purposes of
test a
file
is
to be the
Begin by creating a sorted list of the names of all of the cities and towns in
California. (If time or space is limited, just make a list of names starting
with the letter '5'.)
499
500
HASHING
the sorted list. What patterns do you notice that might
your choice of a hash function?
b. Implement the function hash() in such a way that you can alter
the number of characters that are folded. Assuming a packing density
of 1, hash the entire file several times, each time folding a different
number of characters, and producing the following statistics for
a.
Examine
affect
each run:
The number of collisions; and
The number of addresses assigned
more records.
0, 1, 2,
10,
and 10-or-
Discuss the results of your experiment in terms of the effects of
numbers of characters, and how they compare to
the results you might expect from a random distribution.
c. Implement and test one or more of the other hashing methods described in the text, or use a method of your own invention.
folding different
17.
Using some
set
of keys, such
as the
names of California towns, do
the
following:
a.
Write and
hash
files
test a
program
for loading the keys into three different
using bucket sizes of
1, 2,
and
b.
5,
respectively,
Use progressive overflow
Have your program maintain statistics on
ing density of 0.8.
length, the
maximum
and
pack-
for handling collisions.
the average search
search length, and the percentage of records
overflow records.
a Poisson distribution, compare your results with the
expected values for average search length and the percentage of
records that are overflow records.
that are
c.
Assuming
18.
Repeat exercise
17,
but use double hashing to handle overflow.
19.
Repeat exercise
17,
but handle overflow using chained overflow into
separate overflow area.
number of keys
Assume
to available
program
that the packing density
is
the ratio of
home addresses.
perform insertions and deletions in the file
problem using a bucket size of 5. Have the program
keep running statistics on average search length. (You might also
implement a mechanism to indicate when search length has deteriorated to
a point where the file should be reorganized.) Discuss in detail the issues
you have to confront in deciding how to handle insertions and deletions.
20. Write a
that can
created in the previous
FURTHER READINGS
FURTHER READINGS
There are
number of good surveys of hashing and issues
Knuth (1973b), Severance (1974), Maurer
related to hashing
and SorenTremblay, and Deutscher (1978). Textbooks concerned with file design
generally contain substantial amounts of material on hashing, and they often provide
extensive references for further study. Each of the following can be useful:
generally, including
(1975),
son,
Hanson
(1982)
of the issues
on comparing
with analytical and experimental
is filled
we
introduce, and
different
Bradley (1982) covers
file
file
many more, and
results exploring
also contains a
all
good chapter
organizations.
hashing generally but also includes
much informa-
on programming for hashed files using IBM PL/I.
Loomis (1983) also covers hashing generally, with additional emphasis on
tion
for hashed files in COBOL.
Teorey and Fry (1982) and Wiederhold (1983) will be useful to practitioners
interested in analyses of trade-offs among the basic hashing methods.
programming
One of the
applications of hashing that has stimulated a great deal of interest recently
development of spelling checkers. Because of special characteristics of spelling
checkers, the types of hashing involved are quite different from the approaches we
describe in this text. Papers by Bentley (1985) and Dodds (1982) provide entry into
the literature on this topic. (See also exercise 14.)
is
the
501
Extendible Hashing
11
CHAPTER OBJECTIVES
Describe the problem solved by extendible hashing
and related approaches.
Explain
it
how
combines
Show how
extendible hashing works;
tries
to
with conventional,
show how
static
hashing.
implement extendible hashing,
in-
cluding deletion.
Review
studies of extendible hashing performance.
Examine alternative approaches to the same problem, including dynamic hashing, linear hashing, and
hashing schemes that control splitting by allowing
for
overflow buckets.
503
CHAPTER OUTLINE
11.1
11.2
Introduction
How
11.2.1
11.4.2
Extendible Hashing Works
11.4.3 Collapsing the Directory
Tries
11.4.4 Implementing the Deletion
11.2.2 Turning the Trie into a
Directory
Handle Overflow
Implementation
11.5
Creating the Addresses
11.3.2 Implementing the Top-level
Operations
11.3.3 Bucket and Directory
Operations
11.3.1
11.3.4 Implementation
Extendible Hashing Performance
Space Utilization for Buckets
11.5.1
11.5.2 Space Utilization for the
Directory
11.6 Alternative
Summary
Approaches
Dynamic Hashing
11.6.1
11.6.2 Linear Hashing
11.4 Deletion
11.4.1
Operations
Summary of the Deletion
Operation
11.4.5
11.2.3 Splitting to
11.3
Procedure for Finding
Buddy Buckets
11.6.3 Approaches to Controlling
Overview of
the Deletion
Splitting
Process
11.1
Introduction
In Chapter 8
B-trees.
we began
B-trees
are
with
a historical
review of the work that led up to
such an effective solution to the problems that
stimulated their development that it is easy to wonder if there is any more
important thinking to be done about file structures. Work on extendible
hashing during the late 1970s and early 1980s shows that the answer to that
question
of the
is
file
yes. This chapter tells the story
structures that
emerge from
of that work and describes some
it.
B-trees do for secondary storage what
memory: They provide
dynamic data.
data
set.
By
dynamic
The key
feature of both
AVL
self-adjusting structures that include
As we add and
delete
AVL
trees
do
for storage in
way of using tree structures that works well with
we mean that we add and delete records from the
records,
trees
and B-trees
mechanisms
is
that they are
to maintain themselves.
the tree structures use limited,
local
restructuring to ensure that the additions and deletions do not degrade
performance beyond some predetermined
level.
HOW
Robust, self-adjusting data and
file
EXTENDIBLE HASHING
WORKS
505
structures are critically important to
data storage and retrieval. Judging from the historical record, they are also
hard to develop. It was not until 1963 that Adel'son-Vel'skii and Landis
developed a self-adjusting structure for tree storage in memory, and it took
another decade of
dynamic
work
before computer scientists found, in B-trees,
works well on secondary storage.
B-trees provide 0(\og k N) access to the keys in a file. Hashing, when
there is no overflow, provides access to a record with a single seek. But as
a file grows larger, the need to look for records that overflow their buckets
degrades performance. For dynamic files that undergo a lot of growth, the
tree structure that
performance of a static hashing system such as we described in Chapter 10
typically worse than the performance of a B-tree. So, by the late 1970s,
after the initial burst of new research and design work revolving around
B-trees was over, a number of researchers began to work on finding ways
to modify hashing so that it, too, could be self-adjusting as files grow and
shrink. As often happens when a number of groups are working on the
same problem, several different, yet essentially similar, approaches emerged
to extend hashing to dynamic files. We begin our discussion of the problem
by looking closely at the approach called "extendible hashing" described by
Fagin, Nievergelt, Pippenger, and Strong (1979). Later in this chapter we
compare this approach with others that emerged over the last decade.
is
1 1
.2
How
Extendible Hashing
Works
11.2.1 Tries
idea behind extendible hashing is to combine conventional hashing
with another retrieval approach called the trie. (The word trie is pronounced
so that it rhymes with sky.) Tries are also sometimes referred to as radix
searching because the branching factor of the search tree is equal to the
number of alternative symbols (the radix of the alphabet) that can occur in
each position of the key. A few examples will illustrate how this works.
The key
Suppose
we want to
anderson, andrews,
11.1.
As you can
and
build a
baird.
see, the
trie that stores
the keys able, abrahms, adams,
schematic form of the
trie is
shown
in Fig.
searching proceeds letter by letter through the key.
Since there are 26 symbols in the alphabet, the potential branching factor at
every node of the search is 26. If we used the digits 0-9 as our search
would be
one shown in
alphabet, rather than the letters a-z, the radix of the search
reduced to
Fig. 11.2.
10.
search tree using digits might look like the
506
EXTENDIBLE HASHING
anderson
andrews
baird
FIGURE 11.1
Radix 26
trie
that indexes
names according
trie
that indexes
numbers according
to the letters
of the alphabet.
FIGURE 11.2 Radix 10
digits they contain.
to the
HOW
Notice that
EXTENDIBLE HASHING
507
WORKS
we sometimes
use only a portion of the
need more information to complete the
search. This use-more-as-we-need-more capability is fundamental to the
structure of extendible hashing.
We use
key.
in searching a trie
more of the key
11.2.2 Turning the Trie
We
with
as
we
into a Directory
of two
our approach to extendible hashing:
Furthermore, since we are
retrieving from secondary storage, we will not work in terms of individual
keys, but in terms of buckets containing keys, just as in conventional
hashing. Suppose we have bucket A containing keys that, when hashed,
use
tries
a radix
Search decisions are
made on
in
a bit-by-bit basis.
have hash addresses that begin with the
hash addresses beginning with
10,
bits 01. Bucket B contains keys with
and bucket C contains keys with
addresses that start with 11. Figure 11.3 shows
a trie that
allows us to
retrieve these buckets.
How
should
we
represent the trie? If we represent
it
as a tree structure,
do a number of comparisons as we descend the tree. Even
worse, if the trie becomes so large that it, too, is stored on disk, we are faced
once again with all of the problems associated with storing trees on disk.
We might as well go back to B-trees and forget about extendible hashing.
we
are forced to
So, rather than representing the trie as a tree,
we
flatten
it
into an array
of contiguous records, forming a directory of hash addresses and pointers to
the corresponding buckets. The first step in turning a tree into an array
involves extending it so it is a complete binary tree with all of its leaves at
is enough
the same level as shown in Fig. 11.4(a). Even though the initial
to select bucket A, the new form of the tree also uses the second address bit
so both alternatives lead to the same bucket. Once we have extended the
tree this
11.4(b).
we can collapse it into the directory structure shown in Fig.
Now we have a structure that provides the kind of direct access
way,
associated with hashing:
10 2
th
Given an address beginning with the
bits 10, the
directory entry gives us a pointer to the associated bucket.
FIGURE 11.3 Radix 2 trie that
provides an index to buckets.
508
EXTENDIBLE HASHING
00
J
01
10
11
(a)
FIGURE 11.4 The
trie
from
(b)
11.3 transformed
Fig.
first into
complete binary
and then
tree,
flattened into a directory to the buckets.
11.2.3 Splitting to Handle Overflow
key
any hashing system
issue in
overflows.
The
is
what happens when
goal in an extendible hashing system
is
to find a
bucket
way
to
increase the address space in response to overflow, rather than responding
by creating long sequences of overflow records and buckets
that
have to be
searched linearly.
Suppose we
overflow. In
insert records that cause bucket
in
Fig.
this case the solution is simple: Since addresses
11.4(b)
to
beginning with
we can split bucket A by putting
new bucket D, while keeping only the 00 addresses
in A. Put another way, we already have two bits of address information,
but are throwing one away as we access bucket A. So, now that bucket A
is overflowing, we must use the full two bits to divide the addresses
between two buckets. We do not need to extend the address space; we
simply make full use of the address information that we already have.
00 and 01 are mixed together in bucket A,
all
the 01 addresses in a
Figure 11.5 shows the directory and buckets after the
Let's consider a
more complex
case.
split.
Starting once again with the
directory and buckets in Fig. 11.4(b), suppose that bucket
overflows.
How do we split bucket B and where do we attach the new bucket after the
split?
Unlike our previous example,
we do
not have additional, unused
bits
of address space that we can press into duty as we split the bucket. We now
need to use three bits of the hash address in order to divide up the records
that hash to bucket B. The trie illustrated in Fig. 11.6(a) makes the
distinctions required to complete the split. Figure 11.6(b)
trie
looks like once
leaves at the
form of the
same
trie.
it is
extended into
level,
and
Fig.
completely
11.6(c)
shows
full
shows what
this
binary tree with
all
the collapsed, directory
00
<
01
10
11
<
^
(
FIGURE 11.5 The directory
from Fig. 11.4(b) after bucket
/I
overflows.
FIGURE 11.6 The results of an overflow of bucket B in Fig. 11.4(b), represented
then as a complete binary tree, and finally as a directory.
(a)
000
001
;>
CZEZJ
CZD
(b)
(c)
first
as a
trie,
510
EXTENDIBLE HASHING
By
used in
building on the
search,
trie's ability to
we have doubled
from 2 2 to 2 3 cells. This ability to
shrink) the address space gracefully is what extendible hashing is
therefore, of our directory), extending
grow
all
(or
extend the amount of information
the size of our address space (and,
it
about.
We
have been concentrating on the contribution that
tries
make
to
extendible hashing; one might well ask where the actual hashing comes into
play. Why not just use the tries on the bits in the key itself, splitting buckets
and extending the address space as necessary? The answer to this question
grows out of hashing's most fundamental characteristic: A good hash
function produces a nearly uniform distribution of keys across an address
space. Notice that the trie shown in Fig. 11.6 is poorly balanced, resulting
is twice as big as it actually needs to be. If we had an
uneven distribution of addresses that placed even more records in buckets B
and D without using other parts of the address space, the situation would
get even worse. By using a good hash function to create addresses with a
nearly uniform distribution, we avoid this problem.
in a directory that
11.3
Implementation
11.3.1 Creating the Addresses
Now that we have a high-level overview of how extendible hashing works,
let's
look
at
The
pseudocode
in Fig. 11.7 that describes the algorithms in
more
with the functions that create the addresses, since
the notion of an extendible address underlies all other extendible hashing
detail.
place to start
is
operations.
The hash
function itself
hashing algorithm
we
is
simple variation on the fold-and-add
used in Chapter
10.
The only
difference
is
that
we do
not conclude the operation by returning the remainder of the folded address
divided by the address space.
We
don't need to do that, since in extendible
we don't have a fixed address space, instead using as much of the
address as we need. The division that we do perform in this function, when
we take the sum of the folded character values modulo 19,937, is to make
hashing
summation stays within the range of a signed 16-bit
machines that use 32-bit integers, we could divide by a larger
number and create an even larger initial address.
Since extendible hashing uses more bits of the hashed address as they
are needed to distinguish between buckets, we need a make_address function
that extracts just a portion of the full hashed address. We also use the
make_address function to reverse the order of the bits in the hashed address,
making the lowest-order bit of the hash address the highest-order bit of the
value used in extendible hashing. To see why this reversal of bit order is
sure that the character
integer. For
511
IMPLEMENTATION
FUNCTION hash(KEY)
set
set
set
if
SUM to
J
to
LEN to the length of the key
LEN is odd, concatenate a blank
to make the length even
while (J < LEN)
SUM := (SUM + 100*KEY[J]
J by 2
i nc rement
endwh i 1
to
the key
KEYCJ+13) mod 19937
return SUM
end FUNCTION
FIGURE
1.7 Function hash(KEY) returns an integer hash value for
KEY
for a
15-bit address space.
which is a set of keys and binary hash addresses
produced by our hash function. Even a quick scan of these addresses reveals
that the distribution of the least significant bits of these integer values tends
to have more variation than the high-order bits. This is because many of the
addresses do not make use of the upper reaches of our address space; the
desirable, look at Fig. 11.8,
high-order
By
bits often turn
out to be zero.
reversing the bit order,
working from
advantage of the greater variability of low-order
right to
bit values.
left,
we
take
For example,
we want to avoid having the addresses of bill,
and pauline turn out to be 0000, 0000, and 0000. If we work from right
to left, starting with the low-order bit in each address, we get 0011 for bill,
0001 for lee, and 1010 for pauline, which is a much more useful result.
given a four-bit address space,
lee,
The
make_address function, described in Fig. 11.9, accomplishes this bit
extraction and reversal.
number of address
bill
lee
pauline
alan
julie
mike
elizabeth
mark
The
0110
0100 0010
1111 0110
1100 1010
00101110 0000
0000 0111 0100
1100 0110
001
01
0000
0000
0000
0000
0000
0100
0011
DEPTH
argument
tells
the function the
bits to return.
1100
1000
FIGURE 11.8 Output from the
hash function for a number
0101
of keys,
001
1 001
1101
1010
01
512
EXTENDIBLE HASHING
FUNCTION make_address(KEY, DEPTH)
set
RETVAL to
/*
accumulate reversed
* /
string
0...001 mask to extract
*
low bit from N
to
bit
set MASK
to
HASH_VAL
:=
/*
/*
hash(KEY)
Summary of loop logic:
Shift RETVAL one position
Then move
left, to make room for a new low bit.
Then
HASH_VAL's low bit to RETVAL s low bit.
shift HASH_VAL in the opposite direction, to the
'
* *
right, so we can look at the next lowest bit.
Keep doing this until we have moved as many bits
as we need from HASH_VAL to RETVAL in reverse
order
.
*/
for
:=
to
RETVAL
LOWBIT
RETVAL
HASH_VAL
next
DEPTH
RETVAL left shifted one position
HASH_VAL bitwise ANDed with MASK
RETVAL bitwise ORed with LDWBIT
= HASH_VAL right
shifted one position
return RETVAL
end FUNCTION
FIGURE 11.9 Function make_address (DEPTH) gets a hashed address, reverses the
bits, and returns an address of DEPTH bits.
order of the
FIGURE 11.10
BUCKET and DIR_CELL
record structures.
Record Type: BUCKET
DEPTH
integer count of the number of bits used
"in common" by the keys in this bucket
COUNT
integer count of the number of keys in
the buc ket
KEYM
array
MAX_BUCKET_S ZE
hold k eys
[ 1
of
strings to
Record Type: D RECT0RY_CELL
BUCKET_REF relative record number or other reference
to a specific BUCKET record on disk
I
IMPLEMENTATION
513
11.3.2 Implementing the Top-level Operations
Our
extendible hashing scheme consists of
a set of buckets and a directory
them. Each bucket is a record that contains the information
shown in Fig. 11.10. These bucket records are stored in a file; we retrieve
that references
them as necessary.
Each cell in the directory
Because
we
use
consists of a reference to a
direct access to find
directory as an array of these cells in
directory records,
RAM. The
BUCKET
we implement
than the directory
From
the
address values returned by
make_address are treated as subscripts for this array, ranging from
less
record.
to
one
size.
view of the driver function, use of the system
RAM from
disk, a set of calls from the user to find or add keys, and a final, closing step
that writes the possibly modified directory back to disk. Pseudocode for the
driver, initialization, and close functions are shown in Fig. 11.11.
the high-level
consists of an initialization step that reads the directory into
FIGURE 11.11 The driver, ex_init, and ex_close functions provide a high-level view
hashing program operation.
of the extendible
FUNCTION driverO
ex_i n i t (
call op_add() and op_find() as directed by the user
ex_c 1 ose(
end FUNCTION
FUNCTION ex_init()
open (or create, as necessary) the directory
and buc ket files
if the hash file already exists
read directory records into the array DIRECTORY
DIR_DEPTH := log 2 (size of DIRECTORY)
else
allocate an initial directory consisting of a
single cell
set DIR_DEPTH to
allocate an initial bucket and assign its address
to the directory cell
end i f
end FUNCTION
FUNCTION ex_close()
write the directory back to disk
close files
end FUNCTION
514
EXTENDIBLE HASHING
Note
that the
DIR_DEPTH
directly
is
related
to
the size of the
directory, since
2
If
dir_depth =
we are starting
we are using
the
hash
file,
the
ln
DIRECTORY.
DIR_DEPTH
same bucket, no matter what
which means
the keys go
get the address of
zero,
We
their address.
everything-goes-here bucket and assign
initial,
is
no bits to distinguish between addresses;
that
into the
new
number of cells
the
it
all
to the single directory
cell.
Given a way to open and close the file, we are ready to add records. The
op_add and op_find functions are outlined in Fig. 11.12.
The op_find function turns the key into a directory address. Given this
address,
we do
and assign
a direct
FOUND_BUCKET
return
calls
contains the key,
we
return success; otherwise,
we
FAILURE.
The op_add
in the
lookup of the bucket location, retrieve the bucket
and then search for the key. If
FOUND_BUCKET,
to
it
hash
file,
function begins by calling op_find. If the key already exists
op_add returns immediately;
bk_add_key
to insert
if
the key
is
not found, op_add
it.
11.3.3 Bucket and Directory Operations
When
not
op_add
calls
bk_add_key
bk_add_key
full,
bucket, however,
it
passes a bucket and a key. If the bucket
11.13) simply inserts
(Fig.
requires a
split,
which
is
it
into the bucket.
where things
start
is
full
to
get
interesting.
What we do when we split a bucket depends on the relationship
between the number of address bits used in the bucket and the number used
in the directory as a whole. The two numbers are often not the same. To see
this,
look
at Fig. 11.6(a).
The keys
The
directory uses three bits to define
its
address
from keys in other
buckets by having an initial bit. All the other bits in the hashed key values
in bucket A can be any value; it is only the first bit that matters. Bucket A
space (8
is
cells).
using only one
in
bucket
are distinguished
bit.
The keys in bucket C all share a common
with 11. The keys in buckets B and D use
first
two
bits;
they
all
begin
three bits to establish their
bucket locations. If you look at Fig. 11.6(c),
you can see how using more or fewer address bits changes the relationship
between the directory and the bucket. Buckets that do not use as many
address bits as the directory have more than one directory cell pointing to
identities and, therefore, their
them.
If
we
directory,
one of the buckets that is using fewer address bits than the
which
therefore is referenced from more than one directory
and
split
515
IMPLEMENTATION
FUNCTION: op_add (KEY)
/* if we find the key, we do not add
if op_find(KEY, FOUND_BUCKET)
return FAILURE
/*
* *
second copy */
otherwise, add the key to the bucket that op_find
ret r i eved
*/
bk_add_key(FOUND_BUCKET, KEY)
return SUCCESS
end FUNCTION
FUNCTION: op_find (KEY, FOUND_BUCKET)
/*
**
*/
uses DIR_DEPTH, the number of bits used to create
the addresses in the directory
/* create an address based on directory depth
ADDRESS := ma k e_addr es s ( KE Y DIR_DEPTH)
*/
/*
**
*/
get the bucket that will
key exists in the file
FOUND_BUCKET
if
:=
contain the key,
if
the
bucket referenced by
DIRECTORYCADDRESS] .BUCKET_REF
FOUND_BUCKET contains the KEY
return SUCCESS
else
return FAILURE
end FUNCTION
FIGURE 11.12 op_add and op_find functions.
FUNCTION bk_add_key(BUCKET, KEY)
(BUCKET. USED
add the key
else
if
<
MAX_BUCKET_S ZE )
bk_split(BUCKET)
op_add(KEY)
endi
end FUNCTION
FIGURE 1 1.13 bk_add_key
function adds the key to the
existing bucket if there is
room.
splits
key.
If
it
the bucket
is full,
and then adds the
it
516
EXTENDIBLE HASHING
FUNCTION bk_split(BUCKET)
/* if the depth used for the BUCKET addresses is
** already the same as the address depth in the
** directory, we must first split the directory
** to double the directory address space
*/
if
(BUCKET. DEPTH == DIR_DEPTH)
di r_double( )
allocate NEW_BUCKET
/*
**
*/
f
find the range of directory entries for the new
bucket, given the depth and keys in the old bucket
ind_new_range(BUCKET, NEW_START, NEH_END)
/*
the new bucket over this range */
insert
dir_ins_bucket(NEW_BUCKET, NEW_START, NEN_END)
/*
change the address depths in the buckets to
reflect the split
*/
increment BUCKET. DEPTH
NEULBUCKET. DEPTH := BUCKET. DEPTH
redistribute the keys between the two buckets
end FUNCTION
FIGURE 11.14 bk_split function divides keys between an existing bucket and a new
bucket. If necessary, it doubles the size of the directory to accommodate the new
bucket.
cell,
the
we
can use half of the directory
split.
cells to
Suppose, for example, that
we
point to the
split
bucket
new
bucket
in Fig.
after
11.6(c).
Before the split only one bit, the initial zero, is used to identify keys that
belong in bucket A. After the split, we use two bits. Keys starting with 00
(directory cells 000 and 001) go in bucket A; keys starting with 01 (directory
cells 010 and Oil) go in the new bucket. We do not have to expand the
directory because the directory already has the capacity to keep track of the
additional address information required for the
If,
on the other hand, we
as the directory,
such
as
split a
split.
bucket that has the same address depth
B or D in Fig. 11.6(c), then there are no
we can use to reference the new bucket.
we have to double the size of the directory,
for every one that is currently there, so we
buckets
additional directory entries that
Before
we
creating a
can
new
split the
bucket,
directory entry
can accommodate the
new
address information.
517
IMPLEMENTATION
we
Figure 11.14 shows the bucket-splitting logic in pseudocode. First
compare
number of bits used
the
we double
new bucket
depths are the same,
Next we
get the
we need
addresses
for
to double the directory. If the
the directory before proceeding.
we need for
we will use
the
that
range of directory addresses that
instance,
number used
for the directory with the
the bucket to determine whether
split.
Then we
new
for the
find the
bucket.
For
when we split bucket A in Fig. 11.6(c), the range of directory
for the new bucket is from 010 to Oil. We attach the new bucket
to the directory over this range, adjust the bucket address depth information
in
both buckets to
of an additional address
reflect the use
redistribute the keys
from
the original bucket across the
The most complicated operation supporting
find_new_range, which finds the range of directory
the
new
bucket instead of the old one
pseudocode
in Fig. 11.15.
To
see
how
it
bit,
the bk_split function
cells that
after the split.
It
is
described in
works, return, once again, to
FIGURE 11.15 find_new_range function finds the start and end directory addresses
the new bucket by using information from the old bucket.
f
md_new_range(OLD_BUCKET
NEN_START, NEW_END)
/* find the shared address for the OLD bucket
SHARED_ADDRESS := ma k e_addres s (any KEY from
OLD_BUCKET, OLD_BUCKET DEPTH
*/
/*
**
**
**
**
**
*/
shift everything one bit to the left, then put
in the lowest bit. This is the shared address
for the new bucket. Fill the new shared address on
the right with zero bits until we have reached the
directory depth. This is the start of the range.
Fill it with
bits -- this is the range's end.
NEW_SHARED
NEW_SHARED
:=
:=
SHARED_ADDRESS left shifted
NEW_SHARED bitwise ORed with
place
1)
BITS_TO_FILL := DIR_DEPTH - ( OLD_BUCKET DEPTH
set NEW_START and NEW_END to the NEW_SHARED value
for J :=
to BITS_TO_FILL
place
NEH_START := NEN_START left shifted
place
NEU_END := NEU_END left shifted
NEW_END := NEH_END bitwise ORed with
.
next
end FUNCTION
is
should point to
for
FUNCTION
and then
two buckets.
Fig.
518
EXTENDIBLE HASHING
Assume that we need to split bucket A, putting some of the keys
new bucket E. Before the split, any address beginning with a leads
11.6(c).
into a
address of the keys in bucket A is 0.
add another address bit to the path leading
to the keys; addresses leading to bucket A now share an initial 00 while
those leading to E share an 01. So, the range of addresses for the new bucket
is all directory addresses beginning with 01. Since the directory addresses
to
A. In other words, the shared
When we
split
bucket
A we
FIGURE 11.16 Directory operations to support bk_split: the dir_double and
dir_ins__bucket functions.
FUNCTION dir_double()
/* calculate the current size and new size */
CURRENT_SIZE := 2 DIR - DEPTH
NEH_SIZE := 2 * CURRENT_SIZE
allocate memory for the new, larger directory
temporarily call it NEW_DIR
/*
**
**
**
*/
--
Transfer the bucket addresses from the old
directory to the new one. Each cell in the
original is copied, into two cells of the
expanded directory
for
:=
to CURRENT_SIZE neh_dirc2*i .bucket_ref
d rectory
neh_dir[2*i+1 .bucket_ref
directory:
I
next
.bucket_ref
:=
.bucket_ref
free memory for old DIRECTORY
rename NEN_DIR to DIRECTORY
increment DIR_DEPTH
end FUNCTION
FUNCTION dir_ins_bucket(BUCKET_ADDRESS, START, LAST)
for J := START to LAST
DIRECTORYC J] .BUCKET_REF := BUCKET_ADDRESS
next
end FUNCTION
519
IMPLEMENTATION
bits, the new bucket is attached to the directory cells starting with
010 and ending with Oil.
Suppose that the directory used a five-bit address instead of a three-bit
address. Then the range for the new bucket would start with 01000 and
would end with 01111. This range covers all five bit addresses that share 01
as the first two bits. The logic for finding the range of directory addresses
use three
for the
new
bucket.
It
bucket, then, starts by finding shared address bits for the
then
fills
the address out with zeroes until
used in the directory. This
bits
is
we have
the
new
number of
the start of the range. Filling the address
out with ones produces the end of the range.
The
directory operations required to support bk_split are easy to
implement. They are outlined in pseudocode in Fig. 11.16. The first,
dir_double, simply calculates the new directory size, allocates the required
memory, and writes the information from each old directory cell into two
successive cells in the new directory. It finishes by freeing the old space
associated with the name DIRECTORY, renaming the new space as the
DIRECTORY, and increasing the DIR_DEPTH value to reflect the fact
that the directory
The
range of directory
make
is
now
using an additional address
bit.
dir_ins_bucket function, used to attach a bucket address across a
cells, is
simply
loop that works through the
cells to
the change.
11.3.4 Implementation Summary
Now that we have assembled all of the pieces necessary to add records to an
extendible hashing system,
The op_add
let's
see
how
the pieces
work
together.
function manages record addition. If the key already exists,
op_add returns immediately. If the key does not exist, op_add calls
bk_add_key, passing it the bucket into which the key is to be added. If
bk_add_key finds that there is still room in the bucket, it adds the key and
the operation
is
complete. If the bucket
is full,
bk_add_key
calls bk_split to
handle the task of splitting the bucket.
The bk_split function starts by determining whether the directory
large
enough
to
accommodate
the
new
larger, bk_split calls a function that doubles the directory size.
then allocates
new
bucket, attaches
it
is
bucket. If the directory needs to be
The
function
to the appropriate directory cells,
and divides the keys between the two buckets.
When bk_add_key regains control after bk_split has allocated a new
bucket, it calls op_add to try to place the key into the new, revised directory
structure. The op_add function, of course, calls bk_add_key again, recursively.
new
This cycle continues until there
key.
is
bucket that can accommodate the
520
11.4
EXTENDIBLE HASHING
Deletion
11.4.1 Overview of the Deletion Process
If
extendible hashing
trees,
it
must be
AVL
to be a truly dynamic system, like B-trees or
is
able to shrink
files
gracefully as well as
grow them. When
we
delete a key, we need a way to see if we can decrease the size of the file
system by combining buckets and, if possible, decreasing the size of the
directory.
As with any dynamic system,
the important question during deletion
concerns the definition of the triggering condition:
When do we combine
buckets? This question, in turn, leads us to ask,
Which buckets can be
combined? For B-trees the answer involves determining whether buckets
are siblings and whether they are at the leaf level of the tree. In extendible
hashing
we
Look
use a similar concept: buckets that are buddy buckets.
again
the
at
combined? Trying
to
trie
in
Fig.
11.6(b).
Which buckets could be
A would mean
combine anything with bucket
collapsing everything else in the trie
first.
Similarly, there
bucket that could be combined with bucket C. But buckets
no
is
B and
single
D are in
same configuration as buckets that have just split. They are ready to be
combined; they are buddy buckets. We will take a closer look at the
question of finding buddy buckets as we consider implementation of the
deletion procedure; for now let's assume that we combine buckets B and D.
After combining buckets, we examine the directory to see if we can
make changes there. Looking at the directory form of the trie in Fig.
1 1 .6(c), we see that once we combine buckets B and D, directory entries 100
and 101 both point to the same bucket. In fact, each of the buckets has at
least a pair of directory entries pointing to it. In other words, none of the
buckets requires the depth of address information that is currently available
in the directory. That means that we can shrink the directory and reduce the
the
address space to half
its size.
Reducing the size of the address space restores the directory and bucket
structure to the arrangement shown in Fig. 11.4, before the additions and
splits that produced the structure in Fig. 11.6(c). Reduction consists of
collapsing each adjacent pair of directory cells into a single
since both cells in each pair point to the
nothing more than a reversal of the directory
when we need to add new directory cells.
11.4.2 A Procedure
for Finding
cell.
This
same bucket. Note
splitting
is
easy,
that this
procedure that
is
we use
Buddy Buckets
overview of how deletion works, we begin by focusing on
buddy buckets. Given a bucket, how do we find its buddy? Figure 11.17
Given
this
521
DELETION
FUNCTION bk_f i nd_buddy(BUCKET)
/* NOTE: this function uses DIR_DEPTH -- we
** assume this value is available globally or
** through a function call
*/
/*
**
*/
if
There
/*
**
*/
if
unless the bucket has the same depth as the
directory, there is no single bucket to pair with
is
is no buddy if the DIR_DEPTH is
just a single bucket)
(there
(DIR_DEPTH == 0)
return N0_BUDDY
(BUCKET. DEPTH < DIR_DEPTH)
return N0_BUDDY
/* find the shared address for this bucket */
SHARED_ADDRESS := ma k e_addr ess(any KEY from
BUCKET, BUCKET. DEPTH)
/*
**
*/
flip the last bit -- that
buddy bucket
BUDDY_ADDRESS
:=
is
the address of
the
SHARED_ADDRESS exclusive ORed with
return BUDDY_BUCKET found at BUDDY_ADDRESS
end FUNCTION
FIGURE 11.17 The bk_find_buddy function returns a buddy bucket or the special
NO_BUDDY if none is found.
signal
The procedure works by checking
whether it is possible for there to be a buddy bucket. Clearly, if the
directory depth is zero, meaning that there is only a single bucket, there
cannot be a buddy.
The next test compares the number of bits used by the bucket with the
number of bits used in the directory address space. A pair of buddy buckets
is a set of buckets that are immediate descendents of the same node in the
trie. They are, in fact, pairwise siblings resulting from a split. Going back
to Fig. 11.6(b), we see that asking whether the bucket uses all the address
bits used in the directory is another way of asking whether the bucket is at
the lowest level of the trie. It is only when a bucket is at the outer edge of
the trie that it can have a single parent and a single buddy.
describes the procedure in pseudocode.
to see
522
EXTENDIBLE HASHING
Once we determine
we
that there
is a
buddy bucket, we need
to find
its
we have at hand;
this is the shared address of the keys in the bucket. Since we know that the
buddy bucket is the other bucket that was formed from a split, we know
address. First
that the
Once
find the address used to find the bucket
buddy has
the
same address
again, this relationship
11.6(b). So, to get the
buddy
is
in
all
illustrated
address,
we
regards except for the
last bit.
in Fig.
by buckets B and
flip
the last
bit.
We
return the
buddy bucket.
11.4.3 Collapsing the Directory
The
other important support function used to implement deletion
the
is
function that handles collapsing the directory. Downsizing the directory
is
one of the principal potential benefits of deleting records. In our implementation we use one function to check to see whether downsizing is possible
and, if it is, to actually collapse the directory. Figure 11.18 shows
pseudocode for this function, called dir_try_collapse( ).
The function begins by making sure that we are not at the lower limit
of directory size. By treating the special case of a directory with a single cell
here, at the start of the function, we simplify subsequent processing: With
the exception of this case, all directory sizes are evenly divisible by two.
The actual test for the COLLAPSE_CONDITION consists of examining each pair of directory entries. We assume at the outset that we can
collapse the directory and then look for a pair of directory cells that do not
both point to the same bucket. As soon as we find such a pair, we know that
We set the value of the COLLAPSE_
and break out of the test loop. If we get all the way
through the directory without encountering such a pair, then we can
we
cannot collapse the directory.
CONDITION
to false
collapse the directory.
The
actual collapsing operation consists of allocating space for a
directory that
is
references shared
new
half the size of the original and then copying the bucket
by each
cell pair to a single cell in
the
new
directory.
11.4.4 Implementing the Deletion Operations
Now
that
we have
deletion, finding
an approach to the two
buddy buckets and
critical
support operations for
collapsing the directory,
we
are ready
of the deletion operation.
The highest-level deletion operation, op_del, is very simple. We first try
to find the key to be deleted. If we cannot find it, we return failure; if we
do find it, we call a service function to remove the key from the bucket. We
to construct the higher levels
DELETION
FUNCTION dir_try_collapse()
/* the directory is already at minimum size when
** the depth is zero
*/
if
/*
**
**
*/
(DIR_DEPTH == 0)
return FAILURE
check each pair of directory cells to see whether
each member references the same bucket -- if so,
we can collapse the directory.
DIR_SIZE := 2 DIR - DEPTH
COLLAPSE_CDNDITION := TRUE;
for J
to DIR_SIZE
:=
if
/*
by
assume the best,
try to disprove
then
it
*/
(DIRECTORY!! J] .BUCKET_REF
!=
directory: j+1 .bucket_ref)
c0llapse_c0nditi0n := false
break out of the loop
]
next
/*
**
**
*/
if
endi f
J by
if we have a collapse condition, create a new
directory that is half the size of the original,
and transfer the bucket references
(C0LLAPSE_C0NDITI0N)
NEN_DIR_SIZE := DIR_SIZE / 2
allocate memory for NEW_DIR
for
:=
to
NEW_DIR[
next
J]
NEH_DIR_SIZE .BUCKET_REF :=
DIRECT0RY[2*J] .BUCKET_REF
1
free memory for old DIRECTORY
rename NEN_DIR to DIRECTORY
decrement DIR_DEPTH
endif
return C0LLAPSE_C0ND
ON
end FUNCTION
FIGURE 11.18 The dir_try_collapse function first tests to see whether the directory
can be collapsed. If the test succeeds, the directory is collapsed.
523
524
EXTENDIBLE HASHING
FUNCTION: op_del (KEY)
if (op_f indCKEY, FOUND_BUCKET) == FAILURE)
return FAILURE
found it -- now delete it */
return ( b k_de l_k ey ( FOUND_BUCKET
/*
KEY))
end FUNCTION
FUNCTION bk_del_key(BUCKET, KEY)
set KEY_REMOVED to FALSE
Look
/*
**
*/
if
for KEY in BUCKET -- if found
remove the KEY
set KEY_REMOVED to TRUE
decrement BUCKET. COUNT
if a key was removed, see whether we can combine
this bucket with its buddy bucket
(KEY_REMOVED)
bk_try_combine( BUCKET)
return SUCCESS
else
return FAILURE
endi
end FUNCTION
FIGURE 11.19 The op_del and bk_del_key functions.
return the value reported back from the service function. Figure 11.19
describes op_del and the service function, bk_del_key
The bk_del_key
function does
consists of finding the
second
which
its
work
key and physically
in
pseudocode.
two steps. The first step
removing it from the bucket. The
in
is removed, consists of calling
key has decreased the size of the bucket
enough to allow us to combine it with its buddy.
Figure 11.20 shows the pseudocode for bk_try_combine and its service
function, bk_combine. Note that when we combine buckets, we reduce the
address depth associated with the bucket: Combining buckets means that
we use one less address bit to differentiate keys.
After combining the buckets, we call dir_try_collapse to see if the
decrease in the number of buckets enables us to decrease the size of the
step,
takes place only if a key
bk_try_combine to see
if deleting the
d!
DELETION
525
directory. If we do, in fact, collapse the directory (dir_try_collapse succeeds),
bk_try_combine
created a
calls itself recursively.
new buddy
for the
Collapsing the directory
BUCKET;
it
may
may have
do even more
recursive combining and
be possible to
combination and collapsing. Typically, this
collapsing happens only when the directory has a number of empty buckets
that are awaiting changes in the directory structure that finally produce a
buddy
to
combine with.
FIGURE 1 1.20 The bk_try_combine function tests to see whether a bucket can be
combined with its buddy. If the test succeeds, bk_try_combine calls bk_combine to do the actual combination.
FUNCTION bk_try_combine(BUCKET)
/* If there i5 no baddy return right away
BUDDY := b k_f i nd_buddy(BUCKET)
if (BUDDY == ND_BUDDY)
return
/*
if
*/
see if we can combine buckets */
(BUDDY. COUNT + BUCKET. COUNT <= MAX_BUCKET_S ZE
I
bk_combine(BUCKET
BUDDY)
free memory used by the BUDDY bucket
reassign the DIRECTORY value for the BUDDY so
that it now references the BUCKET
/*
**
*/
if
see if we can collapse the directory -- if
there may be a new buddy to combine with
so,
(di r_t ry_col lapse(
bk_try_combine( BUCKET)
endi
end FUNCTION
FUNCTION bk_combine(BUCKET, BUDDY)
for J :=
to BUDDY. COUNT
increment BUCKET. COUNT
BUCKETCBUCKET. COUNT] .KEY = BUDDYCJ3.KEY
1
next
decrement BUCKET. DEPTH
end FUNCTION
526
EXTENDIBLE HASHING
11.4.5 Summary
of the Deletion Operation
Deletion begins with
a call to op_del that passes the key that is to be deleted.
key cannot be found, there is nothing to delete. If the key is found, the
bucket containing the key is passed to bk_del_key
The bk_del_key function deletes the key and then passes the bucket on
to bk_try_combine to see if the smaller size of the bucket will now permit
combination with a buddy bucket. The bk_try_combine function first checks
to see if there is a buddy bucket. If not, we are done. If there is a buddy, and
if the sum of the keys in the bucket and its buddy is less than or equal to the
size of a single bucket, we combine the buckets.
The elimination of a bucket through combination might enable
collapsing of the directory to half its size. We investigate this possibility by
calling bk_try_collapse. If collapsing succeeds, we may have a new buddy
bucket, and so bk_try_combine calls itself again, recursively.
If the
1.5
Extendible Hashing Performance
Extendible hashing
is
an elegant solution to the problem of extending and
contracting the address space for a hash
shrinks.
How
must consider the
The time dimension
hashing can be kept in
it
is
RAM, two
RAM,
accesses
as the file itself
grows and
the answer to this question
space.
easy to handle: If the directory for extendible
a single access
retrieve a record. If the directory
of
file
work? As always,
trade-off between time and
well does
may
is
is all
so large that
it
that
is
ever required to
must be paged
in
and out
be necessary. The important point
extendible hashing provides O(l) performance: Since there
is
these access time values are truly independent of the size of the
file.
for extendible hashing
are
Questions about
complicated than questions about access time.
space utilization
is
that
no overflow,
more
We need to be concerned
about two uses of space: the space for the buckets and the space for the
directory.
11.5.1 Space Utilization
for
Buckets
In their original paper describing extendible hashing, Fagin, Nievergelt,
and Strong include analysis and simulation of extendible
hashing performance. Both the analysis and simulation show that the space
utilization is strongly periodic, fluctuating between values of 0.53 and 0.94.
The analysis portion of their paper suggests that for a given number of
Pippenger,
527
EXTENDIBLE HASHING PERFORMANCE
r and a block size of
approximated by the formula
records
the average
b,
N
Space utilization, or packing density,
to the total
is
2'
b In
number of records
number of blocks
is
defined as the ratio of the actual
number of records
that could be stored in the
allocated space:
Utilization
Substituting the approximation for
Utilization
we
So,
bN
N gives
us:
0.69.
In 2
expect average utilization of 69%. In Chapter
space utilization for B-trees,
we found
8,
where we looked
that simple B-trees tend to
have
at
a
of about 67%, but this can be increased to over 85% by
redistributing keys during insertion, rather than just splitting when a page
utilization
is
full.
So, B-trees tend to use less space than simple extendible hashing,
typically at a cost of requiring a
The average
few extra
seeks.
space utilization for extendible hashing
is
only part of the
story; the other part relates to the periodic nature of the variations in space
utilization.
It
turns out that if
we
have keys with randomly distributed
fill up at about
addresses, the buckets in the extendible hashing table tend to
the
same time and therefore tend
to split at the
same
time. This explains the
As the buckets
up, space utilization
can reach past 90%. This is followed by a concentrated series of splits that
reduce the utilization to below 50% As these now nearly half-full buckets
fill up again, the cycle repeats itself.
large fluctuations in space utilization.
fill
11.5.2 Space Utilization
for the Directory
directory used in extendible hashing grows by doubling its size. A
prudent designer setting out to implement an extendible hashing system
will want assurance that this doubling levels off for reasonable bucket sizes,
The
even when the number of keys is quite large. Just how large
should we expect to have, given an expected number of keys?
Flajolet (1983) addressed this question in a lengthy, carefully
paper that produces
size.
number of
Table 11.1, which
is
different
ways
directory
developed
to estimate the directory
taken from Flajolet's paper, shows the expected
value for the directory size for different numbers of keys and different
bucket
sizes.
528
EXTENDIBLE HASHING
TABLE 11.1
Expected directory size
records
for a given
bucket size b and
total
number
of
20
10
100
50
200
10
10
10
10
10
1
68.20
M
M
M
11.64 M
25.60
424.10
6.90
111.11
K =
From
K
K
K
1.50
10
3
,
0.30
4.80
K
K
K
0.10
1.70
16.80
62.50
K
K
K
K
16.80
K
K
K
K
0.52
0.26
0.00
0.50
4.10
M
2.25 M
0.26
1.02
10
K
K
K
0.00
0.20
2.00
0.13
0.00
1.00
6
.
Flajolet, 1983.
Flajolet
also
provides
the
following
formula
for
making rough
estimates of the directory size for values that are not in this table.
that this
formula tends to overestimate directory
Estimated directory
1 1
8.10
K
K
K
K
0.00
.6
size
size
-^j r (1
+ 1/
by
a factor
He
notes
of 2 to
4.
Alternative Approaches
11.6.1 Dynamic Hashing
and Strong produced their
paper describing a scheme
called dynamic hashing. Functionally, dynamic hashing and extendible
hashing are very similar. Both use a directory to track the addresses of the
buckets, and both extend the directory through the use of tries.
In 1978, before Fagin, Nievergelt, Pippenger,
paper on extendible hashing, Larson published
The key
difference
between the approaches
like conventional, static hashing, starts
address space of a fixed
size.
with
is
that
dynamic hashing,
hash function that covers an
As buckets within
that fixed address space
forming the leaves of a trie that grows down from the
original address node. Eventually, after enough additions and splitting, the
buckets are addressed through a forest of tries that have been seeded out of
overflow, they
split,
the original static address space.
Let's look at an example. Figure 11.21(a) shows an initial address space
of four, and four buckets descending from the four addresses in the
529
ALTERNATIVE APPROACHES
(a)-
)(
Original address space
)C
40
X XIX
41
Original address space
FIGURE 11.21 The growth of index
in
dynamic hashing.
directory. In Fig. 11.21(b)
the
two buckets
resulting
we have
split the
from the
split as
bucket
address
at
40 and 41.
We
of the directory node at address 4 from a square to a
changed from an external node, referencing a bucket,
that points to
two
We address
circle
because
to an internal
it
has
node
child nodes.
In Fig. 11.21(c)
new
4.
change the shape
we
split the
external nodes 20 and 21.
downward
bucket addressed by node
We
2, creating the
also split the bucket addressed
by
41,
410 and 411. Because the directory
node 41 is now an internal node rather than an external one, it changes from
a square to a circle. As we continue to add keys and split buckets, these
directory tries continue to grow.
Finding a key in a dynamic hashing scheme can involve the use of two
extending the
trie
to include
hash functions, rather than just one.
covers the original address space. If
First,
you
external node, and therefore points to
However,
if the
directory node
is
there
is
the hash function that
find that the directory
a
bucket, the search
is
node
is
an
complete.
an internal node, then you need additional
address information to guide you through the ones and zeroes that form the
530
EXTENDIBLE HASHING
trie.
Larson suggests using a second hash function on the key and using the
of this hashing as the seed for a random-number generator that
result
produces
sequence of ones and zeroes for the key. This sequence describes
the path through the
It is
trie.
interesting to
compare dynamic hashing and extendible hashing.
but illuminating, characterization of similarities and differences is that
while both schemes extend the hash function locally, as a binary search trie,
brief,
in
order to handle overflow, dynamic hashing expresses the extended
directory as a linked structure while extendible hashing expresses
perfect tree,
which
is
Because of this fundamental
utilization within
the buckets
Moreover, since the
it
as a
in turn expressible as an array.
similarity,
is
the
it is
not surprising that the space
same (69%)
for
both approaches.
directories are essentially equivalent,
expressed differently,
and are just
follows that the estimates of directory depth
it
developed by Flajolet (1983) apply equally well to dynamic hashing and
extendible hashing. (In section 11.5.2 we talk about estimates for the
directory size for extendible hashing, but we know that in extendible
hashing directory depth = log 2 directory size.)
The primary
difference
between the two approaches
that
is
dynamic
hashing allows for slower, more gradual growth of the directory, wmereas
it. However, because
dynamic hashing must be capable of holding pointers
to children, the actual .size of a node in dynamic hashing is larger than a
directory cell in extendible hashing, probably by at least a factor of two. So,
the directory for dynamic hashing will usually require more space in
memory. Moreover, if the directory becomes so large that it requires use of
extendible hashing extends the directory by doubling
the directory nodes in
virtual
memory,
extendible hashing offers the advantage of being able to
access the directory with
hashing uses
incur
no more than
a single
page
linked structure for the directory,
more than one page
fault to
move through
it
fault.
may
Since
dynamic
be necessary to
the directory.
11.6.2 Linear Hashing
The key
feature of both extendible hashing and
dynamic hashing
is
that they
use a directory to direct access to the actual buckets containing the key
modify the hashed
After expanding
the directory, more than one directory node can point to the same bucket.
However, the directory adds an additional layer of indirection which, if the
directory must be stored on disk, can result in an additional seek.
Linear hashing, introduced by Litwin in 1980, does away with the
directory. An example, developed in Fig. 11.22, shows how linear hashing
records. This directory
makes
it
possible to expand and
address space without expanding the
number of buckets:
ALTERNATIVE APPROACHES
(a)
00
01
10
11
FIGURE 11.22 The growth of
address space in linear hashing. Adapted from Enbody
and Du (1988).
w
i
(b)
000
(c)
000
10
11
100
001
10
11
100
101
100
101
110
b
1
I
X
i i
(d)
000
001
010
d
1 1
z
i k
(e)
000
001
010
Oil
100
1( )1
110
D
111
531
532
EXTENDIBLE HASHING
works. This example
Enbody and Du
is
adapted from
a description
Linear hashing, like extendible hashing, uses
address space of four, which means that
with two
that produces addresses
that
we
developed
a
more
grows. The example begins
as the address space
key and
of linear hashing by
(1988).
earlier in this
second argument of 2.
h 2 (k) address function.
Note
we
of hashed value
bits
(Fig.
11.22a) with an
are using an address function
of depth. In terms of the pseudocode
chapter, we are calling make_address with a
For this example we will refer to this as the
bits
that the address space consists of four buckets,
rather than four directory nodes that can point to buckets.
As we add records, bucket b overflows. The overflow forces a split.
However, as Fig. 11.22(b) shows, it is not bucket b that splits, but bucket
a. The reason for this is that we are extending the address space linearly, and
bucket a is the next bucket that must split to create the next linear extension,
which we call bucket A. A three-bit hash function, h 3 (k), is applied to
buckets a and A to divide the records between them. Since bucket b was not
the bucket that
we
split,
the overflowing record
is
placed into an overflow
bucket w.
We add
more records, and bucket d overflows. Bucket b is the next one
and extend the address space, so we use the h 3 (k) address function
to divide the records from bucket b and its overflow bucket w between b and
the new bucket B. The record overflowing bucket d is placed in an overflow
bucket x. The resulting arrangement is illustrated in Fig. 11.22(c).
Figure 11.22(d) shows what happens when, as we add more records,
bucket d overflows beyond the capacity of the overflow bucket w. Bucket
to split
c is
the next in the extension sequence, so
to divide the records
Finally,
in
assume
between
that bucket
the overflow bucket z.
we
use the h 3 (k) address function
and C.
overflows.
The overflow
The overflow record
is
placed
also triggers the extension to
and y between buckets d and D. At
this point all of the buckets use the h 3 (k) address function, and w e have
finished the expansion cycle. The pointer for the next bucket to be split
bucket D, dividing the contents ofd,
x,
returns to bucket a to get ready for a
function to reach
new
new
cycle that will use an h 4 (k) address
buckets.
Since linear hashing uses
two hash
functions to reach the buckets during
an expansion cycle, an h d (k) function for the buckets
at
the current address
depth and an h d+l (k) function for the expansion buckets, finding
requires
knowing which function
to use. \{p
is
record
the pointer to the address of
the next bucket to be split and extended, then the procedure for finding the
address of the bucket containing
key k
is
as follows:
ihjk)> = P
address
h d (k)
ALTERNATIVE APPROACHES
533
else
address
hd +
(k)
Litwin (1980) shows that the access time performance of linear hashing
is no directory to access or maintain, and since we
extend the address space through splitting every time there is overflow, the
is
quite good. There
overflow chains do not become very large. Given a bucket size of 50, the
average number of disk accesses per search approaches very close to one.
Space utilization, on the other hand, is lower than it is for extendible
hashing or dynamic hashing, averaging around only 60%.
11.6.3 Approaches to Controlling Splitting
We know
from Chapter 8 that we can increase the storage capacity of
by implementing measures that tend to postpone splitting,
redistributing keys between pages rather than splitting pages. We can apply
B-trees
similar logic to the hashing schemes introduced in this chapter, placing
records in chains of overflow buckets to postpone splitting.
Since linear hashing has the lowest storage utilization of the schemes
introduced here, and since
buckets,
its
it is
already includes logic to handle overflow
it
an attractive candidate for use of controlled splitting logic. In
uncontrolled-splitting form, linear hashing splits
bucket and extends
the address space every time any bucket overflows.
triggering event for splitting
the bucket that splits
is
is
arbitrary, particularly
utilization
Suppose we
reaches
some
let
when we consider that
we
split a
such
75%
and even 85%, the
We
still
average number of
stays
as
an alternative
below
as
75%. Every time
the
bucket and extend the address
space. Litwin simulated this kind of system and
unsuccessful searches
file
the buckets overflow until the space
desired figure,
utilization exceeds that figure,
of
typically not the bucket that overflows. Litwin
(1980) suggests using the overall load factor of the
triggering event.
This choice of
found
that for load factors
accesses for successful and
2.
can also use overflow buckets to defer splitting and increase space
utilization for
which use
attraction
dynamic hashing and extendible hashing. For
these methods,
directories to the buckets, deferring splitting has the additional
of keeping the directory
size
down. For extendible hashing
it
is
particularly advantageous to chain to an overflow bucket and therefore
avoid
a split
when
the split
would cause
the directory to double in size.
Consider the example that we used early in this chapter, where we split the
bucket B in Fig. 11.4(b), producing the expanded directory and bucket
structure shown in Fig. 11.6(c). If we had allowed bucket B to overflow
instead, we could have retained the smaller directory. Depending on how
much space we allocated for the overflow buckets, we might also have
534
EXTENDIBLE HASHING
improved space
utilization
ments, of course,
is
among
the buckets.
The
cost of these
improvedue to the overflow
a potentially greater search length
chains.
Studies of the effects of different overflow bucket sizes and chaining
mechanisms has supported a small industry of academic research during the
early and mid-1980s. Larson (1978) suggested the use of deferred splitting in
his original paper on dynamic hashing but found the results of some
preliminary simulations of the idea to be disappointing.
developed
refinement of
this idea in
Scholl (1981)
which overflow buckets
are shared.
by Chang (1985) tested Scholl's suggestions
empirically and found that it was possible to achieve storage utilization of
Master's
about
thesis
81%
research
while maintaining search performance in the range of
1.1 seeks
per search. Veklerov (1985) suggested using buddy buckets for overflow
rather
than
allocating
chains
of
new
buckets.
This
is
an
attractive
since splitting buckets without buddies can never cause a
suggestion,
doubling of the directory in extendible hashing. Veklerov obtained storage
of about 76% with a bucket size of 8.
utilization
SUMMARY
Conventional,
dynamic, that
static
hashing does- not adapt well to
grow and
file
structures that are
shrink over time. Extendible hashing
is
one of
several hashing systems that allow the address space for hashing to
grow
and shrink along with the file. Because the size of the address space can
grow as the file grows, it is possible for extendible hashing to provide
hashed access without the need for overflow handling, even as files grow
many
times beyond their original expected
The key
to extendible hashing
is
size.
the idea of using
more
bits
of the
hashed value as we need to cover more address space. The model for
extending the use of the hashed value is the trie: Every time we use another
bit of the hashed value, we have added another level to the depth of a trie
with a radix of two.
In extendible hashing we fill out all the leaves of the trie until we have
a perfect tree,
The
and then
we
collapse that tree into a one-dimensional array.
array forms a directory to the buckets, kept on disk, that actually hold
is managed in RAM, if possible.
no room for it in a bucket, we split the
bucket. We use one additional bit from the hash values for the keys in the
bucket to divide the keys between the old bucket and the new one. If the
the keys and records.
If
we add
The
directory
record and there
is
address space represented in the directory can cover the use of this
new
bit,
KEY TERMS
no more changes
however, the address space is using
we double the
address space to accommodate the use of the new bit.
fewer
are necessary.
If,
than are needed by our splitting buckets, then
bits
Deletion reverses the addition process, recognizing that
combine the records
is
for
two buckets only
if
of buckets that resulted from
to say that they are the pair
Access performance for extendible hashing
directory can be kept in
it is
RAM.
possible to
they arc buddy buckets, which
If the directory
is
a split.
single seek
must be paged off
if
the
to disk,
is two seeks. Space utilization for the buckets is
approximately 69%. Tables and an approximation formula developed by
Flajolet (1983) permit estimation of the probable directory size, given a
worst-case performance
bucket
and
size
There are
number of records.
number of other approaches
total
extendible hashing.
Dynamic hashing uses
to the
problem solved by
very similar approach but
The
more cumbersome but grows more smoothly. Space
expresses the directory as a linked structure rather than as an array.
linked structure
utilization
is
and seek performance for dynamic hashing are the same
as for
extendible hashing.
Linear hashing does
address space
overflow of
away with
the directory entirely, extending the
by adding new buckets
in a linear sequence.
Although the
bucket can be used to trigger extension of the address space
in linear hashing, typically the bucket that overflows is not the one that is
a
and extended. Consequently, linear hashing implies maintaining
overflow chains and a consequent degradation in seek performance. The
degradation is slight, since the chains typically do not grow to be very long
before they are pulled into a new bucket. Space utilization is about 60%.
Space utilization for extendible, dynamic, and linear hashing can be
improved by postponing the splitting of buckets. This is easy to implement
for linear hashing, since there are already overflow buckets. Using deferred
splitting, it is possible to increase space utilization for any of the hashing
split
schemes described here to 80% or better while still maintaining search
performance averaging less than two seeks. Overflow handling for these
approaches can use the sharing of overflow buckets.
KEY TERMS
Buddy
bucket. Given a bucket with an address uvwxy, where u, v, w,
or 1, the buddy bucket, if it exists,
and y have values of either
has the value uvwxz, such that
x,
XOR
1.
535
536
EXTENDIBLE HASHING
Buddy
buckets are important in deletion operations for extendible
hashing since,
if
enough keys
are deleted, the contents of
buddy
buckets can be combined into a single bucket.
Deferred splitting.
It is
possible to
improve space
utilization for dy-
namic hashing, extendible hashing, and linear hashing by postponing, or
deferring, the splitting of buckets, placing records into overflow
buckets instead. This
is
a classic
space/time trade-off in which
accept diminished performance in return for
more compact
we
storage.
Directory. Conventional, static hashing schemes transform a key into a
bucket address. Both extendible hashing and dynamic hashing introduce
an additional layer of indirection, in which the key is hashed to a
directory address.
The
directory, in turn, contains information about
the location of the bucket. This additional indirection
ble to extend the address space
by extending the
makes
it
possi-
directory, rather
work with an address space made up of buckets.
hashing. Used in a generic sense, dynamic hashing can refer to
any hashing system that provides for expansion and contraction of
the address space for dynamic files where the number of records
changes over time. In this chapter we use the term in a more specific
sense to refer to a system initially described by Larson (1978). The
system uses a directory to provide access to the buckets that actually
contain the records. Entries in the directory can be used as root
nodes of trie structures that accommodate greater numbers of buckets
than having to
Dynamic
as
buckets
split.
Extendible hashing. Like dynamic hashing, extendible hashing is sometimes used to refer to any hashing scheme that allows the address
space to grow and shrink so it can be used in dynamic file systems.
Used more
precisely, as
it is
used in
this chapter, extendible hashing
files that was
proposed by Fagin, Nievergelt, Pippenger, and Strong (1979).
Their proposal is for a system that uses a directory to represent the
address space. Access to buckets containing the records is through
the directory. The directory is handled as an array; the size of the
array can be doubled or halved as the number of buckets changes.
Linear hashing. An approach to hashing for dynamic files that was first
proposed by Litwin (1980). Unlike extendible hashing and dynamic
refers to
an approach to hashed retrieval for dynamic
first
hashing, linear hashing does not use a directory. Instead, the actual
is extended one bucket at a time as buckets overflow.
Because the extension of the address space does not necessarily correspond to the bucket that is overflowing, linear hashing necessarily
involves the use of overflow buckets, even as the address space ex-
address space
pands.
EXERCISES
Splitting.
new
The hashing schemes described
chapter
in this
records by splitting buckets to form
new
make room
tending the address space to cover these buckets. Conventional,
hashing schemes rely
strictly
for
buckets, and then exstatic
on overflow buckets without extending
the address space.
Trie.
key
search tree structure in which each successive character of the
used to determine the direction of the search at each successive
of the tree. The branching factor (the radix of the trie) at any
is potentially equal to the number of values that the character
is
level
level
can take.
wmmmmmmmam
EXERCISES
ex
1.
Briefly describe the differences
hashing, and linear hashing.
What
between extendible hashing, dynamic
and weaknesses of each
are the strengths
approach?
2.
The
tries
that
are
the basis
change
3.
if
we
use
a larger
procedure
the extendible hashing
for
described in this chapter have a radix of two.
How
does performance
radix?
what would happen if we did not reverse
number of low-order
same left-to-right order that they occur in the address? Think
way the directory location would change as we extend the
In the make_address function,
the order of the bits but just extracted the required
bits in the
about the
implicit trie structure to use yet another bit.
language that you are using to implement the make_address
function does not support bit shifting and masking operations, how could
you achieve the same ends, even if less elegantly and clearly?
4.
If the
we
between the original
for this
implementation
bucket and a new one. Outline a possible
redistribution. How do you decide whether a key belongs in the new bucket
5.
In the bk_split function,
redistribute keys
or the original bucket?
6.
Suppose the redistribution of keys
in bk_split
any keys into the new bucket. Under
happen? How will the program handle
does not result
in
moving
what conditions could such an event
7.
we
The bk_try_combine function
described
situation in
is
this?
potentially recursive. In section 11.4.4
which there
combined with other buckets through
are
a
empty buckets
series
that can be
of recursive
calls
to
537
538
EXTENDIBLE HASHING
two
bk_try_combine. Describe
in the
8.
produce empty buckets
Deletion occasionally results in collapsing the directory. Describe the
conditions that must be
9.
situations that could
hash structure.
met before
the directory can collapse.
Deletion depends on finding buddy
buckets.
Why
does the address depth
have to be the same as the address depth for the directory
bucket to have a buddy?
tor a bucket
order for
10. In the extendible hashing
procedure described in
directory can occasionally point to
that can
produce empty buckets.
empty
How
this
buckets. Describe
could
we modify
chapter,
two
in
the
situations
the procedures to
avoid empty buckets?
11. If buckets are large, a
less
wasteful than an
bucket containing only
empty bucket.
How
could
few records
we minimize
is
not
much
nearly empty
buckets?
makes use of overflow records. Assuming an unconimplementation where we split and extend the address
space as soon as we have an overflow, what is the effect of using different
bucket sizes for the overflow buckets? For example, consider overflow
buckets that are as large as the original buckets. Now consider overflow
buckets that can only hold one record. How does this choice affect
performance in terms of space utilization and access time?
12. Linear hashing
trolled splitting
13. In section
we
11.6.3
described an approach to linear hashing that
controls splitting. For a load factor of 85%, the average
for a successful search
is
number of accesses
1.20 (Litwin, 1980). Unsuccessful searches require
an average of 1.78 accesses.
Why
is
the average search length greater for
unsuccessful searches?
Because linear hashing splits one bucket at a time, in order, until it has
reached the end of the sequence, the overflow chains for the last buckets in
the sequence can become much longer than those for the earlier buckets.
Read about Larson's approach to solving this problem through the use of
"partial expansions," originally described in Larson (1980) and subsequently summarized in Embody and Du (1988). Write a pseudocode
description of linear hashing with partial expansions, paying particular
14.
attention to
how
addressing
15. In section 11.6.3
splitting
of buckets
utilization.
What
larger ones?
is
How
we
is
handled.
discussed different mechanisms for deferring the
in extendible hashing in order to increase
storage
the effect of using smaller overflow buckets rather than
does the use of smaller overflow buckets compare with
the idea of sharing overflow buckets?
FURTHER READINGS
Programming Exercises
Write
16.
version of the make_address function that prints out the input
key, the hash value, and the extracted, reversed address. Build a driver that
allows you to enter keys interactively for this function and see the results.
Study the operation of the function on different keys.
Write
17.
in
a simplified version
pseudocode
Keep
Hold
of the extendible hashing program described
in this chapter. This simplified version should
the directory and buckets in
RAM
rather than
on
disk;
three keys per bucket;
Find and add keys, but not delete them;
Accept keys entered interactively; and
Display the resulting directory structure and buckets so you can see
how the directory references the buckets and can see which buckets
contain which keys.
Once you
as
buckets
it to see how the directory grows
program developed in exercise 16 to develop
hash to the same bucket. Enter such sequences and
build this program, play with
split.
Use
sequences of keys that
the
all
watch what happens.
Extend exercise 17 to include deletion. Once again, experiment with
program to see how deletion works. Try deleting all the keys. Try to
create situations where the directory will recursively collapse over more
18.
the
than one level.
19.
Write an extendible hashing program that stores and retrieves buckets
from disk rather than from
RAM.
Using the information in Enbody and Du (1988) and Litwin
implement a simple, RAM-based linear hashing program.
20.
(1980),
FURTHER READINGS
For information about hashing for dynamic
here,
you must turn
to journal
articles.
files
The
beyond what we present
summary of the different
article titled "Dynamic Hashing
that goes
best
is Enbody and Du's Computing Surveys
Schemes," which appeared in 1988.
The original paper on extendible hashing is "Extendible Hashing A Fast
Access Method for Dynamic Files" by Fagin, Nievergelt, Pippenger, and Strong
(1979). Larson (1978) introduces dynamic hashing in an article titled "Dynamic
Hashing." Litwin's initial paper on linear hashing is titled "Linear Hashing: A New
Tool for File and Table Addressing" (1980). All three of these introductory articles
approaches
539
540
EXTENDIBLE HASHING
are quite readable; Larson's paper
especially
and Fagin, Nievergelt, Pippcnger, and Strong are
recommended.
Michel Scholl's 1981 paper titled "New File Organizations Based on Dynamic
Hashing" provides another readable introduction to dynamic hashing. It also
investigates implementations that defer splitting by allowing buckets to overflow.
Papers analyzing the performance of dynamic or extendible hashing often
derive results that apply to either of the
careful analysis
of directory depth and
two methods.
size.
Flajolet (1983) presents a
Mendelson
(1982) arrives at similar
and goes on to discuss the costs of retrieval and deletion as different design
parameters are changed. Veklerov (1985) analyzes the performance of dynamic
hashing when splitting is deferred by allowing records to overflow into a buddy
results
bucket. His results can be applied to extendible hashing as well.
number of papers
building
ideas associated with linear hashing. His 1980 paper titled "Linear
Hashing
After introducing dynamic hashing, Larson wrote a
on the
with Partial Expansions" introduces an approach to linear hashing that can avoid the
uneven distribution of the lengths of overflow chains across the cells in the address
He followed up with a performance analysis in a 1982 paper titled
"Performance Analysis of Linear Hashing with Partial Expansions." A subsequent,
Handling by Linear Probing"
1985 paper titled "Linear Hashing with Overflow
introduces a method of handling overflow that does not involve chaining.
space.
Appendix A
File Structures
on
CD-ROM
OBJECTIVES
Introduce the commercially important characteristics
of
CD-ROM
storage.
medium with performance charvery different from those o{
magnetic disks; show how to apply good file structure design principles to develop solutions that are
appropriate to this new medium.
Examine
storage
acteristics that are
Describe the directory structure of the
system and show how
acteristics of the medium.
file
it
CD-ROM
grows from the char-
541
OUTLINE
A.l Using this Appendix
A.2 Introduction to
A. 2.1
A. 2. 2
A. 5. 2 Block Size
A. 5. 3 Special Loading Procedures and
Other Considerations
A. 5. 4 Virtual Trees and Buffering
CD-ROM
CD-ROM
Short History of
CD-ROM
as a File Structure
Blocks
A. 5. 5 Trees as Secondary Indexes on
Problem
A.3 Physical Organization of
CD-ROM
CD-ROM
A.6 Hashed
A. 3.1 Reading Pits and Lands
A.3.2 CLV Instead of CAV
A. 3. 3 Addressing
A. 3. 4 Structure of a Sector
A.4
CD-ROM
1
A. 4. 2
A. 4. 3
A. 4. 4
A. 4. 5
on
CD-ROM
A. 6.1 Design Exercises
A. 6. 2 Bucket Size
A. 6.3 How the Size of
Helps
A.6.4 Advantages of
Strengths and
Weaknesses
A.4.
Files
Read-Only
Seek Performance
Data Transfer Rate
Storage Capacity
Read-Only Access
Asymmetric Writing and
Reading
A.5 Tree Structures on
A.7 The
CD-ROM
CD-ROM
CD-ROM's
Status
File
System
A. 7.1 The Problem
A. 7. 2 Design Exercise
A. 7. 3 A Hybrid Design
CD-ROM
A. 5.1 Design Exercises
A.1
Using this Appendix
This appendix has two purposes.
performance
characteristics
of
The
first
CD-ROM,
is
information distribution medium. The second
designing
file
structures for
and techniques presented
CD-ROM
to
review
to
tell
you about
commercially
is
to use the
many of the
the
important
problem of
design issues
in the text.
We begin by introducing CD-ROM. We explain how CD-ROM
works and enumerate the features that make file structure design for
CD-ROM a different problem than file structure design for magnetic
media.
Once we have examined CD-ROM's performance, we provide
high-level look at
how this
performance
affects the
design of tree structures,
hashed indexes, and directory structures for CD-ROM. These discussions
of trees and hashing do not present new information; they review material
that has already been developed in detail. Since you already have the tools
543
INTRODUCTION TO CD-ROM
required to think through these design problems,
we
introduce exercises
and questions throughout this discussion, rather than holding them to the
end. We encourage you to stop at these blocks of questions, think carefully
about the answers, and then compare results with the discussion that
follows.
A.2
CD-ROM
Introduction to
CD-ROM
CD audio
CD-ROM
is
an acronym for
disc
is
contains
that
digital
data
than
rather
commercially interesting because
can be reproduced cheaply.
That
Compact Disc Read-Only Memory.
it
can hold
digital
1"
It
is
sound.
of data and
megabytes of
a lot
single disc can hold over 600
approximately 200,000 printed pages, enough storage to hold
size of this one. Replicates can be stamped from a
master disc for about only a dollar a copy.
data.
is
almost 400 books the
CD-ROM
is
cannot record on
read-only in the same sense as
It
it.
many
is
CD
publishing medium,
audio
disc:
You
used for distributing
and retrieval medium
magnetic disks. Currently, CD-ROMs are often used to publish
database information such as telephone directories, zip codes, and demographic information. There are also many CD-ROM products that deliver
textual data, such as bibliographic indexes, abstracts, dictionaries, and
encyclopedias, often in association with digitized images stored on the disc.
They are also used to publish video information and, of course, digital
information to
users, rather than a data storage
like
audio.
A.2.1 A Short History of
CD-ROM
is
CD-ROM
the offspring of videodisc technology developed in the late
1960s and early 1970s, before the advent of the
to store
movies on
methods
for storing video signals, including
disc.
respond mechanically to grooves
does.
home VCR. The
Different companies developed
The consumer products
in a disc,
one that used
much
industry spent
like a vinyl
great
deal
goal
was
number of
needle to
LP record
of money
developing the different technologies, including several approaches to
optical storage, and then spent years fighting over the question of which
approach should become standard. The surviving format is one called
LaserVision. By the time LaserVision emerged as the winner of these
""Usually
spell
it
we
with
spell disk
a
c.
with
a k,
but the convention
among
optical disc manufacturers
is
to
544
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
standards battles, the competing developers had not only spent enormous
sums of money, but had also lost important market opportunities. These
hard lessons were put to use in the subsequent development of CD audio
and
CD-ROM.
From the outset, there was interest in using LaserVision discs to do
more than just record movies. The LaserVision format supports recording
in
both
CLV
capacity, and a
(Constant Linear Velocity) format that maximizes storage
CAV
seek performance.
frames quickly,
(Constant Angular Velocity) format that enables fast
using the
format to access individual video
CAV
By
number of organizations,
produced prototype interactive video
including the
MIT Media
Lab,
discs that could be used to teach
and
entertain.
In the early 1980s, a
number of firms began looking
at the possibility
of
storing digital, textual information on LaserVision discs. LaserVision stores
data in an analog form;
it
is,
after
storing an analog video signal.
all,
came up with different ways of encoding digital information
analog form so it could be stored on the disc. The capabilities
Different firms
in
demonstrated in the prototypes and early, narrowly distributed products
were impressive. The videodisc has a number of performance characteristics
that
make
it
a technically
more
desirable
medium
than the
CD-ROM;
in
one can build drives that seek quickly and deliver information
from the disc at a high rate of speed. But, reminiscent of the earlier disputes
over the physical format of the videodisc, each of these pioneers in the use
of LaserVision discs as computer peripherals had incompatible encoding
schemes and error correction techniques. There was no standard format,
and none of the firms was large enough to impose their format over the
others through sheer marketing muscle. Potential buyers were frightened
by the lack of a standard; consequently, the market never grew.
During this same period the Philips and Sony companies began work
on a way to store music on optical discs. Rather than storing the music in
the kind of analog form used on videodiscs, they developed a digital data
format. Philips and Sony had learned hard lessons from the expensive
particular,
standards battles over videodiscs. This time they
worked with other
in the consumer products industry to develop
players
licensing system that
emergence of CD audio discs as a broadly accepted, standard
format as soon as the first discs and players were introduced. CD audio
appeared in the United States in early 1984. CD-ROM, which is a digital
data format built on top of the CD audio standard, emerged shortly
thereafter. The first commercially available CD-ROM drives appeared in
resulted in the
1985.
Not
surprisingly,
LaserVision discs
first
the
saw
firms
that
CD-ROM
were delivering
digital
data
as a threat to their existence.
on
They
545
INTRODUCTION TO CD-ROM
however,
also recognized,
always eluded them
CD-ROM
CD-ROM
that
in the past: a
was guaranteed
manufactured by any
drive
of any disc
promised
what had
to provide
standard physical format.
Anyone with
that they could find and read a sector off
medium
firm. For a storage
such
to be used in
fundamental level is essential.
What happened next is remarkable in the history of standards and
cooperation within an industry. The firms that had been working on
products to deliver computer data from videodiscs recognized that a
publishing, standardization
at
CD-ROM, was not
standard physical format meant that everyone was guaranteed to
standard physical format, such as that provided by
enough.
be able to read sectors off of any
work
disc.
But computer applications do not
terms of sectors; they store data in files. Having an agreement
about finding sectors, without further agreement about how to organize the
in
sectors into
settled
on
files, is like
how
relatively
moving
small,
into the
system to be
be organized into words on
from the
the firms emerging
were
everyone's agreeing on an alphabet without having
letters are to
together
called
CD-ROM
built
industry
on top of the
many of the much
to begin work on a
CD-ROM
format. In
file
now emerged
as
official international
of which
standard
file
display of
a rare
system standard by early summer of 1986;
an
all
larger firms
worked out
cooperation, the different firms, large and small,
features of a
page. In late 1985
videodisc/digital data industry,
that
the
main
work
standard for organizing
has
on
files
CD-ROM.
The
has
CD-ROM
begun
to
industry
show mature
is still
matters such as disc formats to
in the past two years it
moving away from concentration on
young, though
signs of
concern with
CD-ROM
applications;
on the new medium in isolation, vendors are seeing
an enabling mechanism for new systems. As it finds more uses in
rather than focusing
as
broader array of applications,
CD-ROM
it
looks like an optical publishing
technology that will be with us over the long term.
A. 2.
CD-ROM
CD-ROM
as a File Structure Problem
presents
medium with
include the fact that
durable.
slow,
interesting
it
structure problems
The key weakness
often taking
and showed
The
from
is
that if
that
is
seek performance on
RAM
access
is
RAM
it
is
CD-ROM
inexpensive, and
a
half second to a second
we compared
because
strengths of
has a lot of storage capacity,
introduction to this textbook
access
file
great strengths and weaknesses.
CD-ROM
per seek.
is
In
is
very
the
access and magnetic disk
analogous to your taking 20
seconds to look up something in the index to this textbook, the equivalent
disk access would take 58 days, or almost two months. With a
CD-ROM
546
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
the analogy stretches the disc access to over two and a half years! This kind
of performance, or lack of it, makes intelligent file structure design a critical
concern for CD-ROM applications. CD-ROM provides an excellent test of
our ability to integrate and adapt the principles we have developed in the
preceding chapters of this book.
A.3
Physical Organization of
CD-ROM
is
is
the child of
CD
CD-ROM
audio. In this instance, the impact of heredity
strong, with both positive and negative aspects. Commercially, the
audio parentage
the market.
it is
It is
possible to
CD
probably wholly responsible for CD-ROM's viability in
because of the enormous size of the CD audio market that
is
make
CD-ROM discs so inexpensively.
Similarly, advances
and decreases in the costs of making CD audio players affect
performance and price of CD-ROM drives. Other optical disc media that
have not enjoyed the benefits of this parentage have not experienced the
commercial success of CD-ROM.
On the other hand, making use of the manufacturing capacity
associated with CD audio means adhering to the fundamental physical
organization of the CD audio disc. Audio discs are designed to play music,
in the design
not to provide
fast,
random
access to data.
This difference in design
objective biases
CD toward having high storage capacity and moderate data
transfer
but against decent seek performance.
requires
rates,
good random-access performance,
from our
in the
file
structure design efforts;
medium
it
If
an application
performance has to emerge
won't come from anything inherent
that
itself.
A.3.1 Reading Pits and Lands
CD-ROM
stamped from a master disc. The master is formed by
we want to encode to turn a powerful laser on and
off very quickly. The master disc, which is made of glass, has a coating that
is changed by the laser beam. When the coating is developed, the areas hit
by the laser beam turn into pits along the track followed by the beam. The
smooth, unchanged areas between the pits are called lands. The copies
formed from the master retain this pattern of pits and lands.
When we read the stamped copy of the disc, we focus a beam of laser
light on the track as it moves under the optical pickup. The pits scatter the
light, but the lands reflect most of it back to the pickup. This alternating
pattern of high- and low-intensity reflected light is the signal used to
reconstruct the original digital information. The encoding scheme used for
discs are
using the digital data that
547
PHYSICAL ORGANIZATION OF CD-ROM
this signal is
not simply
the Is are represented
matter of calling
by the
transitions
a pit a
from
and
pit to
land
a 0. Instead,
land and back again.
Every time the light intensity changes, we get a 1. The zeroes arc
represented by the amount of time between transitions; the longer between
more zeroes we have.
you think about this encoding scheme, you realize that it is not
possible to have two adjacent Is
Is are always separated by zeroes. In fact,
due to the limits of the resolution of the optical pickup, there must be at
least two 0s between any pair of Is. This means that the raw pattern of Is
transitions, the
If
and 0s has
to be translated in order to get the 8-bit patterns
of
Is
and
()s
that
form the bytes of the original data. This translation scheme, which is done
through a lookup table, turns the original 8 bits of data into 14 expanded
bits that can be represented in the pits and lands on the disc; the reading
process reverses this translation. Figure A.l shows a portion of the lookup
table values. Readers
players
may have
who
have looked closely
encountered the term
EFM
at
the specifications for
encoding.
EFM
CD
stands for
"eight to fourteen modulation" and refers to this translation scheme.
It is
important to
realize that since
we
represent the zeroes in the
EFM-
the length of time between transitions, our ability to read the
encoded data by
data is dependent on moving the pits and lands under the optical pickup at
a precise and constant speed. As we will see, this affects the CD-ROM
drive's ability to seek quickly.
A.3.2 CLV Instead
Data on
CD-ROM
three miles
of
is
CAV
stored in a single, spiral track that winds for almost
from the center
to the outer
edge of the
disc.
This spiral pattern
which
from CD
requires a lot of storage space, we want to pack the data on the disc as
tightly as possible. Since we "play" audio data, often from start to finish
is
part of the
CD-ROM's
heritage
Decimal Original Translated
bits
value
bits
00000000 01001000100000
00000001 10000100000000
2
00000010 10010000100000
3
00000011 10001000100000
4
00000100 01000100000000
5
00000101 00000100010000
6
00000110 00010000100000
7
00000111 00100100000000
8
00001000 01001001000000
1
audio. For audio data,
FIGURE A. 1 A portion
EFM encoding table.
of the
548
APPENDIX
STRUCTURES ON CD-ROM
A: FILE
Constant angular
velocity
Constant linear
velocity
FIGURE A.2 CLV and
CAV
recording.
without interruption, seeking is not important. As Fig. A.2 shows, a spiral
pattern serves these needs well. A sector toward the outer edge of the disc
takes the same amount of space as a sector toward the center of the disc.
This means that we can write all of the sectors at the maximum density
permitted by the storage medium. Since reading the data requires that it
pass under the optical pickup device at a constant rate, the constant data
density implies that the disc has to spin more slowly when we are reading
at the outer edges than when we are reading toward the center. This is why
the spiral is a Constant Linear Velocity (CLV) format: As we seek from the
center to the edge, we change the rate of rotation of the disc so the linear
speed of the spiral past the pickup device stays the same.
By contrast, the familiar Constant Angular Velocity (CAV) arrangement shown in Fig. A.2, with its concentric tracks and pie-shaped sectors,
writes data less densely in the outer tracks than in the tracks toward the
center. We are wasting storage capacity in the outer tracks but have the
advantage of being able to spin the disc at the same speed for all positions
of the read head. Given the sector arrangement shown in the figure, one
rotation
reads
Furthermore,
start
of
no matter where we
timing mark placed on the disk makes
eight
sectors,
are
it
on the
disc.
easy to find the
a sector.
The CLV format is responsible, in large part, for the poor seeking
performance of CD-ROM drives. The CAV format provides definite track
boundaries and a timing mark to find the start of a sector. The CLV format,
549
PHYSICAL ORGANIZATION OF CD-ROM
on the other hand, provides no straightforward way
location. Part of the problem is associated with
rotational speed as
that
is
we
seek across the disc.
adjust the speed,
know where we
we need
to a specific
need to change
we need
the correct speed. But to
to be
moving
know how
to
we
to be able to read the address information so
How
are.
at
jump
read the address information
stored on the disc along with the user's data,
the data under the optical pickup
this
To
to
the
does the drive's control mechanism break out of
loop? In practice, the answer often involves making guesses, finding the
correct speed through
trial
and error. This takes time and slows
down
seek
performance.
On the positive side, the CLV sector arrangement
CD-ROM's large storage capacity. Given a CAV
CD-ROM
A. 3.
would have only
a little better
than half
its
contributes to the
arrangement,
the
present capacity.
3 Addressing
The use of
sector
the
Instead,
CLV
organization means that the familiar cylinder, track,
CD-ROM.
we use a sector-addressing scheme that is related to the CD-ROM's
way of
identifying
sector address will not
work on
roots as an audio playback device. Each second of playing time on
CD
is
divided into 75 sectors, each of which holds 2 Kbytes of data. According to
the original Philips/Sony standard, a
CD-ROM,
disc
is
CD
disc,
whether used
for audio or
contains at least one hour of playing time. That means that the
capable of holding
at least
540,000 Kbytes of data:
60 minutes x 60 seconds/minute x 75 sectors/second = 270,000 sectors.
In fact, since
it is
possible to put over 70 minutes of playing time on
the capacity of the disk
is
CD,
over 600 Mbytes.
We address a given sector by referring to the minute,
second, and sector
of play. So, the 34th sector in the 22nd second in the 16th minute of play
would be addressed with the three numbers 16:22:34.
A. 3. 4 Structure of a Sector
It is
interesting to look at the
way
that the
fundamental design of the
CD
designed for delivering digital audio information, has been
adapted for computer data storage. This investigation will also help answer
the question, "If the disc is capable of storing a quarter of a million printed
pages, why does it hold only an hour's worth of Roy Orbison?"
disc, initially
we need to convert a wave pattern into
wave.
At any given point in time, the
digital form. Figure A. 3
digitize
the wave by measuring the
wave has a specific amplitude. We
When we want
to store sound,
shows
550
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
32767
actual
wave
wave reconstructed
from sample data
sampling frequency
-32767
FIGURE A.3 Digital sampling
amplitude
at
of a
very frequent intervals and storing the measurements. So, the
question of how
turns into
we need to represent a wave digitally
questions: How much space does it take to store each
and how often do we take samples?
much
storage space
two other
amplitude sample,
CD audio uses
that
wave
16 bits to store each amplitude measurement; that
our "ruler" that
different gradations.
we use to measure the height of the wave has
To accurately approximate a wave through
means
65,536
digital
we
need to take the samples at a rate that is more than twice as
frequent as the highest frequency that we want to capture. This makes sense
if you look at the wave in Fig. A. 4. You can see that if we sample at less than
twice the frequency of the wave, we lose information about the variation in
the wave pattern. The designers of CD audio selected a sampling frequency
of 44.1 KHz, or 44,100 times per second, so they could record sounds with
frequencies ranging up to 20 KHz (20,000 cycles per second), which is
toward the upper bound of what people can hear.
So, if we are taking a 16-bit, or 2-byte sample 44,100 times per second,
sampling,
we need to store 88,200 bytes per second. Since we want to store stereo
sound, we need double this, storing 176,400 bytes per second. You can see
why storing an hour of Roy Orbison takes so much space.
551
PHYSICAL ORGANIZATION OF CD-ROM
actual
wave
wave reconstructed
from sample data
*
sampling frequency
FIGURE A.4 The effect of sampling at less than twice the frequency of the wave.
If
you divide
the 176,400-byte-per-second storage capacity of the
into 75 sectors per second,
divides
up
this
"raw"
sector
CD
you have 2,352 bytes per sector. CD-ROM
storage as shown in Fig. A. 5 to provide 2 K of
user data storage, along with addressing information, error detection, and
error correction information.
because, although
CD
The
error correction information
not adequate to meet computer data storage needs.
correction
discs.
would
The
result in an average
additional
error
2,352-byte sector decreases
20,000
12 bytes
necessary
The audio
it is
error
of one incorrect byte for every two
information stored within the
correction
this error rate to
one uncorrectable byte
in
every
discs.
FIGURE A.5 Structure of a
synch
is
audio contains redundancy for error correction,
CD-ROM
4 bytes
sector ID
sector.
2,048 bytes
user data
4 bytes
8 bytes
error
detection
null
276 bytes
error
correction
552
A.4
APPENDIX
STRUCTURES ON CD-ROM
A: FILE
CD-ROM
Strengths and Weaknesses
As we say throughout this book, good file design is responsive to the nature
of the medium, making use of strengths and minimizing weaknesses. We
begin, then, by cataloging the strengths and weaknesses of CD-ROM.
A.4.1 Seek Performance
chief weakness of CD-ROM
Current magnetic disk technology
The
random
the
is
is
random-access performance.
such that the average time for
data access, combining seek time and rotational delay,
is
about 30
On a CD-ROM,
this average access takes 500 msec, and can take up
more. Clearly, our file design strategies must avoid seeks to
an even greater extent than on magnetic disks.
msec.
to a second or
A.4. 2 Data Transfer Rate
A CD-ROM
drive reads 75 sectors, or 150 Kbytes of data per second. This
data transfer rate
is
part of the fundamental definition of
CD-ROM;
it
can't
be changed without leaving behind the commercial advantages ot adhering
to the
CD audio standard.
It is
modest
transfer rate, about five times faster
than the transfer rate for floppy disks, and an order of magnitude slower
than the rate for good Winchester disks.
makes
itself felt
when we
The inadequacy of the
are loading large
files,
such
as
transfer rate
those associated
On the other hand, the transfer rate is fast enough
CD-ROM's seek performance that we have a design
with digitized images.
relative
to
the
incentive to organize data into blocks, reading
with the hope that
we
can avoid as
much
more
seeking
data with each seek
as possible.
A. 4. 3 Storage Capacity
A CD-ROM
to use
up
holds
images, 600 Mbytes
to
more than 600 Mbytes of data. Although
this storage area
is
big
very quickly, particularly
when
download 600 Mbytes of
it
text
it is
possible
if you are storing raster
comes
to text applications. If you decide
with
2,400-baud
modem,
it
will take
about three days of constant data transmission, assuming errorless transmission conditions. Many typical text databases and document collections
published on
CD-ROM
The design
use only
benefit arising
a fraction
from such
of the
disc's capacity.
large capacity
is
that
it
enables us
and other support structures that can help overcome some
of the limitations associated with CD-ROM's poor seek performance.
to build indexes
553
TREE STRUCTURES ON CD-ROM
A. 4. 4
From
Read-Only Access
design standpoint, the fact that
CD-ROM
storage device that cannot be changed
significant advantages.
We
organization.
through
file
structures
to optimize our index structures
We know
later additions
is
publishing medium,
provides
manufacture,
never have to worry about updating. This not
only simplifies some of the
worthwhile
after
but also
means
that
is
it
and other aspects of
file
that our efforts to optimize access will not be lost
or deletions.
Asymmetric Writing and Reading
A. 4. 5
For most media,
files
are written
and read using the same computer system.
Often, reading and writing are both interactive and are therefore con-
by the need
strained
to provide quick response to the user.
CD-ROM
is
We create the files to be placed on the disc once; then we distribute
is accessed thousands, even millions, of times. We are in a
and
different.
the disc
it
position
to
bring
substantial
computing power
to
the
task
of
file
when the disc will be used on systems with
we can use extensive, batch-mode processing
organization and creation, even
much
on
less capability. In fact,
large computers to try to provide systems that will perform well
small machines.
file
We make
on
the investment in intelligent, carefully designed
structures only once; users can enjoy the benefits of this investment
again and again.
A.5
CD-ROM
Tree Structures on
A. 5.1
Design Exercises
Tree structures are a good way to organize indexes and data on CD-ROM.
+
Chapters 8 and 9 took a close look at B-trees and B trees. Before we
discuss the effective use of trees on CD-ROM, think through these design
questions:
1.
2.
3.
How
How
big should the block size be for B-trees and
B+
trees?
should you go in the direction of using virtual tree structures? How much memory should you set aside for buffering blocks?
+
How could you use special loading procedures to advantage in a B
far
implementation? Are there similar procedures that will assist in
the loading of B-trees?
Suppose we have a primary index and several secondary indexes to
tree
4.
set
of records.
How
should you organize these access mechanisms
554
APPENDIX
tor
A: FILE
CD-ROM?
your
A.
STRUCTURES ON CD-ROM
Address the issues of binding and pinned records
in
reply.
5.2 Block Size
Avoiding seeks
Consequently,
is
the key strategy in
+
and
B-tree
tree
CD-ROM
file
structure design.
good choices for
As we showed in Chapters 8
structures
are
implementing index structures on CD-ROM.
and 9, given a large enough block size, B-trees and B + trees can provide
access to a large number of records in only a few seeks.
How large should the block size be? The answer, of course, depends on
the application, but
it is
possible to provide
since the sector size of the
be
less
than 2 Kbytes.
consequently,
it
sector
does not
Since the
CD-ROM's
especially
when viewed
CD-ROM
make
is
is
some
general guidelines. First,
2 Kbytes, the block size should not
sequential reading performance
attractive to use a block
on the
the smallest addressable unit
disc;
sense to read in anything less than a sector.
relative to
its
is
moderately
seeking performance,
composed of several
sectors.
it
is
Once you have
the better part of a second seeking for the sector and reading
it,
fast,
usually
spent
reading an
make an 8-Kbyte block takes only an additional 40
added fraction of a second can contribute to avoiding another
additional 6 Kbytes to
msec. If
this
it is time well spent.
Table A.l shows the maximum number of 32-byte records that can be
contained in a B-tree as the tree changes in height and block size. The
seek,
dramatic effect of block
trees
size
on the record counts
for
two- and
suggests that large tree structures should usually use
three-level
at
least
an
8-Kbyte block.
TABLE
A.
The maximum number
of
32-byte records that can be stored
in
a B-tree of
giver height a nd block size
Tree Height
One
Block
size
Block
size
Block
size
=
=
=
4
8
K
K
K
Level
Two
Levels
Three Levels
64
4,224
274,624
128
16,640
2,146,688
256
66,048
16,974,592
555
TREE STRUCTURES ON CD-ROM
A. 5.3 Special
Loading Procedures and Other Considerations
trees are commonly used in CD-ROM applications because they
provide both indexed and sequential access to records. If, for example, you
are building a telephone directory system for CD-ROM, you will need an
index that can provide fast access to any one of the millions of names that
appear on the
You
will also want to provide sequential access so once
name, they can browse through records with the same
name, checking addresses, to make sure they have the right phone number.
disc.
users have found a
B+
trees are also attractive in
CD-ROM
applications because they can
of sequenced records. As we
+
the content of the index part of a B tree can consist
provide very shallow, broad indexes to
showed
a set
in Chapter 9,
of nothing more than the shortest separators required to provide access to
lower levels of the tree and, ultimately, to the target records. If these
shortest separators are only a few bytes long, as is frequently the case, it is
often possible to provide access to millions of records with an index that is
only tw o levels deep. An application can keep the root of this index in
RAM, reducing the cost of searching the index part of the tree to a single
seek. With one additional seek we are at the record in the sequence set.
+
Another attractive feature of B trees is that it is easy to build a
two-level index above the sequence set with a separate loading procedure
T
from the bottom up. We described this operation in
Chapter 9. The great advantage of this kind of loading procedure, as
opposed to building the tree through a series of top-down insertions, is that
we can pack the nodes and leaves of the tree as fully as we wish. With
CD-ROM, where the cost of additional seeks is so high, and where there is
absolutely no possibility that anyone will make additional insertions to the
tree, we will want to pack the nodes and leaves of the tree so they are
completely full. This is an example of a design decision that recognizes that
the CD-ROM is a publishing medium that, once constructed, is used only
that builds the tree
and never for additional storage.
This kind of special, 100%-full loading procedure can also be designed
for B-tree applications. The procedure for B-trees is usually somewhat
more complex because the index will often consist of more than just a root
for retrieval,
node and one
level
manage more
levels
of children. The loading procedure for B-trees has to
of the tree at a time.
This discussion of indexes, and the importance of packing them as
tightly as possible, brings home one of the interesting paradoxes of
CD-ROM
The
CD-ROM
large
storage
capacity that usually gives us a great deal of freedom with regard to
how we
design.
disc
has
relatively
on the disc; a few bytes here or there usually doesn't matter much
when you have 600 Mbytes of capacity. But when we design the index
store data
556
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
structures for
CD-ROM, we
even counting
bits as
reason for this
is
we pack
not, in
most
find ourselves counting bytes,
information into
cases, that
we
a single
are
sometimes
byte or integer.
The
running out of space on the
but because packing the index tightly can often save us from making
file design the cost of seeks adds up very
quickly; the designer needs to get as much information out of every seek as
disc,
CD-ROM
an additional seek. In
possible.
A.
5.4 Virtual Trees and Buffering Blocks
Given the very high cost of seeking on
blocks in
RAM
CD-ROM, we
node should always be buffered. As we indicated
trees in Chapter 8, buffering nodes below
contribute
will
for as long as they are likely to be useful.
significantly
buffering
is
intelligent
Buffering
is
most
to
in
useful
in
selecting
when
the
node
to
keep
tree's root
our discussion of virtual
root
the
reducing seek time,
want
The
can
particularly
to
sometimes
when
the
replace in the buffer.
successive accesses to the tree tend to be
clustered in one area.
Note
we
that packing the tree as tightly as possible during loading,
discussed earlier as a
way
likelihood that an index block in
to reduce tree height,
which
also increases the
RAM will be useful on successive accesses
to the data.
A. 5.
5 Trees as Secondary Indexes on
Typically,
the data
CD-ROM
on the
disc.
CD-ROM
applications provide
more than one
access route to
For example, document retrieval applications usually
give direct access to the documents, so you can page through them in
them up by name, chapter, or section while also providing
access through an index of keywords or included terms. Similarly, in a
telephone directory application you would have access to the database by
name, but also by location (state, city, zip code, street address). As we
sequence or
call
described in Chapter
6,
secondary indexes provide these multiple views of
the data.
Chapter 5 raised the design issue of whether the secondary indexes
should be tightly bound to the records they point to, or whether the binding
should take place at retrieval time, through the use of a common key
Viewed another way, the issue is
whether the target records should be pinned to a specific location through
references in secondary indexes, or whether they should be left unpinned so
accessed through yet another index.
they can be reorganized.
Records will never be reorganized on a CD-ROM; since it is a
disc, there is no disadvantage to having pinned records. Further,
read-only
HASHED
FILES
ON CD-ROM
557
minimizing the number of seeks is the overriding design consideration on
CD-ROM. Consequently, secondary index designs for CD-ROM should
usually bind the indexes to the target records as tightly as possible, ensuring
that once you have found the correct place in the index, you are ready to
retrieve the target with, at most, one additional seek.
One objection to this bind-tightly approach to CD-ROM index design
is that, although it is true that the indexes cannot be reorganized once
CD-ROM,
written to the
they
are, in fact, quite
frequently reorganized
between successive "editions" of the disc. Many CD-ROM publications are
reissued to keep them up to date. The period between successive versions
may
be years, or
may
be
as short as a
week. So, although pinned records
cause no problem on the finished disc, they
difficulty in the files
used to prepare the
may
cause a great deal of
disc.
There are a number of approaches to resolving this tension between
what is best on the published disc and what is best for the files used to
produce it. One solution is to maintain loosely bound records in the source
database, transforming them to tightly bound records for publication on
CD-ROM. CD-ROM
product designers often
on the
fail
to realize that the
file
and often should, be different than the
structures used to maintain the source data and produce the discs. Another
solution, of course, is to trade off performance on the published disc for
decreased costs in producing it. Production costs, time constraints, user
acceptance, and competitive factors interact to determine which course is
structures placed
best.
The key
issue
from
disc can,
the
file
designer's standpoint
is
to recognize that
the alternatives exist, and then to be able to quantify the costs and benefits
of each.
A.6
Hashed
A. 6.1
Files
on CD-ROM
Design Exercises
Hashing, with
its
promise of single access
organize indexes on
CD-ROM. We begin
retrieval,
is
an excellent
way
to
with some design questions that
your knowledge of hashing with what you now know about
CD-ROM. As you think through your answers, remember that your goal
should be to avoid any additional seeking due to hash bucket overflow. As
in any hashing design problem, the design parameters that you can
intersect
manipulate are
Bucket size;
Packing density for the hashed index; and
The hash
function
itself.
558
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
The following
efficient
which you should try to answer before you
think about ways to use these parameters to build
questions,
you
read on, encourage
CD-ROM
to
applications.
What
2.
3.
considerations go into choosing a bucket size?
does the relatively large storage capacity of
developing efficient hashed retrieval?
How
CD-ROM
Since a
CD-ROM
is
read-only,
you have
hashed before you create the
to be
disc.
complete
How
list
assist in
of the keys
can this assist in reduc-
ing retrieval costs?
Bucket Size
A. 6. 2
Chapter 10 we showed how to reduce overflow, and therefore retrieval
time, by grouping records into buckets, so each hashed address references an
entire bucket of records. Since any access to a CD-ROM always reads in a
minimum of a 2-Kbyte sector, the bucket size should be a multiple of 2
Kbytes. Having the bucket be only a part of a sector would be
counterproductive. As we described in Chapter 3, transferring anything less
than a sector means first moving the data into a system buffer, and from
there into the user's data area. With transfers ot a complete sector, many
operating systems can move the data directly into the user area.
How many sectors should go into a bucket? As with trees, it is a
trade-off between seeking and sequential reading. In addition, larger
buckets require more searching and comparing to find the record once the
bucket is read into RAM. In Chapter 10 we provided tools to allow you to
calculate the effect of bucket size on the probability of overflow. For
In
CD-ROM
you
applications,
will
want
to use these tools to reduce the
probability of overflow to almost nothing.
How
A.6.3
Packing
the Size of
hashed
additional seeking.
bucket
we
loosely
good
rule
is
Helps
another
of thumb
is
way
that,
all
see that for
to
avoid overflow and
even with only
keeping the packing density below
size,
overflow almost
10,
file
CD-ROM
60%
moderate
will tend to avoid
the time. Consulting Tables 10.4 and 10.5 in Chapter
randomly
distributed keys, a packing density of 60% and
of 10 will reduce the percentage of records that overflow to
1.3% and will reduce the average number of seeks required for a successful
bucket
size
search to 1.01.
When
there
disadvantage to
unused space available on the disc, there is no
expanding the size of the hashed index so overflow is
is
virtually eliminated.
THE CD-ROM
A. 6. 4
Advantages
of
CD-ROM
collections often use
more
the
is
CD-ROM
90%
full?
discs, this situation
most of the
images along with the
digitized
559
SYSTEM
CD-ROM's Read-Only Status
What if space is at a premium on
a way to pack your index so it
capacity of
FILE
is
disc,
and you need to find
Despite the relatively large
fairly
common. Large
disc just for text. If the product
text
file
storing
is
the available space disappears even
text,
two discs at once are much
and deliver than a single disc application; when a disc is
already nearly full of data, the index files are always a target for size
quickly. Applications requiring the use of
harder to
sell
reduction.
The
space.
we do
calculations that
packing density assume
If
we
could find
uniformly, rather than
a
a
to estimate the effects of bucket size and
random distribution of keys across the address
hash function that would distribute the keys
randomly,
we
could achieve
100% packing
density
and no overflow.
that
Once again, the fact that CD-ROM is
would not be available in a dynamic,
we produce
CD-ROM, we have all
This means that
we do
expecting
a distribution that is
that
read-write environment.
When
the keys that are to be hashed at hand.
not have to choose
whatever distribution of keys
read-only opens up possibilities
it
hash function and then
settle for
produces, hoping for the best, but
merely random. Instead,
function that provides the performance
we
we
can select
hash
need, given the set of keys
we
have to hash. If our performance and space constraints require it, we can
develop a hash function that produces no overflow even at very high
packing densities. We identify the selected hash function on the disc, along
with the data, so the retrieval software knows how to locate the keys. This
relatively expensive and time-consuming function-fitting effort is worthwhile because of the asymmetric nature of writing and reading CD-ROMs;
the one-time effort spent in making the disc is paid back many times as the
disc
A.7
is
distributed to
The CD-ROM
many
File
A. 7.1
The Problem
When
the
users.
System
firms involved in developing
together to begin
work on
confronted with an interesting
common
file
CD-ROM
applications
came
system in late 1985, they were
structures problem. The design goals and
file
constraints included the following:
Support hierarchical directory structures;
560
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
Find and open any one of thousands of
and
Support the use of generic
with only one or two
files
seeks;
file
names,
as in "file*.c",
during directory
access.
The
way
usual
more than
directories as nothing
you
notation,
support hierarchical directories
to
are looking for a
a special
file
with the
kind of
full
file.
is
If,
to
the
treat
using
UNIX
path
/usr/home/mydir/fi lebook/cdrom/part3. txt
you look
in the root directory
usr to find the location
it
(/)
to find the directory file usr, then
of the directory
file
you open
home, you seek to home and open
and so on until you finally open the directory file named
where you find the location of the target file part3.txt. This is a very
to find mydir,
cdrom,
simple, flexible system;
and many
CD-ROM developer,
must seek
it is
the approach used in
other operating systems.
to,
CD-ROM,
that before
is
we
the standpoint of a
can find the location ofpart3.txt,
open, and use six other
such
MS-DOS, UNIX, VMS,
The problem, from
files.
At
we
half-second per seek on
a directory structure results in a
very unresponsive
file
system.
A. 7.
2 Design Exercise
At the time of the
meetings to begin looking
initial
directory structure and
file
system,
at a
standard
CD-ROM
number of vendors were using
this
magnetic disc directwo alternative approaches
treat-directories-as-files approach, literally replicating
tory systems on
that
ROM. One
child,
left
CD-ROM.
were commercially
There were at least
and more specifically
available
placed the entire directory structure in
right sibling tree to
CD-
building a
express the directory structure. Given the
directory hierarchy in Fig. A. 6, this system produced a
shown
tailored to
a single file,
file
containing the
A. 7. The other system created an index to the file
locations by hashing the full path names of each file. The entries in the hash
tree
in Fig.
table for the directory structure in Fig. A. 6 are
shown
in Fig. A. 8.
Considering what you know about CD-ROM (slow seeking, readonly, and so on), think about these alternative file systems and try to answer
the following questions. Keep in mind the design goals and constraints that
were facing the committee (hierarchical structure, fast access to thousands
of
files,
1.
2.
use of generic
file
names).
and disadvantages of each system.
Try to come up with an alternative approach that combines the best
features of the other systems while minimizing the disadvantages.
List the advantages
THE CD-ROM
FILE
SYSTEM
561
ROOT
REPORTS
LETTERS
CALLS.LOG
WORK
SCHOOL
ZATX
S2.RPT
Sl.RPT
WLRPT
Wl.LTR
P2.LTR
Pl.LTR
FIGURE A.6 A sample directory hierarchy.
FIGURE A.7 Left child, right sibling tree to express directory structure.
ROOT
I
REPORTS
fc
LETTERS
CALLS.LOG
I
PERSONAL
ISC
fr
ABC.LTR
SCHOOL
h WORK
I
I
Sl.RPT
Wl.RPT
S2.RPT
XYZ.LTR
fc
WORK
i
Pl.LTR
b>
P2.LTR
Wl.LTR
562
APPENDIX
A: FILE
STRUCTURES ON CD-ROM
/REPORTS/SCHOOL/S 1 .RPT
/REPORTS/SCHOOL/S2.RPT
/REPORTS /WORK/W1 .RPT
Hash
/LETTERS/PERSONAL/P 1 .LTR
/LETTERS/PERSONAL/P2.LTR
Table
of
Path
Hash Function
/LETTERS/ABC.LTR
Names
/LETTERS/XYZ.LTR
/LETTERS/WORK/W1 .LTR
/CALLS.LOG
FIGURE A.8 Hashed index of
file
pathnames.
A.7.3 A Hybrid Design
Placing the entire directory structure into a single
right sibling tree,
works well
as
long
file,
as
with the
left-child,
as the directory structure is small. If
into a few kilobytes, the entire directory
and can be accessed without any seeking at
all. But if the directory, structure is large, containing thousands of files,
accessing the various parts of the tree can require multiple seeks, just as it
the
file
containing the tree
structure can be held in
does
when
fits
RAM
each directory
is
a separate file.
Hashing the path names, on the other hand, provides single-seek access
to any file but does a very poor job of supporting generic file and directory
names, such as prog* .c, or even a simple command such as Is or dir to list
all
the
files
in a given subdirectory.
distribution of the keys, scattering
all
of the
files
By
definition, hashing
them over
in a given subdirectory, say the
randomizes the
the directory space. Finding
letters
subdirectory for the tree
shown in Fig. A. 6, requires a sequential reading of the entire directory.
What about a hybrid approach, in which we build a conventional
file for each directory and then supplement
hashed index to all the files in all directories? This
approach allows us to get to any subdirectory, and therefore to the
information required to open a file, with a single seek. At the same time, it
directory structure that uses a
this
by building
provides us with the ability to
using generic
a
file
work with
names and commands such
conventional directory structure to get
all
the
as
files
Is
and
inside each directory,
dir.
In short,
we build
the advantages of that approach,
and then solve the access problem by building an index for the subdirectories.
563
SUMMARY
Parent
RRN
1
3
4
5
FIGURE A.9 Path index table of directories.
-1
Root
Report 5
Letters
School
Work
Persona 1
Hork
2
2
This
very close to the approach that the committee
is
they went one step further.
settled on.
Since the directory structure
is
But
highly
organized, hierarchical collection of
special index that takes advantage
files, the committee decided to use a
of that hierarchy, rather than simply
hashing the path names of the subdirectories. Figure A.9 shows what
index structure looks like
when it is
A. 6. Only the directories are
this
applied to the directory structure in Fig.
listed in the index; access to the data files
is
through the directory files. The directories are ordered in the index so
parents always appear before their children. Each child is associated with an
integer that is a backward reference to the relative record number (RRN) of
directory
the parent. This allows us to distinguish between the
directory under LETTERS. It also
under REPORTS and the
allows us to traverse the directory structure, moving both up and down
with a command such as the cd command in DOS or UNIX, without
having to access the actual directory files on the CD-ROM. It is a good
example of a specialized index structure that makes use of the organization
inherent in the data to produce a very compact, highly functional access
WORK
WORK
mechanism.
Summary
CD-ROM
an electronic publishing medium that allows us to replicate
and distribute large amounts of information very inexpensively. The
primary disadvantage of CD-ROM is that seek performance is relatively
slow. This is not a problem that can be solved simply by building better
is
performance grow directly from the fact that
CD-ROM is built on top of the CD audio standard. Adherence to this
standard, even given its limitations, is the basis for CD-ROM's success as
a publishing medium. Consequently, CD-ROM application developers
drives; the limits in seek
must look
software.
to careful
file
structure design to build
fast,
responsive retrieval
564
APPENDIX
STRUCTURES ON CD-ROM
A: FILE
B-tree and
B+
on
sector size
work
tree structures
provide access to
ability to
CD-ROM
is
many
it is
few
because of their
Because the
seeks.
2 Kbytes, the block size used in a tree should be
2 Kbytes or an even multiple of this sector
seek so slowly,
CD-ROM
well on
keys with just
size.
Because
CD-ROM
drives
usually advantageous to use larger blocks consisting of
Kbytes or more. Since no additions or deletions will be made to a tree
once it is on CD-ROM, it is useful to build the trees from the bottom up
8
so the blocks are completely
filled.
indexes, the read-only nature of
When
using trees to create secondary
CD-ROM
makes
possible to bind the
it
indexes tightly to the target data, pinning the index records to reduce
seeking and increase performance.
Hashed indexes
are often a
good choice
for
CD-ROM because they can
provide single-seek access to target information. As with
trees, the
The bucket
sector size affects the design of the hashed index:
size
2-Kbyte
should be
one or more full sectors. Since CD-ROMs are large, there is often enough
space on the disc to permit use of packing densities of 60% or less for the
hashed index. Use of packing densities of less than 60%, combined with a
bucket size of 10 or more records, results in single-seek access for almost all
records. But it is not always possible to pack the index this loosely. Higher
packing densities can be accommodated without loss of performance if we
hash function to the records in the index, using
tailor the
provides
more
nearly uniform distribution. Since
we know
function that
that there will
be no deletions or additions to the index, and since time spent optimizing
the index will result in benefits again and again as the discs are used,
often worthwhile
functions. This
of
densities
In 1985
is
90%
to invest the effort in finding the best
especially true
there
to support higher packing
or more.
companies trying
faced an interesting
common
when we need
it is
of several hash
file
to build the
structure problem.
directory structure and
file
CD-ROM
They
system for
were no directory structure designs
in use
publishing market
needed a
At the time,
realized that they
on
CD-ROM.
CD-ROM that provided
nearly optimal performance across a wide variety of applications.
Directory structures are usually implemented
Moving from
another
it
CD-ROM,
This is not a good design for
of several seconds just to locate and open
file.
in a wait
as
directory to a subdirectory beneath
of
series
since
it
files
to
could result
a single file.
alternatives, such as putting the entire directory structure in a single
hashing the path names of all the
files.
means seeking
Simple
file,
or
on a disc, have other drawbacks. The
problem emerged with a design that
committee charged with solving this
combined a conventional hierarchical directory of files with an index to the
directory. The index makes use of the structure inherent in the directory
hierarchy to provide
very compact, yet functional
map of the
directory
SUMMARY
565
CD-ROM indexing problems, this directory
importance of building indexes very tightly on
CD-ROM, despite the vast, often unused capacity of the CD-ROM disc.
Tight, dense indexes work better on CD-ROM because they require fewer
seeks to access. Avoiding seeks is the key consideration for all CD-ROM
structure. Typical
index
file
illustrates
of other
the
structure design.
Appendix B
ASCII Table
Dec. Oct. Hex.
nul
Dec. Oct. Hex.
sp
40
20
41
21
42
43
44
45
46
47
22
23
24
25
26
27
28
29
64
65
66
67
68
69
70
71
2F
N
O
72
73
74
75
76
77
78
79
60
30
61
31
62
63
64
65
66
67
32
33
34
35
36
37
70
38
39
3A
3B
stx
etx
cot
enq
ack
&
bel
bs
10
40
50
ht
11
41
51
nl
10
12
11
13
np
12
14
cr
13
15
so
14
16
si
15
17
42
43
44
45
46
47
52
vt
die
16
20
10
del
17
21
11
dc2
dc3
dc4
18
12
13
51
20
14
nak
21
15
syn
22
23
22
23
24
25
26
27
48
49
50
16
17
52
53
54
55
30
18
31
19
32
33
34
35
36
37
1A
"
'
etb
19
rs
24
25
26
27
28
29
30
us
31
can
cm
sub
esc
fs
gs
566
1C
ID
<
56
57
58
59
60
61
IE
>
IF
IB
32
33
34
35
36
37
38
39
sol
Dec. Oct. Hex.
62
63
53
54
55
56
57
71
72
73
74
75
76
77
2A
2B
2C
2D
2E
3C
3D
D
E
42
43
44
45
46
47
110
48
49
68
69
111
160
70
51
P
q
112
113
161
71
122
52
123
53
54
55
56
57
114
115
116
117
118
119
162
163
164
165
166
167
72
73
74
75
76
77
58
59
120
170
121
171
78
79
122
172
173
174
175
176
177
130
150
4F
88
89
90
62
63
64
65
66
67
82
83
84
85
86
87
61
142
143
144
145
146
147
4E
60
152
153
154
155
156
157
4A
121
140
141
104
105
106
107
108
109
110
111
81
3E
3F
102
103
104
105
106
107
Q
T
102
103
50
101
41
120
101
80
96
97
98
99
100
40
4C
4D
100
112
113
114
115
116
117
Dec. Oct. Hex.
124
125
126
127
131
4B
k
1
u
V
132
133
5A
91
5B
92
93
94
95
134
135
136
137
5C
5D
123
124
5E
125
126
5F
del
127
151
6A
6B
6C
6D
6E
6F
7A
7B
7C
7D
7E
7F
Appendix C
String Functions
in
Pascal: tools. pre
Functions and Procedures Used to Operate on strng
The following
make up
functions and procedures
the tools for operating
on
variables that are declared as:
TYPE
strng
packed array
The length of
the strng
is
MAX_REC_LGTH
of
char;
stored in the zeroth byte of the array as a
character representative of the length. Note that the Pascal functions CHR(
and ORD() are used to convert integers to characters and vice versa.
Functions include:
Returns the length of str.
len_str(str)
by
dear_str(str)
Clears
copy _jtr(str l,str2)
Copies contents of str2
cat_str(strl,str2)
str
setting
Concatenates
str2 to
Puts result in
strl.
Reads
write _str(str)
Writes contents of
Reads
str as
a str
Trims
trim_str(str)
first,
to the screen.
str
with length
trailing blanks
Returns length of
Converts
ucase(strl ,str2)
key)
strl to
Combines
0.
end of strl.
Igth
Writes contents of str to
fwrite_str(fd,str)
makekey (last,
length to
to strl.
input from the keyboard.
read_str(str)
fread_str(fd,str, Igth)
its
last
from
file fd.
file fd.
from
str.
str.
uppercase, storing result in
and first into key
str2.
in canonical
form, storing result in key.
minfintl ,int2)
cmp_jstr(strl ,str2)
Returns the
Compares
If strl
If strl
If strl
<
>
minimum
of two integers.
strl to str2:
str2,
cmp_str returns
0.
str2,
returns a negative number.
str2,
returns a positive
number
567
568
APPENDIX
C:
STRING FUNCTIONS
IN
PASCAL: tools.prc
FUNCTION len_5tr Cstr: 5trng): integer;
{
len_str() returns the length of 5tr >
BEGIN
len_str := QRDCstrCO]
END;
PROCEDURE clear_str(VAR str: strng);
{
A procedure that clears str by setting its length to
BEGIN
str CO]
CHR(O)
=
>
END;
PROCEDURE copy_str(VAR strl: strng; str2: strng);
>
{
A procedure to copy str2 into strl
VAR
i
eger
BEGIN
for
:=
strl
strl CO]
en_s t
str2C
str2[01
to
1
[
:=
r ( s t r
i
DO
END;
PROCEDURE cat_str (VAR strl: strng; str2: strng);
{
cat_str() concatenates str2 to the end of strl and stores
the resu
VAR
i
in
It
strl
integer
>
BEGIN
for
:=
to 1 en_s t r (
strl [(len_str(str1
strHO]
:=
DO
5 t r 2 )
) +
i)l
CHR(len_str(str1
:=
)
str2Ci];
1 en_s t r (
s t r 2 ) )
END;
PROCEDURE read_str (VAR str: strng);
A procedure that reads str as input from the keyboard
VAR
1
gt h
nt eger
BEGIN
lgth
:=
while (not EOLN) and (lgth <= MAX_REC_S ZE ) DO
BEGIN
I
lgth
=
lgth +
read (strtlgthl)
:
END;
read 1 n
str CO]
END;
;
:
CHR(lgth)
>
FUNCTIONS AND PROCEDURES USED TO OPERATE ON
PROCEDURE write_str (VAR str: strng);
write_str() writes str to the screen
{
strng
569
>
VAR
i
eger
BEGIN
for
wr
:=
to
wr i t e(
t eln
s t r
len_str(str) DO
[ i 3
END;
PROCEDURE fread_str (VAR fd: text; VAR str: strng; lgth:
fread_str() reads a str with length lgth from fd >
{
VAR
i
nt eger
BEGIN
for
str [0]
to
readC
f d
lgth DO
s t r
CHR(lgth)
END;
PROCEDURE fwrite_str (VAR fd: text; str
{
fwrite_str() writes str to file fd }
strng);
VAR
i
eger
BEGIN
for
:=
wr
i t
e(
to
f d
len_str(str) DO
s t r
END;
FUNCTION trim_str (VAR str: strng): integer;
{
trim_str() trims the blanks off the end of str and
returns its new length >
VAR
lgth
nt eger
BEGIN
lgth := 1 en_5 t r (
while strtlgth]
s t r )
=
'
lgth
=
lgth strCO]
=
CHR(lgth);
lgth
t r im_5 t r
=
:
END;
DO
integer);
570
APPENDIX
C:
STRING FUNCTIONS
IN
PASCAL: tools.prc
PROCEDURE ucase (strl: strng; VAR str2: strng);
{
ucaseO converts 5tr1 to uppercase letters and stores the
capitalized string in str2 }
VAR
i
integer
BEGIN
to 1 en_s t r ( s t r ) DO
for i :=
BEGIN
i
) <= ORDCz
if CDRDCstrl Ci]) >= ORD('a')) AND CORDC s t r
str2Ci] := CHR(0RD(str1 til)- 32)
else
str2[i] := str! [i]
:
END;
str2[0]
:=
)) then
strl [0]
END;
PROC EDURE makekey (last: strng; first: strng; VAR key: strng);
{
ma kekey() trims the blanks off the ends of the strngs last and
f i r st concatenates last and first together with a space
se parating them, and converts the letters to uppercase }
VAR
integer
lenl
integer;
lenf
blank_str: strng;
BEGIN
lenl := t r im_s t r ( las t )
copy_str (key last);
blank_str[0] := CHRC1 );
;
blank_str[1 ] :=
:
'
cat_str(key blank_str)
T
lenf
:=
t r
im_s t r ( f
rst )
cat_5 1 r(key,first);
ucase( key key)
,
END;
FUNCTION min (int1,int2: integer): integer;
min() returns the minimum of two integers
BEGIN
if
intl
<=
int2 then
min
nt
min
nt
else
END;
FUNCTION cmp_str (strl: strng; str2: strng):
{
integer;
If strl = str2, then
function that compares strl to str2.
If strl < str2, then cmp_str returns a
cmp_str returns 0.
A
FUNCTIONS AND PROCEDURES USED TO OPERATE ON strng
negative number.
positive number.
Or
if
strl
>
str2,
then cmp_str returns a
VAR
i
integer
length
i nt eger
BEGIN
if len_str(str1 )
BEGIN
:
:=
while strl
:=
if
[1
len_str(str1
cmp_5 t r
str2Ci] DO
1)
(i
len_s t r ( s t r2) then
then
else
cmp_str := (ORDC s t r 1 [ i ]
END
else BEGIN
length := mi n( len_s t r ( s t r 1
i
:=
if
len_s t r (
s t r
2)
while (striCil
:=
(ORDC s t r2
) )
str2Ci]) and (i <= length) DO
length then
cmp_str := len_s t r ( s t r
cmp_str
) )
>
len_s t r ( s t r2)
else
END
END;
:=
(0RD( s t r 1
(0RD(
s t
r2
i ]
57
Appendix D
Comparing Disk Drives
There are enormous differences among different types of drives in terms of
the amount of data they hold, the time it takes them to access data, overall
cost, cost per bit, and intelligence. Furthermore, disk devices and media are
evolving so rapidly that the figures on speed, capacity, and intelligence that
apply one month may very well be out of date the next month.
Access time, you will recall, is composed of seek time, rotational delay,
and transfer time.
Seek times are usually described in two ways: minimum seek time and
average seek time. Usually, but not always,
time
it
takes for the head to accelerate
settle to a stop.
from
as these since their
Average seek time
sector
is
seek time includes the
a standstill,
move one
Sometimes the track-to-track seek time
separate figure for head settling time.
such
minimum
as likely to
One
the average time
track,
and
given, with a
has to be careful with figures
meanings are not always stated
is
is
it
clearly.
takes for a seek if the desired
be on any one cylinder
as
it
is
on any
other. In a
completely random accessing environment, it can be shown that the
number of cylinders covered in an average seek is approximately one-third
of the total number of cylinders (Pechura and Schoeffler, 1983). Estimates
of average seek time are commonly based on this result.
Certain disk drives, called fixed head disk
drives,
require no seek time.
Fixed head drives provide one or more read/write heads per track, so there
is
no need
very
fast,
There
to
move
the heads
from
track to track. Fixed head disk drives are
but also considerably more expensive than movable head drives.
are generally
no
significant differences in rotational delay
among
similar drives. Most floppy disk drives rotate between 300 and 600 rpm.
Hard disk drives generally rotate at approximately 3600 rpm, though this
will increase as disks decrease in physical size. There is at least one drive that
rotates at 5400 rpm, and speeds of 7200 rpm are possible. Floppy disks
usually do not spin continuously, so intermittent accessing of floppy drives
might involve an extra delay due to startup of a second or more. Strategies
572
COMPARING DISK DRIVES
such
as sector interleaving
some circumstances.
The volume of data
years, thereby focusing
rate
from
a single
drive
can mitigate the effects of rotational delay
to be transferred has increased
much
is
attention
on
enormously
data transfer rate.
in
in recent
Data transfer
constrained by rotation speed, recording density
itself, and the speed at which the controller can pass data
through to or from RAM. Since rotation speeds vary little, the main
differences among drives are due to differences in recording density. In
recent years there have been tremendous advances in improving recording
densities on disks of all types. Differences in recording densities are usually
expressed in terms of the number of tracks per surface, and the number of
bytes per track. If data are organized by sector on a disk, and more than one
sector is transferred at a time, the effective data transfer rate depends also on
the method of sector interleaving used. The effect of interleaving can be
substantial, of course, since logically adjacent sectors are often widely
on the disk
separated physically.
A
from
different
approach to increasing data transfer
A technology
different places simultaneously.
rate
is
called
to access data
PTD
(parallel
and writes data simultaneously from multiple read/write
heads. The Seagate Sable PTD reaches a transfer rate of over 20 Mbytes per
second using eight read/write heads.
Another promising technology for achieving high transfer rates is
RAID (redundant arrays of inexpensive disks), in which a collection of
small inexpensive disks function as one. RAIDs allow the use of several
separate I/O controllers operating in parallel. These parallel accesses can be
coordinated to satisfy a single logical I/O request, or can service several
independent I/O requests simultaneously.
Although it is very possible that most of the figures in Table D.l will
be superseded during the time between the writing and the publication of
this text, they should give you a basic idea of the magnitude and range of
performance characteristics for disks. The fact that they are changing so
rapidly should also serve to emphasize the importance of being aware of
disk drive performance characteristics when you are in a position to choose
transfer disk) reads
among
Of
different drives.
course, in addition to the quantitative differences
among
drives,
there are other important differences. The IBM 3380 drive, for example, has
many built-in features, including separate actuator arms that allow it to
perform two accesses simultaneously. It also has large local buffers and a
it to optimize many operations that,
have to be monitored by the central
great deal of local intelligence, enabling
with less sophisticated drives,
computer.
573
sC
Is
-w
<J so
C/5
C/3
co
r^
rf
in
cm
oc
vc
CM
in
"O
O.SO
< s
z 8
m in
m o>
c
13 JZ
cs
.* 13++
J2 -S
t-h
vC
00
0 CO
o co
ffl
oc
CO
oc
sc
r-
moo
r-
tN
co
C<
Efl
-^
lib
73
m
CO
CM
oc
CM
CM
-3
_
ed
C/5
C
u
r-,
nj
V5
&
CM
CM
^^
(N
3\
rn .5 uu
o
oc
y:
^
4-1
y:
O
>
n
t/5
Si
O
>
T3
'
in
7^-
^j
n j
p
e a
T3
O
C/5
(/;
t/5
(U
'-*-
<u
"5 5c
TO
Q.
Tj co
.s
E
o
Ih
CJ
T3
-^
c/3
574
;/i
i-
<
-=
H U
>-
-^
~
r;
J-i
W)*T3
H u
"T3
to
s-
.s
pq
ij
CJ
c/J
o
>
>
>
o
o
3 S
m a,
u tfl w
(U
>
'o
"T3
4-1
'
(j
r;
T3
X)
^ H u 2
u
C/5
t/5
(J
(/5
u u >
'13
Bibliography
AT&T.
System
V Interface
Definition. Indianapolis, IN:
AT&T,
1986.
Computer Algorithms: Introduction to Design and Analysis. Reading, Mass.:
Addison-Wesley, 1978.
+
Batory, D.S. "B trees and indexed sequential files: A performance comparison."
SIGMOD (1981): 30-39.
Bayer, R., and E. McCreight. "Organization and maintenance of large ordered
Baase,
S.
ACM
indexes." Acta Informatica
1,
no. 3 (1972): 173-189.
Bayer, R., and K. Unterauer. "Prefix B-trees."
Systems
Bentley,
J.
2,
no.
(March
ACM
Transactions on Database
1977): 11-26.
"Programming pearls: A spelling checker." Communications of the
no. 5 (May 1985): 456-462.
ACM 28,
Bohl,
M.
Introduction to
IBM
Direct Access Storage Devices. Chicago: Science Re-
search Associates, Inc., 1981.
Borland. Turbo Toolbox Reference Manual. Scott's Valley, Calif: Borland International, Inc., 1984.
Bourne, S.R. The Unix System. Reading, Mass.: Addison-Wesley. 1984.
Bradley, J. File and Data Base Techniques. New York: Holt, Rinehart, and Winston, 1982.
Chaney, R., and B. Johnson. "Maximizing hard-disk performance." Byte 9, no.
5 (May 1984): 307-334.
Chang, C.C. "The study of an ordered minimal perfect hashing scheme." Communications of the
27, no. 4 (April 1984): 384-387.
Chang, H. "A Study of Dynamic Hashing and Dynamic Hashing with Deferred
Splitting." Unpublished Master's thesis, Oklahoma State University, De-
ACM
cember 1985.
"Minimal
Chichelli, R.J.
the
ACM 23,
no.
perfect hash functions
made simple." Communications
of
(January 1980): 17-19.
Comer, D. "The ubiquitous B-tree."
ACM Computing
Surveys 11, no. 2 (June
1979): 121-137.
Cooper, D. Standard Pascal User Reference Manual.
New
York:
W.W. Norton &
Co., 1983.
575
576
BIBLIOGRAPHY
Crotzer, A.D. "Efficacy of B-trees in an information storage and retrieval envi-
ronment." Unpublished Master's
thesis,
Oklahoma
State University, 1975.
Davis, W.S. "Empirical behavior of B-trees." Unpublished Master's thesis,
Oklahoma
State University, 1974.
H. An Introduction to Operating Systems. Revised 1st Ed. Reading, Mass.:
Addison-Wesley, 1984.
Digital. Introduction to VAX-11 Record Management Services. Order No. AADeitel,
D024A-TE.
Equipment Corporation, 1978.
Equipment Corporation, 1981.
RMS-II User's Guide. Digital Equipment Corporation, 1979.
VAX-U SORT/MERGE User's Guide. Digital Equipment Corporation,
Digital
Digital. Peripherals Handbook. Digital
Digital.
Digital.
1984.
VAX Software Handbook. Digital Equipment Corporation, 1982.
Dodds, DJ. "Pracnique: Reducing dictionary size by using a hashing technique."
Digital.
Communications of the
ACM 25,
Dwyer, B. "One more time
the
ACM 24,
no.
how
no. 6 (June 1982): 368-370.
to update a master file." Communications of
(January 1981): 3-8.
ACM
Computing
and H.C. Du. "Dynamic Hashing Schemes."
Qune 1988): 85-113.
Fagin, R., J. Nievergelt, N. Pippenger, and H.R. Strong. "Extendible hashTransactions on Database
ing
a fast access method for dynamic files."
Systems 4, no. 3 (September 1979): 315-344.
Computing Surveys 17, no. 1
Faloutsos, C. "Access methods for text."
(March 1985): 49-74.
Flajolet, P. "On the Performance Evaluation of Extendible Hashing and Trie
Searching." Acta Informatica 20 (1983): 345-369.
Enbody,
R.J.,
Surveys 20, no. 2
ACM
ACM
Flores,
I.
Peripheral Devices.
Englewood
Cliffs, N.J.: Prentice-Hall, 1973.
Gonnet, G.H. Handbook of Algorithms and Data Structures. Reading, Mass.: Addison-Wesley, 1984.
Hanson, O. Design of Computer Data Files. Rockville, Md.: Computer Science
Press, 1982.
Held, G., and
ACM 21,
M.
Stonebraker. "B-trees reexamined." Communications of the
no. 2 (February 1978): 139-143.
Hoare, C.A.R. "The emperor's old clothes." The C.A.R. Turing
dress. Communications of the
ACM 24,
Award
ad-
no. 2 (February 1981): 75-83.
IBM. DFSORT General Information. IBM Order No. GC33-4033-11.
IBM. OS/VS Virtual Storage Access Method (VSAM) Planning Guide. IBM Order
No. GC26-3799.
Jensen, K., and N. Wirth. Pascal User Manual and Report, 2d Ed. Springer Verlag,
1974.
Keehn, D.G., andJ.O. Lacy.
"VSAM
data set design parameters."
IBM
Systems
fournal 13, no. 3 (1974): 186-212.
Kernighan, B., and R. Pike. The
UNIX Programming
Environment.
Englewood
Cliffs, N.J.: Prentice-Hall, 1984.
Kernighan, B., and D. Ritchie. The
N.J.: Prentice-Hall, 1978.
Programming Language. Englewood
Cliffs,
577
BIBLIOGRAPHY
Kernighan, B., and D. Ritchie. The
wood
Programming Language, 2nd Ed. Engle-
Cliffs, N.J.: Prentice-Hall, 1988.
Knuth, D. The Art of Computer Programming. Vol.
Ed. Reading, Mass.: Addison-Wesley, 1973a.
1,
Fundamental Algorithms. 2d
Knuth, D. The Art of Computer Programming. Vol. 3, Searching and Sorting. Reading, Mass.: Addison-Wesley, 1973b.
Lang, S.D., J.R. Dnscoll, andJ.H. Jou. "Batch insertion for tree structured file
improving differential database representation." C.^-TR-85,
Department of Computer Science, University of Central Florida, Orlando,
organizations
Flor.
Lapin, J.E. Portable
and
UNIX
System Programming. Englewood Cliffs, N.J.:
Prentice-Hall, 1987.
"Dynamic Hashing." BIT
Larson, P.
18 (1978): 184-201.
ACM
Larson, P. "Linear Hashing with Overflow-handling by Linear Probing."
Transactions on Database Systems 10, no.
(March
75-89.
1985):
Larson, P. "Linear Hashing with Partial Expansions." Proceedings of the 6th Conference on Very Large Databases. (Montreal, Canada Oct 1-3, 1980) New
ACM/IEEE: 224-233.
York:
Larson, P. "Performance Analysis of Linear Hashing with Partial Expansions."
ACM
Laub, L.
S.
Transactions on Database Systems 7, no. 4
"What
is
CD-ROM?"
Ropiequet, eds.
(December
1982):
566-587.
CD-ROM: The New Papyrus. S. Lambert
Redmond, WA: Microsoft Press, 1986: 47-71.
In
and
M.K. McKusick, M. Karels, andJ.S. Quarterman. The Design and
4.3BSD UNIX Operating System. Reading, Mass.: Addi-
Leffler, S.,
Implementation of the
son-Wesley, 1989.
Levy, M.R. "Modularity and the sequential
file
update problem." Communications
ACM
25, no. 6 (June 1982): 362-367.
of the
Litwin, W. "Linear Hashing: A New Tool for File and Table Addressing." Proceedings of the 6th Conference on Very Large Databases (Montreal,
1-3, 1980)
Litwin,
W.
New
York:
"Virtual Hashing:
the 4th Conference on
Canada, Oct
ACM/IEEE: 212-223.
Dynamically Changing Hashing.
Very Large Databases (Berlin 1978)
New
"'
Proceedings oj
York:
ACM/
IEEE: 517-523.
Loomis, M. Data Management and
File Processing.
Englewood
Cliffs, N.J.:
Pren-
tice-Hall, 1983.
Lorin, H. Sorting and Sort Systems. Reading, Mass.: Addison-Wesley, 1975.
Lum, V.Y.,
P.S. Yuen, and M. Dodd. "Key-to-Address Transform Techniques,
Fundamental Performance Study on Large Existing Formatted Files."
Communications of the
14, no. 4 (April 1971): 228-39.
ACM
Lynch, T. Data Compression Techniques and Applications.
trand Reinhold
Madnick, S.E., and
Company,
J.J.
Inc.,
Donovan.
New
York: Van Nos-
1985.
Operatifig Systems.
Englewood
Cliffs. N.J.:
Prentice-Hall, 1974.
Maurer, W.D., and T.G. Lewis. "Hash table methods."
7, no. 1 (March 1975): 5-19.
ACM Computing
Surveys
578
BIBLIOGRAPHY
McCrcight, E. "Pagination of
cations
of the
ACM 20,
McKusick, M.K.,
W.M.
ACM
UNIX."
B*
trees
with variable length records." Communi-
no. 9 (September 1977): 670-674.
Joy,
S.J. Leffler,
Transactions on
and R.S. Fabry.
Computer Systems
2,
"A
fast file
system for
no. 3 (August 1984):
181-197.
Mendelson, H. "Analysis of Extendible Hashing." IEEE Transactions on Software
Engineering 8, no. 6 (November 1982): 611-619.
Microsoft, Inc. Disk Operating System. Version 2.00.
Language Series. IBM, 1983.
Morgan, R., and H. McGilton. Introducing
UNIX
IBM
Personal
System V.
New
Computer
York:
Mc-
Graw-Hill, 1987.
Murayama,
K., and S.E. Smith. "Analysis of design alternatives for virtual
memory
indexes." Communications of the
ACM 20,
no. 4 (April 1977):
245-254.
Nievergelt,
J.,
H. Hinterberger, and K. Sevcik. "The grid
metric, multikey
1
file
structure."
ACM
file:
an adaptive sym-
Transactions on Database Systems 9, no.
(March 1984): 38-71.
Ouskel, M., and P. Scheuermann. "Multidimensional B-trees: Analysis of dynamic behavior." BIT 21 (1981):401-418.
Pechura, M.A., and J.D. Schoeffler. "Estimating file access of floppy disks."
Communications of the
26, no. 10 (October 1983): 754-763.
ACM
Peterson, J.L., and A. Silberschatz. Operating System Concepts, 2nd Ed. Reading,
Mass.: Addison- Wesley, 1985.
Peterson,
W.W.
"Addressing for random access storage."
and Development
1,
IBM fournal
of Research
no. 2(1957):130-146.
and T. Sterling. A Guide to Structured Programming and PE/I. 3rd Ed.
York: Holt, Rinehart, and Winston, 1980.
Ritchie, B., and K. Thompson. "The UNIX time-sharing system." Communications of the
17, no. 7 (July 1974): 365-375.
Pollack, S.,
New
ACM
Ritchie,
D. The Unix I/O System. Murray
Hill, N.J.:
AT&T
Bell Laboratories,
1979.
Robinson, J.T. "The K-d B-tree:
dynamic indexes."
search structure for large multidimensional
ACM SIGMOD
1981 International Conference on Manage-
ment of Data. April 29-May 1, 1981.
Rosenberg, A.L., and L. Snyder. "Time and space optimality in B-trees."
ACM
(March 1981): 174-183.
Sager, T.J. "A polynomial time generator for minimal perfect hash functions."
Communications of the
28, no. 5 (May 1985): 523-532.
Salton, G., and M. McGill. Introduction to Modern Information Retrieval. McGrawTransactions on Database Systems 6, no.
ACM
Hill, 1983.
Salzberg, B. File Structures.
Salzberg, B., et
al.
Englewood
"FastSort:
Cliffs, N.J.: Prentice-Hall, 1988.
Distributed Single-Input, Single-Output Sort."
ACM SIGMOD International Conference on Management
SIGMOD RECORD, Vol. 19, Issue 2, (June 1990): 94-101.
Proceedings of the 1990
of Data,
Scholl,
M. "New
tions
file
organizations based on dynamic hashing."
on Database Systems
6,
no.
(March
1981): 194-211.
ACM
Transac-
579
BIBLIOGRAPHY
Severance, D.G. "Identifier search mechanisms:
ACM Computing Surveys 6,
model."
"On
Snyder, L.
survey and generalized
no. 3 (September 1974): 175-194.
B-trees reexamined." Communications of the
ACM 21,
no. 7 (July
1978): 594.
J. P. Tremblay and R.F. Deutscher. "Key-to-Address Transformation Techniques." INFOR (Canada) Vol. 16, no. 1 (1978): 397-409.
Spector, A., and D. Gifford. "Case study: The space shuttle primary computer
system." Communications of the
27, no. 9 (September 1984): 872-900.
Standish, T.A. Data Structure Techniques. Reading, Mass.: Addison-Wcsley, 1980.
Sun Microsystems. Networking on the Sun Workstation. Mountain View, CA: Sun
Microsystems, Inc., 1986.
Sussenguth, E.H. "The use of tree structures for processing files." Communications of the
6, no. 5 (May 1963): 272-279.
Sorenson, P.G.,
ACM
ACM
"Keyfield design." Datamation (October
Sweet,
F.
Teory,
T.J.,
and
1,
1985): 119-120.
Fry. Design of Database Structures.
J. P.
Englewood
Cliffs, N.J.:
Prentice-Hall, 1982.
The Joint ANSI/IEEE
Pascal Standards Committee. "Pascal:
SIGPLAN Notices
Forward
to the can-
28-44.
and P.G. Sorenson. An Introduction to Data Structures with Applications. New York: McGraw-Hill, 1984.
Ullman, J. Principles of Database Systems, 2d Ed. Rockville, Md.: Computer Scididate extension library."
Tremblay,
19, no. 7 (July 1984):
J. P.,
ence Press, 1980.
Ullman, J.D.
Principles of Database Systems,
3d Ed. Rockville, Md.: Computer
Science Press, 1986.
U.C. Berkeley.
UNIX Programmer's
Reference Manual. University
of California
Berkeley, 1986.
VanDoren,
the
"Some
J.
NSF-CBMS
empirical results on generalized
zation and Retrieval. University
VanDoren,
J.,
and
AVL
trees." Proceedings of
Regional Research Conference on Automatic Information Organi-
J.
Gray.
In Information Systems,
"An
of Missouri
at
Columbia
(July 1973):
algorithm for maintaining dynamic
COINS
IV,
New
46-62.
AVL
trees."
York: Plenum Press, 1974:
161-180.
Veklerov, E. "Analysis of Dynamic Hashing with Deferred Splitting."
Transactions on Database Systems 10, no. 1 (March 1985): 90-96.
ACM
Wagner, R.E. "Indexing design considerations." IBM Systems Journal 12, no. 4
(1973): 351-367.
Wang, P. An Introduction to Berkeley Unix. Belmont, CA: Wadsworth Publishing
Co., 1988.
Webster, R.E.
sity,
"B +
trees."
Unpublished Master's
thesis,
Oklahoma
State Univer-
1980.
Welch, T.
"A Technique
for
High Performance Data Compression." IEEE Com-
puter, Vol. 17, no. 6 (June 1984):
Wells, D.C.,
8-19.
E.W. Greisen and R.H. Harten. "FITS:
Flexible
Image Transport
System." Astronomy and Astrophysics Supplement Series, no. 44 (1981):
363-370.
Wiederhold, G. Database Design, 2d Ed. New York: McGraw-Hill, 1983.
580
BIBLIOGRAPHY
Wirth, N.
"An
assessment of the programming language Pascal."
IEEE
Transac-
Sofiware Engineering SE-1, no. 2 (June 1975).
tions on
Yao, A. Chi-Chih.
159-170.
Zocllick, B.
"On random 2-3
"CD-ROM
trees." Acta Informatica 9, no. 2 (1978):
software development." Byte
11, no. 5
(May
1986):
173-188.
System Support for CD-ROM." In
Lambert and S. Ropiequet, eds. Redmond,
Zoellick, B. "File
rus. S.
CD-ROM: The New PapyWA: Microsoft Press,
1986: 103-128.
Zoellick, B. "Selecting an
ume
Approach
2: Optical Publishing. S.
1987: 63-82.
to
Document Retrieval." In CD-ROM, VolRedmond, WA: Microsoft Press,
Ropiequet, ed.
Index
Abstract data models
explanation
of,
FITS image
as
128
Access. See
124-125, 132
example
Random
Record
of,
access;
access
variable order, 422-425, 437
trees
explanation
B*
trees,
B+
trees
447, 493
overflow
sector, 46
LRU replacement, 376
simple prefix, 429-430. See
Simple prefix
use of,
114, 115
352-362
Assign statement, 9
list
explanation of, 193, 217
of fixed-length records, 193
195
of variable-length records,
196-198
Average search length
of,
492
number of collisions
and, 476
progressive overflow and,
469-471
record turnover and, 482
433
trees vs.,
and,
of,
553-555
347
deletion, redistribution,
and
concatenation in, 366-372
depth of, 364-366
explanation
558
explanation
trees
B-trees
construction
versions of, 137
Avail
431-433
insertion,
and hex values, 107-109
Baver, R., 334-335, 337, 347,
348, 363, 371-372, 431
Berkeley UNIX, compression
of,
tor indexes, 234
and information placement,
377-379
invention
of,
334-336
383
of order m, 364, 382
order of, 362-364, 364, 382.
383
page structure used by, 253.
352
splitting and promoting, 347leaf of,
189
fit
of, 217
placement strategies, 202
Better-than-random, 492-493
Binary encoding, 137-138
Binary search
explanation of, 204. 2(>5. 217
of index on secondary
storage, 234, 336
limitations of, 207-208, 228
explanation
sequential vs., 204-206
on variable-length entities,
422
Binary search
trees
balanced. 4
explanation
of,
337340
heap and, 280- 2S1
paged, 343-347. 352, 353
Binding
explanation of, 252
in
indexing,
Bkdddkey
249-250
function. 514. 515.
519
Bk del key function. 524. 526
Bkfind buddy function. 521
Bk<plit function,
516-519
524-
BktryColl<ip<c function.
351
underflow
in,
Best
and
CD-ROM
table,
436
429-431
ASCII
UNIX,
of, 4, 6, 413,
algorithms for searching and
126
431-433
virtual, 373-377, 383
Balanced merge
explanation of, 312-314, 325
improving performance of,
315, 316
general discussion regarding,
B+
in headers,
553-555
and,
explanation
also
4,
K-wav, 314-315
Adel'son-Vel'skh, G. M., 341
ASCII
340-343,
372-373, 382
CD-ROM
46-48
indexes to keep track of, 102,
103
open. See Progressive
in
of, 6,
382
and files, 4
B-trees vs., 433
buckets and, 471-479
extendible hashing and, 510
513
hashing and, 452-466
home,
use of,
AVL
access; Sequential
Access mode, 29
Addresses
block,
Average seek time, 572
in,
408
526
581
582
INDEX
Bk try combine
function,
524-
526
Block addressing, 46-48, 471
Block device, 82
Block I/O
explanation of, 83
UNIX. 46
use of,
Block
and
Byte count
Byte offset
file
78-79
and order
strengths and weaknesses of,
of,
109
63-68
of,
number
a predictable
of, 101
on performance, 53-54
of, 3,
45-47
sequence sets, 407-413,
417-421
Boolean functions, 18
in
Bpi, 82
392-393
394-396
Btutil.prc, 400-404
Btutil.c,
Buckets
buddv. 520-522, 535-536
and effect on performance,
472-476
explanation
493
closing
LIST program
tries
and,
507-509
of,
535-536
procedure for finding, 520522
Buffer pooling, 70-71
Buttering
bottlenecks
69
in,
double, 70, 311
explanation
of, 29, 38,
68
282-283, 287
multiple, 69-72, 283
disks and cache
input,
RAM
memory as, 55
during replacement selection,
303-304
and virtual
Collision resolution
155-156
trees,
373-377
156-157
94-95,
154-155
CD-ROM,
556
488
of, 447, 493
and extra memory, 462-466
velocity),
544^547-549
Comm, 322. 325
Compact disc read-onlv
memory (CD-ROM).
CD-ROM
Compaction
62
explanation
448-449
methods of reducing, 449450, 462-466
predicting, 457, 461-466
Color lookup table, 128
Color raster images, 128-129
Comer, Douglas, 334-336, 363
in hashing,
98, 99,
Cache, 55h
Canonical form
explanation of, 144
for keys, 110
in secondary indexes, 237
Cascade merge, 316
CAV (constant angular
of, 543,
563-565
problem.
545-546
system and, 79, 559-563
hashed files and, 557-559
file
tables, 487,
Collisions
explanation
as file structure
Buffering blocks, and
162-
103-105, 107, 109,
writstrm.c,
CD-ROM,
by chained progressive
overflow, 484-486
by chaining with separate
overflow area, 486- 487
by double hashing, 483
by progressive overflow,
466-471
and scatter
162
166
writrec.c,
(constant linear velocity),
Coalescing holes, 201
update. c, 119, 120, 123.
Buddy buckets
explanation
strjuncs.c,
from
547-549
Cmp, 320-322, 325
Coalescence, 217-218
106, 107, 158
readstrm.c, 99,
411
544,
389-396
391-392
readrec.c,
block size and, 411
effect on records, 469
explanation of. 42-43, 83,
CLV
keys into B-tree,
makekey.c, 161
13-14
use of, 45
159
to insert
29
14,
internal fragmentation
19-20
C programs
btio.c, 392-393
btutil.c, 394-396
driver. c, 390-391
fileio.h, 153-154
find.c, 160-161
520
and implementation
479
527, 534
15-18
in,
352,
space utilization for, 526-
in,
record length and, 105
insert. c,
476-
in,
),
files,
Clusters
14
in,
extendible hashing and, 512of,
Closing
119
hashing fold and add step
451
getrf.c,
of, 450, 471, 472,
in,
direct access in, 117, 123
seeks
Btio.c,
484-486
Chang, H., 534
CLOSE(
character strings
file
553-557
Character I/O, 83
Character I/O system, 78
Character strings, in Pascal and
C, 119
portability and, 141
organization
tree structure and,
Chained progressive overflow,
per track, 573
stream of, 146
Blocking factor, 46, 57-59
Blocks
explanation of, 82-83, 144
grouping records into, 113
546-
552-553
dump
making records
554
543-545
551
to calculate, 116
journey
choice of, 410-411
effect
29
of,
Bytes
size
CD-ROM,
history of,
physical organization of,
explanation
RRN
144
field,
explanation
storage,
of,
218
190-192
Compar( ), 320
Compression. See Data
compression
See
583
INDEX
Computer hardware, sort time
and. 293-295
Computer Systems Research
Group (CSRG), 53, 54
Concatenation
370
in B-trees, 367, 369,
due to insertions and
deletions. 408. 409
explanation of, 382
nominal. 59. 60, 85
Da tarec,
of block
performance
effect
117
Deletion. See Record deletion
Delimiters
organization
end of records. 102-103
ot.
144
and nondata overhead. 47-49
45-47
organizing tracks by sector.
speed
applied to general ledger
Descriptor table, 83
types of. 37
program, 268-276
and matching, 259-263
and merging, 263266
and multiway merging. 276-
Device driver. 76, 79. 83
Difi 321-322. 326
Dir_double function, 518. 519
285-286
of, 266-268
of.
of.
(DASDs),
83
Direct
42
Conversion
text,
138-140
m UNIX.
318-320
320-322
utilities for.
Count subblocks.
),
turning
Cylinders
a.
of.
tries into. 507,
508
Dir_ifi<-.bucket function. 518.
computing capacity
of,
of, 38, 40,
40
83
519
Dir_.try collapse function, 522.
523
Disk access
Dangling pointers, 213
Data
decreasing
application-oriented view of.
125
number
of. 5
rotational delay and, 50
49-50
seek access and, 37,
standardization of. 136-139
for,
188-189
218
irreversible, 189
and simple prefix method to
produce separators. 431
of,
suppressing repeated
sequences for, 186-188
UNIX. 189-190
using different data
186
for.
185
Data files, 212. 213. 239
Data transfer rate. 552. 573
Data transmission rate
estimating,
5960
explanation
of.
84
Record
distribution
Double buffering. 70, 311
Double hashing, 483, 493
Drive capacitv, 40
390-391
397-399
Driver. pa<,
C,
Dynamic
531, 532
hashing. 528-530.
Disk bound. 54
Disk cache. 55. 84
Disk controller. 67
Disk drives, 37
comparison of. 572-574
dedicated. 294
fixed head. 572
replacement selection and use
of two. 307, 309
use of multiple. 309-31
Disk packs
explanation
removable.
Disks
EBCDIC
(extended binary
coded decimal interchange
code), 136, 137
Effective recording densitv. 59.
84
Effective transmission rate.
of. 38.
;
84
as bottleneck.
59-
60, 84
EFM encoding, 547
80/20 rule of thumb. 488-489,
493
Enbodv, R. J., 531. 532
End-of-file (EOF), 28. 29
EndPosition(f 21
Entropy reduction, 189m
Entrv-sequenced files
.
basic operations on.
transfer time and, 51. 112
assigning variable-length
explanation
timing computations and,
51-53
Data compression
in
522
536
extendible hashing and. 513
519. 527-528. 530
collapsing
space utilization for. 527-528
46, 83
29
explanation
.See
317-318
536
explanation
Cosequential processing
codes
(DMA),
67m. 83
number and
Distribution.
Du, H.
access
of. 2
tape vs.. 61-62.
Driver. c,
37, 83
memory
41-45
Directory
structure, 139. 141
CREATE(
144-145
Direct access storage devices
explanation
file
of.
use of, 115-117, 123
Controller
speed
Direct access
explanation
37-39
of,
organizing tracks by block,
separating fields with, 97-99
Density, packed. See Packed
density
279,
53-54
estimating capacities and
space needs of, 38, 40-41
at
summary
on
of.
Davis. W. S.. 372
Dedicated disk drives. 294
Deferred splitting. 536
explanation
Consequential operations, 258.
325
Consequential processing model
size
explanation
of,
230-234
252
simple indexes with. 227230
Extendible hashing
and controlling
splitting.
533-534
and deletion. 520-526
and dynamic hashing, 528530
explanation ot. 6. 505510,
536
implementation of. 510-519
and linear hashing, 530-533
use ot.
4-5
Extendible hashing performance
and space utilization for
54-55
buckets.
526-527
584
INDEX
and space
merge
utilization for
527-528
directory,
Extensibility,
External fragmentation
218
methods to combat, 201
placement strategies and, 203
of,
tools for,
317-318
96-99
145
of, 96,
190-203
work
in,
organization and, 122
CD-ROM,
79,
with mixtures of data objects,
kernel and,
79-80
483-487
448-450
deletions and, 479-483
22-23,
559-563
131-132
object-oriented, 132-133,
141, 145
dynamic, 528-530, 536
79, 84, 141
explanation of, 84
UNIX counterpart
to,
76
File descriptor table,
218
placement strategy,
201-202
FITS (flexible image transport
system), 126-129, 136-137
102,
118-119
Fixed-length records
29
File descriptor,
74-75
dump, 107-109
manager
clusters and, 42-43
and
access, 123,
File
organization
access and,
527-528, 530
Floppy disks, 37, 572
Fold and add, 451-452, 493
Formatting
method
of,
explanation
pre-,
145
File protection,
of, 5, 6, 75,
84
closing,
data, 212, 213,
239
displaying contents of,
end
of,
84,
218
450-453
(hierarchical data format),
Header
files
explanation
of,
29-30
FITS, 126, 127, 130
self-describing, 125
UNIX,
in
26
Header records,
120, 122, 145
281-284
283, 284
Heapsort
of, 304, 326
use of, 280-281, 287, 291,
explanation
311
Height-balanced
trees, 341,
382-383
Hex dump
15-18
18
logical, 9, 30. See also Logical
files
44-45,
44-45, 198-200,
203, 218
storage, 198-201
Frames, 56, 84
13-14
4-5
writing out in sorted order,
of,
internal,
Files
use of,
Hashing algorithms
perfect, 449, 494
properties of, 280
external, 201, 203, 218
153-154
462
with simple indexes, 234
building,
74//
explanation
for, 139, 141
historv of, 3-5, 124
Fileio.h,
),
and, 462-466
record access and, 488-489
record distribution and, 453-
Heap
Fragmentation
File structures
conversions
explanation
84
47
Fprintf(
13
of,
530-533, 536
memory
130
Flajolet, P.,
122-123
linear,
192-196
deleting,
explanation of, 145
use of, 101, 102, 118-119
names, 76-78
hashing
steps in simple,
File
File
446-
indexed, 493
indexing vs., 447
HDF
204
File
explanation of, 84
function of, 64, 66, 68
of, 6, 431,
448, 493
extendible. See Extendible
Fixed disk, 84
Fixed head disk drives, 572
Fixed-length fields, 96-98, 101,
method, 145
allocation table (FAT)
File-access
file
double, 483, 493
explanation
160-161
Find.new range function, 517
Find. pas, 175-176
fit,
557-559
466-
471,
using indexes, 249
First-fit
CD-ROM,
collisions and,
29
of,
First
123
85
buckets and, 471-479
and
21-22
in,
explanation
UNIX,
link, 77,
Hardware. See Computer
hardware
Hashing
Find.c,
File access
File
in,
disks, 37
collision resolution and,
contained
Filesystems
making records a predictable
number of, 101-102
reading stream of, 99-100
file
files
78
special,
on
early
Hard
Hard
physical, 8-9, 30. See also
Physical
Gray, J., 343
Grep, 115
129-
special characteristics
Fields
explanation
in,
self-describing, 125
310-311
Field structures,
285-
132
normal, 78
opening, 9-13
reclaiming space
External sorting. See also
Sorting
tapes vs. disks for,
size of,
mixing object types
133-134, 145
Extents, 43-44, 84
explanation
and
sorts
311
Gather output, 72
Get. pre,
Getrf.c,
174-175
159
explanation
of,
107-109
portabilitv and, 135
HIGH_VALUE,
265-266, 326
585
INDEX
Home
address, 447, 493
Huffman
explanation
code, 188, 218
to
of, 226-227, 252
keep track of addresses,
102, 103
I/O
approaches
|,
paged, 383
primary, 237
in different
Keys
explanation
block, 46, 78-79, 83
secondary. See Secondary
indexes
character, 78, 83
selective, 248,
processing as, 14
overlapping processing and,
simple, 227-230, 234-235,
languages
16-17
to,
file
RAM
buffer space in
performing, 61
scatter/gather, 86
252
I/O buffers, 64-65, 69
I/O channels
transmission time and, 294295
Insert.c,
391-392
Insertions
+
of,
145
hashing methods and, 455456
and index content, 413-415
indexing to provide access by
'
multiple,
Inode table, 76
Input buffers, 282-283, 287
Insert( ) function, 357, 359
UNIX, 72-80
in
252
Indexing, hashing vs., 447
Inodc. See Index node
280-281, 283
Key held. 252
Key subblocks, 46-47, 85
KEYNOI)ES|
209-212
235-239
placement of information
associated with, 377-379
primary, 110, 111, 146
promotion
of, 383
sequence set, 430
secondary. See Secondary
keys
role in
418-421
355-360, 371-372
block splitting and, 408, 409
as separators,
in B-trees,
separators instead of,
variable-length records and,
description of, 67
236-237
random, 429
explanation
tombstones and, 481-482
use of, 311
of,
85
Insert. pre,
use of multiple, 309-310
I/O redirection
explanation of, 30
IBM,
examples
standardization issues and,
IEEE Standard
data
format, 137, 138
files
files
memory,
Inverted
contents of, 413
of deletions on, 417,
role
Index
of, 416, 433,
of separators
set
in,
422-425
244-248
Irreversible compression, 189
explanation
of,
218
to
files,
3-4
binding and, 249-250
Landis, E. M., 341
Larson, P., 528, 530. 534
LaserVision, 543-544
Leaf
of B-tree, 363, 383
Least-recently-used
(LRU)
strategy, 71
Ledger program, application of
consequential processing
Job control language (JCL), 9
model to, 268-276
Lempel-Ziv method, 189
Linear hashing, 530-533, 536
/C-way merge
Linear probing. See Progressive
421-422
Index tables, 130
Indexed hashing, 493
Indexed sequential access
explanation of, 406-407, 437
and sequence sets, 412
Indexes
added
lists
blocks
internal structure of,
size of,
437
430
373
134-135
Match
explanation of, 252
secondary index structure
and,
418
211-213, 285
pinned records and, 213-214
Keywords, 129, 130
Knuth, Donald, 301-302. 311.
limitations of,
Languages, portability and,
206
operation
Index node, 76, 77, 85
Index set
explanation
207-208
Intersection, 259. See also
335. See also B-trees
208-211, 218,
Lands, 546-547
limitations of,
of,
of,
312, 317, 342, 363, 372.
44-45, 218
minimization of, 198-200
placement strategies and, 203
use
234-235,
effect
of,
Internal sort
and, 212, 213
too large to hold in
85
Internal fragmentation
explanation
135-139
42
of,
Kcysort
289-290
56-57, 85
of,
of,
explanation
415
explanation
Interleaving factor
3380 drive, 573
portability and
Index
399-400
explanation
430-431
413-
379-380
Interblock gap, 47
UNIX, 25-26
IBM
trees,
index,
I/O processors
in
in
balanced,
314-315
explanation of, 326
use of, 276-279, 293, 295
Kernel
explanation of, 85
and filesystems, 79-80
Kernel I/O structure, 72-76
overflow
Linked lists
explanation of. 218
use of, 192-193, 245-247
LIST program
explanation
in Pascal
of.
15.
24-25
and C, 15-18
586
INDEX
Lists
inverted,
244-248
Linked
linked. See
lists
Litwin, W., 530, 533
Loading
of CD-ROM, 555
+
of simple prefix B
maintaining
match
311,
30
Machine architecture, 135-136
Magnetic disks, 37. See Disks
Magnetic tape
applications for, 60-61
disks vs., 61-62
and estimated data
transmission times, 5960
organizing data on, 56-57
sorting files on, 311-318
and tape length requirements,
Makekey.c, 161
Mass storage system, 85
Match operation
of,
326
merge vs., 264-265
for names in two lists, 259
334-335, 337,
347, 348, 363, 371-372,
380
E.,
Memory
and
collisions,
index
in,
files
462-466
too large to hold
234-235
loading index files into, 231
rewriting index file from,
231-232
Merge
balanced, 312-314
cascade, 316
fc-way,
285-
290-292
explanation
of, 145
image, 128
125-126, 128
and
raster
use
of,
Mid-square method, 456, 494
Minimum hashing, 494
Minimum seek time, 572
Mod
operator, 451-452
Morse code, 188
Move mode, 71
multistep. See Multistep
merge
9-13
files,
Operating systems, portability
and, 134
Opfind function, 514, 515
Optical discs, 37
Order
of B-trees, 362-364, 383,
422-425, 437
file dump and byte, 109
of merge, 327
Overflow. See also Progressive
of,
494
508-510
Overflow records
buckets and, 473-475, 532
expected number of, 464-465
techniques for handling, 466splitting to handle,
54
467
310
Multistep merge
number of seeks
using, 295-298, 311
explanation of, 326
replacement selection using,
304, 306, 307
Multiway merge
consequential processing
model and, 276-279
files, 285-
for sorting large
286
Packing density. See
also
Space
utilization
average search length and,
470
buckets and, 472-473
explanation of, 462-463, 494
overflow area and, 486-487,
558
predicting collisions for
different,
463-466
Page fault, 375
Paged binary trees
Network I/O system, 78
Nodes
in B-trees, 347-349
index, 76, 77, 85, 421
Nominal recording density, 85
Nominal transmission rate
explanation
of,
343-345
structure of, 253, 352
top-down construction
345-347
128
Parallel transfer disk
explanation
Parallelism, 54
85
Nondata overhead, 47-49
Nonmagnetic disks, 37
of,
of,
Paged index, 383
Palette,
computing, 59
(PTD), 573
Pareto Principle, 488-489
Parity, 86
Parity bit, 56
276-279
multiphase, 326
75-76, 85
13, 76
overflow
use of, 315-317
Multiprogramming
decreasing
30
file table,
explanation
Multiphase merges
explanation of, 326
effects of,
12,
Open() function,
Opening
time involved in, 287-290,
308
of two lists, 263-266
to avoid disk bottleneck,
57-59
and UNIX, 80
Makeaddress function, 510513, 532
263
McCreight,
files,
),
addressing. See
Progressive overflow
Open
Metadata
LOW_VALUE, 326
LRU replacement, 375-377
explanation
Open
through, 208
files
for sorting large
Locality, 55, 252
9,
OPEN(
264-265
vs.,
56
function, 514, 515, 519
OpJel function, 526
OpJir function, 522, 524
numbers of lists,
278-279
trees,
parity,
Op^add
for large
425-429
two-pass, 485-486
Locate mode, 71
Logical files
explanation of,
in UNIX, 23
Odd
order of, 327
polyphase, 316, 317, 327
Merge operation
explanation of, 326
O(l) access, 446-447
Object-oriented file access, 132133, 141, 145
Pascal
character strings
in,
119
direct access in, 117, 123
587
INDEX
hashing fold and add step in,
451
header records and, 120, 122
LIST program
opening
in,
files in,
15-18
10-11
record length and, 105
20-21
Pascal programs
btutil.prc, 400-404
driver.pas, 397-399
find. pas, 175-176
get. pre, 174-175
methods
182
Position(f), 21
Prefix, 416. See also
176-182
171-172
94-95,
writstrm.pas,
98, 99,
168-169
Pathnames, 30, 562, 563
Perfect hashing algorithm, 449,
494
files
of,
Simple
division,
452-453, 494
Process, 86
Progressive overflow
chained, 484-486
of, 466-467, 494
and open addresses, 480
and search length, 468-471,
476, 477
Promotion of key, 355-357,
383
Protection mode, 30
Pseudo random access devices,
335
PTD (parallel transfer disk), 573
of,
explanation
of,
30
25-26
546-547
use of,
Pixels, 128
Placement strategies
explanation of, 218
selection of, 203
types of,
201-202
chained progressive
overflow, 484-486
dangling, 213
Poisson distribution
in
applied to hashing, 460-461,
473
457-460, 494
packing density and, 463
Polyphase merge, 316, 317, 327
explanation
of,
Portability
explanation
Radix searching. See Tries
Radix transformation, 456
RAID (redundant arrays of
inexpensive disks), 573
RAM
RAM
146
86
disks, 55,
51-53
access,
access
196
from hashed file, 479-483
index, 233, 237-238
storage compaction and, 190192
of variable-length records,
also
vs. tape, 61,
storage
in,
304-306
545
36
54-55
amount of, 293-
increased use of,
increasing
294
in, 206-208, 211,
279-285, 287-290
Random hash functions, 454456
Randomization, 455, 456, 494.
See also Hashing
sorting
READ(
distribution
uniform, 454
117-121
and header records, 120, 122
methods of organizing, 101
103
that use length indicator,
103-107
Record updating, index, 233234,
238-239
Records
explanation of, 100, 101, 146
in Pascal,
96n
reading into
RAM,
287-288
record structure and length
of,
117-121
Redistribution. See also Record
C, 16
explanation
hashing functions and, 453462
Poisson. See Poisson
choosing record length and,
memory (RAM)
access time using, 2,
in
of,
526
of fixed-length records, 192
Record keys, 109-111
Record structures
buffer space, 61
Random
Random
file
and B-trees, 347-349
extendible hashing and, 520-
Redistribution
and disk
86
Pointers
RAM
and
sort, 304-306
Record additions. See Insertions
Record blocking, 112-113
Record deletion
+
in B
trees, 418-421
in B-trees, 366-368, 370
196-198
Record distribution. See
320, 327
of,
Pipes
Platter,
),
218
213-214, 235
explanation
use
Qsort(
172-173
155-156
Readstrm.pas, 169-172
Record access, 3-4
file access and, 51-53
hashing and, 488-489
patterns of, 488-489
Readrec.pas,
using replacement selection
for, 111
UNIX, 23-26
in
99
Readstrm.c, 99,
8-9, 30
Pinned records
Pits,
requirements
Prime
),
Readrec.c, 106, 107, 158
explanation
update. pas, 119, 120, 122, 123,
explanation
Readfieldf
explanation of, 110, 146
in index files, 244, 246, 247
397-404
insert. pre, 399-400
readrec.pas, 172-173
readstrm.pas, 169-172
16
sequential search and, 112
use of, 14-15, 113
and, 141-142
Primary indexes, 237
Primary keys
binding, 249
352,
Physical
136-
prefix B-trees
to insert keys into B-tree,
writrec.pas,
in Pascal,
for achieving,
141
UNIX
seeks in,
stod.pre,
134-136
factors of,
of,
30
distribution
588
INDEX
370-372,
in B-trees, 367,
408, 410, 425
explanation
of,
383
Redundancy reduction,
187, 188,
185,
219
Redundant arrays of
inexpensive disks (RAID),
573
Reference field, 228-229, 252
Relative block number (RBN),
423
Relative record number (RRN)
access by, 116, 204, 207
explanation of, 146
hashed files and, 476-477
in stack, 193, 194
and variable-length records,
196
Replacement
based on page height, 376377
LRU, 375-377
heapsort and, 280
Sequential search
Search length, 469. See also
for
Average search length
Secondary indexes
on CD-ROM, 556-557
improving structure of, 242248
primary vs., 237
236-237
to, 237-238
record updating to, 238-239
retrieval and, 239-241
record addition
record deletion
306-308
to,
235-236
use of,
Secondary key fields, 235
Secondary keys
binding, 249
index applications
111,
of,
Retrieval, using combinations
of secondarv keys, 239242
Rewrite statement, 10
Rotational delav, 50, 86, 572-
573
Run-length encoding
explanation of, 219
use of. 186-188
Runs
10
235-238
retrieval using
239-242
of,
327
298-303
285-289
length of,
use of,
Scatter/gather I/O, 86
Scatter input,
Scatter tables,
71-72
487-488
M., 534
Seagate Sable
PTD, 573
Selective indexes, 248, 252
Self-describing
files,
125. 146
Separators
explanation
of, 433, 437
and index content, 413-415
index
425
set
blocks and. 422-
413-415
430-431
instead of keys,
keys
as,
shortest, 437
Sequence checking, 327
Sequence set
adding simple index to, 411413
and block size, 410-411
blocks and, 407-410, 417-
425-429
explanation
of,
407, 433,
437
Secondary storage
Sequences, suppressing
access to, 36,
336-337
paged binary
344
trees and, 343,
repeating,
186-188
Sequential access,
explanation
3-4
of, 6,
146
indexed. See Indexed
simple indexes on, 234
Sector addressing, 46, 471
sequential access
Sectors
explanation of, 86
organization of, 86
organizing tracks by, 41-45
41-42
SEEK(
time computations and, 5253
use of, 122, 291
Sequential access device, 86
explanation of, 30-31
use of, 18-19
UNIX
Sequential processing,
tools for,
114-115
Sequential search
best uses of, 114
Seek and rotational delav, 288,
292-294
binary vs., 204-206
evaluating performance
CD-ROM,
552
explanation
of,
explanation
49-50,
86,
of,
146
use of record blocking with,
112-113
112-113
types of, 572
Serial devices,
SGML
37
(standard general
markup
Seeks
in C, 19-20
language), 130
131
excessive, 61
Shortest separator, 437
explanation of, 38
multistep merges to decrease
number of, 295-298, 311
Sibling, 367
in Pascal,
of,
111-112
Seek time
See'kRead(f,n), 21
explanation
merging large numbers of
278-279
lists,
421,
combinations
phvsical placement of,
Reset statement, 10, 11
248
selective indexes from.
of,
Replacement selection
average run length for, 301303
cost of using, 303-305
explanation of, 327
increasing run lengths using,
298-301
for initial run formation, 311.
312
plus multistep merging, 304,
Scholl,
Search. See Binary search;
20-21
Simple indexes
with entrv-sequenced
252
SeekWnte(f,n), 21
explanation
Selection tree
too large to hold in
explanation
of,
327
files,
227-230
234-235
of,
memory,
589
INDEX
Simple prefix
B+
B+
B+
also
STDIN, 24-25,
trees
STDOUT,
429-430. See
trees vs.,
trees
Stod.prc,
changes involving multiple
blocks in sequence set and,
418-421
changes localized to single
blocks in sequence set and,
417-418
explanation
of,
416-417, 437
425-429
loading,
use of, 431-432, 434
31,
Storage capacity, of CD-ROM,
552
Storage compaction, 190-192
Storage fragmentation, 198-201
Stream file, 94-96
Stream of bytes, 146
Streaming tape drive, 60-61, 86
Soft links, 77-78. See also
bottleneck, 54, 55
Stmg, 567-571
Subblocks
explanation of, 86
types of,
and cosequential processing
in
UNIX, 318-322
disk
files in
merging
RAM,
for large
206-208
285-
on
for directory,
Special
file,
527-528
86
Split ( ) function, 360,
361
Splitting
in B-trees, 355, 356, 360,
367, 425
Synchronization loop, 260-262,
267, 276, 327
Symbolic links, 77-78, 86
sort. See
Keysort
Tags
advantages of using, 133
explanation of, 129-131
132-133
specification of,
Tape. See Magnetic tape
Temporal locality, 376
Theorem A (Knuth), 328
deferred, 536
Tombstones
189-190
22-23
of block size on
performance, 53-54
and file dump, 108
effect
header
files,
26
commands, 26-27
72-80
filesystem
I/O in,
magnetic tape and, 80
physical and logical files
in,
23-26
explanation
and sequential processing,
114-115
sort utility for, 206
sorting and cosequential
processing
in,
318-322
standard I/O in, 31
Unterauer, K., 431
Update. c, 119, 120, 123, 162-
file
166
format), 130
Update.pas, 119, 120, 122. 123,
176-182
of,
480-481, 495
for handling deletions,
480-
481
Stack
explanation
use of,
in,
portability and, 141
Tag
TIFF (tagged image
explanation of, 383, 537
to handle overflow, 508-510
distribution, 454, 455
file-related
408-410
control of, 533-534
block,
balanced merging,
312
directory structure,
310-311
while writing out to file, 283,
284
Space utilization. See also
Packing density
for buckets, 526-527, 534
Two-way
compression
46-47
chained progressive
overflow, 484
explanation of, 448, 464, 494
System call interface, 74
System V UNIX, 189
311-318
UNIX
in
tape,
508
Turbo Pascal, 9Two-pass loading, 485-486
Uniform
Synonyms
tools for external,
of, 505-507, 537
turned into directory, 507,
explanation
Uniform, 495
file,
311
Tries
162
Striping, to avoid disk
Sorting
234
for indexes,
62-63
Sockets, 78, 86
Symbolic link
Sort, 319-320, 322, 327
Sort-merge programs, 318
553-557
382-383
height-balanced,
74
182
Storage, as hierarchy,
Strfuncs.c,
CD-ROM,
on
74
17, 25, 31,
of,
and insertions, 481-482
performance and, 482-483
219
193-194
Standard I/O, 31
Standardization
of data elements, 137-138
of number and text
conversion, 138-139
of physical record format,
136-137
Standish, T. A., 342
Static hashing,
STDERR,
74
J.,
343
Variable-length codes, 188-189,
219
Variable-length records
379-380
196-198
B-trees and,
Tracks
deleting,
of, 37-40, 87
organizing by sector, 41-45
per surface, 573
Transfer time, 51, 87
explanation of, 146
internal fragmentation and,
explanation
Tree structure
447
24, 25, 31,
Tools, pre, 167
Total search length, 469
Track capacity, 40
VanDoren,
application of, 4
199
methods of handling, 102
Variable order B-tree, 422-425,
437
590
VAX.
INDEX
135.
138-139
Veklerov, E., 534
of,
importance
of,
219
fit,
Writstrm.c,
Worst-fit placement strategies,
202
Virtual B-trees
explanation
Worst
373-377, 383
377
WRITE(
Writstrm.pas,
115
explanation
Webster. R. E.. 375, 377
White-space characters. 97-98
Writrec.c,
98, 99, 154-
94-95,
98, 99,
168-169
of, 31
use of, 15-18,
IIV.
94-95,
155
63-65
103-105, 107, 109,
156-157
Writrec.pas, 171-172
XDR
(external data
representation),
137-139
Yao, A. Chi-Chih, 371
Computer
Science/File Structures
Structures
Bill
Zoellick
Michael
J.
Second Edition
Avalanche Development Company
Folk National Center for Supercomputing Applications
This second edition of the leading
file structures book currently on the market has been
thoroughly revised and updated to instruct readers on the design of fast and flexible file
structures. The new edition now includes timely coverage of file structures in a UNIX
environment in addition to a new and substantial appendix on CD-ROM. Other modern file
structures such as extendible hashing methods are also explored.
This book develops a framework for approaching the design of systems to store and retrieve
information on magnetic disks and other mass storage devices. It provides a fundamental
collection of tools that
any user needs
appropriate solutions to
file
in
order to design intelligent, cost-effective, and
structure problems.
Highlights
Discusses a "toolkit" of approaches to
retrieve file records: simple indexes, paged
indexes (e.g. B-trees), variations on paged
indexes (e.g. B + trees, B trees), and hashing
Includes a
new
chapter on extendible hashing
Uses pseudocode extensively, particularly
where the procedures are complex and where
it
is
important to avoid the distractions
in actual compilable code
Emphasizes the building of conceptual tools
and retrieval of information
from files
for the design
Provides complete examples
and Turbo Pascal 6.0
in
both ANSI C
Introduces UNIX concepts and utilities that
apply directly to file structures and file
management
inherent
Second Edition
an invaluable resource for computer science professionals
UNIX environment. It will also be of interest to professionals
interested in learning about the design of file structures and the retrieval of records. Students
majoring in computer science will benefit from this book's emphasis on fundamental concepts
and its inclusion of C and UNIX.
File Structures,
using
file
and
is
data. structures in a
About the Authors
Bill
Zoellick
is
Vice President and Chief
Scientist at the
Avalanche Development
Michael J. Folk is currently a Senior Software
Engineer at the National Center for
in Boulder, Colorado, a leading
producer of text conversion software.
Previously, he was the Director of
Technology for the Alexandria Institute, a
nonprofit organization working to resolve the
Supercomputing Applications
problems associated with electronic
Computer Science for fifteen years at
Oklahoma State and Drake Universities.
Company
publishing.
writer
He
is
a frequent lecturer
on CD-ROM
and
at the University
three years
he has been responsible for developing
general purpose scientific data file formats.
of Illinois
in
Urbana. For the
Prior to this, Dr. Folk
was
last
a Professor of
issues.
90000>
Addison-Wesley Publishing Company
780201"557138
ISBN D-ED1-55713-4
You might also like
- The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good LifeFrom EverandThe Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life4/5 (6407)
- The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You AreFrom EverandThe Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are4/5 (1173)
- Never Split the Difference: Negotiating As If Your Life Depended On ItFrom EverandNever Split the Difference: Negotiating As If Your Life Depended On It4.5/5 (990)
- The Hard Thing About Hard Things: Building a Business When There Are No Easy AnswersFrom EverandThe Hard Thing About Hard Things: Building a Business When There Are No Easy Answers4.5/5 (361)
- Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space RaceFrom EverandHidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race4/5 (1015)
- Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New AmericaFrom EverandDevil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America4.5/5 (278)
- The World Is Flat 3.0: A Brief History of the Twenty-first CenturyFrom EverandThe World Is Flat 3.0: A Brief History of the Twenty-first Century3.5/5 (2283)
- Microsoft Access Practice Exam 1: Instructions To Download and Unzip The File Needed To Perform This Practice Exam100% (10)Microsoft Access Practice Exam 1: Instructions To Download and Unzip The File Needed To Perform This Practice Exam2 pages
- A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True StoryFrom EverandA Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story3.5/5 (233)
- Top 40 Oracle DBA Interview Questions and Answers - TechgoeasyNo ratings yetTop 40 Oracle DBA Interview Questions and Answers - Techgoeasy1 page
- Chapter 3 Databases and Data Warehouses3883No ratings yetChapter 3 Databases and Data Warehouses388345 pages
- Assessment - Oracle SPARC M5-32 Server Overview (WBT) - 2No ratings yetAssessment - Oracle SPARC M5-32 Server Overview (WBT) - 22 pages
- Upgrading To Ifix 5.0: Installing The Hardware KeyNo ratings yetUpgrading To Ifix 5.0: Installing The Hardware Key9 pages
- Wcdma Dual-Sim Wifi Router Technical Specification: GeneralNo ratings yetWcdma Dual-Sim Wifi Router Technical Specification: General4 pages
- Delta HMI Seminar DOP-B & DOPSoft IntroductionNo ratings yetDelta HMI Seminar DOP-B & DOPSoft Introduction29 pages
- UL Cumulative Quality Ratio Period Start Time BSC Name BCF Name BTS Name TRX Name TRX IdNo ratings yetUL Cumulative Quality Ratio Period Start Time BSC Name BCF Name BTS Name TRX Name TRX Id31 pages
