Chapter 13: Data Storage Structures

Database System Concepts, 7th Ed.

©Silberschatz, Korth and Sudarshan
See www.db-book.com for conditions on re-use
 File Organization

 The database is stored as a collection of files.

Each file is a sequence of records. A record is a
sequence of fields.
 One approach
• Assume record size is fixed
• Each file has records of one particular type
only
• Different files are used for different relations
This case is easiest to implement.
 Assume that records are smaller than a disk block

Database System Concepts - 7th Edition 13.2 ©Silberschatz, Korth and Sudarshan
 Fixed-Length Records

 Simple approach:
 Store record i starting from byte n  (i – 1), where n is the size of
each record.
 Record access is simple but records may cross blocks
 Modification: do not allow records to cross block boundaries

Database System Concepts - 7th Edition 13.3 ©Silberschatz, Korth and Sudarshan
 Fixed-Length Records
 Deletion of record i: alternatives:
1. move records i + 1, . . ., n to i, . . . , n – 1
2. move record n to i
3. do not move records, but link all free records on a free list
Record 3 is to be deleted Record 3 deleted

Database System Concepts - 7th Edition 13.4 ©Silberschatz, Korth and Sudarshan
 Fixed-Length Records
 Deletion of record i: alternatives:
1. move records i + 1, . . ., n to i, . . . , n – 1
2. move record n to i
3. do not move records, but link all free records on a free list

Record 3 is to be deleted Record 3 deleted and replaced by record 11

Database System Concepts - 7th Edition 13.5 ©Silberschatz, Korth and Sudarshan
 Fixed-Length Records
 Deletion of record i: alternatives:
 move records i + 1, . . ., n to i, . . . , n – 1
 move record n to i
 do not move records, but link all free records on a free list

Database System Concepts - 7th Edition 13.6 ©Silberschatz, Korth and Sudarshan
 Variable-Length Records
 Variable-length records arise in database systems in several ways:
 Storage of multiple record types in a file.
 Record types that allow variable lengths for one or more fields such as
strings (varchar)
 Record types that allow repeating fields (used in some older data models).
 Attributes are stored in order
 Variable length attributes represented by fixed size (offset, length), with
actual data stored after all fixed length attributes
 Null values represented by null-value bitmap
 0 indicates that the field has a valid (non-null) value.
 1 indicates that the field contains a NULL value.

Database System Concepts - 7th Edition 13.7 ©Silberschatz, Korth and Sudarshan
 Variable-Length Records: Slotted Page Structure

 Slotted page header contains:

 number of record entries
 end of free space in the block
 location and size of each record
 Records can be moved around within a page to keep them contiguous
with no empty space between them; entry in the header must be updated.
 Pointers should not point directly to record — instead they should point to
the entry for the record in header.

Database System Concepts - 7th Edition 13.8 ©Silberschatz, Korth and Sudarshan
 Storing Large Objects
 E.g., Binary Large Object(BLOB) or
Character Large Object(CLOB) types
 Records must be smaller than pages
 Alternatives:
• Store as files in file systems
• Store as files managed by database
• Break into pieces and store in multiple tuples
in separate relation
 PostgreSQL TOAST
(The Oversized-Attribute Storage
Technique)
Database System Concepts - 7th Edition 13.9 ©Silberschatz, Korth and Sudarshan
 Organization of Records in Files
 Heap – record can be placed anywhere in the file where there is
space
 Sequential – store records in sequential order, based on the value
of the search key of each record
 In a multi-table clustering file organization records of several
different relations can be stored in the same file
 Motivation: store related records on the same block to minimize
I/O
 B+-tree file organization
 Ordered storage even with inserts/deletes
 More on this in next chapter (Chapter 14)
 Hashing – a hash function computed on search key; the result
specifies in which block of the file the record should be placed
 More on this in next chapter ( Chapter 14 )

Database System Concepts - 7th Edition 13.10 ©Silberschatz, Korth and Sudarshan
 Heap File Organization
 It is the simplest and most basic type of organization.
 It works with data blocks.
 In heap file organization, the records are inserted at the file's
end.
 When the records are inserted, it doesn't require the sorting and
ordering of records.
 When the data block is full, the new record is stored in some
other block. This new data block need not to be the very next
data block, but it can select any data block in the memory to
store new records.
 The heap file is also known as an unordered file.
 In the file, every record has a unique id, and every page in a file
is of the same size.
 It is the DBMS responsibility to store and manage the new
records.
Database System Concepts - 7th Edition 13.11 ©Silberschatz, Korth and Sudarshan
 Heap File Organization

Database System Concepts - 7th Edition 13.12 ©Silberschatz, Korth and Sudarshan
 Heap File Organization
 Records can be placed anywhere in the file where there is free
space
 Records usually do not move once allocated
 Important to be able to efficiently find free space within file
 Free-space map
• Array with 1 entry per block. Each entry is a few bits to a byte,
and records fraction of block that is free
• A free-space map for a file with 16 blocks is

• We assume that 3 bits are used to store the occupancy

fraction; the value at position i should be divided by 8 to get
the free-space fraction for block i.
• A value of 4 indicates that at least 4/8th of the space in the
block is free.

Database System Concepts - 7th Edition 13.13 ©Silberschatz, Korth and Sudarshan
 Heap File Organization
• To find a block to store a new record of a given size, the
database can scan the free-space map to find a block
that has enough free space to store that record.
• If there is no such block, a new block is allocated for the
relation.
• While such a scan is much faster than actually
fetching blocks to find free space, it can still be very
slow for large files.
• Can have second-level free-space map
• In example below, each entry stores maximum from 4
entries of first-level free-space map
 Free space map written to disk periodically, OK to have
wrong (old) values for some entries (will be detected and
fixed)
Database System Concepts - 7th Edition 13.14 ©Silberschatz, Korth and Sudarshan
 Heap File Organization
Advantages of Heap File Organization
 Very good method of file organization for bulk insertion. i.e.; when
there is a huge number of data needs to load into the database at
a time, then this method of file organization is best suited. They are
simply inserted one after the other in the memory blocks.
 It is suited for very small files as the fetching of records is faster in
them. As the file size grows, linear search for the record becomes
time consuming.
Disadvantages of Heap File Organization
 If the database is very large then searching, updating or deleting of
record will be time-consuming because there is no sorting or
ordering of records. In the heap file organization, we need to check
all the data until we get the requested record.
 Proper memory management is required to boost the performance.
Otherwise there would be lots of unused memory blocks lying and
memory size will simply be growing.

Database System Concepts - 7th Edition 13.15 ©Silberschatz, Korth and Sudarshan
 Sequential File Organization
 Suitable for applications that require sequential processing of the entire file
 The records in the file are ordered by a search-key

Database System Concepts - 7th Edition 13.16 ©Silberschatz, Korth and Sudarshan
 Sequential File Organization (Cont.)
 Deletion – use pointer chains
 Insertion –locate the position where the record is to be inserted
 if there is free space insert there
 if no free space, insert the record in an overflow block
 In either case, pointer chain must be updated
 Need to reorganize the file
from time to time to restore
sequential order

Database System Concepts - 7th Edition 13.17 ©Silberschatz, Korth and Sudarshan
 Multitable Clustering File Organization
Store several relations in one file using a multi-table clustering file
organization

department

instructor

Multi-table clustering
of department and
instructor

Database System Concepts - 7th Edition 13.18 ©Silberschatz, Korth and Sudarshan
 Multitable Clustering File Organization (cont.)

 good for queries involving department ⨝ instructor,

and for queries involving one single department and
its instructors
 bad for queries involving only department
 results in variable size records
 Can add pointer chains to link records of a particular
relation

Database System Concepts - 7th Edition 13.19 ©Silberschatz, Korth and Sudarshan
 Partitioning
 Table partitioning: Records in a relation can be partitioned into
smaller relations that are stored separately
 E.g., transaction relation may be partitioned into
transaction_2018, transaction_2019, etc.
 Queries written on transaction must access records in all
partitions
• Unless query has a selection such as year=2019, in which
case only one partition in needed
 Partitioning
• Reduces costs of some operations such as free space
management
• Allows different partitions to be stored on different storage
devices
 E.g., transaction partition for current year on SSD, for
older years on magnetic disk

Database System Concepts - 7th Edition 13.20 ©Silberschatz, Korth and Sudarshan
Database System Concepts - 7th Edition 13.21 ©Silberschatz, Korth and Sudarshan
 Data Dictionary Storage
The Data dictionary (also called system catalog) stores
metadata; that is, data about data, such as
 Information about relations
 names of relations
 names, types and lengths of attributes of each relation
 names and definitions of views
 integrity constraints
 User and accounting information, including passwords
 Statistical and descriptive data
 number of tuples in each relation
 Physical file organization information
 How relation is stored (sequential/hash/…)
 Physical location of relation
 Information about indices

Database System Concepts - 7th Edition 13.22 ©Silberschatz, Korth and Sudarshan
 Relational Representation of System Metadata

 Relational
representation
on disk
 Specialized data
structures
designed for
efficient access,
in memory

Database System Concepts - 7th Edition 13.23 ©Silberschatz, Korth and Sudarshan
 Storage Access
 Blocks are units of both storage allocation and data transfer.
 Database system seeks to minimize the number of block transfers
between the disk and memory. We can reduce the number of disk
accesses by keeping as many blocks as possible in main memory.
 Buffer – portion of main memory available to store copies of disk
blocks.
 Buffer manager – subsystem responsible for allocating buffer
space in main memory.

Database System Concepts - 7th Edition 13.24 ©Silberschatz, Korth and Sudarshan
 Buffer Manager
 Programs call on the buffer manager when they need a
block from disk.
• If the block is already in the buffer, buffer manager
returns the address of the block in main memory
• If the block is not in the buffer, the buffer manager
 Allocates space in the buffer for the block
• Replacing (throwing out) some other block, if
required, to make space for the new block.
• Replaced block written back to disk only if it was
modified since the most recent time that it was
written to/fetched from the disk.
 Reads the block from the disk to the buffer, and returns
the address of the block in main memory to requester.

Database System Concepts - 7th Edition 13.25 ©Silberschatz, Korth and Sudarshan
 Buffer Manager
 Buffer replacement strategy
 Pinned block: memory block that is not allowed to be written back to disk
• Pin done before reading/writing data from a block
• Unpin done when read /write is complete
• Multiple concurrent pin/unpin operations possible
 Keep a pin count, buffer block can be evicted only if pin count = 0
 Shared and exclusive locks on buffer
• Needed to prevent concurrent operations from reading page contents
as they are moved/reorganized, and to ensure only one
move/reorganize at a time
• Readers get shared lock, updates to a block require exclusive lock
• Locking rules:
 Only one process can get exclusive lock at a time
 Shared lock cannot be concurrently with exclusive lock
 Multiple processes may be given shared lock concurrently

Database System Concepts - 7th Edition 13.26 ©Silberschatz, Korth and Sudarshan
 Buffer-Replacement Policies
 Most operating systems replace the block least recently used
(LRU strategy)
• Idea behind LRU – use past pattern of block references as a
predictor of future references
• LRU can be bad for some queries
 Queries have well-defined access patterns (such as sequential
scans), and a database system can use the information in a user’s
query to predict future references
 Mixed strategy with hints on replacement strategy provided
by the query optimizer is preferable
 Example of bad access pattern for LRU: when computing the join
of 2 relations r and s by a nested loops

for each tuple tr of r do

for each tuple ts of s do
if the tuples tr and ts match …

Database System Concepts - 7th Edition 13.27 ©Silberschatz, Korth and Sudarshan
 for each tuple tr of r do
for each tuple ts of s do
if the tuples tr and ts match.
Why this is a Bad Access Pattern for LRU?
 LRU (Least Recently Used) page replacement algorithm
removes the least recently used page when the buffer is
full.
 This join pattern results in poor locality of reference,
causing excessive page evictions.
 Assume r and s are both large and do not fit in memory.
 Each iteration reads a new page of r, but loops through all
of s.
 Since s is read repeatedly for each tuple of r, but the LRU
cache keeps evicting pages of s, we end up with a high
page miss rate.

Database System Concepts - 7th Edition 13.28 ©Silberschatz, Korth and Sudarshan
 for each tuple tr of r do
for each tuple ts of s do
if the tuples tr and ts match.
 Example : r has 1000 pages, s has 5000 pages
• Buffer can hold only 10 pages
• A new page of r is brought into memory.
• All 5000 pages of s are accessed in the inner loop,
but only 10 can stay in the buffer.
• By the time the next tuple of r is processed, LRU has
already evicted the previously loaded pages of s.
• Each page of s gets reloaded 1000 times!
• This results in a massive number of disk I/Os,
making LRU perform poorly.
• This leads to thrashing

Database System Concepts - 7th Edition 13.29 ©Silberschatz, Korth and Sudarshan
 Buffer-Replacement Policies (Cont.)
 Toss-immediate strategy – frees the space occupied by a block as
soon as the final tuple of that block has been processed
 Most recently used (MRU) strategy – system must pin the block
currently being processed. After the final tuple of that block has been
processed, the block is unpinned, and it becomes the most recently
used block.
 Buffer manager can use statistical information regarding the probability
that a request will reference a particular relation
• E.g., the data dictionary is frequently accessed. Heuristic: keep
data-dictionary blocks in main memory buffer
 Operating system or buffer manager may reorder writes
• Can lead to corruption of data structures on disk
 E.g., linked list of blocks with missing block on disk
 File systems perform consistency check to detect such situations

• Careful ordering of writes can avoid many such problems

Database System Concepts - 7th Edition 13.30 ©Silberschatz, Korth and Sudarshan
 Optimization of Disk Block Access (Cont.)
 Buffer managers support forced output of blocks for the
purpose of recovery ( Read Chapter 19 more details)
 Nonvolatile write buffers speed up disk writes by writing
blocks to a non-volatile RAM or flash buffer immediately
• Writes can be reordered to minimize disk arm movement
 Log disk – a disk devoted to writing a sequential log of block
updates
• Used exactly like nonvolatile RAM
 Write to log disk is very fast since no seeks are required
 Journaling file systems write data in order to NV-RAM or log
disk
• Reordering without journaling: risk of corruption of file
system data

Database System Concepts - 7th Edition 13.31 ©Silberschatz, Korth and Sudarshan
 Column-Oriented Storage
 Also known as columnar representation
 Store each attribute of a relation separately
 Example

Database System Concepts - 7th Edition 13.32 ©Silberschatz, Korth and Sudarshan
Database System Concepts - 7th Edition 13.33 ©Silberschatz, Korth and Sudarshan
 Columnar Representation
 Benefits:
• Reduced IO if only some attributes are accessed
• Improved CPU cache performance
• Improved compression
• Vector processing on modern CPU architectures
 Drawbacks
• Cost of tuple reconstruction from columnar representation
• Cost of tuple deletion and update
• Cost of decompression
 Columnar representation found to be more efficient for decision support
than row-oriented representation
 Traditional row-oriented representation preferable for transaction
processing
 Some databases support both representations
• Called hybrid row/column stores

Database System Concepts - 7th Edition 13.34 ©Silberschatz, Korth and Sudarshan
 Columnar File Representation
 Optimized Row Columnar(ORC) and
Parquet: file formats with columnar storage
inside file.
 ORC file is a data storage format designed
for Hadoop and other Big Data processing
systems.
 Apache Parquet is a columnar open source
storage format that can efficiently store
nested data which is widely used in Hadoop
and Spark.
Database System Concepts - 7th Edition 13.35 ©Silberschatz, Korth and Sudarshan
 Columnar File Representation

Database System Concepts - 7th Edition 13.36 ©Silberschatz, Korth and Sudarshan
 Row-Based Storage vs. Columnar Storage
ID Name Age Salary
1 John 25 50000
2 Arun 30 70000
3 Aman 28 65000

In a row-oriented storage it is stored on disk as:

1, John, 25, 50K | 2, Arun, 30, 70K | 3, Aman, 28, 65K
In columnar storage, data is stored column-wise:
ID: 1, 2, 3 | Name: John, Arun, Aman |Age: 25, 30, 28
Salary: 50K, 70K, 65K

This structure makes column scans and aggregations faster.

Database System Concepts - 7th Edition 13.37 ©Silberschatz, Korth and Sudarshan
 Traditional Row-Based Storage vs. Columnar Storage

Row-Based Storage Columnar Storage

Aspect
(Row Store) (Column Store)
Stores all attributes of a Stores values of each
Storage Format
row together column together
Good for OLAP
Good for OLTP queries
Read Efficiency queries
(single-row lookups)
(aggregations)
Slower for writes but
Faster for
Write Performance better for read-heavy
inserts/updates
workloads
Highly effective
Compression Less effective (similar data in
columns)
Apache Parquet,
Example MySQL, PostgreSQL
Amazon Redshift,
Databases (row-oriented)
ClickHouse
Database System Concepts - 7th Edition 13.38 ©Silberschatz, Korth and Sudarshan
 Storage Organization in Main-Memory Databases

 Can store records directly in

memory without a buffer
manager
 Column-oriented storage can
be used in memory for
decision support applications
• Compression reduces
memory requirement

Database System Concepts - 7th Edition 13.39 ©Silberschatz, Korth and Sudarshan
 End of Chapter 13

Database System Concepts - 7th Edition 13.40 ©Silberschatz, Korth and Sudarshan