Transaction Management

CSE 444: Database Internals

Two parts: Concurrency control: ACID Recovery from crashes: ACID We already discussed concurrency control You will get to implement locking in lab3 Today, we start recovery More details in the book in Chapter 17
Magda Balazinska - CSE 444, Spring 2012 1 Magda Balazinska - CSE 444, Spring 2012 2

Lectures 17 Transactions: Recovery

Problem Illustration
Client 1: START TRANSACTION INSERT INTO SmallProduct(name, price) SELECT pname, price FROM Product WHERE price <= 0.99 Crash ! DELETE Product WHERE price <=0.99 COMMIT

Recovery
From which events below can DBMS recover ? Wrong data entry Disk failure Fire / earthquake / etc. Systems crashes
Software errors Power failures

What do we do now?
Magda Balazinska - CSE 444, Spring 2012 3 Magda Balazinska - CSE 444, Spring 2012 4

Recovery
Type of Crash
Wrong data entry Disk crashes Fire or other major disaster System failures

System Failures
Each transaction has internal state When system crashes, internal state is lost
Dont know which parts executed and which didnt Need ability to undo and redo

Prevention
Constraints and Data cleaning Redundancy: RAID, backup, replica Redundancy: Replica far away DATABASE RECOVERY
5 Magda Balazinska - CSE 444, Spring 2012 6

Most frequent

Magda Balazinska - CSE 444, Spring 2012

Buffer Manager Review

READ WRITE
Page requests from higher-level code Files and access methods Buffer pool manager Buffer pool Main memory
choice of frame dictated by replacement policy

Buffer Manager Review

Enables higher layers of the DBMS to assume that needed data is in main memory Caches data in memory. When crash occurs:
Problem if committed data was not yet written to disk Problem if uncommitted data was flushed to disk

Disk page Free frame

INPUT OUTPUT
Disk = collection of blocks

Disk

1 page corresponds to 1 disk block

7 Magda Balazinska - CSE 444, Spring 2012 8

Data must be in RAM for DBMS to operate on it! Buffer pool = table of <frame#, pageid> pairs

Buffer Manager
DBMSs build their own buffer manager and dont rely on the OS. Why? Reason 1: Performance
DBMS may be able to anticipate access patterns Hence, may also be able to perform prefetching May select better page replacement policy

Transactions
Assumption: db composed of elements
Usually 1 element = 1 block Can be smaller (=1 record) or larger (=1 relation)

Assumption: each transaction reads/writes some elements

Reason 2: Correctness
DBMS needs fine grained control for transactions Needs to force pages to disk for recovery purposes
Magda Balazinska - CSE 444, Spring 2012 9 Magda Balazinska - CSE 444, Spring 2012 10

Primitive Operations of Transactions

READ(X,t)
copy element X to transaction local variable t

Example
START TRANSACTION READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); COMMIT;
11

WRITE(X,t)
copy transaction local variable t to element X

INPUT(X)
read element X to memory buffer

Atomicity: BOTH A and B are multiplied by 2

OUTPUT(X)
write element X to disk

Magda Balazinska - CSE 444, Spring 2012

Magda Balazinska - CSE 444, Spring 2012

12

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

Mem A

Mem B

Disk A 8

Disk B 8

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A)

Mem A 8

Mem B

Disk A 8

Disk B 8

Magda Balazinska - CSE 444, Spring 2012

13

OUTPUT(B)

Magda Balazinska - CSE 444, Spring 2012

14

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

Mem A 8

Mem B

Disk A 8 8 8

Disk B 8 8 8

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A)

Mem A 8

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8

8 16

8 8

8 16 16

8 8 16

Magda Balazinska - CSE 444, Spring 2012

15

OUTPUT(B)

Magda Balazinska - CSE 444, Spring 2012

16

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

Mem A 8

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8 8

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A)

Mem A 8

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8 8 8 8

8 16 16 16

8 8 16 16 8

8 16 16 16 8 16

8 8 16 16 16 16 8 8 8

8 8 8

Magda Balazinska - CSE 444, Spring 2012

17

OUTPUT(B)

Magda Balazinska - CSE 444, Spring 2012

18

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

Mem A 8

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8 8 8 8 8

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A)

Mem A 8

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8 8 8 8 8 8
20

8 16 16 16 8 16 16

8 8 16 16 16 16 16 8 8 8 16

8 16 16 16 8 16 16 16

8 8 16 16 16 16 16 16 8 8 8 16 16

8 8 8 8

8 8 8 8 16

Magda Balazinska - CSE 444, Spring 2012

19

OUTPUT(B)

Magda Balazinska - CSE 444, Spring 2012

READ(A,t); t := t*2; WRITE(A,t); READ(B,t); t := t*2; WRITE(B,t); Transaction Buffer pool Disk

Action INPUT(A)
Disk B 8 8 8 8 8 8 8 8 8 16
21

t 8 16 16 16 8 16 16 16 16

Mem A 8 8 8 16 16 16 16 16 16 16

Mem B

Disk A 8 8 8 8

Disk B 8 8 8 8 8 8 8 8 8

Action INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

Mem A 8

Mem B

Disk A 8 8 8 8

READ(A,t) t:=t*2 WRITE(A,t) INPUT(B) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B)

8 16 16 16 8 16 16 16

8 8 16 16 16 16 16 16 8 8 8 16 16

8 8 8 16 16 16

8 8 8 8 16 16

8 8 8 8 16

Crash !
16

16 Magda Balazinska - CSE 444, Spring 2012 16 16 16

Crash occurs after OUTPUT(A), before OUTPUT(B) We lose atomicity

22

Buffer Manager Policies

STEAL or NO-STEAL
Can an update made by an uncommitted transaction overwrite the most recent committed value of a data item on disk?

Solution: Use a Log

Log = append-only file containing log records Note: multiple transactions run concurrently, log records are interleaved After a system crash, use log to:
Redo some transactions that did commit Undo other transactions that did not commit

FORCE or NO-FORCE
Should all updates of a transaction be forced to disk before the transaction commits?

Easiest for recovery: NO-STEAL/FORCE Highest performance: STEAL/NO-FORCE

Magda Balazinska - CSE 444, Spring 2012 23

Three kinds of logs: undo, redo, undo/redo

Magda Balazinska - CSE 444, Spring 2012

24

Undo Logging
Log records <START T>
Transaction T has begun

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) 8 16 16

8 8 8 16

8 8 8 8

8 8 8 8 <T,A,8>

<COMMIT T>
T has committed

INPUT(B)
READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B) COMMIT
25

16
8 16 16 16 16

16
16 16 16 16 16

8
8 8 16 16 16

8
8 8 8 16 16

8
8 8 8 8 16 <COMMIT T>
26

<ABORT T>
T has aborted

<T,B,8>

<T,X,v> -- Update record

T has updated element X, and its old value was v
Magda Balazinska - CSE 444, Spring 2012

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) 8 16 16

8 8 8 16

8 8 8 8

8 8 8 8 <T,A,8>

INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) 8 16 16

8 8 8 16

8 8 8 8

8 8 8 8 <T,A,8>

INPUT(B)
READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B) COMMIT

16
8 16 16 16 16

16
16 16 16 16 16

8
8 8 16 16 16

8
8 8 8 16 16

8
8 8 8 8 16 <T,B,8>

INPUT(B)
READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B) COMMIT

16
8 16 16 16 16

16
16 16 16 16 16

8
8 8 16 16 16

8
8 8 8 16 16

8
8 8 8 8 16 <COMMIT T> <T,B,8>

Crash !
<COMMIT T>
27

WHAT DO WE DO ?

WHAT DO WE DO ?

Crash !

28

After Crash
In the first example:
We UNDO both changes: A=8, B=8 The transaction is atomic, since none of its actions have been executed

Undo-Logging Rules
U1: If T modifies X, then <T,X,v> must be written to disk before OUTPUT(X) U2: If T commits, then OUTPUT(X) must be written to disk before <COMMIT T> Hence: OUTPUTs are done early, before the transaction commits

In the second example

We dont undo anything The transaction is atomic, since both its actions have been executed

Magda Balazinska - CSE 444, Spring 2012

29

Magda Balazinska - CSE 444, Spring 2012

30

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

Recovery with Undo Log

After systems crash, run recovery manager

INPUT(A) READ(A,t) t:=t*2 WRITE(A,t) 8 16 16

8 8 8 16

8 8 8 8

8 8 8 8 <T,A,8>

INPUT(B)
READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A) OUTPUT(B) COMMIT

16
8 16 16 16 16

16
16 16 16 16 16

8
8 8 16 16 16

8
8 8 8 16 16

8
8 8 8 8 16 <COMMIT T>
31

Idea 1. Decide for each transaction T whether it is completed or not

<START T>.<COMMIT T>. = yes <START T>.<ABORT T>. = yes <START T> = no

<T,B,8>

Idea 2. Undo all modifications by incomplete transactions

Magda Balazinska - CSE 444, Spring 2012 32

Recovery with Undo Log

Recovery manager: Read log from the end; cases:
<COMMIT T>: mark T as completed <ABORT T>: mark T as completed <T,X,v>: if T is not completed then write X=v to disk else ignore <START T>: ignore

Recovery with Undo Log

<T6,X6,v6> <START T5> <START T4> <T1,X1,v1> <T5,X5,v5> <T4,X4,v4> <COMMIT T5> <T3,X3,v3> <T2,X2,v2> Question1 in class: Which updates are undone ? Question 2 in class: What happens if there is a second crash, during recovery ? Question 3 in class: How far back do we need to read in the log ?
34

Magda Balazinska - CSE 444, Spring 2012

33

crash

Recovery with Undo Log

Note: all undo commands are idempotent
If we perform them a second time, no harm done E.g. if there is a system crash during recovery, simply restart recovery from scratch

Recovery with Undo Log

When do we stop reading the log ? We cannot stop until we reach the beginning of the log file This is impractical Instead: use checkpointing

Magda Balazinska - CSE 444, Spring 2012

35

Magda Balazinska - CSE 444, Spring 2012

36

Checkpointing
Checkpoint the database periodically Stop accepting new transactions Wait until all current transactions complete Flush log to disk Write a <CKPT> log record, flush Resume transactions

Undo Recovery with Checkpointing

<T9,X9,v9> (all completed) <CKPT> <START T2> <START T3 <START T5> <START T4> <T1,X1,v1> <T5,X5,v5> <T4,X4,v4> <COMMIT T5> <T3,X3,v3> <T2,X2,v2>

During recovery, Can stop at first <CKPT>

other transactions

transactions T2,T3,T4,T5

Magda Balazinska - CSE 444, Spring 2012

37

38

Nonquiescent Checkpointing
Problem with checkpointing: database freezes during checkpoint Would like to checkpoint while database is operational Idea: nonquiescent checkpointing

Nonquiescent Checkpointing
Write a <START CKPT(T1,,Tk)> where T1,,Tk are all active transactions. Flush log to disk Continue normal operation When all of T1,,Tk have completed, write <END CKPT>. Flush log to disk

Quiescent = being quiet, still, or at rest; inactive Non-quiescent = allowing transactions to be active
Magda Balazinska - CSE 444, Spring 2012 39

Magda Balazinska - CSE 444, Spring 2012

40

Undo Recovery with Nonquiescent Checkpointing

<START CKPT T4, T5, T6> <END CKPT>

Implementing ROLLBACK
Recall: a transaction can end in COMMIT or ROLLBACK Idea: use the undo-log to implement ROLLBACK How ?
LSN = Log Sequence Number Log entries for the same transaction are linked, using the LSNs Read log in reverse, using LSN pointers

During recovery, Can stop at first <CKPT>

earlier transactions plus T4, T5, T6

T4, T5, T6, plus later transactions

later transactions
41 Magda Balazinska - CSE 444, Spring 2012 42

Q: why do we need <END CKPT> ?

Redo Logging
Log records <START T> = transaction T has begun <COMMIT T> = T has committed <ABORT T>= T has aborted <T,X,v>= T has updated element X, and its new value is v

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

READ(A,t) t:=t*2 WRITE(A,t) READ(B,t) t:=t*2 WRITE(B,t)

8 16 16 8 16 16

8 8 16 16 16 16 8 8 16

8 8 8 8 8 8

8 8 8 8 8 8 <T,B,16> <COMMIT T> <T,A,16>

OUTPUT(A) OUTPUT(B)
Magda Balazinska - CSE 444, Spring 2012 43

16 16

16 16

16 16

16 16

8 16
44

Magda Balazinska - CSE 444, Spring 2012

Redo-Logging Rules
R1: If T modifies X, then both <T,X,v> and <COMMIT T> must be written to disk before OUTPUT(X)

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

READ(A,t) t:=t*2 WRITE(A,t) READ(B,t) t:=t*2 WRITE(B,t)

8 16 16 8 16 16

8 8 16 16 16 16 8 8 16

8 8 8 8 8 8

8 8 8 8 8 8 <T,B,16> <COMMIT T> <T,A,16>

Hence: OUTPUTs are done late

OUTPUT(A) OUTPUT(B)
Magda Balazinska - CSE 444, Spring 2012 45

16 16

16 16

16 16

16 16

8 16
46

Magda Balazinska - CSE 444, Spring 2012

Recovery with Redo Log

After systems crash, run recovery manager Step 1. Decide for each transaction T whether it is completed or not
<START T>.<COMMIT T>. = yes <START T>.<ABORT T>. = yes <START T> = no

Recovery with Redo Log

<START T1> <T1,X1,v1> <START T2> <T2, X2, v2> <START T3> <T1,X3,v3> <COMMIT T2> <T3,X4,v4> <T1,X5,v5>

Step 2. Read log from the beginning, redo all updates of committed transactions

Magda Balazinska - CSE 444, Spring 2012

47

48

Nonquiescent Checkpointing
Write a <START CKPT(T1,,Tk)> where T1,,Tk are all active transactions Flush to disk all blocks of committed transactions (dirty blocks), while continuing normal operation When all blocks have been written, write <END CKPT>

Redo Recovery with Nonquiescent Checkpointing

<START T1> <COMMIT T1> <START T4> <START CKPT T4, T5, T6> <END CKPT> <START CKPT T9, T10>

Step 1: look for The last <END CKPT>

All OUTPUTs of T1 are known to be on disk

Magda Balazinska - CSE 444, Spring 2012

49

Cannot use

Step 2: redo from the earliest start of T4, T5, T6 ignoring transactions committed earlier
50

Comparison Undo/Redo
Undo logging:
Steal/Force OUTPUT must be done early If <COMMIT T> is seen, T definitely has written all its data to disk (hence, dont need to redo) inefficient No-Steal/No-Force OUTPUT must be done late If <COMMIT T> is not seen, T definitely has not written any of its data to disk (hence there is not dirty data on disk, no need to undo) inflexible

Undo/Redo Logging
Log records, only one change <T,X,u,v>= T has updated element X, its old value was u, and its new value is v

Redo logging

Would like more flexibility on when to OUTPUT: undo/redo logging (next) Steal/No-Force
Magda Balazinska - CSE 444, Spring 2012 51 Magda Balazinska - CSE 444, Spring 2012 52

Undo/Redo-Logging Rule
UR1: If T modifies X, then <T,X,u,v> must be written to disk before OUTPUT(X) Note: we are free to OUTPUT early or late relative to <COMMIT T>

Action

Mem A

Mem B

Disk A

Disk B

Log <START T>

REAT(A,t) t:=t*2 WRITE(A,t) READ(B,t) t:=t*2 WRITE(B,t) OUTPUT(A)

8 16 16 8 16 16 16

8 8 16 16 16 16 16 8 8 16 16

8 8 8 8 8 8 16

8 8 8 8 8 8 8 <COMMIT T> <T,B,8,16> <T,A,8,16>

OUTPUT(B)
Magda Balazinska - CSE 444, Spring 2012 53

16

16

16

16

16

Can OUTPUT whenever we want: before/after COMMIT 54

Recovery with Undo/Redo Log

After systems crash, run recovery manager Redo all committed transaction, top-down Undo all uncommitted transactions, bottom-up

Recovery with Undo/Redo Log

<START T1> <T1,X1,v1> <START T2> <T2, X2, v2> <START T3> <T1,X3,v3> <COMMIT T2> <T3,X4,v4> <T1,X5,v5>

Magda Balazinska - CSE 444, Spring 2012

55

Magda Balazinska - CSE 444, Spring 2012

56

10