Session: C06

LOBs : Stronger and Faster

in DB2 9 for z/OS (Part 2 of 2)

Davy Goethals
ArcelorMittal

May 20, 2008 • 10:30 a.m. – 11:30 a.m.

Platform: DB2 for z/OS

In the second part of this presentation, we will discuss the way DB2 utilities
have been enhanced in DB2 9 for z/OS to fully support table spaces that
contain tables with LOB columns.
Much of this information is also applicable to the DB2 9 XML table spaces.

1
 Overview
• Loading and unloading LOB data using file
reference variables
• Enhancements to DSNTIAUL
• Online REORG of LOB table spaces
• Making consistent backups using COPY
• Recovery of LOB data using RECOVER/REBUILD
• Online CHECK utilities in DB2 9
• Altering tables with LOB data

Large Objects were introduced in DB2 V6 and have really evolved since then.
This presentation provides experiences and lessons learned while writing the
IBM Redbook 'LOBs with DB2 for z/OS: Stronger and Faster‘ (SG24 –
7270 - 00) . It looks at the new utility features coming along with DB2 9,
finally ensuring full support for LOBs. Please note that DB2 data management
and application enhancements for LOBs in DB2 9 are covered in "LOBs :
Stronger and Faster in DB2 9 for z/OS - Part 1 of 2"

2
 LOBs : Stronger and Faster in DB2 9

Loading and Unloading LOB data using

File Reference variables

In this section we will discuss the use of File Reference variables in the DB2
LOAD and UNLOAD utilities.

3
 Lob support in DB2 : architecture
Base table space
base table
key col col2 col3 rowid lob col
key a data a data a rowid a -
key b data b data b rowid b -

Lob table space

auxiliary table
rowid lob data
rowid a lob data a
rowid b lob data b
auxiliary index

Although a lob column is conceptually part of the base table, it is physically stored in a
separate table which is called the auxiliary table for that lob column. This auxiliary table
resides in a separate table space which is called the lob table space for that lob column.
Both base table space and lob table space must reside in the same database.

If the base table space is a partitioned table space , each lob column has a separate
auxiliary table in a separate lob table space for each partition. So if I have a partitioned
table space with a base table in it with 5 partitions and 2 lob columns, I will end up with 10
auxiliary tables in 10 lob table spaces.

The link between the base table and auxiliary table is done by a ROWID column. For
quickly finding the lob data in a lob table space, a special index called the auxiliary index
must be created on the auxiliary table.

There are 2 main reasons for separating the lob data from the data of the other columns :
performance (think about table space scans on the base table) and the fact that the maximum
row size of a classic table is 32K.

4
 Loading Lob data in DB2 for z/OS
key col col2 col3 rowid lob col
key a data a data a rowid a -
key b data b data b rowid b -
lobs z/OS host
Cobol host appl DB2 table
drda
file

ftp
client appl Java or UDB import

client

In practice, most lob data will be created on a client platform

Conceptually, there are 2 ways for loading this data into a DB2 table on the
mainframe z/OS server.

1) send the data with a file transfer to the host and LOAD the data in the DB2 table
with a host application (written in COBOL,PL/1…) or DB2 utility.

2) send the data via drda to the host DB2 table with a client application (written in
C, Visual Basic, Java…) or a UDB import utility

5
 Lob support in DB2
• LOAD/UNLOAD utility: restrictions before PK10278

• Only SLOBs allowed

• Always unload/load of base table (not auxiliary table)
• Lob columns treated as normal data columns
• Normal data and lob data in same input/output record
with a limit of 32K (MVS limit for sequential data set)
• Not sufficient in most cases !

• No cross loading (INCURSOR) before PQ90263

• To load data from another DB2 table

The use of File Reference variables in DB2 9 utilities was also retrofitted earlier to
DB2 V7 and DB2 V8 through apar PK10278 (may 2006) . Before it was only possible
to load or unload lob data from/to a sequential file when the total input record size,
containing both normal columns and lob columns , was smaller than 32 k .
In most cases this 32k limit is not enough as it limits the use of Lobs to Slobs .

Support for the cross loader for Lobs > 32K was introduced with PQ90263 (may
2005)

6
 Unload Lobs using file ref variables
• Unload lob data to separate file instead of the
normal unload file (with max 32K record length)
• Member of a pds or pdse
• Hfs file
• Store file name in the normal unload file as char
or varchar field
• 1 separate file per lob value

UNLOAD is changed to store the value of a LOB column in a separate file and
record the name of the file in the unloaded record of the base table.

7
 Unload Lobs using file ref variables
• Implemented through :
• New keywords in the field specification for char
and varchar (file reference variables)
• BLOBF template name
• CLOBF template name
• DBCLOBF template name
• New keywords in TEMPLATE definition :
• DIR
• DSNTYPE
• (&UNIQ) or /&UNIQ. automatically appended

The new file reference variables BLOBF,CLOBF and DBCLOBF specify

that the output field in the normal unload file will contain the name of a file
where the actual LOB value will be stored (in CHAR or VARCHAR format)
For BLOBF the BLOB will be stored without CCSID conversions.
For CLOBF and DBCLOBF the CLOB or DBCLOB will be stored with any
required CCSID conversions specified in the UNLOAD clause
If you use a TEMPLATE to dynamically allocate the output lob files the new
keyword DSNTYPE allows you to specify the data set type LIBRARY, PDS
or HFS. If you omit the DSNTYPE keyword the file will be allocated as
DSORG=PO and using the default SMS rules active .
The keyword DIR can be used to specify the number of directory blocks for a
new PDS (default = number of records divided by 20)
The member name for PDS,PDSE or data set name for the HFS directory is
based on the value of the new template variable &UNIQ. or &UQ. . It is a
unique 8 characters string derived from the system clock beginning with an
alpha followed by 7 alpha or numeric characters. .

8
 Unload Lobs using file ref variables

CREATE TABLE A. IMAGES TEMPLATE TSYSPUN DSN('WRK.PUNCH')

DISP(MOD,CATLG,CATLG)
(IMAGE_ID VARCHAR(30) NOT NULL, TEMPLATE TSYSREC DSN('WRK.SYSREC')
IMAGE_TYPE VARCHAR(10) , DISP(MOD,CATLG,CATLG)
VERSION VARCHAR(30) , UNLOAD DATA FROM TABLE A. IMAGES
……………………………………. UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN)

ROW_ID ROWID NOT NULL

GENERATED ALWAYS ,
IMAGE BLOB(100M) )
DSNU1218I - DSNUULIA - LOGICAL RECORD
LENGTH OF OUTPUT RECORD EXCEEDED THE
IN IMAGES.IMAGES LIMIT FOR TABLE A.IMAGES
DSNU012I DSNUGBAC - UTILITY EXECUTION
TERMINATED, HIGHEST RETURN CODE=8

Let’s consider a table which contains a CLOB column with a maximum length
of 100 Mb to store text documents . In this example we show how normal
unload of a table failed because the sum of all columns exceeds 32K .

9
 Unload Lobs using file ref variables

CREATE TABLE A. IMAGES TEMPLATE TSYSPUN DSN('WRK.PUNCH')

DISP(MOD,CATLG,CATLG)
(IMAGE_ID VARCHAR(30) NOT NULL, TEMPLATE TSYSREC DSN('WRK.SYSREC')
IMAGE_TYPE VARCHAR(10) , DISP(MOD,CATLG,CATLG)
VERSION VARCHAR(30) , TEMPLATE TSYSLOB DSN(‘WRK.SYSLOB’)
……………………………………. DSNTYPE(LIBRARY)
DISP(MOD,CATLG,CATLG)
ROW_ID ROWID NOT NULL UNLOAD DATA FROM TABLE A. IMAGES
GENERATED ALWAYS , IMAGE_ID, IMAGE_TYPE, ……
IMAGE BLOB(100M) ) IMAGE VARCHAR BLOBF TSYSLOB )
UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN)
IN IMAGES.IMAGES

Members : AYPPXJRD
AYPPXJ3H
AYPPXJ6K
10
…….

With File Reference variables you can instruct the UNLOAD utility to unload
the LOB data to separate data sets. In this example a PDSE was chosen using
the BLOBF file reference variable and a template TSYSLOB. As a result a
new PDSE with DSNAME=WRK.SYSLOB will be dynamically allocated and
for each lob value a new member with a unique name will be created for
storing the lob value.

10
 Unload Lobs using file ref variables

CREATE TABLE A. IMAGES TEMPLATE TSYSPUN DSN('WRK.PUNCH')

DISP(MOD,CATLG,CATLG)
(IMAGE_ID VARCHAR(30) NOT NULL, TEMPLATE TSYSREC DSN('WRK.SYSREC')
IMAGE_TYPE VARCHAR(10) , DISP(MOD,CATLG,CATLG)
VERSION VARCHAR(30) , TEMPLATE TSYSLOB DSN(/u/davy)
……………………………………. DSNTYPE(HFS)
UNLOAD DATA FROM TABLE A. IMAGES
ROW_ID ROWID NOT NULL IMAGE_ID, IMAGE_TYPE, ……
GENERATED ALWAYS , IMAGE VARCHAR BLOBF TSYSLOB )
IMAGE BLOB(100M) ) UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN)

IN IMAGES.IMAGES

Files /u/davy/ AYPPXJRD

/u/davy/ AYPPXJ3H
/u/davy/AYPPXJ6K
11
…….

In this example a HFS directory was chosen using the BLOBF file reference
variable and a template TSYSLOB. As a result for each lob value a new file
with a unique name will be created in this directory for storing the lob values.

11
 Unload Lobs using file ref variables
• Other :
• RUNSTATS the base table space before
• A null lob will be represented by a null output file name
(null indicator byte)
• Unloading lob data from image copies is not allowed
• HFS directory must exist and template must specify full
path name (i.e. not relative)
• Use uppercase when using template variables
• CCSID conversions are possible for CLOB
• Unloading to sequential files is not possible
• Use WHEN clause to UNLOAD to multiple files

12

Run RUNSTATS on the base table space before UNLOAD to allow DB2 to
estimate the size of the PDS directory when PDS .
A null lob is represented in the normal unload data set by a null-character
string (null indicator byte being set)
There are still some restrictions to take into account when unloading LOB
data.
The keywords EBCDIC,ASCII,UNICODE or CCSID can be used to convert
the encoding scheme of the output data (CLOB)

12
 Load Lobs using file ref variables
• Load lob data from separate file instead of the
normal sysrec input file (with max 32K record
length)
• Sequential file
• Member of a pds or pdse
• Hfs file
• File name present in the input file as char or
varchar field
• 1 separate file per lob value

13

LOAD is also enhanced to allow an input field value to contain the name of a
file containing a LOB column value . The LOB column will be loaded from
that file directly. The normal load input file will contain the normal data fields
and the name of the lob file associated with that record . It will contain
multiple file names when the table has multiple LOB columns.

13
 Load Lobs using file ref variables
• Implemented through :
• New keywords in the field specification for char
and varchar (file reference variables)
• BLOBF
• CLOBF
• DBCLOBF

• CCSID conversions apply for CLOBF and

DBCLOBF
• NULL input file name results in null lob value

14

The new file reference variables BLOBF,CLOBF and DBCLOBF specify that
the input field in the normal input file contains the name of a file (in CHAR or
VARCHAR format) where the actual LOB value is stored .

For BLOBF the BLOB will be loaded without CCSID conversion.

For CLOBF and DBCLOBF the required CCSID conversions will be done
during the load.

14
 Load Lobs using file ref variables
CREATE TABLE A. DOCUMENTS TEMPLATE TSYSREC DSN('WRK.SYSREC')
DISP(OLD,KEEP,KEEP)
(DOC_ID VARCHAR(30) NOT NULL, LOAD DATA INDDN TSYSREC LOG NO REPLACE
DOC_TYPE VARCHAR(10) , INTO TABLE A.DOCUMENTS
VERSION VARCHAR(30) , (DOC_ID POSITION( 1 : 31) VARCHAR ,
……………………………………. …………
DOCUMENT POSITION( 122:378) VARCHAR CLOBF
ROW_ID ROWID NOT NULL NULLIF(00121)=X'FF' )
GENERATED ALWAYS ,
DOCUMENT CLOB(100M) )

IN DOCUMENTS.DOCUMENTS

15

Let’s for example take the same table which contains a CLOB column with a
maximum length of 100 Mb to store text documents. The actual documents are
loaded from a file which name is contained in the normal input file as a
VARCHAR field

15
 Load Lobs using file ref variables

1 121
DOC001 ………WRK.SYSLOB(AYPPXJRD) ………
DOC002 ……….WRK.SYSLOB(AYPPXJ3H)………
DOC003 ……… WRK.SYSLOB(AYPPXJ6K)………

NULLIF(00121)=X'FF

16

The normal input file for the LOAD utility will contain a file name where the
actual lob data can be found. In this example it will be a member of a PDS.
If the indicator variable contains a -1 (null value) the lob will be null.

16
 Load Lobs using file ref variables

1 121
DOC001 ………/u/davy/AYPPXJRD ………
DOC002 ………./u/davy/AYPPXJ3H………
DOC003 ……… /u/davy/AYPPXJ6K………

NULLIF(00121)=X'FF

17

The normal input file for the LOAD utility will contain a file name where the
actual lob data can be found. In this example it will be a file on a HFS
directory.

17
 Xload Lob enhancements PQ90263
• Cross load : new V7 feature of the LOAD utility
• input is not a sequential file but the result set of
a dynamic SQL query
• source data in query are local or remote DB2
table(s)
• source can be on any drda server or the result
of Websphere Information Integrator
• Replaces unload, file transfer, load

18

Websphere Information Integrator was formerly called Datajoiner

18
 Xload Lob enhancements PQ90263
EXEC SQL
DECLARE c1 CURSOR FOR
select query
ENDEXEC
LOAD DATA
INCURSOR c1
INTO TABLE creator.table

19

Before PQ60923 in DB2 V7 and DB2 V8 it was only possible to cross load
data if the sum of the columns did not exceed 32K.

19
 Xload Lob enhancements PQ90263
• Architectural limit of 32767 bytes bypassed by
using a separate buffer for LOB columns above
16Mb line
• Only 8 bytes stored per lob column in 32767 byte
buffer
• Cross loading now limited by available memory
above 16Mb (region size)
• Message DSNU1178I explanation changed

20

Since PQ90263 in DB2 V7 and DB2 V8 the explanation text of message

DSNU1178I has been changed to reflect the fact that cross loading of Lob
columns will only be limited by the available memory above the 16Mb line. If
the user receives this message then increasing the region size will likely result
in successful execution of the LOAD utility.

20
(Un-)Load Lobs using file ref variables
• Generate LOAD statements using UNLOAD
• syspunch file
• Generate UNLOAD statements using rexx or
equivalent
• UNLOAD DATA FROM TABLE
• With BLOBF,CLOBF,DBCLOBF
• From sysibm.syscolumns
• type in (‘blob’,’clob’,’dbclob’)
• From sysibm.sysdatatypes
• Sourcetype if type = ‘distinct ‘
21

21
 (Un-)Load Lobs using file ref variables
• Unload lob file allocated as
• DSORG =PO
• RECFM=V,LRECL=27994,BLKSIZE=27998

• TEMPLATE variable &UNIQ. or UQ.

• Generate unique name of 8 chars based on
system clock starting with alphanumeric
• Can be used as data set qualifier

22

Be sure to have PK27029 installed on your system to avoid the lob file to be
allocated as a file with LRECL=0 .

By default DB2 will allocate the PDS(E) with BLKSIZE=27998 which is the
optimal blksize for 3390 devices. Preallocate the PDS(E) before if you are
using other device types to have optimal performance (ex: BLKSIZE=23040
for 3380)

22
(Un-)Load Lobs using file ref variables
• What if lob data sets in SYSREC do not exist ?
• LOAD abends :
• Abend S04E RC 00E40350 in dsnurwbx+5D3e
• reg2 = 00000001 if the pds does not exist
• reg2 = 00000002 if the pds member does not
exist
• RC 00E40350 : error has been returned from LOB
MANAGER insert.
• More user friendly messages in DB2 9 (new
SQLCODE=-452 )
23

23
 Using V9 DSNTIAUL to unload lobs
• V9 UNLOAD utility :
• unload lob fields to PDS(E) or HFS directory
• seq. files not supported (performance)

• V9 DSNTIAUL sample program :

• Enhanced to unload lob fields to seq files
• PARMS(‘SQL’) : unload lobs as normal data (with
truncation if record > 32K )
• PARMS(‘LOBFILE(prefix)’) : unload lobs to seq files

24

The DB2 sample program DSNTIAUL is enhanced , using file reference

variables itself, to allow LOB data to be unloaded to separate sequential files. :

RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB91) PARMS(‘SQL’) :

invokes DSNTIAUL with the old behavior :
- SYSIN contains one or more SQL SELECT statements
- all the data , including LOB data, is unloaded to the //SYSRECqq file.
- truncation occurs to 32K if the length of the result set exceeds 32 K.

RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB91)

PARMS(‘LOBFILE(prefix)’) :
- SYSIN must contain the name(s) of the base table
- each lob is unloaded to a separate seq. file <prefix>.Q<l>.C<j>.R<k>
- Q = querynumber C=columnnumber R=rownumber

-RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB91)

PARMS(‘SQL,LOBFILE(prefix)’) :
- SYSIN contains one or more SQL SELECT statements
- the lob data is unloaded to separate seq. files

24
 LOBs : Stronger and Faster in DB2 9

Online REORG of LOB Table spaces

25

In this section we will discuss the use of REORG SHRLEVEL REFERENCE

of LOB table spaces (new DB2 9) and the benefits compared to the old
REORG method .

25
 Online Reorg of LOB table spaces
• V6,V7,V8 REORG :
• Partial implementation of REORG
• Aim : improve prefetch performance of lob data
• Make lob pages as contiguous as possible
• Avoid fragmentation caused by inserts,updates
• Reduce the number of allocated chunks to the
minimum corresponding to orgratio = 100

26

Before DB2 9 the REORG utility run on a lob table space was completely
different from a REORG of a normal table space.

The main objective was to improve read I/O.

Because LOG NO lob table spaces should be consistent after rollback
operations, each time a lob is updated a new version of the lob is created
allocating new chunks of data. The space of old versions of lobs is only
reusable after the rollback or commit of the logical-unit-of-work. When
afterwards new versions of lobs are inserted, fragmentation occurs and on
average more chunks have to be read to retrieve the lob data.

26
 Online Reorg of LOB table spaces
rowid1
Auxiliary
rowid2 rowid points to first lob map page of lob
index
rowid3

rowid4

16 pages 16 pages 16 pages 16 pages 16 pages

6 2 8 5 7 4 ● ● ● ●

chunk1 chunk2 chunk6 chunk7 chunk13

First lob map page of lob with rowid1 points to chunk1 (partially allocated from page 1 with 6 pages)
and chunk2 (partially allocated from page 6 with 7 pages)

27

DB2 allocates space for LOB data in chunks. A chunk is 16 contiguous pages
(16 immediately adjacent pages). A single LOB value can be stored in multiple
chunks which will be fully or partially allocated to that LOB.

When an application wants to access a certain lob, the rowid stored in the base table
will be used to locate the first lob map page via the auxiliary index. This lob map page
contains a table with all the chunks that are fully or partially allocated to this lob .

A lob table space is considered to be perfectly organized when the number of chunks
allocated to a lob is minimal (no extra chunks needed) . This is expressed by the
orgratio factor which is 100 in the perfect case. In the above example the lob pointed
by rowid1 is not perfectly organized because it is allocated to 2 chunks (6 pages in
chunk 1 and 7 pages in chunk2) where one chunk (13 out of 16 pages) would be
sufficient.

ORGRATIO is the percentage of organization in the LOB table space. A value of

100 indicates perfect organization of the LOB table space. A value of 1 indicates that
the LOB table space is disorganized. A value of 0.00 indicates that the LOB table
space is totally disorganized.

27
 Online Reorg of LOB table spaces
• V6 REORG :
• Only improved prefetch performance of lob data
• Rebuild lob table space in place (no unload-
reload)
• No reclaim of free space
• No delete/define cluster : RECOVER utility or
UNLOAD/RELOAD was needed to change priqty
& secqty allocated values and number of extents

28

The V6 REORG TABLESPACE utility could not be used to change the

allocated space because the underlying VSAM data sets were not deleted and
redefined during reorg. The only way to do this before DB2 9 was using the
RECOVER utility.

UNLOAD, drop table space, recreate table space, RELOAD was also possible
but not recommended (because of invalidation of packages etc ..)

28
 Online Reorg of LOB table spaces
• V6 REORG :
• Other restrictions :
• No concurrent access to LOB data
• Always LOG YES
• No inline image copy possible
• Utility not restartable (RECP status)
• Before image copy recommended

• DB2 9 : REORG SHRLEVEL NONE (default)

29

Considering some other restrictions as well, REORG of a LOB table space

was not often done by customers.

IN DB2 9 the old REORG method has been retained for compatibility reasons
and is called REORG SHRLEVEL NONE . Although it is the default ,it is no
longer recommended by IBM.

29
 Online Reorg of LOB table spaces
• DB2 9 REORG SHRLEVEL REFERENCE:
• Uses shadow datasets
• PRIQTY,SECQTY changes are honored
• Read access allowed (except switch phase)
• Free space is reclaimed
• Always LOG NO
• Inline image copy mandatory
• Primary datasets available after failure

30

The new online REORG of LOB table is very similar to REORG SHRLEVEL
REFERENCE of a normal table space. Most disadvantages are gone
One drawback is the temporary need of disk space for allocating the shadow
datasets. The REORGLOB phase is not restartable but leaves the original
datasets available after a failure

30
Online Reorg of LOB table spaces
• Conclusions :
• Use REORG SHRLEVEL REFERENCE in DB2 9
• Better availability (no R/W access )
• Allows free space reclaim
• Easier to handle in case of failure
• Plan for enough DASD

31

31
 LOBs : Stronger and Faster in DB2 9

COPY/QUIESCE
RECOVER/REBUILD

32

In this section we will discuss the use of the COPY and QUIESCE utilities to
create backups with common points of consistency and how to RECOVER
your LOB data.

32
 Making consistent backups using COPY
• Be prepared for recovery of LOB data
• Use TEMPLATES and LISTDEF to backup all at
once and to create a common recoverable point
of consistency using SHRLEVEL REFERENCE

LISTDEF MYLIST INCLUDE TABLESPACES TABLE creator.table ALL

INCLUDE INDEXSPACES COPY YES TABLE creator.table ALL
COPY LIST MYLIST FULL YES SHRLEVEL REFERENCE PARALLEL COPYDSN(TSYSCOPY)

33

A common recoverable point of consistency is a point where both base table

data and LOB data are consistent and to which a point in time recovery can be
done.
Because a non-partitioned or partitioned base table can contain many
auxiliary objects (one lob table space and one auxiliary index per lob column
and per partition) it is recommended to use a LISTDEF to generate a list of all
these objects.
Use the lob indicator keywords BASE , LOB or ALL if you want to include
the base objects only, the auxiliary objects only or both in your list. Use the
INDEXSPACES COPY YES keyword if you do not want to include the base
table indexes for which the COPY YES attribute is disabled. Use the
PARALLEL keyword to take the copies in parallel.

If you specify a LISTDEF with SHRLEVEL REFERENCE , all copy data sets
will have the same START_RBA value in the DB2 catalog table
SYSIBM.SYSCOPY resulting in a good consistency point.

33
 Creating common quiesce points
• Be prepared for recovery of LOB data
• Use LISTDEF to create a common quiesce point.

LISTDEF MYLIST INCLUDE TABLESPACES TABLE creator.table ALL

QUIESCE LIST MYLIST

• A common quiesce point is a common recoverable point

of consistency when both the base table space and LOB
table space are defined as LOG YES or LOGGED .

34

A common quiesce point is a point where both base table data and LOB data
are consistent. The quiesce utility drains all writers on the table space objects
and by default (WRITE YES) writes all changed pages from the table spaces
and index spaces in the buffer pool to disk..
Because a non-partitioned or partitioned base table can contain many
auxiliary objects (one lob table space and one auxiliary index per lob column
and per partition) it is recommended to use a LISTDEF to generate a list of all
these objects.
For a common quiesce point , all table spaces (base and LOB) will have the
same START_RBA value in the DB2 catalog table SYSIBM.SYSCOPY.
A common quiesce point is a common recoverable point of consistency when
both the base table space and LOB table spaces were created as LOG YES (V7
and V8) or LOGGED (DB2 9). Be aware that when you recover to a common
quiesce point, you are not able to recover the LOBs if the LOB table space is
defined as LOG NO (V7 and V8) or NOT LOGGED(DB2 9) because of the
missing log records. Also, if the base table space is defined as NOT LOGGED
, the quiesce point is not a recoverable point of consistency.

34
 Recovery of LOB data
• Recovery of LOB data is similar to recovery of RI
related data because there is a relationship
between the base table space and LOB table
space(s).

• Both base table space and LOB table space(s)

must always be recovered to the same point of
consistency.

35

The RECOVER utility recovers data to the current state or to a previous point
in time by restoring a copy and then applying log records if available.
In some ways, planning for LOB recovery is similar to that for user-defined
referential integrity, because you have to remember that there is a relationship
between a table with a LOB column and the associated LOB table space (s). It
is true that tables involved in referential integrity relationships must be
considered as part of the same table space set for recovery purposes. Similarly,
a base table space and its associated LOB table spaces are also part of a table
space set. That set, too, is reported using REPORT table space.
When recovering, all objects of the table space set should be recovered to the
same point of consistency to finish with consistent data.

35
 Recovery to a prior point in time
• Use a LISTDEF with TOLOGPOINT or TORBA to recover to
• A common set of SHRLEVEL REFERENCE image copies
with the same START_RBA
• A common QUIESCE point

• If the point in time is not a common recoverable point of

consistency :
• Base table space can become ACHKP
• Lob table space can become AUXW
• LOBS can be marked invalid in LOB table space

36

A common point of consistency is a not a common recoverable point of

consistency when the RECOVER utility is unable to recover all objects
because of missing log records.
If you do not recover the base table space and LOB table spaces together to a
common point of consistency, the base table space is marked as auxiliary
check pending (ACHKP) and unavailable for SQL.
Furthermore, if the LOB table space is defined as LOG NO (V7 and V8) or
NOT LOGGED (DB2 9), all LOBs with missing log records between the last
image copy and the recover-to-point are marked invalid in the LOB table
space (but not in the base table space) and the LOB table space is put in the
auxiliary warning state (AUXW). Indexes are put in rebuild pending (RBDP)
or check pending state (CHKP) when the indexes are not recovered together
with their table spaces to a point in time.
You should use the CHECK LOB utility on the LOB table space to find the
invalid LOBs in the LOB table space. You can correct them using the SQL
update of the LOB values or SQL delete of the entire rows.

36
 Recovery to the current point in time
• If LOB table space is defined as LOG NO :
• Lobs are marked invalid in LOB table space
• LOB table space is set AUXW

• Recovery LOB pages on the LPL :

• Start database(..) spacenam(..)
• Lob table space can become CHKP if log
records are missing

37

Recovering to the current point in time for a base table containing a LOB
column is no different than it was in the past. DB2 applies the appropriate
image copy to the base table space and reads the log since copy time until
present, redoing all the changes. This is also true for the LOB table space.

However, if the LOB table space is defined with LOG NO and log records
must be applied to the LOB, the LOB is marked invalid and DB2 sets the
auxiliary warning state (AUXW) for that LOB table space (The invalid flag is
set in the LOB table space but not in the base table space). You should use the
CHECK LOB utility on the LOB table space to find the invalid LOBs in the
LOB table space.

If the base table space is defined as NOT LOGGED (DB2 9) and you try to
recover to the current point in time, the RECOVER utility only recovers to the
last recoverable point with the RBA or LRSN recorded in SYSIBM.
SYSCOPY (for example, the START_RBA of the last full SHRLEVEL
REFERENCE image copy). And, the base table space is put in the auxiliary
check pending status (ACHKP).

37
Recovery of the indexes
• Do not forget to recover the indexes as well :
• Using RECOVER INDEX if image copies avail.
• Using REBUILD INDEX

• Use LISTDEFS to include all objects :

LISTDEF MYTSLIST INCLUDE TABLESPACES TABLE creator.table ALL
LISTDEF MYIXLIST INCLUDE INDEXSPACES TABLE creator.table ALL
RECOVER LIST MYTSLIST PARALLEL
REBUILD INDEX LIST MYIXLIST SORTDEVT 3390 SORTNUM 6

38

38
 LOBS and system level recovery
• BACKUP SYSTEM utility (DB2 V8)
• RESTORE SYSTEM utility
• New DB2 9 : RECOVERY can use backups of
system-level BACKUP for object-level recovery

• Not logged LOB table spaces are marked RECP

after RESTORE SYSTEM when updated between
the system level backup and SYSPITRT

39

System level backup/restore was introduced in DB2 V8 (requires z/OS V1R5)

It allows a complete DB2 subsystem or data sharing group to be backuped and
restored to a previous point-in-time. In DB2 9 it is also possible to use the
system level backups to recover a subset of table spaces and index spaces.

When the RESTORE SYSTEM utility discovers that NOT LOGGED objects
(base table spaces or LOB table spaces) have been updated between the system
level backup and the point in time log truncation point (SYSPITR) specified in
the CRESTART conditional restart , the objects are put in recover pending
state and must be individually recovered using RECOVER/REBUILD.

There are no other special considerations when your DB2 data contains LOB
table spaces in regard to the BACKUP system and RESTORE system utilities.

39
 LOBs : Stronger and Faster in DB2 9

Online CHECK utilities in DB2 9

40

In this section we will discuss the new online CHECK LOB , CHECK DATA
and CHECK INDEX utilities in DB2 9.

40
 CHECK LOB utility
• CHECK LOB can be used for :
• Report invalid LOBs in lob table space
• Report defective LOBs
• Remove CHKP or AUXW states when no errors
• CHECK LOB does not access base table or
auxiliary index
• CHECK LOB must be run before CHECK DATA

41

The CHECK LOB online utility can be run against a LOB table space to
identify any structural defects in the LOB table space and signal any invalid
LOB values. Run CHECK LOB when you suspect there might be structural
defects or when the LOB table space is in the check pending state (CHKP) or
auxiliary warning state (AUXW).
The CHECK LOB utility can only be run on a LOB table space, not on a base
table space. The CHECK LOB utility only accesses the LOB table space. It
does not access the base table space or the auxiliary index. Therefore it will
not detect missing LOBs.
CHECK LOB reports the following errors:
- Invalid LOBs : An invalid LOB is the status of a LOB as set by RECOVER
when an uncorrected LOB column error is found. CHECK LOB examines the
invalid flag in the LOB table space, but it never sets it.
-Defective LOBs : A defective LOB is a logically inconsistent LOB with a
structural defect (most probably as the result of an operational problem that
left the LOB table space in an inconsistent state or as the result of a software
defect).
Always run CHECK LOB before CHECK DATA because CHECK DATA
relies on the information in the LOB table space being correct.

41
 CHECK LOB shrlevel reference
• Drains readers/writers and puts lob table space in
CHKP which makes it unavailable for sql
applications (also readers !)
• If defective lobs : CHKP remains
• Remains unavailable for SQL applications
• If only invalid lobs : AUXW
• Invalid lobs can be updated/deleted by SQL
• New DRAIN options available in DB2 9

42

The following actions are performed when using CHECK LOB SHRLEVEL
REFERENCE:
- DB2 drains all SQL readers and writers and set the LOB table space in check
pending (CHKP) status, which makes it unavailable for all applications. The
LOB table space is put in UTRO status, and the applications get SQLCODE-
904 resource unavailable with reason code 00C900A3.
-If CHECK LOB encounters no more invalid LOBs and no other errors, the
LOB table space is reset in no pending state. If invalid LOB columns remain,
CHECK LOB sets the LOB table space to the auxiliary warning (AUXW)
status and the table is available for applications. You can use SQL to populate
an invalid LOB by updating or deleting the entire row; however, any other
attempt to access the column results in a -904 SQL return code (reason code
00C900D0). But the base table and all other non-invalid LOBs are available
for applications.
- If CHECK LOB encountered defective LOBs, the LOB table space is left in
CHKP status and is unavailable for applications.
Since DB2 9 you can now specify drain options such as DRAIN_WAIT,
RETRY, and RETRY_DELAY to have the utility retry the drain operation if it
fails.
CHECK LOB SHRLEVEL REFERENCE is very disruptive !

42
 CHECK LOB shrlevel change
• Online CHECK LOB new DB2 9
• Runs on shadow datasets created by flashcopy
snapshot
• SQL writers only drained during flashcopy
• Primary LOBs remain available for SQL
• No CHKP or AUXW status afterwards
• REPAIR delete statements generated if
defective or invalid lobs
• Invalid lobs can also be updated/deleted by
SQL
43

The CHECK LOB SHRLEVEL CHANGE utility leaves the LOB table space
available for applications during its processing by running on a shadow copy
of the LOB table space taken by a flashcopy snapshot. These shadow copies
are deleted at the end of the utility (UTILTERM).
Once CHECK LOB has created the shadow data set, it drains all writers until
the snapshot is complete. The LOB table space is in UTRO during that time.
You can specify drain options to have the utility retry the drain operation if it
fails. After the snapshot is complete, the LOB table space is put in UTRW.
CHECK LOB processes the snapshot data set, and during its execution, the
LOB table space remains available for applications if it was not yet in the
CHKP status. If you provide a SYSPUNCH data set, the CHECK LOB utility
generates REPAIR DELETE statements to delete the bad LOBs afterwards.
You can also use SQL to update or delete these LOB columns.
CHECK LOB SHRLEVEL CHANGE never sets or resets CHKP or AUXW
states on the LOB table space.
If your DASD hardware does not support the FlashCopy Version 2 capabilities
needed for SHRLEVEL CHANGE, DFSMSDSS uses the standard REPRO
utility to copy the shadow data set and the utility does not fail. However, the
UTRO phase, in which SQL writers are not allowed to update the data, is
significantly longer.

43
 CHECK DATA utility
• CHECK DATA can be used for :
• Report orphan, missing, out-of-sync or invalid
LOBs in base table space
• Remove ACHKP or AUXW states when no errors
• CHECK DATA does not access the lob table space,
only the auxiliary index
• CHECK DATA must be run after CHECK LOB and
after CHECK INDEX of the auxiliary index

44

The CHECK DATA utility checks for consistency between a base table space
and the corresponding LOB table spaces. Run CHECK DATA after a
conditional restart or a point in time recovery where base tables and auxiliary
tables might not be synchronized or when the base table space is in the
auxiliary check pending state (ACHKP) or auxiliary warning state (AUXW).
The CHECK DATA utility can only be run on a base table space, not on a
LOB table space The CHECK DATA utility accesses the base table space and
all related auxiliary indexes of the LOB table spaces. CHECK DATA relies on
information in the LOB table space and the auxiliary indexes being correct. It
might be necessary to first run CHECK LOB on the LOB table space and
CHECK INDEX and REBUILD INDEX on the auxiliary indexes to ensure the
validity of the LOB table spaces and auxiliary indexes.
CHECK DATA reports the following errors:
- Orphan LOBs : LOBs found in the auxiliary index but not referenced by the
base table space.
- Missing LOBs : LOBs referenced by the base table space but not in the
auxiliary index.
- Out-of-sync LOBs : LOBs with a different version in the base table space and
the auxiliary index.
-Invalid LOBs in the base table space as a result of a previous execution of
CHECK DATA AUXERROR INVALIDATE.

44
 CHECK DATA shrlevel reference
• AUXERROR REPORT
• Drains writers and puts base table space in
ACHKP when errors which makes it unavailable
for sql
• AUXERROR INVALIDATE
• Drains readers/writers , invalidates LOBs in
error and puts table space in AUXW afterwards
which makes it available again for sql
• New DRAIN options available in DB2 9

45

CHECK DATA with AUXERROR REPORT : If the base table space is not
yet in auxiliary check pending (ACHKP) status, DB2 drains all SQL writers
and sets the base table space and auxiliary table space in UTRO. The data is
still available for SQL readers. If errors are found, then the base table space is
set to the auxiliary check pending (ACHKP) status. The whole table, or
partition, if the table space is partitioned, is now unavailable for applications.
The applications get SQLCODE – 904 resource unavailable reason code
00C900C5. If CHECK DATA encounters only invalid LOB columns, the base
table space is set to the auxiliary warning status (AUXW), and the table is
available for applications.
CHECK DATA with AUXERROR INVALIDATE : DB2 drains all SQL
readers and writers, sets the base table space and auxiliary table space in
UTUT, and the data is no longer available for SQL. DB2 invalidates the LOB
columns that are in error (it sets the invalid flag in the base table space for the
LOB value). DB2 resets the invalid status of LOB columns that have been
corrected. If CHECK DATA encounters no more invalid LOBs, then the table
space is reset in no pending state. If invalid LOB columns remain, CHECK
DATA sets the base table space to the auxiliary warning (AUXW) status and
the table is available for applications. You can use SQL to populate invalid
LOBs using update or delete the entire row..

45
 CHECK DATA shrlevel change
• Online CHECK DATA new DB2 9
• Runs on shadow datasets created by flashcopy
snapshot
• SQL writers only drained during flashcopy
• Primary data remains available for SQL
• No ACHKP or AUXW status afterwards
• AUXERROR INVALIDATE :
• LOBs not invalidated in the base table space
• REPAIR statements generated to invalidate
lobs afterwards
46

The CHECK DATA SHRLEVEL CHANGE utility leaves the data available
for applications during its processing by running on a shadow copy of the base
table space taken by a flashcopy snapshot. These shadow copies are deleted at
the end of the utility (UTILTERM).
Once CHECK DATA has created the shadow data set, it drains all writers until
the snapshot is complete. The base table space is in UTRO during that time.
You can specify drain options to have the utility retry the drain operation if it
fails. After the snapshot is complete, the base table space is put in UTRW.
CHECK DATA AUXERROR INVALIDATE does not invalidate LOBs in the
base table space . If you provide a SYSPUNCH data set, the CHECK DATA
utility generates REPAIR statements to invalidate the bad LOBs afterwards.
CHECK DATA SHRLEVEL CHANGE never sets or resets ACHKP or
AUXW states on the base table space.
If your DASD hardware does not support the FlashCopy Version 2 capabilities
needed for SHRLEVEL CHANGE, DFSMSDSS uses the standard REPRO
utility to copy the shadow data set and the utility does not fail. However, the
UTRO phase, in which SQL writers are not allowed to update the data, is
significantly longer.

46
 CHECK INDEX shrlevel change
• Introduced in V8 with PQ96956
• Can also be used to check auxiliary indexes

• Runs on shadow datasets created by flashcopy

snapshot
• SQL writers only drained during flashcopy
• Primary index remains available for SQL writers

47

The CHECK INDEX SHRLEVEL CHANGE utility leaves the LOB table
space and auxiliary index available for SQL writers during its processing by
running on a shadow copy of the LOB table space and auxiliary index taken by
a flashcopy snapshot. These shadow copies are deleted at the end of the utility
(UTILTERM).
Once CHECK INDEX has created the shadow data sets, it drains all writers
until the snapshot is complete. The LOB table space and auxiliary index are in
UTRO during that time. You can specify drain options such as
DRAIN_WAIT, RETRY, and RETRY_DELAY to have the utility retry the
drain operation if it fails. After the snapshot is complete, the LOB table space
is put in UTRW.
If your DASD hardware does not support the FlashCopy Version 2 capabilities
needed for SHRLEVEL CHANGE, DFSMSDSS uses the standard REPRO
utility to copy the shadow data set and the utility does not fail. However, the
UTRO phase, in which SQL writers are not allowed to update the data, is
significantly longer

47
 LOBs : Stronger and Faster in DB2 9

Altering tables with Lob data

48

In this last section we will discuss the benefits of the new UNLOAD /
RELOAD support for LOBs when changing tables that contain LOB columns.
.

48
Altering tables with LOB data
• TABLE changes that require UNLOAD/RELOAD
• Column changes that required drop/create table
were painful if the table also contained lob columns :
• Must also recreate all auxiliary tables and
indexes
• How to unload/load the lob data if lob columns >
32 K ?
• Same for ddl changes to the base table space that
required a recreate

49

49
 Altering tables with LOB data
• TABLE changes that require UNLOAD/RELOAD
• Before : use shadow table :
• Create identical shadow table (like a clone)
• Copy lob data using insert
• Repopulate after drop/create with insert from
shadow table
• Now : use unload/reload with full lob support
• Tools should start to support this

50

The easiest way to unload/load lob data if no proper tool available was to keep
the lob data in DB2 format by copying it to an identical shadow table.

50
 LOBs : Stronger and Faster in DB2 9

Conclusions

51

51
 Conclusions
• Loading and unloading LOB data using file
reference variables
• Enhancements to DSNTIAUL
• Online REORG of LOB table spaces
• Making consistent backups using COPY
• Recovery of LOB data using RECOVER/REBUILD
• Online CHECK utilities in DB2 9
• Altering tables with LOB data

52

We have explained the new utility features coming along with DB2 9, finally
ensuring full support for LOBs.

Please note that DB2 data management and application enhancements for
LOBs in DB2 9 are covered in "LOBs : Stronger and Faster in DB2 9 for z/OS
- Part 1 of 2"

52
Reference material
• DB2 V7, V8 and V9 manuals for z/OS
• APAR text PQ90263 and PK22910

• IBM redbooks
• SG24-6571-00 Large Objects with DB2 for z/OS
and OS/390 (V7)
• SG24-7270-00 LOBs with DB2 for z/OS:
Stronger and Faster (V9)

53

53
54

54
Session C06
LOBs : Stronger and Faster in DB2 9 for z/OS
(Part 2 of 2)

Davy Goethals
ArcelorMittal
[email protected]

55

55