Front cover

ICF Catalog Backup

and Recovery
A Practical Guide
Learn how to be prepared when catalogs
break -- and they do break!
Know the tools for detecting
catalog errors
Look at case studies of
common error situations

Yolanda Cascajo Jimenez

Ronald K. Ferguson
Eileen A. McClintock
Robert A. Matthews

ibm.com/redbooks

International Technical Support Organization

ICF Catalog Backup and Recovery: A Practical Guide
December 2001

SG24-5644-01

Take Note! Before using this information and the product it supports, be sure to read the
general information in Special notices on page 153.

Second Edition (December 2001)

This edition applies to Version 1, Releases 1 and 2 of z/OS, Program Number 5694-A01.
This document was created or updated on July 23, 2002.
Comments may be addressed to:
IBM Corporation, International Technical Support Organization
Dept. 471F Building 80-E2
650 Harry Road
San Jose, California 95120-6099
When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the
information in any way it believes appropriate without incurring any obligation to you.
Copyright International Business Machines Corporation 1999, 2001. All rights reserved.
Note to U.S Government Users Documentation related to restricted rights Use, duplication or disclosure is subject to
restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
The team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
IBM trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Comments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 ICF catalog structure and evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Catalog structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Where a data set is described. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.3 Master and user catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.4 Catalog Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Tape volume catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 The ITSO test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Chapter 2. Backing up catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Backup strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.1 Backup frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.2 Backup techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Backup utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.1 IDCAMS EXPORT command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 DFSMSdss logical dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.3 DFSMShsm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 Advanced Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Considerations for the backup process . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 SMF data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Master catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.3 Tape volume catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.4 VSAM volume data set (VVDS) backup . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 3. Analyzing catalog integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1 General diagnostic considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 Utilities and tools available for diagnosis. . . . . . . . . . . . . . . . . . . . . . . . . . 23

Copyright IBM Corp. 1999, 2001

3.2.1 VSAM Knowledge Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3 Structural errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3.1 Checking the structural integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.2 The consequences of neglecting structural integrity . . . . . . . . . . . . . 26
3.4 Content errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4.1 DIAGNOSE of a BCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.4.2 DIAGNOSE of a VVDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5 Synchronization errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.5.1 Detecting synchronization errors with DIAGNOSE . . . . . . . . . . . . . . 30
3.5.2 Repairing synchronization errors . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.6 Catalog sharing problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.6.1 A review of the SHAREOPTIONS parameter . . . . . . . . . . . . . . . . . . 37
3.6.2 Determining if a catalog is actually shared . . . . . . . . . . . . . . . . . . . . 38
3.6.3 Error symptoms due to sharing problems . . . . . . . . . . . . . . . . . . . . . 40
3.7 CAS problems due to catalog errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.7.1 CAS ABEND or error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.7.2 CAS hang or lockout situations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.7.3 Working with the catalog address space. . . . . . . . . . . . . . . . . . . . . . 44
Chapter 4. Recovering catalogs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.1 Recovery strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.1.1 Determining when to recover a catalog. . . . . . . . . . . . . . . . . . . . . . . 50
4.1.2 Recover, restore, repair, or salvage? . . . . . . . . . . . . . . . . . . . . . . . . 50
4.1.3 Determining the most recent backup version . . . . . . . . . . . . . . . . . . 51
4.1.4 Recovery techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1.5 Preventing catalog activity during restore . . . . . . . . . . . . . . . . . . . . . 52
4.2 Recovery utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2.1 IDCAMS IMPORT command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.2 DFSMSdss logical restore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.3 DFSMShsm recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2.4 Using REPRO MERGECAT to salvage a catalog . . . . . . . . . . . . . . . 54
4.3 Considerations for the recovery process . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.1 Master catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.2 User catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3.3 Tape volume catalogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3.4 VVDS recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3.5 IMBED and REPLICATE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.4 Forward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.4.1 Manually checking for missing updates . . . . . . . . . . . . . . . . . . . . . . 58
4.4.2 SMF data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4.3 Using DFSORT in the checking process. . . . . . . . . . . . . . . . . . . . . . 58
Chapter 5. Using ICFRU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

ICF Catalog Backup and Recovery: A Practical Guide

5.1 ICFRU basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5.2 Recovering an ICF catalog using ICFRU . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.3 Post recovery actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Chapter 6. Using Catalog RecoveryPlus . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.1 Introduction to Catalog RecoveryPlus facilities . . . . . . . . . . . . . . . . . . . . . 70
6.2 Catalog RecoveryPlus ISPF interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.3 RACF FACILITY class profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.4 BACKUP command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.4.1 Specifying BCSs to be backed up . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.4.2 Specifying VVDSs to be backed up . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.4.3 Diagnosing BCS or VVDS integrity prior to backup . . . . . . . . . . . . . 80
6.4.4 Restricting access to the BCS or volume during backup . . . . . . . . . 81
6.4.5 Creating multiple backup copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.4.6 Alias processing with BACKUP BCS . . . . . . . . . . . . . . . . . . . . . . . . 85
6.4.7 BACKUP BCS bypasses index processing . . . . . . . . . . . . . . . . . . . . 85
6.5 RECOVER command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.5.1 Restore versus forward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.5.2 Specifying the BCS or VVDS to restore . . . . . . . . . . . . . . . . . . . . . . 87
6.5.3 Explicit/implicit DEFINE of the BCS or VVDS to restore . . . . . . . . . . 87
6.5.4 Locking versus unlocking the BCS . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.5.5 Alias handling during BCS restore/recovery . . . . . . . . . . . . . . . . . . . 88
6.5.6 Renaming the BCS during restore . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.5.7 SMF data for forward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
6.5.8 PRINTing BCS records during a RECOVER . . . . . . . . . . . . . . . . . . 93
6.5.9 Simulation as a BCS or VVDS recovery testing tool . . . . . . . . . . . . . 94
6.5.10 Re-sizing the VVDS during RECOVER processing . . . . . . . . . . . . 95
6.6 DIAGNOSE command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.6.1 DIAGNOSE ALIAS command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.6.2 DIAGNOSE BCS-VVDS and VVDS-BCS command. . . . . . . . . . . . . 99
6.7 MERGECAT command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.7.1 Alias handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.7.2 Using the journal file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.7.3 RESTART and BACKOUT capability . . . . . . . . . . . . . . . . . . . . . . . 104
6.7.4 SIMULATE capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.7.5 Catalog activity during MERGECAT processing . . . . . . . . . . . . . . . 105
6.7.6 Creating an alternate master catalog . . . . . . . . . . . . . . . . . . . . . . . 106
6.8 SUPERCLIP command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.8.1 Changing a volser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.8.2 Caution when using SUPERCLIP . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.9 ZAP command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.9.1 Printing BCS and VVDS records . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.9.2 Deleting a BCS and VVDS record. . . . . . . . . . . . . . . . . . . . . . . . . . 108

Contents

6.9.3 Patching a BCS and VVDS records . . . . . . . . . . . . . . . . . . . . . . . . 109

Chapter 7. Case studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.1 Case study #1: Ensuring good backups . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.2 Case study #2: BCS backup and forward recovery. . . . . . . . . . . . . . . . . 116
7.3 Case study #3: Synchronizing aliases. . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.4 Case study #4: VVDS backup and forward recovery . . . . . . . . . . . . . . . 125
7.5 Case study #5: Enlarging VVDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.6 Case study #6: Changing a disk volser . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.7 Case study #7: Alternate master catalog . . . . . . . . . . . . . . . . . . . . . . . . 135
7.8 Case study #8: Print and delete BCS/VVDS records . . . . . . . . . . . . . . . 138
7.9 Case study #9: Preventive maintenance . . . . . . . . . . . . . . . . . . . . . . . . . 143
Related publications . . . . . . . . . . . . . . . . . . . . . .
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Other resources . . . . . . . . . . . . . . . . . . . . . . . .
Referenced Web sites . . . . . . . . . . . . . . . . . . . . . .
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . .
IBM Redbooks collections . . . . . . . . . . . . . . . . .

......
......
......
......
......
......

.......
.......
.......
.......
.......
.......

......
......
......
......
......
......

.
.
.
.
.
.

151
151
151
152
152
152

Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

ICF Catalog Backup and Recovery: A Practical Guide

Figures
1-1
1-2
1-3
2-1
2-2
2-3
3-1
3-2
3-3
3-4
3-5
3-6
3-7
3-8
3-9
3-10
3-11
3-12
3-13
3-14
3-15
3-16
3-17
3-18
3-19
3-20
4-1
4-2
4-3
4-4
4-5
5-1
5-2
5-3
5-4
5-5
6-1
6-2

Logical and physical description of data . . . . . . . . . . . . . . . . . . . . . . . . . 3

CAS usage for data set reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Structure diagram of the ITSO parallel sysplex . . . . . . . . . . . . . . . . . . . . 9
Sample IDCAMS EXPORT output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Sample DFSMSdss logical dump messages . . . . . . . . . . . . . . . . . . . . . 15
Sample DFSMSdss logical dump of a VVDS. . . . . . . . . . . . . . . . . . . . . 19
Sample JCL to VERIFY and EXAMINE a BCS . . . . . . . . . . . . . . . . . . . 25
DIAGNOSE BCS output showing content errors . . . . . . . . . . . . . . . . . . 28
DIAGNOSE VVDS output showing content errors . . . . . . . . . . . . . . . . . 29
DIAGNOSE processing: BCS compared to VVDS . . . . . . . . . . . . . . . . 31
Sample JCL to DIAGNOSE a BCS and compare to two VVDSs . . . . . . 33
DIAGNOSE processing: VVDS compared to BCS . . . . . . . . . . . . . . . . 34
Printing the VVCR showing BCS names . . . . . . . . . . . . . . . . . . . . . . . . 35
Sample JCL to DIAGNOSE a VVDS and compare to a BCS . . . . . . . . 36
Catalog status display for allocated catalogs . . . . . . . . . . . . . . . . . . . . . 39
Checking catalog attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Results of improper sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Catalog task display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Step 1: TSO user IDs waiting for BCS ENQ . . . . . . . . . . . . . . . . . . . . . 44
Step 2: TSO user ID cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Step 3: End one catalog request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Step 4: TSO user IDs cancelled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Step 5: Catalog request abended . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Step 6: Display GRS information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Step 7: Restart CAS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Step 8: No contention on CAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Sample JCL to lock a catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Sample JCL for DFSMSdss RESTORE of a BCS . . . . . . . . . . . . . . . . . 53
DFSORT statements to extract SMF record type 61 . . . . . . . . . . . . . . . 59
DFSORT statements to extract SMF record types 61, 65, and 66 . . . . 59
DFSORT statements to extract SMF record type 60 . . . . . . . . . . . . . . . 60
Integrated Catalog Forward Recovery Utility logic flow . . . . . . . . . . . . . 63
Sample JCL for running ICFRU (Part 1 of 2) . . . . . . . . . . . . . . . . . . . . . 64
Sample JCL for running ICFRU (Part 2 of 2) . . . . . . . . . . . . . . . . . . . . . 65
ICFRU ICFRRSV output report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
ICFRU ICFRRAP output report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
CR+ ISPF Main Menu panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
CR+ BACKUP BCS/VVDS command panel . . . . . . . . . . . . . . . . . . . . . 72

Copyright IBM Corp. 1999, 2001

6-3
6-4
6-5
6-6
6-7
6-8
6-9
6-10
6-11
6-12
6-13
6-14
6-15
6-16
6-17
6-18
6-19
6-20
6-21
6-22
6-23
6-24
6-25
6-26
6-27
6-28
6-29
6-30
6-31
6-32
6-33
6-34
6-35
6-36
6-37
6-38

CR+ RECOVER BCS command panel . . . . . . . . . . . . . . . . . . . . . . . . . 73

CR+ RECOVER VVDS command panel . . . . . . . . . . . . . . . . . . . . . . . . 74
CR+ FACILITY class profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Back up a single BCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Back up multiple BCSs, explicitly-specified . . . . . . . . . . . . . . . . . . . . . . 76
Back up multiple BCSs, with a mask filter . . . . . . . . . . . . . . . . . . . . . . . 77
Back up multiple BCSs, excluding one . . . . . . . . . . . . . . . . . . . . . . . . . 77
Back up all BCSs in the environment. . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Back up a single VVDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Back up multiple VVDSs, explicitly-specified . . . . . . . . . . . . . . . . . . . . . 79
Back up multiple VVDSs, with a mask filter . . . . . . . . . . . . . . . . . . . . . . 79
Back up all VVDSs in the environment . . . . . . . . . . . . . . . . . . . . . . . . . 79
Back up multiple VVDSs, excluding one . . . . . . . . . . . . . . . . . . . . . . . . 80
Sample duplex BACKUP command. . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Catalog RecoveryPlus duplex backup, duplex cataloging . . . . . . . . . . . 83
RECOVER command for a BCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
RECOVER command for a VVDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
RECOVER BCS with NEWNAME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
RECOVER BCS FORWARD(SMFFILE) command . . . . . . . . . . . . . . . . 91
RECOVER VVDS FORWARD(SMFFILE) command. . . . . . . . . . . . . . . 91
Catalog RecoveryPlus BACKUP/RECOVER BCS logic flow. . . . . . . . . 92
Catalog RecoveryPlus BACKUP/RECOVERY VVDS logic flow . . . . . . 93
RECOVER BCS SIMULATE(FORWARD) command . . . . . . . . . . . . . . 95
RECOVER VVDS SIMULATE(FORWARD) command . . . . . . . . . . . . . 95
DIAGNOSE ALIAS PEER mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
DIAGNOSE ALIAS NONPEER mode . . . . . . . . . . . . . . . . . . . . . . . . . . 98
DIAGNOSE ALIAS with INCLUDE-USERCATALOG option . . . . . . . . . 98
DIAGNOSE BCS-VVDS ALLRELATED example . . . . . . . . . . . . . . . . 101
DIAGNOSE VVDS-BCS ALLRELATED example . . . . . . . . . . . . . . . . 101
MERGECAT command to move all BCS records . . . . . . . . . . . . . . . . 102
MERGECAT command to move BCS records by LEVEL . . . . . . . . . . 103
MERGECAT command to move BCS records by ENTRIES . . . . . . . . 103
SUPERCLIP example syntax. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
ZAP PRINT of BCS and VVDS records . . . . . . . . . . . . . . . . . . . . . . . . 108
ZAP DELETE of a BCS and VVDS record. . . . . . . . . . . . . . . . . . . . . . 109
ZAP PATCH of a BCS and VVDS record . . . . . . . . . . . . . . . . . . . . . . 110

ICF Catalog Backup and Recovery: A Practical Guide

Tables
1-1
2-1
3-1
4-1
4-2
6-1
6-2

Location of data set information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

SMF record types for ICF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Useful IDCAMS commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Backup and recovery utilities relationship . . . . . . . . . . . . . . . . . . . . . . . 51
SMF record types for ICF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Duplex BCS backup overhead statistics . . . . . . . . . . . . . . . . . . . . . . . . 84
Duplex VVDS backup overhead statistics . . . . . . . . . . . . . . . . . . . . . . . 84

Copyright IBM Corp. 1999, 2001

10

ICF Catalog Backup and Recovery: A Practical Guide

Examples
7-1
7-2
7-3
7-4
7-5
7-6
7-7
7-8
7-9
7-10
7-11
7-12
7-13
7-14
7-15
7-16
7-17
7-18
7-19
7-20
7-21
7-22
7-23
7-24
7-25
7-26

Verifying the structure and content of the BCS . . . . . . . . . . . . . . . . . . 113

Backing up the BCS with EXPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Backing up the BCS with CR+ BACKUP . . . . . . . . . . . . . . . . . . . . . . . 114
Backing up the BCS with CR+ BACKUP . . . . . . . . . . . . . . . . . . . . . . . 116
BCS forward recovery with CR+ RECOVER . . . . . . . . . . . . . . . . . . . . 118
DIAGNOSE ALIAS master catalog job stream . . . . . . . . . . . . . . . . . . 122
DIAGNOSE ALIAS master catalog compare report . . . . . . . . . . . . . . . 122
DIAGNOSE ALIAS master catalog generated fixes. . . . . . . . . . . . . . . 124
BACKUP VVDS job stream and output listing . . . . . . . . . . . . . . . . . . . 126
RECOVER VVDS job stream. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
RECOVER VVDS job stream and output listing . . . . . . . . . . . . . . . . . 128
BACKUP VVDS job stream and output listing . . . . . . . . . . . . . . . . . . . 129
IDCAMS DELETE/DEFINE job stream and output listing . . . . . . . . . . 130
RECOVER VVDS job stream and output listing . . . . . . . . . . . . . . . . . 130
SUPERCLIP job stream and output listing. . . . . . . . . . . . . . . . . . . . . . 132
SUPERCLIP SIMULATE feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
REPRO NOMERGECAT to create an alternate master catalog . . . . . 135
CR+ MERGECAT to create an alternate master catalog . . . . . . . . . . . 136
DIAGNOSE VVDS to identify duplicate VVR in VVDS . . . . . . . . . . . . 139
VVDSFIX DELVVR command to delete a VVDS record . . . . . . . . . . . 140
CR+ ZAP VVDS DELETE to delete a VVDS record . . . . . . . . . . . . . . 141
CR+ ZAP PRINT of a VVDS record . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
VVDSFIX DELBCSR command to delete a BCS record . . . . . . . . . . . 142
CR+ ZAP BCS DELETE to delete a BCS record . . . . . . . . . . . . . . . . . 143
CR+ DIAGNOSE BCS-VVDS to search for errors . . . . . . . . . . . . . . . . 144
CR+ DIAGNOSE VVDS-BCS to search for errors . . . . . . . . . . . . . . . . 147

Copyright IBM Corp. 1999, 2001

11

12

ICF Catalog Backup and Recovery: A Practical Guide

Preface
ICF catalogs are essential system data sets. Even with todays high availability
storage subsystems and processors, there are still situations in which you need
to recover your catalogs. You should keep your catalogs healthy and be prepared
for a recovery situation. Also, you need to be sure that your recovery procedures
do not have a big impact on your production environment. To minimize the
recovery process, you should have a clear backup and recovery strategy, and the
proper utilities.
IBM Integrated Catalog Forward Recovery Utility (ICFRU) and Mainstar Catalog
RecoveryPlus are two of the available tools you can use to recover your catalogs.
ICFRU is a basic tool to help you in a forward recovery situation. It does not offer
a wide range of features, but is useful for catalog recovery. Mainstar's Catalog
RecoveryPlus offers a set of features to help you in your ICF catalog environment
maintenance.
This IBM Redbook provides information and practical examples on how to use
each of these products in a catalog recovery situation. It also provides useful
recommendations for storage administrators in implementing a catalog backup
and recovery plan. It is not intended to compare the two products but to show
how each of them work.
This redbook also provides a variety of practical tests to help you with the
different error scenarios you may find in your daily production activities.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization (ITSO), San Jose
Center.
Yolanda Cascajo Jimenez is a Enterprise Storage Software Project Leader at
the International Technical Support Organization, San Jose Center. She has
eight years of experience in storage systems, including DASD, tapes, and
storage management software. Before joining the ITSO in year 2000, Yolanda
worked as I/T Specialist in Spain, acting as pre-sales technical support and
consultant in storage systems. She has worked at IBM for 13 years.

Copyright IBM Corp. 1999, 2001

13

Ronald K. Ferguson is the President and CEO of Mainstar Software

Corporation, headquartered in Bellevue, WA, USA. He has over 30 years of
experience in IBM mainframe operating systems. His areas of expertise include
VSAM and ICF catalogs, as well as lecturing and writing extensively on these
topics.
Eileen A. McClintock is an independent contractor from Glendale, CA, teaching
MVS courses for IBM and its training partners. She has 25 years of experience in
IBM mainframe operating systems and worked at IBM for 14 years. Her last
assignment in IBM was large systems and storage technical support in the
Western Area System Center. Eileens areas of expertise include SMP/E, MVS,
and ICF catalogs.
Robert A. Matthews is a Senior Performance Analyst with IBM Global Services
Australia. He has over 30 years of experience in IBM mainframe operating
systems. His areas of expertise include tuning VSAM applications and ICF
catalogs.
The authors of the first edition of this book are:
Jon Tate, IBM UK
Frank Byrne, IBM UK
Klaus Stanislawiak, Software AG Germany
Wolfgang Wettig, IBM Global Services Germany
Thanks to the following people for their contributions to this project:
Charlie Burger, ATS
Mark Thomen, Catalog Development
SSG, San Jose
Elliot Hamilton
Elizabeth McGhee
Dave Roeser
Mainstar Software Corporation
Bob Haimowitz
ITSO, Pougkeepsie
Also, special thanks to the following people for their valuable contribution:
Cookie Bovier
Ron Ratcliffe

14

ICF Catalog Backup and Recovery: A Practical Guide

Notice
This publication is intended to help system programmers to ensure the good
health of their catalog environment, show how to use the correct tools, and
explain the recommended procedures in case recovery is necessary. The
information in this publication is not intended as the specification of any
programming interfaces that are provided by ICFRU or Mainstar Catalog
RecoveryPlus. See the PUBLICATIONS section of the IBM Programming
Announcement more information about what publications are considered to be
product documentation.

IBM trademarks
The following terms are trademarks of the International Business Machines
Corporation in the United States and/or other countries:
e (logo)
CICS
DB2
DFS
DFSMS/MVS
DFSMSdfp
DFSMSdss
DFSMShsm
DFSMSrmm
DFSORT
Enterprise Storage Server
FlashCopy
IBM
Magstar

Redbooks Logo
MORE
MVS
MVS/ESA
MVS/XA
Perform
RACF
RAMAC
Redbooks
SP
System/390
VTAM
z/OS

Comments welcome
Your comments are important to us!
We want our Redbooks to be as helpful as possible. Send us your comments
about this or other Redbooks in one of the following ways:
Use the online Contact us review redbook form found at:
ibm.com/redbooks

Send your comments in an Internet note to:

[email protected]

Mail your comments to the address on page ii.

Preface

15

16

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 1.

Introduction
As the number of systems and amount of data have grown over time, and with
them the number of catalogs, a successful catalog backup and recovery plan is
critical. This redbook discusses techniques and considerations for maintaining
the structural integrity of your catalogs, allowing for recovery should the catalogs
ever be compromised.
This introductory chapter describes:
The Integrated Catalog Facility (ICF) structure and evolution
Tape volume catalogs (VOLCATs)
ITSO test environment
Throughout this redbook, the term catalog refers to the Integrated Catalog
Facility (ICF) catalog. As of 01 January 2000, Virtual Storage Access Method
(VSAM) catalogs and OS CVOLs are not supported.

Copyright IBM Corp. 1999, 2001

1.1 ICF catalog structure and evolution

To introduce the concepts associated with backup and recovery, we revisit some
of the terminology associated with data set access and give a high-level overview
of the catalog structure associated with the Integrated Catalog Facility
environment.
To process a data set, you need to know its location and its physical attributes
such as record length and data set organization. Historically, volume location for
non-VSAM data sets has been obtained either from a catalog or from a DD
statement in JCL you provide. For VSAM or system-managed non-VSAM data
sets, it must be obtained from a catalog entry. The information required to identify
the volume where a data set resides can be thought of as a logical description of
the data. This is the information required to complete the allocation process.
The physical description of the data set includes such information as data set
organization, record length, blocksize, and exact location on the volume. This
information is not needed until you actually open the data set. For non-VSAM
data sets, that description has always been in the VTOC on the volume with the
data. The major change in philosophy introduced by ICF was the placement of
the physical description of VSAM components on the volume with the data
instead of in the catalog. This feature led to significant advantages in flexibility,
performance, and recoverability over the old-style VSAM catalogs.
An understanding of the structures involved, the information each contains, and
when that information is used is essential to recognizing when something has
gone wrong.

1.1.1 Catalog structure

The ICF catalog information is stored in two components:
Basic catalog structure (BCS): Can be considered the catalog
VSAM volume data set (VVDS): Can be considered an extension of the
volume table of contents (VTOC)

ICF Catalog Backup and Recovery: A Practical Guide

Figure 1-1 illustrates the relationship between the ICF catalog structures.

VOL002
VTOC

VOL00X

F1 - SYS1.VVDS.VVOL002
F1 - MY.KSDS.D
F1 - MY.KDSD.I
F1 - MY.NVSAM1

portable

BCS1

backup copy
BCS1
BCS1
created by
export
MY.NVSAM1
MY.NVSAM1

MY.NVSAM1
MY.KSDS
- MY.KSDS.D
- MY.KSDS.I
MY.KSDS
- MY.KSDS.D
- MY.KSDS.I
MY.KSDS - MY.KSDS.D - MY.KSDS.I

VVDS

MY.NVSAM1

NVR - MY.NVSAM1
VVR - MY.KSDS.I
VVR - MY.KSDS.D

MY.KSDS.I
MY.KSDS.D

Logical description
Allocation processing
Physical description
Open processing
On volume with data
Figure 1-1 Logical and physical description of data

The BCS is a VSAM KSDS. It functions as a directory simply indicating the

volumes on which a data set is located. A BCS is created when a user catalog
(UCAT) or master catalog (MCAT) is defined using Access Method Services
(AMS, commonly referred to as IDCAMS). A BCS does not have to be on the
same volume as the data sets it references, and there can be more than one
BCS on a volume. A BCS can contain pointers to data sets on any number of
different volumes, and a VVDS can describe data sets cataloged in any number
of different BCSs. There is no restriction imposed by the catalog structure on
where data sets can be placed.
The VVDS is a VSAM ESDS. It contains the information required to process
VSAM data sets in records known as VSAM volume records (VVRs). In a System
Managed Storage (SMS) environment, the VVDS also contains the SMS class
information for non-VSAM system-managed data sets in records known as
non-VSAM volume records (NVRs). There is one VVDS on each disk volume that

Chapter 1. Introduction

contains a VSAM or system-managed data set cataloged in an ICF catalog. Any

volume containing a BCS also contains a VVDS, because the BCS is itself a
VSAM data set. The VVDS is always on the same volume as the data sets it
describes and must be named SYS1.VVDS.Vvolser.
The records in both the VVDS and BCS are made up of variable length cells and
subcells grouped into various record types. The cell type of the first cell is
referred to as the record type. The ICF introduced the concept of the sphere
record, wherein one logical record in the catalog contains the information needed
to locate all related components of a data set.
There are four basic record types in a BCS:
Cluster: A sphere record containing information to locate related VSAM
components. For example, a cluster record for a KSDS contains information
to locate both the data and index components as well as the data and index
components of any alternate index that may be defined.
Non-VSAM data set: Contains name, volume, and if applicable, alias name
for a non-VSAM data set that is not a generation data set.
Generation data group: A sphere record containing all the information about
the group itself, such as limit, scratch/noscratch, and empty/noempty
processing options, as well as the information needed to locate each
generation data set in the group.
User catalog connector: Identifies the volume where the catalog is located
and its aliases. Created by the IDCAMS DEFINE UCAT | VOLCAT or IDCAMS
IMPORT CONNECT commands. It must be present in the master catalog for
a user catalog to be available to the system.
Three other record types should only exist if associated with one of the above
types:
Truename: Relates a VSAM component name to its cluster name
Path: Associated with an alternate index or cluster name
Alias: Can be associated with a non-VSAM data set, generation data set, or
user catalog. It is most commonly thought of as a high-level qualifier directing
catalog management to a particular user catalog for a given data set name. It
can also be simply an alternate name for a non-VSAM data set.
Two record types, library and volume, are unique to tape volume catalogs
(VOLCATs) and are discussed in 1.2, Tape volume catalogs on page 7.

ICF Catalog Backup and Recovery: A Practical Guide

1.1.2 Where a data set is described

The BCS, VVDS, and VTOC each supply a portion of the information required to
process a data set as illustrated in Table 1-1. Damage to any one of the pieces or
a mismatch between them may prevent you from accessing your data even if
there is no problem with the data set itself.
Table 1-1 Location of data set information
Information

VSAM data
set

System-managed
non-VSAM data
set

Non-systemmanaged
non-VSAM
data set

Uncataloged
non-VSAM
dataset

Volume,
data set type,
association,
ownership

BCS

BCS

BCS

n/a

SMS class info

BCS & VVDS

BCS & VVDS

n/a

n/a

Data set attributes

VVDS

VTOC

VTOC

VTOC

Extent description

VVDS &
VTOC

VTOC

VTOC

VTOC

Catalog name

VVDS

VVDS

n/a

n/a

Most data sets have entries in only one VVDS or VTOC. The exception is a
multi-volume data set, which has entries in the VVDS or VTOC of each volume it
occupies. Because the data set, VTOC, and VVDS are on the same volume,
backups of the volume contain all of the information necessary to recover the
data in the event of a volume failure. Because the BCS does not contain any
physical description of the data sets, it can be recovered without requiring data
sets to be restored to the same point in time.

1.1.3 Master and user catalogs

Master and user catalogs are identical in structure. They differ in how they are
used and what data sets are cataloged in them.
A master catalog is required to IPL. It is identified by a SYSCAT statement in the
LOADxx member of SYS1.PARMLIB or a SYSCATxx member of
SYS1.NUCLEUS. It must contain connector records for all user catalogs to be
accessed and aliases for them. Master catalogs on systems that share a given
user catalog must each have a connector record and should have all the same
aliases as well. The only data sets in the master catalog should be the SYS1 or
system data sets required for proper system initialization.

Chapter 1. Introduction

User catalogs provide access to application and user data sets. They should only
be accessed by alias, following your installations data set naming conventions.
There are no set rules as to how many you should have or how large they should
be. It depends entirely on your environment. Cataloging data sets for two
unrelated applications in the same catalog creates a single point of failure for
them that otherwise may not exist. An assessment of the impact of outage of a
given catalog may help determine if it is too big or would impact too many
different applications.

1.1.4 Catalog Address Space

Prior to DFP/XA 2.1, each user had the overhead of allocating, opening, and
closing each catalog referenced. Catalog management modules and control
blocks were located in common areas of virtual storage (LPA and CSA).
Significant virtual storage constraint relief and performance gains were realized
with the introduction of the Catalog Address Space (CAS).
In the CAS environment, the request for a catalog function is passed from the
users address space to the CAS using MVS cross memory services. Figure 1-2
shows how the CAS is utilized upon data set reference.

Catalog Address Space

Process
Request
User A Address Space

Reference
Data Set

User B Address Space

Control Blocks
and Buffers

Reference
Data Set

Call CAS

Call CAS

Wait
Continue

Wait
Continue

Figure 1-2 CAS usage for data set reference

The CAS processes requests concurrently, up to a user-selected value. Once

this threshold is reached, the caller has to wait for another task to complete.

ICF Catalog Backup and Recovery: A Practical Guide

A catalog is opened at the first reference and remains open until one of the
following actions occurs:
The operator issues a command to close the catalog.
The CAS address space terminates or is restarted by the operator.
The maximum number of catalogs allowed to be open is reached. In this
case, the least-recently used catalog is closed and the required catalog is
opened.
A control block in the CAS is built for each catalog and VVDS as it is opened.
Keep this in mind if you find attempts to access a particular BCS or VVDS are
resulting in recurrent errors. Recovery may be as simple as closing and
reopening the BCS or VVDS involved to rebuild the control blocks.
To further improve performance, two types of record caching were introduced:
In-storage catalog cache (ISC): With MVS/XA
Catalog data space cache (CDSC): With MVS/ESA
ISC is the default and caches catalog records within the CAS address space.
CDSC caches records in a data space associated with the CAS using the virtual
look-aside facility (VLF).
With shared catalogs, each systems CAS has its own cache, which creates a
potential for integrity problems. Solutions to these possible exposures are
discussed in later topics.
Catalog problems may necessitate your ending a request, closing or unallocating
a BCS or VVDS, or restarting CAS. Excellent information about the CAS and
communicating with it via the MVS MODIFY command can be found in DFSMS
Managing Catalogs, SC26-7409. This publication should be your first source of
information on anything regarding your catalog environment.

1.2 Tape volume catalogs

A tape volume catalog (VOLCAT) is a user catalog defined with the keyword
VOLCAT. It has the same internal structure as all other ICF catalogs. The
difference between a VOLCAT and other catalogs is its content. A VOLCAT
contains only tape library and tape volume records.
There are two varieties of tape volume catalog: general and specific. In a
system-managed tape (SMT) environment, you are required to have at least one
general VOLCAT. You may have one or several specific VOLCATs, or none at all.

Chapter 1. Introduction

Unlike other user catalogs, VOLCATs have required naming conventions. The
general VOLCAT is named hlq.VOLCAT.VGENERAL. The system default for the
high level qualifier (hlq) is SYS1. If any hlq other than SYS1 is used, it must be
specified in the SYSCAT statement in the LOADxx member of SYS1.PARMLIB
as well as in the SYSCATxx member of SYS1.NUCLEUS, if used in your
environment. To guarantee accuracy, all systems connected to a Tape Library
Dataserver should use the same high level qualifier for their tape volume
catalogs.
The general VOLCAT contains a tape library record for each system-managed
tape library. The record contains information such as the library name, library ID,
and device type. The general VOLCAT also contains tape volume records that
are not included in a specific VOLCAT.
A specific VOLCAT is named hlq.VOLCAT.V*, where the asterisk (*) in the third
qualifier is the first character of the volume serial numbers to be cataloged in it. A
specific VOLCAT contains only tape volume records. Tape volume records
include data such as the volser, recording technique, media type, SMS storage
group, and whether the data on the volume is compacted.
The tape configuration database (TCDB) is the collection of all VOLCATs. There
are IDCAMS commands to create, alter, or delete library and volume records in
the TCDB. They should be used only in a recovery situation and with caution
since they cannot update the library managers inventory or the tape
management systems database.
See DFSMS OAM Planning, Installation and Storage Adminstration Guide for
Tape Libraries, SC35-0427, and DFSMSrmm Implementation and Customization
Guide, SC26-7405, for more information on VOLCATs and system-managed
tape libraries.

1.3 The ITSO test environment

In our ITSO test environment, the sysplex consisted of three z/OS systems with
DFSMS. The examples provided in this redbook apply to all currently supported
releases of DFSMS. The system IDs were SC63, SC64, and SC65. We also had
two coupling facilities: CF03 and CF04. The structure of our sysplex is shown in
Figure 1-3. Two of the three systems share a master catalog.

ICF Catalog Backup and Recovery: A Practical Guide

CF03

Coupling
Facilities

SC63

SC64

z/OS 1.1
JES2

z/OS 1.2
JES2

SC65
z/OS 1.2
JES3

Shared
Master
Catalog

MCAT2

MCAT1

UCAT1

CF04

UCAT2

UCAT3

Figure 1-3 Structure diagram of the ITSO parallel sysplex

The examples used throughout this redbook relate to entries that were defined
and tested within this environment.

Chapter 1. Introduction

10

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 2.

Backing up catalogs
This chapter discusses the importance of an effective backup strategy for
catalogs in your environment. It also describes the techniques and utilities that
are available for performing a successful backup.
The following sections cover:
Backup strategy
Backup utilities
Considerations for the backup process

Note: Throughout this book, the backup of a catalog refers to a BCS.

Copyright IBM Corp. 1999, 2001

11

2.1 Backup strategy

Catalogs are essential system data sets. Backups exist solely to support
recovery requirements and, hopefully, will never be needed. The loss of a single
catalog can result in the loss of access to thousands of data sets. The loss of a
master catalog means an IPL of at least one system. In a sysplex environment
where the master catalog is shared, a broken master catalog may mean an
outage of the entire sysplex. Therefore, the most important attributes of any
backup are that it is readily available and usable when needed. The topics that
follow discuss a general backup strategy.

2.1.1 Backup frequency

The main factor in determining how frequently a given catalog should be backed
up is an assessment of the impact of an outage and how quickly it must be
recovered. A recoverability objective for any data set is a statement of the level of
currency to which it must be recovered and how long it can take. For example, an
objective to recover to point-of-failure in fifteen minutes would require a different
degree of preparedness than the much less demanding objective to restore last
weekends backup in a period of six hours.
In most installations, some BCSs are very volatile and some are relatively static
in terms of the amount of allocate and delete activity. A daily backup may be
sufficient for some catalogs while others may need to be backed up every few
hours. Restoring a catalog can be a relatively trivial task. Recovering it to an
acceptable level of currency can be very complex and time consuming.
Therefore, the time needed to recover a catalog may depend on how recently the
backup copy was made.

2.1.2 Backup techniques

There are a number of different techniques to back up catalogs. The following
sections discuss various backup utilities that you can employ in your
environment, along with considerations that are unique to each. Usually the
storage administrator decides which backup technique is appropriate. This can
be based on speed, utilities available, or other business factors such as auditor
requirements. The most important consideration when deciding upon a backup
technique is that you perform some form of checking to ensure that the backup of
the catalog was successful. We recommend that you test your backup and
recovery procedures to be sure that the backup copy you created is usable in the
event it is needed. You may also want to try different utilities to see which one
best suits your environment and the ease with which post-recovery updates can
take place.

12

ICF Catalog Backup and Recovery: A Practical Guide

2.2 Backup utilities

A backup copy of a catalog can be created by one of the following utilities:

IDCAMS EXPORT command

DFSMSdss logical dump
DFSMShsm BACKDS command
DFSMSdss physical dump
Advanced Copy Services (including FlashCopy, Snapshot, XRC, and PPRC)
Mainstar Catalog RecoveryPlus (see Chapter 6, Using Catalog
RecoveryPlus on page 69)

The following sections discuss each of the backup utilities. To help you verify that
the backup of your catalog has been successful, they also include sample control
statements for IDCAMS and DFSMSdss and the messages that you should
check.

2.2.1 IDCAMS EXPORT command

The syntax of the IDCAMS EXPORT command and the expected output are
shown in Figure 2-1. You should check that the following message is issued for
each catalog you backed up:
IDC0594I PORTABLE DATA SET CREATED SUCCESSFULLY

Only one catalog can be backed up at a time.

Chapter 2. Backing up catalogs

13

IDCAMS

SYSTEM SERVICES

TIME: 12:47:33

09/25/01

PAGE 1

EXAMINE NAME(UCAT.CRPLUS1) INDEXTEST

IDC01700I INDEXTEST BEGINS
IDC01724I INDEXTEST COMPLETE - NO ERRORS DETECTED
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
DIAGNOSE ICFCATALOG INDATASET(UCAT.CRPLUS1)
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
EXPORT UCAT.CRPLUS1 OUTFILE(OUT1) TEMPORARY
IDC0005I
IDC0594I
IDC1147I
IDC1147I
IDC0001I

NUMBER OF RECORDS PROCESSED WAS 8274

PORTABLE DATA SET CREATED SUCCESSFULLY ON 09/25/01 AT 12:48:12
IT IS RECOMMENDED THAT DIAGNOSE AND EXAMINE BE RUN BEFORE
IMPORT OF CATALOG
FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0

Figure 2-1 Sample IDCAMS EXPORT output

Notice the IDCAMS EXAMINE and DIAGNOSE commands that precede the
EXPORT in the example. We strongly recommend you include at least an
EXAMINE INDEXTEST in your backup job, regardless of how the backup is
done. These diagnostic commands are discussed in Chapter 3, Analyzing
catalog integrity on page 21. Catalog RecoveryPlus automatically invokes them
as part of its BACKUP command processing. See 6.4, BACKUP command on
page 75.
In the EXPORT command, the keyword TEMPORARY should be specified. If
PERMANENT is coded or allowed to default, the catalog is still exported as
temporary, but a warning message and non-zero return code are issued. Try not
to be complacent about non-zero return codes!
To ensure the integrity of the copy, access to the BCS is serialized, preventing
update access but allowing read access from the system performing the
command. Other systems will be unable to access the catalog, unless you are
using Global Resource Serialization (GRS) or an equivalent product to convert
the catalog RESERVE to a GLOBAL ENQUEUE.
Upon completion of a successful export, the catalogs self-describing record is
marked to indicate that another copy exists and that the original can be replaced.
Aliases are also backed up.

14

ICF Catalog Backup and Recovery: A Practical Guide

2.2.2 DFSMSdss logical dump

For an example of the DFSMSdss command to perform a logical dump of a
catalog, see Figure 2-2. Check that the following message is issued and shows
each catalog you specified in the INCLUDE statement:
ADR454I (001)-DTDSC(01), THE FOLLOWING DATA SETS WERE SUCCESSFULLY PROCESSED

All catalog names must be fully qualified. Multiple catalogs may be backed up
with a single command, and to the same output data set. For more information
about DFSMSdss, refer to DFSMSdss Storage Administration Guide,
SC35-0423.

PAGE 0001

5695-DF175

DFSMSDSS V2R10.0 DATA SET SERVICES

2001.268 13:14

DUMP DATASET(INCLUDE(UCAT.CRPLUS1
UCAT.CRPLUS2
MCAT.SANDBOX.R10.VSBOX11)) OUTDDNAME(OUTDD1)
ADR101I
ADR109I
ADR016I
ADR006I
ADR801I

(R/I)-RI01 (01),
(R/I)-RI01 (01),
(001)-PRIME(01),
(001)-STEND(01),
(001)-DTDSC(01),

ADR454I (001)-DTDSC(01),

ADR006I (001)-STEND(02),
ADR013I (001)-CLTSK(01),
ADR012I (SCH)-DSSU (01),

TASKID 001 HAS BEEN ASSIGNED TO COMMAND 'DUMP '

2001.268 13:14:02 INITIAL SCAN OF USER CONTROL STATEMENTS COMPLETED.
RACF LOGGING OPTION IN EFFECT FOR THIS TASK
2001.268 13:14:02 EXECUTION BEGINS
DATA SET FILTERING IS COMPLETE. 3 OF 3 DATA SETS WERE SELECTED:
0 FAILED SERIALIZATION AND 0 FAILED FOR OTHER REASONS.
THE FOLLOWING DATA SETS WERE SUCCESSFULLY PROCESSED
CLUSTER NAME
UCAT.CRPLUS1
CATALOG NAME
MCAT.SANDBOX.R10.VSBOX11
COMPONENT NAME UCAT.CRPLUS1
COMPONENT NAME UCAT.CRPLUS1.CATINDEX
CLUSTER NAME
UCAT.CRPLUS2
CATALOG NAME
MCAT.SANDBOX.R10.VSBOX11
COMPONENT NAME UCAT.CRPLUS2
COMPONENT NAME UCAT.CRPLUS2.CATINDEX
CLUSTER NAME
MCAT.SANDBOX.R10.VSBOX11
CATALOG NAME
MCAT.SANDBOX.R10.VSBOX11
COMPONENT NAME MCAT.SANDBOX.R10.VSBOX11
COMPONENT NAME MCAT.SANDBOX.R10.VSBOX11.CATINDEX
2001.268 13:14:07 EXECUTION ENDS
2001.268 13:14:07 TASK COMPLETED WITH RETURN CODE 0000
2001.268 13:14:07 DFSMSDSS PROCESSING COMPLETE. HIGHEST RETURN CODE IS 0000

Figure 2-2 Sample DFSMSdss logical dump messages

The catalogs aliases are saved in the backup copy. The lock attribute of the BCS
is also dumped. For a discussion of locking catalogs, see 4.1.5, Preventing
catalog activity during restore on page 52.

Chapter 2. Backing up catalogs

15

2.2.3 DFSMShsm
When utilizing DFSMShsm availability management or the DFSMShsm BACKDS
command, you must check the DFSMShsm activity logs for any error messages
related to your catalogs. If the catalog resides on a system-managed volume, the
management class is used to control the backup frequency and the number of
backup copies retained. This allows you to define different management classes
to achieve your needs for the backup frequency, and the number of backups you
want to retain for different catalogs. SMS invokes DFSMShsm, which in turn uses
IDCAMS EXPORT to perform the backup. Aliases are included in the backup
copy.

2.2.4 Advanced Copy Services

Advanced Copy Services is a collection of functions that provide solutions for
disaster recovery, data migration, and data duplication. Advanced Copy Services
uses the system data mover (SDM) as the movement engine, to efficiently and
reliably move large amounts of data between storage devices.
We can classify the different functions in terms of the nature of the copy they
create:
Dynamic copy functions

These functions constantly update the secondary copy as applications make

changes to the primary data source. Included in this group are the remote
copy functions:
Extended Remote Copy (XRC)
Peer-to-Peer Remote Copy (PPRC)
You can ensure a faster disaster recovery if you use remote copy functions to
copy volumes that contain the master catalog, key user catalogs, and system
control data sets to the recovery system.
Point-in-time copy functions

These functions provide an instantaneous copy, or view, of what the original

data looked like at a specific point in time. Included in this group are:
SnapShot for the RAMAC Virtual Array (RVA)
FlashCopy for the Enterprise Storage Server (ESS)
Both functions work on a volume basis and can be compared with the
DFSMSdss full volume copy. SnapShot does not support the copy of an ICF
catalog as a single data set.
When using any of the copy services, be aware that the only kind of error you are
prepared for is a media error. If the catalog contains logical or structural errors,
so does the copy.

16

ICF Catalog Backup and Recovery: A Practical Guide

2.3 Considerations for the backup process

It is important to understand the requirements that relate to your environment
and choose the backup method that satisfies those requirements. There are a
number of items that we feel are important for you to consider. They are worthy of
special mention because they are sometimes overlooked and only become
apparent during the recovery phase. Things to consider for all catalogs are:
Include diagnostics in the backup job.
Make more than one backup.
Catalog the backups in different catalogs.
Put the backups on different volumes and behind different control units to
avoid the exposure of a single point of failure.
Use different programs to make backups (see 7.1, Case study #1: Ensuring
good backups on page 113).

Each of the above items helps ensure the integrity and accessibility of your
backup when, and if, the need for it arises.

2.3.1 SMF data

SMF can be used to record certain catalog events. These records can be used
later during catalog recovery. Be sure that you are collecting the record types
shown in Table 2-1 on all systems and that you have backups of the data sets
where they are written.
Table 2-1 SMF record types for ICF
Record type

Event recorded

36

BCS successfully exported

60

VVR or NVR inserted, updated or deleted

61

BCS entry defined

65

BCS entry deleted

66

BCS entry altered

Chapter 2. Backing up catalogs

17

2.3.2 Master catalogs

Since you cannot IPL without a master catalog, we recommend that a procedure
be implemented to ensure an alternate master catalog is always available. For
information on defining and using an alternate master catalog, refer to DFSMS
Managing Catalogs, SC26-7409. The active master catalog sees the alternate
master as a user catalog.
An online copy of the master catalog is not sufficient preparation. You must also
create a LOADxx member in SYS1.PARMLIB or a SYSCATxx member in
SYS1.NUCLEUS that points to it. Your procedures should also include
maintaining it to reflect any updates made to the live master catalog. Consider
the following example:
DEFINE USERCATALOG (NAME(NEWUCAT).....

This results in a connector record being built in the active master catalog. But the
command necessary to keep the alternate master in sync should be:
IMPORT CONNECT OBJ((NEWUCAT) DEVT(devt) VOL(volser)) CAT(ALT.MCAT)

Periodic full-volume dumps of the master catalog volume could provide the ability
to perform a restore by using the stand-alone version of DFSMSdss. This is
especially useful if your site runs a single MVS image or does not share disk
between systems.

2.3.3 Tape volume catalogs

All VOLCATs must be backed up at the same point in time. Taken all together,
they represent the inventory of the tape library. Ensure that backup copies of
VOLCATs are not stored on a virtual tape residing in a 3494 Virtual Tape Server
(VTS). In addition, never store a backup copy of a VOLCAT to a physical
cartridge if you do not have read-compatible tape drives outside a 3494 or 3495
Automated Tape Library (ATL). The reason for this is that you will not be able to
access a system-managed tape volume if the VOLCAT is broken and needs to be
recovered.
A significant advantage of using DFSMSdss to perform the backups of your
VOLCATs is that it can backup all VOLCATs in one job step to one output data
set, regardless of the number of VOLCATs you have.

18

ICF Catalog Backup and Recovery: A Practical Guide

2.3.4 VSAM volume data set (VVDS) backup

It is important to remember that a backup copy of your catalog only contains data
from the BCS. A BCS record for a data set, even an entire BCS, could be deleted
and recreated without affecting the integrity of your data set. However, the VVDS
entry contains physical description details that must accurately describe the data
set. Restoring the VVDS, as a data set, could lead to unpredictable results if
existing information no longer agreed with the VTOC or attributes, such as
high-used relative byte address (HURBA), did not match the actual data.
A VVDS was traditionally not recovered, but rebuilt by recovering the data sets it
describes. Using this approach, a damaged VVR necessitated restoring a
back-level version of the data set and then doing forward recovery. Recovering
an entire VVDS entailed restoring all the VSAM and system-managed
non-VSAM data sets on the volume. A single damaged VVR on one volume of a
multi-volume database required restoring the entire database.
It is certainly understandable why we want to recover a VVDS independently of
the data sets. General recovery considerations and an ad hoc tool for repairing a
VVDS are discussed in 4.3.4, VVDS recovery on page 57. A comprehensive
approach to VVDS backup and recovery is discussed in Chapter 6, Using
Catalog RecoveryPlus on page 69.
According to DFSMS Managing Catalogs, SC26-7409, a VVDS should never be
backed up as a data set. It cannot be backed up using IDCAMS EXPORT or
REPRO commands. DFSMSdss can be used to back up a VVDS. The fully
qualified name and a DD statement allocating the volume must be specified as
shown in Figure 2-3.

//DFDSS
EXEC PGM=ADRDSSU,PARM='UTILMSG=YES'
//SYSPRINT DD SYSOUT=*
//INVOL DD UNIT=SYSDA,VOL=SER=SBOX75,DISP=SHR
//OUT1 DD DSN=YCJRES3.VVDSBACK,DISP=(,CATLG),UNIT=SYSDA,SPACE=(CYL,(2,2))
//SYSIN DD *
DUMP DATASET(INCLUDE(SYS1.VVDS.VSBOX75)) INDDNAME(INVOL) OUTDDNAME(OUT1)
PAGE 0001
5695-DF175
ADR101I (R/I)-RI01 (01),
ADR109I (R/I)-RI01 (01),
ADR006I (001)-STEND(01),
ADR378I (001)-DTDS (01),

DFSMSDSS V2R10.0 DATA SET SERVICES

2001.270 11:31
TASKID 001 HAS BEEN ASSIGNED TO COMMAND 'DUMP '
2001.270 11:31:16 INITIAL SCAN OF USER CONTROL STATEMENTSCOMPLETED.
2001.270 11:31:16 EXECUTION BEGINS
THE FOLLOWING DATA SETS SUCCESSFULLY PROCESSED FROM VOLUME SBOX75
SYS1.VVDS.VSBOX75
ADR006I (001)-STEND(02), 2001.270 11:31:20 EXECUTION ENDS
ADR013I (001)-CLTSK(01), 2001.270 11:31:20 TASK COMPLETED WITH RETURN CODE 0000
ADR012I (SCH)-DSSU (01), 2001.270 11:31:20 PROCESSING COMPLETE. HIGHEST RETURN CODE IS 0000

Figure 2-3 Sample DFSMSdss logical dump of a VVDS

Chapter 2. Backing up catalogs

19

20

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 3.

Analyzing catalog integrity

This chapter describes why it is necessary to analyze the structural integrity of
your catalogs and how to ascertain if the data set information in the BCS, VVDS,
and VTOC is correct and synchronized. It shows what utilities are available and
how to recognize different types of errors.
The sections in this chapter cover:

General diagnostic considerations

Utilities and tools available for diagnosis
Structural errors
Content errors
Synchronization errors
Catalog sharing problems
CAS problems due to catalog errors

Copyright IBM Corp. 1999, 2001

21

3.1 General diagnostic considerations

The purpose of diagnostics is to identify which structure (BCS, VVDS, VTOC, or
VTOCIX) is in error and the extent of the damage (entries or entire structure) in
preparation for recovery. You have arrived at this point because of some kind of
symptom. It is important to keep the big picture in mind.
For example, a user reports getting a data set not found condition. You need to
know more. A JCL error data set not found is very different from an 'x13' abend
or IDCAMS message IEC143I or a non-zero return code out of a utility. The
important question to ask is: What was happening when the message was
issued? This type of JCL error occurs when JES has invoked MVS allocation
routines before your program even has control. Simply put, allocations
responsibility is to complete the blanks in the following message for each DD
statement in the JCL:
IEF237I

device

ALLOCATED TO

ddname

If VOL=SER= is not specified in the JCL, or the data set has not been passed
from a prior step, allocation invokes catalog management to do a locate. The only
structure involved at this point is the BCS. Therefore, it would be the focus of your
diagnostics. You have no reason to suspect a problem in the VVDS or VTOC
because you did not go that far.
After control is passed to the program on your EXEC statement, there are more
possibilities as to what was happening when the data set was not found.
Programs, such as IDCAMS, allow you to put data set names or references to
DD names in the command stream. If you are referring to a DD name, allocation
has already occurred. If you use the data set name, IDCAMS dynamically
invokes allocation. In order to know what was happening, you need to look at the
message prefixes:

IEF messages indicate allocation.

IKJ messages indicate dynamic allocation.
IEC3 messages indicate catalog management.
IEC1 messages indicate open processing, and:
If non-VSAM, IEC6 messages indicate processing the VTOC.
If VSAM, IEC3 return code 50 messages indicate processing the VVDS.

Such utilities as IDCAMS, IEBGENER, or IEBCOPY hardly ever abend. They are
written to set return codes in situations where your application programs might
abend. The messages they direct to SYSPRINT are the most informative. Check
them before worrying about messages in the job log.

22

ICF Catalog Backup and Recovery: A Practical Guide

So step back, take a deep breath and look around before you spend time
diagnosing or recovering the wrong thing. Think about what was happening and
what structures are involved at that time.

3.2 Utilities and tools available for diagnosis

A variety of utilities and tools are available to diagnose the different ICF
components and their relationship to each other. We list some of those utilities
below. For all references to Catalog RecoveryPlus, see Chapter 6, Using
Catalog RecoveryPlus on page 69.
Catalog

IDCAMS

LISTCAT VOLUMES
EXAMINE
DIAGNOSE
PRINT

Catalog RecoveryPlus

DIAGNOSE
ZAP PRINT

DFSMSdss

PRINT

ISPF/PDF
VVDS

IDCAMS

LISTCAT ALL
DIAGNOSE
PRINT

Catalog RecoveryPlus

DIAGNOSE
ZAP PRINT

DFSMSdss

PRINT

Chapter 3. Analyzing catalog integrity

23

 VTOC / VTOCIX

IEHLIST

LISTVTOC DUMP | FORMAT | INDEXDSN

Catalog RecoveryPlus

DIAGNOSE
ZAP PRINT

DFSMSdss

PRINT
DEFRAG

ICKDSF

BUILDIX
BUILDOS

ISPF/PDF

3.2.1 VSAM Knowledge Database

You may also find the Catalog and VSAM Knowledge Base to be helpful in
problem resolution. It is a question-and-answer driven knowledge database that
resides on the DFSMS/MVS Technical Support Web site under Technical
Database at: http://knowledge.storage.ibm.com/

3.3 Structural errors

A structural error in a catalog will most likely result in the failure of your catalog
backup job. It may result in a lack of access to data sets defined in the damaged
catalog. A structural error is a problem with the BCS as a VSAM KSDS, not as a
catalog. Possible causes could be an interrupted control interval (CI) or control
area (CA) split or improper sharing.
You should suspect the structural soundness of a catalog if non-zero return
codes are issued during backup or if a catalog is causing jobs to fail and you
cannot trace the failure to unsynchronized entries. We suggest that you regularly
run jobs to check the integrity of your catalogs. This creates additional overhead,
but may decrease your chances of suffering an outage if an error can be fixed
before it becomes more serious. Like a computer virus, a structural error may
exist and do no harm until a certain condition occurs.

24

ICF Catalog Backup and Recovery: A Practical Guide

3.3.1 Checking the structural integrity

The IDCAMS EXAMINE command analyzes and reports on the structural
integrity of the index and data components of any KSDS. Therefore, it can be
used to analyze a BCS but not a VVDS. The EXAMINE command includes two
possible tests:
INDEXTEST: This test is the default. It evaluates the full index component
(index set and sequence set). It checks vertical and horizontal pointers from
one index record to the next and correlates index records with data control
areas.
DATATEST: This test evaluates the index sequence set and the data
component by sequentially reading all data control intervals. Tests are
conducted to ensure record and control interval integrity, including the
connection between sequence set entries and data control interval contents.
Such conditions as duplicate records or records out of sequence can be
detected.

Figure 3-1 shows the JCL to invoke IDCAMS and do a VERIFY and EXAMINE of
a BCS. Sample output illustrating various error conditions can be found in
Chapter 7, Case studies on page 111.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
VERIFY DATASET(UCAT.VSBOX09)
EXAMINE NAME(UCAT.VSBOX09) INDEXTEST DATATEST ERRORLIMIT(100)

Figure 3-1 Sample JCL to VERIFY and EXAMINE a BCS

To use EXAMINE on a BCS, you must have READ authority to the RACF
STGADMIN.IDC.EXAMINE.DATASET profile, if it is defined.
The ERRORLIMIT keyword is used to limit the number of error messages
printed. It defaults to over two billion so you may want to specify something to
prevent runaway output! The parameter only affects message output and does
not stop the EXAMINE processing. Specifying a value of zero suppresses the
printing of detailed error messages.
We recommend that you issue an IDCAMS VERIFY command prior to
EXAMINE. There have been cases where EXAMINE reports serious errors when
there is no problem with the catalog. The symptoms vary, but if you have
encountered the problem, you usually see the following error messages:
IDC11700I HIGH-LEVEL INDEX STRUCTURE IS NOT UNIQUE
IDC21701I MAJOR ERRORS FOUND BY INDEXTEST

Chapter 3. Analyzing catalog integrity

25

If you run a manual IDCAMS VERIFY on the catalog, a subsequent EXAMINE

will most likely run without errors.
In most cases, a catalog structural error means that you need to recover the BCS
from the most recent backup copy.

3.3.2 The consequences of neglecting structural integrity

Backup utilities may fail if they detect a structural error and leave you without a
backup copy of the catalog. This may require some explaining in a recovery
situation! Even worse, there is the chance that backup utilities will not detect a
structural error in your catalog and you will later discover that the backup copies
taken are unusable. This will result in an additional, unnecessary outage if you
need to recover the catalog. You need to find the last usable backup copy, restore
it, and begin what may be a great deal of forward recovery as described in 4.4,
Forward recovery on page 57.

3.4 Content errors

Structural errors are generally detected by the system or access method.
Content errors are detected by the application program, in this case, catalog
management. Using the IDCAMS DIAGNOSE command, you can analyze the
validity of the content of BCS and VVDS records.
To use DIAGNOSE on a BCS, you must have READ authority to the RACF
STGADMIN.IDC.DIAGNOSE.CATALOG profile, if it is defined. For a VVDS, you
must have READ authority to the STGADMIN.IDC.DIAGNOSE.VVDS profile, if it
is defined.
Be aware that DIAGNOSE assumes a BCS is structurally correct. It is possible
for DIAGNOSE to complete with a return code of zero when EXAMINE shows
structural errors.
Also be aware that DIAGNOSE does not use the in-storage control blocks but
does its own open of the BCS or VVDS. Users on a system may experience
problems accessing data sets through a given BCS or VVDS, but DIAGNOSE
completes with a return code zero. Simply refreshing the control blocks, by
closing the BCS or VVDS with the MVS MODIFY CATALOG command, may
quickly solve your problems. The reverse may also be true. Users may have no
problems, but a routine diagnostic job reports that DIAGNOSE is unable to open
the BCS or VVDS. Then you have a serious problem. Your users are just not
experiencing it yet!

26

ICF Catalog Backup and Recovery: A Practical Guide

For a thorough discussion of the IDCAMS DIAGNOSE command, see Analyzing

Catalogs for Errors and Synchronization in DFSMS Managing Catalogs,
SC26-7409. Catalog RecoveryPlus provides similar function in its DIAGNOSE
command, which is discussed in 6.6, DIAGNOSE command on page 96. This
chapter addresses the use of the IDCAMS DIAGNOSE command.

3.4.1 DIAGNOSE of a BCS

DIAGNOSE of a BCS checks each record for invalid data. For example, finding a
GDG name cell in the middle of a VSAM cluster record would be an error. It also
checks for invalid relationships between entries. For example, truename records
must exist for each component of a cluster and must point back to the cluster
record. If an alias record exists for a non-VSAM data set, the non-VSAM record
must also name the alias.
Figure 3-2 shows a simple DIAGNOSE of a BCS showing both content and
relational errors. Records where errors are encountered are displayed in dump
format. Only the interpreted side of the dump output is reproduced in the figure.

Chapter 3. Analyzing catalog integrity

27

DIAGNOSE ICFCATALOG INDATASET(UCAT.CRPLUS2)

IDC21364I ERROR DETECTED BY DIAGNOSE:
ICFCAT ENTRY: YCJRES2.CRPLUS.TESTE.E000026.DATA (T)
RECORD: YCJRES2.CRPLUS.TESTE.E000026.DATA /00
OFFSET: X'0036'
REASON: 26 - CELL TYPE INVALID IN CONTEXT
IDC21365I ICFCAT RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS.TESTE.E000026.DATA /00
*.!..T..D.YCJRES2.CRPLUS.TESTE.E0*
*00026.DATA
........YCJ*
*RES2.CRPLUS.TESTE.E000026.
*
IDC21364I ERROR DETECTED BY DIAGNOSE:
ICFCAT ENTRY: YCJRES2.CRPLUS.TESTK.K000026.DATA (D)
RECORD: YCJRES2.CRPLUS.TESTK.K000026 /00
OFFSET: X'004A'
REASON: 23 - TRUENAME LOOP FAILURE
IDC21365I ICFCAT RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS.TESTK.K000026 /00
*....C....YCJRES2.CRPLUS.TESTK.K0*
*00026
...........*
*............D.$..YCJRES2.CRPLUS.*
*TESTK.K000026.DATA..............*
*...........SBOX75........}......*
*.......I.*..YCJRES2.CRPLUS.TESTK*
*.K000026.INDEX..................*
*.......SBOX75........}..........*
IDC21365I ICFCAT RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS.TESTK.K000026.DATA /00
*.!..T..D.YCJRES2.CRPLUS.TESTK.K0*
*00026.DATA
........YCJ*
*RES2.CRPLUS.TESTZ.K000026.
*
IDC21364I ERROR DETECTED BY DIAGNOSE:
ICFCAT ENTRY: YCJRES2.CRPLUS.TESTK.K000026.DATA (T)
RECORD: YCJRES2.CRPLUS.TESTK.K000026.DATA /00
OFFSET: X'0036'
REASON: 20 - ASSOCIATION NOT FOUND
IDC01371I RECORD DISPLAY SUPPRESSED, ALREADY DUMPED.
IDC21363I THE FOLLOWING ENTRIES HAD ERRORS:
YCJRES2.CRPLUS.TESTE.E000026.DATA (T) - REASON CODE: 26
YCJRES2.CRPLUS.TESTK.K000026.DATA (D) - REASON CODE: 23
YCJRES2.CRPLUS.TESTK.K000026.DATA (T) - REASON CODE: 20

Figure 3-2 DIAGNOSE BCS output showing content errors

28

ICF Catalog Backup and Recovery: A Practical Guide

None of the above errors give you any reason to suspect the integrity of your
data sets. Recovery consists simply of deleting and recreating the catalog
entries. The IDCAMS DELETE NOSCRATCH, DELETE TRUENAME (if
necessary), and DEFINE RECATALOG commands can accomplish that without
touching the data set. You may be able to use ISMF or ISPF/PDF Data Set List
Utility (3.4) option u (uncatalog) followed by option c (catalog).

3.4.2 DIAGNOSE of a VVDS

DIAGNOSE of a VVDS checks the content of each VVR and NVR and can detect
duplicates. For each entry, it also checks for the presence of a F1 DSCB in the
VTOC. Extent information in the VVR and VTOC must match for VSAM
components (NVRs do not contain extent information). Figure 3-3 shows a
simple DIAGNOSE of a VVDS with duplicate VVRs and a discrepancy in extent
information between the VVDS and VTOC. Notice the VVDS is allocated on a DD
statement and referenced with the INFILE parameter. Records with errors are
displayed in dump format but have not been reproduced in the figure.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//D1 DD UNIT=SYSDA,VOL=SER=SBOX36,DISP=SHR,DSN=SYS1.VVDS.VSBOX36,AMP='AMORG'
//SYSIN
DD *
DIAGNOSE VVDS INFILE(D1)
IDC21364I ERROR DETECTED BY DIAGNOSE:
VVDS ENTRY: YCJRES2.CRPLUS4.VVDSTST1.DATA (Z)
RECORD: X'0000B160'
OFFSET: X'0002'
REASON: 41 - DUPLICATE VVR/NVRS IN VVDS
IDC21365I VVDS RECORD DISPLAY:
RECORD: X'0000B160'
(dump of VVR)
IDC21364I ERROR DETECTED BY DIAGNOSE:
VVDS ENTRY: YCJRES2.CRPLUS4.VVDSTST1.DATA (Z)
RECORD: X'0000B160'
OFFSET: X'014B'
REASON: 16 - VVDS AND VTOC STARTING CCHH UNEQUAL
IDC21365I VVDS RECORD DISPLAY:
RECORD: X'0000B160'
(dump of VVR)
IDC21365I VTOC RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS4.VVDSTST1.DATA
(dump of F1 DSCB)

Figure 3-3 DIAGNOSE VVDS output showing content errors

Chapter 3. Analyzing catalog integrity

29

3.5 Synchronization errors

In addition to the internal structure and content of catalogs, you should also
check for synchronization errors. Inconsistencies between the BCS, VVDS, and
VTOC may make a data set inaccessible or otherwise unusable.
Synchronization errors may be the result of the IDCAMS DEFINE, DELETE, or
ALTER jobs that were canceled by force, or due to system errors not allowing a
job to complete successfully. In either case, the necessary cleanup actions may
not have been done. Errors could also result from someone scratching a DCSB
from the VTOC rather than deleting a data set with a utility that takes care of
BCS, VVDS and VTOC entries. Most frequently they are simply the result of
having restored a BCS or volume and not yet having completed forward recovery
to resynchronize all structures with reality.
The two most common synchronization errors are:
BCS entry missing in catalog named in VVR or NVR
VVDS entry missing on volume named in BCS

Note that the first condition does not actually mean the data set is not cataloged.
It simply is not in the BCS where the VVR or NVR says it is cataloged. It could be
cataloged in a different BCS. Note also that the second condition does not mean
the data set does not exist. It is simply not on the volume named in the catalog. It
could be elsewhere.
You may also discover orphan VVRs or NVRs with no BCS or VTOC entries, or
a situation where there are BCS and VVDS entries but no entry in the VTOC. In
either of these cases, the data set no longer exists on the volume where it is
expected to be. Recovering requires you to clean up the left over pieces and then
find the data set and recatalog it or restore it from a backup.
VSAM data sets and non-VSAM system-managed data sets with a high level
qualifier of SYS1 can be cataloged in more than one BCS. This can be done with
IDCAMS DEFINE RECATALOG. In this case, the VVR or NVR still points to the
original catalog. This is a situation which might be detected by DIAGNOSE, but it
is not considered a structural or synchronization error.

3.5.1 Detecting synchronization errors with DIAGNOSE

Using the IDCAMS DIAGNOSE command, you can analyze the content and
validity of related BCS and VVDS records. Specify the COMPAREDD or the
COMPAREDS keyword with the DIAGNOSE command to indicate which data
sets are to be checked to confirm that they point to the BCS or VVDS being
diagnosed. DIAGNOSE does not fix anything. It simply identifies any
discrepancies.

30

ICF Catalog Backup and Recovery: A Practical Guide

To compare a BCS against the VVDS, specify:

DIAGNOSE ICFCATALOG INDATASET(bcsname) COMPAREDD(voldd)

To compare a VVDS against a BCS, specify:

DIAGNOSE VVDS INFILE(voldd) COMPAREDS(bcsname)

A VVDS must be identified with the INFILE or COMPAREDD parameter and a

DD statement. A BCS may be referenced using INDATASET or COMPAREDS. To
make sure that all possible errors will be detected, you must perform both
checks. Only one BCS or VVDS can be diagnosed at a time, but up to 99 BCSs
or VVDSs can be specified on the compare parameter.

DIAGNOSE BCS compare

Figure 3-4 illustrates processing that occurs when you diagnose a BCS
comparing to a VVDS using the command:
DIAGNOSE ICFCATALOG INDATASET(BCS1) COMPAREDD(VOLDD1)

VOL001
BCS1
DS1 - VOL001
DS2 - VOL002
DS3 - VOL001
DS4 - VOL003
DS5 - VOL999

SYS1.VVDS.VVOL001

DS1 - BCS1
DS7 - BCS3
DS8 - BCS3
DS9 - BCS1
VTOC
DS9
DS8
DS7
DS1

Figure 3-4 DIAGNOSE processing: BCS compared to VVDS

Note the following points:

Every record in BCS1 is diagnosed.
Only the VVDS entries on VOL001 pointed to by BCS1 are diagnosed.

Chapter 3. Analyzing catalog integrity

31

 Any diagnose of a VVDS entry includes comparing information with the

corresponding VTOC entry.
The command would report a missing VVDS entry for DS3.
The fact that data set DS9 on VOL001 is not cataloged in BCS1 would not be
detected.

If you are running DIAGNOSE because of a problem you encountered, you

generally know which VVDSs you want to name on the COMPAREDD parameter.
However, if you are doing a general analysis, you can determine which VVDSs
are related to a given BCS by using the IDCAMS command:
LISTCAT LEVEL(SYS1.VVDS) CATALOG(bcsname)

This LISTCAT command shows only the volumes for which a

SYS1.VVDS.Vvolser entry exists. There may be VVDS entries for volumes on
which the catalog has no data sets. There may also be entries in the BCS for
data sets that reside on volumes for which there is no SYS1.VVDS.Vvolser entry.
Either of these conditions results in a non-zero return code. A simple DIAGNOSE
of the BCS identifies them with the following messages:
IDC11374I THESE ADDITIONAL CATALOG REFERENCED VOLUMES WERE ENCOUNTERED
IDC11362I THE FOLLOWING CATALOG REFERENCED VOLUMES WERE NOT ENCOUNTERED

The situations indicated by these messages may be temporary as data sets are
allocated and deleted on the volume. However, you can remedy them by
executing the following IDCAMS commands.
In the case of IDC11374I message, create the VVDS entry in the BCS and
add the BCS name to the VVCR with:
DEFINE CLUSTER (NAME(SYS1.VVDS.Vvolser) VOLUME(volser) NONINDEXED RECATALOG) CATALOG(bcsname)

In the case of IDC11362I message, delete the extraneous VVDS entry from
the BCS and remove the BCS name from the VVCR with:
DELETE SYS1.VVDS.Vvolser NOSCRATCH CATALOG(bcsname)

32

ICF Catalog Backup and Recovery: A Practical Guide

Figure 3-5 shows a diagnosis of the BCS UCAT.CRPLUS2 checking the

dependent records in the VVDS on volume SBOX38. No VVR is found for
YCJRES2.CRPLUS.TESTK.K000950.DATA. Only the interpreted side of the
dump of the BCS record is shown. The data component of the record indicating
that it resides on volume SBOX38 is highlighted.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//D1 DD UNIT=SYSDA,VOL=SER=SBOX38,DISP=SHR,DSN=SYS1.VVDS.VSBOX38,AMP='AMORG'
//SYSIN
DD *
DIAGNOSE ICFCATALOG INDATASET(UCAT.CRPLUS2) COMPAREDD(D1)
IDC21364I ERROR DETECTED BY DIAGNOSE:
ICFCAT ENTRY: YCJRES2.CRPLUS.TESTK.K000950.DATA (D)
RECORD: YCJRES2.CRPLUS.TESTK.K000950 /00
OFFSET: X'0087'
REASON: 51 - VVDS ENTRY NOT FOUND. SCAN VVDS FAILED.
IDC21365I ICFCAT RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS.TESTK.K000950 /00
*....C....YCJRES2.CRPLUS.TESTK.K0*
*00950
...........*
*............D.$..YCJRES2.CRPLUS.*
*TESTK.K000950.DATA..............*
*...........SBOX38........{......*
*.......I.*..YCJRES2.CRPLUS.TESTK*
*.K000950.INDEX..................*
*.......SBOX38........{..........*
*.
*

Figure 3-5 Sample JCL to DIAGNOSE a BCS and compare to two VVDSs

Chapter 3. Analyzing catalog integrity

33

DIAGNOSE VVDS compare

Figure 3-6 illustrates processing that occurs when you diagnose a VVDS
comparing to a BCS using the command:
DIAGNOSE VVDS INFILE(VOLDD1) COMPAREDS(BCS1)

VOL001
BCS1
BCS1

DS1 - VOL001
DS2 - VOL002
DS3 - VOL001
DS4 - VOL003
DS5 - VOL999

SYS1.WDS.VVOL001

DS1 - BCS1
DS7 - BCS3
DS8 - BCS3
DS9 - BCS1
VTOC
DS9
DS8
DS7
DS1

Figure 3-6 DIAGNOSE processing: VVDS compared to BCS

Note the following points:

Every record in SYS1.VVDS.VVOL001 is diagnosed. Diagnosis includes
comparing information with the corresponding VTOC entry.
Only BCS entries that are diagnosed are those in BCS1 pointed to by VVDS
entries.
The command would report a missing BCS entry for DS9.
The BCS1 entry for DS3 pointing to a data set that is not on VOL001 would
not be detected.

If you are running DIAGNOSE because of a problem you encountered, you

generally know which BCSs you want to name on the COMPAREDS parameter.
Otherwise, to determine which BCSs are related to a VVDS, you can print the
first record of the VVDS, the VSAM volume control record (VVCR). It contains up
to 36 related BCS names. If the number of BCS names in the VVCR exceeds 36,

34

ICF Catalog Backup and Recovery: A Practical Guide

a BCS name extension record (VVCN) is constructed and stored within the
VVDS, with its RBA address stored at offset x'0000C' in the VVCR. You can tell if
you have a VVCN if this word is not zeroes, in which case, you will need to print
the record at that RBA.
There may be entries for catalogs that no longer have data sets on the volume.
There may also be VVRs or NVRs in the VVDS that are cataloged in BCSs for
which there is no entry in the VVCR. Either of these conditions will result in a
non-zero return code. A simple DIAGNOSE of the VVDS identifies them with the
following messages:
IDC11375I THESE ADDITIONAL VVDS REFERENCED CATALOGS WERE ENCOUNTERED
IDC11367I THE FOLLOWING VVDS REFERENCED CATALOGS WERE NOT ENCOUNTERED

There is no need to do a comparison to the BCSs identified on IDC11367I.

However, you should include those on IDC11375I, even though they do not
appear in the VVCR. Figure 3-7 shows how to print the VVCR and what appears
in the interpreted side of the dump output. The VVCR in this sample names nine
BCSs.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//INDD1
DD DSN=SYS1.VVDS.VSBOX03,DISP=SHR,UNIT=SYSDA,VOL=SER=SBOX03
//SYSIN
DD *
PRINT INFILE(INDD1) COUNT(1)
LISTING OF DATA SET -SYS1.VVDS.VSBOX03
RECORD SEQUENCE NUMBER - 1
*.8..VVCR................UCAT.VSB*
*OX01
*
*
............MCAT.SANDBOX.R10*
*.VSBOX11
....*
*........UCAT.VSBOX09
*
*
............*
*MCAT.SANDBOX.Z02.VSBOX11
*
*
............UCAT.CRP*
*LUS3
*
*
..&... .....UCAT.CRPLUSA
*
*
..-.*
*.. .....UCAT.CRPLUSB
*
*
...... .....*
*UCAT.CRPLUSC
*
*
...... .....UCAT.CRP*
*LUSD
*
*
...... .....
*

Figure 3-7 Printing the VVCR showing BCS names

Chapter 3. Analyzing catalog integrity

35

Figure 3-8 shows a diagnose of the VVDS on volume SBOX38 checking the
dependent records in the BCS UCAT.CRPLUS2. No BCS entry is found for the
cluster YCJRES2.CRPLUS.TESTE.E00001. Only the interpreted side of the
dump of the VVR is shown. The VVR name is the data component name. The
fields in the VVR that name the cluster and the BCS where it should be cataloged
are highlighted.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//D1 DD UNIT=SYSDA,VOL=SER=SBOX38,DISP=SHR,DSN=SYS1.VVDS.VSBOX38,AMP='AMORG'
//SYSIN
DD *
DIAGNOSE VVDS INFILE(D1) COMPAREDS(UCAT.CRPLUS2)
IDC3302I ACTION ERROR ON UCAT.CRPLUS2
IDC3351I ** VSAM I/O RETURN CODE IS 16 - RPLFDBWD = X'93080010'
IDC21364I ERROR DETECTED BY DIAGNOSE:
VVDS ENTRY: YCJRES2.CRPLUS.TESTE.E000001.DATA (Z)
RECORD: X'0003716B'
OFFSET: X'0002'
REASON: 19 - CATLG ENTRY NOT FOUND
IDC21365I VVDS RECORD DISPLAY:
RECORD: X'0003716B'
*.,..Z......YCJRES2.CRPLUS.TESTE.*
*E000001.DATA..YCJRES2.CRPLUS.TES*
*TE.E000001..UCAT.CRPLUS2.YCJRES2*
*.CRPLUS.TESTE.E000001.... ......*
*..................&.............*
*.....+/.........................*
*.....+/......-..-...............*
*........&......................+*
*/...............................*
*................................*
*..........................J...J.*
*...........
*

IDC11367I THE FOLLOWING VVDS REFERENCED CATALOGS WERE NOT ENCOUNTERED:

UCAT.CRPLUS3

Figure 3-8 Sample JCL to DIAGNOSE a VVDS and compare to a BCS

3.5.2 Repairing synchronization errors

Different errors may force you to take different repair actions. Your goal is to have
BCS, VVDS, and VTOC entries that allow you to successfully allocate, open, and
process the data set. If a data set must be restored, be sure to clean up all
pieces related to the existing data set. For example, a leftover VVR does not

36

ICF Catalog Backup and Recovery: A Practical Guide

cause reallocation to fail, but results in the existence of a duplicate VVR. The
most frequently used IDCAMS commands and their functions are listed in
Table 3-1. Some of these actions can also be done with ISPF/PDF Data Set List
Utility (3.4) or ISMF.
Table 3-1 Useful IDCAMS commands
Command

Action

DELETE NOSCRATCH

Removes only BCS entries; VVDS and

VTOC not affected.

DELETE TRUENAME

Removes unrelated truename record;

associated cluster record does not exist.

DELETE VVR or DELETE NVR

Removes unrelated VVR or NVR; also

removes VTOC entry, if present.

DEFINE RECATALOG

Creates a BCS entry pointing to existing

VVRs or NVR only in the BCS they name.

DEFINE NONVSAM

Creates BCS entry pointing to

non-system-managed non-VSAM data
set.
Attention: The existence of data set on
named volume is not verified.

3.6 Catalog sharing problems

Catalog structural and synchronization errors may result from failures in
establishing the proper catalog sharing environment.

3.6.1 A review of the SHAREOPTIONS parameter

The IDCAMS DEFINE parameter SHAREOPTIONS indicates the level of shared
access control that catalog management is to provide for a catalog. There are
only two allowable specifications of SHAREOPTIONS for a catalog (3 3) or
(3 4). The default is (3 4).
Whichever is coded has nothing to do with whether the catalog is actually
shared. Rather, it indicates whether catalog management is to assume sharing
and provide shared access integrity support. The importance of this distinction
will become apparent as we continue with this review.

Chapter 3. Analyzing catalog integrity

37

Defining the level of sharing within a system

The first parameter defines the level of shared access control that catalog
management is to provide among regions within the same system. The value 3
indicates that any number of tasks can have the catalog open for both read and
write, but it is the users responsibility to provide protection against multiple
updating. For a catalog that resides on a non-shared disk volume, this is the
correct value to specify since all catalog updates take place from within a single
address space catalog management (CAS).
This could get you into trouble if a user program opens the catalog as a data set,
in update mode, and updates it at the same time as catalog management is
updating it. With SHAREOPTIONS value 3 specified, there is no integrity control
provided by catalog management. User programs that ignore write integrity
guidelines could cause lost or inaccessible records, uncorrectable catalog errors,
program checks, or other damage to catalogs.

Defining the level of sharing across systems

The second parameter specifies the level of shared access control that catalog
management is to provide across systems. The value 3 indicates that shared
access integrity support is not provided across systems, and 4 indicates shared
access integrity is provided across systems.
With parameter value 3, all previously read buffers are assumed to be valid and
will not be refreshed from the physical catalog records. This gives good
performance. However, if the catalog is actually being shared, corruption or the
use of records that are no longer valid could occur. It should be selected only
when you can guarantee that the catalog does not reside on a physically shared
device.
A value of 4 specifies that the catalog can be fully shared, and the integrity of the
buffers and control blocks is maintained by ICF catalog management. Specifying
a value of 4 does not, in itself, ensure that a catalog is safely shared. It must also
reside on a disk volume that was defined as shared to every system. If there are
non-catalog management tasks using the catalog, damage can still result if they
are not coordinated and safely performed.

3.6.2 Determining if a catalog is actually shared

There is a very convenient way to verify that a catalog is shared. You can issue
the following command on all sharing systems:
MODIFY CATALOG,ALLOCATED

MODIFY can be abbreviated as F, which results in the command:

F CATALOG,ALLOCATED

38

ICF Catalog Backup and Recovery: A Practical Guide

The status display (Figure 3-9) shows you if a catalog is considered shared,
among other things. You need to check the fifth character position under FLAGS
for either E, indicating Enhanced Catalog Sharing (ECS) mode, or R,
indicating VVDS mode sharing. If neither of the sharing flags is shown, you can
be certain that the two conditions for sharing a catalog are not met. It may be that
on one of the sharing systems the disk volume is not defined as shared in HCD.
In this case, you see the R flag on the systems that meet the conditions, but not
on the systems that do not.

F CATALOG,ALLOCATED
IEC351I CATALOG ADDRESS SPACE MODIFY COMMAND ACTIVE
IEC348I ALLOCATED CATALOGS
*CAS***************************************************************
* FLAGS -VOLSER-USER-CATALOG NAME
*
* Y-I-R- SBOX01 0001 UCAT.CRPLUS2
*
* Y-I-R- SBOX01 0001 UCAT.CRPLUS1
*
* Y-IAR- SBOX11 0001 SYS1.VOLCAT.VGENERAL
*
* Y-I-R- SBOX11 0001 MCAT.SANDBOX.R10.VSBOX11
*
*******************************************************************
* Y/N-ALLOCATED TO CAS, S-SMS, V-VLF, I-ISC, C-CLOSED, D-DELETED, *
* R-SHARED, A-ATL, E-ECS SHARED, K-LOCKED
*
*CAS***************************************************************
IEC352I CATALOG ADDRESS SPACE MODIFY COMMAND COMPLETED

Figure 3-9 Catalog status display for allocated catalogs

In our case, we can see the R flag on all catalogs that are listed.
The E flag is only seen if ECS is active for that catalog. This means that the
conditions for shared catalogs are met on all systems in the sysplex. ECS is
thoroughly discussed in Enhanced Catalog Sharing and Management,
SG24-5594.
Figure 3-10 shows the display for one particular user catalog following various
actions with IDCAMS or the MODIFY command. Lines have been deleted from
the output. Note the absence of the R in the fifth character position under
FLAGS in the last display. This system no longer considers UCAT.CRPLUS1 to
be shared. The catalog is on a physically shared volume and catalog sharing
continues, but without integrity.

Chapter 3. Analyzing catalog integrity

39

****************** After IDCAMS ALTER LOCK ******************

F CATALOG,ALLOCATED
* FLAGS -VOLSER-USER-CATALOG NAME
* Y-I-RK SBOX01 0001 UCAT.CRPLUS1
********************* Closing the catalog *******************
F CATALOG,CLOSE(UCAT.CRPLUS1)
F CATALOG,ALLOCATED
* FLAGS -VOLSER-USER-CATALOG NAME
* Y-C-RK SBOX01 0001 UCAT.CRPLUS1
********** After IDCAMS ALTER SHAREOPTIONS to (3 3) *********
**** Reallocate catalog to CAS to refresh control blocks ****
F CATALOG,ALLOCATE(UCAT.CRPLUS1)
F CATALOG,ALLOCATED
* FLAGS -VOLSER-USER-CATALOG NAME
* Y-I--- SBOX01 0001 UCAT.CRPLUS1

Figure 3-10 Checking catalog attributes

3.6.3 Error symptoms due to sharing problems

A typical symptom of a catalog sharing problem is when catalog entries created
on one system cannot be found on another system that shares the same catalog.
Assume, as illustrated in Figure 3-11, that system A and system B each do a
locate that results in their retrieving the same control interval of the BCS (1 and
2). System A creates a new data set X and rewrites the CI to disk (3). Then
system B does a locate for the same data set X (4). If the catalog has
SHAREOPTIONS (3 3) and is not, therefore, flagged as shared, system B will
use the copy of the CI it already has in a buffer, not find the record, and fail
allocation.

40

ICF Catalog Backup and Recovery: A Practical Guide

SYSTEM A

BCS

CI

1. Create X
2.

ad
Re

SYSTEM B

W Y Z

CI

2. R
ead

SHAREOPTIONS (3 3)

1. Locate Z
CI

W Y Z

W Y Z

Buffer

Buffer

BCS
BCS

Write X
3.

W X Y Z

I
r ite C
R ew

4. Locate X
(Use same buffer)

W X Y Z

Not found at allocation

SHAREOPTIONS (3 3)

Buffer

5. Delete Y

BCS
7. Locate Y
(Use same buffer)

Found at allocation
Not found at open

W Z
W Z

te
6. Rewri

CI

Buffer

SHAREOPTIONS (3 3)

Figure 3-11 Results of improper sharing

To compound the situation, assume system B then deletes data set Y and
rewrites its version of the CI to disk (5 and 6), overlaying what system A just
wrote. The result would be that data set X exists, but the BCS no longer has an
entry for it. Imagine how messy it would be if the insert of a new record resulted
in a CI or CA split! Improper sharing such as this is the most common cause of
structurally damaged catalogs, where the index and data no longer match and
records are duplicated in different CIs.
Symptoms continue to appear. System A does a locate for the data set Y (7).
Since the catalog is not flagged as shared, system A uses the copy of the CI it
already has in a buffer, finds the record, and completes allocation. At open
processing, however, data set Y will not be found.
The conclusion to be drawn from this is to make sure that catalogs are defined
with SHAREOPTIONS (3 4) and reside on a volume defined as shared to every
system connected to it.

Chapter 3. Analyzing catalog integrity

41

3.7 CAS problems due to catalog errors

Catalog functions are performed in the catalog address space. When a user
requests a catalog function, a service task is assigned with a CAS ID. If you are
receiving error messages, requests are not being satisfied, tasks are abending,
or the catalog appears damaged, your diagnostics may lead you to the CAS
itself. The MVS MODIFY CATALOG command is your tool to communicate with
CAS. It is thoroughly documented in DFSMS Managing Catalogs, SC26-7409,
which should be your first source of information.

3.7.1 CAS ABEND or error messages

The most frequently issued catalog management error message is IDC3009I:
IDC3009I VSAM CATALOG RETURN CODE IS rtn - REASON CODE IS IGGOCLaa - rsn

There are almost 100 valid return codes, one of which has about 100 reason
codes associated with it! Although its identifier ends in I, indicating it is
informational rather than a warning or error message, it can report some very
serious conditions. It is frequently the message you are directed to by other error
messages.
Most occurrences of IDC3009I are related to individual catalog entries and do not
indicate that recovery of an entire BCS or VVDS is necessary. If you need
additional documentation to use for later problem source identification or to
provide to the support center, you can force CAS to create a dump. Use the
following command to create a dump whenever a given return code, reason
code, and module identifier are provided in the IDC3009I message:
MODIFY CATALOG,DUMPON(retcode,rsncode,modid)

For unexpected abends in CAS, a dump is always created. There are certain
forced (expected) abends in CAS, such as abend code 81A during CAS restart,
for which no dump is created. Using the F CATALOG,DUMPON command with
no additional operands specified creates a dump for expected abends and would
be an extreme action to take. It is unlikely you will need a CAS dump to resolve
your catalog problem.
Keep in mind that an abend in CAS does not mean CAS itself has abended. Like
a batch job may abend but JES and the initiator can go on to do other work, an
individual CAS task may abend, but CAS can go on to service other tasks.
Also remember there are in-storage control blocks for BCSs and VVDSs. Users
on one system may experience problems accessing data sets through a given
BCS or VVDS, but users on other systems may not and DIAGNOSE completes
with a return code zero. Simply refreshing the control blocks by closing the BCS

42

ICF Catalog Backup and Recovery: A Practical Guide

or VVDS with the appropriate MVS command may quickly solve your problems.
Try the CLOSE commands before using UNALLOCATE. In any case, the BCS or
VVDS is opened by the next request that accesses it, and the control blocks are
rebuilt at that time.
MODIFY
MODIFY
MODIFY
MODIFY

CATALOG,CLOSE(catname)
CATALOG,UNALLOCATE(catname)
CATALOG,VCLOSE(volser)
CATALOG,VUNALLOCATE

3.7.2 CAS hang or lockout situations

If a user address space is waiting and CAS does not respond due to a hang or
lockout situation, you can try different MODIFY CATALOG commands to attempt
to alleviate the situation.
Use the MODIFY CATALOG,LIST command for a list of catalog tasks and their
status. Each task has a unique ID. Figure 3-12 shows sample output from this
command. Two TSO users, YCJRES1 and YCJRES3, are waiting for a catalog.

F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -W-E-007AF090
YCJRES3 / IKJACCT
00.02.12
01 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.02.12
*
* -W-E-00784288
YCJRES1 / IKJACCT
00.01.36
02 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.01.36
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-12 Catalog task display

With the information from the CAS task and status display, you might consider
canceling one of the jobs that is holding the ENQ. To clear this condition, for one
of the tasks, issue the command:
F CATALOG,END(id)

If that does not end the task, issue the command:

F CATALOG,ABEND(id)

In most cases, this solves the hang or lockout situation.

Chapter 3. Analyzing catalog integrity

43

The F CATALOG,RESTART command should be used only when your only other
option is to IPL the system. CAS is designed to be restarted with minimal impact
on your system. Any catalog requests in process at the time are redriven.
However, in the unlikely event that restart fails, it will be necessary to IPL the
system to recover.

3.7.3 Working with the catalog address space

Figure 3-13 through Figure 3-20 show selected SYSLOG messages that
illustrate using the MVS MODIFY command to list tasks, attempt to end a task,
and restart CAS. We created the hang situation by canceling TSO user IDs
whose data sets are in the same catalog. Prior to the point in time where this
segment of SYSLOG begins, we forced off the user IDs several times. The log
begins after user IDs YCJRES1 and YCJRES3 attempt to logon again and hang.
1. The F CATALOG,LIST command shows user IDs YCJRES1 and YCJRES3
waiting for a BCS ENQ (Figure 3-13).

F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -W-E-007AF090
YCJRES3 / IKJACCT
00.02.12
01 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.02.12
*
* -W-E-00784288
YCJRES1 / IKJACCT
00.01.36
02 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.01.36
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-13 Step 1: TSO user IDs waiting for BCS ENQ

2. We cancel user YCJRES1, who tries to log on again. The F CATALOG,LIST

command in Figure 3-14 shows user ID YCJRES1 is hung again. Note the
different task address: 007827C0 versus 00784288 in step 1. The task ID for
user ID YCJRES1 waiting is 02.

44

ICF Catalog Backup and Recovery: A Practical Guide

C U=YCJRES1
F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -W-E-007AF090
YCJRES3 / IKJACCT
00.03.48
01 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.03.48
*
* -W-E-007827C0
YCJRES1 / IKJACCT
00.00.50
02 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.00.50
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-14 Step 2: TSO user ID cancel

3. In Figure 3-15, we attempt to end and redrive the user IDs YCJRES1s
catalog request by issuing the F CATALOG,END(02) command. A subsequent F
CATALOG,LIST command shows that the previous command did not work. Task
02 is still waiting for a BCS ENQ.

F CATALOG,END(02)
F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -W-E-007AF090
YCJRES3 / IKJACCT
00.06.04
01 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.06.04
*
* -W-E-007827C0
YCJRES1 / IKJACCT
00.00.44
02 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.00.44
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-15 Step 3: End one catalog request

4. We cancel users YCJRES1 and YCJRES3 by using the CANCEL commands.

The F CATALOG,LIST command in Figure 3-16 shows that the CANCEL
command ended the CAS task for YCJRES3, but not for YCJRES1.

Chapter 3. Analyzing catalog integrity

45

C U=YCJRES1
C U=YCJRES3
F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -W-E-007827C0
YCJRES1 / IKJACCT
00.02.51
02 *
* WAITING FOR BCS ENQ Shr
FROM 0FC172E0 FOR 00.02.51
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-16 Step 4: TSO user IDs cancelled

5. We attempt to abend YCJRES1s catalog request, identified with task ID 02.

Figure 3-17 shows that the F CATALOG,ABEND(02) command worked, and CAS
now shows no active tasks.

F CATALOG,ABEND(02)
F CATALOG,LIST
IEC347I LIST CATALOG TASK(S)
*CAS****************************************************************
* FLAGS - TASK ADDRESS - JOBNAME / STEPNAME - ELAPSED TIME - ID *
* -----NOACTIVE / NONE
00.00.00
*
********************************************************************
* O-OLDEST, W-WAIT, A-ABEND, E-ENQ, R-RECALL, L-RLS
*
*CAS****************************************************************

Figure 3-17 Step 5: Catalog request abended

6. User YCJRES1 attempts to logon and hangs again. In Figure 3-18, a display
of GRS data shows two tasks in the CATALOG address space that do not
appear in Figure 3-17.

46

ICF Catalog Backup and Recovery: A Practical Guide

D GRS,C
ISG343I 20.24.49 GRS STATUS
S=SYSTEMS SYSIGGV2 UCAT.CRPLUS1
SYSNAME
JOBNAME
ASID
TCBADDR
EXC/SHR
SC63
CATALOG / *UNKNOW 002F/0000 007AF700 EXCLUSIVE
SC63
CATALOG / *UNKNOW 002F/0000 00782640
SHARE
NO REQUESTS PENDING FOR ISGLOCK STRUCTURE
NO LATCH CONTENTION EXISTS

STATUS
OWN
WAIT

Figure 3-18 Step 6: Display GRS information

7. In Figure 3-19, we try to restart CAS.

F CATALOG,RESTART
IEF450I IEESYSAS CATALOG - ABEND=S81A U0000 REASON=00000005
IEC355I IDACAT13, CATALOG ADDRESS SPACE IS RESTARTING
IEC357I IDACAT13, CATALOG ADDRESS SPACE RESTARTED
IEF403I IEESYSAS - STARTED - ASID=0050.
IEC350I CATALOG ADDRESS SPACE MODIFY COMMAND AVAILABLE
IEC377I ENHANCED CATALOG SHARING: CONNECT COMPLETE
IXL014I IXLCONN REQUEST FOR STRUCTURE SYSIGGCAS_ECS 660
WAS SUCCESSFUL. JOBNAME: CATALOG ASID: 0050
CONNECTOR NAME: IXCLO00E0001 CFNAME: CF01
IEC377I ENHANCED CATALOG SHARING: AUTOADD IS NOT CURRENTLY ENABLED
IEC368I - CATALOG INITIALIZATION 663
THE MULTILEVEL ALIAS FACILITY HAS BEEN INITIALIZED
THE NUMBER OF LEVELS OF QUALIFICATION IS 1
IEC378I UCAT.VSBOX01 IS NOT USING ENHANCED CATALOG SHARING

Figure 3-19 Step 7: Restart CAS

8. CAS was successfully restarted. The display of GRS data in Figure 3-20
shows the catalog ENQ resource contention is cleared. After that, TSO users
YCJRES1 and YCJRES3 could logon.

D GRS,C
ISG343I 20.27.25 GRS STATUS
NO ENQ RESOURCE CONTENTION EXISTS
NO REQUESTS PENDING FOR ISGLOCK STRUCTURE
NO LATCH CONTENTION EXISTS

Figure 3-20 Step 8: No contention on CAS

Chapter 3. Analyzing catalog integrity

47

48

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 4.

Recovering catalogs
This chapter discusses the importance of an effective recovery strategy for
catalogs in your environment. It also describes the techniques and utilities that
are available for performing a successful recovery.
The sections in this chapter cover:

Recovery strategy
Recovery utilities
Considerations for the recovery process
Forward recovery

Copyright IBM Corp. 1999, 2001

49

4.1 Recovery strategy

If a catalog is broken due to a structural error, or is not accessible because of a
hardware or media problem, it is likely that the catalog needs to be recovered. To
reduce the outage and its subsequent impact, it is essential to implement a
recovery methodology that reduces any down time to a minimum. To help you
accomplish that, the following sections describe considerations relative to the
recovery process.
Regardless of the techniques you employ, be sure to test your procedures before
you need them. Collecting SMF records and having a product, such as IBM
Integrated Catalog Forward Recovery Utility (ICFRU) or Mainstar Catalog
RecoveryPlus, installed are far from adequate preparation if you do not know
how to use them. Rehearse!

4.1.1 Determining when to recover a catalog

If a catalog is not accessible, or if a catalog error cannot be repaired by
recovering the damaged records using IDCAMS DELETE and DEFINE
commands, it is likely that a full recovery of the BCS will be necessary. Part of the
error diagnosis process is to determine the extent of damage and whether the
entire structure or only some entries need to be recovered. You do not want to
recover more than is necessary.
Be sure to remember the in-storage pointers. A control block in the CAS is built
for each catalog and VVDS as it is opened. Keep this in mind if you find attempts
to access a particular BCS or VVDS are resulting in recurrent errors. Recovery
may be as simple as closing and reopening the BCS or VVDS involved.

4.1.2 Recover, restore, repair, or salvage?

Once you decide what is broken and needs to be fixed, you need to determine
what degree of recovery is acceptable and what can be done. Recovery up to the
point of failure is the goal. The ability to do that implies you have a good backup
copy, a complete record of changes that were made since it was taken, and a
procedure for using them.
You may find that you can only restore a backup copy and have no
straightforward way to recover it to the point of failure. Or you may find that you
do not have a recent enough good backup copy and must repair or salvage the
damaged catalog. Repair implies ZAP and time consuming bit-level diagnostics.
In this time of object code only (OCO), you may not even have the documentation

50

ICF Catalog Backup and Recovery: A Practical Guide

you need to do it. The ZAP facility of Catalog RecoveryPlus, discussed in 6.9,
ZAP command on page 107, and the ad hoc tool VVDSFIX, discussed in 4.3.4,
VVDS recovery on page 57, can help you when you need to attempt to repair a
BCS or VVDS.
That leaves you with salvage. IDCAMS REPRO MERGECAT can be used to
move entries out of a broken catalog. The potentially lengthy and tedious process
is discussed in 4.2.4, Using REPRO MERGECAT to salvage a catalog on
page 54.

4.1.3 Determining the most recent backup version

Many installations define generation data groups for their backup data sets so
the latest one is always generation zero. If you are using EXPORT/IMPORT
commands, you can use SMF records to determine your most recent backup
version. For each successful export of a catalog, an SMF type 36 record is
created. If your backup was done by DFSMShsm, version numbers are available
to keep track of the most recent backup version.
The catalog containing the entry for your backup data set must be usable. As
discussed in 2.3, Considerations for the backup process on page 17, making
multiple backup copies cataloged in different catalogs can help ensure your
ability to access one when needed. If the backup was written to tape, the tape
management system may be used to locate it if you know the data set name.

4.1.4 Recovery techniques

The technique used to recover a catalog depends on the technique used for
backup. The relationship between backup and recovery utilities is shown in
Table 4-1.
Table 4-1 Backup and recovery utilities relationship
Backup utility

Recover utility

Alias handling

IDCAMS EXPORT

IDCAMS IMPORT

Aliases are defined if ALIAS is

specified with IMPORT

DFSMSdss logical dump

DFSMSdss logical restore

Aliases are defined

DFSMSdss full volume dump

DFSMSdss full volume or

physical data set restore

No alias handling

DFSMShsm BACKDS

DFSMShsm RECOVER

Aliases are defined

Regardless of the technique used for backup and recovery, you need to check for
changes to the BCS that are not reflected in the restored copy, as well as alias
changes that were made after the backup.

Chapter 4. Recovering catalogs

51

4.1.5 Preventing catalog activity during restore

We strongly recommend that access to the catalog is restricted during any
recovery activity by using IDCAMS ALTER LOCK. Figure 4-1 shows a sample job
to lock an existing catalog. To use the LOCK and UNLOCK parameters for
catalogs, you must have RACF profile IGG.CATLOCK defined. In addition, you
must have READ authority to the profile IGG.CATLOCK and ALTER authority to
the catalog being locked, not only to lock it but also to access it while it is locked.

//STEP1
EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
ALTER UCAT.CRPLUS2 LOCK

Figure 4-1 Sample JCL to lock a catalog

The LOCK parameter is also available on the DEFINE UCAT and IMPORT
commands. Once you complete forward recovery and are ready to allow general
access, unlock the catalog with the ALTER UNLOCK command. The catalog can
be locked/unlocked from any one system and does not have to be unlocked from
the same system that locked it.
The lock attribute is recognized by all systems and eliminates the need to
quiesce applications, vary volumes offline to sharing systems, or terminate users
during catalog recovery. While the catalog is locked, unauthorized requests to
access it simply fail with a return code indicating the catalog is temporarily
unavailable.

4.2 Recovery utilities

The first step in recovering the catalog is to restore it. You may explicitly delete it
before running your restore job. The command you use is:
DELETE bcsname USERCATALOG RECOVERY

This results in the BCS being deleted, as well as the connector record and
aliases for it in the master catalog on the system where the command is
executed. Other master catalogs are not affected. The DELETE UCAT
RECOVERY command is like a big DELETE NOSCRATCH. The entire catalog is
deleted, but the data sets are not touched. Any VSAM or system-managed
non-VSAM data sets are simply temporarily inaccessible.

52

ICF Catalog Backup and Recovery: A Practical Guide

If you are using IDCAMS IMPORT, DFSMSdss RESTORE, or DFSMShsm

RECOVER, you do not need to delete and redefine the catalog unless you want
to change its size or other attributes such as CISIZE or FREESPACE. If a BCS
acquired a large number of extents, you may want to delete it and redefine it with
a larger primary space allocation. If you do not need to define the catalog
yourself, using specific attributes, any of these utilities can delete and redefine it
with the attributes of the backup copy.

4.2.1 IDCAMS IMPORT command

After verifying that there is an EXPORTed version, the IDCAMS IMPORT
command deletes the existing catalog, redefines it with the attributes of the
EXPORTed copy, and loads it. The recovered catalog is reorganized. If you
pre-allocated the catalog, you must specify the INTOEMPTY keyword. You do not
have to restore the catalog to the source volume.
UAliases are backed up as part of EXPORT processing. However, they are only
restored if you specify the ALIAS keyword on the IMPORT command. If you did
not delete the catalog, all existing aliases are maintained and any aliases in the
backup copy are redefined.

4.2.2 DFSMSdss logical restore

With DFSMSdss RESTORE, you must specify the fully qualified name to restore
a catalog. Backup copies of catalogs created by DFSMSdss can only be restored
to a volume with the same VOLSER as the source volume. DFSMSdss does not
reorganize the catalog during logical restore. Figure 4-2 shows a sample JCL to
restore the most recent backup copy of UCAT.CRPLUS2 from a GDS.

//STEP1
EXEC PGM=ADRDSSU
//SYSPRINT DD SYSOUT=*
//INDD
DD DSN=CRPLUS2.BACKUP(0),DISP=SHR
//OUTDD
DD UNIT=SYSDA,VOL=SER=SBOX01,DISP=SHR
//SYSIN
DD *
RESTORE DATASET(INCLUDE(UCAT.CRPLUS2)) INDDNAME(INDD) OUTDDNAME(OUTDD)

Figure 4-2 Sample JCL for DFSMSdss RESTORE of a BCS

DFSMSdss logical restore handles aliases differently depending on whether the

catalog is pre-allocated. If the catalog to be restored is pre-allocated but not
empty, aliases are not restored.
Note: A catalog is empty when it contains only its self-describing record.

Chapter 4. Recovering catalogs

53

If the catalog to be restored has been pre-allocated, you must specify the
REPLACE keyword on the DFSMSdss RESTORE job.

4.2.3 DFSMShsm recovery

DFSMShsm RECOVER can restore the catalog to any volume, and the catalog is
reorganized in the process. Aliases preserved in the backup copy are defined if
they do not already exist.
The DFSMShsm RECOVER command can also be used to move a catalog to a
different volume with a different volume serial number. The TOVOLUME keyword
is available to specify the target non-system-managed volume. For
system-managed catalogs, the TOVOLUME keyword is only honored if a storage
class with the GUARANTEED SPACE attribute is assigned. Refer to DFSMShsm
Storage Administration Reference, SC35-0422, and DFSMShsm Storage
Administration Guide, SC35-0421, for more detailed information.

4.2.4 Using REPRO MERGECAT to salvage a catalog

It is possible that structural damage in a catalog makes it impossible for you to
recover with your usual procedures. It may be that you discover your backup copy
is unusable or that the last good backup copy is so old that forward recovery
would be impossible. You can use the following steps to salvage as many entries
as possible, reducing the number that may have to be individually recovered.
1. Define the target catalog using IDCAMS command:
DEFINE USERCATALOG

2. Copy the contents of the source catalog into the target catalog using the
command:
REPRO MERGECAT INDATASET(BCS.source) OUTDATASET(BCS.target)

The job may fail when it encounters the structural error.

3. Move entries by high-level qualifier or qualifiers as long as possible using the
command:
REPRO MERGECAT IDS(BCS.source) ODS(BCS.target) LEVEL(hlq)

4. Move individual entries using the command:

REPRO MERGECAT IDS(BCS.source) ODS(BCS.target) ENTRIES(dsn)

5. Delete the original catalog using the command:

DELETE BCS.source UCAT RECOVERY

6. Redefine the original catalog using the command:

DEFINE UCAT (NAME(BCS.source).....)

54

ICF Catalog Backup and Recovery: A Practical Guide

7. Copy the contents of the target catalog to the new source catalog using the
command:
REPRO MERGECAT IDS(BCS.target) ODS(BCS.source)

The process may take a great deal of time and should only be considered when
recovery is not possible.
See Chapter 6, Using Catalog RecoveryPlus on page 69, for information on
how that product can help you salvage a BCS.

4.3 Considerations for the recovery process

You do not need to know what caused a problem in order to recover from it.
However, it is important to do further analysis of the failure to try to prevent future
occurrences. Attempt to make a copy of the damaged structure and be sure to
collect documentation before you delete it. That documentation may include:

SYSLOG output
JOBLOG output
LOGREC output
Dumps
IEHLIST LISTVTOC output
DIAGNOSE and EXAMINE output
Print of BCS or VVDS records
Print of an entire BCS or VVDS

These items may be necessary for you or IBM support to determine what
happened.
Although the basic structure of the master, user, and tape volume catalogs is the
same, their requirements for recovery differ enough to make it worthwhile to
discuss them separately.

4.3.1 Master catalogs

Recovery of a master catalog must be performed from a different system. During
the time that the catalog is being used as a master catalog, no recovery can be
performed. For this reason, recovery of a master catalog always results in a
system outage. If the master catalog is shared in a sysplex, this means an
outage for the whole sysplex. If there is no other system available to perform the
recovery of the master catalog, a DFSMSdss stand-alone restore of the volume
containing the master catalog may be the only way to recover.

Chapter 4. Recovering catalogs

55

As long as a catalog is used as a master catalog it cannot be locked. While a

catalog is locked, any unauthorized requests to access the catalog will fail. Only
users who have READ access authority to RACF profile IGG.CATLOCK can
access a locked catalog.
When an alternate master catalog is available, an IPL can be performed using
the alternate master catalog.
To recover a broken master catalog, you need to submit the restore job from a
different system to recover the master catalog from the portable backup copy.
If you do not have a usable portable backup copy of your master catalog
available, you must either perform a full volume recovery from a full volume
backup of the master catalogs volume, or you can perform a physical restore of
the tracks containing the master catalog. You need to check for recent changes
that are not reflected in the backup copy that was just restored.
Note: To minimize update activity to the master catalog, and to reduce the
exposure to breakage, we recommend that only SYS1 data sets, user catalog
connector records, and the aliases pointing to those connectors be in the
master catalog.

4.3.2 User catalogs

The loss of a user catalog results in the loss of access to all data sets cataloged
in it. Therefore, the availability requirements of user catalogs are as high as the
applications cataloged within it. The active nature of most user catalogs also
means that a forward recovery to as near the point-of-failure as possible should
be attempted.

4.3.3 Tape volume catalogs

VOLCATs are ICF catalogs but should be thought of more in conjunction with the
tape library function than with other catalogs. IDCAMS cannot change the library
manager inventory in an automated tape library. ISMF should be used for normal
functions. The IDCAMS commands to create, alter, or delete library and volume
records should be used only in a recovery situation and with caution.
See DFSMS OAM Planning, Installation and Storage Adminstration Guide for
Tape Libraries, SC35-0427, and DFSMSrmm Implementation and Customization
Guide, SC26-7405, for comprehensive information on VOLCATs and tape
libraries.

56

ICF Catalog Backup and Recovery: A Practical Guide

4.3.4 VVDS recovery

Traditionally the VVDS was not recovered but rebuilt by recovering all the data
sets it describes. You cannot use IDCAMS EXPORT, IMPORT, or ALTER
commands on a VVDS. After relabeling or clipping a volume, the VVDS no
longer has the correct name and all VSAM or system-managed non-VSAM data
sets defined in it are inaccessible. Seemingly minor damage in it can make data
sets or an entire volume inaccessible.
In response to the need for a way to recover VVDS entries when the integrity of
the data was not in question, an ad-hoc tool to check, read, and modify the
VVDSs was developed. VVDSFIX can be downloaded from the Download
Catalog Utilities section of the Catalog and VSAM Knowledge Base at:
http://knowledge.storage.ibm.com/vsam/vsaminfo/downloadutilitys.shtml
An example of using VVDSFIX tool can be found in 7.8, Case study #8: Print
and delete BCS/VVDS records on page 138.
With Catalog RecoveryPlus BACKUP/RECOVER commands, it is possible to
restore and perform forward recovery on a VVDS as well as a BCS. See
Chapter 6, Using Catalog RecoveryPlus on page 69, and 7.4, Case study #4:
VVDS backup and forward recovery on page 125.

4.3.5 IMBED and REPLICATE

The use of IMBED and REPLICATE is no longer valid if you are using
DFSMS/MVS 1.5 or later. These keywords are ignored if specified when defining
a new catalog. For compatibility reasons, IMBED and REPLICATE are still
supported for IMPORT and DFSMSdss RESTORE. However, we suggest that a
catalog be pre-allocated that has no imbedded index to simplify any recovery
action. This means that you need to specify the INTOEMPTY keyword for an
IDCAMS IMPORT process or the REPLACE keyword for a DFSMSdss logical
restore.

4.4 Forward recovery

The version of the catalog that you just restored reflects the status of the catalog
at the time the backup copy was taken. If you are not using ICFRU, Catalog
RecoveryPlus, or other vendor utilities, you must manually check for changes to
the catalog that are not reflected in the restored copy.

Chapter 4. Recovering catalogs

57

4.4.1 Manually checking for missing updates

To check for changes that are not reflected, you can use the IDCAMS
DIAGNOSE command to compare the VVDSs to the BCS and vice versa. See
3.5, Synchronization errors on page 30. You need to list VVDSs connected to
the catalog to find out which VVDSs you need to compare against the BCS. This
only works for VSAM and system-managed data sets because
non-system-managed and tape data sets are not reflected in a VVDS. Issuing
the IDCAMS LISTCAT NONVSAM command before and after the recovery may
help to find changes for non-system-managed and tape data sets.

4.4.2 SMF data

A more efficient way to check for changes since the last backup was taken is to
collect and check SMF records generated by catalog management. You must
make sure that you are collecting SMF record types 60, 61, 65, and 66 for all
jobs, including subsystems and started tasks, on all systems with access to the
catalog (Table 4-2). Check the contents of member SMFPRMxx of
SYS1.PARMLIB to make sure these record types are not being suppressed.
Table 4-2 SMF record types for ICF
Record type

Event recorded

60 (X3C)

VVR or NVR inserted, updated or deleted

61 (X3D)

BCS entry defined

65 (X41)

BCS entry deleted

66 (X42)

BCS entry altered

4.4.3 Using DFSORT in the checking process

You can use the IBM product DFSORT to filter out the SMF records for a specific
catalog corresponding to the time frame since the last backup copy was made.
Figure 4-3 shows an example of DFSORT control statements to collect only
catalog type 61 records. Similar steps are required to collect the type 65 and 66
records.

58

ICF Catalog Backup and Recovery: A Practical Guide

//SYSIN DD *
OPTION VLSHRT
RECORD TYPE=V
INCLUDE COND=((6,1,BI,EQ,X'3D'),AND,
(11,4,BI,GT,X'0101269F'),AND,
(76,44,CH,EQ,C'UCAT.CRPLUS1'))
SORT FIELDS=(11,4,BI,A,7,4,BI,A)

Figure 4-3 DFSORT statements to extract SMF record type 61

Note the following points:

The first comparison checks for SMF record type 61 (X'3D').
The second comparison checks records created after day 2001269.
The third comparison checks for a 44-character catalog name
UCAT.CRPLUS1.
The SORT FIELDS statement sorts the records by date and time in
ascending order.

Figure 4-4 shows an example of DFSORT control statements to collect all BCS
related record types (types 61, 65, and 66).

//SYSIN
DD *
OPTION VLSHRT
RECORD TYPE=V
INCLUDE COND=(((6,1,BI,EQ,X'3D'),OR,
(6,1,BI,EQ,X'41'),OR,
(6,1,BI,EQ,X'42')),AND,
(11,4,BI,GT,X'0101269F'),AND,
(76,44,CH,EQ,C'UCAT.CRPLUS1'))
SORT FIELDS=(11,4,BI,A,7,4,BI,A)

Figure 4-4 DFSORT statements to extract SMF record types 61, 65, and 66

Note the following points:

The first comparison checks for SMF record type 61 (X'3D') or 65 (X'41') or
66 (X'42').
The remaining fields are the same as in Figure 4-3.

SMF records are also written as VVDS entries are inserted, deleted, or altered.
Figure 4-5 shows the control statements to select type 60 records.

Chapter 4. Recovering catalogs

59

//SYSIN DD *
OPTION VLSHRT
RECORD TYPE=V
INCLUDE COND=((6,1,BI,EQ,X'3C'),AND,
(11,4,BI,GT,X'0101269F'),AND,
(76,44,CH,EQ,C'SYS1.VVDS.VSBOX03'))
SORT FIELDS=(11,4,BI,A,7,4,BI,A)

Figure 4-5 DFSORT statements to extract SMF record type 60

Note the following points:

The first comparison checks for SMF record type 60 (X'3C').
The second comparison checks records created after day 2001269.
The third comparison checks for a 44-character VVDS name
SYS1.VVDS.VSBOX03.

For more information with respect to the format and contents of the SMF records,
see MVS System Management Facilities (SMF), SA22-7630. For additional
information on DFSORT parameters, see DFSORT R14 Application
Programming Guide, SC33-4035.

60

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 5.

Using ICFRU
This chapter explains how to use the IBM Integrated Catalog Forward Recovery
Utility (ICFRU), an ICF catalog recovery tool. With ICFRU, you can forward
recover ICF catalogs using selected SMF records.
ICFRU requires an error-free copy of a backup of the ICF catalog being
recovered. The backup must be created by the IDCAMS EXPORT command.

Copyright IBM Corp. 1999, 2001

61

5.1 ICFRU basics

Integrated Catalog Forward Recovery Utility is a catalog recovery tool that helps
you recover a damaged catalog to a correct and current status. All types of
catalog entries that may exist in the BCS are supported, including those for
VSAM, non-VSAM, and generation data group (GDGs). A catalog to be
recovered may be shared by multiple systems. A master catalog may be
recovered, provided it is not in use as a master catalog.
An error-free backup of an undamaged BCS is used as the base for recovery.
SMF records keep the catalog changes and are selected and applied to this
backup to create a new image of the catalog as it should now exist. This image of
the catalog can be then reloaded to produce a current BCS.

5.2 Recovering an ICF catalog using ICFRU

ICFRU needs a backup copy of the BCS, created by an IDCAMS EXPORT
command, known as the portable backup copy as its base file. If this portable
copy is not available, then ICFRU cannot be used.
To forward recover an ICF catalog, you also require a file that contains the SMF
record types 61, 65, and 66 from all systems to which the ICF catalog being
recovered is connected. These SMF records contain the information needed to
update the ICF catalog to the point of failure. ICFRU uses these records to create
a new file that contains the updated catalog records. This new file can then be
used to create an error free copy of the ICF catalog using the IDCAMS IMPORT
command.
ICFRU uses two programs to create an error free file that can be used to restore
the ICF catalog to the point of failure. The first program, the Integrated Catalog
Forward Recovery Record Selection and Validation (ICFRRSV), extracts the
SMF records that contain the information that is required to create an updated
version of the ICF catalog being recovered. The SMF records are selected based
on the name of the ICF catalog being recovered, the start date and time that is
passed to the program, along with the end date and time. The selected SMF
records are sorted, and this sorted file is used as input to the second program.
The second program, the Integrated Catalog Forward Recovery Record Analysis
and Processing (ICFRRAP), uses the sorted SMF file, along with the backup file
that was created by the IDCAMS EXPORT command, to create a new file
containing the updated ICF catalog records.

62

ICF Catalog Backup and Recovery: A Practical Guide

Figure 5-1 shows the ICFRU forward recovery process.

SMF DUMP DATA SETS
SC63

SC64

SC65

REPORTS

SMFIN

(SYSPRINT)

ICFRRSV

PARAMETERS

MESSAGES

SMFOUT

(SYSLOG)

SELECTED SMF
CATALOG
RECORDS
SORTIN

SYSIN

SYSPRINT

SORT
SORTOUT

OLD CATALOG
EXPORTED COPY

SORTED SMF
CATALOG
RECORDS

EXPIN

SMFIN

REPORTS
(SYSPRINT)

PARAMETERS

ICFRRAP
MESSAGES

EXPOUT

(SYSLOG)

NEW SORTED
EXPORT COPY
INFILE

SYSIN

IDCAMS
IMPORT

SYSPRINT

OUTDATASET
RECOVERED
CATALOG

Figure 5-1 Integrated Catalog Forward Recovery Utility logic flow

To create an error free copy of the ICF catalog, use the file that was created by
ICFRU as input to the IDCAMS IMPORT command. This new version of the ICF
catalog is now error free.

Chapter 5. Using ICFRU

63

ICFRU does not provide any mechanism to lock an ICF catalog. To lock an ICF
catalog prior to commencing the recovery, use the IDCAMS ALTER command,
specifying the lock parameter. This ensures that no updates take place to the ICF
catalog during the recovery process.
Figure 5-2 and Figure 5-3 show an example of the JCL that is required to run
ICFRU to forward recover an ICF user or master catalog.

/YCJRES1Z JOB MSGCLASS=X,NOTIFY=&SYSUID,RESTART=*

//SET0
SET ICFCAT='UCAT.CRPLUS2'
//SET1
SET SYSNAME1=SC63
//SET2
SET SYSNAME2=SC64
//SET3
SET SYSNAME3=SC65
//SET4
SET STARTDTE='09/21/01'
//SET5
SET STARTIME='20:16:47'
//SET6
SET ENDDATE='09/21/01'
//SET7
SET ENDTIME='20:30:00'
//SET8
SET GAPTIME=30
//SET9
SET CLOCKDIF=0
//S1
EXEC PGM=ICFRRSV,
// PARM=('&ICFCAT.,&STARTDTE.,&STARTIME.,&ENDDATE.,&ENDTIME.,&GAPTIME.',
//
'&CLOCKDIF.')
//SYSPRINT DD SYSOUT=*
//SYSLOG
DD SYSOUT=*
//SMFIN
DD DSN=YCJRES1.&SYSNAME1..SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.&SYSNAME2..SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.&SYSNAME3..SMFDATA,DISP=SHR
//SMFOUT
DD DSN=YCJRES1.SMFCAT.DATA,DISP=(,CATLG,DELETE),
//
SPACE=(CYL,(10,10),RLSE),UNIT=SYSALLDA
//
IF (S1.RC = 0) THEN
//S2 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD DSN=YCJRES1.SMFCAT.DATA,DISP=(OLD,DELETE)
//SORTOUT DD DSN=YCJRES1.SMFCAT.DATA.SORTED,DISP=(,CATLG,DELETE),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
//SYSIN DD *
OPTION DYNALLOC=SYSDA,FILSZ=E10000
SORT FIELDS=(218,44,CH,A,262,1,BI,A,11,4,PD,D,7,4,BI,D)
/*
//
ENDIF

Figure 5-2 Sample JCL for running ICFRU (Part 1 of 2)

64

ICF Catalog Backup and Recovery: A Practical Guide

//
IF (S2.RC = 0) THEN
//S3 EXEC PGM=ICFRRAP,
// PARM=('&ICFCAT.,&STARTDTE.,&STARTIME.,&ENDDATE.,&ENDTIME.,&GAPTIME.',
//
'&CLOCKDIF.')
//SYSPRINT DD SYSOUT=*
//SYSLOG
DD SYSOUT=*
//SMFIN
DD DSN=YCJRES1.SMFCAT.DATA.SORTED,DISP=(OLD,DELETE)
//EXPIN
DD DSN=YCJRES1.CRPLUS1.EXPORT.BACKUP(0),DISP=SHR
//EXPOUT
DD DSN=YCJRES1.CRPLUS1.EXPORT.BACKUP(+1),
//
DISP=(,CATLG,DELETE),UNIT=SYSALLDA,
//
SPACE=(CYL,(10,10),RLSE)
//
ENDIF

Figure 5-3 Sample JCL for running ICFRU (Part 2 of 2)

The catalog to recover is UCAT.CRPLUS2. The result of the ICFRU program is

the file YCJRES1.CRPLUS1.EXPORT.BACKUP(+1).
Figure 5-4 shows the output report from ICFRRSV. You can see that the SMF
record selection start date was 09/21/01, with a commencing time of 14:24:26,
and an end date of 09/21/01 and time of 15:15:00.

INTEGRATED CATALOG FORWARD RECOVERY UTILITY R1M0

ICFRRSV SYSPRINT
09/21/01 (01.264) 17:04:05
RECORD SELECTION AND VALIDATION REPORT
EXECUTION PARAMETERS
CATALOG NAME
UCAT.CRPLUS2
RECORD SELECTION START
09/21/01 (01.264) 14:24:26
RECORD SELECTION STOP
09/21/01 (01.264) 15:15:00

PAGE 01

REPORT FOR ALL SYSTEMS

RECORD SELECTION AND VALIDATION CONDITION CODE IS 0
274 RECORDS SELECTED FOR UCAT.CRPLUS2
273 DEFINE (TYPE 61) RECORDS SELECTED
0 DELETE (TYPE 65) RECORDS SELECTED
1 ALTER (TYPE 66) RECORDS SELECTED
RECORD SELECTION AND VALIDATION CONDITION CODE IS 0

Figure 5-4 ICFRU ICFRRSV output report

Figure 5-5 shows the output report from ICFRRAP. You can see that 272 records
were selected from the SMF data and used in the creation of the new export data
set.

Chapter 5. Using ICFRU

65

ICFRRAP SYSPRINT
7,868 TOTAL

7,596

272
7,598 TOTAL
7,596

2
0

274 TOTAL
272
2
0

7,868 TOTAL
4 TOTAL
7,872 TOTAL

09/21/01 (01.264) 17:04:07

PAGE 01
REPORT OF RECORDS BY DATA SET
RECORDS IN THE NEW EXPORT DATA SET (EXPOUT)
10 CONTROL RECORDS
7,858 CATALOG RECORDS
RECORDS FORWARDED FROM THE OLD EXPORT DATA SET (EXPIN)
10 CONTROL RECORDS
7,586 CATALOG RECORDS
CATALOG RECORDS SELECTED FROM THE SMF DATA SET (SMFIN)
RECORDS FROM THE OLD EXPORT DATA SET (EXPIN)
RECORDS CARRIED FORWARD TO THE NEW EXPORT DATA SET
10 CONTROL RECORDS
7,586 CATALOG RECORDS
RECORDS SUPERSEDED OR DELETED (BASED ON SMF DATA)
RECORDS REJECTED BECAUSE OF ERRORS
0 INVALID LENGTH
0 UNRECOGNIZED CATALOG RECORD TYPE
RECORDS FROM THE SMF DATA SET (SMFIN)
RECORDS CARRIED FORWARD TO THE NEW EXPORT DATA SET
RECORDS SUPERSEDED OR DELETED BY NEWER SMF RECORDS
RECORDS REJECTED
0 NOT AN MVS SMF RECORD
0 NOT AN SMF CATALOG RECORD
0 NOT AN SMF CATALOG RECORD FOR THIS CATALOG
0 DATE/TIME EARLIER THAN EFFECTIVE START TIME
0 DATE/TIME LATER THAN EFFECTIVE STOP TIME
OF ALL OUTPUT RECORDS
OF ALL RECORDS DISCARDED
OF ALL INPUT RECORDS

Figure 5-5 ICFRU ICFRRAP output report

5.3 Post recovery actions

When you complete the recovery of the ICF catalog using ICFRU, you should
complete the following tasks:
1. List the recovered ICF catalogs self describing record using the IDCAMS
LISTCAT command. The format of the LISTCAT command is:
LISTCAT ENT(UCAT.CRPLUS2) ALL CAT(UCAT.CRPLUS2)

If the job completes with a condition code of zero (0), it means that you
successfully read a catalog record from the recovered ICF catalog.
2. Unlock the ICF catalog using the IDCAMS ALTER command. If you do not
unlock the recovered ICF catalog, all users and batch jobs are denied access
to any file that is cataloged in it. The format of the ALTER command is:
ALTER UCAT.CRPLUS2 UNLOCK

66

ICF Catalog Backup and Recovery: A Practical Guide

3. Ensure that the structural integrity of the recovered ICF catalog is intact by
using the IDCAMS EXAMINE INDEXTEST command. This validates that the
recovered ICF catalogs vertical and horizontal pointers in the index
component are correct. The format of the EXAMINE command is:
EXAMINE NAME(UCAT.CRPLUS2) INDEXTEST

4. Create a current backup copy of the recovered catalog. You may use the
IDCAMS EXPORT command to use ICFRU again in case a new problem is
detected.

Chapter 5. Using ICFRU

67

68

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 6.

Using Catalog RecoveryPlus

This chapter explains how to use Catalog RecoveryPlus (CR+), an ICF catalog
management tool from Mainstar Software Corporation (more information on this
product is located at http://www.mainstar.com). Catalog RecoveryPlus can be
licensed directly from Mainstar, or from IBM, Program Number 5620-FGY.
With Catalog RecoveryPlus, you can backup and recover ICF catalogs, both the
BCS and VVDS, including forward recovery with SMF records. You can also
perform ICF catalog diagnostic and troubleshooting functions, as well as
day-to-day ICF catalog maintenance.
This chapter provides basic how-to information on Catalog RecoveryPlus
commands. It also briefly discusses the ability to execute all CR+ commands
through its ISPF interface and the RACF FACILITY class profiles that are
available.
The information in this chapter about Catalog RecoveryPlus is not intended to be
complete and up-to-date with the most recently issued product features. You
should consult Mainstar Catalog RecoveryPlus User Guide, document number
006-0202, for complete details. This document can be obtained from Mainstar
Software Corporation, for use during trial evaluation and subsequent licensing of
the software product.

Copyright IBM Corp. 1999, 2001

69

6.1 Introduction to Catalog RecoveryPlus facilities

As a reminder, ICF catalogs do break. If you are not prepared for it, your system
could have a serious outage in accessing thousands, tens of thousands, or
hundreds of thousands of data sets. ICF catalogs, including both BCSs and their
related VVDSs, are software structures. This means they are accessed and
updated by software catalog management, to be exact and this is very
complicated software.
Catalogs are often shared across multiple systems, where in-storage control
blocks and buffers are maintained and accessed by each individual system
during open, close, and update processing of your data sets. This creates a very
complex environment in which ICF catalog management operates. All it takes is
the tiniest failure in the software code, and you risk a catalog failure. Catalog
backups, taken frequently and accurately, are your first level of insurance against
a failure. Your second level of insurance is knowing how to recover an ICF
catalog from a failure, and completing that recovery in the shortest possible time.
To accomplish this, you need to ensure that all ICF catalogs in your environment
are being backed up, as frequently as you can possibly afford to. At a minimum,
make sure that you back up your catalogs at least once a day. For heavily
updated catalogs, consider backups that are taken multiple times a day.
Do not think that the BCS is the only part of your ICF catalog environment that
needs to be backed up. If you have a complex DB2 environment, for example,
spanning across dozens of disk volumes, you should consider backing up the
VVDSs on those volumes. This provides a safeguard against a VVDS failure on
one volume that would otherwise require you to recover the databases across all
of those volumes, and then bring them up to date from those backups. VVDS
backups with Catalog RecoveryPlus are very fast, proving to be a small cost
compared to the time it would take to recover from a failure situation of a VVDS.
Finally, the only way to know how to recover an ICF catalog, whether it is a BCS
or VVDS, is through practice. Your installation almost certainly practices its
disaster recovery plan, and you should practice ICF catalog recovery as well. It is
advised that you not practice this recovery while at your DR site, but rather,
practice recovery at your primary site, and on your production catalogs and
volumes. This can most easily be done through simulation capabilities, and that
is possible with Catalog RecoveryPlus.
The Catalog RecoveryPlus commands covered in this chapter are:

70

BACKUP
RECOVER
DIAGNOSE
MERGECAT

ICF Catalog Backup and Recovery: A Practical Guide

 ZAP (PRINT, DELETE, PATCH functions)

SUPERCLIP

There are several more commands and features in Catalog RecoveryPlus that
are not covered in this redbook, because they are relevant to day-to-day ICF
catalog maintenance, and not to ICF catalog backup and recovery. These
include, for example, the ALTER command to make data value changes to BCS
and VVDS records.

6.2 Catalog RecoveryPlus ISPF interface

All Catalog RecoveryPlus commands can be executed interactively from ISPF
panels, or from batch jobs. For the examples and case studies in this redbook,
we have run all of the test and examples in batch, since they are easier to
illustrate in the book.
Figure 6-1 shows the Main Menu panel, which is the starting point into the ISPF
selection screens, where you can specify the command that you want to execute.

Menu Diagnostics Preferences

-----------------------------------------------------------------------------V6R101A
Catalog RecoveryPlus - Main Menu
Command ==>
Enter an option from the list below:
1
2
3

Search for Catalogs

Search for VVDS's
Search for Storage Groups

Utilities:
4 Backup
5 Recover BCS
6 Recover VVDS
7 Mergecat
8 Superclip
9 ZAP
X

10
11
12
13
14
15

Alter BCS-BACK-POINTERS
Alter BCS-DEVICETYPE
Alter BCS-VOLSER
Diagnose Alias
Diagnose BCS-VVDS
Diagnose VVDS-BCS

Exit
Copyright (C) 2001 Mainstar Software Corporation
All Rights Reserved

Figure 6-1 CR+ ISPF Main Menu panel

Chapter 6. Using Catalog RecoveryPlus

71

The CR+ Submit Backup panel is shown in Figure 6-2. It provides the ability to
specify all keywords and parameters as on a batch-submitted command. After
the panel is completed, a batch job is created, ready for submission of the
BACKUP command.

Menu Diagnostics Preferences

------------------------------------------------------------------------------CAT530
CR+ - Submit Backup (BCS/VVDS)
Command ===> ________________________________________________________
(S) SELECT AN OPTION:
_ Build JCL/Edit for Submit
BACKUP OPTIONS:
BCS/VVDS
(S) Selected List
OUTPUT Dataset Name
OUTFILE (DDName)
ADVANCED OPTIONS:
DIAGNOSEBCS
ACCEPTDIAGNOSE
DISPOSITION
RESERVE
MESSAGETEXT
EXCLUDEBCS(..)

=>
=>
=>
=>

B
B -BCS, V -VVDS
1 CATALOGS SELECTED
'EXAMPLE.ODS.CATALOG.BACKUP.DSN'
________
OUTPUT dataset must not exist

(BCS/VVDS)
=> N
=> E
=> O
=> N
=> N
=> N

Y
I
O
Y
N
Y

or N
-Informational, W -Warnings, E -Errors
-OLD, S -SHR
or N
-None, F -Full
or N

ADVANCED OPTIONS: (BCS Only)

EXAMINE with DATATEST => N
Y or N
EXAMINE with INDEXTEST => Y
Y or N
ACCEPTEXAMINE
=> E
I -Informational, W -Warnings, E -Errors
Include ALIASes
=> Y
Y or N
MASTERCATALOG
=> 'SYS.MASTER.CATALOG'

Figure 6-2 CR+ BACKUP BCS/VVDS command panel

Figure 6-3 and Figure 6-4 show the CR+ Submit Recover BCS and VVDS
panels, which also create a batch job for submission of the RECOVER
command.

72

ICF Catalog Backup and Recovery: A Practical Guide

Menu Diagnostics Preferences

------------------------------------------------------------------------------CAT540
CR+ - Submit Recover BCS
Command ===> ______________________________________________________________
(S) SELECT AN OPTION:
_ Build JCL/Edit for Submit
RECOVER OPTIONS:
BCS Name
=> 'USERCAT.UCAT2'____________________________
INFILE (DDName)
=> ________
INDATASET
=> ___________________________________________
RCVDD1 Dataset => ___________________________________________
ALIAS
=> A
A -ALIAS, N -NOALIAS, O -ALIASONLY
ADVANCED OPTIONS:
SIMULATE
=> D
D -DEFINE, R -RESTORE, F -FORWARD
PRINT
=> N
N -None, D -Data, K -Key
INTOEMPTY
=> N
Y or N
LOCK
=> Y
Y or N
Mode
=> _
R -RESTART, B -BACKOUT
VVDSUPDATE
=> Y
Y or N
NEWNAME DATASET => ___________________________________________
JOURNAL DATASET => ___________________________________________
MASTERCATALOG
=> 'SYSD80.MASTER.CATALOG'
ADVANCED FORWARD RECOVERY OPTIONS:
Recover FORWARD
=> N
Y or N
- SMF Dataset => ___________________________________________
- FROM Date/Time => 2000001 00:00:00
yyyyddd hh:mm:ss
- To Date/Time
=> 2000001 00:00:00
yyyyddd hh:mm:ss
- RCVMRG Dataset => ___________________________________________
- RCVSMF Dataset => ___________________________________________
- SMFERR Dataset => ___________________________________________

Figure 6-3 CR+ RECOVER BCS command panel

Chapter 6. Using Catalog RecoveryPlus

73

Menu Diagnostics Preferences

-----------------------------------------------------------------------------CR+ - Submit Recovery VVDS
Command ===> _____________________________________________________________
(S) SELECT AN OPTION:
Build JCL/Edit for Submit
RECOVER OPTIONS:
VVDS Name
INFILE (DDName)
INDATASET
RCVDD1 Dataset

=>
=>
=>
=>

BKP001
________
__________________________________________
__________________________________________

ADVANCED OPTIONS:
SIMULATE
=> D
PRINT
=> N
INTOEMPTY
=> N

D -DEFINE, R -RESTORE, F -FORWARD

N -None, D -Data
Y or N

ADVANCED FORWARD RECOVERY OPTIONS:

Recover FORWARD
=> N
Y or N
- SMF Dataset
=> __________________________________________
- FROM Date/Time => 2000001 00:00:00
yyyyddd hh:mm:ss
- To Date/Time
=> 2000001 00:00:00
yyyyddd hh:mm:ss
- RCVMRG Dataset => __________________________________________
- RCVSMF Dataset => __________________________________________
- SMFERR Dataset => __________________________________________

Figure 6-4 CR+ RECOVER VVDS command panel

6.3 RACF FACILITY class profiles

Catalog RecoveryPlus provides the ability to control user access to command
execution, through FACILITY class profiles that are recognized by RACF and
other security products.
The profiles have been established at the command level, and sometimes, at the
subcommand level. This gives you maximum flexibility to control who will be
authorized to use Catalog RecoveryPlus, or for the potentially destructive
command functions, who can execute certain commands or subcommands.
Figure 6-5 shows the FACILITY class profiles that are available.

74

ICF Catalog Backup and Recovery: A Practical Guide

MAINSTAR.CR+
MAINSTAR.CR+.BACKUP
MAINSTAR.CR+.BACKUP.BCS
MAINSTAR.CR+.BACKUP.VVDS
MAINSTAR.CR+.DIAGNOSE
MAINSTAR.CR+.DIAGNOSE.BCS-VVDS
MAINSTAR.CR+.DIAGNOSE.VVDS-BCS
MAINSTAR.CR+.DIAGNOSE.ALIAS
MAINSTAR.CR+.EXPLORE
MAINSTAR.CR+.ALTER
MAINSTAR.CR+.ALTER.BCS-DEVICETYPE
MAINSTAR.CR+.ALTER.BCS-VOLSER
MAINSTAR.CR+.ALTER.BCS-BACK-POINTERS
MAINSTAR.CR+.MERGECAT
MAINSTAR.CR+.RECOVER
MAINSTAR.CR+.RECOVER.BCS
MAINSTAR.CR+.RECOVER.VVDS
MAINSTAR.CR+.SUPERCLIP
MAINSTAR.CR+.ZAP
MAINSTAR.CR+.ZAP.VVDS
MAINSTAR.CR+.ZAP.VVDS.DELETE
MAINSTAR.CR+.ZAP.VVDS.PATCH
MAINSTAR.CR+.ZAP.VVDS.PRINT
MAINSTAR.CR+.ZAP.BCS
MAINSTAR.CR+.ZAP.BCS.DELETE
MAINSTAR.CR+.ZAP.BCS.PATCH
MAINSTAR.CR+.ZAP.BCS.PRINT
MAINSTAR.CR+.ZAP.DSN
MAINSTAR.CR+.ZAP.DSN.DELETE
MAINSTAR.CR+.ZAP.DSN.PATCH
MAINSTAR.CR+.ZAP.DSN.PRINT

Figure 6-5 CR+ FACILITY class profiles

6.4 BACKUP command

The Catalog RecoveryPlus BACKUP command provides the ability to create
backup copies of both parts of an ICF catalog the BCS and the VVDS. One or
more BCSs or VVDSs can be specified on each BACKUP command, and with
mask filtering values, groups of BCSs and VVDSs can be selected in a single
command. If you desire, all connected catalogs, including the systems master
catalog, can be backed up with one execution of the BACKUP command. In all
cases, all selected BCSs or VVDSs in one command are stacked on a single
output file.

Chapter 6. Using Catalog RecoveryPlus

75

The BACKUP output file is designated by either the OUTFILE or OUTDATASET

keyword. If OUTFILE is coded, it points to an external DD statement, on which
the output file data set name and other attributes are specified. If the output data
set has been pre-allocated and cataloged, it can be dynamically allocated with
the OUTDATASET keyword.
The BACKUP command has several features that allow you to analyze the
integrity of BCSs and VVDSs just prior to their backup, by invoking IDCAMS
EXAMINE or DIAGNOSE commands under the covers. Execution of these
commands is done by user keyword specification, and the output reports from
these commands are subsequently inserted into the user output. Also, a return
code keyword allows you to designate whether backup processing is to continue,
based on the level of return code returned from these IDCAMS commands.
In addition to backing up the BCSs or VVDSs records, the BACKUP command
backs up the necessary information to automatically delete/define the BCS or
VVDS at the time of recovery.

6.4.1 Specifying BCSs to be backed up

The BACKUP BCS command allows you to specify a single BCS, as illustrated in
Figure 6-6, to be backed up to the output file designated by the OUTFILE
keyword.

//BCKCPY DD DSN=BKUP.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(UCAT.CRPLUS1) OUTFILE(BCKCPY)

Figure 6-6 Back up a single BCS

Or, you can back up multiple, explicitly-specified BCSs, all of which are backed
up to the single output file designated by the OUTFILE keyword, as illustrated in
Figure 6-7.

//BCKCPY DD DSN=BKUP.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(UCAT.CRPLUS1 UCAT.CRPLUS2 UCAT.CRPLUS3) OUTFILE(BCKCPY)

Figure 6-7 Back up multiple BCSs, explicitly-specified

Or, you can back up multiple BCSs, by specifying a mask filter value. Again, all of
the BCSs selected by the mask filter are backed up to the single output file. In
Figure 6-8, all BCSs that match the mask value UCAT.CRPLUS* are backed up.

76

ICF Catalog Backup and Recovery: A Practical Guide

//BCKCPY DD DSN=BKUP.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(UCAT.CRPLUS*) OUTFILE(BCKCPY)

Figure 6-8 Back up multiple BCSs, with a mask filter

When using mask filter values to identify BCSs, you can also use the
EXCLUDEBCS keyword, as shown in Figure 6-9, to specifically exclude those
BCSs that you do not want to include in the selection list created by the mask
filter.

//BCKCPY DD DSN=BKUP.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(UCAT.CRPLUS*) EXCLUDEBCS(UCAT.CRPLUS10) OUTFILE(BCKCPY)

Figure 6-9 Back up multiple BCSs, excluding one

Backing up all BCSs in the system

A valuable feature of the BACKUP command allows you to specify an
all-powerful mask of ** (double asterisk), where the ** value indicates all
catalogs on the system. This technique, illustrated in Figure 6-10, includes in its
selection list all connected catalogs in the master catalog on the system on
which the BACKUP command is run. This includes the master catalog itself
(which includes all other master catalogs that are connected to this master
catalog as user catalogs).

//BCKCPY DD DSN=BKUP.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(**) OUTFILE(BCKCPY)

Figure 6-10 Back up all BCSs in the environment

The reason this technique is so valuable is that you can add and remove user
catalogs, or change names of existing catalogs, without having to worry about
updating the catalog backup jobs.
A real-world example of how things can go wrong may illustrate the value of this.
The system programmer at an installation discovered a broken catalog when
they ran an EXAMINE INDEXTEST on all of their catalogs. The EXAMINE error
message indicated duplicate key and out-of-sequence records within a data CI of
his catalog. They searched for the backup of that particular BCS, so that a

Chapter 6. Using Catalog RecoveryPlus

77

recovery could be started, only to find there was no backup. Two years earlier,
when the catalog was first allocated, someone forgot to add the appropriate new
EXPORT step in the catalog backup job for this catalog, and all this time passed
without any backups being taken. With the double-asterisk facility in the Catalog
RecoveryPlus BACKUP command, any new, deleted, or renamed catalogs are
automatically put into the selection list the next time BACKUP is executed.

Backing up BCSs at different frequencies

In many ICF environments, some BCSs are volatile. Others are relatively static,
depending on frequency of allocations and deletions of data sets through those
BCSs. Also, the speed of recovery required for one BCS might be significantly
different than that for other BCSs. For these reasons, you should consider setting
up multiple backup jobs, where some are run at different frequencies than others,
and each job identifies the specific BCSs and/or VVDSs to be backed up at that
time.
For backing up your BCSs, the ease with which this can be done depends on
how good your BCS naming conventions are, by allowing you to specify mask
values for specific groups of BCSs to be backed up in each job.
For example, you might want all BCSs that catalog application development data
sets (assume these are highly active) backed up every four hours, whereas
those BCSs that catalog stable production data sets could be backed up once a
day. You could specify the following mask in your command that executes every
four hours:
BACKUP BCS(UCAT.DEV.**) ...

Or, you code the following mask in your command that executes once a day:
BACKUP BCS(UCAT.PROD.**) ...

6.4.2 Specifying VVDSs to be backed up

With the BACKUP VVDS command, you have similar options for specification of
one or more VVDSs to be backed up in a single command, and all selected
VVDSs will be backed up to a single output file.
To make coding simpler, the actual data set name of the VVDS is not specified,
but rather, you specify the volser for the VVDS to be backed up. Since the
standard (and required) data set name of a VVDS is SYS1.VVDS.Vvolser, it is a
simple matter for CR+ to convert the specified volser list into VVDS names.
You can specify a single VVDS, as illustrated in Figure 6-11, to be backed up to
the output file designated by the OUTFILE keyword.

78

ICF Catalog Backup and Recovery: A Practical Guide

//BCKCPY DD DSN=BKUP.VVDSBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP VVDS(VBOX01) OUTFILE(BCKCPY)

Figure 6-11 Back up a single VVDS

Or, you can specify multiple, explicit VVDS names, as illustrated in Figure 6-12,
all of which will be backed up to the single output file designated by the OUTFILE
keyword.

//BCKCPY DD DSN=BKUP.VVDSBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP VVDS(VBOX01 VBOX02 VBOX03) OUTFILE(BCKCPY)

Figure 6-12 Back up multiple VVDSs, explicitly-specified

Or, you can specify a volser mask, and all of the VVDSs selected by the mask
filter will be backed up to the single output file. In Figure 6-13, all VVDSs that
match the mask value VBOX* are backed up.

//BCKCPY DD DSN=BKUP.VVDSBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP VVDS(VBOX*) OUTFILE(BCKCPY)

Figure 6-13 Back up multiple VVDSs, with a mask filter

Or, you can back up the VVDSs on all volumes in your entire system, as
illustrated in Figure 6-14.

//BCKCPY DD DSN=BKUP.VVDSBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP VVDS(**) OUTFILE(BCKCPY)

Figure 6-14 Back up all VVDSs in the environment

When using mask filter values to identify VVDSs, you can also use the
EXCLUDEVVDS keyword, to specifically exclude those VVDSs that you do not
want to include in the selection list created by the mask filter, as shown in
Figure 6-15.

Chapter 6. Using Catalog RecoveryPlus

79

//BCKCPY DD DSN=BKUP.VVDSBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP VVDS(VBOX*) EXCLUDEVVDS(VBOX10) OUTFILE(BCKPY)

Figure 6-15 Back up multiple VVDSs, excluding one

6.4.3 Diagnosing BCS or VVDS integrity prior to backup

The structural integrity of BCSs and VVDSs is vital to successful operation of
your system. Yet, people frequently say catalogs do not break anymore, or I
have RAID storage now, and with a mirrored copy, I never have catalog failures.
This ignores the fact that many ICF catalog failures are soft in nature, where
there are problems within the BCS or VVDS, but you simply do not know about
them yet.
With some catalog errors, you can run for weeks or months, without even
realizing there is a problem. Eventually, though, some required process will fail as
a result of that error. Worse, when the error finally manifests itself, it could take
the form of a catastrophic failure, and it could occur at the worst possible time.
The sooner you identify the failure, and fix it, the more likely you will recover with
less difficulty, or recover more successfully.
For that reason, the absolute minimum you should perform is an IDCAMS
EXAMINE, with the INDEXTEST keyword, on every BCS prior to backing it up.
At some regular frequency, you should also run a DIAGNOSE ICFCATALOG,
without the COMPAREVVDS keyword specified. This performs an
inside-the-BCS record structure and relationship check to determine if data
content within the BCS is becoming corrupted.
The Catalog RecoveryPlus BACKUP command provides the ability to issue
these commands, at your discretion, just prior to starting the backup. The
ACCEPTEXAMINE and ACCEPTDIAGNOSE keywords control whether an error
should stop your backup process. The warning and error messages from the
diagnostic commands that you see in your output listing give you a clearer picture
of whether the BCSs and VVDSs that you are backing up are sound:
EXAMINE(INDEXTEST | DATATEST)

The IDCAMS EXAMINE command performs its structure-test at least as well

as any software can, so by default, Catalog RecoveryPlus internally issues an
IDCAMS EXAMINE INDEXTEST just prior to backing up each BCS. If you
request it with the keyword EXAMINE(DATATEST), the CR+ BACKUP
internally issues EXAMINE with this option. If neither is desired, you always
have the option to specify NOEXAMINE.

80

ICF Catalog Backup and Recovery: A Practical Guide

 DIAGNOSEBCS

This keyword indicates that the IDCAMS DIAGNOSE ICFCATALOG

command, without the COMPAREVVDS keyword, is to be invoked by the CR+
BACKUP BCS command just prior to backing up each BCS. When specified,
DIAGNOSE ICFCATALOG reads through every VSAM and non-VSAM
catalog record in the BCS, checking for consistency and accuracy of all data
fields in each record, as well as checking relevant relationships across all
records throughout the BCS.
DIAGNOSEVVDS

This keyword indicates that the IDCAMS DIAGNOSE VVDS command,

without the COMPAREBCS keyword, is to be invoked by the CR+ BACKUP
VVDS command just prior to backing up each VVDS. When specified,
DIAGNOSE VVDS reads through every record in the VVDS, checking for
consistency and accuracy of all data fields in each VVR and NVR.
You can find sample output from these keywords in Chapter 3, Analyzing catalog
integrity on page 21. It illustrates the benefit of frequent and regular diagnostic
processing against your BCSs and VVDSs.

6.4.4 Restricting access to the BCS or volume during backup

IDCAMS EXPORT processing issues an exclusive ENQ for update throughout
the time that a BCS is being backed up. By contrast (and by default), the CR+
BACKUP BCS and VVDS commands do not issue a similar ENQ, and therefore,
do not restrict update or locate access during backup processing. This means
that while BACKUP is running, DELETE, DEFINE, and ALTER processing can
occur coincidentally, with the slight possibility that a small amount of fuzziness
occurs on the backup. This can be identified by running two-way DIAGNOSE
commands following any catalog recovery, and then corrected by execution of
any fix commands that are generated from the recovery.
The benefit of this logic with Catalog RecoveryPlus is that catalog backups can
be scheduled more frequently, with less impact on your environment. For this
reason, we recommend you run BACKUP BCS with NORESERVE in effect
(which is the default), unless you can specifically identify a risk factor that is
unacceptable to you.
If you prefer to restrict catalog access during backup, this can be done with the
RESERVE keyword on the BACKUP command (as previously mentioned,
NORESERVE is the default). The RESERVE keyword, in the case of the
BACKUP BCS command, does not issue a volume reserve (as its name implies),
but rather the same exclusive ENQ for update that EXPORT issues.

Chapter 6. Using Catalog RecoveryPlus

81

When backing up the VVDS, similar considerations apply. But there is one
difference. When RESERVE is specified for VVDS backup processing, an ENQ
with the RESERVE option is in effect throughout the time of the backup. This
locks up the entire volume from any I/O for the duration of the VVDS backup
(which is only 1 to 2 seconds), and is something you should take into account.

6.4.5 Creating multiple backup copies

With the CR+ BACKUP command, you can automatically create multiple backup
copies as many copies as the output files you include in your OUTDATASET or
OUTFILE keyword specification. Figure 6-16 shows how simple it is to code a
BACKUP command for multiple copies of a backup.

//BCKCPY1 DD DSN=BKUP1.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
//BCKCPY2 DD DSN=BKUP2.CATBAKS(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(**) OUTFILE(BCKCPY1,BCKCPY2)

Figure 6-16 Sample duplex BACKUP command

The various reasons for creating multiple backup copies are thoroughly
discussed in Chapter 2, Backing up catalogs on page 11. But it might help if
one of those reasons is reinforced again. Do not forget that backup files need to
be cataloged in order to locate them when a recovery is necessary. If the catalog
that needs to be restored happens to be the one where the backup is cataloged,
you could be in big trouble. If the backup is on disk, even if you can locate the
specific disk volser on which it resides, you need to recatalog it before it can be
accessed in the catalog recovery job. Recataloging could be difficult, since
catalog management forces it to a catalog of the same name (and right now, that
catalog is the one that requires recovery).
Perhaps the best and easiest solution is to establish a fundamental catalog
backup methodology that plans ahead for this situation. To do this, first create
two dedicated catalogs, to be used just for cataloging the backups of your ICF
catalog environment. Assign each catalog a single alias that identifies the catalog
backups that will be cataloged in it. With two catalogs, where you catalog only
the catalog backups in each catalog, there is less risk of failure to your backup
catalog. You would then run the BACKUP command specifying backup copies,
with different high-level qualifiers in their data set name, so that each backup
copy is cataloged in its respective catalog.
With this technique of multiple backup copies, each cataloged in a different
catalog. If a failure occurs in one catalog where your BCS or VVDS backups are
cataloged, you can use the other copy to recover.

82

ICF Catalog Backup and Recovery: A Practical Guide

Cataloging of your SMF data is another important consideration. Since these

files are required for forward recovery of a BCS or VVDS, what if the failing
catalog points to your SMF files? To ensure accessibility, you should create your
backup methodology in such a way that you can always access your cataloged
backup files and SMF data. Techniques for ensuring this are standard for all
types of backup/recovery utilities, and are discussed in detail in Chapter 2,
Backing up catalogs on page 11.
One concern you may have is how difficult this is to set up. Figure 6-16 shows
how to code a duplex BACKUP step using Catalog RecoveryPlus that matches
our logic flow diagram in Figure 6-17, where all BCSs throughout the entire
system are backed up in a single command, with two copies created, one routed
to the file on the BCKCPY1 DD statement, and the other routed to the BCKCPY2
DD statement. The high-level qualifier on the data set name for each, BKUP1
and BKUP2, would be the aliases of different user catalogs, causing each
backup copy to be cataloged in a different location.

Assigned
Alias: BKUP1

Assigned
Alias: BKUP2

BCS1

BCS2

...

BCSn

BKUP1.CATBACKS(+1)

BKUP2.CATBACKS(+1)

Copy 1 of all
BCSs backed
up here

Copy 2 of all
BCSs backed
up here

Figure 6-17 Catalog RecoveryPlus duplex backup, duplex cataloging

Chapter 6. Using Catalog RecoveryPlus

83

Another concern may be how much overhead a duplex backup adds to your BCS
or VVDS backup job. As you would expect, there will be a relatively small
increase in EXCPs, but other performance-related factors should not change. To
illustrate this, we created two sets of tests in our lab the first with duplex BCS
backups and the second with duplex VVDS backups.
In our duplex BCS backup test, we backed up 28 BCSs, where Test1 created a
single backup copy, and Test2 created two backup copies. The 28 BCSs
contained a cumulative total of 63,216 catalog records (which is not very large,
but this example is intended to give you relative differences). As you can see in
Table 6-1, the EXCPs only increased from 4,008 in Test1 to 4,253 in Test2, or
slightly more than 5%, for a test that was a complete duplication of the backup
files.
Table 6-1 Duplex BCS backup overhead statistics
Test

Backup
copies

Elapsed time

CPU time

EXCPs

#1

15 sec

.02 sec

4,008

#2

15 sec

.02 sec

4,253

In our duplex VVDS backup test, we backed up 66 VVDSs, where Test3 created
a single backup copy, and Test4 created two backup copies. During BACKUP
selection processing, the specified volser mask encountered and identified 29
volumes that would not be included, because they were non-system-managed
and did not have a VVDS allocated on them. Our test BACKUP job automatically
ran the IDCAMS DIAGNOSE VVDS on each VVDS (and in fact, encountered one
VVDS that had errors, so the test times included dumping the VVDS record to
SYSPRINT). All VVDSs were allocated at the default size of TRK(10 10). As you
can see in Table 6-2, the elapsed time was exactly the same (recording accuracy
was only to a second), CPU time was the same (recording accuracy of 1/100th of
a second), and EXCPs increased by less than 1%.
Table 6-2 Duplex VVDS backup overhead statistics

84

Test

Backup
copies

Elapsed time

CPU time

EXCPs

#3

1 min, 54 sec

.11 sec

33,667

#4

1 min, 54 sec

.11 sec

33,926

ICF Catalog Backup and Recovery: A Practical Guide

6.4.6 Alias processing with BACKUP BCS

If the default keyword, ALIAS, is in effect for the BACKUP command, all related
aliases for the BCS are copied from the current master catalog to the backup
copy at the start of backup for each BCS. If a subsequent RECOVER process is
executed, these aliases are restored to the master catalog on the system running
the RECOVER command. It is also possible to recover ALIASONLY, allowing you
to use this facility to replicate aliases to multiple master catalogs. If, for some
reason, you do not want aliases backed up at all, you can specify NOALIAS on
the BACKUP command.

6.4.7 BACKUP BCS bypasses index processing

The Catalog RecoveryPlus BACKUP command automatically opens the BCS as
a data set. In other words, the Access Method Control Block (ACB) indicates that
we are opening an ICF catalog, but the open process takes place outside of
catalog management. The command specifically opens the data component of
the BCS for read access, without using the index. With this logic, BACKUP does
not need to worry about index errors. It reads through the data component of the
BCS in physical sequential order, deblocking records from the CIs within the
BCS, reblocking them, and writing them out to the backup.
This is unlike most other BCS backup tools, such as EXPORT, which open the
BCS as a catalog in other words, as a KSDS. If there are corruptions to the
index, the result can be unrecoverable backup failures.
Section 7.1, Case study #1: Ensuring good backups on page 113, illustrates the
importance of this logic, where the specific type of BCS corruption resulted in
IDCAMS EXPORT not being able to back up all of the records in the BCS,
whereas the Catalog RecoveryPlus BACKUP backed up the entire BCS. The
difference was specifically due to this processing logic.

6.5 RECOVER command

The Catalog RecoveryPlus RECOVER command provides the ability to restore
or forward recover either part of an ICF catalog, a BCS or a VVDS, from a backup
copy that was created either by IDCAMS EXPORT (BCS only) or the CR+
BACKUP command.
If the backup was created by CR+ BACKUP, the records are first sorted and then
written to the BCS or VVDS being recovered. This logic alerts you (by way of
informational or error messages) to any duplicate key records that are identified.

Chapter 6. Using Catalog RecoveryPlus

85

The RECOVER input file is designated by either the INFILE or INDATASET

keyword. If INFILE is coded, it points to an external DD statement, on which the
input file data set name and other attributes are specified. If INDATASET is
coded, the data set name is specified, and a dynamic allocation is performed.
Since the backup file contains a copy of all BCSs or VVDSs backed up by a
single BACKUP command, this file is accessed to the point where the relevant
records for the BCS or VVDS to be recovered are found.

6.5.1 Restore versus forward recovery

The RECOVER command is fundamentally a restore BCS or VVDS facility, with
the additional ability to forward recover. It might help to explain these terms,
restore and recover, in some detail,

Restore
The term restore means that RECOVER is to retrieve backup records from the
input file, and write them to the output BCS or VVDS, exactly as they are in the
backup file, without updating them with SMF records. The term forward recover
means that RECOVER not only restores the backup records, but also applies any
updates to them, as indicated by relevant SMF records, to bring the catalog
forward to a more current time than it was at the time the backup was taken.
If that is all you want to do, the restored BCS or VVDS will have identical records
in it to those in the BCS or VVDS at the exact time of BACKUP processing. This
is usually an appropriate function to perform in situations where your BACKUP
and RECOVER command are executed in quick succession. There are no BCS
or VVDS updates that occur between the time of the BACKUP and the time of the
RECOVER. Valid examples of when you may perform just a restore include:
For a BCS, if you decide to reorganize it
For a BCS, if you need to change any of the allocation parameters, such as
allocated size, volser, CI size, the FREESPACE parameter, remove the
IMBED parameter, or other changes
For a BCS, when you are renaming it (see 6.5.6, Renaming the BCS during
restore on page 89, for more details of this facility)
For a BCS, when you determine that the error condition is a structural index
failure, and to repair it, you need to backup and restore, to rebuild the index
(see 7.1, Case study #1: Ensuring good backups on page 113, for a real-life
situation where this occurred)
For a VVDS, if you need to change the physical allocated size (see 7.6, Case
study #6: Changing a disk volser on page 131, for details and an example of
how to do this)

86

ICF Catalog Backup and Recovery: A Practical Guide

Forward recovery
Forward recovery of a BCS or VVDS is the process of applying all updates to the
BCS or VVDS that occurred between the backup start time and the restore, at
the time the restore is taking place. When this process is to be performed, it is
typical for the FROMBACKUPDATE and TOCURRENTDATE options to be in
effect, in which the backup time is retrieved from the backup file, and used as the
starting time from which SMF records are analyzed. The forward recovery ending
time is the time that the RECOVER command was initiated. If desired, alternative
FROM and TO dates and times can be specified.

6.5.2 Specifying the BCS or VVDS to restore

RECOVER allows only a single BCS or VVDS to be specified in one execution of
the command, as illustrated in Figure 6-18 and Figure 6-19.
With RECOVER BCS, the specification must be the fully-qualified name of the
BCS as it was at the time of BACKUP (for details on how to rename the BCS, see
6.5.6, Renaming the BCS during restore on page 89).

//BCKCPY DD DSN=BKUP.CATBAKS(0),DISP=SHR
RECOVER BCS(UCAT.CRPLUS1) INFILE(BCKCPY)

Figure 6-18 RECOVER command for a BCS

For RECOVER VVDS, the specification must be the full volser of the volume on
which the VVDS is to be recovered (and not the actual VVDS data set name).

//BCKCPY DD DSN=BKUP.VVDSBAKS(0),DISP=SHR
RECOVER VVDS(VBOX01) INFILE(BCKCPY)

Figure 6-19 RECOVER command for a VVDS

6.5.3 Explicit/implicit DEFINE of the BCS or VVDS to restore

Prior to beginning the restore, a check is made to determine if a BCS or VVDS of
the same name already exists (and in the case of a BCS, if it is connected to the
current master catalog). If one is found (and INTOEMPTY is not specified),
RECOVER issues an IDCAMS DELETE command (with RECOVERY specified),
to physically delete the BCS or VVDS. The RECOVERY keyword on the DELETE
command tells IDCAMS that a BCS or VVDS is being deleted for recovery

Chapter 6. Using Catalog RecoveryPlus

87

purposes and that all data sets defined in it are to remain untouched. Then, using
the self-describing information that was written to the backup file, an IDCAMS
DEFINE USERCATALOG (in the case of RECOVER BCS) or DEFINE CLUSTER
command (in the case of RECOVER VVDS) is constructed and issued.
If you want, you can issue your own DELETE/DEFINE of the BCS or VVDS, in
which case you then specify INTOEMPTY on the RECOVER command. For a
BCS, this allows you to change the volser, physical allocation values, or any of
the VSAM attribute values, such as CISZ. For a VVDS, this allows you to change
the physical allocation values, but you cannot change anything else. You would
also use this technique if you want to change a BCS name. Refer to 6.5.6,
Renaming the BCS during restore on page 89, for more details.

6.5.4 Locking versus unlocking the BCS

RECOVER BCS, by default of the LOCK keyword, issues an IDCAMS ALTER
LOCK command on the named BCS prior to recovery, to be in effect throughout
the time restore and forward recovery processing is performed. We strongly
recommend you use this facility, because it ensures exclusive access by your
recovery job to the BCS while it is being recovered. You do not want application
production jobs issuing data set searches or data set cataloging (and
uncataloging) while the recovery is in progress. Locking makes certain that no
other accesses are made to the BCS while the RECOVER command is
executing.
For the LOCK to be successful, the RACF profile IGG.CATLOCK must be defined
prior to issuing the RECOVER command. If it is not defined, an IDC3009I
message is issued with a return code 186, reason code 2, and RECOVER is
terminated without further processing.
You can also issue the IDCAMS ALTER LOCK command to the BCS prior to
recovery and would then specify the NOLOCK keyword on the RECOVER
command.

6.5.5 Alias handling during BCS restore/recovery

BACKUP processing dumps all aliases for the BCS to the output file. By default,
they are re-defined to the current master catalog during RECOVER processing.
As a prerequisite of RECOVER processing, whether you explicitly issue the
commands yourself or let RECOVER do it implicitly, the existing version of the
BCS is deleted and disconnected from the master catalog, and all aliases for the
BCS are deleted from the master catalog. Restoring them to the master catalog
from the BCS backup file is the easiest and safest way to establish the aliases for
the newly restored/recovered BCS.

88

ICF Catalog Backup and Recovery: A Practical Guide

If your RECOVER command also performs a forward recovery, any new or

deleted aliases for this BCS that are identified by SMF records that are different
from those on the backup file are also applied during recovery processing.
If your environment has shared access to the BCS that is being restored or
recovered, the CR+ DIAGNOSE ALIAS command can be executed after
RECOVER is complete to compare the aliases for this BCS against those in
another master catalog. Alternatively, if you want to directly update the aliases for
other master catalogs in your environment, you can run the RECOVER
command additional times for the same BCS name, with the keyword
ALIASONLY specified, and with the MASTERCATALOG keyword specified, in
which case all aliases for the specified BCS will be restored to the specified
master catalog.

6.5.6 Renaming the BCS during restore

For many installations, new BCS allocations have occurred over a long period of
time, with various personnel assigned to ICF catalog management. In other
installations, corporate mergers or data center consolidations have caused ICF
catalogs to be imported into the environment from other locations. As a result,
BCS naming convention standards are in great disorder, and it is virtually
impossible to use BCS mask values for day-to-day catalog maintenance
functions.
With the Catalog RecoveryPlus RECOVER BCS ... NEWNAME keyword, it is
possible to rename any BCS, relatively quickly and easily. You simply allocate a
BCS with the new name that you want and then point the RECOVER BCS
command to that BCS by specifying its name in the NEWNAME keyword.
Figure 6-20 illustrates how to code this and the full steps involved in
accomplishing a BCS rename. To illustrate an additional idea of what you can do,
this example also moves the BCS from volume SBOX02 to volume SBOX36. As
you can see, a forward recovery with SMF is not required, provided that during
the time between BACKUP and RECOVER, there are no updates to the BCS
being renamed (in other words, you should ensure that the BCS is quiesced
throughout this entire time period).
Note that a journal file is specified with this version of the RECOVER command
and used for the VVDS update process when all BCS back-pointers are updated
from the old to new BCS name.

Chapter 6. Using Catalog RecoveryPlus

89

//S1 EXEC -- Catalog RecoveryPlus, with all of its regular JCL -//BCKCPY DD DSN=BKUP.CRPLUS1,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
BACKUP BCS(UCAT.CRPLUS1) OUTFILE(BCKCPY)
//S2 EXEC PGM=IDCAMS
//SBOX02 DD VOL=SER=SBOX02,UNIT=SYSALLDA,DISP=SHR
//SBOX36 DD VOL=SER=SBOX36,UNIT=SYSALLDA,DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE UCAT.CRPLUS1 USERCATALOG FILE(SBOX02) RECOVERY
DEFINE USERCATALOG (NAME(UCAT.CRPLUS5) ICFCATALOG CYL(10 ) BUFND(27) STRNO(8) VOLUME(SBOX36)) DATA(CISZ(4096)) INDEX(CISZ(1024) BUFNI(4))
//S3 EXEC -- Catalog RecoveryPlus, with all of its regular JCL -//BCKCPY DD DSN=BKUP.CRPLUS1,DISP=SHR
RECOVER BCS(UCAT.CRPLUS1) INFILE(BCKCPY) INTOEMPTY NEWNAME(UCAT.CRPLUS5) JOURNAL(YCJRES2.CRPLUS1.RECOVER.JOURNAL)

Figure 6-20 RECOVER BCS with NEWNAME

6.5.7 SMF data for forward recovery

Forward recovery of a BCS uses SMF record types 61, 65, and 66, which
represent:
Type 61: A data set DEFINE
Type 65: A data set DELETE
Type 66: A data set ALTER

Forward recovery of a VVDS uses SMF record type 60, which represents a
VVDS record insert, update, or delete, and applies to both VVR changes for
VSAM data set components, and NVR changes for non-VSAM data sets.

90

ICF Catalog Backup and Recovery: A Practical Guide

Only dumped SMF data is acceptable to RECOVER. This requires you to issue
an I SMF switch command for the in-use MANX/Y file, and then dump the SMF
data to a file that can be used as input to the RECOVER process.
With the Catalog RecoveryPlus RECOVER command, it is not necessary to
pre-extract the SMF records that will be used. Instead, you can simply gather up
all SMF dump data from all systems sharing the BCS or VVDS. As illustrated in
Figure 6-21 and Figure 6-22, you then concatenate all of the dumped SMF files
on the DD statement identified in the FORWARD(SMFFILE(ddname)) keyword.

//BCKCPY
//SMFINDD
//
//

DD
DD
DD
DD

DSN=BKUP.CATBAKS(0),DISP=SRH
DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
DSN=YCJRES1.SC65.SMFDATA,DISP=SHR

RECOVER BCS(UCAT.CRPLUS1) INFILE(BCKCPY) FORWARD(SMFFILE(SMFINDD))

Figure 6-21 RECOVER BCS FORWARD(SMFFILE) command

//BCKCPY
//SMFINDD
//
//

DD
DD
DD
DD

DSN=BKUP.VVDSBAKS(0),DISP=SHR
DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
DSN=YCJRES1.SC65.SMFDATA,DISP=SHR

RECOVER VVDS(VBOX01) INFILE(BCKCPY) FORWARD(SMFFILE(SMFINDD))

Figure 6-22 RECOVER VVDS FORWARD(SMFFILE) command

Chapter 6. Using Catalog RecoveryPlus

91

Figure 6-23 shows the BACKUP/RECOVER BCS logic flow. In the first step, the
BACKUP BCS command create a backup copy of multiple BCSs into a single file.
This file is used in the second step to forward recover BCS4. You can see how
the SMF dump files for all the systems are used along with the backup copy of
BCS4. Sample BACKUP BCS commands are shown in 6.4.1, Specifying BCSs
to be backed up on page 76.

VOL001

BCS1

BACKUP
Command
Parameters

VOL002

1
backup
catalog

BCS2
Reports
Messages
(SYSPRINT)

BACKUP
Command
Processor

BCS3
CATVOL

MCATA
VOL003

CATBAK(+1)

BCS4

BCS4
to be

2
recover
catalog

red
Resto

CATBAK(0)

RECOVER
Command
Parameters

CATBAK(-1)

e
na t
cate
Con

RECOVER
Command
Processor

SMF
Dump

s
File
SMF

SMF
Dump

VOL003

New BCS4
Old BCS4

Forward
Recovery
of BCS4

Figure 6-23 Catalog RecoveryPlus BACKUP/RECOVER BCS logic flow

92

ICF Catalog Backup and Recovery: A Practical Guide

SMF

SMF
Dump

Sorted,
Extracted
SMF
Records

Figure 6-24 shows the BACKUP/RECOVER VVDS logic flow. In the first step, the
BACKUP VVDS command creates a backup copy of multiple VVDSs into a single
file. This file is used in the second step to forward recover the VVDS in VOL003.
You can see how the SMF dump files for all the systems are used along with the
backup copy of the VVDS. Sample BACKUP VVDS commands are shown in
6.4.2, Specifying VVDSs to be backed up on page 78.

VOL001

VVDS

BACKUP
Command
Parameters

VOL002

VVDS

1
backup
VVDSs

Reports
Messages
(SYSPRINT)

BACKUP
Command
Processor

CATVOL

VVDS
VOL003

VVDSBAK(+1)

VVDS

VVDS
on
03
VOL0
to b e

VVDSBAK(0)

RECOVER
Command
Parameters

red
Resto

VVDSBAK(-1)

iles
FF
SM
ate
n
e
cat
Con

RECOVER
Command
Processor

SMF
Dump

recover
a VVDS

SMF
Dump

VOL003

New VVDS
OldVVDS

Forward
Recovery
of VVDS

SMF

SMF
Dump

Sorted,
Extracted
SMF
Recprds

Figure 6-24 Catalog RecoveryPlus BACKUP/RECOVERY VVDS logic flow

6.5.8 PRINTing BCS records during a RECOVER

If you want to see a process log printout of the RECOVER command execution,
you can use the PRINT keyword, specified as:
PRINT(NONE | DATA | KEY)

Chapter 6. Using Catalog RecoveryPlus

93

The operand parameters indicate:

NONE: This is the default. It specifies that records from the backup file, and in
the event of a forward recovery, merged records from the SMF file will not be
printed.
DATA: This specifies that the complete record, including key and data from
the backup file. In the event of a forward recovery, merged records from the
SMF file are printed.
KEY: This specifies that only the key from the backup file, and in the event of
a forward recovery, merged records from the SMF file will be printed. KEY is
not a valid specification for RECOVER VVDS, but if coded, it will be changed
to DATA.

6.5.9 Simulation as a BCS or VVDS recovery testing tool

An important key to successful BCS or VVDS recovery helping you to achieve
recovery that is completed in the shortest possible time frame is practice and
testing of your recovery procedures. Testing that is meaningful, although it is
often difficult to perform.
Many people tell us that they frequently test their ICF catalog recovery
procedures at the beginning of their overall DR test. This is definitely not
meaningful, because it is almost certainly testing the successful restore of a
catalog and not a forward recovery. Other people tell us they rarely test ICF
catalog recovery at all, presumably because they have the belief that catalogs
never break. As a result, they never test to determine if their knowledge of ICF
catalog recovery is sufficiently good, nor test whether they have procedures in
place to locate and pull the necessary SMF data together, and other technical
issues that affect successful recovery.
A keyword, SIMULATE, is provided with the Catalog RecoveryPlus RECOVER
command, for both BCS and VVDS recovery, and its subkeyword of FORWARD,
that enables you to fully test a complete forward recovery of either a BCS or a
VVDS. This allows you to test against very realistic situations, including large,
highly active, production BCSs or volumes that would otherwise not be available
for testing. The SIMULATE facility also tests to ensure you have good procedures
for collecting all SMF data, from all systems sharing the BCS or VVDS.
SIMULATE(FORWARD), as illustrated in Figure 6-25 and Figure 6-26, performs
a complete simulation of the restore and forward recovery of the specified BCS
or VVDS. The output report from the command execution is exactly as if the
actual forward recovery was done. As you would expect, SIMULATE does not
delete and define the output BCS or VVDS, nor does it actually write any records
to the output BCS or VVDS. (As a result of this last statement, you should not
use SIMULATE for timing tests of recovery.)

94

ICF Catalog Backup and Recovery: A Practical Guide

//BCKCPY
//SMFINDD
//
//

DD
DD
DD
DD

DSN=BKUP.CATBAKS(0),DISP=SRH
DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
DSN=YCJRES1.SC65.SMFDATA,DISP=SHR

RECOVER BCS(UCAT.CRPLUS1) INFILE(BCKCPY) FORWARD(SMFFILE(SMFINDD)) SIMULATE(FORWARD)

Figure 6-25 RECOVER BCS SIMULATE(FORWARD) command

//BCKCPY
//SMFINDD
//
//

DD
DD
DD
DD

DSN=BKUP.VVDSBAKS(0),DISP=SHR
DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
DSN=YCJRES1.SC65.SMFDATA,DISP=SHR

RECOVER VVDS(VBOX01) INFILE(BCKCPY)

FORWARD(SMFFILE(SMFINDD)) SIMULATE(FORWARD)

Figure 6-26 RECOVER VVDS SIMULATE(FORWARD) command

6.5.10 Re-sizing the VVDS during RECOVER processing

In certain circumstances, an already allocated VVDS may prove to be too small,
and you want to enlarge it. This can be particularly true for volumes where you
allowed the VVDS to be automatically allocated when the first data set is
allocated on the volume that requires the VVDS. In that situation, the default
allocation size for a VVDS is TRK(10 10).
If there are data sets already on the volume that have records in the VVDS, your
only option with standard MVS facilities is to unload all data sets off the volume
that have records in the VVDS, then delete and re-define the VVDS at your
chosen size, and finally reload the data sets.
While this sounds like an easy process, it can be time-consuming and tedious for
volumes with large numbers of existing data sets. With system-managed
volumes, both VSAM and non-VSAM data sets have records in the VVDS,
making this process even more difficult.
With the Catalog RecoveryPlus BACKUP and RECOVERY commands, you can
accomplish this with the following command sequence, without having to remove
any data sets from the volume:

Chapter 6. Using Catalog RecoveryPlus

95

1. BACKUP the VVDS.

2. Run an IDCAMS DELETE (with RECOVERY specified) of the VVDS, to
physically remove it from the volume.
3. Run an IDCAMS DEFINE CLUSTER to re-allocate the VVDS at your desired
larger size.
4. RECOVER the VVDS from the backup (forward recovery is not required).
To see actual commands and job stream output from this process, refer to 7.5,
Case study #5: Enlarging VVDS on page 129.
Note: Before you attempt to enlarge a VVDS, first determine from a LISTCAT
the current total allocated size of the VVDS, and factor that into the new, larger
size that you are contemplating. Also, determine whether the existing VVDS
has acquired multiple extents, and make certain that you include all of these
into your size calculation. The reason for this is that, during Catalog
RecoveryPlus RECOVER VVDS processing, it is not possible to allocate
secondary extents, and if there is not sufficient space in the VVDSs primary
allocation, RECOVER will fail with an error message and you must
DELETE/DEFINE, and start over.

6.6 DIAGNOSE command

Catalog RecoveryPlus provides its own version of a DIAGNOSE command, with
certain functions that might be similar to those in IDCAMS, but with large
differences. The CR+ DIAGNOSE command does not call IDCAMS DIAGNOSE,
but instead processes the selected BCSs and VVDSs with its own logic.
The primary design of the CR+ DIAGNOSE command is to provide an
easy-to-code and extremely high-performance ICF catalog diagnostic facility that
can be run on a frequent and scheduled basis, to help you keep your BCSs and
VVDS as clean and error-free as possible. To assist you, each DIAGNOSE
command can generate fix commands for all of the types of error situations that
it looks for. These fix commands are in the form of IDCAMS commands, stored in
a physical sequential (flat) file, which you can browse, edit, and then submit for
execution when you are ready and comfortable with them.

96

ICF Catalog Backup and Recovery: A Practical Guide

6.6.1 DIAGNOSE ALIAS command

The DIAGNOSE ALIAS command is designed to compare two master catalogs
to determine if their user catalog connections are consistent and if alias
relationships for their connected user catalogs are synchronized. Section 7.3,
Case study #3: Synchronizing aliases on page 121, shows a complete
execution of the DIAGNOSE ALIAS command, with some of the fix commands
that are generated.
This command is most likely executed:
After master catalog maintenance, where aliases have been updated in one
master catalog and you want to replicate that maintenance to one or more
system master catalogs
Following a user or master catalog recovery
After a MERGECAT of one or more alias groups from one user catalog to
another

PEER versus NONPEER processing

There are two fundamental ways to run the DIAGNOSE ALIAS command, using
the PEER or NONPEER keywords.
In PEER mode, as illustrated in Figure 6-27, both specified master catalogs are
considered as equals. The comparison logic identifies situations in each master
catalog that the other does not have and creates fixes to make each master
catalog look like the other. For example, if either master catalog has an alias
that the other does not have, a DEFINE ALIAS fix command is created.

//FIXDD
//
//

DD DSN=YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE,DISP=(,CATLG),
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE),
DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)

DIAGNOSE ALIAS PEER COMPARE-MASTERCATALOG(MCAT.SANDBOX.R10 MCAT.SANDBOX.Z02) FIXDATASET(YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE)

Figure 6-27 DIAGNOSE ALIAS PEER mode

With NONPEER, as illustrated in Figure 6-28, the first-specified master catalog is

considered to be superior to the second-specified master. The comparison
logic identifies situations where the superior master contains entries that the
subordinate master does not have. For example, if the first master catalog has an
alias that the second does not have, a DEFINE ALIAS fix command is created,
but if the second has an alias that is not in the first, a fix command is not created.

Chapter 6. Using Catalog RecoveryPlus

97

//FIXDD
//
//

DD DSN=YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE,DISP=(,CATLG),
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE),
DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)

DIAGNOSE ALIAS NONPEER COMPARE-MASTERCATALOG(MCAT.SANDBOX.R10 MCAT.SANDBOX.Z02) FIXDATASET(YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE)

Figure 6-28 DIAGNOSE ALIAS NONPEER mode

EXCLUDE/INCLUDE keywords
The DIAGNOSE ALIAS command provides EXCLUDE and INCLUDE keywords,
for both alias and user catalog specification, so that you can tailor the commands
execution more closely to your requirements. The example in Figure 6-29 shows
a situation where we only want to synchronize the master catalogs with respect
to aliases related to the user catalogs under our control.

//FIXDD
//
//

DD DSN=YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE,DISP=(,CATLG),
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE),
DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)

DIAGNOSE ALIAS NONPEER COMPARE-MASTERCATALOG(MCAT.SANDBOX.R10 MCAT.SANDBOX.Z02) INCLUDE-USERCATALOG(UCAT.CRPLUS*) FIXDATASET(YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE)

Figure 6-29 DIAGNOSE ALIAS with INCLUDE-USERCATALOG option

Error analysis
DIAGNOSE ALIAS looks for several types of discrepancies, for which fixes are
generally created, including:
User catalogs that are connected in one master catalog, but not in the other
Alias entries in one master catalog that are not in the other (under the rules of
the PEER and NONPEER keywords)
Aliases that exist, but do not have any data sets cataloged under them in the
related user catalog; these are considered empty aliases
Inconsistent aliases, where an alias related to a user catalog is defined in one
master catalog, but the same alias is defined in the other master catalog and
related to a different user catalog
User catalogs that have no aliases defined; effectively, these would be
unusable user catalogs, since it is not possible to direct cataloging of data
sets to them without an alias (this assumes you are not using
STEPCAT/JOBCAT statements, for which support is being dropped, and

98

ICF Catalog Backup and Recovery: A Practical Guide

should not be used). This situation is not considered an error, so a fix is not
created, but rather an information message is returned to alert you of this.
For more information on the fixes that are created for each of these situations,
refer to 7.2, Case study #2: BCS backup and forward recovery on page 116.
Note that all of the fixes in this case study were generated as comment
statements, and as such, are not directly executable as IDCAMS commands until
the /* at the beginning of each command is removed. Since the fix file is a
physical sequential file, the /* characters can be removed using TSO ISPF edit.
In many situations, this might be a laborious task, so the DIAGNOSE ALIAS
command has the DEFINE-ALIAS and DELETE-ALIAS keywords available. If
you know in advance which command is more appropriate for your environment,
the command processor builds those commands in the fix data set as
ready-to-execute.

6.6.2 DIAGNOSE BCS-VVDS and VVDS-BCS command

The DIAGNOSE command has two modes of operation for comparing entries in
one or more BCSs against one or more VVDSs, or one or more VVDSs against
one or more BCSs.

DIAGNOSE BCS-VVDS logic

One mode, BCS-VVDS, compares entries in the selected BCSs to the selected
VVDSs. In simplest terms, the primary function of this mode is to identify BCS
records that do not have matching and corresponding records in the VVDS. In
other words, this command is looking for orphaned BCS records, and for each
one found, an IDCAMS DELETE clustername NOSCRATCH command is
generated in the fix data set. There are other fix commands that may be
generated, but this is, by far, the most common one. All fix commands are
prefixed by a /*, making them comment statements. You have to either edit the /*
off with TSO EDIT, or you can specify the DELETE-ENTRIES keyword on the
DIAGNOSE command and all fixes are generated as ready-to-execute in the fix
data set.

DIAGNOSE VVDS-BCS logic

The other mode, VVDS-BCS, compares entries in the selected VVDSs to the
selected BCSs. In simplest terms, the primary function of this mode is to identify
VVDS records that do not have matching and corresponding records in the
appropriate BCS. In other words, this command is looking for orphaned VVDS
records. The solution and fix to this is more difficult, though, because there are
two possible interpretations:
The data set is one that you want access to and needs an IDCAMS DEFINE
CLUSTER ... RECATALOG command.

Chapter 6. Using Catalog RecoveryPlus

99

 The data set (or component) was deleted and has now re-appeared, in which
case, it needs to be physically deleted with an IDCAMS DELETE objectname
VVR/NVR command. Since the DIAGNOSE command has no way to know
which fix command is appropriate, it generates both in the fix data set and the
decision of which is correct is left to you. By default, both commands are
prefixed by a /*, to make them into comment statements. You have to either
edit the /* the desired fix command with TSO EDIT, or you can specify the
DEFINE-RECATALOG or DELETE-ENTRIES keyword on the DIAGNOSE
command if you know in advance which fix is appropriate.

In addition, the VVDS-BCS also compares VVDS records to their corresponding

VTOC records, to identify inconsistencies between those two structures, building
appropriate fix commands for any errors found.

Selecting BCSs and VVDSs to DIAGNOSE

In both modes of operation, one or more BCSs or VVDSs can be explicitly
specified, mask values can be specified, storage groups of volsers can be
specified, and even ** (double asterisk) can be specified. EXCLUDE keywords
are provided, so that individual BCSs/VVDSs or masked BCSs/VVDSs can be
removed from the selection list.
Possibly the most useful keyword is ALLRELATED, causing the DIAGNOSE
command processor to automatically determine all of the related VVDSs when
the compare is from BCS-to-VVDS, or related BCSs when the compare is from
VVDS-to-BCS. To determine the valid all related list, this facility does not search
for cataloged SYS1.VVDS.Vvolser entries in the BCSs, nor does it use the BCS
name list in the VVCR record in the VVDS. Instead, it searches through each and
every record in the BCS or VVDS, to identify volume pointers for VVDSs that data
sets reside on and BCS back-pointers in VVR/NVR records. Once the list is built,
the diagnosis is made to every BCS or VVDS in the list.
In Figure 6-30, the DIAGNOSE BCS-VVDS command diagnoses all BCSs (as
indicated by COMPAREBCS(**), against all related VVDSs. All errors identified
by this DIAGNOSE will have fixes created and stored in the FIXDATASET
specified in the command. See 7.7, Case study #7: Alternate master catalog on
page 135, for actual output from a sample execution of this command.

100

ICF Catalog Backup and Recovery: A Practical Guide

//YCJRES1X JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*

//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//ALLOC
DD DSN=YCJRES1.CRPLUS.DIAGNOSE.FIXFILE,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
//SYSIN
DD *
DEFINE CLUSTER(NAME(YCJRES1.CRPLUS.DIAGNOSE.JOURNAL) VOL(SBOX01) CYLINDERS(10 10) KEYS(94 0) SHR(1,3)) DATA(CISZ(4096) RECORDSIZE(240 240)) INDEX(CISZ(4096))
//S2 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
DIAGNOSE BCS-VVDS COMPAREBCS(**) ALLRELATED JOURNAL(YCJRES1.CRPLUS.DIAGNOSE.JOURNAL) FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE)

Figure 6-30 DIAGNOSE BCS-VVDS ALLRELATED example

In Figure 6-31, the DIAGNOSE VVDS-BCS command diagnoses all of our ITSO
sandbox volumes VVDSs (as indicated by COMPAREVVDS(SBOX*) keyword),
against all related BCSs. All errors identified by this DIAGNOSE will have fixes
created and stored in the FIXDATASET specified in the command. See 7.8,
Case study #8: Print and delete BCS/VVDS records on page 138, for actual
output from a sample execution of this command.

//YCJRES1X JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*

//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//ALLOC
DD DSN=YCJRES1.CRPLUS.DIAGNOSE.FIXFILE,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
//SYSIN
DD *
DIAGNOSE VVDS-BCS COMPAREVVDS(SBOX*) ALLRELATED FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE)

Figure 6-31 DIAGNOSE VVDS-BCS ALLRELATED example

Chapter 6. Using Catalog RecoveryPlus

101

6.7 MERGECAT command

The Catalog RecoveryPlus MERGECAT command provides functions that are
similar to the IDCAMS REPRO MERGECAT and NOMERGECAT commands. It
either moves or copies records from one BCS to another, allowing you to merge,
split, or clone BCS records.
In general, the MERGECAT command can be used to move all or some BCS
records to another BCS. Depending on the keywords that you code with this
command, aliases may be changed from the input BCS to the output BCS.
When not using the COPYONLY keyword, the MERGECAT command performs
as a move function, and therefore, VVDS records for all data sets whose catalog
entry is moved will have a BCS back-pointer update. To achieve maximum
performance in this operation, the BCS back-pointer updates are saved until the
end of processing and are then performed a volume at a time, so that all updates
to each VVDS are done at once. This is unlike the logic in IDCAMS REPRO
MERGECAT, where the VVDS updates are done at the time each record is
moved from the source to the target BCS. This logic is one of the primary factors
in the performance obtained by the CR+ MERGECAT command.
Figure 6-32 illustrates the command used for moving all records in a BCS to
another BCS.

MERGECAT INBCS(UCAT.CRPLUS3) OUTBCS(UCAT.CRPLUS5) JOURNAL(YCJRES1.MERGECAT.JOURNAL)

Figure 6-32 MERGECAT command to move all BCS records

Every catalog record in UCAT.CRPLUS3 is moved to UCAT.CRPLUS5, and every

alias for UCAT.CRPLUS3 in the master catalog is moved to UCAT.CRPLUS5. At
completion of this command, UCAT.CRPLUS3 is an empty catalog. For every
data set whose catalog record is moved, all the VVDS records (VVRs and NVRs)
change their BCS back-pointer value from UCAT.CRPLUS3 to UCAT.CRPLUS5.
In Figure 6-33, you see the command used to move all records with a high-level
qualifier of YCJRES3, from user catalog UCAT.CRPLUS3 to UCAT.CRPLUS5. By
using the level keyword, the alias YCJRES3 is also changed in the master
catalog from its relationship to UCAT.CRPLUS3 to UCAT.CRPLUS5.

102

ICF Catalog Backup and Recovery: A Practical Guide

MERGECAT INBCS(UCAT.CRPLUS3) OUTBCS(UCAT.CRPLUS5) LEVEL(YCJRES3) JOURNAL(YCJRES1.MERGECAT.JOURNAL)

Figure 6-33 MERGECAT command to move BCS records by LEVEL

Finally, in Figure 6-34, all catalog entries in UCAT.CRPLUS3 that begin with
ITSO1 (regardless of anything else in their name) are moved to UCAT.CRPLUS5.
This type of process is most likely done when entries such as these are
cataloged incorrectly into the source BCS and need to be moved to the correct
BCS (and even more likely, actually has the alias of ITSO1). With the ENTRIES
keyword, the MERGECAT command does not make any alias changes.

MERGECAT INBCS(UCAT.CRPLUS3) OUTBCS(UCAT.CRPLUS5) ENTRIES(ITSO1.**) JOURNAL(YCJRES1.MERGECAT.JOURNAL)

Figure 6-34 MERGECAT command to move BCS records by ENTRIES

6.7.1 Alias handling

A unique feature of the CR+ MERGECAT command is that it re-associates user
catalog aliases to the target catalog. This only happens when the move is
specified with the LEVEL keyword, and therefore, only when an entire alias
group of records is moved.
The significance of the emphasis is, if you specify the following LEVEL example,
you will successfully move all catalog records that match those two levels of data
set name qualification:
LEVEL(YCJRES1.CRPLUS2)

But, if your system is running with MLA=1 (multi-level alias support of 1), you
would only have the high-level qualifier of YCJRES1 set up as an alias, and it will
not be re-associated to the target catalog. The reason is that you may well have
many other data sets under this same high-level qualifier, such as
YCJRES2.CRPLUS1, and you would make these data sets inaccessible if the
alias of YCJRES1 was re-associated. In fact, by running the example above, you
actually orphan the data sets with high-level qualifier YCJRES1.CRPLUS2, as
they are undoubtedly now located in a BCS that does not (and cannot) have an
alias of YCJRES1. This type of operation should only be done when you are

Chapter 6. Using Catalog RecoveryPlus

103

splitting catalogs apart, and most likely, getting ready for multi-level alias support.
Or you should do it when you are performing system consolidations or workload
balancing where you are moving whole categories of data sets over to user
catalogs that are accessible from different systems.
When you use the ENTRIES keyword, aliases are never re-associated, even if
you move all catalog records for an entire alias group.

6.7.2 Using the journal file

MERGECAT uses a journal file, in part to provide RESTART or BACKOUT
capability if the command should fail during processing. In the event of failure,
you can re-run the MERGECAT command, coded exactly as before, and add
either the RESTART or BACKOUT keyword (see the following section (6.7.3) for
details of the RESTART and BACKOUT keywords).
Also, the CR+ MERGECAT command obtains its tremendous speed and
reliability by using its journal file. As catalog records are moved from the source
to target BCS, the associated VVDS catalog back-pointers in each VVR/NVR
are not updated to the new target BCS name at that time. Instead, information
about each catalog record is stored in the journal file. Upon completion of all
catalog records that are to be moved, the back-pointers are changed in a single
mass update to the VVDSs.
It is common for users to automatically place the DELETE and DEFINE of the
journal file in a step immediately in front of the MERGECAT command, since
MERGECAT needs a clean journal each time it runs. Therefore, it is important
to clearly recognize that this operation must not be done in a RESTART or
BACKOUT situation, allowing the command to use the journal records from the
previous, interrupted run.

6.7.3 RESTART and BACKOUT capability

A second powerful use of the journal file is to provide the ability to restart and
back out a failed MERGECAT operation, simply by running the same command
again, but with the RESTART or BACKOUT keyword coded:
If RESTART is specified, the command resumes at the point of processing
indicated by the current status of the journal file. Assuming the problem is
corrected that caused the interruption, the MERGECAT process should
continue to completion.
If BACKOUT is specified, the command resumes at the point of processing
indicated by the current status of the journal file, but it re-runs all of the
already-completed operations to that point, in reverse processing order, and
undoes each operation.

104

ICF Catalog Backup and Recovery: A Practical Guide

The RESTART or BACKOUT operation is possible if you receive the following

messages in the SYSOUT report on the failing or interrupted job:
CAT12000I
CAT12000I

>>> JOURNAL FILE HAS BEEN POPULATED

>>> MERGECAT IS NOW RESTARTABLE

6.7.4 SIMULATE capability

The SIMULATE keyword provides the ability to test the command in advance,
ensuring that it performs exactly what you expect it to do. The output report from
the command looks exactly as it would if you ran it for real, but there are no I/Os
to the output BCS, aliases are not moved, and VVDS back-pointer updates are
not performed.
That makes it almost imperative for the actual move window to be as short as
possible. The last thing you can tolerate is a failure of the MERGECAT,
particularly if it is due to a command coding failure on your part. The SIMULATE
capability enables you to run the MERGECAT in advance, in a simulation mode,
so that you can see exactly what the command is going to accomplish. Since the
CR+ MERGECAT command provides a line-by-line process report of every data
set moved (which IDCAMS REPRO MERGECAT does not), you can verify
exactly what data sets are to be moved, and information about any and all user
catalog aliases that are to be re-associated to the target catalog. The simulations
can be done as far in advance as you want. When you feel comfortable that your
command is performing exactly as you want, you then have the confidence that it
should work when you do it for real within your downtime window.

6.7.5 Catalog activity during MERGECAT processing

In many instances, particularly with user catalog merging and splitting of records,
you want to run MERGECAT when the source catalog is quiesced. In other
words, all data set records being moved or copied must be in a closed state. Job
termination problems almost certainly occur if you move catalog records from
one BCS to another while the actual data set for that catalog record is open to a
processing job (because CAS has to write CLOSE information to the catalog,
and its in-storage control blocks only know about where the catalog record came
from, not where its now supposed to go). This is a particular problem for online
data sets, which are open for long periods of time, requiring you to carefully
schedule down time when these catalog records to be moved are in a quiesced
condition.
To help you control this, MERGECAT has the option to internally issue an ALTER
LOCK command at the beginning of processing:

Chapter 6. Using Catalog RecoveryPlus

105

 If the LOCK keyword is specified (or left to default), MERGECAT internally

issues an IDCAMS ALTER LOCK command to both the source and target
BCSs, ensuring serialized access throughout execution.
If NOLOCK keyword is specified, MERGECAT does not lock the source and
target BCSs at the start of move or copy processing, allowing other catalog
access within the BCS during the time that MERGECAT is performing its
operations.

6.7.6 Creating an alternate master catalog

The CR+ MERGECAT command has a keyword COPYONLY, which provides the
ability to copy records from one BCS to another. While this can be used in some
situations for setting up test catalogs, it is most frequently used to create and
maintain an alternate master catalog. The command is similar to IDCAMS
REPRO NOMERGECAT, with some major differences. The most notable
difference is, with IDCAMS REPRO NOMERGECAT the BCS back-pointers in
each copied VVDS record copied are updated. With CR+ MERGECAT
COPYONLY, the BCS back-pointers are not updated, even if you specifically
code the VVDSUPDATE keyword on the command. Other major differences
include the ability to RESTART, BACKOUT, and SIMULATE in this mode.
The implication of the VVDS back-pointer update rule above is that, when the
CR+ MERGECAT COPYONLY operation is used to create an alternate master
catalog, the source remains your official production master catalog, and the
target becomes your alternate master catalog. With IDCAMS REPRO
NOMERGECAT, where the VVDS back-pointer entries are re-associated, this
makes the target the new official master catalog, and the source is now the
alternate master catalog, requiring you to re-IPL after the command is completed
to make the change effective. If you are doing this on a regular basis as you
perform master catalog updates, you are forced to flip-flop back and forth with
respect to which master catalog is the official one and which is the alternate. With
the logic in CR+ MERGECAT COPYONLY, your production master catalog
remains the same at all times, and you continually create a new alternate master
with this command.

6.8 SUPERCLIP command

Catalog RecoveryPlus provides the SUPERCLIP command for changing the
volser of disk volumes.

106

ICF Catalog Backup and Recovery: A Practical Guide

6.8.1 Changing a volser

Changing a disk volser with the ICKDSF utility is an operation that is quite
regularly performed in many installations. It is commonly referred to as a volser
clip (which is an old, early S/360 days term meaning change label in process,
but has survived in everyday use, even though many people do not have any idea
what it stands for). It is most commonly done on empty volumes.
Changing a volser of a volume that has data sets already allocated on it is a bit
more troublesome. With standard MVS utility functions, you must first move all
data sets off the volume, by copying the data and then deleting the data set. You
then VARY the volume offline to each system that is online to it. You run ICKDSF
to clip the volser to its new value. Once you issue VARY to bring the volume
online, you can allocate and reload the data sets on the volume.
With the Catalog RecoveryPlus SUPERCLIP command, you simply code a
command as shown in Figure 6-35. SUPERCLIP then performs the entire volser
change for you, without unloading and reloading the data sets on the volume.
The SUPERCLIP command uses a journal file, and this provides you with a
process log of all steps that SUPERCLIP performs, as well as providing a
RESTART, BACKOUT, and SIMULATE capability.

SUPERCLIP OLDVOLSER(oldvol) NEWVOLSER(newvol) JOURNAL(YCJRES1.SUPERCLP.JOURNAL)

Figure 6-35 SUPERCLIP example syntax

You can see the complete output from the SUPERCLIP command in 7.6, Case
study #6: Changing a disk volser on page 131.

6.8.2 Caution when using SUPERCLIP

A couple of caveats should be mentioned about SUPERCLIP. It is an extremely
powerful facility, but one that must be used with utmost caution and should be
considered for a RACF FACILITY class profile definition that restricts who can
use this command. Also, you must make certain that the volume is already varied
offline to all other systems that share the device, because SUPERCLIP does not
issue the VARY to systems other than the one on which the command is running.

6.9 ZAP command

The Catalog RecoveryPlus ZAP command, while sounding a lot like AMASPZAP,
is actually a multi-purpose command that provides the following capabilities:

Chapter 6. Using Catalog RecoveryPlus

107

Print one or more BCS records

Delete a BCS record
Patch a BCS record
Print one or more VVDS records
Delete a VVDS record
Patch a VVDS record

The variations in specification of each capability are quite wide. Refer to Mainstar
Catalog RecoveryPlus User Guide, document number 006-0202, for specific
details of all possible ways that the commands can be coded, as well as details of
exactly how each process works.

6.9.1 Printing BCS and VVDS records

You can print one or more BCS or VVDS records with the ZAP command.
Figure 6-36 shows examples of how you might code this command. In each
example shown, you can specify the output to be in FORMAT, DUMP, or
CELLDUMP layout.

To print a single BCS record, by key specification:

ZAP BCS(UCAT.CRPLUS1) PRINT(KEY(YCJRES1.CRPLUS1.KSDS.CLUSTER))
To print 5 BCS records, in ascending key sequence, by key specification:
ZAP BCS(UCAT.CRPLUS1) PRINT(KEY(YCJRES1.CRPLUS1.KSDS.CLUSTER) COUNT(5))
To print 5 BCS records, by generic key specification:
ZAP BCS(UCAT.CRPLUS1) PRINT(KEY(YCJRES1.CRPLUS1.K**) COUNT(5))
To print a single VVDS record, by component name specification:
ZAP VVDS(SBOX01) PRINT(COMPONENT(YCJRES1.CRPLUS1.KSDS.DATA))
To print a single VVDS record, by RBA offset:
ZAP VVDS(UCAT.CRPLUS1) PRINT(RBA(000096DE))
To print 5 VVDS records, in physical sequence, starting at a component name location:
ZAP VVDS(SBOX01) PRINT(KEY(YCJRES1.CRPLUS1.KSDS.CLUSTER) COUNT(5))

Figure 6-36 ZAP PRINT of BCS and VVDS records

6.9.2 Deleting a BCS and VVDS record

You can DELETE a BCS or VVDS record with the ZAP command. You can only
delete one record in each execution of the command. It is also very important to
understand that this command, if successful, deletes only the single record that
you point it to and does not delete any associated records. For example, if you
delete a BCS cluster sphere record, it does not delete any associated truename

108

ICF Catalog Backup and Recovery: A Practical Guide

records for the clusters components (if they exist). If you delete a VVDS record
(a VVR or NVR), it does not delete an associated VTOC record (as an IDCAMS
DELETER VVR/NVR command does). Therefore, you should be careful when
executing this command, to ensure that you do not cause problems greater than
you are trying to solve. Figure 6-37 shows examples of how you may code this
command.

To delete a BCS record, by key specification:

ZAP BCS(UCAT.CRPLUS1) DELETE(KEY(YCJRES1.CRPLUS1.KSDS.CLUSTER))
To delete a BCS record, but only if it is verified to contain a certain value at a
specified location within the record:
ZAP BCS(UCAT.CRPLUS1) DELETE(KEY(YCJRES1.CRPLUS1.KSDS.CLUSTER) VER(04,C4))
To delete a single VVDS record, by component name specification:
ZAP VVDS(SBOX01) DELETE(COMPONENT(YCJRES1.CRPLUS1.KSDS.DATA))
To delete a VVDS record, by RBA offset within the VVDS, but only if it is verified to
contain a certain value at a specified location within the record:
ZAP VVDS(SBOX01) DELETE(RBA(000096DE) VER(04,F9))
To delete a VVDS record, at an RBA offset within the VVDS:
ZAP VVDS(SBOX01) DELETE(COMPONENT(YCJRES2.CRPLUS.TESTK.K000953.DATA) - RBA(0005C000))

Figure 6-37 ZAP DELETE of a BCS and VVDS record

6.9.3 Patching a BCS and VVDS records

You can PATCH a BCS or VVDS record with the ZAP command. You can only
patch one record in each execution of the command. You should be very careful
when executing this command, because it is extremely powerful, and you are
essentially on your own whenever you zap a BCS or VVDS record.
Figure 6-38 shows examples of how you might code this command.

Chapter 6. Using Catalog RecoveryPlus

109

To patch a BCS record, but only if both VER and REP conditions are met:
ZAP BCS(UCAT.CRPLUS2) PATCH(COMPONENT(YCJRES2.CRPLUS.TESTK.K000954) VER(04,X'E3',37,X'24',58,X'F1')
REP(58,X'F2'))
To patch a VVDS record, but only if both VER and REP conditions are met:
ZAP VVDS(SBOX79) PATCH(COMPONENT(YCJRES2.CRPLUS.TESTK.K000954.DATA) VER(04,X'E9',1F,X'E3E2E3E1')
REP(1F,X'E3E2E3E4))

Figure 6-38 ZAP PATCH of a BCS and VVDS record

110

ICF Catalog Backup and Recovery: A Practical Guide

Chapter 7.

Case studies
This chapter presents several real-life examples of how ICF catalog structures
can break. The studies provide actual job listings that contain specific
command sequences that can be used to analyze each situation and to recover
from the errors.
The case studies in this chapter use commands and facilities from IDCAMS,
DFSMSdss, the IBM Integrated Catalog Forward Recovery Utility (ICFRU),
VVDSFIX (an unsupported ad hoc tool from IBM), and Mainstar Catalog
RecoveryPlus (CR+). The commands in the case studies include:

IDCAMS EXPORT and IMPORT

DFSMSdss DUMP and RESTORE
IDCAMS EXAMINE and DIAGNOSE
ICFRU BCS forward recovery
VVDSFIX DELBCSR and DELVVR
CR+ BACKUP and RECOVER
CR+ DIAGNOSE

The information in this chapter is not intended to be all-encompassing. The ways

that ICF catalogs can break is without limit. Even in two similar breakage
situations, the symptoms can be different and manifest themselves in different
ways. The case studies are intended to be examples of various ways that ICF
catalog failures and breakages can occur, so that you can hopefully develop a
methodology and sequence of analysis to use when debugging ICF catalog and
catalog address space problems.

Copyright IBM Corp. 1999, 2001

111

The case studies that are examined in this chapter include:

112

Case study #1: Ensuring good backups on page 113

Case study #2: BCS backup and forward recovery on page 116
Case study #3: Synchronizing aliases on page 121
Case study #4: VVDS backup and forward recovery on page 125
Case study #5: Enlarging VVDS on page 129
Case study #6: Changing a disk volser on page 131
Case study #7: Alternate master catalog on page 135
Case study #8: Print and delete BCS/VVDS records on page 138
Case study #9: Preventive maintenance on page 143

ICF Catalog Backup and Recovery: A Practical Guide

7.1 Case study #1: Ensuring good backups

This case study illustrates the importance of ensuring that the BCS you are
backing up is structurally sound and that all records from the actual BCS are
contained on the backup file.
Example 7-1 begins with an IDCAMS VERIFY command that does not find
anything out of the ordinary with the BCS. The subsequent IDCAMS EXAMINE
command finds structural errors in its analysis of the data and the index
component. This includes what it considers to be a minor index error and a
major data error, where there is a data component CA that is apparently not
indexed by any level 1 sequence set record. The IDCAMS DIAGNOSE command
that follows does not find any data content errors. The fact that the EXAMINE
command raised a condition code 8 should alert you that this BCS is broken in
some way, either localized around a specific area of the BCS or catastrophically.
Regardless of what occurs in the following steps of this case study, we should
plan to fix whatever is broken in this catalog, as soon as possible.

Example 7-1 Verifying the structure and content of the BCS

//YCJRES1X JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
VERIFY DATASET(UCAT.VSBOX09)
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
EXAMINE NAME(UCAT.VSBOX09) DATATEST
IDC01700I INDEXTEST BEGINS
IDC11724I DATA COMPONENT CA NOT KNOWN TO SEQUENCE SET
IDC21700I MINOR ERRORS FOUND BY INDEXTEST
IDC01701I DATATEST BEGINS
IDC11724I DATA COMPONENT CA NOT KNOWN TO SEQUENCE SET
IDC21703I MAJOR ERRORS FOUND BY DATATEST
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 8
DIAGNOSE ICFCATALOG INDATASET(UCAT.VSBOX09)
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 8

Chapter 7. Case studies

113

Example 7-2 illustrates an EXPORT backup of this BCS, run immediately after
the DIAGNOSE step in Example 7-1. Note that the EXPORT output indicates the
number of BCS records backed up is 4,722. Condition code 0 indicates that no
errors were encountered.

Example 7-2 Backing up the BCS with EXPORT

//YCJRES1X JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//OUT1
DD DSN=YCJRES1.EXPORT.BACKUP2,DISP=(,CATLG,DELETE),
//
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE)
//SYSIN
DD *
EXPORT UCAT.VSBOX09 OUTFILE(OUT1) TEMPORARY
IDC0005I
IDC0594I
IDC1147I
IDC1147I
IDC0001I
IDC0002I

NUMBER OF RECORDS PROCESSED WAS 4722

PORTABLE DATA SET CREATED SUCCESSFULLY ON 09/07/01 AT 20:25:02
IT IS RECOMMENDED THAT DIAGNOSE AND EXAMINE BE RUN BEFORE
IMPORT OF CATALOG
FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0

Example 7-3 illustrates a Catalog RecoveryPlus BACKUP command for this

BCS. The most significant aspect to note in the command output is that it
indicates a total of 26,207 records backed up. The EXPORT output in
Example 7-2 indicates that 4,722 records backed up on the same BCS and with
a condition code of 0.

Example 7-3 Backing up the BCS with CR+ BACKUP

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//OUT1
DD DSN=YCJRES1.VSBOX09.BACKUP2,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN
DD *
BACKUP BCS(UCAT.VSBOX09) OUTFILE(OUT1)
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 07 SEP 2001 PAGE 1
CAT02016I THE FOLLOWING DSNS/MASKS ARE TO BE PROCESSED DURING BCS PROCESSING.
UCAT.VSBOX09
CAT02031I BASED ON THE MASKS/DSNS, THE FOLLOWING CATALOGS HAVE BEEN SELECTED.

114

ICF Catalog Backup and Recovery: A Practical Guide

UCAT.VSBOX09
CAT02045I START OF DATA UNLOAD FOR UCAT.VSBOX09 ON 07 SEP 2001 (2001250) AT
20.27.15 VER 1.1.00
CAT02092I CALLING IDCAMS TO PERFORM EXAMINE
IDC01700I INDEXTEST BEGINS
IDC11724I DATA COMPONENT CA NOT KNOWN TO SEQUENCE SET
IDC21700I MINOR ERRORS FOUND BY INDEXTEST
CAT02050W IDCAMS (4) ISSUED RETURN CODE 4
CAT02046I RECORD SUMMARY FOR UCAT.VSBOX09
CLUSTER GDG E J
NONVSAM TRUENAME PATH UCAT ALIAS OTHER
TOTAL TOTAL BYTES
6,587
1 0 0
6,530
13,089
0
0
0
0
26,207
3,123,309
CAT02051I NUMBER OF SPANNED RECORDS ENCOUNTERED: 0
CAT02047I END OF DATA UNLOAD FOR UCAT.VSBOX09 ON 07 SEP 2001 AT 20.27.18
CAT02009I BACKUP FUNCTION COMPLETE. RETURN CODE 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

The difference is that EXPORT treats the BCS like a KSDS, using the index
component as its reference vehicle to the data records. If there is an error in the
index that is not detected by EXPORT, the result could be an incomplete or
incorrect backup. Catalog RecoveryPlus BACKUP treats the BCS like an
ESDS, bypassing all reference to the index as it backs up the data records in
physical sequence, For this type of BCS error, this logic provides greater
insurance that all records are backed up.
One final detail should be clarified in the output report in Example 7-3. Note that
it indicates a return code 4 from the internally-issued IDCAMS EXAMINE. Recall
that the explicitly-issued EXAMINE, in the Example 7-2 output, contained the
keyword DATATEST, which causes EXAMINE to actually run both an
INDEXTEST and a DATATEST, and it returned a condition code of 8. By default,
the CR+ BACKUP command in Example 7-3 internally issues an EXAMINE
command, with only the keyword INDEXTEST. As you can see from the output in
Example 7-2, the INDEXTEST logic only considered the error to be minor. The
DATATEST logic considered the error to be major. Apparently, this results in a
return code of 4 from INDEXTEST and a return code of 8 from DATATEST, and
only the highest return code is passed back.
Note: This case study does not imply that IDCAMS EXPORT is likely to
produce a bad BCS backup and Catalog RecoveryPlus BACKUP always
produces a good one. In other circumstances, it could happen that CR+
BACKUP has problems and EXPORT does not. Instead, this case study
illustrates the importance of frequent and regular diagnostics on your entire
ICF catalog environment, and then carefully checking the output listings when
there is any indication of problem. It also points out the benefit of using two
different catalog backup methods, increasing your chances that your backup
will be valid when the all-important time comes to restore and recover it.

Chapter 7. Case studies

115

7.2 Case study #2: BCS backup and forward recovery

This case study demonstrates how to use the Catalog RecoveryPlus BACKUP
and RECOVER commands to forward recover a BCS.
Example 7-4 shows a CR+ BACKUP command of the specific BCS, named
UCAT.CRPLUS2, taken at 14:23:11 on 21 September. In a production
environment, it might be more appropriate to backup all BCSs in a single step, by
specifying the keyword as BCS(**), resulting in every BCS in the entire
environment being backed up at once.
An IDCAMS EXAMINE, with the INDEXTEST keyword, is specified (which is also
the default), and the result of this structural diagnostic test of the BCSs index
component shows that it has no errors.
Our backup of the BCS then runs, writing a total of 7,838 catalog records to the
backup file on the DD statement OUT1. This is set up as a GDG, so we identify it
as our newest (+1) generation of the GDG.

Example 7-4 Backing up the BCS with CR+ BACKUP

//YCJRES1A JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//OUT1
DD DSN=YCJRES1.CRPLUS2.BKUP(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
BACKUP BCS(UCAT.CRPLUS2) OUTFILE(OUT1) EXAMINE(INDEXTEST)
/*
FAC01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 21 SEP 2001 PAGE:1
CAT02016I THE FOLLOWING DSNS/MASKS ARE TO BE PROCESSED DURING BCS PROCESSING.
UCAT.CRPLUS2
CAT02031I BASED ON THE MASKS/DSNS ENTERED, THE FOLLOWING CATALOGS HAVE BEEN SELECTED.
UCAT.CRPLUS2
CAT02045I
CAT02092I
IDC01700I
IDC01724I

START OF DATA UNLOAD FOR UCAT.CRPLUS2 ON 21 SEP 2001 AT 14.23.11

CALLING IDCAMS TO PERFORM EXAMINE
INDEXTEST BEGINS
INDEXTEST COMPLETE - NO ERRORS DETECTED

CAT02046I RECORD SUMMARY FOR UCAT.CRPLUS2

CLUSTER GDG E J NONVSAM TRUENAME PATH UCAT ALIAS
2,831
15 0 0
1,273
3,719
0
0
0
CAT02051I NUMBER OF SPANNED RECORDS ENCOUNTERED: 0

116

ICF Catalog Backup and Recovery: A Practical Guide

OTHER
0

TOTAL
7,838

TOTAL BYTES
1,011,303

CAT02047I

END OF DATA UNLOAD FOR UCAT.CRPLUS2 ON 21 SEP 2001 AT 14.23.12

CAT02009I BACKUP FUNCTION COMPLETE. RETURN CODE 0

CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

Following the BACKUP step, considerable processing continued on the system,

including a significant number of DEFINEs, DELETEs, and updates to the data
sets in UCAT.CRPLUS2.
On the assumption that we have later a catastrophic error in the BCS named
UCAT.CRPLUS2, we now prepare for a forward recovery of this BCS. To do this,
we must identify all processing that is currently running through this BCS and
take whatever steps are necessary to quiesce that activity. This could mean
waiting until all current jobs that use this BCS are completed or taking the more
drastic action of canceling them from the system. If we allow currently executing
jobs to continue to completion, one or more may not terminate cleanly, as a result
of errors when data sets cataloged in UCAT.CRPLUS2 are closed. If that occurs,
we need to be prepared for job recovery. We also must stop any new jobs from
initiating execution, through the control of our job scheduler. Note, if you are
allowing currently executing jobs to complete execution, issuing an IDCAMS
ALTER LOCK on the BCS will definitely result in CLOSE failures when these jobs
attempt to complete.
Next, we need to identify the most recent backup that was taken of this BCS.
Since we set up a GDG for our BCS backups, we point to the current (0)
generation in our recovery JCL (see the INFILE1 DD statement in Example 7-5,
where we reference the (0) generation).
Then, we take all steps necessary to gather up all of the SMF data that is
required for the recovery. In our ITSO test environment, we have three systems
where we ran our test jobs, identified as SC63, SC64, and SC65. When we know
that all processing is quiesced, on all three systems that have jobs processing
through UCAT.CRPLUS1, we switch from the current SMF log file to the other, by
issuing the operator command I SMF on each of the three systems. The
switched SMF data collection file must then be dumped. In all likelihood, the
most difficult task of the recovery process must now be done gathering up all of
the dumped SMF files, from all three systems, for the entire time period between
when the most recent BACKUP was taken and now. The less frequently we run
BACKUPs of our BCS, the more systems we have that share the BCS, and how
good the procedures are that help us in this task, are all going to be significant
factors in how easy or difficult this part of the process is. In any case, when we
gather up all of the necessary SMF data, we create a JCL that concatenates
every dump file on the SMFINDD statement in our RECOVER job.

Chapter 7. Case studies

117

Example 7-5 shows the complete RECOVER BCS job, with a RECOVER
command. This command indicates the BCS that is to be restored, where the
backup copy is located, and that we want to perform a forward recovery using
SMF data that is included on the SMFINDD JCL statement.
As the processing begins, the defaults in effect tell RECOVER that alias records
from the backup file are to be applied to the current system master catalog and
that all SMF records, from the exact time of backup to the time of this forward
recovery, are to be applied to the backup records. The PRINT options message
indicates that we do not want a detailed listing of all restored records. If you want
to change this, there is a PRINT keyword on the RECOVER command. See
6.5.8, PRINTing BCS records during a RECOVER on page 93, for details of
what the PRINT keyword shows you.

Example 7-5 BCS forward recovery with CR+ RECOVER

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//INFILE1 DD DSN=YCJRES1.CRPLUS1.BACKUP(0),DISP=SHR
//SMFINDD DD DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.SC65.SMFDATA,DISP=SHR
...plus the DD statements for RCVDD1, RCVMRG, RCVSMF, SMFERR, and others
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
RECOVER BCS(UCAT.CRPLUS2) INFILE(INFILE1) FORWARD(SMFFILE(SMFINDD))
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 21 SEP 2001 PAGE:1
CAT04079I
CAT04079I
CAT04079I
CAT04080I

DEFAULT IN EFFECT: ALIAS

DEFAULT IN EFFECT: FROMBACKUPDATE
DEFAULT IN EFFECT: TOCURRENTDATE
PRINT OPTION IN EFFECT: NONE

CAT04029I 15.50.22 RECOVERY STEP: CHECK BCS STATUS - STARTED

CAT04029I 15.50.22 RECOVERY STEP: CHECK BCS STATUS - COMPLETED
CAT04029I 15.50.22 RECOVERY STEP: LOCK BCS - STARTED
CAT04029I 15.50.22 RECOVERY STEP: LOCK BCS - COMPLETED
CAT04029I 15.50.22 RECOVERY STEP: PROCESSING BACKUP - STARTED
CAT04035I RECOVERY STARTED FOR BCS: UCAT.CRPLUS2
BACKUP DATE/TIME: 2001264/14:23:11
CAT04029I 15.50.24 RECOVERY STEP: PROCESSING BACKUP - COMPLETED
CAT04029I 15.50.24 RECOVERY STEP: PROCESSING SMF - STARTED

118

ICF Catalog Backup and Recovery: A Practical Guide

CAT04019I PROCESSING SMF RECORDS

FROM DATE/TIME: 2001264/14:23:11
TO DATE/TIME: 2001264/15:50:22
CAT04020I SMF RECORDS SELECTED:
1689
CAT04021I SMF RECORDS FOR SYSID: SC63
EARLIEST: 2001263/16:21:47
LATEST: 2001264/15:19:41
CAT04021I SMF RECORDS FOR SYSID: SC64
EARLIEST: 2001263/16:21:47
LATEST: 2001264/15:19:38
CAT04021I SMF RECORDS FOR SYSID: SC65
EARLIEST: 2001263/23:30:00
LATEST: 2001264/14:49:58
CAT04026W ******** GAP IN TIME FOR SYSID: SC63 LATEST TIME GAP
CAT04026W ******** GAP IN TIME FOR SYSID: SC64 LATEST TIME GAP
CAT04026W ******** GAP IN TIME FOR SYSID: SC65 LATEST TIME GAP
CAT04027I RECORD COUNTS FROM APPLYING SMF DATA
7580 UNCHANGED; 4 DELETED;
250 INSERTED;
4 UPDATED
CAT04029I 15.50.26 RECOVERY STEP: PROCESSING SMF - COMPLETED
CAT04029I 15.50.26 RECOVERY STEP: REWRITE TO BCS - STARTED
CAT04029I 15.50.35 RECOVERY STEP: REWRITE TO BCS - COMPLETED
CAT04029I 15.50.35 RECOVERY STEP: UNLOCK OF BCS - STARTED
CAT04029I 15.50.35 RECOVERY STEP: UNLOCK OF BCS - COMPLETED
CAT04029I
CAT04038I
CAT04038I
CAT04039I
CAT04029I

15.50.35 RECOVERY STEP: APPLYING ALIASES - STARTED

ALIAS NAMES FROM BACKUP; TOTAL: 1
YCJRES2
ALIASES APPLIED TO CATALOG: MCAT.SANDBOX.R10.VSBOX11
15.50.35 RECOVERY STEP: APPLYING ALIASES - COMPLETED

CAT04036I RECOVERY COMPLETED FOR BCS: UCAT.CRPLUS2

CAT04009I RECOVER PROCESS COMPLETED; RETURN CODE: 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

The RECOVER output listing is essentially a process log of all steps that the
command performs, including:
CHECK BCS STATUS: In this process, RECOVER determines if a BCS of the
name UCAT.CRPLUS2 is connected to the master catalog and physically
exists on the system. If one is found, a DELETE (with RECOVERY) and
DEFINE USERCATALOG command is submitted under the covers to
IDCAMS, so that the BCS into which the backup records is written is a brand
new, empty BCS (see 6.5.3, Explicit/implicit DEFINE of the BCS or VVDS to
restore on page 87, if you want to perform your own DELETE/DEFINE
command in preparation for RECOVER).
LOCK BCS: An IDCAMS ALTER command is issued, for the BCS named
UCAT.CRPLUS2, with the keyword LOCK specified. This makes the BCS

Chapter 7. Case studies

119

inaccessible to all users, including users sharing the catalog on other

systems. The LOCK is removed as one of the last steps in RECOVER
command processing.
PROCESSING BACKUP ... RECOVERY STARTED FOR BCS: UCAT.CRPLUS2: This
process reads the backup file. If there were multiple BCS backups contained
on this file, it spins down to locate the backup records for UCAT.CRPLUS2.
When they are located, it identifies the date/time of the backup, for use in the
SMF record extract, and is now positioned and ready to restore and forward
recover.
PROCESSING SMF: This step performs the forward recovery with the SMF data,
including the extraction of all SMF records between the backup and restore
times, sorting the records into ascending time sequence, and then applying
all updates to the backup records. If gaps in time of more than 10 minutes
between SMF update records for the BCS are detected, an informational
message is printed to alert you to the potential that dumped SMF records
from one or more systems may not be included in the process.
REWRITE TO BCS: This step writes the updated (forward recovered) BCS
records to the output BCS.
UNLOCK OF BCS: An IDCAMS ALTER for the BCS, with UNLOCK, is now
issued.
APPLYING ALIASES: All aliases for the BCS are then restored to the master
catalog for the system on which the RECOVER command is run.
RECOVERY COMPLETED FOR BCS: UCAT.CRPLUS2 ... RETURN CODE: 4: Indicates
the successful completion of the forward recovery, and the return code value
of 4 in this instance is due to the gap in SMF records.

Upon completion of this forward recovery of the BCS named UCAT.CRPLUS2, it

is a good idea to run the following cleanup steps:
1. Run an IDCAMS EXAMINE INDEXTEST DATATEST command to ensure
there are not any structural problems with either the index or data component
of the newly recovered BCS.
2. Run a DIAGNOSE ICFCATALOG command, without the COMPAREDD
operand, to double-check that the BCS does not contain any data content
errors.
3. Depending on how quickly this BCS needs to be back in production, it may
also be prudent to run the following two CR+ DIAGNOSE commands, to
ensure there are no errors between the BCS and its related VVDSs (see
6.6.2, DIAGNOSE BCS-VVDS and VVDS-BCS command on page 99, for
more details of how to code these commands):
DIAGNOSE BCS-VVDS COMPAREBCS(UCAT.CRPLUS2) -

120

ICF Catalog Backup and Recovery: A Practical Guide

ALLRELATED JOURNAL(YCJRES1.CRPLUS.DIAGNOSE.JOURNAL) FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE)

DIAGNOSE VVDS-BCS COMPAREVVDS(SBOX*) TOBCS(UCAT.CRPLUS2) FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE)

4. You should run a CR+ DIAGNOSE ALIAS command to ensure that all aliases
related to this newly restored user catalog are consistent across all system
master catalogs. Case study #3: Synchronizing aliases on page 121 shows
you how to code this command and how to interpret the output and fix file
created with this command. If we ran the forward recovery on system SC63,
we would have to run the DIAGNOSE ALIAS command twice, the first time
specifying SC63 and SC64 as our two master catalogs, and the second time
specifying SC63 and SG65 as our two master catalogs. Presumably, you
would run this in NONPEER mode.
If all of the steps in this case study were fully tested and practiced on a regular
basis, the whole process should not take more than a few minutes. If it was not
tested, and you do not have good procedures, for example, in collecting the
required SMF data, it could be a long and painful process. To accomplish this
testing, the RECOVER command illustrated in this case study could be run at
any time with the SIMULATE keyword.

7.3 Case study #3: Synchronizing aliases

This case study demonstrates how to use the Catalog RecoveryPlus DIAGNOSE
ALIAS command to synchronize the user catalog aliases in two master catalogs.
Example 7-6 shows the CR+ DIAGNOSE ALIAS control statement, identifying in
the COMPARE-MASTERCATALOG keyword the two master catalogs that will be
compared. The NONPEER keyword is specified, so the first-specified master
catalog will be considered superior to the second-specified master catalog. To
state that another way, the first-specified master catalog will be considered to be
the control catalog, so alias entries in it will be compared against the other to
help you synchronize the second master catalog to the first. The FIXDATASET
keyword indicates that IDCAMS fix commands will be created for each
identified discrepancy and stored in the data set specified by the FIXDD
statement.

Chapter 7. Case studies

121

Example 7-6 DIAGNOSE ALIAS master catalog job stream

//YCJRES1N JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S2 EXEC PGM=CAT00010
//FIXDD
DD DSN=YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE),
//
DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
DIAGNOSE ALIAS COMPARE-MASTERCATALOG(MCAT.SANDBOX.R10 MCAT.SANDBOX.Z02) FIXDATASET(YCJRES1.CRPLUS.DIAG.ALIAS.FIXFILE) NONPEER

Refer to 6.6.1, DIAGNOSE ALIAS command on page 97, for specific details of
what this command is used for and how it works.
In Example 7-7, the output report from DIAGNOSE ALIAS first provides statistics
from each master catalog, showing the total number of records, the number of
aliases, and the number of connected user catalogs.

Example 7-7 DIAGNOSE ALIAS master catalog compare report

CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 24 SEP 2001 PAGE:1
CAT11270I CATALOG MCAT.SANDBOX.R10 WILL BE COMPARED TO MCAT.SANDBOX.Z02
CAT11272I CATALOG MCAT.SANDBOX.R10 WILL BE TREATED AS SUPERIOR TO CATALOG
MCAT.SANDBOX.Z02
THAT IS, DEFINE/DELETE ALIAS COMMANDS WILL BE CREATED FOR ENTRIES
MISSING FROM MCAT.SANDBOX.Z02 BUT PRESENT IN MCAT.SANDBOX.R10
CAT11221I EXTRACTING USERCATALOG AND ALIAS INFORMATION FROM MCAT.SANDBOX.R10
CAT11222I
5,811 RECORDS IN CATALOG
CAT11222I
328 ALIAS RECORDS FOR USERCATALOGS
CAT11222I
31 USERCATALOG RECORDS
CAT11223I THE FOLLOWING CATALOGS IN MCAT.SANDBOX.R10 HAVE NO ALIAS NAMES
CAT11223I
MCAT.OS3R6V01.VTOTCAT
CAT11223I
MCAT.SANDBOX.R9.VSBOX11
CAT11223I
MCAT.SANDBOX.VSBOX11
CAT11223I
MCAT.SANDBOX.Z02.VSBOX11
CAT11223I
MCAT.SBOXALT.SBOX02
CAT11223I
SYS1.VOLCAT.VGENERAL
CAT11223I
UCAT.CRPLUS5
CAT11221I EXTRACTING USERCATALOG AND ALIAS INFORMATION FROM MCAT.SANDBOX.Z02
CAT11222I
4,860 RECORDS IN CATALOG
CAT11222I
324 ALIAS RECORDS FOR USERCATALOGS
CAT11222I
30 USERCATALOG RECORDS
CAT11223I THE FOLLOWING CATALOGS IN MCAT.SANDBOX.Z02 HAVE NO ALIAS NAMES
CAT11223I
MCAT.OS3R6V01.VTOTCAT

122

ICF Catalog Backup and Recovery: A Practical Guide

CAT11223I
CAT11223I
CAT11223I
CAT11223I
CAT11223I

MCAT.SANDBOX.R10.VSBOX11
MCAT.SANDBOX.R9.VSBOX11
MCAT.SANDBOX.VSBOX11
MCAT.SBOXALT.SBOX02
SYS1.VOLCAT.VGENERAL

CAT11231I BEGINNING USERCATALOG COMPARISON

CAT11233I COMPARE-MASTERCATALOG MCAT.SANDBOX.R10 FOUND IN MCAT.SANDBOX.Z02
CAT11233I COMPARE-MASTERCATALOG MCAT.SANDBOX.Z02 FOUND IN MCAT.SANDBOX.R10
CAT11232W UCAT UCAT.CRPLUS5 IN MCAT.SANDBOX.R10 BUT NOT IN MCAT.SANDBOX.Z02
CAT11284I CORRESPONDING FIX: 1
CAT11235I BEGINNING ALIAS COMPARISON
CAT11237W ALIAS ALA110 IN MCAT.SANDBOX.R10 BUT NOT IN MCAT.SANDBOX.Z02
CAT11284I CORRESPONDING FIX: 2
CAT11237W ALIAS AUO110 IN MCAT.SANDBOX.R10 BUT NOT IN MCAT.SANDBOX.Z02
CAT11284I CORRESPONDING FIX: 3
** DIAGNOSE ALIAS OF MCAT.SANDBOX.R10 AND MCAT.SANDBOX.Z02 (NON PEERS) **
CAT11239W ALIAS ICSF HAS UNMATCHED RELATED USERCATALOG NAMES
CAT11239W THE ALIAS ENTRY IN MCAT.SANDBOX.R10 REFERENCES UCAT.VSBOX09
CAT11239W THE ALIAS ENTRY IN MCAT.SANDBOX.Z02 REFERENCES UCAT.VSBOX01
CAT11284I CORRESPONDING FIX: 4
CAT10009I DIAGNOSE FUNCTION COMPLETE. RETURN CODE 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

It also lists any user catalogs that do not have any aliases related to them.
Effectively, these would be unusable user catalogs, since it is not possible to
direct cataloging of data sets to them without an alias. This is not considered to
be an error, so a fix is not created for this situation, but rather an information
message to alert you.
Next, the command processor begins master catalog comparison. For each error
situation identified, a fix is created and stored in the fix data set, and a
corresponding reference number is given, so that you can correlate each error
message to its fix. Example 7-8 shows the fixes that were created as a result of
this command.
Immediately after the BEGINNING... message, note that the output report
comments that each master catalog is found to be connected as a user catalog in
the other. This cross-connection of master catalogs is a common practice and
enables you to perform maintenance on a master catalog from the other system,
where you then treat the master catalog as if it was a user catalog.

Chapter 7. Case studies

123

Example 7-8 DIAGNOSE ALIAS master catalog generated fixes

/*
/*

CATALOG RECOVERYPLUS FIX LIST FROM DIAGNOSE ALIAS OF */

CATALOG MCAT.SANDBOX.R10.VSBOX11 AND MCAT.SANDBOX.Z02 */

/* CATALOG RECOVERYPLUS FIX REFERENCE 1 */

/* IMPORT CONNECT OBJECTS (( UCAT.CRPLUS5 DEVICETYPE ( 3390 ) VOLUMES ( SBOX02 ) )) CATALOG ( MCAT.SANDBOX.Z02 )
/* CATALOG RECOVERYPLUS FIX REFERENCE 2
/* DEFINE ALIAS ( NAME ( ALA110 ) RELATE ( UCAT.VSBOX09 ) ) CATALOG ( MCAT.SANDBOX.Z02 )

*/

/* CATALOG RECOVERYPLUS FIX REFERENCE 3

/* DEFINE ALIAS ( NAME ( AUO110 ) RELATE ( UCAT.VSBOX09 ) ) CATALOG ( MCAT.SANDBOX.Z02 )

*/

/* CATALOG RECOVERYPLUS FIX REFERENCE 4*/

/* DELETE ICSF ALIAS CATALOG ( MCAT.SANDBOX.R10 )
/* DEFINE ALIAS ( NAME ( ICSF ) RELATE ( UCAT.VSBOX09 ) ) CATALOG ( MCAT.SANDBOX.R10 )
/* DEFINE ALIAS ( NAME ( ICSF ) RELATE ( UCAT.VSBOX01 ) ) CATALOG ( MCAT.SANDBOX.R10 )
/* DELETE ICSF ALIAS CATALOG ( MCAT.SANDBOX.Z02 )
/* DEFINE ALIAS ( NAME ( ICSF ) RELATE ( UCAT.VSBOX09 ) ) CATALOG ( MCAT.SANDBOX.Z02 )
/* DEFINE ALIAS ( NAME ( ICSF ) RELATE ( UCAT.VSBOX01 ) ) CATALOG ( MCAT.SANDBOX.Z02 )

Fix #1 indicates a user catalog, UCAT.CRPLUS5, that is found to be connected in

MCAT.SANDBOX.R10, but not in MCAT.SANDBOX.Z02. The generated fix is an
IMPORT CONNECT command for that user catalog in MCAT.SANDBOX.Z02.
Fix #2 and #3 indicate aliases that are in MCAT.SANDBOX.R10, but are not
found in MCAT.SANDBOX.Z02. The generated fix is a DEFINE ALIAS command.

124

ICF Catalog Backup and Recovery: A Practical Guide

Fix #4 is more complicated. In that instance, it was found that the alias ICSF was
found in each master catalog. But in one master, it was related to the user
catalog UCAT.VSBOX09. In the other master, it was related to the user catalog
UCAT.VSBOX01. This is considered to be an error situation, but since there is no
way for the DEFINE ALIAS command processor to know which is correct, it
builds two sets of commands that you can choose from, once you determine
which fix is correct for your situation.
Finally, note that all the fixes in Example 7-8 were generated as comment
statements, and as such, are not directly executable as IDCAMS commands until
the /* at the beginning of each command is removed. Since the fix file is a
physical sequential file, that can be done with TSO ISPF edit. In many situations,
this might be a laborious task, so the DIAGNOSE ALIAS command has the
keywords DEFINE-ALIAS and DELETE-ALIAS. If you know in advance which
command is more appropriate for your environment, the command processor will
build those commands as ready to execute.

7.4 Case study #4: VVDS backup and forward recovery

This case study demonstrates how to use the Catalog RecoveryPlus BACKUP
and RECOVER VVDS command first, to produce backup copies for VVDSs on
critical volumes, and secondly, to forward recover a VVDS if it breaks. The
following examples do not show an actual problem with the VVDS being
recovered, but rather, the job streams that perform VVDS backup and recovery.
It is frequently asked, How often do VVDSs break? The answer is more often
than most people realize, but since a DIAGNOSE VVDS command is rarely
performed, any errors encountered are probably credited to other causes.
Interestingly, during system testing for this redbook, we discovered a broken
VVDS on one of our ITSO volumes. Since this volume was heavily in use by
other redbook projects at the time, and was only causing intermittent failures
during data set allocation, its recovery was left for a later time. Below is a
re-creation of the steps that would have been used in our recovery on the ITSO
system, assuming it was a VVDS failure on volume SBOX79.
As shown in Example 7-9, the VVDSs on all of the SBOXxx volumes were
backed up with a single Catalog RecoveryPlus BACKUP command, with the
backup records for all VVDSs being written out to a single output file identified by
the DD statement OUT1. In a production environment, this would presumably be
a regularly-scheduled job, possibly run at the same time as all BCSs are backed
up.

Chapter 7. Case studies

125

Example 7-9 BACKUP VVDS job stream and output listing

20.11.15
20.13.12
20.13.12
20.13.12
20.13.12

JOB10237
JOB10237
JOB10237
JOB10237
JOB10237

IEF403I YCJRES1Z - STARTED - ASID=0019

IEF404I YCJRES1Z - ENDED - ASID=0019
---TIMINGS (MINS.)--JOBNAME STEPNAME RC
EXCP
CPU
SRB
-YCJRES1Z S1
04 33667
.11
.01

CLOCK
1.9

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*

//S2 EXEC PGM=CAT00010
//OUT1
DD DSN=YCJRES1.BACKUP(+1),DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(10,10),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
BACKUP VVDS(SBOX*) OUTFILE(OUT1)
/*
Output Listing:
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 24 SEP 2001 PAGE:1
CAT02061W VVDS SYS1.VVDS.VSBOX07 IS NOT ON THE VOLUME
CAT02061W VVDS SYS1.VVDS.VSBOX34 IS NOT ON THE VOLUME
... and 29 more volumes where VVDSs were not found on the volume,
... followed by a list of 66 volumes that were selected for the BACKUP
CAT02045I START OF DATA UNLOAD FOR SYS1.VVDS.VSBOX01 ON 24 SEP 2001 AT 20.11.16
CAT02092I CALLING IDCAMS TO PERFORM DIAGNOSE VVDS
IDC11367I THE FOLLOWING VVDS REFERENCED CATALOGS WERE NOT ENCOUNTERED:
CATALOG.TOTICF1.VTOTCAT
CAT02050W IDCAMS (8) ISSUED RETURN CODE 4
CAT02046I RECORD SUMMARY FOR SYS1.VVDS.VSBOX01
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
TOTAL BYTES
295
39
0
1
0
335
179,560
CAT02047I
END OF DATA UNLOAD FOR SYS1.VVDS.VSBOX01 ON 24 SEP 2001 AT 20.11.48
... additional volume unloads removed from listing
CAT02045I START OF DATA UNLOAD FOR SYS1.VVDS.VSBOX79 ON 24 SEP 2001 AT 20.11.51
CAT02092I CALLING IDCAMS TO PERFORM DIAGNOSE VVDS
CAT02046I RECORD SUMMARY FOR SYS1.VVDS.VSBOX79
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
TOTAL BYTES
1,032
0
0
1
0
1,033
378,834
CAT02047I
END OF DATA UNLOAD FOR SYS1.VVDS.VSBOX79 ON 24 SEP 2001 AT 20.11.57
CAT02009I BACKUP FUNCTION COMPLETE. RETURN CODE 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

126

ICF Catalog Backup and Recovery: A Practical Guide

There were a total of 95 SBOXxx volumes, as identified by the mask SBOX*, of

which 29 were found to be non-system-managed, without any VSAM on them,
and therefore, did not contain a VVDS. A total of 66 volumes had VVDSs and
were backed up. As part of BACKUP command processing, an IDCAMS
DIAGNOSE VVDS command was first issued for each volume, followed by a data
unload of all records from the VVDS.
As you can see from the job timings message, the BACKUP of all 66 VVDS
required 33,667 EXCPs and 1.9 minutes of elapsed time to execute. In a
subsequent test to determine what the cost increase would be if the backup was
duplexed, the EXCP cost increased by only 259, and the elapsed time remained
at 1.9 seconds.
Subsequent to the backup, our case study assumes that SBOX79 has an internal
error, necessitating a recovery from the most recent backup. The RECOVER
command job stream in Example 7-10 illustrates forward recovery for this
volumes VVDS, specifying that the SMF data from all systems sharing this
volume are concatenated on the DD statement, SMFINDD.

Example 7-10 RECOVER VVDS job stream

//YCJRES1N JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S2 EXEC PGM=CAT00010
//INFILE1 DD DSN=YCJRES1.VVDS.BACKUP,DISP=SHR
//SMFINDD DD DSN=YCJRES1.SC63.SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.SC64.SMFDATA,DISP=SHR
//
DD DSN=YCJRES1.SC65.SMFDATA,DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
RECOVER VVDS(SBOX79) INFILE(INFILE1) FORWARD(SMFFILE(SMFINDD))

The output from the VVDS forward recovery can be found in the listing report in
Example 7-11. After first sorting the SMF records, RECOVER then deletes and
defines the original VVDS, then writes the new, updated records into the new
VVDS. Notice that it also identified one or more VVR/NVR records that
back-pointed to three BCSs whose names were not in the VVCR record, so
these BCS names were added.

Chapter 7. Case studies

127

Example 7-11 RECOVER VVDS job stream and output listing

CAT01001I
CAT04079I
CAT04079I
CAT04080I

Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 24 SEP 2001 PAGE:1

DEFAULT IN EFFECT: FROMBACKUPDATE
DEFAULT IN EFFECT: TOCURRENTDATE
PRINT OPTION IN EFFECT: NONE

CAT04029I 20.32.26 RECOVERY STEP: READ BACKUP - STARTED

CAT04058I RECOVERY STARTED FOR VVDS: SYS1.VVDS.VSBOX79
BACKUP DATE/TIME: 2001267/20:11:16
CAT04030I RECORD SUMMARY FOR SYS1.VVDS.VSBOX79 RECORDS FROM BACKUP
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
TOTAL BYTES
1,012
0
0
1
0
1,013
371,856
CAT04029I 20.32.26 RECOVERY STEP: READ BACKUP - COMPLETED
CAT04029I 20.32.26 RECOVERY STEP: READ VTOC - STARTED
CAT04029I 20.32.29 RECOVERY STEP: READ VTOC - COMPLETED
CAT04029I 20.32.29 RECOVERY STEP: SORT INPUT RECORDS - STARTED
CAT04029I 20.32.30 RECOVERY STEP: SORT INPUT RECORDS - COMPLETED
CAT04029I 20.32.30 RECOVERY STEP: PROCESSING SMF - STARTED
CAT04019I PROCESSING SMF RECORDSFROM DATE/TIME: 2001267/20:11:16
TO DATE/TIME: 2001267/20:32:26
CAT04020I SMF RECORDS SELECTED:
21
CAT04021I SMF RECORDS FOR SYSID: SC63;EARLIEST: 2001266/20:10:00
LATEST: 2001267/20:23:40
CAT04021I SMF RECORDS FOR SYSID: SC64; EARLIEST: 2001266/18:10:00
LATEST: 2001267/18:30:27
CAT04021I SMF RECORDS FOR SYSID: SC65; EARLIEST: 2001266/18:10:01
LATEST: 2001267/18:00:37
CAT04026W ******** GAP IN TIME FOR SYSID: SC64 LATEST TIME GAP
CAT04026W ******** GAP IN TIME FOR SYSID: SC65 LATEST TIME GAP
CAT04027I RECORD COUNTS FROM APPLYING SMF DATA:
1011 UNCHANGED;
0 DELETED;
20 INSERTED;
0 UPDATED
CAT04029I 20.32.32 RECOVERY STEP: PROCESSING SMF - COMPLETED
CAT04029I 20.32.32 RECOVERY STEP: DEFINE OF VVDS - STARTED
DEFINE CLUSTER(NAME(SYS1.VVDS.VSBOX79) VOLUMES(SBOX79) NONINDEXED TRACKS(00010 00010)
)
CAT04029I 20.32.32 RECOVERY STEP: DEFINE OF VVDS - COMPLETED
CAT04029I 20.32.32 RECOVERY STEP: REWRITE TO VVDS - STARTED
CAT04030I RECORDS WRITTEN TO SYS1.VVDS.VSBOX79
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
1,031
0
0
0
0
1,031
CAT04030I BCS'S ADDED TO VVCR/VVCN:
3
CAT04029I 20.32.32 RECOVERY STEP: REWRITE TO VVDS - COMPLETED

TOTAL BYTES
374,385

CAT04059I RECOVERY COMPLETED FOR VVDS: SYS1.VVDS.VSBOX79

CAT04009I RECOVER PROCESS COMPLETED; RETURN CODE: 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

128

ICF Catalog Backup and Recovery: A Practical Guide

7.5 Case study #5: Enlarging VVDS

This case study demonstrates how to use the Catalog RecoveryPlus BACKUP
VVDS and RECOVER VVDS commands to enlarge a VVDS.
In this example, the VVDS on volume VSBOX36 was allocated using the default
allocation size of TRK(10 10). For todays volumes, where thousands of small
data sets could be allocated, you might fill up the VVDS and put it into secondary
extents (which are not necessarily a problem, but its one more complicating
factor on your volumes that potentially leads to problems).
Example 7-12 illustrates the BACKUP job stream and output that creates the
backup copy of the VVDS, in preparation for recovery.

Example 7-12 BACKUP VVDS job stream and output listing

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//OUT1
DD DSN=YCJRES1.CRPLUS.VVDS.SBOX36.BKUP4,DISP=(,CATLG),
//
UNIT=SYSALLDA,SPACE=(CYL,(1,1),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN
DD *
BACKUP VVDS(SBOX36) OUTFILE(OUT1)
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 28 SEP 2001 PAGE:1
CAT02016I THE FOLLOWING DSNS/MASKS ARE TO BE PROCESSED DURING VVDS PROCESSING.
SYS1.VVDS.VSBOX36
CAT02045I START OF DATA UNLOAD FOR SYS1.VVDS.VSBOX36 ON 28 SEP 2001 AT 12.44.32
CAT02092I CALLING IDCAMS TO PERFORM DIAGNOSE VVDS
CAT02046I RECORD SUMMARY FOR SYS1.VVDS.VSBOX36
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
TOTAL BYTES
101
0
0
1
0
102
39,499
CAT02047I
END OF DATA UNLOAD FOR SYS1.VVDS.VSBOX36 ON 28 SEP 2001 AT 12.44.34
CAT02009I BACKUP FUNCTION COMPLETE. RETURN CODE 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

Example 7-13 illustrates the IDCAMS DELETE and DEFINE CLUSTER

commands that you then run. The keyword RECOVERY tells IDCAMS to
physically delete the VVDS from the volume, without deleting any of the actual
data sets on the volume. The FILE keyword is necessary for this operation, as
the MVS allocation routines use it for the locate operation of the VVDS on the
volume. The DEFINE CLUSTER command that follows reallocates a new VVDS
on the volume, at the size of CYL(2 2).

Chapter 7. Case studies

129

Example 7-13 IDCAMS DELETE/DEFINE job stream and output listing

YCJRES1B JOB MSGCLASS=X,NOTIFY=UCJRES1,RESTART=*
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SBOX36
DD VOL=SER=SBOX36,UNIT=3390,DISP=SHR
//SYSIN
DD *
DELETE SYS1.VVDS.VSBOX36 FILE(SBOX36) RECOVERY
IDC0550I ENTRY (C) SYS1.VVDS.VSBOX36 DELETED
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
DEFINE CLUSTER(NAME(SYS1.VVDS.VSBOX36) VOL(SBOX36) FILE(SBOX36) NONINDEXED CYL(2 2))
IDC0508I DATA ALLOCATION STATUS FOR VOLUME SBOX36 IS 0
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0

Example 7-14 then runs a Catalog RecoveryPlus RECOVER VVDS command,

specifying the keyword INTOEMPTY, to alert the command processor that a
DELETE and DEFINE on the VVDS is already accomplished, and that it is to look
for an empty VVDS on the volume for restore processing. This RECOVER
command only does a restore of the backup records and does not perform a
forward recovery, because the FORWARD keyword is not specified. This is a
valid situation, because the BACKUP and RECOVER commands were executed
in close time sequence, without any data set accesses or updates occurring on
the volume in between. As part of its normal processing, RECOVER sorts the
VVR/NVR records prior to restoring them, so that any duplicate VVR/NVR
records can be detected. The read VTOC messages indicate the command is
checking to ensure that all VVDS records have associated VTOC records.

Example 7-14 RECOVER VVDS job stream and output listing

//YCJRES1K JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//INFILE1 DD DSN=YCJRES1.CRPLUS.VVDS.SBOX36.BKUP3,DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 28 SEP 2001 PAGE:1
RECOVER VVDS(SBOX36)
INFILE(INFILE1) INTOEMPTY
CAT04080I PRINT OPTION IN EFFECT: NONE

130

ICF Catalog Backup and Recovery: A Practical Guide

CAT04029I 12.32.18 RECOVERY STEP: READ BACKUP - STARTED

CAT04058I RECOVERY STARTED FOR VVDS: SYS1.VVDS.VSBOX36 BACKUP DATE/TIME:
2001271/12:10:36
CAT04030I RECORD SUMMARY FOR SYS1.VVDS.VSBOX36
RECORDS FROM BACKUP
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
TOTAL BYTES
101
0
0
1
0
102
39,491
CAT04029I 12.32.18 RECOVERY STEP: READ BACKUP - COMPLETED
CAT04029I 12.32.18 RECOVERY STEP: READ VTOC - STARTED
CAT04029I 12.32.21 RECOVERY STEP: READ VTOC - COMPLETED
CAT04029I 12.32.21 RECOVERY STEP: SORT INPUT RECORDS - STARTED
CAT04029I 12.32.21 RECOVERY STEP: SORT INPUT RECORDS - COMPLETED
CAT04029I 12.32.21 RECOVERY STEP: REWRITE TO VVDS - STARTED
CAT04011I DATASET IN VVDS
ALLOCATED TO DDNAME: SYS00001
CAT04030I RECORDS WRITTEN TO SYS1.VVDS.VSBOX36
PRIMARY SECONDARY
NONVSAM VVCR/VVCN
OTHER
TOTAL
100
0
0
0
0
100
CAT04030I BCS'S ADDED TO VVCR/VVCN:
1
CAT04029I 12.32.22 RECOVERY STEP: REWRITE TO VVDS - COMPLETED

TOTAL BYTES
35,090

CAT04059I RECOVERY COMPLETED FOR VVDS: SYS1.VVDS.VSBOX36

CAT04009I RECOVER PROCESS COMPLETED; RETURN CODE: 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

7.6 Case study #6: Changing a disk volser

This case study demonstrates how to use the Catalog RecoveryPlus
SUPERCLIP command.
Without CR+ SUPERCLIP, standard system utilities require you to unload all of
the data sets from the volume, delete (for recovery) any ICF catalogs that reside
on the volume, VARY the volume offline, run the utility ICKDSF to clip the
volser, VARY the volume online, restore any ICF catalogs that previously resided
on the volume, and finally, reload all of the volumes data sets. Depending on how
many data sets are on the volume, this could take a long time and is the primary
reason the process is not performed very often.
With CR+ SUPERCLIP, you simply specify the old volser and the new volser, and
the command processor internally performs all of the functions necessary to
change the volser.

Chapter 7. Case studies

131

Example 7-15 illustrates the SUPERCLIP command, where volser SBOX36 is

changed to CRPL36. The volume currently has data sets physically allocated on
it (154 data sets, according to message CAT14043I), including a combination of
VSAM, non-VSAM, and GDGs. There is also a BCS on the volume,
UCAT.CRPLUS5. Since its connector (U-type) record in the master catalog
points to it by volser, this volser value is also changed.
The output in the example provides a step-by-step process log of all functions
that the SUPERCLIP command provides, including the MVS MODIFY
commands to unallocate the catalog, and VARY OFFLINE/ONLINE commands
surrounding the ICKDSF program execution.

Example 7-15 SUPERCLIP job stream and output listing

18.40.49 JOB11171 IEF403I YCJRES1Z - STARTED - ASID=0019.
18.41.28 JOB11171 F CATALOG,UNALLOCATE(UCAT.CRPLUS5)
18.41.28 JOB11171 V 3F0C,OFFLINE
18.52.48 JOB11171 V 3F0C,ONLINE
18.53.25 JOB11171 ---TIMINGS (MINS.)-18.53.25 JOB11171 -JOBNAME STEPNAME RC
EXCP
CPU
CLOCK
18.53.25 JOB11171 -YCJRES1Z S1
00
7255
.01
12.5
18.53.25 JOB11171 IEF404I YCJRES1Z - ENDED - ASID=0019.
//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
SUPERCLIP OLDVOLSER(SBOX36) NEWVOLSER(CRPL36) JOURNAL(YCJRES1.SUPERCLP.JOURNAL)
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 02 OCT 2001 PAGE:1
CAT14014I SUPERCLIP PROCESSING STARTED
CAT14020I 18.40.49 SUPERCLIP STEP: DSNAME COLLECTION - STARTED
REPORT OF CATALOGS
UCAT.CRPLUS1
UCAT.CRPLUS2
UCAT.CRPLUS5
CATALOG AFFECTED
ENTRY'S NAME IN CATALOG
** CATALOG N/A **
SYS1.VVDS.VSBOX36
SBOX36
UCAT.CRPLUS1
YCJRES1.BCSNEW01
SBOX36
YCJRES1.BCSNEW02

132

** ON OLDVOLSER **

F1 DSCB NAME
SYS1.VVDS.VSBOX36

YCJRES1.BCSNEW01
YCJRES1.BCSNEW02

ICF Catalog Backup and Recovery: A Practical Guide

SBOX36
YCJRES1.BCSNEW03
SBOX36
YCJRES1.BCSNEW04
SBOX36

YCJRES1.BCSNEW03
YCJRES1.BCSNEW04

... a hundred (or so) additional data set names removed from the listing
UCAT.CRPLUS2
YCJRES2.CRPLUS.VVDSTST0
SBOX36
YCJRES2.CRPLUS.VVDSTST0
SBOX36
YCJRES2.CRPLUS.VVDSTST1
SBOX36
YCJRES2.CRPLUS.VVDSTST1
SBOX36

YCJRES2.CRPLUS.VVDSTST0.DATA
YCJRES2.CRPLUS.VVDSTST0.INDEX
YCJRES2.CRPLUS.VVDSTST1.DATA
YCJRES2.CRPLUS.VVDSTST1.INDEX

... plus several additional data set names removed from the listing
UCAT.CRPLUS5
UCAT.CRPLUS5
SBOX36
UCAT.CRPLUS5
SBOX36

UCAT.CRPLUS5
UCAT.CRPLUS5.CATINDEX

REPORT OF VOLUME SERIALS

SBOX36
CAT14042I VSAM DATASETS IN CATALOGS ON OLDVOLSER SHOULD BE CHECKED
FOR DISCRETE RACF PROFILES
CAT14043I NUMBER OF DATASETS ON OLDVOLSER: 154
CAT14020I 18.40.53 SUPERCLIP STEP: DSNAME COLLECTION - COMPLETED
CAT14020I 18.40.53 SUPERCLIP STEP: F1 DSCB UPDATE - STARTED
CAT14020I 18.40.54 SUPERCLIP STEP: F1 DSCB UPDATE - COMPLETED
CAT14020I 18.40.54 SUPERCLIP STEP: BCS ENTRY UPDATE - STARTED
CAT14020I 18.40.55 SUPERCLIP STEP: BCS ENTRY UPDATE - COMPLETED
CAT14020I 18.40.55 SUPERCLIP STEP: VVDS UPDATE - STARTED
CAT14020I 18.40.55 SUPERCLIP STEP: VVDS UPDATE - COMPLETED
CAT14020I 18.40.55 SUPERCLIP STEP: VVDS RENAME - STARTED
CAT14020I 18.40.55 SUPERCLIP STEP: VVDS RENAME - COMPLETED
CAT14020I 18.40.55 SUPERCLIP STEP: CONVERT TO OS VTOC - STARTED
CAT14020I 18.41.03 SUPERCLIP STEP: CONVERT TO OS VTOC - COMPLETED
CAT14020I 18.41.03 SUPERCLIP STEP: VTOCIX RENAME - STARTED
CAT14020I 18.41.03 SUPERCLIP STEP: VTOCIX RENAME - COMPLETED
CAT14020I 18.41.28 SUPERCLIP STEP: SBOX36 IS NOW OFFLINE

Chapter 7. Case studies

133

CAT14020I 18.41.28 SUPERCLIP STEP: ICKDSF REFORMAT - STARTED

CAT14020I 18.52.48 SUPERCLIP STEP: ICKDSF REFORMAT - COMPLETED
CAT14020I 18.52.48 SUPERCLIP STEP: CRPL36 IS NOW ONLINE
CAT14020I 18.53.19 SUPERCLIP STEP: CONVERT TO IX VTOC - STARTED
CAT14020I 18.53.24 SUPERCLIP STEP: CONVERT TO IX VTOC - COMPLETED
CAT14020I 18.53.24 SUPERCLIP STEP: IMPORT CONNECT - STARTED
MCAT'S TO BE UPDATED:
MCAT.SANDBOX.R10.VSBOX11
BCS'S IMPORTED WITH NEWVOLSER:
UCAT.CRPLUS5
CAT14020I 18.53.24 SUPERCLIP STEP: IMPORT CONNECT - COMPLETED
CAT14020I 18.53.24 SUPERCLIP STEP: DEFINE VVDS TO BCS - STARTED
UCAT.CRPLUS2
UCAT.CRPLUS5
CAT14020I 18.53.25 SUPERCLIP STEP: DEFINE VVDS TO BCS - COMPLETED
CAT14014I SUPERCLIP PROCESSING ENDED
CAT14014I SBOX36 SUCCESSFULLY CHANGED TO CRPL36
CAT14009I SUPERCLIP FUNCTION COMPLETE. RETURN CODE 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

As the example shows, the job began at 18.40.49 and ended at 18.53.25,
including 7,255 I/Os. Most of the time and I/O cost was used updating the volume
cell in the associated catalog record for each of the 154 data sets. SUPERCLIP,
as part of its processing, does a Catalog Search Interface (CSI) LOCATE on
each of the 154 data sets on the volume to identify the BCSs in which each data
set is cataloged. It then updates the volser pointer in those catalog records, as
they point to volser SBOX36, and changes it to CRPL36. Also for the BCS that
resides on the volume (UCAT.CRPLUS5), its connector record in the master
catalog must be updated, as that record indicates SBOX36, to be CRPL36.
Prudence dictates that you should consider running the SUPERCLIP command
in advance, in SIMULATE mode, enabling you to see for yourself that the
command is going to do exactly what you expect. The output from this execution
is exactly the same as without SIMULATE, but does not perform any of the actual
change operation that a real SUPERCLIP command does.
If your SUPERCLIP window is very short, there is a possible way that you can
use the SIMULATE facility to complete the long-running tasks of SUPERCLIP
ahead of time, and then perform the actual clip process in a very short time. To
do this, you first run SUPERCLIP with SIMULATE, as shown in Example 7-16.
This process does the LOCATE to the BCS on all data sets on the volume,
storing the information in the journal file. You then replace the word SIMULATE
on the command with the word RESTART. At the beginning of your clip window,

134

ICF Catalog Backup and Recovery: A Practical Guide

you re-run the SUPERCLIP command. SUPERCLIP uses the journal data set
records to drive the processing to actually perform the volser change,
completing this in a very short time. For this to work correctly, though, it is
imperative that there are no allocations.

Example 7-16 SUPERCLIP SIMULATE feature

SUPERCLIP OLDVOLSER(SBOX36) NEWVOLSER(CRPL36) JOURNAL(YCJRES1.SUPERCLP.JOURNAL) SIMULATE

7.7 Case study #7: Alternate master catalog

This case study demonstrates two techniques for setting up and maintaining an
alternate master catalog.
One technique, illustrated in Example 7-17, uses IDCAMS REPRO
NOMERGECAT to create a clone of your current master catalog. In this example,
INFILE represents the source master catalog and is identified as
MCAT.SANDBOX.VSBOX01 on the INDD1 DD statement. OUTFILE represents
the target master catalog for the NOMERGECAT operation and is identified as
MCAT.SANDBOX.VSBOX03 on the OUTDD1 DD statement.
Note message IDC11468I in the output report in this example, indicating that
NVR/VVR records have been updated to point to the new target catalog. The
effect of this is to create the requirement that you consider the target to be your
official master catalog (and which you should now IPL from), and the source
catalog becomes your alternate master catalog.

Example 7-17 REPRO NOMERGECAT to create an alternate master catalog

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//INDD1
DD DSN=MCAT.SANDBOX.VSBOX01,DISP=SHR
//OUTDD1
DD DSN=MCAT.CRPLUS.VSBOX03,DISP=OLD
//SYSIN
DD *
REPRO INFILE(INDD1) OUTFILE(OUTDD1) NOMERGECAT
/*
IDCAMS SYSTEM SERVICES
TIME: 20:08:36

10/02/01

PAGE

IDC11468I NVR/VVR NOW POINTS TO TARGET CATALOG.

IDC0005I NUMBER OF RECORDS PROCESSED WAS 4431

Chapter 7. Case studies

135

IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0

The second technique uses the Catalog RecoveryPlus MERGECAT command,

with the COPYONLY keyword. This command copies all records in the source
master catalog named on INBCS to the target master catalog named on
OUTBCS (MERGECAT normally performs a move of a catalogs records; but with
COPYONLY, it becomes a copy command.)
A fundamental difference in CR+ MERGECAT COPYONLY from that of IDCAMS
REPRO NOMERGECAT is that it does not update the BCS back-pointer in the
VVDS records, where REPRO does. The result is that, with the CR+ logic, your
source master catalog remains your official master catalog, and the target copy
master catalog is the alternate master catalog.
Example 7-18 shows how to use the second technique. Note that the output
listing from MERGECAT automatically provides a list of every data set record that
is copied, enabling you to verify the specific data set records that were copied if
you desire.

Example 7-18 CR+ MERGECAT to create an alternate master catalog

19.55.12 JOB09725 IEF403I YCJRES1B - STARTED - ASID=0019.
19.56.01 JOB09725 IEF404I YCJRES1B - ENDED - ASID=0019.
//YCJRES1B JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN
DD *
MERGECAT INBCS(MCAT.SANDBOX.VSBOX01) OUTBCS(MCAT.CRPLUS.VSBOX01) JOURNAL(YCJRES1.MERGECAT.JOURNAL) COPYONLY UNLOCK
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 25 SEP 2001 PAGE:1
CAT12000I >>> MERGECAT
STARTING
CAT12000I
CAT12000I **************************************
CAT12000I * ALLOCATE FILES
*
CAT12000I **************************************
CAT12000I
>>> ALL FILES ALLOCATED
CAT12000I
CAT12000I **************************************
CAT12000I * BUILD PROCESS JOURNAL
*
CAT12000I **************************************
CAT12000I **************************************
CAT12000I * USER CATALOG EXTRACT STARTED
*
CAT12000I * THIS PROCESS MAY TAKE A WHILE
*

136

ICF Catalog Backup and Recovery: A Practical Guide

CAT12000I **************************************
CAT12000I
>>> JOURNAL FILE HAS BEEN POPULATED
CAT12000I
>>> MERGECAT IS NOW RESTARTABLE
CAT12042I
>>>
4,367 CATALOG ENTRIES WILL BE COPIED TO THE OUTPUT CAT12000I
**************************************
CAT12000I * COPY CATALOG ENTRIES
*
CAT12000I **************************************
CAT12009I
PROCESS STARTED FOR SYS1.AADFMAC1
CAT12011I
PROCESS ENDED FOR SYS1.AADFMAC1
CAT12009I
PROCESS STARTED FOR SYS1.AADRLIB
CAT12011I
PROCESS ENDED FOR SYS1.AADRLIB
... plus several thousand data set names removed from the listing
CAT12000I
>>> SELECTED ENTRIES HAVE BEEN COPIED
CAT12000I
CAT12000I **************************************
CAT12000I * COPY SYS1.VVDS.VXXXXXX ENTRIES
*
CAT12000I **************************************
CAT12035I
SYS1.VVDS.VHG6601
CAT12035I
SYS1.VVDS.VHG6605
... and VVDS names continue for 19 more lines that are removed
CAT12000I >>> MERGECAT
ENDED SUCCESSFULLY
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

Note: You may wonder why you should care which master catalog is the
official one, which is the alternate master catalog, and what is the
importance of these BCS back pointers? The simple answer is, in normal
day-to-day system processing (whether it is for ordinary data sets or critical
system data sets), the BCS to which the back-pointer points is largely
irrelevant, because it is not used or checked during data set LOCATE or
OPEN/CLOSE processing. However, if you attempt to delete the data set, the
BCS back-pointer in the data sets VVR/NVR is checked to ensure that the
BCS through which you are attempting DELETE is the same one as named in
the BCS back-pointer. If it is not, the DELETE operation fails.

For that reason, when you use IDCAMS REPRO NOMERGECAT to clone a
master catalog, and it updates the BCS back-pointers, that makes the target
master catalog your official master catalog from which you now need to IPL.
When you use CR+ MERGECAT COPYONLY, it does not update the BCS
back-pointers. That leaves your original source master catalog as your official
master catalog, and the target is your alternate master. Either way, it works
OK. You just need to know the differences so that you do not accidentally fall
into a trap of using the wrong catalog as your master catalog.

Chapter 7. Case studies

137

7.8 Case study #8: Print and delete BCS/VVDS records

This case study demonstrates several techniques for printing and deleting
records from the BCS and VVDS.
The facilities that were used in this case study include the IBM ad hoc tool,
VVDSFIX, and the Catalog RecoveryPlus ZAP command.
The scenario in this case study is contrived, in that we intentionally created the
situation and then used VVDSFIX and CR+ ZAP to correct the error.
Nevertheless, the error is one that occurs in real life and illustrates the type of
corrective action that might be necessary.
Example 7-19 illustrates the IDCAMS DIAGNOSE VVDS command. It checks all
records in the named VVDS for validity of data content and with the
COMPAREDS operand, names a BCS to which all relevant VVDS records will be
compared.
The output from this command shows us there is a duplicate VVR/NVR in the
VVDS, for the VSAM component named YCJRES2.CRPLUS4.VVDSTST1.DATA
(the Z following the name indicates this is a primary VVR for a VSAM data set
component). This particular VVR record is identified as the incorrect VVR.
Simply it is the one dumped; the next error tells us why.
Sure enough, the following error identified by DIAGNOSE specifically indicates
that the data sets extent information in this VVR does not match with the
corresponding VTOC record. In case you want to analyze it further, the Format-1
DSCB record from the VTOC record is then dumped. Presumably, the other VVR
of this same component name has extent information that matches with the
corresponding VTOC record.
The conclusion is that we have a duplicate VVR in the VVDS for the data set
name YCJRES2.CRPLUS4.VVDSTST1.DATA. The following steps show the
various ways to correct this, using the VVDSFIX utility, as well as Catalog
RecoveryPlus.
Note again that we do not know anything about the other VVR presumably the
correct one particularly where it is located within the VVDS. The DIAGNOSE
output only indicates the VVR in error is located at RBA offset x0000B160 within
the VVDS. Understanding the implication of this is important in correctly fixing
this problem. Otherwise, you might easily delete the wrong record.

138

ICF Catalog Backup and Recovery: A Practical Guide

Example 7-19 DIAGNOSE VVDS to identify duplicate VVR in VVDS

//YCJRES1V JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//DIAGDD
DD DSN=SYS1.VVDS.VSBOX36,DISP=SHR,
//
VOL=SER=SBOX36,UNIT=SYSALLDA,AMP='AMORG'
//SYSIN
DD *
DIAGNOSE VVDS INFILE(DIAGDD)
/*
IDCAMS SYSTEM SERVICES
IDC21364I ERROR DETECTED BY DIAGNOSE:
VVDS ENTRY: YCJRES2.CRPLUS4.VVDSTST1.DATA (Z)
RECORD: X'0000B160'
OFFSET: X'0002'
REASON: 41 - DUPLICATE VVR/NVRS IN VVDS
IDC21365I VVDS RECORD DISPLAY:
RECORD: X'0000B160'
000 015F0068 E9000000 00001EE8 C3D1D9C5 E2F24BC3
020 E2E3F14B C4C1E3C1 0019E8C3 D1D9C5E2 F24BC3D9
040 E3F1000C E4C3C1E3 4BC3D9D7 D3E4E2F2 19E8C3D1
060 4BE5E5C4 E2E3E2E3 F1000055 21402000 00003000
080 00C00000 0000F000 00FFFFFF FFFFFFFF FF000000
0A0 00000000 00000000 00000000 00000000 00010000
0C0 62608000 60000000 00000800 00000C00 00000000
0E0 00000000 00000000 00000000 00000000 00000000
100 00000000 00000000 00000000 00000000 000000C0
120 00003E23 80010000 00000000 00000000 C0000000
140 00000000 00000000 00001400 01000000 02000000

TIME: 18:20:54

D9D7D3E4
D7D3E4E2
D9C5E2F2
00000100
00080000
01800000
00000010
00000000
00000000
1000000C
02000100

10/02/01

PAGE 1

E2F44BE5
F44BE5E5
4BC3D9D7
00018000
00000000
00000000
00000000
00000000
00000000
00014000
00000000

E5C4E2E3
C4E2E3E2
D3E4E2F4
00000000
00000000
00000000
F0000000
01000000
00000000
0F000000
00BFFF

*...Z......YCJRES2.CRPLUS4.VVDST
*ST1.DATA..YCJRES2.CRPLUS4.VVDSTS
*T1..UCAT.CRPLUS2.YCJRES2.CRPLUS4
*.VVDSTST1.... ..................
*.{....0.........................
*................................
*.-..-.......................0...
*................................
*...................{............
*................{......... .....
*...............................

E2F24040
00E5A200
00000000
00000000

40404065
00010000
00000000
00000000

*1SBOX36......._...IBMOSVS2
.
*........{................V......
*................................
*................................
*.................

IDC21364I ERROR DETECTED BY DIAGNOSE:

VVDS ENTRY: YCJRES2.CRPLUS4.VVDSTST1.DATA (Z)
RECORD: X'0000B160'
OFFSET: X'014B'
REASON: 16 - VVDS AND VTOC STARTING CCHH UNEQUAL
IDC21365I VVDS RECORD DISPLAY:
RECORD: X'0000B160'
record dumped, same as above
IDC21365I VTOC RECORD DISPLAY:
RECORD: YCJRES2.CRPLUS4.VVDSTST1.DATA
000 F1E2C2D6 E7F3F600 0165010E 63016D01
020 01130000 00000008 C0801000 00000000
040 0E000300 0E000300 00000000 00000000
060 00030001 25000000 00000000 00000000
080 00000000 00000000 00000000 00000000

0000C9C2
00128800
00000000
00000000
00

D4D6E2E5
00010000
00000000
00000000

IDC21363I THE FOLLOWING ENTRIES HAD ERRORS:

YCJRES2.CRPLUS4.VVDSTST1.DATA (Z) - REASON CODE: 41
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 8
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 8

Chapter 7. Case studies

139

Example 7-20 illustrates how this problem of the duplicate VVR can be solved
with VVDSFIX. The command (CMD) being used is DELVVR, which means
delete VVR. The specific VVR is identified by component name
YCJRES2.CRPLUS4.VVDSTST1.DATA, and it is the same one identified
previously in the DIAGNOSE output. The operand value CO specified within the
command indicates that we want to delete a COmponent. The last operand,
UCAT.CRPLUS2, and is crucial to the success of the command for this particular
problem (see the description for Example 7-22 on page 141 for more
information). When the BCS name is not specified, the DELVVR command
deletes every VVR within the VVDS that has the specified name, which is not
what we want to do here. By specifying the BCS name, the DELVVR command
can match up the BCS cluster record with the correct VVR and then delete only
the invalid VVR.

Example 7-20 VVDSFIX DELVVR command to delete a VVDS record

//YCJRES1F JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//VVDSGO
EXEC PGM=VVDSFIX
//SYSPRINT DD SYSOUT=*
//VVDS
DD DSN=SYS1.VVDS.VSBOX36,DISP=SHR,VOL=SER=SBOX36,UNIT=3390
//CATDD
DD DSN=UCAT.CRPLUS1,DISP=SHR
//SYSIN
DD *
CMD=DELVVR;YCJRES2.CRPLUS4.VVDSTST1.DATA;Y;CO;UCAT.CRPLUS2;
/*
Enter command for VVDS fixer
Enter VVR component or Cluster name.
The name entered =" YCJRES2.CRPLUS4.VVDSTST1.DATA ". Is this correct?(Y/N)
Is this a CLuster or a COmponent (data or index) name ? (Enter CL or CO)
Enter CATALOG name that goes with the VVR (optional)
IEF142I YCJRES1F VVDSGO - STEP WAS EXECUTED - COND CODE 0000

An alternative method to delete the duplicate VVDS record is the Catalog

RecoveryPlus ZAP VVDS DELETE command, as shown in Example 7-21. The
command specifies the explicit component name, the volser on which the VVDS
resides, and importantly, the RBA offset of the record, at x'0000B160', within the
VVDS. With the RBA specification, we deleted the correct duplicate VVR,
because we told the ZAP DELETE command exactly where to find the record. If
we did not specify the RBA offset, and let the ZAP DELETE command determine
the location of the record by reading sequentially through the VVDS, it would
delete the first record of that component name that it comes to, and that may not
be the one we wanted to delete (the next example shows just how important this
detail is to us).

140

ICF Catalog Backup and Recovery: A Practical Guide

Important: The ZAP VVDS DELETE command is not intended to be a

functional replacement for the IDCAMS DELETE VVR command, because it
only deletes the specific record that it is asked to delete. In other words, if the
VVDS record to be deleted has a corresponding VTOC record, the VTOC
record is not deleted with this command. With IDCAMS DELETE VVR, any
corresponding VTOC record is deleted.

Example 7-21 CR+ ZAP VVDS DELETE to delete a VVDS record

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
ZAP VVDS(SBOX36) DELETE(COMPONENT(YCJRES2.CRPLUS4.VVDSTST1.DATA) RBA(B160))
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 02 OCT 2001 PAGE:1
CAT08213I RECORD DELETED
CAT08000I ZAP FUNCTION COMPLETE. RETURN CODE 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

In Example 7-22, we execute a Catalog RecoveryPlus ZAP VVDS PRINT

command, specifying YCJRES2.CRPLUS4.VVDSTST1.DATA, which again is the
component name of the duplicate VVR. In the messages preceding the dump,
note the RBA offset of the record found, at x'000096DE' within the VVDS.
Remember that the invalid, duplicate record is at RBA offset x'0000B160', at a
location after the valid record. As we can see, the correct record of the duplicate
pair is at a lower offset location, where the invalid record is at a higher offset. In
this instance, if we can delete the invalid record, it is imperative that we explicitly
point to it, whether with the user catalog name specified in the VVDSFIX
DELVVR command or with the CR+ ZAP DELETE command.

Example 7-22 CR+ ZAP PRINT of a VVDS record

//YCJRES1X JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
ZAP VVDS(SBOX36) PRINT(COMPONENT(YCJRES2.CRPLUS4.VVDSTST1.DATA) GENERICKEY DUMP)
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 02 OCT 2001 PAGE:1

Chapter 7. Case studies

141

*******************************************************************************
VVDS DATA FOR DSN: YCJRES2.CRPLUS4.VVDSTST1.DATA
RBA FOUND: 000096DE
TYPE FOUND: Z
*******************************************************************************
00000000
015F0068 E9000000 00001EE8 C3D1D9C5
*.?..Z......YCJRE*
00000010
E2F24BC3 D9D7D3E4 E2F44BE5 E5C4E2E3
*S2.CRPLUS4.VVDST*
00000020
E2E3F14B C4C1E3C1 0019E8C3 D1D9C5E2
*ST1.DATA..YCJRES*
00000030
F24BC3D9 D7D3E4E2 F44BE5E5 C4E2E3E2
*2.CRPLUS4.VVDSTS*
00000040
E3F1000C E4C3C1E3 4BC3D9D7 D3E4E2F2
*T1..UCAT.CRPLUS2*
00000050
19E8C3D1 D9C5E2F2 4BC3D9D7 D3E4E2F4
*.YCJRES2.CRPLUS4*
00000060
4BE5E5C4 E2E3E2E3 F1000055 21402000
*.VVDSTST1.... ..*
00000070
00003000 00000100 00018000 00C00000
*................*
00000080
00C00000 0000F000 00FFFFFF FFFFFFFF
*......0.........*
00000090
FF000000 00080000 00000000 00000000
*................*
000000A0
00000000 00000000 00000000 00000000
*................*
000000B0
00010000 01800000 00000000 00000000
*................*
000000C0
62608000 60000000 00000800 00000C00
*.-..-...........*
000000D0
00000000 00000010 00000000 F0000000
*............0...*
000000E0
00000000 00000000 00080000 00000000
*................*
000000F0
00B686A2 27D36B9E A6000000 01000000
*..FS.L,.W.......*
00000100
64000000 00000000 00000000 00000000
*................*
00000110
000000A0 00000000 00000000 00000000
*................*
00000120
03003E23 80010000 10000000 C0000000
*................*
00000130
C0000000 1000000C 00010000 0F000000
*................*
00000140
00000000 00000000 00001400 01000E00
*................*
00000150
03000E00 03000100 00000000 00BFFF
*............... *
CAT08000I ZAP FUNCTION COMPLETE. RETURN CODE 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

And finally, Example 7-23 illustrates that it is also possible to delete a specific
BCS record, with the VVDSFIX DELBCSR command. In this example, the key of
the record to be deleted is specified (in this case, a cluster sphere name), and
the BCS name from which the record is to be deleted. In the output report, we
are told that a C-type (Cluster) record is deleted, and a T-type (Truename) record
for the accompanying data component is deleted.

Example 7-23 VVDSFIX DELBCSR command to delete a BCS record

//YCJRES1F JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//VVDSGO
EXEC PGM=VVDSFIX
//SYSPRINT DD SYSOUT=*
//VVDS
DD DSN=SYS1.VVDS.VSBOX94,DISP=SHR,VOL=SER=SBOX94,UNIT=3390
//CATDD
DD DSN=UCAT.CRPLUS1,DISP=SHR
//SYSIN
DD *
CMD=DELBCSR;YCJRES1.CRPLUS.TESTE.E000024;Y;CO;UCAT.CRPLUS1;
/*

142

ICF Catalog Backup and Recovery: A Practical Guide

Enter command for VVDS fixer

Enter BCS record name to be deleted.
C type BCS record DELETED. Name=YCJRES1.CRPLUS.TESTE.E000024
T type BCS record DELETED. Name=YCJRES1.CRPLUS.TESTE.E000024.DATA

A BCS record can also be deleted with the Catalog RecoveryPlus ZAP BCS
DELETE command, by specifying the key of the record to be deleted (a cluster
sphere name), and the BCS name from which the record is to be deleted. Only
the record with that key is deleted.

Example 7-24 CR+ ZAP BCS DELETE to delete a BCS record

//YCJRES1Z JOB MSGCLASS=X,NOTIFY=&SYSUID,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSIN
DD *
ZAP BCS(UCAT.CRPLUS1) DELETE(KEY(YCJRES1.CRPLUS.TESTE.E000025))
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 02 OCT 2001 PAGE:1
CAT08213I RECORD DELETED
CAT08000I ZAP FUNCTION COMPLETE. RETURN CODE 0
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 0

7.9 Case study #9: Preventive maintenance

This case study demonstrates using the Catalog RecoveryPlus DIAGNOSE
command in a way that you can establish true ICF catalog preventive
maintenance. It uses two commands that can run on a regularly-scheduled
basis, with fixes automatically created for problems encountered.
The commands are DIAGNOSE BCS-VVDS and DIAGNOSE VVDS-BCS, both
specified with masking values for the BCSs and VVDSs and the ALLRELATED
keyword. These two features permit these commands to be run over and over
again, regardless of changes made to the ICF catalog and disk environment.
Example 7-25 illustrates a CR+ DIAGNOSE BCS-VVDS command. This
command compares all records in the selected BCSs, to determine if they each
have corresponding VVR/NVR records in the appropriate VVDS. In this case, the
COMPAREBCS mask selects all BCSs that match UCAT.CRPLUS*. The

Chapter 7. Case studies

143

TOVVDS mask of SBOX** selects all volsers that begin with the value SBOX. For
all errors identified, an IDCAMS fix command is created and stored in the flat
file identified by the FIXDATASET keyword (for this run, there were no errors, so
no fixes were stored).
The command requires only two small changes to convert it into one that
diagnoses the entire environment:
COMPAREBCS(**)
ALLRELATED (instead of the current TOVVDS keyword)

With these changes, this DIAGNOSE command selects every BCS in your
environment and compares outward to every related VVDS.
How much does this cost? For Example 7-25, the total run time was less than
two elapsed minutes. This is certainly on the low side, because it was the ITSO
test environment, with fairly small BCSs and not a high number of data sets on
each volume.

Example 7-25 CR+ DIAGNOSE BCS-VVDS to search for errors

17.56.28 JOB11471 IEF403I YCJRES1B - STARTED - ASID=001B.
17.58.17 JOB11471 IEF404I YCJRES1B - ENDED - ASID=001B.
//YCJRES1B JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=*
//S1 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
DIAGNOSE BCS-VVDS COMPAREBCS(UCAT.CRPLUS*) TOVVDS(SBOX**) JOURNAL(YCJRES1.CRPLUS.DIAGNOSE.JOURNAL) FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE)
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 04 OCT 2001 PAGE:1
CAT11030I DSNS/MASKS FOR KEYWORD: COMPAREBCS
UCAT.CRPLUS1
UCAT.CRPLUS2
UCAT.CRPLUS3
UCAT.CRPLUS4
CAT11033I VOLSERS OR MASKS FOR KEYWORD: TOVVDS
SBOX**
CAT11031I THE FOLLOWING BCS(S) WILL BE DIAGNOSED:
UCAT.CRPLUS1
UCAT.CRPLUS2
UCAT.CRPLUS3
UCAT.CRPLUS4

144

ICF Catalog Backup and Recovery: A Practical Guide

CAT11032I THE FOLLOWING

SBOX01 SBOX02
SBOX11 SBOX12
SBOX18 SBOX19
SBOX28 SBOX29
SBOX35 SBOX36
SBOX45 SBOX46
SBOX52 SBOX53
SBOX62 SBOX63
SBOX69 SBOX70
SBOX79 SBOX80
SBOX86 SBOX87
SBOX96 SBOX97

VOLSERS ARE ELIGIBLE:

SBOX03 SBOX04 SBOX05 SBOX06
SBOX13 SBOX14 SBOX15 SBOX16
SBOX20 SBOX21 SBOX22 SBOX23
SBOX30 SBOX31 SBOX32 SBOX33
SBOX37 SBOX38 SBOX39 SBOX40
SBOX47 SBOX48 SBOX49 SBOX50
SBOX54 SBOX55 SBOX56 SBOX57
SBOX64 SBOX65 SBOX66 SBOX67
SBOX71 SBOX72 SBOX73 SBOX74
SBOX81 SBOX82 SBOX83 SBOX84
SBOX88 SBOX89 SBOX90 SBOX91

SBOX07
SBOX17
SBOX24
SBOX34
SBOX41
SBOX51
SBOX58
SBOX68
SBOX75
SBOX85
SBOX92

SBOX08 SBOX09 SBOX10

SBOX25 SBOX26 SBOX27
SBOX42 SBOX43 SBOX44
SBOX59 SBOX60 SBOX61
SBOX76 SBOX77 SBOX78
SBOX93 SBOX94 SBOX95

CAT11040I 17.56.32 DIAG BCS-VVDS STEP: BCS READ - STARTED

CAT11040I 17.57.20 DIAG BCS-VVDS STEP: BCS READ - COMPLETED
CAT11040I 17.57.23 DIAG BCS-VVDS STEP: VVDS EXTRACT - STARTED
CAT11040I 17.57.30 DIAG BCS-VVDS STEP: VVDS EXTRACT - COMPLETED
CAT11040I 17.57.30 DIAG BCS-VVDS STEP: VTOC EXTRACT - STARTED
CAT11040I 17.58.13 DIAG BCS-VVDS STEP: VTOC EXTRACT - COMPLETED
CAT11040I 17.58.13 DIAG BCS-VVDS STEP: SORT - STARTED
CAT11040I 17.58.17 DIAG BCS-VVDS STEP: SORT - COMPLETED
BCS-VVDS MAPPING
BCS NAME
UCAT.CRPLUS1

UCAT.CRPLUS2

VOLSER
SBOX01
SBOX02
SBOX03
SBOX04
SBOX11
SBOX20
SBOX26
SBOX27
SBOX36
SBOX38
SBOX75
SBOX77
SBOX79
SBOX81
SBOX94
SBOX01
SBOX02
SBOX20
SBOX36
SBOX38
SBOX75
SBOX77
SBOX79
SBOX81

VVDS IN
BCS
YES
YES
NO
NO
NO
YES
YES
YES
NO
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES

BCS IN
VVDS
YES
YES
NO
NO
NO
YES
YES
YES
NO
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES

DATASETS FIX
ON VOLUME CREATED
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
N/A
YES
YES
YES
YES
YES
YES
YES
YES
YES

N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A

Chapter 7. Case studies

145

SBOX94

YES

YES

YES

N/A

UCAT.CRPLUS3

SBOX01
SBOX02
SBOX03
SBOX04
SBOX20
SBOX38
SBOX75
SBOX77
SBOX79
SBOX81

YES
NO
YES
NO
NO
NO
NO
NO
NO
NO

YES
NO
YES
NO
NO
NO
NO
NO
NO
NO

YES
YES
YES
YES
YES
YES
YES
YES
YES
YES

N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A

UCAT.CRPLUS4

SBOX01
SBOX02
SBOX20
SBOX38
SBOX75

YES
NO
NO
NO
NO

YES
NO
NO
NO
NO

YES
YES
YES
YES
YES

N/A
N/A
N/A
N/A
N/A

Example 7-26 illustrates a CR+ DIAGNOSE VVDS-BCS command. This

command compares all records in the selected VVDSs, to determine if they each
have corresponding catalog records in the appropriate BCS. In this case, the
COMPAREVVDS mask selects all volumes that match SBOX*, which is all of the
volumes available to us in our ITSO test environment. The ALLRELATED
keyword indicates that all BCS that are related to these volumes will be
diagnosed. For all errors identified, an IDCAMS fix command will be created and
stored in the flat file identified by the FIXDATASET keyword.
For this run, a total of 415 errors were identified, which may seem very large. It is
a safe guess to say that almost every installation that runs a command such as
this will encounter a high number of such errors when the command is run for the
first time. For years, there was too little attention given to keeping errors like this
under control; given the current situation, there are likely to be large numbers of
orphaned data sets, wasted disk space, and errors lying around that can cause
problems when you least expect. Once the environment is cleaned up,
subsequent and regularly-scheduled runs should not find so many errors.
This command requires only a single small change to convert it into one that will
diagnose the entire environment:
COMPAREVVDS(**)

This DIAGNOSE command selects every VVDS in your environment and

compares outward to every related BCS.
How much does this cost? For Example 7-26, the total run time was
approximately 6.5 elapsed minutes. This is certainly on the low side, because it
was the ITSO test environment, with a fairly small number of volumes and not a
high number of data sets on each volume.

146

ICF Catalog Backup and Recovery: A Practical Guide

Example 7-26 CR+ DIAGNOSE VVDS-BCS to search for errors

17.53.22 JOB11469 IEF403I YCJRES1C - STARTED - ASID=0019.
17.59.56 JOB11469 IEF404I YCJRES1C - ENDED - ASID=0019.
//YCJRES1C JOB MSGCLASS=X,NOTIFY=YCJRES1,RESTART=S0
//S2 EXEC PGM=CAT00010
//SYSPRINT DD SYSOUT=*
//SYSOUT
DD SYSOUT=*
//SYSIN
DD *
DIAGNOSE VVDS-BCS COMPAREVVDS(SBOX*) ALLRELATED FIXDATASET(YCJRES1.CRPLUS.DIAGNOSE.FIXFILE2)
/*
CAT01001I Catalog RecoveryPlus MAINSTAR SOFTWARE CORPORATION 04 OCT 2001 PAGE:1
CAT11116I THE FOLLOWING ENTRIES/MASKS ARE TO BE PROCESSED DURING COMPAREVVDS
PROCESSING.
SBOX*
CAT11103I THE FOLLOWING VOLUMES HAVE MET ALL SELECTION REQUIREMENTS AND WILL BE
PROCESSED:
SBOX01 SBOX02 SBOX03 SBOX04 SBOX05 SBOX06 SBOX07 SBOX08 SBOX09
SBOX10 SBOX11 SBOX12 SBOX13 SBOX14 SBOX15 SBOX16
SBOX17 SBOX18 SBOX19 SBOX20 SBOX21 SBOX22 SBOX23 SBOX24 SBOX25
SBOX26 SBOX27 SBOX28 SBOX29 SBOX30 SBOX31 SBOX32
SBOX33 SBOX34 SBOX35 SBOX36 SBOX37 SBOX38 SBOX39 SBOX40 SBOX41
SBOX42 SBOX43 SBOX44 SBOX45 SBOX46 SBOX47 SBOX48
SBOX49 SBOX50 SBOX51 SBOX52 SBOX53 SBOX54 SBOX55 SBOX56 SBOX57
SBOX58 SBOX59 SBOX60 SBOX61 SBOX62 SBOX63 SBOX64
SBOX65 SBOX66 SBOX67 SBOX68 SBOX69 SBOX70 SBOX71 SBOX72 SBOX73
SBOX74 SBOX75 SBOX76 SBOX77 SBOX78 SBOX79 SBOX80
SBOX81 SBOX82 SBOX83 SBOX84 SBOX85 SBOX86 SBOX87 SBOX88 SBOX89
SBOX90 SBOX91 SBOX92 SBOX93 SBOX94 SBOX95 SBOX96
SBOX97
CAT11129I PROCESSING STARTING FOR VVDS SYS1.VVDS.VSBOX01
CAT11144I VTOC DATA: F1=300
FX=4,201
CVAF CALLS=91
CAT11122I PRELIMINARY CHARACTERISTICS OF VVDS SYS1.VVDS.VSBOX01
SMS MANAGED NO
CONTROL INTERVALS ALLOCATED 120
CATALOGS REFERENCING 15
CAT11119W VTOC ENTRY ANGIEB.DDIR.I IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 1
CAT11119W VTOC ENTRY DB2QM.UNLOAD.MODEL IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 2
CAT11119W VTOC ENTRY HERING.GDG.TESTDATA IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 3
CAT11119W VTOC ENTRY PAOLOR3.ALLRECS.GDGMODEL IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 4
CAT11119W VTOC ENTRY TATERS3.GDG1 IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 41
CAT11150I ONE OR MORE DELETE...VVR OR DELETE...NVR COMMANDS HAVE BEEN GENERATED
FOR THIS VVDS

Chapter 7. Case studies

147

CAT11123I FINAL CHARACTERISTICS OF VVDS SYS1.VVDS.VSBOX01

PRIMARY VVRS (Z) ENCOUNTERED 296
SECONDARY VVRS (Q) ENCOUNTERED 38
NON VSAMS (N) ENCOUNTERED 0
VVCNS ENCOUNTERED 0
VVCMS ENCOUNTERED 0
** DIAGNOSE VVDS-BCS SYS1.VVDS.VSBOX01 **
UNKNOWN TYPES ENCOUNTERED 0
CAT11125I CATALOG SUMMARY FOR VVDS SYS1.VVDS.VSBOX01
- - - - - C A T A L O G N A M E - - - - - VVR/NVR COUNT
CATALOG.TOTICF1.VTOTCAT
0 NO VVRS OR NVRS REFERENCE THIS CATALOG
MCAT.CRPLUS.VSBOX01
2
MCAT.CRPLUS.VSBOX03
0 NO VVRS OR NVRS REFERENCE THIS CATALOG
MCAT.SANDBOX.R10.VSBOX11 2
MCAT.SANDBOX.R9.VSBOX11 4
MCAT.SANDBOX.VSBOX01
20
MCAT.SANDBOX.VSBOX05
2 COULD NOT BE DYNAMICALLY ALLOCATED
COULD NOT BE OPENED FOR PROCESSING
UCAT.CRPLUS1
51
UCAT.CRPLUS2
6
UCAT.CRPLUS3
2
UCAT.CRPLUS4
2
UCAT.TATERS2
2
UCAT.VSBOX01
36
UCAT.VSBOX09
203
UCAT.VSBOX23
2
CAT11129I PROCESSING COMPLETE FOR VVDS SYS1.VVDS.VSBOX01
CAT11129I PROCESSING STARTING FOR VVDS SYS1.VVDS.VSBOX02
CAT11144I VTOC DATA: F1=2,543
FX=1,955
CVAF CALLS=91
CAT11122I PRELIMINARY CHARACTERISTICS OF VVDS SYS1.VVDS.VSBOX02
SMS MANAGED NO
CONTROL INTERVALS ALLOCATED 120
CATALOGS REFERENCING 15
CAT11132W UNABLE TO LOCATE COMPONENT (Z) IXGLOGR.CICSTT11.DFHLOG.SC63.DATA IN
BCS MCAT.SANDBOX.VSBOX01
CAT11137W UNABLE TO LOCATE BASE CLUSTER IXGLOGR.CICSTT11.DFHLOG.SC63 IN BCS
MCAT.SANDBOX.VSBOX01 FOR COMPONENT IXGLOGR.CICSTT11.D
CAT11137W LOG.SC63.DATA
CAT11184I CORRESPONDING FIX: 42
CAT11132W UNABLE TO LOCATE COMPONENT (Z) IXGLOGR.CICSTT21.DFHLOG.SC63.DATA IN
BCS MCAT.SANDBOX.VSBOX01
CAT11137W UNABLE TO LOCATE BASE CLUSTER IXGLOGR.CICSTT21.DFHLOG.SC63 IN BCS
MCAT.SANDBOX.VSBOX01 FOR COMPONENT IXGLOGR.CICSTT21.D
CAT11137W LOG.SC63.DATA
CAT11184I CORRESPONDING FIX: 43
CAT11119W VTOC ENTRY ANDREVN.BRODCAST IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 44
CAT11119W VTOC ENTRY TROWELL.ISPF.ISPPROF IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 101
CAT11150I ONE OR MORE DELETE...VVR OR DELETE...NVR COMMANDS HAVE BEEN GENERATED

148

ICF Catalog Backup and Recovery: A Practical Guide

FOR THIS VVDS

CAT11123I FINAL CHARACTERISTICS
PRIMARY VVRS (Z)
SECONDARY VVRS (Q)
NON VSAMS (N)
VVCNS
VVCMS
UNKNOWN TYPES

OF VVDS SYS1.VVDS.VSBOX02
ENCOUNTERED 816
ENCOUNTERED 6
ENCOUNTERED 0
ENCOUNTERED 0
ENCOUNTERED 0
ENCOUNTERED 0

CAT11125I CATALOG SUMMARY FOR VVDS SYS1.VVDS.VSBOX02

- - - - - C A T A L O G N A M E - - - - - VVR/NVR COUNT
CATALOG.SHRICF1.VIODFPK
1
CATALOG.TOTICF2.VTOTCAT
0 NO VVRS OR NVRS REFERENCE THIS CATALOG
MCAT.CRPLUS.VSBOX03
0 NO VVRS OR NVRS REFERENCE THIS CATALOG
MCAT.SANDBOX.R10.VSBOX11 74
MCAT.SANDBOX.VSBOX01
3
MCAT.SANDBOX.VSBOX11
1
MCAT.SBOXALT.SBOX02
2
UCAT.CRPLUS1
562
UCAT.CRPLUS2
142
UCAT.CRPLUS5
0 NO VVRS OR NVRS REFERENCE THIS CATALOG
UCAT.TEST.VSBOX02
2
UCAT.VSBOX01
25
UCAT.VSBOX02.TEST01
2
UCAT.VSBOX09
6
UCAT.VSTEST1
2
CAT11129I PROCESSING COMPLETE FOR VVDS SYS1.VVDS.VSBOX02
CAT11129I PROCESSING STARTING FOR VVDS SYS1.VVDS.VSBOX97
CAT11144I VTOC DATA: F1=43
FX=4,458
CVAF CALLS=91
IKJ56228I DATA SET SYS1.VVDS.VSBOX97 NOT IN CATALOG OR CATALOG CAN NOT BE
ACCESSED
CAT11197W ATTEMPT FOR DYNAMIC ALLOCATION (1) HAS FAILED FOR SYS1.VVDS.VSBOX97
CAT11119W VTOC ENTRY BBO.SC64.SBBOHFS IS NOT CATALOGUED
CAT11184I CORRESPONDING FIX: 415
CAT11150I ONE OR MORE DELETE...VVR OR DELETE...NVR COMMANDS HAVE BEEN GENERATED
FOR THIS VVDS
CAT11123I FINAL CHARACTERISTICS OF VVDS SYS1.VVDS.VSBOX97
PRIMARY VVRS (Z) ENCOUNTERED 0
SECONDARY VVRS (Q) ENCOUNTERED 0
NON VSAMS (N) ENCOUNTERED 0
VVCNS ENCOUNTERED 0
VVCMS ENCOUNTERED 0
UNKNOWN TYPES ENCOUNTERED 0
CAT11129I PROCESSING COMPLETE FOR VVDS SYS1.VVDS.VSBOX97
CAT10009I DIAGNOSE FUNCTION COMPLETE. RETURN CODE 4
CAT01009I Catalog RecoveryPlus EXECUTION COMPLETE. HIGHEST RETURN CODE WAS 4

Chapter 7. Case studies

149

150

ICF Catalog Backup and Recovery: A Practical Guide

Related publications
The publications listed in this section are considered particularly suitable for a
more detailed discussion of the topics covered in this redbook.

IBM Redbooks
For information on ordering these publications, see How to get IBM Redbooks
on page 152.
DFSMS/MVS Recovery Management, GG24-3922
Enhanced Catalog Sharing and Management, SG24-5594
IBM Magstar Tape Products Family: A Practical Guide, SG24-4632

Other resources
These publications are also relevant as further information sources:
MVS System Commands, SA22-7629
MVS System Management Facilities (SMF), SA22-7630
DFSMS Access Method Services for Catalogs, SC26-7394
DFSMSrmm Implementation and Customization Guide, SC26-7405
DFSMS Managing Catalogs, SC26-7409
DFSMS Using Data Sets, SC26-7410
DFSMShsm Storage Administration Guide, SC35-0421
DFSMShsm Storage Administration Reference, SC35-0422
DFSMSdss Storage Administration Guide, SC35-0423
DFSMS OAM Planning, Installation and Storage Adminstration Guide for
Tape Libraries, SC35-0427
Integrated Catalog Forward Recovery Utility Program Description/Operations
Manual, SH20-6952
Mainstar Catalog RecoveryPlus User Guide, document number 006-0202

This publication is available from Mainstar Software Corporation, which you

can find on the Web at: http://www.mainstar.com/

Copyright IBM Corp. 1999, 2001

151

Referenced Web sites

You may refer to the following sources for relevant information:
Catalog and VSAM Knowledge Base site:
http://knowledge.storage.ibm.com/
Mainstar Software Corporation: http://www.mainstar.com

How to get IBM Redbooks

You can order hardcopy Redbooks, as well as view, download, or search for
Redbooks at the following Web site:
ibm.com/redbooks

You can also download additional materials (code samples or diskette/CD-ROM

images) from that site.

IBM Redbooks collections

Redbooks are also available on CD-ROMs. Click the CD-ROMs button on the
Redbooks Web site for information about all the CD-ROMs offered, as well as
updates and formats.

152

ICF Catalog Backup and Recovery: A Practical Guide

Special notices
References in this publication to IBM products, programs or services do not imply
that IBM intends to make these available in all countries in which IBM operates.
Any reference to an IBM product, program, or service is not intended to state or
imply that only IBM's product, program, or service may be used. Any functionally
equivalent program that does not infringe any of IBM's intellectual property rights
may be used instead of the IBM product, program or service.
Information in this book was developed in conjunction with use of the equipment
specified, and is limited in application to those specific hardware and software
products and levels.
IBM may have patents or pending patent applications covering subject matter in
this document. The furnishing of this document does not give you any license to
these patents. You can send license inquiries, in writing, to the IBM Director of
Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact IBM Corporation, Dept.
600A, Mail Drop 1329, Somers, NY 10589 USA.
Such information may be available, subject to appropriate terms and conditions,
including in some cases, payment of a fee.
The information contained in this document has not been submitted to any formal
IBM test and is distributed AS IS. The use of this information or the
implementation of any of these techniques is a customer responsibility and
depends on the customer's ability to evaluate and integrate them into the
customer's operational environment. While each item may have been reviewed
by IBM for accuracy in a specific situation, there is no guarantee that the same or
similar results will be obtained elsewhere. Customers attempting to adapt these
techniques to their own environments do so at their own risk.
Any pointers in this publication to external Web sites are provided for
convenience only and do not in any manner serve as an endorsement of these
Web sites.

Copyright IBM Corp. 1999, 2001

153

The following terms are trademarks of other companies:

Mainstar, FastAudit/390, and Catalog RecoveryPlus are registered trademarks of
Mainstar Software Corporation in the United States and/or other countries.
C-bus is a trademark of Corollary, Inc. in the United States and/or other
countries.
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States and/or other
countries.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of
Microsoft Corporation in the United States and/or other countries.
PC Direct is a trademark of Ziff Communications Company in the United States
and/or other countries and is used by IBM Corporation under license.
ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel
Corporation in the United States and/or other countries.
UNIX is a registered trademark in the United States and other countries licensed
exclusively through The Open Group.
SET, SET Secure Electronic Transaction, and the SET Logo are trademarks
owned by SET Secure Electronic Transaction LLC.
Other company, product, and service names may be trademarks or service
marks of others

154

ICF Catalog Backup and Recovery: A Practical Guide

Glossary
This glossary contains a list of terms and
abbreviations used within this redbook.
A
access method control block (ACB) A
control block that links an application program
to VSAM or VTAM.
access method services (AMS) A
multifunction service program that manages
both VSAM and non-VSAM data sets and ICF
catalogs. It defines data sets and allocates
space for VSAM data sets and ICF catalogs. It
converts indexed-sequential (ISAM) data sets
to key-sequenced data sets, modifies data set
attributes in the catalog, reorganizes data sets,
facilitates data portability among operating
systems, creates backup copies of data sets
and indexes, helps make inaccessible data
sets accessible, lists the records of data sets
and catalogs, defines, and builds alternate
indexes.

automated tape library (ATL) A device

consisting of robotic components, cartridge
storage areas, tape subsystems, and
controlling hardware and software, together
with the set of tape volumes that reside in the
library and can be mounted on the library tape
drives.
automatic class selection (ACS) A
mechanism for assigning SMS classes and
storage groups.
automatic class selection routine A
procedural set of ACS language statements.
Based on a set of input variables, the ACS
language statements generate the name of a
predefined SMS class, or a list of names of
predefined storage groups, for a data set.
availability For a storage subsystem, the
degree to which a data set or object can be
accessed when requested by a user.
B

ACS routine See automatic class selection

routine.
alias An alternative name for an entry or for a
member of a partitioned data set (PDS).
alias entry. An entry that relates an alias to
the real entry name of a user catalog or
non-VSAM data set.

Copyright IBM Corp. 1999, 2001

backup data set A copy that can be used to

replace or reconstruct a damaged data set.
base cluster A key-sequenced data set or
entry-sequenced data set over which one or
more alternate indexes are built.
basic catalog structure (BCS) The name of
the catalog structure in the integrated catalog
facility environment. See also ICF catalog.

155

C
catalog A data set that contains extensive
information required to locate other data sets,
to allocate and deallocate storage space, to
verify the access authority of a program or
operator, and to accumulate data set usage
statistics. See also master catalog and user
catalog.
catalog address space (CAS) A separate
address space in virtual storage that contains
catalog management modules and control
blocks.
catalog connector A catalog entry, called
either a user catalog entry or a catalog
connector entry, in the master catalog that
points to a user catalog. Contains information
on aliases for the user catalog.

component The components of an object

are usually referred to as the objects data
component and index component. Also, the
cluster, data, or index fields of a subrecord.
control area (CA) A group of control
intervals used as a unit for formatting a data
set before adding records to it. Also, in a
key-sequenced data set, the set of control
intervals pointed to by a sequence-set index
record adjacent to its data.
control interval (CI) A fixed-length area of
auxiliary storage space in which VSAM stores
records. It is the unit of information (an integer
multiple of block size) transmitted to or from
auxiliary storage by VSAM.
CR+ Catalog RecoveryPlus.
D

cell An occurrence of information such as

passwords, volume information, or
associations.
cluster In VSAM, a named structure
consisting of a group of related components.
For example, when the data is
key-sequenced, the cluster contains both the
data and the index components; for data that
is entry-sequenced, the cluster contains only a
data components.
cluster entry A catalog entry that contains
information about a key-sequenced or
entry-sequenced VSAM cluster: ownership,
cluster attributes, and the clusters passwords
and protection attributes. A key-sequenced
cluster entry points to both a data and an
index entry. An entry-sequenced cluster entry
points to a data entry only.

156

ICF Catalog Backup and Recovery: A Practical Guide

DASD volume A DASD space identified by a

common label and accessed by a set of
related addresses.
data component The part of a VSAM data
set, alternate index, or catalog that contains
the objects data records.
data entry A catalog entry that describes the
data component of a cluster, alternate index,
page spaces, or catalog. A data entry contains
the data components attributes, allocation and
extent information, and statistics. A data entry
for a clusters or catalogs data component can
also contain the data components passwords
and protection attributes.

Data Facility Sort (DFSORT) An IBM

licensed program that is a high-speed data
processing utility. DFSORT provides an
efficient and flexible way to handle sorting,
merging, and copying operations, as well as
providing versatile data manipulation at the
record, field, and bit level.
data facility storage management
subsystem (DFSMS) An operating
environment that helps automate and
centralize the management of storage. To
manage storage, SMS provides the storage
administrator with control over data class,
storage class, management class, storage
group, and automatic class selection routine
definitions.
data set In DFSMS, the major unit of data
storage and retrieval, consisting of a collection
of data in one of several prescribed
arrangements and described by control
information to which the system has access. In
non-z/OS UNIX System Services/MVS
environments, the terms data set and file are
generally equivalent and sometimes are used
interchangeably. See also file.
DFSORT See Data Facility Sort.
DFSMS See data facility storage
management subsystem.
DFSMSdfp A DFSMS functional component
or base element of z/OS that provides
functions for storage management, data
management, program management, device
management, and distributed data access.
DFSMSdss A DFSMS functional component
or base element of z/OS, used to copy, move,
dump, and restore data sets and volumes.

DFSMShsm A DFSMS functional

component or base element of z/OS, used for
backing up and recovering data, and
managing space on volumes in the storage
hierarchy.
DFSMSrmm A DFSMS functional
component or base element of z/OS, that
manages removable media.
E
enhanced catalog sharing (ECS) Sharing
protocol that stores the information that
describes changes to a shared catalog in the
coupling facility. The I/O required for the VVDS
mode protocol is eliminated, resulting in better
sysplex-wide performance.
entry A collection of information about a
cataloged object in a master or user catalog.
entry name A unique name for each
component or object as it is defined in a
catalog. The entry name is the same as the
dsname in a DD statement that describes the
object.
entry sequence The order in which data
records are physically arranged (according to
ascending RBA) in auxiliary storage, without
respect to their contents.
entry-sequenced data set (ESDS) In
VSAM, a data set whose records are loaded
without respect to their contents, and whose
RBAs cannot change. Records are retrieved
and stored by addressed access, and new
records are added at the end of the data set.
export To create a backup or portable copy
of a VSAM cluster, alternate index, or ICF
catalog.

Glossary

157

extended addressability (EA) The ability to

create and access a VSAM data set that is
greater than 4 GB in size. Extended
addressability data sets must be allocated with
DSNTYPE=EXT and EXTENDED
ADDRESSABILITY=Y.
extended format (EF) The format of a data
set that has a data set name type (DSNTYPE)
of EXTENDED. The data set is structured
logically the same as a data set that is not in
extended format but the physical format is
different.
extent A continuous space allocated on a
DASD volume occupied by a data set or
portion of a data set. An extent of a data set
contains a whole number of control areas.
F
field In a record or a control block, a
specified area used for a particular category of
data or control information.
file A collection of information treated as a
unit. In z/OS non-UNIX environments, the
terms data set and file are generally
equivalent and are sometimes used
interchangeably.
filtering The process of selecting data sets
based on specified criteria. These criteria
consist of fully or partially-qualified data set
names or of certain data set characteristics.
G
generation data group (GDG) A collection
of historically related non-VSAM data sets that
are arranged in chronological order; each data
set is called a generation data set.

158

ICF Catalog Backup and Recovery: A Practical Guide

generation data set (GDS) One of the data

sets in a generation data group; it is
historically related to the others in the group.
global resource serialization (GRS) A
component of z/OS used for serializing use of
system resources and for converting hardware
reserve on DASD volumes to data set
enqueues.
I
ICF catalog A catalog that is composed of a
basic catalog structure (BCS) and its related
volume tables of contents (VTOCs) and VSAM
volume data sets (VVDSs).
import To restore a VSAM cluster, alternate
index, or integrated catalog facility catalog
from a portable data set created by the
EXPORT command.
index entry A catalog entry that describes
the index component of a key-sequenced
cluster, alternate index, or catalog. An index
entry contains the index components
attributes, passwords, allocation and extent
information, and statistics.
indexed VTOC A volume table of contents
with an index that contains a list of data set
names and free space information, which
allows data sets to be located more efficiently.
Integrated Catalog Forward Recovery
Utility (ICFRU) Program offering that
provides a mechanism to forward recovery an
ICF catalog from a backup copy.

K
key One or more characters within an item of
data that are used to identify it or control its
use. As used in this publication, one or more
consecutive characters taken from a data
record, used to identify the record and
establish its order with respect to other
records.
key sequence The collating sequence of
data records, determined by the value of the
key field in each of the data records. It can be
the same as, or different from, the entry
sequence of the records.
key-sequenced data set (KSDS) A VSAM
data set whose records are loaded in key
sequence and controlled by and index.
Records are retrieved and stored by keyed
access or by addressed access, and new
records are inserted in the data set in key
sequence because of free space allocated in
the data set. Relative byte addresses of
records can change because of control
interval or control area splits.

multilevel alias facility (MLA) A function in

catalog address space that allows ICF catalog
selection based on one to four data set name
qualifiers.
N
non-SMS volume A volume that is not
controlled by SMS.
non-VSAM entry A catalog entry that
describes a non-VSAM data set. A non-VSAM
entry contains the data sets volume serial
number and device type. If the data set
resides on a magnetic tape volume, the entry
can also identify the data sets file number.
non-VSAM volume record (NVR) A VVDS
record that contains SMS-related information
about a non-VSAM, system-managed data
set.
O
object A named style stream having no
specific format or record orientation.

linear data set (LDS) A VSAM data set that

contains data but no control information. A
linear data set can be accessed as a
byte-addressable string in virtual storage.

object access method (OAM) An access

method that provides storage, retrieval, and
storage hierarchy management for objects
and provides storage and retrieval
management for tape volumes contained in
system-managed libraries.

LRECL Logical record length.

partitioned data set (PDS) A data set on

direct access storage that is divided into
partitions, called members, each of which can
contain a program, part of a program, or data.

master catalog A catalog that contains

extensive data set and volume information that
VSAM requires to locate data sets, to allocate
and deallocate storage space, to verify the
authorization of a program or operator to gain
access to a data set, and accumulate usage
statistics for data sets.

Glossary

159

partitioned data set extended (PDSE) A

system-managed data set that contains an
indexed directory and members that are
similar to the directory and members of
partitioned data sets. A PDSE can be used
instead of a partitioned data set.
portability The ability to use VSAM data sets
with different operating systems.
portable data set A data set that can be
transported between systems using access
method services.
R
record A set of data treated as a unit.
record level sharing (RLS) An extension to
VSAM that provides direct record-level sharing
of VSAM data sets from multiple address
spaces across multiple systems. Record-level
sharing uses the System/390 Coupling Facility
to provide cross-system locking, local buffer
invalidation, and cross-system data caching.
recovery The process of rebuilding data
after it is damaged or destroyed, often by
using a backup copy of the data or by
reapplying transactions recorded in a log.
relative byte address (RBA) The
displacement of a data record or a control
interval from the beginning of the data set to
which it belongs; independent of the manner in
which the data set is stored.

system-managed storage Storage

managed by SMS. SMS attempts to deliver
required services for availability, performance,
and space to applications.
system-managed tape library A collection
of tape volumes and tape devices, defined in
the tape configuration database. A
system-managed tape library can be
automated or manual.
system management facilities (SMF) A
component of z/OS that collects input/output
(I/O) statistics, provided at the data set and
storage class levels, which helps you monitor
the performance of the direct access storage
subsystem.
T
tape configuration database (TCDB) One
or more volume catalogs used to maintain
records of system-managed tape libraries and
tape volumes.
tape library A set of equipment and facilities
that support an installations tape environment.
This can include tape storage racks, a set of
tape drives, and a set of related tape volumes
mounted on those drives.

160

storage management subsystem (SMS) A

DFSMS facility used to automate and
centralize the management of storage. Using
SMS, a storage administrator describes data
allocation characteristics, performance and
availability goals, backup and retention
requirements, and storage requirements to the
system through data class, storage class,
management class, storage group, and ACS
routines definitions.

ICF Catalog Backup and Recovery: A Practical Guide

U
user catalog An optional catalog used in the
same way as the master catalog and pointed
by the master catalog. It lessens the
contention for the master catalog and
facilitates volume portability.
user catalog connector See catalog
connector.
V
volume table of contents (VTOC) A table
on a direct access volume that describes each
data set on the volume.
virtual storage access method (VSAM) An
access method for indexed or sequential
processing of fixed and variable-length
records on direct access devices. The records
in a VSAM data set or file can be organized in
logical sequence by a key field (key
sequence), in the physical sequence in which
they are written on the data set or file
(entry-sequence), or by relative-record
number.
VSAM record level sharing See record level
sharing.
VSAM volume control record (VVCR) The
first logical record in the VVDS that contains
information to manage DASD space and the
BCS back pointers.
VSAM volume data set (VVDS) A data set
that describes the characteristics of VSAM
and system-managed data sets residing on a
given DASD volume; part of an ICF catalog.
VSAM volume record (VVR)
logical record within a VVDS.

A VSAM

Glossary

161

162

ICF Catalog Backup and Recovery: A Practical Guide

Index
Numerics
3494 Automated Tape Library 18
3494 Virtual Tape Server 18
3495 Automated Tape Library 18

A
Access Method Services 3
alias 4, 6, 52, 88
ALIAS keyword 53
allocation process 2
alternate index 4
alternate master catalog 18, 106, 135
AMS
See Access Method Services
ATL
See Automated Tape Library
Automated Tape Library 18

B
backup
frequency 12, 16
integrity 14
number of copies 16
portable copy 62
recent version 51
strategy 12
techniques 12
utilities 1213
VOLCAT 18
VVDS 19
basic catalog structure 2
BCS
See basic catalog structure
BCS backup and forward recovery 116
blocksize 2
buffers 70

C
CA
See control area
CAS
See Catalog Address Space

Copyright IBM Corp. 1999, 2001

case studies 111

catalog 1
allocation parameters 86
back-pointers 104
close 7
delete 53
diagnosis 22
lock 64
locking 52
maintenance 69
missing updates 58
open 7
physical allocated size 86
recovery 17
renaming 86
reorganize 86
repair 86
RESERVE 14
salvage 54
self-describing record 14
shared access control 37
shared access integrity 37
sharing 7, 37
structure 2
tasks list 43
unlock 52
catalog access 52
Catalog Address Space 6, 42
restart 7
catalog address space ABEND 42
catalog data space cache 7
catalog management 4, 6, 22, 26
Catalog RecoveryPlus 50
BACKUP BCS command 76
BACKUP command 75, 114
BACKUP VVDS command 78
DIAGNOSE ALIAS command 89
DIAGNOSE command 96
introduction 70
ISPF interface 71
Main Menu panel 71
MERGECAT command 102
RACF FACILITY class profiles 74
RECOVER command 85

163

recovery simulation 94
Submit Backup panel 72
Submit Recover BCS panel 73
Submit Recovery VVDS panel 74
SUPERCLIP command 106
ZAP command 107
ZAP facility 51
CDSC
See catalog data space cache
changing a disk volser 131
CI
See control interval
CISIZE attribute 53
cluster 4
compaction 8
COMPAREDD keyword 30
COMPAREDS keyword 30
connector record 5, 18, 52
content errors 26
control area 24
control blocks 6
control interval 24
coupling facilities 8
CR+ BACKUP BCS command
alias processing 85
backing up all catalogs 77
different frequencies 78
EXCLUDEBCS keyword 77
explicitly-specified BCSs 76
OUTFILE keyword 76
processing 85
CR+ BACKUP command
ACCEPTDIAGNOSE keyword 80
ACCEPTEXAMINE keyword 80
DIAGNOSEBCS keyword 81
DIAGNOSEVVDS keyword 81
duplex BCS 84
duplex VVDS 84
EXAMINE keyword 80
multiple backup copies 82
NORESERVE keyword 81
OUTDATASET keyword 76
OUTFILE keyword 76
RESERVE keyword 81
specifying BCSs 76
CR+ BACKUP VVDS command
EXCLUDEVVDS keyword 79
OUTFILE keyword 78
specifying VVDSs 78

164

ICF Catalog Backup and Recovery: A Practical Guide

CR+ DIAGNOSE ALIAS command

DEFINE-ALIAS keyword 99
DELETE-ALIAS keyword 99
error analysis 98
EXCLUDE keyword 98
INCLUDE keyword 98
NONPEER mode 97
PEER mode 97
CR+ DIAGNOSE command
ALLRELATED keyword 100
DEFINE-RECATALOG keyword 100
DELETE-ENTRIES keyword 99
DIAGNOSE ALIAS 97
DIAGNOSE BCS-VVDS 99
DIAGNOSE VVDS-BCS 99
fix commands 96
selecting BCSs and VVDSs 100
CR+ MERGECAT command
alias handling 103
alternate master catalog 106
BACKOUT keyword 104
COPYONLY keyword 102, 106
ENTRIES keyword 103
journal file 104
LOCK keyword 106
NOLOCK keyword 106
RESTART keyword 104
SIMULATE keyword 105
CR+ RECOVER command
alias handling 88
ALIASONLY keyword 89
BCS rename 89
explicit/implicit DEFINE 87
FORWARD keyword 91
INDATASET keyword 86
INFILE keyword 86
LOCK keyword 88
MASTERCATALOG keyword 89
NEWNAME keyword 89
NOLOCK keyword 88
PRINT keyword 93
PRINTing BCS records 93
re-sizing the VVDS 95
SIMULATE keyword 94
specifying BCS or VVDS 87
CR+ SUPERCLIP command, changing a volser
107
CR+ ZAP command
deleting a BCS and VVDS record 108

patching a BCS and VVDS records 109

printing BCS and VVDS records 108
create VVDS entry 32

D
daily backup 12
data component 4, 25
data component name 36
data control areas 25
data control intervals 25
data set description 5
data set naming conventions 6
data set organization 2
DATATEST keyword 25
delete VVDS entry 32
dependent records 33, 36
device type 8
DFSMSdss
logical dump 15
logical restore 53
stand-alone 18
stand-alone restore 55
DFSMShsm
activity logs 16
availability management 16
BACKDS command 16
backup 51
recovery 54
DFSORT 58
DIAGNOSE BCS compare 31
DIAGNOSE of a BCS 27
DIAGNOSE of a VVDS 29
DIAGNOSE VVDS compare 34
diagnosis utilities 23
diagnostic commands 14
disaster recovery plan 70
down time 50
DSCB 29
duplicate key 77
duplicate records 25
duplicate VVR 29, 37
duplicated records 41

E
ECS
See Enhanced Catalog Sharing
Enhanced Catalog Sharing 39
enlarging VVDS 129

ensuring good backups 113

entry missing 30
error symptoms 40
error-free backup 62
ERRORLIMIT keyword 25
exclusive enqueue 81
extent information 29

F
forward recover 86
forward recovery 19, 52, 57, 62, 87
FREESPACE attribute 53

G
GDG
See generation data group
generation data group 4
GLOBAL ENQUEUE 14
Global Resource Serialization 14
GRS
See Global Resource Serialization
guaranteed space attribute 54

H
high level qualifier 4
high-used relative byte address 19
horizontal pointers 25
HURBA
See high-used relative byte address

I
ICF
See Integrated Catalog Facility
ICFRRAP program 62, 65
ICFRRSV program 62, 65
ICFRU
See Integrated Catalog Forward Recovery Utility
ICKDSF utility 107
IDCAMS commands 3, 8
ALTER LOCK 52, 88
ALTER UNLOCK 52, 66
DEFINE CLUSTER 32, 88
DEFINE RECATALOG 29
DEFINE UCAT 52
DEFINE USERCATALOG 18, 54, 88
DELETE 32
DELETE NOSCRATCH 29

Index

165

DELETE RECOVERY 87
DELETE TRUENAME 29
DELETE UCAT RECOVERY 52
DIAGNOSE 14, 26, 30, 58, 113
DIAGNOSE ICFCATALOG 28, 31, 81
DIAGNOSE VVDS 29, 81
EXAMINE 14, 25, 113
EXAMINE INDEXTEST 67, 80
EXPORT 13, 62
IMPORT 5253, 62
IMPORT CONNECT 18
LISTCAT 32
LISTCAT ALL 66
LISTCAT NONVSAM 58
REPRO MERGECAT 51, 54
VERIFY 25, 113
IGG.CATLOCK profile 52, 88
INDATASET keyword 31
index component 4, 25
index sequence set 25
index set 25
INDEXTEST keyword 25, 80
INFILE keyword 31
in-storage catalog cache 7
in-storage control blocks 26, 70
in-storage pointers 50
Integrated Catalog Facility 2
Integrated Catalog Forward Recovery Utility 62
record analysis and processing 62
record selection and validation 62
integrity check 24
integrity exposures 7
INTOEMPTY keyword 53
ISC
See in-storage catalog cache
ITSO test environment 8

L
library ID 8
library manager 8
library name 8
LOADxx PARMLIB member 5, 8
lock attribute 15, 52
lockout situations 43
logical description 2

M
management class 16

166

ICF Catalog Backup and Recovery: A Practical Guide

master catalog 3, 5, 18
media type 8
messages
ADR454I 15
CAT12000I 105
IDC0594I 13
IDC11362I 32
IDC11367I 35
IDC11374I 32
IDC11375I 35
IDC11700I 25
IDC21701I 25
IDC3009I 42, 88
IEC143I 22
IEF237I 22
prefixes 22
multi-volume data set 5
MVS allocation routines 22
MVS cross memory services 6
MVS MODIFY CATALOG command 26, 42
ABEND 43
ALLOCATED 38
CLOSE 43
END 43
LIST 43
RESTART 44
UNALLOCATE 43
VCLOSE 43
VUNALLOCATE 43
MVS MODIFY command 7

N
non-VSAM data set 4
non-VSAM volume record 3
NVR
See non-VSAM volume record

O
orphan NVR 30
orphan VVR 30
out-of-sequence records 77

P
path 4
physical attributes 2
physical cartridge 18
physical description 2

point-of-failure 12, 56
post recovery actions 66
post-recovery updates 12
preventive maintenance 143
primary space allocation 53
print and delete BCD/VVDS records 138
problem resolution 24

R
record caching 7
record length 2
record type 4
recording technique 8
records out of sequence 25
recover 12
recovery
considerations 55
master catalog 55
process 50
strategy 50
tape volume catalog 56
techniques 51
user catalog 56
utilities 52
VVDS 57
Redbooks Web site 152
Contact us 15
repair 50
repair actions 36
restore 12, 50, 52, 86
restored copy 51
return codes 22

S
salvage 50
sequence set 25
shared device 38
SHAREOPTIONS keyword 37
sharing across systems 38
sharing problems 40
sharing systems 39
sharing within a system 38
SMF records 17, 50, 58, 62, 90
type 36 51
type 60 60
type 61 59
SMFPRMxx PARMLIB member 58
SMS

See system managed storage

SMS storage group 8
sphere record 4
split 24, 41
STGADMIN.IDC.DIAGNOSE.CATALOG profile 26
STGADMIN.IDC.DIAGNOSE.VVDS profile 26
STGADMIN.IDC.EXAMINE.DATASET profile 25
storage administrator 12
storage class 54
structural error 24, 113
structural integrity 25, 80
synchronization errors 30
synchronizing aliases 121
SYSCAT statement 5, 8
SYSLOG messages 44
sysplex environment 12
system initialization 5
system managed storage 3
system-managed tape 7
system-managed tape library 8

T
tape configuration database 8
Tape Library Dataserver 8
tape library record 78
tape management system 8
tape volume catalog 7
general VOLCAT 8
specific VOLCAT 8
tape volume catalogs 4
tape volume record 78
TCDB
See tape configuration database
terminology 2
truename 4

U
user catalog 3
user catalog connections 97
user catalog connector 4

V
vertical pointers 25
virtual look-aside facility 7
virtual storage 6
Virtual Tape Server
VLF

Index

167

See virtual look-aside facility

VOLCAT
See tape volume catalog
volser 8
volser change 107
volser clip 107
volume
clipping 57
non-shared 38
relabeling 57
volume reserve 81
volume table of contents 2
VSAM component name 4
VSAM Knowledge Database 24
VSAM volume control record 34
VSAM volume data set 2
VSAM volume record
VTOC
See volume table of contents
VTS
See Virtual Tape Server
VVCR
See VSAM volume control record
VVDS
See VSAM volume data set
VVDS backup and forward recovery 125
VVDS rebuild 19
VVDSFIX tool 51, 57
VVR
See VSAM volume record

168

ICF Catalog Backup and Recovery: A Practical Guide

ICF Catalog Backup and Recovery: A Practical Guide

(0.2spine)
0.17<->0.473
90<->249 pages

Back cover

ICF Catalog Backup

and Recovery
A Practical Guide
Learn how to be
prepared when
catalogs break -and they do break!
Know the tools for
detecting catalog
errors
Look at case studies
of common error
situations

ICF catalogs are essential system data sets. Even with todays
high availability storage subsystems and processors, there are
still situations in which you need to recover your catalogs. You
should keep your catalogs healthy and be prepared for a recovery
situation. Also, you need to be sure that your recovery procedures
do not have a big impact on your production environment. To
minimize the recovery process, you should have a clear backup
and recovery strategy, and the proper utilities.
IBM Integrated Catalog Forward Recovery Utility (ICFRU) and
Mainstar Catalog RecoveryPlus are two of the available tools you
can use to recover your catalogs. ICFRU is a basic tool to help you
in a forward recovery situation. It does not offer a wide range of
features, but is useful for catalog recovery. Mainstar's Catalog
RecoveryPlus offers a set of features to help you in your ICF
catalog environment maintenance.
This IBM Redbook provides information and practical examples on
how to use each of these products in a catalog recovery situation.
It also provides useful recommendations for storage
administrators in implementing a catalog backup and recovery
plan. It is not intended to compare the two products but to show
how each of them work. This redbook also provides a variety of
practical tests to help you with the different error scenarios you
may find in your daily production activities.

INTERNATIONAL
TECHNICAL
SUPPORT
ORGANIZATION

BUILDING TECHNICAL
INFORMATION BASED ON
PRACTICAL EXPERIENCE
IBM Redbooks are developed by
the IBM International Technical
Support Organization. Experts
from IBM, Customers and
Partners from around the world
create timely technical
information based on realistic
scenarios. Specific
recommendations are provided
to help you implement IT
solutions more effectively in
your environment.

For more information:

ibm.com/redbooks
SG24-5644-01

ISBN 0738423467