IBML

Continuous Availability S/390 Technology Guide

Gregor Neaga, Pierluigi Buratti, Christian Labauve, Gordon Raffel,
Allan Sarlo, Maryela Weihrauch

International Technical Support Organization

http://www.redbooks.ibm.com
This book was printed at 240 dpi (dots per inch). The final production redbook with the RED cover will
be printed at 1200 dpi and will provide superior graphics resolution. Please see “How to Get ITSO
Redbooks” at the back of this book for ordering instructions.

SG24-2086-00
IBML
SG24-2086-00
International Technical Support Organization

Continuous Availability S/390 Technology Guide

December 1998
 Take Note!

Before using this information and the product it supports, be sure to read the general information in
Appendix A, “Special Notices” on page 291.

First Edition (December 1998)

Comments may be addressed to:

IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
522 South Road
Poughkeepsie, New York 12601-5400

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any
way it believes appropriate without incurring any obligation to you.

 Copyright International Business Machines Corporation 1998. All rights reserved.

Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is
subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.
Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
The Team That Wrote This Redbook . . . . . . . . . . . . . . . . . . . . . . . . . xv
Comments Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi

Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Continuous Availability Systems Design . . . . . . . . . . . . . . . . . . . . 1

Chapter 2. S/390 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Summary of S/390 Processor Continuous Availability Attributes . . . . . . 5
2.2 S/390 CMOS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 CMOS Packaging Advantage . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 S/390 G4 Model Logical Flow . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.3 9672 High Availability Features . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.4 9672 Coupling Facility Features . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.5 9674 Standalone Coupling Facility . . . . . . . . . . . . . . . . . . . . . 19
2.2.6 Coupling Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Other S/390 Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.1 S/390 Multiprise 2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.2 S/390 Microprocessor Complex PCI Card . . . . . . . . . . . . . . . . . 21
2.4 OSA-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.1 What is OSA-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.2 OSA-2 TCP/IP Networking Protocol . . . . . . . . . . . . . . . . . . . . 23
2.4.3 OSA-2 SNA/APPN Networking Protocol . . . . . . . . . . . . . . . . . . 24
2.4.4 OSA/SF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Hardware Management Console (HMC) . . . . . . . . . . . . . . . . . . . . 26
2.5.1 HMC Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 HMC Security Considerations . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6 Functional Differences between IBM 9672-, 9021-, and 9121-Based . . . . 30

Chapter 3. S/390 ESCON Architecture . . . . . . . . . . . . . . . . . . . . . . . . 35

3.1 Summary of Continuous Availability Attributes . . . . . . . . . . . . . . . . 35
3.2 Introduction to ESCON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.1 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.2 Speed and Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3 ESCON Channel Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.1 Normal (or Native) Operation (CNC) . . . . . . . . . . . . . . . . . . . . 38
3.3.2 Channel-to-Channel (CTC) . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.3 Converter Channel (CVC) and Byte Multiplexer Channel (CBY) . . . 38
3.3.4 Changing Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 ESCON Multiple Image Facility (EMIF) . . . . . . . . . . . . . . . . . . . . . 39
3.5 Path Configuration for Availability . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5.1 Configuration Hints: Channel Paths . . . . . . . . . . . . . . . . . . . . 44
3.5.2 Configuration Tips: Channel-to-Channel . . . . . . . . . . . . . . . . . . 45
3.6 ESCON Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.6.1 ESCON Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.6.2 ESCON Converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.3 ESCON Channel Extender . . . . . . . . . . . . . . . . . . . . . . . . . . 51

 Copyright IBM Corp. 1998 iii

 3.6.4 Wavelength Division Multiplexor . . . . . . . . . . . . . . . . . . . . . . 51
3.6.5 ESCON-Capable Control Units . . . . . . . . . . . . . . . . . . . . . . . 51
3.7 Path Configuration for Availability . . . . . . . . . . . . . . . . . . . . . . . . 52
3.7.1 Single ESCON Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.7.2 Multiple ESCON Directors . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.7.3 Chained ESCON Directors . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.7.4 Disaster Recovery Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.7.5 ESCON Director Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.8 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.8.1 ESCON Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.9 Advantages and Benefits Summary . . . . . . . . . . . . . . . . . . . . . . . 58

Chapter 4. OS/390 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1 OS/390 Availability Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1.1 Availability Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.2 OS/390 - Continuous Availability . . . . . . . . . . . . . . . . . . . . . . 64
4.1.3 SmartBatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.1.4 DFSMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.1.5 JES3 Improved Availability in OS/390 R4 . . . . . . . . . . . . . . . . . 74
4.1.6 Recoverable Resource Management Services (RRMS) . . . . . . . . 75
4.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Chapter 5. OS/390 Open Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.1.1 OS/390 UNIX Availability Perspective . . . . . . . . . . . . . . . . . . . 79
5.1.2 OpenEdition Data Availability . . . . . . . . . . . . . . . . . . . . . . . . 80
5.1.3 HFS Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.1.4 ADSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.1.5 File Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.1.6 UNIX Processes and PR/SM . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.1.7 UNIX Processes and OS/390 . . . . . . . . . . . . . . . . . . . . . . . . 86
5.1.8 Verifying UNIX Availability . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.1.9 Continuous Availability and Internet Servers . . . . . . . . . . . . . . . 89
5.1.10 Windows NT Applications through Bristol Technology . . . . . . . . 94

Chapter 6. IBM DASD Subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . 97

6.1 Summary of DASD Continuous Availability Attributes . . . . . . . . . . . . 97
6.2 Major High Availability/Continuous Operations Functions . . . . . . . . . . 98
6.2.1 (RAID) Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.3 Major IBM Disk Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.3.1 IBM S/390 DASD Offering . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.3.2 RAMAC 3 Array Storage . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.3.3 RAMAC Subsystem 9394 . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.3.4 RAMAC Virtual Array 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.3.5 RAMAC Scalable Array . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.3.6 RAMAC Electronic Array . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.3.7 S/390 Multiprise 2000 Server Internal Disk . . . . . . . . . . . . . . . 131
6.4 Non-Disruptive DASD Migration Techniques . . . . . . . . . . . . . . . . . 132
6.5 DASD Matrix Function Table . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Chapter 7. Parallel Sysplex Architecture . . . . . . . . . . . . . . . . . . . . . 137

7.1 Summary of Continuous Availability Attributes . . . . . . . . . . . . . . . 137
7.2 Parallel Sysplex Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.2.1 Operability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3 Typical Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3.1 Coupling Facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

iv Continuous Availability S/390 Technology Guide

 7.3.2 Coupling Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.3.3 Sysplex Timer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.3.4 Sysplex Couple Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.3.5 Data Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.4 Datasharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.5 Dynamic Workload Balancing . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.6 Potential Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.6.1 Internal Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.6.2 Mixed Coupling Facilities . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.6.3 External Facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.7 Management and Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.8 Non-Disruptive Software Changes . . . . . . . . . . . . . . . . . . . . . . . 151
7.9 Test Sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.9.1 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.10 Parallel Sysplex Implementation Service Offering (EPSO) . . . . . . . . 153
7.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Chapter 8. Continuous Operation on a Single Processor . . . . . . . . . . . . 155

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.2 Configuration 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.2.2 Non-Disruptive Software Changes . . . . . . . . . . . . . . . . . . . . 157
8.2.3 Processor Resource Considerations . . . . . . . . . . . . . . . . . . . 158
8.2.4 I/O Sharing Considerations . . . . . . . . . . . . . . . . . . . . . . . . 158
8.2.5 Coupling Facility Performance Considerations . . . . . . . . . . . . . 159
8.3 Configuration 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.3.1 Non-Disruptive Software Changes . . . . . . . . . . . . . . . . . . . . 160
8.3.2 Multiple Sysplex Considerations . . . . . . . . . . . . . . . . . . . . . 161
8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Chapter 9. CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

9.1 Summary of CICS Continuous Availability Attributes . . . . . . . . . . . . 163
9.2 Isolation and Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
9.2.1 CICS Intercommunication . . . . . . . . . . . . . . . . . . . . . . . . . 164
9.2.2 Facilities Exploiting CICS Intercommunication . . . . . . . . . . . . . 166
9.2.3 Design for Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
9.3 Dynamic Transaction Routing and Balancing in a CICSplex . . . . . . . 168
9.3.1 Dynamic Transaction Routing . . . . . . . . . . . . . . . . . . . . . . . 168
9.3.2 Dynamic Transaction Balancing . . . . . . . . . . . . . . . . . . . . . 169
9.3.3 CICSPlex/SM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
9.4 CICS Parallel Sysplex Exploitation . . . . . . . . . . . . . . . . . . . . . . . 171
9.4.1 VSAM Record Level Sharing . . . . . . . . . . . . . . . . . . . . . . . 171
9.4.2 Temporary Storage Data Sharing . . . . . . . . . . . . . . . . . . . . 173
9.4.3 VTAM Generic Resource Support . . . . . . . . . . . . . . . . . . . . 173
9.4.4 VTAM Persistent Session Support . . . . . . . . . . . . . . . . . . . . 174
9.5 CICS Extended Recovery Facility (XRF) . . . . . . . . . . . . . . . . . . . . 175
9.6 Improved CICS Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 175
9.6.1 New Domains introduced in CICS/ESA 4.1 . . . . . . . . . . . . . . . 176
9.6.2 New Domains introduced in CICS TS . . . . . . . . . . . . . . . . . . 176
9.7 CICS Resource Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
9.7.1 Resource Definition Online (RDO) . . . . . . . . . . . . . . . . . . . . 179
9.7.2 Autoinstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
9.8 Other CICS Continuous Availability Features . . . . . . . . . . . . . . . . 179
9.8.1 Transaction Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
9.8.2 ARM Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
9.8.3 DBCTL as Interface to IMS . . . . . . . . . . . . . . . . . . . . . . . . 180

Contents v
 9.8.4 DB2 Attachment Facility . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.8.5 CICS Interface to MQSeries . . . . . . . . . . . . . . . . . . . . . . . . 182
9.8.6 Internet Access to CICS Applications . . . . . . . . . . . . . . . . . . 182
9.8.7 Backup-While-Open (BWO) . . . . . . . . . . . . . . . . . . . . . . . . 183

Chapter 10. IMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

10.1 Summary of IMS Continuous Availability Attributes . . . . . . . . . . . . 185
10.2 IMS Full Function and Fast Path Database . . . . . . . . . . . . . . . . . 185
10.3 IMS Recovery Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
10.3.1 Synchronization Points and Backup Copies . . . . . . . . . . . . . . 186
10.3.2 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
10.3.3 Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
10.3.4 Database Recovery Control . . . . . . . . . . . . . . . . . . . . . . . 188
10.3.5 Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10.4 IMS Parallel Sysplex Exploitation . . . . . . . . . . . . . . . . . . . . . . . 190
10.4.1 Data Sharing in Parallel Sysplex . . . . . . . . . . . . . . . . . . . . 190
10.4.2 Common Queue Server . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10.4.3 Generic Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10.5 Other IMS Continuous Availability Features . . . . . . . . . . . . . . . . 194
10.5.1 Fast Path DEDB Online Change . . . . . . . . . . . . . . . . . . . . . 194
10.5.2 Workload Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10.5.3 IMS Partition DB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
10.5.4 Extended Terminal Option (ETO) . . . . . . . . . . . . . . . . . . . . 197
10.5.5 ARM Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.5.6 Distributed Sync Point using OS/390 RRS . . . . . . . . . . . . . . . 197
10.5.7 Daylight Saving Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.5.8 IMS WWW Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

Chapter 11. DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

11.1 Summary of DB2 Continuous Availability Attributes . . . . . . . . . . . 199
11.2 DB2 Recovery Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
11.2.1 Copy Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
11.2.2 LOAD and REORG with Inline Copy . . . . . . . . . . . . . . . . . . 201
11.2.3 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11.2.4 Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11.2.5 Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
11.3 DB2 Data Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
11.3.1 Recovery in a Data Sharing Environment . . . . . . . . . . . . . . . 204
11.3.2 Why Choose Data Sharing . . . . . . . . . . . . . . . . . . . . . . . . 205
11.3.3 Configuration Consideration for High Availability . . . . . . . . . . 205
11.4 Other DB2 Continuous Availability Features . . . . . . . . . . . . . . . . 206
11.4.1 Type 2 Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.4.2 Row Level Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.4.3 Read-Through Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
11.4.4 Online Reorg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
11.4.5 Partition Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 208
11.4.6 RRS Attachment Facility . . . . . . . . . . . . . . . . . . . . . . . . . 209
11.4.7 ARM Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
11.4.8 Daylight Saving Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
11.4.9 Cancel Thread Command . . . . . . . . . . . . . . . . . . . . . . . . 210

Chapter 12. MQ Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

12.1 Summary of MQSeries Continuous Availability Attributes . . . . . . . . 212
12.2 What is Message Queueing . . . . . . . . . . . . . . . . . . . . . . . . . . 212
12.2.1 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

vi Continuous Availability S/390 Technology Guide

 12.2.2 Queue Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12.2.3 Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12.2.4 Message Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12.2.5 Other Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12.3 Program-to-Program Communication . . . . . . . . . . . . . . . . . . . . 216
12.4 Recovery Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12.4.2 How Changes are Made to Data . . . . . . . . . . . . . . . . . . . . 220
12.4.3 How Consistency is Maintained . . . . . . . . . . . . . . . . . . . . . 221
12.4.4 Recovering Distributed Queueing Errors . . . . . . . . . . . . . . . . 222
12.5 How MQSeries Contributes to Continuous Availability . . . . . . . . . . 222
12.5.1 Connectionless and Time-Independent Communication . . . . . . 222
12.5.2 Distribute Programs/Functions across Networks . . . . . . . . . . . 223
12.5.3 Conversational Communication . . . . . . . . . . . . . . . . . . . . . 224
12.5.4 Parallel Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
12.5.5 Segregation of Message Traffic . . . . . . . . . . . . . . . . . . . . . 225
12.6 Continuous Availability Hints and Tips . . . . . . . . . . . . . . . . . . . 226
12.6.1 Dual Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
12.6.2 Restarting a Queue Manager on Another Processor . . . . . . . . 227
12.6.3 Local and Remote Queue Definitions . . . . . . . . . . . . . . . . . . 227

Chapter 13. Systems Management . . . . . . . . . . . . . . . . . . . . . . . . . 229

13.1 Summary of Continuous Availability Attributes . . . . . . . . . . . . . . 229
13.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
13.3 OS/390 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
13.3.2 Additional OS/390 Products . . . . . . . . . . . . . . . . . . . . . . . 233
13.4 Tivoli Management Environment (TME) 10 . . . . . . . . . . . . . . . . . 234
13.4.1 TME 10 NetView for OS/390 . . . . . . . . . . . . . . . . . . . . . . . 235
13.4.2 System Automation for OS/390 . . . . . . . . . . . . . . . . . . . . . 236
13.4.3 Performance Reporter for MVS (PR/MVS) . . . . . . . . . . . . . . . 238
13.4.4 NetView Performance Monitor (NPM) . . . . . . . . . . . . . . . . . 238
13.4.5 TME 10 Operations Planning and Control (OPC) . . . . . . . . . . . 239
13.4.6 Information/Management (Info/Man) . . . . . . . . . . . . . . . . . . 240
13.4.7 Adstar Distributed Storage Manager (ADSM) . . . . . . . . . . . . . 241
13.4.8 TME 10 Global Enterprise Manager (GEM) . . . . . . . . . . . . . . 242
13.5 Batch Interference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
13.5.1 Smartbatch for OS/390 . . . . . . . . . . . . . . . . . . . . . . . . . . 243
13.6 Parallel Sysplex Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
13.6.1 Sysplex Failure Manager (SFM) . . . . . . . . . . . . . . . . . . . . . 245
13.6.2 Workload Manager (WLM) . . . . . . . . . . . . . . . . . . . . . . . . 245
13.6.3 Coupling Facility Resource Manager (CFRM) . . . . . . . . . . . . . 246
13.6.4 CICSPlex Systems Manager (CP/SM) . . . . . . . . . . . . . . . . . 247
13.7 Availability Measurement And Tools . . . . . . . . . . . . . . . . . . . . 248
13.7.1 Service Level Agreements . . . . . . . . . . . . . . . . . . . . . . . . 249
13.7.2 Availability Measurement . . . . . . . . . . . . . . . . . . . . . . . . 249
13.7.3 Capacity Planning and Performance Management . . . . . . . . . . 250
13.7.4 Resource Management Facility (RMF) . . . . . . . . . . . . . . . . . 250
13.7.5 Performance Reporter for MVS . . . . . . . . . . . . . . . . . . . . . 251
13.7.6 Service Level Reporter V3 . . . . . . . . . . . . . . . . . . . . . . . . 252
13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

Chapter 14. Sharing Data among Heterogeneous Platforms . . . . . . . . . . 255

14.1 Universal Data Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
14.2 Data Copy Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14.2.1 File Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
14.2.2 Serial Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Contents vii
 14.2.3 Data Piping Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
14.3 Storage Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
14.4 Real Data Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
14.5 The Seascape Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

Chapter 15. Applications Design for Continuous Availability . . . . . . . . . 267

15.1 C/A Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
15.2 Design for Fault Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . 268
15.2.1 Object-Oriented Programming . . . . . . . . . . . . . . . . . . . . . . 269
15.2.2 Quality Software Engineering . . . . . . . . . . . . . . . . . . . . . . 270
15.2.3 Test Extensively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
15.3 Design for Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . 272
15.3.1 Redundancy/Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . 272
15.3.2 Failure Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
15.3.3 Recovery/Bypass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
15.4 Design for Failure Resistance . . . . . . . . . . . . . . . . . . . . . . . . . 273
15.4.1 Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
15.4.2 User Environment Preservation . . . . . . . . . . . . . . . . . . . . . 275
15.4.3 Fast Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
15.5 Design for Availability Management . . . . . . . . . . . . . . . . . . . . . 275
15.5.1 Tracking and Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 275
15.6 Design for Non-Disruptive Changes . . . . . . . . . . . . . . . . . . . . . 276
15.7 Design for Non-Disruptive Maintenance . . . . . . . . . . . . . . . . . . . 276
15.7.1 Co-existence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
15.7.2 Online Redefinition of Applications . . . . . . . . . . . . . . . . . . . 277
15.8 Design for Continuous Applications . . . . . . . . . . . . . . . . . . . . . 278
15.8.1 No Batch Interference . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
15.8.2 Online Database Reorganization and Copies . . . . . . . . . . . . . 279

Chapter 16. Continuous Availability and Disaster Recovery . . . . . . . . . . 281

16.1 What is Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
16.1.2 Data Loss and Service Loss . . . . . . . . . . . . . . . . . . . . . . . 285
16.2 Relationship Between Continuous Availability and Disaster Recovery 286
16.3 Geographically Dispersed Parallel Sysplex . . . . . . . . . . . . . . . . . 287
16.4 The Total Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . 288

Appendix A. Special Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Appendix B. Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . 293

B.1 International Technical Support Organization Publications . . . . . . . . 293
B.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
B.3 Other Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293

How to Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295

IBM Redbook Fax Order Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

ITSO Redbook Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

viii Continuous Availability S/390 Technology Guide

Figures

1. A Structured Approach to Continuous Availability Design . . . . . . . . . 3

2. 9672 Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3. S/390 G4 Model Logical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4. Using an Internal Coupling Facility . . . . . . . . . . . . . . . . . . . . . . 17
5. Dynamic Coupling Facility Dispatching . . . . . . . . . . . . . . . . . . . . 18
6. S/390 Open Systems Adaptor 2: Open Platform . . . . . . . . . . . . . . . 23
7. S/390 OSA-2: TCP/IP Networking Protocol . . . . . . . . . . . . . . . . . . 24
8. S/390 OSA-2: SNA/APPN Networking Protocol . . . . . . . . . . . . . . . . 25
9. HMC Configuration for Local Site: Protocol Token Ring . . . . . . . . . . 28
10. HMC Configuration for Remote Site: Protocol Token Ring . . . . . . . . . 29
11. Parallel and ESCON Connections . . . . . . . . . . . . . . . . . . . . . . . 37
12. EMIF Channel Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
13. CTC Connections and EMIF . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
14. 9672 Processor with I/O Cage Arrangement . . . . . . . . . . . . . . . . . 42
15. I/O Cage Arrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
16. Multi-System Connection through an ESCON Director . . . . . . . . . . . 47
17. CTC over ESCON Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
18. Separated Computing Facilities . . . . . . . . . . . . . . . . . . . . . . . . 50
19. Port Card Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
20. Multi-Director Port Configuration . . . . . . . . . . . . . . . . . . . . . . . . 54
21. Chained Director Configurations . . . . . . . . . . . . . . . . . . . . . . . . 55
22. Operating System Co-Existence within a Sysplex, N, N+1, N+2 . . . . 68
23. Rolling System Maintenance into 7 X 24 Availability . . . . . . . . . . . . 69
24. UNIX 95 Hierarchical File System . . . . . . . . . . . . . . . . . . . . . . . 81
25. Reasons for Planned Application Data Outages . . . . . . . . . . . . . . 102
26. IBM S/390 Disk Offering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
27. IBM′s S/390 DASD Offerings . . . . . . . . . . . . . . . . . . . . . . . . . 105
28. RAMAC 3 Array Storage Building Blocks . . . . . . . . . . . . . . . . . . 107
29. Peer-to-Peer Remote Copy . . . . . . . . . . . . . . . . . . . . . . . . . . 110
30. PPRC Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
31. P/DAS and MVS Control Blocks before the Swap . . . . . . . . . . . . . 112
32. P/DAS and MVS Control Blocks after the Swap . . . . . . . . . . . . . . 113
33. Extended Remote Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
34. XRC Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
35. RAMAC 3 Drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
36. Concurrent LIC Download and Activation . . . . . . . . . . . . . . . . . . 119
37. RAMAC Virtual Array 2 Turbo . . . . . . . . . . . . . . . . . . . . . . . . 122
38. Virtual Disk Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
39. SnapShot Copy: Impact on Batch . . . . . . . . . . . . . . . . . . . . . . 128
40. RSA-2: Control Unit Design . . . . . . . . . . . . . . . . . . . . . . . . . . 129
41. Parallel Sysplex Components . . . . . . . . . . . . . . . . . . . . . . . . . 140
42. Internal Coupling Facility Connections . . . . . . . . . . . . . . . . . . . 143
43. Separated Data Centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
44. Reversed Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
45. Mixed Coupling Facility Configuration . . . . . . . . . . . . . . . . . . . . 149
46. External Coupling Facilities . . . . . . . . . . . . . . . . . . . . . . . . . . 150
47. Test and Production Sysplex Configuration . . . . . . . . . . . . . . . . 152
48. Single System Sysplex 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
49. Single System Sysplex 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
50. CICS Intercommunication Features . . . . . . . . . . . . . . . . . . . . . 165
51. CICS Intercommunication Facilities . . . . . . . . . . . . . . . . . . . . . 166

 Copyright IBM Corp. 1998 ix

 52. CICSPlex/SM Interaction in a CICSplex . . . . . . . . . . . . . . . . . . . 171
53. Conceptual View of Parallel Sysplex with RLS . . . . . . . . . . . . . . . 172
54. Shared Message Queue Environment . . . . . . . . . . . . . . . . . . . . 173
55. VTAM Generic Resource Support in CICS . . . . . . . . . . . . . . . . . 174
56. CICS Recovery Domain-CICS XRF Configuration . . . . . . . . . . . . . 175
57. CICS Recovery Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
58. CICS Logging Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 178
59. The CICS Transaction Server and the Internet . . . . . . . . . . . . . . . 182
60. Catastrophic Disaster Scenario 1 . . . . . . . . . . . . . . . . . . . . . . 188
61. Catastrophic Disaster Scenario 2 . . . . . . . . . . . . . . . . . . . . . . 189
62. IMS Data Sharing Configuration . . . . . . . . . . . . . . . . . . . . . . . 191
63. IMS Fast Path DEDB Configuration . . . . . . . . . . . . . . . . . . . . . 192
64. IMS Fast Database Recovery Environment . . . . . . . . . . . . . . . . . 192
65. IMS Shared Message Queue Configuration . . . . . . . . . . . . . . . . 193
66. IMS Fast Path Online Change . . . . . . . . . . . . . . . . . . . . . . . . . 195
67. IMS Workload Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
68. IMS Partition DB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
69. IMS Internet Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 198
70. DB2 Recovery Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
71. DB2 Data Sharing Group Structure . . . . . . . . . . . . . . . . . . . . . 204
72. Data Sharing Improves Availability during Outage . . . . . . . . . . . . 205
73. Application Scenario using Read-Through Lock Isolation . . . . . . . . 207
74. Online Reorganization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
75. Partition Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
76. OS/390 RRS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
77. Traditional Applications Structure . . . . . . . . . . . . . . . . . . . . . . 211
78. Message Queueing Applications Structure . . . . . . . . . . . . . . . . . 211
79. Applications Communication using Message Queueing . . . . . . . . . 217
80. Flexible Client/Server Structure . . . . . . . . . . . . . . . . . . . . . . . 218
81. Units of Work and Syncpoints . . . . . . . . . . . . . . . . . . . . . . . . . 220
82. Distributed Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
83. MQSeries and Parallel Processing . . . . . . . . . . . . . . . . . . . . . 225
84. Segregation of Message Traffic . . . . . . . . . . . . . . . . . . . . . . . 226
85. Continuous Availability Requirements . . . . . . . . . . . . . . . . . . . . 230
86. Data Copy Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
87. File Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
88. Faster File Transfer with CNT FileSpeed . . . . . . . . . . . . . . . . . . 258
89. Serial Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
90. Serial Transfer with IBM CLIO/S . . . . . . . . . . . . . . . . . . . . . . . 260
91. Data Piping Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
92. Data Piping with CNT FileSpeed . . . . . . . . . . . . . . . . . . . . . . . 261
93. Storage Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
94. Data Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
95. Homogeneous Platform Data Sharing . . . . . . . . . . . . . . . . . . . . 264
96. Heterogeneous Platform Data Sharing . . . . . . . . . . . . . . . . . . . 265
97. Continuous Availability Perspective in Application Development . . . . 268
98. Tier 0 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
99. Tier 1 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
100. Tier 2 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
101. Tier 3 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
102. Tier 4 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
103. Tier 5 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
104. Tier 6 Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
105. Data Loss and Service Loss . . . . . . . . . . . . . . . . . . . . . . . . . 286
106. Geographically Dispersed Parallel Sysplex . . . . . . . . . . . . . . . . 288

x Continuous Availability S/390 Technology Guide

107. Financial Impact of a Disaster and Cost of Recovery . . . . . . . . . . . 289
108. Total Recovery Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Figures xi
xii Continuous Availability S/390 Technology Guide
Tables

1. S/390 Processor Features Continuous Availability Matrix . . . . . . . . . 5

2. Functional Differences between IBM 9672-, 9021-, and 9121-Based CPCs 30
3. ESCON C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . 35
4. OS/390 C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . 59
5. IBM DASD Continuous Availability Matrix . . . . . . . . . . . . . . . . . . 97
6. IBM DASD Matrix Function Table . . . . . . . . . . . . . . . . . . . . . . 135
7. C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . . . . . 137
8. CICS C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . . 163
9. IMS C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . . 185
10. DB2 C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . . 199
11. MQSeries C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . 212
12. C/A Attribute Summary Matrix . . . . . . . . . . . . . . . . . . . . . . . . 229
13. S/390 Tivoli Framework Products . . . . . . . . . . . . . . . . . . . . . . 234
14. SmartBatch Functions for Various OS/390 Environments . . . . . . . . 244

 Copyright IBM Corp. 1998 xiii

xiv Continuous Availability S/390 Technology Guide
Preface

This publication, Continuous Availability S/390 Technology Guide, is the second

of two recent redbooks about the topic of designing continuously available
information systems. The purpose of these books to provide information and
guidance for the development, implementation, and ongoing operation of a
continuous availability solution.

The first volume, Continuous Availability Systems Design Guide , discusses the
basic concepts of continuous availability and describes a structured approach to
developing and implementing a continuous availability solution. It is strongly
recommended that Continuous Availability Systems Design Guide , be read as a
prerequisite to the material presented here.

The objective of this book is to list all System/390 related products and product
features that can contribute to a continuous availability solution, and describe
their functional capabilities in relation to continuous availability. However, this
book does not just deal with technology and solution components. It also
discusses key theories, philosophies, and technical challenges that are of
importance to continuous availability systems design.

This publication is aimed at individuals who are involved in the design and
implementation of continuous availability solutions, for the IBM S/390 platform. It
will assist them in:
• Designing an effective continuous availability solution to meet predetermined
business requirements and associated DP requirements.
• Selecting suitable products to match the design.
• Implementing and executing the continuous availability solution.

The information in this publication reflects the present functions and capabilities
of IBM products. However, where possible, future trends and directions have
been taken into account.

The Team That Wrote This Redbook

This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization Poughkeepsie
Center.

Gregor Neaga is a Systems Management, Continuous Availability, and Disaster

Recovery Specialist at the International Technical Support Organization,
Poughkeepsie Center. He has been with IBM for 29 years in various positions
and foreign assignments. He has worked as a systems programmer, systems
engineer, storage management expert, and, for the last eight years, as a
systems design consultant. Before joining the ITSO in 1996, he worked as a
senior consultant at IBM UBG, the Enterprise Consulting subsidiary of IBM
Germany.

Pierluigi Buratti is a Advisory I/T Specialist in Italy. He has 12 years of

experience in Large Systems and MVS-based products, where he has worked as
a system programmer, system engineer, storage management expert. For the
last four years, he has been engaged as a BRS specialist in many customer

 Copyright IBM Corp. 1998 xv

 projects. His areas of expertise include design and implementation of disaster
recovery and high availability solutions in the S/390 platform. He has written
extensively and teaches IBM classes on storage management, high availability
and disaster recovery solutions design and implementation.

Christian Labauve is an Availability Manager in France. He has eight years of

experience in the Large Systems field. His areas of expertise include storage
management, data migration, the RAMAC Array Family, and providing
professional services to MVS customers.

Gordon Raffel is a Availability Specialist working in the UK. He has 21 years of

experience in the Large Systems field, and has worked at IBM for 23 years. He
is presently a member of the UK Product Services Support Technical Council.
His areas of expertise include planning for availability using S/390 hardware and
software products.

Allan Sarlo is an Availability Manager in the USA. He has 26 years of

experience in Large Systems and Process Management. He holds an MBA
degree from the University of Florida. His areas of expertise include software
technical support and business process re-engineering.

Maryela Weihrauch is a systems consultant in Germany. She has 9 years of

experience in DB/DC field. She holds a degree in Computer Science.

Comments Welcome
Your comments are important to us!

We want our redbooks to be as helpful as possible. Please send us your

comments about this or other redbooks in one of the following ways:
• Fax the evaluation form found in “ITSO Redbook Evaluation” on page 299 to
the fax number shown on the form.
• Use the online evaluation form found at http://www.redbooks.ibm.com/
• Send your comments in an Internet note to [email protected]

xvi Continuous Availability S/390 Technology Guide

Chapter 1. Introduction

This publication, Continuous Availability S/390 Technology Guide - , together with

the prerequisite redbook Continuous Availability Systems Design Guide ,
discusses the topic of designing continuously available information systems.
The objective of this book is to be a compendium of System/390 related products
and product features that can contribute to a continuous availability solution, and
describe their functional capabilities in relation to continuous availability.

However, a continuously available system is not simply built from hardware and
software stock components. More than that, it is the result of an individual
systems design process that researches your business needs, and creates a
solution individually tailored for your individual organization.

1.1 Continuous Availability Systems Design

The companion book Continuous Availability Systems Design Guide , discusses
the basic concepts of continuous availability and describes a structured
approach to developing and implementing a continuous availability solution. In
this section, we will briefly summarize the six steps in this structured approach.

Designing and implementing a suitable continuous availability solution is not a

simple task. It can involve considerable effort and expense. A solution involves
the following activities:
• It must be designed and built to match the business requirements.
• It may require additional computing and infrastructure resources.
• It will certainly require the development and testing of many new
procedures, and these new procedures and processes must be compatible
with existing operations. Staff from many different departments will be
involved, and they must work together when developing and implementing
the solution.
• It will involve a trade-off between cost, recovery speed, and the scope of
outages covered.

As with any design project, a structured approach helps to ensure that all of
these factors are considered and suitably addressed.

Figure 1 on page 3 shows the main activities required in the planning and
implementing of a continuous availability capability. These activities are:
1. Determining what the business requires
During this initial part of the process, you need to determine which of the two
aspects of continuous availability, namely, high availability and continuous
operation, are required to which degree for which business processes.
Many business processes are so dependent upon information technology (IT)
resources that they come to a complete standstill in the event of an outage.
There might be regulatory reasons, too, that require a business process to
be available at all times.
By performing a business impact analysis , the negative impact of such
outages, that is, loss of business and revenue, must be evaluated.

 Copyright IBM Corp. 1998 1

 Another result will be the potential business benefit from an extension of
service hours. This analysis will tell you what your business process
availability requirements are in terms of service hours, as well as what
outages can be tolerated at what times, and what recovery time scale is
required in case of an unplanned outage. The results can be used in the
next step to determine and define the data processing (DP) requirements.
2. Defining the data processing requirements
Once the business requirements have been established, you must convert
them into data processing terms, or into a context that the system designer
can use. The result are service level objectives . This could be a matrix
showing, for each system, subsystem, or application, the required service
hours, the time required for system or data maintenance, and the maximum
allowable unplanned outage time.
While the availability requirements per application might be easy to define, it
is not so trivial to determine what might be preventing your organization
from achieving them. This requires a series of analysis projects to be
performed.
An availability management analysis should be performed to measure current
or recent systems availability, and to look at the reasons for past outages.
This analysis should be done by looking at historic evidence in terms of
problem reporting data.
A component failure impact analysis is required to determine potential
causes of unplanned outages and their impact. It is done by analyzing the
existing configuration and its potential single points of failure.
In addition, an analysis of service downtime caused by system and data
maintenance needs to be performed. Often, there is a conflict between the
service hours required by the business and the downtime required for
system and data maintenance purposes. If such a conflict cannot be
resolved, a redesign of one or several aspects of the system, such as
infrastructure, configuration, application, systems management, systems
integration, application isolation, or component redundancy is required.
3. Designing the continuous availability solution
After the DP requirements have been established and agreed upon, you
must design a technical solution that helps to achieve these requirements.
In this context, design means to define the overall characteristics and major
elements of the solution. These elements describe, in a generic way, any
special hardware, software, and systems management functions required for
the desired level of availability. It includes things like redundant or
fault-tolerant hardware components, automatic active and passive systems
monitoring, automatic operation, systems integration, application isolation,
concurrent systems and data maintenance.
Often, a high level design is sufficient in order to estimate the cost of the
solution. If the cost is too high, the requirements will need to be lowered,
and the solution reworked until both the cost and the solution are agreed
upon. Once the solution decision is definite, a more detailed design is
required.
4. Selecting products to match the design
Having built the design for the availability solution, you are ready to select
the products required to implement that solution. Some consideration of
products was probably done during the initial design phase, but now you

2 Continuous Availability S/390 Technology Guide

 must select products that can be used together to support the recovery
design solution you developed. This selection also influences the cost of the
solution.
5. Implementing the continuous availability solution
You are now ready to implement the availability solution according to the
design that was developed. To do this, you install any required hardware
and software components, improve or introduce appropriate systems
management procedures, and develop appropriate documentation.
6. Keeping the solution up-to-date
After implementing the continuous availability solution, you must also put
procedures in place to ensure that the solution remains viable regardless of
changes of the computing environment. You do this through systems
management procedures that that help define, verify, or implement
availability measures at each system change or introduction of new
applications.

Figure 1 illustrates this structured approach to continuous availability design.

Figure 1. A Structured Approach to Continuous Availability Design

Chapter 1. Introduction 3
 1.1.1.1 Iterative Design
Often, the activities we just described do not follow each other strictly in
sequence.

In some instances, implementation work may start even while portions of the
solution design are still being developed.

In other instances, the activities associated with a certain step are achieved in
an earlier step. For example, an organization may decide to use fault-tolerant
disk subsystems even before the full business requirements are established.

More importantly, you will often reach a point in the design process where some
aspect of the solution causes you to rework an earlier stage. In this case, the
process flow may have to loop back and modify the requirements or the design.

A very common requirement is to estimate the costs of various options.

Typically, a high-level pass is taken through the requirements, design, and
product selection steps to estimate the cost of various options. This allows
certain options to be ignored and the more plausible options to be reworked.

Understandably, you may not agree with the proposed design sequence. For
instance, some people may say that in many cases, you have to look at available
products (step 4) before you can come up with the overall design (step 3). We
suggest leaving these arguments to the experts (for instance, availability design
consultants) who do the design in each individual case. This redbook is less an
exercise in system design theory than a comprehensive discussion of planning
topics pertaining to continuous availability.

4 Continuous Availability S/390 Technology Guide

Chapter 2. S/390 Processors

This chapter describes the availability attributes of IBM′s current S/390

processor families, including various coupling facilities, OSA-2, and the HMC.

2.1 Summary of S/390 Processor Continuous Availability Attributes

Table 1 shows which availability attribute applies to the frequency, duration, and
scope of an outage and which outage type it belongs to. In this table we
distinguish between planned (continuous operation) outages and unplanned
(high availability) outages.

Table 1 (Page 1 of 2). S/390 Processor Features Continuous Availability Matrix. Legend: X = A p p l i e s ,
Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Processor Design X HA
Fault-Tolerant Design X HA
Concurrent CP Sparing X HA
Dynamic SAP Sparing/Reassignment X HA
Dynamic Coupling Facility Dispatching X X X CO
Error Correction Code X HA
Dynamic Memory Sparing X HA
Sybsystem Storage Protection X X HA
Subspace Group Facility X X HA
Up to 15 LPAR: Software Isolation X X HA
LPAR Dynamic Storage Reconfiguration X CO
LPAR Time Management Reporting CA
ESCON Multiple Image Facility X CO
Automatic Reconfiguration Management X CO
Scrubbing X HA
Partial Memory Restart X HA
Hot-Plugging Channels X CO
Dynamic I/O Reconfiguration Management X CO
Partial I/O Restart X X CO
Concurrent Maintenance X CO
Independent Dual Power Feed X HA
N + 1 Power Supply Technology X HA
Internal Battery Feature X HA
Concurrent Licensed Internal Code Path X CO
Console Integration X CO
Serviceability X HA
Internal Coupling Facility X X X CA

 Copyright IBM Corp. 1998 5

 Table 1 (Page 2 of 2). S/390 Processor Features Continuous Availability Matrix. Legend: X = A p p l i e s ,
Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Internal Coupling Migration Facility X CO

2.2 S/390 CMOS Server

The current S/390 servers employ the latest CMOS technology, which provides
very dense, high performance logic and array chips.

2.2.1 CMOS Packaging Advantage

CMOS technology has evolved very quickly over the past five years in high-end
processing systems. The CMOS advantage over bipolar comes from the large
number of circuits that can be placed in one chip. Its distinguishing
characteristics are:
• Low power per circuit which allows a lot of circuits per chip
• Lower total system heat dissipation
• Significantly less heat removal hardware
• Extremely good reliability

The CMOS chip, at 2,000,000 circuits, contains more than enough circuits to build
a S/390 processing unit. The bipolar 711-based technology, in comparison, takes
somewhere in the neighborhood of 400 chips to build a S/390 PU.

From the point of view &ca, this new packaging lessens the complexity of
installing, monitoring and controlling your data processing environment.
Because of the smaller overall package and fewer parts, maintenance
complexity and cost are also greatly reduced. In addition, the inherent low
power required by CMOS technology implies a significant reduction in
environmental costs.

Figure 2 on page 7 illustrates how a CMOS processor is packaged.

6 Continuous Availability S/390 Technology Guide

Figure 2. 9672 Packaging

There are four types of CMOS chips that are packaged onto the multi-chip
module (MCM). Each of these chips has different functions.

There is the Processing Unit (PU) chip, L2 chip (L), the Bus Switching Network
(BSN) chip and the Memory Bus Adapter (MBA) chip. The following sections
explain these chips in more detail. The PU chip contains the Level 1 cache (64
Kbytes), the control store (32 Kbytes) and the execution unit. Part of the
execution unit is the floating point coprocessor.

The PU chips are mounted on the 127x127 mm multi-chip module (MCM). Using
the 10-way RX5 and RY5 models as an example, there are twelve of these chips
mounted on the MCM; ten of the PUs would be S/390 central processors (CPs),
and two PUs would be system assist processors (SAPs).

The SAPs are S/390 PUs with special Licensed Internal Code (LIC) to allow these
processors to perform dedicated I/O function (similar to the IOP in previous
water-cooled processors).

2.2.1.1 L2(L) Cache Chip

The L2 cache chip provides 384 Kbytes of storage. Two of the 384 Kbyte chips
are combined to make a 768 Kbyte cache which is shared by a cluster of three
PUs. In a fully configured RX5 or RY5 system, there are four of these clusters,
with a total of eight cache chips on the module.

2.2.1.2 Bus Switching Network (BSN) Chip

The BSN chip contains a high speed matrix bus switch and a portion of a new
shared cache. This chip provides the high speed interconnect between
processor storage and the PUs and the MBAs. There are a total of eight BSN
chips in all G4 modules which, in total, contain 2Mb of shared cache (256Kb per
BSN chip).

Chapter 2. S/390 Processors 7

 2.2.1.3 Memory Bus Adapter (MBA) Chip
There are two MBA chips in all G4 processors. The MBA chip provides a path
between the processor storage (via the BSN chips) to the S/390 channels, OSA-2
adapters, and coupling facility links. These chips are similarly mounted on the
MCM.

For G4 systems, the multi-chip module (MCM) is the critical package for the
processor. This substrate is mounted on the processor board along with power,
memory, and some I/O capability. A total 10-way model (RX5 or RY5) can be
contained within one or two frames, depending on the I/O configuration. This
includes all power and cooling facilities.

2.2.2 S/390 G4 Model Logical Flow

Figure 3. S/390 G4 Model Logical Flow

S/390 G4 servers have up to 12 processing units (PUs). Note the following

points:
• At most, ten of the PUs can be used for normal ESA/390 instruction
execution.
• At least one PU must act as a system assist processor (SAP). By default,
there is one SAP on models up through the 7-way, and two SAPs on the
8-way through 10-way.
• More SAPs can be configured for I/O-intensive workloads, such as
Transaction Processing Facility (TPF).
• You can also configure up to two of the PUs as dedicated internal coupling
facility processors except on the 10-way model, where there are no spare
processors (10 CPs + 2 SAPs).

The S/390 G4 server processors have a three-level high speed buffer (HSB)
structure:
• 64KB L1 cache on the PU Chip
• 768KB L2 cache per three processors
• 2MB shared cache

8 Continuous Availability S/390 Technology Guide

 The L1 cache is a store-through buffer, and the L2 cache is a store-in buffer. As
can be observed on Figure 3, the L2 cache is shared by these PUs. The cache
associated with the BSN is shared by all PUs to provide faster response to
memory requests at this level. This shared cache is a store-through cache to
central storage.

There are four high-speed memory buses that connect processors and I/O
devices to processor storage (PS), so that up to four processing units can
transfer data to/from storage concurrently.

Processor storage (PS) can be assigned as a combination of central storage (CS)

and expanded storage (ES) at power-on reset (POR) time.

Storage access is interleaved between the storage cards, and this tends to
equalize storage activity across the cards to minimize storage contention.

The PUs′ high-performance memory muses, BSNs, storage controllers (STC),

and MBAs work in a peer relationship, providing function similar to the system
control element (SCE) used in a 9021 711-based system.

I/O devices pass data to central storage through the memory bus adapter (MBA).
The physical path from the channel includes the planar board, the channel
adapter card (CHA-IBB not shown) the internal bus buffer (IBB), the self-timed
interface (STI) bus, the bus switching network (BSN), and the storage controller
(STC).

2.2.3 9672 High Availability Features

The S/390 G4 family of processors employs state-of-the-art design and
technology as standard features to provide high levels of reliability, availability,
and serviceability.

2.2.3.1 Processor Design

All G4 servers are provided with an enhanced processor design. Each central
processor contains dual instruction/execution units, which operate
simultaneously. Results are compared, and in the event of a miscompare,
instruction retry is invoked. This design simplifies checking and virtually
eliminates CP failures due to soft errors.

2.2.3.2 Fault-Tolerant Design

Fault-tolerant design allows hardware recovery to be performed in most cases
totally transparent to customer operation and eliminates the need for a repair
action, or defers a repair action to a convenient time scheduled by the customer.

2.2.3.3 Enhanced Recovery for Cache and Directory Errors

In the event of cache and directory errors, the enhanced recovery design of the
G4 processor provides line and cache segment deletes to remove the affected
areas from the configuration. Processing will continue and the service action, if
required, will be scheduled for a convenient time.

Chapter 2. S/390 Processors 9

 2.2.3.4 Concurrent CP/SAP Sparing
In the event of a CP failure on a G4 System, and when a spare PU is available,
in most cases the system will initialize a spare PU to take over as a CP. If the
failure occurred in a shared LPAR environment, the sparing is done dynamically
and is transparent to the customer. For multiprocessor systems, in Basic or
dedicated LPAR mode, the system will send a message to the operator to
configure the new CP online, if the software supports re-configuration of CPs. In
these environments, sparing takes place concurrently with system operation.

As a further enhancement on G4 multiprocessor models, in most cases the

application that was running on the failed CP will be preserved (this known as
application preservation) and will continue processing on another CP with no
customer intervention required.

For Basic mode and for dedicated logical partitions in LPAR mode, the following
software is required to support application preservation:
• OS/390 Release 1 or later; MVS/ESA SP Version 5.1 or 5.2
• MVS/ESA SP Version 4.2 or 4.3; MVS ESA SP Version 3.1.3
• VM/ESA Release 2.1 or 2.2

VSE/ESA does not support this function. There are no software corequisites
when running shared CPs in LPAR mode.

Application preservation capability eliminates the need for customer intervention

in the recovery process and preserves the customer′s application processing.
Note that for uni-processor models and uni-processor dedicated partitions, a
failed CP means that the application and operating system are going to go down.
Refer to S/390 PR/SM Planning Guide for additional information.

S/390 G3 models have concurrent CP sparing also, but do not support

application preservation. In the rare event of a CP failure on a G3 system, and
when a spare PU is available, the system will initialize a spare PU to take over
as a CP. In LPAR and Basic modes, the new CP must be configured online (by
software that supports reconfiguration) and the application restarted. For
uni-processor models and uni-processor dedicated partitions, a failed CP means
that the application and operating system are going to go down.

All G4 servers, with the exception of the maximum configured models (for
example, the 10+2 model), provide at least one spare PU. The rules for the
allocation of CPs and SAPs are as follows:
• Standard CPs are allocated starting at the lowest PU number and continue in
ascending order.
• SAPs are allocated starting at the highest available PU number and continue
in descending order.
• ICFs follow standard CPs and continue in ascending order.
Note: Model R45 with the Cryptographic Coprocessor function has
non-sequential CP addresses (in order to use both coprocessors); model R45 has
CP 0/1/2/4 (CP 0 and CP4 have the coprocessor attached).

10 Continuous Availability S/390 Technology Guide

2.2.3.5 Dynamic SAP Sparing/Reassignment
Dynamic recovery is provided for failure of the System Assist Processor (SAP).
In the event of a SAP failure, if a spare processor unit (PU) is available, in most
cases the spare PU will be dynamically activated as a new SAP. If there is no
spare PU available, and the CPC has more than one central processor (CP), an
active CP will be reassigned as a SAP. In either case, no customer intervention
is required. This capability eliminates an unplanned outage and permits a
service action, if necessary, to be deferred to a more convenient time.

2.2.3.6 Processor Availability Facility (PAF)

This facility enhances application availability on n-way CPC models. If an CP
fails, this facility allows the application that was running on that CP to continue
execution on another CP. (On systems without this facility, that would often
result in application termination.) The Processor Availability Facility is supported
natively by OS/390 and VM/ESA, and by PR/SM LPAR in shared partitions
regardless of the operating system.

2.2.3.7 Dynamic Coupling Facility Dispatching

The Dynamic Coupling Facility Dispatching function helps enable continuous
computing in the event of a coupling facility failure without requiring a
standalone backup coupling facility. Enhanced dispatching algorithms enable
you to define a backup coupling facility in a logical partition (LPAR) on your
system. This backup facility shares CP resources with the logical partitions
executing active workloads, and uses only minimal CP resources until the
primary CF fails. When the backup CF is active, only the resource necessary to
provide coupling is allocated.

2.2.3.8 Error Correction Code (ECC)

Memory error checking and correction code is enhanced to detect and correct
not only single bit errors, but also up to four bit errors in a single memory chip.
This enhancement further reduces the possibility of an outage due to a memory
chip failure. During normal operations, the system monitors and records
accumulation of failing bits in memory chips that are corrected by error
correction code (ECC). Before a failure threshold is reached that could result in
an uncorrectable error, during a planned power-on reset, the system invokes a
spare memory chip in place of the one with accumulated failing bits. This action
may prevent an unscheduled outage or a need for replacement of the memory
card.

2.2.3.9 LPAR Dynamic Storage Reconfiguration (DSR 2)

Enhanced dynamic storage reconfiguration of both central and expanded storage
is available on all G4 servers. Previously introduced on G3 servers and MP
2000, this capability removes the restriction of storage reconfigurations only
being possible from an adjacent and above logical partition. More flexible
storage reconfigurations can now occur without an outage to either the donor or
the receiver logical partition with a cooperating guest.

For more information on DSR, refer to LPAR Dynamic Storage Reconfiguration .

Chapter 2. S/390 Processors 11

 2.2.3.10 Dynamic Memory Sparing
The processor storage cards on the Parallel Enterprise servers are equipped
with spare memory chips. During normal operations, the system monitors and
records the accumulation of failing bits in memory chips that are corrected by
error correction code (ECC). Before a failure threshold is reached which could
result in an uncorrectable error, the system invokes a spare memory chip in
place of the one with the accumulated failing bits. This action may prevent an
unscheduled outage for replacement of the memory card.

2.2.3.11 Subsystem Storage Protection

Subsystem Storage Protection (SSSP) provides a mechanism to protect
subsystem code. It prevents applications which are running in the subsystem ′ s
address space from overwriting the subsystem code, control blocks, and the
subsystem data. This function is implemented using a special storage key
(key_9). For example, applications will run under key_9 and cannot overwrite the
subsystem code which runs using key_8.

Currently, CICS V3 (and above) is the only user of SSSP. It is supported by the
9021 711-based, 9021 520-based, 9121, 9221, 9672 processors, and the S/390
G3/G4 server. It protects your applications from some CICS outages due to
overwriting CICS code.

2.2.3.12 Subspace Group Facility

The Subspace Group Facility is an hardware feature used to implement a new
CICS/ESA function called Transaction Isolation, to build upon CICS Subsystem
Storage Protection (SSSP) functions. It provides storage protection at the
transaction application level so that one transaction cannot inadvertently
damage the code or data belonging to other transactions (storage overlay) and
thereby reduces system outages. The function is exploited by CICS/ESA V4.1
and MVS/ESA 5.1 (also as VM/ESA guests). Other subsystems may make use of
this new function in the future.

2.2.3.13 Up to 15 LPARs
PR/SM supports 15 LPARs in a SI-mode (only 10 LPARs on 9672 R1, R2, R3
models). Fifteen LPARs are supported on S/390 G3/G4 processors only.

2.2.3.14 LPAR Time Management Reporting

LPAR utilization reporting enhances the information available in RMF′s partition
data report. The enhanced report provides LPAR management time and helps to
report any low utilization effect, or to measure availability.

2.2.3.15 ESCON Multiple Image Facility (EMIF)

EMIF offers you improved configuration flexibility, a possible reduction in
channels required, and possible reduced cost for the addition of logical partitions
or I/O devices. EMIF may allow you to explore new applications and increase
the processing capacity of your system without requiring channel upgrades to
support the workload.

12 Continuous Availability S/390 Technology Guide

2.2.3.16 Automatic Reconfiguration Facility (ARF)
Based on an installation-defined policy, a partition can internally invoke the
Storage Reconfiguration Facility of PR/SM to add storage to a standby partition.
The standby partition can then acquire the workload of another failed partition.
A higher level of availability is achieved by reducing the time to reconfigure
storage.

2.2.3.17 Scrubbing
Storage background scrubbing provides continuous monitoring of storage for the
correction of detected faults before the storage is used.

2.2.3.18 Partial Memory Restart

This enhancement minimizes the length of an outage caused by a memory card
failure. In the event of a memory card failure, the system can be restarted with
half of the original memory. Processing can be resumed until a replacement
memory card is installed (this installation is disruptive).

2.2.3.19 Hot-Plugging of Channels, Coupling Links, and OSA-2

Adapters
Hot-plugging of channels (ESCON and parallel), coupling links and OSA-2
adapters on the S/390 Parallel Enterprise Server models allow configuration
changes non-disruptively during the system operation. CHA and FBI cards
installation is disruptive.

2.2.3.20 Dynamic I/O Reconfiguration Management (DRM)

DRM (with MVS/ESA V4.2 or higher) allows an installation to modify its hardware
and software I/O configuration definitions dynamically for devices, control units,
and channel paths without requiring a power-on reset (POR) of the hardware or
initial program load (IPL) of the operating system. Dynamic I/O reconfiguration
enables the HCD function provided in MVS/ESA V4.1.

2.2.3.21 I/O Interface Reset

This is a reset that quickly and automatically removes outstanding RESERVEs for
DASD volumes in a shared DASD environment. This reset is initiated by 1) the
CPC hardware when it enters a checkstop state, and 2) by MVS, OS/390, or VM
SCPs as their last action before loading a disabled wait state PSW.

Timing is important because, until a reset is issued that resets any outstanding
allegiance and releases outstanding RESERVEs to DASD (for example, such a
reset is issued implicitly when an operator IPLs a system), other systems
sharing that DASD could be locked out. This facility is also known as
system-initiated reset.

2.2.3.22 Partial I/O Restart

In the event of a failure of one of the two memory bus adapters, the system may
be restarted with half the I/O connections available. With the system configured
for maximum availability, access to critical I/O should be maintained. To obtain
details on how configure I/O for availability, refer to 3.5, “Path Configuration for
Availability” on page 41. This capability enables the system to run in a
degraded mode until the part is replaced, restoring full capacity.

Chapter 2. S/390 Processors 13

 2.2.3.23 Independent Dual Power Feed Capability
The power system offers dual primary (AC) power feeds. Each feed is
electrically isolated and enables redundant power paths to each server.
Customers may elect to provide a dual electrical service to the server, further
minimizing any outage due to a single path power interruption.

2.2.3.24 N + 1 Power Supply Technology

The AC and DC power subsystems are designed with N+1 redundancy. Failure
of a power thermal component does not cause a system outage. Concurrent
replacement of the failed component results in an avoidance of a planned
outage.

2.2.3.25 Internal Battery Feature (IBF)

The Internal Battery Feature (IBF) provides the function of a local uninterruptible
power source (UPS) within the processor frame. This feature further enhances
the robustness of IBM CMOS power design, increasing power line disturbance
immunity.

The IBF enables between 3.5 to 20 minutes (3 to 15 minutes for the RY5) of full
power capability for the S/390 G3/G4 server models and the coupling facility
model C04/C05.

For the C04/C05 models, there is an additional function called power save mode.
Power save mode suspends all coupling facility operations, but preserves
memory content for up to one hour. With the installation of an IBF, no additional
floor space is required on the raised floor or in facilities areas.

The IBF is fully redundant and complements the N+1 front-end power
technology. The IBF design is consistent with IBM′s CMOS server power
technology and has no energy conversion losses experienced between a
conventional independent UPS support of a server installation. The IBF has
self-testing capability and is integrated into the diagnostics, including remote
service facility (RSF) support. The IBF is an optional feature. It can be used with
a UPS to provide additional protection.

2.2.3.26 Local Uninterruptible Power Supply

The optional local uninterruptible power supply, machine type 9910, can be
installed as a supplemental or as an alternative to a central UPS to secure
system availability.

2.2.3.27 Concurrent Hardware Maintenance

Concurrent hardware maintenance allows the replacement of a failing unit with
no requirement to power off the system. This function enhances processor
availability, since no system outage is required for the repair action. Concurrent
maintenance capability exists on the following:
• ESCON channels
• Parallel channels
• Coupling links
• S/390 Open Systems adapter 2
• Power elements
• Thermal elements
• Support element
• Hardware management console

14 Continuous Availability S/390 Technology Guide

2.2.3.28 Concurrent Licensed Internal Code (LIC) Patch
Concurrent Licensed Internal Code (LIC) allows the activation of a patch
concurrent with the system operation, thereby further increasing the availability
of the processor and reducing the planned outage necessary for LIC
maintenance. This capability exists on the following:
• PR/SM LPAR code
• CP code
• SAP code
• Integrated Coupling Migration Facility (ICMF)
• Coupling facility control code (CFCC)
• ESCON channels
• Parallel channels
• Coupling links
• Power/thermal control code
• Support element code
• Hardware management console code
• S/390 Open Systems adapter 2
Note: Not all patches are non-disruptive. Some patches still require a
power-on reset for activation.

2.2.3.29 Console Integration

Console Integration allows MVS/ESA to use the hardware system console for
both hardware and software functions. The hardware system console can be
used for normal operator communications as a full function console. This may
improve system availability.

Multiple MVS/ESA systems (running under PR/SM) can share the same console,
which eliminates the need for multiple backup control units and consoles.

Multiple MVS/ESA systems in a sysplex on different processors can be operated

from one console (except during IML/IPL). The hardware master console has
been designed for ease of use.

2.2.3.30 Reliability
The standard features that provide a high level of reliability include:
• High-reliability technology components
• Parts integration to reduce the number of parts in the machine

2.2.3.31 Serviceability
The standard features that provide a high level of serviceability include:
• Automatic error detection and fault isolation concurrent with system
operation
• Automatic remote support capability
• Multiple channel swap: an enhancement for channel problem determination,
allowing up to four channels to be swapped concurrently with system
operation.
• High degree of concurrent maintenance capability in hardware and code
• A status panel showing the status of N+1 power system
• Enhanced diagnostics for coupling links

Chapter 2. S/390 Processors 15

2.2.4 9672 Coupling Facility Features
A coupling facility is a set of functions and services provided by Licensed
Internal Code running in a partition that is connected to Parallel Sysplex
members via coupling links. A coupling facility is an integral part of the S/390
Parallel Sysplex architecture. With the appropriate level of software, it allows
data management subsystems to communicate so that they can directly share
data.

The coupling facility is one of the most critical components of a Parallel Sysplex
that has production data-sharing implemented. The coupling facility holds the
shared data and locks required for the systems in the Parallel Sysplex to share
the workload. If the coupling facility fails, and its data can no longer be shared,
the throughput of the Parallel Sysplex may be severely impacted.

For this reason, the configuration of the coupling facility is an important

consideration for a high availability sysplex, and it is strongly recommended you
have two coupling facility functions in a production data-sharing sysplex.

You must avoid the use of an active coupling facility function in a LPAR of a
Central Electronics Complex (CEC) where other LPARs contain active MVS
members. A failure of this CEC will cause the loss of two major components,
which means it will be impossible to rebuild structures in another coupling
facility.

On the other hand, it is possible to run with an LPAR coupling facility operating
only as a backup, and carrying no active workload. It is also possible to use
dynamic coupling facility dispatching as a backup to a standalone coupling
facility.

If you only have a unique CEC running two or more OS/390 LPARs and want to
take advantage of the continuous operation software provided by a production
Parallel Sysplex, what should you do?

ICF and ICMF, working together, allows you this opportunity at a low cost.

The following section describes how it is possible to set a coupling facility

function in a 9672 server. For more information about Parallel Sysplex, refer to
Chapter 7, “Parallel Sysplex Architecture” on page 137.

2.2.4.1 Internal Coupling Facility

As shown in Figure 4 on page 17, the internal coupling facility (ICF) is an
optional feature of the S/390 G3 and G4 servers that provides the capability of
trading off up to two of the spare PUs as a coupling facility, without increasing
the software licensing cost, which enables a lower point of entry into Parallel
Sysplex and a reduced cost for a backup coupling facility.

16 Continuous Availability S/390 Technology Guide

Figure 4. Using an Internal Coupling Facility. No single point of failure

One or two ICF features can be ordered, utilizing spare PUs as ICMF or as a
coupling facility. This internal coupling facility is defined as a LPAR with spare
PUs dedicated, and it uses either external coupling links or can run as an ICMF
which emulate the coupling links across LPARs. ICF is significantly less
expensive than a separate, entry-level coupling facility.

As we have just seen, the recommendation for a Parallel Sysplex environment is

to use two (or more) Coupling Facilities for no single point of failure. This
necessitates having more than one external standalone CF (9674), for primary
and for backup purposes.

Another way, with the optional priced ICF feature, is to move the backup
coupling facility inside the S/390 G3 or G4 Server when a standalone coupling
facility is used as a primary, reducing the need for a second standalone coupling
facility in a multi-system Parallel Sysplex configuration. All of the workload must
be carried by the standalone coupling facility during normal operation.

With ICF, you can also get into the Parallel Sysplex environment to take
advantage of its Continuous Operation protection from software outages. You
can use ICF as a production coupling facility even in a single-CEC Parallel
Sysplex configuration running two or more OS/390 logical partitions sharing
non-database data (VTAM, XCF, JES2, RACF and GRS). Individual OS/390
partitions can be taken down for maintenance or release upgrade without
suffering an application outage, through the data-sharing provided by the
remaining LPAR in the CEC. By using the ICF as an ICMF, both the benefits of
reduced software license charges and reduced hardware cost are afforded, while
providing the software availability features of the Parallel Sysplex.

2.2.4.2 Dynamic Coupling Facility Dispatching

Chapter 2. S/390 Processors 17

 Figure 5. Dynamic Coupling Facility Dispatching

As shown in Figure 5, dynamic coupling facility dispatching is, like the internal
coupling facility, another option to implement a Parallel Sysplex environment
with only one external standalone CF (9674), and to have a backup coupling
facility inside a G3/G4 Server. This function is standard on all G3/G4 Servers
and it does not have processors allocated exclusively for its use.

This feature provides a standby capability for the backup CF. It runs in an LPAR
with shared CPs on a system which also runs production LPARs. The new
coupling facility control code (CFCC) eliminates the need to dedicate an entire
processor for CF use because it is no longer running in active wait mode. CFCC
now uses minimal processor resources in normal operations. If the primary CF
fails, the systems will automatically allocate and re-build structures in the
backup CF LPAR, increasing its processor resources as required (storage is not
dynamically adjusted). When functions are returned to primary CF, the
processor capacity allocated to backup CF will be reallocated to production.

It requires physical coupling links to all system images, including the ones on
the same CEC where the coupling facility LPAR is running (it does not emulate
coupling links between LPARs like ICMF does).

This function is available on the 10-way models, even though the ICF is not
available.

2.2.4.3 Integrated Coupling Migration Facility

To implement your continuous availability solution, you will have to build and
test lots of new operational procedures.

Integrated Coupling Migration Facility (ICMF) provides a cost effective alternative

for the testing and migration of workloads to a Parallel Sysplex. ICMF, which
includes the Coupling Facility functions and simulation of Coupling Links,
provides an environment for migration activities. Using this facility the
installation, development and testing of software needed for a Parallel Sysplex
using the coupling facility can be completed prior to the installation of the
Coupling Links and a Coupling Facility. One or two coupling facility logical
partitions can be defined to connect to other integrated logical partitions (or
LPAR′s) running MVS/ESA on the same Central Electronics Complex (CEC) to
assist in the test and development of data sharing applications designed for
future coupled production work.

18 Continuous Availability S/390 Technology Guide

 ICMF runs in a PR/SM LPAR environment, with PR/SM providing the necessary
emulation of physical Coupling Links to allow interconnected communications
between MVS/ESA systems in logical partitions and a new logical partition type
in which the LPAR Coupling Facility is running. When using ICMF, the logical
partitions used for the MVS/ESA systems and the Coupling Facility are defined
as a group assigned to the ICMF. Since ICMF simulates the Coupling Links and
no physical Coupling Links are used, all logical partitions assigned to the ICMF
group must be on the same S/390 CEC. ICMF provides a low-cost PR/SM-based
test and migration vehicle to be used for software development, installation, and
test before moving applications into production in a Parallel Sysplex.

You can install ICMF prior to installing Coupling Facility channel hardware and
continue to use it after adding Coupling Facility channel hardware to the CEC.
Integrated Coupling Facility logical partitions and integrated logical partitions
running MVS can coexist on the same CEC with:
• Coupling Facility logical partitions that use Coupling Facility channel
hardware
• Logical partitions running MVS that use Coupling Facility channel hardware
• Logical partitions running non-coupled workloads

Combining with ICF, you can also get into a production Parallel Sysplex
environment to take advantage of its Continuous Operation protection from
software outages. Refer to 2.2.4, “9672 Coupling Facility Features” on page 16
for more information about ICF.

2.2.5 9674 Standalone Coupling Facility

The C05 model is configured from a 1-way to a 6-way, with up to 16GB of
storage. High performance coupling links (HiPerLinks) provide improve
response times in processing coupling facility requests. All availability features
are the same than S/390 G4 CMOS Enterprise Server, except for the IBF.

The IBF enables between 3.5 to 20 minutes of full power capability for Coupling
Facility Model C04/C05. There is also an additional function called Power Save
Mode. Power Save Mode suspends all Coupling Facility operations but
preserves memory content for up to one hour.

2.2.6 Coupling Links

Coupling Links are high bandwidth fiber optic links that provide the connectivity
between a coupling facility and the Central Electronic Complex (CEC) attached to
it.

There are two types of coupling links:

• Coupling facility sender link (CFS), which are defined to OS/390 partition.
CFS links could be shared between PR/SM partitions in the same SYSPLEX
connected to a coupling facility.
• Coupling facility receiver link (CFR), which are defined to coupling facility
partition. CFR cannot be shared between coupling facility LPARs.

From a continuous availability point of view, and without taking into account
performance considerations, a minimum of two links between each system and
the coupling facility is strongly recommended. This potentially removes a single

Chapter 2. S/390 Processors 19

 point of failure for the processor′s data sharing capability in the Parallel Sysplex
environment.

For more information about how to estimate the number of coupling facility links
you need, see OS/390 MVS Parallel Sysplex Configuration Cookbook .

2.3 Other S/390 Implementations

Besides IBM′s powerful S/390 CMOS servers, there are other IBM
implementations of the S/390 architecture. In this section, the S/390 Multiprise
2000 and the S/390 Microprocessor Complex PCI Card are described.

2.3.1 S/390 Multiprise 2000

The S/390 Multiprise 2000 is available as a packaged solution offering. It
continues to maintain the same high level of data integrity and security as other
S/390 CMOS processors. Numerous characteristics of their design assist in
providing a high degree of availability and minimizing user downtime.

Many features of the S/390 Multiprise 2000 are the same as those of the CMOS
G4 server, and here we just list high availability function names. To obtain more
details about a particular function, refer to 2.2.3, “9672 High Availability
Features” on page 9.

The following high availability functions are available on the latest S/390
Multiprise 2000:
• Partial memory restart
• Error correction code
• Dynamic memory sparing
• Dynamic SAP reassignment on models having more than one CP
• Independent dual power feed
• N + 1 power supply technology
• Concurrent repair for:
− ESCON channels
− Parallel channels
− HDDs, disk drawers and disk (SCSI) adapters
− S/390 Open Systems adapter 2
− Power element
− Thermal elements
− Service element
− Hardware management console
• Concurrent Licensed Internal Code for some patches. Not all patches are
non-disruptive.
• Concurrent channel upgrade
• Dynamic Reconfiguration Manager
• Subsystem Storage Protection Facility

20 Continuous Availability S/390 Technology Guide

 2.3.1.1 Internal Disk for the S/390 Multiprise 2000
This technology gives S/390 Multiprise internal storage capacity of up to 288GB
providing more storage in less space because of the integration of the server
with its storage controller and disks.

The internal disk drawer advances the use of industry standard disks, or JBODs
(Just a Bunch of Disks), in a RAID 1 (mirrored) configuration. With internal disk,
you can attach your disks by a SCSI 2 Fast/Wide adaptor card (eliminating
separate external control units previously required for S/390 storage).

PR/SM LPAR sharing for internal disk enables the sharing of logical volumes of
disk storage across multiple partitions. The sharing of internal disk between
Multiprise 2000 systems or other S/390 systems is not possible.

2.3.1.2 Mapping of Logical Volumes on the Physical Hard Disk

Drive
The large capacity of the Ultrastar 2XP devices allows multiple logical volumes
to be mapped onto a physical HDD. The number of logical volumes on an HDD
varies from two to eight depending on the format (device type) and logical
volume size (model) chosen. You can emulate any of the following devices:
• 3380 model J, E, or K
• 3390 model 1, 2, 3, or 9

2.3.2 S/390 Microprocessor Complex PCI Card

The S/390 Microprocessor Complex is a PCI card containing the S/390 processor,
S/390 integrated memory, and S/390 Licensed Internal Code (LIC) that runs the
full S/390 instruction set. It can be combined either with a PC Server 330 RAID
or with a RS/6000 7025 Model F50 server.
• The integrated product composed of a PC server, 330 RAID, and the S/390
Microprocessor Complex is called PC Server System/390 .
• The integrated product composed of a RS/6000 7025-F50 server and the S/390
Microprocessor Complex is called RS/6000 with S/390 Server-on-Board .

Both machines are ideal platforms for identifying and developing necessary
changes without impacting the host system. Therefore, the availability of the
host production system is increased, dependencies on the host are eliminated,
and risks to the production system are reduced. This way, these systems could
be part of your continuous availability solution.

Subject to availability of open slots, a maximum two S/390 parallel channel

adapter cards are supported for channel-attached S/370 and S/390 devices. The
following attachments are supported:
• Line and page printers such as the IBM 3800 Model 1 and the IBM 4248
• Tape drives such as the IBM 3420, 3480, and 3490
• Communication controllers such as the IBM 3720 and 3745
• Display controllers such as the 3174

Those two servers offer a wide variety of possibilities and creative uses for
business needs.

For application development, these types of combined servers are a

cost-effective platform used to develop client/server applications in the S/390
environment. This workstation is ideal for small application development teams,

Chapter 2. S/390 Processors 21

 because it exploits the productivity of either S/390 and OS/2 or S/390 and UNIX
development tools and provides immediate access to the real S/390 architecture
for testing and debugging applications. Developers can use UNIX-based or
OS/2-based system design and re-engineering tools along with S/390 3GL and
4GL tools and object-oriented technology to shorten the development cycle for
S/390 environments.

Because OS/390 and MVS/ESA 5.2.2 provide the XPGA base that allows easy
porting of UNIX applications, you can consider one of these machines for
developing and porting your applications to the OS/390 environment.

The time-of-day (TOD) on the IBM PC Server System/390 is easily changed. The
result is a dedicated platform for date-reliant testing of system applications.

With the rapid approach of the year 2000, it is important that you begin to assess
and plan a course of action to address potential challenges during this transition.
Both machines support the year 2000. When used in accordance with their
associated documentation, both machines correctly process, supply, and accept
date information within and between the twentieth and twenty-first centuries.

2.4 OSA-2
OSA-2 incorporates the functions of a channel adapter, a control unit and
network interfaces into an integrated S/390 feature that provides industry
standard connectivity to Ethernet, Token Ring, and FDDI LANs and ATM
networks, simplifying enterprise networks, and reducing the number of physical
boxes that must be managed.

2.4.1 What is OSA-2

OSA-2 is directly attached to the S/390 channel subsystem, offering potentially
improved throughput and cost-effective connectivity for end user access to
TCP/IP and SNA/APPN applications on the S/390 host server. You can continue
to bring to the network the strengths of S/390 and choose the media type;
Ethernet/Token Ring, FDDI, ATM with multimode fiber connector, or ATM with
single mode fiber connector.

22 Continuous Availability S/390 Technology Guide

 Figure 6. S/390 Open Systems Adaptor 2: Open Platform. Industry Standard Seamless
Connectivity

When the S/390 server is running in logical partition (LPAR) mode, OSA-2
resources can be shared among the defined LPARs (OSA/SF required). OSA-2
resources consist of the TCP/IP and SNA/APPN applications (TCP/IP mode and
SNA mode) and the ports on the OSA-2 features. Both modes can share access
to an OSA-2 feature and can also access the same port on an OSA-2 feature.
Both modes can also be running concurrently.

The OSA-2 ENTR (Ethernet/Token Ring) feature has two independent ports which
can be configured as either Ethernet or Token Ring. Thus, you can have two
Ethernet ports, two Token Ring ports, or one Ethernet and one Token Ring port.
In addition, the OSA-2 ENTR feature supports full duplex/switched environments,
offering configuration flexibility with minimal disruption to the infrastructure.

A FDDI feature supports attachment to a 100 Mbps FDDI LAN. One dual-ring or
single-ring attachment is supported, as well as attachment to an optical bypass
switch.

The ATM OSA-2 feature delivers integrated connectivity to ATM networks,

delivering a cost-effective migration path from limited, shared LAN environments
to switched, dedicated LAN environments.

For information on the specified operating environment and the hardware and
software requirements, refer to Planning for the S/390 Open Systems Adapter
Feature .

2.4.2 OSA-2 TCP/IP Networking Protocol

TCP/IP has become the de facto standard open networking protocol. TCP/IP is
supported on OS/390, MVS/ESA, and VM/ESA. OSA-2 Ethernet, Token-Ring,
FDDI, and ATM Forum-compliant LAN Emulation (ATM LANE) supports each of
these environments.

Chapter 2. S/390 Processors 23

 Figure 7. S/390 OSA-2: TCP/IP Networking Protocol. Connectivity to TCP/IP Resources

TCP/IP protocol processing is performed on the host (TCP/IP for MVS and VM).
With the addition of OSA-2 to the IBM family of products that provide network
connectivity to distributed systems, connectivity can now be accomplished in a
more integrated manner.

OSA-2 provides TCP/IP clients connectivity to TCP/IP resources on the S/390

server.

TCP/IP is not supported in the VSE/ESA environment.

2.4.2.1 Fault-Tolerant Network Attachment

With virtual IP addressing (VIPA) in TCP/IP for MVS Version 3 Release 2, you can
establish fault-tolerant, redundant, backup paths, and applications become
insensitive to the condition of the network. You can identify virtual IP addresses
and advertise the availability of that virtual host.

Primary and backup paths can be configured using multiple OSA-2s or a

combination of OSA-2s, gate ways, or communication controllers.

This assumes you are also using RouteD which provides automatic and
transparent recovery from controller, interface, and network failures.

For more information on implementing OSA2 function, refer to Open Systems

Adapter 2 Implementation Guide .

2.4.3 OSA-2 SNA/APPN Networking Protocol

OSA-2 is an addition to the IBM family of products that provide SNA/APPN
connectivity to Ethernet, Token-Ring, and FDDI LANs and ATM Forum-complaint
LAN emulation and native ATM networks from IBM S/390 servers.

24 Continuous Availability S/390 Technology Guide

Figure 8. S/390 OSA-2: SNA/APPN Networking Protocol. A SNA/APPN LAN Gateway

OSA-2 provides SNA/APPN (ACF/VTAM) clients with connectivity to SNA/APPN

resources on the S/390 server in the OS/390, MVS/ESA, VM/ESA, and VSE/ESA
environments.

S/390 applications can now take advantage of native, high speed, Asynchronous
Transfer Mode (ATM) connections into the ACF/VTAM network without
application change. The Communications Server Release 2, in conjunction with
OSA-2 and OSA/SF provide an ATM Forum user-to-network interface
(UNI)-compliant, native ATM capability for the S/390 server. This support is
available for the APPN environment only.

Native ATM coupled with VTAM′s High Performance Data Transfer (HPDT) and
APPN High Performance Routing (HPR) gives you the ability to utilize advanced
networking functions in the S/390 server. APPN HPR class of service (COS) is
mapped to ATM virtual circuit connection characteristics. Existing applications
require no changes and can fully utilize the capabilities of the ATM network.

HPR provides improved performance, nondisruptive accommodation of network

failures, and congestion control. HPDT services include reduction in data
movement within S/390, buffer management, data packing, and improved
scheduling of I/O channel programs.

2.4.3.1 SNA/APPN Availability for Token-Ring and FDDI

The following support is enabled via OSA/SF and is available for OS/390,
MVS/ESA, VM/ESA, and VSE/ESA environments via an OSA/SF PTF:
• Load balancing
A single medium access control (MAC) address can now be defined for
attached OSA-2 SNA/APPN physical units (PUs), even though connections
may be via multiple physical ports.
The number of sessions established through a port is monitored, and user
session loads are evenly distributed across the equally configured ports.
• Redundant/backup connections
A secondary path between the LAN workstation and the host system can
now be configured. If the primary path becomes unavailable, the secondary
path will receive the LAN traffic. This increases system availability and
simplifies network management.
• Automatic handling of session overflows

Chapter 2. S/390 Processors 25

 User sessions flow through the primary OSA-2 port until the session capacity
has been reached. Additional user sessions will automatically flow to the
next OSA-2 port. Since all user workstations are identically configured,
network administration is simplified and the network becomes more
scalable. New users can be added nondisruptively. This support is offered
for source-route bridged environments only (Token-Ring and FDDI).
For more information on implementing OSA2 function, refer to Open Systems
Adapter 2 Implementation Guide .

2.4.4 OSA/SF
The S/390 Open Systems Adapter Support Facility (OSA/SF) is an application for
customizing and managing OSA-2 features to support different modes of
operation for host-to-LAN services:
• TCP/IP providing communications between TCP/IP clients and host TCP/IP
applications
• SNA/APPN providing communications between SNA/APPN clients and host
SNA/APPN applications

OSA/SF allows you to change port parameters and obtain status about an OSA-2
feature. OSA/SF can also provide direct management to an OSA-2 defined to an
LPAR.

OSA/SF delivers a simple means to configure, manage, and download software

updates and program fixes.

OSA/SF includes software to install on an OS/2 workstation that communicates

with OSA/SF on the host and the OSA-2 features.

Access to OSA/SF is also available via a command line interface through a REXX
exec.

A call from a REXX exec can be made to the application programming interface
(API) of an OSA/SF application. You can, therefore, automate OSA/SF
procedures.

OSA/SF is required when using the 155 ATM OSA-2, when using any OSA-2
mode other than TCP/IP, or when sharing OSA-2 LAN ports among multiple
logical partitions (LPARs).

A single copy of OSA/SF running in one LPAR is all that is required to configure
all OSA-2 features associated with all of the defined LPARs running on the S/390
Server. For continuous availability purposes, it may be advantageous to have a
second copy of OSA/SF in a second LPAR, in the event the first LPAR is
unavailable.

2.5 Hardware Management Console (HMC)

The hardware management console provides a single point of control and single
system image for managing local or remote hardware elements such as 9672,
9674, and Multiprise/2000. The hardware management console employs a
direct-manipulation object-oriented graphical user interface supporting exception
based real-time system status reporting via icons and colors, consolidated
hardware messages, consolidated operating system messages, and consolidated

26 Continuous Availability S/390 Technology Guide

 service support (including modems and phone lines) and, of course, full
operation of the managed systems. The HMC is an IBM PC running an OS/2
platform, Communication Server, Distributed Console Access Facility and the
IBM Hardware Management Console Application (HWMCA).

The support element (SE) provides the supporting hardware interface controls to
the CPC functional element. It is also the interface used by the hardware
management console for the control, monitoring and service functions on the
9672 and 9674 products. This unit is physically mounted within the 9672 or 9674
A-Frame.

The HMC uses a Token-Ring LAN connection to communicate with the SE. When
tasks are performed at the HMC, the commands are sent to one or more SEs
through the LAN. The SEs then issue commands to their associated CPCs.
CPCs can be grouped at the HMC, so that a single command can be passed
along to all the CPCs in the LAN, if desired. This can help you to reduce the
number of HMCs, and to consolidate operations. For example, if you have a
configuration with four 9672s and two 9674s, then there will be six CPCs and SEs
that can be operated across the LAN by one HMC.

Each HMC can be connected to up to 32 SEs, and each SE can be connected to

up to 16 HMCs.

The following section discusses the recommendations for configuring the

hardware management console for High Availability, and how to handle timer
and ESCON director consoles.

2.5.1 HMC Recommendations

To achieve maximum availability, it is strongly recommended that you have at
least two hardware management consoles. Since HMCs operate as peers, this
provides full redundancy. It is also important to remember that any failure in an
SE or HMC will not bring down the operating system or systems running on the
CPC or CPCs.

The hardware management console is used for the following functions:

• Performing such tasks as load, activate, and reset normal
• Displaying exceptions, for example as indicated through the operating
system messages icon and the hardware messages icon
• Observing status, for example by using the system activity display and the
alter display function
While it is possible for CPCs to continue normal operations without a working
hardware management console, problem determination and recovery time may
be impacted without access to the HMC functions. For this reason, access to an
HMC is regarded as a critical element of your CPC′s configuration.

Figure 9 on page 28 shows the recommended HMC configuration for a local site.
Four HMCs are used to ensure that the hardware management console functions
are available when needed.

Chapter 2. S/390 Processors 27

 Figure 9. HMC Configuration for Local Site: Protocol Token Ring

An HMC is required in each machine room for use by service personnel for
installation and maintenance.

In the operations area, two HMCs are recommended for awareness of status
changes and management of the configuration. One is the primary HMC and the
other is used for backup.

Systems support personnel also require access to some HMC functions for
customizing profiles. This access can be provided through DCAF. Note that the
systems support DCAF should access to HMC4, the backup HMC, so that the
operator access to the HMC is not disrupted.

Figure 9 also shows ESCON Director 9032-3 and Sysplex Timer 9037-2 connected
to the HMC LAN. However, note that their console applications are not placed
on the PC that supports the HMC in the machine room. IBM recommends that
you do not place the extra applications on the same PC that supports the HMC
application because when the machine room HMC is unavailable for service
reasons, access to the extra applications′ functions is lost.

Figure 10 on page 29 shows the recommended HMC configuration for a remote

site.

28 Continuous Availability S/390 Technology Guide

 Figure 10. HMC Configuration for Remote Site: Protocol Token Ring

An HMC is required in each machine room for use by service personnel. In the
operations area, two HMCs are recommended. One is the primary HMC and the
other is used for backup.

Several different connected LANs are required to support full connectivity to the
HMCs in this configuration.

To obtain more information on console configuration, refer to IBM S/390 Harware

Console Configuration Design and Implementation .

2.5.2 HMC Security Considerations

To ensure total security, connect all HMCs, CPCs, and attaching control units to
a private LAN. Using a private LAN for your configuration offers several security,
availability, and performance advantages, as follows:
• Direct access to the LAN is limited to those HMCs and CPCs attached to it.
Outsiders cannot connect to it.
• Traffic disruption due to temporary outages on the LAN is reduced, including
minor disruptions caused by plugging in and powering on new devices on
the LAN, and such catastrophic disruptions as LAN adapters being run at the
wrong speed.
• LAN traffic is minimized, reducing the possibility of delays at the HMC user
interface. Additionally, HMC and SE activity, if included on a non-private
LAN, could potentially disrupt other LAN activity.
• If using a private LAN is not practical, isolate the LAN used by the HMCs,
SEs, and control units by providing a LAN bridge between the isolated LAN
and the backbone LAN to provide an intermediate level of security.
• Assign a unique domain name that includes all the CPCs controlled from one
or more HMCs.

Chapter 2. S/390 Processors 29

 • Install one or more HMCs that have defined to them all the CPCs you want to
control. Place at least one of these HMCs in the machine room near the
CPCs that form its domain.
• Create user IDs and passwords for the HMC, or change the passwords of the
default user IDs.
• Control DCAF access to the HMC by setting the appropriate enablement
option on the HMC.
• Change the DCAF passwords on the HMC as appropriate.
• Do not publicize the SNA names or TCP/IP addresses of the HMCs or the
SEs.
• Do not modify the HMC communications configuration to make it a network
node.

Normal PC security issues should also be satisfied, such as ensuring the PC is

in a secure area, and that the keyboards are locked when not in use. It is
clearly possible to cause significant service disruption from the HMC console if
unauthorized access is possible.

2.6 Functional Differences between IBM 9672-, 9021-, and 9121-Based

Keys to Table 2:
S Standard
O Optional
- Not applicable or supported
*M XDF Withdrawn from marketing
*N CFS only
*O For HMC and SE
*P For HMC, channels (ESCON, parallel, coupling facility), service
element, power control, and some OSA-1 and OSA-2 components
*Q Newly built and MES
*R RY5 only
*S Up to 2GB central storage and up to 8GB expanded storage
*T R3 only

Table 2 (Page 1 of 4). Functional Differences between I B M 9672-, 9021-, and 9121-Based CPCs
ES/9000 ES/9672
Multiprise
Function 9021 9121 R2,
R1 G3 G4 2000
711 511 R3
Parallel Sysplex:
ICMF S S S S S S S
Dynamic CF Dispatching - - O O O S -
CF Links O O*N O O O O -
Reconfigurable CFR Channel Paths - - - - S S -
HiPerLinks - - - - O*Q O -
CFCC in LP O - S S S S -
CF Shared CP O - S S S S -
CF Level 0, 1, 2, or 3 O - O O O - -
CF Level 4 O O O O O O ICMF

30 Continuous Availability S/390 Technology Guide

Table 2 (Page 2 of 4). Functional Differences between I B M 9672-, 9021-, and 9121-Based CPCs
ES/9000 ES/9672
Multiprise
Function 9021 9121 R2,
R1 G3 G4 2000
711 511 R3
Internal Coupling Facility (ICF) - - - - O*Q O -
Sysplex Timer Adapter Attachment O O O O O O O
Reliability, Availability, Serviceability
Application Preservation - - - - - S -
CP Dual Instruction/Execution Unit - - - - - S -
Partial Memory Restart - - - - S S S
Concurrent CP/SAP Sparing - - - - S S -
Dynamic SAP Sparing - - - - S S -
Dynamic SAP Reassignment - - - S S S S
Dynamic Memory Sparing S - - - S S S
Spare Memory Chips S - S S S S S
Concurrent Channel Installation - - S S S S S
Concur. Channel, OSA 2, CF Link Maint. S - S S S S S
Subsystem Storage Protection S S S S S S S
Subspace Group Facility S S S S S S S
Processor Availability Feature (PAF) S S - - - S -
Multi-level HSB S - S S S S S
Concurrent LIC Maintenance S - S*O S*P S*P S*P S*P
Concurrent Power Maintenance S - S S S S S
Concurrent HDD SCSI maintenance - - - - - - S
Dual Utility Power Input - - - S S S S
Internal Battery Feature (IBF) - - - - O O O
Local UPS - - O O O O O
Battery Backup (BBU) - - O - - - -
50/60 Hz Power S S S S S S S
N+1 Power Supplies S - S S S S S
Console Integration S S S S S S S
Concurrent CP TCM Maintenance S - - - - - -
I/O Interface Reset S S S S S S S
Dynamic Physical Partitioning Performance S S - - - - -
Duplexed PCE-Single Image S - - - - - -
HMC/SE Concurrent Maintenance - - S S S S S
Concurrent CP Sparing - - - - S S -
Performance:
High Performance Chiller - - - - - S*R -
Perform Locked Operation (PLO) - - - - - S -
TCP/IP Checksum Facility - - - - S S S
Cryptographic Coprocessor Feature O - - - O S O

Chapter 2. S/390 Processors 31

 Table 2 (Page 3 of 4). Functional Differences between I B M 9672-, 9021-, and 9121-Based CPCs
ES/9000 ES/9672
Multiprise
Function 9021 9121 R2,
R1 G3 G4 2000
711 511 R3
VM Data Spaces S S S S S S S
DB2 Sort Assist S S S S S S S
SIE Assist S S S S S S S
Enhanced Move Page S S S S S S S
Hiperbatch S S S S S S S
Data Spaces S S S S S S S
Hiperspaces S S S S S S S
Move Page S S S S S S S
Logical String Assist S S S S S S S
Asynchronous Pageout Facility S S S S S S S
Asynchronous Data Mover Facility (ADMF) S S S S S S S
Data Compression S S S S S S S
Suppression on Protection S S S S S S S
Vector Facility O O - - - - -
Nine Vector Instructions S S - - - - -
Scalar Square Root Instructions S S S S S S S
PR/SM:
Multiple LPAR 2GB Central Storage - - - - S S S
ESCON Multiple Image Facility (EMIF) S S S S S S S
Up to Fifteen Logical Partitions - - - - O S -
Up to Ten Logical Partitions S - S S S - S
Shared Internal Disk - - - - - - S
Expanded Storage > 8GB - - - - - O -
Dynamic Storage Reconfiguration (DSR 1) S S S S S S S
Enhanced DSR Expanded S - - - S S S
Enhanced DSR Central S - - - S S S
Preferred Path S S S S S S S
LPAR Definitions Retained S S S S S S S
LPAR Management Time Reporting S S S S S S S
Automatic Reconfiguration Facility (ARF) S S S S S S S
Vary CP ON/OFF (logical and physical) S S S S S S S
Resource Capping S S S S S S S
LPAR Recovery (PAF) S S - - - S -
I/O:
ESCON Basic Mode CTC S S S S S S S
8 Path Dynamic Reconnect S S S S S S S
ESCON Channels S O S S S S S
17MB/sec S - S S S S S

32 Continuous Availability S/390 Technology Guide

Table 2 (Page 4 of 4). Functional Differences between I B M 9672-, 9021-, and 9121-Based CPCs
ES/9000 ES/9672
Multiprise
Function 9021 9121 R2,
R1 G3 G4 2000
711 511 R3
10MB/sec (-) S (-) (-) (-) (-) (-)
ESCON XDF O*M O*M O*M - - - -
Dynamic Reconfiguration Management S S S S S S S
Parallel Channels O O O O O O O
Cancel Subchannel S S S S S S S
ESCON Byte Channel (CBY) with 9034 O O O O O O O
Open Systems Adapter 1 (OSA 1) O O O O - - -
Open Systems Adapter 2 (OSA 2) - - - O S S S
Processor Storage:
Up to 2GB Processor Storage O - O O O O O
Up to 4GB Processor Storage O*S - - O*T O O O
Up to 8GB Processor Storage O*S - - - O O -
Up to 16GB Processor Storage - - - - - O -

Chapter 2. S/390 Processors 33

34 Continuous Availability S/390 Technology Guide
Chapter 3. S/390 ESCON Architecture

The ESCON (Enterprise Systems Connection) Architecture was introduced in 1990

as a feature on S/390 processors. It is the first major revision of the channel
architecture announced with S/360 in 1964, and offers new range, performance,
and connectivity options over the old channel architecture.

3.1 Summary of Continuous Availability Attributes

Table 3 shows some areas of continuous availability that ESCON is designed to
address.

Table 3. ESCON C/A Attribute Summary Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e

Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
Non-Disruptive Device Changes X X X CO
Dynamic Path Sharing between X X X CO
LPARs
Configuration Checking after Path X CO
Failure
Enhanced Configuration X X X CO
Management

3.2 Introduction to ESCON

The main change is the adoption of fiber optic transport to perform the physical
connection between processors and their associated peripherals. Light is now
used to carry the control and data packets exchanged between the source and
destination of each operation, although the signal is converted back into
electrical form in all ESCON devices for processing. ESCON uses two fibers to
form a send and return pair for each channel path. This is often referred to as a
link .

The traditional S/360 channel cables contained a separate path for each bit, and
therefore transmitted all nine bits of a data byte in parallel. ESCON channels
have only a single transmit fiber, so all data bits are transmitted serially. The
terms parallel and serial are often used to distinguish between these two types
of architecture.

Fiber optic transport has several advantages over the existing metal cable used
in processor-to-I/O communications. Among these are:
• Fiber optic transport is much smaller, lighter and more easily handled.
• It is more secure; there is no electromagnetive field around the cable, and
therefore no pickup of the radiated signal from the cable.
• It is not affected by other electromagnetic sources (electrical noise), so no
special cable routing is required to avoid such sources.

The ESCON architecture implemented in S/390 also enhances the level of error
checking and recoverability for channel connections over the parallel
environment.

 Copyright IBM Corp. 1998 35

 Removal or breakage of a parallel cable results in a stream of error messages
which continue to report the event until an action is taken to resolve this.

However, while removal or breakage of an ESCON cable results in a message

informing of the failure, the console is not swamped with updates.

On replacing the cable, the ESCON architecture ensures that the environment is
not restarted without full checking. The host is forced to check that the
environment it is now reconnected to is the same environment that existed
before the path error event. The ESCON control unit prevents all operation on
this path until this check has been completed successfully by the host. This
offers full protection against accidental wrong connection, and was not possible
under the parallel channel architecture.

Architectural design points like these ensure that the ESCON channel
environment offers the best possible support for continuous availability.

More detail on the general function and specification of ESCON can be found in
the redbook entitled Enterprise Systems Connection (ESCON) Implementation
Guide . This chapter will focus on ESCON′s contribution to Continuous
Availability, although some background is given to assist understanding.

3.2.1 Topology
The parallel channel architecture was capable of supporting the connection of
several control units to each channel, sometimes called daisy-chaining or
multi-dropping.

The ESCON Architecture cannot support multi-dropped control units. It can

connect either to a single control unit interface (which is called a point-to-point
connection), or by using a high capacity, high speed switch (which is called a
switched point-to-point connection).

Figure 11 on page 37 shows the differences between these connection types.

36 Continuous Availability S/390 Technology Guide

 Figure 11. Parallel and ESCON Connections

the ESCON Director (ESCD) product. In a multi-system environment, and in

particular the Parallel Sysplex environment described later in this book, the
ESCON Director becomes important. It allows the increased connectivity needed
for the wider sharing of devices, while giving additional path management
facilities.

3.2.2 Speed and Distance

The ESCON channel is capable of operating at up to 17 MB/second, compared
with 4.5MB/second for a traditional parallel channel.

The maximum distance allowed for a point-to-point connection depends on the

fiber standard used.
For 50 micron fiber, the distance is limited to two km (1.24 miles).
For 62.5 micron fiber, the distance is limited to three km (1.86 miles).
These grades of fiber are known as multi-mode fiber.

Consider the three types of connection shown in Figure 11. For the parallel
channel the maximum distance is around 120 metres (400 feet). For ESCON
point-to-point, the maximum distance is three km. For the switched
point-to-point configuration shown, the ESCON Director can be placed at up to
three km from the processor, and each control unit can then also be placed at
up to three km. This gives a total separation between processor and control unit
of six km.

A maximum of two ESCON Directors can be chained together on a serial link,

which will allow a separation of nine km, if 62.5 micron fiber is used.

The normal light-transmitting component used is the light emitting diode (LED).
Another transmitter using a laser and operating over nine micron fiber is

Chapter 3. S/390 ESCON Architecture 37

 available for the ESCON Director. This grade of fiber is known as single mode
fiber. The laser or Extended Distance Feature (XDF) offers a maximum of 20 km
between the Directors. Now the potential maximum separation of processor and
control unit is 26 km.

This compares very favorably with the traditional parallel channel′s maximum of
400 feet. In summary, the distance and speed attributes alone of ESCON open a
completely new range of possibilities when designing for both continuous
availability and also disaster recovery.

Now let us consider some additional benefits.

3.3 ESCON Channel Types

The ESCON channel can be defined to work in three different modes of
operation. The definition is effective in both the channel subsystem microcode
and the operating system.

3.3.1 Normal (or Native) Operation (CNC)

In normal operation, the ESCON channel is given the definition abbreviation of
CNC. This will only allow it to work with an ESCON-capable control unit, or
through an ESCON Director.

3.3.2 Channel-to-Channel (CTC)

An ESCON channel can also be defined to operate as a channel-to-channel
adapter (CTC). To form a CTC connection between processors, all that is
required is their connection via a single ESCON cable and one of the ESCON
channels to be configured as a CTC. When compared to existing technology, for
example, the IBM 3088, this offers a reduction of components in the path and is
therefore statistically a higher availability solution.

3.3.3 Converter Channel (CVC) and Byte Multiplexer Channel (CBY)

To allow connection of traditional parallel devices, an ESCON converter unit is
available, and the ESCON channel connected to it carries the definition of CVC.
This allows customers an opportunity to migrate to full ESCON over a period of
time, and does not destroy their investment in parallel devices. When
connecting parallel devices to a converter, the devices can be multi-dropped just
like a normal parallel channel.

A second definition for slow speed parallel devices exists for connection through
an ESCON converter. The byte multiplexor (CBY) definition supports devices that
can only work in byte multiplexer or burst mode. This permits the connection of
several slow speed devices to operate at the same time.

3.3.4 Changing Attributes

The type of ESCON channel required is defined to the processor channel
subsystem microcode and to the operating system using the Hardware
Configuration Dialog (HCD) component of OS/390 or MVS. All processors,
operating systems, and peripherals, including parallel channels and devices, are
defined using the same tool.

38 Continuous Availability S/390 Technology Guide

 At first it may seem unnecessary to use an ESCON channel and converter to
attach a traditional parallel control unit. Parallel channels are still available as a
standard feature on S/390 processors, so why consider using ESCON?

The aim is continuous availability, or to be more precise, in this case the aim is
continuous operation - the reduction of scheduled outages. It is possible to
replace or upgrade the subsystem attached to the converter without a
closedown. Any recabling work can be performed without the need for a service
interruption. This work has been possible for several years, but the final step of
introducing the change to the channel subsystem microcode and the operating
system has always required a service closedown.

Now, to change an ESCON channel from operating in converter (CVC) mode to

native ESCON (CNC), HCD has the capability to allow this change to be made
dynamically. This means that the complete change can be carried out without
scheduling a closedown and power-on reset to activate the new definitions.

This is not possible when using native parallel channels.

The new configuration information is stored separately from the active

configuration for protection, until the changeover is made.

HCD has a graphics capability to display the new configuration from several
viewpoints. It creates a block diagram of the newly defined configuration paths
and allows a simple comparison to be made. We strongly recommend that you
use this feature after each change in the channel subsystem definition. It helps
to eliminate potential human error or misunderstandings between departments
before activation of the new definitions.

3.4 ESCON Multiple Image Facility (EMIF)

ESCON Multiple Image Facility (EMIF) allows an ESCON (CNC) channel or an
ESCON CTC channel to be shared between multiple logical partitions (LPAR) on
the same physical processor complex. The CVC type channel is not supported
by EMIF because the parallel devices do not support this dynamic sharing.

From a continuous availability point of view, EMIF allows LPARs on a processor

to back each other up without requiring an excessive amount of channels and
channel interfaces to be installed. Figure 12 on page 40 shows a simple
comparison between parallel channels and serial channels using EMIF.

Chapter 3. S/390 ESCON Architecture 39

 Figure 12. EMIF Channel Sharing

Access to all devices attached to a failing system is essential when recovering

workloads. Often a high availability connectivity design is reduced to save cost.
EMIF delivers the opportunity to design in all the additional paths you may
require within a single physical processor complex, at no extra cost. EMIF is a
standard feature on S/390 processors, operating within the Licensed Internal
Code or microcode of the processor.

Figure 12 shows a connection to a control unit. EMIF also supports the sharing
of an ESCON CTC channel between LPARs. Figure 13 shows the same four
LPAR configuration, and the number of CTC links required to support an
any-to-any connection.

Figure 13. CTC Connections and EMIF

Figure 13 shows that 12 channels are required to form this configuration. For
the mathematically inclined, the formula is n(n-1), where n is the number of

40 Continuous Availability S/390 Technology Guide

 images or systems to be connected. EMIF allows both the CTC channel and the
normal CNC channel to be shared, therefore only two channels are required for
any-to-any communication.

High availability design requires redundancy for each path. Without EMIF, 24
paths would be required for a high availability design. EMIF reduces this to only
four.

3.5 Path Configuration for Availability

Although technology and manufacturing techniques are advancing constantly, at
this point it is not possible to guarantee that any piece of hardware will not fail.
When aiming for continuous availability, this fact must be taken into account. By
a little planning, it is possible to design a channel subsystem layout to:
• Minimize the impact of channel path failure
• Minimize the impact of concurrent maintenance following a failure
• Make future I/O installation easier (less disruptive)
• Increase the availability of essential services

The small increments for channel upgrades and the variation of I/O cards
available tend to make each 9672 configuration unique, but by looking at the
hierarchy within the channel subsystem, it is possible to offer some rules for
configuring for higher availability.

The channels for the 9672 processors are arranged in groups called I/O cages.
Figure 14 on page 42 shows a 9672 with two I/O cages labelled I/O Cage 1 and
I/O Cage 2.

These labels have been chosen for this example only; refer to the documentation
for your machine for the labeling associated with your hardware. The number of
I/O cages and the number and type of cards installed in your machine depends
on which features were ordered.

ESCON channels, parallel channels, Open Systems Adapter (OSA), and coupling
links are all connected through this arrangement.

Chapter 3. S/390 ESCON Architecture 41

 Figure 14. 9672 Processor with I/O Cage Arrangement

The fine detail of the internal connections is not important at this point. You can
see that the channel subsystem has a hierarchy within each I/O cage.

Each cage consists of one to three Fast Internal Bus (FIB) Cards, which are
connected to the processors. This connection is also known as the Self Timed
Interface (STI).

Each FIB card supports one or two channel driver cards (CHA), and each CHA
card supports one to four channel cards.

This grouping of an FIB card with its CHA cards and channel cards is called an
I/O domain.

The number of domains and cages varies according to the features installed on
your machine; the configuration shown here is for guidance only.

42 Continuous Availability S/390 Technology Guide

The maximum number of channels available decreases when parallel, OSA, or
coupling links are installed. The number of slots where channel cards can be
installed is fixed, therefore installing an OSA card decreases the maximum
number of channels by four.

Each parallel channel card supports three channels. An OSA-2 card supports
one OSA channel. A coupling link card supports two coupling channels.
(Coupling links do not connect through the CHA card, but connect directly to the
FIB card.)

The STI connections are presented to the processors through the memory bus
adapter components. There are two MBAs, numbered MBA 0 and MBA 1.
There is an affinity between MBA 0 and the even-numbered STIs, and an affinity
between MBA 1 and the odd STIs.

This relationship is important to fully use the Partial I/O Restart feature of the G4
CMOS processor. Partial I/O Restart allows the system to restart and process
following the loss of an MBA. This means that multiple paths to the same
control units must be balanced between MBAs, or more accurately odd and even
STIs, for best connectivity to critical devices.

Figure 15 on page 44 shows more detail about the internal connections from the
FIB card down to the channel card for an eight card group. The internal
connections between the channel cards and their CHA “mother” cards is not a
simple pattern. Care must be taken to avoid placing multiple paths within the
same failure boundary.

When your machine order is processed, a report called the CHPID Report maps
out the exact layout of your machine. This report describes the relationship
between the FIB, CHA, and channel cards in your machine.
Note: Channel cards 1, 2, and 3 are labelled CC instead of CH. This is to
indicate that this position can be used for either a channel card or a coupling
link.

A strong recommendation is to draw a clear connection map of the FIB/STI -

CHA - channel card - CHPID relationship for your machine before starting to
assign channel paths to control units.

Further information on the internal layout of a 9672 is available in System

Overview, S/390 9672 Generation 4 .

Chapter 3. S/390 ESCON Architecture 43

 Figure 15. I/O Cage Arrangement

3.5.1 Configuration Hints: Channel Paths

1. For subsystems with multiple paths:
a. Distribute paths across I/O cages (evenly, if possible), if multiple I/O
cages exist.
b. Distribute multiple paths within an I/O cage across STI/FIB groups
(remember the even/odd affinity between the STI number and the MBA
component on the processor card).
c. Distribute multiple paths within an STI/FIB group across CHA groups.
d. Distribute multiple paths within a CHA group across different channel
cards.
e. Distribute paths to subsystems of the same type across different channel
cards.
f. Try to leave spare channel paths distributed as evenly as possible
across all channel cards.
Point e. is designed to reduce the scope of concurrent maintenance.
Although the failure boundary is a single channel path on a channel
card, replacement of the card requires all paths on that card to be
stopped.

44 Continuous Availability S/390 Technology Guide

 If you have four DASD subsystems and you place a path to each on the
same channel card, a scheduled replacement of this card will affect all
four DASD subsystems.
This may impact the performance of the services offered to users, and
this impact possibly could have been avoided. Each installation should
take its own view on this point for best support of their workloads.
Point f. is designed to assist with both concurrent maintenance and future
upgrades.
It is easier to schedule a stop of three channel paths than four.
Spreading any spare channel paths across as many boundaries as
possible allows the best distribution of paths for any new subsystems,
and minimizes reconfiguration of existing channels.
Reconfiguration and redefinition can be performed without closing down
services, but channel paths will have to be stopped while cables are moved.
Good planning can minimize this disruption.
Two other factors should also be taken into account.
• Remember the architecture of the control unit. Modern control units are
often broken into replicated functional units for better availability. Paths
to the same unit should be distributed across different I/O cages, STI/FIB
cards, CHA cards, and channel cards.
• If your processor has multiple System Assist Processors (SAP)
configured, then there is an affinity between STI/FIB connections and
SAP number. Any SAP can address any channel, but observing these
affinities will produce an even balance across the SAP I/O processors,
which may help performance.
2. For subsystems with only one path, the only real advice is to try and
distribute them across I/O cages, FIB, CHA, and then channel card, in that
order. The aim is to try to place as few of these paths as possible on the
same failure boundary.

3.5.2 Configuration Tips: Channel-to-Channel

For clarity, the configuration diagrams used in this chapter tend to be simplified.
Remember that having only one path from a processor to a device is not good
design practice. When planning a CTC/CNC connection, design in a second path
for backup and performance.

• CTC/CNC between LPARs of the same processor

When forming a CTC/CNC pair on the same physical processor, a cable must
be installed between the two channel connectors. The connection is not
emulated within the microcode, and requires the physical connection to be
made.
Contrary to subsystem configuration guidance, consider placing the CTC and
CNC ends of the same link on the same channel card. If either channel path
associated with the CTC link fails, the whole link has failed anyway. It now
becomes a little easier to schedule the replacement, as only two other paths
are affected by the concurrent maintenance.
Note: This applies only to LPAR CTCs on a single processor.

Chapter 3. S/390 ESCON Architecture 45

 Backup CTC/CNC links should be distributed across I/O cages, FIB cards,
CHA cards, and finally channel cards. Avoid placing them in the same
failure boundary.
• CTC/CNC links between processors
When designing CTC/CNC pairs between processors, think of each as a
single path subsystem. Distribute each link across I/O cages, FIBs, CHAs,
and finally channel cards, just as before. Avoid placing backup links in the
same failure boundary.

3.6 ESCON Devices

This segment describes function and configuration aspects of the interconnection
and switching devices used in an ESCON infrastructure. This includes ESCON
directors, converters, channel extenders, multiplexors, as well as peripheral
devives.

3.6.1 ESCON Director

3.6.1.1 Model Types

The ESCON Director (ESCD) was introduced briefly earlier in this chapter.
Explaining its role at the heart of a modern mainframe configuration and the part
it plays in moving towards continuous availability requires a little more detail.

The ESCON Director is currently available in two models:

9033 model 4 The 9033 is a small capacity switch with a maximum of 16
connections (ports).
9032 model 3 The 9032 is a larger capacity switch with a maximum of 124
connections (ports).
Ports can be ordered in increments of four for each model, although the
minimum configuration for the 9032 is 28.

Following are ESCD continuous availability attributes:

• Ports can be added or removed non-disruptively.
HCD is used to define each switch configuration and allows each new
configuration to be built and then activated without the need for a power-on reset
and IPL.

• In the case of a hardware failure, any spare port can be re-assigned as the
failing port.
Simply swap the cable from the failing port to the new port and the path can be
recovered by reassigning port numbers. This temporary reassignment of ports
is performed through the ESCON Director console. The port hardware can then
also be maintained concurrently, and when convenient, the cable can be
returned and the temporary reassignment of ports can be removed.

If the affected path is one of a group connecting to the same control unit, then
performance may be affected during re-assignment actions, but no break in
service should be seen by the users.

Optional continuous availability features for the 9032:

• Spare Token Ring card (for console attachment)

46 Continuous Availability S/390 Technology Guide

 • Spare power supply
• Spare control processor
• Spare matrix controller and switch
The use of these features reduces the chance of the ESCON Director causing a
major outage to services.

3.6.1.2 Multi-System Use

The real value of an ESCON Director is its ability to manage the connection of
multiple control units to multiple processors. This can be in either full dynamic
sharing, or in a strictly controlled fashion, according to service requirements.

Each path to a control unit can be totally open for all to access, or it can be
restricted to a subset of channels only. This subset could be only a single
channel on a single processor.

The paths to the control unit are very effectively shared by the dynamic
switching action of the ESCON Director, but they can also be manipulated and
regulated in a variety of ways. Figure 16 shows a typical connection scenario.

Figure 16. Multi-System Connection through an ESCON Director

The figure shows two processors, each with a potential connection to both
control units. The ports used by the connections are labelled A, B, C, and D.
Although each machine is physically connected into the switch, a connection
from a processor to a control unit across the switch must be allowed within the
switch port map (or matrix).

Where a port is allowed to switch is determined by the attributes set on both the
input and the output port. The attributes are set either by HCD definition, or from
the ESCON Director console.

The default is to allow any port to switch to any other (except itself). This means
that the channel from Processor 1 (ESCD Port A) is allowed to connect to either
Control Unit 1 (ESCD Port D) or Control Unit 2 (ESCD Port C). Processor 2 (ESCD
Port B) has the same access unless the defaults are overwritten.

Chapter 3. S/390 ESCON Architecture 47

 Other levels of control are:
• Prohibit specified pairs of ports from connecting.
Prohibit Ports B and C. This prevents Control Unit 2 from being accessed
from Processor 2, but does not affect Processor 1′s access.
• Dedicate pairs of ports together.
(This can be thought of as being like a cable bypassing the switch, because
no other ports can access either of the dedicated ports.)
Dedicate Ports A and D; this prevents these ports from communicating with
any other. Control Unit 1 cannot be accessed from Processor 2, and
Processor 1 cannot access any other port.
• Block a port from communicating with all others.
Block Port A. This stops its access to or from either Control Unit 1 or 2, but
does not affect any other port connection in the matrix.

The attribute set on any port can be changed dynamically when required.

This means that by connecting all processors to all directors, a fully redundant
physical configuration can be prepared, yet this configuration can be controlled
at a very fine level. For example, it is possible to prevent backup processors
from accessing critical devices by simply changing the attribute set on the
ESCON Director ports. Full connectivity can be granted by simply reversing this.
This allows very fast reconfiguration when trying to recover from a failure, or
when scheduling a workload move to allow a processor to be closed down.
Note: Reducing this interruption time while changing configuration improves
service recovery time and overall availability time.

3.6.1.3 Channel-to-Channel over ESCON Director

In the same way that control unit paths can be shared through an &ESCD, CTC
links between processors (and LPARs) can be shared. Figure 17 on page 49
shows an example of an ESCON Director being used to form the CTC path, and
to reduce the number of channels needed.

48 Continuous Availability S/390 Technology Guide

Figure 17. CTC over ESCON Director

Points to note from Figure 17 are:

• The CTC channel on Processor 1 is shared by Processors 2 and 3, and a
possible connection is also shown for Processor 1. This allows all LPARs on
Processor 1 to also use the same CTC.
• The CNC channel on Processor 2 can also be used to access a normal
control unit. (A CTC channel can only behave as a CTC.)
• There is no CTC/CNC pair available for Processor 2 and 3 to communicate
over. This would require a CTC to be defined on another ESCON channel on
either of these systems, and a connection made to the switch.

3.6.1.4 Chained Directors

It is possible to connect ports on two different directors together. This action,
called chaining. , allows the distance between the host and the target device to
be increased as outlined earlier.

The use of standard LED port cards allows up to three km between the ESCON
Directors; this may be suitable for campus environments.

The use of an XDF card allows much larger distance between processor and its
peripherals.

This now allows the establishment of a second processing facility for backup and
disaster recovery, at a reasonably safe distance, while still allowing all control
units to be connected to all processors.

The foundation can now be laid for a good continuous availability design that can
tolerate the loss of a complete site. Figure 18 on page 50 shows a very simple
example of such a design.

Chapter 3. S/390 ESCON Architecture 49

 Figure 18. Separated Computing Facilities

This configuration allows the establishment of two data centers up to 20 km

apart, yet all peripherals at each site can be accessed by each processor (and
all LPARs) at each site. This design offers a very high degree of availability
when constructed, and it is no longer only a theoretical exercise. Several key
installations around the world have taken the decision to adopt this approach
and are successfully processing their normal everyday workloads on a
configuration like this.

The 3990 Peer-to-Peer Remote Copy (PPRC) feature referred to in Chapter 6,

“IBM DASD Subsystems” on page 97 can also be implemented over directors
and chained directors. PPRC allows current copies of critical DASD volumes to
be maintained at each site. This facility is also being actively exploited in
configurations like the one described. For further information about disaster
recovery options, see Chapter 15, “Applications Design for Continuous
Availability” on page 267 and also refer to Disaster Recovery Library - S/390
Technology Guide .

3.6.2 ESCON Converter

3.6.2.1 Model Types

The ESCON Converter:
• Connects a parallel control unit to an ESCON channel. This type of converter
is classified as a 9034.
• Connects an ESCON 3990 model 2 or 3 to a parallel channel. This type of
converter is classified as a 9035.

Both types of converters can be used in conjunction with the 9032 or 9033
ESCON Director, but some limitations are placed on this use. For example, an
ESCD port connected to a 9034 must be dedicated to a channel port in order to
function. No dynamic switching of the channel connection is possible. Also, the
maximum distance at which devices can be situated when using converters may

50 Continuous Availability S/390 Technology Guide

 be reduced. If you are considering using these units in a high availability
design, check the planning information for any devices to be connected through
them for full details on limitations or restrictions.

3.6.3 ESCON Channel Extender

An ESCON channel can be extended by using the 9036 Remote Channel
Extender. This may be an alternative to the ESCD when only a small number of
channels need to extend beyond three km, and the benefit of dynamic switching
or additional path management is not required.

The 9036 is available in four models:

• Model 001 converts between 62.5 micron (or multi-mode) fiber and 9 micron
(or single mode) fiber and repeats the signal.
• Model 002 supports single mode-to-single mode repeating.
• Model 003, which is a special RPQ for extending Sysplex Timer links.
• Model 004 supports the function of models 1 and 2, but allows the
containment of up to 14 links in a single chassis.

Not all models may be available in all countries; check the status accordingly.

Multiple 9036 units can be placed in a single path, but the minimum is two model
001s. The 9036 can also be used in conjunction with a 9032 or 9033. A maximum
of three 9036 ESCON Remote Channel Extenders may be present in any
end-to-end link, in addition to the two ESCON Directors currently available in a
link.

The 9036 can also extend the PPRC distance to 43 km. Refer to the Remote
Copy Administrators Guide and Reference for further information in this area.

3.6.4 Wavelength Division Multiplexor

The Wavelength Division Multiplexor, which carries an IBM machine type of 9729,
reduces the number of physical links needed to connect two remote sites. It
allows the traffic from up to 10 ESCON channels to be concentrated onto a single
fiber, and can carry this traffic up to 50 km. It allows bi-directional data flow on
the same link. A 9729 is required at each end of the link to correctly perform the
concentration and expansion of the individual data streams.

This would appear to be an extreme single point of failure to introduce into the
configuration compared with individual links, but a backup fiber can be installed
and dynamic switching can occur if the first link fails. This enables some degree
of backup in the interests of high availability.

3.6.5 ESCON-Capable Control Units

The list of ESCON-capable control units is too extensive to include here. All new
developments from IBM and the major OEM companies include ESCON
interfaces. Check each device you are considering for its individual
characteristics, because these can vary, as follows:
• Maximum number of ESCON interfaces supported
• Distance limitations
• Maximum number of hosts supported dynamically

Chapter 3. S/390 ESCON Architecture 51

3.7 Path Configuration for Availability
Just as with the 9672 processor, it is worth considering a few points when
designing a configuration using ESCON Directors. The design concept of
any-to-any switching is very simple, but when designing a channel subsystem, a
few points need to be considered in order to achieve the best availability.

3.7.1 Single ESCON Director

For the purposes of this section, let us assume that the failure boundary within
the ESCON Director is the port card. This is chosen not because a failure will
break all ports on the card, but because concurrent replacement of the port card
requires all paths to be stopped. Features can be installed to make the rest of
the machine fault-tolerant to a certain degree. Any machine that does not use
these features will create a much larger continuous availability design issue, as
a much larger number of paths, perhaps even all paths, will be affected by a
failure.

The ports within an ESCON Director are installed on a card with four ports on
each card. These ports can be either LED or XDF, but not mixed on the same
card.

Any port on a director could be attached to either an ESCON channel or to an

ESCON control unit. There is no restriction made that forces any special
arrangement; the design is entirely open.

Let us call a port attached to an ESCON channel an input port, and a port
attached to a control unit an output port. Any director port card could contain:
• Four input ports
• Four output ports
• A mixture of both
But which arrangement is best? The following sections answer this question.

For high availability, paths to the same subsystem are normally distributed
across as many failure boundaries as possible. With the ESCON Director′ s
switching capability however, a single input port may be allowed to connect to
many output ports. Also, a single output port may be allowed to connect to
many input ports. Mapping the extents of a failure boundary is complex, and
designing a configuration to minimize impact for concurrent maintenance is now
also made more complicated.

If a port card is configured to carry four input ports, then replacement of the card
requires all four channels to be stopped while the replacement is carried out.
This may affect many paths to many different control units. These paths may
also be spread across up to four different processors, each with several LPARs
sharing each path. Scheduling the closedown of all paths is not trivial. If a port
card is configured to carry four output ports, then replacement of the card means
a similar exercise.

If the port card is configured to carry both input and output ports from the same
path, then the impact of replacing this card is minimized. Figure 19 on page 53
shows this situation.

52 Continuous Availability S/390 Technology Guide

 Figure 19. Port Card Configuration

This configuration shows two processors each sharing two control units with four
paths. Each input path from each processor is connected to a different port card,
and each connection to each control unit also sits on a different port card. This
is good separation from the view of either the processor or the control unit.

However, each input and its associated outputs are grouped on the same port
card. If any path is broken, its associated inputs or outputs are affected anyway,
so now the number of additional paths required to be closed for the repair action
is reduced.

3.7.2 Multiple ESCON Directors

A single ESCD can be considered a single point of failure, even with high
availability options installed, so if the design is to support continuous availability,
multiple directors should be used. Now multiple paths should be spread across
all directors to minimize the impact of the loss of a director, and the guidelines
from the single director section can still apply. Figure 20 on page 54 shows the
same configuration now spread across two ESCON Directors.

Chapter 3. S/390 ESCON Architecture 53

 Figure 20. Multi-Director Port Configuration

3.7.3 Chained ESCON Directors

When directors are chained together, one of the director links is required to be
dedicated. This means that only one of the directors is capable of performing
the dynamic switch function. Figure 21 on page 55 shows the two situations
created by this. The solid line in one of the directors represents the dedicated
connection.

54 Continuous Availability S/390 Technology Guide

Figure 21. Chained Director Configurations

There is no hard rule that says that the dedicated link must be placed on the first
or second director. The choice is best made based on what connectivity is
required from the channel.

Placing the dedicated link in the first director allows devices attached to the
second director to be shared. This is an effective way of connecting devices in a
second site to a single processor in the first site. No connection to the second
site can access any device at the first site, so good separation of the
environments is maintained.

Placing the dedicated link in the second director allows the link to the second
director to be shared. This is good for sharing links to a second site between
multiple processors at the first site (reduces the number of inter-site fibers), or
for sharing a channel path between both sites.

The channel connection in this case could also be shared with 3990 Peer-to-Peer
Remote Copy (PPRC) between compatible 3990s at each site. In the lower
diagram of Figure 21, if control units 1 and 2 are both 3990 model 6s, then by

Chapter 3. S/390 ESCON Architecture 55

 allowing the port connection between Control Unit 1 and the link to the second
director, a PPRC session can be established. This can now share the same link
as normal processor traffic to Control Unit 2.
Note: These points apply to both LED cards and XDF cards.

3.7.4 Disaster Recovery Site

The most common use for chained ESCON Directors is when configuring for
disaster recovery. The XDF card allows the sites to be separated by up to 20
km, which gives a reasonable chance of survivability of one site from most
natural disasters. When designing this type of configuration, and also aiming for
continuous availability, we strongly recommend that you ensure the fiber links
between the sites run in at least two different routes with good separation
between them. They should not even share a common point of entry to either
building or computer room.

Another point to consider is the number of ESCON Directors to use. For a

disaster recovery configuration, each site will have at least one processor, and if
this is used in normal workload sharing, data will move in both directions
between the sites. A single ESCON Director at each site now becomes an
extremely critical point. A failure of either ESCON Director would probably
cause both processors to stop. In this situation, continuous availability demands
that at least two ESCON Directors should be installed at each site.

3.7.5 ESCON Director Control

Each ESCON Director contains its own control unit, which is connected to a
reserved port address. Although this control unit is transparent to users, it
requires a defined path to each host that the director will be managed from.
This path can be shared with other connections; it does not need a separate
channel. This system connection allows the management of the switch matrix
and other functions to be performed from the mainframe, using ESCON Manager
or HCD, without access to the director′s console.

This is a very useful facility to build into your operation. It makes it much easier
to co-ordinate system definition changes and any switch matrix changes that are
associated with them. It is strongly recommended to have this facility on all
attached processors in addition to the ESCON Director Console. This
combination makes a loss of ESCON Director management very unlikely.

When chained directors are used, the only way that a processor can control the
second director in the chain is to have a dedicated link in the first director. A
dedicated link in the second director cannot dynamically switch and connect to
the internal control unit.

This control facility must be considered when designing a chained director

configuration.

3.8 Management
When considering a hardware configuration to support continuous availability, it
soon becomes obvious that it probably becomes more complex than those built
simply for service delivery from a single processor.

56 Continuous Availability S/390 Technology Guide

 This complexity brings the challenge of safe and effective management to ensure
that unwanted outages are not introduced because the environment was not fully
understood and a wrong action taken.

It has been shown earlier in this chapter that HCD is the main tool to build and
dynamically change the ESCON environment, but in day-to-day operation,
another tool is required. ESCON Manager was developed to perform this task.

3.8.1 ESCON Manager

3.8.1.1 History
ESCON Manager was originally introduced as a product in its own right
(5688-008) and developed through three releases. It has now been incorporated
into the System Automation for OS/390 product (5645-005). The function
described applies to both the final release of &escm, and to the component of
System Automation for OS/390 .

3.8.1.2 Function
ESCON Manager works dynamically with all OS/390 systems, HCD, the channel
subsystem microcode, and the installed hardware to build and maintain a picture
of the current configuration and other legitimate connections. It also works with
other ESCON Manager components on other OS/390 images to obtain their view
of the same hardware components. This view can be filtered and manipulated
by system operators using menus supplied with the product. ESCON Manager
performs these tasks, regardless of whether the component is attached to a
parallel channel or to an ESCON channel. It can also discover and display
information on other hardware items associated with a Parallel Sysplex (coupling
facility and links), and the Open Systems Adapter (OSA) used on S/390 systems.

When the status of an individual item of hardware has been determined, ESCON
Manager can also perform any change (vary offline or online) decided by the
operator, and then schedule this change across all sharing systems.

The scope of this vary command can also be limited, if required. It contains a
very sophisticated checking system to ensure that the command is valid and
legal for the device, and also a backout facility to restore the status of the device
if anything prevents the command from completing successfully. In addition, it
can also be used to check the switch port connections and attributes in an
ESCON Director, and then change them as required.

ESCON Manager also contains a documented interface to allow it to combine

with mainframe automation products to achieve a high degree of console
automation. It now is part of IBM′s prime mainframe automation product,
System Automation for OS/390.

With its (multi-system) dynamic search and control ability, ESCON Manager is
uniquely placed as a tool to aid operator knowledge and decision-making.
Better operator tools and aids help achieve the target of high availability, by
avoiding unscheduled outages. ESCON Manager can also influence continuous
operation by managing the efficient reconfiguration of hardware paths for
workload switches between processors, or similar activity. ESCON Manager
therefore has an extremely important role to play in the goal of continuous
availability.

Chapter 3. S/390 ESCON Architecture 57

3.9 Advantages and Benefits Summary
This chapter provides an introduction to how the ESCON Architecture and
ESCON devices contribute towards continuous availability. For further
information, refer to the publications cited and also to the bibliography at the end
of this book.

In summary, the ESCON Architecture offers the following continuous availability

benefits:
• Longer distance connections (enhances disaster recovery)
• Higher speed data transfers
• New connectivity standards (ESCD & EMIF)
• New path management possibilities (ESCD)
• Dynamic re-definition (HCD)
• Non-disruptive upgrades and fault-tolerant features (ESCD)
• Better operator feedback and control (ESCM)

58 Continuous Availability S/390 Technology Guide

Chapter 4. OS/390

OS/390 represents a new way of doing business. It is delivered to you in a new

way. Its release schedule is regular and predictable. You will be able to install
it easily and in a shorter amount of time. Combine this with its track record of
99.99% continuous availability, and the tradition extends.

S/390 Parallel Sysplex continues to provide the basis for OS/390 availability
improvements. OS/390 builds on the high availability inherent in a sysplex
because its functions have been tested in an integrated fashion.

The long term OS/390 trend is to continue to provide functional consistency,

extensive pre-testing, and continued integration of products and features.
Migration to new versions and releases will continue to require less time, while
providing a more stable and available environment at production
implementation.

4.1 OS/390 Availability Attributes

Table 4 shows you which availability attribute applies to frequency, duration, and
scope of an outage, and which outage type it belongs to. We distinguish
between planned (continuous operation) and unplanned (high availability)
outages.

Table 4 (Page 1 of 2). OS/390 C/A Attribute Summary Matrix. Legend: X = A p p l i e s ,

Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Function Integration X X X CA
Pre-Tested System Software X X X CA
Dynamic Link Pack Area X CO
RRST X CO
Sysplex Star Topology X CA
Work Load Manager X CA
Dynamic Dump X CO
HCD Dynamic I/O Verification X CA
SmartBatch X CO
DFSMS X X CA
JES3 X CO
Recoverable Resource X X X CO
Management Services
Open & Industry Standards X X X CA
Integrated Server Platform X X X CA
UNIX 95 Certification X X X CA
Year 2000 Certified X X X HA
Windows NT Support X X X CA

 Copyright IBM Corp. 1998 59

 Table 4 (Page 2 of 2). OS/390 C/A Attribute Summary Matrix. Legend: X = A p p l i e s ,
Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Force Sysclone Uniqueness X CO
Peer-to-Peer Remote Copy X CA

4.1.1 Availability Enhancements

4.1.1.1 OS/390 V1R2 Availability Enhancements:

• Global Resource Serialization (GRS) Star Topology
• XCF Coupling Facility Failure Policy
• XES lock support
• Parmlib concatenation
• LNKLST and LPALST enhancements
• PARMLIB Symbolic Pre-processor
• Force Sysclone Uniqueness
• ICKDSF R16
• UNIX 95 certified
• DCE Distributed File Service (DFS)
• DCE Network File System (NFS)
• DFSMS/MVS Network File System
• Access Method Support

Global Resource Serialization (GRS): To better manage serialization in a

sysplex, the GRS component is changing from a ring to a star topology. This
change greatly reduces the impact of losing a system in the sysplex and reduce
the amount of real storage consumed in each system as the number of systems
in the sysplex increases. The coupling facility acts as the hub of the star.
Performance improvements will be evident in system initialization, sysplex
partition, and ENQ responses.

XCF Coupling Facility Failure Policy: This policy is enhanced to optimize the
selection of a coupling facility during structure allocation. It also gives
applications a way to define their connectivity requirements during an initial
connect. The rebuild process is also enhanced to guarantee that the rebuilt
structure has better connectivity to the systems in the sysplex than the old
structure.

XES lock support: Locking is enhanced to support the GRS star topology.

Parmlib concatenation: This support allows the concatenation of ten additional

data sets to SYS1.PARMLIB at IPL, providing greater flexibility in the
management of PARMLIB in the installation. For example, multiple PARMLIB
data sets could isolate certain members that are frequently changed or require
RACF protection on a more granular basis.

60 Continuous Availability S/390 Technology Guide

LNKLST and LPALST enhancements:
• The number of LNKLST and LPALST extents has been increased to 255.
• The ability to concatenate in front of system libraries in the LNKLST and
LPALST concatenations is provided.

PARMLIB Symbolic Pre-processor: This support allows a customer to inspect

the actual input that is used for an IPL. The ability to use symbolics in PARMLIB
reduces the effort required to manage and maintain PARMLIB members. This
support is enhanced to support PARMLIB concatenation and allows customers to
see what a selected PARMLIB member looks like with all symbolics resolved.

Force Sysclone Uniqueness: In MVS/ESA SP 5.2, sysclone was introduced to act

as a short and unique identifier of a system in a sysplex. The customer had an
option about whether to specify uniqueness. To support VTAM, which expects
sysclone values to be unique so that it can generate communications addresses
within the sysplex, OS/390 Release 2 enforces uniqueness.

ICKDSF R16: Release 16 Refresh with Peer-to-Peer Remote Copy adds a new
command, PPRCOPY, to ICKDSF. On those subsystems that have Peer-to-Peer
Remote Copy, PPRCOPY supports synchronous copying of a DASD volume from
one subsystem to another subsystem volume, without host system
dependencies. Use the PPRCOPY command to:
• Establish ESCON paths between two subsystems
• Establish remote copy device pairs

UNIX 95 Certified: In order to achieve full UNIX 95 branding from the X/Open
Company, OS/390 has added new programming and shell interfaces. In addition,
changes have been made to improve performance.

DCE Distributed File Service (DFS): With the DFS client, DFS becomes an
integral part of OS/390 OpenEdition DCE, offering a seamless data repository.
Distributed data with DFS confers such benefits as consistency among
distributed files, improved response time and data availability, access to the
OpenEdition Hierarchical interrupting end users, and directory and security
services that are integrated with OpenEdition DCE.

The DFS client communicates with file server machines to access files for
application programs and to provide local data storage. Also called the cache
manager, the DFS client translates file requests made by application programs
on a client machine into remote procedure calls (RPCs) to file exporter
processes on file server machines. The cache manager caches all data it
requests by storing it on disk or memory before passing it on to the requestor.
In addition, DFS ensures that the cache manager always has access to the most
current copy of the data. If the data changes, the cache manager retrieves the
most current version, automatically and totally transparently to the user.

DCE Network File System (NFS): In addition, for OS/390 Release 2, OpenEdition
DFS provides an NFS-to-DFS gateway. The gateway will allow NFS clients
access to the MVS system by using the MVSLOGIN command and RACF security
services. Users of NFS clients, once authenticated by enrolling on an MVS
machine, can access data in the DFS filespace.

DFSMS/MVS Network File System V1R3 (NFS): A new Network File System client
is provided by the DFSMS/MVS Network File System on the OS/390 platform.

Chapter 4. OS/390 61
 The NFS client program on the client system maps local file system calls into
network calls and sends a request for data or control information to the NFS
server. In this way, NFS clients and servers can deliver remote data to an
application without special interfaces in the application. The client program
supports the SUN NFS Version 2 protocols. It uses RPC requests to
communicate with remote NFS servers over a TCP/IP network.

DFSMS/MVS provides data set attribute accessibility for DFSMS-managed data

sets, even when they have been migrated by DFSMShsm. This means that a
DFSMS/MVS NFS server is able to obtain the attributes of migrated data sets
without physically recalling them.

Access Method Support For OpenEdition MVS Files: Many applications written to
use the BSAM/QSAM access methods, and some applications written to use the
VSAM ESDS access method, now have transparent access to files associated
with OpenEdition. With this support, the conventional MVS application, using the
normal record or block interfaces provided by the MVS access methods, can
access files residing on remote NFS server platforms.

4.1.1.2 OS/390 V1R3 Availability Enhancements:

• LNKLST
• TSO, DB2 Workload Management (WLM) support
• TSO sysplex support
• DB2 Stored SQL Procedures support
• DB2 TCP/IP support
• Sysplex I/O priority management
• System Logger enhancements
• Sysplex availability
• XCF system initialization
• TSO/E session distribution
• SAP R/3 application enablement
• UNIX services
• DFS server export record data
• DFS OS/390 configured using ISPF
• VTAM multi-node persistent sessions (MNPS)
• ICSS MVS dataset support
• ICSS S/390 cryptographic hardware support
• ICSS is Workload Manager(WLM)-enabled

LNKLST - Further enhancements to LNKLST processing allow LNKLST to be

changed without requiring an IPL.

Workload Management (WLM) support by TSO, DB2 is provided.

TSO sysplex support for VTAM Generic Resource, which distributes TSO logons
to available capacity in a coupled systems environment, is provided.

62 Continuous Availability S/390 Technology Guide

DB2 Stored SQL Procedures support is available, which provides the capability
for DB2 to allow its stored procedures to run in multiple server spaces. Enclaves
have been enhanced to support TCB work units in addition to SRB work units.
(These services are also available to other work managers that need a dynamic
pool of server address spaces.)

DB2 TCP/IP support is available, which provides the ability to spread work
across a DB2 datasharing group using the WLM recommendation given in the
sysplex routing selection service.

Sysplex I/O priority management is available, which provides dynamic

management of I/O priority for work across the sysplex to address DASD usage
and delays.

System Logger: System Logger enhancements include remote site recovery,

which allows log stream data at a local or primary site to be “shadowed” at a
remote or secondary site.

Sysplex Availability: Availability is enhanced by providing a descriptive ENF

signal whenever a system joins or leaves a sysplex, providing more extensive
information than before to automation and monitoring programs.

XCF system initialization: Processing is enhanced to eliminate, in many cases,

operator intervention when a system has been re-IPLed without having been
partitioned from the sysplex. XCF signalling also provides extensions to allow
components or applications to broadcast requests to multiple recipients in a
sysplex and to consolidate the responses.

TSO/E session distribution: High availability for mission-critical applications in

the network is enhanced through the communications server′s extension of
generic resource support to TSO/E. This provides the capability to balance
session distribution across TSO systems in the sysplex via the coupling facility.
This also includes support for the reconnection to the original task if the TSO/E
user loses connection to the systems and logs back before the reconnect interval
has expired.

SAP R/3 application enablement: Through OpenEdition, OS/390 is now a

database server for SAP R/3, allowing SAP R/3 applications to profit from
client/server technology benefits such as distributed processing and extensive
scalability. SAP R/3 applications, user data, and process models are stored on
the database server. SAP R/3 uses DB2 as the database server for OS/390.

UNIX services: The OpenEdition kernel starts at IPL so that UNIX C services are
always available. Users no longer need to write code to deal with kernel
termination and restart, simplifying server application programming. Additional
enhancements include increased NLS support. Also included is Web thread
support, allowing a server to dynamically change its environment without having
to be restarted.

Distributed Computing Environment (DCE) Distributed File Service (DFS) (OSF

DCE level 1.1):
• The DFS Server can now export OS/390 record data, including access to
VSAM, sequential, and PDS and PDS/E types of MVS files. With this
enhancement, MVS record data is now available to DFS client applications.

Chapter 4. OS/390 63
 • DFS for OS/390 can now be configured using ISPF dialogs. This improvement
will provide ISPF panel-driven configuration and unconfiguration that users
are accustomed to from DCE base addition services.

VTAM 4.4: High availability for mission-critical applications in the network is

addressed through multi-node persistent sessions (MNPS). MNPS provides for
the transparent recovery of VTAM, MVS, hardware, or application failures by
restarting the critical application on another host in the sysplex.

Internet Connection Secure Server (ICSS) MVS dataset support: With this
feature, World Wide Web documents may be retrieved from native MVS datasets
as well as from the MVS/Open Edition Hierarchical File System.

S/390 cryptographic hardware support provides the option of using hardware

instead of software to perform the encryption and decryption if the required
hardware is installed on the host machine.

ICSS 2.2 is Workload Manager(WLM)-enabled, which allows:

1. ICSS tasks to be managed by WLM to customer-specified goals and policies
2. ICSS work to be spread across multiple address spaces and systems in a
sysplex, which will improve the number of requests processed per second
3. Improved ICSS availability, because WLM will re-start address spaces that
have abended

4.1.1.3 OS/390 V2R4 Availability Enhancements:

• A new facility, the Dynamic Link Pack Area (DLPA), allows optional and new
products to be loaded into the system without an IPL.
• DCE (DFSMS) NLS enablement
• DCE (DFSMS) Parallel Sysplex enablement
• DCE (DFSMS) performance enhancements
• DCE (DFSMS) RAS enhancements
• DCE (DFSMS) S/390 hardware crypto enablement
• OpenEdition file caching: This support allows files to be cached in virtual
storage in a data space associated with the kernel.

4.1.2 OS/390 - Continuous Availability

4.1.2.1 Function Integration

OS/390 delivers an integrated set of MVS, UNIX, LAN, distributed computing, and
application enablement services through its base elements. These base
services provide a rich set of operating system functions that enable open,
distributed processing, and offer a foundation for object-ready, mission-critical
support.

Communication server:

The communications server includes VTAM (including the AnyNet function),

TCP/IP, TIOC, and FFST. It provides SNA (3270), APPC, high performance
routing, ATM support, sockets, and RPC. Some of the benefits are:
• Easy deployment of client/server applications

64 Continuous Availability S/390 Technology Guide

 • Function for new network computing applications
• Multi-vendor, multi-platform connectivity
• High performance
• High availability
• Network choice

Security server:

The security server includes RACF and the OpenEdition DCE security server.
• It allows a single sign-on to the RACF and DCE domain
• It gives enterprise-wide security
• It enables porting of applications that use DCE security services to OS/390
• It allows DCE application services to associate a DCE user with a
corresponding RACF-defined user

IBM Internet Connection Security Server for OS/390:

Businesses increasingly wish to use the Internet to market products and conduct
business with suppliers and customers. The IBM Internet Connection Secure
Server for OS/390 enables the use of S/390 as a Web server.

The benefits of the IBM Internet Connection Secure Server for OS/390 are that
you have S/390 security, the utilization of large storage capacity on S/390, the
use of centralized skills, a single point of entry and control, consolidation of
multiple Web sites, and secure Internet transactions.

4.1.2.2 Functional Groups

All services can be divided into the following functional groups:
• System services provide the classic strengths of MVS, rock-solid reliability
and availability, and support for high transaction workloads and many users,
with optimum performance.
• Security server provides full function security services.
• Systems management services provide a window to enterprise-wide systems
management.
• Application enablement services allow not only a broad application execution
environment, but facilitate rapid development of applications, improving the
time-to-market for new business function.
• UNIX services give UNIX applications S/390 “industrial strength.”
• Distributed computing services provide application interoperability and
client/server processing.
• Communications server opens OS/390 for networking applications.
• LAN services exploit S/390 as a LAN application and data server.
• Network computing services give secure access to the Internet.
• Softcopy services improve productivity in systems installation and
management.

Chapter 4. OS/390 65
 OS/390 is the base for future enhancements to S/390 software, and in
combination with IBM′s new CMOS parallel enterprise servers, provides you with
the opportunity to improve your continuous availability.

4.1.2.3 System Pre-Tested

IBM′s OS/390 systems integration test environment mirrors that of a large
computing enterprise. To ensure that these tests benefit a wide range of
customers, the environment simulates a variety of diverse applications such as:
data entry, hotel reservations, inventory tracking, order entry, product
specifications, stock control, a teller system, and image storage, retrieval and
manipulation.

The environment simulates between 8500 and 12000 end users and performs
between 400-to-520 transactions per second with an internal response time goal
of 80% of CICS transactions completed in less than 0.6 seconds. There are 108
CICS regions, 27 IMS regions and nine DB2 regions.

The hardware configuration is a 10-way Parallel Sysplex, of which nine system

images run production work and one is a test system. The configuration
includes both bi-polar and CMOS machines. The hardware configuration
includes 1.6 terabytes of DASD, 3990 control units, 3490 tapes, 2 coupling
facilities, ESCON Directors, ESCON channels and terminals.

Each of the integrated elements and optional features of OS/390 have been put
through a traditional product development test cycle. This ensures that each
element and feature, when considered on its own, provides the function defined
in its specification. In addition, the elements and features are combined
(integrated) at one time and run in the simulated production environment. This
test ensures that the elements and features can be installed and run together
and have the robustness that is characteristic of the MVS platform. Particular
attention is paid to the following in a Parallel Sysplex environment:
• Continuous operation
• Recovery from hardware and software failures
• High transaction rates
• High CPU utilization

Additional emphasis in the integration test is placed on key subsystems. For

OS/390 R3 these are:
• CICS Transaction Server for OS/390
• IMS
• IRLM 2.1 (IMS Resource Lock Manager)
• DB2

The focus of this testing is Parallel Sysplex datasharing with workloads of the
following type:
• CICS with DL/I
• CICS with DB2
• CICS VSAM RLS (record level sharing)
• DB2 batch
• IMS BMP
• IMS DL/I batch
• IMS TM with DL/I
• IMS TM with DB2

66 Continuous Availability S/390 Technology Guide

Also installed in the OS/390 integration test environment are the products that
make up the SystemView for MVS suite and the Internet Connection Secure
Server. Some of the SystemView for MVS products are utilized in the
management of the Parallel Sysplex, and several are integrated into the OS/390
base as elements or as optional features.

4.1.2.4 Reduced Customer Testing

This new, more extensive testing of OS/390 helps customers to move more
quickly to new functions by eliminating multiple install iterations of IBM products,
and reduces customer testing. System integration testing is provided in Release
3 and all subsequent releases of OS/390.

All of the functions necessary for Parallel Sysplex have been tested in a Parallel
Sysplex environment.

Smaller enterprises, with a single system and just a few programmers, will find
that they too can keep current with new functions delivered in OS/390 releases
with far less work than before because of OS/390 systems integration testing of
each release.

There are detailed procedures documented in OS/390 R3 Planning for

Installation , GC28-1726, to help customers test the system.

Additionally, OS/390 Release 3, like Release 2, offers customers two new ways to
test applications on the system, particularly applications changed for the year
2000.

Customer feedback has indicated that the systems integration testing can save
them several months in moving to a parallel environment.

One of the OS/390 optional features is a CD-ROM pre-configured with OS/390

Release 3 that you can use on a PC Server S/390 or on a RS/6000 system with
S/390 Server-on-Board (R/390) installed.

A report on systems integration testing of OS/390, which focuses on installing

and running OS/390 in a Parallel Sysplex, is updated quarterly. It is available via
MKTTOOLS as OS390TST PACKAGE and on the OS/390 Home Page at
http://www.s390.ibm.com/os390.

4.1.2.5 OS/390 Delivery and Service

New releases of OS/390 are scheduled approximately every six months. Service
upgrades are integrated into each new release. Releases do not need to be
installed sequentially, that is, releases can be skipped. And although many of
the existing products and features whose functions are integrated into OS/390
remain available as stand-alone products at their current release levels, future
functional enhancements (with some exceptions) will be shipped only as part of
OS/390.

Because of your ability to “clone” images in a Parallel Sysplex, the rippling of

software changes across images does not inhibit high application availability.
While software changes are made to one image, an application can continue to
run on other images in the complex.

Figure 22 on page 68 shows how operating systems co-exist within a Sysplex.

Chapter 4. OS/390 67
 Figure 22. Operating System Co-Existence within a Sysplex, N, N + 1 , N + 2

4.1.2.6 N+2 Operating System Co-Existence within a Sysplex

Three consecutive releases of OS/390 can coexist in a multi-system complex or
sysplex, providing optimum compatibility and flexibility.

Subsystems such as DB2, CICS, and IMS will allow two consecutive releases to
coexist.

While it is expected that most installations will migrate to all elements of OS/390
at the same time, the migration of JES2 and JES3 can be staged. The OS/390
levels of JES2 and JES3 are contained in their own libraries and, beginning with
Release 3, in their own SMP/E zones. This makes it possible for you to use your
current copies of JES2 and JES3 more easily, if you so choose. For full details,
refer to OS/390 Planning for Installation Release 3 .

OS/390 provides the base for the terms and conditions of a Parallel Sysplex
environment. Its functions have been tested in a sysplex environment. It is
estimated that with OS/390, moving to a parallel environment can be done
significantly sooner than it can be done using a piecemeal approach. You also
have the ability to clone, allowing your applications to remain available to your
end users. Changing a release causes an IPL of only one system; the others
remain available to your users and applications.

68 Continuous Availability S/390 Technology Guide

4.1.2.7 Rolling OS/390 Across a Sysplex
To ensure that you maintain continuous availability while installing new versions
of OS/390 in your sysplex, you can stage the IPLing across the sysplex as shown
in Figure 23. While each installation will have its own schedule, the timing of
this staging could range from over a weekend to over a couple of weeks. It
should also be noted that while this example shows actions at the CPU level, the
example also applies to updates at the LPAR level.

Figure 23. Rolling System Maintenance into 7 X 24 Availability

In this example, we have a Parallel Sysplex that consists of four MVS images,
SYS1, SYS2, SYS3, and SYS4, using a coupling facility.
• SYS1 is at operating system Level N
• SYS2 is at operating system Level N
• SYS3 is at operating system Level N+1
• SYS4 is the test system at operating system Level N+2

SYS4 has been removed from the sysplex for a system software upgrade. For
the purposes of our example, it is at OS/390 V1R3 (or N+2, as relative to the
lowest operating system level installed in the sysplex).

Figure 23 shows the steps you need to follow when you want to propigate
production to this newer level of OS/390.
1. Add the new system to the Parallel Sysplex as a participating system and
not a test system.
2. Drop SYS1 from the sysplex, keeping SYS4 (the former test OS/390 system)
as a production system.

Chapter 4. OS/390 69
 3. IPL SYS1 with the OS/390 N + 1 or N + 2 level of code and re-connect to
sysplex.

Repeat this process for SYS2 and SYS3.

This roll-out process is conceptual. Whether you are running OS/390 or current
levels of the stand-alone products, you might have to perform certain migration
actions as you upgrade to OS/390 R3. For migration information, refer to OS/390
Planning for Installation V1R3, GC28-1726 .

4.1.3 SmartBatch

4.1.3.1 SmartBatch Goals

• Obtain elapsed time savings by running today′s serial batch jobs in parallel.
• Decrease batch job elapsed time by transparently splitting batch jobs into
units of work that can be executed in parallel on single system images and
multiple system images in a sysplex.
• Improve batch job throughput by dynamically balancing a batch workload in
a sysplex based on CPU and system resources.
• Improve resource utilization in a Parallel Sysplex by distributing a batch
workload across multiple systems.
• Decrease batch job elapsed time through the use of advanced I/O
improvement techniques.
• Eliminate I/O by passing intermediate data through memory instead of using
DASD or tape data sets.
• Reduce tapes/tape mounts, DASD usage, and reruns due to media failure by
eliminating intermediate data sets.

IBM SmartBatch for OS/390 is a major enhancement to, and a replacement for,
BatchPipes/MVS. It maintains the single system data piping capabilities of
BatchPipes/MVS and provides the following enhancements.

4.1.3.2 BatchPipePlex
BatchPipePlex extends the data piping capabilities of BatchPipes/MVS to batch
jobs operating in a Parallel Sysplex environment. By using the coupling facility,
BatchPipePlex can enable parallelism across OS/390 and MVS/ESA images and
provide an opportunity to distribute a batch workload to improve resource
utilization.

Use of BatchPipePlex support will help some installations to move from the large
water-cooled mainframe to the less expensive CMOS mainframes. Since
BatchPipePlex support provides installations with the option of spreading their
batch pipelines across a Parallel Sysplex, they can use the smaller, less
expensive CMOS mainframes to meet their data processing needs.

In cases where an installation has a large pipeline of jobs that could execute
concurrently, but the installation is unable to schedule the sequence of jobs all
together on a single system because of resource constraints, the installation can
establish a BatchPipePlex and spread the pipeline of jobs across more than one
system.

70 Continuous Availability S/390 Technology Guide

In the case of installations that want to write new applications that exploit the
Parallel Sysplex environment, and in doing so need to pass data between
systems, the BatchPipePlex support provides a very simple and natural way of
passing the data. Using the BatchPipePlex support, pipes can be established
between the systems in the Parallel Sysplex and messages or blocks of data can
be sent in sequential order using familiar access method interfaces such as
QSAM, BSAM, or GSAM-BSAM.

If an installation has applications that produce data on one system image

because of resource or data availability and then transports the data to another
image for processing, the data could be piped between systems. An example of
this use is where data is collected on one system, written to tape, and then the
tapes are transported to another system for reading and processing the data.
The use of BatchPipePlex support can avoid staging the data to tape.

4.1.3.3 Batch Accelerator

Batch Accelerator transparently splits batch jobs into units of work which can be
executed in parallel on multiple OS/390 and MVS/ESA systems. It determines
what jobsteps can and should be executed in parallel, splits the job into
individual units of work, and delivers them to alternate initiators on the same or
different systems. During execution, Batch Accelerator manages the I/O piping
and/or data sharing and merges the results back into a single job image.

Batch Accelerator uses a heuristic approach to jobstep scheduling. It “learns”

by watching data access patterns and recording them in a history repository. It
uses this information to make intelligent decisions on where and when to split a
batch job. It also has an option to make real-time job splitting decisions, and
can be used to target high CPU utilization job steps to high-speed processors.

Batch Accelerator provides the necessary administrative framework to integrate

and automate the IBM SmartBatch for OS/390 components. It analyzes data
access patterns and the dynamics of the current environment to determine which
jobsteps should be executed on which system. It automatically compensates for
differences in CPU speeds and I/O configurations to balance the workload.

4.1.3.4 Data Accelerator Performance Component

The IBM SmartBatch for OS/390 Data Accelerator performance component uses
memory and CPU to improve the efficiency of application I/O requests to VSAM
and non-VSAM data sets. Data Accelerator reduces the number of accesses to
disk and reduces or eliminates unnecessary wait times which subsequently
reduces job elapsed time.

Data Accelerator provides two levels of I/O performance processing, basic and
advanced.

At the basic level, Data Accelerator automatically determines and sets the
optimum buffering values for the utility or application.

At the advanced level, Data Accelerator optimizes I/O by using buffer

adjustments and an advanced combination of the following:
• Acquiring and managing its own buffers
• Decreasing I/O pathlengths
• Optimizing channel programs
• Controlling overlapping I/O

Chapter 4. OS/390 71
 • Providing larger data transfers per I/O operation
• Learning an application′s access patterns to restructure I/O requests

Data Accelerator provides selection criteria to control how I/O performance

processing occurs. These criteria include the following items:
• Type of file to be used (VSAM, non-VSAM, or both)
• Data definition name (DDNAME)
• Data set name (DSNAME)
• Job name
• Program name
• SMS data class name
• SMS storage class name
• RACF (TM) group ID
• RACF user ID

Data Accelerator also provides I/O performance options to control how I/O
performance processing occurs. The Resource Usage Bias option controls how
Data Accelerator obtains and uses resources to provide performance benefits.
The Dynamic Region Adjustment option allows Data Accelerator to automatically
modify region specifications to compensate for additional I/O-related storage
areas obtained on the application′s behalf. These and other performance
options can be used selectively to fine-tune application processing.

These Data Accelerator functions do not require any application or JCL changes.

4.1.3.5 Abend Detection and I/O Error Propagation Enhancements

Enhancements to the error detection and propagation functions of IBM
SmartBatch for OS/390 expand the scope of error detection and propagation to
include abends that occur between allocation to jobstep termination.

These enhancements add function to allow IBM SmartBatch for OS/390 users to
optionally synchronize the allocation and termination of their jobs, to ensure that
IBM SmartBatch for OS/390 propagates errors to all jobsteps that execute as a
unit of work. Users can specify the new subparameters, AllocSync, AllocNow,
and TermSync, on the JCL SUBSYS DD statement in their input jobstreams.

IBM SmartBatch for OS/390 is one of several software and hardware functions
that IBM offers to improve the elapsed time of batch jobs that run on OS/390 and
MVS/ESA. Others include:
• VIO to expanded storage
• Hiperbatch
• Sequential data striping

A comparison of these functions to IBM SmartBatch for OS/390 follows:

VIO to expanded storage:

Both VIO to expanded storage and the IBM SmartBatch for OS/390 BatchPipes
component eliminate the I/O wait times that occur when a job waits for I/O
transfers to and from storage devices. In addition, BatchPipes allows the writer

72 Continuous Availability S/390 Technology Guide

 and reader jobs to run concurrently, enabling additional elapsed time
improvements over VIO since VIO datasets cannot be shared between jobs.

However, VIO supports both random and sequential access to data, while
BatchPipes supports only sequential access.

4.1.3.6 Hiperbatch
Hiperbatch targets a different type of batch application from the IBM SmartBatch
for OS/390 BatchPipes component. With Hiperbatch, the target application has
many readers simultaneously accessing data from an already existing dataset.
The shared data all resides in processor storage and on DASD; the data persists
after being read. In contrast, for BatchPipes, the target application has one
writer passing data to one reader. BatchPipes holds very little of the data in
processor storage at one time, and the output from the writer job does not
reside on tape or DASD.

With Hiperbatch, readers run simultaneously with each other, but not with the
writer; with BatchPipes, writer and reader jobs run simultaneously.

4.1.3.7 Data Accelerator

Data Accelerator can be used for VSAM data sets being managed by Hiperbatch
for possible additional performance improvements.

4.1.3.8 Sequential Data Striping

Both sequential data striping and the IBM SmartBatch for OS/390 BatchPipes
component can improve the processing of batch jobs that spend much of their
elapsed time sequentially reading or writing large DASD files.

Sequential data striping substantially improves I/O access times to DASD. For
sequential data sets, BatchPipes can eliminate I/O and enable concurrent
processing.

Data accelerator can be used for data sets being accessed by sequential data
striping for possible additional performance improvements.

For further information, refer to IBM SmartBatch for OS/390 User ′ s Guide ,
GC28-1640. This book is softcopy only. You can find it on the OS/390 V2R6
online library, SK2T-6700.

4.1.4 DFSMS
DFSMSdss is a component of DFSMS/MVS which works with DFSMShsm to
provide the following availability management functions. Applications that
demand total availability need advanced capabilities like these:

4.1.4.1 Concurrent Copy

This hardware and software solution allows you to back up a database or
collection of data at a point in time with minimum down time for the data. The
data is unavailable only long enough for DFSMSdss to initialize a concurrent
copy session, which is a small fraction of the time the complete backup takes.

Chapter 4. OS/390 73
 4.1.4.2 S/390 Remote Copy
S/390 has two remote copy functions that will write data to a remote secondary
disk as it is written to a local primary disk. If a failure occurs on the primary
disk, up-to-the-minute data can be recovered from the copy. That copy can be
many miles away, allowing you to have a remote site for disaster recovery
purposes. These solutions provide backup copies with no loss of service.

4.1.4.3 Peer-to-Peer Remote Copy (PPRC)

Peer-to-Peer Remote Copy (PPRC) is a synchronous copy facility that ensures
that the data at the remote site is always synchronized with the local copy. It is
designed for those applications that need to ensure that the data is committed to
both copies before completing the I/O request. This involves a small increase in
I/O response time, depending on the distance to the remote site. However, it
keeps the primary and backup data synchronized to allow immediate recovery in
the event of a disaster.

4.1.4.4 Extended Remote Copy (XRC)

Extended Remote Copy (XRC) is an asynchronous copy facility that has minimal
performance impact on most applications during normal operation. Data is sent
to the secondary copy after the I/O to the primary copy has completed. There is
minimal increase in I/O response time. The secondary copy will be a few
seconds behind the primary copy, so there may be a small amount of data loss
when a failure occurs.

4.1.4.5 Backup Retry

This option allows you to specify that the system should delay backing up TSO
data sets which are open, and to retry later.

• HSM CDS in CF
• RMM/BTLS

4.1.5 JES3 Improved Availability in OS/390 R4

This provides support improves system availability in an OS/390 Parallel Sysplex
system by reducing/shortening outages needed to make JES3 configuration
changes.

This support is comprised of three parts:

1. Dynamic update support,
2. Refresh with hotstart support
3. Faster restart support

4.1.5.1 Dynamic Update Support

In OS/390 Release 4, JES3 will support dynamic update for adding or modifying
initialization statements for:
• SNA RJP
• RJPWS -- SNA RJP workstation characteristics
• CONSOLE -- SNA RJP consoles
• DEVICE -- SNA RJP devices (readers, printers and punches)
• VTAM Attached FSS printers

74 Continuous Availability S/390 Technology Guide

 • FSSDEF -- Functional Subsystem Definition
• DEVICE -- for VTAM attached FSS printers

4.1.5.2 REFRESH with HOTSTART

One of the major problems customers have with JES3 is the fact that it takes a
warm start to change many of the JES3 initialization parameters. The warm start
is very disruptive because not only must the JES3 global address space be
brought down, but all processors in the JES3 complex must be IPLed.

In OS/390 Release 4, JES3 will be changed to read the initialization stream

during a hotstart without IPL to allow many of the parameters to be changed. A
new start type called HOTSTART with REFRESH will be created to read the
initialization stream and process certain initialization statements (for example,
FSSDEF, RJPWS, non-execution DEVICE).

4.1.5.3 FASTER RESTART

There are occasions when you must restart the global processor, such as when
you need to implement certain configuration changes or to apply service. While
the global processor is down, jobs in execution may experience delays when
they request global services such as spool space allocation, opening SYSOUT
data sets, and so on. The HOTSTART described in 4.1.5.2, “REFRESH with
HOTSTART” that is needed to make the configuration changes will cause such
delays. JES3 RESTART processing is being enhanced to reduce the time it takes
the JES3 global processor to reinitialize.

4.1.6 Recoverable Resource Management Services (RRMS)

RRMS is IBM′s Strategy for extending OS/390 transaction processing capabilities
system services.

The following describes products and functions that could utilize RRMS to extend
transaction processing on OS/390.

In OS/390 V1R3, IBM provided as an integral part of the system, commit

coordination or syncpoint management as part of RRMS. The transactional
capabilities provided by RRMS and APPC/MVS are enabled through the service
stream via PTF for APAR OW23450. These services enable transactional
capabilities in recoverable resource managers. Additionally, these services
provide Protected Communication Resource Managers distributed transactional
capabilities in many OS/390 application environments.

The Recoverable Resource Management Services consist of three parts for

managing transactional work in OS/390:
1. Context Services for the identification and management of work requests.
2. Registration Services for identifying a resource manager to the system.
3. Resource Recovery Services or RRS to provide services and commit
coordination for all the protected resource managers participating in a work
request′s transactional scope in many of OS/390 application environments.

Where existing distributed processing capabilities do not exist, these new system
services enable OS/390′s existing OLTP information systems, IMS and CICS, to
extend their distributed processing capabilities such that clients and distributed
servers have transactional access to the OS/390 IMS/ESA Transaction Server
and the OS/390 CICS Transaction Server through the protocol that is appropriate

Chapter 4. OS/390 75
 for the distributed environment. Multiple client/server platforms have distributed
transactional access to new or existing OS/390 transaction and database servers
through one of the following distributed communication protocols:
• APPC/MVS support for SNA′s LU6.2 syncpoint architecture (introduced in
OS/390 Release 3)
• Transactional RPC support on OS/390
• Object Transaction Service based on OMG′s CORBA specification
• Message and Queuing based on MQSeries for MVS/ESA

IMS/ESA Transaction Manager Version 6.1, ACF/VTAM Version 4, APPC/MVS,

and RRMS in V1R3 provide distributed transaction support for IMS applications
over LU 6.2 conversations. This enables new and existing IMS applications to
participate in a distributed 6.2 commit scope. Through the use of the system
logger, ARM, WLM, and data sharing, distributed IMS transactions can take
advantage of work scheduling, balancing and recovery management of the
Parallel Sysplex. In addition to the OS/390 IMS/ESA Transaction Server, new
servers like DB2 Stored Procedures and OS/390 Object Servers as well as
existing application environments like batch, TSO/E and APPC/MVS, have
transactional capabilities for applications through these new system services.

Resource managers like DB2, MQSeries for MVS/ESA, transaction monitors like
IMS and CICS, and multiple communications protocols like APPC_PC,
Transactional RPC, and OTS can use these services to provide new transactional
applications, as well as combinations of new and existing transactional
applications. These capabilities allow new applications to take advantage of
existing investments in business applications, new application technologies,
distributed protocols of choice, the Parallel Sysplex, and the strengths of OS/390
for scalability, reliability, availability, security, integrity and the system
management of a distributed enterprise environment.

In the OS/390 Release 3 timeframe, DB2 provided a new OS/390 Attach Facility
that uses the Recoverable Resource Management Services. This enables DB2
to participate in commit scopes with other participating resource managers for
most OS/390 application environments. DB2 Stored Procedures also uses the
Recoverable Resource Management Services to participate in commit scopes
with other participating resource managers, as well as the commit scope of
other transaction servers.

Transactional RPC for OS/390 IMS/ESA Transaction Server can also be provided
for distributed transactions by an OS/390 Encina Toolkit Executive using
Recoverable Resource Management Services. Through similar extensions, DCE
servers and any OS/390 participating resource manager or transaction monitor
are able to participate in TRPC-initiated transactions. The distributed transaction
processing capabilities of OS/390 CICS Transaction Server can also be extended
to support transactional RPCs. This allows clients on heterogeneous platforms
and workstations access to the existing CICS and IMS transaction processing
environments for new and existing applications using TRPC. Since IMS and
CICS can participate as resource managers, this extends the transactional RPC
scope to existing transaction applications and potentially any of their databases
or resource managers currently accessible by them.

Object Transaction Service uses these same base system services to provide
distributed transaction capabilities for OS/390 object servers. The object servers
can include both OO resource managers as well as procedural resource

76 Continuous Availability S/390 Technology Guide

 managers as participants in a transaction through these same services. Existing
transaction monitors like IMS can participate as resource managers, thus
extending OO capabilities to existing procedural transaction applications as well
as to database managers.

MQSeries for MVS/ESA uses the Recoverable Resource Management Services

to enable transactional participation in most OS/390 application environments.
When combined with MQ′s guaranteed delivery, an asynchronous distributed
transaction processing model is available.

The integration of WLM and the Parallel Sysplex with DB2, OS/390 IMS/ESA
Transaction Server, OS/390 CICS Transaction Server, OS/390 object servers,
OS/390 application servers and RRMS, together with the work routing
capabilities of APPC/MVS, DCE, TCP/IP and VTAM, allows installations more
control and management of their business applications and the total resources of
the sysplex, consistent with the priorities and policies of their business
objectives.

In summary, these capabilities will, over time, allow new and existing
applications to take advantage of existing investments in business applications,
new application technologies, distributed protocols of choice, the Parallel
Sysplex, and the strengths of OS/390 for scalability, reliability, availability,
security, integrity and the system management of a distributed enterprise
environment for their transaction processing requirements.

4.2 Summary
The future for OS/390 is continued expoitation of Parallel Sysplex and ever
increasing improvements in continuous availability. The future for OS/390 is
more integration, more function, and more synergy between your applications.
OS/390 is a growing platform. It gives you continuous availability, with capacity
you cannot outgrow, at low incremental cost. Strategic initiatives include base
business enhancements in systems management and security, more and more
application enablement across secure networks, whether Internet or intranet,
and server consolidation, which means integrating smaller and multiple servers
such as file serving and print serving into OS/390. These initiatives ensure that
the goals you have set for your business are directly supported by the
technology you have installed.

Chapter 4. OS/390 77
78 Continuous Availability S/390 Technology Guide
Chapter 5. OS/390 Open Edition

OpenEdition offers OS/390 users a true UNIX environment, making it possible to

develop and port applications and data between OS/390 and any other standard
operating system on a workstation or a large computer.

Having achieved XPG4 UNIX 95 certification, OS/390 now provides customers

with full UNIX capabilities built directly into the operating system. This is a
critical differentiator from competing systems because only OS/390 can provide
UNIX facilities in tandem with those S/390 “classic strengths” (for example,
“continuous availability,” reliability, security, scalability) which businesses have
come to rely on. In fact, OS/390 is the IT industry′s first large server operating
system to achieve this level of UNIX certification. With UNIX 95 certification,
OS/390 customers are now provided with increased flexibility and more software
options than ever before. UNIX code can now more readily be accessed from a
variety of Internet sources for quick and seamless deployment on OS/390.

5.1.1 OS/390 UNIX Availability Perspective

Until UNIX became available in OS/390, there was no concept or implementation
of logical partioning (LPAR) for UNIX. For an alternate platform UNIX system to
achieve anything close to 99.9% availability, that UNIX system needed to be
clustered &semi,. that is, one system processing and a second system sitting idle
waiting to take over in case the first fails. For 24x7x365 availability, there is no
alternate platform UNIX solution.

Cost comparisons between S/390 and alternative platforms are affected by

customers expectations for the enterprise server. Customers expect the same
level of service and reliability on the alternative platform as they have on the
S/390 platform; however this is not the case.

S/390 is well known as having the right attributes to serve the enterprise, but it
is perceived to cost much more than alternative (UNIX) platforms. However, a
single alternate platform UNIX server for a single department running one
application is a low-cost platform, but to perform as an alternative to a single
S/390, a UNIX system must be able to:
• Handle multiple diverse workloads
• Provide high availability (99.9%)
• Give fast consistent response times (<3 seconds)
• Handle large amounts of data (GB to multi-TB databases)

With S/390 and OS/390, these attributes are builtin. In contrast, alternate
platform UNIX systems must supplement their basic configuration significantly,
which results in a more complex and expensive configuration that is difficult to
manage.

A customer study which compared the costs of an alternate platform UNIX

system with that of a S/390 was surprised to find that the costs for a UNIX
alternative were much higher as database size increased .
• 100GB S/390 costs 32% more than an alternate platform UNIX system
• 1000GB S/390 costs 37% less than an alternate platform UNIX system

 Copyright IBM Corp. 1998 79

 Nowhere does the difference between alternate platform UNIX systems and
S/390 appear more obvious than the ability to scale capacity in order to support
additional clients. S/390 with OS/390 is documented to support thousands of
concurrent users as is the case with our OS/390 Systems Integration test with
simulates 9000 to 12000 concurrent users, and provides less than 1 second
response time. Alternate platform UNIX systems have difficulty supporting more
than a few hundred concurrent users.

Most alternative platforms perform well with decision support workloads which
are reading data intensively. However, for mission-critical workloads like online
transaction processing (OLTP), fast processing, fast I/O, large buffers, and the
ability to maintain data integrity when multiple users request the same data
concurrently, a single S/390 performs best. To provide the same service,
alternative platforms must duplicate processors and DASD, increasing the
complexity and cost of their solution.

Customers have come to expect excellent (sub-second) response time from

S/390 as this is a key element of end-user productivity. Yet, in the 1990′s,
alternative platforms offer 5 - 10 second response time as their norm.

Just as important as response time is delivering a consistent response time.

OS/390 has a workload manager which has the ability to manage diverse
workloads while delivering consistent response times to the most important
workloads, and still utilize 95% or better of the processor. Alternate platform
UNIX systems cannot drive processor utilization at high levels for OLTP because
they cannot handle the sharing of resources well. Alternate platform UNIX
systems do not have a workload manager to balance workloads. To manage
multiple workloads, UNIX systems typically break up the workload onto separate
servers; one application to one server.

An alternative platform environment with 10 to 20 servers to manage is not

uncommon. A S/390 with OS/390 can process multiple applications and still
have sub-second OLTP response time.

5.1.2 OpenEdition Data Availability

OpenEdition MVS provides both user interfaces of MVS and POSIX. You can
move data between MVS data sets and the OS/390 OpenEdition hierarchical file
system (HFS). For example, you can copy or move MVS data sets into the file
system and you can copy or move HFS files into MVS data sets.

You can use TSO/E commands or shell commands. to work with the HFS. If you
have access to ISPF, you can use the panel interface of the OpenMVS ISPF shell.
For example, you can create a directory with the TSO/E command mkdir, or with
the shell
mkdir
.

Furthermore, you can issue TSO/E commands from the shell command line, from
a shell script, or from a program. Reference OS/390 OpenEdition User ′ s Guide ,
SC28-1891 for a list of TSO/E commands you can use to work with the file
system.

You can write MVS job control language (JCL) that includes shell commands.

80 Continuous Availability S/390 Technology Guide

 To edit HFS files, you can use the ISPF/PDF full-screen editor or one of the
editors available in the shell.

OpenEdition MVS extensions to the Restructured Extended Executor (REXX)

language enable REXX programs to access OpenMVS callable services. You
can run REXX programs using these extensions from TSO/E, batch, the shell, or
a C program.

You can use the OSHELL REXX exec to run a non-interactive shell command or
shell script from the TSO/E READY prompt and display the output to your
terminal. This exec is shipped with OS/390 OpenEdition MVS.

5.1.3 HFS Availability

HFS files are POSIX-conforming files which reside in HFS data sets, also referred
to as file systems. Figure 24 shows hierarchical file system (HFS).

Figure 24. UNIX 95 Hierarchical File System

Programs access the information in HFS files through OpenEdition system calls
instead of through standard MVS access methods and macro instructions. Data
files are identified and accessed by specifying the path leading to them.

HFS data sets appear to the MVS system much as a PDSE does, but the internal
structure is entirely different. DFSMS/MVS manages HFS data sets, providing
access to the files of data within them as well as backup/recovery and
migration/recall capability.

HFS data sets have the following requirements and restrictions:

• They must be system-managed, reside on DASD volumes managed by the
system, and be cataloged.
• They cannot be processed by standard open, end-of-volume, or access
methods; POSIX system calls must be used instead.
• They are supported by standard DADSM create, rename, and scratch.
• They are supported by DFSMShsm for dump/restore and migrate/recall if
DFSMSdss is used as the data mover.
• They are not supported by IEBCOPY or the DFSMSdss COPY function.
• An HFS file can not exceed the size of the (one) physical disk.

Chapter 5. OS/390 Open Edition 81

 5.1.3.1 Backing Up Open HFS Data Sets
DSS supports HFS for copy, dump and restore, except for logical data set copy.
Use DSS in the standard fashion; there are no special commands or parameters.

You can back up the datasets while open. You do not have to shut down the
OpenEdition address space. However, the same rules apply to backing up HFS
files as for backing up anything else; if they are open, they could be becoming
updated just before or just after they are copied for backup.

When DSS is the data mover in HSM, HSM can also back up HFS data sets;
however it backs them up successfully only if they are not open.

To determine when the HFS file is open, since the interface is through
OpenEdition, start an OpenEdition shell session and issue the df command. It
displays information about all mounted files and the relative MVS dataset names.
For example:
#df
/u/gronenb (OMVS.GRONENB.HFS) 14336/14400
/u/mmcdavi (OMVS.MMCDAVI.HFS) 14288/14400
/u/linetti (OMVS.LINETTI.HFS) 14360/14400
/u/vanaers (OMVS.VANAERS.HFS) 14336/14400
/u/newling (OMVS.NEWLING.HFS) 14320/14400
/ (OMVS.ROOT) 184/37440

The OMVS.** datasets are the MVS files that are HFS datasets. HFS files can be
mounted dynamically in OpenEdition by any authorized user, so you need to
check in a running system. This information is documented in more detail in the
OpenEdition manuals.

5.1.3.2 Exchangability
With OS/390 OpenEdition services, workstation users can copy data from MVS
data sets into hierarchical files for use at the host or, through Network File
System server support, at the workstation. Users can also upload
ASCII-encoded hierarchical files to the host, where they can be translated into
EBCDIC encoding for the host. These files can be stored or processed either as
hierarchical files, or copied to an MVS sequential data set or partitioned data set
member.

In OS/390, a hierarchical file system (HFS) is treated as a new kind of data set.
Like MVS data sets, HFS data sets can be allocated, backed up, restored, and
deleted.

In the event that the HFS is exposed to a power failure, it is designed to maintain
its data structure. You can lose recent data that is still buffered, but the file
system structures, directories, inodes and such will not be damaged. There is a
shadow writing technique that is used to ensure structural changes are always
committed atomically. The HFS does its own repair, as needed, on each mount
of a file system. This is based on records it keeps of changes in progress.

The same application program can access both traditional MVS data sets and
the hierarchical files within HFS data sets. In addition to the C language
interface, a callable service interface is provided for invocation from Assembler
H applications and subroutines.

Security for hierarchical files is provided through OS/390 system authorization

facility (SAF) interfaces used by RACF or any equivalent security product.

82 Continuous Availability S/390 Technology Guide

5.1.3.3 Individual Data Set Availability
IBM OpenEdition DCE Distributed File Service for OS/390 Release 3 (DFS for
OS/390 Release 3) provides additional function that allows workstation users and
applications efficient and secure access to OS/390 record data sets. Combined
with the DFS performance and error message improvements that are also
included in this release, the potential for use of DFS for OS/390 to support
workstation applications and user access is increased. Additionally,
configuration enhancements reduce the effort required to do required
administration tasks. DFS for OS/390 Release 3 includes the following new or
enhanced function:

Record Data Access: In addition to the OE HFS and the DCE Local File System
(DCE LFS) byte file data that DFS for OS/390 previously supported, DFS for
OS/390 Release 3 now supports DFS Client Access (TM) to the Sequential,
VSAM, PDS, and PDSE data sets that are allocated to direct access storage and
cataloged using the Integrated Catalog Facility (ICF).

DFS support refers to the Sequential, VSAM, PDS, and PDSE data sets as record
data sets, and to the data in these data sets as record data . DFS for OS/390
Release 3 can export a record data set for access through DFS client support on
workstations or other S/390 systems.

Network access to record data sets through DFS for OS/390 has the benefit of
Distributed Computing Environment (DCE) security based on Kerberos, and DFS
client caching for improved performance. The record data is presented to the
DFS client as byte stream data. EBCDIC to/from ASCII translation is also
supported for single byte code page (SBCS) file data.

5.1.3.4 Distributed File Service

OS/390 provides DCE Distributed File Service (DFS) client and server
functionality.

For end users, DFS provides the following benefits:

• Access to DFS data on OS/390 without regard to physical location from
anywhere in the distributed computing environment.
• Consistency among distributed files.
• Improved response time as a result of local caching.
• Improved availability of data.
• Improved security.
• The DFS server also supports export of MVS Record Data, Sequential (SDS),
Partitioned (PDS & PDSE) and VSAM (KSDS, ESDS, RRDS).

For DFS administrators, DFS provides:

• File sharing between the OS/390 Hierarchical File System (HFS) and the DCE
Local File System (DCE LFS).
• Directory and Security Services integrated with OS/390 DCE.
• Improved availability of data.
• The ability to perform routine server maintenance, such as load balancing
and data backups, while the server data is still in use by end users.

Chapter 5. OS/390 Open Edition 83

 • The ability to use S/390 system management software and high-speed
hardware devices.
• DCE Access Control List (ACL) support on LFS.

DFS server machines run processes that provide services such as making data
available and, on an administrative level, monitoring and controlling other
processes. Server machines are defined by the processes/daemons they run.
Among the processes DFS server machines run are these:
• The File Exporter (DFSKERN) fills requests for data from client machines
anywhere in the network.
• DFSXPORT determines the data to be made available to DFS clients
anywhere in the network.
• The Basic OverSeer Server (BOSERVER) monitors the DFS processes that
run on a server and restarts them when needed.
• The Backup Tape Coordinator (BUTCnn) controls a backup tape drive and
accepts service requests for backup.
• The Replication Server (RPSERVER) copies read-only replicas of read/write
data onto other file server machines.
• The System Control Server (UPSERVER) updates other server machines with
identical versions of system configuration files.
• The System Control Client (UPCLIENT) retrieves system configuration files
from the system control server.
• The Fileset Server (FTSERVER) provides an interface to the DFS commands
and components used to manipulate filesets.
• The Backup Database Server (BAKSERVER) houses the master and replica
versions of the Backup Database, where information used to back up and
restore system and user files resides.
• The Fileset Location Server (FLSERVER) houses the master and replica
versions of the Fileset Location Database (FLDB), where information about
the location of systems and user files is maintained.

DFS client machines, generally workstations, provide users with computational

power, access to DFS files, and other general-purpose tools. The Cache
Manager requests data for users from the processes running on file server
machines. A client Cache Manager on a non-OS/390 DFS system can work in
conjunction with OS/390 DFS to improve file access performance.

When the Cache Manager receives a packet of data for an HFS or DCE LFS file
from a file server machine, it stores copies of the data in a local storage area
called the cache. The cache is an area reserved for data storage on the local
disk or in memory on the client machine. The Cache Manager uses the local
copy of the cached file; it does not continue to send network requests to file
server machines for data it has stored locally. Access to locally stored data is
much quicker than access to data stored across the network.

As you save the changes you make to data, the Cache Manager periodically
sends the changed data back to the appropriate file server machine, where your
changed version replaces the data stored on the server. When the central
version of the file stored on the file server machine changes, DFS advises all
other Cache Managers with copies of the file that their versions are no longer

84 Continuous Availability S/390 Technology Guide

 current. When other users access the file, their Cache Managers use the newer
version of the data.

DFS User Environment

An OS/390 user can get access to DCE from either the TSO or the UNIX
environments in OS/390. Access to the UNIX environment is provided either
directly by using a Telnet session, or through TSO.

Since the UNIX environment in OS/390 provides all the native DCE tools and
utility programs, the UNIX environment is the best place to function for DCE
application developers and DCE administrators.

5.1.3.5 Transporting Hierarchical File System (HFS) with DFSMS

Some MVS installations install products and service on one system and then
transport this system image to the rest of their enterprise. Using the DFSMSdss
DUMP and RESTORE utilities, you can dump individual product libraries or full
volumes, transport them to other systems, and restore them.

You can do the same thing with the hierarchical file system (HFS). However,
since the individual HFS data sets that make up the active file hierarchy must be
on SMS-managed volumes, there are some special considerations for making a
transportable copy:
1. Dump each HFS data set to be transported into individual sequential data
sets using the DFSMSdss dump utility. These sequential data sets contain
all the necessary information about the files and can also exist with other
product libraries that need to be transported.
2. After the system image has been transported to the target system, restore
individual product libraries or full volumes using the DFSMSdss restore
utility.
3. After the data sets have been unloaded on the target system, use the DFDSS
restore utility to restore the sequential data sets into individual HFS data
sets. These HFS data sets will make up the active file hierarchy on the
target system.

Refer to the following documentation for information on how to dump and restore
a hierarchical data set:
• DFSMS/MVS Planning for Installation
• DFSMS/MVS DFSMShsm Storage Administration Guide
• DFSMS/MVS DFSMSdss Storage Administration Reference

5.1.4 ADSM
ADSTAR Distributed Storage Manager (ADSM) offers two types of backup for
OpenEdition MVS clients:
1. Selective, in which the user backups up specific files.
2. Incremental, in which all new or changed files are backed up.

Backup can be performed automatically or at your request. You can initiate a

specific type of backup or start the scheduler, which will run whatever action the
ADSM administrator has scheduled for the user′s machine.

Chapter 5. OS/390 Open Edition 85

 Information about using the OpenEdition UNIX client is described in; ADSM Using
the UNIX Backup-Archive Clients , SH26-4052 and ADSM Installing the Clients ,
SH26-4049.

5.1.5 File Caching

OS/390 Release 4 provides new high availabilty through data caching for HFS
files. This new capability allows files that are selected by the user to be cached
in virtual storage in a data space associated with the kernel. A considerable
reduction in pathlength, multiple concurrent use and a faster access improves
performance for frequently used read-only files. Significant performance benefits
may be obtained by caching the data. Many files that are read-only and are
accessed with high frequency will benefit, such as C header files. Internet and
intranet applications will benefit, because numerous files are accessed
frequently but rarely updated.

5.1.6 UNIX Processes and PR/SM

PR/SM allows a single S/390 processor to run multiple operating systems and
workloads that are isolated from each other. This allows users to separate
workloads based on business needs, or to restrict access to workloads where
different security clearances are required.

For a business that needs an extra measure of security, the Internet Connection
Secure Server for OS/390 can be run in a PR/SM logical partition environment to
separate public network connections from a private customer environment.

For example, a user who wants to connect to the Internet could run the Internet
Connection Server, their Web server and their corporate information systems on
the same physical system with assurance that the workloads will remain
separate. PR/SM on ES/9000 processors (9021 and 9121) achieved an E4 rating
against the Information Technology Security Evaluation Criteria (ITSEC), the
highest achieved by any vendor. There is no concept like PR/SM in any
RISC-based server. Therefore there is no RISC platform that can be used for
combining workloads with different security classifications.

Processor Resource/System Manager (PR/SM) provides a mechanism to

completely isolate up to 32 Internet Connection Secure Servers for OS/390 from
the rest of your S/390 workload.

This is accomplished by using logical partitioning (LPAR). A “secure” private

communications environment is used to connect ICSS and the production
environment connecting existing applications and data. Data and queues can be
shared across logical partitions. The public partition can have read/only access
to prevent “vandals” on the public network from corrupting production data,
while allowing the production partition to have read/write access to all the data.

5.1.7 UNIX Processes and OS/390

5.1.7.1 S/390 Gateways and Application Enablement

The fundamental value of network computing is the ability for any client to be
able to do business with any application on the network. Since the majority of
operational business data today resides on S/390, it makes sense to use this
platform as the hub to this connectivity, thereby exploiting your existing
infrastructure and application investment. The main tasks for Web-enabling
applications are to provide a secure Web server environment and to provide a

86 Continuous Availability S/390 Technology Guide

set of gateway interfaces from the Web server to back-end transaction or data
base servers with no application rewrite. Alternatively, the S/390 may wish to
connect to non-S/390 servers to provide a service to a client. Using S/390 and
ICSS for OS/390, IBM provides mechanisms to achieve this connectivity for your
enterprise.

5.1.7.2 CICS/ESA Internet Gateway

Customers with CICS/ESA and ICSS for OS/390 installed can access CICS
applications by writing programs using the Common Gateway Interface (CGI).
The gateway function uses the CICS External Presentation Interface (EPI) to
request CICS 3270 data, and translates it into HyperText Markup Language
(HTML), the format recognized by Web browsers. The CICS/ESA Gateway is an
IBM-provided CGI script that makes a Web browser appear like a 3270 terminal
with a more appealing interface. This means that CICS dialogs can now be
conducted via the Internet. Information and applications are immediately
available where they are needed most - on the desktop of the end-user.

It also means that your CICS applications, even when used internally (no Internet
connection), can have a much improved user interface, using the Web browser
as an alternative to the 3270 interface. No CICS specific software is needed on
the end user′s system. Further information on this capability is available on the
Internet at the CICS Web page:
http://www.hursley.ibm.com/cics/internet/index.html

5.1.7.3 CICS Gateway for Java

The IBM CICS Gateway for Java provides a new and powerful way to access the
established and popular transaction capabilities of CICS servers from the
Internet. It combines the portable, architecture neutral, object-oriented strengths
of the Java programming environment with the power, high integrity, robustness
and flexibility of CICS to bring state-of-the-art, open, easy, access from the
Internet to CICS applications running on a wide variety of server platforms.

The CICS Gateway for Java enables Java applications to communicate with CICS
servers through a CICS Client. It is itself a Java application which provides an
API that enables conversations between an application or applet in Java and a
transactional application running in a CICS server. Further information on this
capability is available on the Internet at the CICS Web page:
http://www.hursley.ibm.com/cics/internet/index.html

5.1.7.4 DB2 WWW Connection and Net.Data

DB2′s World Wide Web Connection makes it possible to turn your ICSS for
OS/390 web server into a simple, yet powerful SQL gateway for building
applications which query your DB2 data. IBMs Net.Data application is an
upwardly-compatible follow-on version of DB2 WWW Connection presently
available for download and evaluation purposes that allows application
developers to transform static HTML Web pages into dynamic Web applications
using Web macros. Web macros have the simplicity of HTML with the
functionality of CGI-BIN applications and power of SQL, making it easy to add
live data to static web pages. Live data includes information stored in
databases, flat files, registries, applications and system services. Net.Data
supports client-side processing and server-side processing with Java, REXX,
Perl, and C++, conditional logic and a rich macro language.

Chapter 5. OS/390 Open Edition 87

 Further information on these capabilities is available on the Internet at the DB2
Web page:
http://www.software.ibm.com/data/db2/db2wgafs.html
.

Information can also be found at the Net.Data Web page:

http://www.software.ibm.com/data/net.data/

5.1.7.5 IMS WWW Connection

Like the CICS/ESA and DB2 gateways, IBM also offers a direct connect facility for
the IMS/ESA Version 4 and Version 5 environments. Customers can use the CGI
interface to communicate with IMS transactions, and the information can be
returned to the Web browser for display and manipulation.

The IMS World Wide Web Connection provides a connection from Web browsers
into IMS/ESA through ICSS for OS/390 or workstation web servers. Further
information on this capability is available on the Internet at the IMS home page:
http://www.software.ibm.com/data/ims
.

5.1.7.6 MQSeries Internet Gateway

As part of IBM′s ongoing commitment to Internet-enabling our middleware,
applications, and development tools, there is a download and evaluation test
program for the MQSeries Internet Gateway. This gateway provides a bridge
between the synchronous World Wide Web and asynchronous MQSeries
applications.

With the gateway, an Internet-connected Web browser has access to MQSeries

applications through interaction with HTML fill-out form POST requests.
MQSeries applications respond by returning HTML pages to the gateway via the
MQSeries queue. Further information on this downloadable beta evaluation
program is available on the Internet at the MQSeries web page:
http://www.hursley.ibm.com/mqseries/beta/mqig/
.

5.1.7.7 AIF Internet Gateway

The Application Integration Feature (AIF) contains support for the AIF Internet
Gateway, which links the ICSS for OS/390 to AIF. With this capability, businesses
can use the Internet to access, via AIF, existing data and applications on S/390
or platforms connected to it. Additionally, they can initiate a complex business
transaction on S/390 with a single request from a Web front-end. AIF invokes the
necessary back-end applications to process the request, aggregates the results,
and returns a single reply.

AIF applications do not have to be aware of HTML format and vice versa. All
HTML features, including any kind of graphics and animation, can easily be
combined with AIF-based S/390 business applications. Further information on
this capability is available on the Internet at the AIF home page:
http://www.s390.ibm.com/products/aif/aif.html

88 Continuous Availability S/390 Technology Guide

5.1.8 Verifying UNIX Availability
The OpenEdition Setup Verification Program (SVP) lets you check for
troublesome setup errors before they trip you up. After you have followed the
instructions in OpenEdition Planning , SC28-1890, and completed your
OpenEdition setup and customization (including the shell and utilities), you can
run the SVP.

You can also customize it to your local system and run it whenever you suspect
problems.

Using the SVP, you can:

• Verify that each user has a UID and OMVS segment defined, and each group
has a GID.
• Check for duplicate assignment of UIDs and GIDs.
• Verify that each user has access to and owns a home directory and has
read, write, and search access to it.
• Check the permissions for several directories usually set up at installation.
• Check that files in the /dev directory are defined correctly. Reconcile the
number of pseudo-ttys and file descriptor files with the BPXPRMxx
definitions.
• Verify that the OMVS command will run.
Check customization for utilities. The program checks:
− Files that have been copied from /samples to /etc
− Terminfo files
− Settings for some environment variables
− Ability to compile and run a program
− Performs various other checks
If it detects a problem, the SVP warns you about it and, if you request,
corrects the problem. We estimate the SVP can take up to one-half hour to
complete; but the exact amount of time depends on your system.

Loading and Running the Program:

You must download the Setup Verification Program. Perform an anonymous-ftp:

1. ftp www.s390.ibm.com
2. Enter anonymous for user name.
3. cd os390/oe/toys/oesvp

5.1.9 Continuous Availability and Internet Servers

As more and more people become network-connected and businesses adopt
network computing models to launch electronic commerce, collaborative, and
information-rich content applications, the need for highly reliable and secure
enterprise Web servers such as S/390 and the Internet Connection Secure
Server for OS/390 will increase.

Businesses can rest assured that as they evolve their network computing
strategies and seek to revitalize existing applications and deploy new ones

Chapter 5. OS/390 Open Edition 89

 quickly for competitive advantage, the reliability, scalability, and security of a
S/390 Web server will be there to make this transition to e-Business a reality.

5.1.9.1 Internet Connection Secure Server (ICSS)

Internet Connection Secure Server for OS/390 enables S/390 to become a
scaleable Web server with up to 1,000 address spaces, each capable of
supporting 100 concurrent requests. This means that S/390 can have up to 1,000
times more capacity than any other computing platform to handle Web requests,
including commerce-based transactions. ICSS, working with other OS/390
platform products, affords the capability of large numbers of concurrent
connections with dynamic and scaleable Web site management based on
customer goals and priorities.

The use of threaded transaction programming integrated into the Internet

Connection Secure Server for OS/390 allows it to support small, large and global
Internet services with ease. The latest release, ICSS V2.2, has been
re-architected to exploit OS/390′s Workload Manager. Through classification and
goal-oriented scheduling, OS/390 Workload Manager and ICSS will ensure that
Internet-based workloads are satisfied to level of your specification and that
customers are not receiving a “server is busy” message.

Additionally, IBM has announced the intention to enable ICSS for OS/390 to
participate in the S/390 Parallel Sysplex environment. Given this degree of
flexibility, Internet projects can start with minimal configurations and can be
scaled to handle thousands of users with demanding e-business workloads.

An organization can serve up one or many Web sites from a single copy of ICSS
for OS/390. This foundation product provides a high performance Web/proxy
serving solution that continues to be enhanced with added security,
management, monitoring, and application development features, along with
state-of-the-art Internet protocol support.

ICSS for OS/390 may be configured to operate as either a Web server or proxy
server. A proxy server, which makes requests on behalf of clients, can simplify
network design, provide additional privacy, and improve responsiveness by
caching frequently accessed information.

ICSS is also an essential base product when used in conjunction with IBM′ s
Net.Commerce for OS/390.

The Net.Commerce server enables businesses to open up a virtual store front on

the Web to merchandise products and services, plus accept payments via a
secure payment system that will utilize the Secure Electronic Transaction (SET)
protocol.

In a survey conducted last year by technology consultancies Arthur D. Little and

the Giga Information Group(*), nearly half of the respondents rated security as
one of the most significant barriers to consumer acceptance of electronic
commerce.

The Internet was not initially designed to protect confidential or sensitive

information. Internet providers and users are not regulated. Anyone can use
the Internet or become an Internet Service Provider. Internet users have no
control over the path across which their information flows. Computers
connected to the Internet can be accessed by anyone. Unless some form of

90 Continuous Availability S/390 Technology Guide

security is used, both information being communicated and resources used for
communication are unprotected from mischief, malice or inadvertent misuse.

The Internet Connection Secure Server for OS/390 supports the latest industry
standard RSA public key cryptography algorithm in the form of Secure Sockets
Layer (SSL). This means that security can be applied to transactions to prevent
tampering and keep dialogs private when working over the Internet or intranets.
In addition, each party in the dialog can be authenticated, thus removing the fear
of conducting business with impostors. However, cryptography does have an
associated system cost. Each outgoing and in coming transaction that requires
this level of security must be encrypted and decrypted. This operation is
generally performed in software that will have an impact on processor
performance.

The S/390 Parallel Enterprise Server′s G3 & G4 and S/390 Multiprise 2000
servers address this challenge by performing cryptographic functions in an
exclusive specialized hardware function called the Cryptographic Coprocessor
feature. As a result, the main processor(s) is freed to concentrate on application
functions, and is not burdened with the encryption/decryption process. This new
capability, coupled with resource managers such as RACF or the OS/390 security
server, ensures that system resources are protected from unauthorized access,
making S/390 one the most secure environments in which to conduct electronic
commerce over the Internet.

System security starts with the integrity of the operating system. Resource
protection managers like RACF are used to protect the OS/390 system
authorization libraries from unauthorized access.

The Internet Connection Secure Server for OS/390 has extensive access control
support when the option to use Resource Access Control Facility (RACF) is
chosen to perform validation of user IDs and passwords and access control
through surrogate user ID support. RACF has proven to be the method of choice
over the use of basic server security.

Basic server security is implemented using the configuration facility which is

common on most Web servers. Basic server security, which typically uses
statements in the configuration file called directives or sub-directives , examines
header information from an incoming request from a Web browser.

However, it is possible for a hacker coming from a network that you do not
control (for example, the Internet), to forge the header information so that it
falsely identifies a trusted client. It is virtually impossible to detect when
someone impersonates another user unless you are using a resource protection
facility like RACF.

Through security authentication mechanisms, RACF, one of IBM′s leading

SecureWay products, and the Internet Connection Secure Server for OS/390,
protect resources, making it almost impossible for unauthorized users to
compromise OS/390 resources, including Web pages.

With the introduction of Internet Connection Secure Server for OS/390, Version 2,
Release 2, IBM is continuing to deliver on its promise and commitment to
provide customers with industry leading network computing technology and
solutions for the S/390 environment.

Chapter 5. OS/390 Open Edition 91

 5.1.9.2 ICSS Support for S/390 Features
Up to now, the ICSS for OS/390 has used software algorithms to encrypt and
decrypt data flows. In this release, the server has been enhanced to detect the
presence of the Cryptographic Coprocessor feature on the S/390 system. If
present, the encryption and decryption functions will be performed using the
hardware rather than software. This provides higher performance and increased
data security capabilities to customers.

Using the Workload Manager (WLM):

1. Allows tasks to be managed by WLM to customer-specified goals and
policies
2. Allows work to be spread across multiple address spaces which will
significantly improve the number of requests processed per second
3. Improves availability because WLM will re-start address spaces that have
abended

Dynamic workload management allows policies to be established for different

response time criteria and system workload adjustments during differing periods
of the day (for example, between first and second shift).

OS/390 System Measurement Facilities (SMF) log support providing configuration

data to be written at startup, and performance data to be written at
user-specified intervals.

OS/390 Dataset Support for World Wide Web documents is provided as well as
OS/390 OpenEdition Hierarchical File System document support.

Additional OS/390 Console Support allows the server administrator to shut down
the ICSS in an orderly manner and use the OS/390 Modify Command to pass
requests to the server. An example of this is turning debug on or off.

Web Usage Mining provides user-based statistics on path traversal, session

data, and grouped pages per session. This capability piggybacks on the
enhanced logging and reporting. Some uses of Web Usage Mining might be to
help organize a Web site more efficiently, determine the relative value of pages,
and target marketing based on page groupings.

The Internet Connection Application Program Interface (ICAPI) has been

improved by providing additional messages and improved trace capability.
Users may use existing HTTP methods or tailor existing HTTP methods to satisfy
their particular requirements. The IBM httpd API includes a NSAPI compatibility
module that supports the Netscape API. Using this module, you can easily port
your existing NSAPI programs to run on the IBM Internet Connection Servers
without any loss of function. Use of this module results in performance
improvements with CPU requirement reductions, higher throughput, and shorter
response time.

HTTP support now includes full HTTP 1.1 compliance. Improvements in this
version include persistent connections and virtual hosts. Persistent connections
allow the server to accept multiple requests and to send responses over the
same TCP/IP connection. Virtual hosts allow the use of one IP address to serve
multiple files instead of having different IP addresses for different files.

92 Continuous Availability S/390 Technology Guide

The SSL V3 specification is supported in ICSS for OS/390. In addition to
providing data encryption and server authentication, client authentication and
additional Message Digest Hashing algorithms (SHA) will be provided for
additional security. SSL is used by today′s most popular browsers for secure
transactions. Beginning with this release, Secure Hypertext Transfer Protocol
(S-HTTP) will not be supported by ICSS for OS/390.

Automatic browser detection allows the server to respond to requests with the
version of a Web page or document appropriate for the requesting browser.
Support for Year 2000 operations and content delivery is provided. Page counter
and date/time information can be displayed on a page as graphical images.
Logging and reporting are enhanced.

The previous version of ICSS for OS/390 provided standard reporting such as
number of hits and bytes transferred. The new Java Graphical User Interface
(GUI) version supports URLs, host names, method, or return code reports.
Additionally, associations between these report types can be generated, for
example, URL hits by host, host hits by URL, return codes by URL, and methods
by URL.

The Common Gateway Interface (CGI) support has been expanded to include the
Java programming language (in addition to the other languages such as C,
REXX, and Perl which are already supported).

Platform for Internet Content Selection (PICS) provides a way for webmasters to
provide ratings for the documents they own, allowing Internet users to filter
material based on those ratings.

Client Authentication provides the server with the capability to determine, with a
high degree of confidence, that the client is who it says it is. An example of how
this might be used is a medical clinic where a patient′s files could be viewed
only by physicians containing private patient data.

ICSS for OS/390 will now have a Simple Network Management Protocol (SNMP)
subagent which will maintain ICSS information and performance data in an
SNMP Management Information Base (MIB). Now, from any SNMP-capable
network manager, like NetView/AIX, you can display, monitor, and threshold on
your server performance.

SOCKS support provides the capability to use ICSS for OS/390, when located
behind a firewall, as a proxy server to the destination host via a SOCKS server
on the other side of the firewall. SSL tunneling allows the server to act as a
proxy server for secure transactions. Using a browser, you can observe the
status of the server through the Server Activity Monitor. Examples of the
categories of data you can view are the Thread Counts, Server Request
Statistics, Server Throughput Statistics and Connection Counts.

For more information about IBM′s Internet Connection Secure Server for OS/390,
visit the ICS web site at:
http://www.ics.raleigh.ibm.com/icserver
. You can also visit the S/390 web site at:
http://www.s390.ibm.com/nc/
.

Chapter 5. OS/390 Open Edition 93

 5.1.9.3 OS/390 - The Operating System for e-Business
As the Internet becomes a more integral part of conducting electronic business,
OS/390, the strategic operating system of S/390, is making it easier and safer to
put your business data on the Web for competitive advantage. These functions,
along with the proven strengths of S/390 - high availability, large volumes of
transactions and business databases, scaleable servers, and security - allow you
to build a secure presence on the Internet, extend the value of existing content,
and expand network applications by providing better service and more
capabilities to your customers.

The S/390 server, combined with OS/390, offers an enterprise-wide computing

platform, uniquely capable of meeting today′s most demanding Internet, intranet
and e-business applications. It can help your business to:
• Explore the advantages of establishing a web presence
• Seek a solution that consolidates multiple web servers to reduce site
management costs and improve administrative control
• Offer enterprise data access for read-only or transactional applications
• Establish new electronic commerce channels

S/390 has the technology and products to meet your business requirements.
Some of these capabilities include:
• A network-ready family of servers with direct ATM connectivity
• An integrated server-capable operating system with X/Open XPG4 UNIX
branding
• World-class hardware and software security
• Secure and scaleable Web serving
• Object technology and native Java support

It offers enhanced application interfaces and gateways to connect to existing

applications, data, and other platforms, and is enabled for Lotus Domino
groupware, Firewall, Net.Commerce, and the IBM Network Station. In addition,
S/390 provides continuous computing operations through Parallel Sysplex
clustering technology.

5.1.9.4 S/390 OpenEdition Web Page

You can visit the OpenEdition Web Page at:
www.s390.ibm.com/products/oe/index.html

5.1.10 Windows NT Applications through Bristol Technology

On May 28, 1996, S/390 and Bristol Technology announced a relationship that will
result in the availability of Bristol′s Wind/U development environment on OS/390.

This relationship will enable C and C++ programs written to the Windows NT
APIs (application programming interfaces) to be ported to OS/390. Customers
and independent solution developers will be able to leverage OS/390′s unique
scalable and robust strengths for their Windows NT applications.

Bristol Technology′s Wind/U enables developers of Windows NT applications to

port their applications to UNIX environments, which will include OS/390.
Developers will be able to recompile and link their application source code with

94 Continuous Availability S/390 Technology Guide

the Wind/U Library by using IBM′s development tools on OS/390 to generate a
native version of their application. The Wind/U-ported applications have the
same functionality as the original Windows NT programs. These applications
can be further enhanced by exploiting OS/390 interfaces and middleware.

Windows NT programming interface support on OS/390 will also give IBM

customers who have Windows NT as part of their distributed architecture,
greater flexibility to deploy their applications on the appropriate platform for their
computing needs. For those applications that must scale with availability and
execute in a secure environment, S/390 is an ideal platform choice.

In addition, customers whose Windows NT programs access S/390 transactions

and data will be able to house these applications and data on the same server.
Associated benefits include improved application availability and reduced
network overhead and complexity.

A Joint Study is currently underway with a selected customer. Additional

information will be available at a later time.

Chapter 5. OS/390 Open Edition 95

96 Continuous Availability S/390 Technology Guide
Chapter 6. IBM DASD Subsystems

This chapter discusses several continuous availability solutions for DASD

subsystems. continuous availability features are presented in terms of the
functionality offered by IBM products, and information is provided about specific
products. In addition, different ways to manage data migration with the least
possible outage time are described.

6.1 Summary of DASD Continuous Availability Attributes

Table 5 shows you which availability attribute applies to frequency, duration, and
scope of an outage, and which outage type it belongs to. We distinguish
between planned (continuous operation) and unplanned (high availability)
outages.

Table 5. IBM DASD Continuous Availability Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e Type,

HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Dual Copy X CA
Remote Copy PPRC X X CA
Remote Copy XRC X X X CA
Concurrent Copy X CO
SnapShot Copy X CO
CUIR X CO
RAID 1 X X CA
RAID 5 X HA
RAID 6 X HA
FFST X HA
Virtual Disk Architecture X CO
Concurrent Microcode Update X X X CO
FlexVolumes X CO
128 Logical Channel Path Addressing X CO
SIM Messages X CO
Non-Disruptive DASD Installation X CO
Concurrent Maintenance X X X HA
Remote Service Connectivity X X HA

 Copyright IBM Corp. 1998 97

6.2 Major High Availability/Continuous Operations Functions
In a computing center, the most valuable resource you have is unquestionably
data . In case of unavailability, all other resources, such as processors,
communication controllers, software, and cooling units can be replaced, but your
data is unique.

6.2.1 (RAID) Implementations

6.2.1.1 A Brief History of RAID

The concept of using arrays is not a totally new idea. IBM was granted a patent
for an array in 1977. However, at that time neither the technology nor the
customer requirements supported the approach. In 1987, IBM helped fund a
group of scientist at the University of California at Berkeley who studied the
potential use of arrays. The result was a paper entitled “A Case for Redundant
Arrays of Inexpensive Disks (RAID).”

The Berkeley paper hypothesized that a collection of small inexpensive (of low
$/MB, low capacity, low reliability, and low performance) devices could provide
improved performance relative to high-end devices (of high reliability, high
capacity, and high performance) by aggregating the data rates of the several
devices in an array. Today, “inexpensive” has been dropped from the RAID
acronym. The Berkeley authors changed it to “independent” in acknowledgment
of the need for highly reliable devices to meet the requirements of today′ s
customer environments.

A Redundant Array of Independent Disks (RAID) is any disk subsystem

architecture that combines two or more physical disk storage devices into a
single logical device to achieve data redundancy. RAID types have been
categorized by the Berkeley team into 5 levels: RAID-1 through 5. (RAID-2 and
RAID-4 have limited practical value because they store redundancy information
less efficiently.) In addition to these five redundant array architectures, it has
become popular to refer to a nonredundant array of disk drives as a RAID-0
array. In addition, a new RAID-6 implementation which uses dual redundancy
with floating parity is now commonly used.

By definition, RAID has three attributes:

1. It is a set of disk drives viewed by the user as one or more logical drives.
2. Data is distributed across the set of drives in a defined manner.
3. Redundant capacity or data reconstruction capability is added to recover
data in the event of a disk failure.

Each RAID level explores these attributes in a different way.

For more information, refer to Continuous Availability Systems Design Guide .

IBM has implemented four RAID architectures in various DASD products, as

follows:

98 Continuous Availability S/390 Technology Guide

6.2.1.2 RAID Level 0
RAID-0 is typically defined as a nonredundant group of striped disk drives
without parity. RAID-0 arrays are usually configured with large stripes, but may
be sector-striped with synchronized spindle drives for single-user environments
which access long sequential records. If one drive in a RAID-0 array crashes,
the entire array crashes. This means that in terms of continuous availability,
RAID-0 is not a solution at all. However, RAID-0 arrays deliver the best
performance and data storage efficiency of any array type.

On S/390 architecture, RAID-0 is a combination of software and hardware

functions. The SMS-managed Extended Format Data Set function enables
sequential data striping. To exploit this function, you must have one of the
following hardware control units:
• 9394 RAMAC Array subsystem
The RAMAC Array subsystem allows RAID-5 operation in each drawer. This
provides outstanding fault tolerance to protect against disk drive failure.
• 3990/9390 control unit
To improve data availability in conjunction with data striping, you should
consider the use of the RAMAC Drawer (RAID 5).
• RAMAC Virtual Array
IBM RVA 2 is recognized by SMS as a continuous availability device because
of its RAID-6 characteristics. Extended Format Data Sets can be defined to
enable use of advanced SMS functions like data striping.
• IBM RAMAC Scalable Array Storage 2
The RAMAC Scalable Array 2 is recognized by SMS as a continuous
availability device because of its RAID-5 characteristics. Extended Format
Data Sets can be defined to enable use of advanced SMS functions like data
striping.

6.2.1.3 RAID Level 1

RAID-1 is the simplest form of RAID. It entails using disk mirroring (shadowing)
to duplicate the information stored on a disk. RAID-1 read performance can be
very good. When used in conjunction with an intelligent controller, multiple read
commands can be processed simultaneously by a shadow set. It also is
possible to select the disk whose read/write heads are closest to the desired
data, thereby reducing access time and improving performance.

Conversely, write performance of a RAID-1 system is slightly worse than with a

single disk write operation, because both disks in the shadow set must be
written to for each write operation.

RAID-1 delivers high availability because there is a second copy of the data that
can be accessed if the first copy cannot. The disadvantage of RAID-1 is the cost
of providing two copies of data, thus doubling the required disk space.

IBM has implemented several types of RAID-1:

• S/390 Multiprise 2000
All internal disks of the S/390 Multiprise 2000 are mirrored to protect against
data loss and provide continuous availability.
• Dual Copy

Chapter 6. IBM DASD Subsystems 99

 IBM 3990, 9390, 9394 support Dual Copy operations.
• Remote Copy
Remote Copy allows you to duplicate data between several control units.
IBM has implemented two forms of Remote Copy:
− Peer-to-Peer Remote Copy:
PPRC provides synchronous data copying capability from one 3990-6 to
another 3990-6.
− Extended Remote Copy (XRC)
XRC provides an asynchronous data copying capability with data on the
secondary volume typically maintained a few seconds to a few minutes
behind the primary copy. This function is implemented through a
combination of the 3990 model 006 and DFSMS/MVS software. Updates
to data are copied asynchronously by DFSMS/MVS from a 3990 model
006 at the primary site to DASD at a recovery site. For example, IBM
RVA is supported as a secondary disk of XRC pairs.

6.2.1.4 RAID Level 5

RAID-5 stores data on multiple disk drives, with a portion of each drive used to
maintain parity information for the other drives.

The access arms can move independently of one another. This array type is
intended to support multiple concurrent accesses to the array devices to satisfy
multiple concurrent I/O requests and thus provide high transaction throughput.

Data is striped in record increments across the array of devices. This enables
an access by one actuator to complete a request from the host. Depending on
the striping increment, one full record or block can be accessed by any of the
array device actuators. Thus the arrays can support concurrent transactions or
I/Os, with each actuator capable of satisfying a user request.

Parity is distributed across multiple drives in the array, rather than having a
dedicated parity drive. Each device then contains both data and parity. With
parity distributed, the potential for contention of the parity drive is minimized
when writing multiple blocks. The chances that the parity device for a data
device will be available are much improved because the parity has been
distributed n ways across an array of n devices.

High data availability is provided through the use of parity devices. In the event
that data becomes unavailable from a disk drive in the RAID-5 array, the
subsystem can recreate the missing data through the use of stored parity
information. For data to become unavailable in a RAID-5 system, two drives in
the array must fail.

Independent access, striping, and distributed parity permit a very high bandwidth
for large block transfers, while at the same time providing high I/O rates for
short records.

When writing using RAID-5, the system must update both the data and parity (to
maintain recoverability). The new parity is calculated by the subsystem. To
update parity, the subsystem must perform four I/O operations: two reads (read
old data and old parity so that the new parity can be calculated) and two writes
(new data and new parity). Highly sophisticated destage algorithms limit the
impact of the RAID-5 write penalty.

100 Continuous Availability S/390 Technology Guide

For S/390, IBM has implemented RAID-5 architecture in three products:
• 9392 and 9395 RAMAC Drawers
The RAMAC drawer is an independent component, capable of handling many
of the complex operations required with RAID-5 storage. The RAMAC
drawer is common to both the IBM RAMAC Array DASD and IBM RAMAC
Array Subsystem. The order numbers and feature codes are different, but
the basic hardware is identical.
Each RAMAC drawer has four disk drives and a nonvolatile drawer cache,
and it supports electronics and specialized hardware to manage the
independent functions of the drawer. The drawer manages the CKD-FBA
mapping independently of the subsystem or the host. The drawer handles
all RAID-5 functions, including parity processing and data striping. The host
and the subsystem cache are not involved. The drawer performs the device
emulation function, once again, independent of the subsystem or host.
Each drawer has its own cache used for read caching and nonvolatile write
caching. The drawer manages the cache on a least recently used (LRU)
algorithm. The drawer performs all of its own error recovery, including data
regeneration in the event of HDA failure. The RAMAC drawer uses recovery
techniques that ensure that no single component malfunction results in data
being unavailable.
• IBM RAMAC Scalable Array
The IBM RAMAC Scalable Array combines mirrored electronic cache and
RAID-5 arrays of IBM-manufactured Ultrastar 2XP disk drives. RAMAC
Scalable Array 2 provides reliable access to data.

6.2.1.5 RAID Level 6

RAID-6 was not part of the original Berkeley definitions, but was added later. Tt
is identical to RAID 5 except for the added data protection of two levels of parity
information or dual redundancy.

RAID-6 provides independent access to the disk with floating parity. Multiple
concurrent accesses to the array devices are supported to satisfy multiple
concurrent I/O requests. Data is striped in blocks across the disk drives.
Therefore, single records or tracks can be read from one disk drive without
accessing other disk drives in the array.

Parity is distributed over different disks in the array like in a RAID-5

implementation. Data availability is ensured by provided parity information on
two disk drives. This parity information is used to recover from one or two disk
failures. When writing using RAID-6, the system must update both the data and
parities (to maintain recoverability). To update parity, the subsystem must
perform six I/O operations: Three reads (read old data, old parity 1 and old
parity 2 so that the new parity can be calculated) and three writes (new data,
new parity 1 and new parity 2). With this parity scheme, three device failures
must occur within the same group before data is lost.

Chapter 6. IBM DASD Subsystems 101

 6.2.1.6 Concurrent Backup
As shown in Figure 25, the most important contributor of planned data
unavailability is data backup.

Figure 25. Reasons for Planned Application Data Outages

To provide more continuous availability to your data, you need to find ways to
reduce the impact of backup, while still providing your end users with data
recoverability.

IBM has developed two methods of concurrent backup to help you reach this
goal:
• Concurrent Copy
• SnapShot Copy

Another solution is to use Remote Copy secondary volumes to have a

instantaneous image copy of the primary disk. Remote Copy has not been
designed to do such work, but it could be an interesting solution.

Concurrent Copy: Concurrent Copy creates a backup copy of data while

application processing is occurring against that data. This is accomplished with
the coordination of a 3990-6 or -3 Extended Platform subsystem which maintains
the original data, as needed. Figure 26 on page 103 shows you the main steps
of a Concurrent Copy operation.

102 Continuous Availability S/390 Technology Guide

Figure 26. IBM S/390 Disk Offering

The time between the start of the backup operation and the issuance of the
Logical Complete message is required to prepare an extent table in the 3990
cache. Except for this brief period of time required to initiate the Concurrent
Copy session, Concurrent Copy enables applications to continue running while
data set backups are performed concurrently. The extend table maintains
information about which tracks have been backed up.

In regard to Figure 26, when the application attempts to change a track that has
not yet been backed up (1), the original track is copied into a side file (2) and
sent to data mover in DFSMS/MVS (3). The track is held in dataspace until it is
merged into the correct place in the backup (4).

As soon as the original track is copied to the side file, the application update is
allowed to proceed. Concurrent Copy preserves the point-in-time state logically,
while the physical disk always contains the current state of the file.

If a specific point-in-time backup is mandatory, Concurrent Copy may not be the

right option. If something should cause the backup to fail, a new backup
operation would have to be started and that specific point-in-time image of the
database would be different. No data would have been lost, but the new
database copy would reflect the state of the database at say, 12:47 a.m. instead
of 12 midnight.

DB/2, IMS and CICS have some ability to use the Concurrent Copy function
provided by DFSMS.

SnapShot Copy: SnapShot, a high-speed duplication solution that operates on

the data set and volume levels, provides MVS/ESA users with a means to rapidly
duplicate views to data. SnapShot is an exploitation of the IBM RAMAC Virtual
Array Storage′s virtual disk architecture.

SnapShot works in conjunction with IBM Extended Facilities Product and the IBM
RAMAC Virtual Array Storage.

Chapter 6. IBM DASD Subsystems 103

 It duplicates views to data content within the IBM RAMAC Virtual Array Storage
by creating multiple pointers to one functional track or group of tracks. Unlike
traditional copy methods, SnapShot does not perform any data movement. It is a
virtual data duplicator because it creates multiple views of data. Your host can
see and access the original and the snap copies separately.

From a continuous availability point of view, SnapShot allows you to redesign

your data movement processes to reduce recovery windows, reduce the risks
associated with current methods of testing application changes using a subset of
production data, and increase the time that data is available for production
application access.

SnapShot can duplicate individual data sets, entire volumes, and VM minidisks
within the same RAMAC Virtual Array storage subsystem.

SnapShot supports sequential (PS), partitioned (PO), direct access (DA), and
partitioned extended (PDSE) data sets.

SnapShot has always been capable of snapping VSAM data sets, but previously
was restricted to the volume level and required manual actions to use the VSAM
data sets. It recently has been improved to support VSAM files at the individual
data set level. This new release also fully supports multi-volume data sets,
which previously had the same restrictions as VSAM file, as mentioned earlier.

Copying a Remote Copy′ s secondary volume: Another way to obtain an

instantaneous image of a volume is to use the contents of a remote copy′ s
secondary volume, while your application keeps on working with the primary
volume.

This function can be delivered by both XRC and PPRC, although there are some
differences in both functionality and usability.
Using PPRC
Using PPRC, the main problem is to avoid obtaining fuzzy copies of data,
such as the one that results if you do not freeze your primary applications′
updates.
After being sure that the application stopped the updates, you can suspend
one or more PPRC pair by issuing the CSUSPEND command. You can
suspend in one command all PPRC active pairs attached to a primary control
unit by entering a wrong link path with CESTPATH. This will lead the
primary 3990 to truncate all the communication with the secondary 3990,
suspending at the same instant all the PPRC pairs. From this moment you
can resume your production activities, and all the updates of the suspended
pairs will be kept in a bitmap format in the 3990 nonvolatile storage (NVS).
You have to reset the secondary volume into a simplex state (CRECOVER) to
copy or dump it from another MVS partition.
When the copy processing ends, you can re-estabilish the links and
re-synchronize PPRC pairs. All the updates kept in the primary′s NVS will
be transferred and applied on the secondary volumes, thus re-syncronizing
the PPRC pairs.
Using XRC

104 Continuous Availability S/390 Technology Guide

 The consistency group mechanism. is used by XRC to make sure a usable,
consistent group of secondary images of the primary disks is available at all
times.
All you have to do is to stop the XRC session while your applications are still
running. At the end of the session, the System Data Mover informs you of
the consistency time of the secondary disk images, and you can start a copy
of these disks.
At the end of the copy, you must start the XRC session again.

6.3 Major IBM Disk Products

6.3.1 IBM S/390 DASD Offering

Facing the challenging customer requirements for 24 hours/7 days data
availability, the industry-leading IBM RAMAC Array family provides the widest
range of functions and most reliable technology for high availability, capacity,
disaster recovery, and performance available in the marketplace today; while
meeting the demands of uninterrupted data availability at highest levels of
performance. This product range is shown in Figure 27.

Figure 27. IBM ′ s S/390 DASD Offerings. The broadest range of S/390 Disk Arrays

The RAMAC Array family offers you a broad range of products. Each product
has many continuous availability features.
• RAMAC 2 Subsystem (9394) offers 181GB in a compact frame. It continues to
be an entry-level RAMAC solution for those requiring smaller GB storage
with outstanding availability.
• RAMAC 3 and 3990-6 are examples of IBM protecting your previous
investment dollars. It allows your current 3990-6 to be upgradable, via a only

Chapter 6. IBM DASD Subsystems 105

 a microcode change, to being able to connect to up to 363GB of RAMAC 3
disks.
The 3990-6/RAMAC 2 combination has been proven to have unsurpassed
availability of any disk storage system. The 3990-6/RAMAC 3 combination
carries on that tradition of excellent availability.
• The RAMAC 3 Array Storage continues the exemplary tradition of the
RAMAC family of high performance and the industry′s leading availability
characteristics. If your business requirements dictate high performance and
rock-solid availability, the RAMAC 3 is designed to meet those needs. This
storage solution has up to 726GB of disk storage.
• The RAMAC Virtual Array model 2 is designed for flexibility of configuration
options at an affordable total cost of ownership. The RAMAC Virtual Array
meets the business requirements of high availability, flexible, low-cost S/390
storage. This storage solution has up to 726GB of disk storage.
• RAMAC Scalable Array Storage is designed for high speed disk access and
offers scalability in both disk GB and performance. The RAMAC Scalable
Array fits your requirements for very high throughput. This storage solution
has up to 1,394GB of disk storage.
• RAMAC Electronic Array offers “disk” access at electronic memory speeds.
It is designed for the highest performance data access requirements you
may have. It is also upgradable to the RAMAC Scalable Array. The RAMAC
Electronic Arrays fit the business requirements for raw data access speed.
This storage solution has up to 4GB of “electronic” disk storage.
• IBM S/390 Multiprise 2000 is a CMOS server with an internal disk. It is
designed as a complete MVS server solution in one box. The S/390
Multiprise 2000 fits your business requirements as an entry-level MVS
server, as a MVS server at a disaster recovery site, or where floor space is
at a premium. This storage solution has up to 288GB of disk storage.

6.3.2 RAMAC 3 Array Storage

RAMAC 3 extends the advanced RAID-5 implementation of RAMAC 2 introduced
in October 1995 by using even more highly reliable technology, such as the
latest IBM disk drives and improved logic components. RAMAC′ s
industry-leading data availability (confirmed by consultants like Xephon) is
specifically based on its integrated highly reliable technology (that is, disk drives
and logic cards) and its proven storage control functionality.

Based on the success of the industry-leading RAMAC Array Family and 3990
Model 6 storage control unit, RAMAC 3 Array Storage is an evolutionary step
into the third generation of RAID-5 Disk Arrays. RAMAC 3 incorporates the
experience of more than 11,000 installed RAMAC and RAMAC 2 disk arrays
worldwide. Since the RAMAC and RAMAC 2 disk arrays proved their high
availability design and leading performance, RAMAC 3 and 9390 Storage Control
benefit from this experience and knowledge from thousands of various customer
environments. The maturity of the microcode to properly handle drive failures
and much more, together with complete hardware redundancy and fast, reliable
electronic components gives RAMAC the best data availability in the industry.

The RAMAC 3 Array Storage, as shown in Figure 28 on page 107, consists of

three basic integrated building blocks. The first building block is the highly
reliable IBM 3.5″ disk drive, which utilizes industry-leading technology. It
delivers the highest media data rate in disk technology today.

106 Continuous Availability S/390 Technology Guide

Figure 28. RAMAC 3 Array Storage Building Blocks

The next building block is the drawer; which is an independently-managed

RAID-5 array. Each drawer contains four IBM 3.5″ disk drives. This is the key
element to the RAMAC design, because all 16 drawers within the frame can
work in parallel. This means that, if necessary, all 64 disk drives can run read or
write operations concurrently.

IBM has integrated these RAID-5 drawers into the RAMAC 3 extension frame.
New adapter cards with higher throughput complement the capabilities of the
RAMAC 3 drawers.

The frame can be attached to either the new integrated packaged 9390 control
unit or to installed 3990-6 storage controls. Both solutions deliver great
improvements in performance, capacity and cost of operating, as well as
footprint savings. The new Licensed Internal Code (LIC) running the RAMAC 3
Array contains several enhancements as well, and delivers a RAID-5 storage
system solution of proven robustness and reliability.

6.3.2.1 3990/9390 control unit

The integrated IBM 9390 storage control that comes with RAMAC 3 delivers a
combination of very high performance, high availability, disaster recovery
functions, and high capacity.

The 9390 storage control is available in two models: the 9390-001 and 9390-002.
The name “model 001” stands for a single control unit and the name “002”
stands for a dual controller, each independent of the other. The model 001
controller can be upgraded to an model 002 controller. Model 002 is equivalent
to two 3990-6s. Highly integrated CMOS IV technology has been packaged into
the 9390 controller, significantly improving performance of the 9390 internal
processing as compared to the 3990-6.

Since one of the most important objectives of IBM is to protect your investment,
the broad range of all 3990-6 functions (especially Remote Copy for disaster
recovery) is fully supported by the new 9390 Storage Control. This makes data
migration nearly effortless when integrating the new 9390/RAMAC 3 Disk
Subsystem into your existing installations.

9390 and 3990 both support 3380 and 3390 format. The standard maximum
ESCON distance is 43 km. The 3990/9390 storage control unit is designed to
meet continuous availability requirements. Four storage paths between the
control unit and the disk are available. Two independent storage clusters per
frame provide separate power and service regions. Each control unit supports
128 logical paths to the host.

Chapter 6. IBM DASD Subsystems 107

 3990/9390 incorporate a no-single-point-of-repair design, which means that with
the exception of the Emergency Power Off (EPO) switch and the Power On/Off
switch and their associated cables, no repair action will require that the
3990/9390 be removed from service to effect a repair.

Additionally, 3990/9390 have no components whose failure will cause a total loss
of data availability. The 3990/9390′s fault-tolerant design supports two
independent, point- to-point storage clusters which each contain two storage
path (SP) microprocessors, a support facility (SF) microprocessor, and a disk
drive. This no-single-point-of-failure design continues the evolution toward fully
fault-tolerant DASD subsystems.

3990/9390 incorporate a reliable thermal monitoring system by using fan motion

detection instead of earlier methods of heat sensing. Additionally, the four-fan
subsystems each have a fault-tolerant (n + 1 ) design which allows any one fan in
each of the four sectors to fail with no impact to control unit operation.

A service information message (SIM) is issued in the event of a fan failure, and
the fan can be replaced without impact to the normal operations of the 3990.

Control unit LIC, subsystem status, and diagnostic information are stored on two
hard disks (one per cluster). Two diskette drives (one per cluster) are also
available as a data transport vehicle.

The hard disks contribute to improved control unit availability in two significant
ways:
1. Two different levels of Licensed Internal Code can be stored on the disks,
thus enabling the control unit to switch from one level of LIC to a new level
with minimal disruption.
Because two levels of LIC are stored on the hard disks, the control unit can
rapidly return to the previous level of LIC in the unlikely event that a problem
is discovered with the new level.
2. The space available hard disks enable the control unit to store substantially
more diagnostic information should a hardware or LIC error occur. This
ability supports the control unit′s First Failure Support Technology, and is
designed to enable the vast majority of problems to be diagnosed and
corrected based on information gathered at the time of failure, minimizing
subsystem downtime for problem resolution.

First Failure Support Technology (FFST) is an enhanced approach to availability

and serviceability. FFST is being implemented in multiple IBM product areas
(including processors and operating systems) as part of the increasing emphasis
on availability of both data and functional capabilities to meet the seven day a
week/24 hours per day availability needs of customers. The objective of FFST is
to provide the means to avoid a recurrence of an error by insuring that data is
captured to enable problem diagnosis and resolution based on the first
occurrence of the error.

The control unit implements FFST by incorporating extensive trace hardware

used solely to capture information about the state of the Control Unit during
normal operations as well as during error events. The collected information is
written to the cluster′s internal hard disk drive and is available for use if needed.
This capability enables service personnel to gain access to detailed information
concerning the conditions present in a control unit at the time an event occurred.

108 Continuous Availability S/390 Technology Guide

This greatly enhances the ability to analyze and resolve unexpected error
conditions that were not anticipated in the main line SIM error analysis capability
of the control unit.

SIM messages are used to improve availability and serviceability.

6.3.2.2 Concurrent Copy

Concurrent Copy creates a backup copy of data while application processing is
occurring against that data. This is accomplished with the coordination of a
3990-6 or -3 Extended Platform subsystem, which maintains the original data as
needed.

Refer to “Concurrent Copy” on page 102 for further details about concurrent
copy.

6.3.2.3 Dual Copy

The 3990′s dual copy function can significantly improve data availability. Dual
Copy enables critical data to be stored as two logically identical copies on two
separate IBM 3990/RAMAC DASD drawers. You can use Dual Copy to
dynamically switch from one copy to another in the event of a device failure, or
to keep the data available during system maintenance.

Using dual copy may result in fewer application failures, fewer IPLs, and a
reduced frequency of DASD recovery activity. The implementation of dual copy
is straightforward since normally no application changes are necessary.
Normally, using the dual copy function requires no special attention from the
installation beyond having procedures in place to establish the dual copy pairs
and perform recovery actions when required. DFSMS management classes may
be used to direct the allocation of desired data sets automatically to duplex
pairs.

If you have 3390 DASD, it could be interesting to reserve a “spare” HDA for each
control unit to use concurrently with dual copy in case the microcode detects the
beginning of a production HDA failure. The 3990/3390 subsystem incorporates
extensive self-diagnosis features to detect errors as they occur and analyze the
error information to determine if action is required. A SIM delivers the results of
the subsystem problem determination, providing you to start a dual copy pair.

6.3.2.4 Remote Copy

Typically, the most time-consuming kind of system outage is a defect of magnetic
disk storage devices. It can take many hours to recover the data and bring it
up-to-date. If a reasonably current copy of the critical data can be maintained,
recovery time can be dramatically reduced. The remote copy function of the
3990 model 006 DASD controller enables real-time copies of DASD volumes to be
maintained. Little or no data is lost in the event of a disk failure, and the system
will either continue to function or will simply need to be restarted in order to use
the spare copy of the data.

There are two distinct implementations of remote copy: peer-to-peer remote

copy (PPRC) and extended remote copy (XRC). Both PPRC and XRC enable you
to copy any type of data on a disk volume. PPRC provides maximum currency at
shorter distances, while XRC is geared toward minimal performance impact at
long distances.

Chapter 6. IBM DASD Subsystems 109

 For more information on Remote Copy, refer to Remote Copy Administrators
Guide and Reference .

6.3.2.5 Peer-to-Peer Remote Copy (PPRC)

The configuration for PPRC is illustrated in Figure 29. The primary control units
at your production site are linked through dedicated ESCON links to secondary
control units at your recovery site.

Figure 29. Peer-to-Peer Remote Copy

If ESCON directors are used between the two sites, PPRC links and ESCON
channels can share the same fiber links. Either way, no additional investment is
required on the 3990-6, as PPRC uses the standard ESCON interfaces.

PPRC provides synchronous data copying capability from one 3990 model 006 to
another 3990 model 006. This ensures full DASD data currency in the event of an
outage of the primary copy. Updates are sent from the primary 3990 model 006
directly to the secondary 3990 model 006 through ESCON links between the two
3990 model 006s (the host is not involved in the primary-to-secondary transfer).
The supported distance between DASD and processor, and between control units
for PPRC attachments is up to 43 km (26.7 miles).

To ensure consistency between the two copies of the data, any update is not
considered complete by the application until the data is written to both 3990s.
The unavoidable performance impact of this requirement is mitigated by the
ability of the 3990s to consider a write as complete once the update is in the
controller cache.

6.3.2.6 How PPRC Works

PPRC allows duplication at the volume level. You may attach up to four
recovery site storage controls to each application site storage control, and each
recovery site storage control can be attached to 64 application site storage
controls. In Figure 30 on page 111, two primary 3990 control units are attached
to the same 3990 secondary control units, and four PPRC pairs are defined.

110 Continuous Availability S/390 Technology Guide

Figure 30. PPRC Data Flow

The data flow in PPRC is as follows:

1. The application writes data.
Applications running in the production site write their data on the primary
3990-6, which stores that data in cache and non-volatile storage (NVS).
2. The primary 3990-6 initiates replication of the write operation.
For each write operation directed on a PPRC DASD pair, the primary 3990-6
redrives the write operation towards the secondary 3990-6.
3. The secondary 3990-6 acknowledges receipt.
Once data is stored in its own cache and in NVS, the secondary 3990-6
acknowledges the operation.
4. The primary 3990-6 acknowledges the application.
When the primary receives acknowledgement from the secondary 3990-6, it
signals to the host that the write operation has been completed. At this point
the application can continue processing.

PPRC Dynamic Address Switching (P/DAS): PPRC Dynamic Address Switching

improves the availability of data during migration while solidifying your overall
disaster recovery plan. By enabling non-disruptive data migration between
3990-6 controllers, the P/DAS function can significantly boost the access to its
critical information. As part of a comprehensive high-availability solution within
a System/390 Parallel Sysplex, P/DAS helps move data without interrupting the
system, users, or customers. Besides helping to balance workloads among
processors, P/DAS enables automated data migration to help free up resources
and improve productivity across the entire enterprise.

P/DAS is a software implementation that, in combination with the 3990-6 PPRC

microcode, enables dynamic rerouting of I/O between pairs of DASD volumes
without the need to restart applications using those volumes. P/DAS redirects

Chapter 6. IBM DASD Subsystems 111

 the I/O activity from one volume to another and thus achieves a more efficient
way of performing the address switching with today′s DASD technology.

P/DAS is initiated by operator command, which enables the I/O supervisor (IOS)
to queue application I/O to the primary volume. IOS performs the switching
function, and then another MVS console command is issued to resume I/O to the
secondary volume. During the command execution and switching phases of the
process, IOS queues application I/O, thus avoiding the need to stop and restart
the application.

The main P/DAS requirements are:

• 3990-6 or 9390 with PPRC
• MVS/ESA 5.1 with PTF and above
• DFSMS/MVS 1.2 with PTF or above

6.3.2.7 How a P/DAS Switch Works

Redirection of I/O is performed by the MVS IOS. However, this process is tightly
coupled with PPRC operations, monitoring, and volume pair status. Figure 31
illustrates a configuration where an application system is connected to a 3990-6
at the application site and a 3990-6 at the recovery site. The application site
3990-6 attaches to a PPRC primary volume with device number 100 and VOLSER
VOL001. The recovery site 3990-6 attaches to a PPRC secondary volume with
device number 200 and VOLSER VOL001. The VOLSERs of a PPRC pair are
always identical.

Figure 31. P/DAS and MVS Control Blocks before the Swap

The application system has connectivity to both application site DASD and
recovery site DASD. Four-path connectivity to application site DASD is through
CHP01, CHP02, CHP03, and CHP04, while four-path connectivity to recovery site
DASD is through CHP17, CHP18, CHP19, and CHP20.

112 Continuous Availability S/390 Technology Guide

The application system requires device definitions in order to access both sets of
DASD. That is, devices at the application site and recovery site must be defined
in IOCDS and in an input/output data file (IODF) for each system.

The application site 3990-6 and the recovery site 3990-6 are connected through
four ESCON links that support the logical paths used for PPRC real-time data
shadowing.

Defined in the application′s code is the data control block (DCB), which contains
a description of the characteristics of the data, such as record format, data set
organization, and logical record length. When the application opens a data set,
a data extent block (DEB) is created, and its address is stored in the DCB. The
DEB contains details about the data set, such as access method type, data set
status, and extent information.

To access the data set on a DASD volume, the DEB also contains the address of
the unit control block (UCB), which is the software representation of the device.
The UCBs for all devices are created at IPL time when MVS reads the IODF data
set that is created by the HCD program. The UCBs are stored in the system
queue area of the MVS storage map. They contain details such as subchannel
number, device number, VOLSER, and CHPD array used to help define the
channel paths to access the device. So, the DCB points to the DEB, and the DEB
points to the UCB.

As shown in Figure 31 on page 112, only the primary volume of the PPRC pair is
online, while the secondary volume is normally offline. The primary device
number 100 uses channel paths CHP01, CHP02, CHP03, and CHP04, while the
secondary volume on device number 200 uses CHP17, CHP18, CHP19, and
CHP20.

Figure 32. P/DAS and MVS Control Blocks after the Swap

Chapter 6. IBM DASD Subsystems 113

 The swap process takes place after I/O to the source device (the primary PPRC
volume) has been quiesced momentarily.

The control blocks used by an application during execution of an I/O operation

can be considered as application-owned and system-owned. Application-owned
control blocks reside in the user private area of the MVS storage map, while
system-owned control blocks reside in the system queue area (SQA) of the MVS
storage map.

Redirecting I/O to a device without disrupting application control blocks requires

IOS to swap the physical data in the two UCBs.

While the I/O is queued, IOS swaps the contents of the PPRC primary and
secondary volumes UCBs without interfering with the application control blocks,
which still point to the same UCBs address as before.

Figure 32 on page 113 shows that after the swap, the online device is device
number 200, which uses channel paths defines by CHP17, CHP18, CHP19, and
CHP20.

The application still points to the same UCB address, but now all I/O takes place
to the remote secondary volume as determined by UCB contents.

The entire process is triggered by only three MVS operator console commands,
assuming that PPRC volume pairs have already been established, and that the
primary and secondary volumes are in duplex status. These commands are:
1. IOACTION STOP,DEV=0100
Quiesce I/O to primary device
2. SWAP 0100,0200
Swap primary and secondary UCBs
3. IOACTION RESUME,DEV=0200
Resume I/O to secondary device

6.3.2.8 Extended Remote Copy (XRC)

The configuration for XRC is illustrated in Figure 33 on page 115. XRC provides
an asynchronous data copying capability with data on the secondary volume
typically maintained a few seconds to a few minutes behind the primary copy.

114 Continuous Availability S/390 Technology Guide

Figure 33. Extended Remote Copy

This function is implemented through a combination of the 3990 model 006 and
DFSMS/MVS software. Updates to data are copied asynchronously by
DFSMS/MVS from a 3990 model 006 at the primary site to DASD at a recovery
site.

The DFSMS/MVS system can be located at the same site, or in a separate

system located elsewhere, but it must be connected to both the primary and
secondary DASD controllers.

Primary and secondary DASD controllers can be attached to the DFSMS/MVS

host up to 43 km (26.7 miles) through ESCON links. Parallel channel attachment
is also supported with XRC, but considerably limits the performance and
distance.

Distance limitations are virtually eliminated in XRC environments that utilize

channel extenders such as the CHANNELink systems offered by Computer
Network Technology (CNT) Corporation. This capability is also useful in cases
where an ESCON service is not publicly available.

The performance impact of XRC is typically greatly reduced as compared to

PPRC in the same environments, due to the asynchronous design of XRC.

6.3.2.9 How XRC Works

XRC is provided as a function of the IBM 3990-6 control unit and DFSMS/MVS on
the host. DFSMS/MVS includes a System Data Mover (SDM) component that is
the heart of XRC. When a host application updates data on the local volumes on
the IBM 3990, a copy of that data is kept in the cache. The Data Mover receives
copies of updated data from all control units that have active XRC sessions and
transmits updated data to an MVS system at the recovery site. This system then
maintains the corresponding shadow volumes.

Chapter 6. IBM DASD Subsystems 115

 Since it is a fully asynchronous solution, where the second process runs
independently of the first, XRC has much less impact than PPRC over the host
application performance. When traffic is heavy and write operations cluster (as
they frequently do in real-world situations), XRC avoids additional bottlenecks,
and causes little or no degradation of host application performance.

Figure 34. XRC Data Flow

The data flow in XRC, shown in Figure 34 is as follows:

1. The application writes data.
An application running in the production site writes data on a primary 3990-6.
2. The primary 3990-6 completes the I/O.
The primary 3990-6 keeps the application data update in cache and in NVS
and acknowledges the write operation as complete, in order to allow the
application to continue with its processing.
3. SDM reads all updates for XRC DASD pairs from the 3990-6 cache.
Periodically, or upon request of a primary 3990-6, the System Data Mover
retrieves all updates from the primary 3990-6 cache.
4. SDM builds a consistency group and stores it in its control data sets.
Once it has read all updates from all primary DASD volumes defined in the
XRC Session, SDM builds a consistency group.
A consistency group is built to guarantee that the secondary 3990
subsystems will be updated in the same sequence as the primary ones.
SDM uses the timestamp associated with each write I/O to maintain the right
I/O sequence inside the remote copy session, which implies that all
processors sharing primary DASD that is to be remote-copied must
synchronize their clocks. This can be achieved by means of a Sysplex
Timer.

116 Continuous Availability S/390 Technology Guide

 Once the consistency group has been formed, the System Data Mover stores
it in its journal data set, to minimize the in-flight data kept in memory only.
5. SDM replicates the updates on secondary subsystems.
Once the consistency group is secured in the journal, the System Data
Mover replicates the updates it has in memory on the secondary
subsystems, in exactly the same sequence as they happened on the primary.

6.3.2.10 RAMAC 3 Drawer

The RAMAC 3 drawer, as shown in Figure 35, is designed for outstanding data
availability.

Figure 35. RAMAC 3 Drawer

The drawer offers third generation RAID-5 protection and the most reliable
microcode for its disk drives. The other hardware components, including the
microprocessors, are fully redundant and were developed to deliver 100% fault
tolerance. All electronic components are fully battery-backed up, and the battery
itself is checked by the control unit every 24 hours. If required, components can
be exchanged concurrently during normal system operation. The drawer can
perform data reconstruction on a failed disk drive when dynamic sparing has not
been performed.

The RAMAC 3 drawer incorporates the new IBM Ultrastar 2XP 3.5″ 9.1GB
high-performance disk drives which double the capacity of the previous disk
drives.

Each RAMAC 3 drawer contains eight 3390-3 or 3380-K logical volumes for a total
of 22.7GB of storage. The previous RAMAC 2 drawer offered a data capacity of
11.35GB for four logical volumes.

The RAMAC 3 drawer offers improved performance as compared to the RAMAC

2, based on a faster interface to storage control, up to 15.4MB/sec disk media
data rate, increased SCSI transfer rate and improved sequential processing for
write updates. Data transfer between the drawer and storage control is three
times faster than RAMAC or RAMAC 2. This improvement is enabled by the new
High Speed Device Adapter in the RAMAC 3 frame, and by microcode changes.

RAMAC 3 Drawer Maintenance: A key requirement of storage subsystems today

is continuous operations. To address this requirement, the RAMAC Array DASD
subsystem introduced the concept of concurrent drawer maintenance with the
sparing function. RAMAC 2 Array DASD enhanced the concurrent drawer

Chapter 6. IBM DASD Subsystems 117

 maintenance philosophy further and broadened the maintenance options by
introducing dynamic disk reconstruction.

RAMAC 3 Array Storage and the 3990-6 attaching to the RAMAC 3 drawer retain
this strong focus on concurrent drawer maintenance by supporting both sparing
and dynamic disk reconstruction.
• RAMAC 3 sparing
There are two sparing options: dynamic sparing and manual sparing.
Dynamic sparing has the advantage of initiating automatically on fault
detection, thus providing maximum data loss protection by copying the data
to the spare drawer and restoring RAID-5 redundancy as soon as possible.
Manual sparing offers you the flexibility of invoking sparing for specific
drawers, and provides the choice of not reserving a spare drawer, but
instead, of quiescing a drawer when appropriate and initiating the sparing
process with the help of a service representative. Sparing with a RAMAC 3
configuration is similar to the process we are familiar with on RAMAC and
RAMAC 2 subsystems. The default for RAMAC, RAMAC 2, and RAMAC 3 is
to automatically start the sparing process, assuming that a spare has been
defined to the subsystem.
During the dynamic sparing process, SIMs are generated and sent to the
host so that the operations staff is aware of ongoing status. Up to two
drawers can be defined as spare drawers, but only one can be the target of
the sparing operation at any time.
Sparing is the only drawer maintenance function that protects against all
drawer failures. Dynamic disk reconstruction should be used only for disk
drive module (DDM) failures.
We recommend one spare drawer per RAMAC 3 storage frame.
• Dynamic disk reconstruction
Dynamic disk reconstruction offers an alternative method of concurrent DDM
maintenance. It applies to the DDM component of the drawer only and
enables disks to be replaced concurrently with drawer I/O activity. While the
DDM is being replaced, the enhanced RAID-5 function regenerates the data
for the failed disk. After the DDM has been replaced, the data is
reconstructed and written to the new disk.
Dynamic disk reconstruction does not offer the global “all-component” repair
philosophy of dynamic sparing because the technique applies only to DDM
components, but it does enable installations to consider a “no-spare” policy
for DDM repair scenarios. Dynamic disk reconstruction also avoids the time
taken to copy data to the spare drawer, but remember that dynamic sparing
paces the copy to minimize the impact on application performance.

6.3.2.11 RAMAC 3 Array: Microcode Upgrade

Concurrent LIC download and activation, as shown in Figure 36 on page 119,
help to achieve continuous operations by enabling the 3990-6 LIC to be upgraded
in support of storage subsystem maintenance without stopping and restarting
applications.

118 Continuous Availability S/390 Technology Guide

Figure 36. Concurrent LIC Download and Activation

Depending on the microcode level, you should perform all or some of the
following tasks:
1. 3990-6 LIC download
The 3990-6 enables LIC download to take place independently of LIC
activation. Thus an installation has more flexibility in planning the cut-over
to a new level of 3990-6 LIC by “readying” the subsystem in advance. The
3990-6′s fault-tolerant Each disk drive (one per cluster) can hold two levels of
3990-6 LIC (the current level being executed, and a second level, ready to be
activated). When 3990-6 LIC is downloaded, one copy is stored on each disk
in each cluster.
The 3990-6 LIC can be downloaded concurrently with I/O activity. During the
download operation, SPs, cache, NVS, and all other components continue
operating with no impact on application performance.
The SF performs download independently of SP application I/O processing.
The SF reads the LIC from the diskette and writes it to the hard disk in the
cluster.
Note: The download operation must be executed for each cluster.
2. 3990-6 LIC activation
For concurrent LIC activation, the activation process must be performed one
cluster at a time. As the initial step, the 3990-6 LIC activation process
invokes the control unit-initiated reconfiguration (CUIR) function, if enabled,
to request that all host channel path connections to this storage cluster be
varied offline by the host system. CUIR is a function of the 3990-6 only when
configured in extended mode. MVS/ESA, VM/ESA, and VSE/ESA support the
CUIR process.

Chapter 6. IBM DASD Subsystems 119

 The activation process is done by one SP and one SF. During activation, the
paths to the cluster are varied offline by the host systems in response to the
CUIR function, and the two SPs of the other storage cluster remain active.
Most of the new LIC can be activated concurrently with subsystem cache and
NVS activated. The 3990-6 extended functions that are affected by cache
status include DFW, dual copy, XRC, and PPRC, and these functions can now
continue using cache and NVS during the LIC activation process in most of
the cases.
3. Device LIC download
The download of the device LIC is performed by the RAMAC maintenance
panel code and the 3990-6 SF. The device LIC is written to the RAMAC
drawer. During the device level LIC download process, device caching is
disabled although the 3990-6 subsystem cache is still activated. Other
drawers in the rack are unaffected by the download to this drawer. Because
device level caching is disabled, any volume in the drawer that is part of a
dual copy or PPRC pair must be suspended. By suspending the pair in
advance of device microcode download, you can enable a fast
resynchronization of the volumes through change-recording after the code
activation.
Change-recording is enabled because 3990-6 NVS is still active and a record
of dual copy and PPRC primary volume updated cylinders is maintained. All
XRC pairs must be terminated before device LIC download. Termination is
required because device caching is disabled and the 3990-6 uses cache to
manage the record groups that are eventually transferred to SDM so that
consistency groups can be journaled at the recovery site. Dynamic sparing
must also be disabled before device microcode is downloaded, and all
outstanding SIMs must be closed.
4. Device LIC activation
During device LIC activation, the conditions mentioned for device LIC
download also apply. During device LIC activation, I/O to the device results
in the 3990-6 returning a state change pending (long busy) signal. This long
busy may last for approximately 45 seconds. Before you activate a new level
of device LIC, the condition of the storage subsystem must be well
understood in terms of 3990-6 extended functions that are active.

6.3.3 RAMAC Subsystem 9394

The RAMAC 2 Array subsystem is a RAID-5 storage subsystem with dynamic
sparing, dynamic disk reconstruction and predictive failure analysis, thereby
providing an exceptionally high degree of subsystem fault tolerance. The
subsystem also increases data availability through the use of dual line cords and
redundant power, fans, logic and pathing throughout the subsystem. Host
connectivity options include four or eight parallel channels, four or eight ESCON
channels, and mixed parallel/ESCON channel capability. ESCON configurations
include 128 logical channel path addressing capability, and the ability to operate
at distances of up to 43 km (26.7 miles). Overall, the RAMAC 2 Array subsystem
offers extensive flexibility, as follows:
• 11.35-180GB of data storage in one rack (two B13 drawers minimum).
• Controller cache sizes ranging from 64MB to 2GB.

120 Continuous Availability S/390 Technology Guide

 • 3380-K or 3390-3 DASD emulation modes (3380-K is available with B13
drawers), which can be intermixed in a model 003 rack; drawers can be
reformatted.
• Software transparency which is supported in MVS/ESA, MVS/XA, MVS/370,
VM/ESA, VM/XA SP, VM/SP HPO, VM/SP, VSE/ESA), and VSE/SP
environments.
• Parallel (3.0 and 4.5MB/Sec) and 128 logical path ESCON (10 and 18MB/sec)
channel support.
• A high degree of modularity and upgradeability.

6.3.4 RAMAC Virtual Array 2

RAMAC Virtual Array, a storage subsystem that utilizes an advanced virtual
storage architecture, could help you to reduce your scheduled outage due to
tasks such as backup, volume reorganization, data management or I/O tuning.

The RAMAC Virtual Array offers an integrated storage control and disk array
storage capable of using both Parallel and/or ESCON channels. With 128 ESCON
logical paths capability, the RVA 2 can be fully exploited in a Parallel Sysplex
environment.

Up to four 3990 model 3 storage controllers can be defined, each with up to 64

3390 and/or 3380 volumes, for a total of 256 devices. RVA permits you to
dynamically define and manage 3380 and 3390 volumes to exactly fit your space
requirement.

The RVA 2 models support a minimum of 160GB and a maximum of 726GB of

storage capacity. Storage capacity can be added in increments of 80GB, 130GB
(only available on installed 80GB arrays), and 210GB.

DFSMS/MVS provides Systems Managed Storage (SMS) recognition of the RVA 2

as a member of the RAMAC family of devices. DFSMS/MVS recognizes the RVA
2-as a continuous availability device due to its RAID characteristics. Extended
Format Data Sets (EFDS) are supported.

Figure 37 on page 122 shows you a high level overview of the internal structure
of the RVA T82.

Chapter 6. IBM DASD Subsystems 121

 Figure 37. RAMAC Virtual Array 2 Turbo

On the left is the control function of the subsystem. Eight microprocessors

execute microcoded instructions and control the internal functions of the RVA.

Shared memory contains status information for the functional and physical
devices in the RVA subsystem. This status information is maintained by the
microprocessors. It is used to keep track of I/O operations in progress and to
manage the subsystem. Shared memory also provides a means of
communication between the microprocessors. Shared memory is duplicated for
redundancy.

To the right are the data handling functions of the RVA subsystem. The RVA can
provide host connection through either parallel or ESCON channels. The design
of the RVA allows for up to 32 parallel channels, or up to 16 ESCON channels.

The RVA′s log structured file architecture enables the use of data compression,
which facilitates the most efficient use of subsystem cache, nonvolatile storage
(NVS), internal data paths, and array disk space. Write data arriving over host
channels is compressed before it is stored in subsystem cache and NVS. Read
data coming from cache is decompressed before it is sent to a host channel.

The RVA cache can have a physical capacity from 512MB to 1536MB, increasing
in 128MB increments. Because of the RVA data compression, the effective
capacity of this cache is greatly increased when compared with a traditional
DASD subsystem with similar physical cache size. IBM conservatively rates the
effective cache capacity of the RVA family at twice the physical capacity, to
provide a meaningful comparison with traditional DASD subsystems. The RVA 2
and the RVA 2 Turbo can have an effective cache capacity from 1024MB to
3072MB, increasing in 256MB increments. All internal data transfers in the RVA
go through cache. All write operations are handled as DASD fast writes. Front
end (channel transfer) operations occur independently of back end (device
transfer) operations.

The RVA subsystem contains an NVS of 8MB physical capacity. All write data is
copied to NVS, in addition to being written to cache. Thus two copies of write

122 Continuous Availability S/390 Technology Guide

data are always maintained in the RVA until the data is destaged to the disk
arrays. The effective capacity of NVS is 16MB. As for cache, the effective NVS
capacity is rated at twice the physical capacity. NVS is powered by a pair of
batteries that enable any undestaged write data in NVS to be retained for at
least 72 hours in the event of a power failure.

RVA provides four or eight internal data paths between the channels and cache.
There is one compression engine for each of the data paths.

When destaging write data from cache, the RVA appends dual redundant parity
before writing the data and parity to the disk arrays. Generation of the parity is
done by hardware. The parity enables the regeneration of data when one disk,
or even two disks in the same array, fails.

The RVA provides 16 separate data paths from cache to the disk arrays. Of
these, 14 are used to transfer read and write data. This arrangement makes
possible a high aggregate data transfer rate between the cache and the disk
arrays. Because the data is compressed, the effective transfer rate is
approximately 3.6 times the physical data transfer rate.

6.3.4.1 RVA 2 Disk Arrays

RVA uses IBM Ultrastar 2XP 4.5GB, 3.5-inch disk technology.

Disks can be defined as an eight-disk array group with 80GB capacity (5 data, 2
parity and a spare). This definition supports a minimum RVA 2 two-disk array
configuration of 160GB.

RVA disks can also be defined as a 16-disk array group with 210GB capacity (13
data, 2 parity and a spare). This definition supports a larger two array
configuration of 290GB (one 80GB array and one 210GB array). Capacity
increments of 80GB or 210GB support capacities up to the maximum
addressable 726GB (256 3390 model 3).

Flexible RAID-6 architecture provides high availability with:

• Dual parity disk protection which provides redundancy protection even with a
single disk failure. It continues to operate with two disk failures in a single
redundancy group. No other S/390 disk array offers this level of protection.
• Two hot global spare disks are included to automatically replace any failed
disk in any array.

Since dual parity is provided, the rebuild is done in the background during
periods of low activity.

6.3.4.2 Virtual Disk Architecture

Virtual disk architecture, as shown in Figure 38 on page 124, is an exclusive
feature of the IBM RAMAC Virtual Array.

Chapter 6. IBM DASD Subsystems 123

 Figure 38. Virtual Disk Architecture. Operational Flexibility

Data written to disks is compressed and compacted using full track blocking
without gaps. Since block lengths will change when updates occur, Log
Structured Array (LSA) architecture enables compression and compaction
because updates are never done in place.

The benefit is a reduction in the storage management workload because the

RVA 2 is self-tuning. The logic in the RVA selects the best volume where it
writes the updated data. This allows your staff to engage in more productive
activities.

Best volume utilization is achieved because the RVA does not update data in
place. This is a radical departure from other disk array architectures and
enables usage of unique and powerful storage subsystem functions. Unallocated
space does not use physical disk array space. Allocated but unused space does
not use physical disk array space either. This capability allows you to
over-allocate data sets in order to prevent out-of-space abends, allowing your
batch production to run more efficiently.

No RAID-5 or RAID-6 write penalty is incurred with this architecture because the
updated data and dual parity is always written to a new location. Dual
redundancy is not feasible without LSA because of the prohibitive write
performance penalty. As data is updated, it is moved within the subsystem and
placed on the disks with the least activity, thereby balancing the load of the disk.
The RVA is thus a self-tuning subsystem.

On-demand volume management is a unique feature of virtual disk storage

which allows you to create or delete logical volumes dynamically. Unlike with
fixed mapped disk arrays, no vendor service interruption is required, so online
processing can continue. The ability to easily change between 3380 and 3390
disk formats at the individual volume level greatly simplifies data migration from
older technology. User control is provided with an easy-to-use IBM Extended
Facility Product (IXFP) interface so your operations staff can manage this
process. This is one of the features that makes IXFP a highly recommended part
of the RVA 2 solution.

124 Continuous Availability S/390 Technology Guide

The subsystem uses a table with pointers. Pointers always point to the updated
version of the data. Manipulation of these pointers enables SnapShot. SnapShot
is a way of duplicating logical views of data within seconds instead of minutes or
hours using virtually no additional disk space, CPU resources, or channel
resources.

6.3.4.3 Log-Structured Array Management

As records in an LSA are updated and written to new locations, unused spaces
develop which need to be reclaimed to maintain sufficient free space for
additional write activity. The process is similar to defragmentation of disk
volumes where used space is combined together at the beginning of the disk
and unused space is combined into larger, usable amounts at the end of the
disk. This process is called free space collection .

The IXFP Deleted Data Space Release (DDSR) function makes sure that RVA
knows about potential free space which is created when MVS deletes data sets
or VM deletes minidisks. Data set pointers are removed when data sets or
minidisks are deleted. This process can operate either as dynamic DDSR when
the deletion occurs, or as interval DDSR on a scheduled basis.

The interval DDSR process operates on a volume-by-volume basis. During this

process, VTOCs are scanned and deleted extents that are still mapped within the
RVA are then deleted. Interval DDSR is scheduled during a weekly maintenance
period when the performance impact of reading all of VTOCs can be minimized.

Automatic free space collection operates with self-managed algorithms and is a

threshold-driven process. The status of potential free space is maintained on a
physical cylinder basis (called an array cylinder ). Free space thresholds are
monitored to indicate which cylinders are best candidates for free space
collection processing.

Free space collection is automatically initiated and runs without operator

intervention. It operates as a background process during periods of low activity.
Free space collection selects array cylinders with most free space available for
recovery. Valid virtual tracks are copied into cache, collected into new array
cylinders in cache, and written to disk array as a separate process to minimize
performance impact on the normal write process.

When the valid tracks have been properly relocated, the array cylinders are
added to a collected free space pool. This self-management is transparent to
the users.

6.3.4.4 Advantages of Log Structured Array

The unique LSA architecture offers many exclusive features not possible with
traditional fix mapped disk arrays.

With traditional DASD architecture, data compression and compaction is not

possible since block length might change while update-in- place occurs. As LSA
does not allow updating in place, data compression is made possible and thus
leads to space savings. Also many unique features can be delivered, such as
SnapShot.

Traditional RAID-6 architecture requires six separate I/O operations to update a

single record. Since the dual parity tracks are calculated after the data tracks

Chapter 6. IBM DASD Subsystems 125

 have been collected in cache and the data and parity written in parallel to the
disk arrays, the traditional RAID-6 write penalty is eliminated with LSA.

Automatic load balancing is achieved by spreading the data across all of the
disks in the virtual disk system. This eliminates “hot spots” or disk access skew
which occurs in traditional disk array systems.

Striped writes improve back-end performance. Parallel disk writes are

scheduled when full cylinders of data and parity have been collected. Since full
track blocking is used, only one write (rather than multiple writes) is required.
The combination of parallel writes and full track writes greatly improves the
back-end write performance.

IBM RAMAC SnapShot for MVS and VM, or simply SnapShot, is a duplication
solution only possible with LSA disk array systems. As we discuss later, this
unique, advanced duplication product offers the potential to revolutionize the
batch processing operation.

In traditional disk space management, data sets as well as CMS minidisk

allocation is over-allocated in order to avoid out-of-space abends. This
over-allocation results in allocated and unused data occupying real disk
capacity. With LSA, these datasets can be over-allocated without space penalty.
One of the LSA characteristics is that only written data is stored. RVA 2 space
allocation does not reserve any unallocated or unwritten disk space.

Minimizing out-of-space abends by over-allocation of space without concern for

wasted space is a clear benefit. Since RVA only commits array space when data
is written, over-allocation does not waste space. Data is spread across more
volumes, increasing free space per volume.

The total amount of unused RVA disk free space can provide emergency
capacity for your peak workloads. This feature of the RVA LSA architecture is
unique and cannot be exploited by traditional disk.

Simplification of storage management is a major benefit for storage

management staff.

6.3.4.5 High Data Availability

Disks: IBM Ultrastar 2XP disks use Predictive Failure Analysis (PFA) which
ensures a high level of data availability. PFA monitors key device performance
indicators, including head-to-disk fly height, and reports significant changes and
impending failures when predetermined thresholds are exceeded.

Disk Failures Protection: RAID-6 with dual redundancy is an RVA exclusive.

With RAID-6 an RVA provides dual redundancy within an array unlike any other
traditional disk array subsystem. RAID-6 allows two physical drive failures
within a single array without exposing the customer to data loss.

Hot global disk sparing is another RVA exclusive. RVA Hot Global Disk Sparing
is truly global with a minimal performance impact because of the dual
redundancy design. The global disk sparing allows spare drives be shared
among different disk arrays.

Redundant pointer table protection: Logical disk volume tracks are dynamically
mapped with pointer tables to the physical disk locations. Over time, all logical
volume tracks will be spread across all of the physical disks in the virtual array

126 Continuous Availability S/390 Technology Guide

storage system. Protection of the tables is a major issue to be addressed with
any potential user.

Pointer tables are located in separate electronic base memory and do not reside
in user cache. Since they are stored electronically, they are exposed to loss in
case of a power failure. They are continuously backed up to two separate disk
arrays, each with dual redundancy.

Because there is a delay in updating the disk copies, changes are journaled in
nonvolatile storage (NVS) to enable recovery from a power outage.

In a worst-case scenario where control unit electronics are destroyed, the

physical disk arrays could be relocated to a different control unit and the data
restored. Each disk array contains self-defining structure information so that the
pointer tables may be rebuilt. No known customer event has required this level
of recovery.

This multi-level protection of critical pointer tables is designed to enable data

recovery under even the most difficult situations imaginable.

Concurrent changes: For RVA 2 machines, a new feature called Non-Disruptive

Code Load (NDCL) is introduced. This feature is an enabler for future microcode
releases that will be activated concurrently.

Concurrent LIC activation is enabled by the new microcode level. A new

capability enables follow-on LIC to be activated in the RVA 2 without taking any
subsystem components offline, which improves overall system availability.

Note: The one-time installation of the enablement capability is disruptive. This

capability is being made available for all models of the RVA, demonstrating
customer investment protection.

Hardware maintenance is designed not to impact data availability or

performance. Disk drive replacement, logical and memory cards are
hot-pluggable.

Adding additional arrays independently of existing arrays is concurrent.

6.3.4.6 SnapShot Copy

SnapShot is a priced-optional high speed data duplication function that uses Log
Structured File Architecture. SnapShot addresses the costs associated with
traditional data movement. It dramatically reduces the time, CPU costs, channel
utilization, and storage space expenses normally required for the data
duplication process.

While the initial release of SnapShot enabled the duplication of VSAM data sets,
it implied some catalog manipulation, which made it difficult to use. This
difficulty was eliminated by the new release announced on April 8, 1997.

SnapShot operates on the data set, minidisk, and volume levels, and provides
MVS/ESA and VM/ESA users with a means to rapidly duplicate views to their
data.

SnapShot duplication occurs in seconds, not in the minutes or hours required by

traditional physical data volume or data set copy programs such as DFDSS
(which is shown in Figure 39 on page 128). This almost-instantaneous process

Chapter 6. IBM DASD Subsystems 127

 creates a true point-in-time backup copy for restart of batch processes or
disaster recovery.

Figure 39. SnapShot Copy: Impact on Batch

No physical movement of data is required because only pointer tables. and not
the data, are actually copied. The original volume or data set is mapped to the
physical disks with a set of pointers. A volume snap (or data set snap) is
performed by duplicating the pointers (not by physically duplicating the data),
and then both pointers point to the same physical data.

Easy creation of test data without duplicated space means that your current test
processes which include dedicated disk arrays, host channels and processing
resources may now be reassigned to standard application processing. As an
additional benefit, the availability of complete, current test data may improve the
overall testing process for both Year 2000 testing and application testing which
may bring new applications online faster with fewer quality issues.

Since SnapShot is so quick, it simplifies file transfer for data warehousing

applications. A physical duplicate copy of the extract data files no longer need
to be kept on the mainframe while data is transferred to the data warehouse.

SnapShot permits the deferral of physical copies to off-peak periods. This has a
direct cost savings by reducing the tape processing resource requirement.
Additionally, by shifting peak sequential processing to non-peak periods, the four
or eight paths host channel constraint can be greatly reduced and additional
capacity added to RVA without compromising performance, thus reducing the
overall RVA system cost.

Only SnapShot can duplicate data at electronic speeds, thus creating major
competitive advantages for customers exploiting its revolutionary possibilities.

6.3.5 RAMAC Scalable Array

RAMAC Scalable Array 2 further extends the RAMAC Array with new models of
the most highly scalable RAMAC product. The RAMAC Scalable Array 2
presents choices in balanced scalability, letting you specify the performance and
capacity of the subsystem to address current and future needs while maintaining
high availability.

128 Continuous Availability S/390 Technology Guide

Figure 40 on page 129 shows you a high-level overview of the internal structure
of the RVA 2.

Figure 40. RSA-2: Control Unit Design. Scalable performance and capacity

Using an innovative, multiple bus architecture that permits very high scalability
of capacity and performance, the RAMAC Scalable Array 2 focuses on delivering
high-to-extremely high levels of performance in a disk storage subsystem.

The RSA-2 emulates 3380 J and K volumes as well as 3390 models 1, 2, 3 and 9.
You can also define FlexVolumes, which are allocated with a minimum of five
cylinders and can be increased in nine-cylinder increments.

6.3.5.1 RSA-2: High Data Availability

The design of the RAMAC Scalable Array 2 is based on the premise that the
system must be fully fault-tolerant to ensure no data loss and no loss of access
to data regardless of component failure. For a fully configured system, the
performance impact of any component failure should be small.

Memory card failure will not affect performance because of the 100% mirrored
electronic memory. A channel adapter failure would cause the load from this
channel to be spread across other resources. The net result is a performance
system that is fully fault-tolerant and sets new standards in the industry.

The RAID-5 architecture uses nine data disks plus a parity disk for a 9+1 parity
system. The use of a very large cache allows the RAID-5 write operations to be
done asynchronously and thus eliminates any RAID-5 write penalty seen by the
host. The RSA 2 has combined the advanced RAID-5 disk array technology and
the IBM Ultrastar 2XP 9.1GB Disks.

RAID-5 function provides protection against a disk failure. The RAMAC Scalable
Array 2 also has a sophisticated approach to achieving high availability by
factoring other potential failures.

The RAMAC Scalable Array 2 uses a true dual port architecture that ensures full
data access in case of a device controller or SCSI bus failure. Full function dual
port is not offered by any SCSI disk drive manufacturer, so the RAMAC Scalable
Array 2 uses a custom-designed card that provides two SCSI ports to the single
port SCSI drives.

Chapter 6. IBM DASD Subsystems 129

 The RAMAC Scalable Array 2 uses duplexed logic, channels, and internal paths
and even has duplexed backplanes! This duplexing of hardware allows for
hot-plugging of logic cards, cache cards and disks. It also assists in enabling
concurrent maintenance and servicing.

All power functions are duplexed (two power cords, separate AC/DC converters,
separate battery systems, and separate voltage converters). Apart from the AC
input control, all power system components are hot-pluggable. If an AC input
control should fail, the other AC input control will take over.

The RAMAC Scalable Array 2 keeps two complete copies of all data in separate
cache cards that are serviced by separate memory controller cards, each
powered by separate power supplies and separate battery systems.

All reads occur from both sides of the mirrored cache and the data is compared
bit for bit, giving essentially an infinite bit error detection. If a transient
uncorrectable error does occur during reading from the mirrored cache, there
will be a miscompare at the bit level and the failing side will report an ECC
uncorrectable error. The controller can then tell which data is good and satisfy
the read request with good data. The controller will then overwrite the bad data
with the correct data from the other cache card.

If a cache card should fail, the system will be able to continue at full
performance. The error will be recognized by the controller and the controller
will phone home to report the error. The system will switch off mirrored mode
and just read and write to the good card. After the defective card is replaced,
the controller switches to recovery mode. All reads will go to the good side and
all writes will go to both sides. As a background operation, the system will read
from the good side and copy the data to the replaced memory card. This
recovery operation continues until fully mirrored data resides in both memory
cards and the system switches to full mirrored operation.

The device controllers (DC) are responsible for all media management tasks.
The DC performs the disk-scrubbing and error detection and correction for the
disk device. If a media error occurs, the DC maps around the defective media,
assigns an alternate sector or track, rewrites the data, verifies the write and
increments the soft error counter. When the soft error count exceeds the
threshold, the support controller is notified and a SIM is generated.

The support facility is fully redundant and performs such functions as

configuration changes, initial licensed internal code load, error analysis, logging
and reporting. The support facility provides for high data availability by
continuously monitoring and reporting the status of all key components.

The RAMAC Scalable Array 2 has a complete approach to achieving 7 x 24 x 365

availability and offers uncompromising fault-tolerance.

6.3.6 RAMAC Electronic Array

The REA 2 achieves its ultra-high performance and availability due to two key
factors. First, it emulates disks in solid state memory using no real hard drives
(just logical drives). Unlike a cached disk system that will not achieve 100% hits
for all reads and writes, the REA 2 has no caching algorithms and all reads and
writes occur to fast dynamic random access memory chips (DRAM). This 100%
hit for all reads and writes provides consistent fast response time and is similar
in principal to solid state devices offered by other vendors. Second, the REA can

130 Continuous Availability S/390 Technology Guide

 transmit data concurrently over 16 paths at a full 18 MB/second. The REA
provides support for 512 logical volumes.

But the REA 2 also offers new levels of availability by duplexing hardware
components including the solid state memory. In the event of a power failure
batteries take over, providing 48 hours of power for the largest 4GB duplexed
system and greater than 48 hours for smaller memory configurations.

The basic architecture of the REA 2 is the same as the architecture of the IBM
RAMAC Scalable Array 2. The REA 2 is essentially an RSA 2 without the
attached disk frame and a different Licensed Internal Code (LIC)load. Your
investment in an REA 2 provides you with the ability to upgrade to an RSA 2
simply by adding a disk frame and changing the LIC. This provides superior
investment protection.

6.3.6.1 REA-2: High Data Availability

The design of the REA 2 is based upon two complete copies of all data being
kept in separate memory cards that are serviced by separate memory controller
cards, each powered by separate power supplies and separate battery systems.

All reads occur from both sides of the mirrored memory and the data is
compared bit for bit giving essentially an infinite bit error detection. If a
transient uncorrectable error does occur during reading from the mirrored
memory, there will be a miscompare at the bit level and the failing side will
report an ECC uncorrectable error. The controller can then tell which data is
good and satisfy the read request with good data. The controller will then
overwrite the bad data with the correct data from the other memory card.

If a memory card should fail, the system will be able to continue at full
performance. The error will be recognized by the controller, the system will
switch off mirrored mode and just read and write to the good card. After the
defective card is replaced, the controller switches to recovery mode. All reads
will go to the good side and all writes go to both sides. As a background
operation, the system will read from the good side and copy the data to the
replaced memory card. This recovery operation continues until fully mirrored
data resides in both memory cards and the system switches to full mirrored
operation.

All power function are duplexed (two power cords, separate AC/DC converters,
separate battery systems, and separate voltage converters). Apart from the AC
input control, all power system components are hot-pluggable. If an AC input
control should fail, the other AC input control will take over.

The support facility performs such functions as configuration changes, initial LIC
load, error analysis, logging and reporting. The support facility is a critical
element for high data availability by continuous monitoring and reporting of all
key components. The support facility is also fully redundant.

6.3.7 S/390 Multiprise 2000 Server Internal Disk

This innovative technology gives S/390 Multiprise unmatched internal storage
capacity of up to 288GB, providing more storage in less space because of the
integration of the server with its storage controller and disks.

The internal disk drawer advances the use of industry standard disks, or JBODs
(Just a Bunch of Disks), in a RAID 1 configuration. With internal disk, customers

Chapter 6. IBM DASD Subsystems 131

 can attach their disks by a SCSI 2 Fast/Wide adaptor card, eliminating separate
external control units previously required for S/390 storage.

All internal disks are mirrored (duplicate) to provide the capability to

concurrently maintain HDD′s (hot swap). By configuring disks, drawers and
adapter cards in pairs, the internal disk features are configured to eliminate
single points of hardware failure or repair. In the event of a failure, data is still
available using the other copy. Maintenance can be performed concurrently for
all disks, drawers and adapter cards in all configurations except the entry
configuration. For the entry configuration, disks can be concurrently maintained.
However, changing the cache memory size or adding capacity that requires
additional drawers is disruptive.

A new function, Mirroring Status, provides an indication to software when a

change occurs that causes mirroring to be turned off or back on again. All
operating systems (OS/390, MVS/ESA, VM/ESA and VSE/ESA) will issue a Sense
Subsystem Status command to determine the state and issue a message to the
operator. OS/390, MVS and VSE operating systems also allow the operator to
issue a query at any time to display the current mirroring status. ICKDSF can
also be used to query a volume′s mirroring status for VM, as well as OS/390,
MVS and VSE.

Refer to 2.2.3, “9672 High Availability Features” on page 9 for more detailed
information.

6.4 Non-Disruptive DASD Migration Techniques

Another common reason for extended scheduled outages are disk migrations.
Whether because all data from a disk pack to a complete disk string has to be
moved to other disks for maintenance purposes, or because old disks are being
replaced with a new hardware generation, many hours of planned downtime can
result. In all cases, migration of data should always be carefully planned, and
you should always have a fallback plan.

As we seen, many IBM DASD functions can help you achieve your data
migration requirements in term of continuous availability Let us have a look at
each of them.
• Software Physical Copy
Physical volume copy is the traditional method for migrating data from
similar device geometry. If you are moving volumes to like devices of larger
capacity, generally you need a larger VTOC because the larger device can
hold more data sets. This type of migration is disruptive in most cases
because you have to “freeze” all updates on the migrating volume.
• Software Logical Copy
Logical copy allows you to move data between unlike devices. As a physical
copy, this is a disruptive migration.
• Using SMS
Using SMS to move data provides an easy non-disruptive migration path.
The following procedure could be use to migrate data either between like or
unlike devices with SMS.
1. Define a new storage group.

132 Continuous Availability S/390 Technology Guide

 2. Change ACS routines to reflect the new storage group.
3. Set the old storage group to DISNEW to not allow any more new
allocations to it.
4. ENABLE the new storage group and activate the new configuration that
will direct new allocations to it.
5. Once new allocations are correctly going to the new storage group, you
can then move the existing data sets over to this new SG.
6. Once all the old data sets are migrated over, you can remove this old
storage group from the configuration.
• Dual Copy
As discussed in 6.3.2.3, “Dual Copy” on page 109, Dual Copy is an hardware
feature of the 3990/9390 control unit. You can manage a non-disruptive
migration between two geometry like devices behind a unique 3990 control
unit. After you have migrated your data, you should plan to ask your CE to
reset the control unit at the next IPL.
A full procedure to replace an HDA is described in the WSC flash 9325.
• Sparing a drawer
By using the manual RAMAC Drawer sparing function, it is possible to
dynamically free an entire RAMAC, RAMAC 2 or RAMAC 3 drawer.
• Peer-to-Peer Remote Copy (PPRC)
PPRC is an efficient tool for transferring data between various type of DASD.
It has been successfully used by customers to migrate thousands volumes of
3390 data to RAMAC. Another useful tool, PPRC Migration Manager, is
available to help you to manage large scale migration. PPRC can be used
for:
− Temporarily moving data to a new location (for instance, to
accommodate a repair)
− Migrating data to a new device
− Making a copy of data at any point in time
If you use P/DAS, you can manage disk migration without any outage. The
procedure is as follows:
1. Establish PPRC pairs for the disks to be migrated.
2. Use the P/DAS function to switch the roles of the primary disk and the
secondary disk; this switch will be transparent to any application
programs.
3. Discontinue the PPRC function.
However, to migrate a large number of disks (including Page Dataset) it
might be necessary to plan an IPL to perform the cutover. The procedure is
as follows:
1. Establish PPRC pairs for the disks to be migrated.
2. Stop all systems or subsystems accessing those disks.
3. Delete the PPRC pairs.
4. Restart the systems on the new devices.
• Extended Remote Copy (XRC)

Chapter 6. IBM DASD Subsystems 133

 XRC is an efficient tool for moving data between various DASD types. For
example, you can use either the RAMAC 3 or RAMAC Virtual Array as a
secondary device of an XRC pair.
The procedure to migrate data is quite similar to that of PPRC, but a system
restart is required. The procedure is as follows:
1. Establish XRC pairs for the disks to be migrated.
2. Stop the systems or subsystems that use those disks and discontinue the
XRC function for these disks.
3. Issue an XRECOVER command to change the secondary volume serial
number to be the same as the primary volume serial number.
4. Restart the system with the former secondary disk now being used as
the primary (only) copy of the affected data.
• Data Migrator Service Offering
IBM Migration Services for the RAMAC Array Products provides a
non-disruptive migration solution when converting from 3380 or 3390 DASD to
the RAMAC Array Family of products. IBM Migration Services for RAMAC
improve application availability, with outages occurring only during volume
switch.
By allowing experienced IBM services specialists to perform this service, the
customer′s staff is free to perform more productive work.
At the center of this new offering is the IBM RAMAC Data Migrator, which is
intended to enable migration to or from most existing 3990-compatible
storage subsystems: the IBM 3990 (any model) or the RAMAC Family, and
3990-compatible products from other vendors. Note: 3880
Note: The 3880 is not supported. In addition, this new tool supports 3380 and
3390
In addition, this new tool supports 3380 and 3390 track formats (source and
target must be the same geometry, and the target must be the same size or
larger).
Throughout the operation of the RAMAC Data Migrator, the system and
applications remain available for use while the data migration is taking
place. The tool migrates individual volumes, not subsystems, which allows
for concurrent, multiple volume migration and flexibility .
This enables the IBM services specialist to assist in developing a migration
plan that is tailored to specific system requirements and business needs.
At the conclusion of the service, the IBM services specialist provides a
Migration Control Book that contains the high level migration plan, status
reports, and data migration records.

6.5 DASD Matrix Function Table

Table 6 on page 135 lists IBM DASD matrix functions.

134 Continuous Availability S/390 Technology Guide

Table 6. IBM DASD Matrix Function Table. Legend: X = A p p l i e s
Availability Functions RAMAC RAMAC RAMAC RAMAC RAMAC S/390
3 Array 2 Virtual Scalable Electronic Multiprise
Subsystem Array 2 Array 2 Array 2 2000
Dual Copy X
Primary PPRC X
Secondary PPRC X
Primary XRC X
Secondary XRC X X X
CUIR X
RAID 1 X X
RAID 5 X X X
RAID 6 X
FFST X
Concurrent Copy X
SnapShot Copy X
Virtual Disk Architecture X
FlexVolumes X X
128 Logical Channel Path X X X
512 Logical Channel Path X X
SMS Continuous Availability X X X X
Device
SIM Messages X X X X X X
Non-Disruptive Disk Installation X X X
Concurrent Microcode Update X X X
Concurrent Maintenance X X X X X
Remote Service Connectivity X X X X

Chapter 6. IBM DASD Subsystems 135

136 Continuous Availability S/390 Technology Guide
Chapter 7. Parallel Sysplex Architecture

The Parallel Sysplex Architecture is not shipped in a single product. It is a

combination of functions delivered by hardware, software, and microcode, all
blending together to form a parallel processing environment. Anyone using one
of the services provided by the Parallel Sysplex does not see the clustering, only
a level of service that is better than that of previous experiences.

7.1 Summary of Continuous Availability Attributes

Table 7 highlights some of the availability topics that a Parallel Sysplex is
designed to address. This assumes that the appropriate redundancy is in place
to support the solution. The Frequency, Duration, and Scope columns refer to
the impact of service outages.

Table 7. C/A Attribute Summary Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e Type,

HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Reduce Reduce Reduce Type
Outage Outage Outage
Frequency Duration Scope
Fault Tolerance X X X HA
Automated Failure Detection X X X HA
Automated Failure Bypass X X X HA
Automated Failure Isolation X X X HA
Non-Disruptive Hardware Changes X X X CO
Non-Disruptive Software Changes X X X CO
Non-Disruptive Policy Changes X X X CO
Dynamic Workload Balancing X X X CA
Data Sharing X X X CA

7.2 Parallel Sysplex Concept

Parallel Sysplex is the current step in the evolution of IBM′s multi-processing
mainframe architecture. It is the base for future function, and for reduction in
cost of computing for the OS/390 environment. The S/390 Parallel Sysplex is an
advanced commercial processing clustered system. It creates a processing
facility where data can be shared, with full integrity, between members of the
sysplex. It also allows the workload to be dynamically routed and balanced
among these same members. Through this datasharing and dynamic workload
balancing, high availability and continuous operation can each be greatly
improved over traditional single or even multiple systems.

These terms of datasharing, workload balancing, high availability, continuous

operation, and continuous availability sound very impressive, but what do they
really mean? Let us consider an example from daily life to show what these
terms may mean to you in a common situation. Think about a bank branch.

In this branch, four bank tellers are working at their positions. They each have
access to all the money and other facilities needed to support the services the

 Copyright IBM Corp. 1998 137

 bank provides. A single teller drawing money from the cash pool does not
prevent other tellers from getting money out from a different section or placing
money back into a different section.
• (This can be thought of as datasharing.)
Each of the tellers is able to handle any request from any customer who comes
into the branch. When customers come in, they are held in a single queue and
then directed in turn to the first available position.
• (This can be thought of as dynamic workload balancing.)
If one of the bank tellers suddenly becomes ill and has to stop serving
customers, the work is shared between the remaining tellers.
• (In our context, this is high availability. The unplanned loss of one telling
position has not interrupted services provided by the bank.)
At lunchtime, each teller takes a turn in closing for a lunch break. New
customers are stopped from going to that closed position, and when the other
tellers finish their current work, they can go for their break.
• (This is part of continuous operation, as the planned loss of a teller has not
interrupted services provided by the bank.)

This simple example shows what the Parallel Sysplex approach to continuous
availability is about. The services provided by the bank are viewed from the
outside. The door of the bank is never closed as a result of a single teller
position being unavailable; instead, as long as any position is open, customers
are still allowed to enter and their requests for service will still be handled.

save You can view Parallel Sysplex in the same way. If multiple members are
sharing the workload then the loss of any single member will not cause a loss of
services from the sysplex as a whole. The “door” of the sysplex remains open
to customers, and work is simply re-routed to other active members. In this
way, Parallel Sysplex offers a platform for both high availability and continuous
operation.

One final example demonstrates another important part of the Parallel Sysplex
design.

In the same bank branch, if the workload starts to increase, an additional teller
is required to meet the extra demand. It would be unacceptable to close the
bank to allow this additional teller to help with the workload. Instead, by
ensuring that the additional teller has both the correct skills and access to all the
same facilities as the other tellers, a new position is easily opened to deal with
the load. This occurs without interrupting the service to customers.
• (This is another aspect of continuous operation: non-disruptive change.)

When extra capacity is required in a Parallel Sysplex, new systems (processors

or LPARs) can be added without the need to close the sysplex first. The
important prerequisite is that the new system must have connectivity to all
shared databases and information needed to allow its full participation in the
sysplex. The current maximum is 32 members, and each can be up to a 10-way
S/390 processor, so reaching the maximum capacity limit should not cause a
service loss either.

138 Continuous Availability S/390 Technology Guide

7.2.1 Operability
We have seen how services are thought of as coming from the sysplex as an
entity, not the individual members. The operational view of the sysplex should
be the same. Parallel Sysplex has been designed to appear and work as though
it was a single OS/390 image. This has been achieved by automation and by
elimination of unnecessary operator tasks, to reduce the overall complexity of its
routine operation. The sysplex should be as close to self-managing as possible
in order to achieve the high availability, continuous operation, and continuous
availability challenges with minimum impact to customers and operations
support staff.

Important
The fact that a Parallel Sysplex can meet all of these challenges makes it the
ideal platform for delivering continuously available services.

There are many books on the subject of Parallel Sysplex which describe the
technical details of exactly how data sharing and dynamic workload distribution
are achieved. A short description of each is included later in this chapter after
the main components of a Parallel Sysplex have been described. This chapter
concentrates on examining the sysplex components and the way they combine.
The aim is to review alternative Parallel Sysplex configurations to understand
limitations or strengths when designing for continuous availability. This chapter
also discusses the topic of non-disruptive software upgrades and what must be
done to perform them successfully.

Refer to Sysplex Overview for an introduction to Parallel Sysplex concepts, and

to OS/390 MVS Parallel Sysplex Configuration Cookbook for more detailed
information on building configurations. The Bibliography section of this redbook
also lists other related publications which may be of assistance to you.

The following section gives a short functional description of the hardware

components introduced to support the Parallel Sysplex, and any continuous
availability attributes they bring to the Parallel Sysplex.

7.3 Typical Configuration

Figure 41 on page 140 shows a sample Parallel Sysplex configuration capable of
tolerating the loss of any single major component. For clarity only, a single path
is shown between most components. In reality, multiple paths should be
configured to ensure that the path itself is not a single point of failure.

Chapter 7. Parallel Sysplex Architecture 139

 Figure 41. Parallel Sysplex Components

The processors shown in this figure are S/390 models with multiple processing
units to provide recovery in the event of an error. This configuration may allow
a maintenance action to be deferred to a more convenient time, when full
capacity is not required from the sysplex. Several types of S/390 processors can
participate, from the final generation of the water- and air-cooled ES/9000 range,
through all generations of the 9672 CMOS range. The environment supporting
the sysplex is also assumed to be free of single points of failure (power line, air
conditioning, and so on).

7.3.1 Coupling Facility

The first component within a Parallel Sysplex that may be new is the coupling
facility. This function can be performed by a standalone specialized S/390
processor classified as the 9674, or it can be operated within an LPAR of a
standard S/390 processor.

The objective of the coupling facility is to form a shared storage device for use
by the active members of the sysplex. The information stored in the coupling
facility can be active data, locking information for active data, or
informational-type records and messages required by all sysplex members.
These records in the coupling facility are called structures . By placing the
relevant data and control information in this shared storage, recovery and

140 Continuous Availability S/390 Technology Guide

re-distribution of work can be performed following a failure or a scheduled
removal of a member. If a coupling facility should fail, each member tracks the
information that it has stored in the coupling facility, and is then able to re-write
the same information to a backup coupling facility if one exists.

7.3.1.1 Availability Points

• Hardware Features
The 9674 is based on the 9672 hardware design and it therefore inherits all of
the continuous availability function designed into the 9672. These include
Enhanced Processor Design, Dynamic SAP Sparing, Dynamic Memory
Sparing, and so on. Refer to Chapter 2, “S/390 Processors” on page 5 for
further details of continuous availability 9672/9674 processor-related features.
In addition, the 9674 has an addition to the Internal Battery Feature (IBF)
which normally allows the coupling facility to function for a limited period of
time following a power line break, up to about 20 minutes. In the new
feature, Battery Save Mode, the contents of memory are preserved for over
one hour after the line break. (The exact time depends on the configuration
in use.) This preserves all the structures in the coupling facility and
eliminates the time needed to rebuild them on restart.
Using the Internal coupling facility (ICF) feature of the 9672 allows the same
function to be performed within an LPAR, therefore reducing cost. This can
either be used as a single-system sysplex or, by adding the correct
hardware, the ICF can be accessed by other external members.
ICF can be used within a single system complex to assist development and
migration to a Parallel Sysplex. The feature called Integrated Coupling
Migration Feature (ICMF) allows up to two coupling facility LPARs to be used
by the OS/390 LPARs of the same machine without adding additional
hardware. This could also form a useful test facility for software checkout
before implementation on a production sysplex.
It is worth noting however, that the use of ICMF has a performance penalty
associated with the way that the coupling facility is accessed. Coupling
Facility requests can be handled either synchronously (which is a fast
response), or asynchronously (which is a slower response). Using ICMF, all
requests are handled asynchronously, which carries a performance
overhead. For test and development, this overhead may be acceptable, but
this is another reason why ICMF is not recommended for production use.
• Redundancy and Recovery
From Figure 41 on page 140 it is clear that at least two Coupling Facilities
are required for full redundancy. Loss of the coupling facility in a sysplex
with no backup means loss of the sysplex, and all members stop processing.
This does not mean that a second coupling facility can only be used when a
failure occurs, and normally sits doing nothing. This second coupling facility
can be configured to perform useful work also, but care must be taken to
ensure that sufficient spare capacity is maintained in each to allow the
takeover of the other′s work. This could be following a failure, or following a
planned closedown of the other coupling facility.
Splitting the work between two facilities also allows a faster takeover,
because only half the contents need to be rebuilt, not the complete contents.
Remember that each member of the sysplex is responsible for rebuilding its
associated information.

Chapter 7. Parallel Sysplex Architecture 141

 Note: There is no direct backup capability between coupling facilities.
• Coupling facility types
The next decision point for continuous availability is whether these multiple
facilities should be standalone, LPARs of a S/390 processors (ICF), or a
mixture of both.
The answer lies in a trade-off between cost and how close to 100%
availability is required. The key design point is that the configuration should
tolerate a failure in any single major component. A solution that uses ICF on
a processor that also supports working members of the Parallel Sysplex
breaks this design point. A failure of this processor will cause the loss of
two major components, and this will probably mean a major loss of service
until manual recovery can be performed. This also tends to rule out a
configuration actively using two internal Coupling Facilities. Both of these
options may be fully acceptable for testing environments, or if fast recovery
is not an issue.
However, it is both possible and practical to run with an internal coupling
facility operating as a backup to an active 9674 standalone. Failure of the
9674 can now be compensated for automatically, but recovery time will be
longer than operating with two standalone 9674s, as the full contents of the
9674 need to be rebuilt in the backup ICF.
The dynamic coupling facility dispatching feature described in Chapter 2,
“S/390 Processors” on page 5 is intended for this standalone active/internal
backup configuration. It tries to deliver the most cost-effective solution in
providing a continuous availability configuration. DCFD only allocates
processor resource to the coupling facility when it is required to perform
genuine work. This will almost certainly have an effect on performance from
the sysplex compared with the 9674, but services will still continue.

7.3.2 Coupling Links

7.3.2.1 Description
The hardware which forms the communication path between the coupling facility
and the S/390 processor is known as the coupling link. Fiber optic cable (either
multi-mode or single mode) is used to connect the two devices. This is a high
speed, high bandwidth design and although it uses the same standards of fiber
optic cable used by ESCON channels, it does not use normal ESCON channel
protocols. Links can be added individually from 0 up to a maximum of 16 on a
9672, and from 2 to a maximum of 32 on a 9674 standalone coupling facility.

Each coupling link can be considered as directional, that is, the end connected to
an OS/390 image is defined differently from the end connected to a coupling
facility. These ends are known as CF Send (CFS) and CF Receive (CFR)
respectively. The ESCON Multi-Image Facility (EMIF) allows the CFS hardware of
a link to be shared between OS/390 LPARs.
Note: The CFR link hardware cannot be shared at this time.

A link attached to a normal 9672 can be defined as either a CFS or a CFR if ICF
is being used. This is defined and changed using HCD.
Note: The link attached to a 9674 standalone coupling facility can only be a CFR
type.

142 Continuous Availability S/390 Technology Guide

 7.3.2.2 Availability Points
• Link Redundancy
In a configuration where a coupling facility is connected to an active sysplex
member by only a single coupling link, any failure in this link has exactly the
same result as losing the coupling facility or the complete hardware/software
of the member. That member can no longer communicate with the coupling
facility, so it can no longer be part of the sysplex.
Therefore, it is essential that sufficient links are included for redundancy in
this area. This is relatively easy to plan when using only standalone 9674
facilities, but when using an internal coupling facility and EMIF, a little more
care is required. Figure 42 is an example of this situation.

Figure 42. Internal Coupling Facility Connections

This simple configuration shows OS/390 images 1 and 2 on Processor A

sharing the same CFS link to the internal coupling facility also on Processor
A. (Note that an external connection must be made, it is not emulated.)
OS/390 images 3 and 4 on Processor B also share a (different) CFS link.
This means that a total of three links are needed on Processor A, but only
one is needed on Processor B. For redundancy, this number would now
need to increase to six on Processor A, and two on Processor B.
• Link Performance
Just as with normal ESCON channels, the performance of the coupling link
needs to be monitored. A bottleneck on a coupling link will cause severe
system problems, which will be easily seen by end users. The Generation 3
and 4 S/390 CMOS processors now have a higher bandwidth link (HiPerLink)
available, but performance and capacity of the links need to be monitored to
ensure system availability. RMF provides an activity reporting field.

7.3.3 Sysplex Timer

In order to guarantee the correct order of database updates, all sysplex
members must use exactly the same time source. For this reason, the time
source must be taken from a single point of control outside of all processors.
The 9037 Model 002 Sysplex Timer is the unit designed to perform this task. It is
also sometimes called the External Timer Reference (ETR) in OS/390 or MVS
publications.

Chapter 7. Parallel Sysplex Architecture 143

 The 9037 is a tabletop unit that connects via fiber optic cable to a Sysplex Timer
hardware feature on the processors it is to provide timing signals for. Processor
connections (ports) are installable in groups of four, up to a maximum of 24.
These timer ports can be added dynamically. The 9037 is controlled by a Token
Ring-attached console, but this console function can be shared with the ESCON
Director Console or the Hardware Maintenance Console (HMC) of a S/390
9672/9674.

The standard maximum distance between a 9037 and a processor is 3 km, but an
RPQ is available to extend this distance using the 9036 model 003 Extender. The
Distributed Console Access Facility (DCAF) can also be used to provide a remote
console access to the 9037 timer. This can be over a Token Ring or via a dial-in
connection.

The 9037 can also use the Coordinated Universal Time (UTC) signal as a further
check on its accuracy and status.

Note that the timer is not used by a coupling facility.

7.3.3.1 Availability Points

• Redundancy
Parallel Sysplex needs the 9037 to synchronize all clocks in all S/390
processors connected to the sysplex. This means that if the 9037 fails, the
sysplex also fails. The 9037 is now a single point of failure for the Parallel
Sysplex.
An Expanded Availability Feature can be ordered for the 9037 which
eliminates the 9037 as a single point of failure. This Expanded Availability
Feature provides a second 9037 and special communication cards to allow
both timers to check the other timer′s status. Each timer requires
connection via fiber optic cable to each processor in the sysplex, and two
fiber optic cables form the communications link between the timers. For
processors with dual sides, a connection from each timer should be made to
each side of the processor for further redundancy. The timer pair normally
operates with one timer active and the other timer in backup mode, ready to
take over if any error appears.
• Console Support
Both timers can share the same console, and the same console
configuration rules apply as for a single 9037. A backup console is strongly
recommended.
• Timer Separation
The timers can be placed up to 3 km apart using the normal multi-mode
fiber, but with the extended distance RPQ, they can be separated by up to 26
km. The extended distance RPQ uses the 9036 Model 003 Extender unit at
each end of each link. Figure 43 on page 145 shows a configuration using
this RPQ.

144 Continuous Availability S/390 Technology Guide

Figure 43. Separated Data Centers

An important consideration in this diagram is the number of 9036 Model 003

units required to complete the configuration and their associated costs.
Although the console connection is not shown on this diagram, console
access to each timer from each site should also be allowed. This can be
done by using DCAF or by extending the Token Ring to cover both sites.
Extending the ring can be done in several ways. Perhaps the simplest is a
multi-station access unit for the Token Ring, with a fiber ring in-ring out
connection at each site. You should also plan for redundancy or backup for
console access, especially if the console is used to control the HMC, ESCON
Directors, and 9037 timers.

Additional information on Sysplex Timer installation is available in Planning for

the 9037 Model 002 Sysplex Timer . Implementation and management advice can
be found in 9037 Sysplex Timer and S/390 Time Management . Although this book
still refers only to the previous Model 001 Timer, the considerations for time
setting and changing are still valid.

Chapter 7. Parallel Sysplex Architecture 145

 An important point to consider is when planning local clock changes is daylight
saving. When changing the clock back, the possibility of having database
updates with the same time-stamp arises. This problem can be avoided by
keeping the 9037 Timer(s) at a base time like GMT, and using an offset to reflect
local time for operator convenience. IMS is the main subsystem to be affected
by this change, and it is being changed in Version 6 to use the base time, not
local time. This avoids a one-hour outage while waiting to avoid duplicate
time-stamps on database records.

7.3.4 Sysplex Couple Datasets

For effective monitoring and recovery, critical information about the sysplex is
defined, stored, and updated on shared DASD. These groups of information are
known as Sysplex Couple Datasets, and a number of them can be used
depending on which features are activated within the sysplex. The current list of
Sysplex Couple Datasets is:
• Sysplex (or XCF) couple dataset
• Coupling Facility Resource Manager (CFRM) couple dataset
• Sysplex Failure Manager (SFM) couple dataset
• Workload Manager (WLM) couple dataset
• Automatic Restart Manager (ARM) couple dataset
• System Logger (LOGR) couple dataset
A utility is included with OS/390 to perform this task, and both primary and
alternate devices should be defined for availability. When deciding on
placement, some consideration must be given to performance and contention for
certain conditions. This information is available in various publications, but
S/390 MVS Parallel Sysplex Continuous Availability SE Guide is a good reference.

You should ensure that primary and alternate devices are separated between
different DASD subsystems, and are placed on low activity volumes. Avoid
mixing these datasets with other data that may be used by an application using
a Reserve on the volume, for example some HSM functions.

The Sysplex Couple Dataset is used by all members to check that each member
is alive. Each is required to read and write to this dataset within a set time
period. A reserve may prevent this from happening and cause a member
system to be forced out of the sysplex because it has not been able to complete
this operation. Other conditions can cause similar contention. Refer to S/390
MVS Parallel Sysplex Continuous Availability SE Guide for further information in
this area.

7.3.5 Data Connectivity

Another important point when designing a Parallel Sysplex is the area of I/O
sharing to support data sharing. Data sharing cannot be activated on all sysplex
members if they do not have access to the same databases and other
information required to process the arriving workload. With the potential of up to
32 OS/390 systems requiring access to the same devices, connectivity can be
difficult unless ESCON Directors are used to assist.

ESCON Directors reduce the need for permanent connections between each
processor and all devices, by allowing a dynamic connection only when
required. They also make reconfiguration and upgrade much easier as they

146 Continuous Availability S/390 Technology Guide

 separate the processor connections from the devices. New subsystems can be
installed, connected into the Directors and accessed from existing processor
channels. Similarly new processors can be installed and connected in the way
to access exiting subsystems. This approach helps the non-disruptive change
demands of continuous availability. Further information is available in
Chapter 3, “S/390 ESCON Architecture” on page 35.

7.4 Datasharing
Datasharing is the ability of the Parallel Sysplex to allow its members read/write
access to the same databases, while preserving the sequence of updates and
ensuring the integrity of each database. This must also be performed with
minimum performance overheads. The coupling facility is used extensively to
solve each of these challenges by using the different types of structures to lock a
record during update. It is also used to make the updated record available to all
without waiting for normal I/O write operations.

Chapter 10, “IMS” on page 185 and Chapter 11, “DB2” on page 199 explain
how IMS and DB2 make use of the coupling facility to achieve datasharing, and
no further technical explanation is included here.

Datasharing is a key element in the Parallel Sysplexes continuous availability

design. It allows the redundancy needed to overcome both the failure of a
member which is processing database updates, and also allows the scheduled
removal of a member for service or for a similar action. In each case, the
service provided by the sysplex is unaffected. A user may experience a request
to resubmit a transaction, depending on when a failure occurred, but this is far
superior to previous continuous availability capabilities, without having fully
duplicated facilities lying dormant but ready to take over following a failure.

7.5 Dynamic Workload Balancing

Dynamic workload balancing or routing is the ability of the Parallel Sysplex to
move work around members of the sysplex regardless of where that work
entered the sysplex. This normally applies to transactions arriving from a
network which have fairly short duration and need a quick response, but as the
Parallel Sysplex evolves, similar decisions are being made for longer-running
applications like batch.

Sharing the incoming workload normally requires the formation of a common

input queue that work is taken from and a common output queue that completed
work is sent to. The coupling facility structures are used to provide this shared
service and also deliver the performance necessary to run this without a
performance penalty.

Chapter 9, “CICS” on page 163 and Chapter 10, “IMS” on page 185 describe
how each of these transaction management systems exploits workload balancing
in a Parallel Sysplex, so no further technical information is included here.

Dynamic workload balancing is the other important function that a Parallel

Sysplex offers to provide continuous availability. Again, any failure or scheduled
closedown is automatically compensated for by the surviving members of the
sysplex, and no loss of service is seen by the users.

Chapter 7. Parallel Sysplex Architecture 147

7.6 Potential Configurations
The following sections illustrate several Parallel Sysplex configuration variations.

7.6.1 Internal Only

Many multiple processor installations plan their configurations so that Processor
B is capable of supporting the two OS/390 LPARs and also the ICF. Using the
example in Figure 42 on page 143, this would allow Processor A to be closed
down for regular computer hall power safety checks. Figure 44 shows the
configuration reversed. Redundant links have been deliberately omitted for
clarity.

Figure 44. Reversed Configuration

The figure shows that this now requires four coupling links on each processor
(or eight, for redundancy), and the cables between the two machines can be left
permanently connected,

You may ask, is a power-on reset required to change the LPAR definitions? This
is a situation that the Dynamic coupling facility Dispatching (DCFD) feature can
assist with. Without the DCFD, an ICF backup like the one shown in Figure 44
would take CP cycles away from the OS/390 LPARs sharing the CP resource.
With DCFD however, the ICF only runs when there is a request for it to process.
It now offers a solution to providing a test and development facility that can
tolerate planned outages.
Note: This configuration is not recommended to support production workloads,
because using only an an ICF creates a single point of failure that can affect two
major components.

7.6.2 Mixed Coupling Facilities

Figure 45 on page 149 shows an expanded configuration with a standalone
coupling facility, and an internal coupling facility used for backup. This is the
minimum recommended configuration to support an active production sysplex.

148 Continuous Availability S/390 Technology Guide

Figure 45. Mixed Coupling Facility Configuration

In this configuration, EMIF is still used to share CFS ends of each coupling link,
but again no link redundancy has been shown. Full redundancy would require
all numbers to double again, needing eight links on Processor A, and four links
each on Processor B and the 9674.

Again, consider the possibility that the function of Processor A may need to be
switched to Processor B. In a Parallel Sysplex, the workload can be seamlessly
moved if Processor has enough capacity to carry it all.

Many installations, however, may plan configurations that allow the full function
of Processor A to be swapped with Processor B and vice versa. This means that
the full hardware configuration of Processor A must be repeated on Processor B,
including all coupling links. This approach requires careful planning and
assessment, to see if the additional hardware and reconfiguration effort required
to make the change is justifiable. If the configuration shown was intended to be
built across two different sites, then placing the 9674 and Processor A on the
same site may give enough resilience to meet the service requirements.

Also, the maximum number of coupling links that can be installed on a 9672 is
16, which can be rapidly used up in a configuration like this.

Chapter 7. Parallel Sysplex Architecture 149

7.6.3 External Facilities
Figure 46 shows a fully redundant configuration in terms of both coupling facility
and coupling links.

Figure 46. External Coupling Facilities

In this configuration, the workload can be split not only between the two
processors, but also can be split between the two coupling facilities. This offers
the best combination of resilience and performance available to support a full
data-sharing, workload balancing Parallel Sysplex. It also uses the minimum
number of links on each S/390 9672 processor.

7.7 Management and Use

The advantages of running a Parallel Sysplex may appear very convincing and
attractive, but the question “How easy is this to use?” is often asked. A major
design consideration was that regardless of the number of images involved, a
Parallel Sysplex should appear like a single MVS or OS/390 image to users and
operators. This means that from a single workstation, the complete sysplex can
be monitored and managed effectively and efficiently.

The hardware, microcode, and software products associated with a Parallel

Sysplex have all been designed with availability and ease of management in
mind. The aims of automated detection, analysis, and either resolution or
bypass are key focus areas of the Parallel Sysplex solution, but these must
always follow guidelines set by each installation.

150 Continuous Availability S/390 Technology Guide

 For example, the Sysplex Failure Manager (SFM) is used to describe a set of
actions that the sysplex should follow in the event of certain failures. These can
range from the loss of a LPAR (the remaining active LPARs can be allowed to
automatically take the storage from the failing LPAR), to failures within database
subsystems. The active set of instructions is known as an SFM Policy, and this
policy can be changed dynamically without a service interruption.

A similar facility is available to control workload resource allocation within the

sysplex. Again, the guidelines are contained in a prepared policy, in this case
the Workload Manager (WLM) Policy, and again, it can be changed dynamically.
This allows the members of a sysplex to comply with one set of rules during the
online transaction day, and a different set of rules to favor overnight batch
processing. This can be used to ease the conflict and pressure that exists to
ensure batch work meets its daily deadline.

7.8 Non-Disruptive Software Changes

The target of non-disruptive software change at first appears relatively
straightforward. This target however, has been the reason that new standards
have been enforced in Software Development organizations to comply with
Parallel Sysplex needs. Continuous operation means that individual instances of
an element can be upgraded by removing that element from the sysplex and
adding the upgraded element back when ready. This demands that both the old
and the new versions must happily co-exist and work together within the sysplex.
This places more emphasis on co-existence testing for the development
organizations.

At present, OS/390 has documented that it will tolerate a range of three

consecutive releases. For example, if you installed OS/390 Release 3 after it
became generally available in March/April 1997, this release can co-exist in a
Parallel Sysplex with the next two releases of OS/390. This range rolls on with
each new release of OS/390. The major subsystems, IMS, CICS, and DB2, have
all announced that they can support two consecutive releases. The convention
for describing this tolerance in many of the product documentation is n + 1 or
n + 2 , where n is the base.

New releases of OS/390 are currently scheduled for April and October of each
year. This probably places the biggest burden on systems programming staff to
keep within the thresholds. Releases of the major subsystems are more
variable, and are generally not superceded for at least two years. Service will
still be released for all products and should not be ignored.

An intensive amount of work is being undertaken in all S/390 software

development to achieve a greater range of tolerance than this, but care must be
taken when planning software upgrades in a Parallel Sysplex. This is especially
true if a jump of more than one release is planned.

The same constraints must apply when redesigning old or developing new
applications to suit the Parallel Sysplex environment. If multiple versions of the
same application are intended to be active concurrently in the sysplex, their
ability to tolerate each other must have been considered and proved.

Again the subject of effective local testing is important to ensure that changes to
the software inventory are fully checked before introducing them into the
production sysplex. It is impractical for the full combination of IBM and OEM

Chapter 7. Parallel Sysplex Architecture 151

 products to be fully tested for each installation′s environment. The responsibility
for final testing must lie with each installation.

7.9 Test Sysplex

The goal of continuous operation places greater emphasis on effective testing of
all changes before introducing them into the Parallel Sysplex. It defeats the
purpose of the exercise if changes that have not been properly tested are
allowed to affect online services. The user community sees no benefit from any
investment in the technology and effort needed to build a production sysplex.

IBM strongly recommends that a second test sysplex be available to perform

final co-existence testing of new versions of applications and subsystems. This
also requires that test databases and sample representative workloads are
available to carry out the testing effectively. The more alike the test and
production environments are, the better are the chances that any problems will
be detected during testing, and not production.

Figure 47 shows a sample combination production sysplex and test sysplex.

Figure 47. Test and Production Sysplex Configuration

The configuration shown adds the following elements:

Another OS/390 LPAR for test purposes on each processor.
(In reality, this resource would probably have been required anyway.)
Another coupling facility LPAR on Processor A as backup for the test
sysplex.

152 Continuous Availability S/390 Technology Guide

 Another coupling facility LPAR on the 9674 for the test sysplex.
Additional coupling links on all machines.
Again, multiple links to provide fault tolerance have not been shown. The CFR
end of a coupling link cannot be shared, therefore the pair of links from either
Processor A or B must be split into production CFR and test CFR. Redundancy
for the test sysplex can be left out to save cost, but redundancy for the
production sysplex is very important.

In addition, the additional storage resource and processor resource needs to be

estimated to ensure that sufficient capacity is available on all machines.

This configuration allows:

• Standalone testing of new subsystems and applications
• Multi-system sysplex testing of new subsystems and applications
• Full failure and recovery testing of all components
It therefore only varies from the production sysplex in terms of the capacity that
is given to it. The DCFD feature could again be used to ensure that the test
coupling facilities only take CP resource when required. Storage
Reconfiguration could also be used to change the size of storage allocated to
both the test OS/390 LPARs and the test ICF on Processor B.

The major hardware purchase is probably the additional coupling links that are
needed on all machines to build this configuration. The alternative is to add
additional processors, which increases the software licensing costs more than
this approach does.

7.9.1 Other Considerations

Some rules and additional points to consider when planning a production and
test sysplex are:
• Coupling facilities cannot be shared between sysplexes.
• Coupling links cannot be shared between sysplexes.
• Sysplex Couple Datasets cannot be shared between sysplexes.
• A GRS ring can extend to systems outside of the sysplex, but these systems
cannot be part of another sysplex.
• DASD shared between sysplexes must use reserve/release locking.
These final points suggest that the easiest way to manage DASD between
sysplexes is to separate the production subsystems from the test systems
completely, and then no sharing and serialization will need to be managed.

7.10 Parallel Sysplex Implementation Service Offering (EPSO)

EPSO delivers IBM Services to assist you in the implementation of Parallel
Sysplex. These services enable you to achieve a phased exploitation, and range
from new hardware leasing through to onsite assistance and education. The two
services modules that may be of interest at this point are:
• EPSO System Enablement
• EPSO Application Enablement

Chapter 7. Parallel Sysplex Architecture 153

 You can choose between either or both of these services as a means to achieve
a productive Parallel Sysplex faster.

7.10.1.1 EPSO System Enablement Summary

System Enablement services include the following items:
• Project management
• Conducting a migration planning session
• Developing a detailed System Enablement implementation plan
• Validating the hardware configuration
• Verifying the customer MVS software installation
• Performing non-IBM software compatibility research
• Providing Hardware Management Console (HMC) training
• Activating the Parallel Sysplex
• Exploiting the Coupling Facility for XCF, RACF, JES2 checkpoint, and tape
sharing
• Planning implementation for the production systems
• Assisting you in implementing the Parallel Sysplex in production

7.10.1.2 EPSO Application Enablement Summary

Application Enablement services include the following items:
• Application selection and planning
• Implementation of one or more of the following:
- CICS
- IMS Transaction Manager
- TSO
- DB2 datasharing
- IMS datasharing
- VSAM Record Level Sharing

7.10.1.3 Other Services

Under the banner of EPSO, other services can be provided, for example:
• Assistance in the customization of IBM System Management products such
as NetView, OPC/ESA, INFOMAN, and AOC/MVS
• Implementation of System-Managed Storage
• PM/MVS performance management
• Conversion offerings for migration to products such as RACF

7.11 Summary
The Parallel Sysplex is IBM′s most extensive solution to address the challenge
of continuous availability. It offers an environment that fully supports all the
demands of eliminating both unscheduled outages and scheduled outages, and
yet this environment can be easier to operate and run than most multiple
processor configurations in use today.

154 Continuous Availability S/390 Technology Guide

Chapter 8. Continuous Operation on a Single Processor

Continuous operation attempts to increase service availability by eliminating

scheduled outages for events like maintenance, upgrade, and installation. This
probably occurs more regularly for software than for hardware, but the ability to
perform non-disruptive actions on both is important to increase service
availability.

This chapter opens a discussion on whether a target of continuous operation is

realistic for installations that have only one processor complex. It is not
intended as a full implementation guide, but offers points you should consider if
you are planning a configuration like this.

8.1 Introduction
To this point, most approaches to continuous availability, high availability, and
continuous operation have concentrated on multiple systems. The best solution
for any platform to attain continuous availability appears to be having a Parallel
Sysplex built around multiple processors. However, multiple processor
footprints and LPARs, with multiple coupling facilities for redundancy and high
availability, cannot be cost-justified by many single processor installations.

Therefore, you need to ask: Are there are any benefits associated with Parallel
Sysplex to consider its use on a single processor?

A Parallel Sysplex contained on a single processor complex does not supply

independence from hardware failures in the processor. It may even be thought
of as introducing further complexity for recovery and restart to the existing
environment. However, it may address many key issues associated with
software upgrades and maintenance. This applies not only to the operating
system and its various subsystems, but also to new and upgraded applications.

This approach may allow a faster response to new business requirements, by

allowing new applications to be introduced without shutting down the existing
application suite. This can be done in a very selective and controlled manner to
reduce the risk to the minimum possible at each stage.

If there is no pressure on your operation to provide a service 7 days a week with

no scheduled closedown or quiesced time, then continuous operation is probably
not an issue for you. In many installations however, scheduled outages are now
causing more impact than unscheduled outages, so a Parallel Sysplex built using
LPARs on a single S/390 processor may start to offer a solution.

8.2 Configuration 1
Figure 48 on page 156 shows a basic arrangement of LPARs within a S/390
processor to support this approach.

 Copyright IBM Corp. 1998 155

 Figure 48. Single System Sysplex 1

The configuration shown makes use of the Integrated Coupling Migration Facility
(ICMF), which is a standard feature of the S/390 PR/SM microcode. As well as
allowing a number of LPARs to be built, PR/SM allows up to two coupling facility
LPARs to be created within the same machine. It also emulates the Sysplex
Timer function and coupling links between OS/390 and the coupling facility. This
means that a sysplex can be built without acquiring additional hardware,
although capacity (storage size and the number of CPs) may need to be
reviewed.

Under normal conditions, the production workload is carried by LPAR 1, which is

connected to the coupling facility to form a Parallel Sysplex supporting the
day-to-day production services. This can be operated successfully as a single
system sysplex, and some advantage may be seen from sysplex-only functions
like Workload Manager (WLM). Refer to section 8.2.1.1, “Workload Manager” on
page 157 for a brief description of WLM. LPAR 2 provides a test and
development facility, to either upgrade software or allow maintenance of
software.

Testing in non-sysplex mode only is a limitation of this configuration. Although

two coupling facility LPARs can be defined using ICMF, the emulated coupling
links cannot be redefined dynamically. This prevents the test OS/390 LPAR from
being detatched from a test coupling facility and reconnected to the production
coupling facility. Redefinition would require a power-on reset of the machine
and our target of continuous operation is already missed.

156 Continuous Availability S/390 Technology Guide

 The risk associated with non-sysplex testing can be managed however by good
change management procedures. This point will be expanded on later in this
chapter. Effective testing is extremely important, even in non-sysplex mode, to
ensure that no problems are introduced by the new changes.

8.2.1.1 Workload Manager

WLM is an enhancement to the System Resource Manager (SRM) component of
MVS. It allows much more dynamic and automated control of work that is
executing and waiting to execute on the system. By analyzing delays within the
active subsystems, WLM can alter internal controls and push more work through
the system in a shorter time. This in turn may mean better response times to
users, or meeting business deadlines more consistently: both results represent
improved service availability.

The targets or goals set for WLM can be altered without closing and reloading
the system, which may also eliminate one or two scheduled outages each year.
WLM can be used in a single system without migrating to a full Parallel Sysplex,
that is, it can be activated without the use of a coupling facility. Refer to OS/390
MVS Parallel Sysplex Configuration Cookbook for further information on WLM.

8.2.2 Non-Disruptive Software Changes

Although LPAR 1 is the only image carrying the production workload, the
datasharing facility of Parallel Sysplex must be used. It is this facility, in
combination with dynamic workload balancing, that allows the non-disruptive
introduction of new software. Datasharing needs a suitable database manager
at the appropriate version/release level for correct support. Refer to Chapter 9,
“CICS” on page 163, Chapter 10, “IMS” on page 185, and Chapter 11, “DB2” on
page 199 for information relating to IBM′s IMS/CICS/DB2 datasharing products.
Enabling datasharing is probably the most intensive project associated with
using a Parallel Sysplex, and it requires careful control.

When all testing is complete, the OS/390 test system can be defined to allow it to
become a member of the production sysplex. The coupling link connection can
remain associated with the test OS/390, because until the system is defined to
become part of a sysplex it will not use the link. This means that no redefinition
is required to switch the Test OS/390 image into and out of the Parallel Sysplex.

Assuming datasharing is active and workload balancing is allowed, introducing

this test LPAR into production will enable it to start taking workload from the
queues and process this work. There will always be an amount of work that is
not suitable for dispersal around a sysplex, therefore this work must be
scheduled to run on the upgraded OS/390 image to ensure full testing.

The two OS/390 LPARs will now be sharing the workload. This should continue
until the test LPAR has been accepted as a solid base for supporting the
production workload. The length of this period depends on the installation, but a
full month should cover most business cycles to prove the new system.

8.2.2.1 Managing the Change

A choice exists when introducing the new OS/390 image into the sysplex:
• Either you can allow it to fully participate in the sysplex and take its full
share of all dynamically routed work, or
• You can control the scheduling of work into the new system to ensure a
systematic approach to reduce risk

Chapter 8. Continuous Operation on a Single Processor 157

 The second approach is probably more favored. This control can range from the
individual transaction level to the subsytem level, depending on the scope of the
changes being introduced. If a severe problem appears to be starting, it is
possible to reverse the last action, or to even take an extreme action such as
quiescing the new system image. Using a phased approach, it should be
possible to introduce the new image into the sysplex in an organized way and
limit the impact of any problem on business services.

When the new system is accepted as stable, the closedown procedure for the old
production LPAR (OS/390 1) can be started. When drained, it can be removed
from the sysplex. Now the upgrade/maintenance process can begin again using
this drained system as the new test facility. Its libraries should be dumped to
tape before making any new changes as a precaution.

8.2.3 Processor Resource Considerations

1. CP Allocation
When defining the resource for each LPAR, a choice needs to be made
concerning the coupling facility: should a dedicated CP be allocated to the
coupling facility LPAR or not?
It is possible to share CPs between OS/390 partitions and the coupling
facility, and the new hardware feature of Dynamic CF Dispatching could also
be used to ensure that the coupling facility makes minimal impact on the
resource required by the production image.
We strongly recommend, however, that you allocate a dedicated CP to the
coupling facility to ensure that work requests to it are not delayed by waiting
for the coupling facility to be dispatched. Such delays could result in poor or
inconsistent response times and have the effect of making user service
worse.
2. Storage Allocation
You also need to allocate storage to the coupling facility. The size of the
storage depends on the workload it is intended to carry, and on the
subsystems that will use the coupling facility. The documentation associated
with each subsystem guide you in estimating the amount of storage required.
In turn, these estimates can be totalled to decide how much storage can be
given to the coupling facility partition.

8.2.4 I/O Sharing Considerations

Access to all peripherals that support the business is important, and the ESCON
Multi-image Facility (EMIF) allows the sharing of all ESCON channels between
LPARs of the same processor complex. This means that no extra channels need
be installed to achieve this connectivity. It is probably better to keep the
production environment fenced off as much as possible from the test
environment in order to ensure no undesired interaction. This can be managed
by simple vary off commands or by automation at system startup time if separate
I/O subsystems cannot be given to each.

When combining the two images in the Parallel Sysplex, some form of resource
serialization is needed to safely share the production I/O subsystems. In a
single processor installation, this may not have been required before. The
OS/390 component, Global Resource Serialisation (GRS), allows this I/O sharing
to be performed correctly, and is also a pre-requisite for sysplex activation. It

158 Continuous Availability S/390 Technology Guide

 can also be used to control access when the test system is not attached to the
Parallel Sysplex.

Network control takeover depends on the machine type and features used for
connection to the network. If a native ESCON channel interface is available, then
this can simply be shared between the LPARs and EMIF. If the network front end
processor (FEP) is parallel channel-attached, then several options are available.
• The simplest option is to use a second channel adapter on the FEP. This
allows the network to be shared easily between the LPARs.
• If using a second channel adapter is not an option for you, then
non-disruptive change becomes impossible. Instead, you can perform
takeover by using CTCs between the LPARs, so that the test LPAR can use
the CTC for VTAM when joining the sysplex. This would be fine until the
time arrives to close the production LPAR. If the parallel channel to the FEP
is defined as reconfigurable, then it can be switched from the production to
the test system, but this means an interruption to services through the FEP.
Your choice of solution really depends on what hardware you are using.

A similar switching problem probably exists for all printers, and 3x74 local
terminal device controllers.

8.2.5 Coupling Facility Performance Considerations

The use of ICMF to emulate coupling links carries a performance overhead.
Without going into technical detail, a coupling facility can handle requests in two
ways: synchronously , which gives a very fast turnaround time, and
asynchronously , which gives a slower response. ICMF emulating coupling links
can only operate asynchronously, so every communication between an LPAR
and the coupling facility will be subject to this slower response.

CTC links can be used between the LPARs to overcome this response problem
for XCF traffic, and are probably useful for other traffic (for example, VTAM).

DB2 in particular makes extensive use of the coupling facility when working in
datasharing mode, so heavy users of DB2 may find this performance overhead
unacceptable. This penalty can be overcome with the use of configuration 2.

8.3 Configuration 2
The next step up the evolutionary scale is shown in Figure 49 on page 160. It
uses four coupling links to form the communication path between the OS/390
LPARs and coupling facility LPAR. This overcomes the performance penalty
associated with the emulated links. All coupling facility requests that can be
handled synchronously will now be processed in this way. Another advantage of
using the external links is that they can be dynamically redefined, if required.

Chapter 8. Continuous Operation on a Single Processor 159

 Figure 49. Single System Sysplex 2

The EMIF feature of the S/390 allows the CFS end of the link to be shared
between OS/390 LPARs. This means that, for the period that both LPARs need
to access the CF, both links can be shared for both resilience and performance.

Alternatively, only one link can be defined to the test LPAR, and then the second
added after testing in the sysplex has shown the new test image to be stable.
IPL′ing an LPAR with no sysplex definitions will stop the LPAR from using the
coupling link.

8.3.1 Non-Disruptive Software Changes

The steps involved in migrating the test system to production are very similar to
the previous example.

An extra step is now possible after testing the new system in standalone mode.
If no devices are shared between the two LPARs, it is possible to operate the
test OS/390 in a Monoplex. (See section 8.3.2, “Multiple Sysplex Considerations”
on page 161 for restrictions on sharing devices outside of a sysplex.) This is not
a full Parallel Sysplex configuration, but simply the connection of the test LPAR
to a different set of sysplex couple datasets. This could allow testing with WLM,
which is an important aspect of the workload balancing and takeover between
images when joining the production Parallel Sysplex.

Some additional considerations are required when planning for this

configuration.

160 Continuous Availability S/390 Technology Guide

8.3.2 Multiple Sysplex Considerations
Although GRS can extend to systems outside of a sysplex, the other systems
cannot be part of another sysplex. This means that the production and test
sysplexes cannot be part of the same GRS configuration.

DASD shared between sysplexes must use reserve/release serialization. This is

important when considering placement of resources like shared libraries, and
follows on from the previous point.

The sysplex couple datasets that form a non-volatile source of configuration and
control information for each sysplex are also non-sharable between sysplexes.
A group of sysplex couple datasets (primaries and alternates) is needed for each
sysplex. Care needs to be taken when placing these control datasets, to avoid
possible interaction between the two sysplexes. Guidance is printed in many
sysplex manuals on where to place these datsets to best advantage; you can
also refer to S/390 MVS Parallel Sysplex Continuous Availability SE Guide .

8.4 Summary
As the examples have shown, management of the LPARs in this way makes it
possible to introduce software changes without impacting business services.
Other variations, with different number of LPARs and coupling links, are also
possible.

We emphasize effective testing to ensure that no problems are introduced into

the production environment. This emphasis is starting to increase in normal
operation anyway, as problems with service are less tolerated by business
users. You should also understand the scope of the changes and managing the
workload takeover on the new system. Both of these points contribute to a
non-disruptive change activation process, which provides a major step towards
continuous operation.

You should also note that, although simple examples are provided, planning and
constructing a Parallel Sysplex is not a trivial exercise. Project durations of six
to 12 months are normal. The success of the project depends upon the effort put
into it by your technical support team, especially at the planning stage. The
more effective the planning is in anticipating and understanding obstacles, the
greater and earlier the success will be.

IBM now has staff with a high level of experience in planning and constructing
Parallel Sysplexes in different situations. Their skills and experiences are
available at every stage of your project. Refer to section 7.10, “Parallel Sysplex
Implementation Service Offering (EPSO)” on page 153 for a sample of services
available.

Chapter 8. Continuous Operation on a Single Processor 161

162 Continuous Availability S/390 Technology Guide
Chapter 9. CICS

The CICS system is, beginning with Version 3, redesigned to:

• Improve availability of the CICS system themselves
• Avoid single points of failure in the CICS configuration
• Provide new function designed for high availability and continuous operation

With each new CICS, version the number of operations required to shut down the
CICS system is reduced.

9.1 Summary of CICS Continuous Availability Attributes

Table 8 shows you which availability attribute applies to frequency, duration, and
scope of an outage and which outage type it belongs to. We distinguish between
planned (continuous operation) and unplanned (high availability) outages.

Table 8 (Page 1 of 2). CICS C/A Attribute Summary Matrix. Legend: X = A p p l i e s ,

Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
Multi-Region Operations (MRO) X HA
Inter-System Communications X HA
(ISC)
Transaction Routing (TR) X HA
Function Shipping (FS) X HA
Distributed Transaction X HA
Processing (DTP)
External CICS Interface (EXCI) X HA
Support for DCE Remote X HA
Procedure Call (DCE RPC)
Asynchronous Processing (AP) X HA
Dynamic Transaction Routing X X CA
Dynamic Transaction Balancing X X CA
CICSPlex/SM X X CA
VSAM Record Level Sharing X X CA
Temporary Storage Data Sharing X X X CA
VTAM Generic Resource Support X X X CA
VTAM Persistent Session Support X X X HA
Improved CICS Architecture X X X HA
Resource Definition Online (RDO) X CO
Autoinstall X X CO
Subsystem Storage Protection X HA
Transaction Isolation X HA
A R M Support X HA
Use of DBCTL X CA

 Copyright IBM Corp. 1998 163

 Table 8 (Page 2 of 2). CICS C/A Attribute Summary Matrix. Legend: X = A p p l i e s ,
Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
DB2 Attachment Facility X CA
Interface to MQSeries X CO
Internet Access X HA
Backup-While-Open (BWO) X CO

9.2 Isolation and Distribution

CICS availability attributes are based on the ability to configure multiple isolated
CICS subsystems within the same host or separate hosts, to communicate
between CICSs or other non-CICS systems, and to distribute functions across
these multiple subsystems.

Isolation is supported by two CICS intercommunication methods:

• Multi-Region Operation (MRO)
• Inter-System Communication (ISC)

Distribution resides in the exploitation of these intercommunication methods:

• Transaction Routing (TR)
• Function Shipping (FS)
• Asynchronous Processing (AP)
• Distributed Transaction Processing (DTP)
• Distributed Program Link (DPL)
• External CICS Interface (EXCI)
• DCE Remote Procedure Calls (DCE RPC)

9.2.1 CICS Intercommunication

As we will see, a CICSplex can be split into a set of independent and
interdependent CICS regions, while retaining a single system image from the end
user′s point of view. It can be achieved via two methods: multi-region operation
(MRO) and inter-system communication (ISC).

MRO is used to connect CICS regions that are within the same MVS image or
that are using the cross-system coupling facility (XCF) of MVS/ESA 5.1 within the
same Parallel Sysplex (MRO/XCF).

The following five CICS regions (building blocks) can be combined to implement
a CICS architecture for a specific installation:
• Terminal owning region (TOR)
• Application owning region (AOR)
• File owning region (FOR)
• Data owning region (DOR)
• Queue owning region (QOR)

164 Continuous Availability S/390 Technology Guide

You must specify the functions a given region can perform, the resources it uses,
and the means of communication between regions.

ISC allows CICS to communicate with other CICS or IMS systems in a different
MVS system. You normally require an SNA access method, such as ACF/VTAM,
to provide the necessary communication protocols. ISC allows the
implementation of the same kind of regions that can be used with MRO.
Figure 50 shows an example of connected CICS regions using MRO, MRO/XCF,
and ISC.

Figure 50. CICS Intercommunication Features

When MRO, MRO/XCF, and ISC are not exploited, terminal management, data
management and application belong to the same CICS region. A functional split
is a partitioning of these functions into separate CICS regions. For example, the
terminal management function can be isolated in a TOR, file management can
be isolated in a FOR, and application can be isolated in an AOR.

From an availability point of view, you get the following advantages:

• Transparent to application program
With this approach there is no need to change applications to implement a
MRO or ISC structure.
• Isolation through distribution of resources
This approach allows the distribution of resources (applications, files, and so
on) as a way to isolate functions.
• Safer testing of new applications
Using this approach CICS applications can be tested concurrently with a
production CICS, using a separate test region. This improves integrity and
minimizes any impact on production. It also allows selective access to files,
databases, and processing on connected systems without requiring terminals
to be dedicated to the test region.
• Faster application recovery

Chapter 9. CICS 165

 Using this approach speeds up the restart time following a CICS failure.
Re-establishing sessions takes a long time, so terminals, applications and
files should be isolated in regions of their own. If an application region fails,
it can be restarted quickly without the overhead of re-establishing terminals
and the integrity of files.
Section 9.4, “CICS Parallel Sysplex Exploitation” on page 171 provides more
information on how to avoid the overhead of re-establishing terminals and
the integrity of files by using a Parallel Sysplex.

9.2.2 Facilities Exploiting CICS Intercommunication

CICS intercommunication provides different types of facilities to implement a
multi-system CICS environment. These facilities are not all available for all
forms of intercommunication. The circumstances under which they can be used
are described in CICS Transaction Server for OS/390 CICS Intercommunication
Guide . Figure 51 shows examples of intercommunication facilities.

Figure 51. CICS Intercommunication Facilities

Transaction routing (TR) enables a terminal that belongs to a TOR to run a

transaction owned by another CICS region (AOR). CICS handles all the routing
of requests and replies between the two CICS systems. The distribution of
processing over independent regions supported by TR allows the design of high
availability CICS architectures.

Function shipping (FS) lets an application access a resource owned by, or

accessible to, another CICS system. Both read and write access are permitted,
and facilities for exclusive control and recovery and restart are provided. The
remote resource can be:
• A file or database
• A transient-data queue
• A temporary-storage queue
Applications that access remote resources can be designed and coded as if the
resources were owned by the system in which the transaction is to run. During

166 Continuous Availability S/390 Technology Guide

 execution, CICS ships the request to the appropriate system. The major
availability attribute of FS comes from the isolation of data. For example there is
no DBCTL code in the AOR: DBCTL code, control blocks, buffers, and database
recovery control (DBRC) are in the data owning region (DOR).

Asynchronous processing (AP) allows a CICS transaction to initiate a transaction

in a remote system and to pass data to it. The remote transaction can then
initiate a transaction in the local system to receive the reply. The reply is not
necessarily returned to the task that initiated the remote transaction, and no
correlation between requests and replies is possible. The processing is
therefore called asynchronous.

Distributed transaction processing (DTP) enables a CICS transaction to

communicate with a transaction running in another system. DTP allows the
logical split of a transaction into complementary transactions executing
synchronously in connected regions. The transactions are designed and coded
to specifically communicate with each other, and thereby to use the intersystem
link with maximum efficiency. The logical distribution of processing allows a
modular design of transactions into transaction groups, to achieve higher
availability.

Distributed program link (DPL) enables a CICS program (the client) to call
another CICS program (the server program) in a remote CICS region. Using
DPL, you can separate the end-user interface from the application business
logic, such as accessing and processing data, to enable parts of the applications
to be ported from host to workstation more readily. In many cases, DPL offers a
simple alternative to writing distribute transaction processing (DTP) applications.

The external CICS interface (EXCI) is an application programming interface (API)

that enables an MVS batch application (running in an MVS address space) to call
a CICS application (running in a CICS Transaction Server for OS/390 Release 1
address space) and to pass and receive data using a communications area. The
CICS program is invoked as if linked to by another CICS program. You can think
of the external CICS interface as a specialized form of distributed program link.

CICS support for DCE remote procedure calls (RPCs) enables a non-CICS client
program running in an open systems Distributed Computing Environment (DCE)
to call a server program running in a CICS Transaction Server for OS/390
Release 1 system, and to pass and receive data using a communications area.
The CICS program is invoked as if linked to by another CICS program.

9.2.3 Design for Availability

In section 9.2.2, “Facilities Exploiting CICS Intercommunication” on page 166, we
reviewed the independent virtues of the different intercommunication facilities.
Now we see how they could be used together to design a CICS system that
provides an application with an HA/CO environment.

The functional split, supported by TR, FS and AP allows the isolation of functions
into TOR, AOR, FOR, QOR, and DOR, and provides:
• Stable TORs, FORs, QORs and DORs containing no user code
• Very fast restartable AORs without logs and independent of the TOR, which
takes much longer to restart
• Stable DORs, FORs, QORs handling DASD, buffers, and logs for increased
data integrity ans availability

Chapter 9. CICS 167

 The logical split uses FS, DTP, DPL, EXCI, and DCE RPC. It is based on the
aphorism that one must not put all one′s eggs in one basket.
• The logical split into multiple TORs and distribution of TORs over multiple
MVS images increases availability, since only portions of the network are
impacted in the failure and recovery will be faster. You can further increase
the availability of TORs by implementing Generic Resources and VTAM
Persistent Session as described in 9.4.3, “VTAM Generic Resource Support”
on page 173.
• The logical split into AORs allows a modular design of transactions into
transaction groups. This increases system availability by keeping a large
subset of the functions available in the event of a failure of one transaction
group. Additionally, you can increase the availability of AORs by cloning the
AOR regions and using dynamic transaction routing and balancing to route
the transaction to another CICS region in a failure case as described in 9.3,
“Dynamic Transaction Routing and Balancing in a CICSplex.”
• The logical split into FORs, QORs, and DORs allows the separation of the
different data management systems like VSAM, IMS, DB2, and data queues
for reasons similar to the AOR split. By implementing data sharing in a
Parallel Sysplex, you are able to clone the CICS regions and provide an
alternate access path to the data.
For more information on the following subjects, refer to the appropriate
sections:
− For VSAM Record Level Sharing, see 9.4.1, “VSAM Record Level
Sharing” on page 171.
− For Shared Data Queues, see 9.4.2, “Temporary Storage Data Sharing”
on page 173.
− For IMS data sharing, see 10.4.1, “Data Sharing in Parallel Sysplex” on
page 190.
− For DB2 data sharing, see 11.3, “DB2 Data Sharing” on page 203.

9.3 Dynamic Transaction Routing and Balancing in a CICSplex

The usage of dynamic transaction routing and balancing between TORs and
AORs can improve the availability of CICS transactions from an end-user point of
view. CICSPlex/SM gets system health information from OS/390 WLM that it can
use to make the best routing decisions.

9.3.1 Dynamic Transaction Routing

As described in 9.2.2, “Facilities Exploiting CICS Intercommunication” on
page 166, transaction routing is an intercommunication facility that allows
terminals or logical units connected to one CICS region (TOR) to route
transactions in another CICS region (AOR). To initiate TR, you must define your
transaction as remote. CICS supports two forms of transaction routing for
remote transactions, static and dynamic.

With static transaction routing , the transaction is predefined with the name of a
remote CICS region to which it is to be routed for execution.

With dynamic transaction routing , one of the remote attributes on the transaction
resource definition specifies DYNAMIC(YES), indicating that the transaction to be

168 Continuous Availability S/390 Technology Guide

 routed dynamically. In this case, the name of the remote destination is
determined by a user-written dynamic transaction routing program at the time it
is invoked from a user′s terminal. CICSPlex/SM includes the dynamic
transaction routing program. See 9.3.3, “CICSPlex/SM” on page 170 for more
information about CICSPlex/SM. Using dynamic transaction routing across a
CICSplex, compared with static routing, can:
• Improve performance because of increased throughput and improved
response times
• Improve availability
• Simplify systems management

Improved availability can be achieved in the form of both high availability and
continuous operations:
With multiple AORs available to choose from, the dynamic transaction
routing program bypass any failed or “sick” AOR. Faster restart is possible
for smaller AORs, and fewer end users are affected by any single failure. All
these factors contribute to:
• Reducing the number of failures seen by end users
• Reducing the length of outage as seen by end users
• Reducing the incidence of “sympathy sickness” between CICS regions
Dynamic transaction routing helps you to achieve continuous operations by
allowing you to:
• Quiesce an AOR and direct transactions to an alternate AOR
• Stop and restart an AOR to apply service changes to the AOR when no
transactions are active
• Add an AOR clone and notify the dynamic transaction routing program
that a new target AOR is available

9.3.2 Dynamic Transaction Balancing

When you split CICS regions into TOR and AOR combinations, you have to
address the task of how to spread the workload from one TOR to multiple AORs,
or from multiple TORs to multiple AORs. With static transaction routing, the
system designer has to try and design the CICSplex in such a way that, by
allocating applications or sets of transactions to specific AORs, the workload is
evenly distributed across the CICSplex. This form of static workload balancing is
not very efficient, and does not allow peaks and troughs within the workload. It
cannot cater to those situations when some regions come under stress caused
by peak loads.

To implement dynamic workload balancing, you need a dynamic routing program

that makes the routing decision based not only on defined CICS systems; you
need a program that gets the information it needs about the actual workload
situation in the defined CICS regions. With workload balancing, you create a full
dynamical transaction routing environment, including the ability to monitor and
manage the state of the CICSplex.

CICS 4.1 allows a dynamic transaction routing program to track CICS-initiated

transactions, and transactions that abend an application-owning region (AOR).
This function can be used in your own transaction routing programs and is
exploited by workload balancing routines in CICSPlex SM Release 1.

Chapter 9. CICS 169

9.3.3 CICSPlex/SM
IBM CICSPlex System Manager for MVS/ESA (CICSPlex SM) is a
system-management tool that enables you to manage multiple CICS systems as
if they were a single system. Enterprises in which a CICS system-management
tool may be needed range from those running 10 or 15 CICS systems, to those
running two or three hundred (or more) CICS systems: in the latest MVS sysplex
environment, having such a large number of CICS systems to support a
transaction-processing workload is becoming increasingly common.

CICSPlex SM′s workload management function provides a dynamic transaction

routing (DTR) program that can route eligible transactions (those defined as
dynamic) from a terminal-owning region (TOR) to a suitable application-owning
region (AOR) selected at the time the transaction is initiated.

CICSPlex SM′s DTR program supports both workload separation and workload
balancing:
Workload separation The routing of particular transactions to an particular
group of AORs based on any combination of user ID,
terminal ID, and transaction name. For example, using
CICSPlex SM′s workload separation function, you can
specify that transactions beginning where the characters
SAL and initiated by members of your sales department
must be routed to the group of AORs, SALESGRP,
allocated to that department.
Workload balancing The routing of transactions among a group of AORs
according to the availability and activity levels of those
AORs. Workload balancing can be used in addition to,
or in place of, workload separation. For example,
CICSPlex SM can balance the transaction workload
among the SALESGRP AORs by selecting, as each
transaction is initiated, the AOR that is likely to deliver
the best performance.

CICSPlex SM′s workload management function is of particular benefit in those

enterprises that are running CICS/ESA on Parallel Transaction Servers (PTSs),
because CICSPlex SM can route transactions throughout the sysplex. Figure 52
on page 171 shows you how CICSPlex/SM interacts in a CICSplex environment.

170 Continuous Availability S/390 Technology Guide

 Figure 52. CICSPlex/SM Interaction in a CICSplex

The benefit of CICSPlex SM is that it can balance the enterprise workload

dynamically across the available AORs, thereby enabling you to manage a
variable workload without operator intervention, while maintaining consistent
response times. CICSPlex SM routes transactions away from busy regions and
from those are failing or likely to fail, giving improved throughput and concealing
problems from end users. Region failures that themselves result from temporary
resource constraints are thereby much reduced in both number and frequency.
Furthermore, planned service is made much easier: you can remove a CICS
region for maintenance without having to worry about tha region′s workload,
because CICSPlex SM can simply route the work dynamically to another region.

9.4 CICS Parallel Sysplex Exploitation

In this section we describe MVS sysplex exploitation in CICS within the following
areas:
• VSAM Record Level Sharing
• Temporary Storage Data Sharing
• VTAM Generic Resource Support
• VTAM Persistent Session Support

9.4.1 VSAM Record Level Sharing

VSAM record-level sharing (RLS) is a VSAM function available with DFSMS
Version 1 Release 3 and exploited by CICS Transaction Server for OS/390. RLS
enables you to share VSAM data with full update capability between many
applications running in different CICS regions, possibly on several different MVS
images within a single MVS sysplex. RLS also provides some benefits when
data sets are being shared between CICS regions and batch jobs.

DFSMS 1.3 supports a new data-sharing subsystem, SMSVSAM, which runs in its
address space to provide the RLS support required by batch jobs and AORs or
cloned FORs within each MVS image in a Parallel Sysplex environment. An
SMSVSAM server, used by the resident AORs, is shown in each of the MVS
images in Figure 53 on page 172. The SMSVSAM server, which is generally

Chapter 9. CICS 171

 initialized automatically during MVS IPL, uses the coupling facility for its cache
structures and lock structures.

Figure 53. Conceptual View of Parallel Sysplex with RLS

In earlier CICS releases, VSAM recovery attributes for a file are defined on the
file′s resource definition in the CSD. To support RLS access, VSAM supports
some new data set attributes like “recoverable” and “backup-while-open
(BWO).” These attributes are stored in the ICF catalog.

A recoverable data set participates in syncpoint activity, which is eligible for

changes to be backed out in the event of a transaction failure. When several
changes are made to recoverable data sets from within a unit of work (UOW),
these changes are either all committed together, or all backed out together.

RLS provides many benefits to CICS applications, and improves availability in a

number of ways:
• It eliminates the FOR as a single point of failure. With an SMSVSAM in each
MVS image in the sysplex, work can be dynamically routed to another MVS
image in the event of a system failure.
• It does not take a data set offline in the event of a backout failure. If a
backout failure occurs, only the records affected within the unit of work
remain locked; the data set remains online.
• It reduces the scope for deadlocks. Locks are applied to records and not to
an entire control interval, so records are more readily available for VSAM
requests.
• It participates in the new in-doubt support provided by the CICS recovery
manager.
• It provides support for off-site recovery products in a special form of
emergency restart that preserves data integrity for RLS-accessed data sets.
• It improves sharing between CICS and batch, so the batch window can be
shorter. Concurrently with CICS, batch jobs can read (but not update),
recoverable data sets that are opened by CICS in RLS mode.

172 Continuous Availability S/390 Technology Guide

9.4.2 Temporary Storage Data Sharing
The CICS temporary storage data sharing facility, introduced with CICS
Transaction Server, supports the Parallel Sysplex environment by providing
shared access to CICS temporary storage queues. This enables you to share
temporary storage queues between many applications running in different CICS
regions across a sysplex. CICS uses temporary storage data sharing to store
shared queues in a structure in a coupling facility, access to which is provide by
a CICS temporary storage server address space. Figure 54 shows you a
configuration with shared temporary storage.

Figure 54. Shared Message Queue Environment

The benefits of TS data sharing are:

• Improved performance compared with using remote queue-owning region.
Access to queues stored in the coupling facility is quicker than function
shipping to a QOR.
• Improved availability compared with a QOR. The availability of a TS server
is better than a QOR because you can have more than one TS server for
each pool (typically one server in each MVS image in the sysplex). If one TS
server or MVS image fails, transactions can be dynamically routed to
another AOR on a different MVS image and can use the TS server in that
MVS image.
• Elimination of intertransaction affinities, because different transactions use
the same local storage queue.

9.4.3 VTAM Generic Resource Support

VTAM Generic Resource Support is introduced in CICS 4.1. A VTAM application
program such as CICS can now be known by its generic resource name, in
addition to its own application program network name defined on the APPL
definition statement. A number of CICS regions can use the same generic
resource name. VTAM keeps a map of the CICS APPL that are members of a
generic resource name.

Chapter 9. CICS 173

 Each CICS region that uses a given generic resource name is a member of that
generic resource name set. A terminal user wishing to start a session with a
CICSplex that has several terminal-owning regions can use the generic resource
name in the logon request. Using the generic resource name, VTAM establishes
the session with one of the member CICS regions that is registered as a
member of the generic resource name. Figure 55 shows you an example of how
a CICS TOR is selected.

Figure 55. VTAM Generic Resource Support in CICS

The main benefits accrue when VTAM generic resource registration is applied to
CICS terminal owning regions. This enables VTAM to perform dynamic workload
balancing of the terminal sessions across the available TORs. If a TOR is not
available for any reasons, VTAM does not try to establish a session to that TOR.

9.4.4 VTAM Persistent Session Support

Persistent session support is introduced in CICS 4.1 and improves the availability
of CICS. It benefits from VTAM 3.4.1 persistent LU-LU session improvements to
provide restart-in-place of a failed CICS without rebinding.

CICS support of persistent sessions includes the support of all LU-LU sessions
except LU0 pipeline and LU6.1 sessions. CICS determines for how long the
sessions should be retained from the PSDINT system initialization parameter.
This is a user-defined time interval. If a failed CICS is restarted within this time,
it can use the retained sessions immediately; there is no need for network flows
to rebind them.

The end user of a terminal sees different symptoms of a CICS failure following a
restart, depending on whether VTAM persistent sessions, or XRF, are in use:
• If CICS is running without VTAM persistent sessions or XRF and it fails the
user sees the VTAM logon panel followed by the “good morning” message
(if AUTOCONNECT(YES) is specified for the TYPETERM resource definition).
• If CICS does have persistent session support and fails, the user perception is
that CICS is hanging and what is on the screen at the time of the failure
remains until persistent session recovery is complete.

174 Continuous Availability S/390 Technology Guide

9.5 CICS Extended Recovery Facility (XRF)
The CICS extended recovery facility (XRF) provides a high level of availability to
its end users by employing a degree of automated takeover from an active CICS
region to an alternate CICS region, which can then effect a rapid emergency
restart. When the active region fails, it has to be taken off-line for maintenance
or moved to another MVS image. The alternate CICS is a partially initialized
duplicate of the active, which cannot do any normal processing, but does not
take up much storage or Central Processor Complex (CPC) time. Figure 56
shows a CICS XRF configuration.

Figure 56. CICS Recovery Domain-CICS XRF Configuration

While the active CICS is working, the alternate system is monitoring it for
failures and can detect failures more quickly than an operator. When the
alternate has detected a failure, the automatic takeover time is faster than an
ordinary restart, because the system is already partly initialized.

If you do not use dynamic transaction routing, you cannot get all the availability
benefits of that facility. Implementing XRF may be an alternative if you
nevertheless want high availability.

9.6 Improved CICS Architecture

CICS/ESA introduced a new CICS architecture in which the CICS system is split
into functional parts or domains. A domain is the encapsulation of function code
and data. This architecture, with its strictly defined interfaces between domains,
allows a domain to be completely rewritten, or a new domain to be added,
without any interference with other domains. This re-engineering improves CICS
integrity and serviceability for higher availability.

Chapter 9. CICS 175

9.6.1 New Domains introduced in CICS/ESA 4.1
There are five new domains introduced in CICS/ESA 4.1:
• Directory manager domain
The directory manager domain provides resource-table lookup services for
the CICS/ESA 3.3 components that are now restructured into new domains
such as transaction manager, program manager, and user domains. This
provides internal services to other domains with improved performance
compared with DFHTMP. Like all the restructured components of CICS, it
also provides higher reliability and serviceability.
• Program manager domain
The program manager domain provides support for program control,
transaction abend and condition handling functions, and the Autoinstall for
programs, mapsets, and partitionsets. The main benefits are in the
autoinstall facility for programs, mapsets, and partitionsets as described in
9.7, “CICS Resource Definition” on page 178.
• Transaction manager domain
The transaction manager domain provides transaction-related services like
creating and terminating tasks, and managing transaction definitions
transaction classes. The new transaction manager domain is designed to
provide greater reliability and improved function; it has minimal impact on
end users.
• Security manager domain
There are two aspects to security processing in releases of CICS before
CICS/ESA 4.1: identifing terminal users of the CICS system and permitting
them to sign on to CICS. These are stored in the signon table (SNT).
The restructure of these two functional areas of CICS creates two
independent domains: one encapsulates users′ capabilities or authorities
(the security manager domain), and the other to encapsulates users ′
attributes (the user domain).
This approach decouples security checks from the terminal to enable CICS to
support non-terminal security and to provide security checks for the
introduction of work for new user IDs without introducing a non-password
signon function.
• User domain
The user domain is responsible for identifying users and recording their
non-security attributes. It supplies a “user token” to represent each user
that signs on. By implementing the user domain, it is now possible to
control multiple signons by the same user ID or to allow a RACF group name
at signon.

9.6.2 New Domains introduced in CICS TS

These new domains are introduced in the CICS Transaction Server:
• Temporary storage domain
The temporary storage (TS) domain is restructured to conform to the CICS
domain architecture. It provides the addition of the required interfaces to the
recovery manager domain. Significant performance improvements are made
for both main and auxiliary requests, and emergency restart time has been
reduced by eliminating the need to scan the TS dataset.

176 Continuous Availability S/390 Technology Guide

 • Recovery manager domain
The recovery manager is a new domain of this CICS Transaction Server for
OS/390. Its purpose is to ensure the integrity and consistency of resources,
such as files and databases, both within a single system an distributed over
interconnected systems in a network. Figure 57 shows you how the
recovery manager domain interacts with the other CICS domains.

Figure 57. CICS Recovery Domain

CICS has always allowed you to create transactions with ACID properties (4)
(atomicity, consistency, isolation, and durability). In particular, the atomic
property of a CICS transaction means that any changes the transaction
makes to recoverable resources (such as files, temporary storage, or DB2)
are carried out in such a way that either all changes happen or none
happen.
However, in previous CICS releases, restrictions were imposed on
distributed units of work; that is, on units of work that update resources on
two or more systems. These restrictions were necessary because of the
difficulties of coordinating updates to distributed resources.
In CICS Transaction Server for OS/390, the CICS recovery manager allows
you to distribute recoverable resources in a network while ensuring that all
updates made by a distributed unit of work are coordinated, thus preserving
the transaction′s atomicity.
• Log manager domain
The CICS log manager is a new domain that replaces the journal control
management function of earlier releases. It works with the MVS system
logger, which is introduced in MVS/ESA 5.2, to provide a focal point for all
system log, forward recovery log, and user journal output within an single
MVS sysplex. It provides flexible mapping of CICS journals onto MVS log
streams, which you can use, for example, to merge all the forward recovery
logs for a given VSAM data set from many CICS systems on several MVS
images.

Chapter 9. CICS 177

 The MVS system logger also enables faster CICS restart, dual logging, and
flexible log, and journal archiving. Figure 58 on page 178 shows you how
the logging environment looks now.

Figure 58. CICS Logging Environment

9.7 CICS Resource Definition

Your CICS system has to know which resources to use, what their properties
are, and how they are to interact with each other. You supply this information to
CICS by using one or more of the four methods of resource definition that are to
a lesser or greater degree:
• Macro definition: You can use assembler macro source code to define
resources. Definitions are stored in assembled tables in a program library,
from where they are installed during CICS initialization. To make changes
active, you have to stop and start the CICS region. This is the “old” method
and will be replaced by the other methods.
• DFHCSDUP offline utility: This method stores definitions in the CSD.
DFHCSDUP allows you to make changes to definitions in the CICS system
definition file (CSD) by means of a batch job submitted offline.
• Resource definition online (RDO): This method uses CICS-supplied online
transactions. Definitions are stored in the CSD, and are installed into an
active CICS from the CSD.
• Automatic installation (autoinstall): Autoinstall minimizes the need for a large
number of definitions by dynamically creating new definitions based on a
“model” definition provided by you.

178 Continuous Availability S/390 Technology Guide

9.7.1 Resource Definition Online (RDO)
RDO for transactions, programs, map sets, terminals, MRO and ISC links, and
system definitions are introduced in Version 3. CICS TS introduces RDO for the
destination control table (DCT). Using RDO, transient data queues can be
defined without having to bring down the CICS system. This improves
availability because of fewer scheduled outages.

9.7.2 Autoinstall
Before the introduction of autoinstall, all resources had to be defined individually
to CICS before they could be used. With the introduction of autoinstall for
terminals in CICS/OS/VS 1.7, VTAM terminals could be defined dynamically on
their first usage, thereby saving on storage for terminal entries and saving on
time spent creating the definitions (and possibily making definiton failures).

The autoinstall function is extended in CICS/ESA 4.1 to include user programs,

map sets, and partition sets, and LU6.2 parallel sessions. This extends the CICS
autoinstall facility to enable CICS to install programs, map sets, partition sets,
and LU6.2 parallel sessions without prior definition of the resource. This
autoinstall facility is similar to that already supported for terminals.

The major availability benefit of autoinstall is the elimination of the planned

outages necessary to maintain the definitions. If autoinstall is not used, the CICS
system has to be taken off-line and cold started, just to remove unwanted
definitions.

9.8 Other CICS Continuous Availability Features

In this section, we discuss other important continuous availability functions
introduced or enhanced in CICS V4.1 and CICS TS. There are also many other
features and enhancements regarding this release; you can find more
information in CICS/ESA Release Guide and CICS Transaction Server for OS/390
Release Guide .

9.8.1 Transaction Isolation

CICS storage protection, introduced in CICS/ESA 3.3, uses the extension MVS
key-controlled storage protection that is provided by the subsystem storage
protection facility. CICS storage protection is controlled by programs being
defined to run in either user key or CICS key.

Transaction isolation extends storage protection by offering storage protection

between transactions, thereby ensuring that a program of one transaction does
not accidentally overwrite the storage of another transaction. Accidental
overwrites of the transaction data of an application program can affect the
reliability and availability of a CICS system, and the integrity of data in the
system.

To provide protection for the transaction data of application programs, the CICS
transaction isolation facility requires hardware and MVS facilities that support
subspace groups. For more information about subspace groups, see 2.2.3.12,
“Subspace Group Facility” on page 12.

Transaction isolation prevents unplanned CICS system outages caused by

coding errors in user-key application programs that cause the storage user-key

Chapter 9. CICS 179

 transactions to be accidentally overwritten. Prevention of accidental transaction
data overwrites significantly improves the reliability and availability of CICS
regions.

It is also important to protect the application data. If an application program

overwrites another application program′s code, then that other application
program is likely to fail. While this is a serious interruption in a production
region, the effect is immediate and the program can generally be recovered so
that the terminal user can retry the failed transaction. However, if an application
program overwrites the data of another application program, the results often
are not immediately apparent; the erroneous data can be written to a database
and the error can remain undetected until later, when it may be impossible to
determine the cause of the error. The consequences of a data overwrite are
often much more serious than a code overwrite.

9.8.2 ARM Support

CICS supports the automatic restart manager (ARM) component of MVS to
increase the availability of your systems. The main benefits of the MVS
automatic restart manager are that it:
• Enables CICS to preserve data integrity automatically in the event of any
system failure.
• Eliminates the need for operator-initiated restarts, or restarts by other
automatic packages, thereby:
− Improving emergency restart times
− Reducing errors
− Reducing complexity
• Provides cross-system restart capability. It ensures that the workload is
restarted on MVS images with spare capacity, by working with the MVS
workload manager.
• Allows all elements within a restart group to be restarted in parallel. Restart
levels (using the ARM WAITPRED protocol) ensure the correct starting
sequence of dependent or related subsystems.

MVS automatic restart management is available only to those MVS subsystem

that register with ARM. CICS regions register with ARM automatically as part of
CICS system initialization. If a CICS region fails before it has registered for the
first time with ARM, it will not be restarted. After a CICS region has registered,
it is restarted by ARM according to a predefined policy for the workload.

You cannot use MVS automatic restart for CICS regions running with XRF; it is
available only to non-XRF CICS regions. If you specify XRF=YES, CICS
deregisters from ARM and continues initialization with XRF support.

9.8.3 DBCTL as Interface to IMS

IMS ESA 3.1 introduced DBCTL as an interface from CICS to IMS. The
restructuring of the DL/I subsystem (which was buried in the CICS address
space) into a separate subsystem called Data Base Control (DBCTL) and
functioning as a database resource manager, provides CICS with important
availability attributes that were previously available only to IMS DC
environments.

180 Continuous Availability S/390 Technology Guide

 • DBCTL supports shared access to an IMS database from multiple CICS
systems, which with IMS/ESA Data Management Version 5 Release 1 can
reside on different MVS images within a single MVS sysplex.
• DBCTL offers increased data availability, with the ability to schedule PSBs
even when some databases are not available.
• Using batch message processing programs (BMPs), you can share DBCTL
databases between CICS and batch programs, providing a performance
advantage.
• CICS applications may use IMS system service requests, in addition those
related to data availability.
• DBCTL offers access to data entry databases (DEDBs). DEDBs are highly
available, may be very large, are flexible in design, and provide improved
performance.
• Removal of local DL/I means that CICS is supplied completely pregenerated,
thus simplifying the install process. The removal of source-maintained CICS
DL/I modules also means that the process of applying corrective
maintenance is simplified.

The CICS-DBCTL interface is enhanced in CICS TS to use new RMI protocols.

The CICS-DBCTL task-related user exit is enabled with the new INDOUBTWAIT
keyword. At phase 2 syncpoint time, if CICS is in doubt about the outcome of the
unit of work, it invokes the exit program with the UERTWAIT verb. The exit
program issues a term thread call to DBCTL, which terminates the thread but
creates a recoverable in-doubt structure (RIS) for the unit of work and maintains
any locks acquired on behalf of the transaction.

9.8.4 DB2 Attachment Facility

CICS Version 4 includes an integrated DB2 adapter that was previously part of
the DB2 product known as the CICS attachment facility. The new integrated DB2
adapter, which ships with CICS Version 4, offers many improvements for making
the operations between DB2 and CICS easier:
• Reduced virtual storage constraints
• Improved trace information
• Better performance
• Better thread management

The new adapter must be used with CICS Version 4 and works with all current
releases of DB2. The CICS attachment facility from previous releases is still
delivered with DB2.

CICS TS improves the CICS-DB2 attachment. It can now remain active if DB2
fails, and automatically reconnect when DB2 recovers. Previously, if DB2 failed,
the CICS-DB2 attachment was stopped and had to be manually restarted when
recovered.

If the CICS-DB2 attachment is active but DB2 is not active (STRTWT=AUTO), the
attachment can return a SQL return code to an application that tried to issue a
SQL request, rather than to the application abending with abend code AEY9.

The CICS-DB2 attachment facility has been enhanced to support the new
INDOUBTWAIT RMI protocol. The CICS-DB2 task-related user exit (TRUE) is

Chapter 9. CICS 181

 enabled with the INDOUBTWAIT keyword and accepts the new UERTWAIT verb
when a UOW is shunted. In response to a UERTWAIT, the TRUE causes the
thread to be terminated but ensures that DB2 maintains its database locks for
the UOW, and also records that the UOW is indoubt.

The CICS-DB2 attachment has been enhanced to issue calls to the MVS
workload manager if a severe error occurs, or if the CICS region is active but not
connected to DB2, so that CICS-DB2 work is not routed to this CICS region.

9.8.5 CICS Interface to MQSeries

CICS recognizes MQSeries for MVS/ESA as a non-CICS resource and includes
MQSeries as an agent in any syncpoint coordination requests using the CICS
resource manager interface (RMI).

9.8.6 Internet Access to CICS Applications

The CICS Web Interface allows web browsers to call programs in a CICS system
using the hypertext transfer protocol (HTTP). Figure 59 shows shows the
relationship between the CICS Transaction Server, the CICS Web Interface, and
the Web browsers in the network.

Figure 59. The CICS Transaction Server and the Internet

The flexible use of CICS programs to supply transaction processing services to

different types of requestors depends on a two-tier programming model for CICS
applications. In a two-tier design, an application′s presentation logic is
separated from its business logic. The presentation logic is found in such
commands as EXEC CICS RECEIVE MAP and EXEC CICS SEND MAP, while the
business logic is found in commands like EXEC CICS READ FILE UPDATE and
EXEC CICS REWRITE FILE, and in the commands that modify data in databases.

By packaging the presentation logic and business logic in separate programs

and by using EXEC CICS LINK and a communication area interface to the
business logic, the business logic can be made available to a wide range of

182 Continuous Availability S/390 Technology Guide

 requestors. In the CICS Web Interface, the presentation logic is packaged in
user-replaceable programs called converters .

The benefit from an availability point of view is that you make your business
applications available through a new and additional access path.

9.8.7 Backup-While-Open (BWO)

Many CICS applications depend on their data sets being open for update over a
long period of time. Normally, you cannot take a backup of the data set while
the data set is open. Thus, if a failure occurs that requires forward recovery, all
updates that have been made to the data since it was opened must be
recovered. This means that you must keep all forward-recovery logs that have
been produced since the data set was opened. A heavily used data set that has
been open for update for several days or weeks may need much forward
recovery.

The BWO facility, together with other system facilities and products, allows you
to take a backup copy of a VSAM data set while it remains open for update.
Then, only the updates that have been made since the last backup copy was
taken need to be recovered. This could considerably reduce the amount of
forward recovery that is needed.

Chapter 9. CICS 183

184 Continuous Availability S/390 Technology Guide
Chapter 10. IMS

IMS has always been known for high availability. As the product has evolved, it
has become more reliable and more available for your end users. The time
needed to run batch jobs (the “batch window”) has been reduced because more
of these tasks can be done online.

10.1 Summary of IMS Continuous Availability Attributes

Table 9 shows you which availability attribute applies to frequency, duration, and
scope of an outage, and which outage type it belongs to. We distinguish between
planned (continuous operation) and unplanned (high availability) outages.

Table 9. IMS C/A Attribute Summary Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e Type, H A = U n p l a n n e d ,

CO=Planned, C A = B o t h
Availability Attribute Frequency Duration Scope Type
Concurrent Image Copy (CIC) X CO
DFSMS Concurrent Copy Support X CO
Database Recovery Control (DBRC) X CO
IMS/XRF X CA
Remote Site Recovery X CA
IMS Data Sharing X X X CA
Fast Database Recovery X HA
Common Queue Server - Shared Queues X X X CA
VTAM Generic Resource Support X CA
Workload Router X X CA
DEDB Online Change X CO
Extended Terminal Option (ETO) X CA
A R M Support X CO
Distributed Sync Point X X HA
Daylight Saving Time X CO
IMS WWW Connection X HA

10.2 IMS Full Function and Fast Path Database

In this section we briefly describe the components of an IMS environment. The
objective is to help you to better understand the IMS availability attributes that
are reviewed in later sections.

IMS supports four types of application programs:

1. Online message processing programs (MPPs) process messages to and from
IMS message queues.
2. Interactive fast path programs (IFPs), which are similar to MPPs, quickly
process and reply to messages from terminals.

 Copyright IBM Corp. 1998 185

 3. Batch Message Processing Programs (BMPs) perform message processing
or batch processing on-line. Unlike MPPs, BMPs can access non-IMS
database files, in addition to IMS data bases.
4. Batch programs.

IMS Transaction Manager supports four types of databases:

1. Full Function DataBase
The data in a full function database is grouped into a series of data base
records. Each record is composed of smaller groups of data called
segments . Segments are made up of one or more fields .
2. Data Entry Database (DEDB)
A fast path DEDB database can be partitioned into areas . Each area contains
a different collection of database records and is independent of the other.
3. Main Storage Database (MSDB)
A main storage database stores and provides access to the most frequently
used data. MSDBs reside in virtual storage during execution.
4. DB2 Tables

10.3 IMS Recovery Features

IMS is designed to be recoverable . Recovery is concerned with the availability
and correctness of the work the system handles. More specifically, recovery
involves:
• Databases
• User requests to process data
• Programs that do the processing (application programs)
• Output sent back to the users

A recoverable system ensures two kinds of safety:

• The data is not lost
Without recovery features, data in a system can be physically lost; the disk
on which it resides can be damaged or misappropriated.
• Incomplete changes to a database are not saved
Without recovery features, data in a system can be “logically” lost, which
means the data becomes incorrect or loses its relationship to other data.
A system that ensures data integrity ensures that data cannot be lost or saved
incompletely. IMS has a number of mechanisms to help you in these areas.
Some of these function automatically while others require your involvement.

10.3.1 Synchronization Points and Backup Copies

In IMS there are two types of sync point. IMS itself creates periodic sync points
(called system checkpoints), in which it records the current status of the system.
In addition, applications can cause sync points. These are points at which the
work done by the application is assumed to be complete.

186 Continuous Availability S/390 Technology Guide

 You can also make backup copies of databases to serve a function similar to
sync points. The backup copies are image copies of your data. If, for instance,
a batch application fails, you can undo all database changes done by the failed
application and restart processing. Similarly, you can have IMS make backup
copies of the message queue data sets, which contain the input and output
messages.

10.3.1.1 Concurrent Image Copy

Concurrent image copy (CIC) is a batch utility, delivered with IMS 4.1, which
allows you to make an image copy of OSAM data sets and VSAM ESDS DBDSs,
whether or not IMS is running and the database is online. This is a significant
advantage over prior IMS releases because it increases database availability
and helps reduce the need for stand-alone batch image copies.

10.3.1.2 DFSMS Concurrent Copy Support

IMS 6.1 provides DBRC and utility support for image copy processing when
accessing the concurrent copy feature of Data Facility Storage Management
Subsystem (DFSMS). Using the DFSMS concurrent copy feature, you can take a
fast copy of a data set with a very short downtime.

The Database Image Copy 2 utility (DFSUDMT0) performs the tasks required to
take image copies of IMS databases. The databases must therefore registered
in the DBRC.

10.3.2 Logging
In the online environment, IMS keeps a log called the online log data set (OLDS).
The OLDS is on a direct access storage device (DASD) and is later written to
another data set called the system log data set (SLDS). This SLDS can be on
DASD, tape, or mass storage. In the batch environment, log records are written
directly to an SLDS. IMS also keeps another log called the restart data set
(RDS). The RDS contains information needed for restart. The writing of logs is
handled totally and automatically by IMS.

10.3.3 Recovery
There are two major forms of recovery: one involves reconstruction of
information (called forward recovery) and the other involves removal of bad or
incomplete information (called backout or backward recovery). Wit forward
recovery, you reconstruct information or work that has been lost. With backward
recovery or backout, you remove incorrect or unwanted changes from
information.

In IMS, you perform the task of forward recovery (although IMS does supply
utilities to help).

Backward recovery, or backout of transactions and the associated database data

sets (DBDSs), is usually handled automatically by IMS. Backout usually does not
require your involvement. Backout of batch programs can be done either
automatically or by using a utility.

In order to get maximium availability, you need to develop a plan for backup and
recovery that minimizes the duration of the outage because data are not
available. You must decide how to use the tools that IMS provides for operating
your system. This includes choosing, for example, whether to use dual logging

Chapter 10. IMS 187

 when setting up your log, how often to make backup copies of your database,
and whether to use DBRC to control recovery of databases.

10.3.4 Database Recovery Control

The IMS DBRC facility extends the capabilities of IMS utilities for database
recovery to allow easier recovery of IMS databases including DL/I databases,
data entry databases (DEDBs), and databases accessed through CICS. DBRC
offers three levels of control:
1. Log control to control the use and reuse of OLDSs for IMS.
2. Recovery control, in addition to log control, to record recovery-related
information in the Recovery Control (RECON) data set. It also assists in
recovering databases and generates job control language (JCL) for
recovery-related utilities.
3. Share control, in addition to log control and recover control, in which DBRC
registers and controls the scheduling of databases.

10.3.5 Disaster Recovery

In an IMS environment, you have at least two possible ways to prevent an
outage in case of a disaster: by using the Extended Recovery Facility (XRF) or
the Remote Site Recovery (RSR).

10.3.5.1 Extended Recovery Facility (XRF)

The prime objective of Extended Recovery Facility (XRF) is to significantly reduce
the impact of both planned and unplanned outages in order to get service back
as quickly as possible, as in the scennerio shown in Figure 60.

Figure 60. Catastrophic Disaster Scenario 1

IMS supports XRF running under MVS/ESA to provide an alternate IMS

subsystem that monitors the log data of the active IMS subsystem, and takes
over its processing load if the active subsystem, or certain dependent service
elements of the active subsystem, experiences a failure. Everything about

188 Continuous Availability S/390 Technology Guide

non-XRF recovery mechanisms is also true in an XRF complex, but with XRF,
IMS can do more to reduce the time that data is unavailable to users after an
abnormal event occurs.

10.3.5.2 Remote Site Recovery

The Remote Site Recovery (RSR) feature for IMS Version 5 lets you recover
quickly from an interruption of computer services at an active site, as shown in
Figure 61.

Figure 61. Catastrophic Disaster Scenario 2

RSR supports recovery of IMS resources at a remote site. IMS DB full-function

databases and fast path DEDBs can be kept up-to-date by tracking them while
the remote site is in tracking mode. Otherwise, they can be recovered via
normal customer procedures for database recovery at a later time. IMS TM
message queues and the IMS TM network can be recovered by using the /ERE
BUILDQ command when the remote site becomes the active site (at remote
takeover time).

RSR has these recovery features:

• It provides a remote copy of the necessary IMS DB and IMS TM log records
for database and message queue recovery at the tracking site.
• It supports these DL/I database access methods: HDAM, HIDAM, HISAM, and
SHISAM, and supports fast path DEDBs.
• It supports IMS DB, IMS DBCTL, and batch workloads.
• It maintains remote copies of full-function databases and fast path DEDBs.
• It recognizes that IMS database recovery control (DBRC) is operating at the
active site and, separately, at the tracking site.
• It supports data sharing at the active site.
• It coexists with the IMS extended recovery facility (XRF).
• It lets you filter out log records that are not needed to support the defined
critical environment. An added benefit of this is reduced line traffic.
• It continues to operate when the active or tracking sites, or the RSR
transmission facility, become temporarily unavailable, and provides a way to
resynchronize the sites.
• It provides transaction consistency between the active and tracking sites.

Chapter 10. IMS 189

 • It supports standard VTAM communication protocols, so new technology is
not required for data transmission.

10.4 IMS Parallel Sysplex Exploitation

The following functions were introduced in IMS 5.1 and 6.1 to support the Parallel
Sysplex environment:
• Data sharing of full function databases (5.1)
• Data sharing of DEDB (6.1)
• Fast database recovery
• Common queue server
• VTAM Generic Resource Support

10.4.1 Data Sharing in Parallel Sysplex

Data sharing support makes it possible for application programs in separate IMS
systems to have concurrent access to databases. To ensure that database
changes at the segment level originating from one program are fully committed
before other programs can access that segment′s data, IMS systems use lock
management.

IMS DB Version 5 was the first to support sysplex data sharing. This means that
multiple IMS systems (up to 32) can now share data across multiple MVS images
(also up to 32). Before this enhancement, you could only share data across two
MVS images.

10.4.1.1 Data Sharing of Full Function Data Bases

Figure 62 on page 191 shows you the major components of a IMS data sharing
environment. IRLM is needed to manage the global locking between the IMS
systems. A set of DBRC RECON data sets and a lock and a cache structure are
also shared within the data sharing group.

190 Continuous Availability S/390 Technology Guide

Figure 62. IMS Data Sharing Configuration

10.4.1.2 Data Sharing of DEDB

In IMS Version 5, if multiple IMS subsystems in a sysplex environment wish to
access data from a single DEDB, only database-level sharing is allowed This
means that no more than one IMS subsystem can read and update the DEDB
data at one time.

Block-level sharing of VSO DEDB areas in IMS Version 6 allows multiple IMS
subsystems to concurrently read and update VSO DEDB data. The three main
participants are the coupling facility hardware, the coupling facility policy
software, and the XES and MVS services. Figure 63 on page 192 shows the
major components of this environment.

Chapter 10. IMS 191

 Figure 63. IMS Fast Path DEDB Configuration

10.4.1.3 Fast Database Recovery

If you have a sysplex data sharing environment that multiple IMS subsystems
access, and one of the IMS subsystems fails while it has a lock on the database,
the other IMS subsystems must wait until the failed IMS is restarted and the
locks on the resource are released. Because an emergency restart can take a
significant amount of time, waiting for a full restart is unacceptable in situations
that require continuous availability of database resources. Figure 64 shows you
the scenario just described.

Figure 64. IMS Fast Database Recovery Environment

192 Continuous Availability S/390 Technology Guide

 The Fast DB Recovery solution creates a separate IMS control region (the Fast
DB Recovery region) that monitors an IMS subsystem, detects failure, and
recovers any database resources that are locked by the failed IMS, making them
available for other IMS subsystems. In this way, it provides a quick access to
shared database resources in a sysplex environment that might otherwise be
locked by a failed IMS until the failed system is restarted.

10.4.2 Common Queue Server

Common Queue Server (CQS) is a new generalized queueing facility that
shipped with IMS Version 6. It manages shared message queues in a sysplex
environment on behalf of multiple clients. CQS receives, maintains, and
distributes data objects from shared queues on behalf of its clients. Each client
has its own CQS to access the shared queues. CQS runs in a separate address
space that is started by its client, as shown in Figure 65.

CQS uses the MVS coupling facility as a repository for the data object. The
coupling facility stores and arranges the data according to list structures. The
shared queues are kept in a primary list structure and optionally, in an overflow
list structure. The two list structures are known as a structure pair .

Figure 65. IMS Shared Message Queue Configuration

CQS maintains the following components for each structure pair it manages:
• A primary structure that contains the shared message queue.
• An overflow structure (optionally).
• An MVS log stream that contains all CQS log records from all CQSs
connected to a structure pair. This log stream is important for recovery of
shared queues, if necessary. There is one log stream for each structure
pair.
• A checkpoint data set that contains CQS system checkpoint information.

Chapter 10. IMS 193

 • Structure recovery data sets (SRDS) that contain structure checkpoint
information for shared queues on a structure pair. There are two SRDSs for
each structure pair.

Using shared queues enables automatic workload balancing across all IMS
subsystems in the sysplex. Also, new IMS subsystems can be added to the
sysplex, and share the queues as workload increases, thus resulting in
incremental growth. And using shared queues provides increased reliability and
failure isolation: if one IMS subsystem in the sysplex fails, any of the remaining
IMS subsystems can process the work that is waiting in the shared queues.

10.4.3 Generic Resource

The IMS Generic Resources enhancement complements a VTAM V4R2 function
of the same name. Together they enable VTAM to automatically distribute
terminal sessions among a cooperative set of IMS systems known as a generic
resource group .

Distributing sessions among the IMS systems in a generic resource group has
the following advantages:
• It avoids bottlenecks within any one IMS system,
• It uses the available capacity in other systems to share the work.
• It provides increased availability by allowing sessions to be re-established
with another IMS, upon the failure of an IMS image.
Previously, in order to balance sessions across IMS systems, you had to devise
a means of assigning terminals to specific systems. Now VTAM performs that
task for you.

10.5 Other IMS Continuous Availability Features

This section describes additional IMS Continuous Availability Features, including
DEDB Online Change, the Workload Router, Database Partitioning, Extended
Terminal Option (ETO), ARM Support, Distributed Sync Point using OS/390 RRS,
Daylight Saving Time, and IMS WWW Connection.

10.5.1 Fast Path DEDB Online Change

The DEDB Online Change enhancement is an extension of the IMS Online
Change facility. This function allows you to add, change, and delete fast path
data entry databases (DEDBs) at the database or area level without shutting
down and restarting IMS, as shown in Figure 66 on page 195. The IFP or MPP
regions that access the DEDBs need not be stopped during online change
processing. DEDB online change can be used in XRF, RSR, and sysplex
environments.

194 Continuous Availability S/390 Technology Guide

 Figure 66. IMS Fast Path Online Change

10.5.2 Workload Router

The IMS workload router is an optional utility that allows you to distribute IMS
transactions via predefined paths through MSC links, as shown in Figure 67.

Figure 67. IMS Workload Router

It provides a weighted distribution in order to balance the transaction workload

in a sysplex automatically:
• By detecting inactive resources and rerouting transactions

Chapter 10. IMS 195

 • By reconfiguring workload distribution among available systems if an outage
occurs

The benefits of this utility are:

• It balances the transaction workload in a sysplex to optimize workload
management.
• It facilitates flexibility in configuring workloads.
• It provides easier way of managing to peak periods.
• It increases availability for planned or unplanned outages of IMS resources.

10.5.3 IMS Partition DB

Many availability and performance attributes of IMS Fast Path are derived from
data partitioning, which is based on DEDB and areas. However, for full function
databases, there was no partitioning attribute available. Now the IMS/ESA
Partition Support Product for MVS (5697-A06) allows you to split your database
up into up to 32 partitions for full function databases, transparent to the
application. Using the IMS Partition DB function, you are able to use all current
IMS functions and utilities, and get equivalent or better performance. The
DBCTL interface from CICS is also supported.

Figure 68. IMS Partition DB

The IMS Partition DB, and the additional utilities supporting operations on
partition level provide the following benefits:
• Capacity for growth
• Manageability
• Higher availability
• Potential for better performance
• Ease of use
• Transparency to applications
• Simple conversion

196 Continuous Availability S/390 Technology Guide

10.5.4 Extended Terminal Option (ETO)
The Extended Terminal Option (ETO), introduced in IMS 4.1, improves your
system′s availability by reducing scheduled outages for the addition or deletion
of ACF/VTAM local and remote terminals or LTERMs. This feature also enables
you to add additional ACF/VTAM terminals, and users for these terminals.
Another benefit is that you get greater user flexibility in the ETO environment to
control session termination and user signoff conditions. You get also improved
network availability for terminals shared between multiple systems in the
network.

10.5.5 ARM Support

IMS Version 5 is the first version of IMS to support the MVS automatic restart
management (ARM) function. ARM is an MVS recovery function that can
improve the availability of started tasks. When a task fails, or the system on
which it is running fails, automatic restart management can restart the task
without operator intervention.

10.5.6 Distributed Sync Point using OS/390 RRS

Distributed Sync Point support enables IMS and remote application programs to
participate in a 2-phase-commit protocol coordinated by the OS/390 Recoverable
Resource Manager Services (OS/390 RRS). Before this support, IMS always
acted as the sync-point manager. In this new scenario, MVS manages the
sync-point process on behalf of the conversation participants: the application
program and IMS (now acting as a resource manager).

10.5.7 Daylight Saving Time

The IMS internal time-stamp format has been increased in size to accommodate:
• The 4-digit representation of the year
• The increased precision of the time stamp
• The offset between local time and Coordinated Universal Time (UTC)
In addition, IMS no longer uses local time in its internal time stamps. In Version
6, IMS uses UTC as a time base. This new timestamp format and UTC time base
satisfies three customer requirements:
1. To remove the restriction that IMS must be shut down during the transition
between daylight saving time and standard time
2. To operate in and beyond the year 2000
3. To support cross-time-zone operation between IMS systems for recovery
purposes

10.5.8 IMS WWW Connection

The IMS WWW Connection is a major step for IMS customers wanting to provide
the World Wide Web as an additional access path to business applications. A
common gateway interface (CGI) program requests information from IMS TM.
The requested information is mapped to Hypertext Markup Language (HTML)
and returned to a Web browser for display or manipulation, as shown in
Figure 69 on page 198.

Chapter 10. IMS 197

 Figure 69. IMS Internet Configuration

198 Continuous Availability S/390 Technology Guide

Chapter 11. DB2

DB2 was designed so that the operator should not have to take DB2 down in
order to perform traditional database activities. Every new version of DB2
delivers new functions that are designed to ensure high availability.

In this chapter we discuss the main procedures and features of DB2 that are
introduced in Version 4.1 or 5.1 to improve both high availability and continuous
operation. The main areas we cover are:
• Summary of DB2 continuous availability attributes
• DB2 recovery features
• DB2 Parallel Sysplex exploitation
• Other DB2 continuous availability features

11.1 Summary of DB2 Continuous Availability Attributes

Table 10 shows you which availability attribute applies to frequency, duration
and scope of an outage, and which outage type it belongs to. We distinguish
between planned (continuous operation) and unplanned (high availability)
outages.

Table 10. DB2 C/A Attribute Summary Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e

Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
CONCURRENT/SHRLEVEL X CO
CHANGE Option in COPY
CHANGELIMIT Option in COPY X X CO
In-Line Copy (COPYDDN Option in X X C/A
LOAD/REORG)
Multiple Image Copies in One X CO
Pass
Command to Force Active-Log X C/A
Archive
Data Sharing X X X C/A
Type 2 Index X CO
Row-Level Locking X X HA
Read-Through Locks X CO
Online Reorg X CO
RRS Attachment Facility X CO
Partition Independence X X CO
A R M Support X HA
Daylight Saving Time X CO
CANCEL THREAD X C/A

 Copyright IBM Corp. 1998 199

11.2 DB2 Recovery Features
DB2 has a comprehensive and integrated set of facilities to support recovery.
This includes dual logging, automatic recovery during restart of DB2 and the
transaction manager, and extensive utility support such as QUIESCE, REPORT,
COPY, RECOVER, and MERGECOPY. The basic recovery information is kept in
the DB2 catalog, the logs, and the boot strap data set (BSDS), and is used as the
base for the recovery process. Like IMS, DB2 recovery facilities place the
emphasis on maintaining data integrity, regardless of the type of failure that
might occur.

DB2 protects data from three types of failures:

• Subsystem Failures
When a subsystem failure occurs, a restart of DB2 automatically restores
data to a consistent state by backing out uncommitted changes and
completing the processing of the committed changes. This is possible
because both before and after update images of the changed data are kept in
the log.
• Media Failure
DB2 makes provisions for media recovery, for example failure of a disk
device or read/write failure to disk, by providing log data sets. Using the
information in the DB2 log and the latest image copy, the DB2 RECOVER
utility applies all changes made to the user data set.
• Application Program Abend
If an application ends abnormally, DB2 isolates the work associated with the
failing program and then rolls back all uncommitted changes to the data,
without interfering with other subsystem activities.

11.2.1 Copy Utility

One of the most significant continuous operation features of DB2 is the COPY
utility. It can be used to produce a copy of the data, while the applications
continues to read and write the data.

You can use:

• SHRLEVEL CHANGE option
The use of SHRLEVEL CHANGE produces a fuzzy image copy where
uncommitted data might also be copied. To recover to a point of
consistency, you may apply necessary log records. Therefore image copies
taken using SHRLEVEL CHANGE are not recommended for use with
RECOVER TOCOPY.
• CONCURRENT option (DB2 4.1 and higher)
The CONCURRENT option on the COPY utility invokes the concurrent copy
function of the data facility storage management subsystem (DFSMS), and
records the resulting image copies in the catalog. DFSMS lets you update
data after a very brief freeze of the data set during a copy. Image copies
taken by DFSMS are full image copies. For more detailed information about
concurrent copy, see section 6.2.1.6, “Concurrent Backup” on page 102.

The CHANGELIMIT option on the COPY utility lets DB2 determine when an image
copy should be performed and whether a full or incremental image copy should

200 Continuous Availability S/390 Technology Guide

 be taken. Using the CHANGELIMIT option you can, for example, avoid running
image copy jobs when very few or no pages of a table space have changed
since the last image copy was taken. You can use the saving in time to
maximize the use of batch windows.

11.2.2 LOAD and REORG with Inline Copy

In general, the more often you make image copies, the less time recovery takes,
but of course, the more time is spent making copies. If you use LOG(NO)
without the COPYDDN option when you run the LOAD or REORG utilities, DB2
places the table space in copy-pending status. You must remove the
copy-pending status of the table space by making a image copy before you can
further change the table space.

If you run REORG or LOAD REPLACE with the COPYDDN option, a full image
copy is produced during the execution of the utility and DB2 does not place the
table space in copy-pending status. This eliminates the period of time when the
table space is in copy-pending status and a separate COPY step is not required.
Therefore the data is available sooner.

If you use LOG(YES) and log all updates, then an image copy is not required for
data integrity. However, the utility job elapsed time increases because of the
additional logging activity, and in a recovery case, the recovery process using
the image copy is much more efficient than applying a large number of log
records.

11.2.3 Logging
DB2 provides a common log to record data changes for all activities running in
the DB2 subsystem. Since the information in the log is extremely critical for
preserving data integrity, provision is made for writing two copies of the log, for
either the active log and/or the archive log.

DB2 also provides an automatic archiving facility. The active logs are
maintained on disk and as an active log fills up, it is automatically archived to
tape or disk. An active log space cannot be reused until the existing log data has
been archived. DB2 will stop processing if all active logs become full before this
archiving can take place. It is also very important, therefore, to have enough
active logs specified and to make sure that the tape or DASD used for archiving
is available promptly. If dual archive logs are specified, the active log is written
to two archive log data sets.

11.2.4 Recovery
To recover a table space, the RECOVER utility uses these items:
• A full image copy, that is, a complete copy of the table space
• Any later incremental image copies, each ow which summarizes all changes
made to the table space from the time the previous image copy was made
• All log records created since the most recent image copy

Figure 70 on page 202 shows an overview of the recovery process. The

RECOVER utility uses information stored in the catalog table to restore the most
recent full image copy FIC1, apply changes from any later incremental image
copies IIC1 and IIC2, and apply all log records since the last-applied image copy
to the point of recovery.

Chapter 11. DB2 201

 Figure 70. DB2 Recovery Process

In order to get maximium availability, you need to develop a plan for backup and
recovery that minimizes the duration of the outage because data is not available.
Some factors to consider are:
• Backup frequency
Use your recovery criteria to decide how often to make copies of your
database. You should make copies after jobs that make large numbers of
changes. Recovering from an image copy is faster than to applying a large
amount of log records.
• The right characteristics for your logs
For example, if you have enough DASD space, use more and larger active
logs. Recovery from active logs is faster than from archive logs.

For more information about recovery consideration, refer to DB2 for OS/390 V5.1
Administration Guide .

11.2.5 Disaster Recovery

In the case of a total loss of a DB2 computing center, you can recover on
another DB2 system at a recovery site. To do this, you must regularly back up
the data sets and the log for recovery. As with all data recovery operations, the
objectives of disaster recovery are to lose as little data, workload processing
(updates), and time as possible.

There are several levels of preparation for disaster recovery, such as:
• Dump the whole DB2 environment.
This is a very cheap and easy way to prepare for disaster recovery, but it
can only be used to recover to the point in time when the dump was taken.
In the case when your disaster backup belongs to a point in time long before
the point of failure, you may lose a large amount of workload that had been
done. In order to obtain consistent data, you have to shut down DB2 during
the back.
• Create a system-wide point of consistency for more important table spaces
by using the QUIESCE utility.

202 Continuous Availability S/390 Technology Guide

 The QUIESCE utility requires a list of all objects which have to survive a
disaster. The maintenance on this list is an ongoing task. You run the
QUIESCE utility when the DB2 system is running in a low workload period.
The point in time of your disaster recovery can be at the quiesce point.
• Create a system-wide point of consistency using the ARCHIVE LOG
MODE(QUIESCE) command.
This is a better way of achieving a system-wide point of consistency. The
impact on your applications is very small because when competing with your
applications, this command is designed to be the loser, unlike with the
QUIESCE utility, which could force your application to receive an unavailable
condition.
• Transfer DB2 log data continuously offside.
This can be achieved by using a DB2 log capture exit. You need to write this
exit, which captures and transmits log data offside. Log data can be applied
to an existing DB2 system that is a replica of your existing system. This
procedure enables you to recover to within seconds of the failure, but you
will have to back out incomplete units of work if the software you have does
not cater for this.
• Mirror the data to a secure location.
This is the ultimate of disaster recovery in that you can have a dual copy of
your DB2 data at a remote site. Products as PPRC and XRC can enable
remote dual copy. This procedure can be very expensive if you have a large
amount of data that needs to be recovered, but the benefit could be that you
have a hot site waiting for you in case of disaster.

For more information about DB2 Disaster Recovery, see DB2 Data Sharing
Recovery and Disaster Recovery Library - S/390 Technology Guide .

11.3 DB2 Data Sharing

Before DB2 V4.1, only the distributed function of DB2 allowed for read and write
access among multiple DB2 subsystems. This solution requested that the DB2
subsystem that owned the data is active and running. Data Sharing
implemented in DB2 V4.1 allows for read and write access to DB2 data from
different DB2 subsystems residing on multiple central processor complexes
(CPC) in a Parallel Sysplex. For a detailed description of the Parallel Sysplex
architecture, refer to Chapter 7, “Parallel Sysplex Architecture” on page 137.

Figure 71 on page 204 shows the major components of a DB2 data sharing
group. The two DB2 subsystems accessing the same data are called members
of a data sharing group . They can be running on the same MVS image or
different MVS images in the same Parallel Sysplex.

Chapter 11. DB2 203

 Figure 71. DB2 Data Sharing Group Structure

All members of the DB2 data sharing group can read and update the same
tables at the same time. Therefore all data must reside on shared DASD. The
members of a DB2 data sharing group uses coupling facility services to
communicate and move data between the individual members.

The individual members of a data sharing group need the connection to the
coupling facility (CF) to exchange locking information, system and data object
status information, and to cache shared data pages with the other members of
the group.

11.3.1 Recovery in a Data Sharing Environment

From an application recovery perspective, not much has changed from the
non-data sharing environment when it comes to recovering your data. However,
behind-the-scenes recovery in a data sharing environment is far more complex.
During implementation of data sharing you have to consider:
• The logging environment
In a data sharing environment, each DB2 member has his own log data sets.
During a recovery process, the RECOVER utility might need log information
from other DB2 members that have updated data. Recovery process
performance, and therefore the period of time of an outage, depends on
whether or not logs have been archived, and where they have been archived
(DASD or tape).
• Prevention of syplex failures and recovery scenarios
Failure of Parallel Sysplex components such as the coupling facility or the
connection to it can mean serious outages for users. Therefore, careful
preparation for structure and connection failures can greatly reduce the
impact of outages to the end user.
For a detailed description of recovery consideration in a data sharing
environment, refer to DB2 Data Sharing Recovery .

204 Continuous Availability S/390 Technology Guide

11.3.2 Why Choose Data Sharing
Data sharing offers a number of advantages, such as :
• Additional capacity and scalability
• Improved price performance because you can run business applications on
less expensive S/390 microprocessors and grow in steps you need
• Workflow management can correlate better to the business requirements

In this section we review the major advantages of data sharing from an

availability point of view. More DB2 users are demanding access to DB2 data
every hour of the day, every day of the year. DB2 data sharing helps you meet
this service objective, because the DB2 subsystem is not a a single point of
failure any more. As Figure 72 illustrates, if one subsystem shuts down for
some reason, users can access their DB2 data from another subsystem.
Transaction managers are informed of the failure and can switch workloads to
another DB2 subsystem in the group. You can automate restart recovery if you
are running MVS Version 5 Release 2 or later.

Figure 72. Data Sharing Improves Availability during Outage

During planned outages for maintenance or other work, your users can keep
working. You can shut down a subsystem while switching your users to another
subsystem in the group.

11.3.3 Configuration Consideration for High Availability

TSO and batch applications have two methods for specifying the DB2 to which
they want to connect. The first method is to specify the particular subsystem
name in the job for TSO and batch, and on an explicit CONNECT for CAF. The
second method, new with data sharing, is to use the group attachment name
instead of a specific subsystem name.

The group attachment name acts as a generic name for the DB2 subsystem in a
data sharing group. It substitutes for the DB2 subsystem name running on the
MVS the job was submitted from. Utility jobs can also use the group attachment
name. By using the group attachment name, the TSO user or batch job does not

Chapter 11. DB2 205

 have to be sensitive to the particular subsystem. This makes it easier to move
jobs around the sysplex as needed.

You cannot use the group attachment for CICS and IMS applications because
those transaction managers must be aware of the particular DB2 subsystem to
which they are attached so they can resolve in-doubt units of recovery in the
event of a failure.

If you use CICS as transaction manager and CICPlex/SM to route transactions in

either of the running CICS regions, you can shut down the DB2 subsystem
without outage to the end user. The running transaction will finish. New
incoming transactions will route to another CICS region that is connected to
another DB2 member, because the CICSPlex/SM gets the information from
workload manager that the DB2 attachment is not active anymore.

If you use IMS as transaction manager, you should define all DB2 subsystems in
the control region of IMS. Then you are able to connect message regions to
different DB2 subsystems. Before you stop the DB2 subsystem, you should stop
the message regions that the DB2 subsystem is connected to. Message regions
connected to another DB2 system will care of the new incoming transactions.

11.4 Other DB2 Continuous Availability Features

Because you need access to your data every hour of every day, every new
version of DB2 delivers new functions that are designed to ensure continuous
availability. The functions discussed in this chapter are the most important
continuous availability functions introduced in DB2 V4.1 and DB2 V5.1, but there
are many other DB2 features and enhancements you may want to be aware of.
For more information, refer to DB2 for MVS/ESA V4.1 What is New and DB2 for
OS/390 V5.1 Release Guide .

11.4.1 Type 2 Indexes

Until Version 4, DB2 supported a single index structure, now known as a type 1
index. Version 4 introduces a new type of index known as a type 2 index.

Type 2 indexes eliminate locks on index pages. Because there are usually fewer
rows to a data page than there are index entries on a index page, locking only
the data when you lock pages is likely to cause less contention than locking the
index. This way you can avoid most of the deadlock and timeout problems on
the index that often caused application abends because data was not available.

Another advantage of type 2 indexes is that they let you use other function such
as processing multiple parallel tasks, improved partition independence, row
locking, and the read-through locks. All of these functions also improve
concurrency in accessing data, and therefore avoid outages due to data not
being available to applications.

11.4.2 Row Level Locking

Depending on the length of a row, a 4k data page may contain multiple rows of a
table. If you use page level locking as the locking granularity, and you access
one row, you lock all rows stored in the page. Applications accessing central
key tables such as customer or material information very often run into a data
not available condition because another application is locking the whole data
page. A possible solution to the concurrency problem is to use a type 2 index,

206 Continuous Availability S/390 Technology Guide

 which allows you to lock every row in the table separately (by using LOCKSIZE
ROW).

11.4.3 Read-Through Locks

Uncommitted read (UR) isolation lets an application read uncommitted data even
though it might be locked. Uncommitted read can improve the concurrency of
queries that read uncommitted data at the cost of less protection from other
application processes. The query can get result rows that are written by a
concurrent running application which has not been committed and rolled back.
Figure 73 shows an example where APPL1 inserts in T1 and APPL2 runs a query
against this table. After APPL2 finishes APPL1 rolls the changes back for some
reasons. However, APPL2 got result rows that were never visible to applications
using other isolation levels.

Figure 73. Application Scenario using Read-Through Lock Isolation

There are different kinds of applications that, in the past, caused or met data not
available condition during online processing. These applications, as listed, can
tolerate the previously described condition:
• A reference table, such as a table of descriptions of parts by part number,
because it is rarely updated.
• When, for security reasons, the updater is also the only reader of the table.
The access can be serialized within the application.
• All statistical kinds of applications; for example, if someone wants to do
statistical analysis on their employee data and reading an accasional
uncommited record cannot affect the averages much. A typical question
might be “What is the average salary by sex within education level?”

Chapter 11. DB2 207

11.4.4 Online Reorg
DB2 V5.1 REORG utility offers great improvement in the availability of data to
applications during reorganization. A new keyword, SHRLEVEL, is added to the
REORG utility control statement to allow you to choose standard, read-only
online, and read-write online reorganization.

With the SHRLEVEL CHANGE option, the applications have read and write access
to the data being reorganized through almost the entire process. The process
involves these activities:
1. The utility unloads the data from the original tablespace, sorts the data by
clustering key, and reloads the data into a shadow copy. Concurrently,
applications have read and write access to the original tablespace and
changes are recorded in the log.
2. The utility reads the log records and applies them to the shadow copy
iteratively. During the last iteration, they have only read access.
3. The utility switches the application access to the shadow copy by renaming
the data sets for the table and the indexes. During the actual switch,
applications have no access to the data.
4. The applications read and write to the renamed shadow copy.
Figure 74 gives you a view of this process.

Figure 74. Online Reorganization

The utility has additional options that helps you to improve runtime. The
keyword SORTKEYS allows you to specify that sorting is to be done in memory,
rather than in writing and reading workfiles. You also may now have an image
copy taken at the time of loading or reorganizing the tablespace.

11.4.5 Partition Independence

A key availability feature of DB2 is the ability to partition a DB2 table space into
as many as 254 partitions. A table can hold up to a terabyte of data in a
compressed or an uncompressed format.

208 Continuous Availability S/390 Technology Guide

 Distributing large amounts of data over more partitions can reduce outages or
the scope of an outage and allow more concurrent processing for utilities. And
while smaller amount of data are unavailable when utilities are run, processing
time is shortened.

For example, the RECOVER utility can run against a single partition instead of
recovering the whole table space. Figure 75 shows an example of how a
partition tablespace can be accessed. The first partition is not accessed at all,
the second partition is stopped for any reason, the third partition is running an
SQL statement and the fourth partition is in recovery pending mode.

Figure 75. Partition Independence

Serialization of SQL and utility requests on a partition table space is done by a

drain/claim mechanism. This drain/claim mechanism is related to physical
objects, and it
works fine for the table space themselves and the partition index. At most, the
definition of additional non-partition indexes are required. If you use type-2
indexes, DB2 can internally create a logical partition to serialize the access.

11.4.6 RRS Attachment Facility

The Recoverable Resource Manager Services attachment facility (RRSAF) is a
DB2 attachment facility that relies on an OS/390 component called OS/390
Transaction Management and Recoverable Resource Manager Services (OS/390
RRS). OS/390 RRS provides system-wide services for coordinating two-phase
commit operations across MVS products like VSAM, MQSeries and so on.

DB2 already supports a two-phase commit protocol with transaction mangers

like CICS or IMS. By providing this service in an OLTP environment, RRSAF
enables access to resources such as SQL tables, DL/I databases, MQSeries
messages, and recoverable VSAM files within a single transaction scope.
Figure 76 on page 210 shows an example.

Chapter 11. DB2 209

 Figure 76. OS/390 RRS

The OS/390 RRS can be used for an application that connected to DB2 using
RRSAF, and for DB2 stored procedures that execute in an MVS WLM-managed
stored procedures address space.

11.4.7 ARM Support

If you are using MVS/ESA Version 5 Release 2 or later, you can use automatic
restart. The purpose of using automatic restart is to reduce the time a particular
system is down. When DB2 stops abnormally, the surviving MVSs analyze the
situation to determine whether MVS has failed too and where DB2 should be
restarted. It then restarts DB2 appropriately.

11.4.8 Daylight Saving Time

In many installations turning daylight saving time on and off caused a
system-wide outage for at least one hour. By using Greenwich Mean Time as
your time-of-day clock (TOD) and an offset for the local time, you can avoid this
outage.

11.4.9 Cancel Thread Command

Before DB2 4.1 the only way you could cancel a DB2 thread was to use the MVS
cancel command to cancel the application. The -CANCEL THREAD command
now allows you to cancel an application thread that hangs without DB2.

210 Continuous Availability S/390 Technology Guide

Chapter 12. MQ Series

Traditional application design is based on having an application support platform

that starts from the end-user terminal and ends up with the application data,
through a number of different components that must be all present to guarantee
the application functionally.

All applications designed so far have been “conscious” of this platform since
normally it is hardcoded in the application, for example, as in the 3270
communication protocol of an IMS application.

Figure 77. Traditional Applications Structure

A failure in a single component of this application platform structure results in

total unavailability of the service, from the user′s perspective.

Message Queueing allows you to design applications that are independent of

how service is provided. This means that if a single component of the
application structure is unavailable, the user loses only the service delivered by
that component.

Figure 78. Message Queueing Applications Structure

Moreover, an application designed using Message Queueing is also independent

of the communication protocol, since each application talks directly with the local
Queue Manager, which is then responsible for delivering the application data to
the right destination.

This new design methodology allows you to build applications that are
continuously available, thus you need to understand what Message Queueing is
and how it works.

 Copyright IBM Corp. 1998 211

12.1 Summary of MQSeries Continuous Availability Attributes
Table 11. MQSeries C/A Attribute Summary Matrix. Legend: X = A p p l i e s ,
Type=Outage Type, HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
Connectionless Communication x x CO
Time-Independent Processing x x CA
Programs/Functions Distributions x x CA
Avoid Batch Interference x x x CA
Load Balancing x CA
Dual Logging x x x CA
Parallel Processing x x CO
Triggering Events x x HA
Segregation of Message Traffic x HA

12.2 What is Message Queueing

Message queueing is a method of program-to-program communication.

It allows programs to communicate by sending each other data in messages

rather than by calling each other directly. These messages are places on
queues, so that programs can run independently of each other, at different
speeds and times, in different locations, and without having a logical connection
between them.

IBM MQSeries is a family of products for cross-network communication that

enable programs to talk each other across a network of unlike components
(processors, operating systems, subsystems, and communication protocols)
using a simple and consistent application programming interface.

It can be considered as formed by the following major parts:

• Messages
• Queue Managers
• Queues
• Message Channels

12.2.1 Messages
A message is a string of bits and bytes that has meaning to one or more
application programs. It comes into existence when it is passed to the local
queue manager by the sending application ( putting the message ) and it ceases
to exist when the receiving application removes the message from the queue
( getting the message ).

A message consist of two main parts:

Application data: The programs view the string as consisting of a sequence of

“items,” each item having a particular data type and meaning. There are no
constraints on the nature of the data in the message. It is possible for a
message to contain no application data; although it might not seem very useful,

212 Continuous Availability S/390 Technology Guide

 sometimes it is all that is needed to signal to the receiver that some particular
event has occurred.

The message-queueing service does nothing with the application data other than
to deliver it safely to the receiving application.

Control information: The data defines various properties of the message, and is
used by the message-queueing service to decide how the message should be
processed and it is returned by the queue manager to the receiving application.

The control information is contained in a data structure called the message

descriptor (MQMD) and it defines some important properties of the message,
such as:
• Message type
This describes the type of message and if it requires a reply by the receiving
application.
• Message identifier
This uniquely identifies a particular message, so that the receiving
application can retrieve a specific message from the queue.
• Priority
This indicates the importance of a message relative to other messages in the
queue.
• Expiry time
This specifies the lifetime of the message. If the message has not been
retrieved by the receiver before that time expires, the queue manager
discards the message and the receiving application can be notified about it.
• Persistence
This protects the message from a system or network failure, and survives
either until it is retrieved by the receiving application, or until its expiry time
is exceeded, at which point the queue manager discards it.

12.2.2 Queue Managers

A queue manager is the system service that provides the message-queueing
facilities used by applications. Applications obtain message-queueing services
by using the message-queueing calls to communicate with a local queue
manager.

Each queue manager is known by its name which, generally, must be unique
within the network of interconnected queue managers, so that one queue
manager can unambiguously identify the target queue manager to which any
given message should be sent.

12.2.3 Queues
A queue is a type of list that is used to store messages until they are retrieved
by an application. Its physical representation depends on the environment, but
can be as follows:
• A buffer or buffers in main storage
• A file or files on disk or other permanent storage device
• Both of the above

Chapter 12. M Q Series 213

 Each queue belongs to a queue manager , which is responsible for maintaining
the queue and putting received messages onto the appropriate queue.

Messages can be retrieved from a queue by the applications according to these

retrieval algorithms:
• First in, first out (FIFO)
• Message priority, as defined in the message descriptor
• A program request for a specific message

Queues can be either local or remote.

• A local queue belongs to the queue manager to which the application is
connected, and it can be opened for input or for output.
• A remote queue belongs to a queue manager other than the one to which
the application is connected, and it can be opened only for output. It can be
opened for input by the applications that are connected to that queue
manager, as it would appear to be a local queue for them. Applications can
open a remote queue in two ways:
− By using a Local Definition
This is like using an alias to access the queue, and it allows you to move
the queue to another queue manager without changing the application.
− Without using a Local Definition
In this case, the application must specify both the the name of the
remote queue manager that owns the queue and the name by which the
queue is known to that queue manager.

The different types of queues available with MQseries are as follows:

• Message queues: receives data messages from applications.
• Event queues receive event messages, which indicate that a particular type
of instrumentation event has occurred during the execution of an application
program.
• Initiation queues receive trigger messages that are read by a trigger monitor
application, which then starts the appropriate application to process the
message.
• Transmission queues temporarily store messages that are destined for a
remote queue manager. There must be a transmission queue for each
remote queue manager to which the local queue manager is to send
messages.
• Reply-to queues receive the reply that the sender of a message has asked
for on a data message.
• Dead-letter queues, which are also known as undelivered message queues ,
receive messages that cannot be routed to their correct destinations.
• Command queues hold MQSeries administration commands that only
suitably authorized applications can send. The commands in these
messages are processed by the command server part of the queue
manager.
• Alias queues are special kinds of objects that can be used to access another
queue, as well as to set up different attributes to the same queue.

214 Continuous Availability S/390 Technology Guide

 • Model queues are used to generate a new queue, called a dynamic queue.
In turn, dynamic queues can be Temporary or Permanent . While temporary
queues are deleted as soon as the application closes them, permanent
queues are not deleted until an application closes and deletes them.

12.2.4 Message Channels

A message channel provides a one-way communication path between two queue
managers on the same, or different, platforms. It is used for the transmission of
messages from one queue manager to another, and shields the application
programs from the complexities of the underlying networking protocols.

A message channel consists of three main components:

• A transmission queue
• A message channel agent, which is a program that controls communications.
• A communications link, which is a platform-dependent connection that
provide the physical links between the local queue manager and any remote
queue managers that it wants to exchange messages with.

For a detailed description of how these components allow messages to be

exchanged between systems, and how to define them, refer to MQSeries
Distributed Queueing Guide .

The movement of messages between queue managers is handled by the

MQSeries Distributed Queue Management (DQM) facility, which basically works
in two ways, regardless of whether the queue managers are on the same, or
different, platforms.
• Simple transfer is used between two queue managers that are connected by
an MQSeries message channel.
• Staged transfer is used to interconnect queue managers that are located in
nodes that are not adjacent to the sending node, and can only be reached by
staging through an adjacent queue manager.

12.2.5 Other Features

12.2.5.1 Triggers
Some applications run continuously, and so are always available to read a
message as soon as it arrives on the application′s input queue. However,
having the application active consumes system resources, even when the
application is waiting for a message to arrive. This additional load on the
system is not desirable when the volume of message traffic fluctuates wildly.
Instead of the application running continuously, it is better for the application to
run only when there are messages to be processed. The queue-manager′ s
triggering facility can be used to help make this happen.

The queue manager defines some conditions as trigger events. When trigger
events occur, the queue manager sends a trigger message to an initiation
queue, which will be processed by a trigger-monitor application in order to take
appropriate action. Normally this action would be to start some other application
to process the queue that caused the trigger message to be generated.

The trigger events defined by the queue manager are:

Chapter 12. M Q Series 215

 • When the queue becomes nonempty
• Every time a message arrives on a queue
• When a certain number of messages are on the queue

Triggering can be also controlled by the minimum priority that a message must
have in order to be able to cause triggering. If a message arrives with a lower
priority, the message is ignored for the purposes of deciding whether a trigger
message should be generated.

12.2.5.2 Application Data Conversion

The representation of character and numeric data is different on the various
platforms for which MQSeries products are available.

The message descriptor in all messages contains the coded character set
identifier (CCSID) and encoding information, which identifies the representation
used for the character and numeric data that is in the data portion of the
message.

Some MQSeries products can convert the data within messages from one
representation to another, and this operation is performed at the following times:
• By a queue manager during receipt of a message, if the data conversion
option is included in the request.
• By a message channel as it transmits a message to a remote queue
manager, if the data conversion option is included in the channel definition.

For further information about data conversion, refer to MQSeries Application

Programming Guide .

12.3 Program-to-Program Communication

Applications communicate by agreeing to use particular named message
queues. The location of these queues is not apparent to the applications which
send the messages; each application interacts only with its local queue
manager, and it is the network of interconnected queue managers that is
responsible for moving the messages to the intended queues.

There are several types of application communications, which can be

summarized as follows:
• Unidirectional communication
With this type, Program A puts a message in a queue which is the queue
that program B has agreed to read from.
• Bidirectional communication
With this type, Program A puts a message in a queue which is the queue
that Program B has agreed to read from, while Program B puts its messages
in another queue which is the one that Program A has agreed to read from.
Figure 79 on page 217 shows examples of these types of communication.

216 Continuous Availability S/390 Technology Guide

Figure 79. Applications Communication using Message Queueing

• Remote communication
With this type, if Programs A and B reside on different systems (that is, if one
is remote from the other), then they interact with different queue managers.
From the point of view of Programs A and B, there is no difference between
the situations described in the previous two points, because each program
interacts with only its local queue manager.
• Client-server applications
The previous points illustrate the simple communication of messages
between two programs. However, message queueing is also ideal for the
client-server type of application, where many client programs send
messages to a single server program. It is also possible for several
programs to get messages from the same queue, that is, to have several
servers processing the queue. This is useful in situations where the volume
of message traffic exceeds the processing capacity of one server; additional
servers can be started during peak hours to cope with the extra traffic, but
without the client programs needing to be made aware of this.
Figure 80 on page 218 shows a client/server structure.

Chapter 12. M Q Series 217

 Figure 80. Flexible Client/Server Structure

12.4 Recovery Features

A prime requirement for a message delivery system is that it must be reliable.
Many functions are built into MQSeries to ensure the following:
• That messages are not lost, despite system failures
• That messages are not delivered more than once
• That messages are not accessed by, or delivered to, unauthorized persons
or applications
These functions are based on:
• Logging (including options for dual logging and archiving)
• Automatic recovery from transaction, system, and storage media failures (for
which suitable backups are needed)
• Restart from backup files

These functions use data sets to keep track of ongoing activities, such as:

218 Continuous Availability S/390 Technology Guide

 • Page data sets, which are used primarily to store messages, object
definitions, and other important information relevant to the queue manager.
• Bootstrap data sets, which are used to keep an inventory of all active and
archived log data sets and a wrap-around inventory of all recent activity.
MQSeries can be configured with dual BSDSs. This is known as running in
“dual mode.”
• Log data sets, which record all persistent messages in the active log as they
are put onto queues by applications, as well as the information needed to
recover all messages, queues, and the queue manager.
As for the BSDS, MQSeries can have either single logging or dual logging.

Log data sets also contain checkpoint records , which are used to reduce restart
time.

At a checkpoint, MQSeries logs its current status and registers the RBA of the
checkpoint in the bootstrap data set (BSDS).

12.4.1.1 The Log Archive Process

When the active log is full, MQSeries switches to the next available log data set
and copies the contents of this log to a new archive-log data set, along with a
copy of the BSDS. The archive-log data sets are used by MQSeries to restore
the message queues, should any problem occur.

12.4.1.2 Other Recovery Considerations

In addition to the logging and archiving facilities provided by MQSeries, you
should also consider other recovery facilities that are provided in your
enterprise, like:

CICS recovery: CICS recognizes MQSeries as a non-CICS resource (or external

resource manager), and includes MQSeries as an agent in any syncpoint
coordination requests using the CICS resource manager interface (RMI).

For more information about CICS recovery, refer to CICS for MVS/ESA Recovery
and Restart Guide . For information about the CICS resource manager interface,
refer to CICS for MVS/ESA Customization Guide .

IMS recovery: IMS/ESA recognizes MQSeries as an external subsystem and as

a participant in syncpoint coordination. IMS recovery for external subsystem
resources is described in IMS/ESA Version 4 Customization Guide: System .

Using Extended Recovery Facility (XRF): MQSeries can be used in an Extended

Recovery Facility (XRF) environment. All MQSeries-owned data sets (executable
code, BSDSs, logs, and page sets) must be on DASD shared between the active
and alternate XRF processors. If you use XRF for recovery, you must stop
MQSeries on the active processor and start it on the alternate, and this must be
coordinated with the Transaction Manager that you are using, either IMS or
CICS.

Chapter 12. M Q Series 219

12.4.2 How Changes are Made to Data
You need to understand how MQSeries queue managers interact with other
programs to keep all the data consistent.

12.4.2.1 Units of Work

Most applications need to access resources of one form or another, and a
common requirement is to be able to make a coordinated set of changes to two
or more resources; that means either all of the changes made to the resources
take effect, or none of them do.

Queues are no exception to this; applications need to be able to get and put
messages (and possibly update other resources, such as databases), and know
that either all of the operations take effect, or that none of them does. The set of
operations involved in this is called a unit of work (UOW).

A unit of work starts when the first recoverable resource is affected. For
message queueing, this means when a message is put or retrieved as part of a
unit of work.

The unit of work ends either when the application ends, or when the application
declares a syncpoint , at which time any party that has an interest in the unit of
work has to commit about the work they did so far, and the changes that were
made as part of the unit of work become permanent. If any part that is involved
in the unit of work does not commit, the unit of work is backed out; that is, all
changes to data are discarded.

If the unit of work is ended by the application declaring a syncpoint, another unit
of work can then start, so that one instance of an application can be involved
with several sequential units of work. Figure 81 shows an example of UOWs and
syncpoints.

Figure 81. Units of Work and Syncpoints

Putting messages within a unit of work: If a message is put as part of a unit of

work, the message does not become generally available for retrieval by
applications until that unit of work is committed successfully. If the destination
queue belongs to a remote queue manager, the message is not available to be
sent from the local queue manager until the unit of work is committed. This
means that it is not possible to send a request message and receive the reply to

220 Continuous Availability S/390 Technology Guide

 that request as part of the same unit of work; the unit of work containing the
request message must be committed before it is possible to receive the reply.

Getting messages within a unit of work: If a message is retrieved as part of a

unit of work, the message is removed from the queue and so is no longer
available to other applications. However, the message is not discarded but it is
retained by the queue manager until the unit of work is either committed or
backed out. If the unit of work is committed successfully, the queue manager
then discards the message. If the unit of work is backed out, the message is
reinstated in the queue in its original position, and so becomes available once
again to be browsed or retrieved by the same or another application.

Triggering messages within a unit of work: If the message that caused the
trigger condition is put as part of a unit of work, the trigger message does not
become available for retrieval by the trigger monitor application until the unit of
work completes, whether the unit of work is committed or backed out.

12.4.2.2 Units of Work and Message Persistence

Units of work can be used to protect against failures of the application, or
against failures of other resource managers operating within the same unit of
work.

Message persistence, on the other hand, protects against failures of the

message queueing queue manager.

Many applications that use units of work will also want to use persistent
messages. However, there are some situations in which it may be beneficial to
use one without the other.

For example, if the application contains logic to recover after a failure of the
queue manager, using units of work with nonpersistent messages gives a
performance benefit in the normal, nonfailure case. This combination can also
be used to ensure that a final (nonpersistent) message is not sent if the unit of
work is backed out before it reaches the syncpoint.

12.4.3 How Consistency is Maintained

If the data being managed by a queue manager is to be consistent with the data
in other subsystems, any data changed in one must be matched by a change in
the others. However, before one system commits the changed data, it must
know that the other system, or systems, can make the corresponding changes.
This process is also known as two-phase commit .

When the queue manager is restarted after an abnormal termination, it must

determine whether to commit or to back out units of work that were active at the
time of the error. For certain units of work, the queue manager has enough
information to make the decision, while for others it must get information from
the coordinator when the connection is reestablished.

Syncpointing cannot occur directly across remote links, but if you use the
assured message delivery feature of MQSeries, a commit request is sent in a
message to a remote system, where it remains until it can be delivered. When
delivery eventually occurs, any desired commitments can then be made.

Chapter 12. M Q Series 221

 In this situation, we are dealing with time-independent applications. It is for the
application designers to consider how, and when, they require the data shared
between their remote systems to be consistent.

12.4.4 Recovering Distributed Queueing Errors

If any error occurs while sending or receiving a message, the transaction is
terminated and error messages are sent to both the local and remote system
consoles. When you restart remote queueing, the queue manager checks for
and resolves any in-doubt messages caused by the previous termination.

An in-doubt message is one where the point of consistency prior to the message
is known, but it is not known with certainty whether the new point of consistency
that the message creates has been reached, or that a backout to a previous
point of consistency has been completed.

The application must decide what action to take to resolve the in-doubt situation.

12.5 How MQSeries Contributes to Continuous Availability

Because of their asynchronous nature, application systems based on message
queueing are inherently more reliable than traditional transaction processing
systems.

12.5.1 Connectionless and Time-Independent Communication

Message queueing is connectionless, thus programs communicate without
having to establish a mutual and private connection. Instead, Program A and
Program B gives and gets messages from the message-queueing service, which
acts as an intermediary. The mechanism by which the message is transmitted
from Program A to B is completely hidden from both application programs.

With message queueing, the exchange of messages between the sending and
receiving programs is also time-independent.

This means that the sending and receiving applications are decoupled so that
the sender can continue processing without having to wait for the receiver to
acknowledge the receipt of the message. The receiving application may be busy
or, indeed, it does not even need to be running. MQSeries holds the message in
the queue until it can be processed.

This concept is similar to posting a letter. The letter is delivered to the

recipient′s mail box whether or not there is anyone there to receive it; it will be
picked up later when the mail is collected.

You can now design your applications such that they can improve the total
system availability by exploiting this new way of communication, as described in
the following section.

12.5.1.1 Load Balancing

In situations where the message traffic exceeds the processing capability of a
single instance of the application, the queue manager, by means of the
triggering events, can start more instances of the same application in order to
compensate for the message traffic increment.

222 Continuous Availability S/390 Technology Guide

 12.5.1.2 Avoid Batch Interference
MQSeries allows you to design applications which avoid online and batch
interference by having them exchange data using queues instead of datasets.
This allows your online and batch processes to run concurrently, thus improving
total system availability.

12.5.1.3 Internet Interface

MQSeries provide a bridge between the synchronous World Wide Web and
asynchronous MQSeries applications, so that an Internet-connected Web
browser has access to MQSeries applications through interaction with HTML
fill-out form POST requests. MQSeries applications respond by returning HTML
pages to the gateway via the MQSeries queue.

12.5.2 Distribute Programs/Functions across Networks

MQSeries allows you to distribute application functions across networks. For
example, if your application needs a data entry function, that function does not
have to be implemented at the host site; it can run on the end-user workstation,
and it can run even if the host fails. All data entered from that point will be
stored by the queue manager in its queues.

When the host is again available, and the accounting function of the application
is started up, the host will reconnect with the end-user workstation queue
manager to gather the data.

Chapter 12. M Q Series 223

 Figure 82. Distributed Program

From the user perspective, the total application was continuously available
because it kept working even when the host site was unavailable. Indeed, the
user might be completely unaware of what happened.

12.5.3 Conversational Communication

Message queueing also supports the conversational style of program-to-program
communication in which programs take turns sending and receiving.

The conversational style of communication is similar to making a telephone call.

The person you want to speak to has to be there to answer the telephone, in
order for the conversation to take place.

224 Continuous Availability S/390 Technology Guide

12.5.4 Parallel Processing
Message queueing can also be used to achieve parallel processing. Instead of
Program A performing all parts of a task sequentially, the task (if suitable) can
be split into several smaller independent subtasks that can performed in parallel
by other programs, as shown in Figure 83

Figure 83. MQSeries and Parallel Processing

12.5.5 Segregation of Message Traffic

Sometimes it is useful to be able to segregate message traffic into two or more
classes, so that they can be controlled independently. For example, you might
want to keep development work separate from production work, or be able to
stop movement of one class of message traffic at certain times.

One way to segregate message traffic is to use different transmission queues for
the different classes of message by using queue-manager aliases. The
applications themselves need not be aware that message traffic is being

Chapter 12. M Q Series 225

 segregated in this way; that is, they would not have to be changed if the
segregation conditions change. Figure 84 on page 226 shows message traffic
segregation.

Figure 84. Segregation of Message Traffic

12.6 Continuous Availability Hints and Tips

The MQSeries restart process recovers its data to a consistent state by applying
log information to the page sets. In case a page set is unavailable or damaged,
a backup copy of this can be used.

Thus it is very important, in order to achieve a better availability, that all the
required data sets are duplicated and that the backup copies are also available
in case a restart is needed.

12.6.1 Dual Logging

MQSeries provides a dual logging and dual bootstrap feature that allows it to
duplicate that critical information directly. For this reason, you should carefully
plan where to allocate the second copy of the data set, in order to have at least
one of the two copies available.

MQSeries does not provide a duplication of page data sets by itself, but these
sets can be duplicated using other techniques such as IBM 3990-6 Remote Copy.

226 Continuous Availability S/390 Technology Guide

12.6.2 Restarting a Queue Manager on Another Processor
Take care to prevent MQSeries from starting on the alternate processor before
the MQSeries system on the active processor terminates, because a premature
start can cause severe integrity problems in data, in the catalog, and in the log.

Using global resource serialization (GRS) helps avoid integrity problems by

preventing simultaneous use of MQSeries on the two systems. The BSDS must
be included as a protected resource, and the active and alternate XRF
processors must be included in the GRS ring.

12.6.3 Local and Remote Queue Definitions

It is valid for a remote queue to be defined at the remote queue manager as
being another remote queue, thus using Local Definitions. This allows queues to
be moved from one queue manager to another without disrupting the queue
definitions at the other queue managers.

Chapter 12. M Q Series 227

228 Continuous Availability S/390 Technology Guide
Chapter 13. Systems Management

This chapter describes how good systems management can help you achieve
continuous availability. It shows that availability cannot be bought as an
off-the-shelf technology solution, but that instead you need good processes and
tools to increase availability.

13.1 Summary of Continuous Availability Attributes

Table 12 lists some of the actions and disciplines associated with Systems
Management, and indicates their affect on service availability.

Table 12. C/A Attribute Summary Matrix. Legend: X = A p p l i e s , T y p e = O u t a g e Type,

HA=Unplanned, CO=Planned, CA=Both
Availability Attribute Frequency Duration Scope Type
Automated Operations X X X CA
Automated Failure Detection, X X X HA
Recovery/Bypass, Reconfiguration
Security Management X CA
Application Design Standards X X X CA
System Design Standards X X X CA
SLA Management X X X CA
Corrective Action Management X X X CA
Change Management X X X CA
Operations standards X X X CA
Non-Disruptive Change X X X CA
Implementation
Batch Management X X X CA
Online Housekeeping X X X CA

13.2 Introduction
Let us assume you have bought the latest generation S/390 processors and have
attached RAID DASD through ESCON and ESCON Directors in a fully
fault-tolerant configuration. You have also started to use techniques like Remote
Copy to ensure access to the latest copies of your data. You have introduced a
Parallel Sysplex to support critical business services. So, will these steps
automatically deliver continuous availability?

The answer is no !

All of these technology solutions supply important facilities to help achieve

continuous availability, but systems management is another key factor in the
equation. In fact, it can be argued that systems management is the most
important factor in achieving continuous availability.

Systems management is a high-level set of applications required to run a

computer complex. They represent the manual processes and procedures that
have gradually been moved online to be self-managed by the IT environment. If

 Copyright IBM Corp. 1998 229

 there are weaknesses in this area, you will not achieve continuous availability
through buying new technology alone. Systems management underpins all of
your other efforts to meet (or exceed) your service targets. As Table 12
suggests, systems management has a much wider influence on all aspects of
continuous availability than any new technology.

Figure 85 is taken from System/390 Continuous Systems Availability published in

1993. This diagram positions the various elements that combine to deliver
continuous availability.

Figure 85. Continuous Availability Requirements

The diagram indicates that continuous availability is built from two main
elements, high availability and continuous operation. These elements are then

230 Continuous Availability S/390 Technology Guide

broken down into smaller groups such as Fault Avoidance, Fault Tolerance, and
so on. These groups are then divided again into smaller topics.

A chain of dependence is formed reading from left to right. For example,

continuous availability cannot be achieved unless both high availability and
continuous operation can be successfully performed. In turn, continuous
operation requires that availability management, non-disruptive changes and
maintenance, and continuous applications can all be met before it can be fully
implemented, and so on. Some availability management topics are common to
both high availability and continuous operation.

Looking at the right hand column of Figure 85 on page 230, systems

management influences the following essential availability topics:
• Automated operations
• Automated failure detection, recovery/bypass, reconfiguration
• Security
• Application design standards
• System design standards (designed for HA)
• Service Level Agreement (SLA) management
• Corrective action management
• Change management
• Operation standards (designed for CO)
• Non-disruptive change implementation (software and hardware)
• Batch interference
• Online housekeeping (DB re-orgs/copies and application)
You may recognize these topics from Table 12 on page 229, as well.

At first, some of these challenges appear to be associated more with human

actions than with software or hardware products, but generally speaking, the
more automated control that is active in a system, the more effective the overall
systems management is. A correctly designed and constructed control and
management system will recognize and react to events much faster than a
human, and will also react in a predictable and consistent way--this is essential
for Continuous Availability.

Over the last few years, the growth of IT resource outside of the traditional
mainframe has multiplied the problem of effective systems management. It is no
longer acceptable to consider managing all of these resources independently.
They each support an important facet of your business and are increasingly
being combined for a service that needs two or more of these resources for its
delivery. In this distributed environment, an effective management solution must
also span the same resources. An integrated toolset that makes you as
comfortable controlling a S/390 Parallel Sysplex as an OS/2 Server from the
same terminal with the same guidance is no longer a luxury; indeed, continuous
availability cannot be achieved without it.

In order to implement an effective self-managing control system, the underlying

processes and procedures must be totally understood and documented. IBM
Global Services has many experienced consultants and practice leaders in this
area to ensure that a systems management project delivers the success that is
expected of it. Contact your marketing representative if you would like to
discuss this approach.

This chapter does not describe how to select and implement systems
management products or projects. It concentrates mainly on the S/390

Chapter 13. Systems Management 231

 mainframe arena and shows how some of the main IBM and Tivoli products can
help you achieve better availability. It shows how they relate to each other to
provide an integrated toolset for the S/390 mainframe.

The Tivoli products have traditionally been focused on the distributed

environment, and some of the products that extend support to these
environments are included, but this is by no means an exhaustive list. The
products are broken into the following categories, although sometimes there is
some overlap:
• OS/390
• Tivoli
• Batch
• Parallel Sysplex

Database management systems normally have their own specialized toolsets to

perform the online housekeeping tasks required by continuous operation These
are not covered in this chapter; refer to the chapters on IMS and DB2 for a
summary of IBM tools for these products.

Also, the subject of application design standards for continuous availability is

handled by the Application Design for Continuous Operation chapter in this book.

13.3 OS/390
The foundation for good systems management is designed into OS/390 and its
optional elements. These in turn are built on the 20-year experience of MVS as
a commercial platform supporting many different businesses around the world.
OS/390 contains several key components that provide facilities for the additional
systems management products to connect to and interact with. These
components and elements include:
• System Services
JES
DFSMS
TSO/E
ISPF
Automatic Restart Manager (ARM)
Workload Manager (WLM)
Sysplex Failure Manager (SFM)
Coupling Facility Resource Manager (CFRM)
• Systems Management Services
Resource Measurement Facility (RMF)
System Modification Program (SMP/E)
System Display and Search Facility (SDSF)
Hardware Configuration Dialogs (HCD)
Recoverable Resource Management Services (RRMS)

232 Continuous Availability S/390 Technology Guide

 Most of these are probably well known from the traditional MVS environment,
but perhaps ARM and RRMS are worth a few more words. WLM, SFM and
CFRM are included in 13.6, “Parallel Sysplex Control” on page 244.

13.3.1.1 Automatic Restart Manager (ARM0

ARM is a component that helps availability by providing a fast, efficient restart of
critical applications, such as batch jobs or started tasks. ARM is used to restart
them in the event of an abend, a system failure or, if used in a sysplex, when a
system is removed from the sysplex. It can be integrated with other automation
and control products (for example System Automation for OS/390 and OPC,
which are described later) to ensure that only a single product takes control of
restart and recovery.

13.3.1.2 Recoverable Resource Management Services

RRMS extends transaction processing services by providing coordination of
2-phase commit for distributed transaction, and resource recovery backout,
forward recovery, and multi-system restart.

IMS, CICS, and DB2 provide their own equivalents of these services, but can
make use of this OS/390 facility to extend commit processing management
across a distributed environment. It also means that batch programs can make
use of this and take regular syncpoints. This may make each job run slightly
longer, but in the event of a failure, the recovery is much faster. It may also
mean fewer tape copies, as the updates are being more carefully managed now
during the batch process, and regular incremental tape copies are no longer
needed.

13.3.1.3 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, ARM delivers function to assist in:
• Automated operations
• Automated failure recovery/bypass or reconfiguration
• System design standards
• SLA management
• Operations standards

RRMS delivers function to assist in:

• Automated failure recovery/bypass or reconfiguration
• System design standards
• SLA management
• Operations standards

13.3.2 Additional OS/390 Products

13.3.2.1 Hardware Configuration Manager

Hardware Configuration Manager (HCM) is not part of OS/390 and is a separately
orderable product, but it can be thought of as an extension to HCD. It allows you
to use the facilities of a graphics workstation when building or changing the I/O
definition file used by HCD. It is much easier to see and check what is being
prepared when working in this way than when using the HCD text panels.

HCM can also store information with each connection and unit that is not related
to the system definition. For example, this means that information like asset

Chapter 13. Systems Management 233

 numbers or cable and patch panel numbers and so on can be built into the
workstation view of the system configuration. HCD does not use these, but an
operator or hardware engineer would find this information essential when trying
find a problem on a channel path. It makes HCM a much more useful tool than
simply a new front-end for HCD, as it also allows some physical site information
to be recorded and used.

13.3.2.2 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, HCM
(and HCD) delivers function to assist in:
• SLA management
• Change management
• Operations standards
• Non-disruptive change implementation

13.4 Tivoli Management Environment (TME) 10

Tivoli Systems, Inc. was founded in Austin, Texas in 1989. The TME product line
was developed and had become a leader in distributed systems management
when the company merged with IBM in March of 1996. Before the merger, IBM
had its own systems management product line known as SystemView. The
merger allowed the best of both product lines to be combined into a single
family called TME 10.

TME 10 is an open, highly scalable, cross-platform management solution. It

ranges from the mainframe to the desktop, offering management of all systems,
applications, and the network that together support a typical modern business.

Many products have and will be merged and then supported as a single
application. The offering keeps the TME name due to the fact that the products
will continue to be built around the object-oriented (based on CORBA standards)
Tivoli Management Platform. Some products still hold the SystemView reference
in their name, but this will disappear with future releases.

The Tivoli framework breaks systems management into four distinct areas:
• Security management
• Operations and administration
• Deployment management
• Availability management
For S/390 mainframe management, the products that fit in these categories are
shown in Table 13

Table 13 (Page 1 of 2). S/390 Tivoli Framework Products

Availability Operations & Security Deployment
Management Administration Management Management
TME 10 NetView TME 10 OPC OS/390 Security NetView D M
for OS/390 Server
System Info/Man
Automation for
OS/390

234 Continuous Availability S/390 Technology Guide

 Table 13 (Page 2 of 2). S/390 Tivoli Framework Products
Availability Operations & Security Deployment
Management Administration Management Management
Performance ADSM
Reporter for MVS
NetView
Performance
Monitor

Further information on Tivoli and TME 10 products for other platforms is

available at the World Wide Web site:
http://www.tivoli.com

The systems management products that contribute most to continuous

availability are described briefly in 13.4.1, “TME 10 NetView for OS/390,” and a
summary of which continuous availability areas they influence is added at the
end of each. Only the products with most impact on continuous availability are
included here. These come from the availability management and operations &
administration groups.

13.4.1 TME 10 NetView for OS/390

TME 10 NetView for OS/390 is a program for managing systems and networks
through graphical display and automation. It combines the functions of Tivoli′ s
S/390-based network management products, IBM NetView for MVS/ESA, IBM
NetView MultiSystem Manager, and IBM Automated Operations Network/MVS,
into a single integrated product.

TME 10 NetView for OS/390 provides:

• Management of a heterogeneous network environment
• Graphical views of the network′s health at a single glance
• Management of failed resources directly through exception views
• Extensive drop-in network automation capabilities
• Extensible support in both automation and topology discovery
• Dynamic topology and status information to NetView′s Resource Object Data
Manager in real time for both mainframe and distributed resources
• Collection of non-SNA topology data and automatic population of the
centralized RODM

NetView has been the core product of IBM′s mainframe and network
management for over 10 years, and its position has been kept in the Tivoli
portfolio. It provides the TME 10 automation services for S/390, and through the
TME 10 Global Enterprise Manager it can exchange events, commands, and
topology data with the TME 10 Enterprise Console and other TME 10 distributed
products. Many of the other S/390 management products build on the NetView
platform to extend its scope and capability as a manager. It delivers the function
to perform tasks like events management, topology management, and network
management on a single mainframe, a Parallel Sysplex, or a complex network.

Chapter 13. Systems Management 235

 13.4.1.1 Systems Management Availability Areas
Using the list of availability topics shown in Table 12 on page 229 of this
chapter, NetView delivers function to assist in:
• Automated operations
• Automated failure detection, recovery/bypass
• SLA management
• Operations standards
• Corrective action management

13.4.2 System Automation for OS/390

System Automation for OS/390 allows OS/390 and MVS/ESA V5 installations to
manage system operations, remote processors, and I/O operations from a single
point of control.

System Automation for OS/390 is built from:

• Automated Operations and Control
• ESCON Manager
• Target System Control Facility
• Automation Towers for CICS/IMS/OPC

AOC, along with the set of Automation Towers, aims to deliver a complete
automation solution for the mainframe environment. Instead of providing an
environment for you to develop automation routines, it delivers drop-in
automation for starting, shutting down, and re-starting MVS address spaces and
many other operator tasks. It also delivers an automation framework which
ensures that defined critical resources receive the correct levels of service.
These resources include things like systems, DASD volumes, or tapes.
Information is collected from RMF for them and used to drive automation. This
means that both performance-driven automation and event-driven automation
can be used to improve the management of your system.

ESCON Manager works dynamically with all OS/390 systems, HCD, the channel
subsystem microcode, and the installed hardware to build and maintain a picture
of any I/O device selected. It also works with other ESCON Manager
components on other OS390 images to obtain their view of the same hardware
components. For example, by selecting a DASD device, the display shows
whether the device is online or offline, shared between multiple systems,
whether it is attached to an ESCON channel or parallel channel, whether it is
attached through an ESCON Director, whether or not the ESCON channel is
shared between multiple LPARs, and so on. This information is displayed in a
set of colors and shaped icons to represent the different components. Selecting
individual icons allows more detailed information to be displayed. ESCON
Manager performs these tasks regardless of whether the component is attached
to a parallel channel or to an ESCON channel. It can also discover and display
information on other hardware items associated with a Parallel Sysplex (coupling
facility and links), and also the Open Systems Adapter (OSA) used on S/390
systems.

When the status of a device has been determined, ESCON Manager can manage
any change (vary offline or online) decided by the operator and then schedule
this change across all sharing systems. The scope of this vary command can
also be limited if required. It contains a very sophisticated checking system to

236 Continuous Availability S/390 Technology Guide

ensure that the command is valid and legal for the device, and also a backout
facility to restore the status of the device if anything prevents the command from
completing successfully. It can also be used to check the switch port
connections and attributes in an ESCON Director and then change them as
required. This makes ESCON Manager the safest way to handle vary off/on
changes in a multi-system environment.

The Target System Control Facility (TSCF) extends the control, monitoring, and
automation capabilities of NetView. NetView provides tools to manage systems
that are up and running. With TSCF, it then has the ability to initialize, configure,
recover, and shut down systems, and to monitor multiple systems and respond
to a variety of detected conditions.

You designate one processor called the focal-point system to control your
complex. Each system that the TSCF focal-point system manages is called a
target system. Target systems can be placed at dial-up line distances, so the
extent of control is potentially large. Some examples of remote control provided
by TSCF are:
• Perform a power-on reset of a target system
• Perform an initial program load (IPL) of a target system
• Set the target system′s time-of-day (TOD) clock
• Synchronize clocks on all target systems
• Configure the target system
• Detect and respond to target system messages
• Detect disabled console communications facility (DCCF) situations
• Detect wait states at the target system
• Use commands to simulate console keys
• Shut down the target system
These functions require a connection to the hardware management console of
the processor being controlled, and also a connection to any LPARs that need to
be monitored and controlled. Many different types of connection are possible to
perform this remote connection including SDLC links and LANs. TSCF Planning
and Installation gives detailed information of the different configurations that
TSCF can use.

Individually, these products offered a comprehensive range of effective

mainframe management services, but when integrated, they represent the basic
building block of single point of control operation.

13.4.2.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, System Automation for OS/390 delivers function to assist with:
• Automated operations
• Automated failure detection, recovery/bypass
• SLA management
• Operations standards

Chapter 13. Systems Management 237

13.4.3 Performance Reporter for MVS (PR/MVS)
PR/MVS, part of SystemView for MVS, is a systems management tool that
processes and transforms detailed measurement data recorded by other
products. It supports the MVS Workload Management concept to provide
customers with performance data that relates to business-oriented workloads
and assists in managing service levels efficiently. It complements other IBM
strategic products in the same area, for example RMF and NPM, but also
collects these products′ performance log data together with other log data from
all parts of the enterprise. This is used as the basis for historic and trend
reporting and analysis. PR/MVS is part of SystemView for MVS. The features
for reporting on are:
• System Performance
• IMS Performance
• CICS Performance
• Network Performance
• AS/400 System Performance (SP400)
• UNIX Performance
• Capacity Planner
• Reporting Dialog/2
The analyzed data is stored in a DB2 database. Once stored, the data can be
extracted and formatted into reports that focus on key business areas such as
availability management, performance management, and service-level reporting.

Service Level Reporter Version 3 (SLR) will be maintained for managing and
reporting of systems measurements data in a non-DB2 environment.

13.4.3.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, PR/MVS delivers function to assist with:
• Automated operations
• SLA management

13.4.4 NetView Performance Monitor (NPM)

NetView Performance Monitor (NPM) manages performance and accounting data
for your communications networks, and also assists with problem determination.
Networks are becoming more complex and varied, with an increasing number of
devices and protocols being introduced into the enterprise. This places greater
pressure on performance and capacity monitoring and management tasks to
ensure that service meets end-user requirements, and to plan upgrades or
remedial action before users are impacted.

NPM can perform the data collection task from VTAM on the mainframe out
across the network, and with PR/MVS this can be interpreted and displayed to
help you keep track of resources.

238 Continuous Availability S/390 Technology Guide

 13.4.4.1 Systems Management Availability Areas
Using the list of availability topics shown in Table 12 on page 229 of this
chapter, NPM delivers function to assist with:
• Automated operations
• SLA management

13.4.5 TME 10 Operations Planning and Control (OPC)

TME 10 OPC provides solutions to problems in production workload
management. It maximizes the throughput of work, optimizes resources, but lets
you intervene manually when required. It constructs operating plans based on
your descriptions of the operations department and its production workload.
These plans then provide the basis for your service level agreements and give
you a picture of the production workload at any point in time. You can also
simulate the effects of changes to your production workload and resource
availability by generating trial plans. Effective planning is a major step to
achieving higher availability.

From a single point of control, OPC analyzes the status of the production work
and drives the processing of the workload according to installation business
policies. OPC also includes a graphical user interface for application description
that can significantly improve productivity. It supports a multiple-end-user
environment, enabling distributed processing and control across sites and
departments within the enterprise.

OPC also offers:

• Agents for controlling the workload on non-S/390 platforms
• Integration with other systems management applications
• Interfaces to applications running in Systems Application Architecture
environments

To form an integrated Systems Management solution, OPC interfaces directly

with a number of IBM/Tivoli and non-IBM products. A selection is shown below.
• NetView
OPC lets you schedule WTO operations in conjunction with the production
workload processing. Alerts are passed to NetView in response to situations
that occur in processing the production workload. NetView can trigger
OPC/ESA to perform actions in response to situations it detects.
• RODM
The Resource Object Data Manager (RODM) provides a central location for
storing, retrieving, and managing the operational resource information
needed for network and systems management. You can map an OPC/ESA
special resource to a RODM object. This function lets you schedule the
production workload considering actual resource availability, dynamically
updated.
• Systems Automation for OS/390
System Automation for OS/390 initiates automation procedures that perform
operator functions to manage MVS components, datasets, and subsystems.
Additionally, SAfOS390 includes an automation feature for OPC.
• Info/Man
Info/Man supports the administration of the system management process of
an enterprise′s hardware and software resources. An interface to

Chapter 13. Systems Management 239

 Information/Management is provided for reporting problems detected in the
production workload processing.
• CICS and IMS
OPC lets you schedule the starting and stopping of started tasks. Because
OPC tracks the status of started tasks, you can serialize work, such as
backups of your transaction databases, according to the status of your CICS
or IMS subsystems.
• OS/390 and JES
OPC uses standard interface points to obtain information about the OS/390
workload.

13.4.5.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, OPC delivers function to assist with:
• Automated operations
• SLA management
• Batch interference
• Automated failure detection, recovery/bypass
• Corrective action management

13.4.6 Information/Management (Info/Man)

Info/Man is an integrated platform of tools, services, and interfaces for
implementing and enforcing policies and processes. It includes sample
applications supporting the management of:
• Call center/help desk
• Problem management
• Change management
• Configuration management

It gives you facilities to help:

• Manage your problems and changes
• Integrate your problem and change management processes
• Maintain your inventory and configuration in an orderly way
• Integrate with other systems management products (such as NetView and
other Tivoli TME 10 components)
• Maintain an organized framework so you can do more than react to
day-to-day problems--you can plan ahead
As a result, you can avoid service interruptions, maintain and control planned
changes, and better manage your software inventory.

Problem management automation from NetView is provided using the

Info/Man-NetView AutoBridge feature. Information/Management also has built-in
automation interfaces to these IBM products:
• Operations/Planning & Control/ESA (OPC(TM)/ESA)
• Performance Reporter for MVS
• Graphical Data Display Manager (GDDM(R)).
Additional automation can be built using Info/Man client APIs. These allow
customer and vendor client applications to access the Information/Management

240 Continuous Availability S/390 Technology Guide

 server on MVS. Client Info/Man APIs are available for Microsoft Windows NT,
AIX, HP-UX, Sun Solaris UNIX, OS/2, and CICS/ESA(R).

The Info/Man Web connector feature allows a graphical user interface to

Info/Man without requiring TSO, from an end-user′s platform of choice, using a
Web browser.

13.4.6.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, Info/Man delivers function to assist with:
• SLA management
• Corrective action management
• Change management

13.4.7 Adstar Distributed Storage Manager (ADSM)

ADSM is a family of software products offering an enterprise-wide unattended
backup and archive capability. It provides backup and archive services to more
than 25 multi-vendor client platforms, including industry-leading databases and
applications such as SAP. Data is stored on an ADSM storage management
server running on a variety of IBM and non-IBM platforms, from the S/390
mainframe to midrange, and UNIX. In addition to multiple client and server
platforms, ADSM supports multiple devices and communication protocols. This
broad range of support makes ADSM a comprehensive storage management
solution for heterogeneous networks.

ADSM provides significant systems management function in the area of

automation of data backups and storage management. A Hierarchical Storage
Manager service is available to move inactive data to a cheaper medium where
it can be recalled transparently to the user on demand. This offers a very
convenient automatic space management function, which can lower data storage
costs and operational costs.

ADSM also offers help in the area of disaster recovery. When preparing for the
possibility of severe service interruptions, you need to maintain a backup copy of
your vital business data offsite (offsite recovery media). When disaster strikes,
you need to know where your backup data is and what to do with it (recovery
plan) and what your vital business applications and their requirements are
(client recovery information).

Disaster Recovery Manager (DRM) is an optional feature that assists with

planning, preparing, and executing a disaster recovery plan. DRM enables an
ADSM-based recovery of business applications from offsite backup data. The
disaster recovery plan can be used to guide an administrator through disaster
recovery, as well as for audit purposes to certify the recoverability of the ADSM
server.

13.4.7.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, ADSM delivers function to assist with:
• Automated operations
• SLA management
• Corrective action management
• Operations standards

Chapter 13. Systems Management 241

13.4.8 TME 10 Global Enterprise Manager (GEM)
The TME 10 GEM is not a S/390 product but has been included here because of
its significance in overall end-to-end system management. The TME 10 Global
Enterprise Manager (TME 10 GEM) integrates the function of S/390 system and
network management solutions and Tivoli′s distributed enterprise solution. This
integration enables true network computing management across the enterprise
from a single console. Traditionally S/390 management has been performed
from a console directly connected to a S/390 mainframe. The GEM terminal can
be connected anywhere on a different platform and manage S/390 as another
service resource to support the overall business.

TME 10 GEM is the first software specifically designed to manage the strategic
business systems that run your company and make it competitive. It provides
management for business-critical applications that span both the S/390 and
distributed environments. The management and control is presented in a
business context and features:
• Application Policy Management, a Java-based application for managing
business systems that does the following:
Provides a business system view of all resources required to deliver
specific business services
Allows control of components of each business system with
cross-platform, single-action management
Automatically discovers and builds graphical views of the business
system components and their relationships
Allows a view of the business system from the standpoint of a specific
management discipline
Provides pre-built business views of TME 10 systems management
products and of Lotus Notes E-mail systems when used in conjunction
with TME 10 Module for Lotus Domino/Notes
Works with existing applications and management tools instead of
replacing them
• TME 10 GEM Management Integration Services, a bi-directional
communication between management centers of both S/390 and distributed
environments

13.4.8.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, TME 10 GEM delivers function to assist with:
• SLA management
• Operations standards

13.5 Batch Interference

One of the topic headings in Figure 85 on page 230 is Batch Interference. This
means that batch should consistently finish with the correct outputs before the
online transaction managers are due to be enabled for the next day′s work. In
many installations there is constant pressure on the Operations staff to ensure
that this target is met, particularly at peak business periods (for example, at
month end). Many operations are under pressure to extend the online services
availability period to take advantage of new markets or new ways of doing

242 Continuous Availability S/390 Technology Guide

 business with their customers. Again, this places more pressure on ensuring
that the batch job suite not only finishes on time, but takes less time.

In these cases, a product such as IBM′s Smartbatch for OS/390, which reduces
the elapsed time of batch jobs without requiring a major redesign or rewrite, is
of real benefit in improving availability.

13.5.1 Smartbatch for OS/390

IBM′s Smartbatch for OS/390 helps to meet these availability targets by using
techniques to speed up the flow of batch work through a processor or a group of
processors. These techniques are parallelism, I/O optimization, and workload
distribution.

Smartbatch for OS/390:

• Allows two or more jobs that formerly processed consecutively to now
process at the same time
• Allows individual job steps to run in parallel
• Reduces the number of physical I/O operations when possible
• Optimizes I/O for DASD and tape
• Routes job steps to a sysplex image best suited for that step
A job suite that can take advantage of these techniques can have its elapsed
time significantly reduced.

Even an installation with single processor may be able to find some advantage
from Smartbatch. The ability to run job steps or jobs at the same time, and I/O
reduction and optimization can be performed on a single processor. The overall
gain, however, may be dependent on CPU stress and storage use at the time of
the job run.

Full control can be retained, allowing a job to bypass Smartbatch processing if

required. An analysis tool within Smartbatch can also be used to gain an
observation of how the job normally processes, its data access pattern, and CPU
utilization. Using this information, which is retained in a history dataset,
Smartbatch can then decide how best to operate on the job when activated. The
Batch Accelerator component knows:
Which steps can be run in parallel
Which steps are CPU-intensive
Which steps are I/O-intensive
It attempts to achieve the best possible throughput for this job.

13.5.1.1 Function Cross-Reference

The following table shows which environments can use the various Smartbatch
functions. For more information about Smartbatch, refer to IBM Smartbatch for
OS/390 Overview .

Chapter 13. Systems Management 243

 Table 14. SmartBatch Functions for Various OS/390 Environments
OS/390 Environment
Sysplex
SmartBatch Function Single System
Monoplex Multisystem Parallel
JES2 JES3 JES2 JES3 JES2 JES3 JES2 JES3
Job Parallelism with
Yes Yes Yes Yes Yes Yes Yes Yes
Local-System Data Piping
Job Parallelism with
No No No No No No Yes Yes
Cross-System Data Piping

Step Analysis No No Yes No Yes No Yes No

Step Targeting No No No No Yes No• Yes No•

Step Splitting without Data

No No Yes No• Yes No• Yes No•
Piping
Step Splitting with
Local-System Data Piping No No Yes No• Yes No• Yes No•
Only
Step Splitting with both
Local-System and No No No• No No• No Yes No•
Cross-System Data Piping
I/O Performance
Yes Yes Yes Yes Yes Yes Yes Yes
Processing

Shared Record Positioning No No Yes No Yes No Yes No

BatchPipeWorks
Yes Yes Yes Yes Yes Yes Yes Yes
Commands
• This function is not automatic; your installation must manually modify JCL
and job scheduling to some extent, such as converting job steps to jobs.

Note: The source of this data is IBM Smartbatch for OS/390 Overview , GC28-1627

There is no conflict here with TME 10 OPC, since OPC controls when a job is
being presented for processing, and Smartbatch influences the way that a job
processes when released. The products are complementary and are both of
benefit for higher availability.

13.5.1.2 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, Smartbatch delivers function to assist with:
• Operations standards
• Batch interference

13.6 Parallel Sysplex Control

A fundamental design point in the development of Parallel Sysplex was that it
should appear as a single system image, and be easier to manage (to the
operator) than previous multiple systems. This places a higher demand on the
use of systems management products within the Parallel Sysplex. The products
covered so far form the basis for good systems management, either within a
single processor installation or in a Parallel Sysplex. This section highlights

244 Continuous Availability S/390 Technology Guide

 some of the additional systems management facilities that can be used to
effectively manage a sysplex, and get better availability for business services.

13.6.1 Sysplex Failure Manager (SFM)

SFM has a very fixed purpose in a Parallel Sysplex. It is directly designed to
support continuous availability. The SFM couple data set contains a selection of
SFM policies (although only one can be active at any one time), which define
how the sysplex should react when certain failures occur. Policies can be
switched dynamically. SFM allows automatic handling of:
• Connectivity failure, which is the inability of a sysplex member to
communicate with the control structures in the coupling facility.
• Systems failure, which is the inability of a member to update its status
record on the sysplex couple dataset within the allowed time interval.

Some of the actions that may be taken are:

• LPAR reset
• LPAR deactivation
• Redistribution of storage to other active LPARs
• Coupling facility structure rebuild
• Repartition the sysplex
S/390 MVS Parallel Sysplex Continuous Availability SE Guide contains guidance
on implementing SFM in a Parallel Sysplex.

13.6.1.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, SFM delivers function to assist with:
• Automated operations
• Automated failure detection, recovery/bypass, reconfiguration
• SLA management
• Operations standards

13.6.2 Workload Manager (WLM)

WLM is not simply a sophisticated performance manager, but plays an important
role in the area of continuous availability. It is because WLM can dynamically
adjust to changes in the workload arriving at the sysplex, or to a loss of
resources within the sysplex, that it becomes so critical to us.

For unplanned outages, WLM can redistribute the arriving work around the
remaining members of the sysplex. Then the dispatching priorities, storage
allocations, and other parameters affecting individual workload performance can
be rearranged to compensate for this extra work. The net result is continued
service for the end user, and with a transaction response as close to their
requirements as capacity allows. No other system complex has this ability to
react in such a positive way to major events without manual intervention.

For a scheduled shutdown of a major component, WLM works in the same way,
but in this case the changeover is probably much less dramatic. You will
probably use a management product like CP/SM to stop workloads from
processing in a much more controlled manner. This also makes the

Chapter 13. Systems Management 245

 redistribution a little more leisurely and controlled, but the end result is the
same: there is no loss of service for the end user.

In both cases, when the resource is ready to be returned to the sysplex, WLM
allows it to be brought into use transparently. Workload is sent to the new
resource and the workload automatically rebalances across the active members
of the sysplex.

An interesting side effect of WLM is the protection from “hung” systems resulting
from a CPU-intensive job being released at the wrong time. Because of the way
that WLM works, it prevents such a job from completely taking over the
processor on which it has been released. This means that although
performance will certainly be impacted on this processor, other work will
continue to process successfully, and the console will not be locked out. This
may not be a frequent occurrence in your installation, but preventing even one
such hit may be worthwhile.

13.6.2.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, WLM delivers function to assist with:
• Automated operations
• Automated failure detection, recovery/bypass, reconfiguration
• SLA management
• Operations standards
• Non-disruptive change implementation

13.6.3 Coupling Facility Resource Manager (CFRM)

CFRM controls the way that the coupling facility is used in the sysplex. The
CFRM couple dataset contains a set of policies, of which only one can be active
at any one time, which describe these actions. Again, the policy can be
switched dynamically. The policy describes the following parameters:
• Each coupling facility
• Each structure
• The size of each structure
• Preference list
• Exclusion list
• Rebuild percentage
• Dump space
All of these have some influence on availability of the services running on the
sysplex.
Coupling facility list All coupling facilities to be used
(including backups) have to be identified
here or they cannot be used.
Structures and size The information to be used by all
members and applications running on the
sysplex must be defined for use in the
coupling facility, along with its size.
(Storage needs to be managed on a
coupling facility, just as with normal
processors.)

246 Continuous Availability S/390 Technology Guide

 Preference and exclusion lists A preference list is a list of coupling
facilities in which a structure can be built.
An exclusion list is a list of coupling
facilities in which a structure should not
be built. These controls are for
performance and capacity.
Rebuild percentage This is a threshold that determines when
a structure should be rebuilt in an
alternate coupling facility because of
connectivity problems.
Dump space This is an allocated space to which a
structure can be dumped in the event of
a problem. This is important for
resolving what the problem was in a later
analysis.
Failing to specify any of these parameters correctly will influence how your
sysplex reacts, either on attempted startup, or after an error that should have
been transparent to users. It may cause recovery to be impossible (leaving the
whole sysplex down because an alternate coupling facility was not specified
correctly), or it may affect the recovery of only one application. This is one
reason why you should ideally have a test sysplex. It is important to check that
the effects of a dynamic policy change are as expected.

S/390 MVS Parallel Sysplex Continuous Availability SE Guide contains more

information on how to prepare a CFRM policy.

13.6.3.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, CFRM delivers function to assist with:
• Automated operations
• Automated failure detection, recovery/bypass, reconfiguration
• SLA management
• Operations standards
• Non-disruptive change implementation

13.6.4 CICSPlex Systems Manager (CP/SM)

Note: CP/SM is now integrated into the CICS Transaction Server for the OS/390
product.

Chapter 9, “CICS” on page 163 describes CICS can work in multi-region

operation. Within a Parallel Sysplex, the image of TORs and AORs can be
multiplied significantly across the members of the sysplex. The CICSplex idea
solved many problems associated with resource constraints in a single address
space and then within a single system, but it introduced other problems in its
management. For example, disabling a transaction when it may be eligible to
run in many different CICS regions on many systems becomes a complex task.

CP/SM was designed to perform the management of this new multi-region,

multi-system environment, but again a key design feature was that it should
appear as a single image tool. It provides function to:
• Monitor and manage all CICS regions
• Assist with workload balance across regions

Chapter 13. Systems Management 247

 • Dynamically map CICS resources
• Analyze and react to CICS events and events in related resources
• Set thresholds for event triggering
• Automate CICS operation
• Integrate with other management and automation tools

From the high availability point of view, the key functions of internally detecting,
analyzing, and reacting to events that may cause a loss of service are essential.
This can be done without reference to an external product, although generic
alerts for NetView can still be flagged. If the event cannot be handled internally,
then NetView may be the normal route to signal that additional handling is
required. CP/SM can also use the ARM component of OS/390 to handle the
restart of a CICS region according the the ARM policy that is active.

From the continuous availability point of view, the dynamic workload

management function is essential. CP/SM works very closely with WLM to meet
the transaction response goals set in the WLM Policy. WLM monitors and
manages the system resources and the priority of individual CICS regions in the
sysplex. It can also understand other dependencies. For example, if a CICS
transaction makes a DB2 call, WLM can influence the priority and resources
given to DB2 to assist the CICS transaction response goal. CP/SM can take
another approach: it can schedule the startup of additional regions to handle a
sudden high demand.

This workload management is important when adapting to the removal of

systems from the sysplex for planned or unplanned service. Assuming that
there is a reasonable amount of spare capacity still available within the sysplex,
not only will the business user′s transactions continue to be processed, but they
may even process with a very similar response time.

Another important function of CP/SM is the single point of control which allows a
view of all CICS systems and resources no matter which processor or image
these resources are running on. This is essential for the human interaction that
cannot be avoided in day-to-day work. The task of trying to work out what is
running where is taken away, leaving it much easier for people to understand
and manage CICS MRO operations.

13.6.4.1 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, CP/SM delivers function to assist with:
• Automated operations
• Automated failure detection, recovery/bypass, reconfiguration
• Operations standards
• Non-disruptive change implementation

13.7 Availability Measurement And Tools

248 Continuous Availability S/390 Technology Guide

13.7.1 Service Level Agreements
In this section, the function of service level agreements is discussed, as well as
tools to measure availability.

The human view of the performance of a system is often subjective, emotional

and difficult to manage to. However, meeting the business needs of the users is
the reason the system exists.

To solve the problem of matching business needs with subjective perception, the
concept of Service Level Agreements (SLA) was introduced.

First of all, you will have to define, for each on-line service, your own
requirements. This requirements, which can be called service level objectives ,
can include:
• The hours of service
• Maximum/minimum response time for different applications
• Maximum number of outages per time interval
• Mean Time To Repair (MTTR) and recovery time
• Process or print turnaround times
These service level objectives serve as input for determining your data
processing requirements. The service level objectives will have to be negotiated
with the end users and eventually be turned into Service Level Agreements. It is
recommended that objectives be documented in existing service level
agreements, or that new agreements be created if none exist.

To obtain more information on defining Service Level Agreements, refer to

Continuous Availability Systems Design Guide .

13.7.2 Availability Measurement

The lack of availability measurements opens the door to controversy. The
duration of an outage is subjective. A 10-minute outage today can easily
become a 20-minute outage three weeks from now in a service-level discussion
when no automatic availability measurement is provided to the end user.

Measuring availability allows you to answer the following questions:

• Which service level objectives can we commit to?
• Are service level objectives being met?
• Is it reasonable to raise the availability objectives?
• Where is the failure occurring?
• What is the cause of failure?
• Where should the effort be applied to improve availability?

It is obvious that without measurements, it is impossible to commit to service

level objectives. It is also impossible to be sure that the service levels are being
met, or to agree to raising them.

System-wide availability measurement lets you more quickly determine the

cause of failures, and establish a diagnosis. You can then recommend how and
where to apply your efforts. This is the approach commonly used in availability
reviews.

Chapter 13. Systems Management 249

13.7.3 Capacity Planning and Performance Management
An effective capacity planning process provides:
• A mapping of business objectives (user requirements) into quantifiable
information technology (IT) resources.
• Management-oriented reporting of service, resource usage and cost. This
makes clear, in an objective way, what is involved in providing users with
good performance.
• Input for making business decisions which involve IT.
• A way to avoid surprises.

Good capacity planning can contribute to your achieving your continuous

availability requirements.

Performance management and capacity planning have much in common.

Performance management concentrates on allocating existing resources to meet
service objectives. Capacity planning is a means of predicting resources needed
to meet future service objectives.

Similar approaches can be used for both. Here are some examples:
• Rules of thumb
• Comparison with other systems
• Parametric methods
− A transaction profile (10 read calls, two update calls, eight physical I/Os)
− Cost of function (CICS: 15ms per physical I/O in a 3390 non-cached)
• Analytic (queueing theory) models (for example, the IBM capacity planning
tool CP90)
• Simulation using a computer program that has the essential logic of the
system (for example, the Snap/Shot modelling system from IBM)
• Benchmarks, Teleprocessing Network Simulator (TPNS)

In addition, performance and capacity work tend to require similar input data
from the system. In particular, both require resource consumption by workload
data

13.7.4 Resource Management Facility (RMF)

Resource Management Facility (RMF) issues reports about performance
problems as they occur, so that your installation can take action before the
problems become critical.

Your installation can use RMF to:

• Determine that your system is running smoothly
• Detect system bottlenecks caused by contention for resources
• Evaluate the service your installation provides to different groups of users
• Identify the workload delayed and the reason for the delay
• Monitor system failures, system stalls, and failures of selected applications

250 Continuous Availability S/390 Technology Guide

 RMF comes with three monitors, Monitor I, II and III. Monitor III, with its ability to
determine the “cause of delay,” is discussed first:
• RMF Monitor III
Monitor III provides short-term data collection and online reports for
continuous monitoring of system status and solving performance problems.
Monitor III is a good place to begin system tuning. It allows the system tuner
to distinguish between delays for important jobs and delays for jobs that are
not as important to overall system performance.
• RMF Monitor I
Monitor I provides long-term data collection for system workload and
resource utilization. The Monitor I session is continuous , and measures
various areas of system activity over a long period of time. You can get
Monitor I reports directly as real-time reports for each completed interval
(single-system reports only), or you can run the postprocessor to create the
reports, either as single-system or as sysplex reports. Many installations
produce daily reports of RMF data for ongoing performance management.
• RMF Monitor II
Monitor II provides online measurements on demand for use in solving
immediate problems. A Monitor II session can be regarded as a snapshot
session. Unlike the continuous Monitor I session, a Monitor II session
generates a requested report from a single data sample. Since Monitor II is
an ISPF application, you might use Monitor II and Monitor III simultaneously
in split-screen mode to get different views of the performance of your
system.

In addition, you can use the Spreadsheet Converter for further processing the
measurement data on a programmable workstation with the help of spreadsheet
applications.

13.7.5 Performance Reporter for MVS

Complex computing environments require effective management and control of
system resources. Performance Reporter for MVS provides the information that
is needed to maintain a high level of service.

Performance Reporter for MVS is a systems management tool that collects and
transforms detailed measurement data into meaningful reports that focus on
such key business areas as availability and performance management, and
service level reporting. This information is available at various levels (user,
application, subsystem, processor and network). Reports are available for
management and technical specialists.

Performance Reporter for MVS′s use of DB2 provides easy access to the stored
performance data, and makes access to distributed data possible. Performance
Reporter for MVS significantly improves user productivity and ease of
implementation through powerful dialogs and online help facilities. Based on the
user′s definition, Performance Reporter for MVS selects which data to collect
and how long to keep it. It also includes a set of reports that address the user′ s
requirements.

Performance Reporter for MVS consists of a base, and several features

addressing system, network, CICS, IMS, AS/400, and UNIX performance, a
capacity planning feature, and an OS/2 reporting dialog feature.

Chapter 13. Systems Management 251

 • The Performance Reporter for MVS base provides:
− Log and record definitions for all products supported
− Data collection and database maintenance
− Dialogs for reporting and administration
• The Performance Reporter for MVS, IMS, CICS, and network performance
features provide application task support such as:
− Reports
− Help and documentation of methods for data analysis

Users can choose to use either a non-programmable workstation or an OS/2

programmable workstation when performing the application tasks, such as
performance analysis and utilization analysis. Performance Reporter for MVS
can also create information/management problem records from the data
collected. The user defines the conditions for a problem (for example, response
time...one second).

When processing System Management Facility (SMF) data from MVS/ESA

Version 5, Performance Reporter for MVS accepts new data from the Workload
Manager (WLM) and the coupling facility, as recorded by MVS and Resource
Management Facility (RMF*). Performance Reporter for MVS processes data for
both modes of operation (goal mode and compatibility mode). This is
implemented so that a workload can be tracked across a shift from compatibility
mode to goal mode and vice versa. Consolidated sysplex-wide reports can be
produced and viewed on either the RD/2 workstation (OS/2) or on the host.

To allow for improved and more efficient service level management,

Performance Reporter for MVS supports the new MVS Workload Manager, which
allows you to explicitly state your system performance expectations using
common business terminology generally used to document service level
agreements.

Performance Reporter for MVS is the follow-on product to Service Level Reporter
(SLR, 5665-397). Service Level Reporter Version 3 will be kept up-to-date for
supported products and is the preferred solution for managing and reporting of
systems measurements data in a non-DB2 environment.

13.7.6 Service Level Reporter V3

SLR helps you to measure all important aspects of your computer system:
central, distributed, and network.

By selecting the areas and level of detail that need attention, you can set
objectives for system management areas such as:
• Service levels
• Availability
• Performance management
• Capacity planning
• Operations
• Problem and change management
• Accounting

252 Continuous Availability S/390 Technology Guide

 By giving these areas the proper attention, you get a smoother-running
organization with improved cost effectiveness and user satisfaction.

13.7.6.1 Availability Report

SLR measures availability and presents the results in reports that are
meaningful to managers both within and outside the data processing
department. These reports show availability of services at the users′ locations,
and within the host processor.

To produce these reports, SRL tracks the availability of hardware and software
starting at the processor and ending at the users′ terminals.

If you describe your resource hierarchy to SLR, it identifies a resource as

unavailable whenever a resource higher in the hierarchy goes down. For
example, SLR can identify batch job processing as unavailable if JES2 becomes
unavailable.

SLR also identify a resource as unavailable if it is inactive too long. For

example, paging activity records may indicate that RMF is active. If no such
records occur during a given period, RMF is marked as unavailable.

Availability data, to be most meaningful, must show whether a user service was
available when it was supposed to be. SLR availability reporting gives you this
information. If you enter your availability schedules in SLR, your reports will
show availability during these scheduled periods.

Your SLR availability reports can include data such as:

• Details and overviews for any critical item of hardware or software
• Periods when the resources is planned to be up
• Availability objectives for each resource
• The times that resources are unavailable, even when the resource was not
able to send a down record

You can report on, for example:

• An 3174 subsystem control unit, or equivalent, using data from NetView
• A Network Control Program (NCP)
• A group of CICS systems
• TSO

13.7.6.2 Systems Management Availability Areas

Using the list of availability topics shown in Table 12 on page 229 of this
chapter, RMF, PR and SLR deliver function to assist with:
• SLA tracking and reporting

13.8 Summary
It would be convenient to be able to draw a map at this point to chart progress
and directions on the road to systems management suitable for supporting
continuous availability. Unfortunately, it is not so simple.

The OS/390 base and optional products are an obvious first and second step.
The IBM Open Blueprint shows the systems management facilities, along with
the operating system resources, as forming the backbone to support all other

Chapter 13. Systems Management 253

 services. OS/390 with its integrated function (including the optional components)
fills this position very well.

Using NetView is a good third step, with its mainframe and network monitoring
and control abilities, but from this point on the priorities are probably personal.

Products like ADSM, OPC, and System Automation may seem obvious next
targets, but many would argue that the creation of a sound problem and
management database on Info/Man is the next best step (assuming you do not
already have an effective one in operation).

The important point is that all of the topics in Table 12 on page 229 need to be
covered by a process, ideally an analytical, reactive, system-based process, in
order to achieve the combination of high availability and continuous operation.
This chapter highlights some of the IBM S/390 software products that do this.

254 Continuous Availability S/390 Technology Guide

Chapter 14. Sharing Data among Heterogeneous Platforms

In the dedicated computing model of the 1970s and 1980s, storage requirements
were dominated by centralized data and centralized management within an
environment of homogeneous platforms.

In the late 1980s and early 1990s, the client/server computing model distributed
data across the enterprise on a wide variety of heterogeneous multi-platform
environments. This raised new requirements for accessing the same data from
this heterogeneous world. These requirements were usually addressed by
copying the host (server) data and transmitting it to the different distributed
platforms across the enterprise.

Nowadays, the networking computing model and data mining applications

require increasing access to the same data and at the same time as the host
environment, resulting in a new requirement for universal data access.

14.1 Universal Data Access

The challenge of universal data access requires storage solutions with broad
adaptability, yet which are tuned to specific storage solutions. Today these
storage solutions are no longer just disk, or just tape, or just software tools, but
powerful storage servers where multiple media types such as disk and tape are
integrated with rich software functions.

Universal data access requires not only the physical connection and
communication between many types of technologies, but also the ability to
actually make sense of the data once you get to it. Therefore, universal data
access means being able to get to any data anywhere at any time, while
protecting data integrity.

Universal data access can be achieved through:

• Data Copy Sharing
This is a process of copying data between multiple systems, typically with
unlike data structures (for example, copying data from an OS/390 to a UNIX
system to enable a data mining application).
• Storage Sharing
This is the physical partitioning of capacity in a single storage complex for
multiple servers to allow dynamic allocation or re-allocation of storage even
between dissimiliar disk architectures such as UNIX and AS/400.
• Real Data Sharing
This represents the ultimate in data access and requires only a single copy
of the data for multiplatform access and update.

While there are many products that support data copy sharing, both storage
sharing and data sharing among heterogeneous platforms cannot be achieved
by any of the S/390 devices available today. However, we will also discuss these
ways of sharing data in order to have a better understanding about the possible
evolution of the products.

 Copyright IBM Corp. 1998 255

14.2 Data Copy Sharing
In data copy sharing, neither the device capacity nor the data are shared
between different platforms; only a copy of the data is moved from one platform
to the other.

This implies two problems in terms of availability:

• Time is required to produce and transmit the copy of the data.
This directly affects both data and application availability at the source site,
but the applications at the destination site are affected for only the length of
time it takes to switch the data.
• The copy of the data is outdated at the same time it has been produced.
This obviously influences the possible use of the data at the destination site
since it can only be used for reporting purposes. The data represents the
source data situation at the time of the beginning of the copy operation, and
all modifications that occurred to the source data after this point is
unavailable at the destination site.

Figure 86 shows a data copy sharing example.

Figure 86. Data Copy Sharing

The current implementation of data copy sharing can be ranked in three

categories, as discussed in the following sections:
• File transfer
• Serial transfer
• Data piping transfer

256 Continuous Availability S/390 Technology Guide

14.2.1 File Transfer
This technique is normally used when neither system can access the other′ s
devices directly through their channels.

The data transfer process is performed in three different steps:

1. Data extraction at the source system
A database extract program filters and selects records from one or more
databases on one or more storage subsystems.
For example, a 1TB DB2 database may create a 150GB sequential “flat file”
stored on disk at the source site. Assume that this operation takes about
two hours.
2. Data transmission between source and destination systems
Step 2 cannot start until Step 1 has completed. The file transfer transfers the
data from the source system to the destination system via a
program-to-program communication (either by network or by channel) with
its image that is running on the destination system.
The length of time it takes for this operation to complete depends on the
network speed, but if it is a fast telecommunication line (for example,
34Mbps) is used, the 150GB flat file can be transferred in about 10 hours.
3. Database reload
At the destination system, the Oracle database load program cannot start
until the entire file has been transferred and closed.
Assume that the load process takes another two hours.

Figure 87 shows an example of such a file transfer. Id 15is example, both the
source system and the destination data were unavailable for about two hours
each. Moreover, at the end of the process, the destination system application is
working on a data image that is as old as the duration of the entire process (14
hours).

Figure 87. File Transfer

Chapter 14. Sharing Data among Heterogeneous Platforms 257

 Although this technique has been in use a long time, the only way to improve the
total availability is to reduce the time required to extract, transfer, and reload the
data; that is, by adopting faster data transfer solutions. This is discussed in the
following sections.

14.2.1.1 CNT FileSpeed

FileSpeed moves the flat file at standard ESCON/SCSI speeds and eliminates the
host TCP/IP workloads, as shown in Figure 88. Compared with the preceding
example, at ESCON speed (17 MB/sec) the transfer completes after about 1.5 to
2 hours; thus the entire transfer process takes only six hours.

Figure 88. Faster File Transfer with CNT FileSpeed

Moreover, the combination of CNT FileSpeed with IBM RAMAC Virtual Array
SnapShot high-speed database replication unlocks the OS/390 database for
extraction without impacting normal batch or online processing at the point in
time when the service is required.

In the previous example, the database data extract operation took two hours and
the S/390 DB was unavailable for use by batch or online applications during that
entire length of time.

In contrast, while this operation still takes two hours with SnapShot, it does not
require taking the S/390 DB offline, thus improving the S/390 data and
application availability.

14.2.2 Serial Transfer

This technique is normally used when one of the two systems can access the
other′s devices directly through one or more of their channels, as shown in
Figure 89 on page 259.

In this case, the data transfer process is performed as a two-step process:

1. Data extraction at the source system

258 Continuous Availability S/390 Technology Guide

 A database extract program filters and selects records from one or more
databases on one or more storage subsystems. This operation takes the
same time as in the previous FTP example, that is, a 1TB DB2 database may
create a 150GB sequential “flat file” stored on disk in about two hours.
2. Database reload
At the destination system, the Oracle database load program cannot start
until Step 1 is ended, but the load operation is now performed by directly
reading the flat file on the source system disk at channel speed.
Assume that the load process takes another two hours.

Figure 89. Serial Transfer

Compared with the example in 14.2.1, “File Transfer” on page 257, this solution
has not improved the availability of both systems, but the image of the data in
this case is only four hours old.

Also in this case, a copy of the source database using the SnapShot function of
IBM RAMAC RVA allows the extract program to work on an immediate copy of
the source database, without requiring you to take both the database and the
application offline.

14.2.2.1 IBM Client Input/Output Sockets (CLIO/S)

IBM Client Input/Output Sockets is a high-speed data replication solution
between RS/6000 AIX and OS/390 using a dedicated ESCON channel; thus, the
RS/6000 requires an ESCON adapter.

Data transmission is controlled by API functions on both the S/390 side and the
RS/6000 side. The AIX API allows the application to “think” it is writing data to a
local tape, but actually the data is written to tape or disk storage on the S/390 at
very high ESCON speeds.

The process of this operation in the opposite direction is similar.

Chapter 14. Sharing Data among Heterogeneous Platforms 259

 Figure 90. Serial Transfer with I B M CLIO/S

14.2.3 Data Piping Transfer

This technique is an evolution of the previous one and implies not only that the
two systems can access each other′s devices through their channels, but also
that there is a connection between the two database managers (or applications).

It is a single-step process: the data extracted by the database extraction

application on the source side is transferred through a “pipe” directly to the
database load application on the destination side, as shown in Figure 91.

Figure 91. Data Piping Transfer

260 Continuous Availability S/390 Technology Guide

Compared with the previous solutions, data piping does not need a temporary
copy (flat file), and it usually performs the copy process in less time than file
transfer or serial transfer (which is two hours, in our example).

14.2.3.1 CNT High Speed Piping FileSpeed

FileSpeed offers a special high-speed piping solution that can directly transfer
data from an OS/390 server database extraction program to a UNIX/Intel server
database loader program in a single step without intermediate flat file creation
and movement.

How does this application-to-application process really work?

1. The OS/390 extraction program is modified using the PDM Application
Program Interface (API). This supports writing data directly from the OS/390
application to PDM and thus through the Channellink hardware to the UNIX
platform. The source data that is being transferred can reside in any storage
subsystem attached to the S/390 server.
2. The UNIX platform uses a UNIX function called “ n a m e d p i p e s ” to directly
route the data to the Oracle (or any other) database load program without
the creation of a flat file. PDM synchronizes the data transfer, so no manual
coordination is required.
Figure 92 shows an example of data piping with CNT FileSpeed.

Figure 92. Data Piping with CNT FileSpeed

Again, the ability to snap a copy of the source database using the IBM RVA
SnapShot function not only allows continuous availability for both data and
application at the source side, but at the end of the process, the data at the
destination side is only two hours old.

Chapter 14. Sharing Data among Heterogeneous Platforms 261

14.3 Storage Sharing
In storage sharing, only the capacity of a specific device is shared between two
or more different platforms; the data is not shared.

Using storage sharing is a good way to lower the price and total cost of
ownership. However, at the moment, there are no devices available to attach
S/390 and the other platforms directly.

In terms of continuous availability, storage sharing allows the other platforms to

benefit from some of S/390 storage devices continuous availability features such
as Remote copy, Shapshot, and Concurrent Copy.

Figure 93 shows an example of storage sharing.

Figure 93. Storage Sharing

An example of storage sharing is the possibility of having UNIX data coexist on

the same S/390 devices as MVS data by using OS/390 Open Edition Hierarchical
Structured File. This would allow the UNIX data to benefit from all the
continuous availability features now available to the MVS world.

14.4 Real Data Sharing

In “real” data sharing, only one copy of the data exists, and all applications
running on the different platforms which share that device can access that data
in both read and write mode.

Having a single copy of data is a great improvement in term of continuous

availability since it avoids all the delay involved in producing and transmitting
copies of data on different platforms.

Figure 94 on page 263 shows an example of real data sharing.

262 Continuous Availability S/390 Technology Guide

Figure 94. Data Sharing

Real data sharing means that all applications are able to read the other′s data
structure, or they use a common data structure, but they must also guarantee
data integrity on write operations.

In a shared environment, the multiple copies of the database manager uses two
methods to provide this guarantee: locking, and buffer consistency. These are
explained as follows:

Locking: When a database is accessed, the database manager uses locks to

ensure that applications work with a valid copy of the data. If one application is
updating a section of data, a lock mechanism prevents other applications from
accessing that section until the update completes.

Buffer Consistency: To improve performance, database managers hold data in

buffer pools in each system where the database manager is active. If the data
contained in the buffer is updated by an application that shares it, the database
manager must have mechanisms to invalidate the copy of the data it keeps in
storage.

In a shared environment, these data integrity mechanisms must be reflected

across all systems.

These facilities are quite easy to implement in a homogeneous Advanced

Coupled System; that is, both S/390 and RS/6000 platforms can work in data
sharing.

Figure 95 on page 264 shows an example of homogeneous platform data

sharing.

Chapter 14. Sharing Data among Heterogeneous Platforms 263

 Figure 95. Homogeneous Platform Data Sharing

In contrast, sharing data among different platforms is not merely a matter of

sharing devices. Instead, all mechanisms that ensure data integrity must be
implemented by all systems that intend to share data. Figure 96 on page 265
shows an example of data sharing among different platforms.

This will only be possible with industry-wide cooperation to achieve open

standards. Future devices supporting this “real” data sharing will not just be
hardware devices, but rather a combination of hardware and software functions
joined together to provide a storage server.

264 Continuous Availability S/390 Technology Guide

 Figure 96. Heterogeneous Platform Data Sharing

14.5 The Seascape Vision

IBM′s new Storage Enterprise Architecture, named Seascape, provides the base
storage support for heterogeneous data sharing.

Seascape is a modular architecture composed of software and hardware

building blocks which are high-level interchangeable components that can be
snapped together in different configurations to create a total storage solution.

There are basically five categories of building blocks: RISC processors and
memory, network and platform attachments, storage adapters, storage building
blocks (such as disks, tapes, and libraries), and software.

Chapter 14. Sharing Data among Heterogeneous Platforms 265

 This is not a future architecture. This architecture already exists and is used by
IBM storage solutions which are already on the market, such as the IBM
Magstar Virtual Tape Server and the IBM Network Storage Manager.

Furthermore, it was stated in the announcement letter that IBM also plans to
improve current S/390 DASD devices by extending the RAMAC array to support
UNIX and Windows NT data, as well as planning an open systems disk array that
can store UNIX, AS/400, and Windows NT data simultaneously.

266 Continuous Availability S/390 Technology Guide

Chapter 15. Applications Design for Continuous Availability

So far we have limited our scope to the “system side” of continuous availability.
We stressed the need to select components with appropriate C/A attributes, the
need to exploit these attributes when necessary, the need to design the system
for availability, the need for system-wide automation, and the need to measure
and manage availability.

In addition, continuous systems availability implies having applications which are

designed for continuous availability. To use an image to crystallize this idea, we
could compare the whole availability solution to an edifice resting on a tripod,
which is then itself resting on a systems management base.
• The first leg of the edifice is system design.
• The second leg is automation.
• The third leg is application design.
The implication here is that the three legs must all exist and be of equal length
in order to have a balanced solution. The three disciplines must be
implemented at a similar level of sophistication to achieve the desired
objectives.

It is obvious, just as with the system, that applications must also be designed
and implemented to reduce or eliminate the duration and frequency of both
scheduled and unscheduled outages, and to limit their scope.

In this section we briefly review application and development, and provide tips
on exploiting the availability attributes of the S/390 line of products in order to
improve application availability.

15.1 C/A Perspective

Figure 97 on page 268 illustrates the continuous availability attributes that affect
application design.

 Copyright IBM Corp. 1998 267

 Figure 97. Continuous Availability Perspective in Application Development

These aspects should be considered in every stage of the application

development process, because they influence design, coding, and testing.

15.2 Design for Fault Avoidance

The main outages to applications result from “bugs,” so the main improvement
to continuous availability results from designing error-free applications that meet
very stringent quality criteria.

In the following list we discuss four of the most important criteria: correctness,
robustness, extendibility, and reusability.
• Correctness

268 Continuous Availability S/390 Technology Guide

 This is the ability of a software product to perform its functions exactly
according to its functional specifications. Correctness is the prime quality
criterion, which must be addressed by a quality-control development
process.
• Robustness
This is the ability of a software product to continue functioning under
conditions not explicitly covered by its functional specifications. Robustness
is an essence a fuzzy concept, as its role is to make sure that if a situation
arises where an unspecified task is involved, no catastrophic event occurs
that brings the system down, and the current activity terminates cleanly with
the ability to resume rapidly without data loss. An advanced testing process
is required to achieve robustness.
• Extendibility
This is the measure of adaptability of software products to specification
changes. Extendibility is a function of the design simplicity and the degree of
isolation of the modules making up an application. The more autonomous
the modules, the higher the probability that changes to one module will be
localized to that module, or to only a small number of modules. Extendibility
reduces the instability of an application following a change.
• Reusability
This is the ability to reuse correct and robust software modules in new
applications. Reusability reduces the amount of code to be written and
improves the reliability of new applications. When correct and robust
modules can be reused, new applications can be developed more rapidly,
and more time can be devoted to improving the reliability criteria of new
modules.

Reusability and extendibility require architectural techniques to produce simple

designs with distributed modules connected by a small number of well-defined
interfaces. Both object-oriented programming and Message Queueing
programming techniques can provide this.

On the other hand, correctness and robustness can only be achieved with a
quality-controlled development process, coupled with an advanced testing
process.

15.2.1 Object-Oriented Programming

It is much easier to develop an object-oriented application in an environment
designed for this purpose; however, this is not a requirement for developing an
object-based structure.

CICS/ESA, since Version 3.1, has been substantially re-engineered and has an
object-based structure. It is structured in domains , where each domain (also
referred to as a module or an object ) contains a data structure , that is, control
blocks and the function required to change their contents. A domain can send
and receive service requests to/from other domains by using a central
dispatching domain. This architecture reduces the interfaces to the minimum n-1
if the number of domains is n .

Although this programming technique is not new anymore, it is nevertheless

useful to list those characteristics it possesses which contribute to improved
application continuous availability, as follows:

Chapter 15. Applications Design for Continuous Availability 269

 • Modular Object Structure
In a pure object-oriented environment, every component is an object,
including the windows and the window panels of the user interfaces. An
object is a data structure and the processes that are able to use or modify
this data structure to provide service to other objects.
An object is known to other objects by the name of the services it provides,
which can be requested by means of messages. The processing in
object-oriented programming is performed by objects exchanging messages.
• Data Abstraction
With data abstraction, the owner of an object is the only one that knows
about the data structure of that object, while the other objects only care
about the services that the object provides.
• Inheritance
Objects are instances of classes , which are organized in a hierarchy. When
an object is created. it automatically inherits the data structure and the
processes of the class it has been created from, as well as all the
characteristics of the super classes of its class. This concept is similar to
using subroutines, but the data structure and the processes can be
superseded at object level. For example, my bank account is an instance of
the class bank account.
• Polymorphism
This refers to the possibility of sending the same message to different
objects in order to elicit different behavior. For example the message print
can be sent to two different objects, which will then print themselves,
according to their own formats, on different printers.

15.2.2 Quality Software Engineering

A formal quality-controlled development process is required in order to provide a
methodology for designing applications for high availability. With quality and
high reliability being the primary concerns, the process has to focus on defect
detection and elimination, root cause removal, and quality prediction (that is, the
correctness and robustness of software).

The IBM laboratories have developed a strategy to address the zero-defect

software requirement. This strategy, even if it cannot be fully implemented by
smaller development shops, illustrates what must be done to reach the software
quality objectives. The IBM development process, known as the IBM
Programming Process Architecture, is documented in a workbook supported by
development and tracking tools, and a process management database.

The strategic steps in this development process are as follows:

• To agree to a set of parameters defining the software quality as perceived by
our customers, for example, problems by KLOC (thousands of lines of code).
• To define formal development processes that facilitate the achievement of
this objective. This phase process, which covers the overall development
cycle, is split into discrete manageable pieces with checkpoints and
independent quality assurance assessments. The phase process includes:
− Inspection of specifications, designs, and code by formal inspection
meetings. These are chaired by a moderator with a small group of peer

270 Continuous Availability S/390 Technology Guide

 experts who methodically inspect the components presented by the
designer or an independent reader.
− Inspection of test plans and test cases.
− Formal software test entry and exit criteria based on the evaluation of
the quality level achieved.
− A strong management focus on process compliance.

15.2.3 Test Extensively

The application testing process is as important as quality software engineering,
since it must guarantee that the new application can provide service according
to its specifications (correctness), even under conditions that are not specified in
its specifications (robustness).

For this reason, the application testing should be as near as possible to the
production conditions that the application must sustain.

A complete test of the availability design should cover these areas:

• All change and maintenance scenarios
• All failure scenarios
• Concurrent business processes
• Alternate site switch over/back
• Failure during stress operations
• Stress operation during failure

As the last phase of testing, you also need to establish a test environment for
future changes; the following items can help you create such an environment:

Test with Teleprocessing Network Simulator (TPNS): TPNS is a very

sophisticated telecommunications testing package that enables an application
developer to test application programs very accurately and efficiently. The
TPNS/NetView Operator Interface provides for automation of TPNS execution.

TPNS can be used for performance and stress testing of transaction-oriented

application programs, to evaluate and to improve their reliability, and to
approximate performance characteristics under expected operating conditions.
TPNS Version 2 Release 4 can be executed as an ACF/VTAM application
program without any communication hardware requirements.

Testing with CICS EDF: The CICS Execution Diagnostic Facility (EDF), introduced
in CICS/VS V1R3 and enhanced in CICS/MVS and CICS/ESA with a major subset
in CICS OS/2, is an application development tool that can be used to test
command-level programs interactively, without having to modify the code or
preparation procedure.

With the enhancements to the CICS attachment code in DB2 V2R3, the
programmer can now also see information both before and after each SQL
statement. The use of the debugging capability in facilities such as EDF will
potentially result in more thorough testing and more rapid debugging, thus
leading to faster problem resolution and a decrease in future software-related
outages.

Chapter 15. Applications Design for Continuous Availability 271

15.3 Design for Fault Tolerance
When you design for fault tolerance, you must include in your application
specifications all the algorithms and functions to allow the application which is
determining that the problem happened to do the following:
• Isolate the failure component.
• Report about the problem.
• Provide a quick recovery or bypass of the problem.

These can only be achieved if your application design supports such features as
redundancy, failure detection, and recovery or bypass algorithms.

15.3.1 Redundancy/Modularity
Redundancy is the main feature that guarantees your having spare components
that can take over the failed component′s activities.

Where it is not possible to have redundant components, you can use modularity
to guaranteed that only a subset of the total application functions are affected by
the failure.

Modularity by itself does not provide fault tolerance for the single failed
component, but it is useful in limiting the impact of the problem, from the
perception of the user.

Message Queueing: Message Queueing programming techniques can be used

to design applications which have both redundancy and modularity.

For more information about these subjects, refer to 12.5, “How MQSeries
Contributes to Continuous Availability” on page 222.

15.3.2 Failure Detection

Failure detection algorithms should be included in your application in order to
report, with problem and availability management products, that the event
happened due to either code failure or data failure.

Code Failure: In case of code failure, the detection algorithm must provide all
information useful in isolating and identifying both the failure and the component
that is affected.

Data Failure: In case of a data failure, the detection algorithm, before reporting
the problem, may have to identify the reason for the failure. The failure may be
due to:
• Security problems (insufficient authority)
• Physical problems (a hardware error, data corruption, and so on)
• Logical unavailability (data locked by others, and so on)

15.3.3 Recovery/Bypass
This feature allows your application to do a fast recovery or bypass of the
problem; in this case also, we can separate the algorithms into two main parts
as follows:

272 Continuous Availability S/390 Technology Guide

 Code Failure Recovery: In code failure recovery, the algorithm can either retry
the failed operation or it can re-direct the operation to a redundant component.
In both cases, from the user point of view, your application must preserve the
maximum usability for all non-failing paths or functions. It is also important, in
case of a failure, that your recovery actions provide a very fast restart, even by a
fallback to an older level of code.

Data Failure Recovery: Recovery from data failure requires different

considerations, depending on which operation (read or write) the problem was
detected.

When read operations are unsuccessful, your application design should evaluate
the following possibilities:
• Are there any alternate sources of data?
• Are there any default values available?
• Can a subset of the business process be satisfied without the data?
• Would a day-old copy of the data be satisfactory or useful?

When write operations are unsuccessful, your design should consider:

• Can an alternate or temporary database scheme be used?
• Can a temporary flat file or log be used to capture data, for later database
update?
• What part of a user′s request can be satisfied without the write?

15.4 Design for Failure Resistance

Designing failure-proof applications means providing them with such features as
isolation, end-user environment preservation, and fast recovery processes.

15.4.1 Isolation

15.4.1.1 Application Isolation

Application isolation plays a main role in providing continuous availability
application design since it allows your application to be isolated from any
problem happening to others.

On the other hand, most of the applications must intercommunicate in order to

provide their service correctly.

During your design, you must define very strictly the interface protocols and data
structures that these applications use to communicate both with others and
within themselves.

Moreover, your application design should consider being able to isolate the
applications from external dependencies, including operator interventions,
because the service that your application needs may:
• Be unavailable when invoked
• Be unable to process your request
• Fail while processing your request
• Never reply to your request

Chapter 15. Applications Design for Continuous Availability 273

 Your design have to be prepared to handle every occurrence, therefore at a
minimum, you must consider the following:
• Setting time-outs on external resource calls
• Retrying on failures, or briefly waiting and retrying
• Evaluating any alternate services available
• Determining what subset of user requests can be satisfied without this
service

As discussed in 15.3.2, “Failure Detection” on page 272, in this case also your
design must interface with problem and availability management in order to
report any problems that are encountered.

Message Queueing: Message Queueing programming techniques can be used

to isolate an application′s single components, while still providing total
connectivity among them.

For more information about Message Queueing, refer to 12.5, “How MQSeries
Contributes to Continuous Availability” on page 222.

You may also take advantage of the following strategies when designing for
Application Isolation.

Exploit Subsystem Storage Protection (SSSP): Subsystem Storage Protection, a

hardware feature or an RPQ of selected ES/9000 processor models, allows
subsystems executing in a single address space (such as CICS) to use a
different storage protect key from the one the applications are using.

SSSP can be used by CICS/ESA to prevent CICS system code, control blocks, or
data from being overwritten by application code. It enables subsystems to
isolate the failure at the application level.
Note: CICS applications can take instant advantage of this feature without any
modification.

Using SSSP reduces the the time required to diagnose errant application
programs that might potentially overwrite CICS system storage, to solve the
problem, and to recover the application. SSSP can also help application
programmers identify potentially faulty programs before they are put in
production. This is a fault-avoidance attribute at the application level.

Exploit Subsystem Availability Characteristics: For example, under CICS, an

application should exploit Multi-Region Operation (MRO) to isolate its major
functions into independent application regions, terminal-owning regions, and
data-owning regions, so that the unavailability of one function does not impact
the entire application. With IMS TM, the dependent region architecture itself
provides a high degree of isolation, without the need for application design to
enforce it.

15.4.1.2 Data Isolation

When designing your application data structure, you should again keep in mind
that isolation is the only reliable way to limit data unavailability to just a subset
of your application data. This can be achieved in two ways:

274 Continuous Availability S/390 Technology Guide

 Use Logical DB Descriptions The main advantage of using logical database
descriptions is that it allows your applications to be independent from any data
structure modification. This can be achieved by using either Views in DB2 or
Logical DBDs in IMS.

Use Database or Table Space Partitioning: The main advantage of using

partitioning is that utility functions can be run at the partition level, to reduce the
duration of the outage. DB2 and IMS Fast Path users should use partitioning
when designing data bases, to limit the duration of utilities, and (in the case of
Fast Path), to reduce the impact when data has to be taken down. IMS Full
Function users should design their own partitioning, where appropriate.

15.4.2 User Environment Preservation

Applications should be designed to continue to provide end users with some
percentage of service, even if not all databases are available.

Provide Stand-in Processing Alternatives: When applications are designed and

data is placed so that host access is not required to provide service, then users
can keep running a subset of the functions of an application even when the host
or the connection to the host is out of service. Both MQSeries and CICS provide
the application environments on different platforms, so that your processes can
be designed to exploit stand-in processing.

15.4.3 Fast Recovery

In the event of an outage, the application must provide a fast and friendly restart.
Fast restart can be obtained by parallel processing in a multi-processor
environment, and it depends on the modularity of the application. Friendly
restart can be provided by the automatic saving and restoration of the end-user
environment.

Exploit Subsystem Recovery Features: IBM subsystems provide many of these

features, such as the support for checkpoints. Other features are more specific,
such as the use of transaction abend exits in CICS. The use of these features
reduces the time needed to resume service.

15.5 Design for Availability Management

As previously mentioned, the application design process should also include
availability management, which is perceived as tracking and reporting.

15.5.1 Tracking and Reporting

Design algorithms that provide your application with a problem reporting feature,
as well as availability reporting features, for measuring the availability provided
both by your application (as perceived by the end-user) and by the services
requested by your application.

These measures can then be used for:

• Maintaining counters and metrics, to establish expected volumes and detect
growing problems
• Maintaining response time measurements of outside services and detecting
service failures

Chapter 15. Applications Design for Continuous Availability 275

 • Coordinating business measurements with system management processes,
to calculate the impact of outages
• Providing audit trails to improve recovery capability

15.6 Design for Non-Disruptive Changes

As with having a code failure (also known as a “bug”), applying application
changes also usually causes your application to be unavailable.

During the design phase, you should try to provide your application with features
that allow you to dynamically modify its environment, parameters, and data.

End User: The considerations include both logical and physical views of the
end-user. At a minimum, your design should consider:
• Adding/deleting authorized users
• Changing a user′s authority
• Changing a user′s location
• Terminal types and software levels
• Terminal location and hours of operation
• Terminal connection protocols and speeds

Application: These considerations include both environmental and application

structures. At a minimum, your design should consider:
• Adding/deleting/changing transactions
• Adding/changing global variables and values

Time, Date, and Parameters: These considerations include all paramenters that
normally affect applications, such as:
• Date and time
• Year 2000
• Timezone and daylight saving anomalies
• Clock corrections
• Application-related parameters without restart

Application Data: Finally, your design should also provide the ability to
dynamically change all or part of the application data. This can be achieved by
both IMS and DB2, using the Database Partitioning feature. For more
information, refer to 10.5.1, “Fast Path DEDB Online Change” on page 194, and
11.4.5, “Partition Independence” on page 208.

15.7 Design for Non-Disruptive Maintenance

Designing for non-disruptive maintenance means that you can modify the coding
of your application without interrupting the service it provides.

Earlier in this chapter we discussed two features that contribute to non-disruptive

maintenance (redundancy/modularity and isolation), but to actually achieve this
goal, your application design should also provide co-existence.

276 Continuous Availability S/390 Technology Guide

15.7.1 Co-existence
This feature allows multiple versions of the same program/function to run
concurrently. It can be provided by both DB2 and by Message Queueing.

Use DB2 Package Binding and Versioning: The package concept, introduced
since DB2 2.3, is a result of binding a Database Request Module (DBRM).

A DBRM is one of two outputs from the DB2 pre-compiler, and it contains the
SQL statements for a given program module. The new application plan can now
consist solely of a list of the packages (bound DBRMs) that are used by the
application. This three-level hierarchy, DBRM-package-plan, makes the
re-binding process which results from SQL changes to the program much less
disruptive.

With the package bind, only the changed DBRMs need be rebound. In addition,
with the support of multiple versions of packages (DB2 2.3), the binding of
changed DBRMs can be done ahead of time, without disrupting the production
application.

Multi-versioning support also eliminates the need to re-bind in order to fall back
to a previous version if problems arise with modules newly introduced into
production. The package bind, together with the multi-versioning support,
provides very significant improvement in the continuous operation capabilities of
DB2 applications.

15.7.2 Online Redefinition of Applications

Also in regard to non-disruptive maintenence, it is important that the programs
(or functions) that compose the application are able to be updated without
interrupting the service provided by the application. You can accomplish this in
the following ways:

Use XRF to Dynamically Update Application Programs: A new version of your

application can be put into production using the Extended Recovery Facility
(XRF) function of both CICS and IMS. This function provides a non-disruptive
takeover of the workload from different subsystem regions.

Possible implementations of this technique could be:

• Updating the program library on the alternate subsystem
• Transferring the workload to the alternate subsystem
• Using the updated application programs without service interruption
• Updating the program library on the old active subsystem

However, to be transparent to the user, you need also to apply VTAM Persistent
Session Support, which provides you with dynamic reconnection of the end user
to the application. With this feature, the end user perceives the application
switching as a “wait” on the terminal.

For more information on XRF, VTAM Generic Session, and their usage with CICS
and IMS, refer to 9.4, “CICS Parallel Sysplex Exploitation” on page 171 and 10.4,
“IMS Parallel Sysplex Exploitation” on page 190

Chapter 15. Applications Design for Continuous Availability 277

 Message Queueing: Using Message Queueing, you can start multiple instances
of your applications that pull message from the same queue.

For more information, refer to 12.5, “How MQSeries Contributes to Continuous

Availability” on page 222.

15.8 Design for Continuous Applications

Other ways to enhance continuous operation through proper application design,
are avoiding iteference of batch and online processing of the same data, as well
as using database technologies that allow for concurrent reorganization and
backup.

15.8.1 No Batch Interference

Your application should be designed to try to avoid batch interference, because
the application cannot provide online services if it has to be brought down every
night to allow the batch to process. S/390 DB/TM subsystems have
characteristics that can be exploited to reduce or eliminate the batch window.

15.8.1.1 Batch Window Elimination

Message Queueing: By means of Message Queueing, you can design
applications that have concurrent processing of batch and online phases,
because they use a queueing mechanism to exchange data.

For more information, refer to 12.5, “How MQSeries Contributes to Continuous

Availability” on page 222.

VSAM Record Level Sharing: Using VSAM record level sharing, you can share
VSAM files between CICS online transactions and batch processing. For more
information on VSAM Record Level Sharing, refer to 9.4.1, “VSAM Record Level
Sharing” on page 171.

IMS and BMPs: Since IMS/ESA 3.1, DBCTL makes batch message processing
(BMP) available for concurrent updating of data by both online and batch
programs. IMS BMP is used to perform batch-type processing online.

IMS Data Sharing: IMS Data Sharing offers an alternative approach to the
concurrent sharing of IMS data between batch and online. In addition, it
provides the ability to share IMS data between two separate IMS TM and/or
CICS subsystems, which can reside in separate MVS images.

15.8.1.2 Batch Window Reduction

Although you may not be able to completely eliminate the interference of batch
processing, you can use the following methods to reduce the batch window.

Concurrent Backup: This feature provides a background point-in-time backup

copy of designated data sets or databases, while the online application
continues normal operation, accessing and updating the data being copied as
needed. A point-in-time copy can be used to “freeze” an IMS database or CICS
VSAM dataset in order to permit the running of batch reports without impacting
the online system.

For more information, refer to 6.2.1.6, “Concurrent Backup” on page 102.

278 Continuous Availability S/390 Technology Guide

 Sequential Data Striping: Sequential data striping provides significant potential
throughput improvement without requiring new or additional hardware beyond
that required for ESCON capability. Most BSAM and QSAM sequential
processing applications can take advantage of sequential data striping without
change.

Since most batch jobs use sequential data sets and are I/O-bound, sequential
data striping should help reduce the batch window and enhance the continuous
operation and continuous availability characteristics of the service provided by
an installation.

For more information, refer to 6.2.1.2, “RAID Level 0” on page 99.

Hiperbatch: Use of the Hiperbatch data-in-memory facility, available since

MVS/SP 3.1.3, has resulted in a batch elapsed-time reduction of up to 60%,
depending on contention and re-use characteristics of the individual workloads.
This can provide a significant reduction of the batch window elapsed run-time,
and a corresponding increase in availability of OLTP applications.

Smartbatch for OS/390: IBM′s Smartbatch for OS/390 helps reduce the batch
window by using techniques to speed up the flow of batch work through a
processor or a group of processors. These techniques are parallelism, I/O
optimization, and workload distribution.

For more information, refer to 13.5.1, “Smartbatch for OS/390” on page 243.

15.8.2 Online Database Reorganization and Copies

A number of technologies exist that allow the execution of database
reorganizations, as well as making database backup copies, while the online
system is active.

15.8.2.1 Reduce or Eliminate Disruptive Data Reorganizations

A key aspect of continuous applications is the elimination, masking, or reduction
of data outages caused by database reorganizations. These outages can be
eliminated by using DBMS online reorganization features such as IMS Fast Path
DEDB online reorganization or DB2 V5.1 Online Reorg.

For more information, refer to 11.4.4, “Online Reorg” on page 208

If these reorganizations cannot be eliminated, a point-in-time copy of data sets

or databases can be reorganized while the online application is running. When
the reorganization is complete, the log data sets can be run against the
reorganized copy to get a logically consistent copy. It is most probable that the
forward recovery process will be faster than the recovery process, and the result
will be a shorter planned outage.

For more information, refer to 6.2.1.6, “Concurrent Backup” on page 102.

15.8.2.2 Use Non-Disruptive Copy Facilities

IMS and DB2 both provide facilities for non-disruptively making copies of data
while the data is being updated by user applications. DFSMS/MVS Concurrent
Copy drastically reduces data unavailability, while providing a point-in-time
backup copy.

Chapter 15. Applications Design for Continuous Availability 279

280 Continuous Availability S/390 Technology Guide
Chapter 16. Continuous Availability and Disaster Recovery

Continuous availability is the property of a system designed such that users

experience neither scheduled nor unscheduled outages, by employing hardware
components, software, and operational procedures that mask outages from the
user. It usually requires that the recovery from an outage has to be performed
so quickly that the user does not perceive it as an outage. It also frequently
requires the use of redundant components, so that an alternate component can
be used in case of a permanent component failure, or while a component is in
maintenance.

Many major components of today′s computer systems are fault-tolerant to some

degree, which means that they will tolerate some faults through the use of:
• Redundant sub-components
• Error checking and correction for data
• Retry capabilities for basic operations
• Alternate paths for I/O requests
• Duplexed data facilities on DASD

Fault-tolerant systems are often designed so that they will not stop working if
they have an internal component failure or if one of their devices suffers a
failure. This not only requires a suitable degree of redundancy, but it requires
constantly checkpointing work as the state of the work being done changes so
that in the event of a failure, that work can be rescued by a back-up facility.

16.1 What is Disaster Recovery

Disaster recovery is the process of reacting to a disaster by being able to
provide computing services from another location. In most cases, the
countermeasures you employ to be able to recover from a disaster are entirely
different from the solution you use to achieve continuous availability

The publication Fire in the Computer Room, What Now? covers all the design
aspects of disaster recovery, while Disaster Recovery Library - S/390 Technology
Guide describes all the S/390 technologies available and how you can apply
them to implement the recovery solution.

In a disaster situation, users normally are aware that an outage has happened to
the central computer facility, and the duration of the outage is mainly dependent
on the recovery solution.

Usually this duration is measured in two different components:

1. Data Loss
This represents the loss of data you have, that is, how much work you must
re-execute once your system is recovered,
2. Service Loss
This represents the loss of computing you experienced from the moment of
disaster up to the moment when your system has been recovered.

At SHARE 78, Anaheim, 1992, in session M028, the Automated Remote Site
Recovery Task Force presented seven tiers of recoverability, which were ranked

 Copyright IBM Corp. 1998 281

 based on the recovery method and recovery time. The following sections
describe these tiers.

16.1.1.1 Tier 0 - No Offsite Data

This tier provides no preparation in saving information, determining
requirements, establishing a backup hardware platform, or developing a
contingency plan.
Typical Recovery Time: The length of time till recovery is unpredictable.
Indeed, you cannot be sure of being able to recover at all.

Figure 98. Tier 0 Recovery Solution

16.1.1.2 Tier 1 - Pickup Truck Access Method (PTAM)

To be at Tier 1, an installation would need to develop a contingency plan, back
up required information and store it in contingency storage (off-site location),
determine recovery requirements, and optionally establish a backup platform
supporting a conditioned facility without processing hardware.
Typical Recovery Time: The length of time till recovery is usually more than
a week.

282 Continuous Availability S/390 Technology Guide

Figure 99. Tier 1 Recovery Solution

16.1.1.3 Tier 2 - PTAM + Hot Site

Tier 2 encompasses all requirements of Tier 1 and also requires a backup
platform to have sufficient hardware and network to support the installation′ s
critical processing requirements. Processing is considered critical if it must be
supported on hardware that exists at the time of the disaster.
Typical Recovery Time: The length of time till recovery is usually more than
one day.

Figure 100. Tier 2 Recovery Solution

16.1.1.4 Tier 3 - Electronic Vaulting

Tier 3 encompasses all the requirements of Tier 2 and, in addition, supports
electronic vaulting of some subset of the information. The receiving hardware
must be physically separated from the primary platform and the data stored for
recovery after the disaster.
Typical Recovery Time: The length of time till recovery is usually about one
day.

Figure 101. Tier 3 Recovery Solution

Chapter 16. Continuous Availability and Disaster Recovery 283

 16.1.1.5 Tier 4 - Active Secondary Site
Tier 4 introduces the requirements of active management of the recovery data by
a CPU at the recovery site, and bi-directional recovery. The receiving hardware
must be physically separated from the primary platform.
Typical Recovery Time: The length of time till recovery is usually up to one
day.

Figure 102. Tier 4 Recovery Solution

16.1.1.6 Tier 5 - Two Site Two Phase Commit

Tier 5 encompasses all the requirements of Tier 4 and, in addition, will maintain
selected data in image status (updates will be applied to both the local and
remote copies of the databases within a single commit scope). Tier 5 requires
both the primary and secondary platforms′ data to be updated before the update
request is considered satisfied. Tier 5 requires partially- or fully-dedicated
hardware on the secondary platform, with the capability to automatically transfer
the workload to the secondary platform.
Typical Recovery Time: The length of time till recovery is usually less than 12
hours.

Figure 103. Tier 5 Recovery Solution

16.1.1.7 Tier 6 - Zero Data Loss

Tier 6 encompasses zero loss of data and immediate and automatic transfer to
the secondary platform. Data is considered lost if ENTER has been accepted (at
the terminal) but the request has not been satisfied.
Typical Recovery Time: The length of time till recovery is usually a few
minutes.

284 Continuous Availability S/390 Technology Guide

 Figure 104. Tier 6 Recovery Solution

16.1.2 Data Loss and Service Loss

The Typical Recovery Time associated with each tier is just a rough indication of
the time that an installation usually needs to restore its computing services.
However, in a disaster situation, there are many other points to consider.

For example, some installations can tolerate resuming their services after longer
periods of time, but with maximum data currency. Other installations must
resume their services as soon as possible, regardless the currency of their data.
Still others need both a short recovery time and maximum data currency.

Figure 105 on page 286 illustrates the data loss and service loss aspects of each
recovery solution.

Chapter 16. Continuous Availability and Disaster Recovery 285

 Figure 105. Data Loss and Service Loss

16.2 Relationship Between Continuous Availability and Disaster Recovery

All the components that make up continuous availability in a computer system
are usually situated in the same building. Therefore, the building itself can
represent a single point of failure. In this case, and despite all the continuous
availability you have designed into the system, a disaster could cause you to
lose all your computing services.

While you must of course be prepared to react to a disaster, the solution you
would apply in this case may be more of a recovery solution than a continuous
availability solution. A recovery solution can be defined by making a trade-off
among implementation costs, maintenance costs, and financial impacts resulting
from performing a business impact analysis of your business.

Furthermore, as you can see from the tiers definitions, only a Disaster Recovery
Tier 6 solution can be compared to a continuous availability solution, although
currently there is no technology available that fulfill that definition.

However, with the current S/390 technologies, it is possible to achieve a Tier 6

workload transfer if you have enough time to do it in a controlled fashion before
all computing capabilities are lost in a disaster at your production site.

A combination of S/390 technologies such as Parallel Sysplex, IBM 3390-6

Remote Copy and P/DAS, together with a good set of system automation

286 Continuous Availability S/390 Technology Guide

 products that include site and environmental control, can allow you to achieve a
Tier 6 workload transfer.

16.3 Geographically Dispersed Parallel Sysplex

A Parallel Sysplex environment can be set up so that the computer room is not a
single point of failure, even in a disaster situation. To achieve this, you will need
another building to contain all devices necessary to run at least the most critical
business processes.

These two (or more) buildings should also be located in such way that they will
not be impacted by the same disaster; for example if they are located 100 km
apart, but are situated next to the same river, it is likely they could both be
damaged by the same flood.

These two (or more) buildings must also be connected with high bandwidth
connections to provide device sharing among all processors. The best way to do
this is to use the flexibility provided by ESCON channels and ESCON Directors
together. Moreover, you can limit the number of fiber connections between the
sites by using the IBM 9729, which lets you share the same fiber trunk among 10
ESCON channels.

For more information about these connections, refer to Chapter 3, “S/390 ESCON
Architecture” on page 35.

The last consideration concerns the distance between the sites. The further
apart the sites are, the longer it takes to access data since the signals travel at
a fixed speed.

The most limiting factors in extending a Parallel Sysplex configuration are:

• All processors in the complex must have their clocks synchronized by an
external device (the Sysplex Timer).
• All processors in the complex must use the coupling facilities to guarantee
data integrity on datasharing operations.
Both of these facilities can now be located up to 23 km from the processors.
Figure 106 on page 288 shows a typical configuration.

With a geographically dispersed Parallel Sysplex, you have a processing

platform that can transfer the workload among all sites in the configuration.
However, since your application data still resides in each site, you will still lose
it if a disaster occurs.

To avoid this, you need to combine a geographical dispersed Parallel Sysplex

with a total mirroring of your critical data by using a remote copy solution such
as PPRC and P/DAS that allow you to switch instantaneously between the
primary and the secondary image of your data without stopping your application
processing.

For more information on PPRC and P/DAS, refer to 6.3.2.4, “Remote Copy” on
page 109.

Under these conditions you can afford a controlled workload and processing
transfer between the two sites. However, in a disaster situation, the time that
your operators need to perform this transfer might not be available.

Chapter 16. Continuous Availability and Disaster Recovery 287

 Therefore, you must include in your solution a complex pre-emptive automation
package that can initiate the transfer as soon as it detects something unusual
happening in one of the two sites. This automation package must monitor not
only I/S devices, but also many environmental controls in order to at least detect
smoke, fire, water, and room temperature.

Figure 106. Geographically Dispersed Parallel Sysplex

The cost associated with such a solution is very high and few installations can
justify it by having very stringent recovery needs, even in case of disaster.
However, the S/390 platform provides you with this solution by using the same
technologies you may already be using every day in your business.

16.4 The Total Recovery Solution

As you may be aware, the higher the recoverability tier, the more expensive the
cost of implementing the recovery solution.

The only way to keep the implementation cost at an acceptable level is to

identify the real recoverability requirements of each business process, and then
adopt a suitable recovery solution that meets these requirements.

288 Continuous Availability S/390 Technology Guide

This can only be achieved by performing a business impact analysis. Such an
analysis maps the financial impact of a disaster to each business process, and
helps you determine the maximum tolerable cost of the solution for recovering
each business process. Figure 107 on page 289 illustrates the relationship
between the financial impact of a disaster and the cost of recovery.

Figure 107. Financial Impact of a Disaster and Cost of Recovery

Next you should map each business process to the I/S applications that support
it, and identify how important the application is to that business process.

Figure 108. Total Recovery Solution

Chapter 16. Continuous Availability and Disaster Recovery 289

 At the end of the process, the total recovery solution might be a mixture of
different solutions that provide you with the business recoverability you need at
the minimum affordable cost.

290 Continuous Availability S/390 Technology Guide

Appendix A. Special Notices

This publication is intended to help IBM customers and system engineers

selecting products and features that pertain to continuous availability. The
information in this publication is not intended as the specification of any
programming interfaces that are provided by these products.

References in this publication to IBM products, programs or services do not

imply that IBM intends to make these available in all countries in which IBM
operates. Any reference to an IBM product, program, or service is not intended
to state or imply that only IBM′s product, program, or service may be used. Any
functionally equivalent program that does not infringe any of IBM′s intellectual
property rights may be used instead of the IBM product, program or service.

Information in this book was developed in conjunction with use of the equipment
specified, and is limited in application to those specific hardware and software
products and levels.

IBM may have patents or pending patent applications covering subject matter in
this document. The furnishing of this document does not give you any license to
these patents. You can send license inquiries, in writing, to the IBM Director of
Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.

Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact IBM Corporation, Dept.
600A, Mail Drop 1329, Somers, NY 10589 USA.

Such information may be available, subject to appropriate terms and conditions,

including in some cases, payment of a fee.

The information contained in this document has not been submitted to any
formal IBM test and is distributed AS IS. The information about non-IBM
(″vendor″) products in this manual has been supplied by the vendor and IBM
assumes no responsibility for its accuracy or completeness. The use of this
information or the implementation of any of these techniques is a customer
responsibility and depends on the customer′s ability to evaluate and integrate
them into the customer′s operational environment. While each item may have
been reviewed by IBM for accuracy in a specific situation, there is no guarantee
that the same or similar results will be obtained elsewhere. Customers
attempting to adapt these techniques to their own environments do so at their
own risk.

Any pointers in this publication to external Web sites are provided for
convenience only and do not in any manner serve as an endorsement of these
Web sites.

The following document contains examples of data and reports used in daily
business operations. To illustrate them as completely as possible, the examples
contain the names of individuals, companies, brands, and products. All of these
names are fictitious and any similarity to the names and addresses used by an
actual business enterprise is entirely coincidental.

 Copyright IBM Corp. 1998 291

 The following terms are trademarks of the International Business Machines
Corporation in the United States and/or other countries:

AnyNet ACF/VTAM
ADSTAR AIX
APPN AS/400
BatchPipes CICS
CICS OS/2 CICS/ESA
CICS/MVS CICSPlex
DB2 DFSMS
DFSMS/MVS DFSMSdss
DFSMShsm ES/9000
ESA/390 ESCON
ESCON XDF First Failure Support Technology
FFST GDDM
Hiperbatch IBM
IMS IMS/ESA
Magstar Multiprise
MQ MQSeries
MVS/ESA MVS/SP
MVS/XA Net.Data
Network Station NetView
Open Blueprint OpenEdition
OS/2 OS/390
Parallel Sysplex Predictive Failure Analysis
PR/SM PROFS
RACF RAMAC
RMF RS/6000
S/370 S/390
S/390 Parallel Enterprise Server Seascape
SecureWay Sysplex Timer
System/390 SystemView
SP Ultrastar
VM/ESA VSE/ESA
VTAM 400

The following terms are trademarks of other companies:

C-bus is a trademark of Corollary, Inc.

Java and HotJava are trademarks of Sun Microsystems, Incorporated.

Microsoft, Windows, Windows NT, and the Windows 95 logo are trademarks
or registered trademarks of Microsoft Corporation.

PC Direct is a trademark of Ziff Communications Company and is used

by IBM Corporation under license.

Pentium, MMX, ProShare, LANDesk, and ActionMedia are trademarks or

registered trademarks of Intel Corporation in the U.S. and other
countries.

UNIX is a registered trademark in the United States and other

countries licensed exclusively through X/Open Company Limited.

Other company, product, and service names may be trademarks or

service marks of others.

292 Continuous Availability S/390 Technology Guide

Appendix B. Related Publications

The publications listed in this section are considered particularly suitable for a
more detailed discussion of the topics covered in this redbook.

B.1 International Technical Support Organization Publications

For information on ordering these ITSO publications see “How to Get ITSO
Redbooks” on page 295.
• Fire in the Computer Room, What Now? , SG24-4211
• Disaster Recovery Library - S/390 Technology Guide , SG24-4210
• S/390 MVS Parallel Sysplex Continuous Availability SE Guide , SG24-4503
• Continuous Availability Systems Design Guide , SG24-2085
• Enterprise Systems Connection (ESCON) Implementation Guide , SG24-4662
• OS/390 MVS Parallel Sysplex Configuration Cookbook , SG24-4706
• Comparison of S/390 Configurations - Parallel and Traditional , SG24-4514
• DB2 Data Sharing Recovery , SG24-2218
• OS/390 MVS Parallel Sysplex Configuration Overview , SG24-2075
• OS/390 MVS Parallel Sysplex Configuration Cookbook , SG24-2076
• OS/390 MVS Parallel Sysplex Configuration Connectivity , SG24-2077
• Open Systems Adapter 2 Implementation Guide , SG24-4770

B.2 Redbooks on CD-ROMs

Redbooks are also available on CD-ROMs. Order a subscription and receive
updates 2-4 times a year at significant savings.

CD-ROM Title Subscription Collection Kit

Number Number
System/390 Redbooks Collection SBOF-7201 SK2T-2177
Networking and Systems Management Redbooks Collection SBOF-7370 SK2T-6022
Transaction Processing and Data Management Redbook SBOF-7240 SK2T-8038
Lotus Redbooks Collection SBOF-6899 SK2T-8039
Tivoli Redbooks Collection SBOF-6898 SK2T-8044
AS/400 Redbooks Collection SBOF-7270 SK2T-2849
RS/6000 Redbooks Collection (HTML, BkMgr) SBOF-7230 SK2T-8040
RS/6000 Redbooks Collection (PostScript) SBOF-7205 SK2T-8041
RS/6000 Redbooks Collection (PDF Format) SBOF-8700 SK2T-8043
Application Development Redbooks Collection SBOF-7290 SK2T-8037

B.3 Other Publications

These publications are also relevant as further information sources:
• Sysplex Overview , GC28-1208
• System/390 Continuous Systems Availability , GG66-3147
• DB2 for OS/390 V5.1 Release Guide, SC26-8965
• DB2 for OS/390 V5.1 Administration Guide , SC26-8957

 Copyright IBM Corp. 1998 293

 • IMS/ESA V5 Release Planning , GC26-8031
• CICS Transaction Server for OS/390 Release Guide , GC33-1570
• CICS/ESA Release Guide , GC33-1161
• CICS Transaction Server for OS/390 CICS Intercommunication Guide ,
SC33-1695
• CICSPlex SM for MVS/ESA Concepts and Planning , GC33-0786
• System Overview, S/390 9672 Generation 4 , GA22-7154
• Sysplex Timer Planning , GA23-0365
• Planning for the 9037 Model 002 Sysplex Timer , SA22-7233
• 9037 Sysplex Timer and S/390 Time Management , GG66-3264
• The Importance of Systems Management for a Parallel Sysplex , Systems
Journal Vol. 36, No.2, G321-5645
• IBM Smartbatch for OS/390 Overview , GC28-1627
• TSCF Planning and Installation , GC28-1065
• S/390 PR/SM Planning Guide , GA22-7236
• LPAR Dynamic Storage Reconfiguration , GG66-3262
• Planning for the S/390 Open Systems Adapter Feature , GC23-3870
• An Introduction to Message and Queueing , GC33-0805
• MQSeries Planning Guide , GC33-1349

294 Continuous Availability S/390 Technology Guide

How to Get ITSO Redbooks
This section explains how both customers and IBM employees can find out about ITSO redbooks, redpieces, and
CD-ROMs. A form for ordering books and CD-ROMs by fax or e-mail is also provided.
• Redbooks Web Site http://www.redbooks.ibm.com/
Search for, view, download or order hardcopy/CD-ROMs redbooks from the redbooks Web site. Also read
redpieces and download additional materials (code samples or diskette/CD-ROM images) from this redbooks
site.
Redpieces are redbooks in progress; not all redbooks become redpieces and sometimes just a few chapters
will be published this way. The intent is to get the information out much quicker than the formal publishing
process allows.
• E-mail Orders
Send orders via e-mail including information from the redbook order form to:

IBMMAIL Internet
In United States: usib6fpl at ibmmail [email protected]
I n Canada: caibmbkz at ibmmail [email protected]
Outside North America: dkibmbsh at ibmmail [email protected]

• Telephone Orders

United States (toll free) 1-800-879-2755

Canada (toll free) 1-800-IBM-4YOU

Outside North America (long distance charges apply)

(+45) 4810-1320 - Danish (+45) 4810-1220 - French (+45) 4810-1270 - Norwegian
(+45) 4810-1420 - Dutch (+45) 4810-1020 - German (+45) 4810-1120 - Spanish
(+45) 4810-1540 - English (+45) 4810-1620 - Italian (+45) 4810-1170 - Swedish
(+45) 4810-1670 - Finnish

This information was current at the time of publication, but is continually subject to change. The latest information
for customers may be found at http://www.redbooks.ibm.com/ and for IBM employees at
http://w3.itso.ibm.com/.

IBM Intranet for Employees

IBM employees may register for information on workshops, residencies, and redbooks by accessing the IBM
Intranet Web site at http://w3.itso.ibm.com/ and clicking the ITSO Mailing List button. Look in the Materials
repository for workshops, presentations, papers, and Web pages developed and written by the ITSO technical
professionals; click the Additional Materials button. Employees may also view redbook, residency and
workshop announcements at http://inews.ibm.com/.

 Copyright IBM Corp. 1998 295

IBM Redbook Fax Order Form
Fax your redbook orders to:

United States (toll free) 1-800-445-9269

Canada 1-403-267-4455
Outside North America (+45) 48 14 2207 (long distance charge)

Please send me the following:

Title Order Number Quantity

First name Last name

Company

Address

City Postal code Country

Telephone number Telefax number VAT number

• Invoice to customer number

• Credit card number

Credit card expiration date Card issued to Signature

We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card not
available in all countries. Signature mandatory for credit card payment.

296 Continuous Availability S/390 Technology Guide

Index
9672 (continued)
Numerics Independent Dual Power Feed 14
3088 Internal Battery Feature (IBF) 14
see Channel to Channel Internal Coupling Facility use 141
3990 99, 107 Internal Coupling Migration Facility use 141
9032 Local Uninterruptible Power Supply 14
see ESCON Director LPAR Dynamic Storage Reconfiguration (DSR
9033 2) 11
see ESCON Director LPAR Time Management Reporting 12
9034 MBA channel affinity 43
see ESCON Converter N + 1 Power Supply Technology 14
9035 OSA card 42
see ESCON Converter OSA-2 22
9036 P-based models 30
see ESCON Channel Extender Partial I/O Restart 13, 43
9037 Partial Memory Restart 13
see Sysplex Timer Processor Availability Facility (PAF) 11
9390 99, 107 Processor Design 9
9392 101, 117 R1-based models 30
9394 99, 120 R2-based models 30
9395 101, 117 R3-based models 30
9672 Scrubbing 13
Automatic Reconfiguration Facility (ARF) 13 Serviceability 15
CHA card 42 STI card 42
Channel card 42 Subspace Group Facility 12
channel path configuration 44 Subsystem Storage Protection 12
CMOS availability features 9 Summary of S/390 Processor Continuous
CMOS packaging Advantage 6 Availability Attributes 5
Concurrent CP/SAP Sparing 10 Up to 15 LPARs 12
Concurrent Hardware Maintenance 14 9674 19
Concurrent Licensed Internal Code (LIC) Patch 15 9729
configuring channels for availability 41 see Wavelength Division Multiplexer
Console Integration 15
Coupling link card 42
Dynamic Coupling Facility Dispatching 11 A
Dynamic I/O Reconfiguration 13 Access Method
Dynamic Memory Sparing 12 OpenEdition MVS Files 62
Dynamic SAP Sparing/Reassignment 11 ADSM
Dynamic Storage Reconfiguration 11 see ADSTAR Distributed Storage Manager
E-based models 30 ADSTAR Distributed Storage Manager 85, 241
Enhanced Recovery for Cache and Directory application design
Errors 9 application isolation 273
Error Correction Code (ECC) 11 availability management 275
ESCON Multiple Image Facility (EMIF) 12 continuous applications 278
fault-tolerant design 9 data isolation 274
FIB card 42 failure detection 272
functional differences 30 fault avoidance 268
G3-based models 30 fault tolerance 271
G4 Processor 6 non-disruptive changes 276
G4-based models 30 non-disruptive maintenance 276
hardware features 30 object-oriented programming 269
Hot-plugging 13 processing alternatives 275
I/O cage layouts 41 recovery/bypass 272
I/O domain 42 redundancy 272
I/O interface reset 13 software quality 270

 Copyright IBM Corp. 1998 297

Application Integration Feature (AIF) 88 CICS (continued)
ARM distribution (continued)
see Automatic Restart Manager Support for DCE remote procedure calls (DCE
ARM couple dataset 146 RPC) 164
Asynchronous Processing (AP) 164, 167 Transaction routing (TR) 164, 166
Automated Operations and Control 236 Domains
Automatic Reconfiguration Facility (ARF) 13 Directory manager domain 175
Automatic Restart Manager (ARM) 233 Program manager domain 175
CICS support 180 Security manager domain 175
DB2 support 210 Transaction manager domain 175
IMS support 197 User domain 175
Availability measurement 249 Extended Recovery Facility (XRF) 174
external resource manager
syncpoint with MQseries 219
B gateway for Java 87
Backward Recovery internet access 87, 182
IMS 187 isolation
Batch Accelerator 71 Inter-System Communication (ISC) 164, 165
batch management 242 Multi-Region Operation (MRO) 164
batch window Parallel Sysplex exploitation
eliminating 278 Temporary Storage Data Sharing 172
reducing 278 VSAM Record Level Sharing 171
BatchPipePlex 70 VTAM Generic Resource 173
Berkeley 98 VTAM Persistent Session 174
bibliography 293 recovery with MQSeries 219
Bristol Technology Resource Definition Online (RDO) 178
Windows NT application support for OS/390 94 resource manager interface (RMI)
recovery with MQSeries 182
syncpoint with MQSeries 182, 219
C single system image 164
Capacity Planning 250, 251, 252
Transaction isolation 179
CFRM couple dataset 146
unit of work (UOW) 172, 183
Channel-to-Channel
VSAM recovery attributes 172, 183
3088 38
CICS/TS
CTC path configuration 45
DB2 Attachment Facility 181
ESCON CTC 38
DBCTL as interface to IMS 181
ESCON CTCs with EMIF 40
Domains
CICS
Log manager domain 176
ARM support 180
Recovery manager domain 176
Autoinstall 179
Temporary storage domain 176
backup-while-open (BWO) 172, 183
resource manager interface (RMI) 181
CICS regions
CICSplex
Application Owning Region (AOR) 164
Dynamic Transaction Balancing 169
Data Owning Region (DOR) 164
Dynamic Transaction Routing 168
File Owning Region (FOR) 164
System Manager 169
Queue Owning Region (QOR) 164
CICSPlex System Manager 169
Terminal Owning Region (TOR) 164
CICSPlex Systems Manager 247
DB2 Attachment Facility 181
CLIO/S
DBCTL as interface to IMS 180
see IBM Client Input/Output Sockets
Design for Availability
CNT Filespeed
functional split 167
for Data Copy Sharing 258
logical split 167
High Speed Piping 261
distribution
Co-existence
Asynchronous processing (AP) 164, 167
CICS 68
DCE Remote Procedure Calls (DCE RPC) 167
DB2 68
Distributed program link (DPL) 164, 167
IMS 68
Distributed transaction processing (DTP) 164,
Operating System 68, 69
167
External CICS interface (EXCI) 164, 167
Function shipping (FS) 164, 166

298 Continuous Availability S/390 Technology Guide

code failure 272 Database Partitioning
Concurrent Backup 102 DB2 Table Space 208, 275
Concurrent Copy 73, 102, 109 IMS Data Entry Database (DEDB) 186, 275
Concurrent CP/SAP Sparing 10 Daylight Saving considerations 145
Configuration Tips DB2
channel path configuration 44 Continuous Availability Features 206
CTC path configuration 45 Automatic Restart Manager (ARM) 210
ESCD path availability configuration 51 Cancel Thread Command 210
Continuous Availability Daylight Saving Time 210
design 1 Online Reorg 208
implementation 1 Partition Independence 208
iterative design 4 Read through Locks 207
planning 1 Row Level Locking 206
Coupling Facility RRS Attachnment Facility 209
availability considerations 141 Type 2 Indexes 206
coupling facility 140 Copy Utility 200
Coupling Links 19 Data Sharing 203
Dynamic Coupling Facility Dispatching 11, 17 Disaster Recovery 202
Integrated Coupling Migration Facility 16, 18 High Availability Configuration 205
internal coupling facility 16 Inline Copy
redundancy 141 LOAD and REORG 201, 208
Standalone Coupling Facility 9674 19 Logging 201
types 142 Recovery 201
coupling facility Resource Manager 246 in a Data Sharing Environment 204
Coupling Links 19 Recovery Features 199
CTC Stored SQL Procedures 62
see Channel to Channel TCP/IP support 63
why choose Data Sharing 204
DB2 Attachment Facility
D CICS 181
DASD Matrix Function Table 134 CICS/TS 181
DASD Subsystems 97 MVS Workload Manager 182
data abstraction 270 DBCTL
Data Accelerator 71, 73 as CICS/IMS interface 180
data isolation DCE Remote Procedure Calls (DCE RPC) 164, 167
Database partitioning 275 designing, Continuous Availability 1
logical DB definitions 274 DFSMS 73
Data Migrator 134 Concurrent Copy 73
Data Sharing DFSMS/MVS Network File System (NFS) 61
Buffer Consistency 263 Disaster Recovery 281
Data Copy Sharing 255, 256, 258, 259, 260, 261 and continuous availability 286
CNT FileSpeed 258 Data loss 281, 285
CNT High Speed Piping 261 DB2 202
Data Piping Transfer 260 IMS
File transfer 256 Extended Recovery Facility (XRF) 188
IBM Client Input/Output Sockets 259 Remote Site Recovery (RSR) 189
IBM RVA Snapshot feature 258 Service loss 281, 285
Serial Transfer 258 Disk for the S/390 Multiprise 2000 21
DB2 203 Disk migration 132
DB2 Recovery 204 Disk: major IBM disk products 105
in a Parallel Sysplex 147 Distributed Computing Environment (DCE) 63
Locking 263 Distributed File Service (DFS) 61, 63, 83
Real Data Sharing 262 OpenEdition Hierarchical File System (HFS) 61
among heterogeneous platforms 264 User Environment 85
among homogeneous platforms 263 Distributed Program Link (DPL) 164, 167
Storage Sharing 261 Distributed Transaction Processing (DTP) 164, 167
Universal Data Access 255 Dual Copy 99, 109, 133
database
logical definitions 274

Index 299
Dynamic Enterprise Systems Connections
Application update with XRF 277 and Geographically Dispersed Parallel
Coupling Facility Dispatching 11, 17 Sysplex 287
DASD address switching (P/DAS) 111 Error Correction Code 11
ESCD port reassignment 46 ESCD
ESCD port upgrade 46 see ESCON Director
I/O Reconfiguration Management (DRM) 13 ESCM
Memory Sparing 12 see ESCON Manager
SAP Sparing / Reassignment 11 ESCON
Storage Reconfiguration (DSR 2) 11 see Enterprise Systems Connection
Transaction Balancing 169 ESCON Channel Extender
Transaction Routing 168 9036 and PPRC 51
Workload Balancing 147 models 51
Dynamic disk reconstruction 118 ESCON Converter
types 50
ESCON Director
E ESCON Manager 236
ECC 11 ESCON Manager 236
eliminating batch window 278 ESCON Multiple Image Facility
EMIF sharing ESCON channels 39
see ESCON Multiple Image Facility sharing ESCON CTCs 40
Enterprise Systems Connection ESCV
9032 46 see ESCON Converter
9033 46 Extended Distance Feature
9034 50 Extended Recovery Facility (XRF)
9035 50 recovering CICS 174
9036 and PPRC 51 recovering IMS 188
9036 ESCON extender 51 recovering MQSeries 219
9036 models 51 Extended Remote Copy
9729 carrying ESCON channels 51 see XRC
Byte Multiplexer Channel 38 External CICS Interface (EXCI) 164, 167
CBY 38
chained ESCDs 37
channel path configuration 44 F
CNC 38 failure detection 272
configuring channels for availability 41 code failure 272
Converter Channel 38 FFST 108
CTC 38 Fiber Optic
CTC path configuration 45 advantages 35
CVC 38 ESCON Channel 35
Distance 37 multi-mode 37
error checking 35 single mode 38
ESCD control 56 Forward Recovery
ESCD path availability configuration 51 IMS 187
ESCON Channel Types 38 Function Shipping (FS) 164, 166
ESCON Converter 50
ESCON CTCs with EMIF 40
introduction 35 G
LED card 37 Global Enterprise Manager 241
Non-disruptive change 38 Global Resource Serialization (GRS)
Parallel Channel Conversion 38 ring configuration 60
PPRC and ESCON Directors 50 star configuration 60
Serial Channel 35
sharing ESCON channels with EMIF 39
Speed 37
H
Hardware Configuration Dialog
Topology 36
changing ESCON Channel types 38
use of ESCON Manager 57
ESCD control 56
use of HCD 38
XDF card 38

300 Continuous Availability S/390 Technology Guide

Hardware Configuration Manager 233 IMS (continued)
Hardware Management Console Programming support (continued)
see HMC Message Processing Programs (MPPs) 185
HCD Recovery
see Hardware Configuration Dialog Backward Recovery 187
Hiperbatch 73 Forward Recovery 187
HMC 26 Recovery features
HMC Recommendations 27 Backup Copies 186
Local Site 27 Concurrent Image Copy 187
Remote Site 28 DFSMS Concurrent Copy 187
Security Considerations 29 Fast Database Recovery 192
hot-plugging 13 Logging 187
Synchronization Points 186
recovery with MQSeries 219
I Remote Site Recovery (RSR) 189
IBM Client Input/Output Sockets 259 Synchronization Points 186
IBM Net.Commerce for OS/390 90 syncpoint coordination
Secure Electronic Transaction (SET) 90 with MQseries 219
IBM RAMAC Virtual Array two-phase commit 197
and CNT FileSpeed 258 VTAM Generic Resource 194
and CNT High Speed Piping 261 Workload router 195
ICKDSF IMS Transaction Manager
Peer to Peer Remote Copy 61 Data Entry Database (DEDB) 186
ICSS Database Partitioning 186
see Internet Connection Secure Server Databases support
implementing, Continuous Availability 1 DB2 Tables 186
IMS Full Function DataBase 186
and OS/390 RRS 197 Main Storage Database (MSDB) 186
ARM support 197 Full Function DataBase 186
Backup Copies 186 Info/Man 240
Common Queue Server 193 Integrated Coupling Migration Facility 16, 18
Concurrent Image Copy 187 Inter-System Communication (ISC) 164, 165
Data Entry Data Base (DEDB) 191 Internal Battery Feature (IBF) 14
Data Entry Database (DEDB) internal coupling facility 16
online add, change and delete 194 Internet
Data Sharing see Network Computing
Data Entry Data Base (DEDB) 191 internet connection 87
Full Function Database 190 Internet Connection Secure Server (ICSS) 64, 90
in Parallel Sysplex 190 Common Gateway Interface (CGI) 93
Database partitioning 196 HTTP 92
Database Recovery Control (DBRC) 188 IBM Net.Commerce for OS/390 90
Daylight Saving Time (DST) 197 Secure Electronic Transaction (SET) 90
DEDB Online Change 194 Internet Connection Application Program Interface
DFSMS Concurrent Copy 187 (ICAPI) 92
Disaster Recovery 188 Internet Content Selection (PICS) 93
Distributed Sync Point 197 Java 93
Extended Recovery Facility (XRF) 188 OpenEdition Hierarchical File System (HFS) 92
Extended Terminal Option (ETO) 196 S/390 Criptographic Hardware 64
Fast Database Recovery 192 Simple Network Management Protocol (SNMP) 93
Full Function Database 190 SOCKS 93
IMS/ESA Partition Support Product 196 System Measurament Facility (SMF) 92
Internet Web Usage Mining 92
connection from IMS 88 Workload Manager 92
Internet Connection 197 Workload Manager (WLM) 64
Logging 187 Introduction 1
Programming support Systems Design 1
Batch message processing programs
(BMPs) 185
Batch programs 186
Interactive Fast Path Programs (IFPs) 185

Index 301
 MQSeries (continued)
J Recovery Features (continued)
Java Page data sets 218
CICS/ESA internet gateway 87 recovery with XRF 219
Syncpoints 220
two-phase commit 221
L Unit of works 220
LOGR couple dataset 146
Remote queue 214
Reply-to queue 214
M Simple and Staged Transfer 215
Message and Queueing Syncpoints 220
see MQSeries Temporary Dynamic queue 214
Microprocessor Complex PCI Card 21 Transmission queue 214
Migration: DASD 132 Trigger events 215
MQSeries Triggers 215
, and CICS recovery 219 two-phase commit 221
, and IMS recovery 219 Undelivered message queue 214
, Extended Recovery Facility (XRF) 219 Unit of works 220
Alias queue 214 , and message persistence 221
Bootstrap data set 219 getting messages 221
Checkpoint records 219 putting messages 220
Command queue 214 triggering messages 221
Data conversion 216 Multi-Region Operation (MRO) 164
Dead-letter queue 214 Multiprise 2000 20, 99
Distributed Queue Management (DQM) 215 Internal disk 21
Simple and Staged Transfer 215 Muxmaster
Event queue 214 see Wavelength Division Multiplexer
Initiation queue 214
Internet gateway 88
Local queue 214
N
Net.Data
Log data set 219
DB2 WWW connection 87
Message channel agent 215
NetView for OS/390 235
Message channels 215
NetView Performance Monitor 238
Message queue 214
Network Computing
Messages
access from CICS applications 182
Application data 212
AIF gateway 88
Control Information 213
CICS/ESA gateway 87
Expiry time 213
CICS/ESA gateway for Java 87
Message identifier 213
connection from IMS 197
Message persistence 213
DB2 WWW connection 87
Message type 213
e-Business 94
Priority 213
IMS WWW Connection 88
Model queue 214
Internet Connection Secure Server (ICSS) 64, 90
Page data sets 218
MQSeries gateway 88
Permanent Dynamic queue 214
S/390 Gateways 86
Programs communication 216
Secure Electronic Transaction (SET) 90
Bidirectional 216
Network File System (NFS) 61
Client-server 217
Remote 217
Unidirectional 216 O
Queue Managers 213 object-oriented programming
Queue Types 214 application design 269
Queues 213 data abstraction 270
Recovery Features 218 OpenEdition
Bootstrap data set 219 Setup Verification Program (SVP) 89
Checkpoint records 219 OpenEdition Hierarchical File System (HFS) 61, 80
CICS recovery 219 Backuping up Open HFS Data sets 82
IMS recovery 219 Distributed File Service (DFS) 61
Log data set 219

302 Continuous Availability S/390 Technology Guide

OpenEdition Hierarchical File System (HFS) OS/390 (continued)
(continued) Function Groups (continued)
excengability 82 Network Computing Services 65
HFS Availability 81 Security Server 65
Individual data set availability 83 Softcopy Services 65
Security 82 System Services 65
transporting with DFSMS 85 Systems Management Services 65
OpenEdition MVS Files UNIX Services 65
Access Method Support 62 Function Integration 64
Operation Planning and Control 239 Communication server 64
OS/390 59 Internet Connection Security Server 65
Abend Detection Enhancements 72 Security Server 65
Availability Enhancements Hiperbatch 73
DB2 Stored SQL Procedures 62 I/O Error Propagation Enhancements 72
DB2 TCP/IP support 63 Internet
DFSMS/MVS Network File System (NFS) 61 AIF gateway 88
Distributed Computing Environment (DCE) 63 CICS/ESA gateway for Java 87
Distributed File Service (DFS) 61, 63 CICS/ESA Internet gateway 87
Global Resource Serialization (GRS) 60 DB2 WWW connection 87
ICKDSF 61 IMS WWW connection 88
Internet Connection Secure Server (ICSS) 64 MQSeries gateway 88
LNKLST 60, 62 Net.Data 87
LPALST 60 S/390 Gateways 86
Network File System (NFS) 61 Internet Connection Secure Server (ICSS) 90
OpenEdition MVS Files 62 JES3 Availability
PARMLIB Concatanation 60 Dynamic Update Support 74
PARMLIB Symbolic Pre-processor 61 FASTER RESTART 75
S/390 Criptographic Hardware 64 REFRESH with HOTSTART 75
SAP R/3 Application Enablement 63 Operating System Co-existence 68, 69
Sysclone Uniqueness 61 Recoverable Resource Management Services
Sysplex Availability 63 (RRMS) 75
Sysplex I/O Priority Management 63 Rolling OS/390 across a Sysplex 69
System Logger 63 Sequential Data Striping 73
TSO/E Session Distribution 63 SmartBatch 70
Unix 61 System Pre-Tested 66
Unix Services 63 Verifying UNIX Availability 89
VTAM Generic Resource 62 Windows NT application support 94
VTAM multi-node persistent session (MNPS) 64 OS/390 File Caching 86
Workload Manager (WLM) 62 OS/390 Open Edition 79
XCF Coupling Facility Failure Policy 60 OS/390 RRMS
XCF system initialization 63 see Recoverable Resource Manager Services
XES Lock support 60 OS/390 System Authorization Facility (SAF) 82
Batch Accelerator 71 OS/390 UNIX Availability
BatchPipePlex 70 ADSM 85
Bristol Technology 94 Data Availability 80
Data Accelerator 71, 73 Distributed File Service (DFS) 83
Delivery and Service 67 File Caching 86
DFSMS 73 HFS Availability 81
Backup retry 74 OpenEdition Hierarchical File System (HFS) 80
Concurrent Copy 73 Backup HFS data sets 82
Extended Remote Copy (XRC) 74 exchangability 82
Peer-to-Peer Remote Copy (PPRC) 74 Individual data set 83
Remote Copy 74 transporting with DFSMS 85
e-Business 94 Process Availability
Function Groups UNIX Processes and MVS 86
Application Enablement Services 65 UNIX Processes and PR/SM 86
Communications Server 65 OSA-2 22
Distributed Computing Services 65 Automatic handling of session overflows 25
LAN Services 65

Index 303
OSA-2 (continued) Peer-to-Peer Remote Copy (continued)
Load Balancing 25 P/DAS 111
OSA/SF 26 P/DAS Control Bloc 112
Redundant/backup connections 25 PPRC 100
SNA/APPN Networking Protocol 24 PPRC and ESCON Directors 50
TCP/IP Networking Protocol 23 shsring paths over chained ESCDs 55
virtual IP addressing 24 Peer-to-Peer Remote Copy (PPRC) 74
OSA/SF 26 Performance Management 250
Performance Reporter for MVS 237, 251
planning, Continuous Availability 1
P PPRC 104, 110, 133
P/DAS 111 see Peer to Peer Remote Copy
and geographically dispersed Parallel Sysplex 287 PPRC Dynamic Address switching
Parallel Channel 35 see P/DAS
Parallel Sysplex PR/SM
9037 sysplex timer 143 Dynamic Storage Reconfiguration 11
asynchronous CF requests 159 EMIF 12
capacity 138 Time Management Reporting 12
CFRM policy overview 246 UNIX Processes 86
components 139 Up to 15 LPARs 12
configuration discussions 147 Processor 5
coupling facility 140 Processor Availability Facility (PAF) 11
Coupling Links 142
CP/SM 247
datasharing 137 R
daylight saving considerations 145 RAID 98
EPSO service offering 153 Berkeley 98
Geographically dispersed 287 History 98
ICMF 156, 159 Level 0 99
in a Parallel Sysplex 147 Level 1 99
Internal Coupling Facility use 141 Level 5 100
Internal Coupling Migration Facility use 141 Level 6 101
introduction to 137 RAMAC 3 Array 106
monoplex 160 RAMAC 3 Array: microcode upgrade 118
non-disruptive change 151 RAMAC Array Family 105
SFM 151 RAMAC Array subsystem 99
sharing DASD with GRS 161 RAMAC Drawers 101, 117
Single System Parallel Sysplex 154 RAMAC Scalable Array 99, 101, 128
synchronous CF requests 159 RAMAC subsystem 120
Sysplex Couple Datasets 146 RAMAC Virtual Array 99, 121
Sysplex Failure Manager 245 Deleted Data Space Release 125
systems management 244 Disk Array 123
testing 152 Internal Structure 121
WLM 157 Log-Structured Array 125
WLM overview 245 SnapShot Copy 127
WLM policy 151 Recoveable Resource Manager
workload balancing 137 and IMS recovery 197
PARMLIB Recoverable Resource Management Services 233
Concatanation 60 Recoverable Resource Manager
Symbolic Pre-processor 61 and DB2 recovery 209
Partial I/O Restart 13 recovery
Partial Memory Restart 13 code failure 272, 273
PC Server System/390 21 data failure 273
Peer to Peer Remote Copy Recovery Tiers
ICKDSF 61 tier 0 - No Offsite Data 282
Peer-to-Peer Remote Copy tier 1 - PTAM 282
9036 and PPRC 51 tier 2 - PTAM + Hot Site 283
and geographically dispersed Parallel Sysplex 287 tier 3 - Electronic Vaulting 283
Data Flow 110 tier 4 - Active Secondary Site 283

304 Continuous Availability S/390 Technology Guide

Recovery Tiers (continued) Sysplex Failure Manager
tier 5 - Two Site Two Phase Commit 284 connectivity failure 245
tier 6 - Zero Data Loss 284 o v e r v i e w 151
reducing batch window 278 SFM overview 245
Remote Copy 74, 100, 102, 104, 109 systems failure 245
Remote Site Recovery (RSR) 189 Sysplex Timer
Resource Definition Online (RDO) 178 daylight saving considerations 145
Resource Management Facility 250 description 143
Resource Manager Interface (RMI) System Automation for OS/390 236
CICS-DB2 Attachment Facility 181 System Data Mover 114
CICS/TS 181 Systems Automation for OS/390
RMF 250 ESCON Manager 57
RMI Systems Management
see Resource Manager Interface Adstar Distributed Storage Manager 241
RRMS Automated Operations and Control 236
see Recoverable Resource Manager Services Automatic Restart Manager 233
RS/6000 with S/390 Server-on-Board 21 Batch Accelerator 243
RSA 128 batch management 242
RVA 121 CFRM policy 246
continuous availability topics 231
CP/SM 247
S ESCON Manager 236
S/390 Microprocessor Complex PCI Card 21 Global Enterprise Manager 241
SAP R/3 Application Enablement 63 Hardware Configuration Manager (HCM) 233
SDM 114 in a Parallel Sysplex 244
Seascape Architecture in OS/390 232
IBM Magstar Virtual Tape Server 265 Info/Man 240
IBM Network Storage Manager 265 introduction 229
Secure Electronic Transaction (SET) 90 NetView 235
Sequential Data Striping 73 NetView Performance Monitor 238
Service Level Agreement (SLA) 249 Operation Planning and Control 239
Service Level Reporter (SLR) 252 Performance Reporter 237
SET Recoverable Resource Management Services
see Secure Electronic Transaction (RRMS) 233
SHARE 78 Recovery Tiers Service Level Reporter (SLR) 238
see Recovery Tiers Smartbatch 243
Single System Parallel Sysplex 154 Sysplex Failure Manager 245
SmartBatch 70, 243 System Automation for OS/390 236
Abend Detection Enhancements 72 Target System Control Facility 236
I/O Error Propagation Enhancements 72 Tivoli 234
SMS 132 TME 10 234
SnapShot Copy 103, 127 TME framework 234
SnapShot feature Workload Manager 245
and Data Copy Sharing 258, 261
see IBM RAMAC Virutal Array
sparing 118, 133 T
Standalone Coupling Facility 19 Target System Control Facility 236
Striping 99 Tiers
Subspace Group Facility 12 see Recovery Tiers
Subsystem Storage Protection 12 Time Management Reporting 12
Support Element (SE) 26 Tivoli Management Environment 10
Sysplex Couple Datasets Adstar Distributed Storage Manager 241
ARM couple dataset 146 CORBA 234
CFRM couple dataset 146 Global Enterprise Manager 241
description 146 Info/Man 240
LOGR couple dataset 146 introduction 234
SFM couple dataset 146 NetView 235
WLM couple dataset 146 NetView Performance Monitor 238
XCF couple dataset 146 Operation Planning and Control 239

Index 305
Tivoli Management Environment 10 (continued)
Performance Reporter 237
System Automation for OS/390 236
TME framework 234
Transaction Manager
see CICS/TS
Transaction Routing (TR) 164, 166

U
Unix
Services in OS/390 63
Verifying Availability 89

V
VTAM Generic Resource
and CICS support 173
and IMS support 194
and TSO support 62
VTAM Persistent Session 64
and CICS support 174

W
Wavelength Division Multiplexer
and Geographically Dispersed Parallel
Sysplex 287
Wavelength Division Multiplexor
backup fibre 51
carrying ESCON channels 51
WLM couple dataset 146
Workload Manager
DB2 Attachment Facility 182
Internet Connection Secure Server (ICSS) 92
policy 151
Workload Manager (WLM)
OS/390 Availability Enhancements 62

X
XCF couple dataset 146
XRC 74, 100, 105, 114, 133
Data Flow 116
XRF
see Extended Recovery Facility

306 Continuous Availability S/390 Technology Guide

ITSO Redbook Evaluation
Continuous Availability S/390 Technology Guide
SG24-2086-00

Your feedback is very important to help us maintain the quality of ITSO redbooks. Please complete this
questionnaire and Fax it to: USA International Access Code + 1 914 432 8264 or:
• Use the online evaluation form found at http://www.redbooks.ibm.com
• Send your comments in an Internet note to [email protected]

Which of the following best describes you?

__Customer __Business Partner __Solution Developer __IBM employee
__None of the above

Please rate your overall satisfaction with this book using the scale:
(1 = very good, 2 = good, 3 = average, 4 = poor, 5 = very poor)
Overall Satisfaction ____________

Please answer the following questions:

Was this redbook published in time for your needs? Yes____ No____

If no, please explain:

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

What other redbooks would you like to see published?

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

Comments/Suggestions: (THANK YOU FOR YOUR FEEDBACK!)

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

_____________________________________________________________________________________________________

 Copyright IBM Corp. 1998 307

Continuous Availability S/390 Technology Guide SG24-2086-00

Printed in the U.S.A.

IBML
SG24-2086-00