We’ve Got a Problem…

An Introduction to z/OS Dumps and Debugging

Steve Pryor
[email protected]
1.919.833.8426 x0162

11/30/2021 1
 ●
Types of Errors
●
Return code / reason code
●
ABEND – user / system
●
Loop / hang / wait – S322, S522
●
Abend reason code
●
Exception condition

11/30/2021 2
 Interrupts and Exceptions
●
Classes of Interrupts ●
Program Exceptions
●
SVC ●
Addressing
●
Program ●
Protection / Translation
●
Machine-check ●
Operation / Specification
●
External ●
Divide / Overflow / Data
●
I/O ●
others
●
Restart

WLIC field in dump, e.g. 00060011:

Wait code (WC): indicates whether RB is in a WAIT
Interrupt Code: (IC): 2 bytes indicating type of exception
Instruction Length Code (ILC): length of last instruction executed

11/30/2021 3
 Common Types of Errors
●
Addressing Errors
●
Storage Overlay
●
Addressing inaccessible storage (0C4, 0Cx)
●
Often indexing error
●
Data errors
●
Divide / overflow error (often due to bad data) (0C7, 0C9)
●
Instruction errors
●
Bad branch (0C1, 0Cx)
●
Privileged instruction / wrong key (0C2, 0C4)
●
Other – timing, loop, wait...
11/30/2021 4
 Tools – Service Aids
●
System tools
●
AMATERSE (TRSMAIN)
●
Compress (or decompress) dumps (or other datasets) for transfer
●
PACK (simple) or SPACK (complex) format
●
AMBLIST
●
Map Load Modules and Program Objects – useful in locating failing CSECT
●
AMASPZAP
●
Display / update load modules, VTOC records
●
AMAPDUPL
●
upload data to IBM for analysis

11/30/2021 5
 ABEND Dumps
●
ABEND, CALLRTM or SETRP macros, or CANCEL,DUMP
●
Formatted Dumps
●
Symptom dump (Message IEA995I) – basic error information
●
SYSUDUMP – simple program / system information
●
SYSABEND – more detailed system information
●
Unformatted dumps
●
SYSMDUMP (SVC dump) – detailed program and system info
●
Format / analyze with IPCS

Dump content defaults: SYS1.PARMLIB(IEADMP00, IEAABD00, IEADMR00)

Console command: DISPLAY DUMP,OPTIONS

11/30/2021 6
 Other Dumps
●
SLIP dump – SVC dump resulting from SLIP console command
●
DUMP command dump – result of DUMP console command
●
SNAP dumps – SNAP macro displays virtual storage
●
Transaction dump – IEATDUMP macro
●
Standalone dump – for catastrophic system errors

11/30/2021 7
 Examining the Dump
●
PSW – Program Status Word
●
Address of most recently executed/current instruction, key, amode, ...
●
General Purpose Registers
●
Program base register (often R12), other registers 4K apart may be add’l base
●
R14 – often return address after BAL
●
Request Blocks (Rbs) – registers in succeeding RB
●
PRB – created at program interrupt
●
SVRB – created at SVC interrupt
●
CDE / LPDE chain – programs in use

11/30/2021 8
 Sample Program Check – ABEND 0Cx
●
PIC 4 – protection exception
●
PIC 10, 11, 38, 39, 3A, 3B – translation fault – PSW addr NSI
00004E 5800 CC48 00C48 60 L R0,NRECS NUMBER OF RECORDS
000052 4130 C068 00068 61 LA R3,RECORDS POINT TO FIRST RECORD
000056 62 COMPLOOP DS 0H
000056 D504 CC94 3000 00C94 00000 63 CLC =C'STEVE',0(R3) IS IT THE RECORD WE WANT?
00005C 4780 CC4C 00C4C 64 BE GOTIT IF SO->
000060 4130 3050 00050 65 LA R3,80(,R3) BUMP TO NEXT RECORD
000064 47FO C056 00056 67 B COMPLOOP UNTIL EOF

000068 69 RECORDS DS 0H
000068 D3D6C9E240404040 70 DC CL80'LOIS 168251'
0000B8 C2D9C5D5C4C14040 71 DC CL80'BRENDA 051063'
000108 D1D6C54040404040 72 DC CL80'JOE 050155'
000158 C6D9C5C440404040 73 DC CL80'FRED 123456'
0001A8 C4C9C1D5C5404040 74 DC CL80'DIANE 180294'

11/30/2021 9
 Symptom Dump – Basic Error Information
IEA995I SYMPTOM DUMP OUTPUT 216
SYSTEM COMPLETION CODE=0C4 REASON CODE=00000011
TIME=17.29.12 SEQ=01137 CPU=0000 ASID=0037
PSW AT TIME OF ERROR 078D2000 9FA00056 ILC 6 INTC 11
ACTIVE MODULE ADDRESS=00000000_1FA00000 OFFSET=00000056
NAME=READLIST
DATA AT PSW 1FA00050 - CC484130 C068D504 CC943000
GR 0: 00000026 1: 00007018
2: 00000040 3: 1FA01008
4: 00006FF8 5: 008F81A0
6: 008C8FC8 7: 00FB4880
8: 008FE820 9: 008F83D8
A: 01DA0400 B: 1FA01000
C: 9FA00000 D: 00007018
E: 80FD42E0 F: 00000000
END OF SYMPTOM DUMP

11/30/2021 10
 SYSUDUMP / SYSABEND RB Structure
PRB: 008D4C98

-0020 XSB...... 7FFFDAC0 FLAGS2... 00 RTPSW1... 078D2000 9FA00056 RTPSW2... 00060011

-000C 1FA01800 FLAGS1... 00000000 WLIC..... 00060011

+0000 RSV...... 00000000 00000000 SZSTAB... 00110082 CDE...... 008FE7F0

+0010 OPSW..... 078D2000 9FA00056 SQE...... 00000000 LINK..... 008D4D20
+0020 GPR0-3... 00000064 00006FF8 00000040 008D8D64
+0030 GPR4-7... 008D8D40 008F81A0 008C8FC8 00FB4880
+0040 GPR8-11.. 008FE820 008F83D8 01DA0400 00000001
+0050 GPR12-15. 04499CA0 00006F60 008FE86C 008FE850

SVRB: 008FF708

-0020 XSB...... 7FF991D0 FLAGS2... 00 RTPSW1... 00000000 00000000 RTPSW2... 00000000

-000C 00000000 FLAGS1... 22000000 WLIC..... 00020033
+0000 RSV...... 00000000 00000000 SZSTAB... 001ED022 CDE...... 00000000
+0010 OPSW..... 070C1000 849EBFA4 Q........ 00000000 LINK..... 008D4C98
+0020 GPR0-3... 00000026 00007018 00000040 1FA01008
+0030 GPR4-7... 00006FF8 008F81A0 008C8FC8 00FB4880
+0040 GPR8-11.. 008FE820 008F83D8 01DA0400 1FA01000
+0050 GPR12-15. 9FA00000 00007018 80FD42E0 00000000

11/30/2021 11
 SYSABEND / SYSUDUMP Control Blocks
●
CDE/LPDE/XTLST – locations of programs in storage
008FE7F0 NAME..... READLIST ENTPT.... 9FA00000 CHAIN.... 00000000 RRBP..... 008D4C98 XLMJP.... 008FE958
USE...... 0001 SP....... FC R020..... 00000000 R024..... 00000000 R028..... 00000000
REENTERABLE. REUSABLE. APF LIBRARY.
PM LOADER WAS UP WHEN CREATED. LONGPARM OK.
CDE EXTENSION EXISTS.
CDX...... 7F6AEB28

●
Virtual Storage Map – virtual storage addresses, lengths, subpools
●
TIOT – Task I/O Table – DDnames and volumes
●
Data Management Control Blocks (DCBs & DEBs)
●
TCB Summary, Save Area Trace
TCB COMPLETION TCB PRIMARY TCB TCB SECONDARY RTM2 WA TCB
ADDRESS CODE PKF DISP FLAGS LMP DSP DISP FLAGS ADDRESS FBYT
-------- ---------- --- ----------- --- --- ---------- --------- ----
008FD520 00000000 00 00008004 00 FF FF 00000000 00000000 0000
008FF068 00000000 00 00008004 00 FF FF 00000000 00000000 0004
008FED90 00000000 00 00008004 00 FF FF 00000000 00000000 0004
008F81A0 00000000 80 00000004 00 FF FF 00000000 00000000 0004
008D4D20 940C4000 80 01000000 00 FF FF 00000000 7F6A2CF0 0804
11/30/2021 12
 Fixing the Error - SPZAP
●
Display or update
●
Load library members
●
Program Objects
●
Disk records (typically DSCBs)
//ZAP EXEC PGM=IMASPZAP,PARM=(IGNIDRFULL,PRECHECK)
//SYSPRINT DD SYSOUT=*
//SYSLIB DD DISP=SHR,DSN=SJP.TEST71.LOAD
//SYSIN DD *
NAME READLIST READLIST
VER 0064 47F0,C056 B COMPLOOP<--- ORIGINAL
VER 0CA0 0000,0000,0000,0000
REP 0064 47F0,CCA0 B PATCH <--- REPLACEMENT
REP 0CA0 4600,C056 PATCH BCT R0,COMPLOOP
REP 0CA4 47F0,CC66 B QUIT
CHECKSUM ACC9B166

11/30/2021 13
 SYSMDUMP – SVC Dumps
●
IPCS
●
Manage dump inventory
●
Copy, delete, list
●
Analyze dumps interactively
●
ISPF interactive commands (or batch)
●
Browse, format, analysis commands

11/30/2021 14
 SYSMDUMP – Data Exception
●
Invalid zoned decimal data
//SORT EXEC PGM=SORT
//SORTIN DD *
VAL0 123
VAL0 456
VAL3 789
VAL3 90. FBS/4160/4160
VAL8 654
VAL8 321
//SYSMDUMP DD DSN=SJP.SYSMDUMP,DISP=SHR
//SORTOUT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(1,4,CH,A)
SUM FIELDS=(7,3,ZD)

11/30/2021 15
 IPCS Commands
●
IP STATUS FAILDATA (option 2.2, Status)
●
IP SUMMARY FORMAT ASID(x’nn’) (option 2.4 - Format)
●
IP SYSTRACE ASID(x’nn’) – (option 2.7.4 - Trace)
●
IP WHERE addr , IP LIST addr opt
●
IP CPF addr STR(cbname) – (option 1 – Browse)
----------------------- IPCS Subcommands and Abbreviations ------------------
ADDDUMP | DROPDUMP, DROPD | LISTDUMP, LDMP | RENUM, REN
ANALYZE | DROPMAP, DROPM | LISTMAP, LMAP | RUNCHAIN, RUNC
ARCHECK | DROPSYM, DROPS | LISTSYM, LSYM | SCAN
ASCBEXIT, ASCBX | EPTRACE | LISTUCB, LISTU | SELECT
ASMCHECK, ASMK | EQUATE, EQU, EQ | LITERAL | SETDEF, SETD
CBFORMAT, CBF | FIND, F | LPAMAP | STACK
CBSTAT | FINDMOD, FMOD | MERGE | STATUS, ST
CLOSE | FINDUCB, FINDU | NAME | SUMMARY, SUMM
COPYDDIR | GTFTRACE, GTF | NAMETOKN | SYSTRACE
COPYDUMP | INTEGER | NOTE, N | TCBEXIT, TCBX
COPYTRC | IPCS HELP, H | OPEN | VERBEXIT, VERBX
CTRACE | LIST, L | PROFILE, PROF | WHERE, W

11/30/2021 16
 Traces
●
Master Trace – system messages in wraparound table
●
System Trace – h/w and s/w events in table for ea. Processor
●
Interrupts, SVCs, I/Os, et al – PSW and related info
●
Component and Transaction Trace
●
Component-level trace for JES, LLA, LOGR, WLM, XCF, etc
●
GTF trace
●
specified events (SVC, interrupt, other) to trace dataset
●
GFS trace
●
Storage usage (GETMAIN/FREEMAIN)

11/30/2021 17
 Getting a SLIP Dump
●
PER SLIPs
●
Instruction fetch, storage alteration, successful branch, et al
●
Non-PER (Error event) SLIPs
D SLIP=DTS1
●
Abend, message IEE735I 22.56.54 SLIP DISPLAY 449
ID=DTS1,PER-IF,ENABLED
●
Job, asid, other selection criteria ACTION=SVCD,SET BY CONS
IBMUSER$,PRCNTLIM=10,00
MATCHLIM=1,0
●
ACTION RANGE=CURRENT.B49FE8
EVALUATED
●
SVCDump, CMD, trace/gtf options RANGE=00000000_00B49FE8,00000000_00B49FE8
JOBNAME=SJP1

●
D SLIP=id

11/30/2021 18
 AMBLIST – Map a Load Module

//AMB1 EXEC PGM=AMBLIST If LOAD pt ACC0000 = 00B38BB0

//SYSLIB DD DISP=SHR,DSN=SJP.TEST71.LOAD then
//SYSPRINT DD SYSOUT=* EPA of ACC2060 = 00B38BB0 + 11438
//SYSIN DD * = 00B49FE8
LISTLOAD DDN=SYSLIB,OUTPUT=XREF,MEMBER=ACC0000
// ISRDDN can be useful to display
locations in virtual storage
CONTROL SECTION
LMOD LOC NAME LENGTH TYPE RMODE
00 ACC0000 56C SD 31
570 ACC2016 138 SD 31
6A8 ACC0235 285 SD 31
930 ACC0711 C4 SD 31
9F8 ACC2220 1CA SD 31
BC8 ACC2124 1A4 SD 31
….
10F68 ACC2062 394 SD 31
11300 ACC2061 138 SD 31
11438 ACC2060 627 SD 31
11A60 ACC2056 2D4 SD 31

11/30/2021 19
 Console Commands
●
CHNGDUMP – defaults for dumped data
●
DISPLAY – display system dumps status, titles, options
●
DUMP – create SVC dump
●
DUMPDS – SYS1.DUMPxx auto allocation, clear, delete, names
●
SLIP – create SVC dump on abend, message, instruction, et. al.
●
TRACE – start/stop/change traces (system, master, component)

11/30/2021 20
 Securing Sensitive Information
●
Data Privacy for Diagnostics (DpfD)
●
OA57570 – for z15 machines and up
●
IARV64 SENSITIVE = YES/NO/UNKNOWN tags 64-bit memory objects
●
Redact data tagged as sensitive
●
Copy to ‘redacted’ dump using E35 exit BLSRTE35
●
OA58114 – DPfD Analyzer
●
IPCS Dialog and zfs database
●
Scan for and ‘learn’ to redact sensitive information
●
ANALYZE / REDACT / REPORT / FEEBACK / INGEST / EXTRACT

11/30/2021 21
 Additional Debugging Tools
●
Colesoft z/XDC, c/XDC, DUMP/XDC
●
IBM z/OS Debugger
●
BMC Abend-Aid
●
TSO ‘TEST’ command

11/30/2021 22
 Additional Topics
●
AR mode, XMEM mode, generic tracker, GTF
●
Dump supression (DAE), dump exit routines (IEAVTABx et al)
●
LE, Java, application-level debugging
●
REXX
●
And much more…

11/30/2021 23
 Documentation
●
MVS Diagnosis: Tools and Service Aids GA32-0905
●
All about dumps, traces, SPZAP, AMBLIST, et al
●
MVS Diagnosis: Reference
●
SVC nos. and plists, ENQ/DEQ, logrec, SYSEVENTs and more
●
MVS Interactive Problem Control System (IPCS)
●
Commands SA23-1382, User’s Guide SA23-1384, Customization SA23-1383
●
MVS System Codes SA38-0665
●
ABEND codes, wait state codes
●
MVS System Commands
●
console commands
11/30/2021 24
 Summary / Q&A
●
Next webinar: December 2021
●
Introduction to z/OS Utilities

●
Reminders
●
DTS Products release 7.1 available; incl support for z/OS 2.5
●
Product use available for a year – just ask!
●
Send your DFSMS ACS routines or ACC/SRS rules for analysis

11/30/2021 25