Advance SVCDump Analysis
Advance SVCDump Analysis
Advance SVCDump Analysis
STRATEGIES
MVS/XA/ESA Problem
Solving: Part III —
Advanced SVCDUMP
Analysis
T
o recap this series so far, I’ve examined using IPCS for MVS/ESA
Version 5 (Part I, Technical Support, October 1996); and Part II
This article will take us (November 1996) examined a technique called one-page
beyond the “one-page MVS/XA/ESA abend analysis. This article will take us beyond
the “one-page MVS/XA/abend analysis” to gather the relevant information
MVS/XA/abend analysis”
and quickly determine where to go for further help with problems other than
to gather the relevant
the simple abend. Future articles will examine slip traps/GTF tracing and
information and quickly standalone dump analysis.
determine where to go
for further help with
problems other than the Once upon a time, I believed that I ask that you commit to memory the com-
MVS/XA/ESA debugging (what I referred to mands to list system contention. The payback
simple abend. it as then) was about nailing down the exact will be immense. You will save a great deal of
problem to the instruction level. With the time solving problems and avoid that churning
increasing complexity of MVS/ESA Version 5 feeling in your stomach when a problem does
and the many other platforms supported, occur. The most important command to know
this methodology had to change. This article is the one that lets you display MVS/ESA
will examine: ENQ contention. In IBM systems with Global
Resource Serialization (GRS) or equivalent
■ displaying/correcting problems with third-party software, this command is the
system contention; DISPLAY GRS,CONTENTION or D GRS,C.
■ how to know when you must use The normal output of this command should be
the MVS/ESA FORCE command; the following two messages:
■ how to dump an address space
and get just the data you need; NO ENQ RESOURCE CONTENTION EXISTS
NO LATCH CONTENTION EXISTS
■ the two types of MVS/ESA SVCDUMPs
and how to tell them apart;
■ SVCDUMP structures (ASCB, ASXB, If you see these two messages, you can relax.
TCB, RB, CDE, LLE, and XL); Besides ENQ contention, GRS will also
■ the IBM contents directory entries (CDE) report any latch contention, if found.
you should know; The GRS latch manager is new with
■ using the trace table when you are ready MVS/ESA Version 5 and is used to serialize
for it; and resources within an address space. The GRS
■ a case study. latch manager provides a more specific
resource serialization mechanism and incurs
IDENTIFYING SYSTEMS much less overhead for address space serial-
CONTENTION CONDITIONS ization than using the MVS/ESA LOCAL
The essence of MVS/ESA problem solving LOCK. Many of the new MVS/ESA Version 5
is identifying system contention conditions. components, such as Workload Manager, use
If you remember nothing else from this article, GRS latches for low-overhead serialization.
Figure 3: Display Active With a Non-Zero ENQ SMC Value c o m p romise processing and sch e d u l e s
abnormal termination of the task via the
D A,TSOUSER SDUMP BRANCH=YES or ABEND macro.
IEE115I 14.55.03 96.131 ACTIVITY 633 The current task and request block infor-
JOBS M/S TS USERS SYSAS INITS ACTIVE/MAX VTAM mation in a scheduled dump is for the dump
00015 00085 00148 00023 00050 00148/00300 task. The dump must be analyzed to find the
TSOUSER IN PR A=00C8 PER=YES SMC=001 PGN=N/A DMN=N/A failing TCB and RB.
CT=099.775S ET=05.07.18
To find out what kind of dump you have,
WUID=TSU21757
WKL=TSO SCL=TSO P=1 execute the IPCS STATUS SYSTEM com-
SYSTEM ID PT11 DATE 5/10/96 96.131 LINE 90,313 mand. In the field PROGRAM REQUESTED
RGP=N/A SRVR=NO QSC=NO SVCDUMP you will see the name IEAVTSDT,
ADDR SPACE ASTE=1FD20200 which indicates a scheduled SVCDUMP.
In most situations with a scheduled SVC-
Figure 4: FORCE Out a TSO User With Address Space Destruction DUMP you will have to look at the current
and/or error address space(s) followed by the
FORCE U=TSOUSER task control blocks (TCBs) and request blocks
(RBs) to get a clear indication of what was
IEE301I TSOUSER FORCE COMMAND ACCEPTED
*ISG033E UNIDENTIFIED TASK JS FAILED WHILE IN MUST COMPLETE STATUS going on. Remember, your job is to narrow
ISG032E RESOURCE NAMED BLXDAS, Q % MAY BE DAMAGED the scope of the problem and first determine
ISG032E RESOURCE NAMED SYSIKJUA,TSOUSER MAY BE DAMAGED what vendor owns the code (not always IBM).
ISG032E RESOURCE NAMED SPFEDIT,TSOUSER.ISPPROF MAY BE DAMAGED
ISG032E RESOURCE NAMED SPFEDIT,TSOUSER.ISPPROF MAY BE DAMAGED For an IBM problem, the next goal is to be
ISG032E RESOURCE NAMED SPFEDIT,TSOUSER.ISPPROF MAY BE DAMAGED able to provide a good search argument so you
ISG032E RESOURCE NAMED SYSDSN,TSOUSER.SPFTEMP1.CNTL MAY BE DAMAGED can narrow down the number of possibilities.
IST804I CLOSE IN PROGRESS FOR TSO0012 OPENED BY TSOUSER The IPCS CBSTAT and IPCS VERBEXIT
ISG032E RESOURCE NAMED SPFEDIT,TSOUSER.ISPPROF MAY BE DAMAGED
IST400I TERMINATION IN PROGRESS FOR APPLID TSO0012 SRMDATA commands are good ways to
ISG032E RESOURCE NAMED SPFEDIT,TSOUSER.ISPPROF MAY BE DAMAGED check the health indicators of our current
$HASP310 TSOUSER TERMINATED AT END OF MEMORY and/or error address space. The IPCS
IEF743I IKJACCNT.TSOUSER FORCED - CODE SA22 - IN ADDRESS SPACE 00C8
IST805I VTAM CLOSE COMPLETE FOR TSO0012 CBSTAT command takes an ASCB address as
$HASP250 TSOUSER PURGED — (JOB KEY WAS ACD86C3F) input. For example, you would enter IPCS
CBSTAT XXXXXX. (where XXXXXX is the
ASCB address (make sure to include the
me to select a TCB for recovery termination. R ??,JOBNAME=HOGJOB,CONT period). You can find out if a address space is
TSO uses the CALLRTM macro when you swapped in or out, and the reasons why such
reply FSTOP to the TSO termination message. to make sure MVS will prompt us again with as input terminal wait (user hasn’t pressed
This program can be beneficial, but it can the IEE094D message (the CONT parameter enter) or detected wait (e.g., ENQ or DFHSM
also bring your system down in a heartbeat if does this). Our next reply to the IEE094D recall). You will also get information on
used improperly. See IBM APAR II05723 for message will tell MVS what areas we want address space status such as if the address
more details. dumped. I usually reply as follows: space was forced out (S0A22 abend). The
ASCB fields ASCBJBNI (jobname field for
R
CONSOLE DUMPS ??,SDATA=(LPA,LSQA,PSA,RGN,SQA,SUM,TRT,GR
initiated programs) and ASCBJBNS (jobname
There are certain circumstances where you SQ,CSA) field for start/mount/logon) will indicate what
will need to take a console dump of one or type of job you are looking at. If you look at
more address spaces for IBM or another vendor. which means I want just about every data area the eight bytes off of the address in ASCBJBNI
It’s important to make sure that you have all except all of the DAT-on nucleus (which I and see INIT, and look at ASCBJBNS and see
the pertinent information in the dump. The haven’t needed as yet). MVS will first capture YOURJOB, then you know that the system
IBM defaults for the areas to be dumped are the dump in a dataspace and issue the message: initiator just started job YOURJOB. I will
on the lean side. To be sure, I thought it would cover more on types of jobnames/recognition
be best to write down the steps on how to IEA794I SVC DUMP HAS CAPTURED: of dumps later in this article.
DUMPID=??? REQUESTED BY JOB (*MASTER*)
dump an address space. Let’s say we want to The IPCS VERBEXIT SRMDATA com-
DUMP TITLE=HOGJOB NOT CANCELLING
dump the address space of the batch job called mand was primarily designed to let us check
HOGJOB. To do this, issue the MVS command health indicators for the System Resources
DUMP COMM=(HOGJOB NOT CAN- where ??? is a generated number for the dump Manager (SRM) responsible for performance
CELLING). The COMM value will be our identifier. If you issue the DISPLAY management decisions in MVS/XA/ESA.
dump title and should reflect something DUMP,TITLES (D D,T) command you By using the IPCS VERBEXIT SRMDATA
meaningful. MVS will prompt you for more should see the dump in an appropriate command, we can see much more information
information with the following message: DUMP?? dataset (where ?? is from 00 to 99). than by using the IPCS CBSTAT command
If you’re using MVS dynamic dump datasets, which predominantly checks indicators in
*?? IEE094D SPECIFY OPERAND(S) FOR DUMP
COMMAND
then issue D D,T,AUTODSN=3 to list the last the ASCB. Since the output from IPCS
three dynamic dump datasets created. VERBEXIT SRMDATA is quite large, I just
where ?? is the outstanding reply number. We perform a FIND command for ASID in column
Identifying SVCDUMPs
wish to indicate that HOGJOB is to be two and continue the finds while I look at
dumped and also want to specify our SDATA This section examines the sch e d u l e d the data. See macro IRAOUCB in
dumped areas options. Since the SDSF line (asynchronous) dump where a system routine SYS1.MODGEN for more information on the
may not be long enough, we reply as follows: detects an abnormal situation that would SRM user control block (OUCB) fields.
SYSTEM STRATEGIES
Once you are done looking at the address Figure 5: IPCS SUMMARY KEYFIELD Output (A Section)
space, it’s time to look over the task control
block (TCB) and request block (RB) structures TCB: 008FE240
CMP. . . . . 00000000 PKF. . . . . 00 LMP. . . . FF
to see what was going on. Practice this first on DSP. . . . . FF
small problems so you can get a feel for what TSFLG. . . 00 STAB. . . . 008FF6E0 NDSP. . . 00000000
types of work (and loaded programs) are JSCB. . . . 008FFDFC BITS . . . . 00000000 DAR. . . . 00
present. The IPCS SUMMARY KEYFIELD RTWA . . . 00000000 FBYT1. . . 00
Task non-dispatchability flags from TCBFLGS4:
command shows you key fields in the ASCB, Top RB is in a wait
TCB and RBs. For the TCB you get the
completion code (COMP) where (for the most
part) any non-zero completion code should be Figure 6: Common IBM CDE Entries (Field FLCDE)
reviewed. The IPCS SUMMARY KEYFIELD IEAVAR00 - Region Control Task RCT initialization and termination.
command will automatically issue the IPCS
CBSTAT command so you see TCB status IEAVTSDT - SVC Dump Task.
along with the TCB. See Figure 5.
Request blocks are simply the representation IEFSD060 - Initiator Control Initialization.
of programs in stor age. There are some types IEESB605 - Job Scheduling routine. (If the next CDE NAME is an application name then this
of RBs which need to be clarified. When you jobname must be a started task.)
execute your program, a program request
block (PRB) is created. On the other hand, if IEFIIC - Initiator Interface Control (batch jobs only)
MVS has to execute a program on your IKJEFT01 - Terminal Monitor Program Initialization routine (TSO usage only). If previous
behalf, a supervisor request block (SVRB) is cdename is IEFIIC then TSO was run in batch.
created. The RB WLIC field is very important,
since it shows the wait count and interrupt IKJEFT02 - Terminal Monitor Program Mainline routine (TSO usage only). You may see this name
code (which is usually, but not always, the twice due to parallel tmp!
abend code). When the interrupt code (the IC IKJEFT09 - Terminal Monitor Program Second Level routine (TSO usage only). Unauthorized routines
of WLIC) is from a PRB, your program issued only.
the SVC. When the IC is from a SVRB, the
system issued a SVC on your behalf.
Another type of RB is an interrupt request the IPCS SUMMARY KEYFIELD command, manuals were discontinued in MVS/ESA
block. This block is created by the create only the first CDE is shown for a given RB in Version 4 and higher (to my dismay), so if
interrupt request block (CIRB) authorized the dump. If you want to see all of the CDEs for you have access to earlier logic manuals,
macro. The CIRB creates an interrupt queue a given RB then execute: keep them handy.
element which is high priority work tacked on
to a TCB. I will provide an example of an IRB RUNCHAIN ADDRESS(XXXXXX.) NAME(CDE) COMPONENT ANALYSIS
LINK(0) +
problem later in the case study. The contents The last section I want to cover in this article
CHAIN(999) LENGTH(32)
directory entry (CDE) is the physical repre- EXEC((CBFORMAT X STRUCTURE(CDE))) before the case study is component analysis.
sentation of a module, its entry point and use The IPCS VERBEXIT TRACE command
count. If you see multiple copies of the same or SYSTRACE TIME(LOCAL) for IPCS
CDE in a dump, you can be sure that the load I would recommend getting used to IPCS Version 5 and higher will give you the in-
module does not have the re-entrant and SUMMARY KEYFIELD, since it has less storage trace table. I find the trace table
reusable (RENT,REUS) linkage editor indi- output and is easier to use. If this is your invaluable when, for example, I know the area
cators on. The CDE is pointed to by the first time in an MVS/XA/ESA dump, the where the problem is and want to see the SVC
RBCDE field, which you will have to format amount of data presented even using IPCS instruction. From a SAS regression of many
with IPCS CBFORMAT since it is not listed S U M M A RY KEYFIELD can be ove r- trace tables, I have determined the frequency
in IPCS SUMMARY KEYFIELD. Two other whelming. I have seen many first- and sec- of trace table entries and the ones you should
fields of note are the load list element (LLE), ond-time dump readers looking over the be able to recognize. See Figure 8.
which is used to control load and delete refer- TCB/RB fields for CDE IEAVAR00. This is In life there are always exceptions, and the
ences to a load module, and the extent list the MVS/XA/ESA region control task pro- recovery event (RCVY) is one of them. A
(XL), which contains the load module address gram and in no way is where their problems RCVY PROG means that recovery termination
and size. I will not cover them further since I are. I have found that most dumps contain management phase 1 (RTM1) was entered for a
have not had to use them very often. TCB/RB information you need to be aware program check. RCVY PERC means that a
When you execute the IPCS SUMMARY of when locating your application or system given recovery routine could not correct the
KEYFIELD command , your RB information problem. See Figure 6. Once you learn these abend and is passing the error to the next recov-
is presented backwards. For a given TCB, if names, it will be easier to determine the type ery routine (percolation) from RTM1 to RTM2.
you want the last (most recent) RB in the of job you are looking at. RCVY FRR means that RTM1 is invoking a
dump you will go to the last RB for this TCB. Figure 7 presents an example of four SETFRR macro instruction which places the
For the times you use IPCS SUMMARY types of jobs and their CDE entries. The one address of a recovery routine in the FRR stack.
FORMAT (hopefully not too often due to the that is somewhat different is the batch job RCVY ABT is a request for abnormal end-of-
amount of output) this order is the other way running a DB2 command under the TSO task by the CALLRTM TYPE=ABTERM
around in that the most recent RB is presented terminal monitor program. The documenta- macro. If you’ve ever replied FSTOP to a P
immediately after the TCB. I thought I was tion for IEAVAR00, IEAVTSDT, IEFSD060 TSO request and pulled the rug out from many
going crazy until I realized this simple fact and IEESB605 is presented in the TSO users, then you initiated a CALLRTM
and verified it with IBM Level 2. When using MVS/ESA Version 3 logic manuals. Logic TYPE=ABTERM routine.
SYSTEM STRATEGIES
REGISTERS 0-7
GR: 00000000 000084B8 000084D4 00008904 00008918 000084D0 005D1230 005FF510
AR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
REGISTERS 8-15
GR: 005E8AF8 00008770 000084D8 8260F530 00008438 00008438 8260FAC6 80000000
AR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
RECOVERY ENVIRONMENT
RECOVERY ROUTINE TYPE: ESTAE RECOVERY ROUTINE
RECOVERY ROUTINE ENTRY POINT: 82611F58
AN IRB PRECEDED THE RB THAT IS ASSOCIATED WITH THIS TASK.
AN SVC DUMP WAS SCHEDULED BY A PREVIOUS RECOVERY ROUTINE.
SYSTEM STRATEGIES
TCB: 005F99F0
CMP . . . . . . . 940C1000 PKF. . . . . . . . 80 LMP. . . . . . . . FF
NDSP. . . . . . . 00000000 JSCB. . . . . . . 005EE6A4 BITS. . . . . . . . 00000000
SVRB: 005FD330
WLIC. . . . . . . 0002008A FLCDE. . . . . . 00000000 OPSW . . . . . . 070C1000 8217C906
PRB: 005911F8
WLIC. . . . . . . 00020033 FLCDE. . . . . . 14000000 OPSW . . . . . . 070C1000 82613B42
SVRB: 005FD4A0
WLIC. . . . . . . 0002000C FLCDE. . . . . . 00000000 OPSW . . . . . . 070C1000 8222FD50
IRB: 005D1230
WLIC . . . . . . . 00020001 EPA . . . . . . . . 829916C1 OPSW . . . . . . 078C0000 80000004
PRB: 005FF510
WLIC . . . . . . . 00020001 FLCDE. . . . . . 00C29D40 OPSW . . . . . . 078C1000 82608954
CDE: 00C29D40
NAME. . . . . . . IKJEFT02 ENTPT. . . . . . 82608000
0260F430 . . . . . . &. . . . . . . .|
0260F440 | B4&. . . . .C.&. . . . .C%&. . . . . . . &. . . . }. . . \. . .0\.. . 0H. . . . B>.0B. .-4. . . . . |
0260F480 | IRXTVARS. . .0B@. . . . . . . . . 0.H. . . . . . . . . 0. . . . . . BQ. .B0.0. . . . . . . . PATC |
0260F4C0 | H AREA - IKJEFTP8 91.077B. B . B . B . B{BBBDBFBHB.B.B.B}BKBMBOBQB.B.B. |
0260F500 | B\BSBUBWBY. . . . . . . . }. . . . . . . .QSYSIKJUA. . . . . . . . . . . 00. . IKJEFT03-TM |
0260F540 | P EXIT ATTENTION ROUTINE 910 |
IRB: 005D1230
WLIC.. . . . 00020001 EPA. . . . . . 829916C1 OPSW. . . . . 078C0000 80000004