NB Frogs to the Rescue

Using WAPL to solve life’s little problems

Copyright © 2018 HCL Technologies Limited | www.hcltech.com

So what is WAPL

 Made from many ingredients

Data commands (PIF) Batch Loader

LIST, SELECT, DELETE ADSTART, CLSTART,

MODIFY, INSERT etc ETTSTART, WSSTART etc

Current Plan Commands Function Commands

Batch Jobs
STC workstations
HOLD, RELEASE, NOP, ADD, LISTJOB, LISTSTAT,
UNNOP, ALTER, BIND etc GETDATES etc TSO/REXX
Load Modules
TSO commands Occurrence Commands
REST API
OPSTAT, SRSTAT, RUNIF & ALTIF as start-up Automation tools
WSSTAT, OPINFO etc actions in user fields

Communication Programming Language

Variables, Functions,
SENDMAIL, SENDMSG, IF/THEN/ELSE, REXX,
CONSOLE, DISPLAY etc DO/UNTIL/WHILE etc

2 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

Frogs to the rescue – REST API jobs

Post Processing

3 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

REST API jobs
▪ ZWS supports many job types, one of which is a REST API job

▪ What exactly is a RESTful API?

• An Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data
• Effectively a Web Service, designed around a set of standards
• Removes a lot of compatibility issues found with traditional Program to Program interfaces

▪ What is a REST API job type

• Quite simply a job that allows you to launch and track a REST API job
• If the job issues an RC>=300 it will be considered a failure

▪ However, there is a catch –

• The REST API was never designed to run in batch
• It doesn’t follow all the rules
4 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
The rules of being a batch job
▪ Batch jobs are fundamentally simple beasts, with simple rules
• Once launched they will start, execute and end without any user interaction
• They will be considered to have worked unless they issue some non-zero return code
• Even some non-zero return codes can be made to be considered OK

▪ The key thing to consider is that a batch job must do all of its own checking and set return codes
• There are no processes reading output files looking for problems (though it can be done)
• Only return codes will make a job fail in ordinary circumstances

If you want your job to fail, you MUST set a return code

5 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

REST API return codes
1xx Informational

100 Continue 301 Moved permanently 408 Request timeout 428 Precondition required
101 Switching protocols 302 Found 409 Conflict 429 Too many requests
102 Processing 303 See other 410 Gone 431 Request header fields too big
103 Early hints 304 Not modified 411 Length required 451 Unavailable for legal reasons
2xx Success 305 Use proxy 412 Precondition failed 5xx Server error
200 OK 306 Switch proxy 413 Payload too large 500 Internal server error
201 Created 307 Temporary redirect 414 URI Too long 501 Not implemented
202 Accepted 308 Permanent redirect 415 Unsupported media type 502 Bad Gateway
203 Non-authoritative information 4xx Client error 416 Range not satisfiable 503 Service unavailable
204 No content 400 Bad request 417 Expectation failed 504 Gateway timeout
205 Reset content 401 Unauthorized 418 I’m a teapot 505 HTTP version not supported
206 Partial content 402 Payment required 421 Misdirected request 506 Variant also negotiates
207 Multi-status 403 Forbidden 422 Unprocessable entry 507 Insufficient storage
208 Already reported 404 Not found 423 Locked 508 Loop detected
206 IM used 405 Method not allowed 424 Failed dependency 510 Not extended

3xx Redirect 406 Not acceptable 425 Too Early 511 Network authentication required

300 Multiple choices 407 Proxy authority required 426 Upgrade required

6 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

Lack of options
▪ There are no return codes or ranges reserved for exceptions
• 100-299 means I worked (ZWS will not flag these as an error)
• 300-499 means I couldn’t really start
• 500-599 means I failed to finish in a controlled manner
• No space for completing in a controlled manner, but having found exceptions during the process
▪ Why shouldn’t you just make one up
• Future standards may clash with your choice
• System monitoring processes may not be able to distinguish batch errors from system errors

▪ Maybe a teapot may help

• 418 will never be issued for real
• You could perhaps make it an exception for system monitoring

7 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

How does the REST API expect to handle these errors
▪ The API assumes there is a calling program that will know to look in the response file for exceptions
• There are no standards to define how these exception should be coded in the response file
• So no scheduling product can be designed to automatically detect exceptions

▪ Enter the world of post processing

• IWS has return code mapping to spot patterns in the output and set return codes
• ZWS has Job Completion Checker to spot patterns in the output and set return codes for z/OS jobs
• However z-centric jobs cannot use either technique

▪ WAPL to the rescue

• WAPL can fetch job logs
• WAPL can search through them
• WAPL can set return codes

8 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

How to check a joblog with WAPL Put this job in an object variable

VARSUB SCAN
Find predecessor
SELECT CPOP ADID(!OADID) IA(!OYMD1.!OHHMM.) OPNO(!OOPNO.) OBJECT(THISJB)
VARSET PREDMAX @V(@THISJB-#CPPRE) Get number of predecessors
IF "!PREDMAX" <> 1 THEN DO
DISPLAY "## UNABLE TO IDENTIFY A SINGLE PREDECESSOR"
Assumptions
DISPLAY "## PREDCESSOR COUNT = !PREDMAX."
• WAPL job only has 1 predecessor
EXIT 20 • It is the job to check
END • It is in the same application
VARSET PREDOP @V(@THISJB-CPPRE-1-CPPRENO) Get 1st predecessor

OPTIONS CONTENTION(5,12) Set wait and retry Get job log

SELECT JL ADID(!OADID) IA(!OYMD1.!OHHMM.) OPNO(!PREDOP.) OBJECT(PREDJB) Put joblong into object

DO X = 1 TO !@PREDJB-#JLT Loop through all lines Look for errors

VARSET CHKLINE @V(@PREDJB-JLT-!X.-JLTTEXT) Get single line of joblog
IF POS("ERROR","!CHKLINE.") > 0 THEN DO
Look to see if ERROR is in that line
DISPLAY "## ERROR FOUND ON LINE !X"
DISPLAY "## !CHKLINE"
EXIT 20
END You could even Also works for z/OS jobs as
long as Data Store is active
END send an email

9 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Frogs to the rescue – Keeping your frogs quiet

Cleaning the Current Plan

10 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Sometimes you don’t want your scheduler to schedule

 In production you always want to scheduler

▪ Not much point buying IZWS if you didn’t

 Business Contingency Planning (Disaster recovery) is a bit different

▪ For testing you may want to just run a few select occurrences
▪ A noisy current plan is not a good place to start

 For testing you want to schedule on demand

▪ Keeping a full production plan running would be overkill

 Sometimes you want things to be a little quieter

11 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Cleaning out the crap

 A full restore of production inevitably comes with a full current plan

▪ Letting it run could make it difficult to perform meaningful tests

 WAPL can clear out the plan for you in a could of steps
//RUNWAPL EXEC EQQYXJPX
//OUTDEL DD DISP=SHR,DSN=MY.DELETE.STMTS
//SYSIN DD * DELETE CPOC ADID('Z2D#PROD') IA('1910280643')
OPTIONS DELETE(D) DEFERRED

LIST CPOC ADID(Z2D*) STATUS-NE(C)

 The output can then be run into WAPL to delete the occurrences

 With a large current plan this can take a while to run

12 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Knocking them out – Keeping them awake

 What if you could stop stuff being planned in the first place
▪ RESET the plan
▪ Set all your applications PENDING
▪ Create you plans, and they’re empty – Yay!
▪ BUT you can’t add PENDING applications to the plan

 WAPL can however put all run cycles out of effect

▪ When the plan is reset and rebuilt they won’t appear
Sets 720101 out-of-effect
▪ But they can be manually added Processes stack
ACTIVATE to reverse when done
//RUNWAPL EXEC EQQYXJPX
//SYSIN DD *
OPTIONS RUNSTAT(SUSPEND) DBMODE(UPDATE) POSTPROC(Y)
SHOWDFLT(N) STRIP(Y) Writes output
LOADDEF ADCOM DATA(-) LOADER(*) to the stack With a large database
LOADDEF ADRUN DATA(-) LOADER(*) this can take a while
LIST ADCOM ADID(*) SELECT(Y) TYPE(*)

13 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Engineer your batch for silent running

 Create a new period e.g. ALLSTOP

▪ Add negative run cycles to all your applications
▪ This is a big, but one-off job

 Set ALLSTOP to 720101 – 720101 in production

 When doing BCP testing

▪ Set ALLSTOP to 720101 – 711231
▪ Reset and regenerate the plans
▪ Empty CP
▪ No long job to run

 The same techniques can be applied by business

▪ To allow individual streams to be turned on and off

14 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Frogs to the rescue – Huge Current Plan

Dynamic Intervals?
This is a
Work in Progress

15 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Back in the good old days

 Back in the old days, Batch was Batch

▪ The term comes from collecting all data for one process into large batches
▪ Feeding that data into a single long running job
▪ Running many such processes for different types of data over night

 Over the years the nature of batch has changed

▪ Data quantities get smaller
▪ Frequencies of batch jobs increased to run multiple times a day
▪ Batch has become almost transactional

 Data used to be “punched” once a day by the data prep department

▪ Now data can arrive from anywhere But efficiency is still
very important
▪ At any time of the day
Data currency is
▪ Many times a day now more important
than batch efficiencies
16 ▪ And your business can’t wait Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 Enter the world of Interval Scheduling

 Business Goal: To obtain and process data with maximum currency

▪ This means you have to look for data at regular intervals
▪ If you are lucky this will be hourly (24 repeats a day)
▪ Now every 5 minutes is not uncommon (288 repeats a day)
▪ In extreme cases I’ve seen every 2 minutes (720 repeats a day)

 This blurs the line between batch and online

▪ Some suggest this should be system driven, not scheduled
▪ But can a scheduler cope?

17 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Fixed interval scheduling
------------------------------ MODIFYING A RULE -------------------------------
Command ===>

Interval Enter the GENDAYS command to display the dates generated by this rule
settings Enter the E command to specify EVERY options
Enter S and user data in the fields below to define a rule

Application : DH#INTERVAL Demo interval processing

Rule : INTERVAL

--- Frequency --- --- Day --- --- Cycle Specification ---
Rule once -------------------------------------------------------------------------------
a day _ Only | S Day | S Week _ January _ July
S Every | _ Free day | _ Month _ February _ August
| _ Work day | _ Year _ March _ September
_ First _ Last | _ Monday | _ April _ October
_ Second _ 2nd Last | _ Tuesday | _ May _ November
_ Third _ 3rd Last | _ Wednesday | _ June _ December
_ Fourth _ 4th Last | _ Thursday | Week number __ __ __ __ __ __
_ Fifth _ 5th Last | _ Friday | Period/RG ________ ________
___ ___ ___ ___ | _ Saturday | name ________ ________
___ ___ ___ ___ | _ Sunday |
___ ___ ___ ___ | | Shift default origin by ___ days
-------------------------------------------------------------------------------

18 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Fixed interval scheduling
-------------------------------- EVERY OPTIONS --------------------------------
Command ===>

REPEAT EVERY ===> 00.05 Repeat every HH.MM

Runcycle
IA
FROM ===> 00.00 Input Arrival time

UNTIL ===> 23.55 Repeat end time in the HH.MM format

Number of occurrences can be reduced by manually

Only one application to set up creating individual time dependent operations within
a few applications.

288 occurrences Whilst this can drastically decrease the number of

On plan all day occurrences setup time is much longer, more complex
Backlogging can occur and still suffers from backlogging.
19 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 What is Backlogging?

00:00 00:05 00:10 00:15 00:20 00:25 00:30 00:35 00:40 00:45 00:50 00:55 01:00
----+----+----+----+----+----+----+----+----+----+----+----+----
00.00 X
00:15 run
00.05 X overruns
00.10 X
00:20 – 00:40 run
00.15 XXXXXXXXXXXXXXXXXXXXX in quick succession
00.20 X
00.25 X
00.30 X
00.35 X
00.40 X
00.45 X
00.50 X
00.55 X
20 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 Dynamic interval scheduling (WAPL)
--------------------------------- OPERATIONS ----------------- Row 1 to 3 of 3
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete
S - Select operation details, J - Edit JCL
Enter the TEXT command above to include operation text, or,
enter the GRAPH command to view the list graphically.

Application : DH#WAPLRESUBMIT Demo WAPL timed restart

Row Oper Duration Job name Internal predecessors Morepreds No.of

cmd ws no. HH.MM.SS -IntExt- Conds
'''' NONR 001 00.00.01 ZFIRST__ ___ ___ ___ ___ ___ ___ ___ 0 0 0
'''' CPU1 005 00.00.01 JOB001 _ 001 ___ ___ ___ ___ ___ ___ 0 0 0
'''' CMD1 255 00.00.01 TWSXCMD1 005 ___ ___ ___ ___ ___ ___ 0 0 0
******************************* Bottom of data ********************************

• Create a single application to run for the first instance (just like in the Fixed method)
• Add a WAPL job/task as the last application
• This will add a new occurrence when this one finishes
• If delays have occurred any backlog will be skipped – new occurrence will wait for next interval
21 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 Simple WAPL resubmit command
---------------------------- OPERATION USER FIELDS ----------- Row 1 to 1 of 1
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete

Application : DH#WAPLRESUBMIT Demo WAPL timed restart

Operation : CMD1 255
Jobname : TWSXCMD1

Row User Field Name User Field Value

cmd ----+----1----+----2----+----3----+----4----+----5----
'''' EQQ-SYSIN-01 ADD FROM(0000) UNTIL(2355) EVERY(005)
******************************* Bottom of data ********************************

• Whilst this doesn’t fill the plan with 288 occurrences all day, by the end of the day there may still be that many
• Though less might be in the plan if backlog gets skipped
• If you only expect 1 file per day you can make the WAPL step only run when a file isn’t found
• Then it could run considerably less occurrences per day

22 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Dynamic interval restart (Planned new WAPL feature)
---------------------------- OPERATION USER FIELDS ----------- Row 1 to 1 of 1
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete Tells WAPL to RESTART
Application : DH#WAPLRESUBMIT Demo WAPL timed restart
at operation 001 instead
Operation : CMD1 255 of submitting new one
Jobname : TWSXCMD1

Row User Field Name User Field Value

cmd ----+----1----+----2----+----3----+----4----+----5----
'''' EQQ-SYSIN-01 ADD FROM(0000) UNTIL(2355) EVERY(005) RESTART(001)
******************************* Bottom of data ********************************

• A new WAPL feature will allow the current occurrence to be restarted for the next interval
• This means only 1 application and only 1 occurrence per day
• You schedule it once, for the 1st instance and it stops restarting itself once it reaches the until time
• This method also skips backlogs when delays occur
• Because of the way it restarts, traditional conditional dependencies cannot be used within it

23 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Injecting extra jobs

 The methods discussed so far only repeats a single job File

Monitor
 Getting files from other platforms often need a combination of multiple jobs
▪ A File Monitor job to look for a file arriving No
Found?
▪ An FTP job to transfer the file
Yes
▪ A processing job to do something with the transferred file

 Whilst conditional dependencies can’t be used WAPL can conditionally restart

FTP
▪ WAPL can restart if a file isn’t found, avoiding the FTP job and file process
Job
▪ Then if it gets as far as the FTP job and file process a second WAPL step can restart to look for
the next file
File
 If only one file is expected per day the final WAPL step can be omitted Process
▪ This will keep looking using the intervals until a file is found, then end after processing

24 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Conditional restart
--------------------------------- OPERATIONS ----------------- Row 1 to 3 of 3
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete
S - Select operation details, J - Edit JCL
Enter the TEXT command above to include operation text, or,
Processing enter the GRAPH command to view the list graphically.
only
gets this far if Application
a : DH#WAPLRESTART Demo WAPL timed restart
file is found
Row Oper Duration Job name Internal predecessors Morepreds No.of
cmd ws no. HH.MM.SS -IntExt- Conds
'''' NONR 001 00.00.01 ZFIRST__ ___ ___ ___ ___ ___ ___ ___ 0 0 0
'''' LNX1 005 00.00.01 FILEMON_ 001 ___ ___ ___ ___ ___ ___ 0 0 0
'''' CMD1 010 00.00.01 TWSXCMD1 005 ___ ___ ___ ___ ___ ___ 0 0 0
'''' LNX1 015 00.00.01 FTPJOB__ 010 ___ ___ ___ ___ ___ ___ 0 0 0
'''' CPU1 020 00.00.01 FILEPROC 015 ___ ___ ___ ___ ___ ___ 0 0 0
'''' CMD1 255 00.00.01 TWSXCMD1 020 ___ ___ ___ ___ ___ ___ 0 0 0
******************************* Bottom of data ********************************

If only one file is expected per day

this second WAPL step can be omitted
25 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 First WAPL job
---------------------------- OPERATION USER FIELDS ----------- Row 1 to 1 of 1
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete
WAPL gets the details
Application : DH#WAPLRESTART Demo WAPL timed restart of the previous job
Operation : CMD1 255 and stores it in an
Jobname : TWSXCMD1 object - CHECKOP
Row User Field Name User Field Value
cmd ----+----1----+----2----+----3----+----4----+----5----
'''' EQQ-SYSIN-01 VARSUB SCAN RC0004 indicates no
'''' EQQ-SYSIN-02 SELECT CPOPCOM ADID(!OADID) IA(!OYMD1!OHHMM)
'''' EQQ-SYSIN-03 OPNO(005) OBJECT(CHECKOP) file found, so restarts
'''' EQQ-SYSIN-04 IF '!@CHECKOP-CPOPORIGRC' = '0004' THEN DO before getting to FTP
'''' EQQ-SYSIN-05 ADD FROM(0000) UNTIL(2355) EVERY(005) and File Processing
'''' EQQ-SYSIN-06 RESTART(001,FLUSH)
'''' EQQ-SYSIN-07 END
******************************* Bottom of data ********************************

FLUSH tells WAPL to NOP

remaining jobs if UNTIL limit has been
reached and no restart will happen
26 Copyright © 2018 HCL Technologies Limited | www.hcltech.com
 Second WAPL job
---------------------------- OPERATION USER FIELDS ----------- Row 1 to 1 of 1
Command ===> Scroll ===> CSR

Enter/Change data in the rows, and/or enter any of the following

row commands:
I(nn) - Insert, R(nn),RR(nn) - Repeat, D(nn),DD - Delete

Application : DH#WAPLRESTART Demo WAPL timed restart

Operation : CMD1 010
Jobname : TWSXCMD1

Row User Field Name User Field Value

cmd ----+----1----+----2----+----3----+----4----+----5----
'''' EQQ-SYSIN-01 ADD FROM(0000) UNTIL(2350) EVERY(005) RESTART(001)
******************************* Bottom of data ********************************

No need for FLUSH as

this is the last operation

27 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Other possibilities to experiment with

 Business Scenario: To process a file within 5 minutes of it arriving, without restarting every 5 minutes

 Theoretically to reduce number of jobs running

▪ Use WAPL dynamic restart FROM(0000) UNTIL(2355) EVERY(005) RESTART(001)
▪ Use File Monitor with a large value for –timeout (e.g. –timeout 3600)
▪ Use JCL directives to drive a shorter –timeout towards the end of the day, to avoid overrunning TO time

 This way the occurrence will spend most of its time waiting for files
▪ When a timeout occurs, it will restart at the next interval (within 5 minutes)
▪ When a file is found, it will process the file immediately and restart at the next interval (within 5 minutes)
▪ But the large timeout will reduce the likelihood of running 288 times a day

28 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Performance

 Whilst the process works well running this method via WAPL incurs an overhead
▪ Not much for a few instances
▪ The large amount needed by the customer may not scale well

 Where next?
▪ WAPL is a generic tool so does many things to set up its environment
▪ There may be scope to improve performance by making some start-up processes “just in time”
▪ It may also be possible to emulate what we are doing with WAPL in the product directly

Watch this
space

29 Copyright © 2018 HCL Technologies Limited | www.hcltech.com

 Any questions?

Please submit your session feedback!

 Do it online at http://conferences.gse.org.uk/2019/feedback/nb

Session NB

https://www.ibm.com/developerworks/community/groups/community/zGlue

30 Copyright © 2018 HCL Technologies Limited | www.hcltech.com