CA AutoSys Workload Automation r11 Troubleshooting

Presenters: Dan Shannon, Elizabeth Dexter Shannon

Terms of This Presentation

This presentation was based on current information and resource allocations as of October 2009 and is subject to change or withdrawal by CA at any time without notice. Notwithstanding anything in this presentation to the contrary, this presentation shall not serve g y g p y, p to (i) affect the rights and/or obligations of CA or its licensees under any existing or future written license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. The development, development release and timing of any features or functionality described in this presentation remain at CAs sole discretion. Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA will make such release available (i) for sale to new licensees of such product; and (ii) to existing licensees of such product on a when and if-available basis as part of CA maintenance and support, and in the form of a regularly scheduled major product release. Such releases may be made available to current licensees of such product who are current subscribers to CA maintenance and support on a when and if-available basis. In the event of a conflict between pp the terms of this paragraph and any other information contained in this presentation, the terms of this paragraph shall govern.
2 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA

For Informational Purposes Only

Certain information in this presentation may outline CAs general product direction. All information in this presentation is for your informational purposes only and may not be incorporated into any contract. CA assumes no responsibility for the accuracy or completeness p y p y y p of the information. To the extent permitted by applicable law, CA provides this document as is without warranty of any kind, including without limitation, any implied warranties or merchantability, fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, damage indirect document including without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised of the possibility of such damages.

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Agenda:
>Preparing for the Installation g >Installation and Initial Configuration >Customisation and Tuning >Common Issues

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Preparing for Installation

> Architecture
What are you going to install, and where? What are you planning to support?

> System
Machine sizing M hi i i

> OS requirements
Patches, libraries

> Networking
Can the machines communicate?

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Installation
> Verify that the common component installs were successful > Verify that you can communicate between machines

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Architecture

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

WCC Ports
Services Launcher Server: Related Applications WCC Main WCC Main Login Web Services Web Services Samples Application Credential Application Start 8080 Stop 8005 SSL 8443

Configuration S C fi ti Server: Event Server: Job Status Console Servant:

Configuration C fi ti Event Console Quick Edit

10132 10138 10143

10133 10139 10144

10134 10140 10145

Job Status Console Server:

Job Status Console Job Status Views

10122

Monitoring Server:

Job Flow Design Job Flow Monitoring Critical Path Monitoring

10135

10136

10137

Scheduling Server: h d l

Job Editor Job Editor Plus (BLOB editor) Application Editor

10129

10130

10131

UI Framework Server:

Quick View Quick Start Enterprise Command Line High Availability Reporting Collectors

10146

10147

10148

HA Server:

10149

10150

10151

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Installation and Initial Configuration

> Installing the Common Components
SSA EEM CCS/Event Management

> C Connectivity ti it
EEM y / / AutoSys/WCC/SSA

> Multicast (WCC)

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Common Issues
> Collecting Information for Support
Release, maintenance level, logs, system information

> SSA Issues > EEM Issues > AutoSys Issues > WCC Issues > Multiple NIC cards > Command Sponsor is not installed

10

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Logging and Debugging

> CA Workload Automation AutoSys Logging gg g > EEM Logging > WCC Logging > SSA Logging

11

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

CA Workload Automation AutoSys Logging

> ISDBGACTIV g > Log Files
univagent.out as_server.INS event_demon.INS t d INS auto_rem.joid.runnum

12

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

EEM Logging (SDK)

> New With SDK 8.4 SR01 > EIAMCONFIG Environment > Configuration Files
eiam.config logger.config

13

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

WCC Logging
> WCC Logging > Location: <WCC Install Root> Logs folder
Uses log4j
Access (not currently used) Application
Admin, config, event, haserver, jobcontrolservant, jobcontrolserver, launcher, monitoring, reporting, uiframework Log4j.properties Log4j properties

Install Services

> T Tomcat L t Logs

<WCC Install Root>/xxx/logs
14 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA

Determining the Version

> AutoSys
autoflags

> WCC
uejmver.bat (Windows), uejmver.sh (*NIX)

15

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Logging
> Environment Variables
CSAM_SSL_DEBUG CSAM_DEBUG CSAM_LOG

16

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Directory Layout
/o p t/C A /U n ic e n te r A u to S y s J M $ A U T O U S E R a r c h iv e o u t $ A U T O S Y S o u t b in lib
a g e n t

tx
IN G

jo b s t d b o b j s n m p
O R A

re q u e s t
S Y B

re s p o n s e d o c m a n 8 d a ta b in jil n ite ly
b in _ b a k ... d e m o
O u tT h e D o o r

in s ta ll

m e s s a g e s C o n f ig u r a tio n * * * * * e c o a u a u a u a u tc . n fig . $ to s y s to s y s to s y s to s y s A .s .c .k .b F ile s :

U T O S E R V h .h o s t s h .h o s t s h .h o s t a s h .h o s t

te s t

lib _ b a k ..

17

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Related Files

18

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Configuration Global

19

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Configuration Port (Cont)

20

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Application Logic

21

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

SSA Configuration Port

22

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

AutoSys Communication Problems

> CAUAJM_E_10029 Communication attempt with the Unicenter AutoSys JM Application Server has failed! [host:port].
Can you ping the App Server host? Is the App Server running? Are AutoServer/AutoServerPort values correct? Is port 7163 (SSA) blocked by firewall? Are SSA port configurations the same on client & App Server machines the same?
EnablePmux E bl SSL EnableSSL

23

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

AutoSys Communication Problems

> CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Server [host:port]
Is the App Server running? Are there messages in the App Server log file?
ISDBGACTIV=LIGHT,LOG

Is the SSA PortRange for 49152-50176 configured correctly?

EnablePmux EnableSSL

Do the App Server and RDBMS machines have adequate resources? Does increasing DB_CONNECTIONS reduce problem?
24 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA

Troubleshooting the Scheduler

Job stuck in STARTING > Did Scheduler complete the Agent handshake?
CAUAJM_I_10082 [POLAL02-GX280 connected for job1 626.225.1]

> What does corresponding Job Agent log file show?

CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Server Server. CAUAJM_E_10221 Exhausted list of application servers. Failing request.

25

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Troubleshooting the Agent

CAUAJM_E_50026 ERROR: AutoPing WAS NOT SUCCESSFUL! > Does univagent log file show activity under ISDBGACTIV=LIGHT,JOB traces? > Do OS network tools reach Agent machine? (ping, nslookup, tracert/traceroute) > Is firewall blocking SSA physical port 7163? > Do SSA settings for AutoServerPort match (autoping D)? > Do SSA settings for PortRange=49152-50176 match?
EnablePmux EnableSSL
26 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA

EEM Troubleshooting
> Performance Issues
Check logging levels Active Directory connectivity Linux kernel Cache settings

> Unexpected permissions errors

Check for mixed case directory entries y

> Active Directory Group Retrieval

Filters

27

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

EEM Troubleshooting
CAUAJM_E_10448 FATAL SECURITY ERROR: Tampering or corruption of the security-related database keys detected > Did someone install a new instance using an MDB from a previous instance installation? > Did someone directly modify the ujo_alamode or ujo_keymaster keys? j _ y y > What is the state of the database tables? Did some unknown corruption of keys take place?

28

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

EEM Troubleshooting
CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file > Is the EEM backend running? > Do OS network tools reach EEM backend machine? (ping (ping, nslookup, tracert/traceroute) > Is firewall blocking EEM physical port 5250? > Was the proper EEM backend host name entered? > Are you able to connect to the EEM backend via the as_safetool? as safetool? > Does the AUTOUSER folder contain a valid certificate file?
29 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA

EEM Troubleshooting
Policy problems > Does the EEM web UI Permission Check work as expected? > Does autosec test work as expected? autosec_test > Does as_safetool work?

30

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

WCC - Other
> Job Flow Monitoring
Monitor ID Effective limit on the number of jobs in a flow

> Job Status Console

Global S Gl b l Session/Global ID i /Gl b l Prior to WCC r11.1 SP1, check alarms/alerts filters

> Unexpected login prompts

IAM Security setting in server definition

31

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Command Sponsor Troubleshooting

> Required for Quick View and ECLI in WCC q > Requires a credential user that is authorised on the OS to execute the command > Restricted to the commands in the AutoSysCommandISponsorFilters.txt AutoSysCommandISponsorFilters txt file > Uses iGateway

32

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA

Questions?

33

October 2009

[CA AutoSys Workload Automation r11 Troubleshooting]

Copyright 2009 CA