Troubleshooting for Performance Management Version 8.

1
(On Premises)


Troubleshooting for Performance Management Version 8.1
(On Premises)


ii Troubleshooting for Performance Management Version 8.1 (On Premises):
Contents
Chapter 1. Troubleshooting and support 1 Multiple agents: Installation failed on AIX . . . . 16
Collecting Performance Management infrastructure Multiple agents: Agent installation failed for SaaS
logs for IBM Support . . . . . . . . . . . 1 agents with an existing IBM Tivoli Monitoring
Collecting monitoring agent logs for IBM Support . . 4 V6.2.3 framework . . . . . . . . . . . . 16
Performance Management server installation failure
Chapter 2. Installation . . . . . . . . . 7 and the RPM database. . . . . . . . . . . 17
Agent installation fails . . . . . . . . . . . 7
Uninstallation of monitoring agents fail . . . . . 7 Chapter 3. Integration . . . . . . . . 19
Agent Subscription Facility (ASF) activity log Hybrid Gateway: Unable to connect to the Tivoli
customization parameter . . . . . . . . . . 7 Enterprise Portal Server . . . . . . . . . . 19
DataPower agent: Configuration failed with SSL Hybrid Gateway: Tivoli Monitoring domain
security error . . . . . . . . . . . . . . 8 managed systems are not visible in the Application
DB2 agent: Configuration failed with permission Performance Dashboard . . . . . . . . . . 19
error for non-root user after installation with root
user . . . . . . . . . . . . . . . . . 9 Chapter 4. Agent . . . . . . . . . . 21
DB2 agent: Agent uninstallation command reports Active Directory agent: Dashboard errors . . . . 21
failed uninstallation . . . . . . . . . . . . 9 LDAP Successful Bind attribute appears with
HMC Base agent user: Configuration . . . . . . 9 value 0 on the dashboard. . . . . . . . . 21
Linux KVM agent: Installation errors on UNIX Dashboard does not display the updated list of
systems. . . . . . . . . . . . . . . . 10 instances . . . . . . . . . . . . . . 21
Microsoft IIS agent: When Windows OS agent is DataPower agent: AIX Installation . . . . . . . 21
uninstalled, the Microsoft IIS agent is also DB2 agent: Memory leak . . . . . . . . . . 22
uninstalled . . . . . . . . . . . . . . 11 DB2 agent: No restart after unexpected termination 22
Microsoft SQL Server agent: Installation failure with Linux KVM agent: Attributes not available on RHEL
KGL and KGS components on 32-bit Windows 2003 5 and 6 systems . . . . . . . . . . . . . 22
system . . . . . . . . . . . . . . . . 11 Linux KVM agent: Errors with negative values . . 24
Microsoft SQL Server agent: Historical Data Linux KVM agent: oVirt API error in the RHEVM
Collection collects data after a slight delay . . . . 12 environment . . . . . . . . . . . . . . 24
Microsoft SQL Server agent: Database Server Microsoft Exchange Server agent: Exchange Server
Properties window does not display the upgraded 2013 component shows Exchange 2007 and 2010
version of the SQL Server . . . . . . . . . 12 instances . . . . . . . . . . . . . . . 24
Microsoft SQL Server agent: In Windows 2003 Microsoft Exchange Server agent: Dashboard defects 25
system, a non-administrator user cannot configure Microsoft Exchange Server agent: Low mailbox
the SQL Server agent services . . . . . . . . 12 space . . . . . . . . . . . . . . . . 25
Microsoft SQL Server agent: DB Microsoft Hyper-V Server agent: Agent log files are
INCLUDE/EXCLUDE configuration panel supports not in the CANDLEHOME/tmaitm6/logs directory . 25
only 256 characters . . . . . . . . . . . . 13 Microsoft Hyper-V Server agent: Memory leak is
Microsoft SQL Server agent: Historical Data observed . . . . . . . . . . . . . . . 26
Collection is not supported for the Customized SQL Microsoft Hyper-V Server agent: Process name gets
Query Result data set . . . . . . . . . . . 13 truncated on the dashboard . . . . . . . . . 26
Oracle Database agent: Cannot read remote alert log Microsoft Hyper-V Server agent: Rollup errors occur
file on Windows systems . . . . . . . . . . 13 in the agent log file. . . . . . . . . . . . 26
Oracle Database agent: Cannot load the Microsoft Hyper-V Server agent: VHD File Size (GB)
configuration file . . . . . . . . . . . . 13 displays 0 in the Virtual Machine Details group
Oracle Database agent: Uninstallation command widget . . . . . . . . . . . . . . . . 26
does not uninstall agents . . . . . . . . . . 14 Microsoft Hyper-V Server agent: WMI Error
SAP agent: Monitoring agent does not start in a 0x80041006 . . . . . . . . . . . . . . 26
non-ASCII environment . . . . . . . . . . 14 Microsoft Hyper-V Server agent No data is
SAP agent: Agent repeatedly restarts . . . . . . 14 displayed in the dashboard for the Linux virtual
SAP agent: Agent transport errors . . . . . . . 14 machine . . . . . . . . . . . . . . . 27
SAP agent: Monitoring agent cannot connect to the Microsoft IIS agent: Managed system seems to be
mySAP System . . . . . . . . . . . . . 15 offline . . . . . . . . . . . . . . . . 28
SAP agent: Cannot connect to the SAP system by Microsoft IIS agent: Monitoring data fails to be
using the Logon Group mode . . . . . . . . 15 displayed . . . . . . . . . . . . . . . 28
SAP agent: Cannot locate the KDCB0_HOSTNAME setting 16

iii
Microsoft IIS agent: Availability status shows Microsoft SQL Server agent: The configuration file
PROCESS_DATA_NOT_ AVAILABLE. . . . . . 28 of the MS SQL Problem Detail data set cannot
Microsoft IIS agent: Dashboard errors . . . . . 28 handle non-English ERRORLOG files . . . . . . 38
Dashboard shows a blank row for the newly Microsoft SQL Server agent: Attributes do not allow
added performance counters. . . . . . . . 29 non-ASCII input in the Situation Editor . . . . . 38
No data is displayed in the Error Statistics Multiple agents: High CPU usage . . . . . . . 39
(history) group widget and Request Rate Multiple agents: Log data accumulates too rapidly 39
(history) group widget. . . . . . . . . . 29 Multiple agents: Situation does not trigger . . . . 40
Error Statistics (history) group widget and Multiple agents: Situation is not displayed in the
Request Rate (history) group widget displays event tab . . . . . . . . . . . . . . . 40
incorrect or shuffled data for some attributes . . 29 Node.js agent: URL filtering, incorrect URL can be
Microsoft .NET agent: The CTIRA_LOG_PATH displayed . . . . . . . . . . . . . . . 41
variable does not work . . . . . . . . . . 29 Node.js agent: No data in deep-dive Request
Microsoft .NET agent: Dashboard errors . . . . . 30 Summary widget . . . . . . . . . . . . 41
Negative value for the Number of Current Node.js agent: Y-axis labels not displayed in
Physical Threads attribute . . . . . . . . 30 dashboard charts . . . . . . . . . . . . 41
Zero value for the NET CLR Networking data set 30 Node.js agent: Configuration file not generated after
Application instances are not displayed . . . . 30 configuring and starting the agent . . . . . . . 41
Data is not displayed for .NET resources of 32-bit Oracle Database agent: Some columns for monitored
applications . . . . . . . . . . . . . 31 Oracle RDBMS 10g instance display 0 . . . . . 42
Data is not displayed for Windows Workflow Oracle Database agent: Processes consume high
Foundation data set . . . . . . . . . . 31 CPU . . . . . . . . . . . . . . . . . 42
Data is not displayed for some data sets. . . . 31 Oracle Database agent: Memory usage for krzstart
Data is not displayed for the ASP.NET or krzclient processes increases . . . . . . . . 42
Applications Request Status (websockets) data set 32 Oracle Database agent: Instances not listed in My
Dashboards do not update or display data . . . 32 Components . . . . . . . . . . . . . . 43
Microsoft SQL Server agent: Dashboard errors . . . 34 PHP agent: injection codes not removed after
Negative value is returned for Transactions per uninstalling the agent . . . . . . . . . . . 43
Second in the Database Detail data set . . . . 34 Python agent: injection codes not removed after
Collection Status attribute returns a value of uninstalling the agent . . . . . . . . . . . 43
Inactive. . . . . . . . . . . . . . . 34 Ruby agent: Instance not discovered . . . . . . 44
Incorrect data is displayed for two SQL Server SAP agent: Data collection stops or runs sluggishly 44
databases with identical names . . . . . . . 34 SAP agent: ITM raised alert message in the SAP
Dashboard shows no data for the primary replica syslog . . . . . . . . . . . . . . . . 44
in the Availability Group details data set . . . 34 SAP agent: Incorrect parameters . . . . . . . 45
Dashboard displays incorrect value for some data SAP agent: Managed system names that show up
sets when a custom query is applied . . . . . 35 under SAP agent in the dashboard are incorrect . . 45
For the Stolen Pages Growth attribute negative SAP agent: Syslog messages and alert messages do
values are displayed on the dashboard . . . . 35 not show correctly in non-English languages . . . 45
For some attributes of the MS SQL Device Detail SAP agent: mySAP application server is not
and MS SQL Filegroup Details data sets incorrect discovered by the SAP agent . . . . . . . . 46
data is displayed on the dashboard . . . . . 35 SAP agent: mySAP server does not start when using
No data is collected for the Table Detail attribute 36 port 3661 . . . . . . . . . . . . . . . 46
For some attributes no data is displayed on the SAP agent: Non-ASCII characters that are entered
dashboard . . . . . . . . . . . . . . 36 into the configuration window do not show up or
Dashboard displays incorrect value for the Audit are not the correct characters . . . . . . . . 47
name attribute of the Audit Details data set . . 36 SAP agent: Attributes do not allow non-ASCII input
Microsoft SQL Server agent: Database errors . . . 36 when you are creating an eventing threshold . . . 47
Dashboard displays incorrect data for databases SAP agent: Return code 8 or return code 12 occurs
with identical names . . . . . . . . . . 36 on the main import step . . . . . . . . . . 47
Dashboard displays no data for databases with SAP agent: Agent does not start if RFC library is not
table level locking . . . . . . . . . . . 37 copied to the correct path. . . . . . . . . . 47
SQL Server agent takes long time to display data 37 SAP agent: TSV_TNEW_PAGE_ALLOC_FAILED ABAP
No data is collected for the Table Detail attribute 37 runtime error . . . . . . . . . . . . . . 48
Microsoft SQL Server agent: Eventing threshold SAP agent: Runtime errors in relation to transport
errors . . . . . . . . . . . . . . . . 37 on the SAP system . . . . . . . . . . . . 48
Monitoring interval is too long . . . . . . . 37 SAP agent: Same version installation warning
Situation does not activate at startup . . . . . 37 during command-line installation on UNIX . . . . 48
Microsoft SQL Server agent: Agent does not start . . 38 SAP agent: Tivoli Monitoring alert 9912 and SAP
Microsoft SQL Server agent: Handle leak occurs syslog message . . . . . . . . . . . . . 48
during the collector process . . . . . . . . . 38

iv Troubleshooting for Performance Management Version 8.1 (On Premises):

SAP agent: Transaction not valid error when you Chapter 6. Dashboard . . . . . . . . 55
run the /IBMMON/ITM_* transactions . . . . . 49 Application Dashboard takes a long time to display 55
SAP agent: Values shown are not correct when you Dashboard error occurred while loading data . . . 55
edit the configuration for an existing monitoring Dashboards show no data, partial data, or incorrect
agent . . . . . . . . . . . . . . . . 49 data from the monitoring agent. . . . . . . . 56
SAP agent: Monitoring agent does not start in an Dashboard shows no events or event status . . . 58
AIX V6.1 environment . . . . . . . . . . . 49 Historical data doesn't change after you adjust the
SAP agent: Language errors . . . . . . . . . 49 time selector . . . . . . . . . . . . . . 58
SAP agent: Value lists are shown in English in
the mySAP configuration panels . . . . . . 49 Chapter 7. Eventing thresholds . . . . 59
SAP agent configuration panels on a mySAP
Threshold events have unknown severity after agent
system does not show text in a multi-byte
recycle . . . . . . . . . . . . . . . . 59
language . . . . . . . . . . . . . . 50
Thresholds are not applied to the monitoring agent 59
Tomcat agent: Tomcat server not monitored . . . 50
VMware VI agent: Dashboard errors . . . . . . 50
ESX Server component events are not displayed 50 Chapter 8. Performance Management
Attribute groups for the ESX Server component Cognos Reports. . . . . . . . . . . 61
are not displayed . . . . . . . . . . . 51 Reports: Charts in PDF Reports in Firefox are Pink 61
Situations in the Severity column display Reports: Y-axis shows negative values . . . . . 61
incorrect status . . . . . . . . . . . . 51 Reports: Y-axis shows duplicate values . . . . . 61
Group widget titles are not globalized . . . . 51 Reports: The Close and Cancel buttons shown on
Columns in some group widgets are not Reports login screen do not work . . . . . . . 61
highlighted . . . . . . . . . . . . . 51 Reports: Application names are inconsistent in the
WebSphere Applications and Microsoft .NET agents: All My Applications report . . . . . . . . . 62
Missing transaction data in topology views . . . . 51 Reports: Application Performance Usage report
returns: XQE-GEN-0018 . . . . . . . . . . 62
Chapter 5. Component . . . . . . . . 53 Reports: Logging out in APM UI generates: Error
Getting Started links blocked in Chrome. . . . . 53 403: AuthenticationFailed . . . . . . . . . . 62
Network connection error in the Performance
Management console . . . . . . . . . . . 53
Connection to the Performance Management console
fails in Firefox . . . . . . . . . . . . . 53

Contents v
vi Troubleshooting for Performance Management Version 8.1 (On Premises):
Chapter 1. Troubleshooting and support
Review the troubleshooting information for problems you might experience with installing, configuring,
or using IBM® Performance Management.

Additional agent-specific troubleshooting information is available at the following links:

v For the Linux OS agent and Windows OS agent, see the IBM Tivoli® Monitoring V6.3 FP2
documentation on the wiki.
v For the Response Time Monitoring Agent, see the Response Time agent troubleshooting documentation.
v For the WebSphere® Applications agent, see the WebSphere Applications agent troubleshooting
documentation.

When working with IBM Support, gather required logs and other troubleshooting information by using
the pdcollect tool.

Collecting Performance Management infrastructure logs for IBM

Support
Gather the log files from the Performance Management infrastructure when requested by IBM Support to
help diagnose issues.

Before you begin

Root permission is required to collect system information from the Performance Management server.

About this task

To run the log collection command, complete the following steps:

Procedure
1. On the command line, change to the Performance Management server directory: /opt/ibm/ccm/
2. Run the following command:
./collectLogs.sh

A tar.gz file with a time stamp in the file name is generated in the /tmp directory, such as
/tmp/SCMAI_Logs_min_hostname_20150207_105132.tar.gz.
You might receive a message that the system cannot find the path specified. You can ignore this
message.
3. Send the output file to your IBM Support representative.

Results

The script gathers the following component logs:

Table 1. Log files that are collected by the collectLogs.sh script.
Component Log directory Log file name Description
Performance /opt/ibm/wlp/usr/ messages.log The main monitoring
Management server servers/min/logs infrastructure node log file.
console.log Startup, session, transaction, and
shutdown information.

1
Table 1. Log files that are collected by the collectLogs.sh script (continued).
Component Log directory Log file name Description
System Information /opt/ibm/wlp/usr/ machine.info Date and time of collection,
of Performance servers/min/logs/ hostname, and network interface
Management server sysinfo configuration.
system.info System hardware and
configuration information.
cpu.info, memory.info, CPU, memory, and disk
disk.info configuration information.
mount.info Information about mounted
volumes.
nsswitch.conf Copy of etc/nsswitch.conf.
hosts Copy of etc/hosts.
resolve.conf DNS information generated by
the Network Manager.
tasklist.info Task list at time of collection.
maint.info Software packages installed at
the time of collection.
OSLC Service /opt/ibm/ccm/oslc_pm/ candle_installation.log Include installation environment
Provider logs information that is collected at
each restart.
UpdateAutoRun.log Sets up automation environment.
hostname_as.inv Index of log files.
hostname_as_NNNNNN.log Trace file for OSLC Performance
Monitoring service.
Service Component /opt/ibm/ccm/SCR/ msgGTM_CI.log.N
Registry XMLToolKit/log
msgGTM_XT.log.N
msgGTM_XT.log.N.lck
traceGTM_CI.log.N
traceGTM_XT.log.N
traceGTM_XT.log.N.lck
/opt/ibm/ccm/SCR/ xmltoolkitsvc.properties XML Toolkit configuration
XMLToolkit/bin information.
/opt/ibm/ccm/SCR/ DB maintenance scripts and SQL
XMLToolkit/log/ information.
install/maintenance
MongoDB /opt/ibm/mongodb/logs mongodb.log The main MongoDB log.

2 Troubleshooting for Performance Management Version 8.1 (On Premises):

Table 1. Log files that are collected by the collectLogs.sh script (continued).
Component Log directory Log file name Description
Kafka Message /opt/ibm/kafka/log controller.log Log of “controller” subsystem in
Broker Kafka.
kafka-request.log Log of “request” subsystem in
Kafka.
server.log The main Kafka log.
state-change.log Log of “state-change” subsystem
in Kafka.
kafka-server.err Standard error stream of Kafka
process.
zookeeper-server.err Standard error stream of
Zookeeper process.
Monitoring /opt/ibm/gaian gaiandb.log Node startup and performance
Infrastructure Node information.
Derby database
derby.log Derby database startup
information.
*.properties Database configuration
information.
Performance Management console backend (server1)
Server1 Main /opt/ibm/wlp/usr/ server.xml, jvm.options, Server1 server JVM and
servers/server1 bootstrap.properties configuration information.
Server1 Logs /opt/ibm/wlp/usr/ messages.log Server1 standard messages.
servers/server1/logs
trace.log Server1 trace logs.
kd8collectdate_ Archive that contains Data
time.tar.gz Provider debug information.
console.log Server1 standard output
containing startup, session,
transaction, and shutdown
information.
w7_audit.log Role Based Access Control audit
log of changes to RBAC-related
artifacts, such as creating
permissions and roles, assigning
permissions to roles, and
assigning users and user groups
to roles.
scrderby.log Service Component Repository
Derby startup information.
scrserver.log SCR Server process logs.
scrserver-errors.log SCR Server process error
messages.
Server1 FFDC /opt/ibm/wlp/usr/ ffdc_date_time.N.log First failure data capture logs.
servers/server1/logs/
exception_summary_date Exception summaries from FFDC
ffdc
_time.N.log processing.
Performance Management console (APM UI)
APM UI Server /opt/ibm/wlp/usr/ server.xml, jvm.options, Performance Management
Main servers/apmui bootstrap.properties console (APM UI) server JVM
and configuration information.

Chapter 1. Troubleshooting and support 3

Table 1. Log files that are collected by the collectLogs.sh script (continued).
Component Log directory Log file name Description
APM UI Server Logs /opt/ibm/wlp/usr/ messages.log APM UI server standard error
servers/apmui/logs messages.
trace.log APM UI server trace log.
console.log APM UI server standard output
containing startup, session,
transaction, and shutdown
information.
APM_UI.log APM UI process log.
derby.log APM UI Derby log.
itp.log IT Portal log containing client
and server information.
APM UI FFDC /opt/ibm/wlp/usr/ ffdc_date_time.N.log First failure data capture logs.
servers/apmui/logs/
exception_summary_date Exception summaries from FFDC
ffdc
_time.N.log processing.
/opt/ibm/wlp/usr/ components.log List of installed components in
servers/apmui/apps APM UI.
Agent Interface Agent interface main: server.xml, jvm.options, Agent interface JVM and
(asfrest) /opt/ibm/wlp/usr/ bootstrap.properties configuration information.
servers/asfrest
Agent interface logs: messages.log, Agent interface standard
/opt/ibm/wlp/usr/ console.log,trace.log messages and traces.
servers/asfrest/logs
Agent interface FFDC: ffdc_date_time.N.log, Agent interface standard
/opt/ibm/wlp/usr/ exception_summary_date messages and traces.
servers/asfrest/logs/ _time.N.log
ffdc
Performance /opt/ibm/ccm/logs smai_oslc_pm-rpm Installation log files.
Management _inst_trace.log
infrastructure install IA_SCR_Install6.1.2-00.log
deploymentResults.log
apm-server-install
_20150217_110258.log
smai-apmui-configuration
_trace.log smai-oslc-
prestart.log configureMIN.log
itmcdp.log db2setup_wse.log
ksy agent /opt/ibm/sy/logs ksy agent log files
DB2 server /home/db2apm/sqllib/ DB2 server log files
db2dump

Collecting monitoring agent logs for IBM Support

Use the problem determination collection tool, pdcollect, to gather required logs and other problem
determination information that is requested by IBM Support for monitoring agents.

The PD collector tool is installed with each monitoring agent.

4 Troubleshooting for Performance Management Version 8.1 (On Premises):

Before you begin

Root or administrator permission is required for the PD collector tool to collect system information from
the monitoring agents. You can review the agent logs individually in the following folders:
v install_dir\TMAITM6_x64\logs
v install_dir/logs

Restriction: It is only possible to run one instance of the pdcollect script.

About this task

The default location of install_dir is:

v C:\IBM\APM
v /opt/ibm/apm/agent
v /opt/ibm/ccm/agent

To run the PD collector tool, complete the following steps:

Procedure
1. On the command line, change to the agent directory:
v install_dir/bin
v install_dir\BIN
2. Run the following command:
v ./pdcollect
v pdcollect
A file with a time stamp in the file name is generated in the tmp directory, such as
/tmp/pdcollect-nc049021.tar.Z.
3. Send the output files to your IBM Support representative.

Chapter 1. Troubleshooting and support 5

6 Troubleshooting for Performance Management Version 8.1 (On Premises):
Chapter 2. Installation
Review the installation troubleshooting symptoms and solutions for help with installation and upgrade
issues.

Agent installation fails

If the monitoring agent fails to install, you can run commands to clean up the environment, and start the
installation again.

Also, check the installation path. The agent installation fails if parentheses or spaces are included in the
installation path. For example, C:\my downloads\APMaaS_Agent_Install_1.1(1) must be amended to
C:\mydownloads\APMaaS_Agent_Install_1.1 for a successful installation.

If a monitoring agent installation or upgrade from a previous version fails, the installation might have
been terminated abnormally. For example, you might close the command window while the monitoring
agent is being upgraded. An error message is displayed, such as KCICF9004E: Installation failed. See
C:\IBM\APM\InstallITM\Abort*.log file for more details.
Agent installation or upgrade fails
If the installer reports that the agent installation or upgrade fails, use the following command to
uninstall the monitoring agent, and start the installation again:
name_agent.sh uninstall
name_agent.bat uninstall
where name is the type of monitoring agent, such as os or rt.

If the uninstall command fails, clean up the environment and run the smai-agent.sh
ininstall_all.sh or agent_cleanup.bat script and start the installation again.
Framework installation or upgrade fails
If the installer reports that the framework installation or upgrade fails, clean up the home
directory and use the following command to uninstall all monitoring agents, and start the
installation again:
smai-agent.sh uninstall_all
smai-agent.bat uninstall_all

Uninstallation of monitoring agents fail

If the monitoring agents fail to uninstall, you can check whether or not the rpm database is corrupted.

Agent Subscription Facility (ASF) activity log customization parameter

You can change the default number of wrap-around files or the maximum file size limit in the ASF
activity log for the agent. Change these settings if it takes a long time to reproduce a problem scenario
during problem determination, or if you are short on disk space and want to reduce the log file size.

The activity log file includes all time-stamped interaction between the agent and the MIN server, so the
data exchange details can be examined, verified, and analyzed for correctness and for problem
determination. The log contains the following information:
v All agent and server send and receive buffers.
v Agent Service Interface, Private Situations, Event Exporter, Centralized Configuration Facility, EIF
emitter, and Agent Subscription Facility operation log messages.
7
v Centralized Configuration file download status.
v Private situation event state status.
v Configuration files processing and validation status.
v Other agent operation status messages.

To change the number of wrap-around files or maximum file size limit in the activity log, insert the
environment configuration parameter in the following files:
pc.ini file or global.environment file
KpcENV file

For the ASF-enabled agent, specifying Y (default) for the IRA_ASF_ACTIVITY_LOG configuration parameter
instructs ASF to create the following activity log file in the installation log directory or in member
KpcACTSF in the RKANDATU data set: pc_instance_asfActivity_agent-start-date_agent-start-time-
xx.log. For example:
lz_asfActivity_20140715_123015-01.log
r6_DTYA_asfActivity_20140707_201452-05.log
The following configuration parameters can be used:
IRA_ASF_ACTIVITY_LOG=
Y instructs ASF to create an activity log file and output all Subscription Network Task send and
receive data buffer content and other status data to the file, up to 1024 bytes in length. The default is
Y.
IRA_ASF_ACTIVITY_LOG_LIMIT=
Specifies the number of activity log files and the file size limit in MB. The default is
IRA_ASF_ACTIVITY_LOG_LIMIT=n,m. Where:
n Number of wrap-around files that the agent maintains. The maximum number of files is 50.
The default is 3.
m Number of MB for each agent run (start). The maximum number of MB is 99. The default is
10.
The agent manages a maximum of nine run sets of activity log files.

DataPower agent: Configuration failed with SSL security error

Symptom
The configuration fails with an error message after the DataPower® appliance with SSL version 3
is disabled.
Cause A security issue caused the problem.
Solution
Complete the following steps:
1. Download the cert from the xxx.pem Data Power® appliance file that is used in the SSL profile.
2. Copy the cert file to the agent, and import the cert by using this command:
cd ../JRE/lx8266/bin
./keytool -import -trustcacerts -alias ligitalsso -keystore
/opt/ibm/apm/agent/JRE/lx8266/lib/security/cacerts -storepass
changeit -file /opt/ibm/apm/TLSTEST-sscert.pem
3. Add the pem file during the agent configuration when you complete the SSL profile. The agent
configuration file is shown here:
SECTION=DPS:dp23 [ { DP_PORT=5550 } { DP_UID=admin } { SOAP_HOST=9.123.xxx.xxx }
{ DP_PASSWORD=\{AES256:keyfile:a\}TXw7lXT3aCUO0sMq8y3Asw\=\= }
{ DP_SSL_PROFILE=/opt/ibm/apm/TLSTEST-sscert.pem } ]

8 Troubleshooting for Performance Management Version 8.1 (On Premises):

DB2 agent: Configuration failed with permission error for non-root
user after installation with root user
Symptom
The configuration failed with a permission error when you installed the DB2® agent as a root
user and then switched to non-root user to configure and start agent.
Cause A non-root user does not have permission to write files to the root folder.
Solution
Run the ./secure.sh script with the group name of the non-root user to secure the files and set
the file group ownership to the files. Example: ./secure.sh -g db2iadm1.
For details about the secure.sh script, see Securing the agent installation files on AIX.

DB2 agent: Agent uninstallation command reports failed uninstallation

Symptom
After using the ./smai-agent.sh uninstall_all command, the DB2 agent was uninstalled
successfully, but an error reported that uninstallation failed.
Cause The DB2 agent process that was running was not stopped and file dependencies caused some
common components to be uninstalled.
Solution
Use the DB2 agent process ID for the process that is running and stop it. Then, rerun the
following uninstallation command: ./smai-agent.sh uninstall_all.

HMC Base agent user: Configuration

Symptoms
When you start the agent instance, there are some prompts in the console and the HMC base
agent dashboards do not contain any data as shown here:
Password:
Password:
Password:
username@hmchostname password:
username@hmchostname password:

In the kph_data_provider__instance name_startup.logagent log file, the following logs are

shown:
"Failed to logon to the hmc server, please check if username and password
are correct"
"stopping the hmc base data provider process, please reconfigure with correct username,
then restart agent!"
Cause The agent instance is configured with the incorrect HMC console user or the user does not have
hscviewer authority.
Solution
Complete the following steps:
1. Stop the agent.
2. Run the /aix526/ph/bin/setup_hmc_key.pl perl script and include the correct HMC user
name to generate the ssh keys to access the HMC console server.
3. Configure the agent with the same user name that you applied in step 2.
4. Start the agent.

Chapter 2. Installation 9
Linux KVM agent: Installation errors on UNIX systems
The following messages apply to installation errors on UNIX systems:
v Installing the same version
v Unable to find running CMS
v Agent fails to start return code 11
v Failed to connect to data source
v bash: nc: command not found

See the symptom and solution for the message you received.

Installing the same version warning

Symptom
During a command-line installation, when you install a component that was previously installed,
you see the following warning:
WARNING - you are about to install the SAME version of "component_name"
where component_name is the name of the component that you are attempting
to install.

Important: This problem affects UNIX command-line installations. If you monitor Windows
environments only, this problem occurs if you install a product component, for example, a
monitoring server on a UNIX system.
Solution
You must exit and restart the installation process. You must not return to the list where you
selected the components to install. When you run the installer again, you must not attempt to
install any component that is installed.

Unable to find running CMS message

Symptom
A message that is similar to the following message is displayed in the log file:
"Unable to find running CMS on CT_CMSLIST"
Solution
The agent cannot connect to the monitoring server and you must confirm the following
environment settings:
1. Do multiple network interface cards (NICs) exist on the system?
If multiple NICs exist on the system, you must determine the NIC that is configured for the
monitoring server.
2. Ensure that you specify the correct host name and port settings for communication in the IBM
Tivoli Monitoring environment.

Agent fails to start return code 11

Symptom
After installation, the Linux KVM agent instance fails to start.
The following message is shown in the agent log:
(4CF55620.003F-1:kbbssge.c,52,"BSS1_GetEnv") KBB_SIG1="-asyncoff -syncoff
-dumpoff"
(4CF55620.0040-1:signalmanager.cpp,170,
"startManagerThread") Error starting signal managerthread.
Return code = 11; Resource temporarily
unavailable. Use the return code and message to investigate the failure.
Agent is terminating.

10 Troubleshooting for Performance Management Version 8.1 (On Premises):

Cause The cause of the problem is the pdksh public domain Korn shell.
Solution
You must uninstall the pdksh shell and install the ksh rpm that is included on the Linux
installation media.

Failed to connect to data source message

Symptom
In the agent log, you see the message
SEVERE: DataSource.connect: failed to connect to data source ip address
Solution
1. Find the hypervisor URI of the host that failed to connect. The URI is listed in a message just
before the failed to connect message.
2. On the computer where the agent is installed, enter the export LIBVIRT_DEBUG=yes command
followed by the virsh -chypervisor uri that failed command.
3. Review the extra debug messages for symptoms of an underlying problem.

bash: nc: command not found message

Symptom
You turn on the extra libvirt messages by using the export LIBVIRT_DEBUG=yes command. Then,
when you run the virsh -chypervisor uri command, you see the following message:
bash: nc: command not found
Solution
Install the netcat-openbsd package on the host of the hypervisor that you want to monitor.

Microsoft IIS agent: When Windows OS agent is uninstalled, the

Microsoft IIS agent is also uninstalled
Symptom
If both the Microsoft IIS agent and Windows OS agent are installed on a managed system, then if
you uninstall the Windows OS agent, the Microsoft IIS agent is also uninstalled.
Solution
No solution is available for this problem currently.

Microsoft SQL Server agent: Installation failure with KGL and KGS
components on 32-bit Windows 2003 system
Symptom
The installation of Microsoft SQL Server agent fails with a KGL and KGS component error on a
32-bit Windows 2003 system.
Cause The prerequisite scanner does not work on a 32-bit Windows 2003 system.
Solution
Ensure that the prerequisite scanner does not run during the agent installation. To skip the
prerequisite scanner, add the SET SKIP_PRECHECK=Y variable in the mssql-agent.bat file.

Chapter 2. Installation 11
Microsoft SQL Server agent: Historical Data Collection collects data
after a slight delay
Symptom
Historical Data Collection takes some seconds more than the configured data collection interval
time to finish data collection. This delay gives an impression that some data rows are missing
when the collection interval is 1 minute.
Solution
Keep the collection interval to a minimum of 5 minutes and use a smaller interval for an attribute
group if it is critical.

Microsoft SQL Server agent: Database Server Properties window does

not display the upgraded version of the SQL Server
Symptom
When you upgrade the SQL Server from V2005 to V2008 R2, V2012, or V2014 and configure the
SQL Server agent locally, the database version in the Database Server Properties window does not
display the upgraded version of the SQL Server.
Solution
Remove the earlier instance of the SQL Server agent and create another instance.

Microsoft SQL Server agent: In Windows 2003 system, a

non-administrator user cannot configure the SQL Server agent
services
Symptom
In Windows 2003 system, a non-administrator user cannot configure the SQL Server agent
services through the Change Startup option in the Manage Monitoring Services window.
Solution
To configure the SQL Server agent services on a Windows 2003 system, perform the following
steps:
1. On the Windows Start menu, click Run.
2. Type services.msc, and click OK.
3. In the Services window, perform these steps for the agent service and the agent collector
service:
a. Right-click the service, and click Properties.
b. Click the Log On tab.
c. Click This account.
d. Type the user name.
e. In the Password and Confirm Password field, enter password, and click OK.
4. In the Manage Monitoring Services window, right-click the SQL Server agent service, and click
Recycle.

12 Troubleshooting for Performance Management Version 8.1 (On Premises):

Microsoft SQL Server agent: DB INCLUDE/EXCLUDE configuration
panel supports only 256 characters
Symptom
The DB INCLUDE/EXCLUDE configuration panel supports only 256 characters. If you configure
the SQL Server agent with more than 256 characters, the following error message is displayed
and you cannot reconfigure the agent:
Unspecified error
Solution
There is no solution to this problem.

Microsoft SQL Server agent: Historical Data Collection is not

supported for the Customized SQL Query Result data set
Symptom
For the MS SQL Customized SQL Query Result data set, historical data collection (HDC) is not
supported.
Cause
For the MS SQL Customized SQL Query Result data set, historical data collection (HDC) is not
supported because of the following reasons:
v The customized query results are called from other views through a cross link. To run any
specific query, a specific ID is required. A demand request or a historical data collection request
cannot provide specific IDs.
v If you run a query for which an ID is not assigned, all the queries that are defined in the
KoqCusSQL.properties file are run. This unwanted query execution increases the load on the
SQL Server.
Solution
There is no solution to this problem.

Oracle Database agent: Cannot read remote alert log file on Windows
systems
Symptom
The Oracle Database agent cannot read the remote alert log file on Windows systems.
Solution
When you configure the Oracle Database agent to access the remote alert log file, apply the
following guidelines:
v The remote file path must follow the universal naming convention. For example,
\\tivx015\path\alert_orcl.log. And file path in a mapped network drive is not supported.
v The Oracle Database agent must run under an interactive user account, instead of the system
account.
v The files on the remote server can be accessed without a password, or the remote server is
connected by using the Oracle Database agent user account.

Oracle Database agent: Cannot load the configuration file

Symptom
The agent cannot load the configuration file and cannot be started when the system OS host
name is not consistent with the NetBIOS host name.
Cause This problem happens because the XML configuration tool generates the configuration file by

Chapter 2. Installation 13
 using the OS host name file (OS_hostname_rz_agent_instance.cfg), but the agent loads the
configuration file by using the NetBIOS/NetworkAdapter host name file
(NetBIOS_hostname_rz_agent_instance.cfg).
Solution
Implement one of the following solutions:
v Change the system OS host name to match the NetBIOS host name, and reconfigure the agent.
v Copy the OS_hostname_rz_agent_instance.cfg file to the
NetBIOS_hostname_rz_agent_instance.cfg file.
v On a UNIX system, link the OS_hostname_rz_agent_instance.cfg file to the
NetBIOS_hostname_rz_agent_instance.cfg file.

Oracle Database agent: Uninstallation command does not uninstall

agents
Symptom
When the Oracle Database agent is started before the OS agent was started, the smai_agent.sh
uninstall_all command does not uninstall all agents.
Cause The Oracle Database agent instance was considered as not configured by the installer.
Solution
Stop the Oracle Database agent the first time and then run the smai_agent.sh uninstall_all
command.

SAP agent: Monitoring agent does not start in a non-ASCII

environment
Symptom
The monitoring agent does not start in a non-ASCII environment.
Solution
Check the agent configuration to ensure that all of the values are correctly represented. To view
these parameters, go to the Manage Monitoring Services window, select the agent instance, and
click Reconfigure. In the subsequent windows, review and modify configuration parameters as
needed.

SAP agent: Agent repeatedly restarts

Symptom
The SAP agent repeatedly restarts.
Solution
You can collect data to analyze this problem as follows:
1. Access the install_dir/config/.sa.environment file.
2. Add the following line: KBB_SIG1=trace –dumpoff

SAP agent: Agent transport errors

Symptom
Agent transport errors.
Solution
v Update the following SAP kernel executables to the latest level:
– R3trans

14 Troubleshooting for Performance Management Version 8.1 (On Premises):

 – tp
v Verify that the cofile and data transport files have correct authorizations and owners:
sapsid:sapsys
v Check the syntax of the following commands:
– tp addtobuffer
– tp import
For more information about these commands, see Step 5 in the “Using SAP transport and
defining the user” section in the IBM Application Performance Management (SaaS) Knowledge
Center.
v Ensure that the default user /IBMMON/ITM_AUTH roles were created during transport import.
You can use the CUA to monitor an SAP system. To use the predefined user ID and
authorization role to monitor an SAP system set up with Central User Administration,
complete one of the following steps:
– Install the transport into the Central User Administration parent logical system client.
– Manually create the user ID or role in the client where you want to install the transport. The
user ID or role is in the client where the transport is installed (imported).
– Manually create the user ID or role in the Central User Administration parent logical system
client. Then, distribute the user ID or role to the client where the agent runs.
– Manually create the user ID or role in the Central User Administration parent logical system
client and run the agent in this client.

SAP agent: Monitoring agent cannot connect to the mySAP System

Symptom
The SAP agent cannot connect to the mySAP System. The agent is started but no :Ins or :Sys
managed nodes are shown when you add the “SAP Application” and “SAP System” application.
Solution
Ensure that you configured the agent with the correct mySAP logon information (user ID,
password, client). Reconfigure, and restart.
Check the RAS1 log for connection errors. An RAS1 error such as the following indicates that the
agent could not log on on with the connection parameters specified during agent configuration:
Failure on call to /IBMMON/ITM_VERIFY_LOGON. Verify that all the values are correct. For more
information, see IBM Application Performance Management (SaaS) Knowledge Center.
Ensure that the mySAP system, application server the agent connects to, or both are running and
can accept new connections. Use transactions SMGW and SM04 to determine whether there are
free connections on the application server for the monitoring agent to use.
Use the fully qualified host name or IP addresses if you are configuring the agent by using
simple host names.
Ensure that no firewalls are blocking access to mySAP.

SAP agent: Cannot connect to the SAP system by using the Logon
Group mode
Symptom
You cannot connect to the SAP system by using the Logon Group mode.
Solution
Use the following new environment variables:
v SAPLOGONGROUP: Name of the SAP logon group.

Chapter 2. Installation 15
 v SAPMSGSERVER: Host name of the SAP message server. Alternatively, use an IP address.
v SAPMSGSERVICE: Message service name, for example, sapmsTV1 or a full message service
port number, for example, example: 3601.
v SAPROUTESTRING: Route string to the SAP system.

Note: You must include the service names in the following operating system services files:
v UNIX systems: /etc/services
v Windows systems: \windows\systems32\drivers\etc\services

SAP agent: Cannot locate the KDCB0_HOSTNAME setting

Symptom
Cannot locate the KDCB0_HOSTNAME setting.
Solution
Go to install_dir/config and edit the sa_3-character-id.config file. Set the KDCB0_HOSTNAME
parameter to the IP address of a network card on this computer. If you use multiple network
interface cards (NICs), use the Primary IP address of the network interface.

Multiple agents: Installation failed on AIX

The following agents might have this problem:
v DataPower agent
v Oracle Database agent
Symptom
The installation failed on an AIX system because the default .tar command truncated a long path.
On AIX systems, the existing .tar command has some size limits that restrict file names to 100
characters, and restrict UID and GID values to o77777777.
Solution
Extract the files from the archive file. To extract the archive file, obtain the fix that is provided in
APAR IV20002 (http://www-01.ibm.com/support/docview.wss?uid=isg1IV20002).
Alternatively, you can use GNU tar to extract the files. To install GNU tar, go to
http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/alpha.html
Optionally, you can directly download the tar rpm to the AIX system: ftp://
ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/tar/tar-1.22-1.aix6.1.ppc.rpm
Run the following command:
rpm -i tar-1.22-1.aix6.1.ppc.rpm

Then, use the GNU tar to extract the archive by using the following command:
/opt/freeware/bin/tar -xf package.tar

Multiple agents: Agent installation failed for SaaS agents with an

existing IBM Tivoli Monitoring V6.2.3 framework
This problem occurs with any SaaS agent with an existing IBM Tivoli Monitoring V6.2.3 framework or
lower version.
Symptom
When you install the agent, you receive the following error message and the installation failed:
KCI5092E Unable to install agent on this system. Installation will break existing
IBM Tivoli Monitoring environment in <ITM Home> the default location is /opt/IBM/ITM.
Upgrade IBM Tivoli Monitoring agents to latest version and start installation again.

16 Troubleshooting for Performance Management Version 8.1 (On Premises):

Solution
1. Upgrade the IBM Tivoli Monitoring agents to the latest version:
a. Get the IBM Tivoli Monitoring V6.3.0 or later framework.
b. Run the installation program to upgrade the IBM Tivoli Monitoring framework to the
latest version. Then, restart your existing IBM Tivoli Monitoring agents.
2. After you correct the errors, reinstall the SaaS agent.

Performance Management server installation failure and the RPM

database
If the Performance Management server installation fails, the log files might show a problem with the
RPM database.

When the Performance Management server installation fails, you might encounter the following error:
Mon Feb 9 08:02:56 CST 2015: rpmdb: PANIC: fatal region error detected; run recovery
Mon Feb 9 08:02:56 CST 2015: error: db3 error(-30974)
from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
Mon Feb 9 08:02:56 CST 2015: error: cannot open Packages index using db3 - (-30974)
Mon Feb 9 08:02:56 CST 2015: error: cannot open Packages database in /var/lib/rpm

To solve this problem, complete the following steps:

1. Remove the rm -rf /var/lib/rpm/__db.00* rpm lock files.
2. To rebuild the RPM database, enter the rpm --rebuilddb command.
3. To uninstall all the components that you installed to /opt/ibm/, enter the /opt/ibm/ccm/uninstall.sh
command.
4. To restart the installation, enter the./install.sh command.

Chapter 2. Installation 17
18 Troubleshooting for Performance Management Version 8.1 (On Premises):
Chapter 3. Integration
Review the integration troubleshooting symptoms and solutions for help with issues when integrating
with other products.

Hybrid Gateway: Unable to connect to the Tivoli Enterprise Portal

Server
Symptom
After you configure the Hybrid Gateway a connection is not established with the Tivoli
Enterprise Portal Server.
Cause There is an error in the configuration or communication is blocked.
Solution
v Verify that the address, port, web communications protocol, user name and corresponding
password of the Tivoli Enterprise Portal client are correctly configured.
v Verify that ports 15201 or 15200 are not blocked for http or https communication.
v If access to the portal server goes through a proxy server, verify that the address, port, and
web protocol are correctly configured.
v Confirm that the IBM Tivoli Monitoring dashboard data provider has been enabled on the
portal server. For details, see “Verifying the dashboard data provider is enabled” in the IBM
Tivoli Monitoring Knowledge Center.
v Review the Hybrid Gateway log file at /opt/ibm/wlp/usr/servers/hybridgateway/logs/
message.log.

Hybrid Gateway: Tivoli Monitoring domain managed systems are not

visible in the Application Performance Dashboard
Symptom
After you configure the IBM Performance Management Hybrid Gateway, the managed systems
from your Tivoli Monitoring domain are not displayed in the Application Performance
Dashboard.
Cause The Tivoli Monitoring agents are not online, configured, or supported. The configured user does
not have access to view the monitored applications.
Solution
v Verify that the Tivoli Monitoring agents are included in the managed systems group and that
the group is configured in the Hybrid Gateway. For more information about the Hybrid
Gateway, see Integrating with Tivoli Monitoring.
v Verify that the Tivoli Monitoring agents are online. You can check the status of agents in the
following ways
– View the Managed System Status workspace in the Tivoli Enterprise Portal. For more
information about viewing the Managed System Status workspace, download the Tivoli
Enterprise Portal Users Guide PDF from IBM Tivoli Monitoring Product Publications and
search on Managed System Status.
– Use the Tivoli Monitoring tacmd viewAgent CLI command. For example, use the command
tacmd viewAgent -t LZ to list all Linux OS agents.
v Verify that the Tivoli Monitoring agents are supported. For more information about supported
agents, see Integrating with other products.

19
 v Verify that the Tivoli Enterprise Portal Server user specified in the Hybrid Gateway
configuration has access to view all of the applications. For more information about the Hybrid
Gateway, see Integrating with Tivoli Monitoring.
v Review the Hybrid Gateway log file at /opt/ibm/wlp/usr/servers/hybridgateway/logs/
message.log.

20 Troubleshooting for Performance Management Version 8.1 (On Premises):

Chapter 4. Agent
Review the agent troubleshooting symptoms and solutions for help with agent issues.

For the WebSphere Applications agent, see the WebSphere Applications agent Troubleshooting
documentation.

For the Linux OS agent , UNIX OS agent, and Windows OS agent, see the Version 6.3 Fix Pack 2
documentation documentation on the wiki.

For the Response Time Monitoring Agent, see the Troubleshooting: Web Response Time topic in the
SmartCloud Application Performance Management Knowledge Center.

Active Directory agent: Dashboard errors

You can use the Active Directory agent: Dashboard errors topic to troubleshoot errors that relate to the
Microsoft Active Directory agent dashboard.

LDAP Successful Bind attribute appears with value 0 on the

dashboard
Symptom
LDAP Successful Bind attribute appears with value 0 on the dashboard though this attribute is
absent in Perfmon on Windows 2003 system while it appears as Undefined on Windows 2008
system.
Solution
Currently, there is no workaround for this problem.

Dashboard does not display the updated list of instances

Symptom
When you add or remove instances from the following Perfmon objects, the dashboard does not
display the updated list of instances:
v DFS Replication Connections
v DFS Replication Folders
v DFS Service Volumes
Solution
Recycle the Microsoft Active Directory agent after you add or remove an instance from the
following Perfmon objects:
v DFS Replication Connections
v DFS Replication Folders
v DFS Service Volumes

DataPower agent: AIX Installation

Symptom
The DataPower agent does not install. Also, the DataPower agent crashes after a few hours.
Cause On AIX, ulimit -d unlimited is not set.

21
Solution
On AIX, you must set ulimit -d unlimited.

DB2 agent: Memory leak

Symptom
A memory leak occurs with a long-running DB2 agent.
Cause The memory leak might be caused by APAR IC93170 for the DB2 Server (http://www-
01.ibm.com/support/docview.wss?uid=swg1IC93170).
Solution
Use a later version of the DB2 server with a fix integrated to eliminate the memory leak. Fix
packs are available in APAR IC93170.

DB2 agent: No restart after unexpected termination

If no data is available from the DB2 agent, it might be offline or the agent configuration is invalid.
Review the possible symptoms, causes, and solutions:
No automatic restart
Symptom
The DB2 agent is not automatically restarted after terminating unexpectedly.
Cause The DB2 agent instances were not configured before agent startup, and the agent is not
being monitored.
Solution
Restart the OS agent after configuring the DB2 agents.
Delayed automatic restart
Symptom
The DB2 agent is not restarted until up to 30 minutes after terminating unexpectedly.
Cause If agent instances are not configured at agent startup, they are not discovered until 30
minutes later.
Solution
When the agent instances are configured at startup, the restart occurs after the DB2 agent
instances are discovered, up to 30 minutes later. After discovery, the normal status check
of every 5 minutes takes place.

Linux KVM agent: Attributes not available on RHEL 5 and 6 systems

The following attributes might not be available on Red Hat Enterprise Linux (RHEL) 5.x and RHEL 6.x
systems:
v CPU model and topology attributes, such as, sockets per node, cores per socket, and threads per core
v Host Memory data set attributes
– VM Memory Allocated
– Percent Memory Used
– VM Memory Allocated Fit Estimate
v Virtual Machines data set attributes
– Memory Allocated (GB)
– Memory Used (GB)
– Memory Percent

22 Troubleshooting for Performance Management Version 8.1 (On Premises):

See the following symptoms and solutions for the attributes that are not available to you.

CPU model and topology attributes not available

Symptom
On Red Hat Enterprise Linux (RHEL) 5.x and RHEL 6.x systems, the CPU model and topology
attributes, such as, sockets per node, cores per socket, and threads per core, are not available.
Solution
When the CPU model and topology attributes are not available, use the following solution that is
applicable to your environment:
v Red Hat Enterprise Linux (RHEL) 5.x systems
In the Red Hat Enterprise Linux (RHEL) 5.x environment, you can use the processing
information to create new VMs or domains. Then, going forward, you can ignore previous
VMs. To create these new VMs or domains, as you create a virtual machine or domain,
complete the following steps:
1. Use the virsh command to output xml to a file: > virsh dumpxml > domain1.xml.
2. Manually, update the domain1.xml with the domain name and UUID and add the cpu
model and the following topology elements:
<cpu match="exact">
<model>genuineIntel</model>
<topology sockets="1" cores="2"threads="1"/.
</cpu>

Note: For more information about values that relate to the model element, see CPU model
and topology (http://libvirt.org/formatdomain.html#elementsCPU)
3. Create a new virtual machine or domain by using the following command to update the
xml file: virsh create domain1.xml
v RHEL 6.x systems
1. On the command console, run the > virt-manager command.
2. In the Virtualization Manager window, double-click the VM to configure.
3. From the Virtual Machine window menu, select View > Details.
4. Select Processor. Then, in the Configuration section, set the CPU model, and in the
Topology section, set the topology details.

Host Memory and Virtual Machines data set attributes not available
Symptom
When the agent is installed on an RHEL 5.x system, or when an agent that is installed on an
RHEL5 or an RHEL6 system monitors an RHEL 5 KVM host, the following data sets and
attributes are not available:
Table 2. Data sets and attributes not available in RHEL 5 systems
Data set Attribute Value
Host Memory VM Memory Allocated 0
Percent Memory Used 0
VM Memory Allocated Fit Estimate Unavailable
Virtual Machines Memory Allocated (GB) Unavailable
Memory Used (GB) Unavailable
Memory Percent Unavailable

Chapter 4. Agent 23
Solution
This issue is a known issue with the RHEL 5.x environment. You must install the Linux KVM
agent and the KVM host on the RHEL 6.x. environment.

Linux KVM agent: Errors with negative values

Symptom
Some values, for example, VM Memory Used (GB), or Percent Memory Used, are negative.
Solution
This issue is known for Red Hat Enterprise Virtualization Manager (RHEVM) API 3.2 as
described in the Bug report (https://bugzilla.redhat.com/show_bug.cgi?id=1068884). The
problem is resolved in RHEVM API V 3.3.4 and V 3.4.0. If you cannot upgrade to RHEVM API V
3.3.4 or RHEVM API V 3.4.0, then restart the VM or the host.

Linux KVM agent: oVirt API error in the RHEVM environment

Symptom
For RHEVM systems, an oVirt API error occurs in the log files after you start the Linux KVM
agent. The following message is shown in the log files:
An oVirt API error has occurred while connecting to <host_name>:
code :401
reason: Unauthorized
detail: <html><head><title>JBoss Web/7.3.0Final-redhat-1 - JBWEB000064:
Error report</title><style>...
Solution
1. Ensure that the specified password for the RHEVM connection is correct.
2. Alternatively, ensure that the host name matches the host name in the server security
certificate.
Generally, security certificates from the RHEVM server are qualified by the domain name,
rather than the IP address. You must use the domain name in the certificate to specify the
host name in the configuration.

Microsoft Exchange Server agent: Exchange Server 2013 component

shows Exchange 2007 and 2010 instances
Symptom
On the Application Performance Dashboard, when you select the Microsoft Exchange Server
2013 component under My Components, the Exchange Server 2007 and 2010 instances are
displayed as instances of Microsoft Exchange Server 2013 along with instances of Exchange
Server 2013. The dashboard shows incomplete data for the Exchange Server 2007 and 2010
instances.
Cause This problem occurs when Exchange Server 2007 and 2010 instances are selected as instances of
the Microsoft Exchange Server 2013 component when you create an application.
Solution
Always ensure that you select the correct monitored Exchange Server instances for each
component of Exchange Server when you create an application. For example, if you want to
monitor Exchange Server 2007 or 2010, select Exchange 2007 or 2010 instances for the Microsoft
Exchange Server component. If you want to monitor Exchange Server 2013, select Exchange
Server 2013 instances for the Microsoft Exchange Server 2013 component.

24 Troubleshooting for Performance Management Version 8.1 (On Premises):

Microsoft Exchange Server agent: Dashboard defects
No data for the Last Scan Time (UTC) attribute
Symptom
The dashboard does not display data correctly for the Last Scan Time (UTC) attribute.
Solution
A solution is currently not available.

Incorrect data for the Total Log Bytes Generated attribute

Symptom
The dashboard does not display data correctly for the Total Log Bytes Generated attribute of the
MS Exchange IS HA Active Database attribute group.
Solution
A solution is currently not available.

Microsoft Exchange Server agent: Low mailbox space

Symptom
The available mailbox space is low.
Cause The mailbox accumulates large amount of data because of emails that the Microsoft Exchange
Server agent sends to verify reachability. The agent receives automated responses from the remote
Exchange Servers. Depending on the reachability interval, the number of email responses in the
inbox might become quite large.
Solution
To keep the agent processing time minimum, ensure that you clean the mailbox periodically. You
can use the tool such as MapiEdit from Microsoft to clear your mailbox. You can also use the
built-in option in the Exchange Server agent to delete each automated response after it is
analyzed for response time. This option keeps the mailbox cleared during agent processing. To
enable this option, you must create the KEX_DELETE_EMAILS environment variable, and set the
value of the variable to YES. To activate this option, you must restart the Microsoft Exchange
Server agent.

Microsoft Hyper-V Server agent: Agent log files are not in the
CANDLEHOME/tmaitm6/logs directory
Symptom
Agent log files are not in the CANDLEHOME/tmaitm6/logs directory.
Solution
If you reset the logging level to "none," and then turn logging back on by setting the logging
level to something other than "none," the Manage Monitoring Services utility automatically resets
the log directory to CANDLEHOME/logs.
Logging is not affected. Older logs remain in CANDLEHOME/TMAITM6/logs. New logs are placed in
CANDLEHOME/logs.
You can manually edit the ENV file using the Manage Monitoring Services utility:
1. Right-click the agent. Select Advanced > Edit ENV file.
2. Adjust the line prefixed by KBB_RAS1_LOG to the line in the example. You must substitute
ITM_HOME with the path to your installation directory.
-OR-

Chapter 4. Agent 25
 You can also add "\TMAITM6" before all "\logs" in the KBB_RAS1_LOG line. Example:
KBB_RAS1_LOG=ITM_HOME\TMAITM6\logs\$(computername)_q5_kq5agent_$(sysutcstart)-.log
INVENTORY=ITM_HOME\TMAITM6\logs\$(computername)_q5_kq5agent.inv COUNT=03 LIMIT=5
PRESERVE=1 MAXFILES=9

Microsoft Hyper-V Server agent: Memory leak is observed

Symptom
If the Microsoft Hyper-V Server agent is installed on Windows Server 2008, Service Pack 2, a
memory leak is observed.
Solution
Install the hot fix for Windows Server 2008, Service Pack 2 from Microsoft Support
(http://support.microsoft.com/kb/970838).

Microsoft Hyper-V Server agent: Process name gets truncated on the

dashboard
Symptom
When you use a long process name in the situation, the process name is truncated.
Solution
Truncation of process or service names for situations in the Availability table in the portal display
is the expected behavior. The maximum name length is 100 bytes.

Microsoft Hyper-V Server agent: Rollup errors occur in the agent log
file
Symptom
Rollup Error messages in the agent log file.
Solution
Rollup error messages occur when a base WMI query fails and no source data is available to "roll
up." Check the log for the initial WMI error to resolve. The rollup error disappears when the
initial problem is corrected.

Microsoft Hyper-V Server agent: VHD File Size (GB) displays 0 in the
Virtual Machine Details group widget
Symptom
On the dashboard, in the Virtual Machine Details group widget, the value of the VHD file size
(GB) attribute is shown as 0.
Cause The VHD file size is less than 500 MB.
Solution
A solution is not currently available.

Microsoft Hyper-V Server agent: WMI Error 0x80041006

Symptom
WMI Error 0x80041006: There was not enough memory for the operation.
—OR—
WMIPrsve.exe memory increases to 135MB before recycling.

26 Troubleshooting for Performance Management Version 8.1 (On Premises):

Solution
The issue is a known issue with the Microsoft WMI Cluster Association class provider, which
IBM Software Support is working with Microsoft to resolve. Until a fix is available, the following
workaround has been provided from Microsoft. It sets the Cluster WMI provider to run in a
standalone WMIprvse.exe host. The script also lowers the cache timeout to 30 seconds. So, if the
cluster provider is idle for more than 30 seconds, the hosting WMIprvse.exe process is recycled,
minimizing the effect of the leak.
1. Copy and paste the following into a file and save with a .vbs extension (for example,
wmi_fix.vbs). If the comments (lines beginning with ’) cause problems, remove them.
'===[ Script start ]=====================
==============
Set objLocator = CreateObject
("wbemscripting.swbemlocator")
set WMIServices = objLocator.ConnectServer
(".","root")
set cachecontrol = WMIServices.Get
("__ObjectProviderCacheControl=@")
cachecontrol.ClearAfter =
"00000000000025.000000:000"
cachecontrol.Put_ set WMIServices =
objLocator.ConnectServer(".","root\MSCluster")
set ClusProvReg = WMIServices.Get
("__Win32Provider.Name=
""MS_CLUSTER_PROVIDER""")
ClusProvReg.HostingModel =
"NetworkServicehost:ClusterProvider"
ClusProvReg.Put_
'===[ Script stop ]==================
=================
2. Run it on each computer where the agent is running.
3. Restart the system.
If the problem persists (which typically happens if there are multiple objects querying WMI, such
as a second agent), try reducing the cache time from 25 seconds to a lower value.

Microsoft Hyper-V Server agent No data is displayed in the dashboard

for the Linux virtual machine
Symptom
On the Virtual Machine page for the Linux virtual machine, no data is displayed in any of the
group widgets. The Virtual Machine Details group widget shows the no-os value for the
Operating System widget.
Cause This problem occurs because of NetworkManager on the Linux virtual machine that is hosted on
the Hyper-V Server. NetworkManager causes some instability in the ethernet configuration,
which is a known issue with Linux virtual machines that are hosted on Hyper-V Servers.
Solution
Complete the following steps on the virtual machine to turn off NetworkManager and start the
network service:
1. On the command line, enter the following commands:
a. service NetworkManager stop
b. chkconfig NetworkManager off
c. service network start
d. chkconfig network on
2. In the ifcfg-eth0 file, enter NM_Controlled=no to prevent NetworkManager from starting
automatically.

Chapter 4. Agent 27
 If this problem still persists, add the host name of the virtual machine in the host file that is
available at the etc/hosts path, and then refresh the dashboard.

Microsoft IIS agent: Managed system seems to be offline

Symptom
A managed system seems to be offline.
Solution
1. In the Applications pane, click My Components. In the Groups pane, expand Components
and then select Microsoft Internet Information Services. The Instances pane in the lower left
of the window shows the list of IIS server instances and their status.
2. If a system is offline, check network connectivity and the status of the specific system or
application.

Microsoft IIS agent: Monitoring data fails to be displayed

Symptom
Regular (non-historical) monitoring data fails to be displayed.
Solution
Check the formation of the queries you use to gather data. For example, look for invalid SQL
statements.

Microsoft IIS agent: Availability status shows PROCESS_DATA_NOT_

AVAILABLE
Symptom
The process application components are available, but the Availability status shows
PROCESS_DATA_NOT_ AVAILABLE.
Solution
This problem occurs because the PerfProc performance object is disabled. When this condition
exists, dashboard cannot collect performance data for this process. Use the following steps to
confirm that this problem exists and to resolve it:
1. In the Windows Start menu, click Run.
2. Type perfmon.exe in the Open field of the Run window. The Performance window is
displayed.
3. Click the plus sign (+) in the toolbar. The Add Counters window is displayed.
4. Look for Process in the Performance object menu.
5. Complete one of the following actions:
v If you see Process in the menu, the PerfProc performance object is enabled and the problem
is coming from a different source. You might need to contact IBM Software Support.
v If you do not see Process in the menu, use the Microsoft utility from the Microsoft.com
Operations website to enable the PerfProc performance object.
The Process performance object becomes visible in the Performance object menu of the
Add Counters windows, and IBM Tivoli Monitoring is able to detect Availability data.
6. Restart the monitoring agent.

Microsoft IIS agent: Dashboard errors

You can use the Microsoft IIS agent: Dashboard errors topic to troubleshoot errors that relate to the
Microsoft IIS agent dashboard.

28 Troubleshooting for Performance Management Version 8.1 (On Premises):

Dashboard shows a blank row for the newly added performance
counters
Symptom
The IIS agent does not collect data for performance counters that are added after you start the
agent. Therefore, the dashboard shows a blank row for the newly added performance counters.
For example, if FTP or SMTP or NNTP services are installed after you install the IIS agent, the IIS
agent does not retrieve data for the FTP or SMTP or NNTP performance counters.
Solution
Recycle the agent so that the values of the newly added performance counters are retrieved and
displayed on the dashboard.

No data is displayed in the Error Statistics (history) group widget and

Request Rate (history) group widget
Symptom
On the dashboard, no data is displayed in the Error Statistics (history) group widget and Request
Rate (history) group widget. This problem can occur because of one of the following reasons:
v The IIS Manager is configured to log the requests that are not in the W3C format.
v The path of the log file directory was incorrectly specified when the monitoring agent was
configured.
Solution
To resolve this problem, ensure that the following conditions exist:
v The IIS Manager is configured to log the requests that are in the W3C format.
v The path for the log file directory is specified correctly when you configure the Microsoft IIS
agent.

Error Statistics (history) group widget and Request Rate (history)

group widget displays incorrect or shuffled data for some attributes
Symptom
In the dashboard, the Error Statistics (history) group widget and Request Rate (history) group
widget displays incorrect or shuffled data for some attributes. This problem occurs because the
IIS Manager is configured to log the data for some attributes in the log file.
Solution
When you configure the IIS Manager, ensure that all the choices are selected in the W3C Logging
Fields window.

Microsoft .NET agent: The CTIRA_LOG_PATH variable does not work

Symptom
After you modify the CTIRA_LOG_PATH value to change the log path, only part of the log files are
saved in the new directory. Other log files are still saved in the previous directory.
Solution
To customize the log file path instead of using the default path, you must set three environment
variables.
v In the install_dir\TMAITM6_x64\KQEENV file, specify the following two variables:
CTIRA_LOG_PATH=new_log_path
KBB_RAS1_LOG=new_log_path\$(host_name)_qe_kqeagent_$(sysutcstart)-.log

where install_dir is the installation directory of the Microsoft .NET agent; new_log_path is the
customized log path; host_name is the name of the host where the agent is installed.

Chapter 4. Agent 29
 v In the install_dir\TMAITM6_x64\rundotnetagent.bat file, specify the LOGDIR variable as
follows:
set LOGDIR=new_log_path

Microsoft .NET agent: Dashboard errors

You can use the .NET agent: Dashboard errors topic to troubleshoot the errors that are related to the
Microsoft .NET agent dashboard.

Negative value for the Number of Current Physical Threads attribute

Symptom
The dashboard shows a negative value for the Number of Current Physical Threads attribute of
the NET CLR Threads data set.
Solution
A solution is not currently available.

Zero value for the NET CLR Networking data set

Symptom
The dashboard shows the zero value for all attributes of the NET CLR Networking data set.
Solution
Restart the Microsoft .NET agent.

Application instances are not displayed

Symptom
The dashboard does not show the application instances.
Solution
1. Install .NET Framework and check if the .NET applications are running.
2. If you do not see the application instances, reload WMI classes. However, before you reload
the WMI classes, ensure that you back up the WMI repository by running the following
command:
winmgmt/backup filename
This command causes the WMI Service to back up the WMI repository to the specified file
name. The filename argument must contain the full file path.

Remember: You can restore the WMI repository from the specified backup file by running the
following command:
winmgmt/restore filename flag

where flag can either be 1 or 0; where 1 implies that restore the WMI repository after
disconnecting the users and 0 implies that restore the WMI repository, if no users are
connected.
3. Reload the WMI classes. Issue the following commands in the specified order:
Wmiadap /c
This command clears performance library statuses.
Wmiadap /r
This command parses the Windows Driver Model drivers on the system to create
performance objects.

30 Troubleshooting for Performance Management Version 8.1 (On Premises):

 Wmiadap /f
This command parses the performance libraries on the system and refreshes the
Performance Counter Classes.
Net stop winmgmt /y
This command stops the WMI Service.
Net start winmgmt
This command starts the WMI service.
winmgmt /resyncperf
The command registers the system performance libraries with WMI.

Note: The WMI classes take some time to reload. You must wait for at least 2 minutes to see
the updated classes.

Data is not displayed for .NET resources of 32-bit applications

Symptom
If you have installed Microsoft .NET Framework V2.0 or V3.0 on a computer that runs on the
Windows Server 2003, 64-bit operating system, the dashboard does not display data for the .NET
resources of 32-bit applications.
Solution
Install .NET Framework V2.0, Service Pack 1. For more information, see the Microsoft support
website.

Data is not displayed for Windows Workflow Foundation data set

Symptom
The dashboard does not show data for the Windows Workflow Foundation data set.
Solution
In the app.config file or the web.config file of the .NET application, add the following
configuration setting in the configuration tag:
<CustomWorkflowRuntimeSettings Name =
"WorkflowName" EnablePerformanceCounters = "true">

Data is not displayed for some data sets

Symptom
The dashboard does not show any data for the following data sets:
v Service Model Endpoint
v Service Model Operation
v Service Model Service
v Service Model Svc Host
Solution
In the app.config file or the web.config file of the .NET application, add the following
configuration setting in the system.serviceModel tag:
<diagnostics performance Counters = "All"
wmiProviderEnabled = "true">
</diagnostics>

Chapter 4. Agent 31
Data is not displayed for the ASP.NET Applications Request Status
(websockets) data set
Symptom
For Windows Server 2008 R2 and Windows Server 2008 SP2 systems, the dashboard does not
show data for the ASP.NET Applications Request Status (websockets) data set.
Solution
You must reload the Performance Monitor (Perfmon) for ASP.NET Apps v4.0 to retrieve data for
ASP.NET Applications Request Status (WebSockets) data set.
Complete the following steps to reload the Perfmon counters in Windows Server 2008 R2 and
Windows Server 2008 SP2 systems.
1. On the computer, open the command prompt.
2. To go to the System32 directory, run the following command:
cd %Systemroot%\System32
3. To reload the Perfmon counters that are related to ASP.NET Apps v4.0, run the following
command:
Lodctr /R C:\Windows\inf\ASP.NET_4.0.30319\0009\aspnet_perf.ini
Refresh the dashboard to view the data for ASP.NET Applications Request Status (WebSockets)
data set.

Dashboards do not update or display data

Symptom
When you are working with the Microsoft .NET agent and its deep dive dashboards, the group
widgets might either display no data or take a long time to load data. Some of the group widget
do not display data or are slow to update.
Solution
1. Verify that JSO files are being generated in the install_dir\dcruntime\data\request
directory. If JSO files are generated, complete these steps:
a. Modify the JSO file properties in the data collector properties file:
1) From the install_dir\qe\config directory, open the
dotNetDcConfig.properties.inactive file in a text editor.
2) Decrease the maximum number of records stored in a JSO file. Modify the value of the
kqe_svc.requestRecordCount property. The default value is 1000.
3) Decrease the maximum number of JSO files that are maintained in the output
directory. Modify the value of the kqe_svc.maxJsoCount property. The default value is
100.
4) Activate the configuration changes. Enter: cd install_dir\qe\bin configdc
activateconfig
5) Restart the .NET application.
b. Modify the criteria for loading JSO files.
1) From the install_dir\TMAITM6_x64 directory and open the rundotnetagent.bat file in
a text editor.
2) Decrease the time span criteria of the JSO files to load. Modify the value of the
-Dkqe.timespan property in the following line:
"%JAVA_HOME%\bin\java" -classpath %CLASSPATH%
-Dkqe.cache.interval=%KQE_CACHE_INTERVAL% -Xmx1024m
-Dkqe.timespan=1800 -Djlog.propertyFileDir.CYN=%KQE_LOG_CONFIG%

32 Troubleshooting for Performance Management Version 8.1 (On Premises):

 -Dattributegroup.def.jar=%ATTRIBUTE_DEF%
-Ddc.runtime.dir=%DC_RUNTIME_DIR
% com.ibm.tivoli.monitoring.annotation.cpci.CommonAgentCustomClient
> %STDOUT_LOG% 2>&1

In this example, the agent loads JSO files that were generated in the last 30 minutes
(1800 seconds).
3) Restart the agent. Navigate to the install_dir\bin directory and enter the following
commands:
dotnet-agent.bat stop
dotnet-agent.bat start
c. If you registered the data collector profiler to collect all types of diagnostics and
transaction tracking data, increase the maximum heap size of the data collector, if it is not
large enough.
1) Enter cd install_dir\TMAITM6_x64
2) Open the rundotnetagent.bat script in a text editor.
3) Increase the value of the -Xmx parameter in the following line.

Important: The default value of -Xmx384m is provided for an environment where only
request data is collected. If you want to collect all types of data, increase the value to
-Xmx1024m or higher.
%JAVA_HOME%\bin\java" -classpath %CLASSPATH% -Dkqe.cache.interval=%KQE_CACHE
_INTERVAL% -Xmx384m -Djlog.propertyFileDir.CYN=%KQE_LOG_CONFIG%
-Dattributegroup.def.jar=%ATTRIBUTE_DEF%
-Ddc.runtime.dir=%DC_RUNTIME_DIR%
com.ibm.tivoli.monitoring.annotation.cpci.CommonAgentCustomClient
> %STDOUT_LOG% 2>&1
4) Restart the agent. Navigate to the install_dir\bin directory and enter the following
commands:
dotnet-agent.bat stop
dotnet-agent.bat start
5) Restart the .NET application.
If no JSO files are generated, complete these steps:
1. Verify that diagnostics data collection is enabled in the data collector configuration:
a. From the install_dir\qe\bin directory, to view the data collector configuration enter
configdc getconfig
b. Verify that the .NET Data Collector Enabled property is set to True and the HttpModule
property is set to Enabled. For example:
Configuration Enabled: True
.NET Data Collector Enabled: True
Transaction Application Server Enabled: True
Transaction Application Server Host: 127.0.0.1
Transaction Application Server Port: 5456
Profiler: Enabled
SOAP: Enabled
HttpModule: Enabled
ISAPI: Enabled
Trace Log Garbage Collection: Disabled
2. Verify that the profiler service is started. From a command prompt, enter net status
DotNetProfilerService.
If it is stopped, enter net start DotNetProfilerService.

Chapter 4. Agent 33
Microsoft SQL Server agent: Dashboard errors
You can use the Microsoft SQL Server agent: Dashboard errors topic to troubleshoot errors that relate to
the Microsoft SQL Server agent dashboard.

Negative value is returned for Transactions per Second in the

Database Detail data set
Symptom
Negative value is returned for Transactions per Second in the Database Detail data set when the
user database has a similar name to that of a system database.
Solution
There is no solution to this problem.

Collection Status attribute returns a value of Inactive

Symptom
The Collection Status attribute of the Server Summary data set returns a value of Inactive.
A value of Inactive indicates that the SQL Server agent data collector service has stopped. The
collector service might enter this state if:
v The collector service is manually stopped.
v An internal error has occurred.
v The SQL Server is stopped.
Solution
Ensure that the SQL Server is running. If the SQL Server is running and the Collection Status
value is still Inactive, then restart the SQL Server agent.

Incorrect data is displayed for two SQL Server databases with identical
names
Symptom
When you create two SQL Server databases with the same name, but precede one database name
with a space and the other database name without a space, the dashboard does not displays the
data properly.
Solution
The space at the beginning of the database name is truncated by the dashboard, making the two
databases look identical.
Avoid spaces at the beginning of a database name.

Dashboard shows no data for the primary replica in the Availability

Group details data set
Symptom
On the dashboard, no value is displayed for the primary replica in the Availability Group Details
data set.
Cause This problem occurs when you pause or stop the cluster service on the node that hosts the SQL
Server agent.
Solution
Complete the following steps to resolve the problem:
1. In theFailover Cluster Manager, right-click the cluster node that hosts the SQL Server agent.
34 Troubleshooting for Performance Management Version 8.1 (On Premises):
 2. If you have paused the cluster service on the node, click Resume. If you have stopped the
cluster service on the node, click Start.

Dashboard displays incorrect value for some data sets when a custom
query is applied
Symptom
On the dashboard, incorrect values are displayed when a custom query is applied for the
following data sets:
v MS SQL Availability Groups Summary
v MS SQL Availability Database Summary
v MS SQL Availability Database Details
v MS SQL Availability Replicas Status Summary
v MS SQL Availability Replicas Details In Cluster
v MS SQL Service Broker Activation
Solution
Refresh the dashboard after applying the custom query for the following data sets:
v MS SQL Availability Groups Summary
v MS SQL Availability Database Summary
v MS SQL Availability Database Details
v MS SQL Availability Replicas Status Summary
v MS SQL Availability Replicas Details In Cluster
v MS SQL Service Broker Activation

For the Stolen Pages Growth attribute negative values are displayed
on the dashboard
Symptom
On the dashboard, the Stolen Pages Growth attribute displays negative values. The value for the
Stolen Pages Growth attribute is calculated on the basis of the previous and the current sample
data.
Solution
There is no solution to this problem.

For some attributes of the MS SQL Device Detail and MS SQL

Filegroup Details data sets incorrect data is displayed on the
dashboard
Symptom
When the device type is FILESTREAM for an SQL Server database, the following attributes of the
MS SQL Device Detail and MS SQL Filegroup Details data sets display negative or incorrect
values for the database:
v Device Size
v Device Free Space
v Device Free Space Percent
v Filegroup Max Growth Size
v Filegroup Size
v Filegroup Space Percent Used

Chapter 4. Agent 35
Solution
There is no solution to this problem.

No data is collected for the Table Detail attribute

Symptom
No data is collected for the Table Detail attribute when the table is created with a new index
name. A new row is not added in the table when a new index is created.
Solution
There is no solution to this problem.

For some attributes no data is displayed on the dashboard

Symptom
On the dashboard, no data is displayed for some attributes. This problem occurs if a database
name consists of Unicode characters.
Solution
There is no solution to this problem.

Dashboard displays incorrect value for the Audit name attribute of the
Audit Details data set
Symptom
On the Tivoli Enterprise Portal, the Audit name attribute of the Audit details attribute group
displays an incorrect value when you specify the audit name in the following ways:
v If you specify the audit name with double quotation marks such as "TestAudit", then the audit
name on the portal is displayed without any quotation marks as TestAudit.
v If you specify the audit name with double quotation marks in the middle of the string such as
"Test"Audit, then the audit name on the portal is displayed with a single quotation mark and
the last character is truncated. The audit name is displayed as Test'Audi.
v If you specify the audit name with a single quotation mark as 'TestAudit', then the audit name
on the portal is displayed without any quotations marks as TestAudit.
Solution
There is no solution to this problem.

Microsoft SQL Server agent: Database errors

You can use the Microsoft SQL Server agent: Database errors topic to troubleshoot errors that relate to the
databases.

Dashboard displays incorrect data for databases with identical names

Symptom
When you create two SQL Server databases with the same name, but precede one database name
with a space and the other database name without a space, the portal does not display data
properly.
Solution
The space at the beginning of the database name is truncated by the portal, making the two
databases look identical.
Avoid spaces at the beginning of a database name.

36 Troubleshooting for Performance Management Version 8.1 (On Premises):

Dashboard displays no data for databases with table level locking
Symptom
If the table level locking hint (Holdlock, Tablock, and so on) is specified with the SELECT, INSERT,
UPDATE and DELETE statements on multiple tables in an SQL Server database, the portal does not
display data till the lock is released.
Solution
Release the locks.

SQL Server agent takes long time to display data

Symptom
The SQL Server agent takes time to display data when a large number of databases (typically,
more than 6000) are created on the SQL Server. The agent data collector service stops running if
the memory used by the data collector exceeds the threshold value because of a large number of
databases on the SQL Server.
Solution
There is no solution to this problem.

No data is collected for the Table Detail attribute

Symptom
No data is collected for the Table Detail attribute when the table is created with a new index
name. A new row is not added in the table when a new index is created.
Solution
There is no solution to this problem.

Microsoft SQL Server agent: Eventing threshold errors

You can use the Microsoft SQL Server agent: Eventing threshold errors topic to troubleshoot errors that
relate to the situation editor.

Monitoring interval is too long

Symptom
The monitoring interval is too long.
Solution
Access the Situation Editor view for the situation that you want to modify. Check the Sampling
interval area in the Formula tab. Adjust the time interval as required.

Situation does not activate at startup

Symptom
The situation does not activate at startup
Solution
Manually recycle the situation as follows:
1. Right-click the situation and select Stop Situation.
2. Right-click the situation and select Start Situation.

Note: You can permanently avoid this problem by selecting the Run at Startup check box of the
Situation Editor view for a specific situation.

Chapter 4. Agent 37
Microsoft SQL Server agent: Agent does not start
Symptom
Agent does not start.
Solution
Recycle the agent.

Microsoft SQL Server agent: Handle leak occurs during the collector
process
Symptom
For the SQL Server agent, a handle leak occurs during the collector process when the Active
Server Pages performance object uses the aspperf.dll file.
Solution
The Active Server Pages performance object loads on your local computer. If you are not using
this performance object for any other application, disable this performance object by completing
the following steps:
1. Add the Disable Performance Counter (DWORD) registry entry with value equal to 1.
2. Add the DWORD registry entry to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\
services\ASP\Performance registry subkey.
For more information about disabling the performance object, see Microsoft Support Knowledge
Base article 2962962 http://support.microsoft.com/kb/2962962

Microsoft SQL Server agent: The configuration file of the MS SQL

Problem Detail data set cannot handle non-English ERRORLOG files
Symptom
The configuration file of the MS SQL Problem Detail attribute group cannot handle non-English
ERRORLOG files by default.
Solution
Update the koqErrConfig.ini file to resolve this issue. For example, for the French language,
update the koqErrConfig.ini file to reflect the following format:
[French_CI_AS]
Error=Erreur :
Severity=Gravité :
State=État:

For the English language, update koqErrConfig.ini the file to reflect the following format:
[SQL_Latin1_General_CP1_CI_AS]
Error=Error:
Severity=Severity:
State=State:

Note: If you are using the 32-bit Microsoft SQL Server agent, the koqErrConfig.ini file is saved
in the candle_home\TMAITM6 directory. If you are using the 64-bit SQL Server agent, the
koqErrConfig.ini file is saved in the candle_home\TMAITM6_x64 directory.

Microsoft SQL Server agent: Attributes do not allow non-ASCII input in

the Situation Editor
Symptom
Attributes do not allow non-ASCII input in the Situation Editor.

38 Troubleshooting for Performance Management Version 8.1 (On Premises):

Solution
None. Any attribute that does not include "(Unicode)" might support only ASCII characters. For
example "Attribute (Unicode)" will support unicode but "Attribute" without "(Unicode)" might
only support ASCII characters.

Multiple agents: High CPU usage

The following agents might have this problem:
v Oracle Database agent
v DB2 agent
v Microsoft Exchange Server agent
v Microsoft SQL Server agent
v Microsoft IIS agent
v Microsoft .NET agent
Symptom
The system is experiencing high CPU usage.
Solution
Agent process
View the CPU usage of the process:
v Oracle agent: KRZCMA process
v DB2 agent: kuddb2 process
v Microsoft Exchange Server agent: KEXCMA process
v Microsoft SQL Server agent: KOQCMA process
v Microsoft IIS agent: KQ7CMA process
v Microsoft .NET agent: KQECMA process
If CPU usage seems to be excessive, restart the monitoring agent.

Note: As the number of remote systems is increased, the CPU, memory, and network
utilization on the agent server also increase. A dedicated agent server might be added to
the environment to handle a large agentless monitoring environment.
Network cards
The network card configurations can decrease the performance of a system. Each stream
of packets that a network card receives (assuming that it is a broadcast or destined for the
under-performing system) must generate a CPU interrupt and transfer the data through
the I/O bus. If the network card in question is a bus-mastering card, work can be
offloaded and a data transfer between memory and the network card can continue
without using CPU processing power. Bus-mastering cards are 32 bit and are based on
PCI or EISA bus architectures.

Multiple agents: Log data accumulates too rapidly

The following agents might have this problem:
v Microsoft Exchange Server agent
v Microsoft Hyper-V Server agent
v Microsoft IIS agent
v Microsoft .NET agent
Symptom
Log data accumulates too rapidly.

Chapter 4. Agent 39
Solution
Check the RAS trace option settings. The trace option settings that are specified on the
KBB_RAS1= and KDC_DEBUG= lines generate large amounts of data.

Multiple agents: Situation does not trigger

The following agents might have this problem:
v Microsoft IIS agent
v Microsoft SQL Server agent
v Microsoft Hyper-V Server agent
Symptom
The situation does not trigger.
Solution
This problem can be caused when incorrect predicates are present in the formula that defines the
situation. For example, the managed object shows a state that normally triggers a monitoring
event, but the situation is not true because the wrong attribute is specified in the formula.
In the Formula tab, analyze predicates as follows:
1. Click the fx icon in the Formula area. The Show formula window is displayed.
a. Confirm the following details in the Formula area of the window:
v The attributes that you intend to monitor are specified in the formula.
v The situations that you intend to monitor are specified in the formula.
v The logical operators in the formula match your monitoring goal.
v The numeric values in the formula match your monitoring goal.
b. (Optional) Select the Show detailed formula check box to see the original names of
attributes in the application or operating system that you are monitoring.
c. Click OK to dismiss the Show formula window.
2. (Optional) In the Formula area of the Formula tab, temporarily assign numeric values that
immediately trigger a monitoring event. The triggering of the event confirms that other
predicates in the formula are valid.

Note: After you complete this test, you must restore the numeric values to valid levels so that
you do not generate excessive monitoring data based on your temporary settings.

Multiple agents: Situation is not displayed in the event tab

The following agents might have this problem:
v Microsoft SQL Server agent
v Microsoft IIS agent
Symptom
Situation is not displayed in the event tab even though the predicate was correctly specified.
Solution
Check the logs and reports.

40 Troubleshooting for Performance Management Version 8.1 (On Premises):

Node.js agent: URL filtering, incorrect URL can be displayed
Symptom
An incorrect URL can be displayed in the dashboard when you use URL filtering. URL filtering
can be specified in the runtime configuration file of the Node.js agent. This function can be used
to filter the URL path that is used for measurements. When you use URL filtering the dashboard
is expected to display the filtered URL, however it might continue to display the original
non-filtered URL.
Solution
If the dashboard does not display your filtered URL when you expect it to, the incorrectly
displayed URL can be ignored, your filter is still applied.

Node.js agent: No data in deep-dive Request Summary widget

Symptom
No data is displayed in the Request Summary widget of the deep-dive dashboard.

Note: You might also see an OutOfMemoryError message in the knj_std_output.log file similar to
the following example:

Exception in thread "com.ibm.log.cmd.LogCmdServer" java.lang.OutOfMemoryError: Java heap

space at com.ibm.oti.vm.VM.getClassNameImpl(Native Method) at
com.ibm.oti.vm.AbstractClassLoader.getPackageName(AbstractClassLoader.java:395) at
com.ibm.oti.vm.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:65) at
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:802) at
java.lang.ThreadGroup.uncaughtExceptionJVMDUMP013I Processed dump event "systhrow",
detail "java/lang/OutOfMemoryError". (ThreadGroup.java:796) at
java.lang.Thread.uncaughtException(Thread.java:1308)
Solution
The request frequency is too large to be contained within the default size of the Java Virtual
Machine (JVM). Increase the JVM heap size of the agent and then restart the agent.

Node.js agent: Y-axis labels not displayed in dashboard charts

Symptom
Y-axis labels are not displayed in the charts of the detailed dashboard.
Solution
This issue is caused by a limitation of the charting toolkit that is used for the dashboard charts.
After you wait for the agent to collect sufficient data from the Node.js application, the y-axis label
is shown correctly.

Node.js agent: Configuration file not generated after configuring and

starting the agent
Symptom
After configuring and starting the agent, the configuration file plugin_3000_conf.json is not
generated in the directory install_dir/lx8266/nj/bin/plugin/lib, where install_dir is the
installation directory of Node.js agent.
Cause The environment variable AGENT_BIN_DIR is not correctly set.
Solution
Correct the value of the environment variable AGENT_BIN_DIR by running the following commands
in install_dir/bin directory:

Chapter 4. Agent 41
 ./nodejs-agent.sh stop
unset CANDLEHOME
./nodejs-agent.sh start

Oracle Database agent: Some columns for monitored Oracle RDBMS

10g instance display 0
Symptom
A 0 (zero) is displayed in the following columns for the monitored Oracle RDBMS 10g instance:
v Unused Capacity column and the % Free column in the ASM Disk Group Capacity workspace
under the RDBMS subnode
v Unused Capacity column and the % Free column in the ASM Disk Capacity workspace under
the RDBMS subnode
Cause 0 is the value that is reported by the Oracle database. The value of the free_mb column in the
Oracle views, v$asm_disk and v$asm_diskgroup, is 0 if the value is queried from a database
instance. This problem exists for Oracle RDBMS 10g. For detailed information, see Oracle
metalink 294325.1.
The free_mb attribute value is the free space in an ASM DISKGROUP (V$ASM_DISKGROUP) or
in an ASM DISK (V$ASM_DISK).
Solution
Configure the agent and connect it to an ASM instance. The following correct values are
displayed under the ASM subnode:
v In the ASM Disk Group Capacity workspace, in the Unused Capacity column and % Free
column
v In the ASM Disk Capacity workspace, in the Unused Capacity column and % Free column

Oracle Database agent: Processes consume high CPU

Symptom
Oracle Database agent processes consume high CPU.
Cause Detailed RAS1 tracing might cause CPU and I/O overhead and impact the performance of the
monitoring agent.
Solution
Check whether the RAS1 tracing is set to a detailed option, for example, KBB_RAS1=ERROR
(UNIT:krz ALL). Restore the RAS1 tracing to the minimal level of KBB_RAS1=ERROR after problem
diagnosis is completed, or change the trace level to KBB_RAS1=ERROR (UNIT: krz ERROR), and
restart the Oracle Database agent.

Oracle Database agent: Memory usage for krzstart or krzclient

processes increases
Symptom
The memory usage of the Oracle Database agent krzstart or krzclient processes increases
continually when the Oracle Database agent instance has inactive database connections.
Solution
Configure Oracle Database agent instances with the Oracle database or Oracle instant client
version 11.1.0.6 or later.

42 Troubleshooting for Performance Management Version 8.1 (On Premises):

Oracle Database agent: Instances not listed in My Components
Symptom
When the Oracle Database agent is monitoring the Oracle Database servers remotely, and the OS
Agent does not run on the computer where the Oracle Database server runs, the Oracle server
instance is not listed in the ORACLE node of the My Components application.
Cause The Oracle Database agent uses the IP address of the computer where the Oracle Database
instance runs to correlate the runsOn relationship with the computer. However, the OS Agent is
not installed on this computer to provide the IP address.
Solution
Install the OS Agent on the computer where the Oracle Database server runs. Then, the Oracle
instance is listed correctly in the My Components application.

PHP agent: injection codes not removed after uninstalling the agent
Symptom
After you uninstall PHP agent, the WordPress agent plug-in installed by PHP agent is not
removed.
Cause Before you uninstall the agent, the injection codes were not successfully removed by running the
command install_dir/bin/lx8266/pj/lib/uninstall.instance_name.sh, or the command
uninstall.instance_name was not run before you uninstall the agent.
Solution
Remove the injection codes manually.
1. Remove the wordpress_dir/wp-content/plugins/wp-agent folder.
2. Remove the injection code from the wordpress_dir/wp-includes/load.php file. Here is the
code sample:
$insertTime="Tue Dec 30 02:41:10 UTC 2014";
$nowTime=exec("date -u");
$plgs= get_option( ’active_plugins’);
if(strtotime($nowTime) <= strtotime($insertTime)){
if(!$plgs||!in_array(’wp-agent/agent.php’, $plgs)){
array_push( $plgs, ’wp-agent/agent.php’ );
update_option( ’active_plugins’, $plgs );
}
}
3. Drop the following tables in the MySQL database:
v prefix_wp_stat_log
v prefix_req_stat_log

Python agent: injection codes not removed after uninstalling the agent
Symptom
After you uninstall Python agent, the injection codes are not removed.
Cause Before you uninstall the agent, the injection codes were not removed by running the command
install_dir/lx8266/pg/bin/uninstall.sh.
Solution
Go to the application folder and remove the injection codes manually.
1. Remove the kpg_project_id.py and kpg_project_id.pyc files.
2. Remove settings.py_kpg.bak and settings.py.project_id files.
3. Remove the line 'app_folder_name.kpg_project_id.KPG_Middleware' from
MIDDLEWARE_CLASSES in the settings.py file.

Chapter 4. Agent 43
 4. Remove the wsgi.py_kpg.py file.
5. Go to the Apache configuration folder, for example /usr/local/apache2/conf, and recover the
httpd.conf file by removing the following section from the file:
<Location /server-status>xxx</Location>

Ruby agent: Instance not discovered

You can create logical applications for displaying in the Application Performance Dashboard that include
your monitored Ruby on Rails applications.

If you have a Ruby application but do not see it in the list of available instances for adding to an
application, read the symptoms, possible cause, and solution.
Symptom
While attempting to add a Ruby instance to an application (instructions in Managing
applications), you do not see a started instance in the list.
The Ruby instance was added to an application but does not appear in the Application
Performance Dashboard.
Cause The Ruby instance was started with daemonized Ruby processes. For example, you started the
Ruby on Rails application using the rails server command with a -d argument.
Solution
Do not run Ruby applications in a daemon process.

SAP agent: Data collection stops or runs sluggishly

Symptom
Data collection stops or runs sluggishly on your SAP systems with Oracle databases.
Solution
Data collection problems might occur when the SAP program, RSDB_TDB, which collects the
Oracle statistics, does not work correctly. Too many data rows are stored in MONI. Collection
might stop or run sluggishly on busy systems. See SAP Notes: 591801, 713211.
To correct this problem, perform the following steps:
1. Have your SAP Administrator implement these SAP notes.
2. Run the specified program, RSORAUD0, with the recommended cleanup options.
3. Update the Oracle statistics manually through transaction DB02.
After you implement these steps, the number of rows that are returned to the agent is correct, the
volume of data in MONI does not increase, and agent data collection periods are normal.

SAP agent: ITM raised alert message in the SAP syslog

Symptom
Many of the following messages are in the SAP syslog: ITM raised alert 9911 -
Excessive data collected for R/3 Database Detail.
Solution
This alert message is generated by the monitoring agent and indicates that the number of
database detail rows that are generated by the ABAP data provider exceeded the configured
maximum number of rows to return.
The R/3_Data_Base_Detail data set returns a row for every object with more than 20 extents or
for every row that is in a space critical condition. Reduce the number of rows that are reported

44 Troubleshooting for Performance Management Version 8.1 (On Premises):

 by removing all space critical conditions in the database and reorganizing to less than 20 extents.
This reduction improves the database performance, and reduces the number of database detail
rows returned.

SAP agent: Incorrect parameters

Symptom
Incorrect parameters:
v Incorrect SAP host name, Gateway host name, Gateway port, user ID, password, or client
v SAP user specified does not exist
v SAP user password is incorrect
v SAP user is locked (disabled)
Solution
Change the parameter that is in error and confirm with your SAP Administrator that your
parameters are correct. These problems are based on the RFC connection parameters.

SAP agent: Managed system names that show up under SAP agent in
the dashboard are incorrect
Symptom
The managed system names that show up under SAP Agent in the dashboard are incorrect.
Your MSN is not in this form: SID-SAP-Host:mySAP
Your MSN is some other string, and all instances of the monitoring agent show up under this one
managed system name, for example:
agent host
SAP agent
managed system name
SAP
TV2-amsaix25_TV2_00:Ins
TV2-amsaix25:Sys
TV2-amshp8_TV2_22:Ins
TV3-amsaix26_TV3_01:Ins
TV3-amsaix26:Sys
TV3-amssol19_TV3h_10:Ins
Solution
Check to see whether CTIRA_HOSTNAME is set, either globally or in the agent configuration
file, and if the MSN you see in the dashboard under the SAP agent is the value of
CTIRA_HOSTNAME. If so, do not set the environment variable CTIRA_HOSTNAME for the SAP
agent. The monitoring agent cannot properly create the managed system names when this
environment variable is set. If you need to set CTIRA_HOSTNAME for other monitoring agents,
set the variable in the agent configuration file instead of setting it globally.

SAP agent: Syslog messages and alert messages do not show

correctly in non-English languages
Symptom
Text strings, such as syslog messages and alert messages do not show correctly in non-English
languages. This problem is more likely to occur with double-byte languages.
Solution
Set the SAP_CODEPAGE environment variable.
Manually edit the RAS1 trace logging parameters, using the following steps:
1. Specify RAS1 trace options by changing trace parameters in a control file.

Chapter 4. Agent 45
 2. Open the trace options file.
v On Windows systems:
install_dir\TAMITM6_x64\KSAENV
v On UNIXsystems:
export KBB_RAS1=’ERROR (UNIT:ksa ALL) (UNIT:kra ALL)’
3. Edit the line that begins with KBB_RAS1= to set trace logging preferences.
For example, if you want detailed trace logging, set the Maximum Tracing option:
v On Windows systems:
KBB_RAS1=ERROR (UNIT:ksa ALL) (UNIT:kra ALL)
v On UNIX systems:
export KBB_RAS1=’ERROR (UNIT:ksa ALL) (UNIT:kra ALL)’
4. Edit the line that begins with KBB_RAS1_LOG= to manage the generation of log files:
v Edit the following parameters to adjust the number of rolling log files and their size.
– MAXFILES: the total number of files that are to be kept for all startups of a program.
When this value is exceeded, the oldest log files are discarded. Default value is 9.
– LIMIT: the maximum size, in megabytes (MB) of an RAS1 log file. Default value is 5.
IBM Software Support might guide you to modify the following parameters:
v COUNT: the number of log files to keep in the rolling cycle of one program startup. Default
value is 3.
v PRESERVE: the number of files that are not to be reused in the rolling cycle of one program
startup. Default value is 1.
The KBB_RAS1_LOG parameter also provides for the specification of the log file directory, log file
name, and the inventory control file directory and name. Do not modify these values or log
information can be lost.
5. Restart the monitoring agent so that your changes take effect.

SAP agent: mySAP application server is not discovered by the SAP

agent
Symptom
A mySAP application server is not discovered by the SAP agent.
Solution
Check to make sure that there is at least one (preferably two) dialog process on that instance. The
SAP agent requires a dialog work process in which to run the agent supplied ABAP that monitors
the instance.

SAP agent: mySAP server does not start when using port 3661
Symptom
The mySAP server does not start when using port 3661.
Solution
Change the current setting of KDC_FAMILIES=$NETWORKPROTOCOL$ to
KDC_FAMILIES=HTTPS:0 in both of the SAP agent.configuration files (*.config and *.ini).

46 Troubleshooting for Performance Management Version 8.1 (On Premises):

SAP agent: Non-ASCII characters that are entered into the
configuration window do not show up or are not the correct characters
Symptom
Non-ASCII characters that are entered into the configuration window for the monitoring agent do
not show up or are not the correct characters.
Solution
Enter only ASCII characters into these fields.

SAP agent: Attributes do not allow non-ASCII input when you are
creating an eventing threshold
Symptom
When you are creating an eventing threshold, attributes do not allow non-ASCII input.
Solution
None. Any attribute that does not include “(Unicode)” might support only ASCII characters. For
example, “Attribute (Unicode)” supports Unicode but “Attribute” without “(Unicode)” might
support ASCII characters only.

SAP agent: Return code 8 or return code 12 occurs on the main import
step
Symptom
Return code 8 or return code 12 occurs on the main import step.
Solution
This return code is related to DYNPRO format incompatibility or export/import (specifically table
EUDB) incompatibility. These errors occur if the R3trans program is old or the Basis support
package maintenance is low.
Upgrade R3trans program or Basis support level SAP Notes that describe the minimum R3trans
and Basis support packages are documented in the following OSS Notes: 330267, 454321, 743155.

SAP agent: Agent does not start if RFC library is not copied to the
correct path
Symptom
If you do not copy the RFC library to the correct path, the agent does not start and the following
error is reported in the agent log:
Error in agent log: /opt/ibm/apm/agent/lx8266/sa/bin ksaagent: error while loading
shared libraries: libsapnwrfc.so: cannot open shared object file: No such file or
directory
Solution
Information about copying the RFC library to the correct path is as follows:
v For Windows environment:
Copy the RFC library at %CANDLE_HOME%/TMAITM6_x64 for a 64-bit agent on a 64-bit Windows
computer.
v For non-Windows environment:
Copy the SAP NetWeaver RFC SDK V7.20 libraries into the <<INSTALL_DIR>>/<<intrp>>/sa/lib
where

Chapter 4. Agent 47
 – INSTALL_DIR is SaaS Agent installation directory
– intrp is lx8266 for Linux 64-bit and aix526 for AIX® 64-bit

SAP agent: TSV_TNEW_PAGE_ALLOC_FAILED ABAP runtime error

Symptom
The TSV_TNEW_PAGE_ALLOC_FAILED ABAP runtime error occurred in SAP because of huge data
volume.
Solution
To resolve this problem, you must increase the ABAP heap size. To increase the heap size in SAP,
complete the following steps:
1. Run T-Code SE38.
2. In the Program name text box, enter RSMEMORY, and execute.
3. Under the Other parameters section, set the value for abap/heap area dia parameter, and
then, click Copy.

Note: The recommended size of ABAP heap size is 2 GB. However, you can increase the size.

SAP agent: Runtime errors in relation to transport on the SAP system

Symptom
Runtime errors in relation to transport on the SAP system.
Solution
When you import the transport on the SAP system, you must not start the SAP agent instance
that is configured to monitor that SAP system.
Before you delete the transport from the SAP system, you must stop the SAP agent instance that
is configured to monitor that SAP system.

SAP agent: Same version installation warning during command-line

installation on UNIX
Symptom
During the command-line installation on UNIX, you choose to install a component that is already
installed, and you see the following warning:
WARNING - you are about to install
the SAME version of "component"

where component is the name of the component that you are attempting to install.
Solution
You must exit and restart the installation process. You cannot return to the list where you selected
components to install. When you run the installer again, do not attempt to install any component
that is already installed.

SAP agent: Tivoli Monitoring alert 9912 and SAP syslog message
Symptom
You receive Tivoli Monitoring alert 9912: CCMS alerts collection did not complete, SAP syslog
message ??ccms_rfc_error????, or both of these alerts. You might also receive ABAP dumps from
the /IBMMON/ITM_ALERTS and /IBMMON/ITM_CCMS_ALERTS function modules that are provided by the
SAP agent.

48 Troubleshooting for Performance Management Version 8.1 (On Premises):

Solution
Tivoli Monitoring Alert 99112 typically indicates a problem with the mySAP CCMS rather than
with the SAP agent. Review the syslog message to help troubleshoot the problem in the SAP
CCMS environment. Verify that CCMS performs properly.
Contact IBM Software Support to get instructions on how to reduce or prevent the monitoring
agent ABAP dumps while CCMS problems are investigated.

SAP agent: Transaction not valid error when you run the
/IBMMON/ITM_* transactions
Symptom
When you run the /IBMMON/ITM_* transactions, you get an error that indicates that the
transaction is not valid.
Solution
Preface all /IBMMON/ITM_* transactions with /n or /o.

SAP agent: Values shown are not correct when you edit the
configuration for an existing monitoring agent
Symptom
When you edit the configuration for an existing monitoring agent, the values that are shown are
not correct.
Solution
The original configuration settings might include non-ASCII characters. These values were stored
incorrectly and result in the incorrect display. Enter new values by using only ASCII characters.

SAP agent: Monitoring agent does not start in an AIX V6.1

environment
Symptom
The SAP agent does not start in an AIX V6.1 environment that has a Technology Level (TL) less
than TL8 and a time zone of Asia/Calcutta.
Solution
To resolve this problem, complete one of the following steps:
v Change the time zone to TZ=CST6CDT, and then start the agent.
v Upgrade the AIX 6.1 environment to include TL8 level, and then start the agent.

SAP agent: Language errors

You can use the SAP agent: Language errors topic to troubleshoot errors that relate to the desired
language not appearing on the SAP agent. For example, you can find solutions to problems that relate to
desired language not appearing on the mySAP configuration panel.

SAP agent: Value lists are shown in English in the mySAP

configuration panels
Symptom
Value lists such as report names, monitors, and monitor sets are shown in English in the
following mySAP configuration panels:
v Maintain Default Sample Periods

Chapter 4. Agent 49
 v Maintain Log File Names
v Maintain ITM Managed Groups Definitions
v Select CCMS Monitor Sets and Monitors
Solution
This outcome is expected. These value lists are presented in English only regardless of the SAP
logon language.

SAP agent configuration panels on a mySAP system does not show

text in a multi-byte language
Symptom
Text in the SAP agent configuration panels on a mySAP system does not show in the multi-byte
language with which you logged on to mySAP.
Solution
Ensure that you logged on to the mySAP system or SAPGUI by using a supported language. See
the “Language” section in the “Configuring the SAP agent” in the IBM Application Performance
Management (SaaS) Knowledge Center for a list of the languages that the SAP agent supports.

Tomcat agent: Tomcat server not monitored

Symptom
In case of certain configuration errors, the Tomcat server is not monitored after you have
configured the Tomcat agent.
Solution
Ensure that the agent is started. To check the status of the agent, change to the install_dir/bin
and enter the following command:
./tomcat-agent.sh status

To start the agent, enter the following command:

/install_dir/bin/tomcat-agent.sh start instance_name, where instance_name is the name of the
instance that you have set during configuration.

Ensure that the JMX remote is enabled in the Tomcat server. For instructions, see Enabling JMX
Remote (https://tomcat.apache.org/tomcat-6.0-doc/monitoring.html#Enabling_JMX_Remote).
If, after these checks, the server is still not monitored, rerun the configuration for the agent,
ensuring that you enter correct information.

VMware VI agent: Dashboard errors

You can use the VMware VI agent: Dashboard errors topic to troubleshoot errors that relate to the
VMware VI agent dashboard. For example, you can find solutions to problems that relate to the ESX
Server dashboard.

ESX Server component events are not displayed

Symptom
Events for the ESX Server component are not automatically displayed on the dashboard.
Cause The corresponding ESX Server is not added to the application.
Solution
For instructions on adding, removing, and editing applications, see:

50 Troubleshooting for Performance Management Version 8.1 (On Premises):

 v The “Managing applications” topic under the “Configuring” topic in the Application
Performance Management help, if you are using the lightweight infrastructure.
v The “Managing applications” topic in the SmartCloud Monitoring - Application Insight help, if
you are using the Tivoli® Monitoring infrastructure.

Attribute groups for the ESX Server component are not displayed
Symptom
Attribute groups for the ESX Server component are not displayed at the VMware Virtual
Infrastructure level in the Attribute Details tab.
Solution
A solution is not currently available.

Situations in the Severity column display incorrect status

Symptom
The situations that are highlighted in green color display the “Unknown” status in the Severity
column.
Solution
A solution is not currently available.

Group widget titles are not globalized

Symptom
For the globalized dashboards, some group widget titles are displayed in English.
Solution
A solution is not currently available.

Columns in some group widgets are not highlighted

Symptom
In some group widgets, the column names that you can click to open another page are not
highlighted or underlined.
Solution
To know whether you can click a column name to open the detailed dashboard, complete any
one of the following steps:
v Hover the mouse pointer over the column name. The change in the mouse pointer indicates
that you can click the particular column name.
v See the online help for the group widget. The online help specifies whether you can click a
particular column name to open the detailed dashboard.

WebSphere Applications and Microsoft .NET agents: Missing

transaction data in topology views
Symptom
If nodes or transactions are missing from topology views on the Performance Management
console, you can enable metrics logging in the Transaction Framework Extension of the agent.
Solution
Enable metrics logging in the RAS1 log file either by modifying the file directly or, to avoid a
restart of the agent, by modifying the trace settings through the Service Console web browser.
v Edit the KBB_RAS1 environment variable by modifying the environment configuration file.
1. Open the agent environment configuration file in a text editor.

Chapter 4. Agent 51
 Linux systems: From the install_dir/config directory, open the yn.ini file for
a WebSphere Applications agent.
Windows systems: From the install_dir\TMAITM6_x64 directory, open the
KYNENV file for a WebSphere Applications agent or the KQEENV file for a Microsoft .NET
agent.
2. Set the KBB_RAS1 environment variable to ERROR (COMP:TFE ERROR METRICS)
3. Restart the agent to implement your changes.
Linux systems: Change to the /install_dir/bin directory, and enter the
command ./was-agent.sh start for a WebSphere Applications agent.
Windows systems: Change to the install_dirbin directory, and enter the
command was.bat start for a WebSphere Applications agent or the dotnet.bat start
command for a Microsoft .NET agent.
v Edit the KBB_RAS1 environment variable through the Service Console web browser.
1. From a web browser, open the Service Console by entering the URL http://
agent_hostname:1920 in the address field of the browser.
2. From the service index, click the link below the agent whose trace settings you want to
modify.
3. Enter the user ID and password of the agent when prompted.
4. In the command field, enter ERROR (COMP:TFE ERROR METRICS)
The RAS1 log files are updated regularly with logCount() entries for various metrics.
If you see too few incoming EventServer events in the log, your agent may not be reporting
transactions properly.
If you see high number of drop counts or large queue sizes in the log, your agent may have a
performance issue or one of the services might be down.
To view logCount() messages more frequently, increase the frequency with which metrics are
logged by the agent.
1. Open the Transaction Framework Extension configuration file, tfe.cfg, in a text editor.
Linux systems: From the install_dir/localconfig/yn directory for a WebSphere
Applications agent, open the tfe.cfg file.
Windows systems: From the install_dir\localconfig\yn directory for a
WebSphere Applications agent or the C:\IBM\APM\qe\config directory for a Microsoft .NET
agent agent, open the tfe.cfg file.
2. Decrease the value of the DIAGNOSTIC_SLEEP_MINS parameter to log metrics more frequently.
The default value is 30 minutes.

52 Troubleshooting for Performance Management Version 8.1 (On Premises):

Chapter 5. Component
Review the component troubleshooting symptoms and solutions for help with issues, for example,
browser issues.

Getting Started links blocked in Chrome

Symptom
When using the Chrome v21 or later browser, you are blocked from opening the links in the
Getting Started page.
Cause Starting with Chrome v21, any unsecured (HTTP rather than HTTPS) or unauthenticated browser
links are blocked.
Solution
Click the shield tool and select “Load unsafe script”" to continue to the page. Alternatively,
right-click the link and select “Open in new tab” or “Open in new browser” to continue to the
page.

Network connection error in the Performance Management console

Symptom
You receive a Network Connection Error in the Performance Management console.
Cause
The window where you are expected to reauthenticate is shown 3 minutes before the session
expires. You must reauthenticate within 3 minutes or a network error is shown and you are
logged out.
Solution
You must log back in to start a new session.

Connection to the Performance Management console fails in Firefox

Symptom
The connection to the Performance Management console fails with the Firefox browser.
Cause
When the connection fails, you encounter the following error:
An error occurred during a connection to xx.xx.xx.xx:9443.
Issuer certificate is invalid.
(Error code: sec_error_ca_cert_invalid)
Solution
You can choose a solution from one of the following procedures:
Complete this procedure:
1. Check the web browser settings to manage the certificates. To open the Config page, in the
address bar of the browser, enter the following:
about:config
2. Change the Firefox settings as shown here:
security.use_mozillapkix_verification = true
Alternatively, complete this procedure:

53
 1. Use custom certificates that are signed by an official CA authority, for example, GeoTrust. See
the“Configuring a CA custom certificate” topic in the IBM Monitoring Knowledge Center.

54 Troubleshooting for Performance Management Version 8.1 (On Premises):

Chapter 6. Dashboard
Review the dashboard troubleshooting symptoms and solutions for help with issues in the Application
Performance Dashboard.

Application Dashboard takes a long time to display

Symptom
Any discovered managed systems are shown in an application named My Components in the
Application Performance Dashboard. After you click My Components in the Applications section
of the navigator, it takes a long time for the My Components Application Dashboard to display.
Cause When a large number of components is stored in the application repository of Monitoring
Infrastructure Node, enabling the My Components application might cause performance issues.
Solution
You can disable the My Components application to improve performance if your application
repository has many components. Complete the following steps to disable the My Components
application:
1. On the system where the Performance Management server is running, change to the following
directory:
/opt/ibm/wlp/usr/servers/apmui/apps/customCfg
2. Open apmui.cfg in a text editor and set ENABLE_MY_COMPONENTS to false:
ENABLE_MY_COMPONENTS=false
3. Run the following command for the changes to take effect:
./apmcfg.sh -o APM_UI_port -u APM_admin -p APM_password

where
APM_UI_port is 9443 for HTTPS (or 8080 for HTTP)
APM_admin is a user ID with administrative authority (apmadmin is the default)
APM_password is the password that is associated with the user ID

Dashboard error occurred while loading data

If the dashboard for the selected data source displays an error message instead of metrics, the operating
system user credentials for the monitoring agent might need to be added to the Access Authorization
Group Profile (AAGP).
Symptom
After you respond to a lock indicator by entering the operating system user credentials for the
data source, data can be sent to the Monitoring Infrastructure Node dashboards. If you enter a
non-root user ID, such as apmadmin, no data is displayed on the dashboard. Instead, you get a
message that an error occurred while loading data sources.
Diagnosis
By default, monitoring agents are configured to require root user credentials to access Agent
Service Interface (ASI) transactions.
Solution
To change this default, you can customize the IBM Tivoli Monitoring Access Authorization Group
Profile (AAGP) to specify user IDs to be included in the Administrative group, so that the user
has access to all ASI transactions. The non-root user must have a valid user ID on the target
system that they can use to log on and does not require root authority or membership in any
specific group. To customize the agent AAGP, take these steps:

55
 1. Log on to the ASI using the root ID (such as http://agentIPaddress:51920).
2. Select Service Interface Request.
3. Enter ListAAGP in the text box and click Submit the Request to get the default AAGP
specification.
4. Copy the default specification that is displayed in the Agent Response Payload box, and paste
into a text editor. Sample payload:
<AAGP>
<AAGROUP>
<GROUPNAME>Administrative</GROUPNAME>
<CLI>ExecCommand</CLI>
<REFLEXAUTO>ExecAction</REFLEXAUTO>
<SIAPI>AAGP</SIAPI>
SIAPI>AgentInfo</SIAPI>
SIAPI>AttrList</SIAPI>
SIAPI>CnfgCommand</SIAPI>
SIAPI>CnfgControl</SIAPI>
...
5. Edit the text to add the smadmin to the AD (administrative) group, as shown in the following
example:
<ASSIGN>OP</ASSIGN>
</AAUSER>
<AAUSER>
<ID>root</ID>
<ASSIGN>AD</ASSIGN>
</AAUSER>
<AAUSER>
<ID>smadmin</ID>
<ASSIGN>AD</ASSIGN>
</AAUSER>
</AAGP>
6. To reconfigure the agent without restarting it, change the first line of the file to <AAGP
REFRESH="Y">.
7. Save the text file (for example AAGPupdate.txt).
8. Return to the Service Interface Request page and paste the updated AAGP specification from
the text file to the input box and click Submit the Request.
After the “Request completed successfully” response is displayed, the agent has a new file named
lz_aagpcnfg.txt in the $CANDLEHOME$/localconfig/lz directory. It has the configuration necessary
for the user (such as smaiuser) to access ASI transactions and for the Performance Management
server to connect to the agent and retrieve data.

Dashboards show no data, partial data, or incorrect data from the

monitoring agent
If you do not see data in the Application Performance Dashboard for a particular monitoring agent, check
for time out errors and ensure you are using NTP (Network Time Protocol) on the monitored system.

Additionally, review the agent-specific solutions in this topic.

Symptom
In the Application Performance Dashboard ( Performance > Application Performance
Dashboard), you see no data, partial data, or incorrect data for a specific agent or agents after
selecting an item in the Groups or Instance section of the navigator.
Diagnosis and Solution
This problem can occur when the monitoring agent's clock is not synchronized with your

56 Troubleshooting for Performance Management Version 8.1 (On Premises):

Application Performance Management console. If the agent's clock is set ahead of the
infrastructure clock, the agent might not download the latest configuration from the
infrastructure.
Use NTP (Network Time Protocol) on monitored systems to ensure the time on the system
running the agent is accurate. Time zones must be set correctly for the time zone they are in, such
as UTC-3 for Brasilia and UTC +9 for Seoul.
If your problem was not solved by using NTP, review the logs to investigate further.
All monitoring agents:
On the system where the monitoring agent is installed, review the most recent agent log
for any command timeout messages. Agent log names are
host_name_pc_agent_instance_name_timestamp.log
where
host_name is the name of the system on which the agent is installed
pc is the two-character product code, such as kj or km
agent_instance_name is the name that was given to the instance during agent
configuration
timestamp is the time and date when the log was saved
and the path is
install_dir/logs
install_dir\TMAITM6_x64\logs
If you see that command timeout messages appear frequently, the agent's shell script might
be taking too long to return with information and is timing out. The time out period is
controlled in the pc_agent_instance_name.config file by the CDP_DP_SCRIPT_TIMEOUT
environment variable, which has a default value of 30 seconds. Set
CDP_DP_SCRIPT_TIMEOUT and CDP_DP_REFRESH_INTERVAL to higher values to give the scripts
enough time to complete successfully.
PHP agent:
Your Apache version might be lower than the supported version. Check the Apache
HTTP Server httpd.conf configuration file and ensure that both the mod_status and
ExtendedStatus On options are enabled. For more information, see Configuring the PHP
agent.
Ensure the PHP agent plug-in in WordPress is activated. For more information, see
Configuring the PHP agent.
In the WordPress List widget, you might see PHP or a different text string listed under
WordPress Document Root. This text string does not follow the standard format for the
WordPress Document Root. This error might be caused by a PHP module exception. To
ensure the PHP modules are working, execute the php -m command. Correct any modules
with warnings or exceptions.
Ruby agent:
On the system where the monitoring agent is installed, review the
km_agent_instance_name_script.log for any entries with CMD=CONNECTION that start with
app_name#app_root_directory#IP_address#port_number#pid#mem.
Each line following the timestamp corresponds to a different Ruby application. For
example, if a user has an application named “MongoStore” with root directory
/root/Ruby/mongodb_store, one of the rows might look like this example:
MongoStore#/root/Ruby/mongodb_store#9.42.14.228#4444#20522#4959124
9.42.14.228 corresponds to the IP address of the system that the application is on
4444 corresponded to the application's listening port number

Chapter 6. Dashboard 57
 20522 corresponds to the product ID of the application
4959124 corresponded to the system's memory
If the line starts with the pound sign (#) instead of the app_name, you must reconfigure
the Ruby agent instance to include the Ruby runtime binary location.

Dashboard shows no events or event status

If you can see metrics from a data source but no severity indicator is displayed in the Application
Performance Dashboard Events tab, or no threshold events are being reported, you might need to restart
the monitoring agent.
Diagnosis and Resolution
The monitoring agents for Performance Management server require a certain version number, see
the Dependencies topic in the IBM Monitoring Knowledge Center. If you have an earlier version
of the monitoring agent installed, you must stop and restart the agent before it can send events to
the Performance Management server. You must also restart the agent after updating the
thresholds or historical data collection before you can see the effect of your changes.
If the thresholds are for the Monitoring Agent for Ruby, edit the application to confirm that
Enable Ruby App Situations is set and the instances that you want to monitor are selected. For
more information, see install/min/min_admin_managing_apps.dita and the Ruby agent note after
step 5 in the “Managing Applications” topic in the IBM Monitoring Knowledge Center.

Historical data doesn't change after you adjust the time selector
Some of the dashboard widgets show metrics that are based on a time range. You can adjust the time
selector for widgets whose values are derived from historical data. If you adjusted the time but the
widgets do not change to show the new time range, adjust the time for All Applications.
Symptom
In the displayed dashboard, you adjusted the time selector from the default Last 4 Hours or other
value but the historical widgets are not updated to the chosen time range.
Solution
Open the time selector, choose the time range, and select All Applications. The All Applications
option applies the time range to all defined applications in your monitored environment.

58 Troubleshooting for Performance Management Version 8.1 (On Premises):

Chapter 7. Eventing thresholds
Review the eventing thresholds troubleshooting symptoms and solutions for help with issues in the
Threshold Manager.

Threshold events have unknown severity after agent recycle

If the monitoring agent is stopped and restarted, the event severity information is lost and any new
events show a severity of Unknown.

Thresholds are not applied to the monitoring agent

Symptom
After a new threshold is created for a data source in the Threshold Manager, the threshold is
never applied to the monitoring agent.
Cause This problem can occur when the monitoring agent's clock is not synchronized with your
Application Performance Management console. If the agent's clock is set ahead of the
infrastructure clock, the agent might not download the latest configuration from the
infrastructure.
Solution
Use NTP (Network Time Protocol) on monitored systems to ensure the time on the system
running the agent is accurate. Time zones must be set correctly for the time zone they are in, such
as UTC-3 for Brasilia and UTC +9 for Seoul.

59
60 Troubleshooting for Performance Management Version 8.1 (On Premises):
Chapter 8. Performance Management Cognos Reports
There are four predefined historical reports available in Performance Management. The reports are based
on data collected by the Response Time Monitoring Agent. Review the troubleshooting symptoms and
solutions for help with report issues.

Reports: Charts in PDF Reports in Firefox are Pink

Symptom
When you view the Compare Application Performance over Two Time Periods, or Compare
Performance of Multiple Application reports in PDF format in Firefox, you experience formatting
issues. Some of charts in these reports are pink.
Solution
If you experience this problem, choose one of the following alternatives:
v Instead of viewing the PDF report embedded in Firefox, download the PDF report to a
directory in your system. Then, open the PDF report from that directory with a PDF reader.
v View the report in other supported formats, for example, HTML or XLS.
v Use an alternative browser, this problem is not observed in any other supported browsers, for
example, Internet Explorer, Chrome, or Safari.

Reports: Y-axis shows negative values

Symptom
Historical reports that are based on the Response Time Monitoring Agent return no data, the
y-axis range or scale shows negative values.
Solution
This is a known issue. It does not occur when the report returns data.

Reports: Y-axis shows duplicate values

Symptom
Historical reports that are based on the Response Time Monitoring Agent, show duplicate data
points on the y-axis.
Solution
This occurs because the data points are formatted to two decimal points, whereas the actual
numbers might require three decimal places. For example, if the numbers are, 0.00 and 0.005, then
these numbers are formatted to two decimal places and as a result appear as duplicates.

Reports: The Close and Cancel buttons shown on Reports login

screen do not work
Symptom
In the APM UI, when you click Work with Reports, the login window is displayed, you cannot
return to the previous APM UI window by clicking either the Cancel or Close buttons.
Solution
No action is expected when you click Close or Cancel.

61
Reports: Application names are inconsistent in the All My Applications
report
Symptom
When a single Response Time Monitoring Agent application is mapped to multiple Performance
Management applications, only a single (randomly selected) custom Performance Management
name is displayed in the All My Applications report. The data that is displayed is correct, but
application name labels differ for different charts in the report.
Solution
This issue is known. If you select an application in APM UI and then view a report that is based
on this application, the data is displayed correctly.

Reports: Application Performance Usage report returns:

XQE-GEN-0018
Symptom
In the Application Performance and Usage report, on the Select Key Transaction for Application
prompt page, when you select one or more transactions, the following error message is displayed:
XQE-GEN-0018
Query Service internal error has occurred, please see the log for details.
Solution
Wait for weekly aggregation to run, and run the report again. The Application Performance and
Usage report displays data for an application that is based on the weekly aggregated table. When
the weekly aggregated table contains no transactions for the application, for example, if weekly
aggregation did not run yet, the report fails to get data and returns an error.

Reports: Logging out in APM UI generates: Error 403:

AuthenticationFailed
Symptom
In the APM UI, when you click Log out after previously accessing a Cognos report from the same
browser, the following error is displayed:
Error 403: AuthenticationFailed
Solution
This is a known design issue.

62 Troubleshooting for Performance Management Version 8.1 (On Premises):




Printed in USA