Java Monitoring and

Diagnostic Tooling

Iris Baron
IBM Java JIT on System Z
[email protected]

Session ID: 16182

Insert
Custom
Session
QR if
Desired.
 Java Road Map
Java 7.0 Java 8.0**
Language Updates
• Support for dynamic languages • Language improvements
Java 5.0 Java 6.0 • Improve ease of use for SWING • Closures for simplified fork/join
• New Language features: • Performance Improvements
• New IO APIs (NIO2)
• Autoboxing • Client WebServices Support
• Java persistence API
• Enumerated types
• Generics • JMX 2.x and WS connection for
• Metadata JMX agents
• Language Changes

6.0

7.0
5.0

EE 5 EE 6.x

04 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

IBM Java7R1

>= 20 platforms
• Improvements in
18 platforms

20 platforms

SE601/ 7.x
WAS WAS WAS • Performance
SE 6.0

WAS
SE 5.0

6.0 6.1 7.0 • RAS

8.5 • Monitoring
• zEC12™ Exploitation
• zEDC for zip acceleration
• SMC-R integration
IBM Java Runtimes • Transactional Execution
• Runtime instrumentation
IBM Java 5.0 (J9 R23) IBM Java 6.0 (J9 R24) IBM Java 6.0.1/Java7.0 • Hints/traps
• Improved performance • Data Access Accelerator
• Improvements in (J9 R26)
• Generational Garbage Collector • Performance • Improvements in
• Shared classes support • Serviceability tooling • Performance IBM Java7.0SR3
• New J9 Virtual Machine • Class Sharing • GC Technology • Improvements in
• New Testarossa JIT technology • XML parser improvements
• z196™ Exploitation • Performance
• First Failure Data Capture • z10™ Exploitation • OOO Pipeline • zEC12™ Exploitation
• Full Speed Debug • DFP exploitation for BigDecimal • 70+ New Instructions • Transactional Execution
• Hot Code Replace • Large Pages • JZOS/Security Enhancements • Flash 1Meg pageable LPs
• Common runtime technology • New ISA features • 2G large pages
• ME, SE, EE • Hints/traps

2
**Timelines and deliveries are subject to change.
 Linux on System z and Java7SR3 on zEC12
Linux on System z - Multi-Threaded 64 bit Java Workload 16-Way
~12x Improvement in Hardware and Software

160
zEC12 SDK 7 SR3
Aggressive +
140
Normalized Throughput

LP Code Cache
zEC12 SDK 7 SR1
120

100 z196 SDK 7 SR1

80
z196 SDK 6 SR9
60

40 z10 SDK 6 SR4

20
z9 SDK 5 SR4
NO (CR or Heap LP)
0
1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
Threads

(Controlled measurement environment, results may vary)

~12x aggregate hardware and software improvement

comparing Java5SR4 on z9 to Java7SR3 on zEC12
• LP=Large Pages for Java heap
• CR=Java compressed references
• Java7SR3 using -Xaggressive + 1Meg large pages
3
 WAS on zLinux
Aggregate HW, SDK and WAS Improvement:
WAS 6.1 (Java 5) on z9 to WAS 8.5 (Java 7) on zEC12

J9R27 J9R27

(Controlled measurement environment, results may vary)

4x aggregate hardware and software improvement comparing

WAS 6.1 Java5 on z9 to WAS 8.5 Java7 on zEC12

4
 IBM Operational Decision Manager

IBM Operational Decision Management zEC12 16-way

1.4
1.3
1.2
1.1
(Normalized to IBM Java 7 SR4)

1
0.9
0.8
Throughput

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
IBM Java 7 IBM Java 7R1

(Controlled measurement environment, results may vary)

19% improvement to ODM with IBM Java 7R1 compared to IBM Java 7
5
 Java Monitoring and Diagnostic Tooling
Agenda

• IBM Monitoring and Diagnostic Tools for Java

– Why use the tools?
– Where to get the tools?
• IBM Recommended Java Troubleshooting Tools
– Health Center
– Garbage Collector and Memory Visualizer
– Memory Analyzer
• Summary

6
 Java Monitoring and Diagnostic Tooling
Why use the IBM Tools?
Possible problems:
• Tools simplify troubleshooting problems : 
Application coding errors

• IBM provides a free unified suite of tools 

Environment variables
to understand different aspects of Java 
Performance tuning
applications 
Configuration problems

• Fully IBM supported

·

• Tools provide visualizations, analysis and recommendations

Fixing problems ...

... is much easier with the right tool for the job!

7
 Java Monitoring and Diagnostic Tooling
Where to get the IBM Tools?

• IBM Support Assistant

A free application available at:
http://www.ibm.com/software/support/isa

• Eclipse Market Place

Tools available to install directly into Eclipse

8
 Java Monitoring and Diagnostic Tooling
What is IBM Support Assistant?

IBM Support Assistant (ISA) is a free application that :

• Provides the “toolbox” in which analysis and diagnostic tools reside

Over one hundred “add-ons” available for various IBM products

• Provides Serviceability Tools across product families

Simplifies software support

• Provides Search feature to query IBM and non-IBM knowledge banks

• Not a monitoring tool

9
 Java Monitoring and Diagnostic Tooling
ISA Workbench – Diagnostic Tools

10
 Java Monitoring and Diagnostic Tooling
What is IBM Support Assistant?

ISA Workbench 4.1

• Eclipse-based client

• Workbench is installed on each desktop (single user)

• Collect and organize diagnostic data (logs, traces, etc.)

• Find and use Problem Determination tools

• Search and browse support-related information

about IBM products

11
 Java Monitoring and Diagnostic Tooling
What is IBM Support Assistant?

ISA 5.0 Team Server

• Server-based model
• Install once - shared by many team
members via browser
• Web 2.0 browser interface
• Remote execution of PD tools
• Off-load analysis processing
• Collaboration on PD
• Case Management
• Tool Management
• Single-user option available

12
 Java Monitoring and Diagnostic Tooling
Agenda
• IBM Monitoring and Diagnostic Tools for Java
– Why use the tools?
– Where to get the tools?

• IBM Recommended Java Troubleshooting Tools

– Health Center
– Garbage Collector and Memory Visualizer
– Memory Analyzer

• Summary

13
 Java Monitoring and Diagnostic Tooling
Health Center

Motivating questions:
• What is my JVM doing? Is everything ok?
• Why is my application running slowly?
• Why is it not scaling?
• Am I using the right options?

14
 Java Monitoring and Diagnostic Tooling
Health Center

• Live monitoring tool with very low overhead (< 1%)

• Suitable for all Java applications running on IBM’s JVM
• Provides insight into your application behaviour with visualization
• Diagnoses potential problems with recommendations
• Powerful API allowing embedding of Health Center into other
applications

15
 Java Monitoring and Diagnostic Tooling
Health Center
Health Center provides visualization and monitoring in the
following application areas:
• Method profiling
• Lock analysis
• Garbage Collection
• Threading
• Memory Usage
• System environment
• Java class loading
• Object Allocations
• File I/O

16
 Java Monitoring and Diagnostic Tooling
Health Center - Installation
• The tool is provided in two parts:
– An Agent that collects data from a running application
– An Eclipse-based client that connects to the agent

• The Agent ships with the

following vm’s:
– Java 5sr9 and upwards
– Java 6sr3 and upwards

• The latest version of the agent is always available

from within the Health Center Client
– Recommended to always update to the latest version of the agent
– Agent package unzips over the jre directory of the JVM you are using

17
 Java Monitoring and Diagnostic Tooling
Health Center - Enable for Monitoring
• Full instructions are provided within the help shipped with the
Health Center Client but in most cases as simple as:

For Java 5 SR10 and later or Java 6 SR5 and later,

including Java 7 (can be used in production)
java –Xhealthcenter HelloWorld

For Java 5 SR9 and earlier, or Java 6 SR4 and earlier

(not recommended for use in a production environment)

java –agentlib:healthcenter –Xtrace:output=healthcenter.out HelloWorld

18
 Java Monitoring and Diagnostic Tooling
Health Center – Advanced Options

• Headless mode for data collection without connecting the GUI

– Useful for scenarios where firewall blocks connection
– Configurable to limit disk space used
– Timed collections
– Interval based collections
– Started with
-Xhealthcenter:level=headless

• Late attach enabled

• Automated javacore creation

19
 Java Monitoring and Diagnostic Tooling
Health Center – API

• The 2.2 release of Health Center contains a powerful API that

allows Java developers to embed Health Center in their
applications and harness its monitoring power to troubleshoot
problems
• Example:
// Create the connection object:
ConnectionProperties conn1 = new ConnectionProperties("localhost", 1973);

// Connect to the Health Center agent, using the previous connection settings:
HealthCenter hcObject = HealthCenterFactory.connect(conn1, true);

// Get garbage collection data and print:

GCData gcData = hcObject.getGCData();
System.out.println("GC Mode is " + gcData.getGCMode().toString());

20
 Java Monitoring and Diagnostic Tooling
Health Center

DEMO
21
 Java Monitoring and Diagnostic Tooling
Health Center

Environment reporting
•Detects invalid Java options
•Detects options which may hurt performance or
serviceability
•Useful for remote diagnosis of configuration-related
problems

Garbage Collection visualization

•Visualizes heap usage and gc pause times
over time
•Identifies memory leaks
•Suggests command-line and tuning
parameters
•Same recommendation logic as GCMV

22
 Java Monitoring and Diagnostic Tooling
Health Center

Class loading visualization

•Shows all loaded classes
•Shows load time
•Identifies shared classes
•Live class histogram information

CPU usage
•Visualizes overall system CPU use
as well as application process use

23
 Java Monitoring and Diagnostic Tooling
Health Center

I/O
•Monitor application file open/close events as they
occur
•Lists currently open files

Native Memory
•Detect native memory leaks in application
•Determine if external forces are using more
memory
•Memory counters showing which parts of the
JVM are using the most native memory

24
 Java Monitoring and Diagnostic Tooling
Health Center

Method Profiling
•Always-on profiling offers insight into application
activity
•Identifies the hottest methods in an application
•Full call stacks to identify where methods are being
called from and what methods they call
•No byte code instrumentation, no recompiling

Java Lock analysis

•Always-on lock monitoring
•Quickly allows the usage of all locks to be profiled
•Helps to identify points of contention in the
application that are preventing scaling

25
 Java Monitoring and Diagnostic Tooling
Health Center

Threads view
•List of current threads and states
•Deadlock detection and analysis
•Number of threads over time
•See contended monitors

Live control of application

•Trigger dumps
•Enable verbosegc collection

26
 Java Monitoring and Diagnostic Tooling
Agenda
• IBM Monitoring and Diagnostic Tools for Java
– Why use the tools?
– Where to get the tools?

• IBM Recommended Java Troubleshooting Tools

– Health Center
– Garbage Collector and Memory Visualizer (GCMV)
– Memory Analyzer

• Summary

27
 Java Monitoring and Diagnostic Tooling
Garbage Collector and Memory Visualizer (GCMV)

Motivating questions:
– How is the GC behaving? Can I do better?
– How much time is GC taking?
– How much free memory does my JVM have?

Overview
– Analyze Java verbose GC logs, providing insight into application behaviour
– Uses ps -p $PID -o pid,vsz,rss output to plot native footprint
– Visualize a wide range of GC data and Java heap statistics over time
– Provides the ability to detect memory leaks and optimized GC
– Recommendations use heuristics to guide you towards GC performance
tuning

28
 Java Monitoring and Diagnostic Tooling
Garbage Collector and Memory Visualizer (GCMV)
• Views of GCMV

29
 Java Monitoring and Diagnostic Tooling
Garbage Collector and Memory Visualizer (GCMV)
Graphical Display of Data
•Allows graphing of all available data: pause
times, heap size etc
•Allows zoom, cropping and change of axes
value and units
•Allows comparison of multiple files

Analysis and Recommendations

•Provides tuning recommendations based on data and
flags errors.
•Analysis can be limited using cropping.
•Values and units used in analysis can be changed by
changing axes values and units

30
 Java Monitoring and Diagnostic Tooling
Agenda
• IBM Monitoring and Diagnostic Tools for Java
– Why use the tools?
– Where to get the tools?

• IBM Recommended Java Troubleshooting Tools

– Health Center
– Garbage Collector and Memory Visualizer
– Memory Analyzer

• Summary

31
 Java Monitring and Diagnostic Tooling
Memory Analyzer

Motivating questions:
– Why did I run out of Java memory?
– What’s in my Java heap? How can I explore it and get new insights?

Overview
– Tool for analyzing heap dumps and identifying memory leaks from JVMs
– Works with IBM system dumps, heapdumps and Sun HPROF binary dumps
– Provides memory leak detection, footprint analysis:
Objects by Class, Dominator Tree Analysis, Path to GC Roots, Dominator Tree
by Class Loader
– Shows areas of memory wastage:
Collections, duplicate strings, substring/char arrays, constant value primitives
– Displays Stack trace with object references
– Provides SQL like object query language (OQL)
– Provides extension points to write analysis plugins

32
 Java Monitoring and Diagnostic Tooling
Memory Analyzer

Overview:
•Overview of the heapdump including size and total
number of objects.
•Provides links to continued analysis

Path to GC Roots:
•Provides the reference chain that prevents
an object being garbage collected

Dominator Tree grouped by Class Loader:

•Lists the biggest objects using a “keep alive tree” Grouping
by Class
•Loader limits the analysis to a single application in a JEE
environment

33
 Java Monitoring and Diagnostic Tooling
Agenda
• IBM Monitoring and Diagnostic Tools for Java
– Why use the tools?
– Where to get the tools?

• IBM Recommended Java Troubleshooting Tools

– Health Center
– Garbage Collector and Memory Visualizer
– Memory Analyzer

• Summary

34
 Java Monitoring and Diagnostic Tooling
Problem Scenarios and Tools

35
 Iris Baron
[email protected]

36
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

37
Where to find more information
• Documentation
• http://www.ibm.com/developerworks/java/jdk/docs.html
• http://www.redbooks.ibm.com/redpapers/pdfs/redp3950.pdf
• zOS SDK
• http://www.ibm.com/servers/eserver/zseries/software/java
• System z Linux SDK
• http://www.ibm.com/developerworks/java/jdk/linux/download.html
• Java Tuning documentation
• http://www.ibm.com/developerworks/views/java/libraryview.jsp?
search_by=java+technology+ibm+style:
• http://www-01.ibm.com/support/docview.wss?uid=swg27013824&aid=1
• http://proceedings.share.org/client_files/SHARE_in_San_Jose/S1448KI161816.pdf
• IBM Support Assistant
• http://www.ibm.com/software/support/isa/
• IBM Monitoring and Diagnostic Tools for JavaTM
• http://www.ibm.com/developerworks/java/jdk/tools/
• http://pic.dhe.ibm.com/infocenter/isa/v4r1m0/index.jsp
• Health Center API articles
– Monitor a Java application with the Health Center API parts 1 and 2
• http://www.ibm.com/developerworks/library/j-healthcareapi1/index.html
• http://www.ibm.com/developerworks/library/j-healthcareapi2/index.html
38