15 Perf

2006 BEA Systems, Inc.
Monitoring, Tuning, and

Troubleshooting Performance
At the end of this module, you will be able to:
Optimize IPC resources for Tuxedo application
requirements
Configure system and server options for improved
performance
Troubleshoot applications with runtime tracing
Use the Tuxedo Administration Console
Module 15
Monitoring, Tuning, and Troubleshooting Performance-1
2006 BEA Systems, Inc. 456
Road Map
1. IPC Resource Requirements
Understanding Shared Memory and Semaphores
Message and Message Queue Sizing
Performance Monitoring
Tracing ATMI Code Execution
2. Runtime and Configuration Performance Options
3. The BEA Tuxedo Administration Console

Reconfiguring Kernel Parameters
Most UNIX Operating Systems will require tuning.
Refer to the documentation (Tuxedo Platform Data
Sheets) for specific information for your operating
system.
Below is an excerpt from the Sun Microsystems Solaris 9
(32-bit) data sheet :
1024 30 SEMMNU semsys:seminfo_semmnu
2000 25 SEMMSL semsys:seminfo_semmsl
5029 10 SEMMNI semsys:seminfo_semmni
5048 60 SEMMNS semsys:seminfo_semmns
300 100 SHMMNI shmsys:shminfo_shmmni
100 6 SHMSEG shmsys:shminfo_shmseg
67108864 131072 SHMMAX shmsys:shminfo_shmmax
Recommended
Setting
Default
Setting
Traditional
Name
Solaris Name
Determining IPC Requirements
You can determine more specific IPC requirements for
your application:
Use tmloadcf c or tmboot c
Determining IPC Requirements for a Specific Tuxedo Application:
C:\> tmboot -c
Ipc sizing (minimum /T values only) ...
Fixed Minimums Per Processor
SHMMIN: 1
SHMALL: 1
SEMMAP: SEMMNI
Variable Minimums Per Processor
SEMUME, A SHMMAX
SEMMNU, * *
Node SEMMNS SEMMSL SEMMSL SEMMNI MSGMNI MSGMAP SHMSEG
------ ------ ------ ------ ------ ------ ------ ------
MYMACHINE 100 12 95 A + 1 77 154 519K

where 1 <= A <= 8.
Windows IPC Resources Page
The BEAProcMGR service sets resources to values
defined in the IPC Resources Page.
The default
values are
shown here.
Tuning Message and Queue Lengths
When Queues Are Too Long:
Can have a negative impact allowing a backlog of requests to
accrue affecting performance
If a need for large queues is perceived, something else is
probably wrong
When Queues Are Too Short:
Asynchronous calls will suffer performance slow-downs
Larger than average load can cause uneven response times
Large individual requests can cause spooling to disk:
Tuxedo uses file transfer if MSGMAX or 75% of MSGMNB is exceeded
Can be advantageous to prevent very rare
large requests from filling queues.
MSGMAX: Maximum Message Size
MSGMNB: Maximum Message Queue
Size
Tunable Parameters for IBM AIX
The latest supported version is AIX 4.53 (32-bit)
No IPC configuration is required for AIX.
To change the value of the maxuproc parameter:
1. Acquire superuser privileges.
2. Determine the recommended value
3. Change the value of the parameter.
4. Reboot the system.

System-Specific Diagnostic Tools
Some tools are specific to the system platform and are
provided by the operating system vendor.
Use UNIX System utilities such as:
top (Displays system and process information)
sar (System Activity Report)
In Windows:
Event Viewer
Task Manager
Performance Monitor

Windows Task Manager
Unix Monitoring Tools
Most Unix systems provide tools for monitoring
system performance, such as:
sar: system activity (Solaris)
mpstat: per-processor statistics (Solaris)
vmstat: virtual memory statistics
netstat: network statistics
iostat: I/O statistics
Example vmstat Output
Example of vmstat output:

$ vmstat 1
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 120 30180 1736 46696 0 0 424 25 134 77 5 9 86
3 0 0 120 13908 1776 63192 0 0 8220 0 268 275 15 41 44
0 1 1 120 5012 1784 71668 0 0 4344 5196 452 337 9 32 59
0 1 1 120 4756 1784 71796 0 0 128 7824 294 94 0 6 94
2 0 1 120 3200 1788 72436 0 0 1028 3364 266 91 2 6 92
0 2 0 140 3628 1800 70796 0 0 5368 0 216 286 41 30 30
1 0 0 140 23144 1856 51904 0 0 1968 28 339 500 49 16 36
4 1 0 140 20324 1968 53188 0 0 2152 0 210 368 72 7 21
6 0 0 140 15940 2020 54992 0 0 1796 0 413 315 85 10 5
6 3 0 140 12096 2112 56732 0 0 1880 0 468 478 80 19 1
10 0 0 140 8660 2152 58736 0 0 2040 0 232 448 81 19 0
5 3 0 140 4556 2184 61052 0 0 2340 0 482 1198 35 32 34

10
0101
1110
Methodology
A general tuning methodology:
1. State goals and define the system.
2. List services and outcomes.
3. Select metrics.
4. List parameters.
5. Select factors to study.
6. Select an evaluation technique.
7. Select workload.
8. Design experiments.
9. Analyze and interpret data.
10. Present results.
11. Repeat.

Jain, Raj. The Art of Computer Systems Performance Analysis. New York, NY: John
Wiley& Sons, April 1991.
Methodology Tips
Perform benchmarks on:
Functional areas (subsystems) of the application
The full (integrated) application
Test performance early and often, but defer tuning
until forced to do so.
Do the following to maintain test integrity:
Always generate load on separate machines from those
hosting server instances
Minimize the amount of data collection software running on
the machines hosting server instances
Account for the impact of both
Tracing ATMI calls
Tuxedo provides the facility to trace ATMI library code
execution within client and server processes.
Allows the application developer (or administrator) to
trace the flow of execution within a Tuxedo application.
The default ulog receiver outputs trace data:
to include entry and exit time stamps.
written by to the ULOG file.
The utrace receiver outputs trace record information:
by calling a user-defined tputrace() library routine.
Users can customize trace record information and output location.
Version 9.0 + only
Using the tmtrace Facility
tmtrace Environment Variable
setting
To trace a client and all ATMI calls made: TMTRACE=on
To see all client and server trace points,
both explicit and implicit:
TMTRACE="*:ulog:dye"
To trace service requests from a client
but restrict the tracing output from the
client to the bare minimum of information
about tpcall(3c) requests:
TMTRACE=
atmi:/tpcall/ulog:dye
To trace the invocations of all service
requests performed by application
servers:
TMTRACE=
atmi:/tpservice/ulog
changetrace tmadmin sub-command
To change the trace specification of a
specific currently-running server:
changetrace [-g groupname]
[-i srvid] newspec
Section Review
Determine the IPC resource requirements for a Tuxedo
application
Allow for optimum message queue sizing
Trace ATMI code execution within a running
application
In this section, we learned how to:
Road Map
2. Runtime and Configuration Performance
Options
Message Size Compression Thresholds
System and Server Tuning Options
Bundling Services in Servers
Measuring Service Execution Times
Multiple-Server-Single-Queue (MSSQ) Sets

Protecting Shared Memory
In the development cycle, programmer code can
inadvertently corrupt shared memory.
You can protect system tables in shared memory with
the SYSTEM_ACCESS parameter in RESOURCES.
The SERVERS section also provides an override setting.
selected option cannot be changed
NO_OVERRIDE
libraries will not attach
PROTECTED
libraries attach at all times (default)
FASTPATH
Description Option
Protecting Shared Memory:
*RESOURCES
SYSTEM_ACCESS PROTECTED, NO_OVERRIDE
...
UBBCONFIG
SPINCOUNT
Specified in MACHINES section, SPINCOUNT is the
number of attempts made to the BB before sleeping.
Default is 0, which allows each process to use value built into
the binary or use the TMSPINCOUNT environment variable.
Always set SPINCOUNT to 1 for a single CPU machine.
Experiment with a SPINCOUNT between 5,000 and
50,000 for multiple CPU machines.
Tune by trial, while watching a busy server with semOPS
Setting SPINCOUNT per Machine Type:
*MACHINES
gumby # a 4-CPU Solaris machine
SPINCOUNT=25000
PLUTO # a 1-CPU Windows machine
SPINCOUNT=1
UBBCONFIG
Service Lookup Caching
This feature is turned on by default; a performance
improvement over previous versions of Tuxedo.
Saves some BB overhead by caching service (and interface)
entries within the calling process.
Configuration parameter is SICACHEENTRIESMAX
Used in the MACHINES or SERVERS section.
Set value to 0 to turn off or change to an appropriate value
for your application. Default: 500
This feature can also be set/reset per process with the
TMSICACHEENTRIESMAX environment variable.
Versions 8.0 + only
Changing the number of cached services:
*MACHINES
gumby
SICACHEENTRIESMAX=650
UBBCONFIG
Message Compression Thresholds
Remote compression should be based on network
speed and congestion:
Low-speed network compress all messages.
Workstation clients use the compression threshold specified
by the WSL CLOPT -c option.
Local compression is used to prevent disk spooling.
Compress large messages if message queue size is an issue.
Using CMPLIMIT parameter in MACHINES section:

Compressing both remote and local messages:
*MACHINES
gumby
CMPLIMIT=2048,8192
PLUTO
CMPLIMIT=1024,8192
UBBCONFIG
Compress all internal messages larger than 2k
Compress all remote messages larger than 8k
OPTIONS Parameter Values
NO_AA : turns off the extended security plug-in calls:
Username/password authentication or ACL authorization
security is not affected.
NO_XA : turns off XA transactions for the entire
application domain.
Use only if you are not using distributed (XA) transactions or
/Q queuing features.
It is also recommended to set the MAXGTT parameter to 0 in
the RESOURCES section.
Versions 8.0 + only
Protecting shared memory:
*RESOURCES
OPTIONS NO_AA, NO_XA
MAXGTT 0
...
UBBCONFIG
Single-Threaded Option
Use the TMNOTHREADS environment variable for
processes within your application.
Set TMNOTHREADS=Y for all single-threaded clients
and single-threaded servers:
In the runtime environment for remote clients.
In the ENVFILE for servers and native clients.
Results in multi-threaded support code being bypassed
providing improved performance.
Tuxedo versions 8.1+ can still make use of the
BRTHREADS=Y parameter in MACHINES.
Multi-threaded BRIDGE processes ignore TMNOTHREADS.
Versions 8.0 + only
Service/Server Bundling
The following criteria should be considered in bundling
Execution time
Service dependencies
Message buffer sizes and queue sizes
Expected number of requests
Executable size
Shared libraries (BEA Tuxedo/other)
Server/service replication
Process limits
Business rules
The application architect usually decides this !
Repackaging Servers
When service execution time differs by more than
20%, consider re-partitioning server functionality
instead of using LOAD for performance tuning.

Server App 1b
Servant Object A
Method1() [1.0 secs]
Server App 1b

Service1() [1.0 secs]
Server App 1a

After repackaging, overall response times
for requests to Service3 and Service4 will
probably improve because of less wait time
spent on the servers queue
Mix of services in Server with
large variation of service times
Server App 1
Servant Object A
Servant Object A
Server App 1


Repackaging Advertised Services
In most cases, the administrator does not have access to
the source code for the Tuxedo application servers.
Repackaging of services can still be accomplished by
using the CLOPT s option.
Original Configuration:
*SERVERS
serverApp1
SRVGRP=APPGRP SRVID=10
CLOPT=-A MIN=10
New Configuration:
*SERVERS
serverApp1
CLOPT=-s SVC1:foo1
MIN=3
serverApp1
CLOPT=-s SVC2:foo2
MIN=7
Obtaining SERVICE to routine mapping:
C:\> serverApp1 -v
SVC1:foo1
SVC2:foo2
Measuring Service Execution Times
Tuxedo can be configured to measure the time it takes
to perform each service.
Use CLOPT -r option to enable service timings to be
written to stderr.
The -r option should normally be used in test environments.
Use the CLOPT -e option to specify the log file name.
(default is stderr)

*SERVERS
myAppServer
SRVGRP=GROUP1 SRVID=10
RESTART=Y MAXGEN=5 MIN=5
CLOPT=-A -r -e svctimes.txt
UBBCONFIG
Performance Logging Example
This file will contain a line entry for every service call
processed and from every server configured to output
to the log file.
10
0101
1110
Performance Logging Report
Use the txrpt utility to
produce a formatted
report from a service
execution log file.

OPTIONS
-t Sort by descending elapsed service time
-n Report only services specified by name
-d Report only requests occurring on mm/dd
-s Report only requests starting after time
-e Report only requests ending after time
Output the Service Log Report:
C:\> txrpt < svctimes.txt
Multiple-Server-Single-Queue (MSSQ)
Message
Queue
SERVER 1
Services A,B,C
By default, each Tuxedo application server is assigned
its own IPC request message queue.
Multiple-Server-Single-Queue (MSSQ) sets enable two
or more servers to dequeue the same request queue.
SERVER 3
Services A,B,C
SERVER 2
Services A,B,C
When to Use MSSQ
Consider using an MSSQ set for:
High volume (request queue is always busy), quick
turnaround (fast response time)
Moderate number of servers that offer the same set of
services (rule of thumb 2 to 10)
Message request sizes are reasonable relative to the IPC
request queue size limit
MSSQ may not be suitable for servers expecting large
message sizes that may quickly fill the request queue.
Be aware that an MSSQ set still uses the same queue size
limited by the Operating System kernel parameter.
Configuring an MSSQ Set
Request
Queue
MyServer
Services A,B,C
MyServer
Services A,B,C
MyServer
Services A,B,C
Configuring an MSSQ Set:
*SERVERS
MyServer
SRVGRP=APPGRP SRVID=10 MIN=3 MAX=5
RQADDR=myServerQ REPLYQ=Y
UBBCONFIG
When MIN>1, specifying a name for the
common request queue enables an MSSQ
set for this group of servers
REPLYQ is not required, but it is
recommended to avoid possible lost
replies from other service calls.
REPLYQ
Automatic Start of Additional Servers
Use the CLOPT -p option to automatically start
additional servers and to shutdown servers when the
load has abated.
If the number of requests on
the queue exceed 10 for a
period of 7 seconds, an
additional server is started
until MAX servers are active.
If the number of requests on
the queue fall below 5 for a
period of 120 seconds, one
server is stopped until only
MIN servers are active.
Configuring an MSSQ Set:
*SERVERS
MyServer
SRVGRP=APPGRP SRVID=10 MIN=3 MAX=5
RQADDR=myServerQ REPLYQ=Y
CLOPT=-A p 5,120:10,7
UBBCONFIG
Section Review
Configure the additional system and server runtime
configuration options
Optimize performance for application that do not use
all of the Tuxedo architecture
Determine service execution times
Configure and deploy MSSQ sets

Lab Exercise
For details on the exercise, refer to the Lab Guide.
If questions arise, ask the instructor.
The instructor will determine the stop time.
Lab 16 PERF: Configure and Use Additional
Performance Enhancement Options
Road Map
2. Runtime and Configuration Performance Options
Configuring the Web Server and Listener
Launching the Web browser
Connecting to a Domain Configuration
Monitoring and Configuration Options
Creating new Domain Objects
The Tuxedo Administration Console
The Administration Console is a Java-based applet that
you can download into your Web browser.
It can be used from a remote site to:
Monitor system events, logs, and statistics.
Dynamically create and configure administration objects.
Create new domain configurations.
Manage application resources.
Internet Explorer 6.0 using Java 1.4.x plug-in has been
validated by BEA however:
Other web browsers (with a JRE) will work on most
platforms.

Setting Up Your Environment
Any commercial Web server can be used; BEA also
provides tuxwsvr with the system software.
Prior to starting the Web Server, the following
environment variables must be set:
TUXDIR: Set to the Tuxedo product installation directory
PATH: Must include the %TUXDIR%\bin directory
The tuxwsvr process runs in the background until the
machine shuts down or until it is terminated.
In UNIX, use kill -9 on the process id.
In Windows, use the Task Manager > Process Tab (End
Process)
Starting the Web Server tuxwsvr
During the software installation, the tuxwsvr.ini
file is automatically created.
Located in the %TUXDIR%\udataobj directory
You may edit this file if you wish to move the Java files to
another directory.
The l and i command-line options are required.
The l network address will be used for your browser URL.
The i is the initialization file (normally set to the standard
tuxwsvr.ini file).
Starting the tuxwsvr process from the command-line:
C:\> tuxwsvr l //student12:4321
i %TUXDIR%\udataobj\tuxwsvr.ini
Starting the Listener Process - wlisten
During the software installation, the wegui.ini file
is automatically created.
Located in the %TUXDIR%\udataobj\webgui directory
Contains all wlisten initialization parameter options
Edit this file if default values are not appropriate:
Uses port 4003 runs in the background no LLE etc.
The i command-line option is optional.
Used to specify a different file other than the default.
Starting the wlisten process from the command-line:
C:\> wlisten
Launching the Web Browser
Enter the following URL in your browser:
http://machinename:port/webguitop.html
This is the IP Address of your Web Server.
In our example, the l option for the tuxwsvr process:
http://student12:4321/webguitop.html
The BEA Tuxedo
Administration Console
splash screen displays.
Scroll past the licensing
information to the bottom of
the page; click the button to
enter the login page.
Login Information
After a few moments (depends on your machine and
browser):
The BEA Tuxedo Administration Console Login window
should display in a Java applet.
If you do not see the login window, you will need to download a Java
1.4.x plug-in for your browser.
The Password must be present in the tlisten.pw
file located in %TUXDIR%\udataobj
Optionally provide a login name (no spaces)
It must remain active during your
session. You may minimize it to
get it out of your way.
NOTE: After the Console
displays, do not close this
browser window!
Administration Console Main Window
When you first enter the main window there is no
associated Tuxedo domain.
Subsequent logins will remember your last working domain.
Connecting to a Domain Configuration
To associate the Console with a pre-
configured Tuxedo domain:
Click on Domain > Open...
The domain does not have to be running; it is
only looking for the tuxconfig binary file.
A Java Applet window is displayed.
Enter the name and location of TUXCONFIG.
Enter the DOMAINID (if it has been
specified).
Monitoring the Domain
Toolbar Buttons
Invokes the context-sensitive help. Use the cross-hairs cursor to
click on a specific field or area.
Set the default mode (edit or view). Set location for online docs.
Displays the tab pages that allows for graphical monitoring info.
Displays the window for monitoring System Event Broker events.
Displays the ULOG file for a specific machine.
Used to initiative server group migration.
Deactivates (shutdown) all or part of a Tuxedo application domain.
Activates (boot) all or part of a Tuxedo application domain.
Opens a pop-up window to search for an administrative object.
Updates the tree view and configuration tool pane.
Interrupts the current request and returns control to the user.
Description Button
Opens the Admin Console on-line help in a separate Web browser.
Administration Console Tree View
The Tree View pane appears
in the left column of the
main GUI window.
Clicking on any object
corresponds to that MIB
class and retrieves its
attribute values.
Right-click an object to
reveal a quick-edit menu:
Options depend on object type
Icon color reveals the
objects state.
The Configuration Tool Pane
Each object has a corresponding
set of tabs in the Configuration
Tool pane.
The tab pages are electronic forms
that display information about the
administrative object.
The attributes that can be changed
are presented as input form fields.
In some cases, attempting to
change an attributes value will
return a MIB SET error:
Consult the MIB class documentation
Viewing System Statistics...

Each object has a corresponding set of tabs in the
Configuration Tool pane.
...Viewing System Statistics
Tracking client
requests
Watching
selected server
processes
Monitoring load
distribution
across services
Viewing the ULOG
The Logfile Tool provides filtering and formatting
options to view ULOG entries.
Once messages have been downloaded, clicking on a message
will bring up the HTML Message Catalog for that entry.
Creating New Domain Objects
New MIB class objects can be created:
Prior to booting the application
Dynamically while the application is active
The functionality is similar to tmconfig or ud32.
Create a new machine, group, server.
Starting and Stopping Servers
Most of the tmboot and
tmshutdown options are
available.
The console terms are
activate and deactivate.
Section Review
Configure and deploy the Tuxedo Web Server and
Listener.
Launch the Tuxedo Administration Console.
Use the more common features within the
Administration Console.
Lab Exercise
For details on the exercise, refer to the Lab Guide.
If questions arise, ask the instructor.
The instructor will determine the stop time.
Lab 17 PERF: Configure, Deploy, and Test
the Tuxedo Administration Web Console
2006 BEA Systems, Inc.
Module Review
Optimize IPC resources for Tuxedo application
requirements.
Configure system and server options for improved
performance.
Troubleshoot applications with runtime tracing.
Use the Tuxedo Administration Console.
In this module, we learned how to:

15 Perf

Uploaded by

Copyright:

Available Formats

15 Perf

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

15 Perf

Uploaded by

Copyright:

Available Formats

2006 BEA Systems, Inc.

Monitoring, Tuning, and

You might also like