Session id: 1156

Maximizing the Performance of

the Oracle eBusiness Suite

Ahmed Alomari
Applications Performance Group
Oracle Corporation
Agenda
 Architecture
 Application Services Tier
– Forms
– Reports
– Concurrent Manager
– Apache
 Mod PL/SQL
Jserv/JVM
– Portal
– Discoverer
 Network Performance
 Data Server Tier
 Application Tuning
 11i10 Performance Improvements
Q&A
 Architecture
Portal Real Application
Clusters (RAC)

Servlet Engine
JDBC
(Thin)
Java Server Pages (JSP)
Web Listener

http
AOLJ

Discoverer
TNS
PL/SQL Gateway (Net8) Data Server Tier
Clients
Reports

Forms

Concurrent Manager
Application Tier
Application Services Tier

 Forms
– Deploy with socket mode for internal users:
 connectMode=socket (appsweb.cfg)
– Enable Forms Dead Client Detection
 Value specified in minutes
 Terminates f60webmx processes for dead
clients.
 FORMS60_TIMEOUT=10
– Enable Forms Abnormal Termination Handler
 Do not set FORMS60_CATCHTERM
Application Services Tier

 Forms
– Upgrade to the latest Forms patch set (patch set 16)
 Forms patchset 7 (6.0.8.16.X) introduced a regression
which affects scalability (bug 2269913).
­ Results in Forms generating invalid SQL.
­ Fixed in Forms patchset 10 (6.0.8.19.X).
 Refer to MetaLink document 125767.1 for Forms
patchset upgrades.
Application Services Tier

 Forms – Cancel Query

– Should not be enabled unless you are on Forms patchset
14 (Forms version 6.0.8.23.x or higher).
– Refer to MetaLink Note 138159.1 for instructions on how
to enable and tune Cancel query related parameters.
– Cancel Query increases middle-tier CPU as well as DB
CPU.
– To Disable Cancel Query
 Set the Profile “FND: Enable Cancel Query” to No.
Application Services Tier

 Forms
– Ensure users are optimally utilizing the
professional Forms interfaces.
 Avoid Blind queries
 Provide selective criteria in Find windows and
LOVs.
 Avoid opening and closing forms across
transactions.
­ Users which need to switch responsibility in
order to complete a business transaction
should instead keep the relevant Forms open.
Application Services Tier
 Forms (Tracing)
– If the form is slow, or a specific flow in the Form
is slow, first generate a complete SQL trace
using the Trace menu in Help-Diagnostics.
– If the SQL trace does not account for the
majority of the response time experienced by the
user, then generate a Forms Runtime
Diagnostics (FRD) Trace.
Application Services Tier
 Forms (Tracing – SQL Trace)
Application Services Tier
 Forms (Tracing – FRD Trace)
– Logon to Applications 11i with the following
parameters appended to the URL:
play=&record=collect&log=/tmp/oeform.trc
– Navigate through the flows which you would like
to trace.
– Exit from Oracle Applications completely.
– Review the trace file from the application server
(/tmp/oeform.trc).
Application Services Tier
 Reports
– Ensure users provide selective parameters to
the standard Reports.
– Review the Report output:
 Number of rows returned to the report.
 Number of report pages.
– SQL Trace can be generated at Request
Submission Time (SRS form – Debug Options)
or by setting the “Enable Trace” flag in the
Concurrent Program Definition Form.
Application Services Tier
 Reports (Tracing)
– If the SQL trace for the report does not account for the
elapsed time of the request, then generate a Reports
trace by appending tracing parameters the to concurrent
program command line.

– The following are the trace options:

– TRACE_ALL : log all possible trace information in the trace file. (DEFAULT)
– TRACE_APP : log trace information on all the report objects in the trace file.
– TRACE_BRK : list breakpoints in the trace file.
– TRACE_ERR : list error messages and warnings in the trace file.
– TRACE_PLS : log trace information on all the PL/SQL objects in the trace file.
– TRACE_PRF : log performance statistics in the trace file.
– TRACE_SQL : log trace information on all the SQL in the trace file.
– TRACE_TMS : enter a timestamp for each entry in the trace file.
Application Services Tier
 Reports (Tracing)
– Enabling Reports Trace (pre 11i10)
 Sign-on to Apps with System Administrator responsibility
 Navigate to Concurrent -> Program -> Define
 Query the concurrent program (report) which you want to trace.
 Locate the “Options” field and enter the following values :
 TRACEOPTS=<trace option> TRACEFILE=<path/file>
 Save the concurrent program
 Run the report and a trace file will be generated.
 Remember to clear the trace options after running the report.
– Enabling Reports Trace (11i10)
 Use the Debug Options page at request submission.
Application Services Tier
 Reports (Trace Output)
Report: /u02/appl/fnd/11.5.0/reports/US/FNDSCURS.rdf
Logged onto server:
Username:
. . . . . . . .

23:59:57 APP ... ( Generic Graphical Object B_SECURITY_GROUP_NAME1

23:59:57 APP ... ) Generic Graphical Object B_SECURITY_GROUP_NAME1
23:59:57 APP ... ( Generic Graphical Object B_USER_NAME1
. . . . . . . .
+-------------------------------------+
| Report Builder Profiler statistics |
+-------------------------------------+

TOTAL ELAPSED Time: 2243.04 seconds

Reports Time: 1196.62 seconds (53.34% of TOTAL)

ORACLE Time: 1046.41 seconds (46.65% of TOTAL)

UPI: 1046.00 seconds

SQL: 0.40 seconds

TOTAL CPU Time used by process: 904.71 seconds

Application Services Tier
 Concurrent Manager
– Avoid enabling an excessive number of standard
or specialized managers.
– Use specialization rules and work shifts to bind
specific jobs to specific time windows.
 Helps avoid scheduling resource intensive batch
requests during peak activity.
– For jobs which spawn parallel workers such as
Auto Invoice or Payroll, set the sleep time of the
Conflict Resolution Manager (CRM) to null (i.e.
10 seconds). The default value is 60 seconds.
Application Services Tier
 Concurrent Manager
– Database Resource Manager Integration
 Define Resource Consumer Groups and
associate specific concurrent programs with a
resource group.
 Allows an administrator to constrain the amount
of CPU resources used by a concurrent program.
 Resource Groups can also be defined for online
users using the profile “FND: Resource
Consumer Group”.
 Application Services Tier
 Concurrent Manager
– Transaction Managers
 Used for Inventory Transactions as well as other synchronous
online processing.
 Ensure enough transaction managers exist to service the
request load.
 Set the profile “Concurrent:Wait for Available TM” to 1 (second).
 Set “TP:INV Transaction processing mode” to “On-line
processing” for small inventory requests from the UI.
 Set the sleep time on the transaction managers to a high
number (e.g. 10 minutes).
­ Avoids constant polls to check for shutdown requests.
 Transaction Managers can also be traced.
Application Services Tier
 Concurrent Manager
– Enabling Tracing
 Program Level
­ Enable the trace option in the concurrent program
definition form.
 Request Level (new in 11i10)
­ Use the Debug Options button in the Request
Submission Form
­ The profile “Concurrent: Allow Debugging”
should be set to “Yes.”
Application Services Tier
 Concurrent Manager
– Enabling Trace at the Program Level
Application Services Tier
 Concurrent Manager
– Enabling Trace at the Request Level
Application Services Tier
 Concurrent Manager
– Enabling Trace at the Request Level
Application Services Tier
 Apache
– On UNIX, Apache is process based (httpd), and mods
such as mod PL/SQL run within the process address
space of the httpd processes.
– On Windows, Apache is multi-threaded (Apache.exe).
– Tune the number of processes and number of clients
(httpds.conf):
 MinSpareServers 5
 MaxSpareServers 10
 StartServers 5
 MaxClients 512
Application Services Tier
 Apache
– Minimize levels of logging (httpds.conf):
 LogLevel warn
 SSLLogLevel warn
– Enable Caching of non-HTML resources including
images, style sheets, and Java script.
 Caches content in the browser cache.
­ Reduces network round-trips (non SSL)
­ Reduces network bandwidth utilization (SSL)
 Included in AutoConfig templates in 11.5.8 (or higher).
Application Services Tier
 Apache
– Enable Caching of non-HTML resources (httpd.conf
or apps.conf)
<Directory "<physical_path_corresponding_to_the_alias_/OA_HTML/>">
#enable the generation of the Expires header for files under /OA_HTML/
ExpiresActive On
#expire images one month after last client access
ExpiresByType image/gif "access plus 1 month"
#expire stylesheets one week after the last client access
ExpiresByType text/css "access plus 1 weeks"
#expire javascript libraries one day after the last client access
ExpiresByType text/javascript "access plus 1 days“
ExpiresByType application/x-javascript "access plus 1 day"
</Directory>
 Application Services Tier
 Apache
– Review Apache Access Log file to ensure images
are being satisfied from the browser cache.
 Http code 200 (request for document)
 Http code 304 (request for time stamp)
130.35.127.106 - - [23/Oct/2002:19:00:21 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 200 821
130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDWATHS.gif HTTP/1.1" 200 190
130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDREDPT.gif HTTP/1.1" 200 70
130.35.127.106 - - [25/Oct/2002:14:39:23 -0700] "GET /OA_HTML/OA.jsp?
page=/oracle/apps/icx/por/rcv/pages/ReceivingHomePage&OAHP=ICXP
OR_MENU&OASF=ICXPOR_RCV_HOME_PAGE&dbc=ap107fam_ipdev11i&language_code=US&transactionid=70A5819F04C0F411
HTTP/1.1" 200 14
130.35.127.106 - - [25/Oct/2002:14:39:34 -0700] "GET /servlets/PoolMonitor?jvm HTTP/1.1" 200 7954
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_HTML/US/ICXINDEX_ipdev11i.htm HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/logo.gif HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/appslogo.gif HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRL.gif HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRR.gif HTTP/1.1" 304 -
Application Services Tier
 Apache Mod PL/SQL
– Configure a dedicated mod PL/SQL Listener
 Improves performance and scalability
­ Significantly reduces overall number of
sessions/connections.
­ Reduces latency of web requests.
­ Improves cursor sharing.
 Documented in “Oracle 9i Application Server Using the
PL/SQL Gateway Release 1 (v1.0.2.2)”
­ http://technet.oracle.com/docs/products/ias/doc_library/102
2doc_otn/apps.102/a90099/apptroub.htm#634180
 Application Services Tier
 Apache Mod PL/SQL (dedicated listener)
1. For the main Listener running on Port 7000, edit the file
$IAS_HOME/Apache/modplsql/cfg/plsql.conf as follows:

Disable the mod PL/SQL service from the main listener by commenting out the
lines between the two Location parameters as follows:

# <Location /pls>

# SetHandler pls_handler

# Order deny,allow

# Allow from all

# </Location>

Comment out the following line as follows:

# LoadModule plsql_module /d1/ias/Apache/modplsql/bin/modplsql.so

Application Services Tier
 Apache Mod PL/SQL (dedicated listener)

2. Configure the main listener to forward all mod_plsql requests to the

dedicated mod PL/SQL listener by adding the following line:

ProxyPass /pls/ http://sechost.us.oracle.com:8888/pls/

For the dedicated mod PL/SQL Listener running on Port 8888, configure each
DAD to override the default CGI environment variables in order to allow
redirects. Edit the file $IAS_HOME/Apache/modplsql/cfg/wdbsvr.app and add
the following line for each DAD:

cgi_env_list=SERVER_NAME=mainhost.us.oracle.com,SERVER_PORT=7000,HOST=mainho
st.us.oracle.com:7000
Application Services Tier
 Apache Jserv / JVM
– Minimize Jserv logging
 log=true
 log.channel.warning=true
 log.file=/d1/ias/Apache/Jserv/logs/jserv_7000.log
– Review error log files for exceptions.
– Disable auto reload in production environments:
 autoreload.classes=false
 autoreload.file=false
– Use Jserv Auto Load Balancing
 Configure Multiple Zones.
 Provides higher availability and improves scalability.
Application Services Tier
 Apache Jserv / JVM
– Upgrade to the latest JDK (1.4.2_06).
– Do not disable hotspot or the JIT (i.e.
–DCOMPILER=NONE or –Xint).
– Enable verbose GC in order to optimally tune the
heap sizes based on the GC traffic.
 Minor Collections
 Major Collections
 Elapsed Time of GC
Application Services Tier
 Apache Jserv / JVM
– Enable Verbose GC Output
 Overload wrapper.bin with a shell wrapper in order
to redirect stdout to a file.

Existing:
wrapper.bin=/apps/jdk/jdk1.4.2/bin/java

Change To:
Wrapper.bin=/apps/scripts/java.sh

java.sh:
=========

/apps/jdk/jdk1.4.2/bin/java -verbosegc $* >> /apps/logs/java.log

Application Services Tier
 Apache Jserv / JVM (Verbose GC Output)
/apps/logs/java.log:

[GC 30460K->1369K(510848K), 0.1135695 secs]

[Full GC 15135K->1686K(510848K), 0.2700469 secs]
[GC 32123K->2131K(510848K), 0.0203634 secs]
[GC 32595K->2130K(510848K), 0.0113639 secs]
[GC 32593K->2171K(510848K), 0.0129179 secs]
[GC 32635K->2419K(510848K), 0.0567306 secs]
[GC 32881K->3157K(510848K), 0.2906981 secs]
[GC 33620K->3197K(510848K), 0.0320023 secs]
[GC 33661K->3218K(510848K), 0.0103013 secs]
[GC 33674K->3309K(510848K), 0.0487887 secs]
[GC 33769K->3532K(510848K), 0.0531514 secs]
[GC 33983K->3784K(510848K), 0.0552549 secs]
[GC 34248K->4056K(510848K), 0.0624969 secs]
[GC 34520K->4404K(510848K), 0.0555575 secs]
[GC 34868K->4828K(510848K), 0.0587044 secs]
[GC 35292K->5242K(510848K), 0.0945290 secs]
Application Services Tier
 Apache Jserv / JVM
– Utilize the verbose GC output to tune the JVM
heaps (-Xmx and –Xms) accordingly.
– Review the frequency of collections, especially
major collections (i.e. Full GC).
 Start with:
­ -Xms256M and –Xmx512M
­ -XX:NewSize=60M -XX:MaxNewSize=120M
– Above settings have been incorporated in 11i10.
 Application Services Tier
 Apache Jserv / JVM
– Process Identification (V$SESSION)
 Specify the property –DCLIENT_PROCESSID in the
JVM startup shell script.
 Set automatically in 11i10.
Existing:
wrapper.bin=/apps/jdk1.4.2/bin/java

Change To:
Wrapper.bin=/apps/scripts/java.sh

java.sh:
=========
#!/bin/sh
/apps/jdk1.4.2/bin/java -verbosegc -DCLIENT_PROCESSID=$$ $* >> /d2/logs/java.log
 Application Services Tier
 Apache Jserv / JVM
– Process Identification (V$SESSION)
 Allows you to map the JDBC session from
v$session to a particular JVM process.

SID MACHINE PROCESS MODULE LOGON

---- ------------------------- ------------ ------------------ -----------------
41 aptier1.us.oracle.com 28806 JDBC Thin Client 12/02/04 14:26:43
42 aptier2.us.oracle.com 1723 JDBC Thin Client 12/02/04 14:27:01
43 aptier3.us.oracle.com 3201 JDBC Thin Client 12/02/04 14:28:15
44 aptier1.us.oracle.com 28807 JDBC Thin Client 12/02/04 14:29:17
. . . . . . .
. . . . . . .

aptier1{apps_a}-> ps -ef | grep 28806

apps_a 28806 28561 0 14:26:39 pts/20 0:00 /bin/sh ./java.sh

apps_a 28807 28806 53 14:26:40 pts/20 8:55 /jdk1.4.2/bin/../bin/sparc/native_threads/java
Application Services Tier
 Apache Jserv / JVM
– JDK 1.4.2 is Certified with Apps.
 Refer to MetaLink note 246105.1 for details on
upgrading to the JDK 1.4.2.
 ~15% performance improvement.
 Parallel GC (-XX:+UseParNewGC)
 Detail GC Statistics (-XX:+PrintGCDetails)
[GC [DefNew: 2047K->0K(2112K), 0.0012660 secs] 2472K->424K(3520K), 0.0014699 secs]
[GC [DefNew: 2047K->0K(2112K), 0.0012561 secs] 2472K->424K(3520K), 0.0014669 secs]
[GC [DefNew: 2047K->0K(2112K), 0.0012672 secs] 2472K->424K(3520K), 0.0014754 secs]
[GC [DefNew: 0K->0K(2112K), 0.0018657 secs][Tenured: 20227K->227K(21416K), 0.1107343
secs] 20227K->227K(23528K), 0.1387975 secs]
[GC [DefNew: 0K->0K(2112K), 0.0019746 secs][Tenured: 20227K->227K(21416K), 0.1105988
secs] 20227K->227K(23528K), 0.1384847 secs]
[GC [DefNew: 0K->0K(2112K), 0.0019862 secs][Tenured: 20227K->227K(21416K), 0.1105816
secs] 20227K->227K(23528K), 0.1386478 secs]
Application Services Tier
 Apache Jserv / JVM
– JDK 1.4 provides new JVM monitoring tool (jvmstat)
 Download from
http://developers.sun.com/dev/coolstuff/jvmstat/
Application Services Tier
 Java Server Pages (JSPs)
– Precompile the JSPs to avoid dynamic compilation.
 Users experience poor performance for the initial page loads.
 Potential deadlocks if multiple users attempt to compile the
same JSP.
 Potential JVM death due to OutOfMemoryException during
concurrent compilation.
 MetaLink Document 215268.1 provides the instructions and the
patch reference for an automated script to perform the
precompilation.
­ ojspCompile Script
 With the latest version of ojspCompile, it takes 15 minutes to
compile all the JSPs.
Application Services Tier
 Java Server Pages (JSPs)
– Use a separate JVM to perform the JSP
compilation.
 Automatically set in 11i10.

root.properties:

servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,
unsafe_reload=false,
page_repository_root=/appl_top/115/common/html/jsp/pagecache,
alias_translation=true,developer_mode=false,
javaccmd=/usr/jdk142/bin/javac,send_error=true
Application Services Tier
 Portal Tuning
– Tune the number of content fetcher threads for
PPE (poolSize). Default is 25.
– Adjust the fetch request timeout (requesttime).
– Adjust the fetch connection request timeout (stall).

zone.properties:

servlet.page.initArgs=poolSize=50
servlet.page.initArgs=requesttime=200
servlet.page.initArgs=stall=100
Application Services Tier
 Portal Tuning
– Ensure that the mod PL/SQL cache is enabled and
sized appropriately:
$APACHE_TOP/modplsql/cfg/cache.cfg:

[PLSQL Cache]
enabled=yes
total_size=100000000
cleanup_size=75000000
cleanup_interval=86400
;

[Cookie Cache]
enabled=yes
total_size=25000000
cleanup_size=15000000
cleanup_interval=86400
max_size = 0
Application Services Tier
 Discoverer 4i Viewer Tuning
– Disable Query Prediction by setting QPPEnable=0 in pref.txt.
– Set ObjectsAlwaysAccessible=1 to avoid extra workbook
SQL validation, and related dictionary SQL.
– Set the Query Governor option “Limit retrieved query data to:”
to 100 rows.
– Set the option “After opening a worksheet:” to “Don’t run
query.”
– Ensure Custom workbooks define mandatory parameters.
Network Performance
 Network performance is often neglected.
 Networks with poor latency or low bandwidth
impacts user response time.
 Can also impact concurrent programs if the
network between the application tier and data
server tier is not optimal.
 A direct switch should be used between the
application tier and data server tier
– Latency should ideally be ~1-2ms per round-trip.
Network Performance
 Use the ping command to determine the latency of requests. Vary
the packet size to determine the average latency. Do not use the
default ping packet size of 32 bytes.
D:\>ping aptierhost.us.oracle.com -l 2048

Pinging aptierhost.us.oracle.com [139.185.128.27] with 2048 bytes of data:

Reply from 139.185.128.27: bytes=2048 time=371ms TTL=252

Reply from 139.185.128.27: bytes=2048 time=330ms TTL=252
Reply from 139.185.128.27: bytes=2048 time=361ms TTL=252
Reply from 139.185.128.27: bytes=2048 time=360ms TTL=252

D:\>ping aptierhost.us.oracle.com -l 512

Pinging aptierhost.us.oracle.com [139.185.128.27] with 512 bytes of data:

Reply from 139.185.128.27: bytes=512 time=231ms TTL=252

Reply from 139.185.128.27: bytes=512 time=210ms TTL=252
Reply from 139.185.128.27: bytes=512 time=231ms TTL=252
Reply from 139.185.128.27: bytes=512 time=220ms TTL=252
 Network Performance
 Use the ping command to determine the latency between the
application server and the data server. Use a packet size of 2K
since this the default size for SQL*Net traffic.

aptierhost{applmgr}-2-> ping -s dbtierhost 2048

PING dbtierhost.us.oracle.com: 2048 data bytes

2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=0. time=1. ms
2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=1. time=1. ms
2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=2. time=1. ms
2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=3. time=1. ms
2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=4. time=1. ms
2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=5. time=1. Ms

---- dbtierhost.us.oracle.com PING Statistics----

6 packets transmitted, 6 packets received, 0% packet loss
round-trip (ms) min/avg/max = 1/1/1
Network Performance
 Measure the sustained bandwidth on both the
client network (i.e. desktops) as well as the
application-tiers.
 Use traceroute (or tracert) utility to determine
number of hops and impact on latency.
 Monitor network statistics including packet
rates, error rates, collisions, etc.. using tools
such as netstat –i.
Data Server Tier
 Refer to the MetaLink document 216205.1 “Database Initialization
Parameters and Configuration for Oracle Applications 11i”.
– Ensure mandatory parameters are set correctly.
 Optimally configure the buffer cache and shared pool as per the
workload and the number of users:
– [typical configuration for 2,000 Apps users]
 db_block_buffers=800,000
 shared_pool_size=2000M
 A poorly sized buffer cache results in excessive buffer gets and
physical I/O.
 A poorly sized shared pool results in library cache and shared pool
latch contention due to reloads and lack of space.
Data Server Tier
 Enable asynchronous I/O
 Consider using Quick I/O or equivalent if using file system
based files for the DB files.
– Quick I/O improves performance and scalability by simulating
raw devices, and avoiding file system cache traffic for the DB
files.
 Use locally managed temp files (uniform) for the
temporary tablespace.
 Install Stats Pack (spcreate.sql) and use spauto.sql to
automate hourly snapshots.
 Stats Pack reports can be used to trend transaction rates
and business flows.
Data Server Tier
 Stats Pack Report (Instance Summary)
STATSPACK report for

DB Name DB Id Instance Inst Num Release Cluster Host

------------ ----------- ------------ -------- ----------- ------- ------------
GSIAP 317772662 gsi1ap 1 9.2.0.5.0 YES agsidbs1

Snap Id Snap Time Sessions Curs/Sess Comment

------- ------------------ -------- --------- -------------------
Begin Snap: 503400 18-Nov-04 11:00:01 ####### .0
End Snap: 503405 18-Nov-04 16:00:04 ####### .0
Elapsed: 300.05 (mins)

Cache Sizes (end)

~~~~~~~~~~~~~~~~~
Buffer Cache: 5,313M Std Block Size: 8K
Shared Pool Size: 2,864M Log Buffer: 1,024K

. . . . . . . . . . . .
Data Server Tier
 Stats Pack Report (Instance Summary)
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
--------------- ---------------
Redo size: 462,767.56 26,266.16
Logical reads: 54,537.81 3,095.50
Block changes: 3,237.12 183.74
Physical reads: 2,403.22 136.40
Physical writes: 272.19 15.45
User calls: 518.96 29.46
Parses: 223.29 12.67
Hard parses: 1.26 0.07
Sorts: 199.15 11.30
Logons: 1.85 0.11
Executes: 1,566.48 88.91
Transactions: 17.62

% Blocks changed per Read: 5.94 Recursive Call %: 88.46

Rollback per transaction %: 3.18 Rows per Sort: 102.78
. . . . . . . . . . . .
Data Server Tier
 Stats Pack Report (Instance Summary)
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.90 Redo NoWait %: 100.00
Buffer Hit %: 96.03 In-memory Sort %: 99.96
Library Hit %: 99.87 Soft Parse %: 99.44
Execute to Parse %: 85.75 Latch Hit %: 99.82
Parse CPU to Parse Elapsd %: 69.57 % Non-Parse CPU: 97.34

Shared Pool Statistics Begin End

------ ------
Memory Usage %: 90.55 76.14
% SQL with executions>1: 66.70 75.37
% Memory for SQL w/exec>1: 60.86 71.08

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
db file sequential read 16,187,270 134,274 31.12
CPU time 110,764 25.67
global cache cr request 21,410,811 66,129 15.33
db file scattered read 3,184,765 23,571 5.46
KJC: Wait for msg sends to complete 10,722,532 20,446 4.74
Data Server Tier
 Stats Pack Report (Wait Events)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
---------------------------- ------------ ---------- ---------- ------ --------
db file sequential read 16,187,270 0 134,274 8 51.0
global cache cr request 21,410,811 56,241 66,129 3 67.5
db file scattered read 3,184,765 0 23,571 7 10.0
KJC: Wait for msg sends to c 10,722,532 194,069 20,446 2 33.8
row cache lock 74,583 5,842 17,796 239 0.2
wait for unread message on b 73,977 4,624 13,054 176 0.2
SQL*Net more data to client 6,021,483 0 8,873 1 19.0
enqueue 114,204 19,898 7,427 65 0.4
buffer busy global CR 516,382 18 4,735 9 1.6
buffer busy waits 434,054 192 3,857 9 1.4
direct path read 1,239,648 0 3,226 3 3.9
inactive session 1,774 1,774 1,784 1006 0.0
latch free 242,171 19,203 1,694 7 0.8
db file parallel read 121,670 0 1,280 11 0.4
direct path write 1,270,008 0 1,110 1 4.0
log file sync 604,002 244 1,097 2 1.9
SQL*Net message from dblink 20,365 0 1,080 53 0.1
Data Server Tier
 Review Top 5 Timed Events section in the Stats Pack
reports to determine the overall instance utilization.
 For latch free waits, latch contention is often a
symptom due to a legitimate problem such as non-
sharable SQL, sub-optimal SQL which performs full
table or full index scans, dynamic object
creation/removal, etc..
– Review the latch Statistics section of the Stats Pack report to
determine the hot latches.
– Enable tracing for a few of the sessions waiting on the latch
as well as the holder to determine the actual cause.
Data Server Tier
 Stats Pack Report (Expensive SQL)
CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
3,413,015 1 3,413,015.0 7.9 1970.29 6716.03 976039043
Module: ALECDC
SELECT distinct ooha.order_number , fu.user_name , rac.cu
stomer_name , to_char(wctc.last_update_date, 'DD-MON-YY hh24:mi
') , DECODE(SUBSTR(wctc.comments, 1, 21),
'NOTE: This Order-Line', wctc.process_flag || ': manually scre
ened', DECODE(wctc.process_flag ,'101','101: host u

2,229,632 1 2,229,632.0 5.2 2115.49 35448.22 1280227044

Module: ALECDC
select distinct pv.segment1 , nvl(pv.vendor_name_a
lt, pv.vendor_name) , ai.invoice_num , hr.name ,
'[email protected]' from po_vendors pv ,
ap_invoices_all ai , ap_holds_all ah , hr_organi
zation_units hr where ai.payment_status_flag||''='Y' and

1,990,002 1 1,990,002.0 4.6 2321.34 23934.41 4063073269

Module: OKILOADDR
BEGIN OKI_REFRESH_PUB.REFRESH_ADDRS( :errbuf, :rc ); END;
Data Server Tier
 Monitor CPU and Memory Utilization
 Monitor paging
 Monitor I/O statistics, including service times.
 Review Stats Pack reports and correlate top
SQL to the business flows.
 Review expensive custom SQL
Data Server Tier
 Gathering Statistics
– Do not gather statistics excessively on entire
schemas or the entire database such as nightly or
weekly.
– Do not gather statistics on permanent objects
during peak intervals.
 Gathering statistics invalidates cursors
 Gathering statistics requires dictionary and object
level locks.
 Plans are not likely to change if the data distribution
has not changed.
Data Server Tier
 Gathering Statistics
– For tables which are growing at a rapid rate, gather
statistics only on those tables.
– Use only FND_STATS or the Gather Schema and
Gather Table Statistics Concurrent Programs
 Do NOT USE the analyze nor dbms_stats command
directly. It is not supported, and results in sub-
optimal plans.
– Review the table and index statistics for the objects
which appear in the top SQL section of Stats Pack.
Data Server Tier
 Gathering Statistics Enhancements (11i10)
– Gather Schema Statistics
 Auto Gather option
­ Gather Statistics only on tables which have changed
­ Change threshold is user definable.
­ Utilizes the Table Monitoring feature. To enable schema
monitoring:
­ SQL> exec fnd_stats.ENABLE_SCHEMA_MONITORING()
 Auto List option
­ Lists the objects which have changed.
 Maintain history of Statistics Collection
 No Invalidate Option
­ Does not invalidate cursors.
Data Server Tier
 Gathering Statistics Enhancements (11i10)
Data Server Tier
 Verifying the Statistics
SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL');
===================================================================================================
Table OE_ORDER_LINES_ALL
===================================================================================================
last analyzed sample_size num_rows blocks
12-03-2004 22:59 3726829 37268290 3527197

Index name last analyzed num_rows LB DK LB/key DB/key CF

----------------------------------------------------------------------------------------------------
OE_ORDER_LINES_N1 12-03-2004 22:12 36018080 87310 2097282 1 5 11719150
OE_ORDER_LINES_N10 12-03-2004 22:12 26519270 68610 2230418 1 4 10949180
OE_ORDER_LINES_N11 12-03-2004 22:12 16310880 96200 11196380 1 1 7668050
OE_ORDER_LINES_N12 12-03-2004 22:12 50 1 20 1 1 19
OE_ORDER_LINES_N13 12-03-2004 22:12 1363149 3132 138681 1 3 547938
. . .
. . .
OE_ORDER_LINES_N5 12-03-2004 22:12 0 0 0 0 0 0
OE_ORDER_LINES_N6 12-03-2004 22:12 158583 410 158670 1 1 85032
OE_ORDER_LINES_N7 12-03-2004 22:12 103459 238 18873 1 2 48075
OE_ORDER_LINES_N9 12-03-2004 22:12 4276030 11060 3101971 1 1 2444110
OE_ORDER_LINES_U1 12-03-2004 22:12 37510470 91790 37510470 1 1 17574490
----------------------------------------------------------------------------------------------------
Histogram Stats
Schema Table Name Status last analyzed Column Name
----------------------------------------------------------------------------------------------------
ONT OE_ORDER_LINES_ALL present 03-12-2004 22:59 OPEN_FLAG
Data Server Tier
 Verifying the Statistics – Column Statistics
SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL',0,TRUE);
===================================================================================================
Table OE_ORDER_LINES_ALL
===================================================================================================
last analyzed sample_size num_rows blocks
12-03-2004 22:59 3726829 37268290 3527197

----------------------------------------------------------------------------------------------------
Column Stats
Column name sample_size num_distinct num_nulls density last analyzed
----------------------------------------------------------------------------------------------------
ACCOUNTING_RULE_DURATION 0 37219060 0 12-03-2004 22:51
ACCOUNTING_RULE_ID 1882057 72 18398490 .013888888 12-03-2004 22:34
. . .
BOOKED_FLAG 3724692 2 0 .5 12-03-2004 22:51
. . .
CANCELLED_FLAG 3724692 2 0 .5 12-03-2004 22:51
. . .
LINE_ID 3721906 37219060 0 .000000026 12-03-2004 22:34
LINE_NUMBER 3721906 5649 0 .000177022 12-03-2004 22:34
LINE_SET_ID 10420 18873 37114860 .000052985 12-03-2004 22:34
OPTION_NUMBER 1350665 1176 23740270 .00085034 12-03-2004 22:51
ORDERED_ITEM_ID 3210194 50265 5117120 .000019894 12-03-2004 22:34
UPGRADED_FLAG 3125945 2 5987470 .5 12-03-2004 22:51
VEH_CUS_ITEM_CUM_KEY_ID 0 37219060 0 12-03-2004 22:34
VISIBLE_DEMAND_FLAG 668876 2 30558160 .5 12-03-2004 22:51
Data Server Tier
 Enable the Auto memory manager
– Pga_aggregate_target can be set to tune the PGA for the entire instance.
– Automatically tunes hash area and sort area sizes
– Returns unused memory to the OS.
 Enable System Managed Undo
– No more ORA-1555s!!!
– Set undo_retention to the length of the longest running request.
 Utilize Auto Segment Tablespace Management
– No need to manually set Freelist/freelist groups. A bitmap is used to
automatically scale the segment related parameters.
 Enable PL/SQL Native Compilation
– Improves PL/SQL execution performance.
– Improves scalability by removing a lot of stress off of the shared pool.
Data Server Tier
 New Tablespace Model for the eBusiness Suite:
– Consolidates the number of Apps tablespaces from 400
tablespaces to 10.
– Optimizes performance via the wide stripe model.
– Ideal for customers migrating to RAC or migrating to raw
devices.
– Uniform extents, locally managed
 No more fragmentation!
– Migration script can be used to migrate a schema at a time.
Data Server Tier
 New Tablespace Model for the eBusiness Suite:
– Objects are classified by object type and access patterns:

 Transaction tables
 Transaction indexes
 Reference/SEED
 Interface
 Summary Management
 Archive
 NoLogging
 Queue
 Media
 Temporary
 Undo
 System
 Total = 12 Tablespaces
Data Server Tier
 Applications Tablespace Migration Utility
– MetaLink note 248857.1
– ARU # is 3381489
 Migration timings: (complete migration)
– 1.4 Terabyte customer DB
 24 hours
 New size of DB after migration (850 GB).
– 250 GB customer DB
 4 hours
 New size of DB after migration (150 GB).
Application Tuning
 Review the MetaLink note “Recommended
Performance Patches for the Oracle E-
Business Suite”, 244040.1.
– Recommended performance patches for all the
modules and tech. stack components are
consolidated in this note.
Application Tuning
 Workflow
– When starting background engines via Concurrent Manager,
set the Process Stuck parameter to No.
– Start a separate background engine to handle stuck
processes with a very low frequency such as once or twice a
day.
– Use deferred activities to improve online response times and
facilitate asynchronous processing for flows such as
Scheduling, PO Document Approval, etc..
– Establish a regular frequency for purging Workflow runtime
data (FNDWFPR - Purge Obsolete Workflow Runtime Data).
Application Tuning
 Order Management
– OM Pack I contains numerous performance fixes
including Pricing performance fixes.
– Ensure that the profile “OM: Debug Level” is set to
zero (0).
– Ensure that the profile “QP: Debug” is set to “N”.
– Use the streamlined workflows.
Application Tuning
 Payroll
– Tune the number of threads, chunk size and the buffer sizes
(pay_action_parameters):
 THREADS=25
 CHUNK_SIZE = 20
 BAL BUFFER SIZE=550
 EE BUFFER SIZE =550
 RR BUFFER SIZE =550
 COST BUFFER SIZE=550
– Set threads to 1.5-2.0 times the number of data server
processors.
– Set LOW_VOLUME=N to enable optimal plans.
Application Tuning
 OA Framework Applications
– Refer to MetaLink note 123456.1 (Recommended Patches for
Applications) and MetaLink note 275880.1 (Framework Roadmap).
– If running FWK 5.7, ensure that you are running the latest rollup patch
for 5.7H.
 Refer to MetaLink note 258333.1.
– Upgrade to the latest 9iR2 JDBC Drivers (MetaLink note
164317.1)
– Pool Monitor can be used to monitor the resource utilization
 Application Module Pool
 Memory Utilization
 JVM Properties
 Pre 11i10: https://<hostname>/servlet/OAAppModPoolMonitor
 11i10: Login->Diagnostics->Show Pool Monitor.
Application Tuning
 OA Framework Applications (Pool Monitor)
Application Tuning
 OA Framework Applications (Pool Monitor)
Application Tuning (OA)

Tracing Framework Applications

– Set profile FND : Diagnostics to Yes at user level
– Login to Self Service as the above user
– Click on Diagnostics icon at the top of page
– Select ‘Set Trace Level’ and click Go
– Select the desired trace level and click Save
– Perform the activity that you want to trace
– Exit application
Application Tuning (OA)
Tracing Framework Applications
Application Tuning
 AOLJ Connection Pool (dbc configuration)
– $FND_TOP/secure/<hostname_sid>.dbc
 Number of DB connections used by the Java based Self-
Service Applications.
­ Start with a reasonable default such as 100 or 200.
FND_JDBC_MAX_CONNECTIONS=200
­ Tune FND_JDBC_MAX_CONNECTIONS as per the
amount of user concurrency (per JVM).
 AOLJ automatically decays idle connections and resizes the
pool.
 Disable sanity checks in production environments
­ FND_JDBC_USABLE_CHECK=false
­ FND_JDBC_PLSQL_RESET=false
Application Tuning
 AOLJ Connection Pool (dbc configuration)
– AOLJ Supports the use of Services for RAC load
balancing or failover.
 Patch 2566387 allows the use of service names in
the configuration:
­ New dbc parameter APPS_JDBC_URL
­ Used in place of DB_HOST, DB_NAME,
DB_PORT
­ Useful for automatic connection load balancing

APPS_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=ap201ops)(PORT=1526))(ADDRESS=(PROTOCOL=TCP)
(HOST=ap202ops)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=opsperf)))
Application Tuning
 AOLJ Connection Pool Monitoring
– Use the JSP (AoljDbcPoolStatus.jsp) to monitor the
connection pool status.
 Lists DBC configuration parameter settings.
 Connection Statistics
­ Locked Connections
­ Available Connections
­ Closed Connections
­ Allows drill-downs to determine source of the
connection.
– http://apptier.oracle.com:1200/OA_HTML/jsp/fnd/AoljDbcPoolStatus.jsp
Application Tuning
 Connection Pool Status (AoljDbcPoolStatus.jsp)
Application Tuning
 Purging
– Implement a regular purge process including
transient data as well as obsolete transactional
data.
– Archive system related entities prior to purge such
as
 FND_CONCURRENT_REQUESTS
 ICX_SESSIONS
– Purge old data before performing upgrades.
Application Tuning
 Purging
Product Concurrent Program Name

FND Purge Concurrent Request

and/or Manager Data
Purge "Sign on" Audit data
General Ledger GL Archive and Purge
Assets Mass Additions Purge Report
Purge accounting tables
Payables Purge
Receivables Archive and Purge
Order Management Order Purge
11i10 Performance Improvements
Order Management
– Significant improvements in pricing and order
management.

40000 37,500
35000
30000
25000
19,000
20000 11.5.9
11.5.10
15000
10000
5000
0
HVOP Order Import Throughput
11i10 Performance Improvements
Order Management
– New Java Pricing Engine

25,000
25000

20000

15000
11.5.9
9,200 11.5.10
10000

5000

0
Order Throughput with Pricing
11i10 Performance Improvements
Inventory / Shipping
– 20% improvement in Pick Release.
– 25% improvement in Interface Trip Stop.
– 2X improvement in Ship Confirm.
11i10 Performance Improvements
 Workflow Improvements
– Java Mailer
Parallel worker support
15,000 notifications per hour (per worker)
– Workflow engine runtime
Improved Caching
­ Asset Import (5X)
­ Order Management (5%)
– Integration with GSM
Business Event Listener
Mailer
11i10 Performance Improvements
 Payroll Improvements
– New Run Balances Architecture (HRMS FP F)
– Improves reporting performance significantly (~90%):
TRR – 95%+
Payroll Register – 90%+
– US and Canadian Forms and Reports utilize the new Balance
Reporting Architecture.
– Custom reports which utilize the balance user exit also benefit
from the new run balances arch.
– Payroll run improves by 15%.
Suppression of zero and null balances provides an
additional 10% performance improvement.
11i10 Performance Improvements
 TCA Customer Import
– Completely rewritten.
– Requires 9iR2 (or higher).
– Scales well with parallel workers.
– 300K customers per hour per worker
– 1 million customers imported in one hour using 4
workers.
11i10 Performance Improvements
 TCA Customer Import

1,000,000
1000000
900000
800000
700000
600000
500000 11.5.9
400000 11.5.10
300000
200000
100000 10,000
0
Customer Import Throughput
11i10 Performance Improvements
 11i10 Tracing Improvements
– Ability to enable all levels of SQL trace as well as PL/SQL profiler
from Forms, Framework based pages, and Concurrent Programs.
– Help->About Oracle Applications shows performance information
including time spent in the Form and time spent in the DB.
– For concurrent programs, tracing can be enabled at the program
definition level and at the request level.
– Trace file identifier is automatically set to the Applications user name
prod11i_ora_2738_BASMITH.trc
– New Report (PL/SQL Profiler)
Generates an HTML report of the PL/SQL profiler data.
11i10 Performance Improvements
 11i10 Tracing Improvements (PL/SQL Profiler Report)
11i10 Performance Improvements
 11i10 Tracing Improvements (PL/SQL Profiler Report)
11i10 Performance Improvements
 11i10 Tracing Improvements (Performance Info.)
11i10 Performance Improvements
 FND_STATS Improvements:
– No invalidate option can be used to avoid invalidating cursors
while gathering statistics.
– Auto Gather has been added to allow statistics to be gathered
only on those objects which have changed.
Change threshold is configurable.
Default is 10%.
Significantly reduces the time to gather statistics.
Improves manageability as FND_STATS automatically
determines changed object list.
– Maintain history of all gather stats requests.
– Automatically determines degree of parallelism.
11i10 Performance Improvements
 Configuration Enhancements
– Reviewed and optimized default settings for the entire stack via
AutoConfig templates including:
Forms
JDK
DB
Concurrent Manager
Discoverer
AOLJ DBC
– Customer Sequence Cache settings are preserved across
patches/upgrades if current setting exceeds standard default.
– AutoConfig support for complex configurations including RAC,
Load Balancing, Firewalls, etc..
11i10 Performance Improvements
 Significant reduction in patching downtime
– Parallelized large table updates.
– FNDLOAD Improvements
15% for small LDTs
Orders of magnitude for large LDTs.
– Improvements to distributed AD.
– Large indexes are now created in parallel.
– APPS_MRC schema has been eliminated.
– Optimized PL/SQL recompilation order for invalid objects.
Reminder –
please complete the OpenWorld
session survey

Thank you.
Other Relevant Sessions
Partitioning and Purging Best Practices for the
Oracle E-Business Suite
– Thursday December 9, 2004 4:00 PM, Room
3014.
Streamline Maintenance and Reduce
Downtime
– Thursday December 9, 2004 11:00 AM, Room
3014.
Oracle Database 10g Is Ready for ILM: Are
You?
– Thursday December 9, 2004 1:00 PM, Room 102
MetaLink References
Note #: Description:
123456.1 E-Business Suite Recommended Patch List
125767.1 Upgrading Developer 6i with Oracle Applications 11i
138159.1 Canceling Long Running Queries in Oracle Applications 11i
164317.1 Upgrading Oracle JDBC Drivers with Oracle E-Business Suite 11i
215268.1 Implementing and Using the JSP Precompiler
216205.1 Database Initialization Parameters and Configuration for Oracle
Applications 11i
244040.1 Recommended Performance Patches for Oracle E-Business Suite
246105.1 Using J2SE Version 1.4 with Oracle E-Business Suite 11i, Release 11.5.9
or Higher
248857.1 Oracle Applications Tablespace Model Release 11i - Tablespace Migration
Utility
258333.1 About OA Framework Mini-pack 11i.FWK.H
275880.1 Oracle Applications Framework Release 11i Documentation Road Map
Q&
A
QUESTIONS
ANSWERS