MASON Performance Tuning Guide

Author: Manish Pandey

Creation Date: Sep 26th 2018
Last Updated: Nov 09th 2018
Control Number:
Version: 1.1

Approvals:

Rackspace Manager Shyamal Shah

Mason Manager Dan Raffesberger

Document Control

Mason_Performance_Tuning_Guide.docx
Page 1
Change Record
Date Author Version Change Reference

27-Jul-16 Manish Pandey 1.0 Based On performance issue ticket ITK#

1232537
09-Nov-2018 Manish Pandey 1.1 Added Issue from ITK#1260843

Reviewers
Name Position

Shyamal Shah Manager, Rackspace

Amit Raghuvanshi Lead, Rackspace
Venkat Jaganathan Delivery Center Manager, Rackspace

Distribution
Copy No. Name Location
1 Issue Track http://support.tricoresolutions.com

Mason_Performance_Tuning_Guide.docx
Page 2
TABLE OF CONTENTS

TABLE OF CONTENTS................................................................................................................3
Section A. Introduction....................................................................................................................4
Section B. Performance guide.........................................................................................................6

Mason_Performance_Tuning_Guide.docx
Page 3
Section A. Introduction

Objectives and assumptions

The objective of this document is to provide a comprehensive list of all
checklist and steps necessary to check repeating performance issues in
Mason EBS PROD

Document structure
This document has following structure:
Section 1 This section will list all the checks and fixes to be
performed on EBS PROD to resolve performance issue

Related Documents
This guide is based on and should be used together with the standard
Oracle database and Applications document sets provided by the
Oracle Corporation.

Mason_Performance_Tuning_Guide.docx
Page 4
 Performance Tuning Checks and Steps

(1) Issue-1 :

It is seen that at certain times XXMSN_WAVE_RELEASE_CONC concurrent

program and its child program Pick Selection List Generation start queuing up in EBS
PROD. This program usually completes in minutes but at the time of performance
issue it starts taking tens of minutes

Concurrent Program Name Expected Runtime

XXMSN_WAVE_RELEASE_CONC normal is 1-2 minutes
Pick Selection List Generation normal is sub-second

(a) Check for blocking locks and clear them if INACTIVE for long time.

SELECT inst_id,DECODE(request,0,'Holder: ','Waiter: ')||sid sess,id1, id2,

lmode, request, type from gv$lock WHERE (id1, id2, type) IN (SELECT id1,
id2, type FROM V$LOCK WHERE request>0) ORDER BY id1, request;

(b) Check if GSS is run on ALL schemas and custom schemas like XXMSN

SQL> SELECT MAX(LAST_ANALYZED) FROM DBA_TABLES where

OWNER='XXMSN';

MAX(LAST_ANALYZ
---------------
23-SEP-18

(c) Check the concurrent queue pending request count

USER_CONCURRENT_QUEUE_NAME MAX_PROCESSES
RUNNING_PROCESSES COUNT(CWR.REQUEST_ID)
------------------------------------------------------- ------------- -----------------
---------------------
MRP Manager 1 1 1
Conflict Resolution Manager 1 1 140
Mason Trailer Close Manager 7 7 2379
Standard Manager 25 25 7
Mason Report Manager 15 15 2
Mason Wave Manager 50 50 1009

6 rows selected.

Mason_Performance_Tuning_Guide.docx
Page 5
 (d) Check which SQL_ID is being run by concurrent Programs

select sid,serial#,status,last_call_et,module,event,action, sql_id from v$session

where sid='80' order by LAST_CALL_ET desc

select sql_text from V$sqltext where sql_id='1wwj02vwnmpnt' order by piece;

(e) Check is there are more than one execution/hash plans for discovered
specific SQL_ID

SQL> select distinct(PLAN_HASH_VALUE),inst_id from gv$sql_plan where

sql_id='1wwj02vwnmpnt';

PLAN_HASH_VALUE INST_ID
--------------- ----------
2885720150 2
2885720150 1

From OEM check which plan is best and use the best one.

(f) Run SQL Tuning Advisor from OEM for specific SQL_ID to see
suggestions on performance improvement plans . Running sql tuning
advisor may also give a new better plan, for implementing new better
plan, we need to take approval from Ryan before implementing it

(g) Check if any profile is enable on current plan, if so , try disabling it from
OEM ( Please do not delete profile option )

SQL> select SQL_ID,PLAN_HASH_VALUE,SQL_PROFILE from v$sql

where sql_id='1wwj02vwnmpnt';

SQL_ID PLAN_HASH_VALUE SQL_PROFILE

------------- --------------- ----------------------------------------------------------------
1wwj02vwnmpnt 2885720150 SYS_SQLPROF_0346ee0b0628000c

1 row selected.

(h) Check if the sql_plan now for the sql, has picked up new execution plan
automatically or not. Monitor to see if the performance has improved

Mason_Performance_Tuning_Guide.docx
Page 6
 (2) Issue-2 :

Found that Concurrent manager "Mason Trailer Close Manager" has about 4000+
requests pending normal

Found that most of the pending requests are for program "Interface Trip Stop"

FYI – it is normal to see 4000+ of these request queued up at certain times throughout
the day when we close out a large batch of trailers; however, the Interface Trip Stop
concurrent program appears to be running longer than expected (due to system slowness),
which seems to have the queue backed up more than normal today

(a) Check for blocking locks and clear them if INACTIVE for long time.

SELECT inst_id,DECODE(request,0,'Holder: ','Waiter: ')||sid sess,id1, id2, lmode, request,

type from gv$lock WHERE (id1, id2, type) IN (SELECT id1, id2, type FROM V$LOCK
WHERE request>0) ORDER BY id1, request;

(b) Check if GSS is run on ALL schemas and custom schemas like XXMSN

SQL> SELECT MAX(LAST_ANALYZED) FROM DBA_TABLES where

OWNER='XXMSN';

MAX(LAST_ANALYZ
---------------
06-NOV-18

(c) Check the concurrent queue pending request count

USER_CONCURRENT_QUEUE_NAME MAX_PROCESSES PENDING_STANDBY

PENDING_NORMAL RUNNING_NORMAL
---------------------------------------- ------------- --------------- -------------- --------------
Conflict Resolution Manager 1 36 0 0
Inventory Manager 5 0 1 0
Mason Trailer Close Manager 25 0 3896 25
Mason Report Manager 15 0 0 6
Standard Manager 25 36 40 2
MRP Manager 1 0 1 0
OAM Metrics Collection Manager 1 1 0 0

7 rows selected.

(d) Check which SQL_ID is being run by concurrent Programs

Mason_Performance_Tuning_Guide.docx
Page 7
select sid,serial#,status,last_call_et,module,event,action, sql_id from v$session where sid='80'
order by LAST_CALL_ET desc

select sql_text from V$sqltext where sql_id='3j9t85nz92gxj' order by piece;

(e) Check is there are more than one execution/hash plans for discovered specific
SQL_ID

SQL> select distinct(PLAN_HASH_VALUE),inst_id from gv$sql_plan where

sql_id='3j9t85nz92gxj';

PLAN_HASH_VALUE INST_ID
--------------- ----------
18556180 2
18556180 1

(f) Run SQL Tuning Advisor from OEM for specific SQL_ID to see suggestions on
performance improvement plans . Running sql tuning advisor may also give a new better
plan, for implementing new better plan, we need to take approval from Ryan before
implementing it

(g) Tunning Advisor suggested SQL Profile with 99.94 % benefit

(h) Requested approval from Ryan to add suggested profile, once we have approval we can
implement profile

SQL_ID PLAN_HASH_VALUE SQL_PROFILE

------------- --------------- ----------------------------------------------------------------
3j9t85nz92gxj 18556180 SYS_SQLPROF_0146edc8a3360004

SQL> !date
Wed Nov 7 13:56:48 CST 2018

(h) Monitor instance

Mason_Performance_Tuning_Guide.docx
Page 8