Performance test automation using IBM Rational BuildForge and IBM Rational Performance Tester

IBM Testing Services

TTQ
Amitabh Kaushal Performance Test Manager Vivek Sharma Performance Test Lead 20 November 2010
2008 IBM Corporation @ 2009 IBM Corporation

IBM Testing Services

Agenda Case Study Introduction & Background Performance Test Life Cycle Urgence Performance Framework Automation of Performance Test Test Execution Sequence Comparison with Urgence Performance Test Cycle Major Benefits of using Urgence Framework Summary

2009 IBM Corporation

IBM Testing Services

Largest European Bank in UK : Case Study

Client Business Areas Served Engagement Details

Benefits realised
1. 50% Reduction in Performance Regression Test Execution Timelines Indexing/cataloging of Regression Test Pack Complete Automated Solution Effective risk-based approach 1. SLA-Based Performance KPIs Test Quality Schedule Adherence Test Effectiveness 1. Significant Knowledge Management Improvements Reduction in reliance on Customers SMEs Greater flexibility in test execution 1. On-track for 16% reduction in fixed-costs of Service 2. Service Improvement initiatives around Performance Testing effectiveness and methods Further Test Automation in Other projects Driving Quality Methods further up the SDLC

Trades & Settlement Type of Service : Managed Test Services Lifecycle - Trade Capture & vesting, IBM is providing managed testing services, to assist one of the largest European bank in performing Largest Messaging like offshore Managed Testing Services for trading and European SWIFT, CREST settlement application In financial market service. Bank Positions & Balance, IBM responsibility: Settlements, Provide expertise in testing transition in strategic level Statutory reporting Responsible for transition planning and execution and Reconciliations Help obtain knowledge transfer and documentation Global Markets from Customer during transition period Derivatives Designs test process and framework Formulate test strategy, test plan and requirement traceability Develop test scenarios, functional test cases, test scripts, execute testing, record and compile results Performance Testing, Test Automation and Regression improvement Day-to-Day management of testing services IBM has successfully completed testing of several releases of developed applications

2009 IBM Corporation

IBM Testing Services

Introduction and Background

Customers application is a Trades Settlement Platform which processes ~2.5M trades/day, supports 33 Back Offices in all major financial Geographies, Europes biggest Oracle Database IBM currently delivers a range of Testing Services for this Platform. Customer challenged IBM to provide specialised, on-demand, repeatable and cost effective (offshore) performance test service for this application. This will be a different delivery model to SIT, Regression testing etc. IBM did deliver a repeatable, on-demand and cost effective Performance Regression Testing solution with the following attributes: Cover end to end Performance Test Lifecycle (Test Preparation Shakedown Execution Interim Reporting - Final Reporting), using IBM Rational tooling and IBM Research innovation. Test Phase duration got significantly reduced (target was 50%) An automated chassis with rules-based Environment deployment, performance tests automation capabilities, threshold driven automated report generation. Based around the capabilities of IBM Build Forge product which forms the foundation of the Service Uses IBM's Rational Performance Test (RPT) tooling for test execution of the online testing functionality Combines existing monitoring augmented by Unix level Monitoring tools / integration with OptierCorefirst / online transaction timing metrics. Combined uses of Open Source tool with others to monitoring server side metrics for analysis and scalability - PMRE, SQL Developer, Oracle OEM, NaGIOS etc.

2009 IBM Corporation

IBM Testing Services

Performance Test Life Cycle

P E R F O R M A N C E P e r f o r m a n c e R is k W o rk s h o p D e t a ile d P e r f o r m a n c e W o rk s h o p E x it R e p o r t T e s t P la n LC
Test C1 P r e p a r a t io n C1 C1 R e p o rIm p le m e n t t

The performance test lifecycle primarily consists of: 1. Performance Risk Workshop to decide the scope of the performance testing and the test mitigation for the risks the project will introduce. 2. Workshop Exit Report 3. Detailed Performance test Plan which consists of the scope, approach, risks, dependencies and estimated schedule for the performance test.
5

2009 IBM Corporation

IBM Testing Services

Urgence Performance Framework

Test preparation, Execution and Monitoring are automated with Rational Build Forge and Rational Performance Tester
GUI Application Rational Performance Tester
Test Data

Rational Build Forge Define and use Build Forge to set up Test environment Deploy Test Data Install Test Tools Validate Test Environment Readiness Generating Test Reports

Performance Test Environment Performance Test Environment Solaris, Win, Linux boxes with Solaris, Win, Linux boxes with Database Servers Database Servers

Storage of RPT scripts and data Preparation, Execution and monitoring of Performance Test

BF Agent Open Source Tool for monitoring

2009 IBM Corporation

IBM Testing Services

Automation of Performance Test Preparation, Execution, Monitoring

Test Preparation Prepare test input files using IBM Build Forge Execute Performance Health Check using IBM RPT, BuildForge, Open Source tool for monitoring Environment validation Automatically raise JIRA/GIMS (incident management system) if required processes are not running in the test system Test Execution Schedule multiple Test runs for various regions APAC, EUROPE, USA 12-20 hrs online tests running in-parallel with IBM Rational Performance Tester. This whole end-to-end execution is fully automated now. Test Monitoring & Reporting Continuous monitoring of different databases to assess the trade flow using OEM, SQL Developer, build forge Server resource utilization monitoring using PMRE and native Solaris commands - Prstat/Top, IOstat, netstat. Use in-house shell scripts to capture channel throughput. Raise JIRA automatically and send notification mail to stakeholders.

2009 IBM Corporation

IBM Testing Services

Urgence Test Execution Sequence

Following scheduled tasks are driven by Buildforge comprises of the entire performance test steps
T:1 T: 1 T: 1 T: 1 T:1 T: 1 T: 1 T:1 T:1 T: 1 T : 1 1 T : 1 1 T : 1 1T : 1 1 T : 1 1 T : 1 1T : 1 1 T : 1 1T : 1 1 T : 1 1T : 1 1 T : 1 1 T : 1 1T : 1 1 T : 1 1

Server Monitoring 1 minute ? Loop) ( eSpear R equired Process Monitoring 1 1( minute? Loop)

Test Execution Scheduling

Test execution consists of running online tests for 12 hrs in parallel to the intraday tests for various regions at scheduled time. The servers are monitored the entire duration of the tests to view CPU utilisation. These tasks are sequenced as shown and executed at the specified time in Build Forge.
US Batch Intraday Results O nline T est Results Batch R esults

O nline R PT T esting T K Intraday T K Settlement T K Monitoring T K Batch HK Intraday HK Monitoring

H K Batch UK Intraday U K Settlement U K Monitoring U K Batch U S Intraday US Monitoring

2009 IBM Corporation

IBM Testing Services

U R G E N C E P e r f o r m a n c e T e s t E x e c u t io n P l a n & A c t i v i t y C h e c k lis t
P l a n n e d la n n e d A c t u a l A c t u a l P P r e r e q u is it e S t e pS t a r t T im e r a t i oSnt a r t T i m n d T i m e Du Ee O w ner S te p s ( B S T ) ( H H : M M )( B S T ) ( B S T ) 11 11 11 11 00 11 00 11 11 11 11 11 11 11 11 11 11 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 0 00 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : 0 00 : 1 11 : 1 11 : 1 11 : 1 11 : 1 11 : NA 11 NA 11 11 11 11 11 NA 11 11 11

J IR A

E x e c u t io n - O n li n e , In t r a d a y ( T K /H K /U K /U S ) , SBt aat u c h , A ses tutmepmi oenns t/C o m m e n ts S l t s

s R a is e J IRB o to n c e w e b A p p li c a t i o n hso & t e dl uosnt eerS P E A R A p p S e r v e r s fo r A u s C IB M U r g e n c eO P 0 -0 0 0 0 P S C o m p le t e d P e r fo r m a n c e T e s t DSOPS IB M U r g e n c e IB M U r g e n c e D S O P S w ill b o u n c e th e s e r v e r a n d n o tify u s . T h e y w ill ta k e h e lp m p le t e d te a m if n e e d e d . C o fr o m T A

C h e c k B u s in e s s d a t e s f o r a ll r e g io n s C o m p le t e d C h e c k P u b lic H o lid a y fo r c u r r e n t w e e k C o m p le t e d If to d a y 's d a te ( F u ll- r u n ) is a P u b lic H o lid a y f o r a n y r e g io n th e n r a is e J I R A to c h a n g e th a t IB M U r g e n c e NA d a y t o n o r m a l b u s in e s s d a y . DSOPS D S O P S w ill c h a n g e th e d a y a s b u s in e s s d a y f r o m h o lid a y a n d nN A u s . o tify C s a n n e l S p a c rea is IRdAa g a in s t T1 o n T + d a y , u p d a te1it fo r T + d a y h of Je C h e c k s ta tu A IB M U r g e n cO E T 1 -1 1 1 e A C o m p le t e d c h a n n e l s t a tu s TA O E T 1 -1 1 A w ill c h e c k th e c h a n n e l s p a c e a n d p u r g e th e d a ta , if n e e d e C .o m p le t e d A T1 d IB M U r g e n c e C h e c k B F A g e n t s S ta tu s o n a ll b o x e s in U A T C o m p le t e d TA C h e c k B F A g e n t s S ta tu s - I f d o w n , e s c a la t e it t o T A NA IB M U r g e n c e TA C h e c k th e s ta t u s o f A p a c h e S e r v e r s , If s e r v e r s t a tu s is d o w n ,Cr o m e le tE d A J IR A a g a in s t T A a is p O e T

111 1 11 : 111 1 11 :
9

T A w ill b r in g u p t h e A p a c h e s e r v e r a n d n o tify u s NA o R a isO E D B A Jt IR R u nB in v a lid r Roenp U A T e n v a n d m a k e s u r e D a ta b a s e is in e o AD t NA D B A s O E D B1 1 1 1 1 AC o m p le t e d g o o d h e a lt h b y c o n fir m in g w it h C o m p o n e n t T e a m 11 C o m p o n e n ts If D B In v a lid c a n c a u s e p r o b le m , c o m p o n e n t w ill d e p lo y r e c o m p ile p b ats e lin e . C o m le e d A NA IB M U r g e n c e C h e c k Jfo rV A P opl le rc e s s s ta t u s , r a is e J I R A if n e e d e d ro C o m p le t e d sk e e C h e cMk M t a c o u n t in taxbcle p t i o n _ l o g , m m _ o p e n t r a d e , m m _ a lle g e , NA IB M U r g e n c e C o m p le t e d m m _ e n g i n e _ r easnodu trrc e c a t e th e m if n e e d e d . un 11 D B A s O E D B1 1 1 a is1 r e q u e s t t o D B A te a m to t r u n c a t e th o s e ta b le if d o n t h a C eo m cpcle tsesd to d o s o . A- R1 e v a e C h e cTka s k s u b s c r ib eta k e a c t io n to c le a r it, if n e e d e d t a k e s u p p o r t fr o m S e t tle m e n t and 11 IB M U r g e n c e C o m p le t e d te a m U rg e n c e + T A + s 111 M a k e s u r e th o s e T a s k C h a n n e l a r e c le a r e d . NA e t tle m e n t
2009 IBM Corporation

IBM Testing Services

Process Monitoring Reporting Environment Status

10

2009 IBM Corporation

IBM Testing Services

Standard Performance Test Cycle

ID
1 1 1

Ta s k N a m e D e p lo y /R e fre s h Pre p a ra tio n Te st Ex e c u tio n

Sta rt 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 0 / 00 0 00 / 0 0

Fin ish

D u ra tio n 1 d 1 d 1 d 1 d 1 d

1 D ec 1 1 1 11 1 1 1

1Jan 1 1 11 1 1 1 1 1 1 1 1

1 J an 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 Jan 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 Jan 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1F eb 1 1 11 1 1 1 1 1 1

1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1

1 An a ly sis
1

R e p o rts

Urgence Performance
1 D ec 1 1 1 11 1J an 1 1 11 1 1 1 1 1 1 1 1 1 J an 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J an 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J an 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1F eb 1 1 11 1 1 1 1 1 1

ID
1 1 1 1

Ta s k N a m e D e p lo y /R e fre s h Pre p a ra tio n Te s t Ex e c u tio n & An a ly sis R e p o rts

Sta rt

Fin is h

D u ra tio n 1 d 1 d 1 d 1 d

1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 0 / 00 0 00 / 0 0 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 1 / 11 1 11 / 1 1 0 / 00 0 00 / 0 0 1 / 11 1 11 / 1 1

~2.2 weeks cycle time reduced to 5 days 6 days effort saved on plan per Cycle (typically critical-path) 60 days saved on plan per year (based on 4 releases per year) and being utilised for more run now Within Test Execution Task there can be many runs, and therefore more robust performance metrics
11

2009 IBM Corporation

IBM Testing Services

Major Benefits of using Urgence Performance Framework

Re-usability and Modularity - Used Build Forge feature to create common projects across environments wherever it is applicable - Parallel execution possible at a time in 2 or more environments Reduced / Eliminated dependency - Reduce Support Team and SME involvement for providing result. - Usage of licensed software like TeamQuest, Loadrunner etc. - Dependency on Technical Architect and DBA group to provide data and analyse result - Performance tester involvement in checking process down and raising JIRA accordingly, reduce tester effort - Resource Utilization (Server Stats) using external tool like TeamQuest, Analyzer Automation Reduction in manual efforts - Creation of test data (bulk trade files and online data) - Trade Monitoring (to collect vesting results) - Use Multi-threading concept to speed up the execution process - Can run unattended performance test (using scheduling feature in BF) Central Repository for all test results
12

2009 IBM Corporation

IBM Testing Services

With Urgence Framework, Customer has achieved significant

Reduction in test timelines Shorter test execution runs (5 days for 3 run) Fewer attempts to get to first good run Get the analysis done and find the bottleneck by comparing different run results

Increases time dedicated to analysis Automated testing will provide answers for how does release Y equate to release X Opens the door to aggressive tuning activities tweak a variable, run the test, tweak again, run the test. Raise the consistency and confidence in Performance testing.

Reduction in human resource costs Resources running the tests will have lower rates completely offshore SMEs will spend less time on Performance Testing Reduced dependency on Support Team(TA, DBA) to fetch baseline stats Eliminating manual, error prone tasks Repeatable and consistent processes for improved efficiency

13

2009 IBM Corporation

IBM Testing Services

Summary
Using Build Forge, RPT and Open source tool we have automated the end to end performance testing for customers application. We have demonstrated reduction in overall test cycle time, ability to execute the tests 'On Demand' in a cost effective way. We have reduced manual efforts by introducing automation (i.e. creation of test data, Trade Monitoring, JIRA, running unattended performance test with the scheduling feature in Build Forge). Performance test scope is now decided early in the release lifecycle by conducting a risk workshop to discuss the non-functional requirements. Specific deliverables like risk workshop exit report, test strategy, test plan are delivered in different phases of the performance test life cycle and obtained a sign off before the execution of tests.

In Summary, we were able to realise the customer's vision for an 'On Demand' performance test capability.

14

2009 IBM Corporation