Westfield Specialty

Insurance
Specialty

Performance Test
Strategy

October 2022
Performance Test Strategy - Agenda

Topics

• Overview and Scope

• Release Timeline
• Key Milestones
• Performance Test Approach
• Workload Model
• Entry-Exit Criteria
• Roles and Responsibilities
• Tools & Environment Details
• Appendix
 Overview & Scope

Performance Test is essential in fast paced IT transformation programs as it helps with the following:
1. Mitigates the risk of application failure and lowers the cost of operations and ongoing maintenance.
2. Performance bottlenecks are identified and addressed before the application goes live rather than reacting to damaging consequences after launch.
3. Address business growth and understand future infrastructure needs by planning hardware capacity according to the performance test results.
4. Future scalability and reliability plans can be built based on performance measures which allows service quality to remain consistent over time.

Performance Testing Goals:

Release 1 – Establish Performance Baseline.
Release 2 – Performance Tuning and improvement

Release 1 Performance Testing Scope:

In Scope Out of Scope

Product : Excess Casualty Functional Testing will be out of scope

Guidewire Systems: Policy Center, Billing Center Following non-performance NFRs will be out of scope
1. Availability
2. Security
3. Data Creation
Environment : PC (Pre-Prod) and BC (Pre-Prod) Evaluating the performance of any client-side rendering.
Executions: Load, Stress & Endurance Test Tuning / Profiling application code for better performance.
In Scope Scenarios: Accounts Search, Policy Search, Contact Search, Quote, Tuning the web server, application server or the database server
Bind, Issue, View Account Summary, Invoice work queue, Invoice Due work
queue

3
Performance Phase - Release Timeline

Performance Testing Phase timeline along with phase/activities has been enlisted with details.

Inception Development Stabilization / Deployment Activities

Release
Sprint Readiness Sprint 1 (3 Week) Sprint 2(3 Week) Sprint 3 (2 Week) Sprint 4A (1Week) Sprint 4B (2 Weeks) Sprint 5 (2 Weeks) Sprint 6 (2 Weeks) Sprint 7(2 Weeks) Sprint 8 (2 Weeks)

Start 01-Aug 16-Aug 06-Sept 27-Sept 11-Oct 18-Oct 1-Nov 15-Nov 29-Nov 13-Dec

End 15-Aug 05-Sept 26-Sept 10-Oct 17-Oct 31-Oct 14-Nov 28-Nov 12-Dec 26-Dec

Sprint QA

QA Performance Perf Performance

Prep & Scripting Execution Performance
Design Closure (Nov 18)
Go-Live (Dec 13)

UAT In Sprint UAT

Early UAT UAT Prep UAT Execution

Deloitte QA/Westfield UAT & Business Team UAT Team Program QA (Deloitte)

Performance Phase Checkpoints - Assumptions/ Dependencies to support timeline:

• Test Planning – 19th Sept 2022 – 14th Oct 2022 • Guidewire Team to monitor application and database server during
• Performance Scripting : 17th Oct 2022 – 31st Oct 2022 performance execution.
• Performance Execution: 1st Nov 2022 – 15th Nov 2022 • Guidewire / Dev team to tune the application if any performance defects are
identified.
4
Key Milestones

Performance Phase detailed key milestones for all activities with ownership of Planning, Design, Scripting and Execution
from Deloitte team.

Planned Planned
Planned Activities Status Owner
Start Date End Date

Planning and Estimation

Performance Scenarios, Tool and Environment Identification 19 Sep 2022 23 Sep 2022 Completed Deloitte Performance Team

Performance Test Strategy and Planning 26 Sep 2022 14 Oct 2022 In Progress Deloitte Performance Team

Test Designing & Prep

Performance Test Script Development 17 Oct 2022 31 Oct 2022 Not Started Deloitte Performance Team

Script Validation Dry Run 17 Oct 2022 31 Oct 2022 Not Started Deloitte Performance Team

Performance Test Execution

Performance Test Execution (Load, Stress & Endurance) 2 Nov 2022 15 Nov 2022 Not Started Deloitte Performance Team

Performance Analysis Reporting and Sign-Off

Performance Analysis, Recommendations and Sign Off 7 Nov 2022 18 Nov 2022 Not Started Deloitte Performance Team

5
Performance Testing Approach

Key testing activities for performance testing approach is outlined below

Tool Setup HW/Environment Optimize Optimize Environment

• Install Tool • Ensure environment is configured and sized • Optimize HW, OS, App configuration
• Install load injectors properly • Remove problematic pages from test
• Validate connectivity and network latency • Manual exercise of all functions scripts
• Smoke test • Baseline system resources • Rebalance tests
Current State Analysis
• Measurement Areas
• Measurement Tools

Test Plan Design Tests

• Develop Performance Test Plan • Define detailed test cases required for
• Determine use cases, click paths, optimized test
page impressions, etc.… • Determine data and data refresh
• Determine data required requirements
• Determine user count, session • Create or load transactional data
definition, session duration, • Configuration to support Unit Tests
weighted distribution of Test Cases • One user
transactions • Manual test script run • Limited transactions
• Assess existing hardware sizing • Record test scripts • Best page performance
assumptions and/or proportional • Smoke test scripts • Page size
hardware load
• Validate network topology and
network considerations
• Determine test cycle plan Load Tests
• Incremental tests with increasing transaction Analyze Results
and session loads • Identify environment pinch points
• Long run loads • Identify and log problematic
areas
• Break - point loads • Assess test weighting and distribution

6
 Workload Model

To further the process of observability, Guidewire provides ways to monitor certain critical areas in InsuranceSuite
Cloud applications. Guidewire calls these critical areas designated functions and provides monitoring hooks in application
code for the designated functions.

Concurrent Users Concurrent Users

Sr. No. Application Function Name (Normal Load) – 100 (Peak Load) – 300
VUsers VUsers
1 Guidewire Policy Center Accounts Search
16 48
2 Guidewire Policy Center Policy Search
12 36
3 Guidewire Policy Center Contact Search
12 36
4 Guidewire Policy Center Quote
12 36
5 Guidewire Policy Center Bind
12 36
6 Guidewire Policy Center Issue
12 36
7 Guidewire Policy Center View Account Summary
12 36
8 Guidewire Billing Center Invoice work queue
6 18

9 Guidewire Billing Center Invoice Due work queue

6 18

7
Entry-Exit Criteria
Entry or Resumption Criteria

The following bullets list the entry criteria to be fulfilled to start or resume Performance Test:

• Test objectives are developed

• Test Planning Session has occurred
• Functionally stable Application is provided
• Performance testing tool license is active and working
• Performance Testing Environment Should be readily available
• Test Data and Volumetric should be present before the Load Test Execution.
• Test schedule is confirmed and agreed
• Controlled Environment for testing is configured and ready

Exit or Suspension Criteria

The following bullets list down the situations that will conclude Performance Testing:

• 90% of SLA are met and all planned performance execution should be completed successfully.
• All performance testing objectives are achieved as documented in the test plan.
• Performance testing analysis and final summary report should be submitted and published.
• Testing results are documented, reviewed and signed off.

8
Roles and Responsibilities

Role/ Team Responsibilities

• Develop test plans, environment needs and execution plans that includes test iterations and schedules
• Develop testing metrics and status reports including test closure report
Deloitte Performance
• Facilitate defect discussions with relevant stakeholder
Lead
• Validate entry and exit criteria before starting /completing Performance phase
• Lead day to day testing discussions including requirements clarifications sessions with Business SMEs/Analysts

5. Design Test 1.1.Analyze

Analyzethe
the
• Provide inputs for Performance Cases DataData
scenarios identification Model
model
based on Sprint testing knowledge
• Test design and execution for performance test cases
Deloitte Performance
• Bug reporting and retest
Tester
• Highlighting risks/impediments to leads to avoid any delay in Performance closure
• Update test design/execution status

2. Analyze the
Deloitte Business
• Clarify any questions on requirements Target Entity
Target Entity
• Participate in test plan, scenarios, test cases review, and defect triage
Analysts
• Help identify data needs

Westfield Product • Address any requirements clarifications that BA team is unable to answer
Owner • Participate in defect triage and provide decisions on defect prioritization
3. Identify the
Transformation
s
Guidewire / Dev Teams • Performance defect fixes to tune the performance if any defects are identified.

9
Performance Tool – LoadRunner Etnerprise

Loadrunner Enterprise Server1 Controllers2

Provides a web-based interface that enables you to design Controllers are servers that define the number of users to simulate, the
performance tests, configure monitors, reserve testing actions they perform, and the machines on which they generate the load.
resources, run and analyze test results.

Application Lifecycle Management (ALM) MI Listeners2

server1 MI Listeners serve as routers between the Controller and PC agents on the client
Provides a platform for core ALM functionality as user side. They enable running a performance test and collecting server performance
management for PC. 5. Design Test 1.1.Analyze
Analyze the
the
metrics over a firewall.
Cases DataData Model
model

Virtual User Generator (VuGen) Analysis Tool

VuGen or the Virtual User Generator is the coding or The analysis tool allows the tester to download the performance
scripting editor used to create test scripts. test results file to conduct further analysis and create graphs.
2. Analyze the
Target Entity
Target Entity
Load Generators Monitor Over Firewall SiteScope
Load Generators are servers that Monitor over Firewall are servers used to monitor SiteScope is an optional Microfocus monitoring
generate the load for the performance your application servers, database, and other tool that supports advanced monitoring
test. application components capabilities of your network, application,
3. Identifyduring
the testing.
database, and other components during testing.
Transformation
s

Deloitte installs PT Team Member installs SMBC installs 1. Web-based tools set up as part of project provisioning
2. Controllers and MI Listeners have been pre-installed.
3. Can support up to 5000 concurrent users.

10
Performance Status Reporting (Sample Case Study)

Performance Report as below, to be shared with all required Stake holders with below information

Test Phase
Dry Run Cycle 1 | 29 Oct 2020
Target End Date

Type of Perf Test with Load Details, Run ID Load & Volume Test - 400 Vusers | ~50k Records in DB | Performance Center Run ID- 324,329

# of Test Runs Completed Completed 2 performance Load Test Runs – 400 vusers

• Data Base locking issues are observed due to which around 100 transactions didn’t run
• Volume test (RTM) Transaction failed at clicking ‘Configure & Export’ button – 502 Bad Gateway
Summary - Top Findings # 329
• Downstream & Siteadmin – During the load test run was In-Accessible and threw 502 Bad Gateway error
intermittently (noted as auto scaling was spinning new server, we saw application down with 502)
Performance Defects Addressed by DEV/Infra after Last
Auto Scaling Groups for CPU % utilization & DB Connections
Run
- Downloading of excel execution report
- HTTP 404, 500
- Site Admin create role & save create role
Open Issues from DR Cycle 0 - Site Admin Create Project with L1, L2 & L3 user
- Exports Failure, Grid Export Failures
- Server Errors
Perf DB issues have to be fixed in order to get acceptable transaction response time in Load test runs with 400
Issues/Risks If not, it poses risk to meet the planned performance tests in the coming week

11
Thank you