Preparing Applications For Heavy Load

Peak Load High Volume Traffic Planning

About Northway Solutions Group
• HP Elite Solutions Partner and Reseller
• Application Lifecycle Management
• IT Performance Suite Provider
• HP Software Certified (ASE/AIS/CI) Consultants
• Testing Center of Excellence Solution Provider
• Quality Management Practice (Process-Focused)
• Experience With Large, Complex Projects
• A “Value Outsourcer”. We compete on skill sets, not $$$.
• Headquarters: Nashville, TN.

NORTHWAY SOLUTIONS GROUP 2

About The Speaker – Scott Moore

• 18+ years of IT experience

• Tested some of the largest
systems in the world.
• Thought leader for the
Performance Center of
Excellence
• President & CEO of Northway
Solutions Group
• Elite HP Partner/Reseller
• Certified Product Consultant
• Certified Instructor

NORTHWAY SOLUTIONS GROUP 3

About The Speaker – Wilson Mar

• HP Developer Advocate for

Mobile development,
performance, and capacity.
• 30+ years of IT experience
• Tested some of the largest, most
secure systems in the world
• Trained many as Mercury
Certified Instructor and
consultant

NORTHWAY SOLUTIONS GROUP 4

Peak/Point Load

• Sudden spikes or sustained usage volumes on an

application (expected or unexpected)
• Expected
• Seasonal
• Marketing
• Unexpected
• Unscheduled Celebrity Endorsement (TV/Radio)
• Viral Internet Appeal
• Unscheduled news event that affects business
• DDOS Attacks

NORTHWAY SOLUTIONS GROUP 5

Point Load Outages Are Common

NORTHWAY SOLUTIONS GROUP 6

What if we’re on the “Cloud”?

NORTHWAY SOLUTIONS GROUP 7

The Cloud Has Unlimited Scalability?

NORTHWAY SOLUTIONS GROUP 8

Cloudy With A Chance of Slow
• Cloud services are variable
• “Business Jitter”
• Low Tolerance for Jitter?
• Dedicated physical resource
• Virtual private cloud
• Used to be known as HOSTING
• Understand the business impact BEFORE doing a cloud
initiative!

NORTHWAY SOLUTIONS GROUP 9

Some Problems Hardware Can’t Fix

NORTHWAY SOLUTIONS GROUP 10

Does this company value your time?

NORTHWAY SOLUTIONS GROUP 11

Preparing For Heavy Traffic
• Implement effective caching and static content
optimization techniques
• Content Delivery Network (CDN) – distributed cache
• Redirect to static, scaled down page under heavy load
• Don’t Load Shopping Cart From Front Page
• Proactive production monitoring
• Server and Network Resources
• Web Server Logs
• End User Perspective – real browser page rendering
• JVM, SQL, Load Balancers, Gateways
• Long Term Solution = Performance testing in SDLC

NORTHWAY SOLUTIONS GROUP 12

Case Study: Netflix
• Chaos Monkey - randomly disables production instances
• Latency Monkey - induces artificial delays in RESTful client-server
communication (simulate service degradation)
• Conformity Monkey - shuts down instances that don’t adhere to best-
practices.
• Doctor Monkey - runs health checks for each instance. Unhealthy
instances are detected and removed.
• Janitor Monkey - ensures that the cloud environment is clutter and
waste free
• Security Monkey - finds security violations/vulnerabilities, and ensures
all SSL and DRM certificates are valid.
• 10-18 Monkey - detects configuration and run time problems in
instances serving customers in multiple geographic regions/character
sets.
• Chaos Gorilla - simulates an outage of an entire Amazon availability
zone.

NORTHWAY SOLUTIONS GROUP 13

Planning

• The planning phase is the key to load testing

• Planning includes
• Analyzing User Behaviors
• System Information
• Focusing on specific goals and objectives
• A good load test creates chaos in a controlled way
• Chaos = randomization of activity and data
• Controlled = user and transaction volume, environment

NORTHWAY SOLUTIONS GROUP 14

Making Goals Measurable

Performance should be measured on:

• System Capacity – concurrent users or system
activities the application can handle.
• User/Service Experience –response time of user
experience and error rates under load.
• Infrastructure Cost – utilization of critical system
resources such as CPU, memory, disk, and network
throughput.
• Functional Issues Under Load – if a failure is found
only under load, how many are acceptable?

NORTHWAY SOLUTIONS GROUP 15

Concurrency
A set of users acting upon an application in a similar manner and at the same
time.

• Application Level
• How many users are active on the system?
• Business Process Level
• How many users are buying tickets?
• Transaction Level
• How many users are buying tickets NOW?

NORTHWAY SOLUTIONS GROUP 16

Bad Performance Requirements

• “This thing needs to be fast...”

• Sub-second response time for everything, always
• We have a total audience of 10,000 so we need 10,000
users in the load test
• “Let’s just kick it around and see what numbers we get.
Then we’ll make the goals specific....”

CONCEPTUAL GOALS ARE NOT ENOUGH!!!!

NORTHWAY SOLUTIONS GROUP 17

Good Performance Requirements

• Peak load = 500 concurrent users or 50,000 page views

per hour
• Under peak load, all web page requests should return in
under 8 seconds.
• All CPU Utilization metrics should be under 80% under
the peak load conditions.
• Less than 5% failure rate for page request under peak
load conditions

NORTHWAY SOLUTIONS GROUP 18

Determining Volume and Performance
Requirements – Existing Systems
• Find sources of data that have usage patterns:
– Database logs/traces (audit tables, traces from tools)
– Financial systems (sales records, etc)
– Call center records
– Web logs (best of all, if your system has them!)
• Gather information about the users:
– Who are the users? (roles, profiles)
– What are they doing? (business processes)
– How often are they doing it? (volumes per business process)
– Determine weekly, monthly, seasonal trends – will help you identify peak
volumes!
– Acquire data for one year and forecast using trend analysis (don’t forget
seasonal components)

NORTHWAY SOLUTIONS GROUP 19

New Systems
• Involves more uncertainty
• Map use cases to Use Cases

application ‘entry points’

Business Projections Performance / Volume data from
analogous systems

• Developing requirements:
1. Start with any hard data about the system,
then build a model summarizing that data Model Building
(generally in
Excel)

2. Run “what if” situation to determine if this would

cause the data from #1 to change drastically
Sensitivity
3. Define customer performance expectations by Analysis

getting feedback from customers directly.

4. Combine all data to provide a “best guess” for
volume and performance requirements. Expected Volume
Requirements
+
5. Assume a standard deviation of 30% – 70%
Customer Performance
Expectations

Performance and
Volume
Requirements

NORTHWAY SOLUTIONS GROUP 20

What Should We Test?
Use Case Diagram for Simple
• Prioritization and scope reduction Insurance System
• Limited testing time or resources «uses»
Create Claim

• Its not economical to test everything

• Find a compromise that reduces risks «uses» Intake Specialist

Markup Claim
«uses»

• Use a ranking system

«uses»

• What will be used the most? Search

Assign Claim

• What are the most performance sensitive

Intake Coordinator

«uses»

areas? «uses»
Manage Claim

• What areas pose the most business risk if «uses»

they fail under load? Pay Claim

«inherits»
Calculate Payment

• Organize the top areas based on this criteria

Claim Manager

NORTHWAY SOLUTIONS GROUP 21

Roles and Responsibilities
Role: Involved in: Qualities brought: Outputs:
Architect/Technical All aspects of Load Vision, Technical Test Strategy, Success
Contact Test Planning and Guidance, Mentoring, Requirements
Execution Hard-nosed analysis,
Broad knowledge
of system

Business Analyst Testing Strategy, Factual analysis Use Case Prioritization,

Success of marketing data, “Voice of customer” data
Requirements knowledge of
business processes,
knowledge of
customer
Developer Test Execution Detailed knowledge Design and data
Methodology, Data of system and team analyses, Data
Acquisition Requirements
Project Manager Resource Project management Load Test Project Plan,
Management, Project experience, resource Status Reports
plan creation acquisition skills
QA Team Leader All aspects of Load Experience, Load Test Plan
or QA Mgr Test Planning and knowledge of system
Execution failure points,
knowledge of test
case implementations

NORTHWAY SOLUTIONS GROUP 22

Which Load Testing Tool?

• Is it an industry standard or commonly accepted?

• Is it enterprise ready?
• Does it support multiple protocols or technology stacks?
• Does it work with most major platforms?
• Does it have monitoring capabilities? (OS/APP,ETC)
• Does it provide a good way to analyze test results?
• Can you assess the impact of the network for remote
users in a pristine lab (i.e. WAN Emulation)?
• Is there a community of users to share problems,
tips, tricks, etc?

NORTHWAY SOLUTIONS GROUP 23

HP LoadRunner

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LoadRunner Key capabilities

A. Large pool of experienced users world-wide

B. Several forums, new HP Live Network, HP Education
• Powerful correlation studio
C. Broad range of application support
• 50+ Protocols (Mobile, RIA, SAP, Oracle DB, etc.)
• TruClient for automatic JavaScript AJAX handling
• Linux and Windows load generators
D. Monitoring with HP SiteScope & Diagnostics
• Visibility on root cause of performance issues
• Script re-use for production monitoring
E. WAN Emulation in HP Network Virtualization
F. Advanced performance analysis & reporting

25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Workflow and components

End User Client(s) Internet/WAN Application Server(s) Database

C 5 Virtualize Network 4 Virtualize

Services
1 Author
Scripts
Java, C# TruClient 6 Generate
Load
JavaScript 7 Measure Impact
Junit, Nunit

SiteScop
e
8 Analyze, Compare,
Report
HP Live
Network
Test
2 Collaborate Scenarios

3 Plan and Execute

Tests
26 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Triage, isolate and route to effective root-cause analysis

HP Sitescope and HP Diagnostics integrated to HP LoadRunner

• Transaction traceability from end user

response time to the slow method or SQL
query
• Diagnostics across JAVA, .NET, SAP,
SOA, ERP/CRM applications
• Identify performance bottlenecks including
memory leaks
• Diagnose complex issues and reduce
Mean Time to Resolution (MTTR)

27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Summary

• Prepare for the perfect storm before it hits

• The cloud is not a fool-proof solution
• Planning is the key to avoiding performance issues
• Use a lifecycle approach to performance testing
• Select your choice of tool based on industry
standards and best practices

NORTHWAY SOLUTIONS GROUP 28

Additional References

• Northway Solutions Group Blogs

• http://northwaysolutions.com/blog
• @northwaysg and @loadtester
• For this slide deck - [email protected]
• HP LoadRunner Official Web Pages
• http://www.hp.com/go/loadrunner
• HP Blog http://t.co/HourUQikNS
• Perfbytes – www.perfbytes.com
• Episode 17 – “How to Survive Point Load”

NORTHWAY SOLUTIONS GROUP 29

 www.northwaysolutions.com
www.managementsoftware.hp.com

NORTHWAY SOLUTIONS GROUP 30