Testing Software Quality

Characteristics
Performance Testing
Performance Testing - ISO 29119

|Performance testing
- Is aimed at assessing ?
Performance Testing - ISO 29119

|Performance testing
- Is aimed at assessing the performance of the test item
- When it is placed under a “typical” load
Performance Testing

|Verify system meets its performance

requirements
|Quantitative and measurable performance
requirements
|Reasonable stable system
|Test environment representative of customer site
|Tools for load generator
Performance Testing

|Test Data Management

- Growing complexity and volume of data in today’s application
- Replicate real data from operational system
Performance Testing

|Define performance requirements

- What are the characteristics of performance requirements?
Performance Testing

|Define performance requirements

- Quantitative
- Measureable
- For specific functionality
- Related to use case
- Under what circumstances/condition
Performance Testing

|Good Performance requirements

- Core banking application?
Load Specification

|How do we specify load?

Load Specification

|How do we specify load?

- Identify relevant use-cases
- Load representing volumes of an activity
- Load representing mixes of activities
Load Specification

|How do we conduct performance test?

Load Specification

|How do we conduct performance test

- Identify performance requirements
- Identify relevant use-cases
- Identify load
- Vary load on the system
- Trach response time, resource usage, etc
- Identify bottlenecks
- Identify sources of performance problem
- Expected requirements versus actual performance
- Benchmarking
Load Specification
Testing Software Quality
Characteristics
Stress Testing
Stress Testing

|What is stress test

- Verify the behavior of the system meets its requirements when
?
Stress Testing

|What is stress test

- Verify the behavior of the system meets its requirements when
its resources are saturated and pushed beyond limits
- Push to the limits
Stress Testing - ISO 29119

|Stress testing
- Is aimed at assessing the performance of the test item
- When pushed beyond its anticipated peak load
- Or when available resources (e.g. memory, processor, disk) are
reduced
- Below specific min requirements
- To evaluate how it behaves under extreme conditions
Stress Testing

|How do we conduct stress test

a. Identify stress points
b. Develop a strategy to stress the system at stress points
c. Verify the generation of stress
d. Observe behaviour
|But how ?
Stress Testing

|How do we conduct stress test

a. Identify stress points - CPU, memory, buffer
b. Develop a strategy to stress the system at stress points- use
tools to generate load, simulations
c. Verify the generation of stress - is it really stressing, modify
strategy
d. Observe behaviour - stress requirements are met,
functionality is correct
Testing Software Quality
Characteristics
Volume Testing
Volume Testing - ISO 29119

|Volume testing
- Is aimed at assessing the performance of the test item ?
-
Volume Testing - ISO 29119

|Volume testing
- Is aimed at assessing the performance of the test item
- When it is processing specified levels of data (e.g. database
near max capacity)
Volume Testing

|What is volume testing

- Verify the behaviour of system meets its requirements
(functional and performance)
- How is this done?
Volume Testing

|What is stress test

- Verify the behaviour of system meets its requirements
(functional and performance)
- When the system is subjected to a large volume of activity
- Over an extended period of time - 24/7
Volume Testing

|What can happen when systems are used

extended periods of time?
Volume Testing

|What can happen when systems are used

extended periods of time?
- Memory Leak - memory filled up
- Counter overflow -
- Resource depletion - resource used faster than it can be
recovered
Volume Testing - CRM Siebel Example
Testing Software Quality
Characteristics
Configuration Testing
Configuration Testing

|Can tests be repeated for each possible

configuration?
Configuration Testing

|Can tests be repeated for each possible

configuration?
- Can we get access to all configurations?
Configuration Testing

|Can tests be repeated for each possible

configuration?
- Can we get access to all configurations?
- Which tests?
Configuration Testing

|Can tests be repeated for each possible

configuration?
- Can we get access to all configurations? - Not all the time
- Which tests? - functional and nonfunctional requirements
Configuration Testing Steps

|Identify the parameter that define each

configuration
- that could have an impact on the system’s ability to meet its
functional and performance requirements
|Partition (group similar configurations)
|Identify configuration combination to test
- What are the strategies?
Configuration Testing Steps

|Identify the parameter that define each

configuration
- that could have an impact on the system’s ability to meet its
functional and performance requirements
|Partition (group similar configurations)
|Identify configuration combination to test
- Extremes (min and max)
- Risk based
- Design of experiments (pair wise)
Configuration Testing Example
Testing Software Quality
Characteristics
Additional Performance Testing
Terminology
Performance Testing

|What additional terminology are you aware of

related to performance testing:
Performance Testing - ISO 29119

|Additional techniques are:

- Endurance testing (also called Soak Testing) - is aimed at
assessing whether the test item can sustain the required load
for a continuous period of time
- Capacity testing (also called Scalability testing) - is aimed
at assessing how the test item will perform under conditions
that may need to be supported in the future (what levels of
additional resources will be required in the future)
Performance Testing - ISO 29119

|Additional techniques are:

- Memory management testing - is aimed at assessing how the
test item will perform in terms of amount (normally max) of
memory used, the type of memory and or defined level of
memory leakage experienced during testing (e.g. RAM, ROM,
dynamic or allocated/static memory)
Testing Software Quality
Characteristics
Regression Testing
Regression Testing

|Risky task to modify software

- Existing software
- New software
|Why modifications introduce defects?
Regression Testing

|Risky task to modify software

- Existing software
- New software
|Why modifications introduce defects?
- Code ripple effects
• what is the relation between new fix versus new error
- Unintended feature or service interactions
- Changes in performance, resource sharing
Regression Testing

|What is regression testing?

Regression Testing

|What is regression testing?

- Ensure that previously introduced and tested functions
continue to work as specified after software modifications
|What is the strategy?
- Do we run the same tests we ran before?
Regression Testing

|What is regression testing?

- Ensure that previously introduced and tested functions
continue to work as specified after software modifications
|What is the strategy?
- Run regression tests at multiple levels:
• Unit level
• Integration level
• System level
Regression Testing

|Who runs regression testing?

Regression Testing

|Who run regression testing?

|Depends on the strategy
- Run regression tests at multiple levels:
• Unit level - developers
• Integration level - testers
• System level - testers
Regression Testing Process - Defect Repair
Process
Regression Testing Process
Regression Testing Strategy

|Full
- Rerun all tests
|Selective
- Rerun selected subset based on modification
- Rerun selected subset regardless of the modification,
• Standard Confidence Test (Basic Acceptance Test - BAT)
Selective Regression Testing Strategy

|Selective
- Rerun selected subset based on modification
• Requires communication with developer - what is changed
• Requires tools finding code deltas, ripple effect analysis
• Use AI
- Rerun selected subset regardless of the modification,
• Standard Confidence Test (Basic Acceptance Test - BAT)
Selective Regression Testing Strategy

|Tools for Selective Regression Testing

- Code Deltas - keep track of Deltas (what is changed)
• SCM - Software Configuration Management
• Coverage Tools
• Rerun test that traversed changed or deleted code
- Ripple Effect
• Impact of modification to other features/code - who does
this?
• Communication of impacted features/requirements - to who?
Selective Regression Testing Strategy

|Tools for Selective Regression Testing

- Code Deltas - keep track of Deltas (what is changed)
• SCM - Software Configuration Management
• Coverage Tools
• Rerun test that traversed changed or deleted code
- Ripple Effect
• Impact of modification to other features/code - who does this
- developers
• Communication of impacted features/requirements - to who -
test team
Selective Regression Testing Strategy - Code
coverage
Selective Regression Testing Strategy - Ripple
Effect
Selective Regression Testing Strategy

|What is checked by the confidence test?

-
Selective Regression Testing Strategy

|What is checked by the confidence test?

- Confidence test is a subset of tests that is executed to verify
previous functionality
Selective Regression Testing Strategy

|Which tests are selected for confidence test?

-
Selective Regression Testing Strategy

|How to select tests for confidence test?

- High frequency use cases
- Critical functionality
- Functional breath
Selective Regression Testing Strategy

|Can we use the same regression tests in every

release?
Selective Regression Testing Strategy

|Can we use the same regression tests in every

release?
|Can we use the same confidence tests in every
release?
Selective Regression Testing Strategy

|Can we use the same regression tests in every

release?
|Can we use the same confidence tests in every
release?
- Both needs update
- They need to be revalidated (care and maintenance)
Testing Software Quality
Characteristics
Reading - Model Based Regression
Test Selection Technique
Model Based Regression Testing

|Look into deltas in models of the code

|Which models are used?
Model Based Regression Testing

|Look into deltas in models of the code

|Which models are used?
- UML Sequence diagrams
- UML Finite State machines
- Model based control flow diagrams
Operation and Maintenance Phase

|Modifications in code - Delta

|Model 1 versus Model 2 how Regression tests are
created?
Operation and Maintenance Phase

|Modifications in code - Delta

|Model 1 versus Model 2 how Regression tests are
created?
- Model used are Control flow and data flow diagrams
- Map deltas to control flow diagram and create tests cases
Testing Software Quality
Characteristics
Additional Terminology regarding
Regression Test
Performance Testing

|What additional terminology are you aware of

related to regression testing:
Additional terminology related to Regression Testing

|Smoke Tests -
- very high level testing, also called Build Verification Test
- Done mostly by developers ensure the code module is successfully compiling
and the software’s core functionality is working properly
- ensure the build is suitable to begin any other type of testing.
• For example, ensure the build installs and then the Sanity Testing can
begin.
|Sanity Test -
- Done to ensure the product is sane, rational for more detailed testing.
- Also called Surface Level Testing
- ensures that bug fixes, newly added functionality or any other code change
doesn’t affect the stable software, build other functionality or introduce any
type of bug issue.
- a subset of regression testing, some high priority regression test cases
constitute sanity test case usually.
- Just before the deployment on production.
Testing Software Quality
Characteristics
Reading - Testing in O&M
Operation and Maintenance Phase

|Modifications in code - Delta

|No modifications in code but code is in operation
and maintenance
- What are some tests that the regression tests can include?
- Is it subset of existing regression tests?
Operation and Maintenance Phase

|Modifications in code - Delta

|No modifications in code - in operation and
maintenance phase
- Used in a different way versus initial context
• Security and privacy treats
• Safety impact
• Hazards introduced by system changes
• Performance needs changed
• Environment hardware changes
Operation and Maintenance Phase

|As products evolve, it is not enough to select

subset of test but we might need to extend tests
Testing Software Quality
Characteristics
Reading: High Quality Automated
Scripts
High Quality Automated Scripts

|STCE - Software test code engineering

High Quality Automated Scripts

|STCE - Software test

code engineering
- SW engineering in test
code
High Quality Automated Scripts

|Test Automation Process

- Setup
- Exercise/Execute
- Verify quality of test cases - How is this done?
•
- Tear down - prepare environment for new test
High Quality Automated Scripts

|Test Automation Process

- Setup
- Exercise/Execute
- Verify quality of test cases
• Correctness of the test cases
• Effectiveness of the test cases
• Maintainability of test cases
- Tear down - prepare environment for new test
Testing Software Quality
Characteristics
Mobile Testing
Mobile Testing - Test Man
Mobile Testing - Test Man (Paul Marcarelli)

|Test Man - 2002-2011

|Verizon gain 10% new
customer
|Customer turnover,
churn, was down 1.8%
from 2.5%
|Helped move network
reliability up the ranks
Mobile Testing

|How mobile testing different from web

application?
|What are new strategies/tools?
|How far AI is effective in mobile testing?
Mobile Testing
|How mobile testing different from web
application?
- Input
- Context aware - location, time, user depended
- Connectivity - network (many different networks)
- Security and reliability
- Usability - GUI testing
- Diversity of operation systems hardware - configuration, use
emulators and simulators, crowd testing, exploratory testing
- Additional hardware - touch screen, voice recognition
- Languages
- Resource usage - stress and load testing
Testing Software Quality
Characteristics
Mobile Application Testing: Additional
Reading
Mobile Testing
Mobile Testing