Software Quality

Engineering
Lecture No. 11
 Part- 2 (a):
Software Testing
 Summary of the previous lecture

 Testing: Concepts & Process

 Testing Related Questions
 Why, how, when.. etc.
 Major Testing Activities
 Planning and preparation
 Execution
 Analysis and follow-up
 Outlines
 Usage Based Testing
 Coverage Based Testing
 Testing Activities
 Test Management
 Testing Automation
 Objectives

 To be able to learn and understand different testing

related activities, management and automation.

 Functional vs. Structural Testing

• Key distinction: Perspective on what need to be checked/tested.

 
 • Functional testing:
  . tests external functions.
  – as described by external specifications
. black-box in nature;
  – functional mapping: input ⇒ output
– without involving internal knowledge
   • Structural
  testing:
  . tests internal implementations.
  – components and structures.
  . white-box in nature;
– “white” here = seeing through
  ⇒ internal elements visible.
. really clear/glass/transparent box.
 Black-Box vs. White-Box View

   • Object abstraction/representation:
  . high-level: whole system ≈ black-box.
. low-level: individual statements, data, and other elements ≈
white-box.
  . middle-levels of abstraction:
– function/subroutine/procedure, module, subsystem,
etc.
  – method, class, super-class, etc. 
•  Gray-box (mixed black-/white-) testing:
 
  . many of the middle levels of testing.
  . example: procedures in modules
– procedures individually as black box,
– procedure interconnection ≈ white-box at module level.
 Usage-Based Testing and OP

   • Usage-based statistical testing:

  . actual usage and scenarios/information

  . captured in operational profiles (OPs)
. simulated in testing environment
   (too numerous ⇒ random sampling)
  
   • Applicability
  . final stages of testing.
  . particularly system/acceptance testing.
   . use with s/w reliability engineering.
  
 
• Termination criteria: reliability goals
 Coverage-Based Testing

   • Coverage-based testing:
  . systematic testing based on
formal (BBT/WBT) models and techniques
  . coverage measures defined for models
   . testing managed by coverage goals

   • Applicability:

  . all stages of testing.

  . particularly unit and component testing.
   . later phases at high abstraction levels.
• Termination criteria: coverage goals
 Steps in Systematic Testing

  • With a formalized strategies/goals

  . based on formal models and techniques,
   . managed by termination criteria.
  
 
   • Steps in model construction and usage:
. Define the model, usually represented as graphs and relations.
  . “Check” individual elements:
. “Test”: derive (sensitize) test cases and then execute them.
. Result checking and follow-up
 Individual Test Case Preparation

•  Individual test cases (micro-level) vs. test suite (macro-level)

 •
   From multiple sources:
  . actual runs (usage-based).
  . implementation-based (white-box).
  . specification-based (black-box).
  . may use similar/earlier products.
  . (direct) record and replay (less often).
   . (via) formal models (OP, CFT, BT, etc.)
   Defining input values (model ⇒ test cases):
 •

  . initial/intermediate/interactive input
  (expected output too?)
  . exercise path/slice/track/etc
. in testing terminology: sensitization
 Test Cases Based on Formal Models

• Most organized, systematic test cases are derived from

  formal testing models:
  . directly via newly constructed models.
   . indirectly via exist test cases, etc.
 •
   Model construction steps:
. information source identification and data collection
  . analysis and initial model construction
. model validation and improvement
  • Model
   usage:

. defining test cases. (details with individual

  models/techniques)
. indirectly in analysis/followup
  Test Suite Preparation

  • Test suite (macro-level)

  . existing suite: what and where?
  – suitability? selection/screening?
  . construction/generation of new ones
. organization & management: often hierarchical,  
   • Adding new test cases
  . estimate # of new test cases
  . specify new (individual) test cases
   . integrate to existing test cases

  • Allocation to systems/operations
  . OP-/structure-based allocation
. both old and new test cases in suite
 Test Procedure Preparation

   • Key consideration: sequencing:

  . general: simple to complex.
  . dependency among test cases.
  . defect detection related sequencing.
  . sequence to avoid accident.
  . problem diagnosis related sequencing.
   . natural grouping of test cases.

   • Other considerations:
  . effectiveness/efficiency concerns.
  . smooth transition between test runs.
. management/resource/personnel/etc.
 Test Execution
 

   • Major testing activities:

  . test planning and preparation
  . execution (testing)
   . analysis and followup
  

   • Test execution:
  . execution planning and management
  . related activities: important part
  – failure identification and measurement
– other measurement
 Test Execution
  • General steps
  . allocating test time (& resources)
  . invoking test
  . identifying system failures
    (& gathering info. for follow-up actions)

   • Allocating test time

  . OP-based: systems/features/operations
  – also coverage concerns for critical parts
  . coverage-based: func./struc. areas
. alternative: bottom-up approach
– individual test cases ⇒ test time
– sum-up ⇒ overall allocation
– by OP or coverage areas
 Test Execution

  • Invoking test (OP-based)

. OP ⇒ input variables (test points)
  . follow probabilistic distributions
  (could be dynamically determined)
. sequence (what to test first?): COTS, product, super
system
  • Invoking
   test (coverage-based)

. organize sensitized test-cases

  . sequence ⇐ coverage hierarchies
 • Common part: Retest due to
. defect fix ⇒ verify fix
  . code-base or feature change
. general regression test
 Test Execution
  • Identifying system failures (problems):
  . similar for OP-/coverage-based
  . analyze test output for deviations
  . determine: deviation = failure ?
  . handling normal vs. failed runs
– non-blocking failure handling
  • Solving
   problem:

  . theoretically undecidable.
  . some cases obvious: crash, hang, etc.
  . practically based on heuristics:
  – product domain knowledge
  – cross-checking with other products
  – implementation knowledge & internals
– limited dynamic consistency checking
 Testing: Analysis and Follow-up
 
 

   • Major testing activities:

  . test planning and preparation
  . execution (testing)
   . analysis and followup
  

   • Test analysis and followup:

  . execution/other measurement analyzed
  . analysis results as basis for followup
  . feedback and followup:
  – decision making (exit testing? etc.)
– adjustment and improvement.
 Testing: Analysis and Follow-up

    • Input to analysis
  . test execution information
  . particularly failure cases
  . timing and characteristics data
   • Analysis and output
  . basic individual (failure) case
  – problem identification/reporting
  – repeatable problem setup
. overall reliability and other analysis?  
  • Follow-up activities
  . defect analysis and removal (& re-test).
  . decision making and management.
. test process and quality improvement.
 Testing: Analysis and Follow-up

  • For individual test runs:

. success: continue with normal testing.
. failure: see below.
  • Analysis and follow-up for failed runs:
  . understanding the problem
  by studying the execution record.
  . recreating the problem (confirmation).
  . problem diagnosis
  – may involve multiple related runs.
  . locating the faults.
  . defect fixing (fault removal)
  – commonly via add/remove/modify code
  – sometimes involve design changes
. re-run/re-test to confirm defect fixing.
 Test Management

  • People’s roles/responsibilities in formal and informal testing.

 • In informal testing:
  . “run-and-observe” by testers.
  . “plug-and-play” by users.
  . informal testing with ad-hoc knowledge
. deceptively “easy”, but not all failures or problems easy to
recognize.
 • In formal testing:
  . testers, and organized in teams.
  . management/communication structure.
  . role of “code owners” (multiple roles?)
  . 3rd party (IV&V) testing.
. career path for testers
 Test Management
• Test team organization:
  . vertical: project oriented
  – product domain knowledge,
  – staffing/resource management hard.
  . horizontal: task oriented
  – even distribution of staff/resources
  – lack of internal knowledge/expertise
. Mixed models might work better.
• Users and 3rd party testers:
. user involvement in beta-testing and other variations (e.g.,
ECI in IBM)
. IV&V with 3rd party testing/QA
  . impact of new technologies:
– CBSE, COTS impact
– security, dependability requirements.
 Test Automation

 
• Basic understanding:
 

 
. automation needed for large systems.
 
. fully automated: impossible.
. focus on specific needs/areas.
 
 

 
• Key issues to consider:
 

 
. specific needs and potentials.
 
. existing tools available/suitable?
 
– related: cost/training/etc.
 
. constructing specific tools?
 
. additional cost in usage & support.
. impact on resource/schedule/etc.
 Summary of Today’s Lecture

 Test activities:
. Planning & preparation:
. Execution & measurement: common.
.
Analysis & follow-up:
 Test management:
. different roles and responsibilities.

.
good management required.
 Test automation:
. set realistic expectations.
  

.specific areas for automation, esp. in execution,

measurement, and analysis.
 Overview of Next lecture
 Last lecture of Part 2(a)
 Coverage and Usage Testing Based on Checklists and
Partitions.
The End