CHAPTER 1: Fundamentals of Testing

1.1 Why is Testing Necessary (K2)

1.1.1 Software Systems Context (K1)

1.1.2 Causes of Software Defects (K2)

Why do faults occur in software?
 Software is written by human beings
 Who know something, but not everything
 Who have skills, but aren’t perfect
 Who do make mistakes (errors)
 Under increasing pressure to deliver to strict deadlines
 No time to check but assumptions may be wrong
 Systems may be incomplete
Failures must be corrected in the software
The cost of finding and fixing defects rises considerably across the life cycle
1.1.3 Role of Testing
 Rigorous testing of systems and documentation can help to reduce the risk of
problems
 Contribute to the quality of the software system, if the defects found are corrected
before the system is released
  Meet contractual or legal requirements, or industry-specific standards.
1.1.4 Testing and Quality (K2)

–     Testing measures software quality

–     Testing can find faults; when they are removed, software quality  is improved
–     Testing is a part of quality assurance
–     Measure the quality of software in terms of defects found for both functional and
non-functional software requirements and characteristics
–     Give confidence in the quality of a software if it finds few or no defects and we have
good test designs
–     Improve the quality of future system
1.1.5 How Much Testing is enough? (K2)
It depends on RISK
–     Risk of missing important faults
–     Risk of incurring failure costs
–     Risk of releasing untested or under-tested software
–     Risk of losing credibility and market share Risk of missing a market window Risk of
over-testing, ineffective testing
Test time will always be limited – Using RISK to determine
–     What to test first
–     What to test most
–     How thoroughly to test each item
–     What not to test ( this time)
–     Allocate the time available for testing by prioritizing testing
1.2 What is Testing? K2)
–     Test activities exist before and after test execution: planning and control, choosing
test conditions, designing and executing test cases, checking results, evaluating exit
criteria, reporting on the testing process and system under test, and finalizing or
completing closure activities
–     Testing also includes reviewing documents (including source code) and conducting
static analysis.
Test Objectives:
1. Finding defects
2. Gaining confidence about the level of quality
3. Providing information for decision-making
4. Preventing defects
Prevent defects
–     Designing tests early in the life cycle can help to prevent defects from being
introduced into code.
–     Reviews of documents (e.g., requirements) and the identification and resolution of
issues also help to prevent defects appearing in the code.
Different objectives in different phase
–     In development testing: find as many defect as possible
–     In acceptance testing: confirm that the system works as expected, to gain
confidence that it has met the requirements.
–     In some cases: assess the quality of the software to give information to stakeholders
of the risk of releasing the system at a given time.
–     Maintenance testing often includes testing that no new defects have been
introduced during development of the changes.
–     During operational testing: assess system characteristics such as reliability or
availability.
Debugging and testing are different.
–     Dynamic testing can show failures that are caused by defects.
–     Debugging is the development activity that finds, analyzes and removes the cause of
the failure..
–     Re-testing by a tester ensures that the fix does indeed resolve the failure
–     Testers test and developers debug –
1.3 Seven Principles of Testing
1. Testing shows presence of defects
–     Testing can show that defects are present, but cannot prove that there are no
defects.
–     Testing reduces the probability of undiscovered defects remaining in a software but,
even if no defects are found, it is not a proof of correctness
2. Exhaustive testing is impossible
–  Test everything ( all combination of inputs and preconditions) is not feasible
–  Instead of exhaustive testing, risk analysis and priorities should be used to focus
testing efforts
3. Early testing
–  Testing activities should start as early as possible and focus on defined objectives
– Perform the test design and review activities early can finds defects early on when
they are cheap to find and fix
4. Defect clustering
– A small numbers of modules usually contains most of the defects discovered during
pre-release testing, or is responsible for most of the operational failures
– Rule 80/20: Module core often contains 80% defects
5. Pesticide paradox
– If the same tests are repeated over and over again, no new defects can be found
– To overcome this pesticide paradox, test cases need to be regularly reviewed and
revised;
– New and different tests need to be written to exercise different parts of the software
to find potentially more defects
6. Testing is context dependent
– Testing is done differently in different context
– Ex: Safety-critical software is tested differently from an ecommerce site
7. Absence of error fallacy
– Finding and fixing defects does not help if the system built is unusable and does not
fulfill the users’ needs and expectations
1.4 Fundamental test process

1.4.1 Test Planning and Control (K1)

Test planning is the activity of defining the objectives of testing and the specification of
test activities
Test planning tasks:
–           Determine the scope and risks and identify the objectives of testing
–           Determine the test approach
–           Implement test policy and/or test strategy
–           Determine the required test resources
–           Schedule
–           Determine the exist criteria
Concepts:
–           Test policy: A high level document describing the principle, approach and major
objectives of organization regarding testing
–           Test strategy: A high level description of the test types to performed
–           Test approach: The implementation of the test strategy for a specific project
–           Test ware: includes test cases, test plan, test data
Test control is the ongoing activity of :
–           comparing actual progress against the plan
–           reporting the status, including deviations from the plan.
–           taking actions necessary
Test control task:
–           Measure and analysis the results of reviews and testing
–           Monitor and document progress, test coverage and exit criteria
–           Provide information of testing to make evaluation
–           Initiation corrective actions
1.4.2  Test Analysis and Design
is the activity during which general testing objectives are transformed into tangible test
conditions and test cases.
Test Analysis and Design tasks:
–           Reviewing the test basis
–           Evaluating testability of the test basis and test objects
–           Identifying and prioritizing test conditions
–           Designing and prioritizing high level test cases
–           Identifying necessary test data to support the test conditions and test cases
–           Designing the test environment setup and identifying any required infrastructure
and tools
–           Creating bi-directional traceability between test basis and test cases
1.4.3 Test Implementation and Execution (K1)
is the activity where test procedures or scripts are specified, the environment is set up
and the tests are run
Test Implementation and Execution tasks:
–           Finalizing, implementing and prioritizing test cases
–           Developing and prioritizing test procedures, creating test data, writing test
scripts.
–           Creating test suites from the test procedures for efficient test execution
–           Verifying that the test environment has been set up correctly
–           Verifying and updating bi-directional traceability between the test basis and test
cases
–           Executing test procedures either manual or automate
–           Logging the outcome of test execution
–           Comparing actual results with expected results
–           Reporting discrepancies as incidents
–           Repeating test activities in order to confirm a fix
1.4.4 Evaluating Exit Criteria and Reporting (K1)
Evaluating exit criteria is the activity where test execution is assessed against the
defined objectives. This should be done for each test level
–           Checking test logs against the exit criteria specified in test planning
–           Assessing if more tests are needed or if the exit criteria specified should be
changed
–           Writing a test summary report for stakeholders
1.4.5 Test Closure Activities (K1)
Test closure activities collect data from completed test activities to consolidate
experience, testware, facts and numbers.
–           Checking which planned deliverables have been delivered
–           Closing incident reports or raising change records for any that remain open
–           Documenting the acceptance of the system
–           Finalizing and archiving testware, the test environment and the test
infrastructure for later reuse
–           Handing over the testware to the maintenance organization
–           Analyzing lessons learned to determine changes needed for future releases and
projects
–           Using the information gathered to improve test maturity
1.5 Psychology of testing
Level of independence
–           None: tests designed by the person who wrote the software
–           Tests designed by a different person
–           Tests designed by someone from a different department or team ( e.g test team)
–           Tests designed by someone from a different organisation ( e.g agency)
Communication about defects
–           Testing is very constructive in the management of product risks.
–           Looking for failures in a system requires curiosity, professional pessimism, a
critical eye, attention to detail, good communication
–           Need good interpersonal skills to communicate factual information about defects
–           Communicate findings on the product in a neutral, fact-focused way without
criticizing the person who created it.
–           Explain that by knowing about this now we can work round it or fix it so the
delivered system is better for the customer.
–           Start with collaboration rather than battles.