Reviews

A review is a type of static testing during which a work

product or process is evaluated by one or more individuals
to detect issues and to provide improvements.

When done properly, reviews are the single biggest, and

most cost-effective, contributor to overall delivered quality

INFORMAL REVIEWS
characterized by not following a

defined process and not having formal

documented output

FORMAL REVIEWS
characterized by team participation,

documented results of the review and

documented procedures for conducting

the review
Possible reviews
Contractual review

initiated at project inception and at major project

milestones

Requirements reviews

initiated when the requirements are available for review,

which covers functional and non-functional

Top level design reviews

initiated when the overall architectural design is available

for review

Detailed design reviews

initiated when the detailed design

is available for review

Code reviews

carried out as individual modules of software are created,

which may include the unit tests and their results as well

as the code

Test work product reviews

which may cover the test plan(s), test conditions, quality

risk analysis results, tests, test data, test environments,

and test results

Test entry reviews and test exit reviews

§for each test level, which respectively check the test

entry criteria prior to starting test execution and the test

exit criteria prior to concluding testing

Acceptance reviews

used to obtain customer or stakeholder approval for a

system

Audits and management reviews

which focus more on the software process rather than the

software work products

The review leader (sometimes same as the  Test Manager)

should:

implement efficient reviews in their projects

demonstrate the benefits of these reviews

ensure that an environment exists which is conducive to

the implementation of the success factors

devise a measurement plan to ensure that the reviews

provide value

remember that while reviews  should be augmented with

other forms of static testing and dynamic testing of the

code in order to improve test coverage and locate

defects

Different techniques have different focuses:

a review can eliminate an issue at the requirements

level before the problem is implemented in the code

static analysis can help enforce coding standards and

check for problems

Inspections can lead to the discovery and removal of

defects, and also train authors in how to avoid creating

defects
The focus of a review depends on the agreed objective and

the software development lifecycle model should dictate

the formality of a review based on the:

maturity of the development process

complexity of the work product being reviewed

legal or regulatory requirements

need of an audit trail

review success factors

Clear objectives defined in planning and measured

Suitable review types applied per the products and test

levels and executed with proper participants

Review techniques used

Up to date checklists

Small chunks used for large documents so that quality

control is exercised by providing authors early and

frequent feedback

Adequate time to prepare for the participants

Reviews are scheduled with proper notice

Defects found are acknowledged and handled

Management support of the review process

The review is conducted in an atmosphere of trust

 people success factors
right people are involved to meet the objectives

Testers are seen as valued reviewers who contribute

to the review and learn about the work product

Dedicate adequate time and attention to detail

The meeting is well-managed

Adequate training is provided (for ex: inspection)

A culture of learning and improvement is promoted

Please keep in mind that

A single work product may be the subject of
more than one type of review
one of the main objectives is to uncover defects
even though reviews can be used for various
purposes
The types of defects found in a review vary,
depending especially on the work product being
reviewed
All review types can aid in defect detection
the selected review type should be based on
the needs of the project
available resources
product type and risks
business domain
company culture, etc.
INFORMAL REVIEW

is a review without a formal or documented procedure.

Same as: buddy check, pairing, pair review.

Main purpose: detecting potential defects

Can also generate ideas, solutions or quickly solve minor

problems.

During an informal review we may have the below:

review meeting

documented results

use checklists

be performed by a colleague or a group of people

These kind of reviews are common within Agile

develoment

WALKTHROUGH
is a review in which an author leads members of the

review through a work product and the members ask

questions and make comments about possible issues.

Main purpose: find defects, improve the product,

consider alternative implementations, evaluate

conformance to a standard or specification.

Can also: facilitate the exchange of ideas, training the

participants, achieving consensus

A walkthrough can vary from very formal to informal, but

during a walkthrough we must have:

review meeting

scribe for the review meeting

and we may:

have individual preparation before the meeting

use checklists

use potential defect logs and reports

present it as scenarios, dry runs or simulations

TECHNICAL REVIEW
is a formal review type executed by a team of

technically-qualified personnel that examines the

suitability of a work product for its intended use and

identifies discrepancies from specifications and standards

Main purpose: gaining consensus, detecting potential

defects

Can also: evaluate quality and build confidence in the

product, generate new ideas, motivate and enable authors

to improve, consider alternative options

During a technical review we must have:

individual preparation before the meeting

scribe which is not the author

should have:

reviewers as technical peers of the author

 reviewers as technical experts in the same or other

discipline

product potential defect logs and review reports

and may:

hold a review meeting

use checklists

INSPECTION
is a formal review type used to identify issues in a work

product, which provides measurement to improve the

review process and the software development process.

Main purpose: detecting potential defects, evaluating

quality and building confidence in the product,

preventing future similar defects through author learning

and root cause analysis

Can also: motivate and enable authors to improve future

work products and the software development process,

achieving consensus.

There is a defined process with formal documented

outputs, based on rules and checklists. We must:

have clearly defined mandatory roles

have individual preparation

reviewers as peers of the author or experts in

disciplines relevant for the work product

specified entry and exit criteria

And the must continues

have a scribe

have a trained facilitator lead the review meeting (not

the author)

never have the author as leader, reader or scribe

product potential defect logs and review reports

collect metrics and use them to improve the entire

software development process

the next reviews are of more

interest for the Test Manager

MANAGEMENT REVIEW
is a systematic evaluation of:

software acquisition

supply

development

operation

maintenance process

performed by or on behalf of management that:

monitors progress

determines the status of plans and schedules

confirms requirements and their system allocation

evaluates effectiveness of management approaches

To achieve fitness for purpose Management

reviews are used to:

monitor progress

assess status

make decisions
This approach is also used as decision support for:

adapting the level of resources

implementing corrective actions

changing the scope of the project

Management reviews have the following key traits:

Conducted by or for managers having direct

responsibility for the project or system

Check consistency with and deviations from plans

Check if management procedures are adequate

Assess project risks

Evaluate impact of actions and ways to measure them

An output of this review consists of lists of action items,

issues to be resolved and decisions made.

Test Manager should participate in and may initiate

management reviews of testing processed and should

consider such reviews as an integral part of process

improvement (ex: management review of processed,

project retrospectives, lessons learned, etc.).

AUDITS
is an independent examination of a work product,

process, or set of processes that is performed by a third

party to assess compliance with specifications, standards,

contractual agreements, or other criteria

Audits are usually:

performed to demonstrate conformance to a defined set

of criteria

Conducted and moderated by a lead auditor

provide evidence of compliance collected through

interviews, witnessing and examining documents

have documented results

Managing reviews
The review strategy must be coordinated with the test

policy and the overall test strategy.

Reviews should be planned to take place at natural break

points or milestones:

typically after requirement and design specification

start with business objectives and work down to low

level design

often as part of a verification activity before, during,

and after test execution

Before establishing an overall review plan at the project

level, the review leader (may be a Test Manager) should

take into account:

What should be reviewed (product and processes)

Who should be involved in specific reviews

Which relevant risk factors to cover

The review leader should also:

identify the items to be reviewed

select the appropriate review type and level of

formality

identify a budget (time & resource) for the review

create a business case and include a risk evaluation and

return on investment calculation

The return on investment (ROI) for reviews is the

difference between the cost of onducting the review

and the cost of dealing with the same defects at a

later stage (or missing them altogether).

The optimal time to perform reviews depends on:

availability of the items to review in a sufficiently final

format

availability of the right personnel for the review

time when the final version of the item should be

available

time required for the review process of that specific

item

The objective of the review must be defined during test

planning and should also include:

conducting effective and efficient reviews

reaching consensus decisions regarding review

feedback
Adequate metrics also need to be established.

A good tip in case of audit or inspections on big projects

is to use brief inspections/audits conducted at the author's

request as document fragments are completed. This will

help have initial and iterative checks rather than a big

bang approach when all is taken in at once. There are also

options to have an advance audit prior the main

certification audit.

We should not forget about the project reviews which are

frequently held for the overall system and may also be

necessary for subsystems and even individual software

elements

Based on project complexity or product risks we should

adjust:

the number of reviews

the types of reviews

the organization of reviews

the prople involved in reviews

Participants in reviews must:

Have the appropriate level of knowledge, both technical

and procedural

Be thorough and pay attention to details

Have clarity and the use of a correct prioritization

Understand their roles and responsibilities in the review

process
Review planning should address:

The risks associated with technical factors,

organizational factors and people issues

The availability of reviewers with sufficient technical

knowledge

Ensure that each team is committed to the success of

the review process

Ensure that each organization is allocating sufficient

time for required reviewers to prepare for and

participate in the reviews

Time allocation for required technical or process

training for the reviewers

Backup reviewers should be identified in case key

reviewers become unavailable due to changes in

personal or business plans

During execution, a review Leader must ensure:

Adequate measurements are provided by the

participants in the reviews to allow evaluation of

review efficiency

Checklists are created, and maintained to improve

future reviews

Defect severity and priority evaluation are defined for

use in defect management of issues found during

reviews
After each review, the review Leader should:

Collect the review metrics and ensure that the issues

identified are sufficiently resolved to meet the specific

test objectives for the review

Use the review metrics as input when determining the

return on investment (ROI) for reviews

Provide feedback information to the relevant

stakeholders

Provide feedback to review participants

A reviews effectiveness is evaluated by comparing results

from the review with the actual results found in

subsequent testing.

In case a work product is reviewed, but later found

defective, then the review leader should consider ways in

which the review process might have allowed the defects

to escape. Some likely causes:

include problems with the review process (like poor

entry or exit criteria

improper composition of the review team

inadequate review tools (checklists)

insufficient reviewer training and experience

too little preparation and review meeting time