School of Engineering & Applied Science

Department of Engineering Management & Systems Engineering

EMSE 6801
Systems Engineering I
Session – L10: Implementation to Validation

Dr. J.P. Blackford

[email protected]
These slides were developed to present and amplify the content of
INCOSE’s Handbooks and as such incorporate text and graphics
from the handbooks. These items retain INCOSE’s copyrights
statement:

© 2015 INCOSE. All Rights

Reserved. May not be copied,
scanned, or duplicated, in whole
or in part, except for use as
permitted in a license obtained
from INCOSE for classroom use.
EMSE-6801 Systems Engineering I 2
 Agenda

◼ Implementation Process
◼ Integration Process
◼ Verification Process
◼ Transition Process
◼ Validation Process

EMSE-6801 Systems Engineering I 3

 Implementation Process Purpose

◼ The purpose of the Implementation process is to realize

a specified system element.

Source: INCOSE Systems EMSE-6801 Systems Engineering I 4

Engineering Handbook v4
 Implementation Process

Source: EMSE-6801 Systems Engineering I 5

http://dilbert.com/strip/2015-04-22
 Implementation Process

◼ Implementation creates or fabricates a system element

conforming to that element’s detailed description, as
embodied by its
❑ Requirements
❑ Architecture
❑ Design
❑ Interfaces

An element is constructed employing appropriate

technology and industry practices

Source: INCOSE Systems EMSE-6801 Systems Engineering I 6

Engineering Handbook v4
 Implementation Process

◼ Engineers follow the requirements allocated to the

system element to fabricate, code, or build each
individual element using specified
❑ Materials
❑ Processes
❑ Physical or logical arrangements
❑ Standards
❑ Technologies
❑ Information flows outlined in detailed drawings
❑ Other design documentation

◼ System requirements are verified

◼ Stakeholder requirements are validated

Source: INCOSE Systems EMSE-6801 Systems Engineering I 7

Engineering Handbook v4
 Implementation Concepts

◼ Four major forms of system elements are typically

focused upon during the implementation process
❑ Hardware/physical – output is fabricated or adapted hardware or
physical element
❑ Software – output is software code and executable images
❑ Operational resources – output includes procedures and training
❑ Services – output includes specified services
◼ Implementation process supports either the creation
(fabrication or development) or adaptation of system
elements
❑ For reused or acquired system elements, the process allows for
adaptation of the elements to satisfy the needs of the SOI
❑ For newly created products there is generally more flexibility to
be designed and developed to meet the needs of the SOI
Source: INCOSE Systems EMSE-6801 Systems Engineering I 8
Engineering Handbook v4
 Implementation Process

Source: INCOSE Systems EMSE-6801 Systems Engineering I 9

Engineering Handbook v4
 Implementation Process – Major Steps

◼ Prepare for Implementation

◼ Perform Implementation

◼ Manage Results of Implementation

Source: INCOSE Systems EMSE-6801 Systems Engineering I 10

Engineering Handbook v4
 Prepare for Implementation

◼ Define
❑ Fabrication/coding procedures
❑ Tools and equipment to be used
❑ Implementation tolerances
❑ Means and criteria for auditing configuration of resulting
elements to the detailed design
❑ Refine the implementation strategy to achieve consistent and
repeatable element production
◼ Retain in project decision database

◼ Elicit any constraints imposed by implementation

technology, strategy, or implementation enabling
systems
◼ Document the plan for acquiring or gaining access to
resources needed during implementation
Source: INCOSE Systems EMSE-6801 Systems Engineering I 11
Engineering Handbook v4
 Perform Implementation

◼ Develop data for training users on correct and safe

procedures for operating and maintaining the element,
either as stand-alone or as part of a larger system
◼ Complete detailed product, process, material
specifications (“build-to” or “code-to” documents), and
corresponding analyses
◼ Ensure the realization of the system elements per the
detailed product, process, and material specifications
and produce documented evidence of implementation
compliance
❑ Conduct peer reviews and testing
❑ Conduct hardware conformation audits

Source: INCOSE Systems EMSE-6801 Systems Engineering I 12

Engineering Handbook v4
 Perform Implementation

◼ Prepare initial training capability and draft training

documentation
◼ Prepare a hazardous materials log, if applicable
◼ Determine the packaging and storage requirements for
the system element, and ensure initiation of the
packaging and/or storage

Source: INCOSE Systems EMSE-6801 Systems Engineering I 13

Engineering Handbook v4
 Manage Results of Implementation

◼ Identify and record implementation results

◼ Record any anomalies encountered during the
implementation process
❑ Analyze and resolve the anomalies (corrective actions or
improvements) using the quality assurance process
◼ Establish and maintain traceability of the implemented
system elements with the system architecture, design,
and system and interface requirements
◼ Provide baseline information for configuration
management

Source: INCOSE Systems EMSE-6801 Systems Engineering I 14

Engineering Handbook v4
 Agenda

◼ Implementation Process
◼ Integration Process
◼ Verification Process
◼ Transition Process
◼ Validation Process

EMSE-6801 Systems Engineering I 15

 Integration, Verification, and Validation

◼ Vee Concept:
❑ Definition/decomposition/validation
process on left side of the “V”
❑ Corresponding
integration/verification process on
right side of the “V”
◼ Proper development process will
have direct correspondence
between the definition /
decomposition / validation activities
and the integration / verification
activities
◼ For every specification, there
should be an independent
integration and verification activity.

Source: INCOSE Systems EMSE-6801 Systems Engineering I 16

Engineering Handbook v4
 Vee’s Through the Life Cycle

Source: INCOSE Systems EMSE-6801 Systems Engineering I 17

Engineering Handbook v2a
 Moving Through the Left Side of the Vee

Source: INCOSE Systems EMSE-6801 Systems Engineering I 18

Engineering Handbook v4
 Moving Through the Right Side of the Vee

Source: INCOSE Systems EMSE-6801 Systems Engineering I 19

Engineering Handbook v4
 Integration Process Purpose

◼ The purpose of the Integration process is to synthesize a

set of system elements into a realized system (product
or service) that satisfies system requirements,
architecture, and design

Source: INCOSE Systems EMSE-6801 Systems Engineering I 20

Engineering Handbook v4
 Integration Process

◼ Integration process consists of progressively assembling

the implemented system elements (hardware, software,
and operational resources) that compose the SOI and
verifying the correctness of the static and dynamic
aspects of interfaces between the implemented system
elements.

Strong focus on achievement of two areas: 1) interfaces

to ensure the intended operation of the system elements,
and 2) interoperation with other systems

Source: INCOSE Systems EMSE-6801 Systems Engineering I 21

Engineering Handbook v4
 Integration Process
◼ Integration process works closely with the verification
and validation process (V&V) processes.
❑ Iterates, as appropriate, with the V&V processes
❑ For each system element integration, the verification process is
invoked to check the correctness of implementation of
architectural characteristics and design properties
❑ For each system element the validation process may be invoked
to check that the individual system elements provide the function
intended
◼ Integration process checks that all boundaries…have
been correctly identified and described (physical, logical,
and human system interfaces and interactions)
◼ Integration process checks that all system element
functional, performance, and design requirements and
constraints are satisfied
Source: INCOSE Systems EMSE-6801 Systems Engineering I 22
Engineering Handbook v4
 Integration Process

The interaction of the integration process with the system

definition processes (i.e. system requirements definition,
architecture definition, and design definition) early in the
development is essential for avoiding integration issues
during the system realization.

Source: INCOSE Systems EMSE-6801 Systems Engineering I 23

Engineering Handbook v4
 What is the Concept of Aggregate?

◼ The physical integration of a system is based on the

notion of an “aggregate”
◼ An aggregate is made up of several implemented system
elements and their physical interfaces
◼ An aggregate is characterized by a configuration that
specifies the implemented system elements to be
physically assembled (i.e. built) and their configurations
status
◼ Each aggregate has a set of verification actions applied
to it
❑ Verification actions are accomplished using the aggregate plus
verification tools (e.g., simulated system elements, stubs or
caps, activators, harnesses, measuring devices, etc.)
Source: INCOSE Systems EMSE-6801 Systems Engineering I 24
Engineering Handbook v4
 Integration by Level of System

◼ According to the Vee model, the system

definition (top-down branch) is done by
successive levels of decomposition; each
level corresponds to physical architecture
of systems and system elements

◼ Integration (bottom-up branch) consists in

following the opposite way of composition
level by level.
❑ Integration of implemented system elements is
done on the basis of the physical architecture

Source: INCOSE Systems EMSE-6801 Systems Engineering I 25

Engineering Handbook v4
 Integration Strategy and Approaches

◼ Integration of implemented system elements is

performed according to a predefined strategy
❑ Described in an integration plan which defines
◼ Configuration of expected aggregates of implemented system
elements
◼ Order of assembly of these aggregates to carry out efficient
verification actions and validation actions
❑ Elaborated in coordination with selected verification strategy and
validation strategy

Source: INCOSE Systems EMSE-6801 Systems Engineering I 26

Engineering Handbook v4
 Integration Strategy and Approaches: Techniques

◼ Global integration – “big-bang integration”; all delivered

implemented system elements are assembled in only
one step
◼ Integration “with the stream” – the delivered system
elements are assembled as they become available
◼ Incremental integration – In a predefined order, one or a
very few system elements are added to an already
integrated increment of system elements
◼ Subset integration – system elements are assembled by
subsets (a subset is an aggregate), and then subsets are
assembled together; could be called “functional chains
integration”
Source: INCOSE Systems EMSE-6801 Systems Engineering I 27
Engineering Handbook v4
 Integration Strategy and Approaches: Techniques
◼ Top-down integration – system elements or aggregates
are integrated in their activation or utilization order
◼ Bottom-up integration – system elements or aggregates
are integrated in the opposite order of the activation or
utilization
◼ Criterion-driven integration – the most critical system
elements compared to the selected criterion are first
integrated (e.g., dependability, complexity, technological
innovation)
◼ Reorganization of coupling matrices – coupling matrices
are useful for highlighting interfaces during architecture
definition as well as integration
Usually, integration strategy is defined as a combination of these
approaches and techniques in order to optimize the integration
Source: INCOSE Systems EMSE-6801 Systems Engineering I 28
Engineering Handbook v4
 Integration Process

Source: INCOSE Systems EMSE-6801 Systems Engineering I 29

Engineering Handbook v4
 Integration Process

◼ Prepare for Integration

◼ Perform Integration

◼ Manage Results of Integration

Source: INCOSE Systems EMSE-6801 Systems Engineering I 30

Engineering Handbook v4
 Prepare for Integration
◼ Define critical checkpoints to provide assurance of the
correct behavior and operation of interfaces and
functions of the system elements
◼ Establish the integration strategy that minimizes
integration time, costs, and risks
❑ Define an optimized sequence order of assembly aggregates, composed of
system elements, based on the system architecture definition and on appropriate
integration approaches and techniques
❑ Define the configurations of the aggregates to be built and verified
❑ Define the assembly procedures and related enablers
◼ Identify integration constraints on the SOI, arising from the
integration strategy, to be incorporated in the system requirements,
architecture, and design
◼ The acquisition of the enablers can be done through various ways
such as rental, procurement, development, reuse, and
subcontracting
Source: INCOSE Systems EMSE-6801 Systems Engineering I 31
Engineering Handbook v4
 Perform Integration
◼ Successively integrate system element configurations
until the complete system is synthesized
◼ Assemble the verified and validated system elements to
form the incremental aggregate using the defined
assembly procedures, the related integration enabling
systems, and the interface control definitions
◼ Invoke the system V&V processes, as needed, to check
the correct implementation of architectural
characteristics and design properties and to check that
the individual system elements provide the functions
intended

Source: INCOSE Systems EMSE-6801 Systems Engineering I 32

Engineering Handbook v4
 Manage Results of Integration
◼ Identify and record the results of integration
◼ Maintain bidirectional traceability of the updated integrated system
elements with the updated system architecture, design, and system
and interface requirements
❑ Maintain the records, including configuration updates
◼ Record anomalies observed during the integration process
❑ Identify corrective actions or improvements
❑ Resolve them using the quality assurance process
◼ Update the integration strategy and schedule according to the
progress of the project
❑ The order of system elements assembly can be redefined or
rescheduled because of unexpected events or unavailability of system
elements as planned

Coordinate integration activities with development stakeholders

(e.g., project manager, architects, designers, configuration mgr)
Source: INCOSE Systems EMSE-6801 Systems Engineering I 33
Engineering Handbook v4
 Agenda

◼ Implementation Process
◼ Integration Process
◼ Verification Process
◼ Transition Process
◼ Validation Process

EMSE-6801 Systems Engineering I 34

 Verification and Validation

◼ System verification and validation activities are very

similar, but they address different issues:
❑ Verification addresses whether the system, its elements, its
interfaces, and incremental work products satisfy their
requirements
❑ Validation confirms that the system, as built (or as it will be built),
will satisfy the user’s needs
◼ Verification ensures the conformance to those
requirements, and validation ensures the requirements
and the system implementation provide the right solution
to the customer’s problem. (ANSI/EIA-731)
◼ Verification ensures that “you built it right”
◼ Validation ensures that “you built the right thing”
Source: INCOSE Systems EMSE-6801 Systems Engineering I 35
Engineering Handbook v4
 Verification

◼ Verification includes the tasks, actions and activities

performed to:
❑ Evaluate progress
❑ Determine effectiveness of the evolving
system solutions
◼ Could be people, products and processes
❑ Measure compliance with requirements

◼ Analysis (including simulation, demonstration, test and

inspection) are verification approaches used to evaluate:
risk; people, product and process capabilities;
compliance with requirements, and proof of concept

Source: INCOSE Systems EMSE-6801 Systems Engineering I 36

Engineering Handbook v2a
 Verification Process Purpose

◼ The purpose of the Verification process is to provide

objective evidence that a system or system element
fulfils it specified requirements and characteristics

◼ The purpose of the Verification process is to provide

evidence that no error/defect/fault has been introduced
at the time of any transformation of inputs into outputs

◼ The Verification process is used to confirm that this

transformation has been made “right” according to the
requirements and selected methods, techniques, and
standards, or rules

Source: INCOSE Systems EMSE-6801 Systems Engineering I 37

Engineering Handbook v4
 What is a Verification Action?
◼ A verification action describes what must be verified
❑ A requirement
❑ A characteristic
❑ A property as reference
◼ A verification action describes on which item an action is
to be verified
❑ Requirement
❑ Function
❑ Interface
❑ System element
❑ System
◼ A verification action describes the expected result

Source: INCOSE Systems EMSE-6801 Systems Engineering I 38

Engineering Handbook v4
 What is a Verification Action?
◼ A verification action describes the verification technique
to apply when verifying an action
❑ Inspection
❑ Analysis
❑ Demonstration
❑ Test
◼ A verification action describes on which level of
decomposition of the system an action is to be verified
❑ SOI
❑ Intermediate level system element
❑ Leaf level system element
The Performance of a Verification Action onto the submitted item provides
an obtained result which is compared with the expected result. The
comparison allows the correctness of the item to be determined.
Source: INCOSE Systems EMSE-6801 Systems Engineering I 39
Engineering Handbook v4
 What is a Verification Action?

Source: INCOSE Systems EMSE-6801 Systems Engineering I 40

Engineering Handbook v4
 Verification Techniques
◼ Inspection – is based upon visual or dimensional examination of an
element; relies on the human senses or uses simple methods of
measurement and handling
◼ Analysis – is based upon analytical evidence obtained without any
intervention on the submitted element using mathematical or
probabilistic calculation, logical reasoning (including the theory of
predicates), modeling, and/or simulation under defined conditions to
show theoretical compliance
◼ Demonstration – is used to show correct operation of the submitted
element against operational and observable characteristics without
using physical measurements (no or minimal instrumentation or test
equipment)
◼ Test – is performed onto the submitted element by which functional,
measurable characteristics, operability, supportability, or
performance capability is quantitatively verified when subjected to
controlled conditions that are real or simulated

Source: INCOSE Systems EMSE-6801 Systems Engineering I 41

Engineering Handbook v4
 Verification Techniques
◼ Analogy or similarity – is based upon evidence of similar elements to
the submitted element or on experience feedback
❑ Absolutely necessary to show by prediction that the context is invariant
and that the outcomes are transportable
◼ Simulation – is performed on models or mock-ups (not on the
actual/physical elements) for verifying features and performance as
designed
◼ Sampling – is based upon verification of characteristics using
samples

Note: for techniques that do not include stimuli of the system

element, no characteristics (exogenous attributes) can be
observed only properties (endogenous attributes)
Source: INCOSE Systems EMSE-6801 Systems Engineering I 42
Engineering Handbook v4
 Integration, Verification, and Validation of the System
◼ In most cases, it is more appropriate to begin verification
activities during development and to continue them into
deployment and use

◼ Once system elements have been realized, their

integration to form the whole system is performed

◼ Integration assembles developed capabilities (via system

elements) in preparation for verification actions as stated
in the integration process

Misconception: Verification occurs after integration and before

validation
Source: INCOSE Systems EMSE-6801 Systems Engineering I 43
Engineering Handbook v4
 Verification Level per Level

◼ Every system and system

element is verified, and any
findings possibly corrected before
being integrated into the system
of the higher level
◼ Systems and system elements
are partially integrated in subsets
(aggregates) in order to limit the
number of properties to be
verified within a single step
◼ For each level, it is necessary to
make sure by a set of verification
actions that features stated at
preceding level are not adversely
affected

Source: INCOSE Systems EMSE-6801 Systems Engineering I 44

Engineering Handbook v4
 Verification Process

Source: INCOSE Systems EMSE-6801 Systems Engineering I 45

Engineering Handbook v4
 Verification Process

◼ Prepare for Verification

◼ Perform Verification

◼ Manage Results of Verification

Source: INCOSE Systems EMSE-6801 Systems Engineering I 46

Engineering Handbook v4
 Prepare for Verification

◼ Develop a strategy that prioritizes the verification actions

to minimize costs and risks while maximizing operational
behaviors
❑ Establish a list of the items for verification, including
requirements, architectural characteristics, or design properties,
and define the corresponding verification actions
❑ Establish a list of verification constraints that need to be
considered
❑ Considering the constraints, plan for the methods or techniques
that will be applied for each verification action
❑ Establish the scope of the verification

Develop Verification Strategy and Document in Verification Plan

Source: INCOSE Systems EMSE-6801 Systems Engineering I 47
Engineering Handbook v4
 Prepare for Verification
◼ Develop the verification procedures that support the
verification actions
❑ Schedule the execution of verification actions in the project steps and
define the configuration of submitted items to verification actions
◼ Identify verification constraints on the system or system
elements, arising from the verification strategy, that
relate to specific system requirements, architecture
elements, or design elements
◼ Ensure that the necessary enabling systems, products,
or services required for the verification actions are
available, when needed
❑ The planning includes the identification of requirements and interfaces
for the enablers

Develop Verification Procedures that Support the Verification Plan

Source: INCOSE Systems EMSE-6801 Systems Engineering I 48
Engineering Handbook v4
 Perform Verification

◼ Implement the verification plan developed in previous

step
❑ Item to be verified
❑ Expected results and success criteria
❑ Selected verification method or technique
❑ The data needed
❑ The corresponding enabling systems, products, or services
◼ Using the verification procedures, execute the
verification actions and record the results
◼ Analyze the verification results against any established
expectations and success criteria to determine whether
the element being verified indicates conformance

Source: INCOSE Systems EMSE-6801 Systems Engineering I 49

Engineering Handbook v4
 Manage Results of Verification
◼ Identify and record verification results and enter data in the
Requirements Verification and Traceability Matrix (RVTM)
◼ Record anomalies observed during the verification process
❑ Identify corrective actions or improvements
❑ Resolve them using the quality assurance process
◼ Establish and maintain bidirectional traceability of the verified
system elements with the system architecture, design, and system
and interface requirements that are needed for verification
◼ Provide baseline information for configuration management
◼ Update the verification strategy and schedule according to the
progress of the project
❑ Planned verification actions can be redefined or rescheduled as necessary

Coordinate verification activities with development stakeholders

(e.g., project manager, architects, designers, configuration mgr)
Source: INCOSE Systems EMSE-6801 Systems Engineering I 50
Engineering Handbook v4
 Agenda

◼ Implementation Process
◼ Integration Process
◼ Verification Process
◼ Transition Process
◼ Validation Process

EMSE-6801 Systems Engineering I 51

 Transition Process Purpose

◼ The purpose of the Transition process is to establish a

capability for a system to provide services specified by
stakeholder requirements in the operational environment

Source: INCOSE Systems EMSE-6801 Systems Engineering I 52

Engineering Handbook v4
 Transition Process
◼ The Transition process enables the transfer of custody of
the system and responsibility for system support from
one organizational entity to another
❑ Development team to Operations and Support team
◼ The Transition process installs a verified system in the
operational environment along with relevant enabling
systems, products, or services
◼ Upon successful results from the verification process,
the acquirer accepts that the system meets the specified
system requirements in the intended operational
environment prior to allowing change in control,
ownership, and/or custody
The Transition process must be carefully planned to avoid
surprises and recrimination on either of side of the agreement
Source: INCOSE Systems EMSE-6801 Systems Engineering I 53
Engineering Handbook v4
 Transition Process

Source: INCOSE Systems EMSE-6801 Systems Engineering I 54

Engineering Handbook v4
 Transition Process
◼ Prepare for Transition

◼ Perform Transition

◼ Manage Results of Transition

Source: INCOSE Systems EMSE-6801 Systems Engineering I 55

Engineering Handbook v4
 Prepare for Transition
◼ Plan for the transition of the system. The strategy should
include
❑ Operator training
❑ Logistics support
❑ Delivery strategy
❑ Problem rectification/resolution strategy
◼ Develop installation procedures
◼ Ensure that the necessary enabling systems, products,
or services required for transition are available, when
needed
❑ The planning includes the identification of requirements and
interfaces for the enablers

Source: INCOSE Systems EMSE-6801 Systems Engineering I 56

Engineering Handbook v4
 Perform the Transition
◼ Using the installation procedures, install the system
◼ Train the users in the proper use of the system and
affirm users have the knowledge and skill levels
necessary to perform operation and maintenance
activities
◼ Receive final confirmation that the installed system can
provide its required functions and can be sustained by
the enabling systems and services
◼ After the demonstration of functionality in the operational
site and any review for operational readiness, the system
can be placed into service

Source: INCOSE Systems EMSE-6801 Systems Engineering I 57

Engineering Handbook v4
 Manage Results of Transition
◼ Capture post-implementation incidents and problems
❑ May lead to corrective actions and changes in requirements
❑ The quality assurance process is used for incident and problem
resolution that is reported during performance of the transition
◼ Record anomalies observed during the transition
process. These provide awareness of the results,
information needed to address anomalies, and a
historical record
◼ Maintain bidirectional traceability of the transition system
elements with the transition strategy, system
architecture, design, and system requirements
◼ Provide baseline information to configuration
management

Source: INCOSE Systems EMSE-6801 Systems Engineering I 58

Engineering Handbook v4
 Agenda

◼ Implementation Process
◼ Integration Process
◼ Verification Process
◼ Transition Process
◼ Validation Process

EMSE-6801 Systems Engineering I 59

 Validation Process Purpose

◼ The purpose of the Validation process is to provide

objective evidence that the system, when in use, fulfils
its business or mission objectives and stakeholder
requirements, achieving its intended use in its intended
environment

Source: INCOSE Systems EMSE-6801 Systems Engineering I 60

Engineering Handbook v4
 Validation Process

◼ The Validation process is applied to an SOI, or any

system or system element that composes it, at the
appropriate points in the life cycle stages to provide
confidence that the right system (or system element) has
been built
◼ The Validation process can be applied to any system
element or engineering item of the system or its
definition that has been defined or realized
◼ The Validation process helps to ensure that the system
or any system element meets the need of its stakeholder
in the life cycle (i.e., produced what was intended-the
“right” result)

Source: INCOSE Systems EMSE-6801 Systems Engineering I 61

Engineering Handbook v4
 What is a Validation Action?
◼ A validation action describes what must be validated
❑ An operational scenario
❑ A requirement
❑ A set of requirements as reference
◼ A validation action describes on which item an action is
to be validated
❑ Requirement
❑ Function
❑ Interface
❑ System element
❑ System
◼ A validation action describes the expected result

Source: INCOSE Systems EMSE-6801 Systems Engineering I 62

Engineering Handbook v4
 What is a Validation Action?
◼ A validation action describes the validation technique to
apply when validating an action
❑ Inspection
❑ Analysis
❑ Demonstration
❑ Test
◼ A validation action describes on which level of
decomposition of the system an action is to be validated
❑ SOI
❑ Intermediate level system element
❑ Leaf level system element
The Performance of a Validation Action onto the submitted item provides
an obtained result which is compared with the expected result. The
comparison allows the project to judge the element’s acceptability
regarding the relevance in the context of use
Source: INCOSE Systems EMSE-6801 Systems Engineering I 63
Engineering Handbook v4
 Validation Techniques
◼ Validation techniques are the same as those used for verification,
but with different purposes
❑ Verification is used to show compliance with the specified system
requirements and to detect errors/defects/faults
❑ Validation is to prove satisfaction of the desired operational capability
through showing operational scenarios and stakeholder requirements
can be met

◼ A Requirements and Validation Traceability Matrix may be used to

record data such as validation action list, selected validation
method/technique to validate implementation of every engineering
item, the expected results, and the obtained results when a
validation action was performed

Source: INCOSE Systems EMSE-6801 Systems Engineering I 64

Engineering Handbook v4
 Validation, Operational Validation, Acceptance, and
Certification
◼ Validation and Operational Validation validation concerns
the global system seen as a whole
❑ Based upon the totality of requirements (system requirements,
stakeholder requirements)
❑ Is obtained gradually throughout the development stage by
pursuing several nonexclusive ways
◼ Cumulating V&V actions’ results provided by application of the
corresponding processes to every engineering item
◼ Performing final validation actions onto the complete integrated system in an
industrial environment
◼ Performing operational validation actions onto the complete system in its
operation environment
❑ Goal is to completely validate the system capability to meet all
requirements prior to the production and utilization stages
◼ Problems uncovered in these stages are very costly to correct
◼ Validation results are an important element of decision gate reviews

Source: INCOSE Systems EMSE-6801 Systems Engineering I 65

Engineering Handbook v4
 Validation, Operational Validation, Acceptance, and
Certification
◼ Acceptance is an activity conducted prior to transition
such that the acquirer can decide that the system is
ready to change ownership from supplier to acquirer
❑ Exercise of operational validation actions, or
❑ Review of validation results

◼ Certification is a written assurance that the product or

article has been developed, and can perform is assigned
functions, in accordance with legal and industrial
standards
❑ Typically performed by outside authorities, without direction as to
how the requirements are to be verified

Source: INCOSE Systems EMSE-6801 Systems Engineering I 66

Engineering Handbook v4
 Validation, Operational Validation, Acceptance, and
Certification
◼ Readiness for Use assessment may occur many times in
the life cycle, including
❑ the first article delivery
❑ Completion of production
❑ Following maintenance actions

◼ Qualification requires that all V&V actions have been

successfully performed. This includes
❑ The SOI
❑ All the interfaces with its environment
❑ Demonstration that the characteristics or properties of the
realized system, including margins, meet the applicable system
requirements and/or stakeholder requirements
❑ Acceptance review and/or an operational readiness review
Source: INCOSE Systems EMSE-6801 Systems Engineering I 67
Engineering Handbook v4
 Validation Level per Level

◼ Every system and system

element is validated, and possibly
corrected before being integrated
into the parent system of the
higher level
◼ Systems and system elements
are partially integrated in subsets
(aggregates) in order to limit the
number of properties to be
validated within a single step
◼ For each level, it is necessary to
make sure by a set of final
validation actions that features
stated at preceding level are not
adversely affected

Source: INCOSE Systems EMSE-6801 Systems Engineering I 68

Engineering Handbook v4
 Validation Process

Source: INCOSE Systems EMSE-6801 Systems Engineering I 69

Engineering Handbook v4
 Validation Process

◼ Prepare for Validation

◼ Perform Validation

◼ Manage Results of Validation

Source: INCOSE Systems EMSE-6801 Systems Engineering I 70

Engineering Handbook v4
 Prepare for Validation
◼ Establish the validation strategy, which is often part of a validation
plan, that optimizes the number and type of validation actions while
minimizing costs and risks
❑ Identify stakeholders who will be involved in the validation activities and define
their roles and responsibilities
❑ The scope of the validation plan is dependent on the life cycle stage and the
progress within it
❑ Establish a list of validation constraints that need to be considered
❑ With appropriate consideration to the constraints, select suitable validation
approach to be applied (depending upon the life cycle stage)
◼ Inspection
◼ Analysis
◼ Demonstration
◼ Test
❑ Identify any enablers needed
❑ Prioritize the validation actions and evaluate against constraints, risks, type of
system, project objectives, and other relevant criteria

Develop Validation Strategy and Document in Validation Plan

Source: INCOSE Systems EMSE-6801 Systems Engineering I 71
Engineering Handbook v4
 Prepare for Validation
❑ Determine if there are any validation gaps and that the resulting validation
actions will provide an acceptable level of confidence that the system or system
element will meet the identified needs
❑ Ensure appropriate scheduling via the project planning process to meet the
requirements for the execution of the validation actions in the applicable project
steps
❑ Define the configuration of submitted items to validation actions
◼ Identify validation constraints on the system, arising from the
validation strategy, to be incorporated in the stakeholder
requirements
◼ Ensure that the necessary enabling systems, products, or services
required for the validation actions are available, when needed
❑ The planning includes the identification of requirements and interfaces for the
enablers

Source: INCOSE Systems EMSE-6801 Systems Engineering I 72

Engineering Handbook v4
 Perform Validation
◼ Develop the validation procedures that support the
validation actions
◼ Ensure readiness to conduct validation
❑ Availability and configuration status of the system/item
❑ Availability of the validation enablers
❑ Qualified personnel or operators, resources, etc.
◼ Conduct validation actions in accordance with the
procedures
❑ This should include performing the actions in the operational
environment or one as close to it as possible
❑ Record results of the validation actions, as they are performed

Develop Validation Procedures that Support the Validation Plan

Source: INCOSE Systems EMSE-6801 Systems Engineering I 73
Engineering Handbook v4
 Manage Results of Validation
◼ Identify and record validation results and enter data in the validation
report (including any necessary updates to the RVTM)
◼ Record anomalies observed during the validation process
❑ Identify corrective actions or improvements
❑ Resolve them using the quality assurance process
◼ Obtain acquirer acceptance of validation results
◼ Establish and maintain bidirectional traceability of the validated
system elements with the validation strategy, business/mission
analysis, stakeholder requirements, system architecture, design,
and system requirements that are needed for verification
◼ Provide baseline information for configuration management
◼ Update the validation strategy and schedule according to the
progress of the project
❑ Planned validation actions can be redefined or rescheduled as necessary

Source: INCOSE Systems EMSE-6801 Systems Engineering I 74

Engineering Handbook v4
 Test Categories: Development Tests
◼ Conducted on new items to demonstrate proof of concept
◼ Testing may be done on breadboard, brassboard, engineering
prototype, or partial model
◼ Used to reduce risk and prove feasibility

Source: INCOSE Systems EMSE-6801 Systems Engineering I 75

Engineering Handbook v2a
 Test Categories: Qualification Tests

◼ Conducted to prove the

design on the first article
produced
◼ Predetermined margin
above expected operating
conditions, for instance by
using elevated
environmental conditions
for hardware
◼ Hardware qualification test
items cannot generally be
used in an operational test
due to overstress
Source: INCOSE Systems EMSE-6801 Systems Engineering I 76
Engineering Handbook v2a
 Test Categories: Acceptance Test

◼ Conducted to prove workmanship and materials on the

second and succeeding articles
◼ Tests conducted are a subset of the qualification tests,
performed at lower stress levels

Source: INCOSE Systems EMSE-6801 Systems Engineering I 77

Engineering Handbook v2a
 Test Categories: Operational Test
◼ Conducted to verify that the item
meets its specification requirements
when subjected to the actual
operational environment

◼ Some requirements, such as

radiation hardening, may be fully
verified at the parts level by testing

◼ Many requirements at the system

level may be verified only by
simulation (and supported by test
data from lower levels)

Source: INCOSE Systems EMSE-6801 Systems Engineering I 78

Engineering Handbook v2a
 V&V: The Need for Balance
◼ In developing the most cost-effective verification program, a number
of trades must be considered
◼ The objective in conducting these trades is to achieve a minimum
cost verification program while maintaining low risk

Source: INCOSE Systems EMSE-6801 Systems Engineering I 79

Engineering Handbook v2a
Questions?

EMSE-6801 Systems Engineering I 80