Ad 1183573
Ad 1183573
Ad 1183573
ORGANIZATION ORGANIZATION
AC/323(SCI-236)TP/964 www.sto.nato.int
This AGARDograph has been sponsored by the SCI Flight Test Technical
Team (FT3) of the Systems Concepts and Integration Panel (SCI) of STO.
AC/323(SCI-236)TP/964 www.sto.nato.int
This AGARDograph has been sponsored by the SCI Flight Test Technical
Team (FT3) of the Systems Concepts and Integration Panel (SCI) of STO.
Authored by
Dennis Morley, Jeremy Newsome, Dean Moore, Perry Comeau,
Bradford A. Neal, Patrick C. Stoliker, Arun Karwal,
Christian Buck, Önder Kirli, Christian B. Rice,
Fred P. Henderson, Jonathan R. Stevenson,
Joseph A. Mortensen
The NATO Science and Technology Organization
Science & Technology (S&T) in the NATO context is defined as the selective and rigorous generation and application of
state-of-the-art, validated knowledge for defence and security purposes. S&T activities embrace scientific research,
technology development, transition, application and field-testing, experimentation and a range of related scientific
activities that include systems engineering, operational research and analysis, synthesis, integration and validation of
knowledge derived through the scientific method.
In NATO, S&T is addressed using different business models, namely a collaborative business model where NATO
provides a forum where NATO Nations and partner Nations elect to use their national resources to define, conduct and
promote cooperative research and information exchange, and secondly an in-house delivery business model where S&T
activities are conducted in a NATO dedicated executive body, having its own personnel, capabilities and infrastructure.
The mission of the NATO Science & Technology Organization (STO) is to help position the Nations’ and NATO’s S&T
investments as a strategic enabler of the knowledge and technology advantage for the defence and security posture of
NATO Nations and partner Nations, by conducting and promoting S&T activities that augment and leverage the
capabilities and programmes of the Alliance, of the NATO Nations and the partner Nations, in support of NATO’s
objectives, and contributing to NATO’s ability to enable and influence security and defence related capability
development and threat mitigation in NATO Nations and partner Nations, in accordance with NATO policies.
The total spectrum of this collaborative effort is addressed by six Technical Panels who manage a wide range of
scientific research activities, a Group specialising in modelling and simulation, plus a Committee dedicated to
supporting the information management needs of the organization.
• AVT Applied Vehicle Technology Panel
• HFM Human Factors and Medicine Panel
• IST Information Systems Technology Panel
• NMSG NATO Modelling and Simulation Group
• SAS System Analysis and Studies Panel
• SCI Systems Concepts and Integration Panel
• SET Sensors and Electronics Technology Panel
These Panels and Group are the power-house of the collaborative model and are made up of national representatives as
well as recognised world-class scientists, engineers and information specialists. In addition to providing critical
technical oversight, they also provide a communication link to military users and other NATO bodies.
The scientific and technological work is carried out by Technical Teams, created under one or more of these eight
bodies, for specific research activities which have a defined duration. These research activities can take a variety of
forms, including Task Groups, Workshops, Symposia, Specialists’ Meetings, Lecture Series and Technical Courses.
The content of this publication has been reproduced directly from material supplied by STO or the authors.
ISBN 978-92-837-2285-4
Single copies of this publication or of a part of it may be made for individual use only by those organisations or
individuals in NATO Nations defined by the limitation notice printed on the front cover. The approval of the STO
Information Management Systems Branch is required for more than one copy to be made or an extract included in
another publication. Requests to do so should be sent to the address on the back cover.
ii STO-AG-300-V32
AGARDograph Series 160 & 300
Soon after its founding in 1952, the Advisory Group for Aerospace Research and Development (AGARD)
recognized the need for a comprehensive publication on Flight Test Techniques and the associated
instrumentation. Under the direction of the Flight Test Panel (later the Flight Vehicle Integration Panel,
or FVP) a Flight Test Manual was published in the years 1954 to 1956. This original manual was prepared
as four volumes: 1. Performance, 2. Stability and Control, 3. Instrumentation Catalog, and 4. Instrumentation
Systems.
As a result of the advances in the field of flight test instrumentation, the Flight Test Instrumentation Group
was formed in 1968 to update Volumes 3 and 4 of the Flight Test Manual by publication of the Flight Test
Instrumentation Series, AGARDograph 160. In its published volumes AGARDograph 160 has covered
recent developments in flight test instrumentation.
In 1978, it was decided that further specialist monographs should be published covering aspects
of Volumes 1 and 2 of the original Flight Test Manual, including the flight testing of aircraft systems.
In March 1981, the Flight Test Techniques Group (FTTG) was established to carry out this task and
to continue the task of producing volumes in the Flight Test Instrumentation Series. The monographs
of this new series (with the exception of AG237 which was separately numbered) are being published
as individually numbered volumes in AGARDograph 300. In 1993, the Flight Test Techniques Group was
transformed into the Flight Test Editorial Committee (FTEC), thereby better reflecting its actual status
within AGARD. Fortunately, the work on volumes could continue without being affected by this change.
Since that time The Flight Test Editorial Committee has had a number of changes of identity which have
in recent years been closely associated with a series of Task Group identities bearing the acronym FT3.
This period has also seen FT3 migrate from the Air Vehicle Technology Panel to The Systems Concepts
and Integration Panel reflecting the changing nature and focus of NATO Flight Testing. This
AGARDograph is sponsored by FT3 against their current Task Group, SCI-305 Flight Test Technical
Team (FT3).
An Annex at the end of each volume in both the AGARDograph 160 and AGARDograph 300 series lists
the volumes that have been published in the Flight Test Instrumentation Series (AG 160) and the Flight
Test Techniques Series (AG 300), plus the volumes that were in preparation at that time.
STO-AG-300-V32 iii
Flight Test Safety and Risk Management
(STO-AG-300-V32)
Executive Summary
Through a series of Case Studies this AGARDograph brings together the Flight Test Safety and Risk Management
Practices of a number of representative NATO Flight Test Organisations. It then provides an assessment of the
content of the studies looking for common and best practice in this critical area of Flight Testing.
It is the result of an activity, SCI-236, initiated by the NATO STO Flight Test Technical Team, FT3, to capture,
assess and disseminate the Flight Test Safety and Risk Management Practices deployed amongst the Member
Nations.
Through its application both the NATO and wider Flight Test Community will have access to information that
will help to manage flight test risk to the levels and standards that are expected today.
iv STO-AG-300-V32
Sécurité des essais en vol et gestion du risque
(STO-AG-300-V32)
Synthèse
Par le biais d’une série d’études de cas, la présente AGARDographie réunit les pratiques de sécurité
des essais en vol et de gestion du risque d’un certain nombre d’organisations représentatives de l’OTAN
en la matière. Elle fournit ensuite une évaluation du contenu des études, en recherchant les meilleures
pratiques communes dans ce domaine critique des essais en vol.
Cette évaluation résulte de l’activité du SCI-236, entamée par l’équipe technique des essais en vol
de la STO de l’OTAN (FT3), visant à recueillir, évaluer et diffuser les pratiques de sécurité des essais
en vol et de gestion du risque déployées parmi les pays membres.
Ainsi, l’OTAN et la communauté des essais en vol dans son ensemble auront accès à des informations
qui aideront à gérer le risque des essais en vol aux niveaux et selon les normes actuellement attendues.
STO-AG-300-V32 v
Acknowledgements
In addition to the Case Study Authors, both the Lead Author and the FT3 AGARDOgraph Champion wish
to recognise the support offered through the SCI Panel Staff, Ms Carlotta Rossi, and the STO Editorial Team
who made our lives a whole lot easier as this document came to fruition.
vi STO-AG-300-V32
Foreword
It is self-evident that all Flight Test Activities should be conducted safely and without hazarding the aircraft, the
crew or third parties and property on the ground. Far more so than with any other system, airborne test incidents
have the ability to rapidly escalate in seriousness, leading to the potential loss of aircraft and crew.
The history of flight testing from the days of mythology and antiquity to the present day is littered with accidents
many of which could have been avoided if effective safety management practices had been deployed.
Would Icarus have come to such an untimely end if he hadn’t operated outside of the cleared flight envelope,
flying too close to the sun, and melting the wax that secured the feathers to his wings? Might King Bladud not
have plummeted to earth when his wings of chicken feathers failed to support him in his attempt to fly leaping
from The Temple of Apollo in London in 852 BC? It being a well-known fact that chickens cannot fly, a clear
design error.
But for all these failures there have also been successes. The Montgolfier Brothers conducted early unmanned
flights and then flights with animals before themselves ascending in their balloon in 1783. The Wright Brothers
through theoretical study and wind tunnel tests progressed from kites to gliders and eventually manned powered
flight with The Flyer in 1903. Each had adopted a structured approach to design, development and flight testing
that allowed them to manage risk and achieve success.
Since that historic day at Kitty Hawk in 1903 the art and science of flight testing has progressed to become the
highly complex and structured activity we see today. Accidents and incidents have happily become far less
common but sadly they do still occur and more often than is acceptable. To counter this ever-present threat the
International Flight Test Community has established approaches that identify and then mitigate flight test risk.
Some of these are described and assessed in the following pages. These have been offered freely by Organisations
that wish to contribute to the never ending search for the Holy Grail of Safe Flight Testing.
Through this AGARDograph it is hoped that a small contribution may be made to enhancing the safety of future
flight testing across the NATO Nations and the International Flight Test Community. If it successfully guides one
programme away from a potential mishap then all of the effort will have been worthwhile.
STO-AG-300-V32 vii
Preface
Some years ago, the members of the NATO Science and Technology Organisation Flight Test Technical Team
(FT3) were discussing approaches to managing Flight Test Safety and Risk and the systems deployed by the
represented organisations. There was an opinion that the members’ organisations were performing this task in very
similar ways but this could not be substantiated. Through these discussions there was a growing view that there
would be value in formally capturing these approaches and to make these findings available to the wider flight test
community in order to enhance Flight Test Safety and Risk Management. This led to the formation of Task Group
SCI-236, sponsored by the Systems Concepts and Integration Panel, whose mission was to collect the evidence and
to record the findings.
This AGARDograph is the result. It assesses the current state of practice in the field of Flight Test Safety and Risk
Management and provides a number of observations on the approaches taken.
viii STO-AG-300-V32
Biographical Sketches
JEREMY NEWSOME
Mr. Jeremy Newsome is Head of Flight Test Engineering at BAE Systems, Warton, United Kingdom.
He has 28 years of experience in development Flight Testing across a range of manned and unmanned
aircraft, including Tornado, Nimrod, Harrier, Hawk, HERTI, Mantis and Taranis. He holds a Bachelor
of Engineering Degree in Aeronautical Engineering and is a Member of the Royal Aeronautical Society
Flight Test Group. He has led flight trials teams on Tornado, Harrier and Unmanned Air Systems in both
the United Kingdom and at a number of international trials locations. He has been BAE Systems Head of
Flight Test Engineering since 2015 and is a Member of the NATO STO Flight Test Technical Team.
DENNIS MORLEY
Mr. Dennis MORLEY is a Flight Test Engineer with some 45 years of experience in the field. He holds
an Honours Degree in Mechanical Engineering, is a UK Chartered Engineer and a Member of the Royal
Aeronautical Society. He has worked for BAE Systems and its predecessor companies at a number of sites
and on over 30 aircraft programmes. He has led the Flight Test Teams on the Buccaneer, UK Phantom,
Tornado, Hawk, Harrier and UAVs and has also been actively involved with Typhoon, Nimrod MRA4 and
F35 Lightning II. He retired in 2015 as Chief Flight Test Engineer BAE Systems Military Aircraft. He is
a past Member of the NATO STO Flight Test Technical Team, the Royal Aeronautical Society Flight Test
Group and the Board of the European Chapter of the Society of Flight Test Engineers. In retirement he is
an Aerospace Consultant specialising in Test and Evaluation.
DEAN MOORE
Mr. Dean MOORE is the Flight Test Engineer (FTE) Lead for Boeing Test & Evaluation UK. Dean joined
Boeing in September 2013 and led the development of the Contractor Flying Organization approvals with
the UK regulator for Chinook and C-17. Prior to joining Boeing, Dean worked for QinetiQ at MoD
Boscombe Down with his last post being the Principal FTE Tutor at the Empire Test Pilots’ School (ETPS),
2009 to 2013. He previously worked on a range of fixed & rotary wing programs including Puma Mk2,
CONDOR II Technology Demonstration Programme, Typhoon, Chinook and C-130J. Dean holds an MSc
in Flight Dynamics (Cranfield University) & BEng in Aerospace Engineering (University of Hertfordshire)
and graduated from No.25 Flight Test Engineers Course at ETPS. He is also a Boeing Associate Technical
Fellow, Fellow of the Royal Aeronautical Society and member of the NATO Flight Test Technical Team.
PERRY COMEAU
As a Royal Canadian Air Force (RCAF) helicopter pilot, Lt Col Perry Comeau flew operationally
internationally as well as operational test and evaluation on the CH124 SeaKing. Completed test pilot
training at United Stated Naval Test Pilot School and was a test pilot on many projects on the CH124
SeaKing, CH146 Griffon and CH149 Cormorant. He participated in the development and acceptance of
the CH148 Cyclone aircraft and simulator, including as the Combined Test Force Lead of several test pilots
and system evaluators. As Lieutenant Colonel, became the Senior Test Pilot for AETE overseeing all
developmental and engineering flight test for all fixed, rotary and UAVs in the Canadian Armed Forces.
After 28 years, he retired as a Lieutenant Colonel and joined the National Research Council’s Flight
Research Laboratory as a Research Engineering Test Pilot. He has a Bachelor of Science in Physics, and
a Bachelor in Computer Engineering.
STO-AG-300-V32 ix
BRADFORD A. NEAL
Mr. Bradford A. Neal is currently the Chief Engineer at NASA’s Armstrong Flight Research Center
providing independent technical guidance and oversight to flight test programs and projects. Mr. Neal has
more than 35 years of experience in flight research, test and operations at NASA. He leads the AFRC
airworthiness and flight safety review process, reviewing and approving flight research and test activities
such as X-59 and X-57. Mr. Neal began his career as an operations engineer, performing integration and
mission operations for projects such as the SR-71 Linear Aerospike Experiment, X-31, X-43 and
the Stratospheric Observatory for Infrared Astronomy (SOFIA).
PATRICK C. STOLIKER
Mr. Patrick C. Stoliker is currently the Deputy Center Director at the NASA Armstrong Flight Research
Center and maintains daily involvement in the flight activities. Mr. Stoliker has more than 35 years’
experience in flight test and flight research between his time in industry and at NASA. Mr. Stoliker began his
career as a flight controls engineer at Northrop and came to NASA to work the X-31 program. He has served
as the chairman for multiple flight readiness reviews including the X-43, X-38, and Global Observer aircraft.
ARUN KARWAL
Mr. Arun Karwal is a Research Test Pilot at the Dutch Royal Netherlands Aerospace Centre NLR. He holds
a Master of Science degree from Delft University of Technology Aerospace Engineering and is a graduate
of the Royal Netherlands Air Force Officers Training School and KLM Flight Academy. He has 25 years of
experience in research flight test at NLR as test pilot flying the NLR laboratory aircraft on a wide range
of research missions. He combines his work at NLR with a career as airline pilot where he presently is
a Commander on the Boeing 777 and 787 with a major airline. Arun has logged over 13,000 flight hours,
holds an EASA Cat 1 Flight Test Rating and is a Member of the Society of Experimental Test Pilots.
CHRISTIAN BUCK
Mr, Christian Buck is a Senior Advisor within Flight Test Engineering at Flight Test and Verification,
Business Unit Aeronautics, SAAB AB, Sweden. This includes governing processes, methods and tools for
verification and validation at Saab Aeronautics. Mr. Buck has 30 years of experience in development and
systems testing of advanced military combat aircraft systems. Most of the engineering and test leader work
has been on the Gripen fighter system, from envelope expansion of the early digital fly by wire control
system, through weapons integration testing, to an iterative development process based on continues
integration, model validation and flight test.
CHRISTIAN RICE
Mr. Christian Rice, Chief Test Engineer Air Test and Evaluation Squadron Two One. Mr. Rice is a 1987
graduate of U.S. Naval Test Pilot School, holds a B.S. in Aerospace and Ocean Engineering from Virginia
Tech and a M.S. in Aviation Systems from the University of Tennessee. He has over 35 years of experience
in developmental flight test in rotary wing and tilt rotor manned and unmanned aircraft, directly contributing
to all current and developmental USN, USMC, and USCG rotorcraft. Mr. Rice has been the Chief Test
Engineer since 2002.
JONATHAN STEVENSON
Mr. Jonathan Stevenson is the Operations Officer, Naval Test Wing Atlantic, Patuxent River, Maryland.
Mr. Stevenson has 30 years of aviation experience with over 4000 flight hours in 74 different aircraft,
including more than 2000 hours in the F 14, 1500 hours in the F/A 18 and 600 arrested landings.
x STO-AG-300-V32
Mr. Stevenson holds a Bachelor of Science Degree in Mechanical Engineering and is a 1997 graduate of the
U.S. Air Force Test Pilot School. He has16 years of flight test experience as an experimental test pilot in
the F-14, F/A-18 and EA-18G and is a flight instructor with the U.S. Naval Test Pilot School. Mr. Stevenson
provides oversight of all flight test and test pilot training at Patuxent River orchestrating complex operations
on 130 unique fixed- and rotary-wing, manned and unmanned aircraft.
JOSEPH A. MORTENSEN
Mr. Joseph A. Mortensen is the Director, Research, Development, Test, and Evaluation Strategy and
Management for Naval Air Systems Command. He has 22 years in aircraft and weapon system development,
test and evaluation. Mr. Mortensen is a retired Marine Corps Colonel with 27 years of aviation experience
that included 12 years as a test pilot and served as Commander, Naval Test Wing Atlantic and Commanding
Officer/Chief Test Pilot, Air Test and Evaluation Squadron 23. Mr. Mortensen holds an Interdisciplinary
Bachelor of Science Degree in Engineering and Master of Science Degree in Aviation Systems. He is a 1992
graduate of the U.S. Naval Test Pilot School and completed a fellowship with the Asia Pacific Center for
Security Studies. Mr. Mortensen is an Associate Fellow of the Society of Experimental Test Pilots.
STO-AG-300-V32 xi
Table of Contents
Page
xii STO-AG-300-V32
2.2 Boeing Test and Evaluation UK 2-13
2.2.1 Nature of Flight Test Activity 2-13
2.2.2 Background 2-13
2.2.2.1 Boeing Commercial Airplanes 2-13
2.2.2.2 Military Aircraft 2-13
2.2.3 Regulatory Framework 2-13
2.2.3.1 Quality Management System 2-14
2.2.3.2 Product Test Life Cycle 2-14
2.2.3.3 BT&E Safety Policy 2-15
2.2.3.4 Safety Management System 2-15
2.2.4 Organisation and People 2-16
2.2.4.1 Engineering 2-16
2.2.4.2 Test Program Managers (TPM) 2-16
2.2.4.3 Test Value Stream 2-16
2.2.4.4 Flight Operations 2-17
2.2.5 Process and Procedure 2-17
2.2.5.1 Test Planning 2-17
2.2.5.2 Test Hazard Analysis 2-17
2.2.5.3 Risk Management Process 2-18
2.2.5.4 Risk Matrix 2-18
2.2.5.5 Flight Test Mission Assurance Readiness 2-19
Review (FTMARR)
2.2.5.6 Test Readiness Review 2-19
2.2.5.7 Day to Day Supervision 2-19
2.2.6 Facilities and Tools 2-19
2.2.6.1 Test Planning, Execution and Reporting 2-19
Tool (TPERT)
2.2.6.2 Briefing Checklists 2-19
2.2.6.3 Operational Risk Management 2-19
2.2.6.4 Risk Register 2-20
2.2.6.5 Bow Tie Diagrams 2-20
2.2.6.6 Occurrence Reporting Systems 2-20
2.2.6.7 Communications 2-21
2.2.6.8 Air Safety Management System Maturity 2-21
2.2.7 Discussion and Observations 2-21
2.2.7.1 Practices and Principles 2-21
2.2.7.2 Positive Air Safety Culture 2-21
2.2.7.3 Understanding Terminology 2-21
2.2.8 Conclusions and Recommendations 2-22
2.2.9 Appendix 2-22
STO-AG-300-V32 xiii
2.3.5 Process and Procedure 2-25
2.3.5.1 Planning 2-25
2.3.5.2 Flight Test Safety Review 2-27
2.3.5.3 Test Plan Amendments 2-32
2.3.6 Facilities and Tools 2-32
2.3.7 Discussion and Observations 2-32
2.3.8 Conclusions and Recommendations 2-32
2.3.9 Appendix 2-32
xiv STO-AG-300-V32
2.5.8 Conclusions and Recommendations 2-49
2.5.9 Appendix 2-50
2.7 Turkish Air Force Flight Test Center 401ST Test Squadron 2-58
2.7.1 Nature of Flight Test Activity 2-58
2.7.2 Background 2-58
2.7.3 Regulatory Framework 2-59
2.7.3.1 External 2-59
2.7.3.2 Internal 2-59
2.7.4 Organisation and People 2-61
2.7.4.1 Planning and Training Division 2-61
2.7.4.2 Flight Test Engineering Command 2-61
2.7.4.3 Test Flight Command 2-61
2.7.4.4 Test Support Command 2-61
2.7.5 Process and Procedure 2-62
2.7.5.1 Common Procedure 2-62
2.7.5.2 Flight Test Process 2-62
2.7.5.3 Flight Test Safety Review Process 2-63
2.7.5.4 Hazards/Risk Identification 2-64
2.7.5.5 Risk Assessment 2-64
2.7.5.6 Risk Monitoring 2-66
2.7.5.7 Risk Handling 2-66
STO-AG-300-V32 xv
2.7.5.8 Implementation of Risk Handling 2-67
2.7.5.9 Risk Review 2-67
2.7.6 Facilities and Tools 2-67
2.7.7 Discussion and Observations 2-67
2.7.8 Conclusion and Recommendations 2-67
2.7.9 Appendix 2-67
xvi STO-AG-300-V32
3.3 Process 3-6
3.3.1 Overview 3-6
3.3.2 Safety is Fundamental 3-6
3.3.3 Common Process 3-6
3.3.4 Generic Flight Test Process 3-6
3.3.5 Flight Test Safety Process 3-7
3.3.5.1 Long Term Test Planning 3-7
3.3.5.2 Detailed Test Planning 3-7
3.3.5.3 Execution 3-8
3.3.5.4 Analyse 3-9
3.3.5.5 Report 3-9
3.3.5.6 Continuous Process 3-9
3.4 Facilities and Tools 3-9
3.4.1 Overview 3-9
3.4.2 Common Facilities and Tools 3-9
STO-AG-300-V32 xvii
List of Figures
Figure Page
Section 2.7 Turkish Air Force Flight Test Center 401st Test Squadron
Figure 2-7.1 The Organisational Structure of TUTEK – Flight Test 2-60
Technical Board
Figure 2-7.2 The Organisational Structure of TUEK – Flight Test Safety 2-60
Review
Figure 2-7.3 Flight Test Safety Management Chart 2-63
xviii STO-AG-300-V32
List of Tables
Table Page
Section 2.7 Turkish Air Force Flight Test Center 401st Test Squadron
Table 2-7.1 Likelihood Criteria ‒ Risk Occurrence Probability, Risk 2-64
Severity and the Risk Matrix
STO-AG-300-V32 xix
List of Acronyms and Nomenclature
AA Airworthiness Authority
ADB Aircraft Discrepancy Books
AETE Aerospace Engineering Test Establishment
AGARD Aerospace Research and Development
AIA Aircraft Investigative Authority
AFRC Armstrong Research Flight Center
AFSRB Airworthiness and Flight Safety Review Board
ALARP As Low As Reasonably Practicable
AM(MF) Accountable Manager Military Flying
APM(T&E) Assistant Program Manager for T&E
AoA Angle of Attack
AOHL Airfield Operating Hazard Logs
AOSS Angle of Side Slip
ARB Airworthiness Review Board
ASMC Air Supply and Maintenance Centre
ASMP Air System Management Plan
ATP Advanced Turbo Prop
AvP Aviation Publication
AVT Applied Vehicle Technology Panel
xx STO-AG-300-V32
EASA European Union Aviation Safety Agency
EDT External Directed Team
EFP Experimental Flight Permit
EHS Environment Health and Safety
EMC Electro Magnetic Compatibility
ERB Executive Review Board
ET&T Engineering Test and Technology
STO-AG-300-V32 xxi
LOI Letter of Instruction
LTE Lead Test Engineer
OA Operational Airworthiness
OAA Operational Airworthiness Authority
OT&E Operational Test and Evaluation
xxii STO-AG-300-V32
SOFT Safety of Flight Test
SOT Safety of Test
SQEP Suitably Qualified and Experienced Personnel
SRB Safety Review Board
STE Senior Test Engineer
STO Science and Technology Organization
STOVL Short Take Off Vertical Landing
STP Senior Test Pilot
SUAS Small Unmanned Air System
SWE Sweden
UA Unacceptable Risk
UAS Unmanned Air System
UAV Unmanned Aerial Vehicle
UCAS Unmanned Combat Air System
UK United Kingdom
US United States
USA United States of America
USMC United States Marine Corps
USN United States Navy
USNTPS US Naval Test Pilot School
USO Unit Safety Officer
STO-AG-300-V32 xxiii
Author Contact Information
Name: Mr. Jeremy NEWSOME (Flight Test Technical Team Champion
and Case Study Author)
Mailing Address: Principal Flight Test Engineer
Flight Test ‒ W27K
BAE Systems
Warton Aerodrome
Warton
Preston
Lancashire
PR4 1AX
UNITED KINGDOM
Email: jeremy.newsome@baesystems.com
xxiv STO-AG-300-V32
Name: Mr. Bradford A. NEAL (Case Study Author)
Mailing Address: NASA Armstrong Flight Research Centre (AFRC)
NASA AFRC
PO Box 273
Edwards
California 93523
UNITED STATES
Email: bradford.a.neal@nasa.gov
STO-AG-300-V32 xxv
Name: Mr. Christian B. RICE (Case Study Author)
Mailing Address: Naval Systems Air Command
401st Test Squadron
Chief Test Pilot
HX-21
22755 Saufley Rd, Hgr 111
Patuxent River
MD 20670
UNITED STATES
Email: christian.rice@navy.mil
xxvi STO-AG-300-V32
Chapter 1 – INTRODUCTION
1.1 BACKGROUND
It is self-evident that all Flight Test Activities should be conducted safely and without hazarding the aircraft,
the crew, or third parties and property on the ground. Far more so than with any other system airborne incidents
have the ability to rapidly escalate in seriousness, leading to the potential loss of aircraft and crew.
To counter this ever-present threat the International Flight Test Community has established approaches
that identify and then mitigate flight test risk.
This AGARDograph shares practice in this critical area of Flight Testing. Through its application both
the NATO and wider Flight Test Community will have access to information that will help to manage flight
test risk to the levels and standards that are expected today.
STO-AG-300-V32 1-1
INTRODUCTION
The sum total of these is the Organisation’s Safety Management System described in a mechanistic way and
while this is highly relevant and important it comes to nothing if the organisation lacks an effective Safety
Culture.
Culture is characterised by the way things are done in an organisation and Safety Culture is a measure of how
an organisation addresses safety related issues. A good Safety Culture is all pervasive and places safety in
a paramount position at all levels of the organisation. It is reflected in organisational attitudes and influences
all major decisions. Within Flight Test, where catastrophic outcomes are always a possibility, it is vital that
the organisation nurtures a Safety Culture which at all times challenges situations where risk can arise and
addresses them. An effective Safety Culture is founded on a number of attributes:
• Leadership – Managers proactively demonstrate that safety is a core value of the organisation and are
actively and visibly involved in safety management. They establish and live the values of the Safety
Policy.
• Staff Involvement and Attitude – Staff take a personal responsibility for safety and understand
that prevention is better than cure. They both understand and ‘buy into’ the policy and operate
in accordance with approved procedures and practices.
• Commitment – Everyone is empowered to take decisions that affect safety without any fear of censure.
A just culture where anyone can exercise the ‘NO’ vote without fearing reprisal.
• Competency – Is developed at all levels. It is prized and rewarded.
• Policies, Processes and Systems – Are deployed that naturally support the flight test activity and
its safe conduct. Not just in terms of delivery but also in terms of monitoring and reporting safety
performance and initiating appropriate improvement actions.
The very fact that a number of Flight Test Organisations have contributed to this work and are willing to share
practice is testimony to their seriousness in addressing safety issues not only within their own business but also
in the wider Flight Test Community. A review of the individual papers reveals organisations that strive to excel
in pursuing safe flight testing with examples of how they react to the challenges of developing their own
Safety Culture.
The papers perhaps concentrate more on the harder issues of Competency, Organisations and Processes and
are a little light on the softer issues associated with establishing the required Safety Culture of Leadership,
Staff Involvement, Attitude and Commitment but the critically important underlying theme of driving safety
is clear.
An analysis of Flight Test Accidents often reveals a breakdown in the Safety Culture under programme cost
and time pressures especially where management fail to live the values. We should all challenge ourselves
with the questions of how would we behave when such pressures arise? and how would our Safety Culture
1-2 STO-AG-300-V32
INTRODUCTION
withstand the stress test? Many otherwise fine flight test organisations with excellent safety management
systems have failed when these pressures arise. All of the systems can be in place and operated by competent
trained staff but will come to naught if the culture breaks down under pressure.
There is no silver bullet here. It is about how the organisation lives its values. Processes and People are
powerful enablers but the Safety Culture is the glue that binds it all together. Management are key to
developing this and must regard it as much a part of their role as programme delivery.
Whilst the following sections provide information on what others have done to develop and deploy Flight Test
Safety Management Systems and are useful guidance on practice the development of the Safety Culture,
very much belongs to you, the reader, and your organisation. There is no prescription which gives you an
instant answer but without a healthy Safety Culture your search for safety is far less likely to succeed.
STO-AG-300-V32 1-3
INTRODUCTION
1-4 STO-AG-300-V32
Chapter 2 – CASE STUDIES
TURKEY
Christian B. Rice, Fred P. Henderson, Jonathan R. Stevenson
and Joseph A. Mortensen
Naval Air Systems Command (NAVAIR)
UNITED STATES
In support of these it undertakes flight testing using State of the Art Facilities and Tools across the whole of
the product life cycle. This activity encompasses Experimental, Developmental and Production Flight Test.
The Organisation has Flight Test Capabilities covering:
1) Airframe – Handling and Performance, Stability and Control, High Angle of Attack, Spinning, Flutter
Vibration and Loads;
2) Systems Integration – General Systems, Avionic Systems, Weapon Systems; and
3) Weapons Test – Carriage, Jettison, Releases and Firings.
STO-AG-300-V32 2-1
CASE STUDIES
2.1.2 Background
The BAE Systems Flight Test Centre at Warton has a proud heritage that can be traced back to the earliest
days of aviation in the United Kingdom. It’s Corporate Knowledge and Experience originates from a large
number of predecessor Flight Test organisations that have been at the forefront of aeronautical development
for the last 100 years.
Following industry consolidation, it is now the only Flight Test Centre operating within the Company but it
has absorbed the experiences and skills of its forebears. In recent years these have most notably been drawn
from the fast jet and VSTOL experiences of the Dunsfold Flight Test Centre – Hunter, Harrier, Sea Harrier
and Hawk and the large aircraft experiences of the Woodford Flight Test Centre – BAe 146, Jetstream,
ATP and Nimrod.
The Warton Unit itself has a history that begins with the Canberra Bomber of 1949, this continued with the
Lightning Fighter of the 1950s and International Co-operative projects commencing with the Anglo-French
Jaguar in 1968. The Tri National Tornado followed in 1974. A series of Fly By Wire (FBW) Demonstrators,
the FBW Jaguar and Experimental Aircraft Programme, led the way to the Typhoon of 1994. The Hawk and
Harrier were also brought into the site’s Flight Test portfolio following company reorganisation in the late
1980s and early 2000s. In recent years the Nimrod MRA4 Maritime Patrol Aircraft and a series of UAS
Demonstrators have also been tested as has the F35 Lightning II where the organisation has provided domain
expertise to the STOVL testing, supporting the US contractor, Lockheed Martin.
Current Warton activity centres on Typhoon, Tornado and Hawk development for the RAF and International
Customers, the Production of the Typhoon and Hawk and continuing experimental UAS work.
The Organisation conducts some of its trials on a joint basis with customers, partners and the
UK Government’s Official Test Centres. Where these trials take place under BAE Systems governance these
Test Partner’s operate within the Flight Test Engineering procedure set.
In support this work a common Flight Test Safety Management System has been developed and deployed.
2.1.3.1 External
BAE Systems Air and its forebears have historically had the UK Armed Forces as their principal customer.
The Quality Management Systems deployed have therefore primarily been designed to satisfy the requirements
of the UK Ministry of Defence (MoD) and latterly the Military Aviation Authority (MAA). These have then
been reused to support partnership and export activities. For many years this meant that within the Flight Test
domain the controlling instruments were Def Stan 05-123 for Flight Testing and AvP 67 for Test Flying and
the origins of the deployed Flight Test Safety Management System and Quality Management System lie with
these documents.
Following the loss of Nimrod Aircraft XV230 and the subsequent Haddon-Cave Nimrod Review changes were
instituted in the UK MoD that created a new MAA to control military flight safety. Def Stan 05-123 and
AvP 67 were withdrawn and replaced by new, Regulatory Article documents. Many aspects of these are similar
to those replaced. The primary new Document Sets are contained in MAA Regulatory Publications (MRP),
Series 1000 is General, Series 200 covers Flying and incorporates AvP67 and Series 5000 covers Design and
Modification Engineering and incorporates Def Stan 05-123.
It is also a requirement of UK MoD that Approved Suppliers have a Quality Management System that satisfies
the provisions of ISO 9000.The deployed Flight Test Safety Management System satisfies the requirements
of the current MAA Regulations and ISO 9000.
2-2 STO-AG-300-V32
CASE STUDIES
2.1.3.2 Internal
Within BAE Systems safety at all levels is of the highest priority. To support this, the Company has
a mandatory Safety Policy that describes how safety will be managed across all of the Company’s activities
and who is accountable. A specific element considers activities that are considered especially hazardous. Flight
Testing and Flying are two of the identified elements. The Head of Flight Test Engineering (HoFTE) is named
as the accountable person for Flight Test and the Director of Flight Operations (DFO) is accountable for Flying.
Test Flying is addressed as a sub-element of Flying.
2.1.4.1 Engineering
Within BAE Systems Air delivery is the responsibility of project based teams and an aircraft type is typically
a project focus. Within the UK System the Design Authority for an aircraft type is the Manufacturer who is
given this responsibility by the MAA following assessment of organisation’s ability to safely and effectively
manage and develop the design. This leads to certain members of the Design Staff being named and approved
by the UK MAA to fulfil certain duties in relation to the discharge of the Design Authority.
For Flight Test Engineering the accountable person is the Flight Test Manager (FTM) who is named on the
Company’s Design Approval Certificate. The FTM has a team working for him who deliver the engineering
aspects of the Flight Test Programme.
STO-AG-300-V32 2-3
CASE STUDIES
Each Flight Test Manager is responsible for establishing a Team Construct that is best suited to satisfying the
needs of their project. There is no fixed organisational model. There is however commonality of roles and role
responsibilities across the teams and to the assessment and approval of individuals to fulfil their duties.
Each Team consists of two principal skill streams – Operations and Specialists.
Operations staff are responsible for managing a particular test aircraft, planning the details of a sortie in
conjunction with the test aircrew and specialists, defining the required aircraft standard to the Hangar Team,
Preparing Test Cards, leading the Briefing and Debriefing of the crew and leading the Telemetry Team when
used.
Specialists are expert in a particular test domain, e.g., Handling, Weapons, Sensors, etc. and work closely with
Design Departments to establish the detail requirements for a particular flight trial. They capture the required
tests in the Flight Test Plan and act as the lead for the Flight Test Engineering aspects of both Design Review
and Trials Safety Review. They support in the telemetry room and at briefing and debriefing. Post-flight they
are responsible for analysis and reporting.
Against a mandatory procedure, that is part of the common process set, the Flight Test Manager defines what
roles and document approvals can be discharged by all members of his staff. Each Flight Test Engineer has a
personal SQEP Record, which is agreed and approved by the HoFTE and the DFO/AM(MF). This is to satisfy
the UK MAA Regulatory requirement that personnel involved in the planning and execution of Test and
Evaluation (T&E) activities are SQEP against specific T&E categories (in accordance with the MAA MRP).
This delivery accountability is paralleled by a Governance Stream led by an organisation separated from the
delivery streams that is responsible for the definition of the controlling standards that shall be used by the
project and the monitoring and review of the projects to ensure compliance. Within BAE Systems Air the
accountable person for Flight Test is the Head of Flight Test Engineering, supported by the Flight Test
Managers.
The accountable person is the Director of Flight Operations (DFO) who reports to the Engineering Director.
He is identified as the responsible person not only by the Company but also as the AM(MF) by the MAA.
DFO is responsible for the Governance of Flying.
He controls the Flying Operations Manual (inclusive of associated Flying Orders) for BAE Systems Air which
describe the processes to be followed during flying operations. These orders also identify those individuals
approved to perform various types of flight test.
DFO is supported by two Chief Test Pilots. One for Combat Aircraft and a second for Strategic Aircraft and
UAS. They in turn are supported by a team of Test Pilots, Test Aircrew and Flight Test Observers who focus
on particular aircraft types.
2-4 STO-AG-300-V32
CASE STUDIES
different methods of work. In Flight Test Engineering this is not the case. It has been long established that
Flight Test Engineering will operate common process, facilities and tools across all projects. These are owned
and governed by the HoFTE on behalf of the projects and managed between the projects and HoFTE. This
means that irrespective of project Flight Test Engineers only need to operate one method of working. This is
judged to enhance safety and efficiency and also minimises retraining on transfer between Project Teams.
Initial Planning: Every major Flight Test Programme commences with Initial Planning. Its aim is to support
Business Winning at understood levels of commercial and technical risk and the positioning of the Team to do
the detailed work. It is critically important to establishing a Flight Test Programme that can be delivered safely
and effectively. The principal outcomes of Long-Term Planning are twofold:
• The preparation of an Outline Test Plan for the flight trials, that has sufficient detail to support task
estimating in pursuance of a Contract Bid.
• An assessment of the commercial and technical risk inherent in the Outline Test Plan, which will allow
initial risk contingency and risk management measures to be established.
Detailed Planning: Provided the Initial Planning is effective and bid is successful the task will proceed to the
Detail Planning Phase, inclusive of Trials Definition and Aircraft Preparation.
• This is the Phase where the detailed planning of the activities necessary to deliver the trials is
undertaken. It is also the phase where those activities that must be completed ahead of Test Conduct,
e.g., Flight Test Instrumentation design and manufacture, Data Processing system design, detailed
Flight Test Plan preparation, Risk Assessment and Management, etc. are performed.
• It should be recognised that it is not uncommon for this Phase to proceed at commercial risk ahead of
Contract Award in order to mitigate overall programme risk. For a major programme this activity
commences many years before First Flight.
• Detailed Flight Test Risk Review and Safety Management are performed during this Phase and the
outcome is a fully risk mitigated Flight Test Plan approved and authorised for use.
• Within BAE Systems the approval and authorisation Levels for the Flight Test Plan are dictated by
the Trial Risk Category. High Risk Trials are authorised by DFO/AM(MF) and HoFTE. Lower
categorizations are addressed at lower levels in the Organisations.
• The final activity is Test Readiness Review where all aspects of trial preparedness are considered
ahead of the trial. Clearly Test Safety is a major mandatory consideration. The outcome of this Phase
is Approval to Fly.
Trials Execution: The Execution Phase is that part of the Flight Test Programme where the Test Flying is
conducted. For a Major Programme it will involve Envelope Expansion and High Risk Flight Trials.
• The key controlling document is the Test Card Set for a given sortie. Within BAE Systems these are
produced by the Flight Test Operations Team in conjunction with the Test Aircrew.
• Each sortie is subject to formal brief by the relevant approved person with all involved parties.
A Briefing Witness is nominated for each brief who is responsible for ensuring the brief is fit for
purpose and that all sortie elements have been adequately covered. At the conclusion of the brief he
presents a completed checklist to the aircraft captain to be signed off to declare the crew are satisfied
with the Sortie Brief. The Briefing Witness has the authority to suspend a sortie which is judged to be
inadequately prepared.
STO-AG-300-V32 2-5
CASE STUDIES
• Where sorties are supported by Telemetry the Team are present at the brief.
• Formal debrief follows each Sortie. Any safety-related issues receive priority assessment and where
necessary are escalated.
Analysis and Reporting: On completion of the flying tasks the Flight Test team will move into this final
phase of the process.
• Analysis is that part of the Flight Test Programme where test data is processed and assessed both to
enable trials continuation and to support reporting.
• In the Flight Test Domain this will involve not only processing data from the test aircraft but also
other sources especially those which provide independent “Truth” Data.
• Immediate post-flight analysis focuses on test progression and safety-related issues and arisings.
• Reporting covers all of those activities where information and data are released from the Flight Test
Organisation to others. It may be released internally within the Project or externally to Customers.
• Identification and capture of safety-related issues and their escalation for resolution is a priority of
short term reporting.
• Where residual safety issues remain, the Final Report must clearly identify the scale and impact of the
issue and make recommendations for mitigating and controlling the risk.
In the early 2000s it was identified that the company was not following best practice in this area and a specific
process was introduced following research into the approaches taken by other organisations. Since that time,
the basis of the system implemented remains, though it has evolved to encompass changes in MAA Regulatory
requirements, embracing the top level concept of a Test and Evaluation (T&E) ‘Approval Board’.
BAE Systems adopted an approach where trials are pre-assessed by a Risk Categorisation Meeting and given
an indicative Risk Categorisation. This dictates different levels of Risk Review and Risk Management
dependent on the allocated category. This has the advantage of allocating lower categorizations to trials of a
routine low risk nature, e.g., Radio Trials, thereby avoiding the burden of full blooded High Risk Trials Review
in all cases.
The outcome of the board will be the Approval from AM(MF) to proceed to trials execution. This Approval
does not require AM(MF) signature, but instead is granted when the T&E Approval Board process has been
followed and the Flight Test Programme (FTP) is signed by all required signatories.
The first stage of the T&E Approval Board, which for ‘within scope’ T&E activities will occur at an early
stage of the planning process, is identification of the MAA T&E Category (in accordance with Regulatory
Article RA2370) and initial risk categorization of the trial at the Trials Risk Categorization meeting.
2-6 STO-AG-300-V32
CASE STUDIES
At the Trials Risk Categorization meeting a brief outline of each planned trial will be given by the relevant
Flight Test Manager or their nominee. Trials will be categorized and identified against Intolerable, High,
Medium and Low risk categories using the experience of the assessors and with reference to the Risk Matrix
(see Table 2-1.1 in Section 2.1.5.4.3).
A Lead FTE and a Flight Operations Project Pilot (both of whom will act as Trial Supervisors) will be
appointed based on both the agreed Risk and T&E Category, and their competency for conduct of the trial,
i.e., they must each be SQEP in their respective role.
A pre-defined Trials Log Sheet, is employed both as a prompt to ensure all major risk areas are discussed and
also as a record of the categorizations made, the reasons for the decision and any guidance offered to the
associated Safety Review, Trials Readiness Review and FTP authors for Low Risk Trials.
The subsequent stages of the T&E Approval Board depend on the initial risk categorization defined. The stages
are:
• Concept Review (for trials initially Categorized as High Risk)
• Safety Review (for trials initially Categorized as High and Medium Risk)
• Agreeing and mitigating the risks in the FTP to ALARP and Tolerable
• Approval and Authorisation of the FTP
A flowchart showing the T&E Approval Board process is given in the Appendix (Figure 2-1.1).
2.1.5.3.2 Participation
The assessment team will be senior staff from Flight Test Engineering, Flight Operations, Continuing
Airworthiness Management Organisation and Type Airworthiness.
At the outset of each collaborative trial which involves service aircrew the service will be represented primarily
via Officer Commanding 41 Test and Evaluation Squadron who will be involved in the Initial Risk
Categorisation and their involvement will continue through any Safety Review stage. The initial Risk Category
will be continuously reviewed against all MoD or other partner specific requirements and mitigation together
with company requirements; the composite solution will be applied as necessary.
The initial Risk Categorisation attributed to a trial or activity may be changed during the Safety Review Process
or the Trials Readiness Review and also during production of FTPs, if appropriate mitigations or new risks are
identified, or if trials results indicate the need. This principle of continuous review is to be encouraged but the
trial re-categorization must, as a minimum be formally endorsed by the project Flight Test Manager and
appropriate CTP or DFO, but ideally through the T&E Board.
STO-AG-300-V32 2-7
CASE STUDIES
3) High Risk
A high degree of uncertainty exists. Though remote, potential does exist for loss of manned or high
value aircraft and/or loss of life due to predicted risk, unforeseeable or imperfectly understood
actions.
Examples might be:
• High AoA / carefree handling;
• Some engine handling work;
• Concept demonstrator flying;
• First-of-type flying;
• Low level work with degraded situational awareness; or
• The deliberate investigation of major emergency situations.
4) Medium Risk
Uncertainties exist, but experience from similar trials can be used in mitigation. Potential exists for
aircraft, collateral damage or risk to life or injury; risk of catastrophic failure is considered
improbable.
Examples might be:
• Initial carriage/handling/release of unfamiliar stores/weapons;
• Flutter;
• airfield handling and take-off/landing performance definition; or
• Some low level work.
5) Low Risk
Few uncertainties exist and risk of catastrophic failure is considered incredible. Within or close to
established envelopes and with manoeuvres familiar to the test fraternity.
Examples might be:
• Routine air tests;
• Software/hardware revalidation;
• Incremental configuration expansion;
• Handling/carriage/release of comparable stores;
• Data gathering trials within the cleared flight envelope; and
• Most small Unmanned Air System (sUAS) trials.
All the above definitions are guidelines only; the risk categorisation team will be the final arbiters of the
category in all cases.
The Matrix is based on standard industry practice and similar to the 6 x 4 matrix used by QinetiQ and the
Air Warfare Centre to assess trials risk (Table 2-1.1).
2-8 STO-AG-300-V32
CASE STUDIES
Severity
Catastrophic Critical Marginal Negligible
Major damage or severe Minor damage or minor Less than minor damage/
Loss of life or aircraft.
injury. injury. injury.
Probability
Risk
Catastrophic Critical Marginal Negligible
Frequent INTOLERABLE INTOLERABLE INTOLERABLE HIGH
Probable INTOLERABLE INTOLERABLE HIGH MEDIUM
Occasional INTOLERABLE HIGH MEDIUM MEDIUM
Remote HIGH MEDIUM MEDIUM LOW
Improbable MEDIUM MEDIUM LOW LOW
Incredible LOW LOW LOW LOW
Numerical Probabilities are indicative of per event/trial and should be used as guidance to support decision
making and should not diminish the qualitative view of the subject matter experts.
Physical Severity (loss of manned or high value aircraft) and Human Severity (Risk to Life) should be
considered with equal weighting to establish a baseline risk level.
STO-AG-300-V32 2-9
CASE STUDIES
2 - 10 STO-AG-300-V32
CASE STUDIES
The output of this meeting (formal minutes) will provide discussion and agreement of the strategic
way ahead supporting the detailed trials planning, along with some direction/actions with respect to
hazard definition / understanding and potential options for mitigation.
The High Risk Safety Review will be held in similar timescales to the medium risk review i.e., when
the trials team has the relevant information and feels ready, but with sufficient time to amend the FTP
if necessary.
This High Risk Review will be Co-Chaired at HoFTE and DFO level with attendees including:
• Chief Test Pilot;
• Flight Test Manager;
• Airframe/Systems Sponsor. (or their formally appointed nominees);
• Chief Airworthiness Engineer;
• Continuing Airworthiness Management Organisation;
• Test Partners (joint trials);
• FTE and FLOPS Trial Supervisors;
• Contributing Flight Test Engineers;
• Airframe/System Advisors; and
• Independent Subject Matter Experts as appropriate.
The objectives and output from this review will be the same as for the Medium Risk Review.
Trials are ‘Approved’ once the whole Trials Approval Process (see Appendix 2-1.9) has been completed and
the FTP is Approved and Authorised at the appropriate level.
In this context Tools include: Telemetry, Process and Procedures and Practice.
This is particularly relevant to trials of a High Risk nature as we are currently performing fewer of these than
in previous years. Links with other BAE Systems Air facilities are use (i.e., with Simulation) where necessary
in order to ‘work-up’ Flight Test and Flight Ops personnel for specific trials activities.
STO-AG-300-V32 2 - 11
CASE STUDIES
Currently no structured toolsets are deployed to help manage, assess and mitigate risk. Several databases are
in place to manage the Review Process and to capture Arisings and Occurrences. These are used to monitor
overall system performance and to identify areas requiring attention.
2.1.9 Appendix
2 - 12 STO-AG-300-V32
CASE STUDIES
As the centrally-managed test and evaluation organisation of The Boeing Company, Boeing Test and
Evaluation (BT&E) verifies and validates the Company’s products and services, ensuring they operate as
designed and meet the rigorous requirements of regulatory agencies and Boeing customers. Boeing Test and
Evaluation employs approximately 6,000 employees. The organisation has lab and flight operations at more
than 60 sites in 23 states in the United States, with its largest operational centres in Seattle; Southern California;
St. Louis; Philadelphia; Mesa, Ariz.; and Patuxent River, Md. BT&E also has a number of global centres
including the UK, India and Australia. Flight operations personnel conduct an average of approximately
600 flights per month, conducting flight tests for experimental and production aircraft. Flight testing of new
airplanes can occur anywhere in the world, wherever weather conditions exist that will test the airplane to its
performance limits.
2.2.2 Background
STO-AG-300-V32 2 - 13
CASE STUDIES
fleet. It has since been expanded to include C-17 operations, Remotely Piloted Air Systems (under a civil
approval) and continues to develop to enable support to Boeing air systems operated by the UK government.
2 - 14 STO-AG-300-V32
CASE STUDIES
The BT&E safety commitment relies on adherence to the policies and procedures governing commercial and
military flight safety, ensuring safe testing and that our customers receive the best quality and most reliable
product and services.
For the UK regulatory framework an additional supplementary Air Safety Management Plan was generated to
identify the way in which this safety management system was applied to specific regulations applicable to UK
operations; such as generic risk management, occurrence reporting and Post-Crash Management
(or Pre-Mishap Planning in US terminology).
STO-AG-300-V32 2 - 15
CASE STUDIES
2 - 16 STO-AG-300-V32
CASE STUDIES
The UK Military Regulatory Publication requires all T&E activity to have an associated Risk Assessment.
STO-AG-300-V32 2 - 17
CASE STUDIES
2 - 18 STO-AG-300-V32
CASE STUDIES
STO-AG-300-V32 2 - 19
CASE STUDIES
The scores assigned to these are factored and the total is used to identify the level of authorisation required for
the flight test.
2 - 20 STO-AG-300-V32
CASE STUDIES
2.2.6.7 Communications
A key element to safe test conduct is constant positive reinforcement that all employees play a vital role in Air
Safety. This is achieved through a number of routes in BT&E including:
• Leadership webcasts;
• Safety Stand downs / commitments aligned to the Company go4zero initiative;
• Regular Flight Operations meetings globally and locally;
• Recognition of open and honest reporting of occurrences and potential hazards; and
• Regular global e-mail communications from the BT&E Flight Safety Team.
• Test Participants are reminded in pre-test briefings that they always have a ‘no vote’ through the Safety
Andon Practice (encouraging anyone to stop an activity if they see anything that might affect safety).
STO-AG-300-V32 2 - 21
CASE STUDIES
used in relation to safety management and risk mitigation. This ensures that the team have a consistent
application of risk mitigation and safety management.
2.2.9 Appendix
N/A
2 - 22 STO-AG-300-V32
CASE STUDIES
2.3.2 Background
Flight testing did not emerge in Canada until after the First World War, when Parliament created the Air Board
to regulate air navigation. In 1920 an aerodrome in Rockcliffe, Ontario, was stood up and aerospace testing
and evaluation began. Ground and aerial testing in support of Canada’s aerospace program included projects
such as oil dilution systems for engine starts in winter, fog landing equipment, fire resistant flight suits,
parachutes, and even aerial photography for mapping the country.
Canada’s flight test centre proved extremely valuable when the Second World War broke out in 1939. It carried
out testing on electronics, gunnery, navigation, and other aeronautical elements. Due to the ever-growing
demand and the skill and expertise of the small but sophisticated flight test operations, the RCAF officially
reorganized the group into the RCAF Test and Development Establishment in 1940.
Over the next several decades, various test and evaluation units were stood up, each playing an important role
in the development of Canada’s aerospace program. It wasn’t until the consolidation of these units in 1971 that
the Aerospace Engineering Test Establishment (AETE), located in Cold Lake, Alberta, became the premiere
flight test agency for the Canadian Armed Forces (CF).
STO-AG-300-V32 2 - 23
CASE STUDIES
behalf of DND and the CF. As the FTA, CO AETE is also assigned Operational Airworthiness (OA)
responsibilities for AETE or supporting aircrew where flight test activities are required beyond other OAA
delegation.
2 - 24 STO-AG-300-V32
CASE STUDIES
2.3.5.1 Planning
The planning phase involves all activities following the project tasking until the approval of the Safety Review
Board (SRB). Planning is essentially the creation of a specific sequence of events which will produce the data
required to meet the project objectives. Planning is the most critical phase of a project, as the outputs from the
planning phase will have a direct impact on subsequent phases and ultimately on the test results. Furthermore,
and just as importantly, the assessment and mitigation of the risks inherent to a test program is conducted at
the planning phase.
Figure 2-3.1 shows the steps of a typical planning phase. The paths associated with the potential need to return
to a previous step, as well as lateral inputs, are not depicted and are considered to be implied. The planning
phase can be sub-divided chronologically, from top to bottom, and between planning and reviewing activities
(TRB, ARB, SRB). Not all projects will require data planning activities or an Experimental Flight Permit
(EFP) and therefore, the applicability of some of the steps will vary by project. Additionally, projects could
have multiple test plans and associated planning and reviewing steps.
Planning discussions PO
DRP PE
PO TRB ARB PE
SRB PO
STO-AG-300-V32 2 - 25
CASE STUDIES
2 - 26 STO-AG-300-V32
CASE STUDIES
STO-AG-300-V32 2 - 27
CASE STUDIES
CO AETE as the FTA. If necessary, acceptance of “Extremely High” risks lies with the Chief of the Air Staff
(CAS) as the Airworthiness Authority (AA). It is important to distinguish the role of the FTA and CO AETE
in the safety review process. In his absence, CO AETE may assign his FTA responsibilities separately from
his CO AETE responsibilities. The individual assigned the FTA responsibilities is authorised to accept risk up
to and including “High” for authorised flight test programs. Note that the acceptance authority is determined
by the unmitigated risk level. Risk acceptance is based on three critical components, the unmitigated risk level,
the minimising procedures and the residual risk (after applying the minimising procedures).
Utilising inputs from the test team and other available sources, the PO will compile a list of all test-unique
hazards throughout the project planning phase. Once a hazard is identified, it must be analysed using the
following techniques.
Hazard Description: Any real or potential condition that can cause a mishap, degradation, reduction of safety
margins, injury, illness, death or damage to or loss of equipment or property. In describing a hazard, avoid the
tendency to name the effect or outcome, e.g., electrocution, versus the real hazard, which may be an exposed,
energised electrical panel.
Hazard Cause: Anything that could lead to the presence of the hazard (e.g., hard braking at excessive landing
weight).
Hazard Effect: The injury or damage which is being prevented (e.g., major aircraft damage or serious injury).
A methodical approach must be used in the determination of hazard effects requiring sound engineering and
operational judgement. All factors must be considered in identifying credible hazards and effects. Sometimes
multiple effects can result from a single hazard. Each hazard/effect must be listed and considered separately.
The hazard severity and probability must be assigned for each effect. If the mitigating procedures or corrective
actions are the same for some of the effects, they need only be stated for the first effect and then referenced in
the subsequent hazard/effect identified in the risk assessment.
Hazard Severity: The hazard severity category corresponding to the unmitigated effect is assigned according
to the following definitions provided Technical Airworthiness Manual (TAM):
Category A – CATASTROPHIC: Would prevent continued safe flight and landing. Could result in death
of the aircrew, normally with loss of the aircraft.
Category B – HAZARDOUS: Would reasonably be expected to result in a large reduction in safety
margins or functional capabilities, including higher aircrew workload or physical distress such that the
2 - 28 STO-AG-300-V32
CASE STUDIES
aircrew may not be relied upon to perform tasks accurately or completely. Could result in death or major
injury to aircraft occupants or major damage to an aircraft system. Could result in death or major injury to
ground personnel or the general public.
Category C – MAJOR: Would reasonably be expected to result in a moderate reduction in safety margins
or functional capabilities, including a moderate increase in aircrew workload or physical distress impairing
crew efficiency. Possible physical distress, including injuries to occupants or minor damage to an aircraft
system.
Category D – MINOR: Would not significantly reduce aircraft safety but would reasonably be expected
to result in a slight reduction in safety margins or a slight increase in aircrew workload.
Category E – NEGLIGIBLE: No effect on safety. Negligible effect on safety margins. Hazards assessed
as Category E do not need to be included in the risk assessment.
Hazard Probability: The probability of the unmitigated hazard occurring at the previously identified severity
category is assessed according to the following qualitative definitions:
Level 1 – FREQUENT: Likely to occur frequently during the test.
Level 2 – PROBABLE: Expected to occur one or more times during the test.
Level 3 – REMOTE: Unlikely, but possible to occur during the test.
Level 4 – EXTREMELY REMOTE: Not expected to occur during the test.
Level 5 – EXTREMELY IMPROBABLE: So unlikely, it may be assumed that it will never occur during
the test.
Unmitigated Risk Level: The risk level is used to determine the Risk Acceptance Authority and is assigned
according to the Airworthiness Risk Index table provided in the TAM and presented herein as Table 2-3.1.
The unmitigated risk level is based on the hazard severity category and hazard probability level.
Minimising Procedures: Detail ways to minimise the hazard (e.g., flight crew and the Flight Test Control
Room (FTCR) will monitor the brake temperature real-time and fire-fighting vehicles will be positioned at the
planned stop point).
Corrective Actions: Detail actions taken if the hazard occurs to lessen the severity of the mishap (e.g., flight
crew will follow checklist procedures, local first aid personnel and equipment will be available and the fire
department will extinguish the fire).
Residual Hazard Severity: The residual hazard severity category is assigned taking into account the
effectiveness of the minimising procedures.
Residual Hazard Probability: The residual hazard probability is assigned taking into account the
effectiveness of the minimising procedures.
Residual Risk Level: The residual risk level is assigned according to the Airworthiness Risk Index in
Table 2-3.1 and is based on the residual hazard severity category and the residual hazard probability level.
STO-AG-300-V32 2 - 29
CASE STUDIES
Note: Risks below the risk Threshold are considered to be within the Acceptable Level of Safety for all
approved missions and flying operations.
2 - 30 STO-AG-300-V32
CASE STUDIES
STO-AG-300-V32 2 - 31
CASE STUDIES
A minor amendment is an amendment which has a negligible effect on the conduct of test. A major amendment
is an amendment which has an effect on the conduct of test. The determination between a minor and a major
amendment, and whether or not the amendment has an effect on the risk assessment, is made by the test team
with concurrence from the Test Plan Approval Authority.
Requirements for instrumentation changes may require a new ARB. In particular, test plan amendments may
have ARB implications even if no instrumentation changes are made. For example, changes in manoeuvring
loads may reduce or eliminate structural margins of safety.
2.3.9 Appendix
N/A
2 - 32 STO-AG-300-V32
CASE STUDIES
2.4.2 Background
AFRC was originally established in 1946 as the High-Speed Flight Research Station. A contingent of engineers,
pilots, maintainers and administrative support personnel were deployed to Muroc Army Air Base (now Edwards
Air Force Base) from the National Advisory Committee for Aeronautics Langley Memorial Aeronautical
Laboratory to support the first supersonic research flights of the X-1 rocket-powered aircraft. The organisation
grew and expanded its capabilities over the years as it continued to support the development of, and research
associated with aerospace research vehicles. The staff currently consists of 535 civil servants and approximately
600 contractor personnel. In 2006, AFRC established a long-term lease with Los Angeles World Airports for
Building 703 located adjacent to USAF Plant 42 (Palmdale Airport). The majority of the airborne remote sensing
and airborne astronomy is accomplished from this facility. Through agreements with the US Department
Of Defense (DOD), AFRC maintains access to the airfields and airspace that allows the conduct of a full range
of aeronautical flight research and test from both locations. Dryden Aeronautical Test Range (DATR) supplies a
comprehensive set of resources for the control and monitoring of flight activities, real-time acquisition and
reduction of research data, and effective communication of information to flight and ground crews.
STO-AG-300-V32 2 - 33
CASE STUDIES
It is a requirement of NASA Headquarters that the Center maintain a Quality Management System. AFRC’s
Quality Management System is certified through the AS9100 quality system requirements.
2.4.3.2 Internal
To provide airworthiness certifications and review of flight test activities, the Center has developed a robust
airworthiness and flight safety review process. AFRC processes used to support flight test and safety activities
satisfy the AS9100 requirements.
The overarching guidance document is AFG-7900.3-001, Airworthiness and Flight Safety Review,
Independent Review, Technical Brief, and Mini-Tech Brief. The full procedure governing this and the
supporting Airworthiness and Flight Safety Review and Tech Brief and Mini Tech Brief are referred to in
Section 2.4.9.
2 - 34 STO-AG-300-V32
CASE STUDIES
STO-AG-300-V32 2 - 35
CASE STUDIES
successful execution of high risk flight tests. It assesses, communicates and accepts the residual risks inherent
in the operation and test of unique flight vehicles.
2 - 36 STO-AG-300-V32
CASE STUDIES
with particular emphasis on envelope expansions or any unexpected results, and whether or not they
are expected to present problems during future tests. These results should provide a smooth transition
to the objectives of the proposed flight plan.
Residual Risk: Experimental flight often carries higher risk than operational flight. After all appropriate
mitigations have been accomplished the residual safety and technical risks are documented, reviewed and
communicated through the system engineering and airworthiness and flight safety reviews.
Hazard Matrix: There are two Center residual risk Hazard Action Matrices (HAMs) that serve as the primary
means of communicating safety hazard management classification. The purpose of these templates is to relate
human safety hazards, loss of high-dollar value assets, and/or loss of mission in terms of the hazard’s severity
STO-AG-300-V32 2 - 37
CASE STUDIES
and its probability in order to identify the associated overall hazard risk. The HAMs identify the level of
management approval required for actual acceptance of risks (accepted risks) by the solid red and red
cross-hatched areas on the HAMs. The HAM instructions reflect the accepted, Center wording for hazard
probability and severity classifications of mishap occurrence. Projects will not change the substance of the
HAM presentation if it is planned for use as part of the Center airworthiness process without an approved
waiver. Final hazard classifications are determined after the project or program has exhausted all planned
corrective and controlling actions utilising the Hazard Mitigation.
Hazard Probability: The probability categories are derived from NPR 8715.3, NASA General Safety Program
Requirements. “Probability is the likelihood that an identified hazard will result in a mishap, based on an
assessment of such factors as location, exposure in terms of cycles or hours of operation, and affected population.”
The probability is based on the scope and duration of the risk being assessed and presented to Center
management. The probability is determined by quantification (analysis/calculated), or by qualitative means with
appropriate justification (clear rationale) for the assessment. The Hazard Probability categories are:
• Frequent – Likely to occur immediately OR expected to occur often in the life of the project/item.
Controls cannot be established to mitigate the risk.
• Probable – Probably will occur OR will occur several times in the life of a project/item. Controls
have significant limitations or uncertainties.
• Occasional – May occur OR expected to occur sometime in the life of a project/item, but multiple
occurrences are unlikely. Controls have moderate limitations or uncertainties.
• Remote – Unlikely but possible to occur OR unlikely to occur in the life of the project/item, but still
possible. Controls have minor limitations or uncertainties.
• Improbable – Improbable to occur OR occurrence theoretically possible, but such an occurrence is
far outside the operational envelope. Typically, robust hardware/software, operational safeguards,
and/or strong controls are put in place with mitigation actions to reduce risk from a higher level to an
improbable state.
Hazard Severity: Severity can be broken out into personal injury or loss of asset/mission. Personal injury can
be broadened to include death, disability, illness, and several categorisations of injury (life-threatening, lost
time, minor, etc.). Loss of asset/mission can be broadened to include loss of system, substantial system damage,
minor system damage, property damage, and loss or compromise of mission (incomplete mission success).
CLASS I (CATASTROPHIC) Total direct cost of mission failure and property damage of $2M
or more, OR Crewed aircraft hull loss, OR Unexpected aircraft
departure from controlled flight for all aircraft except when
departure from controlled flight has been pre-briefed.
2 - 38 STO-AG-300-V32
CASE STUDIES
CLASS II (CRITICAL) Total direct cost of mission failure and property damage of at least
$500k, but less than $2M.
CLASS III (MODERATE) Total direct cost of mission failure and property damage of at least
$50k, but less than $500k.
CLASS IV (NEGLIGIBLE) Total direct cost of mission failure and property damage of at least
$20k, but less than $50k.
The FLL staff has expertise in ground and flight test design and operations: load, stress, dynamic and thermal
analysis; and instrumentation and measurement systems development. This expertise, coupled with a large
array of capital equipment and advanced data acquisition and control systems, make the FLL and ideal
laboratory for research and testing of aerospace vehicles and structures flying in the subsonic through
hypersonic flight regimes.
STO-AG-300-V32 2 - 39
CASE STUDIES
archived in a post-flight storage area. Audio communication networks support research operations in the
DATR, covering a broad frequency spectrum for transmitting and receiving voice communications and flight
termination signals for unmanned aerial vehicles. Video monitoring provides real-time and recorded data for
the control and safety of flight test missions.
2.4.9 Appendix
NASA Armstrong Flight Research Centre has released a number of complete procedures that provide detail
that complements the preceding text. These are quite substantial and have been added as Annexes at the end
of this AGARDograph:
Annex A: AFG-7900.3-001 “Airworthiness and Flight Safety Review, Independent Review, Technical
Brief and Mini-Tech Brief.”
This is a high level procedure that provides a process overview that describes what the Title Elements are
and how they fit together within the overall process. It contains two very useful Appendices. Appendix A
‒ covering a Completeness Checklist and Appendix B ‒ covering an interesting list of Challenge
Questions. Both are commended to the reader.
Annex B: AFOP-7900.3-023 “Airworthiness and Flight Safety Review Process.”
This is a detailed procedure that flowcharts the Airworthiness and Flight Test Safety Review Process. To
support this it includes a set of amplifying notes.
Annex C: AFOP-7900.3-022 “Tech Brief and Mini-Tech Brief.”
This is a detailed procedure that flowcharts the Tech Brief and Mini-Tech Brief Process. To support this
it includes a set of amplifying notes.
2 - 40 STO-AG-300-V32
CASE STUDIES
Research flight test typically is performed at low TRL levels, well before product development or certification
flight test.
Besides Research Flight Test, the following Specialized Operations also fall inside the scope of NLR Flight
Operations activities:
• NAVAid calibration and procedure validation services for ANSPs;
• Flight Mechanics Flying Classroom (performed by Delft University of Technology); and
• Airborne Remote Sensing.
Typical for Research Flight Test is a frequent, almost weekly, significant change of the aircraft configuration
to prepare for the next mission by accommodating other sensors, displays and data acquisition systems.
To support this, NLR has its own nationally (Dutch) approved Part 21 Design Organisation to develop and
certify new or changed aircraft configurations, as well as EASA and Dutch approved Part M Continuing
Airworthiness Management and Part 145 Maintenance Organisations to implement the required configuration
changes and prepare the aircraft for the next mission.
2.5.2 Background
NLR has been operating dedicated research aircraft since its foundation. The first research aircraft was a
Fokker F.II, operated between 1920 and 1936, subsequent research aircraft included a Fokker F.VII, a Siebel
Si 204, a Fokker S.14, a Hawker Hunter, a Beechcraft Queen Air, a Fairchild Metro II and since 1991 a Cessna
Citation II, operated in conjunction with Delft University of Technology, Faculty of Aerospace Engineering.
Also, NLR supports the RNLAF Flight Test wing, operating and maintaining the data acquisition system on
the dedicated F-16 test aircraft.
NLR Flight Operations has a very close working relation with Delft University of Technology; it co-owns the
Citation II research aircraft and the Flight Operations and Aircraft Maintenance departments are merged. For
brevity “NLR Flight Operations” is used in lieu of “NLR/TUDelft Flight Operations.”
STO-AG-300-V32 2 - 41
CASE STUDIES
The home base is Amsterdam Airport Schiphol, one of the busiest airports in Europe. Flights normally depart
from a separate runway meaning that the advantages of operating at a large airport can be enjoyed (logistics,
24/7 accessibility and ATC support), without having to merge into the main commercial traffic operating at
the airport. In addition, NLR Flight Operations has a second operational base at Rotterdam The Hague Airport
Zestienhoven.
For airspace use, NLR has a close working relation with ATC. The Netherlands and Dutch Military Air
Operations and is able to access civil and military restricted airspace and airfields if required.
NLR as a company has an ISO 9001:2000 Quality Management System, monitored through audits.
Additionally, NLR has introduced an ICAO Annex 6 and 19 compliant Safety Management System (SMS)
to cover Flight Operations. For maintenance, continuing airworthiness management and design national
regulations apply, that either refer to or are similar to the EASA Part 145, Part M and Part 21 regulations.
2.5.3.1 External
Although NLR’s Citation II C550 aircraft originally has a type certificate against civil (FAA, JAA and EASA)
Part 25 standards, because of its specific role and modification as Research Aircraft, it is operated with an
ICAO Certificate of Airworthiness issued by CAA-NL and falls under the provisions of EASA Annex I to
the Basic Regulation (EC).
Since NLR is a non-profit organisation, all flights are considered as non-commercial activities. Most flights
are executed as part of scientific programs related to NLR or associated parties and organisations, other flights
are either training flights or positioning flights.
NLR does not hold an Air Operator Certificate (AOC), prohibiting commercial air transport activities. Because
the aircraft it operates is an Annex I aircraft, operations are governed by national (Dutch) legislation, and not
by EASA Air Operations provisions. Dutch legislation requires operations with Annex I aircraft to adhere to
ICAO Annex 6, Part II, hence that is the regulatory compliance basis. NLR Flight Operations has opted to
fully comply with EASA Part-SPO provisions and by doing so complies with ICAO Annex 6 and Dutch Law.
NLR Flight Operations processes and procedures are described in the (EASA Part-SPO compliant) Operating
Manual, consisting of the following parts:
Part A ‒ Basic Operations Manual;
Part B ‒ Aircraft Operations Manual;
Part C ‒ Route Manual;
Part D ‒ Training Manual; and
Part X ‒ Flight Test Operations Manual (FTOM). The FTOM describes the flight test preparation and
operational standards and risk management procedures applicable to NLR test flights within a framework
of a research or certification project. The FTOM is controlled by NLR Flight Operations, but also serves
as the NLR Part 21 FTOM, complying with Dutch and EASA Part-21 legislation.
To comply with ICAO Annex 6 and Annex 19 and additional EASA and CAA-NL requirements or guidelines,
NLR has developed an integrated Safety Management System (SMS), applicable to NLR Flight Operations,
Aircraft Maintenance, Continuing Airworthiness Management and the Part-21 Aircraft Design Organisation.
2 - 42 STO-AG-300-V32
CASE STUDIES
2.5.3.2 Internal
NLR has opted to comply with EASA Air Operations provisions, including a management structure with an
Accountable Manager and Postholders Flight Operations, Ground Operations and Flight Crew Training.
The FTOM further identifies the Project Pilot (PP) and Project Flight Test Engineer (P-FTE) as designated
functions within the processes leading to an NLR test flight. For every test flight, or series of test flights,
a Flight Test Plan (FTP) is developed under the responsibility of the designated Project Pilot. A Risk
Assessment is part of the development of an FTP, depending on the outcome of this risk assessment the FTP
is signed for release by:
• The Project Pilot for ROUTINE or LOW RISK flight tests.
• The Postholder Flight Operations for MEDIUM RISK flight tests.
• NLR has chosen in principle not to perform HIGH RISK flight tests, but if required the Accountable
Manager in consultation with the NLR General Director and TU Delft Dean (representing the owners
of the aircraft) will need to approve the FTP.
Additionally, the individual organisations have a quality policy in force that is endorsed by the management
and is recorded in the various management system handbooks.
Additionally, the SMS complies with EU regulations on occurrence reporting (Regulation (EU) No 376/2014
and Commission Implementing Regulation (EU) No 2015/1018), EASA Part ORO and Dutch regulations.
The SMS is an integral part of the NLR Management System(s) and is applicable to all personnel,
management, activities and facilities of the NLR Flight Operations, Aircraft Maintenance, Continuing
Airworthiness Management and Design Organisations (i.e., the Flight Test organisations).
STO-AG-300-V32 2 - 43
CASE STUDIES
Due to the nature of the Flight Test organisations (being a subdivision of NLR) and Accountable Manager
being an ancillary function of an NLR manager, a ‘Delegated Accountable Manager’ position has been defined
to alleviate the Accountable Manager of certain tasks (however not of responsibilities).
The Accountable Manager together with the owner of the aircraft and the Chief Executive of the Part-21
organisation constitute the Safety Committee that is responsible for the effectiveness and safety performance
of the SMS, the appropriate allocation of resources and to yearly review the overall safety objectives, set the
yearly safety goals and promote the safety policy.
The postholders (nominated persons) have a role in the SMS as well. They are individually ultimately
responsible for safety performance within their domain (Flight Operations, Ground Operations, Crew
Training). Together they make up the Safety Action Group that has the responsibility to coordinate risk
management activities (e.g., hazard identification and risk assessment, mitigating actions, and management of
change) in case a safety issue covers more than one domain and to oversee safety promotion activities for the
improvement of safety awareness.
A Safety Manager has been appointed, who in short has the overall responsibility to maintain the SMS system,
collect occurrence reports, facilitate hazard identification and risk analysis, monitor mitigating measures and
evaluate their results and provide periodic reports on the organisations safety performance.
2 - 44 STO-AG-300-V32
CASE STUDIES
Test organisations risk matrix (severity x probability = risk level). For Flight Test related hazards (in the
FTOM) this risk matrix is adapted to incorporate the duration of the project.
3) The responsible postholder classifies the risk of a hazard in terms of tolerability (intolerable, tolerable,
acceptable) and may always upscale if deemed necessary. In case of intolerable risks, the Accountable
Manager is involved. The accepted risk classification determines whether mitigating measures are
required to be taken or not.
4) Mitigating measures are taken under responsibility of the postholder who in due time, when the results
of the action taken should be apparent, checks its effectivity. He initiates additional actions as
necessary.
All SMS related data (occurrences, hazards, mitigating measures), including those resulting from Flight Tests
are recorded in a database system for review over time (are implemented measures still effective?) and as a
knowledge base for future re-use.
STO-AG-300-V32 2 - 45
CASE STUDIES
The SPI are measured by the Safety Manager by one or a combination of various means:
• Review meetings of Flight Tests or changes;
• Using the information in the database system;
• Surveys among personnel; using trend information from the reporting system; and
• And audit results.
The Safety Manager reports the safety performance to the Accountable Manager.
All SMS related information is communicated through a common Flight Test organisations digital portal.
Furthermore, safety flyers, minutes of meetings and workshops will be used as communication means as
appropriate.
2 - 46 STO-AG-300-V32
CASE STUDIES
Because NLR does not hold an Aircraft Operator’s Certificate, NLR Flight Operations are not supervised by
EASA or National Authorities but operate under ICAO Annex 6 provisions. The internal NLR Quality System
oversees Flight Operations compliance with legislation and Best Practices.
The Postholder Flight Operations reports to the Accountable Manager Flight Operations.
Test flights are always performed within the framework of a “project” within NLR, governed by ISO9001 quality
standard. Each NLR project will be assigned a Project Manager. For those NLR projects that involve flight test,
a Project Pilot is assigned who is responsible for drafting the Flight Test Plan (FTP). Because the Project Pilot
may be required to engage in the scientific background of the project, NLR pilots are required to have a strong
scientific or engineering background, typically a Masters Degree in Engineering. In addition, pilots will need to
have a flight test background, although it is possible that this experience is gained on the job.
Certification test flights may have to be performed as part of a design change to the aircraft. As these are
governed by EASA FTL rules, NLR pilots will typically also hold an EASA Category 1 or 2 Flight Test Rating.
STO-AG-300-V32 2 - 47
CASE STUDIES
4) Execution Phase: This phase includes the pre-flight go-nogo decision, the pre-flight briefing, the test
execution and the post-flight debriefing.
5) Evaluation Phase: After the flight test campaign has been finished, it will be evaluated in order for
future projects to benefit from the experiences gained. Lessons learned will be identified for each
phase of the project. Within the scope of the Safety Management System this may then result in
updates of Operations Manuals and forms. Typically, after the flight test campaign a Flight Test Report
will be produced. The Flight Test Report does not need to have a specific format as it may be a Project
Document. This report will normally contain the Lessons Learned as outlined above, if this is not the
case, or Lessons Learned can be drawn outside the scope of the Project.
2 - 48 STO-AG-300-V32
CASE STUDIES
2.5.5.2.4 Severity
Severity is expressed as a level based on effect on aeroplane, effect on crew and occupants, effect on test flight,
and financial impact (which includes effects on third parties), and is categorised as Catastrophic, Hazardous,
Major or Minor.
2.5.5.2.5 Frequency
When available, a frequency of occurrence will be established for the hazard under investigation. In-flight test
it may prove difficult to determine the frequency of occurrence as by definition certain aspects cannot be
assessed using previous exposure data. An alternative method to establish frequency may be used based on the
effectiveness of the control measure, in fact the inverse of the frequency. For control measures that are deemed
very effective, e.g., redundancy is available, normal flight crew intervention sufficient to mitigate the risk, the
frequency of a specific hazard leading to an undesired outcome is lower. The effectiveness of a control measure
is categorised as Effective, Limited, Minimal or Not Effective. So even without being able to qualitatively or
quantitatively establish a frequency, with the effectiveness of the control measure it is possible to identify the
risk of a scenario.
NLR has opted to adapt an organisation based on EASA (Part-SPO) Regulations. Pillars of the flight test
organisation are the Flight Operations, Part-145, CAMO and Part-21 organisations. Project Management is
regulated through NLR ISO:9001 Quality Management standards.
Overarching the flight test organisation is an integrated SMS based on ICAO Annex 19 and following ICAO
Doc 9859 guidelines. Typical challenges for a small flight test centre include little data from reporting systems,
and little operational data. It is almost impossible to quantitatively apply data analysis, identify trends and
measure performance on the scarce data available. A more qualitative approach is applied, reverting to e.g.,
awareness campaigns and training.
NLR is a relatively small organisation. As a result, persons may hold more than one responsibility in the
organisation or may have more than one task in the preparation, execution or evaluation of a test flight.
Attention must be given to adequately separate tasks and responsibilities, and always have a separate set of
eyes available for each step along the way.
STO-AG-300-V32 2 - 49
CASE STUDIES
2.5.9 Appendix
2 - 50 STO-AG-300-V32
CASE STUDIES
2.6 SAAB AB
The Flight Test and Verification department is responsible for development test, verification and validation of
experimental, prototypes, pre-production and production aerial vehicles systems through simulator, ground
and flight testing. It also provides support to the marketing department with demonstrations, evaluations plus
land and aerial displays linked to various marketing activities.
The Organisation has Flight Test Capabilities covering the following areas.
2.6.2 Background
SAAB AB is a Swedish Aerospace and Defence Company founded in 1937. It is based at Linköping in
Southern Sweden. Since the 1930s the company has produced a long line of mostly fighter aircraft. Notable
are the Tunnan, Lansen, Draken, Viggen and Gripen. Production today is focused on the Gripen and its
continuing development.
STO-AG-300-V32 2 - 51
CASE STUDIES
2.6.3.2 Internal
2.6.4.1 Foundation/Training
Test pilots graduated from an international test pilot school will have many of the internal education program
elements covered.
2 - 52 STO-AG-300-V32
CASE STUDIES
On top of the general test pilot training is type training requirements for specific aircraft types.
An important part of flight operations is supervision of flying duty and the command chain. All these roles
have training requirements including flight safety aspects.
The program consists of three certification levels Test Engineer, First Test Engineer and Senior Test Engineer
(Figure 2-6.1).
Figure 2-6.1: Internal Education Program for Test Aircraft Maintenance Personnel.
Two of the major goals are to achieve support fulfilment of authorities’ request of formal education and achieve
uniformed, efficient and, above all, safe working methods.
An important part of the program is On the Job Training working according to our Systems and Flight Test
Process. Achieving higher authorisation levels is a prerequisite for leading hazardous flight test.
STO-AG-300-V32 2 - 53
CASE STUDIES
All test flights are carried out under an applicable Flight Test Permit given by the Airworthiness Authority.
Based on Airworthiness Certificates and Operational Plans these Flight Test Permits can span long periods
where the Air Station operates within those permits.
The focus on flight safety regulated in the Air Stations operational manuals and expositions are the most
important foundation for these permits.
Flight Safety Officers covers flight, ground handling, maintenance and test methods.
An important part of the Flight Safety work is to early identify the need for and requirement on safety
equipment and design for test functions.
Envelope Expansion Routines: When the configuration to be tested have several new systems, functionalities
and physical shapes a special frame test plan will govern the overarching configuration steps, envelope
expansion steps as well as risk mitigation requirements. The ultimate example is of course a completely new
aerial vehicle being flown for the first time. The underlying flight test plans will still be assessed regarding
their own risks.
This task is performed by a group of experienced personnel from most involved flight test disciplines,
including maintenance.
The aim of the review is to ensure that the collected experience has been used in order to accomplish the test
with all consideration taken to flight safety.
2 - 54 STO-AG-300-V32
CASE STUDIES
2.6.5.3.1 Events
Dangerous events, which could be conceived as possible during the test flight, are to be identified. Only events
related to the test are to be dealt with. General risks that are always associated with flight activity (e.g., ice
accumulation, collisions and similar) should not be a part of this assessment unless the test itself leads to an
increased probability for these risks.
2.6.5.3.2 Cause
The cause of the event is identified. Here, whatever it is that causes the event (condition, function, pilot actions)
is stated. Not the event, the risk or the consequence.
Consequence
Here, an assessment is made of the consequence of the event in question. The consequence is graded according
to:
CATASTROPHE Malfunction/condition that seriously reduces the aircraft’s function, handling
qualities and/or the pilot’s capability to such a degree that continued safe flight and
landing is not possible, with total loss of material, death and serious damage to the
environment as a result.
CRITICAL Malfunction/condition that reduces aircraft function, handling qualities and/or pilot
capability to such a degree that serious personal injury, severe damage to material
or great environmental damage can result.
MARGINAL Malfunction/condition that degrades aircraft function and handling qualities in a
way that is familiar, and at a level that the pilot can handle, but can lead to minor
personal injury, minor damage to material, minor damage to the environment.
NEGLIGIBLE Malfunction/condition that does not degrade aircraft function and handling qualities
or does not noticeably increase the workload on the pilot.
2.6.5.3.3 Probability
A subjective estimate of the probability of an event actually happening is done.
STO-AG-300-V32 2 - 55
CASE STUDIES
2.6.5.3.5 Results/Actions
A final review is performed and the risk index set by considering the actions that have been taken to reduce
risks (Table 2-6.2).
2 - 56 STO-AG-300-V32
CASE STUDIES
occurs. These test flights are therefore guided and monitored by an applicable test team from a control room
by means of telemetered data and radio communication.
The general roleplay when performing a test flight is that the Test Conductor oversees and guides the Test
Pilot through the whole flight, from start-up to shut-down. The Test Leader focuses on initial conditions,
performance of test cases and quick analysis of online data compared to expected results. Test Engineers
monitors instrumentation data compared to expected behaviour and supports both the Test Conductor and Test
Leader with advice on how to proceed with the test flight.
Generally, only the Test Conductor communicates with the crew on board the test aircraft.
A Safety Pilot will often be present for hazardous testing to help reduce pilot workload both for known
procedures and in unexpected events.
Online data is nearly ever directly relied upon for flight safety, but still can help mitigate risks through being
able to establish known and safe initial conditions before the test point are executed.
2.6.9 Appendix
N/A
STO-AG-300-V32 2 - 57
CASE STUDIES
2.7 TURKISH AIR FORCE FLIGHT TEST CENTER 401ST TEST SQUADRON
2.7.2 Background
Within TurAF, Flight Test Activities started with the establishment of Test and Evaluation Branch within
Technology and Weapon Development Command (TWSDC) which is located in Eskişehir. The TWSDC has
also Avionics and Aeronautical Branches which are capable of F-4 and F-16 avionics and weapon systems
integration.
The T&E Branch initially had no dedicated Test Aircraft, Flight Test Engineers or Test Pilots. It only had
the Flight Test Instrumentation (FTI) capability and coordination role. Early flight test programmes were
accomplished with the support of operational pilots and maintenance people provided by combat bases and
discipline engineers provided by either TWSDC or Contractor Company.
As the complexity and the number of the projects increased, the necessity of establishing a test squadron
emerged. During transition period, T&E Branch was transformed into Developmental Test Detachment
Command in 2013 and then into 401st Test Squadron in 2015. Following the reorganization of TurAF in 2016,
401st Test Squadron was tied to 1st Main Jet Base.
It is now the only Flight Test Centre performing jet aircraft-store compatibility tests within Turkish Air Force
and Turkish Defence Industry.
Its history begins with the Turkish Precision Guided Munition (Hassas Güdüm Kiti – HGK) Programme
in 2006 and since then over fifty different types of inertial aided munitions, cruise missiles, short range and
medium range A-A missiles, reconnaissance pods, targeting pods, gunnery pods, EW Pods integration and
certification tests were accomplished.
With the launch of Turkish National Fighter Aircraft (TFX) Programme, 401st Test Squadron is now being
prepared to lead the flight test of TFX which is planned to be held on a joint basis with industry partners.
In the near future, it is planned to transform Eskişehir into a unique Test and Evaluation Base including Fixed
Wing, Rotary Wing and UAS capabilities.
2 - 58 STO-AG-300-V32
CASE STUDIES
However, Turkish Ministry of Defence is working on the establishment of Turkish Military Airworthiness
Authority. And by the law come into force, The Authority is expected to rearrange and control all airworthiness
and flight test safety issues.
2.7.3.2 Internal
The main Air Force Regulation driving the 401st Test Squadron Flight Test and Evaluation activities is Turkish
Air Force Instruction HKY 165-15 Flight Test and Evaluation. All Safety management and flight release
process are described in HKY 165-15.
Within TurAF, the technical management responsibility of the aircrafts is shared by two Air Supply and
Maintenance Centre (ASMC). The 1st ASMC, which is located in Eskişehir, is responsible for all Fixed Wing
jet aircrafts, and the 2nd ASMC is responsible for all rotary and Fixed Wing propeller type aircrafts.
401st Squadron is located at the ASMC facilities. ASMC is responsible for all modifications and configuration
control of the Aircrafts including FTI, the 401st Test Squadron is responsible for Flight Testing.
For the risk assessment and safety approval of each flight test campaign, a Flight Test Safety Review Board
(Test Uçuşu Emniyet Kurulu – TUEK) is established. After TUEK approval, recommendation for flight release
is submitted to TurAF.
Each Flight Test Project Team has its approved Flight Test Engineers and Test Pilots from 401st Test Squadron
and Systems Engineers and Specialists from TWSDC and Contractor Company. TurAF Flight Test Safety
Process has three stages.
1) Firstly, a Flight Test Technical Review Process is applied to review the flight test design, requirements
and success criteria, test items, required tools, facilities, test ranges, test steps, sortie plans, safety and
security criteria, and to identify and mitigate the risk levels. A specific Flight Test Technical Board
(Test Uçuşu Teknik Kurulu – TUTEK) is established for technical review of every proposed Flight
Test and Evaluation Plan. TUTEK is coordinated by Flight Test Engineering Command and chaired
by the Commander of TWSDC. The organisational structure of the TUTEK is given in Figure 2-7.1
STO-AG-300-V32 2 - 59
CASE STUDIES
Figure 2-7.1: The Organisational Structure of TUTEK – Flight Test Technical Board.
2) Following TUTEK, the TUEK realizes the Flight Test Safety Review Process to review all safety of
flight analyses and tests and to ensure that risk is identified and mitigated to acceptable levels and to
approve safety of flight. TUEK is coordinated by Flight Test Engineering Command and Co-Chaired
by The Commander of ASMC and The Commander of the 1st Main Jet Base. The organisational
structure of the TUEK is given in Figure 2-7.2.
3) After the TUEK, recommendation of the flight tests is submitted to TurAF. The flight test is accepted
and released by The Air Force Chief of Staff.
Figure 2-7.2: The Organisational Structure of TUEK – Flight Test Safety Review.
2 - 60 STO-AG-300-V32
CASE STUDIES
STO-AG-300-V32 2 - 61
CASE STUDIES
2 - 62 STO-AG-300-V32
CASE STUDIES
5) Following the flight test briefing, test pilot conducts a separate flight briefing with his/her crew,
chase and/or support A/C about inter-flight operations.
6) After each sortie a formal debrief is done. All test events are reviewed by the team and a post-flight
report is prepared. All deficiencies are categorised into three groups. The first category deficiencies
have the highest priority and unless solved the test is suspended. All safety critical issues are assessed
as category one.
Flight Test Safety Management Chart followed by 401st Test Squadron is shown in Figure 2-7.3.
STO-AG-300-V32 2 - 63
CASE STUDIES
To identify risk levels, the nature of the test, flight test profile/envelope, environmental issues and mission
requirements are investigated. Controls and Actions for each risk are defined according to Risk Management
Plan and a watch list is initiated for each risk.
Ratings are indications of potential impacts of risks on a test program. They are a measure of the likelihood of
a hazards/mishaps occurring and consequences of them.
Risk categories are expressed as High, Medium and Low. The final risk categorisation is reviewed and
approved by TUEK.
The Risk Matrix used is derived from Air Force Risk Management Regulation.
Likelihood Criteria and the Risk Matrix are given in Table 2-7.1.
Table 2-7.1: Likelihood Criteria ‒ Risk Occurrence Probability, Risk Severity and the Risk
Matrix.
Likelihood Criteria
Level Risk Occurrence Probability
1 Remote Can assume will not occur (Possible, but improbable).
2 Unlikely Possible to occur (Remote chance of occurrence).
3 Likely Occurs sometime (Sporadically).
4 Highly Likely Occurs several times (Frequently).
5 Nearly Certainly Occurs often (Continuously).
2 - 64 STO-AG-300-V32
CASE STUDIES
Likelihood Criteria
Level Risk Severity
E Very Low Less than minor mission degradation, injury, occupational illness,
or minor system damage.
D Low Minor mission degradation, injury, minor occupational illness, or
minor system damage.
C Medium Key mission degradation, injury, minor occupational illness, or
minor system damage.
B High Major mission degradation, severe injury, occupational illness or
major system damage.
A Very High Complete mission failure, death, or loss of system.
Risk Matrix
SEVERITY/ E D C B A
OCCURANCE
PROBABILITY
5 5E 5D 5C 5B 5A
4 4E 4D 4C 4B 4A
3 3E 3D 3C 3B 3A
2 2E 2D 2C 2B 2A
1 1E 1D 1C 1B 1A
The risk level is determined by entering the risk assessment chart with both the occurrence severity and
probability, and seeing what risk category it fits into:
HIGH RISK ‒ Test or activities which present a significant risk to personnel, equipment, or property, even
after all precaution measures have been taken. This necessitates close oversight.
MEDIUM RISK ‒ Test or activities which present a greater risk to personnel, equipment, or property than
normal operations and require more than routine oversight.
LOW RISK ‒ Test or activities which present no greater risk to personnel, equipment, or property than normal
operations.
It is believed that Risk/Hazards/Mishaps can result from Human Factors, Environmental Factors, Logistic
Factors, and Management factors. These factors are described below.
Human Factors: Answers to the following questions should be “yes” to minimise the flight testing risks:
• Flight Test Crews and Pilots are medically and professionally qualified?
• FTC and Pilots are trained in all relevant flight safety procedures, especially those specific to the test
aircraft?
STO-AG-300-V32 2 - 65
CASE STUDIES
• Do they have enough knowledge about conducting test and test platform?
• Do Crew members’ have enough previous experience?
Environmental Factors: Flight testing environment is one of the most important risk factors, Experienced
Flight Test Crews should consider the weather condition on Flight Testing Day, NOTAM’s, NOTMAR’s FOD
protections, Runway lighting Condition, etc.
Logistic Factors: The materials, arms, and other equipment used in flight testing should be controlled and it
should be ensured to have no risk from that parts. Before Flight testing reliability analyses and maintenance
documents should be carefully checked.
Maintenance Factors: All the Flight Test Crews should be aware of having the answers to following
questions:
• Powers and Responsibilities are known at each level?
• Test plans, risk documentations and checklists are available and well prepared?
• Does TurAF FTC have Risk Emergency Plans for that test?
1) Risk Assumption: is an acknowledgment of the existence of a particular risk situation and a conscious
decision to accept the associated level of risk, without engaging in any special efforts to control it. It is
most suited for those situations that have been classified as low risk.
2) Risk Avoidance: involves a change in the concept, requirements, specifications, and/or practices that
reduce risk to an acceptable level. Eliminates the sources of high or possibly medium risk and replaces
them with a lower risk solution. The risk avoidance is a decision process among the effected parties.
The required measures should be defined during test design phase.
3) Risk Control (Mitigation): This is the process that identifies, evaluates, selects, and implements
options in order to set risk at acceptable levels given program constraints and objectives. This includes
what should be done, when it should be accomplished, who is responsible, and associated cost and
schedule.
Risk Control does not attempt to eliminate the source of the risk. It handles the risk in a manner that
reduces the likelihood of its severity and/or occurrence.
4) Risk Transfer: is reallocating risk during the concept development and design processes from one
part of the system to another.
2 - 66 STO-AG-300-V32
CASE STUDIES
Flight Test Personnel at every level must fulfil their respective roles in assuring controls. Final controls and
reviews are made in pre-flight briefings.
Once controls are in place, the process must be periodically revaluated to ensure their effectiveness.
Feedback informs all parties about how the implementation process is working, and whether or not the controls
were effective. Feedback can be in the form of briefings, lessons learned, risk documentations, test reports, etc.
2.7.9 Appendix
N/A
STO-AG-300-V32 2 - 67
CASE STUDIES
2.8.2 Background
NAVAIR is responsible to support the Naval aircraft acquisition for the US Navy and US Marine Corps.
The Naval Air Warfare Centers in locations identified in Section 2.8.1, provide the ranges, facilities,
infrastructure, personnel, operational and risk management oversight to support the acquisition process.
The workforce is comprised of a mixture of uniformed military, government civil service, and contractors.
Work to support external customers is also conducted using existing processes.
2.8.3.1 External
US Department of Defense Instruction 5000 series governs the acquisition process for the US military.
Some aircraft acquisitions are a combination of DOD and US Federal Aviation Administration policy.
2.8.3.2 Internal
Risk management and safety is integral for the Navy and Marine Corps, and NAVAIR follows that model.
NAVAIR policy for flight test risk management is the purview of the Engineering Department. Naval Air
Systems Command Instruction (NAVAIRINST) 3960.4C “Project Test Plan Policy for Testing Air Vehicles,
Air Vehicle Weapons, and Installed Air Vehicle Systems” provides policy and process for the execution
of testing for US Naval aviation.
2 - 68 STO-AG-300-V32
CASE STUDIES
A key component of flight testing at NAVAIR is the project officer project engineer team. The Project Officer /
Test Pilot brings knowledge of how the fleet uses aircraft and the Flight Test Engineer brings detailed
knowledge of how the systems work along with the system integration to form a very effective test team.
This teaming aspect is applied throughout the organization, up to and including senior leadership where the
Commanding Officer (CO) or Chief Test Pilot (CTP), depending on risk, and Chief Test Engineer approve the
test plans and ensure the spirt and intent of the test plan instruction is followed.
STO-AG-300-V32 2 - 69
CASE STUDIES
level and the appropriate risk mitigations are in place. There are three sections in NAVAIR test plans
specifically addressing flight test safety risk, risk management, test hazard analysis, and the safety checklist.
NAVAIR also uses the “No Vote” as a safety tool that is embraced throughout the organization. Additionally,
Go-No-Go criteria are required in the test plan.
2 - 70 STO-AG-300-V32
CASE STUDIES
Standardization (NATOPS) or Naval Aviation Technical Information Products (NATIP). The THA shall
address those hazards which are directly associated with the testing (test-specific). “Generic” hazards
associated with normal operation of the aircraft or test equipment should not be included.
2.8.6.1.4 Firebreaks
Frequently, testing involves the actuation of weapons release controls either in a simulated launch condition
or during actual releases. Firebreaks instruction or local ordnance instructions were developed to ensure an
inadvertent release of the store under test or other loaded stores do not occur. These instructions are very
specific about which weapons release actuations are allowed under which conditions. For all tests involving
the actuation of weapons release controls, either during a simulated or actual release, a statement regarding
adherence to the local Firebreaks instruction shall be made. Tests which do not adhere to Firebreaks
requirements shall be specifically addressed with risk mitigation measures discussed.
STO-AG-300-V32 2 - 71
CASE STUDIES
will be aborted, and the airplane will Return To Base (RTB) as soon as practical. Maneuver
termination will be briefed.” “For any non-emergency termination of manoeuvres, engineers at
Real-Time Processing System (RTPS) will call “KNOCK IT OFF.” This usually applies to loss of
Telemetry (TM) or approaching test limits (Angle of Attack (AOA), Angle of Side Slip (AOSS), etc.).
If emergency termination of the maneuver is required or if the airplane appears to be out of control,
engineers at RTPS will call, “ABORT, ABORT, ABORT,” and give altitude calls in 5,000 ft increments
until 10,000 ft Mean Sea Level (MSL) and in 1,000 ft increments below 10,000 ft MSL.”
2.8.6.2.1 Purpose
The Test Hazard Analysis (THA) is the primary tool for determining the risk associated with the conduct of a
project. The THA process is designed to clearly identify and minimize existing and potential hazards inherent
in the test planning and execution process.
2.8.6.2.2 Background
The THA is an essential part of the overall risk management process to identify and mitigate hazards unique
to the planned test. Risk may be defined as “an expression of possible loss in terms of mishap severity and
mishap probability.” It is useful to expand somewhat on this definition of risk. Loss is measured in lives,
dollars, equipment, and mission capability. Risk assessment, therefore, involves determining the hazard
involved, predicting resulting frequency of occurrence, assessing severity or consequences, determining
exposure, and identifying action to avoid or minimize the risk. The THA is required to discuss potentially
hazardous conditions created by testing the item in question. When defining potential hazards, consideration
should be given to the specific test item, the test maneuvers and flight conditions planned, and the environment
in which the test will be conducted. Ensure that workload during critical flight maneuvers is taken into
consideration when determining risk category.
2.8.6.2.3 Process
The following outline provides the desired information, which should be incorporated in the THA. The tabular
presentation of the THA (example in the following pages) is the preferred format. Different formats are
acceptable, with prior approval of the TECT, but all of the below elements must be included. Test teams shall
use the THA as a risk management tool during the conduct of the test, to include reviewing the THA during
pre-test briefings:
Step 1: Identify the hazards associated with the test. Some methods to identify potential hazards include
test team discussions, conducting fault tree analysis, reviewing historical data, and reviewing hazard
analysis and flight test lessons learned databases maintained by US Navy, US Air Force, and Society of
Experimental Test Pilots.
Step 2: Identify the root cause(s) and their associated effect(s). The root cause is anything that could lead
to the presence of the hazard identified in step 1. What is the root cause of the hazard? What is the effect of
the hazard being uncontrolled? What aircraft system and subsystem failure modes can be identified?
2 - 72 STO-AG-300-V32
CASE STUDIES
Hazard Severity
Hazard Level Severity Definition
I Catastrophic May cause death or aircraft/system loss.
II Critical May cause severe injury or major aircraft/system damage.
III Marginal May cause injury or minor aircraft/system damage.
IV Negligible Will not result in injury or aircraft/system damage.
Hazard Probability
Hazard Probability Definition
A Frequent Likely to occur immediately, or during an individual test event.
B Probable Probably will occur during this evaluation.
C Occasional May occur during this evaluation.
D Remote Unlikely to occur during this evaluation.
Note: Loss of a Group 1 – 3 UAV does not necessarily constitute a catastrophic event. Severity of loss of
a Group 1 – 3 UAV shall be based on UAV cost and potential for injury.
STO-AG-300-V32 2 - 73
CASE STUDIES
Hazard I II III IV
Probability
Catastrophic Critical Marginal Negligible
A ‒ Frequent UA3 UA3 Category C4 Category B5
B ‒ Probable UA3 Category C4 Category C4 Category A6
C ‒ Occasional Note 1 Category C4 Category B5 Category A6
D ‒ Remote Note 2 Note 2 Category A6 Category A6
Notes:
1) For test results with a residual risk assessment of I/C, discussions with the TECT will be required
prior to proceeding with the test program under development.
2) For assessments that result in I/D or II/D, coordination with the TECT (prior to an ERB) will
determine assignment of Category A, B, or C testing classification.
3) Unacceptable Risk (UA) means that the project is considered too high risk to proceed with testing.
4) Test Category C: Test or activities which present a significant risk to personnel, equipment, or
property even after all precautionary measures and corrective actions would be taken.
5) Test Category B: Test or activities which present a greater risk to personnel, equipment, or
property than normal operations.
6) Test Category A: Test or activities which present no greater risk than normal operations.
Step 7: Assign the test plan risk category(s). A description of risk categories, (Exhibit 1), examples of
possible test risk categories (Exhibit 2) are presented on the following pages.
The Risk Category Matrix shall be used to make the final assessment of the appropriate project risk category.
Category D will encompass all ground and flight tests of prototype/pre-production aircraft and will not be
determined by the Risk Category Matrix. Empowerment for test plan approval includes the responsibility for
determining project category.
2 - 74 STO-AG-300-V32
CASE STUDIES
CATEGORY A: Ground tests or project flights not involving potential or known hazardous operations. This
includes flights within the NATOPS flight envelope not involving testing of critical safety of flight
components.
Examples include:
• Antenna patterns (specific category can be A/C dependent).
• Ordnance lot testing.
• Cruise performance tests.
• Pace flight at altitude with non-critical avionics.
• Generally, most ground and laboratory tests.
• Sensor evaluation (not including night vision devices).
• Inert Missile Functional Carriage Tests.
• Government Lot Acceptance Tests (GLAT).
Examples include:
• Automatic Carrier Landing Systems ‒ Shipboard.
• Engine Stall Susceptibility.
• Stores Separation of non-standard or modified stores.
• Air Start Envelope Definition ‒ Multi-engine.
• Accelerated Service Testing.
• Engine Component Improvement.
• Engine-Out Testing: One engine on three- or four-engine aircraft.
• Catapult and Arresting Gear Certification.
• Mission software not yet flown that could affect flight-related displays.
• Navigation/bombing accuracy.
• Tower fly-by tests.
• Air Combat Maneuvering (ACM).
• Night Bombing Test.
• Captive carry of live stores.
• Engine performance.
• Countermeasures towing of untested stores.
• Decoy Flare Lot Acceptance flight tests.
• Initial Instrument Meteorological Conditions (IMC).
• Safety of Flight (SOF) software checks.
• Lab tests that intentionally induce faults on power lines.
STO-AG-300-V32 2 - 75
CASE STUDIES
Examples include:
• Flutter Testing.
• First flight of new/modified aircraft configuration.
• Aborted takeoffs.
• Ground and air minimum control speed determination.
• Spins.
• High Angle of Attack Evaluations.
• Air Start Envelope Definition ‒ Single Engine.
• Minimum End-speed Catapult Shots.
• Carrier Suitability Structural Testing.
• Helicopter/Ship Dynamic Interface Testing.
• Envelope Expansion.
• Full Autorotation.
• Flight Control Software.
• Stores separation for envelope definition or expansion.
• Hazardous stores jettison tests.
• Missile gas ingestion engine tests.
CATEGORY D: Ground tests on, or all flights in, prototype aircraft including all pre-production aircraft and
any other aircraft whose unique configuration or value warrants CATEGORY D designation by the Directors,
Flight Test Engineering and Test Wing Commanders.
Note: AIRCREW QUALIFICATION: Due to the hazards involved with certain project flights and the
increased level of aircrew experience required to safely conduct certain flight test projects, flights have been
divided into categories. Aircrew minimum qualification and currency required for each category are defined
in wing and squadron Standard Operating Procedures (SOP). TECT shall ensure, with the assistance of the
squadron’s operations officer, that the minimum aircrew qualifications specified are met for the type category
of test to be conducted.
2.8.6.3.1 Purpose
A Safety Checklist is required for all test plans. The purpose of the checklist is to stimulate thought in the area
of safety and to ensure no critical aspects of test planning are omitted.
2.8.6.3.2 Background
Most of the questions in the Safety Checklist result from lessons learned from past mishaps in the RDT&E
community. Flight test organizations external to NAVAIR that have their own safety assessment may submit
them to TECT for review. If deemed adequate, the TECT may accept these documents in lieu of the NAVAIR
Safety Checklist.
2 - 76 STO-AG-300-V32
CASE STUDIES
2.8.6.3.3 Scope
The Safety Checklist should be included as an appendix to the test plan, and the questions may be answered
either in the checklist or in the test plan body. If the question is covered in the main body of the test plan,
the applicable section or paragraph number should be placed in the “Response” column. If the question is
answered in the Response column in the checklist, the answer should be brief and to the point, and may include
a one-word response such as “no”, as long as the question is comprehensibly answered. Questions requiring
detailed answers should generally be included in the test plan body and only referenced in the checklist to
ensure maximum clarity of the test plan. Do not be redundant by answering the question in both places. If the
question does not apply, indicate in the “Response” column. There should not be any blank responses boxes
in the completed checklist.
2.8.6.3.4 Contents
The Safety Checklist is broken out into five sections:
1) General;
2) Ground, EMC, and Anechoic Chamber Tests;
3) Ordnance, Stores, or Expendables Tests;
4) Airborne Tests; and
5) UAS Tests.
The general section addresses common concerns and must be included in all test plans. The remaining sections
address specific types of testing. For test plans including “Ground, EMC, and Anechoic Chamber”, “Ordnance,
Stores, or Expendables”, “Airborne” or “UAS” testing, only those applicable sections of the Safety Checklist
must be included. Any changes to the Test Plan require a review of the Safety Checklist. The safety checklist
is presented in Table 2-8.3.
STO-AG-300-V32 2 - 77
CASE STUDIES
2 - 78 STO-AG-300-V32
CASE STUDIES
40 Are any special procedures required (other than normal loading or release checklist
adherence) to guard against in-flight loss of ordnance or aircraft equipment? In the
STO-AG-300-V32 2 - 79
CASE STUDIES
2 - 80 STO-AG-300-V32
CASE STUDIES
2.8.6.4 No Vote
If anyone with information, understanding or a belief that conditions exist that could lead to injury or
equipment damage are obligated to exercise their “No Vote” at any point during test planning or test execution.
Once leadership receives a “No Vote” they are obligated to respond with mitigation or information that
addresses the “No Vote”. In addition, there are four broad classes of issues or challenges within NAVAIR and
who the responsible party for resolution is, depends on the context of the question being raised. These classes
of challenges are discussed in the following paragraphs.
STO-AG-300-V32 2 - 81
CASE STUDIES
2.8.6.6 Training
The flight testers at NAVAIR are trained several ways that provide a focus on flight test safety. The test pilots
are training at a recognized military test pilot school, most commonly, the US Naval Test Pilot School
(USNTPS) at Patuxent River, Maryland, USA. The Flight Test Engineers receive training through the College
of Test and Evaluation (part of NAVAIR University), USNTPS, and on the job training.
2.8.9 Appendix
Chase aircraft requirement for Unmanned Aerial Systems (UAS) is determined by an appropriate authority in
consideration of, but not limited to, the following:
• UAS maturity and reliability,
• UAS operator situational awareness, and
• UAS capability to integrate into the see and avoid environment including the visual detection of the air
vehicle from other aircraft.
A chase aircraft provides essential risk mitigation for high risk test missions as an additional cockpit resource
enhancing the test pilot’s situational awareness (SA). High risk test points are often very high workload
maneuvers that saturate the test pilot’s ability to maintain SA outside of his aircraft. Additionally, many test
aircraft used to conduct high risk test points are prototype or pre-production aircraft with limited mission
systems (e.g., radar, position displays) available to the test pilot for SA. The safety chase pilot enhances the test
pilot’s SA in the following areas:
• Traffic deconfliction
• Airspace management
• Monitoring test aircraft flight parameters
2 - 82 STO-AG-300-V32
CASE STUDIES
Test safety increases during tanker operations by pre-checking the tanker’s refueling system prior to
engagement by the test aircraft. Test aircraft are often configured with a flight test noseboom that significantly
increases the risk of in-flight refueling. Ensuring the proper operation of the drogue by the chase aircraft
minimizes the risk of damage to a potentially one-of-a-kind test asset.
A chase aircraft in close proximity to the test aircraft permits real-time monitoring of test aircraft parameters
during dynamic testing. The chase aircraft relays radio transmissions from the control room that are missed by
the test aircraft antenna positioning during dynamic maneuvers. An immediate call to abort the test can make
the difference between a mishap and simply having to repeat a test point. Should a mishap occur, the chase
pilot is immediately on-station to initiate a rapid search and rescue response.
In addition to the safety enhancements provided by a chase aircraft, test efficiency improves as well. A visual
inspection by the chase aircraft provides information not obtainable by the test pilot or the control room through
any other means, allowing the test team to make sound risk decisions to continue testing or return to base.
During a previous major test program, an estimated 10 – 15 % of all high risk test sorties required a safety
chase visual inspection during the flight. Examples include:
• Detection of deformed or missing panels.
• Detection of missing telemetry antennas and towed decoys.
• Detection of missing weapons fins and loose arming wires.
In order for the safety chase to provide the maximum effectiveness, it must be able to maintain close proximity
to the test aircraft and have sensors to increase SA. High-performance chase aircraft provide the following
advantages:
• Provides the optimum balance of performance and maneuverability characteristics to enable precision
and accuracy during demanding chase missions across the envelope of nearly all tactical aircraft
operating regions (for example, low speed to supersonic, low to high altitude, and less than zero-g up
to high normal acceleration (Nz)).
• Allows extended on-station time to improve efficiency through in-flight refueling and external fuel
tanks configurations.
• On-board sensor capability allows range clearance, airspace monitoring, and other test- unique target
requirements through organic on-board sensors.
• Provides the capability to carry a variety of external stores such as the airborne refueling store,
Telemetry (TM) and Time Space and Position Information (TSPI) relay pods, and unique test
measurement equipment such as capability for in-flight infrared measurement.
• Provides the necessary performance and maneuverability to permit close aboard collection of
photogrammetric data for myriad test points such, weapons carriage and release, airflow visualization,
and in-flight vibrations.
Using organic test pilots to chase test flights brings the collective capability of the test team to a level that
would likely be unachievable with external chase assets. Chase pilot qualification in the test aircraft is the ideal
situation, but not required. Familiarity with the test aircraft provides enhanced SA and improves the chase
pilot’s ability to assist the test pilot. Additionally, the experience gained as a chase pilot augments that pilot’s
competence during subsequent missions as the test pilot. Finally, chase missions provide a necessary means to
keep pilot’s aviation skills sharp.
STO-AG-300-V32 2 - 83
CASE STUDIES
2 - 84 STO-AG-300-V32
Chapter 3 – OBSERVATIONS AND DISCUSSION
3.1.1 Overview
Every Flight Test Quality Management System must not only satisfy the internal requirements of the Host
Organisation and the Specific Project but also the external requirements placed by International/National
Standards and International/National Regulatory Requirements. For any Aerospace Organisation there are
many standards and requirements that must be satisfied in order to certify and qualify a product and to trade
successfully.
In the area of Flight Test Safety, the Case Studies give examples of the specific practices adopted by
the contributing organisations. There is no standard model and each organisation has put in place a system that
satisfies the requirements of the local regulatory regime. Anyone endeavouring to put in place a new system
must therefore carefully study their local requirements and establish the system in conjunction with
the Regulator. It is the Regulator who will grant the final approval.
Special care must be taken on International Projects where the requirements of several different National
Regulatory Regimes may need to be satisfied.
3.1.2 International
There are no internationally recognised standards that universally govern flight test safety. Neither are there
any prospects of there being any.
In the Civil Domain both FAA and EASA are recognised certifying authorities representing the US and Europe
respectively and offer codes that are widely used across The World. Both codes are utilised internationally and
do have elements governing the safe flight testing of the product. A number of the Case Studies describe
compliance with both FAA and EASA requirements.
In the Military Domain there are no similar internationally recognised codes. There is a move for a common
Military Airworthiness Approach in Europe which would cover flight test in a similar manner to EASA but
at the time of writing, 2019, this is regarded as some way off.
There are internationally recognised standards that apply to Quality Management Systems and Safety
Management, e.g., ISO 9000, that will drive how the associated Quality Management System is put together
and for Health and Safety Management ISO 45001 but these are more about quality system content,
development and assurance. Flight Test Safety becomes an element covered but is not specifically identified.
All of the Case Studies describe ISO 9000 compliant systems.
3.1.3 National
The majority of the Case Studies describe Safety Management Systems that have been designed to be
compliant with National Military Requirements, the exceptions being the Research Establishments, NASA
and NLR. Again, there is no universally recognised approach but there are significant similarities in
the responses described in the Case Studies.
So, whilst there is no common requirement there appears to be a shared generic approach across
the organisations represented. Hence in developing a new system there is practice which can inform the system
developer of elements that be incorporated to satisfy the Local Regulator. For the US MIL Standard 882
the US DoD Standard Practice for System Safety describes the generic approach to be taken.
STO-AG-300-V32 3-1
OBSERVATIONS AND DISCUSSION
The Case Studies identify a number of different approaches. That offered by NLR is judged to be particularly
complete and is recommended.
• It is a standalone document that can easily be placed where it can be seen.
• It is signed and endorsed by Senior Management.
• It declares in simple terms:
• NLR’s higher level key safety statement;
• The responsibility of the Accountable Manager; and
• The Organisation’s commitments and principles in relation to delivering Flight Test Safety.
3.1.5 Internal
Each of the Case Studies describes a Flight Test Safety Management System that has been designed to satisfy
the specific Higher Level Requirements and the Safety Policy pertinent to that organisation and produced
in accordance with the Quality Management Standards of the organisation. Each is therefore bespoke and
specific to the particular organisation. Despite this there is significant commonality at the higher level in
the systems described despite their diverse national and domain origins. Each differs in detail but the basic
approaches appear consistent.
3.1.6 Project
None of the Case Studies refer to Project Specific approaches. Indeed, a number go so far as to state that
a common approach will be adopted across all projects in order to enhance safety through the use of the same
processes, tools and facilities. This is highly recommended.
3.2 PEOPLE
3.2.1 Introduction
The most significant elements in any Flight Test Activity are the individuals who deliver the programme and
ensure that the flight testing is planned and executed in a safe and effective way. Each of the Case Studies to
a greater or lesser degree recognises this and provides detail of the specific approaches of the various
organisations. A number of themes emerge as do different approaches to satisfying resulting emergent
requirements. Above all things it is the way in which these individuals operate within the organisation and
against its processes that establishes the approach to safety through the organisation’s Safety Culture. Without
an embedded high level Safety Culture which survives under adverse pressures and influences safe flight
testing is highly likely to be seriously compromised. If there are any doubts about this a study of many
flight test accidents will clearly demonstrate what can happen when Safety Cultures fail.
3-2 STO-AG-300-V32
OBSERVATIONS AND DISCUSSION
the flight test programme to achieve its objectives with the need to deliver the flight testing safely. They, above
all others in the team, have the responsibility for resisting those programme pressures that demand too much
and begin to compromise safety.
3.2.3 Organisation
The Case Studies all provide an overview of the organisational approaches taken to satisfy the requirement to
deliver safe and effective flight testing. They describe the elements of the organisation and the ways in which
they come together and interact. For most organisations the elements described comprise:
• Management – Usually led by the Accountable Person.
• Flight Operations – Test Pilots, Aircrew, Airfield Management and Air Traffic Control.
• Flight Test Engineering – Detailed test planning, analysis and reporting.
• Flight Test Instrumentation – Instrumentation design, installation and support.
• Flight Test Data Processing – Real Time and Post Flight data replay and analysis.
• Maintenance Organisation – Aircraft Maintenance and Servicing.
Principal amongst these in relation to Flight Test Safety is the way in which the organisation addresses
the interactions which are necessary to manage the general flight test process and the test safety management
process. Here the elements must naturally work together in a co-operative way with the major interface being
between Management, Flight Operations and Flight Test Engineering. All of the Case Studies describe how
this is done within the subject organisations.
A number of the Case Studies refer to documents that describe the organisation and how it works,
its composition and what roles they undertake. This is recommended, especially in large organisations,
as a means of clearly identifying accountability chains and chains of command.
The Flight Test Organisation should have a level of independence from the project that leaves it free to raise
safety issues without project interference.
3.2.4 Resourcing
It is the role of Management to ensure that adequate numbers of trained and competent staff are available
to deliver the flight test programme. Failure to do this will result in an organisation that is stretched in its ability
to deliver. Against this pressure the focus is generally one which concentrates on delivery and the attention
drifts away from safety. During estimating and the planning of the resource, safety management requirements
must be an embedded element of the calculations. It needs to be planned in at the start to ensure adequate
resource. It must never be an afterthought.
STO-AG-300-V32 3-3
OBSERVATIONS AND DISCUSSION
Within the Military Organisations this is generally captured in the Chain of Command and Approvals are
granted by the relevant National Military Authority.
For Civil Organisations Approvals are more likely to come from the governing National Airworthiness
Authority. FAA, EASA, etc.
This covers both the operational requirements of the job and the safety involved requirements of the job. These
requirements should be clearly described in the job description topped up where necessary with a formal
statement of additional specific requirements relating to a particular task.
Management are responsible for ensuring a system is in place that not only captures the requirements of the job
but also demonstrates that individuals are competent to satisfy those requirements. This acts as a prompt for
training for those undergoing development and as a tool for demonstrating that and individual can be regarded
as SQEP. A number of the Case Studies describe the approaches taken.
Supported by this evidence the Accountable Manager is able to grant Approvals to staff who hold significant
Safety Responsibilities that are traceable and relate to accepted standards.
3.2.8 Competency
Competency is a measure of someone’s ability to perform a task and in the field of flight testing it is essential
that the activity is only conducted by competent people who are able to deliver not only effective testing but
also safe testing. Within a Flight Test Organisation, you will need people at differing competency levels
to deliver the business. It is the role of management to ensure that adequate numbers of staff are available with
the right skills.
For the purpose of this assessment we shall use the following. Competency in any field is the summation of:
• Aptitude – an individual’s natural ability to do something;
• Education – in a technical subject relevant to the field and at an appropriate level;
• Training – in the specifics of the subject domain on and off the job;
• Experience – of working in the subject domain gaining practical exposure to all its fields; and
• Currency – Exercising the skill set regularly so that it is second nature.
3-4 STO-AG-300-V32
OBSERVATIONS AND DISCUSSION
It is generally accepted that it takes 10000 hours to become expert in any field of endeavour. Assuming
1600 effective hours per year on the job this equates to some six years from starting a flight test career to being
judged truly competent in one’s technical specialism. Management experience typically comes later and
is additional to this. Intensive training can accelerate this but that means time away from the workplace gaining
experience and flight test is a field where experience counts a great deal. It is in truth a skill which is largely
acquired through experience.
In relation to the initial judgement of aptitude this is an outcome of the recruitment process where
the interviewer makes a judgement based on perception. The true assessment of aptitude comes on the job and
judgement by peers normally suffices to identify those with special aptitude who are likely to develop to higher
levels of competency.
Similarly, initial education to technician, engineer or piloting skill level is identified as part of the recruitment
process and is a matter of record.
Training is an area where different organisations have varied approaches to developing skills and the responses
identify some differences. SAAB is judged unique in establishing an internal training scheme for both pilots
and engineers that leads to a recognised qualification level within the company. Other organisations have
tended towards training on the job by giving exposure at increasingly more complex levels as the career
progresses. This approach is only possible in an organisation which can provide such exposure. With
the reduction in the number of active programmes this approach is becoming more difficult as opportunities
become fewer. This has caused a number of organisations to challenge this historic approach and to engage in
more structured training akin to the SAAB Model. Clearly the ultimate solution could be Test Pilot School
training but this can only be a solution for the few because of the costs and therefore is likely to be targeted
at candidates who are displaying high levels of aptitude. With the regulatory shift towards qualification routes
based on Test Pilot School training as a measure of competence, especially in Europe, this will prove
a challenge. There is a view that such training doesn’t necessarily make you better but gets you to the end
point faster.
Experience comes with time on the job and structured exposure to the different parts of the flight test process.
It is the classical way of developing flight test competency. The management team should carefully consider
the staff and ensure that they gain appropriate experience targeting this, if necessary, at those with the greatest
aptitude who are likely to rise to positions of responsibility.
Currency must be maintained if one is to be effective and especially so if one is to test safely. Where an
organisation does not maintain its currency and that of its staff it must seriously consider how it approaches
the flight test task. Recognising and accepting a currency gap is becoming an issue in some organisations
where workloads are falling and where new products are not passing through the system. In these instances,
requalification of the team and the individuals is required when a tasking that has not been performed recently
arises. Revalidation of currency may be achieved via a number of routes, through targeted formal training,
cross project working, planned flying work-up or synthetic trials exercises (i.e., simulator linked to flight test
telemetry). Regardless of the method consideration should be given to ‘team requalification’ as well as
individuals, in order to maximise benefit.
These criteria give a framework against which individual competencies can be judged in determining who are
Suitably Qualified and Experienced People, SQEP. These can then be used to support Role Approvals and
applications for Accountable Persons using formally recorded levels of experience and competence.
STO-AG-300-V32 3-5
OBSERVATIONS AND DISCUSSION
and to bring this to the attention of the management stream. To support this management must create an
environment where team members have no fear of sanction when they raise such issues and must be seen to
act effectively when such issues are raised.
In the extreme, where there is judged to be a safety issue which is imminently likely to prove hazardous, any
team member should feel empowered to exercise THE NO VOTE and call a stop to testing. In doing this they
must not feel the threat of censure and must believe they will be supported by their management. The NAVAIR
Case Study has a representative list of circumstances where an individual should feel entitled to exercise their
NO VOTE and is recommended.
3.3 PROCESS
3.3.1 Overview
The Case Studies generally concentrate on this aspect of Flight Test Safety Management and provide a great
deal of individual detail. The approaches taken within the Case Studies are, at the higher level, broadly similar
across the organisations irrespective of their nationality or domain. This substantiates the originally held view
that there was similarity of practice.
These are conducted in accordance with standard procedures and processes, using common facilities and tools
and managed and delivered by trained and approved competent staff.
3-6 STO-AG-300-V32
OBSERVATIONS AND DISCUSSION
The respondees typically describe a Generic Flight Test Safety Management Process which is run in parallel
with a Specific Flight Test Safety Control Process.
The Generic Process is applied to all trials and describes the process that is used to manage the Risk
Management Activity. This is very much about how the activity is assessed and monitored at a higher level.
Operating within this is a Specific Flight Test Safety Control Process that is applied to individual trials which
analyses the trial in detail and is much more involved with what is being undertaken and how it is being
controlled at the operational level.
A number of the respondees emphasise that the overall process is continuous starting at the design phase and
following the trial through to its completion. Constant supervision and revision is a feature of an effective
safety management process.
Throughout the overall process the responses reiterate the need for developed Safety Cultures and
the application of the NO VOTE without censure
Both of these processes map onto the phases of the Flight Test Lifecycle. The list below endeavours to capture
at, a high level, the activities contained in the Case Studies.
STO-AG-300-V32 3-7
OBSERVATIONS AND DISCUSSION
• Perform hazard analysis and establish the possible mitigations. Are these practical?
• Post mitigation establish the residual risk. What else can be done? Is this the final position? Is this ALARP
and Tolerable?
• Present the evidence to a Review Board. Support with evidence and present Risk Matrices. The risk
matrices presented in the Case Studies are of different formats. Notwithstanding this they all do exactly
the same thing and allow a residual risk level (High – Medium – Low) to be ascribed based on assessment
of severity and probability.
• Board membership must be appropriate to the perceived risk. Higher risk assessments need increasingly
more senior and experienced attendees. Involve junior staff; they need to learn.
• Approval is granted from the Board. For higher risk activities the Board Recommendation may well need
higher level approval within the organisation.
• The Approved Flight Test Plan must reflect the outcome of the Board and have sections clearly capturing
the safety related findings, notably risks and mitigations.
• Ensure a mishap plan is in place and has been exercised and advertised before conducting the trial.
3.3.5.3 Execution
• The Test Cards must only contain tests from an Approved Flight Test Plan.
• Safety advice must be included against the relevant test points.
• Every Test Flight should have a Briefing involving all of the participants. It is recognised that for large
multi-platform trials this may not be possible and that it may have to be a representative population.
• All participants must understand the purpose and conduct of the flight and their role in supporting it.
• At least one person present at the brief must be someone who is not part of the flight crew performing
the duty.
• The Brief should be formally assessed for effectiveness by an independent witness who judges whether
the brief has been complete and thorough and that there are no uncertainties. If the witness is not satisfied
with the quality of the brief they have the authority to suspend the sortie.
• The Brief must consider the conditions of the day. Are they acceptable?
• The Brief may make minor changes to the Test Cards in response to operational circumstances
e.g., weather. This can only be done by approved people in accordance with an approved procedure.
No Freelancing, no unapproved test points.
• Safety of the test flight can be enhanced by appropriate support measures:
• Team workup using Ground Environments. e.g., Simulator connected to Telemetry.
• Rehearsal.
• Real Time Monitoring.
• Real Time Analysis.
• Work up to critical conditions and demonstrations.
• Chase Aircraft – The NAVAIR Case Study has an appendix discussing the use of chase aircraft
(see Appendix 2-8.9).
3-8 STO-AG-300-V32
OBSERVATIONS AND DISCUSSION
• Safety related arisings and issues will take priority and will be resolved before continuing testing.
3.3.5.4 Analyse
• Initial analysis must prioritise safety related assessments.
• Where a safety related issue exists analysis must be satisfactorily completed before further testing
is performed.
3.3.5.5 Report
• Safety reporting will take priority at all times.
• A system must be in place that escalates safety issues to the appropriate part of the organisation so that
they can be managed and addressed.
3.4.1 Overview
As originally envisioned this section was to capture those Facilities and Tools specifically developed and
targeted at managing the Flight Test Safety Process. The Case Studies do not identify any systems of this type.
They do describe generic tools e.g., Bow Tie, Hazard Analysis but do not identify special tools.
STO-AG-300-V32 3-9
OBSERVATIONS AND DISCUSSION
3 - 10 STO-AG-300-V32
Chapter 4 – CONCLUSIONS
4.1 OVERVIEW
This AGARDograph originated from the interest that the STO Flight Test Technical Team (FT3) had in Flight
Test Safety and Risk Management and the practices deployed by the NATO Nations. They wished
to substantiate a view that, despite the disparate origins of the NATO Nation’s Flight Test Organisations,
there was largely common practice in relation to the approached deployed.
It is judged from the Case Studies provided that this view is, in the main, substantiated, and that Practice is
Common.
No diversions from this common practice have been identified that indicate that there is any alternative
approach that could be judged to be a better way.
Whilst there are detailed differences the Case Studies demonstrate high levels of commonality.
4.1.2 People
It is recognised that the staff who deliver the Flight Test Product are the most important single element in
the overall system.
• There must be adequate numbers of trained and competent staff with clearly defined roles and
responsibilities who can deliver the product. Key is the ability to demonstrate that staff are SQEP
Suitably Qualified and Experienced People and can fulfil their roles effectively.
• They must operate in a defined organisation which has clear chains of delivery responsibility and
accountability in relation to safety.
• It is a management responsibility led by the Accountable Person to ensure that these criteria are
satisfied.
4.1.3 Process
The processes described in the Case Studies are broadly similar at high and intermediate level. They deviate
at the detailed level something that is to be expected. The studies provide example of practice that will inform
those who wish to challenge an existing system or develop a new one.
STO-AG-300-V32 4-1
CONCLUSIONS
An accident is traumatic and brings human, programme, financial and reputational costs that dwarf
those associated with effective flight test safety and risk management practices.
4-2 STO-AG-300-V32
Annex A – AFG-7900.3-001 “AIRWORTHINESS AND FLIGHT
SAFETY REVIEW, INDEPENDENT REVIEW, TECHNICAL
BRIEF AND MINI-TECH BRIEF”
STO-AG-300-V32 A-i
ANNEX A – AFG-7900.3-001
“AIRWORTHINESS AND FLIGHT SAFETY REVIEW,
INDEPENDENT REVIEW, TECHNICAL BRIEF AND MINI-TECH BRIEF”
A - ii STO-AG-300-V32
Armstrong Flight Research Center AFG-7900.3-001, Baseline-9
Edwards, California 93523 Expires April 1, 2018
SUBJECT:
Airworthiness and Flight Safety Review,
Independent Review, Technical Brief, and Mini-
Tech Brief
RESPONSIBLE OFFICE:
Office of the Director
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A-1
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 2 of 36
CONTENTS
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A-2 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 3 of 36
This document presents the information needed to maximize the effectiveness of the
airworthiness and flight safety review processes as practiced at the Dryden Flight
Research Center.
Responsibilities of the Airworthiness and Flight Safety Review Board
(AFSRB), the AFSRB Chairperson, and the Dryden Flight Readiness
Review (DFRR) Committee when one is formed.
A sample DFRR outline as a guide for the DFRR Chairperson’s
consideration during the review process.
Items that should be covered in the DFRR Committee report to the AFSRB.
Technical Brief and Mini-Tech Brief guidelines.
The DFRC Center Director appoints the chairperson and the members of the AFSRB.
The AFSRB members are the line organizational Directors, Associate Director for
Programs, the Chief Pilot, the Chief of Safety and Mission Assurance, and ex-Officio
members. Other U.S. Government personnel may be appointed to the AFSRB as
necessary to provide a thorough review.
In order to implement the assigned task, the AFSRB is given the authority and
responsibility to perform reviews. The AFSRB Chairperson, in consultation with the
AFSRB members and the Project under consideration, determine the appropriate level
of review to be performed. There are four levels of review that will vary depending upon
the complexity and the criticality of the project.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A-3
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 4 of 36
The first, though least extensive, level of the AFSRB review is that
conducted solely by the AFSRB Chairperson. The chairperson is
responsible for determining whether a specific project need be reviewed in
any further depth or by any committee. If project plans and preparations
are adequate for performance of their proposed operation with the
necessary level of safety, the chairperson has the authority to cease
reviews at that point. This will be documented in an approved-to-proceed
memo.
The second level of review is one step beyond the sole review of the
AFSRB Chairperson. If the chairperson decides that a specific project
needs further review but does not require the full airworthiness board
review, the chairperson may convene a small team of Dryden experts,
independent of the project, to assist in determining whether the proposed
project is cleared for flight. If the chairperson and the small team agree
that the project should be cleared for flight, this will be documented in an
approved-to-proceed memo.
The third level of review is to have the plans and proposed conduct of the
project presented to the entire AFSRB for review. In this case, the entire
board will make a judgment as to whether a particular project has
adequately considered and integrated flight safety into its proposed plans.
This determination will be based upon a presentation to the AFSRB by the
project. The recommendation of the board to the Center Director will be
based upon the general agreement of the members, with each major
objection addressed and resolved or a minority report included with the
recommendation. A quorum consists of the chairperson and
representatives of Codes M, O, R, S, and XP.
The fourth level of review is to have the plans and proposed conduct of
the project presented to the AFSRB by a team of experts, independent of
the project, to determine whether the proposed project is cleared for flight.
This team is called a Dryden Flight Readiness Review (DFRR)
Committee. The entire committee will render a judgment as to whether a
particular project has adequately considered and integrated flight safety
into its proposed plans. The findings and recommendations of the DFRR
team are typically presented to the AFSRB by the DFRR chairperson.
The recommendation of the AFSRB to the Center Director will be based
on the general agreement of the members, with each major objection
addressed and resolved or a minority report included with the
recommendation. A quorum consists of the chairperson and
representatives of Codes M, O, R, S, and XP.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A-4 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 5 of 36
In any of the four review types, the AFSRB Chairperson has the authority
to obtain assistance from any part of Dryden or any outside help that may
be necessary to ensure that the project will be conducted in the safest
manner possible. This assistance can take many forms, such as the
hiring of a consultant, using the aircraft manufacturer’s expertise, using
experts in various fields, or forming ad hoc committees to assess any or
all parts of the proposed program.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A-5
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 6 of 36
2) Verifying that the approved System Safety Plan has been followed and
that all analyses and results have been properly integrated into the
project’s planning and tracking documentation.
3) Ensuring that all identifiable risks have been identified, assessed, and
either adequately controlled or presented to the Center Management
as risks that must be accepted in order to conduct the program.
4) Providing engineering and technical recommendations to program
personnel throughout the life of the DFRR, while recognizing that it is
not a function of the DFRR to direct actual work effort.
5) Maintaining ongoing communication among DFRR members, program
personnel, DFRC management, and the AFSRB Chairperson.
6) Submitting a final report on Board activity, findings, and
recommendations to the AFSRB Chairperson.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A-6 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 7 of 36
Constant communication between the DFRR Board and the project team
can provide benefits in both directions. A concern or recommendation
voiced to the project team in a timely manner may allow the project to take
action without delaying the project. Likewise, the proposed action of the
project team, communicated to the DFRR Board in a timely manner, may
expose areas of confusion or misunderstanding on the part of either the
board or the project that could lead to unnecessary expenditure of
valuable time and/or resources.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A-7
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 8 of 36
The DFRR oral briefing to the AFSRB should include the material
presented in the written report. Typically, the DFRR chairperson and
DFRR board members will present the briefing. Project team members
should be present to answer very specific questions that may arise. Hard
copies of the oral presentation should be prepared and presented to the
Project Manager and AFSRB members 24 hours prior to the AFSRB
meeting.
Along with the presentation of the DFRR Board’s final report, the Project
Manager of the affected project will submit a report to the AFSRB
chairperson addressing any open action items or recommendations that
may have been in the DFRR report that require action before the first flight
or significant project operation. Following these two report submissions,
the AFSRB will make final recommendations as to whether the project
should be allowed to continue on the planned course or should undergo
some plan modification before continuing.
The Technical Briefing, or Tech Brief, is one of the more important tools
used by Dryden to ensure the safe and efficient conduct of the flight test
mission. Its major function is to continue the review process after the
AFSRB has made its final recommendations and a program moves into
the flight or test phase.
There are two primary purposes for holding Tech Briefs. First, the
individual Project Office is given the opportunity to present its goals and
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A-8 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 9 of 36
A Tech Brief may be held for a block of flights of a research aircraft. This
is typically allowed for more well-established research projects if the flights
being conducted contain similar maneuvers and are deemed to be low risk
by the Project and Chief Engineer. This can allow the project to proceed
through the flight test plan more efficiently and present a more complete
picture of the flight test results. A Tech Brief may be called for mid-block if
unexpected results encountered or flight test plan changes are proposed.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A-9
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 10 of 36
C. Flight Plan
The planned approach to obtaining the data maneuvers should be
explained with emphasis on the technique and rationale for using it.
Any risks, limits, or constraints on the aircraft or maneuvering should
be presented and clearly explained with no assumptions made as to
understanding of these critical areas. Preplanned alternatives should
be presented to allow for unforeseen contingencies that may occur
during flight. This plan should cover the entire flight period from takeoff
to landing and give a clear and concise understanding of the pilot's
duties at all times. If there is to be a period of pilot familiarization
during flight, that should be briefed at the Tech Brief. This is not meant
to limit the pilot's freedom, but to constrain all research aircraft flying to
activity that has been preplanned and briefed.
D. Configuration Changes
A brief review should be made of the configuration that the aircraft will
be in for flight. This is particularly important where there has been a
change made to the aircraft between flights, no matter how small or
seemingly unimportant. The status of the configuration documentation
and waivers should be briefed to verify the completion of the changes
or to identify any incomplete work and its effect on the proposed flight
test. Additional risks perceived to have been incurred because of the
changes are to be briefed in the Tech Brief.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 10 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 11 of 36
explained and justified. This list often takes the form of a Hazard
Action Matrix that shows risk to human safety and risk to assets
supported with a table of risk title, category and probability, causes,
and mitigations.
G. Mandatory Requirements
Every flight of a research aircraft will have a specific set of conditions,
personnel, instrumentation, and equipment required in order to conduct
the flight as planned. These lists must be presented at the Tech Brief
along with the action to be taken in the event a condition is not met or
a person or item is not present or not operating. These could include
cancellation, flight abort, or deletion of a specific maneuver or series of
tests, but the goal is that all possibilities will be given detailed
consideration in advance of the mission and precise alternatives
planned and prepared for. These lists often take the form of the
following:
1) Mission Rules – Required facilities, test systems, and their
constraints
2) Aircraft Operating Limitations – Test specific aircraft system or
maneuver limits
3) Weather Constraints – Test specific weather related limits or
constraints
4) Go/No-Go Instrumentation – Safety or Mission critical
instrumentation required to conduct the mission or specific portions
of the mission
5) Required Documentation – Specific documentation, checklists, or
procedures for the mission
6) Required Personnel – In addition to control room and flight crew,
any ground crew required to conduct the operation.
H. Open Items
Occasionally, items may represent a major problem area and the
Project is delayed until the items can be closed out satisfactorily. More
often, the items are less severe and simply lack the necessary
information at the time of the Tech Brief. These may normally be
carried forward and closed out with the DFRC Chief Engineer or at the
Crew Brief before the Project is cleared to proceed.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 11
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 12 of 36
It is desirable for DFRR Board members to attend the first Tech Brief after
their report to the AFSRB to ensure that actions directed by the AFSRB
have been complied with by the Project. It is the responsibility of the
person chairing the DFRR to notify the members regarding the Tech Brief.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 12 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 13 of 36
Any of these rules may be altered to fit a special case through negotiation
with the Chief Engineer's Office. One example of a rule change that is
permitted is the "Block Tech Brief," where a series of flights is briefed
collectively. This would also include aerial refueling of a research aircraft
where "one" flight is, in effect, two or three normal ones.
Although block briefing is often allowed, there is good reason and benefit
from having the Project take the necessary time between flights to analyze
data before proceeding with the flight program. This is especially true
where an envelope is being expanded and data maneuvers proposed for a
flight are highly dependent upon results from a previous flight. The usual
technique is to expand the envelope on the first flight of a series and then
use the remaining flights to fill in data points, or to expand an envelope in
a different disciplinary area. A Tech Brief is then conducted before further
expansion takes place.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 13
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 14 of 36
immediately prior to the Crew Brief for most block-briefed flights, after the
first flight.
The final decision on what will or will not be allowed for any given project
remains a decision to be made by the DFRC Chief Engineer, a decision
based on what will facilitate the safest and most efficient flight test
program possible.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 14 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 15 of 36
The objectives of a review are to establish that all interfaces are compatible and
function as expected, confirm that the system and support elements are properly
configured and ready for flight, and receive assurance that flight operations can proceed
with acceptable risk.
This checklist provides a partial list of items to address for review team guidance when
conducting an independent review. The team may select only those items that apply to
the project reviewed. The list draws heavily from the Mars Climate Orbiter investigation.
1.0 Personnel
A. Leadership
1) Emphasis on safety as the primary concern
2) Experience level of personnel
3) Clear line of authority to person in charge
4) Examine team working and external interfaces
5) Teamwork promotion
6) Training opportunities provided
7) Mentoring of new or inexperienced personnel
C. Communication
1) Ranking of safety and mission success over cost and schedule
2) Free exchange of information, opportunity to be heard
3) Tracking of top ranked issues and their resolution to everyone’s
satisfaction
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 15
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 16 of 36
D. Project Team
1) Key positions filled and continuity encouraged
2) Experience level of team members
3) Adequacy of project team’s reviews: PDR, CDR, Wind tunnel, test
readiness, simulation
4) Customer involvement in decision-making and trade-offs
5) Team acceptance of external ideas
6) Team metrics relation to requirements
B. Requirements
1) Mission success criteria established and baselined
2) Requirements level sufficiently detailed
3) Change process used and effective
4) Derived requirements flow from base requirements
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 16 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 17 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 17
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 18 of 36
G. Independent Reviews
1) Review conducted by technical peers or experts
2) Sustained support for review members
3) Review independence from common management
4) Review results reported to top management
H. Operations
1) Contingency planning validated and tested (simulated)
2) Contingency training of personnel
3) Mission rules formulation and reasonableness
4) Telemetry and health monitoring during critical operations
I. Center Infrastructure
1) Senior management mechanisms for visibility into the project
2) Line organization accountability
J. Documentation
1) Documentation of design decisions and limitations
2) Decisions communicated to all concerned
3) Documentation process must be continuous
4) Electronic documentation distribution availability
L. Mission Assurance
1) Adequate mission assurance staffing
2) Mission success processes in place and followed
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 18 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 19 of 36
3.0 Technology
Technology adequately matured
Technology solutions alternatives considered
Risk level of new technology
New technology use and limitations
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 19
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 20 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 20 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 21 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 21
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 22 of 36
o Operational restrictions
Operations
o Checklists
o Emergency procedures
o Fact Sheet
o Manuals
Parachutes, vehicle
o Construction
o Pyrotechnics, mortar
Pilot training (ground and flight)
Project overview
o Experiments planned
o Facilities required
o Hardware, software
o Objectives
o Procedures used
Propulsion
o Launch vehicle
o Research vehicle
Range requirements
Range safety
o Abort landing sites
o Beacons
o Command destruct system
o Encryption
o Expected casualty calculations
o Flight termination system
o Operating area
o Trajectory
Recommendations by the Review Board
o Action Items
Research vehicle
o Vehicle purge
o Landing gear
o Mass properties
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 22 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 23 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 23
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 24 of 36
1. Can the type and amount of power available support the electrical
requirements of the installations?
2. Have operating procedures and an inspection checklist been developed for the
installation?
3. Is cooling air adequate to properly cool avionics in flight and on the ground?
4. Have partial flight manuals and checklist been prepared and approved?
5. Have weight and balance figures been computed and are they within
recommended limits?
6. Does the installation of test equipment in the aircraft interior keep aisles and
emergency exits clear for evacuation?
7. Do installed racks and test equipment have projections (bolts, rivets, knobs,
handles) that could cause injury to aircrew personnel?
9. Is the aircraft properly placarded and has the test instrumentation in the cockpit
been properly identified and marked?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 24 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 25 of 36
11. Have magnetic interference (EMI) ramifications been considered? Will flight
day EMI be different from other days?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 25
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 26 of 36
INSTRUMENTATION
1. Has the proposed and/or completed installation been inspected by the project
test aircrew to ensure that it offers the safest possible installation? Has a
cockpit safety design board approved the changes and documented approval?
3. Are the instrumentation appendages (nose boom pitot head, vanes, etc.)
ahead of the engine checked regularly for structural integrity?
5. Have provisions been made for coordinating the data when more than one
recording device is to be used?
8. Are you reasonable certain that this test can be conducted safely?
10. Are black boxes instrumented to reveal elapsed operating hours? On/off
cycles? Are hours and cycles frequently monitored and documented?
11. Are film/tape time limits on recorders and cameras understood? Speeds?
Initiation and shutoff times?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 26 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 27 of 36
MAINTENANCE
1. Are there any special maintenance procedures that will be required to support
the test? Are they published as a requirement?
3. Have the aircraft and, in particular, the modification areas, been thoroughly
inspected for foreign objects?
4. Have closeout photos been taken of areas that are difficult to access or of all
areas for vehicles that are to be unrecoverable post flight?
6. Are you reasonably certain that the test can be conducted safely?
3. Are you satisfied with the limits and accuracy of the monitored parameters?
With interfaces with other monitored parameters?
4. Have you checked scaling and sensing (direction) of the parameters you are to
monitor?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 27
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 28 of 36
6. Are flight envelope limits clearly defined and understood before flight by
necessary persons?
AERODYNAMICS
1. Have all aspects of new design or modification been considered for effect on
Aerodynamics/drag
Acoustics/vibration
Surface shielding/ineffectiveness due to boundary layer/wake shedding
Weight
CG
Inertia
Exterior Configuration
Shock interaction
Control surface movements
Pitot-static system
Other instrumentation (hot wires, pressure sensors/taps, hot films, etc.)
Etc.
5. Is instrumentation and its calibration satisfactory? Does it tell you all you need
to know for safety and mission accomplishment? What are the shortcomings?
6. Do you have any undue concerns about questions in the “Flight Control Room
Flight Ops” section of this document?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 28 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 29 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 29
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 30 of 36
AEROSTRUCTURES
1. Have all aspects of new design or modification been considered for effect on
structure and vice versa?
2. Are ground load and ground vibration tests adequate? Any evidence of
airframe vibration (flutter, buffet, acoustics)?
3. Is instrumentation satisfactory? Does it tell you all you need to know for safety
and mission accomplishment? What are the shortcomings?
4. Do you have any undue concerns about questions in the “Flight Control Room
Ops” section of this document?
5. Have all safety and mission concerns been adequately addressed? What
factor of safety in design or test? What Margin of Safety?
1. Have all “fail to operate” and full hardover impacts been assessed?
3. Have end-to-end tests been conducted on the full-up total system? Have all
credible inputs been accomplished to observe system response?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 30 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 31 of 36
9. Has consideration been given to using parallel-active dual systems rather than
primary-active, backup-benign systems?
11. Do you have any undue concerns about questions in the “Flight Control Ops”
section of this document?
12. Have all safety and mission concerns been adequately addressed? Has a
system safety assessment been accomplished?
1. Have all aspects of new design or modification been considered for effect on
Dynamics and vice versa
Weight
CG
Inertia
Exterior configuration
Surface control movements
Pitot-static system
Other instrumentation
Etc.
4. Is instrumentation satisfactory? Does it tell you all you need know for safety
and mission accomplishment? What are the shortcomings?
5. Do you have any undue concerns about questions in the “Flight Control Ops”
section of this document?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 31
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 32 of 36
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 32 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 33 of 36
PROPULSION
6. Are flight control and electrical/hydraulic power adequate for power-off landing?
PROJECT MANAGEMENT
1. Have all the policies of Dryden Management System Manual (DMSM) been
addressed?
2. Have all project documents been completed at the appropriate life-cycle gates?
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 33
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 34 of 36
Forms
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 34 STO-AG-300-V32
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 35 of 36
APPENDIX D: Acronyms
AFSRB Airworthiness and Flight Safety Review Board
AIL Aircraft In the Loop
CDR Critical Design Review
CG Center of Gravity
DFRR Dryden Flight Readiness Review
DFFRB Dryden Flight Readiness Review Board
DMSM Dryden Management System Manual
DOM Dryden Organizational Manual
EMI Electromagnetic Interference
FADS Flush Air Data System
FMEA Failure Mode and Effects Analysis
FTA Fault Tolerance Analysis
GVT Ground Vibration Test
HIL Hardware In the Loop
HR Hazard Report
IV&V Independent Verification and Validation
MSR Mission Success Review
PDR Preliminary Design Review
PRA Probabilistic Risk Analysis
RPV Remotely Piloted Vehicle
SMI Structural Mode Interaction
UAV Uninhabited Aerial Vehicle
V&V Verification & Validation
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 A - 35
Airworthiness and Flight Safety Review, AFG-7900.3-001, Baseline-9
Independent Review, Technical Brief, Expires April 1, 2018
and Mini-Tech Brief Page 36 of 36
Baseline, 10-15-10
Replaces DHB-X-001
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
A - 36 STO-AG-300-V32
Annex B – AFOP-7900.3-023 “AIRWORTHINESS
AND FLIGHT SAFETY REVIEW PROCESS”
STO-AG-300-V32 B-i
ANNEX B – AFOP-7900.3-023
“AIRWORTHINESS AND FLIGHT SAFETY REVIEW PROCESS”
B - ii STO-AG-300-V32
Armstrong Flight Research Center AFOP-7900.3-023, Revision F-8
Edwards, California 93523 Expires June 1, 2018
Compliance is mandatory.
SUBJECT:
Airworthiness & Flight Safety Review Process
RESPONSIBLE OFFICE:
Office of the Chief Engineer
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B-1
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 2 of 13
CONTENTS
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B-2 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 3 of 13
This process applies to all flight activities and hazardous ground tests
involving aircraft, critical flight systems, and/or experimental facilities for
which the Center has any airworthiness, ground, flight or range safety
responsibility or that involve Center personnel utilizing non-NASA assets.
2.2 Applicability
2.3 Waiver
This procedure may be waived by the Center Director or the Center Chief
Engineer for projects conducted jointly with organizations having a Center
recognized safety review process (for example, other NASA Centers, the
Air Force Flight Test Center, United States Navy test organizations, etc.)
or where the safety review process is covered in a Memorandum of
Agreement (MOA). Typically, the Center is represented in safety review
processes conducted by other organizations for joint projects. An
informational briefing may be requested by the Chief Engineer to inform
Center management of the project content, test plan, and hazard analysis.
The Center accepted risks must be approved by the Center Director or
NASA HQ, as appropriate.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B-3
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 4 of 13
Objective: Ensure that test and airborne science research projects comply with
Center policies and procedures.
Target: All Center policies and procedures followed.
Metric: Briefings in accordance with applicable policies and procedures.
Objective: Review flight activities and tests involving all aircraft, critical flight
systems, and experimental facilities.
Target: Provide required technical and safety information during briefings.
Metric: Briefings in accordance with applicable policies and procedures.
4.0 FLOWCHART
The AFRR can be required independent of the AFSRB, and is frequently done this way
for less technically intensive projects such as the SBLT pylon on the F-15B. The AFRR
will write a report to the Chief Engineer and attend the tech brief in lieu of the AFSRB to
answer questions from the tech brief committee.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B-4 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 5 of 13
Project Manager /
Mission Manager
Airworthiness and Flight
START Safety Review Board
(AFSRB) Chair
3
1
Review and evaluate project plans
Develop or redevelop and present
and preparations for readiness to
project plans and preparations to
fly.
AFSRB Chair.
Note B. (See Note C for
Note A
non-NASA aircraft.)
4 Yes
AFSRB
required?
No
Yes 5
To block 17 Recommend
approval to
fly?
No
6
Inform project manager
2 Yes
Can issues 7
be resolved? Provide guidance.
No
8
END Yes DFRR
required?
Notes D & E
No
9
Convene AFSRB.
Note F
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B-5
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 6 of 13
AFSRB
12
16
No Recommend Review and evaluate flight plans
approval to and preparations for readiness to
fly? fly.
Yes
13 From block 5
From block 20 Write and sign recommendation
memo and forward to Center
Director for approval.
Center Director
14 No 17
Inform project manager. Approved?
Project Manager/
Mission Manager Yes
18
Sign and forward copy to
project manager or mission
10 manager.
Yes 15
Can issues
be Provide guidance.
resolved?
No
END To block 9
11
File original approval directive
memo with Center Director's
Office, file copy in project file, and
forward copy to each AFSRB
member.
END
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B-6 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 7 of 13
From block 8
DFRR Committee
23
Use G-7900.3-001 as guidance
19 for investigating all matters that
Select formal DFRR committee affect airworthiness and safety.
with advice of AFSRB members.
Notes D & E
24
Report findings and
recommendations in writing.
Note G
AFSRB
No 20 Yes 25 No
To block 13 Open action Recommend
items? approval?
Yes
21
Write and sign recommendation
memo subject to tech brief
closeout of action items, and
forward to Center Director for
approval. Center Director
No 26
22 Approved?
Provide guidance.
Yes
27
Sign and forward copy to
project manager / mission
manager.
To block 30
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B-7
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 8 of 13
Project Manager/
Mission Manager
From block 27
Airworthiness AFSRB
and Flight Safety
Review Board 35
28 (AFSRB) Chair Address closure of open action
Respond to open action items and items at tech brief per
recommendations. DCP-X-008.
Note H
31
Open items No
closed out
at tech
brief?
29 Yes
File original recommendation
memo with Center Director's 32
Office, file copy in project file, and Formally endorse flight request.
forward copy to each member of
AFSRB.
END
33
Inform project manager.
30 Yes
Can issues 34
be Provide guidance.
resolved?
No
END
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B-8 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 9 of 13
Note B
The Center Chief Engineer serves as the AFSRB Chair, and the Center Deputy Chief
Engineer serves as the alternate chair.
In accordance with AFFTCI 91-105, the Chief Engineer will notify AFFTC/SET of any
potentially high risk test activity or any test activity that will affect normal AFFTC
operations prior to the start of testing. AFFTC/SET participation in the AFSRB
accomplishes this requirement.
In accordance with AFFTC 91-105, joint projects that involve AFFTC assets (other than
airspace, range, and airfield support of normal flight operations) require AFFTC review
and approval. AFFTC/SET participation in the AFSRB may meet this requirement. The
AFFTC Form 5028 cover sheet, as a minimum, is used by AFFTC/SET to document the
process and gain approval for AFFTC participation. Projects may provide AFFTC/SET
with the AFSRB briefing materials to attach to the AFFTC Form 5028 to provide
additional information to the AFFTC approval chain of command. Projects should plan
for the additional coordination time required for AFFTC approval.
Note C
Projects and tests that involve NASA assets (flight crew and/or high value equipment)
and are flown on non-NASA aircraft will follow guidelines and policy set forth in
NPD 7900.4, NASA Aircraft Operations Management, and NPR 7900.3. Additionally,
the requirements of AFOP-7900.3-006, Chapter 10, Joint Flight Operations, will be
followed when applicable.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B-9
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 10 of 13
Note D
Criteria for chartering Armstrong Flight Readiness Review (AFRR)
A. New project or operation with assumptions of risk
B. Phased project requiring approval to enter succeeding phase
C. Project exceeding limit previously approved by AFSRB - After major
modification(s) to aircraft
Note E
AFRR composition depends upon vehicle, mission, and technologies involved.
AFRR chair is an expert senior technical person.
Note F
Membership of the AFSRB is appointed in by the Center Director and typically includes
the following:
1. Chief Engineer (Chair)
2. Associate Center Director for Programs
3. Director for Flight Operations
4. Director for Research & Engineering
5. Project Mission Director
6. Chief, Safety and Mission Assurance
7. Director, Mission Information and Test Systems
8. Chief Pilot
9. Aviation Safety Officer
A quorum consists of the Chair and representatives from Codes M, O, R, S, and XP.
Note G
A. The written report will be delivered to the AFSRB and the project manager at
least 48 hours prior to the AFSRB meeting.
B. Hard copies of the oral presentation will be made available to project manager
and AFSRB members 24 hours prior to the AFSRB. Copies of slides may be
made available using online resources available at the Center, such as email or
computer servers accessible by the AFSRB membership. At least five hard
copies are provided to the AFSRB Chair and membership at the briefing to
facilitate documentation of discussion items.
C. The final AFRR briefing to the AFSRB must precede the project’s technical
briefing by a minimum of three workdays. This time may be compressed with the
concurrence of the AFSRB Chair.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B - 10 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 11 of 13
Note H
Representatives of the AFRR Board will be present at the technical briefing in order to
concur on closures of any issues that were deferred to the tech brief.
See the Code 100 record types and records plan for storage location, retention
requirements, and ultimate disposition for the following management records associated
with this procedure
Records of the required meetings are filed in the Center Chief Engineer’s Office. An
electronic version will be filed by the project on the MARS server in the Center’s tech
brief folder under the appropriate project.
Records of required meetings are retained for the duration of a project. They are further
retained in the Center Chief Engineer’s Office at the discretion of the Chief Engineer to
provide guidance to future projects with similar technical or safety related challenges.
AFOP-7900.3-022 Tech Brief (T/B) & Mini Tech Brief (Mini T/B)
Baseline, 01-04-99
Revision A, 04-15-99
Modified signature block from "Approved" to "Electronically Approved by"
In all blocks containing the words "Approval Directive", the word "memo" was added
Modified block 4 of the "Project Manager/Mission Manager" on page 1 and block 3 on page 2
Modified block 1 of "FRR Committee" on page 2
Revision B, 10-03-00
Page 1 modified block 5 of "AFSRB Chair"and changed "Flight Readiness Review (FRR)" to "Dryden
Independent Review (DIR)" in block 6, added 2nd block to "Center Director", and modified block 4 of
"Project Manager/Mission Manager".
Page 2 modified blocks 3 & 6 of "AFSRB Chair", added 2nd block to the "Center Director", and
modified block 3 of "Project Manager/Mission Manager".
Page 3 added block 1 for "Center Director", modified block 2 of "Project Manager/Mission Manager",
and changed "FRR" to "DIR" in Notes 2 & 3.
Revision C, 01-27-03
Replaced "Dryden Independent Review (DIR)" with "Dryden Flight Readiness Review (DFRR)" in all
references.
Changed "Approval Directive" to "recommendation" in all references, including column title p. 2.
Moved Notes 1, 2 and 3 to within proximity of where each is referenced.
Additional minor edits throughout.
Revision D, 11-26-03
Note 3 that describes how the airworthiness process will apply to non-NSA aircraft
Revision E, 08-06-10
Extended expiration date by 6 months.
Revision F, 01-01-11
Updated format to current template.
Added required paragraphs.
Modified Section 3.0.
Minor revisions of procedures.
Moved flowchart notes to separate section.
Rebuilt flowchart.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
B - 12 STO-AG-300-V32
Airworthiness and Flight Safety Review Process AFOP-7900.3-023, Revision F-8
Expires June 1, 2018
Page 13 of 13
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
Center.
STO-AG-300-V32 B - 13
B - 14
Annex C – AFOP-7900.3-022 “TECH BRIEF AND MINI-TECH BRIEF”
STO-AG-300-V32 C-i
ANNEX C – AFOP-7900.3-022 “TECH BRIEF AND MINI-TECH BRIEF”
C - ii STO-AG-300-V32
Armstrong Flight Research Center AFOP-7900.3-022, Revision I-1
Edwards, California 93523 Expires March 1, 2021
Compliance is mandatory.
SUBJECT:
Tech Brief (T/B) & Mini Tech Brief (Mini T/B)
RESPONSIBLE OFFICE:
Office of the Chief Engineer
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C-1
C-2 STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 3 of 15
CONTENTS
1.0 PURPOSE OF DOCUMENT ................................................................................. 4
2.0 SCOPE, APPLICABILITY, & WAIVER .................................................................. 4
2.1. Scope ................................................................................................................. 4
2.1.1. Scope Exceptions ........................................................................................ 4
2.2. Applicability ........................................................................................................ 4
2.3. Waiver ................................................................................................................ 4
3.0 OBJECTIVES & METRICS .................................................................................... 5
4.0 ROLES & RESPONSIBILITIES ............................................................................. 5
5.0 T/B or MINI-T/B FLOWCHART .............................................................................. 6
6.0 FLOWCHART REMARKS ..................................................................................... 7
7.0 MANAGEMENT RECORDS & RECORDS RETENTION .................................... 11
8.0 RELEVANT DOCUMENTS ................................................................................. 11
8.1. Authority Documents ........................................................................................ 12
8.2. Referenced Documents ................................................................................... 12
8.3. Informational Documents ................................................................................. 12
8.4. Forms ............................................................................................................... 12
APPENDIX A, Definitions .............................................................................................. 12
APPENDIX B, Acronyms ............................................................................................... 13
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C-3
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 4 of 15
This procedure describes requirements for technical briefings (T/Bs) for senior level
management review of Armstrong Flight Research Center (AFRC) (hence forth referred
to as the Center) flight research projects as part of the overall Center airworthiness and
flight safety review process established in AFCP-X-009, Airworthiness and Flight Safety
Review Process. Technical briefings are used to:
A. Continue safety and technical review processes after the AFSRB has made final
recommendations and the flight program has moved into the flight phase;
B. Ensure project presents goals and plans for peer review;
C. Update current project goals, plans, and risks to the Center Management Team.
This procedure applies to all elevated risk ground tests and all flight
research flown at the Center, whether resident or deployed. It also covers
other flight research or test projects in which Center personnel participate
or research aircraft or other high value assets are used.
2.2. Applicability
This procedure applies to all project managers and T/B Board members.
2.3. Waiver
This procedure may be waived by the Center Director or the Center Chief
Engineer for projects conducted jointly with organizations having a Center-
recognized safety review process (for example, other National Aeronautics
and Space Administration (NASA) Centers, the United States Air Force or
United States Navy test organizations, etc.) or where the safety review
process is covered in a Memorandum of Agreement (MOA).
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C-4 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 5 of 15
The requirements of time between chart delivery and tech brief, and tech
brief and flight, may be waived upon consultation with the Tech Brief
Board Chair (Section 6.0, Remark A).
Tech Brief Board Members: Serve as signatories on the Flight Request. Review and
assess the project team’s plans and approach to accomplish their objectives safely.
Evaluate the project’s residual hazard stance and its acceptability to the Center.
Project Team: Present the project’s objectives, plans, and approach to accomplish the
project’s goals and objectives. Conduct hazard analysis and present results to the Tech
Brief Board. Coordinate meeting agenda with the Chief Engineer.
Center Director: Establish the Center’s airworthiness and flight safety review process.
Receives the recommendation of the Tech Brief Board. Evaluate and accept or reject
residual hazards that fall into the accepted risk category.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C-5
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 6 of 15
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C-6 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 7 of 15
Remark A
Technical briefings will be scheduled at least two working days in advance of the
proposed flight date.
Briefing materials will be distributed to mandatory attendees two days before briefing.
Hard copies of the presentation should be provided at the T/B to facilitate Board
member note taking and situational awareness.
One paper copy of the briefing material presented will be provided by the project to the
Tech Brief Board Chair for the record.
Remark B
Mini-T/Bs are typicallty less than 20 minutes and may be combined with the crew brief
to explain minor anomalies and to brief incremental flight plans.
One paper copy of the briefing material presented will be provided by the project to the
Tech Brief Board Chair for the record.
Remark C – Attendees
1. Mandatory Board Attendees or Designee
a) Center Chief Engineer (Chair)
b) Director for Research and Engineering
c) Director for Flight OperationsDirector for the Office of Safety & Mission
Assurance
d) Chief Pilot
e) Aviation Safety Officer (ASO)
f) Range Safety Officer (RSO) (UAS activities only)
g) Other Organization Representatives, for joint projects
2. Mandatory Project Attendees or Designee
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C-7
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 8 of 15
a) Project Manager
b) Project pilot
c) Project Chief Engineer (if assigned)
d) Project Operations Engineer (if assigned)
3. Desired Attendees
a) Principal investigator
b) Appropriate discipline representatives / technical monitors from Research
Engineering
c) Director for Mission Information and Test Systems
4. Director for Aerospace Projects
The Chief Pilot and Aviation Safety Officer (ASO) position may be filled by the qualified
Director or designee. The Range Safety Office is a required attendee only when
unmanned aerial system (UAS) operations are being conducted.
The Chief Engineer has the authority to augment the T/B mandatory attendee list, as
appropriate, to cover project areas where additional expertise is warranted.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C-8 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 9 of 15
The above checklist is tailored for typical research test missions. Airborne science
research missions and some research test missions may not include all of the elements
above, or may have additional elements that should be briefed. Use the checklist as a
guide for required topics to be presented.
With Chief Engineer approval, a T/B may cover a block of flights. Additional envelope
expansion requires an additional T/B.
Mini-T/Bs are used to address minor system anomalies, review flight test results, and
cover changes to briefed blocks of flights or for other purposes. Mini-T/Bs may be used
by management to provide increased supervision of more hazardous testing. There is
no briefing guide. Each meeting agenda is determined by the project manager in
consultation with the Chief Engineer and other Tech Brief Board members.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C-9
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 10 of 15
Mitigation types fall into one or more of the following categories: design, safety device,
procedure, training, or warning/placard.
For all accepted risks, provide very brief discussion on how residual risk levels were
derived (i.e., through what level(s) in the hierarchy of risk management: design, devices,
processes / procedures, training, placards).
Provide depiction of accepted risks and remaining residual hazards based on phase of
flight (if appropriate). This is generally done using the HAM (Note E).
Test point risk level will be assessed as HIGH, MEDIUM, or LOW. MEDIUM risk test
points will have an accompanying accepted risk requiring Center Director approval for
flight. HIGH risk test points will have an accompanying accepted risk requiring NASA
Headquarters approval for flight.
Provide assessment of probability of achieving technical objectives for flight / flight block
being briefed.
Multiple versions of the HAM (human safety and loss of asset) are presented. There
may be HAMs for each flight phase for complex, multi-phased flight activity.
Remark G
Sufficient paper copies should be provided at the T/B by the project for the mandatory
attendees at the briefing to facilitate note taking.
One paper copy of the briefing material presented at the T/B will be provided by the
project to the Tech Brief Board Chair for the record.
Remark H
Verification of closure may be provided via a mini-T/B or email to the Chair. The
method of reporting closure of action items will be provided to the project at the end of
the T/B by the Tech Brief Board Chair.
Remark I
Tech Brief Board approval refers to concurrence from the following directors (or
designated representative):
1. Director for Research and Engineering
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C - 10 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 11 of 15
At the conclusion of a mini-T/B, all required attendees will either sign the unsigned or
initial and date an already signed Flight Request (AFRC Form 70129).
Tech Brief and mini-Tech Brief records will be kept per the Code X record types and
records plan for the records associated with this procedure.
An electronic version of the presentation material will be filed by the project on the
MARS server in the Center’s Tech Brief folder under the appropriate project. The Flight
Request is kept as part of the Project’s records.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C - 11
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 12 of 15
8.4. Forms
APPENDIX A, Definitions
Can or may Denotes discretionary privilege or permission.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C - 12 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 13 of 15
APPENDIX B, Acronyms
AD Airworthiness Directive
AFCP Armstrong Flight centerwide procedure (temporary document acronym)
AFG Armstrong Flight guidance (document type)
AFPR Armstrong Flight procedural requirements (document type)
AFRC Armstrong Flight Research Center
AFSRB Airworthiness & Flight Safety Review Board
ASO Aviation Safety Officer
CM configuration management
HAM Hazard Action Matrix
MOA Memorandum of Agreement
NASA National Aeronautics and Space Administration
NF NASA form
NPR NASA procedural requirement
SB service bulletin
T/B Tech Brief
TCTO time compliance technical order
TD technical directive
UAS unmanned aerial system
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C - 13
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 14 of 15
Baseline, 01-04-11
Revision A, 04-15-99
Modified signature block from "Approved" to "Electronically Approved by"
Separated responsibilities of "Technical Support and Coordination Office" to "Flight Scheduling
Office" and "Project Administrative Support" on page 1 of flowchart.
Modified the last block of "Project Manager/Mission Manager" on page 2 of flowchart.
Page 2 of flowchart: Combined last two blocks of "Attendees" and moved the responsibility to
"Directors For".
Revision B, 11-17-99
Page 1: Revised Note 1 and Note 2.
Revision C, 05-05-00
Page 1: Deleted "Each mandatory attendee shall provide the Chief Engineer with a list of 3
acceptable alternates Chief Engineer will publish the list" from Note 2.
Revision D, 07-26-00
Page 1: Added "or DOP-Y-003" to block 2 of Project Manager / Mission Manager and modified Note
2.
Revision E, 10-02-00
Page 1 of flowchart: Added "(Chair of T/B Committee)" after "DFRC Chief Engineer" in Note 1and
modified Note 2.
Page 2 of flowchart: Changed "Directors For" to "Tech Brief Committee"
Modified Note 5.
Revision F, 07-31-01
Page 1: Added “or a designee” to Mandatory Attendees in Note 2.
Revision G, 02-14-06
Rebuilt flowchart
Removed reference to cancelled document
DCP-O-009
Inserted flowchart in Word template
Revision H, 02-26-10
Updated format to current template
Added required sections
Amplified checklist of briefing topics
Updated flowchart and added flowchart notes section to accommodate large number of notes
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
C - 14 Center. STO-AG-300-V32
Tech Brief (TB) & Mini Tech Brief (Mini TB) AFOP-7900.3-022, Revision I-1
Expires March 1, 2021
Page 15 of 15
Revision I, 03-04-16
Removed DFRC/Dryden references.
Updated to current format.
Added concurrence signatures.
Updated flowchart.
Updated flowchart notes.
Removed metrics.
Added authority documents.
Updated referenced documents.
Added informational documents.
Updated form numbers.
Before use, check the Master List to verify that this is the current version. For reference only when
printed. This document does not contain export-controlled content and may be distributed outside the
STO-AG-300-V32 Center. C - 15
STO-AG-300-V32 C - 16
Annex D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATIONAND FLIGHT
TEST TECHNIQUES SERIES
1. Volumes in the AGARD, RTO and STO Flight Test Instrumentation Series, AGARDograph 160
STO-AG-300-V32 D-1
ANNEX D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATION AND FLIGHT TEST TECHNIQUES SERIES
D-2 STO-AG-300-V32
ANNEX D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATION AND FLIGHT TEST TECHNIQUES SERIES
2. Volumes in the AGARD, RTO and STO Flight Test Techniques Series, AGARDograph 300
The remaining volumes are published as a sequence of Volume Numbers of AGARDograph 300.
STO-AG-300-V32 D-3
ANNEX D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATION AND FLIGHT TEST TECHNIQUES SERIES
‡
Superseded by Volume 28.
†
Volume 25 has been published as RTO AGARDograph AG-SCI-089.
D-4 STO-AG-300-V32
ANNEX D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATION AND FLIGHT TEST TECHNIQUES SERIES
STO-AG-300-V32 D-5
ANNEX D – AGARD, RTO AND STO FLIGHT TEST
INSTRUMENTATION AND FLIGHT TEST TECHNIQUES SERIES
D-6 STO-AG-300-V32
REPORT DOCUMENTATION PAGE
1. Recipient’s Reference 2. Originator’s References 3. Further Reference 4. Security Classification
of Document
STO-AG-300-V32 ISBN
AC/323(SCI-236)TP/964 978-92-837-2285-4 PUBLIC RELEASE
5. Originator
Science and Technology Organization
North Atlantic Treaty Organization
BP 25, F-92201 Neuilly-sur-Seine Cedex, France
6. Title
Flight Test Safety and Risk Management
7. Presented at/Sponsored by
This AGARDograph has been sponsored by the SCI Flight Test Technical Team
(FT3) of the Systems Concepts and Integration Panel (SCI) of STO.
8. Author(s)/Editor(s) 9. Date
Multiple 210
12. Distribution Statement
There are no restrictions on the distribution of this document.
Information about the availability of this and other STO
unclassified publications is given on the back cover.
13. Keywords/Descriptors
Flight Test Risk Management; Flight Test Safety; Flight Test Safety Management; Safety
Management Systems
14. Abstract
This AGARDograph investigates the current approaches across the NATO Nations in the field of
Flight Test Safety and Risk Management. It originates from a view held by the NATO STO Flight
Test Technical Team (FT3) that there was broadly common process and a wish to demonstrate this
through capturing examples of practice and analysing them. This became the mission of Task Group
SCI-236 and this AGARDograph records the findings. It offers examples of practice captured in a
broadly common format from a representative sample of Flight Test Organisations and an analysis
of their content. It concludes that the views of FT3 are substantiated and that at the baseline level
there is significant commonality of approach in relation to people, process, facilities and tools.
Detailed content is clearly different, but the similarities offer a framework that can be used to
support those embarking upon the creation of a Flight Test Safety Management System or desiring
to compare and contrast an existing one.
STO-AG-300-V32
STO-AG-300-V32
NORTH ATLANTIC TREATY ORGANIZATION SCIENCE AND TECHNOLOGY ORGANIZATION
Les demandes de documents STO, RTO ou AGARD doivent comporter la dénomination « STO », « RTO » ou « AGARD » selon le cas, suivie du
numéro de série (par exemple AGARD-AG-315). Des informations analogues, telles que le titre et la date de publication sont souhaitables. Des références
bibliographiques complètes ainsi que des résumés des publications STO, RTO et AGARD figurent dans le « NTIS Publications Database »
(http://www.ntis.gov).
NORTH ATLANTIC TREATY ORGANIZATION SCIENCE AND TECHNOLOGY ORGANIZATION
BP 25
DISTRIBUTION OF UNCLASSIFIED
F-92201 NEUILLY-SUR-SEINE CEDEX • FRANCE
Télécopie 0(1)55.61.22.99 • E-mail mailbox@cso.nato.int STO PUBLICATIONS
AGARD, RTO & STO publications are sometimes available from the National Distribution Centres listed below. If you wish to receive all STO
reports, or just those relating to one or more specific STO Panels, they may be willing to include you (or your Organisation) in their distribution.
STO, RTO and AGARD reports may also be purchased from the Sales Agencies listed below.
Requests for STO, RTO or AGARD documents should include the word ‘STO’, ‘RTO’ or ‘AGARD’, as appropriate, followed by the serial number.
Collateral information such as title and publication date is desirable.
If you wish to receive electronic notification of STO reports as they are published, please visit our website (http://www.sto.nato.int/) from where you
can register for this service.
Requests for STO, RTO or AGARD documents should include the word ‘STO’, ‘RTO’ or ‘AGARD’, as appropriate, followed by the serial number
(for example AGARD-AG-315). Collateral information such as title and publication date is desirable. Full bibliographical references and abstracts of
STO, RTO and AGARD publications are given in “NTIS Publications Database” (http://www.ntis.gov).
ISBN 978-92-837-2285-4