Ada 464724

Developing Effective Adaptive Missile Crews and Command
and Control Teams for Air and Missile Defense Systems
by John K. Hawley and Anna L. Mares
ARL-SR-149 March 2007
Approved for public release; distribution unlimited.

NOTICES
Disclaimers
The findings in this report are not to be construed as an official Department of the Army position
unless so designated by other authorized documents.
Citation of manufacturer’s or trade names does not constitute an official endorsement or

approval of the use thereof.
Destroy this report when it is no longer needed. Do not return it to the originator.
Army Research Laboratory
Adelphi, MD 20783-1197
ARL-SR-149 March 2007
Developing Effective Adaptive Missile Crews and Command

and Control Teams for Air and Missile Defense Systems
John K. Hawley and Anna L. Mares

Human Research and Engineering Directorate, ARL

Form Approved
REPORT DOCUMENTATION PAGE OMB No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the
data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the
burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302.
Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently
valid OMB control number.
PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To)
March 2007 Final

4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Developing Effective Adaptive Missile Crews and Command and Control 5b. GRANT NUMBER
Teams for Air and Missile Defense Systems 5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
John K. Hawley and Anna L. Mares 5e. TASK NUMBER

5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
REPORT NUMBER
U.S. Army Research Laboratory
ATTN: AMSRD-ARL-HR-ME ARL-SR-149
Fort Bliss Field Element
Ft. Bliss, TX 79916
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)
U.S. Army Research Laboratory

Human Research and Engineering Directorate 11. SPONSOR/MONITOR'S REPORT
NUMBER(S)
Aberdeen Proving Ground, MD 21005
12. DISTRIBUTION/AVAILABILITY STATEMENT

13. SUPPLEMENTARY NOTES
14. ABSTRACT
From the fall of 2005 through the summer of 2006 during the New Equipment Training (NET) and unit train-up period for the
Patriot Post-Deployment Build 6 (PDB-6) Limited User Test (LUT), the progress of training for the test unit sounded an alarm
bell loudly for knowledgeable observers. PDB-6 training was not progressing as it should have. Training events were being
completed, but individual and crew performance objectives were not being met. Many of the training issues identified and
discussed in earlier training-related reports produced under the Patriot Vigilance project (e.g., Hawley, Mares, & Giammanco,
2006) were surfacing and were not being addressed adequately by the NET process or follow-on collective training conducted
by the test unit. Clearly, more applied training guidance for the emerging class of knowledge-intensive air and missile defense
(AMD) systems represented by Patriot and PDB-6 was required. This report is an attempt to meet the requirement for more
applied AMD training guidance. It extends concepts originally introduced in Hawley, Mares, and Giammanco (2006), but is
more hands-on and practical. The report is intended as a primer on advances in training technology and methodologies for
AMD unit commanders and training managers (usually battalion or brigade S-3s). In keeping with this objective, the report is
not intended to be a technical document. But it is not possible to completely avoid a technical discussion of selected human
performance, learning, or training topics. It is sometimes necessary to explain why selected training practices are emphasized
over others. However, these technical discussions are brief and to-the-point. The report also includes background references
for a number of the key technical topics addressed. These key technical references are readily available, selected to be
comprehensible to a lay audience, and should be consulted if readers are interested in additional information on any of the
various human performance topics addressed. In addition to the practical guidance provided, the report addresses two
questions central to effective training for the contemporary AMD operating environment: (1) What makes the emerging AMD
operating environment particularly challenging from a training perspective? and (2) How does AMD training have to change
to meet these challenges? Understanding and addressing the issues underlying these questions is critical to developing missile
crews and C2 teams able to handle the complex environments of today’s and future conflicts.
15. SUBJECT TERMS
Patriot, human supervisory control. Training, job preparation, routine expertise, adaptive expertise
17. LIMITATION 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON
OF ABSTRACT OF PAGES
16. SECURITY CLASSIFICATION OF: John K. Hawley
a. REPORT b. ABSTRACT c. THIS PAGE SAR 44 19b. TELEPHONE NUMBER (Include area code)
Unclassified Unclassified Unclassified 915-568-2896
Standard Form 298 (Rev. 8/98)
Prescribed by ANSI Std. Z39.18
ii
Contents
List of Figures v
1. Background 1
2. The AMD Training Problem and Its Solution 5
3. Turning Theory into Practice 9

3.1 Training Preliminaries.....................................................................................................9
3.2 Training for Routine Expertise......................................................................................12
3.2.1 Scenario Content ...............................................................................................13
3.2.2 Performance Standards: Task Standards and Job Competencies .....................13
3.2.3 Time to Reach Desired Competency Levels .....................................................13
3.2.4 Qualified Instructors..........................................................................................13
3.2.5 Performance Diagnosis and Feedback ..............................................................14
3.3 Training for Adaptive Expertise....................................................................................14
3.3.1 Preliminary Issues .............................................................................................14
3.3.2 The Role of Crew and Team Training in Developing Adaptive Expertise .......15
4. Putting It All Together 17

4.1 An Overview of 5-7ADA’s Gunnery Certification Program ........................................17
4.1.1 Basic Skills (Tables I–IV) .................................................................................17
4.1.2 Intermediate Skills (Tables V-VIII) ..................................................................18
4.1.3 Advanced Skills (Tables IX-XII) ......................................................................19
4.1.4 Sustainment Training ........................................................................................19
4.2 Suggestions for Enhancement and Improvement..........................................................19
5. Key Points 21
6. References 23
Appendix A. Task Matrix 25
Appendix B. Progress Report 29
iii
Appendix C. Crew Progress Report 31
Distribution List 33
iv
List of Figures
Figure 1. The recognition-primed decision model......................................................................... 7

Figure 2. Scenario-based training cycle....................................................................................... 11
v
INTENTIONALLY LEFT BLANK
vi
1. Background
During the combat operations phase of Operation Iraqi Freedom (OIF)—March-April 2003,
Army Patriot units were involved in two fratricide incidents. In the first, a British Tornado was
misclassified as an anti-radiation missile (ARM) and subsequently engaged and destroyed. The
second fratricide incident involved a Navy F/A-18 that was misclassified as a tactical ballistic
missile (TBM) and also engaged and destroyed. Three flight crew members lost their lives in
these incidents. OIF involved a total of 11 Patriot engagements by U.S. units. Of these 11, nine
resulted in successful TBM engagements; the other two were fratricides. Various Boards of
Inquiry (BOIs) were convened to investigate these incidents, and they did an excellent job of
piecing together and describing the technical and operational circumstances that led to the
fratricides.
In the spring of 2004, personnel from the Army Research Laboratory’s Human Research and
Engineering Directorate (ARL HRED) began looking into Patriot and air and missile defense
(AMD) performance and training issues at the invitation of the then Ft. Bliss Commander, Major
General (MG) Michael A. Vane. After reviewing the conclusions of the various BOIs, he was
convinced that human performance issues were part of the problem leading to the fratricides.
MG Vane was interested in operator vigilance and situation awareness (SA) as they relate to the
performance of automated AMD battle command systems. [Note: The generally accepted
definition of SA is from Endsley, Bolte, and Jones (2003) who define it as the perception of
elements in the environment, the comprehension of their meaning, and the projection of their
status in the near future.] MG Vane was particularly concerned by what he termed a “lack of
vigilance” on the part of Patriot operators along with an apparent “lack of cognizance” of what
was being presented to them on situation displays with a resulting “absolute trust in automation.”
Accordingly, he requested that HRED conduct a human-performance-oriented critical incident
assessment to complement the BOI investigations and report back to him regarding potential
problems and solutions.
HRED’s project team spent most of the summer and fall of 2004 performing the requested
human-performance-oriented critical incident assessment of the OIF fratricides—reading
documents, interviewing knowledgeable personnel in the Ft. Bliss area, and observing Patriot
training and operations. An initial assessment was delivered to MG Vane in October 2004. A
discussion of the results of what was termed the Patriot Vigilance project is given in Hawley and
Mares (2006) and is not repeated here, other than in a summary manner. HRED’s report to MG
Vane in October 2004 recommended two primary actionable items to redress the performance
problems identified during the Patriot Vigilance effort:
1
1. Re-examine automation concepts, operator roles, and command and control (C2)
relationships in AMD battle command systems to emphasize effective human supervisory
control (HSC); and
2. Develop more effective missile crews and C2 teams, particularly with respect to Air
Defense Operations—re-look the level of expertise required to operate such a lethal system
on the modern battlefield.
In present usage, the term effective human supervisory control refers to a situation in which
Soldiers and not the automated system are the ultimate decision makers in AMD firing decisions.
Uncritical acquiescence to the automated system’s recommendations is not effective human
supervisory control.
In a report on Patriot system performance requested by the Under Secretary of Defense for
Acquisition, Technology, and Logistics, the Defense Science Board (DSB) reinforced HRED’s
recommendations with the following comments (DSB, 2004). Although the full DSB report is
classified, these extracts are not.
The Patriot system should migrate to more of a “man-in-the-loop” philosophy

versus a fully automated philosophy—providing operator awareness and control
of engagement processes.
and
Patriot training and simulations should be upgraded to support this man-in-the-
loop protocol including the ability to train on confusing and complex scenarios
that contain unbriefed surprises.
A summary of the DSB report on Patriot system performance is available for download on the
DSB’s web site.
The key notion in the first Defense Science Board recommendation is captured in the phrase,
“providing operator awareness and control of engagement processes.” Simply put, Soldiers and
not the automated system must be the ultimate decision makers in AMD engagements.
Decisions to shoot or not to shoot must be made by crews having the technical potential for
adequate SA and the expertise to understand the significance of the information available to
them. The DSB’s first recommendation is synonymous with HRED’s first actionable item
concerning establishing effective human supervisory control of Patriot and other AMD systems.
The second Defense Science Board recommendation having major significance for Soldier
performance in contemporary AMD operations concerns operator training and professional
development. Here, the DSB supported HRED’s conclusion that it is necessary to re-look the
level of expertise necessary to operate such a lethal system on the modern battlefield. To
highlight the importance of the training issue, the Army Board of Inquiry investigating the OIF
fratricides stated bluntly “The system [Patriot] is too lethal to be placed in the hands of crews
trained to such a limited standard.”
2
The Board of Inquiry investigating the OIF fratricides specifically criticized pre-deployment
Patriot training for emphasizing “rote drills” versus the “exercise of high-level judgment.” The
essence of this criticism is that the AMD user community approaches training for a complex,
knowledge-based function like Air Defense Operations in much the same manner as linear, skill-
and rule-based activities like March Order and Emplacement or System Set-up. The emphasis is
on operating equipment and mastering routines rather than adaptive problem solving. However,
the range of actions required in routine drills is narrower and more predictable than those
encountered in combat operations.
The U.S. Navy faced a similar reconsideration of training practices in the aftermath of the shoot-
down of an Iranian airbus by the USS Vincennes in 1988. After more than 10 years of research,
the Navy reached several conclusions that are also relevant to the contemporary AMD setting.
First, the Navy’s research concluded that situation awareness is the key factor underlying
decision quality in battle command. Situation awareness is built upon in-depth technical and
tactical expertise. The primary implication of this conclusion is that marginally-skilled operators
cannot develop the SA necessary for effective human supervisory control, regardless of the
sophistication of the battle command hardware suite provided to them. Systems provide data,
but only users develop situation awareness. Technology can amplify human expertise, but
cannot substitute for it. Relevant operator expertise is an equal contributor to effective HSC of
system operations.
The Navy also concluded that Aegis operator training must emphasize the development of
adaptive decision-making skills. Adaptive decision-making skills, or the ability to think outside
the box defined by routine crew drills, are a key aspect of effective operator performance in
ambiguous situations. The Navy’s third major conclusion was that shipboard (i.e., unit) training
must address team in addition to individual performance. Competent crews are the basis of
effective unit performance, and crews are more than the sum of their individual members.
The Defense Science Board’s recommendation to include unbriefed surprises in training does not
mean that it is sufficient merely to insert anomalous events like those encountered in OIF into
training scenarios. In advanced AMD training, scenarios comprise most of the curriculum. To
properly prepare operators for combat, scenario designers must bear in mind that the surprises of
OIF are representative of a class of potential anomalies. Selected anomalies occurred then;
others—some similar, some different—will occur on future battlefields. It is thus necessary that
operators be imbued with a sense of mindfulness that automated battle command systems are
fallible. The system’s recommendations will be correct most but not all of the time. Training
must foster the development of the adaptive expertise essential to recognize potential anomalies
and the skills necessary to determine an appropriate course of action. Operators must walk a fine
line between blind faith in the system and wholesale mistrust. AMD commanders and training
managers must not underestimate the difficulties associated with adequately meeting this
challenge.
3
In addition to the briefing to MG Vane, the initial phase of the Patriot Vigilance project also
resulted in an ARL technical report titled The Human Side of Automation: Lessons for Air
Defense Command and Control (Hawley, Mares, & Giammanco, 2005). After reviewing results
from the first phase of the project, the TRADOC System Manager for Lower Tier AMD systems
(TSM-LT), COL Rob Jassey, requested that the Patriot Vigilance project continue into a second
year. COL Jassey specifically requested that HRED’s project staff expand on the material
presented in Hawley, Mares, and Giammanco (2005) and prepare two, more-detailed reports, one
concerned with design for effective human supervisory control and a second addressing training
for the emerging class of AMD systems. The intent of these reports was to inform the AMD
community on “what right looks like” in each of these topic areas. The results were the two
reports: Developing Effective Human Supervisory Control for Air and Missile Defense Systems
(Hawley & Mares, 2006), and Training for Effective Human Supervisory Control of Air and
Missile Defense Systems (Hawley, Mares, & Giammanco, 2006).
In the late summer of 2005 after MG Vane had left Ft. Bliss for another assignment, the project
staff briefed his replacement, MG (then Brigadier General) Robert P. Lennox, on the status and
results of the Patriot Vigilance project. MG Lennox requested that the project be continued for
at least another year so that the technical staff could work with the AMD community on
implementing selected results. HRED’s project staff also would participate as the MANPRINT
(Manpower and Personnel Integration) evaluator during a Limited User Test (LUT) of the Post-
Deployment Build 6 (PDB-6) software suite for the Patriot system. PDB-6 was developed to
address many of the Patriot system’s operational deficiencies that had surfaced during OIF and
were generally considered to have contributed to the unacceptable fratricide rate.
From the fall of 2005 through the summer of 2006 during the New Equipment Training (NET)
and unit train-up period for the PDB-6 LUT, the HRED project staff’s observations regarding the
progress of training for the test unit sounded an alarm bell loudly. PDB-6 training was not
progressing as it should have. Training events were being completed, but individual and crew
performance objectives were not being met. Many of the training issues identified and discussed
in Hawley, Mares, and Giammanco (2006) were surfacing and were not being addressed
adequately by the NET process or follow-on collective training by the test unit. Clearly, more
applied training guidance for the emerging class of knowledge-intensive AMD systems
represented by Patriot and PDB-6 was required.
This report is an attempt to meet the requirement for more applied AMD training guidance. It
extends concepts originally introduced in Hawley, Mares, and Giammanco (2006), but is more
hands-on and practical. The report is intended as a primer on advances in training technology
and methodologies for AMD unit commanders and training managers (usually battalion or
brigade S-3s). In keeping with this objective, the report is not intended to be a technical
document. But it is not possible to completely avoid a technical discussion of selected human
performance, learning, or training topics. It is sometimes necessary to explain why selected
training practices are emphasized over others. However, these technical discussions are brief
4
and to-the-point. The report also includes background references for a number of the key
technical topics addressed. These key technical references are readily available, selected to be
comprehensible to a lay audience, and should be consulted if readers are interested in additional
information on any of the various human performance topics addressed.
The next section presents a more detailed assessment of the AMD training problem and its
general solution. It is intended to explore two questions central to effective training for the
contemporary AMD operating environment:
1. What makes the emerging AMD operating environment particularly challenging from a
training perspective?
2. How does AMD training have to change to meet these challenges?

All of the material to follow is intended to provide unit-level trainers (e.g., battalion commanders
and their S-3s) with practical advice concerning how to develop missile crews and C2 teams able
to handle the complex environments of today’s and future conflicts.
2. The AMD Training Problem and Its Solution
One of the defining properties of current and future AMD battle command systems is an
increasing reliance on automation. Technological opportunities and an increasingly complex
operating environment have created a situation where AMD operators must be provided with
automated decision support to meet mission objectives. There is an unfortunate tendency among
system developers, trainers, and users to assume that automation is innately beneficial. Research
and experience in a numbers of areas suggest, however, that such is not always the case. To
begin, automation elevates operators into system monitors rather than active controllers.
Operators are removed from moment-to-moment active control and become monitors and
managers of subordinate automated processes. It is a well established fact that humans make
very poor system monitors.
Beyond classical vigilance, automation applied to real-time C2 brings into play what has been
called the Catch-22 of human supervisory control: Automation has been introduced because it
can do the job better than human operators, but human operators have been left in the control
loop to “monitor” that the automated system is performing correctly and to override the
automated system when it is “wrong.” The unstated assumption is that operators can properly
decide when the automation’s decisions should be overruled. Human operators are expected to
compensate for automation unreliability, but research and experience have consistently shown
that humans suffer from a variety of physical and cognitive limitations that make this assumption
unrealistic. Automation researchers have concluded that while the risks associated with the
5
Catch-22 of human supervisory control can never be eliminated entirely, they can be managed
more effectively through a number of positive actions directed at supporting effective control.
These actions generally fall into one of two categories: (1) design to support effective HSC, and
(2) training focused on developing operator expertise. The thrust of this report is practical ways
to pursue the second objective.
To cope with the Catch-22 of human supervisory control and properly structure an effective
training program for AMD, it is first necessary to consider how operators actually make
decisions in an air battle setting. The conventional wisdom in military circles is that decision
makers routinely follow what is termed the classical decision making model. That is, the
“textbook” way to make an important decision is to (1) list the different options, (2) evaluate
those options using a common set of criteria, (3) determine how important each criterion is, (4)
rate each option on each criterion, (5) do the math, and (6) determine the optimal choice. The
conventional notion of decision making is thorough, systematic, rational, and scientific. But it is
also by and large a myth, particularly in real-time C2.
A growing body of research and experience indicates that human experts do not follow the
classical decision making approach in real-time decision making. Rather, human experts use
what is termed “pattern matching” to quickly understand a situation and select an appropriate
course of action. Klein (2003) refers to this pattern matching-response selection process as
Recognition-Primed Decision Making, or RPD. The RPD process works much as described in
figure 1. The tactical situation generates a set of cues. This cue set lets the expert recognize a
pattern: “I’ve seen this before.” Patterns are associated with what are termed “action scripts”:
“And these are the actions that I took when I saw this pattern before.” Alternative action scripts
are assessed using mental simulation based on the expert’s mental model of the controlled
process. The mental simulation process leads to the selection of one action script as preferred.
A mental model is the expert’s internal understanding of how the controlled process “works,” or
fits together. Norman (2002) remarks that a “good” mental model permits equipment users to
predict the effects of their actions. Without a good model, users perform as they are told without
really knowing why. As long as things work, they can manage. However, when things go
wrong or when the unexpected happens, users frequently are at a loss concerning how to
proceed.
The recognition-primed decision making cycle is executed rapidly and internally. It is also
highly dependent on operator expertise: knowledge (what I know about), skill (what I know how
to do), and experience (what I’ve seen before). Experts immediately “know what to do” in a
given situation, and the RPD model explains how they arrive at a preferred course of action so
quickly.
6
Tactical
Situation
generates
to affect the
Mental
Simulation
Cues
which you using
assess by… your
Action
Scripts
Mental
Models that let you
recognize
Patterns
that activate
Source: Adapted from Klein (2003)
Figure 1. The recognition-primed decision model.
Endsley and her colleagues (Endsley, Bolte, & Jones, 2003) argue that situation awareness drives
the RPD process and is the key factor determining decision quality in battle command. Operator
situation awareness is influenced by factors such as:
1. Preconceptions (what operators have been led to believe about the system and its
capabilities)
2. Tactical objectives
3. Operator ability levels
4. Training
5. Experience
6. Doctrine
7. Formal guidance (e.g., TTPs [tactics, techniques, and procedures] and TSOPs [tactical
standing operating procedures])
8. Immediate cognitive workload.
7
Moreover, operator tactical and technical expertise is the dominant factor in establishing Level 2
(Comprehension, sometimes called Situational Understanding) and Level 3 (Projection) situation
awareness. Comprehension and projection underlie the operator’s ability to notice patterns,
judge typicality, spot anomalies, and have a “feel” for what is happening around them (Klein,
2003). Dekker (2002) summarizes the recognition-primed decision situation quite succinctly by
noting that what distinguishes “good” decision makers from “poor” decision makers in
knowledge-intensive job settings is their ability to make sense of situations using a highly
organized experience base of relevant knowledge. Sensemaking ability is the link between
awareness (what operators see on their screens or obtain from other sources) and understanding.
Because of the realities discussed above, training for the new class of AMD systems must be
qualitatively different from the era of simpler systems. The issue here is that the new class of
technology-dominated systems is complex and knowledge-intensive. Knowledge-intensive
systems shift the focus of user performance from what are referred to as skill- and rule-based
performances to knowledge-based performances (Dekker, 2002; Rasmussen, 1986). Operating
equipment is an example of a skill-based performance, and following procedures is an example
of a rule-based performance; decision making and problem solving are knowledge-based
performances. As a caution, readers should not confuse operating equipment with using that
equipment to meet mission objectives. Knowing how to operate an item of equipment is a
necessary but not sufficient condition for knowing how to use that item to meet a mission
objective.
Most current Army training stresses skill- and rule-based performances but overlooks
knowledge-based performance requirements. Recall that the board of inquiry looking into the
Patriot fratricides during OIF criticized pre-deployment Patriot training for emphasizing “rote
drills versus high-level judgment.” Rote drills focus on skill- and rule-based performances like
operating equipment or following procedures; exercising high-level judgment is a knowledge-
based performance.
The crux of the previous discussion is that knowledge-intensive systems place a premium on user
expertise. In present usage, the term expertise refers to a capability for consistently superior
performance on a specified set of representative tasks for a domain (Ericsson & Charness, 1994).
Expertise is a function of operator (1) knowledge, (2) skill, and (3) job-relevant experience.
Given the centrality of user expertise in the emerging warfighting environment, an obvious
follow-on question is “How is such expertise developed?” Norman (1993) notes that there are at
least three phases of learning leading to expertise as defined above. These are (1) accretion, (2)
tuning, and (3) restructuring. Accretion is the accumulation of facts. Tuning refers to the
process of translating knowledge into skill. The final stage of learning is restructuring, or
forming and reforming the proper conceptual framework for performing as an expert—the
sensemaking ability referred to previously.
8
Norman further remarks that accretion and tuning are primarily experiential—they take place
actively in an experience-based learning environment. Restructuring is reflective. It involves
exploring the domain in depth, forming comparisons, and integrating across related domains.
The complete process requires a hands-on learning environment and hours and hours of practice
under the supervision of a coach or mentor. Such feedback-intensive training is referred to as
deliberate practice. How many hours are necessary? Norman asserts that for any complex
activity, a minimum of 5,000 hours of practice—two years of full-time effort—are required to
turn a novice into an entry-level expert. Expert, in this context, refers to a user who has
developed the sensemaking capability necessary to perform appropriately in a knowledge-
intensive job setting. It should be noted, however, that the 5,000-hour rule is cumulative. It
applies to all training and job preparation relevant to a performance domain: institutional
training, on-the-job-training (OJT), special skills training, and so forth.
Dekker (2002) argues that complex systems like those used in AMD require an “overwhelming
human contribution” for their effective operation. He states that “people are the only ones who
can hold together the patchwork of technologies in their worlds; the only ones who can make it
work in actual practice” (p. 103). A paradox of the emerging high-technology warfighting
environment is that automation and other advanced technologies reduce the moment to moment
need for humans while simultaneously increasing their criticality to overall system effectiveness.
In a nutshell, this statement summarizes the problem facing AMD commanders and training
managers. Advanced technology enables more capable systems to match the requirements of a
more complex battlespace. But this capability comes at a price, and the price is a requirement
for more sophisticated user personnel. Moreover, the performance situation is likely to become
even more demanding with the fielding of the Integrated Air and Missile Defense concept and
other so-called systems of systems.
The preceding paragraphs provide a theory-based answer to the two questions that opened the
discussion on developing effective training for contemporary AMD systems. The next step is to
begin turning theory into practice. That is, to describe how to use the concepts discussed above
in a real-world collective training environment with a goal of developing effective and adaptive
crews and teams.
3. Turning Theory into Practice
3.1 Training Preliminaries

Training is a process by which job-relevant knowledge, skills, and competencies are acquired by
individuals, crews, and multi-echelon units. In a military setting, its purpose is to deliver warfare
competence where and when it is needed. Operationally, training consists of job-relevant
practice with feedback. Without proper feedback, job-relevant learning might not take place.
Putting both definitions together, training is a process by which job-relevant learning at various
levels (individual, crew, or multi-echelon unit) is facilitated.
9
Training is not simulations, networks, virtual environments, or games even if they are labeled as
training or training devices. All of these technical mechanisms can be used to craft an
environment in which learning might take place, but they are not training per se. This is an
important distinction: Various methods and technologies such as those mentioned above can be
used to support training if they provide an environment in which job-relevant practice with
appropriate feedback can occur. Training managers must, however, first know what must be
practiced and the appropriate form of essential feedback.
A good instructional program is defined as one that supports relevant practice with appropriate
feedback leading to desired warfare competencies. Developing a good instructional program is a
relatively simple process. However, the devil can be in the details of how the various steps in the
process are carried out. The steps in proper instructional design are listed as follows (Whitmore,
2002):
1. Specify the job performances we want trainees to learn to do.

2. Break those job performances down successively into their component activities (including
mental activities) until we reach the level of skill already possessed by the trainees.
3. Develop situations (scenarios) that lead trainees through those things we want them to learn
to do.
4. Lead trainees through practicing the new activities, beginning with the lowest level and
successively assembling higher-level activities from lower-level components.
5. Provide trainees with sufficient practice—with feedback—at each level to reach desired
competency levels.
Taken together and without all of the detail, this process is conceptually similar to what is called
for in the Instructional Systems Development (ISD) process or its Army variant, the Systems
Approach to Training (SAT).
It has been noted that for a dynamic, performance-based setting like Air Defense Operations,
scenarios are most of the curriculum. That is, the job-relevant practice and feedback essential to
developing warfare competence takes place within a simulation-based learning environment. In
this setting, training scenarios provide the structure within which the stimulus and response
aspects of learning are presented and evaluated. The overall process by which effective
scenario-based training is developed and implemented is depicted in figure 2.
In figure 2, results from the system’s job and task analysis (JTA) are used to prepare a skill
inventory, or list of job skills to be trained. These job-relevant skills are used to develop a list of
learning objectives and associated competencies. Learning objectives then drive the
development of scenario events, or scripts. If you want a trainee to learn how to do something,
then the training scenario must provide the stimuli necessary to elicit that behavior. Next, the
training developer must define performance measures and standards. A performance measure is
10
a behavioral description that permits an evaluator to unambiguously decide whether a
performance took place. The performance standard provides a way of judging whether that
performance was acceptable. In keeping with the overall view of training advocated in this
paper, performance measures and standards are developed first for the task and skill level and
then for the job or job segment level. It is necessary that trainees master the individual tasks and
skills that comprise job performance, and they must also know how to compose these individual
skills to perform acceptably in the whole-job setting.
Source: Adapted from Cannon-Bowers, Burns, Salas, & Pruitt (1998)
Figure 2. Scenario-based training cycle.
In performance-oriented training, feedback is just as important as performance evaluation. For

learning to occur, trainees must be provided with feedback regarding their performance along
with corrective guidance. Accordingly, the next step in the Scenario-Based Training Cycle is
performance diagnosis: If the performance was not acceptable, why not? Performance
diagnostic information provides the basis for the feedback and debrief stage, or After Action
Review (AAR). Performance diagnosis must be objective and specific to the performances
involved. Moreover, for best effect, feedback should be immediate with an opportunity to
practice less-than-successful performance elements until individual, crew, or team performance
is acceptable.
There is increasing evidence in the training literature that training for a complex performance
setting like AMD is best approached as a continuum of developmental experiences that occur
across a series of different environments (Kozlowski, 1998). Hawley, Mares, and Giammanco
(2006) advocate this segmented approach for AMD and argue for training to be conceptually
structured into two separate parts: (1) training for Routine expertise, and (2) training for
11
Adaptive expertise. Training for Routine expertise is directed at improving effectiveness and
reliability in situations involving little ambiguity or uncertainty. The intent is to develop
individuals and crews well practiced in the basics of Air Defense Operations. Training for
Adaptive expertise is based upon competent baseline performance by individuals and crews. It
might be termed advanced training for situations that are non-routine or out-of-the-ordinary. As
the name implies, it is intended to foster the adaptability necessary to cope with the inevitable
surprises of combat operations. Each of these stages is discussed in more detail in the sections to
follow.
3.2 Training for Routine Expertise

As noted previously, training for Routine expertise is intended to provide practice on well-
established, normative problem situations. It is the starting point for AMD collective training
and is intended to lay the groundwork for the more intensive training to follow. One must learn
to walk before one can run.
Training for Routine expertise consists of two parts. The first portion is initial knowledge and
skill acquisition. Initial knowledge and skill acquisition is intended to provide individual
trainees with basic AMD concepts and the knowledge necessary to take advantage of later
performance-oriented training. This phase of training for Routine expertise corresponds to the
initial stages of Norman’s (1993) accretion phase in the development of expertise.
The first part of training for Routine expertise also is intended to provide individuals with initial
training on equipment operations and the basics of how to use the system to fight an air battle.
This portion is not intended to produce individuals capable of serving on operational crews.
Rather, it is intended to produce individuals fully ready to take advantage of later crew-oriented
training. This first part of training for Routine expertise could take place in a formal school
setting.
The second part of training for Routine expertise is concerned with the development of what are
termed “novice” crews. In current usage, a novice crew is one that has demonstrated its
proficiency on a range of normative air battle operations scenarios. No tricks and few
ambiguities; just the basics of Air Defense Operations. But the basics are well-practiced and
verified.
During the subject matter expert (SME) interviews conducted as part of the Patriot Vigilance
project, a frequent comment was that Table VIII certification (the entry-level standard for Patriot
crews) really amounts to a “learners permit.” The concept of a novice crew is consistent with
this view. A Table VIII certified crew is able to handle itself in normative situations where
things are as they appear to be. But such crews might have difficulties in more challenging and
complex situations. It is the authors’ view—and one shared by others in the AMD community—
that this observation regarding Table VIII certification and what it really means to be “qualified”
is one of the significant lessons coming out of OIF. Recall the board of inquiry remark that the
12
Patriot system is too lethal to be placed in the hands of crews trained to such a limited standard.
The limited standard referred to in this statement is Table VIII certification. Once again,
certified does not necessarily mean fully qualified.
On the surface, training for Routine expertise as described here will not appear to be different
from many current school and unit practices. And to some extent this observation is true.
However, there are several key differences in training practices that must be incorporated for
training for Routine expertise to be effective and lay a suitable groundwork for later training for
Adaptive expertise. These modified practices are discussed in the sub-sections to follow.
3.2.1 Scenario Content

In training for a job like AMD operator, the scenario is most of the curriculum. This means that
training scenarios must be carefully scripted to cue the decisions and performances indicated in
the system’s job and task analysis as being critical. Training managers also must be aware that
training scenarios have a limited shelf life. After more than about three uses with the same group
of trainees, they lose their training value. Trainees recognize the scenario and begin to “game”
it. Gaming scenarios destroys their learning value, since gaming rather than learning becomes
the objective of the exercise.
3.2.2 Performance Standards: Task Standards and Job Competencies

Training managers must attend to the development of operational performance standards for
scenarios and for success at the various certification levels. Skill checks for individual tasks or
portions of tasks are important in skill development, but they are not sufficient for judging job
competence. Job standards must be objectively stated in terms of successful performance against
benchmark scenarios having stated performance objectives and known difficulty levels. In
training for Air Defense Operations, the whole defined as competent job performance is more
than the sum of its individual task parts.
3.2.3 Time to Reach Desired Competency Levels

Time to train to desired competency levels is one of the recurring problems in Army and military
training in general. Training times must be determined on the basis of how long individuals take
to reach desired competency levels. A situation must not be allowed to exist in which training is
shoehorned into available times or is fixed arbitrarily on the basis of bureaucratic or
administrative considerations. If such considerations are permitted to dictate training times, the
result will be incomplete training and inadequate on-the-job performance. To the extent
practical, training must be self-paced and performance-oriented.
3.2.4 Qualified Instructors

There is an old adage that it is hard for an apprentice to learn if there are no masters. Personnel
assigned to training jobs in the institution and in units must be expert job performers (EJPs). An
EJP is a subject matter expert who also has recent and relevant on-the-job experience. Their job
13
is to teach trainees how to do the job, not just to teach them about the job. Consideration must
also be given to the teaching ability of potential instructors. System experience is not a
guarantor of effective instructional capability. Norman (1993) cites results indicating that a
course’s reputation is a major determinant of trainee motivation. If a course has a good
reputation that word will spread and trainees will come to the course motivated and prepared to
learn. On the other hand, if a course has a bad reputation, that fact will be reflected in an
unwillingness to come or attend to the material. Instructor quality is a major factor in training’s
reputation and thus trainee motivation. Trainee motivation cannot be commanded.
3.2.5 Performance Diagnosis and Feedback

A generally accepted axiom of learning theory is that little job-relevant learning takes place in
the absence of focused feedback. This simple fact underlies much of the success attributed to the
Army’s training at Combat Training Centers (CTCs) such as the National Training Center
(NTC). At the CTCs, considerable effort goes into the post-exercise AAR. Operational training
practices can be fully satisfactory, but if the AAR process is deficient, performance will not
improve. The same relationship applies to lower echelon training conducted in institutions and
units outside the CTC context. AMD units do not routinely participate in NTC rotations, but the
importance of proper AARs to development of organizational effectiveness applies to them as
well.
Good AARs go well beyond a mere replay or summary of what went on. They consist of a
behavioral critique focused on specific tasks, skills, or job segments. AARs are intended to
change behavior and the way trainees think about the performances in question. Trainees must
be told not only that a particular performance was right or wrong, but also why that performance
was right or wrong. Good AARs are critical to tuning the mental models upon which future
performance will be based.
3.3 Training for Adaptive Expertise
3.3.1 Preliminary Issues

As noted previously, training for Adaptive expertise is intended to foster the ability to think
outside the box defined by routine crew drills. It is intended to provide crews and teams with the
ability to cope with unreliable automation and the inevitable unforeseen events that characterize
combat operations. Adaptive expertise involves going well beyond applying procedural
knowledge of an automatic sort—what the OIF board of inquiry referred to as rote drills without
active thinking. Research and experience clearly show that stopping at the level of Routine
expertise produces operators who have trouble dealing with ill-structured and novel problems
(Kozlowski, 1998). Routine experts are unable to recognize when the underlying structure of the
tactical situation has changed and they must shift to a problem-solving or active thinking mode.
This is the essence of MG Vane’s comments regarding operator vigilance problems and lack of
cognizance of the tactical situation.
14
In one sense, the mechanisms underlying training for Adaptive expertise are straightforward. As
the Defense Science Board recommended, trainees must face confusing and complex scenarios
that contain unbriefed surprises. Such scenarios must push crews and teams outside their
comfort zone and stress problem solving over routine operational processes. Crews working
together must learn to recognize when a situation is not ordinary and requires problem-solving
intervention. However, the solution is not quite that simple. Problem-based training, such as
that suggested by the DSB, is not a total remedy if trainees are led to believe or are allowed to
view the limited experiences they see in training as representative of all instances they will
encounter in the operational environment (Dekker, 2002).
Developing adaptive capabilities is a long-term process that provides trainees with extensive
deliberate practice—relevant practice with expert and immediate feedback. Adaptive
performance capabilities build on a foundation of Routine expertise and are enhanced by
scenario variability and novelty that challenge routine skills. Expertise as defined previously
will develop as a natural consequence of the long-term application of this basic strategy
(Kozlowski, 1998). This is an important point and bears repeating: Adaptive expertise will
develop over time as a natural consequence of rigorously following a rather basic training
strategy involving increasing levels of scenario novelty and challenge coupled with immediate
and focused feedback. However, as repeatedly emphasized in this report, time is a critical factor
in developing expertise. Expert performance cannot be developed in a few short weeks or
months of training. Training for Adaptive expertise also must focus explicitly on developing
sensemaking skills: recognizing when to shift from automatic processing (being on mental
“cruise control,” so to speak) to active thinking and problem solving.
Readers should note that training for Adaptive expertise coincides with the latter stages of
Norman’s (1993) tuning and restructuring phases of learning. This stage of training stresses
generalizing routine skills to more fluid situations and developing the skills necessary to execute
complex judgments—sensemaking ability. The reflection that goes on as part of the
restructuring phase is an important facilitator of this process. Recall that reflection and
restructuring involve (1) exploring the problem domain in depth, (2) forming comparisons, and
(3) integrating across related domains. Restructuring is central to the development of Dekker’s
(2002) “organized experience base of relevant knowledge” that underlies an operator’s
sensemaking ability. The goal of training for Adaptive expertise is to advance trainees’ skill
level from the surface comprehension that characterizes Routine experts to the deep
understanding of the performance domain that characterizes Adaptive experts.
3.3.2 The Role of Crew and Team Training in Developing Adaptive Expertise
It has been noted that units, not individuals are the basis of warfare competence. This is not to
say that individual performance is not important. Individual performance is an important
component of crew and unit performance. But for the most part, individuals do not perform
missions. Crews, teams, and units do.
15
There is an increasing body of evidence that Adaptive expertise is best developed within the
context of intact teams operating in a realistic performance setting (Kozlowski, 1998). It is also
an accepted fact that a crew is more than the sum of its individual parts, and melding the
disparate parts into a functioning team takes time working together plus specific interventions
focused on developing teamwork skills. Proper performance at the crew level requires that
individual crewmembers be aware of their interdependencies and be able to back each other up.
Highly skilled teams also are “workload sponges” in the sense that team workload is less than the
sum of the individual parts.
A variety of recent research and operational experience suggests that managing crew
interdependencies and developing teamwork skills may require team process training similar to
the Crew Resource Management (CRM) programs prevalent in the aviation community. Once
almost exclusively an aviation program, CRM is now broadly viewed as the utilization of all
available human, informational, and equipment resources toward effective and efficient
operations in domains dependant on crew or team performance (Helmreich, Merritt, & Wilhelm,
1999). In the aviation world, CRM skills are viewed as a primary line of defense against human
error and its consequences. The Federal Aviation Administration (FAA) uses CRM principles to
foster cooperation and coordination among the controllers in air traffic control centers. Also, the
Army’s Aviation branch uses CRM principles in its crew coordination training for aircrews.
Army-sponsored research has shown that crew coordination training significantly improves
aircrew performance (Grubb, Simon, Leedom, & Zeller, 1995).
Properly-implemented CRM focuses on enhancing crew and team behaviors that support mission
success. Specific elements of CRM training include the following behavioral categories:
1. Mission planning and evaluation
2. Task management
3. Enhancing situation awareness
4. Crew coordination
5. Communications
6. Risk management
7. Tactics employment
Given the centrality of these concepts to effective missile crew and C2 team performance, it
might prove useful to adapt and apply CRM principles within the AMD collective training
environment. Without such teamwork-oriented training, teams of proficient individuals will not
necessarily evolve into expert crews and teams.
16
Recent research also highlights the importance of team leaders in the development of effective
crews. Effective crew and team leadership is a key factor in melding individual technical experts
into a high-performing team. Kozlowski (1998) notes that effective team leaders foster effective
team performance by (1) melding individuals into a coherent team, (2) fostering the development
of an adaptive network of roles, and (3) assisting the crew in becoming a self-learning system.
These three features—coherency, or a clear understanding of team roles and interdependencies;
an adaptive network of roles; and a self-learning system—differentiate effective teams from a
simple aggregation of individuals. Effective teams work together toward common goals, learn as
a group, and adjust their intra-team roles to reflect changing operational circumstances. Team
leaders facilitate the development of these capabilities by performing both as intra-team
instructors and team mentors.
4. Putting It All Together
To illustrate how the training structure described in the previous section could be applied
practically, these concepts are next integrated with an actual AMD collective training program.
The collective training program used as an example is the Battalion Gunnery Certification
Program developed by the 5th Battalion 7th Air Defense Artillery (ADA) Regiment (5-7 ADA).
The sections to follow describe the overall structure of 5-7 ADA’s gunnery certification program
and then offer suggestions on how to modify this baseline program to take better advantage of
the theory-based guidelines presented previously. Readers should note that the material to
follow pertains only to the portions of 5-7 ADA’s gunnery certification program that address Air
Defense Operations.
4.1 An Overview of 5-7ADA’s Gunnery Certification Program

5-7 ADA’s gunnery certification program is comprised of four consecutive blocks of instruction:
Basic Skills, Intermediate Skills, Advanced Skills, and Sustainment Training. Each of these
blocks is described in the subsections to follow.
4.1.1 Basic Skills (Tables I–IV)

5-7 ADA’s gunnery certification program notes that “Soldiers must build a solid foundation
during basic gunnery tables if they are to be successful during intermediate and advanced table
training.” Block 1 is intended to establish this foundation. Specific elements of the Basic Skills
block include the following:
● Knowledge of basic AMD tactics, communications, and system capabilities.
● A “Go” rating on all tasks applicable to their crewmember position.
17
● Ability to fight as part of crew in a low intensity air battle—a Reticule Aim Level (RAL) 5
scenario. RALs refer to the level of difficulty of the scenario.
The individual tasks comprising each of the Tables in 5-7 ADA’s gunnery certification program
are shown in appendix A. Tables refer to progressive task groupings. The individual tasks in
appendix A are derived from the Patriot system’s job and task analysis. Within 5-7 ADA, the
Basic Skills block must be successfully completed within 90 days of a Soldier’s arrival in the
unit. Progress on completing each Table or performance “gate” is recorded and tracked on the
Progress Report form shown as appendix B.
During the interviews conducted as part of the Patriot Vigilance project, a frequent comment by
unit personnel was that Soldiers completing Advanced Individual Training (AIT) for the 14E
Military Occupational Specialty (MOS) were Table IV certified. However, unit training
personnel generally did not think that MOS qualified personnel right out of AIT met that
standard. (Note: The same comment was also made regarding officers exiting the Officers’
Basic Course.) This paper will not take a position one way or the other on this point. However,
if the Basic Skills block is properly conducted, new Soldiers will be ready to take advantage of
the Intermediate Block following unit-level Table IV certification. Given the complexity of
AMD concepts and equipment, it will be beneficial for new Soldiers to go over this foundational
material more than one time before beginning crew-oriented OJT. This point is particularly
important if any significant time (e.g., more than one month) has elapsed between the completion
of AIT and the start of crew-oriented training in the assigned unit. The introduction to 5-7
ADA’s gunnery certification program properly notes that air battle skills are “highly perishable.”
4.1.2 Intermediate Skills (Tables V-VIII)

Following Table IV individual certification, Soldiers begin the Intermediate Skills block. The
Intermediate Skills block is crew-oriented. Its objective is to turn collections of individuals into
novice crews. In the terminology of the previous section, Table VIII certified crews are referred
to as novice crews. They are able to handle normative AMD problems, but might have difficulty
with difficult or ambiguous situations.
Specific elements of Table VIII certification include the following:
● Demonstrated knowledge of battalion and brigade tactics.
● A “Go” rating on all tasks applicable to their crewmember position.
● Ability to fight as part of a designated crew in a mid-intensity air battle (RAL 11).
Soldiers must be Table VIII certified within 180 days of arrival in the unit. Once individual
Soldiers are formed into crews, each crew’s status is tracked and reported on the Crew Progress
Report shown as appendix C.
18
4.1.3 Advanced Skills (Tables IX-XII)
The Advanced Skills block is a continuation of the crew development process begun during the
previous block. Specific elements of this block include:
● Demonstrated knowledge of advanced battalion and brigade tactics.
● A “Go” rating on all tasks for their crewmember position.
● Ability to fight as part of a designated crew in a high-intensity air battle (RAL 16).
The Advanced Skills block must be completed within 270 days of arrival in the unit.
4.1.4 Sustainment Training

Finally, the Sustainment Training block is designed to maintain and improve the skills of crews
that have achieved Table XII status. A major aspect of the Sustainment block is professional
development in the form of reading (e.g., technical and field manuals) and attendance at and
participation in battery- and battalion-level Tactical Seminars (TACSEMs). Professional
development activities such as these are a key aspect of Norman’s restructuring phase in the
development of expertise. Sustainment training coupled with periodic refresher training on
previously learned knowledge, skills, and competencies continues over the course of a Soldier’s
assignment to the unit.
4.2 Suggestions for Enhancement and Improvement

The 5-7 ADA’s gunnery certification program is an excellent structure within which to
implement most if not all of the training practices advocated in previous sections. First, the
program’s structure and concept matches closely the segmented training strategy advocated in
the previous section: training for Routine expertise followed by training for Adaptive expertise.
The strategy is progressive (simple to complex performance requirements) and emphasizes
knowledge, task-related skills, and job-related competencies. Individual and crew skill
development also is explicitly tracked and reported.
One area in which the program does not match the suggestions outlined in previous sections is
training time. Operators and crews are expected to be Table XII certified within 270 days of
arrival in the unit (180 days for Table VIII). It is also implied that Table XII is the last formal
level of certification for crews. Norman’s 5,000-hour rule (two years of full-time effort) for the
development of entry-level expertise suggests that 270 days is insufficient for operators and
crews to develop the level of expertise required to perform competently in a contemporary air
battle setting. Crew certification at these points in time (180 and 270 days) might be a practical
necessity for unit status reporting purposes, but rigorous development of operator and crew
knowledge, skills, and competencies must continue into the Sustainment block for the duration
of a Soldier’s assignment to the unit. Unit commanders and training managers must not equate
certification and qualification.
19
To highlight this point, Hawley, Mares, and Giammanco (2006) report training time results for
two benchmark comparison cases: Israeli Patriot and FAA en route air traffic controllers
(ATCs). In the case of Israeli Patriot, a minimum of 31 months of progressive training
interspersed with unit experience is required before full certification as a Tactical Control
Officer. For FAA ATCs, job preparation requirements are similar. Following successful
completion of four months of mostly hands-on training at the FAA Academy, ATC candidates
return to their assigned center for an additional two to three years of apprenticeship. The length
of the apprenticeship period depends on the complexity of the ATC sectors at that center. Total
job preparation time from entry into training until full certification as an en route ATC is 28-40
months. Readers should note that job preparation times in both of these comparison cases are
consistent with Norman’s 5,000-hour rule.
A second area of concern is the quality of training and evaluation within each of the instructional
blocks comprising the gunnery certification program. 5-7 ADA’s documentation calls for the
proper things to be done in each these areas. However, units are turbulent places, and conflicting
and over-stressing time demands are the norm. It is thus essential that commanders and training
managers institute an effective training quality assurance (QA) oversight program to ensure that
the right things are done in spite of the impediments that characterize life in contemporary Army
units. Key points of emphasis in the QA program must include:
● Training delivery and instructor quality
● Scenario content
● Explicit task performance standards
● Guidelines for assessing hands-on job competence at each of the certification gates
● The integrity of the unit’s performance evaluation program: No rubber rulers!
● The quality and conduct of AARs

● The development of crew and team leaders
Recall that job expertise is a function of knowledge, skills, and experience. Time is an important
consideration in the development of expertise. Expert job performers cannot be produced in a
few short weeks or months of job preparation. The quality of instructional experiences also is an
important consideration in the development of expertise. Inadequate scenario content or
ineffective feedback will not lead to desired levels of expertise, regardless of the time devoted to
training.
20
5. Key Points
Achieving the objectives advocated in the previous sections will not be easy. To begin,
commanders and training managers must come to grips with issues of time and the quality of
training experiences: How much training time and what kinds of experiences are required to
produce qualified (as opposed to certified) operators? They must also recognize that the Army’s
crew drill mentality is a major part of the problem associated with preparing Soldiers for
knowledge-intensive jobs. The crew drill mentality discourages adaptive problem solving and
almost guarantees a drift toward automatic, unthinking operating procedures. Crew drills are
appropriate for some AMD job situations, but inappropriate in others. They are not a suitable
training method for Air Defense Operations.
In the report Training for Future Conflicts, the Defense Science Board asserts that the future will
require that more of our people do new and more complicated things (DSB, 2003). That same
report also remarks that meeting this challenge amounts to a “qualitative change in the demands
upon our people that cannot be supported by traditional training practices” (p. 6). The DSB
report concluded that training transformation to support warfare transformation will be a
challenging undertaking. Old training concepts and practices will have to change. It has been
noted that we often resist changing how we implement change. And it is too easy to fall back
into old, familiar behavior patterns. Real change requires sustained real changes. If training
changes are to persist, they must be clearly linked to unit procedures and routines.
The key points for AMD commanders and training managers to take away from this report are
summarized as follows:
● Training individuals, crews, and teams to be flexible, observant, and adaptive is a rigorous
and time-consuming undertaking for any organization. It requires commitment, discipline,
and the proper resources.
● Training for adaptive expertise must be viewed as a series of developmental experiences
that occur across a series of different environments. These developmental experiences are
intended to support Norman’s (1993) phases in the development of expertise: accretion,
tuning, and restructuring. Moreover, if highly-skilled human performance is required to
exploit a system’s capabilities, there is no way to avoid Norman’s 5,000 hour rule for the
development of entry-level expertise. In training, you get what you pay for and are willing
(or required) to settle for.
● Adaptive expertise and team coordination are contextually based. Adaptive skills are best
developed and refined using intact crews and teams working in the job performance
environment. Research has consistently shown that the majority of critical job-related
learning happens, always has happened, and always will happen on the job (Whitmore,
2002).
21
● Effective crew and team training is founded on solid individual technical training.
● Teams of experts do not necessarily evolve into expert teams. Targeted interventions are
required to achieve this status. These interventions include training in teamwork and
coordination skills like that found in Crew Resource Management. Grubb et al. (1995) also
report results from Army aviation indicating that crew coordination training produced
results equivalent to the sustained use of intact aircrews (so-called “battle rostering”).
These results suggest that if it is not possible to rely upon intact crews and teams (due to
personnel turbulence, etc.), intensive individual preparation coupled with crew
coordination training based on CRM principles might provide a suitable alternative. Strong
individual taskwork training coupled with teamwork training appears to compensate
somewhat for the performance benefits obtained from using intact crews consistently
training and working together.
● As the focus of individual, crew, and team skill acquisition shifts to the job performance
setting, increased emphasis must be placed on the role of crew and team leaders as mentors
and team developers.
One can argue that the most important step in reforming AMD training is to adjust current
practices to reflect emerging realities. However, to bring about long-lasting training reform, it is
necessary to correct the thinking that underlies current training practices. Immediate technical
corrections affect only the problem that is fixed and then only for the moment. Thought-process
corrections affect an organization’s ability to plan, adapt, and succeed in future actions.
Commanders and training managers must understand (1) why training change is necessary, (2)
what must be done to reform training, and (3) why those actions must be taken. The intent of
this report has been to contribute to these objectives.
In training, there are few silver bullets. Long experience across multiple performance settings
indicates that learning is hard work. It requires diligence, discipline, and a commitment to
performance excellence. Commanders and training managers deceive themselves if they believe
that high-performing individuals and crews can be developed quickly without an extended period
of hard work and discipline, or that technology will provide a way to avoid that hard work and
discipline. Technology can provide a means to develop skills that might otherwise be
problematic and might improve training efficiency, but it does not eliminate the underlying
requirement for hard work and discipline.
22
6. References
Cannon-Bowers, J. A.; Burns, J. J.; Salas, E.; Pruitt, J. S. Advanced technology in scenario-
based training. In J. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress:
Implications for individual and team training. Washington, DC: American Psychological
Association, 1998.
Defense Science Board. Training for future conflicts. (Final Report of the DSB Task Force on
Training for Future Conflicts). Washington, DC: Office of the Undersecretary of Defense
for Acquisition, Technology, and Logistics, 2003.
Defense Science Board. Patriot system performance. (Final Report of the DSB Task Force on
Patriot System Performance). Washington, DC: Office of the Undersecretary of Defense for
Acquisition, Technology, and Logistics, 2004.
Dekker, S. The field guide to human error investigations. Burlington, VT: Ashgate, 2002.
Endsley, M. R.; Bolte, B.; Jones, D. G. Designing for situation awareness: An approach to
user-centered design. New York: Taylor & Francis, 2003.
Ericsson, K. A.; Charness, N. Expert performance: Its structure and acquisition. American
Psychologist 1994, 49 (8), 725-747.
Grubb, G. N.; Simon, R. A.; Leedom, D. K.; Zeller, J. L. Effect of crew composition on AH-64
attack helicopter mission performance and flight safety; ARI Research Report 1674;
Alexandria, VA: US Army Research Institute for the Behavioral and Social Sciences, 1995.
Hawley, J. K.; Mares, A. L.; Giammanco, C.A. (). The human side of automation: Lessons for
air defense command and control; ARL-TR-3468; U.S. Army Research Laboratory:
Adelphi, MD, 2005.
Hawley, J. K.; Mares, A. L. Developing effective human supervisory control for air and missile
defense systems; ARL-TR-3742; U.S. Army Research Laboratory: Adelphi, MD, 2006.
Hawley, J. K.; Mares, A. L.; Giammanco, C. A. Training for effective human supervisory
control of air and missile defense systems; ARL-TR-3765; US Army Research Laboratory:
Adelphi, MD, 2006.
Helmreich, R. L.; Merritt, A. C.; Wilhelm, J. A. The evolution of Crew Resource Management
in commercial aviation. International Journal of Aviation Psychology 1999, 9 (1), 19-32.
Klein, G. Intuition at work. New York: Doubleday, 2003.
23
Kozlowski, S.W.J. Training and developing adaptive teams: Theory, principles, and research.
In J. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for
individual and team training. Washington, DC: American Psychological Association, 1998.
Norman, D. A. Things that make us smart. Cambridge, MA: Perseus Books, 1993.
Norman, D. A. The design of everyday things. New York: Basic Books, 2002.
Rasmussen, J. Information processing and human-machine interaction: An approach to
cognitive engineering. New York: North Holland, 1986.
Whitmore, P. G. How to make smart decisions about training: Save time, money & frustration.
Atlanta, GA: CEP Press, 2002.
24
Appendix A. Task Matrix
Appendix 4 (Task Matrix-FU) to (5-7 ADA Battalion Gunnery Certification Check Ride Program)
Number Table Basic System Skills (Individual)

1 1 Perform Emplacement Drills IAW ECS and RS ARTEP 44-635-13 Drill
2 1 Perform initialization drills IAW ECS ARTEP 44-635-13 Drill
3 1 Perform TWUD/Baseline Fault Recognition IAW TM 9-1430-600-10-1
4 1 Perform March Order Drills IAW ECS and RS ARTEP 44-635-13 Drill
5 1 Perform ECS PMCS JAW TM 9-1430-600-10
6 1 Perform RS PMCS IAW TM 9-1430-601-10.
7 1 Perform EPP PMCS IAW TM 9-6115-669-13&P
8 1 Perform Communications PMCS JAW applicable TMs
9 1 Perform MO&E Of MCPE/GPFU And Corner Reflectors
10 1 Perform March Order And Emplacement Of The ECS
11 1 Perform Automatic Emplacement Initialization Drills
12 1 Perform Manual Emplacement Initialization Drills
13 1 UHF Initialization IAW Communications Plan
Number Table Ready for Action Drill (Individual)
1 2 Learn the function of all Tabs IAW FM 3-01.87, ST 44-85-3, and TM 9-1430-600-10-1
2 2 Learn the functions of all switches/indicators IAW FM 3-01.87, ST 44-85-3, and TM 9-1430-600-10-1
3 2 Recognize Patriot symbols IAW FM 3-01.87, ST 44-85-3, and TM 9-1430-600-10-1
4 2 Learn situational Display IAW TM 9-1430-600-10-1
5 2 Learn alert states, SSTO, and ACO IAW T S O P
6 2 Perform “READY FOR ACTION” (RFA) drills IAW TSOP
Number Table Basic Air Battle Management (Individual)
1 3 Learn the TSOP
2 3 Learn the Fire Unit capabilities IAW FM 3-01.87, ST 44-85-3, and TM 9-1430-600-10-1
3 3 Learn basic TBM/ABT defense design IAW FM 3-01.87, ST 44-85-3, and TSOP
4 3 Conduct Air Defense operations against ABT/TBM (RAL 1-4)
5 3 Learn Reporting Requirements IAW TSOP, FSOP and ARTEP 44-637-30-MTP Task 44-4-9023
6 3 Learn and Apply Fix or Fight Criteria
Number Table Basic Gunnery Certification (Individual)
1 4 Perform operator level PMCS on the M927 5 Ton Truck IAW applicable TM
2 4 ECS crew members perform operator and organizational level PMCS on the ECS IAW applicable TM
3 4 All crewmembers licensed to drive their equipment regardless of assigned crewmember position.
4 4 Initialize the PATRIOT system with Mapping as required by the terrain. (IV-A)
5 4 Emplace/Prepare the ECS for Tactical Operations. (IV-A)
6 4 Conduct “ready for action” drills (IV-A)
7 4 Conduct Air Defense operations against hostile aircraft and TBMs, (RAL 5). (IV-B)
8 4 Demonstrate Knowledge of proper Reporting Requirements. (IV-C)
9 4 Demonstrate knowledge of PATRIOT system and TSOP. (IV-C)
25
Number Table Intermediate Air Battle Management Under Varying NBC Conditions (Collective)
1 5 Create an ABT/TBM defense design for Table 5-8 RAL scenario
2 5 Process Air Space Coordination Order (ACO) and MEZ information IAW ATO/SPINS
3 5 Create Site Data Books and System Data Book
4 5 Conduct Air Defense operations against hostile aircraft/TBMs. (RAL 6-9)
5 5 Perform Missile Misfire Procedures IAW
Number Table Daytime March Order and Emplacement Under Varying NBC Conditions (Collective)
1 6 March Order of the PATRIOT Missile System
2 6 Emplace the PATRIOT Missile System and prepare for action
3 6 Establish UHF/FM communications
4 6 Establish and Operate a Command Post
5 6 Conduct Reconnaissance Selection Occupation Position (RSOP)
Number Table Practice Tables V&VI Under Varying NBC Conditions (Collective)
1 7 Plan and Conduct RSOP
2 7 Perform Unit March Order
3 7 Perform Unit Emplacement
4 7 Conduct Air Defense operations against hostile aircraft
5 7 Conduct Air Defense operations against TBMs
6 7 Perform Missile Reload
7 7 Set up and Operate a CP
Number Table Intermediate Level Gunnery Certification Under Varying NBC Conditions (Collective)
Number Table Advanced Air Battle Management Under Varying NBC Conditions (Collective)
1 9 Create an ABT/TBM defense design for Table 9-12 RAL scenario
2 9 Process Air Space Coordination Order (ACO) and MEZ information IAW ATO/SPINS
3 9 Create Site Data Books and System Data Book
4 9 Conduct Air Defense operations against hostile aircraft/TBMs. (RAL 12-15)
5 9 Perform Missile Misfire Procedures IAW
Number Table Night Time March Order and Emplacement Under Varying NBC Conditions (Collective)
1 10 March Order of the PATRIOT Missile System
2 10 Emplace the PATRIOT Missile System and prepare for action
3 10 Establish UHF/FM communications
4 10 Establish and Operate a Command Post
5 10 Conduct Reconnaissance Selection Occupation Position (RSOP)
26
Number Table Practice Tables IX&X Under Varying NBC, Conditions (Collective)
Number Table Advanced Level Gunnery Certification Under Varying NBC Conditions (Collective)
27
28
Appendix B. Progress Report
29
Appendix 3 (Progress Report) to (5-7 ADA Battalion Gunnery Certification Check Ride Program)
Unit: C/BTRY 6-43 ADA TCO/TCA PROGRESS REPORT Date: 30 OCTOBER 94

Arrival T-I T-II T-III T-IV T-V T-VI T-VII T-VIII T-IX T-X T-XI T-XII
Name:
Date: Date: Date: Date: Date: Date: Date: Date: Date: Date: Date: Date: Date:
SGT 04 May 09 May 24 May 10 June 20 June 06 Aug 29 Aug 10 Sept 24 Sept 28 Sept 30 Sept
4 APR 94 94 94 24 Oct 94
JONES 94 94 94 94 94 94 94 94 94
SAMPLE FORM
30
Appendix C. Crew Progress Report
31
Appendix 3 (Progress Report) to (5-7 ADA Battalion Gunnery Certification Check Ride Program)
Unit: C/BTRY 6-43 ADA CREW PROGRESS Date: 30 OCTOBER 94
Crew: CREW #3
Name/Rank
Arrival Date: Table-VIII Date: Table-XII Date:
:
TCO
2LT Farrell 17 Mar 94 10 Sept 94 24 Oct 94
TCA
SGT Jones
SAMPLE FORM 04 Apr 9 10 Sept 94 24 Oct 94
32
31R
SPC Lease 26 Oct 93 10 Sept 94 24 Oct 94

Distribution List
ADMNSTR ARMY RSCH LABORATORY - HRED

DEFNS TECHL INFO CTR ATTN AMSRD-ARL-HR-MZ A DAVISON
ATTN DTIC-OCP (ELECTRONIC COPY) 199 E 4TH ST STE C TECH PARK BLDG 2
8725 JOHN J KINGMAN RD STE 0944 FT LEONARD WOOD MO 65473-1949
FT BELVOIR VA 22060-6218
ARMY RSRCH LAB-HRED
DARPA ATTN AMSRD-ARL-HR-MU
ATTN IXO S WELBY M SINGAPORE
3701 N FAIRFAX DR 6501 E 11 MILE RD MS 284 BLDG 200A 2ND
ARLINGTON VA 22203-1714 FL RM 2104
WARREN MI 48397-5000
OFC OF THE SECY OF DEFNS
ATTN ODDRE (R&AT) ARMY RSRCH LABORATORY-HRED
THE PENTAGON ATTN AMSRD-ARL-HR-MQ
WASHINGTON DC 20301-3080 M R FLETCHER
AMSRD-NSC-SS-E BLDG 3 RM 341
ARL HRED AMEDD FLD ELMT NATICK MA 01760-5020
ATTN AMSRD-ARL-HR-MM V RICE
BLDG 4011 RM 217 1750 GREELEY RD ARMY RSRCH LABORATORY-HRED
FT SAM HOUSTON TX 78234-5094 ATTN AMSRD-ARL-HR-ML J MARTIN
MYER CENTER RM 2D311
ARL-HRED AMCOM FLD ELMT FT MONMOUTH NJ 07703-5601
ATTN AMSRD-ARL-HR-MO J MINNINGER
BLDG 5400 RM C-242 ARMY RSRCH LABORATORY-HRED
REDSTONE ARSENAL AL 35898-7290 ATTN AMSRD-ARL-HR-MG R SPINE
BUILDING 333
US ARMY RSRCH DEV AND ENGRG CMND PICATINNY ARSENAL NJ 07806-5000
ARMAMENT RSRCH DEV AND ENGRG CTR
ARMAMENT ENGRG AND TECHNLGY CTR ARMY RSRCH LABORATORY-HRED
ATTN AMSRD-AAR-AEF-T J MATTS ATTN AMSRD-ARL-HR-MK J REINHART
BLDG 305 10125 KINGMAN RD
ABERDEEN PROVING GROUND MD 21005- FT BELVOIR VA 22060-5828
5001
ARMY RSRCH LABORATORY-HRED
ARMY G1 ATTN AMSRD-ARL-HR-M M STRUB
ATTN DAPE-MR B KNAPP 6359 WALKER LANE SUITE 100
ARMY G1 MANPRINT DAPE MR ALEXANDRIA VA 22310
300 ARMY PENTAGON RM 2C489
WASHINGTON DC 20310-0300 ARMY RSRCH LABORATORY-HRED
ATTN AMSRD-ARL-HR-MN R SPENCER
ARMY RSCH LAB-HRED JFCOM JOINT DCSFDI HF
EXPERIMENTSATION J9 JOINT FUTURES HQ USASOC BLDG E2929
LAB FT BRAGG NC 28310-5000
ATTN AMSRD-ARL-HR-MJK J
HANSBERGER ARMY RSRCH LABORATORY-HRED
115 LAKEVIEW PARKWAY STE B ATTN AMSRD-ARL-HR-MW E REDDEN
SUFFOLK VA 23435 BLDG 4 RM 332
FT BENNING GA 31905-5400
33
ARMY RSRCH LABORATORY-HRED DIRECTOR
ATTN AMSRD-ARL-HR-MY J CHEN DIRECTORATE OF COMBAT
12423 RESEARCH PARKWAY DEVELOPMENTS
ORLANDO FL 32826-3276 ATTN COL H L COHEN
5800 CARTER RD
ARMY RSRCH LABORATORY-HRED FT BLISS TX 79916-7001
ATTN AMSRD-ARL-HR-MD T COOK
BLDG 5400 RM C242 DIRECTOR
REDSTONE ARSENAL AL 35898-7290 DIRECTORATE OF TRAINING, DOCTRINE,
& LEADER DEVELOPMENT
ARMY RSRCH LABORATORY-HRED ATTN COL R K CARL
ATTN AMSRD-ARL-HR-MH C BURNS 2 SHERIDAN RD, BLDG 2
BLDG 1467B RM 336 THIRD AVE FT BLISS TX 79916-7001
FT KNOX KY 40121
OFFICE, CHIEF OF AIR DEFNS ARTILLERY
ARMY RSRCH LABORATORY-HRED ATTN LTC J H JENKINS III
ATTN AMSRD-ARL-HR-MP 2 SHERIDAN RD BLDG 2
D UNGVARSKY FT BLISS TX 79916-7001
BATTLE CMD BATTLE LAB
415 SHERMAN AVE UNIT 3 PM TIMS, PROFILER (MMS-P) AN/TMQ-52
FT LEAVENWORTH KS 66027-2326 ATTN B GRIFFIES
BUILDING 563
ARMY RSRCH LABORATORY-HRED FT MONMOUTH NJ 07703
ATTN AMSRD-ARL-HR-MV
HQ USAOTC S MIDDLEBROOKS SMC/GPA
91012 STATION AVE RM 111 2420 VELA WAY STE 1866
FT HOOD TX 76544-5073 EL SEGUNDO CA 90245-4659
ARMY RSRCH LABORATORY-HRED TRADOC CAPABILITY MANAGER-LOWER
ATTN AMSRD-ARL-HR-MY M BARNES TIER
2520 HEALY AVE STE 1172 BLDG 51005 ATTN COL R L DELGADO
FT HUACHUCA AZ 85613-7069 BLDG 12, PERSHING RD
FT BLISS TX 79916-7001
ARMY RSCH LABORATORY - HRED
AVNC FIELD ELEMENT TRADOC CAPABILITY MANAGER-UPPER
ATTN AMSRD-ARL-HR-MJ D DURBIN TIER
BLDG 4506 (DCD) RM 107 ATTN COL S PETERS
FT RUCKER AL 36362-5000 BLDG 12, PERSHING RD
FT BLISS TX 79916-7001
US ARMY TRADOC
BATTLE LAB INTEGRATION & TECHL COMMANDING GENERAL
DIRCTRT US ARMY AIR DEFNS ARTILLERY CTR
ATTN ATCD-B AND FT BLISS
10 WHISTLER LANE ATTN MG R P LENNOX
FT MONROE VA 23651-5850 FT BLISS TX 79916-7001
COMMANDANT USAADASCH COMMANDING GENERAL
ATTN AMSRD-ARL-HR-ME US ARMY AVN & MIS CMND
A MARES (10 COPIES) ATTN AMSAM-RD W C MCCORKLE
ATTN ATSA-CD REDSTONE ARSENAL AL 35898-5000
5800 CARTER RD
FT BLISS TX 79916-3802 US ARMY INFO SYS ENGRG CMND
ATTN AMSEL-IE-TD F JENIA
FT HUACHUCA AZ 85613-5300
34
US ARMY RSCH LABORATORY
ATTN AMSRD-ARL-CI-OK-TP S FOPPIANO
BLDG 459
ABERDEEN PROVING GROUND MD 21005
US ARMY RSRCH LAB
ATTN AMSRD-ARL-CI-OK-TP
TECHL LIB T LANDFRIED (2 COPIES)
BLDG 4600
ABERDEEN PROVING GROUND MD 21005-
5066
US GOVERNMENT PRINT OFF
DEPOSITORY RECEIVING SECTION
ATTN MAIL STOP IDAD (UNLIMITED SR &
TR REPORTS ONLY) J TATE
732 NORTH CAPITOL ST., NW
WASHINGTON DC 20402
US ARMY RSRCH LAB
ATTN AMSRD-ARL-HR-S L PIERCE
ATTN AMSRD-ARL-HR-SE D HEADLEY
ATTN AMSRD-ARL-HR-SE K COSENZO
BLDG 459
ABERDEEN PROVING GROUND MD 21005
US ARMY RSRCH LAB
ATTN AMSRL-ARL-HR-MR F PARAGALLO
BLDG 459
ABERDEEN PROVING GROUND MD 21005-
5066
DIRECTOR
US ARMY RSRCH LAB
ATTN AMSRD-ARL-RO-EV W D BACH
PO BOX 12211
RESEARCH TRIANGLE PARK NC 27709
US ARMY RSRCH LAB
ATTN AMSRD-ARL-HR-ME
J HAWLEY (10 COPIES)
FT BLISS TX 79916
US ARMY RSRCH LAB
ATTN AMSRD-ARL-D J M MILLER
ATTN AMSRD-ARL-CI-OK-T
TECHL PUB (2 COPIES)
ATTN AMSRD-ARL-CI-OK-TL
TECHL LIB (2 COPIES)
ATTN IMNE-ALC-IMS
MAIL & RECORDS MGMT
ADELPHI MD 20783-1197
35
36

Ada 464724

Uploaded by

Copyright:

Available Formats

Ada 464724

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ada 464724

Uploaded by

Copyright:

Available Formats

Developing Effective Adaptive Missile Crews and Command

and Control Teams for Air and Missile Defense Systems

by John K. Hawley and Anna L. Mares

ARL-SR-149 March 2007

Approved for public release; distribution unlimited.

Citation of manufacturer’s or trade names does not constitute an official endorsement or

ARL-SR-149 March 2007

Developing Effective Adaptive Missile Crews and Command

John K. Hawley and Anna L. Mares

Approved for public release; distribution unlimited.

March 2007 Final

6. AUTHOR(S) 5d. PROJECT NUMBER

John K. Hawley and Anna L. Mares 5e. TASK NUMBER

U.S. Army Research Laboratory

Approved for public release; distribution unlimited.

2. The AMD Training Problem and Its Solution 5

3. Turning Theory into Practice 9

4. Putting It All Together 17

Appendix A. Task Matrix 25

Appendix B. Progress Report 29

Figure 1. The recognition-primed decision model......................................................................... 7

The Patriot system should migrate to more of a “man-in-the-loop” philosophy

2. How does AMD training have to change to meet these challenges?

2. The AMD Training Problem and Its Solution

Source: Adapted from Klein (2003)

Figure 1. The recognition-primed decision model.

3. Operator ability levels

8. Immediate cognitive workload.

3. Turning Theory into Practice

3.1 Training Preliminaries

1. Specify the job performances we want trainees to learn to do.

Source: Adapted from Cannon-Bowers, Burns, Salas, & Pruitt (1998)

Figure 2. Scenario-based training cycle.

In performance-oriented training, feedback is just as important as performance evaluation. For

3.2 Training for Routine Expertise

3.2.1 Scenario Content

3.2.2 Performance Standards: Task Standards and Job Competencies

3.2.3 Time to Reach Desired Competency Levels

3.2.4 Qualified Instructors

3.2.5 Performance Diagnosis and Feedback

3.3 Training for Adaptive Expertise

3.3.1 Preliminary Issues

1. Mission planning and evaluation

3. Enhancing situation awareness

4. Putting It All Together

4.1 An Overview of 5-7ADA’s Gunnery Certification Program

4.1.1 Basic Skills (Tables I–IV)

● Knowledge of basic AMD tactics, communications, and system capabilities.

● A “Go” rating on all tasks applicable to their crewmember position.

4.1.2 Intermediate Skills (Tables V-VIII)

Specific elements of Table VIII certification include the following:

● Demonstrated knowledge of battalion and brigade tactics.

● A “Go” rating on all tasks applicable to their crewmember position.

● Demonstrated knowledge of advanced battalion and brigade tactics.

● A “Go” rating on all tasks for their crewmember position.

4.1.4 Sustainment Training

4.2 Suggestions for Enhancement and Improvement

● Training delivery and instructor quality

● Explicit task performance standards

● The integrity of the unit’s performance evaluation program: No rubber rulers!