Naval Medical Research Unit Dayton: D S E T A M T E
Naval Medical Research Unit Dayton: D S E T A M T E
Enclosure (2)
Form Approved
OMB No. 0704-0/88
Pu~lic ~~rting burden for the collection of information is estimated to average I hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
~am~mmg the d~ta needed, an~ co~pleting and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
mcludmg suggesttons for reducmg thiS burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 121 S Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it
does not display a currently valid OMB control number.
I. REPORT DATE
3. DATES COVERED
2. REPORT TYPE
28JAN 2013
4. TITLE AND SUBTITLE
6. AUTHOR(S)
70705
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
NAMRU-D-13-22
10. SPONSOR/MONITOR'S ACRONYM(S)
II. SPONSOR/MONITOR'S REPORT
NUMBER(S)
All U.S. Marines must qualify as marksman; as a result the training demand on live fire rifle ranges is
significant. Marksmanship simulators such as the Indoor Simulated Marksmanship Trainer (ISMT) may
help relieve some of this pressure on traditional ranges, but the training effectiveness of such systems must
be evaluated and verified. The goal of the current effort was to develop a subjective survey for evaluating
marksmanship training systems, as a possible alternative to lengthy and resource-demanding training
effectiveness evaluations. A task analytic approach was used to break down the marksmanship domain, as
presented in the USMC Rifle Marksmanship Manual, into sub-tasks that were converted into training-task
statements which were arranged into a survey format. The survey asked USMC marksmanship instructors
to rate each task statement on Importance, Difficulty, Known Distance (KD) Range Training Effectiveness,
and ISMT Training Effectiveness.
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF:
17. LIMITATION
OF ABSTRACT
a. REPORT
b. ABSTRACT
c. THIS PAGE
unclassified
unclassified
unclassified
IS. NUMBER
OF PAGES
19a. NAME OF
RESPONSIBLE PERSON
46
________________________________________
C. Douglas Forcino, CAPT, MSC, USN
Commanding Officer
The views expressed in this article are those of the author and do not necessarily
reflect the official policy or position of the Department of the Navy, Department of
Defense, nor the U.S. Government.
This work was funded by work unit number 70705 .
The study protocol was approved by the Naval Medical Research Unit Dayton Institutional Review Board in compliance with all applicable Federal regulations governing the protection of human subjects.
I am an employee of the U.S. Government . This work was prepared as part of my
official duties. Title 17 U.S.C. 105 provides that Copyright protection under this
title is not available for any work of the United States Government. Title 17 U.S.C.
101 defines a U.S. Government work as a work prepared by a military service
member or employee of the U.S. Government as part of that persons official duties.
Enclosure (2)
Acknowledgements
This effort was funded by ONR Code 30, and the authors wish to express their appreciation to Mr.
George Solhan, Dr. Rudy Darken, Dr. Roy Stripling, Dr. Peter Squire, Dr. Joan Johnston, and LT Lee
Sciarini, USN, for providing sponsorship, direction, and insightful guidance.
The authors also express appreciation to: Dr. Richard Arnold and Dr. Jeffrey Phillips for laying a proper
foundation for the project; Dr. Phil Mangos for his expertise in survey development and, along with Dr.
Joseph Chandler, for guidance on the analysis of a complicated data set; Mr. Clarke Lethin, Dr. William
Becker, Ms. Rebecca Riffle, Mr. Davey Lind, and Gunner Christian Wade, USMC, for sharing their
considerable knowledge of USMC marksmanship, and for their help in finding subject matter experts to
complete the Marksmanship Training Survey. Finally, we thank the dedicated marksmanship instructors
for finding the time to complete the survey. This was truly a team effort.
Executive Summary
All U.S. Marines must qualify as marksman; as a result the training demand on live fire rifle ranges is
significant. Marksmanship simulators such as the Indoor Simulated Marksmanship Trainer (ISMT) may
help relieve some of this pressure on traditional ranges, but the training effectiveness of such systems
must be evaluated and verified. The goal of the current effort was to develop a subjective survey for
evaluating marksmanship training systems, as a possible alternative to lengthy and resource-demanding
training effectiveness evaluations. A task analytic approach was used to break down the marksmanship
domain, as presented in the USMC Rifle Marksmanship Manual, into sub-tasks that were converted into
training-task statements which were arranged into a survey format. The survey asked USMC
marksmanship instructors to rate each task statement on Importance, Difficulty, Known Distance (KD)
Range Training Effectiveness, and ISMT Training Effectiveness. Surveys were distributed to 39
instructors and 22 of these Subject Matter Experts (SMEs) returned completed surveys.
The interrater agreement for the survey was generally good, which is a basic requirement for this tool to
have utility in assessing marksmanship trainers. Importance ratings were high, with a mean of 4.3 (all of
the scales on the survey ranged from 1 to 5). This was not a surprising result since the task statements
were generated from the USMC Rifle Marksmanship Training Manual. The mean Difficulty rating was
2.8, near the scale midpoint, and respondents used a fairly wide range of the available scale indicating
that the SMEs were able to successfully rate tasks relative to each other and in a meaningful manner.
The mean KD Range training effectiveness rating was 4.4, which was significantly greater than ISMTs
mean of 2.9. Although it scored lower than the KD Range, ISMTs score indicates that it is still a
moderately effective trainer, at least. Response patterns indicated that some instructors do not use
ISMT to train certain categories of marksmanship, such as ballistics and zeroing tasks.
Based on the survey results, the following conclusions and recommendations are offered: 1) The survey
exhibited favorable characteristics as a marksmanship trainer evaluation tool, but an important future
step would be to compare the subjective effectiveness ratings from this survey to objective training
results, as they become available. 2) The survey identified a clear SME preference for the KD Range over
ISMT as a training tool for marksmanship qualification. 3) Despite this preference, the SMEs still viewed
ISMT as a useful training tool. 4) Based on the response patterns and associations between training
effectiveness, task importance, and task difficulty, the KD Range should be used to train items that are
more important and difficult, such as aiming tasks. When KD Range time is particularly scarce, a good
use of limited assets would be to divert less important and less difficult marksmanship tasks, such as
Weapons Handling items, to ISMT. 5) There was some disagreement among SMEs regarding whether or
not ISMT was used to train certain tasks (e.g., Ballistics and Zeroing tasks). It would probably be useful
for USMC training experts and decision-makers to specifically look into which ISMT features are used
and how they are used, and conversely, which features are not used, and why they are not used.
Perhaps some ISMT features are underutilized, and/or simple improvements can be made to improve
ISMTs ability to train certain tasks, ultimately easing the demand on KD Ranges.
INTRODUCTION
Marksmanship is a fundamental skill for all Marines to master. Ongoing military operations require
the United States Marine Corps (USMC) to train Marines as rapidly and effectively as possible, but the
opportunity to train on live fire ranges is limited. Marksmanship simulators such as the Indoor
Simulated Marksmanship Trainer (ISMT) may help relieve some of this pressure on traditional live fire
ranges, (a.k.a., Known Distance (KD) ranges). Simulators can offer several advantages over the
traditional KD range. Some simulators are compact and portable enough to allow Marines to train in
settings such as ships or embassies that are too confined for a KD Range (Training and Education
Command, 2010). Simulators are typically less expensive to operate because they save on the cost of
ammunition, targets, and other supplies (Training and Education Command, 2010). The ability to place
sensors on the trainees weapon also allows instructors to provide more specific feedback compared to
a KD Range, where the instructor may not be able to pinpoint deficiencies in technique so quickly and
easily.
Despite these potential benefits, determining whether simulators offer training that is comparable or
even superior to a KD Range can be difficult. A full training effectiveness evaluation (TEE) comparing a
new system to an established training method often requires a substantial amount of time and
resources, and can disrupt training schedules. Current demands on the training pipeline make such
evaluations even more challenging. The amount of time required to evaluate a new system can
approach the time required to design and build it, meaning that new systems come online before the
old system has been evaluated. As a result, improvement can be sluggish and trainers are implemented
without a full understanding of the new systems effectiveness or advantages over the system being
replaced. The USMC requires a method to evaluate training systems that is faster, less expensive, and
less disruptive while also being applicable to a wide range of simulators.
The goal of this effort is to determine if task analytic and survey techniques can be used to evaluate
the effectiveness of training systems when time and resources do not permit a full-fledged TEE. This
report describes the development, administration, and results of a survey comparing the KD Range and
ISMT as marksmanship training systems. The results of the survey, which was administered to
marksmanship subject matter experts (SMEs), will eventually be compared to objective training
outcomes from a companion study, as those results become available. That comparison will ultimately
determine the utility of this type of survey-based evaluation for assessing a training systems
effectiveness.
METHOD
The survey developed in this project is called the Marksmanship Training Survey (MTS). The MTS is
based upon an analysis of the Marine Corps Rifle Marksmanship Manual (USMC, 2001) focusing on both
the KD Range and ISMT as facilities for training marksmanship. Each facility is described below.
ISMT
ISMT is a weapons training simulator built for indoor use (see Figure 2). The training system consists
of modified M-16 rifles, a firing line, an instructor station, and an audio/visual system for displaying
simulated targets. While ISMT is compatible with a wide range of weapon systems for training
marksmanship skill, shoot/no-shoot judgment, and weapons tactics (Training and Education Command,
2010; Indoor Simulated Marksmanship Trainer Enhanced, 2008), the current effort focuses on ISMT as
used for Training Tables 1 and 2 with the M-16 rifle. For training purposes, the rifles are modified to fire
laser light instead of live ammunition. Trainees occupy positions along the firing line, and a large display
screen is located 20 feet downrange, upon which the simulated targets are projected. When the
trainee fires at the target, ISMT registers and records the point of impact of the simulated round with an
accuracy of two minutes of angle (Yates, 2004). Point of impact and important marksmanship variables
that can affect it, such as point of aim, trigger pressure, buttstock pressure, and barrel cant, can be
displayed immediately to the instructor and student. Trainee scores can be tracked over time and ISMT
can replay a trace of the shooters point of aim prior to and immediately after firing the weapon,
providing feedback to the trainee. This type of immediate feedback is designed to help instructors
correct trainees errors and improve their technique. Training in ISMT is typically conducted with one
instructor per group of approximately eight trainees, but the size of a group varies based on the size of
the ISMT facility (W. Becker, personal communication, August 23, 2012).
1) Importance for Accuracy, defined in the survey instructions as The degree to which incorrect
performance of the task would result in reduced ability to place rounds effectively. Importance was
rated from 1 (not important) to 5 (extremely important), or not applicable (NA).
2) Difficulty to Learn, defined as: Difficulty to learn reflects the total amount of time and effort
required to learn to perform a task successfully and independently, relative to all other marksmanship
training. Difficulty was rated from 1 (one of the easiest tasks to learn) to 5 (one of the most difficult to
learn of all tasks), or NA.
3) ISMT Effectiveness, defined by the question: How effective is ISMT in training the Marine to perform
this task?. ISMT effectiveness was rated from 1 (not effective) to 5 (extremely effective), or NA (the
task is not trained using this method).
4) KD Range Effectiveness, defined by the question: How effective is the KD Range in training the
Marine to perform this task?. KD Range effectiveness was rated from 1 (not effective) to 5 (extremely
effective), or NA (the task is not trained using this method).
Each of the 48 items also provided space for general open-ended comments.
The two final questions on the survey asked participants to rate the overall effectiveness of ISMT, as
well as the overall effectiveness of the KD Range, for training marksmanship. The effectiveness of each
method was rated from 1 (not effective) to 5 (extremely effective). The entire set of instructions with all
of the response options can be found in the MTS in Appendix A.
Survey Administration Procedure
With prior agreement and arrangement, an electronic copy of the survey was e-mailed to a point of
contact (POC) at the Weapons Training Battalion Quantico, Virginia. The POC then forwarded the survey
to 39 USMC M-16 marksmanship SMEs (instructors). After reading through the Instructions,
Informed Consent, and Privacy Act worksheets, the SMEs continued on to fill out the
Marksmanship Experience and Survey worksheets. The SMEs then sent completed surveys back to
the POC, who in turn forwarded them to the research team at NAMRU-D.
Participants
Twenty-six of the 39 marksmanship instructors completed and returned the MTS, for a response rate
of 67%. Three surveys were dropped due to a probable file-saving-error that resulted in identical
responses on each of 194 answers. One additional survey was dropped because the SME reported no
experience with ISMT. The remaining 22 SMEs had a mean of 6.0 years of experience with the M-16
rifle, and had been training others to use the M-16 for an average of 2.6 years. SMEs had an average of
2.2 years of experience working with ISMT.
Interrater Agreement
The MTS data were first analyzed to gauge interrater agreement, since the usefulness of a subjective
assessment tool like this one is limited by the extent to which expert raters can agree upon task ratings.
A high degree of disagreement would indicate that the outcome of the assessment will be dependent on
the particular sample of raters selected, limiting the utility of the evaluation tool. Assessing agreement
can be facilitated by the use of multiple methods, as different indices tend to yield convergent results
(LeBreton & Senter, 2008). Although the choice of index is largely a matter of personal preference
(LeBreton & Senter, 2008), we used the rwg and Average Deviation (AD) indices of agreement per item
following the advice of Burke, Finkelstein, and Dusig (1999). The rwg statistic examines the distribution of
raters responses compared to a hypothetical null distribution. We used a null distribution composed of
uniform random responses because we lacked a sound basis to identify subgroups of raters a priori
(LeBreton & Senter, 2008) or determine the nature of any potential ratings bias and could not otherwise
model the response variability (Burke, Finkelstein, & Dusig, 1999). Interrater agreement was tested at
the p = .05 level.
The AD statistic evaluates the average deviation of each rater from the mean or median of all raters.
We computed the AD statistic as an additional measure of agreement to strengthen our interpretation
of the rwg statistic. Significance of agreement was tested at the p = .05 level for individual items using
established critical values (Dunlap, Burke, & Smith-Crowe, 2003).
The AD statistic can be computed based on either mean rating scores or median rating scores. For
the purposes of measuring agreement, we computed the AD statistic using median rating scores
because these values can be more robust and sensitive in detecting agreement than values computed
using the mean (Burke et al., 1999). Appendix C presents both the rwg and AD values for each item, as
well as the critical values to reach significance at the p = .05 level.
To evaluate aggregated marksmanship categories, we examined the number of individual items
showing agreement out of the total number of items in that category. There is no widely accepted
significance test for the rwg and AD statistics at the aggregated level.
In our MTS dataset we found that SMEs agreed in their task ratings as a whole, but agreement varied
between marksmanship categories and on different dimensions (Importance, Difficulty, ISMT
Effectiveness, and KD Range Effectiveness). These variations can affect the interpretability of the findings
and will be discussed in the appropriate section of the results.
Overall Results
To gain a composite picture of the Importance and Difficulty ratings for the 48 tertiary tasks, we
calculated the overall mean for each of these two ratings. Across the 48 items, the 22 SMEs gave
Importance a mean rating of 4.31 (SD = 0.33). The mean rating for Difficulty was 2.84 (SD = 0.58). The
Top Five and Bottom Five items for Importance and Difficulty are rank ordered in Tables 1 and 2,
respectively. The entire rank orderings for Importance and Difficulty are presented in Appendices D and
E, respectively.
Table 1
Items Rated Most Important for Accuracy and Least Important for Accuracy
Importance
Rank
Item #
Task statement
Category
Mean
35
Zeroing
4.91
12
Aiming
4.86
Aiming
4.82
Aiming
4.82
Aiming
4.73
44
Aiming
3.91
45
15
46
19
47
41
48
11
Weapons
handling
Weapons
handling
3.68
3.68
Ballistics
3.59
Trigger control
3.55
10
Table 2
Items Rated Most Difficult to Learn and Least Difficult to Learn
Difficulty
Rank
Item #
Task statement
Category
Mean
44
Ballistics
4.14
48
Aiming
4.06
10
Trigger control
4.05
37
Aiming
3.86
40
Ballistics
3.86
44
15
45
19
46
25
Firing positions
2.18
47
Aiming
2.00
48
24
Firing positions
1.82
Weapons
handling
Weapons
handling
2.27
2.18
The two overarching questions at the end of the survey asked SMEs to provide an overall rating of KD
Range Effectiveness, as well as ISMT Effectiveness. The two overall questions demonstrated good rater
agreement (rwg and AD were 0.61 and 0.82 for ISMT Effectiveness, and 0.92 and 0.41 for KD Range
Effectiveness, respectively). The mean rating for the KD Range Effectiveness was 4.81 (SD = 0.40), versus
3.52 (SD = 0.87) for ISMT Effectiveness. A paired sample t-test showed that this difference was
significant, t(20) = 6.16, p < 0.01.
A second way to compare KD Range and ISMT Effectiveness ratings is to look at means for these
variables across all 22 respondents and all 48 tertiary tasks. When calculated in this manner, the KD
Range still had the higher rating, but ratings for each facility were lower. The mean effectiveness rating
for the KD Range dropped somewhat to 4.40 (SD = 0.24), and that for ISMT dropped to 2.92, (SD = 0.61).
The overall mean ISMT Effectiveness ratings must be interpreted with caution, however, since they
demonstrated poor rater agreement in the Firing Position and Ballistics task categories (discussed
11
below). In a later section we also apply t-tests to these KD Range vs. ISMT differences within each of the
six marksmanship categories.
Analysis by Marksmanship Category
For each of the six marksmanship categories, we calculated the average rating of Importance and
Difficulty, and these data are graphed in Figures 3 and 4, respectively. Similarly for each category, mean
ratings for KD Range Effectiveness versus ISMT Effectiveness are shown in Figure 5. In all cases, means
were calculated across all subjects and items within each category.
Weapons
handling
4.08
Firing
positions
4.34
Aiming
4.52
Trigger
control
4.11
Ballistics
4.08
Zeroing
4.58
0
Importance
12
Weapons
handling
2.67
Firing positions
2.60
Aiming
3.17
Trigger control
2.96
Ballistics
3.84
Zeroing
2.98
0
Difficulty
3.16
Weapons handling
2.97
Firing positions
4.49
4.23
3.26
Aiming
3.48
Trigger control
2.16
Ballistics
4.54
4.34
KD range
effectiveness
4.13
2.25
Zeroing
ISMT
effectiveness
4.54
3
and 7 show the percentage of questions within each category that received NA responses for the
Importance and Difficulty ratings, respectively. Likewise, Figure 8 shows these percentages for KD Range
Effectiveness and ISMT Effectiveness ratings. NA responses were mostly reserved for ISMT Effectiveness
ratings for the Zeroing and Ballistics categories, indicating that in the experience of some SMEs, ISMT is
not used to train these components of marksmanship.
Weapons handling
0.0%
Firing positions
0.0%
Aiming
2.3%
Trigger control
0.0%
Ballistics
0.0%
Zeroing
0.6%
0%
10%
20%
30%
40%
50%
20%
30%
40%
50%
Weapons handling
0.0%
Firing positions
0.0%
Aiming
1.5%
Trigger control
0.0%
Ballistics
0.0%
Zeroing
0.6%
0%
10%
14
4.0%
Weapons handling
0.0%
5.7%
Firing positions
0.0%
6.4%
Aiming
1.5%
ISMT effectiveness
0.0%
0.0%
Trigger control
KD range effectiveness
31.8%
Ballistics
0.0%
27.8%
Zeroing
2.3%
0%
10%
20%
30%
40%
50%
15
2.97
Weapons handling
4.22
3.16
Firing positions
4.49
3.26
Aiming
3.48
Trigger control
2.16
Ballistics
4.57
4.34
KD range
effectiveness
4.32
2.23
Zeroing
ISMT
effectiveness
4.66
3
Figure 9. Average Ratings of KD Range Effectiveness and ISMT effectiveness with Missing Data Pairs
Omitted
Mean Importance, Difficulty, ISMT Effectiveness, and KD Range Effectiveness means are reviewed
below for each of the six marksmanship categories. Paired sample t-tests results are also presented to
compare effectiveness ratings of KD Range versus ISMT training. Because each of the six marksmanship
categories was composed of a different number of survey items, and because the number of omitted
pairs varied, degrees of freedom for each t value vary as well. Ratings within each category
demonstrated good agreement unless otherwise noted.
Weapons Handling
Weapons Handling received a mean Importance rating of 4.08 (SD = 0.86) and a mean Difficulty rating
of 2.67 (SD = 0.98). Mean Importance and Difficulty ratings are shown in Figures 3 and 4, respectively.
For the KD Range versus ISMT effectiveness comparison, a paired samples t-test demonstrated that the
KD Range (M = 4.22, SD = 0.75) was rated as more effective at training Weapons Handling tasks than
ISMT (M = 2.97, SD = 1.00), with t(188) = 16.12, p < .001 (see Figure 9).
Firing Positions
Firing Positions received a mean Importance rating of 4.34 (SD = 0.89) and a mean Difficulty rating of
2.60 (SD = 1.23). Only four out of eight items demonstrated good agreement for Difficulty ratings on
both rwg and AD, with an additional item showing good agreement on the AD index but not rwg. Difficulty
ratings for Firing Positions as a category therefore demonstrate only modest agreement and should be
interpreted carefully. Regarding effectiveness ratings, a paired samples t-test demonstrated that the KD
Range (M = 4.49, SD = 0.73) was rated as more effective at training Firing Position tasks than ISMT (M =
3.16, SD = 1.24), with t(165) = 11.78, p < .001 (see Figure 9). These results should be interpreted with
16
caution, however, since none of the eight items in the Firing Positions category demonstrated
agreement among the SMEs on ISMT Effectiveness ratings.
Aiming
Aiming received a mean Importance rating of 4.52 (SD = 0.85) and a mean Difficulty rating of 3.17 (SD
= 1.12). For the Effectiveness comparison, a paired samples t-test demonstrated that the KD Range (M =
4.56, SD = 0.67) was rated as more effective at training Aiming tasks than ISMT (M = 3.28, SD = 1.15),
with t(246) = 16.23, p < .001 (see Figure 9).
Trigger Control
Trigger Control received a mean Importance rating of 4.11 (SD = 0.97) and a mean Difficulty rating of
2.96, (SD = 1.08). A paired samples t-test demonstrated that the KD Range (M = 4.34, SD = 0.77) was
rated as more effective at training Trigger Control tasks than ISMT (M = 3.48, SD = 1.02), with t(131) =
9.44, p < .001 (see Figure 9).
Ballistics
Ballistics received a mean Importance rating of 4.08, SD = 0.94 and a mean Difficulty rating of 3.84, SD
= 0.90. A paired samples t-test demonstrated that the KD Range (M = 4.32, SD = 0.84) was rated as
more effective at training Ballistics tasks than ISMT (M = 2.16, SD = 1.23), with t(74) = 12.80, p < .001
(see Figure 9). ISMT Effectiveness ratings should be interpreted with caution, as only one out of five
Ballistics items demonstrated good agreement.
Zeroing
Zeroing received a mean Importance rating of 4.58, SD = 0.76 and a mean Difficulty rating of 2.98, SD
= 1.13. A paired samples t-test demonstrated that the KD Range (M = 4.66, SD = 0.61) was rated as
more effective at training Zeroing tasks than ISMT (M = 2.23, SD = 1.10), with t(124) = 20.02, p < .001
(see Figure 9).
Correlations
In an exploratory effort to identify any associations between the dimensions of item Importance,
Difficulty, ISMT Effectiveness, and KD Range Effectiveness, we constructed the correlation matrix for
these four dimensions. This matrix is shown in Table 3. The significant positive correlation between
Importance and KD Range Effectiveness was fairly strong (r = .68, p < .001); items rated higher in
Importance also tended to be rated high for KD Range Effectiveness. The negative correlation between
Difficulty and ISMT Effectiveness was moderate but not quite significant at the p = .05 level (r = -.41, p =
.06), indicating that more difficult items tended to receive lower ISMT Effectiveness ratings. There was
some indication of an association between Importance and Difficulty ratings, but it was not statistically
significant (r = .37, p = .10). None of the other correlations approached significance.
17
Table 3
Correlations between Importance, Difficulty, KD Range Effectiveness, and ISMT
Effectiveness
Importance
Difficulty
Difficulty
0.37
KD Range Effectiveness
-0.20
-0.41
0.68***
-0.01
ISMT Effectiveness
KD Range Effectiveness
-0.29
Open-Ended Comments
Five SMEs provided a total of 11 general open-ended comments. The comments were wide-ranging and
included suggestions for useful teaching aids, general areas of difficulty for students (e.g., sight
alignment, eye relief), and areas where ISMT might be improved (e.g., more realistic recoil, ability to
teach various shooting positions). All of the comments are presented in Appendix F.
DISCUSSION
The goal of this effort was to determine the utility of a quickly administered subjective evaluation
tool for assessing the effectiveness of marksmanship trainers and as a possible alternative to
comprehensive TEEs. Using a task analytic approach, marksmanship training materials, and SME input,
we developed a survey which instructors and SMEs could complete in under 30 minutes. Twenty-two
USMC marksmanship instructors returned valid surveys and we discuss those results here. These data
will eventually be analyzed further to see if they are predictive of the objective training results of a
companion study comparing the effectiveness of ISMT and KD Range training methods, when those
results become available.
Interrater Agreement
Overall interrater agreement was good, indicating reliable results, with a few exceptions. Agreement
was low for ISMT Effectiveness ratings within the Firing Positions and Ballistics categories, which makes
those results more difficult to interpret. Although low agreement makes system evaluation more
challenging, it does not necessarily mean that an item is not informative. Low agreement could indicate
legitimately different groups of raters, as would result from different instructors using the system
differently, for example. Perhaps some SMEs used ISMT to train some tasks while other SMEs did not.
SMEs who did not use ISMT to train certain tasks may have rated it poorly or NA, while SMEs who used
ISMT rated it highly. This explanation seems plausible given the high agreement among SMEs on KD
18
Range effectiveness, but low agreement among the same SMEs in certain categories on ISMT
effectiveness. There may be some inconsistencies in ISMTs use, leading to different training depending
on a particular instructor or facility. Future work should investigate how ISMT is actually used by
different instructors in order to identify best practices to improve standardization and training
effectiveness.
Agreement was also low for Difficulty ratings for Firing Positions, but it is not clear why. One possible
explanation is again that instructors are using the systems differently. These differences could have
caused some trainees to learn the tasks more easily than others, depending on how instructors used the
systems.
Agreement scores must also be interpreted with special care for items or categories with a high
number of NA responses, as was the case with ISMT Effectiveness ratings for the Ballistics and Zeroing
categories. Numerous NA responses in the presence of other ratings may indicate poor agreement
despite good values for rwg and AD. Any follow on work should try to determine why some SMEs feel
that certain items are NA, whereas other SMEs have no problems in evaluating those same items.
Importance Ratings
Overall, the 48 survey items, or tertiary tasks, received high ratings for Importance. The mean
Importance rating across all items was 4.31, and the mean rating for each of six categories was above 4
(see Figure 3), which corresponded to Highly Important. High Importance ratings are not surprising
here since all items on the survey can be traced back to the Marine Corps Rifle Marksmanship Manual,
which focuses on tasks important to marksmanship. The top five items for Importance (see Table 1)
were all from the categories of Zeroing or Aiming. This result makes sense since a rifle that is not
properly zeroed and properly aimed cannot be expected to place a round accurately on target
(especially the first round, when there is not yet any feedback on point of impact). The item with the
highest Importance rating, Establish battlesight zero, had a mean rating of 4.91.
The bottom five items for Importance in Table 1 were a bit more heterogeneous, originating from the
categories of Weapons Handling, Trigger Control, Ballistics, and Aiming. It is worth noting that even the
item with the lowest rank order Importance rating, Maintain proper interrupted trigger control, still
received a mean rating 3.55. This falls approximately midway between the descriptors 3 - Moderately
Important and 4 - Highly Important, again indicating that all of the survey items were viewed as
important for marksmanship.
Difficulty Ratings
The overall mean Difficulty rating was 2.84, or just below the midpoint descriptor 3 - Approximately
half the tasks are easier to learn and half are more difficult to learn. The item rated as most difficult,
Use multiple methods to determine distance to the target, had a mean rating of 4.14. A rating of 4
corresponded to Harder to learn than most other tasks. The item with the lowest Difficulty rating was
Assume the prone position, which received a mean rating of 1.82, or close to a rating of 2 - Easier to
learn than most other tasks. For Difficulty ratings, SMEs used a fairly wide range of the available scale,
19
and their average rating was close to the scale midpoint. These two characteristics of the data indicate
that with the exception of Firing Position Difficulty ratings as discussed above, the SMEs were able to
successfully rate tasks relative to each other and in a meaningful manner.
The top five items for Difficulty (see Table 2) were from the categories of Ballistics, Aiming, and
Trigger Control. Items in the Ballistics category involved making distance judgments, which is known to
be a difficult perceptual task (Allen & Rashotte, 2006). The Aiming and Trigger Control categories
contained challenging items that require fine motor control and precise hand-eye coordination, so it is
logical that these items and categories were rated among the most difficult.
The bottom five items for Difficulty in Table 2 were from the categories of Firing Positions, Aiming,
and Weapons Handling. These five items involved fairly gross motor movement and positioning tasks
(e.g., Assume the prone position). It is likely that the gross motor and fairly simple procedural nature
of these tasks (especially as compared to the five most difficult items) resulted in their lower Difficulty
ratings.
ISMT vs. KD Range Ratings
Overall Effectiveness Ratings
When SMEs were asked to provide one overall rating for KD Range Effectiveness and one for ISMT
Effectiveness, they clearly rated the KD Range (M = 4.81) as more effective than ISMT (M = 3.52). The
same pattern of results emerged when Effectiveness means were calculated across all 48 tertiary tasks.
The mean for the KD Range was 4.40 while that for ISMT was 2.92. It is unclear exactly why ratings were
lower when means were calculated across the 48 individual tertiary items. Perhaps the increased detail
and specificity at the item level prompted SMEs to think more thoroughly and critically in their ratings.
What is clear is that the SMEs consistently rated the KD Range as more effective.
There are at least two potential explanations for the higher KD Range Effectiveness ratings. First,
since Marines are tested on the KD Range itself, not ISMT, for their marksmanship qualification, the KD
Range is certainly a good choice for learning and practicing the required skills. For example, on the KD
Range Marines learn to deal with real world weather effects (e.g., wind, atmospheric obscuration, etc.)
and experience the actual noise and recoil of firing live ammunition in their own weapons. While the
weapons converted for ISMT do produce some recoil, it is only about one-third as strong as that
produced by live ammunition, and one SME did comment on the non-realistic recoil provided by ISMT.
Thus it may be argued that tasks like re-establishing sight alignment and sight picture are not as
challenging or realistic with ISMT, and therefore the KD Range produces better training for qualification.
A second factor that may have contributed to a higher rating for the KD Range is a subjective
preference or bias for firing actual, versus simulated, weapons and ammunition. Simply put, live fire is
probably more rewarding and fun for the Marines than ISMT, and this may be a bias that affects their
ratings.
20
One point that should be kept in mind is that even though ISMT received lower mean Effectiveness
ratings than the KD Range, it still received intermediate to favorable ratings. ISMTs mean rating of 2.92
across the 48 items is slightly below, but still close to, a rating of 3 Moderately Effective. ISMTs
mean score of 3.52 for the overall question is a favorable rating, falling between Moderately Effective
and Highly Effective. The SMEs still see ISMT as a useful training tool.
21
experts may want to examine further, both for standardization purposes, and possibly for improving the
way certain features of ISMT are used.
In the previously mentioned companion study, a separate project team has plans to collect objective
data on KD Range and ISMT training Effectiveness. In that effort, two groups of Marines will each
participate in a different method of training. One group will train traditionally on the KD Range
exclusively, while the other group will replace some KD Range sessions with ISMT sessions. After
training, the marksmanship qualifying scores of the two groups will be compared. Those results will also
be compared to the results from the current survey to see to what degree, if any, our SME ratings are
predictive of qualification scores.
By several measures, SMEs in the present study rated the KD Range as the more effective training
tool. In the planned companion study, if the group trained exclusively on the KD Range has reliably
higher qualification scores, those results would be an indicator that the MTS has potential merit as a
predictor of training system effectiveness. The future project is to be conducted on a not-to-interfere
basis, and extremely busy training schedules have not yet afforded the opportunity for data collection.
Provided that study can be completed we will work with that project team to combine the results of our
efforts.
Correlations
When correlation coefficients were calculated among the dimensions of Importance, Difficulty, ISMT
Effectiveness, and KD Range Effectiveness, the only significant association was between Importance and
KD Range Effectiveness (see Table 3). Tasks that SMEs viewed as high in Importance were rated highly in
terms of KD Range Effectiveness as well. So not only did SMEs rate the KD Range as more effective, they
saw it as particularly effective for more important items. Although not quite significant at the p = .05
level, there was also a trend in the opposite direction for the correlation between Difficulty and ISMT
Effectiveness (r = -.41, p = .057), indicating that ISMT was viewed as less effective in training the more
difficult tasks. A recommendation emerging from these results would be to give KD Range priority to
tasks rated as more important and difficult. A review of Tables 1 and 2 shows that Aiming tasks tended
to fall into this category.
One other interesting correlation result was that between task Importance and Difficulty (r = .37, p =
.095). Although not statistically significant, there was some tendency for items rated as important to
also be rated as difficult. One simple explanation would be that the important items are indeed more
difficult. Another possible explanation is that the structure of the survey may have induced a confound
between the dimensions of Importance and Difficulty, as follows. The most efficient way to complete
the survey was to read the task statement, and then rate it in terms of Importance, Difficulty, ISMT
Effectiveness, and KD Range Effectiveness. The fact that Importance was immediately followed by
Difficulty may have caused the former rating to bias the latter in the same direction. If follow on work
were to be conducted it may be worthwhile to investigate new methods of survey administration, such
as having the SMEs rate all items on one dimension before moving on to the next. More data on this
topic may provide a clearer picture about any relationship between Importance and Difficulty.
22
CONCLUSIONS
Several conclusions and recommendations that can be drawn from this study are listed below.
The survey exhibited favorable characteristics as a marksmanship trainer evaluation tool, but an
important future step would be to compare the subjective effectiveness ratings from this survey
to objective training results, as they become available.
The survey identified a clear SME preference for the KD Range over ISMT as a training tool for
marksmanship qualification.
Despite this preference, the SMEs still viewed ISMT as a useful training tool. Depending on how
the data were analyzed, ISMT received mean ratings that ranged from just below Moderately
Effective to ratings that fell between Moderately Effective and Highly Effective.
Based on the response patterns and associations between training effectiveness, task
importance, and task difficulty, the KD Range should be used to train items that are more
important and difficult, such as aiming tasks. When KD Range time is particularly scarce, a good
use of limited assets would be to divert less difficult and less important tasks, such as Weapons
Handling items, to ISMT.
There was some disagreement among SMEs regarding whether or not ISMT was used to train
certain tasks, particularly Ballistics and Zeroing tasks. It would probably be useful for USMC
training experts and decision-makers to specifically look into which ISMT features are used and
how they are used, and conversely, which features are not used, and why they are not used.
Perhaps some ISMT features are underutilized, and/or simple improvements can be made to
improve ISMTs ability to train certain tasks, ultimately easing the demand on KD Ranges.
References
Allen, G.A. & Rashotte, M.A. (2006). Training metric accuracy in distance estimation skill: pictures versus
words. Applied Cognitive Psychology, 20, 171-186.
Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating
interrater agreement. Organizational Research Methods, 2(1), 49-68.
Dunlap, W.P., Burke, M.J., & Smith-Crowe. (2003). Accurate tests of statistical significance for rwg and
average deviation interrater agreement indexes. Journal of Applied Psychology, 88(2), 356-362.
Galesic, M & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of
response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360.
Indoor Simulated Marksmanship Trainer Enhanced (ISMT-E) (2008). Retrieved from
http://www.marines.mil/unit/pandr/Documents/Concepts/2008/PDF/CP08Ch3P7_Indoor_Simu
lated_Marksmanship_Trainer%E2%80%93Enhanced.pdf on January 6, 2012.
LeBrenton, J.M. & Senter, J.L. (2008). Answers to 20 questions about interrater reliability and interrater
agreement. Organizational Research Methods, 11(4), 815-852.
23
24
Appendix A
Survey instructions
NOTE: This survey is presented in an Excel workbook (see the multiple colored tabs below). Be sure to
read the brief "Informed Consent" information and the Privacy Act Statement, and to fill out the
short Marksmanship Experience worksheet as well. Please save the file often so you dont lose your
work. Finally, if you have any questions, call Henry Williams (937-938-3880), Cristina Kirkendall (937938-3913), or Eric Robinson (937-938-3919). They will be happy to help with any problems.
You will probably find it helpful to print out these instructions and rating scales and refer back to
them as you fill out the survey. You can also see the ratings scales by hovering the cursor over cells
in the top row on the "Survey" worksheet.
On this survey, you will see a variety of tasks related to marksmanship. We would like you to provide
ratings for each task statement using the scales provided below. Hovering the cursor over a task
statement cell will give you additional detail about that item. Please review all task statements and
type the number corresponding to your rating in the appropriate column. If you think that a task is not
relevant to USMC marksmanship duties, select NA.
Scales 1 & 2 concern the characteristics of the tasks themselves while Scales 3 & 4 relate to how well
ISMT and the live fire range train these tasks. Please read the definitions of the scales and rating
options provided below. When providing ratings for the tasks, you may also review the definitions and
ratings scales by hovering the cursor over the cell containing the name of the rating scale (top row).
There is also space at the end of each row for any additional comments you may have on a task.
There are two questions at the end of the survey asking you to rate ISMT and the Live Fire Range in
their general ability to train marksmanship tasks. For these items please think about the overall
capabilities of both training methods and choose the best response for each item from the dropdown
menu.
Not important
Slightly important
Moderately important
Highly important
Extremely important
Not applicable
25
Not effective (ISMT is used to train this task, but does not train the task
effectively)
Slightly effective
Moderately effective
Highly effective
Extremely effective
ISMT is not used to train this task
Not effective (The live fire range is used to train this task, but does not train
the task effectively)
Slightly effective
Moderately effective
Highly effective
Extremely effective
The live fire range is not used to train this task
26
Marksmanship Experience
1
2
Please list the weapon(s) which you train others to use (e.g., M-16
rifle, M9 pistol).
How long have you been an M-16 rifleman? Please provide both the
Year(s) and Month(s). Enter "0" if necessary.
Year(s):
Month(s):
How long have you worked with ISMT? Please provide both the
Year(s) and Month(s). Enter "0" if necessary.
Year(s):
Month(s):
27
Marksmanship Survey
Task Statement
Importance
Live Fire Additional
Difficulty to
ISMT
for
Range
Comments
Learn
Effectiveness
Accuracy
Effectiveness (Optional)
11
12
13
28
22
23
32
33
29
40
41
42
43
44
45
46
30
Appendix B
Tertiary Tasks by Category
Category
Item #
Firing
positions
14
24
25
26
27
28
29
30
Weapons
handling
15
16
17
19
20
21
22
23
39
Aiming
Trigger
control
Task Statement
Select and assume a firing position based on mobility, stability, and observation of
the target
Assume the prone position
Assume the sitting position
Assume the kneeling position
Assume the high kneeling position
Assume the medium kneeling position
Assume the low kneeling position
Assume the standing position
Don the loop sling
Use proper left hand position
Use proper placement of the rifle butt in the pocket of the shoulder
Use proper positioning of the right elbow
Maintain firm and consistent placement of the cheek against the stock from shot to
shot
Use skeletal structure to support the rifles weight when using the loop sling
Relax the muscles when using the loop sling
Adjust the body to achieve proper point of aim when using the loop sling
Use environmental features to support the weapon
1
2
3
4
5
6
12
13
37
38
47
48
If the strike of the round is known, aim the next shot an equal distance from center
mass in the opposite direction to compensate
Execute a controlled pair to a target
Engage targets while wearing the field protective mask
9
10
11
18
Ballistics
40
41
42
43
44
Zeroing
31
32
33
34
Use the windage knob to adjust the strike of the round right or left
Establish initial sight settings to serve as the starting point for initial zeroing and
subsequent adjustments
Establish BZO
Rezero in order to compensate for the effects of wind, temperature, precipitation,
and light
Use environmental cues to estimate wind direction, angle, and velocity
Use the angle between a flag and the flagpole in order to estimate wind velocity
35
36
45
46
32
Appendix C
rwg and AD statistics for each item, by category
Shaded values not significant at the p = .05 level
Importance
N
rwg
AD
22 0.27 1.05
22 0.68 0.64
22 0.65 0.68
Difficulty to
Learn
N
rwg
AD
22 0.47 0.64
22 0.49 0.82
22 0.78 0.59
ISMT
Effectiveness
N
rwg
AD
22 0.20 1.09
22 0.68 0.45
22 0.58 0.64
KD Range
Effectiveness
N
rwg
AD
22 0.48 0.77
22 0.74 0.64
21 0.83 0.59
22
0.79
0.50
22
0.73
0.45
22
0.66
0.64
22
0.76
0.45
22
0.64
0.68
22
0.65
0.59
22
0.64
0.73
22
0.79
0.50
22
0.72
0.55
22
0.45
0.77
22
0.80
0.45
22
0.79
0.50
22
0.72
0.59
22
0.49
0.73
22
0.57
0.59
22
0.79
0.41
22
0.78
0.59
22
0.64
0.73
22
0.61
0.73
22
0.79
0.50
22
0.58
0.73
22
0.60
0.68
14
0.57
0.64
22
0.64
0.73
22
0.79
0.50
22
0.60
0.59
16
0.28
1.00
22
0.68
0.45
22
22
0.68
0.73
0.55
0.45
22
22
0.64
0.30
0.64
0.91
22
22
0.15
0.18
1.05
1.05
22
22
0.77
0.73
0.45
0.55
Category Item
Weapons Don the loop sling.
Handling Use proper left hand position.
Use proper placement of the rifle butt
in the pocket of the shoulder.
Use proper positioning of the right
elbow.
Firing
Positions
33
Aiming
22
22
22
22
22
0.68
0.69
0.60
0.53
0.14
0.55
0.64
0.68
0.77
1.00
22
22
22
22
22
0.27
0.35
0.35
-0.07
0.20
0.95
0.91
0.91
1.27
0.95
22
22
22
22
18
0.19
0.15
0.22
0.18
0.27
0.95
1.05
1.05
1.05
1.06
22
22
22
22
22
0.77
0.83
0.73
0.59
0.77
0.50
0.41
0.59
0.64
0.45
22
22
22
22
22
0.83
0.92
0.48
0.77
0.85
0.18
0.18
0.82
0.50
0.27
22
22
22
22
22
0.30
0.67
0.57
0.72
0.71
1.00
0.45
0.82
0.41
0.45
22
22
22
22
22
0.64
0.56
0.51
0.34
0.43
0.55
0.64
0.77
0.86
0.64
22
22
22
22
22
0.79
0.90
0.72
0.69
0.73
0.32
0.27
0.59
0.59
0.50
22
0.54
0.41
22
0.32
0.95
19
0.52
0.84
22
0.94
0.14
22
0.64
0.36
22
0.38
0.95
19
0.84
0.32
22
0.85
0.27
22
0.38
0.77
22
0.45
0.82
22
0.32
0.95
22
0.73
0.55
22
0.94
0.14
22
0.71
0.45
21
0.27
1.00
22
0.89
0.32
22
0.73
0.45
22
0.75
0.45
22
0.36
0.86
22
0.73
0.50
20
18
0.76
0.03
0.50
0.94
22
18
0.45
0.50
0.82
0.72
19
15
0.22
0.44
1.00
0.93
22
18
0.74
0.58
0.32
0.61
34
Trigger
Control
Ballistics
Zeroing
22
22
0.79
0.55
0.41
0.77
22
22
0.49
0.73
0.68
0.59
22
22
0.39
0.38
0.91
0.86
22
22
0.74
0.86
0.55
0.32
22
22
0.64
0.68
0.64
0.55
22
22
0.44
0.60
0.77
0.68
22
22
0.44
0.60
0.73
0.59
22
22
0.46
0.68
0.68
0.50
22
0.01
1.18
22
0.57
0.77
22
0.54
0.77
22
0.67
0.68
22
0.70
0.68
22
0.52
0.73
22
0.68
0.50
22
0.78
0.59
22
0.69
0.59
22
0.32
0.95
15
0.40
0.87
22
0.62
0.64
22
0.49
0.77
22
0.56
0.77
15
0.30
1.07
22
0.49
0.82
22
0.46
0.86
22
0.74
0.50
15
0.22
1.07
22
0.62
0.68
22
0.56
0.77
22
0.74
0.55
15
0.13
1.13
22
0.67
0.68
22
0.73
0.50
22
0.60
0.68
15
0.01
1.20
22
0.64
0.64
22
0.78
0.41
22
0.51
0.77
16
0.54
0.75
22
0.68
0.55
22
0.79
0.32
22
0.39
0.91
16
0.65
0.44
22
0.83
0.41
22
0.79
0.32
22
0.49
0.82
17
0.63
0.59
22
0.89
0.32
22
0.63
0.50
22
0.32
0.95
16
0.70
0.50
20
0.83
0.45
35
Overall
Scores
Establish BZO.
Rezero in order to compensate for the
effects of wind, temperature,
precipitation, and light.
22
21
0.96
0.33
0.09
0.67
22
21
0.39
0.50
0.91
0.67
17
17
0.38
0.38
0.94
0.88
21
20
0.87
0.78
0.19
0.57
22
0.68
0.45
22
0.35
0.82
14
-0.22
1.14
22
0.60
0.68
21
0.72
0.77
21
0.42
1.05
13
-0.04
1.36
22
0.68
0.55
N
21
rwg
0.61
AD
0.82
21
0.92
0.41
36
37
Appendix D
Items Ranked by Importance Rating
Importance
Rank
1
Item #
Task Statement
Category
Mean
35
Establish BZO.
Zeroing
4.90
12
Aiming
4.85
Aiming
4.80
Aiming
4.80
Aiming
4.70
32
Zeroing
4.65
33
Zeroing
4.65
38
Aiming
4.60
31
Zeroing
4.55
10
37
Aiming
4.55
11
13
Aiming
4.50
12
25
Firing positions
4.50
13
45
Zeroing
4.50
14
Aiming
4.45
15
10
Trigger control
4.45
16
34
Zeroing
4.45
17
44
Ballistics
4.45
18
47
Aiming
4.44
38
19
24
Firing positions
4.40
20
26
Firing positions
4.40
21
46
Zeroing
4.37
22
23
Weapons handling
4.35
23
Trigger control
4.30
24
27
Firing positions
4.30
25
36
Zeroing
4.26
26
14
Firing positions
4.25
27
17
Weapons handling
4.25
28
18
Trigger control
4.25
29
28
Firing positions
4.25
30
Aiming
4.15
31
16
Weapons handling
4.15
32
22
Weapons handling
4.15
33
29
Firing positions
4.15
34
Trigger control
4.05
35
20
Weapons handling
4.05
36
42
Ballistics
4.05
37
43
Ballistics
4.05
38
21
Weapons handling
4.00
39
39
Weapons handling
4.00
40
40
Ballistics
3.95
39
41
48
Aiming
3.94
30
42
Firing positions
3.90
43
Trigger control
3.85
44
Aiming
3.80
45
11
Trigger control
3.80
46
15
Weapons handling
3.75
47
19
Weapons handling
3.65
48
41
Ballistics
3.45
40
Appendix E
Items Ranked by Difficulty Rating
Difficulty
Rank
1
Item #
Task Statement
Category
Mean
10
Trigger control
4.10
44
Ballistics
4.10
48
Aiming
4.06
41
Ballistics
3.80
37
Aiming
3.75
40
Ballistics
3.75
12
Aiming
3.70
Aiming
3.70
38
Aiming
3.65
10
45
Zeroing
3.65
11
42
Ballistics
3.65
12
43
Ballistics
3.60
13
30
Firing positions
3.50
14
46
Zeroing
3.47
15
11
Trigger control
3.35
16
23
Weapons handling
3.30
17
Aiming
3.30
18
35
Establish BZO.
Zeroing
3.20
41
19
22
Weapons handling
3.05
20
Aiming
3.00
21
36
Zeroing
3.00
22
14
Firing positions
2.95
23
21
Weapons handling
2.95
24
18
Trigger control
2.90
25
31
Zeroing
2.80
26
29
Firing positions
2.80
27
Aiming
2.75
28
34
Zeroing
2.75
29
13
Aiming
2.70
30
26
Firing positions
2.70
31
39
Weapons handling
2.65
32
Aiming
2.60
33
Trigger control
2.60
34
28
Firing positions
2.60
35
16
Weapons handling
2.55
36
27
Firing positions
2.50
37
32
Zeroing
2.45
38
33
Zeroing
2.40
39
47
Aiming
2.40
42
40
17
Weapons handling
2.35
41
Trigger control
2.35
42
20
Weapons handling
2.35
43
Trigger control
2.35
44
25
Firing positions
2.30
45
19
Weapons handling
2.20
46
15
Weapons handling
2.10
47
Aiming
2.05
48
24
Firing positions
1.90
43
Appendix F
Open-ended comments from the demographics questionnaire and survey
Comment
Item Task Statement
NA (Demographics)
I like to shoot
SME
4
NA
(Demographics)
NA
(Demographics)
We must teach; "perfect practice makes perfect" and not "practice makes perfect"
Lots of shooters make mistakes with this on RCO (Rifle Combat Optic)
16
Diagrams for the coachs (sic) to keep on them to show proper aiming would greatly help
22
1
2
22
16
13
22
24
16
30
22
48
22
44