Concept Inventories: Tools For Uncovering

STEM Students’ Misconceptions

Jim Richardson, University of Alabama, Tuscaloosa, AL

Concept inventories are receiving increased interest from Hestenes and his graduate students administered a
STEM faculty. What are concept inventories, why the Mechanics Diagnostic Test to students in Introductory
interest, and what do I need to know about concept inven- Physics courses (4,5). Initial versions were not multiple
tories? This chapter answers these questions in the following choice, but required students to write out the answers.
order. In the first section, you will read a brief history of Common but wrong answers were used to construct the
STEM concept inventory development, which should answer multiple-choice wrong answers (termed “distractors” by
the question, “Why the interest in concept inventories?” In Hestenes) for later versions of the Mechanics Diagnostic Test
the next two sections, you will read first a short discussion and eventually the FCI.
on the theory of assessment as it applies to concept inven- While some may argue with the validity of the FCI (Does
tories and then a description of how to construct a concept it really measure what it purports to measure?), everyone
inventory. Together, these two sections should answer the agrees that the FCI was the impetus for substantial innova-
question, “What is a concept inventory?” And finally, you will tion in physics instruction. Key in this process was the “con-
read how others have used concept inventories and related version experience” of a highly regarded physicist at Harvard,
tools to improve their teaching effectiveness. Eric Mazur. The story is best told in his own words (6).
“For the past 8 years, I have been teaching an
Introductory Physics course for engineering and sci-
An Abridged History of STEM Concept ence concentrations at Harvard University. I used to
Inventories teach a fairly traditional course in an equally
The story of STEM concept inventories begins with the Force traditional lecture-type of presentation, enlivened by
Concept Inventory (FCI) (1,2), which was developed in the classroom demonstrations. I was generally satisfied
late 1980s and early 1990s by David Hestenes and several of with my teaching during these years—my students
his graduate students at Arizona State University. Hestenes did well on what I considered pretty difficult
gave the following account about the origin of the FCI (3). problems, and the feedback I received from them was
“One of my graduate students approached me with a positive.
set of questions related to Newtonian mechanics and About a year ago, however, I came across a series
asked me to give these questions to my Physics I stu- of articles by David Hestenes of Arizona State
dents. I looked at the questions and told him, ‘These University, which completely and permanently
questions are trivial. This is a waste of time.’ He finally changed my views on teaching. In these articles,
succeeded in convincing me to give the questions to Hestenes shows that students enter their first physics
my students. When I looked at the student responses, course possessing strong beliefs and intuitions about
I was astonished. Large numbers of my students had common physical phenomena. These notions are
failed to answer the questions correctly.” derived from personal experiences and color stu-

Assessment and Education Research 19

 dents’ interpretations of material presented in the dents in classes using active engagement techniques had
introductory course. Instruction does very little to larger gains on the FCI than students in traditional lecture-
change these ‘common-sense’ beliefs. Hestenes pro- style classes.
vides many examples in which students are asked to In the late 1990s, others began emulating Hestenes and
compare the forces of different objects on one developed concept inventories for electromagnetic waves
another. When asked, for instance, to compare the (9,10), signals and systems (11), strength of materials
forces in a collision between a heavy truck and a light (12–14), thermodynamics (15), materials science (16,17),
car, a large fraction of the class firmly believes the statistics (18), heat transfer (19), fluid mechanics (20),
heavy truck exerts a larger force on the light car than chemistry (21), electromagnetics (21), and circuits (21).
vice versa. My first reaction was, ‘Not my students…!’ A team of researchers at Colorado School of Mines is cur-
I was intrigued, however, and [tested my own stu- rently developing a concept inventory for thermal and trans-
dents]. The results of the test were undeniably eye- port science (22,23). This team includes assessment experts
opening. and psychologists as well as engineering professors. One of the
I spent many, many hours discussing the results unique aspects of this concept inventory is that the developers
of this test with my students one-on-one. The old identified a fundamental and pervasive misconception before
feeling of satisfaction turned more and more into a beginning work on the inventory. Ron Miller, a project prin-
feeling of sadness and frustration. How could these cipal investigator, had an “aha” experience while listening to a
undoubtedly bright students, capable of solving talk by Michelene Chi, a noted cognitive psychologist at the
complicated problems, fail on these ostensibly University of Pittsburgh. Micki Chi (as she’s known) was
‘simple’ questions? Slowly the problem revealed itself: describing how students have difficulty understanding “emer-
many students concentrate on learning ‘recipes,’ or gent” processes in science, frequently confusing them with
‘problem-solving strategies’ as they are called in text- “causal” processes. She explains the difference between causal
books, without bothering to be attentive to the and emergent processes below (24).
underlying concepts. Many pieces of the puzzle sud- “As it turns out, many science processes that students
denly fell into place: The continuing requests by stu- fail to understand (such as heat transfer, natural
dents to do more and more problems and less and selection) tend to be an emergent kind of process and
less lecturing—doesn’t the traditional lecture not the causal kind. In causal processes, the process
overemphasize problem-solving over conceptual itself can be explained by identifiable actions, with
understanding? temporal and spatial contiguity, and with an identifi-
Just a year ago, I was entirely oblivious to this able beginning and ending. For example, a baseball
problem. I now wonder how I could be fooled into game is a causal event. There is a beginning and an
thinking I did a credible job teaching Introductory ending, with temporal and spatial sequence. A team’s
Physics. While several leading physicists have written wins or losses can be attributed to certain causal
on this problem, I believe most instructors are still actions, such as how well the pitcher pitches.
unaware of it. A first step in remedying this situation An emergent kind of process, on the other hand,
is to expose the problem in one’s own class. The key, I does not have any of these properties. Instead, an
believe, is to ask simple questions that focus on single emergent process is caused by the collective aggre-
concepts. The result is guaranteed to be an eye- gation of multiple, independent causal events. For
opener even for seasoned teachers.” example, birds flocking in a V-formation may look like
Eric Mazur went on to develop a method for teaching a causal event (in that the lead bird shows the others
physics concepts more effectively in large lecture classes, where to fly). It is actually an emergent process, in
described in his book Peer Instruction (7). Another physics that each bird tends to fly in a location that has the
professor, Richard Hake, compiled FCI results from 6,000 stu- least resistance and drag. If each bird independently
dents. The students took the FCI at the beginning and again follows such a simple rule, it will result in the flock
at the end of the first physics course (8). He showed that stu- flying in a V-formation.”

20 Assessment and Education Research

The thermal and transport science concept inventory cur- Cognitive Model
rently under development at the Colorado School of Mines is
one of the best-planned and most thoroughly documented
engineering concept inventories to date. The project website,
http://www.mines.edu/research/cee/Misconceptions.html,
provides links to the project proposal (funded under the
National Science Foundation’s CCLI Assessment of Student
Achievement program) and other useful information.
Observations Interpretation
Elements of a Concept Inventory
The book Knowing What Students Know (25) by the National
Figure 1. Assessment triangle.
Research Council describes three interconnected compo-
nents of assessment: 1) a cognitive model of student
learning, 2) observations of student responses, and 3) inter-
pretation of those responses in light of the cognitive model. The third and final corner of the assessment triangle is
These components of assessment (represented in Figure 1) interpretation. Interpretation of the concept inventory is per-
are not a recipe for constructing a concept inventory, but formed at two levels. First, the developers of the concept
rather serve as criteria against which concept inventory inventory perform statistical analyses on a large number of
developers can check their progress toward developing a student responses to establish the validity (Does the inven-
quality assessment. tory measure what it says it does?) and the reliability (Are the
The cognitive model for a concept inventory is much more results repeatable?). The next paragraph describes an
than a list of topics or concepts. The cognitive model must example of the use of one statistical technique, factor
incorporate knowledge of the student learning process. It is analysis, to help determine the validity of a concept inventory.
the cognitive model that allows one to meaningfully inter- Factor analysis is a statistical technique to detect struc-
pret the results from the observations. Interpreting results ture in the relationships between variables. Imagine that a
from a concept inventory that lacks a good cognitive model concept inventory was intended to probe student under-
is analogous to a doctor interpreting a set of X-rays without standing on five distinct concepts (concepts 1 through 5).
a good understanding of human physiology. And for each concept, four questions were written to probe
The bottom-left corner of the assessment triangle in Figure 1 student understanding on that concept only (questions 1.1,
is observations. Observations of student performance in a 1.2, 1.3, and 1.4 for concept 1, etc.). The order of the ques-
concept inventory are the concept questions. Desirable qual- tions is randomized, and the concept inventory is given to a
ities of assessment observations are described in the fol- large number of students. A factor analysis is performed on
lowing excerpt from Knowing What Students Know (25). the student responses and returns five factors (factors A, B,
“The tasks to which students are asked to respond on C, D, and E). For a perfect concept inventory administered in
an assessment are not arbitrary. They must be care- a perfect world, factor A would be composed of questions
fully designed to provide evidence that is linked to the 3.1, 3.2, 3.3, and 3.4, and factor B would be composed of
cognitive model of learning. …In assessment, one has questions 5.1, 5.2, 5.3, and 5.4, and so on, with each factor
the opportunity to structure some small corner of the mapping exclusively to those questions intended to probe a
world to make observations. The assessment designer single concept.
can use this capability to maximize the value of the The results of a concept inventory are also interpreted
data collected, as seen through the lens of the under- by the instructor. For example, Hestenes and Halloun relate
lying beliefs about how students learn in the the following experience in their article, “Interpreting the
domain.” FCI” (2).

Assessment and Education Research 21

 “However, as a placement exam for accelerated or concepts. Each of the five steps is expanded in a paragraph
advanced courses, the FCI may be very useful. We below.
expect students testing below the Newtonian entry 1. Determine the concepts to be included. Because most
threshold (60%) to have difficulty with such courses. concept inventories are designed to be administered in
Our limited data support this expectation. We have about 30 minutes, the number of concepts examined by
used the FCI several times as a pretest for an Honors the inventory is limited. The developers of the thermal
section of University Physics, wherein all the students and transport science concept inventory followed a
have exceptional academic records. There is a wide process called the Delphi method, in which they asked
distribution of their FCI scores, however. Without experienced teachers to list important concepts with
exception, the students testing below the 60% which students frequently struggle. “The Delphi method
threshold had difficulty with the course, while those is based on a structured process for collecting and dis-
above did not. tilling knowledge from a group of experts by means of a
series of questionnaires interspersed with controlled
opinion feedback” (22). Selecting the content of the con-
Process for Constructing a Concept Inventory cept inventory based on expert opinion is important to
The steps for constructing a concept inventory are listed help establish the concept inventory’s validity.
below. 2. Articulate the student learning process. This is a difficult
1. Determine the concepts to be included in the inventory. step, one that requires extensive observations of the stu-
2. Study and articulate the student learning process dent thought process and thoughtful synthesis of those
regarding those concepts. observations into a cognitive model. The developers of
3. Construct several multiple-choice questions for each the thermal and transport science concept inventory
concept. incorporated cognitive psychologist Micki Chi’s cognitive
4. Administer the beta version of the inventory to as many model of emergent versus causal processes. In work on a
students as possible. Perform statistical analyses on the strength of materials concept inventory, Paul Steif iden-
results to establish validity, reliability, and fairness. tified two underlying mental constructs (see Table 1) that
5. Revise the inventory to improve readability, validity, reli- are taken for granted by most experts, but that are at the
ability, and fairness. root of many student learning difficulties in the subject
Construction of a concept inventory will involve iteration on (14). Concept inventory developers have used several
the steps outlined above. For example, the list of concepts to methods for probing student thought processes,
be examined by the inventory may change after gaining including personal interviews, focus groups, and written
insight to the student learning process. And student answers open-ended questions.
to concept questions without multiple-choice answers will 3. Construct multiple-choice questions. This is a two-step
likely provide insight regarding how students learn the process. In the first step, questions without multiple-

Table 1. Strength of Materials “Root” Constructs

Concept Description

Internal versus external force Internal rather than external forces determine stress and strain.

Displacement versus deformation Relative rather than absolute displacements determine deformation and strain.

22 Assessment and Education Research

 choice answers are generated following Eric Mazur’s 5. Revise the inventory to improve readability, validity, reli-
advice, “The key, I believe, is to ask simple questions that ability, and fairness. Analysis of the results from the beta
focus on single concepts” (6). This is harder than it version will prompt changes in the inventory to improve
sounds, but is very important because questions based the qualities listed in the sentence above. The revised
on more than one concept will make it difficult or impos- concept inventory will need to be reanalyzed, which may
sible to determine which concept the student did not prompt further changes. This is a continual process for
understand when he or she answered the question. The large, widespread, high-stakes assessments such as the
second step of writing multiple-choice questions is to SAT and ACT.
generate the multiple-choice answers. Many concept
inventory developers use the student responses from the
previous step to construct false answers based on Using Concept Inventories to
common student misconceptions (1,21,23). Student Improve Teaching
interviews and focus groups are also frequently used to The first step in advancing skills and abilities in an area is to
identify misconceptions and to ensure that the questions recognize the need for improvement. This is the principal
are worded clearly. contribution of the FCI. Eric Mazur articulately expressed his
4. Administer the beta version of the concept inventory and experience of realizing that his students weren’t learning
analyze the results statistically to establish validity, reli- what he thought they were learning. His frank description of
ability, and fairness. Validity addresses the question, his awakening, coupled with his stature in the physics
“Does the concept inventory measure what it purports to research community, encouraged many other physics
measure?” Validity is established in several ways, instructors to reexamine their teaching effectiveness. Eric
including the following: incorporating the knowledge of Mazur went on to develop his own teaching techniques (7)
experts in the field during the development process, and is currently working on a physics textbook.
checking that questions intended to probe the same con- Once you are aware that your students struggle to learn
cept illicit similar student responses, and comparing stu- the concepts in your course at a “deep” level, you can use a
dents’ concept inventory scores with observations of concept inventory to help you modify your teaching.
their ability to use concepts on traditional exam-style Assessment experts never base their evaluation on a single
problems. assessment, but rather triangulate between several inde-
Reliability addresses the question, “Does the concept pendent assessments. You too should base your evaluation of
inventory yield similar scores when repeated by the same your students’ conceptual knowledge on several assessments
student?” Reliability can be established using statistical (one of which can be a concept inventory). In the next para-
procedures. Fairness addresses the question, “Does the graph, the strengths of a concept inventory as a teaching
concept inventory produce comparably valid inferences improvement tool are discussed, and in the following para-
between students of different gender or ethnic groups?” graph, other assessment techniques that can complement a
Because the number of engineering students from concept inventory are briefly discussed.
underrepresented groups may be too small at a single A properly constructed concept inventory is based on a
university to yield statistically significant results, beta- cognitive model of student conceptual understanding in an
testing the concept inventory should be performed at area of focus. The developers should communicate that cog-
several universities. Concept inventory development nitive model to a teacher interested in using his or her con-
teams need to have experts in assessment and test con- cept inventory. Also, the concept inventory has been
struction involved at every stage of the project, espe- extensively tested to establish that it is a valid and reliable
cially the planning stages. See the project website for the measure of a student’s conceptual knowledge, as articulated
thermal and transport science concept inventory for a in the cognitive model. And finally, a properly constructed
good example of incorporating assessment expertise in database of concept inventory results can help a teacher
all phases of development (http://www.mines.edu/ interpret the significance of his or her students’ perform-
research/cee/Misconceptions.html). ance. The database needs to include results from thousands

Assessment and Education Research 23

of students on dozens of campuses if it is meant to reflect a Summary and Conclusions
national average. A concept inventory administered over the Concept inventories bring to light the disparity between
web would facilitate the collection of information about stu- what we want our students to learn and what they really
dent demographics, class size, type of school, etc., and would learn. Spurred by the success of the FCI in communicating
allow teachers to compare their students’ performance the existence of this disparity to physics teachers, education
against the performance of similar students. researchers are developing concept inventories for many dif-
Concept inventories provide a narrow view of students’ ferent subjects in science and engineering. Constructing a
conceptual knowledge, but one that is anchored to a concept inventory is a multi-year process involving a team of
coherent theory of student learning and can be interpreted professionals with expertise in both teaching the subject and
in relation to thousands of other students’ performances. in educational assessment. Teachers interested in improving
Assessments complementary to a concept inventory can pro- the effectiveness of their teaching can stand on the shoul-
vide a broader and more personal (albeit a less transferable) ders of the concept inventory developers and catch a glimpse
view of students’ conceptual knowledge. One assessment of the typical student’s thought process. Teachers can also
technique emulated by many was originally developed by use a concept inventory as a reference point for gauging
Eric Mazur. In this technique, he stops his lecture, he asks his their progress in closing the gap between actual and desired
students to discuss and then answer a multiple-choice ques- learning.
tion focusing on a single concept, and then he polls his stu- As helpful as concept inventories can be, they cannot
dents and records the number of students selecting each alone provide sufficient guidance for changing one’s
answer. Another assessment technique, described in the teaching to improve students’ conceptual knowledge. Such
excerpt below from Knowing What Students Know (25), guidance must be based on a more intimate knowledge of
focuses on how to ask students questions. The technique, students’ thought processes. The teacher must become a
suitable for smaller classes, illuminates the change in per- student of his or her pupils’ learning process. Time con-
spective required to effectively probe our students’ concep- suming? Yes. Rewarding? Definitely.
tual knowledge.
“The teachers altered their practice [of asking ques-
tions] to give students extended time to think about
the question posed, often asking them to discuss
their ideas in pairs before calling for responses. The
teachers did not label answers as right or wrong, but
instead asked a student to explain his or her reasons
for the answer offered. Others were then asked to say
whether they agreed and why. Thus, questions
opened up discussion that helped expose and explore
students’ assumptions and reasoning. At the same
time, wrong answers became useful input, and the
students realized that the teacher was interested in
knowing what they thought, not in evaluating
whether they were right or wrong.
The quality of the questions posed to the students is partic-
ularly important to the success of both techniques described
above.

24 Assessment and Education Research

 16. Krause, S., J. Decker, J. Niska, T. Alford, and R. Griffin. 2003.
REFERENCES Identifying student misconceptions in introductory materials
engineering classes. In Proceedings of the 2003 ASEE
1. Hestenes, D., M. Wells, and G. Swackhamer. 1992. Annual Conference. Washington, DC: ASEE.
Force concept inventory. The Physics Teacher. 30: 141–158.
17. Krause, S., J. Decker, and R. Griffin. 2003. Using a materials
2. Hestenes, D., and I. Halloun. 1995. Interpreting the FCI. concept inventory to assess conceptual gain in introductory
The Physics Teacher. 33: 502–506. materials engineering courses. In Proceedings of the 2003
FIE Conference. Champaign, IL: Stipes Publishing L.L.C.
3. Personal communication, Tempe AZ, March 2000.
18. Stone, A., K. Allen, T. Rhoads, T. Murphy, R. Shehab, and C.
4. Halloun, I., and D. Hestenes. 1985. The initial knowledge
Saha. 2003. The statistics concept inventory: a pilot study.
state of college physics students. Am J Phys 53: 1043–1055.
In Proceedings of the 2003 FIE Conference. Champaign, IL:
5. Halloun, I., and D. Hestenes. 1985. Common sense concepts Stipes Publishing L.L.C.
about motion. Am J Phys 53: 1056–1065.
19. Jacobi, A., J. Martin, J. Mitchell, and T. Newell. 2003.
6. Mazur, E. 1992. Qualitative versus quantitative thinking: A concept inventory for heat transfer. In Proceedings of the
are we teaching the right thing? Optics and Photonics News 2003 FIE Conference. Champaign, IL: Stipes Publishing L.L.C.
3:38.
20. Martin, J., J. Mitchell, and T. Newell. 2003. Development of a
7. Mazur, E. 1997. Peer Instruction. Upper Saddle River, NJ: concept inventory for fluid mechanics. In Proceedings of the
Prentice Hall. 2003 FIE Conference. Champaign, IL: Stipes Publishing L.L.C.

8. Hake, R. R. 1998. Interactive-engagement vs traditional 21. Evans, D., G. Gray, S. Krause, J. Martin, C. Midkiff,
methods: a six-thousand-student survey of mechanics test B. Notaros, M. Pavelich, D. Rancour, T. Reed-Rhoads, P. Steif,
data for introductory physics courses. Am J Phys 66: 64–74. R. Streveler, and K. Wage. 2003. Progress on concept inven-
tory assessment tools. In Proceedings of the 2003 FIE
9. Roedel, R., S. El-Ghazaly, T. Rhoads, and E. El-Sharawy. 1998. Conference. Champaign, IL: Stipes Publishing L.L.C.
The Wave Concepts Inventory: An Assessment Tool for
Courses in Electromagnetic Engineering. In Proceedings of 22. Streveler, R., B. Olds, R. Miller, and M. Nelson. 2003.
the 1998 FIE Conference. Champaign, IL: Stipes Publishing Using a Delphi study to identify the most difficult concepts
L.L.C. for students to master in thermal and transport science.
Proceedings of the 2003 ASEE Conference. Washington, DC:
10. Rhoads, T., and R. Roedel. 1999. The wave concept inven- ASEE.
tory: a cognitive instrument based on Bloom’s taxonomy.
In Proceedings of the 1999 FIE Conference. Champaign, IL: 23. Olds, B., R. Streveler, R. Miller, and M. Nelson. 2004.
Stipes Publishing L.L.C. Preliminary results from the development of a concept
inventory in thermal and transport science. In Proceedings
11. Wage, K., and J. Buck. 2001. Development of the signals and of the 2004 ASEE Conference. Washington, DC: ASEE.
systems concept inventory (SSCI) Assessment Instrument.
In Proceedings of the 2001 FIE Conference. Champaign, IL: 24. Michelene Chi’s website: http://www.pitt.edu/~chi
Stipes Publishing L.L.C.
25. National Research Council. 2001. Knowing What Students
12. Richardson, J., J. Morgan, and D. Evans. 2001. Development Know. Washington, DC: Board on Testing and Assessment,
of an engineering strength of materials concept inventory National Academies Press.
assessment instrument. In Proceedings of the 2001 FIE
Conference. Champaign, IL: Stipes Publishing L.L.C.

13. Richardson, J., P. Steif, J. Morgan, and J. Dantzler. 2003.

Development of a concept inventory for atrength of
materials. In Proceedings of the 2003 FIE Conference.
Champaign, IL: Stipes Publishing L.L.C.

14. Issen, K., P. Steif, and J. Richardson. 2004. Probing concep-

tual knowledge of students in strength of materials. In
Proceedings of the 17th ASCE Engineering Mechanics
Conference. Newark, NJ: University of Delaware.

15. Midkiff, K., T. Litzinger, and D. Evans. 2001. Development of

engineering thermodynamics concept inventory assessment
instruments. In Proceedings of the 2001 FIE Conference.
Champaign, IL: Stipes Publishing L.L.C.

Assessment and Education Research 25