Focus on Formative

Feedback

March 2007
RR-07-11
Research
Report
Valerie J. Shute

Research &
Development

Focus on Formative Feedback
Valerie J. Shute
ETS, Princeton, NJ
March 2007

As part of its educational and social mission and in fulfilling the organization's nonprofit charter
and bylaws, ETS has and continues to learn from and also to lead research that furthers
educational and measurement research to advance quality and equity in education and assessment
for all users of the organization's products and services.
ETS Research Reports provide preliminary and limited dissemination of ETS research prior to
publication. To obtain a PDF or a print copy of a report, please visit:
http://www.ets.org/research/contact.html
Copyright 2007 by Educational Testing Service. All rights reserved.
ETS and the ETS logo are registered trademarks of
Educational Testing Service (ETS).


Abstract
This paper reviews the corpus oI research on Ieedback, with a particular Iocus on Iormative
IeedbackdeIined as inIormation communicated to the learner that is intended to modiIy the
learner`s thinking or behavior Ior the purpose oI improving learning. According to researchers in
the area, Iormative Ieedback should be multidimensional, nonevaluative, supportive, timely,
speciIic, credible, inIrequent, and genuine (e.g., Brophy, 1981; Schwartz & White, 2000).
Formative Ieedback is usually presented as inIormation to a learner in response to some action
on the learner`s part. It comes in a variety oI types (e.g., veriIication oI response accuracy,
explanation oI the correct answer, hints, worked examples) and can be administered at various
times during the learning process (e.g., immediately Iollowing an answer, aIter some period oI
time has elapsed). Finally, there are a number oI variables that have been shown to interact with
Iormative Ieedback`s success at promoting learning (e.g., individual characteristics oI the learner
and aspects oI the task). All oI these issues will be discussed in this paper. This review concludes
with a set oI guidelines Ior generating Iormative Ieedback.
Key words: Assessment, Iormative Ieedback, directive Ieedback, Iacilitative Ieedback, learning,
perIormance, individual diIIerences, goal orientation, motivation, task characteristics
i
Acknowledgments
I would like to acknowledge the valuable assistance I received in this literature review starting
with Namrata Tognatta and Waverely Van Winkle Ior collecting the mountain oI research papers
(etc.) and then summarizing many oI them in EndNote and Excel. Waverely additionally assisted
with organizing and Iormatting the numerous reIerences used in the paper and Namrata with
counting and classiIying the reIerences. I also am indebted to Waverly, as well as Eric Hansen,
Peggy Redman, Jody Underwood, and Diego Zapata-Rivera, Ior generating two-page summaries
oI Ieedback articles that helped me decide which papers to read Iirst (and more thoroughly) than
others. And Iinally, I`m grateIul to my very smart and generous colleagues who reviewed earlier
draIts oI this review: in addition to Eric, Diego, and Jody, Jim FiIe, Malcolm Bauer, Richard
Roberts, Madeline Goodman, Pat Kyllonen, and Aurora GraI also provided valuable comments.
ii
Table of Contents
Page
1. Introduction................................................................................................................................. 1
1.1 DeIinition oI Formative Feedback........................................................................................ 1
1.2 Goals oI the Review.............................................................................................................. 2
1.3 Focus oI the Review ............................................................................................................. 3
2. Method........................................................................................................................................ 4
2.1 Procedure .............................................................................................................................. 4
2.2 Inclusion Criteria .................................................................................................................. 5
3. Literature Review........................................................................................................................ 5
3.1 Feedback Purposes................................................................................................................ 6
3.2 Cognitive Mechanisms and Formative Feedback................................................................. 6
3.3 Feedback SpeciIicity............................................................................................................. 7
3.4 Features oI Formative Feedback........................................................................................... 8
3.5 Feedback Complexity/Length............................................................................................... 9
3.6 Goal-Directed Feedback and Motivation............................................................................ 12
3.7 Formative Feedback as ScaIIolding.................................................................................... 13
3.8. Timing................................................................................................................................ 15
3.9 Feedback and Other Variables............................................................................................ 18
4. Toward a Framework oI Formative Feedback.......................................................................... 20
4.1 Kluger and DeNisi (1996)................................................................................................... 20
4.2 Bangert-Drowns, Kulik, Kulik, and Morgan (1991) .......................................................... 23
4.3 Narciss and Huth (2004) ..................................................................................................... 25
4.4 Mason and Bruning (2001) ................................................................................................. 27
5. Summary and Discussion.......................................................................................................... 29
5.1 Recommendations and Guidelines Ior Formative Feedback.............................................. 29
5.2 Future Research .................................................................................................................. 34
ReIerences..................................................................................................................................... 36
Notes ............................................................................................................................................. 47


iii
List of Tables
Page
Table 1. Feedback Types Arrayed Loosely by Complexity...................................................... 10
Table 2. Formative Feedback Guidelines to Enhance Learning (Things to Do) ...................... 30
Table 3. Formative Feedback Guidelines to Enhance Learning (Things to Avoid) ................. 31
Table 4. Formative Feedback Guidelines in Relation to Timing Issues ................................... 32
Table 5. Formative Feedback Guidelines in Relation to Learner Characteristics .................... 33
iv
List of Figures
Page
Figure 1. Abstract hierarchy oI processing. ............................................................................... 21
Figure 2. FI moderators and their relationships to learning/perIormance. ................................ 23
Figure 3. Five-stage model oI the learner during a Ieedback cycle. .......................................... 24
Figure 4. Factors interacting with Ieedback to inIluence learning............................................. 26
Figure 5. Feedback variables Ior decision making in computer-based instruction.................... 28
v
1. Introduction
It is not the horse that draws the cart, but the oats. Russian proverb
Feedback used in educational contexts is generally regarded as crucial to improving
knowledge and skill acquisition (e.g., Azevedo & Bernard, 1995; Bangert-Drowns, Kulik, Kulik,
& Morgan, 1991; Corbett & Anderson, 1989; Epstein et al., 2002; Moreno, 2004; Pridemore &
Klein, 1995). In addition to its inIluence on achievement, Ieedback is also depicted as a
signiIicant Iactor in motivating learning (e.g., Lepper & Chabay, 1985; Narciss & Huth, 2004).
However, Ior learning, the story on Ieedback is not quite so rosy or simple.
According to Cohen (1985), Ieedback '. is one oI the more instructionally powerIul and
least understood Ieatures in instructional design (p. 33). In support oI this claim, consider the
hundreds oI research studies published on the topic oI Ieedback and its relation to learning and
perIormance during the past 50 years (Ior excellent historical reviews, see Bangert-Drowns et al.,
1991; Kluger & DeNisi, 1996; Kulhavy & Stock, 1989; Kulhavy & Wager, 1993; Mory, 2004;
and Narciss & Huth, 2004). Within this large body oI Ieedback research, there are many
conIlicting Iindings and no consistent pattern oI results.
Consider just one Iacet oI Ieedback: elaboration (i.e., explanatory inIormation within a
Ieedback message). Some studies report that elaborative Ieedback produces signiIicantly greater
learning among students compared with Ieedback containing less inIormation (e.g., Albertson,
1986; Grant, McAvoy, & Keenan, 1982; HannaIin, 1983; Moreno, 2004; Pridemore & Klein;
1995; Roper, 1977; Shute, 2006). However, other studies show that increasing the amount oI
Ieedback inIormation has no eIIect on learning or perIormance (e.g., Corbett & Anderson, 1989,
1990; Gilman, 1969; Hodes, 1985; Kulhavy, White, Topp, Chan, & Adams, 1985; Merrill,
1987). The goal oI this paper is to try to make sense oI the tangled web oI Iormative Ieedback
research and its relationship to learning. I start by deIining relevant terms.
1.1 Definition of Formative Feedback
Formative Ieedback represents inIormation communicated to the learner that is intended
to modiIy the learner`s thinking or behavior Ior the purpose oI improving learning. And while
the teacher may also receive Iormative Ieedback and use it as the basis Ior altering instruction, I
Iocus on the student (or more generally, the learner) as the primary recipient oI Iormative
Ieedback in this review.
1
InIormation within the Ieedback may address the accuracy oI a response to a problem or
task and may additionally touch on particular errors and misconceptions (Azevedo & Bernard,
1995; Birenbaum & Tatsuoka, 1987; Cheng, Lin, Chen, & Heh, 2005; Cohen, 1985; Kulhavy,
1977; Sales, 1993; Sleeman, Kelly, Martinak, Ward, & Moore, 1989), the latter representing
more speciIic or elaborated types oI Ieedback. To be eIIective, Iormative Ieedback should permit
the comparison oI actual perIormance with some established standard oI perIormance (Johnson
& Johnson, 1993). The deIinition oI Iormative Ieedback may be Iurther reIined as
multidimensional, nonevaluative, supportive, learner-controlled, timely, speciIic, credible,
inIrequent, contingent, and genuine (e.g., Brophy, 1981; Schwartz & White, 2000).
In technology-assisted instruction, similar to classroom settings, Iormative Ieedback
comprises inIormationa message, display, and so onpresented to the learner Iollowing the
learner`s input (or upon request, iI applicable), with the purpose oI shaping the perception,
cognition, or action oI the learner (e.g., Moreno, 2004; Schimmel, 1983; Wager & Wager, 1985).
The main goal oI Iormative Ieedbackwhether delivered by a teacher or computer, in the
classroom or elsewhereis to enhance learning and/or perIormance, engendering the Iormation
oI accurate, targeted conceptualizations and skills. Such Ieedback may be used in conjunction
with low- or medium-stakes assessments, include diagnostic components, and even be
personalized Ior the learner (Albertson, 1986; Azevedo & Bernard, 1995; Narciss & Huth, 2004;
VanLehn, 1982).
The premise underlying most oI the research conducted in this area is that good Ieedback
can signiIicantly improve learning processes and outcomes, iI delivered correctly. Those last
three wordsiI delivered correctlycomprise the crux oI this review. This paper reviews the
literature on Ieedback, with the goal oI cataloging and categorizing key Iacets (i.e., types and
timing), in addition to other variables with which they are known to interact (e.g., learner states
and traits and task complexity).
1.2 Goals of the Review
The dual aims oI this paper are to (a) present Iindings Irom an extensive literature review
oI Ieedback to gain a better understanding oI the Ieatures, Iunctions, interactions, and links to
learning and (b) apply the Iindings Irom the literature review to create a set oI guidelines relating
to Iormative Ieedback. The overarching goal is to identiIy the particular Ieatures oI Iormative
2
Ieedback that are the most eIIective and eIIicient in promoting learning and determine under
what conditions that learning support holds.
This is not an easy task. The vast literature reveals dozens oI Ieedback types that have
been subjected to experimental scrutinyaccuracy oI the solution, topic contingent, response
contingent, attribute isolation, worked examples, hints, and partial solutions. However, diIIerent
studies report disparate Iindings regarding the same Ieedback variable. In addition, Iormative
Ieedback variables have been shown to interact with other variables, such as student achievement
level, task level, and prior knowledge. Again, my goal is to capture and integrate these Ieedback
Ieatures so to begin to deIine prescriptions relating to Iormative Ieedback and its appropriate
delivery.
1.3 Focus of the Review
This review Iocuses on task-level Ieedback as opposed to general summary Ieedback.
Task-level Ieedback typically provides more speciIic and timely (oIten real-time) inIormation to
the student about a particular response to a problem or task compared to summary Ieedback, and
it may additionally take into account the student`s current understanding and ability level. For
instance, a struggling student may require greater support and structure Irom a Iormative
Ieedback message than a proIicient student. On the other hand, summary inIormation is useIul
Ior teachers to modiIy instruction Ior the whole class and Ior students to see how they are
generally progressing.
Some oI the major questions addressed in this review include: What are the most
powerIul and eIIicient types oI Iormative Ieedback and under what conditions do these diIIerent
types oI Ieedback help a learner revise a skill or improve understanding? What are the
mechanisms by which Ieedback Iacilitates the transIormation oI rudimentary skills into the
competence oI a more expert state? Answers to these questions can Iacilitate the design and
development oI teacher-delivered or automated Ieedback to support learning.
This paper begins with a summary oI the methods used to accomplish the literature
review, Iollowed by an extensive review oI Iormative Ieedback research, which comprises the
bulk oI the paper. AIterwards, I showcase Iour important Ieedback articles, each associated with
a theoretically and empirically based model oI Iormative Ieedback. I conclude with speciIic
recommendations Ior using Iormative Ieedback that are supported by the current literature review
and discuss Iuture research in the area.
3
2. Method
2.1 Procedure
I began my research by working with colleagues to identiIy seminal articles in the
Ieedback literature (i.e., Irom sites that provide indices oI importance such as CiteSeer) and then
collecting the relevant documents. The bibliography compiled Irom this initial set oI research
studies spawned a new collection-review cycle, garnering even more articles and continuing
iteratively throughout the review process.
The Iollowing online databases were employed in our search-collection eIIorts:
ERIC (Educational Resources InIormation Center) contains inIormation on
educational reports, evaluations, and research, including data Irom Resources in
Education Index and Current Index to Journals in Education.
PsycINFO, Irom the American Psychological Association, includes citations and
summaries oI scholarly journal articles, book chapters, books, and dissertations in
psychology and related disciplines.
PsycARTICLES is a source oI Iull-text, peer-reviewed scholarly and scientiIic articles
in psychology. The database covers general psychology and specialized, basic,
applied, clinical, and theoretical research. It contains articles Irom 56 journals (45
published by the American Psychological Association and 11 Irom allied
organizations).
Academic Search Premier oIIers inIormation in many areas oI academic study
including computer science, engineering, physics, language and linguistics, and so
Iorth.
MasterFILE Premier, designed speciIically Ior public libraries, covers a broad range
oI disciplines including general reIerence, business, education, health, general
science, and multicultural issues.
In addition to these databases, other research sources included the online catalogs at the
libraries at ETS and the University oI Pennsylvania, which were used to access their electronic
collections oI journals and research studies. Another source was Google Scholara Web site
providing peer-reviewed papers, theses, books, abstracts, and articles Irom academic publishers,
4
proIessional societies, preprint repositories, universities, and other scholarly organizations, to
search Ior and acquire speciIic reIerences.
2.2 Inclusion Criteria
The review Iocused on Iull-text documents, accessed using various search terms or
keywords such as Ieedback, Iormative Ieedback, Iormative assessment, instruction, learning,
computer-assisted/based, tutor, learning, and perIormance. Searches were not limited to a
particular date range, but did include the most recent research in the literature review. In all,
approximately 170180 articles, dissertations, abstracts, books, and conIerence proceedings were
collected. From this larger set, a total oI more than100 documents met the study`s criteria Ior
inclusion into the literature review. Inclusion criteria consisted oI topical relevance, use oI
experimental design, and/or meta-analytic procedures (albeit, this selection process was more
akin to that oI community colleges than ivy-league universities). The majority oI the documents
were journal articles (103), Iollowed by books and book chapters (24), conIerence proceedings
(10), and other (e.g., research reports, 4).
3. Literature Review
As noted at the beginning oI this paper, hundreds oI articles have been written about
Ieedback and its role in knowledge and skill acquisition. Many oI these papers describe the
results Irom experimental tests examining diIIerent Ieatures oI Ieedback, and several represent
important historical reviews (a Iew going back to the early 1900s, such as Kluger & DeNisi,
1996; Kulhavy & Stock, 1989; and Mory, 2004). Despite the plethora oI research on the topic,
the speciIic mechanisms relating Ieedback to learning are still mostly murky, with very Iew (iI
any) general conclusions. Researchers who have tackled the tough task oI perIorming meta-
analyses on the Ieedback data use descriptors such as inconsistent, contradictory, and highly
variable to describe the body oI Ieedback Iindings (Azevedo & Bernard, 1995; Kluger &
DeNisi). Ten years later those descriptors still apply.
Feedback has been widely cited as an important Iacilitator oI learning and perIormance
(Bandura, 1991; Bandura & Cervone, 1983; Fedor, 1991; Ilgen, Fisher, & Taylor, 1979), but
quite a Iew studies have reported Ieedback as having either no eIIect or debilitating eIIects on
learning (Ior examples oI nonIacilitative eIIects oI Ieedback on learning, see Bangert-Drowns et
al., 1991; Kluger & DeNisi, 1996; and Mory, 2004). In Iact, about one third oI the total studies
5
reviewed in two landmark meta-analyses (i.e., Bangert-Drowns et al.; Kluger & DeNisi)
demonstrate negative eIIects oI Ieedback on learning. For instance, Ieedback that is construed as
critical or controlling (Baron, 1993) oIten thwarts eIIorts to improve perIormance (Fedor, Davis,
Maslyn, & Mathieson, 2001). Other Ieatures oI Ieedback that tend to impede learning include
providing grades or overall scores indicating the student`s standing relative to peers and coupling
such normative Ieedback with low levels oI speciIicity (i.e., vagueness) (Butler, 1987; Kluger &
DeNisi, 1998; McColskey & Leary, 1985; Wiliam, in press; Williams, 1997). In addition,
interrupting a student who is actively engaged in problem solving with Ieedback Irom an external
source has too has been shown to inhibit learning (Corno & Snow, 1986). In line with the
deIinition in this review, Ieedback that has negative eIIects on learning is not Iormative.
3.1 Feedback Purposes
The main aim oI Iormative Ieedback is to increase student knowledge, skills, and
understanding in some content area or general skill (e.g., problem solving), and there are
multiple types oI Ieedback that may be employed toward this end (e.g., response speciIic, goal
directed, immediately delivered, and so on). In addition to various Iormats oI Ieedback, there are
also diIIerent Iunctions. According to Black and Wiliam (1998), there are two main Iunctions oI
Ieedback: directive and Iacilitative. Directive Ieedback tells the student what needs to be Iixed or
revised. Such Ieedback tends to be more speciIic than Iacilitative Ieedback, which provides
comments and suggestions to help guide students in their own revision and conceptualization.
The next section describes some oI the ways that Ieedback may exert inIluences on student
learning.
3.2 Cognitive Mechanisms and Formative Feedback
There are several cognitive mechanisms by which Iormative Ieedback may be used by a
learner. First, it can signal a gap between a current level oI perIormance and some desired level
oI perIormance or goal. Resolving this gap can motivate higher levels oI eIIort (Locke &
Latham, 1990; Song & Keller, 2001). That is, Iormative Ieedback can reduce uncertainty about
how well (or poorly) the student is perIorming on a task (AshIord, 1986; AshIord, Blatt, &
VandeWalle, 2003). Uncertainty is an aversive state that motivates strategies aimed at reducing
or managing it (Bordia, Hobman, Jones, Gallois, & Callan, 2004). Because uncertainty is oIten
6
unpleasant and may distract attention away Irom task perIormance (KanIer & Ackerman, 1989),
reducing uncertainty may lead to higher motivation and more eIIicient task strategies.
Second, Iormative Ieedback can eIIectively reduce the cognitive load oI learners,
especially novice or struggling students (e.g., Paas, Renkl, & Sweller, 2003; Sweller, Van
Merrienboer, & Paas, 1998). These students can become cognitively overwhelmed during
learning due to high perIormance demands, and thus they may beneIit Irom supportive Ieedback
designed to decrease the cognitive load. In Iact, Sweller et al. provided support Ior this claim by
showing how the presentation oI worked examples reduces the cognitive load Ior low-ability
students Iaced with a complex problem-solving task. Moreno (2004) provided additional support
using explanatory Ieedback to support novice learners.
Finally, Ieedback can provide inIormation that may be useIul Ior correcting inappropriate
task strategies, procedural errors, or misconceptions (e.g., Ilgen et al., 1979; Mason & Bruning,
2001; Mory, 2004; Narciss & Huth, 2004). The corrective Iunction eIIects appear to be
especially powerIul Ior Ieedback that is more speciIic (Baron, 1988; Goldstein, Emanuel, &
Howell, 1968), which is described next.
3.3 Feedback Specificity
Feedback speciIicity is deIined as the level of information presented in Ieedback
messages (Goodman, Wood, & Hendrickx, 2004). In other words, speciIic (or elaborated)
Ieedback provides inIormation about particular responses or behaviors beyond just their accuracy
and tends to be more directive than Iacilitative.
A number oI researchers have reported that Ieedback is signiIicantly more eIIective when
it provides details oI how to improve the answer, rather than when it just indicates whether the
student`s work is correct or not (e.g., Bangert-Drowns et al., 1991; Pridemore & Klein, 1995).
Feedback lacking in speciIicity may cause students to view it as useless and/or Irustrating
(Williams, 1997). It can also lead to uncertainty about how to respond to the Ieedback (Fedor,
1991) and may require greater inIormation processing activity on the part oI the learner to
understand the intended message (Bangert-Drowns et al.). Uncertainty and cognitive load can
lead to lower levels oI learning (Kluger & DeNisi, 1996; Sweller et al., 1998), or even reduced
motivation to respond to the Ieedback (AshIord, 1986; Corno & Snow, 1986).
In an experiment that tested Ieedback speciIicity and its relationship to learning, Phye and
Sanders (1994) tested two types oI Ieedback (i.e., general advice versus speciIic Ieedback, the
7
latter providing the learner with the correct answer). Students were assigned to one oI the two
learning conditions, and they received either general advice or speciIic Ieedback as part oI a
verbal analogy problem-solving task. In line with the research cited above, Phye and Sanders
Iound that the more speciIic Ieedback was clearly superior to general advice on a retention task.
However, they Iound no signiIicant diIIerences between Ieedback types on a transIer task. They
cautioned against assuming that procedures that enhance perIormance during acquisition (e.g.,
providing speciIic Ieedback) will necessarily enhance transIer to new tasks.
In summary, providing Ieedback that is speciIic and clear Ior conceptual and procedural
learning tasks is a reasonable, general guideline. However, this practice may depend on other
variables, such as learner characteristics (e.g., ability level, motivation) and diIIerent learning
outcomes (e.g., retention vs. transIer tasks). In addition, the speciIicity dimension oI Iormative
Ieedback itselI is not very speciIic, as described in the literature. More Iocused Ieedback Ieatures
are reviewed in the next section.
3.4 Features of Formative Feedback
In an excellent historical review on Ieedback, Kulhavy and Stock (1989) reported that
eIIective Ieedback provides the learner with two types oI inIormation: veriIication and
elaboration. VeriIication is deIined as the simple judgment oI whether an answer is correct, and
elaboration is the inIormational aspect oI the message that provides relevant cues to guide the
learner toward a correct answer. Researchers appear to be converging toward the view that
eIIective Ieedback should include elements oI both veriIication and elaboration (e.g., Bangert-
Drowns et al., 1991; Mason & Bruning, 2001). These Ieatures are now described in more detail.
Verification. ConIirming whether an answer is correct can be accomplished in several
diIIerent ways. The most common way involves simply stating whether the answer is correct or
incorrect. With computers, there are more optionssome oI which are explicit and some more
implicit. Among explicit veriIications, highlighting or otherwise marking a response to indicate
its correctness (e.g., with a check mark) can convey the inIormation. Implicit veriIication can
occur when, Ior instance, a student`s response yields expected or unexpected results (e.g., within
a simulation). This review Iocuses more on explicit than implicit Ieedback as the Iormer is more
readily subject to experimental controls.
Elaboration. Feedback elaboration has even more variations than veriIication. For
instance, elaboration can (a) address the topic, (b) address the response, (c) discuss the particular
8
error(s), (d) provide worked examples, or (e) give gentle guidance. The Iirst three types oI
elaborated Ieedback are more speciIic and directive, while the last two types are more general
and Iacilitative.
In general, elaborated Ieedback usually addresses the correct answer and may
additionally explain why the selected response is wrong and indicate what the correct answer
should be. There seems to be growing consensus that one type oI elaboration, response-speciIic
Ieedback, appears to enhance student achievement, particularly learning eIIiciency, more than
other types oI Ieedback, such as simple veriIication or answer until correct (e.g., Corbett &
Anderson, 2001; Gilman, 1969; Mory, 2004). However, as will be discussed in Section 3.9 (see
'Goal Orientation), Ieedback speciIicity has been shown to aIIect perIormance by way oI an
interaction with learners` goal orientations.
3.5 Feedback Complexity/Length
While more speciIic Ieedback may be generally better than less speciIic Ieedback (at least
under certain conditions), a related dimension to consider in generating Ieedback is one oI length
or complexity oI the inIormation. For example, iI Ieedback is too long or too complicated, many
learners will simply not pay attention to it, rendering it useless. Lengthy Ieedback can also
diIIuse or dilute the message. Feedback complexity thus reIers to how much and what
inIormation should be included in the Ieedback messages.
Many research articles have addressed the issue oI Ieedback complexity, but only a Iew
have attempted to array the major variables along a dimension oI complexity (albeit, see
Dempsey, Driscoll, & Swindell, 1993; Mason & Bruning, 2001; and Narciss & Huth, 2004). I
have aggregated inIormation Irom their respective lists into a single compilation (see Table 1),
arrayed generally Irom least to most complex inIormation presented. Terms appearing in the
Ieedback type column are used throughout the remainder oI this paper.
II Iormative Ieedback is to serve a corrective Iunction, even in its simplest Iorm it should
(a) veriIy whether the student`s answer is right or wrong and (b) provide inIormation to the
learner about the correct response (either directive or Iacilitative). Studies that have examined the
type and amount oI inIormation in Ieedback, however, have shown inconsistent results (see
Kulhavy, 1977, and Mory, 2004, Ior summaries oI the range oI results). Some speciIic Iindings
on the Ieedback complexity issue are described below.
9
Table 1
Feedback Types Arrayed Loosely by Complexity
Feedback type Description
No Ieedback ReIers to conditions where the learner is presented a question and is
required to respond, but there is no indication as to the
correctness oI the learner`s response.
VeriIication Also called knowledge of results (KR), or knowledge of outcome, it
inIorms the learner about the correctness oI her response(s), such
as right/wrong or overall percentage correct.
Correct response Also known as knowledge of correct response (KCR), it inIorms
the learner oI the correct answer to a speciIic problem with no
additional inIormation.
Try-again Also known as repeat-until-correct Ieedback, it inIorms the learner
about an incorrect response and allows the learner one or more
attempts to answer the question.
Error-Ilagging Also known as location of mistakes (LM), error-Ilagging highlights
errors in a solution, without giving correct answer.
Elaborated A general term, it reIers to providing an explanation about why a
speciIic response was correct, and it might allow the learner to
review part oI the instruction. It also might present the correct
answer (see below Ior six types oI elaborated Ieedback).
Attribute isolation Elaborated Ieedback that presents inIormation addressing central
attributes oI the target concept or skill being studied.
Topic-contingent Elaborated Ieedback that provides the learner with inIormation
relating to the target topic currently being studied. This might
entail simply re-teaching material.
Response-contingent Elaborated Ieedback that Iocuses on the learner`s speciIic response.
It may describe why the answer is wrong and why the correct
answer is correct. This does not use Iormal error analysis.
Hints/cues/prompts Elaborated Ieedback that guides the learner in the right direction
(e.g., strategic hint on what to do next or a worked example or
demonstration). It avoids explicitly presenting the correct answer.
Bugs/misconceptions Elaborated Ieedback that requires error analysis and diagnosis. It
provides inIormation about the learner`s speciIic errors or
misconceptions (e.g., what is wrong and why).
InIormative tutoring The most elaborated Ieedback (Irom Narciss & Huth, 2004), this
presents veriIication Ieedback, error-Ilagging, and strategic hints
on how to proceed. The correct answer is not usually provided.
10
No effect of feedback complexity. Schimmel (1983) perIormed a meta-analysis on
Ieedback as used in computer-based instruction and programmed instruction. He analyzed the
results Irom 15 experimental studies and Iound that the amount oI inIormation (i.e., Ieedback
complexity) was not signiIicantly related to Ieedback eIIects. He also Iound that Ieedback eIIects
were signiIicantly larger in computer-based than in programmed instruction.
Sleeman et al. (1989) examined conIlicting Iindings in the literature concerning the
diagnosis and remediation oI students` errors. They noted that Iew studies have systematically
compared the eIIects oI diIIerent styles oI error-based Ieedback, and oI those that have, the
results are inconclusive. For instance, Swan (1983) Iound that a conIlict approach (pointing out
errors made by students and demonstrating their consequences, classiIied in Table 1 as
bugs/misconceptions) was more eIIective than re-teaching (classiIied in Table 1 as topic-
contingent), but Bunderson and Olsen (1983) Iound no diIIerence between these two Ieedback
approaches.
To untangle these conIlicting Iindings, Sleeman et al. (1989) conducted three studies that
explicitly compared error-speciIic or model-based remediation (MBR, bugs/misconceptions)
with simply re-teaching the algebra content (topic-contingent). MBR bases Ieedback on a model
oI student errors, while re-teaching simply shows students a correct procedure and answer
without addressing speciIic errors. Their results showed that MBR (a more complex approach)
and re-teaching (a simpler approach) are both more eIIective than no tutoring; however, MBR
was not more eIIective than re-teaching. The results are discussed in terms oI stability oI errors
and their relevance to educational practice and to intelligent tutoring systems (ITS). Although the
studies used human tutors, the results suggest that, Ior procedural skills in algebra, Ieedback
based on just re-teaching content was as eIIective as Ieedback based on more expensive error
analyses.
Negative effects of feedback complexity. Kulhavy, White, Topp, Chan, and Adams (1985)
similarly examined the Ieedback complexity issue. They tested a group oI college
undergraduates who read a 2,400-word passage, responded to 16 multiple-choice questions about
it, and received aIter responding one oI Iour types oI Ieedback that increased in complexity.
Feedback complexity was systematically varied. The lowest level was simply veriIication
Ieedback with the correct response, and the most complex included a combination oI
veriIication, the correct answer, and an explanation about why the incorrect answer was wrong
11
with a pointer to the relevant part oI the text passage where the answer could be Iound. The main
Iinding was that complexity oI Ieedback was inversely related to both the ability to correct errors
and to learn eIIiciently (i.e., the ratio oI Ieedback study time to posttest score). SpeciIically, the
authors showed that more complex versions oI Ieedback had a small eIIect on students` ability to
correct their own errors, and the least complex Ieedback demonstrated greater learner beneIits in
terms oI eIIiciency and outcome than complex Ieedback.
In summary, the inconclusive Iindings on Ieedback complexity described above suggest
that there may be other mediating Iactors involved in the relationship between Iormative
Ieedback and learning. For instance, instead oI Ieedback complexity, a more salient Iacet oI
Ieedback may be the nature and quality oI the content, such as providing inIormation about
learning goals and the attainment oI those goals.
3.6 Goal-Directed Feedback and Motivation
Goal-directed Ieedback provides learners with inIormation about their progress toward a
desired goal (or set oI goals), rather than providing Ieedback on discrete responses (i.e.,
responses to individual tasks). Research has shown that Ior a learner to remain motivated and
engaged depends upon a close match between a learner`s goals and expectations that these goals
can be met (Fisher & Ford, 1998; Ford, Smith, Weissbein, Gully, & Salas, 1998). II goals are set
so high that they are unattainable, the learner will likely experience Iailure and become
discouraged. When goals are set so low that their attainment is certain, success loses its power to
promote Iurther eIIort (Birney, Burdick, & Teevan, 1969).
According to Malone (1981), goals must have certain Ieatures to make them challenging
Ior the learner. For example, goals must be personally meaningIul and easily generated, and the
learner must receive perIormance Ieedback about whether or not the goals are being attained.
Hoska (1993) classiIied goals as being oI two types: acquisition (i.e., to help the learner acquire
something desirable) and avoidance (i.e., to help the learner avoid something undesirable).
Moreover, acquisition and avoidance goals can be either external or internal.
Motivation has been shown to be an important mediating Iactor in learners` perIormance
(Covington & Omelich, 1984), and Ieedback can be a powerIul motivator when delivered in
response to goal-driven eIIorts. Some researchers suggest that the learner`s goal orientation
12
should be considered when designing instruction, particularly when Ieedback can encourage or
discourage a learner`s eIIort (Dempsey et al., 1993). Goal orientation describes the manner in
which people are motivated to work toward diIIerent kinds oI goals. The idea is that individuals
hold either a learning or perIormance orientation toward tasks (e.g., Dweck, 1986). A learning
orientation is characterized by a desire to increase one`s competence by developing new skills
and mastering new situations with the belieI that intelligence is malleable. In contrast,
perIormance orientation reIlects a desire to demonstrate one`s competence to others and to be
positively evaluated by others, with the belieI that intelligence is innate (Farr, HoImann, &
Ringenbach, 1993).
Research has shown that the two types oI goal orientation diIIerentially inIluence how
individuals respond to task diIIiculty and Iailure (Dweck & Leggett, 1988). That is, individuals
with a learning orientation are characterized by persistence in the Iace oI Iailure, the use oI more
complex learning strategies, and the pursuit oI challenging material and tasks. PerIormance
orientation is characterized by a tendency to withdraw Irom tasks (especially in the Iace oI
Iailure), less interest in diIIicult tasks, and the tendency to seek less challenging material and
tasks on which success is likely. Consistent with these labels, research has generally shown that
learning orientation is associated with more positive outcomes and perIormance orientation is
related to either equivocal or negative outcomes (e.g., Button, Mathieu, & Zajac, 1996; Fisher &
Ford, 1998; VandeWalle, Brown, Cron, & Slocum, 1999).
One way to inIluence learners` goal orientations (e.g., to shiIt Irom a Iocus on perIorming
to an emphasis on learning) is via Iormative Ieedback. Hoska (1993) showed how goal-
orientation Ieedback can modiIy a learner`s view oI intelligence, by helping a learner see that (a)
ability and skill can be developed through practice, (b) eIIort is critical to increasing this skill,
and (c) mistakes are part oI the skill-acquisition process. Feedback can also serve as a cognitive
support mechanism, described next.
3.7 Formative Feedback as Scaffolding
Like training wheels, scaIIolding enables learners to do more advanced activities and to
engage in more advanced thinking and problem solving than they could without such help.
Eventually, high level Iunctions are gradually turned over to the students as the teacher (or
computer system) removes the scaIIolding and Iades away Irom the process (see Collins, Brown,
& Newman, 1989; Graesser, McNamara, & VanLehn, 2005). For instance, Graesser, McNamara,
13
and VanLehn described a theoretically based approach to Iacilitating explanation-centered
learning via scaIIolding, including (a) pedagogical agents that scaIIold strategies, metacognition,
and explanation construction; (b) computer coaches that Iacilitate answer generation to questions
that require explanations by using mixed-initiative dialogue; and (c) modeling and coaching
students in constructing selI-explanations. Their systems (i.e., Point&Query, AutoTutor, and
iSTART) that were built with these components have shown promising results in tests oI
learning gains and improved learning strategies.
In their book, How People Learn, BransIord, Brown, and Cocking (2000) described how
psychological theories and insights can be translated into actions and practices. In relation to
Ieedback, they suggested a goal-directed approach to learning using scaIIolding (or scaIIolded
Ieedback) that (a) motivates the learner`s interest related to the task, (b) simpliIies the task to
make it more manageable and achievable, (c) provides some direction to help the learner Iocus
on achieving the goal, (d) clearly indicates the diIIerences between the learner`s work and the
standard or desired solution, (e) reduces Irustration and risk, and (I) models and clearly deIines
the expectations (goals) oI the activity to be perIormed.
Conventional wisdom suggests that Iacilitative Ieedback (providing guidance and cues, as
illustrated in the research cited above) would enhance learning more than directive Ieedback
(providing corrective inIormation), yet this is not necessarily the case. In Iact, some research has
shown that directive Ieedback may actually be more helpIul than Iacilitativeparticularly Ior
learners who are just learning a topic or content area (e.g., Knoblauch & Brannon, 1981;
Moreno, 2004). Because scaIIolding relates to the explicit support oI learners during the learning
process, scaIIolded Ieedback in an educational setting may include models, cues, prompts, hints,
partial solutions, as well as direct instruction (Hartman, 2002). ScaIIolding is gradually removed
as students gain their cognitive Iooting, thus directive Ieedback may be most helpIul during the
early stages oI learning. Facilitative Ieedback may be more helpIul later on, and the question is
when. According to Vygotsky (1987), external scaIIolds can be removed when the learner
develops more sophisticated cognitive systems, where the system oI knowledge itselI becomes
part oI the scaIIold Ior new learning. The issue oI Ieedback timing is discussed in more detail in
the Iollowing section.
14
3.8. Timing
It was my teacher's genius, her quick sympathy, her loving tact which made the first years
of my education so beautiful. It was because she seized the right moment to impart
knowledge that made it so pleasant and acceptable to me. Helen Keller
Similar to the aIorementioned Ieedback variables (e.g., complexity and speciIicity), there
are also conIlicting results in the literature relating to the timing oI Ieedback and the eIIects on
learning outcome and eIIiciency. Researchers have been examining the eIIects oI immediate
versus delayed Ieedback on learning Ior decades (e.g., Clariana, 1999; Jurma & Froelich, 1984;
Pound & Bailey, 1975; Prather & Berry, 1973; Reddy, 1969). The timing-oI-Ieedback literature
concerns whether Ieedback should be delivered immediately or delayed. Immediately may be
deIined as right aIter a student has responded to an item or problem or, in the case oI summative
Ieedback, right aIter the student has completed a quiz or test. Delayed is usually deIined relative
to immediately, and such Ieedback may occur minutes, hours, weeks, or longer aIter the student
completes some task or test.
Regardless oI the particular unit oI time, the eIIects oI the Ieedback timing variable are
mixed. Again, while there appears to be no consistent main eIIect oI timing, there are
interactions involving the timing oI Ieedback and learning. Some researchers have argued Ior
immediate Ieedback as a way oI preventing errors being encoded into memory, while others have
argued that delayed Ieedback reduces proactive interIerence, thus allowing the initial error to be
Iorgotten and the correct inIormation to be encoded with no interIerence (Ior more on this
debate, see Kulhavy & Anderson, 1972).
Support for delayed feedback. Researchers who support using delayed Ieedback generally
adhere to what is called the interIerence-perseveration hypothesis proposed by Kulhavy and
Anderson (1972). This asserts that initial errors do not compete with to-be-learned correct
responses iI corrective inIormation is delayed. This is because errors are likely to be Iorgotten
and thus cannot interIere with retention.
The superiority oI delayed Ieedback, reIerred to as the delay-retention eIIect (DRE), was
supported in a series oI experiments by Anderson and his colleagues (e.g., Kulhavy & Anderson,
1972; Surber & Anderson, 1975) that compared the accuracy oI responses on a retention test
with the accuracy oI responses on an initial test. Although there are many studies in the literature
that do not support the delay-retention eIIect (e.g., Kippel, 1974; Newman, Williams, & Hiller,
15
1974; Phye & Baller, 1970), delayed Ieedback has oIten been shown to be as eIIective as
immediate Ieedback.
Schroth (1992) presented the results Irom an experiment that investigated the eIIects oI
delayed Ieedback and type oI verbal Ieedback on transIer, using a concept Iormation task. The
Iour conditions oI delayed Ieedback were: 0 seconds, 10 seconds, 20 seconds, and 30 seconds.
The verbal Ieedback conditions were (a) correct-incorrect (veriIication Ieedback), (b) correct-
nothing (i.e., where nothing means that no Ieedback was presented iI the student solved an item
incorrectly), and (c) nothing-incorrect (i.e., no Ieedback was presented iI the student answered
correctly). All participants were tested seven days Iollowing an initial learning trial. The Iinding
relevant to this paper is that although delayed Ieedback slowed down the rate oI initial learning,
it Iacilitated transIer aIter the delay.
Support for immediate feedback. Supporters oI immediate Ieedback theorize that the
earlier corrective inIormation is provided, the more likely it is that eIIicient retention will result
(Phye & Andre, 1989). The superiority oI immediate over delayed Ieedback has been
demonstrated Ior the acquisition oI verbal materials, procedural skills, as well as some motor
skills (Anderson, Magill, & Sekiya, 2001; Brosvic & Cohen, 1988; Corbett & Anderson, 1989;
2001; DihoII, Brosvic, Epstein, & Cook, 2003).
Corbett and Anderson (2001) have been using immediate Ieedback successIully in their
programming and mathematics tutors Ior almost two decades (see Anderson, Corbett, Koedinger,
& Pelletier, 1995). For instance, they used their ACT Programming Tutor to examine diIIerential
timing eIIects on students` learning. The study involved Iour Ieedback conditions, the Iirst three
oI which oIIered the student diIIerent levels oI control over error Ieedback and correction: (a)
immediate Ieedback and immediate error correction (i.e., the tutor intervened as soon as students
made errors and Iorced them to correct the error beIore moving on), (b) immediate error-Ilagging
and student control oI error correction, (c) Ieedback on demand and student control oI error
correction, and (d) no-tutor condition and no step-by-step, problem-solving support (the control
condition). The immediate Ieedback group with greatest tutor control oI problem solving yielded
the most eIIicient learning (i.e., condition a). These students completed the tutor problems
Iastest, and their perIormance on criterion tests was equivalent to that oI the other groups
(excluding the control group). Furthermore, questionnaires showed no signiIicant diIIerences in
16
terms oI preIerence among the tutor conditions. This study demonstrated that immediate error
Ieedback helped with immediate learning.
Azevedo and Bernard (1995) conducted a meta-analysis on the literature concerning the
eIIects oI Ieedback on learning Irom computer-based instruction (CBI). They noted that despite
the widespread acceptance oI Ieedback in computerized instruction, empirical support Ior
particular types oI Ieedback inIormation has been inconsistent and contradictory. EIIect size
calculations were perIormed on 22 CBI studies comparing Ieedback versus no-Ieedback relating
to immediate outcomes. This resulted in a mean weighted eIIect size oI 0.80. The results Irom 9
studies employing delayed outcome conditions resulted in a mean weighted eIIect size oI just
0.35. This provides support Ior the strength oI Ieedback in relation to immediate outcome
administrations, at least in CBI.
Conjoining feedback timing findings. A preliminary conclusion derived Irom both the
Schroth (1992) and Corbett and Anderson (2001) Iindings is that delayed Ieedback may be
superior Ior promoting transIer oI learning, especially in relation to concept Iormation tasks,
while immediate Ieedback may be more eIIicient, particularly in the short run and Ior procedural
skills (i.e., programming and mathematics). This proposition has some support. For instance,
Schmidt, Young, Swinnen, and Shapiro (1989) conducted an experiment that provided
veriIication Ieedback Iollowing a set oI trials relating to a relatively simple ballistic-timing task.
Feedback timing consisted oI one oI Iour lengths: 1 (i.e., veriIication aIter every trial), 5
(veriIication aIter 5 trials), 10, and 15 trials. During the acquisition phase when Ieedback was
present, all groups showed general improvements in perIormance across practice, although those
in the longer length conditions showed worse perIormance relative to the shorter length
conditions. In a delayed test, Schmidt et al. Iound an inverse relation between the timing variable
(1, 5, 10, or 15 trials between Ieedback) and error rates. That is, longer delays between Ieedback
episodes resulted in relatively poorer perIormance during acquisition, but better retention
compared with shorter delay conditions.
Mathan and Koedinger (2002) reviewed various studies on the timing oI Ieedback and
concluded that the eIIectiveness oI Ieedback depends not on the main eIIect oI timing, but on the
nature oI the task and the capability oI the learner. They called Ior Iurther exploration on
possible interactions involving timing eIIects and optimal ways to match Ieedback (type and
timing) to learning tasks and students` individual needs and/or characteristics (e.g., Schimmel,
17
1988; Smith & Ragan, 1999). One such interaction reported in the literature concerns Ieedback
timing and task diIIiculty. That is, iI the task is diIIicult, then immediate Ieedback is beneIicial,
but iI the task is easy, then delayed Ieedback may be preIerable (Clariana, 1999). This is similar
to the ideas presented earlier in Section 3.7.
Summary of feedback timing results. Research investigating the relationship oI Ieedback
timing and learning/perIormance reveals inconsistent Iindings.
1
One way to resolve the
inconsistency is by considering that immediate Ieedback may activate both positive and negative
learning eIIects. For instance, the positive eIIects oI immediate Ieedback can be seen as
Iacilitating the decision or motivation to practice and providing the explicit association oI
outcomes to causes. On the downside, immediate Ieedback may lead to reliance on inIormation
that is not available during transIer, and it also may promote less careIul or mindIul behavior. II
this supposition is true, then the positive and negative eIIects oI immediate Ieedback could
potentially cancel each other out. Alternatively, either the positive or negative eIIects may come
to the Iore, depending on the experimental context. A similar argument could be made Ior
delayed Ieedback eIIects on learning. For example, on the positive side, delayed Ieedback may
encourage learners` engagement in active cognitive and metacognitive processing, thus
engendering a sense oI autonomy (and perhaps improved selI-eIIicacy). But on the negative side,
delaying Ieedback Ior struggling and less motivated learners may prove to be Irustrating and
detrimental to their knowledge and skill acquisition.
3.9 Feedback and Other Variables
So Iar, Iormative Ieedback types and timing have been discussed in relation to their
eIIects on learning. This section examines other variables that may interact with Ieedback
Ieatures, such as learner-ability level, response certitude, goal orientation, and normative
Ieedback.
Learner level. As alluded to in Section 3.8, some research has suggested that low
achieving students may beneIit Irom immediate Ieedback, while high achieving students may
preIer or beneIit Irom delayed Ieedback (Gaynor, 1981; Roper, 1977). Furthermore, when testing
diIIerent types oI Ieedback, Clariana (1990) argued that low ability students beneIit Irom
correct-response Ieedback more than Irom try-again Ieedback. Hanna (1976) also examined
student perIormance in relation to diIIerent Ieedback conditions: verification, elaboration, and
no feedback. VeriIication Ieedback produced the highest scores Ior high-ability students, while
18
elaborated Ieedback produced the highest scores Ior low ability students. There were no
signiIicant diIIerences between veriIication and elaborated Ieedback Ior middle ability students,
but both oI these types oI Ieedback were superior to no Ieedback. These Iindings support the
research and suppositions presented earlier in Section 3.7.
Response certitude. Kulhavy and Stock (1989) examined Ieedback and response certitude
issues Irom an inIormation processing perspective. That is, they had students provide conIidence
judgments (response certitude ratings) Iollowing each response to various tasks. They
hypothesized that when students are certain that their answer is correct, they will spend little time
analyzing Ieedback, and when students are certain that their answer is incorrect, they will spend
more time reviewing Ieedback. The implications oI this are straightIorward; that is, provide more
elaborated Ieedback Ior students who are more certain that their answer is wrong, and deliver
more constrained Ieedback Ior those with high certitude oI correct answers. While Kulhavy and
Stock`s research supported their hypotheses, other studies did not replicate the Iindings. For
instance, Mory (1994) tried to replicate the response-certitude Iindings and Iound that while
there were diIIerences in the amount oI time needed to study the Ieedback, there was no
signiIicant learning eIIect Ior Ieedback tailored to response certitude and correctness.
Goal orientation. Davis, Carson, Ammeter, and Treadway (2005) reported the results oI a
study testing the relationship between goal orientation and Ieedback speciIicity on perIormance
using a management decision-making task. (Feedback speciIicity was discussed in Section 3.3.)
In short, they Iound that Ieedback speciIicity (low, moderate, and high levels) had a signiIicant
inIluence on perIormance Ior individuals who were low on learning orientation (i.e., high
Ieedback speciIicity was better Ior learners with low learning orientation). They also reported a
signiIicant inIluence oI Ieedback speciIicity on perIormance Ior persons high in perIormance
orientation (i.e., this group also beneIited Irom more speciIic Ieedback). The Iindings support the
general positive eIIects oI Ieedback on perIormance and Iurther suggest the use oI more speciIic
Ieedback Ior learners with either high perIormance or low learning goal orientations.
Normative feedback. According to research cited in Kluger and DeNisi (1996), when
Ieedback is provided to students in a norm-reIerenced manner that compares the individual's
perIormance to that oI others, people who perIorm poorly tend to attribute their Iailures to lack
oI ability, expect to perIorm poorly in the Iuture, and demonstrate decreased motivation on
subsequent tasks (i.e., similar to learners with a perIormance orientation, as described earlier).
19
McColskey and Leary (1985) examined the hypothesis that the harmIul eIIects oI Iailure might
be lessened when Iailure is expressed in selI-reIerenced terms (i.e., relative to the learner`s
known level oI ability as assessed by other measures). In their study, learners received Ieedback
indicating that they did well or poorly on an anagram test, and this Ieedback was described as
either norm-reIerenced (comparing the individual`s perIormance to that oI others) or as selI-
reIerenced (comparing perIormance to other measures oI the individual`s ability). They Iound
that, compared to norm-reIerenced Ieedback, selI-reIerenced Ieedback resulted in higher
expectancies regarding Iuture perIormance and increased attributions to eIIort (e.g., 'I succeeded
because I worked really hard). Attributions to ability (e.g., 'I succeeded because I`m smart)
were not aIIected. The main implication is that low achieving students should not receive
normative Ieedback but instead, selI-reIerenced IeedbackIocusing their attention on their own
particular progress.
This review has presented research Iindings covering the gamut oI Iormative Ieedback
variables. As with earlier reviews, this one has unearthed mixed Iindings regarding learning
eIIectswhether examining Ieedback speciIicity, timing oI Ieedback, and so on. The next
section presents Iour inIluential Ieedback research papers that have attempted to integrate
disparate Iindings into preliminary theories (or models) via large literature reviews and/or meta-
analyses. The papers summarized in this section include Kluger and DeNisi (1996), Bangert-
Drowns et al. (1991), Narciss and Huth (2004), and Mason and Bruning (2001).
4. Toward a Framework of Formative Feedback
To understand the world, one must not be worrying about ones self. Einstein
4.1 Kluger and DeNisi (1996)
Ten years ago, Kluger and DeNisi (1996) examined and reported on the eIIects oI
Ieedback interventions (FIs) on perIormance Irom multiple perspectives and spanning decades oI
researchback to Thorndike`s classic research in the early 1900s. Kluger and DeNisi conducted
an extensive review oI the literature, perIormed a meta-analysis on reported experimental
Iindings, and constructed a preliminary theory based on a number oI variables, or moderators.
Their preliminary FI theory (FIT) oIIers a broad approach to investigating FI eIIects, including
such Ieedback moderators as praise, written or verbal Ieedback, task novelty and complexity,
20
time constraints, and types oI tasks such as physical, memory, knowledge, and vigilance tasks.
The basic premise underlying FIT is that FIs change the locus oI a learner`s attention among
three levels oI control: (a) task-learning, (b) task-motivation, and (c) meta-task processes (see
Figure 1).

Figure 1. Abstract hierarchy of processing.
The general pattern oI results Irom their large meta-analysis oI FI studies was consistent
with the argument that all else being equal, FI cues aIIect perIormance by changing the locus oI
attention. The lower in the hierarchy the FI-induced locus oI attention is, the stronger the beneIit
oI an FI Ior perIormance. In other words, Iormative Ieedback that Iocuses the learner on aspects
oI the task (i.e., the lower part oI Figure 1) promotes learning and achievement compared to FIs
that draw attention to the selI (i.e., the upper box in Figure 1), which can impede learning. This
phenomenon was described earlier.
FIT consists oI Iive basic arguments: (a) behavior is regulated by comparisons oI
Ieedback to goals or standards, (b) goals or standards are organized hierarchically, (c) attention is
limited and thereIore only Ieedback-standard gaps (i.e., discrepancies between actual and desired
perIormance) that receive attention actively participate in behavior regulation, (d) attention is
normally directed to a moderate level oI the hierarchy, and (e) FIs change the locus oI attention
and thereIore aIIect behavior. These arguments are interdependent, and each consecutive
argument is built on the preceding argument.
21
SpeciIic results Irom Kluger and DeNisi`s (1996) meta-analysis showed that Iour
moderators (Ieedback variables) demonstrated signiIicant relationships with d (eIIect size) at the
p 0.01 level: (a) discouraging Ieedback interventions (FIs) reduce FI eIIects, (b) velocity FIs
(i.e., selI-reIerenced Ieedback that addresses a change Irom the learner`s prior perIormance), (c)
correct response FIs increase eIIects, and (d) FI eIIects on perIormance oI physical tasks are
lower than FI eIIects on cognitive tasks.
Six more moderators became signiIicant aIter excluding biased studies Irom the meta-
analysis. OI those six, three oI them were shown to reduce FI eIIects: (a) praise, (b) FIs that
threaten selI-esteem, and (c) orally delivered FIs (Irom the instructor). Feedback interventions
that provide Irequent messages enhance FI eIIects, and FI eIIects are stronger Ior memory tasks
and weaker Ior more procedural tasks. Finally, other variables showing signiIicance at p 0.05
level include the Iollowing: computerized FIs, which yield stronger eIIects than
noncomputerized FIs; FIs in the context oI complex tasks, which yield weaker eIIects than Ior
simpler tasks; and FIs with a goal-setting intervention, which are more eIIective than in the
absence oI goal setting. Figure 2 summarizes the main Iindings. This Iigure represents my
interpretation (and categorization) oI data presented in Kluger and DeNisi (1996).
One important Iinding Irom these results concerns the attenuating eIIect oI praise on
learning and perIormance, although this has been described elsewhere in the literature in terms oI
a model oI selI-attention (Baumeister, Hutton, & Cairns, 1990), attributions oI eIIort (Butler,
1987), and control theory (Waldersee & Luthans, 1994). Also, Balcazar, Hopkins, & Suarez
(1985) reported that praise was not as widely eIIective a reinIorcer as previously believed.
Perhaps the most surprising Iinding that emerged Irom the Kluger and DeNisi (1996)
meta-analysis is that in over one third oI the 607 cases (eIIect sizes), FIs reduced perIormance.
Furthermore, most oI the observed variability cannot be explained by sampling or other errors.
In conclusion, and as the authors observe in a later paper on the topic (Kluger & DeNisi,
1998), FIs may be viewed as double-edged swords. Care should be taken to know which
interventions increase perIormance and under which particular conditions.
22

Figure 2. FI moderators and their relationships to learning/performance.
4.2 Bangert-Drowns, Kulik, Kulik, and Morgan (1991)
Bangert-Drowns et al. (1991) examined 40 research studies on Ieedback using meta-
analysis techniques. They examined such variables as type oI Ieedback, timing oI Ieedback, and
error rates in terms oI their respective eIIect sizes. This widely cited article describes both
behavioral and cognitive operations that occur in learning. The basic idea is that in order to direct
behavior, a learner needs to be able to monitor physical changes brought about by the behavior.
That is, learners change cognitive operations and thus activity by adapting to new inIormation
and matching the results oI activities to their expectations about perIormance. Bangert-Drowns et
al. emphasized that, 'any theory that depicts learning as a process oI mutual inIluence between
learners and their environments must involve Ieedback implicitly or explicitly because, without
Ieedback, mutual inIluence is by deIinition impossible. Hence, the Ieedback construct appears
oIten as an essential element oI theories oI learning and instruction (p. 214). To make this point
more concrete, imagine trying to learn something new in the absence oI any Ieedback (explicit or
implicit).
23
Most oI the variables they analyzed comprised text-based Ieedback, which they organized
into a Iive-stage model. This model describes the state oI learners as they move through a
Ieedback cycle and emphasizes the construct oI mindIulness (Salomon & Globerson, 1987).
MindIulness is 'a reIlective process in which the learner explores situational cues and underlying
meanings relevant to the task involved (Dempsey et al., 1993, p. 38).
The Iive stages are depicted in Figure 3 and are similar to other learning cycles (e.g.,
Kolb, 1984, and Gibbs, 1988), particularly in relation to the importance oI reIlection.

Figure 3. Five-stage model of the learner during a feedback cycle.
As described by Bangert-Drowns et al. (1991, p. 217), the Iive states oI the learner
receiving Ieedback include:
1. The initial or current state oI the learner. This is characterized by the degree oI
interest, goal orientation, degree oI selI-eIIicacy, and prior relevant knowledge.
2. Search and retrieval strategies. These cognitive mechanisms are activated by a
question. InIormation stored in the context oI elaborations would be easier to locate in
memory due to more pathways providing access to the inIormation.
3. The learner makes a response to the question. In addition, the learner Ieels some
degree oI certainty about the response and thus has some expectation about what the
Ieedback will indicate.
4. The learner evaluates the response in light oI inIormation Irom the Ieedback. The
nature oI the evaluation depends on the learner`s expectations about Ieedback. For
24
instance, iI the learner was sure oI the response and the Ieedback conIirmed its
correctness, the retrieval pathway may be strengthened or unaltered. II the learner was
sure oI the response and Ieedback indicated its incorrectness, the learner may seek to
understand the incongruity. Uncertainty about a response with Ieedback conIirmation
or disconIirmation is less likely to stimulate deep reIlection unless the learner was
interested in acquiring the instructional content.
5. Adjustments are made to relevant knowledge, selI-eIIicacy, interests, and goals as a
result oI the response evaluation. These adjusted states, with subsequent experiences,
determine the next current state.
Overall, the meta-analysis by Bangert-Drowns et al. (1991) Iound generally weak eIIects
oI Ieedback on achievement. More speciIically (but not surprisingly), the authors Iound that
veriIication Ieedback (correct/incorrect) resulted in lower eIIect sizes compared to correct
response Ieedback (i.e., providing the correct answer). Also, using a pretest within a study
signiIicantly lowered eIIect sizes, as did uncontrolled presearch availability oI answers (i.e.,
ability to locate an answer prior to responding to a question). These last two Iindings may be
because pretests and presearch availability may be seen as advance organizers, which may
support short-term retention, but undermine overall Ieedback eIIects in studies that employ them.
The main conclusion Irom the meta-analysis by Bangert-Drowns et al. (1991) and
subsequent Iive-cycle model is that Ieedback can promote learning iI it is received mindIully.
Conversely, Ieedback can inhibit learning iI it encourages mindlessness, as when the answers are
made available beIore learners begin their memory search, or iI the Ieedback message does not
match students` cognitive needs (e.g., too easy, too complex, too vague).
4.3 Narciss and Huth (2004)
Narciss and Huth (2004) outlined a conceptual Iramework Ior the design oI Iormative
Ieedback. This Iramework is based on the body oI research relating to elaborated Ieedback types.
Cognitive task and error analyses served as the basis Ior the design oI the Iramework. The impact
oI the Ieedback on learning and motivation was ultimately examined in two computer-based
learning experiments. The results oI these studies showed that systematically designed Iormative
Ieedback has positive eIIects on achievement and motivation.
25
In general, they assert that designing and developing eIIective Iormative Ieedback needs
to take into consideration instructional context, as well as characteristics oI the learner, to
provide eIIective Ieedback Ior complex learning tasks. The conceptual Iramework Ior the design
oI Iormative Ieedback is depicted in Figure 4, based on inIormation presented in Narciss and
Huth (2004).

Figure 4. Factors interacting with feedback to influence learning.
Each oI the three Iactors is examined in more detail below.
Feedback. The Ieedback Iactor consists oI three main elements: (a) the content oI the
Ieedback (i.e., evaluative aspects, such as veriIication, as well as inIormative aspects,
such as hints, cues, analogies, explanations, and worked out examples); (b) the
function oI the Ieedback (i.e., cognitive, metacognitive, and motivational); and (c) the
presentation oI the Ieedback components (i.e., timing, schedule, and perhaps
adaptivity considerations).
Instruction. The instructional Iactor or context also consists oI three main elements:
(a) the particular instructional objectives (e.g., learning goals or standards relating to
some curriculum), (b) the learning tasks (e.g., knowledge items, cognitive operations,
and metacognitive skills), and (c) errors and obstacles (e.g., typical errors, incorrect
strategies, and sources oI errors).
Learner. InIormation concerning the learner that is relevant to Ieedback design
includes: (a) learning objectives and goals; (b) prior knowledge, skills, and abilities
26
(e.g., domain-dependent, such as content knowledge, and domain-independent, such
as metacognitive skills); and (c) academic motivation (e.g., one`s need Ior academic
achievement, academic selI-eIIicacy, and meta-motivational skills).
The authors contend that adapting the content, Iunction, and presentation Iormat oI the
Ieedback message should be driven by considerations oI the instructional goals and learner
characteristics to maximize the inIormative value oI the Ieedback. SpeciIic steps Ior generating
eIIective Iormative Ieedback include selecting and speciIying learning objectives (concrete
learning outcomes), identiIying learning tasks, matching to learning outcomes, and aIter
conducting cognitive task and error analyses, speciIying inIormation (i.e., Iormative Ieedback)
that addresses speciIic, systematic errors or obstacles.
4.4 Mason and Bruning (2001)
Mason and Bruning (2001) reviewed the literature on Ieedback that is delivered via
computer-based instructional systems and presented a theoretical Iramework intended to help
designers, developers, and instructors build their own computer-based instructional tools. Mason
and Bruning`s theoretical Iramework, depicted in Figure 5, is based on research that has
examined types oI Ieedback and levels oI elaboration in relation to student achievement level,
task complexity, timing oI Ieedback, and prior knowledge. The general recommendation they
have drawn Irom the Iramework is that immediate Ieedback Ior students with low achievement
levels in the context oI either simple (lower level) or complex (higher level) tasks is superior to
delayed Ieedback, while delayed Ieedback is suggested Ior students with high achievement
levels, especially Ior complex tasks.
The Iollowing research supports their Iramework. First, signiIicant learning gains oIten
show up in response to various types oI elaboration Ieedback (e.g., Clariana, 1990; Pridemore &
Klein, 1991, 1995; Morrison, Ross, Gopalakrishnan, & Casey, 1995; Roper, 1977; Waldrop,
Justen, & Adams, 1986). Second, research conducted in classroom settings seems to suggest that
response-contingent Ieedback enhances student achievement more than other types oI Ieedback
(e.g., Whyte, Karolick, Neilsen, Elder, & Hawley, 1995). Third, they report that the level oI
Ieedback complexity (see Section 3.5) has been shown to both inIluence and not inIluence
learning, and this lack oI eIIect may be due to interactions involving other variables, such as the
nature oI the topic (e.g., memorizing vs. troubleshooting) and the type oI skill measured (e.g.,
27
Hodes, 1985; Park & Gittelman, 1992). In cases where the level oI Ieedback complexity has been
shown to aIIect learning, more elaborative inIormation tends to produce increased understanding
(Gilman, 1969; Pridemore & Klein, 1991; Roper 1997; Waldrop et al.; Whyte et al.). For
instance, while veriIication Ieedback did not improve learning, correct response, response-
contingent, and a combination oI the other levels oI Ieedback have been shown to signiIicantly
improve student learning (e.g., Gilman, 1969). This may be due to the extra inIormation
available in elaboration Ieedback, which allows students to correct their own errors or
misconceptions. InIormation on the correctness oI an answer (i.e., veriIication Ieedback) does
not have much utility Ior learning.

Figure 5. Feedback variables for decision making in computer-based instruction.
Note. Adapted Irom 'Providing Feedback in Computer-Based Instruction: What the Research
Tells Us by B. J. Mason and R. Bruning, 2001, Center Ior Instructional Innovation, University oI
Nebraska-Lincoln. Copyright 2001 by B. J. Mason and R. Bruning. Reprinted with permission.
28
5. Summary and Discussion
I'm trying to free your mind, Neo. But I can only show you the door. You're the one who
must walk through it. The Matrix (1999)
Morgan (2006) likened Iormative Ieedback to a good murder in that eIIective and useIul
Ieedback depends on three things: (a) motive (the student needs it), (b) opportunity (the student
receives it in time to use it), and (c) means (the student is able and willing to use the Ieedback).
However, even with motive, opportunity, and means, there is still large variability oI Ieedback
eIIects on perIormance and learning, including negative Iindings that have historically been
ignored in the literature (see Kluger & DeNisi, 1996).
Despite this variability, several meta-analyses Iound that Ieedback generally improves
learning, ranging Irom about .40 oI a standard deviation (Guzzo, Jette, & Katzell, 1985) to .80
and higher (Azevedo & Bernard, 1995; Kluger & DeNisi, 1996), compared to control conditions.
But major gaps remain in the Ieedback literature, particularly in relation to interactions among
task characteristics, instructional contexts, and student characteristics that potentially mediate
Ieedback eIIects. So while there is no simple answer to what Ieedback works, some preliminary
guidelines can be Iormulated based on the Iindings reported in this review.
5.1 Recommendations and Guidelines for Formative Feedback
Tables 25 present suggestions or prescriptions based on the current review oI the
Iormative Ieedback literature. These are intended to provide a point oI departure Ior more
comprehensive and systematic prescriptions in the Iuture. Equivocal Iindings are not presented,
and the reIerences are not exhaustive, but representative. The tables diIIer in terms oI Iormative
Ieedback guidelines Ior: (a) things to do, in Table 2, (b) things to avoid, in Table 3, (c) timing
issues, in Table 4, and (d) learner characteristics, in Table 5.
29
Table 2
Formative Feedback Guidelines to Enhance Learning (Things to Do)
Prescription Description and reIerences
1 Focus Ieedback on the task,
not the learner.
Feedback to the learner should address speciIic Ieatures oI the
learner`s work in relation to the task, with suggestions on how to
improve (e.g., Butler, 1987; Corbett & Anderson, 2001; Kluger &
DeNisi, 1996; Narciss & Huth, 2004).
2 Provide elaborated
Ieedback to enhance
learning.
Feedback should describe the what, how, and/or why oI a given
problem. This type oI cognitive Ieedback is typically more eIIective
than veriIication oI results (e.g., Bangert-Drowns et al., 1991;
Gilman, 1969; Mason & Bruning, 2001; Narciss & Huth, 2004;
Shute, 2006).
3 Present elaborated Ieedback
in manageable units.
Provide elaborated Ieedback in small enough pieces so that it is not
overwhelming and/or discarded (BransIord et al., 2000; Sweller et
al., 1998). Presenting too much inIormation may not only result in a
superIicial learning, but also invoke cognitive overload (e.g., Mayer
& Moreno, 2002; Phye & Bender, 1989). A stepwise presentation oI
Ieedback oIIers the possibility to control Ior mistakes and gives
learners suIIicient inIormation to correct errors on their own.
4 Be speciIic and clear with
Ieedback messages.
II Ieedback is not speciIic or clear, it can impede learning and can
Irustrate learners (e.g., Moreno, 2004; Williams, 1997). II possible,
try to link Ieedback clearly and speciIically to goals and
perIormance (Hoska, 1993; Song & Keller, 2001).
5 Keep Ieedback as simple as
possible but no simpler
(based on learner needs and
instructional constraints).
Simple Ieedback is generally based on one cue (e.g., veriIication or
hint) and complex Ieedback on multiple cues (e.g., veriIication,
correct response, error analysis). Keep Ieedback as simple and
Iocused as possible. Generate only enough inIormation to help
students and not more. Kulhavy et al. (1985) Iound that Ieedback
that was too complex did not promote learning compared to simpler
Ieedback.
6 Reduce uncertainty between
perIormance and goals.
Formative Ieedback should clariIy goals and seek to reduce or remove
uncertainty in relation to how well learners are perIorming on a task
and what needs to be accomplished to attain the goal(s) (e.g.,
AshIord, Blatt, & VandeWalle, 2003; Bangert-Drowns et al., 1991).
7 Give unbiased, objective
Ieedback, written or via
computer.
Feedback Irom a trustworthy source will be considered more seriously
than other Ieedback, which may be disregarded. This may explain
why computer-based Ieedback is oIten better than human-delivered
in some experiments in that perceived biases are eliminated (see
Kluger & DeNisi, 1996).
8 Promote a learning goal
orientation via Ieedback.
Formative Ieedback can be used to alter goal orientationIrom a
Iocus on perIormance to a Iocus on learning (Hoska, 1993). This
can be Iacilitated by craIting Ieedback emphasizing that eIIort yields
increased learning and perIormance and that mistakes are an
important part oI the learning process (Dweck, 1986).
9 Provide Ieedback aIter
learners have attempted a
solution.
Do not let learners see answers beIore trying to solve a problem on
their own (i.e., presearch availability). Several studies that have
controlled presearch availability show a beneIit oI Ieedback, while
studies without such control show inconsistent results (Bangert-
Drowns et al., 1991).
30
Table 3
Formative Feedback Guidelines to Enhance Learning (Things to Avoid)
Prescription Description and reIerences
10 Do not give normative
comparisons.
Feedback should avoid comparisons with other studentsdirectly
or indirectly (e.g., grading on the curve). In general, do not draw
attention to selI during the course oI learning (Kluger & DeNisi,
1996; Wiliam, in press).
11 Be cautious about providing
overall grades.
Feedback should note areas oI strength and provide inIormation on
how to improve, as warranted and without overall grading.
Wiliam (in press) summarized the Iollowing Iindings: (a)
students receiving just grades showed no learning gains, (b)
those getting just comments showed large gains, and (c) those
with grades and comments showed no gains (likely due to
Iocusing on the grade and ignoring comments). EIIective
Ieedback relates to the content oI the comments (Butler, 1987;
McColskey & Leary, 1985).
12 Do not present Ieedback that
discourages the learner or
threatens the learner`s selI-
esteem.
This prescription is based not only on common sense, but also
research reported in Kluger and DeNisi (1996), which reports
Ieedback interventions that undermine learning as it draws Iocus
to the selI and away Irom the task at hand. In addition, do not
provide Ieedback that is either too controlling or critical oI the
learner (Baron, 1993; Fedor, Davis, Maslyn, & Mathieson,
2001).
13 Use praise sparingly, iI at all. Kluger & DeNisi (1996), Butler (1987), and others have noted that
use oI praise as Ieedback directs the learner`s attention to selI,
which distracts Irom the task and consequently Irom learning.
14 Try to avoid delivering
Ieedback orally.
This also was addressed in Kluger & DeNisi (1996). When
Ieedback is delivered in a more neutral manner (e.g., written or
computer-delivered), it is construed as less biased.
15 Do not interrupt the learner
with Ieedback iI the learner is
actively engaged.
Interrupting a student who is immersed in a tasktrying to solve a
problem or taskcan be disruptive to the student and impede
learning (Corno & Snow, 1986).
16 Avoid using progressive hints
that always terminate with the
correct answer.
While hints can be Iacilitative, they can also be abused. II they are
employed to scaIIold learners, make provisions to prevent their
abuse (e.g., Aleven & Koedinger, 2000; Shute, Woltz, & Regian,
1989). Consider using prompts and cues (i.e., more speciIic
kinds oI hints).
17 Do not limit the mode oI
Ieedback presentation to text.
Exploit the potential oI multimedia to avoid cognitive overload due
to modality eIIects (e.g., Mayer & Moreno, 2002). Do not deIault
to presenting Ieedback messages as text. Instead, consider
alternative modes oI presentation (e.g., acoustic, visual).
18 Minimize use oI extensive
error analyses and diagnosis.
The cost oI conducting extensive error analyses and cognitive
diagnosis may not provide suIIicient beneIit to learning (Sleeman
et al., 1989; VanLehn et al., 2005). Furthermore, error analyses
are rarely complete and not always accurate, thus only are
helpIul in a subset oI circumstances.
31
Table 4
Formative Feedback Guidelines in Relation to Timing Issues
Prescription Description and reIerences
19 Design timing oI Ieedback
to align with desired
outcome.
Feedback can be delivered (or obtained) either immediately
aIter some activity or delayed. Immediate Ieedback can help
Iix errors in real-time, producing greater immediate gains
and more eIIicient learning, (Corbett & Anderson, 2001;
Mason & Bruning, 2001), but delayed Ieedback has been
associated with better transIer oI learning (e.g., Schroth,
1992).
20 For diIIicult tasks, use
immediate Ieedback.
When a student is learning a diIIicult new task (where diIIicult
is relative to the learner`s capabilities), it is better to use
immediate Ieedback, at least initially (Clariana, 1990). This
provides a helpIul saIety net so the learner does not get
bogged down and/or Irustrated (Knoblauch & Brannon,
1981).
21 For relatively simple tasks,
use delayed Ieedback.
When a student is learning a relatively simple task (again,
relative to capabilities), it is better to delay Ieedback to
prevent Ieelings oI Ieedback intrusion and possibly
annoyance (Clariana, 1990; Corno & Snow, 1986).
22 For retention oI procedural
or conceptual knowledge,
use immediate Ieedback.
In general, there is wide support Ior use oI immediate Ieedback
to promote learning and perIormance on verbal, procedural,
and even tasks requiring motor skills (Anderson et al., 2001;
Azevedo & Bernard, 1995; Corbett & Anderson, 1989,
2001; DihoII et al., 2003; Phye & Andre, 1989).
23 To promote transIer oI
learning, consider using
delayed Ieedback.
According to some researchers (e.g., Kulhavy et al., 1985;
Schroth, 1992), delayed may be better than immediate
Ieedback Ior transIer task perIormance, although initial
learning time may be depressed. This needs more research.

32
Table 5
Formative Feedback Guidelines in Relation to Learner Characteristics
Prescription Description and reIerences
24 For high-achieving learners,
consider using delayed
Ieedback.
Similar to the Clariana (1990) Iindings cited in Table 4, high-
achieving students may construe a moderate or diIIicult task as
relatively easy and hence beneIit by delayed Ieedback (see
also Gaynor, 1981; Roper, 1977).
25 For low-achieving learners,
use immediate Ieedback.
The argument Ior low-achieving students is similar to the one
above, only these students need the support oI immediate
Ieedback in learning new tasks they may Iind diIIicult (see
Gaynor, 1981; Mason & Bruning, 2001; Roper, 1977).
26 For low-achieving learners,
use directive (or corrective)
Ieedback.
Novices, or struggling students, need support and explicit
guidance during the learning process (Knoblauch & Brannon,
1981; Moreno, 2004), thus hints may not be as helpIul as more
explicit, directive Ieedback.
27 For high-achieving learners,
use Iacilitative Ieedback.
Similar to the above, high-achieving students or more motivated
ones beneIit Irom Ieedback that challenges them, such as hints,
cues, and prompts (Vygotsky, 1987).
28 For low-achieving learners,
use scaIIolding.
Provide early support and structure Ior low-achieving students
(or those with low selI-eIIicacy) to improve learning and
perIormance (e.g., Collins et al., 1989; Graesser, McNamara,
& VanLehn, 2005).
29 For high-achieving learners,
veriIication Ieedback may be
suIIicient.
Hanna (1976) presented Iindings that suggest that high-achieving
students learn more eIIiciently iI permitted to proceed at their
own pace. VeriIication Ieedback provides the level oI
inIormation most helpIul in this endeavor.
30 For low-achieving learners,
use correct response and
some kind oI elaboration
Ieedback.
Using the same rationale as with supplying scaIIolding to low-
achieving students, the prescription here is to ensure low-
achieving students receive a concrete, directive Iorm oI
Ieedback support (e.g., Clariana, 1990; Hanna, 1976).
31 For learners with low
learning orientation (or high
perIormance orientation),
give speciIic Ieedback.
As described in the study by Davis et al., (2005), iI students are
oriented more toward perIormance (trying to please others)
and less toward learning (trying to achieve an academic goal),
provide Ieedback that is speciIic and goal directed. Also, keep
the learner`s eye on the learning goal (Hoska, 1993).
33
5.2 Future Research
One reason why studies examining Iormative Ieedback eIIects are so inconsistent may be
due to individual diIIerences among motivational prerequisites (e.g., intrinsic motivation, belieIs,
need Ior academic achievement, academic selI-eIIicacy, and metacognitive skills). In Iact,
Vygotsky (1987) noted that the study oI psychology had been damaged by the separation oI the
intellectual Irom the motivational and emotional (or aIIective) aspects oI thinking. CraIting and
delivering Iormative Ieedback may help bridge these aspects oI thinking and enhance learning.
This seems to be supported by a growing number oI researchers (e.g., Goleman, 1995; Mayer &
Salovey, 1993, 1997; Picard et al., 2004) who have argued that emotional upsets can interIere
with mental activities (e.g., anxious, angry, or depressed students do not learn). Thus one
intriguing area oI Iuture research is to systematically examine the relationship(s) between
aIIective components in Ieedback and outcome perIormance. And while there have been inroads
in the area, according to Picard et al., extending cognitive theory to explain and exploit the role
oI aIIect in learning is still in its inIancy.
Schwartz and White (2000), suggested, in general, a multidimensional view oI Ieedback
is needed, where situational and individual characteristics oI the instructional context and learner
are considered, along with the nature and quality oI a Ieedback message. Narciss and Huth
(2004) noted, and I strongly agree, that Iunction, content, and mode oI Ieedback presentation are
important Iacets and should be considered separately, as well as interactively, with learner
characteristics and instructional variables. Cognitive task and error analyses may be used to
match Iormative Ieedback components to (a) learning objectives, (b) skills needed Ior the
mastery oI the task, and to (c) typical errors or incorrect strategies. However, such expensive
analyses and methods may not, in Iact, be necessary to promote learning. (e.g., see Section 3.5
on the Sleeman et al., 1989, Iindings).
In line with the question concerning the added value oI employing error analyses and
more diagnostic types oI Iormative Ieedback, controlled evaluations are needed to systematically
test the eIIects oI Ieedback conditions (as listed in Table 1) on learning, combined with a cost-
beneIit analysis. Some obvious costs include development time Ior speciIying the Ieedback types
and time Ior the student to read the Ieedback. BeneIits relate to improvements in learning
outcome and eIIiciency, as well as possibly aIIective variables. InIormation about the learner
34
would be collected to examine possible interactions. The hypotheses are that (a) more complex
Iormative Ieedback types (e.g., involving extensive and expensive error analyses) do not yield
proportionately greater learning gains, and (b) Ieedback can be made more eIIective iI it can
adapt to the particular needs oI learnerscognitive and noncognitive characteristicsas well as
to diIIerent types oI knowledge and skills. The general question is: What level oI Ieedback
complexity yields the most bang Ior the buck?
Table 5 provides guidelines Ior linking a Iew learner characteristics to diIIerent Ieedback
types. Future research may examine (a) additional learner characteristics, as well as (b) links
between diIIerent types oI knowledge and Ieedback types. For instance, Ieedback to support Iact
learning (declarative knowledge) could reiterate deIinitions or provide the learner with a handy
mnemonic technique; Ieedback to support conceptual knowledge could provide examples,
counter-examples, and big pictures; and Ieedback to improve procedural knowledge could
involve demonstrations, solution paths (complete or partial), and so Iorth. Ultimately,
inIormation about the learner, combined with inIormation about desired outcomes, may inIorm
the development oI adaptive Iormative Ieedback. Various Ieedback types could be generated and
incorporated into a program (or generated on the Ily based on Iormative Ieedback models) and
then accessed and delivered according to the characteristics oI the learner in conjunction with the
nature oI the task and instructional goals.
In closing, the goal oI this review was to summarize research Iindings relating to
Iormative Ieedback to serve as the Ioundation Ior a variety oI Iuture educational products and
services. As evidenced throughout, there is no best type oI Iormative Ieedback Ior all learners
and learning outcomes. However, Iormative Ieedback has been shown, in numerous studies, to
improve students` learning and enhance teachers` teaching to the extent that the learners are
receptive and the Ieedback is on target (valid), objective, Iocused, and clear.
35
References
Albertson, L. M. (1986). Personalized Ieedback and cognitive achievement in computer assisted
instruction. Journal of Instructional Psychology, 13(2), 5557.
Aleven, V., & Koedinger, K. R. (2000). Limitations oI student control: Do students know when
they need help? In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th
international conference on intelligent tutoring systems, ITS 2000 (pp. 292303). Berlin:
Springer-Verlag.
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors:
Lessons learned. Journal of the Learning Sciences, 4, 167207.
Anderson, D. I., Magill, R. A., & Sekiya, H. (2001). Motor learning as a Iunction oI KR schedule and
characteristics oI task-intrinsic Ieedback. Journal of Motor Behavior, 33(1), 5967.
AshIord, S. J. (1986). Feedback-seeking in individual adaptation: A resource perspective.
Academy of Management Journal, 29, 465487.
AshIord, S. J., Blatt, R., & VandeWalle, D. (2003). ReIlections on the looking glass: A review oI
research on Ieedback-seeking behavior in organizations. Journal of Management, 29, 773
799.
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis oI the eIIects oI Ieedback in computer-
based instruction. Journal of Educational Computing Research, 13(2), 111127.
Balcazar, F. E., Hopkins, B. L., & Suarez, Y. (1985). A critical, objective review oI perIormance
Ieedback. Journal of Organizational Behavior Management, 7(34), 6589.
Bandura, A. (1991). Social theory oI selI-regulation. Organizational Behavior and Human
Decision Processes, 50, 248287.
Bandura, A., & Cervone, D. (1983). SelI-evaluation and selI-eIIicacy mechanisms governing the
motivational eIIects oI goal systems. Journal of Personality and Social Psychology, 45(5),
10171028.
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional
eIIect oI Ieedback in test-like events. Review of Educational Research, 61(2), 213238.
Baron, R. A. (1988). Negative eIIects oI destructive criticism: Impact on conIlict, selI-eIIicacy, and
task perIormance. Journal of Applied Psychology, 73(2), 199207.
Baron, R. A. (1993). Criticism (inIormal negative Ieedback) as a source oI perceived unIairness in
organizations: EIIects, mechanisms, and countermeasures. In R. Cropanzano (Ed.), Justice in
36
the workplace: Approaching fairness in human resource management (pp. 155170).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Baumeister, R. F., Hutton, D. G., & Cairns, K. J. (1990). Negative eIIects oI praise on skilled
perIormance. Basic & Applied Social Psychology, 11(2), 131149.
Birenbaum, M., & Tatsuoka, K. K. (1987). EIIects oI 'on-line test Ieedback on the seriousness
oI subsequent errors. Journal of Educational Measurement, 24(2), 145155.
Birney, R. C., Burdick, H., & Teevan, R. C. (1969). Fear of failure. New York: Van Nostrand-
Reinhold.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education:
Principles, Policy & Practice, 5(1), 774.
Bordia, P., Hobman, E., Jones, E., Gallois, C., & Callan, V. J. (2004). Uncertainty during
organizational change: Types, consequences, and management strategies. Journal of
Business and Psychology, 18(4), 507532.
BransIord, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind,
experience, and school (expanded ed.). Washington, DC: National Academies Press.
Brophy, J. E. (1981). Teacher praise: A Iunctional analysis. Review of Educational Research,
51(1), 532.
Brosvic, G. M., & Cohen, B. D. (1988). The horizontal vertical illusion and knowledge oI results.
Perceptual and Motor Skills, 67(2), 463469.
Bunderson, V. C., & Olsen, J. B. (1983). Mental errors in arithmetic skills: Their diagnosis in
precollege students (Rep No. NSF SED 80-12500). Provo, UT: WICAT Education
Institute.
Butler, R. (1987). Task-involving and ego-involving properties oI evaluation: EIIects oI diIIerent
Ieedback conditions on motivational perceptions, interest, and perIormance. Journal of
Educational Psychology, 79(4), 474482.
Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in organizational research:
A conceptual and empirical Ioundation. Organizational Behavior and Human Decision
Processes, 67, 2648.
Cheng, S. Y., Lin, C. S., Chen, H. S., & Heh, J. S. (2005). Learning and diagnosis oI individual
and class conceptual perspectives: An intelligent systems approach using clustering
techniques. Computers & Education, 44(3), 257283.
37
Clariana, R. B. (1990). A comparison oI answer-until-correct Ieedback and knowledge-oI-
correct-response Ieedback under two conditions oI contextualization. Journal of
Computer-Based Instruction, 17(4), 125129.
Clariana, R. B. (1999, February). Differential memory effects for immediate and delayed
feedback: A delta rule explanation of feedback timing effects. Paper presented at the
annual Association oI Educational Communications and Technology convention,
Houston, TX.
Cohen, V. B. (1985). A reexamination oI Ieedback in computer-based instruction: Implications
Ior instructional design. Educational Technology, 25(1), 3337.
Collins, C., Brown, J., & Newman, S. (1989). Cognitive apprenticeship: Teaching the craIts oI
reading, writing, and mathematics. In L. Resnick (Ed.), Knowing, learning, and
instruction: Essays in honor of Robert Glaser (pp. 453494). Hillsdale, NJ: Lawrence
Erlbaum Associates.
Corbett, A. T., & Anderson, J. R. (1989). Feedback timing and student control in the LISP
intelligent tutoring system. In D. Bierman, J. Brueker, & J. Sandberg (Eds.), Proceedings
of the fourth international conference on Artificial Intelligence and Education (pp. 6472).
SpringIield, VA: IOS.
Corbett, A. T., & Anderson, J. R. (1990). The eIIect oI Ieedback control on learning to program
with the LISP tutor. In Proceedings: Twelfth annual conference of the Cognitive Science
Society (pp. 796803). Hillsdale, NJ: Lawrence Erlbaum Associates.
Corbett, A. T., & Anderson, J. R. (2001). Locus oI Ieedback control in computer-based tutoring:
Impact on learning rate, achievement and attitudes. In Proceedings of ACM CHI 2001
conference on human factors in computing systems (pp. 245252). New York: ACM Press.
Corno, L., & Snow, R. E. (1986). Adapting teaching to individual diIIerences among learners. In
M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 605629). New
York: Macmillan.
Covington, M. V., & Omelich, C. L. (1984). Task-oriented versus competitive learning
structures: Motivational and perIormance consequences. Journal of Educational
Psychology, 76(6), 10381050.
38
Davis, W. D., Carson, C. M., Ammeter, A. P., & Treadway, D. C. (2005). The interactive eIIects
oI goal orientation and Ieedback speciIicity on task perIormance. Human Performance,
18(4), 409426.
Dempsey, J., Driscoll, M., & Swindell, L. (1993). Text-based Ieedback. In J. Dempsey & G.
Sales (Eds.), Interactive instruction and feedback (pp. 2154). Englewood CliIIs:
Educational Technology Publications.
DihoII, R. E., Brosvic, G. M., Epstein, M. L., & Cook, M. J. (2003). The role oI Ieedback during
academic testing: The delay retention test revisited. The Psychological Record, 53, 533
548.
Dweck, C. S. (1986). Motivational processes aIIecting learning. American Psychologist, 41,
10401048.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and
personality. Psychological Review, 95, 256273.
Epstein, M. L., Lazarus, A. D., Calvano, T. B., Matthews, K. A., Hendel, R. A., Epstein, B. B., et
al. (2002). Immediate Ieedback assessment technique promotes learning and corrects
inaccurate Iirst responses. The Psychological Record, 52, 187201.
Farr, J. L., HoImann, D. A., & Ringenbach, K. L. (1993). Goal orientation and action control
theory: Implications Ior industrial and organizational psychology. In C. L. Cooper & I. T.
Robertson (Eds.), International review of industrial and organizational psychology (pp.
193232). New York: Wiley.
Fedor, D. B. (1991). Recipient responses to perIormance Ieedback: A proposed model and its
implications. Research in Personnel and Human Resources Management, 9, 73120.
Fedor, D. B., Davis, W. D., Maslyn, J. M., & Mathieson, K. (2001). PerIormance improvement
eIIorts in response to negative Ieedback: The roles oI source power and recipient selI-
esteem. Journal of Management, 27(1), 7997.
Fisher, S. L., & Ford, J. K. (1998). DiIIerential eIIects oI learner eIIort and goal orientation on
two learning outcomes. Personnel Psychology, 51, 397420.
Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships oI
goal orientation, metacognitive activity, and practice strategies with learning outcomes
and transIer. Journal of Applied Psychology, 83, 218233.
39
Gaynor, P. (1981). The eIIect oI Ieedback delay on retention oI computer-based mathematical
material. Journal of Computer-Based Instruction, 8(2), 2834.
Gibbs, G. (1988). Learning by doing: A guide to teaching and learning methods. London: Further
Education Unit.
Gilman, D. A. (1969). Comparison oI several Ieedback methods Ior correcting errors by
computer-assisted instruction. Journal of Educational Psychology, 60(6), 503508.
Goldstein, I. L., Emanuel, J. T., & Howell, W. C. (1968). EIIect oI percentage and speciIicity oI
Ieedback on choice behavior in a probabilistic inIormation-processing task. Journal of
Applied Psychology, 52(2), 163168.
Goleman, D. (1995). Emotional intelligence. Bantam Books, New York.
Goodman, J., Wood, R. E., & Hendrickx, M. (2004). Feedback speciIicity, exploration, and
learning. Journal of Applied Psychology 89(2), 248262.
Graesser, A. C., McNamara, D., & VanLehn, K. (2005). ScaIIolding deep comprehension
strategies through AutoTutor and iSTART. Educational Psychologist, 40, 225234.
Grant, L., McAvoy, R., & Keenan, J. B. (1982). Prompting and Ieedback variables in concept
programming. Teaching of Psychology, 9, 173177.
Guzzo, R. A., Jette, R. D., & Katzell, R. A. (1985). The eIIects oI psychologically based
intervention programs on worker productivity: A meta-analysis. Personnel Psychology,
38(2), 275291.
Hanna, G. S. (1976). EIIects oI total and partial Ieedback in multiple-choice testing upon
learning. Journal of Educational Research, 69(5), 202205.
HannaIin, M. J. (1983). The eIIects oI systemized Ieedback on learning in natural classroom
setting. Educational Research Quarterly, 7, 2229.
Hartman, H. (2002). Human learning and instruction (pp. 2369). New York: City College oI
City University oI New York.
Hodes, C. L. (1985). Relative eIIectiveness oI corrective and noncorrective Ieedback in computer
assisted instruction on learning and achievement. Journal of Educational Technology
Systems, 13(4), 249254.
Hoska, D. M. (1993). Motivating learners through CBI Ieedback: Developing a positive learner
perspective. In V. Dempsey & G. C. Sales (Eds.), Interactive instruction and feedback (pp.
105132). Englewood CliIIs, N.J.: Educational Technology Publications.
40
Ilgen, D. R., Fisher, C. D., & Taylor, M. S. (1979). Consequences oI individual Ieedback on
behavior in organizations. Journal of Applied Psychology, 64, 349371.
Johnson, D., & Johnson, R. (1993). Cooperative learning and Ieedback in technology-based
instruction. In J. Dempsey & G. Sales (Eds.), Interactive instruction and feedback (pp.
133157). Englewood CliIIs, NJ: Educational Technology Publications.
Jurma, W. E., & Froelich, D. L. (1984). EIIects oI immediate instructor Ieedback on group
discussion participants. Central States Speech Journal, 35(3), 178186.
KanIer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An
integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied
Psychology, 74(4), 657690.
Kippel, G. M. (1974). InIormation Ieedback schedules, interpolated activities, and retention.
Journal of Psychology, 87, 245251.
Kluger, A. N., & DeNisi, A. (1996). The eIIects oI Ieedback interventions on perIormance: A
historical review, a meta-analysis, and a preliminary Ieedback intervention theory.
Psychological Bulletin, 119(2), 254284.
Kluger, A. N., & DeNisi, A. (1998). Feedback interventions: Toward the understanding oI a
double-edged sword. Current Directions in Psychological Science, 7, 6772.
Knoblauch, C. H., & Brannon, L. (1981). Teacher commentary on student writing: The state oI
the art. Freshman English News, 10(2), 14.
Kolb, D. (1984). Experiential learning. Englewood CliIIs, NJ: Prentice-Hall.
Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47,
211232.
Kulhavy, R. W., & Anderson, R. C. (1972). Delay-retention eIIect with multiple-choice tests.
Journal of Educational Psychology, 63(5), 505512.
Kulhavy, R. W., & Stock, W. (1989). Feedback in written instruction: The place oI response
certitude. Educational Psychology Review, 1(4), 279308.
Kulhavy, R. W., & Wager, W. (1993). Feedback in programmed instruction: Historical context
and implications Ior practice. In J. Dempsey & G. Ales (Eds.), Interactive instruction and
feedback (pp. 320). Englewood CliIIs, NJ: Educational Technology Publications.
41
Kulhavy, R. W., White, M. T., Topp, B. W., Chan, A. L., & Adams, J. (1985). Feedback
complexity and corrective eIIiciency. Contemporary Educational Psychology, 10(3), 285
291.
Kulik, J. A., & Kulik, C. C. (1988). Timing oI Ieedback and verbal learning. Review of
Educational Research, 58(1), 7997.
Lepper, M. R., & Chabay, R. W. (1985). Intrinsic motivation and instruction: ConIlicting views
on the role oI motivational processes in computer-based education. Educational
Psychologist, 20(4), 217230.
Locke, E. A., & Latham, G. P. (1990). A theory of goal setting & task performance. Englewood
CliIIs, NJ: Prentice Hall.
Malone, T. W. (1981). Toward a theory oI intrinsically motivating instruction. Cognitive Science,
5(4), 333370.
Mason, B. J., & Bruning, R. (2001). Providing feedback in computer-based instruction: What the
research tells us. Retrieved February, 15, 2007, Irom
http://dwb4.unl.edu/dwb/Research/MB/MasonBruning.html
Mathan, S. A., & Koedinger, K. R. (2002). An empirical assessment oI comprehension Iostering
Ieatures in an intelligent tutoring system. In S. A. Cerri, G. Gouarderes, & F. Paraguacu
(Eds.), Intelligent tutoring systems, 6th international conference ITS 2002: Vol. 2363.
Lecture notes in computer science (pp. 330343), New York: Springer-Verlag.
Mayer, R. E., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning and
Instruction, 12(1), 107119.
Mayer, J. D., & Salovey, P. (1993). The intelligence oI emotional intelligence. Intelligence,
17(4), 433442.
Mayer, J. D., & Salovey, P. (1997). What is emotional intelligence? In P. Salovey & D. Sluyter
(Eds.), Emotional development and emotional intelligence: Implications for educators (pp.
331). New York: Basic Books.
McColskey, W., & Leary, M. R. (1985). DiIIerential eIIects oI norm-reIerenced and selI-
reIerenced Ieedback on perIormance expectancies, attribution, and motivation.
Contemporary Educational Psychology, 10, 275284.
Merrill, J. (1987). Levels oI questioning and Iorms oI Ieedback: Instructional Iactors in
courseware design. Journal of Computer-Based Instruction, 14, 1822.
42
Moreno, R. (2004). Decreasing cognitive load Ior novice students: EIIects oI explanatory versus
corrective Ieedback in discovery-based multimedia. Instructional Science, 32, 99113.
Morgan, A. (2006). Feedback: Assessment for rather than of learning. Retrieved June 14, 2006,
Irom http://riel.bangor.ac.uk/the/documents/FEEDBACKJanuary06.ppt
Morrison, G. R., Ross, S. M., Gopalakrishnan, M., & Casey, J. (1995). The eIIects oI Ieedback
and incentives on achievement in computer-based instruction. Contemporary Educational
Psychology, 20(1), 3250.
Mory, E. H. (1994). Adaptive Ieedback in computer-based instruction: EIIects oI response
certitude on perIormance, Ieedback-study time, and eIIiciency. Journal of Educational
Computing Research, 11(3), 263290.
Mory, E. H. (2004). Feedback research review. In D. Jonassen (Ed.), Handbook of research on
educational communications and technology (pp. 745783). Mahwah, NJ: Erlbaum
Associates.
Narciss, S., & Huth, K. (2004). How to design inIormative tutoring Ieedback Ior multimedia
learning. In H. M. Niegemann, D. Leutner, & R. Brunken (Ed.), Instructional design for
multimedia learning (pp. 181195). Munster, New York: Waxmann.
Newman, M. I., Williams, R. G., & Hiller, J. H. (1974). Delay oI inIormation Ieedback in an
applied setting: EIIects on initially learned and unlearned items. Journal of Experimental
Education, 42(4), 5559.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent
developments. Educational Psychologist, 38, 14.
Park, O., & Gittelman, S. S. (1992). Selective use oI animation and Ieedback in computer-based
instruction. Educational Technology Research and Development, 40(4), 2738.
Phye, G. D., & Andre, T. (1989). Delayed retention eIIect: Attention, perseveration, or both?
Contemporary Educational Psychology, 14(2), 173185.
Phye, G. D., & Baller, W. (1970). Verbal retention as a Iunction oI the inIormativeness and delay
oI inIormative Ieedback: A replication. Journal of Educational Psychology, 61(5), 380
381.
Phye, G. D., & Bender, T. (1989). Feedback complexity and practice: Response pattern analysis
in retention and transIer. Contemporary Educational Psychology, 14(2), 97110.
43
Phye, G. D., & Sanders, C. E. (1994). Advice and Ieedback: Elements oI practice Ior problem
solving. Contemporary Educational Psychology, 19(3), 286301.
Picard, R. W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., et al. (2004).
AIIective learningA maniIesto. BT Technology Journal, 22(4), 253269.
Pound, L. D., & Bailey, G. D. (1975). Immediate Ieedback less eIIective than delayed Ieedback
Ior contextual learning? Reading Improvement, 12(4), 222224.
Prather, D. C., & Berry, G. A. (1973). Delayed versus immediate inIormation Ieedback on a
verbal learning task controlled Ior distribution oI practice. Education, 93(3), 230232.
Pridemore, D. R., & Klein, J. D. (1991). Control oI Ieedback in computer-assisted instruction.
Educational Technology Research and Development, 39(4), 2732.
Pridemore, D. R., & Klein, J. D. (1995). Control oI practice and level oI Ieedback in computer-
based instruction. Contemporary Educational Psychology, 20, 444450.
Reddy, W. B. (1969). EIIects oI immediate and delayed Ieedback on the learning oI empathy.
Journal of Counseling Psychology, 16(1), 5962.
Roper, W. J. (1977). Feedback in computer assisted instruction. Programmed Learning and
Educational Technology, 14(1), 4349.
Sales, G. C. (1993). Adapted and adaptive Ieedback in technology-based instruction. In J. V.
Dempsey & G. C. Sales (Eds.), Interpretive instruction and feedback. Englewood CliIIs,
NJ: Educational Technology Publications.
Salomon, G., & Globerson, T. (1987). Skill may not be enough: The role oI mindIulness in
learning and transIer. International Journal of Educational Research, 11(6), 623637.
Schimmel, B. J. (1983, April). A meta-analysis of feedback to learners in computerized and
programmed instruction. Paper presented at the annual meeting oI the American
Educational Research Association, Montreal, Canada.
Schimmel, B. J. (1988). Providing meaningIul Ieedback in courseware. In D. Jonassen (Ed.),
Instructional designs for microcomputer courseware (pp. 183195). Hillsdale, NJ:
Lawrence Erlbaum Associates.
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations oI practice: Common principles
in three paradigms suggest new concepts Ior training. Psychological Science, 3(4), 207
217.
44
Schmidt, R. A., Young, D. E., Swinnen, S., & Shapiro, D. C. (1989). Summary knowledge oI
results Ior skill acquisition: Support Ior the guidance hypothesis. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 15(2), 352359.
Schroth, M. L. (1992). The eIIects oI delay oI Ieedback on a delayed concept Iormation transIer
task Contemporary Educational Psychology 17(1), 7882.
Schwartz, F., & White, K. (2000). Making sense oI it all: Giving and getting online course
Ieedback. In K. W. White & B. H. Weight (Eds.), The online teaching guide: A handbook
of attitudes, strategies, and techniques for the virtual classroom (pp. 5772). Boston: Allyn
and Bacon.
Shute, V. J. (2006, April). Assessments for learning: Great idea, but do they work? Paper presented
at the annual meeting oI the American Educational Research Association, San Francisco,
CA.
Shute, V. J., Woltz, D. J., & Regian, J. W. (1989, May). An investigation of learner differences in
an ITS environment: Theres no such thing as a free lunch. Paper presented at the 4th
international conIerence on artiIicial intelligence and educationAI-ED `89,
Amsterdam, Holland.
Sleeman, D. H., Kelly, A. E., Martinak, R., Ward, R. D., & Moore, J. L. (1989). Studies oI
diagnosis and remediation with high school algebra students, Cognitive Science, 13, 551
568.
Smith, P. L., & Ragan, T. J. (1999). Instructional design (2nd ed.). Upper Saddle River, NJ:
Prentice Hall, Inc.
Song, S. H., & Keller, J. M. (2001). EIIectiveness oI motivationally adaptive computer-assisted
instruction on the dynamic aspects oI motivation. Educational Technology Research and
Development, 49(2), 522.
Surber, J. R., & Anderson, R. C. (1975). Delay-retention eIIect in natural classroom settings.
Journal of Educational Psychology, 67(2), 170173.
Swan, M. B. (1983). Teaching decimal place value. A comparative study of conflict and positively-
only approaches (Research Rep. No. 31). Nottingham, England: University oI
Nottingham, Sheel Center Ior Mathematical Education.
Sweller, J., Van Merrienboer, J., & Paas, F. (1998). Cognitive architecture and instructional
design. Educational Psychology Review, 10, 251296.
45
VandeWalle, D., Brown, S. P., Cron, W. L., & Slocum, L. W. (1999). The inIluence oI goal
orientation and selI-regulation tactics on sales perIormance: A longitudinal Iield test.
Journal of Applied Psychology, 84, 249259.
VanLehn, K. (1982). Bugs are not enough: Empirical studies oI bugs, impasses and repairs in
procedural skills. The Journal of Mathematical Behavior, 3(2), 371.
VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The
Andes physics tutoring system: Lessons learned. International Journal of Artificial
Intelligence in Education, 15(3), 147204.
Vygotsky, L. S. (1987). The collected works of L. S. Vygotsky. New York: Plenum.
Wager, W., & Wager, S. (1985). Presenting questions, processing responses, and providing
Ieedback in CAI. Journal of Instructional Development, 8(4), 28.
Waldersee, R., & Luthans, F. (1994). The impact oI positive and corrective Ieedback on
customer service perIormance. Journal of Organizational Behavior, 15(1), 8395.
Waldrop, P. B., Justen, J. E., & Adams, T. M. (1986). A comparison oI three types oI Ieedback
in a computer-assisted instruction task. Educational Technology, 4345.
Whyte, M. M., Karolick, D. M., Neilsen, M. C., Elder, G. D., & Hawley, W. T. (1995).
Cognitive styles and Ieedback in computer-assisted instruction. Journal of Educational
Computing Research, 12(2), 195203.
Wiliam, D. (in press). Keeping learning on track: Iormative assessment and the regulation oI
learning. In F. K. Lester, Jr. (Ed.), Second handbook of mathematics teaching and learning.
Greenwich, CT: InIormation Age Publishing.
Williams, S. E. (1997, March). Teachers written comments and students responses: A socially
constructed interaction. Paper presented at the annual meeting oI the conIerence on
college composition and communication, Phoenix, AZ.
46


Notes
1
An interesting observation is that many Iield studies demonstrate the value oI immediate
Ieedback (see Kulik & Kulik, 1988), while many laboratory studies show positive eIIects oI
delayed Ieedback (see Schmidt & Bjork, 1992; Schmidt et al., 1989).
47