Assessing the Psychometric Properties of the Achievement Goals

Questionnaire Across Task Contexts

Krista R. Muis

McGill University

Philip H. Winne

Simon Fraser University

Author Note
Correspondence concerning this article should be addressed to Krista R. Muis, McGill
University, Faculty of Education, Department of Educational and Counselling Psychology, 3700
McTavish St., Montreal, Quebec, H3A 1Y2, or via e-mail [email protected].

Support for this research was provided by grants to Philip H. Winne from the Social Sciences
and Humanities Research Council of Canada (410-2007-1159), the Canada Research Chair
Program and Simon Fraser University.

CANADIAN JOURNAL OF EDUCATION 35, 2 (2012): 232-248

©2011 Canadian Society for the Study of Education/
Société canadienne pour l’étude de l’éducation
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 233

Abstract
A program of research is necessary to examine the psychometric properties of instruments
designed to measure individuals’ achievement goal orientations. Recently, research on
achievement goal orientation has examined the stability of achievement goals to assess how
context might influence individuals’ achievement goals. Accordingly, studies are necessary to
establish factorial invariance across contexts. We examined the psychometric properties of the
Achievement Goal Questionnaire (AGQ; Elliot & McGregor, 2001) across task contexts within a
single classroom environment. We tested the factor structure by comparing five competing
models and evaluated the invariance of the factor structure across four task contexts. Results
revealed that the hypothesized four-factor structure was replicated, construct- and discriminant-
related evidence of validity were supported, and both internal consistency and test-retest
reliability estimates were satisfactory. Moreover, invariance held at all levels across the various
contexts.

Keywords: Achievement goal orientation, measurement, factorial invariance, psychometrics,

reliability.

Résumé
Un programme de recherche est nécessaire afin d'examiner les propriétés psychométriques
d'instruments conçus pour mesurer les orientations des individus envers des objectifs de réussite.
Récemment, la recherche sur l'orientation envers des objectifs de réussite s'est intéressée à la
stabilité de ces objectifs afin d'évaluer comment le contexte pouvait influer les orientations
envers des objectifs de réussite chez divers individus. C'est d'ailleurs ceci qui justifie les études
permettant d'établir l'invariance factorielle quels que soient les contextes examinés. À cet effet,
nous avons examiné les propriétés psychométriques du questionnaire sur les objectifs de réussite
(AGQ ; Elliot et McGregor, 2001) dans différents contextes d'activité au sein d'un seul même
environnement en salle de classe. Nous avons étudié la structure factorielle en comparant cinq
modèles différents et avons évalué l'invariance de cette structure selon quatre contextes
d'activités. Les résultats indiquent que la structure à quatre facteurs proposée a été reproduite,
que la validité discriminante et des constructs ont été soutenues, et que les coefficients estimés de
l'uniformité interne et de la fiabilité test-rest sont satisfaisants. D'ailleurs, l'invariance a été
soutenue à tous les niveaux quels que soient les contextes examinés.

Mots-clés: Les orientations envers des objectifs de réussite, la mesure, les propriétés
psychométriques, l'invariance factorielle, la fiabilité.
234 K. R. MUIS & P. WINNE

Assessing the Psychometric Properties of the Achievement Goals Questionnaire Across

Task Contexts

Introduction

Imagine a teacher assigns students a task that provides opportunities for students to work
together, allows them to select a topic for the task, and allows them to submit the assignment for
formative feedback prior to submission for summative assessment. Three weeks later, the same
teacher tells students they will be given an exam graded on normative standards, and scores will
be posted in the class. Would students’ achievement goals be similar for both tasks?
Theoretically, the likely answer is no. As Ames (1992) proposed, the various tasks and learning
activities that teachers set for their students can have powerful influences on how students
engage with the tasks, the amount of effort they expend, and the strategies they choose to
complete the tasks. Specifically, tasks can influence learners’ orientations toward differing
achievement goals. Midgley, Kaplan, and Middleton (2001) define achievement goals as “the
purposes for behavior that are perceived or pursued in a competence-relevant setting” (p.77).
Initially, theorists proposed two types of achievement goals: a mastery goal and a
performance goal (e.g., Anderman & Maehr, 1994; Dweck & Leggett, 1988; Elliot & Dweck,
1988; Maehr & Pintrich, 1991; Meece, 1991). Today, theorists conceptualize achievement goals
within a trichotomous or 2 x 2 framework. Within the trichotomous framework (e.g., Elliot &
Church, 1997; Middleton & Midgley, 1997; Pintrich, 2000), three distinct achievement goal
orientations are proposed: a mastery goal, performance-approach goal, and a performance-avoid
goal. A mastery goal orientation (or mastery-approach orientation) describes learners who strive
to develop competence and task mastery. Learners with a mastery goal orientation are theorized
to believe effort and outcome co-vary. In contrast, a performance-approach goal orientation
characterizes learners who strive to demonstrate aptitude and seek favourable judgments;
demonstrations of competence are in comparison to others. The third goal is a performance-
avoidance orientation, whereby learners strive to avoid appearing unable and avoid negative
judgments. Like the performance-approach orientation, comparisons of competence are made
with other individuals.
The 2 x 2 achievement goal framework (Elliot & McGregor, 2001) divides goals into a
mastery-performance dichotomy plus an approach-avoidance dichotomy. This adds a fourth goal
orientation, a mastery-avoidance orientation, whereby a learner’s goal is to avoid failure rooted
in an intrapersonal perspective (relative to oneself, like the mastery-approach orientation) rather
than in comparison to others. For the mastery-avoid goal construct, incompetence is the focus.
A mastery-avoid oriented learner, for example, may strive to avoid misunderstanding or failing
to learn course material, or strive not to forget what has been learned (Elliot & McGregor, 2001).
Conceptually, the mastery component in a mastery-avoid goal orientation emerges from optimal
antecedents (e.g., motive dispositions, implicit theories, socialization histories) that may
facilitate positive consequences (such as mastery-approach goals; see Elliot & McGregor, 2001,
for a complete discussion). The avoidance component, however, is hypothesized to emerge from
non-optimal antecedents and may result in negative consequences (such as performance-
avoidance goals).
Several studies have examined the consequences of endorsing various goals and how
those goals relate to achievement (Elliot, 1999). For example, performance-approach goals
correlated with positive factors such as absorption during task involvement (Elliot &
Harackiewicz, 1996), high performance outcomes (Elliot & Church, 1997), task value (Wolters,
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 235

Yu, & Pintrich, 1996), and intrinsic motivation (Elliot & Harackiewicz, 1996; Elliot & Church,
1997). Performance-approach goals also have been linked to negative outcomes such as test
anxiety (Elliot & McGregor, 1999; Middleton & Midgley, 1997), low self-efficacy (Skaalvik,
1997), and higher avoidant help-seeking (Middleton & Midgley, 1997; Ryan & Pintrich, 1998).
Performance-avoidance goals have been associated with negative outcomes, such as low
absorption during task engagement (Elliot & Harackiewicz, 1996), an unwillingness to seek help
with schoolwork (Middleton & Midgley, 1997), and reduced intrinsic motivation (Elliot &
Church, 1997). Mastery-approach goals have been found to relate to positive outcomes, such as
long-term retention of information (Elliot & McGregor, 1999), intrinsic motivation (Elliot &
Church, 1997; Elliot & Harackiewicz, 1996), help-seeking (Ryan & Pintrich, 1998), and
persistence (Pintrich, 2000). Finally, a mastery-avoid orientation has been found to correlate
with negative outcomes such as low achievement (Crippen, Biesinger, Muis, & Orgill, 2009),
disorganization and emotionality (Elliot & McGregor, 2001), and positively relate to outcomes
such as fear of failure (Fryer & Elliot, 2007).
Studies have also examined psychometric properties of various types of instruments used
to measure achievement goals (Elliot & McGregor, 2001; Finney, Pieper, & Barron, 2004;
Jagacinski & Duda, 2001; Midgley et al., 1998). For example, Finney et al. (2004) examined the
factor structure of the Achievement Goal Questionnaire (AGQ; Elliot & McGregor, 2001) in a
general academic context. A sample of 2111 freshman undergraduate students completed the
AGQ along with other various instruments. Results supported the four-factor structure that Elliot
and McGregor (2001) hypothesized. Moreover, correlations between each of the four
dimensions were low, and Cronbach alpha reliability coefficients were greater than .70, with the
exception of the performance avoidance subscale (.54).
Our review of the literature suggests the majority of studies have examined relations
between achievement goals and other cognitive, affective, and achievement outcomes.
Psychometric assessments of the various scales also have been conducted on specific samples
(Elliot & McGregor, 2001), between groups to assess factorial invariance (Midgley et al., 1998),
or across general or domain-specific contexts (Finney et al., 2004; Jagacinski & Duda, 2001).
More recently, however, studies have explored the stability of goal orientations across contexts
(Fryer & Elliot, 2007; Muis & Edwards, 2009; Senko & Harackiewicz, 2005; Winne, Muis, &
Jamieson-Noel, 2005). As Pintrich (2000) stated, studies are needed to determine whether
achievement goals are constant across contexts or whether they change as contexts change.
Accordingly, theorists have suggested that achievement goals may not be as stable as theorists
typically assume (Fryer & Elliot, 2007; Muis & Edwards, 2009; Senko & Harackiewicz, 2005;
Winne et al., 2005). Results from the few studies that have examined goal stability found
support for both stability and change (Fryer & Elliot, 2007; Muis & Edwards, 2009; Senko &
Harackiewicz, 2005; Winne et al., 2005).
For example, Senko and Harackiewicz (2005) examined whether individuals who
received ongoing performance feedback on a series of tasks engaged in goal-switching (e.g.,
switching from one goal type to another) or goal intensification (e.g., to strengthen or reduce the
pursuit of one goal). In the first study, tasks included four non-cumulative multiple-choice
exams. For the second study in a laboratory setting, feedback was manipulated such that
participants performed either “well below average” or “well above average” on a first
mathematics task, and “well above average” on the second set of tasks. Results from Senko and
Harackiewicz’s (2005) first study revealed that individuals’ goal orientations remained stable
throughout the semester, although poor exam performance predicted a significant decrease in
mastery-approach and performance-approach goals, and an increase in performance-avoidance
236 K. R. MUIS & P. WINNE

goals. In the second study, they found that negative feedback reduced individuals’ mastery-
approach goals.
In another study, Muis and Edwards (2009) examined the stability and change in
students’ achievement goal orientations across varying tasks over the course of two
undergraduate classes. For both classes, achievement goals were measured four times: prior to
two assignments and two exams. Using four different complimentary techniques, results across
both studies revealed evidence for stability and change across tasks. Moreover, the level of
change that occurred across the different tasks was similar, which the authors argued suggested
that task specificity was not a key factor in the amount of change that occurred over the course of
the semester. Based on these results, they proposed that other factors, such as fear of failure,
anxiety, or interest in a specific task may be more predictive of goal change than the task itself.
We question, however, whether changes in students’ mean level of achievement goals
were a function of true differences or a function of fluctuations in students’ interpretations of
items across the task contexts. In particular, given that few studies have conducted psychometric
assessments of the instruments researchers typically use to measure achievement goals, and
given the new direction in this line of work, more research is needed to further establish
construct validity and reliability. Particularly, it is imperative that factor analytic work is
conducted to establish factorial invariance across contexts. Typically, factorial invariance is
conducted across groups to assess whether group differences are meaningful and valid (i.e., ‘Can
items be interpreted similarly across groups? [Chan, 2000]). Measurement equivalence needs to
be established prior to examining mean differences across groups. If items are not equivalent,
then differences between groups’ means cannot be meaningfully established—the construct may
not be the same across groups.
A parallel argument applies to differences in achievement goals across time or contexts.
When students respond to questionnaire items after receiving performance feedback (Senko &
Harackiewicz, 2005; Winne et al., 2005) or are responding to individual items that focus
specifically on a particular task (Fryer & Elliot, 2007; Muis & Edwards, 2009), context and
content changes. Theoretically, because there may be cumulative effects of feedback or differing
effects due to task specificity on item interpretation, it is not assured a priori that data on
students’ goal orientations reflect the same constructs from time to time.
We addressed these issues by investigating whether students’ goal orientation exhibited
invariant factorial structure across various tasks after receiving feedback about those tasks. As
Chan (2000) argued, this is essential before comparing observed data in exploring for differences
across time and tasks. As well, we add to the empirical literature by further exploring the
psychometric properties of one of the most widely used instruments, the Achievement Goal
Questionnaire (AGQ; Elliot & McGregor, 2001). Specifically, we explored factorial invariance
across four different task contexts, and examined construct- and discriminant-related evidence of
validity and reliability of the items with a sample of 99 university undergraduate students
enrolled in an introductory educational psychology course. Given the small sample size,
precision of estimates may be affected and, as such, we suggest that results are interpreted with
caution.

Method

Participants
Participants were 99 students—13 males and 86 females—taking an educational
psychology course. The mean age of students was 23.12 years (SD = 6.97), and the mean self-
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 237

reported GPA was 3.42 (SD = 2.96, N = 85). The sample was participating in a multi-faceted
study and responded to other questionnaires about study tactics that are not relevant here.

Measures
Goal orientation. We used the 12-item Achievement Goal Questionnaire (AGQ; Elliot
& McGregor, 2001) to assess students’ achievement goals for their educational psychology
course. Students indicated their agreement with each statement on a scale of 1 (not at all true of
me) to 7 (very true of me). The AGQ generates four theoretically orthogonal subscales of three
items each: mastery-approach, mastery-avoidance, performance-approach, and performance-
avoidance. Sample items are: “I desire to completely master the material presented in this class”
(mastery-approach), “I worry that I may not learn all that I possibly could in this class” (mastery-
avoid), “It is important for me to do better than other students” (performance-approach), and
“My goal in this class is to avoid performing poorly” (performance-avoid).

Procedure
In this course, students wrote two five-page papers that required them to reason with
course content and apply it to teaching. They also took two multiple-choice exams, a midterm
covering the fist half of the course, and a final covering the second half of the course. In week 2
of the semester, before participants were given any assignments or exams, they completed the
AGQ. Each time an assignment or exam was handed back with feedback, they completed the
AGQ again. Instructions and items for the AGQ were the same across all tasks. Students were
asked to indicate how well each statement best described them.

Results

Descriptive Statistics and Discriminant Validity

Using SPSS 19 (IBM), we first examined sub-scales of the AGQ for normality. Kline
(1998) suggested using absolute cut-off values of 3.0 for skewness and 8.0 for kurtosis. All
items on the AGQ were well within these ranges (ranging from –1.79 to 0 for skewness and from
–1.18 to 4.88 for kurtosis). Means, standard deviations, and Cronbach alphas are reported in
Table 1. To examine discriminant-related validity, we examined correlations between each of the
theoretically defined goal orientation sub-scales. Correlations should be low to provide evidence
of four distinct dimensions. As shown in Table 2, most correlations between sub-scales across
the four time points were near zero. The exceptions were correlations between the two mastery
orientation sub-scales and between the two avoidance sub-scales, which were moderate. We also
examined correlations between and within each of the goal orientation sub-scales across time. As
shown in Table 3, correlations within each goal orientation ranged from moderate to high,
whereas correlations between each of the goal orientations were predominantly low, although
some moderate correlations were found between the two mastery orientations.
238 K. R. MUIS & P. WINNE

Table 1.
Descriptive Statistics and Cronbach αs for Goal Orientations.

Time 1 Time 2 Time 3 Time 4

Start of Course Think Paper 1 Midterm Think Paper 2
M SD α M SD α M SD α M SD α
Scale

Mastery 5.77 .89 .70 5.54 1.12 .86 5.34 1.20 .87 5.20 1.17 .89
Approach

Mastery-Avoid 4.26 1.49 .85 4.34 1.44 .87 4.29 1.50 .93 4.36 1.46 .90

Performance 4.36 1.40 .90 4.31 1.57 .96 4.22 1.62 .95 3.99 1.53 .96
Approach

Performance- 4.47 1.43 .67 4.36 1.49 .84 4.14 1.47 .87 4.14 1.43 .89
Avoid

Note: Ns varied across times. N = 99, N = 71, N = 76, and N = 53 for times 1, 2, 3, and 4, respectively.
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 239

Table 2.
Correlations Between Each Goal Orientation Across Tasks.

1 2 3
Think Paper 1

MAP1

MAV2 .43**

PAP3 -.05 .08

PAV4 -.01 .41** .32**

1 2 3
Midterm

MAP1

MAV2 .56**

PAP3 .03 .02

PAV4 -.04 .39* .12

1 2 3
Think Paper 2

MAP1

MAV2 .50**

PAP3 -.07 -.12

PAV4 -.10 .39** .13

Note: MAP = mastery approach, MAV = mastery avoidance, PAP = performance approach,
PAV = performance avoidance, * p < .05, and ** p < .01.
240 K. R. MUIS & P. WINNE

Table 3.
Correlations Between Each Goal Orientation Across Time.

Time 2 Time 3 Time 4

1 2 3 4 1 2 3 4 1 2 3 4
Time 1

MAP1 .75 b .44 b .00 -.10 .66 b .40 b -.07 -.12 .66 b .54 b -.13 -.03

MAV2 .37 b .72 b -.04 .32 b .28 a .66 b -.13 .35 b .37 b .68 b -.22 .20

PAP3 -.05 -.05 .77 b .04 -.12 -.15 .74 b .04 -.03 -.01 .74 b -.01

PAV4 .01 .31 b .20 .67b -.08 .23a .15 .73 b -.07 .15 .16 .60 b

Time 2

MAP .89 b .53 b -.08 -.04 .92 b .57 b -.06 -.03

MAV .50 b .86 b .01 .46 b .58 b .89 b -.18 .36 b

PAP -.01 -.13 .92 b .06 -.04 -.06 .80 b -.01

PAV -.12 .35 b .05 .88 b -.06 .34 b .03 .86 b

Time 3

MAP .85 b .51 b -.12 -.07

MAV .44 b .80 b -.16 .36 a

PAP -.05 -.05 .83 b .02

PAV -.05 .36 a .14 .88 b

Note: MAP = mastery approach, MAV = mastery avoidance, PAP = performance approach,
PAV = performance avoidance, a p < .05, and b p < .01.
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 241

Evaluating Fit of the Factor Structure: Construct Validity

EQS 6.0 (Multivariate Software Inc) was used to cross-validate Elliot and McGregor’s
(2001) 2 x 2 model of goal orientation. We used data from the start of the course to test five
competing models also tested by Elliot and McGregor (2001) and Finney et al. (2004) that might
explain relationships among item responses. Model A was a four-factor model based on the 2 x
2 framework for goal orientation. Model B was a three-factor model that included a performance
approach factor, a performance-avoid factor, and an overall mastery factor. Model C was a
three-factor model that included a mastery approach factor, a performance approach factor, and
an overall avoidance factor. Model D was a two-factor model that included an overall mastery
factor and an overall performance factor. Finally, Model E was a two-factor model that included
an overall approach factor and an overall avoidance factor. Given that alternative models B
through E are nested within the four-factor model (the hypothesized model, model A), a chi-
square difference test (Δχ2) can be conducted between Model A and each respective model. If
the Δχ2 is significant, the more complex model (Model A) fits significantly better than the
alternative model. Accordingly, differences between chi-square statistics and the comparative fit
index (CFI) were used to compare models. The CFI is particularly sensitive to misspecified
factor pattern correlations and is a useful fit index when sample sizes are not large (Tabachnick
& Fidell, 1996). A value of .95 or higher indicates good fit (Hoyle & Panter, 1995).
Table 4 displays chi-square, Δχ2, and CFI values for all five models. The chi-square and
CFI values for model A were 83.63 (df = 48, N = 99, p < .01) and .95, respectively. This
suggests a good fit to the data. For model B, the chi-square and CFI values were 127.63 (df =
51, p < .01) and .85, suggesting a moderate fit to the data. Chi-square and CFI values for model
C were 127.54 (df = 51, p < .01) and .85, again, suggesting a moderate fit. Model D resulted in a
chi-square of 176.02 (df = 53, p < .01) and a CFI of .75. Finally, model E resulted in a chi-
square of 204.33 (df = 53, p < .01) and a CFI of .70. Consistent with Elliot and McGregor’s
(2001) and Finney et al.’s (2004) results, our data validate the four-factor model of goal
orientation for mastery approach, mastery avoidance, performance approach, and performance
avoidance.
242 K. R. MUIS & P. WINNE

Table 4.
Fit Statistics for the Five Goal Orientation Models.

Model χ2 Δχ2 Δdf CFI ΔCFI

Four-Factor Model 83.63 ----- ----- .93 -----

(df = 48)

Overall Mastery 127.63 44 3 .85 -.08

(df = 51)

Overall Avoidance 127.54 43.91 3 .85 -.08

(df = 51)

Overall Mastery and 176.02 92.39 5 .75 -.18

Overall Performance (df = 53)

Overall Approach and 204.33 120.70 5 .70 -.23

Overall Avoidance (df = 53)

Invariance of the Factor Structure

We then investigated whether students’ goal orientations exhibited invariant factorial
structure across the different time points. Five steps are involved in testing for factorial
invariance. In step one, the data are tested separately at each time for fit to the model (Byrne,
1998), in our case, the 2 x 2 model of goal orientation. If this test is passed, four more tests are
applied to establish that differences across time can be validly attributed to genuine differences
rather than to differences in the architecture of the construct in different times, or to varying
biases in measurements (Cheung & Rensvold, 2002). First, configural invariance is tested to
examine whether the 2 x 2 architecture of the latent traits that gives rise to item responses does
not vary across time. Second, metric invariance is tested by adding a constraint that loadings of
the item responses on the latent constructs are invariant across time. Third, scalar invariance is
examined to test whether the observed variables have the same degree of statistical bias—the
distance of a mean from a fixed point—over time. This is accomplished by setting the influence
of a latent trait to zero and computing predicted values of the observed variables at each time.
There should be no differences (Hancock, 1997). Last, testing for measurement error invariance
examines invariance of the factor variances and co-variances over time. Invariance of factor co-
variances indicates that relationships among factors are similar across time, and invariance of
factor variances indicates the ranges of scores on factors are similar over time. Because these
two tests are statistically independent, order of assessment is irrelevant.
Separate estimations of the fit of the data to Elliot and McGregor’s 2 x 2 model revealed
adequate fit at each time. The chi-square and CFI values for the factor structure at time 1 (start
of the course) were 83.63 (df = 48, N = 99) and .95, respectively, as we reported previously. At
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 243

time 2, the chi-square and CFI values were 69.91 (df = 48, N = 71) and .97, respectively. At time
3, chi-square and CFI values were 84.04 (df = 48, N = 76) and .95, respectively. Finally, at time
4, chi-square and CFI values were 74.26 (df = 48, N = 53) and .95, respectively.
Because our data adequately fit the 2 x 2 model of goal orientation at each time, we
proceeded to test for invariance of measurement properties by adding successive constraints on
parameters. The fit of the model after adding a constraint was compared to the model at the
previous step in terms of differences in chi-square and CFI (Cheung & Rensvold, 2002). A
ΔCFI value less than or equal to -.01 indicates that the null hypothesis of invariance should not
be rejected (Cheung & Rensvold, 2002). If the fit of the more constrained model is statistically
or practically worse than the less constrained model at the prior step, it would be concluded the
parameters being constrained differ across time; that is, they are not invariant. In Table 5, we
report statistics for these tests of configural invariance, metric invariance, scalar invariance, and
measurement error invariance. All four levels of invariance were satisfied across the four points
in time.

Table 5.
Tests of Invariance Across Contexts.

Model χ2 Δχ2 Δdf CFI ΔCFI

Configural Invariance 310.74 ----- ----- .95 -----

(nothing held invariant) (df = 192)

Metric Invariance 330.61 19.87 24 .95 .00

(coefficients held (df = 216)
invariant)

Scalar Invariance 359.91 29.3 24 .95 .00

(coefficients and (df = 240)
intercepts held invariant)

Measurement Error 369.42 9.51 18 .95 .00

Invariance I (df = 258)
(coefficients, intercepts
and factor covariances
held invariant)

Measurement Error 388.53 19.11 12 .95 .00

Invariance II (df = 270)
(coefficients, intercepts,
factor covariances and
factor variances held
invariant)
244 K. R. MUIS & P. WINNE

Reliability
To assess the reliability of each of the four dimensions, Cronbach’s co-efficient alpha
was calculated within each. With the exception of two of the estimates at the start of the course
(for the mastery approach and performance-avoid subscales), reliabilities were all greater than
.80, which indicated good internal consistency. Test-retest reliabilities were also computed
across the four contexts. Values ranged from .60 to 74 (half of which were greater than .70),
which indicates good test-retest reliability. Reliability statistics are presented in Tables 1 and 6
(for test-retest).

Table 6.
Test-Retest Reliability Coefficients for Each Goal Orientation.

rTime 1 Time 2
• max rTime 1 • rTime 1 Time 3 max rTime
• rTime 1 Time 4
• max rTime 1 •

Time 2 •
1 Time 3 Time 4

Mastery .75 .78 .66 .78 .66 .79

Approach

Mastery- .72 .86 .66 .89 .68 .87

Avoid

Performance .77 .93 .74 .92 .74 .93

Approach

Performance- .67 .75 .73 .76 .60 .77

Avoid

Note: max rtime 1 Time n represents the maximum correlation possible between time 1 and each subsequent time,
•

computed by √(α1•αn).

Discussion
This study contributes to the literature on achievement goal theory by providing further
evidence of construct and discriminant validity, reliability, and factorial invariance across tasks
within a single classroom environment. For construct validity, results from our study
demonstrate support for the four-factor model of achievement goals that Elliot and McGregor
(2001) proposed. Compared to the four competing models, the 2 x 2 framework resulted in a
much better fit than the trichotomous or dichotomous frameworks. The generalizability of the
factor structure with our sample is an important addition to the few psychometric studies that
have been done with achievement goal instruments in general.
The low to moderate correlations between each of the achievement goals across the
various contexts provides evidence of discriminant-ralted validity. Based on our results, we
posit that each of the goal orientations represents a distinct construct. Like Elliot and McGregor
(2001) and Finney et al. (2004), participants in our study endorsed each of the goal orientations
across the various task contexts. Moreover, given that test-retest reliabilities were within an
acceptable range, individuals’ levels of endorsements did not vary much over the different
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 245

contexts, nor did their interpretations of the items designed to measure the four dimensions. That
is, our results unambiguously show that over time and across various contexts, students’
interpretations of the 12 items do not change.
Substantively, results from the tests for factorial invariance indicate that items do not
function differentially for any of the sub-scales across tasks. The range of each latent trait scale,
which measures individuals’ levels of endorsement, is similar over time. Accordingly, full
measurement equivalence at the item level is established across tasks and any direct mean
comparisons to assess changes in levels of goal endorsement over time can be meaningfully
made. Establishing factorial invariance across tasks and time is important given that research in
this area is moving toward examining the stability of goal orientations over tasks, time, and as a
function of feedback. In these situations, it is pertinent that research is conducted to examine
item functioning over time. Our study addressed this gap in the literature.

Limitations and Future Directions

One limitation of our study is sample size. Although data from our sample fit the
hypothesized model quite well, future research could use larger sample sizes. However, we note
an important value that our study adds is the longitudinal and naturalistic nature of the design:
students estimated their achievement goals over time and tasks within one classroom setting.
Given that attrition is problematic with longitudinal designs, we recommend researchers begin
with larger sample sizes to provide more power to detect slight variations in item interpretations.
We also recommend more psychometric work on other instruments designed to measure
students’ achievement goals like the Patterns of Adaptive Learning Scales (Midgley et al., 2000).
Studies are also needed to explore whether slight changes to items that are written for specific
tasks, rather than for a specific course, influence item interpretation or construct validity.
Psychometrically sound instruments that can be modified slightly without altering theoretical
frameworks are needed to further explore the nature of achievement goals and how those goals
influence important educational outcomes.
246 K. R. MUIS & P. WINNE

References
Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational
Psychology, 84, 261-271.
Anderman, E. M., & Maehr, M. L. (1994). Motivation and schooling in the middle grades.
Review of Educational Research, 64, 287-309.
Byrne, B. M. (1998). Structural Equation Modeling with LISREL, PRELIS, and SIMPLIS. NJ:
Lawrence Erlbaum Associates.
Chan, D. (2000). Detection of differential item functioning on the Kirton Adaptation-Innovation
Inventory using multiple-group mean and covariance structure analyses. Multivariate
Behavior Research, 35, 169-199.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing
measurement invariance. Structural Equation Modeling, 9, 233-255.
Crippen, K., Biesinger, K., Muis, K. R., & Orgill, M. K. (2009). The role of goal orientation and
self-efficacy in learning from Web-based worked examples. Journal of Interactive
Learning Research, 20, 385-403.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and
personality. Psychological Review, 95, 256-273.
Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational
Psychologist, 34, 169-189.
Elliot, A. J., & Church, M. A. (1997). A hierarchical model to approach and avoidance
achievement motivation. Journal of Personality and Social Psychology, 72, 218-232.
Elliot, A. J., & Dweck, C.S. (1988). Goals: An approach to motivation and achievement. Journal
of Personality and School Psychology, 54, 5-12.
Elliot, A. J., & Harackiewicz, M. (1996). Approach and avoidance achievement goals and
intrinsic motivation: A mediational analysis. Journal of Personality and Social
Psychology, 70, 461-475.
Elliot, A. J., & McGregor, H. (1999). Test anxiety and the hierarchical model of approach and
avoidance achievement motivation. Journal of Personality and Social Psychology, 76,
628-644.
Elliot, A. J., & McGregor, H. (2001). A 2 X 2 achievement goal framework. Journal of
Personality and Social Psychology, 80, 501-519.
Finney, S. J., Pieper, S. L., & Barron, K. E. (2004). Examining the psychometric properties of
the achievement goal questionnaire in a general academic context. Educational and
Psychological Measurement, 64, 365-382.
Fryer, J. W. & Elliott, A. J. (2007). Stability and change in achievement goals. Journal of
Educational Psychology, 99, 700-714.
Hancock, G. R. (1997). Structural equation modeling methods of hypothesis testing of latent
variable means. Measurement and Evaluation in Counseling and Development, 30, 91-
105.
PSYCHOMETRIC ASSESSMENTS OF ACHIEVEMENT GOALS 247

Hoyle, R. H., & Panter, A. T. (1995). Writing about structural equation models. In R. H. Hoyle
(Ed.), Structual equation modeling: Concepts, issues and applications (pp. 159-176).
Thousand Oaks, CA: Sage.
Jagacinski, C. M., & Duda, J. L. (2001). A comparative analysis of contemporary achievement
goal measures. Educational and Psychological Measurement, 61, 1013-1039.
Kline, R. (1998). Principles and practice of structural equation modeling. New York: Guilford
Press.
Maehr, M. L., & Pintrich, P. R. (1991). Advances in motivation and achievement: Goals and
self-regulatory processes. Greenwich, CT. JAI Press.
Meece, J. L. (1991). The classroom context and students’ motivational goals. In M. L. Maehr &
P. R. Pintrich (Eds.), Advances in motivation and achievement: Goals and self-regulatory
processes (pp. 261-285). Greenwhich, CT: JAI Press.
Middleton, M. J., & Midgley, C. (1997). Avoiding the demonstration of lack of ability: An
unexpected aspect of goal theory. Journal of Educational Psychology, 89, 710-718.
Midgley, C., Kaplan, A., & Middleton, M. (2001). Performance-approach goals: Good for what,
for whom, under what circumstances, and at what cost? Journal of Educational
Psychology, 93, 77-86.
Midgley, C., Kaplan, A., Middleton, M., Maehr, M. L., Urdan, T., Anderman, L. H., Anderman,
E., & Roeser, R. (1998). The development and validation of scales assessing students’
achievement goal orientations. Contemporary Educational Psychology, 23, 113-131.
Midgley, C., Maehr, M. L., Hruda, L. Z., Anderman, E., Anderman, L., Freeman, K. E., et al.
(2000). Manual for the Patterns of Adaptive Learning Scales (PALS). Retrieved from
www.unmich-edu/~pals/pals.
Muis, K. R., & Edwards, O. V. (2009). Examining the stability of achievement goal orientations.
Contemporary Educational Psychology, 34, 265-277.
Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts,
P. R. Pintrich & M. Zeidner (Eds), Handbook of Self-regulation. San Diego, CA:
Academic Press.
Ryan, A. M., & Pintrich, P. R. (1998). Achievement and social motivational influences on help
seeking in the classroom. In S. A. Karabenick (Ed.), Strategic help seeking: Implications
for learning and teaching (pp. 117-139). Mahwah, NJ: Erlbaum.
Senko, C., & Harackiewicz, J. M. (2005). Regulation of achievement goals: The role of
competence feedback. Journal of Educational Psychology, 97, 320-336.
Skaalvik, E. M. (1997). Self-enhancing and self-defeating ego orientation: Relations with task
and avoidance orientation, achievement, self-perceptions, and anxiety. Journal of
Educational Psychology, 89, 71-81.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). New York:
Harper Collins.
248 K. R. MUIS & P. WINNE

Wolters, C., Yu, S., & Pintrich, P. R. (1996). The relation between goal orientation and students’
motivational beliefs and self-regulated learning. Learning and Individual Differences, 8,
211-238.
Winne, P. H., Muis, K. R., & Jamieson-Noel, D. J. (2005). Relationships among achievement
goal orientation, calibration bias and performance in response to successive feedback in
an undergraduate course. Paper presented at the annual American Educational Research
Association.