Available online at www.sciencedirect.

com

Procedia - Social and Behavioral Sciences 46 (2012) 3807 – 3811

wces 2012

A comparison between two methods of measurement of meta-cognitive

awareness on mathematical problems solving of university students

Farhad Kazemi1
Department of mathematics,Ghorveh branch, Islamic Azad University, Ghorveh, Kurdistan, Iran

Abstract

The purpose of this study was measurement of university students' meta-cognitive awareness in mathematical
problem solving by two different methods; include "protocols analysis and self-questionnaire". The subjects were
asked to write their total mental processes during solving a non-routine problem. Their papers were analyzed by
Foong's model. Immediately after solving the problem, the students were asked to respond to the valid meta-
cognitive inventory (reliability obtained 0.722 by Cronbach's alpha). The results revealed a moderate and significant
correlation between students meta-cognitive awareness which obtained via meta-cognitive inventory and protocol
analysis (r [64] =0.417, p‫ޒ‬0.05). Consequently, this study suggests that use of multiple methods for measuring meta-
cognition provide a more reliable picture of the phenomena under investigation.
© 2012
© 2010Published
Elsevier Ltd. All rights
by Elsevier Ltd.reserved.
Selection and/or peer review under responsibility of Prof. Dr. Hüseyin Uzunboylu

Keyword: Metacognitive awareness; Mathematical problem solving; Non-routine problems; Protocol-analysis;

1. Introduction

The important of meta-cognitive awareness in learning of students has been widely acknowledged
(Anderson & Walker, 1991, Baird, 1998, Biggs, 1987, Birenbaum, 1996, Brown & De Loache, 1983, Campione,
1987, Gourgey, 1998, Pintrich, & De Groot, 1990, Schraw & Moshman, 1995, Schoenfeld, 1987, and Wilson &
Wing Jan, 1993, 1998). The role of metacognition in teaching and learning was reported by Hartman (1998) as an
especially important issues because it will affects so many elements, such as acquisition, comprehension, retention
and application of what is learned (Wilson, 1999).
Baker and Brown (1984) defined meta-cognition as awareness and control of one’s learning. In another
definition which defined by Flavell (1979) meta-cognition was known as awareness of how one learns, awareness of
when one does and does not understand, knowledge of how to use available information to achieve a goal; ability to
judge the cognitive demands of a particular task, knowledge of what strategies to use for what purposes and
assessment of one’s progress both during and after performance. The cognitive strategies enable one to make
progress to build knowledge, in contrast meta-cognitive strategies enable one to monitor and improve one’s progress
to evaluate understanding and apply knowledge to new situations. Thus, metacognition is vital to cognitive
effectiveness. According to Brown (1987) meta-cognition is divides into two broad categories: (1) knowledge of

1
Farhad Kazemi., Tel.: +989188734486
E-mail address: [email protected]

1877-0428 © 2012 Published by Elsevier Ltd. Selection and/or peer review under responsibility of Prof. Dr. Hüseyin Uzunboylu
doi:10.1016/j.sbspro.2012.06.151
3808 Farhad Kazemi / Procedia - Social and Behavioral Sciences 46 (2012) 3807 – 3811

cognition, as activities that involve conscious reflection on ones cognitive abilities and activitieVDQGUHJXODWLRQ
of cognition, as activities regarding self-regulatory mechanisms during an ongoing attempt to learn or solve
problems. She reported that these two forms of metacognition are closely related, each feeding on the other
recursively, although they can be readily distinguishable.
One of the main problems in each empirical research and especially on the field of metacognition is to
develop and use of valid instruments and methods to measure meta-cognition. According to Georgiades (2004) the
study of meta-cognition is heavily dependents of the development of valid measuring instruments and specifically
appropriate tasks to measure metacognitive ability. There are two main sources which arise the complexity of this
task, first the lack of a generally accepted conceptualization of what really the construct means and second the fact
that metacognition is an inner awareness or process rather than an overt behavior and consequently individuals
themselves are often not conscious of these processes.
There are some empirical studies which conducted by Garner and Alexander (1989) on the measurement of meta-
cognition which suggested that the following questions: “How can we measure knowledge about knowledge more
accurately?”; “How can we measure the effects of strategy training?”
Many researches carried out by researchers to find answer for these questions, designing instruments and
methods to measure metacognition as a whole or components of it; those were then tested with learners in different
domains. These methods are consisting self-questionnaire, where learners themselves rate their metacognitive skills
and knowledge, to interviews or verbal-reports, in which the learners recall what they did and what they thought
during learning experience. All researchers believed that all such methods are fallible, not least because measuring
metacognition is a very difficult task. Thus, many researchers have recommended the use of multiple methods that
do not share the same source of error to provide a more reliable picture of the phenomena under investigation for
measuring meta-FRJQLWLRQ *DUQHU  0HLFKHQEDXP HW DO  6FKRHQIHOG   $FFRUGLQJ WR %URZQ
(1987) ‘theory, using the term meta-cognition to refer to two distinct areas of research makes the research procedure
more difficult and creates confusion clouding interpretation of research findings. Flavell' (1987) believed that
methods for measuring and assessing meta-cognitive awareness would soon be developed, each of the proposed so
far methods has different strengths and weaknesses. For instance, make interview is one of the most popular
methods in measuring metacognition, research has convincingly shown that verbal reports of all types are subject to
many constraints and limitations (cited in Panaoura & Philippou2005). Schoenfeld (1985) stated that there are many
methodologies which will illuminate some aspects of behavior but distort others. The subject's tendency to perform
in front of a microphone, their beliefs about the research requirements or about the discipline as environmental
factors can affect the results.

2. Methodology

Wherefore complexity nature of meta-cognition, there is broad consensus among researchers that all
methodologies applied in this area of research are fallible, have strengths and weaknesses, and feel that the strengths
of one single methodology can complement the weaknesses of another methodology. Thus, we used mixed
methodology (protocols analysis and self-questionnaire) for measuring of meta-cognitive awareness. To this end, we
studied some of inventories that have been used for the measurement of metacognition. For instance, Fortunato,
Hecht, Tittle and Alvarez (1991), Schraw and Sperling-Denisson (1994), Goos, Galbraith and Renshaw (2000)
Sperling et al. (2002), Panaoura& Philippou (2005), and Biryukov (2004). Finally we have developed an inventory
based on the inventories that used by Fortunato, Hecht, Tittle and Alvarez (1991) with seventh-grade students and
Goos et.al (2000). In order to make the inventory more appropriate for older students, we modified the original
version by deleting some items, rewording others and including some new items. In the present study the inventory
consisted of 25-items and students according to their mental processing in during solving a non-routine problem
responded by ticking boxes marked Yes=2, No=0, or Unsure=1, for each item.
For appraisal reliability of this instrument, we used Cronbach's alpha values to evaluate its internal
consistency. After statistic analyzing, the Cronbach’s alpha value of 25 items was 0.722, that indicate the inventory
had a good internal consistency. The participants of this study were 64 university students (34 boys and 30 girls)
who enrolled in calculus course in the Kurdistan university in western of Iran. 30 minutes was allocated for solving
a non-routine mathematics problem.
In this research the students have been asked to write their total mental processes during solving the non-routine
problem. Immediately after solving the problem, the students have been given the inventory to answer the questions
according to their mental processes during solving of the problem. The Problem solving protocols were initially
 Farhad Kazemi / Procedia - Social and Behavioral Sciences 46 (2012) 3807 – 3811 3809

analyzed using a taxonomy modified from the one previously derived in the study on problems solving (Foong,
1993). Taxonomy of problem solving behaviours based on Foong’s taxonomy involved five phases: Problem
orientation behaviour, Heuristic problem solving behaviour, Domain-specific behaviour, Affective behaviour, and
Meta-cognitive behaviour. In this taxonomy meta-cognitive behaviours LQYROYHVWDWLQJDSODQ0FODULI\LQJWDVN
requirements (M2), reviewing progress (M3), recognizing error (M4), and detecting new development (M5). The
students who solved the problem correctly called successful students and students who were failed in solving the
problem called unsuccessful students. The design of this study is independent two sample t-test which compared two
groups of successful and unsuccessful students on meta-cognitive awareness which obtain through protocol analysis
and meta-cognitive inventory as two methods of measurement of meta-cognitive awareness.

3. Data analysis

According to Table 1, which is based on students’ meta-cognitive strategies which obtained through analysis of
solution protocols, among 64 students, 31 students solved the problem correctly and complete (successful students)
and 33 Students were failed in solving the problem(unsuccessful students). This table shows the total number of
meta-cognitive strategies which used by successful students is 132 and this number for unsuccessful students is 118.
In addition, the number of strategies of clarifying task requirements, reviewing progress and detecting new
development which used by successful students are greater than unsuccessful students.

Table 2 : Student’s meta-cognitive strategies frequency according to Foong’s taxonomy

Meta-cognitive Successful
Un successful students
behaviors students
Stating a plan 31 33
Clarifying task
10 6
requirements
Reviewing progress 60 49
Recognizing error 23 23
Detecting new
development 8 7

According to table 2, we can see that base on protocol analysis; the meta-cognitive strategies frequency mean of
successful students was more than unsuccessful students (as much as 0.68). In addition, the results of T-independent
samples test revealed that there is a significant difference at the level p<0.05 between meta-cognitive strategies
mean of successful and unsuccessful students (t[62]=2.11, p<0.05).

Table 2: Results of T-independent samples test for comparing of meta-cognitive strategies frequencies mean between two
groups based on protocol analysis method
groups N Mean Std. Deviation t df Sig(2-tailed)
successful 31 4.25 1.12
2.11 62 0.032
Unsuccessful 33 3.57 1.09

In addition, the frequency of students’ responses to meta-cognitive inventories which consisted of percentage and
number of students who responded to Yes, No or unsure, gathered in table 3. The next row of each statement shows
the number of successful and unsuccessful students who responded to Yes, No or unsure options. For instance, in
question 2, 25% of all students responded to Yes, 61% responded to No and 14% of them responded to unsure and
in next row of this question, 9 successful students and 7 unsuccessful students responded Yes, 19 successful
students and 20 unsuccessful students responded No and 3 successful students and 4 unsuccessful students
responded unsure.

Table 3: frequency of students' responses to statements of meta-cognitive inventory

Statements Yes No Unsure
Successful unsuccessful Successful unsuccessful Successful unsuccessful
1- I read the problem more than once. 100% 0% 0%
31 33 0 0 0 0
3810 Farhad Kazemi / Procedia - Social and Behavioral Sciences 46 (2012) 3807 – 3811

2- I assessed that how much time is needed to solve the 25% 61% 14%
problem. 9 7 19 20 3 4
3- Before solving the problem I assessed my ability for 50% 33% 17%
solving the problem (I can/ I don't can). 17 15 8 13 6 5
4- I tried to find any information that was given in the 81% 7% 12%
problem that needed special attention. 25 27 1 4 5 2
5- I tried to put the problem into my own word. 94% 2% 4%
31 29 0 1 0 3
6- I tried to remember the solution of similar problems. 87% 2% 11%
30 26 1 0 0 7
7- I understood completely that, what the problem is 98% 2% 0%
asking. 31 32 0 1 0 0
8- I tried to use strategies that I knew. 72% 9% 19%
27 19 0 6 4 8
9- When I tried to solve the problem I pose questions to 41% 51% 8%
myself in order to concentrate more. 15 11 14 19 2 3
10- I tried to break the problem into some small 73% 8% 19%
problems and then solve them. 28 19 0 5 3 9
11- During solving the problem, I stopped and checked 67% 7% 26%
the solution method or strategy to make sure it is correct. 25 18 1 3 5 12
12- I checked my work step by step as I was going 78% 11% 11%
through the problem. 28 22 2 5 1 6
13- I asked myself whether I was getting any closer to a 42% 40% 18%
solution. 17 10 8 18 6 5
14- After solving the problem I checked all the steps and 64% 25% 11%
calculation to make sure they were correct. 23 18 8 8 0 9
15- After solving the problem I asked myself whether 42% 46% 12%
my answer made sense. 17 10 11 19 3 4
16- After solving the problem I tried to find some other 19% 73% 8%
solution for it. 6 6 22 25 3 2
17- I looked back over my solution method to check that 55% 26% 19%
I had done what the problem asked. 20 15 4 13 7 5
18- For the better understanding of a problem I usually 86% 4% 10%
draw a picture or diagram. 29 26 0 2 2 5
19- For the better understanding of a problem I usually 67% 16% 17%
write down its important data. 23 20 1 9 7 4
20- While I am solving a problem I try to realize which 42% 28% 30%
are its aspects that I cannot understand. 13 14 7 11 11 8
21- When I encounter a difficulty in during solving a 98% 0% 2%
problem I reread the problem. 31 32 0 0 0 1
22- If I couldn’t progress in solving a problem, I try to 34% 49% 17%
use some other strategies. 14 8 10 20 7 5
23- I usually try to memorize those strategies that I use 61% 20% 19%
for problems solving. 22 17 6 7 3 9
24- When I encounter a difficulty in solving a problem I 82% 14% 4%
try to seek help. 27 26 4 5 0 2
25- When I cannot solve a problem, I identify the factors 50% 26% 24%
of the difficulty. 19 13 5 12 7 8

The results of analysis of students' responses to the metacognitive inventories showed that, the mean of meta-
cognitive awareness of successful students is 38.35 and this mean for unsuccessful students is 32.06. Also The result
of T-independent samples test showed that there is a statically significant difference between mean of meta-
cognitive awareness of successful and unsuccessful students in mathematical problem solving and this difference
was significant at p<0.01 level (t[62]=4.104, p<0.01). the results summarized in table 5.

Table 5: Results of T-independent samples test for comparing of meta-cognitive awareness's mean between two
groups based on inventory method
groups N Mean Std. Deviation t df Sig(2-tailed)
successful 31 38.35 5.51 4.10 62 0.000
 Farhad Kazemi / Procedia - Social and Behavioral Sciences 46 (2012) 3807 – 3811 3811

Unsuccessful 33 32.06 6.65

Another interest finding of this study was, the exits of a moderate and significant correlation between students meta-
cognitive awareness which obtained via meta-cognitive inventory and protocol analysis (r [64]=0.417, p=0.011).
This correlation shows, these two measurement methods are appropriate for measuring meta-cognitive awareness
and also both methods are correlated together.

4. Conclusion

The results of this study showed that the use of different methods for measuring of meta-cognition makes
different results. As mentioned, the means difference of successful and unsuccessful students' meta-cognitive
awareness base on inventory was significant at 0.01 level, while means difference was significant at 0.05 level in
protocols analysis. Although the obtained results by inventory were more accurate rather than the results which
obtained by protocols analysis, but we can't confirm that reliability of this results is completely high; because
answers may be given to please the investigator/ teacher; difficult to answer about at least partially automated
processes; while protocols of problem solving are document and testable. In addition, one of the main problems in
protocols analysis is that there isn't completely agreement between analyses by different researchers.
Although we believe that the inventory has been developed can be used for the measurement of university students
meta-cognitive awareness in mathematical problems solving, but a second study can examine the validity of this
inventory.

References

Baker, L. and A.L. Brown (1984). Metacognitive skills and reading. In P. D. Pearson, R. Barr, J. L. Kamil, and P.
Rosenthal (Eds), Handbook of Reading Research. New York: Longman Press.
Brown, A. (1987). Metacognition, excutive control, self-regulation and other more mysterious mechanisms. In
Frann Weinert & Rainer Kluwe (Eds), Metacognition, Motivation and Understanding, pp. 65-115. London
Flavell, J. (1987). Speculations about the nature and development of metacognition. In F. Weinert, & R. Kluwe
(Eds), Metacognition, Motivation and Understanding, pp. 21-29. London: LEA.
Foong,P.Y. (1993). Development of a framework for analyzing mathematics problem solving behaviors. Singapore
Journal of Education, 13(1), pp.61-75.
Fortunato, I., Hecht, D., Tittle, C., and Alvarez, L. (1991). Metacognition and problem solving, Arithmetic teacher,
39(4), pp. 38-40.
Garner, R., & Alexander, P. (1989). Metacognition: answered and unanswered questions. Educational Psychologist,
24, pp. 143-158.
Georgiades, P. (2004). From the general to the situated: three decades of metacognition, International Journal of
Science Education, 26(3), pp. 365-383.
Goos, M., Galbraith, P., and Renshaw, P. (2000). A money problem: a source of insight into problem-
solving action. Electronic Journal: International Journal for Mathematic Teaching and
Learning, April, 2000. http://www.intermep.org
Hartman, H. (1998). Metacognition in teaching and learning. Instructional Science, 26, pp.1-3.
Panaoura, A, Philippou, G, (2005), The measurement of Young Pupils Metacognitive Ability in Mathematics: the case of
self- Representation and Self- evaluation. European Research in Mathematics Education. Pp. 255. http://
cerme4.crm.es/Papers%20definitius/2/panaoura.philippou.pdf
Sperling, R., Howard, L., Miller, L., and Murphy, C. (2002). Measures of children's knowledge and regulation of
cognition, Contemporary Educational Psychology, 27, pp.51-79.
Schoenfeld, A. H. (1985). Mathematical problem solving, Academic Press, INC, Harcourt Brace Javanovich,
Publishers.
Scraw, G. and Sperling-Dennison, R.(1994). Assessing metacognitive awareness. Contemporary Educational
Psychology, 19, pp. 460-475.
Wilson, J. and L. Wing Jan (1998). Self-assessment for students. Melbourne, Eleanor Curtain.