International Journal of Science Education

ISSN: 0950-0693 (Print) 1464-5289 (Online) Journal homepage: http://www.tandfonline.com/loi/tsed20

Development of an instructional model for higher

order thinking in science among secondary school
students: a fuzzy Delphi approach

G. A. M. Saido, S. Siraj, D. DeWitt & O. S. Al-Amedy

To cite this article: G. A. M. Saido, S. Siraj, D. DeWitt & O. S. Al-Amedy (2018): Development
of an instructional model for higher order thinking in science among secondary school
students: a fuzzy Delphi approach, International Journal of Science Education, DOI:
10.1080/09500693.2018.1452307

To link to this article: https://doi.org/10.1080/09500693.2018.1452307

Published online: 21 Mar 2018.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=tsed20
INTERNATIONAL JOURNAL OF SCIENCE EDUCATION, 2018
https://doi.org/10.1080/09500693.2018.1452307

Development of an instructional model for higher order

thinking in science among secondary school students: a fuzzy
Delphi approach
G. A. M. Saidoa, S. Sirajb, D. DeWittb and O. S. Al-Amedyc
a
Basic Science Department, College of Medicine, University of Duhok, Duhok City, Kurdistan Region, Iraq;
b
Department of Curriculum & Instructional Technology, Faculty of Education, University Malaya, Kuala
Lumpur, Malaysia; cRapporteur of Nursing Department, College of Nursing, University of Duhok, Duhok City,
Kurdistan Region, Iraq

ABSTRACT ARTICLE HISTORY

It is important for science students to develop higher order thinking Received 28 May 2017
(HOT) so that they can reason like scientists in the field. In this study, Accepted 11 March 2018
a HOT instructional model for secondary school science was
KEYWORDS
developed with experts. The model would focus on reflective Higher order thinking; Fuzzy
thinking (RT) and science process skills (SPS) among Grade 7 Delphi Method; reflective
students. The Fuzzy Delphi Method (FDM) was employed to thinking; science process
determine consensus among a panel of 20 experts. First, semi- skills; instructional model
structured interviews were conducted among the experts to
generate the elements required for the model. Then, a
questionnaire was developed using a seven-point linguistic scale
based on these elements. The defuzzification value was calculated
for each item, and a threshold value (d ) of 0.75 was used to
determine consensus for the items in the questionnaire. The
alpha-cut value of >0.5 was used to select the phases and sub-
phases in the model. The elements in the model were ranked to
identify the sub-phases which had to be emphasised for
implementation in instruction. Consensus was achieved on the
phases of the HOT instructional model: engagement,
investigation, explanation, conclusion and reflection. An additional
24 learning activities to encourage RT skills and SPS among
students were also identified to develop HOT skills in science.

Introduction
Science as an inquiry involves reflection on one’s experiences when reasoning about a
phenomenon. The process of scientific inquiry and reasoning produces new scientific
knowledge (Gyllenpalm, Wickman, & Holmgren, 2010). Scientific inquiry has been
defined as a systematic approach used by scientists in finding the answers to questions
about a certain phenomenon (Lederman, 2004). It involves the use of the science
process skills (SPS) for observation, measurement, experimentation, and reasoning.
However, scientific inquiry has often been referred to a set of skills to be learned
through experimental research, using the manipulation of variables to determine causal
relationships (Gyllenpalm et al., 2010). This view of scientific inquiry neglects the

CONTACT D. DeWitt [email protected]

© 2018 Informa UK Limited, trading as Taylor & Francis Group
2 G. A. M. SAIDO ET AL.

understanding of the nature of science (Gyllenpalm et al., 2010). When inquiry is used as a
pedagogical strategy, teachers do not provide students with opportunities for reflection
and model construction (Gyllenpalm et al., 2010; Sun, Wang, Xie, & Boon, 2014).
Scientific inquiry should focus on the nature of science. Gyllenpalm et al. (2010) have
shown that scientific inquiry with scientific reasoning and reflection develops cognitive
understanding. Hence, learners should have opportunities for scientific inquiry similar
to how scientists in real life solve problems and build understandings of phenomena.
Reflective thinking (RT) aids scientific reasoning using cognitive skills for understanding
and evaluating scientific information, theory, statistics and causal hypothesis (Marušić &
Sliško, 2012).
Higher order thinking (HOT) includes scientific reasoning. HOT also involves the
higher level cognitive skills of critical and evaluative thinking, decision-making,
problem-solving and transfer to other situations (Bramwell-Lalor & Rainford, 2014).
HOT is important for applying one’s understanding of science in solving science and tech-
nology-related issues related to the community, the environment or society. For this
reason, the National Research Council’s Study (NRCS) recommends that teachers
should facilitate developing students’ HOT (Yeh, 2012). However, not all of HOT skills
seem to be used during scientific inquiry.
HOT skills can include the SPS. The HOT skills of reasoning for scientific inquiry, as
well as creative thinking and problem-solving are required for the SPS of formulating
hypothesis and models, interpreting data and making predictions (Özgelen, 2012) (see
Figure 1). This suggests that inculcation of SPS for instruction could develop HOT
among students. However, there has been little done on incorporating HOT with SPS
for teaching science and it is not known if incorporating SPS for HOT would produce a
suitable instructional model, and which of the SPS could be included. This study hopes
to fill this gap and to determine the SPS which could be used to improve students’ HOT.
Although SPS are important for scientific reasoning and HOT, students seem to lack
scientific reasoning and HOT. Secondary school students have difficulty in constructing
scientific explanations of phenomena and are not able to make logical relationships
between evidence and explanations (Wu & Hsieh, 2006). Students seem to have difficulty

Figure 1. The relationship between science process skills and HOTS in scientific inquiry.
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 3

in extending their thinking skills to higher levels of thinking when they progress to second-
ary school (Bramwell-Lalor & Rainford, 2014). International assessments among second-
ary students also indicate that students lack HOT skills in science. In the Programme for
International Student Assessment (PISA) 2012 among 15-year-olds for science, less than a
third (19 out of 64) of the countries showed improved performance in science, while more
than half (37 countries) had a mean score below the Organisation for Economic Co-oper-
ation and Development (OECD) average (2014). In fact, for the majority of countries with
comparable data, students’ performance in science has remained unchanged since 2006.
This indicated students were at the lower levels of science proficiency and they lack scien-
tific reasoning and HOT (OECD, 2016).
The lack of proficiency in science might be due to instructional methods. Teachers
seem to emphasise the memorisation of scientific facts and concepts for knowledge acqui-
sition (DeWitt, Alias, & Siraj, 2014; Saido, Siraj, Nordin, & Al-Amedy, 2015; Sun et al.,
2014). In addition, problem-solving and hands-on activities were seldom employed for
teaching (Saido et al., 2015). Further, the scientific inquiry process was taught as an
instructional strategy and neglected the nature of science (Gyllenpalm et al., 2010; Sun
et al., 2014). The reason teachers seem to focus on teacher-centred instructional strategies
to deliver knowledge might be because of the lack of skills and ineffective training for con-
ducting scientific inquiry and HOT (Bramwell-Lalor & Rainford, 2014; Gyllenpalm et al.,
2010; Sun et al., 2014). Teacher-centred instructional strategies were used when teachers
had more teaching hours, and lacked sufficient teaching resources (Sun et al., 2014). Tea-
chers’ beliefs on the usefulness of scientific inquiry and problem-solving also influenced
how they teach science (Gyllenpalm et al., 2010).
Research has suggested that activities which require students to use SPS, RT, critical
thinking for decision-making and problem-solving enables thinking at a higher level
and can be applied to solving complex problems in real-life situations (Gillies, Nichols,
Burgh, & Haynes, 2014; Harle, 2001; Zhai, Jocz, & Tan, 2014). Hence, it is believed that
scientific inquiry that applies SPS for solving relevant and authentic issues can develop stu-
dents’ scientific reasoning and HOT skills (Anastopoulo et al., 2012; Jones et al., 2011).
In order to encourage the processes of scientific reasoning, students need to be provided
opportunities to reflect on the phenomena investigated (Gyllenpalm et al., 2010). Sins,
Savelsbergh, and van Joolingen (2005) elaborate that the skills required for reasoning as
students develop their scientific models are analysis, inductive reasoning, quantify,
explain, and evaluate. Hence, students would be able to make use of scientific reasoning
skills to develop their own models when they analyse the phenomena being investigated,
perform inductive reasoning to develop hypotheses on a model’s behaviour, quantify the
model by expressing it in a mathematical format, explain and document possible reasons
for the model’s elements to be related and evaluate the model from data and results of
experiments (Sins et al., 2005). In summary, scientific reasoning involves the skills of
scientific inquiry: experimentation, evaluation of evidence and inference for conceptual
change or scientific understanding (Marušić & Sliško, 2012; Zimmerman, 2007). In
addition, scientific reasoning can only occur with RT.
RT is an effective strategy for developing HOT. When more science inquiry through
authentic problem-solving tasks, field work, data analysis, and synthesis were combined
with RT in an introductory environmental science program, the undergraduates per-
formed better in the tasks which required HOT skills (Dresner, de Rivera, Fuccillo, &
4 G. A. M. SAIDO ET AL.

Chang, 2014). Hence, activities which encourage RT should be encouraged (Gyllenpalm

et al., 2010). The stimulus during the lesson should enable students to reflect and evaluate
the impact of science and technology and explore possible solutions for building scenarios
for the future (Jones et al., 2011).
Hence this study aimed to develop an instructional model to develop HOT skills which
incorporates RT and SPS for teaching science among seventh-grade students in secondary
schools. This model was developed with consensus among a panel of experts. The model
was developed for teaching seventh-grade students as students at this level had difficulty in
progressing to higher levels of thinking (Bramwell-Lalor & Rainford, 2014).
This model will be significant to assist teachers in developing HOT skills among stu-
dents. Policy planners will be able to use the model to determine the requirements for
teacher training to develop students with HOT skills. HOT would enable students to trans-
fer their knowledge and solve problems in real life.

Science process skills

SPS include the scientific reasoning and thinking processes in scientific inquiry. SPS are
the thinking skills that scientists use to construct scientific knowledge and solve problems
(Özgelen, 2012). There are two types of SPS: basic and integrated. The basic SPS consist of
processes such as observation, forming relationships of space and time, making inferences,
measurement, communication, classification, and prediction, while the integrated SPS
consist of controlling variables, forming operational definitions, formulation of hypoth-
eses and models, interpretation of data, experimentation, and presentation of information
(Cecen, 2012; Özgelen, 2012). In order to formulate hypothesis and models and interpret
data in the integrated SPS, scientific inquiry and reasoning are used. Özgelen’s (2012)
believes that scientific inquiry, scientific reasoning, and HOT are included in SPS. The
skills of reasoning, inquiry, creative thinking, and problem-solving are within the SPS
(Özgelen, 2012). During the proses of experimentation and presentation of information,
HOT skills such as problem-solving and creativity may be used. Hence, HOT skills are
part of the integrated SPS which can support students’ thinking and reasoning.
In addition to critical thinking and RT in the SPS (Kizilkaya & Askar, 2009), HOT
skills enable students to transfer the scientific knowledge and apply to new situations
(Gillies et al., 2014). Hence, activities which involve investigating and solving open-
ended problems, rather than rote learning of facts and science concepts, enable the
development of HOT among students (Krajcik & Mamlok-Naaman, in press). The
active participation of students enables a better understanding (Anastopoulo et al.,
2012). Hence, the SPS for scientific inquiry should include HOT skills for evaluation
and problem-solving.
There seems to be little research investigating the relationship between SPS and HOT
(Özgelen, 2012). Research on SPS have focused on the need to develop basic SPS among
primary and secondary school students (Akinbobola & Afolabi, 2010; Padilla, Okey, & Dil-
lashaw, 1983), and development of integrated SPS among high school students (Aktamis,
2012; Balfakiha, 2010; Caliskan & Kaptan, 2012; Cecen, 2012; Hafizan, Halim, & Meerah,
2012; Kose & Bilen, 2012). However, research on strategies to enhance the development of
SPS seem to be lacking (Aydinli et al., 2011; Karar & Yenice, 2012). Hence, this study seeks
to develop an instructional model which can integrate the SPS to engage students in the
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 5

construction of scientific knowledge and HOT. Therefore, appropriate instructional strat-

egies which are important for developing HOT need to be identified.

Reflective thinking
In the process of scientific reasoning, one needs to reflect for thinking. Hence, the
possibility of incorporating RT in the instructional model for HOT should be con-
sidered. RT is a cognitive strategy which could enable students to develop HOT
skills for solving problems, making decisions, and applying knowledge to real-life
situations (Kizilkaya & Askar, 2009). RT can be used with SPS to improve students’
understanding of science (Seung & Choi, 2016). This is because combining RT with
SPS in activities enables students to form arguments to support their claims with evi-
dence and hence, develop HOT (Seung & Choi, 2016). In this study, an instructional
model which incorporated RT into the SPS for developing HOT in science was
developed.

Instructional models
Past research on science education seem to emphasise the importance of changing the
science curriculum for developing students’ HOT skills (Burke & Williams, 2008; Cru-
jeiras & Jiménez-Aleixandre, 2013; Du, Su, & Liu, 2013; Fensham & Bellocchi, 2013; Mon-
hardt & Monhardt, 2006; Osborne, 2013; Zohar, 2013). There have been some studies
which examined the effect of different instructional methods for improving HOT skills
among high school students (Ersoy & Başer, 2014; Szabo & Schwartz, 2011; Şendağ &
Ferhan Odabaşı, 2009; Yang & Chang, 2013; Zohar, 2013). However, these studies have
noted that there are insufficient instructional models for teaching for HOT which may
have been attributed to the lack of HOT skills among students.
Instructional models to develop HOT skills cannot be based on the traditional theories
of instruction but should be centred on the learner, the learning process, and the learning
environment (Driscoll & Burner, 2005). The present inquiry-based learning approach is
applied to teaching science for the student to develop their SPS such as generating hypoth-
esis, experimenting or observing and evaluating while using inductive, deductive, and
transitional reasoning (Arslan, Göcmencelebi, & Tapan, 2009; Lawson et al., 2000; Mäki-
talo-Siegl, Kohnle, & Fischer, 2011; Sandoval & Morrison, 2003; Zimmerman, 2007).
However, these existing instructional models seem to be insufficient for developing
HOT skills (Dwyer, Hogan, & Stewart, 2010). Hence, there is a need for a more compre-
hensive model for developing HOT skills for a culture of thinking among students in
science (Zohar, 2013).
A survey of teachers’ instructional strategies showed that problem-solving strategies
and hands-on activities were seldom employed (DeWitt, Alias, Palraj, & Siraj, 2017;
Saido et al., 2015). Instructors need to teach cognitive strategies to develop students’
HOT skills for students to do RT, solve problems, and make decisions (Ramirez &
Ganaden, 2008; Zohar & Schwartzer, 2005). An effective student-centred instructional
model allows students to be engaged in the construction of knowledge and generate
new knowledge through activities by providing opportunities to engage in HOT (Miri,
David, & Uri, 2007). Hence, this study aims to develop a HOT instructional model
6 G. A. M. SAIDO ET AL.

which incorporates RT and SPS for teaching science among seventh-grade students, with
consensus among a panel of experts.

The 5E instructional model

The 5E instructional model is used as a framework of the study. This model has been used
to develop the scientific reasoning for scientific inquiry in constructivist learning environ-
ments through the development of science curriculum materials (Bybee et al., 2006). The
model was originally developed for inquiry-based learning by the Biological Science Cur-
riculum Study (BSCS) group and was called the five-step model (Singer & Moscovici, 2008).
The 5E model consists of five phases: engagement, exploration, explanation, elabor-
ation, and evaluation (Bybee et al., 2006). The engagement phase is related to promoting
curiosity and eliciting prior knowledge while engaged with a new concept, while the
exploration phase provides the base for activities to explore current concepts, processes,
and skills to facilitate conceptual change and discover new knowledge (Bybee et al.,
2006). New patterns may be modified or changed during this phase (Singer & Moscovici,
2008). In the explanation phase, the focus is on particular aspects of engagement and
exploration which provides the opportunity for students to demonstrate their conceptual
understanding, process skills, or behaviours in new situations (Bybee et al., 2006; Singer &
Moscovici, 2008). The fourth phase, elaboration, is for teachers to challenge and extend
students’ skills and conceptual understanding for deeper understanding. Finally, in the
evaluation phase, students assess their understanding and abilities, and teachers are pro-
vided opportunities to evaluate their students’ progress in achieving the educational goals.
Evaluation does not indicate the end of the process but should encourage the learner to
think forward for other research opportunities (Singer & Moscovici, 2008).
However, Bybee et al. (2006) noted that his model was not exhaustive and could be
further refined. For this purpose, the 5E model was selected to develop a more comprehen-
sive model for SPS and RT for developing HOT skills among seventh-grade students.

Bruner’s cognitive theory

Bruner’s cognitive theory promotes inquiry through discovery learning for improving stu-
dents’ higher level cognitive skills for HOT. Learning is an active process that learners use
new information to build new ideas and concepts from their prior knowledge (Gredler,
1997). This process of making interconnections between new experiences and prior
knowledge results in the reorganisation of the cognitive structure, which allows the lear-
ners to create new meanings for themselves. Bruner suggested that children progressively
acquire cognitive skills, which he refers to as modes of thinking. These modes are the ways
in which knowledge is kept and encoded into memory. The modes are combined and are
loosely successive as they integrate with each other (Olson, 2007). Based on this, Bruner’s
discovery learning provides the opportunity for students to construct their own meanings
rather than simply memorising.
The process of discovery learning goes through three stages: firstly, the inactive stage, in
which the teacher presents a concept involving direct operation of objects without any
interior representation; secondly, the iconic stage characterised by interior representation
of external objects such as re-introducing a concept using some form of imagery; and
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 7

thirdly, the symbolic stage involving illustration of external objects through words, for-
mulas, or other symbolic means (Bibergall, 1966; Künsting, Kempf, & Wirth, 2013).
When the instructional process goes through these stages, the students’ ability to think
at a higher level will be improved as the new information is integrated with their prior
knowledge. This is done through categorisation and solving problems, and thus improves
the students’ cognitive processes for HOT.
Past studies have revealed the impact of discovery learning as an inquiry in science.
Children who learned in discovery learning environments made better scientific judge-
ments than when learning with direct instruction (Klahr & Nigam, 2004), In addition,
it increased students’ success and inquiry learning skills more than the traditional
method of teaching (Balım, 2009). Therefore, Bruner’s cognitive learning theory has
been adopted in this study to guide experts in selecting appropriate learning activities
for discovery learning with SPS to encourage students HOT.

Methodology
In this study, the Fuzzy Delphi Method (FDM) was used as an analytical method for the
decision-making process to achieve consensus on the phases and sub-phases of an instruc-
tional model for developing HOT among a panel of experts.

Fuzzy Delphi Method

First used by Kaufmann and Gupta (1988) for decision-making, the FDM incorporates
Fuzzy Theory with the classical Delphi method (Hsu, Lee, & Kreng, 2010). The Delphi
method is a decision-making approach that obtains consensus among a panel of
experts by using a sequence of questionnaires with controlled feedback on input from
the experts (Spinelli, 1983). The Delphi method has been effective for achieving consensus
among experts with differing opinions to resolve complex issues and enables a group
decision to be made, rather an individual decision (Baumfield, Conroy, Davis, &
Lundie, 2012). The Delphi method has been used for predicting advent of new technol-
ogies and has been used for the development of new models and products. The logical con-
sensus among a panel of experts is obtained after the experts are given sufficient prospects
to consider their views (Buckley, 1994).
However, the Delphi method requires multiple rounds of questionnaire and feedback
from experts to reach a consensus, and this may be a problem as there may be too
many repeated rounds. To addresses this problem, the FDM was developed by applying
Fuzzy Theory to take into account the fuzziness in group decision-making in the
Delphi Method (Lin & Lu, 2013; Murray, Pipino, & van Gigch, 1985).
Fuzzy logic is used in computers to make decisions based on imprecise quantities in a
similar process in the human brain. Hence, the FDM is an effectual tool for gathering data
subjective data generated from opinions which involves uncertainty and imprecision, to
transform it into quasi-objective quantitative data for easier decision-making in contro-
versial issues (Wu, Wu, Chen, & Chen, 2014). The method has been used widely in devel-
oping products in different research areas such as engineering, business and tourism, and
town planning (Hsu, Hu, Chiou, & Chen, 2011; Kardaras, Karakostas, & Mamakou, 2013;
Wang, Lee, Peng, & Wu, 2013). Thus, because it is an established decision-making
8 G. A. M. SAIDO ET AL.

approach that relies on experts’ opinions to make decisions, the FDM was adopted in this
study to develop the HOT instructional model.

Participants
The focus of the FDM is on the development of a HOT instructional model for science
among seventh-grade students and the views and opinions of a panel of experts in
science instruction will be employed. The criterion of selection of the experts for the
panel is important. The number of experts in the panel may vary substantially from 10
to 50 (Damigos & Anyfantis, 2011). As 15 to 20 experts are recommended for product
development (Kuo & Chen, 2008), this study utilises 20 experts. The experts who were
from the fields of science education and psychology, as well expert teachers and prac-
titioners, had a minimum of 15 years’ experience. The experts were selected based on
their expertise and contributions in their field. In addition, the practitioners who were
expert teachers had to have a Bachelor of Science with Education degree and be involved
in training junior teachers.

Instruments
The instruments in this study were an interview protocol for a semi-structured interview
among experts, and an FDM questionnaire which was developed from the themes which
emerged from the analysis of the interview with experts. For the interview protocol, eight
experts were presented with the 5E model and were asked for their views on the possibi-
lities for the phases of the model and the activities for developing HOT using SPS and RT.
The FDM questionnaire consisted of 29 items in the form of seven-point Likert scale
divided into two sections: the first section was about experts’ views about the model’s
phases and the second section was regarding the experts’ views on the sub-phases for
each phase of the model. In the first section, the experts were questioned as whether
they agreed with the phases for developing HOT: engagement, investigation, conclusion,
reflection, and explanation. In the second section, the experts were asked their agreement
on the sub-phases or activities in each phase. Example, for engagement, the sub-phases
listed were asking critical questions, making estimations, formulating the problem,
making comparisons, and identifying relationships. The questionnaire was piloted
among 10 experts in science education with an alpha Cronbach coefficient of 0.842, indi-
cating high reliability (Santos, 1999).

Data analysis
The interviews with eight experts were transcribed and analysed thematically to form the
items of the FDM questionnaire. The FDM questionnaire was administered to the panel of
20 experts and the analysis was conducted according to the following steps:
Step 1: Determining the linguistic scale: The linguistic scale is a Likert scale with the
addition of fuzzy numbers (Hsieh, Lu, & Tzeng, 2004). In order to address the issue of
fuzziness among experts, three fuzzy values are given for each response to form a triangu-
lar Fuzzy number. In the triangular Fuzzy number, m1 is the minimum value, m2 is the
medium value, and m3 is the maximum value. These numbers m1, m2, and m3 are fuzzy
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 9

numbers between the range of 0 and 1. Hence, for every response in the Likert scale, there
will be three values. For a seven-point linguistic scale, the fuzzy numbers are shown in
Table 1.
Step 2: Computing the average fuzzy number: This is done by identifying the average
responses for every fuzzy number (Benitez, Martín, & Román, 2007).
Step 3: Identifying threshold value ‘d’: The threshold value is important to determine
the consensus level among experts using the equation below:


















































m) 1
d(M,
= (M1 − m1 )2 + (M2 − m2 )2 + (M3 − m3 )2 .
3
According to Chang, Hsu, and Chang (2011), the experts are considered to have achieved a
consensus if the threshold value is less than or equal to 0.2, and the overall group consen-
sus should be more than 75%. Otherwise, the FDM survey should be repeated until a con-
sensus is achieved.
Step 4: Identifying alpha-cut level: In order to select the elements for the HOT instruc-
tional model, most studies used the alpha-cut level which equals to 0.5 as the range of the
fuzzy number is between 0 and 1 (Abdelgawad & Fayek, 2011; Mourhir, Rachidi, & Karim,
2014). Thus, the alpha-cut of 0.5 is used as a cut level to select the elements for the HOT
instructional model. Elements above 0.5 will be selected and the elements below 0.5 will be
omitted.
Step 5: The defuzzification process: In order to justify experts’ consensus on the
elements for the phases and sub-phases in the model, a defuzzification is required. Defuz-
zification is a technique to convert fuzzy number into crisp real number (Thomaidis, Niki-
takos, & Dounias, 2006). The defuzzification value (DV) for each questionnaire item is
calculated using the following equation:
DV = 1/3∗ (m1 + m2 + m3).
The values of m1, m2, and m3 are the mean value of fuzzy number for each expert. The
range of accepted value as reaching the consensus among experts in this study is between
10 and 19. A DV of 10 is the minimum agreement which corresponds to ‘Slightly agree’
while 19 is the maximum value (strongly agree). However, the range of the DV for this
study is shown in Table 2.
Step 6: Ranking the phases and sub-phases (elements) of the model. The ranking is used
to prioritise the elements based on the DV in the model for implementation of instruction
in the science class. The element with the highest DV is ranked highest in priority (For-
temps & Roubens, 1996).

Table 1. Linguistic scale.

Fuzzy number
7-point linguistic scale m1 m2 m3
Strongly agree 0.90 1.00 1.00
Agree 0.70 0.90 1.00
Moderately agree 0.50 0.70 0.90
Slightly agree 0.30 0.50 0.70
Slightly disagree 0.10 0.30 0.50
Disagree 0.00 0.10 0.30
Strongly disagree 0.00 0.00 0.10
10 G. A. M. SAIDO ET AL.

Results
During the interviews, all eight experts from the panel agreed that the five phases of
engagement, exploration, explanation, elaboration, and evaluation should be included in
the model. However, during the interviews, several themes emerged to indicate that
there was a need to change some of the phases.
Firstly, the experts stressed the importance of scientific inquiry for scientific reasoning.
Expert 3 stated, ‘Students need to undergo the scientific inquiry process. They need to
carry out scientific investigations before they can have HOT.’ Hence, the exploration
phase in the initial model was changed to investigation.
In addition, the experts suggested a conclusion phase to replace elaboration. Expert 6
elaborated, ‘We need to have more structured investigations. The SPS are important. So
we should have the students making conclusions.’ Expert 2 said, ‘What is more important
is for students to see what they have learnt and make conclusions on the activities done.
Then they can share their results.’ Hence, the elaboration phase was replaced with
conclusion.
The final phase in the initial model was evaluation. However, the experts felt that
this should be replaced by Reflection. Expert 7, ‘Evaluation seems to indicate assessment
of the learning. This phase should be a review of all the scientific inquiry processes to
reflect on the process and suggest new strategies to solve problems.’ At the same time
Expert 3, ‘When we provide the opportunities to reflect, they will be able to use their
HOT. They need to solve problems.’ Hence, the evaluation phase was replaced with
reflection.
Hence, the phases of engagement, investigation, explanation, conclusion, and reflection
were included in the FDM questionnaire to determine consensus among the panel of
experts. Similarly, the other items for the FDM questionnaire were determined based
on the emergent themes during the interview and included to determine consensus
among the experts.
Other concerns that arose during the interviews were the need for SPS to be taught
in hierarchical order and the feedback mechanism. All the experts agreed that SPS
were essential but had to be appropriate to the learners’ age. The experts believed
it was important to develop the basic skills before the more complex skills. Expert
7 commented, ‘Science teachers should start with improving students” basic skills
and move to more complex skills.’ In addition, Expert 4 stated, ‘you may have to
elaborate and rearrange the skills in a hierarchical order, from simple to complex’.
Hence, in the FDM questionnaire, the SPS were arranged from basic to more
complex skills.

Table 2. Experts’ agreement based on the defuzzification value.

Consensus agreement Defuzzification value (DV)
Strongly agree 19
Agree 17.3
Moderately agree 14
Slightly agree 10
Slightly disagree 6
Disagree 2.6
Strongly disagree 0.66
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 11

As for the feedback mechanism for students, some experts voiced their concern. Expert
3 stated ‘It is important for students to get feedback about what they learned in RT’ while
Expert 4 stated ‘The selected skills are suitable for enhancing students RT and HOT. But
this is not enough. There are other skills which can be added for RT such as getting feed-
back on what they have learned.’ Accordingly, feedback and other appropriate skills were
identified for RT and included in the FDM questionnaire.
After administering the FDM questionnaire, the data collected was analysed according
to the steps in the FDM.

The threshold level

In order to determine consensus among the experts on each item in the questionnaire, the
threshold value d was calculated based on the experts’ responses to the seven-point linguis-
tic scale on the FDM questionnaire. The threshold value, d, equalled 86.20% (more than
75%), and indicated that the required consensus was reached. Hence, since consensus had
been achieved, there was no need to repeat the FDM.

The alpha-cut
The experts were of the opinion that the phases of engagement, investigation, conclusion,
reflection, and explanation should be included in the HOT instructional model (see
Table 3). The decision to include these phases was determined using the alpha-cut level.
As determined earlier, the accepted value for alpha-cut was 0.5. The results show that all
the items achieved the required acceptance level (see Table 3). The maximum value was
obtained for engagement, with a value of 0.82 and the minimum value was for reflection,
at 0.76. Based on the results, all five phases were selected for inclusion in the HOT instruc-
tional model. Similarly, the sub-phases were selected according to the values of alpha-cut
(see Table 4). The results indicated 24 sub-phases or learning activities for encouraging RT
and SPS were selected for the HOT model.

Defuzzification and ranking

Next, the selected elements were ranked for priority for implementation in the science
class. This was done from the DV where the highest ranking was for element with the
highest DV. The phases were ranked in order: Engagement (DV = 16.56), Investigation
(DV = 16.26), Explanation (DV = 15.9), Conclusion (DV = 15.8), and Reflection (DV =
15.26) (see Table 5). Table 6 shows the rankings of the sub-phases of each phase of the
model.
The final model developed from the ranking of the elements is shown in Figure 2.

Table 3. Alpha-cut value for the proposed phases in the HOT teaching model.
Phase Average response Alpha-cut
1.1 Engagement 0.690 0.850 0.945 0.8283
1.2 Investigation 0.680 0.835 0.925 0.8133
1.3 Conclusion 0.650 0.810 0.910 0.7900
1.4 Reflection 0.610 0.775 0.905 0.7633
1.5 Explanation 0.650 0.815 0.920 0.7950
12 G. A. M. SAIDO ET AL.

Table 4. Selection of sub-phases of the HOT instructional model.

Phase Sub-phase Average response Alpha-Cut
A Engagement A1: Asking critical questions 0.700 0.865 0.950 0.842
A2: Estimation 0.630 0.795 0.905 0.777
A3: Formulating the problem precisely 0.700 0.850 0.935 0.828
A4: Making comparisons 0.700 0.860 0.955 0.838
A5: Identifying relationships 0.710 0.860 0.940 0.836
B Investigation B3: Writing procedures 0.750 0.900 0.975 0.876
B2: Planning 0.650 0.800 0.900 0.783
B4: Controlling variables 0.710 0.870 0.950 0.843
B1: Formulating hypothesis 0.800 0.940 0.985 0.908
B5: Measuring 0.700 0.855 0.940 0.831
C Conclusion C1: Expanding 0.770 0.910 0.970 0.883
C2: Generating ideas 0.810 0.945 0.990 0.915
C3: Defining the concept 0.750 0.975 0.900 0.875
C4: Making judgements 0.770 0.915 0.980 0.888
C5: Producing 0.730 0.880 0.955 0.855
D Reflection D1: Identifying experimental error 0.750 0.900 0.975 0.875
D2: Summarise results in a graph 0.830 0.955 0.99 0.925
D3: Suggesting another procedure to solve the problem 0.720 0.885 0.975 0.86
D4: Evaluation of argument 0.760 0.905 0.965 0.876
E Explanation E4: Critiquing 0.790 0.935 0.985 0.903
E1: Organising data 0.760 0.910 0.975 0.881
E2: Checking 0.750 0.905 0.975 0.876
E5: Identifying assumption 0.610 0.785 0.910 0.768
E3: Comparing results 0.720 0.870 0.945 0.845

Table 5. Ranking of the phases in the HOT instructional model.

Phases Fuzzy evaluation Defuzzification Ranking
Engagement 13.8 17 18.9 16.566 1
Investigation 13.6 16.7 18.5 16.266 2
Conclusion 13.0 16.2 18.2 15.800 4
Reflection 12.2 15.5 18.1 15.266 5
Explanation 13.0 16.3 18.4 15.900 3

Discussion
The HOT instructional model developed was based on the 5E model and integrated the
SPS and RT in the phases. The experts agreed that engagement should be the first
phase as it is important to promote curiosity to initiate learning and activate students’
prior knowledge (Bybee et al., 2006). The experts identified that prior knowledge could
be elicited in the engagement phase using the basic SPS such as asking questions,
making comparisons, identifying relationship, formulating the problem, and predicting
the solution (Cecen, 2012; Özgelen, 2012). There is an element of discovery in the instruc-
tional process in this phase as students attempt to identify relationships and make analo-
gies when comparing with other models when formulating the problem. This is the
systemic process of scientific inquiry (Lederman, 2004). The sub-phases in the engage-
ment phase enable teachers to assess students’ understanding and prior knowledge on
the topic, and to identify cognitive conflicts to be resolved and create new meaning
(Gredler, 1997). Similar to Bybee et al.’s (2006) model, this phase motivates students’
learning.
Asking critical questions and making comparisons are essential for scientific inquiry to
understand concepts (Gillies et al., 2014). These cognitive skills help students to attempt to
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 13

Table 6. Ranking of the sub-phases in the HOT instructional model.

Sub-phases Fuzzy evaluation Defuzzification Ranking
A. Engagement phase
A1: Asking critical question 14.2 17.3 19.0 16.83 1
A2: Estimation 12.6 15.9 18.1 15.53 5
A3: Formulating precisely problem 14.0 17.0 18.7 16.57 4
A4: Making comparison 14.0 17.2 19.1 16.77 2
A5: Identifying relationships 14.2 17.2 18.8 16.73 3
B. Investigation phase
B3: Writing procedure 15.1 18.0 19.0 17.37 2
B2: Planning 13.0 16.0 18.0 15.67 5
B4: Controlling variable 14.2 17.4 19.0 16.87 3
B1: Formulating hypothesis 16.0 18.8 19.7 18.17 1
B5: Measuring 14.0 17.1 18.8 16.63 4
E. Explanation phase
E4: Critiquing 15.5 18.2 19.0 17.57 2
E1: Organising data 15.2 18.2 19.5 17.63 1
E2: Checking 15.0 18.1 19.5 17.53 3
E5: Identifying assumption 12.2 15.7 18.2 15.37 5
E3: Comparing result 14.4 17.4 18.9 16.90 4
C. Conclusion Phase
C1: Expanding 15.4 18.2 19.4 17.67 3
C2: Generating idea 16.2 18.9 19.8 18.30 1
C3: Defining the concept 15.0 18.0 19.5 17.50 4
C4: Summarise results in graph 15.4 18.3 19.6 17.77 2
C5: Producing 14.6 17.6 19.1 17.10 5
D. Reflection Phase
D1: Identifying experimental error 15 18 19.5 17.50 3
D2: Making judgement 16.6 19.1 19.8 18.50 1
D3: Suggesting another procedure to solve the problem 14.4 17.7 19.5 17.20 4
D4: Evaluation of argument 15.2 18.1 19.3 17.53 2

represent phenomena and physical processes objectively and logically (Krajcik & Mamlok-
Naaman, in press). Miri et al. (2007) also support the first phase of the model should be
engagement as instruction for enhancing students’ HOT should start with activities to
develop students’ curiosity and initiate problems that lead to conflict. This initial stage
is the inactive stage where the student observes the operations and develops his curiosity.
In the second phase, investigation involves formulation of the problem. This phase
differs from the exploration phase in the 5E model as the experts wanted a more structured
investigation for inquiry in this phase (Bybee et al., 2006; Jones et al., 2011). The experts
suggested developing students’ understanding to a higher level (Singer & Moscovici, 2008)
through the following sub-phases: writing the procedures for investigation, planning the
investigation, controlling variables, and formulating hypothesis in this phase. These
sub-phases are the integrated SPS which required reasoning and HOT (Cecen, 2012;
Özgelen, 2012).
Research has advocated scientific inquiry which allows students to access a variety of
resources related to the topic, with cues and hints for exploration to facilitate the students’
quest for knowledge. This means that one should avoid explaining the evidence or defining
terms until the students have made sufficient trials for investigation in order to solve the
problem (Gyllenpalm et al., 2010; Sun et al., 2014). The teacher plays the role of a facil-
itator for the investigation in this phase. The teacher initiates the activity and gives the
students the time and opportunity to investigate the scientific concept using the student’s
own ideas of the phenomena or problem solution. In this phase, the student attempts to
14 G. A. M. SAIDO ET AL.

Figure 2. The elements in an instructional model for HOT in science for seventh grade.

represent internally the external objects in the investigation in the iconic stage (Bibergall,
1966; Künsting et al., 2013).
The third phase is explanation. In this phase, a specific aspect of the students’ engage-
ment and investigation is emphasised so that students are provided the opportunity to ver-
balise their understanding and explain the concepts in their own words. This is similar to
Bybee et al.’s (2006) model where one particular aspect of engagement and exploration is
focused upon so that students can demonstrate their conceptual understanding and skills.
This occurs when students connect their prior knowledge in the first phase to the new dis-
coveries through forming and communicating new understandings in the investigation
phase. In the explanation phase, the experts identified the following sub-phases: organis-
ing data, critiquing, checking, comparing results, and identifying assumptions. These cog-
nitive skills are the integrated SPS or HOT skills which help students form hypotheses and
models and interact in a positive, supportive manner in explaining the concept (Cecen,
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 15

2012; Özgelen, 2012). Through these activities, students will be able to represent the exter-
nal objects by an interior representation as they attempt to construct their own under-
standing of the concept and verbalise the concepts using their own words in the
symbolic stage (Bibergall, 1966; Künsting et al., 2013).
The fourth phase in the HOT instructional model is the conclusion. The aim of this phase
is to extend students’ conceptual understanding and allow students opportunities to clarify
newly discovered concepts, and to make conclusions through these new experiences. This
differs from the 5E model where during elaboration, teachers challenge and extend students’
conceptual understanding and skills so that they develop deeper understandings (Bybee
et al., 2006). Instead, the conclusion phase consists of the following sub-phases: generating
ideas, summarising the results in a graph, expanding, defining the concept and production.
The SPS for critical and RT are used in making conclusions (Kizilkaya & Askar, 2009). As
students generate new ideas, they move from concrete experiences to abstract experiences
and attempt to produce a symbolisation of their understanding (Bibergall, 1966; Künsting
et al., 2013). The production sub-phase requires the student to use HOT to produce an appli-
cation of their understanding, example a scientific model.
The final phase is Reflection. In this phase, feedback on the processes in previous
phases is made by taking time to review the initial problem, the path taken to solve it,
and review the conclusions. The experts’ reached consensus on the four RT skills to assist
students in reflection on their experience: evaluating the argument, identifying experimental
error, thinking of new ways to solve the problem by suggesting alternative procedures, and
making judgement on the solutions (Yeh, 2012). In this phase, activities such as journal
writing, group discussion and presentations, and creating students’ examples of solutions
to discuss can be used for reflection. This phase differs from the evaluation phase of the
5E model where the focus is on evaluating students’ progress (Bybee et al., 2006; Singer
& Moscovici, 2008). In addition, the sub-phases are not SPS, but activities which support
scientific reasoning for thinking critically and reflectively (Kizilkaya & Askar, 2009).
Hence, the HOT instructional model developed through experts’ consensus can be used
for science instruction using SPS and RT skills for scientific reasoning and HOT skills for
solving complex problems (Gillies et al., 2014; Harle, 2001; Zhai et al., 2014).

Conclusion
In this study, an instructional model to develop HOT skills was developed based on the
consensus among experts. The 5E instructional model for scientific reasoning was initially
used and the SPS and RT were integrated for a comprehensive model (Arslan et al., 2009;
Bybee et al., 2006; Lawson et al., 2000; Mäkitalo-Siegl et al., 2011; Sandoval & Morrison,
2003; Zimmerman, 2007; Zohar, 2013).
Although the 5E model was used, the experts agreed that the phases needed to be
changed to incorporate SPS and HOT. The phases identified were Engagement, Investi-
gation, Explanation, Conclusion, and Reflection. Each of the phases had several sub-
phases as learning activities for students to develop higher level cognitive skills in
science. Based on this framework, the experts achieved consensus on the appropriate
SPS and RT skills which could be used to improve HOT among seventh-grade students.
This study has several contributions to the field of science education. Firstly, the
instructional model developed integrated the SPS and RT for developing HOT skills
16 G. A. M. SAIDO ET AL.

among students, thus filling the gap on this area of knowledge. This model emphasises scien-
tific inquiry and production through the investigation, explanation, and the conclusion
phases. The experts required a more structured investigation phase for scientific inquiry
as compared to the exploration phase initially, and a conclusion phase for production to
exhibit their ideas symbolically, instead of the initial elaboration phase. Finally, a reflection
phase to evaluate progress and encourage reasoning rather than the initial evaluation phase
was agreed upon. The SPS detailed for each phase differs from previous models as it empha-
sised HOT through additional processes: asking critical questions and formulating problems
(during the Engagement phase), formulating hypothesis (during the Investigation phase),
critiquing the data and checking the results (during the Explanation phase), making judge-
ment and producing models (during the Conclusion phase) and evaluating an argument,
identifying experimental errors, and suggesting alternative procedures in solving a
problem (during the Reflection phase) (Cecen, 2012; Özgelen, 2012).
Secondly, the FDM which was used to achieve consensus among experts could be used for
the development of instructional models. The experts agree that scientific inquiry was impor-
tant for scientific reasoning for the development of HOT, and hence, the model which was
developed was a systematic approach to finding answers to questions about a phenomenon
(Lederman, 2004). The instructional model developed was a scientific inquiry based on the
nature of science as the first phase involved asking critical questions and formulating pro-
blems (Gyllenpalm et al., 2010; Sun et al., 2014). A conclusion phase was added as the
experts believed that scientific inquiry should be systematic and hence include a conclusion
(Lederman, 2004). In addition, the experts identified skills for the reflection phase to include
evaluation, problem-solving, and decision-making for HOT (Bramwell-Lalor & Rainford,
2014). Hence, the FDM proved to be a valuable process to achieve consensus among
experts in developing an instructional model. In future, studies should be done to explore
the implementation and effectiveness of the instructional model developed, which employed
SPS for scientific inquiry and RT for developing HOT among students.
The results of this study are significant for teachers and curriculum designers in science
education. The HOT instructional model could be used during professional development
programs for both preservice and in-service teachers in science education. This is impor-
tant as teachers need to know the processes and skills required for developing students’
HOT. Thus, it is crucial for teachers to have effective instructional models in order to
create a supportive environment for improving HOT skills among students.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
The work was supported in part by the Fundamental Research Grant Scheme (FRGS), Ministry of
Higher Education Malaysia [grant FP011-2014B].

References
Abdelgawad, M., & Fayek, A. R. (2011). Fuzzy reliability analyzer: Quantitative assessment of risk
events in the construction industry using fuzzy fault-tree analysis. Journal of Construction
Engineering and Management, 137(4), 294–302.
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 17

Akinbobola, A. O., & Afolabi, F. (2010). Analysis of science process skills in West African senior
secondary school certificate physics practical examinations in Nigeria. American-Eurasian
Journal of Scientific Research, 5(4), 234–240.
Aktamis, H. (2012). Determination of the effect of the science and technology curriculum on devel-
oping students’ science process skills: A Turkish case study. Energy Education Science and
Technology Part B-Social and Educational Studies, 4(1), 419–432.
Anastopoulo, S., Sharpels, M., Ainsworth, S., Crook, C., O’Malley, C., & Wright, M. (2012).
Creating personal meaning through technology-supported science inquiry learning across
formal and informal settings. International Journal of Science Education, 34(2), 251–273.
Arslan, C., Göcmencelebi, S. I., & Tapan, M. S. (2009). Learning and reasoning styles of pre service
teachers’: Inductive or deductive reasoning on science and mathematics related to their learning
style. Procedia – Social and Behavioral Sciences, 1(1), 2460–2465.
Aydinli, E., Dokme, I., Unlua, Z. K., Ozturk, N., Demir, R., & Benli, E. (2011). Turkish elementary
school students’ performance on integrated science process skills. Procedia - Social and
Behavioral Sciences, 15, 3469–3475. doi:10.1016/j.sbspro.2011.04.320
Balfakiha, N. M. (2010). The assessment of the UAE’s in-service and pre-service elementary science
teachers in the integrated science process skills. Procedia - Social and Behavioral Sciences, 2,
3711–3715.
Balım, A. G. (2009). The effects of discovery learning on students’ success and inquiry learning
skills. Eurasian Journal of Educational Research, 35, 1–20.
Baumfield, V. M., Conroy, J. C., Davis, R. A., & Lundie, D. C. (2012). The Delphi method: Gathering
expert opinion in religious education. British Journal of Religious Education, 34(1), 5–19. doi:10.
1080/01416200.2011.614740
Benitez, J. M., Martín, J. C., & Román, C. (2007). Using fuzzy number for measuring quality of
service in the hotel industry. Tourism Management, 28(2), 544–555.
Bibergall, J. A. (1966). Learning by discovery: Its relation to science teaching. Educational Review,
18(3), 222–231. doi:10.1080/0013191660180307
Bramwell-Lalor, S., & Rainford, M. (2014). The effects of using concept mapping for improving
advanced level biology students’ lower- and higher-order cognitive skills. International Journal
of Science Education, 36(5), 839–864. doi:10.1080/09500693.2013.829255
Buckley, C. C. (1994). Delphi technique supplies the classic result? The Australian Library Journal,
43(3), 158–164. doi:10.1080/00049670.1994.10755684
Burke, L. A., & Williams, J. M. (2008). Developing young thinkers: An intervention aimed to
enhance children’s thinking skills. Thinking Skills and Creativity, 3(2), 104–124.
Bybee, R. W., Taylor, J. A., Gardner, A., Van Scotter, P., Powell, J. C., Westbrook, A., & Landes, N.
(2006). The BSCS 5E instructional model: Origins and effectiveness. Colorado Springs, CO: BSCS.
Caliskan, I. O., & Kaptan, F. (2012). Reflections of performance assessment on science process
skills, attitude and retention in science education. Hacettepe Universitesi Egitim Fakultesi
Dergisi-Hacettepe University Journal of Education, 43, 117–129.
Cecen, M. A. (2012). The relation between science process skills and reading comprehension levels
of high school students. Energy Education Science and Technology Part B-Social and Educational
Studies, 4(1), 283–292.
Chang, P. L., Hsu, C. W., & Chang, P. C. (2011). Fuzzy Delphi method for evaluating hydrogen
production technologies. International Journal of Hydrogen Energy, 36(21), 14172–14179.
Crujeiras, B., & Jiménez-Aleixandre, M. P. (2013). Challenges in the implementation of a compe-
tency-based curriculum in Spain. Thinking Skills and Creativity, 10, 208–220. doi:10.1016/j.tsc.
2013.07.001
Damigos, D., & Anyfantis, F. (2011). The value of view through the eyes of real estate experts: A
Fuzzy Delphi approach. Landscape and Urban Planning, 101(2), 171–178.
DeWitt, D., Alias, N., Palraj, S., & Siraj, S. (2017). Problem solving strategies among science teachers
in the state of Selangor, Malaysia. In K. Persichitte, A. Suparman, & J. M. Spector (Eds.),
Educational technology to improve quality and access on a global scale: Papers from the
Educational Technology World Conference (ETWC 2016) (pp. 123–141). New York, NY:
Springer.
18 G. A. M. SAIDO ET AL.

DeWitt, D., Alias, N., Siraj, S. (2014). The design and development of a collaborative mLearning
prototype for Malaysian secondary school science. Education Technology Research and
Development. doi.10.1007/s11423-014-9340-y
Dresner, M., de Rivera, C., Fuccillo, K. K., & Chang, H. (2014). Improving higher-order thinking
and knowledge retention in environmental science teaching. BioScience, 64(1), 40–48. doi:10.
1093/biosci/bit005
Driscoll, M. P., & Burner, K. J. (2005). The cognitive revolution and instructional design. In J. M.
Royer (Ed.), The cognitive revolution in educational psychology (pp. 199–229). Connecticut:
Information Age.
Du, X., Su, L., & Liu, J. (2013). Developing sustainability curricula using the PBL method in a
Chinese context. Journal of Cleaner Production, 61, 80–88. doi:10.1016/j.jclepro.2013.01.012
Dwyer, C. P., Hogan, M. J., & Stewart, I. (2010). The evaluation of argument mapping as a learning
tool: Comparing the effects of map reading versus text reading on comprehension and recall of
arguments. Thinking Skills and Creativity, 5(1), 16–22.
Şendağ, S., & Ferhan Odabaşı, H. (2009). Effects of an online problem based learning course on
content knowledge acquisition and critical thinking skills. Computers & Education, 53(1),
132–141. doi:10.1016/j.compedu.2009.01.008
Ersoy, E., & Başer, N. E. (2014). The effects of problem-based learning method in higher education
on creative thinking. Procedia - Social and Behavioral Sciences, 116, 3494–3498.
Fensham, P. J., & Bellocchi, A. (2013). Higher order thinking in chemistry curriculum and its
assessment. Thinking Skills and Creativity, 10(0), 250–264. doi:10.1016/j.tsc.2013.06.003
Fortemps, P., & Roubens, M. (1996). Ranking and defuzzification methods based on area compen-
sation. Fuzzy Sets and Systems, 82(3), 319–330.
Gillies, R. M., Nichols, K., Burgh, G., & Haynes, M. (2014). Primary students’ scientific reasoning
and discourse during cooperative inquiry-based science activities. International Journal of
Educational Research, 63, 127–140. doi:10.1016/j.ijer.2013.01.001
Gredler, E. G. (1997. Learning and instruction: Theory into practice (3rd ed.). Upper Saddle River,
NJ: Merrill Prentice Hall.
Gyllenpalm, J., Wickman, P., & Holmgren, S. (2010). Teachers’ language on scientific inquiry:
Methods of teaching or methods of inquiry? International Journal of Science Education, 32(9),
1151–1172.
Hafizan, E., Halim, L., & Meerah, T. S. (2012). Perception, conceptual knowledge and competency
level of integrated science process skill towards planning a professional enhancement pro-
gramme. Sains Malaysiana, 41(7), 921–930.
Harle, W. (2001). Research in primary science education. Journal of Biological Education, 35(2), 61–
65. doi:10.1080/00219266.2000.9655743
Hsieh, T. Y., Lu, S. T., & Tzeng, G. H. (2004). Fuzzy MCDM approach for planning and design
tenders selection in public office buildings. International Journal of Project Management, 22
(7), 573–584.
Hsu, C. W., Hu, A. H., Chiou, C. Y., & Chen, T.-C. (2011). Using the FDM and ANP to construct a
sustainability balanced scorecard for the semiconductor industry. Expert Systems with
Applications, 38(10), 12891–12899. doi:10.1016/j.eswa.2011.04.082
Hsu, Y.-L., Lee, C.-H., & Kreng, V. B. (2010). The application of Fuzzy Delphi Method and Fuzzy
AHP in lubricant regenerative technology selection. Expert Systems with Applications, 37(1),
419–425. doi:10.1016/j.eswa.2009.05.068
Jones, A., Buntting, C., Hipkins, R., McKim, A., Conner, L., & Saunders, K. (2011). Developing stu-
dents’ futures thinking in science education. Research in Science Education, 42, 1–22. doi:10.
1007/s11165-011-9214-9
Karar, E. E., & Yenice, N. (2012). The investigation of scientific process skill level of elementary
education 8th grade students in view of demographic features. Procedia – Social and
Behavioral Sciences, 46(0), 3885–3889. doi:10.1016/j.sbspro.2012.06.166
Kardaras, D. K., Karakostas, B., & Mamakou, X. J. (2013). Content presentation personalisation and
media adaptation in tourism web sites using Fuzzy Delphi Method and fuzzy cognitive maps.
Expert Systems with Applications, 40(6), 2331–2342. doi:10.1016/j.eswa.2012.10.031
 INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 19

Kaufmann, A., & Gupta, M. M. (1988). Fuzzy mathematical models in engineering and management
science. New York, NY: Elsevier Science Inc.
Kizilkaya, G., & Askar, P. (2009). The development of A reflective thinking skill scale towards
problem solving. Egitim Ve Bilim-Education and Science, 34(154), 82–92.
Klahr, D., & Nigam, M. (2004). The equivalence of learning paths in early science instruction:
Effects of direct instruction and discovery learning. Psychological Science, 15(10), 661–667.
Kose, S., & Bilen, K. (2012). The effect of laboratory activities based on a POE strategy for class
teacher candidates’ achievements, science process skills and understanding the nature of
science. Energy Education Science and Technology Part B-Social and Educational Studies, 4(4),
2357–2368.
Krajcik, J. S., & Mamlok-Naaman, R. (in press). Using driving questions to motivate and sustain
student interest in learning science. In K. Tobin (Ed.), Teaching and Learning Science: An
Encyclopedia (pp. 317–327). Westport, CT: Greenwood Publishing Group.
Künsting, J., Kempf, J., & Wirth, J. (2013). Enhancing scientific discovery learning through meta-
cognitive support. Contemporary Educational Psychology, 38(4), 349–360. doi:10.1016/j.
cedpsych.2013.07.001
Kuo, Y. F., & Chen, P. C. (2008). Constructing performance appraisal indicators for mobility of the
service industries using Fuzzy Delphi Method. Expert Systems with Applications, 35(4), 1930–
1939.
Lawson, A. E., Clark, B., Cramer-Meldrum, E., Falconer, K. A., Sequist, J. M., & Kwon, Y. J. (2000).
Development of scientific reasoning in college biology: Do Two levels of general hypothesis-
testing skills exist? Journal of Research in Science Teaching, 37(1), 81–101.
Lederman, N. G. (2004). Syntax of nature of science within inquiry and science instruction. In N.
Lederman (Ed.), Scientific inquiry and the nature of science (pp. 301–317). London: Kluwer.
Lin, L.-Z., & Lu, C.-F. (2013). Fuzzy group decision-making in the measurement of ecotourism sus-
tainability potential. Group Decision and Negotiation, 22(6), 1051–1079. doi:10.1007/s10726-
012-9305-7
Mäkitalo-Siegl, K., Kohnle, C., & Fischer, F. (2011). Computer-supported collaborative inquiry
learning and classroom scripts: Effects on help-seeking processes and learning outcomes.
Learning and Instruction, 21(2), 257–266.
Marušić, M., & Sliško, J. (2012). Influence of three different methods of teaching physics on the gain
in students’ development of reasoning. International Journal of Science Education, 34(2), 301–
326. doi:10.1080/09500693.2011.582522
Miri, B., David, B.-C., & Uri, Z. (2007). Purposely teaching for the promotion of higher-order think-
ing skills: A case of critical thinking. Research in Science Education, 37(4), 353–369.
Monhardt, L., & Monhardt, R. (2006). Creating a context for the learning of science process skills
through picture books. Early Childhood Education Journal, 34(1), 67–71.
Mourhir, A., Rachidi, T., & Karim, M. (2014). River water quality index for Morocco using a fuzzy
inference system. Environmental Systems Research, 3(1), 84–12.
Murray, T. J., Pipino, L. L., & van Gigch, J. P. (1985). A pilot study of fuzzy set modification of
Delphi. Human Systems Management, 5(1), 76–80.
Olson, D. R. (2007). Jerome Bruner: The cognitive revolution in educational theory (Vol. 3).
New York, NY: Continuum International Publishing Group.
Organisation for Economic Co-operation and Development. (2014). A profile of student perform-
ance in science in PISA 2012 results: What students know and can do (volume 1, revised
edition, February 2014): Student performance in mathematics, reading and science. Paris:
Author. doi.10.1787/97789264208780-9-en
Organisation for Economic Co-operation and Development. (2016). PISA 2015: Results in focus.
Retrieved from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf
Osborne, J. (2013). The 21st century challenge for science education: Assessing scientific reasoning.
Thinking Skills and Creativity, 10(0), 265–279. doi:10.1016/j.tsc.2013.07.006
Özgelen, S. (2012). Students’ science process skills within a cognitive domain framework. Eurasia
Journal of Mathematics, Science & Technology Education, 8(4), 283–292.
20 G. A. M. SAIDO ET AL.

Padilla, M. J., Okey, J. R., & Dillashaw, F. G. (1983). The relationship between science process skill
and formal thinking abilities. Journal of Research in Science Teaching, 20(3), 239–246.
Ramirez, R. P. B., & Ganaden, M. S. (2008). Creative activities and students’ higher order thinking
skills. Education Quarterly, 66(1), 22–33.
Saido, A. M. G., Siraj, S., Nordin, A. B., & Al-Amedy, O. S. (2015). Teaching strategies for promot-
ing higher order thinking skills: A case of secondary science teachers. Malaysian Online Journal
of Educational Management, 3(4), 16–30.
Sandoval, W. A., & Morrison, K. (2003). High school students’ ideas about theories and theory
change after a biological inquiry unit. Journal of Research in Science Teaching, 40(4), 369–392.
Santos, J. R. A. (1999). Cronbach’s alpha: A tool for assessing the reliability of scales. Journal of
Extension, 37(2), 1–5.
Seung, E., & Choi, A. (2016). University students’ understanding of chemistry processes and the
quality of evidence in their written arguments. Eurasia Journal of Mathematics, Science &
Technology Education, 12(4), 991–1008.
Singer, F. M., & Moscovici, H. (2008). Teaching and learning cycles in a constructivist approach to
instruction. Teaching and Teacher Education, 24(6), 1613–1634.
Sins, P. H. M., Savelsbergh, E. R., & van Joolingen, W. R. (2005). The difficult process of scientific
modelling: An analysis of novices’ reasoning during computer-based modelling. International
Journal of Science Education, 27(14), 1695–1721. doi:10.1080/09500690500206408
Spinelli, T. (1983). The Delphi decision-making process. The Journal of Psychology, 113(1), 73–80.
doi:10.1080/00223980.1983.9923559
Sun, D., Wang, Z. H., Xie, W. T., & Boon, C. C. (2014). Status of integrated science instruction in
junior secondary schools of China: An exploratory study. International Journal of Science
Education, 36(5), 808–838. doi:10.1080/09500693.2013.829254
Szabo, Z., & Schwartz, J. (2011). Learning methods for teacher education: The use of online discus-
sions to improve critical thinking. Technology, Pedagogy and Education, 20(1), 79–94. doi:10.
1080/1475939X.2010.534866
Thomaidis, N. S., Nikitakos, N., & Dounias, G. D. (2006). The evaluation of information technology
projects: A fuzzy multicriteria decision-making approach. International Journal of Information
Technology & Decision Making, 5(01), 89–122.
Wang, W. M., Lee, A. H. I., Peng, L. P., & Wu, Z. L. (2013). An integrated decision making model
for district revitalization and regeneration project selection. Decision Support Systems, 54(2),
1092–1103. doi:10.1016/j.dss.2012.10.035
Wu, H., & Hsieh, C. (2006). Developing sixth graders’ inquiry skills to construct explanations in
inquiry-based learning environments. International Journal of Science Education, 28(11),
1289–1313. doi:10.1080/09500690600621035
Wu, H. Y., Wu, H. S., Chen, I. S., & Chen, H. C. (2014). Exploring the critical influential factors of
creativity for college students: A multiple criteria decision-making approach. Thinking Skills and
Creativity, 11(0), 1–21. doi:10.1016/j.tsc.2013.09.004
Yang, C. Y. T., & Chang, C. H. (2013). Empowering students through digital game authorship:
Enhancing concentration, critical thinking, and academic achievement. Computers &
Education, 68, 334–344. doi.10.1016/j.compedu.2013.05.023
Yeh, Y. C. (2012). A co-creation blended KM model for cultivating critical-thinking skills.
Computers & Education, 59(4), 1317–1327. doi:10.1016/j.compedu.2012.05.017
Zhai, J., Jocz, J. A., & Tan, A. L. (2014). ‘Am I like a scientist?’: Primary children’s images of doing
science in school. International Journal of Science Education, 36(4), 553–576. doi:10.1080/
09500693.2013.791958
Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle
school. Developmental Review, 27(2), 172–223.
Zohar, A. (2013, December). Challenges in wide scale implementation efforts to foster higher order
thinking (HOT) in science education across a whole school system. Thinking Skills and
Creativity, 10, 233–249. doi:10.1016/j.tsc.2013.06.002
Zohar, A., & Schwartzer, N. (2005). Assessing teachers’ pedagogical knowledge in the context of
teaching higher-order thinking. International Journal of Science Education, 27(13), 1595–1620.