See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/329584112

Conceptualizing ELT Research: Research Process in Quantitative and

Qualitative Study

Chapter · August 2018

CITATIONS READS

0 5,329

1 author:

Parlindungan Pardede
Universitas Kristen Indonesia
52 PUBLICATIONS   236 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Scientific Writing View project

Blended Learning for ELT View project

All content following this page was uploaded by Parlindungan Pardede on 12 December 2018.

The user has requested enhancement of the downloaded file.

Instructional Material Updated July 2018

Conceptualizing ELT Research:

Research Process in Quantitative and Qualitative Study

Parlindungan Pardede
[email protected]
English Education Department
Universitas Kristen Indonesia

Abstract
Research is essentially a process involving six steps. However, conducting a research is more
than engaging in those six steps, because the investigator should also design and write the
research in either quantitative or qualitative approach. After the problem identification,
literature review, and problem statement formulation, the investigator should decide whether
the problem best fits the quantitative or qualitative approach. The chosen research design
then will then inform the procedures involved in it, including sampling, data collection
instruments (protocols), the procedures, the data analysis, and the final interpretation of
results. After studying this module, the reader will be able to: (1) Analyze the steps of the
research process in the two research articles selected, and (2) Describe the elements of the
research process steps in the research articles selected.

Keywords: quantitative, qualitative, mixed methods, research process, problem identification,

literature review, data collection.

INTRODUCTION
In Module 1 (Pardede, 2018, p. 3), the research process is explicitly described in six
steps. In spite of this, conducting research is more than engaged in that procedure, because
investigators should also design and write research in one of two research approaches
(quantitative and qualitative). After identifying the research problem, reviewing the literature
to establish the importance of the problem, and formulating the problem statement, they must
decide whether the problem better fits the quantitative or qualitative approach.
The research process is generally similar in both quantitative and qualitative
approaches. However, they are different in terms of methods of data collection, data
processing and analysis procedures and styles for communicating the findings. For example,
if your research problem is more appropriate for a qualitative inquiry mode, interviews and
observations are more likely to be used in data collection. On the other hand, if your research
problem is suitable for a qualitative research, tests or surveys will be a method for collecting
data. When analyzing data in qualitative research, you go through the process of identifying
themes and explaining what you find during interviews or observations rather than subjecting
your data to statistical procedures.
After deciding the research approach (quantitative or qualitative) that fits best with the
problem, the questions, and the literature reviews, the researcher can proceed designing the
procedures involved in the study, including sampling, data collection instruments (protocols),
the procedures, the data analysis, and the final interpretation of results. Creswell (2012, p. 12)
Parlindungan Pardede Research in ELT (Module 2) 8
uses Figure 1 to illustrate this through the flow of the research process.
This is the second module of the Research in English Language Teaching (RELT) class
which is used as the main source of material. The purpose of this module is to show how the
six steps involved in the research process are implemented in quantitative and qualitative
studies. It begins by exploring the characteristics of quantitative and qualitative research
approaches regarding each step of the research process. After that, the discussion focuses
on the nature and type of mixed method approaches, procedures in which the elements of the
qualitative and quantitative research approaches are combined in one study or series of
studies. To strengthen and deepen your understanding of the concepts being discussed, four
real research articles are used as examples, i.e. Shams-Abadi, Ahmadi and Mehrdad (2015),
Tajzad and Ostovar-Namaghi (2014), Nazara (2011). and Pardede (2011). Thus, before you
begin studying the following sections, it is necessary for you to read these articles first. After
learning this module, you will be able to: (1) Explain the difference between quantitative
and qualitative research process and (2) explain the nature of mixed method research, its
type, and process.

Research Process

Research
Problem

Literature
Review

Research
Questions

Quantitative Qualitative
Research Research

Research
Design

Quantitative Design Combined Design Qualitative Design

 Experimental  Mixed Methods  Grounded Theory
 Correlational  Action Research  Ethnography
 Survey  Narrative

Sampling Instruments/ Data Analysis Interpretation

Protocols

Discussions, Conclusion, Limitations, Future Research

Figure 1. Flow of the Research Process through Quantitative and Qualitative Research
(Creswell, 2012, p. 12)

Parlindungan Pardede Research in ELT (Module 2) 9

 What are the characteristics of quantitative and qualitative research at each step in the
research process? Table 1 presents quantitative and qualitative research differences viewing
from the six steps of the research process. However, since these differences are described in
a very wide spectrum, it must be discussed in detail in order to get a comprehensive
understanding. While discussing each characteristic, it is helpful to see it in actual research.
Thus, you need to first examine the work of Shams-Abadi, Ahmadi, and Mehrdad (2015) and
Tajzad and Ostovar-Namaghi (2014). Because former article reports quantitative research
and the latter reports qualitative research, they can be used to illustrate within them good
procedures of research. They will be used as a frame of reference for each step in the
research process for the quantitative and qualitative tracks.

Table 1. Major Differences of Quantitative and Qualitative Research in Research Process

(Creswell, 2012, pp. pp. 13-19)

Aspects Quantitative Qualitative

Research problem Is described through a description of Is explored to develop a detailed
trends or a need for an explanation of understanding of a central
the relationship among variables phenomenon (key concept, idea, or
process being studied)

Literature review Plays a major role in justifying the Plays a minor role in justifying the
research problem and in suggesting research problem, because qualitative
potential purposes and research research relies more on the views of
questions for the study. participants in the study and less on
the direction identified in the
literature

Purpose statements, Must be specific, narrow, Stated in a general and broad way in
research questions, measurable, and observable order to be able to catch the
and hypotheses participants’ experiences.

Data Collected in numeric form from a Collected based on words from a

large number of people using small number of individuals so that
instruments with preset questions and the participants’ views are obtained.
responses.

Analysis trends Comparing groups or relating Analyzed data for description and
variables using statistical analysis, themes using text analysis and
and interpreting results by comparing interpreting the larger meaning of the
them with prior predictions and past findings.
research.

Research report Written in standard, fixed structures Written by using flexible, emerging
and evaluation criteria, and taking an structures and evaluative criteria, and
objective, unbiased approach including the researchers’ subjective
reflexivity and bias

Quantitative Research Characteristics

Quantitative studies are generally based on the positivist or scientific paradigm which
regards the world as composed of observable and measurable facts. As a consequence, the
investigators in quantitative research identify a research problem based on a description of

Parlindungan Pardede Research in ELT (Module 2) 10

trends or a need to explain the relationship among variables. Thus, the identified problem is
something that can be answered best by a study in which the researcher seeks to establish
the overall tendency of responses from individuals and to note how this tendency varies
among people. For example, the researcher might seek to learn how students describe their
attitudes toward the use of songs in pronunciation practice. The results from this study can
inform how a large population of students view the practice and the variety of these views.
Variables, whose relationship an investigator tries to explain, are basically any involved
factors that can be controlled, changed, or measured in a study. A variable is stated using the
relevant attribute. For example, if the variable is a test score, its attribute might be 70, 75, 80,
90, etc. If sex is the variable, the attributes are male and female. By explaining a relation
among variables, the researchers would like to determine whether one or more variables
might influence another variable. For example, quantitative researchers may seek to know
why certain students were not interested to use songs in pronunciation practice. The variables
of gender and attitude toward the use of songs in pronunciation practice may influence
individuals’ engagement in learning activities.
In their research article, Shams-Abadi, Ahmadi and Mehrdad (2015) identified their
research problem based on the growing importance for students to communicate through
writing and the accelerating trend of using technologies in language learning, including in
writing classrooms. The authors are less interested in describing what students and teachers
think of the use of Edmodo (as one of educational technologies). They were more interested
to seek the effect of using Edmodo on students’ writing performance. Thus, the problem being
addressed concerns with the effect of using Edmodo on students’ writing performance--
something that the researchers know very little about. Such research problem is very suitable
to study in the quantitative research.
In quantitative research, the literature review is very important. First, it is required to
justify the need for the research problems. Second, it is essential to suggest potential aims
and research questions for the study (Creswell, 2012, p. 13). Justifying the research problem
means that the researcher uses the literature to document the importance of the problems
examined in the study. To achieve this, researchers searches the literature, locates relevant
studies (i.e., those which identify the problem as important to examine) and then quotes this
literature in the opening section of the research report. Literature review also creates a need
for research, as stated specifically in the statement of purpose and research question or
hypothesis. When studying the literature, key variables, relations, and trends are identified
and used to provide direction for the research questions and the research hypotheses to be
tested.
In the literature review section of their research article, Shams-Abadi, Ahmadi and
Mehrdad (2015) showed that we have known quite much about the teaching of writing in the
classroom. The use of technology in language classrooms has also been widely known. It is
shown that the activities of writing could be effectively facilitated by using Edmodo, an
educational website that takes the ideas of a social network and is appropriate for a classroom.
Current studies and experiences included in the literature review section also indicate that
students and learners had positive attitudes towards the use of technology in learning
languages. They viewed Edmodo as a means of enlivening instruction, improving students’
writing skills, and facilitating collaboration and interaction both within and beyond the
classroom. However, little is known about the effect of using Edmodo towards students writing
performance. Studies about this problem are still rare. That is why this problem is important
Parlindungan Pardede Research in ELT (Module 2) 11
to study. In this way, the literature in a quantitative study both documents the need to study
the problem and provides direction for the research questions.
The research questions in a quantitative research are stated specifically and narrowly
so that researchers can obtain measurable and observable data on variables. The major
statements and questions of direction in a study (the purpose statement, the research
questions, and the hypotheses) should be specific and narrow because they must identify only
a few variables to study. From a study of these variables, measures or assessments on an
instrument or record scores on a scale from observations are obtained. For example, to carry
out a study about the factors affecting students’ reading habit, the factors must be narrowed
to a specific variable. This could achieved by choosing one from many relevant variables (e.g.
students’ family background, presence of library, students’ perception of reading, types of
books assigned by teachers, etc.). If the researchers decide to focus on students’ perception
of reading, they can examine it by obtaining data from the students.
In their research article, Shams-Abadi, Ahmadi and Mehrdad (2015) narrowed the
research problem to the effect of Edmodo on students’ writing performance. At the end of their
introduction section (p. 89), they stated the purpose of the study as “an attempt to investigate
the effect of Edmodo on EFL learners’ writing performance”. Thus, their research questions
are specific to two factors (use of Edmodo and students’ writing performance), and later in the
method section, they explain how they measured these factors.
Quantitative study generally tries to find answers to concrete questions by generating
numbers and facts. To be more specific, a quantitative research is conducted to achieve one
of the following purposes: (1) to discover the characteristics of an observed phenomenon, (2)
to explore correlations between two or more entities, (3) to identify causal relationships among
concepts, or (4) to make predictions. A study conducted to discover the characteristics of an
observed phenomenon is carried out by collecting particular factors of behavior which are
quantified. For example, a researcher wanted to discover the perception of a group of students
of the implementation of blended learning in their reading classroom. To achieve this, he
collected data by asking the students to respond to a questionnaire by choosing one of the
options in Linkert scales ranging from “Strongly Agree”, “Agree”, “Disagree”, and “Strongly
Disagree”. Every statement responded with “Strongly Agree” is given 4 score. A statement
responded with “Agree” got 3; Disagree”, 2; and “Strongly Disagree”, 1. In such a case, the
students’ responses were quantified. When a research is directed to examine whether
students who often read research articles can write good research proposals, he is conducting
a study for the purpose of exploring correlation between the variable of frequent reading of
research articles and the variable of writing good research proposals. Because Shams-Abadi,
Ahmadi and Mehrdad’ (2015) focused their study on investigating the effect of Edmodo on
EFL learners’ writing performance, their research took the fourth purpose above. A researcher
who has collected data about the habits, attitudes, motives, and learning activities of some
successful language learners can use his findings to predict that if other people develop those
successful language learners’ traits, they can be successful too.
To collect quantitative data, an instrument is used to measure the variables in the study.
An instrument is a tool for measuring, observing, or documenting quantitative data. It contains
specific questions and response possibilities which are established or developed in advance
of the study. Survey questionnaires, standardized tests, and checklists used to observe a
student’s or teacher’s behaviors are some common instruments used in ELT quantitative
research. These instruments are administered to participants and collect data in the form of
Parlindungan Pardede Research in ELT (Module 2) 12
numbers. For instance, the researchers might collect responses based on students choices
on a questionnaire, or from checklists the researchers complete as they are watching a
student perform a task in the classroom. The intent of this process is to generalize the results
from a small number of people to a large number. The larger the number of individuals studied,
the stronger the case for applying the results to a large number of people.
In Shams-Abadi, Ahmadi and Mehrdad’ (2015) research, the variable of the use of
Edmodo was measured by giving different treatments to the 20 students in the experimental
group and 20 students in the control group. Both groups learnt paragraph writing using the
same materials however, the experimental group was taught using Edmodo. They uploaded
assignments, received and provided comments, and discussed with classmates in Edmodo.
In contrast, the control group students did not experience the activities in Edmodo. The second
variable, students’ writing performance, was measured through a pretest conducted before
the treatment and a posttest administered after 12 sessions of treatment. To ensure the
validity and reliability of the tests, they were taken from Cambridge IELTS 9 (2013), and the
students writing samples were scored based on IELT’S Holistic Scoring Rubric for Writing.
The results of the pretest showed no significant differences between the groups (p. 93). That
means, both groups had similar writing performance before the treatment. The results of the
pretest of each group were than compared to each posttest results to see the gain scores
(Table 1, p. 93). Mann-Whitney U test, both groups’ gain scores were compared to see the
difference significance. The results showed that the experimental group outperformed the
control group (Table 3, p. 94).
In quantitative research, the data are generally analyzed using mathematical
procedures, called statistics. Statistics is differentiated into two main types: descriptive and
inferential. Descriptive statistics is used to summarize our group of data using a combination
of tabulated description (i.e., tables), graphical description (i.e., graphs and charts) and
statistical commentary (i.e., a discussion of the results). Descriptive statistics also provides
measures of frequency (count, percent, and frequency) which help the researchers to show
how often something occurs; measures of central tendency (mean, median, and mode)
necessitated to locate the distribution by various points; measures of dispersion or variation
(range, variance, standard deviation) which help to identifies the spread of scores by stating
intervals; and measures of position (percentile ranks and quartile ranks) needed to describes
how scores fall in relation to one another.
Many studies, however, necessitate the researchers to draw conclusions such as
making comparisons across time, comparing different groups, or trying to make predictions
based on the collected data. To do this, inferential statistics are needed. Some of the most
common inferential statistics are t¬¬-test, which is used to compare means; ANOVA
(employed to compare multiple means at the same time); and regression, which is used to
make a prediction about an outcome (or criterion) variable based on knowledge of some
predictor variable. Technological advancement has made statistical analysis easy. Therefore,
there is no need for any researcher, including those who conduct quantitative research in ELT,
to worry with statistical analysis. With the help of SPSS (Statistical Package for Social
Sciences), statistical analysis could be effortlessly performed.
Shams-Abadi, Ahmadi and Mehrdad (2015) analyzed the data they obtained (results of
the pretest and posttest) by using SPSS statistical package version 22 (SPSS Inc., 2013) by
running a Mann-Whitney U Test. The result of the analysis, showing that Edmodo created a
significant difference in EFL learners’ writing performance, was then interpreted in the
Parlindungan Pardede Research in ELT (Module 2) 13
discussion section (pp. 94-95) by confronting it to the hypothesis and comparing it to the
results of some prior studies.
The overall format used to report and evaluate a quantitative research follows a
predictable pattern: introduction, review of the literature, methods, results, and discussion.
This form creates a standardized structure for quantitative studies. In addition, it also leads to
specific criteria used to evaluate the quality of a quantitative research report. Using this format,
it is quite easy to evaluate a quantitative study by examining whether it has an extensive
literature review; tests good research questions and hypotheses; uses rigorous, impartial data
collection procedures; applies appropriate statistical procedures; and forms interpretations
that naturally follow from the data. The format also makes it easy to see whether the
instruments employed have proven value and reliable and valid scores from past uses or not.
Looking at the research design section, it is quite easy to check whether all variables were
controlled to prevent bias or not. Another important feature of a quantitative research report is
that it is written without referring to the researchers’ personal reaction.
Shams-Abadi, Ahmadi and Mehrdad (2015) subdivided their research article into
standard sections typically found in quantitative studies. The study begins with an introduction
that includes the purpose statement. What follows is the literature review, the methods
(including participants, materials, and procedures), the results, the discussion; and, finally, the
discussion. Entirely, the article conveys an impersonal, objective tone, and they do not bring
either their biases or their personal opinions into the study. They use proven instruments to
measure variables, and they employ multiple statistical procedures to build objectivity into the
study.
Comparing to the general format of quantitative research article, certain differences are
found in Shams-Abadi, Ahmadi and Mehrdad (2015) article. For instances, the literature
review which commonly integrates to the introduction is separated by Shams-Abadi, Ahmadi
and Mehrdad (2015); the hypothesis is commonly placed at the end of the literature review
because hypotheses are formulated along with literature review, while in this article the
hypothesis is placed in the result section; and the authors did not include the implications and
limitations in their conclusion section.

Qualitative Research Characteristics

In ELT, the qualitative research could be seen as the research where the investigator relies
on the views of the participants. It is best suited to address a phenomenon whose variables
are unknown and need to explore because literature might have little information about the
phenomenon. Therefore, it is necessary to learn more from participants through exploration.
For instance, the literature may not adequately address the practice of online learning in
language skills development in English as a Foreign Language (EFL) setting. A qualitative
research study is needed to explore this phenomenon from the perspective of EFL learners.
Unquestionably, online learning implementation in language skills classes may cause
difficulties to the students and teachers. Such perception may not have been examined in the
prior literature. In a qualitative study, such problem is called the central phenomenon, i.e. the
key concept, idea, or process studied in qualitative research. Thus, the research problem of
online learning implementation in language skills classes in EFL setting requires both an
exploration (because we need to better know how to implement the new learning mode) and
an understanding (because of its complexity) in the process of language skills teaching and
learning.
Parlindungan Pardede Research in ELT (Module 2) 14
 The authors in the sample article on EFL learners’ perceptions of the integrated skills
approach to language teaching (Tajzad and Ostovar-Namaghi, 2014) argue that although the
Audio-lingual method have been replaced by the Communicative Approach in popularity, the
former is still praciced in many EFL contexts. Consequently, although students have learned
language skills and components, they find it difficult to use them to communicate. So, to help
the incompetent learners to communicate, they suggest, practitioners should present
language skills and components in an integrated fashion, which is known as the integrated
skills approach. The authors then present the results of some current studies which show
many advantages of the integrated skills approach implementation.
However, these previous studies collected the data mainly based on the researchers’
preconceived academic knowledge about the phenomena identified in the literature. None of
those studies relied the data collection on the students’ perspectives, whereas the students
are active participants in the learning process. So, the students’ perceptions should also be
considered in order to optimize their participation in the learning activities. If the students’
views are ignored, they might find integrated skills approach implementation uninteresting.
Consequently, effective and enjoyable teaching and learning will not take place and learning
will be up to failure. Tajzad and Ostovar-Namaghi (2014) accentuated that having taken
students’ perception of the integrated skills approach into account, they believed the problem
of communicative incompetence among EFL learners could be alleviated. Thus they directed
the study to present EFL learners with the integrated skills approach, and explore their
perceptions of this approach. More specifically, they aims at evaluating the efficacy of this
approach by answering the general question, “How do you perceive the integrated skills
approach? (p. 93).
If in quantitative research the literature review plays a major role to justify the need for
the research problem and to suggest potential purposes and research questions for the study,
in a qualitative research, it plays a less substantial role at the beginning of the study. Although
the literature review may be used to justify the need to study the research problem in
qualitative research, it does not provide major direction for the research questions because
qualitative research relies more on the views of participants in the study and less on the
direction identified in the literature. Thus, to use the literature to foreshadow or specify the
direction for the study is inconsistent with the qualitative approach of learning from
participants. For example, in her study on how Chinese students perceived peer review
activities and how they viewed the roles of their Chinese identities played in their interaction
and negotiation in a freshmen composition class at a Midwest university in the US, Wang
(2016) cited several studies at the beginning of the research to provide evidence for the
problem. However, the literature was not used to specify the research questions.
In their research article, Tajzad and Ostovar-Namaghi (2014) begin by citing numerous
studies from the literature. This literature review is not to identify specific questions that need
to be answered; instead, it shows that segregated skills approach is still implemented in many
EFL classes despite the current researches’ findings revealing that the integrated skills
approach is the best to use for enabling students to use their language skills to communicate.
They also cited several studies which show the benefits and usefulness of the skills integration
approach.
In qualitative research, the purpose statement and the research questions are stated so
that you can best learn from participants. The investigators research a single phenomenon of
interest and state this phenomenon in a purpose statement. A qualitative study that examines
Parlindungan Pardede Research in ELT (Module 2) 15
the “professionalism” of EFL teachers, for example, asks high school EFL teachers, “What
does it mean to be a professional?” This question focuses on understanding a single idea—
being a professional—and the responses to it will yield qualitative data such as quotations.
In Tajzad and Ostovar-Namaghi’s (2014) study, the authors say that the study emerged
from their concern with the fact that the segregated language skills approach is still
implemented in many in EFL classrooms. Since that approach does not facilitate students to
use the skills they have mastered in real communication. The authors stated the research
purpose, “to present EFL learners with the integrated skills approach, and to explore their
perceptions of this approach”, and raise the research question, “How do you perceive the
integrated skills approach?” (p. 93).
In qualitative research data could be obtained from many sources and be in many forms.
They may be collected through interviews, observation, or content analysis of existing
materials (e.g. documents, videos), and may include interview tapes or transcripts,
observation field notes, notes on interview context and process, analytic notes and memos,
or journal entries. Data, therefore, could be gathered in the forms of text (word), image
(picture), audio and video recordings. The level and extent of analysis of data from the various
sources will depend on the purpose for which the data was collected and involves choices
that need to be made for each project. The most important thing is that the data should as
explicitly as possible describe about what is being done and why. Taylor and Bogdan suggest
that all qualitative research reports should ‘provide enough information about how your
research was conducted to enable readers to discount your account or to understand it in the
context of how it was produced’ (1988, p. 168).
In qualitative research data analysis starts by creating a text database by dividing the
texts into groups of sentences, called text segments, and determining the meaning of each
group of sentences. Rather than using statistics, you analyze words or pictures to describe
the central phenomenon under study. The result may be a description of individual people or
places. In some qualitative studies, the entire report is mostly a long description of several
individuals. The result may also include themes or broad categories that represent your
findings. In qualitative studies in which you both describe individuals and identify themes, a
rich, complex picture emerges. From this complex picture, you make an interpretation of the
meaning of the data by reflecting on how the findings relate to existing research; by stating a
personal reflection about the significance of the lessons learned during the study; or by
drawing out larger, more abstract meanings. In short, Bogdan and Biklen (cited by Hoepfl
1997, p. 54) accentuate that qualitative data analysis is “working with data, organizing it,
breaking it into manageable units, synthesizing it, searching for patterns, discovering what is
important and what is to be learned, and deciding what you will tell others”.
In Tajzad and Ostovar-Namaghi’s (2014) study, data was obtained from a group of 30
participants who had been introduced and experienced the integrated skills approach
implementation. To collect the data, those who were willing to share their views concerning
this approach and interviewed. The obtained data was then analyzed through “coding”
techniques, including open coding, axial coding, and selective coding. The open coding, or
line-by-line analysis, was employed to identify provisional explanatory concepts. In the axial
coding, the researchers tried to elaborate the concepts and categories by specifying their
dimensions and properties and then fining the interrelationships between concepts and
categories. Finally, in the selective coding the researchers found the core category, i.e.,
learners’ perceptions of the integrated approach. Credibility of the findings was established
Parlindungan Pardede Research in ELT (Module 2) 16
through: (1) constant comparison, i.e., the provisional concepts and categories were
constantly compared with transcripts in order to ensure consistency; and (2) member
checking, i.e., provisional categories as well as the final conceptualization were shown to the
participants and modifications were made for the best fit. According to the original rules of
grounded theory, the researchers did not leave the field and stop sampling until theoretical
saturation was reached (p. 94).
There is a wide range of formats employed to report qualitative research. These formats
certainly follow the standard steps in the process of research, but the sequence of these
“parts” of research tends to vary from one qualitative report to another. A study may begin
with a long, personal narrative told in story form or with a more objective, scientific report that
resembles quantitative research. With such variability, it is not surprising that the standards
for evaluating qualitative research also are flexible. Good qualitative reports, however, need
to be realistic and persuasive to convince the reader that the study is an accurate and credible
account. Qualitative reports typically contain extensive data collection to convey the
complexity of the phenomenon or process. The data analysis reflects description and themes
as well as the interrelation of themes. In addition, the reports also discuss the researchers’
role or position in the study, called being reflexive. This means that the researchers reflect on
their own biases, values, and assumptions and actively write them into the research. This may
also involve discussing personal experiences and identifying how the researchers
collaborated with participants during phases of the project. They may also discuss how their
experiences and cultural backgrounds (e.g., Batak perspectives) affect the interpretations and
conclusions drawn in the study.
Although Tajzad and Ostovar-Namaghi’s (2014) study belongs to the qualitative
research, they used scientific structure, not a literary structure, for reporting their article. This
may have been done because of the requirements of the journal to address certain aspects
(e.g., methods, results, discussion). The authors also never employed the personal pronoun
“we” in referring to themselves in the study. They used “the researchers” instead. They might
have done this to emphasize objectivity. However, it prevents the subjective orientation
typically associated with qualitative, literary writing.

Quantitative and Qualitative Research Similarities and Differences

Based on the discussion above, it’s obvious that both forms quantitative and qualitative
research follow the six steps in the process of research. However, there are some minor
differences, in the introduction, especially in the research problem section. In quantitative
research the research problem section is used to direct the types of questions or hypotheses
asked in the study, whereas in qualitative research the research problem discussion is
typically used to establish the importance of the central phenomenon. These differences are
apparent in the comparison of the introduction section of the two sample articles a frame of
reference for each step in the research process above. Shams-Abadi, Ahmadi and Mehrdad
(2015) identified their research problem based on growing importance for students to be able
to communicate through writing and the accelerating trend of using technologies in language
learning, including in writing classrooms. On the other hand, Tajzad and Ostovar-Namaghi
(2014) began by showing the importance of the central phenomenon, i.e. the application of
the integrated skills approach to enable EFL learners to use the language skills and
components they have learnt to communicate.
Both quantitative and qualitative research are similar in data collection, because both
Parlindungan Pardede Research in ELT (Module 2) 17
may employ similar approaches, such as interviews or observations. However, since
quantitative approaches should gather quantified information, it uses more closed-ended
approaches in which the researchers identify set response categories (e.g., strongly agree,
strongly disagree, and so forth), whereas qualitative approaches use more open-ended
approaches in which the inquirer asks general questions of participants, and the participants
shape the response possibilities (e.g., in an interview with an English teacher, a qualitative
researcher might ask: What does a professional English teacher mean to you?).
Another difference exists in data analysis procedures. Quantitative research investigator
relies on statistical analysis (mathematical analysis) of the data, which is typically in numeric
form. In qualitative research, statistics are not used to analyze the data; instead, the inquirer
analyzes words (e.g., transcriptions from interviews or notes of observations) or images (e.g.,
photographs) or videos. Rather than relying on statistical procedures, the qualitative
researcher analyzes the words to group them into larger meanings of understanding, such as
codes, categories, or themes.
The reporting formats are also typically different, with the quantitative structure following
the typical introduction, literature review, methods, results, and conclusion sections. In
qualitative research, some of these sections, literature review, for instance, may be missing,
and the format may be more of a literary opening with a personal passage, an unfolding story,
the use of extensive quotes from participants, and personal reflections from the researcher.
Despite some differences existed in quantitative and qualitative research, both
approaches should essentially viewed as different points on a continuum rather than as two
end points in a dichotomy. Studies may contain some elements of the characteristics of
quantitative research and some elements of qualitative research. However, studies actually
tend to lean toward one approach or the other, and knowing the characteristics associated
with each type of research enables a researcher to assess whether a particular study favors
either quantitative or qualitative research.

Mixed Methods Approach

Many of current investigators have combined the qualitative and quantitative
approaches in many phases of their research process. That mixture emerged what is called
as the mixed methods approach. Creswell and Clark (2007, p. 5) state, that as a research
design, mixed method is “a procedure for collecting, analyzing, and “mixing” both quantitative
and qualitative methods in a single study or a series of studies to understand a research
problem”. This is in line with Johnson, et.al. (2007, p. 123) who define mixed methods as “…
the type of research in which a researcher or team of researchers combine elements of
qualitative and quantitative research approaches (e.g., use of qualitative and quantitative
viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth
and depth of understanding and corroboration.” Although mixed method is accepted as a new
research methodology, many investigators have collected quantitative and qualitative data for
many years.
The goal of combining qualitative and quantitative research components in mixed
methods is to expand and strengthen a study’s conclusions. Hurmerinta-Peltomakl and
Nummeia (2006) argued that the mixed method approach is necessary because using one
method alone would only provide a small view of the whole picture when studying complex
issues. Mixed methods provide information on different levels of understanding. When
methods are combined, qualitative methods may provide in-depth understanding of the
Parlindungan Pardede Research in ELT (Module 2) 18
variables that lead to quantitative numerical findings. Mixed methods may also be used for
triangulation or to improve the validity of research. In a more detailed fashion, Creswell (2012,
p. 535) provide three reasons for conducting the mixed methods design. First, a mixed
methods is used when the researchers have both quantitative and qualitative data and both
types of data, together, provide a better understanding of the research problem than either
type by itself. Mixed methods research is a good design to use if the researchers seek to build
on the strengths of both quantitative and qualitative data. Quantitative data, such as scores
can be statistically analyzed to produce the frequency and magnitude of trends. Such results
can provide useful information if trends about a large number of participants is needed to
describe. However, qualitative data, such as open-ended interviews that provide actual words
of people in the study, offer many different perspectives on the study topic and provide a
complex picture of the situation. When quantitative and qualitative data is combined, more
comprehensive understanding could be obtained.
The second reason for using a mixed methods study is when one type of research
(qualitative or quantitative) is not enough to address the research problem or answer the
research questions. More data is needed to extend, elaborate on, or explain the first database.
For example, a researcher may want to first explore the data qualitatively to develop an
instrument or to identify variables to test in a later quantitative study. One engage in a mixed
methods study when he wants to follow up a quantitative study with a qualitative one to obtain
more detailed, specific information than can be gained from the results of statistical tests.
Finally, mixed methods is conducted when a researcher wants to provide an alternative
perspective in a study. An example of this would be an experimental study in which the
experiment yields useful information about outcomes, but the additional collection of
qualitative data develops a more in-depth understanding of how the experimental intervention
actually worked. Another example would be when a policymaker wants both the “numbers”
and the “stories” about an issue. These different sources of information provide both a
condensed understanding of a problem as well as the detail. On a practical level, you use
mixed methods research for studies in graduate programs in which qualitative research has
yet to be fully accepted and in which quantitative approaches are the norm.
Mixed methods approach refers to the research that mixes or integrates both qualitative
and quantitative approach within a single study to produce a fuller account of the research
problem, the integration can occur at any stage(s) of the research process (Glogowska, 2011).
The six steps of research process we have discussed so far, can be reasonably used to
conduct and to evaluate a mixed methods research, because the features of quantitative and
qualitative approach could be integrated in one or more of the steps. Schoonenboom and
Johnson (2017) highlighted that “Each true mixed methods study has at least one “point of
integration” … at which the qualitative and quantitative components are brought together.
Having one or more points of integration is the distinguishing feature of a design based on
multiple components. It is at this point that the components are “mixed”, hence the label “mixed
methods designs”.
However, it is not necessary to make the integration of both features in equal portion. A
mixed method study could be qualitative dominant [or qualitatively driven] in which the
researcher relies on a qualitative view of the research process, and quantitative data and
approaches are added as assisting elements to strengthen the study’s conclusion. A mixed
method study could also be quantitative dominant [or quantitatively driven] in which the
researcher relies on a quantitative research process, and quantitative data and approaches
Parlindungan Pardede Research in ELT (Module 2) 19
are added to support the study for getting more robust conclusion (Johnson et al., 2007, p.
124).
Ponce and Pagán-Maldonado (2015) suggest five possible points in the research
process to integrate or combine quantitative and qualitative approaches. First, in the planning
phase where the research plan is developed, to make what is investigated and how
quantitative and qualitative approaches are used clear. Second, combining or integrating
research questions from quantitative and qualitative approaches to guide the researcher into
the complexity of the problem studied. Third, using quantitative measurement instruments with
qualitative research techniques to generate quantitative and qualitative data for the research
problem. Fourth, combining or integrating quantitative and qualitative data in the analysis of
study data. Fifth, combining or integrating quantitative and qualitative data in the presentation
of the study findings.
While planning a mixed methods research, it is important to consider what design, which
serves as a guiding practice, one is going to use. Creswell and Clark’s (2011) proposed six
“major mixed methods designs” to choose. The first, convergent (or parallel or concurrent)
mixed methods design is aims at collecting both quantitative and qualitative data in one phase.
The data is analyzed separately and then compared and/or combined. For example, a
researcher collected survey data and interview data at the same time and compared the
results. This method is used to confirm, cross-validate or corroborate findings. It is often used
to overcome a weakness in one method with the strengths of another. It can also be useful in
expanding quantitative data through collection of open-ended qualitative data.

Quantitative data
collection &
Analysis
Followed Interpret-
up with ation
Qualitative data
collection &
Analysis

Figure 5. Convergen/Parallel Mixed Methods Design (Creswell, 2012)

The second, explanatory sequential mixed methods design (also called a two-phase
model) consists of first collecting quantitative data and then collecting qualitative data to help
explain or elaborate on the quantitative results. For example, a survey may be used to collect
quantitative data from a larger group. Members of that group may then later be selected for
interviews where they can explain and offer insights into their survey answers. For example,
say that the English department in a university decides to study the levels of student’s
satisfaction with the internet connection. A survey with questionnaires to students is performed
(phase I). This survey asks how satisfied residents are with the internet connection, using a
Linkert scale of 1 to 4. After the structured quantitative survey the study continues with a
qualitative interview for trying to understand the reasons for the initial response (phase II).
Each resident is asked to explain or qualify his answer. This allows generation of a list of
strengths and weaknesses of the internet connection as perceived by the students.
Pardede’s (2011) study which investigated the interest and perceptions of the teachers
training students majoring in English on the inclusion of short story in language skills
classrooms, is an example of the explanatory sequential mixed methods design. The

Parlindungan Pardede Research in ELT (Module 2) 20

researcher began the study by asking 45 students to fill in a questionnaire to obtain their
perceptions and attitudes on the inclusion of short story in their language skills classes. Then,
to understand the respondents‟ point of view about the phenomenon, focused semi structured
open-ended interviews were conducted involving five volunteers out of the 45 participants.

Quantitative data Qualitative data Interpret-

collection & Followed
up with collection & ation
Analysis Analysis

Figure 6. Explanatory Sequential Mixed Methods Design (Creswell, 2012)

The third, exploratory sequential mixed methods design is also a two phase design. The
objective of this design is the exploration of the research problem. Exploration is used when
very little is known about the research problem. The qualitative data is collected first to explore
the experience of participants with the phenomenon under study (e.g., the culture or values
of the group, or the structure of the institution being studied. After analyzing the qualitative
data, the researcher now has better understanding of the research problem. Having that
understanding, he can design a quantitative study (phase II) to define or measure the findings
of the qualitative phase (phase I) in a sample of the universe under study. For example, say
that the principle of a school wants to change all computers used by more than 100 teachers
in the school. As he does not know the teachers’ needs and interests, the principle starts a
study by interviewing 10 teachers randomly selected (phase I). Based on the interview results,
he designs a questionnaire (quantitative phase II) to be administered to all teachers.

Qualitative data Quantitative data Interpret-

collection & Builds to collection & ation
Analysis Analysis

Figure 7. Explanatory Sequential Mixed Methods Design (Creswell, 2012)

Fourth, the embedded design, aims at collecting quantitative and qualitative data
simultaneously or sequentially in which priority is given to one approach that guides the
project, while the other approach is embedded or nested into the project and provides a
supporting role. The embedded approach is often addressing a different question then the
primary research question. For example, a researcher conducted an experimental study to
test the effectiveness of using English movies technique to develop students’ speaking
performance. The experiment did reveal that the technique was significantly effective.
However, it did not provide an explanation of how the technique worked and how the students
viewed it. To investigate them, a qualitative research approach was carried out by interviewing
the students. In this case, the experiment is the primary method of research and the qualitative
approach is the complementary method because it is used to compensate for the
methodological deficiencies of the experiment.

Parlindungan Pardede Research in ELT (Module 2) 21

 Qualitative (or Quantitative)
data collection & analysis

Qualitative (or Quantitative) Interpretation

data collection & analysis
(before, during, or after)

Figure 8. Embedded Mixed Methods Design (Creswell, 2012)

Fifth, the transformative mixed methods design, aims at using one of the four designs
(convergent, explanatory, exploratory, or embedded) but encasing the design within a
transformative framework or lens.

Quantitative Qualitative data Interpret-

data collection & Followed
up with collection & ation
Analysis Analysis

Figure 9. Transformative Mixed Methods Design (Creswell, 2012)

Sixth, the multiphase mixed methods designs, is used when researchers or a team of
researchers examine a problem or topic through a series of phases or separate studies. This
design requires different research approaches and different groups or samples. To illustrate,
the chief of an English department decides that blended learning will be implemented in all
classes. Every class will run in 40-60% face-to-face and 40-60% online sessions. To assess
the extent of this decision prior to implementation, the measure should be understood from
different perspectives of what it will entail for faculty, students and parents. To study a problem
like this, it is necessary to use a multilevel design or a study that uses various quantitative and
qualitative approaches using different population groups or samples, as part of the research
design.

Overall Study 1: Study 1: Study 3: Mixed

Program informs Quantitative informs
Quantitative Methods
Objective

Figure 10: Multiphase Mixed Methods Designs (Creswell, 2012)

To see how the research process is implemented in a mixed methods research, the
following analysis of research process in Nazara’s (2011) Research article might be helpful.
To get the optimum benefits, you are suggested to study the research article first. After that
examine what the analysis reveals about the six steps of research in the article.

Parlindungan Pardede Research in ELT (Module 2) 22

 Johan Franklin Pardede
1512152007

An Analysis of Research Process in

Nazara’s (2011) Students’ Perception on EFL Speaking Skill Development

Introduction
Students’ Perception on EFL Speaking Skill Development is a mixed methods carried out to
investigate the perceptions of the English Teaching Study Program students of their English
speaking skill development. It employed the explanatory mixed methods design to enable the
researcher to gather qualitative input to explain and extend quantitative results in order to gain
a comprehensive insight concerning students’ perception towards English speaking
development.
This paper discusses the process taken up by the researcher in conducting this study. In
light of the six steps of research process (Cresswell, 2012), the activities carried out Nazara in
this study could be described as follows.

Discussion
A. Identifying Research Problem
In the Introduction section of his article, Nazara identified the research problem based on the
trend of the growing importance for students to master English speaking but researches that
investigated the students’ perception are still rare. Realizing this, it is obvious that the author
employed the quantitative approach in identifying his research problem.
The author began his problem identification by emphasizing the high importance of
mastering English speaking due to the strengthening position of English as the main
international language. He then related this to that fact, the best instructional methods,
materials, activities, media, and other requirements that facilitate the learners to master
English speaking skill is highly necessary. Unfortunately, current studies on the teaching and
learning of speaking mainly dealt with the linguistic aspects of second language acquisition.
Researches concerning students’ perspectives on the learning of speaking in the EFL
classroom are very rare, whereas students’ view is an important aspect to be considered.
That’s why he was interested to investigate students’ perceptions on their English speaking
skill development (pp. 29-30).

B. Reviewing the Literature

In his literature review, Nazara also employed the quantitative approach. Citing many
relevant studies, he used the literature review to justify the need for the research problem and
to suggest potential purposes and research questions for the study. The need for the research
problem was justified by arguing that speaking is very complex. So, to effectively tech it
requires good teaching approaches, facilities, opportunities for students to practice, and some
other aspects. One of the most important aspects to consider is students’ perception of the
learning of speaking, because how the teachers work with the students, how they motivate
speech activities, and the teacher’s ability to relate them to students’ personal interests and
needs, are vital factors for the improvement of speaking. He created need to conduct the study
by emphasizing that teacher;s awareness of students’ belief in speaking development will
surely beneficial in designing the strategies and activities for the promotion of their speaking
skill. (pp. 30-33).
(continued to the next page)

Parlindungan Pardede Research in ELT (Module 2) 23

 C. Specifying the Purpose
Nazara narrowed his research purpose by specifying the participants and the place of research.
At the end of Literature Review he stated “The present study was conducted to investigate the
perceptions of the students of the English Teaching Study Program of FKIP-UKI Jakarta on
their English speaking skill development” (p. 33). Narrowing and specifying the research
problem is typical of the quantitative research. Thus, Nazara employed the quantitative
approach in this step.

D. Collecting Data
Nazara began his Method section by stating that his study employed the explanatory mixed
method design which enables him to gather qualitative input to explain and extend quantitative
results in order to gain a comprehensive insight of the research. In relation to this, he described
the questioner used in phase 1 to collect quantitative data concerning the students’ perceptions
of EFL Speaking development. He then continued with the description of a focused semi
structured open-ended interview guide used in phase 2 to collect qualitative data which would
be used to get insights to understand their responses to the survey more comprehensively (p.
34). Thus, Nazara employed the mixed methods approach in the data collection step.
E. Analyzing and Interpreting Data
The data in this study was analyzed by administering descriptive statistics (for quantitative data)
and the descriptive analysis technique (for qualitative data). Nazara organized the quantitative
data in five tables, and every major finding in each table is is supported and clarified by means
of relevant qualitative data (pp. 35-40).
Based on the analysis, Nazara then drew some conclusions, i.e. (1) The majority of the
students thought speaking is very important to master and they were willing to deal with
necessary activities to master it; Most of the respondents thought speaking learning materials
and activities they got were interesting, however the time provided was too limited and the
learning facilities need to be improved; they insisted on having more intensive use of English
in non-speaking classes; many of them believed that providing a friendly and conducive
environment in the classroom should be made a priority (pp. 40-41).
F. Reporting and Evaluating the Research
Overall, the format used by Nazara to report this research follows the standardized structure for
quantitative studies. The research article was divided into introduction, review of the literature,
methods, results, and discussion sections.
Conclusion
Based on the discussion above, it can be safely concluded seeing from its process, Nazara’s
“Students’ Perception on EFL Speaking Skill Development” was properly conducted by employing
the explanatory mixed method design. This study was qualitative dominant [or qualitatively driven)
because the quantitative research approach was employed in four steps: Research Problem
Identification, Literature Review, Research Purpose Specification, and Research Report and
Evaluation. Mixed methods approach was employed only in Data collection and Data Analysis
steps.

References
Nazara, S. (2011). Students’ Perception on EFL Speaking Skill Development. Journal of English
Teaching, 1(1), pp. 28-42. https://media.neliti.com/media/publications/256868-students-
perception-on-efl-speaking-skil-e0384502.pdf

Parlindungan Pardede Research in ELT (Module 2) 24

Factors to Consider in Choosing Research Approach
To choose the most suitable research approach (a quantitative, a qualitative, or a mixed
methods) for your study, you should consider three factors. First, suit your approach to your
research problem. Note that the problems best suited for quantitative research are those in
which trends or explanations need to be made. For qualitative research, the problems need
to be explored to obtain a deep understanding. Second, your approach should fit the
audience(s) for the research report. Undergraduate final project (or thesis) is written for the
graduate committees, journal editors and review boards and individuals in schools or
educational settings. It is important that the audience(s) be familiar with the approach used in
a study. Third, relate your approach to your personal experience and training. A quantitative
researcher typically has taken some courses or training in measurement, statistics, and
quantitative data collection (e.g. experiments, correlational designs, or survey techniques).
Qualitative researchers need experience in field studies in which they practice gathering
information in a setting and learning the skills of observing or interviewing individuals.
Coursework or experience in analyzing text data is helpful, as well as in research designs
such as grounded theory, ethnography, or narrative research. Some individuals have
experience and training in approaches to research that combine both quantitative and
qualitative methods, such as mixed methods research or action research.

Quiz 2
To every item in this quiz, choose the most suitable option. To every correct
answer for items 1-10, you get 5 points. To every correct answer for items 11-
14, you get 12.5 points.

To take Quiz 2 click: https://www.edmodo.com/home#/quiz/grade/quiz_run_id/15567791

Follow-up Activity 2
This activity is meant to solidify and depend your understanding of the
topic you have just studied. It also aims to facilitate you to apply the
new knowledge you have just taken. So, try your best to finish this
activity.

To access Follow-up Activity 2, click this link:

https://www.edmodo.com/home#/assignment?mid=675478027

Mixed Method Design in YouTube.

To watch the mixed method design in YouTube, click this link:

https://youtu.be/A7YYA9qQjJA

Parlindungan Pardede Research in ELT (Module 2) 25

CONCLUSION
Although quantitative and qualitative characteristics need to be seen as points on a continuum
rather than opposites, the choice of research between the two is based on matching the
approach to a research problem, fitting the approach to your audience, and relating the
approach to your experiences. The major characteristics of the quantitative researches are:
(1) they describe a research problem through a description of trends or a need for an
explanation of the relationship among variables; (2) the literature plays a major role in
suggesting the research questions to be asked, justifying the research problem and creating
a need for the direction (purpose statement and research questions or hypotheses); (3) they
state the purpose, research questions, and hypotheses in a specific, narrow, measurable, and
observable way; (4) they collect numeric data from a large number of people using instruments
with preset questions and responses; (5) they analyze trends, compare groups, or relate
variables using statistical analysis, and interpret results by comparing them with prior
predictions and past research; (6) the report is written using standard, fixed structures and
evaluation criteria, and takes an objective, unbiased approach. In the qualitative researches,
we see different major characteristics at each stage of the research process, i.e.: (1) they
explore a problem and develop a detailed understanding of a central phenomenon; (2) the
literature review plays a minor role but justifies the problem; (3) the purpose and research
questions are stated in a general and broad way so as to the participants’ experiences; (4)
data were collected in words from a small number of individuals so that the participants’ views
are obtained; (5) data are analyzed for description and themes using text analysis, and the
findings are interpreted in the larger meaning; and (6) the report is written by using flexible
and emerging structures, employing evaluative criteria, and including the researchers’
subjective reflexivity and bias.

REFERENCES
Creswell, J.W. (2012). Educational research: Planning, conducting, and evaluating
quantitative and qualitative research. (4th Ed.). Boston, MA: Pearson.
Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research.
Thousand Oaks, CA: Sage Publications.
Glogowska, M. (2011) Paradigms, pragmatism and possibilities: Mixed-methods research in
speech and language therapy. International Journal of Language & Communication
Disorders, 46, pp. 251-260.
Hoepfl, M. (1997).Choosing Qualitative Research: A Primer for Technology Education
Researchers. Journal of Technology Education, 9(1), pp. 47-63.
Hurmerinta-Peltomaki, L. and Nummela, N. (2006). Mixed methods in international business
research: A value-added perspective. Management International Review, 46(4), pp.
439-59.
Johnson, R.B., Onwuegbuzie, A.J., Turner, L.A. (2007). Toward a definition of mixed methods

Parlindungan Pardede Research in ELT (Module 2) 26

 research. Journal of Mixed Methods Research, 1, pp. 112-133. doi:
10.1177/1558689806298224
Nazara, S. (2011). Students’ Perception on EFL Speaking Skill Development. Journal of
English Teaching, 1(1), pp. 28-42.
Pardede, P. (2018). Conceptualizing ELT Research: Research as a Journey. Available online
at: https://www.researchgate.net/publication/329092492_Conceptualizing_ELT_
Research_ Research_as_a_Journey
Pardede, P. (2011). Short Stories Use in Language Skills Classes: Students’ Interest and
Perception. In Zacharias, N.T. & Manara, C. (Eds.) (2011). Bringing Literature and
Linguistics into EFL Classrooms: Insights from Research and Classroom Practice.
Cambridge: Cambridge Scholars Publishing, pp. 101-108.
Ponce, O. A., & Pagan-Maldonado. (2015). Mixed methods research in education: Capturing
the complexity of the profession. International Journal of Educational Excellence, 1(1),
pp. 111-135.
Schoonenboom, J., & Johnson, R. (2017). How to Construct a Mixed Methods Research
Design. Cologne Journal for Sociology and Social Psychology, 25.
Shams-Abadi, B.B. Ahmadi, S.D. & Mehrdad, A.G. (2015). The Effect of Edmodo on EFL
Learners’ Writing Performance. International Journal of Educational Investigations,
2(2), pp. 88-97.
Tajzad, M. & Ostovar-Namaghi, S.A. (2014). Exploring EFL Learners’ Perceptions of
Integrated Skills Approach: A Grounded Theory. English Language Teaching, 7(11), pp.
92-98.

Parlindungan Pardede Research in ELT (Module 2) 27

View publication stats