International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056

Volume: 04 Issue: 07 | July -2017 www.irjet.net p-ISSN: 2395-0072

Quiz in MOOC: An Overview

Jyoti Chauhan1
1Department of Computer Science, University of Delhi, New Delhi, India

---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - Quiz is a one of the primary elements of Massive Generally, a MOOC course is structured in several learning
Open Online Course (MOOC) used for evaluating the units, where each unit having content in form of lecture
knowledge of the learner. A quiz provides feedback as video, text books, tutorial notes, blog post, and many more.
correct/incorrect response or grades that help the learner in The content is followed by the assessment activities that are
designing their own learning paths. In this paper, an overview delivered as quiz, assignment, or exam.
of the MOOC quiz is provided by covering various aspects of Quiz is one of the key elements of assessment in MOOC.
quiz including the users of quiz and types of quiz, basis for Several studies show that the inclusion of quiz has positive
selection of quiz type, and different assessment methods used impact over certain learning related factors, such as,
in MOOC. Also, there are several tools and service that helps in attentiveness of the learners [3] and learning outcomes [4].
offering quiz in MOOC. Here, some of the most common tools Quiz in MOOC is delivered mainly to perform two functions
and services are explained with the suggestions for future evaluation of the overall learning outcome and performance
direction in improving the quiz services in MOOC. of the learner; and, to provide feedback for learners
responses. Quiz responses by the learner are evaluated using
Key Words: MOOC, Quiz, Quiz Types, Assessment Method, the assessment method such as, self, peer, or automated
Automatic Assessment, Authoring Tool, QTI assessment supported by a platform. Though each quiz
evaluation method has its pros and cons, an automated
1. INTRODUCTION assessment with grading is commonly used in MOOC.
However, learner prefers mix of all assessment methods due
The fusion of education with technology is considered as to variety of activity they are used for.
most promising development in learning space. Online
In this paper, the focus is on providing an insight of quiz
learning uses technology for delivering the education. With
used in MOOC. Firstly, an introduction of quiz in MOOC is
technology globalization, the concept and methodology of
given with a brief of MOOC quiz, its use, benefits, and how a
learning, and teaching has undergone a tremendous change.
learner uses it. The author explains various aspects of quiz in
Technological usage in education provides the global
context of MOOC. It includes the discussion of users of quiz,
learning environment that allows accessing the course
types of quiz used, and services required for quiz, in MOOC.
material anytime, anywhere, connect other learners, and get
Author has classified the type of quiz used in MOOC in
access to the content without considering any geographical
several categories, depending on different parameters. For
boundary. The significant changes in use of technological in
example, based on the purpose of quiz to be used for, quiz is
online education have contributed to the emergence of the
categorized as- a) Performance based Quiz and b) Practice
concept of Massive Open Online Course (MOOC).
based Quiz; based on the integration mechanism used quiz is
Nowadays, MOOC is the popular way to offer online courses, of two types- a) Independent Quiz and b) Embedded/ In-Video
and used globally by educational organization or institutions. Quiz; and depending on the grading mechanism used quiz
MOOC uses web-based tools and environments to deliver can be - a) Graded Quiz and b) Non-Graded Quiz. Also, there
education [1]. It provides online courses for unlimited are some specific tools or services required in order to
(logically) participation. It has gained lot of popularity since support the quiz in MOOC. In order to cover this dimension,
the time of its development in 2008. As of December 2016, some of the commonly used tool and services are explained
approximately 58 million students are registered for the with a brief introduction, their use, current state in relation
MOOC courses, offered by more than 700 universities and with the MOOC platform with the suggestions for their
approximately 6850 courses [2]. MOOC courses are offered improvement.
by various providers. Some of the popular MOOC providers
In this paper, Section 2 provides an overview of quiz in
are Coursera1, edX2, Udacity3, iversity4, FutureLearn5,
MOOC. Section 3 explains the users of the MOOC quiz.
NovoEd6.
Various types of quizzes used in MOOC are discussed in
Section 4. Section 5 describes the quiz support system
1 www.coursera.org provided in the form of tools or services in MOOC. Section 6
2 www.edx.org states the conclusion.
3 www.udacity.com
4 https://iversity.org/
5 https://www.futurelearn.com/
6 https://novoed.com/

2017, IRJET | Impact Factor value: 5.181 | ISO 9001:2008 Certified Journal | Page 303
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056
Volume: 04 Issue: 07 | July -2017 www.irjet.net p-ISSN: 2395-0072

2. QUIZ IN MOOC [8]. Embedded quizzes are predominantly important in

MOOC as learners interact mainly with videos.
MOOC delivers well-structured courses. The courses are
divided into a list of lecture units, where each unit comprises 3. USERS OF QUIZ
of the lecture content provided in different forms (video, pdf,
ppt, etc.) followed by various assessment activities including The MOOC courses along with course content and various
quiz, assignment, homework, walkthroughs, lab work, final learning activities are delivered via a MOOC platform, a
project and exam. Primarily, MOOC uses the video lecture specially designed software package offered by the provider.
and quizzes as the instructional methods [5] for their course. Primarily, the users either use the MOOC platform to deliver
Moreover, quiz is the basic method used for assessment by a course or to learn from the course. Here, we focus on the
most of the MOOC providers. users who use the features offered by the MOOC providers.
Various studies [6][7][8] in MOOC considered the quiz as In MOOC, there are several kinds of users who interact with
basic contributing course activity that helps in appraising a quiz. For example, learner uses the quiz for the learning
number of factors such as, drop-out rate [6], engagement [7], purpose. We have identified two main users of the quiz in
usefulness, and many other. Therefore, consideration of quiz MOOC, which are as follows-
as a useful metric clearly shows its significance in MOOC.
Moreover, the magnitude of quiz forced providers to work Instructor is a person who is responsible for
towards improvement of quiz in different ways. For example, designing and delivery of the course to their learners.
providing more interactive and better responsive question Instructor is also liable to control the tools, features,
types, support for better assessment methods, delivering and activities in a course. So, instructor is sometimes
mechanism etc. However, there are some characteristics and called as administrator of the course. The instructor
properties commonly found in MOOC quizzes. manages users specific to the course, defines various
settings and parameters of the tools, assessing
Quiz comprises of set of questions of several types like, MCQ, learners. For example, while designing a quiz in
True/False, descriptive, numerical, image based, activity MOOC, instructor defines the type of questions,
based that are delivered with the objective of evaluating question itself, grading policy, assessment method,
learner. The quiz is evaluated using an assessment method and other required parameters for the quiz. Instructor
such as, self-assessment, peer assessment, automated also plays a role of a passive coordinator whose task is
assessment, based on the method accepted by the course to monitor some of the learners activities.
designer (Instructor). Commonly, the mix of different
assessment method is recommended to be used. Learner is a person who simply uses the features
offered as lesson units, activities, tools etc., by the
A learner participates in MOOC quiz usually after completing instructor or course provider. In quiz sometimes,
some of the course lessons like, watching videos, completing learners are allowed to assess their peers, provide
readings. A learner can also choose to attempt quizzes even feedbacks on their assessments, participate in
before having any course lessons (in case learner have prior discussion over specific question or quiz. Learner in
knowledge) by skipping certain lessons and jump directly to quiz can be a single person or a group of learners
quiz, supporting the self-paced learning. Learner can also based on the type of quiz defined by the instructor.
skip some of the quizzes itself. In response to the questions
asked in the quiz the learner register their reply by giving 4. TYPES OF QUIZ IN MOOC
answers or responses. These responses or answers will be
evaluated on the basis of assessment method (self, peer, or
MOOC supports various types of quiz and each one of them
automated) and grading policy used by the course provider.
serves a different purpose. Depending on different basis
Considering a simple case, the answers given will be
used for quiz, it can be divided into several categories.
evaluated online. System records the responses of learner
and compared with the answers contained in the database of Generally, quiz is used basically used for two reasons; first is
the MOOC. In some quizzes, learner needs to complete the for evaluating performance of a learner, secondly for
quiz within a time frame, decided by the course providers. practice purpose to facilitate learners to check their level of
understanding from the MOOC course. So, based on the use
Sometimes, to enhance the interactivity in the course, quiz is
the quiz, it can be divided into two types.
embedded inside video lecture that are titled as embedded
or in-video quiz. It allows the learners to test their level of Performance based quiz helps in evaluating the
understanding even while watching a video lecture. Learner performance of the learner in the form of score or
can rewind or go back to a particular video frame for marks as the learning outcomes.
reinforcement of learning. Inclusion of in-video quiz Practice based quiz used for practice purpose
provides several benefits from increase interaction level of mainly to provide instant feedbacks to the learner
learner with their course content to the long retention rate for self evaluation, without worrying about the
effect of their score on final outcome.
2017, IRJET | Impact Factor value: 5.181 | ISO 9001:2008 Certified Journal | Page 304
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056
Volume: 04 Issue: 07 | July -2017 www.irjet.net p-ISSN: 2395-0072

Moreover, in order to serve for these two different reasons, learners to check their current learning and
nature of quiz as well as integration mechanism may also performance status after learning from the course.
vary. A quiz can be incorporated in MOOC courses either as
an independent course activity that needs to be completed to
attain their learning objectives or incorporated within the
video lecture [9]. Therefore, based on the integration
mechanism used quiz can be of two kinds.
Independent Quizzes are the self sufficient course
activity consists of a set of questions of numerous
types, supporting wide range of question types. Fig. 1
showing an independent quiz that is supporting MCQ
as question type.
Embedded/ In-Video Quizzes have questions (Set)
sandwiched in between a video lecture (as shown in
Fig. 2). It supports a limited question types due to the
complexity of integration and assessment method that
will work from within the integrated activity.
Quiz can also be categorizes based on the grading or
evaluation policy they are supporting. Based on the support
of evaluation practice, a quiz can be of two types.
Graded Quiz provides grades for each response or
answer given by the learner depending on the grading
policy used. As a leaner successfully attempts a Fig -1: Example of Independent Quiz in edX.
graded quiz, at the end marks/grades will be
calculated that will sum up in final grade of the
learner. Fig. 1 showing a graded quiz that will give 1
mark for the correct answer and learner can attempt
it maximum 3 times.
Non-Graded Quiz does not provide any grade for the
responses of the learner. These quizzes do not
participate in the final grading of the course and let
the learner attempt the quiz without worrying about
their final grading. Fig. 2 showing a non-graded quiz
giving correct/incorrect as the response feedback.
Though there are different types of quizzes as explained
above, still each one of them is related with other. So, it is not
possible to categorize MOOC quizzes into independent
categories. The relation and dependency among them is
explained as follows- Fig-2: Example of an embedded quiz in Coursera [14].
Generally, the independent quizzes are used for For computing the grades, different type of assessment
performance evaluation as in performance based quiz, methods are used. Three basic assessment methods are- 1)
while the embedded/ in-video quiz are used for self- Self assessment: done by learner itself (based on rubrics); 2)
evaluation or practice purpose only. Peer assessment: get assessed by colleagues (based on
Independent quizzes can be graded or non-graded. rubrics); and 3) Automated assessment: where the evaluation
Whereas in-videos quizzes are mainly non-graded. is done by some machine learning mechanism (based on
Sometimes automatically graded in-video quizzes are specially designed grading tool or algorithm). The simplest
also used in MOOC. assessment method captures the learner responses and
compare with the answers in the database for exact match.
For evaluating performance of the learner using Each of the assessment types has their own benefits and
performance based quiz, graded quizzes are used. serves for a different learning purpose. A proper
However, non- graded quizzes are used to serve for combination of all assessment types can help in creating a
the practice purpose mainly by providing feedback to better learning environment for engaging the learners.

2017, IRJET | Impact Factor value: 5.181 | ISO 9001:2008 Certified Journal | Page 305
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056
Volume: 04 Issue: 07 | July -2017 www.irjet.net p-ISSN: 2395-0072

5. QUIZ SUPPORT SYSTEM answer. Most of the learners prefer automated assessment
methods as reported by [13].
MOOC courses are delivered using the MOOC platform that Quiz is one of the assessment components in MOOC. The
itself covers a range of tools assisting in the process of process of assessing massive audiences is a very complex
delivering various features including quiz, to the learner. task where complexity even increases if the complex quizzes
However, different MOOC platform have different set of are used in the course. Due to massiveness and complexity of
features and services for their components like, quiz. There the course activities (including quiz) in MOOC, automatic
are some specific services required for the quiz in MOOC, evaluation of the learners can be the best suitable solution.
discussed as follows-
Some of MOOC platform supports the automatic
5.1 Authoring Tool assessment method. For example, automatic assessment for
the programming exercises in MOOC [14]. Though various
These are specialized tool for designing and delivery of methods are already available for automatic grading, yet,
course content, as well as course related activities. It places there is lot more that need to be developed for the same.
the course elements, such as, text, graphics, and questions,
and turn individual screens into a complete course with 5.4 Learning Analytics Support
pages, navigation, menu and buttons.
Learning analytics are the tools designed to use the data
Other than designing a quiz, an instructor have to perform collected from different sources (can be quiz in MOOC) in
several task such as, checking learners progress, ensuring order to measure different learning processes, like,
availability of quiz, provide feedback to learners questions, success/failure, most engaging content, most active learners
moderating course activities. Moreover, various complex
course activities like, a quiz with activity based questions are MOOC courses are complex to design and deliver due to
also introduced make authoring process difficult for the massiveness of audiences. But due to large participation in a
instructor that needs some special assistance methods. course, huge data is generated from the actions of the
participants, like, attempt quiz (skip question, check answer,
Authoring tools are used to make the authoring process number of tries), interaction with the video lecture (pause,
easier and make it possible for the course instructors to play, resume, skip) etc. Numerous studies [6] [7] [15] [16]
create complex question types in MOOC quiz. It helps and have used quiz as the useful metric for identifying some
assists the course instructor to design and implement the learning or course related factors. The MOOC providers also
quizzes with ease. Generally, MOOC platforms provide gather and process this data generated from course activities
authoring tools. For example, Open edX provide authoring including quiz, and extract some valuable information to
tool named as studio for designing MOOC course. help in delivering better learning environment.
5.2 Automatic Quiz Generation Learning analytics is not yet supported by all MOOC
platforms, but the systems are being developed and
It is the process of creation of quiz automatically using enhanced to provide feature of learning analytics. For
different methods, like, semantic based [10], sentence example, in September 2015, Open edX added a module
formation [11] etc. For instance, identify some course related named insight especially for learning analytics, some other
attributes from the plain course content and then the quiz efforts also done to extend the architecture of khan academy
sentences are automatically formed [12]. [17], Open edX [18] to include learning analytics.
A learning system like MOOC can get benefited with
support of such method as it will free an instructor from 5.5 Question and Test Interoperability (QTI)
creating questions for the quiz. As, it automates some of the IMS7 organization provides a standardization specification
authoring tasks, it saves a lot of time of instructor that can be as QTI standard, for questions and tests of any digital test
constructively used for other course related tasks. content. It describes a data model for the representation of
Commonly, MOOC platforms do not comes up with question (assessment Item) and test (assessment Test) data
integrated automatic quiz generation system systems. But and their corresponding results reports [19].
the platforms need to be improved and developed to provide Due to the standardization of the tests and questions, the
such features as done in [10] for AMOL platform. data can be exchanged with other system like, LMS, CMS,
VLE, learning analytics, authoring tool, and many others.
5.3 Automatic Assessment Support Therefore, QTI specification supports the interoperability
It is one of the assessment methods that automatically and reusability to the content that is in the form of questions.
measure performance of learner using different techniques Therefore, in order to provide interoperability to the
of machine learning, artificial intelligence, synaptic method. quizzes in MOOC and make them suitable to exchange with
It provides opportunity to the learners to applying their
learning and get instant responses as correct/incorrect
7 https://www.imsglobal.org/
2017, IRJET | Impact Factor value: 5.181 | ISO 9001:2008 Certified Journal | Page 306
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056
Volume: 04 Issue: 07 | July -2017 www.irjet.net p-ISSN: 2395-0072

multiple learning systems, the questions should be designed practice in long-term retention. Trends in cognitive
according to the QTI specifications. Although many systems sciences, 15(1), pp.20-27..
does not supports QTI specifications but now various [9] Chauhan, J., & Goel, A., 2016. An analysis of quiz in
learning system including MOOC platform, are designed and MOOC. In Contemporary Computing (IC3), 2016 Ninth
developed to support the QTI standards along with the other International Conference on (pp. 1-6). IEEE.
standardization methods.
[10] Miranda, S., Mangione, G.R., Orciuoli, F., Gaeta, M. and
Loia, V., 2013. Automatic generation of assessment
6. CONCLUSIONS objects and Remedial Works for MOOCs. In Information
Technology Based Higher Education and Training
MOOC has become common means for offering online (ITHET), (pp. 1-8). IEEE.
courses worldwide. Primarily, it uses the quiz as the
[11] Kulkarni, C., Wei, K.P., Le, H., Chia, D., Papadopoulos, K.,
assessment activity with the other course content. An
Cheng, J., Koller, D. and Klemmer, S.R., 2013. Peer and
instructor offers the quiz by defining several properties such
self assessment in massive online classes. ACM
as, defining questions, type of questions, answers for
Transactions on Computer-Human Interaction
checking, grading policy, and assessment method. To set up a
(TOCHI), 20(6), p.33.
ground for understanding the MOOC quiz, a discussion is
provided in detail for the kind of users in quiz, different [12] Jain, S., 2015, August. Automated generation of
types of quizzes, basis for selection of type of quiz, and programming language quizzes. In Proceedings of the
different assessment methods used in MOOC. Furthermore, 2015 10th Joint Meeting on Foundations of Software
for delivering the quiz by defining the properties, the Engineering (pp. 1051-1053). ACM.
instructor needs to use some specific services or tools like, [13] Papathoma, T., Blake, C., Clow, D. and Scanlon, E., 2015.
authoring tool. As there are many tools and service available, Investigating learners views of assessment types in
a discussion of the basic ones is provided here. We shall see Massive Open Online Courses (MOOCs). Lecture Notes in
further disruption as well as innovation in terms of the type Computer Science, 9307, pp.617-621.
of quiz supported, their question type, assessment method to [14] Staubitz, T., Klement, H., Renz, J., Teusner, R. and Meinel,
support more complex problems in the quiz in future. C., 2015, December. Towards practical programming
exercises and automated assessment in massive open
REFERENCES online courses. In Teaching, Assessment, and Learning for
Engineering (TALE), 2015 IEEE International Conference
[1] Voss, B.D., 2013. Massive Open Online Courses (MOOCs): on (pp. 23-30). IEEE.
A Primer for University and College Board Members. [15] Ruby, A., Perna, L., Boruch, R. and Wang, N., 2015. Are
Association of Governing Boards (AGB) of Universities There Metrics for MOOCS From Social Media?. Online
and Colleges. Learning, 19(5).
[2] Class Central, MOOC Stats, 2016, https://www.class- [16] Bote-Lorenzo, M.L. and Gmez-Snchez, E., 2017, March.
central.com/report/mooc-stats-2016/ Predicting the decrease of engagement indicators in a
[3] Desouza, E. and Fleming, M., 2003. A comparison of in- MOOC. In Proceedings of Seventh International Learning
class and online quizzes on student exam Analytics & Knowledge Conference (pp. 143-147). ACM.
performance. Journal of Computing in Higher [17] Ruiprez-Valiente, J.A., Muoz-Merino, P.J. and Kloos,
Education, 14(2), pp.121-134. C.D., 2013. An architecture for extending the learning
[4] Wilder, D.A., Flood, W.A. and Stromsnes, W., 2001. The analytics support in the Khan Academy framework.
use of random extra credit quizzes to increase student In Proceedings of the First International Conference on
attendance. Journal of Instructional Psychology, 28(2). Technological Ecosystem for Enhancing
[5] Yuan, L., Powell, S. and CETIS, J., 2013. MOOCs and open Multiculturality (pp. 277-284). ACM.
education: Implications for higher education. [18] Ruiz, J.S., Daz, H.J.P., Ruiprez-Valiente, J.A., Muoz-
[6] Onah, D.F., Sinclair, J. and Boyatt, R., 2014. Dropout rates Merino, P.J. and Kloos, C.D., 2014, October. Towards the
of massive open online courses: behavioural development of a learning analytics extension in open
patterns. EDULEARN14 Proceedings, pp.5825-5834. edX. In Proceedings of the Second International
Conference on Technological Ecosystems for Enhancing
[7] Anderson, A., Huttenlocher, D., Kleinberg, J. and Multiculturality (pp. 299-306). ACM.
Leskovec, J., 2014, April. Engaging with massive online
courses. In Proceedings of the 23rd international [19] IMS Question and Test Interoperability, 2005. Retrieved
conference on World wide web (pp. 687-698). ACM. May 2015, from
https://www.imsglobal.org/question/qtiv2p2/imsqti_v
[8] Koller, D., 2012. MOOCs on the Move: How Coursera Is 2p2_oview.html
Disrupting the Traditional Classroom (text and
video). Knowledge@ Wharton Podcast. Roediger, H.L.
and Butler, A.C., 2011. The critical role of retrieval

2017, IRJET | Impact Factor value: 5.181 | ISO 9001:2008 Certified Journal | Page 307