Enriching the Student Model in an

Intelligent Tutoring System

Submitted in partial fulfillment of the requirements for the degree

of Doctor of Philosophy
of the
Indian Institute of Technology, Bombay, India
and
Monash University, Australia
by
Ramkumar Rajendran

Supervisors:
Sridhar Iyer (IIT Bombay)
Sahana Murthy (IIT Bombay)
Campbell Wilson (Monash University)
Judithe Sheard (Monash University)

The course of study for this award was developed jointly by

the Indian Institute of Technology, Bombay and Monash University, Australia
and given academic recognition by each of them.
The programme was administered by The IITB-Monash Research Academy.

2014
 Dedicated
to
my Parents, Teachers and the Almighty

iii
iv
 Thesis Approval
The thesis entitled

Enriching the Student Model in an

Intelligent Tutoring System
by

Ramkumar Rajendran
(IIT Bombay Roll Number: 08407406,
Monash Student ID Number: 22117954)

is approved for the degree of

Doctor of Philosophy

Examiners

1.

2.

3.

Supervisors

1.

2.

3.

Chairman

1.

Date: Place:

v
vi
 Declaration

I declare that this written submission represents my ideas in my own words and
where others’ ideas or words have been included, I have adequately cited and refer-
enced the original sources. I also declare that I have adhered to all principles of aca-
demic honesty and integrity and have not misrepresented or fabricated or falsified any
idea/data/fact/source in my submission. I understand that any violation of the above
will be cause for disciplinary action by the Institute/the Academy and can also evoke
penal action from the sources which have thus not been properly cited or from whom
proper permission has not been taken when needed.
Notice 1
Under the Copyright Act 1968, this thesis must be used only under normal conditions of
scholarly fair dealing. In particular no results or conclusions should be extracted from it,
nor should it be copied or closely paraphrased in whole or in part without the written
consent of the author. Proper written acknowledgement should be made for any assistance
obtained from this thesis.
Notice 2
I certify that I have made all reasonable efforts to secure copyright permissions for third-
party content included in this thesis and have not knowingly added copyright content to
my work without the owners permission.

Place Signature

Date Name

vii
viii
 Abstract

An intelligent tutoring system is a computer-based self-learning system which pro-

vides personalized learning content to students based on their needs and preferences. The
importance of a students’ affective component in learning has motivated adaptive ITS
to include learners’ affective states in their student models. Learner-centered emotions
such as frustration, boredom, and confusion are considered in computer learning envi-
ronments like ITS instead of other basic emotions such as happiness, sadness and fear.
In our research we detect and respond to students’ frustration while they interact with
an ITS. The existing approaches used to identify affective states include human obser-
vation, self-reporting, modeling affective states, face-based emotion recognition systems,
and analyzing data from physical and physiological sensors. Among these, data-mining
approaches and affective state modeling are feasible for the large scale deployment of ITS.
Systems using data-mining approaches to detect frustration have reported high accuracy,
while systems that detect frustration by modeling affective states not only detect a stu-
dent’s affective state but also the reason for that state. In our approach we combine these
approaches. We begin with the theoretical definition of frustration, and operationalize
it as a linear regression model by selecting and appropriately combining features from
log file data. We respond to students’ frustration by displaying messages which motivate
students to continue the session instead of getting more frustrated. These messages were
created to praise the student’s effort, attribute the results to external factors, to show
sympathy for failure and to get feedback from the students. The messages were displayed
based on the reasons for frustration.

We have implemented our research in Mindspark, which is a mathematics ITS with

a large scale deployment, developed by Educational Initiatives, India. The facial observa-
tions of students were collected using human observers, in order to build a ground truth
database for training and validating the frustration model. We used 932 facial observa-
tions data from 27 students to create and validate our frustration model. Our approach
shows comparable results to existing data-mining approaches and also with approaches
that model the reasons for the students’ frustration. Our approach to responding to frus-
tration was implemented in three schools in India. Data from 188 students from the three
schools, collected across two weeks was used for our analysis. The number of frustration
instances per session after implementing our approach were analyzed. Our approach to
responding to frustration reduced the frustration instances statistically significantly–(p <
0.05)–in Mindspark sessions. We then generalized our theory-driven approach to detect

ix
other affective states. Our generalized theory-driven approach was used to create a bore-
dom model which detects students’ boredom while they interact with an ITS. The process
shows that our theory-driven approach is generalizable to model not only frustration but
also to model other affective states.

x
Publications from the thesis work

• A Theory-Driven Approach to Predict Frustration in an ITS, Ramkumar Rajen-

dran, Sridhar Iyer, Sahana Murthy, Campbell Wilson, and Judithe Sheard, IEEE
Transactions on Learning Technologies, Vol 6 (4), pages 378–388, Oct-Dec 2013.

• Responding to Students’ Frustration while Learning with an ITS, To be submitted

to the IEEE Transactions on Learning Technologies.

• Literature Driven Method for Modeling Frustration in an ITS, Ramkumar Rajen-

dran, Sridhar Iyer, and Sahana Murthy, International Conference on Advanced
Learning Technologies (ICALT), 2012, Rome, Italy.

• Automatic identification of affective states using student log data in ITS, Ramku-
mar Rajendran, Doctoral Consortium in International Conference on Artificial In-
telligence in Education (AIED), 2011, Auckland, New Zealand.

xi
xii
 Contents

Page

Abstract ix

List of Tables xxv

List of Figures xxv

List of Abbreviations xxvi

1 Introduction 1

1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Solution Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.4 Thesis structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Background 11

2.1 Intelligent Tutoring System . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 History of ITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.2 Generic Architecture of ITS . . . . . . . . . . . . . . . . . . . . . . 13

2.1.3 Research areas in ITS . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2 Affective Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2.1 Affective States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 Frustration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.3.1 Rosenweig’s Frustration Theory (1934) . . . . . . . . . . . . . . . . 20

xiii
 2.3.2 Frustration Aggression Hypothesis (1939) . . . . . . . . . . . . . . . 20

2.3.3 Frustration and Goal-Blockage . . . . . . . . . . . . . . . . . . . . . 22

2.3.4 Experimental Research on Frustration . . . . . . . . . . . . . . . . 22

2.3.5 Definition of Frustration Used in Our Research . . . . . . . . . . . 23

2.4 Motivation Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.4.1 Hull’s Drive Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.4.2 Lewin’s Field Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4.3 Atkinson’s theory of achievement motivation: . . . . . . . . . . . . 25

2.4.4 Rotter’s Social Learning Theory: . . . . . . . . . . . . . . . . . . . 26

2.4.5 Attribution Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5 Responding to Frustration . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3 Research Platform 33

3.1 Mindspark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.1.1 Student Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.1.2 Learning Content Database . . . . . . . . . . . . . . . . . . . . . . 38

3.1.3 Browser Log and Student Model . . . . . . . . . . . . . . . . . . . . 39

3.1.4 Adaptation Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.1.4.1 Cluster level adaptation logic . . . . . . . . . . . . . . . . 40

3.1.4.2 Adaptation within a cluster . . . . . . . . . . . . . . . . . 41

3.1.5 Sparkies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4 Related Work of Affective Computing 45

4.1 Affective States Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2 Detecting Frustration from the Student’s Interactions with the System . . 52

4.2.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

xiv
 4.3 Responding to Affective States . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.3.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.4 Motivation for Our Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5 Detect Frustration 63

5.1 Theory-Driven Model to Detect Frustration . . . . . . . . . . . . . . . . . 64

5.2 Theory-Driven Model for Mindspark Log Data . . . . . . . . . . . . . . . . 66

5.2.1 Frustration Model for Mindspark . . . . . . . . . . . . . . . . . . . 66

5.2.2 Feature Selection and Combination in the Theory-Driven Approach 70

5.3 Experiment Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

5.3.1 Human Observation . . . . . . . . . . . . . . . . . . . . . . . . . . 72

5.3.1.1 Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5.3.1.2 Video Recording Procedure . . . . . . . . . . . . . . . . . 73

5.3.1.3 Instrument . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5.3.1.4 Observation Procedure . . . . . . . . . . . . . . . . . . . . 74

5.3.1.5 Labeling Technique . . . . . . . . . . . . . . . . . . . . . . 75

5.3.1.6 Universality in Facial Expressions . . . . . . . . . . . . . . 75

5.3.2 Analysis Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.3.3 Metrics to Validate Frustration Model . . . . . . . . . . . . . . . . 78

5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.5 Performance of Data-Mining Approaches Applied to the Data from Mindspark

Log File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5.6 Performance of Theory-Driven Constructed Features Applied to Other Clas-

sifier Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.7 Generalization of our Model Within Mindspark . . . . . . . . . . . . . . . 89

5.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

6 Responding to Frustration 93

6.1 Our Approach to Respond to Frustration . . . . . . . . . . . . . . . . . . . 93

xv
 6.1.1 Reasons for Frustration . . . . . . . . . . . . . . . . . . . . . . . . . 95

6.1.2 Motivational Messages . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.1.3 Algorithms to Display Motivational Messages . . . . . . . . . . . . 98

6.2 Methodology to Show Motivational Messages in Mindspark . . . . . . . . . 102

6.2.1 Implementation of Affective Computing in Mindspark . . . . . . . . 104

6.3 Impact of Motivational Messages on Frustration . . . . . . . . . . . . . . . 106

6.3.1 Methodology to Collect Data from Mindspark Sessions . . . . . . . 107

6.3.2 Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6.3.3 Data Collection Procedure . . . . . . . . . . . . . . . . . . . . . . . 107

6.3.4 Data Analysis Technique . . . . . . . . . . . . . . . . . . . . . . . . 109

6.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

6.4.1 Validation of Impact of Motivational Messages . . . . . . . . . . . . 113

6.4.2 Detailed Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

6.4.2.1 Impact of Motivational Messages on Learning Achievement 117

6.4.2.2 Impact of Motivational Messages on Average Time to An-

swer the Questions in Mindspark Session . . . . . . . . . . 118

6.4.2.3 Analysis on Ordering Effects - Removal of Motivational

Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

7 Generalizing Theory-Driven Approach 123

7.1 Applying Theory-Driven Approach to Model Boredom . . . . . . . . . . . 123

7.2 Definition of Boredom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

7.2.1 Definition of Boredom Used in Our Research . . . . . . . . . . . . . 126

7.3 Modeling Boredom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

7.3.1 Boredom Model for Mindspark . . . . . . . . . . . . . . . . . . . . 127

7.4 Experiment Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

7.4.1 Self Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

xvi
 7.4.2 Analysis Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.4.3 Metrics to Validate Boredom Model . . . . . . . . . . . . . . . . . . 131
7.4.4 Related Works - Detecting Boredom . . . . . . . . . . . . . . . . . 132
7.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.5.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

8 Summary and Conclusion 137

8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
8.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
8.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Appendix 142

Bibliography 143

Acknowledgement 165

xvii
xviii
 List of Tables

Table Page

4.1 Synthesis of Different Approaches to Detect Affect States . . . . . . . . . . 50

4.2 Research Works, that Identify Frustration Using the Data from Student
Log File, with Number of Features, Detection Accuracy and Classifiers used 57

4.3 Related Research Works to Respond to Student’s Affective States along

with the Theories used, Experiment Method and Results . . . . . . . . . . 60

5.1 Student Goals and Blocking Factors for Mindspark . . . . . . . . . . . . . 68

5.2 An example to illustrate the advantages between selecting features by ap-

plying goal-blockage based theory and by a simple combination of data
from the log file when applied to Mindspark . . . . . . . . . . . . . . . . . 71

5.3 Sample Human Observation Sheet to Record Students’ Facial Observations

and to Label it as Frustration (Frus) or Non-Frustration (Non-Frus) . . . 76

5.4 Contingency Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5.5 Frustration model performance for different alpha values . . . . . . . . . . 81

5.6 Frustration model performance for different threshold values . . . . . . . . 82

5.7 Contingency Table of Our Approach when Applied to Mindspark Log Data 82

5.8 Performance of our Approach Shown Using Various Metrics when Applied
to Mindspark Log Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5.9 Description of Features Selected from Mindspark Log Data . . . . . . . . . 84

5.10 Comparison of our Approach with Existing Data-Mining Approaches Ap-

plied to the Data from Mindspark Log File . . . . . . . . . . . . . . . . . . 85

xix
5.11 Performance of Theory-Driven Features when Applied to Higher Order
Polynomial Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.12 Performance of Theory-Driven Features on Different Classifiers . . . . . . . 89

5.13 Confusion Matrix of Mumbai School Data . . . . . . . . . . . . . . . . . . 90

6.1 Explanation of the Reasons for Goal Failure - Events . . . . . . . . . . . . 96

6.2 Messages to Respond to Frustration with Condition and Justification . . . 97

6.3 Details of the data collected from three schools to measure the impact of
motivational messages on frustration . . . . . . . . . . . . . . . . . . . . . 109

6.4 Median and Median Absolute Deviation (MAD) of number of frustration

instances from the Mindspark session data from three schools . . . . . . . . 110

6.5 Impact of motivational messages on frustration in three schools . . . . . . 112

6.6 Mann-Whitney significance test on frustration instances from Mindspark

sessions without motivational messages . . . . . . . . . . . . . . . . . . . . 114

6.7 Transition matrix of the Mindspark sessions of 88 students, for a number

of frustration instances per session, from without motivational messages to
with motivational messages . . . . . . . . . . . . . . . . . . . . . . . . . . 116

6.8 Percentage of increase and decrease in frustration instances at one and

two steps. The data are from the Mindspark sessions of three schools.
Percentage of no change is not shown . . . . . . . . . . . . . . . . . . . . . 117

6.9 Impact of the motivational messages on students with different learning

achievement. The students are grouped based on their percentage of correct
response in the Mindspark session. Low (< 60%), Medium (60% < and <
80%), and High (> 80%). The percentage of frustration instances which
increased and decreased due to the motivational messages is shown. The
percentage of no change in frustration instances are not shown . . . . . . . 118

xx
6.10 Impact of the motivational messages on the average time spent by the
students to answer the questions in the Mindspark session. The students
are grouped based on their average time spent to answer the questions in
the Mindspark session. Less (< 20 seconds), Average (20 seconds < and
< 30 seconds), and More (> 30 seconds). The percentage of frustration
instances which increased and decreased due to the motivational messages
is shown. The percentage of no change in frustration instances are not shown119

7.1 Contingency Table of Our Approach when Applied to Mindspark Log Data 135
7.2 Performance of our Approach Shown Using Various Metrics when Applied
to Mindspark Log Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

xxi
xxii
 List of Figures

Figure Page

1.1 Focus Area. LMS = Learning Management System . . . . . . . . . . . . . 2

1.2 Proposed Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Generic Architecture of an Intelligent Tutoring System . . . . . . . . . . . 14

3.1 Mindspark screen showing a sample question from class 6 on topic Per-
centages. The Mindspark interface contains student name, class, selected
topic, a question from the topic, progress indicator in the topic, number
of Sparkies collected in the session, options to change the topic or end the
current session, and an option to provide feedback. . . . . . . . . . . . . . 34

3.2 A sample Mindspark screen showing the correct answer with explanation . 35

3.3 Mindspark Architecture - Intelligent Tutoring System . . . . . . . . . . . . 36

3.4 The options in emote toolbar. The options are “Confused”, “Bored”, “Ex-
cited”, “Like”, “Dislike” and “comment” to receive the student’s feedback . 37

3.5 The options for the student’s to enter their feedback when the question
s/he answered is incorrect . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.6 Sparkie is introduced to the students and the rules of earning Sparkies are
described . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.7 Challenge Question from Topic Percentages . . . . . . . . . . . . . . . . . 43

5.1 Steps of the theory-driven approach to create a frustration model using

data from the log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

xxiii
5.2 Convergence of Weights of our Linear Regression Frustration Model using
Gradient Descent Algorithm. The Cost J in the Figure is the error function
(difference between detected observation Pi and human observation Bi ) . . 81

6.1 Steps of our Approach to Respond to Frustration . . . . . . . . . . . . . . 94

6.2 Block Diagram of our Methodology to Detect and Respond to Frustration

in Mindspark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

6.3 Screen-Shot of Mindspark’s Buddy Showing a Motivational Message as

Speech Bubble - 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

6.4 Screen-Shot of Mindspark’s Buddy Showing a Motivational Message as

Speech Bubble - 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6.5 Methodology to collect data for validating our approach to respond to

frustration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

6.6 Box plot of Frustration instances from 188 sessions without and with moti-
vational messages. Box = 25th and 75th percentiles; bars = minimum and
maximum values; center line = median; and black dot = mean. . . . . . . . 111

6.7 (a). Histogram of Number of the Frustration Instances in 188 sessions

without Motivational Messages and (b). Histogram of Number of the Frus-
tration Instances in 188 sessions with Motivational Messages . . . . . . . . 112

6.8 Box plot of Frustration instances from Mindspark sessions in a school for
consecutive weeks without responding to frustration. Box = 25th and 75th
percentiles; bars = minimum and maximum values; center line = median,
and black dot = mean. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

6.9 Box plot of Frustration instances from 42 session in each week. First week
without motivational messages, second week with motivational messages
and third week without motivational messages. Box = 25th and 75th per-
centiles; bars = minimum and maximum values; center line = median; and
black dot = mean. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

xxiv
7.1 Steps of theory-driven approach to create a boredom model using data from
the Mindspark log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.2 EmotToolbar integrated with Mindspark user interface to collect students’
emotions. The emote bar is in right side of the figure. . . . . . . . . . . . . 129
7.3 The EmotToolbar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

xxv
 List of Abbreviations

Commonly used abbreviations

CAI Computer assisted instruction

CBSE Central Board of Secondary Education

DBN Dynamic Bayesian Network

DDN Dynamic Decision Network

FACS Facial Action Coding System

HMM Hidden Markov Model

ICAI Intelligent CAI

ICSE Indian Certificate of Secondary Education

IGCSE International General Certificate of Secondary Education

ITS Intelligent Tutoring System

KNN k-nearest neighbor

LMT Logistic Model Tree

MLP Multilayer Perceptron

NLP Natural Language Processing

SVM Support Vector Machine

Abbreviations defined in this thesis

Fi Frustration Index

RoF Reason for frustration

SDL Sub difficulty level

TT Teacher Topics

xxvi
 Chapter 1

Introduction

E-learning is one of the important fields of research in education. E-learning provides

the same learning content to all users irrespective of their age, experience, knowledge and
the like. To manage the learning content and to help in teaching process the learning
management system (LMS) are used. LMS are used to manage the students’ interaction
with course content, managing the learning content, and allowing interaction among stu-
dents and teachers. To provide learning content to the students at anywhere and anytime,
Ubiquitous Learning is used. Due to availability of mobile devices like mobile phones and
tablet, and high speed internet connection, the learning content can be accessed by the
students using their mobile devices, anywhere and at anytime (Ubiquitous learning). How-
ever, providing the same learning content to all students can lead to cognitive mismatch.
Providing difficult learning content to low performers and providing non challenging tasks
to high performers is known as “cognitive mismatch”. Intelligent Tutoring System (ITS)
is used to provide personalized learning content to each student. The focus area of our
research work is ITS. Our research focus in e-learning is shown in Figure 1.1∗ . ITS pro-
vides personalized learning content to students based on factors like their performance,
prior knowledge, and others [22], [35]. ITS is defined as “computer-based instructional
systems with models of instructional content that specify what to teach, and teaching
strategies that specify how to teach” [117]. The intelligence in ITS refers to “the adapta-
∗
The diagram serves to provide general view about our research area and is not the outcome of our
research. The diagram does not represent all domains in e-learning. Moreover, the interactions between
domains in e-learning may vary.

1
tion of its tutoring, which means to provide different tutoring to the individual student”
[166]. In ITS, the sequencing of the learning content is personalized to avoid a cognitive
mismatch. Adapting the learning content based on the student’s needs, and personalizing
the learning for the student, enables the ITS to work with students of different abilities.

E-Learning

Adaptive System
LMS
ITS

Mobile Learning
Ubiquitous
Learning

Figure 1.1: Focus Area. LMS = Learning Management System

The generic architecture of an ITS consists of a student model, learning content

and an adaptation engine. The student interacts with the ITS by solving problems, or
playing games or by other means of interaction. Students’ interaction with the ITS, such
as responses to questions, number of attempts at a task, and the time taken for various
activities (such as responding or reading) are captured in the ITS log file. The log file
is analyzed to identify the student’s performance and preferences and is stored in the
student model. The student model also contains general information about the students,
such as class level, previous knowledge and background. The adaptation engine in the
ITS tailors the learning content based on the student’s request and the data from his/her
profile.

To improve the performance of the ITS, affective characteristics of the students

should also be considered while tailoring the learning content, as is done in traditional

2
one-on-one learning. D’mello et al.[47] identified the fundamental components of learning
as cognition, motivation and emotions. The importance of a student’s affective component
in learning has led ITS to include learners’ affective states in their student models. The
consideration of affective processes has been shown to achieve higher learning outcomes
in ITS [93], [164].

Recognizing the student’s affective states and responding to it is called affective

computing [126]. Affective computing was first introduced by Picard [126] and considered
in various domains which included gaming, learning, health, entertainment and others
[125], [126]. In the past decade, affective computing research generated great interest in
affect detection [26]. The affective states widely used in affective computing research in
ITS [39], [33], [139], [82], [18], [46] are: frustration, boredom, confusion, delight, surprise
and engaged concentration. In our research,we focus on negative affective states like
frustration and boredom. We start our research with modeling frustration later we apply
our model to detect boredom. To include affective states in the student model, students’
affective states should be identified and responded to, while they interact with the ITS. In
affective computing, detecting affective states is a challenging, key problem as it involves
emotions–which cannot be directly measured; it is the focus of several current research
efforts [26], [167].

Methods that have been implemented in ITS to detect the affective state include
human observation [39], [134], [133], learners’ self-reported data of their affective state
[34], [33], mining the system’s log file [48], [132], [88], [111], by modeling affective states
[33], [139], face-based emotion recognition systems [93], [47], [58], [110], [14], [122], [74],
analyzing the data from physical sensors [46], [52], [89], [34], and more recently, sensing
devices such as physiological sensors [18], [83], [25], [105]. Advances in these methods to
detect affective states look promising in a lab setting. However, they are not yet feasible
in a large scale, real-world scenario [133]. The exceptions are data-mining approaches and
modeling affective states.

Existing systems which use data-mining approaches [48], [111] have reported high

3
detection accuracy (77% - 88 %) in detecting frustration. On the other hand, the advan-
tage of systems which are based on modeling of affective states is that they not only detect
the affective state of the learner but also shed light on the reasons for that state. The
existing system [139], [33] which models the affective states reported the less detection ac-
curacy (28% - 70%). Detecting affective states with high accuracy along with the reasons
for frustration is still an open research problem. Since the reasons for frustration are not
identified in data-mining approaches, the existing system which responds to frustration
provides an ad-hoc responses. Hence, an approach to detecting frustration more accu-
rately along with the reasons for the frustration is needed to have an informed responses.
This has provided the motivation for us to create a new frustration model–using features
constructed from a theoretical definition that will detect frustration with high accuracy
along with the reasons for frustration. Hence, our model will respond to the reasons for
frustration.

Two responses to affective states in ITS are: a) Adapting the learning content to
address the cause of frustration [3] and b) Providing motivational messages to encourage
the students to create new goals and to avoid the negative consequences of frustration [93],
[133] and [164]. In our research we choose to respond frustration by providing motivational
messages.

1.1 Objective

The objective of our research is to create a model to detect and respond to frustration
accurately in real-time when students are working with an ITS. To achieve our objective
we proposed a theory-based approach. Our proposed methodology is shown in Figure 1.2.

We divide our proposed methodology into four phases:

Phase I: Identify the definition of frustration from educational and cognitive psy-
chology. Identify the methods used to detect frustration from facial expressions.
This phase is explained in Chapter 2.

4
 Phase I
Defintion of
frustration
Operationalize for ITS
User Interface Model to predict
Log data
(System) frustration
Student If frustrated
Messages to Reasons for
Phase IV handle frustration frustration.
Phase II
Motivation Theory

Phase III

Figure 1.2: Proposed Methodology

Phase II: Operationalize the definition of frustration using the data from the ITS
log file. Construct the features to detect frustration using the data from the log
file. Create and validate the frustration model using the constructed features. This
phase is explained in Chapter 5.

Phase III: Identify the strategies used to respond the negative consequences of frus-
tration. Create the strategies to respond to frustration using motivational messages.
This phase is explained in Chapter 2 and 6.

Phase IV: Develop an approach to show motivational messages based on the rea-
sons for frustration. Implement the approach in an ITS and validate its performance.
This phase is explained in Chapter 6.

5
1.2 Solution Approach

To achieve the objectives described in the previous section, we surveyed the literature
for definitions of frustration. We considered theoretical and experimental definitions of
frustration to define it. We used the definition of frustration from theory as “an emo-
tion caused by interference preventing/blocking one from achieving the goal.” We have
implemented our research in Mindspark, which is a commercial ITS developed by Ed-
ucational Initiatives, India, and has been used across schools in India. The log data
from Mindspark were analyzed and operationalized, based on the theoretical definitions
of frustration. Hence, we constructed the theory-based features from the Mindspark log
data. The constructed features are used to create a linear regression model to detect
frustration. Human observation to detect frustration was used as an independent method
to train and validate the model. We created, validated, and applied an observation proto-
col based on FACS, and collected the real-time classroom data from 11 students of class
six. The dataset was stratified based on the student’s Mindspark sessions and a ten-fold
cross-validation method was applied to avoid bias in the dataset. The training datasets
were used to learn the weights of the frustration model. The gradient descent method
was used to calculate the weights of the model. The trained model was tested using the
test dataset. The performance measures such as precision, recall and accuracy of the
frustration model were then calculated.

To improve the performance of the model, more qualities of frustration, such as

the cumulative nature of frustration and time spent to achieve the goal were considered
and the frustration model was modified accordingly. Also, we increased our independent
dataset to improve the performance. The data was collected in two different schools from
two different cities (Mumbai, Ahmedabad) in India to generalize the frustration model for
all schools in Mindspark. Cohen’s Kappa, ROC analysis, and the F1 score were calculated
to validate the performance of the frustration model. The constructed features were
applied to a higher order polynomial model and non linear classifiers using Weka–a data-
mining software. To compare the performance of existing data-mining approaches on the

6
Mindspark log data, several features were constructed from the log file without application
of theory; the selected features were applied to the different classifiers in Weka. The results
of our model are relatively equal to existing data- mining methods in terms of accuracy
and relatively better in terms of precision. Also, the reasons for frustration are identified
in our model. The performance of the linear regression model is comparatively similar
to other models. Hence, for ease of understanding and implementation in Mindspark, we
used the linear regression model in our research.
To respond to student’s frustration detected by our model, while they work with
Mindspark, we integrated the trained model in the Mindspark adaptation logic to detect
frustration at real-time. If an instance of frustration is identified by the model, messages
are shown to motivate the students to continue the session instead of getting more frus-
trated. We developed algorithms to show the motivational messages whenever the student
is detected as frustrated. The messages are selected based on the reasons for frustration.
Our motivational messages are based on a motivation theory–the attribution theory [77],
[160], and adheres to the Dweck’s [54], [53] theory of praising the effort. The messages
were created to praise the student’s effort, attribute the results to external factor, to show
the sympathy for failure and to get feedback from the students. Our approach to respond
to frustration was implemented in Mindspark adaptation logic and tested in real-time.
The number of frustration instances per session after implementing the algorithm were
analyzed. The frustration instances reduced significantly statistically (p < 0.05), due to
the motivational messages.
Hence, our objective of detecting frustration accurately along with the reasons for
it is achieved; motivational messages to avoid the negative consequences of frustration
were implemented and tested in a large scale deployment of the ITS.

1.3 Contribution

In this section, we list the major contributions from our research work.

• We have developed the theory-driven approach to detect affective states using data

7
 from the students’ interaction with the system. (Chapter 5 and Chapter 7).

• We have developed a linear regression model to detect frustration in a math ITS–

Mindspark. The detection accuracy of our model is comparatively equal to existing
models to detect frustration and also provides the reasons for why the student is in
that state. (Chapter 5).

• We have provided strategies to avoid the negative consequences based on the reasons
for frustration. Our approach to respond to frustration significantly reduced the
number of frustration instances per session (Chapter 6).

• An approach to detect and respond to frustration is built into a math ITS–Mindspark.

Student’s frustration instances were detected and motivational messages were pro-
vided to avoid frustration in real-time (Chapter 6).

• We found that motivational messages to respond frustration had a relatively high

impact on the following kinds of students: (a) those who spent more time in an-
swering the questions; (b) those who had a low average score in Mindspark sessions;
(c) those who had a high number of frustration instances per session, before imple-
menting the algorithm (Chapter 6).

• Our theory-driven approach is generalized to detect other affective states. We have

developed a model, using theory-driven approach, to detect student’s boredom when
they interact with an ITS (Chapter 7).

1.4 Thesis structure

This report is organized as:

Chapter 2 discusses the background required for our research work. The func-
tions of Intelligent Tutoring System (ITS), affective states used in affective computing
research, the definitions of frustration from educational and cognitive psychology and the
motivation theory used in our research to respond frustration are described.

8
 In Chapter 3, we briefly describe the Intelligent Tutoring System (ITS) used in our
research–Mindspark. Our approach to detect and respond to frustration is implemented
in Mindspark and tested in real-time. The details of Mindspark’s log data, adaptive logic,
learning content and user interface, are described.
Chapter 4 discusses related work on detecting and responding to the affective
states of the users. The existing approaches to detect affective states in educational
systems like games, Intelligent Tutoring Systems (ITS) and search engines, the process by
which frustration is detected and responded to, using data from the students’ interaction
with the ITS, are described.
Chapter 5 discusses the creation of our frustration model using theoretical defini-
tion of frustration. We explain operationalization of our approach using the Mindspark log
data. Then we describe our experimental setup which includes human observation, analy-
sis procedure and metrics used to evaluate our frustration model. The results of our model
and performance of other existing data-mining approaches applied to the Mindspark log
data are discussed. Finally, we describe the performance of different classifier models
using our theory-driven constructed features.
Chapter 6 details the messages and algorithm used to avoid the negative conse-
quences of frustration and its performance. Chapter 7 describes generalizing our theory-
driven approach by modeling boredom. Chapter 8 summarizes this report.

9
10
 Chapter 2

Background

In this chapter, we discuss the background required for our research work. We start with
the functions of an intelligent tutoring system (ITS), and briefly discuss the related re-
search areas in ITS. ITS adapts the learning content based on user preferences and needs.
Affective computing is concerned with adapting the learning content based on a user’s
affective states (emotions), and is one of the research areas in ITS. An overview of the
affective states used in affective computing research is provided. In our research we focus
on affective computing in ITS, and the affective state considered in our research is frus-
tration. Hence, the definitions of frustration from educational and cognitive psychology
are described in detail. Later, we describe the theories and strategies used to respond to
affective states.

2.1 Intelligent Tutoring System

An Intelligent Tutoring System (ITS) is a computer system which provides personalized

learning content to students based on factors like their performance, prior knowledge,
and the like [22], [35]. ITS may be defined as “computer-based instructional systems
with models of instructional content that specify what to teach, and teaching strategies
that specify how to teach” [117]. The intelligence in ITS refers to “the adaptation of its
tutoring, which means to provide different tutoring to the individual student” [166]. In
ITS, the sequencing of the learning content is personalized to avoid a cognitive mismatch

11
which may be caused by providing difficult learning content to low performers and pro-
viding non challenging tasks to high performers. Adapting the learning content based on
the student’s needs, and personalizing the learning for the student, enables ITS to work
with students of different abilities.

2.1.1 History of ITS

Although Charles Babbage envisioned the programmable computer and created the first
mechanical computer in 1800s, the first use of a machine for teaching is reported by
Pressey only in 1926 [145]. Pressey used the system to deliver multiple choice questions
and to provide feedback immediately. However, this system was not an ITS as it was
used only to check whether the response provided by the student was correct or not. In
1960s, programmed instructions (PI) were widely used to teach programming concepts
step by step with immediate feedback and remediation for the students [145]. In PI, the
problem is solved both by the system and the student. Later, the results are compared
and feedback is provided. If the responses do not match, a remedial learning path is
suggested. Computer assisted instruction (CAI) or Computer based instruction (CBT)
is an extension of PI in a computer. However, CAI is able to provide only predefined
learning content based on the student’s response. It is not intelligent enough to analyze
the student’s response and choose the learning content based on the analysis.

Intelligent CAI (ICAI), [161], analyzes the student’s response, diagnoses the error
and provides remediation. For example, when the sum of 56 + 47 is asked, if the student’s
response is 93, then it indicates that s/he has a problem in the carry over digit in addition,
if the student’s response is 87, then it indicates that s/he has a problem in addition.
Based on the student’s response, his/her problem can be diagnosed and then the required
learning content can be provided. The ability to diagnose the student’s problem and
provide remediation is the key difference between ICAI and CAI. The ICAI is also called
an intelligent tutoring system (ITS). Hartley and Sleeman in 1973 [76], suggested that
ITS must have knowledge of teaching strategies (adaptation strategies), knowledge of the

12
domain (learning model) and the knowledge of the learner (student model). These three
features are important features even for modern day ITS. The ’I’ (Intelligent) in ITS is
the ability to diagnose the student’s knowledge, misconceptions, skills and the adapting
of the learning content accordingly, rather than providing preprogrammed responses [32],
[146] and [145].

At present, ITSs are typically web-based learning systems [23], which are used
to teach a wide range of topics from math to medicine [96], [154], [59]. The need for
personalized learning and immediate feedback led many researchers to work on the ITS.
This resulted in a wide range of ITSs in different fields, for example the SHERLOCK
[96] tutor teaches the air-force technicians to diagnose electrical problems in jets, the
Cardiac [59] tutor simulates the function of the heart for medical students, Why2-Atlas
[154] helps in learning physics principles, and AutoTutor [70] assists students in learning
about Internet, operating systems and the like. Math tutors such as Active Math [112],
and the Cognitive tutor [131], are widely used in schools as part of the curriculum. In our
research work we focus on the web-based ITSs, which are used to teach math in schools.
The ITS used for our research, Mindspark, is explained in Chapter 3.

2.1.2 Generic Architecture of ITS

The ITS consists of the user interface, the learning content, the student model, and the
adaptation engine [22]. The generic architecture of the ITS is shown in the Figure 2.1.
These modules are briefly described below.

User Interface and Log file

The user interface delivers the learning content to the student and accepts the students’
responses to the questions posed by the ITS. Based on the nature of the ITS, the learning
content can be delivered as text, voice, simulation or interactive games. The user interface
can be a mobile device (Tablet, Mobile, Laptop) or a desktop. The students’ interaction
with the ITS, such as response to questions, number of attempts and time taken for
various activities (responding, reading and others) is captured in the log file. The log file

13
 Log
Files

Adaptation Learning
Engine Content
DB

User Interface

Student
Model

Figure 2.1: Generic Architecture of an Intelligent Tutoring System

is used to create the student model.

Learning Content
The learning content of the ITS is stored in a database as topics. The topics are divided
into learning units to enable the teaching of a specific concept or a fact. Misconceptions
and common wrong answers of each learning unit are stored in the database. The learning
units can be explanations, examples, hints, quizzes, questions and they can be used to
teach, demonstrate or test the students.

Student Model
The student model contains general information about the students, such as class level,
previous knowledge and background. Apart from this general information, the following
information of the students may be stored in the student model.

• Student’s skills, goals, plans

• Student’s performance such as topic performance and number of questions correctly

14
 answered per session

• Learning characteristics such as the learning rate, the student’s preferences and
learning styles

• Affective states such as engagement, boredom, frustration and confusion

Adaptation Model
The adaptation engine is an algorithm to select (adapt) the learning content based on
the student’s input from the user interface (response to the question) and the information
from the student model.

ITS adapts the learning content based on the learner’s preferences such as:

• The learner’s level of ability, such as, “beginner” or “intermediate” or “expert”

[102].

• The learner’s knowledge, such as, previous knowledge of learning content [80].

• Learning styles such as “visual”, “audio”, and “interactive” [19, 128].

Function of ITS
In ITS, if a student registers for the first time, general information like age, gender and
background are collected and stored in the student model. Based on the student’s request,
the learning content that matches the student’s profile is provided in the user interface.
The student then interacts with the ITS by solving problems, or playing games or by
other ways of interaction. The student’s interaction with the ITS is stored in the log
file, which is analyzed to identify the student’s performance and preferences and is stored
in the student model. The adaption engine in ITS tailors the learning content based on
the student’s request and the data from his/her profile. For example, if the ITS detects
that the student has a misconception on a learning unit, the ITS can suggest that the
student redo the current learning unit, or redo the previous learning unit or resolve the
misconception using remedial content.

15
2.1.3 Research areas in ITS

In the past four decades, the research areas in ITS have evolved according to the available
technology and user needs. In 1970s the main focus of research was on creating an auto-
matic bug library (misconception and common wrong answers) [24], expert systems [144],
tutors (teaching strategies) [31] and a reactive learning environment (system responds to
user’s actions instead of teaching or correcting the user’s mistakes) [20]. In 1980s, along
with these research areas, the research focus in ITS was simulations [78], Natural Lan-
guage Processing (NLP) to improve the student’s interaction and introducing case-based
learning [143]. In 1990s, the main focus were virtual reality [43], and collaborative learn-
ing [90]. In the past decade, the research focus has been on understanding the student’s
learning instead of teaching strategies and domain experts [21]. The key research areas
are the students’ cognitive abilities [131], preferences [27], skills [94], learning style [21],
learning strategies and affective states [68]. The research focus is on improving the stu-
dents’ learning with the ITS. It is done by predicting a) how students learn [38], b) when
students need help [9], c) when to resolve the students’ misconceptions [30].

Recent research includes improving the students’ interaction with the ITS. Re-
searchers have focused on the user interface to recognize non-textual input [69], interact
with students as a human tutor [68], and understand the students’ body language and
respond to it. Another area of research has been to improve the content of the student
model. Students’ preferences are identified from their interaction with the ITS. The stu-
dents’ emotions [133], ability and motivation [156] are detected from the log files [132] or
by analyzing the data from external devices such as video cameras [93], or from physi-
ological sensors [89]. Researchers have also worked on improving the personalization of
the learning content. Students’ preferences from the student model are considered while
adapting the learning content. Adaptation strategies are researched, in order to enable dy-
namic adaption of the available learning content for the wide range of student preferences
[155].

Researchers have extensively evaluated the students’ learning outcomes while learn-

16
ing with the ITS. A recent review article on the impact of the ITS [155], reported that
ITSs improve student learning. Most of the commercial ITSs, such as Active Math [112],
Cognitive tutor [131], [6], Mathematics Tutor [151], [16], and PAT Algebra Tutor [130],
[94] have been tested in schools and have reported an improvement in the students’ learn-
ing. Moreover, detecting the student’s affective states and responding to them (affective
computing) has been shown to achieve higher learning outcomes in ITS [93], [164]. In
affective computing, the key problem of detecting the affective states is challenging as it
involves emotions which cannot be directly measured [26].

2.2 Affective Computing

In traditional learning, affective states, like frustration, may lead the student to quit learn-
ing; hence addressing the affective state is important. In a face-to-face teaching situation,
experienced teachers observe and recognize the students’ affective states and, based on
that, change the teaching style to positively influence the learning experience. In ITS, for
effective tutoring, students’ motivation and affective components should be identified and
considered while tailoring the learning content [164], as it is done in traditional one-on-one
learning.

Affective computing was first introduced by Picard [126] and considered in various
domains including gaming, learning, health, entertainment among others. [125], [126]. In
the past decade, affective computing research generated great interest in affect detection
[26]. In the next section, we describe the affective states considered in affective computing
and the definition of frustration in detail, which is the affective state considered for our
research.

2.2.1 Affective States

The typical affective states are emotions, feelings and moods. However, in affective com-
puting research, only emotions are considered [26]. Traditional emotion theories study
emotions through facial and body expressions [41], [55], [57], [84], [85]. Emotions were

17
first explored scientifically by Darwin, and Darwin’s study reports that some facial and
bodily expressions are associated with essential emotions like fear and rejection [41], [26].
Based on Darwin’s theory of emotions, six universally recognized emotions have been de-
veloped. The six basic emotions are: anger, disgust, fear, joy, sadness and surprise [55],
[57], [85].

Emotions have been further researched by cognitive psychologists [147], [142], [135],
[120], to emphasize the connection between emotion and cognition [109]. According to
cognitive psychology theories such as the appraisal theory, people’s perception of their
circumstances and interpretation of events determine the emotion, not the event itself.
Thus, two people with different appraisals (assessing the outcome of event) or in different
circumstances may feel different emotions for the same event [136]. This is the central
concept of appraisal theories of emotion [60] and they view appraisal as the cause of the
cognitive changes associated with emotions [147].

Later cognitive psychologists [147], [142], [135], [120] explored the cognitive ap-
proaches to emotions. This research of cognitive psychologists aimed to understand the
relationship between variable (circumstances, goal) and emotion labels (joy, fear) [120]
and the relationship between appraisal variables and cognitive responses [99], [147]. The
theories of cognitive psychologists state that emotions are related to the student’s expe-
rience, goal, blockage of goal, achieving of the target, and so on. We discuss two theories
of cognitive approaches to affective states in next paragraphs.

Roseman [135] proposed that five appraisals influence emotions: a) motivational

state - the motivation is to get a reward or to avoid punishment or failure, b) situational
state - whether the motivational state is present or absent, c) probability of achieving the
goal, d) legitimacy - deserved outcome of the event, and e) agency - whether the outcome
is by self or external person or circumstances. Based on these appraisals the emotions
such as joy, pride, distress, and shame are defined.

OCC’s appraisal theory [120], explains emotions as a cognitive appraisal of the

current situation which involves the events, agents and the objects. It states that, the

18
emotions depend on how one’s preferences or goals match with the outcome of that event,
that is, the reaction to the events form the emotions. Similarly, the OCC theory discusses
the emotions that occur due to the agents and the objects. Based on the outcome of the
event, the intervention of the agent, and the objects, the OCC theory defines 22 different
emotions such as “Joy”, “distress”, “hope”, “fear” and others. The cognitive emotions
(fear, distress) are primarily focused on the student’s goals and event outcomes.

Based on research in [44], [101], [45], a review paper on affect detection [26] states
that the emotion which occurs during the learning sessions that lasts for 30 to 120 minutes
has less relevance with the basic emotions. Hence, the basic emotions might not be
relevant to the students’ emotions which occur during interaction with computer. Instead
learner-centered emotion such as frustration, boredom, confusion, flow, curiosity, and
anxiety are more applicable to computer learning environments [26], based on the research
in affective computing [39], [33], [139], [82], [18], and [46]. In learner-centered affective
states identifying and responding to the negative affective states [95] are important as it
might lead the student to drop out of learning. In educational research, there is debate on
whether negative affective states like frustration and confusion are required for learning or
should be addressed to avoid the students’ dropping out [64]. Gee’s research on learning
through games [64], [65] reports that affective states like frustration can facilitate thinking
and learning, hence required while learning. However, Gee [65] further describes that
frustration should be kept below certain level to avoid high stress, powerful anger or
intense fear. Also frustration is a cause of student disengagement and eventually leads to
attrition [88]. Hence, in our research we focus on a negative affective state–frustration.
The definitions of frustration are described in the next section.

2.3 Frustration

Frustration has been researched for more than 80 years, hence, classifying or analyzing the
various definitions of frustration is beyond the scope of this thesis. Hence we consider the
classic frustration theories and experimental definition of frustration in this thesis. We first

19
describe the frustration theories discussed in the book FRUSTRATION: The development
of a scientific concept by Lawson in 1965 [97]. The theories considered from Lawson’s
book are Rosenweig’s Frustration Theory [138] and Frustration Aggression Hypothesis
by Dollard et al. [50]. The definition of frustration from the textbook “Introduction to
Psychology”, by Morgan et al. [115], is also discussed in this section. Later we consider
the definition of frustration from experimental research by Lazara et al. [98].

2.3.1 Rosenweig’s Frustration Theory (1934)

Rosenweig [137] defines frustration as the emotion that an individual feels when a com-
monly available need or the end-state is not available now, or is removed. For example,
a student working in ITS needs help or hint to answer the question, but is not able to
get it in the ITS even though help is easily available to him/her in a classroom learning.
Rosenweig states that frustration can occur due to the external world or or due to one’s
own personal actions, such as, the student is not prepared enough to take test. Addition-
ally Rosenweig asserts that “Frustration tolerance tends to increase with age” [138]. We
infer that frustration occurs more frequently in school students when compared to college
students. Hence, we consider school children in our research.

2.3.2 Frustration Aggression Hypothesis (1939)

The frustration aggression hypothesis [50] states that the existence of frustration always
leads to some form of aggression. Frustration is defined as a “condition which exists
when a goal-response suffers interference.” To understand the definition, we construct
an example. This example is used only to explain the definition of frustration aggression
hypothesis and not related to ITS. Consider a student prepares for the exams. Based on
previous experience, s/he predicts that goal of getting good grades in exams is achievable.
His interest to achieve the goal is conveyed using several indicators like not playing games,
less time on social networking, and his/her preparation for exam. The strength of these
indicators is measured by the duration, force and probability of the occurrence of the

20
goal–getting good grades. In our example, the time spent for preparation (duration), and
the probability of getting good score from previous experiences are used to measure the
strength of his interest to achieve goal. We only provided the explanation for duration
and probability of the occurrence and not force because force is not relevant to measure
the strength of indicators in our example. The act which terminates a student’s predicted
sequence (preparing for exam will lead to good grades in exams) is called goal-response, in
our example the students’ predicted sequence will be terminated after getting good grades.
When the goal-response is interfered with, due to any intervention, such an interference
at the proper time in a predicted sequence, is called frustration [50]. In our example, if
the student’s laptop is being used by someone else like sibling or roommate then s/he
gets frustrated. Now the student might scream or hate the person who is using his/her
laptop; any such behavior in a goal-response sequence shown towards the interference is
called aggression. According to the frustration aggression hypothesis “aggression is the
primary and characteristic reaction to frustration”.

The hypothesis also states that:

• The greater the strength of the goal-response sequence interfered with, the greater is
the frustration; it could affect the strength of the tendency to respond aggressively
to frustration.

• The greater the amount of interference with the goal-response, the greater would
be the tendency to respond aggressively to frustration.

• The effect of combined frustration can induce stronger aggressive reaction than
individual frustration.

In summary, the severity of frustration is determined either by the amount of the inter-
ference or by the strength of the interference or the added effect of several frustrations
(cumulative).

21
2.3.3 Frustration and Goal-Blockage

In the classic textbook “Introduction to Psychology”, Morgan et al. [115] define frus-
tration as, “the blocking of behavior directed towards a goal”. The major source of
this frustration are environmental factors (a physical obstacle or a person who blocks
us achieving our goals), personal factors (not able to attain the goal due to lack of re-
quired ability), or conflict (not able to attain the goal due to other goals). This theory
also supports Rosenweig’s theory, that frustration can occur due to external or personal
factors.

Spector and Paul (1978) [149], further argue that frustration can occur when pro-
cess of maintaining one’s goal is blocked. Frustration occurs when “both the interference
with goal attainment or goal oriented activity and the interference with goal maintenance”
[149]. If any goal or expected outcome is blocked or stopped, then a frustration instance
occurs. Also, if the process of maintaining the goal is interfered then a frustration in-
stance occurs. The factors that affect the strength of the frustration are the importance
of the blocked goal, degree of interference, and number of interferences. The time and the
distance to the goal when it is blocked also affects the strength of the frustration [149].
That is, if the goal is missed narrowly then the strength of the frustration is more, when
compared to the missing of the goal by a broad margin.

Cognitive psychologists [147], [135], [120], view appraisal as the cause of cognitive
emotions occur due to a person’s perspective and expectation of an event. The theories
of cognitive psychologists state that emotions are related to the student’s experience,
goal, blockage of goal, achieving of the target and so on. We consider the cognitive
psychologists’ theory of emotions, that is emotions can occur due to one’s preferences or
goal match with the outcome of that event, in our frustration definition.

2.3.4 Experimental Research on Frustration

Apart from these theories of frustration, the following attribute of frustration has been
found by experimental research and is not mentioned explicitly in frustration theories.

22
Lazara et al. [98] studied the causes of frustration in computer users. They concluded
that, if a task which is of higher importance to the students involves a large amount of
wasted time, then it is directly proportional to a higher level of frustration. It implies
that if an important goal, for which the student spent a lot of time fails, it leads to a
higher level of frustration. Hence, we should consider the time spent to achieve the goal
while detecting frustration.

2.3.5 Definition of Frustration Used in Our Research

From the above research of frustration, the central and primary factor is that frustration
occurs when the goal is blocked. Therefore, it is necessary, first, to identify the primary
factor of frustration [5], which is, the fact that frustration occurred due to goal-blockage.
In addition to the primary factor of frustration, we consider all the characteristics of
frustration from the various theories and studies described in Sections 2.3.1 to 2.3.4.
However, we do not consider the frustration which occurred due to external interference.
Since our focus is to detect frustration that occurs due to a student’s interaction with the
ITS, we do not consider external factors such as, issues in the device used or connectivity
problems.

The following factors of frustration are considered in our research to model the
student’s frustration.

• Frustration is the blocking of a behavior directed towards a goal [115].

• The distance to the goal is a factor that influences frustration [149].

• Frustration is cumulative in nature [50].

• Time spent to achieve the goal is a factor that influences frustration [98].

• Frustration is considered as a negative emotion, because it interferes with a student’s

desire to attain a goal [149], [50].

23
2.4 Motivation Theory

The goal of affective computing is to detect a learner’s affective state and respond to it.
In the previous sections, we described the theoretical definitions required to detect the
affective states. We respond to the student’s affective state by displaying motivational
messages. Motivational messages motivate the student to continue the task instead of
dropping to frustration or goal-failure. In this section, we describe theories behind moti-
vation. Motivation theories are used to motivate the person to get involved in work or to
continue working. Motive is derived from the Latin word for movement (movere). Moti-
vation psychologists in 1930 to 1950 studied the factors that moved the resting organism
(a living creature) to state of activity. They found that the desire for achievement is the
key factor in motivation theory. In 1960s, cognitive motivational theories were developed
by researching the application of the motivation theory to the event’s outcome (success or
failure) [73]. In this section we briefly describe the motivation theories which dominated
the scientific study of motivation [73] till 1990. The motivation theories from 1990 are
briefed later in this section.

2.4.1 Hull’s Drive Theory

Hull’s drive theory was developed in the period of 1940-1960 and is based on the energy
(drive) required to motivate the organism [81]. Also, it is based on the general law
of learning: if stimulus-response (the action towards an event and the response of that
event – goal-response) ends with a satisfying result, the motivation increases, and if the
stimulus-response ends with an annoying result, the motivation decreases. Hull defines the
strength required to increase the motivation, which decreased due to stimulus-response,
as habit. In other words, a habit is the action required for a organism to move towards the
goal. However, habit can provide the directions required to an action but not the drive.
Hence, Hull created the mathematical relation between drive and habit for motivational
behavior which is reproduced below:

Behaviour = Drive . Habit

24
 Behaviour is considered as multiplication of “Drive” and “Habit”. This is to indi-
cate that “Drive” or “Habit” alone cannot motivate the organism. If there is no energy
(Drive = 0), the organism would not act irrespective of the strength of the Habit. Later,
Hull’s drive theory was researched and its results supported the drive theory [10].

2.4.2 Lewin’s Field Theory

Kurt Lewin’s theory was also developed during the period of 1935-1960, just as Hull’s
theory. Lewin’s theory is based on Gestalt psychology (the psychology which analyses
behavior as a whole and is not determined by the summation of individual elements)
to interpret motivational behavior, which is known as the field theory. The field theory
states that behavior is determined by both the person and the environment involved:

Behaviour = f (P erson, environment)

The motivational force of a person is based on three factors: 1) the person’s in-
tent (need) to complete the task, known as tension (t) 2) The magnitude of the goal
(G), which satisfies the need and 3) the psychological distance of the person from the goal
(e). The mathematical function for the motivational force of person is: F orce = f (t, G)/e

In this function the psychological distance from the goal is inversely proportional
to the motivation force, that is, if the distance to achieve the goal is reduced (approaching
zero), then the motivation to achieve the goal is increased.

2.4.3 Atkinson’s theory of achievement motivation:

Similar to Hull and Lewin, Atkinson also developed the mathematical function for achieve-
ment motivation; however, Atkinson focused on individual differences in motivation.
Atkinson’s theory states that the behavior (tendency) to approach an achievement-related
goal (Tt ) is the product of three factors: the need for achievement or motive for success
(Ms ), the probability that a person will be successful at the task (Ps ) and the incentive
for the success (Is ). The mathematical function is:

25
 Ts = Ms . Ps . Is

Ms , the achievement motive, is developed during the early stages of life and shaped
by child-reading practices. The probability of success, Pi , usually defined in terms of the
difficulty of the task. The value of Pi ranges from 0 to 1. The third factor, Is , the incentive
of success, is inversely related to PI : Is = 1 − PS . This theory is developed in the
period of 1960-1980.

2.4.4 Rotter’s Social Learning Theory:

Rotter’s theory is also based on individual differences in behavior like Atkinson’s theory.
The motivational model by Rotter is based on general expectancy (E) and reinforcement
value (RV), and the relationship of these two factors is:

Behaviour = f (E, RV )

Reinforcement value (RV) is a comparative term and is not clearly mentioned in

the theory [73]. The expectancy (E) of success depends on one’s history of the present
situation and similar circumstances. For example, one’s expectancy of success in a event
depends on the history of success or failure in the same event or the result of similar
events. In a situation which requires one’s skill, the expectancy increases after success
and decreases after failure. This theory was developed in the period of 1960-1990.

2.4.5 Attribution Theory

Attribution theory considers humans as researchers who are trying to understand the
world around them using their own techniques, to reach logical conclusions. Attribution
theory, when applied to motivation, considers the person’s expectation and the response
from the event. Attribution theory was constructed by Heider [77] and subsequently
developed by Weiner [160]. Bernard Weiner’s attribution theory [160] relates emotional
behaviors with academic success and failure (cognitive). In attribution theory, the causes

26
of success and failure related to achievement context are analyzed.
Weiner reports that the reaction of the person is related to the outcome of an
event. That is, a person feels happy if the outcome is successful and frustrated or sad if
the outcome of the event is a failure. This is termed as “outcome dependent-attribution
independent” [160]. The learner’s attribution of success or failure are analyzed in three
sets of characteristics: locus, stability, and controllability.

• Locus refers to the location of the cause, which deals with the cause of success or
failure may be internal or external. Locus determines whether the pride and self-
esteem are altered due to outcome of an event (success or failure). If the learner
attributes the success to internal causes, such as, being well prepared for the exam,
and doing more homework, then it will lead to pride and motivates the learner to
set new goals. Whereas, if the learner attributes the failure to internal causes then
it will diminish the self-esteem. Hence the learner’s failure should be attributed to
the external factors such as “tough questions” and “difficulty in maths”, to motivate
the learner to give effort on future event.

• Stability refers to the learner’s performance in the future. If the learner attributes
the success to stable factors such as “low ability”, then the outcome of the future
event will be the same, given the same environment. If the learner attributes the
failure to the stable factors then the future success is unlikely. If the learner at-
tributes the failure to unstable factors such as “less effort” and “luck” then the
learner’s success in future events will be improved [62].

• Controllability refers to the factors which are controllable by the learner and have
ability to alter them. If the learner failed the task and s/he can control the future
outcome by altering them, such as, improving math-solving ability, spending more
time on homework, it will lead to self-motivation. Whereas, if there is failure at a
task which s/he can not control, it will lead to shame or anger.

The attribution theory implies that the person’s attribution towards the success

27
or failure contributes to the person’s effort on future activity. If the learner attributes
the success to internal, stable and controllable factors then it will lead to pride, and mo-
tivation. If the learner attributes the failure to the internal, stable and non-controllable
factors then it will lead to diminishing the self-esteem, shame and anger. Hence, motivat-
ing the students’ failure by messages which attributes the failure to external or unstable
or controllable factors will help them to set a new goal with self-motivation. This theory
was developed in the period of 1970-1990.

2.4.6 Discussion

In the above theories, Hull’s drive theory and Lewin’s Field theory both explain what
determines motivation using the same factors: need of a person (“drive” in Hull’s and
“tension” in Lewin’s), the goal object, and directional value (“habit” in Hull’s and “psy-
chological distance” in Lewin’s). Later, these factors are not considered in expectancy-
value theories by Atkinson’s and Rotter’s and also in the Attribution theory. Atkinson’s
achievement motivation and Rotter’s social learning theory focus on the individual’s mo-
tivation, success rate, and history. However, these theories addressed the broader goals
of motivation and did not provide suggestions to increase classroom performance.

Based on the review article on the above theories by Graham et al. [73], reports
that each theory had a life span of around 20 years and major contributions to theories
made in those 20 years. The grand theories of Hull, Lewin and Atkinson have not been
used after their life span. Also, the research on Rotter’s social learning theory has been
reduced. Research on the Attribution theory and its application to achievement appears
to be dominant in the theory of motivation [73]. Also, the review paper by Graham in 1991
[72], analyzed the papers published in the Journal of Educational Psychology from 1979-
1988, related to motivation theories. The study reports that, a) there were 66 published
studies in that decade and the primary conceptual framework was the attribution theory,
b) “Attribution theory has proved to be a useful conceptual framework for the study of
motivation in educational contexts” [72].

28
 More recently (1990 onwards) motivation theory has been researched for its ap-
plications. For example, the self-determination theory [63], [127], [42] is an application
of motivation theory in organization. This theory discusses the relevance of work mo-
tivation in organizational behavior. ExpectancyValue theory of achievement motivation
(2000) [162], relates the children’s expectancy of success, ability, and subject task to mo-
tivation. The recent article [106] in 2004 discusses the future of (work) motivation theory
for twenty-first century. The article suggest that the extant theories of work motivations
should be integrated, and the work motivation should consider different aspects in orga-
nization such as employee motivation, incentives, and performance. The primary focus of
research on motivation in the last two decades has been on work motivation applied to
organization.

The theory that is still widely used in education research is attribution theory
(for example, in [15], and [157]). Attribution theory is also used in affective computing,
especially in ITS, to address the students’ affective states (for example, in [91], and [45]).
Hence, we choose the attribution theory in our research to create motivational messages
to address affective states.

2.5 Responding to Frustration

In this section we discuss the different approaches used to respond to frustration in com-
puter based learning environments. In a recent research work by Klein, Moon and Picard
[93], listed the strategies to respond to students’ affective states. These strategies are
developed based on research works on active listening [67] and [118]. The guidelines listed
in Klein et.al’s strategy in [93] to respond to affective states are discussed below.

• The system should provide option to receive feedback from the student for their
affective state. This is to show the student that the system is actively listening
to their emotions. Active listening to students’ emotions has shown to alter their
emotions [118].

29
 • The students’ feedback should be requested immediately whenever the student is
detected frustrated. The feedback request when the student is not frustrated will
be ineffective. To report the affective states, the students’ should have list of option
to choose from. This will provide the option to student to react on what emotion
s/he is undergoing.

• The system should provide feedback messages with empathy, which should make
the student feel that s/he is not alone in that affective state. Also the messages
should convey the student that the emotion s/he undergoing is valid. For example
the student should not feel that only s/he got wrong answers to the question given
by the system or only s/he missed the goal.

The other approaches to respond to affective states includes displaying the messages
using agents [129], [79]. The agents are designed to show empathy, and encourage the
students to continue learning. Also, the research in [123] shows that the positive messages
to address the students’ emotion have helped them to improve their performance.
To create motivational messages, we studied the Dweck’s research [53], [54] on
feedback messages to praise the student’s effort instead of student’s intelligence. In her
research, Dweck conducted a nonverbal IQ test on students and provided one of the three
forms of feedback messages. One-third of the students were praised for their intelligence,
one-third of the students were praised for their effort and remaining students were not
praised for effort or intelligence. After providing the feedback message the students were
given second set of problems which are difficult compared to first set of problems. Later
the students were interviewed to know their view on intelligence. The result shows that
the students who were praised for intelligence believes that the intelligence is fixed and
cannot be improved. The students who were praised for effort believes that intelligence can
be improved by more effort. Also, the students who were praised for effort believes that
failure means low effort and displayed more enjoyment in solving difficult problems. The
Dweck’s research on feedback messages is a seminal work in the research area of guidelines
to create feedback messages, and it had been applied to wide range of educational research

30
(examples are motivating school students [163], encourage girl students in maths and
science [75], and to respond to students’ affective states in computer based learning [44],
[39]).
In our research we adapt all the above approaches to respond to frustration. The
content in our motivational messages are based on attribution theory [160]. Based on
Klein’s guidelines [93], we provide the option to students to reflect their feedback, the
feedback is requested after detecting frustration and feedback messages to show empathy
for students’ affective state. Using the recommendation from [129], [79] Our motivational
messages are displayed using agents who communicate empathy in their messages. Based
on Dweck’s research [54], our motivational messages are constructed to praise the effort
not their intelligence. Our strategies to respond to frustration is detailed in Chapter 6.

2.6 Summary

In this chapter, we have discussed the function of ITS with its general architecture and
recent research areas. After this, we have discussed the affective states in general with
a detailed description of frustration. We have discussed frustration from theoretical and
empirical points of view, and considered definitions from educational psychology and
cognitive psychology. We explained the definition of frustration used in our research and
argued how we arrived at the definition. Later, we have discussed five different motivation
theories in brief with a detailed description of the Attribution theory which has been used
in our research. We also discussed the theoretical guidelines of our approach to respond to
frustration. In the next section we describe the functions of the ITS used in our research.

31
32
 Chapter 3

Research Platform

In this chapter, we describe the Intelligent Tutoring System (ITS) used in our research–
Mindspark. Our approach to detect and address frustration is implemented in Mindspark
and tested in real-time. The details of Mindspark’s log data, adaptive logic, learning
content and user interface are described in this chapter.

3.1 Mindspark

Mindspark is a commercial mathematics ITS developed by Educational Initiatives India

(EI-India)∗ [1], and is used in our research to test our proposed approach. Mindspark
has been incorporated into the school curriculum for different age groups (grades 1 to 10)
of students [150]. Mindspark’s learning content is developed to match the school curric-
ula. Schools can select the learning content based on their curriculum like Central Board
of Secondary Education (CBSE† ), Indian Certificate of Secondary Education (ICSE‡ ) or
Mindspark Recommended. Also, teacher can customize the learning content for Inter-
national General Certificate of Secondary Education (IGCSE§ ). Mindspark is currently
implemented in more than 100 schools and is being used by 80,000 students at an average
of four sessions per week, with each session ranging from 30 to 35 minutes. Mindspark
∗
http://www.ei-india.com/
†
www.cbse.nic.in/
‡
www.icseindia.org/
§
www.cie.org.uk/

33
Figure 3.1: Mindspark screen showing a sample question from class 6 on topic Percentages.
The Mindspark interface contains student name, class, selected topic, a question from the
topic, progress indicator in the topic, number of Sparkies collected in the session, options
to change the topic or end the current session, and an option to provide feedback.

is also available as a home-based tutoring system, for a student to work on mathematics

problems outside the classroom.

Mindspark is a web-based self-learning system, in which students learn mathematics

by answering questions posed by the system. A sample question from Mindspark is shown
in Figure 3.1. Mindspark provides detailed feedback and explanations on receiving the
answer from the student. A sample explanation from Mindspark is shown in Figure 3.2.
The explanation is provided to all questions regardless of the answer provided by the
students. Mindspark adaptation logic selects the questions to be presented based on a
student’s response to the current question and his/her overall performance in the topic,
thereby, allowing the student to move at his/her own pace. On the basis of the answer
from student, whether right or wrong, Mindspark adapts itself and provides relevant
intervention to either strengthen understanding or to offer a higher level of content. If a
student performs poorly in the current topic (for example, the student does not answer
sufficient number of questions correctly), s/he will be moved to a lower complexity level
in the same topic.

Mindspark’s architecture is shown in Figure 3.3. Mindspark adapts the learning

content from the learning content database. The adaptation is done based on the data
from the student model which is created using the student’s interaction with the system.

34
 Figure 3.2: A sample Mindspark screen showing the correct answer with explanation

The student’s interactions with the system are stored in log files. In this section we
describe the student interface, the log files, the learning content and the adaptive logic of
Mindspark.

3.1.1 Student Interface

Mindspark is a web-based learning system which can be accessed using Internet browsers
like Mozilla Firefox or Google Chrome. Hence, Mindspark can be accessed using a laptop,
a desktop computer or a tablet. In Mindspark, after successful registration, each student
gets a login ID. The student logs in with his/her unique ID and password. The teacher
assigns the topics to the students based on their curriculum.

The user interface, as shown in Figure 3.1, contains a progress indicator at the top
the screen. The progress indicator shows the student’s progress in the selected topic. The
student’s name, class and topic name are displayed above the question. The students have
the option of changing the topic or ending the session at anytime during their interaction
with Mindspark. A optional emote tool bar is integrated in Mindspark interface (bottom
left in the Figure 3.1 indicated as Rate) to enable the students to provide feedback. The

35
 Log Files

User ID,
Time to answer,
Result, etc.

Mindspark Learning
Adaptation Content DB
Engine
Math Topics,
Teacher topics,
Questions,
Registered Explanations etc.
Student

Student Model

Age, Grade,
Performance,
Curriculum,
etc.

Figure 3.3: Mindspark Architecture - Intelligent Tutoring System

options in the emote toolbar are shown in Figure 3.4. The emote toolbar is used to collect
the student’s emotion during the Mindspark session. The options in the emote toolbar are
“confused”, “bored”, “excited”, “like”, “dislike” and an option to receive the student’s
comments.

Apart from the emote toolbar, based on the student’s response to the question,
students are allowed to provide feedback on questions, as shown in Figure 3.5. The
student can mark whether the question is a repeat question or not. If the student’s
response to the question is incorrect then student can report whether Mindspark marked
his/her response as correct or not. Regardless of the answer provided by the student,
the explanation of the correct answer will be provided in the Mindspark interface after
submitting the answer. The sample explanation is shown in Figure 3.5.

Similar to the student’ interface, Mindspark has a teacher interface for school teach-
ers. Teachers can select and a assign the topic for the students based on his/her class
curriculum. Also, the teacher can track: a) each student’s progress session-wise or topic-

36
Figure 3.4: The options in emote toolbar. The options are “Confused”, “Bored”, “Ex-
cited”, “Like”, “Dislike” and “comment” to receive the student’s feedback

Figure 3.5: The options for the student’s to enter their feedback when the question s/he
answered is incorrect

wise, b) the concept that elicits a low performance from the class, or c) the misconceptions
of the students.

37
3.1.2 Learning Content Database

Mindspark’s learning content is stored in the learning content database and are hierar-
chically arranged in the following order:

Topics − > Teacher Topics − > Clusters − > Sub difficulty Level

• Mathematics is divided into several topics. The topics in Mindspark are: Algebra,
Fractions and Decimals, Geometry, Measurement, Mensuration, Number Theory,
Numbers, Percentages & Commercial Math, Problem Solving, Ratio-Proportion,
Real Numbers, Statistics & Data Analysis and Trigonometry.

• Each topic is divided into Teacher Topics (TT). TT contains subtopics in each
topic. For example, the subtopics of “Real numbers” are: Squares and square roots,
Logarithms, Cubes and cube roots, Integers, Rational numbers, Real numbers and
Exponents and roots.

• Each TT is divided into clusters (learning units). A cluster contains the learning
content in the form of questions. Each cluster has 20 to 40 questions. The questions
are arranged in progressively increasing levels of complexity.

• The questions in the clusters are divided into sub difficulty levels (SDL). For ex-
ample, if the cluster is “Squares” the first SDL is “ the concept of the square”, the
second SDL is “Square of a small number and single digit numbers” and so on. Each
SDL contains three questions with the same level of difficulty.

In Mindspark, questions are designed to identify the causes of error based on the
options students select. The options provided to the students in each Mindspark ques-
tion is created to diagnose the student’s misconception, lack of concept understanding
or the common error in that question. The student’s misconceptions are addressed us-
ing special learning content called a “remedial cluster”. The remedial cluster contains
examples, animated displays, and interactive multimedia to explain the concept and to
resolve the misconception. Apart from the questions and answers, Mindspark’s learning

38
database contains feedback with explanations to each question. Explanations are targeted
at specific mistakes which are diagnosed from the response provided by the student.

Mindspark’s learning content is designed to integrate with the curriculum of a

school. So, Mindspark can be configured to match the curriculum each school uses.
Mindspark allows teachers to customize the topic according to how the topic is being
taught in the school. Teachers can select standard curricula like CBSE, ICSE, Mindspark
Recommended or can customize for IGCSE as per the school curriculum. The teachers
can then select the topic and subtopic to be activated and customize it according to
their requirements. Teachers can view the sample questions from each subtopic before
activating it. The activated topic can be assigned as homework or revision, based on the
students’ performance.

3.1.3 Browser Log and Student Model

The browser log file records the students’ interactions with Mindspark. The following
details are recorded in the log file: TT (Teacher Topic) attempt ID, user ID, TT code,
TT attempt number, attempt time, class level result (values: success, failure, aborted or
null), individual result, cluster ID, failed clusters, last modified date, TT cluster status,
cluster attempt ID, cluster code, cluster attempt number, result, start session ID, end
session ID, serial number, attempted date, question number, answer, time in seconds to
answer the question, result (correct or wrong), and time taken to read the explanation.
The log files are stored as tables in a cloud-based server. The database is archived at the
end of school year (May month).

The student information is captured from the browser log and stored in a separate
database called the student model. The student model contains the student’s performance
in each topic, grade, school curriculum, the student’s overall progress and the personal
details of the students. Using the student model and data stored in the database, teachers
can see the students’ learning patterns in the teacher interface. The teacher can have
detailed insights into the students’ understanding and their misconceptions, right after

39
the topic is taught in the class. It allows the teachers to address the learning gaps of the
class. The teacher can learn exactly which concept a student is struggling with, allowing
them to offer personalized intervention.

3.1.4 Adaptation Engine

Mindspark delivers the learning content based on the student’s performance in the current
question, performance in the current topic and the student’s grade. In Mindspark, adap-
tation happens at two levels–the cluster level and the inside cluster level. This Mindspark
adaptation logic described in this section is based on 2011 Jan. The adaptation logic of
Mindspark is refined later.

3.1.4.1 Cluster level adaptation logic

The students’ performance are monitored in Mindspark by analyzing the log files. Progress
in each topic is measured and used to determine whether the student has learned the
subtopic or not. The adaption logic at the cluster level is given below:

• When a student gets 25% or more questions incorrect in a cluster, it is considered

as a cluster failure. On the first failure of the cluster, Mindspark allows the student
to repeat the cluster.

• On the second failure of the cluster, the cause of the student’s failure is identified
based on student’s performance in the cluster. If the student gave the wrong answers
due to lack of understanding of fundamental concepts, adaptive logic takes the
student to the cluster that was immediately before the current one. If it is the first
cluster in the topic, the student is taken to a lower level of the same topic. If the
student gave the wrong answers due to misconception, then the remedial cluster is
given to the student. Remedial cluster is a cluster which explains the concept with
more hints and explanation. This cluster teaches the concept of using questions like
other clusters. Remedial clusters are designed to resolve the misconception based
on cognitive dissonance theory [19]. In the cognitive dissonance method, the system

40
 conflicts the student’s prior understanding (misconception), then gives the correct
explanation to resolve the misconception.

• On the third failure, the student is given five clusters prior to the current cluster,
irrespective of the position of the cluster and the class level.

• After the fourth failure, the cluster is highlighted in the teacher interface. The
student is allowed to proceed to the next cluster. The teacher can help the student
to understand the concept and clear any misconceptions.

• Success on a cluster wipes out past failures.

If a student on the remedial cluster fails twice, irrespective of whether the remedial cluster
is from the same TT or some other TT, it will take the student to one cluster lower in the
flow. The logic of the third and the fourth failure does not apply to the remedial cluster.

3.1.4.2 Adaptation within a cluster

In Mindspark, when a student answers a question incorrectly, the system provides a de-
tailed explanation to the student, on the concept involved. The student is then presented
another question of the same difficulty level (DL) from the same learning unit, to give
the student another opportunity to demonstrate his/her understanding. The adaptation
logic is given below:

• If a student fails to answer one question in the SDL, the next question in the same
SDL is given. After attempting three questions, the student is moved to the next
SDL, even if the student fails to answer all questions in the SDL.

• If students attempted 50% of questions in the cluster and the total mark percentage
is less than 60% in the 50% of the questions attempted then it is considered as
a cluster failure. Mindspark will not allow the student to complete the remaining
questions.

41
 Mindspark’s adaptive logic diagnoses the learning gaps in each student. Then
Mindspark helps the student to clear the gaps by providing the required learning content.
Also the adaptive logic allows the students to learn beyond their grade level.

3.1.5 Sparkies

Mindspark rewards the students for their performance. Sparkies are the reward points in
Mindspark and they are displayed in the student interface. The Sparkie is introduced to
the student when s/he logs into Mindspark and the rules of earning Sparkies are described
as shown in Figure 3.6.

Figure 3.6: Sparkie is introduced to the students and the rules of earning Sparkies are
described

Sparkies act as motivational factors for the students in Mindspark. If the student
answers three questions consecutively correct, s/he receives a Sparkie (extra motivational
points). If the student answers five questions consecutively correct, s/he receives a chal-
lenging question which is tougher than normal questions and can be from any topic that
the student has covered. The sample challenge question from topic percentages is shown
in Figure 3.7. If the student answers the challenging question correctly, s/he receives five
Sparkies. The Mindspark interface in the figures above, show the number of Sparkies the
student collected in that session. Every week, students with highest number of Sparkies

42
are identified and their names are published on the Mindspark’s login page ¶ .

Figure 3.7: Challenge Question from Topic Percentages

3.2 Discussion

The features of Mindspark, such as, feedback with explanation, adapting questions based
on the students response and logging of the student’s response, time taken to answer and
other common data are available in most of ITS. Also Mindspark is used as part of regular
school curriculum, hence:

1. It has data of more than 80,000 students

2. It is an opportunity to implement our approach and test it in the real world.

Due to these reasons we chose Mindspark as our research platform. In this chapter,
we have described the functions of the research platform used in our research.

¶
http://www.mindspark.in/login

43
44
 Chapter 4

Related Work of Affective Computing

In this chapter we review related work on detecting and responding to the affective states
of the users. First, we start with existing approaches to detect affective states in educa-
tional systems like games, Intelligent Tutoring Systems (ITS) and search engines. Then
we discuss in detail, the process by which frustration is detected and responded using data
from the students’ interaction with the ITS. We conclude the chapter with the motivations
for our solution approach.

4.1 Affective States Detection

Adapting the learning content based on the user’s affective states is called “affective
computing.” In affective computing, systems detect and respond to users’ affective states
[126]. Detecting the affective states has been the focus of the affective computing research
area for the past decade [26]. In this section, we first discuss the different approaches used
to detect the affective states in various systems such as games, ITS and search engines.
Then we discuss existing research to detect frustration using the students’ interaction
with ITS or educational games.

The students’ affective states can be detected by analyzing the data from:

• Facial expressions.

• Body language and postures.

45
 • The physiological sensors.

• Voice or text input.

• Interactions with the system.

The different approaches for affect detection described in this section is based on
the recent (2010) review paper on affect detection [26]. This paper discusses five studies
using facial expressions, two studies using body language and postures, twelve studies
using data from the physiological signals and sixteen studies using voice and text input.
Apart from the approaches given in the review paper [26], we included the approach to
detect affect states using student’s interaction with the system. We also included the
research papers published from the year 2010-2013 for each approach. We conclude this
section with comparative analysis of different approaches and applicability to our research
work.

Facial Expressions

Affective states can be inferred from the users’ facial expressions during their interaction
with the system (ITS, Games, and the like). The users’ facial expressions during their
interaction with the system are captured using a video camera or web camera. Later
the facial expressions are coded as action units (expressions in different parts of our
face, for example, inner brow rise, nose wrinkle, chin rise, jaw drop, and dimpling) [14].
Based on Ekman’s Facial Action Coding System (FACS) [57], emotions are identified
from the action units [110]. However, manual coding of facial expressions is a time-
consuming process [51], and cannot be used in real-time. Hence, automated facial coding
systems are used in affective computing. The videos of facial expressions are analyzed
automatically and the affective states are detected [58]. In a research paper [58], the
mental states like “agreeing”, “concentration”, “disagreeing”, “interested’ ’, “unsure”,
and “thinking” are detected using the head movements and facial expressions. The facial
expressions considered to detect the mental states are “eyebrow rise”, “lip pull”, “lip

46
pucker”, “lips part”, “jaw drop”, “teeth.” Dynamic bayesian networks (DBN) was used
to model the head movement and facial expressions to mental states. This approach
reports an overall detection accuracy of 77.4 % [58]. The research study in [14] and [153]
classifies facial action units instead of emotions. The facial actions units like “lip pull”,
“eyebrow rise” are classified using non-linear classifiers, like the support vector machine
(SVM), Adaboost and the hidden Markov model (HMM); these approaches result in
higher detection accuracy rates–over 90 %. The facial action units are used to classify the
basic emotions such as “anger”, “fear”, and “happy.”

Body Postures

Similar to the facial expressions, body language and postures are also used to detect the
affective states. The research in [36] reports the correlation of emotions like anger and fear
with body movements, body rotations in angles like chest, abdomen, head bend, shoulder
and others. The automated approaches use the signal from pressure sensitive chairs
(pressure pads on the chair) to identify the user’s positions of sitting like leaning forward
or back, sitting upright, or sitting on the edge of the seat. These positions of sitting are
used to classify the user’s interest level (high or low) using the HMM classifier [116] or the
user’s emotions using the non-linear classifiers [49]. These approaches report detection
accuracy in the range of 71 % to 80 %. The users’ postures and body motion are analyzed
automatically to detect the users’ engagement with game companion (robot) [141]. The
body motions like “body lean angle” and “contraction index (degree of contraction and
expansion of the upper body)” are considered to detect the user’s engagement. The
classifier in Weka [108] are used in this research and this approach reports 82% detection
accuracy.

Physiological Sensors

Affective states can be detected by analyzing data from the physical sensors such as the
blue eye camera [46], posture analysis seat [8], the pressure sensitive mouse and skin

47
conductance bracelet [164]. The data from the sensors are compared with the affective
states identified using independent methods to detect the affective states. In Wayang
Outpost, an adaptive tutoring system for geometry [8], a linear regression algorithm was
used to detect affective states using the data from sensors. This study reports the corre-
lation coefficient of multiple features, R = 0.72 for frustration. More recently, the data
from physiological sensors such as electrocardiogram (ECG) [18], facial electromyography
(EMG) [83], galvanic skin response (GSR) [18], electrodermal [92] and respiration changes
[159] are being considered in affective computing. The data from sensors are modeled us-
ing classifiers, to detect the affective states. The classifiers used in this approach are
k-nearest neighbor (kNN), linear discriminant function (LDF) [159], SVM and multilayer
perceptron (MLP). These approaches report the detection accuracy as being in the range
of 80-92 % [159], [105] and [17].

Voice and Text Input

The user’s input to the system, such as voice and text data, are used to detect their
affective states [167]. The following features are considered in voice analysis:

• Voice level, voice pitch and phrase.

• Acoustic-Prosodic features like the speaking rate, the time taken for each speech
turn and the amount of silence in each turn.

• Lexical items in a user’s response or the system’s response.

These voice features are used to classify generic emotions such as anger, fear [100], or to
classify emotions at an abstract level such as positive, negative, emotion, non-emotion
[104].

Similarly, from the user’s text input, features like first sentence, direct speech in
sentence, special punctuation (! and ?), complete upper-case word and sentence length
in words are used to classify the users’ emotions. Classifiers used in this approach are

48
linear classifier [4], Naive Bayes classifier [40], SVM [103] and these approaches report a
detection rate of 70-80%.

Interactions with the System - Log Data

The user’s interaction with the system, such as response time, response to the questions,
questions difficulty level, number of hints used, number of wrong answers in a session, and
goals achieved are stored in the log file. These data from the log the files are analyzed
to detect the affective states. The features (data from the log files) are chosen based
on the correlation, with the affective state measured using an independent method, or
theoretical definitions of the affective states under consideration. The selected features
are then applied to the classification algorithms to classify the affective states. The
classifiers used in this approach are linear regressions classifier, Naive Bayes, decision
tree, support vector machine and rule-based classifier. The research work in [61], models
the user frustration based on the data from the search engine and reports the accuracy
of 87% [61].

Multimodel Approach

In the multimodel approach, the researchers implement in a system, two or more of the
approaches mentioned above and analyze the data to detect affective states. The general
approach is to combine the data from the approaches mentioned above with the data from
the physiological senors and create a model to detect affective states. There is a lot of
research interest in combining data from different channels like facial expressions, body
gestures and the log file to detect the users’ affective states [89], [12]. The data from the
posture sensor, log data, facial expressions and head gestures are combined and used to
detect the affective states in [89], [2]. These studies report a higher detection accuracy
compared to the detection of affective states using individual channels [89], [121].

49
Discussion

A comparative summary of the various approaches discussed so far to detect affective

states is shown in Table 4.1. The comparison is based on factors such as degree of
intrusion, cost, detection accuracy, and scalability. These factors are considered based
on application of our research work to large classrooms. The factor “degree of intrusion”
considers whether any external sensor is attached to the user’s body or not. The factor
“cost” considers the additional cost required to detect the users’ affective states: Low -
no extra cost, Medium - Less and one-time cost, High - relatively high cost and recurring.
The factor “scalability” considers the feasibility of implementing the approach to a large
number of students.

Table 4.1: Synthesis of Different Approaches to Detect Affect States

Data Used to Intrusive Cost Detection Accu- Scalability

Detect Affective racy
States
Facial Expressions No Medium Medium (70–80%) High
[58], [14], & [153]
Body Postures No Medium Medium (70–80%) Medium
[116], [49]
Physiological Sensors Yes High High (> 80%) [159], Low
[105] & [17]
Voice and Text Input No Low Medium (70–80%) High
[4], [40] & [103]
User Interactions No Low High (> 80%) [61] High

In the approaches mentioned above, detecting affective state using physiological

sensors is the best approach based on detection accuracy. However, implementing these
approaches requires very high cost due to the sensors used; most of sensors require a large
space to record the body signals from the user. Hence, most of the research studies in
this approach are done in laboratory settings. Later, by invention of hand-held sensors
like the skin conductance sensor, and pressure sensitive mouse, these approaches can be
implemented in a real-world scenario. However, these sensors are intrusive to the users
while they interact with the system. Moreover, the scalability of this approach to large

50
class is very low due the cost and maintenance required. The facial expressions are tested
using the facial expression videos from adults (> 18 years) [14], [153] and the user must
face the video camera while working with the system. Also, this approach results com-
paratively less detection accuracy and has not been tested in the real-world environment.
Voice and text data analysis is only applicable to the ITS which considers voice input or
long text answers from the users. However, the Math ITSs, which is implemented and
tested in schools (Active Math [112], Cognitive tutor [131], [6], Mathematics Tutor [151],
[16], and PAT Algebra Tutor [130], [94]), neither uses voice nor long text as a input. Also,
the research on detecting the affective state using voice and text data, reported a medium
detection accuracy of 70 to 80 %. In these approaches, detecting the affective states by
analyzing the user interactions is feasible in a large-scale, real-world scenario [133] and is
non-intrusive to the users. In summary:

• Identifying affective states using the sensor signals is possible in laboratory set-
tings, but difficult to implement at a large scale. Also, the physiological sensors are
intrusive to the users.

• Facial analysis methods use a web-cam to analyze the facial expressions of the users.
In the real-world scenario, keeping the camera in the right position, and expecting
users to face the camera all the time is not feasible.

• Voice and text analysis methods can only be used in the ITS that considers voice
and subjective answers as an input from the users.

In affective states, detecting and responding to the negative affective states are
important as it might lead the student to drop out of learning [88]. Moreover, frustration
is a cause of student disengagement and eventually leads to attrition [88]. Hence, in our
research we focus on approaches that use student’s interaction with the system to detect
the students’ frustration.

51
4.2 Detecting Frustration from the Student’s
Interactions with the System

In this section, we review the existing approaches related to our research on modeling
and detecting frustration from the ITS log data. We review the systems which identify
the students’ affective states from the log files of the students’ interactions as well as
systems which do so by modeling the affective states. First we describe three systems–
AutoTutor, a programming lab, and Crystal Island–which detect frustration based on
data from the log file. Some of these systems also use biometric data, but we will focus
only on detection approaches using log data. Later we discuss two systems–Crystal Island,
and Prime Climb–which detects frustration by modeling the students goal and learning
pattern.

AutoTutor [48] is a dialogue-based tutoring system. In AutoTutor, the students’

affective states are detected based on the features from the log data. The affective states
detected are frustration, boredom, flow and confusion. The features are identified based
on the features from the log file, such as, response time, number of characters in a stu-
dent’s response, number of words in student’s response, change in student’s score and
tutor feedback to a student’s response. The data from 28 college students’ interaction
with AutoTutor were considered for the analysis. Large set of features were first identified
from log data and then reduced from 17 to 11 by applying correlation analysis with affec-
tive states identified, using an independent method; the features which are significantly
correlated were selected. To avoid redundancy among the selected features, dimension-
ality reduction techniques such as principle component analysis and linear discriminant
analysis were used. The selected features were applied to a data-mining software Weka
[108], to detect the affective states. The classifiers considered to detect affective states
from Weka are:

• Bayesian classifiers: Naive Bayes and Naive Bayes Updatable.

• Functions: Logistic Regression, Multilayer Perceptron, and Support Vector Ma-

52
 chines.

• Instance based techniques: Nearest Neighbor, K*, Locally Weighted Learning.

• Meta classification schemes: AdaBoost, Bagging Predictors, Additive Logistic

Regression.

• Trees: C4.5 Decision Trees, Logistic Model Trees, REP Tree.

• Rules: Decision Tables, Nearest Neighbor Generalization, PART.

Independent methods used in this research to detect affective states and validate
the results were self reporting, peer reporting and human observation of students’ facial
expressions. The reliability of human observers was measured in mutual agreement of
affective state and the kappa value for agreement is K=0.36. Affective states were de-
tected for every 20-second interval and the results are compared with the affective states
from independent method. This study reports individual detection accuracy of boredom,
confusion, flow, and frustration, when compared with neutral affect are 69%, 68%, 71%,
and 78%, respectively. The AdaBoost classifier provided the best detection accuracy for
boredom and confusion when compared to neutral. Simple logistic regression and C4.5
decision trees provided the best detection accuracy for flow and frustration respectively
when compared to neutral affect. The features response time, tutor feedback, and change
in student’s score were correlated with the affective state frustration.
Crystal Island [111] is a task-oriented learning environment test-bed, featuring a
science mystery in the domain of genetics. In Crystal Island, the students’ affective state
were predicted early based on the features from the log file and the data from physiolog-
ical sensors. The affective states predicted in this system was frustration. The features
considered to predict frustration were temporal features (such as time spent in a current
location, and time spent after), locational features (such as current location, and past
location), intentional features (such as goals being attempted, goals achieved and rate of
goal achieved), and physiological response (such as blood volume and heart rate) from
biofeedback apparatus attached to student hand. The data from 36 students’ interaction

53
with Crystal Island and biofeedback apparatus were considered for the analysis. This
systems used two n-gram affect recognition models to early predict the frustration, a un-
igram model and bigram model. The features selected using n-gram models were applied
to classifiers such as Naive Bayes, decision tree, and support vector machine to model
frustration. The data-mining software, Weka [108] was used to predict frustration. Inde-
pendent method used in this research to detect affective state was student self reporting.
The student’s were asked to report their affective states periodically (75 seconds) in the
“self-report emotion dialog” box. The students were asked to select the emotion from set
of six emotions; excitement, fear, frustration, happiness, relaxation and sadness. The best
reported performance of this system [111] is an accuracy of 88.8%, precision of 88.7% and
recall of 88.9% for predicting frustration using decision tree.

In a programming lab, in [132], the students’ affective state in computer program-

ming exercises across different labs is detected. The affective state detected in this system
was frustration. The features considered to detect frustration were information from com-
piler data such as average time between compilations, total number of compilations, total
number of errors and Error Quotient (EQ) construct. The error quotient construct is
developed based on how students cope with syntax errors [86], this includes the number
of pairs of compilations in error, number of pairs of compilations with the same error,
number of pairs of consecutive compilations with the same edit location, and number of
pairs of consecutive compilations with the same edit location. The data from ten students
across five lab sections were collected for the analysis. The data mining tool Weka [108]
was used to generate linear regression model to detect student average frustration. Inde-
pendent method used in this research to detect affective states and validate the results
was human observation of students’ facial expressions. The reliability of human observers
was measured using Cohen’s kappa (k=0.65). Affective states were detected for every 20
second interval and the results were compared with the affective states from independent
method. The study reports poor correlation coefficient r (r in the range of -0.3472 to
0.0136). The correlation coefficient, r reported across all five labs is r = 0.3178 [132]. The

54
features which correlated with frustration are consecutive pairs of compilations with the
same edit location, consecutive pairs of compilations with the same error, average time
between compilations, and total errors.

In the above three systems, the feature set is constructed from the student interac-
tion with the system (log data). In contrast, the following systems detect frustration by
modeling the student’s goal and the learning pattern, based on the theoretical definitions
of frustration.

Affective state modeling in Crystal Island [139], creates a Dynamic Bayesian Net-
work (DBN) model to capture the students’ affective states. The affective states detected
in this system were anxious, bored, confused, curious, excited, focused, and frustrated.
The features considered to model students affective states are personal attributes of stu-
dents which are identified from students’ scores and personal surveys prior to interaction
with system, observable environment variables such as goals completes, books viewed and
successful tests, and appraisal variables which are student’s cognitive appraisals such as
learning focus and performance focus. The data from 260 students’ interactions with
Crystal Island was considered to model the affective states. Naive Bayes, Bayes net and
dynamic Bayesian net were used to model the affective states. The independent method
used in this research to validate the results is self reporting. The student’s were asked to
self-report their current mood in the dialog box for every seven minutes. The students
were asked to select the emotion from set of seven emotions which are anxious, bored,
confused, curious, excited, focused, and frustrated. The reported detection accuracy of
all affective states is 72.6% using DBN. The individual detection accuracy reported for
anxious, bored, confused, curious, excited, focused, and frustrated are 2%, 18%, 32%,
38%, 19%, 52% and 28%, respectively. The DBN for individual affect states were not
discussed in this research study.

The Prime Climb is an educational game designed to help 6th and 7th grade stu-
dents to practice number factorization. In Prime Climb students’ emotions as described in
OCC theory [120] were modeled using Dynamic Decision Network (DDN). The emotions

55
from OCC theory considered in this system were joy, distress, admiration and reproach.
The features considered to model the emotions were student goals (Have Fun, Avoid
Falling, Beat Partner, Learn Math, and Succeed By Myself) captured using post-game
questionnaire, personality types of students such as agreeableness which were identified
using personality test, and interaction patterns such as move quickly, fall often, and ask
advice often which are captured from log data. Sixty-six 6th and 7th grade students inter-
action with Prime Climb and data collected from survey questionnaire were considered to
model the emotions. Dynamic Decision Network was used to model the emotions. The
independent method used in this research to validate the results is self reporting. The
students were asked to self-report their feeling about game and feeling about the agent.
The students were asked to select a feeling from set of five options which are Very Bad,
Bad, Neutral, Good, and Very Good. Self reporting dialog box was present permanently
on the side of the Prime Climb game window. To collect more data, the self-report dialog
box was designed to pop-up if the student was reporting his/her feeling for long time or
the DBN model has detected a change in emotions. The emotions towards the self are
joy and distress and the emotions towards the agent are admiration and reproach. The
DBN detected the emotions based on whether the goals of the students were satisfied or
not. The system [33] reports that the accuracy in detection of joy is 69%, distress is 70%,
reproach is 47%, and admiration is 66%. The student feels joy when he achieves the goals
and distress when he was not able to achieve the goal.

4.2.1 Discussion

Table 4.2 summarizes the research done in identifying frustration using the log data. The
other affective states such as boredom, confusion were not detected in all the system
described in this section systems hence we are reporting not them in this discussion.
However, we note that the detection accuracy of other affective states are relatively equal
to detection accuracy of frustration.

In the research studies mentioned above, AutoTutor, Crystal Island and Program-

56
Table 4.2: Research Works, that Identify Frustration Using the Data from Student Log
File, with Number of Features, Detection Accuracy and Classifiers used

Ref ITS/Game Features used Method of Detection Classifiers

Num- used selecting the Accuracy used
ber feature
[48] AutoTutor Data from Correlation 78% 17 classifier like
students’ inter- analysis NB, DT from
action Weka[108]
[111] Crystal Island Data from All features 88.8% NB, SVM, DT
students’ in-
teraction and
Physiological
senors
[132] Introductory Data from Correlation Regression Linear regres-
Programming students’ inter- analysis coefficient sion model
Course Lab action r=0.3168
[139] Crystal Island Students’ learn- All features 28% DBN
ing pattern and
data from ques-
tionnaires
[33] Prime Climb Students’ learn- All features For joy = DDN
ing pattern and 69% and for
data from ques- distress =
tionnaires 70%$
NB- Nave Bayes, SVM- Support Vector Machine, DT - Decision Tree, DBN - Dynamic
Bayesian Network, DDN - Dynamic Decision Network, $ = this system was not detecting
frustration

ming Lab detect the students’ frustration based on the students’ interaction with the
system (log file). In these systems, frustration is detected using a combination of corre-
lation analysis and theory. These three systems used a machine learning software called
Weka [108], to create the model to detect affective states. The detection accuracy using
the log data in these systems (AutoTutor [48] and Crystal Island [111]) is in the range of
77% to 88%. In the approach which models the affective states detects frustration and
emotions based on the students’ interaction pattern with the system and their goals. The
reported detection accuracy for frustration [139] = 28% and for emotions [33] in the range
of 47% to 70% which is comparatively lower than the systems which detects frustration
using data from students interactions.

57
 We note that in AutoTutor, Crystal Island and Programming Lab the frustration
was detected in a fixed interval. These systems were not detecting frustration when the
student’s goal failed or other learning patterns. The interval where the affective state
was detected, the student might be reading or doing calculations for his goal and not
necessarily got response for his goal. However, Prime Climb system detects the students’
emotion based on their goal and learning pattern and independent method to collect
emotion is initiated for validation. In Prime Climb, theoretical definitions were used to
model the affective states hence these systems capture not only the affective state but also
the reason why the student is in that state. The reason identified by the system helps to
respond to user’s affective state based on the reasons for it. In our research, we focus on
creating a model which has higher detection accuracy and also provides the reason why
the student is frustrated.

4.3 Responding to Affective States

In this section, we focus on the systems (ITS and Educational games) which detect and
respond to the students’ affective states while they interact with the system. Responding
to affective states in traditional teaching methods are not considered in this section. There
are fewer research studies in responding to affective states than in detecting affective states
in systems like educational games and ITS. We describe three systems in this section:
affect-support computer games [93], Scooter the Tutor [133] and Wayang Outpost [164].

The affect-support computer game [93] induces frustration by freezing the screen
when the students play the game. This system provides three types of response–ignore
students’ frustration, collect feedback from students, and provide messages. “Ignore stu-
dents’ frustration” response provides no motivational messages and does not collect feed-
back from the users. “Collect feedback” response collects the student feedback in terms
of like how they feel and does not provide any motivational messages. In the third re-
sponse, the system provides feedback messages and sympathy messages whenever the user
reports frustration. The impact of these system responses provided to the students was

58
analyzed using the data from 71 students. The system reports that in general students
felt significantly less frustrated when playing the game without freezing than the game
playing with freezing. The students who have got feedback message and empathy mes-
sages played the game significantly more time than the students’ who got no messages.
The study concludes that, any system can alleviate negative emotions by using empathy
messages even if there is a source of negative emotions in the system.

Scooter the Tutor is an ITS to teach how to create and interpret scatter plots of
data [133]. This system detects when the students game the system, using the student’s
interaction with the system. If the student games the system, the agent (scooter) will
show empathy towards the student with motivational messages like “student should work
in order to learn.” If the student does not game the system, the scooter will show the
happy expressions and occasionally provide positive messages. This system was tested
using 124 students. The students were divided into control and experimental groups. The
students’ affective states like boredom, frustration and confusion were observed using
human observers while they interacted with the ITS. This system reported no significant
difference between the control and experimental groups in the observed affective states.
However, when considering only the number of frustration instances, the control group felt
more number of frustration instances than experimental group. In general, the students
reported that they liked the agent.

Wayang Outpost [164], is an interactive math ITS used to teach math to school
students. This system used physiological sensors to detect affective states. The data from
the sensors were applied to data-mining tools to detect the affective states. The affective
states considered in this research were confidence, excitement, boredom, focus, frustra-
tion, and anxiety. Based on the detected affective states, this system provides delayed or
immediate feedback. The agent reflects the student’s emotion when it detects the affective
state and motivational messages were provided. The motivational messages were created
based on Dweck’s [54] motivational message theory–praise the effort instead of the re-
sult. The preliminary analysis of this system is tested using 34 students’ interaction with

59
Wayang Outpost. This system reported that the students change their engagement (from
low to high) with the system based the interventions such as tips to read the problem and
hint carefully, performance graph of last fix questions and asking student to think about
the problem.

4.3.1 Discussion

Table 4.3 summarizes the research done in responding to affective states.

Table 4.3: Related Research Works to Respond to Student’s Affective States along with
the Theories used, Experiment Method and Results

Ref Num- ITS/Game Theory used to Experiment Method Results

ber used respond to frus-
tration
[93] Affect- Active listening,
Factorial study, 2 On an average the affect
Support emotional feed-(level of frustration) support group played more
computer back, sympathy x 3 (interactive de- minutes compared to non-
game statement [118]sign), N = 71. Self affect support group.
reporting using ques-
tionnaire
[133] Scooter the Agents were Control-experiments Reduction in frustration
Tutor given emotions group study. N = 60. instances. There is no
Human observation significant difference in
observed affect between
control and experimental
group.
[164] Wayang Agent to reflect N = 34, physiological Initial studies results that
Outpost student’s affec- sensor data to detect students change their be-
tive states and affective states havior based on digital in-
messages based terventions
on Dweck’s
messages [53],
[54]
N = Number of participants

The systems discussed in this section respond to the student’s frustration by show-
ing motivational messages and reflecting their emotions using agents. Three different
approaches (self reporting, human observation and physiological sensors) are used to de-
tect affective states in these systems. These systems detect only the affective state not

60
its reasons; hence, they provide motivational messages which respond to frustration in
general and may not address the specific cause which induces frustration. These systems
report that responding to frustration positively influences the student’s learning–in areas
such as increased time of playing the game, fewer frustration instances or change in stu-
dent engagement with the system. Our research focus is to respond to the reasons for
frustration using the motivational messages.

4.4 Motivation for Our Work

In this chapter we described our literature survey on detecting and responding to the
affective states of the users. First we described the existing approaches to detect affective
states in educational systems like Games, Intelligent Tutoring Systems (ITS) and search
engines. Based on our synthesis we focused our literature survey on detecting the affective
states by analyzing the log data, since it is a feasible approach in a large-scale, real-world
scenario and non-intrusive to the students. We then discussed the systems which detect
frustration based on the users’ interaction with the system (log file). The existing systems
predict frustration at different intervals and the reasons for the frustration are not ex-
plored. The system which predicts frustration based on theoretical definitions performed
poorly compared to the system which detects frustration using data from log file. Hence,
their is a need for a frustration model with higher detection accuracy and also with the
reason why the student is frustrated. Later, we discussed the systems which respond
to the students’ frustration by motivational messages and reflecting students’ affective
states using agents. The results reported in these systems conclude that the systems
which respond to frustration by motivational messages positively influences the students’
interaction with the system. However, these systems detect only the affective state not its
reasons; hence, they provide motivational messages which respond to frustration in gen-
eral and may not address the specific cause which induces frustration. Hence, we require
a system which address the students’ frustration.

In our research, we design the model to detect frustration based on the students’

61
learning patterns and goals applied to the students’ interactions (log file). We applied
the theoretical definitions of frustration to the students’ interactions (data from log file)
to construct the features, by doing this we constructed the features which captures the
students’ goals and learning patterns. These features are used to create a frustration
model, which detects student’s frustration when the student receives the response about
his/her goals. We used the human observation data from students’ facial expressions as a
independent method to train and validate our model. The reliability of human observers
are established using Cohen’s Kappa. Since, our frustration model is designed based
on the students’ goals and learning patterns, the reason why the student is frustrated
is known. Later, we developed an approach to respond to reasons for frustration. Our
approach is build on the strategies used in existing system to respond to frustration.

62
 Chapter 5

Detect Frustration

Affective computing is concerned with detecting and responding to the affective states.
Our research aim is to create a model to detect and respond to frustration accurately in
real-time when students are working with an ITS. In this chapter we describe our approach
to detect student’s frustration when they interact with the ITS. Then we discuss, how
our approach is applied to an ITS (Mindspark), and its performance compared to other
data-mining approaches. The frustration model is created by constructing features from
the ITS log data based on theoretical definitions of frustration. We focus on the instances
of frustration that occur due to goal blockage. Frustration in this case is considered
to be a negative emotion, as it interferes with a student’s desire to attain a goal [115].
To model the frustration we considered different classifiers used in existing approaches,
including dynamic Bayesian nets; however, we start with the linear regression classifier
because it is easy to build and it is a simple classification model to start with. The
linear regression model is flexible, fast and accurate when cross-validated [29]. Also the
linear regression classifier model, informs us of the factors contributing to frustration. It
helps us to determine which features contribute most to frustration, as well. Thus the
linear regression model can help us to respond to frustration systematically, and identify
potential sources of frustration, thereby, helping students to avoid it. However, later we
apply our constructed features to other classifiers such as decision tree and support vector
machine.

63
5.1 Theory-Driven Model to Detect Frustration

In this section, the theory-driven approach to detect frustration is described. The sequence
of steps to model frustration is shown in Figure 5.1. The selection and combination of
features from the ITS log file is done through a systematic process based on an analysis
of goal-blocking events. Guided by the theoretical definition (Step 1), we first identify
the goals of the student with respect to their interaction with the ITS, and select the top
n goals (Step 2). Based on information from the student log, a blocking factor, bf , for
each of the n goals is identified (Step 3). For example, goalj.bf represents the blocking
factor for the goalj. We formulate a linear model for Fi , the frustration index at the ith
question based on the blocking behaviors of student goals (Step 4). Since the features in
linear regression model are constructed based on the theoretical definition of frustration,
we call this approach as theory-driven approach. We apply a threshold to the frustration
index Fi , to detect whether the student is frustrated or not. The average of values used to
represent frustration and non-frustration, during the training process, is used as threshold.
The analysis on different threshold values is explained later in this chapter. The weights
of the linear regression are determined during the training process (Step 5)–with labeled
data from human observation–which is an independent method to identify affective states.
The performance of the model is validated by detecting frustration in the test data and
comparing results with independent method (Step 6).

The proposed theory-driven linear regression model to detect frustration is given

below:

Fi = α[w0 + w1 ∗ goal1.bf + w2 ∗ goal2.bf + ....

+wn ∗ goaln.bf + wn+1 ∗ ti ] + (1 − α)[Fi−1 ] (5.1)

The weights w0 , w1 , w2 , ..., wn in the equation above are determined by linear re-
gression analysis–explained later in this chapter. As explained in the previous paragraph,
the terms goal1.bf , goal2.bf , . . . , goaln.bf , are the blocking factors for goals goal1,

64
 1. Define Frustration: An emotion caused by
interference preventing/blocking one from
achieving the goal

System
2. Identify the students’ goals while they
under study
interact with the system (goal1, goal2,...,goaln)
(ITS)

3. List the blocking factors of each identified

goal (goal1bf, goal2bf, ..., goalnbf). Log data
from System
Operationalize it for the system using
log data

4. Create a linear regression model for

frustration index (Fi) with the
blocking factors identified

5. Learn the weights of the linear regression

model using labeled human observation data

6. Validate the performance of model with test

data and compare the results with labeled
human observation data

Figure 5.1: Steps of the theory-driven approach to create a frustration model using data
from the log file

goal2, . . . , goaln, respectively. The term ti , the time spent by the student to answer the
question i., is included on the basis of work done by Lazara et al. [98], which states that
time spent to achieve the goal is an important factor in frustration. The last term in the
equation, (1 − α)[Fi−1 ] accounts for the cumulative effect of frustration. We include this
term on the basis of [93], which states that frustration is cumulative in nature. The value
of α, determines the contribution of frustration at (i − 1)th question to frustration at ith
question, α which ranges from 0 to 1. We assume that the student is not frustrated at

65
the beginning of their interaction with the ITS, and hence, choose Fi = 0 for i = 1, 2, 3.

We restrict the scope of our approach to identify frustration that occurs due to
students’ goal blockage while interacting with the ITS. We do not include frustration that
might have occurred due to other factors external to students’ interactions with the ITS.
Hence, we are primarily concerned with how correct our detection (precision) is, even
though we might not be able to find all the frustration instances that exist (recall).

5.2 Theory-Driven Model for Mindspark Log Data

In this section we explain the application of the theory-driven approach to the Mindspark
log data. Our research goal is to detect frustration of the students while they interact
with the Mindspark.

5.2.1 Frustration Model for Mindspark

We create the linear regression model based on steps given in Figure 5.1.

Step 1. Frustration definition: We begin with the definition of frustration from

theory as “an emotion caused by interference preventing/blocking one from achieving the
goal”. The definition of frustration was discussed in Chapter 2 and it is reproduced below:

• Frustration is the blocking of a behavior directed towards a goal [115].

• The distance to the goal is a factor that influences frustration [97], [149].

• Frustration is cumulative in nature [137], [50].

• Time spent to achieve the goal is a factor that influences frustration [98].

• Frustration is considered as a negative emotion, because it interferes with a student’s

desire to attain a goal [124].

Step 2. Students’ Goals: We identified the four most common goals of students
while interacting with Mindspark. To identify these goals, we conducted interviews with

66
the staff of EI-India. The staff of EI-India conduct student interviews every month, as
part of their job. We requested them to interview the students, about their goals while
they interacted with the Mindspark. The interview was informal and conversational.
The questions emerged from the immediate context and were based on the student’s
answers; there were no predetermined questions. Eight students were interviewed by
random selection, during their interaction with Mindspark. The interview details are
given below.

Student Interview Details

• Goal: To identify the students’ goals while interacting with the Mindspark.

• Class: Sixth standard in Indian Certificate of Secondary Education (ICSE)∗ board.

• Subject: Student’s were learning Maths using Mindspark.

• Location: Mumbai, India.

• Time per student: Three to five minutes, and five questions per student.

• Data Collection: Voice recorder was used to record the interaction with the students.

We transcribed the interviews, and analyzed the transcripts to identify the goals
of the students. The important goals of the student were collecting Sparkies (reward
points in Mindspark), completing the current question correctly and quickly completing
the topic. These goals and corresponding blocking factors are discussed in next step.

Step 3. Blocking Factors: The goals goal1, goal2, goal3, goal4 and the cor-
responding blocking factors goal1.bf , goal2.bf , goal3.bf , goal4.bf are given in the Table
5.1. To model the blocking factor (bf ) of each goal, we consider students’ response to
Mindspark questions, a feature captured in the Mindspark student log file.

For goal1 of “to get the correct answer to the current question” the blocking factor
is getting the wrong answer to the current question. We use ai to represent the answer
∗
http://www.cisce.org/

67
 Table 5.1: Student Goals and Blocking Factors for Mindspark
Student Goal Blocking factor
goal1: To get the correct goal1.bf : Answer to the current question is wrong
answer to the current ques-
tion
goal2: To get a Sparkie (an- goal2a.bf : Answers to two previous questions are cor-
swer three consecutive ques- rect and to the current question is wrong
tions correctly)
goal2b.bf : Answer to the previous question is correct
and to the current question is wrong
goal3: To reach the Chal- goal3a.bf : Answers to four previous questions are cor-
lenge Question (answer five rect and to the current question is wrong
consecutive question cor-
rectly)
goal3b.bf : Answers to three previous questions are cor-
rect and to the current question is wrong
goal4: To get the cor- goal4.bf : Answer to the Challenge Question is wrong
rect answer to the Challenge
Question

to the current question; ai = 1 if correct, ai = 0 if wrong. The blocking factor of goal1 is

captured using:
goal1.bf = (1 − ai ) (5.2)

For goal2, “to get a Sparkie” the student should answer three consecutive questions cor-
rectly. This goal can be blocked, if a student gets any question wrong in a sequence of
three questions. Since the blocking factor by getting the wrong answer to the current
question is already addressed in goal1.bf , we consider only the blocking factor by getting
the second and third answer wrong, in a sequence of three questions. Hence goal2.bf has
two components. One way in which goal2 can get blocked is, if the student answers first
two questions correctly in a sequence of three questions and the third question wrongly.
This is captured by blocking factor goal2a.bf :

goal2a.bf = (ai−2 ∗ ai−1 ∗ (1 − ai )) (5.3a)

68
The second way in which goal2 can get blocked is, if the student answers only the first
question correctly in a sequence of three and the second question wrongly. This is captured
by blocking factor goal2b.bf :

goal2b.bf = ai−1 ∗ (1 − ai ) (5.3b)

The blocking factor of goal2 is:

goal2.bf = goal2a.bf + goal2b.bf (5.4)

For goal3, “to reach the Challenge Question”, the student should answer five con-
secutive questions, correctly. This goal can be blocked, if the student gets any question
wrong in a sequence of five questions. Since the blocking factor by getting the wrong
answer to the first, second and third question in a sequence of five questions is already
addressed in goal1.bf , and goal2.bf , we consider only the blocking factor by getting the
fourth and fifth answers wrong in a sequence of five questions. Hence, goal3.bf has two
components. One way in which goal3 can get blocked is, if the student answers only first
four questions correctly, in a sequence of five questions and the fifth question wrongly.
This is captured by goal3a.bf :

goal3a.bf = (ai−4 ∗ ai−3 ∗ ai−2 ∗ ai−1 ∗ (1 − ai )) (5.5a)

The second way in which goal3 can get blocked is, if the student answers only first three
questions correctly, in a sequence of five questions and the fourth question wrongly. This
is captured by goal3b.bf :

goal3b.bf = (ai−3 ∗ ai−2 ∗ ai−1 ∗ (1 − ai )) (5.5b)

69
The blocking factor of goal3 is:

goal3.bf = goal3a.bf + goal3b.bf (5.6)

For goal4: “To get the correct answer to the Challenge Question” the blocking
factor is getting the answer to Challenge Question wrong. The blocking factor of goal4 is
captured using:
goal4.bf = I ∗ (1 − ai ) (5.7)

where, I is the indicator for whether the current question is the Challenge Question or
not. I = 0 for normal question and I = 1 for Challenge Question.

Step 4. Linear Regression Model: The mathematical model to detect frustra-

tion for the Mindspark data is given in Equation 5.8, with the individual terms goal1.bf ,
goal2.bf ,...,goal4.bf , being defined in Equations 5.2-5.7:

Fi = α[w0 + w1 ∗ goal1.bf + w2 ∗ goal2.bf + w3 ∗

goal3.bf + w4 ∗ goal4.bf + w5 ∗ ti ] + (1 − α)[Fi−1 ] (5.8)

Step 5 and Step 6 of Fig. 5.1 are explained later in this Chapter.

5.2.2 Feature Selection and Combination in the Theory-Driven

Approach

The features (that is, the goal-blocking factors) in our linear regression model for the frus-
tration index Fi , are not the features directly available in the log file. Instead, they are cre-
ated by selecting and appropriately combining the available features from the Mindspark
log file (Table 5.2) using theory. We illustrate the advantages of selecting features by
applying a goal-blockage based theory, by comparing a feature with a simple combination
of data from the log file. Simple combination would be the most obvious way and the

70
first step in creating features from the log data. Both methods start with the same raw
data–features from the log file which indicate students’ response to the current question
(ai ), and the last two questions (ai−1 ) and (ai−2 ).

In our approach, we use a complex combination of the previous three responses

(goal2.bf in this example). This combination, goal2b f , is calculated using equation (5.4).
If ai = 1, indicates that the current question is correct and ai = 0 indicates that the
current question is wrong. When the goal of achieving a Sparkie (goal2), is blocked near
the goal (student answers first two questions correctly, in a sequence of three questions),
then the blocking factor goal2.bf , which contributes to the frustration index is high.
Similarly, when the goal is blocked midway to the goal (student answers only the first of
a sequence of three questions correctly) then the value of goal2.bf is medium. Finally,
when the goal is blocked far from the goal (student answers the first of a sequence of three
questions wrongly), the goal2.bf has a low value. The difference between goal-blocking
near the goal and far from the goal is captured in our approach which is not possible in
simple combination of the data from the log file (the last column in Table 5.2).

Table 5.2: An example to illustrate the advantages between selecting features by applying
goal-blockage based theory and by a simple combination of data from the log file when
applied to Mindspark
Theory-Driven Approach Simple Combination of
Data from Log File
ai−2 ai−1 ai eq eq eq Sum of last 3 responses
5.3a 5.3b 5.4
(goall2.bf )
0 0 0 0 0 0 0
0 0 1 0 0 0 1
0 1 0 0 1 1 1
0 1 1 0 0 0 2
1 0 0 0 0 0 1
1 0 1 0 0 0 2
1 1 0 1 1 2 2
1 1 1 0 0 0 3

71
 In the Table 5.2, the goal2.bf is to capture whether the goal of getting a Sparkie is
blocked or not. As explained in the previous paragraph the goal can be blocked by two
factors. For the response pattern (0,0,0) the student’s goal to get a Sparkie is neither
started nor blocked hence the blocking factor for goal2 (goal2.bf ) is 0. In the second
response patter (0,0,1) the student’s goal to get a Sparkie is started at the current response
but not blocked. Hence goal2.bf = 0. In third response pattern (0,1,0) the student’s goal
of getting a Sparkie is started in the last question but blocked due to incorrect answer
in the current question. Hence the goal2.bf is 1 (midway to the goal). In the fourth
response pattern (0,1,1) the goal of getting a sparking is started in the last questions
and not blocked, hence there is no blocking factor to this goal, goal2, bf = 0. Similarly
the value of goal2.bf for other response patterns are calculated based on the blocking
factors of the goal not by a combination of last three questions. Since the reasons for goal
blocking are identified in the finer detail, it may help us in avoid the negative consequences
of frustration in its detailed aspects–facilitating the prevention of frustration instances.

5.3 Experiment Methodology

In this section, we discuss the independent method we used to identify frustration: hu-
man observation. We conducted human observations of students interacting with the
Mindspark to validate our frustration model and to determine the weights of our model.
We describe the observation details, labeling of the observations and the metrics used
to compare the results of our frustration model with those from the human observation
method.

5.3.1 Human Observation

The goal of human observation was to observe the students’ facial expressions and label
them as frustrated or non-frustrated. In the human observation method, the observer
observes a student’s facial expression while s/he interacts with the ITS. In this subsection,
we discuss the expressions observed and affective states identified from facial expressions.

72
5.3.1.1 Sample

We recorded videos of 27 students’ (13 female, 14 male) facial expressions; six students
from a school in Mumbai, India and 21 students from a school in Ahmedabad, India
(both, urban areas in India). The students were fifth or sixth standard students in the
age range of 10-12. We recorded the video of students’ expressions while they interacted
with a Mindspark during a session in their school. Each video was 25-30 minutes, and on
average, it contained a student’s facial expressions over 30 to 40 questions.

5.3.1.2 Video Recording Procedure

We contacted two schools which were already using Mindspark in their curriculum. We
explained our objectives to students and their parents, and requested their consent to
record students’ facial expressions. The recording of the facial expression and the storing
of the videos adhered to ethics committee guidelines. The students’ facial expressions
were recorded using a webcam and the students’ interaction with the computer were
recorded using Camstudio† . The collected videos were used for the human observation.
The students’ facial expressions were coded, after they received feedback to the response
they submitted for each question. Our goal was to capture students’ expressions at the
point where they learn whether their answer to Mindspark’s question was correct or wrong.

5.3.1.3 Instrument

We used an observation protocol based on a facial analysis coding system [87], [152]. The
data collection sheet contained the following information: Student ID (Mindspark login
ID), Question Number and observation made by the observer. A sample observation that
records a student’s expressions is given in Table 5.3.

†
www.camstudio.org

73
5.3.1.4 Observation Procedure

The observers were Ph.D. students in Educational Technology at the Indian Institute
of Technology (IIT) Bombay. All observers had taken the ‘Research Methodology in
Education’ course, and hence had an understanding of the observation data collection
method. All the observers had practiced facial expression coding [87], [152] during a
pilot observation, prior to the actual study. After the pilot study, the observations were
checked for inter-observer reliability. The observers agreed eighty percent of the time with
other observers’ facial expression coding and Cohen’s κ was found to be 0.74–a substantial
agreement in the pilot study. In our actual study, the observers observed students’ facial
expression from the video whenever the student submitted an answer to a question. The
recorded videos helped the observers to pause the video and note down the expression. The
observations were recorded on the data collection sheet. In our study, only one observer
out of four was involved in building the model. Moreover, the student’s interaction with
the computer is used only to identify when to observe the facial expressions and is not
shown to the observer during observation process. Hence, the observation process is truly
an independent method of identifying frustration.

The facial expression observed are:

Face: Inner eyebrow rise, outer eyebrow rise, brow raise, brow lower, eyes wide
open, eyes closed, mouth open, jaw drop, mouth funnel-rounding, tongue show,
teething, smile, lip tightener and blink.

Head Movement: Left (with reference to the student), right, head up, head low,
head forward, head backward, head shake up and down and head shake side to side.

Speech: all words, for example yes, no, shhh and other were noted.

Hands Gesture: Hand on chin, scratching of head, tightening of fingers and raising
of fist.

Others: Leaning forward, relaxed, sitting at edge of the seat, mouse tapping,

74
 keyboard banging and chair movement.

5.3.1.5 Labeling Technique

The observers classified the students’ facial expressions into frustration and non-frustration,
based on the guidelines given in [46], [134] and [71].
The key behaviors to express frustration are:

• Outer brow raise

• Inner brow raise

• Pulling of hair

• Statements like “what”, “this is annoying”, “arey (Hindi word for expressing disap-
pointment)”

• Banging on the keyboard or the mouse

• Cursing

We show labeling of the students’ affective states along with sample observation data in
Table 5.3.

5.3.1.6 Universality in Facial Expressions

Facial emotions have been reported by Ekman to be universal [56]. The study by Hillary
et al., [7] reviewed 87 articles, related to cross-culture facial expression, and reports that

• Certain core components of emotions are universal

• “Emotions may be more accurately understood when they are judged by members
of same national, ethnic or regional group that had expressed the emotion”

The study by MK Mandal et al. checks the observer-expressor culture differences.

The study in [107] includes 43 university students from Canada and 43 university stu-
dents from India. All the participants were asked to recognize the emotions from facial
expressions. The study reports that:

75
Table 5.3: Sample Human Observation Sheet to Record Students’ Facial Observations
and to Label it as Frustration (Frus) or Non-Frustration (Non-Frus)
Question Observation Classification
Number
1 Mouth open, lower lip down Non-Frus
2 Reading aloud Non-Frus
3 Making noise (cha, arey), resting forehead Frus
with fingers
4 Lips tightening Non-Frus
5 Two hands raised up to to chest Non-Frus
6 Hands clamped Non-Frus
7 Reading Non-Frus
8 Head lean towards screen immediately Non-Frus
Challenge Inner brow up, eyes wide open, noise (hindi Frus
Question cursing words)
9 No expression Non-Frus
10 Mouth little open, eyes wide open Non-Frus
11 Inner eye brows lowered (shrink) Non-Frus

• “Happy, sad, and disgusted expressions are judged in the same way regardless of
the culture of the expressor or that of the observer”

• For surprise and anger there is a mismatch if the observer-expressor culture differs.
However, these emotions are correctly classified if the observer-expressor are the
from same culture. This supports the argument by Hillary et al., [7].

Based on the research results reported in the above articles we conclude that, (i)
the facial expressions are universal for frustration [56], and (ii) emotion recognition from
facial expression is done better if the observer-expressor are from the same culture/nation
(which is the case in our study). Hence, our facial expression observation is a valid
independent analysis to validate our model.

76
5.3.2 Analysis Procedure

We recorded 932 observations from 27 students. Among those, 137 observations were
classified as frustration (Frus) and remaining as non-frustration (Non-Frus). The dataset
is stratified at the student level. We represent the values obtained from human observation
as Bi at the ith instance, Bi = 0 for non-frustration and Bi = 1 for frustration.

To validate our model, we need to calculate the Frustration Index, Fi (using Equa-
tion 5.8), for the students whose facial expressions were observed. In order to calculate
Fi , the weights of the frustration model need to be learned. The procedure to learn the
weights of our frustration model, corresponding to Step 5 in Fig. 5.1, is given below:

1. To maintain a uniform scale among all the features in the frustration model, we apply
normalization to all the features. We used the following normalization equation.

X − M ean(X)
Xnew =
M ax(X) − M in(X)

Here, Xnew is the normalized value of feature X. We used the range (M ax(X) −
M in(X)) of the feature in the denominator, instead of standard deviation. This is
because the feature “response time” had a wide range of data (from 1 second to 200
seconds) compared to other features. Hence standard deviation in the denominator
will not normalize all the features in a uniform scale.

2. The data from the Mindspark log file is divided into two groups–training and testing–
to create and validate our frustration model, respectively. We use cross-validation
technique [66] to check how generalizable our frustration model is, when we apply
it to independent data. In this report, we used a tenfold cross-validation technique
[66] for all the experiments. In tenfold cross-validation the data is divided into 10
subsets. The model is trained using the nine subsets of data and tested on the
remaining data. This process is repeated to test all sets of data. We divided the
data based on number of students.

77
 3. We use linear regression analysis to identify the values for weight by assigning, 0
and 1 to represent non-Frus and Frus, respectively, in the training dataset.

4. We apply threshold to Fi , to classify the frustration index values as frustration and

non-frustration; we call this value as predicted value Pi .

5. We use the trained model (weights learned from linear regression analysis), to detect
frustration on the test dataset. Compare our detection with human observation.

5.3.3 Metrics to Validate Frustration Model

The metrics used to measure the performance and to validate our frustration model are
discussed in this subsection. Based on the feature from the log file, we detect students’
affective state at a given instant, as frustration or non-frustration; hence, we consider it
as a binary classification problem. In binary classification, most of the evaluation metrics
are based on a contingency table [140], [165]. The 2x2 contingency table for our research
is shown in Table 5.4:

Table 5.4: Contingency Table

Actual Frus Actual Non-Frus
Pred Frus True Positive (TP) False Positive (FP)
Pred Non-Frus False Negative (FN) True Negative (TN)

where, Actual Frus and Actual Non-Frus are results from human observation based on
students’ facial expressions while they interact with the ITS. Pred Frus and Pred Non-Frus
are detected values of frustration and non-frustration, respectively, using our frustration
model on the data from the Mindspark log file.

• True Positive (TP) is the number of frustration instances correctly detected as

frustration.

• True Negative (TN) is the number of non-frustration instances correctly detected

as non-frustration.

78
 • False Positive (FP) is the number of non-frustration instances detected as frustra-
tion.

• False Negative (FN) is the number frustration instances detected as non-frustration.

The most common metric used in the classification problem based on contingency
table is accuracy [113].

TP + TN
Accuracy = (5.9)
TP + FP + FN + TN

Accuracy measures the ratio of correctly detected instances (TP + TN) to total instances.
Other standard measures in classification problems are precision and recall based
on the contingency table. Precision measures the ratio of correctly detected frustration
instances (TP) to the total number of detected frustration instances (TP + FP).

TP
P recision = (5.10)
TP + FP

Recall measures the ratio of correctly detected frustration instances (TP) to the actual
number of frustration instances identified from human observation (TP + FN).

TP
Recall = (5.11)
TP + FN

Since our dataset is an unbalanced distribution of frustration and non-frustration, we

calculate the F1 score and Cohen’s kappa to measure the performance of our model
compared to random guesses.
As mentioned earlier, it is not feasible to detect all types of frustration experienced
by a student from all sources, especially those extrinsic in nature. In our research, we are
interested in detecting students’ frustration arising from their interactions with the ITS.
Hence, we report precision and recall of detecting frustration (instead of detecting non-
frustration). Our goal is to ensure the correctness of our detection of frustration, instead
of being able to detect all frustration instances encountered by students while interacting

79
with the ITS. Hence, high precision, and not high recall, is the important metric.

5.4 Results

In this section we describe the performance of our frustration model. In the frustration
model, our goal is to minimize the difference in detected frustration Pi and corresponding
human observation values Bi , (Pi − Bi )

min(Pi − Bi )2

by varying w0 , w1 , w2 , w3 , w4 , w5

To solve the above problem, we used a GNU Octave‡ , an open source numerical
computation software. The values of weights W0 to W5 are assigned equally during the
training phase of our model. This indicates that all the goals are assigned equal weights
during training of our model. In our experiments, to train the weights to optimum value,
we used the gradient descent algorithm with step size = 0.001. Our approach leads to a
converged set of weights after 70000 iterations, as seen in Fig. 5.2. Although our data
set is not large, to generalize our linear model we used gradient descent algorithm to find
the global optimum value for our weights.

The frustration model with trained weight values is shown in Equation 5.12. The
features are normalized as explained in the previous section.

Fi = 0.8[0.147 + 0.423 ∗ F eature1 − 0.0301 ∗ F eature2

+ 0.0115 ∗ F eature3 + 0.8359 ∗ F eature4

+ 0.1864 ∗ F eature5] + 0.2 [Fi−1 ]. (5.12)

To choose the best alpha value we find the performance of our model for different
‡
http://www.gnu.org/software/octave/

80
Figure 5.2: Convergence of Weights of our Linear Regression Frustration Model using
Gradient Descent Algorithm. The Cost J in the Figure is the error function (difference
between detected observation Pi and human observation Bi )

alpha values from 0.3 to 0.9. The results of frustration model for different alpha values
are given in table 5.5.

Table 5.5: Frustration model performance for different alpha values

Alpha TN TP FP FN Precision Recall Accuracy Kappa
0.3 0 0 137 795 0.00% 0.00% 85.30% 0.0000
0.4 2 1 135 794 66.67% 1.46% 85.41% 0.0224
0.5 5 2 132 793 71.43% 3.65% 85.62% 0.0560
0.6 41 8 96 787 83.67% 29.93% 88.84% 0.3939
0.7 44 10 93 785 81.48% 32.12% 88.95% 0.4118
0.8 45 12 92 783 78.95% 32.85% 88.84% 0.4132
0.9 45 13 92 782 77.59% 32.85% 88.73% 0.4099

From the analysis it is clear that alpha = 0.8, gives better performance based
on kappa values. Hence, we use alpha = 0.8. To determine the threshold value, which
classifies frustration and non-frustration in our model, we calculated the average of the
values we used to represent frustration and non-frustration in the training dataset. The
threshold value used is 0.5 (average of 0 and 1). Since we used the linear regression
classifier, we consider the mid-value as the threshold. However we find the performance
of the model for different threshold values from 0.2 to 0.9. The results of the frustration

81
model for different threshold values are given in table 5.6.

Table 5.6: Frustration model performance for different threshold values

Threshold TN TP FP FN Precision Recall Accuracy Kappa
0.2 134 141 3 654 48.73% 97.81% 84.55% 0.5652
0.3 130 132 7 663 49.62% 94.89% 85.09% 0.5683
0.4 79 64 58 731 55.24% 57.66% 86.91% 0.4873
0.5 45 12 92 783 78.94% 32.85% 88.84% 0.4132
0.6 38 5 99 790 88.37% 27.74% 88.84% 0.3786
0.7 20 5 117 790 80.00% 14.60% 86.91% 0.2111
0.8 2 0 135 795 10.00% 1.46% 85.52% 0.0247
0.9 2 0 135 795 10.00% 1.46% 85.52% 0.0247

We choose the threshold value based on the kappa value. From the threshold
analysis table, we observe that the kappa values are best for threshold values 0.2, 0.3, 0.4,
and 0.5, however the precision value is very low compared to the threshold value = 0.5.
The threshold value of 0.5 gives the best precision and recall combination, based on our
research interest. Our research interest is to achieve better precision than recall. Hence
we choose the average of 0 and 1 (values assigned for frustration and non-frustration,
respectively, to train the model) = 0.5 as threshold for our frustration model. The selected
threshold gives a better balance of precision and recall compared to other threshold values.

The performance of our frustration model on Mindspark data, using tenfold cross-
validation, compared to human observation is given in Table 5.7.

Table 5.7: Contingency Table of Our Approach when Applied to Mindspark Log Data
Human Observation
Frustrated Non-Frustrated
Pred Frustrated 45 12
Result Non-Frustrated 92 783

The values from Table 5.7 are used to calculate the performance of our model. The
results are given in Table 5.8.

82
Table 5.8: Performance of our Approach Shown Using Various Metrics when Applied to
Mindspark Log Data
Metrics Results
Accuracy 88.84%
Precision 78.94%
Recall 32.85%
Cohen’s kappa 0.41
F1 Score 0.46

From the above results, the accuracy and precision of our frustration model are
high compared to recall. As we mentioned earlier, we are interested in how correct our
detection of frustration is (measured by Precision) instead of detecting all frustration
instances encountered by students (reflected by Recall). Thus, the results are aligned
with our research goals. The Cohen’s kappa (0.41) and F1 measure (0.46) are found and
it is an acceptable value from affective state detection in ITS.

5.5 Performance of Data-Mining Approaches

Applied to the Data from Mindspark Log File

While the theory-driven approach gave high accuracy and precision for detecting frustra-
tion, we would like to compare how the theory-driven approach compares with data-mining
approaches. In this section, we compare the results of the theory-driven approach to de-
tect frustration, with some existing data-mining approaches, applied to the data from the
Mindspark log file. However, given the limited information available from the Mindspark
log data it is not easy to construct a Decision Network, as done in [33] and [139]. Hence,
we consider the approaches used in AutoTutor [48], Crystal Island [111], and approach in
[132], and apply them to the same Mindspark data.

We identified students’ frustration using the data from the Mindspark log file by ap-
plying the approaches in [48],[111] and [132]. We identified 14 features from the Mindspark
log file related to the students’ responses, time spent to answer and time spent on reading

83
explanation. We captured these features, after the students answered each question in
the session. Features selected from the Mindspark log data with description are given in
Table 5.5. According to the approaches in AutoTutor [48] and programming lab [132], we
did a correlation analysis of these 14 features with the observed affective state to select
those features that correlated with observed frustration. We identified 10 such features
that were correlated with observed frustration, and omitted the remaining uncorrelated
features. To avoid redundancy and to reduce the number of features, we did correlation
analysis among features and removed the strongly correlated features (Pearson’s r > 0.7)
as suggested in [48]. If two features are highly correlated then the feature which has a
higher correlation with the affective state is preserved [48].

Table 5.9: Description of Features Selected from Mindspark Log Data

Feature Description
Ques No Current question number in the session
Response The response given by student is correct or not
Res Time Time taken by the student to answer the question
Expln Time Time spent on reading explanation after answering the
question
Chall Res The response by the student for the challenge question
is correct or not
Prev Ques Res The response to previous question
Pre Expln Time Time spent on explanation of previous question
Last 2 Res Sum of response to last two questions
Last 3 Res Sum of response to last three questions
Last 4 Res Sum of response to last four questions
Last 5 Res Sum of response to last five questions
Last 6 Res Sum of response to last six questions
Last 3 Res Time Average response time for last three questions
Last 5 Res Time Average response time for last five questions

After this analysis, seven features were selected: i) response to answer the question
(whether the response provided by a student is correct or not), ii) response time to
answer the question, iii) time spent on explanation of the answer, iv) response to answer

84
the Challenge Question, v) sum of responses to the previous two questions, vi) sum of
responses to the previous four questions and vii) average response time to answer the
previous three questions. We used these seven features to repeat the analysis done in [48]
and [132]. We applied the data to all the classifiers used in AutoTutor [48], to identify
frustration from the log file. We report only the best result of the classifiers in each
category in Table 5.10.

Similarly, we used all 14 features to repeat the analysis as done in Crystal Island
[111], and we did experiments on Naive Bayes, SVM and Decision Tree classifiers as
mentioned in [111]. The results, along with the theory-driven approach are shown in
Table 5.10. We used the tenfold cross validation method in all our analyses.

Table 5.10: Comparison of our Approach with Existing Data-Mining Approaches Applied
to the Data from Mindspark Log File
System Classifiers Accuracy Precision Recall in
in % in % %
AutoTutor ap- Naive Bayes 82.83 40.94 37.95
proach [48]
(Selected Features) MLP 86.59 55.76 42.33
K* 87.02 56.89 48.17
Bagging Pred 87.55 57.89 56.20
Logistic Model 88.63 65.97 46.71
Tree
PART 87.23 60.97 36.49
Crystal Island ap- Naive Bayes 81.12 38.72 48.90
proach [111]
(All Features) Decision Tree 86.05 52.63 51.09
Introductory pro- Selected features used r = 0.583
gramming lab to form a linear regres-
approach [132] sion model
Our Approach Linear Regression 88.84 78.94 32.85
Model
Bold – Best results obtained in each approach

In Table 5.10, the best results by each approach are highlighted. The detailed
description of each approach is given here.

85
Autotutor Approach

We have identified seven features by correlation analysis as mentioned in Autotutor [48]

from the Mindspark log data. We applied these seven features in weka [108] and used all
classifiers mentioned in [48] to identify frustration from the log data. The classifiers used
are given below with categories:

• Bayesian classifiers

– Naive Bayes, Naive Bayes Updatable

• Functions

– Logistic Regression, Multilayer Perceptron, Support Vector Machines

• Instance based techniques

– Nearest Neighbor, K* (K Star), Locally Weighted Learning

• Meta classifiers

– AdaBoost, Bagging Predictors, Additive Logistic Regression

• Tree based classifiers

– C4.5, Decision Trees, Logistic Model Trees (LMT), REP Tree

• Rule based classifiers

– Decision Tables, Nearest Neighbour Generalization, PART

In Table 5.10 we have shown only the best result of the classifiers in each category.
Within the AutoTutor [48] approach, the Logistic Model tree performs comparatively
better than the other classifiers on the Mindspark dataset with an accuracy of 88.63%, a
precision of 65.97%, and a recall of 46.71%.

86
Crystal Island Approach

We have applied all the features identified from Mindspark log data in weka [108]. We
have done experiments on Naive Bayes, SVM and Decision Tree classifiers as mentioned
in [111]. We used tenfold cross validation to avoid bias in the training dataset. In
the Crystal Island [111] approach, we observed that the Decision Tree classifier gives
maximum accuracy of 86.05%, precision of 52.63%, and a recall of 51.09% as compared
to other classifiers.

Programming Lab Approach

We used seven selected features to form a multiple linear regression model and tested the
model using tenfold cross validation. Weka [108] is used to create and test the model as
suggested in [132]. The results are correlation co-efficient r = 0.583.

From the Table 5.10 we observe that the accuracy and precision of our frustration
model are high compared to the other approaches. However, the theory-driven approach
performed poorly in recall of 32.85% (best result in data-mining approach is recall of
56.2%). As we mentioned earlier, we are interested in how correct our detection of frus-
tration is (measured by Precision) instead of detecting all frustration instances encoun-
tered by students (reflected by Recall). Hence, our goal of achieving best precision and
accuracy is achieved in the theory-driven approach. The reason for better precision and
poor recall could be that the features are selected based only on goal-blockage type of
frustration, and hence, other types of frustration might have been missed.

5.6 Performance of Theory-Driven Constructed

Features Applied to Other Classifier Models

In our research, we started with the linear regression classifier model as our frustration
model. To test, if different models for frustration (other than a linear model) perform
better, we applied our features to non-linear models such as second order, and third order

87
polynomial models. For the second order polynomial model, we combined the theory-
driven features for the second order. Consider our features goal1.bf as f 1, goal2.bf as f 2,
and so on, then our initial feature set will be X = [f 1, f 2, f 3, f 4, f 5]. We combined these
five features into second order features as X2 = [f 1f 2, f 1f 3, f 1f 4, f 1f 5, f 2f 3, f 2f 4, f 2f 5,
f 3f 4, f 3f 5, f 4f 5, f 1f 1, f 2f 2, f 3f 3, f 4f 4, f 5f 5, f 1, f 2, f 3, f 4, f 5] and applied the linear
regression algorithm to create the frustration model. The trained model is shown in
Equation 5.13.

Fi = 0.8 ∗ [−0.013 ∗ f 1 − 0.0984 ∗ f 2 + 0.0766 ∗ f 3 + 1.0002 ∗ f 4 − 0.0019 ∗ f 5

−0.0984 ∗ f 1f 2 + 0.0766 ∗ f 1f 3 + 0.0292 ∗ f 1f 4 + 1.7798 ∗ f 1f 5 + 0.0766 ∗ f 2f 3

+0.0292 ∗ f 2f 4 + 0.8444 ∗ f 2f 5 + 0.0146 ∗ f 3f 4 + 0.3325 ∗ f 4f 5 + −0.0012 ∗ f 4f 5

+ − 0.013 ∗ f 1f 1 + 0.0363 ∗ f 2f 2 + −0.1807 ∗ f 3f 3 + 1.0002 ∗ f 4f 4

+0.0022 ∗ f 5f 5 − 0.9997] + 0.2[Fi−1 ] (5.13)

From the above second order model, we observed that the weights of feature 5
(f5), which is time spent to answer the questions, is increased when combined with other
features. This indicates the importance of time spent to achieve the goal in detecting
frustration. For the third order polynomial model, the feature set will further add fifteen
features to the second order feature set. To avoid handling the higher number of features,
we used the polynomial kernel function in the support vector machine (SVM) to compute
the performance of third and fourth order polynomial models. The polynomial kernel in
SVM computes all the possible features based on the exponent (order) value given to it.
The results of the second, the third and the fourth order polynomial models are shown in
Table 5.11

From the results, we observe that the second and the third order polynomial model
perform relatively better in precision, compared to the linear regression frustration model.
However, the overall performance value (kappa) is comparatively lesser than the linear
regression model. For the better understanding of cause of frustration, we use the linear
regression model.

88
Table 5.11: Performance of Theory-Driven Features when Applied to Higher Order Poly-
nomial Models
Order of Polynomial Model Precision Recall Accuracy Kappa
First 78.94% 32.85% 88.84% 0.41
Second 85.1% 29.2% 88.84% 0.3889
Third 82.4% 30.7% 88.84% 0.3989
Fourth 77.4% 29.9% 88.4% 0.3808

Table 5.12: Performance of Theory-Driven Features on Different Classifiers

Classifiers Precision in % Recall in % Accuracy in % Kappa
Naive Bayes 55.24% 57.66% 86.91% 0.4873
Logistic Regression 77.94% 38.69% 89.38% 0.4649
Bagging Pred 60.18% 49.64% 87.77% 0.4741
Logistic Model Tree 79.69% 37.23% 89.38% 0.4566
Decision Table 68.97% 43.80% 88.84% 0.4759

Further to measure the performance of theory-driven constructed features on non-

linear classifiers, we applied our features to the non-linear classifiers in Weka[108]. The
best result of the classifiers in each of the categories using tenfold cross validation method
is shown in Table 5.12.

From the performance of theory-driven constructed features on non-linear classi-

fiers, the Logistic regression model and the Logistic Model Tree outperformed our model
in recall of 38.69%; the other metrics are performed equal to the linear regression frustra-
tion model. For a simpler model we considered linear regression model in our research.
We note that logistic regression is the best method for our theory-driven approach. We
use logistic regression model to generalize our theory-driven approach in Chapter 7.

5.7 Generalization of our Model Within Mindspark

We have collected human observation data of class six students from two schools, hence,
the frustration model may not be used as it is for all age groups and all school cur-
riculum. In this section we discuss the limitations of generalizing our frustration model.
We have collected data from two schools–one from Mumbai, India and the other from

89
Ahmedabad, India. Both the schools are following the Indian Certificate of Secondary
Education (ICSE)§ curriculum. Hence, we cannot use our model to all curriculum across
Indian Schools. In order to apply our model to students of different age and other school
curriculum, revalidation of human observation is required.

We have collected the data from 27 students–14 male and 13 female. We restrict
our generalization only to the following in Mindspark log data:

• ICSE school curriculum

• Different locations across India

• For both genders

• Age Group of 10-13

To verify the above claims we trained our model with the Ahmedabad school data
and tested the model with the Mumbai school data. The results are given below. The
confusion matrix is shown in Table 5.13

Table 5.13: Confusion Matrix of Mumbai School Data

Human Observation
Frustrated Non-Frustrated
Model Frustrated 15 0
Data Non-Frustrated 18 212

The proposed model classified the frustration in new school data with Accuracy
of 92.65%, Precision of 100%. In the Ahmedabad school data the system identified
43 frustration instances from 687 observations, 43/687 = 6.2%. When we applied our
trained model to data from a new school, we identified 15 instances of frustration out of
245 observations, 15/245 = 6.12%. It proves that our model can be generalized for the
data from ICSE standard schools across Mindspark for the age group of 10-13.
§
http://www.cisce.org/

90
5.8 Discussion

In this chapter, we proposed and validated a theory-driven approach to detect frustration

of a student working with an ITS to understand the reasons for why the student is
frustrated. We focused on how correctly the frustration instances are detected, rather
than detecting all frustration instances encountered by the students. The performance of
theory-driven approach compared with existing data mining approaches when applied to
Mindspark log data. The results performed comparatively better in precision of 78.94%
(best result from data-mining approach is precision of 65.97%) and comparatively equal
in accuracy with 88.84% (best result from data-mining approach is accuracy of 88.63%).
Hence, our goal of achieving best precision and accuracy is achieved in the theory-driven
approach. However, the theory-driven approach performed poorly in recall of 32.85%
(best result in data-mining approach is recall of 56.2%). The reason for better precision
and poor recall could be that the features are selected based only on goal-blockage type of
frustration, and hence, other types of frustration might have been missed. A significant
advantage of the theory-driven approach is that the features identified provides the reasons
for students’ frustration and can be useful for informed responding. This knowledge can
give information on which variables to control while responding to students’ frustration.

The frustration model discussed in this Chapter is specific to Mindspark. In order

to apply our frustration model to other systems, the theory-driven approach explained
in Section 5.1 should be used. To apply our theory-driven approach to other systems,
careful thought is required to operationalize the blocking factors of goals. The goals of
the students when they interact with the system should be captured; this is a limitation in
the scalability of our approach. The results of the theory-driven approach are dependent
on how well the goals are captured and how well the blocking factors of the goals are
operationalized. In this research we observed students from an homogeneous group. The
human observation is valid for those students that belong to our group. In order to apply
our approach to different group of students, revalidation of human observations is required
for student from different group.

91
 In next chapter, we describe the strategies to avoid the negative consequences of
frustration in a timely manner, during the student’s interaction with the ITS.

92
 Chapter 6

Responding to Frustration

Affective computing is detecting and responding to the affective states of the user. Our
research aim is to detect and respond to the frustration of the students when they interact
with the ITS. In the previous chapter we described our theory-driven approach to detect
frustration of the students while they interact with the Mindspark–a web-based math
ITS. In this chapter, we describe our approach to respond to frustration, and the results
of our approach.

We detect frustration using the features constructed from the Mindspark log data
by applying theoretical guidelines. We respond to frustration using the messages which
are created based on theoretical definitions.

6.1 Our Approach to Respond to Frustration

The steps of our approach to respond to frustration is shown in Figure 6.1.

Detect the students’ frustration with its reasons when the students interact with
Mindspark (Step 1). We described the Step 1 in Chapter 5. Create the motivational
messages to respond to frustration when system detects frustration (Step 2). The messages
are displayed to the students based on the reasons for frustration. The algorithm to display
the messages is created in Step 3. Implement the approach to respond to frustration in
Mindspark and collect students’ log data for validation (Step 4). Impact of our approach
to respond to frustration is analyzed in Step 5. The Step 2 of our approach is described

93
 1. Detect frustration with its The theoy-driven
reasons model

2. Create motivational Strategies to res-

messages to respond pond and reasons
to frustraiton for Frustration

Log data and

3. Develop the algorithm
reasons for
to show messages
frustration

4. Collect data
for validation

5. Validate the impact of

motivational messages on
students' frustration

Figure 6.1: Steps of our Approach to Respond to Frustration

in following subsections.

In Chapter 2, we describe in detail the theoretical basis of our strategy to respond

to frustration. To summarize: the content in our motivational messages is based on
attribution theory [160]. The attribution theory implies that motivating the students
failure by messages which attributes the failure to external factors (such as math, difficulty
of the question) will motivate them to set a new goal. We use Klein’s guidelines [93], we
provide the option to students to reflect their feedback, the feedback is requested after
detecting frustration and feedback messages to show empathy for students’ affective state.
Using the recommendation from [129], [79] our motivational messages are displayed using
agents who communicate empathy in their messages. Dweck’s research [54] recommends
that messages should be constructed to praise the effort not their intelligence.

Our strategy to respond to frustration consists of following aspects.

94
 • Create motivational message to attribute the students’ failure to achieve the goal
to external factors [160].

• Create messages to praise the students’ effort instead of outcome [54].

• Create messages with empathy, which should make the student feel that s/he is not
alone in that affective state [93].

• Create message to request student’s feedback [79].

• Display messages using an agent [129], [79].

6.1.1 Reasons for Frustration

We create and display the messages to motivate the students based on the reasons for why
the student is frustrated. The prime reason for frustration is goal failure. The possible
reasons for goal failure are identified from the students’ goal while they interact with
the ITS. We represent these reasons as “events.” To create and display the messages we
consider the events in Mindspark as listed in the Table 6.1. We modified our frustration
model to identify the reasons for frustration (RoF ) as shown in equation 6.1.

RoF = goal1.bf + goal2.bf + goal3.bf + goal4.bf (6.1)

The values of RoF and its corresponding reasons for failure are detailed in Table
6.1. The value of RoF will be in the range of 0 to 5. If the goal1, that is getting correct
answer to current question, is blocked then it is identified in goal1.bf which is answer to
current question is wrong. The current question can be a normal question or challenge
question, which is identified the using the indicator I (I = 0 for normal question and
I = 1 for Challenge Question).

95
 Table 6.1: Explanation of the Reasons for Goal Failure - Events
Event RoF Value Answering Pattern
E1 0 The student’s response to current question, ai , is correct
E2 1 The student’s response to current question, ai , is wrong
Normal and Challenge both
E3 2 The student’s response to current question, ai , is wrong and to
previous question, ai−1 , was correct
E4 3 The student’s response to current question, ai , is wrong and to two
previous questions, ai−2 and ai−1 , were correct
E5 4 The student’s response to current question, ai , is wrong and to
three previous questions, ai−3 , ai−2 and ai−1 , were correct
E6 5 The student’s response to current question, ai , is wrong and to four
previous questions, ai−4 , ai−3 , ai−2 and ai−1 , were correct

6.1.2 Motivational Messages

Our messages are created using the strategies discussed in previous section, the reasons
for frustration and Mindspark log data. The following parameters are identified from the
Mindspark log data, which are considered while creating the motivational messages. The
messages are given in the Table 6.2 with condition to display the message and the reason
for creating the message.

• Average Response Time (Res Time) is the average time taken to answer the ques-
tions in Mindspark by students. The average response time from Mindspark’s ex-
isting log data, which we calculated, is 22 seconds.

• The Response Rate is the percentage of instances when students answered the ques-
tion correctly. We calculate the response rate for all questions using the Mindspark’s
existing log data. We represent the response rate as RR. However, we found that
RR of normal questions are higher then a Challenge question hence the RR is con-
sidered only in Challenge question.

• Frustration instances in the current session is represented by FrusInst. The FrusInst

counts the number of frustration instances detected in the session.

The messages discussed in the Table 6.2 will be concatenated based on the condi-
tions and displayed to the students. The message will appear as a speech bubble from a

96
 Table 6.2: Messages to Respond to Frustration with Condition and Justification
Condition Message Justification
to Display
Message
E2 Challenge You did well in the last four
questions
E3 You did well in the last The reason for frustra-
question tion and to praise the
E4 You did well in the last two student’s effort
questions
E5 You did well in the last
three questions and got a
Sparkie!
FrusInst = 1 E6 You did well in the last four
questions
Res Time > Av- You tried hard to get the Praising the students’
erage response correct answer effort.
time
Res Time < Av- Try hard
erage response
time
Normal Ques- I am sure you will do well in To motivate the stu-
tion the next questions dent
Challenge Ques- You may solve it next time.
tion
Challenge Ques- Dont worry, this is a tough Attributing the failure
tion question for many of your to difficulty of math
friends too. You can at- question and motivat-
tempt it again. ing the student.
FrusInst = 2 Normal question It is okay to get the wrong To share the students
and Response answer sometimes. You feelings - show empa-
Rate > 50% may have found the ques- thy
tion hard, but practice will
make it easier. Try again
Normal question It seems this is a tough Attributing the failure
and Response question for many of your to difficulty of math
Rate < 50% friends too. Try again question
FrusInst = 3 All questions Would you like to give your To receive student’s
feedback? feedback

buddy (an agent–shown in the Figure 6.3.) The algorithm to concatenate and display the
messages is discussed next.

97
6.1.3 Algorithms to Display Motivational Messages

In this section, we describe step 3 of our approach, that is the algorithm to show mo-
tivational messages. For the events listed in Table 6.1, that is for each goal failure, we
show the messages based on the student’s response time in answering the questions from
Mindspark, and nature of the question, that is how tough it is (how other students re-
sponded to the same question). We restrict the number of messages per Mindspark session
to 3. This is to reduce the number of interventions to the students during their interaction
with Mindspark. The messages are concatenated from the messages we created based on
the conditions. We created algorithms to concatenate the messages from Table 6.2 for
each event and described in this section in detail. We consider the following factors in all
algorithms:
a) For the first instance of frustration, we choose the message based on the time spent
by the student to answer the question, that is, Res Time. If the student spent more than
an average response time then, based on the event, the message to praise the student’s
effort of answering the question will be shown. If the student spent less than an average
response time then, the message to motivate the student will be shown. This is to praise
the students effort to answer the question [54].
b) For the second instance of frustration, we choose the message based on the difficulty
of the question, that is RR. If the RR is more than 50% then the message to motivate
the student will be shown. If the RR is less than 50% then the message to attribute the
failure to the difficulty of question will be shown. We assume that if the RR is < 50%
then this question might be difficult for many of the students. This is to attribute the
students’ failure to difficulty of the question, hence student will be motivated for the next
question [160].
c) For the third instance of frustration, the student’s feedback is asked. The pseudo code
for each algorithm is given below.

98
Event E1

The Student response to the current question is correct, so there will be no goal blockage
and hence no motivational messages.

Event E2: The Student’s Response to the Current Question is Wrong

If our model detects that the student is frustrated and the reason for frustration is that
the student’s response to the current questions is wrong, then the messages based on
the following algorithm are shown. The current question can be a normal question or a
challenge question. If the current question is a normal question, then the algorithm to
show the message based on the instance of frustration in the current session is given in
Algorithm 1.

Algorithm 1 To select the messages for Event 2 and Question Type is Normal
Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: Good, you tried hard to get the correct answer. I am sure you
will do well in the next questions.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: Try hard. I am sure you will do well in the next questions.
else if FrusInst = 2 & RR > 50% then
Message: It is okay to get the wrong answer sometimes. You may have
found the questions hard, but practice will make it easier. Try again.
else if FrusInst = 2 & RR < 50% then
Message It seems this question is tough, your friends also felt this question
is tough, try again
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

If the current question is a challenge question then student must have answered the
last four questions correctly. So, the message to praise the student’s effort on achieving
the challenge question will be shown. Challenge questions are generally difficult questions,
so, the RR of these questions is always less than 50%. If the current question is a challenge
question, then the algorithm to show the message based on the instance of frustration in
the current session is given in Algorithm 2.

99
Algorithm 2 To select the messages for Event 2 and Question Type is Challenge
Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: You did well in the last four questions, and have tried hard to
answer this challenge question too. Keep trying, you may solve it next
time.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: You did well in the last four questions. Try hard; you may solve
it next time.
else if FrusInst = 2 then
Message Don’t worry, this is a tough question for many of your friends too.
You can attempt it again.
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

Event E3: The Student’s Response to the Current Question is Wrong and
to the Previous Question was Correct

The messages based on the Algorithm 3 are shown if our model detects that the student
is frustrated and the reason for frustration is that the student’s response to the current
question is wrong and to the previous question was correct. Since the student performed
well in the previous question, the message praises the student’s effort in the previous
question.

Event E4: The Student’s Response to the Current Question is Wrong and
to the Two Previous Questions was Correct

If our model detects that the student is frustrated and the reason for frustration is that
the student’s response to the current question is wrong and to two previous questions was
correct, then messages based on the Algorithm 4 are shown. Since the student performed
well in two previous questions, the message praises the student’s effort in the previous
questions.

100
Algorithm 3 To select the messages for Event 3
Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: You did well in the last question, and have tried hard to answer
this question too. I am sure you will do well in the next questions.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: You did well in the last question. Try hard; I am sure you will
do well in the next questions.
else if FrusInst = 2 & RR > 50% then
Message: It is okay to get the wrong answer sometimes. You may have
found the questions hard, but practice will make it easier. Try again.
else if FrusInst = 2 & RR < 50% then
Message It seems this question is tough, your friends also felt this question
is tough, try again
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

Algorithm 4 To select the messages for Event 4

Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: You did well in the last two questions, and have tried hard to
answer this question too. I am sure you will do well in the next questions.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: You did well in the last two questions. Try hard; I am sure you
will do well in the next questions.
else if FrusInst = 2 & RR > 50% then
Message: It is okay to get the wrong answer sometimes. You may have
found the questions hard, but practice will make it easier. Try again.
else if FrusInst = 2 & RR < 50% then
Message It seems this is a tough question for many of your friends too. Try
again. You may get a Sparkie next time.
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

Event E5: The Student’s Response to the Current Question is Wrong and
to Three Previous Questions was Correct

If our model detects that the student is frustrated and the reason for frustration is that the
student’s response to the current question is wrong and to three previous questions was
correct, then messages based on the Algorithm 5 are shown. Since the student performed

101
well in three previous questions, the message praises the student’s effort in the previous
questions.

Algorithm 5 To select the messages for Event 5

Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: You did well in the last three questions and got a Sparkie! You
tried hard to answer this question too. I am sure you will do well in the
next questions.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: You did well in the last three questions. Try hard; I am sure
you will do well in the next questions.
else if FrusInst = 2 & RR > 50% then
Message: It is okay to get the wrong answer sometimes. You may have
found the questions hard, but practice will make it easier. Try again.
else if FrusInst = 2 & RR < 50% then
Message It seems this is a tough question for many of your friends too. Try
again.
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

Event E6: The Student’s Response to the Current Question is Wrong and
to Four Previous Questions was correct

If our model detects that the student is frustrated and the reason for frustration is that
the student’s response to the current question is wrong and to four previous questions was
correct, then messages based on the Algorithm 6 are shown. Since the student performed
well in four previous questions, the message praises the student’s effort in the previous
questions.

6.2 Methodology to Show Motivational Messages in

Mindspark

The methodology to show the messages is explained in this section. The block diagram of
the methodology to detect and respond to frustration is shown in Figure 6.2. As described

102
Algorithm 6 To select the messages for Event 6
Require: Res Time, RR, FrusInst.
return Message
if FrusInst = 1 & Res Time > Average response time in seconds then
Message: You did well in the last four questions, and have tried hard to
answer this question too. I am sure you will do well in the next questions.
else if FrusInst = 1 & Res Time < Average response time in seconds then
Message: You did well in the last four questions. Try hard; I am sure you
will do well in the next questions.
else if FrusInst = 2 & RR > 50% then
Message: It is okay to get the wrong answer sometimes. You may have
found the questions hard, but practice will make it easier. Try again.
else if FrusInst = 2 & RR < 50% then
Message It seems this is a tough question for many of your friends too. Try
again. You may get a Challenge Question next time.
else if FrusInst = 3 then
Message Would you like to give your feedback?
end if

in the previous section, the messages based on the event and the frustration instances in
the current session are shown.

Defintion of
frustration
Applied to

Features constructed Frustration Model (Fi)

Log data
from the log data Chapter 5

If frustrated
Mindspark Create and display the Reasons for
user interface motivational message frustration (Events).
Student
Attribution theory &
Dweck's message

Figure 6.2: Block Diagram of our Methodology to Detect and Respond to Frustration in
Mindspark

103
 As shown in Figure 6.2, the student’s interactions with the Mindspark user inter-
face are stored in the log file. From the log data the features to detect frustration are
constructed using the definition of frustration. The constructed features are used to create
the frustration model. The definition of frustration, the theory-driven approach to con-
struct features from the log data, and validation of the frustration model are described in
Chapter 5. If the student’s frustration instances is detected by our frustration model, then
the reasons for frustration are identified. The reasons for frustration are represented as
events. The appropriate motivational message based on the events and the data from log
file is selected, as explained in the Section 6.1.3. The implementation of our methodology
in Mindspark is explained below.

6.2.1 Implementation of Affective Computing in Mindspark

In order to detect and respond to frustration in Mindspark, we integrated the frustration

model and motivational messages in the Mindspark code. The following parameters are
calculated from the Mindspark log data.
Data Required

• The student’s response to Mindspark’s questions (the result of the last five ques-
tions,) R in log data. We represent it as ai .

• Time spent to answer the current question, S in log data.

• Question type, challenge or normal.

We use R = ai to represent the result to the current question; ai = 1 if correct,

ai = 0 if wrong.
ai−1 is result of previous.
ai−2 is result of previous to previous question. And so on for, ai−3 , ai−4 , ai−5 , ....
Using the above data, we calculate following features:

• Feature 1: goal1.bf (i) = (1 − ai ) ∗ (1 − I)

104
 • Feature 2: goal2.bf (i) = ((ai−2 ∗ ai−1 ∗ (1 − ai )) + ai−1 ∗ (1 − ai )) ∗ (1 − I)

• Feature 3: goal3.bf (i) = ((ai−4 ∗ ai−3 ∗ ai−2 ∗ ai−1 ∗ (1 − ai )) + (ai−3 ∗ ai−2 ∗ ai−1 ∗
(1 − ai ))) ∗ (1 − I)

• Feature 4: goal4.bf (i) = I ∗ (1 − ai )

• Feature 5: time spent to answer the question, goal5.bf = S

Where I is to check whether the current question is a challenge question or not.

I = 0 for normal question and I = 1 for a challenge questio

The features are used to calculate the frustration index. The frustration model was
trained using the data collected during human observation. The trained model from the
equation 5.12 in Chapter 5 is reproduced below in the equation 6.2.

Fi = 0.8[0.147 + 0.423 ∗ (F eature1 − 0.25) − 0.0301 ∗ (F eature2 − 0.25)/2

+ 0.0115 ∗ (F eature3 − 0.11)/2 + 0.8359 ∗ (F eature4 − 0.04)

+ 0.1864 ∗ (S − 22.5)/300 ] + 0.2 [Fi−1 ]. (6.2)

Fi = 0, for first three questions.

RoF = Feature1 + Feature2+ Feature3 + Feature4. (Range : 0 - 5)

The algorithm to detect whether the student is frustrated or not, is given in Algo-
rithm 7

The threshold value 0.5 chosen based on our analysis as shown in Table 5.6 in
Chapter 5. From the threshold analysis, we observe that the kappa values are best for
threshold values 0.2, 0.3, 0.4, and 0.5, however the precision value is very low compared
to the threshold value = 0.5. The threshold value of 0.5 gives the best precision and recall
combination.

105
Algorithm 7 To detect whether the student is frustrated or not
Require: Fi , RoF , FrusInst.
return Message
Initialize FrusInst = 0
while After receiving response from student, ai do
Calculate Fi using ai
if The Frustration Index,Fi , is greater than the threshold value and FrusInst < 3
then
FrusInst++
Based on the value of RoF , select the event and the algorithm to show motivational
message
end if
end while

Motivational messages are shown to students in real-time when they interact with
the Mindspark. The sample screen-shots of motivational messages shown to students is
shown in Figure 6.3 and Figure 6.4.

Figure 6.3: Screen-Shot of Mindspark’s Buddy Showing a Motivational Message as Speech

Bubble - 1

6.3 Impact of Motivational Messages on Frustration

To determine the impact of the motivational messages, we compare the number of frus-
tration instances without and with motivational messages in Mindspark sessions. In this
section we discuss our data collection methodology, sample size and analysis techniques.

106
Figure 6.4: Screen-Shot of Mindspark’s Buddy Showing a Motivational Message as Speech
Bubble - 2

6.3.1 Methodology to Collect Data from Mindspark Sessions

The methodology to collect data to validate our approach is described in this section–step
4 of our approach to respond to frustration (Figure 6.1). Our methodology to collect the
data is shown in Figure 6.5

6.3.2 Sample

We have created the frustration model based on the Indian Certificate of Secondary Ed-
ucation (ICSE∗ ) syllabus. Hence, we test the impact of motivational messages in schools
which follow ICSE syllabus. We collected the data from class six of three schools. Three
schools were chosen based on the number of students using Mindspark in class six. Also
we chose the school from different cities (Rajkot, Bangalore, Lucknow) in India.

6.3.3 Data Collection Procedure

The steps to collect data are given below:

• Extract Mindspark session data from one week of class six students in three schools.
∗
http://www.cisce.org/

107
 In the following week,
Select three ICSE board implement addressing
schools. School ID: 1752, frustration algorithms for
153271, 420525 same schools.

Collect class 6 student’s Collect class 6 student’s

log data for one week. log data for one week.

Remove the sessions with Remove the sessions with

no of questions < 10 no of questions < 10

Remove the sessions with Remove the sessions with

average time spent to average time spent to
answer the questions < answer the questions <
11 seconds 11 seconds

Select the unique user ID Select the unique user ID

and corresponding data and corresponding data

Calculate number of frustration

instances per session for the
identical students

Figure 6.5: Methodology to collect data for validating our approach to respond to frus-
tration

• Remove sessions which have less than ten questions. This is to avoid having a small
number of questions in a session.

• Remove those sessions which have less than 11 seconds as a average time spent on
each questions.

108
 • Retain the unique userIDs and their Mindspark sessions’ data.

The selected student’s userIDs were stored. In the following week, we implemented
our approach to show motivational messages to the same schools. Motivational messages
are shown to students through “buddy,” based on the reasons for goal failure. These data
collection steps are repeated to select the unique userIDs and their Mindspark sessions’
data in the second week.

From the two weeks’ data, the identical userIDs (students) and corresponding ses-
sions’ data are considered for analysis. We collected the Mindspark session data 188
students’ from three schools for two weeks. The details of the data collected from the
three schools are shown in Table 6.3.

Table 6.3: Details of the data collected from three schools to measure the impact of
motivational messages on frustration
School Number of Mindspark topic Mindspark topic Number of match-
Code students in in first week in second week ing students’ ses-
Class 6 (Without motiva- (with motiva- sions considered for
tional Messages) tional messages) analysis
1752 326 Integers Integers 54
153271 279 Decimals Decimals 72
420525 164 Algebra Geometry 62
Total 188

6.3.4 Data Analysis Technique

The data from 188 students’ Mindspark sessions are collected across two weeks has been
used for our analysis. We used the trained frustration model, as shown in Equation 6.2
to detect the number of frustration instances in each session. The number of frustration
instances in the first week’s data from 188 students’, without any motivational messages,
was calculated. The number of frustration instances in the second week’s data from 188
students’, with motivational messages to avoid the negative consequences of frustration,
were also calculated. The first Mindspark session of 188 students in each week is considered
in our analysis, hence 188 Mindspark sessions data from two weeks are used in our analysis.

109
To analyze the impact of frustration, we compare the number of frustration instances in
the data from the first and second weeks. The median is used as a metric to compare the
frustration instances.

6.4 Results

The impact of motivational messages on frustration instances is discussed in this section,

which is step 5 in the approach to respond to frustration. The median absolute deviation
of the number of frustration instances in Mindspark sessions of 188 students is calculated
and shown in Table 6.4.

Table 6.4: Median and Median Absolute Deviation (MAD) of number of frustration in-
stances from the Mindspark session data from three schools

Number of Mindspark Ses- Median of Frustration In- MAD of Frustration In-

sions stances stances
188 sessions without moti- 2 2.1942
vational messages
188 sessions with motiva- 1 1.4628
tional messages

The number of frustration instances in 188 sessions are visually represented using
the box plot in Figure 6.6. The figure shows the frustration instances without and with
motivational messages.

The number of frustration instances after implementing motivational messages is

reduced–shown in the box plot. The circle indicates out-lier; this means that the number
of frustration instances per session being equal to six is very less after implementing
motivational messages to respond to frustration. From the results, the median and median
absolute deviation of frustration instances after implementing motivational messages to
avoid frustration is reduced from 2 to 1 and 2.19 to 1.46, respectively.

To analyze the distribution of frustration instances, we plot the histogram of frus-

tration instances without and with motivational messages. The histogram plot of the

110
Figure 6.6: Box plot of Frustration instances from 188 sessions without and with mo-
tivational messages. Box = 25th and 75th percentiles; bars = minimum and maximum
values; center line = median; and black dot = mean.

number of frustration instances without motivational messages is shown in Figure 6.7 (a).
The histogram plot of the number of frustration instances with motivational messages is
shown in Figure 6.7 (b).
The histogram plot shows that due to our approach, number of sessions (Y axis)
with frustration instances per session (X axis) greater than or equal to 2 is reduced com-
pared to the histogram plot without motivational messages. For example, number of
sessions with frustration instances equal to 5 is reduced from 11 to 5 due to motivational
messages. It indicates that the students’ number of frustration instance per session has
reduced. The histogram does not follow a normal distribution, hence, we used Man-
nWhitney (MW) test to validate whether the two distributions are the same or not. The
P value from MW test is = 0.0001 (P < 0.05), which rejects the null hypothesis that two
distributions are same. The mean of ranks of frustration instances in sessions without
motivational messages is high, compared to the sessions with motivational messages, 212.7

111
 (a) (b)
Figure 6.7: (a). Histogram of Number of the Frustration Instances in 188 sessions without
Motivational Messages and (b). Histogram of Number of the Frustration Instances in 188
sessions with Motivational Messages

vs 164.3. This indicates that the number of frustration instances are significantly lesser
when motivational messages are shown to the students.
We further analyzed the significant difference in frustration instances for three
schools separately, and the results of our analysis are shown in Table 6.5.

Table 6.5: Impact of motivational messages on frustration in three schools

School Number of Without Motiva- With Motivational Mann-Whitney’s

Code Sessions tional Message Messages Significance Test
Sum of Median Sum of Median
Frustration Frustration
instances instances
1752 54 92 1 57 0 P < 0.05
153271 72 212 3 148 1 P < 0.05
420525 62 130 2 72 1 P < 0.05

The P value from the MW test is = 0.0136 (P < 0.05) for school 1752, which rejects
the null hypothesis that two distributions of frustration instances are same. Similarly the
P value for school 153271 is 0.0028 and for school 420525 is 0.0062, which are lesser

112
then 0.05. Hence, the data from all schools rejects the null hypothesis. Moreover, the
data from Table 6.5 shows that, in all three schools, the number of frustration instances
after implementing the approach to avoid frustration is reduced. Hence, the number of
frustration instances is reduced statistically significantly in all three schools. The math
topics used in Mindspark sessions are Integers, Decimals and Algebra. Hence, the results
indicate that the motivational messages to respond to frustration are generalizable to all
topics in math and across different schools in Mindspark.

6.4.1 Validation of Impact of Motivational Messages

In the previous subsection, the results indicate that the motivational messages reduced
the number of frustration instances in Mindspark sessions. However, the results do not
show whether the frustration instances reduced due to motivational messages or due to
students’ usage of Mindspark in the second week for same math topic. Hence, in this
subsection we analyzed one school’s data for two weeks without motivational messages.

For the validity analysis we used the data from school 1752. We considered the
Mindspark sessions of 99 students for two weeks. The Mindspark sessions are selected
as described in the previous section. The math topic in Mindspark sessions was Whole
Number Concept. The students’ data from the same school has been analyzed; it showed
(in Table 6.5) a significant reduction in the number of frustration instances with moti-
vational messages. The box plot of frustration instances in Mindspark sessions for two
weeks without motivational messages is shown in Figure 6.8.

From the box plot above, the median value of number of frustration instances per
session is reduced in second week from 2 to 1. However, the box size and mean value
are not reduced and the maximum number frustration instances per session remain same.
Hence, we used the Mann Whitney test, to check whether there is a significant difference
in number of frustration instances from the Mindspark sessions. The details are given in
the Table 6.6

From the Table 6.6, the P value from Mann-Whitney test is = 0.45 (P > 0.05)

113
Figure 6.8: Box plot of Frustration instances from Mindspark sessions in a school for
consecutive weeks without responding to frustration. Box = 25th and 75th percentiles;
bars = minimum and maximum values; center line = median, and black dot = mean.

Table 6.6: Mann-Whitney significance test on frustration instances from Mindspark ses-
sions without motivational messages

School Number of First Week Data Second Week Data Mann-Whitney’s

Code Sessions Significance Test
Sum of Median Sum of Median
Frustration Frustration
instances instances
1752 99 215 2 203 1 P > 0.05

for school 1752, which accepts the null hypothesis that two distributions of frustration
instances are same. This indicated that the number of frustration instances is not signif-
icantly reduced when the student uses the same math topic in Mindspark for two weeks.
However, when the students’ were learning same topic in Mindspark session showed sig-
nificant reduction in number of frustration instances in second week with motivational
messages. This has been established for same students (school 1752) in Table 6.5. Hence,

114
we conclude that our previous results of significant reduction in frustration instances are
due to the motivational messages shown to the students. To understand the impact of
the motivational messages for students with different learning achievements, we analyzed
our method in finer detail–described in the next subsection.

6.4.2 Detailed Analysis

The objective of the detailed analysis is to understand the impact of the motivational
messages on students’ frustration, learning achievement and time spent to answer the
question. We start with the transition matrix of frustration instances for two weeks,
without and with motivational messages. The matrix helps us to identify how the frus-
tration instances of each student change after receiving motivational messages. Then, we
analyze the impact of the motivational messages (to avoid students’ frustration) with the
learning achievement of the students and the average time spent to answer questions in
the Mindspark session.

Motivational messages are shown only to those students who are frustrated. Hence,
we show the transition matrix of sessions with frustration instances more than or equal
to one. From 188 Mindspark sessions, we considered data from 88 Mindspark sessions
which showed frustration instances at-least once in both weeks. Mindspark sessions of 88
students without motivational messages and the Mindspark sessions of the same students
with motivational messages are considered to create a transition matrix.

The transition matrix of the Mindspark sessions of 88 students, for a number of

frustration instances per session, from without motivational messages to with motivational
messages is shown in the Table 6.7. The upper bound is considered as 6, that is the
Mindspark session with more than 6 frustration instances is counted as 6.

The transition matrix Table 6.7, shows how the frustration instances in each stu-
dent’s Mindspark session change after receiving motivational messages. For example,
consider the Mindspark sessions in the first week (without motivational messages) with
number of frustration instances equal to two. From the table, it is 22. During the second

115
Table 6.7: Transition matrix of the Mindspark sessions of 88 students, for a number of
frustration instances per session, from without motivational messages to with motivational
messages

Number of Students’ Mindspark sessions with motivational messages

Frustration 1 2 3 4 5 6 Total
Instances per
session
1 13 5 2 2 22
2 12 5 3 2 22
Number of students’ 3 4 5 1 1 1 12
Mindspark sessions 4 2 1 2 1 3 9
Without Motivational 5 6 1 7
Messages 6 4 3 1 1 3 4 16
Total 41 19 7 5 5 11 88

week, with motivational messages, only 5 out 22 students got the number of frustration
instances as equal to two. Also, the frustration instances of 12 out of 22 students is
reduced to 1 and the remaining 5 students’ per session is increased from 2 to 3 and 4.
This indicates that the students’ frustration instances reduce after receiving motivational
messages. However, the students who got a lesser number of frustration instances per
session without motivational messages are more frustrated in Mindspark sessions with
motivational messages.

We compute the percentage of students’ whose frustration instances are reduced

due to motivational messages and the percentage of students whose frustration instances
increased due to motivational messages. Table 6.8 shows the percentage of increase and
decrease in frustration instances for 188 sessions and three schools separately. Moreover,
we computed the percentage of frustration instances increased or decreased by greater
than one steps, that is number of frustration instances per session change from 3 to 1,
or from 4 to 0, or from 3 to 6, and so on. The percentage of “no change”, that is, the
students whose frustration instances are not changed due to motivational messages are
not considered.

From the Table 6.8, it is evident that the percentage of reduction in frustration
instances is almost double compared to the percentage of increase in frustration instances

116
Table 6.8: Percentage of increase and decrease in frustration instances at one and two
steps. The data are from the Mindspark sessions of three schools. Percentage of no change
is not shown

School Number of Frustration Instances De- Frustration In-

Sessions creased stances Increased
By 1 step By > 1 step By 1 step By > 1 step
Three 188 52.65% 33.51% 26.06% 13.82%
schools
1752 54 44.44% 25.92% 20.37% 9.25%
153271 72 55.55% 34.72% 25% 16.66%
420525 62 53.22% 35.48% 32.25% 14.51%

due to motivational messages. Also, the frustration instances are reduced in all three
schools. In school 1752, the percentage of frustration instances in week two increased less
(9.25%) compared to school 153271 (16.66%). This can be due the math topic used by
students of 1752 (same topic for two weeks). The frustration instances is reduced for all
math topics, which infers that our motivational messages has impact on all math topics.
However, the percentage of frustration decrease is varied for different math topic. We
assume the is due to nature of the math topic considered.

6.4.2.1 Impact of Motivational Messages on Learning Achievement

The objective of this analysis is to check the impact of motivational messages on students
with different learning achievement. We grouped the student ID’s into low, medium, and
high based on their performance in the Mindspark session. The students who got less
than 60% of correct responses in the session are grouped as low, the students with 60% to
80% of correct responses in the session are grouped as medium, and the students with >
80% of correct responses in the session are grouped as high. Table 6.9 shows the impact
of the motivational messages on the students with varying learning achievement.

From Table 6.9, the frustration instances of the students with low learning achieve-
ment are 77.78%, reduced compared to the 8.33% increase. The frustration instances
of the students with medium learning achievement are 47.05% reduced compared to the

117
Table 6.9: Impact of the motivational messages on students with different learning achieve-
ment. The students are grouped based on their percentage of correct response in the
Mindspark session. Low (< 60%), Medium (60% < and < 80%), and High (> 80%). The
percentage of frustration instances which increased and decreased due to the motivational
messages is shown. The percentage of no change in frustration instances are not shown

Learning Achievement Number of Sessions Frustration Instances Frustration Instances

Decreased Increased
Low 36 77.78% 8.33%
Medium 85 47.05% 29.41%
High 67 47.76% 31.34%

29.41% increase. Similarly, the frustration instances of the students with high learning
achievement are 47.76% reduced compared to the 31.34% increase. Hence, the motiva-
tional messages have more impact on students whose correct response percentage is low.

6.4.2.2 Impact of Motivational Messages on Average Time to Answer the

Questions in Mindspark Session

The objective of this analysis is to check the impact of motivational messages on the
average time taken to answer the questions in the Mindspark session. We grouped the
student ID’s into “less”, “average”, and “more” based on average time they spent to
answer the questions. The students who spent less then 20 seconds per question in the
Mindspark session are grouped as “less”, the students who spent between 20 to 30 seconds
per question are grouped as “average”, and the students who spent more then 30 seconds
per question in the Mindspark session are grouped as “more”. Table 6.10 shows the
impact of motivational messages on the average time spent by the students to answer the
questions in Mindspark session.

From the Table 6.10, the frustration instances of students who spent lesser time
in answering the questions are 43.39% reduced compared to a 33.96% increase. The
frustration instances of students who spent an average time to answer the questions are
49.2% reduced compared to 34.92% increase. However, the frustration instances students
who spent more time in answering the questions are 63.88% reduced compared to a 12.5%

118
Table 6.10: Impact of the motivational messages on the average time spent by the students
to answer the questions in the Mindspark session. The students are grouped based on their
average time spent to answer the questions in the Mindspark session. Less (< 20 seconds),
Average (20 seconds < and < 30 seconds), and More (> 30 seconds). The percentage of
frustration instances which increased and decreased due to the motivational messages is
shown. The percentage of no change in frustration instances are not shown

Average time spent to Number of Sessions Frustration Instances Frustration Instances

answer the questions Decreased Increased
in the Mindspark ses-
sion
Less 53 43.39% 33.96%
Average 63 49.2% 34.92%
More 72 63.88% 12.5%

increase. Hence, the motivational messages have more impact on students who spend
average or more time in answering the questions in the Mindspark session.

6.4.2.3 Analysis on Ordering Effects - Removal of Motivational Messages

In our research, the sessions with motivational messages always followed by sessions with
no motivational messages. Hence, to analyze the ordering effect we collected the data
from sessions with no motivational messages followed by the sessions with motivational
messages. We collected 42 students’ Mindspark sessions’ data from a school (153271) for
three weeks. First week data are from sessions with no motivational messages, second
week data are from sessions with motivational messages, and third week data are from
the sessions with no motivational messages.

The number of frustration instances from 42 sessions in each week is visually rep-
resented using the box plot in Figure 6.9.

From the Figure 6.9, it is clear that frustration instances is higher in session with
no motivational messages compared to the sessions with motivational messages. It is valid
for the sessions with no motivational messages followed or preceded by session with mo-
tivational messages. This analysis proves that, there is no ordering effect in our approach
to respond to frustration.

119
Figure 6.9: Box plot of Frustration instances from 42 session in each week. First week
without motivational messages, second week with motivational messages and third week
without motivational messages. Box = 25th and 75th percentiles; bars = minimum and
maximum values; center line = median; and black dot = mean.

6.4.3 Discussion

In this chapter, we described our approach to respond to frustration in Mindspark. We

analyzed the data from three schools to measure the impact of the approach to respond to
frustration. The results are discussed above. From our analysis the following inferences
are made.

• From the histograms in Figure 6.7, the frustration instances of students are reduced
in the sessions with motivational messages.

• There is a statistically significant reduction in the number of frustration instances

per session due to the approach to respond to frustration.

• The significant reduction in the frustration instances is independent of the schools

analyzed and topics used in the Mindspark sessions.

120
 • The approach to respond to frustration has a relatively higher impact on the students
whose performance in the sessions is low.

• The approach to respond to frustration has a relatively higher impact on the students
who spend more time to answer the questions in Mindspark session.

These inferences prove that, the approach to reduce frustration significantly reduces
the number of frustration instances per session. Our research findings are aligned the
existing research findings on responding to frustration. Scooter the tutor [133] reports
that, due to motivational messages the number of frustration instances is reduced. Wayang
Outpost [164] reports that, the students change their behavior because of motivational
messages.

Our approach to respond to frustration can be generalized to all math topic and
schools, in Mindspark. However, the impact of motivational messages are analyzed using
students of class 6. The motivational message may have different impact on class 2 or
class 3 students, who may not understand the meaning of the messages shown to them.
Moreover, the higher class students might be self motivated and learn skills to handle
frustration. Hence, this approach cannot be generalized to students of all ages.

Motivational messages, avoid the negative consequences of frustration and hence

helps the students to continue learning and achieve their learning goal. Detecting and
responding to frustration enriches the student model of ITS.

6.5 Summary

In this chapter we have discussed our approach to respond to frustration using the attribu-
tion theory. The approach we developed for Mindspark was based on students’ goal failure
and has been implemented and tested successfully. The results show that motivational
messages using the attribution theory reduced the number of frustration instances per
session. The reduction in the number of frustration instances is statistically significant.
Also the approach can be generalizable to Math topics and class 6 students in schools, in

121
Mindspark.

122
 Chapter 7

Generalizing Theory-Driven Approach

In this chapter, we motivate generalization of our theory-driven approach to detect other

cognitive affective states. We discussed modeling frustration in Chapter 5. We now
describe modeling “boredom” using the theory-driven method. Boredom is one of the
cognitive affective state considered in the computer learning environments [26], that has
been studied with respect student learning in ITS [39]. Our goal in this chapter is to show
proof of concept of how the theory-driven approach we developed can be used to model
the other affective states. We do not attempt to create a state of art model to detect
boredom in ITS.

7.1 Applying Theory-Driven Approach to Model

Boredom

In chapter 5, we described our theory-driven approach to model frustration in an ITS.

The generic theory-driven approach to detect affective states is given below:

1. Operationalize the theoretical definition of affective state for the system under con-
sideration.

2. Construct features from the system’s log data; based on the theoretical definition
of affective state.

123
 3. Create a logistic regression model using the constructed features to detect the af-
fective state.

4. Conduct an independent method to detect affective state and use the data from
independent method to train the weights of logistic regression model.

5. Validate the performance of the model by detecting the affective state in the test
data and compare the results with the data from independent method.

In this chapter, we modified the generic theory-driven approach to model boredom

and is shown in Figure 7.1.

7.2 Definition of Boredom

In this is section we discuss the various existing definitions of boredom. Then we discuss
the definition of boredom within educational settings.

In 1938, Barmack [13] defines boredom as a conflict between the tendency to con-
tinue the situation and tendency to escape from it due to lack of motivation. For example,
if a student is not motivated to complete a assignment given to him/her, and if s/he is
not interested in the subject, then the conflict arises whether to complete and submit the
assignment or not to submit. That state of conflict is called as boredom. It is the first
published scientific study on boredom.

In 1981, the review article by Smith [148], reviewed the psychological and psychi-
atric study of boredom from 1926 to 1979. The review article reports that, the common
factors to generate boredom are repetitiveness, lack of novelty, and monotony.

In another review article [119], Hanlon reviews the research articles from 1930-1980
and lists the general agreement regarding boredom. They are:

• Boredom is a reaction to the repeated activity and monotonous stimulation.

• Degrees of boredom vary greatly for different individuals in the same working envi-
ronment.

124
Figure 7.1: Steps of theory-driven approach to create a boredom model using data from
the Mindspark log file

• The person can escape boredom by changing the situation or escaping from the
situation.

• Boredom is situation specific and it is reversible if the situation changes.

More recently, Mikulas and Vodanovich (1993), [114] defines boredom as “a state of
relatively low arousal and dissatisfaction, which is attributed to an inadequately stimulat-
ing situation [114].” For example, boredom occurs if the student is in “non interesting” or
“non challenging” situation which fails to keep his/her motivation. Flow theory (1997) by

125
Csikszentmihalyi [37] states that boredom occurs when the students’ skill and challenge
mismatches (high skill, less challenge). For example, if the student is asked to answer less
challenging questions compared his skill then s/he might get bored. Carriere, Cheyne,
and Smilek (2008), [28] states that boredom occurs due to the inability to engage or sus-
tain attention in the current situation. Boredom is outcome of (a) forcing not to do the
desirable action or (b) forced to do undesirable action.

Another recent article [158] in Educational Psychology Review journal, reviewed

the definitions of boredom in educational settings. The article proposed a definition
of boredom from their review of definitions of boredom in educational setting. “State
boredom occurs when an individual experiences both the (objective) neurological state
of low arousal and the (subjective) psychological state of dissatisfaction, frustration, or
disinterest in response to the low arousal.” This definition states that both the state of
low arousal, that is less interest to the outcome of an event, and dissatisfaction with the
outcome of an event leads to boredom.

7.2.1 Definition of Boredom Used in Our Research

From the above overview of boredom, the most common feature in all existing work on
boredom is repetitiveness and monotonous stimulation [148], [119]. The other key features
of boredom are

1. Conflict between whether to continue the current situation or not due to lack of
motivation [13].

2. The student is forced to do the an uninteresting activity. Non-interest occurs when

the student not challenged enough [114], [37].

3. The student is prevented from doing a desirable action or forced to do an undesirable

action [119].

4. The student lost the interest in outcome of the event [158].

126
 Out of these above features Mindspark captures log data related only to repeti-
tiveness, less challenging activity and students’ non-interest to the outcome of the event.
Hence we consider only these features to model boredom.

7.3 Modeling Boredom

Guided by the theoretical definition (Step 1), we construct features from the ITS log data
(Step 2). Based on information from the student log, the the features are constructed
f 1, f 2, ..f n (Step 3). We formulate a logistic regression model for Bi , the boredom index
at ith question based on the constructed features (Step 4). The weights of the logistic
regression classifier are determined during training process (Step 5) with labeled affective
states from student’s self reported data – an independent method to identify the affective
states. The performance of our model is validated by detecting boredom in the test data
and comparing the results with the student’s self reported data (Step 6).

The logistic regression model to detect boredom is given below:

Bi = w0 + w1 ∗ f 1 + w2 ∗ f 2 + w3 ∗ f 3 + ... + wn ∗ f n (7.1)

The weights, w0 , w1 , w2 , ..., wn , in the equation above are determined by the logistic
regression analysis.

7.3.1 Boredom Model for Mindspark

The definitions of boredom are operationalized to construct features from the Mindspark
log data. In this section we describe how we operationalize the definition of boredom for
Mindspark log data and construct the features. The logistic regression model is used to
detect student’s boredom while they interact with the Mindspark.

We created the logistic regression model based on steps given in Figure 7.1.

Step 1. Boredom: We consider that boredom occurs due to following features:

a) repeated activities [148], [119], b) disinterest in outcome of an event [158], and c) less

127
challenging activity [114], [37].

Step 2. Operationalize: We operationalize the following from the Mindspark

log data:

• Repeated activities: In Mindspark a student can get repeated activities in two

following situations

If the student fails in the topic then s/he will get the same identical question.

If the student fails in the subtopic then s/he will get the similar questions with
different numerical values in the question.

• Disinterest: The student interest to the outcome considered to be less, if the student
spent less time to answer the question and less time to read the explanation if the
question is wrongly answered. Hence we consider the response to the current ques-
tion, the time taken to answer the question and time spent to read the explanation
in the last question are considered.

• Less challenging activity: If the student is answering the questions correctly without
much effort (less time) then we consider that the student is not challenged enough.
In Mindspark we consider the student’s performance in last three questions and
the average time taken to answer the last three questions. These parameters will
determine whether the student is challenged less or not.

Step 3. Boredom Variables: We construct the following features from Mindspark

to model Boredom: repeating the same questions Ra1, repeating similar questions Ra2,
the average time spent to respond to the last three questions At, the students’ response
in the last three questions, ai , ai−1 , ai−3 , the time spent to answer the current question
Ti , and the time spent to read the explanation of the last question ETi−1 , are constructed
from the student’s log data. i denotes the current question.

Step 4. Logistic Regression Model: The mathematical model to detect bore-

128
dom for Mindspark data is given in Equation 7.2.

Bi = w0 + w1 ∗ Ra1 + w2 ∗ Ra2 + w3 ∗ At + w4 ∗ ai +

+w5 ∗ ai−1 + w6 ∗ ai−2 + w7 ∗ Ti + w8 ∗ ETi−1 (7.2)

Step 5 and Step 6 of Figure 7.1 are explained in the following sections.

7.4 Experiment Methodology

In this section, we discuss the independent method we used to identify boredom: the
student’s self reported data. We describe sample data, and the metrics used to compare
the results of our boredom in this section.

7.4.1 Self Reporting

Mindspark has an emotToolbar integrated with its user interface. It is optional and can be
activated by mouse-over operation. The student can choose to report his/her emotions at
any time. Minspark will not force the students to report their emotions. The emotToolbar
is shown in Figure 7.2.

Figure 7.2: EmotToolbar integrated with Mindspark user interface to collect students’
emotions. The emote bar is in right side of the figure.

129
 The emotToolbar consists of six options for the students to choose from as

shown in the Figure 7.3.

• Confused

• Bored

• Excited

• Like

• Dislike Figure 7.3: The EmotToolbar

• Option to give comments.

The emotToolbar takes only one option at a time as an input from students, for
example a student cannot choose Bored and Confused at the same time.

Sample

The student’s self reporting on emotToolbar is optional in Mindspark, hence the question
not marked as bored should not be considered as non-bored if there is no input from the
student in the emotToolbar. However, to train a logistics regression classifier, both bored
and non-bored data are required to create a model.

To address the issue mentioned above, we choose only those sessions which have
five or six entries in the emotToolbar. The students who report their emotions (confused,
bored, excited), for two or three times per session may not be reporting all the emotions
they had undergone during their interaction with Mindspark session. Hence, we consider
sessions where the students reported their emotions for five or six times. We collected
90 sessions data, which have reported emotions five or six times, from the Mindspark
database. The sessions include different school boards (ICSE, CBSE) and different schools
across India.

130
7.4.2 Analysis Procedure

We collected 1617 instances of student’s answering the questions in Mindspark. Out of

1617, 442 instances are self reported as boredom (Bored) by students, the remaining
instances are marked as (Non-Bored). The dataset is stratified at questions (instances)
level. Hence, our unit of analysis is the instances where students respond to questions in
Mindspark. We represent the values obtained from the emotToolbar as Bori at the ith
instance, Bori = 0 for non-bored and Bori = 1 for bored.
To create a boredom model using the logistic regression classifier (Step 5 in Figure
7.1 ), we followed the similar steps that were used to model frustration in Chapter 5. The
modified steps to model boredom are given below:

1. To maintain a uniform scale among all the features in the boredom model, we apply
normalization to all the features. We used the following normalization equation:

X − M ean(X)
Xnew =
M ax(X) − M in(X)

Here, Xnew is the normalized value of feature X.

2. We use cross-validation technique [66] in our analysis, to train and validate our
model. In this research, we used the tenfold cross-validation technique [66] for all
the experiments.

3. We use logistic regression analysis to identify the values for weight, by assigning 0
and 1 to represent non-bored and bored, respectively, in training dataset.

4. The trained model is used to detect boredom on test dataset. Validate our detection
by comparing with student’s self reported data.

7.4.3 Metrics to Validate Boredom Model

We use the same metrics used to measure the performance of our frustration model to
measure the performance of the boredom model. However, we briefly discuss the metrics

131
used in this section. The most common metric used in the classification problem is
accuracy [113], precision and recall.

TP + TN
Accuracy = (7.3)
TP + FP + FN + TN

Accuracy measures the ratio of correctly detected instances (TP + TN) to total instances.

TP
P recision = (7.4)
TP + FP

Precision measures the ratio of correctly detected boredom instances (TP) to total number
of detected boredom instances (TP + FP).

TP
Recall = (7.5)
TP + FN

Recall measures the ratio of correctly detected boredom instances (TP) to the actual
number of boredom instances self reported by the students (TP + FN).

Since our dataset is an unbalanced distribution of bored and non-bored, we calcu-

late the F1 score and Cohen’s kappa to measure the performance of our model compared
to random guesses.

7.4.4 Related Works - Detecting Boredom

We discuss the related works in detecting boredom using the data students’ interaction.
We discuss three such systems in this section: AutoTutor [48], Crystal Island [139] and
Cognitive Tutor Algebra I [11]. In AutoTutor, boredom was detected using the data
from log file and also using the data from external sensors such as body posture, and eye
tracking [44]. In this section we discuss only the results of boredom detection using the
data from log file.

AutoTutor [48] is a dialogue-based tutoring system. The affective states detected

are frustration, boredom, flow and confusion. The features are identified based on the
features from the log file, such as, response time, number of characters in a student’s

132
response, number of words in student’s response, change in student’s score and tutor
feedback to a student’s response. The data from 28 college students’ interaction with
AutoTutor were considered for the analysis. Independent methods used in this research
to detect affective states and validate the results were self reporting, peer reporting and
human observation of students’ facial expressions. Affective states were detected for every
20-second interval and the results are compared with the affective states from indepen-
dent method. This study reports individual detection accuracy of boredom, confusion,
flow, and frustration, when compared with neutral affect are 69%, 68%, 71%, and 78%,
respectively. The detection accuracy for boredom is 69%.

Affective state modeling in Crystal Island [139], creates a Dynamic Bayesian Net-
work (DBN) model to capture the students’ affective states. The affective states detected
in this system was anxious, bored, confused, curious, excited, focused, and frustrated.
The features considered to model students affective states are personal attributes of stu-
dents which are identified from students’ scores and personal surveys prior to interaction
with system, observable environment variables such as goals completed, books viewed
and successful tests, and appraisal variables which are student’s cognitive appraisals such
as learning focus and performance focus. The data from 260 students’ interactions with
Crystal Island was considered to model the affective states. Naive Bayes, Bayes net and
dynamic Bayesian net were used to model the affective states. The independent method
used in this research to validate the results is self reporting. The student’s were asked to
self-report their current mood in the dialog box for every seven minutes. The students
were asked to select the emotion from set of seven emotions which are anxious, bored,
confused, curious, excited, focused, and frustrated. The individual detection accuracy
reported for anxious, bored, confused, curious, excited, focused, and frustrated are 2%,
18%, 32%, 38%, 19%, 52% and 28%, respectively. The detection accuracy for boredom is
18%.

Cognitive Tutor Algebra I [11] is a algebra tutor used is schools as part of regular

133
mathematics curriculum. The affective states detected in this system are boredom, con-
fusion, engaged concentration, and frustration. Total of 58 features were created from the
students’ behavior during interaction with system and past performance. The features are
collected for every 20 second window. The features like student response to the current
question, help request, and previous performance in current skill were considered to de-
tect the affective states. The data from 89 students’ interaction with the Cognitive Tutor
Algebra I are considered for the analysis. Independent method used in this research to
detect affective states and validate the results was human observation of students’ facial
expressions. Affective states were detected for every 20-second interval and the results
are compared with the affective states from independent method. The Kappa score was
used to measure the goodness of detection model for each affective states. The reported
Kappa scores were 0.31 for engaged concentration, 0.40 for confusion, 0.23 for frustration,
and 0.28 for boredom. For boredom, the best algorithm reported was Nave Bayes with
Kappa = 0.28.

From the above related works the detection accuracy of boredom is comparatively
lesser than detection accuracy of frustration. The best detection accuracy of boredom is
69% as reported in [48]. The best reported Kappa value of boredom model is 0.28 [11].

7.5 Results

In this section we describe the performance our boredom model. The extracted features
are applied to weka [108]. The logistic regression model to classify boredom is:

Bi = −0.66 + 0.05 ∗ Ra1 − 0.06 ∗ Ra2 + 0.94 ∗ At − 0.48 ∗ ai +

−0.27 ∗ ai−1 + 0.314 ∗ ai−2 + 7.6 ∗ Ti − 2.73 ∗ ETi−1 (7.6)

The performance of our boredom model on the Mindspark data, using tenfold
cross-validation, compared to self reported observation is given in Table 7.1.

134
 Table 7.1: Contingency Table of Our Approach when Applied to Mindspark Log Data
Self Reported Data
Bored Non-Bored
Pred Bored 98 46
Result Non-Bored 344 1129

The values from Table 7.1 are used to calculate the performance of our model. The
results are given in Table 7.2.

Table 7.2: Performance of our Approach Shown Using Various Metrics when Applied to
Mindspark Log Data
Metrics Results
Accuracy 75.88%
Precision 68.1%
Recall 22.22%
Cohen’s kappa 0.23
F1 Score 0.33

From the results, the accuracy and precision of our boredom model are high, com-
pared to recall.

7.5.1 Discussion

Our boredom model performed relatively better compared to the existing research work
in accuracy with 75.88%. Precision and recall are not reported in the existing research
work. The value of F1 score and Kappa in our approach indicates that our model can be
improved.

Moreover, compared to our frustration model, boredom model performed poorly

in accuracy with 75% vs 88%, in precision 68% vs 79% and in recall 22% vs 32%. The
reason for poor performance can be attributed to consideration of few factors of boredom
in operationalization and validity of self reporting. The students’ age is in the range of
10-12, due to this, the students might have wrongly reported other emotion as boredom
or vice versa. The advantage of our model is that the features identified gives the reasons

135
for students’ boredom and it is useful while addressing the student’s boredom.

7.6 Conclusion

To check the generalization of our theory-driven approach to model other affective states,
in this chapter, we applied our theory-driven approach to detect boredom of the students
while working with an ITS. The process shows that our theory-driven approach is general-
izable to model other affective states and not only frustration. However, the performance
of the model is lesser compared to the theory-driven frustration model. The reason for the
lower precision, recall and kappa can attributed to the operationalization of features from
the Mindspark log data and independent method used to detect boredom. Improving
the operationalization of the theory for the ITS, and independent method such as human
observation might improve the performance of the boredom model. However, our main
goal of theory-driven approach, which is understanding the reasons for boredom, is shown
in our model. Our boredom model gives the reasons for students’ boredom and it can be
useful to respond to boredom.

136
 Chapter 8

Summary and Conclusion

In this chapter we summarize our research work with results, contributions and describe
possible future works from our research.

8.1 Summary

In this research work, we propose to detect and respond to students’ frustration while
working with an ITS. We developed a frustration model using the features constructed
from the log data by applying theoretical definitions of frustration. We implemented our
model in an ITS, Mindspark. By modeling frustration using theoretical definitions, the
reasons for frustration can be inferred. Since the reasons for frustration is known, it led to
informed responding to frustration. The data from independent method (human obser-
vation) was used to create and validate the frustration model. The results are described
in Chapter 5.

In our research, what is important is how accurately the frustration instances are
detected rather than detecting all frustration instances encountered by the students. Our
theory-driven approach performed comparatively better than other approaches, with a
precision of 78.94% (the best result from the data-mining approach with a precision
of 65.97%) and comparatively equal in accuracy with 88.84% (the best result from the
data-mining approach is with an accuracy of 88.63%). Hence, our goal of achieving the
best precision is achieved in our theory-driven approach. However, the theory-driven

137
approach performed poorly in recall of 32.85% (the best result from the data-mining
approach with a recall of 56.2%). The reason for better precision and poor recall could
be that our frustration model detected the students’ frustration which occurred only due
to interaction with Mindspark not the frustrations which occurred due to external factors
such as Internet speed and interaction with friends.

In order to apply our frustration model to other systems, careful thought is required
to operationalize the factors that block the goals. Moreover, the goals of the students when
they interact with the system should be determined. These are the limitations in the
scalability of our model. The results of the frustration model are dependent on how well
the goals are captured and how well the blocking factors of the goals are operationalized.
Another limitation of our model is, we used the data from class six students from two
schools. Hence, our model can be generalized only for the students with the age group of
10–13, across Mindspark.

Our theory-driven approach can be generalized to detect other affective states. In

Chapter 7, we applied our theory-driven approach to detect boredom of students while
they interact with the ITS. The results suggest that our model can be generalized to
detect other affective states in ITS. The results of detecting boredom using our theory-
driven approach in chapter 7, are comparatively equal to the existing methods. However,
the kappa value (k = 0.23) indicates poor performance of our approach when applied to
detect boredom. The reason for the lower kappa value can be attributed to the theo-
ries of boredom considered to operationalize the features from the Mindspark log data
and the independent method used to detect boredom. More research on improving the
operationalization of the theory for the ITS and independent methods, such as human
observation, will improve the performance of the boredom model. Our boredom model
gives an interpretation of the reasons for students’ boredom and it can be useful to provide
informed responses.

The aim of our research is to avoid the negative consequences of frustration (de-
tected by our model) in real-time, at Mindspark’s production environment. To achieve our

138
aim, we integrated our model with the Mindspark code to check the students’ frustration
index after every response and trigger an appropriate algorithm whenever the frustration
index is above the threshold value (0.5). We developed the algorithms to respond to the
frustrations in a timely manner using motivational messages, during the students’ inter-
action with the ITS. The algorithms are developed based on the reasons for frustration,
the time taken to answer the question, the difficulty level of the question and the number
of frustration instance in that session by the student.

The results of our strategies to respond to frustration are described in Chapter 6.

The results show that the number of frustration instances is reduced due to motivational
messages. The reduction in the number of frustration instances per session is statistically
significant. Our approach can be generalized to schools across Mindspark and it is inde-
pendent of schools analyzed and Mindspark topics used in the Mindspark sessions. Our
approach to respond to frustration has a relatively higher impact on the students whose
performance in the sessions is low and the students who spend more time in answering
the questions in the Mindspark session.

The results provide an evidence that the approach, detecting and responding to
frustration, significantly reduces the students’ frustration. The reduction in students’
frustration can lead the students to continue learning instead of dropping out the session.
Moreover, the motivational messages help the students to create new goals while inter-
acting with the ITS, thus enabling to avoid the negative consequences of frustration, such
as dropping out of Mindspark.

8.2 Contributions

In this section, we list the contributions from our research work. The major contributions
from our research work are a theory-driven approach to detect affective states, a frustra-
tion model to detect frustration with the reasons for that state, an approach to respond
to the frustration and an insight into the impact of motivational messages to frustration
instances.

139
• Theory-driven Approach: We developed an approach to detect affective states using
data from the students’ interaction with the system. Our approach uses only the
data from log files, hence, it can be implemented in the large scale deployment of
ITS. We have tested our approach on a math ITS to detect frustration. Moreover,
we validated the likelihood of generalizing the theory-driven approach to detect
other affective states by creating a model to detect boredom in an ITS.

• Frustration Model: We developed a linear regression model to detect frustration

in a math ITS – Mindspark, using the theory-driven approach. The theoretical
definition of frustration is operationalized for the ITS log data. The detection
accuracy of our model is comparatively equal to the existing approaches to detect
frustration. Additionally, our model provides the reasons for the frustration of the
students. Based on our results, the linear regression model with a ten-fold cross-
validation method performs relatively better than other complex classifier models
like Support Vector Machine (SVM) and Logistic Model Tree (LMT).

• Respond to Frustration: We provided an approach to avoid the negative conse-

quences of frustration, such as dropping out, by using the motivational messages.
The messages to respond to frustration are created based on the reasons for frus-
tration. The software code to detect and respond to frustration is added to the
Mindspark’s adaptive logic. Students’ frustration instances were detected and mo-
tivational messages were provided to avoid frustration in real-time. The impact of
motivational messages was analyzed and it was found that our approach significantly
reduced the number of frustrations per session.

• Impact of Motivational Messages: We provide that the motivational messages to re-

spond to frustration had a relatively high impact on the following kinds of students:
(a) those who spent more time in answering the questions; (b) those who had a low
average score in Mindspark sessions; (c) those who had a high number of frustration
instances per session, before implementing the algorithm.

140
8.3 Future Work

In this section, we discuss the possible future work from our research. The typical exten-
sion of our theory-driven approach is to detect other affective states like confusion and
surprise using theoretical definitions and to study the interaction among affective states.
Later, we aim to create a comprehensive model to detect all cognitive affective states
of the students for adaptation. The model will help in detecting the students’ affective
state and provide a possibility of adapting the learning content based on specific affec-
tive states, thus improving the students’ interaction with the ITS/Gaming system, using
learning analytics.

In order to create a comprehensive model, the first step will be based on our
research outcome–that a theory-driven model can perform equal to data-driven models
and also show the reasons for frustration. We plan to apply the theory-driven research
method to other cognitive affective states such as boredom, confusion and delight and
in turn to identify the features associated with them. We also plan to detect the other
affective states using an independent method such as human observation, self reporting
and the like. Using the features extracted and the independent data of emotions, we
plan to train the theoretical model to identify affective states. The next step will be
the forming of the Bayesian network for each of the affective states using the associated
features. The interaction among affective states can be modeled as a transition matrix
of affective states. Using the transition matrix and features associated with the affective
states, one can model the Hidden Markov Model (HMM) for cognitive affective states. In
HMM, the affective state at time n is dependent on the affective state at time n-1 and
the features f1, f2, f3 of time n. Hence, the detection accuracy of affective states will
be improved. HMM will provide details about the interaction between affective states
and the features associated with each affective state. Hence, the adaptation logic can be
enhanced for the better performance.

The second possible extension of research work is to investigate whether adapting

the learning content based on students’ affective states improves the students’ learning or

141
not. The first step in this research will be, detecting different cognitive affective states
like boredom, confusion, engagement and the like. The next step will be to perform the
correlation analysis of each of the affective states with the student learning. If the results
suggest that there is a negative correlation between negative affective states like confusion
and boredom, then we aim to change the adaptation logic or create strategies to cope with
these affective states. Lastly, we plan to analyze whether adapting the learning content
or strategies to avoid negative affective states has an impact on student learning or not.
Another possible extension to our research is developing a hybrid system which
includes web-cam, eye-movement tracking system, pressure sensitive chairs, etc. which
are not expensive and non-intrusive to capture the student’s goals and emotions. Such
a type of system can improve the performance of the theory-driven approach and would
address the limitation of our approach in scalability.
Specific to Mindspark, the course of future work could be the generalization of the
frustration model in Mindspark for the different age groups. For this, more data has to
be collected from students of different age groups and schools. We also aim to work on
detecting different affective states using our theory-driven approach in Mindspark and
changing the Mindspark adaptation logic to help students to avoid these affective states.

142
 Appendix

Ethics Documents

Ethics Committee approval, school and company permission for data collection, consent
forms for students and parents, and explanation statement for students and general au-
dience are given in this section.

143
Monash University Human Research Ethics Committee (MUHREC)
Research Office

Human Ethics Certificate of Approval

Date: 20 October 2011

Project Number: CF11/2358 - 2011001355

Project Title: Enriching student model in intelligent tutoring system

Chief Investigator: Dr Campbell Wilson

Approved: From: 20 October 2011 to 20 October 2016

Terms of approval
1. The Chief investigator is responsible for ensuring that permission letters are obtained, if relevant, and a copy
forwarded to MUHREC before any data collection can occur at the specified organisation. Failure to provide
permission letters to MUHREC before data collection commences is in breach of the National Statement on
Ethical Conduct in Human Research and the Australian Code for the Responsible Conduct of Research.
2. Approval is only valid whilst you hold a position at Monash University.
3. It is the responsibility of the Chief Investigator to ensure that all investigators are aware of the terms of approval
and to ensure the project is conducted as approved by MUHREC.
4. You should notify MUHREC immediately of any serious or unexpected adverse effects on participants or
unforeseen events affecting the ethical acceptability of the project.
5. The Explanatory Statement must be on Monash University letterhead and the Monash University complaints clause
must contain your project number.
6. Amendments to the approved project (including changes in personnel): Requires the submission of a
Request for Amendment form to MUHREC and must not begin without written approval from MUHREC.
Substantial variations may require a new application.
7. Future correspondence: Please quote the project number and project title above in any further correspondence.
8. Annual reports: Continued approval of this project is dependent on the submission of an Annual Report. This is
determined by the date of your letter of approval.
9. Final report: A Final Report should be provided at the conclusion of the project. MUHREC should be notified if the
project is discontinued before the expected date of completion.
10. Monitoring: Projects may be subject to an audit or any other form of monitoring by MUHREC at any time.
11. Retention and storage of data: The Chief Investigator is responsible for the storage and retention of original data
pertaining to a project for a minimum period of five years.

Professor Ben Canny

Chair, MUHREC

cc: Dr Judithe Sheard; Dr Sridhar Iyer; Dr Sahana Murthy; Mr Ramkumar Rajendran

Postal – Monash University, Vic 3800, Australia

Building 3E, Room 111, Clayton Campus, Wellington Road, Clayton
Telephone +61 3 9905 5490 Facsimile +61 3 9905 3831
Email [email protected] www.monash.edu/research/ethics/human/index/html
ABN 12 377 614 012 CRICOS Provider #00008C

144
145
146
 Consent Form - Students

Title: Enriching student model in Intelligent Tutoring System

NOTE: This consent form will remain with the Monash University researcher for their records

I agree to take part in the Monash University research project, “Enriching student model in Intelligent
Tutoring System”. I have had the project explained to me, and I have read the Explanatory
Statement, which I will keep for my records. I understand that agreeing to take part means that:

I agree to allow the researcher to video-tape my facial expressions while interacting with Mindspark, ,
a math system which I use in my math class. Yes No
I agree to record my use of Mindspark in a computer log file Yes No

and

I understand that my participation is voluntary, that I can choose not to participate in the project, and
that I can stop at any stage of the project without being penalised or disadvantaged in any way.

and

I understand that any data that the researcher extracts from the video-tape/ computer log files for use
in reports or published findings will not, under any circumstances, contain my name or identifying
characteristics.

and

I understand that data from the video–tape and computer log files will be kept in a secure storage and
accessible only to the research team. I also understand that the data will be destroyed after a 5 year
period.

Participant’s name
Signature

Date

147
 Consent Form - Parents

Title: Enriching student model in Intelligent Tutoring System

NOTE: Signed written consent will remain with the Monash University researcher for their records.

I agree that _________________________ (insert full name of participant) may take part in the
Monash University research project, “Enriching student model in Intelligent Tutoring System”.
The project has been explained to _______________ (insert name of participant) and to me,
and I have read the Explanatory Statement, which I will keep for my records.

I understand that agreeing to take part means that I am willing to allow __________ (insert full name
of participant) to:

 Video-tape his/her facial expressions while interacting with Mindspark, a math Intelligent
Tutoring System used in his/her regular math class Yes No
 Record the interactions with Mindspark, in a computer log file Yes No

Participant’s name:

Participant’s Age:

Parent’s / Guardian’s Name:

Parent’s / Guardian’s relationship to participant:

Parent’s / Guardian’s Signature:

Date

148
Date
Explanatory Statement - parents

Title: Enriching student model in Intelligent Tutoring System

This information sheet is for you to keep.

My name is Ramkumar Rajendran and I am conducting a research project with Dr Campbell Wilson, a senior lecturer
in the Faculty of Information technology, Monash University, and Dr. Sridhar Iyer, Assoc Prof in computer science and
engineering department of IIT Bombay, towards a PhD at IITB-Monash Research Academy, a joint program of Indian
Institute of Technology Bombay, India and Monash University, Australia. This means that I will be writing a thesis
which is the equivalent of a 250 page book.

Please read this Explanatory Statement in full before making a decision.

I am looking for the students who use Mindspark, a math Intelligent Tutoring System (ITS), in their school as part of
their curriculum. We obtained the information about the users of Mindspark from the company “Educational Initiatives
(EI)”, India.

The aim of the research

The aim of this research is to identify a set of affective states (namely confusion, frustration and boredom) from data
collected on computer log files. The log files contain data from the students’ interaction with ITS.
The affective states identified from the log file data will be validated through comparison with the results obtained from
observation of students’ facial expressions while they interact with ITS.

Possible benefits
Participants will not directly benefit from this research. If our model identifies the affective states automatically only
using the student interaction with ITS, it will help to improve the function of ITS thereby student learning experience.

Participation Involves:
The study involves video recording the participants’ facial expressions while they interact with ITS.
The participants’ interaction with ITS is recorded in computer log files.

Time for this research:

Participants do not need to spend extra time for this research; participants are video recorded during their one regular
math tutoring session which will be 40 minutes.

Participation and Withdrawal:

Participation in this study is completely voluntary and the participant is free to choose whether to be or not part of this
study. Parents can decide whether their child participates in this study or not. If parents consent to their child
participating, they may withdraw their child from the study at any stage. Furthermore, if parents give consent to their
child participating, then their child can withdraw from the study at any stage.

Confidentiality
We obtain student information as de-identified data from database, so personal information will not be stored in the
data.
Storage of data
Data collected will be stored in accordance with Monash University regulations, kept on University premises, in a
locked room for 5 years. A report of the study may be submitted for publication in scientific conferences, but individual
participants will not be identifiable in such a report.

149
Results
If you would like to be informed of the aggregate research finding, please contact Ramkumar Rajendran on +61 3 990
31217 or email [email protected].

If you would like to contact the researchers about any If you have a complaint concerning the manner in
aspect of this study, please contact the Chief which this research CF11/2358 - 2011001355 is being
Investigator: conducted, please contact:
Dr Campbell Wilson Executive Officer
Deputy Head of School, Monash University Human Research Ethics
Faculty of Information Technology, Committee (MUHREC)
Caulfield Campus, Monash University Building 3e Room 111
PO Box 197, Caulfield East Victoria 3145, Australia. Research Office
Email: [email protected] Monash University VIC 3800
Phone: +61 3 990 31142
Tel: +61 3 9905 2052 Fax: +61 3 9905 3831
Email: [email protected]

Thank you.

Ramkumar Rajendran

Caulfield school of IT
Faculty of Information Technology
PO Box 197, Caulfield East Victoria 3145, AustraliaBuiling H 7.24, Caulfield Campus, Monash University
Telephone +61 3 9903 1217 Facsimile +61 3 9903 1077
Email [email protected] Web www.infotech.monash.edu.au/about/staff/Ramkumar-Rajendran
ABN 12 377 614 012 CRICOS provider number 00008C

150
Date

Explanatory Statement - Students

Title: Enriching student model in Intelligent Tutoring System

This information sheet is for you to keep.

1. My name is Ramkumar Rajendran and I am conducting a research project with Dr Campbell Wilson,
a senior lecturer at Monash University, Australia and Associate Professor Sridhar Iyer, IIT Bombay,
India. This research is for my PhD studies at IITB-Monash Research Academy.
2. Please read this Explanatory Statement in full before making a decision.

3. We are asking you take part in a research study because we are trying to automatically identify
emotions while you use Mindspark in your math class.
4. If you agree to be in this study,
a. We will record your facial expressions while you use Mindspark.
b. Your use of Mindspark will be recorded on the computer.

5. We will conduct this study in your regular math class. It will not take any extra time to participate in
this study.

6. Please talk to your parents before you decide whether or not to participate. We also explained this
research study to your parents and asked them to give permission for you to take part of this study.
But even if your parents agree, you can still decide not to do this.

7. If you don’t want to be in this study, you don’t have to participate. There is no compulsion, and no
one will upset if you do not participate. However, if you do agree to participate, you may stop
participation at any time during the session.

8. If you have questions, you can ask your math teacher. If you have any question later that you did
not think of now, you can call me on +61 3 9903 1217 or email me at [email protected].

151
If you would like to contact the researchers about any If you have a complaint concerning the manner in
aspect of this study, please contact the Chief which this research CF11/2358 - 2011001355 is being
Investigator: conducted, please contact:
Dr Campbell Wilson Executive Officer
Deputy Head of School, Monash University Human Research Ethics
Faculty of Information Technology, Committee (MUHREC)
Caulfield Campus, Monash University Building 3e Room 111
PO Box 197, Caulfield East Victoria 3145, Australia. Research Office
Email: [email protected] Monash University VIC 3800
Phone: +61 3 990 31142
Tel: +61 3 9905 2052 Fax: +61 3 9905 3831
Email: [email protected]

Thank you.

Ramkumar Rajendran

Caulfield school of IT
Faculty of Information Technology
PO Box 197, Caulfield East Victoria 3145, Australia
Builing H 7.24, Caulfield Campus, Monash University
Telephone +61 3 9903 1217 Facsimile +61 3 9903 1077
Email [email protected] Web www.infotech.monash.edu.au/about/staff/Ramkumar-Rajendran
ABN 12 377 614 012 CRICOS provider number 00008C

152
 Bibliography

[1] Mindspark: Improve your math skills. http://www.mindspark.in/. Accessed Feb 20, 2014.

[2] M. Adibuzzaman, N. Jain, N. Steinhafel, M. Haque, F. Ahmed, S. I. Ahamed, and R. Love.

Towards in situ affect detection in mobile devices: a multimodal approach. In Proceedings of the
2013 Research in Adaptive and Convergent Systems, pages 454–460. ACM, 2013.

[3] L. Agnieszka. Affect-awareness framework for intelligent tutoring systems. In Human System
Interaction (HSI), 2013 The 6th International Conference on, pages 540–547. IEEE, 2013.

[4] C. O. Alm, D. Roth, and R. Sproat. Emotions from text: machine learning for text-based emotion
prediction. In Proceedings of the conference on Human Language Technology and Empirical Methods
in Natural Language Processing, pages 579–586. Association for Computational Linguistics, 2005.

[5] A. Amsel. Frustration theory: Many years later. Psychological bulletin, 112(3):396–399, 1992.

[6] J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier. Cognitive tutors: Lessons

learned. The journal of the learning sciences, 4(2):167–207, 1995.

[7] H. Anger Elfenbein and N. Ambady. On the universality and cultural specificity of emotion recog-
nition : A meta-analysis. Psychology Bulletin, 128(2):203–235, 2002.

[8] I. Arroyo, D. G. Cooper, W. Burleson, B. P. Woolf, K. Muldner, and R. Christopherson. Emotion

sensors go to school. In Artificial Intelligence in Education, volume 200, pages 17–24, 2009.

[9] I. Arroyo and B. P. Woolf. Inferring learning and attitudes from a bayesian network of log file data.
In AIED, pages 33–40, 2005.

[10] J. W. Atkinson. An introduction to motivation. Van Nostrand, 1964.

[11] R. S. J. d. Baker, S. M. Gowda, M. Wixon, J. Kalka, A. Z. Wagner, A. Salvi, V. Aleven, G. W.

Kusbit, J. Ocumpaugh, and L. Rossi. Towards sensor-free affect detection in cognitive tutor algebra.
International Educational Data Mining Society, 2012.

[12] T. Bänziger, D. Grandjean, and K. R. Scherer. Emotion recognition from expressions in face, voice,

153
 and body: the multimodal emotion recognition test (mert). Emotion, 9(5):691–704, 2009.

[13] J. E. Barmack. The effect of benzedrine sulfate (benzyl methyl carbinamine) upon the report of
boredom and other factors. The Journal of Psychology, 5(1):125–133, 1938.

[14] M. S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan. Fully automatic
facial action recognition in spontaneous behavior. In 7th International Conference on Automatic
Face and Gesture Recognition., pages 223–230, 2006.

[15] S. Batool, M. I. Yousuf, and Q. Parveen. A study of attribution patterns among high and low
attribution groups: An application of weiners attribution theory. Anthropologist, 14(3):193–197,
2012.

[16] C. Beal, J. Beck, and B. Woolf. Impact of intelligent computer instruction on girls math self
concept and beliefs in the value of math. In Poster presented at the annual meeting of the American
Educational Research Association, San Diego, 1998.

[17] N. Bianchi-Berthouze and C. L. Lisetti. Modeling multimodal expression of users affective subjective
experience. User Modeling and User-Adapted Interaction, 12(1):49–84, 2002.

[18] K. Brawner and B. Goldberg. Real-time monitoring of ecg and gsr signals during computer-based
training. In Intelligent Tutoring Systems, pages 72–77, 2012.

[19] E. J. Brown, T. J. Brailsford, T. Fisher, and A. Moore. Evaluating learning style personaliza-
tion in adaptive systems: Quantitative methods and approaches. IEEE Transactions on Learning
Technologies, 2:10–22, 2009.

[20] J. S. Brown, R. R. Burton, and A. G. Bell. Sophie: A step toward creating a reactive learning
environment. International Journal of Man-Machine Studies, 7(5):675–696, 1975.

[21] P. Brusilovsky. Adaptive hypermedia. User modeling and user-adapted interaction, 11(1-2):87–110,
2001.

[22] P. Brusilovsky and E. Milln. User models for adaptive hypermedia and adaptive educational
systems. In The Adaptive Web, LNCS 4321, pages 3–53, 2007.

[23] P. Brusilovsky, E. Schwarz, and G. Weber. Elm-art: An intelligent tutoring system on world wide
web. In Intelligent tutoring systems, pages 261–269, 1996.

[24] R. R. Burton and J. S. Brown. An investigation of computer coaching for informal learning activ-
ities. International Journal of Man-Machine Studies, 11(1):5–24, 1979.

[25] R. A. Calvo, I. Brown, and S. Scheding. Effect of experimental factors on the recognition of affective

154
 mental states through physiological measures. In AI 2009: Advances in Artificial Intelligence, pages
62–70. Springer, 2009.

[26] R. A. Calvo and S. K. D’Mello. Affect detection: An interdisciplinary review of models, methods,
and their applications. IEEE Transactions on Affective Computing, 1:18–37, 2010.

[27] N. Capuano, M. Marsella, and S. Salerno. Abits: An agent based intelligent tutoring system
for distance learning. In Proceedings of the International Workshop on Adaptive and Intelligent
Web-Based Education Systems, ITS, 2000.

[28] J. S. Carriere, J. A. Cheyne, and D. Smilek. Everyday attention lapses and memory failures: The
affective consequences of mindlessness. Consciousness and cognition, 17(3):835–847, 2008.

[29] R. Caruana and A. Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In

Proceedings of the 23rd international conference on Machine learning, pages 161–168. ACM, 2006.

[30] L.-H. Chen, Y.-C. Lai, and Y.-H. Weng. Intelligent e-learning system with personalized miscon-
ception diagnose and learning path guidance. In International Conference on Electronic Business,
2009.

[31] W. J. Clancey. Tutoring rules for guiding a case method dialogue. International Journal of Man-
Machine Studies, 11(1):25–49, 1979.

[32] W. J. Clancey. Intelligent tutoring systems: A tutorial survey. Technical report, DTIC Document,
1986.

[33] C. Conati and H. Maclaren. Empirically building and evaluating a probabilistic model of user
affect. User Modeling and User-Adapted Interaction, 19(3):267–303, 2009.

[34] D. Cooper, I. Arroyo, B. Woolf, K. Muldner, W. Burleson, and R. Christopherson. Sensors model
student self concept in the classroom. In User Modeling, Adaptation, and Personalization, pages
30–41. Springer, 2009.

[35] A. T. Corbett, K. R. Koedinger, and J. R. Anderson. Intelligent tutoring systems. Handbook of

humancomputer interaction, pages 849–874, 1997.

[36] M. Coulson. Attributing emotion to static body postures: Recognition accuracy, confusions, and
viewpoint dependence. Journal of nonverbal behavior, 28(2):117–139, 2004.

[37] M. Csikszentmihalyi. Finding flow: The psychology of engagement with everyday life. Basic Books,
1997.

[38] R. S. J. d Baker, A. T. Corbett, and V. Aleven. More accurate student modeling through contextual

155
 estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring
Systems, pages 406–415. Springer, 2008.

[39] R. S. J. d. Baker, S. K. D’Mello, M. M. T. Rodrigo, and A. C. Graesser. Better to be frustrated

than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during
interactions with three different computer-based learning environments. International Journal of
Human-Computer Studies, 68(4):223 – 241, 2010.

[40] T. Danisman and A. Alpkocak. Feeler: Emotion classification of text using vector space model.
In AISB 2008 Convention Communication, Interaction and Social Intelligence, volume 2, pages
53–59, 2008.

[41] C. Darwin. The expression of the emotions in man and animals. Oxford University Press, 1998.

[42] E. L. Deci and R. M. Ryan. Self-Determination. Wiley Online Library, 2010.

[43] C. Dede, M. C. Salzman, and R. B. Loftin. Sciencespace: Virtual realities for learning complex and
abstract scientific concepts. In Virtual Reality Annual International Symposium, 1996., Proceedings
of the IEEE 1996, pages 246–252, 1996.

[44] S. DMello, R. Picard, and A. Graesser. Towards an affect-sensitive autotutor. IEEE Intelligent
Systems, 22(4):53–61, 2007.

[45] S. K. D’Mello, S. D. Craig, K. Fike, and A. C. Graesser. Responding to learners’ cognitive-

affective states with supportive and shakeup dialogues. In Human-Computer Interaction. Ambient,
Ubiquitous and Intelligent Interaction, pages 595–604, 2009.

[46] S. K. D’Mello, S. D. Craig, B. Gholson, and S. Franklin. Integrating affect sensors in an intelligent
tutoring system. In In Affective Interactions: The Computer in the Affective Loop Workshop at
2005 Intl. Conf. on Intelligent User Interfaces, 2005, pages 7–13. AMC Press, 2005.

[47] S. K. D’Mello, S. D. Craig, J. Sullins, and A. C. Graesser. Predicting affective states expressed
through an emote-aloud procedure from autotutor’s mixed-initiative dialogue. Int. J. Artif. Intell.
Ed., 16:3–28, January 2006.

[48] S. K. D’Mello, S. D. Craig, A. Witherspoon, B. Mcdaniel, and A. Graesser. Automatic detection of

learner’s affect from conversational cues. User Modeling and User-Adapted Interaction, 18:45–80,
February 2008.

[49] S. K. D’Mello and A. C. Graesser. Automatic detection of learner’s affect from gross body language.
Applied Artificial Intelligence, 23(2), 2009.

156
[50] J. Dollard, N. E. Miller, L. W. Doob, O. H. Mowrer, and R. R. Sears. Frustration and aggression.
1939.

[51] G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski. Classifying facial actions.
Pattern Analysis and Machine Intelligence, IEEE Transactions on, 21(10):974–989, 1999.

[52] T. Dragon, I. Arroyo, B. P. Woolf, W. Burleson, R. Kaliouby, and H. Eydgahi. Viewing student
affect and learning through classroom observation and physical sensors. In Proceedings of the 9th
international conference on Intelligent Tutoring Systems, ITS ’08, pages 29–39, 2008.

[53] C. S. Dweck. Motivational processes affecting learning. American psychologist, 41(10):1040–1048,

1986.

[54] C. S. Dweck. Messages that motivate: How praise molds students’ beliefs, motivation, and perfor-
mance (in surprising ways). In Improving Academic Achievement, pages 37 – 60. Academic Press,
2002.

[55] P. Ekman. Universals and cultural differences in facial expressions of emotion. In Nebraska sym-
posium on motivation. University of Nebraska Press, 1971.

[56] P. Ekman. Strong evidence for universals in facial expressions: a reply to Russell’s mistaken critique.
Psychology Bulletin, 115(2):268–287, 1994.

[57] P. Ekman and W. V. Friesen. Unmasking the face: A guide to recognizing emotions from facial
clues. Malor Books, 2003.

[58] R. El Kaliouby and P. Robinson. Real-time inference of complex mental states from facial expres-
sions and head gestures. Springer, 2005.

[59] C. Eliot and B. P. Woolf. Reasoning about the user within a simulation-based real-time training
system. In Fourth International Conference on User Modeling, Hyannis Cape Cod, Mass, pages
15–19, 1994.

[60] P. C. Ellsworth and K. R. Scherer. Appraisal processes in emotion. Handbook of affective sciences,
pages 572–595, 2003.

[61] H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In Proceedings of the 33rd
international ACM SIGIR conference on Research and development in information retrieval, pages
34–41. ACM, 2010.

[62] F. Försterling. Attributional retraining: A review. Psychological Bulletin, 98(3):495, 1985.

[63] M. Gagné and E. L. Deci. Self-determination theory and work motivation. Journal of Organizational

157
 behavior, 26(4):331–362, 2005.

[64] J. P. Gee. What video games have to teach us about learning and literacy. Computers in Enter-
tainment (CIE), 1(1):20–23, 2003.

[65] J. P. Gee. Learning and games. The ecology of games: Connecting youth, games, and learning,
3:21–40, 2008.

[66] S. Geisser. Predictive Inference: An Introduction. Monographs on Statistics and Applied Proba-
bility. Chapman & Hall, 1993.

[67] T. Gordon. PET: Parent Effectiveness Training. New American Library, 1970.

[68] A. C. Graesser, P. Chipman, B. C. Haynes, and A. Olney. Autotutor: An intelligent tutoring

system with mixed-initiative dialogue. Education, IEEE Transactions on, 48(4):612–618, 2005.

[69] A. C. Graesser, K. VanLehn, C. P. Rosé, P. W. Jordan, and D. Harter. Intelligent tutoring systems
with conversational dialogue. AI magazine, 22(4):39–51, 2001.

[70] A. C. Graesser, K. Wiemer-Hastings, P. Wiemer-Hastings, and R. Kreuz. Autotutor: A simulation

of a human tutor. Cognitive Systems Research, 1(1):35–51, 1999.

[71] J. F. Grafsgaard, J. B. Wiggins, K. E. Boyer, E. N. Wiebe, and J. C. Lester. Automatically

recognizing facial expression: Predicting engagement and frustration. In Proceedings of the 6th
International Conference on Educational Data Mining, 2013.

[72] S. Graham. A review of attribution theory in achievement contexts. Educational Psychology Review,
3(1):5–39, 1991.

[73] S. Graham, B. Weiner, D. Berliner, and R. Calfee. Theories and principles of motivation. Handbook
of educational psychology, 4:63–84, 1996.

[74] H. Gunes and M. Piccardi. Bi-modal emotion recognition from expressive face and body gestures.
Journal of Network and Computer Applications, 30(4):1334–1345, 2007.

[75] D. F. Halpern, J. Aronson, N. Reimer, S. Simpkins, J. R. Star, and K. Wentzel. Encouraging girls
in math and science. 2007.

[76] J. Hartley and D. H. Sleeman. Towards more intelligent teaching systems. International Journal
of Man-Machine Studies, 5(2):215–236, 1973.

[77] F. Heider. The Psychology of Interpersonal Relations. Lawrence Erlbaum Associates, 1958.

[78] J. D. Hollan, E. L. Hutchins, and L. Weitzman. Steamer: An interactive inspectable simulation-

158
 based training system. AI magazine, 5(2):15–27, 1984.

[79] K. Hone. Empathic agents to reduce user frustration: The effects of varying agent characteristics.
Interacting with Computers, 18(2):227–245, 2006.

[80] C.-M. Hong, C.-M. Chen, M.-H. Chang, and S.-C. Chen. Intelligent web-based tutoring system
with personalized learning path guidance. Advanced Learning Technologies, IEEE International
Conference on, 0:512–516, 2007.

[81] C. Hull. Principles of behavior. 1943.

[82] M. S. Hussain, O. AlZoubi, R. A. Calvo, and S. K. D’Mello. Affect detection from multichan-
nel physiology during learning sessions with autotutor. In Proceedings of the 15th international
conference on Artificial intelligence in education, AIED’11, pages 131–138, 2011.

[83] M. S. Hussain, H. Monkaresi, and R. A. Calvo. Categorical vs. dimensional representations in

multimodal affect detection during learning. In Intelligent Tutoring Systems, pages 78–83, 2012.

[84] C. E. Izard. The face of emotion. Appleton-Century-Crofts, 1971.

[85] C. E. Izard. Innate and universal facial expressions: evidence from developmental and cross-cultural
research. Psychological Bulletin, 115(2):288–299, 1994.

[86] M. C. Jadud. An exploration of novice compilation behaviour in BlueJ. PhD thesis, University of
Kent, 2006.

[87] T. Kanade, Y. Tian, and J. F. Cohn. Comprehensive database for facial expression analysis. In Pro-
ceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition,
pages 46–53, 2000.

[88] A. Kapoor, W. Burleson, and R. W. Picard. Automatic prediction of frustration. Int. J. Hum.-
Comput. Stud., 65:724–736, August 2007.

[89] A. Kapoor and R. W. Picard. Multimodal affect recognition in learning environments. In Proceed-
ings of the 13th annual ACM international conference on Multimedia, pages 677–682, 2005.

[90] S. Katz and A. Lesgold. The role of the tutor in computer-based collaborative learning situations.
Computers as cognitive tools, pages 289–317, 1993.

[91] F. A. Khan, S. Graf, E. R. Weippl, and A. M. Tjoa. An approach for identifying affective states
through behavioral patterns in web-based learning management systems. In Proceedings of the
11th International Conference on Information Integration and Web-based Applications & Services,
iiWAS ’09, pages 431–435, 2009.

159
 [92] K. H. Kim, S. Bang, and S. Kim. Emotion recognition system using short-term monitoring of
physiological signals. Medical and biological engineering and computing, 42(3):419–427, 2004.

[93] J. Klein, Y. Moon, and R. W. Picard. This computer responds to user frustration: Theory, design,
and results. Interacting with Computers, 14:119–140, Jan 2002.

[94] K. R. Koedinger, J. R. Anderson, W. H. Hadley, M. A. Mark, et al. Intelligent tutoring goes

to school in the big city. International Journal of Artificial Intelligence in Education (IJAIED),
8:30–43, 1997.

[95] B. Kort, R. Reilly, and R. W. Picard. An affective model of interplay between emotions and learn-
ing: Reengineering educational pedagogy-building a learning companion. In Advanced Learning
Technologies, IEEE International Conference on, pages 0043–0046. IEEE Computer Society, 2001.

[96] S. P. Lajoie and A. Lesgold. Apprenticeship training in the workplace: Computer-coached practice
environment as a new form of apprenticeship. Machine-Mediated Learning, 3(1):7–28, 1989.

[97] R. Lawson. Frustration; the development of a scientific concept. Critical issues in psychology series.
Macmillan, 1965.

[98] J. Lazar, A. Jones, M. Hackley, and B. Shneiderman. Severity and impact of computer user
frustration: A comparison of student and workplace users. Interact. Comput., 18(2):187–207, mar
2006.

[99] R. S. Lazarus. Emotion and adaptation. Oxford University Press New York, 1991.

[100] C. M. Lee and S. S. Narayanan. Toward detecting emotions in spoken dialogs. Speech and Audio
Processing, IEEE Transactions on, 13(2):293–303, 2005.

[101] B. Lehman, M. Matthews, S. DMello, and N. Person. What are you feeling? investigating student
affective states during expert human tutoring sessions. In Intelligent Tutoring Systems, pages 50–59.
Springer, 2008.

[102] C. Leung, E. Wai, and Q. Li. An experimental study of a personalized learning environment through
open-source software tools. IEEE Transactions on Education, Vol 50(No. 4), Nov 2007.

[103] W. Li and H. Xu. Text-based emotion classification using emotion cause extraction. Expert Systems
with Applications, (0):–, 2013.

[104] D. J. Litman and K. Forbes-Riley. Predicting student emotions in computer-human tutoring dia-
logues. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics,
pages 351–358, 2004.

160
[105] C. Liu, K. Conn, N. Sarkar, and W. Stone. Physiology-based affect recognition for computer-assisted
intervention of children with autism spectrum disorder. International journal of human-computer
studies, 66(9):662–677, 2008.

[106] E. A. Locke and G. P. Latham. What should we do about motivation theory? six recommendations
for the twenty-first century. Academy of Management Review, 29(3):388–403, 2004.

[107] M. K. Mandal, M. P. Bryden, and M. B. Bulman-Fleming. Similarities and variations in facial

expressions of emotions: Cross-cultural evidence. International Journal of Psychology, 31(1):49–
58, 1996.

[108] H. Mark, F. Eibe, H. Geoffrey, P. Bernhard, R. Peter, and I. H. Witten. The weka data mining
software: An update. SIGKDD Explorations, 11, 2009.

[109] S. Marsella, J. Gratch, and P. Petta. Computational models of emotion. A Blueprint for Affective
Computing-A Sourcebook and Manual, pages 21–46, 2010.

[110] B. T. McDaniel, S. K. D’Mello, B. G. King, P. Chipman, K. Tapp, and A. C. Graesser. Facial

features for affective state detection in learning environments. In Proceedings of the 29th Annual
Cognitive Science Society, pages 467–472, 2007.

[111] S. W. Mcquiggan, S. Lee, and J. C. Lester. Early prediction of student frustration. In Proceedings
of the 2nd international conference on Affective Computing and Intelligent Interaction, ACII ’07,
pages 698–709, 2007.

[112] E. Melis and J. Siekmann. Activemath: An intelligent tutoring system for mathematics. In Artificial
Intelligence and Soft Computing-ICAISC 2004, pages 91–101. Springer, 2004.

[113] D. Michie, D. Spiegelhalter, C. Taylor, and J. Campbell. Machine learning, neural and statistical
classification. Ellis Horwood London, 1994.

[114] W. L. Mikulas and S. J. Vodanovich. The essence of boredom. Psychological aspects, 43(1):3, 1993.

[115] C. T. Morgan, R. A. King, J. R. Weisz, and J. Schopler. Introduction to Psychology. McGraw-Hill

Book Company, seventh edition edition, 1986.

[116] S. Mota and R. W. Picard. Automated posture analysis for detecting learner’s interest level. In
Computer Vision and Pattern Recognition Workshop, 2003. CVPRW’03. Conference on, volume 5,
pages 49–49. IEEE, 2003.

[117] T. Murray. Authoring intelligent tutoring systems: An analysis of the state of the art. International
Journal of Artificial Intelligence in Education, 10:98–129, 1999.

161
[118] W. R. Nugent and H. Halvorson. Testing the effects of active listening. Research on Social Work
Practice, 5(2):152–175, 1995.

[119] J. F. O’Hanlon. Boredom: Practical consequences and a theory. Acta psychologica, 49(1):53–82,
1981.

[120] A. Ortony, G. L. Clore, and A. Collins. The cognitive structure of emotions. Cambridge university
press, 1990.

[121] A. Panning, I. Siegert, A. Al-Hamadi, A. Wendemuth, D. Rosner, J. Frommer, G. Krell, and

B. Michaelis. Multimodal affect recognition in spontaneous hci environment. In Signal Processing,
Communication and Computing (ICSPCC), 2012 IEEE International Conference on, pages 430–
435, 2012.

[122] M. Pantic and I. Patras. Dynamics of facial expression: recognition of facial actions and their
temporal segments from face profile image sequences. Systems, Man, and Cybernetics, Part B:
Cybernetics, IEEE Transactions on, 36(2):433–449, 2006.

[123] T. Partala and V. Surakka. The effects of affective interventions in human–computer interaction.
Interacting with computers, 16(2):295–309, 2004.

[124] R. Pekrun. The impact of emotions on learning and achievement: Towards a theory of cogni-
tive/motivational mediators. Applied Psychology, 41(4):359–376, 1992.

[125] R. W. Picard. Affective computing. Technical report, Cambridge, MA, 1997.

[126] R. W. Picard. Affective computing. MIT press, 2000.

[127] C. C. Pinder. Work motivation in organizational behavior . Psychology Press, 2008.

[128] E. Popescu. Evaluating the impact of adaptation to learning styles in a web-based educational
system. In ICWL ’009: Proceedings of the 8th International Conference on Advances in Web
Based Learning, pages 343–352, 2009.

[129] H. Prendinger and M. Ishizuka. The empathic companion: A character-based interface that ad-
dresses users’ affective states. Applied Artificial Intelligence, 19(3-4):267–285, 2005.

[130] S. Ritter, J. Anderson, M. Cytrynowicz, and O. Medvedeva. Authoring content in the pat algebra
tutor. Journal of Interactive Media in Education, 98(9), 1998.

[131] S. Ritter, J. R. Anderson, K. R. Koedinger, and A. Corbett. Cognitive tutor: Applied research in
mathematics education. Psychonomic bulletin & review, 14(2):249–255, 2007.

[132] M. M. T. Rodrigo and R. S. J. d. Baker. Coarse-grained detection of student frustration in an

162
 introductory programming course. In Proceedings of the fifth international workshop on Computing
education research workshop, ICER ’09, pages 75–80, 2009.

[133] M. M. T. Rodrigo, R. S. J. d. Baker, J. Agapito, J. Nabos, M. C. Repalam, S. S. Reyes, and M. O.

C. Z. S. Pedro. The effects of an interactive software agent on student affective dynamics while
using an intelligent tutoring system. IEEE Transactions on Affective Computing, 3(2):224 –236,
2012.

[134] M. M. T. Rodrigo, R. S. J. d. Baker, M. C. V. Lagud, S. A. L. Lim, A. F. Macapanpan, S. A. M. S.

Pascua, J. Q. Santillano, L. R. S. Sevilla, J. O. Sugay, S. Tep, and N. J. B. Viehland. Affect and
usage choices in simulation problem-solving environments. In Proceeding of the 2007 conference
on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work,
pages 145–152, 2007.

[135] I. J. Roseman. Cognitive determinants of emotion: A structural theory. Review of Personality &
Social Psychology, 1984.

[136] I. J. Roseman, M. S. Spindel, and P. E. Jose. Appraisals of emotion-eliciting events: Testing a

theory of discrete emotions. Journal of Personality and Social Psychology, 59(5):899–915, 1990.

[137] S. Rosenzweig. A general outline of frustration. Journal of Personality, 7(2):151–160, 1938.

[138] S. Rosenzweig and G. Mason. An experimental study of memory in relation to the theory of
repression. British Journal of Psychology. General Section, 24(3):247–265, 1934.

[139] J. Sabourin, B. Mott, and J. C. Lester. Modeling learner affect with theoretically grounded dynamic
bayesian networks. In International Conference on Affective Computing and Intelligent Interaction,
pages 286–295, 2011.

[140] G. Salton. Automatic Text Processing; the Transformation, Analysis, and Retrieval of Information
by Computer. Addison-Wesley, 1989.

[141] J. Sanghvi, G. Castellano, I. Leite, A. Pereira, P. W. McOwan, and A. Paiva. Automatic analysis of
affective postures and body motion to detect engagement with a game companion. In Human-Robot
Interaction (HRI) 2011, 6th ACM/IEEE International Conference on, pages 305–311, 2011.

[142] S. Schachter and J. Singer. Cognitive, social, and physiological determinants of emotional state.
Psychological review, 69(5):379, 1962.

[143] R. C. Schank. Dynamic memory: A theory of reminding and learning in computers and people.
Cambridge University Press, 1982.

163
[144] E. H. Shortliffe. Computer-based medical consultations, MYCIN. Elsevier Science Publishers, 1976.

[145] V. J. Shute and J. Psotka. Intelligent tutoring systems: Past, present, and future. Technical report,
DTIC Document, 1994.

[146] D. Sleeman and J. S. Brown. Intelligent tutoring systems. Computer and people series. London
Academic Press, 1982.

[147] C. A. Smith and P. C. Ellsworth. Patterns of cognitive appraisal in emotion. Journal of personality
and social psychology, 48(4):813–838, 1985.

[148] R. P. Smith. Boredom: A review. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 23(3):329–340, 1981.

[149] P. E. Spector. Organizational frustration: A model and review of the literature. Personnel Psy-
chology, 31(4):815–829, 1978.

[150] S. Srinivas, M. Bagadia, and A. Gupta. Mining information from tutor data to improve pedagogical
content knowledge. In Educational Data Mining, 2010.

[151] M. Stern, J. Beck, and B. P. Woolf. Adaptation of problem presentation and feedback in an
intelligent mathematics tutor. In Intelligent tutoring systems, pages 605–613. Springer, 1996.

[152] Y. Tong, W. Liao, and Q. Ji. Inferring facial action units with causal relations. In IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, pages 1623 – 1630, 2006.

[153] M. F. Valstar and M. Pantic. Fully automatic recognition of the temporal phases of facial actions.
Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 42(1):28–43, 2012.

[154] K. VanLehn, P. W. Jordan, C. P. Rosé, D. Bhembe, M. Böttner, A. Gaydos, M. Makatchev,

U. Pappuswamy, M. Ringenberg, A. Roque, et al. The architecture of why2-atlas: A coach for
qualitative physics essay writing. In Intelligent tutoring systems, pages 158–167. Springer, 2002.

[155] E. Verdú, L. M. Regueras, M. J. Verdú, J. P. De Castro, and M. Á. Pérez. An analysis of the research
on adaptive learning: the next generation of e-learning. WSEAS Transactions on Information
Science and Applications, 5(6):859–868, 2008.

[156] A. d. Vicente and H. Pain. Motivation diagnosis in intelligent tutoring systems. In Proceedings of
the 4th International Conference on Intelligent Tutoring Systems, ITS ’98, pages 86–95, 1998.

[157] E. Vockell. Educational psychology: A practical approach. Purdue University, 2004.

[158] J. J. Vogel-Walcutt, L. Fiorella, T. Carper, and S. Schatz. The definition, assessment, and mitiga-
tion of state boredom within educational settings: A comprehensive review. Educational Psychology

164
 Review, 24(1):89–111, 2012.

[159] J. Wagner, J. Kim, and E. André. From physiological signals to emotions: Implementing and
comparing selected methods for feature extraction and classification. In Multimedia and Expo,
2005. ICME 2005. IEEE International Conference on, pages 940–943. IEEE, 2005.

[160] B. Weiner. An attributional theory of achievement motivation and emotion. Psychological review,
92(4):548–573, 1985.

[161] E. Wenger. Artificial intelligence and tutoring systems. International Journal of Artificial Intelli-
gence in Education, 14:39–65, 2004.

[162] A. Wigfield and J. S. Eccles. Expectancy–value theory of achievement motivation. Contemporary

educational psychology, 25(1):68–81, 2000.

[163] A. Wigfield and K. R. Wentzel. Introduction to motivation at school: Interventions that work.
Educational Psychologist, 42(4):191–196, 2007.

[164] B. Woolf, W. Burleson, I. Arroyo, T. Dragon, D. Cooper, and R. W. Picard. Affect-Aware Tutors:
Recognising and Responding to Student Affect. Int. J. Learn. Technol., 4(3/4):129–164, 2009.

[165] Y. Yang. An evaluation of statistical approaches to text categorization. Information retrieval,

1(1):69–90, 1999.

[166] Z. Ying, Q. Bingzhe, and J. Chunzhao. On intelligent tutoring system. CNKI Journal on Computer
Science, 04, 1995.

[167] Z. Zeng, M. Pantic, G. Roisman, and T. Huang. A survey of affect recognition methods: Au-
dio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 31(1):39–58, 2009.

165
166
 Acknowledgement

First and foremost, I wish to express my utmost gratitude and heartfelt appreciation to
my dedicated research advisors Prof. Sridhar Iyer, Prof. Sahana Murthy, Prof.
Campbell Wilson and Prof. Judy Sheard for their guidance, continuous support and
discussions throughout the course of my research work. Without my supervisors advice
this thesis would not have been possible.
I thank Prof. Sridhar Iyer and Prof. Sahana Murthy for their encouragement,
motivation and freedom they gave me to choose my research area and their guidance to
achieve my goal. I thank Prof. Judy Sheard and Prof. Campbell Wilson for their
support, guidance during my Monash stay and for their constant guidance for improving
my thesis work.
I sincerely thank Prof. Krithi Ramamritham for his guidance, support during
my initial days of research study and continuous guidance as my Research Progress Com-
mittee member. His comments helped me to explore the possibilities of a wider research
domain and at the same time details to the finest granularity. I also take this opportunity
to thank my other Research Progress Committee member Ms. Janet Fraser for her
valuable comments that helped to improve my research work.
I also thank Mr. Sridhar Rajagoplan, CEO of Educational Initiatives, India,
for his support and permission to use the Mindspark data for my research work. Also
I would like to thank staff of Mindspark especially Mr. Muntaquim for his help in
implementing my research work in Mindspark. I thank Ms. Rajini Mitra, teacher
and school co-coordinator of Mindspark, for her support and extra effort she gave for my
experiments. I am grateful to all the students who accepted to participate in my study.
Their smiles made my experiments enjoyable.
I take the privilege to acknowledge my fellow researchers and facial expression ob-
servers Aditi Kothhiyal, Anura Kenkre, Gargi Banerjee, Jagadish, Kirankumar
Eranki, Neelamadhav and Yogendra Pal for their help in facial observation coding
and their support during my work. I thank my friend Aditya Iyer for helping us to learn
and train ourselves for facial observation. I am grateful for his timely help and the time
we spent together.

167
 I am thankful to Prof. Mohan Krishnamurthy, CEO, IITB-Monash Research
Academy, for his support, advice and guidance to handle tough situations during my
research work. I thank all the members of Department of Computer Science, IIT Bombay
and Caulfield school of IT, Monash University, who provided a friendly environment for
research. My special thanks to the staff at IITB-Monash Research Academy, IIT Bombay,
and Monash University. The most important names to be mentioned are Mrs. Kuheli
Mukerjee, Mr. Rahul Krishna, Mr. Rahul Ingle, Mr. Bharat Ingle, Mr.
Adrian Gertler, Mr. Vijay Ambre, Mrs. Alpana Athavankar, Mrs. Sunanda
Ghadge, Mrs. Subhadra, Mr. Thushar, Mrs. Gaikwad, and Mrs. Gayathri. I
thank Ms Sheba Sanjay, Communications and Training Services Consultant at IITB-
Monash Research Academy, for doing the language review for my thesis.
My heart filled thanks to my lab-mates Ajay Nagesh, Pratik Jawanpuria in
IIT Bombay and Dwi A. P. Rahayu, Varsha Mamidi in Monash University for their
help and support during my work. I specially thank my friend Naveen Nair for his
guidance, help and advice during my research. I am thankful to my friends Anu Thomas
and Chitti Babu for their help and support, particularly during my stay at Monash
University. I thank my hostel friends Balamurali, Siddharth Gadkari, and Bharat
Padeker for their help and support during my stay in IIT Bombay. I duly thank my
friends: Ashutosh Namdeo, Asifkhan Shanavas, Kapil Kadam, Kunal Kumar,
Ramesh K. V., Rupesh Bobade, Thaneshan Sapanathan, Wasim Feroze and all
my IIT Bombay campus friends.
I dedicate this thesis to my parents Mr. Rajendran and Mrs. Vatchala Devi
who encouraged me to continue my education and for the support they provided during
tough phases of my life. I thank my brother Mr. Santhosh Kumar and his wife Mrs.
Madhumathi for their motivating words to take up big challenges in my life. I thank
my grandma, grandpa, uncle and aunt for their care and affection during my early stages
of life. I am grateful to all my family members especially Krishanth, Vikranth and
Lesanth for their time and support.
I take privilege to thank people who helped and supported me directly or indirectly
throughout my academic career. Last but not least, I thank the Almighty for giving me
the strength and capability for this long journey in the pursuit of knowledge.

Date:
IIT Bombay, India Ramkumar Rajendran

168