IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO.

2, APRIL-JUNE 2015 201

Constructing Interpretative Views of Learners’

Interaction Behavior in an Open Learner Model
Kyparisia A. Papanikolaou

Abstract—In this paper, we discuss how externalizing learners’ interaction behavior may support learners’ explorations in an adaptive
educational hypermedia environment that provides activity-oriented content. In particular, we propose a model for producing
interpretative views of learners’ interaction behavior and we further apply this model to INSPIREus for visualizing specific indicators.
In the proposed approach, we collect raw data from learners’ interaction at various grains, model the state of interaction using a set of
indicators that combine temporal, navigational and performance data with semantic data of content and available tools, and visualize
this information alongside with comparative information coming from the instructor or peers. In this way, we provide users (learners,
tutors, peers) with a mirror of learners’ behavior and a point of reference such as the instructor’s proposal or peers’ behavior, in order to
enable monitoring and reflection. An empirical study with students is also described investigating how they interpret the visualizations
of their interaction data provided by INSPIREus, the metacognitive skills they cultivate, and the personal data of peers they value for
selecting collaborators. Preliminary results provide evidence about the understandability and expressiveness of the indicators of effort,
progress, working style, and the visualizations used.

Index Terms—Adaptive hypermedia, interaction analysis, open learner modeling, visualization

Ç
1 INTRODUCTION

T RACKING and analyzing learners’ experience during

their interaction within a learning environment is a
challenging target for the area of technology enhanced
critical issues to this end such as contents and user control
over the learner model, as well as in which ways and by
whom such a model could be used in order to promote indi-
learning. Various types of learning environments keep track vidual and social learning [10], [11], [12], [35], [47], [56].
of learners’ interaction behavior [1] such as adaptive learn- Thus, a challenging target for both the areas of Open
ing environments [2], collaborative learning environments Learner Modeling and Interaction Analysis is the produc-
[3], [4], course management systems [5], [6], [7], intelligent tion of meaningful externalizations of learners’ interaction
tutoring systems [8], [17]. Interaction data are usually used behavior allowing different stakeholders such as learners,
for various purposes in order for the environment to main- peers, instructors to ‘explore’ the learning process from
tain the current state of learners and accordingly adapt its their own perspective [14], [15], [56].
output or open it to the various stakeholders of a virtual The task of collecting data in a web-based adaptive
class as evidence for learners’ work and performance [1], learning environment that correctly and accurately repre-
[9], [18], [48]. In the last cases, specific aspects or results sents learners’ interaction with the system and storing
of learners’ interaction behavior are usually externalized them in a well-defined and expandable database format is
focusing on learners’ knowledge state/level, navigation or not as straightforward as it might seem at first, for various
aspects of social interaction. Learners’ observable behavior technical and non-technical reasons [1], [9]. Moreover,
is rarely exploited to reflect learners’ cognitive activity and interpreting and visualizing learners’ interaction raw data
cultivate metacognitive skills such as reflection, planning, in meaningful and useful ways is another area of research,
and self-monitoring. An open issue remains how interaction with great potential regarding the impact these methods
data could be used as evidence for the learning process. can have on understanding learning processes and behav-
How could we extend the learner model with representa- ior patterns [20], [6], [4], [33]. Especially in a hypermedia
tions of personal interaction data that reflect the processes learning environment, a main issue is that learners make
that the learner follows in order to attain specific outcomes explicit decisions repeatedly during interaction that usually
or cultivate skills? This could be based on a theoretical results in complex interaction protocols. These protocols
model that augments interaction data with learning design refer to the series of events, which occur during hyperme-
characteristics of content and available tools giving meaning dia usage, with corresponding time stamps [16]. However,
to learners’ actions and linking them with specific out- collecting learners’ actions is the first step for re-construct-
comes. The Open Learner Modeling area investigates ing a view of learners’ activity able to promote learners’
reflection on their explorations. Then, heterogeneous data
 The author is with the Department of Education, School of Pedagogical and included in interaction protocols must be carefully chosen
Technological Education, Heraklion 14121, Greece. and handled in order to yield meaningful information and
E-mail: [email protected]. build a thorough view of learners’ activity. To this end,
Manuscript received 11 June 2013; revised 2 Oct. 2014; accepted 3 Oct. 2014. contextual data about the content, available tools and
Date of publication 23 Oct. 2014; date of current version 16 June 2015. learning design issues of the learning environment is nec-
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference the Digital Object Identifier below. essary. Finally, this information may be used or analyzed
Digital Object Identifier no. 10.1109/TLT.2014.2363663 with different purposes depending on the context and the
1939-1382 ß 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
202 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

end user. For learners, it may be useful to compare their instructors in monitoring the learning process and gaining
state with an ‘ideal’ model proposed by the instructor or an understanding of their learners [5], [6], [7], [13], [19], [20].
an ‘actual’ model of interaction coming from individual For example, CourseVis [6] use web log data from online
peers or their group in order to detect possible mismatches course management systems to visualize interaction in dis-
that can promote reflection on their learning process. For cussions, quiz performance and page access. By using such
tutors, it may be useful to have access to learners’ interac- tools, teachers could get a view of students’ work more
tion behavior in order to get insight into the learner’s state quickly. Moodog [19] is a Moodle plugin that extends the
and be able to intervene accordingly. For an adaptive sys- Moodle log facility by visualizing data from the activity logs
tem the whole procedure may result in a set of recommen- to allow students to compare their progress to others and
dations or corrective actions which could support learners teachers to get insight into the student interactions with the
in reaching the desired state. online course. It provides aggregated and meaningful statis-
It is important that interpretative views of learners’ inter- tical reports, showing the number of unique users, view
action produced by system designers and their expressive counts and popularity of resources. SAM (Student Activity
power of learners’ cognitive processes be evaluated by the Meter) [13] is a tool that captures the interactions of users
learners themselves [49], [50]. Learners’ personal views on with resources and tools and provides simple metrics, statis-
their interaction patterns or those of their peers will prove tics and visualizations of learners’ activities, using similar to
what these patterns actually reveal. Opening the learner Moodog metrics but not committed to one learning platform.
model is the first step to this end. Then, learners should be GISMO—Graphical Interactive Student Monitoring Sys-
prompt to offer their own interpretations, additional or cor- tem—[20] provides graphical representations of data col-
rective information, leading to a more precise adaptive lected from real courses reflecting student attendance, access
interaction [11], [46]. In this context, it is worthwhile to fur- to resources, overview of discussions, and results on assign-
ther investigate (a) how learners’ interaction behavior could ments and quizzes, i.e. information interesting to instructors.
be tracked, interpreted and visualized to provide a compre- In the area of computer supported collaborative learning
hensive view of their cognitive activity, meaningful for dif- (CSCL), several studies have explored the issue of analyzing
ferent types of users or uses depending on the type of learners’ interaction in a social context to assess group/
learning environment or the task, and then (b) how users individual performance and collaboration dynamics more
interpret visualizations of their learning experience. objectively [21], [3], [4], [23]. Several systems that external-
In this paper we propose an approach to constructing ize these data to learners in multiple ways in order to pro-
interpretative views of learners’ interaction behavior in a mote learning and enhance participation and collaborative
hypermedia learning environment. The way learners use skills such as Participation Tool [22], DIAS [24], team.sPace
and mainly interpret these visualizations is investigated. [15], [25], COMTELLA [26], CoolModes [27], i-bee [28], ACT
The paper is structured as follows. The literature review [29] have also been reported. For example, the Participation
section presents the scope of using interaction data from Tool visualizes students’ contribution to their group’s
three different types of learning environments. In the next online communication aiming to promote group awareness.
section, an approach to modeling and visualizing interac- Each student is represented by a sphere and group mem-
tion data is proposed. In Section 4, the application of the bers’ spheres are grouped together. i-Bee uses the analogy
approach in the adaptive educational hypermedia system of flowers and bees to visualize relationships between users
INSPIREus is analyzed. Section 5 presents an empirical and keywords in online discussions. It represents several
study investigating how students interpret views of their discussion indicators, such as keyword usage frequency
interaction data. The paper ends with discussion of the and recent user activity, through the distance between flow-
study results and future plans. ers and bees, their status such as flying or sleeping bee,
blossomed or closed flower, and their orientation. Although
the context and the study objectives of collaborative learn-
2 LITERATURE REVIEW ing systems are quite different compared to intelligent and
Different types of learning environments that keep track of adaptive learning environments, research studies in this
interaction data, such as course management systems, com- area could be used as valuable resources for modeling and
puter supported collaborative learning systems, intelligent visualizing learners’ interaction. Useful approaches that
and adaptive learning environments, usually process them have been proposed on how interaction behavior might be
in order to diagnose learners’ characteristics and accord- used to support interaction and collaboration, are the fol-
ingly adapt the interaction, or provide evidence of their lowing [30]: (a) reflecting learners’ actions by collecting raw
individual/social characteristics. In some cases, they inter- data and displaying it to the collaborators, (b) monitoring
pret interaction data providing visualizations of actual the state of interaction by aggregating the interaction data
learners’ interaction behavior aiming to prompt them to into a set of high-level indicators or by comparing the cur-
reflect on their actions or tutors to monitor this behavior. In rent state of interaction to a model of ideal interaction
this section, we discuss representative paradigms of sys- enabling learners to self-diagnose the interaction, (c) analyz-
tems and tools that exploit interaction data in multiple ways ing the state of collaboration using a model of interaction,
focusing on learners’ interaction with the content or on and offer advice.
social interaction. Learners’ interaction data are usually processed by adap-
In the area of course management systems, visualisation tive learning environments in order to built the knowledge
techniques have been adopted to analyze learners’ interac- state or navigation paths of learners and accordingly pro-
tion behaviour aiming to support learners and/or vide guidance in the form of graphical annotations, or
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 203

communicate it to learners through appropriate visualiza- communication, for involving users in learning about them-
tions included in an open learner or group or social model selves and other users through reflection, validation, assess-
such as ELM-ART [31], KnowledgeTree [32], ViSMod [33], ment and diagnosis. In our case, the current purpose of the
Flexi-OLM [34], MyProject [54], JavaGuide [55], Progressor interaction modeling approach is to support learners’ reflec-
[35]. For example, ELM-ART and KnowledgeTree maintain tion and help them learn about themselves and other peers.
data about learners’ interaction with the system and exter- It is then up to the learner to further elaborate on the visuali-
nalize information, in a text-based form, about the learning zation and decide what action (if any) to take.
resources and various tools that they worked with, such as Our proposal for designing interpretative views of
time of access, learning activity, activity ID, visited pages, learners’ interaction behavior in an AEHS combines and
learning time, results. ViSMod is an interactive visualization expands ideas coming from the areas of hypermedia learn-
tool for the representation of Bayesian learner models. ViS- ing [16], open learner modeling [11], [10], [47], and interac-
Mod uses various visualization techniques such as color, tion analysis [3], [30], [37]. The learning analytics process as
size proximity link thickness, and animation, in order to described in [37] is represented as an iterative cycle carried
represent the overall belief of a student knowing a particu- out in three steps: data collection and pre-processing, ana-
lar concept taking into account the student’s opinion, the lytics and action, and post-processing. Post-processing
instructor’s opinion, and the influence of social aspects of relates to the continuous improvement of the analytics,
learning on each concept. Flexi-OLM also provides multiple which is out of the scope of this paper. Following the first
representations of learners’ knowledge based on Felder and two steps, we initially collect appropriate data from
Silverman’s style categorization. Learners’ understanding learners’ interaction with the content and system tools/
of each topic and misconceptions on specific topics are illus- modules at various levels of granularity (Phase 1: Data col-
trated through appropriate coloring of nodes. An interest- lection and Data pre-processing). Then, we transform data
ing direction on the crossroads of adaptive learning into suitable format producing several indicators of
environments and social learning is the adaptive social nav- learners’ cognitive and social activity (Phase 2: Analytics
igation support as implemented by Progressor. Progressor and Action).
allows learners to share their navigation and progress data The process of modeling learners’ interaction has the
through a tabular interface that accommodates a sizable col- double aim of providing a mirror of the learner’s actions
lection of content augmented with appropriate graphical augmented in some cases with the desired state of interac-
annotations. In the line of sharing personal data among tion. Another possibility, not examined in this paper, is the
peers, the UMPTEEN approach [56] investigates students’ system-generated recommendations whenever a perturba-
views about opening their learner model more widely as tion arises for example among the actual and desired state
well as how they use their own and peer models. of interaction or in cases of failure. In more detail, the pro-
In the above studies, aspects of learners’ interaction cess of modeling learners’ interaction is deployed at the fol-
behavior with the content are externalized in a text-based or lowing phases:
graphical form illustrating their knowledge state and/or Phase 1—Data collection and pre-processing. The data collec-
navigation at individual or social level allowing learners to tion phase involves observing and recording the interaction.
reflect on their own data and on specific occasions compare Important decisions when analyzing data coming from
them with those of their peers. In this paper, we build on interaction with a hypermedia environment are the selec-
the above studies in order to further exploit learners’ tion of appropriate type of data, as well as the definition of
observable behavior and construct interpretative views of the appropriate observation grain, i.e. the precision of the
their cognitive activity as it unfolds. To this end, a qualita- events considered as units in the analysis [16].
tive description of the interaction that could provide mean- Regarding the type of data, it concerns (a) data collected
ing to specific learner’s actions with the educational content from learners’ interaction with the content and system tools
and system functions is proposed. at various grains, as well as (b) variables reflecting the learn-
ing design of the environment, putting learners’ interaction
in context. Following [1], specific data that are available for
3 PROPOSING A MODELLING APPROACH FOR
mining in the educational area have intrinsic semantic infor-
LEARNERS’ INTERACTION BEHAVIOUR mation, relationships with other data, and multiple levels of
Adaptive Educational Hypermedia Systems (AEHSs) usu- meaningful hierarchy such as the domain model. Further-
ally track learners’ interaction and content usage in order to more, it is also necessary to take into account the pedagogi-
dynamically adapt the content presentation, topic sequenc- cal aspects of the learner and the system, making
ing, navigation or collaboration support. Interaction data interaction data valuable for extending the learner model.
may also be carefully handled in order to yield meaningful In particular, data coming from learners’ interaction are
views of learners’ strategies. In this process, key issues are navigational and temporal data such as visits on various
the selection of the appropriate data, their interpretation, types of resources, sequence of resources, time spent on
and the way this info is conveyed to the learner. The task of resources, actions performed with available tools, as well as
interpretation demands the construction of a model of the performance data such as type of assessment, attempts on
interaction, which is instantiated to represent the current various types of self-assessment questions, performance,
state of interaction, and possibly the desired state [3], [30]. and progress. However, concerning temporal data, long
In [36], several purposes for user modeling are considered time intervals or too short ones should be eliminated in an
in I-Help, such as user modeling for locating appropriate attempt to deal with the problem of time when working
resources, for facilitating interpersonal and inter-agent with digital learning environments where it is difficult to
204 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

identify what the learners actually do. Variables related to purpose indicators. The cognitive purpose indicators reflect cog-
the learning design of the environment such as pedagogical nitive engagement that concerns the actions of the individu-
characteristics of the content and the available tools are con- als in the learning environment referring to the process or
sidered as context variables that affect learners’ behavior in the product of their activities [3]. In particular, these indica-
a particular environment with specific affordances. tors reflect the mode or quality of learners’ individual activ-
The observation grain relates to the events that are selected ity with content of a particular pedagogical profile related
to be analyzed, ranging from global activity patterns (coarse to specific outcomes, the type of messages that they
grain) to specific aspects of the interaction (intermediate or exchange with peers, as well as their individual activity
fine grain) [16]. This relates to the principle of ‘overview, with various types of tools, supportive aids (notes, interac-
zoom and filter, details on demand’ proposed by Shneider- tion data), and learner control opportunities provided by
man [52] since the various grains can be considered as over- the system (open learner model, adaptive controls).
view and zoom levels. Specifically, at the coarse grain, the Learners’ interaction with the particular indicators is
activity is considered as a whole, providing the overview expected to cultivate metacognitive skills such as self- and
that reflects global features of learners’ work. Large sequen- task- knowledge, strategic knowledge and knowledge of
ces of actions are taken as data units, whilst events that may plans and goals [39], [40]. To this end, they should be
have occurred within each sequence are not considered. For designed to assist learners in recognising their preferences
instance, the total time taken to perform a task may be con- / strengths / weaknesses and evaluating themselves (self-
sidered as an indicator of task difficulty. The sequence of knowledge), understanding the demands of tasks and what
units visited during a session of hypermedia navigation can they require (task-knowledge), setting and maintaining
also be divided into coarse segments, i.e. areas of applica- goals and recording what they intend to do through their
tion like a set of particular pages as opposed to each indi- learning (knowledge of plans & goals), but also assessing
vidual page. At the fine grain level, each event recorded the usefulness of their strategies for achieving specific out-
during the interaction will be considered, whilst at the inter- comes (strategic knowledge). The social purpose indicators
mediate level only some significant events are considered, reflect various aspects of social behavior in the learning
both constituting different zoom levels [52]. At the interme- environment that refer to communication, cooperation or
diate level, critical events of interest may be selected within collaboration of individuals as members of groups or com-
the interaction protocol making appropriate computations munities [3]. They may reflect the level of forum participa-
such as total time spent on specific resources or frequency tion, the use of sharing opportunities of their learner model
of visits of particular tools. This level is useful when testing and/or notes and/or forum discussions.
specific hypotheses about the cognitive processes at work Regarding the contents of indicators, these may be simply
during interaction. For instance, moves from one particular one type of data or combinations of various types of data
node to another can be considered, regardless of what hap- such as navigational and temporal data along with learning
pens in each node. An important feature of the intermediate design variables. A critical issue here is the definition of the
grain is that it allows the researcher to summarize informa- appropriate observation grain which can be linked with dif-
tion as a set of numerical parameters like the number of ferent purposes and metacognitive activities. The scope of
times learners accessed the learner model or a node with the indicators may vary from an overview of learners’ work
particular characteristics. or a more detailed observation of their work on particular
Lastly, at the fine grain level, all the observable actions are tasks to a deeper view of the way learners use the resources
taken into account. At this level the complete sequence of for achieving specific outcomes. Visualisations of the above
events included in raw interaction protocols is analyzed. In data may focus on a learner’s interaction or on comparative
this case, the researcher focuses on meaningful patterns, in information coming from tutors or peers to support moni-
order to achieve a global understanding of learners’ activity. toring and reflection. For example, it may represent the nav-
Phase 2—Analytics and action. Analytics and action phase igational path of a learner or augment the navigational path
includes analysis and visualization of information as well as with temporal data reflecting time spent on various steps,
actions on that information focusing, in our case, on moni- or combine a learner’s temporal (time spent on resources),
toring and reflection [37]. The approach adopted at this navigational (number of visits) and performance data with
phase is mainly inspired by the collaboration management a ‘desired’ state of interaction coming from peers or the
cycle as described in [30]. At this phase we aggregate the tutor. Such a ‘desired’ state may be the pedagogical dura-
interaction data into a set of high-level indicators. These tion proposed by the tutor [45], [53] or the mean time spent
indicators combine one or more types of interaction data on specific resources by selected peers. Visualisations may
with learning design variables aiming to put learners’ be static allowing learners to inspect the indicators or adapt-
actions in context. Important design decisions at this phase able allowing learners to intervene and select options such
are the type of indicators (such as cognitive, social), their as the observation grain or features of the ‘desired’ state e.g.
contents (data and observation grain) and scope, the focus choice with whom to compare.
of the visualization, the mode of use (static, adaptable) and The system can further analyze the above data taking into
intended users. account the learners’ current state, their individual charac-
Interaction analysis indicators, in our case, describe teristics and the contextual variables influencing their
aspects of learners’ interaction related to the domain model actions, in order to advise the learner how to proceed or pro-
or the process or the quality of the learning activity (task pose appropriate peers. This analysis of learners’ actions is
related process) [3] reflecting the current state of interaction. based on their interaction with the resources and tools in the
The indicators proposed at this phase are cognitive and social current context at multiple observation grains. For example,
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 205

information about the type of educational resources that the can be reused by learners of different profiles. The notion of
learner selects, combined with learners’ progress and/or educational scenarios is used to underline a learner-centred
learning style, provides an indication of their preferences for content design approach. In particular, each scenario is
educational resources. Thus, in case of low performance the associated with a conceptual structure that includes all the
system may recommend another route in the content based necessary domain concepts and their relationships—out-
on the desired interaction model or on a ‘real’ interaction comes, prerequisites, related concepts- providing learners
model coming from peers with specific characteristics like with an overview of how all the relevant information fits
those having high performance and similar style. together. This structure is provided in a hypermedia form
The main idea behind such an interaction modeling to enable the learners to freely navigate and to use the con-
approach is to (i) enable mirroring of learners’ interaction tent in accomplishing the role(s) they undertake in the con-
and cultivate metacognition, (ii) support the provision of text of the scenario. In this new version, the educational
meaningful recommendations to learners, (iii) guide system material pages of each concept may consist of a variety of
adaptive behavior in case of an agreement between the content modules according to the learning design adopted.
learner and the system, (iv) provide tools for tutors to evalu- The design of scenarios is activity-oriented aiming to
ate students’ work and the content. encourage learners to use tools, generate and test hypothe-
sis in real contexts, solve open problems, explore alternative
4 THE CASE OF INSPIREUS perspectives, work individually or in groups. To this end,
different types of content modules such as examples, exer-
INSPIRE [41] is an adaptive educational hypermedia envi- cises, triggering or assessment questions, theoretical tips,
ronment that allows learners to freely explore the content activities, can be combined in educational material pages of
offering them individual advice. INSPIRE provides adap- various categories according to the learning design proto-
tive support based on learners’ individual characteristics type proposed by the system or created by the content
by: (a) structuring the content around specific outcomes author. Authors are encouraged to develop content of high
augmented with visual cues that inform learners about the interactivity incorporating simulations located on the Inter-
content that they are ready to study based on their knowl- net, video, communication tools, and Web 2.0 tools. For
edge level (adaptive navigation technique), (b) providing example, in Fig. 1 the educational material page ‘Decimal—
individualized versions of the educational material pages Binary Numbering Systems’ of the concept ‘Computer
with alternative sequencing of the modules involved based memory organisation’ (scenario ‘How is data represented
on instructional strategies that suit learners’ learning style inside the computer?’) that promotes conceptualisation and
preferences as proposed by Honey and Mumford (adaptive application of the particular numbering systems appears at
presentation technique). the Content Area. The particular page consists of several
INSPIREus is the latest version of INSPIRE, extended with modules, most appearing as links and only one being fully
collaborative functionalities and a flexible authoring process opened, i.e. links to an activity, an example, theoretical
that allows users to reflect their pedagogical perspective on issues, and an exercise, and the module fully appearing is a
content development. In INSPIREus users comprise an theoretical presentation about decimal binary, octal and
online community, having one or more roles with different hexadecimal number systems which provides the necessary
rights, such as learners, tutors, authors, reviewers. Learners information through a video and prompts students to watch
can enroll in specific virtual classes, attend online lessons at it and then share ideas in the forum.
their pace, personalize the interaction, view interaction data, Authors are allowed to select or create learning design
participate in groups, communicate and share their notes prototypes reflecting their own pedagogical perspective.
and/or discussions and/or model with their group or class. They can propose specific categories of educational material
When students communicate through a forum, they can pages that comprise appropriate modules and focus on spe-
characterize the messages they exchange as well as their cific learning outcomes according to the learning theory
peers’ posts as a way to better communicate their ideas. The they adopt. They can also propose an adaptation algorithm
characterizations are taken from a list based on the interac- linked with learners’ individual characteristics.
tion analysis model of [42], which examines the social con- Learning design prototypes already incorporated in the
struction of knowledge in computer conferencing. system have been inspired by the inquiry based learning
A main challenge for the new version of INSPIREus is and the ‘New Learning’ model. These prototypes propose
the visualization of interaction analysis in a meaningful the development of various types of educational material
way for learners and tutors. In this paper we elaborate on pages that promote specific outcomes or knowledge pro-
the new type of support provided by INSPIREus through cesses. For instance, the inquiry based learning model is
modeling the learners’ interaction and visualizing this organized in four phases [43]. Each phase or their combi-
information in a meaningful way. By opening this info to nations have inspired the design of specific categories of
learners we extend the opportunities of interaction with educational material pages. The theory of ‘New Learning’
the learner model aiming to cultivate metacognition [12], introduces eight ‘knowledge processes’ (i.e. forms of action
[38], [39], [40]. inspiring various types of activities), each one representing
a different way of making knowledge [44]: (i) Experiencing
4.1 Modeling the Content the known and the new, (ii) Conceptualizing by naming
INSPIREus provides learners with structured content, as in or with theory, (iii) Analyzing functionally or critically,
the previous version, which is comprised of units, such as (iv) Applying appropriately or creatively. Learning design
scenarios, concepts and educational material modules that prototypes that are based on ‘New Learning’ consist of
206 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

Fig. 1. The educational material page ‘Decimal—Binary Numbering Systems’ appears at the ‘Content area’ of INSPIREus consisting of multiple types
of modules: links to an Activity and Example, Theoretical issues that embed a video (this module appears open as it is currently being selected by the
user), link to theoretical issues and an exercise. Each open module has its own toolbar (‘Tools’) with the ‘Start/Stop’ button allowing learners to
declare ‘idle’ time.

specific categories of educational material pages that stim- very short and long intervals in learners’ work are elimi-
ulate a particular knowledge process or combinations of nated—and performance data such as type of assessment,
processes. attempts on assessment questions, performance on various
types of questions, are recorded and calculated at scenario,
concept and page level.
4.2 Producing Interpretative Views of Learners’ Aiming to set the data recorded from learners’ interac-
Interaction Behavior
tion behaviour (navigational, temporal, performance data)
INSPIREus gathers data from learners’ interaction and visu- in context, specific variables reflecting the learning design
alizes them augmented with contextual information, in of the environment are also considered. These variables are
order to support learners in gathering evidence and evaluat- used to augment the data visualisation. Currently as such
ing the efficacy of their moves. Key issues in this process are variables are considered:
the selection of the appropriate data (learners’ actions and
contextual information) and the production of interpretative  pedagogical characteristics of the content such as the
views along with a meaningful way of conveying them to pedagogical duration, type, learning outcomes in
the learner. To this end, the approach described in Section 3 terms of knowledge process/level of performance/
has been adopted. phase of inquiry according to the learning design
Data collection and pre-processing. A combination of fine, prototype adopted,
intermediate and coarse observation grain data has been  type of available tools providing learners with
considered in INSPIREus, to construct meaningful views of (a) controls over the learning and adaptation process
learners’ interaction with the system. These data range such as the open learner model, adaptation controls,
from global activity patterns at scenario and concept level (b) social opportunities such as forums, sharing
(coarse grain), and specific aspects of the interaction with notes and learner models, (c) reflection opportunities
resources of particular type (intermediate grain), to all the like note keeping tool, interaction analysis data.
observable actions where the analysis focuses on meaning- The two types of variables are considered valuable for
ful patterns at content page level (fine grain). In particular, tracking learners’ behaviour and building the individual
navigational data such as number of hits, frequency of visits and desired model of interaction. At a next phase, learners’
reflecting resources (content and tools) usage, temporal data characteristics such as learning style, knowledge level (as
such as time spent on various types of resources—cases of this is defined by the learning design prototype) and social
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 207

Fig. 2. Indicator of learners’ effort mirrors learners’ engagement: (a) illustrates learners’ effort at scenario level for a particular work period (‘Work
period’) at coarse grain providing temporal, navigational, and performance data (Part I) about the three concepts of the scenario “How is data repre-
sented inside the computer?”, (b) illustrates how the learner has distributed his/her time on the concept ‘Computer and memory organisation’ among
modules of various types.

interaction data can be used to assess the interaction effec- corresponds to a page category of the scenario aiming
tiveness and to guide system recommendations. to provide progress data according to the learning
Analytics and action. A set of cognitive and social pur- design prototype of the content. Another visualiza-
pose indicators has been created. In this paper, we focus tion that reflects the particular indicator at intermedi-
on indicators that reflect cognitive engagement by repre- ate grain is the pie chart (see Fig. 2b). It presents how
senting data that concern (a) learners’ interaction with learners’ time is distributed across modules of vari-
the content and system affordances such as opportunities ous types at two segments, concept or a single page.
for learner control and (b) individual contributions in The particular visualization is adaptable to the loca-
social knowledge construction at various observation tion from which the learner accesses it, depending on
grains. Currently, the indicators computed and visualized whether the learner is on the first page of a concept or
are: any other page of a concept. In the first case, total data
about the learner’s interaction with all the content of
a) indicator of learners’ effort that reflects the learners’ the concept are visualized.
engagement with the particular scenario. This is visu-
b) indicator of learners’ progress that reflects the mode or
alized through the ‘analytical presentation’ of Fig. 2a
quality of learners’ individual activity at intermedi-
that illustrates navigational and temporal interaction
ate grain with content of particular pedagogical
data at coarse grain (scenario level) along with perfor-
‘profile’ reflecting also the learning outcomes that
mance data based on the learning design prototype
students have to attain. To this end, navigational and
adopted for the particular scenario. This visualisation
temporal interaction data on resources of specific
provides an overview of learners’ activity with the
type like the various page categories (introductory
domain concepts of the scenario ‘How is data repre-
pages, pages covering specific types of outcomes
sented inside the computer?’. In particular, informa-
based on the learning design prototype, self-assess-
tion illustrated on the top of Fig. 2a (see ‘Work
ment pages, recall pages) and the various types of
period’) represents the total period of interaction
material (activities, examples, exercises, questions,
with the content of the scenario (work period is
theory) at scenario or concept level are illustrated
selected by the learner from a list where s/he can
along with performance data. For instance, informa-
select a particular period, or the total period working
tion illustrated on Fig. 3 reflects learners’ activity
with the scenario named ‘Whenever’). Then, in Part I
with various types of content at concept level. In par-
of Fig. 2a, learners’ activity with the content of
ticular, in Part I interaction data with each of the
the domain concepts (i.e. ‘Computer memory
pages of the concept ‘Computer Memory Organ-
organisation’, ‘Hexadecimal number system’,
isation’ are illustrated. Then in Part II, interaction
‘Representation of text’) is depicted. For each concept,
data are added up for each category of educational
we provide: (a) the time spent by the learner along
material pages (Recall, Assessment, Category A,
with the pedagogical duration of the resource (see
Category B, Category C) illustrating (a) the time
‘Time/Proposed’ column), (b) learners’ visits along
spent along with the pedagogical duration of all the
with total number of hits on the content (see ‘Visits/
resources of each particular category (see Fig. 3—
Total’ column), (c) level of performance on various
Part II ‘Time/Proposed’ column) and (b) visits along
types of questions (see ‘Performance per type of
with total number of learners’ visits on the content of
questions’ column) where each type of questions
208 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

Fig. 3: Indicator of learners’ progress mirrors learners’ work with particular types of content at intermediate grain for a particular work period (see
‘Work period’) depicting their activity with various categories of content pages (Part II) and particular types of modules (Part III) for the ‘Computer
Memory Organisation’ concept.

the particular concept (see Fig. 3—Part II ‘Visits/ resources, although Student A seems more concen-
Total’ column). Moreover, in Part III, temporal and trated on the question, whilst Student B seems to
navigational data are provided for each type of con- prepare himself before visiting and probably
tent modules included in the educational material answering the question. Lastly, in Fig. 4, by combin-
pages of the concept like activities, examples, exer- ing information from two representations (learner
cises, theory and questions (see Fig. 3, Part III ‘Time/ navigation and time distribution—pie chart) for a
Proposed’ and ‘Visits/Total’ columns). The particu- particular student such as Student B, we can also
lar visualisation is adaptable allowing learners to have a view of how a learner distributes her time to
select the working period, with whom to compare various types of resources.
(tutors or selected peers), as well as if the visualisa- d) indicator of learner control that reflects learners’ indi-
tion will reflect their interaction behaviour at sce- vidual activity with specific system tools that allow
nario or concept level based on their location when learners to personalise the interaction at three coarse
they visit interaction analysis data. So, if learners are segments (i.e. scenario, concept, page), through the
on the first page of the scenario, then the visualisa- particular actions (view/update) they performed
tion will reflect their interaction at scenario level pro- with their learner model, the note keeping tool and
viding data for each concept of the scenario, and the the various interaction data visualizations.
various page categories of all the concepts of the sce- e) indicator of learners’ contribution in social knowledge con-
nario. Accordingly, if they are on the first page of a struction reflecting the type of messages that the
concept, like in Fig. 3, the visualisation will reflect learner exchanges with peers at forum level. The par-
their interaction at concept level. ticular indicator focuses on individual contributions.
c) indicator of learners’ working style that reflects It can be adapted to a single forum of a module
learners’ strategies while working with content of selected by the learner, or to all the forums of a page,
particular pedagogical ‘profile’ related to specific concept or scenario.
outcomes at fine grain. The ‘learner navigation’ All the above indicators can be used by individual learn-
representation visualizes the learners’ navigation ers, groups, and tutors. Learners are allowed to share inter-
path on content pages and particularly the sequence action data with peers (at group and/or class level) so as to
of learners’ visits on various types of modules compare themselves to others (see on the top of Fig. 3,
(activities, examples, exercises, questions, theory) at Access to peers’ interaction data). Such information coming
a fine grain by recording every visit and its duration from peers can also give them new ideas and encourage
at page level. In Fig. 4 the navigation history of two deeper thought about the implications of their own strate-
students in a particular content page that contains a gies or support them find appropriate peers when seeking
question, an example and an exercise is depicted. for help. It may also support tutors in acquiring an image of
Both students spent time on all the available the learners’ activity, progress and needs as well as in
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 209

Fig. 4. Indicator of learners’ working style mirrors learners’ activity on the page “Data representation inside the computer” at fine grain providing data
about the sequencing of learners’ visits and their duration on particular type of modules illustrated through various colours i.e. red for theory, blue for
examples, pink for questions. At this illustration the navigation history of Student A and the navigation history and time distribution (pie chart) to vari-
ous types of resources of Student B appear.

evaluating the resources offered to learners with particular formed). The groups of students had to select between two
profiles. To this end learners’ self reports would be also scenarios, both of which were based on the same learning
valuable since many visits on particular content may reflect design. They had two weeks to perform the tasks asynchro-
a preference for that content but also difficulty to deal with nously, complete and submit the worksheet. Group work
the particular task (this is also supported by the results of was organized and scheduled mainly by the members of
the study presented in Section 5). each group whilst group collaboration was done through
the forum of INSPIREus or other communication tools
5 EMPIRICAL STUDY selected by the group like Skype or email. The first time
they logged on INSPIREus, they submitted the question-
The empirical study aimed at investigating how students
naire of Honey and Mumford in order for the system to
interpret specific visualizations of their interaction behavior
automatically identify their learning style. During the inter-
provided by INSPIREus. This study was conducted in the
action, they were allowed to change it manually through
context of a technology enhanced learning course, offered
the learner model. Finally, students were asked to answer
to graduates of a variety of disciplines that attend the one-
an evaluation questionnaire with closed questions (see
year postgraduate certificate in education of the School of
Appendix A), in order to reflect on and evaluate their learn-
Pedagogical and Technological Education (ASPETE). One
ing experience.
class of 50 students participated in the study. The study
focused on the following research questions:
5.2 Data Collection and Tools
 How do students interpret visualizations of their The data collected were the students’ worksheets, log data
interaction behavior? and questionnaires.
 Which evaluation criteria do learners use for select- Worksheet. The worksheet included guidelines recom-
ing collaborators based on their interaction data? mending specific content to study and indicators (called vis-
ualizations in the worksheet) to reflect on. Throughout the
5.1 Procedure guidelines, open questions were introduced asking students
Students had to follow a structured usage protocol in order to visit interaction analysis data and interpret the particular
to work with the main functionalities of INSPIREus, the visualizations. The structure of the worksheet is presented
learning design of the environment, the educational mate- in Table 1.
rial structure and contents, and specific indicators. Appro- Questionnaire. The questionnaire included open and close
priate guidelines and open questions were included in a ended questions, and it was structured in two Sections,
worksheet, leading students to work individually with the each one having a different focus (see Appendix A where
first concept of a scenario, visit specific indicators and try to the question appears at the first column, the number of stu-
interpret them, then perform a collaborative task, working dents that answered ‘Yes’/’No’ at the second/third col-
in groups of three or two (only two groups of two were umns, percentage of positive answers at the forth column):
210 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

TABLE 1 Since the indicator of learners’ effort reflects learners’

Interaction Protocol for Evaluating Visualizations engagement with the content focusing on the time that the
learner devotes to various types of resources, in the analysis
Indicator of learners’ effort. Guideline 1 (G1): Study the first page
of category A of the first concept of the scenario. process we cross-check students’ perceptions about the time
Question 1 (Q1): Estimate how much time you devoted to the they spent on resources with actual log data as well as with
various types of modules. their interpretations of the correlation of time with their
G2: Visit the ‘pie chart’ visualization and observe which type of way of studying. Thus, by comparing students’ answers to
module you spent more and less time on. the first question of the worksheet (see Table 1, Q1) to their
G3: Study the second page of category B of the first concept. log data reflected on the pie chart of the indicator (see
G4: Submit the assessment test of the concept and study the
Table 1, G2 and Fig. 2b), we conclude that 20.5 percent of
summary.
G5: Visit the ‘pie chart’ visualization and observe which type of the students had a different impression compared to the
modules you spent more and less time on. actual one depicted on the pie chart (another 20.5 percent
had a small declination whilst the 59 percent just copied the
Indicator of learners’ working style. G6: Check your
navigation history visualization for the first concept, observe pie chart). This provides evidence for the potential of the
the sequence you followed through the various types particular representation of the indicator of learners’ effort to
of content. increase students’ self-knowledge and support self-monitoring.
Q2: Interpret the strategy you followed while studying. The first two questions of the evaluation questionnaire pro-
Indicator of learners’ progress. G7: Check the analytical presenta- vide more evidence in this direction. Both questions aim at
tion of the first concept and record the time that you spent eliciting students’ interpretation of time. In Question 1 of
along with the pedagogical duration, and the number of your the evaluation questionnaire appearing in Appendix A
visits, at (a) the various types of modules, (b) each page cate- (study time of content), most students suppose that when
gory related to specific knowledge processes. they spend short/long time on studying specific content,
Q3: Write your reflections and try to interpret the way you
worked. this behavior reflects how easy/difficult the particular con-
Q4: Guess your next movement based on this data. tent is for them as well as their effort to study, whilst 25 per-
cent of the students don’t seem to be concerned about time.
Especially, 59 percent of the students link long study time
with hard study. In Question 2 of the evaluation question-
 Part A stimulated learners to interpret aspects of
naire appearing in Appendix A (study time of content of par-
their interaction behavior represented in the
ticular type), 77 percent of the students relate again (as in
indicators.
Question 1) their study time to the difficulty of the tasks
 Part B aims at eliciting if students would share their
they work with, whilst half of the students (50 percent) sup-
interaction behavior data and how they would use
pose that the time they spend on content of specific type
their peers’ data in order to select collaborators.
reflects their preferences for the particular type of content.
However, 39 percent of the students seem not to be inter-
5.3 Data Analysis and Results ested in the time.
Data gathered were 44 completed worksheets, log data of Since the indicator of learners’ working style focuses on the
these 44 students, and their evaluation questionnaires. Six learners’ navigation through the resources, students’ own
out of 50 students did not manage to complete the work- interpretations about the strategies they adopt were consid-
sheet mainly due to their own time restrictions. Based on ered important. Students were asked to study particular
demographic data, fifteen men and twenty-nine women content, and then to try to interpret their interaction data
participated in the study. Nine had a strong preference for (see Table 1, G6 and Fig. 4). By analyzing students’ reflec-
the ‘activist’ learning style, ten for the ‘theorist’ style, tions to the second question of the worksheet (see Table 1, Q2)
twenty-three for the ‘reflector’ style, and two for the we observe that most students interpret the sequence they
‘pragmatist’ style. Based on log data, two activists changed follow based on their style characteristics although they
their style during the interaction as they thought that the weren’t asked to. It is encouraging that all the students
instructional strategy adopted for presenting the content understand the visualizations. In particular, we distinguish
didn’t suit their preferences. four categories of answers: (a) descriptions of the sequence
they followed without interpretations of their selections
 How do students interpret visualizations of their interac- (5 students - 11.4 percent), (b) reflections on their preferred
tion behavior? style of studying which is in accordance with the system
Students’ answers to the worksheets and question- proposal (“I need to fully understand the theory/examples
naires were collected and categorized. The analysis and then go into action”) (31 students - 70.5 percent),
focused on the expressiveness of the interaction analysis (c) reflections on the type of the content they worked mainly
indicators based on the students’ self reports coming from with and the knowledge processes they intended to develop
their own answers to similar questions from the work- (e.g. “I focused on application and analysis of concepts in
sheet and the questionnaire, and combining them with order to accomplish the task”) or their needs ignoring the
real log data. We also investigated their self reports gath- system proposal which is based on their style characteristics
ering evidence for the metacognitive skills cultivated. (e.g. “I tried to do the activity but I needed more informa-
Below data analysis is presented about each of the various tion so I moved on to theory”) (8 students—18.2 percent).
indicators following the structure of the interaction proto- What is obvious in students’ reflections is the increased self-
col of Table 1. knowledge, as learners seem to recognise and comment on
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 211

TABLE 2 with the system (10 students in total: 9 from the second
Students’ Interpretations and Plans Inspired by the Indicator of group and 1 from the third group) (see Table 2, second col-
Learners’ Progress
umn, lack of familiarity with INSPIREus). Only students of the
Study time vs Interpretation Next movement first group seem to doubt the accuracy of the time recorded
Pedag. duration since “time runs even if the students work out of the sys-
tem” (see Table 2, second column, Doubt about data accuracy).
Students that No comments: 1 No comments: 3
spend less time Doubt about data Plan next steps focusing The students’ reflections also included comments about the
than the proposed accuracy: 4 on type of material: 5 strategies they adopt while studying particular content
one: 18.2% Strategy adopted focusing on (see Table 2, second column, strategy adopted): (a)
(8 out of 44) (style): 3 preferences for particular content that sometimes were also
linked to learning style preferences or to the usefulness of
Students that No comments: 3 Time restrictions: 2 the content, (b) difficulties that they faced with particular
spend more time Lack of familiarity No comments: 3
content, (c) demanding tasks asking for higher order think-
than the proposed with INSPIREus: 9 Focus on time
one: 56.8% Strategy adopted improvement: 2 ing skills and deep study. However, it is worth noting that
(22 out of 44) (style, deep study, Plan next steps focusing students of the first group link their way of studying only
difficulties): 13 on type of material: 5 with their style preferences (3 students out of 8). Moreover,
More focused study: 9 these visualizations seem to help students organize their
study focusing on specific type of content (see Table 2, third
Students that spend No comments 0: 1 Time restrictions: 1 column, plan next steps). Specially those students of the sec-
more time in some Lack of familiarity No comments: 4
ond and third groups elaborate on changes to their way of
cases and less in with INSPIREus: 1 Focus on time
other: 31.8% Strategy adopted improvement: 3 studying towards the desired one and goal setting related
(14 out of 44) (style, deep study, Plan next steps focusing to the improvement of the time they spent on various types
difficulties): 12 on type of material: 4 of resources leading to more focused study (see Table 2,
More focused study: 1 third column, more focused study). However, in three cases
(see Table 2, third column, time restrictions) the particular
data about time seem to have a negative impact on students’
their strengths/weaknesses /preferences and evaluate attitudes, making them focus on staying within the pro-
themselves. They also recognize the strategies they follow posed time by restricting their explorations only to the
and interpret them based on their style characteristics, goals available content.
or needs. Their answers to Question 3 and 4 of the evalua- This provides evidence for the potential of the indicator of
tion questionnaire appearing in Appendix A support the learners’ progress to increase the students’ strategic-knowledge
above findings. In Question 3 (navigation history in various as it stimulates conscious thinking about the way they use
types of content) students seem to relate their navigation resources and their usefulness in achieving outcomes.
and the time they spent on various resources to their pre- Moreover, students’ plans about their next step provide evi-
ferred studying/learning strategy (Question 3, 63 percent) dence for increased knowledge of plans and goals referring to
but also to the planning of their study (Question 3, 52 per- learners’ capacity to set what they intend to do through
cent). In Question 4 (high frequency of visits on specific their learning.
type of content), most students (61 percent) agree that such The above findings are corroborated through the
behavior is related to difficult content and few (27 percent) students’ answers to the six and seven questions of the eval-
relate it to understandability/usefulness of the content. uation questionnaire appearing in Appendix A. In Ques-
To assess the impact of the indicator of learners’ progress tion 6 (study time compared to time proposed by teacher)
we initially used log data to categorize students based on students again (as in Table 2) relate study time that exceeds
their behavior compared to the ‘desired’ one, resulting to the time proposed by the tutor with (a) demanding content
three categories of students, those that spent on studying (82 percent), i.e. focusing on the characteristics of the con-
the resources (see Table 2, column 1): (a) less time than the tent, (b) deep study of the content (79 percent), i.e. focusing
proposed one (18.2 percent), (b) more than the proposed on themselves and the way they study, (c) their interest in
one (50 percent) and (c) more time in some cases and less in the material (52 percent), i.e. focusing on themselves and
others (31.8 percent). Then, students’ reflections on their their learning preferences. Moreover, some students doubt
studying behavior and progress towards specific outcomes the accuracy of the time as a point of reference but in this
of particular page categories (see Table 1, G7, Q3 and Fig. 3) case they are referring to the time proposed by the tutor
were further analyzed and categorized in (a) the interpreta- which they consider “may be inaccurate due to wrong esti-
tions that appear in Table 2, column 2, and (b) the plans that mation of the tutor”; however only 63 percent of the stu-
appear in Table 2, column 3. The interpretations of the par- dents answered this option. Accordingly in Question 7 of
ticular indicator are quite interesting as they focus on the the evaluation questionnaire (study time compared to the mean
strategies that the students adopted seeking the origins of time of their group) students, by comparing themselves to
their behavior. We also observed that students’ interpreta- their peers, acknowledge the same origins of their behavior
tions of the first group differ from those of the second and as when having the tutor’s time as a point of reference, but
third ones as well as their plans. Specially students whose in different percentages, i.e. 64 percent of the students
behavior is ‘worse’ than the ‘desired’ one (i.e. those that report that spending more time studying the content than the
spent more time than the proposed one) try to argument mean time of their group reflects demanding content, 64
about this behavior usually linking it with lack of familiarity percent deep study of the content, and 45 percent of the
212 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

students report that it reflects their interest in the material. It metacognitive activities. It is worth noting that the indicator
is worth noting that 48 percent of the students would like to of learners’ effort helped them acquire an overview of how
know those peers with less time and good results. they organize/distribute their studying time. This informa-
Lastly, assessment data seem also to support students in tion seems to support them in evaluating their strengths/
organizing their study. In Question 5 (assessment tests evalu- weaknesses as well as the demands of the tasks they work
ating specific outcomes), many students state that this infor- on increasing their self and task knowledge. Furthermore,
mation helps them identify their level of performance (72 students’ reflections on the indicator of working style focus
percent) and weaknesses (82 percent), increasing their self on the evaluation of their behavior towards the attainment
knowledge. Moreover, 81 percent of the students suppose of specific outcomes making them identify their needs in
that this information helped them to change their planning order to deal with specific tasks (focus on tasks) as well as
and to focus on content that supports specific outcomes. the strategies they adopt and their preferences for particular
types of tasks (focus on themselves). These data provide evi-
 Which evaluation criteria do learners use for selecting col- dence for the impact that the particular indicator may have
laborators based on their interaction data? on their self and strategic knowledge. As a result, a useful
Most of the students (93 percent) answering Questions 9 guideline is that the visualizations should represent the con-
and 10 of the evaluation questionnaire would agree to open tent augmented with pedagogical information. In this way,
this information at class level and all of them (100 percent) learners are supported in relating their selections to the
to their group. Concerning the selection of collaborators expected outcomes and evaluating the role of various types
based on interaction data, various approaches are proposed of content and their own strategies in achieving them.
in Question 8 of the evaluation questionnaire. In particular, Lastly, students’ reflections on the indicator of progress
52 percent of the students would prefer as collaborators show that they use and interpret similar interaction data in
peers with a profile similar to their own, whilst 43 percent multiple ways. The comparison of their behavior to a
prefer peers with a totally different profile but great perfor- desired one seems to play a critical role as students interpret
mance. However, students’ performance compared to the data in different ways in cases of success or failure. On the
time proposed by the teacher seems not to be considered as one hand, success seems to make students concentrate on
a criterion for selecting collaborators since only 18 percent their way of working, helping them recognize the strategies
care about this information. Finally, it is interesting that they adopt, whilst in cases of failure the ‘desired’ state moti-
62 percent of the students do not care about study time and vates them to change their plans and strategies. As a result,
performance of their peers and focus on their answers to another useful guideline is that the ‘desired’ state should
questions, exercises and activities. follow learners’ movements providing a point of reference
like the state of an expert-tutor or real data coming from
6 DISCUSSION AND FUTURE PLANS peers who are worth following.
Furthermore, this research contributes to the investiga-
In this paper, we propose a model for producing interpre- tion of students’ attitudes towards viewing the learner
tative views of learners’ interaction behavior. This model model of others and making their models accessible to
focuses on the definition of relevant indicators of learners’ others. Most students are willing to share interaction data
interaction with the content and system tools which with peers. This information looks useful for selecting col-
can be further linked to specific learner characteristics, laborators, although various approaches were observed,
leading to useful recommendations. Challenging research reflecting differences in students’ preferences. Moreover,
goals in this process are selecting appropriate data from most students prefer their own behavior as a point of refer-
learners’ interaction, describing the ‘context’ that affects ence for selecting collaborators to the time proposed by the
learners’ actions and visualizing this information in a instructor. They also seem to appreciate their peers’ contri-
meaningful way. The proposed modeling approach can butions to various tasks more than time information. This is
be implemented in any adaptive and/or hypermedia edu- an argument for allowing and encouraging learners to share
cational system that has data with semantic information their ideas and answers. The design of social indicators will
and multiple levels of meaningful hierarchy such as the focus on this direction.
domain model. Currently, the evaluation of the learner model of INSPIR-
The empirical study performed with students working Eus is on progress. The next step will be to evaluate the
with the indicators of INSPIREus, provided evidence for the impact of the indicators in a real context involving also
expressiveness of the indicators. Preliminary results show tutors and allowing learners to freely interact with the con-
that students want to have access to information maintained tent. We also intend to further work on building interpreta-
by the system for them and their peers. In particular, based tive views of learners’ social interaction. We plan to involve
on the results of the study, most students managed to inter- learners in the interpretation of their interaction data in
pret their interaction data following a quite directive usage order to minimize arbitrariness in the identification of
scenario. This usage scenario proved to be quite useful in meaningful patterns [46] and link interaction behaviour
guiding students to use the various functionalities of the with their individual characteristics. This is considered criti-
system as well as in interacting with the visualizations cal for the system in order to personalise tasks, tools or
offered. This is in line with other studies focusing on how to study advice to individuals or groups, and to encourage
support learners in interpreting the contents of their model social interaction, providing a basis on which learners will
and using them creatively [51]. Actually, while students share their experiences as well as for group formation and
observe the particular indicators, they engage in several group development purposes.
PAPANIKOLAOU: CONSTRUCTING INTERPRETATIVE VIEWS OF LEARNERS’ INTERACTION BEHAVIOR IN AN OPEN LEARNER... 213

REFERENCES [21] J. Kay, N. Maisonneuve, K. Yacef, and P. Reiman, “The big five
and visualisations of team work activity,” in Proceedings of the 8th
[1] C. Romero and S. Ventura, “Educational Data mining,” IEEE International Conference on Intelligent Tutoring Systems (ITS2006),
Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 40, no. 6, pp. 601–618, M. Ikeda, K. Ashley, and T.-W. Chan, Eds., Lecture Notes in Com-
Oct. 2010. puter Science, Berlin, Germany: Springer, pp. 197–206, vol. 4053,
[2] R. Farzan, P. Brusilovsky, “Social navigation support in e-learn- 2006.
ing: what are the real footprints? ,” Intell. Techniques Web Personali- [22] J. Janssen, G. Erkens, and P. A. Kirschner, “Group awareness
sation, vol. 8, no. 1, pp. 49–80, 2005. tools: It’s what you do with it that matters,” Comput. Human Behav.
[3] A. Dimitrakopoulou, A. Petrou, A. Martinez, J. A. Marcos, V. Edu., vol. 27, pp. 1046–1058, 2011.
Kollias, P. Jermann, A. Harrer, Y. Dimitriadis, and L. Bollen. [23] D. Clauzel, K. Sehaba, and Y. Pri
e, “Enhancing synchronous col-
(2006). “State of the art of interaction analysis for Metacognitive laboration by using interactive visualisation of modelled traces,”
Support & Diagnosis”.[Online]. Available: http://hal.archives- Simulation Modelling Practice and Theory, vol. 19 no. 1, pp. 84–97,
ouvertes.fr/hal-00190146/ 2011.
[4] J. Janssen, G. Erkens, P. A. Kirschner, “Group awareness tools: It’s [24] Th. Bratitsis and A. Dimitrakopoulou, “Interpretation issues in
what you do with it that matters,” Comput. Human Behav., vol. 27, monitoring and analyzing group interactions in asynchronous
no. 3, pp. 1046–1058, 2011. discussions,” Int. J. e-Collaboration, vol. 4, no. 1, pp. 20–40, 2008.
[5] L.P. Macfadyen and S. Dawson, “Mining LMS data to develop an [25] C. Glahn, M. Specht, and P. Koper, “Smart Indicators on Learning
“early warning system,” Comput. Edu., vol. 54 no. 2, pp. 588–599, Interactions,” in Creating New Learning Experiences on a Global Scale,
2010. Lecture Notes in Computer Science 4753, E. Duval, R. Klamma, M.
[6] R. Mazza and V. Dimitrova, “CourseVis: A graphical student Wolpers (eds.), Berlin, Germany: Springer, 2007.
monitoring tool for supporting instructors in web-based distance [26] J. Vassileva and L. Sun, “Evolving social visualization design aimed
courses,” Int. J. Human-Comput. Studies, vol. 65, no. 2, pp. 125–139, at increasing participation in a class-based online community,” Int.
2007. J. Cooperative Inf. Syst., vol. 17, no. 4, pp. 443–466, 2008.
[7] A. Adesina and D. Molloy, “Virtual Learning Process Environ- [27] K. Gabner, M. Jansen, A. Harrer, K. Herrmann, and H.U. Hoppe,
ment: Cohort Analytics for learning and learning processes,” “Analysis methods for collaboration models and activities,” pre-
World Acad. Sci., Eng. Technol., vol. 6, no. 5, pp. 55–64, 2012. sented at the Comput. Supported Collaborative Learn. Conf., Ber-
[8] B. M. McLaren, K. R. Koedinger, M. Schneider, A. Harrer, and gen, Norway, 2003.
L. Bollen, “Bootstrapping novice data: Semi-automated tutor [28] T. Mochizuki, H. Kato, S. Hisamatsu, K. Yaegashi, S. Fujitani, T.
authoring using student log files,” presented at the Workshop Nagata, J. Nakahara, T. Nishimori, and M. Suzuki, “Promotion of
Analyzing Student-Tutor Interaction Logs Improve Educ. Out- Self-assessment for learners in online discussion using the visuali-
comes, 7th Int. Conf. Intell. Tutoring Syst., Macei
o, Brazil, zation software,” presented at the Comput. Supported Collabora-
Aug./Sep. 2004. tive Learn. Conf., Taiwan, 2005.
[9] M. Bienkowski and M. B. Feng, “Means, Enhancing Teaching and [29] A. Gogoulou, E. Gouli, and M. Grigoriadou, “Adapting and per-
Learning Through Educational Data Mining and Learning Analyt- sonalizing the communication in a synchronous communication
ics: An Issue Brief,” U.S. Dept. Edu., Washington, DC, USA, 2012. tool,” J. Comput. Assisted Learn., vol. 24, pp. 203–216, 2008.
[10] A. Mitrovic and B. Martin, “Evaluating the effect of open student [30] A. Soller, A. Martinez, P. Jermann, and M. Muehlenbrock, “From
models on self-assessment,” Int. J. Artif. Intell. Edu., vol. 17, no. 2, Mirroring to Guiding: A review of state of the art technology for
pp. 121–144, 2007. supporting collaborative learning,” Int. J. Artif. Intell. Edu., vol. 15,
[11] S. Bull and J. Kay, “Student Models that Invite the Learner In: The no. 4, pp. 261–290, 2005.
SMILI Open Learner Modelling Framework,” Int. J. Artif. Intell. [31] Weber, G. and Brusilovsky, P. “ELM-ART: An adaptive versatile
Edu., vol. 17, no. 2, pp. 89–120, 2007. system for web-based instruction,” Int. J. AI Educ. , vol. 12, no. 4,
[12] S. Bull and J. Kay, “Open learner models as drivers for metacogni- pp. 351–384, 2001.
tive processes,” Int. Handbook Metacog. Learning Technol., Springer [32] P. Brusilovsky, “KnowledgeTree: A distributed architecture for
International Handbooks of Education, vol. 28, pp. 349–365, adaptive e-learning,” in Proceedings of the Thirteenth International
Springer New York, 2013. World Wide Web Conference. New York, NY, USA: ACM Press,
[13] S. Govaerts, K. Verbert, J. Klerkx, and E. Duval, “Visualizing activ- pp. 104–113, 2004.
ities for self-reflection and awareness,” International Conference on [33] J. D. Zapata-Rivera and J. Greer, J., “Inspecting and visualizing
Web-based Learning 2010, X. Luo et al. (Eds.), Lecture Notes in distributed bayesian student models,” in Intelligent Tutoring Sys-
Computer Science, vol. 6483, pp. 91–100, Springer Berlin Heidel- tems, G. Gauthier, C. Frasson, and K. VanLehn (Eds.) Berlin,
berg, 2010. Germany: Springer, pp. 544–553, 2000.
[14] R. Mazza, L. Mazzola, C. Glahn, D. Verpoorten, A. Nussbaumer, [34] A. Mabbott and S. Bull, “Alternative views on knowledge: Presen-
C. Steiner, and D. Heckmann, “Design of interactive visualization tation of open learner models,” in Proc. 8th Int. Conf. Intel. Tutoring
of models and students data,” Deliverable 4.5a of GRAPPLE, FP7 Syst. (ITS2004), J. C. Lester, Eds., Lecture Notes in Computer Sci-
European Project, 2009. ence, vol. 3220, Berlin, Heidelberg: Springer, 2004, pp. 689–698.
[15] C. Glahn, M. Specht, and R. Koper, “Visualisation of interaction [35] I. H. Hsiao, F. Bakalov, P. Brusilovsky, and B. K€ onig-Ries,
footprints for engagement in online communities,” Educational “Progressor: Social navigation support through open social stu-
Technology & Society, vol. 12 no. 3, pp. 44–57, 2009. dent modeling,” New Review Hypermedia Multimedia, vol. 19, no. 2,
[16] J-F. Rouet and J.-M. Passerault, “Analysing learner-hypermedia pp. 112–131, 2013.
interaction: An overview of on-line methods,” Instructional Sci., [36] J. Vassileva, G. McCalla, and J. Greer, “Multi-agent multi-user
vol. 27, pp. 201–219, 1999. modeling in i-help,” User Model. User-Adapted Interaction, vol. 13,
[17] J. Mostow and J. Beck, “Some useful tactics to modify, map and no. 1–2, pp. 179–210, 2003.
mine data from intelligent tutors,” natural language engineering, [37] M. A. Chatti, A. L. Dyckhoff, U. Schroeder and H. Th€ us. “A refer-
vol. 12, no. 2, pp. 195–208, Cambridge University Press UK, Jun. ence model for Learning Analytics,” Int. J. Technol. Enhanced
2006. Learn., vol. 4, nos. 5–6, pp. 318–331, 2012.
[18] N. Avouris, V. Komis, G. Fiotakis, M. Margaritis, and E. Voyiat- [38] J. D. Zapata-Rivera and J. Greer, “Analyzing learner reflection in
zaki, “Logging of fingertip actions is not enough for analysis of the learning game,” in Proc. Workshop Learner Model. Reflection,
learning activities,” presented at the Workshop Usage Anal. AIED Conf., pp. 288–298, 2003.
Learn. Syst., Amsterdam, he Netherlands, July. 2005. [39] P.R. Pintrich, “The role of metacognitive knowledge in learning,
[19] H. Zhang, K. Almeroth, A. Knight, M. Bulger, and R. Mayer, teaching, and assessing,” Theory Practice, vol. 41, no. 4, pp. 219–
“Moodog: Tracking students’ Online Learning Activities,” in 225, 2002.
Proceedings of the World Conference on Education, Multimedia, [40] C. McLoughlin and R. Hollingworth, “The weakest link: Is web-
Hypermedia and Telecommunications 2007, C. Montgomerie and based learning capable of supporting problem-solving and meta-
J. Seale, eds., Chesapeake, VA, USA: AACE, pp. 4415–4422, cognition?,” in Proc. ASCILITE, pp. 117–120, 2001.
2007. [41] K.A. Papanikolaou, M. Grigoriadou, H. Kornilakis, G.D.
[20] R. Mazza and L. Botturi, “Monitoring an online course with the Magoulas, “Personalizing the interaction in a Web-based educa-
GISMO Tool: A case study,” J. Interactive Learn. Res., vol. 18, no. 2, tional hypermedia system: The case of INSPIRE,” User-Modeling
pp. 251–265, 2007. User-Adapted Interaction, vol. 13, no. 3, pp. 213–267, 2003.
214 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 8, NO. 2, APRIL-JUNE 2015

[42] C. N. Gunawardena, K. Carabajal, and C. A. Lowe, “Critical anal- Kyparisia A. Papanikolaou is currently an
ysis of models and methods used to evaluate online learning Assistant Professor in the Department of Educa-
networks,” in American Educational Research Association Annual tion, School of Pedagogical and Technological
Meeting. Seattle, WA, USA: American Educational Research Asso- Education, Heraklion, Greece. Her research work
ciation, 2001. has contributed to the design of web-based adap-
[43] D.R. Garrison and N.D. Vaughan, Blended learning in higher educa- tive learning environments (personalization at
tion: Framework, principles and guidelines. New York, NY, USA: individual and group level, open learner model-
Wiley, 2008. ing, group formation), and web-based learning.
[44] M. Kalantzis and B. Cope, New Learning: Elements of a Science She has published over 50 papers in international
of Education, 2nd ed., Cambridge, United Kingdom: Cambridge journals and conferences. She is a member of
University Press, 2012. the editorial board of the International Journal
[45] ARIADNE project. [Online]. Available at: http:// ariadne-eu.org, of Learning Technology and a member of the executive review board of
Oct. 2014. the Educational Technology and Society journal.
[46] C. Brooks and G. McCalla, “Towards flexible learning object meta-
data,” Int. J. Continuing Eng. Edu. Life-Long Learn., vol. 16, no. 1–2,
pp. 50–63, 2006.
[47] V. Dimitrova, G. McCalla, S. Bull, “Open learner models: Research
Questions,” J. Artif. Intell. Edu., vol. 17, no. 2, pp. 217–226, 2007.
[48] A. Dimitracopoulou, “Computer based interaction analysis sup-
porting self-regulation: Achievements and prospects of an emerg-
ing research direction,” Technol. Instruction Cognition Learn., vol. 6,
no. 4, pp. 291–314, 2009.
[49] N. Van Labeke, P. Brna, R. Morales, “Opening up the interpreta-
tion process in an open learner model,” Int. J. Artif. Intell. Edu.,
vol. 17, no. 3, pp. 305–338, 2007.
[50] D. Zapata-Rivera, E. G. Hansen, V. J. Shute, J. S. Underwood, and
M.I. Bauer, “Evidence-based approach to interacting with open
student models,” Int. J. Artif. Intell. Edu., vol. 17, no. 3, pp. 273–
303, 2007.
[51] D. Zapata-Rivera and J. Greer, “Exploring various guidance
mechanisms to support interaction with inspectable learner mod-
els,” Proc. Intell. Tutoring Syst., pp. 442–452, 2002.
[52] B. Shneiderman, “The eyes have it: A task by data type taxonomy
for information visualizations,” in Proc. IEEE Symp. Vis. Lang.,
pp. 336 –343, 1996.
[53] D. Roy, S. Sarkar, and S. Ghose, “A comparative study of learning
object metadata, learning material repositories, metadata
annotation and an automatic metadata annotation tool,” M. Joshi
H. Boley, and R. Akerker (Eds.), Adv. Semantic Comput., vol. 2,
pp. 103–126, TMRF e-Book, 2010.
[54] K. Papanikolaou and M. Grigoriadou, “Combining adaptive
hypermedia with project and case based learning,” Int. J. Edu.
Multimedia Hypermedia, vol. 18, no. 2, pp. 191–220, 2009.
[55] I.-H. Hsiao, S. Sosnovsky, and P. Brusilovsky, “Guiding students
to the right questions: Adaptive navigation support in an E-Learn-
ing system for Java programming,” J. Comput. Assisted Learn.,
vol. 26, no. 4, pp. 270–283, 2010.
[56] S. Bull, A. Mabbott, and A. Abu-Issa, “UMPTEEN: Named and
anonymous learner model access for instructors and peers,” Int.
J. Artif. Intell. Edu., vol. 17, no. 3, pp. 227–253, 2007.