IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO.

2, APRIL-JUNE 2020 387

Recommender Systems and Scratch: An

Integrated Approach for Enhancing
Computer Programming Learning
Jesennia C
ardenas-Cobo , Amilkar Puris , Pavel Novoa-Hern
andez ,
Jos
e Angel Galindo , and David Benavides

Abstract—Learning computer programming is a challenging I. INTRODUCTION

process. Among the current approaches for overcoming this
challenge, visual programming languages (VPLs), such as Scratch,
have shown very promising results for beginners. Interestingly,
some higher education institutions have started to use VPLs to
N OWADAYS, learning computer programming remains a
challenging task. It generally comprises of hard skills
related to problem solving (e.g., modeling, abstraction, logical
introduce basic programming concepts, mainly in CS1 courses. thinking, and coding), which are difficult to teach or learn. To
However, an important issue regarding Scratchs usage in higher cope with such difficulties, several approaches have been sug-
education environments is that students may feel unmotivated gested in the past, which include collaborative work [1], simu-
being confronted by programming exercises that do not fulfill their lation tools [2], games [3], pair programming [4], and role
individual expectations. To try and overcome this barrier, we
games [5].
propose CARAMBA, a Scratch extension including an exercise
recommender system. Based on features, such as taste and In this context, visual programming languages (VPLs) have
complexity, CARAMBA is able to personalize student learning proven to be important tools for beginners with little or no expe-
with Scratch by suitably suggesting exercises for students. An in- rience in programming. This is the case among most children
depth evaluation was conducted about the effects of our proposal and high-school students. By mapping conventional program-
on both the learning of basic concepts of CS1 and the overall ming concepts into visual metaphors, VPLs have significantly
performance of students. We adopted an equivalent pretest- impacted the learning of computer programming [6]. Popular
posttest design with 88 college students at an Ecuadorian VPLs like Alice [7], RAPTOR [8] and Etoys [9] have been used
university. Results confirm that recommending exercises in at different educational stages. Scratch is perhaps the most
Scratch had a positive effect on students programming learning widespread VPL today [10], [11]. Although it was initially con-
abilities in terms of pass rates. In totality, the pass rate achieved by ceived for children, few higher-education institutions have
our proposal was over 52%, which is 8% higher than the rate adopted it for introductory programming courses [12]–[14].
achieved during a previous experience using only Scratch (without
recommendation) and 21% higher than the historical results of
This was the case with the Universidad Estatal de Milagro
traditional teaching (without Scratch). Furthermore, we analyzed (UNEMI) in Ecuador, where Scratch was employed as a com-
the degree of exploitation of CARAMBA by students to portray plementary tool to consolidate introductory courses on pro-
two facts: students actually used CARAMBA and there was a gramming (CS1). The decision to use Scratch in UNEMI was
significant, positive correlation between the utilization of based on the following two reasons:
CARAMBA and the scores obtained by the students.
the reported success of Scratch in similar educational
Index Terms—Scratch, recommender systems, visual contexts; and
programming languages, programming learning.
the students low performance in CS1 since the begin-
ning of the Computer Science Engineering program in
2003 (as the pass rate in UNEMI has historically been
Manuscript received August 21, 2018; revised February 8, 2019; accepted under 50%).
February 18, 2019. Date of publication February 24, 2019; date of current ver-
sion June 18, 2020. This work was supported by the Project FOCICYT: Using Scratch over the last two semesters, the pass rate has
Soft Computing Applications in Higher Education Environments, funded not only increased, but it has also stabilized. Notwithstanding
by the Universidad T
ecnica Estatal de Quevedo (2017–2018); in part, by the positive experience, some issues regarding its utilization
Spanish government under the Juan de la Cierva Postdoctoral program; by
the Andalussian governement under the METAMORFOSIS project; by the were observed. We found that certain students felt unmoti-
project FOCICYT (2017-2018): Soft Computing Applications in Higher vated with the assignments (exercises) suggested by the lec-
Education Environments, financed by the Universidad T
ecnica Estatal de turer during the course. For instance, when high-performance
Quevedo (Ecuador) and; by the Universidad Estatal de Milagro (Ecuador). students were faced with simple exercises or vice versa. A
(Corresponding author: Jos
e Angel Galindo.) similar experience was reported by [15].
J. C
ardenas-Cobo is with the State University of Milagro, Milagro 091050,
Ecuardor (e-mail: [email protected]). To overcome this issue and exploit the benefits of Scratch, in
A. Puris and P. Novoa-Hern
andez are with the Technical State University of this study we have presented a novel system called CARAMBA.
Quevedo, Quevedo, Ecuador, and also with the State University of Milagro, It extends Scratch by including a recommender system for sug-
Milagro EC120503, Ecuardor (e-mail: [email protected]; [email protected]). gesting exercises (problem statements) to the student. Based on a
J. A. Galindo and D. Benavides are with the Department of Computer
Languages and Systems, Universidad de Sevilla 41012, Seville,
collaborative-filtering approach, this intelligently suggests exer-
Spain (e-mail: [email protected]; [email protected]). cises to students based on their interests and level of assessment.
Digital Object Identifier 10.1109/TLT.2019.2901457 During proposal, we perceived high levels of student satisfaction
1939-1382 ß 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
388 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

regarding the proposed system. Through simple questionnaires, concluded that “the use of Scratch in the very beginning of the
the students believed that Scratch and a recommender system for CS1 course promoted a high level of motivation, thus a positive
exercises would help improve their overall performance in CS1. perception of learning programming”. However, the author did
Despite such a positive reception, it was considered that a more not observe “any measurable improvement of the results
serious analysis was needed to gain better insights on the true obtained by students who used the tool compared with the nor-
impact of the proposal. Thus, we have statistically validated the mal course”. The same holds true for the pass rates, where no sig-
effects of the recommending exercises with Scratch on both the nificant differences existed. The author justified these outcomes
learning of programming concepts and the overall performance by the fact that students of the Scratch course were affected by
of the students. An equivalent group design was adopted to ana- two (learning) “jumps”: Scratch-to-Java, whilst the control group
lyze the three different learning strategies: was only affected by one “jump”: Java. In other words, it seems
1) traditional; that the benefits acquired by starting with Scratch disappeared
2) Scratch-only; and when students had to face Java in the middle of the course. This
3) Scratch plus recommending of exercises (CARAMBA). seems to suggest that a transitional education strategy from
It is worth mentioning that this study complements our pre- Scratch to Java (or any other conventional programming lan-
liminaries results [16] in two major directions: 1) the develop- guage) is required to overcome the “two-jump” syndrome.
ment of an improved (and more mature) system, and 2) the The attitudes and opinions of future teachers about Program-
prosecution of a deeper analysis about the effects of this sys- ming and Information and Communications Technology (ICT)
tem on student learning. in Education were explored by [17] using Scratch in a CS1. Per-
From the results obtained we concluded that recommending haps the most relevant aspect of this research is that it was
exercises in Scratch has a significant and positive impact on restricted to women studying preschool education and educa-
the learning of CS1 concepts in college students. Moreover, it tional design. The authors observed that Scratch was considered
improves the overall performance of the students. We espe- user-friendly by students and helped to increase their interest in
cially showed that CARAMBA surpasses the traditional exploiting ICT in education. Consequently, the stress and anxiety
approach by 21% and the Scratch-only approach by 8%. related to using ICT in educational practices tended to decrease.
This paper is structured as follows: Section II reviews related In [18], the role of Scratch in teaching CS1 was discussed.
studies, while Section III presents the main features of CAR- The author specifically explored how Scratch could be used in
AMBA as an integrated approach, including technical aspects business programming. Although the paper reported prelimi-
of the implemented recommender system. Section IV is devoted nary results, the author confirmed that Scratch “allows educa-
to the validation of CARAMBA, while Section V outlines the tors to reduce the cognitive load that students experience
discussion of the results and a conclusion of the research. when first introduced to programming”.
A broader research, as presented in [13], assessed the impact of
a short intervention by Scratch in a CS1 course. The idea was to
II. RELATED WORK use Scratch to facilitate the introduction of a more complex pro-
In this Section, we go through related works about both gramming language, in this case C++. The study involved 332
Scratch and Recommender Systems at the university level. It is first-year undergraduate engineering students and the authors
important to note that most literature related to Scratch is about recorded both the quantitative and qualitative data for assessing
experiences with elementary and high school students. This is the impact of Scratch. Analysing the collected data, the authors
to be expected, since Scratch was conceived for children and concluded that a two-week Scratch intervention, along with spe-
teenagers. However, as we will show in the next section, from cific educational resources (e.g. labs and projects), would be ben-
the very beginning, few researchers have been interested in eficial for learning basic programming concepts. Thus, both the
how to exploit it for enhancing programming learning in CS1. novices and advanced learners could be satisfied. However, it
was additionally noted that such an intervention did not help
novices obtain grades similar to those of advanced learners when
A. Scratch in College Environments faced with typical programming exam questions.
Perhaps, the first study using Scratch in higher education A similar study was conducted by [19] aiming to facilitating
was [10]. Here, the authors included Scratch in the Harvard the learning of programming concepts by undergraduate Com-
Summer School: The Great Ideas in Computer Science (a puter Engineering students. The study was developed at the
summer school version of a course at Harvard College). The State University of Feira de Santana and involved a particular
goal was to improve first-time programmers experiences in learning approach “based on peer support, game development,
CS1 by devoting one week to explaining general CS concepts a challenge-response strategy and Scratch”. Through several
using Scratch. Afterwards, students would transition to Java surveys and the analysis of source code the authors found that
for the remainder of the course. From conducted surveys, the students considered Scratch to be “user-friendly... with good
authors concluded that “Scratch excites students at a critical working mechanics, especially the Lego-style blocks”.
time (i.e. their first foray into computer science)”. Scratch also Two positive experiences were reported in [19] and [20]
“familiarizes the inexperienced among them with fundamen- regarding introducing programming to pre-service teachers
tals of programming without the distraction of syntax”. with Scratch. After attending theoretical lessons, students
In a similar study, [12] reported an experience at an developed an educational app and wrote a report highlighting
Uruguayan university. This study aimed to analyze the use of both the negative and positive impressions on their individual
Scratch in a CS1 course, before introducing the Java program- experiences. As a result, it was observed that students posi-
ming language. To improve learning with Scratch, the author tively assessed important Scratch’s features.
proposed specific exercises to be solved with the same. By con- Recently, Martinez et al. [21] presented a negative experi-
ducting both a qualitative and quantitative analysis, the author ence in which authors used the extension Dr. Scratch in the

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 389

TABLE I improve students’ autonomous learning. While in e-commerce

CARAMBA VS. OTHER CONTRIBUTIONS FROM LITERATURE
domains RSs suggest products or services to clients, in e-learn-
ing environments RSs suggest educational resources (e.g.,
papers, books, or courses) to educational participants (e.g., stu-
dents and/or teachers).
Related literature shows a wide variety of works proposing
RSs for e-learning environments. Few important works in the
field are briefly explained further. For an in-depth survey, the
reader is to refer to [30] and, more recently, to [31].
The RS CourseAgent was conceived with the idea of
enabling students to provide feedback in implicit and explicit
ways [32]. This RS allowed students to directly evaluate
first weeks of a video-games course. Results showed that stu-
courses with respect to (i) their relevance regarding career
dents were not additionally motivated using Scratch as stu-
goals and (ii) the level of difficulty of the course. Both parame-
dents associated Scratch with pre-university level courses and
ters provided implicit feedback when students planned or regis-
not serious programming.
tered for a course. The basic and evident benefits of this RS for
Few important conclusions can thus be drawn from the
students were as follows: (i) it constituted a course manage-
above experiences:
ment system that retained information about the classes taken,
1) The common practice in college education when using and (ii) it facilitated communication with student advisors.
Scratch is to perform a short intervention, as the first The Virtual University of Tunis develops automatic recom-
interface between a novice and complex-to-learn pro- mendations in e-learning platforms [33]. These are composed
gramming languages; of two modules: an offline module that pre-processes data to
2) Additional educational strategies are required to per- build learner and content models; and an online module that
form an effective intervention with Scratch. In other uses those models on the fly to recognize students’ needs and
words, using Scratch as it was originally conceived for goals to predict suitable recommendation lists.
children is not enough to cope with the expectations of It has been argued that traditional RSs are not suitable for
college students regarding computer programming; and supporting e-learning since they have not yet taken into account
3) No previous literature exists about extending Scratch tech- two important mechanisms: the learning processes and the
analysis of social interaction [34]. To deal with these issues,
nologically to enhance programming learning in colleges.
the authors of the argument proposed a flexible approach
Although there are similar experiences, like Dr. Scratch involving a multidimensional recommendation model and a
[22], [23], they are not employed in college scenarios. Markov Chain model. Results showed that more suitable rec-
Thus, to provide better insights about the features of available ommendations could be obtained from such an approach. Simi-
systems for enhancing computer programming through VPLs, lar research with a personalized approach was proposed, which
in the Table I we compared the same including our proposal relied on data mining and natural language process technolo-
CARAMBA. The features for consideration were: 1) the pres- gies to determine learner relationships based on the learning
ence of a recommendation system, 2) whether the research was processes and learning activities [35].
evaluated or validated, 3) whether it extended the employed To guide learners in personalized, inclusive e-learning sce-
VPL, and 4) which VPL was used. narios, an important analysis was conducted on how RSs could
be applied to e-learning systems [36]. Here, three technologi-
B. Recommender Systems and Learning Environments cal requirements for developing semantic education RSs were
This Section is inspired by the related work of our previous provided. Other authors have additionally reported successful
work presented in [16]. An RS is a software tool that deter- experiences using similar ideas. See for example [38]–[40].
mines and suggests what a particular user would find useful A framework for the rapid prototyping of knowledge-based
[24]. It is also considered to be a part of the so-called informa- RSs was reported in [41]. This was used for recommending
tion filtering system, which exploits user information for pre- learning objects. From a software development perspective, the
dicting ratings or preferences that the user would give to an proposed framework was flexible enough to implement new
actual item [25], [26]. Thus, the basic benefit of an RS is that approaches and included default implementations of alternative
it finds the most suitable set of items for a target user by maxi- strategies for each of its five stages. Two RSs were imple-
mizing its rating prediction. mented to illustrate the benefits of this framework.
According to [24] five types of RSs exist: content-based, Related to the technological benefits of RS in education, there
knowledge-based, demographic, community-based, collabora- are subjective dimensions to this topic, which are also important
tive, and hybrid. Collaborative Filtering RSs (CFRSs) have for study. For instance, a psychological view of learning with
probably been the most widely used [27], [28]. These were personalized RSs was provided by [42]. Here, a very good fit
built on the assumption that one user might like items that between the main features of RSs (collective responsibility, col-
other users with similar tastes had liked in the past. This lective intelligence, user control, guidance, and personalization)
assumption was adopted for the RS that was implemented in and the principles in learning sciences was demonstrated. How-
the present work. CFRSs involve two major approaches: user- ever, the authors claimed that a “recommender systems should
user and item-item [28]. In general, both approaches rely on not be transferred from commercial to educational contexts on a
the Nearest Neighbors algorithm [27]. one-to-one basis, but rather need adaptations in order to facilitate
A major area where RSs have been broadly applied is learning.” In this context, few potential adaptations were
e-learning environments, within the context of TEL [29], to grouped into system-centered adaptations (e.g., for enabling

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
390 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

Fig. 1. Comparison between the Scratch-only approach and the CARAMBA approach.

proper functioning in educational contexts) and social adapta- III. CARAMBA: AN INTEGRATED APPROACH
tions (e.g., for addressing typical information processing biases).
The proposed system aims to improve the current state of
Similar to the previous work, in [43] a conceptual framework
teaching Foundations of Computer Programming at Universi-
was proposed to explain how evolving recommender-driven
dad Estatal de Milagro, Ecuador, with Scratch. Fig. 1 helps
online learning systems (ROLS) support students, course
demonstrate both the current and the proposed approaches for
authors, course instructors, system administrators, and policy
such a process.
makers in development. Moreover, this framework involved
In the first approach (Fig. 1(a)), the professor interacts with
two important perspectives in the constructivist paradigm of
students exclusively by means of classroom lectures (formal
learning: cognitive and situative.
learning). The students use Scratch in the traditional form, by
Additionally, an interesting approach to enhance RSs in col-
creating or modifying projects (informal learning). Despite the
laborative learning environments has been presented, which
benefits of this method, the professor often encounters difficul-
consists of an influence diagram including observable varia-
ties in controlling the efficiency of the students interaction with
bles for assessing the collaboration among users [44]. By
Scratch (i.e., whether the students are properly traversing the
applying machine learning techniques, the influence diagram
knowledge levels in which the course is organized). In light of
was refined to increase its accuracy. The main outcome was
these facts, a gap between the informal and formal learning
the development of an automatic RS alongwith a pedagogical
support system in the form of a decision tree, which provided approaches is observed. Additionally, as observed over the
visual explanation to the user. course of one year, certain students have not been satisfied with
Furthermore, a generic cloud-based architecture for a sys- the complexity of the exercises they were asked to solve with
tem that recommends learning elements according to the Scratch. For instance, the more experienced students were
affective state of the learner was presented [45]. The authors faced with Scratch projects that they deemed too easy.
also provided some use cases, explaining implementation. A possible solution for these issues is a personalized set of
Undoubtedly, this is an interesting technological solution for exercises for students. (An exercise is defined in the context of
exploiting cloud-based learning environments, which is a this research as a problem statement that the student can solve
common feature in many education institutions. in Scratch.) However, this demands for the individual charac-
An important survey on how to evaluate RSs was conducted terization of students to assign to them the most suitable set of
in the context of TEL [46]. From an in-depth survey obtained exercises, according to their knowledge level and expecta-
from 235 works on the subject, it was concluded that there tions. This is a difficult task for the professor, mainly because
exists an important interest to design better strategies for eval- there are too many students and exercises to assign. Moreover,
uating RSs in TEL. Future trends and research opportunities since student learning is a dynamic process, both the student
were also highlighted in the study. characterization and the suggested exercises are expected to
Summarizing the above review, three important conclusions evolve over time. Hence, this assignation process is to be
can be drawn: repeated over and over.
1) using RS in educational environments is a popular An alternative solution is to increase the amount of time in labo-
ratory practice with the professor being present. Thus, the professor
research topic with increasing associated studies;
could control the student’s interaction with Scratch. However, this
2) most of the existing works are technology-based, where would put too much emphasis on formal learning, which contra-
RSs are proposed for enhancing the learning process. dicts our education goals. The challenge is thus how to improve
Contrarily, just a few works employ a subjective perspec- the current approach with the least level of professor inter-
tive like [42] to analyze the role of RS in this context; and vention possible. More specifically, it is important to find
3) as per our knowledge there are no RSs supporting pro- solutions to issues with the current approach while main-
gramming learning with Scratch. taining benefits of the employed learning approaches.
Conclusions 1 and 3 thus provide additional reasons for pro- In keeping with this consideration, we propose CAR-
posing the system explained next. AMBA, a Web Application composed by Scratch as a project

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 391

The student profile is thus built based on his/her assess-

ments (Taste and Complexity) about the solved exercises. In
the next subsection we provide additional technical descrip-
tions about the RS for exercises. Following this, we briefly
describe the main functionalities of CARAMBA.

A. Technical Aspects of the Recommender System

The RS included in our Web Application is based on a col-
laborative filtering (CF) approach, using both users and items
[25], [26] and relying on the surprise library implementation
artefacts (http://surpriselib.com/). In this context, the users are
the students, while the items are the exercises to be solved in
Scratch. The aim of this RS is to exploit the experience of
existing students and suggest suitable exercises to a certain
student. In the following sections, we refer to such a student
as an active student.
To build the CF model, we assume that we have a set of n
students fu1 ; u2 ; u3 ; . . . ; un g and a set of m exercises fe1 ; e2 ;
e3 ; . . . ; em g, with each student assessing an exercise according
to the following two criteria:
1) Taste (I 2 f1; 2; 3; 4; 5g). This measures how interest-
ing the student found the exercise. A value close to 1
means that the exercise is not interesting at all for the
student, while a value of 5 means the opposite.
2) Complexity (C 2 f1; 2; 3; 4; 5g). This states the com-
plexity level of the exercise from the student’s point of
view. For this variable, 1 means a lower complexity, 2 a
Fig. 2. Main flow of CARAMBA. medium complexity, and 3 a higher complexity.
In this manner, each student would have a record of
viewed exercises along with their corresponding evalua-
editor along with an RS for exercises (see Fig. 1(b)). Please tions. By using this record, the goal is to determine which
notice that this proposal not only allows for the interactions in students have common exercises and an evaluation that’s
the current approach, but also personalizes the learning pro- similar to other students. Once this first filter is applied, the
cess of students using Scratch. By this, the professor is able to next step is to recommend the exercises that are evaluated
control students learning processes by creating exercises and by other students but not yet viewed by the active student
including them in the system. Furthermore, students have the in question. Such a recommendation system involves the
opportunity to assess the exercises in order to inform the sys- following processes:
tem of their personal preferences. Using this information, the 1) Cold Start: During the implementation of an RS in real
system suggests new exercises to the student, assuming that environments, a common difficulty is how to recommend
students with similar tastes and complexity perceptions about exercises when no user experience or data exists. This issue is
exercises are a good source for recommendations. known as Cold start [47]. In the case of our RS, we have two
From a pedagogical perspective, this proposal is a new scenarios:
mediator between professors and students. In the following
the system has no recorded user experience. In this
section, the proposal is presented in detail, where the devel- case, a random set of exercises is proposed to the active
oped Web Application is described through its main modules
student from the pool of all available exercises; and
and features and then the technical aspects of the implemented
RS are explained.
the active student has no experience recorded by the
The main workflow of the interaction between the student system. This is the case for new users. Here, the system
and the proposed system is depicted in Fig. 2. After logging recommends exercises that are most frequently evalu-
in, the system checks whether the student has previously seen ated by the community with I ¼ 5 and C ¼ 1. This
an exercise. If they have, 10 exercises are recommended fol- assures that new users start with the most popular and
lowing a collaborative filtering approach. Otherwise, the sys- easiest exercises.
tem selects the easiest and most interesting exercises among To reduce the lack of exercises in the early stages of the
all available exercises. The system additionally builds a list of CARAMBA adoption we have manually created sets of exer-
10 exercises that are to be presented to students. cises based on interactions from previous years before intro-
The following steps are easy to understand by referring to ducing the recommender system. We first obtained the sets of
the diagram. However, it is important to note that after submit- exercises used before using the recommender system within
ting the exercise evaluation, the system applies a collaborative CARAMBA. Secondly, we asked students to evaluate each
filtering to provide a new pool of recommended exercises to exercise and then introduced these exercises in the system to
the student. Consequently, the student may face different exer- reduce the number of cold starts during the course. It is worth
cises according to their individual experiences. highlighting that the students who evaluated our initial

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
392 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

batch of exercises did not participate in the study presented in TABLE II

MAIN STEPS OF THE RECOMMENDER SYSTEM INCLUDED IN CARAMBA
Section IV-B.
2) Neighborhood Computation: A user-user collaborative
filtering approach is adopted here. As mentioned above, the
first step consists of finding the students who are the most sim-
ilar with respect to the active user k. To this end, the Cosine
Similarity function between two vectors [48] is considered.
These vectors contain evaluations made by two students with
common exercises, according to a certain criteria (e.g., taste
or complexity). This measure is then applied to each evalua-
tion criterion independently.
More formally, S ðtÞ and S ðcÞ are values of similarity com-
puted for the evaluation of criteria Taste (t) and Complexity
(c) respectively, and vk and vi are the evaluation vectors for
the exercises that the active student uk and the student ui have
in common. Thus, the corresponding similarities between stu-
dents uk and ui are computed as:

ðtÞ ðtÞ
vk  vi
S ðtÞ ðuk ; ui Þ ¼





ðtÞ

ðtÞ
(1)

vk

vi

and
ðcÞ ðcÞ
vk  vi
S ðcÞ ðuk ; ui Þ ¼





ðcÞ

ðcÞ
(2)

vk

vi
3) Building the List of Recommended Exercises: This is a
key stage in the usage of recommender systems. CARAMBA
relies on the Weighted Sum of Others Ratings [49] to calculate
S ðtÞ and S ðcÞ take values in the range [0,1]. A value closer to 1
a value (considering taste and complexity) for each exercise
means a high similarity between the students, while a value
ruk )
not evaluated by uk . Equation (5) shows this task, where (
closer to 0 means the opposite.
is the averaged value of uk for each problem solved and ( rui )
It is important to highlight that Equation (1) is not able to
is the averaged value for each similar user ui .
express the significance of the number of common exercises
with respect to all previouslt recorded exercises. Since Equa- P
tion (1) computes similarities using information from the exer- u2 ðrui ;e rui ÞSðuk ; ui Þ
ruk ;e ¼ ruk þ P (5)
u2 Sðuk ; ui Þ
cises that both the students have in common, it does not take
into account their corresponding records of exercises. We argue
that this information is crucial to sutably compute similarities Where:
between two given students. In this regard, Equation (3) was
ruk ;e : estimated value for the exercise e by the user uk ,
employed:

Sðuk ; ui Þ: similarity result 4

rui ;e : estimated value of the user ui for the exercise e
jHk \ Hi j
S ðpÞ ðuk ; ui Þ ¼ (3) and,
jHk j

: set of similar users to uk ðSðuk ;ui Þ > 0Þ.
where, Hk and Hi are the sets of exercises seen by students uk Later, the recommendation algorithm selects the top 10
and ui respectively. It is easy to note that Equation (3) quanti- exercises with more value and generates the recommendation
fies the significance of the number of exercises that the active list for uk .
user k has in common with user i by computing the percentage Table II summarizes the main steps of the recommendation
of common exercises regarding the record of the active user. process, which corresponds to Recommend in Fig. 2(b).
This expression is also defined in the range ½0; 1 and possible
B. Main Functionalities of CARAMBA
values have the same meaning as Equation (1).
Thus, we have three sources for computing the similarity for CARAMBA is a web application developed to manage the
every pair of students: S ðiÞ , S ðvÞ , and S ðpÞ . The next question is process of recommendation, solution, and assessment of exer-
how to aggregate them to obtain a single value portraying over- cises. This web application has an easy-to-use graphic user
all similarity. Several alternatives exist to deal with this. For interface (GUI) and the current version was developed using
instance, an average or weighted sum of the three similarity Python and the PostgreSql Server as a database system.
values could be used. Another approach is to multiply them: Among the main features of CARAMBA, one can mention
that:
Sðuk ; ui Þ ¼ S ðtÞ ðuk ; ui Þ  S ðcÞ ðuk ; ui Þ  S ðpÞ ðuk ; ui Þ (4)
It presents a user authentication area , where you can
enter previously registered students and teachers.
Notice that since S ðtÞ , S ðcÞ and S ðpÞ take values in [0, 1],
In the student interface, an informational panel (see left
then S will also take values in this range. panel Fig. 3) where students can manage their profiles

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 393

Fig. 3. CARAMBA showing the description of a selected exercise and its evaluation parameters.

and obtain a summary of the exercises carried out and and Industrial Engineering, as part of an exploratory study. Both
the assessments of those exercises these areas of study include the Fundamentals of Computer Pro-

The central area ((Fig. 3) was designed to show the gramming course in their curricula. These students used the CAR-
description of the selected exercise and its evaluation AMBA tool for autonomous learning for a period of 3 months.
parameters. After using CARAMBA, we gave the students a question-
naire asking for their opinion on nine assertions. Table III

The area to the right (Fig. 3) presents a list of recom-
shows these assertions according to three evaluation goals: RS
mended exercises, a list of all exercises, and a chronom-
performance, user-centric effects, and learning effects as sug-
eter that is activated with the start button and measures gested in [46]. Additional emphasis was put on evaluating
the solution time for the selected exercise. both learning and user-centric effects (thus, more assertions

The Scratch tool is activated in a pop-up window (Fig. 4) are included for assessing these goals).
through the ”Start” button (see central area in Fig. 4). Two assertions related to the accuracy and response time of

In the teacher profile teachers can add new exercises the system have been included (e.g., assertions A1 and A2). It
and obtain statistical information about the exercises is clear that both are less precise than those obtained from an
and the students. off-line experiment, since they were measured based on the
Finally, is important to mention that the RS was included in opinions of real users. However, we are aware that a greater
CARAMBA using Surprise,1 which is a Python Scikit (short for number of technical tests will be needed and this will be the
SciPy Toolkits) for building and evaluating recommender systems. subject of future works.
The assertion A1 links the recommender system with the stu-
dents programming skills refers to the fact that CARAMBA rec-
IV. EVALUATION
ommends exercises to the active user from the preferences of
The studies carried out in the Universidad Estatal de Milagro other users. However, this assertion was made under the assump-
assess the effects of the computational tools on learning. They tion that the active user will be comfortable with CARAMBAs
were organized in two stages. At the first stage, we developed a recommendations close to his/her current programming skills.
qualitative study based on satisfaction levels in using the tool; The nine exposed assertions were responded to using one of
this was applied in the 2015-2016 period. At the second stage these five options: strongly agree, agree, neither agree nor
(2016-2017), we conducted a quantitative analysis of the effect disagree, disagree, and strongly disagree.
on the development of a skill set and its influence on pass rates. The results of the questionnaires (Fig. 5) were organized
These studies are presented in detail below. into three groups: (a) Satisfaction of Computer Science stu-
dents, (b) Satisfaction of Industrial Engineering students, and
A. Motivation and Satisfaction Study (c) Overall satisfaction. The last is the aggregation of the first
two groups. A generally acceptable degree of satisfaction is
In this section, we will show the results of a study Real-Life appreciated. For instance, when more than 50% of the students
Testing [46] applied to 64 students studying Computer Science have at least agreed with all the assertions.
However, differences exist between the first two groups. As
1
http://surprise.readthedocs.io/en/stable/index.html expected, Computer Science students were more critical than

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
394 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

Fig. 4. Popup page of Scratch application inside CARAMBA.

the Industrial Engineering students (Fig. 5(a) and Fig. 5(b)). In The 88 students were randomly divided into three homoge-
both cases, a significant number of students did not agree with neous groups according to the learning strategy, as shown in
A1, indicating that more work had to be done regarding sys- Table V. The same teachers were attending each group. To per-
tem accuracy. A similar conclusion can be derived from the form a fair comparison among the three groups, we employed
system time response (A2). the same type and quantity (109) of exercises in all three. Fig. 6
As for the user-centric assertions (A3, A4, and A5), more shows the distribution of the exercises according to the concept
than 60% of the students from both courses at least agree and (Variable and Initialization, Cycles, Conditionals, and Concur-
40% strongly agree with assertion A5: that the proposed sys- rency). See that a similar distribution exists among the con-
tem is better. cepts, which provides enough diversity to the student. In order
Finally, regarding the assertions for evaluating the learning to avoid the Hawthorne effect [50], students and teachers were
effects (A6, A7, A8, and A9), a clear difference exists between not informed about their participation in the study.
both areas of study. For instance, about 60% of Computer Sci- The study was organized to answer the following research
ence students agree with those indicators, while 75% of Indus- questions:
trial Engineering students agree.
RQ1 (Concept Learning): To what extent will Scratch’s
In general, there was a suitable satisfaction level from the learning strategy and an exercise recommendation sys-
students of the proposed system (Fig. 5(c)). tem impact the students ability to learn programming
basics?
B. Effect on Learning
RQ2 (Final Performance): To what extent will
Motivated by the satisfactory conclusions obtained in the Scratch’s learning strategy and a exercise recommen-
previous section, we decided to carry out a quantitative study dation system impact the students’ final performance
to measure the effect of the CARAMBA tool on the concept in CS1?
of students. The experiment was carried out with 88 students For both questions (RQ1 and RQ2), we selected the learning
enrolled in the CS1 course, taught in the first semester of the strategy applied to each group as the independent variable (see
Computer Science Engineering program at Universidad Esta- Table V). The following is a brief explanation of what each
tal de Milagro. Before attending CS1, the students received 64 consists of:
hours of basic preparation in algorithms, corresponding to the
Traditional: The teacher applies traditional teaching
pre-university stage2. The Structure of Programming Funda- methods without using technology. The exercises used
mentals is shown in Table IV. in the practices and independent study are proposed and
controlled by the teacher.
2
An intensive preparation course applied by the Ecuadorian Ministry of
Scratch-only: The teacher uses the Scratch tool to intro-
Education to compensate for the steep learning curve upon entering higher
education. duce and practice programming concepts. The exercises

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 395

TABLE III TABLE IV

LIST OF ASSERTIONS EMPLOYED IN THE QUESTIONNAIRES COURSE SYLLABUS BY WEEK
FOR ASSESSING CARAMBA

1) RQ1 (Concept Learning): To provide answers to RQ1,

we focused on the first eight weeks of the CS1 course. To mea-
sure the level of each group, an initial test (pre-test) was applied,
taking into account four basic concepts: Variables, Cycles, Con-
ditionals and Concurrency. At the end of week eight another test
(post-test) was applied evaluating the same concepts.
The dependent variables for this RQ were the learning lev-
els achieved by the students in each of the concepts. These
levels were determined by ratings obtained in the post-test,
applied after eight weeks of the experiment.
A similar knowledge level was assessed in the groups at the
beginning of the study. Table VI shows the corresponding sta-
tistics for the groups in the pre-test. In these results, it can be
seen that the mean values achieved by the three groups were
low (on a scale of 0-10).
used in the independent study are proposed and con- Table VII presents the descriptive statistics of the varia-
trolled by the teacher. bles measuring the concept learning of the students (with-
out distinction of group), both before and after applying

CARAMBA: As with the previous strategy, the
the learning strategies. A previous analysis allowed us to
teacher uses the Scratch tool to introduce and prac-
determine:
tice concepts in the classroom, but independent study
the strength of the applied learning strategies. This is
is personalized by means of an exercise recommen- because the highest mean values are reached after
dation system. applying these strategies (in the post-test).
Notice that our main goal is to find out whether the person-

that the greatest learning was obtained in the concept
alized way of facing exercises provided by the recommender
system of CARAMBA has a significant impact on the learners. Variable and initialization, the difference between the
While traditional and Scratch-only approaches provided an pre-test and post-test was the largest.
organized path, where the exercises are revealed to students
Concurrency is the most difficult concept to learn,
corresponding to the order of topics, with CARAMBA the stu- because the values of the post-test were the lowest
dents are free to build their own paths, by choosing to select of all.
the recommended exercises. A preliminary reliability analysis across the Cronbach
Below, we will discuss the results obtained for these research alpha revealed acceptable internal concordance within the
questions. In all cases, the assessment scale of the exams was allowed ranges (0.77). Alternately, the consistency study
0-10, where 0 is the lowest grade and 10 is the highest. among evaluators from Kendall’s W [51] also determined

Fig. 5. Satisfaction of students with the system according to the questions.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
396 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

TABLE V TABLE VI
STRUCTURE OF CONSIDERED GROUPS DESCRIPTIVE STATISTICS OF PRE-TEST RESULTS FOR EACH GROUP

In order to determine if a learning strategy significantly influ-

enced the students’ cognitive development, we then applied an
ANCOVA test by selecting results obtained in the pre-test as
co-variables.
b) Results: For this study, four hypotheses were defined
(one for each concept studied) as shown below:

H1: Using the Scratch tool with an Integrated Exercise
Recommendation System as a learning strategy has a
significant impact on students identifying and initializ-
ing variables.

H2: Using the Scratch tool with an Integrated Exercise
Recommendation System as a learning strategy has a
significant impact on students identifying and using dif-
ferent types of cycles.

H3: Using the Scratch tool with an Integrated Exercise
Recommendation System as a learning strategy has a
significant impact on students identifying and using the
different types of conditionals.
Fig. 6. Distribution of the number of exercises according to the concept.
H4: Using the Scratch tool with an Integrated Exercise
Recommendation System as a learning strategy has a
significant impact on students identifying and using
a positive result (w = 0.685) with a significance level of
6.33E-80. Concurrency.
Note that we applied both reliability and concordance tests to When applying the ANCOVA test (Table IX), we found
the overall dataset used for the subsequent statistical analysis, that the results achieved by the learning strategies that were
that is, involving all the evaluations performed during pretest applied in each concept differed significantly ðSig: < 0:05Þ.
and post-test stages and the three groups (traditional, Scratch- Thus, we proceeded with post-hoc analysis to identify where
only, and Caramba). So, what we consider as scorers in our con- these differences existed.
cordance test (Kendall w) were the students, but not the teachers. The Dunnett test represents a post-hoc alternative using
a control treatment and comparing it with other treatments.
a) Data Analysis: In this study, we applied an experimental In our case, we selected the results achieved by the CAR-
design of pre-test/post-test to the corresponding groups. AMBA learning strategy as base of comparison (control
Table VIII shows a descriptive analysis of the post-test varia- group in Dunnett’s test) to compare it with the results of
bles depending on the learning strategy. It can be seen that the the other strategies.We selected this test because our
groups which used Scratch with the exercise recommendation hypotheses were enunciated to determine if the use of
system obtained the highest average values for each concept CARAMBA siginificantly improved the learning of pro-
as well as the lowest values of standard deviation. gramming languages.

TABLE VII
DESCRIPTIVE STATISTICS

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 397

TABLE VIII
DESCRIPTIVE STATISTICS FOR POST-TEST VARIABLES

TABLE IX
RESULTS OF THE ANCOVA TEST

TABLE X
RESULTS OF THE DUNNETT TEST FOR EACH CONCEPT

Table X summarizes the values obtained by this test, which 2) RQ2 (Final Performance): In this RQ2, we focused on
shows that: the students’ final grades, which were calculated by adding

The CARAMBA strategy achieved significantly higher together the three evaluations of the course: cumulative grade up
results than the traditional strategy ðSig: < 0:05Þ in all to week eight, cumulative grade up to week 16, and final exam
concepts. grade.

The CARAMBA strategy also achieved significantly Thus, the level of knowledge acquired in the subject by the
higher results than the Scratch strategy but only in student was considered as the dependent variable. The values
of this variable, as mentioned above, represented the final
concepts of variables and initialization and that of
grade of each student.
Cycles and Concurrency. In cases of conditional
concepts, the differences found were not significant a) Data Analysis: In this case, a comparative analysis
ðSig: > 0:05Þ. was applied between groups to identify whether the

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
398 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

TABLE XI TABLE XII

DESCRIPTIVE STATISTICS BY LEARNING STRATEGY RESULTS OF THE DUNNETT TEST FOR ACADEMIC PERFORMANCE

C. Analysis of CARAMBA Utilization

In this Section we present evidences supporting how
learning strategy had a significant impact on students assisted learning using CARAMBA helps students to signifi-
advancement. cantly improve their results. Concretely, we focus on utiliza-
Analyzing the descriptive statistics presented in tion, which is the level of practice of the students with the
Table XI, it can be seen that the group who used the CAR- exercises.
AMBA tool was the one with the highest average advance- Concretely, we will try to answer the following questions:
ment value (7.5083) and the lowest standard deviation
Q1: Did students actually use CARAMBA?
(0.92). This result shows that this was the only group
Q2: Did students practice by using CARAMBA recom-
whose general average exceeded the threshold allowed to mended exercises?
pass a subject (greater than or equal to 7).
Q3: How does doing sets of concept learning problems
We will then apply a one-factor ANOVA analysis to deter- organized in levels help students to pass the course of
mine if the differences between the means are significant. CS1?

Q4: How did CARAMBA help students to pass the
b) Results: In this study, the following research hypotheses course of CS1?
were defined: To answer Q1, we analyzed the number of exercises done

H5: Using the Scratch tool with an Integrated Exercise by students. Fig. 8 shows the average number of exercises
Recommendation System as a learning strategy has a done by the students in the 16-week course. We see that the
significant impact on the final class results. exercise rate per student-week remained above 4 throughout
The ANOVA test of an executed factor determined the exis- the course. Note that the tendency (continuous line) of this
tence of significant differences between the means of the 3 rate fluctuated across the course and had an error of approxi-
groups (p value = 8.838E-10). Furthermore, the Dunnett multi- mately one exercise.
ple comparison test (Table XII) showed that the control group The optimal rate is 6.81 which is calculated from the total
achieved significantly better results than other treatments in number of exercises available (109) per student in the 16
the study ðSig: < 0:05Þ. weeks, assuming that each student would ideally complete the
In another analysis, we calculated the pass rate, or per- same number of exercises per week. We observe that the values
centage of students who passed in the group that used the obtained by the students represented more than the 58%. This
CARAMBA tool, and compared it with the historic results is not only an affirmative answer to question 1, but also indi-
of the Fundamentals of Programming course. In Fig. 7, it cates that the level of practice with CARAMBA was important.
can be observed that: it was the first time that the pass To answer the second question we evaluate the students’
rate was over 52%, the value reached was 8% higher than interactions with the exercices. Fig. 9 shows the distribution
that of the 2014-2015 (2nd) period pass rate (which was of the students’ interactions with the exercises included in
the highest value prior to this study) and 21% higher than CARAMBA, taking into account the origin of the exercise
the program’s historical average (30.38%). (Pool of exercises and Recommender system). Please note

Fig. 7. Historical pass rate of UNEMI. The semester where the present study took place is denoted as 16–17(1st) in boldface.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 399

Fig. 8. Rate of exercises evaluated per student/week (points). The continu- Fig. 9. Distribution of the type of students’ interactions with CARAMBA
ous line correspond to the conditional mean using a Local Polynomial Regres- according to source: Pool of exercises (without recommendation) and Recom-
sion Fitting. The shaded area around the line correspond to the confidence mender system.
interval (level = 0.95).

here that an exercise can be selected by the student in two the student’s success/failure. For instance, one can see that
ways using CARAMBA: from the set of all available exercises excellent and average students are associated with high usage
(i.e. without recommendation) or from the set created by the levels. In the case of the failed students, the great dispersion
recommendation system. Firstly, Fig. 9, shows that most of between the levels of student utilization does not allow us to
the interactions (about 80%) of the students with the system identify this relationship.
were due to the interest in the exercises recommended by We have proceeded in a similar manner for Q4. Fig. 11
CARAMBA. Secondly, only 20% are interactions that came shows the same analysis but considers the overall utilization
out of non-recommended exercises. This means that the stu- level of each student. In this plot, the pattern of correlation
dents practiced with CARAMBA mostly through the recom- (positive) is much better appreciated.
mendation system. To support our perception that the level of use of CAR-
Questions Q3 and Q4 are closely related. Both questions AMBA has a positive/negative influence on the student’s
depend on the definition of the system’s level of use system. In success/failure in the subject, we have proceeded with a correla-
this case, we have considered that the use of CARAMBA by a tion test. Here, we have used the Pearson’s correlation test,
student can be defined as the percentage of exercises performed which was applied by using the level of use of the system by
from those totally available (regardless of origin: recom- each student and the corresponding grade as variables. The
mended or not). Specifically, for question Q3 this percentage results shown in the Table XIII consider each concept and all
was calculated on the basis of the number of exercises available exercises (overall). Please note here that the p-value of the test,
for each concept. Thus, each student would have five levels of the correlation coefficient, and the minimum and maximum val-
use associated for each concept (Variable and Initialization, ues of the confidence interval (for 95%) have been included.
Cycles, Conditionals, and Concurrency) and one overall (tak- From Table XIII it can be concluded that there is a positive
ing into account all available exercises). A higher value for the correlation between the level of use and the student’s grade.
level of use indicated that the students interacted with a lot of However, in the case of variable and initialization, this corre-
exercises, while a low value meant the opposite. lation is not significant (p  value > 0:05). Questions Q3
In Fig. 10, the level of use of the system (x-axis) and the and Q4 can therefore be answered in an affirmative way.
final grade of the students (y-axis) were summarized for each We can thus conclude that the primary cause of
concept. The grade is in the range of 0 to 10, where 0 is the CARAMBA’s success is the high level of practice achieved
worst possible grade and 10 the best. To facilitate the visual by the students with the tool. However, an unanswered ques-
comparison in this graph, we have identified each student tion is what causes students to exhibit such a high level of
with a category according to their grade: failed students practice with CARAMBA. In our opinion, this is mainly
(FAILED with a grade below 7), average students (Average caused since students are very motivated with CARAMBA.
with a grade between 7 and 8.5), and finally, outstanding stu- We have observed that today’s students, like most people in
dents (Excellent). Fig. 10 allows us to draw some important modern society, are linked to information and communication
conclusions. Initially, it can be seen that the students had technologies. Thus, we assume that by using a tool like CAR-
usage levels above 50% in all concepts. It is interesting to AMBA, which allows students to learn autonomously, stu-
see that there was one student who used all the exercises of dents would feel more motivated. Consequently, this
the concept in the Concurrency level case. However, the motivation would lead to a higher level of practice, which
most important conclusion here is that for at least three of would lead to a higher probability of success in the subject.
the concepts (Cycles, Conditionals, and Concurrency) there We recognize that such an assumption implies an in-depth for-
is a correlation between the level of use of the system and mal study that explains these and other issues.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
400 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

Fig. 10. System utilization vs. Score according to the Concept (Var. and Init., Cycles, Conditionals, and Concurrency). Each point represents a student of a cer-
tain group (see legend). The continuous line corresponds to the conditional mean using a Local Polynomial Regression Fitting, where the shaded area around the
line corresponds to the confidence interval (level = 0.95).

V. FUTURE WORK the order in which the topics are taught to improve
In this Section we present the future work related to this students’ learning. To this end, we will study content-
research. based recommendation systems. These systems recom-

Recommendation of controlled exercises: We will study mend items with similar characteristics to those already
whether building recommendations take into account evaluated by a user.

Scalability of the system: This tool is being adapted to
other study scenarios with more users and exercises. In
this sense, we are studying the incorporation of a collabo-
rative filtering system based on the [52] model. This type
of system reduces the size of the valuation matrix by
transforming it into characteristics that represent common
factors present in the original matrix and allow the system
to recognize patterns, which may be hidden in the data set.

CARAMBA Functionality: Incorporating other statisti-
cal details and improving the user interface.

TABLE XIII
PEARSON’S PRODUCT-MOMENT CORRELATION TEST FOR
SYSTEM UTILIZATION VS. SCORES

Fig. 11. Overall system utilization vs. Score. Each point represents a student
of a certain group (see legend). The continuous line corresponds to the condi-
tional mean using a Local Polynomial Regression Fitting, where the shaded
area around the line corresponds to the confidence interval (level = 0.95). *95% confidence interval

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 401

VI. CONCLUSION REFERENCES

In this study we present CARAMBA, an easy-to-use Web [1] I. Huet, O. Pacheco, J. Tavares, and G. Weir, “New challenges in teaching
introductory programming courses: A case study,” in Proc. 34th Annu.
Application involving Scratch alongside a recommender sys- Frontiers Educ., 2004, pp. 286–290.
tem for exercises. Our goal has been to enhance the learning of [2] H. C. Jiau, J. C. Chen, and K.-F. Ssu, “Enhancing self-motivation in
computer programming at the college level. The experimental learning programming using game-based simulation and metrics,” IEEE
study was developed in two different stages and periods: Trans. Educ., vol. 52, no. 4, pp. 555–562, Nov. 2009. [Online]. Avail-
able: http://ieeexplore.ieee.org/document/5164890/

In the first stage, a group of computer science and engi- [3] T. M. Connolly, M. Stansfield, and T. Hainey, “An application of games-
neering students used the application for independent based learning within software engineering,” Brit. J. Educ. Technol.,
study. These students were asked to assess nine indica- vol. 38, no. 3, pp. 416–428, 2007.
[4] D. Tsompanoudi, M. Satratzemi, and S. Xinogalos, “Evaluating the
tors regarding three goal areas: (1) recommender system effects of scripted distributed pair programming on student performance
performance; (2) user-centric effects; and (3) learning and participation,” IEEE Trans. Educ., vol. 59, no. 1, pp. 24–31, Feb.
effects. In general, a significant level of satisfaction 2016. [Online]. Available: http://ieeexplore.ieee.org/document/7089313/
[5] S. K. Andrianoff, D. B. Levine, S. K. Andrianoff, and D. B. Levine, “Role
among the students was observed. playing in an object-oriented world,” ACM SIGCSE Bulletin, vol. 34,

In the second stage, we conducted a quantitative no. 1, pp. 121–125, Mar. 2002. [Online]. Available: http://portal.acm.org/
study regarding concept learning and academic per- citation.cfm?doid=563517.563386
[6] J.-M. S
aez-L
opez, M. Rom
an-Gonz
alez, and E. V
azquez-Cano, “Visual
formance in CS1 students. For this, 3 learning meth- programming languages integrated across the curriculum in elementary
odologies were applied: the traditional method, school: A two year case study using scratch in five schools,” Comput.
Scratch only, and the CARAMBA application. The Educ., vol. 97, pp. 129–141, 2016. [Online]. Available: https://www.sci-
encedirect.com/science/article/pii/S0360131516300549
results are as follows: [7] S. Cooper, W. Dann, R. Pausch, S. Cooper, W. Dann, and R. Pausch,
– Concept learning: The study confirmed that Scratch “Teaching objects-first in introductory computer science,” in Proc. 34th
with an exercise recommendation system signifi- SIGCSE Tech. Symp. Comput. Sci. Educ., 2003, vol. 35, pp. 191–195.
[Online]. Available: http://portal.acm.org/citation.cfm?doid=611892.611966
cantly enhances the learning of basic programming [8] M. C. Carlisle, T. A. Wilson, J. W. Humphries, and S. M. Hadfield,
concepts. This tool provided a customized environ- “RAPTOR: Introducing programming to non-majors with flowcharts,”
ment for the study, which helped to develop auto- J. Comput. Sci. Coll., vol. 19, no. 4, pp. 52–60, 2004. [Online]. Avail-
able: http://dl.acm.org/citation.cfm?id=1050231.1050238
nomous learning amongst students based on their [9] C. J. Bouras, V. Poulopoulos, and V. Tsogkas, “Squeak etoys: Interactive
cognitive preferences. Statistical analysis allowed us and collaborative learning environments,” in Handbook of Research on
to validate the formulated hypotheses (H1, H2, H3 Social Interaction Technologies and Collaboration Software: Concepts
and Trends, T. Dumova and R. Fiordo, Eds. Hershey, PA, USA: IGI
and H4), showing that the effect of the proposal was Global, 2010, pp. 417–427. [Online]. Available: http://services.igi-global.
significantly superior than other applied strategies. com/resolvedoi/resolve.aspx?doi=10.4018/978- 1-60566-368-5.ch037
– Final performance: The performance study statisti- [10] D. J. Malan, H. H. Leitner, D. J. Malan, and H. H. Leitner, “Scratch for
budding computer scientists,” in Proc. 38th SIGCSE Tech. Symp. Comput.
cally corroborated with the H5 research hypothesis, Sci. Educ., 2007, vol. 39, pp. 223–227.
showing that the learning strategy significantly [11] J. Maloney, M. Resnick, and N. Rusk, “The scratch programming lan-
influences rates of student advancement. The results guage and environment,” ACM Trans. Comput. Educ., vol. 10, no. 4,
pp. 1–15, 2010.
showed that the group which used the Scratch tool [12] I. F. de Kereki, “Scratch: Applications in computer science 1,” in Proc.
with the exercise recommendation system obtained 38th Annu. Frontiers Educ. Conf., Oct. 2008, pp. T3B-7–T3B-11.
the highest scores among all experimental groups. [Online]. Available: http://ieeexplore.ieee.org/document/4720267/
[13] S. Mishra, S. Balan, S. Iyer, and S. Murthy, “Effect of a 2-week Scratch
– CARAMBA utilization: results show that there is a intervention in CS1 on learners with varying prior knowledge,” in Proc.
significant positive correlation between the high Conf. Innov. Technol. Comput. Sci. Educ., 2014, pp. 45–50. [Online].
level of practice with CARAMBA and the possibil- Available: http://doi.acm.org/10.1145/2591708.2591733
[14] U. Wolz et al., “Starting with scratch in CS 1,” in Proc. 40th ACM Tech.
ity of success in the subject. This is consistent with Symp. Comput. Sci. Educ., 2009, pp. 2–3. [Online]. Available: http://por-
the study on cognitive learning and performance, tal.acm.org/citation.cfm?doid=1508865.1508869
which points out that students feels motivated when [15] E. Tanrikulu and B. C. Schaefer, “The users who touched the ceiling
of scratch,” in Proc. World Conf. Educ. Technol. Res., H. Yalin,
using CARAMBA. F. Adiloglu, H. Boz, S. Karata, and F. Ozdaml, Eds., 2011, vol. 28,
Additionally, we observed that the pass rate of students pp. 764–769.
using CARAMBA surpassed the historic pass rate for the [16] J. C
ardenas-Cobo, P. Novoa-Hern
andez, A. Puris, and D. Benavides, Rec-
subject of Fundamentals of Programming. Concretely, the ommending Exercises in Scratch: An Integrated Approach for Enhancing
pass rate achieved by our proposal was over 52%, which is the Learning of Computer Programming. Cham, Switzerland: Springer,
2018, pp. 255–271.
8% higher than the rate achieved during a previous experi- [17] G. Fesakis and K. Serafeim, “Influence of the familiarization with
ence using only Scratch (without recommendation) and 21% “scratch” on future teachers’ opinions and attitudes about programming
higher than the historical results of tradition teaching (with- and ICT in education,” ACM SIGCSE Bull., vol. 41, no. 3, pp. 258–262,
out Scratch). 2009.
[18] S. Garner, “Learning to program from scratch,” in Proc. 9th IEEE Int.
Lastly, we expected that our findings would increase research- Conf. Adv. Learn. Technol., 2009, pp. 451–452.
ers interest in this topic. Extending Scratch with suitable ICT [19] L. A. Vaca-C
ardenas et al., “Coding with scratch: The design of an edu-
progresses to enhance programming learning in college students cational setting for elementary pre-service teachers,” in Proc. Int. Conf.
seems to be a promising alternative to traditional approaches of Interact. Collab. Learn., Sep. 2015, pp. 1171–1177. [Online]. Available:
teaching. We believe that todays students need technology- https://doi.org/10.1007/978-3-319-60937-9_20
[20] L. A. Vaca-C
ardenas et al., “An educational coding laboratory for elemen-
based learning strategies. Our future work will be oriented for tary pre-service teachers: A qualitative approach,” Int. J. Eng. Pedagogy,
developing and assessing these technological solutions. vol. 6, no. 1, Feb. 2016, Art. no. 11.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.
402 IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. 13, NO. 2, APRIL-JUNE 2020

[21] J. A. Mart
ın
ez-Vald
es, J. Ang
el V
elazqu
ez-Iturbid
e, and R. Hijon-N
eira, [45] D. Leony, H. Parada G
elvez, P. Mnoz-Merino, A. Pardo, and C. Kloos,
“A (relatively) unsatisfactory experience of use of Scratch in CS1,” in “A generic architecture for emotion-based recommender systems in cloud
Proc. ACM Int. Conf. Series, 2017, paper 8. learning environments,” J. Universal Comput. Sci., vol. 19, no. 14,
[22] J. Moreno-Le
on and G. Robles, “Dr. scratch: A web tool to automatically pp. 2075–2092, 2013.
evaluate scratch projects,” in Proc. Workshop Primary Secondary Com- [46] M. Erdt, A. Fern
andez, and C. Rensing, “Evaluating recommender sys-
put. Educ., 2015, pp. 132–133. tems for technology enhanced learning: A quantitative survey,” IEEE
[23] J. Moreno-Le
on, G. Robles, and M. Rom
an-Gonz
alez, “Towards data- Trans. Learn. Technol., vol. 8, no. 4, pp. 326–344, Oct.–Dec. 2015.
driven learning paths to develop computational thinking with scratch,” [47] J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl,
IEEE Trans. Emerg. Topics Comput., to be published, doi: 10.1109/ “Evaluating collaborative filtering recommender systems,” ACM Trans.
TETC.2017.2734818. Inf. Syst., vol. 22, no. 1, pp. 5–53, Jan. 2004.
[24] F. Ricci, L. Rokach, and B. Shapira, “Recommender systems handbook,” [48] M. Bramer, Introduction to Data Mining. London, U.K.: Springer, 2013,
in Introduction to Recommender Systems Handbook. Boston, MA, USA: pp. 1–8.
Springer, 2011, pp. 1–35. [49] X. Su and T. M. Khoshgoftaar, “A survey of collaborative filtering
[25] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutierrez, “Recommender techniques,” Adv. Artif. Intell., vol. 2009, Jan. 2009, Art. no. 421425.
systems survey,” Knowl.-Based Syst., vol. 46, pp. 109–132, 2013. [Online]. Available: http://dx.doi.org/10.1155/2009/421425
[26] J. Lu, D. Wu, M. Mao, W. Wang, and G. Zhang, “Recommender system [50] R. Spano, “Observer behavior as a potential source of reactivity:
application developments: A survey,” Decis. Support Syst., vol. 74, Describing and quantifying observer effects in a large-scale observa-
pp. 12–32, 2015. tional study of police,” Sociol. Methods Res., vol. 34, no. 4, pp. 521–553,
[27] C. Desrosiers and G. Karypis, “A comprehensive survey of neighborhood- 2006.
based recommendation methods,” in Recommender Systems Handbook. [51] J. Kendall, “A new measure of rank correlation.” Biometrika, vol. 30,
Boston, MA, USA: Springer, 2011, pp. 107–144. pp. 81–93, 1938.
[28] M. Elahi, F. Ricci, and N. Rubens, “A survey of active learning in col- [52] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-based collabora-
laborative filtering recommender systems,” Comput. Sci. Rev., vol. 20, tive filtering recommendation algorithms,” in Proc. 10th Int. Conf.
pp. 29–50, 2016. World Wide Web, 2001, pp. 285–295.
[29] N. Manouselis, H. Drachsler, R. Vuorikari, H. Hummel, and R. Koper,
Recommender Systems in Technology Enhanced Learning. Boston, MA,
USA: Springer, 2011, pp. 387–415. Jesennia C
ardenas received the B.S. degree in infor-
[30] N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, Survey and mation systems from the Escuela Politcnica del
Analysis of TEL Recommender Systems. New York, NY, USA: Springer, Litoral, Guayaquil, Ecuador, the Diploma in Higher
2013, pp. 37–61. Education by Competences from the Technical Uni-
[31] A. Klasnja-Mili
cevi
c, M. Ivanovi
c, and A. Nanopoulos, “Recommender versity of Ambato, Ambato, Ecuador, and the M.Sc.
systems in e-learning environments: A survey of the state-of-the-art and
degree in business administration from the Business
possible extensions,” Artif. Intell. Rev., vol. 44, no. 4, pp. 571–604, 2015.
Technological University of Guayaquil, Guayaquil,
[32] R. Farzan and P. Brusilovsky, Social Navigation Support in a Course Ecuador. She is currently working toward the Ph.D.
Recommendation System. Berlin, Germany: Springer, 2006, pp. 91–100. degree in software engineering with the University of
[33] M. Khribi, M. Jemni, and O. Nasraoui, “Automatic recommendations for Seville, Seville, Spain. She is a full-time Professor
e-learning personalization based on web usage mining techniques and and the Dean of the Faculty of Engineering Sciences,
information retrieval,” Educ. Technol. Soc., vol. 12, no. 4, pp. 30–42,
State University of Milagro (now UNEMI). She has more than 16 years of pro-
2009.
fessional experience in higher education. Her main research interests include
[34] X. Wan and T. Okamoto, “Utilizing learning process to improve recom- software products and artificial intelligence applied to engineering education.
mender system for group learning support,” Neural Comput. Appl., vol. 20,
no. 5, pp. 611–621, 2011.
[35] X. Wan, Q. Jamaliding, and T. Okamoto, “Analyzing learners’ relation-
ship to improve the quality of recommender system for group learning
support,” J. Comput., vol. 6, no. 2, pp. 254–262, 2011. Amilkar Puris received the B.S. degree in computer
[36] O. C. Santos and J. G. Boticario, “Requirements for semantic educa- science, and the M.Sc. and Ph.D. degrees from the Uni-
tional recommender systems in formal e-learning scenarios,” Algo- versidad Central de Las Villas, Santa Clara, Cuba,
rithms, vol. 4, no. 2, pp. 131–154, 2011. 2004, 2006, and 2010, respectively. He was an Associ-
[37] K. Ghauth and N. Abdullah, “The effect of incorporating good learners’ ate Professor and Researcher with the Universidad
ratings in e-learning content based recommender system,” Educ. Technol. Estatal de Milagro, Ecuador, from 2016 to 2018 and
Soc., vol. 14, no. 2, pp. 248–257, 2011. the Universidad T
ecnica Estatal de Quevedo, Ecuador,
[38] G. Lee, M. Salehi, and I. N. Kmalabadi, “International conference on in 2013. He has had considerable experience in teach-
future computer supported education, August 22- 23, 2012, fraser place ing computer programming and operation research at
central - Seoul a hybrid attribute based recommender system for e-learning the college level. His research interest involve meta-
material recommendation,” IERI Procedia, vol. 2, pp. 565–570, 2012. heuristics, data science, evolutionary computation, and
[39] P. Dwivedi and K. K. Bharadwaj, “Effective trust-aware E-learning rec- decision making in complex scenarios.
ommender system based on learning styles and knowledge levels,” Educ.
Technol. Soc., vol. 16, no. 4, pp. 201–216, 2013.
[40] A. S. Tewari, A. Saroj, and A. G. Barman, “E-learning recommender
system for teachers using opinion mining,” in Information Science and
Applications (Lecture Notes Elect. Eng.), vol. 339. Berlin, Germany: Pavel Novoa-Hern
andez received the B.S. degree in
Springer, 2015, pp. 1021–1029. computer science engineering from the University of
[41] A. Ruiz-Iniesta, G. Jim
enez-D
ıaz, and M. G
omez-Albarr
an, “A frame- Holguin, Holgun, Cuba, in 2007, and the Ph.D.
work for the rapid prototyping of knowledge-based recommender sys- degree in information and communication technolo-
tems in the learning domain,” J. Res. Pract. Inf. Technol., vol. 44, no. 2, gies from the University of Granada, Granada, Spain,
pp. 167–181, 2012. in 2013. He is currently with the Universidad T
ecnica
[42] J. Buder and C. Schwind, “Learning with personalized recommender Estatal de Quevedo, Quevedo, Ecuador, where he is
systems: A psychological view,” Comput. Human Behav., vol. 28, no. 1, an Associate Professor and Researcher, and a Lec-
pp. 207–216, 2012. turer in discrete mathematics and computer program-
[43] K. Peiris and R. C. Gallupe, “A conceptual framework for evolving, rec- ming. He also collaborates with the Universidad
ommender online learning systems,” Decis. Sci. J. Innov. Educ., vol. 10, Estatal de Milagro as a Guest Lecturer. His research
no. 3, pp. 389–412, 2012. interests involve soft computing, metaheuristics,
[44] A. R. Anaya, M. Luque, and T. Garc
ıa-Saiz, “Recommender system in dynamic optimization, and problem-solving in higher
collaborative learning environment using an influence diagram,” Expert education environments.
Syst. Appl., vol. 40, no. 18, pp. 7193–7202, 2013.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.


CARDENAS-COBO et al.: RECOMMENDER SYSTEMS AND SCRATCH: AN INTEGRATED APPROACH FOR ENHANCING COMPUTER PROGRAMMING... 403

Jos
e Angel Galindo received the Ph.D. degree with David Benavides received the B.S. degree in informa-
honors from the University of Seville, Seville, Spain, tion systems from the Institute Superieur d’Electronique
and the University of Rennes 1, Rennes, France, in de Paris, Paris, France, in 2000, and the M.Sc. degree in
March 2015. He has developed his professional activ- computer engineering and Ph.D. degree in software
ity in the United States, France, and Spain. His engineering from the University of Seville, Seville,
research areas include product lines software and the Spain, in 2001 and 2007, respectively. He has been
configuration of such products. He received the an Associate Professor with the University of
award for the best national thesis by SISTEDES. He Seville, Seville, Spain, since 2010. His main research
developed his postdoctoral research activity at interests include software product line and artificial
INRIA, France. He is currently a Juan de la Cierva intelligenceappliedtoengineeringeducation.
Researcher with the University of Seville, where he
continues his line of research on configuration, test-
ing, and the evolution of highly configurable systems.

Authorized licensed use limited to: Chung Yuan Christian University. Downloaded on November 25,2024 at 07:01:27 UTC from IEEE Xplore. Restrictions apply.