DOI: 10.1111/j.1467-8659.2010.01829.

x COMPUTER GRAPHICS forum

Volume 30 (2011), number 1 pp. 61–73

Experimental Feedback on Prog&Play: A Serious Game

for Programming Practice

M. Muratet1 , P. Torguet1 , F. Viallet2 and J.P. Jessel1

1 IRIT/VORTEX, Université Paul Sabatier, Toulouse, France

{mathieu.muratet, patrice.torguet, jean-pierre.jessel}@irit.fr

2 CREFI-T/DiDiST, Université Paul Sabatier, Toulouse, France

[email protected]

Abstract
This paper presents an experimental feedback on a serious game dedicated to strengthening programming skills.
This serious game, called Prog&Play, is built on an open source real-time strategy game. Its goal is to be compatible
with different students, teachers and institutions. We based its evaluation on an iterative process that allows to
implement the game and carry out experimentations in several contexts. Through this assessment, we define a
framework which has been tested by third parties and we analyse both positive and negative points to improve
the project. Evaluation is indeed beneficial and enables you to establish communication about the implemented
practices.
Keywords: serious game, video game, RTS, programming, Prog&Play
Categories and Subject Descriptors (according to ACM CCS): K.3.2 [Computer and Education]: Computer
and Information Science Education–Computer science education I.3.0 [Computer Graphics]: General—

1. Introduction U.S. movie market [Num] ($10 billion in 2008). Students cur-
rently in universities were born in the video games era. Thus,
In many countries, students are becoming less and less inter- those games are as much a part of their culture as TV, movies
ested in science. In computer science, for example, accord- or books. We think that the use of video games technologies
ing to Crenshaw et al. [CCMT08] and Kelleher [Kel06], the in a serious game context to practice programming could be
number of students has been shrinking. Moreover, ‘colleges a solution to attract and retain students in computer science.
and universities routinely report that 50% or more of those Currently, serious games exist in several fields such as edu-
students who initially choose computer science study soon cation, government, health, defence, industry, civil security
decide to abandon it’ [ACM05, p. 39]. Our university has and science. Let us first define a serious game.
been experiencing the same phenomenon with a decrease of
16.6% over the last 4 years in students studying computer For Zyda [Zyd05], a serious game is ‘a mental contest,
science. played with a computer in accordance with specific rules,
that uses entertainment to further government or corporate
To find a solution to this problem, we bet on serious games. training, education, health, public policy, and strategic com-
Since the first boom in video games in the 1980s, the gaming munication objectives’. Serious games use entertainment to
industry has held an important place in the world market. pursue different learning objectives. For example: ‘Darfur
According to the Entertainment Software Association fig- is dying’ [Dar] tries to raise public awareness; ‘Tactical
ures [Ent09], in 2008 the market of U.S. computer and video Language & Culture’ [Tac] aims to teach foreign languages
games amounted to $11.5 billion. This has overwhelmed the and cultures; ‘America’s Army’ [Ame] tries to recruit young


c 2010 The Authors
Computer Graphics Forum  c 2010 The Eurographics
Association and Blackwell Publishing Ltd. Published by
Blackwell Publishing, 9600 Garsington Road, Oxford OX4
2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
61
62 M. Muratet et al. / Experimental Feedback on Prog&Play

people for the U.S. Army. Serious games should be created games are Marvin’s Arena [Mar] using any .NET compatible
according to the needs and expectations of different sectors, language, Gun-Tactyx [Gun] using SMALL and Robot Battle
and within the available resources (physical and financial) [Robb] using a specific script language.
for their implementation.
Colobot [Col] is the only example we know of a com-
plete video game which mixes interactivity, storytelling and
1.1. Related work programming. In this game, the user must colonise a planet
using some robots that s/he is able to program in a specific
Learning programming is the basis of computer science train- object-oriented programming language similar to C++.
ing and an important topic in many scientific courses. How-
ever, programming fundamentals are hard to learn, especially
for novices. Several approaches exist to motivate students. 1.2. Overview
For example, Stevenson and Wagner [SW06] analyse assign-
ments from textbooks and historical usage to look for stu- Our approach consists in reusing existing games as the basis
dents’ problems and propose a ‘good programming assign- of a serious game and making it compatible with a maxi-
ment’ in computer science. These exercises could be com- mum of teaching contexts. There are many open source video
pleted by using block-based graphical languages like Scratch game projects available on the Internet. We think that reusing
[MBK*04], StarLogo [KY05] or Alice2 [KCC*02]. Indeed, them offers advantages in playing and robustness. To build
these novice-programming environments allow students to our serious game, we used an open source real-time strat-
forget syntax and directly experiment with programming. egy game. Building an efficient tool for a specific teaching
course is interesting, and it can often be reused in other con-
Another approach uses video games to hook the player texts. Our approach consists in working on a serious game
and bring him/her into programming. Two uses have been compatible with different students, teachers and institutions.
experimented: implementing new video games and play- This serious game is called Prog&Play and is introduced in
ing video games. For example, Chen and Cheng [CC07] Section 2.
asked students to implement in C++, through a collabora-
tive project, a small-to-medium scale interactive computer The evaluation of this type of tools in real contexts is a
game in one semester, using a game framework. Gestwicki complex task mainly because of the large number of un-
and Sun [GS08] based a case study on EEClone. This game controllable parameters. An iterative evaluation is proposed
is an arcade-style computer game implemented in Java: stu- (Section 3) and results are presented in the context of several
dents analyse various design patterns within EEClone, and experimentations.
from this experiment, learn how to apply design patterns in
their own game software. Leutenegger and Edgington [LE07]
used a ‘Game First’ approach to teach introductory program- 2. Prog&Play
ming. These authors believe that game programming moti-
vates most new students. They have used two-dimensional We consider Prog&Play as a serious game dedicated to pro-
game development as a unifying theme. gramming practice. Prog&Play is based on an open source
real-time strategy (RTS) game called Kernel Panic [Ker].
Another solution is to let students learn when they play Kernel Panic uses computer science metaphors, such as bits
a game. Wireless Intelligent agent Simulation Environment and pointers, as units (i.e. graphical objects which are con-
(WISE) [CHYH04] combines activities from virtual and trolled by the player). It is a simplified RTS with the follow-
physical versions of the Wumpus World game. It enables ing features: there is no resource management except for time
physically distributed agents to play an interactive game and space; all units are free to create; it has a small technol-
and provides a dynamic learning environment that can en- ogy improvement tree with fewer than 10 units; and it uses
hance a number of computer science courses: it can be used low-end vectorial graphics which match the universe. Owing
as a medium to demonstrate techniques in lectures; during to these characteristics, differences between two players are
classes, students can carry out practical work that tests, ex- about strategies and tactics used (and not about knowledge
pands, or modifies the simulator. The Wumpus World game of units features and relative advantages). Thus, the game is
can be played cooperatively or competitively. action-oriented while always remaining user friendly.
Robocode [Roba] is a Java programming game, where Kernel Panic takes place inside a computer where play-
the goal is to develop a robot battle tank to fight against ers command one of the three available factions: ‘Systems’,
other tanks programmed by other players. It is designed to ‘Hackers’ and ‘Networks’. Each of them offers units, such
help people learn Java programming. The robot battles are as Bits, Bytes and Assemblers for the System side, Virus,
running in real time and on-screen. It is suitable for all kinds Bugs and Worms for the Hacker side and Ports, Firewalls
of programmers from beginners (simple robot behaviour can and Packets for the Network side. Figure 1 shows the hier-
be written in just a few minutes) to experts (perfecting an archy of unit development for System factions. The Kernel
AI—Artificial Intelligence—can take months). Other such (which is the main unit of Systems) can build Bits, Bytes,

c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 63
Table 1: Specification of the Prog&Play API.

Operators Descriptions

Open Opens Prog&Play API

Refresh Loads the last state of the game
Close Shuts down and cleans up the Prog&Play
API
Changeover Indicates if the game is over or not
Map Size Returns the map size
StartPosition Returns the player starting position on the
map
NumSpecialAreas Returns the number of special areas on the
map
SpecialAreaPosition Returns the position of a specific special
area
Resource Returns the level of a specific resource
NumUnits Returns the number of visible units being a
member of a specific coalition
GetUnitAt Returns a specific unit which is a member
Figure 1: Units development hierarchy for system factions.
of the indicated coalition
GetUnitCoalition Returns the coalition of a specific unit
Pointers and Assemblers. The latter can build Sockets (which GetUnitType Returns the type of a specific unit
can solely build Bits) and Terminals. GetUnitPosition Returns the position of a specific unit
GetUnitHealth Returns the health of a specific unit
Starting with Kernel Panic, we designed an applicative GetUnitMaxHealth Returns the maximum health of a specific
programming interface that enables students to interact with unit
the game through programming (Section 2.1). The functional GetUnitGroup Returns the group of a specific unit
architecture of this library is detailed in Section 2.2. Follow- GetUnitNumPending Returns the number of pending commands
ing these technical details, we point out two solutions to map Commands of a specific unit
learning objectives into the game and we detail the one we GetUnitPending Returns a specific pending command
used during experimentations in Section 2.3. CommandAt which is associated to the indicated unit
SetUnitGroup Defines the group of a specific unit
SetUnitAction Defines an action for a specific unit
2.1. Applicative programming interface

In RTSes, a player gives orders to his/her units to carry out

operations (i.e. moving, building, and so forth). Typically,
API. With Kernel Panic these special areas are used to spec-
these instructions are given by clicking on a map with the
ify zones where factory building is allowed.
mouse. We modified the game to allow the player to give these
instructions through a program. Thus, students interact with The Prog&Play API is available in different programming
the game using the Prog&Play Applicative Programming languages: C (Appendix gives a program example), C++,
Interface (API). This API simplifies programming as much Java, OCaml, Ada, Scratch and an interpreted language called
as possible. It hides the game synchronization complexity and ‘Compalgo’ (used in a specific course at our university).
gives access to a subset of the game data. Table 1 details its Thus, the serious game is adaptable to teaching choices and
specification. Using this API, programs can load game data is usable in different teaching approaches (imperative, object-
such as unit features (such as number, position and type), map oriented, functional and graphical).
size, etc. Using these data, the player program can create a
set of commands and send it to the game. When the game
receives these commands, it executes them, modifying the 2.2. Functional architecture
game state.
From a technical point of view, communication between the
The specification detailed in Table 1 uses concepts called student program and the video game is carried out through a
coalition and special areas. Three coalitions are available shared memory. Figure 2 shows the functional architecture
to structure units: MY_COALITION representing units con- of the Prog&Play API which is an interface to manage this
trolled by the player, ALLY_COALITION representing units shared memory to simplify communication and synchroni-
controlled by allies of the player and ENEMY_COALITION sation between student programs and Kernel Panic. Two in-
representing units controlled by enemies of the player. Spe- terfaces are available: the ‘Client’ is used by the player to
cial areas allow transferring a set of positions through the code his/her programs and the ‘Supplier’ is integrated into


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
64 M. Muratet et al. / Experimental Feedback on Prog&Play

own AIs to use them in a multiplayer session (the student

motivation is maintained by competition between players).

Currently, we have only tested the first solution (i.e. the

campaign). As the original Kernel Panic was only a multi-
player game and did not provide campaigns, we had to build
one dedicated to our educational objectives. We have taken
advantage of the Kernel Panic universe and offered students
the following scenario: ‘For a certain number of years, a se-
cret war has been rife inside computers. Steady attacks have
been led against innocent victims. Today is your turn. Your
aggressor captured your mouse controller. You must recover
it. Your only solution: programming’. To achieve this objec-
tive, five missions were created. During the first, the player
Figure 2: Functional architecture of the Prog&Play API. controls only one Bit and has to move it to a specific posi-
tion to retrieve a lost Byte. In the second mission, the player
controls two units (one Bit and one Byte) and has to place
them in two different positions to find more units. Thus, in
the game engine. On student’s program demand, pertinent this mission, the conditional control structure is introduced
data of the game are copied into the shared memory. To to give a target position to each unit according to their types
avoid incoherent situations, the student’s program works on (Bit or Byte). In the third mission, the player controls a weak
this copy. The student’s program reads data and writes com- army which is made up of seven Bits and three Bytes. To
mands through the ‘Client’ interface. The shared memory strengthen his/her army, the player has to move it to meet up
is regularly checked by the game engine to carry out pend- with an Assembler. To do this, the iterative control structure
ing commands. With this solution, at any time, the player is introduced to loop through each unit and move them to-
can stop his/her program execution, modify it and execute wards the right position. In mission four, the player has to
it again. Thus, the program is able to connect to the shared use the Assembler capability to repair all the army. This is
memory without disturbing the game simulation. the most complex mission and it requires overlapping iter-
The ‘Supplier’ is responsible for shared memory manage- ative and conditional control structures. Finally, during the
ment. First, the ‘Supplier’ creates and initializes the shared last mission, the player launches an attack against opponents
memory so that it is available to the ‘Client’. After this ini- to retrieve the mouse controller.
tialization, the game enters the simulation loop and for each Thus, the serious game presented in this paper is a combi-
iteration consults the shared memory to know if an update nation of three entities (Figure 3): the first one is a standard
is required. If this is the case, the ‘Supplier’ computes the video game (Kernel Panic); the second one, the Prog&Play
update according to the current game state and copies this API, provides the programming facilities; the third one, the
update into the shared memory. Then, the ‘Supplier’ pro-
cesses pending commands defined by the ‘Client’. Finally,
when the game is over, the ‘Supplier’ frees and cleans up the
shared memory.
On the ‘Client’ side, calling API’s operators can be carried
out after a first call to the Open and Refresh operators. If
there is a problem, API’s operators return an error code or an
exception depending on the language used. When the ‘Client’
asks for a refresh, this call is blocking until the ‘Supplier’
has detected and carried out the update.

2.3. Mapping learning objectives into the game

In our previous paper [MTJV09], we identified two solutions

to map learning objectives into the game. First a campaign
can be used, divided in missions (equivalent to exercises),
to gradually introduce learning topics and enable students to
learn how to play and program AIs (student motivation is
maintained by the campaign story). Second skirmishes can
be used as a project approach where students program their Figure 3: Serious game composition.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 65
game mode (campaign or skirmish), transforms the program- iment, the research team has to define a theoretical intent and
ming game into a pedagogical artefact, that is a serious game. specify disciplinary ideas and forms of teaching which con-
As each entity can be changed as long as it interfaces with stitute the prospective goals or endpoints for student learn-
the two others, new versions of the serious game can be ing. The challenge is to formulate a design which embodies
built. For example, we can replace Kernel Panic with an- testable conjectures about both significant shifts in student
other game, create new campaigns adapted to object-oriented learning and the specific means of supporting those shifts.
programming teaching or work with different programming In our experiments, the theory we attempt to develop is the
languages. process of learning programming through serious games.

3. Evaluation 3.2. Iterative evaluation

The conception of an evaluation to assess the positive or The design experiment is based on an iterative process, which
negative impacts of our serious game on students is a com- allows the serious game and its evaluation to evolve. Each
plex task. This is due to the large number of uncontrollable iteration is made up of three stages: preparing for a design
parameters inherent to experiments in a real context. Never- experiment, conducting a design experiment and conducting
theless, an evaluation is necessary for several reasons. First, retrospective analysis. In the first stage, the theoretical intent
it allows defining a framework which will be tested by third has first to be clarified (what is the point of this iteration?
parties in different contexts. Secondly, results of evaluation What will be tested?). After this we need to select a course
show negative points and mistakes and will be analysed to to conduct experimentation corresponding to the objectives
improve pedagogical aspects of our serious game. Indeed, of this iteration. Then, according to the conjectured starting
evaluation is beneficial and allows communication about the points (the environment where Prog&Play will be used), el-
implemented practices. ements of trajectory (how the missions will be fulfilled) and
In Section 3.1, we define the methodology of design exper- prospective endpoints (the improvement of students’ pro-
iments that serves as theoretical framework. Then, we present gramming skills or motivation), an appropriate design exper-
in Section 3.2 an iterative implementation of this framework. iment has to be formulated either by us, the teachers or both.
Finally, we define evaluation criteria and we give experimen- Finally, to adequately document the learning ecology, we
tation results in Section 3.3. have to choose the evaluation criteria, associated indicators
and the way to support them. The second stage carries out
the experimentation and collects a maximum amount of data
3.1. Theoretical framework to ensure that retrospective analyses will result in rigourous,
empirically grounded claims and assertions. Finally, the third
We propose to study whether serious games could be useful stage analyses experimentation data to verify the theoretical
to teach programming, to attract and retain computer science intent and optionally proposes new experiments. Three iter-
students. The question is: Is it interesting to use a serious ations have so far been carried out with Prog&Play.
game to teach programming?
To achieve this goal, we propose to use the methodol-
ogy of design experiments [CCD*03]: ‘prototypically, de- 3.2.1. First iteration
sign experiments entail both “engineering” particular forms The first iteration’s objective is to observe student behaviour
of learning and systematically studying those forms of learn- in relation to the serious game to determine its motiva-
ing within the context defined by the means of supporting tional potential and influence on teacher activity. To carry
them. This designed context is subject to test and revision, out this first iteration, we have chosen to test the seri-
and the successive iterations that result play a role similar ous game in a familiar environment during some practical
to that of systematic variation in experiment’. The aim of work. The experimentation took place with first-year stu-
this methodology in educational research is to investigate dents learning computer science at an institute of technology
the possibilities for educational improvement by bringing in Toulouse (IUT A). Among 196 students, we chose 15 stu-
about new forms of learning to study them. Because de- dents from 40 volunteers. Students were novices: at the time
signs are typically test-beds for innovation, the nature of the of the experiment, they did not know any programming lan-
methodology is highly interventionist, involving a research guage, except ‘Compalgo’, an algorithmic language devel-
team, one or more teachers, at least one student and possibly oped by teachers of IUT A. During five sessions of an hour
school administrators. Design contexts are conceptualized as and a half each, students used Compalgo in the Windows
interacting systems and are implemented with a hypothesis environment.
ed learning process and the means of supporting it. Design
experiments are conducted in a limited number of settings To select these students, we designed a questionnaire to
and aim to develop a relatively humble theory which targets a evaluate their motivation to play video games and learn
domain specific learning process. To prepare a design exper- programming. We gave priority to students who were not


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
66 M. Muratet et al. / Experimental Feedback on Prog&Play

motivated by programming but very motivated by video The teaching team’s required condition to validate the
games. This questionnaire is based on the goals of Viau game’s integration into the curriculum concerns the ability
[Via97]; the value and success expectation model of Ban- of the serious game to preserve basic knowledge processed
dura [Ban97]; and the causal model of Pintrich and Schunk during standard teaching. Table 2 shows teaching objectives
[PS96]. Each model suggests specific indicators which we of standard practical work (PW) sessions.
adapted to students’ motivation in learning programming
and practising video games. We drew inspiration from the PW sessions one, three and four were modified. After
‘motivated strategies for learning’ questionnaire [PMB93]. validation by the teaching team, modifications are as follows:

For this experiment, we structured lessons in two phases.

In the first, students play the game in a multiplayer session • PW 1: In addition to the working environment presen-
without programming. Thus, they become familiar with the tation, this first session presents the game universe. An
game universe and units. The second phase is a presentation Appendix to the teaching documentation describes the
of the Prog&Play API which students learn to use through process of creating a multiplayer game. Students are in-
mission solving. After these two phases, students should be vited to test the game by themselves before the third
able to achieve small-scale programming compatible with session.
Kernel Panic.
• PW 3: The third session is centred on library use. The
teaching documentation has been completely rewritten to
present the Prog&Play API and the campaign. To match
3.2.2. Second iteration this with the original PW, a new mission (called 1a) has
been added to the campaign between the first and the sec-
The second iteration’s objective is to study the implemen- ond missions. Now, at the end of the first mission, the lost
tation of our serious game on a large scale to evaluate its Byte is not present at the indicated position. Thus, in this
influence on students’ results and teachers’ assessment. As a new mission, additional information is supplied to find
matter of fact, teachers involved in this experiment were not the Byte. The player controls only one Bit (the same as
members of our research team and had no previous experi- in the first mission) and knows the Byte’s direction (ex-
ence of serious games. To carry out this second iteration, we pressed in degrees, 0 being north) and relative distance.
introduced Prog&Play in the first semester of the computer The player has to compute the Byte position according
science core curriculum (bachelor’s degree—Paul Sabatier to the above information. During this session, students
University). In this course, computer programming is dealt have to complete missions 1, 1a, 2 and 3. Thus, students
with a functional approach. Students apply basic knowledge learn how to use the library with mission 1, how to carry
of this programming paradigm during six practical work ses- out a trigonometrical exercise on mission 1a and how to
sions of two hours each using the ‘OCaml’ programming handle pattern matching with mission 2 and recursion in
language on Linux. mission 3.
The class is made up of more than 300 students structured • PW 4: Finally, the teaching aid of the fourth session has
in practical work groups of about 15. Students’ distribution also been rewritten. Here, students work on the fourth
into those groups is achieved randomly. About 20 teachers mission. They start by creating functions to manage lists
work in this course. To evaluate the influence of the serious of units and use them to solve the mission simply. With
game on students’ results, half the groups were reference sorting algorithms, students code a second, more effi-
groups which followed the standard teaching whereas the cient, solution and glimpse the optimization principles
other half used the serious game. which was removed from PW 3.

Table 2: Teaching objectives of the practical work sessions (PW).

PW Objectives

1 Working environment presentation: operating system (Linux - Mandriva 2008), window manager (KDE), OCaml interactive
loop and word processor (Kate).
2 Arithmetical functions: pattern matching, n-tuple parameter, closure and recursion; introduction to dichotomy.
3 Use of the graphical library: definition of a library, trigonometrical arithmetic, optimisation and recursion.
4 Lists handling: list management and construction; pattern matching and recursion with lists; introduction to predicates;
sorting.
5 and 6 Data structures management: reuse of previous functions; projection, doublet removal, selection, update and sorting; user data
type manipulation.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 67

Owing to the number of teachers taking part in this ex-

periment, a teacher training session took place to present the
serious game, the teaching aids modifications and solutions
to the new exercises.

3.2.3. Third iteration

Prog&Play can be seen as an artefact. We have built a Website
where teachers can have access to the serious game and
adapt it to their own pedagogy by themselves. The third
iteration aims to test the usability of this artefact by third
parties.
The first experimentation has been carried out in collab-
oration with Jérémie Guiochet, teacher at the department of
electrical engineering, electronics and industrial informatics
at an institute of technology in Toulouse (IUT A). He chose to
use the game with students in difficulty. The teacher expected
this could motivate them to practice programming. Fifteen Figure 4: Synthesis of means used to collect data during
students used the C++ language to complete the campaign experimentations.
over nineteen hours.
The second experimentation has been carried out in rion allows obtaining a qualitative evaluation of the serious
collaboration with Elisabeth Delozanne and Françoise Le game by teachers.
Calvez from Pierre and Marie Curie university (Paris VI). Before presenting each criterion and associated results,
They have deployed the game in a transversal teaching we synthesise all the means we have used during experi-
unit based on information and communication technologies mentations to collect data (Figure 4). The latter operation is
in education. In this teaching unit, teachers propose a set organised in three stages: before, during and after the use of
of projects, one of which is based on our serious game. the serious game. For example, we massively used question-
Among volunteers, eight students who have followed in- naires instead of interviews to make data processing easier
troductory courses in imperative programming have been and reuse them for several iterations. Please note that we did
retained. Teachers supervised two sessions of two hours to not use all these solutions for each iteration.
introduce the game and programming concepts not studied
during lectures (library use and data structures). Following
this presentation, the first three missions have been com- 3.3.1. Criterion 1: enhancement of programming skills
pleted using a pure algorithmic language and then coded
again using the C language. Then, students had two months The learning evaluation is based on works of Chen and
to complete the campaign and prepare a presentation of their Cheng [CC07], Gestwicki and Sun [GS08] and Leutenegger
achievements. and Edgington [LE07]. We define three indicators: quan-
tity of work achieved, acquired knowledge and students’
future courses. Quantity of work achieved is evaluated us-
3.3. Evaluation criteria and results ing the game progress (number of missions completed). Ac-
quired knowledge is evaluated with students’ results of mid-
To define the evaluation criteria, we summarise objectives semester and final examinations. These examinations are de-
revealed from iterations: determine the motivational potential signed by teachers external to the research team and are the
of the serious game; evaluate its influence on students’ results same for all students.
and retrieve teachers’ views.
During the first iteration, we analyse the first indicator
From these objectives, we define four evaluation criteria: (quantity of work achieved) by processing data produced by
enhancement of programming skills; system usability; en- students during experiments. We count the number of mis-
tertainment; and teachers’ assessment. The first and third sions completed by each student and for each mission the
criteria evaluate the “serious game” concept. Indeed, it is number of compiling instances and executions necessary to
fundamental to check the “serious” side (enhancement of perfect their solution. These data have been collected thanks
programming skills) and the “game” side (entertainment) of to a software spy integrated into the development environ-
Prog&Play. The second criterion allows identifying bottle- ment used during sessions. With these data, we can define a
necks which could hinder students (installation problems, difficulty ratio (noted ‘d m ’) for each mission ‘m’ shown in
hardware requirements, software bugs, etc.). The last crite- Figure 5. This difficulty ratio is computed as indicated in


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
68 M. Muratet et al. / Experimental Feedback on Prog&Play

Figure 5: Difficulty ratio for each mission.

formula 1, where ‘nbStudents m ’ is the number of stu-

dents who have completed mission ‘m’, ‘nbCompim ’ and Figure 6: Average marks for each examination by groups
‘nbExecim ’ are, respectively, the number of compiling in- (20 being the top mark).
stances and the number of executions required for the student
‘i’ to complete mission ‘m’.
lable parameters. Nevertheless, we consider that the serious
 m
nbStudents
game is partly responsible for this positive result.
dm = nbCompim + nbExecim
i=1 Concerning the ‘students’ future courses’ indicator, we
nbStudentsm (1)
used students’ choices for their second semester. Indeed,
after the first semester, nine major courses are available, one
Contrary to what we expected, the difficulty caused by the of which is dedicated to computer science. Among students
various missions is not progressive. There is an important dif- from the serious game groups 55% have chosen this major
ference in difficulty between the first three missions and the course compared to 49% in the reference groups.
fourth. For future experiments, we plan to split mission 4 into
two new missions to propose a more progressive challenge. Thus, these results show that even if missions’ difficulty is
The players getting familiar with the API seems to lessen the less progressive than it was expected, the campaign supplies
difficulty between missions 1 and 3. Finally, we will update a sufficient difficulty to encourage work and stimulate stu-
mission 5 to offer a better challenge to end the campaign. dents’ interest. From a statistical point of view, we note that
Indeed several students who reached this mission have been the serious game has contributed to reduce students’ failure
frustrated by its simplicity. Johnson et al. [JVM05] argue compared to reference groups and to recruit more students
that story maintains user interest, encourages the player to for computer science for the second semester.
progress constantly and links actions and future objectives.
Challenges are introduced with the story and must match
players’ experience. Greitzer et al. [GKH07] highlight the 3.3.2. Criterion 2: system usability
difficulty to build a suitable scenario which is neither too System usability is based on a post-questionnaire. We ask
easy nor too difficult to submit a challenge without discour- students to evaluate the serious game. Using works of Siang
aging the player. and Rao [SR03], we define questions to evaluate the hier-
The other indicators (acquired knowledge and students’ fu- archy of the players’ needs to identify which part of our
ture courses) have been evaluated during the experimentation serious game needs improvement. This hierarchy is divided
of the second iteration. Concerning the ‘acquired knowledge’ into seven levels and lower levels are to be fulfilled before
indicator, we have collected students’ results of mid-semester moving on to the higher levels in the pyramid. The seven
and final examinations. Figure 6 shows the average marks for levels by priority order are as follows: Rules need, play-
reference and serious game groups. Please note that we did ers seek information to understand the basic rules of the
not have those marks when we started experimentation. game; Safety need, players need guidance about the game
software; Belongingness need, players need to become fa-
First, we notice a decline in marks between mid-semester miliar with the game to feel capable to achieve objectives;
and final examinations whatever group we take. However, Esteem need, players need to be motivated by the game (feed-
this decline is less important for the serious game groups backs, scores, competition, etc.); Need to know and under-
(1.25 points) than reference groups (1.55 points). We cannot stand, players need to understand and know more informa-
link this result only to the serious game because experimenta- tion about the game (e.g. different strategies or hidden items)
tions have been carried out with a large number of uncontrol- to reuse it during play time; Aesthetic need, players need good


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 69

• Q1: In your opinion, is it interesting to use a video game Table 3: Average and median answers of questions presented Fig-
to learn how to program? (1=“not at all”, 7=“essential”) ure 7 for the two first iterations (highest possible value being 7).
• Q2: Do you think that practical work sessions based on
a video game are adapted to this course? (1=“not at all”, 1st iteration 2nd iteration
7=“perfectly adapted”)
Q1 Q2 Q1 Q2

Figure 7: Opinion-related questions in the post- Average 6.17 6 5.05 5.18

questionnaire. Median 6.83 7 5 5

• Q3: Do you appreciate the campaign story (missions)?

(1=“not at all”, 7=“a lot”)
• Q4: Do you think that using programming in Kernel Panic
increases entertainment?
Reduces entertainment Do not change enter-
tainment
Enhances entertainment I don’t know

Figure 9: Questions for the entertainment post-

questionnaire.

The main criticisms are levelled at the teaching aids that are
considered too descriptive and over-prescriptive. We suggest
Figure 8: Students’ satisfaction for the hierarchy of the play- that this feeling has counterbalanced the game understanding.
ers’ needs. Thus, this experiment highlights that results obtained depend
not only on the serious game but also on the implementation,
supervision, teaching aids, etc.
graphics and visual effects, appropriate music, sound effects,
etc; Self actualization need players need to be able to transfer Nevertheless, whatever iteration, students have appreci-
their creativity and imagination into the game as long as it ated the initiative and are interested in this kind of pedagogy.
conforms to the game rules. The average and median answers to the two additional ques-
tions are presented in Table 3 for the first two iterations.
In addition, we ask students to express their criticisms,
remarks and suggestions about the serious game as well as In conclusion about this criterion, the game is functional
their point of view about the interest of a serious game to because no critical bugs were revealed during experimenta-
learn computer programming. Questions are presented in tions. The only issue is that some teachers have encountered
Figure 7. difficulties to configure their operating system to use the seri-
ous game correctly. The analysis of the hierarchy of players’
During the first two iterations, we evaluate the hierarchy needs shows that students’ satisfaction heavily depends on
of players’ needs. Figure 8 shows students’ satisfaction for implementation. This confirms the importance of supervision
each level of this hierarchy. for our serious game.
Concerning the first iteration, the lowest satisfaction level
concerns the aesthetic need with an average value of 47%. 3.3.3. Criterion 3: entertainment
The low-end vectorial graphics of Kernel Panic seem disturb-
ing to our students. Nevertheless, this simplified RTS allows This criterion has been evaluated with a questionnaire dis-
a quick learning curve which takes part in the positive satis- tributed to students at the end of experiments. Questions are
faction of lower needs. Therefore, we still think that Kernel presented in Figure 9.
Panic is not a bad choice to support Prog&Play.
The fourth question (Q4, Figure 9) was crucial for us.
For the second iteration, we observe lower satisfaction Introducing algorithmic concepts into the game should not
rates although the serious game is the same except for the new make it any less entertaining. This requirement is due to
mission (1a). To explain differences with the first iteration, our vision of serious games which have to be, above all,
we analyse remarks and criticisms expressed by students. entertaining.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
70 M. Muratet et al. / Experimental Feedback on Prog&Play

Concerning the first iteration, the average and median an- Table 4: Time spent by the teacher on each activity during one
swers to the third question (Q3, Figure 9) are, respectively, session.
5.85 and 6. These results show that students liked playing
the campaign. We think that missions’ interest is partly due Activity Time spent
to the first phase of the experimentation. The multiplayer
session allows students to become familiar with the universe Game usability 22 m 13 s
of the game and makes students’ immersion into the story Teaching contents 1 h 3 m 42 s
easier. Other tasks 41 m 27 s

For the fourth question (Q4, Figure 9), 100% of students

found that using programming in the game enhances en-
tertainment. We found out during this first experiment that
the game is fun and rewarding: it is fun because all stu- difficulties with the serious game and what they would advise
dents found that programming enhances entertainment; and regarding a possible renewal of the experiment.
it is rewarding because even though we initially planned, for The observation of the teacher’s activities has been car-
the final session, to let students play a normal multiplayer ried out during the second practical work session of the first
game, more than half of the students preferred to continue iteration. The time spent by the teacher on each activity is de-
programming. tailed in Table 4 (please note that some activities are counted
Concerning the second iteration, results are slightly lower. in several categories, which explains why the amount of time
The average and median answers to the third question (Q3, spent is higher than the duration of a session).
Figure 9) are, respectively, 4.14 and 4. This is the same for the As can be seen in the table, the longest activity deals
answers to the fourth question (Q4, Figure 9). Fifteenth per- with ‘teaching contents’. This distribution is appropriate, be-
cent of students found that the introduction of programming cause, the game’s explanations should not excessively reduce
into the game reduces entertainment, 22% indicate there is teaching activities. Moreover, the time spent on game-related
no change, 42% identify an enhancement and 21% do not activities diminishes when students become expert in game
give their opinions. We attribute these results to the same handling. Nevertheless, ‘game usability’ is an important ac-
reason identified in criterion 2 analysis: the teaching aids tivity mainly for the first sessions. Indeed, the second level
were perceived as too descriptive and over-prescriptive, they of the hierarchy of players’ needs (safety need) has been re-
have counterbalanced the entertainment. solved, in a large part, by teachers. Currently, Prog&Play is
Thus, the campaign is entertaining and it is a real success a serious game for programming practice; information about
from this point of view. Moreover, students appreciate using programming, learning contents and game usability are not
their programming knowledge in a real context. included in the game and must be added through teachers’
speeches or teaching aids.

Concerning the second iteration, we take advantage of the

3.3.4. Criterion 4: teachers’ assessment important number of teachers by asking them about their
To observe teachers’ activities during experiments, we pro- perception of video games and serious games before the ex-
pose to film sessions. Particular emphasis has been laid on perimentation. Three women and 12 men answered our ques-
activities dedicated to game usability and teaching contents. tions. Nine people said they played video games. Teachers
The activity called ‘game usability’ concerns explanations have mainly pointed out three qualities of such games: they
on the game’s control (how to launch the game? How to encourage thinking through strategic developments, they are
move the camera?) and on the game’s interaction with stu- entertaining and allow immersion into virtual worlds. To a
dents’ programs (how to select or command units? How to lesser extent, competition with multiplayer sessions, story-
check if programs are correct?). The activity called ‘teaching telling and creativity are identified as intrinsic qualities of
contents’ deals with explanations on programming obsta- video games. The main drawback is addiction. Some teach-
cles like variables (types, assignments, records), functions ers find video games too complex and associate them with
(call, argument passing) and control structure (conditional, a waste of time. Concerning serious games, seven teachers
iterative). The activity called ‘other tasks’ concerns adminis- who have a positive perception think that they can moti-
trative tasks (e.g. welcoming students, roll call), explanations vate students to practice computer science activities. More-
on operating system usability, etc. over, they consider serious games as concrete tools close to
students’ interests. The two teachers who have a negative
Furthermore, we submit a questionnaire to teachers about perception fear that serious games misrepresent computer
their perception of video games and serious games before ex- science as only made of video games, give an advantage to
perimentation. Then, each teacher has to file a report, which boys, are a waste of time (i.e. explaining the game instead of
enables us to collect their points of view about the past experi- the teaching content) and that playing distracts students from
mentation. On this occasion, we ask them if they encountered basic knowledge acquisition. The six teachers who had no


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 71

opinion said they would consider results before expressing campaign story which has a twofold objective. First, it moti-
their convictions. vates the player by making him/her protagonist in the story
development. Secondly, it gradually introduces the pedagog-
Again from the second iteration, teachers’ reports are ical contents of the serious game. At the end of the story,
mixed. On the performance side, they thought that sessions students master the programming interface and are able to
based on the serious game have been counter-productive for program their own AIs.
several students. They do not attribute this consequence to
the serious game but to the teaching aids. Like students, they For pedagogical reasons, we made the Prog&Play API
felt this documentation was too descriptive. On the plus side, available in different programming languages. Specifically,
most teachers have noticed a positive influence of the serious Scratch compatibility opens up new research vistas. Indeed, it
game on students. will be interesting to study the impact of these two technolo-
gies (block-based graphical languages and serious games)
Concerning the third iteration, teachers, who have carried on students’ motivation. Another future project consists in
out experimentations by themselves, give positive reports. adapting the serious game to other RTSes. Indeed, with the
First, Mr Guiochet said that the serious game campaign was quick evolution of video games engines, its integration into
well adapted to his students (they needed basic program- new RTSes is essential to continue entertaining students. It
ming practice and had a lot of time for this course). Indeed, would also be interesting to evaluate this approach with an-
students have shown their interest through some personal other video game genre and to compare it with our RTS-based
work which was not required. Secondly, Mrs Delozanne and serious game.
Mrs Le Calvez have been impressed by the wide range of
strategies planned and the quality of presentations prepared The Prog&Play system with Kernel Panic and com-
by students. They report that students have appreciated the patible interfaces are downloadable at http://www.irit.fr/∼
projects because they have programmed something concrete. Mathieu.Muratet/progAndPlay_en.php. If you are interested
Mrs Delozanne and Mrs Le Calvez added that their sessions in our serious game, please do not hesitate to get in touch
preparation time was quite long due to their inexperience with us!
of video games. In both instances, teachers plan to use our
serious game again next year.
In conclusion about teachers’ assessment, we can say that Acknowledgements
globally they are positive about the influence of the serious
game on students’ work. Indeed, in the second iteration the This work would not have been possible without the col-
negative evaluations are mainly attributed to the teaching laboration of several learning institutions. The authors thank
aids and not to Prog&Play. This is very encouraging for the following people and institutions: Christian Percebois
us. and Max Chevalier (department of computer science) and
Jérémie Guiochet and André Lozes (department of electrical
engineering, electronics and industrial informatics) at IUT A
4. Conclusion and future work in Toulouse; Mathias Paulin, Véronique Gaildrat and all Paul
Sabatier university teachers who took part in our experimen-
This paper describes Prog&Play, a serious game aiming to en- tation; and Elisabeth Delozanne and Françoise Le Calvez at
courage students to persevere in computer science. Through Pierre and Marie Curie university in Paris.
a theoretical framework, we designed an iterative evaluation
based on several experiments. During the first iteration, we
References
observed the serious game used by students in a real con-
text. On this occasion, we identified that Prog&Play is really [ACM05] ACM/IEEE-CURRICULUM 2005 TASK FORCE (Ed.):
appreciated by students. Following this first iteration, the Computing Curricula 2005, The Overview Report. IEEE
portability of the game was questioned. Thus, a second iter- Computer Society Press and ACM Press, New York,
ation was designed to study the game’s implementation on 2005.
a large scale. Despite poor teaching aids, the serious game
has helped reduce students’ failure compared to reference [Ame] AMERICA’S ARMY: http://www.americasarmy.com/.
groups. Then, the last iteration was meant to let teachers im- Accessed 26 August 2010.
plement the game by themselves in their courses. Reports
of these teachers are positive because they plan to continue [Ban97] BANDURA A.: Self-efficacy: The Exercise of Control.
using Prog&Play. Worth Publishers, New York, 1997.

Thus, the serious game is functional and motivates stu- [CC07] CHEN W.-K., CHENG Y. C.: Teaching object-
dents. We attribute this success to the original approach of oriented programming laboratory with computer game
Prog&Play which allows the use of programs to interact with programming. IEEE Transactions on Education, 50, 3
a RTS. The second element taking part in this success is the (Aug. 2007), 197–203.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
72 M. Muratet et al. / Experimental Feedback on Prog&Play

[CCD*03] COBB P., CONFREY J., DISESSA A., LEHRER R., [KY05] KLOPFER E., YOON S.: Developing games and sim-
SCHAUBLE L.: Design experiments in educational research. ulations for today and tomorrow’s tech savvy youth.
Educational Researcher, 32, 1 (Jan. 2003), 9–13. TechTrends: Linking Research and Practice to Improve
Learning, 49, 3 (2005), 33–41.
[CCMT08] CRENSHAW T. L., CHAMBERS E. W., METCALF H.,
THAKKAR U.: A case study of retention practices at the [LE07] LEUTENEGGER S., EDGINGTON J.: A games first ap-
university of illinois at urbana-champaign. In Proceedings proach to teaching introductory programming. SIGCSE
of the 39th ACM Technical Symposium on Computer Sci- ’07: Proceedings of the 38th SIGCSE Technical Sympo-
ence Education (New York, NY, USA, 2008), ACM Press, sium on Computer Science Education, 39, 1 (Mar. 2007),
pp. 412–416. 115–118.

[CHYH04] COOK D. J., HUBER M., YERRABALLI R., HOLDER L. [Mar] MARVIN’S ARENA: http://www.marvinsarena.com/.
B.: Enhancing computer science education with a wireless Accessed 26 August 2010.
intelligent simulation environment. Journal of Computing
in Higher Education, 16, 1 (2004), 106–127. [MBK*04] MALONEY J., BURD L., KAFAI Y., RUSK N.,
SILVERMAN B., RESNICK M.: Scratch: A sneak preview. In
[Col] COLOBOT: http://www.ceebot.com/colobot/index-e. Proceedings of the 2nd International Conference on Cre-
php. Accessed 26 August 2010. ating Connecting, and Collaborating through Comput-
ing (Keihanna-Plaza, Kyoto, Japan, Jan. 2004), pp. 104–
[Dar] DARFUR IS DYING: http://www.darfurisdying.com/. Ac- 109.
cessed 26 August 2010.
[MTJV09] MURATET M., TORGUET P., JESSEL J.-P., VIALLET
[Ent09] ENTERTAINMENT SOFTWARE ASSOCIATION: Essential F.: Towards a serious game to help students learn com-
facts about the computer and video game industry. http:// puter programming. International Journal of Comput-
www.theesa.com/facts/pdfs/ESA_EF_2009.pdf (2009). erised Games Technology 2009 (2009), 1–12.
Accessed on 30 September 2010.
[Num] THE NUMBERS: http://www.the-numbers.com/mark
[GKH07] GREITZER F. L., KUCHAR O. A., HUSTON K.: Cog- et/. Accessed 26 August 2010.
nitive science implications for enhancing training effec-
tiveness in a serious gaming context. Journal Education [PMB93] PINTRICH P., MARX R. W., BOYLE R. A.: Beyond
Resource Computation, 7, 3 (2007), Article No. 2. cold conceptual change: The role of motivational beliefs
and classroom contextual factors in the process of contex-
[GS08] GESTWICKI P., SUN F.-S.: Teaching design patterns tual change. Educational Research, 630, 2 (1993), 167–
through computer game development. ACM Journal on 199.
Educational Resources in Computing, 8, 1 (2008), 1–22.
[PS96] PINTRICH P. R., SCHUNK D. H.: Motivation in Educa-
[Gun] GUN-TACTYX: http://apocalyx.sourceforge.net/gun tion: Theory, Research and Applications. Prentice Hall,
tactyx/. Accessed 26 August 2010. Englewood Cliffs, 1996.
[JVM05] JOHNSON W. L., VILHJALMSSON H., MARSELLA S.: [Roba] ROBOCODE: http://robocode.sourceforge.net/. Ac-
Serious games for language learning: How much game, cessed 26 August 2010.
how much AI?. In Proceeding of the 2005 conference
on Artificial Intelligence in Education (Amsterdam, The [Robb] ROBOT BATTLE: http://www.robotbattle.com/. Ac-
Netherlands, The Netherlands, 2005), IOS Press, pp. 306– cessed 26 August 2010.
313.
[SR03] SIANG A., RAO R. K.: Theories of learning: A com-
[KCC*02] KELLEHER C., COSGROVE D., CULYBA D., FORLINES puter game perspective. In Multimedia Software Engineer-
C., PRATT J., PAUSCH R.: Alice2: Programming without ing, 2003. Proceedings of the Fifth International Sympo-
syntax errors. In Proceedings of the 15th Annual Sym- sium (Dec. 2003), pp. 239–245.
posium on the User Interface Software and Technology
(Paris, France, Oct. 2002). [SW06] STEVENSON D. E., WAGNER P. J.: Developing real-
world programming assignments for cs1. In ITICSE ’06:
[Kel06] KELLEHER C.: Alice and the sims: the story from Proceedings of the 11th Annual SIGCSE Conference on
the alice side of the fence. In The Annual Serious Games Innovation and Technology in Computer science Educa-
Summit DC Washington (DC, USA, Oct. 30–31, 2006). tion (Bologna, Italy, June 2006), pp. 158–162.

[Ker] KERNEL PANIC: http://springrts.com/wiki/Kernel_Panic. [Tac] TACTICAL LANGUAGE AND CULTURE: http://www.tactical
Accessed 26 August 2010. language.com/. Accessed 26 August 2010.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.
 M. Muratet et al. / Experimental Feedback on Prog&Play 73

[Via97] VIAU R.: La motivation en contexte scolaire. De

Boeck, Bruxelles, 1997. (in French).
[Zyd05] ZYDA M.: From visual simulation to virtual reality
to games. IEEE Computer, 38, 9 (2005), 25–32.

Appendix: Example solution

We present in Figure 10 a solution (in C) to the following sce-
nario: ‘You need to find an allied unit near to the position of
your unit. Tracks indicate that it has moved away at a distance
of 1060 units and an orientation of 209 degrees’. In this con-
text, we have used the native Prog&Play API. In ‘C’, a unit is
an abstract type that can be consulted with a set of functions.
For example, the function PP_Unit_GetPosition(u) allows
getting the position of a unit. With the assistance of teachers,
students have to find in the documentation which function is
useful to complete this exercise. Figure 10: A solution written in C.


c 2010 The Authors
Computer Graphics Forum 
c 2010 The Eurographics Association and Blackwell Publishing Ltd.