Paee Ale 2022 Proceedings

TITLE
International Symposium on Project Approaches in Engineering Education

Volume 12 (2022) ISSN 2183-1378
Proceedings of the PAEE/ALE’2022, International Conference on Active Learning in Engineering Education

14th International Symposium on Project Approaches in Engineering Education (PAEE)
19th Active Learning in Engineering Education Workshop (ALE)
Alicante - Spain, 06-08 July 2022
Editors
Rui M. Lima, Montse Farreras, Miguel Romá, Valquíria Villas-Boas
Publisher
Department of Production and Systems – PAEE association
School of Engineering of University of Minho, Campus de Azurém, 4800-058 Guimarães, Portugal
Graphic Design
Local Organizing Committee
ISSN
2183-1378
PAEE/ALE’2022, International Conference on Active Learning in Engineering Education, 14th International Symposium on Project
Approaches in Engineering Education (PAEE) and 19th Active Learning in Engineering Education Workshop (ALE), was organized by
Active Learning in Engineering Education Network (ALE) and PAEE – Project Approaches in Engineering Education Association.
http://paee.dps.uminho.pt/
This is a digital edition.

WELCOME TO PAEE/ALE’2022
Dear Participants,
Welcome to the International Conference on Active Learning in Engineering Education (PAEE/ALE’2022). This
is the seventh collaboration of the International Symposium on Project Approaches in Engineering Education
(PAEE) and the Active Learning in Engineering Education Workshop (ALE).
The theme of the conference is “Planting the seed: promoting engineers with global awareness”. The world
needs engineers that know how to establish a collaborative culture in their workplace and how to
deal/communicate/negotiate with people from different countries and cultures. Engineering schools and
faculty need to help their students to develop socio-emotional and intercultural skills that will provide the
future engineer with the competence needed to work in complex global contexts. Active learning strategies
and methods can be great tools for the development of these skills. By defining this theme, we would like to
raise awareness to ensure a guarantee of personal dignity, the free flow of persons without any
discrimination, and the right to effective equality between women and men.
Renowned keynote speakers from the Engineering Education and Sustainability areas will be present in this
PAEE/ALE edition sharing their experience with the participants. Besides, the offer of hands-on sessions will
generate learning-by-doing experiences that will allow experimentation and discussion of the proposals
presented.
May the International Conference on Active Learning in Engineering Education (PAEE/ALE’2022) be a fruitful
forum where participants, in their permanent need for continuing education, will have the opportunity to
discuss research and current practice under this challenging theme of promoting engineers with global
awareness.
We would like to express our sincere gratitude to the participants that made this event possible and for all
the support that we had during this last year from different people and organizations.
We hope you will enjoy the conference that the PAEE/ALE team has prepared for you.
Miguel Romá & Valquíria Villas-Boas

(Chairs of the PAEE/ALE’2022)
iv
PAEE/ALE’2022 Organization
PAEE/ALE’2022 Chairs
Conference Chairs
Miguel Romá, University of Alicante, Spain
Valquíria Villas-Boas, Universidade de Caxias do Sul, Brazil
Program Chairs
David Ballester, University of Alicante, Spain
Rui M. Sousa, University of Minho, Guimarães, Portugal
Scientific Committee Chairs

Rui M. Lima, University of Minho, Guimarães, Portugal
Montse Farreras, Universitat Politècnica de Catalunya, Spain
Student Session Chairs

Jens Myrup Petersen, Aalborg University, Denmark
Vicente Morell, University of Alicante, Spain
PAEE/ALE’2022 Organising Committee

Local Organizing Committee
Miguel Romá, David Ballester, Gabriel J. García, María José Blanes, Vicente Morell, Tomás Martínez
University of Alicante, Spain
Montse Farreras, Pau Bofill, Jesús Armengol

Universitat Politècnica de Catalunya, Spain
PAEE Steering Committee
Rui M. Lima, Dep. of Production and Systems, Algoritmi Center ‐ Engineering School, University of Minho, Guimarães, Portugal
Anabela Alves, Dep. of Production and Systems, Algoritmi Center ‐ Engineering School, University of Minho, Guimarães, Portugal
Diana Mesquita, Faculty of Education and Psychology, Universidade Católica Portuguesa, Porto, Portugal
Dinis Carvalho, Dep. of Production and Systems, Algoritmi Center ‐ Engineering School, University of Minho, Guimarães, Portugal
Nélson Costa, Dep. of Production and Systems, Algoritmi Center ‐ Engineering School, University of Minho, Guimarães, Portugal
Rui M. Sousa, Dep. of Production and Systems, Algoritmi Center ‐ Engineering School, University of Minho, Guimarães, Portugal
Sandra Fernandes, Department of Psychology and Education, Portucalense University, Porto, Portugal
ALE Steering Committee
Miguel Romá, University of Alicante, Spain (chair)

Jens Myrup Petersen, Aalborg University, Denmark (vice-chair)
Valquíria Villas-Boas, Universidade de Caxias do Sul, Brazil
Erik de Graaff, Aalborg University, Denmark
Jennifer DeBoer, Purdue University, USA
Lamjed Bettaieb, ESPRIT School of Engineering, Tunisia
Marietjie Havenga, School for Mathematics, Science and Technology Education, North-West University, South Africa
Pisut Koomsap, Asian Institute of Technology, Thailand
Rui M. Lima, Universidade do Minho, Portugal
v
PAEE/ALE’2022 Scientific Committee
Scientific Committee Affiliation
Aida Guerra Aalborg UNESCO Centre for PBL on Engineering Education and Sustainability, Aalborg University, Denmark
Ana Teresa Oliveira Technology and Management School, Viana do Castelo Polytechnic Institute, Viana do Castelo, Portugal
Anabela Alves Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
André Luiz Aquere Department of Civil and Environmental Engineering, Faculty of Technology, University of Brasília, Brazil
André Uébe Mansur Instituto Federal Fluminense e Universidade Federal Fluminense, Brazil
Andromeda Menezes Instituto Federal do Espírito Santo - Campus Cariacica, Brazil
Athakorn Kengpol King Mongkut's University of Technology North Bangkok, Thailand
Benedita Malheiro Instituto Superior de Engenharia do Porto, Portugal
Carina Pimentel Departamento de Economia, Gestão, Engenharia Industrial e Turismo, Universidade de Aveiro, Portugal
Celina Pinto Leão Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
Choon Seng Yap Republic Polytechnic (RP), Singapore
Crediné Menezes Universidade Federal do Espírito Santo, Brazil
Cristian Mustață Faculty of Engineering in Foreign Languages, University Politehnica of Bucharest, Romania
Cristiano Jesus Algoritmi Center, University of Minho, Braga, Portugal
David Ballester University of Alicante, Spain
Diana Mesquita Universidade Católica Portuguesa, Faculty of Education and Psychology, Porto, Portugal
Dianne M. Viana Department of Mechanical Engineering, Faculty of Technology, University of Brasília, Brazil
Erik de Graaff Department of Development and Planning, Aalborg University, Denmark
Filomena Soares Department of Industrial Electronics, Engineering School, University of Minho, Guimarães, Portugal
Gabriel J. García University of Alicante, Spain
Huynh Trung Luong Asian Institute of Technology, Thailand
Isabel Loureiro Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
Jens Myrup Pedersen Aalborg University, Dept. of Electronic Systems, Denmark
Jesús Armengol Universitat Politècnica de Catalunya, Spain
José Dinis-Carvalho Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
José Manuel Oliveira Higher Education Polytechnic School of Águeda, University of Aveiro, Portugal
José Souza Rodrigues Faculdade de Engenharia de Bauru, Universidade Estadual de São Paulo (UNESP)
Leidy Johanna Castrillón Centro de Estudios y de Investigación en Biotecnología (CIBIOT), Universidad Pontificia Bolivariana
Luiz Carlos Campos Faculty of Science and Technology, PUC São Paulo, Brazil
Marco Antonio Pereira Escola de Engenharia de Lorena (EEL), Universidade de São Paulo (USP), Brazil
Maria Catalina Cajiao Universidad de los Andes, Colombia
Mauricio Duque Departamento de Ingeniería Eléctrica, Universidad de los Andes, Colombia
Messias Borges Silva School of Engineering of Lorena, São Paulo University
Miguel Romá University of Alicante, Spain
Montse Farreras Universitat Politècnica de Catalunya, Spain
Natascha vanHattum-Janssen Saxion Research & Graduate School, Saxion University of Applied Sciences, Enschede, the Netherlands
Nélson Costa Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
Nicolae Ionescu University Politehnica of Bucharest, Romania
Octavio Mattasoglio Neto Instituto Mauá de Tecnologia, Escola de Engenharia Mauá - São Caetano do Sul - SP - Brazil
Pau Bofill Universitat Politècnica de Catalunya, Spain
Renata Chagas Perrenoud UNISAL - Lorena, Brazil
Rosane Aragon Universidade Federal do Rio Grande do Sul (UFRGS), Brazil
Rui M. Lima Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
Rui M. Sousa Department of Production and Systems, Engineering School, University of Minho, Guimarães, Portugal
Sandra Fernandes Department of Psychology and Education, Portucalense University, Portugal
Simone Borges Monteiro Department of Production Engineering, Faculty of Technology, University of Brasilia (UnB), Brasília, Brazil
Sonia Gomez Puente Technical University of Eindhoven, The Netherlands
Stefan Grbenic Institute of Business Economics and Industrial Sociology, Graz University of Technology, Austria
Stevan Stankovski Industrial Engineering and Management Department, Faculty of Technical Sciences, University of Novi Sad, Serbia
Subha Kumpaty Milwaukee School of Engineering, USA
Thanate Ratanawilai Prince of Songkla University, Thailand
Tomasz Nitkiewicz Częstochowa University of Technology, Poland
Valquiria Villas-Boas Centro de Ciências Exatas e Tecnologia, Universidade de Caxias do Sul, Brazil
Vicente Morell University of Alicante, Spain
Walter Nagai Universidade Federal de Itajubá (UNIFEI) - Campus de Itabira, Itabira, Brazil
Wichai Chattinnawat Chiang Mai University, Thailand
Zied Alaya ESPRIT Ingénieur, Tunisia
vi
PAEE/ALE’2022 Invited Speakers
PAEE/ALE’2022 attracted renowned keynote speakers from different sectors to share their viewpoints on the direction of education,
especially in engineering education in this new era. We are honoured to have the following inspiring keynote speakers:
● Antonio Turiel (Nacional Research Council, Spain)
● Shannon Chance (TU Dublin, UK)
● Jordi Segalàs (Univesitat politècnica de Catalunya, Spain)
Antonio Turiel
National research council (CSIC), Spain
Title of the keynote – Science curricula in the time of resource scarcity
Short bio
Antonio Turiel was born in León, Spain, in 1970. He is a Senior Scientist at the Institute for Marine
Sciences of Barcelona (CSIC, Spain) where he carries out research on physical oceanography using the
principle of fluid mechanics and thermodynamics. He has been also engaged on a intense scientific
communication activity regarding the depletion of conventional fossil fuel resources and the
anthropogenic causes of climate change as well as the failure of the capitalist mode of production. He
is the editor and main author of the scientific blog The Oil Crash, which became a benchmark on these
topics.
ShannonChance
TU Dublin, UK
Title of the keynote – Engineering knowledge, skills and values: keys to fostering
globally responsible changemakers
Short bio
Shannon Chance is a fully qualified Architect, licensed to practice in the Commonwealth of Virginia and a
Council Record Holder in the National Council of Architectural Registration Boards since 2005, meaning that
she is eligible for reciprocity/licensure in any state within the USA. She has also been a LEED Accredited
Professional (LEED-AP) under the United States Green Building Council’s Leadership in Energy and
Environmental Design program since 2009. Shannon’s PhD thesis, in the area of Higher Education Policy,
Planning and Leadership, looked at how universities have been using green building rating systems to
develop their built assets; she assessed to what degree their efforts constituted actual “leadership” in
environmental sustainability. She presented findings at NASA Langley. Her thesis earned the Outstanding
Dissertation Award from the International Society for Educational Planning and she graduated with the sole
2010 Dean’s Award for PhD students in her School. Aiming to work at the cutting edge of architecture and
building environment, Shannon recently completed a Postgraduate Certificate in Building Information Modeling (BIM) in Ireland.
Shannon has accepted global leadership roles, serving as Associate Editor for IEEE Transactions on Education, an Editorial Board
member of the European Journal of Engineering Education, and 2020-2022 Chair of the global Research on Engineering Education
Network which organizes a bi-annual symposium (upcoming in Australia and India) and publishes special focus journal issues.
Shannon is lead guest editor on a new special focus issue on ethics in engineering education and practice to be published spring 2021.
She is also a mentor for TU Dublin researchers as well as the Journal of Engineering Education.
vii
Jordi Segalàs
Universitat Politècnica de Catalunya (UPC), Spain
Title of the keynote – The ‘whys’, ‘whats’, ‘hows’ and ‘whos’ of Sustainability in
Engineering Edication
Short bio
Jordi Segalàs works as associate professor at the Research Institute of Sustainability Science and
Technology in the Universitat Politècnica de Catalunya UPC-Barcelona Tech. He is the head of the
Research Group on Sustainability Education and Technology in Higher Education. He has been the
Director of the Catalan Network of Education for Sustainability. He obtained his PhD in Sustainability
Education in Engineering from Barcelona Tech University. He has been working in curriculum greening
policies and actions plans at the Barcelona Tech University since 2000. He is also working in TEMPUS
(trans-European cooperation scheme for higher education) projects related to sustainable development
in higher education. He has published more than 60 articles on higher education and sustainability. He
is the chair of the Sustainability working group of SEFI. He has been working in curriculum greening
policies and action plans since 2000.
viii
PAEE/ALE’2022 Programme
ix
PAEE/ALE’2022 Paper Sessions, Hands-On Sessions and Students
Sessions
Day 1
Day 1: Wednesday, 6 July 2022
8:30-9:30 Opening ceremony & Ice-breaking activity (Room A)
Hands On Session 1 (Room C) Hands On Session 2 (Room E)
9:30-11:00
Miguel Romá Montse Farreras, Armengol Jesús, Pau Bofill, Àngels Hernández
20 Do we, as educators, have to be neutral in our 29 Game-based Learning vs Gamification: A Hands-On
classroom?
11:00-11:30 Coffee break (Inner courtyard)
Papers Session 1 (Room B) – Student Engagement
Session Chair: Anabela Alves
Jose Ignacio Pagan, Pablo Ortiz, Isabel Lopez
7 The use of Kahoot! to promote Active Learning in Port Management and Operation subject in Civil Engineering degree
Jose Ignacio Pagan, Isabel Lopez, Pablo Ortiz
9 Use of video tutorials for Active Learning and PBL in Coastal Engineering subject
Sandra Fernandes, Anabela Alves, Celina P. Leao, Ana Pereira
11:30-13:00
49 Promoting assessment as learning in PBL: findings from blogs created by first year students
Sandra Fernandes, Marta Abelha, Ana Sílvia Albuquerque
50 The student journey in PBL: using individual portfolios to promote self-reflection and assessment as learning
Anabela Alves, Sérgio Oliveira, Celina P. Leão, Sandra Fernandes
45 IEM@ProjectNetworking revisited: freshmen students closer to professional practice
Javier Ramirez Angulo, Margarita Portilla Pineda, Maria del Carmen González Cortés
61 Conducting a Home Experiment Related to the Colligative Properties of Chemical Solutions
13:15-14:30 LUNCH (Inner courtyard)
Papers Session 2 (Room B) – Research on Engineering Education
Session Chair: Shannon Chance
Miguel Romá, Josep David Ballester, Francisco Pedro Vives
10 Strengths and dangers of self and peer assessment in engineering learning. Teachers' and students' perspective
Dianne Viana, Valquíria Villas-Boas
13 Do Brazilian Engineering Professors Do Engineering Education Research in Active Learning? A Case Study
14:30-16:00 Rosane Aragón, Crediné Menezes, Alberto Castro Junior
48 Learning Assessment in a "Theses Debate"
Teresa Hattingh
67 Are your rubrics hitting the mark? An exploration of the use of rubrics for project-based learning in engineering
Daniela Pedrosa, Leonel Morgado and José Cravino
72 A strategy to support Engineering Education teaching staff monitoring students' learning process: Metacognitive
Challenges

KEY-NOTE Session (Room A)
16:30-18:00 Antonio Turiel
Science curricula in the time of resource scarcity
18:00-20:00 PAEE/ALE Challenge + Welcome cocktail
x
Day 2
Day 2: Thursday, 7 July 2022
9:30-11:00 Shannon Chance
Engineering knowledge, skills, and values: keys to fostering globally responsible changemakers

Hands On Session 3 (Room D) - laptop required Hands On Session 4 (Room C)
11:30-13:00 Jens Myrup Pedersen Elisandra Martins, Valquiria Villas-Boas

54 Haaukins: Cyber Security Training with Gamification 31 Engineers working as a team: socio-emotional and
intercultural skills for multicultural environments
13:15-14:30 LUNCH (Inner courtyard)
14:30-16:00 Jordi Segalàs
The ‘whys’, ‘whats’, ‘hows’ and ‘whos’ of Sustainability in Engineering Education

Students Session – Local/Online (Room B) Students Session – Online
Session Chair: Jens Myrup Pedersen Session Chair: Vicente Morell
Bruna Andrade Machado, Bárbara Hotta, Gabriel Lino Garcia, Fernando

(On-site) Pablo Ortiz, Jose Ignacio Pagan and Isabel Lopez
Bernardi Souza, José De Souza Rodrigues
8 Entrepreneurship as a tool to facilitate job placement for civil engineering 25 The use of games in the teaching of Production Engineering: a catalog
students based on publications in Scopus
(On-site) Lluis Bernat, Carlos A. Jara, Jose L. Ramon, Jorge Pomares, Gabriel Bárbara Yumi Hotta, Fernando Bernardi Souza, Bruna Andrade Machado,
J. Garcia, Andres Ubeda José De Souza Rodrigues
22 Exoforge: Interdisciplinary teaching laboratory for the development of 26 Evaluation of the effectiveness of the use of games in the teaching of
assistive technology projects Production Engineering: an experiment with Goldratt Simulator
(On-site) Isabel B. A. de Souza, Bruno L. Borba, Alexandre S. C. Costa,

Humberto Arruda, Édison R. Silva
Simone B. S. Monteiro, Edgar C. Oliveira
57 PBL to design a Gamified Financial Management Application for the 34 Lecturer Self-Awareness Index: measuring the alignment between
socially vulnerable lecturer and student perception
Mariana de Azevedo Mello, Júlia Rocha Thomaz Mattoso Salgado,
(On-site) Moises A. Croes, Carlos A. Jara, Jose L. Ramón, Jorge Pomares,
Marcelle Caldas Vicente, Fernando Augusto Ullmann Töbe, Stephany Rie
Gabriel J. García, Andrés Ubeda
Yamamoto Gushiken, Anibal Alberto Vilcapoma Ignácio
62 An advanced application for learning robotics using Augmented Reality 38 Application of Project-Based Learning and Lean tools in a machining
16:30-18:00
area of a welding industry
Miriam Chan-Pavón, Jesús Escalante-Euán, Ileana Monsreal-Barrera, Carlos Amanda Rocha, Ingrid Alves, Larissa Santos, Laura Sousa, Dianne Viana,
Rubio-Atoche Maura Shzu, Simone Lisniowski
23 Exploration of the use of Project Based Learning (PBL) methodology in 39 An approach to encourage girls' protagonism in exact sciences and
two accredited engineering programs at the Universidad Autónoma de engineering
Yucatán
Cláudia Gonçalves, João Neto, Sabrina Oliveira and José Dinis-Carvalho Mariane Souza, Rui Lima, Diana Mesquita, Elida Margalho
14 Project-based learning applied to improve the performance of a family 43 Development and Validation of Scenarios for the assessment of Project
wine bottling unit Management People Competences
Matheus Ferreira Palú, Octavio Mattasoglio Neto, Eduardo Nadaleto da
Jose Cristiano Pereira, Flavio Almeida, Ercilia de Stefano
Matta
28 Multidisciplinary projects for Engineering, Business Administration and 68 Proposal of Method for Risk Assessment of Project-Based Learning
Design programs: Construction and mapping of common skills through an Failure via AHP and BBN aiming at improving the Quality of Engineering
analysis instrument Education – A Case Study
Matheus Ricardo de Avila Santos, Maria Valéria Araújo, John Burgoyne,

Nicole Andressa Michel, Luiz Campos
Dmitryev Silva, Diego Paes, Rebeka Alves and Ana Beatriz Santana
4 Serious Game with emphasis on Global Prosperity: developing the Game 71 Circular Economy Implemented in a Cosmetic Company: A Case Study
Reis Magos from Design Science
Cultural Activity – Visit to Santa Bàrbara Castle (bus ride) and Alicante historic centre
19:00 ->
&
Conference dinner
xi
Day 3
Day 3: Fryday, 8 July 2022
Papers Session 3 (Room B) – Gamification Hands On Session 5 (Room C)
Session Chair: Sandra Fernandes
Pau Bofill, Montse Farreras, Jesús Armengol Jens Myrup Pedersen
30 An Escape Room For Learning Computer 53 Problem Based Learning – Making Blended Learning work!
Programming
Montse Farreras, Pau Bofill, Jesús Armengol
9:30-11:00 33 An escape room for an alternative evaluation
system
Andres Ubeda, Gabriel J. Garcia, Vicente Morell, Jose L.
Ramon, Maria J. Blanes
21 A gamification approach for continuous
engagement in engineering courses
Jesús Armengol, Pau Bofill, Montserrat Farreras
35 Gaming for learning

Papers Session 4 (Room B)
Session Chair: Rui Sousa
Social Development
Tatiane Evangelista
59 Seeding future engineers
Fernando J. Rodriguez-Mesa
66 The Torca Experimen: A model of transdisciplinary project-work
11:30-13:00
Multidisciplinar Approaches
Victor Paiva, Caio Santos
11 Project based learning approach in the heat transfer course for undergraduate students
Luciano Soares, Fabio Orfali
37 Pair Teaching in Computer Graphics
Celina P. Leão, M. Florentina Abreu, Anabela Alves and Sandra Fernandes
55 PBL tutoring dynamics in first-year of Industrial Engineering and Management Program
13:00-13:30 Closing Ceremony (Room A)

13:30-14:30 LUNCH (not included in the fees)
Post-Conference Visit
Human Robotics (HURO): Robotic engineering
15:00-17:00
Big structures (LARGE): Civil engineering
Autonomous industrial vehicles (Quixmind)
xii
List of Authors
Author Affiliation Country #
Alberto Castro Junior Universidade Federal do Amazonas Brazil 48
Alexandre S. C. Costa University of Brasilia Brazil 57
Amanda Rocha University of Brasília Brazil 39
Ana Beatriz Santana Universidade Federal do Rio Grande do Norte Brazil 4
Ana Pereira University of Minho Portugal 49
Ana Sílvia Albuquerque Portucalense University Portugal 50
Anabela Alves ALGORITMI Centre, School of Engineering, University of Minho Portugal 45,49,55
Andres Ubeda Human Robotics Group, University of Alicante Spain 21,22,62
Àngels Hernández UPC Spain 29
Anibal Alberto Vilcapoma Ignácio Fluminense Federal University Brazil 38
Armengol Jesús UPC Spain 29
Bárbara Yumi Hotta UNESP Brazil 25,26
Bruna Andrade Machado UNESP Brazil 25,26
Bruno L. Borba University of Brasilia Brazil 57
Caio Santos Insper Brazil 11
Carlos A. Jara University of Alicante Spain 22,62
Carlos Rubio-Atoche Facultad de Ingeniería Química - Universidad Autónoma de Yucatán Mexico 23
Celina P. Leao ALGORITMI Centre, School of Engineering, University of Minho Portugal 45,49,55
Cláudia Gonçalves University of Minho Portugal 14
Crediné Menezes Universidade Federal do Rio Grande do Sul Brazil 48
Daniela Pedrosa Universidade de Aveiro & CIDTFF Portugal 72
Diana Mesquita Universidade Católica Portuguesa Portugal 43
Dianne Viana University of Brasília Brazil 13,39
Diego Paes Centro Universitário do Rio Grande do Norte Brazil 4
Dmitryev Silva Universidade Federal do Rio Grande do Norte Brazil 4
Edgar C. Oliveira University of Brasilia Brazil 57
Édison R. Silva Universidade Federal do Rio de Janeiro Brazil 34
Eduardo Nadaleto da Matta Instituto Mauá de Tecnologia Brazil 28
Elida Margalho Universidade do Minho (UMinho) Portugal 43
Elisandra Martins University of Caxias do Sul Brazil 31
Ercilia de Stefano Universidade Federal Fluminense Brazil 68
Fabio Orfali Insper Brazil 37
Fernando Augusto Ullmann Töbe Fluminense Federal University Brazil 38
Fernando Bernardi Souza UNESP Brazil 25,26
Fernando J. Rodriguez-Mesa Universidad Nacional de Colombia Colombia 66
Flávia Vieira Centro de Investigação em Educação (CIEd), Institute of Education, University of Minho, Braga Portugal 69
Flavio Almeida Universidade Católica de Petropolis Brazil 68
Francisco Pedro Vives University of Alicante, Spain Spain 10
Gabriel Hornink Educational Media Lab., Dep of Biochemistry, Inst Biomedical Sciences, Federal Univ. of Alfenas Brazil 69
Gabriel J. Garcia Human Robotics Group, University of Alicante Spain 21,22,62
Gabriel Lino Garcia UNESP Brazil 25
Humberto Arruda Universidade Federal do Rio de Janeiro (UFRJ) Brazil 34
Ileana Monsreal-Barrera Facultad de Ingeniería Química - Universidad Autónoma de Yucatán Mexico 23
Ingrid Alves University of Brasília Brazil 39
Isabel B. A. de Souza University of Brasilia Brazil 57
Isabel Lopez University of Alicante Spain 7,8,9
Javier Ramirez Angulo Tecnológico de Monterrey Mexico 61
Jens Myrup Pedersen Aalborg University, Dept. of Electronic Systems Denmark 53,54
Jesús Armengol Universitat Politècnica de Catalunya, Spain Spain 29,30,33,35
Jesús Escalante-Euán Facultad de Ingeniería Química - Universidad Autónoma de Yucatán Mexico 23
João Neto University of Minho Portugal 14
John Burgoyne Lancaster University UK 4
Jorge Pomares University of Alicante Spain 22,62
José Cravino Universidade de Trás-os-Montes e Alto Douro & CIDTFF Portugal 72
Jose Cristiano Pereira UCP - Universidade Católica de Petrópolis Brazil 68
José De Souza Rodrigues UNESP Brazil 25,26
José Dinis-Carvalho ALGORITMI Centre, School of Engineering, University of Minho Portugal 14
Jose Ignacio Pagan University of Alicante Spain 7,8,9
Jose L. Ramon Human Robotics Group, University of Alicante Spain 21,22,62
Josep David Ballester University of Alicante Spain 10
Júlia Rocha Thomaz Mattoso Salgado Fluminense Federal University Brazil 38
Larissa Santos University of Brasília Brazil 39
Laura Sousa University of Brasília Brazil 39
Leonel Morgado Universidade Aberta & INESC TEC Portugal 72
Lluis Bernat University of Alicante Spain 22
Luciano Soares Insper Brazil 37
Luiz Campos Pontifical Catholic University of Sao Paulo Brazil 71
M. Florentina Abreu Centro ALGORITMI, School of Engineering, University of Minho, Guimarães Portugal 55
M. Teresa T. Monteiro ALGORITMI Centre, Dep. of Production and Systems, School of Engineering, University of Minho Portugal 69
Marcelle Caldas Vicente Fluminense Federal University Brazil 38
Margarita Portilla Pineda Universidad Autónoma Metropolitana Mexico 61
Maria del Carmen González Cortés Universidad Autónoma Metropolitana Mexico 61
Maria J. Blanes IT Service, University of Alicante Spain 21
Maria Valéria Araújo Universidade Federal do Rio Grande do Norte Brazil 4
Mariana de Azevedo Mello Fluminense Federal University Brazil 38
Mariane Souza Universidade do Minho (UMinho) Portugal 43
xiii
Author Affiliation Country #
Marta Abelha Universidade Aberta Portugal 50
Matheus Ferreira Palú Instituto Mauá de Tecnologia Brazil 28
Matheus Ricardo de Avila Santos Universidade Federal de Santa Maria Brazil 4
Maura Shzu University of Brasília Brazil 39
Miguel Romá University of Alicante Spain 10,20
Miriam Chan-Pavón Facultad de Ingeniería Química - Universidad Autónoma de Yucatán Mexico 23
Moises A. Croes HUman RObotics - University of Alicante Spain 62
Montse Farreras Universitat Politècnica de Catalunya, Spain Spain 29,30,33,35
Nicole Andressa Michel Pontifical Catholic University of Sao Paulo Brazil 71
Octavio Mattasoglio Neto Instituto Mauá de Tecnologia Brazil 28
Pablo Ortiz University of Alicante Spain 7,8,9
Pau Bofill Universitat Politècnica de Catalunya Spain 29,30,33,35
Rebeka Alves Universidade Federal do Rio Grande do Norte Brazil 4
Rosane Aragón Universidade Federal do Rio Grande do sul Brazil 48
Rui M. Lima ALGORITMI Centre, School of Engineering, University of Minho Portugal 43
Sabrina Oliveira University of Minho Portugal 14
Sandra Fernandes Portucalense University Portugal 45,49,50,55
Sérgio Oliveira Centro ALGORITMI, School of Engineering, University of Minho, Guimarães Portugal 45
Simone B. S. Monteiro University of Brasilia Brazil 57
Simone Lisniowski University of Brasília Brazil 39
Stephany Rie Yamamoto Gushiken Fluminense Federal University Brazil 38
Tatiane Evangelista Universidade de Brasília - UnB Brazil 59
Teresa Hattingh North West University South Africa 67
Valquiria Villas-Boas University of Caxias do Sul Brazil 13,31
Vicente Morell Human Robotics Group, University of Alicante Spain 21
Victor Paiva Insper Brazil 11
xiv
PAEE/ALE’2022 Submissions
The PAEE/ALE’2022, International Conference on Active Learning in Engineering Education, joins the
International Symposium on Project Approaches in Engineering Education – PAEE, which is being organized by
PAEE association, the Department of Production and Systems, of the School of Engineering, University of
Minho, since 2009, and the ALE workshop, which is being organized by the ALE Network, since 2000.
PAEE/ALE’2022 aims to join teachers, researchers on Engineering Education, deans of Engineering Schools and
professionals concerned with Engineering Education, to enhance engineering education through Active
Learning and Project Approaches through workshops and discussion of current practice and research.
PAEE/ALE’2022 event is hybrid, with both full online and local on site sessions.
The event has three type of submissions in English:

• Hands-on and Workshop submissions, aiming to encourage discussion of current practice and
research on project approaches.
• Full Papers for paper sessions, including standard research submissions, and papers of innovative
experiences describing implementation issues.
• Abstract submissions, which is a short submission that may be included in paper session
presentations or poster sessions presentations.
All full paper submissions were double reviewed by the PAEE/ALE’2022 scientific committee, and in some cases
had a third review. PAEE/ALE use a single blind review procedure. After notification of acceptance authors were
invited to submit a final paper of 6 to 8 pages long in Microsoft Word format, using the available template.
Accepted contributions were invited to make a presentation at the symposium.
The proceedings are published under the Guidelines on Open Access to Scientific Publications and Research
Data in Horizon 2020 (http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/hi/oa_pilot/h2020-hi-
oa-pilot-guide_en.pdf): "Open access to scientific publications refers to free of charge online access for any user."
The authors retain the copyright of their work.
The authors are responsible for the publications which reflect their own viewpoints. The organization cannot
be held responsible for any use which may be made of the information contained therein.
Publication Ethics and Malpractice Statement
The PAEE/ALE editorial board and scientific committee are committed to preventing publication malpractice,
do not accept any kind of unethical behaviour, and do not tolerate any kind of plagiarism. Authors, editors,
and reviewers are to be committed with good practice of publications and accept to fulfil the duties and
responsibilities as set by the COPE Code of Conduct (http://publicationethics.org/resources/code-conduct).
Based on these, PAEE/ALE expects authors, editors, and reviewers to be committed to the following general
guidelines:
• Editors take decisions on the acceptance of papers and compose and evaluate the proceedings quality.
• Ensure that all published papers have been fairly reviewed by suitably qualified reviewers.
• Expect original submissions from the authors and discourage misconduct.
• Expect that authors are responsible for language quality.
• Expect that the authors adequately reference the sources of their work.
• Ensure confidentiality of submissions and reviews.
• Reviewers do a fair and detailed review of paper(s) assigned to them.
1
PAEE/ALE’2022 List of Submissions
PAEE/ALE’2022 Submissions ................................................................................................................... 1

PAEE/ALE’2022 List of Submissions ................................................................................................................................ 2
PAEE/ALE’2022 Full Papers Submissions ........................................................................................................................ 4
Serious Game with emphasis on Global Prosperity: developing the Game Reis Magos from Design Science .......... 5
Matheus Ricardo de Avila Santos¹, Maria Valéria Araújo2, John Burgoyne³, Dmitryev Cyreneu da Silva2, Diego Cristóvão Alves
de Souza Paes2, Rebeka Coelho de Almeida Alves2, Ana Beatriz Cabral Santana 2 ..................................................................... 5
The use of Kahoot! to promote Active Learning in Port Management and Operation subject in Civil Engineering
degree ...................................................................................................................................................................... 16
José Ignacio Pagán1, Pablo Ortiz1, Isabel López1 ....................................................................................................................... 16
Entrepreneurship as a tool to facilitate job placement for civil engineering students ............................................ 22
Pablo Ortiz1, José Ignacio Pagán1, Isabel Lopez1 ....................................................................................................................... 22
Use of video tutorials for Active Learning and PBL in Coastal Engineering subject ................................................. 30
José Ignacio Pagán1, Isabel López1, Pablo Ortiz1,...................................................................................................................... 30
Strengths and dangers of self and peer assessment in engineering learning. Teachers’ and students’ perspective
................................................................................................................................................................................. 38
Miguel Romá1, Josep David Ballester-Berman1, Francisco P. Vives2, Juan Manuel López-Sánchez1, José Antonio Signes2, Tomás
Martinez-Marín1, Enrique Martín2, Jesús Selva1 ....................................................................................................................... 38
Project Based Learning Approach in the Heat Transfer Course for Undergraduate Students................................. 47
Victor A. S. M. Paiva1, Caio F. R. Santos1 .................................................................................................................................. 47
Do Brazilian Engineering Professors Do Engineering Education Research in Active Learning? A Case Study ......... 53
Dianne Magalhães Viana1, Valquíria Villas-Boas2 ..................................................................................................................... 53
Project-based learning applied to improve the performance of a family wine bottling unit .................................. 62
Cláudia Gonçalves1, João Neto1, Sabrina Oliveira1, José Dinis-Carvalho2 ................................................................................. 62
A Gamification Approach for Continuous Engagement in Engineering courses ...................................................... 70
Andres Ubeda1, Gabriel J. Garcia1, Vicente Morell1, Jose L. Ramon1, Maria J. Blanes2 ............................................................ 70
Exoforge: Interdisciplinary teaching laboratory for the development of assistive technology projects ................. 76
Lluis Bernat, Carlos A. Jara, Jose L. Ramon, Jorge Pomares, Gabriel J. Garcia, Andres Ubeda ................................................. 76
Exploration of the use of Project Based Learning (PBL) methodology in two accredited engineering programs at
the Autonomous University of Yucatán. .................................................................................................................. 85
Miriam V. Chan-Pavón1,2, Jesús F. Escalante-Eúan1, Ileana C. Monsreal-Barrera1, Carlos M. Rubio-Atoche1 .......................... 85
The use of games in the teaching of Production Engineering: a list based on publications in Scopus .................... 95
Bruna Andrade Machado1, Bárbara Yumi Hotta¹, Gabriel Lino Garcia², Fernando Bernardi de Souza¹, José de Souza Rodrigues¹
.................................................................................................................................................................................................. 95
Evaluation of the effectiveness of the use of games in the teaching of Production Engineering: an experiment
with Goldratt Simulator ......................................................................................................................................... 102
Bárbara Yumi Hotta1, Fernando Bernardi de Souza1, Bruna Andrade Machado1, José de Souza Rodrigues1......................... 102
Multidisciplinary projects for Engineering, Business Administration and Design programs: Construction and
mapping of common skills through an analysis instrument .................................................................................. 110
Matheus Ferreira Palú1, Octavio Mattasoglio Neto2, Eduardo Nadaleto da Matta3 .............................................................. 110
An Escape Room For Learning Computer Programming ........................................................................................ 119
Pau Bofill1, Montse Farreras2, Jesús Armengol3 ..................................................................................................................... 119
An Escape Room for an alternative evaluation system ......................................................................................... 126
Montse Farreras1, Pau Bofill2, Jesus Armengol3, Adrian Asensio1 .......................................................................................... 126
Lecturer Self-Awareness Index: measuring the alignment between lecturer and student perception ................. 134
Humberto Arruda1, Édison Renato Silva1 ............................................................................................................................... 134
Gaming for learning ............................................................................................................................................... 143
Jesús Armengol 1, Pau Bofill2, Montse Farreras3. ................................................................................................................... 143
Pair Teaching in Computer Graphics ...................................................................................................................... 149
Luciano Soares, Fabio Orfali ................................................................................................................................................... 149
Application of Project-Based Learning and Lean tools in a machining area of a welding industry ....................... 156
2
Mariana de Azevedo Mello¹, Júlia Rocha Thomaz Mattoso Salgado¹, Marcelle Caldas Vicente¹, Fernando Augusto Ullmann
Töbe2, Stephany Rie Yamamoto Gushiken³, Anibal Alberto Vilcapoma Ignácio4.................................................................... 156
An Approach to Encourage Girls' Protagonism in Exact Sciences and Engineering ............................................... 165
Amanda Nunes Rocha1, Ingrid de Castro Alves2, Larissa Pereira da Costa Santos3, Laura Beatriz Lima de Sousa3, Dianne
Magalhães Viana3, Maura Angélica Milfont Shzu4, Simone Lisniowski5 ................................................................................. 165
Development and Validation of Scenarios for the assessment of Project Management People Competences ... 173
Mariane C. Souza 1,2, Rui M. Lima1 , Diana Mesquita3, Elida M. Margalho1 ............................................................................ 173
IEM@ProjectNetworking revisited: freshmen students closer to professional practice ....................................... 182
Anabela C. Alves1, S. Oliveira2, Celina P. Leão1, Sandra Fernandes3,4 ..................................................................................... 182
Learning Assessment in a "Theses Debate" ........................................................................................................... 190
Rosane Aragón1, Crediné Silva de Menezes1, Alberto Nogueira de Castro Júnior2 ................................................................ 190
Promoting assessment as learning in PBL: findings from blogs created by first year engineering students ......... 198
Sandra Fernandes1,2, Anabela C. Alves3, Celina P. Leão3, Ana Pereira3 .................................................................................. 198
The student journey in PBL: using individual portfolios to promote self-reflection and assessment as learning . 207
Sandra Fernandes1,2, Marta Abelha3,4 & Ana Silvia Albuquerque1 ......................................................................................... 207
PBL tutoring dynamics in first-year of Industrial Engineering and Management Program ................................... 212
Celina Pinto Leão1, M. Florentina Abreu1, Anabela C. Alves1, Sandra Fernandes2 ................................................................. 212
PBL to design a Gamified Financial Management Application for the socially vulnerable .................................... 221
Isabel B. A. de Souza1, Bruno L. Borba1, Alexandre S. C. Costa1, Edgard Costa Oliveira1, Simone B. S. Monteiro1 ................. 221
Seeding future engineers ....................................................................................................................................... 230
Tatiane da Silva Evangelista1 .................................................................................................................................................. 230
Conducting a Home Experiment Related to the Colligative Properties of Chemical Solutions. ............................ 237
Javier Ramírez-Angulo1, Margarita Portilla-Pineda2, María del Carmen González-Cortés2 ........................................ 237
An advanced application for learning robotics using Augmented Reality ............................................................. 243
Moisés A. Croes, Carlos A. Jara 1, José L. Ramón1, Jorge Pomares1, Gabriel J. García1 and Andrés Ubeda1 ........................... 243
The Torca Experiment: A model of transdisciplinary project-work ....................................................................... 251
Fernando José Rodríguez-Mesa1 ............................................................................................................................................ 251
Are your rubrics hitting the mark? An exploration of the use of rubrics for project-based learning in engineering
............................................................................................................................................................................... 258
Teresa S. Hattingh1 ................................................................................................................................................................. 258
Proposal of Method for Risk Assessment of Project-Based Learning Failure via AHP and BBN – A Case Study .... 266
J.C Pereira1, E. Stefano2, F. Almeida3 ...................................................................................................................................... 266
Engaging with real-world phenomena through Matlab programming projects .................................................... 277
M. Teresa T. Monteiro1, Gabriel Hornink2, Flávia Vieira3 ....................................................................................................... 277
Circular Economy Implemented in a Cosmetic Company: a Case Study ................................................................ 286
Nicole Andressa Fuoco Michel, Luiz Carlos de Campos .......................................................................................................... 286
A strategy to support Engineering Education teaching staff monitoring students' learning process: Metacognitive
Challenges .............................................................................................................................................................. 295
Daniela Pedrosa1, Leonel Morgado2, José Cravino3................................................................................................................ 295
PAEE/ALE’2022 Hands-on sessions ............................................................................................................................ 304
Do we, as educators, have to be neutral in our classroom? .................................................................................. 305
Miguel Romá1 ......................................................................................................................................................................... 305
Game-based Learning vs Gamification: A Hands-On ............................................................................................. 307
Montse Farreras1, Jesús Armengol2, Pau Bofill3,Angels Hernández1 ...................................................................................... 307
Engineers working as a team: socio-emotional and intercultural skills for multicultural environments .............. 310
Elisandra Martins1, Valquíria Villas-Boas1 .......................................................................................................................... 310
Problem Based Learning – Making Blended Learning work! ................................................................................. 312
Jens Myrup Pedersen1 ............................................................................................................................................................ 312
Haakins: Cyber Security Training with Gamification .............................................................................................. 314
Jens Myrup Pedersen1 ............................................................................................................................................................ 314
3
PAEE/ALE’2022 Full Papers Submissions
Submissions accepted for the PAEE/ALE’2022 papers sessions in English.
4
Serious Game with emphasis on Global Prosperity: developing
the Game Reis Magos from Design Science
Matheus Ricardo de Avila Santos¹, Maria Valéria Araújo2, John Burgoyne³, Dmitryev Cyreneu da Silva2, Diego
Cristóvão Alves de Souza Paes2, Rebeka Coelho de Almeida Alves2, Ana Beatriz Cabral Santana 2
1
Universidade Federal de Santa Maria, Santa Maria, Brazil
2
Universidade Federal do Rio Grande do Norte, Natal, Brazil
3
Lancaster University, Lancaster, United Kingdom
Email: [email protected], [email protected], [email protected], [email protected],

[email protected], [email protected], [email protected],
DOI: https://doi.org/10.5281/zenodo.6916562
Abstract
Serious Games increasingly receive the support and interest from teachers as another active teaching methodology that,
built to achieve educational goals, also manages to motivate and attract the attention of students. The objective of the
research is to describe the development of a serious game used to assist in the teaching-learning process of global
prosperity. To this end, qualitative research was performed, adopting the Design Science model. As for the results, it was
identified that constant testing added value to the construction of the game. Another contribution of the study is the
emphasis on the concern that developers or educators must have regarding some elements in games, such as the rules
and mechanics. The research brought contributions to the educational process, as well as business practice, with the
combination of teaching cases and educational games.
Keywords: Case Method. Global Prosperity. Design Science.
1 Introduction
The social and technological context and new contemporary social habits and practices, in addition to the
generation of digital natives (Prensky, 2003) who have an innate ability to learn how to interact online
(McGonigal, 2011) trigger the growing emergence of new approaches and educational possibilities, allowing
the expansion of pedagogical actions in the classroom (Figueiredo, et al., 2015).
From this perspective, active learning methodologies, such as case studies and serious games, have several
advantages: allowing for the establishment of links between the teaching environment and the real world;
leading students to the variability of solutions; favouring the establishment of relationships between variables;
contributing to the analysis of a problem from different points of view; requiring students to publicly express
their ideas and subject them to criticism; adjusting to different levels of teaching complexity, and they can also
be used both in face-to-face and distance learning (Alves et al., 2015; Bacich & Moran, 2018).
As an active learning methodology, Teaching Cases are commonly used as a complement to other teaching
methods, usually based on the deductive method of teaching-learning, thus, they are based on theories,
models, or concepts (Roesch, 2007).
For Alves et al. (2015), Gamification is an active teaching methodology that arises, in this sense, with the
principle of appropriation of elements used in games for contexts of products and services not focused on
games, but to promote engagement and learning. Unlike games, the goal of gamification is not just having
fun, hence serious games.
The gamification process analyzed here had the pedagogical purpose of teaching and making students aware
of the concept of Global Prosperity. This concept has emerged in recent decades questioning the centrality of
economic growth as a measure of success and well-being of a community, proposing a multifaceted analysis
that considers alternative and contextualized values to judge the quality of life of a society.
Since its establishment, the concept of Sustainable Development, and initiatives derived from it, have become
the target of criticism, either for the false symmetry between the three factors propagated (Flint, 2010; Paes,
5
2017), for the impossibility of reconciling an ecologically viable economy with the objectives of a capitalist
system (Vizeu, et al., 2012), or by the contradictions of equalizing “growth” with the obvious limitations of
resources (Misoczky & Böhm 2012).
In this context, the concept of Global Prosperity emerges as a critical alternative to the unifying perspective of
Sustainable Development, focused on growth. Thus, the concept is particularly averse to the vision of prosperity
in a community defined by the notion of economic wealth (Sender et al, 2020).
In this way, the research aims to describe the development of a serious game to assist in teaching-learning
Global Prosperity from tests with Design Science. To achieve this objective, the following specific objectives
were defined: presenting the process of game building; - presenting the adjustments in the game structure; -
describing the structural elements of the game; - describing the game.
1.1 Theoretical elements

To understand the theoretical elements that would be part of the game, the researchers focused on
comprehending the design that a game should have, and for that, they were based on the Theory of Flow, and
also, to achieve educational goals, the Theory of Global Prosperity was deepened, bringing to the game a
reflexive character about the complex reality surrounding the player.
Kirriemuir and McFarlene (2004) comment that several researchers and video game developers based their
investigations on the theory of flow (figure 1), that shows that the shape or state of the flow depends on a
combination of challenges and skills.
Figure 1. Theory of flow source: Csikszentmihalyi, 1990.
For a person to reach a state of flow or challenge, they must report to the level of their own difficulties. For
example, in a situation (A1), the person is still learning to play with only a few skills at that time, or with reduced
challenge levels, so the degree of difficulty is suited to their skill (Csikszentmihalyi, 1990).
At this point, we are in a flow situation, which is likely to be short-lived, as the player will improve their dexterity
by repeating and adapting to the flow of the game. If no level changes occur in the challenge (A2), the player
will start to feel bored and presumably give up on the task. If the challenges are too far above the obstacles, it
will cause anxiety (A3), which can also lead to abandoning the game.
Lopes and Oliveira (2013) state that for a flow situation to occur, a continuous balance is needed between the
challenges presented and the current capabilities of the player (A1>>A4).
From a game-components perspective, Busarello (2016) highlights that gamification values five structural
elements: 1) learning; 2) game mechanics; 3) in-game thinking; 4) motivation and engagement; 5) narrative.
6
Vianna et al. (2013) and Zichermann and Cunningham (2011) point out that, to keep the student motivated in
the game building process, it is necessary to appropriate its most efficient elements - mechanics, dynamic and
aesthetic. The first comprises the elements of the functioning of the game, that guide the player in their actions;
while the second are the moments of interaction between the player and these mechanics; and, finally, the
aesthetic refers to the emotions of the players in the moments of interaction with the game, also reflecting the
previous relationships with mechanics and dynamics.
Hanus and Fox (2015) emphasize that one must be careful when defining rewards throughout the game,
because certain extrinsic rewards can destroy intrinsic motivations, affecting the motivational aspect of the
individual.
In this way, for the proposed game, a theory that contemplated a model extrapolating the ‘economicist’ metrics
was designed. Sustainability sees growth, in an economic sense, as necessary, especially with regard to solving
poverty and misery, emphasizing metrics such as GDP and family income.
In this sense, the concept of Global Prosperity promotes a multifaceted vision capable of seeing beyond the
economic dimension, incorporating values, quality of life, and well-being in its analysis, given a specific context
(Moore, 2015). For a real Prosperity to be sustainable, it must combine the development of the ability to live
well on a planet with finite resources, with the quality of life and relationships, the resilience of communities,
and a sense of collective and individual meaning (Jackson, 2017).
Thus, Global Prosperity envisages rethinking global society and its priorities, reframing much of what is thought
about life in the global south. Although historically classified as "developing" or "underdeveloped", the rupture
of a uniform vision of what prosperity means, and the end of a linear and evolutionary developmental vision,
makes it possible to focus on what quality of life actually means for a given community.
Many experiments have been performed in the development of Prosperity analysis tools (Sender et al., 2020).
In this sense, the Legatum Institute created a ranking of Global Prosperity among countries, analysing the
variables security, economic quality, governance, business environment, individual freedoms, social capital,
education, health, and environment (Legatum Institute, 2020).
Moore (2015), while recognizing the merit of the UN Sustainable Development Goals in establishing standards
and timelines for implementation, points to the need for more goals to be created considering the necessary
diversity and contextualization of Global Prosperity, which calls for a change in the concept of wealth, values
and social progress.
Thus, we have sought to assemble the elements of gamification for the educational purpose of generating
skills necessary to understand the complexity of decisions in a teaching-learning context that thinks of Global
Prosperity. Lopes and Oliveira (2013) and Alves (et al., 2015) define this type of gamification as serious game.
2 Methodology
This research is of the Design Science type. This term highlights the orientation, which goes from the
development of knowledge to the design of solutions for real-world problems, and the tools necessary to
perform appropriate actions attributable to professionals. In this case, a game that helps the teacher, in a
management context in higher education, to develop specific skills in their students, in particular the ability of
superior thinking based on the reflection of multiple decision-making possibilities.
Sordi, Meireles, and Sanches (2010) state that the process of using knowledge to plan and create an artifact,
when it is carefully, systematically, and rigorously analyzed about the effectiveness with which it achieves its
goal, can be called Design Science and applied in research in Administration.
In the present case, the game developed from this research is characterized as an artifact systematically
developed from an iterative process of constant validation. Venable (2006) proposes a framework for the
procedures of Design Science research, shown in figure 2
7
Figure 2. Framework for the Design Science research. Source: Venable (2006).
In figure 2, problem and solution are placed in the same flow, where it is possible to confront them to propose
new theories or hypotheses. Although the activities can be developed without a theoretical definition, during
the progress of the building-development-validation process, the theoretical construction occurs when
analyzing the interactions to understand the problem, its causes, and consequences; when defining and testing
the artifact and; with the validation of the results obtained by its application.
Based on figure 2, by Venable (2006), we can divide the research into 3 phases: building, natural or artificial
evaluation, and development. It is important to highlight the circular model of the system, a beneficial feature
in model validation.
The building stage seeks to understand how it is done, how it should be done, and the success parameters that
are adopted in the literature. For this, the basis of the work is theoretical, making the test versions of the game
into conceptual work frameworks, trying to create a solid base in which the educational principles of the game
were fostered.
Based on this, there are changes made from an evaluation; whether artificial from a role-playing simulation or
even simulations from action research, when researchers are involved in adaptation meetings; or even, field
studies, as in tests with subjects brought from outside the research.
The results of this iterative model, even if partial in this process, already denote a "design research", while the
result of this type of research can be either the finished artifact or an improvement in an artifact that exists, or
is in development.
2.1 Serious game Reis Magos

The game Reis Magos was inspired by the teaching case Reis Magos Hotel. It portrays a decision-making
situation involving a manager, regarding the fate of an important landmark for tourism, in addition to being a
historical and cultural monument in the city of Natal, in Rio Grande do Norte.
Throughout the case, data demonstrating the golden years of the Hotel are shown, followed by changes in the
places, due to the expansion of tourism to other regions of the city, and the consequent period of decay of the
hotel.
In the teaching case, the student is in the position of the manager, owner of the establishment, and is faced
with quantitative and qualitative data related to historical, cultural, economic, and social issues surrounding
the hotel, as well as possible choices.
8
From these reflections and ideas, there was an interest in the use of gamification. This originated discussions
regarding its applications, bringing important regional and cultural aspects, and a complex vision of a
phenomenon commonly analyzed and decided in favour of profitability and efficiency at any cost, including in
the actual outcome of the case in question.
2.2 Structuring of the artifact.

According to the method for the construction of a Design Research, explained by Venable (2006), the three
revision phases must occur circularly until the artifact requirements are reached.
The building phase, with the insertion of some authors of more current methodology and references, was
basically conducted during the preparation of the Teaching Case and also in the review of the concepts for the
construction of the reference, detailed in the session of the Theoretical Elements of this work.
Having partially overcome the building phase, given its restriction of format, the evaluation and development
phases started. To facilitate the organization of this phase, it was divided into three equivalent fronts,
implemented during developer meetings based on design thinking: ideation, prototyping, and testing, that is,
every meeting occurred iteratively in the construction of what the game would be.
It is important to emphasize that the entire evaluation and development process happened in the context of
the pandemic. Thus, all steps were performed in weekly meetings at distance with the support of online
software as Google Meet.
Regarding ideation, it can be considered a natural assessment, as the researchers were immersed in the process
of developing ideas that would contribute to the construction of an initial prototype.
The requirements adopted in the ideation phase were that the game should supply the need to continue the
Teaching Case; have at least 3 activities, an interactive one, a collaborative one, and a reflexive one, that led to
pondering about the implications of their actions in the game; it should provide conditions for students to
develop skills for complex thinking from the theory of global prosperity; and have a graphic representation
that alludes to the elements of the Teaching Case (the Hotel and the City).
The part of the meeting that included prototyping can also be considered a natural assessment, since the
developers were creating, modifying, or even improving the result of the ideation phase, in the form of the
operationalization of rules, mechanics, and educational objectives.
In the testing phases, there were two types of experiments:
Internal tests, which took place periodically, with the developers themselves. A member would always act as a
tutor, conducting the activity, due to changes in rules and new mechanics. This role was fundamental, mainly
at the beginning.
External testing was on-demand, as soon as we had a first viable prototype that met the requirements. Due to
social distancing, we have performed it at the student residence where one of the researchers lived. Other
researchers observed this test by video conference software. Both types of testing can be considered artificial
evaluations. One of role-playing (internal), and a field study (external).
The main contributions of artificial evaluations were given in 3 dimensions: Game Communication, Mechanics,
Resource Balance.
Regarding game communication, contributions to initial information on the map helped in the preliminary
communication for those who did not yet know the main rules of the game. As well as the inclusion of the
location of the Hotel helped players to create reference and empathy with the city and its culture. The
standardization of nomenclatures, such as “turns”, “acts”, and “resources”.
As for the mechanics, the inclusion of more interactive parts, such as the exchange mechanics were important
in two points: in the creation of a collaborative environment and the balance of points, when in an environment
9
conducive to these exchanges, whether due to the scarcity or abundance of a resource. Another contribution
to the mechanics was the inclusion of luck or bad luck.
In the balance of features, the main factor that led to this fundamental part of the game design was the
repetition of internal tests. The importance of repetition in identifying biases concerning resource imbalances
was easily identified when viewing the annotations made in the tests.
This way, we need to understand what the structural elements that the game implemented in its development
were, and then move on to their description.
3 Describing the structural elements of the game

Prior, the developers intended to make the game as interactive as possible, thus, characteristics common to
Role Play Games (RPG) were used, where the player assumes the role of a character, which has its own history,
characteristics, and motivations.
In this way, the game presents the narrative of the characters through three cards distributed throughout the
gameplay, they contain a description of motivations and objectives, from the characters, about what should be
done with the Hotel. This is also the first contact with the character. This aims to convey a point of view, where
the player can identify or not with the motivations, generating or not an empathetic link with the character.
As the game progresses, the player receives information about what happened to the Hotel, from its
construction to the most current moment in the case, along with other points of view on what should be done
with the organization. This qualitative information ends up influencing the decisions of the winner, as in that
moment the player can use the information along with their persuasion skills to defend the point of view from
their character, winning the game or not.
During the internal validations, the developers noticed that the main moments where the motivation was
highest were at the beginning of each act, when players receive a new card related to their character, delving
deeper into their narrative and, with it, they received a letter presenting the historic moment of the Hotel,
generating a broad vision of the period in which they are inserted.
In addition to the beginning of the acts, the end of the game also represents a peak of motivation, as the
strategy chosen by the player is put to the test and everything, they learned during the game can be used as
an argument in his defence in defining the winner.
As this is based on a true story, the demands for resources vary a lot between the possible decisions of what
will be done with the Hotel, so the researchers have chosen to divide them into 4 types, to measure the
objectives within the game, namely:
1. Monetary Resources: they represent how resources related to money or goods are needed for players
to reach their individual goals;
2. Social Resources: they represent how much of the social demand must be met for players to reach
their individual goals;
3. Media resources: they represent how many media-related activities players need to reach their goal;
4. Political resources: they represent how much political influence players need to achieve their goal.
These features are directly linked to the individual goals and narratives of each character, being part of one of
the main factors of gamification, which is the transformation of actions or events into resources, and thus
motivating players to seek the most demanded resources to achieve their goals. Finally, diversifying strategies
from each of them according to their demands.
10
3.1 Introducing the Reis Magos
The game starts with the delivery of the character sheet (figure 3), in this image we can see, in the upper left
corner, the individual goals of the character, each one represented by a symbol. From top to bottom: monetary
resource, social resource, political, and media resource.
The name of the character is right above its image, and, in the text, part of the story is narrated, which will have
a close relationship with the established goals. During the game, the resources acquired by each player must
be counted individually, for this it is recommended to use a notepad to manage them.
To prepare the cards, the developers used some popular board games as a model. Pandemic and Monopoly
were the main examples; thus, the cards are also one of the attractive aspects of the game. The software’s used
was Microsoft word, PowerPoint and Excel to organize the texts and the cards design.
Figure 3. Businessperson card. Source: Venable (2006).
As inspiration for the design of the map, the researchers chose the game map from The Godfather board Game,
where each region has its resources and characteristics, in this way each area has its peculiarities and, with this,
players must create their strategies considering these factors.
At first, the researchers chose the regions closest to the Hotel location to be present in the game, however,
during the development stage, demands from the group emerged so that all regions of Natal were present on
the map, aiming to represent less privileged regions of the city (figure 4).
At the beginning of the game, only areas 1 and 2 were available for players to select, each one has the number
of moves relative to the number of players, i.e., 3 players 3 moves. The move consists of selecting one of the
empty circles and placing your piece in it, receiving the resources available there. The number of vacancies in
each area is the same as the number of players.
11
Figure 4. Map of the game.
The game has 4 types of cards, they are used to distribute resources, give bonuses or penalties, while they are
also the main narration device of the case.
The resource cards (figure 5) have some resources and tell a little about the history of the Hotel, this information
is read to all players and can be used as an argument for defining the winner:
Figure 5. Resource card.
In addition to the businessperson cards (figure 3), presented at the beginning of the game description, there
are Act cards (figure 6), which report the historical period in which the game takes place, representing the time
of the game advancing.
12
Figure 6. Act cards.
Finally, there are the luck or bad luck cards (figure 7) aimed at randomly giving bonuses or penalizing
players.
Figure 7. Luck and bad luck cards.
There are game moments to help players achieve the goals from their characters, at the beginning and end of
the third act when players can make cards exchanges. As the resources are part of the cards, when an exchange
occurs, the previous resource is lost and the new one is received.
The winner of the game is decided in three moments, the first is the strategic evaluation, where each player
presents their cards and checks whether they reached their individual goal. After that, each player who did it
must defend, in a coherent way, dialoguing with the information he obtained during the game, the reasons
why his character deserves the victory, and which benefits will be brought both to the Hotel region and the
city of Natal.
Following the presentations, all players must vote to define the winner. If the result is not yet determined, an
external person (the teacher, tutor, or a person chosen by the players) must hear the defences presented and
decide the outcome.
After the winner is announced, everyone should read the epilogue cards (figures 8 and 9) related to the
character, keeping in mind that each card has a prerequisite written on it. After this, a joint reflection should
be made as to whether this was the best option for the future of both the Hotel and the city of Natal. This
moment serves to develop a critical view of the social issues that are inherent to the Hotel case.
13
Figure 8. Epilogue cards 1 and 2.
Figure 9. Epilogue card 3.
4 Conclusion
We believe that the objective of this research, which was to describe the development of a serious game to
assist in the teaching-learning of global prosperity from tests with Design Science, was achieved. We could see
that with Design Science methodology, constant modifications and tests were performed, enhancing the
construction of the game and providing changes that have improved its overall quality.
Another contribution of the study is highlighting the concern that developers or educators should have about
game elements during development: Communication of game elements, Mechanics, Balance of Resources.
We believe that research is relevant both for theory and the educational process, as well as for business practice.
This is demonstrated in the possibility of building games from teaching cases, thus contributing to the
development of people through the dissemination and adoption of the combination of these two
methodological teaching strategies.
We recognize that the study had limitations. Despite the game being built, it was not tested with a large sample,
only remotely, due to the pandemic. In this sense, we suggest further testing performed by studies with large
samples, in quantitative research, and, in addition, new research conducted in person.
5 Final considerations
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil
(CAPES) - Finance Code 001
6 References
Alves, F. (2015). Gamification: como criar experiências de aprendizagem engajadoras. DVS editora.
Bacich, L., & Moran, J. (2018). Metodologias ativas para uma educação inovadora: uma abordagem teórico-prática. Penso Editora.
Busarello, R. I. (2016). Gamification: princípios e estratégias. Pimenta Cultural.
Csikszentmihalyi, M., & Csikzentmihaly, M. (1990). Flow: The psychology of optimal experience (Vol. 1990). New York: Harper & Row.
14
Figueiredo, M. Paz, t. Junqueira, e. Gamification and education: a state of the art of research conducted in Brazil. Anais do iv congresso
brasileiro de informática na educação - cbie. 2015.
Flint, R. W. (2010). Seeking resiliency in the development of sustainable communities. Human Ecology Review, 44-57.
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social
comparison, satisfaction, effort, and academic performance. Computers & education, 80, 152-161.
Jackson, T. (2017). Beyond consumer capitalism: foundations for a sustainable prosperity. In Handbook on Growth and Sustainability.
Edward Elgar Publishing.
Kirriemuir, J., & McFarlane, A. (2004). Literature review in games and learning.
London: Legatum Institute. Index, L. P. (2020).
Lopes, N., & Oliveira, I. (2013). Videojogos, Serious Games e Simuladores na Educação: usar, criar e modificar. Educação, Formação &
Tecnologias, 6, 4-20.
McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. Penguin.
Misoczky, M. C., & Böhm, S. (2012). Do desenvolvimento sustentável à economia verde: a constante e acelerada investida do capital sobre
a natureza. Cadernos Ebape. br, 10, 546-568.
Moore, H. L. (2015). Global prosperity and sustainable development goals. Journal of International Development, 27(6), 801-815.
Paes, D. C. A. D. S. (2017). Conhecimento local, tecnologias apropriadas e o desenvolvimento sustentável local na piscicultura familiar do
Vale do Jamari/RO.
Prensky, M. (2003). Digital game-based learning. Computers in Entertainment (CIE), 1(1), 21-21.
Roesch, S. M. A. (2007). Notas sobre a construção de casos para ensino. Revista de Administração Contemporânea, 11(2), 213-234.
Sender, H., Hannan, M. A., Billingham, L., Isaacs, J., & Ocitti, D. (2020). Rethinking prosperity: perspectives of young people living in east
london.
Silva, A. R. L., Catapan, A. H., da Silva, C. H., Reategui, E. B., Spanhol, F. J., Golfetto, I. F., ... & Sartori, V. (2014). Gamificação na educação.
Pimenta Cultural.
Sordi, J. O., Meireles, M., & Sanches, C. (2011). Design Science aplicada às pesquisas em administração: Reflexões a partir do recente
histórico de publicações internacionais. RAI Revista de Administração e Inovação, 8(1), 10-36.
Vianna, M., Vianna, Y., Medina, B. & Tanaka, S., (2013). Gamification, Inc.: como reinventar empresas a partir de jogos.
Venable, J. (2006, February). The role of theory and theorising in design science research. In Proceedings of the 1st International Conference
on Design Science in Information Systems and Technology (DESRIST 2006) (pp. 1-18).
Vizeu, F., Meneghetti, F. K., & Seifert, R. E. (2012). Por uma crítica ao conceito de desenvolvimento sustentável. Cadernos Ebape. br, 10,
569-583.
Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps. " O'Reilly
Media, Inc.".Education, 32(3), 337 - 347.
15
The use of Kahoot! to promote Active Learning in Port
Management and Operation subject in Civil Engineering degree
José Ignacio Pagán1, Pablo Ortiz1, Isabel López1
1
University of Alicante, Department of Civil Engineering, Alicante, Spain
Email: [email protected], [email protected], [email protected]
Abstract
Port Management and Operation subject in Civil Engineering degree of the University of Alicante is a mainly theoretical
subject, full of complex engineering terms. In addition, the schedule of this course is from 15 to 17 h, just after lunchtime.
This causes a decrease in the attendance and motivation of the students throughout the course. To mitigate this problem,
an Active Learning methodology was proposed. Personal response systems are mechanisms with a system of controls that,
in real time, can be used to carry out questionnaires and interpret the results. One of these systems is the platform Kahoot!.
The materials of each lesson were provided to the students so that they could work on the contents before the sessions
(flipped classroom). A quiz is completed before and after each session to determine the degree of involvement of the
students and the level of attention during the session. The first and last 5-10 minutes of each of the theory sessions were
dedicated to the Kahoot! quizzes. Most of the students' connections were made with their mobile phones. The results show
that the level of student engagement increases over the course, as the percentage of correct answers before the explanation
increases over the course. It is also observed that after the lessons, the rate of correct answers increases, which indicates
that the students are more involved and follow the class with greater interest than previously. Moreover, gamification
improves the classroom learning atmosphere. The use of Kahoot! to promote active learning, to reinforce the knowledge
of the concepts seen in class and to create an evaluative test in the form of a game were evaluated in a very positive way.
Keywords: Active Learning; Civil Engineering Education; Kahoot!; Soft Skills.
1 Introduction
Teaching in Spanish faculties generally involves a large number of students, whose active participation in
classes and evaluation involves great difficulties (Pérez-Colodrero, 2020). Thus, one of the main problems in
university teaching nowadays is the lack of motivation and participation on the part of the students (Pintor et
al., 2014). The subject of Management and Operation of Ports of the Degree in Civil Engineering at the
University of Alicante, although it has few students, there is a lack of motivation among them.
At the beginning of the 21st century, the first personal response systems appeared to alleviate this problem.
These were initially called "clickers", which are mechanisms with a system of controls that, in real time, can be
used to perform questionnaires and interpret the results (Pintor et al., 2014). One of these systems is Kahoot!
platform. This platform is a personal or group response system that, based on the game, transforms the
teaching-learning process (da Silva et al., 2018). In this way, students learn from a playful perspective (Jaber et
al., 2016), their motivation, commitment and active participation are fostered (Sharples, 2000; Wang, 2015) and,
at the same time, digital competence is developed (Álvarez-Rosa et al., 2018).
The recreational component of these games promotes student motivation, confirming the existence of a
relationship between gamification and motivation (Kenny & McDaniel, 2011). Therefore, if one of the main
objectives is to increase the motivation of students to achieve meaningful learning (Curto Prieto et al., 2019),
games present a good opportunity to increase the motivation, desire and commitment of students in their
teaching-learning process (Serrano et al., 2011).
Some educators consider game-based learning an effective method because it maintains the purpose of
education, enhances player skill, and can be used in real life (Von Wangenheim & Shull, 2009). In fact,
gamification has increased exponentially in education since 2014 perhaps because educational games put the
learner at the centre of learning, which facilitates more fruitful and interesting learning (Torres-Toukoumidis et
al., 2019). Therefore, the introduction of games in class aims to foster dynamism, engagement and motivation
16
among other factors (Lee & Hammer, 2011). But the current educational landscape demands the inclusion of
technological advances to improve the quality and the learning process, which would be in line with the
preferences of the millennial generation for more active and technological (Jain & Dutta, 2019).
Considering the benefits described above, this methodology has been implemented in the course of Port
Management and Operation of the Degree in Civil Engineering at the University of Alicante to achieve the
results obtained in similar studies in the scientific literature described for this resource. Thus, it is expected to
improve the participation, performance and motivation of students in the subject, mainly in the theoretical part
of it.
2 Method
This project was conducted within the framework of the course "Management and Operation of Ports" of the
Civil Engineering Degree of the University of Alicante. This compulsory course is part of the fourth year of the
degree within the branch of Transport and Urban Services. It is a subject that consists of 1.20 credits of theory,
0.60 credits of problems and 0.60 credits of fieldwork. The inclusion of the Kahoot platform was implemented
in the 2018-2019, 2019-2020 and 2020-2021 courses of the subject of Port Management and Operation of the
fourth year of the Degree in Civil Engineering at the University of Alicante. The experience evaluated a total of
14 students (6 in 2018-19, 4 in 2019-20 and 4 in 2020-21) during the theory hours shared between two
professors every Wednesday of the second term from 15:00 to 17:00.
In this research, the Kahoot! platform was used to evaluate the level of attention, understanding and
involvement of the students, both in their work at home and during the theoretical classes. This platform allows
for questionnaires that can be answered individually or in groups, and also allows setting a time limit for
answering the question. Students earn points for each correct answer as well as for the speed with which they
answer the different questions correctly. For students to participate in the surveys, they must have a device
connected to the network. It is not necessary to have an account on the platform application, simply know the
test code and indicate a name or "nickname" to identify yourself during the duration of the questionnaire.
When the time to answer each question is over or when all the participants have answered, the answer selected
by each contestant, the correct answer, the score obtained in that question by each participant and the
positions in the ranking are displayed. Finally, at the end of the questionnaire, it is possible to include questions
to obtain feedback on the feelings of the students.
In our case, the tests were set to be answered individually by each of the students. Each test had a different
number of questions depending on the length and complexity of the unit. The response time was set at 20
seconds for all cases. To determine the degree of enjoyment of the students in taking these tests, four questions
were established only in the last test that was taken. The questions indicated the level of enjoyment valued
between zero and five, the belief of having learned with the method (yes or no), the possible recommendation
of the method (yes or no), and the level of satisfaction with the methodology, with the possibility of answering
between zero to five.
On the first day of class, the procedure and teaching methodology that would be used during the theoretical
classes of the course was explained. It was indicated that during each session two questionnaires would be
conducted on the Kahoot! platform, one at the beginning and one at the end of the session. During this first
class, a first test was carried out to find out the knowledge that the students had about the subject.
Throughout the following sessions, the first and last 5-10 minutes of each theoretical class were reserved for
the completion of the questionnaires with the Kahoot! platform. This is because in each class two
questionnaires were carried out with two purposes.
1) Before the theoretical explanation (BTE Test). Before the explanation of each lecture, a test was conducted
with questions related to the topic. This made it possible to know the level of knowledge of the subject of the
students, as well as their work at home and involvement with the subject since at the beginning of the course
the theoretical development of each of the topics and their organization and distribution throughout the
course were given.
17
2) After the theoretical explanation (ATE Test). After the explanation of the topics corresponding to each
session, a test was conducted with repeated questions from the initial test and other new questions, but all of
them related to the topic of the session. The purpose of this test is to determine the level of attention,
motivation and learning of the students throughout the class.
Once all the students had answered each of the ATE test questions, the answers chosen by each one were
commented on and discussed, and the correct answer was reasoned. At the end, the teacher of the session
kept the results obtained by the participants in each session.
Finally, during the last session, the initial test (conducted on the first day) was repeated to see the increase in
knowledge of the students concerning the beginning of the course.
3 Results
The first result to be analyzed is the results obtained by the students after the inverted class, that is, the previous
evaluation of the knowledge of the students before the explanation of the unit in class. It should be born in
mind that the students had all the information on the subject from the beginning of the course. Figure 1 shows
the average results obtained by each of the courses evaluated for each of the 14 theoretical class sessions. As
can be observed, in general, the percentage of correct answers before the theoretical explanation is between
54% and 71%, with an average of 61.6% in the three courses. Unit 10 stands out for its low percentage of
correct answers, with an average of 45.7% in the three courses. It is also observed (except for unit 10) that the
percentage of success in the last units tends to increase slightly, which leads to thinking that little by little the
students tend to become more involved in the readings and learning of the material before the explanation in
class. Next, the success rate of the students after the theoretical classes is evaluated. In this case, the percentage
of correct answers is higher than in the questionnaires before the explanation in class, which implies a high
level of student attention during the theoretical sessions. In this case, the percentages of correct answers are
between 61% and 95%, with an average of 79.4% in the three courses. Once again, unit 10 stands out with an
average correct answer of 85.5%, which implies an increase of 87% concerning the percentage of correct
answers before the theory class (the average for the rest of the units is around 26%). This leads to the conclusion
that this is a complex topic for students to understand unless they are correctly guided during its development.
100%
90%
80%
70%
Percentage of success
60%
50%
40%
30%
20%
10%
0%
Unit 1 Unit 2 Unit 3 Unit 4 Unit 5 Unit 6 Unit 7 Unit 8 Unit 9 Unit 10 Unit 11 Unit 12 Unit 13 Unit 14
18-19 BTE 18-19 ATE 19-20 BTE 19-20 ATE 20-21 BTE 20-21 ATE
Figure 1. Percentage of correct answers in each of the Kahoot! questionnaires conducted in each of the courses analyzed
before (BTE) and after (ATE) the explanation of the subject unit.
Figure 2 shows the results of the test conducted on the first day of class to determine the level of student
knowledge of the subject. As can be seen at the beginning of the course, knowledge is very low with an average
18
of 57% of the answers being correct. This is due to the high degree of specialization of the subject, the contents,
and the vocabulary used in the port environment. However, the knowledge at the end of the course as observed
is much higher with an average of 85% of the questions answered successfully. These good results were also
reflected in the final grade of the course, which increased by 1.8 points concerning previous courses in which
the usual methodology was used.
100% 93%
90% 85%
78%
Percentage of success
80%
70% 61%
59%
60% 52%
50%
40%
30%
20%
10%
0%
18-19 19-20 20-21 18-19 19-20 20-21
First day Last day
Figure 2. Results of the initial test conducted on the first day and the last day of class.
The analysis of the four final questions to determine the degree of student acceptance of the Kahoot! platform
(Figure 3) shows that in general, 89% of students consider this learning system to be innovative, fun and
motivating in the classroom. The 2020-2021 course is the one that least values the platform as learning
dynamize with 79% compared to 85% in 2018-2019 and 2019-2020. Analyzing the questions individually, it is
observed that 87% of the students consider that it is a fun way of learning, highlighting the course 2019-2020
with 91%. Interestingly, the same percentage (87%) of students consider that they have learned using this
methodology and that they would recommend it. The percentage of 2020-2021 stands out with only 81% in
the question on whether they have learned something, while the 2018-2019 and 2019-2020 courses present
93% and 88%, respectively. Finally, concerning the question of the level of satisfaction with the methodology,
only 79% of the students said that they had a positive feeling. The 2018-2019 and 2019-2020 academic years
stand out again with 73% and 71%, respectively, while the 2020-2021 academic year shows the lowest
percentage with only 63%. Degree of satisfaction
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Level of enjoyment
Belief of learning
Possible recommendation
Level of satisfaction
Total
18-19 19-20 20-21

Figure 3. Responses to the questions to determine the degree of acceptance of the use of the Kahoot! platform for each of
the implementation courses.
19
4 Discussion
Several studies have shown that the performance of students in class increases at the same time that their
motivation increases (Ausó-Monreal et al., 2020; Sousa, 2016). This seems to be in line with the results obtained
for two reasons: i) the final grade of the courses in which Kahoot is applied as a dynamic tool increases
considerably concerning the courses in which it is not used, and; ii) the success rate of the students in the
different questionnaires increases throughout the sessions (Figure 1). The subject of Management and
Operation of Ports is within the subjects of transport and urban services so that, in general, students do not
show a great interest in this subject (outside the usual scope of this branch), which can be deduced from the
low average grade obtained. However, in the years in which the methodology of using the Kahoot platform
has been implemented, the average grade has increased by about two points.
The motivation of the student cannot be based solely on the teaching methodology, but the teaching staff
plays a very important role (Fernández, 2002; Torelló, 2011). For example, the students in the last course where
Kahoot is applied (2020-2021) have the worst opinion of the platform of the three courses analyzed (Figure 3),
and yet they are the ones with the highest average grade of the eight courses studied. Nevertheless, this group
of students considers that the platform helps them in learning (Figure 3), which is consistent with the results
obtained in other similar studies (Ausó-Monreal et al., 2020; Rodríguez-Fernández, 2017). The difficulty of the
subject matter can also influence student motivation. Thus, in general, the students can understand and
assimilate about 60% of the subject by themselves, while a dynamic and enjoyable explanation by the teacher
makes the percentage of success reach almost 80%. In addition, in unit 10, the explanation provided by the
teacher increases the percentage of correct answers from 45.7% to 85.5% (Figure 1). Curiously, the success rate
in the test on the first day of class (57%, Figure 2) is approximately the same as the final average grade achieved
by the students when the Kahoot! platform was not used in teaching.
Another factor that, according to different authors, can affect the degree of attention is the moment in which
the subject is taught, such as the day or time of the session (Blake, 1967). In this sense, it can be affirmed that
this factor has not influenced the results obtained since the beginning of the course, which has been taught
on Wednesdays from three to six o'clock. Finally, to correctly evaluate the real attention level of the students
in the program, it would be necessary to conduct the questionnaires every fifteen minutes, because as indicated
(among others) by Sousa (2016) throughout the duration of a class, the attention of the student varies, being
on average about 15-20 minutes (Bunce et al., 2010). However, this is not feasible given that the length of the
units and the organization of the subject do not allow this time to be wasted throughout the class.
5 Conclusion
With all the results obtained in this work, it can be concluded that:
• The flipped classroom allows students to achieve a success rate of about 60% (BTE test), approximately
equal to the final grade obtained by students without Kahoot!
• The explanation of the subject matter by the teacher and the use of Kahoot! increases the success rate
to 80% (ATE test).
• Student motivation increases throughout the course when using Kahoot!, as demonstrated by the
increase in the success rate.
• Some units cannot be understood by the students on their own, and the explanation of the concepts
by the teacher is necessary for their correct assimilation.
• In general, students consider that the use of the platform is fun, increases their level of learning and
would recommend its use.
To continue increasing the use of the platform, it is necessary to conduct a study on the response time, since
some of the students indicated that, although the recreational part motivated them to study the subject, the
pressure of having little time to answer the questions made them nervous and stressed them out.
20
6 Acknowledgements
This work was partially supported by the Universidad de Alicante through the project “El empleo de la
plataforma Kahoot! en la asignatura Gestión y explotación de puertos del grado en Ingeniería Civil” (XARXES-
2021-5656).
7 References
Álvarez-Rosa, C. V., Marcos, E. V., Navarro, M. N., Rodríguez, V. M. M., & de la Mata, I. C. (2018). La competencia digital en
la universidad con la aplicación Kahoot! In A. Press (Ed.), Innovative strategies for Higher Education in Spain (pp.
49-57).
Ausó-Monreal, E., García Velasco, J. V., Gómez-Vicente, V., Gutiérrez Flores, E., & Angulo Jerez, A. (2020). Innovación
educativa para valorar el grado de atención y asistencia en titulaciones universitarias. El uso de “Kahoot” en el
aula. In R. Roig-Vila (Ed.), El compromiso académico y social a través de la investigación e innovación educativas en
la Enseñanza Superior (pp. 519-529). Octaedro. http://hdl.handle.net/10045/87519
Blake, M. (1967). Time of day effects on performance in a range of tasks. Psychonomic science, 9(6), 349-350.
Bunce, D. M., Flens, E. A., & Neiles, K. Y. (2010). How long can students pay attention in class? A study of student attention
decline using clickers. Journal of Chemical Education, 87(12), 1438-1443.
Cortizo, J. C., Carrero, F., Monsalve, B., Velasco, A., Díaz del Dedo, L., & Pérez, J. (2011). Gamificación y Docencia: Lo que la
Universidad tiene que aprender de los Videojuegos. Revista Española De Documentación Científica, 34(4), 621-
629.
da Silva, J. B., Andrade, M. H., de Oliveira, R. R., Sales, G. L., & Alves, F. R. V. (2018). Tecnologias digitais e metodologias
ativas na escola: o contributo do Kahoot para gamificar a sala de aula. Revista Thema, 15(2), 780-791.
Fernández, J. T. (2002). El docente universitario ante los nuevos escenarios: implicaciones para la innovación docente. Acción
pedagógica, 11(2), 30-42.
Jaber, J. R., Arencibia, A., Carrascosa, C., Ramírez, A., Rodriguez-Ponce, E., Melián, C., . . . Farray, D. (2016). Empleo de Kahoot
como herramienta de gamificación en la docencia universitaria III Jornadas Iberoamericanas de Innovación
Educativa en el ámbito de las TIC (InnoEducaTIC 2016), Las Palmas de Gran Canaria, España.
Pérez-Colodrero, C. (2020). El empleo de la plataforma Kahoot! en el grado de historia y ciencias de la música de la
Universidad de Granada: La experiencia del alumnado a través de relatos biográfico-narrativos. In S. L. Dykinson
(Ed.), Desafíos de investigación educativa durante la pandemia COVID19 (pp. 591-603).
Pintor, E., Gargantilla, P., Herreros, B., & López del Hierro, M. (2014). Kahhot en docencia: una alternativa práctica a los
clickers XI Jornadas Internacionales de Innovación Universitaria "Educar para transformar", Granada, España.
Rodríguez-Fernández, L. (2017). Smartphones y aprendizaje: el uso de Kahoot en el aula universitaria. Revista mediterránea
de Comunicación, 8(1), 181-189.
Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & education, 34(3-4), 177-
193.
Sousa, D. A. (2016). How the brain learns. Corwin Press.
Torelló, Ò. M. (2011). El profesor universitario: sus competencias y formación. Profesorado. Revista de Currículum y
Formación de Profesorado, 15(3), 195-211.
Wang, A. I. (2015). The wear out effect of a game-based student response system. Computers & education, 82, 217-227.
21
Entrepreneurship as a tool to facilitate job placement for civil
engineering students
Pablo Ortiz1, José Ignacio Pagán1, Isabel Lopez1
1
Department of Civil Engineering, University of Alicante, Alicante, Spain

Abstract
Traditionally, civil engineering professionals have worked for public institutions and large construction companies.
Nowadays, recent graduates of civil engineering have problems finding their first job despite the requirements of the
degree. Creativity, collaboration, persuasion and emotional intelligence are the essential soft skills needed. Unfortunately,
none of them appears in the civil engineering curriculum.
This work proposes to conduct the creation of small cooperative teams of students to face different challenges in line with
the syllabus of one or several subjects. Thanks to this, students can increase the number of competencies developed during
their academic life at the university. Nevertheless, they are able to apply technical knowledge while they gain professional
experiences, improve their soft skills and enhance the ability to understand, use and manage their own emotions through
active learning. Incorporating real-life technical problems, already solved or not, from different companies is the key to
establishing alliances with enterprises that connect hard and soft skills. In this way, the quality of the grade can be improved
while the students keep learning key competencies of the subject allowing them to gain skills and professional experience.
Furthermore, this activity can be the beginning of a start-up in an academic environment.
This research has shown the need for the development of soft skills in civil engineering students, both undergraduate and
master's degrees, to promote entrepreneurship and self-employment as job opportunities. The creation of the teams, their
management, the resolution of the different problems from real-life situations or adapted, the connection with different
companies and institutions and the presentation of the results can be the key to the holistic development of future
professionals. Including the possibility of entrepreneurship, since all the students questioned considered that they have
improved their entrepreneurial competencies thanks to this experience.
Keywords: Active Learning; Engineering Education; Entrepreneurship; Start-Up; Soft Skills.
1 Introduction
Both the current and future generations of civil engineers will not find it sufficient to assimilate the curriculum
content related to the technical knowledge of road design, dam maintenance or port management.
Unemployment rates among civil engineering graduates are on the rise and there are more and more civil
engineering schools. Therefore, it is necessary to introduce knowledge related to entrepreneurship as well as
the development of soft skills as a differentiator for the subsequent employability of graduates.
An important aspect to justify this need is the unemployment rate of close to 5% in the year 2020 along with
the low rate of Spanish university graduates who develop their professional activity from self-employment
(Garcia-Barba et al., 2020). Furthermore, the European Union has set as a strategic objective within the ET2020
document ''Increasing creativity and innovation, including entrepreneurship, at all levels of education and
training'' (Weber, 2012). Therefore, this innovative action is understood and explained within this objective.
Also considering, entrepreneurship is a complex and extensive process ranging from the generation of the
idea, the process of elaboration and maturation of the idea, the output to the real world, as well as the
evaluation of the process to complete the action performed.
From a formal point of view, there is more and more research and in-depth study of the phenomenon of
entrepreneurship and the concept itself, and most developing and developed countries are now beginning to
clearly link entrepreneurship education and training (Matlay, 2008). It is clear, as already mentioned, that the
concept of entrepreneurship education is much broader than just entrepreneurship training content. As Osorio
and Duart (2011) state, economic sciences have a functional view, of what to do; human sciences focus on the
22
subject, they are interested in who and why; whereas management sciences apply to the process, how. We can
complete the definition of the concept of entrepreneurial activity as the management of radical and
discontinuous change, or strategic renewal regardless of whether this strategic renewal occurs within or outside
existing organisations, and regardless of whether it results in a new business (de Haro et al., 2019).
Of course, as Timmons (2003) suggests, the myth that entrepreneurs are born has evolved, leading to the
consensus that entrepreneurship, like any other discipline, can be learned. If we delve a little deeper,
entrepreneurship includes the study of the sources of opportunities, the processes of discovering, evaluating
and exploiting opportunities, and the people who discover, evaluate and exploit them. Entrepreneurship does
not require, but may include the creation of new organisations (Shane & Venkataraman, 2003). Furthermore,
we should understand these entrepreneurial competencies as ''the set of knowledge, skills, abilities and
aptitudes necessary for the effective work of an individual in a specific work environment'' (Savanevičienė et
al., 2008). Entrepreneurship should therefore be approached holistically by acquiring the set of competencies
and we can understand it from Social Learning Theory (Bandura, 1987), the individual's attention is directed
towards the entrepreneurial phenomenon, instilling certain knowledge and skills to be able to create
entrepreneurial activity, which facilitates and supports the emergence of entrepreneurial behaviours. Although
the observation method is not the only approach to internalise and reproducing these behaviours (Arriaga et
al., 2006). For instance, learning by doing is one of the keys to the development of these entrepreneurial skills
(Lăcătuş & Stăiculescu, 2016).
The aim of this research is to evaluate the influence that the development of entrepreneurial projects has on
the development of soft skills for entrepreneurship in students, as well as the improvement of the results in
the subject itself by carrying out a realistic and viable project. Students to promote entrepreneurship and self-
employment as job opportunities. The creation of small cooperative teams of students to face different real-
life challenges while they keep learning key competencies of a subject will improve their soft skills and
professional experience, including the possibility of entrepreneurship as a tool to facilitate job placement
thanks to having an integral vision of the project and not only a technical one. Even, in the future, they will be
able to identify any project as a business opportunity.
2 Method
2.1 Context and student profile
This experience was implemented in the subject Maritime Traffic and Port Operation, optative framed in the
first four-month period of the second year of the Master’s degree in Civil Engineering, during the academic
year 2020-21 and 2021-22. The purpose of this subject is to familiarize and train students in the understanding
of the knowledge derived from maritime traffic and port operations. The course is carried out through
theoretical and practical lessons on problems with practical cases related to maritime traffic and port
operations. It has 3 credits ECTS: 1.20 Practical credits and 1,80 Distance-based hours. The entrepreneurial
project is fully developed during the first quarter of the second academic year of the Master.
The students of Master’s degree in Civil Engineering have a deeper understanding of the multiple constraints
of technical and legal nature arising in public work construction. Moreover, it is designed to boost their ability
to use proven methods and technologies in order to achieve greater efficiency in public work construction. In
addition, the students who are attending this course have already passed subjects in the civil engineering
degree that previously develop the basic knowledge to create entrepreneurial projects, such as Engineering
and Business, Works for organisation and occupational health and safety or Urban planning and environment
(Table 1). Therefore, they should already be able to tackle the professional world of civil engineering and the
construction business.
2.2 Description of the experience

The aim of the experience is to develop teamwork to solve an open problem so that students are able to adopt
a creative and innovative solution ensuring sustainability from an economic, financial and technological point
of view. The problems or projects to be solved are connected to real problems and can lead to public-private
23
collaboration, thus allowing the creation of a learning environment that leads to the development of skills and
attitudes in the profile of civil engineering students.
Through the entity that manages the ports of Valencia, Sagunto and Gandía, the Port Authority of Valencia and
the Open Innovation Hub of Valenciaport, a series of real challenges will be established that must be solved as
sustainable projects from an integral point of view. The Port of Valencia is also one of the leaders in container
transport in the Mediterranean Sea, reaffirming itself as well as "strengthening its leadership position as a
reference and strategic hub of the Mediterranean in the management of goods and traffic, its commitment to
sustainability and the environment, digitalisation and transparency". Therefore, each academic year will include
a series of challenges related to the sustainable mobility of people and goods, the development of a
competitive offer of infrastructures and services, as well as the fight against climate change. In short, the aim
is to develop solutions and business models to achieve a smart, green and resilient port in the future (Figure
1). The different development proposals should be connected to the items given in Figure 1, related to
circularity, planning, management and operations. In this way, we connect the development of entrepreneurial
ideas considering social and environmental aspects.
Table 1: Curriculum of the Undergraduate degree in Civil Engineering at the University of Alicante. In bold are
the subjects that previously develop the basic knowledge to create entrepreneurial projects.
FIRST-YEAR ECTS
FUNDAMENTALS OF MATHEMATICS IN ENGINEERING I 6
FUNDAMENTALS OF PHYSICS IN CIVIL ENGINEERING 6
FUNDAMENTALS OF CHEMISTRY IN CIVIL ENGINEERING 6
FUNDAMENTALS OF INFORMATION TECHNOLOGY 6
ENGINEERING AND BUSINESS 6
FUNDAMENTALS OF MATHEMATICS IN ENGINEERING II 6
MECHANICS FOR ENGINEERS 6
GRAPHIC EXPRESSION I 6
FUNDAMENTALS OF MATHEMATICS IN ENGINEERING III 6
GEOLOGY APPLIED TO CIVIL ENGINEERING 6
SECOND-YEAR
STRUCTURAL ANALYSIS I 7.5
CONSTRUCTION MATERIALS I 6
GRAPHIC EXPRESSION II 7.5
HYDRAULICS AND HYDROLOGY 9
SOIL AND ROCK MECHANICS 6
FURTHER MATHEMATICS 6
STRUCTURAL ANALYSIS II 6
CONSTRUCTION MATERIALS II 6
TOPOGRAPHY AND PHOTOGRAMMETRY 6
THIRD-YEAR
GEOTECHNICS AND FOUNDATIONS 6
ELECTRICAL TECHNOLOGY AND LIGHTING ENGINEERING 6
24
CONSTRUCTION PROCEDURES AND MACHINERY IN PUBLIC WORKS 6
URBAN PLANNING AND THE ENVIRONMENT 6
METALLIC STRUCTURES 6
REINFORCED AND PRE-STRESSED CONCRETE STRUCTURES 6
WORKS ORGANISATION AND OCCUPATIONAL HEALTH AND SAFETY 6
+COMPULSORY SUBJECTS: PUBLIC WORKS, HYDROLOGY OR TRANSPORT AND SERVICES 18
FOURTH-YEAR
FINAL PROJECT 12
+COMPULSORY SUBJECTS: PUBLIC WORKS, HYDROLOGY OR TRANSPORT AND SERVICES 48
•New energy •Optimized

models planning
•Innovative waste •Transparency
solutions and efficiency
CIRCULARITY PLANING
MANAGMENT OPERATIONS
•Technological •Trade facilitation

solutions •Unnecessary
•Management waiting times
Systems
Figure 1: Challenges in which the different projects will be included
The development of the entrepreneurial project can be structured in four steps (Figure 2) It begins with the
presentation of the challenges for the academic year. After the first phase of research on the different proposals
by the teams, the choice of the problem or project is made.
25
Ideation and
Presentation of
Research solving the
challenges
problem
Development of
Project
the project and
Presentation
BMC
Figure 2 Steps of the entrepreneurial project
The next phase has to do with the part of in-depth research of the problem, academic and professional review
of the subject studied and the ideation of proposals that solve the problem with the characteristics mentioned.
The following step is related to the realization of the ideas generated in the previous stage, the development
of the project from the technical point of view and the application of the Business Model Canvas (Figure 3). It
resolves aspects such as the value proposition, or what our solution brings to the table that is different from
what was being done before (Osterwalder & Pigneur, 2010).
Figure 3: Business Model Canvas adapted for the entrepreneurial project.

Moreover, it considers the customer segment or, depending on the type of project, the different users. It also
includes the relationship established with the customers and how our solution reaches these users. This
document reflects the activities and key resources that are necessary for our proposal to work as well as the
possible alliances that can or should be established for it to work properly. Finally, it considers its economic
balance, both in terms of costs and revenues. We will also include the social and environmental impact of the
project itself. In this way, students solve the proposed problems from a technical point of view but are able to
understand their economic feasibility. Thus, they can understand how their solution would work from a
business point of view, giving it the form of a business idea and almost a business plan, one of the first steps
to entrepreneurship through this model. Finally, the project is presented for its evaluation both by the teachers
26
of the course and by the organizations or entities that have proposed the issues and problems to be solved in
order to assess both from the technical and academic point of view the different solutions provided.
2.3 Evaluation
For the global evaluation of the Project, not only concerning the given solution but also to measure the whole
development of the project from an integral point of view, the following evaluation rubric is proposed for the
different teams that carry it out, establishing the score from 0 to 10 and determining a weight percentage of
influence of the value of each item, being the final grade the weighted grade (Table 2).
Table 2: Evaluation rubric for the evaluation of the Project.
n % Item
1 10 Problem characterization and data collection with a supporting methodology
2 10 Data analysis and generation of results lead to conclusions in the development of the project.
Evaluation of the problem from multiple perspectives including the connection between different areas
3 10 and disciplines.
Demonstrate a comprehensive understanding of the problem to be solved, based on critical thinking and
4 10 creative solutions.
5 10 Teamwork, effective management and learning through failure.
6 30 Appropriate, innovative and creative technical solution
7 20 Effective presentation and communication of the proposed solution or project
The rubric was developed by researching the best practices found in Weber (2012), in Lăcătuş and Stăiculescu
(2016) or Garcia-Barba et al. (2020). Therefore, for the design of the rubric we have considered a holistic view
of the development of the project and weighted those items according to the importance they may have when
carrying out another successful entrepreneurial project.
In addition, to evaluate the degree of achievement of the objectives regarding the development of soft skills
with the realization of the project by the participants, a short questionnaire of satisfaction was carried out with
the evaluation of the following questions from 1 to 5, being 1 not at all and 5 very much:
• Have you improved your ability to work in a team?
• Have you improved your conflict resolution skills?
• Have you improved your ability to communicate ideas and concepts?
• Have you improved your ability to adapt your ideas?
• Have you improved your creative capacity and strategic vision?
3 Results
The first aspect to highlight is the positive feedback from the students during the term from a qualitative point
of view. All of them have expressed the great practical application of the work proposed for the course and the
possibility of using different knowledge from other subjects in the development of the project itself, as well as
giving a practical approach to the resolution of the practical part of the course. During the realisation and
subsequent defence of the project they work on the five main competences measured. For example, the
development of teamwork throughout the whole four-month period, adopting different roles during the
project. Creativity and strategic vision to provide solutions to the problems initially posed, as well as the integral
conception of the project so that it is technically and economically viable. The ability to communicate, both
with colleagues, teachers and collaborating companies, at different levels, is important for the project as well
as the resolution of conflicts implicit in any project contextualised in the real world. As I mentioned, these five
competences are key to developing an entrepreneurial attitude and valuing entrepreneurship as a job
opportunity. Secondly, the average grade of the participating students has improved considerably. The
improvement concerning the last academic year without implementation was 16 % (2020-2021) for the first
year with the new methodological proposal and 18% for the second year of implementation (2021-2022).
27
(Figure 4). Students are involved in the subject and the development of the projects by carrying out a realistic,
feasible project designed to simulate a future professional activity.
10.0
9.5
8.8 9
9.0
AVERAGE GRADE
8.5
8.0 7.6
7.5
7.0
6.5
6.0
5.5
5.0
2019-2020 2020-2021 2021-2022
ACADEMIC YEAR
Figure 4. Average grade of the subject for the last three academic years
Finally, the measurement of the results of the perception of the development of soft skills shows the
improvement experienced in the last course concerning them. The assessment of the questionnaires shows
that the students have experienced the development of these skills (Figure 5).
Communication and Work in teams were the soft skills more developed, followed by a significant level of
Creativity and strategic vision. Conflict resolutions and adaptability of ideas were the lesser evaluated skills, but
both with a score of 3.8/5.
Work in team
5
4.2
4
Creativity and 4.0

2
3.8
Conflict
strategic vision resolution skills
1
3.8 4.6
Adapt ideas Communication
Figure 5. Mean of the responses to self-assessment questionnaire competencies
4 Conclusions
This research tackles the need for the development of soft skills in civil engineering students, both
undergraduate and master's degrees, to promote entrepreneurship and self-employment as a job opportunity,
since recent graduates have problems finding their first job. Creativity, collaboration, persuasion and emotional
intelligence are the essential soft skills needed.
28
This work developed a strategy to improve soft skills through the creation of small cooperative teams of
students to face different real-life technical problems, establishing alliances with leading companies in the
sector and connecting hard and soft skills in line with the syllabus of Maritime Traffic and Port Operation of
the Master’s degree in Civil Engineering at the University of Alicante.
The results show that by working on the concepts related to the subject, these soft skills can be acquired and
developed, becoming more attractive to students. This is reflected in the increase in the average grade of the
subject after applying this methodology. Thanks to this, students can increase the number of competencies
developed during their academic life at the university.
Finally, it is worth highlighting the usefulness of this work methodology, since all the students consider that
they have improved in the entrepreneurial competencies evaluated by means of the self-assessment
questionnaire. Therefore, we consider this activity a success as it has improved the job prospects of our students
and can also be the beginning of a start-up in an academic environment.
5 Acknowledgements
This work was partially supported by the Universidad de Alicante through the project “Desarrollo e
implementación de recursos innovadores para la resolución guiada de prácticas con ordenador que mejoren
el proceso de enseñanza-aprendizaje” (XARXES-2021-5649).
6 References
Arriaga, J., Ortega, M. G., Meza, G., Huichán, F., Juárez, E., Rodríguez, A., & Cruz, S. (2006). Análisis conceptual del aprendizaje
observacional y la imitación a conceptual. Revista latinoamericana de psicología, 38(1), 87-102.
Bandura, A. (1987). Teoría del aprendizaje social. Espasa-Calpe.
de Haro, J. M., Pozo-Rico, T., Mira Galvañ, M. J., Andreu, E., Gilar-Corbi, R., Corral-Granados, A., Poveda Brotons, R., &
Castejón, J. L. (2019). Desarrollo de competencias de innovación para el fomento del espíritu emprendedor en
estudiantes de educación superior. In R. c. Roig-Vila (Ed.), Memorias del Programa de Redes-I3CE de calidad,
innovación e investigación en docencia universitaria. Convocatoria 2018-19 (pp. 1253-1275). Institut de Ciències
de l’Educació (ICE) de la Universitat d’Alacant.
Garcia-Barba, J., Cano, M., Tomás, R., Ivorra, S., Neipp, C., Reyes, J. A., Reyes Márquez, A., & Soriano-Vicedo, J. (2020).
Criterios de diseño para la mejora de la docencia en el Máster en Ingeniería de Caminos, Canales y Puertos a partir
de encuestas de empleabilidad sobre los egresados. In R. c. Roig-Vila (Ed.), Memorias del Programa de Redes-I3CE
de calidad, innovación e investigación en docencia universitaria. Convocatoria 2019-20 (pp. 635-643). Institut de
Ciències de l’Educació (ICE) de la Universitat d’Alacant.
Lăcătuş, M. L., & Stăiculescu, C. (2016). Entrepreneurship in Education. International conference KNOWLEDGE-BASED
ORGANIZATION, 22(2), 438-443.
Matlay, H. (2008). The impact of entrepreneurship education on entrepreneurial outcomes. Journal of Small Business and
Enterprise Development, 15(2), 382-396.
Osorio, L. A., & Duart, J. M. (2011). Análisis de la interacción en ambientes híbridos de aprendizaje Comunicar, 37(19), 65-
72.
Osterwalder, A., & Pigneur, Y. (2010). Business model generation: a handbook for visionaries, game changers, and challengers
(Vol. 1). John Wiley & Sons.
Savanevičienė, A., Stukaitė, D., & Šilingienė, V. (2008). Development of strategic individual competences. Engineering
economics, 58(3).
Shane, S., & Venkataraman, S. (2003). Guest editors’ introduction to the special issue on technology entrepreneurship.
Research policy, 32(2), 181-184.
Timmons, S. (2003). Nurses resisting information technology. Nursing inquiry, 10(4), 257-269.
Weber, R. (2012). Evaluating Entrepreneurship Education. Gabler Verlag.
29
Use of video tutorials for Active Learning and PBL in Coastal
Engineering subject
José Ignacio Pagán1, Isabel López1, Pablo Ortiz1
1
University of Alicante, Department of Civil Engineering, Alicante, Spain
Abstract
In the past, workshops lessons on the Coastal Engineering subject were held in computer classrooms, where different
problems related to the subject were resolved. The professor guided the students through each of the sessions, while at
the same time the students solved them on their computers, asking any questions that arose and, finally, completing the
tasks individually, either in the classroom or at home. However, this methodology usually presented serious difficulties to
students, particularly the impossibility in some cases to follow the synchronised rhythm of the explanations, which led to a
reduction in attendance and a drop in academic performance. To improve it, a teaching model based on Active Learning
and Project-Based Learning (PBL), supported by of Information and Communication Technologies (ICT) resources, was
proposed. Students now acquire the competencies by carrying out their vulnerability analysis projects. Audio-visual material
(video tutorials) recorded by the professors solving a case of study in a guided way, covering the computer sessions were
created and shared. Students therefore can view those videos asynchronously as many times as necessary, using the device
that suits them best (TV, computer, tablet or mobile phone) at the same time as they carry out their work. The practical
sessions are now dedicated to the development of a vulnerability analysis project. Assignments and supplemental materials
from instructors keep learners engaged and accountable. The results obtained show an increase in attendance and follow-
up of the computer sessions, as well as a reduction in the drop-out rate of the subject and an improvement in the quality
of the project submitted by the students at the end of the course. In addition, an increase in academic performance and
global students’ satisfaction has been detected.
Keywords: Engineering Education; Video Tutorial; Computer Lesson; Project-Based Learning.
1 Introduction
The development of the European Higher Education Area (EHEA) has led to updating the teaching
methodologies traditionally used. Its implementation has implied a real educational innovation in the use of
more active teaching-learning methodologies, supported by the special relevance of Information and
Communication Technologies (ICT) as a systematic resource to help improve the training process of the
students (Cañete et al., 2012). The number of hours of autonomous work required for students to assimilate
the contents developed is explicitly stated in the methodological planning of each subject, contained in the
corresponding teaching guide. As Aguila et al. (2013) indicate, in order to facilitate the autonomous work of
students, there may be a multitude of curricular material available on the Internet (notes in digital format,
exercise resolution, guided computer practice, video tutorials, etc.) that adapts to the learning pace of the
students, without time restrictions, and that allows reinforcing and complementing the contents developed in
the classroom lessons, promoting the acquisition of knowledge, skills and professional competences. Similar
experiences have been succeeded developed in the field practices sessions with success (Pagán et al., 2021).
The Active Learning and Project-Based Learning (PBL) methodology have been implemented in recent years in
different European degrees and universities, especially in the field of engineering (Lehmann et al., 2008).
Compared to traditional lectures, this new approach offers several advantages: on the one hand, students have
a leading role in on-site sessions, increasing their motivation and satisfaction. On the other hand, active
methodologies and ICTs are naturally incorporated into learning processes, and various transversal
competencies can be developed and assessed in the classroom (Zabalza et al., 2016). In this methodology,
learning is organised through a real problem or project proposals. The problem or project is the starting point
of the learning process, and its resolution involves integrating contents and procedures in a learning activity,
30
facilitating a better understanding of the contents of the subject and working on their application to real
situations (Lehmann et al., 2008). Its main characteristics are:
• It focuses on the development of projects whose characteristics are often defined by the students
themselves, based on guidelines and main aims set by the tutor.
• The project is presented at the beginning of the process, becoming the guiding thread of the learning
process, developing the students' competencies. Thus, the concepts, procedures and attitudes
included in the learning objectives of the subject will be assimilated.
• Learning activities are carried out in teams, in a collaborative and active environment among students.
• There is continuous monitoring of student learning, both at the group and individual level,
encouraging critical reflection, self-assessment and co-assessment, with the tutor acting as a guide
and advisor in the process.
• It involves the development of specific skills and abilities, such as leadership, the capacity for
coordination, consensus, decision-making, communication skills, and the sharing of roles and
responsibilities.
• These abilities and new skills are acquired through active learning.
For this purpose, video tutorials are a key ICT resource that makes it easier for students to learn the contents
(Colomo & Aguilar, 2017). Video tutorials can be reproduced as many times as the students require, stopped
during playback to repeat a specific part, and viewed from anywhere on computers, TVs or mobile devices
depending on the needs of the students, which makes the video format an essential resource for offering
quality training (Brame, 2016). The duration of video tutorials is an element of discussion (González et al., 2010).
While some experiences have been developed during a traditional practice session lasting between 1h - 1h
30min, capturing the screen using specific software and recording the audio with the teacher's explanations
and the doubts or questions that arise for students during the practical session (González et al., 2010), most
focus on the development of videos specifically to support lessons. Knowledge pills allow access to concrete
information very quickly, but with the restriction that their content is very limited and, therefore, their learning
activities are very targeted. The availability of a good collection of short videos as part of the didactic material
of a subject, made available to students, allows them to execute and consume them autonomously as a training
complement, thus improving the effectiveness of knowledge transfer (Alonso-González et al., 2021).
This research focuses on the development and implementation of strategies and methodologies in the
implementation of formative assessment in practical subjects, assessing them based on objective quality
criteria, such as academic results, and subjective criteria like the anonymous opinion collected from students
and their own experience of the teaching staff throughout its implementation. Specifically, a new methodology
is proposed for the development and evaluation of the computer sessions on the subject Coastal and Oceanic
Engineering for the Degree in Marine Sciences at the University of Alicante. The incorporation of new tools in
our teaching practice, many of them based on ICTs, will improve the independent learning of students. Thus,
the specific objectives to be achieved by this teaching experience are the following: (i) to investigate the
different resources available offered by ICTs for the guided resolution of computer-based practical sessions
and project-based learning, (ii) to design materials for the guided resolution of the practical sessions adapted
to the subject Coastal and Oceanic Engineering of the degree in Marine Sciences of the University of Alicante,
(iii) to implement the use of the designed resources in the teaching of the current course and finally, (iv) to
evaluate the results of the experience using the information adequately collected using questionnaires to
students.
2 Method
2.1 Context and participants
This educational experience was implemented in the subject Coastal and Oceanic Engineering, which is part of
the 4th year of the Degree in Marine Sciences, pursued during the first semester and with six European Credit
Transfer and Accumulation System (ECTS) credits. It aims to provide students with the acquisition and
application of the necessary knowledge involved in coastal phenomena, as well as in the design and
31
construction of facilities and engineering works in the maritime and coastal environment. The development of
the subject is carried out through theoretical classes, practical problems, computer practice and field visits. In
the academic year 2021-2022, there are a total of 18 students enrolled. The computer lessons are conducted
over 5 sessions of one hour each, distributed throughout the course so that the concepts learnt during the
theory sessions can be applied.
Throughout the sessions, the teacher made an example in class of obtaining the Coastal Vulnerability Index
(CVI) on a coastal strip, obtaining the geomorphological, oceanographic and marine community parameters.
To do this, he used a geographic information system, QGIS, a new software for students, which can be complex
without some previous knowledge. The students were, at the same time, doing their work on their computers,
asking questions and, finally, completing the tasks autonomously, either in the classroom or, in most cases, at
home, given the lack of time to complete them in the classroom due to the slow pace of the explanations.
Moreover, students usually complained about the impossibility in some cases to follow the synchronised
rhythm of the explanations, which led to a reduction in attendance and a drop in academic performance.
As a method of evaluation, at the end of the course the students must hand in a report describing the
procedure followed and the results obtained from the calculation of the CVI. The mark obtained represented
10% of the total of the course.
2.2 Description of the experience

Firstly, a design phase was carried out in the initial weeks of the course and before the start of the computer-
based practice sessions. All available information on Active Learning and PBL, supported by ICT resources, was
compiled. After exploring the different methods and resources available (notes in digital format, existing videos
on the Internet, creation of specific video tutorials, etc.), scenarios for testing new methodological tools and
innovative techniques for improving the teaching-learning processes were developed and implemented. The
aim is to achieve greater student engagement, which should have a positive impact on the results of the
continuous assessment throughout the course.
The best way to implement the PBL and Active Learning methodology was discussed among all the participants
in the network, and the conclusion was to carry out group workshops on obtaining CVI in different sectors of
the coast. Professors met to establish the quantity and appropriate content of each of the video tutorials
necessary to carry out this experience, in line with the development of the contents of the subject and with the
programming of the computer-based practices foreseen in the schedule. Two video tutorials were created
containing the guided development of the computer exercises, with the step-by-step resolution of a practical
case of obtaining the CVI in a coastal stretch different to those assigned to the groups. For each video tutorial,
files were prepared with the title, description, contents, format and duration, as well as screenshots of the cover
and part of the video (Figure ). In addition, to reflect the real workload, the weight of this part of the course
has been increased to 30% of the total mark, with 60% for the report to be submitted with the methodology
and results obtained and 40% for the presentation and oral defence of the project carried out.
For the evaluation of the experience, the instrument to be used will be mainly a questionnaire designed to find
out the students' assessment of the new methodology implemented and the video tutorials, including possible
answers where appropriate. Moreover, we will also compare the attendance and the results of the evaluation
of the current academic year in which the experience will be implemented with the reports of previous years.
As described above, the new methodology based on PBL and Active Learning has been implemented in the
academic year 2021/22. The students will carry out a project in groups of between 2 and 3 people whose
objective is to obtain the Coastal Vulnerability Index of various stretches of coastline of similar length and
complexity. Sessions 1, 3 and 5 will be done in classic workshop lessons, while for sessions 2 and 4 the viewing
of the video tutorials is implemented.
This activity was planned to be carried out asynchronously, although it was recommended to be carried out
before the date assigned to the respective sessions in order to be able to dedicate the session (1 hour each)
to the resolution of doubts and/or problems that arose for each student. The video tutorials have been created
with the OBS Studio software, and have been published on Youtube with access only to the students' e-mail
32
addresses. This platform was chosen for its versatility and accessibility from different devices (PC, Tablets,
Smartphones, TVs...) as well as for providing higher quality viewing and for being widely known by all students,
regardless of their digital skills.
Figure 1: Sheet describing the video tutorials
Finally, after the implementation of the experience, the degree of student satisfaction was evaluated using the
evaluation instrument described in section 2.2 above. To ensure maximum participation, a 10 min slot was set
aside at the end of the last practical session where all the students were present and were asked to fill in the
questionnaire anonymously. Once the answers were collected, they were analysed to draw the results and
conclusions shown below. In addition, an evaluation was made of the overall quality of the work presented in
comparison with that of previous years, as well as an analysis of academic performance and attendance and
monitoring of the practical sessions.
3 Results
It is important to highlight the high participation of the students in answering the questionnaire, achieving 100
% participation. This was influenced by the fact that a few minutes were spent in a class in which all 18 students
were present and asked to fill in the questionnaire, emphasising its anonymous nature.
The results of the questions asked are shown below. Inquired about their opinion on the use of PBL & video
tutorials (Figure 2a), 100 % of the students considered their use to be of interest, as well as that it has helped
them in the learning process. As for the teaching methodology (Figure 2b), 89 % opt for the one followed in
this course, viewing the videos asynchronously at home before the workshop session and dedicating that hour
to progressing with the assigned work and resolving doubts. Only 2 student indicates that he/she prefers to
receive a classic synchronous explanation and to have the videos as support, while not having the video
tutorials is an option that nobody chooses.
33
Regarding the reasons why the students consider that the methodology implemented in this course (video
tutorials for the guided resolution of computer exercises and PBL) has improved the learning process (Figure
2c), all indicate as advantages of having the video tutorials that when developing the project, they have been
able to consult the explanations again, that they have been able to repeat the explanations as many times as
they have needed and that in this way it has become easier to follow the course at their own pace. 78 % percent
consider that this gives them more time to understand the exercises proposed, and only 61 % see the
advantage of the video tutorials as being able to follow the explanations if they do not attend the workshop
in person.
a) Opinion on the use of PBL + video tutorials b) Which teaching methodology do you prefer?
No Yes Watching the videos before
at home and developing the
0% 20% 40% 60% 80% 100%
11% work during the workshop
(PBL+Videotutorials).
Do you consider the use of PBL + video Developing an example in

tutorials for solving computer 100% class and doing the work at
exercises to be of interest? home without videotutorials
(traditional)
That explanations are given

Do you consider that the use of PBL + in class and that the videos
video tutorials has helped you in the 100% 89% serve as support for any
learning process?
doubts that may arise at
home (both).
c) Reasons why students consider that the PBL methodology and video tutorials have improved their learning process:
I was able to refer back to the explanations when developing the group
100%
work.
I have more time to be able to understand the exercises 61%
I can follow the explanations even if I do not attend the workshop. 78%
I have been able to repeat the explanations as many times as I needed to. 100%
It makes it easier to follow the course at my own pace. 100%
0 2 4 6 8 10 12 14 16 18
Number of responses
Figure 2: Responses on the use of video tutorials and PBL methodology
Regarding the perception of how the video tutorials influence the consolidation of the concepts and knowledge
explained (Figure 3a), 63 % consider that they influence Very, while the remaining 38 % indicate that they
influence them Extremely. In other words, all the students considered that the use of video tutorials had a
positive or very positive influence on the teaching-learning process. When asked about the aspects that they
consider most relevant to be present in the video tutorials and that help them to learn (Figure 3c), 100 % agree
that the accuracy of the explanations is the most relevant, followed by 78 % who mark the graphic design as
an aspect to be considered, and only two students indicate that the clarity language used is relevant. This
implies that the students attach importance to receiving correct and precise explanations, and also to the
graphic format of the video, whereas they dominate the technical terms used by not giving them so much
importance.
About the advantages they have observed in the use of video tutorials in the learning process (Figure 3c), the
students could choose multiple predefined answers, as well as add other advantages that they considered were
not present, which none of them did. The only answer that everyone ticked, and which is therefore shown to
be the greatest advantage of this methodology, is Being able to repeat the explanation and the commands used
as many times as one needs. This is followed by Following the pace of the class in a personalised way (89 %), and
this response corresponds to one of the findings of this research. Traditionally, teachers were faced with the
problem of having to adapt the pace of their lessons to students who had more difficulties in following them
synchronously or who had less digital skills, causing, on the one hand, that the time allocated to each lesson
34
was often scarce because they could not advance all the desired concepts and procedures, and on the other
hand, that the students were frustrated, causing a reduction in attendance and a decrease in academic
performance. At the same time, more advanced students or those with greater digital competencies saw their
learning progress delayed, with abundant time-outs that also caused the same effect of disconnection,
sometimes even generating some friction between classmates.
a) b) What aspects of the video tutorial do you

consider relevant and helpful to your learning?
Graphic design 78%
Clarity of language 11%
Accuracy of explanations 100%
0 2 4 6 8 10 12 14 16 18
Number of responses
c) What do you consider to be the advantages of using video tutorials in learning?
Repeat the explanation and the commands used as many times

100%
as you need.
Follow the pace of the lessons in a personalised way. 89%
Better visualisation of content (better image quality). 39%
Follow the explanations asynchronously, when it suits me best. 78%
Use other devices to watch the videotutorial (tablet, mobile, TV,

78%
other screen) while practising.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Number of responses
Figure 3: Responses on the advantages and influences of the proposed methodology on learning.
This negative effect has disappeared with the proposed methodology, as viewing the video tutorials allows
each student to follow the pace of the explanations they need, even if they are viewed during the workshops,
and the teacher can deal with any doubts that arise in a personalised way without slowing down the teaching
pace. 78 % of students agree that this is an advantage, since being able to view the video tutorials at any time
makes it easier to follow their own pace of learning at the time when they are most inclined. Comments not
included in the survey but communicated to the teaching staff by the students indicated that they watched the
videos on their smartphones/tablets on public transport on the way to class, during the dead time between
subjects or even while doing their housework. This possibility of viewing the video tutorials on several devices
is also pointed out as an advantage by 78 % of respondents, as it allows them to have the video tutorial on
one screen to follow the explanations and on the other screen to carry out the procedure at the same time.
Finally, the better visualisation of the content, an aspect that also used to be problematic in workshops sessions,
due to the resolution of the projector, is only indicated as an advantage by 39 % of the students.
The length of the video tutorials was rated as adequate by 72 % of the students, while 28 % considered it
excessive (Figure 4a). It should be recalled that the duration of the video tutorials was about 50 minutes,
practically the same as the face-to-face session. However, for some students this duration is often considered
excessive, as they are used to a faster audiovisual consumption, like knowledge pills (Colomo & Aguilar, 2017).
As a proposal for improvement, for future courses we will consider dividing the video tutorials into blocks of a
shorter duration (10-15 min). It will be the same content but presented in a more convenient way to find the
parts of interest for the students. Even so, 89 % would like to repeat this experience in other subjects (Figure
4b), while 11 % indicate that they might like to do so, which indicates that the methodology we have
implemented has been very well accepted.
35
a) How did you find the length of the videos? b) Would you like to work with PBL + video tutorials
in other subjects?
11%
28%
Short No
Adequate Maybe
Excessive Yes
72%
89%
c) Overall assessment
1. Very poor 2. Poor 3. Neither 4.Good 5.Very good
PBL Methodology + Videotutorials 39% 61%
Quality of explanations 50% 50%
Graphic design 61% 39%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Number of responses
Figure 4: Assessment of the experience.
This is confirmed in Figure 4c, where, when asked about the overall evaluation of the experience, i.e. the
learning based on carrying out a project to calculate the coastal vulnerability index with video tutorials that
solve the computer-based practice sessions in a guided manner, 61 % indicated that the experience was Very
Good, and 39 % that it was Good, with no negative evaluations. The quality of the explanations is rated 50 %
as Good or Very Good, while the graphic design of the tutorials is rated as Good (61 %) or Very Good (39 %).
In other words, in all cases there is a positive assessment by the students. No negative comments or proposals
for improvement have been received from them regarding the experience developed in this course.
Figure 5 show an increase in attendance (100 % in this academic year 2021-22), as well as an improvement in
the quality of the project submitted by the students at the end of the course, even considering the quality
requirements were higher than previous courses as the value of the final mark of the workshop on the overall
subject score was augmented up to 30 %. In addition, an increase in academic performance has been detected,
obtaining the best scores of the last 7 academic courses.
Attendance Workshop Subject

100%
100%
90% 95%
80% 86%
70% 72%
76%
73%
% of succeed
60% 68%
65%
62% 62% 62%
59% 58% 57% 59%
50% 52% 52% 53%
51%
40% 42% 42%
30%
20%
10%
0%
2015-16 2016-17 2017-18 2018-19 2019-20 2020-21 2021-22
Figure 5: Evolution of the percentage of attendance and success at workshop and overall subject score. Students enrolled
were 16, 17, 15, 19, 18, 16 and 18, respectively.
36
4 Conclusion
This research has investigated the different resources available offered by ICTs for the guided resolution of
computer-based practical sessions and project-based learning. Project-Based Learning (PBL) and Active
Learning methodology were implemented, creating two video tutorials for the guided resolution of the
computer sessions adapted to the subject Coastal and Oceanic Engineering of the degree in Marine Sciences
of the University of Alicante. The results obtained show an increase in attendance and engagement in the
workshops, as well as an improvement in the quality of the project submitted by the students at the end of the
course. Students pointed out that the greatest advantages of this methodology are, firstly, being able to repeat
the explanation and the commands used as many times as one needs. Secondly, following the pace of the class
in a personalised way. The PBL approach with video tutorials was also well evaluated as a help to assimilate the
concepts, procedures and attitudes included in the learning objectives of the subject. In addition, an increase
in academic performance and global students’ satisfaction has been detected. Thus, this experience can be
evaluated as a complete success, encouraging the teaching staff to maintain in the following academic years
and even expanding to other subjects where computer practice is relevant.
5 Acknowledgements
This work was partially supported by the Universidad de Alicante through the project “Desarrollo e
implementación de recursos innovadores para la resolución guiada de prácticas con ordenador que mejoren
el proceso de enseñanza-aprendizaje” (XARXES-2021-5649).
6 References
Aguila, R. C., Rodríguez, A. M. M., & Lechuga, A. O. (2013). Un recurso para la enseñanza de la Estadística: video tutoriales
para las prácticas de ordenador. Probabilidad Condicionada: Revista de didáctica de la Estadística(1), 653-654.
Alonso-González, C., Mulero, J., Nueda, M. J., Rodríguez Álvarez, M., & Vicente-Pérez, J. (2021). Creación de videotutoriales
para la iniciación al programa R de Estadística en el Grado de Óptica. In R. c. Satorre Cuerda (Ed.), Memorias del
Programa de Redes-I3CE de calidad, innovación e investigación en docencia universitaria. Convocatoria 2020-21
(pp. 3081-3101). Universitat d’Alacant.
Brame, C. J. (2016). Effective educational videos: Principles and guidelines for maximizing student learning from video
content. CBE—Life Sciences Education, 15(4), es6.
Cañete, A., Garijo, D., & Pérez, R. (2012). Enseñanza basada en contenidos: una experiencia para el desarrollo de
competencias del EEES en grados en Ingeniería Informática. Jornadas de Enseñanza de la Informática (18es: 2012:
Ciudad Real).
Colomo, E., & Aguilar, Á. I. (2017). Píldoras formativas en la educación online: posibilidades y limitaciones. In J. Ruiz-Palmero,
Sánchez-Rodríguez, J. y & E. Sánchez-Rivas (Eds.), Innovación docente y uso de las TIC en educación. (pp. 1-10).
UMA Editorial.
González, M. J., Montero, E., de Heredia, A. B., & Martínez, D. (2010, 14-16 April 2010). Integrating digital video resources
in teaching e-learning engineering courses. IEEE EDUCON 2010 Conference,
Lehmann, M., Christensen, P., Du, X., & Thrane, M. (2008). Problem-oriented and project-based learning (POPBL) as an
innovative learning strategy for sustainable development in engineering education. European journal of
engineering education, 33(3), 283-295.
Pagán, J. I., Antón Sempere, J., Aragonés, L., López, I., Tenza-Abril, A. J., Bañón, L., Navarro-González, F. J., Villacampa, Y., &
Asencio-Gil, E. (2021). Adquisición de competencias en modalidad dual por la COVID19 en las enseñanzas de
ingeniería costera y marítima. In R. c. Satorre Cuerda (Ed.), Memorias del Programa de Redes-I3CE de calidad,
innovación e investigación en docencia universitaria. Convocatoria 2020-21 (pp. 2855-2876). Universitat d’Alacant.
Zabalza, I., Peña, B., Llera, E., & Usón, S. (2016, 4–6 July 2016). Improving the teaching-learning process using educational
videos as reusable learning objects (RLO) in the field of thermal engineering. 8th International Conference on
Education and New Learning Technologies, Barcelona.
37
Strengths and dangers of self and peer assessment in
engineering learning. Teachers’ and students’ perspective
Miguel Romá1, Josep David Ballester-Berman1, Francisco P. Vives2, Juan Manuel López-Sánchez1, José Antonio
Signes2, Tomás Martinez-Marín1, Enrique Martín2, Jesús Selva1
1
Signals. Systems and Telecommunication group - SST, University of Alicante, Spain
2
Department of Physics, Systems Engineering and Signal Theory, University of Alicante, Spain
Email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected],

[email protected]
Abstract
Given the need to use competency assessment systems, a complex problem arises, especially when applying a metric to
transversal competencies is intended. In this context, it is possible to use, among other tools, self and peer assessment, so
that the first-person perspective of the students is considered among the criteria to quantify aspects that are difficult to
measure. Even though it may seem interesting, this strategy is not riskless, at least this is deduced when talking about this
topic with some colleagues. It is intended to address a reflection on the advantages of self and peer assessment within
engineering assessment, but also analysing the risks involved (lack of maturity, assigning too high grades...). This feeling of
risky activity is presented as especially dangerous when it has to be used by teachers who are reluctant to its use. Analysis
of collected assessment data and questionnaires addressed to both teachers and students will provide an insight to their
opinion about these assessment methods.
Keywords: Assessment in active learning; Self and peer assessment in engineering; Competencies assessment.
1 Introduction
A recent study by Halls et al. (2021) suggests that, after a systematic literature review, a greater focus is required
in student engagement in the assessment in engineering education. Involving students in the assessment
process may provide interesting outcomes. If students are aware of the relevance of their role in the evaluation
process, this process can become a true learning opportunity. Even at a professional level, well supported
formative feedback would be required to successfully change workplace culture (Willey & Gardner, 2007). This
change is initiated by facilitating individual self-reflection and ongoing improvement while encouraging teams
to resolve team issues independently. Maybe the most obvious ways to generate this involvement are self and
peer assessment.
As stated in (Boud & Falchikov, 1989) self-assessment refers to the involvement of learners in making
judgements about their own learning, particularly about their achievements and the outcomes of their learning
and is mostly used for formative assessment to foster reflection on one's own learning processes and results
(Dochy et al., 1999). Falchikov (1995) defines peer assessment as the process through which groups of
individuals rate their peers. This exercise may or may not entail previous discussion or agreement over criteria.
It may involve the use of rating instruments or checklists which have been designed by others before the peer
assessment exercise or designed by the user group to meet its particular needs (Dochy et al., 1999).
Besides, both students and teachers find risks in the use of self and peer assessment. Some of the potential
risks, that are listed in (Tennant & Crawford, 2019), include the reaction of colleagues and external examiners,
demands of time, its robustness, and reliability. Students themselves tends to feel confident in traditional
summative assessment methods as unequivocally valid assessment mechanisms (Fernandes et al., 2012),
considering summative assessment is in the centre of their concerns and as one of the most important results
of their learning process.
There are, notwithstanding, studies that conclude that this active role of students in the assessment process is
reliable and fair and contributes to a growth in competence (Dochy et al., 1999), even for first year engineering
students (Van Hattum-Janssen & Lourenço, 2008).
38
Summarizing, some of the commonly cited benefits and risks of self and peer assessment for teachers and
students are listed in table 1.
Table 1. Benefits and risks of self and peer assessment (literature review).
Implication Teacher Student
Improves professional practice in relationship Improves learning

with assessment
Increases opportunities to monitor student Supports generic skill development

progress and identify potential problems
Benefits
Strengthens assessment criteria and marking Enhances motivation and enthusiasm
schemes
Promotes student-centred learning Increases level of feedback
Reactions of Colleagues and External Belief in teacher-centred assessment

Examiners
Time consuming Time consuming
Risks Robustness of Assessment Feeling that traditional summative

assessment is more objective
Reliability Fears about the impartiality of other

students
Lack of confidence in students’ fairness Relationship with other students if

assessment is not favourable
To be honest, when considering that the fact that self and peer assessment can show a lack of objectivity is
perceived as a risk for both parts, we are not much worried about this fact. Moreover, some of the members
of the network are defendants of subjective assessment as presented in a past work (Romá, 2007).
2 Context
This work is being developed under the framework of a network project of the University of Alicante pursuing
improvement of quality and innovation in higher education by means of specific learning experiences. The
main goal of this work is to explore the risks and strengths of self and peer assessment from students’ and
teachers’ perspective together with their feelings in front of that strengths and dangers from a local context,
as explained in following sections. Besides, some data is going to be summarized from courses that use, among
others, self and peer assessment as a part of their evaluation methods.
In sections 2.1 and 2.2 the context in which the study is being conducted is presented.
2.1 Courses
To analyse the results obtained by the use of self and peer assessment, data from five different courses using
these types of assessment in different schemes is going to be used. These are courses from Sound and Image
in Telecommunication Engineering (SITE) and Biomedical Engineering (BE) degrees in the University of Alicante
Polytechnic School.
39
Table 2. Use of self and peer assessment in the courses under study.
Course Year Self Peer Short description
Basic Electronics 1st 5% N. A. General opinion about course and self-performance
Biomedical Signal 3rd 5% N. A. General opinion about course and self-performance

Processing
Video Engineering 3rd 35% 13% Peer assessment of projects and self-assessment of self-
performance (with special focus on transversal skills)
Advanced Audio-visual 4th N. A. 40% Peer assessment of research project and literature
Systems review)
Audio-visual 4th 35% N. A. General opinion about course and self-performance

Production Facilities
Self and peer assessment are used with different objectives and also with different effects in the final marks of
each course. Table 2 summarizes how self and peer assessment is used and their weights in the final course
marks. The first course in which self and peer assessment was introduced is Video Engineering (in 2008) and
the last one Biomedical Signal Processing (in 2016). The weight in which self and peer assessment affects to
the final mark varies from 5% to almost 50% depending on the course design.
Shortly, in first year courses self-assessment is used as a final reflection activity with a modest impact in the
final mark. This process is done around two questions, what are the benefits of taking this course and which
skills have I developed. In 3rd and 4th courses the self-assessment is used goal-based evaluation in with students
have to analyse the degree of fulfilment of transversal skills as stated in course goals.
Using peer-assessment in PBL is interesting as it gives provides students a tool to put their own work in context
(especially if all the working teams are trying to solve the same, or at least a very similar, problem.
The data will be used to check if there is correlation between student and teacher appreciation of learning
outcomes, and to analyse if students tend to over-grade themselves. As students are asked to deliver justified
reports to support the proposed marks, interesting information can be obtained from these reports.
2.2 Perspective
The sources of information to gather teachers’ and students’ perspective in front of self and peer assessment
have been some face-to-face informal interviews (with teachers) and questionnaires sent to both students and
teachers. The justified reports mentioned in the previous section will also be used with the purpose of gathering
information about students’ opinion.
The interviews have been used to collect the general feeling of some colleagues (both using and not using self
and peer assessment in their courses). As the questions were intentionally not previously prepared, the goal of
these meetings was focused on getting subjective information rather than measurable data.
Students’ questionnaire has been sent (sharing a google forms link) mainly to students who are actually
enrolled in the courses mentioned in Table 1. The questions have also been sent to students from the Master´s
degree on Education Research. No personal information is stored to ensuring the anonymity of the
respondents. At the moment of writing this report, 60 students have fulfilled the form. Although there is no
way to know if the answers come from undergraduate or master’s degree students, this is not a key data in the
study as we were not looking for correlation between answers and students’ affiliation. The first intention was
to submit the question only to engineering students, but we finally decided to also include students from the
Education Research program. The questionnaire is composed of 23 multiple choice questions (6 about
assessment in general, 9 about self-assessment and 8 about peer assessment), and one open answer question.
A similar form has been sent to teachers from the Polytechnic School and the Faculty of Education, with the
following structure: 7 multiple choice questions about assessment in general, 5 about self-assessment, 4 about
students’ position in front of self-assessment, 5 about peer assessment, 5 about students’ position in front of
40
peer assessment, 2 about final thoughts and one open answer question. Unfortunately, although not
surprisingly, only 10 people have responded to the form at this moment.
3 Teachers’ perspective
With only 10 people answering the questionnaire, there is, obviously, no possibility of getting statistics from
the data. Notwithstanding, some insights can be obtained from teachers’ opinion that can be correlated with
the answers in the informal meetings. To put answers in perspective, almost any of them had made research
or training on self or peer assessment and its effect on learning outcomes.
One of the most cited risks (and reasons supporting the decision of not using self and peer assessment) is its
lack of reliability or, in other words, not being objective. In fact, only 20% of answers show tendency for
subjective assessment, while the 80% rely on the idea of having an objective assessment method. Surprisingly,
30% of the teachers are not able to say if the assessment methods they are using are summative or formative
ones. There is a higher than desirable number of teachers who is still evaluating in the same way they were
evaluated without considering other options. As can be seen in Figure 1 most of them have never used self or
peer assessment in their courses, so some of the opinions expressed may not being supported by experience.
Figure 1. Use of self and peer assessment.
In general, teachers tend to think that students will over-rate themselves when using self-assessment (80%).
However, most of teachers think that students are mature enough to carry on with this process (90%). When
considering peer assessment there is a slightly lower level of over-rating sensation (70%), while 100% consider
that students are able to evaluate their peers in a mature and thoughtful way.
Asking teachers about under what circumstances they will choose self and/or peer assessment in their courses,
some interesting answers are listed:
• Reduced number of students and over third year.
• Course content not being too technical (more descriptive).
• I would have to feel that these strategies help student development.
• I would use them if I was responsible of the course or having the support of course responsible.
• Not having to worry about others’ opinion.
Our general perception is that teachers who are reluctant to the use of self and peer assessment show a
tendency to think that students will not evaluate themselves fairly, even though they consider students to be
mature enough to take the assessment responsibility. At the same time these teachers are not aware of the
learning outcomes that self and peer assessment can provide.
4 Students’ perspective
In this section some of the students’ answers to the questionnaire are going to be highlighted. To state a
starting point, the answers about the preferred assessment model is presented in figure 2.
41
Figure 2. Preferred assessment model for students.
It is remarkable that near 50% of students don’t know the difference between assessment models. If we want
them to be engaged in the learning process and confident in the way they are evaluated, maybe an effort must
be done in letting them know about the assessment strategies.
When students are asked if self or peer assessment must have a considerable weight in the final mark, similar
results are found when comparing self and peer assessment (Figure 3). It will be interesting to figure out why
over 20% of students fully disagree with the use of self or peer assessment as part of the final mark.
Figure 3. Students’ opinion about using self and peer assessment with a considerable weight in the final mark.
Most of the students are aware that proposing self-assessment implicitly shows a high level of confident in
themselves, as can be seen in figure 4.
Figure 4. Students’ awareness of the confident needed to propose self-assessment.
When considering one of the fears expressed by teachers in relation with the tendency of students to over-
rate themselves, students’ opinion is not the same (figure 5).
42
Figure 5. Students’ awareness of the confident needed to propose self-assessment.
Asking students about under what circumstances they think self and/or peer assessment should be used, some
interesting answers are listed:
• There must be a proper rubric to be used as an assessment guide.
• If self-assessment is used, everyone would be more engaged and active in classroom sessions.
• It will be interesting if the teacher reviews the marks of self and peer assessment.
• Inter-personal relationships can be an issue in peer assessment. The classroom mood must be good
and relaxed.
• In self-evaluation we can be aware, and learn, from our own mistakes. In peer assessment we can learn
from the work done by other students.
• It would be good for students with a well-measured self-criticism. The problem can arise in students
who undervalue themselves, and vice versa, students who are more narcissistic who feel they deserve
a higher grade than they should be.
Summarizing, students consider themselves as ‘fair players’ concerning self and peer assessment. With little
instruction, they are fully aware about the learning possibilities of this kind of assessment strategy. To ensure
they will be confident in this assessment, special care must be taken to ensure personal relationships are not
being considered in the assessment. When asking them for their opinion about self and peer assessment, the
answers that have been obtained tend to be more mature than expected.
5 Analysis of assessment data

In this section we are going to analyse if students’ and teachers’ opinion about self and peer assessment are
correlated with the data obtained from courses using these types of assessment. Of all the courses presented
in Section 2.1, a couple of examples is going to be presented together with some general thoughts.
The way in which self-assessment is addressed is different depending on the course. For first year students, it
is necessary to be more directive in the way the instructions are provided to the students. In the case of the
Basic Electronics (first year) course, the self-assessment is weighted with a 5% in the final mark. Students are
asked to prepare a justified report and a proposed mark, based on the learning goals of the course structured
in two questions:
• What skills have I developed by taking this course?
• What are the benefits of having taken this course?
The difference between the self-assessment mark and the final mark (0-10 scale) is displayed for about 750
students between 2011-12 and 2020-21 academic years, together with the average of this difference per year
is presented in Figure 6.
43
Figure 6. Numeric difference between self-assessment and final mark of a first year engineering course. Pink bars indicate
the average difference per year.
Even this data come from first year students, which are supposed to be less mature, there is clear consistency
in the results. There is a quite stable offset of two points between the marks. The person responsible of the
course is quite satisfied with the self-assessment reports and marks. His opinion of this two points’ difference
is a mix between a slight over-rating and an evaluation scheme that can be improved.
Some results of peer assessment can be obtained from the 3rd year Video engineering course. In this course,
peer assessment is used as the main tool in the process of evaluation of projects. Figure 7 presents the
difference between peer assessments and teacher marks for the 21 working groups in a given year. The data
displayed corresponds with the average of all the marks assigned to each one of the working teams related
with one of the projects developed.
In this case it is interesting to consider that, contrary to reluctant teachers’ opinion, there is not a general
tendency to over-rate if the teacher mark is taken as a reference. Not only is there a high degree of correlation
but in half of the cases, students are marking their peers with a more severe criterion. Although Figure 7 depicts
data from one year, the tendency we have obtained is similar in almost every year. Little exceptions have been
found corresponding with years with reduced number of students (and, logically reduced number of working
teams).
Figure 7. Comparison between peer assessment and teacher’s marks from a 3rd year engineering course.
The last behaviour to explore is the relationship between the self-perception (self-assessment mark) and the
marks assigned to the rest of the teams (peer assessment mark). Figure 8 represents the dispersion comparing
the difference between self-assessment mark with the average of the received peer assessment marks (SA-PAr)
with the difference between assigned peer assessment and the average mark of all the peer assessments (PAa-
PAavg) for the same 21 teams of Figure 7. The data have been normalized in a ±1 range.
44
Figure 8. Relationship between self-perception and assigned peer assessment from a 3rd year engineering course.
Figure 8 shows a clear trend. The groups with the highest level of self-demand (SA-PAr<0) have a tendency to
rate the rest of the groups with higher grades (PAa-PAavg>0). Likewise, groups that rate themselves with higher
grades (SA-PAr>0) tend to be more demanding when rating the rest of the teams (PAa-PAavg<0). The global
reference provided by the implementation of peer assessment generates a consistent marking reference that
it used in the self-assessment when both strategies are used simultaneously.
6 Conclusion
Learning outcomes associated to self and peer assessment have been widely stated in bibliography. However,
it is easy to find teachers who are reluctant to the use of these assessment methods. Besides, a considerable
number of students don’t know the difference between formative and summative assessment. Some effort
must be done in training on assessment strategies addressed to both teachers and students to help them be
confident in ‘non-traditional’ assessment methods.
According to teachers’ perspective, the main risks associated to self and peer assessment are the lack of
reliability, not being confident in student criteria and how to justify evaluation results to other colleagues or
external examiner.
One of the most cited fears by students is related with the risk of being assessed by peers biased by personal
relationships. Apart from this, students are quite receptive to the use of self and peer assessment showing a
clear idea (or intuition) about their benefits.
Acceptable levels of correlation between self and teacher’s assessment has been supported by the data
analysed. A quite consistent behaviour has been observed. Using peer assessment to evaluate projects, with
all working groups solving the same project, has delivered high levels of correlation with teachers’ opinion. If
self-assessment is used in the same framework, the reference of peers’ work becomes a solid basement for the
self-assessment.
7 Acknowledgment
This work has been supported by a grant from the University Teaching Research Networks Program of the
Institute of Education Sciences of the University of Alicante (call for proposals 2021-22). Ref.: [5480].
8 References
Boud, D. & Falchikov, N., (1989). Quantitative studies of self-assessment in higher education: a critical analysis of findings, Higher Education,
18, pp. 529-549.
Dochy, F., Segers, M. & Sluijsmans, D., (1999). The use of self-, peer and coassessment in higher education: A review, Studies in Higher
Education, 24:3, 331-350, DOI:10.1080/03075079912331379935
45
Falchikov, N. (1995). Peer feedback marking: developing peer assessment, Innovations in Education and Training International, 32, pp. 175-
187.
Fernandes, S., Flores, M. A. & Lima, R. M., (2012). Students’ views of assessment in project-led engineering education: findings from a case
study in Portugal, Assessment & Evaluation in Higher Education, 37:2, 163-178, DOI: 10.1080/02602938.2010.515015
Halls, J. G., Tomás, C., Owen, J. S. & Hawwash, K. (2021). Mapping out the landscape of literature on assessment in engineering education,
European Journal of Engineering Education, DOI: 10.1080/03043797.2021.2009775
Romá, M. (2007). Does it make sense to look for an objective assessment in PBL? Proceedings of the 7th international ALE workshop,
Toulouse, France, June 2007.
Tennant, J., Crawford, A. (2019). WebPA Tool: Supplementary information about self and peer assessment. Loughborough University.
Available on-line at: https://webpaproject.lboro.ac.uk/academic-guidance/supplementary-information-about-self-and-peer-
assessment/. Last accessed April 2022.
Van Hattum-Janssen, N., Lourenço, J. M. (2008). Peer and self-assessment for first year students as a tool to improve learning. Journal of
professional issues in engineering education and practise. October 2008, 346-352 DOI: 10.1061/(ASCE)1052-3928(2008)134:4(346)
Willey, K. & Gardner, A., (2007). Building better teams at work using self and peer assessment practices. Proceedings of the 2007 AaeE
Conference, Melbourne.
46
Project Based Learning Approach in the Heat Transfer Course
for Undergraduate Students
Victor A. S. M. Paiva1, Caio F. R. Santos1
1
Insper Instituto de Ensino e Pesquisa, São Paulo, Brazil
Email: [email protected], [email protected]
Abstract
Computer engineering courses often require knowledge from other fields, such as numerical methods and heat transfer. It
is challenging to motivate students and show them how the contents presented in these units are connected to the core
of their course. This paper presents a project based learning experience for third year undergraduate students of computer
engineering at Insper Instituto de Ensino e Pesquisa. The students are required to work in teams to design a cooling system
to a processor in a free version of the finite element software LISA. The theme of the project builds the bridge between
heat transfer and computer engineering and the tool shows the importance of numerical methods in engineering. Teams
are free to select any processor available in the market and choose fictitious parts, such as fans and fins, from a list provided
by the instructor. Each item in the list is associated with a price, so they must propose and simulate numerous designs to
optimize their final cost. Their designs are initially based on observation of the cooling systems available in computers and
smartphones and then improved by their heat transfer knowledge. Students present their solutions to sell their designs in
the end of the project. This experience could be adapted into a contest for the lowest cost if a single processor was provided
to all groups. A contest can motivate the students, but may also worry them, while a free choice has been successful in
making them active and keeping a light environment. This experience can be accomplished remotely without expensive
software and create a bond between heat transfer, numerical methods and computer engineering.
Keywords: Project Based Learning; Heat Transfer; Numerical Methods; Engineering project.
1 Introduction
Many engineering courses have heat transfer and numerical methods units in their programme, but the
importance of those units may not be clear to the students. For example, a computer or electrical engineering
undergraduate may not see a straight connection between heat transfer and the core of their engineering
courses. Also, the role of the engineer has changed lately and companies have been requiring their specialists
to have a broader understanding of the projects (Santos, Ayres, & Miranda, 2018), not only regarding technical
aspects but also about entrepreneurship, design thinking (Goldberg & Somerville, 2014) and soft skills (Kumar
& Hsiao, 2007). Therefore, it is necessary to motivate engineering students to acquire knowledge from other
fields.
Active learning (Christie & Graaf, 2017) strategies and project based learning (Kokotsaki, Menzies, & Wiggins,
2016) approaches have been widely used in engineering courses, especially to develop interdisciplinary tasks.
The laboratory was transformed in an escape room game by (de la Flor, Calles, Espada, & Rodríguez, 2020),
where chemical engineering students needed to answer heat transfer questions correctly to win. A sequence
of tasks under the PBL methodology was designed by (Montero & González, 2009) to first year electronic
engineering students, but their objectives did not include numerical methods. A PBL approach was suggested
by (Zhuge & Mills, 2009) to motivate students to learn basic concepts of the finite element method. They
focused on structural analysis and used a commercial software to develop their models. In this work, numerical
methods and heat transfer are combined in a 1 week project under the PBL approach. Each team of students
must select an existent processor, design a cooler by assembling parts from a list and use a finite element
commercial software to simulate its performance.
This project was designed for third year computer engineering students, but the methodology can be applied
to most engineering modules. Either as an introductory activity in courses that have heat transfer as a core unit
(mechanical engineering, for example) or as a complementary assignment to other engineering curriculums.
47
This project does not require prior knowledge related to the finite element method or any commercial software,
but students will have to analyse and discuss their results, therefore a reference containing such information
has to be indicated.
2 Objectives and expectations

In this project, teams are free to select any processor and design the heat fins as they prefer. We expect this to
enhance the engagement of the students in the project because they can choose a processor they have in their
computers, mobile phones, videogame consoles, etc. Also, the solution space on heat fin designs is too wide,
therefore teams that discuss more and work creatively tend to perform better in such an open-ended situation.
Many undergraduate engineering projects focus too much on the technical aspects and leave the cost out of
the study. By including this aspect, we expect the students begin to consider the price of the components in
their thoughts more often.
The main learning objectives of this project are:
● Research technical data of processor units

● Apply heat transfer knowledge (conduction and convection) in processor cooling systems
● Discuss and create potential configurations of coolers
● Solve heat transfer analyses in a finite element analysis software
● Understand concepts behind numerical solutions (finite element method)
● Eliminate and/or improve inefficient cooling systems based on heat transfer knowledge and simulation
outputs
● Find a configuration that satisfies the maximum operation temperature of the processor
● Design a brochure with fundamental information on the final product
We expect this project to introduce the theoretical studies on conduction, convection and numerical methods.
Depending on the course, it should be followed by a deeper study of the heat transfer governing equations
and numerical methods algorithms.
3 Description of the activity

This project can be divided in four main stages: selecting a processor, assembling a cooling system from
fictitious parts available in the store, using a finite element software to improve the system and presenting a
brochure with the optimized configuration. The due date is within a week from publication of the assignment
text.
3.1 Selecting a processor

Students are free to select any real processor, but their main technical data will be needed in the next stages
of the project, therefore we encourage them to pick a processing unit whose specifications can be easily found.
Most teams select AMD or Intel cores due to their popularity and availability of information, but a few students
have explored smartphone processors and graphics processing units (GPU).
3.2 Assembling a cooling system

Instructors provide a list of fictitious parts and their respective cost (Table 1). Students can assemble their
coolers by selecting a fan (if any), build a dissipator base to place heatfins on and assign one or more materials
to their systems.
48
Table 1. Fictitious hardware store. Material properties retrieved from (Incropera, Dewitt, Bergman, & Lavine, 2007).
Price ($) Hardware
30 Breeze fan (h=30W/m²K)
60 Gust fan (h=50W/m²K)
120 Cyclone fan (h=80W/m²K)
5/heatfin Heatfin manufacturing
0.0005/mm³ Aluminum (k=237W/mK)
0.02/mm³ Copper (k=401W/mK)
3.3 Using a finite element software

To quantify the performance of their coolers, students must model the assemblies in a finite element software.
They are encouraged to use the free version of the LISA FEA software (Figure 1) (Sonnenhof Holdings, 2011),
as it is an easy to learn tool and has enough resources to fulfil the objectives of this project. A video tutorial
was provided by the instructors.
Figure 1. Example of a heat transfer analysis in the LISA FEA software of a cooling system. Temperatures in K.
There is a maximum limit of nodes in the free version of the software, which can be seen as an opportunity to
create a sense of waste of resources. This limitation helps the students learn to refine their meshes around
critical areas only, and not on the whole model.
The processor core is modelled as a heat flow rate on the bottom of the dissipator. Its Thermal Design Power
(TDP) and dimensions depend on which processor was chosen by the team. The design of the dissipator base
and fins are free, but the materials must be listed in the store. The exposed portion of the base is subject to
free convection, while the fins are subject to forced convection, whose coefficient h depends on the fan the
team has selected from the list. They can also decide to not use any fan (which is mandatory for mobile
processors, for example), in this case, fins are also subject to free convection. The main technical objective is to
keep the highest temperature below the maximum operating temperature of the processor while minimizing
the total cost of the system. Therefore, a solid discussion and a few trial-and-error iterations are required to
obtain an improved solution.
3.4 Presenting a brochure

After analysing potential cooling systems, deciding on a better configuration and optimizing it, each team must
create a brochure and pitch their cooler to the teachers. The brochure has to contain the processor’s name,
TDP and junction temperature, simulation parameters and outputs, final cost and all relevant information about
the cooling system.
49
3.5 Assessment rubrics
To make the grading process more objective and accurate, a few topics are assessed and the grades are
calculated under the rubrics Invalid source specified. presented in Table 2. The main topics to be assessed
are:
1. Brochure: showed technical data of the selected processor unit

2. Brochure: showed a finite element analysis (graph/figure) of their design
3. Brochure: satisfied the maximum operation temperature of the processor
4. Pitch: mentioned previous configurations of coolers
5. Pitch: discussed heat transfer processes and their importance on the design
Table 2. Assessment rubrics.
Incomplete (0%) Developing (25%) Essential (50%) Proficient (75%) Distinction (100%)
Items 1 or 2 are Cleared only items 1 Cleared only items 1, Cleared only items 1,
Cleared all items.
incomplete. and 2. 2 and 3. 2, 3 and 4.
4 Student submissions
This section contains two examples of student submissions we found to be interesting. In the first example
(Figure 1), all mandatory information is present in the brochure and the team managed to keep the highest
temperature within the limit specified by the manufacturer. Even though their design could have been further
optimized, e.g. removing cold spots from the external fins to save material, it is safe to state that the
requirements of the assignment have been attended. Contrarily, the second example (Figure 2) lacks the final
cost of the system, also their fins are poorly distributed as the hottest region of the dissipator does not have a
fin and the fins on the corners are cold. This kind of submission suggests the team knows how to operate the
software and understood the main objectives, but their learning experience would benefit from a richer
discussion and more trials.
Figure 2. Student submission 1.
50
Figure 3. Student submission 2.
5 Performance of the students

Only 50% of the teams presented a brochure in the end, the other groups organized the results in different
formats. The proposal of using a brochure to advertise their coolers must be reinforced in a future iteration of
this project. Additionally, during the group discussions, a few students wanted to explore different processor
units but did not manage to convince their partners. If the teams had been divided according to processor
preference, maybe more diversity of coolers would have appeared. Only one team chose a mobile Snapdragon
processor while all other groups selected regular CPU from Intel or AMD.
However, all teams developed cooling systems that satisfied the maximum temperature allowed by the
manufacturer, which indicates they understood the basic objective of the project and all of them obtained at
least a 5/10 grade. Around 30% of the teams truly dived in the problem, presented fine results and obtained a
maximum 10/10 grade.
6 Teachers' perception
The first designs of the students were mostly based on observation of coolers and slight improves on the
tutorial we provided. But many groups managed to discuss and provide better results based on heat transfer
knowledge. The activity can run remotely or in person as long as the teachers can enter the group discussions
to hear and eventually steer the conversation into fruitful topics. Also, each team has access to at least one
computer. We found it may become too demanding for a single teacher to interact at a personal level with the
teams if there are more than 20 students in the class, it is ideal to have at least two instructors to follow the
group talks in that case.
If all teams were forced to use the same processor, this project could be transformed in a contest where the
team with the lowest price cooler (that satisfies the maximum temperature) would win. The competition would
favour the optimization step and could extrinsically motivate a few groups, therefore it is worthy being
considered, but we felt most of the students were already intrinsically motivated by the free choice of processor.
The non-competitive atmosphere foments cooperation between groups, which enhances the learning
51
experience. Additionally, in a scenario where all groups have the same processor, plagiarism would become a
concern for the teachers and among teams.
7 Conclusion
An open-ended PBL activity on heat transfer and numerical methods has been presented. The experience was
executed on a computer engineering class at Insper Instituto de Ensino e Pesquisa, but any undergraduate
engineering course can run this project. Students worked in teams to develop a processor cooler using parts
from a fictitious store and a finite element software. The final handout was a brochure containing information
about their designs, two student submissions were presented as examples. The first example may not be
optimal, but it surely had been improved to a certain extent and attends the expectations. The second example
indicates the team could have discussed better and tried more configurations to show a more efficient cooler.
A table for assessment rubrics is presented to increase the accuracy and objectivity of the grading process.
Future steps involve reinforcing to the students the value of the brochure format, applying the project
methodology on students from other engineering modules and running a questionnaire to accurately assess
the students’ perception. From personal interaction with the teams, the project was well accepted by most of
the students, who understood the importance of heat transfer concepts, teamwork and numerical methods to
engineers of all fields.
8 References
Christie, M., & Graaf, E. (2017). The philosophical and pedagogical underpinnings of Active Learning in Engineering Education. European
Journal of Engineering Education, 5-16.
de la Flor, D., Calles, J., Espada, J., & Rodríguez, R. (2020). Application of escape lab-room to heat transfer evaluation for chemical engineers.
Education for Chemical Engineers, 9-16.
Goldberg, D., & Somerville, M. (2014). A whole new engineer. The coming revolution in Engineering Education.
Incropera, F., Dewitt, D., Bergman, T., & Lavine, A. (2007). Fundamentals of Heat and Mass Transfer. John Wiley & Sons.
Intel corporation. (s.d.). Acesso em 01 de May de 2022, disponível em https://ark.intel.com/
Kokotsaki, D., Menzies, V., & Wiggins, A. (2016). Project-based learning: A review of the literature. Improving Schools, 267-277.
Kumar, S., & Hsiao, J. K. (2007). Engineers Learn “Soft Skills the Hard Way”: Planting a Seed of Leadership in Engineering Classes. Leadership
and management in engineering, 18-23.
Malini Reddy, Y., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 435-448.
Montero, E., & González, M. (2009). Student Engagement in a Structured Problem-Based Approach to Learning: A First-Year Electronic
Engineering Study Module on Heat Transfer. IEEE Transactions on Education, 214-221.
Santos, C. F., Ayres, F., & Miranda, F. (2018). Desenvolvimento de Software como motivador para Mecânica dos Sólidos. Proceedings of the
PAEE/ALE’2018, 10th International Symposium Project Approaches in Engineering Education, PAEE, and 15th Workshop on Active
Learning in Engineering Education, ALE (pp. 825-831). University of Minho.
Sonnenhof Holdings. (2011). LISA 8.0.0. Retrieved May 01, 2022, from https://www.lisafea.com/
Zhuge, Y., & Mills, J. (2009). Teaching Finite Element Modelling at the Undergraduate Level: A PBL Approach. Proceedings of the 20th
Annual Conference for the Australasian Association for Engineering Education (AaeE 2009) (pp. 105-110). University of Adelaide.
52
Do Brazilian Engineering Professors Do Engineering Education
Research in Active Learning? A Case Study
Dianne Magalhães Viana1, Valquíria Villas-Boas2
1
University of Brasília, Brasília, Brazil
2
University of Caxias do Sul, Caxias do Sul, Brazil
Abstract
While in countries like Brazil, the concern is still the training of instructors in the use of active learning strategies and
methods, in countries such as the United States, Denmark, Sweden, Australia, for at least two decades, research has been
placed at the service of teaching and teacher professional development and projects are developed so that research in
Engineering Education is carried out with greater rigor. In this context, this research work aims to evaluate a set of
interventions presented at the 43rd edition of the Brazilian Congress in Engineering Education (COBENGE), from the year
2015, because of the theme of the event: “Active Learning: Collaborative Engineers for a Competitive World. Thirty-six
papers presented in this edition of COBENGE and that were related to active learning were analyzed to determine whether
they are papers that address interventions such as an effective teaching (ET), a scholarly teaching (ST), a scholarship of
teaching and learning (SoTL) or an engineering education research (EER) work. Our results show that from the 36 papers
analyzed 25 are ET, 1 is ST, 2 are SoTL, 4 are EER works, and 4 could not be characterized in any of the levels. We present
and discuss the approach built for the analysis performed based on the characteristics pointed out by the literature for ET,
ST, SoTL and EER. It is worth pointing out that ST and SoTL are concepts little known to the Brazilian EE community. This
research work is a starting point to evaluate the state of engineering education research in Active Learning in Brazil.
Keywords: Active Learning; Effective Teaching; Scholarly Teaching; Scholarship of Teaching and Learning; Engineering
Education Research.
1 Introduction
According to de Graaff and Christensen (2004), active learning (AL) and Engineering Education (EE) are a perfect
match, and thus engineering professors can be expected to employ active learning strategies and methods in
their classes and courses. After all, the engineer must be educated to design and create solutions to practical
real-world problems. Over the past two decades, research has shown that students prefer classes with active
learning strategies and methods over traditional classes and that the adoption of AL instructional practices in
STEM courses results in improvements in student learning, contributes to decrease dropout rates and reduce
the performance gap between different student populations (McConnell et al., 2017; Freeman et al., 2014). In
these research works, student performance was also evaluated and it was found that many AL strategies are
comparable to traditional classes in promoting content mastery, but superior to promoting student skills and
competence development (Freeman et al., 2014; Kay & MacDonald, 2016).
Unfortunately, even today, in many engineering schools in Brazil, the dominant pedagogy remains “chalk and
talk” and curricula with a few project courses and laboratory practices, which are not enough to develop the
skills desired by the world of work. In Brazil, both the possibilities and the challenges related to EE are debated
by the participants of the Brazilian Congress in Engineering Education (COBENGE), which is considered the
most important forum for reflection on EE in Brazil. The event is annually promoted by the Brazilian Association
of Engineering Education (ABENGE), with the objective of bringing together schools and instructors to, together
with government agencies and other entities interested in EE, share experiences, promote debates and propose
strategies to train professionals increasingly more qualified to meet the country's needs.
Considering the obvious need for stronger actions regarding the creation of engineering programs that
prepare engineers in tune with the problems of the 21st century, the Active Learning Working Group in
53
Engineering Education (GTAAEE1) was created during COBENGE 2014. The GTAAEE has as main objectives: to
disseminate the knowledge generated in active learning in engineering education (ALEE); to promote the
creation of a research network in ALEE in Brazil; to foster the formalization of research groups in the area, in
the most diverse engineering schools; and to promote the training of teachers to design AL environments
(GTAAEE, 2022).
In 2019, seeking to update data on the use of AL strategies and methods in Brazil, previously surveyed by
Oliveira, Pinto & Santos (2017), some GTAAEE researchers presented a study, from a systematic mapping, of
articles published in the proceedings of COBENGE on AL strategies and methods from 2007 to 2019 (Pinto et
al., 2019). The results obtained have shown an exponential growth in the number of publications of ALEE-
related articles in COBENGE. This reveals the increasingly intense search by instructors and researchers for a
teaching process focused on the development of skills and competencies.
In countries such as the United States, Denmark, Sweden, Australia, EE and research in EE are very advanced.
For example, while in countries like Brazil, the concern is still the training of instructors in the use of AL strategies
and methods, in the United States and in the aforementioned countries, for at least two decades, research has
been placed at the service of teaching and teacher professional development and projects are developed so
that research in EE is carried out with greater rigor (Shuman, & Besterfield-Sacre, 2019; Borrego & Bernhard,
2011; Streveler, Borrego & Smith, 2007).
Also, in 2019, the new National Curriculum Guidelines (BRASIL, 2019) for Engineering programs in Brazil were
published, and it is believed that their implementation is a unique opportunity for more programs to have their
curricula designed for competency-based training once and for all, which inevitably demands the incorporation
of AL strategies and methods so that these curricula are, in fact, executed in their essence.
Allied to this, one must consider that, researching and reflecting on what one does in the classroom and then
sharing it with peers is not a common practice among engineering professors. One of the reasons why this
kind of research is often underestimated is that EE is not considered a research area in Brazil. Although the
professors are teachers, they are not researchers in teaching and learning. For the most part, they are not
trained in pedagogical methods, or self-identified with the professional community dedicated to research in
education. Although the engineering professor may wish to engage in educational research, more often than
not, he or she gets frustrated and gives up because they have no idea where to start. This becomes clear in the
evaluation process of the papers at COBENGE. The great majority of the papers presented at the event do not
present well-defined educational objectives, nor research objectives, and some do not even mention the term
"engineering education".
In this context, and based on the objectives of GTAAEE, researchers of the group started a research, whose
main objective was beyond the survey of the number of papers related to the application of AL strategies and
methods, to determine the nature of the pedagogical interventions reported in the papers presented at
COBENGE and to determine how much of these papers are research papers, minimally rigorous, in EE. Thus,
this research paper aims to evaluate a set of interventions presented at the 43rd edition of COBENGE, held in
the year 2015, due to the event's theme: "Active Learning: Collaborative Engineers for a Competitive World”, a
suitable starting point to evaluate the state of engineering education research in Active Learning in Brazil.
Thirty-six papers presented at this edition of COBENGE, and which were related to AL, were analyzed to
determine whether they are papers that address interventions such as effective teaching (ET), scholarly teaching
(ST), scholarship of teaching and learning (SoTL), or research in engineering education (EER).
It is important to point out that research in engineering education starts with experiences of effective teaching
(ET) and as the research becomes more mature, it progresses to ST, SoTL and EER. In other words, we can
consider that ET, ST and SoTL are progressive levels of maturity of EER.
The following topics are presented in this article: a brief theoretical framework for ET, ST, SoTL and EER, the
research methodology employed, the results and some final considerations.
1
in this paper, the acronym used will be the acronym in Portuguese.
54
2 Theoretical Framework
In this section, the goal is to distinguish reports of pedagogical experiences from research associated with
pedagogical interventions and from more complex research in EE. To this end, a brief description of the EER
levels named ET, ST, SoTL will be given, and the basic principles of more complex EER will be presented.
2.1 ET (Effective Teaching)

An Effective Teaching intervention requires the use of good content and good teaching methods and tends to
promote desirable learning outcomes. Kipper & Rüütmann (2012) point out models and strategies for Effective
teaching in order to provide successful learning and the development of critical thinking in engineering
students. They agree with Eggen and Kauchak (2006) that mastery of four types of knowledge by instructors is
essential: (i) knowledge of content - what to teach; (ii) pedagogical content knowledge - how to teach a certain
content; (iii) general pedagogical knowledge – understanding of general principles of instruction and classroom
management; and (iv) knowledge of learners and learning – instructor’s ability to adapt their instruction based
on what learners already know. Thus, when the intent is to characterize an Effective Teaching intervention,
these elements should be identified in the teaching practice, through an assessment of student satisfaction,
peer observation appraisal, self-reflection, among others.
2.2 ST (Scholarly Teaching)

A Scholarly Teaching intervention is nothing more than academic teaching. It implies the adoption of an
academic approach to teaching, just as it is adopted for other areas of knowledge. It requires research and
includes reflection. The instructor needs to have expertise in teaching and learning, to experiment with new
pedagogical strategies and methods, to study and apply the literature on teaching and learning in the courses,
to reflect on his/her teaching. According to Richlin (2001), "Scholarly teachers are those who consult the
literature, select and apply appropriate information to guide the teaching-learning experience, conduct
systematic observations, analyze the outcomes, and obtain peer evaluation of their classroom performance”.
2.3 SoTL (Scholarship of Teaching and Learning)

A Scholarship of Teaching and Learning intervention involves high level academic teaching and learning. It
addresses university teaching and learning by investigating student learning in the classroom. According to
Rüütmann & Saar (2017):
In general, SoTL includes rigorous, systematic, and evidence-based study of student
learning; the understanding and improvement of student learning and/or teaching
practice; commitment to disciplinary and/or interdisciplinary peer review and
appropriate public dissemination; impact beyond a single course, program, or
institution – advancing the field of teaching and learning to build collective knowledge
and ongoing improvement. (Rüütmann & Saar, 2017, p.214)
It emphasizes the identification and definition of a problem and demands systematic collection of evidence
that will be the results from which conclusions will be drawn (Dewar, Bennet & Fisher, 2018). The instructor
needs to have knowledge of the disciplinary content, of effective pedagogical practices, and of discipline-
specific pedagogical practices. According to Streveler, Borrego & Smith (2007), the involvement in SoTL usually
begins with faculty's interest in how their own students are learning in their classrooms, and the main purpose
of SoTL is to improve learning by improving teaching. " The results of a Scholarship of Teaching and Learning
intervention should add knowledge to the literature on teaching and learning and contribute to the scientific
productivity of the faculty.
2.4 EER (Engineering Education Research)

EER goes beyond the classroom and student learning and consists of the development of more rigorous studies
related to EE (Streveler, Borrego & Smith, 2007). The ultimate goal of EE is, among several, to influence
classroom practice and policies for conducting engineering programs from its findings. In EER, the researcher
focuses on the "why" and the "how" of the educational phenomena involved in EE. The researcher identifies
relevant, meaningful research questions; reviews the literature, links to appropriate theory, uses appropriate
55
research methods aligned with the research question, interprets the data, disseminates the results in a
generalized way. In general, EER demands a set of methods and processes that can be seen as different from
the usual research in specific engineering disciplines. Borrego (2007, p. 99) claims that "research [in EER] is
fundamentally different from engineering research". This turns out to be a hindrance to the development of
EER, since most engineering professors lack familiarity with literature in the field of education and with research
methodologies suitable for educational studies (Froyd & Lohmann, 2014).
Having presented the main characteristics related to ET, ST, SoTL and EER, the next section will present the
methodological procedures developed in this research.
3 Methodological Procedures
The present research is characterized as qualitative research. It investigates the nature of papers presented at
COBENGE 2015, regarding the classification of these papers as being pedagogical interventions, such as ET, ST,
SoTL, or research in EE. The qualitative character of this research is explained by Deslauriers (1991, p. 58) when
he says that "The purpose of the sample is to produce in-depth and illustrative information: whether it is small
or large, what matters is that it is capable of producing new information."
The survey, which identified the 36 studies involving AL strategies and methods published in the annals of
COBENGE 2015 and analyzed in this work, was carried out by Pinto et al. (2020) with the main objective of, as
mentioned before, verifying the growth of AL in EE in Brazil in the period from 2007 to 2019 in COBENGE’s
proceedings.
After identifying the papers, their abstracts were read for an initial classification in terms of ET, ST, SoTL, and
EER. Subsequently, a more in-depth reading of each of the papers was necessary for the final classification
presented in this paper. For the classification into ET, ST, SoTL or EER, the characteristics described in sections
2.1, 2.2, 2.3 and 2.4 were taken into consideration. The researchers randomly divided the 36 papers into two
groups of 18 papers. Each analyzed 18 papers. In case of doubt, the work was read by the other researcher,
and this second reading was followed by a discussion between the two researchers. The characteristics
described in sections 2.1, 2.2, 2.3, and 2.4 were sought in each of the papers. When found in the papers, these
characteristics were highlighted and noted.
In the next section, the results obtained through the described procedures are presented and discussed.
4 Results and Discussion

Table 1 shows the number of papers classified in each of the levels, namely ET, ST, SoTL or EER.
Table 1. Classification of papers in ET, ST, SoTL and EER levels.
ET ST SoTL EER Other
25 1 2 4 4
The general analysis of the classification shows an imbalance between the quantity of ET papers, which include
experience reports, 69% of the total number of papers, and scientific research papers, which include the ST,
SoTL and EER levels, and represent 19% of the total number of papers analyzed. In addition, 11% of the papers
do not fall into any of the levels. The most relevant results for each level are presented and discussed below.
4.1 ET
COBENGE accepts two types of papers, "experience report" and "scientific research", however, there are no
clear criteria for what is an "experience report". Of the 25 papers initially classified as ET, 72% focused on
technical issues of the project/experiment developed, while 28% focused on the methods and strategies of ET.
In both cases, there were papers with: evidence of student learning gains, mostly results of grades from
evaluative activities, such as traditional tests; results from satisfaction surveys; and statements from the authors
56
that the learning was effective, although without verifiable data. These are all papers that claim to present an
innovation in the classroom. However, as pointed out by Wankat, Felder, Smith, & Oreovicz (2002, p. 217), the
evaluation of innovation is typically something like "We (the teachers) tried it and liked it, as did most of the
students”. Based on these characteristics, we defined the experience reports (ER) intervention as: descriptions
of methods applied in the classroom and/or development of experiments or prototypes to support teaching.
In general, the ER interventions do not present a consistent theoretical review of the AL strategies and methods
nor do they provide criteria, data, or evidence of learning gains. In this sense, in this research, it was decided
to formalize the ER level for the papers presenting these characteristics. From the 25 papers initially classified
as ET, 18 papers were reclassified as ER, which means 50% of all 36 AL papers.
Among the six works classified as ET, we highlight the work of Guimarães et al. (2015), in which the authors
describe an activity developed and coordinated by students of a group from the Tutorial Education Program,
under faculty mentoring, directed to freshmen and inserted in the “Introduction to Electrical Engineering”
course. From instructions and materials prepared by the students who organized the activity, teams assembled
line following robots and, in the end, participated in a competition. It is worth mentioning that there are
learning gains for all involved, students who are the organizers and freshmen, and that competitions of this
type generate engagement and emotional involvement, contributing to the consolidation of new knowledge.
The authors explain that the activity involves technical knowledge, mainly related to electronics, as well as
attitudes such as proactivity, flexibility, tolerance to uncertainty, and skills such as teamwork, written and oral
communication, planning, management and leadership. The results include team and participants' evaluation
of the activity and the project, reflection on the technical and non-technical learning gains, and the paper also
mentions agreement with the competencies expected and based on the National Curriculum Guidelines2 and
on ASIBEI3. The instrument mentioned for data collection is the questionnaire.
Analyzing in more detail the work of Guimarães et al. (2015), one realizes that experiments, team brainstorms,
reflections, and non-formalized observations were carried out. It is possible to identify characteristics of action
research and the interrelation between teaching, research, and extension. It is observed that there are
numerous instruments that can be used to obtain evidence of effective teaching in addition to those already
mentioned: interviews and focus groups; content analysis; experiments; case studies; multi-method studies and
hybrid methods.
The paper highlights the competences exercised in each phase of the proposed activity, such as planning,
elaboration, testing or execution, and this helps in the definition of learning indicators that can be used to
analyze student performance and generate evidence for evaluating the results of a proposal. A review of the
literature including similar experiences, greater care with methodological procedures regarding the strategies
and methods of AL and the methods of data collection, as well as the evaluation of student performance, would
elevate the work to the ST level.
4.2 ST
In the ST level, a single paper was identified (Spricigo, 2015), which presents the results of the application of
the PBL method in a technical feasibility study of an industrial enterprise in a class of 17 students in a course
of the Food Engineering program, replacing the traditional lectures previously held. The author presents the
method adopted, in which an open problem divided into two parts was chosen. The methodology developed
involved the presentation of the results and a debate with the class about the solution found by each team. To
evaluate the students' motivation the Situational Motivation Scale (SIMS) was used. For the evaluation of
learning gains, the grades of an individual written test on a problem-situation similar to the theme of the
applied PBL were considered. Rubrics were used to correct the test. Results obtained with the use of the SIMS
scale indicated a high level of intrinsic motivation and a low level of demotivation in the class during the PBL.
2
National Curricular Guidelines for Engineering Courses, Resolution CNE/CES 11, of March 11, 2002. The New Curricular Guidelines, of April 24, 2019, are currently
in effect.
3
Declaration of Valparaíso on Generic Competences of the Iberoamerican Engineer
57
However, extrinsic motivation, which indicates the execution of activities to obtain a reward, was representative.
In the individual test results, 90% of the students met all rubrics. The author makes reflections based on the
qualitative results of the teams and mentions gains in complexity in the issues discussed and in the role of the
professor as mediator.
The main characteristics identified in the analyzed work are: adoption of academic approach to teaching; use
of hybrid methods; experimentation with new techniques; application of literature on teaching and learning in
the course; reflection on teaching; evidence-based study of student learning; analysis of understanding and
improvement of learning; impact on a single course.
Although Spricigo's (2015) work presents the characteristics of an ST, the application of the PBL method was
not developed considering all the stages of the method, nor the evaluative instruments usually used when
applying PBL, for example, peer and self-assessment. Recommendations for a paper like Spricigo's (2015) to
be classified as a SoTL also involve sharing experiences and reflections on the results with professors from
other courses and programs; for it to be classified as an EER, it would need to broaden the scope of application
of AL methods and strategies, include real problems involving companies and/or professionals in the field,
and/or promote continuous improvement of the results.
4.3 SoTL
In the SoTL level, the works of Chinaglia & Santos (2015) and Gomes, Lima & Bianchini (2015) were identified.
In the first, the authors present quantitative results on the introduction of AL strategies for large physics classes.
The research involves an experimental group of 247 students who participated in the new teaching proposal,
a control group with 124 students submitted to the traditional teaching method, and 3 instructors responsible
for each group. The new methodological proposal includes lecture, demonstrative simulation, conceptual test,
and group work to solve problems. The traditional method consists of lecture and exercise solving by the
instructor. The authors used the mechanics baseline test, known worldwide as the "Force Concept Inventory"
(Hestenes, Wells, & Swackhamer, 1992) to assess the prior knowledge of all students and found a leveling of
the classes. The effectiveness of the developed methodology is verified by comparing the average scores of
the students' performance on 3 tests and also their attendance in class. Besides the improvement in the average
grades, the experimental class presented a lower rate of absences. After this experience, the proposal started
to be applied in 100% of the physics courses, involving all the physics instructors of the institution. Also in this
paper, despite the good results achieved, a certain lack of knowledge by the authors about the AL strategies
is noted. In this particular context, when using conceptual tests in the development of the classes, the authors
do not mention the Peer Instruction strategy, famous for its excellent results in learning gains in physics
courses, and whose central point is the conceptual test conceived by Mazur (1997).
In the second paper, Gomes, Lima & Bianchini (2015) investigate the learning gains of 25 freshmen of an
engineering program, by means of an intervention carried out with the Jigsaw method of cooperative learning,
as part of a doctoral research of the Graduate Studies Program in Mathematics Education of São Paulo’s
Pontifical Catholic University (PUC/SP). The chosen subject was the study of the line in two- and three-
dimensional spaces. Five study themes were proposed, a final question and an individual evaluation question.
They used as methods of data collection, observations and interviews. All five groups did well in the
development of the activities. The authors infer that "the Jigsaw method of cooperative learning can allow
students to move through other styles of mathematical thinking different from their preferred ones". It is
observed in this paper the authors' deep knowledge of the Jigsaw method (Aronson, & Patnoe, 2010). This
characteristic of the article may be associated with the fact that the research work is linked to the development
of a doctoral thesis.
Common features of the papers are: good description of methodological procedures, including data collection
methods; analysis of results; extrapolation from the classroom; systematic evidence-based study of student
learning; the understanding and improvement of learning; engagement with disciplinary colleagues; impact on
multiple courses.
An important issue to be considered in SoTL interventions is the focus on improving teaching and learning and
contributing to faculty productivity. Thus, works of this nature are closely related to the existence of faculty
58
training programs. In Brazil, teacher training in engineering programs is still incipient, and most of the time,
actions related to improving teaching are generally the teacher's own initiative or rare teacher training sessions.
4.4 EER
In the EER level, four papers were identified, which in common present theoretical review, describe the
methodological procedures, including data collection methods; analyze the results from the data and
demonstrate that the objectives were met. They certainly go beyond the classroom, however, a feature that
draws attention is the fact that all of them address the Project-based learning (PjBL) method. Three papers are
related to the organization of curricula of engineering programs, three are from Brazil and also three had
explicit cooperation with the University of Minho (UMinho), not necessarily the same papers.
It is worth highlighting the work of Schlichting & Heinig (2015), from the Graduate Program in Education at
the Regional University of Blumenau. The authors discuss the implications of AL regarding the uses of orality,
reading and writing in the professional daily life in engineering. This is a qualitative research that involves
discourse analysis from interviews conducted with students from the 7th semester of the Integrated Master in
Industrial Engineering and Management at UMinho. The research brings a contextualization of the scenario
studied at this university from the perspective of PLE (Project-Led Education) and also about its relationship
with the world of work. The authors present a consistent theoretical framework as a basis for the analysis and
discussion of the results. The methodological procedure is presented in a diluted form, throughout the text as
the students' statements are interpreted from their experiences in the PLE and their first professional
experiences. The authors attest to the need of inserting the student in realistic interaction practices and the
importance of a spiral curriculum, in such a way as to allow the student to move from general knowledge to
specialized knowledge. It is interesting to note that the authors' conclusion refers to Bruner's discovery learning
in which the learner is exposed to the same topic in different ways and levels of depth.
Another work of interest is that of Santos and Silva (2015), which presents the experience of the first contact
with the project-based learning method from the perceptions of freshmen of the computer engineering
program at the State University of Feira de Santana, in northeastern Brazil. A questionnaire answered by 14
participants brings the students' impressions. In the first part, questions about satisfaction are raised; in the
second part, the skills developed; in the third part the infrastructure is evaluated, and in the last part the overall
satisfaction with the method is raised. Although it has been classified as an EER, from the analysis of this work
it can be seen that the authors could have mentioned similar applications in the theoretical review in order to
compare the results found with those in the literature and thus be able to better evaluate the contribution.
Also, the PBL model used was not well clarified.
5 Final Remarks
A reflection on such a large quantity of papers (50% of the total, classified as ER) that only describe practices
or procedures without effectively demonstrating the gains in student learning implies a greater attention to
the training of engineering professors for EE issues. In many papers, gaps are observed about several important
aspects of AL strategies and methods, such as the stages of application, the evaluation of learning when
applying such a strategy/method, among others. It is essential that the instructor knows the AL strategy and/or
method that he/she will use in the design of the learning environments in order to be able to choose which
one will help in the development of the intended learning outcomes. Only then, the instructor will be able to
develop a good ET intervention.
The very small number of papers classified as ST and SoTL is closely related to the fact that most engineering
schools in Brazil are not supported by faculty training programs. The results of a survey conducted by the
ABENGE´s Teacher Training Working Group (Battistini & Mattasoglio, 2020) on the training of engineering
instructors confirm that in most higher education institutions in Brazil there is no space for the training of
59
engineering instructors. This data also supports the significant amount of ER papers found in this research. It
is worth pointing out that ST and SoTL are concepts little known to the Brazilian EE community.
In relation to the EER papers, this research did not expect a significant number of papers in this level, after all,
COBENGE is a national event, and EER papers, in general, are submitted for publication in specialized journals.
However, it was expected early stage EER papers or clippings from a more robust research project. Here, it is
worth presenting a result reported by Williams (2018), in which he highlighted that in the period from 2015 to
2018, Brazilian researchers submitted 14 papers to the Journal of Engineering Education in different EE subjects.
Nine of these papers were rejected upon submission, five were revised, and none of them were accepted for
publication. Unfortunately, this result presented by Williams points to a certain unpreparedness of Brazilian
engineering professors for EE research.
Although the analysis carried out was in articles from 2015 and, therefore, can be considered a little outdated,
what matters is that this research considers the theme Active Learning addressed in 2015 in COBENGE and is,
as far as is known, a first step to analyze the state of the art of engineering education research in AL in Brazil.
This research work should be extended to evaluate a more significant number of COBENGE papers on AL
strategies and methods, in principle, from 2015 to 2021. The results presented here point to the need of a
discussion to be developed within the EE community in Brazil. As actions to improve the scenario presented
here, the GTAAEE is working to organize its symposium which will address the theme "Scholarship of Teaching
and Learning" and to create a tutorial for the planning of ET type interventions to assist in the submission of
papers to COBENGE.
This paper will be finished with the following statement:
“Changing practice in engineering education so that faculty members apply findings
in engineering education research, education research, and research in the learning
sciences to their practice in engineering classrooms is a major challenge for
engineering education practice and research (Jamieson & Lohmann, 2012)”
because this statement masterfully describes the great challenge that engineering education research in Brazil
still has to face.
6 References
Aronson, E., & Patnoe, S. (2010). Cooperation in the classroom: The jigsaw method. Pinter & Martin.
Battistini, A., & Mattasoglio, O. (2020). Resultados da Pesquisa nas IES sobre Formação de Professores. Available
in:<http://www.abenge.org.br/arquivos/downloads/atas/20190920_GT%20Forma%C3%A7%C3%A3o_Professore
s_Fortaleza%202019_Novo_2.0_Publicado_Novo.pdf>. Acessed: April 24, 2022.
Borrego, M. (2007). Conceptual difficulties experienced by trained engineers learning educational research methods,
Journal of Engineering Education, Vol. 96, No. 2, pp. 91-102.
Borrego, M., & Bernhard, J. (2011). The emergence of engineering education research as an internationally connected field
of inquiry. Journal of Engineering Education, 100(1), 14-47.
BRASIL (2019) Ministério da Educação. Diretrizes Curriculares Nacionais para os Cursos de Engenharia. Resolução CNE/CES
nº 2, de 24 de abril de 2019 - Institui as Diretrizes Curriculares Nacionais do Curso de Graduação em Engenharia.
2019. Available in: <http://www.in.gov.br/web/dou/-/resolu%C3%87%C3%83o-n%C2%BA-2-de-24-de-abril-de-
2019-85344528>. Acessed: April 24, 2022.
Chinaglia, E. F., & Santos, R. B. B. (2015). Metodologia ativa de aprendizagem para Física básica em cursos de engenharia.
In: Anais do XLIII Congresso Brasileiro de Educação em Engenharia “Aprendizagem Ativa: Engenheiros
colaborativos para um mundo competitivo”, 08 a 11 de Setembro de 2015, São Bernardo do Campo.
De Graaff, E., & Christensen, H. P. (2004). Editorial: Theme issue on active learning in engineering education. European
Journal of Engineering Education, 29(4), 461-463.
Deslauriers, J. P. (1991). Recherche qualitative: guide pratique. Montreal: La Chenelière McGraw-Hill.
Dewar, J., Bennet, C., & Fisher, M, (2018). Understanding the Scholarship of Teaching and Learning (SoTL).The Scholarship
of Teaching and Learning - A Guide for Scientists, Engineers, and Mathematicians. Oxford University Press. 72p.
Eggen, P. D., & Kauchak, D. P. (2006). Strategies and Models for Teachers Teaching Content and Thinking Skills (5th ed.).
Pearson Education Inc.
60
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning
increases student performance in science, engineering, and mathematics. Proceedings of the national academy of
sciences, 111(23), 8410-8415.
Froyd, J. E., & Lohmann, J. R. Chronological and Ontological Development of Engineering Education as a Field of Scientific
Inquiry. (2014). In Johri, A., & Olds, B. M. (Eds.). Cambridge handbook of engineering education research. (pp. 28-
48). Cambridge University Press.
Gomes, E., de Lima, G. L., & Bianchini, B. L. (2015). Percepção dos alunos a respeito de uma atividade utilizando o método
Jigsaw de aprendizagem cooperativa. In: Anais do XLIII Congresso Brasileiro de Educação em Engenharia
“Aprendizagem Ativa: Engenheiros colaborativos para um mundo competitivo”, 08 a 11 de Setembro de 2015,
São Bernardo do Campo.
GTAAEE (2022). http://www.abenge.org.br/Gt_Aprendizagem.php
Guimarães, I. M., Martins, M. F., Sampaio, T., & Gomes, F. J. (2015). Projeto Linus Bot: A Metodologia PjBl na Inserção dos
Novos Alunos no Curso de Engenharia Elétrica. In: Anais do XLIII Congresso Brasileiro de Educação em Engenharia
“Aprendizagem Ativa: Engenheiros colaborativos para um mundo competitivo”, 08 a 11 de Setembro de 2015,
São Bernardo do Campo.
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The physics teacher, 30(3), 141-158.
Jamieson, L. H., & Lohmann, J. R. (2012). Innovation with impact: Creating a culture for scholarly and systematic innovation
in engineering education. Washington, DC: American Society for Engineering.
Kay, R., & MacDonald, T. (2016, June). Comparing flipped, active, and lecture-based teaching approaches in higher
education. In EdMedia+ Innovate Learning (pp. 1559-1565). Association for the Advancement of Computing in
Education (AACE).
Kipper, H., & Rüütmann, T. (2012, September). Teaching for understanding in engineering education. In 2012 15th
International Conference on Interactive Collaborative Learning (ICL) (pp. 1-9). IEEE.
Mazur, E. (1997) Peer instruction: a user’s manual. Upper Saddle River, NJ: Prentice-Hall.
McConnell, D. A., Chapman, L., Czajka, C. D., Jones, J. P., Ryker, K. D., & Wiggen, J. (2017). Instructional utility and learning
efficacy of common active learning strategies. Journal of Geoscience Education, 65(4), 604-625.
Oliveira, J. A., Pinto, G. R. P. R., & Santos, J. M. J. (2017). Uso de Estratégias Ativas na Educação em Engenharia no Brasil: um
mapeamento sistemático de experiências a partir das publicações realizadas no COBENGE. In: Anais do XLV
Congresso Brasileiro de Educação em engenharia “Inovação no Ensino/Aprendizagem em Engenharia”, 26 a 29
de Setembro de 2017, Joinville.
Pinto, G. R. P. R., Oliveira, J. A. C. B., Viana, D. M., das Neves, R., & Villas-Boas, V. (2020, April). Work-in-Progress: A Systematic
Mapping Study of Experiences with Active Learning Strategies and Methods in Brazilian Engineering Education. In
2020 IEEE Global Engineering Education Conference (EDUCON) (pp. 1819-1823). IEEE.
Richlin, L. (2001). Scholarly teaching and the scholarship of teaching. New directions for teaching and learning, 2001(86),
57-68.
Rüütmann T. & Saar, M (2017). Scholarly Teaching and Scholarship of Teaching and Learning in Teaching Engineering. In:
IEEE Global Engineering Education Conference (EDUCON) (pp.213-218). IEEE.
Silva, C. A. S., & Santos, D. M. B. (2015). Avaliação do método PBL por calouros do curso de engenharia de computação da
UEFS. In: Anais do XLIII Congresso Brasileiro de Educação em Engenharia “Aprendizagem Ativa: Engenheiros
Schlichting, T. S., & Heinig, O. L. O. M. (2015). Aprendizagem ativa e suas implicações nas práticas de linguagem
características do mundo do trabalho em engenharia. In: Anais do XLIII Congresso Brasileiro de Educação em
Engenharia “Aprendizagem Ativa: Engenheiros colaborativos para um mundo competitivo”, 08 a 11 de Setembro
de 2015, São Bernardo do Campo.
Shuman, L. J.; Besterfield-Sacre, M. Innovation through Propagation: Future Directions for Engineering Education Research.
Advances in Engineering Education, 2019, 7(2).
Spricigo, C. B. (2015). Aprendizado baseado em problemas no ensino de fundamentos de projetos de processos industriais.
In: Anais do XLIII Congresso Brasileiro de Educação em Engenharia “Aprendizagem Ativa: Engenheiros
Streveler, R. A., Borrego, M., & Smith, K. A. (2007). Moving from the ‘scholarship of teaching and learning’ to ‘educational
research’: An example from engineering. In D. R. Robertson (Ed.), To Improve the Academy, 25(1), 139-149. Bolton,
MA: Anker.
Wankat, P. C., Felder, R. M., Smith, K. A., & Oreovicz, F. S. (2002). The scholarship of teaching and learning in engineering.
Disciplinary styles in the scholarship of teaching and learning: Exploring common ground, 217-237.
Williams, B. (2018) Internacionalização na Formação de Engenharia. In: XLVI Congresso Brasileiro de Educação em
Engenharia e 1º Simpósio Internacional de Educação em Engenharia “Educação Inovadora para uma Engenharia
Sustentável”, 03 a 06 de setembro de 2018, Salvador.
61
Project-based learning applied to improve the performance of a
family wine bottling unit
Cláudia Gonçalves1, João Neto1, Sabrina Oliveira1, José Dinis-Carvalho2
1
Department of Production and Systems, School of Engineering, University of Minho, Guimarães, Portugal
2
ALGORITMI Centre, Department of Production and Systems, School of Engineering, University of Minho, Guimarães, Portugal
Email: [email protected], [email protected], [email protected], [email protected]
Abstract
This article aims to show the effect of the application of concepts and principles of some operational excellence models,
such as Lean Thinking and Shingo Model, in a family wine bottling unit. The project was carried out by a group of students
in a project-based learning mode in the curricular unit "Lean Management and Organization" of the Master’s degree in
Systems Engineering at the University of Minho. The objective of the project is to implement actions and improve the
overall performance of the wine bottling unit. The students started by making an analysis and diagnosis of the initial state
of that production unit, identifying problems and opportunities for improvement, and introducing improvements that led
to better performance. The main concepts and principles applied were value and waste identification; transport, movement
and defect waste reduction; production in flow; and process focus. They managed to better balance the line, improve flow,
reduce defects and increase productivity.
Keywords: Project Based Learning; Excellence Models; Lean Thinking.
1 Introduction
Over the years, there has been an increase in competitiveness in the world market, which together with the
emergence of an increasingly demanding consumer, forced the evolution of production systems. Within that,
the Lean philosophy emerged, which, according to Liker (1997) can be defined as an organizational model that
seeks to eliminate waste and increase the agility and efficiency of the value stream. The concept of waste can
be defined as any activity that uses resources, but doesn’t produce any type of value and can be classified as:
overproduction, when more is produced or before the moment is necessary; transport, when there is
unnecessary movement of products/materials; inventory, when there is an excess of stored materials or
products; waiting, when there is waste time waiting for the next step; overprocessing, when there is more work
or higher quality than required by the customer; defects, when there is effort caused by defects, reworks and
incorrect information; movement, when there is an unnecessary movement of people; human potential, when
talent, skills and knowledge are wasted (Womack and Jones, 2003; Ohno, 1998).
To assist the identification of waste, Value Stream Mapping (VSM) (Rother & Shook, 1999)(Jones & Womack,
2002) can be used as a Lean tool to analyze and model production processes and, consequently, identify
activities that add/don’t add value (Busert & Fay, 2020). In addition, to monitor and evaluate the evolution of
production, a set of performance indicators is used, aimed at eliminating waste, efficient use of resources and
better control of the production process (Mourtzis et al., 2017).
The article describes the application of theoretical knowledge in a practical context with the aim of improving
a family wine bottling line where production problems and operating difficulties were observed. The main
objective is to understand how the application of Lean principles and techniques have an impact on a
production line. In this way, it is intended to carry out an analysis of the production line and identify
opportunities for improvement, collect data and information for analysis, propose and implement
improvements and perform comparisons in the performance before and after the proposed changes.
In a first phase, in order to have a real perception of the production line, the group went to the farm, on a
bottling day, and observed its operation. In addition, processing times were also accounted for in order to
ensure the possibility of comparing performance indicators with the improvements implemented.
62
Then, to assess the state of the bottling line, a VSM was created, and some relevant performance indicators
were established. In this way, it was possible to identify in which sections of the line improvements could be
applied and/or problems that occurred during the analysis of the line could be solved. That said, a set of actions
was developed and later implemented on the line, in order to account for new process performance indicators.
After all the implementations, the new operation of the line was analyzed, and conclusions were drawn to
understand the impact of lean techniques implemented in the bottling line.
The present article was developed by a group of students, within the scope of the curricular unit Management
and Lean Organization, taught in the Master’s degree in Systems Engineering at the University of Minho. The
project was developed during the second semester of the academic year 2021/2022, and a group of three
elements implemented the techniques taught in the curricular unit to this practical case. The curriculum of
Master’s degree in Systems Engineering aims to provide students a strong practical component on all the
following curricular units: logistics, simulation, production management, supply chain optimization, systems
analysis, similarity based systems, advanced database administration and exploration and two integrated
projects. These two integrated projects are carried out over an entire semester and it´s important to mention
that they combine all the other units. The present project developed in the lean management and organization
curricular unit fits in the Master’s curriculum since there is a link between the course, the production and the
industry so it is important to know the techniques learned in this unit. It also provides the practical component
that becomes a valuable point for the students' learning, especially since it is a project-based learning.
At the beginning of the semester, the teacher explained the context of the project to the students, being that
no specific theme was given to them, it was only necessary to apply all the knowledge that were being acquired
throughout the unit to a practical case chosen by the group. The group decided to apply the project in a
company context and the first task was to select it. After the selection, the group defined milestones to achieve
a member responsible for having the most contact between the company and the group/teacher. Given the
characteristics of the business, wine bottling was not yet being executed at the beginning of the project, so the
group decided to visit the farm and study some aspects of wine bottling characteristics according to workers
interviews and through a simulation of the line production. When the line was finally working, the group divided
tasks among the members, to increase productivity, and was given a deadline to every task in order for the
group to fulfill the plan initially defined.
2 Initial State
The Quinta Portal da Veiga is a family wine company located in Guimarães, region in the north of Portugal. The
farm produces red, rosy and white “vinho verde”4. The winemaking process ends with the bottling of the wine,
which takes place from December to March, on previously planned days, because of the long setup times
required to prepare the bottling wine. Another restriction is that when the wine is transferred to the
refrigeration tank, it has to be emptied by the end of each bottling day.
The current bottling line considered in this study needs five operators assigned to five processes. Before each
bottling day, the bottles required for that day are washed in advance and stored in baskets as shown in Figure
1. Left, with fifty bottles each.
4
“Vinho verde” is the type of wine produced in the northwest part of Portugal
63
Figure 1. Storage of washed bottles (left); Process 1 (right).4
In process 1 the operator collects empty bottles from the baskets and places them in the filling machine. Once
inside the machine, the filling is automatic, and each bottle always takes the same amount of wine, which is
mixed with the gas. This process takes about 4.49 seconds per bottle.
The process 2 (Figure 2 on the left) also requires one operator interacting continuously with process 1, since it
consists of removing the bottle from the filling machine and passing it on to the worker of the next process.
This process takes about 3.99 seconds per bottle.
Figure 2. Process 2 (left) and Process 3 (right).
The operator in Process 3 takes the bottle that comes from process 2, places it in the bottle corking machine
and presses a button to initiate an automatic introduction of a cork stopper in the bottle. This process takes in
average 3.99 seconds per bottle.
In Process 4 (see left side of Figure 3) the operator collects the bottle that comes from the previous process,
places a plastic capsule on the bottle neck and inserts the bottle into the bottle capsulator machine for a short
period of time, so that the plastic melts and stays fixed to the bottle. The time of this process is 5.84 seconds
per bottle.
Figure 3. Process 4 (left) and Process 5 (right).
Finally, in Process 5 an operator transports six bottles at a time taking them by hand to the storage location.
This process has a time per bottle of 3.53 seconds.
64
2.1 KPIs Analysis
The line configuration presented has a bottleneck in process 4, resulting in a cycle time of 5.84 seconds per
bottle. Table 1 presents data and KPI results from the observation and analysis of one day. Due to 5 workers
hired, the productivity of 123 bottles/man.hour and the labor utilization rate of 60.53% was determined. The
processing time, that is, the sum of times of all processes is 20.68 seconds, which results in a line efficiency of
70.81%. In this way the line is only able to fill 3785 bottles in an 8-hour shift, equivalent to 400 minutes of work.
On the observed day 84 defective bottles were produced, so a quality factor of 97.8% was determined. The
OEE (Overall Equipment Effectiveness) was also determined resulting in a value of 74.9% with a speed factor of
83.2% and an availability factor of 92.1%.
Table2. Data on the current state of the bottling line
Data collected Value KPI Value
Number of employees 5 Line Cycle Time 5.84 seconds
123
Parts produced 3785 Productivity
bottles/man.hour
Total Processing Time 20.69 seconds Line efficiency 70,81%
Total manpower time 17.68 seconds Labor utilization rate 60,53%
Opening time 400 minutes OEE 92,1%
Unplanned downtime 31.6 minutes Availability factor 83,2%
Defective produced parts 84 Speed factor 97,8%
Quality factor 74,9%
2.2 Waste identification

Once the operation of the line was analyzed, a VSM (Value Stream Mapp) was created, with the objective of
being able to describe the whole process and identify the main problems and some existing types of waste.
Given the characteristics of the business, the VSM only corresponds to the bottling period.
Customers go to the farm on average once a day and buy 2 boxes of 12 bottles. Suppliers, who are contacted
by telephone call by the production manager, go to the farm only once a year and bring all the necessary stock
(capsules and cork stoppers), except for the bottle supplier, which moves once a week.
Regarding the processes, 5 types of waste were identified: motion, defect, transport and overproduction.
Motion waste was observed in processes 1 and 2, because operators are constantly replacing empty baskets
with baskets with bottles. Defective waste occurs in all processes: in process 1 and process 2, it occurs when
the filling of the bottles becomes unbalanced and the bottles become a little empty; in case 3, it is verified
when the bottle explodes at the moment the cork stopper is put; in process 4, it happens when the bottle
explodes, because the operator leaves it in the machine too long; in process 5, occurs when the operator drops
a bottle to the ground and leaves. The waste with transport is observed in process 5, when the operator moves
through the cellar to the storage area of the bottles. Finally, overproduction is wasted when it is produced first
than necessary. However, this waste is considered as necessary due to the characteristics of the business.
Additionally, in addition to these wastes, one of the main problems of the line is the accumulation of WIP
between process 3 and 4, because in process 4, there are actually two operations: putting the capsule and
putting the bottle in the machine. Thus, the operator, before putting the bottle into the machine, always has
to check that the bottle already has capsule and, while putting capsules, has to check that no bottle has
forgotten in the machine, which means that it is a more time-consuming process, which marks the cadence of
the line.
65
Figure 4. Value Stream Mapping (VSM).
3 Proposals for improvements

In view of the identified waste, we sought to propose solutions that would allow its elimination or its reduction.
The first proposed solution is the acquisition of fastrack wine supports, as they take up little space and are quite
stable. Given that the space for basket storage if of about 15 square meters and the proposed solution,
considering 3 levels, needs about 7 square meters and would be possible to be implement. However, an
investment of €660 would be required.
Figure 5. Example of a fastrack wine holder (Fast Brewing & Wine Making, 2018)
In fact, this solution would reduce waste with motion in processes 1 and 2, as operators would not need to be
constantly picking up a heavy basket to replace the empty baskets. In addition, it would reduce the defective
waste that occurs in process 3, because with the new arrangement of the bottles, the bottlenecks would be
away from each other, and the glass would always be intact. It should be noted that the arrangement of the
deflating bottles of the baskets causes the necks to exert pressure on each other, which gives rise to fractures
in the glass that then give in to the pressure of the cork stopper.
The second suggested solution consists of the use of two cases between processes 4 and 5, which make it
possible to carry 12 bottles simultaneously. Thus, it was intended for the worker in workstation 4 to place the
bottles in one of the baskets instead of placing them on the table, whereas during this period the worker in
workstation 5 would be placing the bottles in the storage area. Consequently, when he returned to his post,
he would have a new complete box to make a new transport. However, after implementing this solution, it
66
was found that the worker in post 5 had enough waiting periods and to eliminate this waste, it was considered
that this worker should help the worker in post 4 to put capsules.
Having said that, in process 5, the waste with transport is reduced, since the worker can carry 12 bottles in a
single movement, and the defective waste is eliminated, since the boxes prevent some bottle from falling to
the ground. In addition, there is a reduction of WIP between process 3 and 4 and a decrease in the waste
verified in process 4, since with the help of the worker of process 5, the worker of this process can be more
focused on the bottles he puts in the machine.
In addition to the improvements implemented in the line, the 5S methodology (Hirano, 1995) was also
implemented, in order to keep the winery and materials organized and reduce the cleaning time of the space
whenever an unplanned stop occurs, in view of the occurrence of defects in production.
In the separation (Seiri), the necessary objects were separated from the unneeded ones. For this, the red tag
strategy was used with the identification of the object and the reason for it to be discarded. In the red tag area
defined, the objects must be kept for one month and then the owner of the farm must make a final decision
in relation to their usefulness.
Figure 6. Implemented red tag area (left) and Organization of materials (right)
Then the space was organized (Seitou) so as to be simpler and more intuitive to find the materials, having
placed some identifying papers.
Regarding cleaning (Seiso), despite the care of the workers, they usually used baskets to store the garbage and
therefore the only aspect implemented was the addition of a dustbin.
For the Seiketsu (Standardize) process, visual management was used with instructions in the various spaces
and in relation to the last Shitsuke (Keep), the line is responsible for naming a person who on each day of
bottling, before starting the shift, verify that the 5S rules are being complied with and that the space is properly
organized.
4 Futures State
The new line configuration allowed reducing the cycle time to 4.49 seconds, maintaining the same number of
employees, which improves the various KPIs analyzed previously. Considering the same time of operation of
the line (400 minutes), the most significant improvements are the increase in productivity to 160 bottles/man,
quantity of parts produced for 4981 bottles, reduction of defective parts and the various times of operation of
the line. The positive impacts are reflected in the availability factor, speed and quality, contributing to an
increase in the OEE to 77.5%.
Table 2. Data on the future state of the bottling line
KPI Value Value
Cycle time 5.84 seconds 4.49 seconds
Number of employees 5 5
Productivity 123 bottles/hour. 160 bottles/hour.
Processing time 20.69 seconds 17.33 seconds
67
Total manpower time 17.68 seconds 14.33 seconds
Line efficiency 70,81% 77,18%
Labor utilization rate 60,53% 63,82%
Opening time 400 minutes 400 minutes
Unplanned downtime 31.6 minutes 27 minutes
Parts produced per day 3785 4981
Defective produced parts per day 84 67
Availability factor 92,1% 93,2%
Speed factor 83,2% 84,3%
Quality factor 97,8% 98,65%
Overall Equipment Effectiveness 74,9% 77,5%
5 Conclusion
This article, developed in real context by a group of systems engineering students in a family wine company,
identified opportunities for improvement in the company under study.
The project helped students to understand the impact of lean principles and techniques application in the
production process with a practical use case. Additionally, the experience contributed to understand that a
successful teamwork needs a very solid communication and a distribution of tasks according team’s members
skills.
This work also contributed to the students learning process, with the increase of their motivation as part of a
natural process with an outside collaboration and in the development of important skills for their future
professional path, like critical thinking and teamwork spirit.
In this paper, the following improvements in line performance should be highlighted: decreased cycle time,
significantly increase in line productivity and efficiency, decrease in defective parts per day and a very
satisfactory increase in OEE in the face of implementing improvements.
In addition to the improvements verified with the monitoring of performance indicators, the lean
improvements implemented, allowed workers to operate more safely in their workplace and are less subject to
any accidents that may occur when a bottle leaves. The implementation of the 5S also brought significant
advantages with regard to the organization of the winery, with the main benefit of speeding up unplanned
stops when problems occur on the line.
Since the main objective of this project was to apply the knowledge acquired in lean classes in a practical
context, the work developed falls within that scope and for that the results were successfully obtained.
The project evaluation was divided in two phases. Firstly, in the middle of the semester, the students introduced
the company to the teacher, showed videos about the processes and presented some preliminary results. At
the end of the semester, the final results were discussed with other systems engineering students. On both
occasions, the teacher has given feedback to improve the quality of the project and provided an evaluation
about the topics discussed and the level of the group communication.
Finally, the main challenge faced with the realization of the project was to overcome the resistance of
employees in the face of the improvements that the group intended to implement. However, the project
allowed the group to understand the concepts taught through their application in a real context.
Acknowledgments
This work was partially supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope:
UIDB/00319/2020.
68
6 References
Busert, T., & Fay, A. (2020). Information quality focused value stream mapping for the coordination and control of production processes.
International Journal of Production Research, 1–20. https:10.1080/00207543.2020.1766720
Fast Brewing & Wine Making. (2018). FastRack Product Specifications. https://www.fastbrewing.com/products/fastrack /product-specs
Hirano, H. (1995). 5 pillars of the visual workplace: The source book for 5S implementation. Productivity Press.
Jones, D., & Womack, J. (2002). Seeing the Whole: Mapping the Extended Value Stream. In Lean Enterprise Institute, Brookline. Lean
Enterprises Inst Inc.
Liker, J. K. (1997). Becoming Lean: Inside Stories of U.S. Manufacturers. Productivity Press.
Mourtzis, D., Fotia, S., Vlachou, E., & Koutoupes, A. (2017). A Lean PSS design and evaluation framework supported by KPI monitoring and
context sensitivity tools. The International Journal of Advanced Manufacturing Technology, 94(5-8), 1623–1637.
https:10.1007/s00170-017-0132-5
Ohno, T. (1988). The Toyota Production System: Beyond Large-scale Production. Productivity Press.
Rother, M., & Shook, J. (1999). Learning to see: Value stream mapping to add value and eliminate muda. In The Lean Enterprise Institute.
https://doi.org/10.1109/6.490058
Womack, J., P. e Jones, D., T. (2003). Lean Thinking - Banish waste and create wealth in you corporation. Free Press.
69
A Gamification Approach for Continuous Engagement in
Engineering courses
Andres Ubeda1, Gabriel J. Garcia1, Vicente Morell1, Jose L. Ramon1, Maria J. Blanes2
1
Human Robotics Group, Department of Physics, Systems Engineering and Signal Theory, University of Alicante, Spain
2
IT Service, University of Alicante, Spain
Email: [email protected], [email protected], [email protected], [email protected], [email protected]
Abstract
This paper proposes an approach for continuous engagement in engineering courses using Kahoot! gamification. The main
goal of the approach is increasing weekly interest in the course by providing regular assessment of the student progression
in an entertaining and attractive way. For each different module of the course, a Kahoot! quiz is taken by the students to
score several points proportional to the module length. The quiz includes basic questions on the topic. Points are only
distributed among the top ten participants. At the end of the course, all points are summed to obtain the final classification.
The course grade is increased up to 5% depending on the final position of the student. The experience aims at increasing
the interest of the students and promote continuous learning from the beginning of the course. This paper describes the
organization of the activities, the design of the Kahoot! quizzes, the tentative student survey to evaluate engagement and
satisfaction and the quantitative assessment metrics of the proposed approach.
Keywords: Gamification; Active Learning; Engagement.
1 Introduction
Educational technology has been a matter of great importance in the last two decades, particularly in the STEM
field (Chacko et al., 2015). The incorporation of alternative ways of promoting learning with current available
technologies such as computer-based activities has been a priority of educational institutions. An example of
this is the use of social networks such as Facebook to promote science education (Waghid, 2015) or open-
source databases as Wikipedia (Meishar-Tal, H., 2015).
A very interactive way of promoting learning and engagement is gamification. This method tries to increase
the student motivation by using game-thinking to solve problems (Conolly et al., 2012). Several studies have
used gamification methods to increase engagement in students using a variety of platforms such as Knowingo
(Welbers, et al., 2019), Class Dojo (DiGiacomo et al., 2022) or MinecrafEdu (Cózar-Gutiérrez and Sáez-López,
2016). A very simple and attractive option is the use of the Kahoot! platform. Kahoot! provides a series of
learning games that include multiple-choice questions and puzzles. The use of Kahoot! gamification or similar
systems has shown a significantly better performance in students’ grades than those who had traditional
teaching (Sarkar, Ford, & Manzo, 2017).
In this paper, we propose an approach for continuous engagement in engineering courses using Kahoot!
gamification. The main goal of the approach is increasing weekly interest in the course by providing regular
assessment of the student progression in an entertaining and attractive way. Some works have incorporated
similar approaches to engineering courses including surveys and quantitative evaluation of certain parameters.
These studies do not evaluate engagement elements and are more centered in the effect on final grades
(Chernov et al., 2021) or in student satisfaction (Pertegal-Felices et al., 2020). For this reason, our proposal also
includes possible metrics to evaluate how the gamification activities are affecting weekly engagement in the
course contents and does not focus specifically in the effects on course performance.
This paper is divided as follows: in Section 2, the gamification methodology is explained including the structure
of the activity, the design of quizzes, the scoring systems, the student survey and the evaluation metrics. Section
3 describes the possible additions to the gamification approach. Finally, Section 4 contains the conclusions.
70
2 Methods
In this section, we show the description of the structure of the gamification activities and the general and
specific criteria to create the different quizzes. The satisfaction survey provided to the students at the end of
the course is also described as well as the metrics extracted from the students’ performance.
2.1 Structure of the activity

The gamification activity is divided into several quizzes (one per block) along the course, each of them with a
set of 20 questions. Each course has specific characteristics and length, so the questions, distribution and score
of quizzes are adapted accordingly (Figure 1). The activity is proposed for the following engineering courses in
University of Alicante:
• Service Robotics (BSc in Robotics Engineering): This course describes the fundamentals of service
robotics. A service robot is a robot that operates in a semi or totally autonomous way to provide useful
services for the well-being of human beings or equipment, excluding manufacturing operations. The
course is divided into 4 blocks with a Kahoot! quiz at the end of each. In a first block, the concept of
human-robot interaction and the different methods available to carry it out are analyzed. In a second
block, the main aspects of cognitive robotics are studied, including the analysis of cognitive
architectures and cognitive human-robot interaction. The third block describes humanoid robots and
aspects on locomotion of this type of robots. Finally, the last block presents the applications of service
robotics in different areas, particularly centered in the use of robots for rehabilitation. Course length is
divided into 15 theory lectures of 2 hours and 15 practical sessions of 2 hours (60 hours total).
• Fundamentals of Systems and Instruments (BSc in Biomedical Engineering): This course describes
the use of systems and instrumentation in a clinical context. The course is divided into 4 blocks with a
Kahoot! quiz at the end of each. In a first block, basic concepts of systems theory are introduced
addressing possible clinical situations. A second block describes the main biomedical sensors used to
measure human biosignals, including electroencephalography (EEG), electromyography (EMG),
electrocardiography (ECG) and stimulation systems. The third block covers the application of robotics
in clinical environments as for rehabilitation, assistance, surgery or motor substitution. Finally, the last
block describes visualization systems in medicine and aspects of virtual and augmented reality for
rehabilitation. Course length is divided into 15 theory lectures of 2 hours, 15 problem-case lectures of
1 hour and 10 practical sessions of 1 hour and 30 minutes (60 hours total).
• Robot Control and Programming (MSc in Automation and Robotics): This course is focused on
the control and programming of robots in different environments. The course is divided into five
different fields in robotics: rehabilitation robots, humanoids, robotic prostheses, space robotics and
assistive robots. For each section a theoretical lecture is given followed by video examples of each
topic and the Kahoot! quiz. Each section has a short introduction on the specific type of robot and
aspects on how to interact and control them in practical applications. The practical contents are
evaluated through a group project. Course length is divided into theory lectures up to 15 hours and
15 hours for the development of the practical project (30 hours total).
71
Figure 1. Score ratio across modules for each of the courses where the methodology is or will be applied. The final score
for the course is computed by multiplying quiz position by the score ratio of a particular module and summing up score
for all course quizzes (score range: 0-100 points).
2.2 Design of quizzes

Kahoot! is a game-based learning platform used in educational institutions to promote additional ways of
engaging students. Kahoot! provides a series of learning games that include a variety of questions such as
multiple-choice, true and false, short answer, puzzles and audio questions. It also has the possibility of polling
and brainstorming in a cooperative and intuitive way. Kahoot! can be accessed through a web browser or a
smartphone app and allows the students to easily interact with the quiz from their seats in classroom.
For this educational experience we have proposed the use of only multiple-choice questions (4 options) to
evaluate how students are following the weekly contents of each course. At the end of each block of contents,
a series of 20 questions is presented to the students. Questions are divided into three categories:
• Basic knowledge (B): Includes basic concepts that should be known if the student has a minimal
understanding of the contents of the block. This kind of questions are not expected to be included in
final evaluation.
• Advance knowledge (A): Includes concepts that only students with a more detailed comprehension
of the contents are able to know as they require a more complex reasoning. Questions of similar
difficulty are included in the final evaluation.
• Professor remarks (P): Includes aspects that the professor has mentioned in class and that are not
necessarily included in written content, i.e., slides or documentation.
A proposed question distribution for each Kahoot! quiz would be 10B, 5A and 5P questions. For quizzes that
include a module with different submodules a proportional distribution of contents is advisable.
2.3 Scoring system

The proposed scoring system considers the length of each evaluated module as defined in Figure 1. For each
quiz only the first 10 best students will score (10 to 1 point in descendent order). The score of the quiz will then
be multiplied by the score ratio (proportion of the block divided by 10) and summed to the scores obtained in
the other quizzes. The final score will give a maximum of 100 points. For example:
• If Student 1 from the Service Robotics course is 2nd in the first quiz, 6th in the second quiz, 14th in the
third quiz and 10th in the fourth quiz, he/she will get a total score of 42.5=9*3+5*2.5+0*1.5+1*3.
• If Student 2 from the Service Robotics course is 3rd in the first quiz, 4th in the second quiz, 5th in the
third quiz and 1st in the fourth quiz, he/she will get a total score of 80.5=8*3+7*2.5+6*1.5+10*3.
72
The final course grade is increased by a 5% if a student has the highest score of the whole course. Up to 10
students get an increase in the course grade. The student with the second highest score gets a 4.5% increase
in the course grade and so on, until the 10th student who gets only a 0.5% increase in the course grade. This
scoring system is intended to be proportional to the course contents and to reward not only the winner of
each quiz but a fair number of students to avoid disengagement in the course of time. An additional prize for
the winner can be optionally awarded. For engineering courses, we propose the 3D-printing of a trophy, medal
or badge shaped in a course-related topic. This award is given to the winner by the end of the last quiz. This
kind of reward is aimed at increasing the interest on this activity in the following academic years.
2.4 Student surveys

By the end of each course, students will answer to a questionnaire to evaluate aspects of engagement and
satisfaction. Table 1 shows the proposed survey given to students. Each question follows a numerical Likert
scale style from 1 to 10, being 1 “Strongly disagree” and 10 “Strongly agree”.
Table 3. Elements of the student survey.
Type Question
Do you think the Kahoot!-based evaluation is a good tool to capture your interest in this course?
Did your interest in the Kahoot!-based quizzes increased in time with each activity?
Engagement
Did this activity make you study in advance or have more interest in the contents to perform
better in the quizzes?
Did you have fun while doing this activity?
Do you think the proposed Kahoot!-based evaluation is well balanced in terms of difficulty?
Do you think the proposed Kahoot!-based evaluation is well balanced in terms of contents?
Do you prefer this kind of activities instead of optional submissions of practical or theoretical
exercises?
Satisfaction Do you recommend doing this kind of activities in other courses?
Do you think your position is fair compared to the effort you did?
Do you think the score prize of this activity is high enough?
Do you agree with the scoring system?
Short final
Do you have any suggestion to improve this activity?
answer
2.5 Evaluation metrics

To evaluate the gamification in a quantitative way, several metrics are proposed:
Course scores
• Average course final score: this metric gives hints of the difficulty of quizzes for a particular group of
students and can be compared to satisfaction questions from the student survey.
• Average module correct questions: this metric stablishes which module has been more difficult for the
students.
• Average correct B questions: this metric is useful to measure the general level of the group. B questions
should be correctly answered by most of the students. If not, it could indicate a low level of competition
or a proof of general disengagement.
73
• Average correct A questions: this metric is useful to measure if the lectures are of special interest to the
students and how they have prepared for the quizzes.
• Average correct P questions: if high, this metric could indicate an especially relevant level of interest of
the students to a particular course.
Fidelity
• Number of students with all quizzes completed: this metric stablishes how many students did all the
activities and shows if the gamification approach is interesting for the students.
• Number of students completing each quiz: together with score metrics, this parameter can help to
determine if disengagement is due to difficulty and the loss of possibilities to win or is merely lack of
interest.
• Score progression of each student: particularized to each student, this metric allows to study
improvements in individual performance.
• Average position of each student: this metric allows evaluating how regular are students along the
course.
3 Future developments
The gamification approach will be applied to the academic year 2022/2023. After evaluating the results, if
positive, a pilot gamification approach will be proposed in combination with the Kahoot! quizzes in the course
Robot Control and Programming within the MSc of Automation and Robotics. This course in particular is
shorter than the other two and intensive, meaning that in only two weeks all the contents are given. Our
proposal will include several elements to engage students:
• Badge system: we will incorporate a badge system to evaluate achievements during the course such as
outstanding scores in specific theory and practical assignments or participation in class.
• Video-based lectures: in person lectures will be changed to video lectures and a flipped classroom
approach will be introduced for the theory assignments.
• Free-to-choose itinerary: course modules will have a recommended order, but students will be free to
choose which module order fits better their own interests.
4 Conclusion
This paper has described the organization of the activities, the design of the Kahoot! quizzes, the tentative
student survey to evaluate engagement and satisfaction and the quantitative assessment metrics of a
gamification approach to engineering courses. The main goal of this proposal is to evaluate how the
gamification activities are affecting weekly engagement in the course contents and does not focus specifically
in the effects on course performance. If the application of this approach is successful, more gamification
elements are intended to be included in a pilot engineering course.
5 References
Chacko, P., Appelbaum, S., Kim, H., Zhao, J., & Montclare, J. K. (2015). Integrating technology in STEM education. Journal of Technology
and Science Education, 5(19; 5-14.
Chernov, V., Klas, S., & Shaharabani, Y. F. (2021). Incorporating Kahoot! in core engineering courses: Student engagement and performance.
Journal of Technology and Science, 11(2).
Conolly, T. M., Boyle, E. A., & MacArthur, E. (2012). A systematic literature review of empirical evidence on computer games
and serious games. Computers and Education, 59(2), 661-686.
Cózar-Gutiérrez, R., & Sáez-López, J. M. (2016). Game-based learning and gamification in initial teacher training in the social
sciences: an experiment with MinecraftEdu. International Journal of Educational Technology in Higher Education,
13(2), 1-11.
74
DiGiacomo, D. K., Greenhalgh, S. & Barriage, S. (2022). How students and principals understand ClassDojo: Emerging
insights. TechTrends, 66, 172-184.
Meishar-Tal, H. (2015). Teachers’ use of Wikipedia with their Students. Australian Journal of Teacher Education, 40(12).
Pertegal-Felices, M. L., Jimeno-Morenilla, A., Sánchez-Romero, J. L., & Mora-Mora, H. (2020). Comparison of the effects of
the Kahoot tool in teacher training and computer engineering students for sustainable education. Sustainability,
12(11), 4778.
Sarkar, N., Ford, W., & Manzo, C. (2017). Engaging digital natives through social learning. Systemics, Cybernetics and
Informatics, 15(2), 1-4.
Waghid, F. (2015). On the democratisation of science education through Facebook: Implications for autonomy, equality and
teacher education at universities. South African Journal of Higher Education, 29(2), 298-314.
Welbers, K., Konijn, E. A., de Vaate, A. B., Eden, A. & Brugman, B. C. (2019). Gamification as a tool for engaging student
learning: A field experiment with a gamified app. E-Learning and Digital Media, 16(2), 92-109.
75
Exoforge: Interdisciplinary teaching laboratory for the
development of assistive technology projects
Lluis Bernat, Carlos A. Jara, Jose L. Ramon, Jorge Pomares, Gabriel J. Garcia, Andres Ubeda
Human Robotics Group, Department of Physics, Systems Engineering and Signal Theory, University of Alicante, Alicante, Spain
Email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Abstract
Today, the aging of the population and the prevalence of neuromuscular diseases make it increasingly necessary to develop
technologies that improve the quality of life of these users and optimize the physical rehabilitation process. The initiative
"Exoforge: Interdisciplinary teaching laboratory for the development of assistive technology projects" is proposed as a
complementary training tool for engineers in the field of health technologies to cover these needs from a multidisciplinary
orientation. From the teaching point of view, this initiative tries to address two essential aspects in the training of future
engineers who work in the field of assistive technologies: multidisciplinarity, through the incorporation of students from
different degrees such as robotics engineering, biomedical engineering, or multimedia engineering; and job orientation
and entrepreneurship, through the development of projects with the possibility of future transfer to a company and useful
from a social point of view. This paper also describes the evaluation of the Exoforge initiative. From its creation, Exoforge
has launched two different projects: ARMIA, a sensorized sleeve for upper limb rehabilitation; and ARES, a lower limb
robotic exoskeleton. The main objective is to establish whether the initiative provides useful skills and tools to students
within a complementary educational framework such as internships in laboratories at the University of Alicante or, where
appropriate, the development of Undergraduate Thesis and Master's Dissertations related to the proposed projects.
Keywords: Active Learning; Engineering Education; Project Approaches.
1 Introduction
The number of people with congenital and/or acquired disabilities in today's Spanish society represents a large
number of dependent people who lack the necessary autonomy for a fully independent life. The health
technology sector is allocating a large part of its funding to provide the highest possible quality of life for these
disabled people and the elderly. One of the most funded topics is the rehabilitation of the affected patient.
However, according to the WHO (World Health Organisation), existing rehabilitation services have been
disrupted in 60-70% of countries due to the Covid-19 pandemic. This disruption, and the need to avoid visits
and attendance of patients in hospital or rehabilitation centres, reflects the importance of having competitive,
reliable and accessible rehabilitation, tele-rehabilitation and home-based rehabilitation systems.
Exoforge is an interdisciplinary teaching laboratory for the development of projects in assistive technologies.
It is a multidisciplinary laboratory in which students from different degrees develop projects in assistive
technologies. Exoforge is a proposal that emerged within the Human Robotics research group at the University
of Alicante during the 2020-21 academic year. Students participate in Exoforge through their internships as
part of the curriculum of their own university degree. The participating students can come from the Degrees
in Biomedical Engineering, Robotics Engineering and Multimedia Engineering, as well as from the Master's
Degree in Automatics and Robotics. The main objectives of the Exoforge intiative are:
1. To develop a multidisciplinary complementary training experience for students in their final years of
undergraduate and master's degrees.
2. To propose bridging experiences between educational sphere, and business and research spheres.
3. To promote innovation and entrepreneurship among students of the University of Alicante Degrees.
4. To enhance general skills such as teamwork, decision-making, personal initiative and social values and
specific skills in different branches of knowledge.
Students at the Polytechnic School of the University of Alicante always have the possibility to do an internship
in a company in the syllabus of their degree courses. Within the research groups of the University of Alicante,
76
students are offered the possibility to carry out these internships within the framework of current research
projects. From the Human Robotics research group, we think that the best way to carry out these internships
is through real projects that could be proposed by any company in the sector. In order to complete these
projects, it is necessary to form a group of students, with great potential and who come from different degrees,
that can collaborate to complement each other. The result of this proposal is Exoforge, which is similar to a
company that "hires" the students. Within Exoforge, new projects are proposed every year. In the first year,
Exoforge launched the ARMIA project. ARMIA is a project that develops a new wearable and wireless
rehabilitation device based on an upper limb sleeve with inertial and electromyographic sensors. This device
allows recording the patient's kinematic and muscular information during rehabilitation tasks and introduces
telerehabilitation, a new paradigm in the process of recovering upper limb motor function. This device has
significant advantages compared to other devices for motor rehabilitation. It includes low cost, small size, light
weight, and wireless portability components. In the current academic year (2021-22), the development of
ARMIA has continued, while a new project has been launched: ARES, a robotic exoskeleton for the lower limb.
These projects are related to the research lines of the Human Robotics research group. They are projects in
which the professors of the research group can act as leaders and advisors at the same time. The idea is to take
advantage of the Project Based Learning (PBL) methodology to enhance students' learning in a real project
that they will be able to find in any company after graduation.
PBL is not a new teaching methodology. In (Postman, 1969) a teaching model was proposed in which lectures
were abandoned and students developed their creative abilities through open questions and problems. These
ideas were applied for the first time at the Mc Master University (Canada) and at the Case Western Reserve
University School of Medicine (USA), where the name of problem-based learning methodology appeared for
the first time. This method quickly spread through European Universities in the 1970s. It is in this decade when
the Danish University of Aalborg developed a new method derived from problem-based learning: project-
based learning (PBL). Currently, the PBL is considered one of the most suitable methods for the new higher
education models based on active learning (Guo, 2020; Bittencourt, 2018; Guerra, 2017). With this
methodology, students must assume greater responsibility, and obtain freedom of action. They will go through
an active learning process that is necessary to solve the projects proposed by the teacher. PBL-based teaching
is based on the development of a project that sets goals, such as the development of the final product, which
achievement will require the learning of technical concepts and attitudes. The PBL methodology will only be in
tune with the objectives of the European Higher Education Area (EHEA) if the student takes an active role in
their learning process.
One of the main advantages of the PBL methodology is that it is developed in a real and experimental
environment. This characteristic helps students to relate the theoretical contents with the real world, thus
improving the acquisition of theoretical concepts. At the same time, the student takes an active role in the
project and sets the pace and depth of their own learning, which makes this methodology perfectly applicable
to groups with disparate base knowledge. The PBL motivates students, therefore, it can be considered as an
instrument to improve academic performance and persistence in studies. Furthermore, the PBL creates an ideal
framework to develop various transversal competences such as teamwork, planning, communication, and
creativity (Lima, Dinis-Carvalho, Flores, & Hattum-Janssen, 2007; Powell & Weenk, 2003).
Only few works about PBL methodology in internship works are found in the literature. In (Johari, A., &
Bradshaw, A. C., 2008) authors considered the roles of task, learner, and mentors as they are needed to make
the most of project-based internship programs. Implications for the design and development of internship
programs, and specifically successful student performance in internship programs, were also considered.
Another paper describes G-DORM (Ueda, Y., et al., 2018). G-DORM is a student exchange project that has
global PBL in an internship by Niigata University with four universities in Mekong countries. The paper describes
the achievements of G-DOMR carried out in 2017, in terms of coordination with companies for program
designing, recruiting and selection of students, and implementation of internship in the student exchange
program and evaluation.
77
2 Methodology
This section describes the methodology used in Exoforge during the last two years. Firstly, the students
participating in each of the courses are described, detailing the grade they are studying. Next, the two projects
launched during each course are described in more detail: ARMIA and ARES.
2.1 Participants
The students who have participated in the two courses in which Exoforge has offered internships come from
the degrees taught at the Polytechnic School of the University of Alicante. The professors of the research group
promoting Exoforge, Human Robotics, mainly teach on the Robotics Engineering Degree. That is why most of
the students interested in participating in Exoforge come from this degree. They also teach on the Biomedical
Engineering Degree, the Multimedia Engineering Degree and the Master's Degree in Automation and Robotics.
In the 2020-21 academic year, Exoforge launched the ARMIA project. The aim of this project is to develop a
sensorised sleeve that allows rehabilitation exercises to be carried out at a low cost and autonomously. The
project mainly attracted the attention of robotic and biomedical engineers. The selection was made mainly
based on the dossier provided by the candidates. Table 1 shows the number of students interested in
participating, and the final number of students, from each grade who ended up participating in Exoforge during
the 2020-21 academic year.
Table 4. Entry profile of students enrolled in the 2020-21 academic year in Exoforge.
Number of Number of
Degree
candidate students selected students
Robotic Engineering Degree 15 5
Biomedical Engineering Degree 8 2
Computer Science Engineering Degree 1 0
Master’s Degree in Automation and Robotics 1 1
In the academic year 2021-22, Exoforge started by offering the extension of the ARMIA project. We already
had a valid prototype, but we wanted to make progress in obtaining a textile sleeve that the user who was
going to carry out the therapy could wear on his or her arm. In addition, the aim was to develop a serious
game that, using the signals measured in the sensorised sleeve as input, would allow the exercises to be
adjusted to the physical and mental fatigue of the user in the rehabilitation activities. This attracted students
from the Multimedia Engineering Degree who had not shown interest in the first-year call. In addition, Exoforge
has launched the ARES project for this academic year 2021-22, which consists of the design, simulation, and
implementation of a knee exoskeleton. This other project is more robotic and attracted many students from
the Robotics Engineering Degree. It also requires knowledge of biomechanics, which can be provided by
Biomedical Engineering students. Table 2 shows the students interested by degree in participating in Exoforge
during this academic year 2021-22, as well as the students who have finally participated in the two projects.
Table 2. Entry profile of students enrolled in the 2021-22 academic year in Exoforge.
Number of Number of
Degree
candidate students selected students
Robotic Engineering Degree 12 8
Biomedical Engineering Degree 6 4
Multimedia Engineering Degree 3 2
2.2 ARMIA project

ARMIA is a sensorized sleeve that allows measuring kinematics and muscular activity during upper limb
movements. To determine arm kinematics three inertial sensors are placed on the kinematic chain: one on the
78
thorax (reference), one on the arm and one on the forearm. These last two sensors provide arm links orientation
and elbow and shoulder joint angles. ARMIA does not provide wrist orientation, but it is possible to obtain
Cartesian coordinates of the arm end effector (hand).
Muscular activity is measured from three of the main muscles in charge of upper limb movements: biceps,
triceps and pronator teres. These sEMG sensors are used to determine the signal amplitude (contraction force)
and other factors such as muscular fatigue. The instrumentation of these sensors is critical, as the bipolar
electrodes must be placed parallel to the muscle fibers and close to the middle of the muscle belly, and the
reference electrode must be firmly in contact with the skin.
ARMIA software is structured into three different levels. The low-level layer stab-lishes the physical
communication protocols between the different sensors and the microcontroller, and the wireless
communication between the microcontroller and the computer. The medium-level layer manages the data
acquisition and storage from the sensors. Finally, the high-level layer develops processing and visualization
modules.
During the 2020-21 academic year, a first ARMIA prototype was developed (see Figure 1). During the academic
year 2021-22, work is being carried out on a textile design for the wearable, as well as on a virtual reality
application that will allow the user to carry out rehabilitation exercises.
Figure 1. ARMIA prototype.

During the first course (2020-21), the students assigned to the ARMIA project were separated into three
different working teams:
• Design Team (2 robotics engineering and 1 Master’s Degree in Automation and Robotics students):
o Design and implementation of the initial prototype.
• Software Team (3 robotics engineering students):
o Acquisition and communication.
• Neuromechanics Team (2 biomedical engineering students):
o Biomechanical study from inertial and sEMG sensors.
During this course (2021-22), the students assigned to the ARMIA project have been separated into two
different working teams:
• Design Team (3 robotics and 2 biomedical engineering students):
o Adaptation of the first prototype to a textile wearable sleeve.
• Software Team (2 multimedia engineering students):
o Design and development of a VR serious game.
2.3 ARES project

ARES is a project that proposes the design and development of a lower limb exoskeleton. The ARES exoskeleton
was created with the aim of giving users with various disabilities the opportunity to regain their mobility,
79
improve their quality of life and increase their autonomy. Specifically, the exoskeleton has been designed to fit
a patient's leg to allow and facilitate their movement. The exoskeleton integrates inertial and
electromyographic sensors to detect the signals that the brain sends to the leg. The exoskeleton processes
these signals to be able to move along with the limb and, in this way, to assist the patient during walking.
The main features of the ARES device are defined below:
• Robotic exoskeleton adaptable to different leg morphologies.
• Full integration of low-cost inertial and electromyographic sensors in the device.
• Control of the exoskeleton based on ROS (Robot Operating System).
• Acquisition and calibration software for intuitive configuration and use.
• Biomechanical study of human gait for planning valid trajectories for the exoskeleton.
• Design of gait patterns to assist the patient in the development of different types of movements.
The ARES device is designed to be used both by private users to recover their mobility and autonomy, and by
hospitals, clinics, therapy centers and nursing homes. ARES is a lower limb exoskeleton designed for the
rehabilitation of patients with different types of disability such as spinal cord injuries, neurodegenerative
diseases or those who have suffered cerebrovascular accidents. In this first phase of development, the
exoskeleton has a motorized kneecap with the aim of enabling functional recovery of the knee and thus
improving their autonomy.
The ARES exoskeleton adjusts to different anthropometric measurements thanks to its adaptable design. It
features an adjustable knee joint extension angle, allowing the full range of knee joint movements to be limited
depending on the patient's characteristics. In addition, the adjustment is simple, and the fitting, adjustment
and set-up can be done in a few minutes.
In addition, the ARES exoskeleton can be used as a rehabilitation therapy to complement traditional
rehabilitation by generating constant gait patterns. Its use can contribute recovering the mobility of the knee
or to maintain its functional capabilities as long as possible.
A first working prototype of the exoskeleton is now available (see Figure 2). All the software development of
the project is being carried out in ROS. The use of ROS, and open-source software, allows the integration of
previously existing developments in perception, planning and artificial intelligence.
Figure 2. ARES prototype.

The students assigned to the ARES project have been separated into three different working teams:
• Neuromechanics Team (2 biomedical engineering students):
o Study of lower limb neuromechanics and design of motor movement patterns.
• Design Team (2 robotics engineering students):
o Adaptation of the orthosis for the motor, design and 3D printing of the robot-orthesis motion
transmission.
80
• Control and actuation Team (3 robotic engineering students):
o ROS control and actuation of the Maxon motor via Ethercat, and motion pattern tracking.
3 Exoforge evaluation
For the drafting of the survey to evaluate different aspects of the methodology, we first thought of the items
that were most interesting to know. A very important aspect to know was the group cohesion achieved, since
all the work for the development of the project is divided into different work teams. Students were also asked
for their opinion on whether they found the Exoforge experience useful, or whether it allowed them to apply
the knowledge acquired in their degree, if the motivation has been increased or not, etc. Another important
question to be evaluated is whether their time at Exoforge has given them any ideas for the development of
their final degree project. The questions asked can be seen in Table 3. The survey was conducted through a
questionnaire using Google Forms. This survey was anonymous, although as described in Section 2.1, the total
sample is known, given that they are the students who have participated in the Exoforge experience. Although
18 students participated in the experience, only 11 answered the survey.
Table 3. Survey for the Exoforge initiative student opinion.
Q1 Do you feel you have learnt anything from working at Exoforge? Yes No
Q2 Have you been able to apply concepts seen in any of the subjects of the Degree in Exoforge? Yes No
Q3 Do you consider that the time spent on external internships is sufficient to carry out your tasks Yes No
within Exoforge?
Q4 Do you consider that the guidance provided by the teachers for the resolution of the project has Yes No
been adequate?
Q5 Doing my external internship at Exoforge has allowed me to define an idea for my Final Degree Yes No
Thesis dissertation
Rate from 1 (a little) to 9 (a lot):
Q6 The work methodology proposed in Exoforge has made me feel motivated. 1 2 3 4 5 6 7 8 9
Q7 The work methodology proposed at Exoforge promoted teamwork. 1 2 3 4 5 6 7 8 9
Q8 I have learned how to make a detailed planning of the time to be spent on each 1 2 3 4 5 6 7 8 9
of the tasks required to complete the project
Q9 I have learnt how to carry out an analysis and design prior to project 1 2 3 4 5 6 7 8 9
implementation
Q10 I have learnt to search for information on my own for the implementation of the 1 2 3 4 5 6 7 8 9
project
Q11 I have learnt to work in a group and actively participate in the project 1 2 3 4 5 6 7 8 9
Q12 I am satisfied with the work we can do in the group formed in Exoforge 1 2 3 4 5 6 7 8 9
Q13 I like the working atmosphere of the group formed in Exoforge 1 2 3 4 5 6 7 8 9
Q14 I am satisfied with the amount of work being done at Exoforge 1 2 3 4 5 6 7 8 9
Q15 I am satisfied with the ideas that the members of the Exoforge group bring to 1 2 3 4 5 6 7 8 9
the table when we work together
Q16 When one member of Exoforge's group doesn't understand something, the 1 2 3 4 5 6 7 8 9
others try to explain it to him or her ---
Q17 The members of my group prefer to work together rather than working 1 2 3 4 5 6 7 8 9
individually
Q18 Doing my internship at Exoforge gives me a work experience close to reality 1 2 3 4 5 6 7 8 9
81
The results of the survey are very encouraging. We will now review the results for each question. In the first
question, the students unanimously indicate that they have learned something during their time at Exoforge
(100%). Also, all students (100%) say that they have been able to apply the concepts seen in their degree
courses during the 150 hours they have spent on the external internships carried out at Exoforge. Once again,
all the students (100%) who answered the survey indicated that the 150 hours were sufficient to carry out the
tasks that were planned at the beginning of the internship at Exoforge. With regard to question 4, the students
also unanimously indicated that the lecturers have given them adequate guidance in the development of the
proposed project (100%). Question 5 is the last Yes/No question. The idea behind this question was to obtain
information on whether the students have seen or detected any ideas for a final degree project to be carried
out based on the developments made in Exoforge. More than half (54.5%) answered in the affirmative, which
seems to us a very interesting aspect, as Exoforge can become a seed for the development of small individual
projects that enhance the experience started in the internships.
The first block of questions (Q6, Q8, Q9, Q10, Q14, Q18) with 11 possible Likert-type answers deals with the
opinion of each student's individual work in Exoforge. For each question, students answer a value from 0 to 10
indicating a little (0) to a lot (10). Figure 3 shows the answers obtained for this first block of Likert-type
questions. The answers are positive for all the questions in this block. Specifically, when asked whether the
Exoforge methodology has motivated them, the majority indicate values between 8 and 10 (90.09%), with one
isolated case indicating 5. All students indicate that the methodology used has helped them to some degree
to learn to plan the time used for each task (with 100% indicating values between 7 and 10). The same happens
when questioned about whether the methodology has helped them learn about the importance of prior design
and analysis before starting to programme or develop tasks (also with 100% in values between 7 and 10,
although in this case more than half 6/11 voted values of 9 and 10). Encouraging responses were also seen in
question 10. The students claim to have learned to search for information on their own to solve the various
tasks assigned in the project (7/11 voted values of 9 and 10). In question 14, the majority of students say that
they are satisfied with the work carried out in Exoforge (9/11 voted 10, the other 2 voted 8). Finally, they were
asked whether they think that the work done at Exoforge gave them an experience as close as possible to what
they might find in the world of work. This is where we found the greatest disparity in the responses, although
always with positive values (between 5 and 10).
Figure 3. Answers for Questions 6, 8, 9, 10, 14 and 18 of the survey.
The second block of questions (Q7, Q11, Q12, Q13, Q15, Q16 and Q17) with 11 possible Likert-type answers
deals with group cohesion and group functioning. Figure 4 shows the answers obtained for this second block
of Likert-type questions. Again, the answers are positive for all the questions in this block. To begin with, in
question 7, 9 of the 11 students who filled in the survey say that working at Exoforge encouraged teamwork
(they voted values of 9 and 10). Of particular interest is question 11, where all students indicate that they have
learned to work in a team because of their experience at Exoforge (all voted 8-10). The same is true for question
12, with all 11 students voting between 8 and 10, with 7 of the 11 voting 10 on the question of whether they
82
are satisfied with the work that can be done by the group formed in Exoforge. The working atmosphere is
undoubtedly the most highly valued aspect of the survey, with 9 of the 11 students voting 10, and the other
two, 8 and 9. The ideas contributed by the other members of the group to solve the problems encountered
throughout the project are also highly valued (9/11 voted 9-10). Question 16 asks about companionship, asking
whether the members of the group help someone who has not understood concepts to understand them (all
students voted 8-10). Virtually all students indicate that others prefer to work in groups rather than individually
(only one student voted 4, the rest 8-10).
4 Conclusion
The Exoforge initiative has allowed students from the Polytechnic School of the University of Alicante to carry
out their internships for two academic years in technological projects for the rehabilitation of upper and lower
Figure 4. Answers for Questions 7, 11, 12, 13, 15, 16, and 17 of the survey.
limbs. The result of the first course (2020-21) was a success, obtaining a valid prototype of a sensorised sleeve
that allows obtaining kinematic and electromyographic information during the rehabilitation task. In addition,
this project, the ARMIA project, won the Impulso 2021 Prize for Accessibility, awarded by the University of
Alicante. In this second course of the Exoforge initiative, ARMIA continues to develop the initial prototype to
make it more "wearable", while a serious game for virtual reality is being developed to gamify the rehabilitation
therapy exercises. In addition, during this academic year, the ARES project has been launched, in which a first
functional prototype of a low-cost knee exoskeleton has already been obtained. This exoskeleton is controlled
by free software and has a series of sensors that allow estimating the user's movement intention to improve
the quality of rehabilitation and/or walking assistance. The survey carried out among the participating students
shows excellent results. A great group cohesion in teamwork is observed, as well as great satisfaction in
participating in the projects offered by Exoforge. More than half of the students are even considering
continuing to develop tasks for Exoforge as part of their own final degree projects.
83
5 References
Bittencourt, A. C., Diniz, A. C., & Macedo S.C. (2018). A review of Problem/Project-based learning approach in engineering education:
motivations, results and gaps to overcome. In: PAEE/ALE, 2018, Brasília. International Symposium on Project Approaches in
Engineering Education, 8, 302-308.
Guerra, A., Ulseth, R., & Kolmos, A. eds. (2017). PBL in Engineering Education. Sense Publishers, Rotterdam.
Guo, P., Saab, N., Post, L. S., & Admiraal, W. I. (2020). A review of project-based learning in higher education: Student outcomes and
measures. International Journal of Educational Research, 102, art. no. 101586.
Johari, A., & Bradshaw, A. C. (2008). Project-based learning in an internship program: A qualitative study of related roles and their
motivational attributes. Education Tech Research Dev, 56, 329-359.
Lima, R. M., Dinis-Carvalho, J., Flores, M. A., & Hattum-Janssen, N. v. (2007). A case study on project led education in engineering: students'
and teachers' perceptions. European Journal of Engineering Education, 32(3), 337-347.
Postman, N., & Weingartner, C. (1969). Teaching as a subversive activity. New York: Dell Publishing Co.
Powell, P. C., & Weenk, W. (2003). Project-Led Engineering Education. Utrecht: Lemma.
Ueda, Y., Tsuboi, N., Suzuki, T., Abe, K., Suzuki, T., Kourakata, I., & Nakano, S. (2018). W-06 Value Creation by Global Project-based Learning
in Internship: Practices in G-DORM 2017, JSEE Annual Conference, Session ID W-06, 34-39.
84
Exploration of the use of Project Based Learning (PBL)
methodology in two accredited engineering programs at the
Autonomous University of Yucatán
Miriam V. Chan-Pavón1,2, Jesús F. Escalante-Eúan1, Ileana C. Monsreal-Barrera1, Carlos M. Rubio-Atoche1

1
Group of Production and Logistics Systems, School of Chemical Engineering, Autonomous University of Yucatan, Yucatán, México
2
Estudiante de Doctorado en Informática, Universidad Americana de Europa, México-España.
Abstract
The aim of this work was an exploratory qualitative investigation of the use of the PBL methodology in two currently
accredited undergraduate programs at the Faculty of Chemical Engineering of the Autonomous University of Yucatan:
Industrial Chemical Engineering (IQI) and Industrial Logistics Engineering (IIL). Project Based Learning (PBL) is considered
today as one of the most effective methods in engineering education, because it allows students to develop competencies
and skills that are increasingly demanded by organizations to new professionals and trainees. The methodology consisted
of consulting the programs of the mentioned degrees and courses, in particular the didactic plans elaborated by the
professors of each subject in order to select those that use project-based learning as an evaluation instrument. An
exploration instrument was applied to students of the last semesters, with the purpose of knowing if this methodology is
reaching its objectives and if the student has knowledge of the project-based learning methodology. The results showed
that 72% of the curricula of the Industrial Logistics Engineering career apply this methodology (PBL) and 52% in Industrial
Chemical Engineering. In conclusion, it was observed that a significant number of subjects in the undergraduate courses
studied use project-based learning as a product for evaluating course performance; however, the PBL methodology is used
empirically.
Keywords: Project Based Learning; Programs; Didactic plans; Accredited; Engineering.
1 Introduction
Traditionally, education has been fundamentally centered on teaching, and with it on the teacher as the axis of
the teaching-learning process. Now, this change entails giving students a greater role, putting all efforts into
their learning, relieving the role of the teacher to that of guide and orientor. Thus, the more behaviorist and
cognitivist theories give way to constructivist theories, developing meaningful learning in students (Carrasco,
Donoso, Duarte-Atoche, Hernández & López, 2015; Muñoz & Díaz, 2009; Navarro, Pertegal, Gil, González &
Jimeno, 2011), in which the knowledge they already possess is integrated with the new knowledge they must
acquire, making what they have learned last over time. It is not so important what the student knows at a given
moment, but what he/she may come to know, which is closely related to his/her learning capacity (Estruch and
Silva, 2006). More important than learning content is learning to learn, fostering continuous learning (Reverte,
Gallego, Molina and Satorre, 2006; Taboada, Touriño and Doallo, 2010). Learning can be seen as a cumulative,
self-regulated, directed, collaborative and individual process (Van den Bergh et al., 2006).
Project-based learning (PBL) appears to be an effective teaching method compared to traditional cognitive
teaching strategies, particularly for the development of real-life problem-solving skills (Willard & Duffrin,
2003).
Project-based learning is a methodology that is developed in a collaborative manner that confronts students
with situations that lead them to put forward proposals to address certain problems. By project we understand
the set of activities articulated with each other, in order to generate products, services or understandings
capable of solving problems or satisfying needs and concerns, considering the resources and time allocated.
Authors and researchers who propose competency-based models in education consider that the project is an
integrative strategy par excellence, and that it is the most appropriate to mobilize knowledge in situation (Díaz
Barriga 2015; Jonnaert et. al. 2006). In this way, students can plan, implement and evaluate activities with
purposes that have real-world application beyond the classroom.
85
Project-based learning has many benefits, among others, the possibility of continuous feedback and evaluation
by the teacher, the establishment of a schedule of activities that show the progress of the different groups,
generating a space for reflection by the student, preparing them for their professional and labor future,
increasing motivation and involvement of both students and teachers. It increases their social and
communication skills, improves problem solving and they learn to work in groups (Collazos, 2009). In addition,
it promotes creative thinking and decision making, fostering higher academic performance. Bearing all the
above in mind, it can be stated that PBL facilitates the learning of new knowledge and allows the application
of the already acquired knowledge, develops transversal skills, including planning, writing, communication, as
well as the responsibility to face a real situation (Carraco et al., 2015).
Nowadays, society requires an innovative engineer, bold in experimentation, with skills of interaction and
exchange of ideas with other professionals from different areas (Duque & Martínez, 2000). This implies the
establishment of a solid academic-cultural community, which breaks with the mental schemes that generate a
presumed separation between scientific knowledge and humanistic knowledge. There is no point in training
academic engineers who are alien to human sensitivity, just as there is no point in graduating artists without
any hint of scientific rigor. Engineering is the conceptualization, design, construction and administration of
projects and products aimed at providing a solution to a need of society or the environment. For this reason,
the engineer must solve problems or provide different solutions, which requires imagination, creativity and
synthesis of knowledge (Duque et al., 1999). Engineering, in general, is a decision-making process for the
solution of problems within a particular field of action. This decision making involves different steps, among
which the following stand out: delimiting the situation, proposing a solution strategy, obtaining experimental
or theoretical information, analyzing the data and results, selecting the valuation criteria for the possible
solutions, choosing the optimum variable and correcting the decision during its implementation (Garza-Rivera,
2001).
In response to the national and global needs of the integral formation of an engineer, capable of developing
in the labor field, so that he develops skills and abilities to solve problems or provide different solutions, given
this is why the Autonomous University of Yucatan (UADY), prioritizes the continuous professional training of
students, through its educational programs taught in the Faculty of Chemical Engineering (FIQ), making use in
recent years of the PBL methodology, as a method to obtain a competent learning for students who graduate
and can perform satisfactorily in the labor field. Therefore, it is necessary to know and identify the results,
effectiveness and contribution of the project-based learning methodology in the accredited programs of the
faculty, taking as a sample the last semesters of the degrees of; Industrial Chemical Engineering and Industrial
Logistics Engineering.
2 Objective
• Identify which subjects of the curriculum of each degree program declare in their didactic plans the
use of the PBL methodology as an evaluation tool.
• Determine if students know the project-based learning methodology and if they have used it in any
subject in order to know the impact and effectiveness it has had in the acquisition of new competencies
for professional development.
3 Methodology
Considering that the research is qualitative and exploratory, it was established that the sample size for this
exploration is 30 cases (Hernandez, Fernandez and Baptista, 2010), by degrees made up of eighth and tenth
semester students, i.e. there will be 30 cases in total for students of the degree of Industrial Chemical
Engineering and Industrial Logistics Engineering.
The first part of the research consisted of the exploration of the descriptive letters of the two accredited
bachelor's degrees under study, which were consulted through the official website of the Faculty of Chemical
Engineering of the UADY, which is shown in the following figure:
86
Figure 1. Portal de los planeaciones didácticas de los programas analizados
Subsequently, the results obtained from the descriptive charts consulted were compared with those found in
the Moodle platform, which is used to support face-to-face classes by both teachers and students. On this
platform, students can consult the guidelines for each of their subjects, including the topics, learning
activities and the final evaluation method of the course.
The second part consisted of the application of a survey-type evaluation instrument using a Likert scale to the
students to determine if they know and use project-based learning and how it impacts their training and the
acquisition of skills.
The survey contains eight questions and was created using the Google Drive forms tool to make it easy to
apply via email. Two important sources were used as a guide for the creation and adaptation of the evaluation
survey (Carrasco, A., et al. (2015) and Rodríguez A., Río R, and Larrañaga J. (2016)). It should be noted that the
exploratory research includes as a sample only students from the last two years of the degree.
3.1 Survey
1. ¿Have you heard of the Project-Based Learning (ABP or PBL) methodology?

• Yes
• No
2. Select the subjects in which you carried out a project in a real company:
▪ for IIL students the possible options were:

• Safety and industrial hygiene
• Methods Engineering
• Project management
• Supply chain management systems
• Warehouses and inventories
• Formulation and evaluation of processes
• Supplying
• Marketing and customer service
• Planning and control of operations
• Strategic planning
▪ for IQI students the possible options were:
• Integrative project I
• Integrative Project II
87
• Industrial engineering
• Economic engineering
• Process control
• Service engineering
• Process design
• Process integration
• Safety and industrial hygiene
• Separations by continuous contact
• Fundamentals of industrial engineering
3. ¿Mention if you have taken the (PBL) in subjects other than those selected above?
▪ Open answer
4. ¿Do you think that teamwork allows you to increase your motivation to learn or do you prefer to work
individually?
▪ Yes
▪ No
▪ I prefer work individually
5. ¿ Your teachers were available to give you support and help you solve any doubts about the project?
▪ Totally agree
▪ I agree
▪ Neither agree nor disagree
▪ Disagree
▪ Strongly disagree
6. ¿What do you think was the biggest obstacle you had to overcome to carry out a project of this type?
▪ Communication skills
▪ Time
▪ Little access to the company
▪ Unknowing of the objectives
▪ Others
7. ¿ What kind of skills do you consider to have developed in the projects in which you were involved?
▪ Leadership
▪ Critical thinking
▪ Autonomous learning
▪ Conflict resolution
▪ Collaboration
▪ Creativity
▪ Troubleshooting
▪ Listen active
8. ¿Do you feel that the projects are relevant to your professional future?
▪ Totally agree
▪ I agree
▪ Neither agree nor disagree
▪ Disagree
▪ Strongly disagree
4 Results
4.1 Analysis of the didactic plans
Once the exploratory research was initiated, the descriptive letters and didactic plans of the five undergraduate
courses offered by the School of Chemical Engineering were consulted, with the intention of finding out which
subjects of their curricula declared the use of Project Based Learning, and the results were the following.
88
The first point was to determine the total percentage of the use of this methodology at a global level, that is
to say, considering all the subjects taught in the five degree courses. It was found that 53% of the subjects
contemplate the implementation of PBL within their descriptive charts, according to the figure below:
Figure 2. Overall percentage of subjects that reported using PBL as a learning technique..
Subsequently, of the total number of subjects that apply the PBL, the total percentage that corresponds to
each degree program was explored, that is, out of a total of 123 subjects that apply this methodology to
evaluate, the Industrial Logistics Engineering (IIL) degree program has 28%, resulting in the fact that,
comparing all the degree programs, this is the one that applies this methodology the most, and Industrial
Chemical Engineering (IQI), 21%, placing it in third place. This distribution is shown in the following figure:
Figure 3. Percentage of subjects that have reported using PBL as a learning technique, in each of the degree programs
offered..
Taking these results into account, we proceeded to explore the undergraduate programs being evaluated,
which are currently accredited by the faculty. Beginning with the analysis of the Industrial Logistics
Engineering degree, the figure below shows the total percentage of use of this methodology by subject. This
analysis shows that 72% of the subjects in its curriculum implement Project Based Learning. That is to say, 34
subjects state in their descriptive letters and didactic plans that they evaluate student learning through this
study method, of the 47 that make up the curriculum.
89
Figure 4. Percentage of subjects that have reported using PBL as a learning technique in the Industrial Logistics Engineering
plan.
In order to have a clearer idea of the distribution throughout the proposed training, the following graph
shows the number of subjects that make use of it by semesters of the bachelor's degree under study.
Figure 5. Percentage of subjects that have reported using PBL as a learning technique in the Industrial Logistics
Engineering plan, by semester.
Continuing with the analysis, we move on to the Industrial Chemical Engineering degree. In the figure below
we can see that 52% of the subjects in its curriculum state that they implement this Project Based Learning
methodology as an evaluation method. That is to say, out of its 50 mandatory subjects, 24 declare that they
use this methodology.
90
Figure 6. Percentage of subjects that have reported using PBL as a learning technique in the Industrial Chemical
Engineering plan.
In order to have a clearer picture of the distribution, the figure below shows the number of subjects that
make use of the methodology (PBL) per semester, in the case of Industrial Chemical Engineering. It can be
seen that the semester in which this evaluation method is most used is the eighth semester.
Figure 7. Percentage of subjects that have reported using PBL as a learning technique in the Industrial Chemical Engineering
plan, by semester..
Once the results of the exploration of the descriptive letters and the didactic plans were obtained, the
evaluation instruments were applied in order to measure their effectiveness and the impact they are having..
4.2 Analysis of student perceptions

Figure 3 shows the percentage of knowledge that students have in the use of this methodology. In the case of
the students of the Logistics Industrial Engineering program (IIL), 58.9% know the PBL methodology, as well as
53.4% of the Chemical Engineering students (IQI).
91
Figure 8. Results of question 1 regarding knowledge of the methodology for IQI and IIL respectively.
The subjects that apply the PBL technique. For the Industrial Logistics Engineering (IIL) degree, Methods
Engineering, Project Formulation and Evaluation and Supply stand out. For Industrial Chemical Engineering
(IQI), none of the subjects presents a response that exceeds 50% as shown in the next figure.
IIL IQI
Figure 9. Asignaturas que en sus planeaciones contemplan la técnica ABP y que los alumnos reportan haberla utilizado. IIL
en azul e IQI en dorado.
According to the results presented in the next figure, more than 75%, in both careers, are convinced that
teamwork motivates them to learn
IQI IIL
Figure 10. Perception of students in relation to intrinsic motivation for teamwork.
The presence of the teacher with the students is necessary when using this technique. The accompaniment of
the teacher provides certainty that translates into better achievement results. That was expressed by students
from both programs.
92
IQI IIL
Figure 11. Presence of teachers during the development of the project.
In the case of the challenges presented by the PBL technique (next figure) for the students of the Industrial
Chemical Engineering program, the predominant factor is access to the company (53.6%) similar to the
appreciation of the students of Logistics Industrial Engineering (58.9 %). On the other hand, for Chemical
Engineers, the second factor that represents a challenge is communication skills (26.8%), which are the third
factor for students of Logistics Industrial Engineering (11%) and vice versa, the third for IQI is the time factor
(12.5%) which for IILs is the second element (24.7%).
IQI IIL
Figure 12. Challenges to overcome in the elaboration of the Project
In relation to the skills developed during the development of the project, although they have different views,
it is interesting to observe that they recognize the development of different skills as shown in the next figure.
IQI IIL
Figure 13. Competences developed during the use of PBL as a learning method.
An important percentage (80.4%) of the IIL recognize that the development of projects as a learning technique
is relevant for their professional future, while only half of the IQI students recognize this relationship.
93
Figure 14. Perception of the relationship between the skills developed and professional practice.
5 Conclusion
In general, we can conclude that the PBL technique is used as a learning model in the programs studied. From
the results obtained we can establish that a greater use of the technique for the acquisition of knowledge is
needed. As well as a more punctual presence of the teacher when this technique is proposed as a learning
method.
In both study plans it is necessary to deepen the use of this technique and to integrate it in the last semesters
so that learning has an even greater impact. It is necessary to reorient the strategies and deepen the research
trying to discriminate those courses that use the methodology and those that remain superficial. This in order
to be able to answer the question generated by the declared percentage of use of the technique versus the
percentage of appreciation on the part of the students.
On the other hand, students recognize that the technique has an impact on several skills that will be useful
during their professional practice. However, it is not used in a systematized way by students or teachers, in
this work we only inquired with students who develop projects in real scenarios as part of the product of their
subjects, but in an empirical way they actually apply it without the knowledge that there are a series of steps
to follow.
6 References
Carrasco, A., Donoso, J. A., Duarte-Atoche, T., Hernández, J. J. y López, R. (2015). Diseño y validación de un cuestionario que mide la
percepción de efectividad del uso de metodologías de participación activa (CEMPA). El caso del aprendizaje basado en proyectos
(ABP) en la docencia de la contabilidad. Innovar, 25(58), 143-158
Estruch, V. y Silva, J. (2006). Aprendizaje basado en proyectos en la carrera de Ingeniería informática . JENUI, 339-346.
Reverte, J. R., Gallego, A. J., Molina, R. y Satorre, R. (2006). El aprendizaje basado en proyectos como modelo docente: experiencia
interdisciplinar y herramientas groupware. Proyecto de inno vación tecnológico-educativo e innovación educativa de la
Universidad de Alicante
Van Den Bergh, V.; Mortermans, D.; Spooren, P.; Van Petegem, P.; Gijbels, D.; Vanthournout, G. New assessment modes within project-
based education – the stakeholders. Studies in Educational Evaluation, 2006, 32, 345-368.
Duque, M.; Gauthier, A.; Gómez, R.; Loboguerrero, J.; Pinilla, A. Formación de ingenieros para la innovación y el desarrollo tecnológico en
Colombia. Revista DYNA, 1999, vol. 128, pp. 63-82.
Duque, M., & Martínez, Ac. Aprender haciendo: una experiencia de un laboratorio diferente. XX Reunión Nacional de Facultades de
Ingeniería. “Ingeniería y Desarrollo Social”. Asociación Colombiana de Facultades de Ingeniería (Acofi). Cartagena de Indias, 2000,
pp. 14-20.
Garza-Rivera, Rg. El rol de la física en la formación del ingeniero. Ingenierías, 2001, vol. IV, No. 13, pp. 48-54.
Duffrin, MW. Integrating problem-based learning in an introductory college food science course. Journal of Food Science Education, 2003,
2, pp. 2-6.
Collazos, C. (2009). Enseñanza de la conservación del momento angular por medio de la construcción de prototipos y el aprendizaje
basado en proyectos. Latin-American Journal of PhysicsEducation, 3(2), 427-432.
Díaz Barriga, Frida (2015) “Estrategias para el desarrollo de competencias en educación superior”. En: CARRILLO, Gabriela (Ed). I Encuentro
Internacional Universitario. El currículo por competencias en la educación superior. Ponencias y debate. Lima. PUCP.
Jonnaert, Phillipe; Barrere, Johanne; Masciotra, Domenico y Yaya, Mane (2006) Revisión de la competencia como organizadora de los
programas de formación: hacia un desempeño competente. Ginebra: International Bureau of Education.
Chan, M., Escalante, J. y Hernández, F. (2018). Utilización del aprendizaje basado en proyectos en escenarios reales como herramienta de
evaluación del aprendizaje, International Symposium on Project Approaches in Engineering Education, en Brasilía, pp. 948-955.
Rodríguez A., Río R, y Larrañaga J. (2016). Aprendizaje Basada en Proyecto (PBL), descripción de una experiencia desarrollada en aula
universitaria y sugerencias para optimizar resultados, Escuela Universitaria de Ingeniería Vitoria-Gasteiz en España.
94
The use of games in the teaching of Production Engineering: a
list based on publications in Scopus
Bruna Andrade Machado1, Bárbara Yumi Hotta¹, Gabriel Lino Garcia², Fernando Bernardi de Souza¹, José de Souza
Rodrigues¹
¹
Production Engineering Department, Faculty of Engineering, UNESP, Bauru, São Paulo, Brazil
²
Computer Science Department, Faculty of Science, UNESP, Bauru, São Paulo

Abstract
The use of games in the teaching of production engineering is on the rise since it allows the presentation of knowledge
and its absorption in a more relaxed way, it helps the development of essential skills for a good professional as well. As a
way of identifying educational games developed, used and/or published in production engineering courses, a qualitative
research was carried out on the Scopus database. The research returned 40 publications, of which 11 could not be accessed
and only 18 had games related to the course in question. As a result, 44 games were compiled, which were listed in order
to facilitate their knowledge by teachers in the area, stimulating their application and the modernization of teaching.
Keywords: Educational Games; Production Engineering; Game-based Learning; Game List.
1 Introduction
The advancement of technologies, the dynamics of the modern world, the sharing of information in real time,
the adaptation needs, among other issues, caused the students' profile to be significantly changed (RUBEUS,
2021).
In this way, the traditional teaching-learning strategies are no longer sufficient for this profile that seeks more
condensed and faster information. Evidence of this fact is that much has been studied and developed in relation
to active methodologies.
The active learning methodology is any educational approach in which students actively participate in the
learning process, becoming the centrepiece of the process and the main responsible for their learning (Lázaro
et al., 2018). They encourage debates and practice, as well as the review of knowledge through explanations
with other students. Some of the main active methodologies are: Project-Based Learning, Problem-Based
Learning, Gamification, Game-Based Learning (GBL).
With the world increasingly dynamic, the GBL has proved to be a good teaching tool, since it uses games to
generate greater engagement, motivate action, promote learning or solve problems in a creative way, in the
case of an excellent way to help students lose resistance in the face of complex topics (Sales et al., 2017).
It turns out that the application of games implies the need for development or knowledge of the existence of
the game on the part of the teacher as well as in its study. Although the absence of games was a problem a
few years ago, today there are already several games that can be applied in teaching different subjects and
courses. However, finding a game suited to a specific subject, discipline or competence is often hard work that
inhibits the adoption of the GBL (Bonetti, 2014).
In order to facilitate and encourage the use of games in the teaching of Production Engineering, this research
presents a list of games found in national and international publications.
2 Methods
This bibliographic research was carried out by searching the titles of Scopus publications, using the following
keywords: game AND engineer* AND (product* OR manufact* OR industrial). The search resulted in 40
95
publications, of which 11 could not be accessed and only 18 had games related to Production Engineering.
After analysing the publications found, 44 games were identified.
Although it would be desirable to present a detailed description of the games, including the number of players,
application time, the skills worked, among others, the lack of standardization in the data presented limited the
study to presenting few characteristics (name, description, access, format and areas/subjects addressed), listed
in table 1.
Therefore, this is a bibliographical, qualitative and exploratory research that aims to identify games that can be
used in Production Engineering courses.
3 Researches about the use of games

The use of games as a tool to help teach specific content in the classroom is not a new idea (HILL et al., 2003).
Education scholars, such as Vygotsky (1989) and Piaget (1972), present games as extremely important tools for
the development of cognitive skills, especially in early childhood and elementary education. Through them, the
child acts and has his/her creativity stimulated, in addition to acquiring self-confidence and concentration.
These same opinions are reinforced by Macedo et al. (2008), according to whom, in addition to promoting
cognitive development, “playing” is engaging, interesting and stimulating.
Even though much of the initial research was focused on early childhood and elementary school, today there
is already a lot of research on the use of games in higher education. Bahadoorsingh, Dyer and Sharma (2016)
states that games provide a different pedagogical perspective within a higher education context. Although
they may not seem natural, they represent two critical factors that need to be considered: they are impactful
and they are emerging as a potential source of disruption in current teaching models.
To be an educational game, games need to be focused on teaching a certain subject, expanding concepts and
improving some skills and attitudes that players/students seek/acquire during the game (Dempsey et. al., 1996).
The use of games for pedagogical purposes has grown and covered several areas, from health (Alloni et al.,
2017; Pires et al., 2019), resource management (Bellotti et al., 2014; Othman et al., 2016) and even learning new
languages (Eltahir et al., 2021).
Several authors such as Kishimoto (2004) and McDowell et al. (2006) reinforce the importance of the game as
a great motivator and facilitator so that concepts can be absorbed more quickly and dynamically. On the other
hand, Bonetti (2014) reports that game development is arduous and costly, which makes it difficult for teachers
to perform them, who need to resort to existing games. According to the author, there are no specific
environments for searching games for educational purposes, these “are scattered on the Internet in a non-
centralized way, without a systematic and uniform representation of their relevant information”, which makes
the search time-consuming, complicated and not very efficient.
From this difficulty arises this research, which identified and listed some games used in the teaching of
Production Engineering, in order to facilitate access to games by interested parties.
4 Results: a list of games for Production Engineering

At first, more than 20 items were defined to describe the characteristics of the games found. However, the lack
of standardization and information meant that only 5 items were presented here, because only these fields
could be filled in, for at least 50% of the games listed.
The listed features are:
● Name: the game's title;
● Description: brief description of the game based on information provided in publications;
● Access: informs the type of access, whether the game can be accessed freely (free), paid or trial (free
to test, but pay to use), also indicates if the game is available or if it is for internal use only /specific;
● Format: indicates whether the game is physical or digital;
96
● Areas: indicates the areas that are worked on by the game, such as: management, marketing, life cycle,
etc.
Characteristics that can be considered important for pedagogical purposes, such as: subjects in which games
are usually used, number of players/teams, time required to apply the games, etc., were not listed due to lack
of information.
The reference work(s) used in the research were included in the results. Table 1 presents the games identified
and applicable to the Production Engineering course.
Table 1 – List of games applied to Production Engineering and their characteristics.
Name Description Access Format Area(s) Cited by
(not indicated - A simulation game that involves product
Gutiérrez & Sastrón
ETO SIM) lifecycle planning in an Engineer-To-Order - Digital life cycle, LCA
(2009)
(ETO) environment.
(not indicated) A virtual reality game in which players "visit" an
industrial environment and need to identify
- Digital manufacturing systems Urgo et al. (2022)
various elements and concepts, as well as
establish hypotheses and models.
(not indicated) A computer simulation game between juice Braghirolli et al.
- Digital business management
companies. (2016)
Beer game A board game that already has a digital version Physical Despeisse (2019),
supply chain
and that simulates the beer supply chain. - and Machado et al.
management, bullwhip
Digital (2021)
Bernard - Industrial A business game in the consumer durables
Simulator sector that reproduces operating conditions of
Igidio et al. (2017),
the main functional areas of an industry. For production, finance,
Paid Digital Ishihara, Neto &
each simulated period it is necessary to plan marketing and logistics
Neumann (2021)
the machines, facilities and raw materials
needed, human resources.
BEWARE Game A risk management simulation game. Hauge & Riedel
- Digital risk management
(2012)
Business Game A business game used in Young Business
Talents (an international competition) whose production, finance,
Free Digital Igidio et al. (2017)
objective is to analyze, plan and control a marketing and logistics
company.
Business A business game in the area of production, finance, Ishihara, Neto &
Paid Digital
Management microcomputers. marketing and logistics Neumann (2021)
Cesim Firm A business game in the pharmaceutical
industry that integrates the functional areas of
production, marketing and logistics, helping Igidio et al. (2017),
production, finance,
participants to plan their strategies and Paid Digital Ishihara, Neto &
marketing and logistics
practice decision making, having interactions Neumann (2021)
between all typical functions within the
company.
CityCar A dynamic simulation game related to the
Despeisse (2019),
development of new products in which teams - Physical product development
Kerga et al. (2014)
of 5 to 7 players compete against each other.
COSIGA A simulation game focused on simultaneous Kerga et al. (2014),
engineering in which players must develop a Hauge & Riedel
Free Digital product development
truck (construction, mobile home or delivery). (2012), Riedel, Pawar
& Barsonn (2001)
Factory Heroes A cooperative RPG-style board game that
works on sustainable management in a - Physical sustainability Despeisse (2019)
manufacturing company.
Fishbanks A board game that already has a digital
Physical
multiplayer version in teams that represent resource management,
Paid and Despeisse (2019)
fishing companies and need to maximize their sustainability
Digital
profits while competing for natural resources.
Furniture Factory A computer simulation game that works basic
manufacturing systems
concepts of production engineering for
- Digital management, Production Bengoa et al. (2013)
freshmen through a scenario referring to a
Engineering concepts
furniture company.
97
Production Game A mobile game developed specifically for the
(Game da Production Engineering course at UnB, in which
Júnior, Simão
Produção) students can simulate their trajectory through
- Digital Education Monteiro & Madeira
the course, understanding the disciplines and
Campos (2019)
activities that must be carried out until
completion.
General A business game in the field of graphic design production, finance, Ishihara, Neto &
Paid Digital
Management industry. marketing and logistics Neumann (2021)
Production A business game whose objective is to
Management determine the quantity of products to be
(Gestão da produced in normal and extended hours and - Digital Igidio et al. (2017)
Produção 1 -GP-1) how much of the production must be
outsourced, so that the profit is maximum.
GLOTRAN A computer simulation game that simulates the
development and production of a product - Digital product development Kerga et al. (2014)
across different teams.
GSim - Goldratt A simulator used as a game to work concepts production management, Machado et al.
Free Digital
Simulator of Theory of Constraints (TOC) theory of constraints (2021)
Industrial A business game in the area of physical goods. production, finance, Ishihara, Neto &
Paid Digital
Simulator marketing and logistics Neumann (2021)
Industrial Strategic A business game in the area of physical goods. production, finance, Ishihara, Neto &
Paid Digital
Simulator marketing and logistics Neumann (2021)
JOGAI - The A game that simulates the supply chain of
Industrial gemstones and jewelry through group
Engineering dynamics, with assembly of parts in which - Physical planning and control Barçante et al. (2011)
Undergraduate different companies compete with each other.
Game
LCA Game An RPG-style computer simulation game that
focuses on analyzing the environmental impact - Digital life cycle, LCA Despeisse (2019)
of the company's coffee machine production.
Lean Board Game A board game that simulates the operation of
Machado et al.
an industry allowing the monitoring of various - Physical production management
(2021)
indicators.
LSSP_PCP* A business game in which the simulated
company produces three different families of
meshes, with sectors of Knitwear. Dyeing and Free Digital demand forecast Igidio et al. (2017)
Finishing. It is necessary to manage the
company that has variable demand.
Moto Cycle A business game in the area of motorcycle production, finance, Ishihara, Neto &
Paid Digital
industry. marketing and logistics Neumann (2021)
NCTB GAME A business (computerized) game in which
students must make weekly decisions related
to different areas (purchasing, production, production, finance, Bringelson et al.
Paid Digital
quality control and marketing) of a packaging marketing and logistics (1995)
company in order to obtain the best possible
results.
One thousand A card game in which the highest score is
kWh sought, which is obtained by reducing the - Physical sustainability Despeisse (2019)
energy saved in the company's plant.
Production A business game in the field of furniture production, finance, Ishihara, Neto &
Free Digital
Management – PM industry. marketing and logistics Neumann (2021)
Production A business game in the area of fictional
production, finance, Ishihara, Neto &
Planning products. Paid Digital
marketing and logistics Neumann (2021)
Simulation – PPS
Production System A business game in the area of knitting
Simulation industry. Free Digital
Laboratory – PSSL
SBCE game A dynamic simulation game in which players
must develop a simplified aircraft structure product development,
- Physical Kerga et al. (2014)
respecting the established parameters, using lean
two different proposals (PBCE and SBCE).
98
Strategic A business game in which participants plan and
Simulation (SDE implement a new venture that will produce and
Simulação market the RPB – Portable Beverage
Estratégica) Refrigerator – a new product, without equal,
Paid Digital entrepreneurship Igidio et al. (2017)
competing with other simulated companies in
different markets. At the end of the simulation,
the performance of the groups is evaluated
based on the analysis of the results obtained.
Simulators and An interactive business simulation applied in
Management the Global Management Challenge
Models (SDG – (international strategy and management
Simuladores e competition) in which each team manages a Paid Digital Igidio et al. (2017)
modelos de gestão) company with the objective of obtaining the
best investment performance for their
company in the market in which it operates.
Shoe Maker A business game in the shoe industry. production, finance, Ishihara, Neto &
Paid Digital
Simulare A business game in which teams compete with
each other, having to make decisions such as
Igidio et al. (2017),
price, investment in advertising, receipt and production, finance,
Paid Digital Ishihara, Neto &
payment terms, purchases of inputs and marketing and logistics
Neumann (2021)
machinery, hiring and firing of personnel, loans,
financing, etc.
Production A business game in which participants act as
Planning leaders of an industrial unit, that produces
Simulation (SPP VERIPEX (fictitious product), which must be
Simulação de shipped according to the quantity requested
Planejamento da by the customer. Participants must make Paid Digital Igidio et al. (2017)
Produção) strategic and managerial production decisions,
simulating results, in quarterly cycles. Who
accumulates the highest net profit at the end
of the 4th period wins the game.
Supply Chain and A business game in the area of
Channel microcomputers. Paid Digital
Management
TFC business game A computer simulation of a supply chain that simulation, operational
(The Fresh must be optimized. - Digital research, production, Mosia (2018)
Connection) quality, logistics
The Product Line A game in the form of dynamics that brings
Planning Game consumers and developers together by
facilitating the presentation of consumer
- Physical agile method Carbon et al. (2008)
requirements and feedback to developers. As a
result of the matches are scenarios/stories that
help in the development of projects.
The Trimrian A dynamic simulation game that depicts a
Factory Game bankrupt company trying to get back on its
- Physical production Jensen (2008)
feet by fulfilling a customer's orders within 6
rounds.
Topaz A business game with 3 different product
Management options. Paid Digital
Simulation
Virtonimics A business game in the clothing field. production, finance, Ishihara, Neto &
Paid Digital
Enterprenuer marketing and logistics Neumann (2021)
Virtual Business A business game in the grocery/convenience production, finance, Ishihara, Neto &
Paid Digital
Retailing store area. marketing and logistics Neumann (2021)
Source: Authors (2022)
Even performing a very limited search, it could be noted that part of the publications found in the research
were not aligned with the scope of this work. Among them is the article by Zhou (2018) that analyses the
scenario of the chemical industry through game theory. Another article that was discarded, although it
presented an analysis of publications in the area of games for Production Engineering, was that of Rosado and
99
de Souza (2021), since the authors carried out a bibliographic mapping, without it being possible to identify
any game.
Contrary to Rosado and de Souza (2018) are the works of Despeisse (2018), Igidio et al. (2017), Ishihara, Netto
and Neumann (2021) and Kerga et al. (2014) which present game relationships with different approaches.
Despeisse (2018) analyses 6 (six) games and relates them to the cognitive development domains of Bloom's
Taxonomy. Igidio et al. (2017) succinctly present 8 (eight) business games and an applied study of the
Management of Production 1 (GP-1), Ishihara, Netto and Neumann (2021) present 16 business games from
the analysis of their manuals, bringing information internals of the games and Kerga et al. (2014) cites three
games related to product development and analyses and presents the results of applying another game in the
area. The games that were only mentioned by these authors are listed in this research, although they have little
general information.
The scarcity of information caused the work of Scholl, Gube and Koppatz (2021) to be disregarded, since the
authors report the development of two physical games (Social Engineering Theater and Risk Roulette) that
could not be properly tested due to the Covid-19 pandemic.
5 Conclusion
This research aimed to present a list of games that can be used in Production Engineering teaching, making
the teaching-learning process more motivating and effective. As a result, 44 games were found and listed.
There was a lack of standardization in the obtained data. which makes it difficult to properly and quickly
evaluate and access the games. The absence of information on locations and access methods also prevents
verifying whether the games are really active. This lack of information implies the need to seek new materials
so that there is better detail and more direct guidance to readers who are interested in making use of games
in their classes.
This research emerged as part of a larger research which aims to establish a methodology for identifying and
making available information of games that facilitate their discovery and application by teachers and other
interested parties. In general, the lack of standardized information that describes games as a learning object is
something that has been studied, with researches related to the adoption of metadata standards, such as the
Learning Object Metadata (from the IEEE) and the Dublin Core (DC). This adoption becomes necessary so that
works like this can be more meaningful, bringing more relevant and updatable information.
As future research, a deeper analysis of the identified games is suggested in order to update the information
and present only games that are active and accessible (free of charge or for a fee) to the public, since it is
believed that a good part of the games listed here are not available or are already obsolete, as in the case of
The Product Line Planning Game (Carbon et al. 2008) which either must have undergone an update, or has
already left the market, since it used floppy disks.
6 References
Alloni, A., Sinforiani, E., Zucchella, C., Sandrini, G., Bernini, S., Cattani, B., ... & Pistarini, C. (2017). Computer-based cognitive rehabilitation:
the CoRe system. Disability and rehabilitation, 39(4), 407-417.
Bahadoorsingh, S., Dyer, R., & Sharma, C. (2016). Integrating serious games into the engineering curriculum-a game-based learning
approach to power systems analysis. International Journal of Computational Vision and Robotics, 6(3), 276-289.
Barçante, L. C., Corrêa, F. R., Maciel, M. S. D., de Souza, F. S., & Alves, C. S. C. (2011). The Industrial Engineering Undergraduate
Game. Proceedings of the International Mult iConference of Engineers and Computer Scientists.
Bengoa, D., Rullán, A., Velez, B., Pomales-García, C., & Zapata, F. (2013). Development of an Educational Game for Industrial Engineering
Freshman Students. In IIE Annual Conference. Proceedings (p. 616). Institute of Industrial and Systems Engineers (IISE).
Bellotti, F., Berta, R., De Gloria, A., Lavagnino, E., Antonaci, A., Dagnino, F., ... & Mayer, I. S. (2014). Serious games and the development of
an entrepreneurial mindset in higher education engineering students. Entertainment Computing, 5(4), 357-366.
Bonetti, T. M. (2014). Proposta de um modelo de repositório colaborativo para compartilhar informações de jogos para o ensino de
computação.
Braghirolli, L. F., Ribeiro, J. L. D., Weise, A. D., & Pizzolato, M. (2016). Benefits of educational games as an introductory activity in industrial
engineering education. Computers in Human Behavior, 58, 315-324.
Bringelson, L. S., Lyth, D. M., Reck, R. L., & Landeros, R. (1995). Training industrial engineers with an interfunctional computer simulation
game. Computers & industrial engineering, 29(1-4), 89-92.
100
Carbon, R., Knodel, J., Muthig, D., & Meier, G. (2008, September). Providing feedback from application to family engineering-the product
line planning game at the testo ag. In 2008 12th International Software Product Line Conference (pp. 180-189). IEEE.
Dempsey, J. V., Lucassen, B., & Rasmussen, K. (1996). The instructional gaming literature: Implications and 99 sources. South Carolina:
University of South Carolina, College of Education.
Despeisse, M. (2018). Games and simulations in industrial engineering education: a review of the cognitive and affective learning outcomes.
In 2018 Winter Simulation Conference (WSC) (pp. 4046-4057). IEEE.
Eltahir, M., Alsalhi, N. R., Al-Qatawneh, S., AlQudah, H. A., & Jaradat, M. (2021). The impact of game-based learning (GBL) on students’
motivation, engagement and academic performance on an Arabic language grammar course in higher education. Education and
Information Technologies, 26(3), 3251-3278.
Gutierrez, M., & Sastron, F. (2009). A computer simulation game for learning product lifecycle planning through the engineer-to-order
case. The International journal of engineering education, 25(5), 864-875.
Hauge, J. B., & Riedel, J. C. (2012). Evaluation of simulation games for teaching engineering and manufacturing. Procedia Computer Science,
15, 210-220.
Hill, J. M., Ray, C. K., Blair, J. R., & Carver Jr, C. A. (2003, January). Puzzles and games: addressing different learning styles in teaching
operating systems concepts. In Proceedings of the 34th SIGCSE technical symposium on Computer science education (pp. 182-186).
Igidio, S. S., Paula, D. D. F. F., Rodrigues, A. C., Gontijo, T. S., & Braga, L. B. M. (2017). A contribuição de jogos de simulação na aprendizagem
de alunos de engenharia de Produção. Espacios, 38(30), 23.
Ishihara, S. E. M. P., Neto, A. A., & Neumann, C. (2021, February). Business Simulation Games: A Comparative Analysis for Teaching of
Production Engineering. In International Joint conference on Industrial Engineering and Operations Management (pp. 237-251).
Springer, Cham.
Jensen, J. P. (2008). Experiences with the Trimrian factory game - A production game within the education of Bachelor Production Engineers
Júnior E. S., Simão Monteiro S. B., Madeira Campos M. R. (2019). Serious game for the production engineering course of the UNB: Student's
trajectory in the course. In: Proceedings of the PAEE ALE.
Kerga, E., Rossi, M., Taisch, M., & Terzi, S. (2014). A serious game for introducing set-based concurrent engineering in industrial practices.
Concurrent Engineering, 22(4), 333-346.
Kishimoto, T. M. (1996). Froebel e a concepção de jogo infantil. Revista da Faculdade de Educação, 22(1), 145-167.
Lázaro, A. C., Sato, M. A. V., & Tezani, T. C. R. (2018). Metodologias ativas no ensino superior: o papel do docente no ensino presencial. CIET:
EnPED.
Machado B.A., De Souza F.B., De Souza Rodrigues J., Hotta B.Y. (2021). Game-based learning in a production engineering course in Brazil.
In: Proceedings of the PAEE ALE.
McDowell, P., Darken, R., Sullivan, J., & Johnson, E. (2006). Delta3D: a complete open source game and simulation engine for building
military training systems. The Journal of Defense Modeling and Simulation, 3(3), 143-154.
Mosia N. (2018). Application digital game based learning in industrial engineering teaching and learning in an online distance electronic
learning institution. In: Proceedings of the International Conference on Industrial Engineering and Operation Management
Pretoria, Johannesburg, South Africa.
Othman, B. Y., Abdelali, S., Amine, B., & Mohamed, B. (2016). Classification, evaluation and assessment of serious games in business
field. Journal of Theoretical and Applied Information Technology, 91(1), 83.
Piaget, J. (1972). Desenvolvimento e aprendizagem. Studying teaching, 1-8.
Pires, M. R. G. M., Guilhem, D., & Göttems, L. B. D. (2013). The (IN) DICA-SUS game: a strategy of game-based learning on the Unified
Health System. Texto & Contexto-Enfermagem, 22(2), 379-388.
Riedel, J. C., Pawar, K. S., & Barson, R. (2001). Academic and industrial user needs for a concurrent engineering computer simulation game.
Concurrent Engineering, 9(3), 223-237.
Rosado, C. A. G., & de Souza, M. C. (2021). A bibliometric study on the use of games in the Production Engineering area. Gestão & Produção,
28.
Rubeus. (2021). O novo perfil dos alunos: 5 comportamentos que sua IE precisa conhecer! [Post da web]. Recuperado de
https://rubeus.com.br/blog/o-novo-perfil-dos-alunos/
Sales, G. L., Cunha, J. L. L., Gonçalves, A. J., da Silva, J. B., & dos Santos, R. L. (2017). Gamificação e ensinagem híbrida na sala de aula de
física: metodologias ativas aplicadas aos espaços de aprendizagem e na prática docente. Conexões-Ciência e Tecnologia, 11(2), 45-
52.
Scholl, M., Gube, S., & Koppatz, P. (2021). Development of Game-Based Learning Scenarios for Social Engineering and Secu-rity Risk
Management for SMEs in the Manufacturing Industry. In 12th International Multi-Conference on Complexity, Informatics and
Cybernetics, IMCIC 2021 (pp. 59-65).
Urgo, M., Terkaj, W., Mondellini, M., & Colombo, G. (2022). Design of serious games in engineering education: An application to the
configuration and analysis of manufacturing systems. CIRP Journal of Manufacturing Science and Technology, 36, 172-184.
Vygotsky, L. S. (1989) A formação social da mente: o desenvolvimento dos processos psicológicos superiores. 3a. ed. São Paulo: Martins
Fontes.
Zhou, X. (2018). Management Mechanism of Chemical Engineering Production Management Based on Game Model. Chemical Engineering
Transactions, 67, 673-678.
101
Evaluation of the effectiveness of the use of games in the
teaching of Production Engineering: an experiment with
Goldratt Simulator
Bárbara Yumi Hotta1, Fernando Bernardi de Souza1, Bruna Andrade Machado1, José de Souza Rodrigues1
1
Production Engineering Department, Faculty of Engineering, UNESP, Bauru, São Paulo, Brazil
Abstract
Traditional and theoretical classes are usually tiring and their contents are poorly absorbed. One way to apply and fix the
contents in a more attractive and motivating way is through the use of active methodologies, whose use has been encouraged
by the new curricular guidelines of engineering courses in Brazil. Game-based learning (GBL) is one of such approach. Although
many games are already used in classroom environments, e.g. Beer Game, Lean Board Game and Goldratt Simulator, there is
still a lack of studies on the effectiveness of its application. In 2020, a study about the use of games in Production Engineering
courses was initiated, which resulted in the development of questionnaires and the suggestion of a methodology for evaluating
the effectiveness of the use of GBL. In order to identify the effectiveness of the use of Goldratt Simulator, as a game, in the
learning process this evaluation methodology was applied during an experiment carried out in the discipline of Production
Management IV, in which two groups were defined (control and experimental). After the classes and the application of the
questionnaires, it became evident: greater engagement and confidence by the experimental group and greater fixation of
concepts by the control group. Furthermore, the evaluative tools proved to be adequate to give feedback to the teacher,
presenting themselves as an important diagnostic tool.
Keywords: Learning evaluation methodology; GBL; Educational games; Production engineering.
1 Introduction
Changes in student profiles, who think that theoretical and traditional classes are usually tiring and
demotivating, have led to new teaching methodologies being developed and applied (Rubeus, 2021).
The use of such methodologies has even been encouraged by teaching guidelines, which suggest that active
methodologies (in which students become protagonists in the learning process) are constantly used (MEC,
2019).
One of the methodologies that promises to stimulate student development is Game Based Learning (GBL),
which uses analogue or digital games in classrooms (Karagioras & Niemann, 2017). The methodology is mainly
used to generate greater engagement, motivate action, promote learning and solve problems in a creative way
(Lester, 2014). It is an excellent way to help students lose resistance in the face of complex topics (Sales et al.,
2017).
There are several studies that present the creation of game-based learning environments for a wide range of
curricula (Warren et al., 2008; Kebritchi & Hirumi, 2010). In the university where this project has been developed,
some analogue games are already used, such as Beer Game and Lean Board Game, and virtual simulators, such
as Goldratt Simulator, to support the Production Management disciplines of the Production Engineering
course. It so happens that, so far, in the course in question, this methodology has been applied without carrying
out a scientific evaluation of the benefits of its use.
According to Lester (2014), research on game-based learning has matured and today there are empirical
studies that qualitatively demonstrate students' learning gains when interacting with educational games in a
variety of subjects, however, these studies often make use of non-standard instruments. Furthermore,
quantitative research is more difficult to identify.
In 2021 a research developed by the authors resulted in the elaboration of a set of tools (questionnaire to
characterise the participants and three instruments evaluation) that allow to analyse the perception of students
and the level of learning obtained by the use of games in the learning process (Machado et al., 2021). During
102
the same period, the Goldratt Simulator was used as a game in the discipline of Production Management IV,
offered to a 4th year class at a Brazilian public university. Due to the importance of analysing the effectiveness
of its application, the authors of this research used the tools previously developed in an attempt to answer the
following question: is the use of Goldratt Simulator effective for greater student motivation and learning?
In order to verify the effectiveness of the use of the Goldratt Simulator as a motivation and learning tool, a
research was conducted applying the tools previously developed. The instruments and results are presented
in this work.
2 Methods
After carrying out a systematic review to identify instruments for evaluating the effectiveness of the use of
games in teaching, the results of which were published by Machado et al. (2021), an action research was carried
out regarding the application of a game already consolidated and commonly applied in the discipline of
Production Management IV of the course studied, which uses the Goldratt Simulator, a simulator of the
production process, which allows the player to identify and deal with problems that can occur in real situations
and that allows the learning of several concepts of Theory of Constraints, an important content for the
Production Engineering course.
According to Oquisit (1978, apud Mello et al, 2012), action research can be defined as the production of
knowledge guided by practice, occurring simultaneously to the modification of a reality that is also part of the
research process. Thiollent (2007, apud Mello et al, 2012) emphasises that for a research to be considered an
action research, it is essential that there is an action on the part of the people involved in the problem under
observation, that this action is non-trivial to the point of needing to investigation and that researchers play an
active role in conducting, monitoring and evaluating the research. It is added that this method is appropriate
when the research question is used to clarify how an action of a person can improve the functioning of the
system and how this process will take place, thus aiming to learn from it (Coghlan & Brannick, 2008). apud
Mello et al, 2012).
Thus, an experiment was carried out in the discipline of Production Management IV of the Production
Engineering course of a Brazilian public institution. In this experiment, the class was divided into two groups,
one group was exposed to a theoretical-expository teaching-learning process, while the other used an
approach along the lines of GBL (Game-based learning). With the purpose of measuring the effectiveness of
each approach, the students of each group were questioned using the evaluation tools developed from the
bibliographic research.
At this stage, of the 32 students in the class, 15 volunteered to participate in the research, the participants were
divided between the control (9 students) and experimental (6 students) groups in order to maintain a certain
homogeneity in relation to the average of ages and gender. However, right at the beginning of the
questionnaires, 5 students in the control group dropped out, leaving 10 participants.
3 Evaluation methodologies
In view of the existence of few validation studies on the effectiveness of the use of games in teaching, the
authors of this work carried out, between the years 2020 and 2021, a research whose result was the elaboration
of a questionnaire to characterise the participants and three instruments evaluation, one of perception, another
of self-evaluation and a third of learning. These three instruments were published in the proceedings of
PAEE/ALE'2021 under the title “Game-based learning in a Production Engineering course in Brazil.” by Machado
et al. (2021) and are the basis of this research, since they were used to measure the effects of applying the
Goldratt Simulator in the discipline of Production Management IV. The instruments created and used
throughout this new research are presented below.
103
3.1 Participant’s characterization questionnaire
The participants' characterization questionnaire begins with the student's acceptance to participate in the
experiment and is applied through “Google Forms”. The questionnaire comprises the following questions:
Gender: ( ) Female ( ) Male ( ) Not informed
Age: _____
What is your level of knowledge about Theory of Constraints (TOC)¹?
( ) I've never heard of it, I don't know what it is
( ) I've heard of it, but I don't know what it is
( ) I've heard of it and I know it's related to production systems
( ) I have read and / or attended classes on the subject
( ) I know and am able to discuss the matter properly
Based on the teaching plan presented for the subject, analyse the following statements using the Likert scale
from 1 (strongly disagree) to 5 (strongly agree):
I believe that subject is important for my training
I would enrol in the course if it was not mandatory
I think the proposed assessment is fair
I think the proposed assessment is challenging
I think the proposed methodology looks interesting
¹ Main concept subject to be worked on in the experiment
Figure 1. Specific participant’s characterization questionnaire
Source: Machado et al (2021)
This questionnaire must be applied at the beginning of the classes, before the first instructions on the content
are offered, so that the groups can be properly organised.
3.2 Perception questionnaire

Perception analyses reach the first two levels proposed by Kirkpatrick (reaction and learning). The first analysis
will begin with the application of a perception questionnaire combined with the Flow Theory.
Using the Likert scale from 1 (strongly disagree) to 5 (strongly agree), students from both groups should answer
the following questions:
Absorption / Immersion:
1. During the activity I lost track of time
2. During the activity, I felt totally immersed
3. The activity made me excited
4. The activity made me feel self-confident
5. The activity stimulated my interest
6. The activity piqued my curiosity
Pleasure:
7. The activity gave me a good feeling
8. I had fun during the activity
9. The activity brought me joy
10. The activity was pleasant
Motivation:
11. This type of activity should be carried out more frequently
12. This is an activity that I willingly performed
13. This is an activity that I would participate in even if it was not linked to the presence
14. This is an activity that I would participate in even if it was not linked to the note
15. This is an activity that I would do even though I didn't receive anything in return
Skills:
16. This activity improved my critical thinking
17. This activity improved my problem-solving ability
18. This activity improved my analytical ability
19. This activity improved my ability to manage time and resources
20. Some problems became clear with this activity
Figure 2. A generic developed perception questionnaire
104
Source: Machado et al (2021)
3.3 Post-test
After conducting the experiment, the following questionnaire must be applied to students in both groups. For
a coherent evaluation it is fundamental that the questions are neutral, that is, they are not directly related to
the game. We understand that the ideal would be the realisation of conceptual questions that could be
analysed in a practical way, for example by a TRUE or FALSE model.
Mark the sentences below with TRUE or FALSE:

1. The Theory of Constraints (TOC) argues that companies have many constraints that make it difficult to reach their goal
2. The Theory of Constraints (TOC) states that local optima result in global optima
3. Capacity constrained resources (RRC) are resources that, if poorly managed, can become bottlenecks
4. Bottlenecks impede demands from being fully met
5. Bottlenecks and RRCs are synonymous
6. The buffer is the stock of materials in process that must be present throughout the entire production line
7. The Rope is the sequencing of material release to the factory based on the discounted Lung Drum
8. The production schedule of the Bottleneck is the Drum and it is from there that the rest of the factory must be
subordinated
9. An hour lost on a non-bottleneck resource is an hour lost on the entire system
10. An hour saved on a non-bottleneck resource is an hour gained system-wide
11. In the classic Drum-Lung-Rope method, in-process buffers are dimensioned and controlled in the form of time
12. Non-bottleneck resources should adjust their production speeds to the Drum
Figure 3. The post-test with specific questions about Theory of Constraints
Source: Adapted from Machado et al (2021)
3.4 Self-assessment questionnaire

As a last tool, a self-assessment is suggested. In this, students will have the chance to review the contents
worked and analyse their own level of understanding.
Analyse your level of understanding by checking 1 (I didn't understand) to 5 (I fully understood).

1. The Theory of Constraints (TOC) argues that companies have very FEW constraints that limit the achievement of the
goal
2. The Theory of Constraints (TOC) states that local optima DO NOT result in global optima
3. Capacity constrained resources (RRC) are resources that, if poorly managed, can become bottlenecks
4. Bottlenecks impede demands from being fully met
5. Bottlenecks and RRCs are NOT synonymous
6. Time buffers are reflected in physical inventories located at strategic points to protect system constraints
7. The Rope is the sequencing of material release to the factory based on the Drum discounted the Buffer
8. The production schedule of the Bottleneck is the Drum and it is from there that the rest of the factory must be
subordinated
9. An hour lost on a NON-bottleneck resource is NOT a system-wide lost hour, but an hour lost on the bottleneck
resource is a system-wide lost hour
10. An hour saved on a NON-bottleneck resource is NOT an hour gained system-wide.
11. In the classic Drum-Buffer-Rope method, in-process buffers are dimensioned and controlled in the form of time
12. Non-bottleneck resources must NOT adjust their production speeds to the Drum. They must operate according to
roadrunner logic
Figure 4. The self-assessment with specific affirmatives about Theory of Constraints

Source: Adapted from Machado et al (2021)
105
4 Results
As for the application of the Goldratt Simulator, the 10 students who participated answered the questionnaires
to characterize the participants and the perception and learning assessment questionnaires, but only 9
responded to the self-assessment.
4.1 Participant’s characterization questionnaire

Regarding gender, 5 men and 5 women participated in the experiment. The average age is 22.8 years old, with
a maximum age of 27 and a minimum age of 21. Regarding the level of knowledge of the Theory of Constraints,
all 10 participants already knew something about the subject. For questions related to expectations of the topic
discussed in the course, the means and deviations were presented in Table 1.
Table 1: Assessment of participants' expectations
Average σ
I believe that this subject is important for my training 4.8 0.6
I would enrol in the course if it was not mandatory 4.0 1.1

I think the proposed assessment is fair 4.4 0.7
I think the proposed assessment is challenging 4.3 0.7
I think the proposed methodology looks interesting 4.4 0.8
Source: authors (2021)
4.2 Perception questionnaire

In the perception assessment, composed of 20 questions whose answers were based on a Likert scale from 1
(strongly disagree) to 5 (strongly agree), the means and deviations between the control and experimental
groups were calculated and are shown in Tables 2, 3, 4 and 5.
Table 2: Participants' perception of Absorption / Immersion
Control Group Experimental Group
Average σ Average σ
Q1. During the activity I lost track of time 1.8 0.5 4.2 0.4
Q2. During the activity, I felt totally immersed 2.8 1.5 4.5 0.5
Q3. The activity made me excited 2.8 1.5 3.7 0.8
Q4. The activity made me feel self-confident 3.3 1.5 4.5 0.5
Q5. The activity stimulated my interest 3.8 1.5 4.8 0.4
Q6. The activity piqued my curiosity 3.8 1.5 4.8 0.4
Regarding immersion, it was noted that the experimental group felt more immersed and motivated, presenting
higher averages and smaller deviations. Attention to the first item, that refers to the loss of the notion of time,
whose difference between the groups is quite significant, corroborating the idea that traditional classes are
more tiring. This statement is also confirmed by the report of one of the participants:
"As a member of the control group, I believe it is worth highlighting the teacher's didactic ability. In the
content presented, the examples used, clarity in speech and attention to students' doubts count a lot for
a very positive evaluation of the activity taught. Without these highlighted points, due to the relatively
long class time (4h), I don't think I would feel involved and excited to make the most of it. As classes are
usually long, they can become tiring and even though there is a good effort from the teacher and an effort
on the part of the students, even so, with only theoretical classes it is more difficult to maintain
concentration at all times and lose this notion. of time. By adding a more dynamic activity, students are
more involved and time goes faster".
106
Table 3: Participants' perception of Pleasure

Q7. The activity gave me a good feeling 3.8 1.5 4.5 0.8
Q8. I had fun during the activity 2.8 1.0 4.8 0.4
Q9. The activity brought me joy 2.8 1.0 4.0 1.1
Q10. The activity was pleasant 3.3 1.5 4.5 0.5
For pleasure, the experimental group also presented higher ratings, with attention to item Q9 in which the
deviation of the experimental group exceeded the control group, even though the average is higher.
Table 4: Participants' perception of Motivation
Q11. This type of activity should be carried out more frequently 3.3 1.0 4.8 0.4
Q12. This is an activity that I willingly performed 4.0 1.4 4.8 0.4
Q13. This is an activity that I would participate in even if it was 3.8 1.5 4.3 0.8
not linked to the presence
Q14. This is an activity that I would participate in even if it was 3.8 1.5 4.5 0.8
not linked to the note
Q15. This is an activity that I would do even though I didn't 3.8 1.5 4.3 0.8
receive anything in return
As for motivation, again, the experimental group showed higher ratings, with lower deviations than the control
group.
Table 5: Participants' perception of improved Skills

Q16. This activity improved my critical thinking 3.8 1.5 4.8 0.4
Q17. This activity improved my problem-solving ability 3.5 1.3 4.5 0.5
Q18. This activity improved my analytical ability 3.5 1.3 4.5 0.8
Q19. This activity improved my ability to manage time and 3.0 1.4 4.5 0.5
resources
Q20. Some problems became clear with this activity 2.5 1.3 4.8 0.4
Regarding skills, the control group presented lower averages with always higher deviations than the
experimental group.
Thus, it can be seen that the experimental group has a higher average for all questions. This shows that, in
general, the group that used the game became more immersed, had more pleasure in performing the activity,
felt more motivated and believed to have improved their skills.
107
4.3 Post-test
For the second instrument, the learning assessment, made up of 12 True/False questions, a different result was
obtained than expected. The control group had a higher average of correct answers than the experimental
group, as shown in Table 6. This can be explained by reasons such as incorrect division of groups, greater ease
on the part of students who belonged to the control group or lack of attention to answer the questionnaire. A
new application, with a larger sample of people, may indicate a different average, since the sample was
composed of the most engaged students in the class.
Another factor that may have been a complicating factor for the learning of the experimental group is that the
interest in simulating the activity and the fact that the students were in environments different from the teacher
(since the classes were held remotely due to the pandemic) prevented the better control of the teacher and
allowed the students to simulate while the teacher performed the explanations, which are of paramount
importance for obtaining the concepts presented.
Table 6: Average of correct answers in the learning evaluation form by group
Group Average
Control Group 9.25
Experimental Group 8.00
A feedback was received on this form:

I found the insights obtained by using the game helped me a lot to understand. Anyway, I think the 4
hours were a bit exhausting and may have hampered learning.
This report brings a reflection: even though the use of the game makes the class "lighter", as observed by the
first evaluation instrument, time is still a critical point.
It is also worth mentioning that questions 1, 6 and 12 were the ones with the highest frequency of errors, with
only 4, 4 and 2 correct answers respectively.
4.4 Self-assessment questionnaire

The self-assessment added to the results of the learning assessment (post-test) serves as a parameter for the
teacher to identify the points of difficulty for students to understand. Thus, following the Likert scale pattern 1
(I did not understand at all) to 5 (I fully understood), the following averages were obtained for the established
statements (Table 7).
Table 7: Average of self-assessment questions per group.
Group Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Total
Control Group 3.75 3.5 4.75 4 4,.5 5 4.5 3.75 5 4.75 5 3.75 4.35
Experimental Group 3.6 4.8 4.6 4.8 4.2 4 4.6 4.8 4.4 4.2 4.4 4 4.37
Source: authors (2021).
In general, a high level of confidence and security in the answers by all students can be observed. An interesting
detail is that the lowest averages of the experimental group are really related to the questions whose students
had greater assimilation difficulties (1, 6 and 12), according to the number of errors in the learning assessment.
5 Conclusion
Following the idea defended by McGonigall (2011 apud Moores, 2016) that participation in games should be
voluntary, participation in the experiment was optional, which implied a low adherence of students. The
108
exceptions refer to students who normally interact more with the teacher and are more willing to participate
in classes. The division of experimental and control groups in an entire room could indicate other results,
especially in the learning assessment instrument. It is also necessary to think that in this freedom of
participation, those who chose not to take part in the research, even because they had never used a game as
a teaching tool, may have missed the opportunity to discover a new method in which they could enjoy more,
have identification with and ease to learn.
The perception assessment instrument showed an acceptance of the use of games, as well as greater
immersion, pleasure, motivation and skills improvement perception on the part of those who participated. It
also indicated greater development of important and expected skills of a production engineer. Participants
were very interested in the proposal to use these means as a form of teaching. The students did not present
technical difficulties to answer the questionnaires. The use of games and simple assessment tools, such as
those developed in this study, are alternatives to measure student performance and perceive difficult points,
without having the pressure of a test, in addition to returning the results more quickly to students.
The learning assessment brought attention for pointing out a different result than expected. Students who did
not use the simulator performed better, even though they had less confidence in their answers, as indicated
by the self-assessment form. It is important that there are new applications with a greater number of students
to verify if the results are maintained in this proportion.
In order to answer the question of this work, by a perception way, the use of the game brings more immersion,
pleasure, motivation and skills improvement. Although, in the learning field, the result has shown to be inferior
to those who used the game. This may be due to the fact that the application was carried out online (not face-
to-face), with the teacher not having control over the students regarding the fact that they stopped using the
simulator so that they could focus attention on the explanations, which were essential for the correct
absorption of the concepts that were collected during the post-test. The results presented by the post-test and
the self-assessment questionnaire were quite consistent and corroborate the evidence of the quality of the
instruments used.
For future research, the application with a larger number of participants is indicated and the need to investigate
the low adherence of students to the research is evidenced, since a lack of time for extra activities was justified,
even when the application was made during the class schedule and questionnaires take very few minutes to
be answered. It is also worth mentioning the opportunity to apply the assessment instruments to other subjects
and even other teaching areas, with the necessary adaptations.
6 References
Karagiorgas, D. N., & Niemann, S. (2017). Gamification and game-based learning. Journal of Educational Technology Systems,
45(4), 499-519.
Kebritchi, M., Hirumi, A., & Bai, H. (2010). The effects of modern mathematics computer games on mathematics achievement
and class motivation. Computers & education, 55(2), 427-443.
Lester, J. C., Spires, H. A., Nietfeld, J. L., Minogue, J., Mott, B. W., & Lobene, E. V. (2014). Designing game-based learning
environments for elementary science education: A narrative-centered learning perspective. Information Sciences,
264, 4-18.
Machado, B. A., de Souza, F. B., Rodrigues, J. De S. & Hotta, B. Y. (2021). Game-based learning in a Production Engineering
course in Brazil. In: Proceedings of the International Conference on Active Learning in Engineering Education.
PAEE/ALE 2021, 7 - 9 July 2021. Braga - Portugal. (Conference website).
MEC – Ministério da Educação (2019). Diretrizes Curriculares Nacionais. Resolução n. 2, de 24 de abril de 2019.
Mello, C. H. P., Turrioni, J. B., Xavier, A. F., & Campos, D. F. (2012). Pesquisa-ação na engenharia de produção: proposta de
estruturação para sua condução. Production, 22, 1-13.
Moores, T. T. (2016). Teaching Tip: An Introduction to the Business Game 'Flowers for the World'. Journal of Information
Systems Education, 27(4), 217.
Rubeus. (2021). O novo perfil dos alunos: 5 comportamentos que sua IE precisa conhecer! [Post da web]. Recuperado de
https://rubeus.com.br/blog/o-novo-perfil-dos-alunos/
Sales, G. L., Cunha, J. L. L., Gonçalves, A. J., da Silva, J. B., & dos Santos, R. L. (2017). Gamificação e ensinagem híbrida na sala
de aula de física: metodologias ativas aplicadas aos espaços de aprendizagem e na prática docente. Conexões-
Ciência e Tecnologia, 11(2), 45-52.
Warren, S. J., Dondlinger, M. J., & Barab, S. A. (2008). A MUVE towards PBL writing: Effects of a digital learning environment
designed to improve elementary student writing. Journal of Research on Technology in Education, 41(1), 113-140.
109
Multidisciplinary projects for Engineering, Business
Administration and Design programs: Construction and
mapping of common skills through an analysis instrument
Matheus Ferreira Palú1, Octavio Mattasoglio Neto2, Eduardo Nadaleto da Matta3
1
Mauá Institute of Technology, Control and Automation Engineering, São Caetano do Sul, São Paulo, Brazil
2
Mauá Institute of Technology. Basic Cycle, São Caetano do Sul, São Paulo, Brazil
3
Mauá Institute of Technology, Basic Cycle, São Caetano do Sul, São Paulo, Brazil
Email: [email protected]; [email protected]; [email protected]
Abstract
In Brazil the Engineering, Business Administration and Design courses have, each one of them, a list of own competences,
or skills, that are established on the curricular guidelines by Ministry of Education, to be developed by students along the
course. This is not a closed list, and specific competences attending specifics courses demands can be included, in order to
training students with innovative and current profile. The engineering courses guidelines point out that students need to
be trained with active learning strategies putting them as protagonists of their training and, besides this, courses need to
make use of projects to develop the soft skills desirable in this training. To achieve an integrative approach, an institution
created a new curriculum in which the students of these three courses interact in multidisciplinary projects. This approach
has with bases the innovation tripod supporting the collaboration of the students of these three courses, aiming training
them to a global scenario that claims professional’s broad vision. Each multidisciplinary project is developed in an academic
semester and the students need to enrol in two projects per semester. In this context emerged the need of a convergent
list of competences that translate the competences of these three courses, without lose the essence of each one. This paper
presents the process of building this unique list of competences, for training professionals with an updated and innovative
profile. From this, a survey was carried out to gather information on the competences developed in these projects. The
research method was the documental analysis for the creation of the list of competences. To survey the competences that
are offered in the projects, a questionnaire sent to the instructors was used. The list of competences allowed the creation
of an activity plan, in which they declare the competences that will be developed by students, ensuring the management
of the learning process.
Keywords: Problem Based Learning; Competences; Multidisciplinary projects; Innovation tripod; Soft skills.
1 Introduction
The Engineering, Design and Business Administration courses at a higher education institution in Brazil have in
common the use of projects that are offered to students throughout the course and in parallel to the subjects
in the curriculum. These projects are "packaged" in a discipline called PAE (Projetos e Atividades Especiais, in
English, Special Projects and Activities) and thus meet the need to be placed as elements of the curriculum. In
this discipline, a list of more than two hundred projects is offered to the students, who have to choose at least
two in each academic semester (Mattasoglio Neto et al, 2019).
The focus of this paper is the competencies that these projects allow be developed by the students. These
projects need to meet two different aims: strictly speaking, the projects must develop technical and transversal
competences. At the same time the competencies developed must be aligned with those defined in the DCNs
(Diretrizes Curriculares Nacionais, in English, National Curriculum Guidelines) for these courses, Engineering,
Business, and Design, promulgated by the Ministry of Education (MEC, 2019), which are the courses in which
these projects are developed.
The projects offered to the students have another important characteristic. The students from all three courses
can enroll in any project, that is, there will be students with different profiles working together in the same
project.
110
The question discussed in this work is: How can we identify and define the competencies that will be developed
in each project whether at the same group, there are students from three different courses, with different
profiles and who should develop specific competencies determined by the National Curriculum Guidelines of
the Ministry of Education.
This work analyzes and compares the competences explored Engineering, Business and Design areas (MEC,
2004; MEC 2019; MEC 2020), investigating what is common and what is specific for each of these programs,
seeking to arrive in a set of competences that are common to these diverse areas of activity. The importance
of the study lies in the fact that entering the job market has become more demanding by the students, both
because the technical knowledge as for more professional attitudes that are being achieved in the curricula at
the start of the 21st century (MESQUITA et al, 2013).
To achieve this end, document analysis was used to compare the DCNs in order to identify which competences
and areas require more exploration, verifying which competences are most demanded in these careers, seeking
to promote the education of a professional with a profile with a broad vision that guarantees the solution of
problems effectively and efficiently, mobilizing knowledge and abilities with an entrepreneurial and innovative
professional attitude. In turn, it is desirable that the competences are aligned with the ONU's sustainable
development objectives that are translated into the Grand Challenges Scholar Program (GCSP) (BOTELHO et
al, 2020).
This nexialist vision (CHINALLI, 2013; SOUZA Jr, 2019a) focuses on the training of a professional with the ability
to connect different types of knowledge, in a current context in which there is a rapid expansion of technologies
in an interdisciplinary way, promoting new learning. In short, it is the professional capable of promoting the
connection of the problem to the knowledge of several areas, ensuring its solution, that is, giving it nexus.
Nexialists are professionals who manage to make these “different” connections
naturally, coherently organizing the relationships between different types of
knowledge. Therefore, they differ from the general brain pattern of knowledge
construction, they think “out of the box” as they say. They see connections where
others do not, see the opposite, escape the linear thinking of the absolute majority
of professionals and think in a systemic way (CHINALLI, 2013).
2 Methodology
The research method used in this work is a document analysis of the DCNs of the three courses. So, the
competences of the engineering, business and design degrees were analyzed and compared. Based on this
process, it was possible to generate a single framework as a result of this work. In addition, the transversal
competences that could best be associated with these three areas were also sought from other sources,
establishing a new set of common competences.
2.1 Study context

In the institution where the research was conducted, the "Innovation Tripod" (Souza Jr., 2019b), is used as a
reference, for the connection between the knowledge of the Business, Design and Engineering areas (Figure
1). Aiming to explore this concept, the institution has the PAEs (Special Projects and Activities) as a tool to
develop new competencies - both technical and transversal.
111
Figure 1 - Tripod of innovation at IMT (Souza Jr, 2019b)
The Figure 1 is about the union between these three courses, so that what is learned in one area can be used
in the other two, for example, the business administration area that makes the products more viable for the
consumption of a target audience (Viability), the design that can make this product aesthetically beautiful and
functional (Desirability), and engineering that deals with the components and operation of the product
(Feasibility). In general terms, the idea is to distribute these three great concepts so that, for example, an
engineer also knows something related to design.
This structuring of the course with the introduction of PAEs provides an opportunity for interaction with current
professionals working in the labor market, further increasing the richness of these experiences. It is worth
pointing out that any professional from the Brazilian or foreign market can propose a PAEs, which, if there are
students interested, can be offered and incorporated into the student's curriculum.
Based on the importance of these projects and on the needs and opportunities for the student's education,
the projects can contribute to, by developing the desired competencies, generating an enrichment of the
education, always based on the DCNs of the courses that guide the curricular planning of a discipline.
2.2 Data collection and analysis

The research developed had a few phases:
1st Step:
In this step, the focus was on the competences stated in the National Curriculum Guidelines for the three
degree programs, Engineering, Business and Design. The method used in this work was document analysis in
order to identify the competences that are intended to be developed in these degree programs. The
competences of the three programs were compared in order to find similarities and differences between them.
This comparison led to a single list that would cover all three degrees simultaneously.
2nd Step:
With the list of common competences, a survey instrument was designed, more precisely a questionnaire with
the aim of identifying which competences would be declared in the projects offered in the PAEs in the first
semester of 2022. This questionnaire was built and validated and structured in Google Forms, in order to
facilitate data collection for the research that was to be developed.
3rd Step:
This step consisted of registering the projects that would be offered to students in the first semester of 2022.
Each teacher or external professional who intended to offer a workshop or project as an PAE would have to fill
out a form in Google forms, in which they would indicate what competencies they would develop for the
students. The teachers need to detail some characteristics of the project for the registration of the activity, we
112
decided to make this registration online through a form and we took the opportunity to ask about the
competences, in other words, they were required to answer the questions desired by the authors for the
registration of the project, although other questions will be asked, the main focus will be on the competencies
part, the vast majority were for the registration of the activity itself.
Once these three steps were completed, the data from step 3 was synthesized in an Excel spreadsheet, which
allowed for its analysis.

3.1 Development
In practical terms, was separated and analyzed the eight DCNs for Engineering. Subsequently, the DCNs for
Business and then the DCNs for Design were together analyzed again. The initial idea was to create a common
list of 8 competences based on the Engineering DCNs, but one detail led to the appearance of more categories
of competences: an engineering competence did not always have a corresponding one for other courses, or
vice versa. This meant that some items in the common competency block had no connection to the 3 courses
in the DCNs. However, because some of them are related in a practical way, they are in the Common
Competencies block. The result was a single block with 10 competencies.
The transversal competences is a common block of 10 competences was also arrived at for the transversal
competences, which were taken from some work in the area of Engineering Education (MESQUITA et al, 2013,
MATTA, et al, 2018).
Summarizing, the technical competences and the transversal competences were indicated in the list of
competences for the graduation programs.
3.2 The competences list

The list of competences considered for the questionnaire is:
Transversal Competences:
• Critical Sense (Be Critical) - The idea is for the student to have a notion of what is tolerable or not
even before solving an exercise or problem, for example. This means that the student will have a
dexterity to solve challenges in a certain area of knowledge, in addition to knowing whether the values
and answers found are acceptable and correct.
• Information Selection (Be Selective) - To solve a project, an assignment, or something else, one
often needs to consult sources for help, be it the teacher, recommended books, or the subject matter
itself and this competence is responsible for selecting this information.
• Knowing how to face challenges - When it comes to real world problems, it is important that the
professional knows how to face these problems, it is important that the individual is courageous and
has solution attitudes, in some circumstances, it is necessary that the student has a solid opinion, a
solution and has the courage to introduce it to his environment.
• Proactivity (Initiative) - It is the ability to anticipate and responsibly target attitudes on imposed
occasions before they even happen.
• Create / Innovate - The ability to create products, assets, services (among others) in an innovative
and creative way, that is, to produce something useful, necessary, and new.
• Organization / Planning - Produce and act in an organized and planned way.
• Interpersonal relationship - Being empathetic and agreeable, that is, thinking of others working
together, acting fully, and making decisions thinking of the whole.
• Ability to deal with the unforeseen / Working in uncertain environments - The student's ability to
deal with problems rationally and calmly, to adapt to change, to withstand various pressures imposed
by the environment, and to overcome obstacles, basically resilience.
• Ability to solve problems - Be effective at solving problems.
• Make decisions - Be decisive and take harsh and determined attitudes.
113
Technical Competences:
• To formulate and design desirable and innovative solutions in your area.
• To analyze and understand the phenomena, events and models in your area based on the sciences
that underlie it.
• Creatively conceive, design and analyze systems, products (goods and services), components or
processes, technically and economically viable.
• Implement, oversee and control solutions in your area.
• Communicate effectively in written, oral and graphic forms, including communication in LIBRAS.
• Working and leading multidisciplinary teams
• Know and ethically apply the legislation and normative acts within the scope of the exercise of the
profession.
• Learn autonomously and deal with new and/or complex situations and contexts.
• Understand the potential of technologies and apply them in solving problems and taking advantage
of opportunities.
• Get to know the productive sector of your specialization, revealing a solid sectorial view, related to the
market, materials, production processes and technologies.
3.3 Survey Development

A questionnaire was created with a list of competencies, as well as other data used in the internal record of the
projects themselves. The importance of having this questionnaire is the easy access to the data and its analysis
in digital form.
Among the questions presented were "Has this PAE been offered sometime?", to give us control over which
ones had already been offered during and before the pandemic, "Choose your PAE category", giving the person
answering a list, making them choose among several areas, and the questions referring to the competencies,
where we put the two lists, each one as a question, and asked the teachers to choose a maximum of 3, in the
case of the transversal competencies, the respondent could choose none, but this was not the case with any
of the answers. In the technical competences analyzed, it was mandatory to choose at least one competence.
3.4 The questionnaire results

The data from the questionnaire are presented in the following.
Figure 2 - Has this project already been offered?
61
120
Yes No
114
It is worth noting that PAEs have existed at Mauá since 2015, and even after almost 7 years, there is always a
high demand for new PAEs, which is the case as more than 30% of these activities offered are new this semester.
This provides current and ever-renewing learning
Figure 3 - Are you already an instructor at IMT?
78
103
Yes No
Some PAEs instructors are professionals in the job market, and they do not work as teachers in the institution.
Therefore, we asked them about their origin, and we can see in Figure 3 - Are you already an instructor at IMT?
that almost 43% of the instructors who apply the PAEs are not professors, that is, they are professionals in the
job market. This is an interesting fact, as it creates an experience and a new window of opportunity with even
more qualified teachers.
Figure 4 - Transversal competences analyzed
100
91
90 83
80 73
QUANTITY FOR EACH SKILL
70
60
51 53
50
47 45
40 36
28
30
20 15
10
0
I II III IV V VI VII VIII IX X
TRANSVERSAL COMPETENCES
Label:
I – Critical Sense - Be Critical
II - Information Selection - Be Selective
III - Knowing how to face challenges
115
IV - Proactivity - Initiative
V - Create / Innovate
VI - Organization / Planning
VII - Interpersonal relationship
VIII - Ability to deal with the unforeseen / Working in uncertain
environments
IX - Ability to solve problems
X - Take decisions
It can be seen that among the transversal competences analysed, those that stood out most were those related
to problem-solving competences, which shows that the projects always have a "hands-on" idea, in other words,
being able to solve real-world problems. Next is the competence to create and innovate, which was also
expected to receive a high number of responses, since most projects have the idea of creating a professional
with new and effective resolutions. In general, it is possible to see that all competencies were met to some
extent, most of the questionnaires had 3 competencies, and in this question it was not even required to select
an item. With this data, we can see that this list of transversal competencies was satisfied.
Figure 5 - Technical competences analyzed
100
90 87
80 77
QUANTITY FOR EACH SKILL
71
70 67
62
60
50
43
40
30 24 26
21
20 15
10
0
I II III IV V VI VII VIII IX X
TECHNICAL COMPETENCES
Label:
I - To formulate and design desirable and innovative solutions in your area.
II - To analyze and understand the phenomena, events and models in your area based on the sciences that
underlie it.
III - Creatively conceive, design and analyze systems, products (goods and services), components or
processes, technically and economically viable.
IV - Implement, oversee and control solutions in your area.
V - Communicate effectively in written, oral and graphic forms, including communication in LIBRAS.
VI - Working and leading multidisciplinary teams.
116
VII - Know and ethically apply the legislation and normative acts within the scope of the exercise of the
profession.
VIII - Learn autonomously and deal with new and/or complex situations and contexts.
IX - Understand the potential of technologies and apply them in solving problems and taking advantage of
opportunities.
X - Get to know the productive sector of your specialization, revealing a solid sectorial view, related to the
market, materials, production processes and technologies.
In relation to these technical competences analyzed, was tried to see, as compactly as possible, the integration
of Engineering, Business and Design. The answers allows to see that all of them are fulfilled. One can see that
some were highlighted, especially those related to the formulation and conception of desirable solutions (I)
and design and analyze solutions technically and economically viable (III), that is, once again, the issue of the
"creative mind".
The Figure 5 also shows what was expected about the projects having "technical" competences to be
developed. An important detail is that even the least explored competences are still a high number in relation
to the total: practically 8.3% of the projects have the objective of knowing and ethically applying legislation
and normative acts within the scope of professional practice (VII).
In a practical way, for example competence VII, which was the least answered, if from these 15 answers about
20 classes are opened, since the same activity can have more than one timetable depending on student
demand, at least 350 students will be reached if these classes are opened, since the activities have about 20
students per class, that is a small demonstration that this "15" is still a very large number and that it can reach
a considerable number of students.
Table 1 - PAEs Category
Category Name Number of answers

Industrial Management 1
Mockups and modeling 1
Citizenship 2
Exact sciences 3
Energy 3
Expression & Imaging 3
Hardware & Embedded Systems 3
Games 3
Robotics & Automation 3
Materials 4
Simulation and computational optimization 4
Cities and urban solutions 5
Data Science 5
Academic competitions 5
Finance 6
Careers 8
Environment and Sustainability 8
Applied sciences 9
Management skills 10
Entrepreneurship 12
Product research and development 12
Engineering solutions 12
Industrial projects and processes 16
Development of socioemotional skills 17
Software training 26
117
The categories indicate to which area the offered project is directed. The idea is that these projects should be
diverse, since we are looking for a nexialist profile and, in this sense, the more areas students are offered, the
more versatile they can become. As it is an institution with a strong emphasis on Technology, it is natural to
have more courses related to software training.
4 Conclusion
The aim of this work was to create a common list of competences for Engineering, Business and Design
programs, so that the competences declared by instructors of projects offered to students from these three
courses can be surveyed. The importance of this common list lies in the fact that students from these three
courses can enroll and participate in these projects and having a common standard of competences helps in
identifying what is being taught in these projects.
The most important initial conclusion is that it was possible to construct this single list, which was validated
and allows the identification of common competences among these three courses. A second conclusion is that
it was possible to construct a data collection instrument on the projects that allows the instructor to state which
competences he intends to develop in the students. The third conclusion was a quantitative survey that allows
us to know which competences are being developed in the projects.
All of this allows for a management of the projects offered to students. In this way, it is possible to act so that
competencies that appear little in the projects offered to students are encouraged, in order to cover any gaps.
This allows a broad vision that helps in the management of the institution's courses.
Acknowledgments: To the teachers that kindly responded to the research. This work was partially supported
by IMT – Instituto Mauá de Tecnologia.
5 References
Botelho, V. B., Mattasoglio Neto, O., & LIMA, João de Sá B. (2020) Comparação entre as competências do Grand Challenges
Scholars Program e das DCNs do curso de graduação de engenharia – Anais: Congresso Brasileiro de Educação
em Engenharia. Bento Gonçalves, RS. 2020.
Chinalli, Didiane V. F. (2013) Nexialistas: Os designers do conhecimento e a nova era da tecnologia da educação. – UNAERP
– Universidade de Ribeirão Preto. https://www.unaerp.br/documentos/1468-166-462-1-sm/file . 05.02.2022.
Matta, E. N. da, Mattasoglio Neto, O., Lima, R. M., & Mesquita, D. (2018) The construction of the project evaluation concept
in a team of teachers. In: Proceedings: Fifth International Symposium on Project Approaches in Engineering
Education, PAEE-LE 2018.
Mattasoglio Neto, O., Lima, R. M., & Mesquita, D. (2019) Changing an Engineering Curriculum through a Co-Construction
Process: A Case Study. International Journal of Engineering Education. V. 35, n. 4, pp 1129-1140.
MEC (2020)– Ministério da Educação e Cultura. Conselho Nacional de Educação. Câmara de Educação Superior. Diretrizes
Curriculares para o curso de administração. Parecer CNE/CES nº 438/2020, aprovado em 10 de julho de 2020.
MEC (2004) – Ministério da Educação e Cultura. Conselho Nacional de Educação. Câmara de Educação Superior. Diretrizes
Curriculares para o curso de design. Resolução CNE/CES nº 5, de 8 de março de 2004.
MEC (2019) – Ministério da Educação e Cultura. Conselho Nacional de Educação. Câmara de Educação Superior. Diretrizes
Curriculares para os cursos de engenharia. Resolução CNE/CES nº 2, de 24 de abril de 2019.
Mesquita, D., Lima, R. M., & Flores, M. A. (2013) Developing professional competencies through projects in interaction with
companies: A study in Industrial Engineering and Management Master Degree. Proceedings: Fifth International
Symposium on Project Approaches in Engineering Education, PAEE'2013. Eindhoven, The Netherlands. 2013. Actas:
PAEE'2013. Eindhoven, The Netherlands.
Souza, Junior, J. C. (2019a) Inovação na Educação em Engenharia Gestão e Ações Institucionais em Prol da Renovação
Sustentável dessa Inovação. Lecture offered at “Engineerign Education for the Future”, 1st edition. ITA. Brasil.
Souza, Junior, J. C. (2019b) Mauá sob a óptica do Nexialismo! Youtube https://www.youtube.com/watch?v=yRUndB0g1gs.
05.02.2022.
118
An Escape Room For Learning Computer Programming
Pau Bofill1, Montse Farreras2, Jesús Armengol3
1
Escola Tècnica Superior d’Enginyeria de Telecomunicació de Barcelona, STH-UPC-Barcelona Tech
2
Escola Politècnica Superior d’Enginyeria de Vilanova i la Geltrú, UPC-Barcelona Tech
3
Group of Applied Optics and Image Processing (GOAPI), Optics and Optometry Department, Technical University of Catalunya UPC-
Barcelona Tech
Abstract
Game-based learning is a strategy where games are used as a challenge for students to learn and apply the contents of a
subject matter. In this sense, game-based learning is an instance of problem-based learning. In this paper we discuss how
game based strategies can be used to motivate students to perform the actions required for each of the learning phases.
Namely: motivation, information, understanding, application and validation (feed-back). Then we present the application
of those strategies to the design of an escape room where computer programs are required to solve the puzzles of the
game. The designed escape room is then used as a game-based strategy in an introductory seminar on the Python
programming language.
Keywords: Game-based Learning; Escape Room; Learning Phases; Learning Computer Programming.
1 Introduction
An escape room is a game activity where a team of players is confined in an enclosed room and needs to find
hints and clues to solve a series of puzzles required to open the locks that open the room. Escape rooms have
become very popular in the last decade. In addition to their original purpose as a game, escape rooms have
been extensively used in areas such as the enterprise (as team-building routines) and education (as learning
environments where puzzles are related to the subject matter) (Call et al, 2021; Lopez-Fernandez et al., 2021).
In the educational field, escape rooms can be used as problem based strategies to guide student teams towards
the completion of a set of goals. In a sense, escape rooms can be viewed as a situation from which “students
cannot escape without learning” (Cowan 1998; Bofill 2006).
In section 2 of this paper we focus on the different phases of learning (namely: motivation, information,
understanding, application and validation or feed-back) (Bofill & Miró, 2007) and, in section 3, we focus on
how the use of escape rooms can be used to induce student activities that enhance each of these phases. We
then describe our escape room (section 4), and its application to a game-based learning introductory seminar
for learning computer programming in Python (section 5). In this escape room, the puzzles require solving
computer programming problems, with hints hidden in the course notes and syllabus. Finally, section 6 is
devoted to conclusions.
2 The phases of learning

In (Bofill & Miró, 2007) we decompose the process of learning in a sequence of phases, in order to be able to
design student activities for each phase. The phases we proposed are the following:
1. Motivation, in two senses:
a. Subjective motivation reflects the attitude of the student towards the subject matter and
towards the activity of learning. A student is motivated when she wants to learn the contents
and, as a consequence, when she is ready to invest some time and effort to this purpose.
Subjective motivation can be intrinsic, when the student wants to learn because she likes it
(the reward for the effort is successful learning), or extrinsic, when studying is felt as an
obligation, and the effort is based on an external reward (or to avoid a punishment). Without
some intrinsic motivation, though, learning is bound to fail.
119
b. Motivation of the subject contents. The subject is well motivated if the student understands
the finality of what she is about to learn, and its relationship with what she already knows.
Subject contents must be contextualised and related to the previous experience of the learner.
Motivation of the contents must be accompanied with a clear formulation of the learning
objectives (what will the student get to know and be able to do after the learning process).
2. Information or acquisition of factual knowledge. The student must be somehow exposed to some
sort of explanation of the contents of what she is going to learn. In that phase, the student must gather
the information and process it. Information can be transmitted by frontal teaching, or with different
kinds of learning materials such as course notes, problem collections, books, tutorials, presentations,
videos and so on.
3. Understanding of the matter. After this phase, the student must be able to explain the subject matter,
and to find the relationships between the concepts introduced, and the relationship between new
concepts and what she already knows. Understanding requires significant learning. Sometimes
understanding is confused with learning as a whole: I understand it, therefore I know it. But often this
is not the case. The following phases are also necessary,
4. Application is the ability to transfer knowledge onto a different context, and it corresponds to deep
learning. Knowledge transference involves activities such as the evaluation of the new situation, the
identification of the required knowledge for the new context, and the translation of this knowledge to
the new requirements. For deep learning to take place, application should go beyond the solution of
prototype problems (procedural knowledge) based on a theory (conceptual knowledge).
5. Validation is the required feed-back to assess that we are following the right learning path. Validation
is required in all the previous phases: validation of the learning objectives, validation of the factual
knowledge, validation of understanding and validation of knowledge transference. Validation requires
assessment and feed-back in order to enhance what we did well, and in order to be able to learn from
mistakes. In an academic context, validation requires formative assessment, as opposed to selective
assessment and grading.
Quoting John Cowan (Cowan 1998):
“I take teaching to be: The purposeful creation of situations from which motivated learners should
not be able to escape without learning or developing”.
The purpose of the above scheme is to help in the design of such situations and activities, for each of the
learning phases.
3 Learning Python with a game-based learning approach

The course we want to design is a 20h introductory course to the Python programming language, for students
with some programming experience with other languages (C and Java). A “frontal” approach, like explaining
the structures and syntax of the language, one after another, would be boring, both for the students and the
teacher. Furthermore, computer programming is a skill and, like bike riding, it should be learned by practice.
Problem based learning is a resourceful alternative approach, well suited for the situation. In problem based
learning, the learner is confronted with a problem that needs to be solved, which requires the study of the
subject matter. In this way, the problem is the conductive thread that motivates the need for the subject matter
(learning phase 1.b).
Game-based learning can be organised in such a way that one or several problems are embedded in the game.
In our case, the student is confronted with an escape room situation where she needs to find clues and solve
programming problems to find the passwords that open the locks and produce the passwords of the game. In
this way, the student gets involved with the challenges and puzzles of the game, which enhances subjective
motivation (learning phase 1.a).
To avoid trivialization, care must be taken not to place too much emphasis on the game: the student must be
aware at all times that this is a “serious” game, and that the actual purpose of the game is learning. To this
120
effect, the programming problems embedded in the game should be sufficiently challenging and attractive by
themselves. In our course, the problems include a Morse decoder, a drawing with turtle graphics, an
hexadecimal to decimal converter and a fractional number calculator.
Information on the Python language (learning phase 2) is provided with a syllabus and a series of course notes
and examples that accompany each of the programming challenges. Besides, in order to motivate students to
study these notes, game-related hints are hidden within the notes. Therefore, students need to read the notes
in order to be able to proceed with the game.
Understanding of the language syntax and structures (learning phase 3) should be straightforward for students
who already know other languages, so the challenge lies in the application of these language structures to the
problems that need to be solved (learning phase 4).
Computer programming is a subject that is easy to validate (learning phase 5). A computer program either
solves the proposed problem or it does not. In this sense, feed-back is self provided and immediate (if the
problem is not too large), and different alternatives can be tested on the spot. Furthermore, the game takes
place in the classroom and the role of the teacher is to provide help in developing the computer problems
(and, eventually, to provide hints for the game when students get stuck).
Finally, playing games in teams is both more fun and more productive. In our course students are organised in
teams of two or three people, and care is taken that each student is involved with the computer programs.
The escape room is designed for 10h of work, altogether. In the remaining 10h the students must invent, design
and implement a small programming project with the help of the teacher (Lima et al, 2017).
4 The bridge over the river Splash, an escape-room
Paris 1901, The game starts at the telegraph office of the railway station. Suddenly an alarm goes off and the
telegraph machine spits this text:
... --- ...

-... .-. .. -.. --. . -... .-. --- -.- . -.
... - --- .--. - .-. .- .. -.
... --- ...
121
The players must understand that this is a Morse encoded message, and they must build a computer program
to decode it. Course notes are provided with all the Python structures required to write the program, and mixed
within the notes, there is a table with the Morse code.
Once they finish the program, the decoded text should be the following:
SOS
BRIDGE BROKEN
STOP TRAIN
SOS
The password to the next screen is the christian name of the inventor of the code.
The players then rush towards the train, which is already leaving the station, but the train platform is too high
to jump on it. They manage to figure out that they need a ladder, and they have to write a computer program
that uses turtle graphics to draw the picture of a ladder (which they show to the game assistants). The following
is one such drawing of a ladder:
Now, they use the ladder to climb onto the train platform but the door is locked. There is a sign that reads:
0x2F3A
The players have to realize that this is a number in hexadecimal format, and they must translate it into decimal
notation. To do so, of course, they have to build a computer program that translates any hexadecimal number
into a decimal base (the course notes include enough hints to solve the problem). Once they find the answer
they introduce it into the lock and they are able to enter the train.
But, elas! Once on the train, the brake gear is quite complicated. There is an instruction panel that says:
Good job! But there is one last problem to solve. The brake works with a gear shift that requires
a precise combination of wheels. The right combination is the simplified fraction resulting from the
following expression:
(8/21 + 3/14) * 3/5
122
Therefore, again, they need to build a computer program that can operate with fractional numbers.
Once they find the solution, they enter it and the train stops.
Figure 1. The escape room has been solved with success!!!
In the last window of the game (figure 1), a teacher-based solution of the programs is provided, so that they
can compare them with their own.
5 The Python seminar. Running and assessment of the escape room

The Python seminar will take place by the end of June (just before the PAEE/ALE conference) at the Telecos-
UPC school of Telecommunication Engineering. Twenty students will attend it. Without much explanation, the
123
students will be confronted with the escape-room and they will use the course material to develop their
programs, with the help of the teacher.
At the end of the seminar they will be asked to answer the following assessment quiz, which is based on the
SEEQ questionnaire (https://teaching.usask.ca/documents/seeq/Standardized_SEEQ_Instrument_at_UofS.pdf).
Statement Strongly Strongly

Disagree Neutral Agree
disagree agree
The game has been a motivation to start working on the subject 1 2 3 4 5
The game has engaged me to practice more 1 2 3 4 5
I have enjoyed learning through play 1 2 3 4 5
The game has helped me to acquire a deeper level of understanding 1 2 3 4 5
I like the fact that I could solve the problems at my own pace 1 2 3 4 5
Mention three positive aspects that you have enjoyed
Mention three shortcomings or things that you would change or

improve
Table 1. Summary of the proposed questionnaire for the assessment of the activity.
The processed results will be presented at the conference.
6 Conclusion
This paper presented the realisation of an escape room as a means of designing student activities to enhance
student involvement in each of the learning phases (motivation, information, understanding, application and
validation). The course will take place just before the conference and results will be presented there.
In the meantime, a short version of the escape room has been tested on a different course with good results:
students reported interest in moving along the windows of the game, and they were excited to escape the
room. They reported that the game provided extra motivation for solving the programming problems involved.
7 Final remark
I asked my friend, Miguel Valero, who has become a reference in cooperative learning and project based
learning (Valero, 2022), how he organises his learning activities so that the students get the most out of them.
He answered that, most of the time, he was happy enough if the students were active in and out of the class.
Learning, he said, is more likely to happen when students are active, rather than sitting and listening passively
to the teacher’s explanations.
Acknowledgement
This work has been supported by the Institut de Ciències de l’Educació (ICE-UPC-Tech) Convocatòria de
projectes d’innovació docent 2021 (Acord CG/2021/02/34, de 9 d'abril de 2021), and by the Escola Tècnica
Superior d'Enginyeria de Telecomunicació de Barcelona (upc telecos-UPC-Tech).
124
8 References
Bofill P. (2006), “Designing Activities from which the Student Cannot Escape without Learning”, in Proceedings of the ALE
2006 Conference.
Bofill P. y Miró J. (2007). “Las fases del aprendizaje: Un esquema para el análisis y diseño de actividades de
enseñanza/aprendizaje”, en Actas XIII Jornadas de Enseñanza Universitaria de la Informática, JENUI 2007, Thomson
Novática, ISBN: 978-84-9732-620-9, pp 559-560, Teruel, 16-18 de julio de 2007.
Call, T., Fox, E. and Sprint, G. (2021). Gamifying Software Engineering Tools to MotivateComputer Science Students to Start
and FinishProgramming Assignments Earlier. IEEE TRANSACTIONS ON EDUCATION, 64, 4, 423, 2021.
Cowan, J. (1998). On Becoming an Innovative University Teacher, Reflection in Action, SRHE and Open University
Press.
Lima, Rui & Dinis-Carvalho, José & Sousa, Rui & Alves, Anabela & Moreira, Francisco & Fernandes, Sandra Raquel &
Mesquita, Diana. (2017). Ten Years of Project-Based Learning (PBL) in Industrial Engineering and Management at
the University of Minho. 10.1007/978-94-6300-905-8_3.
Lopez-Fernandez,D., Gordillo, A, et al. (2021). Comparing Traditional Teaching and Game-Based Learning Using Teacher-
Authored Games on Computer Science Education. IEEE TRANSACTIONS ON EDUCATION, 64, 4, 367, 2021.
Valero, M. (2022). https://personals.ac.upc.edu/miguel/
125
An Escape Room for an alternative evaluation system
Montse Farreras1, Pau Bofill2, Jesus Armengol3, Adrian Asensio1

1
Escola Politècnica Superior d’Enginyeria de Vilanova i la Geltrú (EPSEVG) UPC-Barcelona Tech, Barcelona,Spain
2
Escola Tècnica Superior de Telecomunicació de Barcelona (ETSETB), UPC-Barcelona Tech, Barcelona, Spain
3
Group of Applied Optics and Image Processing (GOAPI), Optics and Optometry Department, UPC-Barcelona Tech. Barcelona, Spain
Abstract
Our traditional education system spins around evaluation and its traditional grading system: summative assessment. In that
system, professors spend time and effort trying to be “fair” when they mark the exercises, while students' main concern is
snatching enough points to pass. We believe learning should be at the center. Students should put their energy on learning
and professors on facilitating this learning process. However, we are still required to give a mark. We propose a game to
achieve formative assessment. Play is a source of motivation to both engage students and enhance learning. The aim is
that students should learn and enjoy their learning. We have designed an escape room with different sets of questions,
each set corresponds to one topic. The student is presented with one activity to solve correctly. There is no grading but
feedback: the answer can either be correct or incorrect. If the answer is correct, the student moves on to the next set of
questions, otherwise the student is challenged with another activity of the same set. This goes on until the student solves
one exercise correctly of each set. The student can only escape the room if one exercise of each topic is solved correctly.
All activities proposed are of a basic level and if the student escapes the room he has a pass. Another bundle of sets can
be done with more advanced exercises and if the student escapes the room he would achieve a higher mark. The method
emphasizes two fundamental (but neglected) aspects of education: (i) the joy of learning; and (ii) diversity, as we give
students as many opportunities as they need to learn at their own pace without penalty.
Keywords: Active Learning; Engineering Education; Formative Assessment; Game based learning
1 Introduction
Our traditional grading system is based on the much-dreaded exams. The exam period is a time of the year
when students are assessed on the knowledge they achieve during the term and it is a time usually marked by
stress, late nights, sore eyes and lots of coffee. With the good intention to alleviate the stress sometimes there
is a mid term exam, or even the so called continuous evaluation with its own shortcomings (Armengol, 2007).
In any case, the grading system is based on exams and summative assessment. Students are presented with
exercises to solve in a limited time and professors check how well they have done and based on that they give
a mark, if the mark is sufficiently high, the student passes the subject.
In this scenario, assessment does not support learning (Gibbs 2005):
● no feedback is given to the students to improve and facilitate their learning,
● Students may not have another chance to test their knowledge or in the best case scenario even if
there is another chance they have already been punished with a bad mark, which means that essentially
students are not allowed to make mistakes during their learning process. We believe it is by making
mistakes and reflecting on them that we learn the most (Farreras 2021).
However, we are still required to give a mark, is there a better way to grade? Yes, Virginia (Linda B.
Nilson,2016).
We propose to challenge the students with a certain amount of exercises that they need to solve correctly to
pass the subject - instead of grading the exercise(s) of a test with a mark. There are different sets of
questions/activities/exercises (i.e., one set for every subject topic). The student is presented with one activity
to solve correctly. There is no grading but feedback: the answer can either be correct or incorrect. If the answer
is correct, the student moves on to the next topic, otherwise the student is challenged with another activity of
the same set. This goes on until the student solves one exercise correctly off each set (or topic). .
126
We present this evaluation method as a Game-based learning (GBL) activity: an escape room. The student can
only escape the room if one exercise of each topic is solved correctly. By introducing Game-based evaluation
(GBE) into our subject we aim to achieve to following:
● Rescue the joy of learning. Learning for the sake of learning is actually a very enjoyable and fulfilling
activity. Joy and fulfillment are undermined by the pressure the traditional evaluation system puts on
the students (Bofill, P. 2005). Learning should not be seen as a sacrifice but as a privilege.
● Enhance learning. Students will gain deeper knowledge as they engage in the activities with the
purpose of solving them correctly, meeting the specifications of each problem. They will work on each
problem until they complete it. As opposed to doing what they can and hopping the mark is over the
pass line.
● Engage students. By engaging them on the challenges of the game, students will practice more and
practice makes better.
● Allow mistakes. Enough exercises are provided so that they can have as many attempts as they need,
allowing them to make mistakes without affecting their mark. Students can learn at their own pace,
accommodating a diversity of learners.
Overall, this paper makes the following contributions:
● We present our design and implementation of a escape room for First Year students of Computer
Structure I course in the Degree in Informatics Engineering curriculum. There are 50 students per
course divided into 3 groups for the practice sessions.
● We present a novel form of evaluation in the form of a game GBE that is stress free, for students
and professors, accommodates diversity of learners, engages students and enhances learning.
● We present our expectations and our plans and methodology to evaluate the activity (the escape
room). The activity is taking place at the time this paper is being written so we have not had the chance
to evaluate the outcomes yet. If the paper is accepted we plan to present the results at the conference.
The rest of the paper is organized as follows: Section 2 presents the design of the escape room game; section
3 discusses the evaluation aspect of the game; section 4 shows it implementation; section 5 evaluates the
outcome of the activity; and concluding remarks are written in section 5.
2 Theoretical framework
Recent research has applied gamification to many educational contexts, including Computer Science courses
(Call, 2021)(Lopez-Fernandez 2021). However not all gamification strategies are equally fruitful. As in (Dave
Eng, 2019), we make the distinction between gamification and Game-based learning (GBL). As opposed to
gamification GBL does not encourage competition, which is not an intrinsic source of motivation. In GBL the
students get involved in studying and solving subject matter problems to get the required hints to continue
the game. In this sense, game-based learning is an instance of problem-based learning as it complies with the
characteristics of PBL (Graaff, E. 2003).
PBL has long been proven to be an efective way to engage students and enhance learning (Yunita 2021)
(Chevalier 2021) (Lima 2017).
According to (Bofill, 2007) learning takes place in five stages. Namely: motivation, information retrieval,
understanding, application (or practice) and feed-back. We have chosen a escape room over other gamification
activities because in a escape room (and in general in GBL) a challenge is set for the students (in the form of a
game) and they have to learn the contents of the subject and apply them in order to solve the riddles to
progress through the game and escape the room. In this way, GBL reinforces the autonomous realization of
each of the learning stages.
Concerning assessment, we aim to achieve the conditions defined by (Gibbs G., Simpson C. (2005)) under which
assessement supports learning. They can be summarized as: Providing the right activities and timely formative
assessment which is used by the students and does not affect the grading. The escape room is a good
framework as feedback is provided immediately at every attempt and there is always another chance. It also
enables us to grade according to the system explained above.
127
3 Designing the game
The definition of an escape room: a game in which participants confined to a room or other enclosed setting
(such as a prison cell) are given a set amount of time to find a way to escape (as by discovering hidden clues
and solving a series of riddles or puzzles).
We have designed an escape room that seeks to have an epic mission, a clear goal, immediate feedback,
another chance to prove it and a positive social dimension (McGonigal 2011). Careful consideration has also
been given to each exercise so that students cannot escape without learning (Bofill, 2007).
3.1 The context

The escape room presented here was designed as a part of a Teaching Innovation Project to introduce Game
Based Learning (GBL) into first year subjects with the aim of fostering (cooperative) learning, engaging students,
and providing a positive social dimension. It is an interdisciplinary project that involves 3 different schools of
Engineering education in our University where GBL has been introduced in 4 different subjects . The project
involves 3 different schools of Enginery education in our University and GBL has been introduced in 4 different
subjects. This paper focuses on the experience in the Computer Structure I course (ESC1), which belongs to the
first year studies of the Bachelor’s degree in Informatics Engineering curriculum offered by the Polytechnic
School of Engineering of Vilanova i la Geltru (EPSEVG).
3.2 The set up or mise en place

The story line is as follows: there is a train at the platform about to take off when suddenly an urgent message
arrives at the station office about a fallen down bridge on the route of that train. The train needs to be stopped!
(epic mission). However it has already left the station and its communication system is broken. So you need to
jump onto the last coach and make progress from coach to coach until you reach the locomotive engine at the
driver’s compartment where the driver is going to test you before taking your advice to stop the train (the
goal).
Each coach has a security system: (i) first there is a password at the door that you need to guess; and (ii) an
alarm that you need to deactivate in a given amount of time, before a bomb explodes and the train blows up.
Each team needs to move ahead from coach to coach, and each coach is locked with a unique password that
all teams need to figure out together.
Once they get to the engine, the teams meet the driver and there is one final challenge as the driver is a
cautious person and gets suspicious when you break into the train engine’s room and ask him to stop the train.
So he locks each person alone into the room, from where you can only escape by answering correctly to 5
questions. Only then will the driver be convinced that you are a trustable person.
3.3 The challenges and the course timeline

The escape room is made of three types of challenges. It is a cooperative game, so the students work in teams
of 2 or 3 students, chosen by themselves. Teams do not compete but work together towards achieving the
goal. The games start in the classroom at the practice sessions, where the whole class is slpit into three smaller
groups, and can be finalized at home.
The first session will be spent presenting the game, signing a contract to set up the rules and responsibilities
and setting up the working environment for the practice sessions with the computer (lab sessions).
3.3.1 Challenge 0: Consult the reference book

The first (preliminary) challenge is to jump into the train before it leaves the station and unlock the door of the
last coach. The riddle is a question which answer can be found in the reference book of the subject so teams
need to visit the school library. Once they have the password they can begin.
Each coach represents a practice session at the lab. These sessions run every fortnight and there are 4 lab
sessions during the course. Each lab session has two parts, that we have linked to our next two challenges:
128
3.3.2 Challenge 1: Deactivate the alarm
(i)First, there is a pre-assignment the students need to complete beforehand. Once in the classroom, answering
correctly to a set of questions related to the pre-assignment will deactivate the alarm. These questions are
the same for all groups and a limited amount of time has been given for them to answer the questions
before the timer expires.
3.3.3 Challenge 2: Unlock the door to the next coach

(ii)During the rest of the lab session the students work on some codes to practice the topics that have been
introduced into the theoretical lessons. They are presented with a problem to solve that is split into tasks of
equivalent difficulty and educational content and each team takes one task (there is a riddle to solve to
figure out which task each team needs to solve). The tasks need to be combined to solve the problem and
only the correct answer to the problem will give the right password to unlock the door to the next coach.
The teams need to work together making the game cooperative and providing a positive social experience.
After answering each question immediate feedback is provided and they have as many attempts as they need
in the given time. Students work in teams in challenge 1 but in challenge 2 teams need to work together
making the game cooperative and providing a positive social experience. All activities proposed for challenge
1 and 2 are of basic level and if the group reach the driver’s compartment, they have a pass.
3.3.4 Challenge 3: The driver’s test

This is a final individual challenge, students reach this point towards the end of the term. In this challenge
another bundle of questions is used of a higher difficulty. There are 5 sets of questions which correspond to
the 5 topics of the subject matter. In order to escape the room each student needs to respond correctly to one
question of each set.
The student is presented with one exercise to solve correctly. If he succeeds in providing the correct answer
(feedback is immediate), he moves on to the next set of questions, otherwise the student is challenged with
another activity of the same set (mistakes are allowed!). The new activity will be a different one but equivalent
in the sense that it relates to the same topic. All activities in one set cover the same educational content. This
goes on until the student solves one exercise correctly off each set. In that way he has covered the full content
of the course.
The students who pass the driver’s test and escape the room will obtain a higher mark.
All challenges preserve our ethos, each team works at their own pace, formative assessment is given and the
teams work in a cooperative way.
4 Game Based Evaluation

Using a escape room game for grading the students progress is a novel form of evaluation as it shifts the focus
from (a) a scenario where students try to solve a task(s) the best they can and are given points according on
how well they did; to (b) a scenario where there is a set of tasks and students are asked to solve one of them
correctly. In (b) they can have as many attempts as they need (limited only by the number tasks on the set), for
each attempt there is formative assessment so the student can progress.
The task can be a single one which covers the full content of the subject matter or several smaller tasks, one
for each topic of the subject. And the level of the tasks may vary, there can be bundles of exercises at different
levels (basic, medium, high) and the mark could be set depending on how many bundles the student can solve
correctly. Students who complete the basic set correctly, but fail to complete the medium set, may have a pass,
but students who complete the basic set plus the medium they may have a higher mark, and the highest mark
can be achieved by completing the 3 sets.
As (Linda B. Nilson,2016) states it: all assignments and tests are graded satisfactory/unsatisfactory (pass/fail),
depending on whether the work meets its specifications. In this way the students need to correctly solve a
certain amount of exercises in order to pass the subject instead of grading the exercise(s).
129
In this way, formative assessment is provided at every attempt and assessment is used to support the learning
(Gibbs G., Simpson C. (2005)).
In our subject we have used the escape room to grade the practical work, which is part of the final mark of
each student. It is a cooperative game, so even if there are teams, the full class (lab groups) either succeeds or
fails. If they succeed to stop the train on time they will be graded the practical work with a pass (minimum).
The students who also pass the driver’s test will have a higher mark (7/10). Students have also delivered tasks
which will be used to grade the teams beyond 7.
This is the first time we implement GBL in the subject, our set of questions covers the basic and intermediate
levels, the former used for challenges 1 and 2, and the later for challenge 3. The game-based Evaluation
methodology presented has only been used to grade the practical sessions for now.
5 The escape room implementation

For the implementation we looked at several available tools: Socrative, padlet, moodle questionnaires among
others.
The logistic requirements of the game are: (i) possibility to play online (if necessary); (ii) engaging for students,
it involves an stimulating and appealing graphical interface; (iii) and it should be possible (and fairly
straightforward) to implement our challenges (escape room style).
We finally settled on a combination between genially (https://genial.ly/) and moodle questionnaires
(https://docs.moodle.org/). Challenge 1 was fully implemented on genially with a nice graphical interface.
Challenges 0 and 2 were synchronizing moodle and genially. And Challenge 3 was fully implemented using
moodle questionnaires.
Figure 1 shows two of the introductory screens of the game; (a) presents the mission and (b) is the interactive
map of the train that they can use to navigate for every lab session. All lab sessions are locked until the
password is obtained, in due course.
Figure 1. Image of the first screens of the game (a) the mission; (b) the interactive map
We found that Genially gave us a nice graphical interface but the type of questions was limited. Multiple choice
question type is available by default (shown in Figure 2 (a)); and we have installed a plugin to support the Short
answer type of questions (shown in Figure 2 (b)). On the contrary moodle has a richer repertory of question
forms which allowed us more elaborate questions (https://docs.moodle.org/400/en/Question_types).
130
Figure 2. Image of the two types of question we used in genially (a) single choice; (b) short answer
Figure 3(a) shows the password page that locks each lab session (the key to open the coach door), which at
the same time, allowed us to synchronize with the timeline of the course using a single genially. Figure 3(b)
shows the final page after challenge 1 is completed on couch 2. In this page we give a small riddle which allows
us to synchronize with the moodle platform.
Figure 3. Image of (a) password that locks the door of coach 2 on the left; (b) Final pages after succeeding to
disconnect the alarm.
Figure 4 shows what happens if things go wrong. Figure 4(a) shows a train accident, the train falling down on
the broken bridge, and 4(b) shows a bomb explosion when failing to disconnect the alarm. The good news is
that this is a game and the student can always go back and have another chance.
Figure 4. Image of (a) a falling down train, when a question is answered incorrectly; and (b) the bomb explosion if the
timer expires.
6 Outcomes and evaluation

At the time the paper is being written we are halfway through the term and the students are carrying on the
activity. At this point all we can say is that they seem to be enjoying themselves with the game and fully
131
engaged with it, we have full attendance to the classroom and we have not had any student withdrawing the
course up to this point.
In order to evaluate the activity and improve it for later editions we have prepared a questionnaire based on
the Student Evaluation of Educational Quality (SEEQ) Standardized Instrument. Table 1 below shows the
questions related to the game and the evaluation methodology. We will ask the students to fill the
questionnaire and if the present paper is accepted we pretend to present the results at the conference.
Strongly Strongly
Statement Disagree Neutral Agree
disagree agree
The game has engaged me to have more practice 1 2 3 4 5
I like the fact that I can have as many attempts at the exercises as I
need 1 2 3 4 5
I think the evaluation methodology of the lab is adequate and fair 1 2 3 4 5
Mention three positive aspects that you have enjoyed of benefit from
regarding the game (lab sessions)
Mention three shortcomings of things that you would improve on the
game (lab sessions)
Table 1. Summary of the questionary based on SEEQ regarding GBL and evaluation methodology
7 Conclusion
This paper presents a Game Based Learning (GBL) activity where we use an escape room not only to engage
students and enhance learning but also as a form of evaluation (Game Based Evaluation). We present the
students with a set of challenges that they need to solve correctly in order to escape the room, save the train
and pass the subject. If they fail to complete one exercise another opportunity is given, with another exercise
covering the same educational content. In this way the students need to correctly solve a certain amount of
exercises in order to pass the subject instead of grading how well they have completed the exercise(s).
GBE provides a stress free evaluation methodology that emphasizes two fundamental (but neglected) aspects
of education: (i) the joy of learning; and (ii) diversity, as we give students as many opportunities as they need
to learn at their own pace without penalty.
8 Acknowledgement
This work has been supported by the Institut de Ciències de l’Educació (ICE) Convocatòria de projectes
d’innovació docent 2021 (Acord CG/2021/02/34, de 9 d'abril de 2021) and by Escola Tècnica d’Enginyeria de
Telecomunicació (upc telecos-Barcelona Tech).
9 References
Armengol, J., (2007). Continuous assessment, pros and cons (or the art to combine summative and formative assessment). Active Learning
in Engineering Education Workshop, 99-104.
Bofill,P. (2007) Designing activities from which the student cannot escape without learning. Active Learning in Engineering Education
Workshop
Bofill,P. (2005) Anything you say can and will be used againt you. Active Learning in Engineering Education Workshop, 113-120.
Call T., Fox E., Sprint G, (2021) Gamifying Software Engineering Tools to Motivate Computer Science Students to Start and Finish
Programming Assignments Earlier. IEEE Trans. Educ Vol 64 no.4
Chevalier A., (2021) Student Feedback on Educational Innovation in Control Engineering: Active Learning in Practice IEEE Trans. Educ. 64(4):
432-437
Dave Eng, 2019. https://medium.com/@davengdesign/game-based-learning-vs-gamification-f44e5b09d808
Farreras M. Bofill P. Armengol J, Roma M. (2021) Plant a seed for the future. PAEE/ALE'2021 International Conference on Active Learning
in Engineering Education
132
Gibbs G., Simpson C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education,
Issue 1, 3-31.
Graaff, E. d., & Kolmos, A. (2003). Characteristics of Problem–Based Learning. International Journal of Engineering Education, 19(5), 657-
662.
Lima R. et al. (2017) Ten Years of Project-Based Learning (PBL) in Industrial Engineering and Management at the University of Minho. PBL
in Engineering Education 37-51
Linda B. Nilson, (2016) Yes, Virginia, There's a Better Way to Grade By https://www.insidehighered.com/views/2016/01/19/new-ways-
grade-more-effectively-essay
Lopez-Fernandez D., Gordillo A., et al. (2021) Comparing Traditional Teaching and Game-based Learning Using Teacher-Authored Games
on Computer Science Education IEEE Trans. Educ Vol 64,no 4.
Yunita Y., Dadang J., et al. (2021) The effectiveness of the Project-Based Learning (PBL) model in students’ mathematical ability: A systematic
literature review Journal of Physics Conference Series 1882(1):012080
133
Lecturer Self-Awareness Index: measuring the alignment
between lecturer and student perception
Humberto Arruda1, Édison Renato Silva1

1
Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil
Abstract
During last decades, there was a transformation in the Engineering Education field that resulted in the widespread use of
Active Learning techniques, with the goal to place the student at the center of the learning process. However, there was a
lack of effective ways to evaluate how a course uses those techniques. Although Active Learning has already been validated
as an effective way to influence student learning and is increasingly being validated into the classroom, there was no way
to qualify and evaluate the use of Active Learning techniques by faculty members, until the proposal of a conceptual model,
called Engineering Education Active Learning Maturity Model (E2ALM2). The application of E2ALM2 uses questionnaires
applied to course lecturers and their students. This article proposes the Lecturer Self-Awareness Index (LSAI), that derives
from E2ALM2, as a complement to measure the discrepancy between lecturer and student perception. LSAI allows a lecturer
to identify items in which his perception diverges from students’. This tool can help improve decisions in course preparation
and lead to a better delivery.
Keywords: Active Learning; Engineering Education; assessment; self-awareness
1 Introduction
One of the great challenges of higher education in the 21st century is to adapt traditional teaching techniques
to the reality of today's students, who have different characteristics from previous generations (Hartikainen,
Rintala, Pylväs, & Nokelainen, 2019). Student-centered strategies increase student engagement, which is linked
to teaching effectiveness. The positive results have already been widely explored (Burke & Fedorek, 2017; Cho,
Mazze, Dika, & Gehrig, 2015; Prince, 2004).
Engineering Schools eager to modernize educational practices need to understand their current maturity of
Active Learning, which can be the first step of a renovation program. To achieve this goal, Arruda & Silva (2021)
present the Engineering Education Active Learning Maturity Model (E2ALM2). This framework processes data
collected from students and lecturers of a course to depict the current maturity level of the Active Learning
implementation in terms of five dimensions, composed of 14 key success factors (KSF) that summarize 41
constructs.
Self-awareness is positively associated with transformational leadership (Titrek & Çelik, 2011), leadership
performance and success (Axelrod, 2012; Caldwell & Hayes, 2016; Showry & Manasa, 2014; Whetten &
Cameron, 2016). It leads to better decision making, once it connects the past and present experience (Kamenov,
2013), and supports effective learning, because the more students know their  ‘self’, the better they increase
their own personal developments (Murphy, 2007). Furthermore, self-awareness increases sympathy (Boyer,
2010; Gair, 2011; Smith, 2011), reduces egocentrism (Gendolla & Wicklund, 2009; Scaffidi Abbate, Boca, &
Gendolla, 2016), and is perceived as central to improving management skills (Whetten & Cameron, 2016).
Regardless of the specific maturity level in active learning implementation, the perceptions of students and
lecturers can be quite misaligned, due to the natural difference in points of view. The aim of the Lecturer Self-
Awareness Index (LSAI) is to depict these possible discrepancies and provide the decision-making process with
feedback between how lecturers and students perceive the benefits and characteristics of a given active
learning implementation.
In this article, the concept of self-awareness is emphasized, bringing some definitions, motivations, and
limitations. Additionally, the concept of Active Learning is presented, followed by E 2ALM2 details. Then, LSAI is
134
defined and detailed. Finally, we conclude with some discussions about the advantages and limitations of the
index.
2 Self-awareness
Although the term self-awareness is widely used in practitioner and scientific publications, the exact meaning
of the term is still unclear (Carden, Jones, & Passmore, 2022). The literature brings several definitions of self-
awareness (Nutt Williams, 2008; Sutton, 2016; Sutton, Williams, & Allinson, 2015), but confusion between the
concepts of self-awareness, self-knowledge and self-consciousness is common (Morin, 2017; Sutton, 2016).
Also, definitions seem to depend on focus and context (Sutton, 2016).
Goleman (1995) defines self-awareness as “knowing one’s internal states, preferences, resources, and
intuitions”. There are statements that self-awareness refers to the constant consciousness of feelings, thoughts
and behaviors (Harrington & Loffredo, 2010; Nutt Williams & Fauth, 2005) and how someone’s behaviors
impact other individuals (Mayer, Salovey, & Caruso, 2004).
In their systematic literature review, Carden et al (2022) offer the following definition:
“Self-awareness consists of a range of components, which can be developed through focus, evaluation
and feedback, and provides an individual with an awareness of their internal state (emotions, cognitions,
physiological responses), that drives their behaviors (beliefs, values and motivations) and an awareness
of how this impacts and influences others.”
However, self-awareness is a difficult journey. Practicing it may feel risky and frightening for some students,
and its development requires a safe learning environment (Feize & Faver, 2019). In  the process of developing
self-awareness, a person might find discrepancies between the  real self and the ideal self (Gallwey, 2001; Silvia
& Duval, 2001).
Finally, Govern and Marsch (2001) discuss the manipulation and measurement of levels of situational self- focus.
Titrek (2011) analyses the Self-Awareness Competence Scale and Demerouti et al (2011) state that a high level
of self-awareness could be considered as a high consistency between self-evaluation and the evaluations of
others.
3 Active Learning and its assessment

Despite the vast literature available on Active Learning, there is still no consensus definition on the topic.
Hartikainen (2019) shows 66 definitions grouped by three main categories: (1) defined and viewed as an
instructional approach; (2) not defined but viewed as an instructional approach; and (3) not defined but viewed
as a learning approach. We highlight three popular definitions: “any instructional method [used in the
classroom] that engages students in the learning process” (Prince, 2004), “an umbrella term for pedagogies
focusing on student activity and student engagement in the learning process” (Roehl, Reddy, & Shannon, 2013)
and “an umbrella term that now refers to several models of instruction, including cooperative and collaborative
learning, discovery learning, experiential learning, problem-based learning, and inquiry-based learning”
(Barkley, 2010).
In the Engineering Education field, it is possible to identify several movements with the objective of
modernizing programs and teaching process, such as the CDIO initiative (Crawley, Malmqvist, Östlund, Brodeur,
& Edström, 2014), the change in ABET's accreditation criteria (called EC2000) (Lattuca, Terenzini, & Volkwein,
2006), and the greenfield creation of engineering colleges with proposals totally different from the traditional
20th Century model, such as Olin College (Goldberg & Somerville, 2014) and Aalborg University (Mohd-yusof,
Arsat, Borhan, Graaff, & Kolmos, 2013).
A common point among the modernization movements in the engineering education field is the
recommendation to place the student at the center of the learning process, which drives the increasing use of
active learning techniques (Hartikainen et al., 2019).
135
From the consensus on the importance of using Active Learning techniques, there are different levels of
maturity in this subject. At one extreme, it is possible to observe isolated actions of teachers who experience a
change in a course. At the other extreme, there are programs designed as project-based, with materials and
assessment methods aligned, in addition to adequate infrastructure for classes in formats different from the
traditional model, in which students attend the class in rows watching a lecturer in front of the classroom.
Considering the growing popularity of these techniques, Arruda & Silva (2021) introduce the Engineering
Education Active Learning Maturity Model (E2ALM2), a framework that allows practitioners to assess the current
maturity level of Active Learning implementation in a course. This framework can support diagnosis and
practical improvements in real settings.
3.1 E2ALM2 method

The Engineering Education Active Learning Maturity Model (E2ALM2) (Arruda & Silva (2021) was built from a
literature review of key success factors (KSF) for Active Learning implementations, which were divided into five
dimensions: content quality, organizational environment, organizational infrastructure, lecturer and
interactions. These five dimensions are subdivided in fourteen KSF. Each KSF is composed of one or more
constructs, adding up to 41 constructs in total. Each construct is detailed in variables, totaling 89 of them.
At the lowest level, each variable has a grade. To apply the model to real cases, two questionnaires are used:
Student Questionnaire (SQ) and Lecturer Questionnaire (LQ). Figure 1 shows the E2ALM2 structure and those
questionnaires.
Figure 1 - E2ALM2 structure and questionnaires
Each question in the questionnaires is linked to a variable, and based on the result obtained in this question, a
value is assigned (Question value). Figure 2 shows the condition to calculate this value. From the Question
Value of all questions, the score for the linked variable is calculated. The construct score is the average of the
variables that compose it. The KSF score is the average of the constructs that compose it. Due to its nature,
there are variables that are present in both questionnaires and others that are present in only one of them.
Figure 2 - Question value condition
136
Then, to calculate the dimension score, the grades of the KSF that compose it and the weights are used,
calculated from the application of the Analytical Hierarchy Process – AHP method (Saaty, 2008), with answers
from paired comparisons of importance between the KSFs that make up the evaluated dimension. These
answers were provided by Active Learning experts consulted specifically for this purpose. Figure 3 shows the
process to calculate the dimension score.
Figure 3 - Dimension score process
4 LSAI Method
As the application of the E2ALM2 is based on the responses of the students and the lecturer, it is possible that
there are divergences between the perceptions of both sides. Due to this fact, the Lecturer Self-Awareness
Index (LSAI) was created as a way of identifying the discrepancy between the two points of view. Figure 4 shows
the E2ALM2 framework and its KSFs that make up the LSAI.
Figure 4 - E2ALM2 framework and LSAI components
Due to its nature, LSAI can only be calculated from KSF that have both student and lecturer measurements. It
is calculated from the difference of each question in the Student Questionnaire and in the Lecturer
Questionnaire. The maximum value (5) indicates that the lecturer's perception is fully aligned with the students'
perception, while the minimum value (0) indicates the total misalignment between the two perceptions.
4.1 LSAI composition

The constructs that compose the LSAI calculation (those that have scores in both the SQ and the LQ) are shown
in the Table 1.
137
Table 1 - LSAI constructs and Key Success Factors
Construct Key Success Factor Dimension
Use of real-life problems Course Activities Content Quality
Application of active experiments Course Activities Content Quality
Variety of instructional resources Course Activities Content Quality
Adequacy to Learning Outcomes (LO) Course Activities Content Quality
Clearness of assessment methods Student Assessment Content Quality
Clearness of criteria for success Student Assessment Content Quality
Communications with students Student Assessment Content Quality
Preparation of students to conduct activities required Learning Facilitation Content Quality
Classrooms designed for improve active learning

experience Classrooms Organizational infrastructure
Classrooms equipped with technologies to enhance

student learning and support teaching innovation Classrooms Organizational infrastructure
Availability of technology Technology Organizational infrastructure
Accessibility of technology Technology Organizational infrastructure
Usability of technology Technology Organizational infrastructure
Interactions students/lecturers Interactions with lecturers Interactions
As shown in the table 1, LSAI is calculated from just six KSF: Course Activities, Student Assessment, Learning
Facilitation, Classrooms, Technology e Interactions with lecturers.
According to Arruda & Silva (2021), those KSF have following specifications:
• Course Activities (problems, projects or cases studied) should: engage students with real-life problems
and active experiences; provide students with a variety of additional instructional resources, such as
simulations, case studies, videos, and demonstrations; be suitable to achieve different targets including
the support of the students’ learning process and establishing learning outcomes requirements; be
clearly written, in the right length, useful, flexible, and provide appropriate degree of breadth; have
suitable intellectual challenge; and begin with an explanation of its purpose.
• Student Assessment needs to be clear, concise, and consistent. This involves instructions, assignments,
assessments, due dates, course pages, and office hours. Furthermore, criteria for success must be
communicated clearly and monitored.
• Learning facilitation includes the preparation of students to conduct activities and tasks required in
addition to activities related to the facilitator guiding the learning process of the students. It also
involves providing students with regular opportunities for formative feedback from the lecturer.
• Classrooms should be designed for improve Active Learning experience and be equipped with
technologies that can enhance student learning and support teaching innovation.
• Technology involves availability, reliability, accessibility, usability of devices, internet (Wi-Fi), learning
support, and inclusive learning environment.
• Interaction between students and lecturer supports knowledge construction, motivation, and the
establishment of a social relationship. Furthermore, constructive, and enriching feedbacks from the
lecturer lead to increasing academic success and feelings of support.
As can be seen in the specifications above, in addition to the nature of the LSAI, all the KSFs that make up this
indicator have the potential for different perceptions by the lecturer and the students. However, the capacity
to trigger actions to improve the course (and the conditions that surround it) is much more concentrated on
138
the lecturer than on the students, because although the lecturer acts as a facilitator and co-constructor of
learning, he has a role in the organization structure. This way, it is possible to identify students as clients of the
service provided by the organization and it is essential that there is a way to assess how much the lecturer's
perception is in fact aligned with the perception of clients (students) in relation to this service.
5 Discussion
The application of E2ALM2 allows the diagnosis of a course in the five dimensions that make up the index, as
already discussed. As a way of representing the result, a radar chart is constructed, as in figure 5.
Figure 5 - Example of E2ALM2 real application
For each KSF, lecturer and student perceptions may converge or diverge. Possible combinations are shown in
figure 6.
Figure 6 - Student and lecturer perceptions
From the answers to the questionnaires, the LSAI in your six KSF is also calculated. The line LSAI = 0 is the
bisector of the quadrant shown in the figure 7. Analyzing the example shown as follows, it is possible to identify
that the KSF in which perceptions are most discrepant is “Classrooms”.
139
Figure 7 - LSAI results
In this example, KSF “Classrooms” is a component of the dimension “Organizational infrastructure”, which had
the lowest grade between five dimensions of E2ALM2. Then, while this dimension is possibly a course weakness,
the lecturer’s and students’ perception are misaligned. This way, any effective improvement action could be
hampered.
Looking at this example, two variables of KSF “Classrooms” are showed: (i) % of activities performed in an
environment suitable for Active Learning and (ii) % of activities performed in a technologically appropriate
environment. So, if the lecturer is not concerned with finding more suitable physical spaces for the activities of
the course, this indicator will hardly improve. Thus, students' perception of quality will remain compromised.
Here, it is important to remember that self-awareness as considered as central to improving management skills
(Whetten & Cameron, 2016).
As a limitation, it is important to highlight that E2ALM2 does not assess the quality of a course but identifies
the maturity level related to active learning. In addition, data collection made only with questionnaires can lead
to bias or have answers that reflect some student dissatisfaction with other factors. Finally, it is essential to
highlight the importance of validation of the questionnaires, to eliminate another kind of bias. This validation
is still planned to future steps.
6 Conclusion
Active Learning is a broad concept that has been boosted during last years, especially in the Engineering
Education field. As a first step to an improvement plan, it is necessary to diagnose the status of a course. Then,
to assess the maturity level of an AL implementation, E 2ALM2 is used.
Additionally, self-awareness is understood as an important capacity to high levels of leadership, better decision
making and management skills. Thus, this concept is intrinsically linked to the path to improvement actions,
with the goal of upgrading course conditions and evaluation.
After E2ALM2 application, some factors can show misalignment between lecturer’s and students’ perceptions.
To identify this discrepancy, this paper presented Lecturer Self-Awareness Index (LSAI). This index supports the
identification process of critical factors that need special attention to get better results to students.
As future work, we suggest a quantitative study, comparing different applications of E 2ALM2. This study is
ongoing and will be the subject of future work. Initially, it was focused on five courses, from Brazil and Portugal.
These cases are being studied and will serve to better understand both the application process and the search
for cause-effect relationships between practices and results, in addition to serving as a basis for the validation
of questionnaires, the main information collection tool.This could be a start of a predictive model, that can
provide hints and suggest approaches to enrich courses.
140
7 References
Arruda, H., & Silva, É. R. (2021). Assessment and evaluation in active learning implementations: Introducing the engineering
education active learning maturity model. Education Sciences, 11(11). https://doi.org/10.3390/educsci11110690
Axelrod, S. D. (2012). “Self-awareness”: at the interface of executive development and psychoanalytic therapy.
Psychoanalytic Inquiry, 32(4), 340–357.
Barkley, E. F. (2010). Student Engagement Techniques: A Handbook for College Faculty (1st ed.). Retrieved from
http://library1.nida.ac.th/termpaper6/sd/2554/19755.pdf
Boyer, W. (2010). Empathy development in teacher candidates. Early Childhood Education Journal, 38(4), 313–321.
Burke, A. S., & Fedorek, B. (2017). Does “flipping” promote engagement?: A comparison of a traditional, online, and flipped
class. Active Learning in Higher Education, 18(1), 11–24. https://doi.org/10.1177/1469787417693487
Caldwell, C., & Hayes, L. A. (2016). Self-efficacy and self-awareness: moral insights to increased leader effectiveness. Journal
of Management Development.
Carden, J., Jones, R. J., & Passmore, J. (2022). Defining Self-Awareness in the Context of Adult Development: A Systematic
Literature Review. Journal of Management Education, 46(1), 140–177. https://doi.org/10.1177/1052562921990065
Cho, C.-S., Mazze, C. E., Dika, S. L., & Gehrig, G. B. (2015). Enhancing Construction Education: Implementing Habitat for
Humanity Projects as Service-Learning for Construction Materials. International Journal of Construction Education and
Research, 11(1), 4–20. https://doi.org/10.1080/15578771.2014.915274
Crawley, E. F., Malmqvist, J., Östlund, S., Brodeur, D. R., & Edström, K. (2014). Rethinking engineering education: The CDIO
Approach (2nd ed.). https://doi.org/10.1109/FIE.2017.8190506
Demerouti, E., van Eeuwijk, E., Snelder, M., & Wild, U. (2011). Assessing the effects of a “personal effectiveness” training on
psychological capital, assertiveness and self‐awareness using self‐other agreement. Career Development
International.
Feize, L., & Faver, C. (2019). Teaching self-awareness: social work educators’ endeavors and struggles. Social Work Education,
38(2), 159–176. https://doi.org/10.1080/02615479.2018.1523383
Gair, S. (2011). Creating spaces for critical reflection in social work education: Learning from a classroom-based empathy
project. Reflective Practice, 12(6), 791–802.
Gallwey, W. T. (2001). The inner game of work: Focus, learning, pleasure, and mobility in the workplace. Random House.
Gendolla, G. H. E., & Wicklund, R. A. (2009). Self-focused attention, perspective-taking, and false consensus. Social
Psychology, 40(2), 66–72.
Goldberg, D. E., & Somerville, M. (2014). A Whole New Engineer. Douglas, MI, USA: ThreeJoy Associates, Inc.
Goleman, D. (1995). Emotional Intelligence. Bantam Books, 2006.–358 p.
Govern, J. M., & Marsch, L. A. (2001). Development and validation of the situational self-awareness scale. Consciousness and
Cognition, 10(3), 366–378. https://doi.org/10.1006/ccog.2001.0506
Harrington, R., & Loffredo, D. A. (2010). Insight, rumination, and self-reflection as predictors of well-being. The Journal of
Psychology, 145(1), 39–57.
Hartikainen, S., Rintala, H., Pylväs, L., & Nokelainen, P. (2019). The concept of active learning and the measurement of
learning outcomes: A review of research in engineering higher education. Education Sciences, 9(4), 9–12.
https://doi.org/10.3390/educsci9040276
Kamenov, K. (2013). Self-awareness and behaviour within social structures. Народностопански Архив, (4), 3–13.
Lattuca, L. R., Terenzini, P. T., & Volkwein, J. F. (2006). Engineering Change - A Study of the Impact of EC2000. In Executive
Summary. Retrieved from http://www.abet.org/wp-content/uploads/2015/04/EngineeringChange-executive-
summary.pdf
Mayer, J. D., Salovey, P., & Caruso, D. R. (2004). AUTHORS’RESPONSES:" A further consideration of the issues of emotional
intelligence". Psychological Inquiry, 15(3), 249–255.
Mohd-yusof, K., Arsat, M., Borhan, M. T., Graaff, E., & Kolmos, A. (2013). PBL Across Cultures. In 4th International Symposium
on Problem Based Learning. Aalborg, Denmark.
Morin, A. (2017). Toward a glossary of self-related terms. Frontiers in Psychology, 8, 280.
Murphy, J. (2007). Putting humor into practice. The ASHA Leader, 12(10), 55.
Nutt Williams, E. (2008). A psychotherapy researcher’s perspective on therapist self-awareness and self-focused attention
after a decade of research. Psychotherapy Research, 18(2), 139–146.
Nutt Williams, E., & Fauth, J. (2005). A psychotherapy process study of therapist in session self-awareness. Psychotherapy
Research, 15(4), 374–381.
Prince, M. (2004). Does Active Learning Work? A Review of the Research. Journal of Engineering Education, 93(3), 223–231.
Retrieved from https://www.proquest.com/docview/217960253?accountid=26624
Roehl, A. M. Y., Reddy, S. L., & Shannon, G. J. (2013). The Flipped Classroom : An Opportunity To Engage Millennial Students
Through Active Learning Strategies. 105(2).
Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83–
141
98.
Scaffidi Abbate, C., Boca, S., & Gendolla, G. H. E. (2016). Self-awareness, perspective-taking, and egocentrism. Self and
Identity, 15(4), 371–380.
Showry, M., & Manasa, K. V. L. (2014). Self-Awareness-Key to Effective Leadership. IUP Journal of Soft Skills, 8(1).
Silvia, P. J., & Duval, T. S. (2001). Objective self-awareness theory: Recent progress and enduring problems. Personality and
Social Psychology Review, 5(3), 230–241.
Smith, E. (2011). Teaching critical reflection. Teaching in Higher Education, 16(2), 211–223.
Sutton, A. (2016). Measuring the effects of self-awareness: Construction of the self-awareness outcomes questionnaire.
Europe’s Journal of Psychology, 12(4), 645.
Sutton, A., Williams, H. M., & Allinson, C. W. (2015). A longitudinal, mixed method evaluation of self-awareness training in
the workplace. European Journal of Training and Development.
Titrek, O., & Çelik, Ö. (2011). Relations between Self-Awareness and Transformational Leadership Skills of School Managers.
The New Educational Review, 23(1).
Whetten, D. A., & Cameron, K. S. (2016). Developing management skills,(9. izd). Michigan: University of Michigan.
142
Gaming for learning
Jesús Armengol 1, Pau Bofill2, Montse Farreras3
1
Group of Applied Optics and Image Processing (GOAPI), Optics and Optometry Department, UPC-Barcelona Tech.
2
Escola Tècnica Superior de Telecomunicació de Barcelona, STH-UPC-Barcelona Tech.
3
Escola Politècnica Superior d’Enginyeria de Vilanova i la Geltrú, UPC-Barcelona Tech.
Email: [email protected], [email protected], [email protected].
Abstract
In the subject of wave optics, we seek to involve students in their learning through active learning techniques. Students
participate in large group classes doing small activities, competition games, solving problems,... They are also very active
in the laboratory sessions (half the total time of the course) where they measure and analyze the contents developed at
the large sessions. The activities they play in the large group are based on gamification principles; students answer group
quizzes and get a ranking based on whether or not they get the right answer. We have set out to better develop the
possibilities of game-based learning by redesigning the simulation activities that students carry out based on the principles
of McGonigal1 so that they are games from which they cannot escape without learning. The simulation activities have been
included in an escape room where students play in groups and to advance in the game they must solve a riddle using the
knowledge developed in the subject. As McGonigal proposes, the escape room seeks to have an epic mission, a clear goal,
immediate feedback and another chance to prove it, and a positive social dimension.
Keywords: Active Learning; game-based learning, serious games, motivation.
1 Introduction
Gamification has entered strongly into many areas of training, training for industry 5, training for the financial
sector 6 , and of course university education (Call, 2021; Lopez-Fernandez, 2021). Game-based learning has
gained significant notoriety in recent years, accompanied by the development of computer tools that facilitate
the implementation of quizzes, competitions, simulations, WebQuests, escape rooms, ...
Both the gamification of educational activities (introducing points, badges, and leaderboards) and game-based
learning activities (where students learn while playing) increase students' motivation to complete learning
activities. However, how does this extra motivation help us as teachers to achieve the learning outcomes
scheduled in the course? Are there any gamified activities that are best suited to achieve some results or others
depending on the level of the goal set?
An interdisciplinary teaching innovation group at the UPC set out to work on game-based activities to achieve
deeper learning by taking advantage of the extra doses of motivation that students show when participating
in gamified activities.
2 The subject
Wave optics is a core subject and is taught in the second semester of the degree in optics and optometry at
the School of optics and optometry in the Universitat Politècnica de Catalunya. The first two semesters of the
degree form the selective phase and therefore, the subject is part of this phase. The selective phase is a set of
subjects that are assessed together and must be passed in a maximum period of twice their duration. In case
of not passing the selective phase, the student cannot continue the degree.
5
https://trainingindustry.com/articles/learning-technologies/game-based-learning-vs-gamification-do-you-know-the-
difference/
6
https://www.forbes.com/advisor/banking/how-gamification-can-help-you-meet-financial-goals/
143
The subject has 6 ECTS, which are transformed into 2 hours a week with the big group and 2 hours of laboratory
work for approximately 15 weeks. The second semester of the degree is characterized by its hardness, many
subjects with new content and many laboratory practice sessions. This fact, together with the fact that the
contents are related to the fundamental aspects of the discipline and that it is difficult for students to see their
application, leads us to the initial motivation for the subject being low.
In the normal period (2nd Q) 90 new students are enrolled. They are divided into two big groups (50 + 40) and
seven groups of laboratories (x15). Attendance is very dissimilar among big groups and very high among all
laboratory groups.
2.1 Contents
The subject has a syllabus that extends over the generation, propagation and detection of light. Themes are
covered by considering light as an electromagnetic wave. The properties of the waves and how they are
presented in the light are studied.
2.2 Methodology
Students have all the material they need to follow the course on the intranet. This material includes a detailed
study guide with the activities to be carried out to learn the contents of the subject. The homework consists
that the students have to write summaries of the contents to be treated in the subject from the reading of the
chosen material, try to solve small exercises, share the exercises with the classmates and give them feedback,
use some simulations to check the phenomena explained in class and solve some problems where the usual
topics of the topic appear.
In the course, we propose for the big group sessions to combine expository classes with informal cooperative
learning activities. Some of these activities are games (group quizzes where students get small rewards such
as winning a race, ...) that seek to keep students up to date.
Completion of activities is also listed on a leaderboard that motivates students to stay up to date.
2.3 The laboratory

In the laboratory sessions, students carry out guided experiments to check the most important laws explained
in the large group sessions.
In the laboratory, each group (3 people) has a set-up to do the experiments of the session and must be able
to deliver at the end of the session a dossier with the completed report of the work. The dossier should consist
of: a summary of the theory related to the experiment they will carry out, the numerical results, graphs, etc.,
the answers to small questions given in advance by the teacher and as a most important element, the
conclusions and final comments of what they worked on in the session.
The groups work cooperatively, they deliver a single consensual dossier for which all members are responsible,
exchange roles within the group (prepare reports alternately), conduct examinations individually, and are
rewarded if the individual performance of all group members is high… The handout of the experiments tries to
foster interaction within the group (discussion of outcome graphs, individual measurements and comparison
of results…). Each week, students receive feedback from the teacher on reports from the previous session
(Gibbs, 2005).
Laboratory sessions allow us to work on transversal competencies over relatively long periods. So, we spent
the first half of the semester working on the written expression. In these laboratory sessions, students should
write comments on the results (in the form of a graph) obtained explaining the observed behaviours. In the
second half, we work on the accuracy and precision of the measurements. Students develop techniques to try
to improve the accuracy of the measurements they make and to know the precision with which they make
these measurements.
The experience of the last courses is that the laboratory sessions are very positive and beneficial for the
performance of the students. The groups in the laboratory work very well and their overall performance
improves a lot.
144
2.4 Assessment
The assessment of the subject is divided equally between the big group sessions and the laboratory sessions,
as the time spent by the students in each one is the same.
The School of Optics and Optometry only reserves a time slot for each subject at the end of the semester to
take exams in the big groups. All other tests are carried out during the hours assigned to the subject. In these
spaces, we propose to the students four controls with a weight each of 10% of the final mark. There are two
that are small multiple-choice tests where they also have to justify the chosen option. We dedicate 30 minutes
to this. And two where in addition to the test they have to solve an exercise and justify the solution, this takes
one hour.
At the end of the course, in the slot reserved by the School for the final assessment, students can repeat any
of the tests they have taken during the semester. On this occasion, all the controls have a multi-response test
part and an exercise.
The activities carried out during the semester (summaries, solving exercises, simulations ...) are assessed by the
teacher at the end of the course if 80% of them have been completed on time. This assessment adds up to
10% of the final grade.
The most experimental part of the course is assessed with two laboratory exams (weighing 20% each) and a
laboratory report grade (10%). Laboratory exams are conducted during a lab session and students must
individually repeat a part of one of the experiments performed, write a small report with the results and answer
some questions. The grade of the laboratory reports (10%) is proposed by the teacher at the end of the
semester based on the reports delivered and the work done by the student in the laboratory sessions.
Figure 1.- Diagram of the assessment
We have tried to design an evaluation system that manages to engage the students in a continuous work from
the beginning of the subject and until the end, some of the elements taken into account are (Armengol, 2007):
▪ Give opportunities to students to pass the subject until the last day.
▪ Not to eject students from the course until the end.
▪ Do not permit a student to pass the subject until the end.
▪ Give them the appropriate workload.
▪ Keep students informed of their performance
▪ Engage them in the course (hear their opinions, take them into account,…)
▪ Explain why the course is active.
145
▪ …
3 Gaming
So far we have done gamification experiences like quizzes in class with electronic platforms, leaderboards of
the homework,… which have shown us that students have an extra dose of motivation to participate in these
activities.
The challenge of the new innovation project is to work on game-based activities in order to achieve deeper
learning.
Although the variety of experiences and theories on which the games are based are very diverse 7, we have set
out to better develop the possibilities of game-based learning by redesigning the simulation activities that
students carry out from the principles of McGonigal (2011) so that they are games from which they cannot
escape without learning. The activity we found most appropriate to achieve this goal was a sequential escape
room (Veldkamp, 2020). The simulation activities have been included in an escape room where students play
in groups and to advance in the game they have to solve an enigma using the knowledge developed in the
subject. As McGonigal proposes, the escape room seeks to have an epic mission, a clear goal, immediate
feedback and another opportunity to prove it, and a positive social dimension.
3.1 The escape room

The escape room is a challenge: Students get on the train to come to the School (located in Terrassa, a city
near Barcelona) and just as the train starts moving they hear a message from the driver that today the train will
not stop in our city!!! "Just today my teammates have the most important commitment of the whole degree!!!".
Students must advance on the train wagons until they reach the locomotive where the driver is to inform him
that he must stop the train so that his classmates can come to school to carry out the most important activity
of the year (Epic Mission). To cross each wagon (clear goal) the reviser asks them some questions that they
must solve. In each car, the questions are on one of the topics of the subject and they can run some simulations
to solve the problem. The questions you ask them are small exercises with multiple-choice options. At the end
of all the questions in the wagon, they receive feedback on the result (immediate feedback) and if they have
not answered all the questions correctly, they can try again (another chance to prove it). The game is played
as a team (social dimension) and cooperation between team members is encouraged. They have one hour to
complete the game, which is the train journey time.
3.2 The evaluation of the experience.

To evaluate the experience, we have added some questions to the traditional assessment surveys of each
subject. Thus, having all the members of the team who participate in different subjects with different studies
run the same questions, we can easily make comparisons.
The questions we propose to ask are:
Statement Strongly
Disagree Neutral Agree
Strongly
disagree agree
The game has engaged me to practice more 1 2 3 4 5
I like the fact that I could solve the problems at my own pace 1 2 3 4 5
Mention three positive aspects that you have enjoyed
Mention three shortcomings or things that you would improve
7
Valero, Miguel. https://personals.ac.upc.edu/miguel/materiales/docencia/articulos/Gamificacion.pdf
146
4 Preliminary results
Preliminary results of the survey are shown in Figure 2.
The game has been a motivation to start working The game has engaged me to practice more
on the subject
7
7 6
6 5
5
4
4
3
3
2 2
1 1
0 0
Strongly Disagree (1) Neutral (2) Agree (3) Strongly agree Strongly Disagree (1) Neutral (2) Agree (3) Strongly agree
disagree (0) (4) disagree (0) (4)
I have enjoyed learning through play The game has helped me to acquire a deeper
level of understanding
9
8 6
7
5
6
5 4
4 3
3
2
2
1 1
0 0
Strongly Disagree (1) Neutral (2) Agree (3) Strongly agree Strongly Disagree (1) Neutral (2) Agree (3) Strongly agree
disagree (0) (4) disagree (0) (4)
I like the fact that I could solve the problems at

my own pace
12
10
8
6
4
2
0
Strongly Disagree (1) Neutral (2) Agree (3) Strongly agree
disagree (0) (4)
Figure 2.- Results of the survey. The vertical axis indicates the number of groups (each group having 3-4 students).
Access to the survey presented in the previous chapter was given once finished the escape room. Groups, the
same ones who had participated in the game, answered it. There is only one answer per group. Each group
consisted of between 3 and 4 students.
The processed results and discussion of the individual and joint surveys will be presented at the conference.
Acknowledgement
d’innovació docent 2021 (Acord CG/2021/02/34, de 9 d'abril de 2021) and by the Escola Tècnica Superior
d'Enginyeria de Telecomunicació de Barcelona (upc telecos-UPC-Tech).
147
5 References
Armengol, Jesús, (2007). Continuous assessment, pros and cons (or the art to combine summative and formative assessment).
Active Learning in Engineering Education Workshop, 99-104.
Call, T., Fox, E., & Sprint, G. (2021). Gamifying Software Engineering Tools to Motivate Computer Science Students to Start and
Finish Programming Assignments Earlier. IEEE Transactions on Education, 64(4), 423–431.
https://doi.org/10.1109/TE.2021.3069945
McGonigal, Jane (2011). The reality is broken: Why Games Make Us Better and How They Can Change the World. The Penguin
Press, New York.
Gibbs G., Simpson C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher
Education, Issue 1, 3-31.
Lopez-Fernandez, D., Gordillo, A., Alarcon, P. P., & Tovar, E. (2021). Comparing Traditional Teaching and Game-Based
Learning Using Teacher-Authored Games on Computer Science Education. IEEE Transactions on Education, 64(4),
367–373. https://doi.org/10.1109/TE.2021.3057849
Veldkamp, Alice, et al. (2020). Escape education: A systematic review on escape rooms in education. Educational Research
Review 31, 100364. https://doi.org/10.1016/j.edurev.2020.100364
148
Pair Teaching in Computer Graphics
Luciano Soares, Fabio Orfali
Insper, São Paulo, Brazil
Abstract
Computer Graphics is a course commonly found on Computer Science and Computer Engineering undergraduate
programs. There are some convenient points on teaching computer graphics in these programs: Computer-Generated
Imagery in movies and videogames are relevant examples of applied engineering and science that students experience in
their lives, and this can be a potential motivation factor for students to get engaged on learning; furthermore, Computer
Graphics relies on several fundamental and complex mathematical concepts, that can also be applied in other science areas,
being an opportune moment to cover and practice mathematical learning goals, all connected with a base on computer
science. Learning Computer Graphics depends on developing and integrating skills on computer science and mathematics,
therefore, a learning plan must cover both topics. However, it must be considered that students need to relate these topics,
and this is the trick, since ideally professors need the knowledge and pedagogy in both areas. The strategy presented in
this document was based on two professors, one mathematician and one computer engineering, planning, and teaching
an elective undergraduate course on Computer Graphics. The course was based on active learning strategies by design,
heavily focused on Project Based Learning through three projects starting from scratch, each week learning and
implementing new features on the projects. Students must program in regular programming languages like Python, C++
and Javascript, and build up their competence in mathematics from basic math concepts. As a result, students were able to
develop a scanline renderer, and a ray-tracing renderer without any graphical Application Programming Interface and finally
a 3D web application based on ThreeJS. Although students had a perception of a demanding course, they were engaged
on proposed activities during all semester, and were able to implement all important features on projects, often above
expectation, showing evidence that learning objectives were achieved.
Keywords: Pair Teaching; Computer Graphics Teaching; Mathematics Teaching; Project Based Learning.
1 Introduction
The foundations of Computer Graphics rely heavily on mathematical knowledge. Therefore, the proposal of a
Computer Graphics course that is not limited to the mere application of ready-made programming codes
requires a plan on how Mathematics will be incorporated into classes. This plan becomes even more important
when considering Computer Graphics as an elective course at the end of an engineering program, as
mathematics is too much concentrated in the beginning of the program, usually in the first semesters. This is
a scenario in which students show less readiness for the application of mathematical concepts in the context
of the discipline. As a result, many students may have difficulties with basic mathematical content, such as
vector operations.
However, the development of a didactic sequence that resumes the mathematical knowledge required by
Computer Graphics may represent a great challenge. As a computer engineer or a computer scientist, this
professor may not master the content and/or pedagogy of the mathematical area. Furthermore, a clear
understanding of the mathematics track in the undergraduate program, will allow course professors to identify
topics of mathematics that were covered in previous semesters of the course, having a better view on what
should be reviewed and what could be an entirely new topic for students.
In order to deal with this problem a pair teaching strategy (Andersson & Lars, 2006; Ott & Meek, 2019; Burden,
Heldal & Adawi, 2012; Zehetmeier, Bottcher & Brüggemann, 2018) was adopted: two professors of different
areas working together throughout the course. It is important to point out that this is not a course split in two
parts having professors teaching their topics, professors have discussed and defined learning goals, planned
classes and activities, and evaluated projects all together. At first this strategy may appear less efficient since
there are two professors at the same time in class, but improvement on learning was very evident.
149
This paper reports the experience of pair teaching in a Computer Graphics course offered as an elective in an
Engineering program. The course, based on an earlier version lectured by only a Computer Engineering
professor, was taught by two professors: a computer engineer; and a mathematician. The remainder of the
paper explains: the main pedagogical strategies in chapter 2, selected results achieved on conducted course in
chapter 3, and final conclusions on chapter 4.
2 Strategy
Working in pairs for teaching can bring several benefits, but a clear alignment must be established between
lecturers to create coherent activities in classes. The starting point for achieving this alignment was the plan
and material produced in the previous version of the course, to ensure that developing computer graphics
skills and course goals were maintained and improved as possible. Then, the mathematical content required
for each course phase was identified. The organisation of mathematical knowledge was defined at the
beginning of the program, to define the most appropriate depth, order, and approach during the course.
Another important constraint is that courses should follow an active learning pedagogy heavily based on
project-based learning.
Since this course is taught at the end of the engineering program, bringing back fundamental mathematical
concepts could be very challenging. In this case, the profile of students enrolled in the course was necessary
to better design activities in the mathematical area. Usually, students are more interested in working on more
advanced projects, and very often they rely on tools or applications that hide the complexity of development,
in this case on mathematics and computer science, but since the beginning students were informed that this
course would dig in the fundamentals of mathematics and low level programming.
Thus, weekly demands were defined in the projects, most of which involving some type of mathematical
knowledge and applied computer science. During classes, students worked in pairs or trios, guided by handouts
that addressed the content required in that week's delivery. In this way, the progress concretely observed in
their projects was the main motivation for studying the mathematics involved.
An example of this strategy was the study of rotations by quaternions. The representation of a three-
dimensional object rotation by quaternions is a technique commonly used in computer graphics, since it avoids
known problems of Euler's angles (Pletinckx, 1989). Its mathematical justification is based on the understanding
of quaternions as a 4-dimensional space that contains complex numbers. More than just applying formulas to
implement quaternion rotations in their projects, the proposed activities allowed students to have a real
understanding of what they were doing in their programming code.
In addition to the weekly deliveries related to the projects, revisions were scheduled at strategic points in the
course, which supported students to understand the big picture related to the procedures previously studied.
These revisions were usually associated with the development of a complex case where students could check
their understanding.
Three main projects were distributed during the semester that were further subdivided. The partial project
deliveries have generally lasted a week, so that students could build their project and see some progress little
by little.
2.1 Scanline Renderer

The first project was a conventional scanline renderer (Foley, van Dam, Feiner & Hughes, 1990). Students would
initially implement an algorithm to identify 2D triangles and generate an image, but soon afterwards they
would work in 3D, calculating the perspective view of a scene, calculating the lighting and traditional rendering
resources such as anti-aliasing. Students had to develop from scratch using Python as a programming
language. A skeleton code was provided by a git repository for them to follow the development and render
some known sample scenes. The scanline render was designed to read X3D files (http://www.web3d.org/), what
is a convenient format since it is open, well documented, and in text format, and all geometry can be
parametrized.
150
The expected deliverables were:
● Render 2D colourful points, lines, and triangles
● Calculate perspective projection
● Render 3D triangles and triangle strips
● Rendering using super sampling
● Scene graph implementation
● Interpolate colours across triangles
● Texture map triangles
● Z-Buffer implementation
● Calculate illumination over triangles
● Render an animated scene
The deliverables were sometimes subdivided, and students were requested to implement and test their code
with the examples. Balancing the level of complexity for the week's task was not simple since previous
experience did not take into account the extra care with mathematical teaching.
2.2 Ray Tracer

The second project was a ray tracing renderer, for which Peter Shirley (2020) books were used as a basis. Initially
students should implement the basics of the algorithm, following the book. The suggested language for this
renderer was C++. In the sequence, challenges were proposed for the implemented ray tracing algorithms that
students should fulfil. Ray tracing algorithms have a different strategy from scanline algorithms, but some
mathematical concepts are reused, and some entirely new mathematical concepts were brought on this project.
One of the biggest differences from the first project, was the fact that ray tracing algorithms expect implicit
representation of geometries, for instance, a sphere centred in the origin is represented by: x 2 + y2 + z2 = r2.
This creates several new challenges for students that must adapt their mindset for this new project proposal.
The main deliverables were:
● Implement a new implicit geometry (like cylinders, torus, among others)
● Develop a texture based on some noise function
● Texture map a 3D object
The time for students to implement the basics of the renderer were 4 classes (classes are two hours long),
although the algorithm documentation was very complete, students should take care of small details in order
to compile the code successfully.
2.3 3D Web Project

The third and final project was the development of a theme free 3D application for web browsers. Students
should implement a 3D scene using the ThreeJS 3D web API (https://threejs.org/) which is convenient as it does
not depend on any specific platform, like Windows or MacOS. At that moment students were very creative,
with curious scenes created.
Students were free to choose direction, but they must discuss with professors next steps, defining an adequate
level of complexity for each intermediary delivery. This project management strategy was based on agile
methodologies (Fowler & Highsmith, 2001), where professors would act as Product Owners.
3 Results
The results presented are based on the comparison to the previous version of the Computer Graphics course.
The last version, in which pair teaching was adopted, allowed a greater advance both in programming and in
mathematical learning evidenced by the quality of project deliverables. In the previous version, students did
not progress as fast as expected because they had to review the mathematical concept by themselves without
151
proper support. For example, there was no time to discuss rotations with quaternions in the first version, as
students were struggling with vector operations in describing rotations. And despite the denser content in the
last version, students were able to fulfil most of the expected learning goals and tasks.
Another important point is how difficult it was for students in the previous version to connect what they did
learn in math and how to use it on a real computer graphics application. Apparently, this connection was more
natural for students that took the course with pair teaching.
These courses (last and previous versions) were taught at Insper (Soares, Achurra, & Orfali, 2016), and the
educational institution collects blind feedback from students. Both courses were well evaluated by the students.
Just, in the last version of the course, students pointed out that the course was more demanding than a regular
elective, but this did not affect their motivation, and learning was enhanced due new activities planned.
There were some complaints about specific maths assignments charged at the beginning of the semester.
Based on feedback, these deliverables have been discontinued, and content has been incorporated in project
deliverables.
For the first part of the course, they were challenged to acquire all the expertise to develop a scanline rendering.
Figure 1 and Figure 2 show some results from student’s projects.
(a) (b) (c)
Figure 1. 2D render of points(a), lines(b) and triangles(c).
(a) (b) (c)
Figure 2. 3D render of triangle strips (a), texture maps (b) and scene graph (c).
Although the requested project was the same for all students, it is possible to see small details different from
project submission to submission. One noticeable difference between each project submission was the time
necessary to render 3D scenes. Rendering time was not a point for assessment, but some students did work
on improving the performance of their algorithm, and at the end some complex scenes took a few seconds to
render in some projects and in other projects took some minutes. This was also important for students to
understand that rendering is a complicated process, and depending on the way you implement, algorithm run
time and space requirements can exponentially grow as the input size grows (Chivers & Sleightholme, 2015).
152
The second project was based on a well-documented ray tracer (Shirley, 2020). Students had to implement the
traditional ray tracer and after implement some kind of improvement for the system. Some of the
implementation's results are presented in Figure 3.
(a) (b) (c) (d)
Figure 3. Ray tracing render of cylinder (a), torus (b), drop (c) and spiral (d).
Shirley’s book (Shirley, 2020) implementation shows only how to render a sphere, this is a convenient geometry
for learning, but students were challenged with other geometries. In the end this was a complex task for
students since it is possible to see some problems in most of the delivered results. We were expecting a cylinder
in Figure 3 (a), although it is possible to recognize it, the calculation for surface normal is not correct, if normal
were correctly calculated it would be possible to identify the interior part. Figure 3 (b), shows a torus that has
a very complex equation to solve; students did an excellent job, but it is possible to visually confirm that
normals were not well calculated. Figure 3 (c), shows a 3D drop, and again normal were not perfect and there
is a small detail failing some render lines at the top of the drop. Finally, in Figure 3 (d) there is a spiral, the most
complex geometry, students were able to correctly calculate normals, but they have to adapt the spiral
equation.
As a second part of the second project, students were challenged on developing different textures, using noise
functions, in particular Perlin Noise (Perlin, 1985), to create different materials. Figure 4 shows some results
implemented by the students.
(a) (b) (c)
Figure 4. Materials created for a ray tracer using Perlin noise function among other techniques.
Students were very creative in creating new textures exploring mathematical functions. Professor’s perception
was that students enjoyed developing this creative work.
At the end, students had a complete free 3D project. This was an opportunity for students to create a more
complex project and publish on the web for anyone to see. Students learned how to use a scene graph, in an
advanced API (Application Programming Interface), in the case ThreeJS. Figure 5 shows some results from
student’s projects.
153
Figure 5. Sample images of theme free computer graphics web project.
There were several deliverables for the students during the semester, and students were a little concerned
about the complexity of the topics in the course, and one student decided to leave the course. The first
deliverables had an extremely high quality, and students were giving their best to accomplish the proposed
challenges as best as they could. During the entire course topics had about the same level of complexity, but
students start to burnout, and the deliverables were not so complete as used to be in the first part of the
course. As professors start to notice that students were not delivering project with the same quality, they
decided to slow down a little. In a future version of the course, professors should better balance how difficult
are the projects and better track if students are burning out.
3.1 Difference between course versions

The most noticeable difficulty some students had in previous course versions was in vector operations. The
course was not advancing as expected since skills in geometry transformations were necessary for the course
and these are based on vector operations. In previous versions of the course this problem was tackled in a
corrective way, as students started to struggle with vector operation, support classes were created. In the last
course version, with two professors, vector and other mathematical topics were treated in a preventive way,
but instead of a pure revision, students were oriented on how to learn the topic, with general exercises,
readings, videos and even lectures to support students for their projects.
At the end, students could deliver projects with more details and quality, for instance in previous versions, the
fourth part of the first project was not even published for students, since some support classes were necessary,
and the original plan was delayed. On the second project in previous versions, only basic resources of the ray
tracer were developed and in the last version not only the basic resources were implemented as some complex
routines were also incorporated.
4 Conclusion
Having two professors on this computer graphics course, one mathematician and one computer engineer,
allowed students to better discuss and learn several details in mathematics and computer science that
otherwise would not be so practical. Classes were planned in an active way where students could discuss with
professors graphical and mathematical techniques, also having studio time (that is time slots for developing
their project in class) allocated for regular activities and project development.
There was a clear improvement on projects quality comparing the pair teaching version with previous version
of the course with a single professor. In the pair teaching edition, students were able to implement more
graphical techniques, and the quality is also noticeable. In the first edition, small mistakes on algorithms and
math routines lead to imagens with some strange artifacts, what was not often noticeable in the pair teaching
version.
The course has yet a lot of opportunities for improvement: for instance, in the first project a better sequence
of topics can be used, in the second project the high dependency of an external book is not creating enough
154
opportunities for learning, and additional motivation for the final project can lead to better projects, what by
consequence should improve learning.
5 References
Andersson, R. & Lars, B. (2006). Pair Teaching – an eXtreme Teaching Practice. In Pedagogiska Inspirationskonferensen. issn:
2003-3761
Burden, H., Heldal, R. & Adawi, T. (2012). Pair Lecturing to Enhance Reflective Practice and Teacher Development, in
Proceedings of Conference on Teaching and Learning.
Chivers, I. & Sleightholme, J. (2015). An Introduction to Algorithms and the Big O Notation. In: Introduction to Programming
with Fortran. Springer, Cham. doi: 10.1007/978-3-319-17701-4_23
Foley, J. D., van Dam, A., Feiner, S. & Hughes, J. (1990). Computer Graphics: Principles and Practice. Reading, MA: Addison-
Wesley. ISBN: 978-0-201-12110-0
Fowler, M., & Highsmith, J. (2001). The agile manifesto. Software development, 9(8), 28-35.
Ott, C. & Meek, N. (2019). Pair Teaching in Action. In Proceedings of the Twenty-First Australasian Computing Education
Conference (ACE '19). Association for Computing Machinery, New York, NY, USA, 87–95. doi:
10.1145/3286960.3286971
Perlin, K. (1985). An Image Synthesizer. SIGGRAPH Computer Graphics 19 (0097-8930): 287-296. doi:10.1145/325165.325247
Pletinckx, D. (1989). Quaternion calculus as a basic tool in computer graphics. The Visual Computer 5, 2–13 doi:
10.1007/BF01901476
Shirley, P. (2020). Ray Tracing in One Weekend. url: https://raytracing.github.io/
Soares, L. P., Achurra, P. & Orfali, F. (2016). A hands-on approach for an integrated engineering education. Proceedings of
the PAEE/ALE 2016, p. 294-302.
Zehetmeier, D., Bottcher, A. & Brüggemann, A. (2018). Designing Lectures as a Team and Teaching in Pairs. In proceedings
of the Fourth International Conference on Higher Education Advances. doi: 10.4995/HEAD18.2018.8103.
155
Application of Project-Based Learning and Lean tools in a
machining area of a welding industry
Mariana de Azevedo Mello¹, Júlia Rocha Thomaz Mattoso Salgado¹, Marcelle Caldas Vicente¹, Fernando Augusto
Ullmann Töbe2, Stephany Rie Yamamoto Gushiken³, Anibal Alberto Vilcapoma Ignácio 4
1
Graduation Student in Production Engineering, School of Engineering of Petrópolis, Fluminense Federal University, Rio de Janeiro, Brazil
2
Master of Science in Production Engineering, School of Engineering of Petrópolis, Fluminense Federal University, Rio de Janeiro, Brazil
3
Graduation in Production Engineering, School of Engineering of Petrópolis, Fluminense Federal University, Rio de Janeiro, Brazil
4
Doctor of Science in Production Engineering, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil. Professor at Fluminense Federal
University, Rio de Janeiro, Brazil.

[email protected], [email protected]
Abstract
This paper has the objective of applying Project-Based Learning to propose improvements and measure the movement
times in the machining area, using the Value Stream Mapping and Spaghetti Diagram as tools. This was a study developed
by Graduation Students in Production Engineering of the Fluminense Federal University in partnership with a welding
industry. The methodology can be classified as exploratory and descriptive, applying the bibliographic method to
understand concepts related to machining, lean manufacturing, Value Stream Mapping (VSM) and Spaghetti Diagram, and
it is also a case study, using the machining area of a welding industry as an analysis unit. The practices of project
management recommended by the PMBOK Guide from Project Management Institute (PMI) were used, as well as the
Project-Based Learning (PBL) methodology to drive the study in terms of research and application in the analysis unit. The
activities of the machining area were mapped through the Value Stream Mapping and Spaghetti Diagram in order to
identify the non-productive movements and time. Considering the results from the bibliographic search and the process
mapping, alterations were proposed in the layout and order of activities and also, removed the ones that do not add value
to the product. The improvements suggested to the welding industry were implemented and resulted in a reduction of
33h, equivalent to around EUR 5.000.
Keywords: Project-Based Learning; Lean Manufacturing; Value Stream Mapping; Machining.
1 Introduction
When a project is presented to students simply “because they will need the knowledge in the future” or
“because they will need to understand it for the next subjects”, there is no natural engagement and motivation
required for students to feel challenged to solve the problems of a given situation (LARMER et al., 2010). In this
way, an approach which offers the investigation of authentic and real problems is needed.
Project-Based Learning (PBL) is a student-centered, teacher-facilitated learning methodology where students
search for knowledge by asking questions. Such questions guide the research, which is supervised by the
professor (BELL, 2010). In addition, PBL can be considered as a model that organizes learning around projects,
(THOMAS, 2000).
In this context, the paper was developed by Graduation Students in Production Engineering from the
Fluminense Federal University (UFF) of Rio de Janeiro, Brazil, with a company in the welding industry.
Considering that there is downtime and non-productive hours in the machining sector, the students defined
the central question as “How to reduce downtime in a machining sector through PBL using Lean Manufacturing
as the main driver?” The students defined as specific questions “How does the machining process work?” and
“Which is the most appropriate Lean tool for reducing downtime?”. Both questions aimed to answer the central
question.
This paper has the following structure: first, the bibliographic research will be addressed. Next, the
methodological description for the academic project is presented and finally, the results and conclusion.
156
2 Bibliographic research
This topic will present the results of the bibliographic research done to provide theoretical basis to all
improvements suggested to the welding company.
2.1 Machining
According to Davim (2008), “machining is the broad term used to describe the removal of material from a
workpiece”. Also, it is called an operation that gives the part shape, dimensions and surface finishes by
removing the material in the form of shavings.
Machining has two types of processes, conventional and unconventional. In conventional processes, cutting
operations use mechanical energy to remove material from the part. In unconventional machining processes,
cutting operations use other forms of energy, such as thermoelectric power (MACHADO et al, 2015).
With the technological advances in machine tools, there was the emergence of automatic machines, and soon
after, of Computer Numerical Control (CNC) machines. According to Silva (2014), CNC lathes transmit
information in the form of movement by activating and deactivating the machine tool's features according to
the sequence that has been programmed without depending on the assistance of an operator.
The question “How does the machining process work?” is answered through this topic, at the same time it
helps the students understand the process in the analysis unit that relies on both conventional and CNC
machining. As the central issue is focused on reducing downtime in this area, it is important to understand how
the steps flow and which activities are essential and which are not.
2.2 Lean Manufacturing

According to Touriki et al (2021), Lean Manufacturing can be seen as “an amalgamated sociotechnical method
aspiring to reduce waste by minimizing the manufacturing variations”. In Lean view, waste can be defined as
anything that does not add value to the product (GUPTA & JAIN, 2013).
The 3 M's (Muda, Mura and Muri) are Japanese terms related to different types of waste. The first, Muda, refers
to uselessness and has 8 types, according to Womack and Jones (2013): excess production, waiting,
unnecessary transport or movement, process excess or incorrect process, excess inventory, unnecessary
movement, defects and misuse of personnel. According to Santos et al. (2011), Mura is waste related to lack of
balance and Muri is waste related to overload.
Lean Manufacturing has several tools, which are applied in companies to reduce waste. Matt et al. (2013, p.
423) developed the matrix presented in Figure 01.
Figure 1. Types of Lean Manufacturing tools related to the organization site. Source: Matt et al. (2013, p. 423).
157
Considering that the company used as a case study is characterized as medium-size and the specific question
“Which is the most appropriate Lean tool for reducing downtime?”, the tools listed in Material flow and layout
type were analysed. Value Stream Mapping was selected as the most appropriate Lean tool for reducing
downtime because is conceptualized in a critical analysis of material and information flows along the value
chain, which makes it possible to highlight waste and opportunities for elimination thereof (ROTHER & SHOOK,
2003).
Typically, the Spaghetti Diagram is used in conjunction with Value Stream Mapping, due to the fact that aim
of tracing a path taken by materials or information in a specific layout (Tapping & Shuker, 2010). In this way,
the diagram allows the visualization of displacements and unnecessary efforts, showing and mapping the waste
of movement and transport. Therefore, both VSM and Spaghetti Diagram were used.
3 Methodology
In the following topics, the research classification, analysis unit, procedures for both collecting data and
bibliographic research as well as the PBL approach will be presented.
3.1 Research classification

The research can be classified as exploratory and descriptive, providing greater familiarity with the problem to
make it more explicit and describing and establishing relationships between variables, respectively (GIL, 2002).
Also, this work is a case study, which is the most appropriate strategy when "how" and "why" questions are
defined about a set of events and something the researcher has little or no control over (YIN, 2015).
3.2 Analysis unit

The analysis unit is a German multinational in Petrópolis - RJ characterized by its productive activity focused
on a cutting and welding portfolio. The company produces welding torches, air or water cooled, for use in
manual, semi, or fully automated operations. The focus of this paper was in the Machining Sector, which is
responsible for manufacturing components used in assembling of welding torches.
3.3 Procedures for collecting data

The students used three main sources of evidence. First, interviews with Machining employees to understand
the process flow and help to answer the question “How does the machining process work?”. Also, interviews
with the Managing Director and Industrial Manager to guide the students in prioritizing solutions.
Second, company documents and databases such as layout drawings and Work Orders, both to comprehend
the process flow and answer the central question. And finally, observations made by the students during
technical visits to the organization. These visits aimed to time each activity of the process flow for VSM.
3.4 Procedures for bibliographic research

The students divided the research into five steps: (i) searching literature on platforms through key words; (ii)
organizing the articles by chronological order of citations, alphabetical order of authors and article coverage;
(iii) selecting and prioritizing by choosing the most cited authors in the theme; (iv) reading and emphasizing
the most relevant parts; and finally (v) writing, showing how the authors studied correlated with the topic of
the research, in this way not simply summarizing their work. (TURRIONI & MELLO, 2012).
3.5 Stages for applying the PBL approach

The practices of project management recommended by the PMBOK Guide from Project Management Institute
(PMI) were used for applying the PBL approach and its stages are described in the following topics.
3.5.1 1st stage – Definition of problem-situation and elaboration of the Project Opening Term
The problem-situation was defined together with the company and the questions were defined in conjunction
with the professor to guide their research. According to Larmer et al (2010), a good guiding question captures
the heart of the project in clear language and is related to the content that the teacher wants the student to
158
learn at that moment. As a deliverable of this stage, the Project Opening Term was elaborated containing the
justifications, objective, benefits, project leader and team, sponsor as well as estimated deadline.
3.5.2 2nd stage – Elaboration of the Project Management Plan

The students prepared the Project Management Plan, which is contained the number of visits in the analysis
unit and deadlines to answer the questions. For Thomas (2000), a PBL project should not be carried out under
controlled conditions, in laboratories, limited by professors or by a script. To be considered a PBL project, there
must not be a pre-determined product nor a pre-defined path to follow, students spend more time without
professor supervision, rather having autonomy to seek the necessary knowledge and skills. Therefore, the
action plan should be defined only as a support for the students during the project.
3.5.3 3rd and 4th stages – Project execution and control

The execution was done with the professor’s support, being necessary for students to acquire the required
knowledge and understanding to define possible solutions and evaluate possibilities according to the literature
and the context of the project. These stages aimed to respond the central question through the bibliographic
research, interviews, company documents and database as well as visits to the organization. Also, it aimed to
elaborate the VSM and Spaghetti Diagram.
3.5.4 5th stage – Project ending

In the Ending stage, the project's product is publicly presented not only to the professor, but also to a relevant
audience – in this case, the company stakeholders. In this way, the meaning of the entire project-based learning
process is reinforced (LARMER et al., 2010).
4 Results
Next, the results of the activities described in Methodology are presented. This topic was structured in two
parts: analysis of the problem situation and proposals for improvement.
4.1 Analysis of the problem-situation

The scope of this study is the processes performed before and after machining a part and not the
manufacturing process itself. The students divided the problem-situation into three processes and analysed its
current scenario. The bibliographic research, interviews and questions guided all the improvements proposed
by the students.
4.1.1 Current scenario – Value Steam Mapping of WIP shelf process

Work in-process (WIP) parts correspond to assembled items that still need to be machined. The activities in
the process flow are represented by boxes in VSM as shown in Figure 2. In VSMs, all the cycle times (CT) were
timed by the students during visits to the organization and the order of activities was collected through the
interviews.
Figure 2. VSM of WIP shelf process – current scenario.
159
According to Figure 2, the last activity highlighted in red means that there is an opportunity for improvement.
In current scenario, the employee takes the parts to the finished products shelf outside the Machining sector.
The students proposed to place this shelf within the sector to reduce movement time.
4.1.2 Current scenario – Value Steam Mapping of raw material shelf process
The raw materials used in the Machining sector are brass or copper bars that are stored inside the Warehouse.
Focusing only on the highlighted activities in Figure 3, after checking if it’s a WIP or raw material Production
Order (PO), the employee waits for the PO booking in the Warehouse. The students proposed this booking
process to be after finishing the PO. And the last activity was covered in the previous topic.
Figure 3. VSM of raw material shelf – current scenario.

Removing the waiting for the PO booking, the students followed the Lean Manufacturing most important
principle of reducing waste and consequently, downtime. It is important to emphasize how valuable is the PBL
projects, due to the fact that the students are able to apply the knowledge learned during bibliographic
research.
4.1.3 Current scenario – Value Steam Mapping of exchange of shavings barrels process
Shavings are produced during the machining and in this case, shavings coming from the Computer Numerical
Control (CNC) Turning Center will be analysed. The students proposed removing the first two activities
highlighted in red, because the pallet trucks and the barrel area should be within the Machining sector,
avoiding movement time between areas.
Figure 4. VSM of exchange of shavings barrels – current scenario.

According to Figure 4, the activities of placing the full barrel and pallet truck in the appropriate area should be
done after restarting the machining process. Then, the parts would continue being manufactured while the
employee does these activities.
160
4.1.4 Current scenario – Spaghetti Diagram
The Spaghetti Diagram was built to analyse the movement and trajectory of employees in the Machining sector
when they need to pick up a tool or instrument that helps them in machining parts. In it, the coloured lines
represent the trajectory of the employees starting from each machine and going to each cabinet. These
movements can be seen in Figure 5.
Figure 5. Spaghetti Diagram – current scenario.

It was identified that the employees needed to move around a lot to find the instruments and tools that were
scattered around the cabinets in the Machining sector. Therefore, it was proposed creating a tooling and
instrumentation room. With this, the employees’ trajectories would start from their machines to a specific place,
not needing to go around the entire sector looking for instruments and tools.
4.2 Improvements proposal

After analysing the current scenarios of all processes selected for this study, improvements were proposed to
the welding company as previously showed. This topic is divided into two parts: compare the results from both
current and future VSMs and present the Spaghetti Diagram after the tooling and instrumentation room.
4.2.1 Future scenario – VSM of all processes

From the comparison of data from the Current and Future Value Stream Maps, it was possible to quantify the
gains obtained by the proposed improvements suggested for the sector. The process efficiency calculation is
obtained by dividing the time that adds value to the process by the lead time. Therefore, the following values
shown in Table 1 were obtained.
Table 1. Comparison of VSM results – current and future scenarios.
Process Criteria Current Future Percentage
Cycle time 637,93 637,93 0,00%
Value added time 600,00 600,00 0,00%
WIP shelf No added value time 98,51 84,42 14,30%
Lead time 698,51 684,42 2,02%
Process cycle efficiency (%) 86% 88% 2%
Cycle time 698,47 637,69 8,70%
Value added time 600,00 600,00 0,00%

Raw material
No added value time 194,40 88,83 54,31%
shelf
Lead time 794,40 688,83 13,29%
Cycle time 147,69 121,02 18,0%
161
Exchange of Value added time 60,00 60,00 0,00%
shavings
No added value time 250,41 93,70 62,58%
barrels
Lead time 310,41 153,70 50,48%
All the results presented on Table 1 are regarding one execution of each process. Together with the Machining
Supervisor during the interview and according to the company’s database, it was identified how many times
each process is done weekly and with this, the annual gain in hours was calculated, which can be used by the
employee in other activities that add value to the organization.
Table 2. Annual gain in hours.
Exchange of
Criteria / Process WIP shelf Raw material shelf Total
shavings barrels
Estimate of process execution / week 20 15 4 39
Estimate of process execution / year 960 720 192 1.872
Annual gain (h) 3,76 h 21,12 h 8,36 h 33,24 h
Value of man-hour (EUR) EUR 151,00 EUR 151,00 EUR 151,00 EUR 151,00
Reduction in costs (EUR) EUR 567,76 EUR 3.189,12 EUR 1.262,36 EUR 5.019,24
As it can be seen on Table 2, all the improvements would result in a reduction of 33,24h annually, which is
equivalent to a cost saving of EUR 5.019,24. These hours can be invested in activities that add value to the
product, considering that all proposals were implemented.
4.2.2 Spaghetti Diagram after improvements

After implementing the tooling and instrumentation room and aiming better visualize the movement flow in
the new layout model, a future Spaghetti Diagram was prepared.
Figure 6. Spaghetti Diagram – future scenario.

Comparing both Figures 5 and 6, the main reason to significantly reduce the movements inside the Machining
sector is the creation of a tooling and instrumentation room. The green highlights are the layout improvements.
5 Conclusion
This paper aimed to conduct a project through PBL by applying the Value Stream Mapping and the Spaghetti
Diagram in the machining process, having as central question “How to reduce downtime in a Machining sector
through PBL using Lean Manufacturing as the main driver?” The students were oriented by the professor to
define specific questions to help answering the central one: “How does the machining process work?” and
“Which is the most appropriate Lean tool for reducing downtime?”.
162
The PBL methodology divided the project into five stages, where the first one defined the problem-situation
with the analysis unit and helped the students to prepare the questions that would guide their learning process.
In the second stage, the students planned the activities within the organization and bibliographic research in
terms of deadlines.
In the third and fourth stages, they had to answer the questions. For “How does the machining process work?”,
the students learned through the bibliographic research, interviews with the Machining employees and visiting
the company. And for “Which is the most appropriate Lean tool for reducing downtime?”, by searching in
literature and interviewing the Managing Director and Industrial Manager, the Value Stream Mapping and
Spaghetti Diagram were selected as the most appropriate Lean tools for reducing downtime.
As a result of elaborating the VSM of each process, it became possible to identify and eliminate some activities
that did not add value to it, reducing the lead time of the processes. Regarding the Spaghetti Diagram, it
helped to define an ideal layout for the machining sector, eliminating unnecessary employee paths. With the
suggested improvement proposals, there was an annual reduction of 33 hours in activities, which can be used
in a better way by the employee, performing activities that add value to the process and that optimize the
productivity of the organization. These 33 hours are equivalent to EUR 5.019,24.
Finally, the fifth stage of presenting the project, it is clear that the central question “How to reduce downtime
in a machining sector through PBL using Lean Manufacturing as the main driver?” was answered. Not only by
following the PBL staged, applying the methodology and answering the specific questions, but effectively
reducing downtime in an average of 50%. This result shows how important is for students in participating in
academic PBL projects and in partnership with industries, because it gives students the experience of how to
solve real problems within an organization. Another important benefit of this type of project is being able to
guide students in determining the area of Production Engineering that they most identify with.
In addition, through the study, it was possible to apply the knowledge obtained in the academic environment
and participate in the implementation process of the proposed solution. In this sense, students had the
opportunity to live experiences in an industry, while still in the academic period, thus providing learning outside
the university, a balance between theory and practice.
This paper was also very beneficial for the students in three main aspects: first, the approach to the academic
environment, in writing an article and looking for the best references on the subject; the deepening of the
vision through the process, increasing criticality and the search for better solutions that fit the specificity of the
process and, finally, the possibility of understanding how the company and the job market work.
6 References
Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The clearing house, 83(2), 39-43.
Davim, J. P. (Ed.). (2008). Machining: fundamentals and recent advances. Springer Science & Business Media.
Gil, A. C. (2002). Como classificar as pesquisas. Como elaborar projetos de pesquisa, 4(1), 44-45.
Gupta, S., & Jain, S. K. (2013). A literature review of lean manufacturing. International Journal of Management Science and
Engineering Management, 8(4), 241-249.
Larmer, J., & Mergendoller, J. R. (2010). Seven essentials for project-based learning. Educational leadership, 68(1), 34-37.
Machado, Á. R., Coelho, R. T., Abrão, A. M., & da Silva, M. B. (2015). Teoria da usinagem dos materiais. Editora Blucher.
Matt, D. T.; Rauch, E. Implementation of Lean Production in small sized Enterprises. In: 8th CIRP Conference on Intelligent
Computation in Manufacturing Engineering. Italy, 2013.
Rother, M., & Shook, J. (2003). Aprendendo a enxergar o fluxo de valor para agregar valor eliminando o desperdício. Rio de
Janeiro: Lean Institute Brasil.
Silva, S. D. (2014) Processos de Programação, Preparação e Operação de Torno CNC. Saraiva Educação SA.
Santos, L. C., Gohr, C. F., & dos Santos, E. J. (2011). Aplicação do mapeamento do fluxo de valor para a implantação da
produção enxuta na fabricação de fios de cobre. Revista Gestão Industrial, 7(4), 118-139.
Tapping, D., & Shuker, T. (2010). Lean Office: Gerenciamento do fluxo de valor para áreas administrativas - 8 passos para
planejar, mapear e sustentar melhorias Lean nas áreas administrativas. São Paulo: Editora Leopardo.
Thomas, J. W. (2000). A review of research on project-based learning. The Autodesk Foundation.
Touriki, F. E., Benkhati, I., Kamble, S. S., & Belhadi, A. (2021). An integrated smart, green, resilient, and lean manufacturing
framework: A literature review and future research directions. Journal of Cleaner Production, 319, 128691.
163
Turrioni, J. B., & Mello, C. H. P. (2012). Metodologia de pesquisa em engenharia de produção: estratégias, métodos e técnicas
para condução de pesquisas quantitativas e qualitativas. Itajubá: Unifei.
Womack, J. P., & Jones, D. T. (2003). Banish waste and create wealth in your corporation. Recuperado de http://www. kvimis.
co. in/sites/kvimis. co. in/files/ebook_attachments/James.
Yin, R. K. (2015). Estudo de Caso-: Planejamento e métodos. Bookman editora.
164
An Approach to Encourage Girls' Protagonism in Exact Sciences
and Engineering
Amanda Nunes Rocha1, Ingrid de Castro Alves2, Larissa Pereira da Costa Santos3, Laura Beatriz Lima de Sousa3,
Dianne Magalhães Viana3, Maura Angélica Milfont Shzu4, Simone Lisniowski5
1
Exact Sciences Institute, University of Brasília, Brasil
2
Chemistry Institute, University of Brasília, Brasil
3
Faculty of Technology, University of Brasília, Brasil
4
Faculty of Gama, University of Brasília, Brasil
5
Faculty of Education, University of Brasília, Brasil

[email protected], [email protected], [email protected]
Abstract
The 21st century has come highlighting the world's concerns regarding competitiveness in face of new demands arising
from accelerated technological advancement and, on the other hand, with inclusion, sustainability and quality of life.
Therefore, gender equity in STEM fields has also become part of the goals of the 2030 agenda for sustainable development,
with the aim of building a less exclusionary world. Faced with this urgent need to design a more inclusive future, several
approaches have been proposed to attract girls to the exact sciences and engineering using active learning strategies and
valuing the protagonism of these students in their school environment. In order to do so, a group of six ninth grade girls
were invited to participate in the pilot project Accelerating Girls in Elementary School, conducted by undergraduate
students in the exact sciences and engineering, and accompanied by professors from the school and from the University
of Brasilia. In it, the Elementary School students were organized into a team to propose and present STEM hands-on
practices, according to curricular contents, to their own class. The steps for designing such activities include weekly
meetings for the development of the flipped classroom, quizzes and hands-on activities. Aspects and outcomes related to
this process are described and discussed based on the literature, motivating reflections on strategies used to develop skills
and competencies in STEM with young girls.
Keywords: Active Learning; Elementary School; Girls Protagonism; STEM Education.
1 Introduction
In recent years, the rapid technological development, on a worldwide scale, has been causing movements in
education and in the job market in search of greater female inclusion, since women were historically placed on
the margins of this academic and professional space. Despite advances, education is still not universal and as
gender inequalities are still significant, and a current worry, not only the girls access to a current school, but
also the limitations they face in the academic environment, more specifically, is reflected in the low number of
girls regarding STEM education.
In line with current demands, the 2030 Agenda was conceived on September 2015 during the United Nations
General Assembly. It is a document that encompasses objectives with a view to tackling global problems for
the next 15 years (2016-2030), with these sustainable development objectives (SDGs) supported by three
dimensions: social, environmental and economic. Among the 17 SDGs, two stand out, aiming at improvements
in terms of quality education and gender equality, SDG4 and SDG5, respectively. In a report made in 2018,
UNESCO — a United Nations agency specializing in education — places education in Science, Technology,
Engineering and Mathematics (STEM) as one of the bases of the 2030 Agenda to achieve its goals, stressing
that “ensuring that girls and women have equal access to STEM education and, ultimately, to STEM careers, is
imperative for human rights, scientific and developmental perspectives” (UNESCO, 2018, p.15). Initiatives that
take place at school, such as extracurricular activities, and joint projects with universities, have positive impacts
on girls' interest in subjects and careers in the STEM field (Oliveira, Unbehaum, & Gava, 2019).
165
In view of this scenario, this study aims at reporting the activities developed by the Accelerating Girls project
in Elementary School — an extension and research project from the University of Brasília developed in the
scope of STEM education — reflecting on its reach and challenges in face of the promotion of female
protagonism in areas of Exact Sciences and their technologies, especially with regard to their approach and
execution with students from the public educational system of the Federal District, Brazil.
It should be noted that extension is one of the pillars of Brazilian federal institutions of higher education and
comprises activities that aim to build paths to tackle social problems and issues, to popularize science, and to
strengthen the interlocution with other knowledge, enhancing the interaction with teaching and research.
Given the context, the present research started from the need to identify more suitable strategies for the
application of Active Learning Methodologies as an incentive and study motivation factor, and at the same
time to stimulate the debate on the promotion of gender equality and empowerment of girls and women
through STEM education.
The objective of this work is to present the aspects related to the development of STEM workshops in the ninth
grade of a public elementary school and the learning gains of female elementary school students, monitors of
this class, who participated in the development and execution of these workshops in the scope of a pilot
proposal.
This is an action-research project in which proposals for the implementation of STEM workshops are carried
out by the group of students and monitored during their realization. Reflections are made based on the results
of the implementations and on interviews with the student monitors.
The work is organized as follows: the second section presents the bibliographic review that supports the
measures adopted in the project; the third section exposes the context and model of the work conducted; the
fourth section brings the partial results; and finally, the fifth section includes the concluding remarks of the
developed research.
2 Active Learning and the Potential in Regard to Academic Protagonism

In the ideal future designed by the world nation, one of the main ideals is the respect for plurality and the
appreciation of education in order to “ensure that all human beings can fulfill their potential in dignity and
equality” (Agenda 2030, 2015, p. 1). It is common for each individual to have their particularities, so, in the field
of education, it is natural that each student has preferences regarding the most effective study methods for
themselves. Thus, seeking to respect students´ singularities, and make our education even more efficient, it is
expected that as the ideal world is transformed into reality, new strategies and teaching-learning methods will
be created and used.
Assuming that education is the main fuel for life improvements, it is important that the teaching-learning
methods and strategies are able to promote the development of students' competencies and skills. Active
learning strategies place the student in a more active role, contributing to the formation of responsible and
critical citizens (Freire, 1997, p. 59). Placing the learner in a protagonist position means breaking away from the
authoritarian model that reserves to the student the role of simply watching, listening and taking notes of pre-
established contents (Felder & Brent, 2009). The report "The Future of Jobs" by the World Economic Forum
(2020) points out some important and valued skills and competencies in the professional of the future. These
skills and competences are: active learning, problem solving, critical thinking and analysis, creativity, originality
and initiative; which are developed in activities such as debates, seminar presentations, experiments and
workshop development.
These are knowledge sharing activities, and several tools can be used to make the process of intellectual gain
more enjoyable. When the exchange is done horizontally, that is, by peers, it eliminates some obstacles that
hinder learning. In this way, placing the learner as an active subject in the dissemination of knowledge
encourages his peers to also feel part of this process.
166
In this dynamic, it is possible to work on communication strategies to promote the engagement of the parties
involved, on research and study methods, on creativity in designing the presentation format of the content,
and on skills with new active learning technologies. Thus, various aspects related to student empowerment,
such as the responsibility to study and understand the subject before presenting the work, since she has the
task of teaching the classmates, giving that student the right motivation to learn. Inverting the teacher-student
role also affects the student's creativity when deciding which methods to use in order to make the presentation
interesting, as he or she can create fun slides, search for videos, create quizzes, etc. In addition, it improves
orality, argumentation and empathy, as Snider and Schnurer (2006) state in relation to debating activities.
There are several definitions and conceptions of active learning. Some definitions are presented by Mahlambi
(2021):
According to Abu Bakar and Ismail (2020) and Jesionkowska et al. (2020), active learning is an action that directly
involves learners in the process of learning. But Hartikainen et al. (2019:2-3) view active learning as “not just
something that learners do independently but is somehow organised and monitored by an instructor, therefore, an
instructional approach that guides learning”. The critical factor to active learning, especially in mathematics, is
collaborative learner involvement which can assist the learner in improving their conceptions by using their peer
explanations to solve problems (Webb et al., 2019). [...] Virtanen et al. (2017, p. 2) concur, citing that active learning
is an “instructional method that engages learners and includes them as active participants in the learning process”.
To enhance active learning, teachers need to guide learners to take responsibility for what they learn (Virtanen et
al., 2017), particularly as active learning can improve learners’ attitude, understanding of the content (Mathias, 2014)
and enhance the development of their mathematical skills. (Mahlambi, 2021, p. 473, 475).
The active learning conceptions comprise a constructivist vision with active student participation. In activities
such as these, the student is not led to repeat the information provided by the teacher in a test, as is done in
a traditional class, but he or she is invited to research the topic of the class by himself or herself, seeking
concepts and information on that topic, develop them and explain them to the colleagues.
If colleagues do not understand, the student is challenged to explain the topic in a different way, making the
student revisit the knowledge already acquired, reassess it and reconstruct a new argument that is more
coherent or does not have logic errors. In case there are, the student responsible for explaining the task will be
forced to question him or herself and re-evaluate the concepts. Thus, the goal is to create strategies to
overcome the conceptual and epistemological difficulties manifested in the learning process.
Such activity, having so many layers, is one of the most powerful and efficient methods used in the teaching-
learning process. Inverting the teacher–student role, and giving the student the opportunity to teach his or her
classmates, enables the student to gain a greater understanding about the subject, since knowing how to
explain a concept or topic in different ways shows that the student has truly studied and knows about the
subject at hand (Srinath, 2014). It is important to support the students in the construction of a coherent
knowledge and that they can actively participate in the recognition and construction of this knowledge
“following the sequence: identification of the phenomenon, identification of the variables/elements involved,
identification of interactions among them; search for regularities in the observed behaviors, conclusion on the
concepts or law involved“ (Bravo et al., 2022, p. 06, translated by the authors).
Thus, by placing students not as listeners, but as lecturers, debaters, monitors, the student almost completes
the role of protagonist in the learning process. It is also understood that elaborating and conducting a
workshop is not limited to explaining the class topic, but also to researching and elaborating didactic material,
making the student-monitor role even more active in the studies, developing a sense of responsibility and
motivation to learn and do a good job.
3 An Initiative that Aims at Gender Equity in STEM

The Accelerating Girls Project in Elementary School (MAF) is an extension project developed with female
students from Elementary School Center 201 in Santa Maria, a public school located in the Federal District. In
view of the low adherence of girls and women to STEM areas in higher education, which is related to the gender
issue in education, the extension project acted in the planning, elaboration and execution of exact science
167
workshops with themes related to the school curriculum of the 9th grade of that elementary school using active
learning methodologies in all activities. The project was conducted in 2021 and then due to the COVID-19
pandemic, took place remotely. Six female students were chosen by the school science teacher, who was a
partner in the project, and the criteria used were based on their performance in science or their interest in
participating in the project.
The direct work focused only on the female gender is based on studies that show that the group composed
solely of women has more positive attitudes towards topics that are considered male-dominated (Cooper and
Weaver, 2003)
The group that assists the students is made up of undergraduate students from the University of Brasília in the
Statistics, Chemistry, Forestry Engineering, Mechanical Engineering and Electrical Engineering courses — they
act as facilitators with the Elementary School students. The project also counts with the support of professors
from the University of Brasília who work in engineering, education, psychology, sociology, design, and
communication. They guide and supervise the activities developed. During the year, this team carried out
workshops in which the main performers were the Elementary School (EF) students, who presented, based on
their lesson plan, workshops to their colleagues in mixed classes, developing theoretical and practical aspects
previously studied by the whole class.
The Accelerating Girls project in Elementary School was born as a segment of the Fast Girls project, which aims
to encourage the entry of girls and women into STEM areas, being also developed with students from public
schools, including High School ones. It is promoted due to the low permanence and interest of women in these
areas.
According to Linda Gottfredson (1981), from the age of 6 to 8, children begin to form ideas about the influence
of gender in relation to professions and this becomes evident when reaching adolescence, when abilities, values
and interests based on childhood are used to make possible professional choices and roles to be taken. Several
factors consolidate ideas on what would or would not be appropriate for a given gender, from stereotypes,
self-esteem, expectations, gender roles to the absence of female role models (Munilla, 2018). This, when
internalized, influences the vocational behavior of young people, limiting their achievements and aspirations
(Betz & Hackett, 1981; Faria, Taveira, & Saavedra, 2008). Therefore, in order to promote gender equity and
inclusion, it is necessary to act since basic education.
In this sense, the Accelerating Girls in Elementary School aims to foster interest and create a safe space for
elementary school students, providing opportunities for the expression of creativity, logical reasoning and
improving their interest in the areas of mathematics, science and technology. Thus, they are encouraged to
face the likely difficulties related to the gender issue in their path, creating conditions so as not to interrupt
this journey through educational practices which are sensitive to this problem.
3.1 Workshops Model and Methodology

Through active and collaborative learning models, development of practical activities and bringing scientific
content closer to everyday life, workshops led by elementary school students were developed. The workshops
followed the same structure, being divided into stages over five weeks: (1) Delimiting the themes of the workshop,
and the beginning of the studies stage; (2) Designing the flipped classroom; (3) Slides preparation; (4) Experiment
characterization. Preparation of quizzes and questionnaires; (5) Testing the workshop. After the stages had been
performed, it was possible to present them to the target audience.
The workshops were planned according to the syllabus studied in the 9th year of Elementary School, covering
topics such as matter and energy, covering areas of physics, biological sciences, mathematics, chemistry and
others. Themes such as density, chemical reactions, waves and Newton's laws were the guiding principles in
the four workshops held. Within them several aspects were discussed in a multidisciplinary way, contemplating
both the theoretical and the practical parts of the deliberate areas. These various aspects can be observed in
Table 1.
After having delimitated the theme, a week was used for studies, carrying out research activities in books,
websites, articles and searching for videos, images and various didactic resources to formulate ideas about the
168
chosen content. This step is essential for the delineation of the workshop's pillars, and it helps the elementary
school monitors to remember and fix the content previously discussed in the classroom.
Perusall, a platform for inserting images, PDF and media files, hyperlinks and other resources, developed by
educational researchers and behavioral and data scientists from Harvard University, was one of the resources
connecting undergraduate monitors and elementary students because, in a versatile way, it was possible to
highlight important parts of the texts, share new content, exchange comments and ask questions collectively,
enabling an individual activity to be carried out in a collaborative way.
Based on the contents studied and the didactic resources that were grouped, a flipped classroom strategy was
designed, preparing students for the contents that would be discussed later in the presentation. The flipped
classroom has proven to be an important strategy for both preparing the elementary school monitors and
delivering the workshop to the class. Wankat and Oreovicz (2015) defined the flipped classroom as a
pedagogical approach that includes a three-step didactic sequence, pre-class, class, and post-class. In training
the students to conduct the workshops, it is possible to distinguish the three stages: (i) previous studies on the
subject; (ii) feedback and development of collaborative work; (iii) individual work.
Through videos, short texts, images and games, the students were expected to begin a brief study of the topic.
The contents studied also contributed to the formulation of the presentation slides. These were developed
from summaries written by the elementary students in their study phase, and included gifs, small texts, images
and references. The last week execution of the proposed steps was related to the formulation and delimitation
of experiments, quizzes and questionnaires.
Table 1. Workshops conducted and their specifications.
Workshop Themes approached Experiment(s)
Concept of weight, mass, volume and density.

Density calculation. Immiscible substances. Factors
Matter properties that can modify the density of a material.
Lava lamp
– Density Relationship between temperature and density.
Relationship between physical state of matter and
its density.
Meaning of chemical transformation. Signs of

Chemical reactions occurring. How chemical reactions are Potassium permanganate solution, water
reactions written. Types of chemical transformations. Factors oxygenated and manganese dioxide
that prove the occurrence of a reaction.
Definition of wave, amplitude, wavelength, period,

speed and frequency of a wave. Wave classification.
Waves and wave Wave machine and experiment with sound
Direction of wave propagation and vibration.
phenomena waves
Examples of wave phenomena. Definition of sound
waves and their properties.
Concept of kinematics and dynamics, and their Experiment with coins (principle of inertia),
differences. The history of Isaac Newton. experiment with eggs (law of inertia),
Newton´s laws Distinction of force and resultant force. experiment with displacement of a rubber
Description of different types of force. Concept (principle of dynamics) and experiment with
behind Newton's laws. balloons (principle of action and reaction)
The Elementary School students, after the steps outlined above, performed the presentation of the chosen
theme. The class given was conducted as followed: theoretical presentation; resolution of doubts in relation to
the theoretical part; quiz; experiment presentation; solving practical doubts.
The quizzes, which served as a way of evaluating the understanding of the workshops, were prepared by the
elementary students. They included five to seven questions which were discussed in class after the theoretical
presentation, in order to contemplate the main points that were exposed in the presentation.
169
Based on the fact that knowledge can only be developed in the individual's own action (Freire, 2001), the
experiments were thought of as a way of correlating the contents taught with the knowledge and narratives
that students observed in their daily lives, in order to make Physics and Chemistry more understandable and
less abstract to students. These experiments were designed with accessible materials, made available to
elementary school monitors by the guiding professors and purchased with resources offered by the Federal
District Research Support Foundation (FAP-DF). All experiments were exposed to the class in order to highlight
the materials needed for the elaboration and their step-by-step assembly. These could be replicated by the
students in their homes.
Before the class presentation, the elementary students presented the complete workshop to the undergraduate
monitors and coordinating professors, so that they could discuss aspects that could be improved and small
adjustments could be made. That served as a rehearsal and the workshop improvement.
Questionnaires were prepared by the undergraduate monitors to assess the impact of the workshops on the
classes where they were taught. They were sent after the end of the presentation. These involved questions
including the flipped classroom, the experiment, and length of presentation, content presented, the quiz, the
presenters’ didactics, among others. The questionnaires contributed to changes throughout the workshops,
making them adequate to the target audience. This moment was also important for their learning and
autonomy in the process of designing and executing the workshops.
4 Results
The six female elementary school students who participated in the project during 2021 conducted four
workshops as class monitors in a remote virtual environment for about 80 students, boys and girls in the ninth
grade of a science class at Elementary School Center 201 in Santa Maria, on the outskirts of Brasília, Federal
District. Figure 1 shows the cover of the material produced for the flipped classroom and screenshots of the
presentation and the experiment performed by the elementary school monitors.
Figure 1. Workshop of Chemical Reactions: flipped classroom material; remote class presentation; and remote experiment.
To analyze the results obtained with the project, interviews were conducted with these six female students. The
interview had 10 questions that addressed the thematic axes: (1) General aspects (issues related to STEM areas
and active learning methodologies); (2) Adherence to the project; and (3) Encouraging protagonism.
When asked about the project's contribution to increasing their interest in STEM fields, all students answered
affirmatively. One of the interviewees is considering studying aerospace engineering, while another reported
an increased interest in robotics. It was also emphasized that, although they had not previously excelled in
mathematics, participating in the project brought them closer to the STEM areas.
Regarding the small age difference between undergraduate and elementary school students, there was
unanimity in highlighting the age range as a positive point, being interpreted as a contributing factor to the
creation of a relaxed environment. When asked about the methods and strategies adopted, experiments and
quizzes, these were positively evaluated, highlighting the playful and interactive nature of these proposals. The
inverted classroom was also mentioned; they emphasized its important role in the study and individual
preparation before the workshop with the undergraduate monitors.
One of the interviewees emphasized that the research activity helped thematic understanding in comparison
with traditional learning methods. All of the interviewees evaluated the experiments as a facilitating strategy
170
for understanding the content. Such active methodology strategies, according to two of the interviewees, were
responsible for their better performance in science assessments.
This effectiveness in performance on assessments changes how female students perceive themselves in the
face of learning in STEM. The issue of vocation is called into question when girls realize that the problem is not
an individual characteristic. The prejudices linked to gender begin to be addressed through the use of an
active methodology that develops autonomy, interaction, and creativity.
Moving on to axis 2, about the questions referring to adherence to the project's gender proposal, when asked
about the motivations needed to continue in the project, one of the answers was "I liked it a lot because there
were only girls”. One of the students commented that being with her classmates was a motivational factor to
continue doing the workshops.
For those who left or thought of leaving the project early, they were asked to explain their motivations. One of
the students explained that the pandemic and, possibly, remote learning, left her demotivated and, for this
reason, she thought of leaving the project, but that after talking to one of the group members she decided to
stay. This student participated in all four workshops and reiterated that, despite not being the central incentive,
the CNPq scholarship she received encouraged her to keep going to the meetings.
All students confirmed their interest in participating in future project actions. In fact, four of them have already
started to participate in the Fast Girls project after entering high school. One of the students said that the
project during the pandemic was an encouraging factor for her, as it fostered interaction with other classmates,
which was hindered by physical isolation during the pandemic.
In axis 3, which dealt with the students' protagonism in the project, all of them answered that they had
autonomy to decide the themes of the workshops and discuss the strategies used, although one of the students
commented that not all of them used this autonomy to give their opinion on the methods and strategies used
in the workshops.
Finally, when asked about the benefits of having participated in the project and what they learned in the process
that they would take back to high school, the answers ranged from improved teamwork dynamics to improved
public speaking and presentation skills.
In this perspective, when acting as monitors, the elementary school students felt protagonists in the
presentations and in carrying out the experiments, and realized greater resourcefulness, greater curiosity in
the classroom and greater ability to understand and correlate content. Through the methods and strategies
used and the production of materials based on the theories studied in class, the students were able to
consolidate their knowledge and instigate classmates to think critically. Understanding that active learning
requires questioning, problematization, and the sensorial and emotional experience of knowledge (Oliveira,
2006), which are not such characteristics explored in the standard learning process in which students absorb
knowledge in a passive way (Leão, 1999). As we could see from the reports, the project correlated the contents
to the students' reality, encouraging them to observe and question phenomena, allowed the appropriation of
methodological tools, of experiences in the construction of scientific knowledge and its articulation with
everyday life, as well as developing skills in organizing, planning, and executing the workshops.
5 Concluding Remarks
The selected students are a vulnerable group - taking into account monetary parameters, housing region and
gender and race identification - in this sense, this is a valuable project in the socioeconomic sphere, being in
line with what is advocated by Adams, Gupta and Cotumaccio (2014), who argue that science programs, outside
the school, help in the trajectory of young people in the scientific world, especially girls from underrepresented
groups in this area. Thus, it is generalized that, although it brings elements that are similar to successful
initiatives regarding the encouragement of girls and women in STEM areas, because it is a project still in its
infancy, the process of interest needs to be maintained and deepened so that it can introduce relevant changes
that enable the promotion of the advancement of women that is necessary in the scientific and technological
field, as well as in educational institutions.
171
The 2030 Agenda, together with UNESCO, has placed STEM education as one of the pillars for achieving this
goal. In this sense, actions to encourage girls to engage in Science, Engineering, Technology and Mathematics
developed while still at school age proved to be positive. In light of this, the project Accelerating Girls in
Elementary School, with the use of Active Learning, developed its activities in order to encourage girls and
women in the areas of STEM, encouraging the permanence of undergraduates in the areas and providing a
more personal and active initiation of Elementary School students with Exact Sciences. In view of what has been
exposed, the results obtained so far have shown to be promising, since even those students who had no initial
interest in STEM remained motivated and participatory during the activities. Currently, the Accelerating Girls in
Elementary School project aims to migrate to the face-to-face format without any loss in the quality of the
workshops, evasion of monitors and losses of its central idea.
Acknowlegments
To Fundação de Apoio a Pesquisa (Research Support Foundation) of the Federal District, FAP – DF, to DEX/UnB,
to DPI/UnB and to DEG/UnB for financial support. To Centro de Ensino Fundamental - CEF 201 for the
partnership and pedagogical support to students in the project.
6 References
Adams, J. D., Gupta, P. & Cotumaccio, A. (2014). Long-term participants: a Museum Program Enhances Girls’ STEM interest, motivation,
and persistence. Afterschool Matters, Wellesley, 20,13-20.
Agenda 2030. (2015). Transformando nosso mundo: A agenda 2030 para o desenvolvimento sustentável. Available at
https://brasil.un.org/pt-br/resources/publications.
Betz, N. E.; Hackett, G. (1981). The relationship of career-related self-effi cacy expectations to perceived career options in
college women and men. Journal of Counseling Psychology, 28, 399-410.
Bravo, B., Pesa, M. & Braunmüller, M.. (2022). MarinéIDAS: una metodología de enseñanza centrada en el estudiante para
favorecer el aprendizaje de la física. Revista Brasileira de Ensino de Física [online], 44. doi:10.1590/1806-9126-RBEF-
2021-0326.
Cooper, J. And Weaver, K. D., 2003. Gender and Computers – Understanding the digital divide. New Jersey, London: Lawrence Erlbaum
Associates. 168p.
Faria, L.; Taveira, M.; Saavedra, L. (2008). Exploração e decisão de carreira numa transição escolar: Diferenças individuais.
Revista Brasileira de Orientação Profissional, 9, 7-30.
Felder, R. M., & Brent, R. (2009). Active learning: An introduction. ASQ higher education brief, 2(4), 1-5.
Freire, P. (1997). Professora sim, tia não- cartas a quem ousa ensinar. São Paulo: Olho d’água, 59, 84 p.
Freire, P. (2001). Educação como prática da liberdade. São Paulo: Paz e Terra, 25, 158 p.
Gottfredson, L. S. (1981). Circumscription and compromise: A developmental theory of occupational aspirations. Journal
of Counseling Psychology, 28, 545-579.
Leão, D. M. M. (1999). Paradigmas contemporâneos de educação: escola tradicional e escola construtivista. Cadernos de Pesquisa, 107,
187-206.
Mahlambi, S. B.. Avaliação para a aprendizagem como driver para a aprendizagem ativa e participação do aluno em matemática. Int J Educ
Methodol. 2021;7(3):473-485. doi: 10.12973/ijem.7.3.473
Munilla, M. S. (2018). The absence of women in STEM careers: a social and gender problem. adolesCiência, Revista Júnior
de Investigação, 5, 14-16.
Oliveira, C. L. (2006). Significado e contribuições da afetividade, no contexto da Metodologia de Projetos, na Educação Básica.
Dissertação de Mestrado – CEFET, Belo Horizonte, MG.
Oliveira, E. R. B., Unbehaum, S., & Gava, T. (2019). A educação STEM e gênero: uma contribuição para o debate brasileiro.
Cadernos de Pesquisa, 171, 130-159.
Snider, A., & Schnurer, M. (2006). Many sides: Debate across the curriculum. IDEA. 321 p.
Srinath, A. (2015). Active Learning Strategies: An Illustrative Approach to Bring out Better Learning Outcomes from Science,
Technology, Engineering and Mathematics (STEM) Students. International Journal of Emerging Technologies in
Learning, 10.
UNESCO (2018). Decifrar o código: educação de meninas e mulheres em ciências, tecnologia, engenharia e matemática
(STEM). Organização das Nações Unidas para a Educação, a Ciência e a Cultura (UNESCO), Brasília, DF.
Wankat, P. C., Oeoviks, F. S. (2015). Teaching engineering. Indiana, USA: Purdue University Press, 450 p.
World Economic Forum. (2020). The Future of Jobs. Report. Geneva, Switzerland: World Economic Forum.
172
Development and Validation of Scenarios for the assessment of
Project Management People Competences
Mariane C. Souza 1,2, Rui M. Lima1 , Diana Mesquita3, Elida M. Margalho1
1
Department of Production and Systems, Algoritmi Centre, School of Engineering, University of Minho, Guimarães, Portugal
2
School of Engineering, Federal University of Rio Grande – Campi Carreiros, Brazil
3
Research Centre for Human Development (CEDH), Faculty of Education and Psychology, Universidade Católica Portuguesa
Email: [email protected], [email protected], [email protected]; [email protected]
Abstract
The development of project management competences, particularly people competences, is a challenging and complex
process. Scenario-based learning is an interesting approach to developing these competences, involving real problems that
allow you to face challenges based on your own experience. In this sense, scenarios act as the starting point for diving into
a real-world problem and fostering the whole process of decision making and searching for solutions. The International
Project Management Association (IPMA) has defined an Individual Competence Baseline (ICB 4.0) standard for the
development of the project management area of knowledge. The standard shows, besides other competences, the
following 10 people competences: personal communication; leadership; teamwork; self-reflection and self-management;
personal integrity and reliability; relationships and engagement; resourcefulness; conflict and crisis; negotiation; and result
orientation. Development and assessment of these competences are increasingly important for the people working on
projects. In addition, there is an accreditation process for these professionals that is increasingly required by organizations.
The process involves an assessment of technical qualifications to improve task performance. In this sense, this paper aims
to describe the development and validation of ten project management scenarios that allow the assessment of people
competences. The methodology of the study comprised six phases. In addition, two pilot applications were conducted with
the proposal to assess competence of the scenarios developed. The findings reveal that the scenarios can be used as an
alternative approach to the accreditation process of professionals who work in project management.
Keywords: Competences Development; Scenario-Based Learning; Project management; Engineering Education.
1. Introduction
The world is in permanent change. People, processes, and environments change and evolve at an ever-increasing rate in a
world characterized by marked interconnections and externalities. Given this, it is quite naive to think that projects, and the
individuals who participate in and manage them, will not need to adapt to and embrace change and uncertainty in order
to successfully achieve their outlined goals (López-Alcarria, Olivares-Vicente, & Poza-Vilches, 2019). The people who
manage projects (the project management community of practice) are increasingly requiring the development and
assessment of these professionals (Alam, Gale, Brown, & Kidd, 2008).
As learning becomes more personalized and requires adjustment to changing conditions and requirements while meeting
rigorous accreditation standards, new forms of assessment are needed (López-Alcarria, Olivares-Vicente, & Poza-Vilches,
2019). The use of scenarios, i.e., challenging situations inspired by project management practices, can be a valuable tool
for both organizations and individuals to develop and assess their competencies. There is a growing interest in development
and assessment of project management competence (Crawford, 2005).
In this regard, the research question of the study sought to understand, "In what ways do scenarios enable the assessment
of project management competency?" The main purpose of this study is to describe the development and validation of
Project Management scenarios for the assessment of competencies defined by the Individual Competence Baseline
2. Background
Recognized as a profession in the mid-20th century (PMI, 2017), project management continues to evolve, and as old
methods undergo modifications, there is the creation of new tools and the emergence of research with new viewpoints in
the profession (Wawak & Woźniak, 2020). This evolution represents the creation of several benchmarks that can be used
as a basis for professional development and assessment. The Individual Competence Baseline (ICB4) is a reference source
173
for those seeking a method option in project management that is more focused on the human aspects of management,
namely people competences (IPMA, 2015).
This benchmark was developed by IPMA (2015) and describes a coherent inventory of competence elements that an
individual needs to have or develop to successfully master the work package, project, program or portfolio they have been
assigned to manage (IPMA, 2015). Furthermore, it does not detail the competences by specific roles (e.g., project manager
or planning specialist) but rather in terms of what is required in the domain of project, program, and portfolio management
(Vukomanović et al., 2016).
The individual "competence" is the application of knowledge, skills and abilities in order to achieve the desired
results (ICB4). The concept of competence is strongly associated with the ability to master complex situations. In the
professional world, accreditation and training of professionals through proficiency testing is strongly demanded by
organizations (Erol et al., 2016). The process involves the assessment of competences required for improvement in task
performance. In the study of (Erol et al., 2016) a scenario-based Industry 4.0 was developed to assess how common problem
engineering competences can improve performance.
Several research papers have focused on the development scenarios to assess competences (Ilahi et al., 2014). However,
competence assessment is not very well researched in the context of project management, and there is a gap in competence
assess models (Ilahi et al., 2014). In this sense, although changes have occurred as a result of the emphasis on competence-
based learning, both in the academic and professional worlds, assessment processes are often based on traditional models
(Ilahi et al., 2014). In fact, in the practical world, professionals or academics are more often faced with situations where they
have to apply their acquired competences, but they are still examined by traditional assessment models (Ilahi et al., 2014).
In the fields of Medicine and Nursing, one of the approaches used to assess competences is centred on the principles of
competence assessment through simulated scenarios (Hagler & Wilson, 2013; Waxman, 2010). In general, this approach of
scenario can be used both for assessment and for learning purposes, where the individual passes through a simulated
situation that is aligned with the behaviour they would adopt in a similar situation in real life (Banerjee, 2019).
As defined by Rosson & Carroll (2009, p. 149): "...like any story, a scenario consists of a context, or state of affairs; one or
more actors with personal motivations; knowledge; capabilities; and various tools and objects that the actors encounter
and manipulate. The narrative describes a sequence of actions and events that lead to an outcome. These actions and
events are related in a context of use that includes the goals, plans, and reactions of the people who are part of the episode."
The use of scenarios has potential application in many contexts, problems, issues, and when used in the classroom provides
a useful means for students to approach the reality of their profession (Errington, 2011) by applying their knowledge and
competences to solve issues in a safe environment (Erol et al., 2016). In this sense, students, as potential professionals, are
confronted with a description of the scenario, where they should assume roles or certain specific perspectives to explore it
in order to enrich and complement the application of their competence in a professional context. Notably, scenarios do
not replace work immersion, but through them, there is the construction and deconstruction of authentic experiences
(Errington, 2011).
A scenario-based assessment involves asking participants to answer questions or challenges related to a short case, where
the assessment is supported by a scenario, several questions or challenges related to it, and a system for assigning scores
(Daniel & Mazzurco, 2019). The application of case studies can be seen as an appropriate methodology for competences
development, and in particular in preparing for the professional world of project management, because it allows the
participant to apply the knowledge learned, enabling reflections of what went well, what went wrong, and what would be
most recommended for improvement in the future (Kerzner, 2006).
According to Hagler & Wilson (2013), there is little research that develops guidelines for writing and using scenarios, also
little has been found in the project management literature assessment models through scenarios. In general, scenarios
should be able to require the participant to perform the competences being assessed and match the level of complexity of
the actual task (Hagler & Wilson, 2013). However, before the competence can be assessed through the scenarios it is
necessary to plan and develop processes, tools, and resources (Hagler & Wilson, 2013).
(O'Brien, Hagler, & Thompson, 2015) highlight a set of best practices that should be considered when developing scenarios
for competence assessments, the first consists of a good theoretical foundation to identify the objectives or competences
that are significant for the assessment, the second best practice refers to the selection of parameters for the scenarios,
these parameters should be evidence-based, i.e., be grounded in practical guidelines that allow the simulation of real
scenarios. The third best practice has to do with the link between the assessment elements, namely: the competences that
are to be assessed, the instrument to be used in measuring the competence, and the scenario itself. Based on these best
practices, the authors proposed an eight-step validation process for scenario design in simulation-based competence
assessment.
174
First define the purpose of the proposed assessment (Step 1), then select a measurement instrument useful for the intended
objectives (Step 2). In designing the scenario, a literature review is required (Step 3), then the scenario must be written and
mapped (Step 4), the validation team must be selected (Step 5), and the method of conducting the validation process must
be developed (Step 6). Finally, test the scenario (Step 7) and review and assess the feedback (Step 8) (O'Brien et al., 2015).
Given this, sought in this study, develop scenarios with real situations of project management, which were able to assess
the people competences of professionals working in the area.
3. Methodology
The methodology of this study comprised six (6) steps for the process of developing, validating and assessing scenarios to
assess on Project Management people competences. The steps of the study were adapted from the study by O'Brien et al.,
(2015). Table 1 shows the steps adapted for the study.
Table 2. Steps of the study
Steps (O’Brien et al., 2015) Adapted steps (O’Brien et al., 2015)

1. Define the Purpose 1. Define the Purpose
2. Select the Tools
2. Review and selection of studies
3. Review the Evidence
3. Development of scenarios
4. Write and Map the Scenario
4. Review Linguistic of scenarios
5. Select a Validation Team
5. Validation of scenarios
6. Seek Consensus
7. Pilot the Scenario
6. Pilot study
8. Assess and revise
Step I. Define the Purpose: the purpose of developing scenarios is to integrate these scenarios into an assessment model
of engineering project management competences (study under development). Assessment through scenarios is
characterized as an approach that makes it possible to represent real situations or situations inspired in the professional
reality of project management, thus, the purpose in developing scenarios is to assess project management competences.
Step II. Review and selection of studies: understanding and selection of studies was defined as the second phase, which
aimed to search for general studies on scenarios, as it allowed identifying information, facts and evidence. This activity was
developed searching the Scopus database. The end result of this phase was a general understanding of scenario-based
learning in a variety of contexts. The authors (Wawak & Woźniak, 2020; Ilahi et al., 2014; Hagler & Wilson, 2013; Erol et al.,
2016; Banerjee, 2019; Errington, 2011; O'Brien et al., 2015) were references in this phase for a general understanding about
scenarios. It is important to mention that other studies contributed with less significance. However, we felt the need to
deepen the analysis considering the engineering area. Following the procedures developed in the Scopus database,
selecting only journals linked to the engineering area, resulting in 44 studies. Additionally, some works from the Project
Management area of knowledge were also analysed: Kerzner (2006); PMI. (2017) A guide to the project management body
of knowledge (PMBOK); IPMA/Individual Competence Baseline for Project Management (ICB4) and Miguel, A. (2019).
The Individual Competence Baseline for Project Management (ICB4) was the work selected as the fundamental work for
this article. Defined as an International standard and developed by more than 150 experts on the subject, it presents a
complete inventory of key individual competences for people working on projects. In this sense, it is characterized as a
validated and complete instrument when seeking to develop and assess competences. It presents three domains of
individual project management competences: perspective, practice and people competences.
For reasons related to time and capacity limitations it was decided to select the people competences domains, including
the 10 (ten) people competences and the assessment criteria (indicators and performance measures), which will be used to
describe and assess the scenarios.
Step III. Scenario development: for the development of the scenarios the 10 (ten) people competences described in the
(ICB4) defined by the International Project Management Association (IPMA) were used in the study. In this sense, each
scenario seeks to assess a specific Project Management competence, as an example: scenario 1 is based on the personal
communication competence, scenario 2 is based on leadership and so on. Thus, each scenario assesses a base competence.
Table 2 shows the scenarios that were developed in the study.
175
Table 3. Scenarios developed
Scenarios Base Competence

1 Self-reflection and self-management
2 Personal integrity and reliability
3 Personal communication
4 Relationships and engagement
5 Leadership
6 Teamwork
7 Conflict and crisis
8 Resourcefulness
9 Negotiation
10 Result orientation
Step IV. Review with a Professional Social Science and Project Management Professional: the third phase of scenario
development, comprised an in-depth review of the scenarios by two scholars, working in the fields of Educational Sciences
and Project Management. The process aimed at two key inputs: (i) Linguistic review, and (ii) Review the content. Suggestions
for improvement were analysed and the main improvements suggested were implemented.
Step V. Validation with professionals: whose objective focused on the collection of the first perceptions about the
scenarios, problems that were exposed and actions for improvements with key informants. The key informants were
professionals who work in the area of project management. Thus, for the selection of the professionals, two basic criteria
was considered: professional experience linked to engineering projects and/or complementary training linked to the area
of project management. With this in mind, a questionnaire survey was developed and sent via email, with open and closed
questions; the questionnaire was created with 20 closed questions and 10 open questions. The closed-ended questions
were answered using an agreement Likert-type scale, ranging from 1 to 5, represented qualitatively by the following
perceptions: "1. totally disagree"; "2. disagree"; "3. neither agree nor disagree"; "4. agree"; "5. totally agree".
An invitation was sent, via e-mail, to 10 professionals, and we obtained the acceptance of all professionals. However, after
sending the survey by questionnaire, a deadline of 30 days was offered, and at the end of the deadline, we obtained a total
of 7 respondents. Table 3 presents a summary of the characterization of the profile of the professionals who were surveyed
in this phase.
Table 4. Characterization of professionals

# Basic Training Training Additional Experience in the area
1 Mechanical Engineering Specialization in Project Management 5 years
MBA in Project Management
3 Plastics Engineering 12 years
MBA in Careers, Leadership and Coaching
4 Business Administration Professional Master's in Administration 10 years
5 Agronomic Engineering Master's in Project Management 2 years
6 Production Engineering 6 years
The qualitative analyses of the survey results focus on the open-ended questions in the questionnaires. For each scenario
presented, a question regarding suggested improvements was asked to the respondents. The quantitative analysis of the
surveys went through statistical treatment which consisted of classification, calculation and analysis procedures. For this,
the program SPSS (Statistical Package for the Social Sciences) version 26 was used. The analysis used was the calculation
of Cronbach's Alpha coefficient.
Cronbach's Alpha coefficient is a technique used to assess the reliability and internal consistency of instruments. The
objective in using the coefficient is to ensure the accuracy/reliability of what has been developed. In this sense, the
coefficient measures the degree of reliability of the information obtained (constancy of the results and homogeneity of the
items).
176
Step VI. Pilot study: with the scenarios statistically analysed from the standpoint of professionals in the field, the fifth
phase of this study sought to apply the scenarios. To this effect, the scenarios were applied, and an assessment of project
management competences was simulated by using scenarios. With this in mind, participants were sought who could
contribute to the analysis of the scenarios. To this end, two criteria were defined: volunteers and students enrolled in the
same academic year of the Integrated Master's Degree in Industrial Engineering and Management (MIEGI) of the University
of Minho, Portugal. The objective in assessing these scenarios was to know the perceptions of the participants about the
understanding, time estimation (reading and response), advantages and limitations identified in the scenarios and in the
assessment of competences by scenarios.
In this sense, the pilot studies were organized in two groups of participants, with pilot study 1 being conducted online and
pilot study 2 in person. It is important to mention that the choice of dividing the application of the scenarios into two
groups of participants was an option of the authors due to the issue of the physical structure that could hold all the
participants at the current moment (world pandemic). In pilot study 1, three participants were involved, and each
participant answered one scenario: personal communication, leadership, and teamwork, and it was carried out in the month
of December/2021. The second pilot contained three participants involved and the data collection process took place, face-
to-face, in a room located at the University of Minho, exclusively for the process. For this group, seven scenarios were
assessed: introspection and personal management, personal integrity, relationships and commitments, conflict and crisis,
ingenuity, negotiation, and finally, results orientation. The pilot study was conducted in the month of March/2022. Figure
1 shows the main activities conducting the study I and II.
Welcome General instructions Initiate the process
Discussion Finish Taking notes
Figure 1. Activities Pilot Studies
In addition to the participants, an evaluator and a mediator were present in the studies. The assessor a professor with a doctorate
and research focus on teaching and project management, with teaching experience in competence development and assessment,
and the mediator was a professor with a focus on assessment studies of project management competences.
4. Development and Validation of scenarios

Developing scenarios for assessing Project Management people competences enables an assessment that mobilizes different
resources, including knowledge, competences and experience in addressing a problem related to professional practice. Creating real
or reality-inspired situations makes the assessment process "more flexible" and real in that one must mobilize different kinds of
knowledge, not just memorization to deal with that problem situation. As an example, a question developed by the evaluator in some
of the scenarios and which makes it possible to mobilize different resources: "Have you been through a situation similar to this one
(exposed in the scenario), either as a leader or as a team member? If yes, what actions did you take that you think could be useful for
this scenario? Each scenario developed reflects a practical project management situation and makes it possible to assess
competences.
4.1 Analysis of Scenarios – a quantitative perspective

The professionals' contributions allowed to integrate a more realistic perspective to the scenarios. The result obtained
indicated a value of the Coefficient Cronbach’s Alpha, considering all dimensions, of 0.89, a value considered to be of
almost perfect reliability, according to parameters used by (Landis & Koch, 1977). According to (Landis & Koch, 1977) the
classification of the coefficient reliability follows the following limits: 0.81 to 1.0 almost perfect; 0.61 to 0.80 substantial;
0.41 to 0.60 moderate; 0.21 to 0.40 reasonable; 0 to 0.20 small.
For the data analysis of the questionnaire applied to the project management practice professionals for each scenario two
closed questions and one open question were developed. The closed-ended questions were answered using an agreement
Likert-type scale, ranging from 1 to 5, represented qualitatively by the following perceptions: "1. totally disagree"; "2.
disagree"; "3. neither agree nor disagree"; "4. agree"; "5. totally agree".
177
Regarding the following question: "Does the scenario represent a situation of Project Management professional practice?".
The data presented in Table 4 indicate that in the self-reflection and self-management and personal integrity and reliability
scenarios 2 out of 7 of the respondents indicate that they do not agree with the representation of professional practice in
the scenarios. In addition, the leadership and teamwork scenarios reveal some indecision on the part of respondents. In
the other scenarios the results indicate agreement in the representation of the practical situation of project management.
Table 4. Answers from 7 professionals regarding the representation of professional practice situations
Neither agree nor
Scenarios Base Competence Totally disagree (1) Disagree (2) Agree (4) Totally agree (5)
disagree (3)
1 Self-reflection and self-management 2 1 2 2

2 Personal integrity and reliability 1 4 2
3 Personal communication 1 4 2
4 Relationships and engagement 2 3 2
5 Leadership 2 3 2
6 Teamwork 1 2 2 2
7 Conflict and crisis 3 4
8 Resourcefulness 3 4
9 Negotiation 1 2 4
10 Result orientation 4 3
Regarding the analysis of the second question "Does the scenario provide the opportunity to demonstrate competence?". In
the personal communication scenario one expert indicated disagreement with the statement. In the scenarios, relationships
and engagement, leadership, conflict and crisis and negotiation there was indecision by one expert. Finally, in the remaining
scenarios, there was agreement on the scenario's opportunity to demonstrate the core competence. The results are shows
in Table 5.
Table 5. Answers from 7 professionals regarding the opportunity to demonstrate the core competence
Neither agree nor Totally agree
Scenarios Base Competence Totally disagree (1) Disagree (2) Agree (4)
disagree (3) (5)
1 Self-reflection and self-management 2 4 1
2 Personal integrity and reliability 4 3
3 Personal communication 1 5 1
4 Relationships and engagement 1 3 3
5 Leadership 1 3 3
6 Teamwork 3 4
7 Conflict and crisis 1 3 3
8 Resourcefulness 4 3
9 Negotiation 1 2 4
10 Result orientation 4 3
4.1 Analysis of Scenarios – a qualitative perspective

The qualitative analysis of this study occurred in two moments: i) analysis of the improvements in the scenarios from the
professionals' point of view, regarding the open question of the questionnaire; ii) analysis of the narratives of the pilot
studies. The main results will be presented regarding the proposed improvements for the scenarios.
Regarding the Self-reflection and self-management scenario, professional 3, indicated:
"Since it is an aid and/or recommendation to the colleague, it seems to me that what we will be able to extract from the
interviewee, will be a transfer of best practices and experiences".
Regarding the communication scenario, two main propositions were evidenced:

"...in previous moments the Project Manager had already mentioned that in situations like this it would be necessary to attend
and consider procedures such as..." (professional 3).
"Emphasize the importance of performing the activities according to the procedures, making it clear that, any problem should
be communicated to the project manager" (professional 7).
For the leadership scenario:
178
"It is necessary to include in the scenario the step that involves eventual attitudes thought and/or designed by the manager, to
be taken or put into practice" (professional 3).
"I think it is important to emphasize in the scenario that the "project manager" is hierarchically superior to the "construction
coordinator". The nomenclature of the position, if misinterpreted, can distort the argumentation" (professional 4).
Regarding the teamwork scenario:

"I would modify in the sense that the team leader is centralizing all the information and decision making, changing the context
that he would be doing everything alone, but centralizing all the decisions with him" (professional 1).
As for the conflict and crisis scenario the inclusion of discordant points was suggested:
"We could include the possibility of the Leader obtaining from the team, alternatives in points where there is no consensus"
(professional 3).
For the negotiation scenario obtained the proposition:

"I would include in the text not only that the "client may not be happy with the deviation" but also, more explicitly, that the
client believed in the schedule and is already starting to have financial losses with this delay. This places on the interviewee a
need to search for "arguments" more connected to the reality and practice of the business and how he would face the situation"
(professional 3).
With the analysis of the results of the ten scenarios developed from the professionals' point of view the main suggestions
were analyzed and included for the pilot studies. That said, the second part of this study, focused on the qualitative analysis
of the pilot studies from the students' point of view. The objective in assessing these scenarios was to know the perceptions
of the participants about the understanding, time estimation (reading and response), advantages and limitations identified
in the scenarios and in the assessment of competences by scenarios.
Regarding the understanding/comprehension of the ten scenarios developed, all participants affirmed that there were
no doubts. Furthermore, the data indicate that 100% affirmed that the time for reading the scenario, estimated at 2
minutes, was adequate. With regard to the time for answering the scenario estimated at 8 minutes, all respondents
indicated that perhaps the time was too long and justified their response.
"8 minutes is a very good time to discuss, but maybe 5 minutes is enough" (student 1).
"In my opinion, a response time to the scenarios of 8 minutes is high since it is only one person talking, discussing about
the best decision to make in a specific situation" (student 2).
"Even with the additional comments and discussion with fellow observers, I think 8 minutes was too long for th e scenarios"
(student 3).
When asked about the main positive aspects of the experience of an assessment using scenarios, the results indicate
that the respondents perceive positive points of an assessment by scenarios, allowing them to explain their arguments
in an open and creative way in specific contexts of the area. Furthermore, they affirmed that the assessment that
includes scenarios becomes diversified and representative and can be an important instrument for managers to present
their competences. Below are some of the participants' descriptive answers.
"It is an experience that allows candidates to explain their reasoning openly and allows for a very diverse and representativ e
assessment" (student 1).
"With the use of scenarios, it is possible for an evaluator to see, in a practical context, what a candidate's action would be
in various situations, which is an advantage compared to the assessment only by competences mentioned in a curriculum,
for example, and not effectively verified" (student 3).
When questioned with the difficulties felt in this assessment experience through the use of scenarios, two students
directed to the lack of practical experience in the area, leading to difficulties in the argumentation in the answer.
"Some lack of knowledge on how to deal with the content of some scenario. From the functional point of view of the
method I had no difficulties" (student 1).
"For me, the biggest difficulty in this experience was to put myself in the position of the project managers of the s cenarios
in which I was allocated and make the best decision. Another difficulty was in the argumentation of my decisions" (student
2).
"The scarce experience in the labor market limited the answers to the scenarios to the use of teamwork experience only i n
the academic context" (student 3).
Finally, participants described in their own words their participation in the study. The responses indicate an enriching
and valuable experience. Unanimously, they enjoyed and felt comfortable in assessment of this type . Below are some
of the responses.
179
"I consider this a very enriching experience in that it allows candidates to try to develop their soft competences, such as
thinking about various possible situations. From the assessment point of view, I think this is a very complete method, which
allows us to make a very representative assessment of the candidates and has a lot of potential to be applied in several
areas. I didn't know about it, and I really enjoyed the experience" (student 1).
"I liked participating in this experience, I think it was well thought out and structured. The different scenarios were
interesting and their contents allowed for engaging discussions" (student 2).
"I found it a very interesting experience, which I had never been exposed to before and it seems like an excellent way to
also prepare even students for the job market/job interviews" (student 3).
5. Discussion
In this study, it can be verified that the assessment of competences using scenarios (problem situations inspired by real
situations of professional practice) was evidenced by the participants as an excellent possibility for the assessment of
competences. The suggestions for improvement from the professionals' point of view sought to align the scenarios
developed with the real situations of the professional practice of project management and also the perception of the
demonstration of the competence
Several authors identified benefits of scenarios in assessment processes (Klassen et al., 2021; Lovell & Khatri, 2021; Naidu,
2010; Seddon, McDonald & Schmidt, 2012): increased motivation and interest, engaging process and learning and more
effective assessment (Ribchester & Healey, 2019) emphasize the realistic factor, encourage reflection and promote
individual responsibility.
In this study, some of the benefits were evidenced. Benefits such as a dynamic and interactive, diversified and representative
assessment, with the possibility of explaining the reasoning in an open manner and also aligned with professional practices
of project management made the assessment approach by scenarios a possibility to assess people competences, since the
assessed could mobilize different resources in solving a problem situation that is presented by the scenario.
Still, the assessment of competences by scenarios can be useful for accreditation, recruitment and team building processes,
as it allows mobilizing different types of competences in simulated situations close to professional practice, diversifying the
competence assessment processes and allowing the assessed a more flexible and real situation.
6. Conclusion
The objective of the study was to develop project management scenarios for assessing people competences. The
importance of assessing these competences in the Project Management area is undeniable, and in this sense, assessment
through scenarios becomes an excellent way to prepare professionals effectively for their practice.
The main results of the study indicate that the ten scenarios developed and assessed present benefits and provide the
participant with the mobilization of resources, competences and experience to solve the problem to which he/she was
exposed. In addition, the statistical analysis show high level of agreement from the point of view of professionals regarding
the representation of project management practices, but show allow the need for review of some of the scenarios before
the application in assessment processes.
As future work, scenarios will be improved, and instruments to support the assessment process will also be developed, i.e.
rubrics for competence assessment will be developed for all people competences scenarios. With the scenarios and rubrics
for competence assessment, a model for assessing Project Management people competences will be developed tested.
Acknowledgments
This work was supported by the Federal University of Rio Grande (FURG), and by FCT – Fundação para a Ciência e Tecnologia
within the R&D Units Project Scope: UIDB/00319/2020.
References
Alam, M., Gale, A., Brown, M., & Kidd, C. (2008). The development and delivery of an industry led project management professional
development programme: A case study in project management education and success management. International Journal of
Project Management, 26(3), 223-237.
Banerjee, H. T. L. (2019). Investigating the construct of topical knowledge in a scenario-based assessment designed to simulate real-life second
language use. Teachers College, Columbia University.
180
Crawford, L. (2005). Senior management perceptions of project management competence. International journal of project management,
23(1), 7-16.
Daniel, S., & Mazzurco, A. (2020). Development of a scenario-based instrument to assess co-design expertise in humanitarian
engineering. European Journal of Engineering Education, 45(5), 654-674.
Erol, S., Jäger, A., Hold, P., Ott, K., & Sihn, W. (2016). Tangible Industry 4.0: a scenario-based approach to learning for the future of
production. Procedia CiRp, 54, 13-18.
Errington, E. P. (2010). Preparing graduates for the professions: achieving employability through the exploration of near-world
scenarios. International Journal of Interdisciplinary Social Sciences, 5, 1-10.
Errington, E. P. (2011). Mission possible: Using near-world scenarios to prepare graduates for the professions. International Journal of
Teaching and Learning in Higher Education, 23, 84-91.
Hagler, D., & Wilson, R. (2013). Designing nursing staff competency assessment using simulation. Journal of Radiology Nursing, 32(4), 165-
169.
Ilahi, M., Cheniti-Belcadhi, L., & Braham, R. (2014, July). Scenario model for competence-based assessment. In 2014 IEEE 14th International
Conference on Advanced Learning Technologies (pp. 344-346). IEEE.
IPMA. (2015). Referencial de Competências Individuais para Gestão de Projetos, Programas e Portefólios. International Project
Management Association.
Klassen, R. M., Rushby, J. V., Maxwell, L., Durksen, T. L., Sheridan, L., & Bardach, L. (2021). The development and testing of an online scenario-
based learning activity to prepare preservice teachers for teaching placements. Teaching and Teacher Education, 104, 103385.
Kerzner, H. R. (2006). Project Management: With Project Management Case Studies, 2r. e. John Wiley & Sons.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of
chiropractic medicine, 15(2), 155-163.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. biometrics, 159-174.
López-Alcarria, A., Olivares-Vicente, A., & Poza-Vilches, F. (2019). A systematic review of the use of agile methodologies in education to
foster sustainability competences. Sustainability, 11(10), 2915.
Lovell, D., & Khatri, C. (2021). Do Early Simulations Work? Simulations in Gateway Political Science Courses at Community Colleges. Journal
of Political Science Education, 17(1), 139-148.
Miguel, A. Modern Project Management: Best Techniques and Practices. 8th Edition. FCA.
Naidu, S. (2010). Using scenario-based learning to promote situated learning and develop professional knowledge. In Preparing graduates
for the professions using scenario-based learning (pp. 39-49): Post Pressed.
O'Brien, J. E., Hagler, D., & Thompson, M. S. (2015). Designing simulation scenarios to support performance assessment validity. The Journal
of Continuing Education in Nursing, 46(11), 492-498.
PMI (2017). A guide to the project management body of knowledge (PMBOK® Guide) (Sixth edit). Project Management Institute, Inc.
Ribchester, C., & Healey, R. L. (2019). Realism, reflection and responsibility: The challenge of writing effective scenarios to support the
development of ethical thinking skills. Journal of Further and Higher Education, 43(1), 101-114.
Rosson, M. B., & Carroll, J. M. (2009). Scenario based design. Human‐computer interaction. boca raton, FL, 145-162.
Seddon, J. M., McDonald, B., & Schmidt, A. L. (2012). ICT-supported, scenario-based learning in preclinical veterinary science education:
Quantifying learning outcomes and facilitating the novice-expert transition. Australasian Journal of Educational Technology, 28(2).
Vukomanović, M., Young, M., & Huynink, S. (2016). IPMA ICB 4.0—A global standard for project, programme and portfolio management
competences. International journal of project management, 34(8), 1703-1705.
Wawak, S., & Woźniak, K. (2020). Evolution of project management studies in the XXI century. International Journal of Managing Projects
in Business, 13(4), 867-888.
Waxman, K. T., & Telles, C. L. (2009). The use of Benner's framework in high-fidelity simulation faculty development: The Bay Area Simulation
Collaborative model. Clinical Simulation in Nursing, 5(6), e231-e235.
181
IEM@ProjectNetworking revisited: freshmen students closer to
professional practice
Anabela C. Alves1, Sérgio Oliveira2, Celina P. Leão1, Sandra Fernandes3,4
1
2
ALGORITMI Centre, Department of Information Systems, School of Engineering, University of Minho, Guimarães, Portugal
3
Portucalense Institute for Human Development, Portucalense University, Porto, Portugal
4
Research Centre on Child Studies – CIEC, University of Minho, Portugal
Email: {anabela, cpl}@dps.uminho.pt, [email protected]; [email protected]
Abstract
This paper describes a challenge given to freshman students of the Integrated Master degree of Industrial Engineering and
Management (IEM) – University of Minho, Portugal. It was developed in the context of the Integrated Project of Industrial
Engineering and Management 1 (IPIEM1). First-year students were challenged to interview an Industrial Engineer, or
someone with this function, in a company. That was the second time the challenge was proposed to students of this
program, after the first time in 2013. The challenge, called “IEM@ProjectNetworking”, has as main idea to allow students,
on their own, to establish network with an IEM active professional and to get a full picture of what could be their future
career. Students worked in pairs and had a semi-structured interview guide to help them to perform the interview. All
interviews were synthetized in a written report and in a summary presented orally. Thirty-three interviews were conducted
and assessed by a group of teachers and researchers. Teachers believe that by presenting this challenge, students will be
more focused and aware of the contents given in the courses of the following semesters. At the same time, teachers obtain
a perspective of what IEM professionals are doing and update their knowledge about industry needs. In fact, many findings
were obtained with this challenge, namely: 1) a company database and the connection to IEM professionals for their future
contact; 2) the stimulus of students’ capacity of initiative; 3) the students learned how to prepare and conduct an interview,
to communicate, to extract the most important from an interview; and; 4) the students acquire knowledge where they can
work and the variety of functions which they may be engaged. The assessment was based on how the task was organised
and accomplished (written report and oral presentation in pitch format). Based on the analysis of results obtained from an
online survey applied to students and their feedback during the oral presentation, it is possible to state that the challenge
has been mastered perfectly, with a “mission accomplished” feeling.
Keywords: Industrial Engineering and Management Education; Active Learning; Networking, Professional profile,
Transversal competencies.
1 Introduction
Normally, first year students (freshman) of Higher Education Institutions do not know much about their future
professional career. This is generally the case in Engineering Education programs but it became more visible in
Industrial Engineering and Management (IEM) programs, as this engineering has been evolved from different
sources (Pimentel et al., 2022). Also, in the future they could perform so many different functions by taking
different roles in the companies, from production systems designers to health and safety experts.
The Institute of Industrial and Systems Engineers (IISE) defined Industrial Engineering as: “… concerned with the
design, improvement and installation of integrated systems of people, materials, information, equipment and
energy. It draws upon specialized knowledge and skill in the mathematical, physical, and social sciences together
with the principles and methods of engineering analysis and design, to specify, predict, and evaluate the results
to be obtained from such systems.” (IISE, 2022).
The IISE is composed of 14 knowledge areas: 1) Work design and measurement; 2) Operations Research &
Analysis; 3) Engineering Economic Analysis; 4) Facilities Engineering & Energy Management; 5) Quality &
Reliability Engineering; 6) Ergonomics & Human factors; 7) Operations Engineering & Management; 8) Supply
Chain Management; 9) Engineering Management; 10) Safety; 11) Information Engineering; 12) Design &
Manufacturing Engineering; 13) Product Design & Development and, finally; 14) System Design & Engineering.
It is expected that IEM students acquire knowledge and act on these areas in their later future professional in
182
any type of company or organization, producing goods, services, or both. Furthermore, they must develop
competencies that go beyond the knowledge, including the skills and attitudes. This means that skills such as
communication, problem solving, critical thinking, teamwork, leadership, capacity of initiative, among others,
must be part of their learning.
With so many areas, it is not a surprise that IEM students do not know what will be their role in the companies.
Moreover, they need to understand the meaning of learning in such knowledge areas that are included in their
curricular plans. Attending to this, a challenge was launched to the IEM first-year, first-semester students called
IEM@ProjectNetworking. This activity was developed in the context of Project-Based Learning (PBL) as a
milestone to accomplish in the Integrated Project of Industrial Engineering and Management course. The
students organized themselves in pairs and had to conduct one interview with an Industrial Engineer or
someone that had this role in a company located in the region. The objective of this paper is to present how
the challenge was launched, planned and executed and to analyse and evaluate the results obtained by
students.
The paper is structured in five sections. After this introduction, the study context and previous work are
presented in section two. Section three presents the research methodology. Section four presents the results
obtained by students and the main findings of the authors about this challenge. Main conclusions are
presented in section five.
2 Study context and previous work

The PBL learning methodology is being used by Industrial Engineering and Management (IEM) program of the
School of Engineering of the University of Minho (UMinho) since 2004 (Alves et al., 2020; Lima et al., 2007).
From a formal structure without a project course (Alves et al., 2014) to the current one, a long path of 19
editions was carried out (Alves et al., 2021). Along this path, a lot of research and continuous improvement has
been done (Alves et al., 2017; Alves & Leão, 2015). This included the challenge of IEM@ProjectNetworking that
was developed, for the first time, in the context of the course of Introduction (or Topics) of Industrial
Engineering and Management (Alves et al., 2013). The setting for the challenge of this academic year was
different and it is explained next.
Imposed by government, this academic year of 2021_22 implied one more change: from a Master Integrated
of five years (300 ECTS) it becomes a Bachelor of three years (180 ECTS). The setting up of this paper is based
on the 2021-2022 curricular structure conducted at the UMinho, as represented in Figure 1. From Figure 2 it is
possible to see that the six courses that are integrated in the curricular plan are from different schools and
departments, having a Science, Technology, Engineering and Mathematics (STEM) structure.
Figure 1. Curricular plan of IEM first-year, first semester, IEM11

Figure 2. Schools and departments involved
in the IEM11
The 67 students of this cohort were organized in eight teams of seven-nine members that need to develop a
project in the context of the course Integrated Project in Industrial Engineering and Management I (IPIEM1),
applying the contents learned in the five different courses. Each course has their own assessment methodology
that includes a project component. Nevertheless, the contents of each course included in the IPIEM1 is assessed
183
in the presentations, reports, blog, prototypes, that the teams delivered to be assessed. Figure 3 shows the
milestones of the IPIEM1 and the weight of each deliverable in the assessment methodology. Individual grade
of each student is the result of (a) team grade influenced by a correction factor that result from the peer
assessment (Fernandes et al., 2009; Fernandes et al., 2020; Uebe-Mansur & Alves, 2018) (90%) and (b) the grade
obtained to the IEM@ProjectNetworking.
Ms Week Deliverable Weight (%)
Nota do Aluno à UC X
1 Week 2 1st presentation not assessed Individual student grade
(exceto PIEGI1)
2 Week 6 2nd presentation 7,5 (team) Avaliação efetuada

docentes da UC X, de acordo
pelos
90% 10%
3 IEM@ProjectNetworking presentation 10 (individual)
com a metodologia definida
Week 8 (discussão, testes, trabalho em
equipa, envolvimento...). Esta
4 Week 11 Preliminary report 25 (team) Individual grade of each IEM@Project avaliação também poderá conter
student networking elementos oriundos do projeto.
(b)
5 Week 13 Final report + Prototypes + blog 30 + 20 + 5 (team) Correction factor
6 Week 14 Final presentation & discussion 12,5 (team) (a)
Figure 3. Milestones (Ms) and weight of each in the assessment Figure 4. Individual grade of each student
Team grade
20+5% 30% 20% 25%
3 Research methodology Preliminary

Final report Presentations
Prototypes &
The research methodology used by the paper authors to analyse the IEM@ProjectNetworking developed by
report blog
the students was a content analysis of the interview transcriptions that were described in a report and personal
(a) Atribuído ao aluno por hetero-avaliação (média dentro do grupo igual a 1.0)
(b) Avaliação efetuada pela coordenadora e outros
(c) Avaliação efetuada por todos os docentes envolvidos no projeto
reflections held by students in the oral presentation. Also, two questions related with this challenge were
(d) Avaliação efetuada pelos docentes de IEGI, AP e por outras UCs
(e) Avaliação pela equipa de coordenação à capacidade de avaliação dos
included in the survey that had been used to evaluate the PBL process every year (Alves et al., 2020). The non-
grupos
parametric test Mann-Whitney (U) for comparison between the ranks of two independent samples, after
verification of normality by Shapiro-Wilk test. The statistical software SPSS 28.0 was used for the analysis, and
the effect is considered statistically significant for a p-value<α (the significance level).The process of data
collection involved 67 students, who mostly worked in pairs, with 33 interviews conducted in a total of 37
companies.
The research involved the four authors of this paper, two of them teachers: one in the course of Introduction
(or Topics) of IEM and simultaneously the coordinator of IPIEM1 and the other is coordinator of the course of
Computer Programming 1. The other two research members had different roles in the project: one is tutor of
a student’s team and the other is an educational researcher. The four assessed different criteria defined for this
challenge. The criteria considered were: 1) accomplishment of the interview guide topics; 2) reflection about
the interview process; 3) auto-reflection; 4) accomplishment of the topic related to the information system; 5)
assessment of the interview contents and finally, 6) accomplishment of the delivery conditions (deadline, all
documents and audio/video delivered).
As defined by the criteria, the data considered in the findings focused on the reflections written by students
concerning the overall process (before, during and after) of the experience of interviewing an IEM professional
in their field, more than just simply describing the content of the interview. The students also discussed the
main benefits and difficulties related to this experience. The assessment was based on how the task was
organised and accomplished (written report and oral presentation in pitch format).
4 IEM@ProjectNetworking
This section describes the purpose of the IEM@ProjectNetworking challenge, the interview guide used and the
results obtained.
4.1 Purpose
In the first stage, the challenge was presented to students with the main goal of explaining the purpose and
what was expected from them. The main purpose was presented as:
• To develop proactive attitude;
• To develop awareness of the importance of entrepreneurial attitude.
Nevertheless, objectives also included:
184
• To bring students closer to their future professional practice;
• To develop a proactive and entrepreneurial attitude, student’s initiative and communication
competences;
• To develop a proactive attitude towards their own future as industrial engineering professionals;
• To develop awareness among the students on the importance of perseverance and proactive attitude;
• To create a network of contacts of IEM professionals and potential employers;
• To get familiar with IEM professionals point of view as well as with their role in the organizations.
4.2 Interview guide

To collect information from the IEM professionals, the interview was considered an appropriate method to give
students a structured tool when approaching the companies. A semi-structured interview guide, adapted from
an existing guide, included eight main sections:
1) Introduction, to inform about the interview objectives, highlight the importance of the collaboration,
assure the confidentiality and ask for authorization to record the interview;
2) Interviewee profile, in order to know the academic and professional background;
3) Transition to workplace, in order to know the difficulties and expectations related to this issue;
4) Activities/functions held in the company, in order to understand what an industrial engineer is able to
do;
5) Importance and recognition of the IEM profession, in order to analyse the employability issues, the
advantages and disadvantages related to being an industrial engineer;
6) Personal satisfaction and professional achievement, in order to understand how professionals manage
their time between personal and professional life;
7) Production Information system, to know more about the software used by the company, information
needs, production management tasks supported by the information system, responsibilities of this task
and access privileges;
8) Final observations, to give opportunity to the interviewees to add more information and acknowledge
the availability.
The semi-structured interview guide document includes also some recommendations and tips of how to
conduct an interview and some relevant literature regarding this topic. With this documentation in their hands,
students had to select a company and manage to schedule an interview with an IEM professional with a relevant
position in terms of production management. At the scheduled moment, students, in pairs, had to perform the
interview, eventually make a small tour in the company and collect as much relevant data/information as
possible. Finally, each student pair had to write a report presenting the interview results as well as a personal
reflection on their experience of carrying out the interview and present the results in a five-minute oral
presentation.
4.3 Results
As expected, this was not an easy task, not for the students neither for the authors that had to collect and
assess 33 interviews, including the reports and audio/videos presented. For the authors, this task involved a
high consuming-time, not considered in the workload defined for the project, that, normally, implies a high
workload (Alves et al., 2009; Alves et al., 2019). Additionally, the content analysis of the report could conduct
to different interpretations and unfair assessment between students grade but this is inherent to all other
assessments(Fernandes et al., 2012, 2021) . Nevertheless, it is of general opinion that should be repeated every
year, since it was assumed as teachers team task (Alves et al., 2021; van Hattum-Janssen et al., 2022). The
students organized themselves to conduct the interviews, some online due to pandemic restrictions, others
had the opportunity to visit the company and conduct the interview face-to-face. Difficulties reported by the
students were, mainly, related with the time management, as they also complain about the time they dedicated
to the IPIEM1. The presentation of the Project Networking took place in an online format by zoom (Figure 5).
185
Figure 5. Some snapshots of the zoom session
Main findings from the report and reflections were grouped in four main topics that will be discussed in the
following sections: 1) a company database and the connection to IEM professionals for their future contact.; 2)
the stimulus of students’ capacity of initiative; 3) the students learned how to prepare and do an interview, to
communicate, to extract the most important from an interview; 4) the students acquire knowledge where they
can work and type of functions they could perform. After the assessment all students obtained a grade higher
than 16 (of 20) values.
The two sentences included in the PBL questionnaire that were related with this challenge, and evaluated on a
scale from 1 (strongly disagree) to 5 (strongly agree), were:
S1: “The IEM@ProjectNetworking project activity helped me to better understand my professional future.”
S2: “I think the weight of the IEM@ProjectNetworking activity (10%) is adequate.”
Based on the number of enrolled students, 72% (48 out of 67) voluntarily completed the questionnaire. The
main descriptive statistics measures obtained are presented in Table 1, and the students’ evaluation distribution
for both sentences S1 and S2, by gender, illustrated in Figure 6.
Table 1. Main descriptive statistics obtained for S1 and S2 evaluation (F – female, M – male students).
Descriptive
S1 S2
Statistics
F 33 34
n M 14 14
Total 47 48
F 4.88  .33 4.24  .82
mean  s.d. M 4.57  .65 4.21  .80
Total 4.79  .46 4.22  ..81
F 5 4
median M 5 4
Total 5 4
F 4 2
min M 3 3
Total 3 2
F 5 5
max M 5 5
Total 5 5
186
(a) (b)
Figure 6. Students’ evaluation for sentence (a) S1 and (b) S2, by gender.
Both sentences, S1 and S2, received a positive evaluation, with mean values higher than 4: 4.79 and 4.22,
respectively (5 corresponds to strongly agree), with the highest value 5 and the lowest 3. Although slightly
differences were observed between the evaluation distribution by gender, namely in sentence S1 with male
students with slightly lower values, they are not statistically significance (U=174.5, p>0.05 for S1, and U=232.0,
p>0.05, for S2). So, students in average considered that the IEM@ProjectNetworking project activity helped to
better understand the future professional career.
4.3.1 Database of companies and networking expansion
As referred, 33 interviewes were conducted by students in groups of two, in a total of 37 companies (some
students decided to conducted more than one interview and four interviewees were from the same two
companies). These companies included international and national companies, small, medium-sized companies
(SME) and large companies. Companies were from different sectors: textile, shoes, software development,
eletronic components, comercial organizations, cooperatives among others. The interviewees belong to the
following list of the companies:
1) PROEF 13) Marley Spoon 24) Tiajo
2) Farfetch 14) BOSCH 25) Leiper
3) BSD - The Best Solution 15) FORteams LAB 26) Paul Stricker
Design 16) Gewiss 27) Continental
4) Fermir 17) Grupo M 28) BorgWarner
5) ACCO Brands Portuguesa 18) Textil Manuel Gonçalves Sa- 29) Amkor
6) Bysteel (DST group) Automotive 30) Off Spin
7) Jordão Cooling Systems 19) AICEP 31) Valerius têxteis SA
8) BytePitch 20) Custódio Castro Lobo 32) Caixindu
9) Aldo Shoes 21) WIM 33) LASA
10) Bysteel (DST group) 22) Cooperativa Agrícola de 34) GKN
11) GreenPelts Tannery Lda Boticas 35) Texteis Penedo
12) Leica 23) AMOB
All addresses and emails from the companies and interviewees were collected for future contacts. This
interaction was also positive to strengthen the connection between the companies and the academy, as
students played the role of academic ambassadors, fostering opportunities for future collaborations in terms
of research and development activities. This networking with the companies also allow to the teachers to know
a little more about perspectives of employers regarding the Industrial Engineers competencies.
It is worth emphasising that the process of contacting the company was the responsibility of the students.
Some through a family member who knew an engineer and suggested contact or through conversations
between them (students). Some of the interviewees are former students of the university and of the same IEM
degree.
4.3.2 Stimulus of students’ capacity of initiative

Related to the stimulus of students’ capacity of initiative, all students developed this capacity. They were
capable of contacting an engineer through their closer contacts (family, friends, among others) or someone
187
they knew and heard in university events or explore other possibilities. It is important to highlight that some
pairs described the interviews in the team blog. Figure 7 presents some snapshots of two team blogs, where
they described the whole process of conducting the interview .
Figure 7. Extract of two teams blogs snapshots of the interviews´ description
Besides a proactive attitude, students developed other transversal competences such as communication
competences and organizational behaviour awareness. Although students had an interview guide with
questions to support their interview, students also had to deal with unforeseen situations, which led them also
to develop the competence of adaptation to new situations, problem solving and resilience in order to proceed
successfully with their objectives for this task.
4.3.3 Knowledge to do and interpret an interview

All students followed the semi-structured interview guide and summarized the interviews conducted very well.
This was verified by the authors of the paper after hearing and seeing the videos. Also, students were capable
of synthesizing the most important contents and recall the interviewees’ advices during the oral presentation,
for example: “Do not forget that you work with the people and for the people. It is not forcing processes that will
change anything, it is with management by example”.
4.3.4 Awareness of their professional practice

Through their reports, students realized what it means to be an Industrial Engineer, an idea for an IEM
profession and how diverse and broad it could be. All knowledge areas discussed in the introduction were
exposed by the interviewees. It was a positive discovery for students, who were amazed by all the professional
areas in which they can enrol in their future.
This challenge was even more relevant for students to realize how important the information systems are for
modern organizations. Industrial Engineers, who frequently assume the leadership of industrial departments,
and even top management positions in the case of industrial SMEs, are often called to specify user
requirements related to the need for the implementation of new strategic business evolutions in their
organizations’ information systems. The students’ answers provided for the point 7 (Production Information
system) of the interview guide were assessed by the teacher of the Computer Programming I course who
established a set of user requirements to be fulfilled by the information system and software to implement in
the enterprises of their projects of IPIEM1. In comparison to the two previous editions of PBL (2019 and 2020),
the task related to Computer Programming I course, in the present year, was accomplished with more pertinent
and valuable content, which confirm the positive contribution of the interviews undertaken by the students in
the fulfilment of the IEM@ProjectNetworking challenge.
5 Conclusion
This paper presents the main findings of the challenge given to freshman IEM students. According to them,
this experience was very rich and useful, allowing them to have a better look at their future professional career.
Based on students’ feedback during and after the oral presentation and also on the analysis of results obtained
from part of the PBL questionnaire, it was possible to state that, despite some difficulties, as mentioned in
section 4.3, it was a worthwhile challenge. Better time management and teamwork of teachers and students
continues being the best solution to overcome the difficulties. This is also evidenced by students’ good
performance, comments and reflections in the IEM@ProjectNetworking report. It was also clear that students
enjoyed the activity very much and listened carefully to the advice from the interviewees and, through the
experiences lived in the first-person, students easily learned some lessons.
188
This experience is also relevant for first-year students as they can achieve greater confidence about their
professional future, at an early stage of their academic degree. The experience is significant, useful, and
impacting for students. Along with the PBL approach, this experience brings forward the meaning of “learning
by doing” in the sense that students are actively engaged in the process of their own learning and development.
Acknowledgments
This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project
Scope: UIDB/00319/2020. Also, the authors would like to express their acknowledgement to all interviewers
and companies who agreed to collaborate in this study.
References
Alves, A. C., Moreira, F., Lima, R. M., Sousa, R, M., Carvalho, D., Mesquita, D., Fernandes, S., & Van-Hattum-Janssen, N. (2014). Aprendizagem
Baseada em Projetos interdisciplinares: análise da implementação em duas estruturas curriculares distintas. In A. Fischer & O. L.
de O. M. Heinig (Eds.), Linguagens em uso nas Engenharias (Edifurb-). FURB.
Alves, A. C., Moreira, F., Sousa, R., & Lima, R. M. (2009). Teachers’ workload in a project-led engineering education approach. International
Symposium on Innovation and Assessment of Engineering Curricula, 14.
Alves, A., Carvalho, J. D., Mesquita, D., Fernandes, S., & Lima, R. M. (2013). IEM@ProjectNetworking: bringing first year students closer to
professional practice. International Symposium on Project Approaches in Engineering Education (PAEE2013), 1–7.
Alves, A.C., & Leão, C. P. (2015). Action, practice and research in project based learning in an industrial engineering and management
program. ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE), 5–2015.
https://doi.org/10.1115/IMECE2015-51438
Alves, A.C., Moreira, F., Fernandes, S., Leão, C. P., & Sousa, R. (2017). PBL in the first year of an industrial engineering and management
program: A journey of continuous improvement. International Symposium on Project Approaches in Engineering Education, 9, 44–
51.
Alves, A.C., Moreira, F., Leão, C. P., Pereira, A. C., Pereira-Lima, S. M. M. A., Malheiro, M. T., Lopes, S. O., & Oliveira, S. (2019). Industrial
engineering and management PBL implementation: An effortless experience? In A. K. Lima R.M., Villas-Boas V., Bettaieb L. (Ed.),
11th International Symposium on Project Approaches in Engineering and 16th Active Learning in Engineering Education Workshop,
PAEE/ALE 2019; Hammamet; Tunisia; 10 June 2019 through 12 June 2019; (Vol. 9, pp. 117–127). University of Minho.
Alves, Anabela C., Moreira, F., Leão, C. P., & Fernandes, S. (2020, November 16). Ten Years of Positive Feedback on Project-Based Learning
From First-Year Engineering Students’ Perspective. Volume 9: Engineering Education. https://doi.org/10.1115/IMECE2020-23212
Alves, Anabela C., van Hattum-Janssen, N., & Fernandes, S. (2021). Teacher collaboration in PBL: setting the example for the students.
PAEE/ALE’2021, International Conference on Active Learning in Engineering Education, 13rd International Symposium on Project
Approaches in Engineering Education (PAEE) and 18th Active Learning in Engineering Education Workshop (ALE), Braga, Portugal, 6–
14. https://doi.org/https://doi.org/10.5281/zenodo.5093791
Alves, Anabela Carvalho, Fernandes, S., Moreira, F., Lima, R. M. S. P., Carvalho, J. D. A., Sousa, R. M. A. S., Mesquita, D. I. A., & Hattum-
Janssen, N. van. (2021). Project-Based Learning: implementação no primeiro ano de um curso de Engenharia. UMinho Editora.
https://doi.org/10.21814/uminho.ed.26
Fernandes, S., Mesquita, D., & Lima, R. et al. (2009). The impact of peer assessment on teamwork and student evaluation. Innovation and
Assessment of Engineering Curricula, 125–136.
Fernandes, Sandra, Alves, A. C., & Uebe-Mansur, A. (2021). Student-Centered Assessment Practices: An Integrated Approach With Project-
Based Learning (PBL). In A. S. Moura, P. Reis, & M. N. D. S. Cordeiro (Eds.), Handbook of Research on Determining the Reliability of
Online Assessment and Distance Learning (pp. 213–243). IGI Global. https://doi.org/10.4018/978-1-7998-4769-4.ch009
Fernandes, Sandra, Alves, A. C., & Uebe-Mansur, A. F. (2020). Student-Centered Assessment Practices: An Integrated Approach With
Project-Based Learning (PBL). In Determining the Reliability of Online Assessment and Distance Learning (p. 340). IGI Global.
Fernandes, Sandra, Flores, M. A., & Lima, R. M. (2012). Students’ views of assessment in project-led engineering education: findings from
a case study in Portugal. Assessment & Evaluation in Higher Education, 37(2), 163–178.
https://doi.org/10.1080/02602938.2010.515015
IISE. (2022). IISE Body of Knowledge. Body of Knowledge. https://www.iise.org/Details.aspx?id=43631
Lima, R. M., Carvalho, D., Assunção Flores, M., & Van Hattum-Janssen, N. (2007). A case study on project led education in engineering:
students’ and teachers’ perceptions. European Journal of Engineering Education, 32(3), 337–347.
https://doi.org/10.1080/03043790701278599
Pimentel, C. M. O., Alves, A. C., Matias, J. C. O., & Azevedo, S. G. (2022). The Challenges of Industrial Engineer Management Skills in Industry
4.0. In A. C. Alves & N. van Hattum-Janssen (Eds.), Training Engineering Students for Modern Technological Advancement (pp. 225–
249). IGI Global. https://doi.org/10.4018/978-1-7998-8816-1.ch011
Uebe-Mansur, A. F., & Alves, A. C. (2018). The importance of peer assessment & self-assessment in PBL applied to an Administration course.
Revista Ibero-Americana de Estudos Em Educação, 13(esp1), 451–467. https://doi.org/10.21723/riaee.nesp1.v13.2018.10347
van Hattum-Janssen, N., Alves, A. C., & Fernandes, S. R. G. (2022). Team Teaching in PBL: a literature review in Engineering Education. In A.
C. . Alves & N. van Hattum-Janssen (Eds.), Training Engineering Students for Modern Technological Advancement (pp. 250–270). IGI
Global. https://doi.org/10.4018/978-1-7998-8816-1.ch012
189
Learning Assessment in a "Theses Debate"
Rosane Aragón1, Crediné Silva de Menezes1, Alberto Nogueira de Castro Júnior2
1
Doctoral Program in Informatics Applied to Education, Federal University of Rio Grande do Sul, Porto Alegre - RS, Brazil
2
Institute of Computing, Federal University of Amazonas, Manaus – AM, Brazil
Abstract
In this scenario of constant expansion of digital information technologies, education saw significant opportunities to review
its transmissive practices, generally limited by physical spaces that restrict communications and by the preponderance of
analogic pedagogical resources. As an alternative to the use of this new scenario, we conceive a theoretical-methodological
framework for the proposition of new active methodologies, which we call Pedagogical Architectures. We define them as
structuring supports for learning, performed from the confluence of different components: strategy constructivist, digital
technology (software, internet, artificial intelligence, and others), and flexible design of time and space. Although we can
practice constructivism, at different levels of teaching, without the use of digital communication technologies, the
introduction of these opens new learning possibilities, facilitating the interaction of students with the environment, with
the objects of study, with their peers, and teachers. In addition, it favors the resignification of the notions of time and space.
With the internet, we interact with objects and people, synchronously or asynchronously, geographically distributed, on a
planetary scale. In this study, we experimented with the Pedagogical Architecture Theses Debate. We carried out the study,
with 22 students of a doctoral program in Informatics in Education, in a Brazilian public university. We aimed to promote
the knowledge construction cooperatively, with the support of a collection of testimonies on the topic Applications of
information technology in education. Based on the students' prior knowledge, we held a collective debate in small groups,
organized in successive stages of deepening and resignification. We realized the evaluation of learning through the analysis
of the textual productions of the individuals, considering different stages of the debate, based on a list of categories created
for this purpose. The results showed that the architecture reached its objective, supporting the reconstruction of knowledge.
Keywords: Active Learning; Pedagogical Architectures; Educational Technologies
1 Introduction
Use of active methodologies in the learning process emerged on the international scene in the first half of the
20th century, based on different theoretical contributions, including the works of Dewey (1930), Piaget (1953,
1971) and Vygotsky (1978). Some important elements are present in these contributions, of which we highlight:
a) the subject does not acquire knowledge from a passive reception of the information presented to him, it is
necessary to build new mental structures from the interaction with the world; b) social interaction is important
to share different perceptions with other subjects and, c) It is not enough to know the world as it is at the
moment, it is necessary to develop our intellect to produce innovations.
Regarding this last element, let's see what Jean Piaget says.
[...] The invention proper to the experimental spirit and to deduction only becomes rational if it is
regulated thanks to the norms of objectivity and logical coherence, and these norms only take on a
living value if they are applied to a constructive activity. [...] Piaget (1998)
The theoretical contributions presented above deal with the process of knowledge construction by individuals.
However, to migrate from transmissive teaching to a constructivist practice, we still need to develop adequate
pedagogical approaches. For the elaboration of constructivist proposals, we need to think about new
pedagogical resources, new ways of organizing the school space, and the availability of technological solutions
to favor interactions.
The organization of the school space is still restricted to physical spaces between four walls. However, the use
of personal digital equipment has become part of the school scenario, giving opportunity to student
production, using different media shared from geographically distributed workstations. Notably, in the context
190
of COVID-19 pandemic, school activities were carried out remotely, in all places where Internet access was
available. However, this was not the reality of many developing countries.
Design and production of tangible teaching materials is too expensive both for the cost of production and for
obsolescence, which has been attenuated using digital resources, allowing teachers to adopt different media
instead of more conventional and expensive ones. In addition, cooperative development of resources, including
methodologies, by a network of teachers and other professionals has also been carried out in an international
movement led by UNESCO (Elder, 2019).
However, there is an obstacle to be overcome for these new materials meet the principles of active learning: a
methodological basis. Our research group has been working in this direction and, in recent years, we have
developed a framework to support the design of new proposals for active methodologies, which we call
Pedagogical Architectures (Menezes and Aragón, 2018). In this article we present an experiment supported by
an active methodology called “Theses Debate”, built within this Framework. The results indicate that this
proposal can contribute to the construction of knowledge in an active and cooperative approach.
2 Pedagogical Architectures
According to genetic epistemology (Piaget, 1971), learning results from the restructuring of a subject's network
of meanings based on a process of rebalancing. In this sense, it is necessary for the subject to enter into
cognitive conflict, which can be understood as an impossibility of explaining a fact of reality with his current
knowledge. Rebalancing is then sought using the metacognitive operations of assimilation and
accommodation. Conflicts can occur from the subject's interactions with the objects of the field of study,
whether physical or conceptual, or through social interaction, which can be understood as a mediation process.
Mediation can be carried out through interaction between students, called distributed mediation, or with the
participation of teachers (Aragón, 2016). In view of our intention to make use of digital technologies, notably
communication networks and artificial intelligence, the framework we propose already considers the
possibilities of communication and mediation made possible by these two elements.
In line with these principles, we define a Pedagogical Architecture as a methodological framework articulated
by the following components: the domain of knowledge; educational goals; students' prior knowledge of a
certain domain; the problem-oriented interactionist dynamics; distributed pedagogical mediations; procedural
and cooperative assessment of learning and support of digital technology (Menezes, Castro and Aragón, 2020).
2.1 Theses Debate

The practice of debate constitutes a socialization process that serves the development of citizenship and
thinking (Parrat-Dayan, 2007). To support this process in education, one of the alternatives is to act in the
perspective of promoting the exchange of points of view, which calls on students to position themselves in
relation to others, but, at the same time, remain open to others views. The Theses Debate architecture is part
of this perspective, constituting itself as a structuring framework for the construction of knowledge.
In this activity, the construction of knowledge is triggered by a set of statements, raised with the students
and/or collected from the paradidactic literature. From that point on, we can ask the participants to take a
stand, whether they agree or not, followed by an argumentative justification. Subsequently, the arguers assume
the role of reviewers and each one must analyse the arguments of one or more of their peers. This review does
not seek to oppose the participants' conceptions, but to present problematizations so that each one can
express their position more clearly. Once the reviews are completed, the participants return to their arguments
and present replies to their reviewers, responding to the problems received and providing clarifications on
points raised by the reviewers. Now, based on the reviews received from their reviewers, on the arguments and
replies of their reviewers, each participant is invited to review their position and rewrite their arguments.
191
STAGE 1 STAGE 2 STAGE 3 STAGE 4
Initial Final
Positioning Reply Positioning
Review
+ +
Initial Final
Argument Argument
Figure 1. Development Flux in Theses Debate
3 The experiment
We have continuously used the Theses Debate pedagogical architecture in different disciplines, mainly in
postgraduate (MSc/PhD) courses, in the training of professionals for the use of digital technologies in
education. The experiment we present in this article deals with the application of this educational technology
during the first semester of 2020, in the context of the Subject “Cooperative Learning in Digital Context”. The
choice was made because it was an edition with a greater number of students, which provides a greater
diversity of situations and because it is the first opportunity to use it during the COVID-19 pandemic.
Twenty-two students (15 women and 7 men) from a doctoral program in Informatics applied to Education
participated in that edition. The domain profile of the participants was: Education (5), Science Teaching (3),
Mathematics Teaching (2), Computing (4), Communication (1), Administration (1), Arts (2) and Educational
Technologies (4).
The discipline's principle is the development of its themes from active methodologies. Thus, to present the
Theses debate Architecture, discuss its foundations and analyse its contributions to the learning process, we
started with a debate on three theses covering “the use of digital technologies in education”. Three theses were
selected by participants from a list previously prepared by the teacher based on topics considered relevant in
the context of the use of digital technologies in educational activities, according to the list presented in Table
1.
Table 1: Theses used in the experiment
Thesis 1: Working in a collaborative environment is not the same of standardizing speech or

thinking, but rather to bring together points of view in favor of everyone.
Thesis 2: Using the internet, students can share results obtained in a problem to their
colleagues, in a process that shortens the resolution of the work.
Thesis 3: Digital games are addictive and distract students from learning; however, building
games mixed with math challenges can increase students' interest.
In agreement with the students, a schedule was prepared and provided 3 days for the first stage (position and
initial argumentation and 1 day for each of the four subsequent stages.
The computational support used was based on individual debate pages, one per participant, available on an
online site, where participants had online access. Each column on the page should only be edited in the time
interval defined in the schedule by the participants involved in the page.
192
Argumentator:
Reviewer 1:
Reviewer 2:
Stage 1 Stage 2 Stage 3 Stage 4
Thesis Initial Initial Review Reply Final Final
positioning Argument positioning Argument
Working in a collaborative environment is not synonymous R1
with standardizing speech or thinking but bringing together
points of view in favor of everyone.
R2
Using the internet, students can pass the results obtained in R1
a problem to their colleagues, in a process that shortens the
resolution of the work.
R2
Digital games are addictive and distract students from R1
learning; however, building games mixed with math
challenges can increase students' interest.
R2
Figure 2: Digital Debate Sheet
4 Argument Analysis
The analysis was carried out using a qualitative approach and focused on understanding the potential of the
Theses debate Pedagogical Architecture (AP) to promote cooperative learning.
The data collection allowed us to propose and proceed with the categorization of the arguments developed
by the 22 students, for each of the three theses, as well as observing the changes between the initial arguments
(step 1) and the arguments elaborated after the stages of review and reply.
To carry out the analysis of the arguments, we defined 10 categories based on the justifications of the students'
positions in relation to the three theses in the initial stage (categories C1 to C4) and in the final stage (categories
C5 to C10) of the Pedagogical Architecture. The categories are presented in Table 2.
Table 2: Categories of arguments
Initial Arguments
C1-Badly developed initial arguments, based on opinions and/or personal experiences (or that
only repeat the thesis);
C2- Arguments developed based on interpretations of concrete evidence;
C3- Arguments based on theoretical/methodological contributions that support them;
C4- Arguments that bring contradiction with the positioning.
Final Arguments
C5- Arguments that repeat the initial ideas, without enrichment;
C6- Arguments that consider the ideas expressed by the reviewers and bring enrichment;
C7-Arguments enriched with new elements, regardless of the ideas expressed by the reviewers;
C8-Arguments that support new positions considering the ideas expressed by the reviewers;
C9- Arguments that support new positions regardless of the ideas expressed by the reviewers;
C10- Arguments that bring contradiction with respect to the positioning.
193
The categorization of the arguments, considering the categories in Table 2, allowed the elaboration of Table 3,
which aims to present an overview of the arguments expressed by the subjects, considering each of the three
theses, in the different stages of the cooperative work “Thesis Debate”.
Table 3: Argument distribution by student category
Initial argument (categories Final arguments

C1-C4)
Students (Categories C5-C10)
Thesis 1 Thesis 2 Thesis 3 Thesis 1 Thesis 2 Thesis 3
S01 C1 C1 C1 C6 C6 C8
S02 C1 C1 C2 C6 C6 C5
S03 C3 C2 C3 C6 C6 C6
S04 C1 C1 C1 C8 C8 C8
S05 C1 C1 C1 C10 C6 C6
S06 C2 C2 C2 C5 C5 C5
S07 C2 C2 C1 C6 C8 C7
SC8 C2 C2 C3 C7 C6 C5
S09 C1 C4 C1 C6 C8 C9
S10 C2 C1 C2 C6 C6 C5
S11 C1 C1 C1 C8 C6 C8
S12 C4 C1 C2 C5 C5 C6
S13 C3 C2 C2 C6 C6 C6
S14 C3 C2 C2 C6 C6 C8
S15 C2 C2 C2 C6 C6 C6
S16 C2 C1 C1 C6 C6 C5
S17 C1 C2 C1 C6 C6 C8
S18 C1 C1 C1 C6 C6 C6
S19 C1 C1 C3 C6 C6 C7
S20 C1 C1 C1 C6 C6 C6
S21 C2 C3 C2 C5 C8 C6
S22 C1 C1 C2 C6 C6 C6
Based on Table 3, we prepared Tables 4 and 5, which allow us to observe the distribution of arguments by
categories, considering the three theses (Thesis 1, Thesis 2, and Thesis 3) adopted for the development of
Pedagogical Architecture. In each of the stages, 22 students argued justifying their positions in front of the
theses, resulting in a total of 66 categorized arguments in each of the stages.
194
Table 4: Category distribution of arguments by thesis (initial stage)
Initial arguments Thesis Thesis Thesis Totals

1 2 3
C1-Badly developed initial arguments, based on

opinions and/or personal experiences (or that only 11 12 10 33
repeat the thesis);
C2- Arguments developed based on interpretations of 7 8 9 24

concrete evidence;
C3- Arguments based on theoretical/methodological 3 1 3 7

contributions that support them;
C4- Arguments that bring contradiction with the 1 1 0 2

positioning.
Looking at Tables 4 and 5, we can say that most students (15 of the 22 students) started the AP, using poorly
developed arguments, based on opinions and/or personal experiences or that only repeated the thesis. In this
category (C1) it is possible to classify 33 justifications of positions, in at least one of the theses, by 15 students
(6 students argued this way for the three theses, 6 students for 2 theses and 3 students for 1 thesis.
Arguments based on interpretations of concrete evidence (C2) were used to justify the positions of 16 students
in 24 theses (2 students used this argument for 3 theses, 6 used it for 2 theses and 6 for 1 thesis). The arguments
based on theoretical-methodological contributions (C3) were used by 7 students to justify their positions in 7
theses (6 subjects used this form of argument, none of them used to justify their position in all theses and only
1 student used argumentation to justify 2 theses).
Table 5 presents the distribution of the number of arguments per thesis.
Table 5: Category distribution of arguments by thesis (final stage)
Thesis Thesis Thesis

Totals
Final arguments 1 2 3
C5- Arguments that repeat the initial ideas, without
3 2 5 10
enrichment;
C6- Arguments that consider the reviewers' ideas
15 16 9 40
and bring enrichment;
C7- Arguments that are enriched with new
elements, regardless of the ideas expressed by the 1 0 2 03
reviewers;
C8- New arguments that support new positions,
2 4 5 11
consider the reviewers' ideas;
C9- Arguments support new positions, regardless
0 0 1 01
of the ideas raised by the reviewers.
C10-Arguments that bring contradiction with the
1 0 0 01
positioning.
195
Regarding the students' arguments, in the final stage, we identified a prevalence of arguments that were
enriched from the debate with the reviewers. In category C6, we identified 18 students who modified their
arguments to support at least one of the three theses, totaling changes in the arguments of 40 of the 66 theses.
Changes in the arguments were identified, linked to a new position (agree, disagree, or partially agree), which
were motivated by exchanges with reviewers who pointed out weaknesses in the initial argument and/or raised
new arguments for the colleague, as well as their reflections from the contact with the arguments of colleagues.
Category C7 was identified in the arguments of 3 students. Each of the students modified their arguments in 1
of the 3 theses.
Of the 22 students, 7 subjects maintained their arguments in at least 1 of the 3 theses, that is, they repeated
the initial arguments and positions (C5). Only one student finished the AP maintaining the same arguments for
the three theses. This student, in the assessment space, expressed not having clearly understood the debate.
Still, 1 student changed his position and arguments without the influence of exchanges with the reviewers (C9)
and 1 student, in one of the theses, presented a contradictory position with his own arguments (C10).
5 Discussion and Final Remarks

The first result to be emphasized refers to the potential of Theses Debate to promote active and collaborative
learning, considering the changes in the arguments related to the enhancement of ideas and changes in
positions regarding the theses debated.
The experiment allowed us to observe qualitative changes in the arguments (due to the need to express them)
and the advances in the understanding of colleagues' ideas and study on the theses under debate. These
changes were predominantly due to improvement of initial arguments, even when there was no change in the
position (agreement, disagreement, and partial agreement). In addition to those improvements, which were
more frequent, changes in positioning were also identified, with the elaboration of new arguments,
predominantly motivated by interactions with the reviewers, but also because of reflections after the first
positioning. In all these cases, advances in the consistency of the arguments were evidenced.
Positions and arguments about thesis 1 (20 of the 22 students in the final stage agreed) and thesis 2 (18 of the
22 students in the final stage) showed us that they consider cooperative work relevant for learning. They
understand that cooperative learning is not reduced to an abbreviation of work, but to a construction that
takes place through the coordination of different points of view.
Considering students' evaluations, we can say that the experience of collaborative work reinforced the
importance of exchanges in active learning. The review stage proved to be relevant for the improvement of
arguments, with advances both in consistency and expression of ideas as well as development of critical
thinking.
Regarding the work dynamics and use of the virtual environment, the evaluations showed that it was positive,
allowing more activity and protagonism from students. Most students considered that the environment was
adequate to support the architecture, although three students found some initial difficulties in using it. Some
students expressed that they perceived possibilities of using Theses Debate in their work contexts, with the
necessary adjustments according to student age and pedagogical objectives. They mentioned, for example,
the adoption of different theses for each subject and division into groups based on the positions favoring the
discussion between those who agreed and those who disagreed.
Regarding criticisms and suggestions, most students expressed that they would have carried out interactions
for longer so that they would became more familiar with work dynamics and would have more time for
reflection. Some students suggested that, in future AP applications, more material should be made available
before all stages; the introduction of a new specific stage for online discussions by the entire group; and further
time to the establishment of connections supported by a theoretical framework. It was also suggested the use
of a specific (custom-made) environment for Theses Debate.
196
As a result of the authors’ findings and suggestions of the students, an updated version of a specific
environment for Theses Debate is being produced, and will have new features to automate the organization of
stages and facilitate teacher mediation. In addition, new stages like search for approximations and differences
between arguments and formation of groups for the elaboration of argumentative texts are to be implemented.
These improvements will be the subject of future work.
6 References
Aragón, R. (2016). Interação e mediação no contexto das arquiteturas pedagógicas para a aprendizagem em rede, Revista de Educação
Pública, Cuiabá - MT, Brasil.
Dewey, J. (1933). How We Think. Boston: D.C. Heath.
Dewey, J. (1938). Experience and Education. New York: Collier Books.
Elder, A.K. (2019). The OER Starter Kit. Ames, IA: Iowa State University Digital Press. DOI: https://doi.org/10.31274/isudp.7
Menezes, C.S., Aragón, R. (2018). Pedagogical Architectures to Active Learning. Proceedings of PAEE/ALE 2018. Brasilia-DF, Brasil.
Menezes, C., Castro-Jr, A. N., Aragón, R. (2020) Arquiteturas Pedagógicas para Aprendizagem em Rede, SBC-CEIE, Porto Alegre - RS, Brasil.
Parrat-Dayan, S. (2007). A discussão como ferramenta para o processo de socialização e para a construção do pensamento. In Educação
em Revista, v. 45. p. 13-23. Belo Horizonte – MG, Brasil.
Piaget, J. (1953). Origins of Intelligence in the Child. London: Routledge & Kegan Paul.
Piaget, J, (1971). The Psycology of Intelligence: Sicth Impression. London: Routledge & Kegan Paul.
Piaget, J. (1998). Sobre pedagogia. Casa do Psicólogo, São Paulo-SP, Brasil.
Vygotsky, L. (1978). Mind and Society: The Development of Higher Psychological Processes. Cambridge, Massachusetts: Harvard University
Press.
197
Promoting assessment as learning in PBL: findings from blogs
created by first year engineering students
Sandra Fernandes1,2, Anabela C. Alves3, Celina P. Leão3, Ana Pereira3
1
2
3
Abstract
This paper aims to present findings from first year engineering students involved in a Project-Based Learning (PBL)
approach developed within the first cycle degree of Industrial Engineering and Management (IEM) – University of Minho,
Portugal. Student assessment in PBL includes both formative and summative assessment, with a variety of different
methods and purposes. One of the assessment methods in this PBL approach is the creation of a blog by each project team
with the purpose of providing student reflection of their work on a regular basis. The blog allows students to share evidence
of the project journey, with particular focus on the student’s learning process. This idea is aligned with the concept of
assessment as learning, where students are able to learn about themselves as learners and take responsibility for their own
learning and monitor future directions. Based on the analysis of both quantitative and qualitative results from an online
survey applied to students and the content analysis of each of the blogs, it is possible to conclude that the assessment task
was considered as a positive learning experience for students. Quantitative data from the questionnaire show that the blog
was a positive educational tool useful for: i) the organization and dissemination of the project team; ii) the selection of the
project content and organization; iii) keeping an up-to-date record of the project's progress; iv) encouraging writing about
project content and other curiosities. The qualitative data based on the content analysis of the blogs revealed students’
capacity to reflect upon their journey, by critically analysing the feelings, achievements, fears, thoughts, and future plans,
throughout the thirteen weeks of the project duration.
Keywords: project-based learning (PBL); student assessment; assessment as learning; blog as educational tool;
1 Introduction
Promoting assessment as a tool for learning is at the core of the higher education agenda (Evans, 2020; Zhang
et al., 2019). Far beyond the traditional perspective of assessment as a task that takes place at the end of a unit,
designed to confirm what students know and verify if the curriculum goals were met, the concept of
assessment has enlarged its perspective to include a type of assessment that is mainly focused on students’
learning process and therefore occurs throughout the learning process and is aimed at improving student
learning through feedback and self-reflection (Sambell et al., 2012). These ideas are based on what Earl and
Katz (2006) identified as the three purposes of classroom assessment: assessment of learning, assessment for
learning, and assessment as learning. Each of these forms of assessment entail different assessment purposes,
types and methods, as show in figure 1.
Student assessment plays a major role in how students learn, their motivation to learn, and how teachers teach
(Earl & Katz, 2006). Assessment promotes awareness of the knowledge, skills, and beliefs that students bring
to a learning task, promoting student learning. Learning is also enhanced when students are encouraged to
think about their own learning, to review their experiences of learning (What made sense and what didn’t?
How does this fit with what I already know, or think I know?), and to apply what they have learned to their
future learning (Earl & Katz, 2006). This type of assessment is usually used in teaching and learning
environments which focus on student centred learning and assessment methods that engage students in
meaningful learning. Project-Based Learning (PBL) is one of those approaches (Alves et al., 2021).
One of the characteristics of student assessment in PBL is the diversity of assessment methods and purposes
(Fernandes et al., 2012a, 2012b, 2021). PBL approaches include both summative and formative assessment
practices. However, the great emphasis that is put in the student’s learning process gives formative assessment
198
a very important role in PBL, as it aims to monitor, guide, improve, support and regulate student learning. To
meet this goal, alternative assessment methods, such as rubrics, portfolios, simulations, blogs, etc. are highly
recommended.
Figure 1: Assessment Purposes, Types and Methods (Fernandes et al., 2021)

Based on this assumption, this paper will present findings from the use of digital blogs in PBL with the purpose
of providing student reflection of their work on a regular basis. The blog allows students to share evidence of
the project journey, with particular focus on the student’s learning process (Alves, Pereira, et al., 2020; Ifinedo,
2017; Jackling et al., 2015; Marinho et al., 2021). This idea is aligned with the concept of assessment learning,
where students are able to learn about themselves as learners and take responsibility for their own learning
and monitor future directions. Student assessment in PBL recognizes the importance promoting assessment
as learning, giving students several opportunities to reflect and take responsibility for their own learning
process.
This paper is structured in five sections. This introduction is followed by the PBL approach adopted in IEM.
Third section presents the methodology. Fourth section shows the results obtained and, finally, fifth section
presents the discussion and final remarks.
2 PBL approach at Industrial Engineering and Management (IEM)

The PBL was introduced in 2004 by a team of teachers of IEM in School of Engineering of the University of
Minho (UMinho) (Alves et al., 2020; Lima et al., 2007). Since then, many improvements were implemented,
including a formal structure change that introduced formally the Integrated Project of IEM1 (Alves et al., 2014).
Nineteen editions were carried out by teams of teachers, tutors and researchers (Alves et al., 2021). A journey
of continuous improvement has been carried out every year (Alves et al., 2017; Alves & Leão, 2015).
The cohort of 2021/22 involved more students than in previous years, i.e. 67 students were enrolled at this
academic year. These students were organized in eight teams of seven-nine members. Each team had to
develop a project in the context of the course Integrated Project in Industrial Engineering and Management I
(IPIEM1). Every year the project theme (Moreira et al., 2011) is different and this year teams had to design of a
production system to produce a more sustainable packing and should apply the contents learned in the five
different courses, presented in Figure 2.
199
Figure 2. Curricular plan of IEM first-year, first semester, IEM11
Each course has their own assessment methodology that includes a project component. Nevertheless, the
contents of each course included in the IPIEM1 is assessed in the presentations, reports, blog, prototypes, that
the teams delivered to be assessed. Table 1 shows the milestones of the IPIEM1 and the weight of each
deliverable in the assessment methodology. Individual grade of each student is the result of team grade
influenced by a correction factor that result from the peer assessment (Fernandes et al., 2009; Fernandes et al.,
2020; Uebe-Mansur & Alves, 2018) (90%) and the grade obtained to the IEM@ProjectNetworking (third
milestone) (Alves et al., 2013, 2022). As can be seen, the blog is part of the fifth milestone and should be
finished until de last day of the project.
Table 1. Milestones (Ms) and weight of each in the assessment
Ms Week Deliverable Weight (%)
1 Week 2 1st presentation not assessed
2 Week 6 2nd presentation 7,5 (team)
3 Week 8 IEM@ProjectNetworking presentation 10 (individual)
4 Week 11 Preliminary report 25 (team)
5 Week 13 Final report + Prototypes + blog 30 + 20 + 5 (team)
6 Week 14 Final presentation & discussion 12,5 (team)
Each team created a blog, that was regularly presented and discussed by the team members and the tutors
and teachers. Table 2 also includes the links to each of the team blogs.
Table 2. Team blogs of IEM first-year students 2021/22
Teams Team name Link to the Blogs
1. KEY https://keykeepthenvironment.wixsite.com/websitekey
2. GreenBox https://grupo2legi1.wixsite.com/website
3. Eco-Sumaco https://eco-sumaco.wixsite.com/eco-sumaco
4. HEARTH https://grupo4egiuminho.wixsite.com/website
5. LEAF https://rosa-gil.wixsite.com/leaf
6. BioPackage https://pleg62122.wixsite.com/biopackage
7. United for Change https://piegigrupo7.wixsite.com/blog
8. E-Thinking https://grupo8piegi.wixsite.com/website
3 Methodology
This paper aims to give answer to the following research questions:
- How do students describe their PBL experience in their team blogs?
- What importance to students give to the blog as a learning tool?
- How do blogs contribute to promote assessment as learning in PBL?
For data collection a questionnaire was used to collect feedback from students at the end of the PBL experience.
This questionnaire was organized based on a set of items that explore the way the PBL process was developed
200
and perceived by students. From the 58 statements/items based a 5-point scale of agreement (1- strongly
disagree, to 5- strongly agree), 4 items were selected from the questionnaire to be analysed for the scope of
this study. These items are related to the importance given by students to the blog as a learning tool. The items
are the following:
Q14 – useful for the organization and dissemination of the team project;
Q15 - helps to select, organize and register the contents related to the project;
Q16 – helps to keep an updated record of the project’s progress; and
Q17 – stimulated writing (in English) about the contents related to the project and some curiosities.
Besides this analysis, the content of the blogs was also carefully analysed, through the identification of the
main categories that relate to the principles of assessment as learning.
4 Results
Based on the analysis of the results from the questionnaire applied to students and the content analysis of
each of the blogs, the following section presents the organization of the analysis of results according to the
main categories identified in the qualitative and quantitative data. For the quantitative analysis the non-
parametric test Mann-Whitney (U) for comparison between the assessment of two independent samples
(students gender), after verification of normality by Shapiro-Wilk test, was used. Data were stored in Excel and
later analysed with the statistical software SPSS 28.0.
4.1 Student satisfaction with blogs as an educational tool: a quantitative analysis

Data in this study, as previously mentioned, was gathered using four items in the PBL questionnaire that 72%
of the enrolled students voluntarily and anonymous filled, and then analysed using statistical methods.
The sample included students from the all teams, however not all the members of each team answered the
questionnaire, varied from 50% (in one team) to 88% (in one team). The majority of participant students (70.8%)
are female, reflecting the gender balance of the course with 63.2% of female students. The main descriptive
statistics measures obtained based on students’ perceptions for the four items Q14, Q15, Q16 and Q17 are
presented in Table 3, total and by gender.
Table 3. Main descriptive statistics obtained for Q14, Q15, Q16 and Q17 students’ perceptions (F – female, M
– male).
Descriptive
Q14 Q15 Q16 Q17
Statistics
F 34 34 34 34
n M 14 13 13 14
Total 48 47 47 48
F 4.03  .83 4.03  .63 4.29  .72 4.50  .62
mean  s.d. M 4.07  .83 4.08  .76 3.69  .85 3.71  1.14
Total 4.04  .82 4.04  .78 4.13  .78 427  .87
F 4 4 4 5
median M 4 4 4 4
Total 4 4 4 4
F 2 3 3 3
min M 3 3 2 2
Total 2 3 2 2
F 5 5 5 5
max M 5 5 5 5
Total 5 5 5 5
Generally, quantitative data from the questionnaire show that the blog was a positive educational tool useful
for: i) the organization and dissemination of the team project (mean=4.04); ii) the selection of the project
201
content and organization (mean=4.04); iii) keeping an up-to-date record of the project's progress (mean=4.13);
and iv) encouraging writing about project content and other curiosities (mean=4.27).
Notwithstanding the overall positive assessment of the blog, when the analysis by gender was made, some
differences between genders have been uncovered (Figure 4). The differences statistically significant
encountered between genders were in the two items Q16 and Q17, with female student assessing these two
items with a higher mean (U=135.0, p<.05, for Q16 and U=142.5, p<.05, for Q17). That is, regarding keeping
an up-to-date record of the project's progress (Q16) and encouraging writing about project content and other
curiosities (Q17) female students attributed more importance. Also, Figure 3 illustrate the students’ assessment
distribution for both genders regarding item Q17 where the highest difference between genders were
occurred.
Figure 3. Students’ assessment by gender: (a) mean values for all items and (b) distribution for Q17 (blue
male and red female students).
To some extent, this positive results seems to be in line with a previous preliminary study regarding gender
difference in peer assessment, where female students obtain peer assessment higher marks in comparison with
their male colleagues (Alves, Moreira & Leão, 2017) and that it is usually the female students who develop
writing tasks and keep the tasks up to date.
4.2 Promoting student self-reflection through blogs: a qualitative analysis

The qualitative data based on the content analysis of the blogs revealed students’ capacity to reflect upon their
journey, by critically analysing the feelings, achievements, fears, thoughts, and future plans, throughout the
fifteen weeks of the project duration. The following sections reveal, through examples extracted from the blogs,
students’ capacity of self-reflection regarding their team roles and the weekly meeting reports.
4.2.1 Team member characteristics and roles

The content of the blogs reveal that the teams focused on presenting information about their initial
expectations and motivations in regard to the project and also identifying their preferred team role, according
to the nine roles considered in the Belbin test (Figure 4). This helped to characterize the team as whole.
“We are a team constituted by 8 members of the engineering and industrial management at the University of
Minho, willing to spread awareness about environmental and sustainability causes”.
“Eco Sumaco is the result of the hardwork of 8 dedicated Industrial Management and Engineering students in
the University of Minho”.
“During the third week, we did some important things for our project. We had to complete the so called "Belbin
tests", which were useful to make us understand the role of each member of the group. Also, besides the natural
organisation that we had to do in all of our social media to keep it updated, we were able to be in a position
where we would have some individual feedback in terms of our contribution to the project, by doing the peer
evaluation.”
202
Figure 4. Examples of how teams presented each team member and their roles in the project
Students also expressed, at an initial phase of the poject, their expectations and motivations in regard to the
project. This reflection shows how students related the objectives of the project with the development of their
individual competences. For students to be able to improve, they must develop the capacity to monitor the
quality of their own work.
"My motivation for this project is to know the skills it will provide me. In addition to being able to improve my
skills for group work, I know that I will develop my speech and begin to get some sense of what the future in
industrial engineering and management will be like."
“Throughout the development of this project I’m hoping to help spread the message of the climate urgency, as
well as acquire skills that will help me in the future as I get into the labour market.”
“With this project I expect to improve my confidence while speaking in public and at the same time upgrade
my team working and leadership skills.”
"In this project, I’m not only expecting to solidify technical knowledge but also to enhance soft skills such as
communicating, organizing and planning tasks, and working as a team."
"I see this project as the perfect way to improve my soft and hard skills, apart from experiencing the methods
and logistics required for the development of a specific product."
“I see this project as a way to improve my creativity, to develop my skills to speak in front of a public and also
to give me experience of team working”
“This project will not only contribute to the environment and sustainable habits but also for my personal
experience and development.”
“I see this project as an opportunity to get out of my comfort zone and work on my weaknesses such as
presenting in public and having to work with different people each with their own personality.”
4.2.2 Weekly meeting records

These weekly reports reveal student’s thinking about his/her learning process and the strategies or mechanisms
that were used to adjust and advance in the project (Figure 5). When students reflect on their own learning
and need to communicate it to others, as they did in their blogs, they are intensifying their understanding
about a topic, their own learning strengths as a team and the areas in which they need to develop further.
“Today was a very special day for the team. We gather for the first time and it was awesome to get to know
each other within the work context. We are looking forward to the change. What about you? Are you with us?”
“Greeting to all readers and welcome to our very first weekly report about our project. Firstly, we met our student
advisor, Margarida Vasconcelos, that introduced us to many resources that we could use, as a team, from
communicating to sharing media files and more. Afterwards she briefly spoke about her experience as a student,
working on a project based on similar bases as ours in past years. Subsequently, we got to know our mentor
Florentina Abreu that discussed what possible paths we could go to. On the second meeting, we split as pairs
(after doing the personality test) working on many areas such as blog pair, research pair, presentation pair and
203
script pair. Here, we divided tasks, organized and built our very own work schedule as well as searching some
topics about our project”. (G2)
“Like every journey ever, this one has come to an end. It was a pleasure, and a big one. When we started this
project the only thing we knew was our main goal, the development of some sort of sustainable packaging. We
had no idea with who we were going to work, we had no idea how to even work in an “company”, this was
something new. The first weeks were rough, we were experiencing something outrageously out of our comfort
zone. We had a chance to meet everyone involved in the project and get to know our future colleagues. This
was the beginning of HEARTH. HEARTH, the combination of the words “earth” and “heart” was our little
invention. A company that would create a sustainable package for cakes and little pastry items. It was exciting,
it was stressful, it was everything. Every emotion possible, we felt it. From panic to laughter, from anxiety to
relief, it was the full package. This was a chapter in our lives that none of us will ever forget. Our hearts are
nothing but filled with joy and love after this semester of absolute madness. We loved it. There is nothing more
to say. We want to thank everyone involved in the making of HEARTH, our tutors, our teachers, and of course
our amazing team. Without them this would be even crazier than it was. Thank you so much, see you soon,
stay safe and remember, Keep earth’s heart beating. HEARTH.”
“After 3 long months of hard work and dedication, we reached the end of our final project. This project has been
a true challenge in so many ways that we couldn’t even imagine. It has been a pleasure to complete all our
given tasks from the various CUs, where we saw the practical use of our theoretical knowledge, improving not
only hard skills but many soft skills, one of the many goals we had predicted. In the beginning we had no idea
this project would have this sort of dimension, but we are glad that we were able to learn so much about
teamwork, leadership, initiative and have a taste of what being an industrial engineer would feel like. In general,
we are thankful for the proposed project, and we would like to thank everyone involved in it, including our
teacher-mentor, Maria Florentina Abreu, our student-mentor, Margarida Vasconcelos, and all our CUs teachers!
We hope you could learn from us to always think outside the box, see you another time. GreenBox”.
Figure 5. Examples of strategies or mechanisms that students adopted to adjust and advance in the project
5 Final remarks
This paper presents findings about the use of blogs to promote assessment as learning in a PBL context, carried
out within an IEM program. The creation of a blog by the project teams as one of the assessment components
of the PBL approach has allowed the first-year engineering students to share the development of the project,
focusing especially on the team members' learning process and promoting the reflection and self-assessment
about their work throughout the thirteen weeks of the project. According to the analysis of both quantitative
and qualitative results, it was possible to address the research questions that guided the development of this
study.
The first research question refers to the student’s experience with their team blogs. Based on the results of the
quantitative data from an online survey, the majority of students expressed a positive judgement about the
blog as a useful educational tool for: i) the organization and dissemination of the project team; ii) the selection
204
of the project content and organization; iii) keeping an up-to-date record of the project's progress; iv)
encouraging writing about project content and other curiosities.
Moreover, the second research question addresses the importance the students give to the blog as a learning
tool. Based on the abovementioned results of the quantitative data, it is possible to conclude that, based on
the overall analysis, the students value more the fact of being able to keep up-to-date record of the project's
progress, as well as, the opportunity to write in English about project content and other curiosities. However,
if the analysis is carried out by gender, it is possible to perceive that female students value more these two
points, while male students attribute more importance to the usefulness of the blog in the organization and
dissemination of the team project, as well as, the ease that the blog gives to organize and register the contents
related to the project.
Finally, the last research question intends to understand how blogs can contribute to promote assessment as
learning in PBL context. Thus, the qualitative data that resulted from blogs’ content analysis, allowed to
conclude that this learning tool promotes students’ capacity to reflect about their work and journey, showing
a strong sense of criticism when analysing their feelings, weaknesses, achievements, strengths and future plans
during the development of the project.
These conclusions are aligned with the concept of assessment as learning, that aims to promote an
environment where students are able to learn about themselves and take responsibility for their own learning
process, while developing abilities to evaluate their own progress and direct their own learning. For graduates
and professionals, the most important assessment is self-assessment. Our students should be helped to
develop appropriate dispositions, attitudes and skills (McDowell & Sambell, 2007; Sambell et al., 2012).
6 Acknowledgments
Scope: UIDB/00319/2020. Also, the authors would like to express their acknowledgement to all students that
answered to the questionnaire.
7 References
Alves, A.C., Moreira, F., & Leão, C.P. (2017). Peer assessment in PBL: Does gender matter? 2017 International Conference on
Engineering, Technology and Innovation (ICE/ITMC), 1357-1361.
Alves, A. C., Fernandes, S., Moreira, F., Lima, R. M. S. P., Carvalho, J. D. A., Sousa, R. M. A. S., Mesquita, D. I. A., & Hattum-Janssen, N. van.
(2021). Project-Based Learning: implementação no primeiro ano de um curso de Engenharia. UMinho Editora.
https://doi.org/10.21814/uminho.ed.26
Alves, A. C., & Leão, C. P. (2015). Action, practice and research in project based learning in an industrial engineering and management
program. ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE), 5–2015.
https://doi.org/10.1115/IMECE2015-51438
Alves, A. C., Moreira, F., Fernandes, S., Leão, C. P., & Sousa, R. (2017). PBL in the first year of an industrial engineering and management
program: A journey of continuous improvement. International Symposium on Project Approaches in Engineering Education, 9.
Alves, A. C., Moreira, F., Leão, C. P., & Fernandes, S. (2020, November 16). Ten Years of Positive Feedback on Project-Based Learning From
First-Year Engineering Students’ Perspective. Volume 9: Engineering Education. https://doi.org/10.1115/IMECE2020-23212
Alves, A. C., Moreira, F., Lima, R. M., Sousa, R, M., Carvalho, D., Mesquita, D., Fernandes, S., & Van-Hattum-Janssen, N. (2014). Aprendizagem
Baseada em Projetos interdisciplinares: análise da implementação em duas estruturas curriculares distintas. In A. Fischer & O. L.
de O. M. Heinig (Eds.), Linguagens em uso nas Engenharias (Edifurb-). FURB.
Alves, A. C., Pereira, A. C., Leão, C. P., Fernandes, S., & Uebe-Mansur, A. F. (2020, November 16). The Use of Blogs in a Project-Based Learning
Context for First-Year Engineering Students’ Teams. Volume 9: Engineering Education. https://doi.org/10.1115/IMECE2020-23768
Alves, A., Carvalho, J. D., Mesquita, D., Fernandes, S., & Lima, R. M. (2013). IEM@ProjectNetworking: bringing first year students closer to
professional practice. International Symposium on Project Approaches in Engineering Education (PAEE2013), 1–7.
Alves, A., Oliveira, S., Leão, C. P., & Fernandes, S. F. (2022). IEM@ProjectNetworking revisited: freshmen students closer to professional
practice. PAEE/ALE’2022, International Conference on Active Learning in Engineering Education.
Earl, L., & Katz, S. (2006). Rethinking Classroom Assessment with Purpose in Mind. In Learning. https://doi.org/10.4135/9781446214695
Evans, C. (2020). Student assessment Thematic Peer Group Report (Issue March). European University Association.
https://eua.eu/downloads/publications/eua report student assessment_web.pdf
Fernandes, S., Alves, A. C., & Uebe-Mansur, A. (2021). Student-Centered Assessment Practices. In Handbook of Research on Determining
the Reliability of Online Assessment and Distance Learning (pp. 213–243). https://doi.org/10.4018/978-1-7998-4769-4.ch009
Fernandes, S., Alves, A. C., & Uebe-Mansur, A. F. (2020). Student-Centered Assessment Practices: An Integrated Approach With Project-
Based Learning (PBL). In Determining the Reliability of Online Assessment and Distance Learning (p. 340). IGI Global.
205
Fernandes, S., Flores, M. A., & Lima, R. M. (2012a). Students’ views of assessment in project-led engineering education: Findings from a
case study in Portugal. Assessment and Evaluation in Higher Education, 37(2), 163–178.
Fernandes, S., Flores, M. A., & Lima, R. M. (2012b). Students’ views of assessment in project-led engineering education: Findings from a
case study in Portugal. Assessment and Evaluation in Higher Education, 37(2), 163–178.
Fernandes, S., Mesquita, D., & Lima, R. et al. (2009). The impact of peer assessment on teamwork and student evaluation. Innovation and
Assessment of Engineering Curricula, 125–136.
Ifinedo, P. (2017). Students’ perceived impact of learning and satisfaction with blogs. International Journal of Information and Learning
Technology, 34(4), 322–337. https://doi.org/10.1108/IJILT-12-2016-0059
Jackling, B., Natoli, R., Siddique, S., & Sciulli, N. (2015). Student attitudes to blogs: a case study of reflective and collaborative learning.
Assessment and Evaluation in Higher Education. https://doi.org/10.1080/02602938.2014.931926
Lima, R. M., Carvalho, D., Assunção Flores, M., & Van Hattum-Janssen, N. (2007). A case study on project led education in engineering:
students’ and teachers’ perceptions. European Journal of Engineering Education, 32(3), 337–347.
Marinho, P., Fernandes, P., & Pimentel, F. (2021). The digital portfolio as an assessment strategy for learning in higher education. Distance
Education, 42(2), 253–267. https://doi.org/10.1080/01587919.2021.1911628
McDowell, L., & Sambell, K. (2007). Putting assessment for learning into practice.
https://www.researchgate.net/publication/265939893_Putting_Assessment_for_Learning_into_Practice
Moreira, F., Mesquita, D., & van Hattum-Janssen, N. Van. (2011). The importance of the project theme in Project-Based Learning: a study
of student and teacher perceptions. Proceedings of the 2011 Project Approaches in Engineering Education, 53(9).
https://doi.org/10.1017/CBO9781107415324.004
Sambell, K., McDowell, L., & Montgomery, C. (2012). Assessment for Learning in Higher Education. In Assessment for Learning in Higher
Education. Routledge. https://doi.org/10.4324/9780203818268
Uebe-Mansur, A. F., & Alves, A. C. (2018). The importance of peer assessment & self-assessment in PBL applied to an Administration course.
Revista Ibero-Americana de Estudos Em Educação, 13(esp1), 451–467. https://doi.org/10.21723/riaee.nesp1.v13.2018.10347
Zhang, L.-Y., Liu, S., Yuan, X., & Li, L. (2019). Standards and Guidelines for Quality Assurance in the European Higher Education Area:
Development and Inspiration. DEStech Transactions on Social Science, Education and Human Science, icesd.
https://doi.org/10.12783/dtssehs/icesd2019/28072
206
The student journey in PBL: using individual portfolios to
promote self-reflection and assessment as learning
Sandra Fernandes1,2, Marta Abelha3,4 & Ana Silvia Albuquerque1
1
2
3
Laboratório de Educação a Distância e eLearning - LE@D, Universidade Aberta, Portugal
4
Centro de Estudos Interdisciplinares - CEIS20, Universidade de Coimbra, Portugal
Email: [email protected], [email protected].; [email protected]
Abstract
This paper aims to present findings based on students’ journey within a Project-based learning (PBL) experience, with
particular emphasis on the development of an individual portfolio by students, as part of the assessment process. The PBL
project involved three curricular units (Research Methods and Techniques II, Ethics and Education and Sociology of
Education), from the field of Education Sciences, integrated in the first year of the study plan of the Social Education degree
programme at Portucalense University, Portugal. This paper describes student assessment within this educational approach,
including both formative and summative methods. It calls attention to the concept of assessment as learning, which can
be understood as a process where students are able to learn about themselves as learners and take responsibility for their
own learning and monitor future directions. One of the assessment methods included in the PBL approach is the
development of an individual portfolio with the purpose of enhancing student self-reflection and assessment of their work
on a regular basis. The portfolio allowed students to share evidence of the project journey, with particular focus on the
student’s learning process. The students were provided with general guidelines to develop their portfolio, but the flexible
nature of this educational resource allowed students to record and highlight their individual and team contributions in a
very creative and personalized way. The analysis of qualitative data from the individual student portfolios reveals student
self-reflection on individual achievements, project milestones, teamwork skills, impact of teacher feedback, celebration
moments, amongst others. Implications of these findings for PBL assessment practices will be discussed further in the
paper.
Keywords: project-based learning (PBL); student assessment; assessment as learning; individual portfolios
1 Introduction
Project-based learning (PBL) is an educational approach which focuses on student-centred learning, the
articulation between theory and practice and the development of student skills (Powell, 1999; Powell & Weenk,
2003). It has been widely used in higher education programs to improve student engagement, promote active
learning and provide significant learning experiences (S. Fernandes et al., 2014; Lima, R. M., Dinis-Carvalho, J.,
Sousa, R. M., Alves, A. C., Moreira, F., Fernandes, S., & Mesquita et al., 2017; Prince, 2004).
Student assessment in PBL differs from traditional assessment. It focuses on assessment as an integrated
process, with the purpose of improving student learning through relevant feedback and self-reflection (S.
Fernandes et al., 2012; S. Fernandes, Alves, et al., 2021; S. R. Fernandes et al., 2010; M. I. R. Ortigão, D. Fernandes,
2019). Far beyond the concept of assessment of learning, student assessment in PBL emphasizes the concept
of assessment for learning or assessment as learning, which can be understood as a process where students
are able to learn about themselves as learners and take responsibility for their own learning and monitoring
future directions (Earl & Katz, 2006; Sambell et al., 2012).
Student portfolios are a formative assessment method that promotes student engagement, motivation, self-
reflection and, consequently, significant learning (Marinho et al., 2021). By using portfolios, teachers can
support students in the construction of their own knowledge through metacognition and self-regulation
processes. This is particularly relevant in PBL approaches where the monitoring and assessment of the students’
learning process is carefully planned and enhanced (S. Fernandes, Alves, et al., 2021).
207
This paper aims to present findings based on students’ journey within a PBL experience, with particular
emphasis on the development of an individual portfolio by students, as part of the assessment process. It
focuses on a case study developed in the field of Education, where PBL has been implemented in the past few
years (Abelha et al., 2020; Abelha & Fernandes, 2021; S. Fernandes et al., 2018; S. Fernandes, Conde, et al.,
2021).
2 Context of Study
The PBL project involved three curricular units (Research Methods and Techniques II, Ethics and Education and
Social and Educational Intervention Models), from the field of Education Sciences, integrated in the first year
of the study plan of the Social Education degree programme at Portucalense University, Portugal. In PBL,
student assessment includes several milestones where the student groups present their projects development
state. These milestones aim to provide students with moments of feedback on the project development and
an opportunity to clarify doubts regarding the integration of the curricular units in the project. Table 2 presents
the milestones of the project, according to the last five editions of PBL projects in the Social Education degree
at UPT.
Table 1. PBL-LES Project Miletones Schedule
# Week Milestone
1 2nd Week Presentation of the PBL-LES Project
2 3rd Week Open Lecture on the Project Theme
3 4thWeek Presentation # 1 (Submission in Moodle)
4 10th Week Presentation # 2 (Submission in Moodle)
5 13th Week Presentation # 3 (Submission in Moodle)
6 13th Week Preliminary Project Report Submission
7 14th Week Preliminary Project Report Feedback (for each CU)
8 15th Week Final Presentation and Discussion of the Project (Submission of the Final Project Report)
9 15th Week Submission of Individual Portfolio
Concerning the project's assessment components, these are distributed at different times, throughout the
semester, including the following elements and weights: Presentation #1, #2 and #3 (15%), Preliminary Project
Report (20%), Final Project Report (30%), Final Presentation and Project Discussion (15 %), Individual Portfolio
(20%). The development of an individual portfolio has the purpose of enhancing student self-reflection and
assessment of their work on a regular basis. The portfolio allows students to share evidence of the project
journey, with particular focus on the student’s learning process. Students are provided with general guidelines
to develop their portfolio, but the flexible nature of this educational resource allows students to record and
highlight their individual and team contributions in a very creative and personalized way.
3 Methods
This paper is based on a qualitative study which aims to give answer to the following research questions:
- How do students describe their journey during the PBL experience?
- How do individual portfolios contribute to promote self-reflection and assessment as learning?
Data analysis is based on the results of a content analysis of the qualitative data available in the individual
student portfolios. For the elaboration of their portfolios, students were provided with some guiding questions,
including the following: What have I learned from this curricular unit? What did I learn from participating in the
PBL project? What motivated me the most in the PBL project? What difficulties have I encountered? Why? How
did I get over them? What could I have done better? What did I learn about myself? What are my strengths
and weaknesses? What did I learn about others? What new experiences have I done? What do I do differently
now from what I have done in the past? What are my plans for the future? What goals do I have for the next
semester?
208
Based on these topics, students based their reflections on their individual achievements, the project milestones,
teamwork skills, impact of teacher feedback, celebration moments, amongst others.
4 Results
As stated previously, this paper aims to present findings based on students’ journey in a PBL experience,
described in an individual portfolio developed by students. The most important topics explored by students in
their self-reflections are presented in the following subtopics.
4.1 What did I learn? / What skills did I develop?

When asked about what they had learned and what type of skills they had the opportunity to develop during
the PBL, the students highlighted the learning outcomes that referred to the technical skills of the course units
of Research Methods and Techniques I and Social and Educational Intervention Models, as can be seen in their
speeches:
With the PBL project (...) I learned how to conduct questionnaire surveys, develop scripts, protocols and
interviews, as well as their transcriptions (...) I learned how to carry out research projects, which should be
included in the structure of a research report (...) (S6)
This project represented an area of growth for me, as it allowed me to understand how a social project should
be developed, addressing the importance of each of the phases that constitute it (S5)
( ...) I also learned how to conduct and transcribe an interview, something I had never done before (S10)
The PBL project was a great opportunity to acquire knowledge and enabled me to broaden my vision of the
area of Social Education. (S3)
Interdisciplinarity was another aspect highlighted by students when they stated that the PBL project:
(...) was very beneficial and enriching in my learning as a Social Education student, because it allowed me
(...) to apply knowledge from several areas at the same time. (S3)
(...) cross knowledge from different curricular units and take advantage of them (...) (S6)
Students mentioned that they had the opportunity to develop several transversal competencies, namely
communication skills, time management, decision-making, interpersonal skills, and teamwork skills, as shown in
the following excerpts:
(...) this project facilitated better communication as a group and a close relationship between students and
teachers (S5)
One of the lessons I learnt was that developing this type of project goes far beyond communicating, it also
involves collaborating, which presupposes the confrontation of ideas, the acceptance of different opinions, as
well as the ability to construct joint solutions as a team. (S4)
With the PBL project, I learned to manage time better, the importance of teamwork and the allied difficulty
(...). (S6)
Throughout this project, I learned to work as a team and that when you work together everything becomes
easier and we can achieve a better result (...). (S10)
Today I feel more able to establish forms of communication appropriate to situations, to work in a team, to
establish a relationship of trust with the other members, to build my professional identity and to present
relational and helping attitudes. (S3)
(...) one of the most relevant aspects of the project was also the opportunity to develop collaborative work,
which presents difficulties, but also many possibilities for growth, because with everyone's contribution, with
their different ideas and critical views, the final product is certainly more enriching. (S4)
(...) allowed for a greater capacity of time management for the tasks in which it would be necessary, through
this capacity of time optimisation, I was able to finish the tasks in a more profitable way (S5)
209
4.2 What do I now do differently from what I have done in the past?
With regard to what they consider that they now do differently from what they did in the past, as a result of
the experience of the PBL project, students highlighted the greater capacity for critical reflection, reasoning,
creativity, the ability to make more constructive criticisms, and the ethical care taken during the preparation
and implementation of the project itself, as evidenced in the students' speeches:
(...) when comparing my pathway last year and currently, my evolution in many aspects is evident, namely
today I am able to analyse more critically any procedure to be performed, as well as the reason for it. (...) (S3)
I became much more careful in the collection and processing of information, looking for suitable and reliable
sources and also in the respect for copyrights and Copyright rules. (S4)
What changed most from my initial work to the final one was my ability to provide a basis, which is strongly
present in the report we prepare and my capacity for creativity (...) (S5)
The way of reflecting on my own work and that of the group (...) (S6)
I learnt to make more constructive criticism on subjects which I did not know so well and which I now know
more about (...) (S7)
4.3 What did I learn about myself? / What did I learn about others?
Resilience and the capacity to overcome oneself were the two aspects most often mentioned by students when
they were asked to reflect on what they learned about themselves during the completion of the PBL project,
as evidenced by the students' speeches:
(...) I managed to know that (...) when faced with a difficulty I do not give up and always try to find a solution.
(S3)
I learned that my effort will always be rewarded and that giving up should not be an option. I also learned to
value the work I do, giving my best, in a demanding context and under less than ideal conditions. (S4)
My capacity for resilience and overcoming, my level of demand, my organizational capacity and my interviewer
side, eheh! (S6)
I have learned that no matter how tiring a job may be, I can find the strength and the will to do anything, that
with dedication I can do anything, all I need is the will, and I have learned that I can be more committed than
I had imagined. (S10)
With regard to what they consider they have learnt about others, the students' reflections focused essentially
on teamwork: its implications, difficulties, added value, as can be seen in the students' discourse:
I learned that group work is not always easy. Working in a team implies communicating and collaborating,
living with very different points of view and opinions, having to overcome problems together and outline
solutions collectively. (S7)
Working in a team also means having the ability, by understanding the strengths and the less strong points of
each member of the group, to seek to establish bridges and consensus, giving our contribution to the final
product. (S4)
Regarding the others, I have learned that together we can overcome any challenge. Everyone has their own and
unique characteristics, which has allowed us to have projects which are so different, but at the same time so
complete. (S5)
5 Final Remarks
Assessment as learning aims to guide and provide opportunities for each student to monitor and critically
reflect on their knowledge and identify the next steps. The use of the portfolio to report a student's learning
process has impact on students as it provides the deepening of knowledge and reflecting on what is learned
and how it is learned, besides also promoting the articulation of theory and practice (Marinho et al., 2021).
210
The portfolio is an assessment method that elicits students' learning and metacognitive processes. Data
collected from student's self-reflections in the portfolio prove the theory about the usefulness of a portfolio,
in the present case, applied in a PBL project, which allowed them to enrich the learning experience. In addition
to technical aspects, they developed transversal competencies, such as communication skills, time
management, decision-making, interpersonal skills, and teamwork skills which are essential in the present days.
Also, the general portfolio guidelines enhanced greater creativity.
Critical reflection and ethical care are also very important achievements. As a matter of fact, when asked about
what they did differently from what they have done in the past, their evolution is obvious, emphasizing their
capacity for constructive criticism either about the other team members or their own performance. PBL's
interdisciplinarity was considered advantageous, crossing knowledge from different curricular units.
Above all, if it would be possible to hierarchize in the present pandemic context, we would highlight the
«resilience» and «capacity to overcome oneself» expressed by several students. This study induces us to
conclude that the permanent contact with each other, either as a teamwork or celebration moments, including
teacher's feedback related to portfolio construction, contributed to motivating and improving assessment as
learning and taking responsibility for their learning and monitoring future directions.
References
Abelha, M., & Fernandes, S. (2021). Aprendizagem baseada em Projetos: o contributo da formação pedagógica docente. In J. Belli, S.
Henriques, & C. Neves (Eds.), Formação de Professores e Inovação Pedagógica: contextos, processos e evoluções (pp. 87–104). Paco
Editorial.
Abelha, M., Fernandes, S., Miguel, I. C., & Sousa, E. (2020). Educação social e aprendizagem e serviço (Aps). LAPLAGE EM REVISTA, 6(3), 38–
53. https://doi.org/10.24115/S2446-6220202063935p.38-53
Earl, L., & Katz, S. (2006). Assessment of Learning What Is Assessment of Learning? In Rethinking Classroom Assessment with Purpose in
Mind.
Fernandes, S., Abelha, M., Fernandes, S. M., & Albuquerque, A. S. (2018). Implementation of PBL in a social education programme at the
Portucalense University. International Symposium on Project Approaches in Engineering Education, 8, 446–455.
Fernandes, S., Alves, A. C., & Uebe-Mansur, A. (2021). Student-Centered Assessment Practices. In Handbook of Research on Determining
the Reliability of Online Assessment and Distance Learning (pp. 213–243). https://doi.org/10.4018/978-1-7998-4769-4.ch009
Fernandes, S., Conde, A., & Abelha, M. (2021). Student motivation in the first year of University: findings from the implementation of a PBL
project for a real context. In V. V.-B. Anabela C. Alves, Natascha van Hattum-Janssen, Rui M. Lima (Ed.), Proceedings of the
PAEE/ALE’2021, International Conference on Active Learning in Engineering Education 13th International Symposium on Project
Approaches in Engineering Education (PAEE) 18th Active Learning in Engineering Education Workshop (ALE) (pp. 301–309).
Department of Production and Systems – PAEE association School of Engineering of University of Minho, Campus de Azurém,
4800-058 Guimarães, Portuga. https://doi.org/10.5281/ZENODO.5096981
Fernandes, S., Flores, M. A., & Lima, R. M. (2012). Students’ views of assessment in project-led engineering education: Findings from a case
study in Portugal. Assessment and Evaluation in Higher Education, 37(2), 163–178. https://doi.org/10.1080/02602938.2010.515015
Fernandes, S., Mesquita, D., Flores, M. A., & Lima, R. M. (2014). Engaging students in learning: Findings from a study of project-led
education. European Journal of Engineering Education, 39(1), 55–67. https://doi.org/10.1080/03043797.2013.833170
Fernandes, S. R., Flores, M. A., & Lima, R. M. (2010). A aprendizagem baseada em projectos interdisciplinares: avaliação do impacto de uma
experiência no ensino de engenharia. Avaliação: Revista Da Avaliação Da Educação Superior (Campinas), 15(3), 59–86.
https://doi.org/10.1590/s1414-40772010000300004
Lima, R. M., Dinis-Carvalho, J., Sousa, R. M., Alves, A. C., Moreira, F., Fernandes, S., & Mesquita, D., Lima, R. M. M., Dinis-Carvalho, J., Sousa,
R. M. M., Alves, A. C. C., Moreira, F., Fernandes, S., & Mesquita, D. (2017). Ten Years of Project-Based Learning (PBL) in Industrial
Engineering and Management at the University of Minho. PBL in Engineering Education: International Perspectives on Curriculum
Change, 33–52. https://doi.org/10.1007/978-94-6300-905-8
M. I. R. Ortigão, D. Fernandes, T. V. P. & L. S. (Ed.). (2019). Para um enquadramento teórico da avaliação formativa e da avaliação sumativa
das aprendizagens escolares. In Avaliar para aprender no Brasil e em Portugal (pp. 139–163). Editora CRV.
Marinho, P., Fernandes, P., & Pimentel, F. (2021). The digital portfolio as an assessment strategy for learning in higher education. Distance
Education, 42(2), 253–267. https://doi.org/10.1080/01587919.2021.1911628
Powell, P. (1999). From classical to project-led education. In António Sérgio pouzada (Ed.), Project Based Learning. Project-led Education
and Group Learning. (pp. 11–42). Thematic Network Plastics in Engineering.
Powell, P., & Weenk, W. (2003). Project-Led Engineering Education (Vol. 53, Issue 9). Lemma Publishers.
https://doi.org/10.1017/CBO9781107415324.004
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231.
Sambell, K., McDowell, L., & Montgomery, C. (2012). Assessment for learning in higher education. In Assessment for Learning in Higher
Education. https://doi.org/10.4324/9780203818268
211
PBL tutoring dynamics in first-year of Industrial Engineering
and Management Program
Celina Pinto Leão1, M. Florentina Abreu1, Anabela C. Alves1, Sandra Fernandes2
1
2
Email: {cpl, d7233, anabela}@dps.uminho.pt, [email protected]
Abstract
Project-Based Learning (PBL) methodology engages students in learning in an active practice promoting ideas discussion
and share among all team members. One of the ways in which this is done is to stimulate students to solve real-world
problems during a semester integrating all the Curricular Units (CU) through a large interdisciplinary project development.
One of the distinguishing elements of the PBL methodology is teamwork. In this process, emerge the tutor, whose main
task is to stimulate discussion and facilitate the process and direction of the students work. Also, tutor report the team
progress to the PBL project coordinator. For each team a set of activities, tasks and milestones are planned in order to
succeed project objectives. Taking this in line, this paper describes the tutor’s role from the students’ viewpoint, within the
first year of the Industrial Engineering and Management degree at the University of Minho based on an online questionnaire
composed by a total of 37 questions. A total of seven tutors-teachers and eight third year tutors-students were evaluated,
from each group, regarding their role and their contribution to the project development. At the end of the semester,
students’ feedback was also collected through a focus group regarding the tutor role. Moreover, a first-person narrative of
two tutors-teachers is presented. One tutor had her first experience as a tutor, the other with seven years of experience in
this role. Main findings highlight a students’ overall satisfaction with tutors’ role and by underlining the importance of both
as a team member. Nevertheless, some fewer positive points are seen as improvements opportunities for the next edition.
Keywords: Active Learning; Engineering Education; Project-based Learning; Tutor role.
1 Introduction
The learning process is an important determinant for the student learning outcomes. Currently, in a world in
constant transition and evolution, teaching needs to adapt to the demands that the working world places and
seeks in professionals. The learning process needs to make its transition from the traditional teaching to a
more active teaching, so as to reflect this transition and embark in a teaching method that meets the learners’
needs in order to develop their skills and competences. This later concept comprehends not only cognitive
components but also the motivational, ethical, social, and behavioural ones, merging stable features, learning
outcomes (e.g., knowledge and skills), belief value systems, habits, and other psychological features (Flumerfelt,
Alves & Khalen, 2014).
According to Prince & Felder (2007), traditionally engineering has been taught deductively, which means
having a lecturer presenting a topic by exposing the meaningful theory and mathematical models,
demonstration and completing the cycle through the assessment of the students’ ability. However, in the last
few years, engineering institutions have been implementing alternative teaching methods, specifically inductive
teaching ones (Graham, 2018; Prince & Felder, 2007). In inductive teaching, students are presented with a
challenge, like an authentic case study, a real-world problem or a real observation. To overcome the challenge,
students rapidly acknowledge the need for facts, skills, and conceptual understanding, which can happen in
one of two ways: the teacher provides instruction or supports students to learn by themselves. Inductive
teaching models’ examples encompass inquiry-based learning, discovery-based learning, problem-based
learning, project-based learning, case-based teaching, and just-in-time teaching (Prince, Felder & Brent, 2020;
Prince & Felder, 2006).
In this context, lectured based classes are not enough and there is a need for more active learning
methodologies, like Project Based Learning (PBL) model. PBL has been implemented in Higher Education in
areas such as social education (Imaz, 2015), higher education (Guo, Saab, Post & Admiraal, 2020), health
212
(Demirören, Turan, & Teker, 2020; Edelbring, Alehagen, Mörelius, Johansson, & Rytterström, 2020; Khamchiyev,
Batyayeva, Shandaulov, Zhashkeyeva, Suleimenova & Sagimova, 2020), e-learning (Amamou & Cheniti-
Belcadhi, 2018), computer engineering and software development (Cornejo-Aparicio, Flores-Silva, Bedregal-
Alpaca & Tupacyupanqui-Jaen, 2019), STEM (Prince, Felder & Brent, 2020) and engineering (Beneroso &
Robinson, 2022; Kuppuswamy & Mhakure, 2020; Rivaldi, Megayanti & Aryanti, 2020).
PBL is sustained by a project brief around which students perform a series of tasks guiding to the delivery of a
final product that can be: a design, a device, a model, or a computer simulation (Guo, Saab, Post & Admiraal,
2020). Therefore, PBL engages students in active learning, promoting ideas discussion and sharing amongst all
team members. Throughout a semester, students are motivated to solve authentic, real-world problems
through a large interdisciplinary project development and integrating all the Curricular Units (CU). Teams are
formed and for each team a set of activities, tasks and milestones are planned. Each team tackles the project,
providing a solution and delivering in the deadline the product developed, as a prototype and a team-report
(Powell, 2004). The students will develop skills such as team work, leadership, ability to listen, participate
actively, collaboration, cooperation, respect other’s views, ability to critically analyse and evaluate literature,
focused study and use of resources, and mastery of presentation skills.
In this methodology, teamwork tutors are a distinguishing element, having as role stimulating discussion,
facilitating the process and reporting the team’s progress to the project coordinator, among others (Alves,
Moreira & Sousa, 2007; Alves, Moreira & Sousa, 2010). By asking the right questions, the tutor will lead the
student in learning, setting appropriate learning objectives and providing feedback on his/her learning. Hence,
PBL demands a mind shift in teachers, transferring in students’ hands their own learning responsibility.
This tutelage can be done by teachers included or not in the CU of the semester, researchers or upper classes
students (Simão, Flores, Fernandes, & Figueira, 2015). Few are the studies reporting on the tutor’s role
performed by students (Abbot, Graf & Chatfield, 2018; Alves, Moreira, Leão & Teixeira, 2017). The literature
reported one more case of PBL in an engineering-design course which presented a sort of peer tutoring
through what it was called “peer interactions”, but in a much more informal basis (Kuppuswamy & Mhakure,
2020). In Alves, Moreira, Leão & Teixeira (2017), each PBL team has two tutors: one teacher and one third year
student.
In this paper, the tutor’s role from the students’ viewpoint is described, within the first year of the Industrial
Engineering and Management (IEM11) degree at the University of Minho. It is based on an online questionnaire
composed by a total of 37 questions to the students that went through the PBL. Towards the semester’s end,
students’ feedback was also collected by means of a focus group concerning the tutor’s role. Furthermore, a
first-person narrative of two tutors-teachers is highlighted. One tutor had her first experience in this role, the
other has seven years of experience.
This paper is organized in five sections. This section introduces the objectives of the paper followed by a brief
literature review regarding the tutor’s role in PBL. Then, the research methodology and the tools used to collect
data are presented. The main results found are presented and discussed. Ending the paper with the final
remarks.
2 PBL: brief description

This section describes the context of PBL in engineering and the corresponding role played by the tutor,
particularly at the first year of the Industrial Engineering and Management program at University of Minho.
2.1 PBL in engineering

PBL model has been implemented at University of Minho since 2004_05, at the Industrial Engineering and
Management Master Integrated program of the first year, first semester. Since then, PBL has been improved
through continuous action-research-practice cycle, engaging all stakeholders, from students (Alves & Leão,
2015; Fernandes, Mesquita, Flores, & Lima, 2014) to researchers and teachers (Alves, Sousa, Fernandes,
Cardoso, Carvalho, Figueiredo & Pereira, 2016). PBL engages students in active learning, emphasizing ideas
213
discussion and sharing amongst team members. The students are motivated to solve authentic, real-world
problems during a semester, integrating all the CU through a large interdisciplinary project development.
Within PBL methodology, teamwork tutors have to be emphasized by their important role: making questions;
stimulating discussion; facilitating the process; motivating the teams and reporting to project coordinator the
team’s progress. The tutor should, as well, keep the team focused on the set of activities, tasks and milestones
established to achieve project objectives.
In 2016_17 academic year, the tutor’s role, which until then had been exclusively performed by teachers, was
also voluntarily assumed by third-year students (Alves, Moreira, Leão & Teixeira, 2017). These authors,
investigated the tutor’s role from two perspectives: 1) the tutored students and 2) the tutors. Additionally, the
differences between the tutored perspectives from tutors-students and tutors-teachers were also analysed.
The teams perceived the tutors-students as “godfathers” and as someone who has carried out similar
experience and difficulties. Therefore, this paper introduces the concept of peer tutors as a novelty, being
carried out by third-year students from the same program.
Falchikov (2001) identified four peer tutoring types in higher education: 1) students tutoring other students in
the same level and class; 2) students tutoring other students in the same class, but with a special status specified
by the course instructor; 3) students tutoring other students in the same institution, but at a different level; and
4) students tutoring students at different levels and from different institutions. The peer tutoring type described
in Alves, Moreira, Leão & Teixeira (2017) is the “students tutoring other students in the same institution, but at
a different level”.
Interestingly, Abbot, Graf & Chatfield (2018) studied the same peer tutoring type through a case study. Within
the peer tutors’ understanding and satisfaction with their tutoring experiences, the authors identified the
importance of three aspects: the professor-student relationship; role clarity and expectations, and tutor
positionality (tutors navigate between students and instructors). They concluded that tutors would support the
students better, enjoy the tutoring experience and feel connected to the course’s purpose, when all the
stakeholders involved (tutors, students and professors) have the perfect notion of the tutor’s role. Additionally,
they emphasized that the peer tutors perceived themselves as liaisons, intermediaries, and connectors, bridging
the world of professor and student.
Kuppuswamy & Mhakure (2020) presented the use of PBL in an engineering-design course, offering students
the opportunity to experience engineering design like the way how it was practiced in real industry context.
The students were supported through formal and informal lecturer’s consultations, workshop activities and
through “peer interactions” - where students use the other student’s thinking as resources, as a kind of peer
tutoring, but on an informal basis. Also, Rivaldi, Megayanti & Aryanti (2020) investigated the digital peer
tutoring in engineering education, particularly in engineering drawing subjects. According to them, with peer
tutoring, students felt more supported when they were ashamed to ask the tutor about some difficulties or
even some non-understanding, because their peers translated knowledge in their language, and
simultaneously the tutors felt helped.
2.2 Tutor Role

In PBL, the tutor’s role is essentially related with following the progress of a student’s team and the associated
activities leading to the suitable project development. The tutor will monitor the team activity, regarding
conflicts amongst teammates and other subjects, will apply a set of activities related to project management,
like the ones associated to project planning, to tasks’ division and allocation, meetings realization, and so on,
in order to accomplish the project objectives. The tutor likewise reports on the teams’ progress to the
coordinating team and offers a complementary perception on the team members’ work behaviour relatively to
the ones of the CU lecturers.
In this context, team success on accomplishing the project to be developed, depends on the tutor’s persistent
and paced activity, ensuring the project progress within safe limits and recording decisions and progress made
(Cornejo-Aparicio et al., 2019; Grunefeld & Silén, 2000). The tutor is a facilitator (Demirören, Turan & Teker,
2020) and should enable a fluid process of project management, as stated by Powell (2004). According to him,
214
the tutor should propose strategies, helping the team to generate ideas and identify important points to follow
and guiding efforts in order to progress the work and triggers its development, instead of contributing with
answers and solutions. Thus, attributes for tutors like being a good listener, making the right questions and
guiding the efforts look to be important attributes.
The tutor making the right questions, in PBL, will guide the students to search for knowledge and thus allowing
them to fully engage in the construction of their knowledge, as they are the focus of the learning activity
(Demirören, Turan & Teker, 2020; Imaz, 2015). This construction will be done through several sources, such as
the CU lecturers’ support, which should be consulted as learning resources during the PBL process deployment.
As well the tutor will support similarly, but his/her main goal is to make the right questions to the team, being
this much more important than give answers, because it will trigger the creative process of the team. The tutors
should guide the team towards the CU lecturers and also to external entities, whereas the lecturers will provide
technical guidance and knowledge. Thus, according to Amamou and Cheniti-Belcadhi (2018), the tutor will act
as a mediator among the students and the objects of knowledge which represent the knowledge to achieve.
In PBL, maintaining a suitable motivation throughout the project is the team’s most significant challenge
(Demirören, Turan & Teker, 2020), because a number of difficulties might happen in some project phases,
leading to disappointment and to frustration. Thus, throughout the project, and in order to guaranteeing a
uniform motivation of the team, one of the tutors’ challenges is to support the team (Alves, Moreira & Sousa,
2007). These authors presented as well the students’ perceptions concerning the practice of the tutor’s role at
the interdisciplinary IEM1 PBL at the University of Minho: a) availability (to meet with team); b) contribution to
solve conflicts amongst team members; c) facilitating debate within the team; motivation and trust on the work
carried out by the team; e) providing directions for pursuing the project tasks. Later on, Alves, Moreira & Sousa
(2010) researched the similarities and dissimilarities of the tutor’s experience between different years of the
IEM curricula. The findings demonstrated that in PBL of advanced academic years, the teams have a tendency
to be more autonomous and have a broader confidence on their solutions. Furthermore, conflicts on the teams
look to be less challenging. They also reported that tutors appear to perform differently between their peers,
which appear to be related to each individual distinctiveness.
According to Weenk, Govers & Vlas (2004) attending that the tutor’s role is not straightforward to achieve, the
tutor should have previous training on tutoring teams, specifically regarding competencies’ development on
problem solving and conflict management, whereas guaranteeing that the process learning within the team is
done on a cooperative basis. The authors also suggest as a main aspect on tutor’s role that his/her job is not
to offer technical solutions nor direct the project, but instead encourage students to learn, think and solve
problem by themselves, in order to stimulate their autonomy.
3 Research Methodology
This section describes the sample and the methodology followed to gather data to fulfil the main objective
obtained from: (i) students’ questionnaire, (ii) focus group; and (iii) tutors-teacher first person narrative.
Based on tutored students’ answers, the main research paper questions focused on this study are:
• What are the relevant tutors’ attitudes in the context of PBL?
• Do teachers and students play tutors’ role in the same way?
The data collected was analysed by non-parametric test (Mann-Whitney, U, for the comparison of two
independent samples means) since they do not follow normality (normality verified by Shapiro-Wilk test). The
statistical software SPSS 28.0 was used for the analysis.
3.1 Sample Characterization

The majority of enrolled students (approximately 65%) voluntarily participated and fulfilled the questionnaire,
83% was of female gender, and at least one student of each team fulfilled the questionnaire.
215
3.2 Questionnaire description
The questionnaire used in this study “Learning by project: tutor’s assessment” was developed and validated in
previous study using the population for which it has been developed and validated (Alves, Moreira, Leão &
Teixeira, 2017). Succinctly, the questionnaire has 37 questions, separated in ten sections according to the topic
in evaluation: 34 closed questions on a 5-levels Likert scale base (1-totally disagree to 5- totally agree), two
open questions and final general question in a 10-levels of agreement. The “not applicable” option was also
considered, upholding specific situations where the topic in evaluation was not applicable to tutors-students.
Each student, voluntarily, fulfilled a questionnaire for each tutor in their group. In Table 1 the most applicable
questions used to answer the research questions are listed.
3.3 Focus group

The focus group that occurred at the end of the semester, allowed to gather students’ feedback regarding the
PBL functioning. The present students, divided into five focus groups, apart from the tutor role, discussed also
other topics too, namely: peer assessment, CU integration in PBL project, project networking, teachers’ support
and feedback, and final marks. All these topics were discussed for four hours following the presentation of the
summary of how the semester run, in a nice and informal environment.
3.4 First person narratives

First person narratives of two tutors-teachers are presented: one tutor had her first experience as a tutor
(identified as Tutor1), the other has seven years of experience (Tutor2). These narratives describe perceptions
regarding PBL knowledge and motivation to be a tutor-teacher, difficulties felt and attitudes to overcome them,
and general opinion. Another worth mention aspect is that these two tutors are two of the authors of this
paper.

This section depicts the results of the study based on the quantitative analysis of the students’ answers and
student’ focus group regarding the tutor’s attitudes and role, and tutors-teacher first person narrative of their
experience as tutor.
4.1 Tutors’ attitudes and role: Questionnaire analysis

The data collected was analysed by non-parametric test Mann-Whitney, U, for the comparison of students’
perceptions regarding the two tutors’ (teacher and student) attitude and role.
Table 1 presents the corresponding statistics in terms of mean and standard deviation, minimum, maximum,
and median values obtained according to the questions defined for the quantitative analysis. On average, and
regarding the topics under study, students demonstrated to be in agreement for both students (T-S) and
teachers (T-T) tutors (mean values around 4 and median values equal 5 in most questions).
However, in three topics, the difference in agreement is statistically significant (based on the obtained 𝑝-value,
where the lower the 𝑝-value, the greater the statistical significance of the observed difference), showing a
different tendency on how students see teachers and students tutors’ role, i.e.:
26. “the tutor discussed with the team the results of peer assessment, of the team and auto-peer
assessment” (𝑝 < 0.005), where tutors-teachers is rated highest at median 5, with mean 4.64 close to "totally
agree" (Figure 1.a);
27. “the tutor emphasized the importance to create internal mechanisms that support the team
working operation” (𝑝 < 0.05), where students evaluate more positively the tutor-student at median 5, with
mean 4.63 close to "totally agree" (Figure 1.b);
17. “the tutor supported the team in the accomplishment of the objectives” (𝑝 < 0.06), nevertheless
students evaluate both students and teachers with the same median (5), tutors-students presents a higher
mean (4.63).
216
Table 1. Questions used in the quantitative study and relevant statistics.
𝑛𝑇−𝑆 𝑛𝑇−𝑇 𝑛𝑇−𝑆 𝑛𝑇−𝑇 𝑛𝑇−𝑆 𝑛𝑇−𝑇
Statistic
Section Questions = 34 = 32 = 34 = 32 = 34 = 32
U
mean s.d. min; max median
The tutor…
4. communicated clearly 4.74 .51 4.69 .74 3; 5 2; 5 5 5 527.5
5. showed enthusiasm in the performance of
4.38 1.2 4.38 .94 1; 5 2; 5 5 5 469.5
his/her role 1; 5 3; 5 4 5 465.0
4.26 .93 4.41 .72
III. Attitudes 8. was prepared for the meetings 3; 5 3; 5 5 5 422.0
4.44 .76 4.65 .66
9. supported, instead, of lecturing
10. stimulated a safety environment where the 4.84 .52 4.68 .70 3; 5 2; 5 5 5 433.5
students feel they are will to express their
ideas
The tutor…
11. put challenges and relevant questions 4.16 1.1 4.39 .76 3; 5 2; 5 4.5 5 464.5
13. stimulated the team thinking in different
4.33 1.1 4.65 .71 1; 5 2; 5 5 5 434
points of view 2; 5 3; 5 5 5 471.5
4.59 .84 4.77 .50
16. provided feedback when
IV. Project necessary/solicited 4.63 .87 4.31 .97 2; 5 2; 5 5 5 396.0 *
monitoring 17. supported the team in the accomplishment
of the objectives 4.44 .96 4.34 1.0 1; 5 2; 5 5 5 514.5
18. stimulated the team members to an active
participation 4.55 1.1 4.39 .96 1; 5 2; 5 5 5 442.0
20. helped the team to distinguish the
essential from the accessories
The tutor…
V. The
22. encouraged the team to critically analyse
critical 4.29 1.0 4.59 .68 1; 5 3; 5 5 5 390.0
the information
thinking
23. emphasized that all members of the team
development
are responsible by the process and project 4.59 .71 4.53 .80 2; 5 2; 5 5 5 503.0
and
final result
problem-
24. stimulated the autonomy development of 4.63 .72 4.45 .85 3; 5 2; 5 5 5 411.0
solving
the students
The tutor…
26. discussed with the team the results of peer
assessment, of the team and auto-peer
3.62 1.4 4.64 .87 1; 5 1; 5 4 5 202.0 ‡
assessment
VI.
27. emphasized the importance to create
Teamwork
internal mechanisms that support the team 4.63 .75 4.21 .94 2; 5 2; 5 5 4 338.5 **
function
working operation 4.53 .90 4.28 .92 1; 5 2; 5 5 4 348.0
29. tried to orient the team when they feel lost
31. had an active role in the dynamism of the 4.09 1.0 4.06 1.1 1; 5 1; 5 4 4 496.0
team
The tutor…
VII.
33. insisted in an equal division and task
Individual 4.24 1.0 4.031.1 1; 5 2; 5 5 4 454.5
turnover in order to all members develop
learning
project competences
How you classified, in general, the tutor 8.54 2.4 8.42 1.9 2; 10 3; 10 10 9 484.0
X. General
performance?
‡ statistical significant differences for 𝑝 < 0.005; ** statistical significant differences for 𝑝 < 0.05; * statistical significant differences for 𝑝 <
0.06; s.d. standard deviation; T-S Tutor-Student; T-T Tutor-Teacher
It is worth to emphasize that questions 26 and 27 are concerning the teamwork function and question 17 to
the project monitoring. The last two results to questions 27 and 17, could refers that students-students present
more facility in dialogue than students-teacher, as if they speak the “same language”.
Concerning the attitudes, students assess positively both students and teachers’ tutors (median 5, with the
exception of 4 for tutors-students in question 8). That means that the communication between students and
tutors is clear, showing enthusiasm in the performance of his/her role, demonstrating an early preparation of
the meetings, stimulated a friendly but focused environment where students feel they are will to express their
ideas, and that a tutor supported, instead, of lecturing. This last topic is very important issue being in
accordance to Amamou and Cheniti-Belcadhi (2018).
217
T-T
Tuto r to be evaluated
T-S
26. “the tutor discussed with the team the results of peer assessment, of 27. “the tutor emphasized the importance to create internal mechanisms
the team and auto-peer assessment” that support the team working operation”
Figure 1. Students’ answers distribution for question 26 (a) and 27 (b) for both tutor-student (T-S) and tutor-teacher (T-T).
Also, the results show how students positively perceive the tutors role regarding critical thinking development
and problem-solving and the importance that this has in continuously motivate the team (Demirören, Turan &
Teker, 2020; Alves, Moreira & Sousa, 2007).
4.2 Focus group

The students present in the closure PBL semester meeting were divided into five focus group, discussed the
tutor role among other topics, as previously mentioned. The following are particular points of focus in respect
of providing the general opinion of students related to tutor role in PBL project development.
Students presented distinct opinions regarding the tutor role, sustaining the answers given in the
questionnaires as previously analysed.
The negative opinions are regarding the less positive experience with the tutor. For example, the absence of
one tutor (particularly the tutor-student) throughout the semester. Nevertheless, broadly and unanimously
students agree on the importance of the tutor presence and role in a PBL project development.
In PBL project, tutor concept encompasses the binomial teacher and third-year student.
4.3 First person narratives

Tutor1, as her first experience as a tutor-teacher, refers that this was a very interesting and challenging one:
“My first contact with PBL was as a teaching professor in a CU, on a previous PBL edition.”
Her knowledge about PBL ranged from this later experience to conversations with the other counterparts
regarding PBL purpose and functioning. Another challenge embraced was tutoring two teams.
By contrast, Tutor2, besides the experience in recent years as tutor of a PBL team, is a teacher from different
engineering area than the Industrial Engineering and Management. This is a challenging and motivating
experience, where the creation of new ideas and to work with freshman students are the reason.
Both agree and identify the difficulty inherent to the first contact. Students show some shyness and felt lost,
not knowing what to do and what is expected to do in a PBL project. It was interesting to acknowledge that
the students’ perception about the tutor’s role was somewhat far from what is a PBL tutor role. As tutor, the
first task is to define the tutor meaning in a PBL project, as students thought that tutor would present them
with solutions and decide the way forward in the project:
“... first, I (Tutor1) began by reading the project’s guide with them, particularly the tutor’s role, the
project description (theme, project context, project objectives) and skills to acquire. Then, I reinforce
that a PBL tutor must be a facilitator, an expert using his/her knowledge mindfully within the project
context, a mentor, a team builder who is there to guide the team for the project solution and, that
is someone they can turn to, in case of doubts, problems and would help in conflicts, if they occur.”
218
After beginning the PBL project, to support the team, weekly meetings were planned. Accordingly, the students’
behaviour, tutors-teacher adjusted their approach and take action. For example, make questions in order to
understand students’ rationale thinking process and to identify the type of research done. Consequently,
suggestions about knowledge supports and database they should delve into. Based on Tutor2 experience, this
questioning process happen or not, in line with the students’ autonomy level. Eventually, the students make a
lot of questions, and discussions in order to drive their path within the project solution, when they were more
comfortable with the PBL project, creating group dynamics and trust between all members.
Tutor1, after this experience, identified some improvement points, namely: scheduling some kind of extra-
curricular activity or some informal event, outside the project room environment, to get to know each other
and to create bonds between all the team members. Regarding this extra-curricular activity, Tutor2 mentioned
that had opposite experiences: one, where all together went for an informal tea/coffee break at the end of the
semester, and another where the semester ended in a normal way.
Tutor1 refers that the PBL tutor role is an opportunity to work closely with students, observing their behaviour,
shaping their learning in real-time and within the dynamic setting of the project room. Both tutor-teachers
agree that:
“... it is an experience to be repeated.”
5 Conclusion
This study presents the main results in order to analyse how PBL students perceived the tutors’ role. Thus, to
fulfilled this objective two research questions were defined: “What are the relevant tutors’ attitudes in the
context of PBL?”, and “Do teachers and students play tutors’ role in the same way?”. The data were obtained in
three different ways: a students’ questionnaire, a focus group and a tutors-teacher first person narrative of
their experience as tutor.
Generally speaking, students perceived positively the tutors' role in the PBL project development where
communication has a large part to play in this process. For this reason, students see the tutor-student more
near them as a partner, and the tutor-teacher as the more-experienced partner. Additionally, tutors are
recognized as a key element, essentially in a first year of a programme, due to the inexperience of students in
a new environment and learning process. However, tutors, students or teachers, performance is always
dependent on the skills and motivation of the tutor to assume this role. Maybe, as future work, a kind of
interview could be made to tutors to realize about their intrinsic motivation and select them based on this
interview.
Acknowledgments
Scope: UIDB/00319/2020.
References
Abbot, S., Graf, A. J., & Chatfield, B. (2018). Listening to Undergraduate Peer Tutors: Roles, Relationships, and Challenges.
International Journal of Teaching and Learning in Higher Education, 30(2), 245–261.
Alves, A. C., & Leão, C. P. (2015). Action, Practice and Research in Project Based Learning in an Industrial Engineering and
Management Program. ASME 2015 International Mechanical Engineering Congress and Exposition, Volume 5:
Education and Globalization. https://doi.org/10.1115/IMECE2015-51438
Alves, A. C., Moreira, F., Leão, C. P., & Teixeira, S. (2017). Tutoring experiences in PBL of industrial engineering and
management program: Teachers vs students. ASME International Mechanical Engineering Congress and Exposition,
Proceedings (IMECE), 5, 1–8. https://doi.org/10.1115/IMECE2017-71306
Alves, A. C., Moreira, F., & Sousa, R. M. (2007). O papel dos tutores na aprendizagem baseada em projectos: três anos de
experiência na Escola de Engenharia da Universidade do Minho. Libro de Actas Do Congreso Internacional Galego-
Portugués de Psicopedagoxía. A.Coruña/Universidade Da Coruña: Revista Galego-Portuguesa de Psicoloxía e
Educación., (1), 1759–1770. Retrieved from http://repositorium.sdum.uminho.pt/handle/1822/18278
219
Alves, A. C., Sousa, R. M., Fernandes, S., Cardoso, E., Carvalho, M. A., Figueiredo, J., & Pereira, R. M. S. (2016). Teacher’s
experiences in PBL: implications for practice. European Journal of Engineering Education, 41(2), 123–141.
Alves, A., Moreira, F., & Sousa, R. (2010). Aprendizagem baseada em Projectos Interdisciplinares em Engenharia Industrial:
dissimilitudes de tutoria entre o início e o final do curso. PAEE2010-International Symposium on Project Approaches
in Engineering Education, 1, 133–139.
Amamou, S., & Cheniti-Belcadhi, L. (2018). Tutoring in Project-Based Learning. Procedia Computer Science, 126, 176–185.
https://doi.org/10.1016/j.procS.2018.07.221
Beneroso, D., & Robinson, J. (2022). Online project-based learning in engineering design: Supporting the acquisition of
design skills. Education for Chemical Engineers, 38(March 2021), 38–47. https://doi.org/10.1016/j.ece.2021.09.002
Cornejo-Aparicio, V., Flores-Silva, S., Bedregal-Alpaca, N., & Tupacyupanqui-Jaen, D. (2019). Capstone courses under the
PBL methodology approach, for engineering. EDUNINE 2019 - 3rd IEEE World Engineering Education Conference:
Modern Educational Paradigms for Computer and Engineering Career, Proceedings.
https://doi.org/10.1109/EDUNINE.2019.8875803
Demirören, M., Turan, S., & Teker, G. T. (2020). Determinants of self-regulated learning skills: The roles of tutors and
students. Advances in Physiology Education, 44(1), 93–98. https://doi.org/10.1152/ADVAN.00121.2019
Edelbring, S., Alehagen, S., Mörelius, E., Johansson, A., & Rytterström, P. (2020). Should the PBL tutor be present? A cross-
sectional study of group effectiveness in synchronous and asynchronous settings. BMC Medical Education, 20(1), 1–
6. https://doi.org/10.1186/s12909-020-02018-3
Falchikov, N. (2001). Learning together: Peer tutoring in higher education. New York, NY.: Routledge.
Fernandes, S., Mesquita, D., Flores, M. A., & Lima, R. M. (2014). Engaging students in learning: Findings from a study of
project-led education. European Journal of Engineering Education, 39(1), 55–67.
Flumerfelt, S., Alves, A. C., & Khalen, F.-J. (2014). Lean Engineering Education: The DNA of Content and Competency.
Proceedings of the 2014 IIE Engineering Lean and Six Sigma Conference Lean.
Graham, R. (2018). The global state-of-the-art in engineering education. Retrieved from
https://rhgraham.org/resources/Phase-1-engineering-education-benchmarking-study-2017.pdf
Grunefeld, H., & Silén, C. (2000). Problem based learning compared to project organized learning. Enschede, Universiteit
Twente.
Guo, P., Saab, N., Post, L. S., & Admiraal, W. (2020). A review of project-based learning in higher education: Student
outcomes and measures. International Journal of Educational Research, 102(May), 101586.
https://doi.org/10.1016/j.ijer.2020.101586
Imaz, J. I. (2015). Aprendizaje Basado en Proyectos en los grados de Pedagogía y Educación Social: “¿Cómo ha cambiado
tu ciudad?” Revista Complutense de Educación, 26(3), 679–696. https://doi.org/10.5209/rev_rced.2015.v26.n3.44665
Khamchiyev, K. M., Batyayeva, Y. K., Shandaulov, A. K., Zhashkeyeva, A. M., Suleimenova, F. M., & Sagimova, G. K. (2020).
The role of the tutor and students in the process of Problem-Based Learning (Pbl). Systematic Reviews in Pharmacy,
11(12), 388–393. https://doi.org/10.31838/srp.2020.12.63
Kuppuswamy, R., & Mhakure, D. (2020). Project-based learning in an engineering-design course - Developing mechanical-
engineering graduates for the world of work. Procedia CIRP, 91, 565–570. https://doi.org/10.1016/j.procir.2020.02.215
Powell, P. C. (2004). Assessment of team-based projects in project-led education. European Journal of Engineering Education,
29(2), 221–230.
Prince, M., Felder, R., & Brent, R. (2020). Active Student Engagement in Online STEM Classes: Approaches and
Recommendations. Advances in Engineering Education, 8(4), 1–25.
Prince, M. J., & Felder, R. M. (2006). To state a theorem and then to show examples of it is literally to teach backwards.
Journal of Engineering Education, 95(2), 123–138. Retrieved from
http://mate.calpoly.edu/media/files/Review_inductive_learning.pdf
Prince, Michael, & Felder, R. (2007). The Many Faces of Inductive Teaching and Learning. Journal of College Science Teaching,
36(5), 14–20.
Rivaldi, D., Megayanti, T., & Aryanti, T. (2020). Digital peer tutoring in engineering education. IOP Conference Series:
Materials Science and Engineering, 830(4). https://doi.org/10.1088/1757-899X/830/4/042070
Simão, A. M. V., Flores, M. A., Fernandes, S., & Figueira, C. (2015). Tutoring in higher education: Concepts and practices.
Sísifo Educ. Sci. J., 7, 73–86.
Weenk, W., Govers, E., & Vlas, H. (2004). Training in project-based education: practise as you preach. European Journal of
Engineering Education, 29(4), 465–475. https://doi.org/10.1080/03043790410001716301
220
PBL to design a Gamified Financial Management Application for
the socially vulnerable
Isabel B. A. de Souza1, Bruno L. Borba1, Alexandre S. C. Costa1, Edgard Costa Oliveira1, Simone B. S. Monteiro1
1
University of Brasilia, Brasilia, Brazil
Abstract
This paper presents the prototype of "MobEdu": a gamified app that responds to the need for financial management
training, for a community in state of vulnerability in Brazil, with low schooling and semi-literacy, by using engagement
strategies. This work was developed in partnership with five different courses conducted via Problem-Based Learning (PBL)
methodology, in the Industrial Engineering course of the University of Brasília. The results presented here were obtained
by students of the Production Systems Project 2 (PSP2) course, who developed exploratory research, with a qualitative
approach, using FIGMA as a prototype design tool. The steps of the research were: requirements elicitation, development
of a system vision document, user flow mapping and use case diagrams, and finally a prototype interface. Gamification
strategies can be key elements to engage people to use an app, and this proposal was developed to promote a high
engagement of the vulnerable community in a financial management application.
Keywords: Mobile Learning, Gamification, Financial Education, Interface Development, Interdisciplinary Project.
1 Introduction
In January 2018, the largest dumpsite in Latin America shut down its activities, submitting more than 1,800
waste pickers to a compensatory monthly salary payment flow of garbage cooperatives. The waste pickers
community was working autonomously before, and now they have been subjected to this new mode of
payment, without prior knowledge of basic financial management. In this context, researchers of the University
of Brasilia identified an opportunity to develop a financial management tool, aimed at communities in a state
of vulnerability, with low education and semi-illiterates.
According to Vieira, Moreira Junior and Potrich (2019), Brazil’s financial education is being highlighted due to
the Estratégia Nacional de Educação Financeira (ENEF), a national strategy that promotes financial education
and social security, contributing to the strengthening of citizenship and conscious decision-making by
costumers (BRASIL, 2010), developed by the Central Bank of Brazil. This was brought to practice together with
another trend in education: gamification. It can be done via a strategy to make people and their organizations
interact with background content and information needed for decision-making, by playing a ludic game that
makes learning a challenging and funny experience (FADEL, 2014).
In addition to the development of the tool's interface, it has been noticed that challenge-based gamification
improves the outcome of learning (Legaki, 2020), therefore, the format of engagement became as essential as
the user interface. As a result, the application was incremented with a gamified journey, aiming at the greater
engagement of the target audience in the usage of the tool´s features.
This paper focuses on the results of the Production System Project 2 (PSP2) course, which has delivered a
prototype of an app, and it is presented in four topics: i) theoretical references, with a theoretical basis; ii)
methodology used to create the application, which unifies mobile education, financial education, and
gamification; iiv) results and iv) conclusions.
2 Theoretical References
A literature review was conducted to show an overview of the subject matter, and to understand the state of
the art and how it could relate to the objective of this research.
221
This project was developed in the context of the New Epic - SDG Challenge8, a program that takes place every
year, coordinated by a group of professors and students from the University of Aalborg, Denmark, and the
University of Brasilia, Brazil. According to the UN (United Nations), the SDGs (Sustainable Development Goals)
are the main focus of the 2030 Agenda, united into 17 goals, each goal represents main activities that should
be considered for the achievement of the 2030 Agenda’s purpose: guarantee human development and the
basic needs of the citizen of the whole world through an economic, political and social process that respects
the environment and sustainability in the present and the future. The projects were developed in accordance
with one of those goals.
This paper presents the project “Mobile Education,” related to target 2 of SDG 10 – Reduced Inequalities - “By
2030, empower and promote the social, economic and political inclusion of all, irrespective of age, sex, disability,
race, ethnicity, origin, religion or economic or another status”. To promote empowerment and social and
economic inclusion, it is necessary to teach waste pickers how to manage their money correctly.
This project was developed in the realm of interdisciplinary projects. Interdisciplinary can be understood as the
crossing of existing disciplines resulting in a new one, especially in the scientific courses in the organizational
and the industrial subject fields. This is the case of Operational Research, which is an operational investigation
area that came across out of the fusion among scientists, engineers, and the military (Pombo, 2008).
Interdisciplinary projects are the result of the fusion of project ideas that float around in different areas, that
when brought together to develop a solution, can be mutually beneficial in finding methods and tools to
achieve their goals. Interdisciplinary projects are a trend in modern education and university courses are being
designed to meet this goal: to bring students and professors of different knowledge areas or courses working
together, in learning by doing manner. The work in an environment that is project-based education and
solution-driven initiatives, with cooperation in many different levels (EVANGELISTA; COLARES; FERREIRA, 2009).
Bayuk (2019) presents how smartphone apps can aid in the financial health of their end-users. The study was
conducted in an interdisciplinary manner, by exposing the group under review to game apps about finance
that has social tools, such as abilities to share achievements, or economic tools, such as the ability to earn real
money. The study found that consumers vary in their preferences for certain features based on their previous
experiences with other financial management apps, with those who had prior contact tending to exhibit greater
subjective knowledge and engage more with "social" and "economic" features of financial apps, while those
with no experience are more motivated by economic features.
Vieira, Bataglia and Sereia (2011) define financial education as measures that aim to create and transmit
financial information to individuals, by providing them the ability to distinguish the advantages and risks of
their choices. Moraes (2019) explains that the main objective of financial education is to instruct people on
how to manage their monetary resources, by helping them make conscious decisions that enable them to save
and invest, and ensuring that they live well financially, whether in the present or the future.
According to Legaki et al. (2020), challenge-based gamification improves learning outcomes. As a result, it
showed that the performance of their study audience was improved by 34% through gamified education and
that better results can be acquired through gamification combined with traditional study methods such as
reading. However, any interaction with a gamified application, in a lecture, for example, can already be
considered beneficial to students. Peixoto et al. (2015) present in his study a systematic mapping of the use of
gamification in educational software in the context of Brazilian research and concluded that only thirteen
Brazilian institutions conduct studies in the field of gamification for education, and the software most used are
web-based systems.
3 Methods used
This paper is an outcome of a project developed under the Mega-theme "Mobile Education" of the SDG
Challenge. It involved five different courses from Production Engineering PSPs courses (i.e., Production Systems
8
The mission of the SDG-Challenge is to connect and mobilize students and organisations to work together on the Sustainable Development Goals (SDGs) of the United
Nations (UN). These goals are set to combat climate change and to reduce the inequalities in the world. New EPIC + SDG Challenge 2020.
https://www.sdgchallenge.com.br/
222
Projects) of the University of Brasilia (UnB): Production Systems Project 1, 2, 3, 5 and 8 (PSP1, PSP2, PSP3, PSP5
and PSP8), in an interdisciplinary manner. PSP1 carried out the data collection via interviews, thus supporting
the Production System Project 2 (PSP2) course in the requirements elicitation with stakeholders. PSP2 students
were responsible for the design of a prototype interface. PSP3 teams created the educational content that is
displayed in the application. PSP5 teams worked on the quality control of the prototype developed by PSP2,
and the PSP8 team managed the integration of the project portfolio, all those five courses were managed by
using PBL (Problem Based Learning) as a learning methodology.
The app´s interface was developed with a focus on a challenge-based gamification journey, to engage and
motivate the target users of the platform, based on the DML (Dynamic Model for Gamification in Learning),
presented by Peixoto and Silva (2015) and Kim (2013).
The focus of this research was the PSP2 course, responsible for the development of the app prototype named
“MobEdu”. An applied research was carried out, with a qualitative approach, and the techniques used for data
collection were based on querying an indexed database for requirements elicitation, in addition to the input
provided by the PSP1 course.
The research methodology was structured in 5 phases: (i) Development of a system vision document, (ii)
Requirements Elicitation, (iii) User Flow Mapping, (iv) Use Case Diagrams and (v) Prototype Interface.
4 Research Results
4.1 System Vision Document
A vision document was written by the PSP2 group in order to provide a high-level system view and more
detailed technical requirements of the application, such as scope and non-scope, app building team, functional
and non-functional requirements (represented visually in Figure 2), constraints, and a primary interface
prototype.
The vision document was essential to present to the stakeholders (professors and students) the steps taken to
elicit the functional and non-functional requirements via user´s needs and a literature review, to meet the
expectations, to adapt them to stakeholders’ suggestions, such as in the application's boundaries and the
interfaces, to build the application's prototype.
4.2 Requirements Elicitation

The system requirements specification was divided in two parts, based on Pressman (2009): the functional
requirements, which refer to all the features that the application must present, described in Table 1, and the
non-functional requirements, which refer to the services and restrictions presented by the system, described
in Table 2.
Table 1. System Functional requirements list.
Nº Name Functional Requirements Description.

RF001 Register and log in It allows performing user registration by storing information, user and password.
RF002 User Validation It allows to view and validate information about application users.
RF003 Change User It allows to view, change and delete information about application users.
RF004 Create Avatar It allows the creation a user's avatar.
RF005 Edit Avatar It allows to view and edits the user's avatar
RF006 Store Avatar It allows the storage and view of accessories purchased with fictional coins.
RF007 Control fictional coins. It allows to record and view historical earnings and spending of fictional coins.
RF008 Include Courses It allows you to insert courses into the app.
RF009 Keep Courses It allows changing existing courses in the app, as well as recording and viewing completed
courses.
RF010 Exclude Courses It allows deleting existing courses in the app.
RF011 Integrate family users It allows to create, change and delete a family group.
223
RF012 Integrate work users It allows to create, change and delete workgroups.
RF013 Keep financial control It allows to record, change, view and delete financial data.
RF014 Share link It allows sharing the application link with a certain personalized identifier for each user so that
you benefit only.
RF015 User Guide It allows solving app usability questions.
RF016 Control Notifications It allows controlling notifications for everyday alerts and fosters user engagement
Table 2. Non-functional requirements list.
Nº Name Non-functional Requirements Description

RNF001 Usability The system must make it easy for the user to understand the functionality available by using symbols. It
should have a simple presentation making the execution of actions intuitive and logical.
RNF002 Performance The system must have a maximum response time of 3 seconds.
RNF003 Storage The system must be able to store data both locally and in the cloud.
RNF004 Security The system must be able to register a password per user.
RNF005 Compatibility Must be compatible with Android 6.0 or higher operational systems.
RNF006 Legality The system must comply with the General Data Protection Law (LGPD) as well as restrictions for an
application by entering the terms of use, terms of the customer.
4.3 User Flow Mapping

The user flow mapping shows the main features that users need order to use the app. It was built according to
the system requirements specifications and shows the required user journey in order to achieve the functions
related to the financial education goals.
Figure 1. User Flow Map
Initially, the user opens the application, to log in or to create a new account. Then user must enter their main
data, create their avatar and search through their contact list to find other users in the app. Then the user is
directed to Home page of the application, and has the view of four main icons on the dashbar: Wallet, Plus,
Teams and Profile. In addition, the user can also navigate through the homepage and access other relevant
information such as: courses, medals, coins, etc
4.4 Use-case Diagram

The use case diagram, shown in Figure 2, was designed out the list of the functional requirements, presented
in section 3.2, and by analysing the business model.
The use-case diagram represents the interaction between the user and the system administrator, and their
main assignments within the application.
224
Figure 2. Use-case diagram.
4.5 Prototype Interface

The application was created to help waste pickers to take better care of their finances. The scenarios presented
in the application's prototype bring features and elements designed exclusively to better meet the needs of
those users, in total, 69 interfaces were created, however, only 22 will be presented below.
The designing and prototyping of the application, throughout the research, were created by using the following
tools: “Figma” - used to effectively design the screens and visualize them as a functional application; “Power
BI” – Used for testing the application's points and data entry system by displaying reports; “Abobe Illustrator”
– Used for designing the application-specific content; and “Miro” – used for the collaborative work of the
prototype development team, of a flowchart of the app´s screens.
4.5.1 Registration and Login

The Welcome screen was built to guide the user during the registration process in the application and to collect
key data, such as name, phone number, and identification number. After registering, it is possible to accept the
option to find friends and family who use the application. The user can also choose to indicate friends and
family by sending a link, from the moment the link is used, the main user obtains their first virtual coins. On
the Login screen, the user should fill in their identification number and password to enter the account. In the
case of forgetting the password, the user should choose the “forgot my password” button, receiving a
confirmation code by message to set a new password.
Figure 3.a Welcome, 3.b Login, 3.c Reward Pop-up
225
4.5.2 Creating an Avatar
In Figure 4, the user can create their avatar by choosing the skin tone, hairstyle and clothing pieces for the top
and bottom parts. When creating the avatar, the user is rewarded with virtual coins to spend on accessories in
the app store.
Figure 4.a Avatar creation, 4.b Avatar creation, 4.b Store, 4.c Reward Pop-up
4.5.3 Home page features

Figure 5. a presents the home screen, the screen offers the user an overview of the main information available
in the app, it presents monthly earnings and expenses, progress in the financial education courses, earned
coins (Figure 6.a), courses (Figure 6.b), medals (Figure 6.c) and store (Figure 4.c). Figure 5.b, presents the
navigation bar, where the user can navigate through other screens, such as the wallet (Figure 8.a), the team’s
screen (Figure 9.a), and the profile configuration screen (Figure 7.a).
Figure 5.a Home, 5.b Overview
4.5.4 Courses and Medals

Figure 6.a presents how the coins were obtained, after the registration, virtual coins can be obtained through
three steps: by indication, by adding data to the earnings or expenses report, or by watching the available
courses in the app. Figure 6.b presents the courses screen, where the user can access the financial education
content of the application. Figure 6.c presents the medals earned by finished courses, the user also has the
226
option to share the medal via WhatsApp (Figure 6.d), medals with a lock symbol can be obtained by completing
the equivalent course.
Figure 6.a Calendar, 6.b Courses, 6.c Medals Overview, 6.d Share your Medal
4.5.5 User´s profile features

Figure 7.a presents the registered user information. Figure 7.b presents frequently asked questions and
instructions for the app usage. Figure 7.c presents the settings option of the app: privacy, notifications, change
password, invite code, search contacts, term of the user and deactivate the account.
Figure 7.a Profile, 7.b Help, 7.c Settings
4.5.6 Financial Information

By clicking on the button with the plus symbol on the home page “+”, it is possible to add the expenses and
earnings data of the day (Figure 8.a). The “General” screen shows the available income, the average weekly
expenditure, and the expected expense for the month (Figure 8.b). Users can delete the financial records (Figure
8.d) by clicking on the pencil and notebook icon in the upper left corner of the dashboard (Figure 8.c). The
dashboard is divided into “Earnings” and “Expenses” and has a table with the date, value and reason for the
expense (Figure 8.d). The user can also listen to the dashboard analysis by clicking on the speaker located in
the upper right corner (Figure 8d).
227
Figure 8.a Financial Input, 8.b General, 8.c Delete Information, 8.c Dashboard
4.5.7 Family and Work Team

The Family and Work Team strategy is responsible for the integration between users. In this section, it is
possible to create teams and have visibility of the number of coins of each player.
Figure 9. Work Team
5 Conclusions
This project resulted in the prototyping of a gamified app that helps socially vulnerable communities develop
critical thinking and financial management abilities through financial management courses and tool that are
made available by the application. Those results were accomplished due to the integration between five
problem-based learning (PBL) classes of Production System Project (PSP) of the Production Engineering course
of the University of Brasília, whose students worked together to design a solution to participate in the Epic and
SDG Challenge, which is a program that takes place every year, coordinated by a group of professors and
students from the University of Aalborg, Denmark, and the University of Brasília, Brazil.
Besides the main objective, the project also had the purpose of developing students skills through problem-
based learning (PBL) and multidisciplinary critical thinking. Those goals were achieved through the integration
228
of courses that applied knowledge about project management, quality management, requirements elicitation
and product development.
The project was developed based on real needs, in addition to developing a solution to a latent social problem
in Brazil, the lack of training on financial management of communities in a state of vulnerability. Students at
the University of Brasília had the opportunity to develop their autonomy and creativity, in addition to applying
theoretical knowledge learned in the classroom, in a solution that brings value to Brazilian society and
cooperates with the strategic objectives of the UN 2030 Agenda.
Due to the social isolation imposed because of the COVID-19 pandemic, the project did not have a validation
phase with the target audience, waste pickers from the SLU. As a next step, the study will continue with a
hands-on real user experience with waste pickers to validate the application outcome and improve the project´s
results.
6 References
BAYUK, J. and Altobello, S.A. (2019), "Can gamification improve financial behavior? The moderating role of app expertise", International
Journal of Bank Marketing, Vol. 37 No. 4, pp. 951-975.
EVANGELISTA, I.A.S.; COLARES, M.L.I.; FERREIRA, M.A.V. Projetos educativos interdisciplinares na prática docente. Piauí: UFPI, 2009.
FADEL, M.L., Ulbricht V.R, Regina C., Vanzin T. (2014). Gamificação na Educação. São Paulo: Pimenta Cultural, 2014. 300p.
LEGAKI, Nikoletta Zampeta & Xi, Nannan & Hamari, Juho & Karpouzis, Kostas & Assimakopoulos, Vassilis. (2020). The effect of challenge-
based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer
Studies.
MORAES, F.A. Educação financeira: curso de capacitação na formação docente inicial. 2019. Dissertação (Mestrado Profissional em Ensino)
– Universidade Estadual do Norte do Paraná, Cornélio Procópio, 2019.
POMBO, OLGA. Epistemologia da Interdisciplinaridade. Ideação Revista do centro de Educação e Letras. UNIOESTENIOESTE Campus Foz
do Iguaçu. Uv. 10 - n1 - p. 9-401. 2008
PEIXOTO, M., Silva, C., Vilela, J., & Gonçalves, E. (2015). Um Mapeamento Sistemático de Gamificação em Software Educativo no Contexto
da Comunidade Brasileira de Informática na Educação.
PEIXOTO, M. M. e Silva, C. (2015) Requisitos para Softwares Educacionais Gamificados: Uma Revisão Sistemática de Literatura. In Anais do
18o Ws em Engenharia de Requisitos, Lima, Perú.
PRESSMAN, R. S. (2005). Software engineering: a practitioner's approach. Palgrave macmillan.
PRESSMAN, R., & Maxim, B. (2016). Engenharia de Software-8ª Edição. McGraw Hill Brasil.
Vieira, K. M., Moreira Junior, F. D. J., & Potrich, A. C. G. (2019). Indicador de educação financeira: proposição de um instrumento a partir da
teoria da resposta ao item. Educação & Sociedade, 40.
KIM, T. J., Lee, W. H. Dynamical model for gamification of learning (DMGL), Multimedia Tools and Applications, Springer US, (2013).
229
Seeding future engineers
Tatiane da Silva Evangelista1
1
Universidade de Brasília, Faculdade do Gama, Brasil
E-mail: [email protected]
Abstract
How to motivate female students to progress in the Engineering course? This question was the motivation for this work,
which aimed to introduce some women who have revolutionized the world around Exact Sciences to female students
attending the Engineering course at the University of Brasilia. This paper presents personal and professional bibliographic
results of four extraordinary women scientists in exact sciences: Ada Lovelace, Katherine Johnson, Marie Curie, and Hedy
Lamarr and, reports the results of the experience of this work developed in which it concludes benefits of self-esteem,
overcoming, courage and appreciation of the female gender, promoting the planting of female engineers.
Keywords: Women Scientists; Female Gender; Motivation; Exact Sciences.
1 Introduction
When we think of physics and mathematics, we remember physicists Galileo Galilei and Albert Einstein, and
mathematicians Leonhard Euler and Gottfried Leibniz, for example. In engineering we remember Leonardo da
Vinci and Gustave Eiffel. What do they all have in common? They are men in exact sciences. Where are the
women in this field? Historical accounts register that it was unusual to have women in the Exact Sciences,
because they were prepared to dedicate themselves to domestic chores and were not suitable for this training;
and those who had the opportunity to study in this area were because they had family financial support, but
the merit of their achievements was not attributed.
Figure 1 presents global statistical data of the most employment sectors for women from 1970 to 2019 (before
the start of the COVID-19 pandemic). In 1970, women made up 38% workers and 8% representatives in the
STEM fields (Science, Technology, Engineering and Mathematics). By 2019, the STEM proportion had increased
to 27% and women made up 48% of all workers, on what women also made up nearly half of those in all math
(47%) and life and physical science (45%) occupations.
Figure 1. Porcentage of women in STEM jobs: 1970-2019.
230
Currently, there has been an increase in the number of female scientists, but they are still a minority and are
still not valued in some fields. Why is this? Because career choice is due to cultural factors acquired in childhood
and adulthood rather than biological factors (Reis & Silva & Carvalhaes, 2016). For example, toys discriminated
by gender, that is, dolls for girls and cars for boys, generating less contact for them in parental activities that
have social culture linked to technological and exact themes (Mascarenhas, 2019).
In the studies by Matusovich & Streveler & Miller (2010), they show that the motivation for choosing an
engineering course is due to emotional values, acquired beliefs and discovery of personal identity, especially
for female students. On the other hand, intrinsic and social motivations are the motivational factors to attract
students to engineering education and in addition, women are significantly influenced by their mentors
(Kolmos, & Mejlgaard, & Haase & Holgaard, 2013).
Given this situation, the idea arose to expose to the female students of the Engineering course at the University
of Brasilia (UnB) some female scientists who revolutionized the world of Exact Sciences with their research,
because inspirations come from examples, right?
Thus, the objective of this article is to expose the personal and professional bibliographies of four women
scientists (Ada Lovelace, Marie Curie, Hedy Lamarr, and Katherine Johnson) and to report the motivational
experience of this exposition for the students of the Engineering course at UnB.
Initially, we will make a documental presentation of these researchers in chronological order. Next, we will
report the characteristics of the students involved in the Engineering course, where the work was developed.
Finally, we will show the results obtained and the conclusions.
2 Four women who changed the world of exact sciences

In the old days, restrictions on women's access to education were customary and the rare ones who had
admission to education could not publish their work because they were expected to be groomed to be only
good wives and good mothers and supported by their husbands and furthermore, society labeled that woman
were not talented in the exact field like men (Ygnotofsky, 2016). The four women in this article broke rules,
learned to listen to themselves and follow their dreams.
2.1 Ada Lovelace

On December 10, 1815, Augusta Ada Byron was born in England-UK, the daughter of poet George Gordon
Byron and mathematician Anne Isabella ("Anabella") Milbake - known as the princess of parallelograms. Her
parents divorced when she turned one year old. Thus, she lost the paternal bond and started living only with
her mother, who was responsible for her education by hiring excellent tutors and for influencing her in exacts.
When she was 8, her father passed away, leaving poetry as her second passion, as numbers became her great
admiration.
At 17, Ada met the old scientist Charles Baggage who was amazed by her intelligence. At the time he had
created the analytical machine (a device like a large clock full of gears for adding and subtracting numbers).
He became her mentor and thus began the emergence of the great mathematician Ada Lovelace.
In July 1935, at the age of 20, Ada married William 8th Baron King, Earl of Lovelace, and became known as Ada
Lovelace. They had three children, but at the age of 36 due to uterine cancer, she died and at his request, was
buried next to her father in Nottinghamshire, a county in England situated in the East Midlands.
After one hundred years of her loss, the poetic scientist's notes on Baggage's analytical machinery have been
recognized as a description of a computer and software, i.e., she is the first person to create a computer
program that took as its inspiration the punched cards used in mechanical looms at the time.
231
In her honor and in her recognition around exact sciences, in 1980 the U.S. Department of Defense created the
ADA programming language, and since 2009, every second Tuesday in October is celebrated as Ada Lovelace
Day, whose goal is to highlight women in science, technology, engineering and mathematics.
Figure 2. Image of Ada Lovelace (Source: https://pt.wikipedia.org/wiki/Ada_Lovelace ).
2.2 Marie Curie

Marie Salomea Skłodowska was born on November 7, 1867, in Warsaw-Poland in a period when her country
was under Russian rule. From an early age the words "How does this work? What is it? Why is it here? I have
an idea" became her jargon. With the death of her mother, at the age of 10, and three years later with the loss
of her older sister, she began to dedicate herself more to her studies and to science, which she was fascinated
with, but a hidden learning, since women were not allowed to attend universities.
In 1891, at the age of 24, Marie gathered money to attend the Sorbonne University in Paris and it was during
this period that she met the scientist Pierre Curie, in which they married in 1895 and became recognized as
Marie Curie and had two daughters. They were a couple with a passion for research and together they formed
a brilliant team. In the year 1903, they won the Nobel Prize in Physics for the discovery of radiation, which made
Marie the first woman to be awarded this prize.
The Curies knew that the effect of the radioactivity was making them ill, yet they did not give up scientific work.
In 1906, Pierre died in a carriage accident, and despite great sadness at the loss of her husband, she continued
research with the radioactive materials they had discovered: polonium and radium, named after the country of
Poland and the sun, respectively. These discoveries led Marie to receive her second Nobel Prize in Chemistry
in 1911, making her the first person to have two Nobel Prizes in different fields.
During World War I in 1914, she served as a volunteer nurse driving x-ray trucks to save and help wounded
French soldiers. In addition, she proved that radium is a powerful material for cancer treatment. She died of
leukemia at the age of 66, leaving great inspirations of courage and fraternity for scientists today.
Figure 3. Image of Marie Curie (Source: https://pt.wikipedia.org/wiki/Marie_Curie ).
2.3 Hedy Lamarr

Hedy Lamarr is the stage name of Hedwig Eva Maria Kiesler, an Austrian born on November 9, 1914.
From an early age, her beauty was admirable, which made her win several beauty contests and start her artistic
career at the age of 16. She had a very controversial personal life, was married six times, and had two children
with her third husband, John Loder. In addition, most of her husbands were wealthy and influential politicians,
for example, her first husband, Friedrich Mandi, in which she attended his meetings with researchers and
scientists that awakened and revived her interest in technological research.
In 1940, during World War II, Lamarr patented her first invention: a radio interference device to alter Nazi
torpedoes; the idea for which came about together with composer and friend George Antheil. This work was
232
the basis for the creation of wi-fi, Bluetooth, GPS, and military communication, i.e., she co-invented the
technology to use frequency hopping spectral scattering (FHSS).
In the last years of her life, she lived alone and did not like visitors, died at the age of 85 residing in Florida-
USA, received several awards in life and fourteen after her death, in 2014, entered the National Inventors Hall
of Fame.
Figure 4. Image of Hedy Lamarr (Source: https://pt.wikipedia.org/wiki/Hedy_Lamarr ).
2.4 Katherine Johnson

On August 26, 1918, African American Katherine Coleman was born in the small town of White Sulphur Springs,
USA. Due to the racial segregation of the time, she had many prejudices and difficulties in accessing education,
but this did not hinder her dedication to her studies.
At 15, she entered the university and was tutored by W. W. Schieffelin Claytor, the third African American to
earn a PhD in Mathematics. In 1939, Katherine at age 18 finished her undergraduate degree and was the first
black woman to complete a college degree.
In 1929, she interrupted her professional career to devote herself to her marriage and the three daughters she
had with her first husband James Francis Goble. In 1956, Katherine was widowed and, remarried in 1959 to
Lieutenant Colonel James A. Johnson, and became known as Katherine Johnson.
She returned to research in 1953 working for the U.S. Aerospace Department (now the National Aeronautics
and Space Administration - NASA), where she was responsible for man's flight to the moon in 1962. He then
dedicated himself to space work for NASA and retired in 1986, after 33 years of dedication to mathematics for
aerospace engineering. He received several awards and honors before he passed away on February 24, 2020,
at the age of 101, including the Presidential Medal of Freedom.
Figure 5. Image of Katherine Johnson (Source: https://pt.wikipedia.org/wiki/Katherine_Johnson ).
3 Description of the experience

In 2021, the idea of planting, sowing, and motivating female students emerged from the author's concern with
their demotivation in dropping out of the Engineering course at Faculdade do Gama (FGA), during the period
of the pandemic caused by COVID-19. FGA is an extension of the University of Brasilia (UnB) of Brazil, in which
it offers five engineering courses: aerospace, automotive, electronics, energy and software.
For this reason, an extension project entitled "Mathematical café for future female engineers" was created,
which took place in the period from May to December 2021 with weekly meetings in virtual format with the
participation of 50 students, in which the themes were directed:
• A bibliographical exposition of personal and professional women who have made historical
contributions and revolutionized the significant role in the engineering courses offered at FGA.
233
• The presentation of curious historical facts in the world of Exact Sciences.
• Emotional themes to mitigate the period of social isolation due to the COVID-19 pandemic, such as:
anxiety, depression, phobia, nostalgia, fear, etc.
Figure 6. Logo of the extension project Math Café for future female engineers.
4 Results and Analysis

To collect data from the extension project presented, the female students who participated in this activity
answered a brief questionnaire (voluntarily) that addressed the following questions:
(i) Did the presentation of women who shone in exact sciences motivate you to continue in the Engineering
course?
(ii) Did your perception of the Engineering course change positively with the exposure of the female scientists
of the project?
(iii) Were your emotional problems arising in the isolation period because of the COVID-19 pandemic
successfully alleviated?
(iv) Did you enjoy participating in the project?
(v) Among the four scientists studied, which one did you like the most? Why?
In addition, the questionnaire had a space for open answers so that the students could give their opinion,
suggest and/or criticize. Thus, the scale dimensions ranged from 1 (strongly disagree) to 4 (strongly agree).
Figure 7. Questionnaire response - questions (i) to (iv).
Based on Figure 7, it can be concluded that most of the students considered that the exemplification of
outstanding women in exact sciences motivated them to continue in the course (35 votes = 70%) and that their
perceptions/notions of the Engineering course became clearer and more motivating (100%). The emotional
themes worked on helped overcome the period of isolation experienced by the COVID-19 pandemic (40 votes
= 80%) and unanimously, all participants enjoyed participating in the project (100%).
234
Figure 8. Answer to question (v).
Figure 8 presents the scientists favored by the participants. Thus, listed in descending order were Katherine
Johnson (17 votes = 34%); Marie Curie (14 votes = 28%); Hedy Lamarr (10 votes = 20%) and Ada Lovelace (9
votes = 18%), respectively, the justifications were: example of race and courage; example of fraternity and love
for science; example of beauty and intelligence; example of achievement and conquest.
Below are four randomly open responses from some participants:

▪ Student 1: “The participation in this project was a great motivation for me to keep in my Engineering
course. In particular, highlight the example of scientist Marie Curie who encouraged me to study what I
love and not be afraid to always research what I don't know.”
▪ Student 2: “The project brought me a different perspective on women's skill in Exacts. I had my hope
renewed and my emotional worth. In addition, meeting all these women scientists are certainly great
motivational examples to attend the Engineering course.”
▪ Student 3: “Katherine Johnson and Hedy Hamarr were the scientists who motivated me the most, as they
were distinct examples of economic class and race, but both had courage and fought for their dreams.
Exacts are also for women!”
▪ Student 4: “The women scientists studied in this project improve my conception of myself, that is, my
values, my beliefs and my goals. In addition, the stories of these great women motivated me to have
courage in my choices and find the strength to never give up on my dreams.”
According to the data presented, it can be concluded that the personal and professional bibliographic study
of women scientists in the Exact Sciences was a great motivator for the permanence of female students in the
Engineering course, because women scientists were sources of inspiration and intrinsic motivation, which
converges with the results of Kolmos, & Mejlgaard, & Haase & Holgaard (2013). In particular, it is highlighted
that the values (captured from the brilliant women scientists studied): autonomy, mastery, purpose, courage,
achievement, perseverance and creativity, were sown, planted and will certainly always be cultivated by the
students participating project.
Therefore, it is concluded that real inspirations from women scientists who made history and shone in the field
of exact sciences can successfully attract women and/or young people to engineering with female
empowerment.
5 Conclusions
With the results obtained in this study, it was possible to observe that the students were motivated and excited
to stay in the Engineering courses at FGA, due to the many real-life accounts of female scientists who
revolutionized the area of Exact Sciences. The approach presented cleared up doubts and exposed curiosities
in the universe of exact sciences that stimulated the participants' curiosity. The emotional themes brought
emotional comfort for the period of isolation due to the 2021 pandemic of COVID-19.
The scientists presented mirrored in the students the joy of fundamental discovery, awakening the feminine
touch of each within themselves (Swaby, 2015). As such, it is believed that exemplifying women in exacta
enables a great benefit to plant and sow fruits of motivation, self-esteem, courage, overcoming, female
empowerment, and confidence so that SHE can follow the brilliant paths that these pioneers once explored.
235
For future activities, it is intended to carry out a local and global statistical survey on factors that are capable
of awakening more women in the exact sciences, especially in engineering.
Acknowledgements
I thank Research Support Foundation of the Federal District (FapDF) and University of Brasilia (UnB) for their
financial support.
References
Ignotofsky, R. (2016). Women in science: 50 fearless pioneers who changed the world. New York: Ten Speed Press.
Kolmos, A., & Mejlgaard, N.., & Haase, S., & Holgaarda, J. E (2013). Motivational factors, gender and engineering education. European
Journal of Engineering Education, 38 (3).
Mascarenhas, Y. P. (2019). Science in my life. Pure and applied chemistry, 91, 751-760. doi: 10.1515/pac-2018-0606
Matusovich, H. M., & Streveler, R. A., & Miller, R. L (2010). Why Do Students Choose Engineering? A Qualitative, Longitudinal Investigation
of Students’ Motivational Values. Journal of Engineering Education, 99 (4).
Reis, E. P., & Silva, G. M., & Carvalhaes, F. (2016). Apresentação: As dimensões sociológicas da desigualdade social no Brasil. Revista
Brasileira de Sociologia, 4(7), 5-12.
Swaby, R. (2015). Headstrong: 52 women who changed science – and the world. Portland: Broodway Books.
236
Conducting a Home Experiment Related to the Colligative
Properties of Chemical Solutions.
Javier Ramírez-Angulo1, Margarita Portilla-Pineda2, María del Carmen González-Cortés2
1
Department of Production and Systems, School of Engineering, Tecnológico de Monterrey, México
2
Department of Basic Science and Engineering, Universidad Autónoma Metropolitana, Mexico City, México
Email: [email protected], [email protected], [email protected],uam,mx
Abstract
The experimental part of teaching chemistry in a series of laboratory activities complements the fundamental learning of
the various topics covered in a chemistry course. However, alternatives are necessary when situations arise where it is
impossible to have a chemistry laboratory. Conducting home experiments can be of great help in teaching various
chemistry topics, as they can be used in tasks, research scenarios, and alternative laboratory practices. They can be applied
to different levels of teaching depending on their orientation and the contents they address. In the absence of a laboratory
to conduct chemistry practice experiments due to Covid-19 faculty and student confinements at home, we developed
feasible practices to do at home with ordinary homemade materials that were not risky or dangerous. We selected solutions
with desired colligative properties. Thirty engineering students taking two university chemistry laboratory classes
participated, coming from programs in the chemical area such as chemical engineering and biotechnology engineering,
among others. The teaching-learning process was conducted remotely via Zoom. The students appreciated the in-home
practices. Motivated to understand the topics, they performed well on the exams and the laboratory reports and issued
high opinions of the classes in the student survey.
Keywords: Concentration, solutions, colligative properties, higher education, educational innovation
1 Introduction
Chemistry courses are part of the curricula of most engineering careers in the various institutions of higher
education, such as Tecnologico de Monterrey or the Autonomous Metropolitan University. They are usually
composed of theoretical and experimental parts, the latter occurring with practices of specific course topics in
a chemistry laboratory.
However, it is not always possible to have a chemistry laboratory for the experimental phase of a chemistry
course due to unforeseen (or even foreseen) situations for social, administrative, or medical reasons.
In this case, technological tools such as the Zoom videoconferencing application can facilitate convenient and
effective remote teaching when the traditional face-to-face teaching modality is impossible.
Also, laboratory activities can be adapted to take place outside of physical laboratories to meet the objectives
of the experimental phase of the chemistry courses. In this study, we sought to work with topics on solutions
and their colligative properties, which are taught in chemistry courses at different levels.
2 Theoretical Framework
Conducting experiments related to the colligative properties of solutions to support the theoretical content is
part of the chemistry curricula at different academic levels.
The four colligative properties are a) the decrease in the vapor pressure of a liquid (P°), b) the increase in boiling
temperature (Tb), c) the decrease in freezing temperature (Tf), and d) osmotic pressure (π). They only depend
on the number of solute particles in the solution and not on the nature of the solute particles (Chang, 2002).
This work focused on practices related to the colligative properties of solutions and specifically on increasing
boiling temperature (ΔTb) and decreasing freezing temperature (ΔTf).
237
Both the increase in boiling temperature and the decrease in the freezing point are proportional to the molal
concentration (m) of the solution and an ionization factor "i" (the van't Hoff factor). The proportionality
constants are Kb (molal constant of boiling point elevation) and Kf (molal constant of the decrease in freezing
point), both with units of °C / m. Therefore:
ΔTb = i Kb m (1)
ΔTf = i Kf m (2)
The Van´t Hoff factor (i) is related to the solutions that form electrolytes since these dissociate into ions,
separating into two or more particles, which are the ones that determine the colligative properties of a solution
and can be defined as:
𝑨𝒄𝒕𝒖𝒂𝒍 𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒑𝒂𝒓𝒕𝒊𝒄𝒍𝒆𝒔 𝒊𝒏 𝒕𝒉𝒆 𝒔𝒐𝒍𝒖𝒕𝒊𝒐𝒏 𝒂𝒇𝒕𝒆𝒓 𝒕𝒉𝒆 𝒅𝒊𝒔𝒂𝒔𝒔𝒐𝒄𝒊𝒂𝒕𝒊𝒐𝒏

i =
𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒎𝒐𝒍𝒆𝒔 𝒅𝒊𝒔𝒔𝒐𝒍𝒗𝒆𝒅 𝒊𝒏𝒊𝒕𝒊𝒂𝒍𝒍𝒚 𝒊𝒏 𝒕𝒉𝒆 𝒅𝒊𝒔𝒔𝒐𝒍𝒖𝒕𝒊𝒐𝒏
Table 2. Regarding the molal constants of boiling point elevation and the freezing point decrease of several common
liquids, both Kb and Kf only depend on the solvent (Chang, 2002)
Solvent Normal freezing Kf (°C / m) Normal boiling Kb (°C / m)

point (°C) point (°C)
Water 0 1.86 100 0.52
Benzene 5.5 5.12 80.1 2.53
Ethanol -117-3 1.99 78.4 1.22
Acetic acid 16.6 3.90 117.9 2.93
Cyclohexane 6.6 20,0 80.7 2.79
This implies that in non-electrolytes (covalent solutes), the van t́ Hoff factor has a value of one. A strong
electrolyte such as CaCl2 will optimally have a value of three because this compound is ionized in a calcium
particle Ca+2 and two particles of chlorine Cl-1.
Table 3. Van't Hoff factor for 0.05 M electrolyte solutions at 25 °C (Chang, 2002)
Electrolyte i measured i calculated

Sucrose (is not an electrolyte) 1.0 1.0
HCl 1.9 2.0
NaCl 1.9 2.0
MgSO4 1.3 2.0
MgCl2 2.7 3.0
FeCl3 3.4 4.0
This table presents the values of i, measured experimentally and those calculated assuming a complete
dissociation. They are similar. There is no appreciable formation of ionic pairs that do not have a net charge
and reduce the number of particles in the solution.
238
3 Background
The colligative properties experiments and others are covered in a laboratory manual for a chemistry laboratory
equipped with the appropriate materials (Serrano, 2020). However, if a physical laboratory is not available,
experiments can be conducted virtually (Infante, 2014), (Vega, 2016). Moreover, we contemplated an alternative
practice that could be carried out in any home (Zapata, 2020).
Zapata (Zapata 2020) promotes experimenting at home with homemade materials usually found in a kitchen.
Although these experiments do not require chemistry laboratory equipment such as beakers and specimens,
they require a thermometer to measure the solution's boiling temperatures and freezing points, which is not
an instrument usually found in a house. If the home does have a thermometer, it is usually a clinical one, and
its temperature measurement range is not sufficient to measure the temperatures involved in experimental
practices.
Well (Well, 2004) and Braga (2022) explain that you do not need a laboratory built for specific purposes or a
closet with specialized and expensive equipment to perform chemistry experiments. Most can be carried out
with simple materials found in the kitchen, the bathroom, and the pharmacy or convenience stores. However,
they do not mention an experiment that involves temperature measurement.
4 Proposal for the home laboratory practice

To develop the various course practices, teachers, as far as feasible, can simulate chemistry laboratory
experiments and provide tutorial videos from various educational institutions (Chen et al. 2013), (Infante 2014),
(Vega et al. 2016) related to the topics to be discussed. These can be available to all on the network. In addition,
educators can design no-risk experiments adapted to the program contents to be performed in the students'
homes with household materials while maintaining the level of knowledge established by the course objectives.
Considering the difficulty of measuring a solution's boiling and freezing temperatures in a home experiment
to teach colligative properties, we looked for a parallel variable proportional to the difference in boiling and
freezing temperatures between a pure liquid and a solution formed by it.
Since the solute content in a solution causes it to boil at a higher temperature than the pure solvent, according
to equation (1), we contemplated measuring the time the pure solvent boils and the time to boil the solution.
This time difference would be proportional to the temperature difference. Thus, using the same solvent and
different solutes, it could be deduced which would have higher ΔTb and, therefore, higher boiling temperature,
considering the same molal concentrations in all cases.
For example, suppose an aqueous solution A takes 400 seconds to start boiling, a solution B 450 seconds and
water alone 300 seconds. In that case, it could be concluded that solution B has a higher boiling point than
solution A and, therefore, a higher ΔTb, which would constitute a starting point for the student to analyze the
observed phenomenon and its implications.
According to equation 2, something similar could be done to determine which solution would have the lowest
freezing point and the highest value of ΔTf. In this case, it would be more complicated to determine the freezing
times of the solvent and the solutions used in the experiment.
5 Methodology
The students in two classes of 30 each studying this subject were instructed to mark a glass at a given height
above the middle and obtain a small measuring spoon in their house. Otherwise, they could get the lid of a jar
as small as possible and three equal soft-drink bottle caps or something similar.
239
Figure 1.
They were told to fill the glass with tap water to the mark on the glass and place it in another similar glass. The
students were then asked to refill the glass with water to the indicated mark, add two sugar measures with the
measuring spoon, shake it to dissolve evenly and place the formed solution in another glass. Next, they were
instructed to repeat the same procedure but add a measure of salt to the marked glass so that the identical
amounts of water, sugar solution, and the saline solution would be in the first, second, and third glasses,
respectively.
Figure 2.
The students were then asked to carefully fill a bottle cap with the liquid from each glass and place them in the
kitchen freezer, long enough for the solutions to freeze.
Figure 3.
240
Next, we instructed the students to pour all the contents of the first glass into a pot and place it on a kitchen
stove burner to observe how long it took for the water to boil. At the end of the experiment, they emptied the
pot, rinsed it, and allowed it and the stove to cool. Then, they were to repeat the same procedure with the
solutions in the other two vessels, trying to maintain as much as possible the same process conditions.
Figure 4.
Finally, they were asked to explain and justify what they had obtained and observed during the performance
of these home chemical experiments in the laboratory report used in each practice.
6 Results
All the students reported that the time to reach boiling in the saline solution was more than the sugary one,
which, in turn, was longer than the pure water. Moreover, the latter was the first to freeze, then the sugar water,
and finally the saline solution much later; some even pointed out that the salt solution did not freeze.
75% of them explained that sugar was a covalent compound and that salt was ionic, which was the reason for
the difference in the times obtained. 62% correctly contemplated the van t́ Hoff factor concept. Only 28 % tried
to apply specific numerical calculations to justify the experimental process, concluding that more particles were
dissolved in the saline solution than in the sugary one.
In a series of interviews, most of the students opined that this activity had awakened their interest in the subject
but found it challenging to understand. Something similar happens in problem-based learning or research,
even when it involves simple things of daily life.
At the end of the course, all the program topics were analyzed. 81% of the students solved the problem related
to colligative properties with a level of difficulty similar to other tasks and exams of various chemistry courses
that most students considered too complex to understand.
7 Conclusions
In the absence of laboratories to conduct experimental practices of a chemistry course for any reason, one can
turn to virtual technological supports such as simulators and descriptive videos, which are undoubtedly very
useful. However, alternative practices with materials commonly used at home can complement these activities
so that the student feels the closest possible to the activities and methodology in a laboratory.
All the students liked doing the home-based activities for the level of rapport and presence they felt, unlike
watching a descriptive video or tutorial where there could be a certain degree of dispersion at any given
moment. The home experiments motivated them to study the subject, investigate, and understand the
241
observed chemical phenomena; some even proposed that similar home activities be developed in other
practices
Conducting chemistry experiments of this type is economical and feasible outside a laboratory's scope.
Depending on the teacher's management and approach, they do not detract from the quality or level of
knowledge in the significant learning of one or several topics.
While in this case, the work was developed as an educational response to unexpected home confinement for
the entire population, including students and university professors, the methodology could also be used in
everyday teaching of theoretical chemistry courses.
It would be convenient to continue developing experimental activities of this type for various topics of
chemistry courses for the educational benefit that they can entail.
Acknowledgements : "The authors would like to acknowledge the financial and the technical support of
Writing Lab, TecLabs, Tecnologico de Monterrey, Mexico, in the production of this work." 
8 References
Braga, X. M; Castro, C.H; Carla, C.H.; Carvalho Delou, C.M. (2022) Chemistry and Giftedness: A Delight Full and Catalytic Mixture in a Home
Enrichment Model. Creative Education. Ol. 13. No 1. January 2022
Chang, R; College, W. (2002). Química. 7ª Edición. McGraw Hill Interamericana Editores S. A. de C.V. México.
Chen, D; Chen, X; Gao, W. (2013) The Application and Perspective of Multimedia Technology in Chemistry Experimental Instruction in
China. Creative Education. Vol. 4, April 2013
Infante Jiménez, Cherlys. (2014). Propuesta pedagógica para el uso de laboratorios virtuales como actividad complementaria en las
asignaturas teórico-prácticas. Revista mexicana de investigación educativa, 19(62), 917-937. Accessed July 6, 2020, from
http://www.scielo.org.mx/scielo.php?script=sci_arttext&pid=S1405-66662014000300013&lng=es&tlng=es.
Rodríguez, E. B. E; Romero, R.L.E. (2020). Experimentación química y pensamiento estadístico. Pearson Educación de México
Serrano, S. T. (2020). Pruebas fisicoquímicas. Editorial Síntesis. Madrid, España.
Vega, O. A., Londoño, H.S.J.; Toro, V.S.; (2016). Laboratorios Virtuales para la Enseñanza de las Ciencias. Ventana Informática. N° 35 –Julio
Diciembre. Accessed July 8, 2020, from https://doi.org/10.30554/ventanainform.35.1849.2016
Zapata, Mariangel. (2020). Laboratorio en casa. Practica de laboratorio: Propiedades coligativas. Universidad Pedagógica Libertador.
Maturín, Venezuela. Accessed July 8, 2020, from https://quimicaencasa.com/practica-de-laboratorio-propiedades-coligativas/
242
An advanced application for learning robotics using
Augmented Reality
Moisés A. Croes1, Carlos A. Jara1, José L. Ramón1, Jorge Pomares1, Gabriel J. García1, Andrés Ubeda1
1
Human Robotics Group, University of Alicante, 03690 Alicante, Spain
Email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Abstract
Robotics is a multidisciplinary branch of engineering that nowadays presents many challenges to overcome for having a
greater presence and impact in our society. Due to its characteristics, usually it is investigated by the integration that it can
have with other technologies and one of these is augmented reality (AR), which consists of combining the real and virtual
world through computer processes, improving the visual experience and the scientific application. The main objective of
this project is to develop an application that implements AR for the visual projection and supervision of robots, where the
solution is based on a client-server architecture. The server is responsible for controlling the robot using ROS (Robotics
Operative System), and the client is based on a platform compatible with Unity that includes the AR Vuforia engine, including
the characteristics of graphic, physical and AR simulation of the robot to be controlled. To bring about this investigation it
was used a configured Windows 10 environment with WSL (Windows Subsystem Linux), allowing ROS and Unity to coexist
on the same machine, meanwhile, to overcome the communication challenge that exists between them, it used the solution
proposed by the Unity Robotics Hub project. This software permits to add three robot models with distinctive characteristics
to recreate some use cases with AR. Finally, it is possible to obtain an independent Unity client based on UWP (Universal
Windows Platform) oriented to desktops and with AR functionalities.
Keywords: Robotics; Augmented reality; Robotic Operative System (ROS); Unity, Vuforia, Ros #, Unity Robotics Hub.
1 Introduction
Robotics is a multidisciplinary branch of engineering that currently presents many challenges to overcome in
order to have a greater presence and impact in our society. The research drives the current state of this branch
with new improvements that are always being sought by collaborating with technologies from other
environments to enhance current robotic solutions and one of them is augmented reality (AR).
Augmented reality (AR) is a variation of Virtual Reality (VR), where the user can see in the same environment
how real objects and the superimposition of virtual objects coexist (Azuma, 1997). This offers the improvement
of our perception by helping us to see, hear and feel our environment in new ways (Krevelen, 2007). The AR
potential can be used in multiple areas and it also helps to improve interaction difficulties in current challenges.
Some examples about this application cases are:
● In an industrial manufacturing environment, the management of a production line can be supported
by visual indicators to know some parameters such as the state of operation, quality control of the
products, alerts that include the location of a possible failure in any component of the line.
● In the field of medicine, doctors can speed up the process of diagnosing a physical injury in a patient
while obtaining real-time visual feedback from a virtual projection over the affected area with the
respective bones in that area.
● For entertainment, focusing on video games, the user experience can be enhanced by including
features that use real physical effects in conjunction with the 2D/3D elements of the video game itself.
● For educative environment, simulations is a commonly tool to avoid some executions of experiments
because doing them with real equipment and components can be significant expensive, so this is
where AR offers an improvement in the student experience that can observe a better physical
interaction between two virtual 3D elements in an already existing and easy to manage real
environment.
243
In the field of applications of AR its notable the intention of integration with robotics, which offer the
enhancement of existing solutions in multiple areas and the experimental research. Currently there are already
several investigations that study these characteristics:
● Improve the navigation control of mobile robots using head mounted displays (HMDs) and AR (Kästner
& Lambrecht, 2019). In this case tests were carried out using a pair of Microsoft Hololens that allow
the user to indicate the displacement positions of the robot with hand gestures, these are transmitted
to the robot's operating system to be processed and executed, in addition to giving user trajectory
planning feedback.
● Facilitate the teleoperation of aerial robots (Hedayati, Walker, & Szafir, 2019). In this case they used a
commercial drone and a pair of Microsoft HoloLens, where a user get this equipment and obtained
details feedback about robot vision with other perspectives that helped flight operation.
● Improve trajectory planning with AR-based interfaces (Fang, Ong, & Nee, 2013). The authors exposed
some of the difficulties of human-machine interaction, making a specific focus on industrial robots and
how these can be very limited to the use of the programming languages of each manufacturer,
especially in relation to trajectory planning, but through AR they managed to develop an interface
oriented to the use of a cubic marker that allows managing the points of the desired trajectory. Two
aspects of this development can be highlighted: first, the use of the cubic marker that has several
targets associated with it, so that when it is identified by the camera, it executes an action on the
indicated point, and the use of a method based on Euclidean distance to resolve and assist to the user
during point generation.
● Planning and simulation of a robotic cell using AR (Pai, Yap, & Singh, 2014), in this research the authors
use some technologies to find alternatives in the planning process of a future robotic cell installation,
offering a cost reduction and increased efficiencies that are representative for small and medium-sized
businesses. The cell that they recreate to simulate the manufacturing plant environment is made up of
a robotic arm, a conveyor belt, a pallet and a numerical control machine (CNC), with the aim of
recreating material handling processes. An interesting point that should not be overlooked are the
technologies that were used, such as a HMD as vision equipment and an interface based on C++ with
OpenGL through ARToolKit to generate the virtual elements, as well as a target for everything related
to the interaction of the user.
● Teleoperation of a manipulator-type industrial robot based on AR (Solanes, et al., 2020), this paper
seeks a considerable improvement in the teleoperation of this type of robots, which are common in
manufacturing areas and for this reason makes it relevant to improve their management. Regarding
the hardware, they use a game controller (gamepad) and a HMD which receive the feedback from the
control carried out on a robotic arm. For the development of the interface, they use Unity and Visual
Studio on Windows 10, managing to generate an application for the HMD with the MixedRealityToolkit
in Unity. The results showed the benefits of a control feedback to enhance the user experience.
2 Software architecture
In this research we used some technologies such as 3D objects render, robotics controls, AR and
communications between software components. Some of these technologies don’t have any compatibilities
between them therefore it was a huge challenge to resolve as real time communication.
2.1 Technologies and tooling

● OS Windows 10 – Enterprise Edition
● Ubuntu 18.04 LTS over WSL (Window Subsystem Linux), specifically WSL 2 in W10
● ROS (Robotic Operative System) Melodic, corresponding version in Ubuntu 18.04
● MoveIt, open-source framework for controlling robots in ROS
● Robot’s models in URDF (Universal Robot Description Format); UR3, HOAP3 and Shadow Hand (Figure
1)
● Unity, a world class game engine
244
● Visual Studio 2019 or latest
● Vuforia, AR engine
● Unity Robotics Hub, library for communicate ROS and Unity, predecessor of RosSharp project
● UWP (Universal Windows Platform), target platform for client out Unity environment
● Webcam (1920 x 1080p, 30 fps)
Figure 1. Real robots model used in this project: (Universal Robots, 2021; Galdeano, Chemori, Krut, & Fraisse, 2014;
Galdeano, Chemori, Krut, & Fraisse, 2014)
2.2 Integration
In design process the best software architecture for this solution has a client-server approach, therefore the
classification for each component was grouped in infrastructure, server, client and communication components
(Figure 2):
● Infrastructure: based in W10 OS and extra layer with WSL 2
● Server: which has the responsibility to process all robot movements and calculate trajectories with
MoveIt in ROS (programmed in Python). This layer run over Ubuntu 18 in WSL into the same machine
● Client: who has the AR features and 3D renders in Unity combined with Vuforia engine in the develop
environment; in production case all this client runs in UWP. All logics for this layer was coding in C#
● Communication: to resolve the interaction between client and server was necessary to use libraries
from Unity Robotics Hub, allowing the communication with TCP socket in real time and low latency.
Figure 2. Architecture solution with all key components
The default robotics models are in URDF, this format is ROS oriented but can be transformed with a package
from Unity Robotics Hub called URDF Imported which generate a robot model into the Unity scene with all
visual and physics properties, including the joints as articulation bodies allowing an easier control. Another key
245
library in this solution is ROS TCP Connector for client side in Unity and ROS RCP Endpoint for ROS server side,
both are from Unity Robotics Hub and allow the communication.
3 Experiments
There were designed three experiments with the aim of testing the previous environment. Attempt to recreate
real life process with some popular robotics model and all cases the logic structure is the same both on the
client and on the server (Figure 3).
For AR features, Vuforia engine has many ways to detect and render the Unity model in real time video from
camera device. The selected way was focused on detect targets features mainly with VuMark (a special image
with high contrast and optimized to be detected) and the render action succeed when the camera’s scan
detected the established target in the real environment.
On the client side the general execution process renders the robotic model when a trigger is activated because
the camera’s scan detects a target on real environment, after this the robot is ready to go to the next state
(movement combination), when pressing the respective button, the actual state and next state data is sent to
the server endpoint. On the other hand, the server waits for client request, once it has arrived the request data
about robot state (robot’s joints and next request state) redirect this data to mover service which processes it
with a robots model previously charged in a MoveIt node (with all movement restriction and config to calculate
trajectories), if the next state doesn’t have collisions or trajectories errors, this service generates an output with
the new position coordinates and return them to the client. Finally, the client executes the movements in visual
model until robot has the final state.
Figure 3. Software components, modules and relationships in client and server
3.1 Pick and place with a robotic arm

A typically pick and place process was designed with a robotic arm UR3 from Universal Robots and pneumatic
vacuum suction cup in the extreme of the robot arm. This experiment has 4 states or positions and just one
trigger. Where robot begins its initial state, it picks a box up, moves it to the opposite direction and places it
in the green zone (Figure 4).
246
Figure 4. Pick and place process with UR3 and target
3.2 Robotic humanoid control

The control for humanoid robot is complex, but an adjustment was made to remain the torso fixed at a certain
coordinate. It allowed to use the robot HOAP3 from Fujitsu. This experiment has 3 independently states
associated to each body part and each one has its trigger, when all triggers are executed, the robot copies an
equilibrium position on one foot (Figure 5).
Figure 5. Robotic humanoid control with HOAP3
3.3 Grasping with robotic hand

Shadow Hand form Shadow Robot was selected to realize the grasping activity. This experiment has 2
independently states, pre-grasp and post-grasp, when all stages were completed, the robot catches the
cylinder from its initial position to another final opposite position (Figure 6).
247
Figure 6. Grasping process with Shadow Hand
3.4 Client app in UWP

Unity is mainly used for development stage, but it can compile to target device when the app will be executed,
in this case the limitation is about Vuforia engine compatibility, this one only works with mobiles devices
(Android, iOS) and UWP, but the research scope is about the development in the same machine, therefore
UWP was selected as production target, because it allows to execute the app in desktop with W10.
To improve the user experience, a main menu was included in the app, which groups all the experiments in AR
and virtual version (Figure 6 and 7).
Figure 6. Main menu for all experiment in UWP
Figure 7. Scene pick and place with robotic arm without AR
248
4 Benefits in the learning process
It is worth pointing that this research also has some positive insights for learning process focussed on robotics
students or similar in related areas, this is the case for the degree in Robotics and the master’s degree in
Automatic and Robotic students in University of Alicante.
The technologies and techniques tested here could be used for designing some new experiments in
laboratories or interesting workshop, being able to have the following advantages:
• Configuring the develop environment with only one personal laptop or desktop, because these
experiments use ROS in WSL as backend and Unity in W10 as client both in the same instance
• Learning about robotics physics with robot models in Unity, this allows to test some full featured
models in an easy way with URDF file from real models
• Practicing the control algorithms over robots with ROS and planning trajectories with MoveIt
framework
• Improving technical skills in programming languages as C# in Unity and Python or C++ in ROS
• Demonstrate a way to build new robotics products with AR, due to is possible generate a standalone
apps with external and enterprise grade frameworks
5 Conclusions
After performing the iterations and preparing the development environment for this research, the conclusions
would be the followings:
• The AR development environment for using W10 as S.O. base to support Unity and WSL, the latter
allows the use of Ubuntu distribution with a ROS respective version, which is a viable alternative to
avoid virtual machines by third parties in Windows, due to its easy configuration and compatibility with
graphics recourses
• Unity and Vuforia are good options to develop interfaces with AR, both are allowed to do a project
with three use cases with different based robotic models: UR3, HOAP3 and Shadow Hand. The
iterations and the proof of concept had generated many scenes with and without AR in a modular way.
It is worth mentioning the power of Unity to export multitarget clients. In this research was generated
an UWP app which is ready for production environment, doesn’t need Unity installation in destiny
device and it is easy to distribute
• The communication between a ROS node or server with an app client based in Unity is not native but
exist some solutions to mitigate this complexity. The projects in Unity Robotics Hub were a key
component as well as it integrates a two-ways communication and uses ROS’s message scheme to
generate standard or custom messages
• The communication components and the architecture for this application allowed the response with
information state and the coordinates to the client (with a previous services validation with MoveIt)
then the client was independently getting this data and executing the movement with scene and
physical restriction in Unity (all workload belongs to client side)
• Around benefits in learning process, this research has many resources to build practices in laboratories
or make some workshops about robotics and AR, where students can learn more about these topics
with practical examples and reinforce other knowledge areas which as: programming languages,
algorithms, networks, 3D modeling and some others.
In future researches and projects will be interesting to consider other topics as: limitations of Articulation Body
components in Unity, improvement of the AR experience with better hardware features, to make tests and
integrations with mobiles devices and to use container technologies as Docker for development and deploy
ROS nodes.
249
6 References
Azuma, R. T. (1997). A Survey of Augmented Reality. In Presence: Teleoperators and Virtual Environments 6, 4, (pages. 355-385).
Galdeano, D., Chemori, A., Krut, S., & Fraisse, P. (2014). Task-based whole-body control of humanoid robots with ZMP regulation, real-time
application to a squat-like motion. Montpellier, Francia.
Hedayati, H., Walker, M., & Szafir, D. (2019). Improving Collocated Robot Teleoperation. Chicago, USA.
Kästner, L., & Lambrecht, J. (2019). Augmented-Reality-Based Visualization of Navigation Data of Mobile.
Krevelen, R. V. (2007). Augmented Reality: Technologies, Applications, and Limitations. Amsterdam, Netherlands.
Pai, Y. S., Yap, H. J., & Singh, R. (2014). Augmented reality–based programming.
Shadow Robot Company. (2021). Shadow Robot. Get from Dexterous Hand Series: https://www.shadowrobot.com/dexterous-hand-series/
Solanes, J. E., Munoz, A., Gracia, L., Martí, A., Girbes-Juan, ·. V., & Tornero, ·. J. (2020). Teleoperation of industrial robot manipulators based.
Universal Robots. (2021). Universal Robots. Get from UR3: https://www.universal-robots.com/es/productos/robot-ur3/
250
The Torca Experiment: A model of transdisciplinary project-
work
Fernando José Rodríguez-Mesa1

1
Universidad Nacional de Colombia, Sede Bogotá, Facultad de Ingeniería, Bogotá, Colombia
Email: [email protected]
Abstract
All of Colombia's major cities have at least one higher education centre. However, this does not necessarily entail fair access
to education for those living outside these cities' main urban areas. Moreover, financial difficulties and the problems
inherent to a long commute are often insurmountable barriers for people who want to study but live in comparatively
remote regions. To tackle this issue, the Universidad Nacional de Colombia (UNAL) created a particular admission program
(PEAMA) to facilitate access to education in this less-favoured population sector. Since the academic interests of any
population will be varied, the UNAL offers selected educational programs o through the PEAMA. Thus, professors and
lecturers are sent to schools that effectively function as nodes of the UNAL in rural areas to allow students to complete the
first four semesters of their program of choice. During the first semester of 2022, 15 people from the rural area of Torca in
Bogotá decided to begin their higher education studies through the PEAMA. In total, seven different academic programs
were chosen. To allow as many students as possible to complete their programs, a transdisciplinary project-based learning
model focused on the particular social problems of the community was designed and implemented for that specific case.
Thus, three multidisciplinary groups were created to accommodate the 15 students enrolled in the university through the
PEAMA: one for students from the literature and philology programs, one for students of economy, agronomy, and the
social sciences, and, finally, one for engineering students. This article discusses the model and the first results obtained.
Keywords: Project-based learning, transdisciplinary, humanitarian education, rural
1 Introduction
In 2007, the Universidad Nacional de Colombia (UNAL) started implementing the PEAMA, a special admissions
program designed to facilitate access to education for specific population sectors in the periphery of its areas
of influence. Initially, the PEAMA was devised to allow students of the periphery campuses of the university,
i.e., those in Arauca, Leticia, San Andrés Islas and San Andrés de Tumaco, to enrol in programs offered only in
its major campuses, i.e., those in Bogotá, Medellín, Manizales and Palmira. Thus, the students of the periphery
campuses would have the chance to study programs that would otherwise be out of reach for them (Acuerdo
025, 2007).
Students admitted to this program completed their undergraduate degrees in three stages. In the first stage,
they took the introductory courses of their program of choice in their place of origin. Then, after their fourth
semester of studies, they went to one of the main campuses of the university to continue their studies there.
Finally, they returned to their places of origin during their last semester to write a thesis or implement a project
focused on their territories.
Due to the positive reception of the program and its favourable impacts on students from the periphery
campuses of the university, the PEAMA was extended in 2015 to start offering the same services to people who
lived in comparatively remote areas that, nevertheless, fell into the regions of influence of the major campuses
of the university (Acuerdo 201, 2015). Since there were no periphery university campuses in those areas,
professors and lecturers from its main campuses were sent to some rural schools that would effectively begin
to function as nodes of the UNAL. Thus, in 2016, the PEAMA Sumapaz was created to allow the people that
lived in Nazareth —about 84 km south of Bogotá, in the lower part of the Sumapaz paramo— to enrol in some
of the programs offered at the Bogotá campus of the UNAL (Alcaldía Local de Sumapaz, 2022). At that time,
the university provided 60 PEAMA spots, of which 18 were taken. Those students chose the following
undergraduate programs: nursing, agricultural engineering, agronomic engineering, veterinary medicine and
zootechnics. Later, in 2018, the PEAMA Sumapaz program expanded further to allow students graduating from
251
several rural schools in Bogotá to study at the UNAL. The number of undergraduate programs that could be
chosen was also increased (Resolución 405, 2016; Resolución 908, 2018). Afterwards, during the first semester
of 2021, a new node of the university was established for the PEAMA Sumapaz in one of the rural schools of
the district of Ciudad Bolívar in Bogotá. Finally, in the first semester of 2022, the newest version of the program
was established in the rural sector of Torca, located almost at the city’s northern border. In this case, 40 courses
were offered for 18 students enrolled in 8 programs: civil engineering, mechanical engineering, mechatronics
engineering, agronomic engineering, economics, linguistics, English philology, and French philology. Like all
students of the PEAMA Sumapaz, the people that graduated from schools in Torca will remain in their place of
origin while they complete the first semesters of their chosen programs. During this period, they will learn the
foundational and part of the disciplinary content of their programs through project-based learning (PBL).
Afterwards, they will go to the Bogotá campus of the UNAL as regular students.
Unlike the original version of the PEAMA, PEAMA Sumapaz is not entirely dependent on the university budget
to run. Indeed, the Secretary of Education of Bogotá agreed with the UNAL to facilitate the access to higher
education in the rural sectors of the metropolitan area of the city that they deemed most in need of the
program. Thus, the funding for the PEAMA Sumapaz of the UNAL comes from an external entity.
The PBL model used in all iterations of the PEAMA Sumapaz promotes learning through participation in
projects that can have concrete impacts in the communities of which students are part. Thus, in Nazareth, for
instance, the participants of the project began their studies by deepening their knowledge of their place of
origin and by identifying the potential of the region and its inhabitants (Cita Triana et al., 2020; Ordóñez-
Ordóñez et al., 2020).
Given the enormous differences between the sectors where PEAMA Sumapaz was implemented, each PBL
model used in the program has had to be developed almost from scratch. The form that the transdisciplinary
PBL model will take is, thus, different according to the necessities, strengths, weaknesses, and potentials of the
stakeholders of the community projects that will be used to teach students the skills that they will need to
complete their programs of choice. This article describes the model used in the Torca version of the PEAMA
Sumapaz and its first learning outcomes. In addition, some of the open-ended interviews with students carried
out to assess the success of the program will be analysed.
2 Torca PBL model

Traditional PBL models use real and concrete problems as the starting point for learning. Problems that may
be addressed or solved in a relatively short time are usually chosen for these models. Pioneering universities,
such as McMaster in Canada, Maastricht in the Netherlands, New Castel in Australia, and Roskilde in Denmark,
have researched and championed PBL models (Neame, 1989; Neufeld et al., 1974; Olsen & Pedersen, 2008;
Schmidt, 1989). More recent and less widely spread variations of the PBL methodology choose long-winded
projects rather than short problems to drive students to learn. Like many other models, the Aalborg University
focus on this type of PBL (Kolmos et al., 2017). The PBL model implemented in Torca described in this paper is
also based on project work.
The following statements were used as the PBL principles of the program:
1. Problems must be addressed and solved by a group.
2. Students are responsible for their own learning (self-directed learning).
3. The problems used for learning must be real problems.
4. The problems used for learning must be exemplary.
5. The solutions to these problems are inherently social and must be implemented with the aid of the
community.
However, unlike traditional PBL practices (e.g. De Graaff & Kolmos, 2003; Savin-Baden & Major, 2004), the
PEAMA students in Torca were comprised of people seeking different learning outcomes since not all of them
enrolled on the same undergraduate programs. However, despite this heterogeneity, they were assigned the
same problem and project. Thus, students of various engineering branches, the social sciences, and the
252
agricultural sciences worked side by side on a project that was important to the community to which all of
them belonged. Since problems are rarely one-sided, the input from students of very different disciplines, far
from hindering processes, often allows for developing more robust solutions.
The transdisciplinary model shown in Figure 1 presents the process followed by everyone involved in the
project at Torca.
Figure 1. Torca PBL model.
This model works within a system of three interrelated layers: the social, the academic or institutional, and the
student. The first layer is comprised of a community with multiple needs and problems that perhaps could be
solved with the aid of higher-education students. The next layer includes the educational organisation and the
people working to facilitate students' learning. Finally, students, in their learning process, are the core of
formulating and solving the problem affecting the community.
Rather than focusing excessively on the idea of a teacher that transfers some ready-made knowledge to passive
students, the role of teachers and other academic staff in the PBL implemented in Torca was to facilitate
learning. Indeed, by making the concept of “educator” more flexible, four education-facilitation roles were
created for the program. The first kind of facilitator resembles more or less a teacher of a subject in a regular
classroom. These facilitators are responsible for concrete courses of the undergraduate programs that the
students choose, but their most important task is to encourage students to come up with their answers. The
second kind of facilitator is a transversal facilitator who supports the project according to planning,
management, and group needs. In addition, they help in the development of soft teamwork skills. The third
facilitator is the project chancellor, a person hired by the university to facilitate the exchange between the
social and institutional layers. Moreover, the chancellor helps students, other facilitators, and the community
to understand, on an academic level, the problems they will be working with. Furthermore, the chancellor works
actively to keep the project on its track, i.e., they make sure that students do not get carried away by secondary
problems while they work towards completing the project. Finally, the fourth kind of facilitator oversees
bringing the literacy or drawing skills of the students to the levels necessary for them to successfully achieve
the intended learning outcomes (ILOs) of their program of choice.
Naturally, even though the PBL method dictates that a common problem must be solved by all of the
participants of the project, each study program has its own particular ILOs. Thus, to aid students in achieving
the ILOs of the program that they chose, the general group of the Torca students was divided into smaller
groups —according to the similarity of the contents and skills that they must learn— for some of the activities
253
that they partake in their day-to-day academic formation. Whether the students achieve these ILOs was
determined on a case-by-case basis by a group of facilitators whose input weighed according to their
involvement with each particular student. To do this, they assessed the material or immaterial artefacts
produced by the students during the development of the project.
Given their importance to the whole process, all the relevant ILOs were explained to the students and the
chancellor at the beginning of the project. This, on the one hand, served as a guideline for students during the
problem-formulation stage of the project, and, on the other hand, allowed the chancellor to formulate his
interpretation of the problems of the community in terms that would be useful for the students to develop the
skills necessary for completing their programs.
The solution will produce a report with a material or immaterial product during the problem's development.
For example, the product could be the specific outcome that students would use to learn in some or all subjects.
In this case, each teacher involved must review the project to assess the achievement of the result of their
matter. Consequently, they award the respective grade. On the other hand, the general project allows the
evaluation of the transversal skills of the students.
Torca's model also uses subjects that are not part of the project due to their high content of propositional
knowledge, such as basic mathematics, principles of chemistry, and computer programming. That cases have
lectures and sessions with active learning.
3 Method
The students of the Torca experience went into three groups for the project work, considering that all the
subjects involved in some projects to get more balance (Table 1). However, Philology and Linguistics had a
group composed of 3 students for a need previously identified as relevant to these careers concerning the
school radio station.
Table 1 Project groups for the Torca Model
Name Curriculum Project Theme
Student 1 Mechanical Engineering
Student 2 Civil Engineering
Student 3 Civil Engineering Reactivation of a scholar greenhouse
Student 4 Economics
Student 5 Mechatronic Engineering
Student 6 Mechatronic Engineering
Student 7 Economics
Student 8 Civil engineering Solid waste management at the Nuevo

Student 9 Agronomic Engineering Horizonte school
Student 12 Philology and Language
Student 13 Philology and Language School Radio Station
Student 14 Linguistics
Students analysed needs during the first six weeks of the project. Identifying the problem is essential that the
problem address social needs, but not a pre-established solution.
254
After the first six weeks of model development, all of them, 14/14 students answered an open-ended interview
by the project group. The interviews lasted around 45 minutes for the first two table groups and 21 minutes
for the third group. The following research question: Have they found learning of the subjects during the
formulation of the problem? Have they managed to identify the project work with the learning outcomes?
4 Results and discussion
The interviews were conducted in Spanish, transcribed verbatim, and analysed contextually. The PBL learning
model has shown favourable results in student learning. However, they generally stated that they had
difficulties formulating the problem.
During the formulation stage of the problem, the students have managed to understand concepts with the
project tools on topic explanations given by the teachers. In this regard, one of the students stated:
"I believe that each subject gives us tools to be able to continue, and one no longer feels so lost
concerning the first problem. It no longer feels like that. I don't know how to say it, like that emptiness,
well, you can see, you can see a big difference. We picked up things more easily than before starting
something before. It was like something. We were lost. So, they have given us excellent tools that are
already helping us understand the things they, for example, explain”. A Student translated from
Spanish.
Despite achieving learning, there was significant difficulty defining the problem during the first six weeks of
project work, representing about 40% of the time to carry out the project. The definition of the problem begins
with the identification of the needs. Then students analyse it for a possible solution and the relevance to the
learning results of the subjects and end with the declaration of research questions and objectives. From this
analysis, teachers qualitatively observe if, with that analysis, the students will meet the ILOs by subject. That is
the most crucial thing in the development of the project.
The most significant difficulties arose in formulating the objectives during the process. The learning model
expected difficulties with problem identification and dilemmas to produce some transformative learning.
Several reasons mainly explain it:
• The subject facilitators (Professors) promoted a solution that favoured the application of the theory of
that subject, and that intention was generalised to all the group members.
• The Chancellor oriented the project toward the primary need identified.
• Students wanted to develop their solutions to the problem.
The different points of view of the teachers as facilitators caused a certain degree of uncertainty in the students.
However, it would foster reflection and argumentation and recognise some topics' lack of depth learning.
“But it is that, for example, the tutoring. It is that they try to lead us along a path with a social stigma
that does not bother us. It is as if we end up choosing that idea. Then another teacher comes with another
idea, and we believe that idea. So, what happens is that we still don't have things very clear and well.
After comes the other, by another side, that we had arrived. It is from one side to the other, and we get a
little lost, and of course, since we don't understand things well, then on that side, it is as if they drag us
to like their ideas and ideas.” A Student translated from Spanish
Many different points of view in identifying the community's needs and how to approach the solution
happened, but it seemed to favour a change in the ways students learn. When students come from
environments where what the teacher says is done, finding several teachers saying such different things about
the same thing produces uncertainty; this setting facilitates reflection (c.f. Rogers, 1951).
“If what Luis says is true, seeing too many points of view. Each one of them...As if he wants to direct us to
his thinks or the place of his eyes. He makes us contradict ourselves. Perhaps, if not, we have changed
the question.” A Student translated from Spanish
255
This process took six weeks to work and not five as initially expected. However, there were times when the
teachers also thought about the inefficiency and lack of clarity in the solution, encouraging behaviourist
practices in them, as they manifested in several teacher meetings.
However, the students managed to move forward, understanding that a transformation in their way of
thinking about one subject, from traditional teacher centred to student centred with PBL. Traditional local
schools are behaviourist. In this regard, a student stated:
“That is called critical thinking, making a significant decision. That is why it makes it necessary, and
that is why we have advanced so much. We put both decisions. For example, in each class, we
change. Still, today we achieve that it will not modify. This situation occurs when the teacher tries to
change the problem, but the students have learned to argue their point of view facing the problem.”
A Student translated from Spanish
Regarding the role of the reading, writing, and drawing facilitator, the students affirmed that they had received
the support, and it has been helpful for the formulation of the problem. Reading and writing usually are difficult
for students. However, Drawing is an engineering subject that often has an excessive workload in engineering
programs. Nevertheless, the students seem to have connected with the other subjects. When asked about it,
they stated:
“As we have been in the introduction, they have put us as terms, all those things, plates, plates that are
worked freehand, plate in a cube, yes as the terms, as the line types. “ A Student translated from Spanish.
The chancellor has been an element with a high degree of influence in the initial part of the project's
development. However, his interventions with the students have caused conflict with the other teachers. The
students stated there was a contradiction between what he said and the teachers, with a mistaken perception
of disorganisation. According to the students' backgrounds, it is better to follow pre-established class
schedules with well-defined projects, but not with the responsibility carried out by PBL.
“Well, we have the topic and the problem question. The subject from auditory pedagogical strategies, but
we had it as inefficient- Then, talking with teacher Anonymous, he told us that the word insufficient was
better. Insufficient auditory pedagogical approaches remained in the Torca institution. The situation is
this. There are not enough auditory pedagogical strategies in the school that promote the participation
of the members of the educational community, as well as the development of listening and speaking
skills in students. Now on Thursday, we will have a class with a teacher who will define that well for us
to see if we can leave it or change it for something else." A Student translated from Spanish
As the previous text marks, at the beginning of the projects, the Chancellor proposed thematic solutions on
details that the students did not take as a recommendation for reflection but as a suggestion that would later
be transferred to the other facilitating teachers, causing conflicts between them.
5 Conclusion and perspectives

The article described Torca's transdisciplinary model and its first results in student learning outcomes during
the problem formulation stage, showing that the model is viable and produces learning but needs minor
adjustments. Since the model involving students, professors, facilitators, chancellor, and the community is
transdisciplinary.
Although the students managed to define the problem and stated that they are learning, the formulation of
the problem occurs in six weeks, which looks pretty long. It is mainly due to various disciplines and their
concepts in identifying issues. Also, the need to link problems with ILOs caused some difficulties.
Difficulties incorporating many viewpoints seemed to foster students' critical thinking and argumentation.
Finally, the students found a problem and its objectives that appropriately fit everyone to the subjects.
However, until the end of the semester, they will show the deep of their learning.
256
6 References
Alcaldía local de Sumapaz. (2022). http://www.sumapaz.gov.co/
Cita Triana, N. C., Sierra Lopez, L. P., Ordoñez Ordónez, C. L., & Cepeda-Valencia, J. (2020). Aprendizaje basado en proyectos
(ABP) para desarrollar habilidades académicas en la educación superior: una experiencia en Sumapaz. Praxis
Educación y Pedagogía, 5, 74–93. https://doi.org/10.25100/praxis_educacion.v0i5.8791
De Graaff, E., & Kolmos, A. (2003). Characteristics of Problem-Based Learning. International Journal of Engineering Education,
19(5), 657–662. https://doi.org/0949-149X/91
Kolmos, A., de Graaff, E., & Du, X. (2017). Diversidad del PBL: Principios y modelos de aprendizaje. In F. Rodriguez-Mesa, A.
Kolmos, & A. Guerra (Eds.), Aprendizaje basado en problemas en ingeniería (pp. 39–57). Aalborg Universitetsforlag.
Neame, R. (1989). Problem-based learning in Medical Education: The Newcastle Approach. In H. G. Schmidt, M. Lipkin, M.
W. de Vries, & J. M. Greep (Eds.), The Evaluation System at the Maastricht Medical School BT - New Directions for
Medical Education: Problem-based Learning and Community-oriented Medical Education (pp. 112–146). Springer.
Neufeld, V. R., Barrows, H. S., & Howard S. (1974). The “McMaster philosophy”: an approach to medical education. Journal
of Medical Education, 49(11), 1040–1050. https://doi.org/10.1111/j.1365-2923.1987.tb00366.x
Olsen, P. B., & Pedersen, K. (2008). Problem-oriented project work- A workbook (P. B. Olsen & K. Pedersen (Eds.); pp. 63–65).
Roskilde University Press. /Users/Fernando/Documents/Project
Ordóñez-Ordóñez, C. L., Cita Triana, N. C., & Sierra López, L. P. (2020). PEAMA Sumapaz: cuatro semestres de aprendizaje
basado en proyectos, ABP. Boletín Pedagógico, 1(1).
Rogers, C. R. (1951). Client-centred therapy. T. Gordon and N. Hobbs.
Savin-Baden, M., & Major, C. H. (2004). Foundations of Problem-Based Learning. Open University Press/SRHE.
Schmidt, H. G. (1989). Part III Problem-based Learning : Rationale and Examples. In H. G. Schmidt, M. Lipkin, M. W. de Vries,
& J. M. Greep (Eds.), New Directions for Medical Education: Problem-based Learning and Community-oriented Medical
Education (pp. 101–105). Springer-Verlag.
Acuerdo 025, (2007). http://www.legal.unal.edu.co/
Acuerdo 201, (2015). http://www.legal.unal.edu.co/
Resolución 405, (2016).
Resolución 908, (2018).
257
Are your rubrics hitting the mark? An exploration of the use of
rubrics for project-based learning in engineering
Teresa S. Hattingh1
1
Centre for Engineering Education, North-West University, Potchefstroom, South Africa
Abstract
Project-based learning is often introduced in engineering contexts to provide students with a better learning experience.
Projects can be more authentic and can create opportunities for students to integrate knowledge and skills from several
modules. Projects can also integrate an engineering curriculum with the simultaneous development and evaluation of
technical and professional competencies. Furthermore, it is often argued that project-based assessments lead to deeper,
more lifelong and self-directed learning. However, the assessment of work done during these projects presents many
challenges. One of these challenges involves marking mechanisms. Project-based learning assessment frequently uses
rubrics – designed to provide transparency regarding criteria used to evaluate students and consistency in the marking
process. Sometimes, these rubrics can even be used as feedback to students. The questions arise, what makes a good
rubric, and how should rubrics be designed to complement the intentions of project-based learning? This paper discusses
findings from a qualitative study that explored lecturer and student experiences of assessment in an engineering school
through interviews and focus groups. The study shows that the design of rubrics needs to be carefully considered to ensure
that suitable types of learning are both encouraged and rewarded. Poorly constructed rubrics can result in dysfunctional
student behaviours that do not support the intended learning outcomes of the assessment. This paper provides some
practical recommendations that can be considered when setting up project-based learning assessments and designing
assessment methods and rubrics to support these.
Keywords: Project-based learning; assessment; rubrics; student learning
1 Introduction
1.1 Background
In project-based learning, rubrics are often used as they are believed to be better suited to open-ended tasks
that engage higher-order thinking (Jonsson & Svingby, 2007) and accreditation body outcomes and critical
thinking (Newell, Dahm & Newell, 2002; Ralston & Bays, 2010). Furthermore, rubrics are often seen as a useful
means to improve the efficiency of grading and providing feedback (Panadero & Jonsson, 2013). The purpose
and function of rubrics vary, including clear expectations and transparency regarding the assessment and
grading process, feedback, and self-evaluation skills. Regardless of the intention, rubrics are intricately woven
into the grading or marks that students receive, and as a result, the power and influence that these rubrics
have in the learning process are unquestionable (Boud, 2007). While rubrics can be carefully designed, how
students respond is a function of how rubrics are perceived and engaged with (Lindberg-Sand & Olsson, 2008).
If project-based learning is to affect the positive and sustainable change in student learning that is intended,
a thorough understanding of what influences student assessment decision-making is required (Prosser &
Trigwell, 1999).
1.2 Purpose of this study

The original study from which this paper draws its findings was designed to understand how assessment was
designed and perceived by students in an Engineering School where student performance was significantly
lower than desired. The larger triangulated study drew on student surveys (Hattingh, Dison & Woollacott,
2019), lecturer interviews (Hattingh & Dison, 2019) and student focus groups (Hattingh & Dison, 2021) to
answer the following over-arching research question: How might current assessment practices be transformed
to improve the quality of student learning in the School? The study adopts a theoretical lens that draws on
learning-oriented assessment (Carless, 2015) and sustainable assessment (Boud, 2007). The intention is to
258
explore how student engagement, performance and success can be positively influenced by adapting
assessment approaches that focus on the learning that is taking place and prepares students for their own
future learning needs.
Many of the findings from the original study referred to rubrics, mainly used for projects, and how these were
designed and used by lecturers and perceived and engaged with by students. The purpose of this paper is,
therefore, to draw on the specific findings around rubrics to explore how rubric design can influence student
learning and, through this, make recommendations for rubric design to support the overall learning intentions
of project-based learning.
2 Literature review
2.1 Purpose of rubrics
Traditionally, the purpose of a rubric is to assign a level of performance or a grade (Goldberg, 2014) or "guide
the analysis of the products or processes of students' efforts" (Moskal & Leydens, 2000). This is known as the
summative purpose of a rubric. When used for a summative purpose, much focus is on the quality of
assessment grading (Panadero & Jonsson, 2013), considering reliability and validity of the rubric itself, inter-
and intra-rater reliability (Moskal & Leydens, 2000) or how accurately a score can reflect a student's abilities
(Newell et al., 2002).
A formative approach to rubric use considers alignment with course objectives, clarifying expectations to
students (Newell et al., 2002), and providing feedback (Catete, Snider & Barnes, 2016). Rubrics can also be used
to provide a quick overview of student progress (Ralston & Bays, 2010) which can be used to adapt the learning
environment.
Rubrics, if effectively designed, can be used both summatively and formatively (Stegeman, Barendsen &
Smetsers, 2016). They should articulate expectations for an assessment task and typically do this through
criteria and descriptions of levels of quality in relation to each criterion (Reddy & Andrade, 2010). Rubrics in
more design or project-oriented contexts can include criteria that consider design process elements such as
problem definition and feasibility, product-oriented criteria such as user experience and professional
competencies such as teamwork and entrepreneurship (Huang & Jong, 2020).
2.2 Impact of rubrics on student learning

Rubrics can positively influence student learning in several ways. Rubrics are a mechanism to make assessment
criteria explicit which provides transparency to students, but it also requires lecturers to reflect on these criteria
when setting up assessment tasks. In this way, rubrics can facilitate constructive alignment (Biggs, 1996) of
assessment design which supports improved student learning. Rubrics can increase transparency, reduce
anxiety and provide feedback to students (Panadero & Jonsson, 2013). For this feedback to be valuable,
students need to identify what good performance is, how their own performance relates to this and how they
can go about closing the gap (Stegeman et al., 2016), promoting sustainable assessment practices (Boud &
Soler, 2016). All of this supports the development of self-regulation skills and self-efficacy (Panadero & Jonsson,
2013). Rubrics are further able to support self-assessment (Andrade & Valcheva, 1999) and related activities
such as peer evaluation (Mullen, 2003) which can support the development of lifelong or sustainable learning
skills (Boud & Soler, 2016).
2.3 Challenges of rubrics

There are, however, many challenges with rubric design. Some of these stem from rubrics that are not well
designed or designed with a predominantly summative focus. Others originate from a lecturer-focused
approach that does not consider how students will engage with and respond to the rubrics and the impact
that this will have on their overall learning experience.
The criteria of many rubrics focus on the product or artefact that is being assessed (Catete et al., 2016; Verleger,
Rodgers & Diefes-Dux, 2016), which can mean that there is limited consideration, visibility or evaluation of the
259
process that is taking place and the assessment task remains product-centred (Gibbs, 1995). While it is often
easier to grade or evaluate something tangible such as a design or a design report, it is the process that
students will take with them into the working world. If rubrics are to be used to provide formative feedback to
students, the criteria and descriptors also need to provide guidance on how the process that leads to the
artefact can be improved in the future. If rubrics are to be used to develop self-evaluation capacity, students
need this visibility to judge their approaches.
Criteria in rubrics often include scale descriptors such as adequate, reasonable or poor (Huang & Jong, 2020),
clear, accurate, complete and fair (Ralston & Bays, 2010) or few, some, most, all, slightly, moderately, mainly
and extremely (Tierney & Simon, 2004). While these may enable raters to be more consistent and reliable, one
may question how useful these descriptors are to a student and whether they enable valuable feedback that
students can engage with and improve. While a grader may be able to develop a sense of what complete or
reasonable is, students need to develop these skills. In some cases, examples can be included in the descriptors,
often to improve reliability and validity or grading, but also to try and provide better guidance to students.
Depending on student dispositions and intentions when using the rubric, this can tempt students to adopt
procedural or tick-boxing approaches, defeating the rubric's objective to develop self-evaluation skills.
Whenever using rubrics, it remains key to consider how the students will perceive and use the rubrics during
the design process. By providing clear expectations and criteria, rubrics provide a very structured framework
that students can use when tackling assessment tasks. Some would argue that students could become
dependent on these rubrics. The clear guidelines could lead students to avoid a trial-and-error approach to
their problem-solving processes (Panadero & Jonsson, 2013), following a more mechanistic procedural-
learning approach (Case & Marshall, 2004). This does mean that rubrics can encourage instrumentalism or
assessment as learning where criteria-compliance dominates the learning process (Torrance, 2007), which
works against the idea of project-based and sustainable assessment practices.
2.4 Opportunities for consideration

The value of rubrics often remains centred on the robustness and inter or intra-rater reliability. Rubric design
principles tend to speak to inconsistencies and redundancy in descriptors, unevenness in increments and
limited routes to partial credit (Goldberg, 2014), language, terminology and phrasing (Stegeman et al., 2016)
and frequency and intensity of descriptor levels (Tierney & Simon, 2004). Even when rubrics are designed to
provide feedback, analysis of the rubric often remains focused on reliability and validity and how the feedback
is perceived and used by students is not evaluated (Catete et al., 2016; Stegeman et al., 2016). While
sophisticated rubrics provide a structure that can be used to provide a holistic and accurate evaluation that
extends beyond the product or artefact to critical thinking (Ralston & Bays, 2010), very rarely do the studies
discuss how the rubrics are perceived or used by students.
Panadero & Jonsson (2013) conducted a review of literature that explores the impact of rubrics on student
learning. Their findings revealed that most research remains focused on the summative aspect of rubrics and
limited studies consider the formative impact of rubrics. For those studies that do look at the formative aspect,
few have been conclusive as case studies that use rubrics formatively typically combine rubric use with other
meta-cognitive activities such as peer-evaluation.
While the value and importance of the summative aspect of rubrics are acknowledged, their influence on
student learning behaviours and, ultimately, student learning cannot be ignored. Learning-oriented and
sustainable assessment thinking requires that the primary purpose of any assessment activity is the support of
student learning and the development of self-judgement and evaluation skills. Since rubrics form an integral
part of this process in project-based learning, it is imperative to understand how rubrics can be designed to
support this.
260
3 Method
3.1 Study context
This study takes place at an engineering school ('the School') in a South African university. Assignments (or
projects), tests, and exams are the primary means of assessing student performance. Larger projects are used
in various courses but mainly in design courses and can be conducted individually or in groups. Although the
predominant means of assessing remains tests and exams, there has been a distinct shift toward the use of
projects. Projects can contribute significantly to the overall course mark in practical and design courses.
3.2 Research approach

The findings presented in this paper draw from two parts of a larger triangulated case study. The original study
consisted of a student survey, lecturer interviews and student focus groups to obtain a holistic view of
assessment practices and how these influence student learning behaviours. The study used a qualitative
exploratory approach to draw on individual and group experiences. This paper will draw specifically on findings
from the lecturer interviews and student focus groups. The study aims to understand the decision-making and
intentions of lecturers when designing and using rubrics and the students' experiences and intentions when
tackling assessment tasks that use rubrics. The study does not focus on a particular module but considers
students' holistic approach to assessment tasks since the orientation of a student towards assessment is
influenced by their prior learning experiences (Biggs & Tang, 2011) and what happens in a particular course
and around it (Boud & Soler, 2016).
3.3 Lecturer interviews data collection

Semi-structured, individual interviews were conducted with ten purposively sampled lecturers in the School.
Lecturers were posed with a series of open-ended questions designed to explore their perceptions and
experiences of the overall purpose of assessment. These included factors that influence the design of
assessment tasks; explicit and implicit criteria used to design and evaluate tasks; communication of
expectations and criteria to students, feedback, experiences of student engagement with tasks, how well
assessments evaluate the intended outcomes and how assessment could be improved. The interviewed
lecturers teach a range of courses across all four years of study, including mechanics, engineering drawing,
mechatronics, engineering design and laboratory courses and complementary courses such as business
management.
3.4 Student focus groups data collection

Focus groups were conducted using a protocol that encouraged students to reflect on both their own
experiences and, where necessary, to comment on what other students might experience or do (Merriam,
2009). The questions probed several issues, including students' overall approach to their studies in respect of
different assessment tasks; how they knew what was expected from them in assessments; an example of a
situation in which they were disappointed by an assessment, and how they reflected on this experience; forms
of received feedback, and their response to feedback. Students were asked to reflect individually by writing
down their thoughts on selected questions before engaging in the group discussion, allowing them to
formulate their thoughts before being influenced by others (Gibbs, 2007). The focus groups were facilitated by
the researcher and observed by a research assistant. The sessions were recorded and later transcribed.
The focus groups sampled all students in the School from the second, third, and final years of study. Four
separate groups of students were chosen for the focus groups using maximum variation sampling (Cohen,
Manion & Morrison, 2011): a mid-performing group (FG1), a high-performing group (FG2), a low-performing
group (FG3) and a group of students categorized as turnaround students (FG4). The turnaround students
performed exceptionally poorly in one year, followed by a year when they performed particularly well. Emails
were sent to students from all groups, inviting them to be part of the focus group on a specified day. The
number of emails sent out was increased until five to ten students (Merriam, 2009; Cresswell, 2012) consented
to be part of each focus group. A total of 22 students participated in the focus groups.
261
3.5 Data Analysis
Recordings of the interviews and focus groups were transcribed and analysed. Focus group analysis included
individual voices, and the sense-making that emerged as a shared understanding was developed in the group
(Cresswell, 2012; Wilkinson, 2004). The original study tracked emerging concepts using a coding system linked
to key supporting quotes (or evidence). This process was repeated for each transcript, comparing and adding
codes when required. The identified codes and evidence were then captured into a case study database (Yin,
2014). A comparative analysis was then carried out, clustered into topical categories informed by the literature
(Merriam, 2009) until a set of emergent themes was obtained. To address the credibility of the findings, rich
descriptions and evidence were used to support the claims. Surprising or 'outlying' evidence was investigated,
and rival explanations were considered when interpreting data by referring to literature and using peer review
with two colleagues (Cohen et al., 2011; Merriam, 2009). For this paper, themes and codes that linked
specifically to the topic of rubrics were extracted and are discussed herein.
4 Findings and discussion

4.1 Introduction to the findings
Although there are indications that assessment is sometimes used as a learning opportunity, the predominant
thinking in the School is assessment of learning, to evaluate the competence of students. This thinking frames
most decisions that lecturers make when designing and using assessments. Lecturers aim to discourage the
use of rote learning strategies and attempt to test if students have a deep understanding of concepts by
requiring students to apply their knowledge and understanding of concepts to new and unseen problems,
often using real-world scenarios. Students indicate a preference for projects as they enable them to better
understand concepts and are more relevant to the real world and the engineering profession. Student learning
behaviours in the School are, however, dominated by a studying for passing paradigm where students
strategically make decisions that will enable them to pass or obtain as many marks as possible, often at the
expense of learning. Within this context, findings of particular interest to the topic of rubrics are discussed. The
quotations are referenced back to the data source, i.e., L1 representing lecturer 1 and FG1 representing focus
group 1.
4.2 Product or output-focused nature

Lecturers describe an underlying set of skills that students are expected to exhibit. These align with the process
of problem-solving and include: to understand and visualise complex problems, to source and understand
relevant information, to apply appropriate concepts to analyse/solve complex problems, to integrate the
problem into related systems and to communicate ideas. However, many lecturers describe the criteria used
to evaluate students in terms of the specific requirements that the product that was being designed needed
to meet or sections of a report rather than the outcomes that the student needed to demonstrate to be able
to design the product so that the product met the requirements. Rubrics reflect this and are often designed in
a way that provides criteria simply as mark allocations for each section of a report or task. Gibbs (1995) warns
against using rubrics in a way that does not provide any indication of the quality of work that is required to
achieve the marks that are allocated for a particular section. Furthermore, as the assessment criteria are
product-oriented, they understate what students need to achieve and provide minimal scaffolding for students
to appreciate the concept of quality in their work (Sadler, 2010). Feedback also typically focuses on the product
of the assessment task, indicating how the answer or design has not met the engineering specifications or
requirements without suggesting what the student could have done differently to improve the quality of their
engagement with the task to deliver a quality final product. The feedback is, therefore typically aligned to
"what" needs to be improved, a focus on the diagnostic element, and not the "how" of getting there, the
bridging-the-gap element (Boud & Molloy, 2013). This simplistic view of criteria used in the rubrics disregards
the process of learning and engagement, which affects feedback and student use of rubrics, including the
development of judging capabilities and self-evaluative expertise (Carless, 2015).
262
The breaking down of criteria into discrete elements or steps is perceived to ensure consistency and objectivity
in marking. However, the criteria that assess quality or understanding underpinning the steps become much
more difficult for lecturers to explain. Some lecturers admit that they find it difficult to explain to students what
is required of them to perform well:
"Students often ask: …how can I pass this course? And even though I get asked this a lot of times, I always struggle
to answer it….have you understood and can your design, do what it needs to do?" (L9)
This inability of some lecturers to explain to students how to improve could stem from a lack of awareness of
the importance of threshold or bottleneck concepts and their role in unlocking understanding and the
construction of knowledge and ideas (Middendorf & Pace, 2004). As a result, rubric criteria and therefore,
feedback does not articulate to students what they need to demonstrate to meet the required outcomes, which
further hampers the development of self-evaluative skills.
4.3 Tick-boxing and instrumentalism

To facilitate a shift to learning-oriented and sustainable assessment, attempts need to be made to develop
students' capacity to self-evaluate. This can be done in several ways, but improving the communication of
expectations and criteria to students is a key priority. However, as students are already operating in a studying
for passing paradigm, there is a risk that increasing the transparency of outcomes and criteria could lead to an
over-reliance on these with criteria compliance (Torrance, 2007) replacing learning. As the content of rubrics
remains product rather than process-centred, this can lead students to use the rubrics as a means of cue-
seeking and mark-hunting, resulting in mechanistic and "tick-boxing" strategies. This lecturer reflects:
"So I almost think that the very detailed rubric can be a disadvantage. Because they're just trying to tick boxes at
the end of the day." (L1)
Although assignments and projects are seen as a better way of encouraging students to develop the necessary
process-type skills, the use of rubrics in this School is potentially encouraging students to go through the
breakdown, ticking off steps in a procedural manner in order to "get marks". This is certainly exacerbated by
the already existing student focus on marks in the School, where students appear to prioritise and focus their
attention based on the marks that are provided for all assessment tasks:
"that's why the interactions that you end up kind of having with the lecturers are okay, why are my marks so
low…it's not okay, help me understand this concept, it's my mark needs to advance…so at the end of the day, it's
all about marks." (FG2)
Students particularly do not see value in feedback for an assignment if they are not going to get something
similar again since the rubric is only used to "tick boxes" against mark allocations rather than change learning
behaviours and approaches to solving problems that are universally useful in future learning.
"Like also in the rubric, it's not always helpful, because some of the lecturers only give it to you when (you've
already done the work) and that doesn't help you because then you're kind of just doing whatever and when you
get the rubric you might have to add something in or take out or stuff. So if they give it to you beforehand you
then know like how to use your time." (FG1)
"I think getting a rubric also helps. Especially if there'll be a follow or similar type thing, then you can see exactly
where the marks are, where you need to improve." (FG2)
The product-centred structure of rubrics and tick-boxing approach of students can also lead to mechanistic or
boxing approaches to tackling assignments:
"…the students have a mechanistic approach …{without} the subtleties as to why you should be doing this or that."
(L8)
This can also be amplified by group projects:
"Students end up splitting parts of a report (between group members). And this is a problem, not only because
you're not actually practising but also because then you're not realising what the link is between everything. And
if you're having to do the entire project that becomes an entirely different story." (L7)
263
The adoption of mechanistic approaches implies a lack of underlying problem-solving skills and strategies and
the development of skills that can integrate elements. This can exist between aspects of an assignment, sections
of reports, problems and surrounding systems and how courses and the degree links together as a whole.
4.4 Agency and self-evaluation

Findings from this study reveal that students are not necessarily developing self-evaluative skills and do not
adopt agency for their own learning. There is a sense that the agency for learning and assessing lies with
lecturers, impeding the ability of students to develop their technical and professional competencies. Students'
ability to self-evaluate and improve is limited by the current product-centred nature of rubrics which do not
enable students to see where they are going wrong or develop ways of reflecting, changing and improving.
The following participant sums up this frustration:
"You work really hard on your assignment and you get it back and you realise you've gone wrong, and it gets you
really down because you spent so much time doing it and it doesn't show you what's wrong." (FG1)
This quote also hints at a lack of agency. It appears that students expect lecturers to "tell them what is wrong"
rather than developing the ability to self-evaluate where they went wrong.
"…lecturers should sit and see where most students are going wrong and try to come to class and work out some
of those mistakes that students make." (FG1)
This is confirmed by lecturers who also discuss their frustrations:
"That is the number one question students always ask is what are you expecting? Where's the rubric? Is this
right?...Their biggest question is, how will I be assessed? Is this work good enough? Have I done enough?" (L4)
The development of self-evaluative skills is a crucial aspect of sustainable assessment and project-based
learning. However, the assessment context needs to provide structure and support for students to identify with,
adopt and exercise agency to create opportunities for developing these skills (Ritchie, 2016).
5 Conclusion
The findings show that rubrics can indeed influence student learning behaviours. Furthermore, they show that
they can have a negative impact on the competencies that project-based learning aims to develop. As a result,
rubrics should be intentionally and carefully designed, and the perception and response of students should be
investigated. Practically, rubrics should incorporate criteria and descriptors that are process-centred and
encourage students to integrate their learning in other projects and courses to develop sustainable, lifelong
learning skills. They should provide sufficient guidance for students to develop self-judgement skills without
becoming over-reliant. It is also recommended that the use of rubrics includes supplementary meta-cognitive
activities that transfer agency and scaffold the development of relevant skills. This requires lecturers to think
beyond the reliability and validity of rubrics, considering the student learning process, threshold concepts and
constructive alignment.
6 References
Andrade, H., & Valtcheva, A. (2009). Promoting Learning and Achievement Through Self-Assessment. Theory Into Practice, 48(1), 12–19.
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher education, 32(3), 347-364.
Biggs, J. & Tang, C. (2011). Teaching for quality learning at university. London: Open University Press.
Boud, D. (2007). Reframing assessment as if learning were important. In. Boud, D., & Falchikov, N. (Eds). Rethinking assessment in higher
education: Learning for the longer term. London: Routledge.
Boud, D., & Molloy, E. (2013). Feedback in Higher and Professional Education: Understanding it and doing it well. Oxon: Routledge.
Boud, D., & Soler, R. (2016). Sustainable assessment revisited. Assessment & Evaluation in Higher Education, 41(3), 400-413.
Carless, D. (2015). Exploring learning-oriented assessment processes. Higher Education, 69(6), 963-976.
Case, J., & Marshall, D. (2004). Between deep and surface: procedural approaches to learning in engineering education contexts. Studies in
Higher Education, 29(5), 605–615.
Cateté, V., Snider, E., & Barnes, T. (2016, July). Developing a rubric for a creative CS principles lab. In Proceedings of the 2016 ACM Conference
on Innovation and Technology in Computer Science Education, 290-295.
Cohen, L., Manion, L., & K. Morrison. (2011). Research Methods in Education, 7th Edition Oxon: Routledge.
264
Creswell, J.W. (2012) Educational Research: Planning, conducting and evaluating quantitative and qualitative research, 4th Edition, Boston:
Pearson Education.
Gibbs, G. (1995). Assessing student centred courses. Oxford. Oxford Centre for Staff Development.
Gibbs, G. (2007). Analyzing qualitative data. London: Sage.
Goldberg, G. L. (2014). Revising an engineering design rubric: A case study illustrating principles and practices to ensure technical quality
of rubrics. Practical Assessment, Research, and Evaluation, 19(1), 8.
Hattingh, T., Dison, L., & Woollacott, L. (2019). Student learning behaviours around assessments. Australasian Journal of Engineering
Education.
Hattingh, T. S., & Dison, L. (2019). Why lecturers in an engineering School assess the way they do. In Research in Engineering Education
Symposium (REES), July 2019, Cape Town, South Africa.
Hattingh, T. S., & Dison, L. (2021). How assessment shapes learning: A perspective from engineering students. SAIEE Africa Research Journal,
112(4), 161-170.
Huang, B., & Jong, M. S. Y. (2020, August). Developing a Generic Rubric for Evaluating Students' Work in STEM Education. In 2020
International Symposium on Educational Technology (ISET), IEEE, 210-213.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational research review,
2(2), 130-144.
Lindberg-Sand, Å., & Olsson, T. (2008). Sustainable assessment? Critical features of the assessment process in a modularised engineering
programme. International Journal of Educational Research, 47(3), 165-174.
Merriam, S.B. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass
Middendorf, J., & Pace, D. (2004). Decoding the disciplines: A model for helping students learn disciplinary ways of thinking. New directions
for teaching and learning, 2004(98), 1-12.
Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical assessment, research, and evaluation,
7(1), 10.
Newell, J., Newell, H., & Dahm, K. (2002). Rubric Development And Inter Rater Reliability Issues In Assessing Learning Outcomes. In
Proceedings of the 2002 American Society for Engineering Education Annual Conference & Exposition, 7-991.
Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational research
review, 9, 129-144.
Prosser, M. & Trigwell, K. (1999). Understanding learning and teaching: the experience in higher education. Buckingham: Open University
Press.
Ralston, P., & Bays, C. (2010). Refining a critical thinking rubric for engineering. In Proceedings of the 2010 American Society for Engineering
Education Annual Conference & Exposition, 15-1022.
Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & evaluation in higher education, 35(4), 435-
448.
Ritchie, L. (2016). Fostering Self-Efficacy in Higher Education Students. London: Palgrave.
Sadler, D.R. (2010). Beyond Feedback: Developing student capability in complex appraisal. Assessment and Evaluation in Higher Education,
35(5), 535-550
Stegeman, M., Barendsen, E., & Smetsers, S. (2016, November). Designing a rubric for feedback on code quality in programming courses.
In Proceedings of the 16th Koli Calling International Conference on Computing Education Research, 160-164.
Tierney, R., & Simon, M. (2004). What's still wrong with rubrics: focusing on the consistency of performance criteria across scale levels.
Practical Assessment, Research, and Evaluation, 9(1), 2.
Torrance, H. (2007). Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post‐
secondary education and training can come to dominate learning. Assessment in Education, 14(3), 281-294.
Verleger, M. A., Rodgers, K. J., & Diefes‐Dux, H. A. (2016). Selecting Effective Examples to Train Students for Peer Review of Open‐Ended
Problem Solutions. Journal of Engineering Education, 105(4), 585-604.
Wilkinson, S. (2004). Focus groups: A feminist method. In. Hesse-Biber, S.N. and Yaiser, M.L. (Eds.). Feminist perspectives on social research.
Oxford: Oxford University Press.
Yin, R.K. (2014). Case study research: Design and methods. 5th edition. Thousand Oaks: Sage.
265
Proposal of Method for Risk Assessment of Project-Based
Learning Failure via AHP and BBN – A Case Study
J.C Pereira1, E. Stefano2, F. Almeida1
1,3
Master in Engineering Department, UCP (Universidade Católica de Petrópolis), Petrópolis – Rio de Janeiro, Brasil
2
Industrial Engineering Department - Universidade Federal Fluminense - Petrópolis – Rio de Janeiro - Brasil
E-mail: [email protected], [email protected], [email protected]
Abstract
Project-based learning has been used in engineering education because it is expected to develop students' professional
knowledge. Project-based learning research and practices in engineering education are growing. The author conducted in-
depth literature research to identify the risk factors in PBL application and observed that risk assessment in implementing
the PBL via Analytical Hierarchy Process and Bayesian Belief Networks had not been addressed. The study aims to propose
a method to identify the risks of using the Project-Based Learning method and involving companies when teaching
engineering disciplines to engineering students. A Case Study was conducted on the application of PBL to support local
companies in a specific region. The risks were identified, a relationship digraph was used to identify the categories of risks,
a survey was conducted to elicit risk probabilities, AHP was used to estimate risk impact, the global risk scores were
estimated, and the responses to the risks were defined. BBN was also used to combine the probabilities of risks and perform
sensitivity analysis. Future research directions for engineering education researchers are proposed to optimize Project-
based learning curriculum design. The study provides a method to be used by professors, students, and decision-makers
to identify risk factors that can impact the quality of teaching in engineering education and implement proper risk
responses to the risks. The contribution is significant since it improves the quality of universities.
Keywords: Active Learning; Engineering Education; PBL; Risk Assessment, BBN, AHP
1 Introduction
Student-centered teaching methods by using Project-based learning (PBL) have been widely used. Still,
universities find it difficult to deal with unexpected issues during the implementation phase and often return
to traditional teaching methods (Henderson, 2012). Therefore, it is essential to identify, describe, and deal with
risk factors that directly impact PBLs. It has been noticed that current literature on the Use of PBL in engineering
education has not addressed the risks of PBL failure, and even less attention has been paid to risk responses
to ensure excellence in projects. In this study, the author conducted a literature review to identify and list key
risk factors when implementing PBL in engineering education. Being aware of these risks and how to respond
to them can improve the chances of success when implementing PBL to enhance student learning and provide
a sustainable teaching method at universities. The objective is to propose a method for risk assessment using
Analytical Hierarchy Process (AHP) and Bayesian Belief Networks (BBN) to identify the risks and responses to
ensure successful PBLs are conducted in collaboration with local companies. If risk factors are not identified,
and proper responses are not provided PBL project can fail. This study assesses the risks and completes a gap
in the literature by assessing the risk factors of using the Project-Based Learning Method in teaching
engineering students. The diversity of risk factors presented here highlights the complexity of implementing
PBL in engineering education. To be addressed, most of these risks require proper responses, and this need
justifies this project. If risks are not identified and appropriately addressed, PBL projects can fail, and the
students, professors, and partner organizations will not obtain the expected benefits. This paper is significant
since it will help the development of skills and practice of professors and students and increase the quality of
PBL. It is noteworthy here that the study focuses on project-based learning involving companies. Successful
PBLs, in this case, will result in a significant reduction of business risks and achievement of excellence in quality
and organizational safety. None of the studies present herein dealt with risk assessment and its impacts on
PBL. The literature revealed that most previous studies addressed PBL and challenges without giving a
methodology for risk prioritization using AHP (Analytical Hierarchy Process) and Bayesian Belief Networks.
266
The paper responds to the following research questions:
Research Question 1: What are the most significant risk factors for using PBL in collaboration with local
companies when teaching engineering students?
Research Question 2: What are the probability and the impact of these risks and the global risk scores?
Research Question 3: What are the responses to high RPN risks, and how to combine the risks and conduct a
sensitivity analysis?
This paper is divided into five sections; the first introduces the concepts of the study, the second describes PBL,
the challenges and Risk Assessment of PBL Failure, BBN (Bayesian Belief Network), AHP (Analytical Hierarchy
Process), and Section 3 describes the Methodology, section 4 results, and section 5 concludes.
2 Project-Based Learning
Previous significant studies about PBL Methodology and Challenges in its implementation are presented in
subsections 2.1 and 2. 2.
2.1 Design of PBL Method

Previous significant studies about PBL are presented herein. Moliner et al. (2019) developed a work focused on
describing the experience of using PBL methodology in Materials Science courses conducted by four different
Spanish universities on different engineering degrees. The author analyzed and evaluated how the PBL was
perceived by the students and lecturers who participated in the PBL process. Setiawan (2019) conducted a
study focusing on the implementation of PBL, specifically on the opportunities and challenges. The students
choose their topic, identify it, explain why they choose it and solve the problem. Thevathayan (2018) presented
an experience evolving a hybrid-teaching model by using the action research cycle plan-act-observe-reflect
over three semesters. The main novelty of the approach was the use of projects with varying levels, which gave
students an enjoyable and beneficial project experience. Marques (2018) proposed a formative monitoring
method to help students be aware of their individual and team performance. The results indicated that PBL
effectively enhanced the learning experience in the instructional scenario studied. Schneider (2020) used PBL
to enhance student engagement, and Daun (2016) discussed results from the long-term application of such a
course design in a graduate setting. In addition, he indicated that project-based learning techniques foster
different teaching goals in graduate and undergraduate settings. Du et al. (2013) developed a framework of
change in educational culture by using a PBL methodology. This framework aims to inspire curriculum design
for education and analyze the implementation of PBL in each cultural context.
Palmer and W. Hall (2011) presented PBL offering in engineering PBL at Griffith University in Australia. The
author observed that students generally enjoyed the experience, and the aspects needing improvement were
listed and documented. García-Martín and E.Pérez (2017) presented a method to guide teachers using PBL
principles and several instructional design models. In particular, the process deals with the definition of a
problem facing three fundamental issues in active learning, especially in PBL: Students' Motivation, Supporting
Students' Work, and Autonomous Working. The authors focused on academic contexts where instructors are
starting to use this Methodology and students are not dealing with ill-structured projects. Du Bani et al. (2018)
presented the main challenges facing PBL. The challenges included the type of projects, how to team up
students, how to proceed with planning, how to swap planning outputs among teams, and how to implement
a Project.
2.2 Challenges and Risk Assessment of PBL Failure

Previous significant studies about the Challenges of PBL are presented herein. The Henderson et al. (2012)
survey mentioned that faculty are aware of student-centered teaching methods but find it challenging to deal
with unexpected issues during the implementation phase. Thus, they often return to traditional teaching
methods. Kjellberg et al. (2015) stated that implementing PBL is the holistic perspective of the project and that
in most projects, the non-technical responsibilities are not clearly defined. The author says the complete
infrastructure is not defined, probably due to a holistic project perspective and project management methods.
267
These authors also stated that novice teams affected knowledge transfer and communication within extended
teams, affecting group dynamics, commitment, and responsibilities. The authors highlighted that lack of
teacher teams leads to one teacher acting as both examiner and PM. The authors also emphasized that the
"two-hats" issue added to the teacher workload and created emotional stress due to the lack of tools and
support and the constant brooding on addressing issues that appear. Beddoes et al. (2010) explained that
challenges to the implementation and execution of PBL are both theoretical and practical. Theoretically,
debates remain over the best approach to incorporate PBL and the performance necessary to benefit students.
Some engineering educators argue that the maximum benefits of PBL will not be obtained unless it is
implemented across the entire curriculum and all at once (Inelmen, 2003).
On the other hand, some argue that due to the significant differences between PBL and traditional methods,
instructors should start with small-scale initiatives to incrementally familiarize themselves with PBL (Hansen,
Cavers, & George, 2003). The changing roles of the teacher and the student are widely recognized as two of
the most significant barriers to the implementation of PBL (Prince & Felder, 2006; Strobel, 2009). PBL can be
difficult for faculty and students "because it challenges them to see learning and knowledge in new ways" and
blurs boundaries (Savin-Baden, 2007, p. 24). For instance, students may be hostile to PBL because they are
unaccustomed to the level of personal responsibility required and may experience conflicts with team members
(Prince & Felder, 2006). Moreover, teachers often find it difficult to adjust to PBL (Prince & Felder, 2006; Thomas,
2000). Furthermore, institutional difficulties include resources, scalability, physical facilities, and management
(Bielefeldt et al., 2009).
2.3 BBN (Bayesian Belief Network)

BBN has been widely used to estimate corrosion risks (Yang et al., 2016). Bayesian Network Networks were
used to ease knowledge acquisition of causal dependence in CREAM (Ashrafi et al., 2016). The Bayesian variable
was used in the selection to analyze regular resolution IV two-level fractional factorial designs (Chipman et al.,
2016). BBN's is a causal structure used by probability risk analysis specialists to obtain information about
important risk events and the necessary interventions to address risks (Rechenthin, 2004; Mosleh, 1992). The
Use of BBN's in safety, maintenance, and reliability has increased quickly (Mahadevan, 2001). Bayesian methods
have been used comprehensively in many applications and provide a structure for addressing the limitations
of human reliability analysis (Podofillini and Dang, 2013; Mosleh and Apostolakis, 1986; Droguett et al., 2004;
Groth and Swiler, 2013). None of the above previous studies deals with the application of BBN to the Risk
Assessment of Project-Based Learning Failure. The objective of the BBN methodology is to allow more
straightforward predictions of risk events and also execution of sensitivity analysis; it is a structure representing
arguments when uncertainty exists.
2.4 AHP (Analytical Hierarchy Process)

Saaty (1980) was the first to use AHP as a decision-making tool to provide the relative weight of criteria based
on a hierarchy structure. The author proposed the use of pairwise comparison to evaluate alternatives. The
method has been used extensively to solve complex decision problems. It divides a problematic issue into
smaller parts aiming at ranking them hierarchically. Thus, the relative importance of alternatives is weighed
accordingly. In this paper, AHP is utilized to consider/prioritize the key risks affecting the PBL result. The AHP
is an excellent tool to provide weight for the different risk levels; the first phase is to create a pairwise evaluation
matrix (A), as introduced by (Saaty 1980), by utilizing the relative importance scale. The matrix A represents a
pairwise evaluation matrix where each element "aij"(i, j = 1, 2, ..., n) represents the proportional importance of
two compared elements (i and j). The higher the value, the stronger the first element preference(i) over the
second (j). (Mls and Otcenaskova, 2013).
3 Methodology
The research was conducted in Google Scholar, Scielo, Scopus, and Web of Knowledge using the keywords:
Active Learning; Engineering Education; PBL; Risk Assessment, BBN, and AHP. The following Journals related to
engineering education listed in JCR Journal Citation Reports Full Journal List were also reviewed for state-of-
the-art papers on the subject: 1 – International journal of engineering education – Ireland; 2 - Journal of
268
engineering education – USA; 3 - Journal of professional issues in engineering education and practice – USA;
4 - Computer applications in engineering education – USA; 5 - International journal of electrical engineering
education – England; 6 - European journal of engineering education. For selecting the publications of interest,
they were searched by title, abstract, and keywords, and the focus was on the papers published in the last five
years. The following keywords were used: Active Learning; Engineering Education; PBL; Risk Assessment, BBN,
AHP. The searched papers were reviewed by reading the abstract and introduction; those relevant to the
research objectives were selected. In this project, 50 research papers were selected among 120 to identify risk
factors in the PBL use and implementation. The papers covering problem-based learning and not covering the
Design of PBL and challenges/risk assessment of PBL failure were disregarded.
A survey was conducted as a case study. The study adopted the approach of building theory from Case Study
Research (Eisenhardt, 1898) and Kin (1999). It combined data from archives, interviews, and observations and
focused on the application of PBL by a university in partnership with a company. A PBL process map was
constructed, and a list of risk factors found in the researched literature and in the process map was then
prepared. A quality management and planning tool named Relationship Digraph was used to cluster the risk
factors into categories and establish the cause-and-effect relationship to the failure of PBL. A survey in Google
forms was created to obtain the probability of students, professors, and organization leaders' risk factors.
Students, professors, and professionals from a local university and its major partner company completed the
survey in 2022. It was responded to by approximately 12 participants, who were asked to estimate the
probability that a given risk could cause the failure of a PBL project. AHP (Analytical Hierarchy Process) was
used to find the relative importance (impact) level of the risk categories. The global risk scores (combination
of probability and impact scores) were obtained by multiplying the probabilities and impact scores. The next
step was implementing risk responses to address the risk categories (Risk Priority Number). The last step was
preparing the BBN and performing sensitivity analysis to identify the risks with a higher impact on the PBL
failure. The steps of the Methodology are shown in Figure 1.
Figure 1. Methodology flowchart
4 Results
This section shows the process map, the list of risk factors impacting PBL failure, the AHP for the Technical
Learning principle, the list of probability and impact scores, the BBN combining all risk factors, and the tornado
chart resulting from sensitivity analysis. Following the sequence defined in the Methodology, the process map
and relationship digraph was used to prepare the list of risk factors organized by risk categories. An AHP matrix
was prepared for each Learning principle to obtain the risk factor's impact values. The probability values for
risk factors were obtained with the survey. The probability and the impact values for each risk factor were
combined to obtain the risk index. The risk index was color-coded as per the risk index value. Finally, a BBN
was also used to identify the most significant risks impacting PBL projects using sensitivity analysis.
269
The flowchart in Figure 2. shows the process map used by universities in conjunction with partner companies
in PBL projects and the risks presented in each step. The letters in red represent the category of risks: C:
Cognitive Learning Failure, S: Social Learning Failure, and T: Theory and Practice Learning.
Figure 2. Structure in the conduction of PBL projects and risks.
The risk factors were clustered into categories to establish the cause-and-effect relationship to the failure of
PBL. The affinity diagram was used to organize the risk factors within three learning principles and nine
categories, as Xiangyun (2013) suggested. Risk factors leading to each risk category were identified based on
the researched literature, as shown in Table 1.
Table 1. Risk Factors impacting PBL failure.
Risks
Type Categories Risk Factors
Identification
C1 Lack of procedure for PBL process
CA: No Standardization of
C2 Students and professors not appropriately trained on the PBL procedure
PBL Procedure
C3 Lack of standard work for the execution of PBL's
C: CB: PBL specific C4 Poor explanation of expectations to students
Cognitive requirements not defined C5 Lack of background definition on principle behind projects
Learning accurately C6 No clear definition of requirements
Failure C7 Project complexity incompatible with time and resources
CC: Wrong Choice of C8 The project is not related to discipline
Project C9 Workload too heavy for the student
C10 The low ability of students (slow learners)
S1 Number of Students in project inadequate (too big or too small)
SA: Team Building
S2 Project team members not equally strong
practices not used
S3 Assign students to teams rather than let them select the team themselves
S: S4 Professor does not give feedback on the project
SB: PBL Professor not
Social S5 Nonexistence of guidelines for team operation in the project
active in the project
Learning S6 Students not encouraged by professors
Failure S7 Some of the students not active in the project
SC: Team lack of S8 No focus on the project
Motivation S9 Relationship professor and student not good
S10 Students and Professor lack patience and enthusiasm
T1 Professor does not support knowledge base construction.
TA: PBL Professor not T2 Professor does not support Argument base construction.
prepared for the project T3 Lack of professor technical content knowledge and experience
T:
T4 Professor has no industrial skills.
Theory and
T5 Lack of definition for the project content organization
Practice TB: No definition of PBL
T6 No definition of project report content
Learning records organization
T7 Problem-solving methods not defined
Failure
T8 Students are not familiar with the specific process theory behind PBL.
TC: Students not
T9 Students have no knowledge of Quality Tools for problem-solving
prepared for the PBL
T10 Students not trained on specific PBL process
270
The probabilities for each risk factor were elicited from students and professors using a Google Form survey.
AHP was utilized to compare the risk factors in each learning principle category pairwise. An interview process
was conducted to obtain from professors and students the degree of impact of each risk on the failure of PBL.
The risk weight values (impact) were quantitatively calculated based on the completed pairwise comparative
matrix. The empirical data was converted into mathematical models using a hierarchy table established by
Saaty (2009). The relative importance of the risks is translated into the numerical pairwise comparison matrix
shown in Table 2.
Table 2. Hierarchy table
Importance Definition
1 Both elements are of equal importance
3 Moderate importance of one element compared to the other
5 Strong importance of one element compared to the other
7 Very strong importance of one element compared to the other
9 The extreme importance of one element over the other
AHP was utilized to obtain the impact for each risk and respective risk category. A pairwise comparative matrix
was prepared, as shown in Table 3, for the Technical Learning principle, and similar tables were prepared for
the other principles. An interview process was conducted to obtain from 3 professors and 6 students the degree
of impact of each risk on the failure of PBL.
Table 3. AHP for the Technical Learning principle
Criteria Comparison Matrix
TA: PBL Professor not prepared for the project
TB: No definition of PBL records organization

W
CB: PBL specific requirements not defined
SB: PBL Professor not active in the project

CA: No Standardization of PBL Procedure
TC: Students not prepared for the PBL

e
SA: Team Building practices not used
i
g
CC: Wrong Choice of Project
Risk Factors
SC: Team lack of Motivation
h
t
s
CA: No Standardization of PBL Procedure 1 1 7 5 7 3 3 5 3 0,26

CB: PBL specific requirements not defined accurately 1 1 3 5 7 3 3 7 3 0,23
CC: Wrong Choice of Project 1/7 1/3 1 1 3 5 3 3 3 0,13
SA: Team Building practices not used 1/5 1/5 1 1 1 1 1/3 3 3 0,07
SB: PBL Professor not active in the project 1/3 1/3 1/5 1 3 1 1 3 3 0,08
SC: Team lack of Motivation 1/7 1/7 1/3 1 1/3 1 1 3 1/3 0,04
TA: PBL Professor not prepared for the project 1/3 1/3 1/3 3 1 1 1 5 3 0,10
TB: No definition of PBL records organization 1/5 1/7 1/3 1/3 1/3 1/3 1/5 1 1/3 0,03
TC: Students not prepared for the PBL 1/3 1/3 1/3 1/3 1/3 3 1/3 3 1 0,06
TOTAL 3,69 3,82 13,53 17,67 15,00 26,33 12,87 33,00 19,67
The impact scores shown in Table 3 were color-coded using Table 4.
271
Table 4. AHP
Impact Level Score

Score Impact Level Impact
5 High More than 0,16
4 Elevated 0,12-0,16
3 Moderated 0,08-0,11
2 Low 0,04-0,07
1 Limited Less than 0,04
The final risk index for each risk factor was obtained in the following way. As an example, considering Table 6
(applicable to Theory and Practice Learning Failure), the probability for T1 is 0.7, and the impact is 0.05; based
on Table 5, the probability rating score is (value: 4). The impact rating score is (value: 2) for the risk factor T1.
Figure 2 is referenced to obtain the final risk index for T1. In this case, the final score is (value: 8), the product
of 4 by 2. The risk index for the other Principles is calculated similarly.
Table 5. Probability and Impact Score.
The global risk score (index) was determined using the risk scoring matrix shown in Figure 2 (Hyun et al., 2015).
The final risk index is obtained by combining the probability and impact.
Figure 2. Final Risk Index
The global risk score (index) for Cognitive Learning Failure, Social Learning Failure was determined using the
same process.
Table 6. Probability and Impact Score Theory and Practice Learning Failure
272
The authors weighted the probability values obtained with the survey based on the respondents' experience,
as shown in Table 7. The probabilities were then loaded into BBN software, and the probabilities of each risk
were combined, allowing sensitivity analysis to define the most significant risk factor.
Table 7. Experience Period (E) & Weight
Experience Period (E) &

Weight
Less than 5 years 0.60
6-10 years 0.75
11-15 years 0.09
More than 15 years 1.00
The probabilities were loaded into BBN software. BBN was used because the statistical causal structure was key
to obtaining information about risk events to mitigate them. Charts in Figures 3 and 4 could be generated
using the software.
Figure 3. BBN combines all risk factors in the software
Figure 4. Tornado Chart
273
The Tornado chart originated from BBN (Figure 4) shows that the risks with the highest probabilities are shows
that the highest risk Index is C4 (Poor explanation of expectations to students), C5 (Lack of background
definition on principles behind projects), and C6 (No clear definition of requirements). The actions to be taken
for risk factor C4 are ensuring the procedure clarifies expectations to the Professor and students. The procedure
needs to be detailed enough to ensure repeatability and reproducibility. The action for risk factor C5 is to
ensure the procedure also covers the principles behind the projects. The detailed explanation of the objectives
and targets of each project is crucial to keep everybody in the same page. The action for risk factor C6 is to
ensure the procedure covers the definition of project requirements. If the project requirements are not defined,
it is difficult to monitor and control the performance.
5 Discussion of Results and Conclusion

As proposed in the introduction, the study demonstrated that AHP in conjunction with BBN could be used to
assess and prioritize the risk factors of PBL failure in future studies. An in-depth analysis of current literature
about the subject allowed the identification of risk factors in this process. The preparation of a global risk matrix
is proposed since it provides critical information considering identifying and prioritizing the potential risks that
could cause PBL failure. It is an effective decision-making process for universities. This study shows evidence
that PBL, in general, is subjected to a great variety of risks, some of them capable of compromising the teaching
quality in the universities. Probabilistic risk analysis plays a significant role in understanding and implementing
risk responses to avoid failure. An in-depth search for previous work dealing with risks in PBL was conducted
and is described herein, and a model is proposed for risk assessment. This study is significant because
understanding the significant risks in the PBL process can influence the decision of professors and university
engineering school coordinators. Applying this innovative risk assessment method in the PBL process fills a
gap in the literature since no previous work dealt with this specific subject. The contribution is significant since
risk assessment in the PBL process permits decision-makers to assign funds for critical activities that can impact
universities' teaching quality. As initially proposed, a model for risk assessment of PBL failure is being proposed
that allows the combined application of AHP and Bayesian Networks to prioritize risks. The proposed
application sought to analyze and define the primary risk factors and critical events that could lead to a failure
in PBL.
In response to the first question, "1 - What are the most significant risk factors for using PBL in collaboration
with local companies when teaching engineering students?" The most significant risks were identified. The
Tornado chart originated from BBN (Figure 4) shows that the risks with the highest risk Index is C4 (Poor
explanation of expectations to students), C5 (Lack of background definition on principles behind projects), and
C6 (No clear definition of requirements). The result of AHP for the Cognitive Learning principle shows that the
most impactful risk factors are C1 (Lack of procedure for the PBL process) and C3 (Lack of standard work for
the execution of PBLs). The AHP for the Social Learning principle shows that the most impactful risk factors are
S3 (Assign students to teams rather than let them select the team themselves) and S6 (Students not encouraged
by professors). The AHP for the Technical Learning principle shows that the most impactful risk factors are T3
(Lack of professor technical content knowledge and experience) and T4 (lack of industrial skills). The AHP for
risk factors categories shows that the most impactful risk factors are CA (No Standardization of PBL Procedure)
and CB PBL specific requirements not defined accurately).
In response to the second question, " What are the probability and the impact of these risks and the global risk
scores? " The probability and impact (global risk scores) for all the risks are provided. The Probability and
Impact Score for Cognitive Learning Failure shows that the highest risk Index is C1 (Lack of procedure for PBL
process), and C2 Students and professors are not appropriately trained on the PBL procedure). The Probability
and Impact Score for the Social Learning Failure shows that the highest risk Index is S3 (Assign students to
teams rather than let them select the team themselves), S6 (Students not encouraged by professors), and S9
(Relationship professor and student not good). The Probability and Impact Score Theory and Practice Learning
Failure shows that the highest risk Index is T4 (Professor has no industrial skills) and T3 (Lack of professor
technical content knowledge and experience).
274
In response to the third question, "3 What are the responses to high Index risks, and how to combine the risks
and conduct a sensitivity analysis with BBN" The responses to high index risks are the following: The actions to
be taken for risk factor C1 are the issue of a procedure to document the PBL process. Risk factor C2 is to train
professors on the documented procedure. The actions to be taken for risk factor S3 are not assigning the
students to the PBL group but letting the student choose the group he wants to work with and the action for
the risk factor. S6 trains professors to encourage students to work on the PBL, and S9 provides training on
team building to professors and students. The actions to be taken for risk factor T3 are ensuring the Professor
assigned to a project has the required technical knowledge and experience. The action to risk factor T4 is to
ensure the Professor has practical experience in the industry where the PBL project will be developed. A
sensitivity analysis with BBN. The Tornado chart shows that the risks with the highest probabilities are shows
that the highest risk Index is C4 (Poor explanation of expectations to students), C5 (Lack of background
definition on principles behind projects), and C6 (No clear definition of requirements). The actions to be taken
for risk factor C4 are ensuring the procedure clarifies expectations to the Professor and students. The action
for risk factor C5 is to ensure the procedure also covers the principles behind the projects. The action for risk
factor C6 is to make sure the procedure covers the definition of project requirements.
The target of the study was to identify the critical risk factors that could affect PBL and propose an optimized
process that has quality in PBLs. The implications are relevant since changes in the PBL process can improve
the result substantially. By following the revised process, PBL failures can be prevented. The proposed
Methodology revealed some critical results, thus contributing to previous studies on the subject and may help
overcome some of the challenges of professors, students, and other professionals looking for quality
education. The study was conducted based on the experience and knowledge of professors and students. It is
believed that the present study will augment the knowledge of professors, engineering students, and
engineering school coordinators and help in the application process. As explained in the Introduction Section,
several papers have been published addressing PBL use in different domains in the latest years. However, no
previous study could be found covering the application of AHP and BBN to identify risks of the application of
PBL. It is noteworthy that this paper proposes an optimized approach that could be used in any university or
teaching organization.
This proposed process can guide the universities under the traditional teaching process to achieve quality
improvement by following the proposed Methodology. That helps to impact results, representing considerable
teaching gains significantly. The proposed method, enhanced by improved communication, enables any
university to increase efficiency in education. This study shows evidence that the quality of the PBL process is
affected by several factors, some of which can compromise the reliability of the teaching institution. In this
regard, process analysis played a crucial role in understanding and implementing actions to improve it.
This study is significant because understanding the most impactful risk factors in the conduction of PBL can
influence professors, students, and decision-makers in universities. As evidenced by the results, the modified
method can help to optimize engineering teaching. As expected, the contribution is significant; it is believed
that the present study will augment the knowledge of professors and students concerning the use of the PBL
process to improve the quality and effectiveness of the teaching. Scope for future research: This study opened
some new research avenues for the future. Opportunities for other case studies are abundant. They could be
related to a broader application of risk analysis of PBL in specific cases, enhancing the current Methodology in
use and reducing the risk of failures.
6 References
Bani-Hani, E., A.Al Shalabi, F.Alkhatib, A.Eilaghi, and A.Sedaghat . 2018. "Factors Affecting the Team Formation and Work in Project-Based
Learning (PBL) for Multidisciplinary Engineering Subjects." Journal of Problem-Based Learning in Higher Education 6 (2): 136–143.
Beddoes, K. D. , B. K.Jesiek, and M.Borrego . 2010. "Identifying Opportunities for Collaborations in International Engineering Education
Research on Problem- and Project-Based Learning." Interdisciplinary Journal of Problem-Based Learning 4 (2): 7–34. doi:
10.7771/1541-5015.1142
Bielefeldt, A., Paterson, K., & Swan, C. (2009, February). The State of Project-Based Service Learn ing in Engineering Education. Report of
the National Summit on Measuring the Impacts of Project-Based Service Learning on Engineering Education, Washington, DC.
275
Chipman, H. and Hamada, M. (2016) 'Using Bayesian variable selection to analyze regular resolution IV two-level fractional factorial
designs', Quality and Reliability Engineering International, Vol. 33, No. 3, pp.493–502.
Daun, M., Salmon, A., Weyer, T., Pohl, K., and Tenbergen, B. (2016). Project-based learning with examples from industry in university courses:
An experience report from an undergraduate requirement engineering course. In 2016 IEEE 29th International Conference on
Software Engineering Education and Training (CSEE&T), pages 184– 193.x
Dong, Q. and Saaty, T.L. (2014) 'An analytic hierarchy process model of group consensus', J. Syst. Sci. Syst. Eng., Vol. 23, No. 3, pp.362–374
Droguett, E.L., Groen, F. and Mosleh, A. (2004) 'The combined use of data and expert estimates in population variability analysis', Reliability
Engineering and System Safety, Vol. 83, No. 3, pp.311–321.
Du, X. , L.Su, and J.Liu . 2013. "Developing Sustainability Curricula Using the PBL Method in a Chinese Context." Journal of Cleaner
Production 61: 80–88.
Figueiredo, M. A. D., Ignacio, A. A. V., Guimarães, M. S. The initial model of the application of PBL in Industrial Engineering. PAEE
García-Martín, J. , and J. E.Pérez-Martínez . 2017. "Method to Guide the Design of Project-Based Learning Activities Based on Educational
Theories." International Journal of Engineering Education 33 (3): 984–999.
Graaff, E., Kolmos, A., 2003. Characteristics of problem-based learning. International Journal of Engineering Education 19 (5), 657e662.
Jucker, R., 2002. "Sustainability? never heard of it!" e some basics we shouldn't ignore when engaging in education for
sustainability. International Journal of Sustainability in Higher Education 3 (1), 8e18.
Groth, K.M. and Swiler, LP (2013) 'Bridging the gap between HRA research and HRA practice: a Bayesian network version of SPAR-H',
Reliability Engineering and System Safety, July, Vol. 115, pp.33–42
Hansen, D., Cavers, W., & George, G. H. (2003). Use of a Physical Linear Cascade to Teach Systems Modelling. International Journal of
Engineering Education, 19(5), 682-695.
Henderson, C., Dancy, M. and Niewiadomska-Bugaj, M. (2012), use of research based instructional strategies in introductory physics: Where
do faculty leave the innovation-decision process? Physical Review Special Topics-Physics Education Research, 8(2), 1-15.
Hyun, k. C, Min, S.,Choi, H., Park, J. Lee, In-Mo. Risk analysis using fault tree analysis (FTA) and Analytic Hierarchy Process (AHP) applicale
to shield TBM tunnels. Tunnelling and Underground Space Technology., 49, 121-129.
Inelmen, E. (2003). Challenging the Administration to Implement Problem-Based Learning in
Kjellberg, K. , T.Adawi, and K.Brolin . 2015. "Challenges in Implementing PBL: Chalmers Formula Student as a Case." In Proceedings of the
43rd Annual SEFI.
Mahadevan, S., Zhang, R. and Smith, N. (2001) 'Bayesian networks for system reliability reassessment', Structural Safety, Vol. 23, No. 3,
pp.231–251.
Marques, M., Ochoa, S. F., Bastarrica, M. C., and Gutierrez, F. J. (2018). Enhancing the student learning experience in software engineering
project courses. IEEE Transactions on Education, 61(1):63–73. ISSN 0018-9359
Mls, K. and Otcenaskova, T. (2013) 'Analysis of complex decisional situations in companies with the support of AHP extension of Vroom-
Yetton contingency model', IFAC Conference on Manufacturing Modelling, Management and Control, June, Vol. 1, pp.549–554.
Moliner, L. , L.Cabedo, M.Royo, J.Gámez-Pérez, P.Lopez-Crespo, M.Segarra, and T.Guraya . 2019. "On the Perceptions of Students and
Professors in the Implementation of an Inter-University Engineering PBL Experience." European Journal of Engineering Education
44 (5): 726–744.
Mosleh, A. (1992) 'Bayesian modeling of expert-to-expert variability and dependence in estimating rare event frequencies', Reliability
Engineering and System Safety, Vol. 38, Nos. 1–2, pp.47–57.
Mosleh, A. and Apostolakis, G. (1986) 'The assessment of probability distributions from expert opinions with an application to seismic
fragility curves', Risk Analysis, Vol. 6, No. 4, pp.447–461.
Palmer, S. , and W.Hall . 2011. "An Evaluation of a Project-Based Learning Initiative in Engineering Education." European Journal of
Engineering Education 36 (4): 357–365.
Podofillini L. and Dang V.N. (2013) 'A Bayesian approach to treat expert-elicited probabilities in human reliability analysis model
construction', Reliability Engineering and System Safety, September, Vol. 117, pp.52–64.
Rechenthin, D. (2004) 'Project safety as a sustainable competitive advantage', Journal of Safety Research, Vol. 35, No. 3, pp.297–308.
Requies, J. M. , I.Agirre, V. L.Barrio, and M.Graells . 2018. "Evolution of Project-Based Learning in Small Groups in Environmental Engineering
Courses." Journal of Technology and Science Education 8 (1): 45–62.
Saaty, T.L. (1980) The Analytic Hierarchy Process, McGraw Hill, New York.
Schneider, B., Krajcik, J., Lavonen, J., Geller, M. J., & Salmela-Aro, K. (2020). Learning science – The value of crafting engagement in science
environments. Yale University Press
Setiawan, A. W. 2019. "Implementation of Project-Based Learning in Biomedical Engineering Course in ITB: Opportunities and Challenges."
Proceedings of IFMBE Proceedings 68/1: 847–850.
Strobel, J., & van Barneveld, A. (2009). When is PBL More Effective? A Meta-synthesis of Meta Analyses Comparing PBL to Conventional
Classrooms. The Interdisciplinary Journal of Problem-bsed Learning, 3(1), 44-58.
Thevathayan, C. (2018). Evolving project-based learning to suit diverse student cohorts. In Proceedings of the 22nd International
Conference on Evaluation and Assessment in Software Engineering 2018, pages 133--138. ACM.
Thomas, J. W. (2000). A Review of Research on Project-Based Learning. Retrieved June 7, 2009, from
www.bobpearlman.org/BestPractices/PBL_Research.pdf
Xiangyun D., Liya S., Jingling L. (2013). Developing sustainability curricula using the PBL method in a Chinese context. Journal of Cleaner
Production. Volume 61, 15 December 2013, Pages 80-88.
Yang, X., Haugen, S. and Li, Y. (2017) 'Risk influence frameworks for activity-related risk analysis during operation: a literature review', Safety
Science, Vol. 96, pp.102–116.
276
Engaging with real-world phenomena through Matlab
programming projects
M. Teresa T. Monteiro1, Gabriel Hornink2, Flávia Vieira3

1
2
Educational Media Laboratory, Department of Biochemistry, Institute of Biomedical Sciences, Federal University of Alfenas, Brazil
3
Centro de Investigação em Educação (CIEd), Institute of Education, University of Minho, Braga, Portugal
Abstract
This presentation aims to report a pedagogical experience carried out in a course unit of a graduate programme in
Computer Engineering at the University of Minho. The course unit, Numerical Methods and NonLinear Optimization
(NMNO), integrates the first semester of the third year. The experience took place in 2021-2022 with 146 students, and it
was supported by Centre IDEA-UMinho within the project 2Be-Learning. The classes were taught face-to-face (theoretical
lectures and lab practice) and several strategies were implemented to support learning: ARS, padlet, videos, storytelling,
and projects. Assessment was diversified and distributed over time to foster ongoing study and progress. It included two
face-to-face written tests, four online multiple choice mini-tests (one per month, lasting about 10 minutes, based on
extensive question banks), and one Matlab project. The focus of the presentation is on the impact of Matlab projects in the
learning process. The projects were carried out by teams of 4 students. Each team could choose one of five proposed topics.
The main challenge was to search for and select a real-world phenomenon where the chosen topic could be applied and
solve a problem that should have an adequate level of complexity. The experience was evaluated on the basis of the quality
of projects, students’ grades and their perceptions collected in a survey at the end of the course unit. Results show that
students developed their creativity through building bridges with other scientific areas and solving problems in innovative
ways. Projects promoted their involvement in learning, autonomy, cooperation and the personal construction of knowledge,
which are essential competences for lifelong learning. Overall, it can be considered that engaging with real-world
phenomena creates conditions for students to connect course-based learning with authentic situations, analyse and solve
problems from a multidisciplinary perspective, mediated by digital technologies, and become pro-active learners.
Keywords: real-world phenomena, projects, problem-solving, maths education.
1 Introduction
The Centre IDEA-UMinho (Centre for the Innovation and Development of Teaching and Learning -
https://idea.uminho.pt/pt) is an academic development centre at the University of Minho that supports the
development of innovative pedagogical projects where faculty assume the role of teacher-researchers,
exploring and assessing their teaching experiences with a focus on student learning. These projects represent
cases of the Scholarship of Teaching and Learning (SoTL), whereby teachers inquire into their practice and
make teaching “community property” by sharing and disseminating results so that others can build on their
work (Shulman 2004). By inquiring into teaching, teachers become agents of change and reshape their
professional identity, which supports the development of culture of innovation that assigns teaching a more
visible role in higher education institutions (Vieira, 2014).
The first author started her journey of pedagogical inquiry in 2020-21, during the COVID-19 pandemics, by
exploring a b-learning approach in a course unit of a master’s programme in Computer Engineering at the
University of Minho, Numerical Methods and Nonlinear Optimization, which is placed in the first semester of
the third year of the programme (Monteiro et al., 2021). Her project was supported by the Centre and the
second and third authors acted as mentors. The goal was to enhance students’ motivation, interaction and
participation in learning. Along with the exploration of digital resources, assessment became more diversified
and distributed over time to foster ongoing study and progress. It included mini-tests and two Matlab projects
carried out in teams with the main challenge of finding a real-world phenomenon for the application of a
course concept, which implied connecting conceptual learning with reality and creating bridges with other
areas of knowledge. The experience was evaluated on the basis of students’ assessment results and their
277
perceptions collected in a survey. The new approach resulted in high levels of student engagement and
satisfaction, promoting cooperation and the personal construction of knowledge, which are essential
competences for lifelong learning. Nevertheless, is was concluded that the development of Matlab projects
required further improvements, not only as regards support to students but also the evaluation of their impact
on learning. Therefore, the project was continued in 2021-22 with a new group of students, and the purpose
of the present paper is to report the improvements made and the impact of Matlab projects on learning. This
project was also supported by the Centre and accompanied by the same colleagues as mentors.
The paper is organized as follows: section 2 describes the context of the teaching experience and the
procedures for developing the Matlab projects; section 3 presents information of those projects, students’
assessment results and their perceptions regarding project development and some statistical data; section 4
presents conclusions and future directions.
2 The development of Matlab projects

The experience took place with 146 students in the first semester of the academic year 2021-2022. The course
unit has 5 ECTS (European Credit Transfer System) with 140 working hours and consists of two modules:
Numerical Methods and Nonlinear Optimization. The learning outcomes are the following: applying computer
tools to model physical problems; developing and applying numerical skills to analyse systems; comparing
different solutions for numerical problems; selecting, using and comparing different optimization algorithms;
developing critical evaluation of results; and using computational tools.
The course unit is taught 4 h a week: 2 h of theoretical lessons and 2 h of practical lessons, all face-to-face. Five
practical lessons were carried out in computational laboratories with about 30 students per group, involving
four teachers. In these classes, the students solve exercises using the calculator and the software Matlab. The
Blackboard eLearning platform (Bb) was used to support the learning process.
One of the learning tasks to assess students on the Matlab programming component is a Matlab project. This
project entails linking conceptual learning with reality and is aimed at promoting problem-solving abilities,
interdisciplinary learning and cooperation, which are important elements of active learning (Graaff & Kolmos,
2003; Prince, 2004). Connecting the curriculum with real world problems enhances authentic learning and deep
understanding (McGregor, 2020), and cooperative learning is especially effective in problem-solving tasks that
involve dealing with multiple sources of knowledge and require students to negotiate ideas and decisions
(Johnson & Johnson 2014).
The projects were carried out outside the classroom in teams of four elements, formed by the students. Five
topics were proposed by the teacher and each team should select one according to their preference.
Project development involved several phases, requiring students to negotiate decisions, do research work,
solve problems by applying mathematical knowledge creatively, analyse results and report their work:
• Project guidelines placed on the Bb platform
• Topic selection (one of the five indicated by the teacher)
• Searching for a real-world phenomenon, using several sources
• Mathematical modelling (complexity analysis and adequacy)
• Matlab programming using the corresponding routine
• Numerical experiments to obtain results
• Critical analysis of results
• Report writing (three pages)
• Electronic submission with deadline on the Bb platform
The main challenge of the project, which students usually find to be rather difficult, is finding a real-world
phenomenon for the application of a course concept. This was observed in the first experience in 2020/21,
where students had to develop two projects, which proved to be time-consuming and too demanding for
many of them. This time, they only had to do one project, and before they began, the teacher explained the
task requirements as usual, but she also showed them examples of well designed projects developed by former
278
students, increasing student’s awareness regarding the quality criteria their projects were expected to meet,
and giving them a more concrete sense of the type of work they were expected to do. During project
development, she held regular support sessions as usual, both online and face-to-face, to help students
monitor their progress and overcome difficulties. These procedures were useful to develop students’
assessment literacy and self-regulation skills (Evans, 2003; 2021). In the end, each team presented a 3-page
report where they synthesized their project.
3 The nature and impact of Matlab projects

A total of 33 project reports were concluded. In this section, we present the topics and the phenomena they
explored, as well as assessment results and the students’ perceptions about the competences they developed.
3.1 Topics and real-world phenomena explored

The five topics teams could choose from are presented in Table 1, as well as the corresponding Matlab routines,
the week when the topics were proposed, which depended on when they were explored in class, and the
number of projects per topic. The first two topics were chosen by a large number of students (79 %). They
found them interesting and it was easier to find real-word phenomena related to them; moreover, the fact they
were proposed earlier in the semester also gave students more time to explore them. The topic Numerical
integration was not chosen, perhaps because it is more difficult to find a real-world phenomenon; also, the fact
that it is very easy to use the corresponding MatLab routine makes it more difficult to be creative and obtain
a good grade, since creativity is one of the assessment criteria.
Table 1. Topics and Matlab routines
Topic Matlab routine Week (1-15) Projects
Nonlinear equations fsolve 4 16
Splines spline 5 10
Numerical integration trapz, quad 6 0
Least squares approximation polyfit, lsqcurvefit 7 4
Nonlinear optimization fminunc, fminsearch 12 3
As pointed out above, the main challenge of the project was to find a suitable real-world phenomenon where
the chosen topic could be explored. Despite the strategies used to support the students, this difficulty was
mentioned in all the reports and the students complained about it during project development, asking for the
teacher’s help. Some ideas were suggested to them by referring to books, articles, magazines, other course
units, databases, etc. The identification of a taught concept (topic) in the in the real-world is a mechanism to
facilitate conceptual understanding, although it may be quite demanding.
Table 2 shows the variety of real-world phenomena explored for each topic and the scientific areas with which
bridges were established. It illustrates the potential value of projects in promoting the connection between
theoretical learning and reality, as well cross-disciplinary learning,
Table 2. Real-world phenomena and scientific areas in Matlab projects
Topics Phenomena Scientific area
Nonlinear Carbon 14 dating Chemistry

equations
Global Positioning System (GPS) using satellites signals (2) Spatial engineering
Temperature distribution in a wire Materials engineering
Tank sizing Civil engineering
American soccer ball throw Physics
279
Topics Phenomena Scientific area
Electric circuits (2) Electrical engineering
Concentration of a polluting bacteria in a lake Environment
Location of a bridge over a river Civil engineering
Orbit of celestial bodies Astronomy
Electrical power system planning Electrical engineering
Efficient square root calculation Mathematical

computational
Splines Objects design Computer graphics
Pathfinding (2) Video games
Cryptocurrency price Economy
Apparent visual magnitude of a star Astronomy
Portuguese students in higher education Sociology
Parachuting jump Physics
Electrocardiographic heart rate monitoring Electronics
Graphic contour of the heart left ventricle Biomedicine
Cod fishing in northeast Atlantic ocean (1960-2019) Business
Least- Average temperature in Lisbon since 1960 Environment

squares
Portuguese population 1960-2020 Sociology
Microchip and processor industry (Moore's law) Electronic engineering
Soccer players running speed Sports
Nonlinear Evaluation of optimization algorithms Informatics

optimization
Iron barrel sales Business
Truck and sedan resale prices Retail
3.2 Assessment results

In each team, all members had the same mark. This was assumed in the beginning of the semester, but it was
also pointed out by a few students as a negative factor in the final assessment, because it did not take account
of individual differences among team members regarding their investment and contribution to projects.
The assessment criteria were explained to the students. A scale from 0 to 3 points was used (the course unit
has a scale from 0 to 20, the project is worth 15 % of the final grade). The quality of projects was assessed
taking into account their originality, complexity, creativity and the computing experiences. Reports were
assessed in terms of rigour, ability to synthesize, organization, presentation and the final conclusions. Grading
projects is not an easy task and has to be done in comparative terms, i.e., each project is compared with the
others, which makes it an iterative, time-consuming process.
The distribution of grades is presented in Figure 1 – 9 teams (27 %) obtained the maximum grade (3). The lower
grading in 8 projects (below 1,5) is related to factors such as inappropriateness of the topic to the phenomenon,
low complexity, or low commitment to report writing. Overall, students’ results were positive and demonstrate
that the learning outcomes for this task were globally achieved.
280
Assessment
10
8
# groups
6
4
2
0
0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 3
Grades
Figure 1. Project grades
The students’ final grades showed an approximately normal distribution (Fig. 2), but project grades were not
normally distributed (Fig. 3). We used a non-parametric test to look for correlations between the two sets of
data. We calculated the Spearman correlation coefficient using ranked project grades and ranked final grades.
The result was 0.52. This means that there is a moderate monotonic relationship between project grades and
final grades. While both grades tend to go up in relation to one another, this relationship is not very strong
because there is a significant number of cases where this is not true.
Figure 2. Final grades histogram
Figure 3. Project grades histogram
281
However, when we consider the project grades average for each range of final grades: [18,20], [14,17], [10,13],
<10, we find out that final grades and project grades are strongly related. An increasing monotonic relationship
between the two sets of values is shown in the Table 3. In other words, if a final grade x is higher than a final
grade y, then the project grades average Ax corresponding to the range to which x is belongs is never lower
than the project grades average Ay corresponding to the range to which y belongs.
Table 3. Project grades average per final grades range
Project grades average ([0,3]) per final grades range
Final grades range [18,20] [14,17] [10,13] <10
Project grades average 2.769 1.877 1.695 1.583
A relationship was found in the results concerning the project topics (cf. Table 1). On average, Splines projects
were much better than any other projects. The difference between the average grade for Splines projects, 2.82,
and the average grade for Nonlinear optimization projects, 1.75 (the second best group of projects), is
approximately 1. On the other hand, the average grade of Nonlinear optimization projects differs only 0.08 and
0.3, respectively, from the average grades of Nonlinear equations projects and Least-squares projects. This may
be due to the fact that Splines are important and useful concepts for the Computer Graphics course and their
application on various areas is easily understood by students.
Another interesting relationship concerns project grades average for a given topic and final grades average of
the students that chose that topic. There is an increasing monotonic relationship between the two sets of
values, as shown in Table 4. In other words, if project grades average for topic x is higher than project grades
average for a topic y, then - on average - students who chose topic x got higher final grades than students
who chose topic y.
Table 4. Grades per topic
Project Grades Average Final Grades Average per

Topic
per topic topic
Nonlinear equations 1.673 13
Splines 2.824 16
Numerical integration - -
Least squares 1.45 12
approximation
Nonlinear optimization 1.75 14
3.3 Student’s perceptions

In the end of the semester, in order to identify the impact of the teaching approach and to collect some
information and suggestions for improvement, a short Google forms survey with some questions was
implemented. 60 students answered the survey (around 40 %). The survey included two questions on the
impact of projects, which were not asked to the students from the previous experience.
One of the questions was related with the usefulness of the project in learning. Students were asked to indicate
how useful the project was for their learning in the course unit, using the following scale: ‘a lot’, ‘a little’, ‘not
at all’. 38 % answered ‘a lot’, 47 % ‘a little’ and 15 % ‘not at all’. These are not very positive findings. However,
the findings from another question regarding cross-disciplinary competences developed through projects
suggest otherwise. That question presented a set of six competences related to project development and
students were asked to indicate whether they developed those competences by doing the project, using the
same scale: ‘a lot’, ‘a little’, ‘not at all’. The results are presented in Table 5.
282
Table 5. Competences developed through projects: students’ perceptions (n=60)
Competences A lot A little Not at all
% % %
Autonomy 51.7 41.7 6.7
Cooperation 55 41.7 3.3
Creativity 63.3 33.3 3.3
Building bridges with other scientific areas 53.3 40 6.7
Personal construction of knowledge 58.3 35 6.7
Solving problems 63.3 33.3 3.3
Around half of the students answered ‘a lot’ on all competences, and almost all students felt they developed
them to a higher or lower degree (‘a lot’ or ‘a little’), which is quite expected given the variability and
idiosyncrasy of competence development. The number of students who answered ‘not at all’ is minimal. These
findings appear to suggest that students acknowledged the role of these projects in developing cross-
disciplinary competences that are crucial for lifelong learning, namely creativity, personal knowledge
construction and problem-solving. A majority of students also considered to have developed autonomy,
cooperation and multidisciplinary learning through projects.
It is possible that, in assessing the usefulness of projects for their learning in the course unit, the students were
thinking more about disciplinary competences, namely those needed for exams and tests, which may explain
the less positive findings on that question. If this is so, then we need to ask whether the students really valued
cross-disciplinary competences in their learning, even though they realised to have developed them. Perhaps
the fact that these projects were not rated high for their overall grades (3 points in 20) conveyed a conflicting
message about their value for learning. These students also took two tests and 4 short multiple choice tests,
which accounted for most of their final grade. One needs perhaps to reconsider the relative weights of
assessment tasks and whether they are aligned with the value assigned to the various competences those tasks
are expected to promote
Let [0,2] be the range of values that represent the degree of students’ perceptions of competence development
in projects, where 0 stands for “no development”, 1 stands for “a little development” and 2 stands for “a lot of
development”. In this range, a competence is considered relevant if its value is no less than 1.5. Table 6
describes how the students of four different grade ranges ([18,20], [14,17], [10,13], <10) evaluated the
development of the several competences through projects.
Table 6. Average development of different competences per final grades range, where development ∈ [0,2]
Degree of involvement ([0,2]) of each

competence per final grades range
Competence [18,20] [14,17] [10,13] <10
Autonomy 1.563 1.536 1.222 1
Cooperation 1.563 1.607 1.333 1.167
Creativity 1.688 1.714 1.444 1
Building bridges with other scientific areas 1.563 1.464 1.556 1
Personal construction of knowledge 1.75 1.464 1.444 1.167
Solving problems 1.75 1.571 1.667 1.167
283
For students in the highest range of grades, all competences were solidly developed through projects.
Problem-solving and personal knowledge construction scored rather higher than others. For students in the
[14,17] range, all competences were developed, with creativity scoring rather higher than others. For students
in the lowest range of grades, only two competences were solidly developed: multidisciplinary learning and
problem-solving. Creativity and personal knowledge construction were fairly developed, but autonomy and
cooperation less so. These findings indicate differentiated perceptions of learning in the three student groups,
suggesting that those who had higher grades also perceived a higher degree of competence development
through projects
Let students’ perceptions of motivation be represented in the scale [0,4], where 0 stands for “not motivated at
all”, and 4 corresponds to maximum motivation. As it would be expected, perceptions of motivation and final
grades are strongly correlated. As motivation increases, final grades increase as well (Table 7).
Table 7. Average motivation per final grades range, where motivation ∈ [0,4]
Degree of motivation ([0,4]) per final grades

range
Final grades range [18,20] [14,17] [10,13] <10
Motivation 3.25 2.893 2 1.333
4 Conclusion
The findings showed that project development enacted some of the conditions for creating ‘situated learning
environments’ in higher education as pointed out by Herrington (2006):
• an authentic context that reflects the way knowledge will be used in real life: by identifying real-world
phenomena and applying acquired knowledge to solving problems, projects enhanced students’ ability
to use that knowledge in an authentic context, and also to undertake interdisciplinary learning;
• authentic activities which have real-world relevance, and which present a single complex task to be
completed over a sustained period of time: project development integrated a set of tasks that can be
transferred to other real-life situations (negotiating decisions, team-building, solving problems, etc.);
• collaborative construction of knowledge, along with coaching and scaffolding by peers and teachers:
in developing their projects in teams and with regular teacher support, opportunities for cooperative
learning were created, which enhanced students’ confidence and motivation;
• authentic assessment by integrating learning and assessment into the same task: projects were
learning tasks and also learning products to be assessed, which means that the focus was not only on
assessing outcomes but also, and most importantly, on creating conditions for the development of
disciplinary and cross-disciplinary competences.
The changes made in students’ preparation for projects were useful, yet they still had difficulties in relating
course contents with real-life phenomena at the initial state of their projects. Further support might be needed
at that stage and during project development, for example through joint seminars on theory-practice
integration with realistic examples. Nevertheless, it is also our conviction that students’ struggle to make
connections between concepts and real life is a necessary part of ‘learning beyond the classroom’.
Assessment results and students’ perceptions of learning through projects were globally positive. However,
more thought is needed on the importance they give to cross-disciplinary competences, and on the role that
assessment might play in valuing those competences in grading systems. These concerns may be the starting
point for further developments in the teaching approach, which may require the involvement of students in
discussing these issues among themselves and with the teacher during the development of projects.
By supporting the development of teaching projects, the Centre IDEA-UMinho enhances collaborative forms
of SoTL where mentors act as critical friends. This process creates conditions for the professional development
of both the teacher and mentors, who in this case belong to different disciplinary fields. It also favours the
enactment of two basic principles of good SoTL practice pointed out by Felten (2013): inquiry focused on
284
student learning and conducted in partnership with students. The process of sharing experiences and
interpretations with others increases our attention to learning and our ability to develop dialogic pedagogies.
Acknowledgments
This work has been supported the Centre IDEA-UMinho and by FCT – Fundação para a Ciência e Tecnologia
within the R&D Units Project Scope: UIDB/00319/2020. The authors are grateful to the reviewers for their
valuable comments.
5 References
Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 82(1), 70-120. .
Evans, C. (2021). The EAT Framework. ERASMUS EAT (©Evans, 2021) Version 4 of EAT. htps://www.researchgate.net/publication/354822146
Felten, P. (2013). Principles of good practice in SoTL. Teaching & Learning Inquiry, 1(1), 121–125.
https://journalhosting.ucalgary.ca/index.php/TLI/article/view/57376/43149
Graaff, E. D., & Kolmos, A. (2003). Characteristics of problem–based learning. International Journal of Engineering Education, 19(5), 657-
662. http://www.ijee.ie/articles/Vol19-5/IJEE1450.pdf
Herrington, J. (2006). Authentic e-learning in higher education: Design principles for authentic learning environments and tasks. World
Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Chesapeake,
Va.https://www.researchgate.net/publication/30388215_Authentic_e-
learning_in_higher_education_Design_principles_for_authentic_learning_environments_and_tasks
Johnson, D. W. & Johnson, R. T. (2014). Cooperative Learning in 21st Century. Anales de Psicología.
https://doi.org/10.6018/analesps.30.3.201241
McGregor, S. L. T. (2020). Emerging from the deep: Complexity, emergent pedagogy and deep learning. Northeast Journal of Complex
Systems (NEJCS), 2(1). Article 2. https://orb.binghamton.edu/nejcs/vol2/iss1/2
Monteiro, M. T., Hornink, G. & Vieira, F. (2021). Innovating to improve – An experience in a computer engineering programme. 13th
International Symposium on Project Approaches in Engineering Education and 18th Active Learning in Engineering Education
Workshop, PAEE/ALE 2021, 11: 192-199. https://doi.org/10.5281/zenodo.5095687
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223-231.
https://doi.org/10.1002/j.2168-9830.2004.tb00809.x
Shulman, L. (2004). Teaching as community property – Essays on higher education. Coletânea ed. por P. Hutchings. San Francisco: Jossey-
Bass.
Vieira, F. (2014). Para uma mudança profunda da qualidade da pedagogia na universidade. Revista de Docencia Universitaria (REDU), 12(2),
23-39. https://doi.org/10.4995/redu.2014.5638
285
Circular Economy Implemented in a Cosmetic Company: a Case
Study
Nicole Andressa Fuoco Michel1, Luiz Carlos de Campos1
1
Faculty of Exact Sciences and Technology, Pontifical Catholic University of Sao Paulo, Sao Paulo, Brazil
Abstract
This research was part of a Scientific Initiation Project developed during last year in the Industrial Engineering Program as
complement of the undergraduate contents. The purpose of this work was to encourage the adoption of the concept of
Circular Economy by the companies regardless of your size in order to reduce environmental impact and help to avoid the
malefic effects produced by the tailings discarded by the majority of the companies. This research was based on an
extensive literature review and on the formulation of a case study, which was implemented and fully assisted by the
Cosmetic Bio Phito Therapeutical Company. Using the assistance offered by the Enterprise Resource Planning tool (ERP), it
was possible to infer a mere substrate whose only purpose was to support the raw material that would effectively be used
to an applicability that, besides being profitable, brought circularity to the operational flow. The results obtained with the
implementation of the Circular Economy brought to the company that was studied the opportunity to avoid inadequate
waste management, in addition to a financial savings of around U$ 2,000 per month due the reduction of expenses with a
material that, instead of being improperly discarded, was reprogrammed for practical use, through recycling. The research
was developed using the Problem Based Learning (PBL) approach and hands on activities during all the work.
Keywords: Circular Economy; raw material; Enterprise Resource Planning; recycled.
1 Introduction
Since the beginning, more specifically around 10 thousand years ago, the history of mankind was in the Upper
Paleolithic period. A milestone in Prehistory in which the Cro-Magnon Man, a primitive population of the
species Homo Sapiens, had already developed their first tools, still with limited sophistication, given the
incipience and simplicity of the techniques used, defining characteristics of this revolutionary moment in
History.
From this point on, great waves were responsible for changing the course of humanity.
From the stone age to the beginning of the First Industrial Revolution. The, until then, hominids left the
savannah and began to compose a new class that emerged in History, called artisans, the first industrialists on
a modest scale.
Urban landscapes were radically changed, just as they made way for a vertiginous, unplanned urbanization,
recurrent throughout Industrialization.
According to Schwab (2018) the Industrial Revolution in the 18th century brought several transformations in
the production means and in the European urban scenario. With the industrial rise, handcrafted manufactures
and productions were quickly replaced by a scenario of incessant search for technological development, arising
from the abundant thirst for productivity.
Baine (2012) said that the Green Engineer must be one of the Engineering careers responsible to save the Earth,
creating the infrastructure and means for impoverished countries to have access to clean water, electricity, and
internet services to maintain the Sustainability of the planet. The circular economy is a concept of a true
sustainable economy that it works without waste, save resources, and works with synergy with the biosphere.
Instead of looking at emissions, subproducts, and damaged or unwanted goods as waste or garbage, these
materials in circular economy become raw materials and inputs for a new production cycle.
286
Throughout the consecutive Industrial Revolutions, certain triumphs became noteworthy, namely: machinery,
the progress of means of transportation, the advance of intercontinental communication, the advent of the
Internet and new technologies, resulting in the expansion of the, until then restricted, Globalization.
From the mitigation of infant mortality, good quality and life expectancy to the approach of the numerical,
Campos et al., (2011), for the levels of education, health, and income, according to the HDI indicator. From high
levels of education and literacy to high per capita income. It is palpable that these Revolutions, including the
most recent one, the so-called Industry 4.0, have been responsible for significant improvements in the welfare
of society, especially in the so-called developed countries. However, their price is being charged.
The human population continues to grow exponentially around the world, with the imminence of reaching a
population of about 10 billion people by the year 2050. Therefore, the combination of population growth and
social upswing has put increasing pressure on essential natural resources.
With the atmosphere and oceans overburdened with carbon, coupled with uncontrolled burning of fossil fuels,
unprecedented climate disruption is manifesting itself. The enormous extraction of resources from nature, with
no or minimal replanting policy, leads to the current emergency of the Brazilian flora, defined by the problems
of deforestation and wildfires.
The ocean covers about 71 % of the planet Earth, and the waste generated by human activities is totally
dispersed in it.
In almost every corner of the ocean, there are already records of the presence of plastic and microplastics.
From the above, science shows that more than 80 % of this material is located decanted on the ocean floor
and has been degrading successively into smaller and smaller particles.
Today, humanity is at a turning point.
It has become imperative to carry out emergency planning to ensure the subsistence of the next generations.
To this end, it is unthinkable to conjecture an eventual renunciation of everything that humanity has already
achieved. Nevertheless, it is feasible to rewrite the formula that will unveil the secret responsible for
perpetuating, in a sustainable way, this new Industrial Age.
According to Stahel (2016), […] “Today’s goods become tomorrow’s resources at yesterday’s price”
2 Scope
The model with which this scientific study was developed, directly implies rigorous exploratory research
through literature review. In addition to the meticulous analysis and investigation of previously published
theoretical material, field research was also proposed.
According to Powell & Weenk (2003); Kolmos et al. (2009), the Project Based Learning approach (PBL) is a very
important method of Engineering Education and put the students in a real contact with the professional
environment where the students will be work in the future. To get this knowledge is necessary that the students
participate from the start of the project to the final step to learn all the steps and learn to read and
implementation of the project.
Campos et al. (2012) show how to create an environment of learning, mainly using, a hands-on activity based
on the PBL approach.
Lima et al. (2014) shows as is important the projects developed with the Industry for the development of
Professional Competences in Industrial Engineering Education.
Based on these concepts we decided to use PBL approach in this research.
This comprises data collection, which suggests an interpretation of the results obtained.
Conducted right in the original environment where your object of study is located, field research results in
accurate, timely, and exceptionally efficient analysis.
287
In line with both pillars, a Case Study was also introduced. As this is still a little explored theme and little diffused
in the market, opportunities for development in techniques and business models that apply Circular Economy
in practice, as an inherent part of the production line, can be evidenced.
A Chemical Industry, from the Cosmetology field, was selected for the deepening of this study and the
introduction of the concepts of circularity in the daily life of a Small Business Company. Small businesses are
made up of micro and small companies and individual micro companies. In Brazil there are on average 6.4
million establishments and, of this total, it is measured that 99% are registered as micro and small company.
That said, it is demonstrated as a niche of strong representation in the Brazilian scenario and as an opportune
chance to be demonstrated through example, data and proven results, that it is an action whose applicability
depends on few resources; low investment cost; and is still able to result in the reduction of materials held in
stock, reducing storage costs, increasing the turnover of inputs and the protection of the amount of its working
capital. Finally, it ensures a better financial planning of the company and a redistribution of investments. Thus,
the focus shifts to improvements in the production belt and, in a possible automation, in the means of
production.
This proves, therefore, the endless returns that are generated for society, in terms of sustainability and
environmental preservation, as well as the positive cycle that is incited.
According to what was said above, this study aims to demonstrate that the practice of Circular Economy should
not be restricted to large industries, nor does it require the export of technologies to be feasible, on the
contrary, implementing a system that brings circularity to the process allows all companies to have an
experience of intelligent manufacturing, in which natural resources are saved, the destination of cash flow is
reconfigured, and therefore new technologies arising from Industry 4.0 can be implemented, democratizing
competitiveness within the national market itself.
3 Applying Research in Engineering Education

When focused on the educational context, this study addresses efforts to improve logical investigation skills
and to improve the search for solutions to overcome the obstacles presented, which are intrinsic characteristics
of an engineering education.
Aiming to acquire this knowledge to conduct efficient research, with results that are tangible to reality, notes
were taken on articles presented at the UNESP Symposium of Industrial Engineering (SIMPEP), in 2020, through
which it was possible to understand more clearly the characteristics that differentiate Circular Economy from a
Recycling Process, for example, among other details that highlight the importance of this concept today.
It´s important to highlight some academic-scientific-cultural activities promoted by the Pontifical Catholic
University of São Paulo, which were of immeasurable contribution to the formulation of this research. These
activities consisted, mainly, of lectures and seminars that dealt with subjects of fundamental importance to the
development and conclusion of the chosen thesis. Some of the subjects approached were The Future of
Industry; Prototyping Technology Applied to Factory 4.0; Use of Technology in Small Business and
Opportunities of Waste Management.
Finally, the experience obtained by following the daily routine of the factory floor, in the industry selected for
the application of the Case Study, was extremely enriching at the juncture of studies, aiming at a future training
in the Engineering career. There, it was possible to learn about the manufacturing processes, how the
organizational structure of a company works, the obstacles faced during production management, planning
strategies and risk analysis, financial control, and, focusing on the theme of this project, the most appropriate
methods of waste disposal and management. The purpose of the study was to reduce the generation of
production waste to as close to zero as possible. This opportunity of proximity with the cosmetics industry Bio
Genetyc, also allowed to acquire the knowledge of using the Corporate Software ERP (Enterprise Resource
Planning). This technology is becoming more and more usual in Engineering and, frequently, required in the
management areas because it makes possible to analyse and follow up each step of a production process,
288
besides allowing a rigid stock control. The ERP tool helped, rigorously, in the monitoring of each result
obtained.
4 The Circular Economy

Due to the usual logic of extraction, production, consumption, and disposal, humanity has submitted itself to
a Status Quo limit situation. The, until then, trivial model of traditional and linear economy has suffocated
natural resources and expressively mitigated the quality of life of the inhabited environment. Emerging as an
alternative to contain the consequences that this and the next generations will be subject to face, a definition
emerges in defence of the operating modes that surround the sustainable model.
In 1989, what is today an integral part of the lexicon of the economy, namely known as Circular Economy, was
born.
Elaborated in an autonomous article by British economists and environmentalists David W. Pearce and R. Kerry
Turner, it was demonstrated, through analysis, that the environment was totally disregarded in the unbridled
search for the consumer market and the desire to gain a position of influence among the capitalist nations, an
episode reinforced and contextualized in the midst of the Fall of the Berlin Wall (1989), which culminated in
the end of the Cold War and caused, subsequently, the dissolution of the Soviet Union.
At the time, the environment was bequeathed a secondary role, as a simple waste reservoir. Also known as
"cradle to cradle", Circular Economy does not work with the idea of waste, after all, everything can be
progressively reused in a new cycle. In other words, by designing the product design in its entirety before
manufacturing it, it was possible to avoid the use of materials that cannot be reused, so that all creation returns
to the production cycle, drastically limiting waste generation.
According to the United Nations (UN) and the Circle Economy report, only 9 % of the global economy is
circular, which in practice confirms that of the 92.8 billion tons of waste generated by industrial and residential
means, less than 10 % returns to the production chain, that is, more than 80 billion tons of plastics, fossil fuels
and biomass are inadequately disposed of to the environment.
Essentially, Circular Economy is based on value creation through a "closed loop", in which industrial and social
evolutionary concepts aim to achieve integral sustainability objectives in relation to a no-waste philosophy,
starting from a regenerative system.
4.1 The Waste Management Issue

Waste, on the other hand, pragmatically suggests an extra cost within a corporation and, accordingly, has a
negative impact on its reputation, depending on the type of disposal that is routinely performed. To this end,
the Federal Accounting Council (CFC) has been orienting accountants to bring these environmental points to
the balance sheet of their partner companies, aiming at the lowest wear and tear not only to the already scarce
natural resources, but also to the very image of the corporation in question and, consequently, aspire to the
protection of the balance sheet in view of inspection agencies, such as: the Civil Police, CETESB, and the Sanitary
Surveillance.
Currently, inadequate areas of final disposal, such as dumps, and controlled landfills receive an average of 80
thousand tons of waste per day. These are 80 thousand tons that could be redirected and better used, by
transforming them into by-products. However, the consequences for this phenomenon called, Inadequate
Waste Management, are impacted soil; impacted groundwater; impacted rivers and lakes; spreading vectors
and, because of these calamities, human beings are fervently impacted.
Between the years 2017 and 2018, the generation of MSW (Urban Solid Waste) in Brazil increased by almost
1%, reaching the mark of 216,629 tons per day. As the population also grew in the period (0.40 %), the per
capita generation obtained a minimally reduced elevation (0.39 %). This means that, on average, each Brazilian
generated a little more than 1 kilo of waste per day.
289
However, taking into account the contemporary atypical situation in which society has been subjected, it is
important to take into account the change in this statistic, given that in 2020 and 2021 the number of food
orders via apps will expand exponentially, causing a predicted increase in the number of disposable packages
that lead to an extreme increase in waste originating in homes and likewise, as a reflection of the pandemic, a
radical demand for hospital supplies.
On August 2, 2010, Law No. 12.305/10, better known as the National Solid Waste Policy, was enacted and is
fully responsible for organizing the way in which the country manages its waste, requiring from the public and
private sectors transparency in the destination and treatment of its waste. This law establishes the principle
that everyone, from the person in charge of receiving the raw material to the dispatch supervisor, is fully
responsible for the management of these remaining materials.
A viable solution to be undertaken and, increasingly, instituted in reference corporations in the market to
encourage their competitors in the industry to apply the same strategy, is the so-called Reverse Logistics.
This instrument consists in the economic and social development, characterized by a set of actions, procedures
and means that enable the collection and return of solid waste to the business sector, for reuse in its production
cycle or even proposing the realization of a new final route, this time, environmentally appropriate.
These procedures consist, respectively, in reusing or reprocessing the raw material for the production chain;
waste generation by the manufacturer (during manufacture); waste generation by the merchant (during
commerce); waste generation by the consumer (after use); disposal; voluntary delivery point; collection and
recycling; reusing or reprocessing the raw material for the production chain.
All phases, whether of manufacture, marketing, or use, can be returned and reprocessed for the creation of a
new raw material, without necessarily having to extract new resources from nature and much less discard the
old ones.
The state of São Paulo has been one of the forerunners of the implementation and obligation of Reverse
Logistics in more and more products with flammability risk or that, today, have an inadequate disposal.
4.2 The Microplastics Issue

Considered one of the most problematic, dangerous, and difficult to solve issues on this subject, special
attention should be paid to the topic of microplastics.
The difficulty of the subject matter becomes clear when, upon reflection, it becomes possible to conclude that
every little piece of plastic that has been created to date still exists. Even after reaching its degradation limit
year, plastic will never completely disappear, it will only decrease in size. The problem lies in the fact that the
smaller the material gets, the easier it is for an organism to ingest it without realizing it, and this can lead to an
interaction with different chemical components as soon as it fragments.
Microplastics, when dispersed in the ocean, are often found in plankton, which provide about 60 % of the
oxygen found in the atmosphere by carrying out photosynthesis. However, since plankton in the pre-existing
food chain relationship also serves as a subsistence for some species of fish, the fish ingest the plastic, and the
plastic is automatically incorporated into man's food chain when he feeds on it.
Recent research conducted by European scientists from the Medical University of Vienna has shown, by
collecting human excrements from different countries, that all the excrements studied contained more than
50,000 microparticles of plastic each.
Demonstrating at last that the contamination of the garbage has reached our intestines.
Actions admittedly capable of mitigating the existence of plastic in the seas consist of investments to enable
the use, productively, of the new bacterium identified in 2016, which is called Ideonella sakaiensis 201-F6 and
has the capacity to degrade PET, using it as its main source of carbon and energy.
According to Bornscheuer (2016), if terephthalic acid can be isolated and reused, it could generate huge savings
in producing new polymers, without the need for petroleum-based feedstocks.
290
Aggravating factors related to socioeconomic and occupational problems should also be assumed, since in
Brazil, the great source of waste that reaches the sea through the watersheds, about 80 %, is located on land
roads, mostly from irregularly occupied areas, because there is no collection service and proper disposal of
solid waste in these regions.
After all, similarly, considered as a viable option for the resolution of the adversities caused by improper
residual disposal, the concept of Industrial Symbiosis is presented. This can be defined as a long-term
association between two organisms of different species, i.e., submerging the concept to the corporate
environment, it is assumed that the waste generated in one company can be used by another, following the
minimization or total elimination of waste.
4.3 Successful Cases

In the domestic sphere, the number of consumers who demand differentiated attitudes is growing, and above
all, from the circular economy perspective, they opt for sustainable products and services, in which the
responsibility for waste management is planned from manufacturing to the final disposal of packaging after
consumption.
Within the cosmetics industry, two companies stand out for their socio-environmental concern: Natura,
responsible for launching a challenge to itself which consists in the search for innovative solutions, such as
zeroing the brand's packaging waste, has already started to demonstrate positive results for the theme, by
declaring that the “Kaiak Oceano” perfume packaging is produced with plastic residue of which, at least 50 %
of the total of this material that is used in the confection of each packaging, is taken from the ocean and
collected on the beaches of the Brazilian coast.
Another even more recent measure taken by the Natura & Co group is the launching of its packaging recovery
program at its branches located in Mexico. Aiming to be able to contribute even more effectively to reducing
the environmental impact generated by improper handling of waste, the Brazilian company has promoted the
action "Reverse logistics of post-consumer packaging", with which it invites consumers and its entire network
of consultants to join the Circular Economy by taking empty, clean, and dry Natura packaging to any of its
physical stores. By participating in the initiative, the contributor will receive 25 % discount on refills for each 5
packages delivered.
"With this proposal we want to invite people to be part of the movement, promoting the circularity of post-
consumption packaging, in line with our commitment to be an agent of change that adds value to society and
generates positive impacts, achieving a world with more beauty and less waste," said Griscelda Ramos, Director
of Sustainability at Natura Mexico.
In June 2020, the 4th largest holding company in the beauty business in the world, which gathers Natura, Avon,
The Body Shop and Aesop, presented its plan "Commitment to Life", to face the most urgent global problems.
Among the commitments highlighted in the document is that of "embracing circular economy and
regeneration", to achieve greater circularity in packaging. A clear example of this is in the constitution of the
container used for the Natura Ekos and Tododia lines, according to Alves (2017) and the Brazilian company
itself, PET plastics from the recovery of bottles that feature this material are used, extending in such a way the
useful life, and contributing to its maintenance within a productive cycle.
By implementing these good practices, packaging material is reduced by up to 80%, avoiding more than 3,000
tons of waste annually, as well as the emission of 8,000 tons of greenhouse gases, according to company data.
In 2020 alone, Natura managed to recover 667 tons of waste.
Another reference brand in the search for sustainable solutions associated with its products is B.O.B.,
responsible for offering a bar alternative, aiming at gradually replacing the plastic packaging of shampoos and
conditioners.
The digital transformation, although it may seem "cleaner" than its predecessor technologies, will generate a
legacy that is even more harmful and difficult to control if not wisely handled. Thus, the Circular Economy
coupled with Industry 4.0 arises, intending a sustainable management of the supply chain, through the Internet
291
of Things, which connects stakeholders to efficient flows of materials and information. Thus, by linking ideas,
people and places, the generation of good opportunities is potentialized as a result.
5 The Case Study

After searching for a practical means and a feasible experience for the application of the concept of Circular
Economy in an Industry, whose manager is configured in the same classification of Micro and Small
entrepreneur, as well as the 99 % of entrepreneurs located in the national territory who, despite all the fiscal
disincentive and the lack of credit stimulus with which they are afflicted, it is important to emphasize that they
continue to leverage the country's economy by sustaining about 27% of the Gross Domestic Product (GDP)
and ensure 54 % of formal employment in Brazil, according to data from “Brazilian Service Support to Micro
and Short Companies” (SEBRAE). The idealization of taking advantage of the representativeness that the
sector enjoys in the private sector sowing sustainable practices that, in parallel, also play a financially attractive
role to the company, was conceived. In response to the above, the structuring of a case study was outlined
that, after all the work of elucidating, theorizing, defining which strategies are more palpable and
experimenting with possibilities, it was possible to attest to the simplicity with which Circular Economy not only
can, but should be present in each stage studied during the formulation of a product and/or in the operational
planning that surrounds a production.
The target company of this study was Cosméticos Bio Phito Terápicos Ltda.
Founded in 1984 in the city of São Paulo, Bio Genetyc, the brand for which it is recognized in the market,
started its activities manufacturing cosmetics for third parties and for the retail market. Today, with over 30
years of continuous experience acquired, its main focus is to offer products geared to the needs of distributors
and clients from the international market, clients who aim to work alongside a company constantly concerned
with the quality, sustainability and technological innovation promoted in its merchandise, whose origin is
entirely from the domestic industry and each production process, from the development of products to the
act of packaging them are thought out and closely monitored by its manager.
With the help of the Enterprise Resource Planning (ERP) tool, it was possible to observe in detail all the
production chain related to the industry in question, in order to establish the procedures that, when optimized
or possibly replaced by more viable techniques, would achieve tangible results for the corporation, serving as
an example for commercial partners and even competitors, whether in the same sector or not, to contemplate
the unmeasured benefits that were added after the experimentation of this pilot, encouraging them in such a
way, to apply similar methods in their production cycles, expanding the benefits gained in an unprecedented
quantitative scale.
For this study, it was selected within the context of raw materials, the roll of labels, thus being elected as the
target object, where the understanding regarding the applicability of Circular Economy would be properly
tested.
Once the first phase of decision making about the most suitable raw material to undergo the readjustment
process was concluded, the possible strategies to implement the concept were defined.
Initially, it was conjectured the use of the smooth roll, making use only of its substrate, without the conjunction
of labels on the surface so that it could be reused to wrap the products already boxed, about to be shipped to
the carrier.
However, it was noticed the low efficiency in what concerns the protection of the products contained in the
box; in the little aesthetic attractiveness given and in the little performance promoted to the material, as much
as it was expected when readjusting the utility of its function.
Thus, the idea of using a paper shredder emerged, one of those easily found in offices and accessible in the
main stationery stores. The objective is to take advantage of the physical concept that comprises, for example,
the springs. By going through the fragmentation process, the paper acquires the property of flexibility, which
292
enables it to store mechanical energy, making it, therefore, a pseudo-spring able to dampen the movement of
objects inside the box during transportation, ultimately avoiding damage to them.
Despite the addition of an extra procedure in the production chain, with the insertion of the fragmentation
action, it was possible to verify a significant improvement and a true optimization regarding the amount of
material spent, given the increase in volume caused, as well as in the improvement of aesthetics and the benefit
of its new assignment.
In short, an investigative process can be evidenced in practice, with a simple, effective, and easily absorbable
solution in the day by day of an organization, in which by recovering all its production, a work of identification
is carried out on which inputs, products, and methodologies are likely to be incorporated into the closed
production cycle.
When one acquires the ability, by observing the whole, to promptly search for the opportunities contained, a
ridiculous raw material, in the eyes of the layman, is equivalent to the master key capable of solving an
important humanitarian setback, in the eyes of the visionary.
Thus, a mere support for the labels is transformed into a resource of unavoidable usefulness, which is
interesting to highlight the fact that it is manufactured with waxed paper or impregnated with impermeable
substances, configuring a non-recyclable waste and difficult to compost.
In other words, the technique of certifying the safety provided in the transportation of goods was improved
and new costs were avoided with the purchase of materials such as bubble wrap, whose average price to cover
a small box is evaluated at US$ 0.04. When considering a production of 50 thousand boxes per month, it is
palpable as a result the conception that just with the implementation of this simple addition of function to a
material that would be discarded in the production cycle, besides avoiding an inadequate waste management
and an accumulation of materials improperly discarded in landfills, it also brings a saving of about US$
24,000.00 for the company per year.
Therefore, the maxim that "waste is nothing more than raw material out of place" is proven.
During the process I was present from the start of the Project till its final, working in all steps in activities hands
on.
6 Conclusion
This work brings to me the opportunity to apply several concepts learned in classes and develop in floor of
company, competences and skills of engineering professional, based on the confluence generated by the
bibliographical review, the field research promoted by the collection of data and experiences of partner
companies in the segment, as well as on the empirical investigation on the others authors of this essay,
demonstrated as a result the initiation of a transformation in the population's view of the Circular Economy
theme, ranging from the micro company to the most knowledgeable CEO in the market.
The consumer market is progressively more demanding in selecting which company value their investment and
help to maintain the sustainability of the planet, preserving the natural resources in the production cycle,
minimizing the impact generated by the discarding and mitigation of the need for extraction of new inputs
taken from the environment. In addition to the sustainable benefit granted to the planet, which will reflect
significantly on society, the implementation and insertion of Circular Economy instinctively to the internal flows,
is also able to promote a boost in its invoicing, from the economy provided in line with the profitability achieved
due to the positive movement that had been promoted, and may even gain important highlights of real
relevance to the agenda, raising and consolidating such a way, a consumer market even more reliable.
Finally, this case study presented its role in offering the necessary subsidies for companies inclined to adopt
such practices, and can be applied due its simplicity and, synchronically, with the endless constructive return
that can be achieved, as evidenced in the calculations and analyses based on the data offered.
Besides that, the implementation of this approach brings financial saves to the companies.
293
7 References
Alves, R. A., Martins, R. C., & Paulista, P. H. (2017). Case Study at Natura Company: sustainable practices and creation of shared value.
University of Vale do Paraíba.
Baine, C. The Green Engineer – Engineering Carees to Save the Earth. Engineering Education Service Center. Springfield, OR – 97477 – USA,
2012.
Bornscheuer, U. T. (2016). Feeding on plastic. Science, Vol 351 - Issue 6278, pp. 1154 - 1155
Campos, L. C., et al. (2011). Challenges of Engineering Education: vocation, training, professional practice, methodological experiences, and
propositions. COBENGE, 113-164.
Campos, L.C., et al. (2012). Project Approaches to Learning in Engineering Education – The Practice of Teamwork. 1 ed. Rotterdam – The
Netherlands: Sense Publishers, 2012. V. 1. 175p.
Du, X., de Graaff, E., Kolmos, A. Research on PBL Practice in Engineering Education. Sense Publishers, 2009.
Rui M. Lima, José Dinis-Carvalho, Luiz Carlos de Campos, Diana Mesquita, Rui M. Sousa, & Anabela Alves. (2014). Projects with the Industry
for the Development of Professional Competences in Industrial Engineering and Management. In Natascha van Hattum-Janssen,
Rui M. Lima, Dinis Carvalho, Sandra Fernandes, Rui M. Sousa, Francisco Moreira, Anabela Alves, & Diana Mesquita (Eds.), Sixth
International Symposium on Project Approaches in Engineering Education (PAEE’2014) (p. [1-11]ID13). University of Minho.
Powell, P & Weenk W. (2003). Project-Led Engineering Education. Lemma Publishers. Ultrech, 2003.
Schwab, K. (2018). The Fourth Industrial Revolution. World Economic Forum, Edipro Publisher
Stahel, W. R. The Circular Economy. Nature News. Nature Publishing Group, 2016
294
A strategy to support Engineering Education teaching staff
monitoring students' learning process: Metacognitive
Challenges
Daniela Pedrosa1, Leonel Morgado2, José Cravino3
1
CIDTFF – Research Centre on Didactics and Technology in the Education of Trainers, Department of Education and Psychology
University of Aveiro, Aveiro, Portugal.
2
INESC TEC - Institute for Systems and Computer Engineering, Technology and Science, LEAD, & Universidade Aberta, Coimbra,
Portugal.
3
CIDTFF – Research Centre on Didactics and Technology in the Education of Trainers, School of Science and Technology, University of a
Trás-os-Montes e Alto Douro, Vila Real.
Abstract
It is increasingly required that Engineering Education courses include activities that promote the development of cognitive
skills, such as metacognition. However, including such activities is challenging for lecturers, particularly in Distance Learning
contexts. It is also complex, when working online, for teaching staff to carry out monitoring of the metacognitive learning
processes of students, understand their difficulties, and provide formative feedback.
In this work, we present the design and discussion of a pedagogical strategy: Metacognitive Challenges (MC), which allows
lecturers to monitor the evolution of students' perceptions regarding their learning process We discuss how lecturers can
use MCs for formative assessment and how to weave this intervention with individual students or groups. The Design
Science Research methodology was adopted for the design, implementation, and demonstration of MCs, applied in a
Software Engineering course within a distance learning Informatics Engineering undergraduate programme. We exemplify
how MCs have the potential to support monitoring of students' cognitive and metacognitive processes and offer a set of
guidelines on how the teaching staff can use them.
In future work, we intend to evaluate the effectiveness of MCs in different teaching contexts, and develop technological
solutions that facilitate the monitoring process (reduce the time and effort required for analysis of MC content).
Keywords: Metacognition; Cognitive process monitoring; Software Engineering Education; Self-regulation and Co-
regulation of Learning.
1 Introduction
Increasingly, Engineering Education is committed to active and situated learning, putting students in contact
with real engineering experiences (Wengrowicz, Dori & Dori, 2018). In software engineering, the novice-to-
expert transition requires students to develop advanced technical skills, namely: large-scale programming and
software development processes (ACM & IEEE, 2016), the ability to think abstractly, and the adoption of
cognitive and metacognitive strategies (Garcia, Falkner & Vivian, 2018). Major professional organizations in the
field (ACM & IEEE, 2016) recommend connecting practical aspects of real practice with educational plans.
Distance learning in universities has expanded. However, this brings novel challenges and dropout is usually
higher (Broadbent, 2017; Pedrosa et al., 2021). Students experience difficulties planning, developing, and using
self and co-regulation learning (SCRL) skills properly (ibid.), which adds new challenges to online teaching, both
regarding course structure and the practice of e-pedagogy (Kara et al., 2019). Strategies that allow lecturers to
overcome such challenges include: 1) provide formative assessment; 2) provide timely, continuous, and
constructive feedback that facilitates the process of planning, managing learning and problem-solving skills;
3) adoption of appropriate assessment tools contributing to a better optimization of learning; and 4) reflect
upon and adjust their pedagogical practices towards enriching students’ learning (Kebritchi, Lipschuetz &
Santiague, 2017; Kara et al., 2019).
295
In Engineering Education, the incorporation of metacognition in curriculum plans is advantageous as it helps
students improve essential skills and their ability to engage in SCRL (Wengrowicz, Dori, & Dori, 2018; Wallin &
Adawi, 2018). Also, metacognitive regulation is a characteristic that distinguishes experts from novices (Kim &
Lim, 2019), hence developed in the novice-to-expert transition.
Metacognition is “thinking about thinking” (Flavell, 1979) and is understood as knowledge and capacity for self-
judgment about one's own cognitive processes and control, that allows identifying successful strategies (e.g.,
planning, analysis, management), emotional self-efficacy monitoring, and evaluating metacognitive knowledge
according to feedback (Wengrowicz, Dori, & Dori, 2018; Prather et al., 2020; Dindar, Jarvela & Jarvenoja, 2020;
Schuster et al., 2020). When the student has metacognitive awareness about these strategies, performance
improves and success in higher (Davis & Hadwin, 2021; Frasier, 2021).
Metacognition can be understood in two dimensions (Wengrowicz, Dori, & Dori, 2018): 1) The regulation of
cognition - which allows students to develop skills, such as: planning, monitoring and evaluation of their tasks,
which allow them to take control of their learning; and 2) Knowledge of cognition – which helps students
improve the way they learn and solve problems, in conjunction with the regulation of cognition that allows
students to acquire key skills in engineering.
Software Engineering requires problem-solving processes; but it is difficult to study cognitive control (Prather
et al., 2020). The use of pedagogical strategies that promote the acquisition and use of metacognitive skills by
students is crucial to support teaching feedback processes that encourage self-analysis of errors and positive
correction (Sáiz-Manzanares & Montero-García, 2015). However, there are few instructional programs that
explicitly focus on teaching control and monitoring skills (Schraw & Gutierrez, 2015) and few studies explore
the use of strategies that facilitate formative assessment processes by teaching staff (Wallin & Adawi, 2018).
In this work, we present the design of a strategy that we developed, the Metacognitive Challenges – MC
(Pedrosa et al., 2021). We discuss how MC can be used by teaching staff to monitor the cognitive and
metacognitive processes of students and to provide formative feedback.
2 Teaching context
In previous work, we explained that the concept the Metacognitive Challenges (MC) results from the
articulation of the different dimensions that involve a reflexive metacognition process with the concept of
challenges (Pedrosa et al., 2021) and reported on the positive perceptions of students about MC: they are
perceived as being innovative, motivational, and useful to develop self-regulating learning strategies and
greater self-awareness about one’s own skills.
The MC were designed within scope of the e-SimProgramming didactic approach (Pedrosa, 2021). This
approach was implemented in an online asynchronous course ("Software Development Laboratory”, LDS in the
Portuguese-language acronym), part of the 2nd semester of the 2nd year of the Informatics Engineering
undergraduate programme at Universidade Aberta (UAb), using the Moodle platform. It is organized along a
six-topic syllabus with the goal of scaffolding undergraduates transitioning from novice programmers into
proficient programmers that acknowledge the relevance of employing engineering structural qualities in the
development of their software programs. The students are typically working students, aged 24-60 years old,
residing in various regions of Portugal and abroad, with different academic backgrounds. The teaching and
learning methodology employ Project-Based Learning, through the development of software projects by
students or teams, integrating concepts sequentially throughout the semester (Pedrosa et al., 2020, 2021).
3 Methodology
This work focuses on the problem that the teaching staff faces when monitoring the evolution of cognitive and
metacognitive processes of engineering students, which are essential for successful learning, particularly in
Distance Learning. We adopted Design Science Research (Hevner & Chatterjee, 2010), which develops
knowledge by embodying it in the design, implementation, and evaluation of an artefact. In this study, we
296
present and demonstrate through examples how Metacognitive Challenges (artefact) can be a strategy that
helps teaching staff monitor students' metacognitive processes. We evaluated the examples by collecting data
on their use with 32 students (out of 50 enrolled in the LDS) who agreed to participate in this research during
the academic year 2019/2020. All students granted their authorization through an informed consent statement.
4 Pedagogical design of Metacognitive Challenges

4.1 Design and implementation:
Two types of Metacognitive Challenges (MC) were designed and implemented (Pedrosa et al., 2021) with
different pedagogical goals, timing, and characteristics (see table 1), allowing both students and lecturer to
monitor the cognitive and metacognitive learning process of students.
Table 1: Differences between the two types of Metacognitive Challenges (MC)
MC Type 1 MC Type 2
Dimensions Metacognitive challenges as fortnightly Metacognitive challenges to promote self-
reflections about the learning progress reflection about programming concepts
Pedagogical Stimulate self-reflection and self-assessment of Promote students' self-reflection on their ability
Goals students about: to apply the knowledge they have acquired
a) the learning progress. throughout the course, regarding the technical
b) the development of the project. aspects of software development processes.
c) self-confidence about their work. Note: For the construction of these MC, in each
topic, Bloom's taxonomy was used.
Components of Regulation of cognition: Knowledge of cognition: Declarative knowledge,
the Planning, monitoring (self-awareness), and Procedural knowledge, and Conditional
metacognitive evaluation (self-assessment). knowledge. Metacognitive experiences: Feelings
domain and judgments.
When it appears At the end of each syllabus topic. At key moments in each syllabus topic
in the course (beginning, middle, or near the end).
Number of MC 6 12
Format Both types of MC have been implemented in Moodle, using the Quiz feature. In the introductory
part, a narrative by the fictional Catmming character is used (Pedrosa et al., 2021) that triggers the
student's reflection to respond and reflect through prompts (questions in the quiz).
The quiz includes Likert-scale closed questions (Very Low; Low; Regular; High; Very High), and open-
ended questions where students justify their choice or explain in detail the answer given in the closed
question.
Questions Standard questions that are adjusted by There are no standard questions. The questions
syllabus topic and the expected development vary according to the metacognitive goals
status of the software project. defined for each syllabus topic.
Usefulness for It provides awareness of the student's It allows the lecturer to infer and formulate
the lecturer perception of the learning regulation processes formative hypotheses regarding expected skills:
adopted throughout each topic. Identify the whether they were developed, if knowledge was
type of difficulties; Provide SRL strategies for the applied correctly, and understand students'
student (e.g., planning, time management, feelings and judgments about their technical and
seeking help); Checking discrepancies knowledge skills.
regarding the level of confidence and self-
assessment with their appreciation of the work.
4.2 Demonstration/Evaluation: How metacognitive challenges help lecturers monitor

and be aware of students' metacognitive learning processes?
The Metacognitive Challenges (MC) provide awareness information to the lecturer about the cognitive and
metacognitive processes of the students regarding their own learning. With this information the lecturer can
define ways of acting/intervene to provide formative feedback for the different situations that occur, in a
personalized way (individually) or at the group (class) level.
297
Ideally, the lecturer should analyse all answers of all students. However, analysing MC is a time-consuming
activity. Thus, it is suggested that the lecturer, to gain some awareness about the general panorama of the
class, generates response graphs that allow understanding the critical aspects to be solved and define a
timeline. We demonstrate and discuss, in the following examples, how to use information from MC to monitor
and intervene with feedback.
Example 1 – Motivational intervention

In the question (MC type 1) “At what level of learning progress do I consider my evolution to be?”, one can see
where most students´ answers are. In the following example (figure 1), most students consider that they are at
a “Regular” level (satisfactory learning progress).
Figure 1: Responses of LDS students for the 2019/2020 academic year to the question “At what level of learning progress
do I consider my evolution to be?”.
In this situation, the lecturer can decide to focus on students who expressed a “Low” level and understand
through their answers to the question “Why?” the reasons that the students provided. Some examples: “I started
with few bases (…)”. S21, March 25, 2021. “I think I still have a long way to go.” S57, March 21, 2021.
Analysing these responses points towards motivational factors and lack of confidence (or modesty). This may
inform enable a teaching decision to provide feedback to the whole class of a motivational nature, weaving a
set of suggestions for self and co-regulation of learning strategies (such as suggesting that students seek help
from the teacher and colleagues). Or instead provide individualized feedback, or yet another approach. The
rationale is that the MC provide a structure for teaching decision-making: identifying focus aspects from the
Likert-scale responses and then analysing concrete responses within those focus aspects.
Example 2 - Awareness of difficulties and help students overcome them

The lecturer can identify difficulties reported by students and try to understand them. By analysing the answers
to the question: “Did I experience difficulties in this initial phase of software development?”, the lecturer may
realize, as in our test case, that most students expressed that they felt difficulties (Figure 2).
Figure 2: Response of LDS students for the 2019/2020 academic year to the question “Did I experience difficulties in this
initial phase of software development?” of MC type 1.
298
The lecturer can try to understand the type of difficulties from the responses to the follow-up question “Why?”.
In our test case, students reported difficulties of various types, e.g.: “I had a hard time understanding how to
use MVC with the Selenium API (…). (…) the interpretation of the texts, the fact that the links to documents were
not highlighted, made me start developing the demonstration application when that was not the goal”. S56, April
5, 2020.
In this example, one can identify two types of difficulties:
a) The student's understanding of the practical application of course material (MVC software architecture) with
an API (software development tool). The instructional intervention may be to promote class sharing of specific
difficulties or offering individualized help, towards better clarification of these concepts to the student or to
the class (knowledge of cognition).
b) Understanding specifically which task to perform, from the material provided. In this situation, the lecturer
may elect to improve the pedagogical resources for by correcting the next tasks through the placement of
visual elements (highlights) and a final task list to help students; or to alert the students in general to check if
such misunderstanding occurred to others, etc.
Other responses have shown us that these MC may also expose difficulties not related to the course itself but
related to personal lives / professional factors of the students, that affect their self-regulation learning
strategies: “Mainly, personal difficulties led me to compromise time management.” S36, April 13, 2020. This
awareness may enable the teacher to recommend activity prioritization for students, plan recovery plans, or
other approaches.
Example 3 - Clarification of aspects related to the syllabus and concepts that may be misunderstood
The question “Were you able to perform all the requested tasks?”, enables understanding the students'
perception of overall task completion. The lecturer can check whether that perception is correct or not. That is,
a student can believe that he/she has completed all tasks, but in reality some task may be missing. This
awareness may recommend alerting the student or the class, in realization that uncompleted tasks are not
simply delayed, but rather disregarded.
Another approach is this case (figure 3), where most students affirmed that they were able to complete all
tasks. If this matches the perception of the lecturer, the focus can shift towards the students who indicated
“No”, to analyse the reasons.
Again, the “Why” questions will be the source of that analysis. Different types of situations can be encountered
requiring interventions, e.g.: “I await feedback (…) to finish this sprint” S8, April 20, 2020. In this circumstance,
there was an expectation of feedback to advance, of which the lecturer might be unaware.
Figure 3: Response of LDS students for the 2019/2020 academic year to the question “Were you able to perform all the
requested tasks?” of the MC type 1.
Another situation found in this process was related to the student's own technical ability to apply course
knowledge in practice: “I am not able to make user interaction and choice with the API, I changed the model, but
it is not easy to be able to make the application according to that design.” S56, May 9, 2020. In this situation, a
technical difficulty was exposed (which may even be impacting other students in the class) and provide the
299
lecturer with an opportunity for general feedback or try to support the student on that specific technical issue
(Knowledge of cognition).
Example 4 - Recommendations for adopting learning SCRL strategies

Another MC type 1 question was: “What are the steps you will take in the next two weeks?”. This enables the
lecturer to verify if students are outlining a plan according to what is expected for the upcoming project phase
or deviating from it and may recommend SCRL strategies accordingly (Regulation of cognition).
Example 5 – Motivational or clarification doubts

The level of confidence that the student has in relation to the work developed is an indicator that allows the
lecturer to perceive whether the student's confidence (high or low) matches its project status, comparing it to
his/her own evaluation. In this example, the answers to the question “What level of confidence do you feel about
the operation of the code of the demonstration application, considering the use of interfaces between components
as required?”, exposed that most of students placed their confidence at a “Regular” level (figure 4).
Figure 4: Response of LDS students for the 2019/2020 academic year to the question “What level of confidence do you feel
about the operation of the code of the demonstration application, considering the use of interfaces between components as
required?”.
The lecturer may elect to prioritize analysis of student who expressed a “Very Low” or “Low” level of confidence
on this aspect. The “why” follow-up question may enable understanding the reasons and allow the lecturer to
intervene accordingly. For instance, if the project work matches expectations, perhaps motivational or
confidence-building feedback is necessary, e.g.: “I went back and lowered the level. I still don't understand how
the example code works (…). I haven't gotten past the classic [approaches] yet.” S44, May 10, 2020.
Example 6 - Clarification of assessment criteria

The MC can also enable lecturers to understand whether a student's self-assessment is similar or different from
the lecturer’s assessment. If the student's self-assessment is very different from the lecturer’s, this awareness
may originate, for instance, feedback reminding or clarifying the assessment criteria. Comparing the students’
confidence level with their self-assessment level may enable the lecturer to focus on discrepant situations, such
as someone who has a “Low” level of confidence about the work developed, but self-assessed as “High”. For
example (table 2).
The lecturer can thus proceed, and possibly explore the reason for this imbalance (regular confidence on a very
high self-evaluation), and act in accordance with what is expected for that phase (Knowledge of cognition).
300
Table 2: Comparison between the student's confidence level and self-assessment regarding an MC.
Student Confidence level Why? Self-evaluation Why?
S1 Regular “I understood the concept of how to Very High “My demo application
implement error handling in the MVC conforms to the MVC style
model. I think I could have and is easily adaptable to
implemented a more adequate next steps.”
treatment, which I intend to do in the
next steps.”
Example 7 – Improvement for the next editions of the course:

The MC also allow the lecturer to visualize the evolution of the class, in each of the dimensions, verifying which
are the critical phases in which he/she will have to act. For example, the lecturer may perceive a reduction in
the students' answering of the MC, along with a predominance of students' perceptions of their learning
evolution as “Regular”. This may anticipate shortcomings that would only be exposed upon later project
delivery and enable pre-emptive action. Also, the lecturer may be able to identify phases where the class have
felt difficulties, by detecting sudden changes in the responses – this may expose issues with the syllabus, the
pedagogical planning, or other transversal problems (figure 5).
Figure 5: Example of how the MC type1 offer an overview of what happened in the class throughout the course.
The lecturer, when noticing these trends in the answers to the MC, may check for correlations with dropout
outcomes (remembering that in Distance Learning dropout rates tend to be high) and intervene. Not just
improving the planning/pedagogy in subsequent years, but for the ongoing year, if detected early enough. For
instance, acting in a motivational manner, or exploring project development status throughout the class and
provide ways to maintain student interest or recover shortcomings. If instead of overall lowering of response
rates it is rather a matter of specific students not wanting to address specific MC, the lecturer could try to
understand why.
5 Conclusions and final thoughts

The Metacognitive Challenges (MC) emerge as a strategy with the potential for the teaching staff to monitor
students' metacognitive learning processes, allowing lecturers to focus their class analysis and feedback effort,
and intervene adequately (regulation of cognition and knowledge of cognition) including: 1) Motivational
interventions; 2) Recommendations for adopting learning self-regulation and co-regulation strategies; 3)
Clarification of doubts due to errors of interpretation, reasoning or resolution; 4) Clarification of assessment
criteria; 5) Correction of content or task specification mistakes and/or aspects for pedagogical improvement in
subsequent editions of the course (self-reflection of the teacher on his/her pedagogical practices); 6)
Understand students' difficulties and help overcome them; 7) Clarification of aspects related to the syllabus
and concepts that may be misunderstood.
301
The awareness of lecturers allows them to make decisions that are critical to the student's learning success.
However, analysing MC requires a high amount of effort, so good planning and time management is
recommended. Future work will focus on the evaluation of the use of MC by teachers as a form of intervention,
verifying their effects on the student's learning process. It is important to develop technological solutions that
allow teachers to analyse the contents of MC efficiently and effectively.
6 Acknowledgments
This work was financially supported by National Funds through FCT – Fundação para a Ciência e a Tecnologia,
I.P., and CIDTFF (UIDB/00194/2020) - Universidade de Aveiro, Portugal, via Stimulus of Scientific Employment
– CEECIND/00986/2017 Individual Support 2017, and via project PTDC/CED-EDG/30040/2017. We would also
like to thank all students and teachers who collaborated on this research.
7 References
ACM & IEEE (2016). Computer Engineering Curricula 2016: Curriculum Guidelines for Undergraduate Degree Programs in
Computer Engineering. ACM & IEEE, New York.
Broadbent, J. (2017). Comparing online and blended learner's self-regulated learning strategies and academic performance.
The Internet and Higher Education, 33, 24-32. https://doi.org/10.1016/j.iheduc.2017.01.004
Dindar, M., Järvelä, S., & Järvenoja, H. (2020). Interplay of metacognitive experiences and performance in collaborative
problem solving. Computers & Education, 154, 103922. https://doi.org/10.1016/j.compedu.2020.103922
Flavell, J. H. (1979). Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. American
Psychologist 34 (10): 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Frazier, L. D., Schwartz, B. L., & Metcalfe, J. (2021). The MAPS model of self-regulation: Integrating metacognition, agency,
and possible selves. Metacognition and learning, 16(2), 297-318. https://doi.org/10.1007/s11409-020-09255-3
Garcia, R., Falkner, K., & Vivian, R. (2018). Systematic literature review: Self-Regulated Learning strategies using e-learning
tools for Computer Science. Computers & Education, 123, 150-163.
Hevner, A., & Chatterjee, S. (2010). Design science research in information systems. In Design research in information systems
(pp. 9-22). Springer, Boston, MA.
Kara, M., Erdoğdu, F., Kokoç, M., & Cagiltay, K. (2019). Challenges faced by adult learners in online distance education: A
literature review. Open Praxis, 11(1), 5-22. https://doi.org/10.5944/openpraxis.11.1.929
Kebritchi, M., Lipschuetz, A., & Santiague, L. (2017). Issues and challenges for teaching successful online courses in higher
education: A literature review. Journal of Educational Technology Systems, 46(1), 4-29.
Kim, J. Y., & Lim, K. Y. (2019). Promoting learning in online, ill-structured problem solving: The effects of scaffolding type
and metacognition level. Computers & Education, 138, 116-129. https://doi.org/10.1016/j.compedu.2019.05.001
Pedrosa, D., Fontes, M. M., Araújo, T., Morais, C., Bettencourt, T., Pestana, P. D., Morgado, L., & Cravino, J. (2021).
Metacognitive challenges to support self-reflection of students in online Software Engineering Education. In 2021
4th International Conference of the Portuguese Society for Engineering Education (CISPEE) (pp. 1-10). IEEE. doi:
10.1109/CISPEE47794.2021.9507230.
Pedrosa, D., Morgado, L., Cravino, J., Fontes, M. M., Castelhano, M., Machado, C., & Curado, E. (2020). Challenges
Implementing the SimProgramming Approach in Online Software Engineering Education for Promoting Self and
Co-regulation of Learning. In 2020 6th International Conference of the Immersive Learning Research Network (iLRN)
(pp. 236-242). IEEE. https://ieeexplore.ieee.org/document/9155183 .
Prather, J., Becker, B. A., Craig, M., Denny, P., Loksa, D., & Margulieux, L. (2020). What do we think we think we are doing?
Metacognition and self-regulation in programming. In Proceedings of the 2020 International Computing Education
Research Conference (ICER ’20). ACM, New York, NY, USA, 2-13. https://doi.org/10.1145/3372782.3406263
Sáiz-Manzanares, M. C., & Montero-García, E. (2015). Metacognition, self-regulation and assessment in problem-solving
processes at university. In Metacognition: Fundaments, Applications, and Trends (pp. 107-133). Springer, Cham.
Schraw, G., Gutierrez, A.P. (2015). Metacognitive Strategy Instruction that Highlights the Role of Monitoring and Control
Processes. In: Peña-Ayala, A. (eds) Metacognition: Fundaments, Applications, and Trends. Intelligent Systems
Reference Library, vol 76. Springer, Cham. https://doi.org/10.1007/978-3-319-11062-2_1
Schuster, C., Stebner, F., Leutner, D., & Wirth, J. (2020). Transfer of metacognitive skills in self-regulated learning: an
experimental training study. Metacognition and Learning, 15(3), 455-477.
Wallin, P., & Adawi, T. (2018). The reflective diary as a method for the formative assessment of self-regulated learning.
European Journal of Engineering Education, 43(4), 507-521. https://doi.org/10.1080/03043797.2017.1290585
302
Wengrowicz, N., Dori, Y. J., & Dori, D. (2018). Metacognition and meta-assessment in engineering education. In Cognition,
metacognition, and culture in STEM education (pp. 191-216). Springer, Cham.
303
PAEE/ALE’2022 Hands-on sessions
Submissions accepted for the PAEE/ALE’2022 hands-on sessions.
304
Do we, as educators, have to be neutral in our classroom?
Miguel Romá1
1
Signals, Systems and Telecommunication Group – SST, University of Alicante, Spain
Abstract
Using controversial issues in classroom can provide interesting learning opportunities. Having to face such situations can
make students being aware about how technical decisions can be biased by underlying personal values.
In this hands-on session attendees will face the question about the convenience, or not, of maintaining neutrality in front
of our students when including values and ethics in engineering.
Keywords: Politics and Education; Ethics and Values in Engineering Education; Promoting Global Awareness.
1 Introduction
The underlying idea of this hands-on session is well described by the topic of the conference, planting the
seed: promoting engineers with global awareness, which can be understood as a statement of intent. If we take
a picture of the global situation, this global awareness we try to seed in our students implies much more than
technical contents.
If we are aware of the social implications of engineering, any decision-making process can be extremely
complicated. As stated in the call for papers, ‘What we do as engineers and why we do it, is based on our
underlying values. However, rarely in engineering education is any consideration taken of our values, what they
are and where they come from’ (Nahar and Baillie, 2009). In this sense, when introducing ethics in engineering
in our courses we need to understand how our own values are affecting our decisions. From what perspective
should we approach this issue?
In the session attendees will face the question about the convenience, or not, of maintaining neutrality in front
of our students when including values and ethics in engineering.
This is, without doubt, a controversial topic. Making a fast search on the Internet, you can find teaching
professionals with three main points of view, against neutrality, pro neutrality or making it content dependant
(left, right and mid sections in figure 1). In short, the answers would be ‘yes’, ‘no’ or ‘it depends’.
Figure 1. Screenshot of some Internet posts reflecting the three main positions about the topic.
The session will be organized (among others) around the following questions:
• Is neutrality the most polite position?
• Can (or must) we share our political position in the classroom?
• Can we include ethics and values from a neutral position?
305
• Do we have to show a political position based on the values of our institution?
• Can role-play be an instrument to include political positions without making it personal?
• How to consider the effect of individual personality (introvert - extrovert)?
• What will happen with students not aligned with our opinion?
2 Activities
The main tasks will be performed in small groups. The session will be organized as follows:
• Topic presentation.
• Warm-up activity.
• Group activities (debates / role-play).
• Discussion.
• Conclusion.
3 Expected results
As the session topic does not have an answer, we will try to explore our own starting position and see if it
changes or not at the end. We will be expecting to have enriching debates.
4 References
Nahar and Baillie, 2009. Engineering values: an approach to explore values in education and practice. Research in Engineering Education
Symposium (REES 2009). Palm Cove. Australia.
(http://rees2009.pbworks.com/f/rees2009_submission_3.pdf)
Internet references:
https://www.nea.org/advocating-for-change/new-from-nea/education-political-neutrality-classroom-shortchanges-
students
https://www.cfmta.org/docs/essays/Music-education-is-a-political-endeavor-590775b170961.pdf
https://truthforteachers.com/truth-for-teachers-podcast/neutral-politics-in-the-classroom/
https://www.amle.org/do-teachers-remain-neutral-or-share-their-beliefs-with-students/
https://hechingerreport.org/should-teachers-be-apolitical/
https://www.adl.org/education/resources/tools-and-strategies/classroom-conversations/when-do-teachers-stay-neutral
https://www.tes.com/magazine/archive/why-teachers-must-remain-neutral-class-discussion
https://www.hiclark.com/blog/why-educators-should-remain-neutral-in-a-politically-charged-environment/
https://blogs.ibo.org/sharingpyp/2015/04/14/teacher-neutrality/
306
Game-based Learning vs Gamification: A Hands-On
Montse Farreras1, Jesús Armengol2, Pau Bofill3,Angels Hernández1
1
Escola Politècnica Superior d’Enginyeria de Vilanova i la Geltrú (EPSEVG) UPC-Barcelona Tech, Barcelona, Spain
2
Group of Applied Optics and Image Processing (GOAPI), Optics and Optometry Department, UPC-Barcelona Tech. Barcelona, Spain
3
Escola Tècnica Superior de Telecomunicació de Barcelona (ETSETB), UPC-Barcelona Tech, Barcelona, Spain
Abstract
An example of gamification is a contest where students get points for solving the usual exercises of the subject matter. An
example of game-based learning is an escape room where students get involved in studying and solving subject matter
problems to get the required hints to continue the game. In this sense, game-based learning is an instance of problem-
based learning. We propose a hand-on sesion where participants will get engaged into: first, a gamification activity; and
later a game-based learning (GBL) activity. They will be encouraged to notice the differences and make a distinction
between them. Afterwards, participants will be required to design a simple escape room situation involving problems for
their own courses.
Keywords: Active Learning; Engineering Education; Project Approaches; Game-based Learning; Gamification.
1 Introduction
Gamification has gained popularity in the last years, it is used in primary and secondary schools, as well as in
companies and also at the university (Call, 2021). Along with this growth in popularity the number of available
computer tools that facilitate the implementation of quizzes, competitions, simulations, WebQuests etc. has
also grown.
Play engages students and enhances learning, however not all sorts of games are equally fruitful. As in (Dave
Eng, 2019), we make the distinction between gamification and Game-based learning (GBL). An example of
gamification is a contest where students get points for solving the usual exercises of the subject matter. An
example of game-based learning is an escape room where students get involved in studying and solving
subject matter problems to get the required hints to continue the game. In this sense, game-based learning is
an instance of problem-based learning as it complies with the characteristics of PBL (Graaff, E. 2003).
The work presented here has been designed as a part of a Teaching Innovation Project to introduce Game
Based Learning (GBL) into first year subjects with the aim of fostering (cooperative) learning, engaging students,
and providing a positive social dimension . It is an interdisciplinary project that involves 3 different schools of
Engineering education in our University where GBL has been introduced in 4 different subjects.
Our goal is to design games from which the students cannot escape without learning (Bofill 2007). As
McGonigal proposes (McGonigal 2011), an escape room seeks to have an epic mission, a clear goal, immediate
feedback, another chance to prove it and a positive social dimension.
From a general point of view, we can say that learning takes place in five stages [Bofill, 2007]. Namely:
motivation, information retrieval, understanding, application (or practice) and feed-back. As we will see, GBL,
reinforces the autonomous realization of each of these stages. Figure 1 shows a screenshot of the Escape
Room that will be used in the workshop: A bridge over the river Splash
307
Figure 1. Screenshot of our escape room: a bridge over the river splash.
2 Activities
In the hands-on session we will ask the participants to work in teams and we will need a computer or laptop
to be available for each team.
• First participants will engage in a gamification activity: they will be required to solve a simple
questionnaire, and we will monitor their progress using Socrative tool (socrative.com).
• Second we will ask them to participate in a game-based learning activity, an escape room implemented
with genially https://genial.ly/es/ where they will be required to solve simple computer programming
exercises to obtain the keywords to progress on the game and exit the escape room. Notice that we
do not expect the participants to know computer programming so they will be allowed to use other
means (aka. cheating) like google to provide the answers to the riddles. The students however are
required to write the programs to solve the questions.
• Third, we will ask participants to reflect on both experiences and notice the difference between
gamification (the first) and GBL (the second).
• Fourth, participants will be invited to design a simple escape room situation involving problems for
their own courses.
• Finally we will end the session with an open debate, where we may discuss topics such as: evaluation,
comparing game-based learning versus gamification strategies, strategies and tools for applying such
gaming techniques in the classroom and online. The debate will be aimed to formulate conclusions
and get feedback and ideas to improve our game-based activities.
Inspirational questions to open debate:

• Which sort of gaming experience provided higher motivation?
• What differences can you see between gamification and GBL?
• Think of ways to introduce GBL into your subject
• Regarding the quality or level of undertanding, which experience do you think provides a deeper level?
308
3 Expected results
We expect a fruitful discussion and each participant should go home with some ideas on how to introduce GBL
into his subjects. We also aim to have fun and spread the joy of learning through play. We will evaluate the
activity with a survey (using the same Socrative tool).
4 Instructions for Participants

The activity is open to everybody who has an interest on the topic. Participants will be required to bring a
laptop and a set of earphones, or otherwise it should be made available a computer per team to be able to
carry on the session.
They should also come with an open mind and willing to have fun.
5 Acknowledgement
d’innovació docent 2021 (Acord CG/2021/02/34, de 9 d'abril de 2021) and by Escola Tècnica d’Enginyeria de
Telecomunicació (upc telecos-Barcelona Tech).
6 References
Bofill,P. (2007) Designing activities from which the student cannot escape without learning. Active Learning in Engineering Education
Workshop
Call, T., Fox, E., & Sprint, G. (2021). Gamifying Software Engineering Tools to Motivate Computer Science Students to Start and Finish
Programming Assignments Earlier. IEEE Transactions on Education, 64(4), 423–431. https://doi.org/10.1109/TE.2021.3069945
Dave Eng, 2019. https://medium.com/@davengdesign/game-based-learning-vs-gamification-f44e5b09d808
Graaff, E. d., & Kolmos, A. (2003). Characteristics of Problem–Based Learning. International Journal of Engineering Education, 19(5), 657-
662.
McGonigal, Jane (2011). The reality is broken: Why Games Make Us Better and How They Can Change the World. The Penguin Press, New
York.
309
Engineers working as a team: socio-emotional and intercultural
skills for multicultural environments
Elisandra Martins1, Valquíria Villas-Boas1
1
University of Caxias do Sul, Caxias do Sul, Brazil
Abstract
This workshop consists of an activity that aims to help engineering students to develop socio-emotional and intercultural
skills. Through a usual situation for engineers at the beginning of their career, the participant will have the opportunity to
experience a situation in which it will be necessary to work as a team, communicate, negotiate, and deal with colleagues of
different nationalities.
Keywords: Teamwork skills; Socioemotional and intercultural skills; Engineering Education; Active Learning.
1 Introduction
Engineering students need more than technical knowledge to face the demands of the 21st century (Downey
et al., 2006; Passow, 2012). They also need to develop lifelong learning skills, such as socio-emotional and
intercultural skills. Socio-emotional skills will enable future engineers to understand the social, moral and
human values of a society. This will help them to work as a team, negotiate, manage, empathize, and deal
better with colleagues, customers and the general public. Developing cross-cultural skills will provide the future
engineer with the competence needed to work on complex global projects, immigrate to find work, collaborate
and compete with colleagues of other nationalities, and work for international companies. Engineering schools
are exploring ways to support and develop socio-emotional and intercultural competences in students.
However, to date, studies on this subject are not as abundant as one would like (Ndubuisi et al., 2020).
Seat, Parsons & Poppen (2001) brought that there is no secret that the quality of interpersonal, communication,
and teaming skills in engineering graduates—termed performance skills—is of concern to both industry
employers and engineering educators. These skills include communication abilities, interpersonal interaction,
conflict mediation, team performance, understanding of technical culture, and sensitivity toward diverse
populations due to race, ethnicity, gender, and socio-economic standing.
Besides learning how to work in teams, engineering students need to learn how to establish a collaborative
culture in their future workplace and how to deal/communicate/negotiate with people from different countries
and cultures (Josefsson, 2010; Rico-García & Fielden Burns, 2019).
2 Activities
The activities in this workshop are:
(i) Getting to know each other.
(ii) Dividing the participants in pairs.
(iii) Developing the activity:
Part 1: a large company sends two engineers (one from country A and one from country B) to
work on the construction of two bridges and an underwater tunnel between Cardiff and Weston-
super-Mare in the United Kingdom. For that, a two-bedroom apartment will be made available
where they will live for a year. The apartments have basic items, but most of the furniture is not
available. Each of the engineers will receive a sum of 2000 euros to equip and furnish the
apartment, considering their personal and cultural characteristics.
310
Part 2: upon arriving in the UK, the two engineers are informed that there has been a logistical
problem in the allocation of the apartments, and they will have to share the apartment with two
more engineers from country C and country D. Therefore, they will have to divide themselves
into pairs that will occupy each dormitory and will have to redo the shopping list, because due
to this relocation, each of the four engineers will receive only 1200 euros.
(iv) Presenting the results obtained by the different teams.
(v) Reflecting on the results obtained by the different teams.
3 Material needed for the workshop

Each participant must bring their notebook to the workshop. The workshop organizers will provide office
supplies.
4 Expected results
By the end of this workshop, we expect that the participants can recognize that besides having skills to work
in teams, future engineers need to be prepared to share a living space with co-workers, negotiate, manage,
empathize and deal better with colleagues, work on complex global projects, immigrate to find work,
collaborate and compete with colleagues of other nationalities, and work for international companies, among
other skills that are important for academic and professional life.
5 References
Downey, G. L., Lucena, J. C., Moskal, B. M., Parkhurst, R., Bigley, T., Hays, C., ..., & Nichols‐Belo, A. (2006). The globally
competent engineer: Working effectively with people who define problems differently. Journal of Engineering
Education, 95(2), 107-122.
Josefsson, T. (2010). Intercultural Competence in Engineering Education. 7th CDIO.
http://wvvw.cdio.org/files/document/file/T2A_Paper_2.pdf
Passow, H. J. (2012) Which ABET Competencies Do Engineering Graduates Find Most Important in their Work? Journal of
Engineering Education, 101(1), 95–118.
Seat, E., Parsons, J. R., & Poppen, W. A. (2001). Enabling engineering performance skills: A program to teach communication,
leadership, and teamwork. Journal of Engineering Education, 90(1), 7-12.
Ndubuisi, A., Khan, R., Marzi, E., & Edun, O. (2020, October). A KCI Approach to Promote Intercultural Competencies for
International Virutal Engineering Student Teams (InVEST). In 2020 IEEE Frontiers in Education Conference (FIE) (pp.
1-5). IEEE..
Rico-García, M., & Fielden Burns, L. V. (2019). Intercultural communication in engineering studies: a key competence in
global labour markets. European Journal of Engineering Education, 45(6), 833-853.
311
Problem Based Learning – Making Blended Learning work!
Jens Myrup Pedersen1
1
Department of Electronic Systems, Aalborg University, Denmark
Abstract
The Covid-19 situation forced many educational institutions to change their educational offer into either blended or fully
online formats. While classical lectures and other “passive learning” methods can be quite easily adapted to this new format
– given that technology and connectivity is in place – it is more challenging when it comes to various active learning
methods such as Problem Based Learning. However, for Problem Based Learning, the blended format also has strong
potentials: It can open for more flexible ways of working together across physical distances, and as such serve as an enabler
for projects based on international collaboration, and it can generally increase the flexibility and accessibility of education.
This workshop will focus on how Problem Based Learning projects can be carried out in a blended setting, and walk through
the different phases: Problem identification, group formation, group collaboration, supervision, and assessment. For each
phase there will be a short introductory presentation, followed by group discussions on challenges in carrying out this
phase in a blended setting, and how these challenges can be overcome.
Keywords: Active Learning; Engineering Education; Project Approaches.
1 Introduction
Problem Based Learning (PBL) is known as an efficient and motivating way of learning in engineering education
that not only provide the learners with technical skills, but also supports the learning of competences related
to teamwork, project management and other transversal skills (Kolmos et al., 2004). In the recent years, various
efforts have been taken to also explore international collaboration, both at a European level with projects such
as EPIC (Pedersen et al., 2019) and COLIBRI (Pedersen et al., 2016), and at global levels e.g. (Britze et.al., 2021).
While these projects are all based on blended learning – with a large virtual component due to the physical
distances – one common learning point is the importance of the physical learning activities and design hereof:
Kick-off seminars, possibly with added extra mobilities, are important for both content discussions and for
establishing good dynamics within and between student groups.
With the lockdowns during Covid-19, remote and virtual learning spaces became the “new normal”, at least for
a while. This added a completely different scale to the experiments with PBL and virtual collaboration, where it
was necessary to carry out all phases of the PBL projects virtually. With the improvement of the Covid-19
situation most universities are back to physical teaching. However, with the experiences from extended periods
of fully online learning activities in mind, the good question is: How can we use these experiences to improve
the blended learning experiences even after Covid? It is interesting to investigate how it can support projects
which by nature need virtual components (such as international projects), but also how it can be used in other
settings, for example to make education more flexible and accessible.
In this workshop, the focus will specifically focus on the blended settings, including the selection of virtual vs.
physical components. The workshop will cover the following phases of a student project:
• Identifying problems in collaboration with internal and external stakeholders.
• Group formation, where students form groups and choose problems.
• Group collaboration.
• Supervision, including supervisor meetings.
• Assessment and examinations.
The purpose of the workshop is to share experiences from both the author and other participants, and based
on these enable the participants to design blended learning settings for PBL projects, with a particular focus
on how to choose and design the virtual and physical components, to reflect on how to benefit from the
312
opportunities offered by the blended settings, and how to identify and deal with some of the most challenging
aspects of blended learning – all in all, enabling the participants to design blended learning that works. The
workshop is related to the conference theme of Active Learning and ICT support, yet also addressing the theme
of Innovative experiences in engineering education.
2 Activities
The session will be hands-on, and the participants will mainly work in smaller groups of 3-4 people. The focus
will be on the phases described above (identifying problems, group formation, group collaboration,
supervision, assessment).
After a 5-minute icebreaker in the groups, for each of the five phases the following is done:
• 2-minute introduction to the phase in plenum, including a short review of existing experiences.
• 8 minutes discussion in groups, where the participants sketch 1-2 ways to carry out the phase in a
blended setting. This is documented in terms of 3 key-words per group per question, submitted on a
Padlet.
Each group will eventually select the keyword, which is found to describe the most challenging aspect.
Eventually there will be 25 minutes where groups are paired (so 6-8 people together) and discussing these
aspects with each other. During the last 10 minutes the facilitator will bring up highlights from the discussions
in each group.
The participants are requested to bring laptop, smartphone, or tablet in order to be able to contribute to the
Padlet documentation.
3 Expected results
At the end of the workshop, the participants will be better prepared to facilitate PBL projects in blended
settings, and the Padlet printouts will serve as concrete sources of inspiration. The identification of challenges
can also support further research and experiments in the field.
4 References
Britze, D., Torres, M.H., Jensen, J.V., Melo, L.C.Q., Lima, T.B.O., Pedersen, J.M., da Silva, J.M., Monteiro, S.B.S., Gomes,
P.C.R. (2021). Sustainability-focused international PBL project: Rethinking digital education for individuals of low socioeconomic
status . International Symposium on Project Approaches in Engineering Education (PAEE), pp. 103-112.
Kolmos, A., Fink, F., & Krogh, L. (2004). The Aalborg PBL Model: Progress, Diversity and Challenges. Aalborg University Press.
Krogh Hansen, K., Otrel-Cass, K., Guerra, A., & Dahms, M-L. (2014). Good Examples Catalogue: “Problem Based Learning and Sustainability
in Engineering and Science Education – Practice and Potential”. Faculty of Engineering and Science, Aalborg University.
Pedersen, J. M., Kuran, S., Frick, J., & Mank, L. (2016). Colibri: An International Blended Learning Experience based on Real-World Problems.
Proceedings of the PAEE/ALE’2016, 8th International Symposium on Project Approaches in Engineering (s. 259-268). University of
Minho. International Symposium on Project Approaches in Engineering Education
(PAEE) http://paee.dps.uminho.pt/proceedingsSCOPUS/PAEE2016+ALE%20proceedings.pdf
Pedersen, J. M., Kirkova, M., Kuladinithi, K., & Janssen, N. V. H. (2019). EPIC: Making Multinational Student Projects Happen. International
Symposium on Project Approaches in Engineering Education (PAEE), 9, 219-228. https://paeeale.esprit.tn/wp-
content/uploads/2019/06/PAEE_ALE_2019_PROCEEDINGS.pdf
313
Haakins: Cyber Security Training with Gamification
Jens Myrup Pedersen1
1
Department of Electronic Systems, Aalborg University, Denmark
Abstract
With increased digitalization, our societies have become more vulnerable to cyber attacks, and we are already seeing both
criminal groups and nation states being very active in the cyber domain. This calls for action in order to better protect our
societies, companies and people against these attacks. One of the main challenges with respect to this is the lack of skilled
people -a gap that is expected to increase in the coming years. To increase the interest in cyber security among young
people, and to increase the competence level among young people, Aalborg University together with a number of partners
have developed the training platform Haaukins. It can easily be setup by teachers, who can select which challenges the
students will work on - a typical session would contain around 10 challenges out of the 180 currently provided. The students
are then presented with these challenges along with a virtual lab, which provides a closed and secure environment for
working with cyber security. Upon solving challenges, the students can gain points and compete with their classmates. The
workshop provides an overview of the platform, and then takes the participants through a typical training session which
require no previous experience with computers, programing, or cyber security. The session consists of a short introduction
to the platform, followed by a mix of short presentations, demonstrations, and independent work on challenges. After the
session, a discussion on how the platform can be integrated in learning activities is facilitated. A particular focus in the
discussion is how more diversity among the students can be achieved.
Keywords: Active Learning; Gamification; Cyber Security; Engineering Education.
1 Introduction
With our societies becoming increasingly digitalised, we are also becoming more exposed and vulnerable to
cyber-attacks. According to FBI’s Internet Crime Complaint Center, 2021 gave new records both in terms of
number of cases (847,376) and the total losses ($ 6.9 billion). These numbers have been consistently increasing
during the last years – in 2017, the total number of cases was 301,580, and the losses $1.4 billion (FBI, 2021).
One of the main problems for companies and societies is the lack of cyber security professionals. According to
the (ISC)2 2021 Cybersecurity Workforce Study ISC2, 2021) there is – despite of many efforts across the global
– a workforce gap of 2.72 million people.
One way to attract more young people into working with cyber security is to introduce cyber security at all
levels in the educational system is to use cyber security training platforms, which can increase awareness by
providing fun and interesting hands-on challenges – often with an element of gamification (Kjorveziroski et al.,
2020). One such platform is the Haaukins platform developed by Aalborg University (Panum et al, 2019).
Haaukins makes it possible for teachers to setup virtual labs in a highly automated fashion, and in a way that
is easily accessible for both students and teachers as everything runs in a browser and no special software is
required. The teacher simply selects the number of labs, the relevant challenges for that class – and then
students can sign up and compete for points among other students in the same class. The gamification is
supported by leader boards and scoreboards, so progression can be tracked by all participants throughout the
session. However, as the uptake of Haaukins is increasing, and as more teachers with different backgrounds –
sometimes with little experience with cyber security – it is becoming even more important that the system is
not only technically user friendly, but also that the pedagogical approaches are clear, and that both teachers
and students are supported by the platform together with other online resources (Menecozzi et al, 2021).
This pedagogical challenge is also the starting point for this hands-on workshop, where we will explore and
discuss the use of Haaukins with an emphasis on good pedagogical practices.
314
2 Activities
The session is divided in two parts, each expected to last around 45 minutes.
The first part is a presentation of the Haaukins platform, where the participants also get the chance to solve
challenges and, in this way, get a good understanding of how the platform works. Participation requires no
previous experience with computers, programing, or cyber security. The first part contains the following
elements:
• Presentation of the Haaukins platform (5 minutes)
• Demonstration and hands-on work on two challenges, organised like a typical training session with (1)
introduction to the concept, (2) introduction to the challenge, (3) participants working two and two on
the challenge, (4) demonstration of the solution. 20 minutes is allocated to each challenge.
• More challenges will be made available for the participants, so in case there are participants with
domain knowledge, they can move faster, and all participants have the possibility to try out more
challenges after the session.
The second part is a discussion session on how the platform can be integrated in learning facilities. The
participants are divided into groups, where each group discuss one of the following topics:
• How to integrate hands-on training into cyber security teaching in the best way?
• How can hands-on training be integrated into cyber security teaching in a cirruculum, in different
programs with students from different backgrounds?
• How to enable the teachers to use the platform, given often limited time constraints?
• How can Haaukins be used in promoting more diversity among cyber security students, e.g. attracting
more women to the field?
For the second part, the groups will have 30 minutes to discuss their topic, and then no more than 3 minutes
per group to present the most interesting findings. The findings for each group will be documented in a Padlet.
The participants are requested to bring a laptop to be able to do the challenges, but no special software is
required. The laptop will also be used for documenting the group discussions in the Padlet
3 Expected results
There are two main results of the workshop:
• The participants will be prepared to facility cyber security trainings using the Haaukins platform.
• We will generate ideas and knowledge about how to improve learning, teaching, and training within
cyber security, including achieving more diversity and better gender balance in the field.
4 References
FBI (2021) Internet Crime Report 2021. Federal Bureau of Investigation, Internet Crime Complaint Center. Available at:
https://www.ic3.gov/Media/PDF/AnnualReport/2021_IC3Report.pdf.
ISC2 (2021) 2021 Cybersecurity Workforce Study. Available at https://www.isc2.org//-/media/ISC2/Research/2021/ISC2-Cybersecurity-
Workforce-Study-2021.ashx.
Kjorveziroski, V., Mishev, A., Filiposka, S. (2020). Cybersecurity Training Platforms Assessment. In: Dimitrova, V., Dimitrovski, I. (eds) ICT
Innovations 2020. Machine Learning and Applications. ICT Innovations 2020. Communications in Computer and Information
Science, vol 1316. Springer, Cham. https://doi-org.zorac.aub.aau.dk/10.1007/978-3-030-62098-1_15
Mennecozzi, G. M., Hageman, K. D., Panum, T. K., Türkmen, A., Mahmoud, R-V., & Pedersen, J. M. (2021). Bridging the Gap: Adapting a
Security Education Platform to a New Audience. I T. Klinger, C. Kollmitzer, & A. Pester (red.), 2021 IEEE Global Engineering Education
Conference (EDUCON) (s. 153-159). [9453985] IEEE. 2021 IEEE Global Engineering Education Conference
(EDUCON) https://doi.org/10.1109/EDUCON46332.2021.9453985
Panum, T. K., Hageman, K. D., Pedersen, J. M., & Hansen, R. R. (2019). Haaukins: A Highly Accessible and Automated Virtualization Platform
for Security Education. I M. Chang, D. G. Sampson, R. Huang, A. S. Gomes, N-S. Chen, I. I. Bittencourt, K. Kinshuk, D. Dermeval, & I.
M. Bittencourt (red.), 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT) (s. 236-238). [8820918]
IEEE. International Conference on Advanced Learning Technologies (ICALT) https://doi.org/10.1109/ICALT.2019.00073
315

Paee Ale 2022 Proceedings

Uploaded by

Copyright:

Available Formats

Paee Ale 2022 Proceedings

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Paee Ale 2022 Proceedings

Uploaded by

Copyright:

Available Formats

TITLE

International Symposium on Project Approaches in Engineering Education

Proceedings of the PAEE/ALE’2022, International Conference on Active Learning in Engineering Education

This is a digital edition.

Miguel Romá & Valquíria Villas-Boas

Scientific Committee Chairs

Student Session Chairs

PAEE/ALE’2022 Organising Committee

Montse Farreras, Pau Bofill, Jesús Armengol

PAEE Steering Committee

ALE Steering Committee

Miguel Romá, University of Alicante, Spain (chair)

Title of the keynote – Science curricula in the time of resource scarcity

16:00-16:30 Coffee break (Inner courtyard)

18:00-20:00 PAEE/ALE Challenge + Welcome cocktail

11:00-11:30 Coffee break (Inner courtyard)

11:30-13:00 Jens Myrup Pedersen Elisandra Martins, Valquiria Villas-Boas

16:00-16:30 Coffee break (Inner courtyard)

Session Chair: Jens Myrup Pedersen Session Chair: Vicente Morell

Bruna Andrade Machado, Bárbara Hotta, Gabriel Lino Garcia, Fernando

(On-site) Isabel B. A. de Souza, Bruno L. Borba, Alexandre S. C. Costa,

Matheus Ricardo de Avila Santos, Maria Valéria Araújo, John Burgoyne,

11:00-11:30 Coffee break (Inner courtyard)

13:00-13:30 Closing Ceremony (Room A)

The event has three type of submissions in English:

PAEE/ALE’2022 Submissions ................................................................................................................... 1

Email: [email protected], [email protected], [email protected], [email protected],

Keywords: Case Method. Global Prosperity. Design Science.

1.1 Theoretical elements

Figure 1. Theory of flow source: Csikszentmihalyi, 1990.

2.1 Serious game Reis Magos

2.2 Structuring of the artifact.

3 Describing the structural elements of the game

Figure 3. Businessperson card. Source: Venable (2006).

Figure 5. Resource card.

Figure 7. Luck and bad luck cards.

Figure 9. Epilogue card 3.

José Ignacio Pagán1, Pablo Ortiz1, Isabel López1

Email: [email protected], [email protected], [email protected]

18-19 19-20 20-21

Pablo Ortiz1, José Ignacio Pagán1, Isabel Lopez1

Email: [email protected], [email protected], [email protected]

Keywords: Active Learning; Engineering Education; Entrepreneurship; Start-Up; Soft Skills.

2.2 Description of the experience

•New energy •Optimized

•Technological •Trade facilitation

Figure 1: Challenges in which the different projects will be included

Figure 2 Steps of the entrepreneurial project

Figure 3: Business Model Canvas adapted for the entrepreneurial project.

1 10 Problem characterization and data collection with a supporting methodology

5 10 Teamwork, effective management and learning through failure.

6 30 Appropriate, innovative and creative technical solution

7 20 Effective presentation and communication of the proposed solution or project

Creativity and 4.0

José Ignacio Pagán1, Isabel López1, Pablo Ortiz1

Email: [email protected], [email protected], [email protected]

Keywords: Engineering Education; Video Tutorial; Computer Lesson; Project-Based Learning.

2.2 Description of the experience

Figure 1: Sheet describing the video tutorials

Do you consider the use of PBL + video Developing an example in