IEEE INTERNATIONAL CONFERENCE ON AUTOMATION (ICA)

XXVI CONGRESO DE LA ASOCIACIÓN CHILENA DE CONTROL AUTOMÁTICO (ACCA)

Effectiveness of Generative
Artificial Intelligence in
learning programming to higher
education students

Rafael Mellado
Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile.
[email protected]
Claudio Cubillos
Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile.
[email protected]
Giovanni Ahumada
Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile.
[email protected]
 INTRODUCTION

• Educational context and challenges in teaching programming:

⎼ High dropout rate in programming courses [1].
⎼ Difficulty in understanding abstract concepts [2].
⎼ Traditional methods could be more interactive [3].
⎼ Lack of personalized tools for students [4].
⎼ Frustration due to debugging errors and lack of practice [5].

[1] P. Kinnunen and L. Malmi, “Why students drop out CS1 course?,” in Proceedings of the second international workshop on Computing education research, in ICER ’06. New York, NY, USA: Association for Computing Machinery, Sep. 2006, pp.
97–108. doi: 10.1145/1151588.1151604.
[2] J. Sorva, “Notional machines and introductory programming education,” ACM Trans. Comput. Educ., vol. 13, no. 2, p. 8:1-8:31, Jul. 2013, doi: 10.1145/2483710.2483713.
[3] A. Robins, J. Rountree, and N. Rountree, “Learning and Teaching Programming: A Review and Discussion,” Computer Science Education, vol. 13, no. 2, pp. 137–172, Jun. 2003, doi: 10.1076/csed.13.2.137.14200.
[4] L. E. Winslow, “Programming pedagogy—a psychological overview,” SIGCSE Bull., vol. 28, no. 3, pp. 17–22, Sep. 1996, doi: 10.1145/234867.234872.
[5] R. McCauley et al., “Debugging: a review of the literature from an educational perspective,” Computer Science Education, vol. 18, no. 2, pp. 67–92, Jun. 2008, doi: 10.1080/08993400802114581. 2
 INTRODUCTION

• Current state:
⎼ The need for more studies on computer programming education in Latin
American countries [6], considering teaching methodologies [6], the digital divide
[7], cultural relevance [8], and technological integration [9].

[6] F. Ferrero, “Learning to Program Computers: Systematic Literature Review and Vygotskian Discussion,” in 2019 XIV Latin American Conference on Learning Technologies (LACLO), Oct. 2019, pp. 182–189. doi:
10.1109/LACLO49268.2019.00040.
[7] J. Simmonds, F. J. Gutierrez, C. Casanova, C. Sotomayor, and N. Hitschfeld, “A Teacher Workshop for Introducing Computational Thinking in Rural and Vulnerable Environments,” in Proceedings of the 50th ACM Technical Symposium on
Computer Science Education, in SIGCSE ’19. New York, NY, USA: Association for Computing Machinery, Feb. 2019, pp. 1143–1149. doi: 10.1145/3287324.3287456.
[8] J. Carroll-Miranda et al., “This is What Diversity Looks Like: Making CS Curriculum Culturally Relevant for Spanish-speaking Communities,” in Proceedings of the 50th ACM Technical Symposium on Computer Science Education, in SIGCSE ’19.
New York, NY, USA: Association for Computing Machinery, Feb. 2019, pp. 647–648. doi: 10.1145/3287324.3287339.
[9] X. B. Olabe and M. E. O. Parco, “Integración de Pensamiento Computacional en Educación Básica. Dos Experiencias Pedagógicas de Aprendizaje Colaborativo online,” Revista de Educación a Distancia (RED), vol. 20, no. 63, Art. no. 63, Apr. 3
2020, doi: 10.6018/red.409481.
 HIGHLIGHTS FROM PREVIOUS STUDIES

• GenAI (Generative Artificial Intelligence):

⎼ Personalization of learning and instant feedback [10].
⎼ Generation of explanations and practical examples [11].
⎼ Potential to increase student motivation [12].

[10] Z. Bahroun, C. Anane, V. Ahmed, and A. Zacca, “Transforming Education: A Comprehensive Review of Generative Artificial Intelligence in Educational Settings through Bibliometric and Content Analysis,” Sustainability, vol. 15, no. 17, Art. no.
17, Jan. 2023, doi: 10.3390/su151712983.
[11] R. Yilmaz and F. G. Karaoglan Yilmaz, “The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation,” Computers and Education: Artificial Intelligence,
vol. 4, p. 100147, Jan. 2023, doi: 10.1016/j.caeai.2023.100147.
[12] D. Andrade-Girón et al., “Generative artificial intelligence in higher education learning: A review based on academic databases,” Iberoamerican Journal of Science Measurement and Communication, vol. 4, no. 1, Art. no. 1, Apr. 2024, doi:
10.47909/ijsmc.101.
4
 HIGHLIGHTS FROM PREVIOUS STUDIES

• Active learning with videos:

⎼ Educational videos improve comprehension and reduce the knowledge gap [13].
⎼ Greater student interaction and engagement [14].
⎼ Allows continuous content review as needed [15].

[13] A. Jones, M. Aryal, J. Selby, and S. Adjei, “Creating Instructional Elementary Programming Videos for Use in an Adaptive Testing and Remediation System,” in Proceedings of the 55th ACM Technical Symposium on Computer Science
Education V. 2, in SIGCSE 2024. New York, NY, USA: Association for Computing Machinery, Mar. 2024, p. 1882. doi: 10.1145/3626253.3635397.
[14] S. Freeman et al., “Active learning increases student performance in science, engineering, and mathematics,” Proceedings of the National Academy of Sciences, vol. 111, no. 23, pp. 8410–8415, Jun. 2014, doi: 10.1073/pnas.1319030111.
[15] J. Liebenberg and S. van der Linde, “Factors Determining the Success of Online Learning Videos for Programming,” in ICT Education, H. E. Van Rensburg, D. P. Snyman, L. Drevin, and G. R. Drevin, Eds., Cham: Springer Nature Switzerland,
2024, pp. 48–63. doi: 10.1007/978-3-031-48536-7_4.

5
RESEARCH OBJECTIVES

• Evaluate the effectiveness of GenAI compared to traditional methods in

programming learning:
⎼ Assess the effectiveness of GenAI vs. videos on academic performance.
• Measure changes in the perception of usefulness, ease of use, and
student motivation:
⎼ Compare the perception of usefulness and ease of use (TAM).
⎼ Measure student satisfaction and motivation (HMSAM).

6
RESEARCH OBJECTIVES

• Research questions:
⎼ Does the academic performance of university students in programming improve
when using a generative artificial intelligence model compared to active learning
based on videos?
⎼ How does the use of generative artificial intelligence influence the perception of
ease of use and perceived usefulness according to the Technology Acceptance
Model compared to active learning methods using videos?
⎼ How does the use of generative artificial intelligence affect student satisfaction
and motivation in the context of the Human Motivation System for Assessment
Model compared to active learning methods using videos?
7
METHOD

• Setup:
⎼ Test subjects: 40 computer engineering students.
⎼ Experimental design: Distribution into two groups (control and experimental).
⎼ Tools used: Educational videos vs. Google Gemini 1.5.
⎼ Questionnaires used: TAM and HMSAM.
• Learning objectives:
⎼ LO1: Identify the fundamental aspects of singly and doubly linked lists to
develop algorithmic solutions for mathematical and real-life problems.
⎼ LO2: Use pointers in singly and doubly linked lists to implement functions and
modularize algorithmic problems.
8
METHOD

9
METHOD

10
RESULTS

• Performance:

11
RESULTS

• Perceived usefulness and ease of use (TAM):

12
RESULTS

• Satisfaction and motivation (HMSAM):

13
RESULTS

• Interpretation of the results:

⎼ There were no significant differences in academic performance (p > 0.05).
⎼ Perception of usefulness and ease of use were similar in both groups (TAM).
⎼ Satisfaction and motivation showed no significant differences (HMSAM).
⎼ Both methods, GenAI and videos, were equally effective.
• Implications for education:
⎼ GenAI is a viable tool that complements traditional methods.
⎼ It can personalize learning and provide instant feedback.
⎼ Teacher training is required for its effective implementation.
⎼ It increases the diversity of pedagogical strategies.
14
RESULTS

• Study Limitations:
⎼ Small sample size (n=40).
⎼ Short experiment duration (12 days).
⎼ Students from a single profile (Computer Engineering).
⎼ Results are not generalizable to other populations.

15
CONCLUSIONS

• Summary of findings:
⎼ GenAI and videos are equally effective in learning.
⎼ There were no significant differences in academic performance.
⎼ Perception of usefulness, ease of use, satisfaction, and motivation were similar.
⎼ Both methods can enhance programming learning.

16
CONCLUSIONS

• Recommendations for future studies:

⎼ Expand the sample size and diversity of students.
⎼ Conduct longitudinal studies to assess long-term impact.
⎼ Explore the combination of GenAI with other pedagogical methods.
⎼ Evaluate the effectiveness of GenAI in different educational disciplines.

17
EFFECTIVENESS OF GENERATIVE ARTIFICIAL INTELLIGENCE
IN LEARNING PROGRAMMING TO HIGHER EDUCATION
STUDENTS
Prof. Dr. (c) Rafael Mellado S.
[email protected]
Pontificia Universidad Católica de Valparaíso

© 2024 Rafael Mellado.

All rights reserved.
The reproduction, distribution, or use of these slides is strictly prohibited without the express written
permission of Rafael Mellado.
Disclaimer: This presentation contains material generated by artificial intelligence.

18