Computers in Human Behavior: Artificial Humans 1 (2023) 100005

Contents lists available at ScienceDirect

Computers in Human Behavior: Artificial Humans

Augmented intelligence in programming learning: Examining student views

A R T I C L E I N F O A B S T R A C T

Keywords: With the diversification of generative artificial intelligence (AI) applications, the interest in their use in every
Generative artificial intelligence segment and field of society in recent years has been increasing rapidly. One of these areas is programming
ChatGPT learning and program writing processes. One of the generative AI tools used for this purpose is ChatGPT. The use
Programming
of ChatGPT in program writing processes has become widespread, and this tool has a certain potential in the
Programming learning
Student opinions
programming process. However, when the literature is examined, research results related to using ChatGPT for
this purpose have yet to be found. The existing literature has a gap that requires exploration. This study aims to
analyze the students’ perspectives on using ChatGPT in the field of programming and programming learning. The
study encompassed a cohort of 41 undergraduate students enrolled in a public university’s Computer Technology
and Information Systems department. The research was carried out within the scope of the Object-Oriented
Programming II course for eight weeks. Throughout the research process, students were given project assign­
ments related to the course every week, and they were asked to use ChatGPT while solving them. The research
data was collected using a form consisting of open-ended questions and analyzed through content analysis. The
research findings revealed both the advantages and disadvantages of ChatGPT usage, as perceived by the stu­
dents. The students stated that the main benefits of using ChatGPT in programming learning are providing fast
and mostly correct answers to questions, improving thinking skills, facilitating debugging, and increasing self-
confidence. On the other hand, the main limitations of using ChatGPT in programming education were get­
ting students used to laziness, being unable to answer some questions, or giving incomplete/incorrect answers,
causing professional anxiety in students. Based on the results of the research, it can be said that it would be useful
to integrate generative AI tools into programming courses considering the advantages they provide in pro­
gramming teaching. However, appropriate measures should be taken regarding the limitations it brings. Based on
the research findings, several recommendations were proposed regarding the integration of ChatGPT into
lessons.

1. Introduction people’s daily lives, the concept of augmented intelligence has started to
be talked about in many areas. One of these areas is the programming
Augmented Intelligence is a concept used to combine human and and software development sector.
artificial intelligence. Augmented Intelligence allows people to make Programming education has made significant progress in the last
more effective and efficient decisions by working with artificial intelli­ decade. While previously programming was considered a skill that a
gence technologies. This approach aims to manage and use artificial small group of people should know, today, it has become an essential
intelligence systems by humans. In this way, people can gain more requirement in many industries. Programming is now a fundamental
confidence and accuracy in decision-making by leveraging artificial in­ tool for solving complex problems and providing innovative solutions in
telligence technologies. By combining artificial intelligence and human many fields, including the healthcare, finance, and transportation sec­
intelligence. Augmented intelligence can complement each other’s tors. Therefore, programming education is vital for anyone who wants to
shortcomings and achieve better results. Today, especially with artificial be successful in many areas, especially in the business world (Nouri
intelligence language models such as ChatGPT finding more place in et al., 2020).

* Corresponding author.
E-mail addresses: [email protected] (R. Yilmaz), [email protected] (F.G. Karaoglan Yilmaz).

https://doi.org/10.1016/j.chbah.2023.100005
Received 14 April 2023; Received in revised form 9 July 2023; Accepted 13 July 2023
Available online 16 July 2023
2949-8821/© 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

Despite the increasing interest in programming education, learning 2.2. Programming learning environments
programming is generally a challenging and complex process for most
people (Tsai, 2019; Yusoff et al., 2020). For this reason, many people In recent years, there has been a serious increase in programming
who start the process of learning programming may give up learning languages and tools. This has made it easier for people to learn and use
programming before completing this process successfully. The reasons software application development skills (Lindberg et al., 2019; Zino­
for this include the need for support and guidance during the program vieva et al., 2021). This trend, especially combined with the increase in
writing process, difficulties in debugging the errors in the program they popularity of online learning platforms, has made it easier for people to
have written, etc. (Gomes & Mendes, 2014; Sun & Hsu, 2019). Today, access programming education. For example, platforms such as Code­
generative AI tools such as ChatGPT can provide solutions to the prob­ cademy, Udemy, and Coursera offer a wide variety of programming
lems that individuals face in programming by giving fast and mostly courses from beginner to advanced (Diaz et al., 2021; Oktavia et al.,
correct answers. It can almost offer them support and guidance. This 2018; Sharov et al., 2021). Therefore, the ease of access to programming
study aims to reveal the advantages and limitations of using ChatGPT in education tools and resources has increased the number of interested
the programming learning process from the student’s perspective. Upon persons.
reviewing the existing literature, no study investigates student views on Programming learning tools and environments aim to make the
benefiting ChatGPT in programming learning. In this respect, the programming learning process of students more effective and enjoyable.
research makes a new and original contribution to the literature. These resources are designed to help students understand programming
concepts more easily. Visual programming tools can help students make
2. Background and literature their programming code easier and more understandable (Sahay et al.,
2020). Interactive environments can help students identify and correct
2.1. Importance of programming education their mistakes by allowing them immediate feedback as they write their
code. In addition, these resources are designed to support students’
Programming education is an important source of skills and knowl­ self-learning processes (Fagerlund et al., 2021; Saito et al., 2017). In this
edge for students today. With today’s rapidly developing technology and way, students can progress at their own pace while learning program­
increasing digitalization in the business world, it can provide a great ming and have learning experiences that suit their learning styles (Chen
advantage for future business opportunities (Qian & Lehman, 2017; et al., 2023; Tian et al., 2023; Yilmaz & Yilmaz, 2023). There are various
Topalli & Cagiltay, 2018). Today, rapid developments in technology and programming tools for example Scratch, Alice, Code, Blockly, Code.org,
digitalization in the business world have made programming education App Inventor, Micro:bit, Greenfoot, Logo, Codecademy, Thimble, Lightbot,
an important source of skills and knowledge for students and a necessary Tynker, Blockly Games, App Lab, Snap!, CodeCombat, MakeCode, Swift
feature to survive in a competitive job market. Learning programming Playgrounds, Repl.it, and Unity which can be used in programming
skills equips students with analytical thinking, problem-solving, logical learning. Their purposes may differ from each other. These program­
reasoning, and algorithmic thinking skills, enabling them to learn ming tools and environments are designed to help students learn pro­
essential skills useful in any area of their lives. These skills can positively gramming concepts in a fun and engaging way. Using these resources,
affect students’ academic achievement, performance at work, and per­ students can improve their coding skills and better understand pro­
sonal development (Agbo et al., 2019; Tikva & Tambouris, 2021). gramming concepts.
Therefore, programming education is necessary to prepare students for
future job opportunities and can increase their professional success. 2.3. ChatGPT as a programming tool
Programming education is more important than providing students with
job opportunities and skills. Learning to program helps students develop ChatGPT is different from existing programming environments and
a deeper understanding of technology and better grasp the inner tools. Because ChatGPT is an artificial intelligence language model that
workings of today’s digital world. In addition, the programming can interact with people using a natural language. Therefore, even
learning process can also improve students’ creativity and ability to people without programming knowledge can easily solve programming
express themselves (Noh & Lee, 2020; Romero et al., 2017). In this way, problems with ChatGPT (Surameery & Shakor, 2023; Yilmaz & Yilmaz,
students consume technology and learn how it works and how they can 2023). Existing programming tools generally consist of software devel­
use it. Programming education can also increase students’ opment environments, programming languages, libraries, tools, and
self-confidence, allowing them to develop their own projects and other related components (Carver et al., 2007; Ray et al., 2014). These
generate innovative ideas (Dorotea et al., 2021). Therefore, program­ tools usually focus on a specific programming language and require
ming education can play a critical role in their future careers by programming knowledge. For example, a Python IDE can only be used
increasing students’ awareness of technology and improving their for the Python programming language, and the user must be familiar
creativity and ability to express themselves. with the basics of Python. However, there are cross IDE that allow
Programming training is offered at many educational institutions develop several languages, for example, Flutter (several platforms) or
around the world as a core part of the computer science curriculum. This Microsoft Visual Studio (several languages such as C++, C#, Visual
course introduces students to basic programming concepts, program­ Basic .NET, F#, Java, Python, Ruby). ChatGPT, on the other hand, uses
ming languages such as Python, Java, and C++. Students gain the ability natural language processing and machine learning technologies to un­
to write simple programs using programming tools and learn the steps in derstand and correctly respond to commands given by the user in a
the problem-solving process through programming. As programming natural language. ChatGPT has been developed to understand the user’s
education progresses, students move towards more complex program­ needs and respond accordingly, without using syntax and concepts
ming concepts, such as algorithms, data structures, and programming specific to programming languages (OpenAI, 2023). Therefore, ChatGPT
paradigms. This course provides students with a strong foundation in offers a different approach to programming education for both students
programming principles and problem-solving techniques and increases and teachers (Jalil et al., 2023; Yilmaz & Yilmaz, 2023).
their ability to develop software for solving real-world problems (Gor­ ChatGPT offers different advantages over other programming tools.
don et al., 2022; Siegfried et al., 2021). Students also gain the skills to Here are some advantages that ChatGPT can provide to those who want
test, debug and maintain software through programming education. to learn programming. Using Natural Language: ChatGPT can interact
This course helps students understand software development processes with people using natural language processing technologies. This allows
for different application areas and prepares them for a successful career even people with no programming knowledge to solve programming
in technology. problems via ChatGPT. Easy Access: ChatGPT can be accessed from any
device with an internet connection. Users do not need to install any

2
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

special software or tools. Quick Response: ChatGPT provides immediate 2.4. Purpose and importance of the research
feedback to users by providing quick response. This can speed up the
learning process and help students understand better. Personalized While ChatGPT offers many advantages to students in terms of pro­
Learning: ChatGPT can provide users with customized learning experi­ gramming learning, it also has several limitations in terms of program­
ence. It can provide users with learning materials, practices, or exam­ ming learning. Upon reviewing the existing literature, research has yet
ples, follow the learning process and provide customized feedback. to be found that examines students’ views on the use of ChatGPT in the
Multi-Language Support: ChatGPT is available in many different lan­ programming learning process. Knowing the students’ views on using
guages. This allows those who want to learn programming to interact ChatGPT in programming learning can provide educators and curricu­
with ChatGPT in their native language. Unlimited Resources: ChatGPT lum developers with many perspectives, especially regarding the inte­
can provide users with unlimited programming resources using the in­ gration of ChatGPT into the course. The purpose of this research carried
ternet’s vast resources. These resources include learning materials, out from this point of view, is to examine the students’ opinions about
programming examples, online courses, apps, and others. Clear Expla­ using ChatGPT for programming learning purposes. The findings of this
nations: ChatGPT provides students with clear explanations about pro­ research will provide valuable insights into the integration of generative
gramming topics. It helps students to understand the topics better and AI tools like ChatGPT into programming courses. These results will
allows them to spend less time. Examples and Applications: ChatGPT contribute to a better understanding of the potential benefits and im­
provides students with programming examples and applications, plications of utilizing such tools in educational settings, offering guid­
enabling them to use theoretical knowledge in practice. This helps stu­ ance for educators and curriculum developers in effectively
dents better understand the topics and have a better learning experience. incorporating them into programming instruction. In line with the main
Inquiry and Search: ChatGPT allows students to ask questions and search purpose of the research, the following research questions were sought to
about programming topics. This helps students research topics they are be answered.
curious about and learn more about. Debugging and Feedback: ChatGPT
helps students identify and fix programming errors. It also provides 1 What are the students’ views on the benefits/advantages of using
feedback to students and can make suggestions for better programming ChatGPT in the programming learning process?
practice. Advanced Topics: ChatGPT helps students advance to advanced 2 What are the students’ opinions about the limitations/disadvantages
levels in programming. Students can reach more advanced levels in of using ChatGPT in the programming learning process?
programming. Research shows that generative AI tools such as ChatGPT
also increase students’ computational thinking, programming self- 3. Method
efficacy, and motivation (Yilmaz & Yilmaz, 2023). According to Chen
et al. (2023), programming training supported by tools such as ChatGPT The following section details the research model, study group, data
can effectively improve students’ programming skills by providing collection tool and data analysis.
benefits such as code explanations and debugging. Similarly, Jalil et al.
(2023) state that software testing training using ChatGPT can be more 3.1. Research model
effective for learners. In addition, researchers state that ChatGPT can be
used effectively in debugging programming and assisting programmers This study examines the opinions of students who use the ChatGPT
(Surameery & Shakor, 2023; Tian et al., 2023). Some of the advantages tool for programming learning purposes about the use of ChatGPT. For
of ChatGPT are that it provides fast and mostly correct answers, can be this reason, the case study method was used in the research. A case study
accessed at any time and place, and is easy to access (Kasneci et al., is a detailed examination and analysis of an individual, group, organi­
2023; Lo, 2023; Qureshi, 2023; Tlili et al., 2023). All these are some of zation, or event. These studies delve deeper into a specific topic, explain
ChatGPT’s benefits in the programming learning process. a phenomenon, or understand what works or doesn’t work in a partic­
There are also limitations to the use of ChatGPT in programming. ular situation. This study aimed to examine the students’ opinions
Because ChatGPT is a language model, it needs some of the features of regarding the use of ChatGPT in the object-oriented programming II
traditional programming tools and environments. Some of the limita­ course within the scope of the case study. Thus, it was tried to determine
tions are as follows. Unstructured Learning: ChatGPT teaches students the situation of the students regarding the use of ChatGPT in the pro­
about a particular programming language or topic in an unstructured gramming course. Both quantitative and qualitative research ap­
way. This can cause students to need help managing the learning process proaches were utilized in the study. The researchers created a
and achieving their learning goals. Lack of Applications and Need for questionnaire to collect quantitative data and a form consisting of open-
Secondary Tools and Environments: ChatGPT does not provide environ­ ended questions to obtain qualitative data.
ments or tools for programming applications. Students must use a
separate tool or environment for coding in programming languages. 3.2. Participants and process
Limited Data Structures and Algorithms: ChatGPT does not support all data
structures and algorithms used in programming languages. Therefore, The research was conducted on 41 s-year undergraduate students
when students want to learn about a particular data structure or algo­ studying object-oriented programming II at a state university in Tur­
rithm they want to know, they may need to use different resources or key’s computer technology and information systems department. The
tools. Lack of User Interface: ChatGPT does not support interfaces (GUI) study included 33 male and 8 female students who voluntarily partici­
used in programming languages. This means students may need to use pated and responded to the data collection tool. The students who took
another tool or environment when they want to develop GUI-based part in the study were aged between 19 and 25. The students took
applications in programming languages. Rahman and Watanobe Introduction to Programming and Algorithms I and II courses in the first
(2023) state in their research that although ChatGPT has several ad­ year of the undergraduate program and the Object-Oriented Program­
vantages in learning programming, it has limitations, such as a lack of ming I course in the fall semester of the second year of the undergrad­
common sense, possible biases, difficulty in complex reasoning, and uate program. From this point of view, the students participating in the
inability to process visual information. The researchers also note that the research are experienced in programming.
ethical implications of ChatGPT (e.g., bias and discrimination, privacy Within the research scope, it aimed to integrate ChatGPT into the
and security, misuse of technology, accountability, transparency, and course. For this purpose, the object-oriented programming II course was
social impact) are complex and multifaceted. taught for eight weeks in the spring semester. The course was taught in a
computer laboratory environment. At the beginning of the course, the
instructor explained each week’s topic theoretically to the students.

3
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

During this lecture process, documents such as presentations and e- environment and the link was sent to the students via e-mail. The stu­
books were also utilized. This took approximately 1 h of the course. The dents answered the questions in the data collection tool by clicking on
students spent the remaining 2 h doing the application assignments the link. By answering open-ended questions, students’ opinions were
related to that week’s topic that the instructor projected on the screen. obtained to reveal the benefits and limitations of using ChatGPT in the
The application assignments were related to that week’s topic and programming course from the student’s perspective.
involved writing programs using the information that the students had
learned in the theoretical course. For example, the following assignment 3.3. Data collection tools
was given on array and collection.
“Market application will be developed using array and collection. You The data collection process involved using a questionnaire and a
need to write a program to keep the list of products in the market and perform form consisting of open-ended questions, both designed by the re­
different operations. Requirements; searchers. The questionnaire and semi-structured form questions were
prepared by reviewing the literature. Then, it was submitted to the
- Product Class: evaluation of five experts in the field of educational technology. In line
Create a "Product" class. Use the appropriate data types to store the with the opinions of the experts, the questions were tried to be prepared
following properties for each grocery product: Product Name as short, simple, and clear as possible in order to be suitable for the
Manufacturer name research and to facilitate students’ responses. Students’ demographic
Price information characteristics were assessed using questions included in the question­
Production date naire. These questions are about age and gender. Questions in the form
It must contain the necessary getter and setter methods. were developed to determine students’ viewpoints on the use of
- Market Class: Create a "Market" class. This class should use an array or ChatGPT for programming learning purposes. For this purpose, two
a suitable collection to store the list of market products. Add the following open-ended questions were included in the semi-structured form. These
functionalities: questions were; a) "What have been the benefits of using ChatGPT while
Add a new product doing the application assignments in the programming course? Explain."
Removing a product b) "What were the limitations/disadvantages of using ChatGPT while
List all products doing the application assignments in the programming course? Explain."
Search for a product by manufacturer After the questions were included in the questionnaire form and the
Search for a product by production date semi-structured form was created, the researchers created the ques­
- Test Application: Write a test application. This application should add, tionnaire electronically using Google Forms. Then, during the data
remove and search products using the market class. collection process, the link to the online form was sent to the students in
Present a menu to the user and allow them to select different actions. the class and the students were asked to answer the form using the
After the user selects an action, perform the action by taking the relevant computers in the laboratory. The results of the content analysis made in
inputs and show the results to the user. line with the questions in the form and the answers given to these
The program should continue to run until the user logs out. questions are given in the findings section of the article.
- Tip: You can use a suitable collection like ArrayList or HashMap to store
the products. 3.4. Data analysis
You can use the Scanner class to get input from the user.”
The qualitative data obtained from the 41 student participants were
As shown in the example, students are given laboratory assignments subjected to analysis through the content analysis method. This method
based on the information related to the subject learned each week within allowed for the identification of key themes, patterns, and insights
a scenario. The purpose of the laboratory assignments is to give the within the collected data, enabling a comprehensive exploration and
students the ability to apply what they have learned about the theo­ interpretation of the students’ experiences and perspectives. Two
retical topic of that week. In this direction, the instructor presents a different researchers took part in the analysis of the qualitative data.
problem situation to the student within a scenario. It was stated that the Based on the students’ responses to each research question, the re­
student could benefit from ChatGPT while performing the programming searchers extracted sub-themes related to the advantages of using
task related to this problem. At the beginning of the research process, ChatGPT and sub-themes related to the limitations/disadvantages of
that is, at the beginning of the academic term, the instructor explained using ChatGPT. Then, the two researchers came together to examine the
the ChatGPT tool and how it can be used within the scope of program­ sub-themes and clarified the sub-themes by reaching a consensus. These
ming education to the students. Students used the ChatGPT tool for sub-themes were written on the student opinion form and guided the
various purposes while doing their weekly laboratory assignments. researchers in terms of classification in analyzing the students’ opinions.
Students do their weekly laboratory assignments individually using After determining the sub-themes, each researcher examined the stu­
the computers in the computer laboratory and ChatGPT. After dents’ opinions to determine the number of codes (frequency). The use
completing the assignment, students send the assignment to the course of ChatGPT for programming learning was examined by gathering the
e-mail address. The instructor then evaluates the correctness of the opinions of students through a form consisting of open-ended questions.
assignment and gives the student a score. The assignments are explained For research reliability purposes, the qualitative data collected from the
to the students by the instructor after the theoretical lecture. Therefore, analysis process were coded by one researcher and cross-checked by a
the same assignments were given to all students in the class using the second encoder. The agreement between the two coders was assessed by
same methodology. In other words, the laboratory assignments and the calculating the percentage of codes they both assigned to the same
way the course is implemented are similar for all students. In this way, it category (Miles & Huberman, 1994). The coding reliability was 95%,
was aimed to control the factors that could threaten the validity and indicating a strong agreement between the two coders. The remaining
reliability of the research. The research continued for eight weeks. 5% was resolved through discussion and consensus. After reviewing the
Students benefited from ChatGPT in the process of doing different students’ written explanations, it was apparent that the coding incon­
application assignments related to the course every week. At the end of sistency arose from some of the responses being classified under various
the eight-week period, the students’ views on the use of ChatGPT in sub-themes. By means of a questionnaire, the study obtained quantita­
programming education became clear. At the end of this process, the tive data from the participants about their gender, age, and other rele­
data collection tool developed by the researchers was applied to the vant details. Descriptive statistics, particularly the mean, were
students. The data collection tool was prepared in a web-based employed to analyze the gathered data. While interpreting the

4
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

qualitative data in the research, quantitative data were used in order to any day. Users can ask their questions at any time and get immediate an­
better determine the study group, reveal demographic information and swers. ChatGPT is available in many languages. Users can ask questions and
create a student profile. get answers in their native language.”
These interventions were made to ensure the validity, reliability, and
generalizability of the study. During participant selection for the 4.2. Views on the disadvantages/limitations of using ChatGPT in the
research, inclusion criteria were applied to identify students who learning programming process
actively utilized the ChatGPT tool as part of their programming learning
process. Thus, it is aimed that the students have a detailed view of the Students’ views on the disadvantages/limitations of using ChatGPT
questions in the context. Two researchers coded the students’ opinions in the programming learning process and doing application assignments
separately to assess the research’s reliability. The reliability coefficient were analyzed. Student opinions are given in Table 2.
between the two coders was then computed. Experts in educational While some of the students stated that there were no disadvantages/
technologies reviewed the research questions to ensure their validity. limitations of using ChatGPT in the programming process, some students
stated that there were various disadvantages/limitations. First of all, one
4. Findings of the points stated by the students is related to the fact that the use of
ChatGPT leads the student to be lazy/easy-going and causes occupa­
4.1. Opinions on the benefits of using ChatGPT in the learning tional anxiety. Some students stated that ChatGPT may not always give
programming process correct answers and that they needed more knowledge to answer some
questions. A small number of students stated that ChatGPT has aspects
Students’ views on the benefits of using ChatGPT in the program­ for improvement as a programming learning tool/environment. Some
ming learning process and doing application assignments were students stated that the use of ChatGPT could weaken students’ thinking
analyzed. Student opinions are given in Table 1. skills and make them dependent on themselves. Some of the student
Students stated that the use of ChatGPT in the programming process opinions are as follows.
provides many advantages. It is noted that the most essential benefits of S12: “ChatGPT may not always give correct answers. In particular, it
ChatGPT in this process are that it responds quickly and effectively to may make mistakes when responding to questions that are misunderstood or
questions, reduces time loss in researching the solution to problems, contain missing information.”
helps in debugging, gives ideas for solving complex problems, and S16: “It is a disadvantage that it negatively affects the algorithmic
provides information on details about the subject. Students state that the thinking ability by taking the easy way out.”
answers given by ChatGPT make the students think and think about how S23: “I think that it will remove the efficiency of professions. I think that
to integrate the given solution into the project and that it will improve after a while there will be no need for people. In the future, it may even be able
the students’ thinking skills. In addition, students see ChatGPT as a to finish coding a website.”
teacher and state that they use it to learn the details of the issues that S34: “The use of ChatGPT in programming can easily familiarize some­
need to be understood about the subject. Some of the student opinions one who is just starting to learn programming. In the past, people who entered
are as follows. the programming sector were preferred in the sector because they matured
S3: “I can easily learn any information I need. It works great for me in the with the experience they gained from the mistakes they made. It is very
coding world.” difficult to make mistakes nowadays because instead of us, there is an arti­
S11: “It is both easier and time-saving to find the answer in one place, ficial intelligence (model) that has already learned (trained) from these
rather than constantly researching from different sites.” mistakes.”
S13: “ChatGPT can write a desired program partially correctly or S38: “It is not a real programming environment. Therefore, it may be
completely if the desired program part is described correctly. For this reason, insufficient to enable students to gain real programming experience.”
it provides great convenience in programming. Likewise, ChatGPT helps in
roadmap when it comes to programming education.” 5. Discussion and conclusion
S24: “I can get fast and accurate answers to my questions about software
development. It allows me to spend my time practicing coding instead of This research was carried out on university students to determine the
looking for time and resources to find answers to my questions.” advantages and limitations of using ChatGPT in the programming
S35: “ChatGPT can answer questions quickly and effectively, thanks to learning and program writing processes. While students state that the
its natural language processing technology. ChatGPT is available at any hour, use of ChatGPT in program writing processes provides many benefits,
they also state that it includes some limitations. Students use ChatGPT to
Table 1 get suggestions for solving complex problems in the process of writing a
Opinions on the benefits of using ChatGPT. program, learning the subjects they do not know, understanding, and
Sub-Themes f obtaining information about the details of a subject. ChatGPT created a
Respond to questions quickly and effectively 39 positive perception in the eyes of the students as it gave quick and often
Preventing wasting time while doing research 38 effective answers to the questions of the students. However, some stu­
Help with debugging 35 dents state that ChatGPT sometimes needs to give correct answers or
Supporting the development of thinking skills 34
Providing the opportunity to reach the solution of complex problems 33
Providing simple and understandable teaching of incomprehensible subjects 32 Table 2
Contributing to improving our programming skills 31 Opinions on disadvantages/limitations of using ChatGPT.
Giving information about the details about the curious subject 28
Sub-Themes f
Being available 24/7 27
Confidence in programming 25 Can lead the programmer to be lazy/easy 16
Helping me see alternative solutions to the problem 23 Causing occupational anxiety 15
Guiding/giving ideas to the student in the process of making the projects 22 It does not have any disadvantages/limitations 13
Presenting the solution of problems from different disciplines such as 19 May not always give correct answers 12
mathematics and physics, which are needed in the program writing process. Not having enough information to answer some questions 9
Providing interaction in our own language thanks to natural language 18 Sometimes his/her answers are not as desired 9
processing technology Lack of a real programming training environment/tool 8
Demonstrating more accurate and appropriate writing of codes 17 Negatively affect the development of thinking skills 7
Providing help when I can’t continue while writing code 12 May increase self-dependence 5

5
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

answer questions. This may be related to the lack of answers to these concludes that the incorporation of ChatGPT in the programming pro­
questions in ChatGPT’s existing data sources. However, this may also be cess enhances students’ self-confidence, learning motivation, and
related to how students should ask questions to ChatGPT. At this point, code-writing skills. In a study by Qureshi (2023), the performance of an
the prompt comes to the fore. experimental group, which utilized ChatGPT, was compared to a control
A prompt is a sentence or short paragraph that forms the beginning of group that did not use ChatGPT in laboratory assignments for a data
a task or text to be completed by the language model using artificial structures and algorithms course. The results revealed that the experi­
intelligence technology. For example, "What are the advantages and mental group students who employed ChatGPT obtained higher scores,
disadvantages of using the K-Nearest Neighbor (KNN) Algorithm to indicating a clear advantage in their academic performance. Still, there
classify people by gender?" prompt could help a machine learning model needed to be more consistency and accuracy in the submitted code that
generate text about using the KNN algorithm to classify people by affected the overall performance. In this respect, the results of our study
gender. When using an artificial intelligence model such as ChatGPT, it are similar to the literature. The research findings suggest that the uti­
is important to type the correct prompt because the model tries to lization of generative AI tools, including ChatGPT, in program writing
generate text based on the given prompt (Liu et al., 2023; Reynolds & and programming learning processes yields overall benefits. These tools
McDonell, 2021). Typing the correct prompt can help the model produce have shown promise in enhancing various aspects of the learning
the correct results. For example, a false prompt such as "Classification of experience, such as improving self-confidence, motivation, code-writing
a human by gender" instead of "Benefits of using the KNN algorithm in skills, and academic performance. Therefore, incorporating generative
classifying people by gender" may cause ChatGPT to respond incor­ AI tools into programming education can be regarded as a valuable
rectly. Also, choosing a prompt can help the model provide helpful or approach to enhancing the learning outcomes and effectiveness of pro­
interesting information for the user. For example, the "How to become a gramming courses. Therefore, it is useful to integrate these tools into
better programmer?" prompt can direct ChatGPT to provide relevant courses. On the other hand, students should also take into account that
and helpful suggestions. Depending on ChatGPT’s answers to this the codes produced by these tools may be faulty. For this reason, it
question, the user can get in-depth information on the subject by asking should be explained to students that they should be made aware of this
more detailed questions (for example, the details of each answer) based and that the results produced by generative AI tools should be checked
on the answers given. When the literature is examined, it has been and verified by themselves. Students can reconstruct the outputs pro­
revealed that prompt type and prompt typing significantly affect in­ duced by generative AI tools and adapt them in their own way. Once
dividuals’ outputs such as writing performance, resource usage strate­ again, it is crucial to emphasize the importance of evaluating the ethical
gies, and monitoring strategies (Nückles et al., 2020; Shi et al., 2020). appropriateness of results generated by generative AI tools like
For this reason, it is very important today that teachers who want to ChatGPT. This evaluation is closely linked to the development of AI
integrate artificial intelligence language models such as ChatGPT into literacy skills among students. It becomes essential to equip students
their lessons provide their students with "prompt literacy" skills. with the knowledge and critical thinking abilities necessary to navigate
The majority of participants in the study reported that incorporating the ethical considerations and potential biases associated with
ChatGPT into their programming endeavors enhances their cognitive AI-generated outputs. By providing students with these skills, educators
abilities. By reducing the time spent on coding tasks, students are able to can empower them to make informed decisions and engage responsibly
allocate more time towards engaging in algorithmic thinking processes with generative AI tools, fostering a more ethical and responsible use of
for effective problem-solving. Students think about which questions to AI technologies in their academic and professional pursuits. (Kong et al.,
ask ChatGPT, how to integrate the answers they receive into their pro­ 2021; Laupichler et al., 2022; Wang et al., 2023). Thus, the integration
jects, and how they can complete the project holistically by bringing of generative AI tools such as ChatGPT into courses can be done
together small code blocks. This can improve students’ thinking skills. correctly and effectively. It should be noted that this research has some
On the other hand, some students state that the use of ChatGPT will limitations. First, the research was conducted within the Object Oriented
negatively affect their thinking skills, because the answer to the desired Programming II course scope. Similarities and differences in the results
question can be obtained from ChatGPT and therefore they cannot can be examined by examining student opinions in different program­
develop their thinking skills. This may be true for answers to questions ming courses (e.g. visual programming, web programming, robotic
with short and simple solutions. For this reason, it will be useful for programming, etc.). The research was conducted on university students.
teachers to give complex and modular projects to the students, and to Similar studies can be conducted on students from different levels (e.g.
ensure that the student tries to reach the whole from modular answers, middle school, high school, etc.) to examine student opinions. The
at the point of development of thinking skills (computational thinking, implementation period of this research is limited to eight weeks. In
creativity, algorithmic thinking, cooperation, critical thinking, and future studies, longitudinal studies can be conducted to analyze the
problem-solving). When the literature is examined, it has been revealed opinions of students who use ChatGPT for longer. Differences in the
that complex and unstructured problems are useful in developing stu­ results depending on the duration of use can be determined. To deter­
dents’ computational thinking skills (Bai et al., 2021; Jiang et al., 2022). mine the effect of generative AI tools such as ChatGPT on students’
When the literature is examined, it is seen that research results are learning processes and outcomes, it would be useful to plan experi­
confirming this view of the students. In the study conducted by Yilmaz & mental studies, especially with experimental and control groups.
Karaoglan Yilmaz (2023), it was concluded that the computational
thinking, programming self-efficacy, and motivation of the students who Ethical approval
received ChatGPT support during the programming learning process
were higher than the students who did not receive this support. Ac­ All procedures performed in these studies were in accordance with
cording to the results of Yilmaz & Karaoglan Yilmaz (2023) study, the the APA ethical guidelines, the ethical standards of the institutional
use of ChatGPT led to a significant increase in students’ creativity, research committee, and the 1964 Helsinki declaration and its later
algorithmic thinking, cooperativity, critical thinking, and amendments.
problem-solving scores. These findings of the study are consistent with
the results of our research. Based on the results of our study, students Code availability
stated that using ChatGPT in the programming process effectively
improved their thinking skills. Huang et al. (2023) conducted a study The authors used AMOS functions for their statistical analyses.
that demonstrated the positive impact of providing AI-based personal­
ized advice and guidance to learners, leading to improved motivation
and learning outcomes. Consistent with these findings, our research

6
R. Yilmaz and F.G. Karaoglan Yilmaz Computers in Human Behavior: Artificial Humans 1 (2023) 100005

Informed consent to participate Nouri, J., Zhang, L., Mannila, L., & Norén, E. (2020). Development of computational
thinking, digital competence and 21st century skills when learning programming in
K-9. Education Inquiry, 11(1), 1–17.
All participants gave full informed consent to participate. Nückles, M., Roelle, J., Glogger-Frey, I., Waldeyer, J., & Renkl, A. (2020). The self-
regulation-view in writing-to-learn: Using journal writing to optimize cognitive load
Consent for publication in self-regulated learning. Educational Psychology Review, 32, 1089–1126.
Oktavia, T., Prabowo, H., & Supangkat, S. H. (2018). The comparison of MOOC (massive
open online course) platforms of edx and coursera (study case: Student of
All participants gave consent for their data to be used in publication. programming courses). In 2018 international conference on information management
and technology (ICIMTech) (pp. 339–344). IEEE.
OpenAI. (2023). Introducing ChatGPT. https://openai.com/blog/chatgpt.
Data availability Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in
introductory programming: A literature review. ACM Transactions on Computing
The authors are willing to share their data, analytics methods, and Education, 18(1), 1–24.
Qureshi, B. (2023). Exploring the use of chatgpt as a tool for learning and assessment in
study materials with other researchers upon request. undergraduate computer science curriculum: Opportunities and challenges. arXiv preprint
arXiv:2304.11214.
Declaration of competing interest Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research:
Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783.
Ray, B., Posnett, D., Filkov, V., & Devanbu, P. (2014). A large scale study of programming
The authors have no conflicting or competing interests to declare. languages and code quality in github. In Proceedings of the 22nd ACM SIGSOFT
international symposium on foundations of software engineering (pp. 155–165).
Reynolds, L., & McDonell, K. (2021). Prompt programming for large language models:
References
Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on
human factors in computing systems (pp. 1–7).
Agbo, F. J., Oyelere, S. S., Suhonen, J., & Adewumi, S. (2019). November). A systematic Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through
review of computational thinking approach for programming education in higher creative programming in higher education. International Journal of Educational
education institutions. In Proceedings of the 19th koli calling international conference on Technology in Higher Education, 14(1), 1–15.
computing education research (pp. 1–10). Sahay, A., Indamutsa, A., Di Ruscio, D., & Pierantonio, A. (2020). Supporting the
Bai, H., Wang, X., & Zhao, L. (2021). Effects of the problem-oriented learning model on understanding and comparison of low-code development platforms. In 2020 46th
middle school students’ computational thinking skills in a python course. Frontiers in euromicro conference on software engineering and advanced applications (SEAA) (pp.
Psychology, 12, Article 771221. 171–178). IEEE.
Carver, J. C., Kendall, R. P., Squires, S. E., & Post, D. E. (2007). May). Software Saito, D., Sasaki, A., Washizaki, H., Fukazawa, Y., & Muto, Y. (2017). Program learning
development environments for scientific and engineering software: A series of case for beginners: Survey and taxonomy of programming learning tools. In 2017 IEEE 9th
studies. In 29th international conference on software engineering (ICSE’07) (pp. international conference on engineering education (ICEED) (pp. 137–142). IEEE.
550–559). Ieee. Sharov, S., Kolmakova, V., Sharova, T., & Pavlenko, A. (2021). Analysis of MOOC on
Chen, E., Huang, R., Chen, H. S., Tseng, Y. H., & Li, L. Y. (2023). GPTutor: A ChatGPT- programming for IT specialist training. TEM Journal: Technology, Education,
powered programming tool for code explanation. arXiv preprint arXiv:2305.01863. Management, Informatics, 10(4), 1884–1894.
Dorotea, N., Piedade, J., & Pedro, A. (2021). Mapping K-12 computer science teacher’s Shi, B., Huang, L., & Lu, X. (2020). Effect of prompt type on test-takers’ writing
interest, self-confidence, and knowledge about the use of educational robotics to performance and writing strategy use in the continuation task. Language Testing, 37
teach. Education Sciences, 11(8), 443. (3), 361–388.
Fagerlund, J., Häkkinen, P., Vesisenaho, M., & Viiri, J. (2021). Computational thinking in Siegfried, R. M., Herbert-Berger, K. G., Leune, K., & Siegfried, J. P. (2021). Trends of
programming with Scratch in primary schools: A systematic review. Computer commonly used programming languages in CS1 and CS2 learning. In 2021 16th
Applications in Engineering Education, 29(1), 12–28. international conference on computer science & education (ICCSE) (pp. 407–412). IEEE.
Gomes, A., & Mendes, A. (2014). A teacher’s view about introductory programming Sun, J. C. Y., & Hsu, K. Y. C. (2019). A smart eye-tracking feedback scaffolding approach
teaching and learning: Difficulties, strategies and motivations. In 2014 IEEE frontiers to improving students’ learning self-efficacy and performance in a C programming
in education conference (FIE) proceedings (pp. 1–8). IEEE. course. Computers in Human Behavior, 95, 66–72.
Gordon, C., Lysecky, R., & Vahid, F. (2022). Programming learners struggle as much in Surameery, N. M. S., & Shakor, M. Y. (2023). Use chatgpt to solve programming bugs.
Python as in C++ or Java. In 2022 ASEE annual conference & exposition. International Journal of Information Technology & Computer Engineering (IJITC), 3(1),
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023). April). Chatgpt and 17–22.
software testing education: Promises & perils. In 2023 IEEE international conference Tian, H., Lu, W., Li, T. O., Tang, X., Cheung, S. C., Klein, J., & Bissyandé, T. F. (2023). Is
on software testing, verification and validation workshops (ICSTW) (pp. 4130–4137). ChatGPT the ultimate programming an assistant–How far is it?. arXiv preprint arXiv:
IEEE. 2304.11938.
Jiang, S., Qian, Y., Tang, H., Yalcinkaya, R., Rosé, C. P., Chao, J., & Finzer, W. (2022). Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through
Examining computational thinking processes in modeling unstructured data. programming in K-12 education: A conceptual model based on a systematic
Education and Information Technologies, 1–25. https://doi.org/10.1007/s10639-022- literature review. Computers & Education, 162, Article 104083.
11355-3 Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., &
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., … Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case
Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large study of using chatbots in education. Smart Learning Environments, 10(1), 15.
language models for education. Learning and Individual Differences, 103, Article Topalli, D., & Cagiltay, N. E. (2018). Improving programming skills in engineering
102274. education through problem-based game projects with Scratch. Computers &
Kong, S. C., Cheung, W. M. Y., & Zhang, G. (2021). Evaluation of an artificial intelligence Education, 120, 64–74.
literacy course for university students with diverse study backgrounds. Computers Tsai, C. Y. (2019). Improving students’ understanding of basic programming concepts
and Education: Artificial Intelligence, 2, Article 100026. through visual programming language: The role of self-efficacy. Computers in Human
Laupichler, M. C., Aster, A., Schirch, J., & Raupach, T. (2022). Artificial intelligence Behavior, 95, 224–232.
literacy in higher and adult education: A scoping literature review. Computers and Wang, B., Rau, P. L. P., & Yuan, T. (2023). Measuring user competence in using artificial
Education: Artificial Intelligence, 3, Article 100101. intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour
Lindberg, R. S., Laine, T. H., & Haaranen, L. (2019). Gamifying programming education & Information Technology, 42(9), 1324–1337.
in K-12: A review of programming curricula in seven countries and programming Yilmaz, R., & Yilmaz, F. G. K. (2023). The effect of generative artificial intelligence (AI)-
games. British Journal of Educational Technology, 50(4), 1979–1995. based tool use on students’ computational thinking skills, programming self-efficacy
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and motivation. Computers and Education: Artificial Intelligence, 4, Article 100147.
and predict: A systematic survey of prompting methods in natural language Yusoff, K. M., Ashaari, N. S., Wook, T. S. M. T., & Ali, N. M. (2020). Analysis on the
processing. ACM Computing Surveys, 55(9), 1–35. requirements of computational thinking skills to overcome the difficulties in learning
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the programming. International Journal of Advanced Computer Science and Applications, 11
literature. Education Sciences, 13(4), 410. (3), 244–253.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded Zinovieva, I. S., Artemchuk, V. O., Iatsyshyn, A. V., Popov, O. O., Kovach, V. O.,
sourcebook. Sage Publications. Iatsyshyn, A. V., … Radchenko, O. V. (2021). March). The use of online coding
Noh, J., & Lee, J. (2020). Effects of robotics programming on the computational thinking platforms as additional distance tools in programming education. In Journal of
and creativity of elementary school students. Educational Technology Research & physics: Conference series (Vol. 1840)IOP Publishing, Article 012029, 1.
Development, 68, 463–484.