Using ChatGPT throughout the Software Development Life Cycle

by Novice Developers
Muhammad Waseem Teerath Das Aakash Ahmad
Faculty of Information Technology Faculty of Information Technology School of Computing and
University of Jyväskylä University of Jyväskylä Communications
Jyväskylä, Finland Jyväskylä, Finland Lancaster University Leipzig
[email protected] [email protected] Leipzig, Germany
[email protected]

Mahdi Fehmideh Peng Liang Tommi Mikkonen

arXiv:2310.13648v1 [cs.SE] 20 Oct 2023

School of Business School of Computer Science Faculty of Information Technology

University of Southern Queensland Wuhan University University of Jyväskylä
Queensland, Australia Wuhan, China Jyväskylä, Finland
[email protected] [email protected] [email protected]

ABSTRACT KEYWORDS
This study investigates the impact of ChatGPT - a generative AI- ChatGPT, Undergraduate Students, Software Development, Learn-
based tool - on undergraduate students’ software development ing Impact, User Experience
experiences. Through a three-month project involving seven un-
dergraduate students, ChatGPT was employed as a supporting tool,
and their experiences were systematically surveyed before and af- 1 INTRODUCTION
ter the projects. The research aims to answer four key questions Artificial Intelligence (AI) and especially Machine Learning (ML)
related to ChatGPT’s effectiveness, advantages, limitations, impact is increasingly being integrated into software development pro-
on learning, and challenges faced. The findings revealed signifi- cesses [5], leveraging ML algorithms and intelligent systems to
cant skill gaps among undergraduate students, underscoring the automate code generation, bug fixing, and testing, thereby enhanc-
importance of addressing educational deficiencies in software de- ing efficiency and reducing manual effort [6, 38]. This technological
velopment. ChatGPT was found to have a positive influence on progression paves the way towards Large Language Models (LLMs)
various phases of the software development life cycle, leading to [17, 26, 40], which have revolutionized textual and coding tasks
enhanced efficiency, accuracy, and collaboration. ChatGPT also by understanding and generating human-like text. Notable devel-
consistently improved participants’ foundational understanding opments within LLMs include BERT [16], GPT [45], and LLaMA,
and soft skills in software development. These findings underscore each having distinct functionalities. Among these, the Generative
the significance of integrating AI tools like ChatGPT into under- Pre-trained Transformer model, for example GPT-3 has garnered
graduate students education, particularly to bridge skill gaps and substantial attention due to its ability to train on extensive hu-
enhance productivity. However, a nuanced approach to technol- man data and generate coherent text, both labelled and unlabelled
ogy reliance is essential, acknowledging the variability in opinions datasets. Moreover, GPT-3 excels in a variety of language-related
and the need for customization. Future research should explore tasks such as answering questions and completing text with excep-
strategies to optimize ChatGPT’s application across development tional accuracy.
contexts, ensuring it maximizes learning while addressing specific Context and Motivation: GPTs have pervaded numerous ap-
challenges. plications, notably AI chatbots, content creation, and diverse areas
within software development, as demonstrated by multiple stud-
CCS CONCEPTS ies and practical applications (e.g., [1, 19, 32, 48]). Nevertheless,
empirical reports specifically detailing the application of GPTs,
• Software and its engineering → Software development tech-
including GPT-3 and the more recent ChatGPT, in the training
niques.
of undergraduate students across various software development
phases remain notably limited. We will use ’students’ for short in
the rest of the paper. For instance, GPT-3 has been investigated for
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed its capabilities in automatic code generation and automated code
for profit or commercial advantage and that copies bear this notice and the full citation documentation [25, 32], and utilized to develop tools like Codex,
on the first page. Copyrights for components of this work owned by others than ACM which converts natural language instructions into code and aids in
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or automatically generating code documentation [25]. Simultaneously,
a fee. Request permissions from [email protected]. students are consistently face challenges such as understanding fun-
Conference’17, July 2017, Washington, DC, USA damental architectural concepts, managing subsystem integrations,
© 2023 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 and confronting debugging issues [14]. A survey involving both
https://doi.org/10.1145/nnnnnnn.nnnnnnn educators and students highlighted common struggles that students
Conference’17, July 2017, Washington, DC, USA M. Waseem et al.

face, including program designing, procedure segmentation, and technical output of the projects, and the user experience and in-
error identification within their code [29], while additional research teraction with ChatGPT. Primary contributions of this study are
has illuminated the occurrence of recurrent code quality issues in to:
novice-authored programs [41] and difficulties in utilizing tools • Evaluate the influence of ChatGPT across all phases of the
like version control systems, performing infrastructure testing, and software development life cycle among students, and identify
managing bug tracking and resolution [10, 15, 22]. its advantages and limitations.
Considering the above challenges, AI tools – such as the recently • Identify the challenges faced by students when using Chat-
introduced ChatGPT – can assist in the training and education of GPT, and assess its educational value in the context of learn-
students. Although researchers have begun exploring ChatGPT’s ing curve and skill development among students engaged in
potential in various software development tasks—including col- software development projects.
laborative software architecture building [1] and software testing • Generate empirically-grounded evidence concerning the in-
education [23]—and its efficacy in addressing programming bugs tegration and effectiveness of ChatGPT in software devel-
has been explore [39], there are few studies that report empirical opment processes, contributing to the broader discourse on
evidence detailing how students have used ChatGPT throughout human-AI collaboration in software engineering.
the software development life cycle.
Objective of the Study: This study explores the efficacy of Paper Organization: The remainder of this paper is organized
ChatGPT utilizing within different phases of software development as follows: Section 2 reviews the related work. In Section 3, we
life cycle undertaken by students. This includes assessing the im- described the research method employed in this study. Section 4
pact on requirements analysis, design, architecture, development, presents the results of our study, which are further elaborated upon
testing, and deployment phases. Additionally, the study aims to in Section 5. Section 6 outlines the potential threats to validity.
outline the advantages and limitations of employing ChatGPT at Finally, Section 7 provides concluding remarks on this work and
each phase, understanding its influence on the learning curve and suggests directions for future research.
skill development, assessing improvement in students’ proficiency
in software development concepts, and identifying the challenges 2 RELATED WORK
encountered by the students throughout the projects. Through a This section provides a brief overview related work concerning
combination of quantitative and qualitative analysis, this study en- the utilization of ChatGPT throughout the various phases of soft-
deavours to furnish comprehensive insights that could contribute to ware development, from requirement gathering to deployment. A
enhancing the understanding of the integration of AI-driven tools notable experimental study by Bencheikh et al.[8] delved into the
in software development processes, especially for individuals at the efficacy of AI tools, particularly ChatGPT, in generating software
early stages of their educational and development careers. Align requirements efficiently. They highlighted the ability of ChatGPT
with objective of the study, we formulated the following Research to emulate human expertise, though emphasizing the indispensable
Questions (RQs). nature of human feedback to enhance requirement quality. More-
over, the time efficiency of ChatGPT was acknowledged, albeit with
• RQ1: How did the utilization of ChatGPT impact the require- a recognition that experienced human participants tend to produce
ments analysis, design, architecture, development, testing, more comprehensive requirements. The study also differentiated
and deployment phases of software development, and what between the premium and free versions of ChatGPT, showing a
are the advantages and limitations of its use at each phase superior consistency and overall quality in the former. On a related
of the software development life cycle? note, White et al.[44] proposed several ChatGPT prompt patterns
• RQ2: How did the involvement in software development to elevate software requirements elicitation. Transitioning to the ar-
projects that incorporated ChatGPT influence the learning chitectural phase of software development, a case study by Ahmad
curve and skill development of the students? et al. [1] shed light on the collaborative efforts between a novice
• RQ3: To what extent did the students’ proficiency in soft- software architect and ChatGPT. The focus was on architectural
ware development concepts improve through their partici- analysis, synthesis, and evaluation in a services-driven software
pation in these projects with the help of ChatGPT? application, revealing ChatGPT’s potential to effectively emulate
• RQ4: What challenges did the students face when using the role of an architect in Architectural Collaborative Software
ChatGPT? Engineering (ACSE). Yet, the necessity of human oversight was
underscored. In a similar way, White et al., [44] exploited ChatGPT
prompt patterns to enhance the overall software design.
To answer these RQs, we engaged seven students as paid devel- In the coding and implementation, several researchers have used
opers for a three-month period to create AI-based software systems ChatGPT. For instance, Al-Khiami et al.[2] conducted a case study
based on requirements coming from public procurement calls or to explore the feasibility of using ChatGPT for generating JavaScript
from external stakeholders, with ChatGPT as their support tool. Sub- code suitable for Android Studio, aimed at creating a functional app.
sequently, we conducted a survey-based study to assess ChatGPT’s Concurrently, Bera et al.[9] employed ChatGPT to support agile
impact and the students’ experiences before and after completing software development. Additionally, the significance of ChatGPT
all three software projects. Our investigation aimed to understand prompt patterns in improving code quality and facilitating refac-
ChatGPT’s role in educating and training students throughout the toring was discussed by White et al. [44]. Beyond these, the GPT-3
various phases of software development, spanning from project model has been utilized for automatic code generation [32] and
requirements to deployment. This encompassed evaluating the ef- automation of code documentation [25], with Tian et al.[42] employ-
fectiveness of ChatGPT, its influence on students’ learning, the ing ChatGPT as a programming assistant, albeit noting limitations
Using ChatGPT throughout the Software Development Life Cycle by Novice Developers Conference’17, July 2017, Washington, DC, USA

Entry Survey
Background
Information
Software
Engineering/Development
Tools Proficiency and
Learning
individuals with limited software development knowledge utilize
( 5 questions) (33 questions) (6 questions)

AI Tools
ChatGPT:
Education Software Process
Requirements
Knowledge

Required
Expereince Design
Implementation
Data
Analysis • AI Procurement Assistant (AIPA): This project aimed
Testing Open-ended
Knowledge/Skills
- Development Process
Deployment
to streamline enterprise procurement processes using an
- Software Design (UML) - Functionality
AI-powered platform, making it more efficient for users to

Development Process
- Computer Programming - Quality
- Software Testing Developer Descriptive
Project - Constraints
Recruitment 1
Project Requirements Design Implementation Testing Deployment
Stastics
identify relevant opportunities. ChatGPT played a key role
(7 UG Students)
Public
Project
2
Project
3 Open Coding
in simplifying procurement options identification, address-
Repository
Development Experts
ing common challenges found in traditional manual bid as-
Impacts of ChatGPT on Software Engineering/Development
(114 questions)
sessments. The client for this project was XYZ software
Survey Data
Requirements Design Implementation Testing Deployment
development company.
Effectiveness Development • AI-based Teacher-Tech Forum: The primary objective
Evaluation Obstacles
(35 questions) Advantges/
Limitations
Learning/
Skills
Development
Proficiency
(16 questions)
(11 questions)
of this project was to establish an online platform where
(36 questions) (12 questions)
teachers and small to medium-sized technology companies
Exit Survey could collaborate and exchange ideas. Teachers had the op-
portunity to submit suggestions for enhancing technology’s
Figure 1: Overview of the Research Method role in education, and these ideas underwent review by tech
companies for potential implementation. The client for this
project was ABC University.
regarding ChatGPT’s attention span. Moving onto the testing and • AI-based Skills Assessment SaaS for Basic Education:
deployment phase, Surameery et al.[39] examined the effectiveness This software system was aimed to evaluate the grades of
of ChatGPT in resolving programming bugs. Moreover, Jalil et al., junior-level school students and provide a concise overview
[23] emphasized the significance of ChatGPT in software testing of their performance in subjects like mathematics, English,
education. history, and more. The results were made accessible to city
In summary, this paper aims to extend the current body of knowl- municipalities and schools, providing valuable insights into
edge by investigating the role of ChatGPT in assisting students and student performance. The requirements for this project came
novice developers throughout the entire software development life from the AXZ public procurement system, and the client
cycle. Our study distinctively encompasses the complete develop- was the city government.
ment cycle of three actual publicly announced software projects,
endeavoring to discuss the learnings, challenges, and achievements These projects serve as the foundation for our study, where
concerning the use of ChatGPT in software development. ChatGPT plays a key role in assisting students in their software
development endeavors.
3 RESEARCH METHOD
The research method of this study consists of four phases, as illus- 3.2 Conducting Entry Survey
trated in Figure 1. Each of these phases is detailed in the following We adopted a cross-sectional survey design, which is appropriate
subsections. for collecting information at one given point in time across a sam-
ple population [27]. Surveys can be conducted in many ways, such
3.1 Developers and Projects Selection as Web-based online questionnaires and phone surveys [30]. We
Developers Selection: In collaboration with the XYZ1 software decided to conduct a Web-based survey because these surveys can
development company, we aim to assess the software development help to (i) minimize the time and cost, and (ii) save the effort of
experience of students with the assistance of ChatGPT (i.e., GPT-3.5 researchers in collecting data in a textual, graphical, or structured
and GPT-4). For this purpose, we selected seven students who have format [30]. We created a survey instrument (see the Entry Survey
completed their first and second years of undergraduate studies in Questionnaire sheet in [43]) based on grey and peer-reviewed lit-
information technology, computer science, and software engineer- erature, alongside feedback from software engineering professor
ing programs. These students were intentionally chosen because who also have rich industrial experience. The collaborative effort
of their limited familiarity with various aspects of the software continued with focus group discussions among the authors, who
development process. They are in the early stages of their academic have both academic and industrial backgrounds, contributing sig-
journey and have yet to delve deeply into practices such as Scrum, nificantly to the instrument’s development. The survey, structured
CI/CD, software design methodologies like UML, programming into three distinct sections with a total of 53 questions, aimed to
languages like Python, software testing techniques like automated explore the participants’ background, their experience in software
testing, and deployment practices. engineering, and proficiency with AI tools. To ensure the instru-
Project Selection: This study focus on three publicly announced ment’s effectiveness and alignment with our objectives, a pilot
software projects assigned to a group of seven students, divided into survey was conducted with two students who were not the part of
two teams. ChatGPT, an AI-powered tool, was utilized extensively development team.
to support students throughout the software development process, Background Information: To gain insights into each partici-
spanning from understanding project requirements to deployment. pant’s background and experience with software engineering activ-
The rationale behind selecting students is to gain insights into how ities, we posed 5 questions aimed at understanding (i) their educa-
tional background, (ii) years of experience in software development,
1 ABC, AXZ, XYZ, etc: Dummy names for company privacy. (iii) skills in the different phases of software engineering, and (iv)
Conference’17, July 2017, Washington, DC, USA M. Waseem et al.

level of experience with commonly used collaboration tools in • Project Development with ChatGPT: The development
software engineering projects. phase was carried out with the support of ChatGPT, which
Software Engineering and Development Experience: To facilitated the coding process. During this phase, the design
gain a comprehensive understanding of the participants’ knowl- blueprints were turned into functional software, adhering
edge, confidence, and practical experience, we administered a series to the defined specifications.
of 33 questions related to the typical phases of the software engi- • System Testing with Potential Clients: Engaging with
neering life cycle and activities. We employed a 5-level Likert scale potential clients for system testing ensured that the soft-
(e.g., ranging from “Not familiar at all” to “Extremely familiar”) ware met the expected standards and functioned as intended
to capture the developers’ responses. This section was organized in a real-world environment, allowing for any necessary
into subdivisions corresponding to the following software engi- adjustments before deployment.
neering life cycle phases: (i) Software Requirements Engineering, • System Deployment with ChatGPT: With the assistance
(ii) Software Design, (iii) Implementation, (iv) Testing and Quality of ChatGPT, the finalized software was deployed to the pro-
Assurance, and (v) Deployment and Release Management. Approx- duction environment, marking the transition from the devel-
imately 5 questions were posed for each phase. Additionally, we opment phase to making the software available for end-users.
included two open-ended questions to gather insights on partici- These activities were carried out iteratively, with continuous in-
pants’ expectations concerning the advantages and challenges of volvement of feedback from all stakeholders. The project initiators
integrating AI tools across different software engineering phases. monitored the development team’s progress every Monday, while
AI Tools Proficiency and Learning Readiness:In this survey potential clients held biweekly meetings with the development
section, we posed twelve closed-ended and 6 open-ended ques- team.
tions with the following objectives: (i) comprehensively evaluate
participants’ familiarity and comfort levels with commonly used
AI tools and technologies in software development, (ii) elicit their
3.4 Conducting Exit Survey
perspectives on the potential benefits, concerns, and integration In a similar way to the entry survey, we constructed and employed
of AI tools in the software development process, and (iii) identify a cross-sectional survey approach [27] for the exit survey (see the
effective approaches to enhance skill acquisition and learning. Exit Survey Questionnaire sheet in [43]), comprising 114 questions
aimed at comprehensively understanding ChatGPT’s impact on
software development. These questions are structured to assess
3.3 Development Process ChatGPT’s effectiveness, advantages, and limitations across vari-
The development process started with the identification of suitable ous software development phases, its educational value, challenges
projects from the AXZ country public procurement system, where encountered, and its potential for enhancing both project outcomes
many business opportunities within the Finland public sector are and developers’ skills. This survey questionnaire was intention-
made available. Project selection involved extensive discussions ally designed to address the Research Questions (RQs) outlined in
between the project initiators (i.e., the authors of this study) and the Section 1. We employed a 5-level Likert scale (e.g., ranging from
recruited developers. We selected one projects from software devel- “Strongly agree” to “Strongly disagree”) to capture the develop-
opment company, one from ABC university, and one from public ers’ responses. We organized the questions into 5 sections, each of
procurement system . The roles (e.g., front-end developer, back- which aligns with key aspects of the impact of ChatGPT in software
end developer) for each member of the development team were development projects, encompassing the following:
defined in advance. After the projects were chosen, consultations • ChatGPT Effectiveness Evaluation: This section contains
took place with representatives from potential clients, including questions that assess how much ChatGPT contributes to
XYZ software development company and ABC university. Upon different stages of software development, such as gathering
mutual agreement among all stakeholders, the recruited developers requirements, designing, coding, testing, and deployment.
performed the following activities in alignment with the software It also looks at how ChatGPT helps with communication,
development life cycle: finding conflicts, setting priorities, improving processes, and
decision-making.
• Collecting Requirements with ChatGPT: With the aid • Advantages and Limitations: In this section, the questions
of ChatGPT, the team gathered both functional and non- aim to find both the strengths and weaknesses of using Chat-
functional requirements, ensuring a comprehensive under- GPT. They explore how ChatGPT can speed up processes,
standing of the project’s needs, which is crucial for the sub- enhance communication, raise the quality of code, stimu-
sequent phases of the development cycle. late creative thinking, and identify challenges in software
• Setting up CI/CD Infrastructure with ChatGPT: Chat- development projects.
GPT’s support was instrumental in establishing the Continu- • Learning Curves and Skill Development: This section
ous Integration/Continuous Delivery (CI/CD) infrastructure, evaluates how interactions with ChatGPT impact the under-
which is essential for automating the deployment process, standing, skills, and self-confidence of individuals involved
ensuring a streamlined, error-free delivery of the software. in software development. It assesses how useful ChatGPT is
• Software Design using ChatGPT and Plant UML: Uti- for both experienced developers and newcomers to the field.
lizing ChatGPT alongside Plant UML enabled the team to • Development Proficiency: The questions in this section
design the software architecture effectively. This phase trans- focus on measuring the improvement of technical skills, es-
lated the gathered requirements into a coherent design blue- pecially in coding and programming languages, as a result of
print, setting a clear path for the development phase. incorporating ChatGPT into software development projects.
Using ChatGPT throughout the Software Development Life Cycle by Novice Developers Conference’17, July 2017, Washington, DC, USA

They also examine how ChatGPT affects problem-solving In the Implementation, Testing, and Quality Assurance stages,
abilities. the comfort and proficiency levels ranged from ‘Somewhat con-
• Development Obstacles: This section explores the difficul- fident’ to ‘Moderately confident’, and from ‘Not confident at all’
ties encountered while using ChatGPT, including challenges to ‘Somewhat proficient’ in activities such as writing code based
related to understanding complex requirements, architec- on design specifications and developing test strategies. The lack of
tural suggestions, and recommendations. It also discusses proficiency became even more pronounced in the Deployment and
Release Management stage, where the majority of the respondents
reported ‘No experience’ or indicated being ‘Not proficient at all’
3.5 Analysis in activities like preparing software applications for deployment
We employed descriptive statistics to analyze the quantitative (i.e., and planning release schedules. This collective data underscores
closed-ended questions). To evaluate the responses to open-ended a significant skill gap among the surveyed developers across all
questions, we employed open coding from grounded theory [20] to stages of software engineering, most notably in the Deployment
segment and label the survey data, thereby identifying and empha- and Release Management stage.
sizing the text fragments in each student’s response that could be Readiness to Learn about AI tools: In Figure 2, we also pro-
treated as distinct data points regarding ChatGPT. This approach en- vide a overview of the participants’ proficiency with AI tools, and
abled us to systematically identify and comprehend the perspective their readiness to learn and acquire new skills in software develop-
of students. ment. A thorough assessment was conducted through 16 questions
to get these aspects among the selected developers. Notably, the
data reveals that, five out of seven respondents, have not previously
4 RESULTS utilized AI tools in their software development projects, with the
4.1 Background, Experience, and Readiness exception of ChatGPT, indicating a relatively limited exposure to
Before initiating the actual development process, we conducted a AI technologies within their professional experiences. When as-
web-based survey with selected developers. The aim was to un- sessing their familiarity with various AI technologies on a scale
derstand their background, experience in software engineering, of 1 to 5, where 1 signifies “Not familiar at all” and 5 indicates
familiarity with various stages of the software development life “Extremely familiar”, participants displayed varying levels of profi-
cycle, and proficiency as well as readiness to learn about AI tools. ciency. They reported a higher familiarity with Machine Learning
Respondents utilized a five-level Likert scale and provided answers (ML) for Predictive Analytics and Deep Learning (DL) for Image or
to open-ended questions. Herein, we present the key results from Audio Processing, both scoring 5 on the familiarity scale.
the survey. Regarding the preferred methods for acquiring AI tool-related
Developers’ Background: The data from the survey, as shown skills in software development, the data shows a clear inclination to-
in Figure 2, indicate that the participants primarily have educational ward hands-on learning. “Hands-on experimentation and projects”
backgrounds limited to high school and possess minimal experience emerged as the top preference, closely followed by “Online tu-
in software development. Specifically, four participants reported torials or video courses”. Collaborating with peers on AI-based
having 1-2 years of experience, while three reported having less projects also garnered notable interest, suggesting a value in collab-
than a year of experience. It is important to clarify that this is not orative learning experiences. In contrast, reading research papers
professional experience, but rather reflects their level of study in un- and attending workshops or seminars on AI tools ranked lower in
dergraduate programs related to Computing. In self-assessing their preference. These findings collectively highlight the participants’
software engineering competencies, most participants categorized readiness to engage with AI tools, particularly through practical
themselves at an ‘Intermediate’ level, signifying a lack of advanced application, emphasizing the importance of accessible and experi-
expertise. Additionally, there was a notable lack of familiarity with ential learning opportunities in this domain.
agile methodologies among the respondents. Moreover, they dis- Perception and Potential of AI Tools: Through open-ended
played little to no proficiency in utilizing commonly employed questions, we also collected perceptions and potential of AI tools in
professional collaboration tools such as Jira. These observations (i) enhancing software development, (ii) concerns and limitations,
suggest that the participants might lack both the educational qual- and (iii) future prospects" of ChatGPT for software development.
ifications and practical skills commonly associated with roles in In the following, we provide a summary of participants’ responses,
advanced software development. denoting the participants with P1 to P7.
Experience in Software Engineering: In this section of the (i) Student participants express varied perceptions of AI tools
entry survey, we posed 35 questions covering various stages of the in software development. Students find AI beneficial, aiding them
software development life cycle, revealing a consistent trend as illus- to undertake complex projects with limited experience, especially
trated in Figure 2. The majority of respondents reported limited ex- with tools like ChatGPT ( P3). AI is recognized for its capability
perience and low comfort levels across different phases. Specifically, to quickly generate code and efficiently discover optimal tools and
in the Software Requirements Engineering phase, most participants algorithms, substantially reducing manual search time ( P4, P5).
indicated being ‘Somewhat familiar’ or ‘Somewhat comfortable’ However, concerns emerge regarding the quality of rapidly pro-
with activities such as analyzing and prioritizing software require- duced outputs and the complexity encountered when integrating
ments and defining functional and non-functional requirements. AI-generated components into a full system ( P1, P2). Hence,
This pattern persisted in the Software Development and Design while AI presents opportunities for enhanced productivity and
stage, where respondents generally felt ‘Somewhat comfortable’ or efficiency in certain development aspects, challenges persist in en-
‘Somewhat proficient’ with tasks like translating requirements into suring quality and seamless system integration. One representative
detailed designs and architecting software systems. quotation is depicted below.
Conference’17, July 2017, Washington, DC, USA M. Waseem et al.

¬ “I think AI tools will enhance productivity and efficiency of the Background Information (5 Questions)
Educational Background High School (7) 7 (100%)
software development process by generating code much faster than Engineering/Development Expereince 1 - 2 Years (4) <1 Years (3)
Education
humans. Also in the planning phase AI can be good tool to boost Skills in SE Phases

creativity by generating different scenarios and perspectives” ( P5, Requirements

Design
Expert (0)
Expert (0)
Novice (2)
Novice (2)
Intermediate (5)
Intermediate (5)
Advanced (0)
Advanced (0) SE Skills
Student Developer). Implementation Expert (0) Novice (3) Intermediate (4) Advanced (0)
Testing Expert (0) Novice (1) Intermediate (6) Advanced (0)
(ii) Student developers showed concerns about AI in software Deployment Expert (0) Novice (7) Intermed (0) Advanced (0) SE Expereince
development, highlighting issues related to privacy, ethics, devel- Familiarty with Agile Development and Relevant Tools

oper education, and reliability. P1 and P2 pointed to potential Kanban/Scrum etc.

Collaboration Tools None (0)
None (4)
Limited (5)
Lim (0) Moderate (3)

Moderate (2)
Ext (0)

Ext (0)
Exp (0)
Exp (0)
Agile
privacy breaches in AI-generated code and risks of unintentionally Version Control None (0) Limited (1) Moderate (6) Ext (0) Exp (0)
Development

sharing sensitive information. P4 discussed ethical issues related Issue Tracking

Proj. Management
None (7)
None (6)
Lim (0) Mod (0)
Limited (1) Mod (0)
Ext (0)

Ext (0)
Exp (0)
Exp (0)
Familiarity
Highest
to code ownership when private code is used in AI training. P3 Collab. Platform None (0) Limited (4) Moderate (2) Ext (0) Exp (0)
High
Neutral

highlighted a risk of hindering new developers’ skill growth due to Software Engineering and Development (33 Questions)
Low
Lowest

over-reliance on AI, while P7 noted concerns about the reliabil- Software Requirements Engineering
Elicit Reqs N.Familer(2) S.Familer(3) M.Familer (2) M.Familer (0) E.Familier (0)

ity of AI-generated code due to its non-deterministic nature. One Prioritise Reqs. N.Comf.(0) S.Comf.(6) M. Comf.(1) V. Comf. (0) E.Comf. (0)

representative quotation is depicted below.

N.Comf.(3) S.Comf.(2) M. Comf.(2) V. Comf. (1) E.Comf. (0)
Functional/Non-Func
Use-cases/Stories N.Exp (3) L.Exp (3) M.Exp. (1) E.Exp (0) Expert (0)

¬ “AI, while helpful for inexperienced developers, might hinder Reqs. Validation N. Skill.(3) S.Skill (2) M.Skill.(1) V.Skill (0) E.Skill (0) Analysis and
Requirements
Software Design and Architecture
their learning of the software development process by encouraging
Reqs. to Design N.Comf. (2) S.Comf. (3) M. Comf.(2) V. Comf. (0) E.Comf. (0)
over-reliance” ( P3, Student Developer). Create Architecture N.Comf. (2) S.Comf. (3) M. Comf.(2) V. Comf. (0) E.Comf. (0)

(iii) Students provided mixed feedback on integrating AI into Sys. Model/Diagram N.Prof. (2) S.Prof. (3) M. Prof.(1) V. Prof. (0) E.Prof. (0)

Identify Components
Comp/ interactions N.Skill. (0) S.Skill (4) M. Skill.(2) V. Skill. (1) E.Skill (0)
software development. P1 expressed concern about the sustain- Arch. Interfaces N.Conf. (0) S.Comf. (6) M. Conf. (0) V. Conf. (1) E.Comf. (0)
Designing
ability of AI solutions in the long run. P2 expects the development Data Flow and Struct N.Prof. (0) S.Prof. (4) M. Prof.(3) V. Prof. (0) E.Prof. (0)
and
Development Plans N.Exp. (2) S.Exp (4)
of AI tools that can read entire project directories to assist with
M. Exp. (0) V. Exp. (1) E.Exp. (0)
Architcting
Software Coding and Implementation
larger projects. P4 suggested developers need to improve in artic- Design-based code N.Conf. (0) S.Conf. (3) V. Conf. (2) V. Conf. (2) E.Conf. (0)

Prog. Lang/Tools
ulating problems for better utilization of Language Models. P5
N.Exp. (0) S.Exp (6) M. Exp. (0) V. Exp. (1) E.Exp. (0)

Coding Modules N.Prof. (0) S.Prof. (3) M. Prof.(3) V. Prof. (1) E.Prof. (0)

highlighted trust issues with AI, referencing its past inaccuracies, Integrate Modules N.Skill. (0) S.Skill (4) M. Skill.(3) V. Skill. (0) E.Skill (0)

despite its ongoing improvements. P6 saw potential in AI tools Code Reviews N.Exp. (3) S.Exp (3) M. Exp.(1) V. Exp. (0) E.Exp. (0)

Test for Code Comp N.Fam. (0) S.Fam. (6) S.Fam. (1) V. Fam. (0) E.Fam. (0) Source Code
to make software development workflows smoother, particularly Software Testing and Quality Assurance
and
Proramming
for new developers. The feedback ranges from recognizing poten- Test Strategy/Plans
Functional Testing
N.Conf. (2)
N.Prof. (0)
S.Conf. (5)
S.Prof. (7)
M. Conf. (0)
M. Prof.(0)
V. Conf. (0)
V. Prof. (0)
E.Conf. (0)
E.Prof. (0)

tial advantages of AI tools to pointing out notable challenges such Non-functional Test N.Skill. (3) S.Skill (4) M. Skill.(1) V. Skill. (0) E.Skill (0)

as trust and effective utilization. One representative quotation is Identify/Report Bugs N.Exp. (0)

N.Fam. (6)
S.Exp (7)

N.Fam. (1)
M. Exp.(0)
M.Fam.(0)
V. Exp. (0)

V. Fam. (0)
E.Exp. (0)

E.Fam. (0)
Regression Testing
depicted below. Acceptance Testing N.Comf. (3) S.Comf. (4) M. Comf.(0) V. Comf. (0) E.Comf. (0)
Testing and
Validation
¬ “Probably something is going to come out that can read the Complinace with Stds. N.Comt. (0) S.Comt. (2) M. Comt.(4) V. Comt. (1) E.Comt. (0)

Software Deployment and Release Management

entire project directory and all it’s files to help with bigger projects Deployment Plan N.Exp. (6) S.Exp (0) M. Exp.(0) V. Exp. (1) E.Exp. (0)

(to my knowledge copilot X does this already to some extent) ” ( P2, Instl./Deploy Script N.Fam. (3) S.Fam. (4) M.Fam.(0) V. Fam. (0) E.Fam. (0)
Deployment
Release Scheduling N.Prof. (6) N.Prof. (0) M. Prof.(1) V. Prof. (0) E.Prof. (0)
Student Developer). Deploy Team Coord N.Comf. (4) S.Comf. (0) M. Comf.(2) V. Comf. (1) E.Comf. (0)
Expereince/
Deploy for Production N.Skill. (5) S.Skill (1) M. Skill.(1) V. Skill. (0) E.Skill (0)
Comfortability
Post Deploy Checks
[ Takeaways N.Conf. (5) S.Conf. (2) M. Conf. (0) V. Conf. (0) E.Conf. (0) Highest
Software Update N.Exp. (5) S.Exp (0) M. Exp.(2) V. Exp. (0) E.Exp. (0) High
Neutral
Post-deploy Issues N.Comf. (4) S.Comf. (3) M. Comf.(0) V. Comf. (0) E.Comf. (0) Low
Significant Skill Gaps: The surveyed developers showed notable Knowledege and Learning for AI Tools (6 Questions) Lowest

gaps in their software engineering skills, especially in deployment Knowledge and Expreince with AI Tools for Software Engineering and Development
Yes Not
Scale of 1 to
Not Sure Automated Code Generation 1 5 1
and release management stages. (1) (1)
Automated Testing/Debugging 1 5 1 2
Familiar 5, with 1
being "Not
Eagerness to Learn AI: Despite minimal previous experience Expereince
with AI NLP for Requirements Engr. 3 3 1 3 familiar at all"
and 5 being
with AI tools, participants demonstrated a strong desire to learn, Development
Tools No ML for Predictive Analytics 5 1 1 4
Extremely
"Extremely
particularly through hands-on experiences and projects. (5) DL for Image/Audio Processing 5 1 1 5 Familiar familiar"
Preference for New Sikills on AI tools in Software Engineering and Development
Most
Not Sure Yes
Online tutorials/resources 1 2 3 1 Preferred With 1 being
(2) (3) Collaborating with peers 4 1 1 1 2 the most
4.2 Effectiveness, Advantages, and Limitations Learning about
AI Skills/Tools
Reading research papers 2 1 4
3 preferred and
5 being the
(RQ1) for Software
Development Sessions AI tools
Hands-on experiment 1
1 1

1
1

3
4

2
4
least
No 5 Least preferred:
To answer the RQ1, we asked 71 survey questions to gather answers (2) Other 1 2 3 1
Preferred

from students who extensively used ChatGPT for three months to

develop three systems as detailed in Section 3. Figure 2: Overview of survey results regarding students’ back-
Effectiveness: The survey results indicate (see Figure 3 that grounds, experience, and readiness
ChatGPT has made a significant positive impact on the initial
phases of software development projects. In requirements analysis,
developers reported that ChatGPT significantly reduced the time helped align architectural decisions with project constraints and
required for defining requirements, with 7 respondents agreeing or effectively resolved architectural risks and issues, as 7 respondents
strongly agreeing. It also facilitated the accurate identification of agreed or strongly agreed. These findings suggest that ChatGPT
needs and requirements, as 6 respondents agreed or strongly agreed. plays a valuable role in streamlining the early stages of software de-
In the design phase, ChatGPT led to clearer and more scalable archi- velopment, improving efficiency, and accuracy. In the development
tectures, with 5 respondents agreeing or strongly agreeing. It also phase, ChatGPT continued to be a beneficial tool. It streamlined
Using ChatGPT throughout the Software Development Life Cycle by Novice Developers Conference’17, July 2017, Washington, DC, USA

the coding process, reducing errors and improving code quality, testing limitations were perceived as an issue by some, with more
as reported by 6 students who agreed or strongly agreed. In the students in the neutral category. Overlooked defect risk had mixed
testing phase, ChatGPT optimized testing processes, enabling early feedback, with some students in agreement and some strongly
defect detection and resolution. It also enhanced test coverage and agreeing. Testing objective misalignment, impractical deployment
effectiveness, with 3 respondents agreeing or strongly agreeing. recommendations, and manual verification neglect were generally
Additionally, it efficiently identified bugs and vulnerabilities, im- met with neutrality but had pockets of agreement. Production en-
proved testing risk management, and enhanced software reliability vironment misalignment received mostly positive feedback, with 5
and stability, as noted by 4 to 6 respondents who agreed or strongly students in agreement and 2 strongly agreeing.
agreed. These results highlight the utility of ChatGPT in enhancing
the software development process from coding to quality assur-
ance. The deployment phase, students also get significant help from [ Takeaways
ChatGPT. It led to a reduction in deployment issues, optimized
deployment processes, and resulted in smoother and more reliable Notable Positive Impact: ChatGPT positively influences all
phases of the software development life cycle, enhancing efficiency,
deployments, as reported by 6 respondents who agreed or strongly
accuracy, and collaboration throughout each stage.
agreed. It also improved deployment risk management, enhanced Varied Perceptions on Utility and Constraints: students ex-
deployment monitoring and support, ensured software compati- hibit mixed opinions on the advantages and limitations of ChatGPT,
bility in production, and enabled quicker rollbacks and updates in appreciating its facilitation of clarity and efficiency, while express-
deployment, with 4 to 7 respondents agreeing or strongly agreeing. ing concerns regarding design simplification and coding standard
Advantages: In the survey assessing the perceived advantages alignment
of using ChatGPT at each phase of the software development life
cycle, students provided responded to its utility (see Figure 3). En-
hanced requirement clarity was identified as a key benefit, with 7 4.3 Learning Curves and Skill Development
students agreeing that ChatGPT contributes to better understand-
(RQ2)
ing of project requirements. Additionally, ChatGPT was seen as a
facilitator for stakeholder communication, although opinions were To get the answer of RQ2, we asked 11 survey questions to gather
mixed, with 4 students neutral and 3 in agreement. Rapid conflict students’ opinions on the influence of ChatGPT on their learning
resolution, innovative design formulation, and efficient design vali- and skill development. The survey results highlight the partici-
dation also received mixed responses, indicating the diversity of pants’ experiences and perceptions regarding several aspects of
opinions within the group. However, user-cantered design promo- using ChatGPT in their software development projects (see Fig-
tion, software architecture robustness, and informed architectural ure 3). For instance, when asked about “Reducing the time needed
decision-making had predominantly positive feedback, with most to understand software development concepts”, the feedback was
students agreeing on their advantages. Clear architecture represen- mostly positive: 3 participants agreed, and 3 strongly agreed. Simi-
tation was another strong point, with 6 students strongly agreeing larly, “Enhancing adaptability to new technology” was positively
on its benefits. In terms of coding, students acknowledged that received with 5 strongly agreeing and 1 agreeing. The aspects “Im-
ChatGPT optimized coding practices, although this received mixed proving practical problem-solving” and “Enhancing coding skills”
opinions. Testing efficiency improvement, early defect resolution, were also viewed favorably, receiving mainly positive feedback with
deployment strategy optimization, smooth software deployments, 4 strongly agreeing and 2 agreeing to the former, and 4 strongly
and deployment risk management were generally seen as advanta- agreeing and 2 agreeing to the latter. However, it is worth noting
geous by many students. that one participant strongly disagreed with the latter point. For
Limitations: The survey also explored the limitations associ- “Improving code efficiency”, 4 participants agreed and 2 strongly
ated with using ChatGPT during software development. Complex agreed, despite 1 participant strongly disagreeing. Feedback on
requirement limitations were met with a mix of opinions, with “Enriching software knowledge” was overall positive and seems to
3 students in disagreement and 4 neutral. Analysis dependency indicate that ChatGPT was helpful in this regard.
risk saw varied responses, with 4 students agreeing and 2 neutrals. Other aspects such as “Enhancing project communication” and
However, objective misalignment was mostly met with neutrality, “Providing unique insights” were perceived variably, though still
as 5 students fell into this category. Design oversimplification was leaned towards a positive view. For example, “Increasing task confi-
perceived as a limitation, although the opinion was divided, leaning dence” had 4 strongly agreeing and 2 agreeing. On the other hand,
slightly towards agreement. Impractical design recommendations “Boosting self-efficacy” saw 2 strongly agreeing and 4 agreeing, with
were generally neutral with a subtle tendency towards agreement, 1 participant disagreeing, showing that while ChatGPT was gener-
with one respondent strongly agreeing. students expressed concern ally seen as a useful tool, experiences varied among the students in
about potential creativity hindrance, with a significant portion in some areas.
disagreement and a few neutral or slightly in agreement. Archi-
tectural expertise neglect and architectural constraint conflict also
received mixed opinions, with the majority neutral. Coding stan- [ Takeaways
dard misalignment was seen as a limitation, as most students fell Positive Impact on Learning and Skill Development: Chat-
into the neutral or agreement categories. The issue of critical think- GPT was generally perceived to positively impact various facets
ing reduction was raised, with a majority expressing disagreement. of software development learning and skill development, despite
Coding efficiency issues garnered agreement from most students, in- some isolated instances of dissent or neutral perspectives from
dicating that ChatGPT can pose challenges in this aspect. Complex participants.
Conference’17, July 2017, Washington, DC, USA M. Waseem et al.

4.4 Proficiency in Software Development (RQ3) challenges, participants showed a high level of satisfaction and
In addressing RQ3, we asked 16 survey questions. A agreed sen- perceived benefit from using ChatGPT. A notable 6 out of 7 respon-
timent of approval was observed concerning ChatGPT’s role in dents were satisfied with ChatGPT’s contributions to their projects,
enriching software knowledge (see Figure 3), with all 7 respondents indicating a high level of acceptance and value despite any hurdles
either agreeing or strongly agreeing. This indicates a consistent they faced. Moreover, when asked whether the benefits of using
acknowledgment of ChatGPT’s capacity to enhance foundational ChatGPT outweighed the challenges faced, 6 out of 7 responded
understanding and proficiency in software development among the with agreement or strong agreement. This highlights that, despite
participants. Further, the efficacy of ChatGPT was evident in sev- the observed challenges, ChatGPT was seen as a beneficial addition
eral other facets of the software development learning process. For to the development projects, assisting students in their developmen-
example, 6 out of the 7 respondents concurred (agree or strongly tal journey, while also providing them with practical, real-world
agree) that the tool positively impacted their adaptability to new challenge experiences.
technologies, boosted practical problem-solving skills, and encour-
aged innovative thinking during their software development. This [ Takeaways
suggests that ChatGPT is not only a knowledge resource but also a
Valuable Learning Despite Challenges: While encountering
catalyst that fosters a conducive environment for cultivating cru-
challenges with ChatGPT, participants unanimously agreed that
cial soft skills such as adaptability and innovative thinking among they got valuable learning and insights from these experiences,
students. suggesting that obstacles faced were constructively impactful on
On the other hand, it is noteworthy that certain aspects yielded their developmental journey.
more varied perceptions, providing a view of the participants’ expe-
riences. Specifically, regarding the statement, “ChatGPT provided
unique insights,” there was an almost equal distribution of responses 5 DISCUSSION
across all available options (strongly disagree to strongly agree), 1 Positive Impact on Software Development Phases: Chat-
showcasing that the participants had diverse experiences and per- GPT’s reported facilitation of various software development phases,
ceptions concerning ChatGPT’s capability in offering unique in- from requirements analysis to deployment, aligns with existing lit-
sights during the software development process. This divergence erature (e.g., [36, 37]) highlighting the beneficial role of AI in stream-
in perceptions reflects the varied experiences developers have with lining development workflows. For example, AI-driven tools have
artificial intelligence tools and underscores the complexity of its been substantiated for enhancing requirement gathering through
application in an educational context. natural language processing capabilities [35] and facilitating design
phases [1]. It is important to acknowledge that the findings are
[ Takeaways based on the experiences of "seven students," which may limit the
generalizability of the insights to more seasoned professionals or
Proficiency in Software Development: ChatGPT consistently diverse development contexts. Also, the participant’s varied famil-
enhanced foundational understanding and soft skills in software iarity and comfort with AI tools might influence their perception
development among participants, but experiences varied notably
and reported impact of using ChatGPT. The positive impact across
in its ability to provide unique insights during the software devel-
development phases prompts deeper explorations into specifying
opment process.
contexts, projects, or phases where ChatGPT’s application could be
maximized. Additionally, studies comparing the utility of ChatGPT
4.5 Challenges (RQ4) across different developer expertise levels and various project com-
In addressing RQ4, we asked 11 survey questions to understand the plexities and scales can provide nuanced insights into its adaptable
difficulties of students might have experienced while using Chat- utility.
GPT in their projects. When it came to difficulties in various phases 2 Diverse Opinions on Advantages and Limitations: The
of project development with ChatGPT, the students generally did presence of both significant advantages and noticeable limitations
not find using ChatGPT hard or tricky. For example, in dealing in using ChatGPT for software development echoes previous lit-
with ChatGPT usage difficulties, 4 out of 7 respondents disagreed, erature (e.g., [21, 24]) on AI tools (e.g., [46]) in development envi-
suggesting that most found it user-friendly. Also, a similar pat- ronments. In terms of advantages, the recognized enhancement in
tern was seen in integration challenges and requirements analysis requirement clarity and promotion of user-centered design correlate
hurdles with ChatGPT, where 5 out of 7 respondents disagreed, with the narrative of AI-driven tools being useful in translating user
showing that most found ease in both integrating ChatGPT and in needs into technical requirements efficiently [4]. The support in
the requirements analysis phases. clear architecture representation and testing efficiency is coherent
Even with the overall positive feedback regarding usability, there with studies that underscore automated testing (e.g., [7]) and model-
were areas where participants felt they learned and improved their driven development (e.g., [34]) facilitated by AI. In contrast, the
skills from the challenges they faced. A key observation was that limitations, particularly around coding efficiency issues and design
all participants (7 out of 7) agreed or strongly agreed that they recommendation impracticalities, mirror concerns found in both
experienced learning and skill growth from the challenges faced in peered review (e.g., [5]) and gray literature (e.g., [12, 28]) that
while using ChatGPT. In the same way, all respondents agreed cautions against over-reliance on AI for complex decision-making
or strongly agreed that they gained valuable insights from the in software development. It’s worth noting the manifestation of
challenges faced with ChatGPT. This consistent positive response coding standard misalignment and concerns over creativity hin-
highlights that challenges can indeed foster skill enhancement drance, which draw parallels with discussions on AI potentially
and generate insights among students. Even though there were stifling innovative problem-solving when it becomes a primary
Using ChatGPT throughout the Software Development Life Cycle by Novice Developers Conference’17, July 2017, Washington, DC, USA

decision-making tool [3, 31]. ChatGPT shows promise for enhanc- might be a limitation in exploring the specific nature and impact of
ing parts of the Software Development Life Cycle (SDLC) but must these challenges due to the small participant group and the quantita-
be used wisely due to its limitations, like possibly oversimplifying tive approach of a survey. Investigating the specific challenges and
complex tasks or suggesting unhelpful designs due to its partial exploring how these were transformed into learning experiences
understanding of context and software tasks. Although particularly through a qualitative lens could be a prospective direction. Addi-
helpful in stages needing clear communication, like requirement tionally, understanding how these challenges align with theoretical
analysis and design, careful strategies are needed to avoid issues frameworks of problem-based learning and how such experiences
in areas like coding where it might be less accurate or efficient. can be scaffolded for optimized learning in future implementations
Developers’ mixed responses highlight its varied efficacy across of ChatGPT in educational settings may be explored.
the diverse and complex tasks within SDLC phases, advocating for
a thoughtful integration in the process. 6 THREATS TO VALIDITY
3 Positive Impact on Learning and Skill Development:
Construct Validity: The primary threat to construct validity lay
The majority of students found ChatGPT beneficial in aiding their
in the accuracy of the survey instruments in measuring develop-
understanding and skill enhancement in various facets of software
ers’ learning experiences and the impact of ChatGPT on software
development, which aligns with literaturesuggesting that AI can
development phases. To mitigate this threat, we refined the survey
expedite the learning curve by providing instant, context-aware
questions through a pilot survey administered with undergradu-
assistance (e.g., [2, 18]). However, the dissenting responses, partic-
ate students who were not involved with the development team.
ularly the one participant who strongly disagreed about enhancing
This ensured a broad range of perspectives in evaluating the clarity
coding skills, indicating that personal experiences with ChatGPT
and unambiguity of the questions. Moreover, two professors spe-
varied. ChatGPT, acting as a supplementary tool, seems to align
cializing in software engineering and educational methodologies
with these findings by providing support and instant feedback, en-
reviewed and provided constructive feedback on the survey instru-
hancing both the theoretical and practical understanding of the
ments, further confirming their appropriateness in measuring the
students. However, it is crucial to consider the small sample size of
intended constructs. We incorporated the feedback to enhance the
7 students, which may limit the generalizability of these findings.
clarity and relevance of our survey questions.
Positive impacts on skill development might be subjected to the
External Validity: While our findings, based on the experiences
initial proficiency level of the participants or other unaccounted
of seven undergraduate students, provide valuable insights, their
contextual factors. Future studies might focus on a more diversified
generalizability to seasoned professionals or diverse development
and larger cohort to validate these findings, and examine the spe-
contexts may be limited. The varied familiarity and comfort with
cific components of ChatGPT that contribute to effective learning
AI tools among participants could have influenced their percep-
and skill development, while also exploring the facets where users
tion and reported impact of using ChatGPT. Although the study
had divergent views.
aligns with existing literature on the positive impacts of AI on
4 Varied Experiences about Software Knowledge and Soft
software development, the specific context and small sample size
Skills Development: Even though ChatGPT positively impacted
may constrain the external validity. Future studies involving larger
software knowledge and soft skills development, it provided varied
and more diverse sample sizes, as well as different developmental
experiences regarding unique insights into the software develop-
contexts, are recommended to enhance the external validity of our
ment process among participants. The varied experiences with
findings.
ChatGPT mirror the existing literature on AI in education, which
Conclusion Validity: Ensuring conclusion validity was crucial
suggests that while AI can offer valuable support, the utility can
to derive reliable conclusions from the data. We employed descrip-
differ based on users’ expectations, existing skills, and the nature
tive statistics and constant comparison techniques, providing a
of tasks [13, 18]. The disparity in how ChatGPT’s capability to
robust methodological framework for analyzing the survey data.
provide unique insights was perceived implies that the tool might
The logical basis for deriving conclusions was clearly delineated,
be interpreted or utilized differently among users. The limitation
supported by adequate evidence from the data and corroborated by
here pertains to understanding the depth and nature of these varied
existing literature, establishing a solid foundation for conclusion va-
experiences, given that the reasoning behind such divergence is not
lidity. Nevertheless, the diverse opinions and experiences reported
explored in detail. A qualitative approach through interviews or
by participants highlight the complexity of drawing conclusive
focus groups in future studies might shed light on why some users
insights. Further studies with larger or more diversified cohorts,
find ChatGPT more insightful than others, and develop strategies
coupled with additional methodological approaches, may help in
or guidelines to optimize the utilization of ChatGPT for a wider
reinforcing the conclusion validity and providing a more compre-
audience.
hensive understanding of the impacts of ChatGPT on software
5 Valuable Learning Despite Challenges: Challenges faced
development phases.
when using ChatGPT were not seen as drawbacks but rather as
valuable learning experiences, aiding skill development and in-
sight generation among students. This aligns with the pedagogical 7 CONCLUSIONS
perspective that views challenges and obstacles as crucial learn- In this study, we explored the impact of ChatGPT on students en-
ing elements that facilitate deep understanding and skills mastery gaged in software development projects. It unveiled significant skill
[11, 33, 47]. Challenges faced, while initially perceived as hurdles, gaps among participants but also their eagerness to learn AI. Chat-
become opportunities for active learning and skill enhancement. GPT was identified as a valuable support tool, enhancing efficiency
Given the unanimously positive feedback regarding learning from and collaboration across software development phases. However,
challenges, there may be potential bias in the responses, or there there were mixed opinions regarding its advantages and limitations,
 0
1
0
3
3
Reduced time for develop: concepts 14% 43% 43%

0
0
1
1
5
ChatGPT adaptability to new tech. 14% 14% 72%

0
1
0
4
6

0
0
0
1
0
0
4
3

1
ChatGPT improved problem-solving 14% 29% 57%
Enhanced Requirement Clarity 86% 14%

1
0
0
2
Less time in requirements definition 57% 43%

4
0
0
4
3
0
0
0
1
6
0
ChatGPT enhanced coding skills 14% 29% 57%
Stakeholder Commun: Facilitation 57% 43%

4
Accurate identification of needs & reqs. 86%

1
0
0
2
14%

0
0
1
6
0
0
2
2
3
0
ChatGPT improved code efficiency 14% 57% 29%
Rapid Conflict Resolution 86%

0
2
14% Stakeholder comm: in req. gathering

0
4
28.5%

1
28.5% 43%

0
1
2
3
1
0
0
3
ChatGPT enhanced project commun: 4
0

29% 57% 14%

Efficient Design Validation 14% 29% 43% 14% Comprehensive req. documentation

0
0
2
43% 57%

0
0
2
0
5
0
1
2
3
1

ChatGPT encouraged innovation 14% 57% 29%

User-Centered Design Promotion 29% 71% Effective req. prioritization

0
0
0
3
14% 28% 43% 14%

4
0
0
1
5
1

Skill Development
0
2
2
3
0

ChatGPT enrich software knowledge 43% 57%

Learning Curves &

Software Architecture Robustness 14% 72% 14% Early detection of conflicting reqs. 28.5%

1
1
2
28.5% 43%

2
1
0
0
2
1

4
0
0
1
2
4

ChatGPT provided unique insights 14% 14% 28% 28% 14%

Informed Architect: Decision-Making 29% 57% 14% Better grasp of scope & objectives 14% 29%

0
0
1
2
57%

4
0
0
0
6
1
0
0
0
2
5
Conference’17, July 2017, Washington, DC, USA

ChatGPT increased task confidence 14% 29% 57%

Clear Architecture Representation 86% 14% Improved software architecture

0
1
0
4
29% 71%

2
0
1
0
2
4
0
0
2
5
0

ChatGPT boosted self-efficacy 14% 57% 29%

Optimized Coding Practices 14% 29% 57% Effective risk resolve in architect: 29% 71%

0
1
4

0
2
0
0
2
3
2

Robust Code Development 14% 57% 29% Scalable, maintainable architecture 28.5% 43% 28.5%

0
1
2
1
0
0
0
4

0
2
0
0
5
3

Grip of software develop: concepts Developer Collaboration Enhancement 14% 43% 29% 14% Better architect: decision-making 57% 43%
28% 72%

0
1
2
0

4
0
0
0
4

0
1
3
3

0
3
Testing Efficiency Improvement 14% 29% 57% Clearer system architect: depiction 57% 43%
Applying development principles 14% 43% 43%

0
0
1
1
0
0
0
5

5
2

0
1
0
1
Early Defect Resolution 14% 72% 14% Architect: decisions with project curb 71% 29%
Proficiency in coding & languages 14% 14% 72%

0
0
3
4
0
0
0
2
3

2
2
2

1
1
1
Testing Risk Management 43% 57% Architect: performance & reliability 28.5% 43% 28.5%

Advantages
Writing efficient, error-free code 14% 29% 29% 14% 14%

0
0
2
4
1
2

0
1
0

0
2
3
4

1
1
Strongly
Disagree

Deployment Strategy Optimization

Effectiveness Evaluation
29% 14%

proficiency in software development, and challenges.

Problem-solving ability 14% 29% 43% 14% 57% Streamlined coding process 14% 29% 57%

0
0
5
2

6
0
0
1
2

0
0
4

1
0
Smooth Software Deployments 72% 29% Reduced coding errors & issues 14% 57%
Critical thinking & effective solutions 14% 86% 29%

0
0
1
4

Proficiency in
2
0
0
3
3
1

0
0
1
4
Deployment Risk Management 14% 57% 29% Improved code quality 43% 43% 14%
Knowledge of development phases 29% 14% 57%

0
0
1
6
0
0
0
0
2

0
1
2
3
5

Software development
Innovative Design Formulation 14% 86% Reduced development time 29% 71%
Understanding of project phases 14% 29% 43% 14%
Disagree

0
3
4
0
0
0
1
2
2
2

1
3

2
1
0
Complex Requirement Limitations 43% 57% Better developer commun & collaborate 14% 28% 29% 29%
Proficiency in testing & debugging 14% 29% 43% 14%

0
0
2
4
1
0
0
0
2
5

1
0
2
3
0
Analysis Dependency Risk 29% 57% 14% Effective code integration 29% 71%
Identifying & resolving defects 14% 29% 43%

0
1
0
5
1
0
0
3
1

0
0
4

2
1
Objective Misalignment 14% 72% 14% Optimized testing process 43% 43% 14%
Understanding design principles 29% 57% 14%

0
2
1
3
1
Neutral

0
1
3
3
0

0
1
2
1
3
Design Oversimplification 29% 14% 43% 14% Early defect detection & resolution 43% 43%
14%

0
0
3
Creating effective software designs

3
14% 29% 14%
1

43%
0
0
4
3
0

Impractical Design Recommendations 43% 43% 14%

0
0
1
3
Improved test coverage & effectiveness 57% 43%

3
1
3
3
0
0
0
0
4

Collaborating & communicating effectively 43%

1
2

14% 43%
Creativity Hindrance 14% 43% 43% Efficient bug & vulnerability detection 57% 29%
14%

0
0
0
7
0
0
2
Agree

4
1
0
0
0
4
3
0

Conveying complex concepts 100% Creativity Hindrance 29% 57% 14% Improved testing risk management 57% 43%

0
1
4
0
1
0

0
2
5
1
0
1
2
3
1

Time management & task prioritization Architectural Expertise Neglect 14% 72% 14% Enhanced software reliability & stability 14% 29% 43% 14%

0
1
2
4
0

0
0
0
1
2

1
3
2
2

3
Productivity in completing tasks 14% 43% 43% Architectural Constraint Conflict 14% 29% 57% Enhanced confidence in software quality 14% 28% 29% 29%

1
0
0
4
Agree

2
0
1
0
4
2
Strongly

Coding Standard Misalignment 14% 57% 29% Reduced deployment issues 14% 57% 29%

1
1
2
3
0

3
0
1
0

1
3
0
0
4
2

Critical Thinking Reduction 14% 14% 29% 43%

43% Optimized deployment process
0
0
2

ChatGPT usage difficulties 14% 43% 14% 57%

5

29%
0

1
4
2
0
0
Coding Efficiency Issues 29%
0

71%
0
1
2
4

57% 29%
0
1

Integration challenges with ChatGPT 14%

4
2
0

Smoother, reliable deployments

0
5
2
0
0
Complex Testing Limitations 14% 57% 29%
72% 14% 29% 57%
0
0
3

Reqs analysis hurdles with ChatGPT 29%

4
0

Limitations

5
0
0
2

1
4
1

0
1
0
Overlooked Defect Risk 43% 57%

Challenges
Improved deployment risk management 29% 57% 14%
0
1
2

Design phase obstacle with ChatGPT 14% 72% 14%

4
0

2
0
0
3

1
0
0
Testing Objective Misalignment 14% 29% 57%
4
0

57% 14%
0
1

Develop: phase difficulties with ChatGPT 29%

4
2
0

Better deployment monitoring & support 43% 57%

1
0
1
0
Impractical Deployment Recommendations 14% 57% 29%
2

ChatGPT's impact on project quality 14% 72% 14%

0
0
0
0
3

3
2
3
1

0
1
2
1
Manual Verification Neglect 29% 43% 29% Ensured software compatibility in production 43% 43% 14%
Reducing ChatGPT-based challenges 14% 43% 29% 14%
0
0
1
5
1

0
0
0
4
3
3

1
3

Production Environment Misalignment 14%

0
0

14% 72%
Learning & skill growth from challenges 57% 43%
Quicker rollbacks & updates in deployment 14% 43% 43%

0
0
0
4
3
Valuable insights from ChatGPT obstacle 57% 43%

0
0
1
1
5
Satisfaction with ChatGPT contributions 14% 14% 72%

0
1
0
5
Benefits outweighing ChatGPT challenge 14% 14% 72%
M. Waseem et al.

Figure 3: Overview of survey results about ChatGPT effectiveness evaluation, advantages, limitations, skill development,
Using ChatGPT throughout the Software Development Life Cycle by Novice Developers Conference’17, July 2017, Washington, DC, USA

particularly concerning design and coding. Notably, ChatGPT con- [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
sistently improved participants’ software development proficiency Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Association
and soft skills, though its ability to provide unique insights varied. for Computational Linguistics: Human Language Technologies (NAACL), Volume 1
Despite facing challenges, all participants considered the experience (Long and Short Papers). ACL, 4171–4186.
[17] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Sid-
beneficial for their learning journey, highlighting the pedagogical dharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaud-
potential of AI tools in overcoming obstacles. hary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard
These findings emphasize the need for a balanced approach in in- Grave, Michael Auli, and Armand Joulin. 2020. Beyond English-Centric Multilin-
gual Machine Translation. arXiv:2010.11125
tegrating AI tools like ChatGPT into software development. While [18] Mohammad Fraiwan and Natheer Khasawneh. 2023. A Review of ChatGPT
they offer substantial benefits, developers should maintain an in- Applications in Education, Marketing, Software Engineering, and Healthcare:
dependent problem-solving mindset. For future research, we are Benefits, Drawbacks, and Research Directions. arXiv:2305.00237 (2023).
[19] Giancarlo Frosio. 2023. The Artificial Creatives: The Rise of Combinatorial
planning to explore deeper into understanding the varied expe- Creativity from Dall-E to GPT-3. Handbook of Artificial Intelligence at Work:
riences, optimizing ChatGPT’s use with the help of multi-agents, Interconnections and Policy Implications (Edward Elgar, Forthcoming) (2023).
[20] Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory:
and developing guidelines for effective AI integration across vari- Strategies for qualitative research. Routledge.
ous software development phases, aligned with undergraduate and [21] Walid Hariri. 2023. Unlocking the Potential of ChatGPT: A Comprehensive
graduate computing courses. Exploration of its Applications, Advantages, Limitations, and Future Directions
in Natural Language Processing. arXiv:2304.02017 (2023).
[22] James D. Herbsleb. 2007. Global Software Engineering: The Future of Socio-
DATA AVAILABILITY technical Coordination. In Future of Software Engineering (FOSE). 188–198.
[23] Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023.
We provide a dataset containing the entry questionnaire, exit ques- Chatgpt and software testing education: Promises & perils. In Proceedings of the
tionnaire, and analyzed data for other researchers to replicate and 2023 IEEE International Conference on Software Testing, Verification and Validation
validate our study [43]. Workshops (ICSTW). IEEE, 4130–4137.
[24] Dinesh Kalla and Nathan Smith. 2023. Study and Analysis of Chat GPT and its
Impact on Different Fields of Study. International Journal of Innovative Science
REFERENCES and Research Technology 8, 3 (2023).
[1] Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, [25] Junaed Younus Khan and Gias Uddin. 2022. Automatic code documentation gen-
Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards human-bot eration using gpt-3. In Proceedings of the 37th IEEE/ACM International Conference
collaborative software architecting with chatgpt. In Proceedings of the 27th on Automated Software Engineering (ASE). IEEE, 1–6.
International Conference on Evaluation and Assessment in Software Engineering [26] Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Sali-
(EASE). 279–285. nas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan
[2] Mohamad Iyad Al-Khiami and Martin Jaeger. 2023. Leveraging ChatGPT in Awadalla. 2021. Scalable and Efficient MoE Training for Multitask Multilingual
Android App Development: A Case Study on Supporting Novice Developers in Models. arXiv:2109.10465
Creating Successful Apps. Preprints (2023). [27] Barbara A Kitchenham and Shari L Pfleeger. 2008. Personal opinion surveys. In
[3] Abdulrahman Al-Surmi, Mahdi Bashiri, and Ioannis Koliousis. 2022. AI based Guide to Advanced Empirical Software Engineering. Springer, 63–92.
decision making: combining strategies to improve operational performance. [28] Anastasia Kovalevskaya. 2023. Pros and Cons of Using AI for Web Development:
International Journal of Production Research 60, 14 (2022), 4464–4486. Stumbling Blocks You Have to Know About. https://litslink.com/blog/using-ai-
[4] Oriana Arnone. 2021. Machine learning for designers. Enable designers to merge for-web-development Accessed: September 12, 2023.
users’ needs with machine learning capabilities in the idea generation and problem [29] Essi Lahtinen, Kirsti Ala-Mutka, and Hannu-Matti Järvinen. 2005. A study of the
framing phases. Master’s thesis. Politecnico di Milano. difficulties of novice programmers. ACM SIGCSE Bulletin 37, 3 (2005), 14–18.
[5] Marco Barenkamp, Jonas Rebstadt, and Oliver Thomas. 2020. Applications of AI [30] Timothy C Lethbridge, Susan Elliott Sim, and Janice Singer. 2005. Studying soft-
in classical software engineering. AI Perspectives 2, 1 (2020), 1. ware engineers: Data collection techniques for software field studies. Empirical
[6] Feras A Batarseh, Rasika Mohod, Abhinav Kumar, and Justin Bui. 2020. The Software Engineering 10, 3 (2005), 311–341.
application of artificial intelligence in software engineering: a review challenging [31] Rebecca Marrone, Victoria Taddeo, and Gillian Hill. 2022. Creativity and artificial
conventional wisdom. Data Democracy (2020), 179–232. intelligence—A student perspective. Journal of Intelligence 10, 3 (2022), 65.
[7] Dhaya Sindhu Battina. 2019. Artificial intelligence in software test automation: [32] Aishwarya Narasimhan, Krishna Prasad Agara Venkatesha Rao, et al. 2021.
A systematic literature review. International Journal of Emerging Technologies CGEMs: A metric model for automatic code generation using GPT-3.
and Innovative Research (2019), 2349–5162. arXiv:2108.10168 (2021).
[8] Leila Bencheikh and Niklas Höglund. 2023. Exploring the Efficacy of ChatGPT in [33] Jon Ohlsson. 2014. Pedagogic challenges in the learning organization. The
Generating Requirements: An Experimental Study. Bachelor’s Thesis, Chalmers Learning Organization 21, 3 (2014), 162–174.
University of Technology. [34] Elena Planas, Gwendal Daniel, Marco Brambilla, and Jordi Cabot. 2021. Towards
[9] Palash Bera, Yves Wautelet, and Geert Poels. 2023. On the Use of ChatGPT to a model-driven approach for multiexperience AI-based user interfaces. Software
Support Agile Software Development. In Short Paper Proceedings of the 2nd Inter- and Systems Modeling 20 (2021), 997–1009.
national Workshop on Agile Methods for Information Systems Engineering (Agil-ISE) [35] Md Saidur Rahaman, MM Tahmid Ahsan, Nishath Anjum, Harold Jan R Terano,
co-located with the 35th International Conference on Advanced Information Systems and Md Mizanur Rahman. 2023. From ChatGPT-3 to GPT-4: a significant advance-
Engineering (CAiSE), Vol. 3414. CEUR-WS.org, 1–9. ment in ai-driven NLP tools. Journal of Engineering and Emerging Technologies 2,
[10] Jacob T. Biehl, Mary Czerwinski, Greg Smith, and George G. Robertson. 2007. 1 (2023), 1–11.
FASTDash: A Visual Dashboard for Fostering Awareness in Software Teams. In [36] Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background,
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems applications, key challenges, bias, ethics, limitations and future scope. Internet of
(CHI). ACM, 1313–1322. Things and Cyber-Physical Systems (2023).
[11] Hee-Seung Henry Bom. 2023. Exploring the Opportunities and Challenges of [37] Daniel Russo. 2023. Navigating the complexity of generative ai adoption in
ChatGPT in Academic Writing: a Roundtable Discussion. Nuclear Medicine and software engineering. arXiv:2307.06081 (2023).
Molecular Imaging (2023), 1–3. [38] Mohammad Shehab, Laith Abualigah, Muath Ibrahim Jarrah, Osama Ahmad
[12] CodeLogic. 2023. The Future of Software Development: Maximizing Benefits and Alomari, and Mohammad Sh. Daoud. 2020. Artificial Intelligence in Software
Minimizing Risks of AI-Assisted Coding. https://codelogic.com/blog/ai-coding- Engineering and inverse: Review. International Journal of Computer Integrated
tools/ Accessed: September 12, 2023. Manufacturing 33, 10-11 (2020), 1129–1144.
[13] Grant Cooper. 2023. Examining science education in chatgpt: An exploratory [39] Nigar M Shafiq Surameery and Mohammed Y Shakor. 2023. Use chat gpt to solve
study of generative artificial intelligence. Journal of Science Education and Tech- programming bugs. International Journal of Information Technology & Computer
nology 32, 3 (2023), 444–452. Engineering 3, 01 (2023), 17–22.
[14] Fulvio Corno, Luigi De Russis, and Juan Pablo Sáenz. 2019. On the challenges [40] NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad,
novice programmers experience in developing IoT systems: A survey. Journal of Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht,
Systems and Software 157 (2019), 110389. Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi
[15] Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herbsleb. 2012. Social Coding Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoff-
in GitHub: Transparency and Collaboration in an Open Software Repository. In man, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit,
Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov,
(CSCW). ACM, 1277–1286. Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn,
Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk,
Conference’17, July 2017, Washington, DC, USA M. Waseem et al.

and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine [45] Gokul Yenduri, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, Rutvij H
Translation. arXiv:2207.04672 Jhaveri, Weizheng Wang, Athanasios V Vasilakos, Thippa Reddy Gadekallu,
[41] Peeratham Techapalokul and Eli Tilevich. 2017. Understanding recurring quality et al. 2023. Generative Pre-trained Transformer: A Comprehensive Review on
problems and their impact on code sharing in block-based software. In Proceedings Enabling Technologies, Potential Applications, Emerging Challenges, and Future
of the 2017 IEEE Symposium on Visual Languages and Human-Centric Computing Directions. arXiv:2305.10435 (2023).
(VL/HCC). 43–51. [46] Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem.
[42] Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques 2023. Demystifying Practices, Challenges and Expected Features of Using GitHub
Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the Ultimate Programming Copilot. International Journal of Software Engineering and Knowledge Engineering
Assistant–How far is it? arXiv:2304.11938 (2023). (2023).
[43] Muhammad Waseem, Teerath Das, Aakash Ahmad, Mahdi Fehmideh, Peng Liang, [47] Gaoxia Zhu, Xiuyi Fan, Chenyu Hou, Tianlong Zhong, Peter Seow, Annabel Chen
and Tommi Mikkonen. 2023. Dataset for the Paper: Using ChatGPT throughout Shen-Hsing, Preman Rajalingam, Low Kin Yew, and Tan Lay Poh. 2023. Embrace
the Software Development Life Cycle by Novice Developers. Opportunities and Face Challenges: Using ChatGPT in Undergraduate Students’
[44] Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. Collaborative Interdisciplinary Learning. arXiv:2305.18616 (2023).
2023. Chatgpt prompt patterns for improving code quality, refactoring, require- [48] Mingyu Zong and Bhaskar Krishnamachari. 2022. A survey on GPT-3.
ments elicitation, and software design. arXiv:2303.07839 (2023). arXiv:2212.00857 (2022).