Capture the Flag with ChatGPT: Security Testing with AI ChatBots

Ellis Casey and David Chamberlain

Airbus, Newport, UK
[email protected]
[email protected]
Abstract: Penetration testing, commonly referred to as pen testing, is a process of assessing the security of a computer
system or network by simulating an attack from an external or internal threat actor. One type of pen testing exercise that
has become popular among cybersecurity enthusiasts is called Capture the Flag (CTF). This involves solving a series of
challenges that simulate real-world hacking scenarios, with the goal of capturing a flag that represents a piece of sensitive
information. Recently, there has been a growing interest in the use of natural language processing (NLP) and machine
learning (ML) technologies for penetration testing and CTF exercises. One such technology that has received significant
attention is ChatGPT, a large language model (LLM) trained by OpenAI based on the GPT-3.5 architecture. The use of ChatGPT
in CTFs has several potential benefits for participants and organisers, including more dynamic and realistic scenarios and
enhanced learning experiences, and enhance the effectiveness and realism of CTFs.. Future research can explore more
sophisticated models and evaluate the effectiveness of ChatGPT in improving the performance of participants in CTFs.
Keywords: ChatGPT, Chatbot, CTF, Penetration testing, Hacking

1. Introduction
In recent years, cyber-attacks are becoming more frequent and sophisticated. To keep pace with these evolving
threats, it is important to continually improve and innovate the tools and techniques used in cybersecurity. One
promising area of innovation is the use of artificial intelligence (AI) in cybersecurity, particularly in the area of
penetration testing. In this essay, the use of ChatGPT, a large language model trained by OpenAI, will be explored
in conducting penetration testing on Capture the Flag (CTF) exercises.
CTF is a popular cybersecurity competition that simulates real-world scenarios for participants to test their skills
in offensive and defensive security. CTFs involve a variety of challenges, ranging from web application exploits
to network and system vulnerabilities. One of the main objectives of CTFs is to provide a safe and controlled
environment for participants to learn about cybersecurity concepts and techniques, and to practice penetration
testing skills.
Penetration testing and CTF exercises are two critical areas of cybersecurity that require constant innovation to
stay ahead of evolving threats. One such innovation that has recently garnered attention in the field is the use
of natural language processing (NLP) and machine learning (ML) technologies to enhance the effectiveness of
these exercises. Specifically, ChatGPT, a large language model trained by OpenAI, has shown promise in
automating the process of generating attack vectors and testing them against target systems in CTFs. This essay
will also explore how to leverage ChatGPT when conducting CTF exercises, including technical requirements and
integration functions. Examples will be discussed on how ChatGPT and other chatbots can be leveraged in CTFs.
ChatGPT can be used to automate the process of generating attack vectors and testing them against a target
system or network. By leveraging its ability to understand and generate natural language, ChatGPT can craft
attack scenarios that are more sophisticated and realistic than traditional approaches. This makes it a powerful
tool for conducting CTF exercises, as it can generate challenges that are tailored to the specific strengths and
weaknesses of the participants.
Using ChatGPT for CTF exercises has the potential to significantly enhance the realism and effectiveness of these
exercises. In addition to generating challenges, ChatGPT can also act as a virtual adversary, providing hints and
feedback to participants based on their progress. This can help to create a more engaging and dynamic CTF
experience, while also providing valuable feedback on participants' strengths and weaknesses, helping them
overcome obstacles and learn new skills.
However, the use of ChatGPT in pen testing and CTF exercises also raises important ethical and legal questions.
The technology can potentially be used to automate and scale attacks against real-world targets, which could
have serious consequences for individuals and organisations. As such, it is important to consider the ethical
implications of using ChatGPT for these purposes and to ensure that its use is in line with established ethical
guidelines and legal frameworks.

43
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

2. Leveraging ChatGPT in CTFs

ChatGPT and other chatbots can be leveraged in CTFs to enhance the realism and effectiveness of the exercises.
For example, ChatGPT can be used to create dynamic challenges that respond to participants' actions and
progress. This can help to create a more engaging and realistic CTF experience, while also providing valuable
feedback on participants' strengths and weaknesses, providing hints and suggestions for participants who are
struggling with a challenge, helping them overcome obstacles and learn new skills.
In addition, ChatGPT can be used to simulate the behaviour of real-world threat actors by generating attack
scenarios that mimic their tactics, techniques, and procedures (TTPs). This can help participants to develop a
better understanding of how threat actors operate and how to defend against them.
Further ways to leverage ChatGPT in CTFs is as a tool to teach participants about cybersecurity concepts and
best practices. For example, ChatGPT can be used to generate interactive tutorials on topics such as password
cracking, buffer overflows, and SQL injection. Providing participants with hands-on experience, they can develop
a deeper understanding of cybersecurity concepts and how to apply them in practice.
The use of ChatGPT in CTFs has several potential benefits for participants and organisers. One of the main
advantages is that ChatGPT can provide more dynamic and realistic scenarios that simulate real-world threats.
ChatGPT can generate responses based on the participants' solutions and adapt to their level of knowledge and
experience, making the challenges more personalised and engaging.

3. Research Plan
Within this research, each step has been manually entered to ensure validity of the results in order to
demonstrate ChatGPT’s practical usage in security testing within cybersecurity, as well as measuring its success
rate in completing various tasks from user prompts.
What is being studied
• Using ChatGPT for assistance in CTF challenges from Hack the Box (HTB), which is a gamified
cybersecurity learning platform
• Assessing its collected training set of cybersecurity concepts, tools and techniques
• Evaluating its ability to process context of the challenges and provide accurate solutions or hints
where possible
• Identifying limitations of ChatGPT’s capabilities, such as potential for misuse, and areas where human
intervention is still necessary
• Is a level of fundamental knowledge of cybersecurity required, in order to interrogate and elicit an
appropriate response from ChatGPT
Potential output of this research
• Gain an understanding of ChatGPT’s capabilities and limitations in the context of cybersecurity, and
how it could be integrated into security testing workflows
• Establishing a performance benchmark in ChatGPT’s current state
• Identifying areas of improvement for the application
• Gain a better understanding of how crafting the commands used will determine the validity or
usefulness of the output.
Considerations
• ChatGPT struggles with identifying and assisting with newer CVEs and exploits
• When queried about a newer CVE, it will pretend to know what it is and ‘hallucinate’ incorrect
information about its usage. These hallucinations refer to the chatbot confidently generating
seemingly realistic responses that do not coincide with real-world scientific knowledge. Since ChatGPT
is an example of an LLM (Large Language Model), it has a tendency to falsify information and present
it as factual. When a user sends a prompt to ChatGPT, it gets broken down into small parts called
tokens, which could be as short as a single character or word. It then processes these tokens, using
patterns it learned during its training to predict what word (token) will come next in the sequence.
Each token iteration is assigned a probability and the token with the highest probability is chosen to
continue the text generation. This process is repeated until the full response has been generated.

44
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
Ellis Casey and David Chamberlain

• This makes the response seem human-like, in the sense that it doesn’t like to say “I don’t know” when
it doesn’t know
• Recent update - Instead of substituting the CVE the user provides by adding it to the end of a CVE
database URL and hallucinating information, it now seems to interpret the date of the CVE given and
tell the user it only has information up to September 2021. This change was observed between
approximately the 9th February 2023 update to the 23rd March 2023 update.
• It states the last known CVE it knows is Log4Shell
• This makes it easier for security testers to know what information is accurate, however adding a
confidence scale will ultimately be the best solution for ensuring accuracy, along with providing a
source(s) for the output it provides

4. Selected HTB Machines

Machines were chosen with generally high user ratings in order to try and give ChatGPT what is perceived as a
greater chance of success. It was also decided to exclusively use Linux machines for consistency and used a mix
of easy and medium challenges involving a variety of vulnerabilities, in order to vary the potential difficulty for
the chat bot. The overall thoughts on ChatGPT’s usefulness and limitations in certain areas of each machine have
been recorded. ‘Boot-to-root machines’ were selected in order to simulate a real life scenario, as opposed to
short challenges. Boot-to-root challenges involve going from booting the machine to escalating to ‘root’ or
‘admin’ level privileges on a vulnerable system. It’s worth noting that during the testing phase, most of the
machines chosen were active, meaning that there were no online write-ups for these challenges at the time.
The following section contains the description and general steps taken during each challenge by HTB.
MetaTwo
Easy — Linux
MetaTwo starts with a basic WordPress (WP) blog using the BookingPress plugin to manage booking events.
There is an unauthenticated SQL injection vulnerability within that plugin, which is used to access the WP admin
panel as an account that can manage media uploads. There is an exploit for an XML external entity (XXE) injection
to read files from the host, WP configuration, and access FTP server credentials. The FTP server contains a script
that sends emails, and credentials can be used to get a shell on the host. There is a tool on the system called
Passpie that stores the root password. The PGP key protecting the password can be cracked, then brute force
the output. This presents a shell as the root user.
Precious
Easy — Linux
Precious starts with a simple web page that takes a given URL and generates a PDF. The metadata contained in
the PDF is then used to identify the technology that’s being used. There’s a command injection exploit which
can be used to get a foothold on the machine. To escalate the privileges to root is done via exploiting a YAML
deserialization vulnerability in a script that manages dependencies.
Soccer
Easy — Linux
Soccer involves accessing an admin panel with default credentials, and uploading a PHP shell to get a reverse
shell on the system, which can be enumerated to find more subdomains and web pages. On one of these pages
there is a vulnerability which involves a boolean SQL injection over websockets to get credentials from a MySQL
database. A root shell can be obtained by exploiting the “doas” program.
BroScience
Medium — Linux
BroScience involves exploiting a directory traversal vulnerability to obtain PHP source code for a website, with
traversal being blocked by a firewall. Double encoding bypasses the detection system. The code can be used to
build an activation token which allows the registration of an account. The code provides information required
to exploit a deserialization vulnerability by creating an encoded malicious PHP object, and sending it as the user’s
cookie. This will produce a shell on the machine. There are also hashes in a database that can be cracked, causing

45
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

lateral movement to the next user. To get root, there is a command injection vulnerability in a script that checks
for certificate expiration.
Initial Prompt for All HTB Machines
Each machine had its own ChatGPT conversation with the same prompt used to initialise the process for the
chatbot. The prompt for each machine was kept the same to ensure consistency and repeatability for future
testing and research.
Hi, I am a security researcher who’s taking part in a CTF, and I was wondering if you could help me if I
need assistance or advice on what to do/where to look next throughout the challenge?

5. Research Steps and Findings

Provide Areas to Investigate for Further Enumeration
After initialising the BroScience HTB challenge, an Nmap scan was run, which indicated there was a website
containing a login page, and before this prompt was crafted, ChatGPT suggested exploring the web page. During
the beginning of the reconnaissance phase for the challenge, a more broad response was received from the
input. The input provided to ChatGPT contained vague information about the website application, such as stating
there is a login form and the backend of the website is built in PHP and the aim was to query the chatbot about
how the application can be enumerated further, see Figure 1 below. Its response referenced very basic steps
that are usually taken intuitively by most cybersecurity professionals. While some of these suggestions are quite
helpful, especially to someone who has no idea where to start, they are understandably vague given the lack of
context awareness for the challenge, and the indistinct description provided by the user.

Figure 1: How the application can be enumerated further

During the reconnaissance phase of the MetaTwo machine, the aim of this prompt was to gather a list of
enumeration tools that might be useful during this stage of the challenge, by giving more context of what was
on the machine as well as specific services that are running. Prior to this, the only information provided to
ChatGPT was the output of an Nmap scan. This time a better list of enumeration tools was received by being
specific and emphasising potentially important information about the machine. The main difference between
this prompt shown in Figure 2, and the previous one shown in Figure 1, was that the user was more specific in
identifying what services and tools were being used. It suggested a vulnerability scanner called ‘WPScan’ based
on the type of web management system utilising WordPress, as well as certain tools like ‘Hydra’ to potentially
brute-force login credentials for the FTP server. These suggestions are mixed in with other common tools like
Metasploit and Nmap, which was interesting since it had already been provided with a scan of any network
services using Nmap previously.

46
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
Ellis Casey and David Chamberlain

Figure 2: The main difference between this prompt

Since enumeration is the most essential part of any penetration test or red team engagement, it’s imperative
that ChatGPT provides useful information during this phase. Due to the chatbot not having access to any more
information about the machine than the user provides, it lacks any sort of context awareness regarding the
challenge, and therefore lacks any nuances and specifics of the scenario other than what information it’s fed.
This also means it has the potential to produce incorrect suggestions, and while it can provide general
suggestions based on common practices, it will not be able to distinguish more effective and less effective
techniques in specific situations.
In its current state, it’s not advisable to heavily rely on ChatGPT to suggest areas for enumeration due to its lack
of context awareness and deep technical knowledge within specific domains of cybersecurity. Its behaviour and
output are shaped by its training set. It can, however, definitely prove useful for someone who is starting out in
cybersecurity or stuck during a penetration test.
Suggesting Attack Vector and Bypass Detection System
ChatGPT provided a seemingly correct exploit for this stage of the BroScience machine, and gave an example to
demonstrate along with a high level explanation of how the exploit works, and how to know if the target machine
is vulnerable to this type of attack. However, it did take a bit of back and forth with the chatbot to get to this
point, having previously gone down rabbit holes that led to dead-ends by suggesting that a login form was
vulnerable to SQL injection or brute forcing credentials. It also previously suggested using several web
vulnerability scanners such as Nikto and Dirbuster, which did find some hidden directories, and led to discovering
the missing path parameter.
ChatGPT provided some good suggestions for this vulnerability, as it noted there was a detection system in place,
and double URL encoding worked in this particular case. However, it did not give sufficient detail on how to
encode the payload or give any more specific examples, only a potential direction to explore.
ChatGPT performed moderately well overall in suggesting attack vectors. However, suggesting this particular
exploit was after a lot of back and forth with the chatbot, to get to the point where sufficient information was
provided to ChatGPT during the enumeration phase of the challenge. Once it was confirmed the attack vector
works, the chatbot was able to provide suggestions for bypassing the detection system. Demonstrating that
proving enough information during enumeration is paramount for ChatGPT to suggest accurate vulnerabilities,
areas for exploitation, and reduces the potential for incorrect suggestions.
Analysing and Deobfuscating Code, and Decoding Base64
With some PHP files retrieved throughout the BroScience machine by injecting the website with an encoded
local file inclusion (LFI) path, ChatGPT was particularly useful at analysing the code and describing how it works.
After giving the contents of one of the PHP files, ChatGPT responds with a description of what the code does
and a high level overview of how the script works. ChatGPT shows promising utility for explaining complex code,
by explaining its logic and what each function does.
One area where ChatGPT proves incredibly useful is code de-obfuscation. In this case, PowerShell de-
obfuscation. Of course, there are some online tools out there that can do this already, but given the extra
explanation ChatGPT provides, it is the preferred tool on the market.

47
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

For the MetaTwo machine, ChatGPT had previously suggested I use WPScan to enumerate the website since it
was using WordPress, and it provided suggestions based on the output of this scan. It was discovered that an
old, vulnerable version of the BookingPress plugin was being used. After asking ChatGPT if there are any known
exploits for that version, it states that it cannot provide any known exploits as it goes against OpenAI’s usage
policy, which will be discussed further later in this paper. After finding a working exploit online, it allowed the
retrieval of the /etc/passwd directory which was Base64 encoded. After asking ChatGPT to decode this, it misses
a crucial part of the CTF in this output, as there is no ‘jnelson’ user displayed in ChatGPT’s result. Decoding it via
the command-line to compare results shows that there are some differences in the way both tools decode it.
After a short back-and-forth with ChatGPT, it states there is no user ‘jnelson’ in the encoded text it was provided
shown in Figure 12, and was then asked to decode it again. After decoding it again, the chatbot outputs the
expected user account with a minor difference - denoting the full name of the user as ‘John Nelson’, which is
not present in the output of the command-line Base64 decoding. To confirm this was a hallucination, it was told
there was another missing user and asked it to decode it again. This time, ChatGPT added ‘jsmith’ as a new user,
which confirms that ChatGPT might not be reliable for decoding various encoding formats, which can be seen in
Figure 13.
Something interesting to note is that ChatGPT redacts the password for a database, as well as removing the
database charset rules and collate type by filtering out potentially sensitive information. It also misses out critical
FTP information, such as the username and password. The password in particular is crucial to the next stages of
the MetaTwo CTF challenge. Ultimately it is useful to decode Base64 quickly, although it’s best not to rely on it
getting the syntax 100% accurate, and would instead prefer to use CyberChef or stick to decoding it via the
terminal. When asking the chatbot to display the password, it refuses for ethical concerns, no matter how the
prompt was phrased.
In regards to code interpretation, ChatGPT does very well. However, it has a tendency to pretend it knows what
it’s saying and can hallucinate information. During the decoding process, it seems to filter and redact certain
information it perceives as sensitive or exploitable. Bypassing the filter to reveal such sensitive information will
be covered later in the paper.
Interpreting Scan Results
After providing the output of the WPScan for the MetaTwo machine to ChatGPT, it noticed that an outdated
version of WordPress was being used, along with an outdated theme version in Figure 3. It also identified two
user accounts “admin” and “manager” and suggested the idea of brute-forcing these credentials. On top of
picking out key information from the scan, the chatbot suggested some steps to take based on the information
it found. The most useful suggestion it gave in this case was to research vulnerabilities for this outdated software.
This was conducted after ChatGPT was provided an Nmap output and told about various services running on the
machine, such as FTP on a ProFTP server. The chatbot had also provided the exact command for WPScan to
enumerate users, plugins, themes, and so on.

Figure 3: An outdated theme version

ChatGPT was also queried about Nikto scan results in Figure 4 for MetaTwo to see if it would offer up any more
suggestions on top of the previous ones. It provided a lot of potential areas to explore based on the output of

48
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
Ellis Casey and David Chamberlain

the scan, which produced more tailored guidance on what to look into next. This step was conducted
immediately after analysing the WPScan output.

Figure 4: Nikto scan results

After providing ChatGPT with MetaTwo’s Nmap scan results in Figure 5, it gives an overview of results, noting
any important information that might be required to progress in the CTF, such as open ports, running services
and domains. Interestingly it then suggested that the next step should be to run a network scanner like Nmap
to identify what services are running on the network, which seemed redundant.

Figure 5: MetaTwo’s Nmap scan results

ChatGPT is particularly useful for attaining a high level overview of scan results, and is especially useful if the
output of the scan is very long and tedious to go through line by line. By providing ChatGPT with scan results, it
can help understand and pick out what parts are worth investigating further which can make the penetration
test process much more efficient, particularly for those who are inexperienced and might not know what to look
out for. These scan results can lead to the chatbot assisting in identifying attack vectors by analysing potential
vulnerabilities in scan results.
While ChatGPT has general knowledge about security standard security tools mentioned in its training set, it
doesn’t necessarily possess deep expertise in specific features or intricacies of every tool. Therefore its guidance
on advanced usage in certain tools may be limited. The analysis will be based solely on the information provided
in the scan results, as well as any other information the user has shared with the chatbot. This means that it may
not have an understanding of the wider context of the target system, and therefore it’s crucial to consider

49
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

utilising human expertise in looking over the output of the scan, in case there is anything that ChatGPT could’ve
missed.
Hallucinations
Some time after the 9th February 2023 update, in Figure 6 ChatGPT was prompted to provide information about
a CVE used in HTB’s MetaTwo machine which was reported in 2022, to see how it would respond. It gave
completely incorrect information about the CVE, stating it’s a remote code execution (RCE) vulnerability for
certain versions of Microsoft Office, SharePoint and Visual Studio. Interestingly, it provided the supposed source
for the incorrect information by substituting the CVE name into the Mitre URL in Figure 7. When navigating to
the URL, it presents the correct page for the exploit, stating it’s vulnerable version of a certain WordPress plugin
which leads to unauthenticated SQL injection, which is incredibly different from the information provided by
ChatGPT.

Figure 6: information about a CVE

Figure 7: Mitre URL

After recently testing this again (March 23rd 2023 update), it seems that OpenAI have fixed this particular
hallucination issue with new CVEs, stating that it doesn’t know about the CVE, without providing false
information and sources. The updated response is shown in Figure 8.

50
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
Ellis Casey and David Chamberlain

Figure 8: updated response

Since the previous exploit used wasn’t working, ChatGPT was prompted to provide an article that can walk
through how to use the exploit, as it would be interesting to see how it would provide an article for a CVE it
doesn’t know about. As expected, navigating to the URL in Figure 9 yields no article but a “404 page not found”.
A brief investigation into whether the article once existed was conducted, with the Internet Archive and
Wayback Machine unable to find any results, despite finding other articles made in the same time period, so it’s
theorised it was hallucinated information.

Figure 9: navigating to the URL

Interestingly, this is completely hallucinated information, which makes sense since the CVE for this challenge is
from 2021 (CVE-2021-40964), and after ChatGPT’s knowledge cut-off date. The CVE ChatGPT referenced in
Figure 10 is for a file sharing tool called Pydio which allows for an attack via PHP object injection. The CVS score
for this CVE is also incorrectly labelled as 8.8 instead of 9.8. The vulnerable version of this file manager is for the
Soccer HTB machine and was found after ChatGPT suggested logging in using default admin credentials that
could be found online.

Figure 10: CVE ChatGPT

51
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

Overall, ChatGPT comes with its fair share of limitations, especially in regards to real-time vulnerabilities. As
mentioned earlier, ChatGPT’s training set cut-off date is September 2021. When queried about a newer CVE, it
will pretend to know what it is and hallucinate incorrect information about its usage. This makes it appear very
human, in the sense that it doesn’t say “I don’t know” when it doesn’t know the answer.
These hallucinations pertain to the chatbot confidently producing responses that appear remarkably realistic,
despite contradicting established scientific knowledge. Being an instance of an LLM, ChatGPT tends to fabricate
information and present it as factual. When a user inputs a prompt to ChatGPT, it breaks down the input into
smaller units called tokens, which can be as short as a single character or word. It processes these tokens,
utilising the patterns acquired during training to predict the subsequent word (token) in the sequence. Each
token iteration is assigned a probability, and the token with the highest probability is selected to continue
generating the text. This iterative process persists until the complete response is generated. It would be useful
for OpenAI to develop and implement a confidence scale for ensuring accuracy for cybersecurity professionals,
along with providing a source(s) for the output provided.
Jailbreaking ChatGPT
Throughout the research period, it was only possible to get ChatGPT to reveal the database password and
various other sensitive information through the use of the DAN prompt. (0xk1h0, n.d.) DAN (do anything now)
is a prompt used to jailbreak ChatGPT into leaking restricted data, and can elicit responses to questions the AI
would usually refuse to answer. These prompts are generated by users of ChatGPT, and are generally disallowed
by OpenAI’s usage policies, since the usage of DAN could enable users to generate malicious code and develop
realistic phishing emails that are designed to damage or gain unauthorised access to a computer system, as well
as breaking any other laws. These prompts get updated quite often as they frequently get patched. The prompt
can be used to bypass OpenAI guardrails and moderation, and breaks the filter that avoids producing content
deemed unsafe as per their usage policies.
Getting ChatGPT to reveal the sensitive information of the database was very straight-forward and successful.
Simply entering the prompt, then providing the encoded data resulted in both the GPT and DAN response
displaying the database password and other sensitive information in Figures 11 and 12. This was tested using
the latest current version of DAN (11.0).

Figure 11: GPT Figure 12: DAN

52
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
Ellis Casey and David Chamberlain

6. Future Work
There are several areas for future research and development related to the use of ChatGPT in CTFs. One area is
the development of more sophisticated models that can generate more complex and diverse responses, and
realistic attack scenarios. Another area is the evaluation of the effectiveness of ChatGPT in improving the
learning experience and performance of participants in CTFs. Additionally, there is potential for the integration
of ChatGPT with other cybersecurity tools and technologies to create more comprehensive and effective security
solutions.
The integration of ChatGPT in CTFs can be achieved through web-based interfaces or integration with existing
CTF platforms. One example of a web-based interface is the CTFd platform, which allows for the creation and
management of CTF competitions. CTFd can be extended to include ChatGPT functionality by creating a plugin
that integrates the API created in the previous step.
Another example of integration with existing CTF platforms is the integration of ChatGPT with a CyberRange,
which is a platform that enables the creation and management of virtual environments for conducting
cybersecurity exercises. CyberRange can be extended to include ChatGPT functionality by creating a module that
integrates the API with the platform's existing functionality.
An automated implementation of ChatGPT in CTFs requires technical requirements such as collecting a dataset
of real-world hacking scenarios, fine-tuning a pre-trained ChatGPT model on the collected dataset, and creating
an API for participants to interact with the model. The integration of ChatGPT in CTFs can be achieved through
web-based interfaces or integration with existing CTF platforms.

7. Conclusion
ChatGPT has the potential to revolutionise the way in which CTF exercises are conducted, in addition to security
and penetration testing. By leveraging its NLP and ML capabilities, automation of the process of generating
attack scenarios and provide personalised feedback to participants can be accomplished.
The use of ChatGPT in CTFs has several benefits, such as enhancing the realism and effectiveness of the exercises,
providing personalised feedback to participants, simulating real-world threat actors and providing interactive
tutorials on cybersecurity concepts. ChatGPT can also simulate real-world threat actors by generating attack
scenarios mimicking tactics procedures, and provides interactive cybersecurity tutorials.
While the use of ChatGPT in CTFs is promising, it is important to consider the ethical and legal implications of its
use and ensure that it aligns with established ethical guidelines and legal frameworks. Overall, ChatGPT has the
potential to revolutionise the conduct of CTF exercises and penetration testing, and its implementation can help
to stay ahead of evolving threats in the cybersecurity field. However, as with any new technology, it is important
to consider the ethical and legal implications of using ChatGPT in these contexts and ensure that its use is in line
with established ethical guidelines and legal frameworks.
User assurance of ChatGPT’s output is required, as any understanding of cybersecurity is by proxy, since the
sequences LLMs are training are created by people who do understand cybersecurity concepts. LLMs are shaped
by this, and do not understand the tokens it uses.
This research paper has provided valuable insights into the capabilities and limitations of ChatGPT in the context
of cybersecurity by evaluating its performance across a range of Hack The Box challenges. Similar limitations
were noted when using ChatGPT for solving programming problems (Surameery, Shakor, 2023). The study
demonstrates that ChatGPT can be a useful tool for certain cybersecurity tasks, as it can offer guidance and
assistance in areas such as programming and web exploitation. However, it also highlights some limitations,
including the need for human expertise in specific domains and potential ethical considerations related to AI
adoption in cybersecurity.
This research suggests that using standard cybersecurity tools with ChatGPT’s assistance has the potential to
integrate with existing security testing workflows, enhancing the problem solving capabilities of cybersecurity
teams. Organisations considering AI adoption for cybersecurity purposes can benefit from the findings of this
study, using them as a foundation for informed decision making and responsible AI usage, since integrating a
tool like this can pose another layer of potential security risks for a business.
Finally, this study identifies areas for future research and development, including improving ChatGPT's training
set, context awareness and adaptability to address the constantly evolving landscape of cybersecurity threats.

53
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024
 Ellis Casey and David Chamberlain

Implementing features such as a confidence scale would also enable cybersecurity professionals to make more
informed decisions based on their knowledge and experience with the assistance ChatGPT provides, as well as
providing sources for the information the chatbot references. Further research on the ethical implications of AI
chatbots in cybersecurity, and the development of best practices for their use will also be paramount to ensure
the responsible integration of these technologies into the cybersecurity domain.

References
0xk1h0. (n.d.). 0xk1h0/ChatGPT_DAN: ChatGPT DAN, Jailbreaks prompt. GitHub. Retrieved June 12, 2023, from
https://github.com/0xk1h0/ChatGPT_DAN
Nigar M. Shafiq Surameery, & Mohammed Y. Shakor. (2023). Use Chat GPT to Solve Programming Bugs. International
Journal of Information Technology & Computer Engineering (IJITC) ISSN : 2455-5290, 3(01), 17–22.
https://doi.org/10.55529/ijitc.31.17.2

54
Proceedings of the 19th International Conference on Cyber Warfare and Security, ICCWS 2024