2024 IEEE/IFIP Network Operations and Management Symposium (NOMS 2024) - Poster

ChatGPT-enabled Network Automation using API-

based Prompts
Olasupo Okunaiya Ron Austin Shao Ying Zhu
College of Computing College of Computing College of Computing
Birmingham City University Birmingham City University Birmingham City University
Birmingham, United Kingdom Birmingham, United Kingdom Birmingham, United Kingdom
[email protected] [email protected] [email protected]
NOMS 2024-2024 IEEE Network Operations and Management Symposium | 979-8-3503-2793-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/NOMS59830.2024.10574943

Abstract—In the rapidly evolving world of network automation scripts to meet network requirements, leading to
provisioning and management, the traditional approach to potential for syntax errors [3].
device configuration has proven to be error-prone and time-
consuming, especially in large-scale deployments. The TABLE I. RELATED WORKS ON NETWORK AUTOMATION
emergence of network automation has sought to address these
challenges by leveraging software to perform tasks which were Reference Python Ansible ChatGPT
traditionally executed manually by network engineers. [1]  
However, utilizing these tools requires technical expertise. This [2]  
study explores the potential of using Large Language Models [3]  
[4]  
(LLMs), specifically ChatGPT, to generate Ansible Playbooks
[5]  
for automating network deployment tasks. The goal is to  
[6]
eliminate the need for network engineers to prepare automation [7]  
scripts. An application named "Bubbln" was developed to [8]  
prompt ChatGPT through APIs and execute the ChatGPT- [9]  
generated playbooks on a network of four Cisco routers. A [10]  
systematic evaluation of this approach reveals impressive results [11]  
including 99% accuracy, 100% reliability, a 62% improvement [12]  
in efficiency compared to traditional approach and a 13.52% [13]  
efficiency gain over manually prepared playbook approach. [14]  
These results highlight ChatGPT as a promising tool to enhance [15]  
existing network automation approaches. [16]  
[17]  
Keywords—ChatGPT, chatbot, network automation, [18]  
provisioning, configuration. [19]  
[20]  
I. INTRODUCTION
The evolution of network provisioning processes has More recently, significant advancements have emerged
transitioned from a traditional, error-prone approach to the in the field of Artificial Intelligence, with a particular focus
use of automation tools. Initially, engineers configured each on Large Language Models (LLMs) such as ChatGPT and
networking device manually through Command Line Bard. These models exhibit the remarkable ability to process
Interfaces, which became impractical for large-scale instructions expressed in natural language to generate desired
networks. The growing complexity of networks necessitated textual outputs [21]. The release of ChatGPT has ushered in
faster setup procedures, leading to the concept of network a new era of LLM applications across various domains,
automation. Network automation involves the utilization of including software coding [22], debugging [23], vulnerability
software tools such as Ansible, Chef, Puppet, and others to detection [24] and more. Considering these advancements,
streamline tasks that were traditionally performed manually research on the use of LLMs to automate the scripting process
by network engineers [1]. did not return any result. This prompted our research to
explore the potential of leveraging ChatGPT to generate
Recent research on network automation approaches has Ansible playbooks for network deployment, eliminating the
primarily focused on the preparation of Ansible [1]-[7] and necessity for manual preparation of automation scripts. This
Python scripts [8]-[20] by the researchers as summarized in approach is expected to improve network automation
Table I, to execute various network tasks. For instance, [2] efficiency and ultimately increase the productivity of
compared the performance of Ansible, Salt and Puppet, by engineers.
preparing scripts for different tools, [3] utilized Ansible
scripts to configure both the interfaces and routing protocols II. METHODOLOGY
on a network of five Cisco and five MikroTik routers, while To assess the potential of ChatGPT to generate Ansible
[4] utilized Ansible scripts to provision and verify routing playbooks, a systematic methodology was devised. The key
protocols on a network of three routers. In a similar manner, steps of the methodology included the following:
Python scripts were prepared by [8] to prove its ability to
verify network states automatically and [9] utilized python in A. Obtain API Keys
a proposed architecture to automate VPN configurations. The first step was to register on the OpenAI portal to
Although this approach of manually preparing automation obtain an API key, which was then utilized in the application
scripts offer numerous advantages over traditional methods, developed for prompting ChatGPT.
including increased efficiency and a reduction in errors, there
is a requirement for technical expertise, for effective use of
the automation tools [2]. Engineers need to prepare or tailor

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO CEARA. Downloaded on January 28,2025 at 20:52:38 UTC from IEEE Xplore. Restrictions apply.
 2024 IEEE/IFIP Network Operations and Management Symposium (NOMS 2024) - Poster

B. Application Development
An application named Bubbln, was developed using
Python programming language to get network parameters
from users, compose prompts, prompt ChatGPT, process the
response from ChatGPT, and finally execute the ChatGPT-
generated playbooks on a network of four Cisco routers.
Bubbln was developed with three main components as
follows:
• Logic and Control – This component manages taking
and validating router parameters from users. It then
combines these inputs with a pre-defined set of
instructions written in natural language to form the
prompts illustrated in Fig. 2, which were then sent to
the OpenAPI Gateway. It also receives ChatGPT’s
raw response from the OpenAPI gateway component
and extracts the playbooks, which are subsequently
sent along with the host file to the Playbook Executor
component for onward execution on the network. This
entire process is illustrated in Bubbln’s architecture in
Fig. 1.
• OpenAPI Gateway – This component is responsible
for prompting and receiving responses from ChatGPT
through APIs. It utilizes GPT-4 model, with a
temperature parameter setting of zero and maximum
token of 1500 and OpenAI’s chatCompletion API for
sending prompts to ChatGPT. The choice of
parameter values was to limit the tendency of
ChatGPT generating unwanted features. It works by
passing prompts received from the Logic and Control
component to ChatGPT and sends the raw response
from ChatGPT, which contains a combination of the
playbooks and explanation notes, back to the Logic
and Control component for processing as illustrated in
Fig. 1.
• Playbook Executor – This component handles the Fig. 2. Prompt for generation of playbook for router four.
execution of the ChatGPT-generated playbooks on the
C. Network Design
network. It makes use of the Python library called
Ansible-runner which takes a hosts file and the A network of four Cisco routers with parameter details
ChatGPT-generated playbooks as input from the displayed in Fig. 3, was set up to test the ChatGPT-generated
Logic and Control component. playbooks. It was also used to conduct traditional and user-
prepared playbook experiments. The physical interfaces,
shown in the topology, static routes, SSH2.0 and Ansible host
file, were manually configured for the ChatGPT-generated
and user-prepared playbook experiments to enable
reachability from the controller machine, which hosts Bubbln
and Ansible installations.
D. Conduct of the Experiment
Three categories of experiments were conducted. The
main experiment, which is ChatGPT-enabled, and two other
benchmark experiments as follows:
• ChatGPT-generated Playbooks: Twenty-five
experiments were conducted to test the defined
performance metrics. It involved taking parameter
input from users, prompting ChatGPT for playbook
generation, automatically executing the generated
playbooks on the network and keeping a record of the
deployment timings. Inputting of parameters was
done only in the first experiment and the loaded
parameters were re-used for the remaining twenty-
four experiments.
Fig. 1. Bubbln's architecture and process flow.

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO CEARA. Downloaded on January 28,2025 at 20:52:38 UTC from IEEE Xplore. Restrictions apply.
 2024 IEEE/IFIP Network Operations and Management Symposium (NOMS 2024) - Poster

Fig. 3. Network topology and parameters used for the experiments.

• User-prepared Playbooks: A single experiment was

conducted, where Ansible scripts were manually
updated with the required network parameters. A
measurement of the time taken to edit the scripts as
well as execution time were recorded.
• Traditional Manual Configuration: A final experiment
was conducted to manually configure the network.
The deployment duration was also recorded.
E. Defining Performance Metrics
Evaluating the potential of the ChatGPT-enabled
approach relies on performance metrics. In this study, three Fig. 4. ChatGPT-generated playbook for router four.
key indicators were utilized to assess the outcome of the
experiment: TABLE II. DETAILED OUTCOME OF EACH EXPERIMENT

• Accuracy: This measures the correctness of the Parame te r

Playbook
Exe cution Deployment
ChatGPT-generated playbooks and was assessed by a Gene ration
Expe rime nt Input Time Time Time
Time
successful execution of the playbooks without (Se conds)
(Se conds)
(Se conds) (Se conds)
runtime errors. 1 238.34 172.63 61.19 472.16
• Efficiency: This quantifies the average time taken to 2 238.34 171.07 62.78 472.19
deploy the network compared to traditional and user- 3 238.34 161.13 64.44 463.91
prepared playbook approaches. Equation (1) was used 4 238.34 151.27 63.63 453.24
to determine the calculation for efficiency. 5 238.34 227.48 62.11 527.93
6 238.34 215.3 61.8 515.44
• Reliability – This evaluates the fulfilment of the 7 238.34 148.1 63.55 449.99
intended purpose for each playbook. It ensured 8 238.34 175.65 62.85 476.84
parameters and features in the generated playbooks 9 238.34 183.06 62.1 483.5
aligned with those provided in the prompt. 10 238.34 191.62 61.82 491.78
F. Time Tracking 11 238.34 186.62 62.65 487.61
12 238.34 146.15 62.08 446.57
The time taken for inputting configuration parameters,
13 238.34 158.28 63.25 459.87
generating playbooks, and executing playbooks were tracked
14 238.34 166.7 61.11 466.15
in Bubbln using the time python library. These served as
15 238.34 167.13 61.49 466.96
input towards evaluating efficiency of this approach.
16 238.34 161 62.48 461.82
III. THE RESULT 17 238.34 123.67 62.19 424.2
18 238.34 173.24 61.55 473.13
This section offers a detailed account of the outcomes
19 238.34 143.92 61.21 443.47
derived from the experiments conducted as outlined in Table
20 238.34 175.68 61.95 475.97
II.
21 238.34 111.73 61.74 411.81
A. Reliability Assessment 22 238.34 144.39 62.34 445.07
Each ChatGPT-generated playbook with zero runtime 23 238.34 183.6 63.72 485.66
error fulfilled its intended purpose, thereby resulting in a 24 238.34 169.47 62.25 470.06
reliability outcome of 100%. A sample of the generated 25 238.34 177.27 64.78 480.39
playbook can be seen in Fig. 4. AVERAGE
(ChatGPT - 238.34 167.45 62.44 468.23
B. Accuracy Evaluation generated)
User-
During the execution of the twelfth experiment, a runtime
prepared 479.54 - 62 541.54
failure was encountered due to a syntax error in the Approach
T raditional
- - - 1219
Approach

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO CEARA. Downloaded on January 28,2025 at 20:52:38 UTC from IEEE Xplore. Restrictions apply.
 2024 IEEE/IFIP Network Operations and Management Symposium (NOMS 2024) - Poster

generation of router two’s playbook. As a result, the overall

accuracy metric dropped to 99%. Deployment Duration
1400
C. Efficiency Analysis
1200
The ChatGPT-enabled approach took an average time of
1000
468.23 seconds, encompassing parameter input, playbook

Time (Seconds)
generation, and execution across twenty-five experiments. In 800

contrast, the user-prepared playbook experiment required 600

541.54 seconds, involving template editing and playbook 400
execution, while the deployment time for the traditional 200
approach was 1219 seconds. The details can be seen in Fig. 0
5. ChatGPT-generated User-generated Traditional
Approach
Based on the deployment timings, an efficiency
calculation was performed with the formula: Fig. 5. Deployment time for each approach.
Time Taken by Approach A
Efficiency = 1 − * 100 Overall Reliability and Accuracy
Time Taken by Approach B
100.00
99.80
(1) 99.60

Percentage
99.40
The ChatGPT-enabled approach is 61.63% more efficient 99.20
than the traditional approach and 13.52% more efficient than 99.00
user-prepared playbook approach. 98.80
98.60
IV. EVALUATION 98.40
Reliability Accuracy
Traditional network deployment approach is error-prone Performance Metric
and inefficient, prompting the adoption of automation tools,
which also require technical expertise to operate and prone to Fig. 6. Experimental outcome of reliability and accuracy of
syntax errors. This led to our investigation into ChatGPT's ChatGPT-generated playbooks.
potential to generate Ansible playbooks. This section
discusses the discovered pros and cons of using a ChatGPT- • Cost: The utilization of ChatGPT APIs involves
enabled tool for automating network configuration tasks. incurring costs based on usage volume. In essence, a
higher frequency of prompts sent to ChatGPT results
A. Pros of A Chat-GPT Network Automation Tool in proportionally increased costs. This financial aspect
The outcome of this research highlights a ChatGPT- must be carefully factored into the decision to employ
enabled approach as a potential tool for enhancing the this approach.
existing network automation tools for the following reasons:
• Rate Limit: The process of transmitting prompts to
• Simplified Automation Process: The ChatGPT- ChatGPT through APIs is subject to predefined rate
enabled approach seamlessly generated Ansible limits, necessitating thoughtful consideration during
playbooks with 99% accuracy and 100% reliability as the design of an automation tool. This is vital to
seen in Fig. 6. This result positions the approach as prevent the unforeseen interruption of automation
promising for eliminating manual scripting, thereby processes.
reducing time spent on preparing Ansible playbooks.
It addresses the limitations identified by [2] and [3] V. CONCLUSION AND FUTURE WORK
and obviates the need for extensive technical In conclusion, the experiments undertaken to assess the
expertise, eliminating limitations encountered by [2]. potential of employing ChatGPT for generating playbooks
for network deployment tasks have yielded promising and
• Efficiency: The ChatGPT-enabled approach exhibited noteworthy results. ChatGPT has exhibited a commendable
notable efficiency gains. On average, it took 468.23 accuracy rate of 99% and an exceptional reliability rate of
seconds, outperforming the user-prepared playbook 100%. Moreover, these findings indicate an appreciable
approach (541.54 seconds) by 13.52% and the improvement in efficiency, with ChatGPT streamlining
traditional approach (1219 seconds) by 61.63%. It is processes by approximately 62% when compared to
noteworthy that enhancing the efficiency of a traditional methods and 14% when compared to user-
ChatGPT-enabled approach could be achieved prepared playbook approaches. The combination of these
through the adoption of a less time-intensive results demonstrates ChatGPT as a potential tool to enhance
parameter input method, such as the direct loading of existing network automation approaches.
parameters from design documents.
In future research, we aim to investigate ChatGPT’s
B. Cons of A Chat-GPT Driven Network Automation Tool potential for network fault monitoring and auto-recovery to
During this experiment, certain issues were encountered simplify network management, enhance resilience, and
which need to be considered before adopting this approach to minimize downtime. This marks a significant advancement
automate a network. These considerations include the in applying AI and natural language processing to network
following: automation, opening avenues for further exploration of
ChatGPT's potential in network management.

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO CEARA. Downloaded on January 28,2025 at 20:52:38 UTC from IEEE Xplore. Restrictions apply.
 2024 IEEE/IFIP Network Operations and Management Symposium (NOMS 2024) - Poster

REFERENCES [20] A. F. Rochim, A. Rafi, A. Fauzi, and K. T. Martono, "As-RaD System

as a Design Model of the Network Automation Configuration System
[1] J. Wijaya, "Network Automation Using Ansible for Cisco Router Basic Based on the REST-API and Django Framework," Kinetik: Game
Configuration," Retrieved from osf. io/u8cdm, 2018. Technology, Information System, Computer Network, Computing,
[2] S. Wågbrant and V. Dahlén Radic, "Automated Network Electronics, and Control, pp. 291-298, 2020.
Configuration: A Comparison Between Ansible, Puppet, and SaltStack [21] O. Suissa, A. Elmalech, and M. Zhitomirsky-Geffet, "Text analysis
for Network Configuration," Mälardalen University Digital Archive, using deep neural networks in digital humanities and information
2022. [Online]. Available: urn:nbn:se:mdh:diva-58886. (OAI: science," Journal of the Association for Information Science and
oai:DiVA.org:mdh-58886) Technology, vol. 73, no. 1, 2021, doi: 10.1002/asi.24544.
[3] M. F. Islami, P. Musa, and M. Lamsani, "Implementation of Network [22] P. Abrahamsson et al., "ChatGPT as a Fullstack Web Developer - Early
Automation using Ansible to Configure Routing Protocol in Cisco and Results," Tampere University, Finland, 2023.
Mikrotik Router with Raspberry PI: Array," Jurnal Ilmiah
KOMPUTASI, vol. 19, no. 2, pp. 127-134, 2020 [23] H. Tian et al., "Is ChatGPT the Ultimate Programming Assistant—How
far is it?" arXiv preprint arXiv:2304.11938 [cs.SE], 2023. Available at:
[4] M. F. Mohd Fuzi, K. Abdullah, I. H. Abd Halim, and R. Ruslan, https://doi.org/10.48550/arXiv.2304.11938.
"Network automation using ansible for EIGRP network," Journal of
[24] M. Omar and S. Shiaeles, "VulDetect: A novel technique for detecting
Computing Research and Innovation (JCRINN), vol. 6, no. 4, pp. 59-
69, 2021. software vulnerabilities using Language Models," 2023 IEEE
International Conference on Cyber Security and Resilience (CSR),
[5] I. Ortiz-Garcés, A. Echeverría, and R. Andrade, "Automation Tasks Venice, Italy, 2023, pp. 105-110, doi:
Model for Improving Hardening Levels on Campus Networks," in 2021 10.1109/CSR57506.2023.10224924.
Fifth World Conference on Smart Trends in Systems Security and
Sustainability (WorldS4), July 2021, pp. 30-35, IEEE.
[6] E. Mulyana and G. Fakih, "Network Automation with a Single Source
of Truth in a Heterogeneous Environment," International Journal on
Electrical Engineering and Informatics, vol. 14, no. 1, 2022.
[7] A. Khumaidi, "Implementation of Devops Method for Automation of
Server Management Using Ansible," Jurnal Transformatika, vol. 18,
no. 2, pp. 199-209, 2021.
[8] A. Elezi and D. A. Karras, "On Detailed Network Systems
Configuration Management Automation using Python," WSEAS
TRANSACTIONS ON COMMUNICATIONS, 2023.
[9] L. Firdaouss, B. Ayoub, M. Manal, and Y. Ikrame, "Automated VPN
configuration using DevOps," Procedia Computer Science, vol. 198,
pp. 632-637, 2022.
[10] O. W. J. Altalebi and A. A. Ibrahim, "Optimization of Elapsed Time of
Automation for Large-Scale Traditional Networks and Proposing New
Automation Scripts," in 2022 International Congress on Human-
Computer Interaction, Optimization and Robotic Applications
(HORA), Ankara, Turkey, 2022, pp. 1-10, doi:
10.1109/HORA55278.2022.9799873.
[11] M. Al-Mekhlal, A. AlYahya, A. Aldhamin, and A. Khan, "Network
Automation Python-based Application: The performance of a Multi-
Layer Cloud Based Solution," in 2022 IEEE International Conference
on Omni-layer Intelligent Systems (COINS), August 2022, pp. 1-8,
IEEE.
[12] A. L. Ptr and B. H. Hayadi, "IOT Network Automation Implementation
in Router Configuration Handling: Implementasi IOT Network
Automation Pada Penanganan Konfigurasi Router," Jurnal MESIL
(Mesin Elektro Sipil), vol. 3, no. 1, pp. 13-21, 2022.
[13] A. M. Mazin, R. Ab Rahman, and M. Kassim, "Performance analysis
on network automation interaction with network devices using python,"
in 2021 IEEE 11th IEEE Symposium on Computer Applications and
Industrial Electronics (ISCAIE), April 2021, pp. 360-366, IEEE.
[14] P. Mihăilă, T. Bălan, R. Curpen, and F. Sandu, "Network Automation
and Abstraction using Python Programming Methods," MACRo 2015,
vol. 2, no. 1, pp. 95-103, 2017.
[15] L. G. Mauboy and T. Wellem, "Studi Perbandingan Library Untuk
Implementasi Network Automation Menggunakan Paramiko Dan
Netmiko Pada Router Mikrotik," JURIKOM (Jurnal Riset Komputer),
vol. 9, no. 4, pp. 790-799, 2022.
[16] K. Nugroho, A. D. Abrariansyah, and S. Ikhwan, "Performance
Comparison between Paramiko and Netmiko Libraries in Network
Automation Process," InfoTekJar: Jurnal Nasional Informatika dan
Teknologi Jaringan, vol. 5, no. 1, pp. 1-8, 2020.
[17] I. Ortiz-Garcés, A. Echeverría, and R. Andrade, "Automation Tasks
Model for Improving Hardening Levels on Campus Networks," in 2021
Fifth World Conference on Smart Trends in Systems Security and
Sustainability (WorldS4), July 2021, pp. 30-35, IEEE.
[18] L. G. Mauboy and T. Wellem, "Studi Perbandingan Library Untuk
Implementasi Network Automation Menggunakan Paramiko Dan
Netmiko Pada Router Mikrotik," JURIKOM (Jurnal Riset Komputer),
vol. 9, no. 4, pp. 790-799, 2022.
[19] M. Shyam and S. Dr.Ambareesh, "NETWORK AUTOMATION,"
International Journal of Innovative Research in Advanced Engineering,
vol. 9, pp. 203-206, 2022. DOI: 10.26562/ijirae.2022.v0908.10.

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO CEARA. Downloaded on January 28,2025 at 20:52:38 UTC from IEEE Xplore. Restrictions apply.