Mastering Python

Network
Automation
Automating Container Orchestration, Configuration,
and Networking with Terraform, Calico, HAProxy,
and Istio

Tim Peters
Copyright © 2023 GitforGits
All rights reserved.
 Content

Preface ................................................................................................................................ xiv

CHAPTER 1: PYTHON ESSENTIALS FOR NETWORKS .............................................. 1
Role of Python in Network Programming ................................................................. 2
Overview ......................................................................................................................................... 2
Factors Benefitting Networking .................................................................................................. 2
Learn to use Data Types ................................................................................................. 3
Numeric Data Types ..................................................................................................................... 3
Boolean Data Type ........................................................................................................................ 4
Sequence Data Types .................................................................................................................... 4
Mapping Data Types ..................................................................................................................... 6
Set Data Types................................................................................................................................ 6
Binary Data Types ......................................................................................................................... 7
Exploring Loops ................................................................................................................ 7
For Loops........................................................................................................................................ 8
While Loops .................................................................................................................................... 9
Working with Functions ................................................................................................ 11
Defining Functions ...................................................................................................................... 11
Calling Functions ......................................................................................................................... 11
Default Arguments ...................................................................................................................... 12
Variable-length Arguments ........................................................................................................ 12
Lambda Functions ....................................................................................................................... 13
Recursion....................................................................................................................................... 14
Global and Local Variables ........................................................................................................ 14
Function Arguments.................................................................................................................... 15
Nested Functions ......................................................................................................................... 16
Summary ............................................................................................................................ 17
CHAPTER 2: FILE HANDLING AND MODULES IN PYTHON .................................. 18
File Handling ................................................................................................................... 19
iii
 Opening and Closing Files ......................................................................................................... 19
Reading from Files ....................................................................................................................... 19
Writing to Files ............................................................................................................................. 20
Appending to Files....................................................................................................................... 21
With Statement ............................................................................................................................. 21
Exception Handling:.................................................................................................................... 22
Utilizing Modules ............................................................................................................23
Creating a Module ........................................................................................................................ 23
Importing a Module..................................................................................................................... 24
Built-In Modules .......................................................................................................................... 24
Creating Packages ........................................................................................................................ 25
Standard Library Modules .......................................................................................................... 25
My First Python Script....................................................................................................26
Summary .............................................................................................................................28
CHAPTER 3: PREPARING NETWORK AUTOMATION LAB...................................... 29
Components of Network Automation Process ........................................................30
Network Devices ......................................................................................................................... 30
Network Emulator....................................................................................................................... 31
Python Environment................................................................................................................... 32
Automation Scripts ...................................................................................................................... 33
Putting It All Together ...................................................................................................33
Benefits of Network Automation Lab ........................................................................34
Install NS3 Network Simulator ....................................................................................34
System Requirements .................................................................................................................. 35
Install Required Dependencies .................................................................................................. 35
Download NS-3 ........................................................................................................................... 36
Install Python ....................................................................................................................37
Update System .............................................................................................................................. 37
Install Python................................................................................................................................ 38
Install pip ....................................................................................................................................... 38
Install paramiko, Netmiko and Nornir..................................................................................... 38

iv
 Install Virtual Environment ......................................................................................... 39
Create Virtual Environment ....................................................................................................... 39
Activate the Virtual Environment ............................................................................................. 39
Install Python Libraries in Virtual Environment .................................................................... 39
Deactivate the Virtual Environment......................................................................................... 39
Install Visual Studio Code ............................................................................................ 40
Download and Install VS Code ................................................................................................. 40
Install Python Extension ............................................................................................................ 40
Configure Python Interpreter .................................................................................................... 40
Create Python Project ................................................................................................................. 41
Write Python Code ...................................................................................................................... 41
Run Python Code ........................................................................................................................ 41
Summary ............................................................................................................................ 42
CHAPTER 4: CONFIGURING LIBRARIES AND LAB COMPONENTS ......................... 43
Nornir ................................................................................................................................. 44
Architecture of Nornir ................................................................................................................ 44
Significance of Nornir ................................................................................................................. 45
Paramiko ............................................................................................................................ 46
Architecture of Paramiko ........................................................................................................... 46
Significance of Paramiko ............................................................................................................ 47
Netmiko ............................................................................................................................. 48
Architecture of Netmiko ............................................................................................................ 48
Significance of Netmiko ............................................................................................................. 49
PyEZ ................................................................................................................................... 50
Architecture of PyEZ .................................................................................................................. 51
Significance of PyEZ ................................................................................................................... 51
Configure nornir, paramiko, netmiko and pyEZ ................................................... 53
Installing and Configuring Nornir............................................................................................. 53
Installing and Configuring Paramiko ........................................................................................ 54
Installing and Configuring Netmiko ......................................................................................... 54
Installing and Configuring PyEZ .............................................................................................. 55

v
 Configure Ports .................................................................................................................56
Configuring Ports on Switches .................................................................................................. 57
Configuring Ports on Routers .................................................................................................... 59
Configure Hosts ...............................................................................................................60
Configuring Hosts on Windows ................................................................................................ 61
Configuring Hosts on Linux ...................................................................................................... 62
Configure Servers .............................................................................................................63
Installing Server Operating System ........................................................................................... 63
Configuring Network Settings ................................................................................................... 63
Installing and Configuring Server Software ............................................................................. 64
Configure Network Encryption ...................................................................................65
SSL/TLS........................................................................................................................................ 65
IPsec ............................................................................................................................................... 65
SSH................................................................................................................................................. 65
VPN ............................................................................................................................................... 66
Testing the Network Automation Environment ....................................................66
Test Connectivity between Hosts ............................................................................................. 66
Test Port Connectivity ................................................................................................................ 67
Test SSH Connectivity ................................................................................................................ 69
Test Network Automation Libraries ........................................................................................ 69
Test NS3 Emulator ...................................................................................................................... 70
Test Network Encryption........................................................................................................... 72
Summary .............................................................................................................................72
CHAPTER 5: CODE, TEST & VALIDATE NETWORK AUTOMATION ....................... 73
Understanding Network Automation Scripts ..........................................................74
Procedure of Network Automation Scripts ..............................................................75
Define Variables for Automation Scripts ..................................................................76
Install Required Libraries ............................................................................................................ 76
Import Libraries ........................................................................................................................... 76
Define Variables ........................................................................................................................... 76
Connect to Device ....................................................................................................................... 77

vi
 Send Configuration Commands ................................................................................................ 77
Close Connection......................................................................................................................... 78
Create Script to Use Variables ................................................................................................... 78
Run the Script ............................................................................................................................... 79
Write Codes using Python Tools ................................................................................ 79
Install Required Libraries and Tools ......................................................................................... 80
Import Libraries ........................................................................................................................... 80
Define Inventory.......................................................................................................................... 80
Define Tasks ................................................................................................................................. 81
Define Playbook........................................................................................................................... 81
Execute the Script ........................................................................................................................ 82
Test and Validate the Script ....................................................................................................... 82
Testing Network Automation Scripts ....................................................................... 83
Set Up a Test Environment ....................................................................................................... 84
Create Test Cases ......................................................................................................................... 85
Run the Code ................................................................................................................................ 85
Document Test Results............................................................................................................... 86
Debug Errors .................................................................................................................... 86
Identify the Error or Issue ......................................................................................................... 86
Review the Code .......................................................................................................................... 87
Use Print Statements ................................................................................................................... 87
Use a Debugger ............................................................................................................................ 87
Fix the Error or Issue .................................................................................................................. 88
Validate Network Automation Scripts ...................................................................... 88
Prepare the Production Environment ...................................................................................... 89
Deploy Code to Production Environment or Devices.......................................................... 89
Run the Code on Production Environment or Devices........................................................ 90
Verify the Output......................................................................................................................... 91
Summary ............................................................................................................................ 91
CHAPTER 6: AUTOMATION OF CONFIGURATION MANAGEMENT ........................ 92
Why Configuration Management? ............................................................................. 93

vii
 Need of Configuration Management ........................................................................................ 93
Role of Python in Configuration Management ....................................................................... 94
Server Provisioning with Terraform ...........................................................................95
Set up AWS Credentials .............................................................................................................. 96
Install Terraform .......................................................................................................................... 96
Define Terraform Configuration............................................................................................... 96
Initialize Terraform...................................................................................................................... 97
Apply Terraform Configuration ................................................................................................ 97
Connect to EC2 Instance ........................................................................................................... 98
Creating Server ............................................................................................................................. 98
Testing Server ............................................................................................................................... 98
Using Python to Automate System Settings ............................................................99
Import Necessary Modules ........................................................................................................ 99
Define Timezone ......................................................................................................................... 99
Execute Command to Change Timezone .............................................................................. 100
Verify Setting the Timezone .................................................................................................... 100
Using Python to Modify Base Configurations ..................................................... 101
Using Terraform to Modify Base Configurations ............................................... 102
Automating System Identification ........................................................................... 103
Install Terraform Module ......................................................................................................... 103
Python Script to Retrieve System Information ..................................................................... 103
Using Python to Automate Patches and Updates ............................................... 104
Install Necessary Libraries ........................................................................................................ 104
Check for Available Updates ................................................................................................... 105
Upgrade the System ................................................................................................................... 105
Reboot the System ..................................................................................................................... 105
Schedule Regular Updates ........................................................................................................ 105
Using Terraform to Roll Patches and Updates .................................................... 106
Create Configuration File ......................................................................................................... 106
Applying Configuration File..................................................................................................... 106
Identify Unstable and Non-compliant Configurations...................................... 107

viii
 Establish Connection with Device .......................................................................................... 108
Retrieve Running Configuration ............................................................................................. 109
Search Non-compliant Interfaces............................................................................................ 109
Fixing Non-compliant Configurations ................................................................................... 110
Summary .......................................................................................................................... 110
CHAPTER 7: MANAGING DOCKER AND CONTAINER NETWORKS ....................... 112
Docker and Containers ................................................................................................ 113
Docker & Container Fundmentals.......................................................................................... 113
Benefits & Applications ............................................................................................................ 113
Role of Python in Containerization .......................................................................... 114
Install and Configure Docker .................................................................................... 115
Install Docker ............................................................................................................................. 115
Install Docker Python Module ................................................................................................ 116
Create Dockerfile ....................................................................................................................... 116
Build Docker Image .................................................................................................................. 117
Run Docker Container .............................................................................................................. 117
Test Docker Container.............................................................................................................. 117
Using Python to Build Docker Images ................................................................... 118
Create DockerFile ...................................................................................................................... 118
Install Dependencies ................................................................................................................. 119
Define Command ...................................................................................................................... 119
Build Docker Image .................................................................................................................. 119
Run Container ............................................................................................................................ 120
Running Containers...................................................................................................... 120
Automate Running of Containers ............................................................................. 121
Install Docker SDK for Python .............................................................................................. 122
Import Docker SDK ................................................................................................................. 122
Connect to Docker Daemon ................................................................................................... 122
Define Container Configuration.............................................................................................. 122
Create Container ........................................................................................................................ 122
Start the Container ..................................................................................................................... 122

ix
 Stop and Remove Containers .................................................................................................. 123
Container Network Management ............................................................................ 123
Overview ..................................................................................................................................... 123
Managing Container Networks with Docker SDK .............................................................. 124
Summary .......................................................................................................................... 125
CHAPTER 8: ORCHESTRATING CONTAINER & WORKLOADS .............................. 127
Container Scheduling and Workload Automation .............................................. 128
Network Service Disocvery ........................................................................................ 128
Understanding etcd ...................................................................................................... 129
Service Discovery using etcd ..................................................................................... 130
Install etcd ................................................................................................................................... 130
Start etcd...................................................................................................................................... 131
Register Services ......................................................................................................................... 131
Discover Services ....................................................................................................................... 131
Automate Service Discovery .................................................................................................... 131
Sample Program to Automate Service Discovery ................................................ 132
Kubernetes Load Balancers ....................................................................................... 134
Exploring HAProxy ...................................................................................................... 135
Manage Load Balancer Servers using HAProxy.................................................. 137
Import Required Libraries ........................................................................................................ 137
Define API Endpoint URLs .................................................................................................... 137
Define Function to Add or Remove Servers ........................................................................ 137
Call Function .............................................................................................................................. 137
Sample Program to Manage Load Balancer Servers .......................................... 138
Automate Add/Manage SSL Certificate ................................................................ 139
Using Cryptography Library to Automate SSL ..................................................................... 139
Step-by-step Illustration of Sample Program ........................................................................ 141
Manage Container Storage ......................................................................................... 142
Sample Program ......................................................................................................................... 142
Step-by-step Illustration of Sample Program ........................................................................ 143
Necessity of Container Performance....................................................................... 143
x
 Why Container Performance? .................................................................................................. 143
Container Performance KPIs .................................................................................................. 143
Setting Up Container Performance Monitoring ................................................... 144
Install the Required Libraries ................................................................................................... 144
Import Required Libraries ........................................................................................................ 145
Connect to Docker API ............................................................................................................ 145
Get Container List ..................................................................................................................... 145
Pull Performance Metrics ......................................................................................................... 145
Print Container Metrics............................................................................................................. 145
Automated Rolling of Updates .................................................................................. 146
Get Current Deployment Object ............................................................................................ 146
Update Deployment Object ..................................................................................................... 147
Check Status of Deployment Rollout ..................................................................................... 147
Clean Up Resources................................................................................................................... 147
Summary .......................................................................................................................... 148
CHAPTER 9: POD NETWORKING ........................................................................ 150
Pods and Pod Networking ......................................................................................... 151
What are Pods? ........................................................................................................................... 151
Pods beyond Containers ........................................................................................................... 151
Networking in Pods................................................................................................................... 152
Setting Up Pod Network ............................................................................................. 153
Choose a Pod Network Provider ............................................................................................ 153
Install Pod Network Provider .................................................................................................. 153
Configure Pod Network ........................................................................................................... 154
Verify the Pod Network ........................................................................................................... 154
Exploring Calico ............................................................................................................ 154
Overview ..................................................................................................................................... 154
Characteristics of Calico ........................................................................................................... 154
Getting Started with Calico ...................................................................................................... 155
Using Calico to Setup Pod Network ........................................................................ 156
Routing Protocols.......................................................................................................... 158

xi
 Border Gateway Protocol (BGP) ............................................................................................ 158
Open Shortest Path First (OSPF) ........................................................................................... 158
Intermediate System to Intermediate System (IS-IS) ........................................................... 159
Routing Information Protocol (RIP) ...................................................................................... 159
Exploring Cilium .......................................................................................................... 159
Key Features of Cilium ............................................................................................................. 160
Cilium Architecture ................................................................................................................... 161
Install Cilium............................................................................................................................... 162
Automation of Network Policies .............................................................................. 163
Overview ..................................................................................................................................... 163
Steps for Network Policies Automation ................................................................................ 163
Using Calico to Automate Network Policies ........................................................ 164
Workload Routing ......................................................................................................... 166
Need of Workload Routing ...................................................................................................... 166
Istio............................................................................................................................................... 167
Linkerd......................................................................................................................................... 167
Consul .......................................................................................................................................... 167
Summary .......................................................................................................................... 168
CHAPTER 10: IMPLEMENTING SERVICE MESH ................................................... 169
Service-to-Service Communication .......................................................................... 170
Remote Procedure Calls (RPCs) .............................................................................................. 170
Message-based Communication .............................................................................................. 170
Need of Service-to-Service ....................................................................................................... 170
Rise of Service Mesh .................................................................................................... 171
Exploring Istio ............................................................................................................... 172
Overview ..................................................................................................................................... 172
Istio’s Capabilities ...................................................................................................................... 173
Installing Istio ................................................................................................................ 174
Cluster Traffic ................................................................................................................ 175
NodePort..................................................................................................................................... 175
LoadBalancer .............................................................................................................................. 175

xii
 Ingress .......................................................................................................................................... 175
Istio Control Plane..................................................................................................................... 176
Using Istio to Route Traffic ....................................................................................... 177
Metrics, Logs and Traces ........................................................................................... 180
Metrics ......................................................................................................................................... 180
Logs .............................................................................................................................................. 180
Traces ........................................................................................................................................... 180
Using Grafana to Collect Metrics ............................................................................. 181
Steps to Collect Metrics ............................................................................................................ 181
Summary .......................................................................................................................... 183

xiii
 Preface
With "Mastering Python Network Automation," you can streamline container
orchestration, configuration management, and resilient networking with Python and its
libraries, allowing you to emerge as a skilled network engineer or a strong DevOps
professional.

From the ground up, this guide walks readers through setting up a network automation
lab using the NS3 network simulator and Python programming. This includes the
installation of NS3, as well as python libraries like nornir, paramiko, netmiko, and PyEZ,
as well as the configuration of ports, hosts, and servers. This book will teach you the skills
to become a proficient automation developer who can test and fix any bugs in
automation scripts. This book examines the emergence of the service mesh as a solution
to the problems associated with service-to-service communication over time.

This book walks you through automating various container-related tasks in Python and its
libraries, including container orchestration, service discovery, load balancing, container
storage management, container performance monitoring, and rolling updates. Calico and
Istio are two well-known service mesh tools, and you'll find out how to set them up and
configure them to manage traffic routing, security, and monitoring.

Additional topics covered in this book include the automation of network policies, the
routing of workloads, and the collection and monitoring of metrics, logs, and traces. You'll
also pick up some tips and tricks for collecting and visualising Istio metrics with the help
of tools like Grafana.

In this book you will learn how to:

 Use of Istio for cluster traffic management, traffic routing, and service mesh
implementation.
 Utilizing Cilium and Calico to solve pod networking and automate network policy
and workload routing.
 Monitoring and managing Kubernetes clusters with etcd and HAProxy load
balancers and container storage.
 Establishing network automation lab with tools like NS3 emulator, Python, Virtual
Environment, and VS Code.
 Establishing connectivity between hosts, port connectivity, SSH connectivity,
python libraries, NS3, and network encryption.

xiv
 GitforGits
Prerequisites
"Mastering Python Network Automation" is an essential guide for network engineers,
DevOps professionals, and developers who want to streamline container orchestration and
resilient networking with the help of Terraform, Calico, and Istio. Knowing Python and
basics of networking is sufficient to pursue this book.

Codes Usage
Are you in need of some helpful code examples to assist you in your programming and
documentation? Look no further! Our book offers a wealth of supplemental material,
including code examples and exercises.

Not only is this book here to aid you in getting your job done, but you have our permission
to use the example code in your programs and documentation. However, please note that
if you are reproducing a significant portion of the code, we do require you to contact us
for permission.

But don't worry, using several chunks of code from this book in your program or answering
a question by citing our book and quoting example code does not require permission. But
if you do choose to give credit, an attribution typically includes the title, author, publisher,
and ISBN. For example, "Mastering Python Network Automation by Tim Peters".

If you are unsure whether your intended use of the code examples falls under fair use or
the permissions outlined above, please do not hesitate to reach out to us at
[email protected].

We are happy to assist and clarify any concerns.

xv