Download the Full Version of textbook for Fast Typing at textbookfull.

com

Python Network Programming Cookbook Kathiravelu

https://textbookfull.com/product/python-network-programming-
cookbook-kathiravelu/

OR CLICK BUTTON

DOWNLOAD NOW

Download More textbook Instantly Today - Get Yours Now at textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Python Network Programming Cookbook Practical solutions to

overcome real world networking challenges 2nd Edition
Pradeeban Kathiravelu
https://textbookfull.com/product/python-network-programming-cookbook-
practical-solutions-to-overcome-real-world-networking-challenges-2nd-
edition-pradeeban-kathiravelu/
textboxfull.com

Python GUI Programming Cookbook Meier

https://textbookfull.com/product/python-gui-programming-cookbook-
meier/

textboxfull.com

Python GUI Programming Cookbook Burkhard A. Meier

https://textbookfull.com/product/python-gui-programming-cookbook-
burkhard-a-meier/

textboxfull.com

Java Network Programming Fourth Edition Harold Elliotte

https://textbookfull.com/product/java-network-programming-fourth-
edition-harold-elliotte/

textboxfull.com
Python Digital Forensics Cookbook Effective Python recipes
for digital investigations 1st Edition Preston Miller

https://textbookfull.com/product/python-digital-forensics-cookbook-
effective-python-recipes-for-digital-investigations-1st-edition-
preston-miller/
textboxfull.com

Network Programming With Rust 1st Edition Abhishek Chanda

https://textbookfull.com/product/network-programming-with-rust-1st-
edition-abhishek-chanda/

textboxfull.com

Python Advanced Programming: The guide to learn pyhton

programming Marcus Richards

https://textbookfull.com/product/python-advanced-programming-the-
guide-to-learn-pyhton-programming-marcus-richards/

textboxfull.com

Learn Python Programming Second Edition Fabrizio Romano

https://textbookfull.com/product/learn-python-programming-second-
edition-fabrizio-romano/

textboxfull.com

Advanced Guide to Python 3 Programming Hunt

https://textbookfull.com/product/advanced-guide-to-
python-3-programming-hunt/

textboxfull.com
Python Network Programming Cookbook
Second Edition

Overcome real-world networking challenges

Pradeeban Kathiravelu
Dr. M. O. Faruque Sarker

BIRMINGHAM - MUMBAI
Python Network Programming Cookbook
Second Edition
Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the authors, nor Packt Publishing, and its
dealers and distributors will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

First published: March 2014

Second edition: August 2017

Production reference: 1080817

Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.

ISBN 978-1-78646-399-9

www.packtpub.com
 Credits

Authors Copy Editors

Pradeeban Kathiravelu Safis Editing
Dr. M. O. Faruque Sarker Juliana Nair

Reviewers Project Coordinator

Dr. S. Gowrishankar Judie Jose
Michael Bright

Commissioning Editor Proofreader

Kartikey Pandey Safis Editing

Acquisition Editor Indexer

Rahul Nair Aishwarya Gangawane

Content Development Editor Graphics

Abhishek Jadhav Kirk D'Penha

Technical Editor Production Coordinator

Mohd Riyan Khan Aparna Bhagat
 About the Author
Pradeeban Kathiravelu is an open source evangelist. He is a Ph.D. researcher at INESC-ID
Lisboa/Instituto Superior Tecnico, Universidade de Lisboa, Portugal, and Universite
Catholique de Louvain, Belgium. He is a Fellow of Erasmus Mundus Joint Degree in
Distributed Computing (EMJD-DC), researching a software-defined approach to quality of
service and data quality in multi-tenant clouds.

Pradeeban holds a master of science degree, Erasmus Mundus European Master in

Distributed Computing (EMDC), from Instituto Superior Tecnico, Portugal and KTH Royal
Institute of Technology, Sweden. He also holds a first class bachelor of science in
engineering (Hons) degree, majoring in computer science and engineering, from the
University of Moratuwa, Sri Lanka. His research interests include Software-Defined
Networking (SDN), distributed systems, cloud computing, web services, big data in
biomedical informatics, Network Functions Virtualizations (NFV), and data mining. He is
very interested in free and open source software development and has been an active
participant in the Google Summer of Code (GSoC) program since 2009, as a student and as a
mentor.

Pradeeban has published several conference papers and co-authored a few book chapters.
He has also worked on OpenDaylight Cookbook and Learning OpenDaylight as a technical
reviewer. Python Network Programming Cookbook, Second Edition (2017) is his first book as an
author, and he is quite excited about it.

I would like to thank my readers for the interest in the book. Please feel free to contact me if
you need any assistance in the topics or the recipes, beyond what we have discussed in the
book. I would like to thank the entire editorial team at Packt, including Abhishek Jadhav,
Rahul Nair, and Mohd Riyan Khan. I would like to extend my thanks to the Linux
Foundation for their open source projects on softwarization of networks and systems. I
would like to thank my friends and colleagues who helped me in various ways. I would like
to thank Prof. Luís Veiga (INESC-ID Lisboa), my MSc and Ph.D. advisor, for sharing his
wisdom and encouragement throughout my stay in Instituto Superior Técnico. I would
like to thank him for being my mentor since 2012. I would also like to thank Prof. Ashish
Sharma (Emory University, Atlanta) for his guidance and motivation.

My special thanks go to my loving wife, Juejing Gu. This book would not be a reality
without her continuous support and creative suggestions. Her tireless efforts helped me
always be on time without missing the deadlines.
I would like to thank my mom, Selvathie Kathiravelu, for her support.
Dr. M. O. Faruque Sarker is a software architect based in London, UK, where he has been
shaping various Linux and open source software solutions, mainly on cloud computing
platforms, for commercial companies, educational institutions, and multinational
consultancies. Over the past 10 years, he has been leading a number of Python software
development and cloud infrastructure automation projects. In 2009, he started using
Python, where he was responsible for shepherding a fleet of miniature E-puck robots at the
University of South Wales, Newport, UK. Later, he honed his Python skills, and he was
invited to work on the Google Summer of Code (2009/2010) programs for contributing to
the BlueZ and Tahoe-LAFS open source projects. He is the author of Python Network
Programming Cookbook and Learning Python Network Programming both by Packt Publishing.

He received his Ph.D. in multi-robot systems from the University of South Wales. He is
currently working at University College London. He takes an active interest in cloud
computing, software security, intelligent systems, and child-centric education. He lives in
East London with his wife, Shahinur, and daughter, Ayesha.

All praises and thanks to Allah, the God who is the Merciful and the Beneficent. I would
not be able to finish this book without the help of God.I would like to thank everyone who
has contributed to the publication of this book, including the publisher, technical reviewers,
editors, my family and friends for their sacrifice of time, encouraging words, and smiles,
especially my wife Shahinur Rijuani for her love and support in my work. I also thank the
readers who have patiently been waiting for this book and who have given me lots of
valuable feedback.
 About the Reviewers
Dr. S. Gowrishankar is currently working as an associate professor in the Department of
Computer Science and Engineering at Dr. Ambedkar Institute of Technology, Bengaluru,
Karnataka, India.

He received his Ph.D. in Engineering from Jadavpur University, Kolkata, West Bengal,
India in 2010, MTech in software engineering and BE in computer science and engineering
from Visvesvaraya Technological University (VTU), Belagavi, Karnataka, India in the year
2005 and 2003 respectively.

From 2011 to 2014 he worked as a senior research scientist and tech lead at Honeywell
Technology Solutions, Bengaluru, Karnataka, India.

He has published several papers in various reputed international journals and conferences.
He is serving as an editor and reviewer for various prestigious international journals. He is
also a member of IEEE, ACM, CSI, and ISTE.

He has delivered many keynote addresses and invited talks throughout India on a variety
of subjects related to computer science and engineering. He was instrumental in organizing
several conferences, workshops, and seminars. He has also served on the panel of a number
of academic bodies of universities and autonomous colleges as a BOS and BOE member.

His current research interests are mainly focused on data science, including its technical
aspects as well as its applications and implications. Specifically, he is interested in the
applications of Machine Learning, Data Mining, and Big Data Analytics in Healthcare.

I would like to acknowledge my earnest gratitude to my wife, Roopa K M, for her constant
source of support and encouragement throughout this assignment. I’m truly thankful to
almighty God for having her in my life and give her my deepest expression of love and
appreciation.
Michael Bright, RHCE/RHCSA, is a solution architect working in the HPE EMEA Customer
Innovation Center.

He has strong experience across Cloud and Container technologies (Docker, Kubernetes,
AWS, GCP, Azure) as well as NFV/SDN.

Based in Grenoble, France, he runs a Python user group and is a co-organizer of the Docker
and FOSS Meetup groups. He has a keen interest in Container, Orchestration, and
Unikernel technologies on which he has presented and run training tutorials in several
conferences.
He has presented many times on subjects diverse as NFV, Docker, Container Orchestration,
Unikernels, Jupyter Notebooks, MongoDB, and Tmux.

Michael has a wealth of experience across pure research, R&D and pre-sales consulting
roles.
 www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and
ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a
print book customer, you are entitled to a discount on the eBook copy. Get in touch with us
at [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a
range of free newsletters and receive exclusive discounts and offers on Packt books and
eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt
books and video courses, as well as industry-leading tools to help you plan your personal
development and advance your career.

Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
 Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial
process. To help us improve, please leave us an honest review on this book's Amazon page
at https://www.amazon.com/dp/1786463997.

If you'd like to join our team of regular reviewers, you can e-mail us at
[email protected]. We award our regular reviewers with free eBooks and
videos in exchange for their valuable feedback. Help us be relentless in improving our
products!
I dedicate this book to the world, in memory of my dad, Kanapathipillai Kathiravelu.
 Table of Contents
Preface 1
Chapter 1: Sockets, IPv4, and Simple Client/Server Programming 11
Introduction 12
Printing your machine's name and IPv4 address 12
Getting ready 12
How to do it... 13
How it works... 15
Retrieving a remote machine's IP address 16
How to do it... 16
How it works... 17
Converting an IPv4 address to different formats 18
How to do it... 18
How it works... 19
Finding a service name, given the port and protocol 19
Getting ready 19
How to do it... 19
How it works... 20
Converting integers to and from host to network byte order 20
How to do it... 20
How it works... 21
Setting and getting the default socket timeout 22
How to do it... 22
How it works... 23
Handling socket errors gracefully 23
How to do it... 23
How it works... 25
Modifying a socket's send/receive buffer sizes 26
How to do it... 26
How it works... 28
Changing a socket to the blocking/non-blocking mode 28
How to do it... 28
How it works... 29
Reusing socket addresses 29
How to do it... 30
 How it works... 32
Printing the current time from the internet time server 32
Getting ready 32
How to do it... 33
How it works... 34
Writing an SNTP client 34
How to do it... 34
How it works... 35
Writing a simple TCP echo client/server application 36
How to do it... 36
How it works... 39
Writing a simple UDP echo client/server application 39
How to do it... 39
How it works... 42
Chapter 2: Multiplexing Socket I/O for Better Performance 43
Introduction 43
Using ForkingMixIn in your socket server applications 44
How to do it... 44
How it works... 47
Using ThreadingMixIn in your socket server applications 48
Getting ready 48
How to do it... 48
How it works... 50
Writing a chat server using select.select 50
How to do it... 51
How it works... 58
Multiplexing a web server using select.epoll 58
How to do it... 58
How it works... 61
Multiplexing an echo server using Diesel concurrent library 62
Getting ready 62
How to do it... 64
How it works... 66
Chapter 3: IPv6, Unix Domain Sockets, and Network Interfaces 67
Introduction 67
Forwarding a local port to a remote host 68
How to do it... 68
How it works... 71

[ ii ]
Pinging hosts on the network with ICMP 72
Getting ready 72
How to do it... 72
How it works... 76
Waiting for a remote network service 77
How to do it... 77
How it works... 80
Enumerating interfaces on your machine 80
Getting ready 81
How to do it... 81
How it works... 82
Finding the IP address for a specific interface on your machine 83
Getting ready 83
How to do it... 83
How it works... 84
Finding whether an interface is up on your machine 85
Getting ready 85
How to do it... 85
How it works... 86
Detecting inactive machines on your network 87
Getting ready 87
How to do it... 87
How it works... 89
Performing a basic IPC using connected sockets (socketpair) 89
Getting ready 89
How to do it... 90
How it works... 91
Performing IPC using Unix domain sockets 91
How to do it... 91
How it works... 94
Finding out if your Python supports IPv6 sockets 94
Getting ready 94
How to do it... 95
How it works... 96
Extracting an IPv6 prefix from an IPv6 address 97
How to do it... 97
How it works... 99
Writing an IPv6 echo client/server 99
How to do it... 99

[ iii ]
 How it works... 102
Chapter 4: Programming with HTTP for the Internet 103
Introduction 104
Downloading data from an HTTP server 104
How to do it... 104
How it works... 107
Serving HTTP requests from your machine 107
How to do it... 108
How it works... 110
Extracting cookie information after visiting a website 110
How to do it... 110
How it works... 113
Submitting web forms 114
Getting ready 114
How to do it... 114
How it works... 116
Sending web requests through a proxy server 116
Getting ready 116
How to do it... 116
How it works... 118
Checking whether a web page exists with the HEAD request 118
How to do it... 119
How it works... 121
Spoofing Mozilla Firefox in your client code 121
How to do it... 121
How it works... 122
Saving bandwidth in web requests with the HTTP compression 122
How to do it... 123
How it works... 125
Writing an HTTP fail-over client with resume and partial downloading 125
How to do it... 126
How it works... 127
Writing a simple HTTPS server code with Python and OpenSSL 128
Getting ready 128
How to do it... 128
How it works... 129
Building asynchronous network applications with Twisted 130
Getting ready 130
How to do it... 131

[ iv ]
 How it works... 134
Building asynchronous network applications with Tornado 134
Getting ready 134
How to do it... 134
How it works... 136
Building concurrent applications with Tornado Future 137
Getting ready 137
How to do it... 137
How it works... 140
Chapter 5: Email Protocols, FTP, and CGI Programming 141
Introduction 141
Listing the files in a remote FTP server 142
Getting ready 142
How to do it... 142
How it works... 144
Common error 144
Uploading a local file to a remote FTP server 146
Getting ready 146
How to do it... 146
How it works... 147
Emailing your current working directory as a compressed ZIP file 148
Getting ready 148
How to do it... 148
How it works... 151
See also 152
Downloading your Google email with POP3 152
Getting ready 152
How to do it... 152
How it works... 153
Checking your remote email with IMAP 153
Getting ready 154
How to do it... 154
How it works... 155
Sending an email with an attachment via Gmail SMTP server 156
Getting ready 156
How to do it... 156
How it works... 158
Writing a guestbook for your (Python-based) web server with CGI 158
Getting ready 158

[v]
 How to do it... 158
How it works... 162
Finding the mail server from an email address 163
Getting ready 163
How to do it... 164
How it works... 166
Writing a simple SMTP server 166
Getting ready 166
How to do it... 166
How it works... 169
Writing a secure SMTP client using TLS 170
Getting ready 170
How to do it... 170
How it works... 172
Writing an email client with POP3 173
Getting ready 173
How to do it... 174
How it works... 175
Chapter 6: Programming Across Machine Boundaries 176
Introduction 176
Executing a remote shell command using telnet 177
Getting ready 177
How to do it... 177
How it works... 179
Copying a file to a remote machine by SFTP 179
Getting ready 179
How to do it... 180
How it works... 181
Printing a remote machine's CPU information 182
Getting ready 182
How to do it... 182
How it works... 186
Installing a Python package remotely 186
Getting ready 187
How to do it... 187
How it works... 189
Running a MySQL command remotely 189
Getting ready 189
How to do it... 190

[ vi ]
 How it works... 193
Transferring files to a remote machine over SSH 193
Getting ready 193
How to do it... 194
How it works... 196
Configuring Apache remotely to host a website 196
Getting ready 197
How to do it... 197
How it works... 199
Chapter 7: Working with Web Services – XML-RPC, SOAP, and REST 201
Introduction 201
Querying a local XML-RPC server 202
Getting ready 202
How to do it... 202
How it works... 204
Writing a multithreaded, multicall XML-RPC server 205
How to do it... 205
How it works... 207
Running an XML-RPC server with a basic HTTP authentication 208
How to do it... 208
How it works... 212
Collecting some photo information from Flickr using REST 213
How to do it... 213
How it works... 217
Searching for SOAP methods from an Amazon S3 web service 217
Getting ready 217
How to do it... 218
How it works... 219
Searching Amazon for books through the product search API 219
Getting ready 220
How to do it... 220
How it works... 222
Creating RESTful web applications with Flask 222
Getting ready 223
How to do it... 223
How it works... 226
Chapter 8: Network Monitoring and Security 227
Introduction 227

[ vii ]
 Sniffing packets on your network 228
Getting ready 228
How to do it... 228
How it works... 230
Saving packets in the pcap format using the pcap dumper 230
How to do it... 231
How it works... 234
Adding an extra header in HTTP packets 235
How to do it... 235
How it works... 236
Scanning the ports of a remote host 237
How to do it... 237
How it works... 239
Customizing the IP address of a packet 239
How to do it... 239
How it works... 241
Replaying traffic by reading from a saved pcap file 241
How to do it... 241
How it works... 243
Scanning the broadcast of packets 244
How to do it... 244
How it works... 246
Chapter 9: Network Modeling 247
Introduction 247
Simulating networks with ns-3 248
Getting ready 248
How to do it... 250
How it works... 252
Emulating networks with Mininet 252
Getting ready 253
How to do it... 253
How it works... 255
Distributed network emulation with MaxiNet 255
Getting ready 256
How to do it... 257
How it works... 258
Emulating wireless networks with Mininet-WiFi 259
Getting ready 259
How to do it... 259

[ viii ]
 How it works... 263
Extending Mininet to emulate containers 264
Getting ready 265
How to do it... 266
How it works... 270
Chapter 10: Getting Started with SDN 271
Introduction 271
SDN emulation with Mininet 272
Getting ready 272
How to do it... 272
How it works... 275
Developing Software-Defined Networks with OpenDaylight controller 276
Getting ready 276
How to do it... 278
How it works... 281
Developing Software-Defined Networks with ONOS controller 281
Getting ready 282
How to do it... 284
How it works... 286
Developing Software-Defined Networks with Floodlight controller 286
Getting ready 287
How to do it... 290
How it works... 293
Developing Software-Defined Networks with Ryu controller 293
Getting ready 293
How to do it... 294
How it works... 298
Developing Software-Defined Networks with POX controller 299
Getting ready 299
How to do it... 301
How it works... 302
Developing Software-Defined Networks visually with MiniEdit 303
Getting ready 303
How to do it... 303
How it works... 307
Chapter 11: Authentication, Authorization, and Accounting (AAA) 308
Introduction 309
Finding DNS names of a network 309

[ ix ]
 Getting ready 309
How to do it... 309
How it works... 311
Finding DNS host information 311
Getting ready 311
How to do it... 311
How it works... 313
Finding DNS resource records 313
Getting ready 315
How to do it... 315
How it works... 318
Making DNS zone transfer 319
Getting ready 319
How to do it... 319
How it works... 323
Querying NTP servers 323
Getting ready 323
How to do it... 323
How it works... 325
Connecting to an LDAP server 325
Getting ready 326
How to do it... 327
How it works... 330
Making LDAP bind 331
Getting ready 331
How to do it... 331
How it works... 333
Reading and writing LDAP 333
Getting ready 333
How to do it... 334
How it works... 335
Authenticating REST APIs with Eve 336
Getting ready 336
How to do it... 336
How it works... 338
Throttling requests with RequestsThrottler 338
Getting ready 339
How to do it... 339
How it works... 342

[x]
Chapter 12: Open and Proprietary Networking Solutions 343
Introduction 343
Configuring Red PNDA 344
Getting ready 344
How to do it... 344
How it works... 347
Configuring VMware NSX for vSphere 6.3.2 348
Getting ready 348
How to do it... 349
How it works... 350
Configuring Juniper Contrail Server Manager 351
Getting ready 351
How to do it... 354
How it works... 355
Configuring OpenContrail controller 355
Getting ready 356
How to do it... 356
How it works... 357
Configuring OpenContrail cluster 357
How to do it... 358
How it works... 363
Interacting with devices running Cisco IOS XR 363
Getting ready 364
How to do it... 364
How it works... 364
Collaborating with Cisco Spark API 365
Getting ready 365
How to do it... 367
How it works... 368
Chapter 13: NFV and Orchestration – A Larger Ecosystem 369
Introduction 369
Building VNFs with OPNFV 370
Getting ready 370
How to do it... 371
How it works... 373
Packet processing with DPDK 376
Getting ready 377
How to do it... 379

[ xi ]
 How it works... 379
Parsing BMP messages with SNAS.io 380
Getting ready 381
How to do it... 382
How it works... 386
Controlling drones with a wireless network 387
Getting ready 387
How to do it... 387
How it works... 390
Creating PNDA clusters 390
Getting ready 391
How to do it... 394
How it works... 395
Chapter 14: Programming the Internet 396
Introduction 396
Checking a website status 396
Getting ready 397
How to do it... 397
How it works... 398
Benchmarking BGP implementations with bgperf 398
Getting ready 398
How to do it... 400
How it works... 400
BGP with ExaBGP 401
Getting ready 401
How to do it... 401
Looking glass implementations with Python 402
Getting ready 404
How to do it... 405
How it works... 407
Understanding the internet ecosystem with Python 407
Getting ready 408
How to do it... 409
How it works... 410
Establishing BGP connections with yabgp 410
Getting ready 411
How to do it... 411
How it works... 413

[ xii ]
Index 414

[ xiii ]
 Preface
It has been more than 3 years since Python Network Programming Cookbook was first
published. In this second edition, we extend our book to discuss the recent advancements in
the networking industry and network softwarization. The widespread use of Software-
Defined Networking (SDN), Network Functions Virtualization (NFV), and orchestration
have been addressed in detail in the latter chapters while the first eight chapters were taken
from the first edition, improved with a few new recipes based on the feedback from the
readers.

This book is an exploratory guide to network programming in Python. It has touched a

wide range of networking protocols such as TCP/UDP, HTTP/HTTPS, FTP, SMTP, POP3,
IMAP, and CGI. With the power and interactivity of Python, it brings joy and fun to
develop various scripts for performing real-world tasks on network and system
administration, web application development, interacting with your local and remote
network, low-level network packet capture and analysis, and so on. The primary focus of
this book is to give you a hands-on experience on the topics covered. So, this book covers
less theory, but it is packed with practical materials.

This book is written with a DevOps mindset, where a developer is also more or less in
charge of operation, that is, deploying the application and managing various aspects of it,
such as remote server administration, monitoring, scaling-up, and optimizing for better
performance. This book introduces you to a bunch of open-source, third-party Python
libraries, which are ideal to be used in various use cases. We elaborate in detail the
configurations of complex networking systems with helpful hints to ensure that the reader
can follow them without getting stuck.

We hope you will enjoy the recipes presented in this book and extend them to make them
even more powerful and enjoyable.

What this book covers

Chapter 1, Sockets, IPv4, and Simple Client/Server Programming, introduces you to Python's
core networking library with various small tasks and enables you to create your first client-
server application.

Chapter 2, Multiplexing Socket I/O for Better Performance, discusses various useful techniques
for scaling your client/server applications with default and third-party libraries.
Preface

Chapter 3, IPv6, Unix Domain Sockets, and Network Interfaces, focuses more on administering
your local machine and looking after your local area network.

Chapter 4, Programming with HTTP for the Internet, enables you to create a mini command-
line browser with various features such as submitting web forms, handling cookies,
managing partial downloads, compressing data, and serving secure content over HTTPS.

Chapter 5, Email Protocols, FTP, and CGI Programming, brings you the joy of automating
your FTP and e-mail tasks such as manipulating your Gmail account, and reading or
sending emails from a script or creating a guest book for your web application. We learn to
write email clients with SMTP and POP3.

Chapter 6, Programming Across Machine Boundaries, gives you a taste of automating your
system administration and deployment tasks over SSH. You can run commands, install
packages, or set up new websites remotely from your laptop.

Chapter 7, Working with Web Services – XML-RPC, SOAP, and REST, introduces you to
various API protocols such as XML-RPC, SOAP, and REST. You can programmatically ask
any website or web service for information and interact with them. For example, you can
search for products on Amazon or Google.

Chapter 8, Network Monitoring and Security, introduces you to various techniques for
capturing, storing, analyzing, and manipulating network packets. This encourages you to
go further to investigate your network security issues using concise Python scripts.

Chapter 9, Network Modeling, introduces you to the world of network simulations and
emulations. You learn to simulate networks with NS-3, and emulate networking systems
with Mininet and its extensions.

Chapter 10, Getting Started with SDN, discusses the enterprise SDN controllers, configuring
them to use in Software-Defined Networks. We learn to develop SDN visually with
MiniEdit, and configure the networks with OpenDaylight, ONOS, Floodlight, Ryu, and
POX controllers.

Chapter 11, Authentication, Authorization, and Accounting (AAA), introduces how the
networks are secured, and discusses configuring LDAP clients with Python, accounting
aspects of the network, and authentication and access of network services.

Chapter 12, Open and Proprietary Networking Solutions, discusses in detail, configuring large-
scale enterprise networking projects, including a few projects from Cisco, Juniper, VMware,
and the Linux Foundation.

[2]
Preface

Chapter 13, NFV and Orchestration – A Larger Ecosystem, discusses configuring complex
NFV and orchestration systems of the Linux Foundation, such as OPNFV, DPDK, SNAS.io,
Dronekit, and PNDA. We elaborate the use of Python in these complex systems.

Chapter 14, Programming the Internet, presents you various Python libraries for BGP
protocol and implementations developed for the internet scale. We learn to use and
benchmark libraries such as exabgp and yabgp, and also discuss the looking glass
implementations with Python.

What you need for this book

You need a working PC or laptop, preferably with a modern Linux operating system. The
installation instructions are written and tested on Ubuntu 16.04 LTS and would work on
any recent Debian-based Linux operating system without modification. We developed for
Python 3. However, we have maintained backward-compatibility with Python 2 in our
recipes as much as we can. On the other hand, some open source projects used in this book
do not yet support Python 3. So, ideally, you will need both Python 2 and Python 3 to test
all the recipes in this book.

Most of the recipes in this book will run on other platforms such as Windows and Mac OS
with some changes in the configuration steps. Some of the recipes require two or more
computers in a cluster to test the distributed systems. You may use Amazon Web Services
(AWS) to initiate a cluster inside a single placement group to test these recipes.

You also need a working internet connection to install the third-party software libraries
mentioned with respective recipes. If you do not have a stable or continuous internet
connection, you can download the third-party libraries and install them in one go.
However, it is highly recommended to test some of these recipes with the internet
connection, as it would make the configuration task minimal and more interesting, than
having to download a bulk of software in bunch. Moreover, testing the application in an
AWS cluster would certainly require the internet connectivity.

The following is a list of the Python third-party libraries with their download URLs:

ntplib: https://pypi.python.org/pypi/ntplib/
diesel: https://pypi.python.org/pypi/diesel/
nmap: https://pypi.python.org/pypi/python-nmap
scapy: https://pypi.python.org/pypi/scapy
netifaces: https://pypi.python.org/pypi/netifaces/

[3]
Preface

netaddr: https://pypi.python.org/pypi/netaddr
pyopenssl: https://pypi.python.org/pypi/pyOpenSSL
pygeocoder: https://pypi.python.org/pypi/pygocoder
pyyaml: https://pypi.python.org/pypi/PyYAML
requests: https://pypi.python.org/pypi/requests
feedparser: https://pypi.python.org/pypi/feedparser
paramiko: https://pypi.python.org/pypi/paramiko/
fabric: https://pypi.python.org/pypi/Fabric
supervisor: https://pypi.python.org/pypi/supervisor
xmlrpclib: https://pypi.python.org/pypi/xmlrpclib
SOAPpy: https://pypi.python.org/pypi/SOAPpy
bottlenose: https://pypi.python.org/pypi/bottlenose
construct: https://pypi.python.org/pypi/construct/
libpcap: https://pypi.python.org/pypi/pcap
setup tools: https://pypi.python.org/pypi/setuptools
exabgp: https://pypi.python.org/pypi/exabgp
traixroute: https://pypi.python.org/pypi/traixroute
dronekit: https://pypi.python.org/pypi/dronekit
dronekit-sitl: https://pypi.python.org/simple/dronekit-sitl/
ryu: https://pypi.python.org/pypi/ryu
Flask: https://pypi.python.org/pypi/Flask
smtpd: https://pypi.python.org/pypi/secure-smtpd
twisted: https://pypi.python.org/pypi/Twisted
tornado: https://pypi.python.org/pypi/tornado
dnspython: https://pypi.python.org/pypi/dnspython
ldap3: https://pypi.python.org/pypi/ldap3
Eve: https://pypi.python.org/pypi/Eve
RequestsThrottler: https://pypi.python.org/pypi/RequestsThrottler
PyNSXv: https://pypi.python.org/pypi/pynsxv
vmware-nsx: https://pypi.python.org/pypi/vmware-nsx

[4]
Preface

Other software needed to run some recipes are as follows:

postfix: http://www.postfix.org/
openssh server: http://www.openssh.com/
mysql server: http://downloads.mysql.com/
apache2: http://httpd.apache.org/download.cgi/
virtualenv: https://virtualenv.pypa.io/
filezilla: https://filezilla-project.org/
vsftpd: https://security.appspot.com/vsftpd.html
telnetd: telnetd.sourceforge.net/
curl: https://curl.haxx.se/
NS-3: https://www.nsnam.org/ns-3-26/download/
Mininet: mininet.org/
Ansible: https://www.ansible.com/
Git: https://git-scm.com/
aptitude: https://www.openhub.net/p/aptitude
Node-ws / wscat: https://www.npmjs.com/package/wscat
MaxiNet: https://github.com/MaxiNet/MaxiNet/
Mininet-WiFi: https://github.com/intrig-unicamp/mininet-wifi
ContainerNet: https://github.com/containernet/containernet.git
Ant: ant.apache.org/
Maven: https://maven.apache.org/
OpenDaylight: https://www.opendaylight.org/downloads
ONOS: https://wiki.onosproject.org/display/ONOS/Downloads
Floodlight: http://www.projectfloodlight.org/download/
POX: http://github.com/noxrepo/pox
libnl-3-dev: https://packages.debian.org/sid/libnl-3-dev
libnl-genl-3-dev: https://packages.debian.org/sid/libnl-genl-3-dev
libnl-route-3-dev: https://packages.debian.org/sid/libnl-route-3-dev
pkg-config: https://www.freedesktop.org/wiki/Software/pkg-config/
python-tz: pytz.sourceforge.net/
libpcap-dev: https://packages.debian.org/libpcap-dev
libcap2-dev: https://packages.debian.org/jessie/libcap2-dev

[5]
Another Random Scribd Document
with Unrelated Content
consider some of the antecedents of our modern approach to the
investigation of nature.
In the first place, there can be no living science unless there is a
widespread instinctive conviction in the existence of an Order of
Things, and, in particular, of an Order of Nature. I have used the
word instinctive advisedly. It does not matter what men say in words,
so long as their activities are controlled by settled instincts. The
words may ultimately destroy the instincts. But until this has
occurred, words do not count. This remark is important in respect to
the history of scientific thought. For we shall find that since the time
of Hume, the fashionable scientific philosophy has been such as to
deny the rationality of science. This conclusion lies upon the surface
of Hume’s philosophy. Take, for example, the following passage from
Section IV of his Inquiry Concerning Human Understanding:

“In a word, then, every effect is a distinct event from its cause. It
could not, therefore, be discovered in the cause; and the first
invention or conception of it, à priori, must be entirely arbitrary.”

If the cause in itself discloses no information as to the effect, so that

the first invention of it must be entirely arbitrary, it follows at once
that science is impossible, except in the sense of establishing
entirely arbitrary connections which are not warranted by anything
intrinsic to the natures either of causes or effects. Some variant of
Hume’s philosophy has generally prevailed among men of science.
But scientific faith has risen to the occasion, and has tacitly removed
the philosophic mountain.
In view of this strange contradiction in scientific thought, it is of the
first importance to consider the antecedents of a faith which is
impervious to the demand for a consistent rationality. We have
therefore to trace the rise of the instinctive faith that there is an Order
of Nature which can be traced in every detailed occurrence.
Of course we all share in this faith, and we therefore believe that
the reason for the faith is our apprehension of its truth. But the
formation of a general idea—such as the idea of the Order of Nature
—, and the grasp of its importance, and the observation of its
exemplification in a variety of occasions are by no means the
necessary consequences of the truth of the idea in question. Familiar
things happen, and mankind does not bother about them. It requires
a very unusual mind to undertake the analysis of the obvious.
Accordingly I wish to consider the stages in which this analysis
became explicit, and finally became unalterably impressed upon the
educated minds of Western Europe.
Obviously, the main recurrences of life are too insistent to escape
the notice of the least rational of humans; and even before the dawn
of rationality, they have impressed themselves upon the instincts of
animals. It is unnecessary to labour the point, that in broad outline
certain general states of nature recur, and that our very natures have
adapted themselves to such repetitions.
But there is a complementary fact which is equally true and
equally obvious:—nothing ever really recurs in exact detail. No two
days are identical, no two winters. What has gone, has gone forever.
Accordingly the practical philosophy of mankind has been to expect
the broad recurrences, and to accept the details as emanating from
the inscrutable womb of things, beyond the ken of rationality. Men
expected the sun to rise, but the wind bloweth where it listeth.
Certainly from the classical Greek civilisation onwards there have
been men, and indeed groups of men, who have placed themselves
beyond this acceptance of an ultimate irrationality. Such men have
endeavoured to explain all phenomena as the outcome of an order
of things which extends to every detail. Geniuses such as Aristotle,
or Archimedes, or Roger Bacon, must have been endowed with the
full scientific mentality, which instinctively holds that all things great
and small are conceivable as exemplifications of general principles
which reign throughout the natural order.
But until the close of the Middle Ages the general educated public
did not feel that intimate conviction, and that detailed interest, in
such an idea, so as to lead to an unceasing supply of men, with
ability and opportunity adequate to maintain a coordinated search for
the discovery of these hypothetical principles. Either people were
doubtful about the existence of such principles, or were doubtful
about any success in finding them, or took no interest in thinking
about them, or were oblivious to their practical importance when
found. For whatever reason, search was languid, if we have regard
to the opportunities of a high civilisation and the length of time
concerned. Why did the pace suddenly quicken in the sixteenth and
seventeenth centuries? At the close of the Middle Ages a new
mentality discloses itself. Invention stimulated thought, thought
quickened physical speculation, Greek manuscripts disclosed what
the ancients had discovered. Finally although in the year 1500
Europe knew less than Archimedes who died in the year 212 B. C.,
yet in the year 1700, Newton’s Principia had been written and the
world was well started on the modern epoch.
There have been great civilisations in which the peculiar balance
of mind required for science has only fitfully appeared and has
produced the feeblest result. For example, the more we know of
Chinese art, of Chinese literature, and of the Chinese philosophy of
life, the more we admire the heights to which that civilization
attained. For thousands of years, there have been in China acute
and learned men patiently devoting their lives to study. Having
regard to the span of time, and to the population concerned, China
forms the largest volume of civilisation which the world has seen.
There is no reason to doubt the intrinsic capacity of individual
Chinamen for the pursuit of science. And yet Chinese science is
practically negligible. There is no reason to believe that China if left
to itself would have ever produced any progress in science. The
same may be said of India. Furthermore, if the Persians had
enslaved the Greeks, there is no definite ground for belief that
science would have flourished in Europe. The Romans showed no
particular originality in that line. Even as it was, the Greeks, though
they founded the movement, did not sustain it with the concentrated
interest which modern Europe has shown. I am not alluding to the
last few generations of the European peoples on both sides of the
ocean; I mean the smaller Europe of the Reformation period,
distracted as it was with wars and religious disputes. Consider the
world of the eastern Mediterranean, from Sicily to western Asia,
during the period of about 1400 years from the death of Archimedes
[in 212 B. C.] to the irruption of the Tartars. There were wars and
revolutions and large changes of religion: but nothing much worse
than the wars of the sixteenth and seventeenth centuries throughout
Europe. There was a great and wealthy civilisation, Pagan, Christian,
Mahometan. In that period a great deal was added to science. But
on the whole the progress was slow and wavering; and, except in
mathematics, the men of the Renaissance practically started from
the position which Archimedes had reached. There had been some
progress in medicine and some progress in astronomy. But the total
advance was very little compared to the marvellous success of the
seventeenth century. For example, compare the progress of
scientific knowledge from the year 1560, just before the births of
Galileo and of Kepler, up to the year 1700, when Newton was in the
height of his fame, with the progress in the ancient period, already
mentioned, exactly ten times as long.
Nevertheless, Greece was the mother of Europe; and it is to
Greece that we must look in order to find the origin of our modern
ideas. We all know that on the eastern shores of the Mediterranean
there was a very flourishing school of Ionian philosophers, deeply
interested in theories concerning nature. Their ideas have been
transmitted to us, enriched by the genius of Plato and Aristotle. But,
with the exception of Aristotle, and it is a large exception, this school
of thought had not attained to the complete scientific mentality. In
some ways, it was better. The Greek genius was philosophical, lucid
and logical. The men of this group were primarily asking
philosophical questions. What is the substratum of nature? Is it fire,
or earth, or water, or some combination of any two, or of all three?
Or is it a mere flux, not reducible to some static material?
Mathematics interested them mightily. They invented its generality,
analysed its premises, and made notable discoveries of theorems by
a rigid adherence to deductive reasoning. Their minds were infected
with an eager generality. They demanded clear, bold ideas, and strict
reasoning from them. All this was excellent; it was genius; it was
ideal preparatory work. But it was not science as we understand it.
The patience of minute observation was not nearly so prominent.
Their genius was not so apt for the state of imaginative muddled
suspense which precedes successful inductive generalisation. They
were lucid thinkers and bold reasoners.
Of course there were exceptions, and at the very top: for example,
Aristotle and Archimedes. Also for patient observation, there were
the astronomers. There was a mathematical lucidity about the stars,
and a fascination about the small numerable band of run-a-way
planets.
Every philosophy is tinged with the colouring of some secret
imaginative background, which never emerges explicitly into its
trains of reasoning. The Greek view of nature, at least that
cosmology transmitted from them to later ages, was essentially
dramatic. It is not necessarily wrong for this reason: but it was
overwhelmingly dramatic. It thus conceived nature as articulated in
the way of a work of dramatic art, for the exemplification of general
ideas converging to an end. Nature was differentiated so as to
provide its proper end for each thing. There was the centre of the
universe as the end of motion for those things which are heavy, and
the celestial spheres as the end of motion for those things whose
natures lead them upwards. The celestial spheres were for things
which are impassible and ingenerable, the lower regions for things
impassible and generable. Nature was a drama in which each thing
played its part.
I do not say that this is a view to which Aristotle would have
subscribed without severe reservations, in fact without the sort of
reservations which we ourselves would make. But it was the view
which subsequent Greek thought extracted from Aristotle and
passed on to the Middle Ages. The effect of such an imaginative
setting for nature was to damp down the historical spirit. For it was
the end which seemed illuminating, so why bother about the
beginning? The Reformation and the scientific movement were two
aspects of the historical revolt which was the dominant intellectual
movement of the later Renaissance. The appeal to the origins of
Christianity, and Francis Bacon’s appeal to efficient causes as
against final causes, were two sides of one movement of thought.
Also for this reason Galileo and his adversaries were at hopeless
cross purposes, as can be seen from his Dialogues on the Two
Systems of the World.
Galileo keeps harping on how things happen, whereas his
adversaries had a complete theory as to why things happen.
Unfortunately the two theories did not bring out the same results.
Galileo insists upon ‘irreducible and stubborn facts,’ and Simplicius,
his opponent, brings forward reasons, completely satisfactory, at
least to himself. It is a great mistake to conceive this historical revolt
as an appeal to reason. On the contrary, it was through and through
an anti-intellectualist movement. It was the return to the
contemplation of brute fact; and it was based on a recoil from the
inflexible rationality of medieval thought. In making this statement I
am merely summarising what at the time the adherents of the old
régime themselves asserted. For example, in the fourth book of
Father Paul Sarpi’s History of the Council of Trent, you will find that
in the year 1551 the Papal Legates who presided over the Council
ordered: ‘That the Divines ought to confirm their opinions with the
holy Scripture, Traditions of the Apostles, sacred and approved
Councils, and by the Constitutions and Authorities of the holy
Fathers; that they ought to use brevity, and avoid superfluous and
unprofitable questions, and perverse contentions.... This order did
not please the Italian Divines; who said it was a novity, and a
condemning of School-Divinity, which, in all difficulties, useth reason,
and because it was not lawful [i.e., by this decree] to treat as St.
Thomas [Aquinas], St. Bonaventure, and other famous men did.’
It is impossible not to feel sympathy with these Italian divines,
maintaining the lost cause of unbridled rationalism. They were
deserted on all hands. The Protestants were in full revolt against
them. The Papacy failed to support them, and the Bishops of the
Council could not even understand them. For a few sentences below
the foregoing quotation, we read: ‘Though many complained here-of
[i.e., of the Decree], yet it prevailed but little, because generally the
Fathers [i.e., the Bishops] desired to hear men speak with intelligible
terms, not abstrusely, as in the matter of Justification, and others
already handled.’
Poor belated medievalists! When they used reason they were not
even intelligible to the ruling powers of their epoch. It will take
centuries before stubborn facts are reducible by reason, and
meanwhile the pendulum swings slowly and heavily to the extreme
of the historical method.
Forty-three years after the Italian divines had written this
memorial, Richard Hooker in his famous Laws of Ecclesiastical Polity
makes exactly the same complaint of his Puritan adversaries.[1]
Hooker’s balanced thought—from which the appellation ‘The
Judicious Hooker’ is derived—, and his diffuse style, which is the
vehicle of such thought, make his writings singularly unfit for the
process of summarising by a short, pointed quotation. But, in the
section referred to, he reproaches his opponents with Their
Disparagement of Reason; and in support of his own position
definitely refers to ‘The greatest amongst the school-divines,’ by
which designation I presume that he refers to St. Thomas Aquinas.
1. Cf. Book III, Section VIII.
Hooker’s Ecclesiastical Polity was published just before Sarpi’s
Council of Trent. Accordingly there was complete independence
between the two works. But both the Italian divines of 1551, and
Hooker at the end of that century testify to the anti-rationalist trend of
thought at that epoch, and in this respect contrast their own age with
the epoch of scholasticism.
This reaction was undoubtedly a very necessary corrective to the
unguarded rationalism of the Middle Ages. But reactions run to
extremes. Accordingly, although one outcome of this reaction was
the birth of modern science, yet we must remember that science
thereby inherited the bias of thought to which it owes its origin.
The effect of Greek dramatic literature was many-sided so far as
concerns the various ways in which it indirectly affected medieval
thought. The pilgrim fathers of the scientific imagination as it exists
today, are the great tragedians of ancient Athens, Aeschylus,
Sophocles, Euripides. Their vision of fate, remorseless and
indifferent, urging a tragic incident to its inevitable issue, is the vision
possessed by science. Fate in Greek Tragedy becomes the order of
nature in modern thought. The absorbing interest in the particular
heroic incidents, as an example and a verification of the workings of
fate, reappears in our epoch as concentration of interest on the
crucial experiments. It was my good fortune to be present at the
meeting of the Royal Society in London when the Astronomer Royal
for England announced that the photographic plates of the famous
eclipse, as measured by his colleagues in Greenwich Observatory,
had verified the prediction of Einstein that rays of light are bent as
they pass in the neighbourhood of the sun. The whole atmosphere of
tense interest was exactly that of the Greek drama: we were the
chorus commenting on the decree of destiny as disclosed in the
development of a supreme incident. There was dramatic quality in
the very staging:—the traditional ceremonial, and in the background
the picture of Newton to remind us that the greatest of scientific
generalisations was now, after more than two centuries, to receive its
first modification. Nor was the personal interest wanting: a great
adventure in thought had at length come safe to shore.
Let me here remind you that the essence of dramatic tragedy is
not unhappiness. It resides in the solemnity of the remorseless
working of things. This inevitableness of destiny can only be
illustrated in terms of human life by incidents which in fact involve
unhappiness. For it is only by them that the futility of escape can be
made evident in the drama. This remorseless inevitableness is what
pervades scientific thought. The laws of physics are the decrees of
fate.
The conception of the moral order in the Greek plays was certainly
not a discovery of the dramatists. It must have passed into the
literary tradition from the general serious opinion of the times. But in
finding this magnificent expression, it thereby deepened the stream
of thought from which it arose. The spectacle of a moral order was
impressed upon the imagination of classical civilisation.
The time came when that great society decayed, and Europe
passed into the Middle Ages. The direct influence of Greek literature
vanished. But the concept of the moral order and of the order of
nature had enshrined itself in the Stoic philosophy. For example,
Lecky in his History of European Morals tells us ‘Seneca maintains
that the Divinity has determined all things by an inexorable law of
destiny, which He has decreed, but which He Himself obeys.’ But the
most effective way in which the Stoics influenced the mentality of the
Middle Ages was by the diffused sense of order which arose from
Roman law. Again to quote Lecky, ‘The Roman legislation was in a
two-fold manner the child of philosophy. It was in the first place
formed upon the philosophical model, for, instead of being a mere
empirical system adjusted to the existing requirements of society, it
laid down abstract principles of right to which it endeavoured to
conform; and, in the next place, these principles were borrowed
directly from Stoicism.’ In spite of the actual anarchy throughout
large regions in Europe after the collapse of the Empire, the sense of
legal order always haunted the racial memories of the Imperial
populations. Also the Western Church was always there as a living
embodiment of the traditions of Imperial rule.
It is important to notice that this legal impress upon medieval
civilisation was not in the form of a few wise precepts which should
permeate conduct. It was the conception of a definite articulated
system which defines the legality of the detailed structure of social
organism, and of the detailed way in which it should function. There
was nothing vague. It was not a question of admirable maxims, but
of definite procedure to put things right and to keep them there. The
Middle Ages formed one long training of the intellect of Western
Europe in the sense of order. There may have been some deficiency
in respect to practice. But the idea never for a moment lost its grip. It
was preëminently an epoch of orderly thought, rationalist through
and through. The very anarchy quickened the sense for coherent
system; just as the modern anarchy of Europe has stimulated the
intellectual vision of a League of Nations.
But for science something more is wanted than a general sense of
the order in things. It needs but a sentence to point out how the habit
of definite exact thought was implanted in the European mind by the
long dominance of scholastic logic and scholastic divinity. The habit
remained after the philosophy had been repudiated, the priceless
habit of looking for an exact point and of sticking to it when found.
Galileo owes more to Aristotle than appears on the surface of his
Dialogues: he owes to him his clear head and his analytic mind.
I do not think, however, that I have even yet brought out the
greatest contribution of medievalism to the formation of the scientific
movement. I mean the inexpugnable belief that every detailed
occurrence can be correlated with its antecedents in a perfectly
definite manner, exemplifying general principles. Without this belief
the incredible labours of scientists would be without hope. It is this
instinctive conviction, vividly poised before the imagination, which is
the motive power of research:—that there is a secret, a secret which
can be unveiled. How has this conviction been so vividly implanted
on the European mind?
When we compare this tone of thought in Europe with the attitude
of other civilisations when left to themselves, there seems but one
source for its origin. It must come from the medieval insistence on
the rationality of God, conceived as with the personal energy of
Jehovah and with the rationality of a Greek philosopher. Every detail
was supervised and ordered: the search into nature could only result
in the vindication of the faith in rationality. Remember that I am not
talking of the explicit beliefs of a few individuals. What I mean is the
impress on the European mind arising from the unquestioned faith of
centuries. By this I mean the instinctive tone of thought and not a
mere creed of words.
In Asia, the conceptions of God were of a being who was either
too arbitrary or too impersonal for such ideas to have much effect on
instinctive habits of mind. Any definite occurrence might be due to
the fiat of an irrational despot, or might issue from some impersonal,
inscrutable origin of things. There was not the same confidence as in
the intelligible rationality of a personal being. I am not arguing that
the European trust in the scrutability of nature was logically justified
even by its own theology. My only point is to understand how it
arose. My explanation is that the faith in the possibility of science,
generated antecedently to the development of modern scientific
theory, is an unconscious derivative from medieval theology.
But science is not merely the outcome of instinctive faith. It also
requires an active interest in the simple occurrences of life for their
own sake.
This qualification ‘for their own sake’ is important. The first phase
of the Middle Ages was an age of symbolism. It was an age of vast
ideas, and of primitive technique. There was little to be done with
nature, except to coin a hard living from it. But there were realms of
thought to be explored, realms of philosophy and realms of theology.
Primitive art could symbolise those ideas which filled all thoughtful
minds. The first phase of medieval art has a haunting charm beyond
compare: its own intrinsic quality is enhanced by the fact that its
message, which stretched beyond art’s own self-justification of
aesthetic achievement, was the symbolism of things lying behind
nature itself. In this symbolic phase, medieval art energised in nature
as its medium, but pointed to another world.
In order to understand the contrast between these early Middle
Ages and the atmosphere required by the scientific mentality, we
should compare the sixth century in Italy with the sixteenth century.
In both centuries the Italian genius was laying the foundations of a
new epoch. The history of the three centuries preceding the earlier
period, despite the promise for the future introduced by the rise of
Christianity, is overwhelmingly infected by the sense of the decline of
civilisation. In each generation something has been lost. As we read
the records, we are haunted by the shadow of the coming barbarism.
There are great men, with fine achievements in action or in thought.
But their total effect is merely for some short time to arrest the
general decline. In the sixth century we are, so far as Italy is
concerned, at the lowest point of the curve. But in that century every
action is laying the foundation for the tremendous rise of the new
European civilisation. In the background the Byzantine Empire,
under Justinian, in three ways determined the character of the early
Middle Ages in Western Europe. In the first place, its armies, under
Belisarius and Narses, cleared Italy from the Gothic domination. In
this way, the stage was freed for the exercise of the old Italian genius
for creating organisations which shall be protective of ideals of
cultural activity. It is impossible not to sympathise with the Goths: yet
there can be no doubt but that a thousand years of the Papacy were
infinitely more valuable for Europe than any effects derivable from a
well-established Gothic kingdom of Italy.
In the second place, the codification of the Roman law established
the ideal of legality which dominated the sociological thought of
Europe in the succeeding centuries. Law is both an engine for
government, and a condition restraining government. The canon law
of the Church, and the civil law of the State, owe to Justinian’s
lawyers their influence on the development of Europe. They
established in the Western mind the ideal that an authority should be
at once lawful, and law-enforcing, and should in itself exhibit a
rationally adjusted system of organisation. The sixth century in Italy
gave the initial exhibition of the way in which the impress of these
ideas was fostered by contact with the Byzantine Empire.
Thirdly, in the non-political spheres of art and learning
Constantinople exhibited a standard of realised achievement which,
partly by the impulse to direct imitation, and partly by the indirect
inspiration arising from the mere knowledge that such things existed,
acted as a perpetual spur to Western culture. The wisdom of the
Byzantines, as it stood in the imagination of the first phase of
medieval mentality, and the wisdom of the Egyptians as it stood in
the imagination of the early Greeks, played analogous rôles.
Probably the actual knowledge of these respective wisdoms was, in
either case, about as much as was good for the recipients. They
knew enough to know the sort of standards which are attainable, and
not enough to be fettered by static and traditional ways of thought.
Accordingly, in both cases men went ahead on their own and did
better. No account of the rise of the European scientific mentality can
omit some notice of this influence of the Byzantine civilisation in the
background. In the sixth century there is a crisis in the history of the
relations between the Byzantines and the West; and this crisis is to
be contrasted with the influence of Greek literature on European
thought in the fifteenth and sixteenth centuries. The two outstanding
men, who in the Italy of the sixth century laid the foundations of the
future, were St. Benedict and Gregory the Great. By reference to
them, we can at once see how absolutely in ruins was the approach
to the scientific mentality which had been attained by the Greeks.
We are at the zero point of scientific temperature. But the life-work of
Gregory and of Benedict contributed elements to the reconstruction
of Europe which secured that this reconstruction, when it arrived,
should include a more effective scientific mentality than that of the
ancient world. The Greeks were over-theoretical. For them science
was an offshoot of philosophy. Gregory and Benedict were practical
men, with an eye for the importance of ordinary things; and they
combined this practical temperament with their religious and cultural
activities. In particular, we owe it to St. Benedict that the monasteries
were the homes of practical agriculturalists, as well as of saints and
of artists and of men of learning. The alliance of science with
technology, by which learning is kept in contact with irreducible and
stubborn facts, owes much to the practical bent of the early
Benedictines. Modern science derives from Rome as well as from
Greece, and this Roman strain explains its gain in an energy of
thought kept closely in contact with the world of facts.
But the influence of this contact between the monasteries and the
facts of nature showed itself first in art. The rise of Naturalism in the
later Middle Ages was the entry into the European mind of the final
ingredient necessary for the rise of science. It was the rise of interest
in natural objects, and in natural occurrences, for their own sakes.
The natural foliage of a district was sculptured in out-of-the-way
spots of the later buildings, merely as exhibiting delight in those
familiar objects. The whole atmosphere of every art exhibited a direct
joy in the apprehension of the things which lie around us. The
craftsmen who executed the late medieval decorative sculpture,
Giotto, Chaucer, Wordsworth, Walt Whitman, and, at the present
day, the New England poet Robert Frost, are all akin to each other in
this respect. The simple immediate facts are the topics of interest,
and these reappear in the thought of science as the ‘irreducible
stubborn facts.’
The mind of Europe was now prepared for its new venture of
thought. It is unnecessary to tell in detail the various incidents which
marked the rise of science: the growth of wealth and leisure; the
expansion of universities; the invention of printing; the taking of
Constantinople; Copernicus; Vasco da Gama; Columbus; the
telescope. The soil, the climate, the seeds, were there, and the
forest grew. Science has never shaken off the impress of its origin in
the historical revolt of the later Renaissance. It has remained
predominantly an anti-rationalistic movement, based upon a naïve
faith. What reasoning it has wanted, has been borrowed from
mathematics which is a surviving relic of Greek rationalism, following
the deductive method. Science repudiates philosophy. In other
words, it has never cared to justify its faith or to explain its meanings;
and has remained blandly indifferent to its refutation by Hume.
Of course the historical revolt was fully justified. It was wanted. It
was more than wanted: it was an absolute necessity for healthy
progress. The world required centuries of contemplation of
irreducible and stubborn facts. It is difficult for men to do more than
one thing at a time, and that was the sort of thing they had to do after
the rationalistic orgy of the Middle Ages. It was a very sensible
reaction; but it was not a protest on behalf of reason.
There is, however, a Nemesis which waits upon those who
deliberately avoid avenues of knowledge. Oliver Cromwell’s cry
echoes down the ages, ‘My brethren, by the bowels of Christ I
beseech you, bethink you that you may be mistaken.’
The progress of science has now reached a turning point. The
stable foundations of physics have broken up: also for the first time
physiology is asserting itself as an effective body of knowledge, as
distinct from a scrap-heap. The old foundations of scientific thought
are becoming unintelligible. Time, space, matter, material, ether,
electricity, mechanism, organism, configuration, structure, pattern,
function, all require reinterpretation. What is the sense of talking
about a mechanical explanation when you do not know what you
mean by mechanics?
The truth is that science started its modern career by taking over
ideas derived from the weakest side of the philosophies of Aristotle’s
successors. In some respects it was a happy choice. It enabled the
knowledge of the seventeenth century to be formularised so far as
physics and chemistry were concerned, with a completeness which
has lasted to the present time. But the progress of biology and
psychology has probably been checked by the uncritical assumption
of half-truths. If science is not to degenerate into a medley of ad hoc
hypotheses, it must become philosophical and must enter upon a
thorough criticism of its own foundations.
In the succeeding lectures of this course, I shall trace the
successes and the failures of the particular conceptions of
cosmology with which the European intellect has clothed itself in the
last three centuries. General climates of opinion persist for periods of
about two to three generations, that is to say, for periods of sixty to a
hundred years. There are also shorter waves of thought, which play
on the surface of the tidal movement. We shall find, therefore,
transformations in the European outlook, slowly modifying the
successive centuries. There persists, however, throughout the whole
period the fixed scientific cosmology which presupposes the ultimate
fact of an irreducible brute matter, or material, spread throughout
space in a flux of configurations. In itself such a material is
senseless, valueless, purposeless. It just does what it does do,
following a fixed routine imposed by external relations which do not
spring from the nature of its being. It is this assumption that I call
‘scientific materialism.’ Also it is an assumption which I shall
challenge as being entirely unsuited to the scientific situation at
which we have now arrived. It is not wrong, if properly construed. If
we confine ourselves to certain types of facts, abstracted from the
complete circumstances in which they occur, the materialistic
assumption expresses these facts to perfection. But when we pass
beyond the abstraction, either by more subtle employment of our
senses, or by the request for meanings and for coherence of
thoughts, the scheme breaks down at once. The narrow efficiency of
the scheme was the very cause of its supreme methodological
success. For it directed attention to just those groups of facts which,
in the state of knowledge then existing, required investigation.
The success of the scheme has adversely affected the various
currents of European thought. The historical revolt was anti-
rationalistic, because the rationalism of the scholastics required a
sharp correction by contact with brute fact. But the revival of
philosophy in the hands of Descartes and his successors was
entirely coloured in its development by the acceptance of the
scientific cosmology at its face value. The success of their ultimate
ideas confirmed scientists in their refusal to modify them as the
result of an enquiry into their rationality. Every philosophy was bound
in some way or other to swallow them whole. Also the example of
science affected other regions of thought. The historical revolt has
thus been exaggerated into the exclusion of philosophy from its
proper rôle of harmonising the various abstractions of
methodological thought. Thought is abstract; and the intolerant use
of abstractions is the major vice of the intellect. This vice is not
wholly corrected by the recurrence to concrete experience. For after
all, you need only attend to those aspects of your concrete
experience which lie within some limited scheme. There are two
methods for the purification of ideas. One of them is dispassionate
observation by means of the bodily senses. But observation is
selection. Accordingly, it is difficult to transcend a scheme of
abstraction whose success is sufficiently wide. The other method is
by comparing the various schemes of abstraction which are well
founded in our various types of experience. This comparison takes
the form of satisfying the demands of the Italian scholastic divines
whom Paul Sarpi mentioned. They asked that reason should be
used. Faith in reason is the trust that the ultimate natures of things
lie together in a harmony which excludes mere arbitrariness. It is the
faith that at the base of things we shall not find mere arbitrary
mystery. The faith in the order of nature which has made possible
the growth of science is a particular example of a deeper faith. This
faith cannot be justified by any inductive generalisation. It springs
from direct inspection of the nature of things as disclosed in our own
immediate present experience. There is no parting from your own
shadow. To experience this faith is to know that in being ourselves
we are more than ourselves: to know that our experience, dim and
fragmentary as it is, yet sounds the utmost depths of reality: to know
that detached details merely in order to be themselves demand that
they should find themselves in a system of things: to know that this
system includes the harmony of logical rationality, and the harmony
of aesthetic achievement: to know that, while the harmony of logic
lies upon the universe as an iron necessity, the aesthetic harmony
stands before it as a living ideal moulding the general flux in its
broken progress towards finer, subtler issues.
 CHAPTER II

MATHEMATICS AS AN ELEMENT IN
THE HISTORY OF THOUGHT

The science of Pure Mathematics, in its modern developments,

may claim to be the most original creation of the human spirit.
Another claimant for this position is music. But we will put aside all
rivals, and consider the ground on which such a claim can be made
for mathematics. The originality of mathematics consists in the fact
that in mathematical science connections between things are
exhibited which, apart from the agency of human reason, are
extremely unobvious. Thus the ideas, now in the minds of
contemporary mathematicians, lie very remote from any notions
which can be immediately derived by perception through the senses;
unless indeed it be perception stimulated and guided by antecedent
mathematical knowledge. This is the thesis which I proceed to
exemplify.
Suppose we project our imaginations backwards through many
thousands of years, and endeavour to realise the simple-
mindedness of even the greatest intellects in those early societies.
Abstract ideas which to us are immediately obvious must have been,
for them, matters only of the most dim apprehension. For example
take the question of number. We think of the number ‘five’ as
applying to appropriate groups of any entities whatsoever—to five
fishes, five children, five apples, five days. Thus in considering the
relations of the number ‘five’ to the number ‘three,’ we are thinking of
two groups of things, one with five members and the other with three
members. But we are entirely abstracting from any consideration of
any particular entities, or even of any particular sorts of entities,
which go to make up the membership of either of the two groups. We
are merely thinking of those relationships between those two groups
which are entirely independent of the individual essences of any of
the members of either group. This is a very remarkable feat of
abstraction; and it must have taken ages for the human race to rise
to it. During a long period, groups of fishes will have been compared
to each other in respect to their multiplicity, and groups of days to
each other. But the first man who noticed the analogy between a
group of seven fishes and a group of seven days made a notable
advance in the history of thought. He was the first man who
entertained a concept belonging to the science of pure mathematics.
At that moment it must have been impossible for him to divine the
complexity and subtlety of these abstract mathematical ideas which
were waiting for discovery. Nor could he have guessed that these
notions would exert a widespread fascination in each succeeding
generation. There is an erroneous literary tradition which represents
the love of mathematics as a monomania confined to a few
eccentrics in each generation. But be this as it may, it would have
been impossible to anticipate the pleasure derivable from a type of
abstract thinking which had no counterpart in the then-existing
society. Thirdly, the tremendous future effect of mathematical
knowledge on the lives of men, on their daily avocations, on their
habitual thoughts, on the organization of society, must have been
even more completely shrouded from the foresight of those early
thinkers. Even now there is a very wavering grasp of the true
position of mathematics as an element in the history of thought. I will
not go so far as to say that to construct a history of thought without
profound study of the mathematical ideas of successive epochs is
like omitting Hamlet from the play which is named after him. That
would be claiming too much. But it is certainly analogous to cutting
out the part of Ophelia. This simile is singularly exact. For Ophelia is
quite essential to the play, she is very charming,—and a little mad.
Let us grant that the pursuit of mathematics is a divine madness of
the human spirit, a refuge from the goading urgency of contingent
happenings.
 When we think of mathematics, we have in our mind a science
devoted to the exploration of number, quantity, geometry, and in
modern times also including investigation into yet more abstract
concepts of order, and into analogous types of purely logical
relations. The point of mathematics is that in it we have always got
rid of the particular instance, and even of any particular sorts of
entities. So that for example, no mathematical truths apply merely to
fish, or merely to stones, or merely to colours. So long as you are
dealing with pure mathematics, you are in the realm of complete and
absolute abstraction. All you assert is, that reason insists on the
admission that, if any entities whatever have any relations which
satisfy such-and-such purely abstract conditions, then they must
have other relations which satisfy other purely abstract conditions.
Mathematics is thought moving in the sphere of complete
abstraction from any particular instance of what it is talking about. So
far is this view of mathematics from being obvious, that we can
easily assure ourselves that it is not, even now, generally
understood. For example, it is habitually thought that the certainty of
mathematics is a reason for the certainty of our geometrical
knowledge of the space of the physical universe. This is a delusion
which has vitiated much philosophy in the past, and some
philosophy in the present. This question of geometry is a test case of
some urgency. There are certain alternative sets of purely abstract
conditions possible for the relationships of groups of unspecified
entities, which I will call geometrical conditions. I give them this
name because of their general analogy to those conditions, which
we believe to hold respecting the particular geometrical relations of
things observed by us in our direct perception of nature. So far as
our observations are concerned, we are not quite accurate enough
to be certain of the exact conditions regulating the things we come
across in nature. But we can by a slight stretch of hypothesis identify
these observed conditions with some one set of the purely abstract
geometrical conditions. In doing so, we make a particular
determination of the group of unspecified entities which are the
relata in the abstract science. In the pure mathematics of
geometrical relationships, we say that, if any group of entities enjoy
any relationships among its members satisfying this set of abstract
geometrical conditions, then such-and-such additional abstract
conditions must also hold for such relationships. But when we come
to physical space, we say that some definitely observed group of
physical entities enjoys some definitely observed relationships
among its members which do satisfy this above-mentioned set of
abstract geometrical conditions. We thence conclude that the
additional relationships which we concluded to hold in any such
case, must therefore hold in this particular case.
The certainty of mathematics depends upon its complete abstract
generality. But we can have no à priori certainty that we are right in
believing that the observed entities in the concrete universe form a
particular instance of what falls under our general reasoning. To take
another example from arithmetic. It is a general abstract truth of pure
mathematics that any group of forty entities can be subdivided into
two groups of twenty entities. We are therefore justified in concluding
that a particular group of apples which we believe to contain forty
members can be subdivided into two groups of apples of which each
contains twenty members. But there always remains the possibility
that we have miscounted the big group; so that, when we come in
practice to subdivide it, we shall find that one of the two heaps has
an apple too few or an apple too many.
Accordingly, in criticising an argument based upon the application
of mathematics to particular matters of fact, there are always three
processes to be kept perfectly distinct in our minds. We must first
scan the purely mathematical reasoning to make sure that there are
no mere slips in it—no casual illogicalities due to mental failure. Any
mathematician knows from bitter experience that, in first elaborating
a train of reasoning, it is very easy to commit a slight error which yet
makes all the difference. But when a piece of mathematics has been
revised, and has been before the expert world for some time, the
chance of a casual error is almost negligible. The next process is to
make quite certain of all the abstract conditions which have been
presupposed to hold. This is the determination of the abstract
premises from which the mathematical reasoning proceeds. This is a
matter of considerable difficulty. In the past quite remarkable
oversights have been made, and have been accepted by
generations of the greatest mathematicians. The chief danger is that
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com