System Programming with

Python in Linux
 |System Programming
System Programming with Python in Linux

| The practice of writing scripts or

programs that provide services to
the computer hardware or the
software applications that run on it.
 |Python and Linux
System Programming with Python in Linux

| Python, with its simplicity (and ease

of learning) and vast range of
libraries, is a popular choice for
system programming.

| Linux, with its customizable and

open-source nature, offers a great
environment for Python system
programming.
 |Python
System Programming with Python in Linux

| Python is an interpreted, high-level,

and general-purpose language that
emphasizes code readability.

| It comes with a large standard library

that includes modules for various
tasks such as file I/O system calls,
sockets, and even interfaces for
email and web services.
Python Modules
 |Python Modules
| Modules in Python are simply Python
files with the .py extension, which
implement a set of functions.

| They can also contain executable

Python Modules

statements.
○ Which are intended to initialize the module.
○ Only executed the first time the module
name is encountered in an import statement.
 |Importing Python Modules
| Modules are imported using the
import command.

| The first time a module is loaded into

a running Python script, it is
Python Modules

initialized by executing the code in

the module once.
 |Singleton
| If another module in your project
imports the same module again, it
will not be loaded twice, but only
once - so local variables inside the
module act as a singleton - they are
Python Modules

only initialized once.

 |A Note on the Examples
| Code examples that follow do not
include the shebang line.

| You must add the shebang line as

the first line in your source code
Python Modules

files:

#!/usr/bin/python3
Python Modules
|Python Modules & Importing
 |Checking if a Module is Imported

| Python uses a special variable called

__name__ to determine if a module is
running independently or if it has
been imported.
Python Modules
 |Checking if a Module is Imported

| When a Python file is run directly,

__name__ is set to '__main__'.

| If it is being imported from another

script, __name__ is set to the name
Python Modules

of that script.
| This can be used to ensure certain code only
runs when the script is being executed
directly (often for testing purposes), not when
it’s imported as a module.
Python Modules
|Checking if a Module is Imported
os Module
 |os Module
| The os module in Python provides a
way of using operating system
dependent functionality.

| This functionality includes file and

directory access, process
management, and environment
os Module

variable management.
 |File Operations
| The os module allows us to interact
with the file system, perform file
operations such as creating, reading,
writing, renaming, and deleting files.
os Module
 |open()
| open() is used to open files
○ You do not need to import the os
module to use open().
○ If the file does not exist, it will
create the file.
○ There are multiple modes:
■ r - read
os Module

■ w - write
■ a - append
■ b - binary
 |os.remove()
| os.remove() is used to delete a file.
○ You must to import the os module
to use os.remove().
os Module
os Module
|open() and os.remove()
 |os.rename()
| os.rename() is used to rename a file
or directory.
○ It takes two arguments:
■ the current filename
■ the new filename
os Module
os Module
|os.rename()
 |os.stat()
| os.stat() is used to retrieve
information about the specified file.
○ The information includes file size,
time of modification, and more.
os Module
os Module
|os.stat()
 |Directory Operations
| The os module also allows us to
interact with directories in the file
system.

| We can perform operations such as

creating, listing, changing, and
removing directories.
os Module
 |os.getcwd()
| os.getcwd() retrieves the current
working directory as a string value.
os Module
 |os.chdir()
| os.chdir() is used to change the
current working directory to the path
specified.
os Module
os Module
|os.getcwd() and os.chdir()
 |os.listdir()
| os.listdir() lists all files and
directories in the specified path.
os Module
 |os.mkdir()
| os.mkdir() creates a directory in the
specified path.
os Module
 |os.makedirs()
| os.makedirs() creates all the
intermediate-level directories needed
to create the full path.
os Module
os Module
|os.listdir(), os.mkdir(), os.makedirs()
 |os.rmdir()
| os.rmdir() removes the specified
directory.
os Module
 |os.removedirs()
| os.removedirs() removes directories
recursively up the path, stopping at
the first non-empty parent directory.
os Module
os Module
|os.rmdir() and os.removedirs()
 |os.walk()
| os.walk() generates the filenames in
a directory tree by walking the tree
either top-down or bottom-up.
os Module
os Module
|os.walk()
 |os.path()
| os.path() includes functions to
manipulate file paths.
○ os.path.join() - combines paths.
○ os.path.basename() - returns the base name of the path.
○ os.path.dirname() - returns the directory name from a path.
○ os.path.split() - splits a path into a pair (head, tail).
○ os.path.exists() - checks if a path exists.
○ os.path.isdir() - checks if a path is a directory.
○ os.path.isfile() - checks if a path is a file.
os Module

○ os.path.splitext() - splits the extension from a path.

os Module
|os.path()
 |Environment Variables
| The os module allows us to retrieve
information about the environment.

| Environment variables.
os Module
 |environ
| os.environ returns a dictionary
containing all of the environment
variables.
os Module
 |environ.get()
| os.environ.get() returns the value of
a specific environment variable.
os Module
os Module
|environ and environ.get()
 |Process Management
| We can manage processes with the
os module. It gives us the capability
to fork child processes, change the
user of a process, and more.
os Module
 |os.fork()
| This is often used when you want a
process to run different code
concurrently.

| For example:
○ The parent process might want to
wait for the child process to
os Module

complete its tasks or proceed with

executing the next lines of code.
 |os.fork() - Child & Parent
| The new process, called a child, is an
exact copy of the calling process,
called the parent.

| Except for a few values having been

changed including the process ID
(PID) and the parent process ID
os Module

(PPID).
os Module
|os.fork() with getuid()

● In this example, os.fork() is used to create a new child process.

● Then the if statement is used to distinguish between parent and child process
based on the PID returned by os.fork().
● The child process has a PID of 0, while the parent process has a PID of the child's
PID.
● os.wait() is used in the parent process to wait for the child process to complete.
● Finally, we get the user’s id (uid).
sys Module
 |sys Module
| The sys module provides access to
some variables used or maintained
by the Python interpreter and to
functions that interact strongly with
the interpreter.
sys Module
 |sys Module
| It provides numerous resources that
can be used to interact with the
Python runtime environment.

| We can read command-line

arguments, manipulate the Python
module search path, and handle
sys Module

errors and exceptions.

 |sys.argv
| sys.argv is a list in Python, which
contains command-line arguments
passed to the script.

| The user can provide various inputs

to the script from the command line.
sys Module

| sys.argv also contains the name of

the script, which is considered an
argument, as its first element.
sys Module
|sys.argv
 |sys.path
| sys.path provides Python’s module
search path.

| It is initialized from the environment

variable PYTHONPATH.
○ Or a built-in default if
PYTHONPATH is not set.
sys Module
 |sys.exc_info()
| sys.exc_info() provides exception
handling information.
sys Module
sys Module
|sys.path and sys.exc_info()
subprocess Module
 |subprocess Module
| This module provides excellent tools
for executing shell commands and
scripts.
subprocess Module
 |subprocess.run()
| subprocess.run() is used for creating
new processes and connecting to
their input/output/error pipes, and
subprocess Module

obtaining their return codes.

○ Must then wait for it to finish
running.
subprocess Module
|subprocess.run()
threading & multiprocessing
Modules
threading & multiprocessing modules
|threading and multiprocessing modules

| These are used for

parallel/concurrent execution.

| Threading is for I/O-bound tasks

while multiprocessing is for
CPU-bound tasks.
 |Disclaimer
threading & multiprocessing modules

| Both threading and multiprocessing

allow for concurrent execution, but
due to Python’s Global Interpreter
Lock (GIL), the threading module
does not truly run in parallel on
multiple cores.
 |Global Interpreter Lock
threading & multiprocessing modules

| Protects access to Python objects.

| Prevents multiple threads from executing
Python bytecodes at once in a single
process.

| It is necessary because Python’s memory

management is not thread-safe.

| This is as bottleneck for multi-threaded

programs - can’t effectively utilize multiple
CPU cores.
 |OK … ?
threading & multiprocessing modules

| But WTH is threading and

multiprocessing?
 |Concurrent Execution
threading & multiprocessing modules

| A property of systems in which

multiple tasks are executing in
overlapping time intervals.

| It’s about dealing with a lot of things

at once.

| In Python, both threading and

multiprocessing allow for concurrent
execution.
 |Threading
threading & multiprocessing modules

| Threads are the smallest units of a

process that can be executed
concurrently.

| They share the same memory space,

which makes communication
between threads relatively easy.

| But again, with the GIL, threads don’t

provide a speedup for CPU-bound
tasks.
 |Multiprocessing
threading & multiprocessing modules

| Processes do not share the same

memory space, making
communication between them a little
more complex.

| They can run on different CPU cores

concurrently.

| They avoid the GIL and do provide a

speedup for CPU-bound tasks.
 |I/O-bound Tasks
threading & multiprocessing modules

| These are tasks or programs that

spend most of their time waiting for
Input/Output (I/O) operations to
complete.
| Such as reading from or writing to a disk or
the network.

| Threading can be beneficial here because if

one thread is waiting or I/O completion, other
threads can continue their computation.
 |CPU-bound Tasks
threading & multiprocessing modules

| These are tasks or programs that

spend most of their time using the
CPU (computing something).
| Multiprocessing can be beneficial here
because it allows for using multiple cores,
hence increasing the computing capacity of
the program.
 |Following Code Example
threading & multiprocessing modules

| Demonstrates the difference between

threading and multiprocessing.
| Remember that threads and processes are
similar in that they both handle tasks
concurrently, but they work differently as
described in the previous slides.
 |Following Code Example
threading & multiprocessing modules

| In the provided script, the task()

function is run in two ways:
○ Once as a thread
○ Once as a process
| You will notice that in the threaded version, the task is
executed in the same process (with the same PID) as the
main script.

| In the multiprocessing version, a new process with a

different PID is created.

| This illustrates the fundamental difference between

threading and multiprocessing in Python.
threading & multiprocessing modules
|threading and multiprocessing
Summary
 |Summary
| We’ve covered some key Python
modules such as os, sys,
subprocess, threading, and
multiprocessing.

| Python’s strength lies in its

simplicity and the wide range of
Summary

modules it offers. This makes it a

powerful language for system
programming in Linux.
 |Further Reading
● Python Modules:
○ https://docs.python.org/3/tutorial/modules.html

● __main__
○ https://docs.python.org/3/library/__main__.html

● os Module
○ https://docs.python.org/3/library/os.html

● sys Module
○ https://docs.python.org/3/library/sys.html

● subprocess Module
○ https://docs.python.org/3/library/subprocess.html
Summary

● threading Module
○ https://docs.python.org/3/library/threading.html

● multiprocessing Module
○ https://docs.python.org/3/library/multiprocessing.html