UNIT-5

Database Programming:
Introduction:
1). Persistent Storage:
In any application, there is a need for persistent storage. Generally, there are three basic storage
mechanisms: files, a database system, or some sort of hybrid, such as an API that sits on top of
one of those existing systems, an ORM (Object-Relational Mapping), file manager,
spreadsheet, configuration file, etc.

2). Basic Database Operations and SQL:

Before we dig into databases and how to use them with Python, we want to present a quick
introduction (or review if you have some experience) to some elementary database concepts
and SQL.
i). Underlying Storage
Databases usually have a fundamental persistent storage that uses the file system, that is,
normal operating system files, special operating system files, and even raw disk partitions.
ii). User Interface:
Most database systems provide a command-line tool with which to issue SQL commands or
queries. There are also some GUI tools that use the command-line clients or the database client
library, affording users a much more comfortable interface.
iii). Databases
A relational database management system (RDBMS) can usually managebmultiple databases,
such as sales, marketing, customer support, etc., all onnthe same server (if the RDBMS is
server-based; simpler systems are usually not).
iv). Components
The table is the storage abstraction for databases. Each row of data will have fields that
correspond to database columns. The set of table definitions of columns and data types per
table all put together define the database schema.
Databases are created and dropped. The same is true for tables. Adding new rows to a database
is called inserting; changing existing rows in a table is called updating; and removing existing
rows in a table is called deleting. These actions are usually referred to as database commands
or operations. Requesting rows from a database with optional criteria is called querying.
When you query a database, you can fetch all of the results (rows) at once, or just iterate slowly
over each resulting row. Some databases use the concept of a cursor for issuing SQL
commands, queries, and grabbing results, either all at once or one row at a time.
v). SQL
Database commands and queries are given to a database via SQL. Not all databases use SQL,
but the majority of relational databases do. Following are some examples of SQL commands.
Note that most databases are configured to be case-insensitive, especially database commands.
The accepted style is to use CAPS for database keywords. Most command-line programs
require a trailing semicolon (;) to terminate a SQL statement.

Creating a Database
CREATE DATABASE test;
GRANT ALL ON test.* to user(s);

The first line creates a database named “test,” and assuming that you are a database
administrator, the second line can be used to grant permissions to specific users (or all of them)
so that they can perform the database operations that follow.

Using a Database
USE test;
If you logged into a database system without choosing which database you want to use, this
simple statement allows you to specify one with which to perform database operations.

Dropping a Database
DROP DATABASE test;
This simple statement removes all the tables and data from the database and deletes it from the
system.

Creating a Table
CREATE TABLE users (login VARCHAR(8), userid INT, projid INT);
This statement creates a new table with a string column login and a pair of integer fields, userid
and projid.

Dropping a Table
DROP TABLE users;
This simple statement drops a database table, along with all its data.
Inserting a Row
INSERT INTO users VALUES('leanna', 2111, 1);

You can insert a new row in a database by using the INSERT statement.You specify the table
and the values that go into each field. For our example, the string 'leanna' goes into the login
field, and 2111 and 1 to userid and projid, respectively.

Updating a Row
UPDATE users SET projid=4 WHERE projid=2;
UPDATE users SET projid=1 WHERE userid=311;
To change existing table rows, you use the UPDATE statement. Use SET for the columns that
are changing and provide any criteria for determining which rows should change. In the first
example, all users with a “project ID” (or projid) of 2 will be moved to project #4. In the second
example,
we take one user (with a UID of 311) and move him to project #1.

Deleting a Row
DELETE FROM users WHERE projid=%d;
DELETE FROM users;

To delete a table row, use the DELETE FROM command, specify the table from which you
want to delete rows, and any optional criteria. Without it, as in the second example, all rows
will be deleted.

3). Databases and Python:

The way to access a database from Python is via an adapter. An adapter is a Python module
with which you can interface to a relational database’s client library, usually in C. It is
recommended that all Python adapters conform to the API of the Python database special
interest group (DBSIG).
The below Figure illustrates the layers involved in writing a Python database application, with
and without an ORM. The figure demonstrates that the DB-API is your interface to the C
libraries of the database client.
The Python DB-API:
The API is a specification that states a set of required objects and database access mechanisms
to provide consistent access across the various database adapters and underlying database
systems. Like most community-based efforts, the API was driven by strong need.
SIG for Python database connectivity was formed, and eventually, an API was born: the DB-
API version 1.0. The API provides for a consistent interface to a variety of relational databases,
and porting code between different databases is much simpler.

i). Module Attributes:

The DB-API specification mandates that the features and attributes listed below must be
supplied. A DB-API-compliant module must define the global attributes as shown in bellow
Table.
Data Attributes:
apilevel
This string (not float) indicates the highest version of the DB-API with which the module is
compliant, for example, 1.0, 2.0, etc. If absent, 1.0 should be assumed as the default value.
threadsafety
This an integer that can take the following possible values:
• 0: Not threadsafe, so threads should not share the module at all
• 1: Minimally threadsafe: threads can share the module but not connections
• 2: Moderately threadsafe: threads can share the module and connections but not cursors
• 3: Fully threadsafe: threads can share the module, connections, and cursors

paramstyle
The API supports a variety of ways to indicate how parameters should be integrated into an
SQL statement that is eventually sent to the server for execution. This argument is just a string
that specifies the form of string substitution you will use when building rows for a query or
command.

Function Attribute(s)
connect() Function access to the database is made available through Connection objects. A
compliant module must implement a connect() function, which creates and returns a
Connection object. Table 6-3 shows the arguments to connect().
You can pass in database connection information as a string with multiple parameters (DSN)
or individual parameters passed as positional arguments (if you know the exact order), or
more likely, keyword arguments. Here is an example of using connect() from PEP 249:
connect(dsn='myhost:MYDB',user='guido',password='234$')

The use of DSN versus individual parameters is based primarily on the system to which you
are connecting. For example, if you are using an API like Open Database Connectivity (ODBC)
or Java DataBase Connectivity (JDBC), you would likely be using a DSN, whereas if you are
working directly with a database, then you are more likely to issue separate login parameters.
Another reason for this is that most database adapters have not implemented support for DSN.
The following are some examples of non-DSN connect() calls. Note that not all adapters have
implemented the specification exactly, e.g., MySQLdb uses db instead of database.

• MySQLdb.connect(host='dbserv', db='inv', user='smith')

• PgSQL.connect(database='sales')
• psycopg.connect(database='template1', user='pgsql')
• gadfly.dbapi20.connect('csrDB', '/usr/local/database')
• sqlite3.connect('marketing/test')

Exceptions
Exceptions that should also be included in the compliant module as globals are shown in
Table 6-4.
ii). Connection Objects:
Connections are how your application communicates with the database. They represent the
fundamental mechanism by which commands are sent to the server and results returned. Once
a connection has been established (or a pool of connections), you create cursors to send requests
to and receive replies from the database.

Connection Object Methods

Connection objects are not required to have any data attributes but should define the methods
shown in Table 6-5.
When close() is used, the same connection cannot be used again without running into an
exception. The commit() method is irrelevant if the database does not support transactions or
if it has an auto-commit feature that has been enabled. You can implement separate methods
to turn auto-commit off or on if you wish. Since this method is required as part of the API,
databases that do not support transactions should just implement “pass” for this method.

Like commit(), rollback() only makes sense if transactions are supported in the database. After
execution, rollback() should leave the database in the same state as it was when the transaction
began. According to PEP 249, “Closing a connection without committing the changes first will
cause an implicit rollback to be performed.”

iii). Cursor Objects:

Once you have a connection, you can begin communicating with the database. As we
mentioned earlier in the introductory section, a cursor lets a user issue database command and
retrieve rows resulting from queries.
Once you have created a cursor, you can execute a query or command (or multiple queries and
commands) and retrieve one or more rows from the results set. Table 6-6 presents Cursor object
data attributes and methods.
The most critical attributes of cursor objects are the execute*() and the fetch*() methods; all
service requests to the database are performed by these. The arraysize data attribute is useful
in setting a default size for fetchmany().

iv). Type Objects and Constructors:

The interface between two different systems are the most fragile (unstable or vulnerable). This
is seen when converting Python objects to C types and vice versa. Similarly, there is also a fine
line between Python objects and native database objects. As a programmer writing to Python’s
DB-API, the parameters you send to a database are given as strings, but the database might
need to convert it to a variety of different, supported data types that are correct for any particular
query.

For example, should the Python string be converted to a VARCHAR, a TEXT, a BLOB, or a
raw BINARY object, or perhaps a DATE or TIME object if that is what the string is supposed
to be? Care must be taken to provide database input in the expected format; therefore, another
requirement of the DB-API is to create constructors that build special objects that can easily be
converted to the appropriate database objects. Table 6-7 describes classes that can be used for
this purpose. SQL NULL values are mapped to and from Python’s NULL object, None.

v). Relational Databases:

Python supports multiple Relational Database Management Systems (RDBMS) through
various database adapters. Here’s a list of relational databases that support Python:

Commercial RDBMSs
• IBM Informix
• Sybase
• Oracle
• Microsoft SQL Server
• IBM DB2
• SAP
• Embarcadero Interbase
• Ingres
Open-Source RDBMSs
• MySQL
• PostgreSQL
• SQLite
• Gadfly
Database APIs
• JDBC
• ODBC
Non-Relational Databases
• MongoDB
• Redis
• Cassandra
• SimpleDB
• Tokyo Cabinet
• CouchDB
• Bigtable (via Google App Engine Datastore API)

vi). Databases and Python: Adapters

For each of the databases supported, there exists one or more adapters that let you connect to
the target database system from Python. Some databases, such as Sybase, SAP, Oracle, and
SQLServer, have more than one adapter available.
You have to keep in mind that most adapters provide just the basic necessities to get you
connected to the database. It is the extras that you might be looking for. Keep in mind that you
are responsible for higher-level code like threading and thread management as well as
management of database connection pools, etc.

vii). Examples of Using Database Adapters

First, let’s look at a some sample code, from creating a database to creating a table and using
it. We present examples that use MySQL, PostgreSQL, and SQLite.

MySQL with Python (MySQLdb & MySQL Connector):

Python provides several libraries to interact with MySQL databases, the most popular being:

MySQLdb (mysqlclient) – Faster, but does not support Python 3 officially.

MySQL Connector/Python – Official MySQL library, supports Python 3.
For modern applications, MySQL Connector/Python is recommended because it is actively
maintained.

If MySQL is not available in Python, you need to install the appropriate MySQL database
software and the Python MySQL connector. Follow these steps:
1. Install MySQL Server (If Not Installed)
If MySQL is not installed on your system, download and install it:
Windows: Download MySQL Installer from MySQL Official Website

2. Install MySQL Connector for Python

You need to install a MySQL connector to interact with MySQL in Python. There are two
popular options:

(A) Install mysql-connector-python (Official MySQL Library)

pip install mysql-connector-python

(B) Install MySQLdb (Faster, Requires Extra Setup)

pip install mysqlclient

3. Verify Installation
To check if MySQL is installed and accessible in Python, run:

import mysql.connector
print("MySQL Connector Installed Successfully!")

OR

import MySQLdb
print("MySQLdb Installed Successfully!")
4. Connect Python to MySQL
Now, test the connection:

import mysql.connector

conn = mysql.connector.connect(
host="localhost",
user="root",
password="your_password"
)

if conn.is_connected():
print("Connected to MySQL successfully!")

conn.close()

If you see "Connected to MySQL successfully!", your installation is complete!

How Python Interacts with MySQL

• Establish a connection to the database.
• Create a cursor object to execute SQL queries.
• Execute queries like SELECT, INSERT, UPDATE, DELETE.
• Commit changes (for INSERT/UPDATE operations).
• Fetch results (for SELECT queries).
• Close the cursor and connection.
Example:
import mysql.connector

# Establish connection to MySQL

conn = mysql.connector.connect(
host="localhost", # MySQL server address
user="root", # MySQL username
password="password", # MySQL password
database="test_db" # Database name (must exist)
)

# Create a cursor object

cursor = conn.cursor()

# Create a new database (if it doesn't exist)

cursor.execute("CREATE DATABASE IF NOT EXISTS test_db")

# Select the database

cursor.execute("USE test_db")

# Create a table (if not exists)

cursor.execute("""
CREATE TABLE IF NOT EXISTS users (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100) UNIQUE
)
""")

# Insert data into the table

insert_query = "INSERT INTO users (name, email) VALUES (%s, %s)"
values = [
("Alice", "[email protected]"),
("Bob", "[email protected]"),
("Charlie", "[email protected]")
]
cursor.executemany(insert_query, values) # Insert multiple rows at once

# Commit changes
conn.commit()

# Fetch data from the table

cursor.execute("SELECT * FROM users")
rows = cursor.fetchall()

print("\nUser Data:")
for row in rows:
print(row)

# Update data
cursor.execute("UPDATE users SET name = 'Alice Updated' WHERE name = 'Alice'")
conn.commit()

# Delete data
cursor.execute("DELETE FROM users WHERE name = 'Charlie'")
conn.commit()

# Using prepared statements (for security)

name = "Eve"
email = "[email protected]"
cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", (name, email))
conn.commit()

# Drop table (optional)

# cursor.execute("DROP TABLE users")
# conn.commit()

# Close cursor and connection

cursor.close()
conn.close()

Explanation of MySQL-Python Methods & Objects

Method / Object Purpose

mysql.connector.connect() Connects to the MySQL database
cursor = conn.cursor() Creates a cursor object to execute queries
cursor.execute(query) Executes an SQL command
cursor.executemany(query, values) Executes multiple SQL commands at once
conn.commit() Saves changes to the database
cursor.fetchall() Fetches all rows from the last executed query
cursor.fetchone() Fetches one row from the last executed query
cursor.close() Closes the cursor
conn.close() Closes the database connection

PostgreSQL with Python (psycopg2):

Python interacts with PostgreSQL databases using the psycopg2 library, which is the most
commonly used PostgreSQL adapter for Python.

Install psycopg2
First, install psycopg2 if it's not installed:
pip install psycopg2
How Python Interacts with PostgreSQL
• Connect to a PostgreSQL database.
• Create a cursor object to execute SQL queries.
• Execute queries such as SELECT, INSERT, UPDATE, and DELETE.
• Commit changes for INSERT or UPDATE operations.
• Fetch results for SELECT queries.
• Close the cursor and connection.

Example
This script demonstrates all methods and objects used to interact with PostgreSQL.

import psycopg2

# Establish connection to PostgreSQL

conn = psycopg2.connect(
dbname="postgres", # Connect to default database first
user="postgres", # PostgreSQL username
password="password", # PostgreSQL password
host="localhost", # PostgreSQL server address
port="5432" # Default PostgreSQL port
)

# Create a cursor object

cursor = conn.cursor()

# Create a new database (if it doesn’t exist)

cursor.execute("SELECT 1 FROM pg_database WHERE datname = 'test_db'")
exists = cursor.fetchone()
if not exists:
cursor.execute("CREATE DATABASE test_db")
# Commit and close connection to switch databases
conn.commit()
cursor.close()
conn.close()

# Reconnect to the newly created database

conn = psycopg2.connect(
dbname="test_db",
user="postgres",
password="password",
host="localhost",
port="5432"
)
cursor = conn.cursor()

# Create a table (if not exists)

cursor.execute("""
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100) UNIQUE
)
""")
conn.commit()

# Insert data into the table

insert_query = "INSERT INTO users (name, email) VALUES (%s, %s) ON CONFLICT
(email) DO NOTHING"
values = [
("Alice", "[email protected]"),
 ("Bob", "[email protected]"),
("Charlie", "[email protected]")
]
cursor.executemany(insert_query, values) # Insert multiple rows
conn.commit()

# Fetch data from the table

cursor.execute("SELECT * FROM users")
rows = cursor.fetchall()

print("\nUser Data:")
for row in rows:
print(row)

# Update data
cursor.execute("UPDATE users SET name = 'Alice Updated' WHERE name = 'Alice'")
conn.commit()

# Delete data
cursor.execute("DELETE FROM users WHERE name = 'Charlie'")
conn.commit()

# Using prepared statements (for security)

name = "Eve"
email = "[email protected]"
cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s) ON CONFLICT
(email) DO NOTHING", (name, email))
conn.commit()

# Drop table (optional)

cursor.execute("DROP TABLE users")
conn.commit()

# Close cursor and connection

cursor.close()
conn.close()

Explanation of PostgreSQL-Python Methods & Objects

Method / Object Purpose
psycopg2.connect() Connects to the PostgreSQL database
cursor = conn.cursor() Creates a cursor object to execute queries
cursor.execute(query) Executes an SQL command
cursor.executemany(query, values) Executes multiple SQL commands at once
conn.commit() Saves changes to the database
cursor.fetchall() Fetches all rows from the last executed query
cursor.fetchone() Fetches one row from the last executed query
cursor.close() Closes the cursor
conn.close() Closes the database connection
Method / Object Purpose
psycopg2.connect() Connects to the PostgreSQL database
cursor = conn.cursor() Creates a cursor object to execute queries

SQLite with Python (sqlite3):

SQLite is a lightweight, file-based database that comes built-in with Python, so you don’t need
to install anything extra. It is ideal for small-scale applications, testing, and development.

Advantages:
• No installation required – Comes with Python by default
• Lightweight – Uses a single file to store data (.db file)
• Cross-platform – Works on Windows, Linux, and macOS
• Easy to use – No need for a dedicated database server

How Python Interacts with SQLite:

• Connect to an SQLite database (creates a .db file if it doesn’t exist).
• Create a cursor object to execute SQL queries.
• Execute queries like CREATE, INSERT, UPDATE, DELETE, and SELECT.
• Commit changes for INSERT or UPDATE operations.
• Fetch results from SELECT queries.
• Close the connection when done.
Example:
import sqlite3

# Connect to SQLite database (Creates 'test.db' if not exists)

conn = sqlite3.connect("test.db")

# Create a cursor object

cursor = conn.cursor()

# Create a table (if not exists)

cursor.execute("""
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL
)
""")
conn.commit()

# Insert data into the table

insert_query = "INSERT INTO users (name, email) VALUES (?, ?)"
values = [
("Alice", "[email protected]"),
("Bob", "[email protected]"),
("Charlie", "[email protected]")
]
cursor.executemany(insert_query, values) # Insert multiple rows at once
conn.commit()

# Fetch all data from the table

cursor.execute("SELECT * FROM users")
rows = cursor.fetchall()

print("\nUser Data:")
for row in rows:
print(row)

# Fetch a single row

cursor.execute("SELECT * FROM users WHERE name = ?", ("Alice",))
single_row = cursor.fetchone()
print("\nSingle User:", single_row)

# Update data
cursor.execute("UPDATE users SET name = 'Alice Updated' WHERE name = 'Alice'")
conn.commit()

# Delete data
cursor.execute("DELETE FROM users WHERE name = 'Charlie'")
conn.commit()

# Using a prepared statement (Prevents SQL injection)

name = "Eve"
email = "[email protected]"
cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", (name, email))
conn.commit()

# Drop table (optional)

# cursor.execute("DROP TABLE users")
# conn.commit()
# Close cursor and connection
cursor.close()
conn.close()

ORMs (Object-Relational Mapping):

As seen in the previous section, a variety of different database systems are available today, and
most of them have Python interfaces with which you can harness their power. The only
drawback to those systems is the need-to-know SQL. If you are a programmer who feels more
comfortable with manipulating Python objects instead of SQL queries, yet still want to use a
relational database as your data back-end, then you would probably prefer to use ORMs.
Object-Relational Mapping (ORM) is a technique that allows developers to interact with
databases using Python objects instead of writing raw SQL queries.

Popular Python ORMs:

• SQLAlchemy – Most powerful & widely used.
• Django ORM – Built into Django framework.
• Peewee – Lightweight ORM for simple projects.
• Tortoise-ORM – Asynchronous ORM for FastAPI, Django, etc.

Why Use an ORM?

• No need to write raw SQL – Queries are written in Python.
• Database-agnostic – Supports multiple databases like MySQL, PostgreSQL, SQLite,
etc.
• Object-oriented – Database tables are represented as Python classes.
• Automatic migrations – ORMs can create/update tables without manual SQL
commands.

Install SQLAlchemy
pip install sqlalchemy

Example:
This example covers all CRUD (Create, Read, Update, Delete) operations using SQLAlchemy.

from sqlalchemy import create_engine, Column, Integer, String

from sqlalchemy.orm import declarative_base, sessionmaker
# Define the database connection (using SQLite for simplicity)
DATABASE_URL = "sqlite:///test.db"
engine = create_engine(DATABASE_URL, echo=True) # echo=True shows SQL queries

# Create a session factory

SessionLocal = sessionmaker(bind=engine)

# Define the base class for ORM models

Base = declarative_base()

# Define a User model (Represents 'users' table)

class User(Base):
__tablename__ = "users"

id = Column(Integer, primary_key=True, autoincrement=True)

name = Column(String, nullable=False)
email = Column(String, unique=True, nullable=False)

# Create the database tables

Base.metadata.create_all(engine)

# Open a new session

session = SessionLocal()

# Insert a new user

new_user = User(name="Alice", email="[email protected]")
session.add(new_user)
session.commit()
# Insert multiple users
users = [
User(name="Bob", email="[email protected]"),
User(name="Charlie", email="[email protected]")
]
session.add_all(users)
session.commit()

# Fetch all users

all_users = session.query(User).all()
print("\nAll Users:")
for user in all_users:
print(user.id, user.name, user.email)

# Fetch a single user

user = session.query(User).filter_by(name="Alice").first()
print("\nSingle User:", user.name, user.email)

# Update a user
user.name = "Alice Updated"
session.commit()

# Delete a user
session.delete(user)
session.commit()

# Close the session

session.close()
Explanation of SQLAlchemy ORM Methods & Objects
Method / Object Purpose
create_engine(DATABASE_URL) Connects to the database
declarative_base() Base class for ORM models
Column(Integer, String, etc.) Defines table columns
Base.metadata.create_all(engine) Creates the database tables
SessionLocal() Creates a new session
session.add(object) Adds a new object (record)
session.commit() Saves changes to the database
session.query(Model).all() Fetches all records from a table
session.query(Model).filter_by(name="Alice").first() Fetches a single record
session.delete(object) Deletes a record
session.close() Closes the session

Non-Relational Databases:
Non-relational databases, also known as NoSQL databases, store data in key-value, document,
graph, or columnar formats instead of traditional tables and rows.

Types of NoSQL Databases & Python Libraries:

NoSQL Type Database Python Library
Key-Value Store Redis redis-py
Document Store MongoDB pymongo
Column-Family Store Cassandra cassandra-driver
Graph Database Neo4j neo4j

1. Redis (Key-Value Store)

Install Redis & Python Client

pip install redis

Example:
import redis

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, db=0)
# Set a key-value pair
r.set("username", "Alice")

# Get the value

print("Username:", r.get("username").decode())

# Store a list
r.rpush("fruits", "apple", "banana", "cherry")

# Get all list elements

print("Fruits:", r.lrange("fruits", 0, -1))

2. MongoDB (Document Store)

Install MongoDB & Python Client
pip install pymongo

Example:
from pymongo import MongoClient

# Connect to MongoDB
client = MongoClient("mongodb://localhost:27017/")
db = client["test_db"]
collection = db["users"]

# Insert a document
user = {"name": "Alice", "email": "[email protected]"}
collection.insert_one(user)

# Fetch all users

for user in collection.find():
print(user)

3. Cassandra (Column-Family Store)

Install Cassandra & Python Client
pip install cassandra-driver

Example:
from cassandra.cluster import Cluster

# Connect to Cassandra
cluster = Cluster(['127.0.0.1'])
session = cluster.connect()

# Create keyspace and table

session.execute("CREATE KEYSPACE IF NOT EXISTS test WITH replication =
{'class':'SimpleStrategy', 'replication_factor' : 1}")
session.set_keyspace("test")
session.execute("CREATE TABLE IF NOT EXISTS users (id UUID PRIMARY KEY, name
text, email text)")

# Insert data
import uuid
session.execute("INSERT INTO users (id, name, email) VALUES (%s, %s, %s)",
(uuid.uuid4(), "Alice", "[email protected]"))

# Fetch data
rows = session.execute("SELECT * FROM users")
for row in rows:
print(row)
4. Neo4j (Graph Database)
Install Neo4j & Python Client
pip install neo4j

Example:
from neo4j import GraphDatabase

# Connect to Neo4j
uri = "bolt://localhost:7687"
driver = GraphDatabase.driver(uri, auth=("neo4j", "password"))

# Create a person node

def create_person(tx, name):
tx.run("CREATE (p:Person {name: $name})", name=name)

# Query the database

def get_people(tx):
result = tx.run("MATCH (p:Person) RETURN p.name AS name")
for record in result:
print(record["name"])

with driver.session() as session:

session.write_transaction(create_person, "Alice")
session.read_transaction(get_people)
WEB Programming:
Web programming with Python involves building dynamic websites, web applications, and
APIs using Python-based frameworks and libraries. Python is widely used for web
development due to its simplicity, readability, and extensive ecosystem of web frameworks.

Web Surfing with Python (Web Scraping & Automation):

Web surfing with Python refers to the process of retrieving, extracting, and interacting with
web content using Python programs. This is done using web scraping, web automation, and
networking libraries.
Python allows developers to navigate the web, collect data, and automate tasks on websites
using powerful tools such as Requests, BeautifulSoup, Selenium, and Scrapy.

1. Web Scraping: Extracting Data from Websites:

Web scraping is the process of programmatically fetching a webpage and extracting specific
information from it.

Key Python Libraries for Web Scraping

Library Purpose
Requests Sends HTTP requests to websites and fetches HTML content.
BeautifulSoup Parses and extracts data from HTML and XML documents.
Scrapy A framework for large-scale web scraping.
Selenium Automates browsers and handles JavaScript-heavy websites.

Example: Extracting Headings from a Webpage with BeautifulSoup

import requests
from bs4 import BeautifulSoup

# Fetch the webpage

url = "https://example.com"
response = requests.get(url)

# Parse the HTML content

soup = BeautifulSoup(response.text, "html.parser")
# Extract and print headings
headings = soup.find_all("h1")
for heading in headings:
print(heading.text)

2. Web Automation: Interacting with Websites

Web automation refers to simulating user interactions with websites, such as filling forms,
clicking buttons, or even running automated tests.

Key Python Libraries for Web Automation

Library Purpose
Selenium Controls browsers for interaction with dynamic websites.
Pyppeteer Automates headless Chrome, similar to Puppeteer in JavaScript.
TagUI A tool for automating repetitive web tasks.

Example: Automating a Google Search with Selenium

from selenium import webdriver

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time

# Set up WebDriver (requires ChromeDriver)

driver = webdriver.Chrome()

# Open Google
driver.get("https://www.google.com")

# Search for a query

search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("Python web scraping")
search_box.send_keys(Keys.RETURN)

# Wait for results

time.sleep(3)

# Print search result titles

results = driver.find_elements(By.CSS_SELECTOR, "h3")
for result in results:
print(result.text)

# Close browser
driver.quit()

3. Accessing Web APIs (RESTful Services):

Web surfing also includes retrieving data from APIs, which provide structured data without
requiring web scraping.

Example: Fetching Data from an API

import requests

# Fetch data from a public API

url = "https://api.github.com/users/octocat"
response = requests.get(url)

# Print response JSON

print(response.json())
Creating Simple Web Clients with Python:
A web client is an application that interacts with web services or APIs to send and receive data
over the internet. In Python, we can create simple web clients using libraries like Requests,
HTTP.client, and urllib.

Methods to Create a Web Client Using the requests Library in Python:

The requests library in Python is widely used to create web clients that interact with web
services using HTTP methods. It allows applications to send and receive data from websites or
API
HTTP Methods in the requests Library

HTTP Method Purpose

GET Fetches data from a URL (read-only).
POST Sends data to a server (e.g., form submission).
PUT Updates existing data on a server.
DELETE Deletes data from a server.
HEAD Fetches headers from a URL (without content).
OPTIONS Retrieves allowed methods for a URL.
PATCH Partially updates data on a server.

Example: Sending a GET Request (Fetching Data)

A GET request is used to retrieve information from a web server.

import requests

# Define the URL

url = "https://jsonplaceholder.typicode.com/posts/1"

# Send GET request

response = requests.get(url)

# Print response data

print(response.json())
Example: Sending a POST Request (Submitting Data)

A POST request is used to send data to a web server.

import requests

# Define the API URL

url = "https://jsonplaceholder.typicode.com/posts"

# Data to send
data = {"title": "New Post", "body": "This is a test post", "userId": 1}

# Send POST request

response = requests.post(url, json=data)

# Print response
print(response.json())

Example: Sending a PUT Request (Updating Data)

A PUT request is used to update an entire resource.

import requests

# Define the API URL

url = "https://jsonplaceholder.typicode.com/posts/1"

# Data to update
data = {"title": "Updated Title", "body": "Updated content", "userId": 1}

# Send PUT request

response = requests.put(url, json=data)

# Print response
print(response.json())

Example: Sending a PATCH Request (Partially Updating Data)

A PATCH request is used to update part of a resource.

import requests

# Define the API URL

url = "https://jsonplaceholder.typicode.com/posts/1"

# Data to update (only title)

data = {"title": "Partially Updated Title"}

# Send PATCH request

response = requests.patch(url, json=data)

# Print response
print(response.json())

Example: Sending a DELETE Request (Deleting Data)

A DELETE request removes data from a web server.

import requests

# Define the API URL

url = "https://jsonplaceholder.typicode.com/posts/1"

# Send DELETE request

response = requests.delete(url)

# Print status code

print(response.status_code) # 200 means success

Example: Handling Headers & Authentication

Some APIs require authentication, which can be handled using headers.

Using Headers for Authentication

import requests

# API URL
url = "https://api.example.com/data"

# Headers with API Key

headers = {
"Authorization": "Bearer YOUR_API_KEY",
"User-Agent": "MyPythonClient/1.0"
}

# Send GET request with headers

response = requests.get(url, headers=headers)

# Print response
print(response.json())
Example: Handling Errors & Timeouts
A good web client should handle errors properly.

Error Handling in a Web Client

import requests

url = "https://www.example.com"

try:
response = requests.get(url, timeout=5)
response.raise_for_status() # Raise an error for bad responses (4xx, 5xx)
print(response.text)
except requests.exceptions.HTTPError as errh:
print("HTTP Error:", errh)
except requests.exceptions.ConnectionError as errc:
print("Connection Error:", errc)
except requests.exceptions.Timeout as errt:
print("Timeout Error:", errt)
except requests.exceptions.RequestException as err:
print("Something went wrong:", err)

Advanced Web Clients:

An advanced web client is a Python application that interacts with web services in a more
sophisticated way, handling authentication, session management, error handling, asynchronous
requests, and working with APIs efficiently.

Key Features of an Advanced Web Client

• Session Management – Maintain cookies and authentication across multiple requests.
• Handling Authentication – Work with OAuth, API keys, JWT tokens, etc.
• Error Handling & Retries – Ensure stability and recover from failures.
• Asynchronous Requests – Improve performance using asyncio and httpx.
• Streaming & Large Data Handling – Efficiently process large files or real-time data.
1. Managing Sessions (Cookies & Authentication)
A session helps maintain a logged-in state while making multiple requests.
Example: Using Sessions for Login and Data Retrieval

import requests

# Start a session
session = requests.Session()

# Login credentials
login_data = {"username": "testuser", "password": "securepass"}

# Perform login
session.post("https://example.com/login", data=login_data)

# Access a protected page after login

response = session.get("https://example.com/dashboard")

# Print response
print(response.text)

# Close the session

session.close()

2. Handling Authentication (API Keys, OAuth, JWT)

Many APIs require authentication using API keys, OAuth, or JWT tokens.
Example: API Key Authentication

import requests

headers = {"Authorization": "Bearer YOUR_API_KEY"}

response = requests.get("https://api.example.com/data", headers=headers)
print(response.json())

Example: OAuth 2.0 Authentication

OAuth is used by APIs like Google, Facebook, and GitHub.

from requests_oauthlib import OAuth1Session

# Setup OAuth1 session

oauth = OAuth1Session(
"client_key",
client_secret="client_secret",
resource_owner_key="token",
resource_owner_secret="token_secret"
)

# Fetch data
response = oauth.get("https://api.example.com/protected-data")
print(response.json())

3. Handling Errors & Retries

Web clients should handle failures gracefully.
Example: Robust Error Handling

import requests
from requests.exceptions import HTTPError, Timeout

url = "https://example.com"

try:
response = requests.get(url, timeout=5)
response.raise_for_status() # Raise error for bad responses
print(response.json()) # Process response if successful
except HTTPError as http_err:
print(f"HTTP error occurred: {http_err}")
except Timeout:
print("Request timed out")
except Exception as err:
print(f"Other error occurred: {err}")

4. Making Asynchronous Requests (Boost Performance)

Synchronous requests can slow down applications. Using asyncio and httpx, we can send
multiple requests at once.

Example: Sending Multiple Requests Asynchronously

import asyncio
import httpx

async def fetch(url):

async with httpx.AsyncClient() as client:
response = await client.get(url)
 return response.json()

async def main():

urls = [
"https://jsonplaceholder.typicode.com/posts/1",
"https://jsonplaceholder.typicode.com/posts/2",
"https://jsonplaceholder.typicode.com/posts/3",
]
results = await asyncio.gather(*(fetch(url) for url in urls))
print(results)

asyncio.run(main())

5. Handling Large Files & Streaming Data

If an API returns large data, handling it in memory can cause crashes. We can stream the
response instead.

Example: Downloading Large Files Efficiently

import requests

url = "https://example.com/large-file.zip"

with requests.get(url, stream=True) as response:

response.raise_for_status()
with open("large-file.zip", "wb") as file:
for chunk in response.iter_content(chunk_size=8192):
file.write(chunk)

print("Download complete!")
6. WebSockets for Real-Time Communication
Web clients can also handle real-time communication using WebSockets.

Example: Real-time WebSocket Client

import asyncio
import websockets

async def connect():

async with websockets.connect("wss://example.com/socket") as ws:
await ws.send("Hello Server!")
response = await ws.recv()
print(f"Received: {response}")

asyncio.run(connect())

CGI (Common Gateway Interface) – Helping Servers Process Client Data

Common Gateway Interface (CGI) is a protocol that enables web servers to execute scripts and
programs to generate dynamic content for web pages. It acts as a bridge between the client
(browser) and the server-side application.
Key Features of CGI:
• Handles client requests (like form submissions).
• Processes user input and generates dynamic responses.
• Runs programs written in Python, Perl, C, etc.
• Executed by the web server, not the browser.

How CGI Works?

CGI enables communication between the web server and a server-side script that processes
client requests.
Step-by-Step Process:
• A user submits a form or sends a request to the server.
• The web server detects the CGI script and executes it.
• The CGI script processes the input (e.g., storing it in a database).
• The CGI script generates an HTML response and sends it back to the client.
• The browser displays the response.

2. Writing a Simple CGI Script in Python

A CGI script is a program that runs on the server to process data.

Example: Basic CGI Script (script.py)

#!/usr/bin/env python3

import cgi

# Required for CGI scripts

print("Content-Type: text/html\n")

# Get user input from the HTML form

form = cgi.FieldStorage()
name = form.getvalue("name")
# Generate response
print(f"<html><body><h1>Hello, {name}!</h1></body></html>")

How It Works:
It retrieves input from a form (name).
Prints the Content-Type header.
Generates an HTML response dynamically.

3. Running the CGI Script

Setting Up the Web Server (Apache)

1. Enable CGI in Apache:

Modify /etc/apache2/sites-available/000-default.conf:

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Require all granted
</Directory>

2. Restart Apache:
sudo systemctl restart apache2

3. Place CGI Script in the cgi-bin Directory:

sudo mv script.py /usr/lib/cgi-bin/

chmod +x /usr/lib/cgi-bin/script.py # Make it executable
4. Access CGI Script in Browser:
http://localhost/cgi-bin/script.py?name=John

Output:
Hello, John!

Advanced CGI:
Processing Form Data
CGI can handle multiple input fields.

Example: Handling Multiple Inputs

#!/usr/bin/env python3

import cgi

print("Content-Type: text/html\n")

form = cgi.FieldStorage()
name = form.getvalue("name")
email = form.getvalue("email")

print(f"<html><body>")
print(f"<h1>Thank You, {name}!</h1>")
print(f"<p>Your email: {email}</p>")
print(f"</body></html>")

Handling POST Requests in CGI:

By default, form data is sent via GET, but POST is used for secure input handling.
Example: Processing POST Requests
#!/usr/bin/env python3

import cgi

print("Content-Type: text/html\n")

# Get form data from POST request

form = cgi.FieldStorage()
message = form.getvalue("message")

print(f"<html><body>")
print(f"<h1>Your Message:</h1>")
print(f"<p>{message}</p>")
print(f"</body></html>")

HTML Form for POST

<form action="/cgi-bin/post_script.py" method="post">
Message: <textarea name="message"></textarea>
<input type="submit" value="Submit">
</form>

Handling Errors in CGI Scripts:

Good CGI scripts must handle errors properly.

Example: Adding Error Handling

#!/usr/bin/env python3

import cgi
print("Content-Type: text/html\n")

try:
form = cgi.FieldStorage()
name = form.getvalue("name")

if not name:
raise ValueError("Name cannot be empty!")

print(f"<html><body><h1>Hello, {name}!</h1></body></html>")
except Exception as e:
print(f"<html><body><h1>Error:</h1><p>{e}</p></body></html>")

Web (HTTP) Servers:

A Web Server is software that handles HTTP requests from clients (browsers or APIs) and
delivers web pages, files, or responses. It acts as a bridge between users and web applications,
processing incoming requests and serving responses accordingly.

What is an HTTP Server?

An HTTP (HyperText Transfer Protocol) Server is a program that:
• Listens for incoming HTTP requests.
• Processes the request (e.g., retrieving a web page).
• Responds with an HTML page, JSON data, files, or an error message.
Basic Working of an HTTP Server

Client (Browser/API) ----> Web Server ----> Application/Database

⬆ | ⬇

| <------------------------- | <------------------ |
Response Sent Request Processed Request Received

How Web Servers Work?

• A client (browser or API) sends an HTTP request (e.g., visiting https://example.com).
• The web server processes the request:

➢ If it's a static file (HTML, CSS, JS, images), it serves the file.
➢ If it's a dynamic request, it forwards it to a web application (e.g.,
Python CGI, Flask, Django).
• The server sends an HTTP response (e.g., an HTML page, JSON data, or an error).
• The client (browser) displays the response.

Popular Web Servers:

Web Server Description Use Case
Apache One of the oldest and most Hosting websites, CGI
widely used open-source web scripts.
servers.
Nginx High-performance, event-driven Load balancing, static
server. Used for reverse content delivery.
proxying.
 Lighttpd Lightweight and fast, used for Small applications, IoT.
embedded systems.
Gunicorn Python-based web server used Running Python web apps.
with Flask/Django.
uWSGI Python application server, often Hosting Django/Flask apps.
used with Nginx.

Setting Up a Simple Python HTTP Server:

Python comes with a built-in HTTP server that serves static files.

Example: Running a Simple HTTP Server

python3 -m http.server 8000

Building a Simple Web Server with Python (http.server)

We can create a basic HTTP server using Python’s http.server module.

Example: Custom Python HTTP Server

from http.server import SimpleHTTPRequestHandler, HTTPServer

# Define server address

server_address = ("", 8000)

# Create a simple HTTP server

httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)

print("Serving on port 8000...")

httpd.serve_forever()

• Run it: python3 server.py

• Access it: http://localhost:8000
Handling Dynamic Requests in Python:
For handling dynamic requests, we use Flask or Django instead of http.server.

Example: Simple Flask HTTP Server

from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
return "Hello, World! This is a Flask HTTP server."

if __name__ == "__main__":
app.run(debug=True)

• Run it: python3 app.py

• Access it: http://127.0.0.1:5000/

Web Server Response Codes:

Web servers return HTTP status codes to indicate the result of a request.
Status Code Meaning
200 OK Request was successful.
301 Moved Permanently Redirected to a new URL.
403 Forbidden Access denied.
404 Not Found Requested resource does not exist.
500 Internal Server Error The server encountered an error.
Status Code Meaning
200 OK Request was successful.