Migration of table from CSVto postgres using Python
Last Updated :
09 Aug, 2024
In this article, we will learn to read data from a CSV file and insert it into a PostgreSQL database table using Python.
Prerequisites
- Python installed on your machine.
- PostgreSQL server and database setup.
- psycopg2 library to connect to PostgreSQL from Python.
- pandas library to handle CSV data.
Let's install the necessary Python libraries:
pip install psycopg2 pandas
Reading Data from CSV
Let's say we have a file called data.csv
name,age,gender
alice,21,female
abul,23,male
pranjali,22,female
priyanka,24,female
suraj,25,male
Now, we will read the data from the CSV file using the pandas library.
Python
import pandas
file = 'data.csv'
data = pandas.read_csv(file)
print(data.head())
Output:
Setting Up PostgreSQL Connection
Now, set up the connection to PostgreSQL database using the psycopg2 library.
Python
import psycopg2
conn = psycopg2.connect(
host="localhost",
database="gfgdb",
user="postgres",
password="1234"
)
cursor = conn.cursor()
Make sure to use your own username and password.
Creating a Table in PostgreSQL
Now, the next step is to create a table in the PostgreSQL database to store the CSV data. The table must have all the columns in the CSV data. In this case, our table will have three columns name, age, and gender.
Python
query = '''
CREATE TABLE employee (
id SERIAL PRIMARY KEY,
name VARCHAR(255),
age INT,
gender VARCHAR(10)
);
'''
cursor.execute(query)
conn.commit()
Inserting Data into PostgreSQL
Inserting the data from the CSV file into the PostgreSQL table.
Python
query = '''
INSERT INTO employee(name, age, gender)
VALUES (%s, %s, %s);
'''
for _, row in data.iterrows():
cursor.execute(query, tuple(row))
conn.commit()
Complete Example:
Python
import pandas, psycopg2
# Step 1: Setting Up PostgreSQL Connection
conn = psycopg2.connect(
host="localhost",
database="gfgdb",
user="postgres",
password="1234"
)
cursor = conn.cursor()
# Step 2: Reading Data from CSV
file = 'data.csv'
data = pandas.read_csv(file)
print(data.head())
# Step 3: Creating a employee Table in PostgreSQL
query = '''
CREATE TABLE employee (
id SERIAL PRIMARY KEY,
name VARCHAR(255),
age INT,
gender VARCHAR(10)
);
'''
cursor.execute(query)
conn.commit()
# Step 4: Inserting Data into PostgreSQL
query = '''
INSERT INTO employee(name, age, gender)
VALUES (%s, %s, %s);
'''
for _, row in data.iterrows():
cursor.execute(query, tuple(row))
conn.commit()
# Step 5: Verifying Data Migration
query = "SELECT * FROM employee;"
cursor.execute(query)
rows = cursor.fetchall()
for row in rows:
print(row)
# Closing the connection
cursor.close()
conn.close()
Verifying Data Migration
Python
query = "SELECT * FROM employee;"
cursor.execute(query)
rows = cursor.fetchall()
for row in rows:
print(row)
Output:
outputBy following the above steps, we can easily read data from our csv file and add to our postgresql database.
Similar Reads
PostgreSQL - Create table using Python Creating tables in a PostgreSQL database using Python is a common task for developers working with databases. This process involves defining the structure of your data and ensuring that your database is optimized for efficient storage and retrieval. In this article, we will walk through the steps of
3 min read
Get column names from PostgreSQL table using Psycopg2 This article is an illustration of how to extract column names from PostgreSQL table using psycopg2 and Python. Used table for demonstration: Example 1: First, we connect the PostgreSQL database using psycopg2.connect()Â method, then we create a cursor using cursor() method, after that we use the cu
1 min read
PostgreSQL - Create Tables in Python Creating tables in PostgreSQL using Python is an essential skill for developers working with databases. This article will explore the process of creating new tables in the PostgreSQL database using Python.Why Create PostgreSQL Tables with Python?Using Python to create PostgreSQL tables is beneficial
4 min read
Perform PostgreSQL CRUD operations from Python The DDL is comprised of the Create, Read, Update, Delete (CRUD) operations which form the backbone of any SQL database system. Let us discuss how to perform CRUD operations on a PostgreSQL database using python. Â Pyscopg2 is the most preferred mode and is widely used to connect the PostgreSQL databa
8 min read
Python Select from PostgreSQL Table using Psycopg2 This article will introduce you to the use of the psycopg2 module which is used to connect to a PostgreSQL database from Python. We will look over how to establish a connection to a database, create a cursor object to execute DBMS SQL statements, execute a SELECT statement to retrieve the data from
7 min read
PostgreSQL Python - Update Data in Table In this article, we are going to see how to update existing data in PostgreSQL tables using the pyscopg2 module in Python. In PostgreSQL, the UPDATE TABLE with where clause is used to update the data in the existing table from the database. Syntax: UPDATE <table_name> SET column1 = value1, c
2 min read
Python - Import CSV into PostgreSQL In this article, we will see how to import CSV files into PostgreSQL using the Python package psycopg2. First, we import the psycopg2 package and establish a connection to a PostgreSQL database using the pyscopg2.connect() method. before importing a CSV file we need to create a table. In the example
2 min read
How to Copy a Table Definition in MySQL Using Python? Python requires an interface to access a database server. Python supports a wide range of interfaces to interact with various databases. To communicate with a MySQL database, MySQL Connector Python module, an API written purely in Python, is used. This module is self-sufficient meaning that it does
6 min read
Save API data into CSV format using Python In this article, we are going to see how can we fetch data from API and make a CSV file of it, and then we can perform various stuff on it like applying machine learning model data analysis, etc. Sometimes we want to fetch data from our Database Api and train our machine learning model and it was ve
6 min read
Python MariaDB - Insert into Table using PyMySQL MariaDB is an open source Database Management System and its predecessor to MySQL. The pymysql client can be used to interact with MariaDB similar to that of MySQL using Python. In this article we will look into the process of inserting rows to a table of the database using pymysql. You can insert o
2 min read