Scribd
Upload
21 views4 pages

Echftp Auto Part

Uploaded by

dbatalks
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
21 views4 pages

Echftp Auto Part

Uploaded by

dbatalks
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 4

####################################################

Automating
Partition Creation and Deletion Procedure

####################################################

Purpose: This document outlines the systematic procedure for configuring auto-
creation and deletion of partitions in your PostgreSQL database environment.

Step 1: Create Configuration Table #Skip this step if it already exist for
routing_entity_log
--------------------------

Connect to the designated database, in this case, the "config" database.

psql> \c config

psql> CREATE TABLE pg_part_maintain (


db_name character varying(100),
schema_name character varying(100),
table_name character varying(100),
partition_prefix character varying(100),
retention integer,
lead_part integer,
part_col character varying(100)
);

Step 2: Insert Table Values for Automatic Partition Setup


-------------------------------------------------
INSERT INTO pg_part_maintain (db_name, schema_name, table_name,
partition_prefix, retention, lead_part, part_col)
VALUES ('config', 'public',
'routing_entity_log', 'p_routing_entity_log', 12, 15, 'time');

Step 3: Copy auto_part.sh Partition Script to Designated Location


---------------------------------------------------------
The script should be placed in the designated directory, such as
/data/scripts and can be copied from 10.60.208.212 placed at /data/scripts
auto_part.sh will drop partitions that are older than retention
mentioned in pg_part_maintain and will create future partitions so would not be
able

Step 4: Update auto_part.sh with required DB details


--------------------------------------------
Update below parameters in the first 10 lines of script as per your
environment:
vi auto_part.sh
DB_PORT=<DB_Port>
DB_NAME=<dbname>
DB_USER=<db_user>
DB_PWD=<db_user_password>
DUMP_DIR=<backup_directory>

mkdir -p /tmp/pdump
--Connect with Config DB and execute below:
-- Grant connection privilege to the database
GRANT CONNECT ON DATABASE config TO dbadmin;

-- Grant USAGE on the schema


GRANT USAGE ON SCHEMA public TO dbadmin;

-- Grant SELECT and EXECUTE privileges for necessary tables and


functions
GRANT SELECT ON ALL TABLES IN SCHEMA public TO dbadmin;

-- Grant EXECUTE privilege for `pg_is_in_recovery()` function


GRANT EXECUTE ON FUNCTION pg_is_in_recovery() TO dbadmin;

Step 5: Schedule Job in Crontab (This will execute daily at 9PM, change as per
requirement)
-----------------------
a) Create directory for logs
mkdir -p /data/scripts/part_log

0 1 * * * /data/scripts/auto_part.sh >>
/data/scripts/part_log/auto_part_$(date +\%Y\%m\%d).log 2>&1
0 2 * * * find /data/scripts/part_log -name "*.log" -type f -mtime +15
-exec rm -f {} \;

Step 6: Create Partitioned Table


------------------------
CREATE TABLE IF NOT EXISTS routing_entity_log_new (
id uuid DEFAULT uuid_generate_v4() NOT NULL,
"routingEntityKey" varchar NOT NULL,
"routingEntityTenantId" varchar NOT NULL,
"routingEntityType" varchar NOT NULL,
operation public.routing_entity_log_operation_enum NOT NULL,
"newRoutingEntityData" json NULL,
"oldRoutingEntityData" json NULL,
"user" varchar NOT NULL,
"time" timestamptz NOT NULL,
CONSTRAINT pk_routing_entity_log_new PRIMARY KEY (id, time)
) PARTITION BY RANGE ("time");

Step 9: Create Partitions


-----------------
* Get the DDL of all partitions available already in DB, first query
will also show the count and 2nd will share only DDL

SELECT 'CREATE TABLE p_routing_entity_log_'||TO_CHAR(date(time),


'YYYY_MM_DD')||' PARTITION OF routing_entity_log_new FOR VALUES FROM ('''||
TO_CHAR(date(time), 'YYYY-MM-DD')||''') to ('''||TO_CHAR(date(time)+1,'YYYY-MM-
DD')||''');', count(*) FROM routing_entity_log GROUP BY DATE(time) order by 1;

SELECT 'CREATE TABLE p_routing_entity_log_'||TO_CHAR(date(time),


'YYYY_MM_DD')||' PARTITION OF routing_entity_log_new FOR VALUES FROM ('''||
TO_CHAR(date(time), 'YYYY-MM-DD')||''') to ('''||TO_CHAR(date(time)+1,'YYYY-MM-
DD')||''');' FROM routing_entity_log GROUP BY DATE(time) order by 1;

Step 10: Apply Constraints to Partitions


-------------------------------
* Get DDL to create constrains on above created partitions
SELECT 'ALTER TABLE p_routing_entity_log_' || TO_CHAR(date(time),
'YYYY_MM_DD') || ' ADD CONSTRAINT p_routing_entity_log_' || TO_CHAR(date(time),
'YYYY_MM_DD') || '_chk CHECK (time >= ''' || TO_CHAR(date(time), 'YYYY-MM-DD') ||
''' AND time < ''' || TO_CHAR(date(time) + INTERVAL '1 day', 'YYYY-MM-DD') ||
''');' FROM routing_entity_log GROUP BY DATE(time) ORDER BY 1;

Step 11: Create Default Partition


------------------------
CREATE TABLE p_routing_entity_log_default PARTITION OF
routing_entity_log_new DEFAULT;

Step 12: Verification of Partitions


--------------------------
SELECT c.oid::regclass AS child_partition,
p.oid::regclass AS parent_tbl,
pg_get_expr(c.relpartbound, c.oid) AS boundaries
FROM pg_class AS p
JOIN pg_inherits AS i ON p.oid = i.inhparent
JOIN pg_class AS c ON i.inhrelid = c.oid
WHERE p.relkind = 'p';

OR

\d+ routing_entity_log

Step 13: Switch an unpartitioned table into a partitioned one


----------------------------------------------------
Take downtime to switch unpartitioned to the partitioned table

a) Rename original unpartitioned table to backup table


ALTER TABLE routing_entity_log RENAME TO routing_entity_log_bkp;

b) Rename paritioned table with the required name


ALTER TABLE routing_entity_log_new RENAME TO routing_entity_log;

c) Move data from unpartitioned table to the paritioned one


INSERT INTO routing_entity_log_new (
id,
"routingEntityKey",
"routingEntityTenantId",
"routingEntityType",
operation,
"newRoutingEntityData",
"oldRoutingEntityData",
"user",
"time"
)
SELECT
id,
"routingEntityKey",
"routingEntityTenantId",
"routingEntityType",
operation,
"newRoutingEntityData",
"oldRoutingEntityData",
"user",
"time"
FROM public.routing_entity_log;

d) Verification of routing_entity_log
SELECT c.oid::regclass AS child_partition,
p.oid::regclass AS parent_tbl,
pg_get_expr(c.relpartbound, c.oid) AS boundaries
FROM pg_class AS p
JOIN pg_inherits AS i ON p.oid = i.inhparent
JOIN pg_class AS c ON i.inhrelid = c.oid
WHERE p.relkind = 'p';

OR

\d+ routing_entity_log

d) After verification(manually executing the cronjob) of processes &


data drop bkp tables:
drop table routing_entity_log_bkp;

Step 14: Schedule Job in Crontab (This will execute daily at 9PM, change as per
requirement)
-----------------------
a) Execute the auto_part job manually from OS, it will create today and
future partitions

sh /data/scripts/auto_part.sh >>
/data/scripts/part_log/auto_part_log_$(date +\%Y\%m\%d).log 2>>
/data/scripts/part_log/auto_part_log_$(date +\%Y\%m\%d).err

You might also like