Vacuum in PostgreSQL
0 likes618 views
This document discusses PostgreSQL's VACUUM utility. It explains that VACUUM is needed to reclaim space from deleted and updated tuples, prevent transaction ID wraparound issues, and update statistics. The document covers various aspects that interact with VACUUM like commit logs, visibility maps, and free space maps. It also describes the tasks performed by VACUUM, options available, and tuning autovacuum. Finally, it provides a high-level overview of the internal workings of VACUUM.
Vacuum in PostgreSQL
- 1. VACUUM in PostgreSQL Zalando SE Rafia Sabih 26-09-2018 1
- 2. 2 Outline ● Why VACUUM ● Things around VACUUM ● VACUUM utility ● Inside VACUUM ● Proposals around VACUUM 2
- 4. 4 Update Tuples in PostgreSQL T1 T2 T3 T4 T1 Old T2 T3 T4 Update T2 New T2 Page Page Both versions of the tuples exist in the page 4
- 5. 5 Delete tuples in PostgreSQL T1 T2 T3 T4 T1 T2 T3 T4 Delete T3 Page Page Tuple is marked deleted but remains in the page and its space remains unused 5
- 6. 6 Why VACUUM ● An UPDATE or DELETE of a row does not immediately removes the old version ○ VACUUM removes dead row versions in tables and indexes ○ Does not return the space to the operating system ■ except in the special case where one or more pages at the end of a table become entirely free and an exclusive table lock can be easily obtained ○ VACUUM FULL minimizes the size of the table by writing a complete new version of the table ● To overcome this, a new storage engine namely zheap based on undo logs is proposed 6
- 7. 7 Why VACUUM: Transaction wraparound ● The value of current txid reaches the maximum allowed value ○ The system remains blocked until old xacts are frozen ○ Frozen_txid ■ To avoid the problem of transaction wraparound ■ Reserved value is 2, meaning always visible ○ Freeze process rewrites the value of Xmin to frozen_txid for the tuples with too old Xmin ○ The xids are checked against a cutoff -- vacuum_freeze_min_age 7
- 9. Around VACUUM: CLOG and MultiXacts ● Commit log(CLOG) is a logical array for the status of the xacts ● Stored in shared memory and used for transaction lifecycle ● At every checkpoint or shutdown, flushed to disk in pg_xact directory ● Multixacts store information about row-level locks, updates etc. when multiple transactions are involved Committed Committed Aborted In-progress In-progress In-progress 1440 1441 1442 1443 1444 1445XIDs status 9
- 10. Around VACUUM: VisibilityMap ● Introduced to help VACUUM process in identifying which pages of the relation has dead tuples ● Provides information on if the tuples in the page are frozen ● Can be found in data directory with the suffix _vm ● Up-to date VMs speed-ups index-only scans ○ No need to visit corresponding heap pages ● Check extension pg_visibility for more information 10
- 11. Around VACUUM: FreeSpaceMap ● Tells about the free space available in the pages of the relation ○ Associated with every relation whether normal or index ● While inserting, it is required to check through FSM to find out which page has enough space ● Pg_freespacemap extension can be used for better visualization 11
- 12. Around VACUUM: HeapOnlyTuple ● A precursor to the VACUUM utility ○ Performs the defragmentation of a page ● Reuse the space of the index entries of the deleted or obsolete tuples ● Only for the case when updated tuple does not change any indexed columns Col1 Col2 Col3 Col4 With HOT A new tuple with unchanged indexed column creates no new index entry Without HOT Every version of a row has its own index entries 12
- 14. Tasks of VACUUM ● Prevent the issues of transaction or multi-xact id wraparound ■ Freeze xids when required ● To reclaim the space of dead tuples and indexes ■ Remove clog entries that are not required ■ Defragment pages with live tuples ● Update statistics of tables, it helps in better query plans ● Update visibility and freespace maps ● Remove CLOGs and multixacts 14
- 15. What can be vacuumed? ● Tuples or indexes that are not visible to any running transactions ○ Avoid long-running transactions ● Streaming replication with hot_standby_feedback enabled might also limit the vacuum on the master ● On an internal level vacuum requires special type of lock on the buffer -- BufferLockForCleanup which locks the buffer and checks that no other process has pinned that buffer ○ Long running select queries can also be blocking 15
- 16. VACUUM options ● Standard vacuum ● VACUUM FULL ○ Rewrites the table and indexes ○ Moderately-frequent standard VACUUM is better than infrequent VACUUM FULL runs for maintaining heavily-updated tables ○ Alternative: pg_repack ● VACUUM FREEZE ○ Aggressively freezes tuples ● Auto-vacuum ○ To run vacuum and analyze commands in the background ● Vacuumdb ○ For multiple databases in a cluster 16
- 17. AUTOVACUUM ● Comprises of a launcher and workers ● Priority for each database ○ XID freeze ○ MXID freeze ○ Least recently auto-vacuumed ○ These priorities are not within database ● Multiple workers can work at the same time on a database ● Auto-vacuum can get terminated if other process needs access to tables ○ Workers tries to get lock, if not available skip the table for now ● Auto-vacuum running for xid-wraparound are more aggressive and can not be canceled ● Temporary tables are ignored 17
- 18. Tuning VACUUM ● For autovacuum ○ Autovacuum_max_workers ○ Autovacuum_naptime ○ Autovacuum_vacuum_threshold ○ Autovacuum_vacuum_scale_factor ■ # dead_tup > (reltuples * scale_factor) + threshold ○ Autovacuum_work_mem ■ Default value -1 means take the value of maintenance_work_mem ● Vacuum_cost_delay 18
- 19. Take-away ● VACUUM ○ Removes dead tuples and indexes ○ Prevents the problem of transaction wraparound ○ No memory is returned to OS! ○ Autovacuum is your friend ● Obstacles for VACUUM ○ Long running transactions ○ Low autovacuum_work_mem ● Bloated indexes are more difficult than bloated relations ○ Reindex! ● pg_repack! 19
- 21. At a glance ● For each target relation ○ Acquire ShareUpdateExclusiveLock ○ Scan all pages to get all dead tuples, and freeze old tuples if necessary ■ The list of dead tuples and the index tuples pointing to dead tuples is limited by maintenance_work_mem ■ If the scan is not complete and amount of memory is consumed, it goes ahead to perform the next work for the listed tuples and continues the scan later 21
- 22. At a glance ○ Remove the index tuples that point to the respective dead tuples if exists ○ For each page of the table ■ Remove the dead tuples and reallocate the live tuples in the page ■ Update FSM and VM 22
- 23. At a glance ● Truncate the last page if possible ○ Update both the statistics and system catalogs of the target table ○ Release ShareUpdateExclusiveLock ● Remove both unnecessary files and pages of the clog if possible 23
- 24. Important routines ● Vacuum ○ Entry point for vacuum and analyze commands ○ Pgstat_vacuum_stat : gives information about the dead objects - db and relations ● Vacuum_rel ○ Checks ownership of the table underway, skips if not allowed ○ Checks if the relation is vacuum-able - ! matview, !partition_table, !temp_table of other backends, etc. ○ Vacuums toast tables if present ○ Calls cluster (for vacuum full) or heap_vacuum_rel as required 24
- 25. Important routines ● Heap_vacuum_rel ○ Scans the heap and indexes ○ Calls lazy_truncate_heap as required ■ Deletes the last pages of the relation if empty ○ Freeze xid or mxid if required ● Lazy_scan_heap ○ Skip all-visible pages ■ But only when the number of pages to be skipped passes SKIP_PAGES_THRESHOLD ■ Skipping a page also means not updating relfrozenxid! ■ In aggressive mode (freeze), skip only if all-frozen 25
- 26. Important routines ● Lazy_vacuum_indexes ○ Deletes index entries corresponding to dead tuples ● Lazy_cleanup_index ○ Calls index specific cleanup method ○ Update the index stats ● Lazy_vacuum_heap ○ Deletes the tuples only after corresponding index entries are deleted ○ Actually removes dead tuples and compacts the space 26
- 27. Important routines ● Vac_update_datfrozenxid ○ Updates datfrozenxid and datminmxid for the database ○ If one of them got updated, truncate pg_xact and/or pg_multixact ○ Vac_truncate_clog ■ Truncates the clog files and pages ● Analyze_rel ○ Collect the statistics of updated tables ○ Update the pg_statistics, pg_class ○ Collect all unique columns and indexes to be analysed ○ Lower bound for sample size is 100 rows 27
- 28. More to come... ● Zheap ○ In-place updates ○ Older versions of tuples are kept in a separate entity - undo logs ● Block level parallel vacuum (and autovacuum) ○ Parallelize vacuum of a table among workers ● Resume vacuum or autovacuum after interruption ○ Save the blocks that were vacuumed previously and resume from there ● Reduce duplicates in BTree indexes ○ For each key that have duplicates, a list is created which contains the info about the duplicate tuples ○ Improves index bloat 28