Administering Your PostgreSQL

Geodatabase
Michael Downey
Jim McAbee

Esri UC 2015 | Technical Workshop |

 Agenda

• News since the last UC

• How do I …
- Configure Postgres to support geodatabases?
- Create geodatabases?
- Control access to my data?
- Make sure that my data is safe?
- Maintain good performance?

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PostgreSQL
A Free Open Source Option

• Introduction to PostgreSQL
- http://www.postgresql.org/
- Open Source Enterprise level RDBMS
- Free, distributed with bsd license
- Supported by an active online development community
• Possible additional cost for administration guidance
- 3rd Quadrant, Enterprise DB, etc…

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 PostgreSQL and ArcGIS Server for Amazon Web Services

• Geodatabases in PostgreSQL included with ArcGIS for Server Amazon

Machine Image (AMI)
- Includes optional PostgreSQL database cluster (Ubuntu Linux)
- Amazon Relational Data Service (RDS) for PostgreSQL (Ubuntu Linux or Windows)

• When ArcGIS Server Cloud Builder for AWS used to launch

- 2 databases in cluster: egdb and geodata
- egdb – registered as a managed database
- geodata – intended for use as a replicated geodatabase

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 How do I…?
Common questions when working with PostgreSQL
databases and geodatabases

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 How do I…
configure PostgreSQL
to support geodatabases?

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 How do I configure PostgreSQL to support geodatabases?

• Install a supported version of PostgreSQL

PostgreSQL database requirements for ArcGIS 10.3.x
• PostgreSQL configuration parameters
• Enabling Connections to PostgreSQL
• PostgreSQL client libraries for ArcGIS
• ArcGIS Spatial Type libraries for PostgreSQL

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Supported versions at 10.3.x

• PostgreSQL 9.3.5 (64 bit) and PostGIS 2.1 (optional)

• PostgreSQL 9.2.9 (64 bit) and PostGIS 2.0 (optional)
• PostgreSQL 9.1.14 (64 bit) and PostGIS 2.0 (optional)

• Customers using 9.0.5 or earlier need to upgrade.

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 ArcGIS and Postgres Changes since last year
What’s new at 10.3.x

Support for PostgreSQL 9.3.5

Support for PostGIS 2.1

New Geodatabase Administration Geoprocessing Tools

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PostgreSQL Configuration Parameters
• postgresql.conf
- most defaults ok, testing and monitoring should be done
• Memory - #shared_buffers
- Set greater than minimum value of 128KB or 16KB times max_connections…
- Windows – best 64MB to 512MB, little benefit to set higher, use OS cache
- Linux – 25% of Physical Memory to start and as possible (no. of instances)
• Connections
- #max_connections=100 (default)
- one ArcGIS connection = multiple PostgreSQL connections
- Geodatabase connections parameter (sde_server_config) – no longer relevant at 10.3

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 More PostgreSQL Initialization Parameters and Environment Variables
• max_locks_per_transaction
- Most cases default value of 64 is sufficient
- If loading several thousand datasets at once may need to be increased and possible OS
shared memory configuration
• cursor_tuple_fraction
- Set to 0.1 (10% of rows returned rapidly) by default
- ArcGIS 10.2 and later use a setting of 1.0 (set on the stream by ArcSDE), data does not
display until 100% has been returned, cursor_tuple_fraction environment variable set for
session or system used to change, not postgresql.conf
• st_geometry_output_format
- If performaing sql queries (outside ArcGIS) against st_geometry, setting this environment
variable ( = TYPE) will cause the query to return st_geometry vs. WKT (well known text)
which could result in improved performance.
• Environment Variables (PG_HOME)

Esri UC 2015 | Technical Workshop |

 PostgreSQL Client and enabling connections

• ArcGIS 10.3 and 10.3.x for Desktop, Server and Engine

include the required PostgreSQL files (4 library files)
•

• Enable connections
pg_hba.conf

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

ST_Geometry: Default Geometry Datatype

• SQL 3 specification of user-defined data types (UDTs) – ISO and OGC compliant
• Provides structured query language (SQL) access to the geodatabase and database.
• Can be used in PostgreSQL databases that contain a geodatabase and those that do
not.
SDE.SDE_DBTUNE
table for storing keywords
and associated parameters

Administering your PostgreSQL Geodatabase

 Spatial Types and Functions

• Creation of Features through SQL

• Analysis through SQL
• Geodatabase behavior not supported
through SQL

Esri UC 2014 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Configuring PostgreSQL for ST_Geometry spatial type
• Copy st_geometry library for PostgreSQL version to PostgreSQL lib directory
• Necessary for everything except a non-Geodatabase with PostGIS.

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Datatype Mapping

• PostgreSQL supports almost 100 datatypes

• ArcGIS has 8
• Some PostgreSQL datatypes are mapped to
one ArcGIS datatype
• Some datatypes are not supported
- Error: “invalid column datatype”
- Documented in online help

Esri UC 2014 | Technical Workshop | Administering your PostgreSQL Geodatabase

 How do I…
create geodatabases?

Esri UC 2015 | Technical Workshop |

 Databases and Geodatabases

• A database is a PostgreSQL object

- There can be many per PostgreSQL cluster (instance)

• A geodatabase is an ArcGIS construct hosted in a database

- Administrative schema
- One allowed in each database

• Options for creating geodatabases

- Use a GP tool to create a new geodatabase from scratch
- Use a GP tool to create a new geodatabase in an existing database

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Database vs. Geodatabase
Behaviors, Complex Features, Versioning and Distributed Data
Database
• Database provides
- Transaction Management
- Authorization/Security
- Data Management
- Spatial Types
A
- Backup B
C

user Geodatabase
data
• Enterprise Geodatabase is an Postgres Database
with an Administrative Schema providing
- Behaviors
- Complex Features A
A
B
C
B
(Topologies, Networks, Parcel Fabrics, etc…) C

admin user
- Versioning (Long Transactions) objects data
- Distributed Data sde user
schema schema
- Archiving

Esri UC 2015 | Technical Workshop |

 Demo

PostgreSQL
Geodatabase
Configuration
Michael Downey

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PG_HBA file location

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Our example is for local testing on our internal network.

• # TYPE DATABASE USER CIDR-ADDRESS METHOD

• # IPv4 local connections:

• host all all 127.0.0.1/32 trust
• host all all 10.0.0.0/8 md5
• # IPv6 local connections:
• #host all all ::1/128 trust
• host all all ::/0 md5

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PostgreSQL administrator connection in ArcGIS Pro

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 ST_Geometry source location

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 ST_Geometry library destination folder

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Create enterprise geodatabase geoprocessing tool

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Create enterprise geodatabase
geoprocessing tool results

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Create database user tool

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Points to remember

• Upper- and mixed-case object names are not supported for

Geodatabases in PostgreSQL.
- ArcGIS will convert them to lower case.
• Use ArcGIS GP Tools to create and enable geodatabases
- Must connect as postgres superuser to create database
- Python script example available in online help
- SDE command lines tools deprecated at 10.3
• More control over storage?
- Use PostgreSQL tools to create database first
• Enable geodatabase tool
- Create a geodatabase in an existing database
• Do not rename a database that contains a geodatabase

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Upper Case Database Identifiers

• PostgreSQL is case sensitive

- It stores all of it’s object identifies in lower case
- Names of: Databases, Tables, Indexes, Column
- SDE/GDB also stores all identifiers in lower case
- User data can be in any case
- To use identifiers in upper case, they need to be quoted “PGGDB”
- PgAdminIII quotes them automatically
- ArcGIS does not look for quoted strings
- Identifiers with upper case names not found

Esri UC 2014 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Upgrading a geodatabase

• All Application Server connections must be migrated to Direct Connect at 10.3.x

• Direct upgrades from 10, 10.1, 10.2, 10.2.1 or 10.2.2 to 10.3.x are supported if
database at a supported release.
• Upgrading PostgreSQL
- Must have same name and owner, set search_path variable to “$user”, public, and sde.
• Update st_geometry (and optionally libst_raster) libraries to appropriate version.
• Permissions
• Order of operations when upgrading both gdb & sql server?
- Recommended best practices

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Client and Geodatabase PostgreSQL Interoperability

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 How do I…
control access to my data?
Access to PostgreSQL objects are managed with
permissions granted to logins (users) and roles

Esri UC 2015 | Technical Workshop |

 Limit Permissions for Most Users

Admin

Data Owners

Data Editors

Data Readers

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PostgreSQL Principals
A role is an entity that can own database objects and have database privileges; a role can be
considered a "user", a "group", or both depending on how it is used.

• Authentication
- Who is connecting?
- Controlled by pg_hba.conf, specific database user name or group

• Login Roles/Users/Groups = Authorization

- PostgreSQL Login Role = User
- What can this person do in the database?

• Schemas = Containers
- What are logical groups of database objects that should be managed as a whole?

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 User-schema relationship

• For users that create data, ArcGIS requires that

user name = default schema name
- Not a PostgreSQL rule

• Data readers & editors do not need a same-named schema

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Database Users - Logins
• SDE Administrative user, Data Owners, Editors, Viewers
• login and schema must be same name for logins that will own objects in the
geodatabase.
• You cannot create a schema for a group role.
• Can rerun tool to create a schema in a second database
ArcGIS Tools PostgreSQL Tools

Administering your PostgreSQL Geodatabase

PostgreSQL User Permissions
Login Roles (Users) and Group Roles (Groups)

• Geodatabase Administrator (SDE)

- Authorization on SDE Schema
- Usage on all other schemas
• Data Owner/Creators
- Usage on SDE Schema, and any data schemas
where access is required
- Authorization on user’s own schema
• Data Editor
- Usage on SDE schema, and on data schemas to be edited
- Grant editing privileges in Geodatabase through ArcGIS
• Data Viewer
- Usage on SDE schema, and on data schemas to be viewed

Administering your PostgreSQL Geodatabase

 How do I…
make sure my data is safe?

Esri UC 2015 | Technical Workshop |

 Backup
Backup – has it been tested?

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Points to remember

Backups are the only way to reliably protect your data

1. Decide how much time you can afford to lose when disaster strikes
and data must be restored
2. Create a restore plan that will achieve that goal
3. Create a backup plan that supports your restore plan
4. Implement your plan
5. Test your recovery plan regularly by using real backup media to
restore to a system capable of being used in production

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 PostgreSQL Recovery Models

• What are needs

- how often does data change?
- how long can application(s) be down?
- how fast does recovery need to be?
• Weekly or nightly backups
- recovery to most recent backup – most common
• Write-Ahead Logging (WAL)
- point-in-time recovery
- must test thoroughly to understand resource requirements (e.g. disk i/o)

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Backup – pg_dump

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Backup Notes

• While there are several methods for creating backup – for PostgreSQL
Geodatabases pg_dump to create an archive file is the recommended method.
• Recovery would be dropping database and recreating with same name.
• PostGIS has further backup recommendations – see PostGIS’s documentation

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Restore – pg_restore

• You must restore the public schema and its contents first. If you do not, some
of your spatial data will not restore. Second, restore the sde schema. If you do
not do this, spatial indexes will not be re-created on the feature classes.

http://desktop.arcgis.com/en/desktop/latest/manage-data/gdbs-in-
postgresql/restore-geodatabase-postgresql.htm

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Moving or Copying a PostgreSQL Database

• Purposes
- cloning to staging or development environments
- migrating to new VM environment or hardware
• Methods
- via ArcGIS to new database – copy/paste, export/import, etc…
- pg_dump/pg_restore commands
- Restore PUBLIC schema first, then SDE, then data owner schemas
- run as superuser
- run ANALYZE after to update statistics
- re-create Tablespaces
- Text version of a table cannot be larger than 8GB if output to TAR
- use –o option if object identifiers (OIDs) in user-defined data

• Multiple PostgreSQL Instances on same Server

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase
 Multiple Postgres instances on same server
• Create Instance
E:\PostgreSQL\9.2\installer\server>initcluster.vbs postgres postgres gisdata.101
“E:\PostgreSQL\9.2" “E:\PostgreSQL\9.2\pgdata3" 5434 DEFAULT
Usage: initcluster.vbs <OSUsername> <SuperUsername> <Password> <Install dir> <Data dir> <Port> <Locale>

• Create Windows Service

E:\PostgreSQL\9.2\installer\server>startupcfg.vbs 9.2 postgres gisdata.101
“E:\PostgreSQL\9.2" “E:\PostgreSQL\9.2\pgdata3" postgresql-9.2-3
Usage: startupcfg.vbs <Major.Minor version> <Username> <Password> <Install dir> <Data dir> <ServerName>

• Environment Variables available

• PGDATABASE
• PGHOST
• PGPORT
• PGUSER

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 How do I…
maintain good performance?

Esri UC 2015 | Technical Workshop |

 Monitoring: Why monitor?

• Establish performance benchmarks to measure impacts:

- upgrades and patches
- new applications or workflows
- new server resources or deployment patterns
• Assist in troubleshooting
- assist in isolating a problem when one takes place

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 How do I maintain good performance?

• Standard maintenance
- Reconcile/Post/Compress
- Rebuild Indexes
- Calculate Statistics
• Layer scale dependencies

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase

 Performance Considerations

• Excessive normalization
- Too many indexes
- No optimizer hints, index use can not be forced
- Need not worry about the Spatial Index
- GIST index used, self correcting

• Can change postgresql.conf initialization parameters

• Issue with long running ArcGIS edit sessions
- The larger the number of states
- The larger the bloat in indexing belonging to the Feature Class
- Key recommendation – save as often as possible as a best practice, combined with
regular vacuum (improvements coming in next release)

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 PostgreSQL Geodatabase Performance
• Proper Maintenance

• Vacuum
- Have a specific plan – let it run automatically or run manually but this is what is
going to keep your indexes usable if there are large amounts of editing.
- removes dead tuples (rows)
- Autovacuum – on by default, can do an analyze
• Statistics (Analyze)
- Statistics – table size, cardinality of joins, distribution of indexes, etc…
- pg_stat_statements (create extension pg_stat_statements)
- module provides a means for tracking execution statistics of all SQL statements
executed by server.
- shared_preload_libraries = '"E:\\PostgreSQL\\9.2\\lib\\pg_stat_statements.dll"'
• Indexes (Rebuild)

Esri UC 2015 | Technical Workshop | Administering your PostgreSQL Geodatabase

 Monitoring and Performance tools
Many tools to assist in performance analysis during testing
ESRI System Monitor

mxdperfstat

PerfQA Analyzer
ArcGIS Desktop
Esri UC 2015 | Technical Workshop |
 Thank you…

Please fill out the session evaluation

Please note PostgreSQL Performance demo

theater – Wednesday in room 6 at 3:30pm

Online – www.esri.com/ucsessionsurveys
Paper – pick up and put in drop box

Esri UC 2015 | Technical Workshop | Administering Your Postgres Geodatabase