Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Module: PGSQL

Declare, deploy, expose, observe, back up, and manage PostgreSQL clusters with Pigsty v4.5.

PGSQL is Pigsty’s core module. Ansible inventory declares PostgreSQL clusters; Patroni and etcd provide HA orchestration; pgBackRest provides backup and PITR; HAProxy, VIP, DNS, PgBouncer, and the observability stack expose and monitor database services.

This page follows the Pigsty v4.5.0 source layout. Exact defaults live only in the parameter reference so the module landing page does not become a second stale parameter snapshot.

Modeling and Configuration

  • Cluster model: clusters, instances, identity, and roles.
  • Architecture: relationships among Patroni, etcd, service access, and observability.
  • Cluster configuration: primary, replica, offline instance, synchronous commit, standby cluster, delayed cluster, and Citus.
  • Kernel: PostgreSQL major version, distribution, and package selection.
  • Users, databases, HBA, and ACL: business objects and access control.
  • Service access: read-write/read-only services, HAProxy, VIP, DNS, and pooling.
  • Extension catalog: the current 575 packaged extensions and platform coverage.

Deployment and Administration

Task Entry point
Initialize a cluster or add an instance Cluster administration · pgsql.yml
Create or change users User administration · pgsql-user.yml
Create or change databases Database administration · pgsql-db.yml
Change HBA or parameters HBA administration · Component administration
Patroni switchovers, maintenance, and incidents Patroni administration
Install, create, update, or remove extensions Extension administration
Monitor an external instance pgsql-monitor.yml
Prepare a migration Migration · pgsql-migration.yml
Remove an instance or cluster Safe removal workflow · pgsql-rm.yml

Real runs of pgsql.yml, pgsql-user.yml, pgsql-db.yml, and related playbooks change the target environment; pgsql-rm.yml can delete data and backups by default. Resolve the exact cluster/node and recent backup first. Removal also requires the operator to type and confirm the exact target.

Backup and Recovery

Restore is destructive. Production recovery requires an independent recent tested backup and separate gates for shutdown, restore, data validation, timeline promotion, DCS rebuild, replica rebuild, and a fresh full backup.

Monitoring

The current source contains 29 PostgreSQL/PGCAT dashboards under files/grafana/pgsql, covering fleet, cluster, instance, database, table, query, session, transaction, replication, service, PgBouncer, PITR, and alerts.

Parameter Groups

The PGSQL parameter reference is the single documentation source for v4.5.0 defaults and semantics:

  • PG_ID: cluster and instance identity.
  • PG_BUSINESS: users, databases, services, and other business objects.
  • PG_INSTALL: kernel, packages, and extensions.
  • PG_BOOTSTRAP: Patroni bootstrap, replication, and database initialization.
  • PG_PROVISION: in-database objects and privileges.
  • PG_BACKUP: pgBackRest and backup repositories.
  • PG_ACCESS: PgBouncer, services, VIP, and DNS.
  • PG_MONITOR: exporters, monitoring registration, and metrics.
  • PG_REMOVE: removal safeguards and cleanup scope.

Further Reading

1 - Configuration

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Pigsty is a “configuration-driven” PostgreSQL platform: all behaviors come from the combination of inventory files in ~/pigsty/conf/*.yml and PGSQL parameters. Once you’ve written the configuration, you can replicate a customized cluster with instances, users, databases, access control, extensions, and tuning policies in just a few minutes.

Configuration Entry

  1. Prepare Inventory: Copy a pigsty/conf/*.yml template or write an Ansible Inventory from scratch, placing cluster groups (all.children.<cls>.hosts) and global variables (all.vars) in the same file.
  2. Define Parameters: Override the required PGSQL parameters in the vars block. The override order from global → cluster → host determines the final value.
  3. Apply Configuration: Run ./configure -c <conf> or bin/pgsql-add <cls> and other playbooks to apply the configuration. Pigsty will generate the configuration files needed for Patroni/pgbouncer/pgbackrest based on the parameters.

Pigsty’s default demo inventory conf/pgsql.yml is a minimal example: one pg-meta cluster, global pg_version: 18, and a few business user and database definitions. You can expand with more clusters from this base.

Focus Areas & Documentation Index

Pigsty’s PostgreSQL configuration can be organized from the following dimensions. Subsequent documentation will explain “how to configure” each:

  • Cluster & Instances: Define instance topology (standalone, primary-replica, standby cluster, delayed cluster, Citus, etc.) through pg_cluster / pg_role / pg_seq / pg_upstream.
  • Kernel Version: Select the core version, flavor, and tuning templates using pg_version, pg_mode, pg_packages, pg_extensions, pg_conf, and other parameters.
  • Users/Roles: Declare system roles, business accounts, password policies, and connection pool attributes in pg_default_roles and pg_users.
  • Database Objects: Create databases as needed using pg_databases, baseline, schemas, extensions, pool_* fields and automatically integrate with pgbouncer/Grafana.
  • Access Control (HBA): Maintain host-based authentication policies using pg_default_hba_rules and pg_hba_rules to ensure access boundaries for different roles/networks.
  • Privilege Model (ACL): Converge object privileges through pg_default_privileges, pg_default_roles, pg_revoke_public parameters, providing an out-of-the-box layered role system.

After understanding these parameters, you can write declarative inventory manifests as “configuration as infrastructure” for any business requirement. Pigsty will handle execution and ensure idempotency.

A Typical Example

The following snippet shows how to control instance topology, kernel version, extensions, users, and databases in the same configuration file:

all:
  children:
    pg-analytics:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica, pg_offline_query: true }
      vars:
        pg_cluster: pg-analytics
        pg_conf: olap.yml
        pg_extensions: [ postgis, timescaledb, pgvector ]
        pg_databases:
          - { name: bi, owner: dbuser_bi, schemas: [mart], extensions: [timescaledb], pool_mode: session }
        pg_users:
          - { name: dbuser_bi, password: DBUser.BI, roles: [dbrole_admin], pgbouncer: true }
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_hba_rules:
      - { user: dbuser_bi, db: bi, addr: intra, auth: ssl, title: 'BI only allows intranet SSL access' }
  • The pg-analytics cluster contains one primary and one offline replica.
  • Global settings specify pg_version: 18 with a set of extension examples and load olap.yml tuning.
  • Declare business objects in pg_databases and pg_users, automatically generating schema/extension and connection pool entries.
  • Additional pg_hba_rules restrict access sources and authentication methods.

Modify and apply this inventory to get a customized PostgreSQL cluster without manual configuration.

1.1 - Cluster / Instance

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

You can define different types of instances and clusters. Here are several common PostgreSQL instance/cluster types in Pigsty:

  • Primary: Define a single instance cluster.
  • Replica: Define a basic HA cluster with one primary and one replica.
  • Offline: Define an instance dedicated to OLAP/ETL/interactive queries
  • Sync Standby: Enable synchronous commit to ensure no data loss.
  • Quorum Commit: Use quorum sync commit for a higher consistency level.
  • Standby Cluster: Clone an existing cluster and follow it
  • Delayed Cluster: Clone an existing cluster for emergency data recovery
  • Citus Cluster: Define a Citus distributed database cluster

Primary

We start with the simplest case: a single instance cluster consisting of one primary:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test

This configuration is concise and self-describing, consisting only of identity parameters. Matching the Ansible group name to pg_cluster remains convenient for -l pg-test, but it is not a hard membership constraint. Current code discovers actual members from each host’s pg_cluster identity, so one PostgreSQL cluster may span multiple inventory groups.

Use the following command to create this cluster:

bin/pgsql-add pg-test

For demos, development testing, hosting temporary requirements, or performing non-critical analytical tasks, a single database instance may not be a big problem. However, such a single-node cluster has no high availability. When hardware failures occur, you’ll need to use PITR or other recovery methods to ensure the cluster’s RTO/RPO. For this reason, you may consider adding several read-only replicas to the cluster.

Replica

To add a read-only replica instance, you can add a new node to pg-test and set its pg_role to replica.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }  # <--- newly added replica
  vars:
    pg_cluster: pg-test

If the entire cluster doesn’t exist, you can directly create the complete cluster. If the cluster primary has already been initialized, you can add a replica to the existing cluster:

bin/pgsql-add pg-test               # initialize the entire cluster at once
bin/pgsql-add pg-test 10.10.10.12   # add replica to existing cluster

When the cluster primary fails, the read-only instance (Replica) can take over the primary’s work with the help of the high availability system. Additionally, read-only instances can be used to execute read-only queries: many businesses have far more read requests than write requests, and most read-only query loads can be handled by replica instances.

Offline

Offline instances are dedicated read-only replicas specifically for serving slow queries, ETL, OLAP traffic, and interactive queries. Slow queries/long transactions have adverse effects on the performance and stability of online business, so it’s best to isolate them from online business.

To add an offline instance, assign it a new instance and set pg_role to offline.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline }  # <--- newly added offline replica
  vars:
    pg_cluster: pg-test

Dedicated offline instances work similarly to common replica instances, but they serve as backup servers in the pg-test-replica service. That is, only when all replica instances are down will the offline and primary instances provide this read-only service.

In many cases, database resources are limited, and using a separate server as an offline instance is not economical. As a compromise, you can select an existing replica instance and mark it with the pg_offline_query flag to indicate it can handle “offline queries”. In this case, this read-only replica will handle both online read-only requests and offline queries. You can use pg_default_hba_rules and pg_hba_rules for additional access control on offline instances.

Sync Standby

When Sync Standby is enabled, PostgreSQL will select one replica as the sync standby, with all other replicas as candidates. The primary database will wait for the standby instance to flush to disk before confirming commits. The standby instance always has the latest data with no replication lag, and primary-standby switchover to the sync standby will have no data loss.

PostgreSQL uses asynchronous streaming replication by default. If the primary fails, WAL that has not yet replicated may be lost. pg_rpo is Patroni’s sampled lag threshold for failover candidates, not a hard upper bound on actual loss; the real window also depends on write rate, replication state, and Patroni sampling timing.

However, in some critical scenarios (e.g., financial transactions), data loss is completely unacceptable, or read replication lag is unacceptable. In such cases, you can use synchronous commit to solve this problem. To enable sync standby mode, you can simply use the crit.yml template in pg_conf.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: crit.yml   # <--- use crit template

To enable sync standby on an existing cluster, configure the cluster and enable synchronous_mode:

$ pg edit-config pg-test    # run as admin user on admin node
+++
-synchronous_mode: false    # <--- old value
+synchronous_mode: true     # <--- new value
 synchronous_mode_strict: false

Apply these changes? [y/N]: y

In this case, the PostgreSQL configuration parameter synchronous_standby_names is automatically managed by Patroni. One replica will be elected as the sync standby, and its application_name will be written to the PostgreSQL primary configuration file and applied.

Quorum Commit

Quorum Commit provides more powerful control than sync standby: especially when you have multiple replicas, you can set criteria for successful commits, achieving higher/lower consistency levels (and trade-offs with availability).

If you want at least two replicas to confirm commits, you can adjust the synchronous_node_count parameter through Patroni cluster configuration and apply it:

synchronous_mode: true          # ensure synchronous commit is enabled
synchronous_node_count: 2       # specify "at least" how many replicas must successfully commit

If you want to use more sync replicas, modify the synchronous_node_count value. When the cluster size changes, you should ensure this configuration is still valid to avoid service unavailability.

In this case, the PostgreSQL configuration parameter synchronous_standby_names is automatically managed by Patroni.

synchronous_standby_names = '2 ("pg-test-3","pg-test-2")'
Example: Using multiple sync standbys
$ pg edit-config pg-test
---
+synchronous_node_count: 2

Apply these changes? [y/N]: y

After applying the configuration, two sync standbys appear.

+ Cluster: pg-test (7080814403632534854) +---------+----+-----------+-----------------+
| Member    | Host        | Role         | State   | TL | Lag in MB | Tags            |
+-----------+-------------+--------------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.10 | Leader       | running |  1 |           | clonefrom: true |
| pg-test-2 | 10.10.10.11 | Sync Standby | running |  1 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.12 | Sync Standby | running |  1 |         0 | clonefrom: true |
+-----------+-------------+--------------+---------+----+-----------+-----------------+

Another scenario is using any n replicas to confirm commits. In this case, the configuration is slightly different. For example, if we only need any one replica to confirm commits:

synchronous_mode: quorum        # use quorum commit
postgresql:
  parameters:                   # modify PostgreSQL's configuration parameter synchronous_standby_names, using `ANY n ()` syntax
    synchronous_standby_names: 'ANY 1 (*)'  # you can specify a specific replica list or use * to wildcard all replicas.
Example: Enable ANY quorum commit
$ pg edit-config pg-test

+    synchronous_standby_names: 'ANY 1 (*)' # in ANY mode, this parameter is needed
- synchronous_node_count: 2  # in ANY mode, this parameter is not needed

Apply these changes? [y/N]: y

After applying, the configuration takes effect, and all standbys become regular replicas in Patroni. However, in pg_stat_replication, you can see sync_state becomes quorum.

Standby Cluster

You can clone an existing cluster and create a standby cluster for data migration, horizontal splitting, multi-region deployment, or disaster recovery.

Under normal circumstances, the standby cluster will follow the upstream cluster and keep content synchronized. You can promote the standby cluster to become a truly independent cluster.

The standby cluster definition is basically the same as a normal cluster definition, except that the pg_upstream parameter is additionally defined on the primary. The primary of the standby cluster is called the Standby Leader.

For example, below defines a pg-test cluster and its standby cluster pg-test2. The configuration inventory might look like this:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is the standby cluster of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # <--- pg_upstream defined here
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

The primary node pg-test2-1 of the pg-test2 cluster will be a downstream replica of pg-test and serve as the Standby Leader in the pg-test2 cluster.

Just ensure the pg_upstream parameter is configured on the standby cluster’s primary node to automatically pull backups from the original upstream.

bin/pgsql-add pg-test     # create original cluster
bin/pgsql-add pg-test2    # create standby cluster
Example: Change replication upstream

If necessary (e.g., upstream primary-standby switchover/failover), you can change the standby cluster’s replication upstream through cluster configuration.

To do this, simply change standby_cluster.host to the new upstream IP address and apply.

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.13     # <--- old upstream
+  host: 10.10.10.12     # <--- new upstream
   port: 5432

 Apply these changes? [y/N]: y
Example: Promote standby cluster

You can promote the standby cluster to an independent cluster at any time, so the cluster can independently handle write requests and diverge from the original cluster.

To do this, you must configure the cluster and completely erase the standby_cluster section, then apply.

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y
Example: Cascade replication

If you specify pg_upstream on a replica instead of the primary, you can configure cascade replication for the cluster.

When configuring cascade replication, you must use the IP address of an instance in the cluster as the parameter value, otherwise initialization will fail. The replica performs streaming replication from a specific instance rather than the primary.

The instance acting as a WAL relay is called a Bridge Instance. Using a bridge instance can share the burden of sending WAL from the primary. When you have dozens of replicas, using bridge instance cascade replication is a good idea.

pg-test:
 hosts: # pg-test-1 ---> pg-test-2 ---> pg-test-3
   10.10.10.11: { pg_seq: 1, pg_role: primary }
   10.10.10.12: { pg_seq: 2, pg_role: replica } # <--- bridge instance
   10.10.10.13: { pg_seq: 3, pg_role: replica, pg_upstream: 10.10.10.12 }
   # ^--- replicate from pg-test-2 (bridge) instead of pg-test-1 (primary)
 vars: { pg_cluster: pg-test }

Delayed Cluster

A Delayed Cluster is a special type of standby cluster used to quickly recover “accidentally deleted” data.

For example, if you want a cluster named pg-testdelay whose data content is the same as the pg-test cluster from one hour ago:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-testdelay is the delayed cluster of pg-test
pg-testdelay:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11, pg_delay: 1d }
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-testdelay }

You can also configure a “replication delay” on an existing standby cluster.

$ pg edit-config pg-testdelay
 standby_cluster:
   create_replica_methods:
   - basebackup
   host: 10.10.10.11
   port: 5432
+  recovery_min_apply_delay: 1h    # <--- add delay duration here, e.g. 1 hour

Apply these changes? [y/N]: y

When some tuples and tables are accidentally deleted, you can modify this parameter to advance this delayed cluster to an appropriate point in time, read data from it, and quickly fix the original cluster.

Delayed clusters require additional resources, but are much faster than PITR and have much less impact on the system. For very critical clusters, consider setting up delayed clusters.

Citus Cluster

Pigsty natively supports Citus. You can refer to conf/ha/citus.yml as a complete example.

To define a Citus cluster, you need to specify the following parameters:

  • pg_mode must be set to citus, not the default pgsql
  • The shard name pg_shard and shard number pg_group must be defined on each shard cluster
  • pg_primary_db must be defined to specify the database managed by Patroni.
  • If you want to use pg_dbsu postgres instead of the default pg_admin_username to execute admin commands, then pg_dbsu_password must be set to a non-empty plaintext password

Additionally, extra hba rules are needed to allow SSL access from localhost and other data nodes. As shown below:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # if using dbsu, need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

On the coordinator node, you can create distributed tables and reference tables and query them from any data node. Starting from 11.2, any Citus database node can act as a coordinator.

SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table($$public.pgbench_accounts$$);
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_branches$$);
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_history$$);
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_tellers$$);

1.2 - Kernel Version

How to choose the appropriate PostgreSQL kernel and major version.

Choosing a “kernel” in Pigsty means determining the PostgreSQL major version, mode/distribution, packages to install, and tuning templates to load.

The Pigsty v4.5 source currently supports PostgreSQL 14-18 and uses 18 by default. The following content shows how to make these choices through configuration files.

Major Version and Packages

  • pg_version: Specify the PostgreSQL major version (default 18). Pigsty will automatically map to the correct package name prefix based on the version.
  • pg_packages: Define the core package set to install, supports using package aliases (default pgsql-main pgsql-common, includes kernel + patroni/pgbouncer/pgbackrest and other common tools).
  • pg_extensions: List of additional extension packages to install, also supports aliases; defaults to empty meaning only core dependencies are installed.
all:
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ postgis, timescaledb, pgvector, pgml ]

Effect: Ansible will pull packages corresponding to pg_version=18 during installation, pre-install extensions to the system, and database initialization scripts can then directly CREATE EXTENSION.

Extension support varies across versions in Pigsty’s offline repository: 14 has relatively fewer available extensions, while 17/18 have the broadest coverage. If an extension is not pre-packaged, it can be added via repo_extra_packages.

Kernel Mode (pg_mode)

pg_mode controls the kernel “flavor” to deploy. Default pgsql indicates standard PostgreSQL. Pigsty currently supports the following modes:

Mode Scenario
pgsql Standard PostgreSQL, HA + replication
citus Citus distributed cluster, requires additional pg_shard / pg_group
gpsql Cloudberry / Greenplum / MatrixDB
mssql Babelfish
mysql OpenGauss/HaloDB compatible with MySQL protocol
polar Alibaba PolarDB (based on pg polar distribution)
ivory IvorySQL (Oracle-compatible syntax)
pgtde Percona PostgreSQL with pg_tde under /usr/pgtde-$v
oriole OrioleDB storage engine
agens AgensGraph graph database kernel
pgedge pgEdge distributed replication kernel

pg_mode determines binary paths, Patroni integration, and some kernel-specific logic; it does not automatically add every required package, extension, and business database. Use the matching conf/*.yml template in real deployments, or explicitly configure pg_packages, pg_extensions, pg_libs, and pg_databases. Here is a minimal Citus example:

all:
  children:
    pg-citus1:
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1, pg_group: 0 }
    pg-citus2:
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2, pg_group: 1 }
  vars:
    pg_mode: citus
    pg_shard: pg-citus
    pg_primary_db: citus
    pg_extensions: [ citus ]
    pg_libs: 'citus, pg_stat_statements'
    pg_databases:
      - { name: citus, extensions: [ citus ] }

conf/ha/citus.yml provides the current complete example. The minimal configuration above explicitly installs Citus packages and creates the extension in the citus database.

Extensions and Pre-installed Objects

Besides system packages, you can control components automatically loaded after database startup through the following parameters:

  • pg_libs: List to write to shared_preload_libraries. For example: pg_libs: 'timescaledb, pg_stat_statements, auto_explain'.
  • pg_default_extensions / pg_default_schemas: Control schemas and extensions pre-created in template1 and postgres by initialization scripts.
  • pg_parameters: Rendered by Pigsty into postgresql.auto.conf during configuration. Do not also manage the same settings manually with ALTER SYSTEM.

Example: Enable TimescaleDB, pgvector and customize some system parameters.

pg-analytics:
  vars:
    pg_cluster: pg-analytics
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_default_extensions:
      - { name: timescaledb }
      - { name: vector }
    pg_parameters:
      timescaledb.max_background_workers: 8

Effect: During initialization, default extensions are created in template1 and postgres; newly created databases based on template1 inherit those objects. pg_parameters is written directly to postgresql.auto.conf.

Tuning Template (pg_conf)

pg_conf points to Patroni templates in roles/pgsql/templates/*.yml. Pigsty includes four built-in general templates:

Template Applicable Scenario
oltp.yml Default template, for 4–128 core TP workload
olap.yml Optimized for analytical scenarios
crit.yml Emphasizes sync commit/minimal latency, suitable for zero-loss scenarios like finance
tiny.yml Lightweight machines / edge scenarios / resource-constrained environments

You can directly replace the template or customize a YAML file in templates/, then specify it in cluster vars.

pg-ledger:
  hosts: { 10.10.10.21: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-ledger
    pg_conf: crit.yml
    pg_parameters:
      synchronous_commit: 'remote_apply'
      max_wal_senders: 16
      wal_keep_size: '2GB'

Effect: Copy crit.yml as Patroni configuration, overlay pg_parameters written to postgresql.auto.conf, making instances run immediately in synchronous commit mode.

Combined Instance: A Complete Example 

pg-rag:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
    10.10.10.32: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-rag
    pg_version: 18
    pg_mode: pgsql
    pg_conf: olap.yml
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ pgvector, pgml, postgis ]
    pg_libs: 'pg_stat_statements, auto_explain'
    pg_parameters:
      max_parallel_workers: 8
      shared_buffers: '32GB'
  • First primary + one replica, using olap.yml tuning.
  • Install PG18 plus common RAG extensions; only libraries that actually require preloading belong in pg_libs.
  • Patroni/pgbouncer/pgbackrest generated by Pigsty, no manual intervention needed.

Replace the above parameters according to business needs to complete all kernel-level customization.

1.3 - Package Alias

Pigsty provides a package alias translation mechanism that shields the differences in binary package details across operating systems, making installation easier.

PostgreSQL package naming conventions vary significantly across different operating systems:

  • EL systems (RHEL/Rocky/Alma/…) use formats like pgvector_18, postgis36_18*
  • Debian/Ubuntu systems use formats like postgresql-18-pgvector, postgresql-18-postgis-3

This difference adds cognitive burden to users: you need to remember different package name rules for different systems, and handle the embedding of PostgreSQL version numbers.

Package Alias

Pigsty solves this problem through the Package Alias mechanism: you only need to use unified aliases, and Pigsty will handle all the details:

# Using aliases - simple, unified, cross-platform
pg_extensions: [ postgis, pgvector, timescaledb ]

# Equivalent to actual package names on EL9 + PG18
pg_extensions: [ postgis36_18*, pgvector_18*, timescaledb-tsl_18* ]

# Equivalent to actual package names on Ubuntu 24 + PG18
pg_extensions: [ postgresql-18-postgis-3, postgresql-18-pgvector, postgresql-18-timescaledb-tsl ]

Alias Translation

Aliases can also group a set of packages as a whole. For example, Pigsty’s default installed packages - the default value of pg_packages is:

pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common

Pigsty will query the current operating system alias list (assuming el10.x86_64) and translate it to PGSQL kernel, extensions, and toolkits:

pgsql-main:    "postgresql$v postgresql$v-server postgresql$v-libs postgresql$v-contrib postgresql$v-plperl postgresql$v-plpython3 postgresql$v-pltcl postgresql$v-llvmjit pg_repack_$v* wal2json_$v* pgvector_$v*"
pgsql-common:  "patroni patroni-etcd pgbouncer pgbackrest pg_exporter pgbackrest_exporter vip-manager"

Next, Pigsty further translates pgsql-main using the currently specified PG major version (assuming pg_version = 18):

pg18-main:   "postgresql18 postgresql18-server postgresql18-libs postgresql18-contrib postgresql18-plperl postgresql18-plpython3 postgresql18-pltcl postgresql18-llvmjit pg_repack_18* wal2json_18* pgvector_18*"

Through this approach, Pigsty shields the complexity of packages, allowing users to simply specify the functional components they want.

Which Variables Can Use Aliases?

You can use package aliases in the following four parameters, and the aliases will be automatically converted to actual package names according to the translation process:

Alias List

You can find the alias mapping files for each operating system and architecture in the roles/node_id/vars/ directory of the Pigsty project source code:

How It Works

Alias Translation Process 

User config alias --> Detect OS -->  Find alias mapping table ---> Replace $v placeholder ---> Install actual packages
     ↓                 ↓                   ↓                                   ↓
  postgis          el9.x86_64         postgis36_$v*                   postgis36_18*
  postgis          u24.x86_64         postgresql-$v-postgis-3         postgresql-18-postgis-3

Version Placeholder

Pigsty’s alias system uses $v as a placeholder for the PostgreSQL version number. When you specify a PostgreSQL version using pg_version, all $v in aliases will be replaced with the actual version number.

For example, when pg_version: 18:

Alias Definition (EL) Expanded Result
postgresql$v* postgresql18*
pgvector_$v* pgvector_18*
timescaledb-tsl_$v* timescaledb-tsl_18*
Alias Definition (Debian/Ubuntu) Expanded Result
postgresql-$v postgresql-18
postgresql-$v-pgvector postgresql-18-pgvector
postgresql-$v-timescaledb-tsl postgresql-18-timescaledb-tsl

Wildcard Matching

On EL systems, many aliases use the * wildcard to match related subpackages. For example:

  • postgis36_18* will match postgis36_18, postgis36_18-client, postgis36_18-utils, etc.
  • postgresql18* will match postgresql18, postgresql18-server, postgresql18-libs, postgresql18-contrib, etc.

This design ensures you don’t need to list each subpackage individually - one alias can install the complete extension.

1.4 - User/Role

How to define and customize PostgreSQL users and roles through configuration?

In this document, “user” refers to a logical object within a database cluster created with CREATE USER/ROLE.

In PostgreSQL, users belong directly to the database cluster rather than a specific database. Therefore, when creating business databases and users, follow the principle of “users first, databases later”.

Pigsty defines roles and users through two config parameters:

The former defines roles/users shared across the entire environment; the latter defines business roles/users specific to a single cluster. Both have the same format as arrays of user definition objects. Users/roles are created sequentially in array order, so later users can belong to roles defined earlier.

By default, all users marked with pgbouncer: true are added to the Pgbouncer connection pool user list.

Define Users

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
      - {name: dbuser_remove   ,state: absent }  # use state: absent to delete user

Each user/role definition is a complex object. Only name is required:

- name: dbuser_meta               # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, user state: create (default), absent
  password: DBUser.Meta           # Optional, password, can be scram-sha-256 hash or plaintext
  login: true                     # Optional, can login, default true
  superuser: false                # Optional, is superuser, default false
  createdb: false                 # Optional, can create databases, default false
  createrole: false               # Optional, can create roles, default false
  inherit: true                   # Optional, inherit role privileges, default true
  replication: false              # Optional, can replicate, default false
  bypassrls: false                # Optional, bypass row-level security, default false
  connlimit: -1                   # Optional, connection limit, default -1 (unlimited)
  expire_in: 3650                 # Optional, expire N days from creation (priority over expire_at)
  expire_at: '2030-12-31'         # Optional, expiration date in YYYY-MM-DD format
  comment: pigsty admin user      # Optional, user comment
  roles: [dbrole_admin]           # Optional, roles array
  parameters:                     # Optional, role-level config params
    search_path: public
  pgbouncer: true                 # Optional, add to connection pool user list, default false
  pool_mode: transaction          # Optional, pgbouncer pool mode, default transaction
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).

Parameter Overview

The only required field is name - a valid, unique username within the cluster. All other params have sensible defaults.

Field Category Type Attr Description
name Basic string Required Username, must be valid and unique
state Basic enum Optional State: create (default), absent
password Basic string Mutable User password, plaintext or hash
comment Basic string Mutable User comment
login Privilege bool Mutable Can login, default true
superuser Privilege bool Mutable Is superuser, default false
createdb Privilege bool Mutable Can create databases, default false
createrole Privilege bool Mutable Can create roles, default false
inherit Privilege bool Mutable Inherit role privileges, default true
replication Privilege bool Mutable Can replicate, default false
bypassrls Privilege bool Mutable Bypass RLS, default false
connlimit Privilege int Mutable Connection limit, -1 unlimited
expire_in Validity int Mutable Expire N days from now (priority)
expire_at Validity string Mutable Expiration date, YYYY-MM-DD format
roles Role array Additive Roles array, string or object format
parameters Params object Mutable Role-level parameters
pgbouncer Pool bool Mutable Add to connection pool, default false
pool_mode Pool enum Mutable Pool mode: transaction (default)
pool_connlimit Pool int Mutable Pool user max connections

Parameter Details

name

String, required. Username - must be unique within the cluster.

Must be a valid PostgreSQL identifier matching ^[a-z_][a-z0-9_]{0,62}$: starts with lowercase letter or underscore, contains only lowercase letters, digits, underscores, max 63 chars.

- name: dbuser_app         # Standard naming
- name: app_readonly       # Underscore separated
- name: _internal          # Underscore prefix (for internal roles)

state

Enum for user operation: create or absent. Default create.

State Description
create Default, create user, update if exists
absent Delete user with DROP ROLE 
- name: dbuser_app             # state defaults to create
- name: dbuser_old
  state: absent                # Delete user

These system users cannot be deleted via state: absent (to prevent cluster failure):

password

String, mutable. User password - users without password can’t login via password auth.

Password can be:

Format Example Description
Plaintext DBUser.Meta Not recommended, logged to config
SCRAM-SHA-256 SCRAM-SHA-256$4096:xxx$yyy:zzz Recommended, PG10+ default
MD5 hash md5... Legacy compatibility 
# Plaintext (not recommended, logged to config)
- name: dbuser_app
  password: MySecretPassword

# SCRAM-SHA-256 hash (recommended)
- name: dbuser_app
  password: 'SCRAM-SHA-256$4096:xxx$yyy:zzz'

When setting password, Pigsty temporarily disables logging to prevent leakage:

SET log_statement TO 'none';
ALTER USER "dbuser_app" PASSWORD 'xxx';
SET log_statement TO DEFAULT;

To generate SCRAM-SHA-256 hash:

# Using PostgreSQL (requires pgcrypto extension)
psql -c "SELECT encode(digest('password' || 'username', 'sha256'), 'hex')"

comment

String, mutable. User comment, defaults to business user {name}.

Set via COMMENT ON ROLE, supports special chars (quotes auto-escaped).

- name: dbuser_app
  comment: 'Main business application account'
COMMENT ON ROLE "dbuser_app" IS 'Main business application account';

login

Boolean, mutable. Can login, default true.

Setting false creates a Role rather than User - typically for permission grouping.

In PostgreSQL, CREATE USER equals CREATE ROLE ... LOGIN.

# Create login-able user
- name: dbuser_app
  login: true

# Create role (no login, for permission grouping)
- name: dbrole_custom
  login: false
  comment: custom permission role
CREATE USER "dbuser_app" LOGIN;
CREATE USER "dbrole_custom" NOLOGIN;

superuser

Boolean, mutable. Is superuser, default false.

Superusers have full database privileges, bypassing all permission checks.

- name: dbuser_admin
  superuser: true            # Dangerous: full privileges
ALTER USER "dbuser_admin" SUPERUSER;

Pigsty provides default superuser via pg_admin_username (dbuser_dba). Don’t create additional superusers unless necessary.

createdb

Boolean, mutable. Can create databases, default false.

- name: dbuser_dev
  createdb: true             # Allow create database
ALTER USER "dbuser_dev" CREATEDB;

Some applications (Gitea, Odoo, etc.) may require CREATEDB privilege for their admin users.

createrole

Boolean, mutable. Can create other roles, default false.

Users with CREATEROLE can create, modify, delete other non-superuser roles.

- name: dbuser_admin
  createrole: true           # Allow manage other roles
ALTER USER "dbuser_admin" CREATEROLE;

inherit

Boolean, mutable. Auto-inherit privileges from member roles, default true.

Setting false requires explicit SET ROLE to use member role privileges.

# Auto-inherit role privileges (default)
- name: dbuser_app
  inherit: true
  roles: [dbrole_readwrite]

# Requires explicit SET ROLE
- name: dbuser_special
  inherit: false
  roles: [dbrole_admin]
ALTER USER "dbuser_special" NOINHERIT;
-- User must execute SET ROLE dbrole_admin to get privileges

replication

Boolean, mutable. Can initiate streaming replication, default false.

Usually only replication users (replicator) need this. Normal users shouldn’t have it unless for logical decoding subscriptions.

- name: replicator
  replication: true          # Allow streaming replication
  roles: [pg_monitor, dbrole_readonly]
ALTER USER "replicator" REPLICATION;

bypassrls

Boolean, mutable. Bypass row-level security (RLS) policies, default false.

When enabled, user can access all rows even with RLS policies. Usually only for admins.

- name: dbuser_myappadmin
  bypassrls: true            # Bypass RLS policies
ALTER USER "dbuser_myappadmin" BYPASSRLS;

connlimit

Integer, mutable. Max concurrent connections, default -1 (unlimited).

Positive integer limits max simultaneous sessions for this user. Doesn’t affect superusers.

- name: dbuser_app
  connlimit: 100             # Max 100 concurrent connections

- name: dbuser_batch
  connlimit: 10              # Limit batch user connections
ALTER USER "dbuser_app" CONNECTION LIMIT 100;

expire_in

Integer, mutable. Expire N days from current date.

This param has higher priority than expire_at. Expiration recalculated on each playbook run - good for temp users needing periodic renewal.

- name: temp_user
  expire_in: 30              # Expire in 30 days

- name: contractor_user
  expire_in: 90              # Expire in 90 days

Generates SQL:

-- expire_in: 30, assuming current date is 2025-01-01
ALTER USER "temp_user" VALID UNTIL '2025-01-31';

expire_at

String, mutable. Expiration date in YYYY-MM-DD format, or special value infinity.

Lower priority than expire_in. Use infinity for never-expiring users.

- name: contractor_user
  expire_at: '2024-12-31'    # Expire on specific date

- name: permanent_user
  expire_at: 'infinity'      # Never expires
ALTER USER "contractor_user" VALID UNTIL '2024-12-31';
ALTER USER "permanent_user" VALID UNTIL 'infinity';

roles

Array, additive. Roles this user belongs to. Elements can be strings or objects.

Simple format - strings for role names:

- name: dbuser_app
  roles:
    - dbrole_readwrite
    - pg_read_all_data
GRANT "dbrole_readwrite" TO "dbuser_app";
GRANT "pg_read_all_data" TO "dbuser_app";

Full format - objects for fine-grained control:

- name: dbuser_app
  roles:
    - dbrole_readwrite                            # Simple string: GRANT role
    - { name: dbrole_admin, admin: true }         # WITH ADMIN OPTION
    - { name: pg_monitor, set: false }            # PG16+: disallow SET ROLE
    - { name: pg_signal_backend, inherit: false } # PG16+: don't auto-inherit
    - { name: old_role, state: absent }           # Revoke role membership

Object Format Parameters:

Param Type Description
name string Role name (required)
state enum grant (default) or absent/revoke: control membership
admin bool true: WITH ADMIN OPTION, false: REVOKE ADMIN
set bool PG16+: true: WITH SET TRUE, false: REVOKE SET
inherit bool PG16+: true: WITH INHERIT TRUE, false: REVOKE INHERIT

PostgreSQL 16+ New Features:

PostgreSQL 16 introduced finer-grained role membership control:

  • ADMIN OPTION: Allow granting role to other users
  • SET OPTION: Allow using SET ROLE to switch to this role
  • INHERIT OPTION: Auto-inherit this role’s privileges
# PostgreSQL 16+ complete example
- name: dbuser_app
  roles:
    # Normal membership
    - dbrole_readwrite

    # Can grant dbrole_admin to other users
    - { name: dbrole_admin, admin: true }

    # Cannot SET ROLE to pg_monitor (only inherit privileges)
    - { name: pg_monitor, set: false }

    # Don't auto-inherit pg_execute_server_program (need explicit SET ROLE)
    - { name: pg_execute_server_program, inherit: false }

    # Revoke old_role membership
    - { name: old_role, state: absent }

set and inherit options only work in PG16+. On earlier versions they’re ignored with warning comments.

parameters

Object, mutable. Role-level config params via ALTER ROLE ... SET. Applies to all sessions for this user.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: 'all'
ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';
ALTER USER "dbuser_analyst" SET "search_path" = 'analytics,public';
ALTER USER "dbuser_analyst" SET "log_statement" = 'all';

Use special value DEFAULT (case-insensitive) to reset to PostgreSQL default:

- name: dbuser_app
  parameters:
    work_mem: DEFAULT          # Reset to default
    statement_timeout: '30s'   # Set new value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

Common role-level params:

Parameter Description Example
work_mem Query work memory '64MB'
statement_timeout Statement timeout '30s'
lock_timeout Lock wait timeout '10s'
idle_in_transaction_session_timeout Idle transaction timeout '10min'
search_path Schema search path 'app,public'
log_statement Log level 'ddl'
temp_file_limit Temp file size limit '10GB'

Query user-level params via pg_db_role_setting system view.

pgbouncer

Boolean, mutable. Add user to Pgbouncer user list, default false.

For prod users needing connection pool access, must explicitly set pgbouncer: true. Default false prevents accidentally exposing internal users to the pool.

# Prod user: needs connection pool
- name: dbuser_app
  password: DBUser.App
  pgbouncer: true

# Internal user: no connection pool needed
- name: dbuser_internal
  password: DBUser.Internal
  pgbouncer: false           # Default, can be omitted

Users with pgbouncer: true are added to /etc/pgbouncer/userlist.txt.

pool_mode

Enum, mutable. User-level pool mode: transaction, session, or statement. Default transaction.

Mode Description Use Case
transaction Return connection after txn Most OLTP apps, default
session Return connection after session Apps needing session state
statement Return after each statement Simple stateless queries 
# DBA user: session mode (may need SET commands etc.)
- name: dbuser_dba
  pgbouncer: true
  pool_mode: session

# Normal business user: transaction mode
- name: dbuser_app
  pgbouncer: true
  pool_mode: transaction

User-level pool params are configured via /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

pool_connlimit

Integer, mutable. User-level maximum pool connections. If omitted, no user-level override is generated and Pigsty’s global pgbouncer.ini default of 100 applies. PgBouncer uses 0 to mean unlimited.

- name: dbuser_app
  pgbouncer: true
  pool_connlimit: 50         # Max 50 pool connections for this user

ACL System

Pigsty provides a built-in access control / ACL model. Assign these default business roles to users as required:

Role Privileges Typical Use Case
dbrole_readwrite Global read-write Primary application accounts
dbrole_readonly Global read-only Read-only application access
dbrole_admin DDL privileges Application administrators and table creation
dbrole_offline Independent read-only; instance scope controlled by HBA Ad hoc users, ETL, and analytics 
# Typical business user configuration
pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]    # Prod account, read-write

  - name: dbuser_readonly
    password: DBUser.Readonly
    pgbouncer: true
    roles: [dbrole_readonly]     # Read-only account

  - name: dbuser_admin
    password: DBUser.Admin
    pgbouncer: true
    roles: [dbrole_admin]        # Admin, can execute DDL

  - name: dbuser_etl
    password: DBUser.ETL
    roles: [dbrole_offline]      # Analytics account; restrict instance scope through HBA

dbrole_offline does not itself restrict a user to offline instances. To establish that boundary, set role: offline on the corresponding HBA rule; see Offline Role and Instance Isolation.

To redesign your own ACL system, customize:

Pgbouncer Users

Pgbouncer is enabled by default as connection pool middleware. Pigsty adds all users in pg_users with explicit pgbouncer: true flag to the pgbouncer user list.

Users in connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="

User-level pool params are maintained in /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

When creating users, Pgbouncer user list is refreshed via online reload - doesn’t affect existing connections.

Pgbouncer runs as same dbsu as PostgreSQL (default postgres OS user). Use pgb alias to access pgbouncer admin functions.

pgbouncer_auth_query param allows dynamic query for pool user auth - convenient when you prefer not to manually manage pool users.

Related Resources

For user management operations, see User Management.

For user access privileges, see Access Control: Role System.

1.5 - Database

How to define and customize PostgreSQL databases through configuration?

In this document, “database” refers to a logical object within a database cluster created with CREATE DATABASE.

A PostgreSQL cluster can serve multiple databases simultaneously. In Pigsty, you can define required databases in cluster configuration.

Pigsty customizes the template1 template database - creating default schemas, installing default extensions, configuring default privileges. Newly created databases inherit these settings from template1. You can also specify other template databases via template for instant database cloning.

By default, all business databases are 1:1 added to Pgbouncer connection pool; pg_exporter auto-discovers all business databases for in-database object monitoring. All databases are also registered as PostgreSQL datasources in Grafana on all INFRA nodes for PGCAT dashboards.

Define Database

Business databases are defined in cluster param pg_databases, an array of database definition objects. During cluster initialization, databases are created in definition order, so later databases can use earlier ones as templates.

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition is a complex object with fields below. Only name is required:

- name: meta                      # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, database state: create (default), absent, recreate
  baseline: cmdb.sql              # Optional, SQL baseline file path (relative to Ansible search path, e.g., files/)
  pgbouncer: true                 # Optional, add to pgbouncer database list? default true
  schemas: [pigsty]               # Optional, additional schemas to create, array of schema names
  extensions:                     # Optional, extensions to install: array of extension objects
    - { name: postgis , schema: public }  # Can specify schema, or omit (installs to first schema in search_path)
    - { name: timescaledb }               # Some extensions create and use fixed schemas
  comment: pigsty meta database   # Optional, database comment/description
  owner: postgres                 # Optional, database owner, defaults to current user
  template: template1             # Optional, template to use, default template1
  strategy: FILE_COPY             # Optional, clone strategy: FILE_COPY or WAL_LOG (PG15+)
  encoding: UTF8                  # Optional, inherits from template/cluster config (UTF8)
  locale: C                       # Optional, inherits from template/cluster config (C)
  lc_collate: C                   # Optional, inherits from template/cluster config (C)
  lc_ctype: C                     # Optional, inherits from template/cluster config (C)
  locale_provider: libc           # Optional, locale provider: libc, icu, builtin (PG15+)
  icu_locale: en-US               # Optional, ICU locale rules (PG15+)
  icu_rules: ''                   # Optional, ICU collation rules (PG16+)
  builtin_locale: C.UTF-8         # Optional, builtin locale provider rules (PG17+)
  tablespace: pg_default          # Optional, default tablespace
  is_template: false              # Optional, mark as template database
  allowconn: true                 # Optional, allow connections, default true
  revokeconn: false               # Optional; when true, retain CONNECT only for owner, admin, monitor, and replication users
  register_datasource: true       # Optional, register to grafana datasource? default true
  connlimit: -1                   # Optional, connection limit, -1 means unlimited
  parameters:                     # Optional, database-level params via ALTER DATABASE SET
    work_mem: '64MB'
    statement_timeout: '30s'
  pool_auth_user: dbuser_meta     # Optional, auth user for pgbouncer auth_query
  pool_mode: transaction          # Optional, database-level pgbouncer pool mode
  pool_size: 50                   # Optional, database-level pgbouncer default pool size
  pool_reserve: 30                # Optional, database-level pgbouncer reserve pool
  pool_size_min: 0                # Optional, database-level pgbouncer min pool size
  pool_connlimit: 100             # Optional, database-level max database connections

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Parameter Overview

The only required field is name - a valid, unique database name within the cluster. All other params have sensible defaults. Parameters marked “Immutable” only take effect at creation; changing them requires database recreation.

Field Category Type Attr Description
name Basic string Required Database name, must be valid and unique
state Basic enum Optional State: create (default), absent, recreate
owner Basic string Mutable Database owner, defaults to postgres
comment Basic string Mutable Database comment
template Template string Immutable Template database, default template1
strategy Template enum Immutable Clone strategy: FILE_COPY or WAL_LOG (PG15+)
encoding Encoding string Immutable Character encoding, default inherited (UTF8)
locale Encoding string Immutable Locale setting, default inherited (C)
lc_collate Encoding string Immutable Collation rule, default inherited (C)
lc_ctype Encoding string Immutable Character classification, default inherited (C)
locale_provider Encoding enum Immutable Locale provider: libc, icu, builtin (PG15+)
icu_locale Encoding string Immutable ICU locale rules (PG15+)
icu_rules Encoding string Immutable ICU collation customization (PG16+)
builtin_locale Encoding string Immutable Builtin locale rules (PG17+)
tablespace Storage string Mutable Default tablespace, change triggers data migration
is_template Privilege bool Mutable Mark as template database
allowconn Privilege bool Mutable Allow connections, default true
revokeconn Privilege bool Mutable Revoke PUBLIC CONNECT privilege
connlimit Privilege int Mutable Connection limit, -1 for unlimited
baseline Init string Mutable SQL baseline file path, runs on every provisioning
schemas Init (string|object)[] Mutable Schema definitions to create
extensions Init (string|object)[] Mutable Extension definitions to install
parameters Init object Mutable Database-level parameters
pgbouncer Pool bool Mutable Add to connection pool, default true
pool_mode Pool enum Mutable Pool mode: transaction (default)
pool_size Pool int Mutable Default pool size, default 50
pool_size_min Pool int Mutable Min pool size, default 0
pool_reserve Pool int Mutable Reserve pool size, default 30
pool_connlimit Pool int Mutable Max database connections, default 100
pool_auth_user Pool string Mutable Auth query user
register_datasource Monitor bool Mutable Register to Grafana datasource, default true

Parameter Details

name

String, required. Database name - must be unique within the cluster.

The current role does not enforce this regular expression, and SQL identifiers are double-quoted. However, the name is also used in temporary file paths and shell/SQL command assembly. For safe operation across the entire automation chain, keep it within 63 bytes, follow ^[A-Za-z_][A-Za-z0-9_$]{0,62}$, and avoid spaces, quotes, slashes, or other special characters.

- name: myapp              # Simple naming
- name: my_application     # Underscore separated
- name: app_v2             # Version included

state

Enum for database operation: create, absent, or recreate. Default create.

State Description
create Default, create or modify database, adjust mutable params if exists
absent Delete database with DROP DATABASE WITH (FORCE)
recreate Drop then create, for database reset 
- name: myapp                # state defaults to create
- name: olddb
  state: absent              # Delete database
- name: testdb
  state: recreate            # Rebuild database

owner

String. Database owner, defaults to pg_dbsu (postgres) if not specified.

Target user must exist. Changing owner executes (old owner retains existing privileges):

Database owner has full control including creating schemas, tables, extensions - useful for multi-tenant scenarios.

ALTER DATABASE "myapp" OWNER TO "new_owner";
GRANT ALL PRIVILEGES ON DATABASE "myapp" TO "new_owner";

comment

String. Database comment, defaults to business database {name}.

Set via COMMENT ON DATABASE, supports Chinese and special characters (Pigsty auto-escapes quotes). Stored in the shared-object comment catalog pg_shdescription, viewable via \l+.

COMMENT ON DATABASE "myapp" IS 'my main application database';
- name: myapp
  comment: my main application database

template

String, immutable. Template database for creation, default template1.

PostgreSQL’s CREATE DATABASE clones the template - new database inherits all objects, extensions, schemas, permissions. Pigsty customizes template1 during cluster init, so new databases inherit these settings.

Template Description
template1 Default, includes Pigsty pre-configured extensions/schemas/perms
template0 Clean template, required for non-default locale providers
Custom database Use existing database as template for cloning

When using icu or builtin locale provider, must specify template: template0 since template1 locale settings can’t be overridden.

- name: myapp_icu
  template: template0        # Required for ICU
  locale_provider: icu
  icu_locale: zh-Hans

Using template0 skips monitoring extensions/schemas and default privileges - allowing fully custom database.

strategy

Enum, immutable. Clone strategy: FILE_COPY or WAL_LOG. Available PG15+.

Strategy Description Use Case
FILE_COPY Direct file copy with checkpoints before and after Large templates, lower WAL volume
WAL_LOG Block-by-block copy written to WAL; PG15+ default Small templates, non-blocking

WAL_LOG doesn’t block template connections during clone but less efficient for large templates. Ignored on PG14 and earlier.

- name: cloned_db
  template: source_db
  strategy: WAL_LOG          # WAL-based cloning

encoding

String, immutable. Character encoding, inherits from template if unspecified (usually UTF8).

Strongly recommend UTF8 unless special requirements. Cannot be changed after creation.

- name: legacy_db
  template: template0        # Use template0 for non-default encoding
  encoding: LATIN1

locale

String, immutable. Locale setting - sets both lc_collate and lc_ctype. Inherits from template (usually C).

Determines string sort order and character classification. Use C or POSIX for best performance and cross-platform consistency; use language-specific locales (e.g., zh_CN.UTF-8) for proper language sorting.

- name: chinese_db
  template: template0
  locale: zh_CN.UTF-8        # Chinese locale
  encoding: UTF8

lc_collate

String, immutable. String collation rule. Inherits from template (usually C).

Determines ORDER BY and comparison results. Common values: C (byte order, fastest), C.UTF-8, en_US.UTF-8, zh_CN.UTF-8. Cannot be changed after creation.

- name: myapp
  template: template0
  lc_collate: en_US.UTF-8    # English collation
  lc_ctype: en_US.UTF-8

lc_ctype

String, immutable. Character classification rule for upper/lower case, digits, letters. Inherits from template (usually C).

Affects upper(), lower(), regex \w, etc. Cannot be changed after creation.

locale_provider

Enum, immutable. Locale implementation provider: libc, icu, or builtin. Available PG15+, default libc.

Provider Version Description
libc - OS C library, traditional default, varies by system
icu PG15+ ICU library, cross-platform consistent, more langs
builtin PG17+ PostgreSQL builtin, most efficient, C/C.UTF-8 only

Using icu or builtin requires template: template0 with corresponding icu_locale or builtin_locale.

- name: fast_db
  template: template0
  locale_provider: builtin   # Builtin provider, most efficient
  builtin_locale: C.UTF-8

icu_locale

String, immutable. ICU locale identifier. Available PG15+ when locale_provider: icu.

ICU identifiers follow BCP 47. Common values:

Value Description
en-US US English
en-GB British English
zh-Hans Simplified Chinese
zh-Hant Traditional Chinese
ja-JP Japanese
ko-KR Korean 
- name: chinese_app
  template: template0
  locale_provider: icu
  icu_locale: zh-Hans        # Simplified Chinese ICU collation
  encoding: UTF8

icu_rules

String, immutable. Custom ICU collation rules. Available PG16+.

Allows fine-tuning default sort behavior using ICU Collation Customization.

- name: custom_sort_db
  template: template0
  locale_provider: icu
  icu_locale: en-US
  icu_rules: '&V << w <<< W'  # Custom V/W sort order

builtin_locale

String, immutable. Builtin locale provider rules. Available PG17+ when locale_provider: builtin. Values: C or C.UTF-8.

builtin provider is PG17’s new builtin implementation - faster than libc with consistent cross-platform behavior. Suitable for C/C.UTF-8 collation only.

- name: fast_db
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin UTF-8 support
  encoding: UTF8

tablespace

String, mutable. Default tablespace, default pg_default.

Changing tablespace triggers physical data migration - PostgreSQL moves all objects to new tablespace. Can take long time for large databases, use cautiously.

- name: archive_db
  tablespace: slow_hdd       # Archive data on slow storage
ALTER DATABASE "archive_db" SET TABLESPACE "slow_hdd";

is_template

Boolean, mutable. Mark database as template, default false.

When true, any user with CREATEDB privilege can use this database as template for cloning. Template databases typically pre-install standard schemas, extensions, and data.

- name: app_template
  is_template: true          # Mark as template, allow user cloning
  schemas: [core, api]
  extensions: [postgis, pg_trgm]

Deleting is_template: true databases: Pigsty first executes ALTER DATABASE ... IS_TEMPLATE false then drops.

allowconn

Boolean, mutable. Allow connections, default true.

Setting false completely disables connections at database level - no user (including superuser) can connect. Used for maintenance or archival purposes.

- name: archive_db
  allowconn: false           # Disallow all connections
ALTER DATABASE "archive_db" ALLOW_CONNECTIONS false;

revokeconn

Boolean, mutable. Revoke PUBLIC CONNECT privilege, default false.

When true, Pigsty executes:

  • Revoke PUBLIC CONNECT, regular users can’t connect
  • Grant connect to replication user (replicator) and monitor user (dbuser_monitor)
  • Grant connect to admin user (dbuser_dba) and owner with WITH GRANT OPTION

Setting false restores PUBLIC CONNECT privilege.

- name: secure_db
  owner: dbuser_secure
  revokeconn: true           # Revoke public connect, only specified users

connlimit

Integer, mutable. Max concurrent connections, default -1 (unlimited).

Positive integer limits max simultaneous sessions. Doesn’t affect superusers.

- name: limited_db
  connlimit: 50              # Max 50 concurrent connections
ALTER DATABASE "limited_db" CONNECTION LIMIT 50;

baseline

String. SQL baseline file path executed while provisioning the database.

Baseline files typically contain schema definitions, initial data, stored procedures. Path is relative to Ansible search path, usually in files/.

Whenever baseline is defined, the current role runs the file on every provisioning pass for that database, even if the database already exists. It also runs after state: recreate. Make the baseline SQL idempotent, or avoid rerunning it against an existing database.

- name: myapp
  baseline: myapp_schema.sql  # Looks for files/myapp_schema.sql

schemas

Array, mutable (add/remove). Schema definitions to create or drop. Elements can be strings or objects.

Simple format - strings for schema names (create only):

schemas:
  - app
  - api
  - core

Full format - objects for owner and drop operations:

schemas:
  - name: app                # Schema name (required)
    owner: dbuser_app        # Schema owner (optional), generates AUTHORIZATION clause
  - name: deprecated
    state: absent            # Drop schema (CASCADE)

Create uses IF NOT EXISTS; drop uses CASCADE (deletes all objects in schema).

CREATE SCHEMA IF NOT EXISTS "app" AUTHORIZATION "dbuser_app";
DROP SCHEMA IF EXISTS "deprecated" CASCADE;

extensions

Array, mutable (add/remove). Extension definitions to install or uninstall. Elements can be strings or objects.

Simple format - strings for extension names (install only):

extensions:
  - postgis
  - pg_trgm
  - vector

Full format - objects for schema, version, and uninstall:

extensions:
  - name: vector             # Extension name (required)
    schema: public           # Install to schema (optional)
    version: '0.5.1'         # Specific version (optional)
  - name: old_extension
    state: absent            # Uninstall extension (CASCADE)

Installation uses IF NOT EXISTS ... CASCADE; PostgreSQL emits a NOTICE and skips an extension that already exists, while automatically installing dependencies when possible. Uninstallation uses CASCADE and deletes dependent objects.

CREATE EXTENSION IF NOT EXISTS "vector" WITH SCHEMA "public" VERSION '0.5.1' CASCADE;
DROP EXTENSION IF EXISTS "old_extension" CASCADE;

parameters

Object, mutable. Database-level config params via ALTER DATABASE ... SET. Applies to all sessions connecting to this database.

- name: analytics
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'

Use special value DEFAULT (case-insensitive) to reset to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # Reset to default
  statement_timeout: '30s'   # Set new value
ALTER DATABASE "myapp" SET "work_mem" = DEFAULT;
ALTER DATABASE "myapp" SET "statement_timeout" = '30s';

pgbouncer

Boolean, mutable. Add database to Pgbouncer pool list, default true.

Setting false excludes database from Pgbouncer - clients can’t access via connection pool. For internal management databases or direct-connect scenarios.

- name: internal_db
  pgbouncer: false           # No connection pool access

pool_mode

Enum, mutable. Pgbouncer pool mode: transaction, session, or statement. Default transaction.

Mode Description Use Case
transaction Return connection after txn Most OLTP apps, default
session Return connection after session Apps needing session state
statement Return after each statement Simple stateless queries 
- name: session_app
  pool_mode: session         # Session-level pooling

pool_size

Integer, mutable. Pgbouncer default pool size, default 50.

Pool size is the regular backend-connection limit for this database’s pool; pool_size_min controls prewarmed connections. Adjust it for the workload.

- name: high_load_db
  pool_size: 128             # Larger pool for high load

pool_size_min

Integer, mutable. Pgbouncer minimum pool size, default 0.

Values > 0 pre-create specified backend connections for connection warming, reducing first-request latency.

- name: latency_sensitive
  pool_size_min: 10          # Pre-warm 10 connections

pool_reserve

Integer, mutable. Pgbouncer reserve pool size, default 30.

When default pool exhausted, Pgbouncer can allocate up to pool_reserve additional connections for burst traffic.

- name: bursty_db
  pool_size: 50
  pool_reserve: 30           # Allow up to 30 more connections after the regular pool is exhausted

pool_connlimit

Integer, mutable. Max connections via Pgbouncer pool, default 100.

This is Pgbouncer-level limit, independent of database’s connlimit param.

- name: limited_pool_db
  pool_connlimit: 50         # Pool max 50 connections

pool_auth_user

String, mutable. User for Pgbouncer auth query.

Requires pgbouncer_auth_query enabled. When set, all Pgbouncer connections to this database use specified user for auth query password verification.

- name: myapp
  pool_auth_user: dbuser_monitor  # Use monitor user for auth query

register_datasource

Boolean, mutable. Register database to Grafana as PostgreSQL datasource, default true.

Set false to skip Grafana registration. For temp databases, test databases, or internal databases not needed in monitoring.

- name: temp_db
  register_datasource: false  # Don't register to Grafana

Template Inheritance

Many parameters inherit from template database if not explicitly specified. Default template is template1, whose encoding settings are determined by cluster init params:

Cluster Param Default Description
pg_encoding UTF8 Cluster encoding
pg_locale C / C-UTF-8 (if supported) Cluster locale
pg_lc_collate C / C-UTF-8 (if supported) Cluster collation
pg_lc_ctype C / C-UTF-8 (if supported) Cluster ctype

New databases fork from template1, which is customized during PG_PROVISION with extensions, schemas, and default privileges. Unless you explicitly use another template.

Deep Customization

Pigsty provides rich customization params. To customize template database, refer to:

If above configurations don’t meet your needs, use pg_init to specify custom cluster init scripts:

Locale Providers

PostgreSQL 15+ introduced locale_provider for different locale implementations. These are immutable after creation.

Pigsty’s configure wizard selects builtin C.UTF-8/C locale provider based on PG and OS versions. Databases inherit cluster locale by default. To specify different locale provider, you must use template0.

Using ICU provider (PG15+):

- name: myapp_icu
  template: template0        # ICU requires template0
  locale_provider: icu
  icu_locale: en-US          # ICU locale rules
  encoding: UTF8

Using builtin provider (PG17+):

- name: myapp_builtin
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin locale rules
  encoding: UTF8

Provider comparison: libc (traditional, OS-dependent), icu (PG15+, cross-platform, feature-rich), builtin (PG17+, most efficient C/C.UTF-8).

Connection Pool

Pgbouncer connection pool optimizes short-connection performance, reduces contention, prevents excessive connections from overwhelming database, and provides flexibility during migrations.

Pigsty configures 1:1 connection pool for each PostgreSQL instance, running as same pg_dbsu (default postgres OS user). Pool communicates with database via /var/run/postgresql Unix socket.

Pigsty adds all databases in pg_databases to pgbouncer by default. Set pgbouncer: false to exclude specific databases. Pgbouncer database list and config params are defined in /etc/pgbouncer/database.txt:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When creating databases, Pgbouncer database list is refreshed via online reload - doesn’t affect existing connections.

1.6 - HBA Rules

Configuration reference for PostgreSQL and PgBouncer Host-Based Authentication (HBA) rules in Pigsty.

Overview

HBA (Host-Based Authentication) controls “who can connect to the database, from where, and how”. See Authentication for the authentication model and default rules. Pigsty manages HBA rules declaratively through pg_default_hba_rules and pg_hba_rules.

Pigsty renders the following config files during cluster init or HBA refresh:

Config File Path Description
PostgreSQL HBA /pg/data/pg_hba.conf PostgreSQL server HBA rules
PgBouncer HBA /etc/pgbouncer/pgb_hba.conf Connection pool HBA rules

HBA rules are controlled by these parameters:

Parameter Level Description
pg_default_hba_rules G PostgreSQL global default HBA
pg_hba_rules G/C/I PostgreSQL cluster/instance add
pgb_default_hba_rules G PgBouncer global default HBA
pgb_hba_rules G/C/I PgBouncer cluster/instance add

Rule features:

  • Role filtering: Rules support role field, auto-filter based on instance’s pg_role
  • Order sorting: Rules support order field, controls position in final config file
  • Two syntaxes: Supports alias form (simplified) and raw form (direct HBA text)

Refresh HBA

After modifying config, re-render config files and reload services:

bin/pgsql-hba <cls>                   # Refresh entire cluster HBA (recommended)
bin/pgsql-hba <cls> <ip>...           # Refresh specific instances in cluster

Script executes the following playbook:

./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true

PostgreSQL only: ./pgsql.yml -l <cls> -t pg_hba,pg_reload -e pg_reload=true

PgBouncer only: ./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload

Don’t edit config files directly

Don’t directly edit /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf - they’ll be overwritten on next playbook run. All changes should be made in pigsty.yml, then execute bin/pgsql-hba to refresh.

Parameter Details

pg_default_hba_rules

PostgreSQL global default HBA rule list, usually defined in all.vars, provides base access control for all clusters.

  • Type: rule[], Level: Global (G)
pg_default_hba_rules:
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

pg_hba_rules

PostgreSQL cluster/instance-level additional HBA rules, can override at cluster or instance level, merged with default rules and sorted by order.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []
pg_hba_rules:
  - {user: app_user, db: app_db, addr: intra, auth: pwd, title: 'app user access'}

pgb_default_hba_rules

PgBouncer global default HBA rule list, usually defined in all.vars.

  • Type: rule[], Level: Global (G)
pgb_default_hba_rules:
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pgb_hba_rules

PgBouncer cluster/instance-level additional HBA rules.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []

Note: PgBouncer HBA does not support db: replication.

Rule Fields

Each HBA rule is a YAML dict supporting these fields:

Field Type Required Default Description
user string No all Username, supports all, placeholders, +rolename
db string No all Database name, supports all, replication, db name
addr string Yes* - Address alias or CIDR, see Address Aliases
auth string No pwd Auth method alias, see Auth Methods
title string No - Rule description, rendered as comment in config
role string No common Instance role filter, see Role Filtering
order int No 1000 Sort weight, lower first, see Order Sorting
rules list Yes* - Raw HBA text lines, mutually exclusive with addr

Either addr or rules must be specified. Use rules to write raw HBA format directly.

Address Aliases

Pigsty provides address aliases to simplify HBA rule writing:

Alias Expands To Description
local Unix socket Local Unix socket
localhost Unix socket + 127.0.0.1/32 + ::1/128 Loopback addresses
admin ${admin_ip}/32 Admin IP address
infra All infra group node IPs Infrastructure nodes
cluster All current cluster member IPs Same cluster instances
intra / intranet 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 Intranet CIDRs
world / all 0.0.0.0/0 + ::/0 Any address (IPv4 + IPv6)
<CIDR> Direct use e.g., 192.168.1.0/24

Intranet CIDRs can be customized via node_firewall_intranet:

node_firewall_intranet:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

Auth Methods

Pigsty provides auth method aliases for simplified config:

Alias Actual Method Connection Type Description
pwd scram-sha-256 or md5 host Auto-select based on pg_pwd_enc
ssl scram-sha-256 or md5 hostssl Force SSL + password
ssl-sha scram-sha-256 hostssl Force SSL + SCRAM-SHA-256
ssl-md5 md5 hostssl Force SSL + MD5
cert cert hostssl Client certificate auth
trust trust host Unconditional trust (dangerous)
deny / reject reject host Reject connection
ident ident host OS user mapping (PostgreSQL)
peer peer local OS user mapping (PgBouncer/local)

pg_pwd_enc defaults to scram-sha-256, can be set to md5 for legacy client compatibility.

User Variables

HBA rules support these user placeholders, auto-replaced with actual usernames during rendering:

Placeholder Default Corresponding Param
${dbsu} postgres pg_dbsu
${repl} replicator pg_replication_username
${monitor} dbuser_monitor pg_monitor_username
${admin} dbuser_dba pg_admin_username

Role Filtering

The role field in HBA rules controls which instances the rule applies to:

Role Description
common Default, applies to all instances
primary Primary instance only
replica Replica instance only
offline Offline instance only (pg_role: offline or pg_offline_query: true)
standby Standby instance
delayed Delayed replica instance

Role filtering matches based on instance’s pg_role variable. Non-matching rules are commented out (prefixed with #).

pg_hba_rules:
  # Only applies on primary: writer can only connect to primary
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer only on primary'}

  # Only applies on offline instances: ETL dedicated network
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl, role: offline, title: 'offline dedicated'}

Order Sorting

PostgreSQL HBA is first-match-wins, rule order is critical. Pigsty controls rule rendering order via the order field.

Order Interval Convention

Interval Usage
0 - 99 User high-priority rules (before all defaults)
100 - 650 Default rule zone (spaced by 50 for insertion)
1000+ User rule default (rules without order)

PostgreSQL Default Rules Order

Order Rule Description
100 dbsu local ident
150 dbsu replication local
200 replicator localhost
250 replicator intra replication
300 replicator intra postgres
350 monitor localhost
400 monitor infra
450 admin infra ssl
500 admin world ssl
550 dbrole_readonly localhost
600 dbrole_readonly intra
650 dbrole_offline intra

PgBouncer Default Rules Order

Order Rule Description
100 dbsu local peer
150 all localhost pwd
200 monitor pgbouncer intra
250 monitor world deny
300 admin intra pwd
350 admin world deny
400 all intra pwd

Syntax Examples

Alias Form: Using Pigsty simplified syntax

pg_hba_rules:
  - title: allow grafana view access
    role: primary
    user: dbuser_view
    db: meta
    addr: infra
    auth: ssl

Rendered result:

# allow grafana view access [primary]
hostssl  meta               dbuser_view        10.10.10.10/32     scram-sha-256

Raw Form: Using PostgreSQL HBA syntax directly

pg_hba_rules:
  - title: allow intranet password access
    role: common
    rules:
      - host all all 10.0.0.0/8 scram-sha-256
      - host all all 172.16.0.0/12 scram-sha-256
      - host all all 192.168.0.0/16 scram-sha-256

Rendered result:

# allow intranet password access [common]
host all all 10.0.0.0/8 scram-sha-256
host all all 172.16.0.0/12 scram-sha-256
host all all 192.168.0.0/16 scram-sha-256

Common Scenarios

Blacklist IP: Use order: 0 to ensure first match

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'block bad ip'}

Whitelist App Server: High priority for specific IP

pg_hba_rules:
  - {user: app_user, db: app_db, addr: '192.168.1.10/32', auth: ssl, order: 50, title: 'app server'}

Admin Force Certificate: Override default SSL password auth

pg_hba_rules:
  - {user: '${admin}', db: all, addr: world, auth: cert, order: 10, title: 'admin cert only'}

Offline Instance Dedicated Network: Only on offline instances

pg_hba_rules:
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl-sha, role: offline, title: 'etl network'}

Restrict Access by Database: Sensitive databases limited to specific networks

pg_hba_rules:
  - {user: fin_user, db: finance_db, addr: '10.20.0.0/16', auth: ssl, title: 'finance only'}
  - {user: hr_user, db: hr_db, addr: '10.30.0.0/16', auth: ssl, title: 'hr only'}

PgBouncer Dedicated Rules: Note no db: replication support

pgb_hba_rules:
  - {user: '+dbrole_readwrite', db: all, addr: world, auth: ssl, title: 'app via pgbouncer'}

Complete Cluster Example 

pg-prod:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
    10.10.10.13: {pg_seq: 3, pg_role: offline}
  vars:
    pg_cluster: pg-prod

    pg_hba_rules:
      # Blacklist: known malicious IP (highest priority)
      - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'blacklist'}

      # App server whitelist (high priority)
      - {user: app_user, db: app_db, addr: '192.168.1.0/24', auth: ssl, order: 50, title: 'app servers'}

      # ETL tasks: offline instances only
      - {user: etl_user, db: all, addr: '172.20.0.0/16', auth: pwd, role: offline, title: 'etl tasks'}

      # Cluster internal monitoring
      - {user: '${monitor}', db: all, addr: cluster, auth: pwd, order: 380, title: 'cluster monitor'}

    pgb_hba_rules:
      # App via connection pool
      - {user: '+dbrole_readwrite', db: all, addr: '192.168.1.0/24', auth: ssl, title: 'app via pgbouncer'}

Verification & Troubleshooting

View Current HBA Rules

psql -c "TABLE pg_hba_file_rules"         # View via SQL (recommended)
cat /pg/data/pg_hba.conf                  # View PostgreSQL HBA file
cat /etc/pgbouncer/pgb_hba.conf           # View PgBouncer HBA file
grep '^#' /pg/data/pg_hba.conf | head -20 # View rule titles (verify order)

Test Connection Auth

psql -h <host> -p 5432 -U <user> -d <db> -c "SELECT 1"

Common Issues

Error Message Possible Cause Solution
no pg_hba.conf entry for host... No matching HBA rule Add corresponding rule and refresh
password authentication failed Wrong password or enc Check password and pg_pwd_enc
Rule not taking effect Not refreshed or order Run bin/pgsql-hba, check order

Important Notes

  1. Order sensitive: PostgreSQL HBA is first-match-wins, use order wisely
  2. Role matching: Ensure role field matches target instance’s pg_role
  3. Address format: CIDR must be correct, e.g., 10.0.0.0/8 not 10.0.0.0/255.0.0.0
  4. PgBouncer limitation: Does not support db: replication
  5. TLS prerequisite: ssl and cert require server-side TLS; clients must still use verify-full to authenticate the server
  6. Test first: Validate in test environment before modifying HBA
  7. Refresh on scale: Rules using addr: cluster need refresh after cluster membership changes

Related Documentation

1.7 - Access Control

Configuration reference for Pigsty built-in roles, users, default privileges, and database ACLs.

Access control combines roles, object privileges, database ACLs, and HBA. This page covers configuration parameters; see Access Control Concepts for design and boundaries.

Pigsty provides a compact ACL model described by these parameters:

  • pg_default_roles: system roles and system users.
  • pg_users: application users and roles.
  • pg_default_privileges: default privileges on objects created by managed administrators and owners.
  • pg_revoke_public, pg_default_schemas, and pg_default_extensions: default behavior for template1.

Manage these parameters together with HBA and database definitions to produce reproducible access-control configuration.

Default Role System (pg_default_roles)

The defaults contain four business roles and four system users:

Name Type Description
dbrole_readonly NOLOGIN Shared read-only role with SELECT and USAGE
dbrole_readwrite NOLOGIN Inherits read-only and adds INSERT, UPDATE, and DELETE
dbrole_admin NOLOGIN Inherits pg_monitor and read-write; can create objects and triggers
dbrole_offline NOLOGIN Independent read-only role; instance scope must be restricted explicitly through HBA
postgres User System superuser; same name as pg_dbsu
replicator User Streaming replication and backup; inherits monitoring and read-only privileges
dbuser_dba User Primary administration account, also synchronized to PgBouncer
dbuser_monitor User Monitoring account with pg_monitor; records slow SQL by default

These definitions live in pg_default_roles. The parameter is a complete list. When customizing it, copy and retain the required default roles and system users, then add new roles in dependency order. If a role name changes, update references in HBA, default privileges, and scripts.

Default Users and Credential Parameters

These parameters control system-user names and passwords:

Parameter Default Purpose
pg_dbsu postgres Database and OS superuser
pg_dbsu_password Empty string dbsu password, disabled by default
pg_replication_username replicator Replication user name
pg_replication_password DBUser.Replicator Replication password
pg_admin_username dbuser_dba Administrator user name
pg_admin_password DBUser.DBA Administrator password
pg_monitor_username dbuser_monitor Monitoring user
pg_monitor_password DBUser.Monitor Monitoring password

After changing these parameters, update the corresponding user definitions in pg_default_roles so user names and role attributes remain consistent.

Application Roles and Grants (pg_users)

Declare application users with pg_users; see User Configuration for field details. The roles field grants business roles.

Example read-only and read-write users:

pg_users:
  - { name: app_reader,  password: DBUser.Reader,  roles: [dbrole_readonly],  pgbouncer: true }
  - { name: app_writer,  password: DBUser.Writer,  roles: [dbrole_readwrite], pgbouncer: true }

Application users inherit default object privileges through dbrole_*. Database CONNECT privileges and pg_hba_rules continue to control which databases and sources can connect.

For finer ACLs, use standard GRANT and REVOKE in baseline SQL or a later playbook, and include those additional grants in reviews.

Default Privilege Template (pg_default_privileges)

pg_default_privileges applies to pg_dbsu, pg_admin_username, dbrole_admin, and every declared database owner. The default template is:

pg_default_privileges:
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Objects created by these identities receive the corresponding privileges automatically. Other object creators need their own ALTER DEFAULT PRIVILEGES configuration.

Additional notes:

  • pg_revoke_public defaults to true, revoking CREATE from PUBLIC on databases and the public schema.
  • pg_default_schemas and pg_default_extensions control schemas and extensions created in template1/postgres, usually for monitoring objects such as the monitor schema and pg_stat_statements.

Common Scenarios

Read-only Account for a Partner 

pg_users:
  - name: partner_ro
    password: Partner.Read
    roles: [dbrole_readonly]
pg_hba_rules:
  - { user: partner_ro, db: analytics, addr: 203.0.113.0/24, auth: ssl }

This adds an HBA rule allowing the partner to reach analytics over TLS from the specified CIDR. pg_hba_rules does not remove broader default rules. If the account must reach only this database, also narrow the default HBA policy and configure database CONNECT privileges.

DDL for an Application Administrator 

pg_users:
  - name: app_admin
    password: DBUser.AppAdmin
    roles: [dbrole_admin]

app_admin inherits DDL privileges from dbrole_admin. To apply the default privileges configured for dbrole_admin to new objects, run SET ROLE dbrole_admin first. If app_admin is a declared database owner, it can also create objects directly as that owner.

Custom Default Privileges 

pg_default_privileges:
  - GRANT INSERT,UPDATE,DELETE ON TABLES TO dbrole_admin
  - GRANT SELECT,UPDATE ON SEQUENCES TO dbrole_admin
  - GRANT SELECT ON TABLES TO reporting_group

This parameter replaces the complete default privilege list. Referenced roles must already exist. Changes affect only objects created afterward; grant privileges separately on existing objects.

Integration with Other Components

  • HBA rules: use pg_hba_rules to bind roles, databases, and sources. To restrict dbrole_offline, set role: offline on its rule.
  • PgBouncer: users with pgbouncer: true are written to userlist.txt; pool_mode and pool_connlimit control pool-level quotas.
  • Database monitoring: dbuser_monitor receives privileges from pg_default_roles. When adding another monitoring user, grant pg_monitor and check access to the monitor schema.

These parameters can be versioned with the inventory. Continue to review effective privileges through PostgreSQL catalogs.

Related Documentation

1.8 - Parameters

Configure PostgreSQL parameters at cluster, instance, database, and user levels

PostgreSQL parameters can be configured at multiple levels with different scopes and precedence. Pigsty supports four configuration levels, from global to local:

Level Scope Configuration Method Storage Location
Cluster All instances in cluster Patroni DCS / Tuning Templates etcd + postgresql.conf
Instance Single PG instance pg_parameters / ALTER SYSTEM postgresql.auto.conf
Database All sessions in a DB pg_databases[].parameters pg_db_role_setting
User All sessions of a user pg_users[].parameters pg_db_role_setting

Priority from low to high: Cluster < Instance < Database < User < Session (SET command). Higher priority settings override lower ones.

For complete PostgreSQL parameter documentation, see PostgreSQL Docs: Server Configuration.

Cluster Level

Cluster-level parameters are shared across all instances (primary and replicas) in a PostgreSQL cluster. In Pigsty, cluster parameters are managed via Patroni and stored in DCS (etcd by default).

Pigsty provides four pre-configured Patroni tuning templates optimized for different workloads, specified via pg_conf:

Template Use Case Characteristics
oltp.yml OLTP transactions Low latency, high concurrency (default)
olap.yml OLAP analytics Large queries, high throughput
crit.yml Critical/Financial Max durability, safety over perf
tiny.yml Tiny instances Resource-constrained, dev/test

Template files are located in roles/pgsql/templates/ and contain auto-calculated values based on hardware specs. Templates are rendered to /etc/patroni/patroni.yml during cluster initialization. See Tuning Templates for details.

Before cluster creation, you can adjust these templates to modify initial parameters. Once initialized, parameter changes should be made via Patroni’s configuration management.

Patroni DCS Config

Patroni stores cluster config in DCS (etcd by default), ensuring consistent configuration across all members.

Storage Structure:

/pigsty/                          # namespace (patroni_namespace)
  └── pg-meta/                    # cluster name (pg_cluster)
      ├── config                  # cluster config (shared)
      ├── leader                  # current primary info
      ├── members/                # member registration
      │   ├── pg-meta-1
      │   └── pg-meta-2
      └── ...

Rendering Flow:

  1. Init: Template (e.g., oltp.yml) rendered via Jinja2 to /etc/patroni/patroni.yml
  2. Start: Patroni reads local config, writes PostgreSQL parameters to DCS
  3. Runtime: Patroni periodically syncs DCS config to local PostgreSQL

Local Cache:

Each Patroni instance caches DCS config locally at /pg/conf/<instance>.yml:

  • On start: Load from DCS, cache locally
  • Runtime: Periodically sync DCS to local cache
  • DCS unavailable: Continue with local cache (no failover possible)

Config File Hierarchy

Patroni renders DCS config to local PostgreSQL config files:

/pg/data/
├── postgresql.conf          # Main config (managed by Patroni)
├── postgresql.base.conf     # Base config (via include directive)
├── postgresql.auto.conf     # Instance overrides (ALTER SYSTEM)
├── pg_hba.conf              # Client auth config
└── pg_ident.conf            # User mapping config

Load Order (priority low to high):

  1. postgresql.conf: Dynamically generated by Patroni with DCS cluster params
  2. postgresql.base.conf: Loaded via include, static base config
  3. postgresql.auto.conf: Auto-loaded by PostgreSQL, instance overrides

Since postgresql.auto.conf loads last, its parameters override earlier files.

Instance Level

Instance-level parameters apply only to a single PostgreSQL instance, overriding cluster-level config. These are written to postgresql.auto.conf, which loads last and can override any cluster parameter.

This is a powerful technique for setting instance-specific values:

  • Set hot_standby_feedback = on on replicas
  • Adjust work_mem or maintenance_work_mem for specific instances
  • Set recovery_min_apply_delay for delayed replicas

Using pg_parameters

In Pigsty config, use pg_parameters to define instance-level parameters:

pg-meta:
  hosts:
    10.10.10.10:
      pg_seq: 1
      pg_role: primary
      pg_parameters:                              # instance-level params
        log_statement: all                        # log all SQL for this instance only
  vars:
    pg_cluster: pg-meta
    pg_parameters:                                # cluster default instance params
      log_timezone: Asia/Shanghai
      log_min_duration_statement: 1000

Use ./pgsql.yml -l <cls> -t pg_param to apply parameters, which renders to postgresql.auto.conf.

Override Hierarchy

pg_parameters can be defined at different Ansible config levels, priority low to high:

all:
  vars:
    pg_parameters:                    # global default
      log_statement: none

  children:
    pg-meta:
      vars:
        pg_parameters:                # cluster override
          log_statement: ddl
      hosts:
        10.10.10.10:
          pg_parameters:              # instance override (highest)
            log_statement: all

Using ALTER SYSTEM

You can also modify instance parameters at runtime via ALTER SYSTEM:

-- Set parameters
ALTER SYSTEM SET work_mem = '256MB';
ALTER SYSTEM SET log_min_duration_statement = 1000;

-- Reset to default
ALTER SYSTEM RESET work_mem;
ALTER SYSTEM RESET ALL;  -- Reset all ALTER SYSTEM settings

-- Reload config to take effect
SELECT pg_reload_conf();

ALTER SYSTEM writes to postgresql.auto.conf.

Note: In Pigsty-managed clusters, postgresql.auto.conf is managed by Ansible via pg_parameters. Manual ALTER SYSTEM changes may be overwritten on next playbook run. Use pg_parameters in pigsty.yml for persistent instance-level params.

List-Type Parameters

PostgreSQL has special parameters accepting comma-separated lists. In YAML config, the entire value must be quoted, otherwise YAML parses it as an array:

# Correct: quote the entire value
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'

# Wrong: unquoted causes YAML parse error
pg_parameters:
  shared_preload_libraries: timescaledb, pg_stat_statements   # YAML parses as array!

Pigsty auto-detects these list parameters and renders them without outer quotes:

Parameter Description Example Value
shared_preload_libraries Preload shared libs 'timescaledb, pg_stat_statements'
search_path Schema search path '"$user", public, app'
local_preload_libraries Local preload libs 'auto_explain'
session_preload_libraries Session preload libs 'pg_hint_plan'
log_destination Log output targets 'csvlog, stderr'
unix_socket_directories Unix socket dirs '/var/run/postgresql, /tmp'
temp_tablespaces Temp tablespaces 'ssd_space, hdd_space'
debug_io_direct Direct I/O mode (PG16+) 'data, wal'

Rendering Example:

# pigsty.yml config (quotes required in YAML)
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'
  work_mem: 64MB
# Rendered postgresql.auto.conf (list params unquoted)
shared_preload_libraries = timescaledb, pg_stat_statements
search_path = "$user", public, app
work_mem = '64MB'

Database Level

Database-level parameters apply to all sessions connected to a specific database. Implemented via ALTER DATABASE ... SET, stored in pg_db_role_setting.

Configuration

Use the parameters field in pg_databases:

pg_databases:
  - name: analytics
    owner: dbuser_analyst
    parameters:
      work_mem: 256MB                              # analytics needs more memory
      maintenance_work_mem: 1GB                    # large table maintenance
      statement_timeout: 10min                     # allow long queries
      search_path: '"$user", public, mart'         # list param needs quotes

Like instance-level params, list-type values must be quoted in YAML.

Rendering Rules

Database params are set via ALTER DATABASE ... SET. Pigsty auto-selects correct syntax:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries, log_destination) without outer quotes:

ALTER DATABASE "analytics" SET "search_path" = "$user", public, mart;

Scalar params with quoted values:

ALTER DATABASE "analytics" SET "work_mem" = '256MB';
ALTER DATABASE "analytics" SET "statement_timeout" = '10min';

Note: While log_destination is in the database whitelist, its context is sighup, so it cannot take effect at database level. Configure it at instance level (pg_parameters).

View Database Params 

-- View params for a specific database
SELECT datname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_database d ON d.oid = drs.setdatabase
WHERE drs.setrole = 0 AND datname = 'analytics';

Manual Management 

-- Set params
ALTER DATABASE analytics SET work_mem = '256MB';
ALTER DATABASE analytics SET search_path = "$user", public, myschema;

-- Reset params
ALTER DATABASE analytics RESET work_mem;
ALTER DATABASE analytics RESET ALL;

User Level

User-level parameters apply to all sessions of a specific database user. Implemented via ALTER USER ... SET, also stored in pg_db_role_setting.

Configuration

Use the parameters field in pg_users or pg_default_roles:

pg_users:
  - name: dbuser_analyst
    password: DBUser.Analyst
    parameters:
      work_mem: 256MB                              # more memory for analytics
      statement_timeout: 5min                      # allow longer queries
      search_path: '"$user", public, analytics'    # list param needs quotes
      log_statement: all                           # log all SQL

Rendering Rules

Same as database-level:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries) without outer quotes:

ALTER USER "dbuser_analyst" SET "search_path" = "$user", public, analytics;

Scalar params with quoted values:

ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';

DEFAULT Value

Use DEFAULT (case-insensitive) to reset a parameter to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # reset to default
  statement_timeout: 30s     # set specific value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

View User Params 

-- View params for a specific user
SELECT rolname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_roles r ON r.oid = drs.setrole
WHERE rolname = 'dbuser_analyst';

Manual Management 

-- Set params
ALTER USER dbuser_app SET work_mem = '128MB';
ALTER USER dbuser_app SET search_path = "$user", public, myschema;

-- Reset params
ALTER USER dbuser_app RESET work_mem;
ALTER USER dbuser_app RESET ALL;

Priority

When the same parameter is set at multiple levels, PostgreSQL applies this priority (low to high):

postgresql.conf           ← Cluster params (Patroni DCS)
       ↓
postgresql.auto.conf      ← Instance params (pg_parameters / ALTER SYSTEM)
       ↓
Database level            ← ALTER DATABASE SET
       ↓
User level                ← ALTER USER SET
       ↓
Session level             ← SET command

Database vs User Priority:

When a user connects to a specific database and the same parameter is set at both levels, PostgreSQL uses the user-level parameter since it has higher priority.

Example:

# Database: analytics has work_mem = 256MB
pg_databases:
  - name: analytics
    parameters:
      work_mem: 256MB

# User: analyst has work_mem = 512MB
pg_users:
  - name: analyst
    parameters:
      work_mem: 512MB
  • analyst connecting to analytics: work_mem = 512MB (user takes precedence)
  • Other users connecting to analytics: work_mem = 256MB (database applies)
  • analyst connecting to other DBs: work_mem = 512MB (user applies)

2 - Service/Access

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

Service is an abstraction: it is the form in which database clusters provide capabilities externally, encapsulating the details of the underlying cluster.

Service is critical for stable access in production environments, showing its value during high availability cluster automatic failovers. Personal users typically don’t need to worry about this concept.

Personal User

The concept of “service” is for production environments. Personal users/single-machine clusters can skip the complexity and directly access the database using instance names/IP addresses.

For example, Pigsty’s default single-node pg-meta.meta database can be directly connected using three different users:

psql postgres://dbuser_dba:DBUser.DBA@10.10.10.10/meta     # Direct connection with DBA superuser
psql postgres://dbuser_meta:DBUser.Meta@10.10.10.10/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

In real-world production environments, we use primary-replica database clusters based on replication. Within the cluster, there is one and only one instance as the leader (primary) that can accept writes. Other instances (replicas) continuously fetch change logs from the cluster leader to stay synchronized. Additionally, replicas can handle read-only requests, significantly offloading the primary in read-heavy, write-light scenarios. Therefore, distinguishing between write requests and read-only requests to the cluster is a very common practice.

Moreover, for production environments with high-frequency short connections, we pool requests through connection pooling middleware (Pgbouncer) to reduce the overhead of connection and backend process creation. But for scenarios like ETL and change execution, we need to bypass the connection pool and directly access the database. At the same time, high-availability clusters may experience failover during failures, which causes a change in the cluster leader. Therefore, high-availability database solutions require write traffic to automatically adapt to cluster leader changes. These different access requirements (read-write separation, pooling vs. direct connection, automatic adaptation to failovers) ultimately abstract the concept of Service.

Typically, database clusters must provide this most basic service:

  • Read-write service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-write service (primary): Write data: Only carried by the primary.
  • Read-only service (replica): Read data: Can be carried by replicas, but can also be carried by the primary if no replicas are available

Additionally, depending on specific business scenarios, there might be other services, such as:

  • Default direct access service (default): Service that allows (admin) users to bypass the connection pool and directly access the database
  • Offline replica service (offline): Dedicated replica that doesn’t handle online read-only traffic, used for ETL and analytical queries
  • Synchronous replica service (standby): Read-only service with no replication delay, handled by synchronous standby/primary for read-only queries
  • Delayed replica service (delayed): Access older data from the same cluster from a certain time ago, handled by delayed replicas

Default Service

Pigsty provides four different services by default for each PostgreSQL database cluster. Here are the default services and their definitions:

Service Port Description
primary 5433 Production read-write, connect to primary pool (6432)
replica 5434 Production read-only, connect to replica pool (6432)
default 5436 Admin, ETL writes, direct access to primary (5432)
offline 5438 OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

From the sample cluster architecture diagram, you can see how these four services work:

pigsty-ha.png

The actual DNS target of pg-meta is controlled by pg_dns_target. The default auto points to the L2 VIP when VIP is enabled; otherwise it points to the inventory primary’s IP. VIP is not enabled by default. See Access Service.

Service Implementation

In Pigsty, services are implemented using haproxy on nodes, differentiated by different ports on the host node.

Haproxy is enabled by default on every node managed by Pigsty to expose services, and database nodes are no exception. Although nodes in the cluster have primary-replica distinctions from the database perspective, from the service perspective, all nodes are the same: This means even if you access a replica node, as long as you use the correct service port, you can still use the primary’s read-write service. This design seals the complexity: as long as you can access any instance on the PostgreSQL cluster, you can fully access all services.

This design is similar to the NodePort service in Kubernetes. Similarly, in Pigsty, every service includes these two core elements:

  1. Access endpoints exposed via NodePort (port number, from where to access?)
  2. Target instances chosen through Selectors (list of instances, who will handle it?)

The boundary of Pigsty’s service delivery stops at the cluster’s HAProxy. Users can access these load balancers in various ways. Please refer to Access Service.

All services are declared through configuration files. For instance, the default PostgreSQL service is defined by the pg_default_services parameter:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

You can also define additional services in pg_services. Both pg_default_services and pg_services are arrays of Service Definition objects.

Define Service

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, with four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for other component access

For PostgreSQL clusters, you typically only need to focus on the first two. Each service definition generates a new configuration file in the configuration directory of all related HAProxy instances: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg Here’s a custom service example standby: When you want to provide a read-only service with no replication delay, you can add this record in pg_services:

- name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
  port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
  ip: "*"                         # optional, service bind ip address, `*` for all ip by default
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, backup server selector, these instances will only be used when default selector instances are all down
  dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default, meaning pg_default_service_dest decides
  check: /sync                    # optional, health check url path, / by default, here using Patroni API: /sync, only sync standby and primary will return 200 healthy status
  maxconn: 5000                   # optional, max allowed front-end connection, default 5000
  balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

The service definition above is rendered as /etc/haproxy/conf.d/pg-test-standby.cfg on the sample three-node pg-test cluster:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds to port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Max connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balance algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connections
    http-check send meth OPTIONS uri /sync   # <---- Using /sync here, Patroni health check API, only sync standby and primary will return 200 healthy status
    http-check expect status 200             # <---- Health check return code 200 means healthy
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector.
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore acts as fallback instance: normally doesn't serve requests, only serves read-only requests after all other replicas are down, maximizing avoidance of read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

Here, all three instances of the pg-test cluster are selected by selector: "[]" and rendered into the backend list of the pg-test-standby service. Because of the /sync health check, the Patroni REST API returns HTTP 200 only on the primary and synchronous standby, so only those members can actually serve requests. Additionally, the primary satisfies the condition pg_role == primary and is selected by the backup selector, marked as a backup server, and will only be used when no other instances (i.e., sync standby) can satisfy the requirement.

Primary Service

The Primary service is probably the most critical service in production environments. It provides read-write capability to the database cluster on port 5433, with the service definition as follows:

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Primary service
  • But only the primary can pass the health check (check: /primary), actually serving Primary service traffic.
  • The destination parameter dest: default means the Primary service destination is affected by the pg_default_service_dest parameter
  • The default value of dest is default which will be replaced with the value of pg_default_service_dest, defaulting to pgbouncer.
  • By default, the Primary service destination is the connection pool on the primary, i.e., the port specified by pgbouncer_port, defaulting to 6432

If the value of pg_default_service_dest is postgres, then the primary service destination will bypass the connection pool and directly use the PostgreSQL database port (pg_port, default value 5432), which is very useful for scenarios where you don’t want to use a connection pool.

Example: pg-test-primary haproxy configuration
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to using Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni’s high availability mechanism ensures that at most one instance’s /primary health check is true at any time, so the Primary service will always route traffic to the primary instance.

One benefit of using the Primary service instead of directly connecting to the database is that if the cluster experiences a split-brain situation (for example, killing the primary Patroni with kill -9 without watchdog), Haproxy can still avoid split-brain in this situation, because it only distributes traffic when Patroni is alive and returns primary status.

Replica Service

The Replica service is second only to the Primary service in importance in production environments. It provides read-only capability to the database cluster on port 5434, with the service definition as follows:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Replica service
  • All instances can pass the health check (check: /read-only), serving Replica service traffic.
  • Backup selector: [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backup servers.
  • Only when all regular replicas are down will the Replica service be served by the primary or offline replicas.
  • The destination parameter dest: default means the Replica service destination is also affected by the pg_default_service_dest parameter
  • The default value of dest is default which will be replaced with the value of pg_default_service_dest, defaulting to pgbouncer, same as the Primary service
  • By default, the Replica service destination is the connection pool on replicas, i.e., the port specified by pgbouncer_port, defaulting to 6432
Example: pg-test-replica haproxy configuration
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service is very flexible: If there are living dedicated Replica instances, it will prioritize using these instances to serve read-only requests. Only when all replica instances are down will the primary serve as a fallback for read-only requests. For the common one-primary-one-replica two-node cluster: use the replica as long as it’s alive, use the primary only when the replica is down.

Additionally, unless all dedicated read-only instances are down, the Replica service will not use dedicated Offline instances, thus avoiding mixing online fast queries with offline slow queries and their mutual interference.

Default Service

The Default service provides service on port 5436, and it’s a variant of the Primary service.

The Default service always bypasses the connection pool and directly connects to PostgreSQL on the primary, which is useful for admin connections, ETL writes, CDC change data capture, etc.

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is changed to postgres, then the Default service is completely equivalent to the Primary service except for port and name. In this case, you can consider removing Default from default services.

Example: pg-test-default haproxy configuration
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service runs on port 5438 and bypasses the connection pool to access PostgreSQL directly. It is normally used for slow or analytical queries, ETL reads, and interactive personal queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

The Offline service routes traffic directly to dedicated offline replicas, or regular read-only instances marked with pg_offline_query.

  • The selector parameter filters two types of instances from the cluster: offline replicas with pg_role = offline, or regular read-only instances marked with pg_offline_query = true
  • The main difference between dedicated offline replicas and marked regular replicas is: the former doesn’t serve Replica service requests by default, avoiding mixing fast and slow queries, while the latter does serve by default.
  • The backup selector parameter filters one type of instance from the cluster: regular replicas without the offline mark, which means if offline instances or marked regular replicas are down, other regular replicas can be used to serve Offline service.
  • Health check /replica only returns 200 for replicas, primary returns error, so Offline service will never distribute traffic to the primary instance, even if only the primary remains in the cluster.
  • At the same time, the primary instance is neither selected by the selector nor by the backup selector, so it will never serve Offline service. Therefore, Offline service can always avoid users accessing the primary, thus avoiding impact on the primary.
Example: pg-test-offline haproxy configuration
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

The Offline service provides restricted read-only service, typically used for two types of queries: interactive queries (personal users), slow queries and long transactions (analytics/ETL).

The Offline service requires extra care. HAProxy’s /replica health check automatically rejects the new primary after a switchover, but selector uses static pg_role / pg_offline_query labels from the inventory. In a one-primary-one-replica cluster where only the replica serves Offline queries, a switchover may temporarily leave no eligible backend. Reloading an unchanged inventory does not add the old primary to the Offline backend list. First update the inventory labels (or pg_offline_query) to match the new plan and then reload service, or switch the primary back.

If your business model is relatively simple, you can consider removing Default service and Offline service, using Primary service and Replica service to directly connect to the database.

Reload Service

Reload services when cluster membership changes, service definitions or static selector labels change, or relative weights are adjusted. Normal Primary/Replica switchover is handled by Patroni health checks and does not require a separate reload.

bin/pgsql-svc <cls> [ip...]         # reload service for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # the actual ansible task to reload service

Access Service

The boundary of Pigsty’s service delivery stops at the cluster’s HAProxy. Users can access these load balancers in various ways.

The typical approach is to use DNS or VIP access, binding to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

Type Example Description
Cluster Domain Name pg-test Access via cluster domain name (resolved by dnsmasq @ infra nodes)
Cluster VIP Address 10.10.10.3 Access via L2 VIP address managed by vip-manager, bound to primary
Instance Hostname pg-test-1 Access via any instance hostname (resolved by dnsmasq @ infra nodes)
Instance IP Address 10.10.10.11 Access any instance IP address

Port

Pigsty uses different ports to distinguish pg services

Port Service Type Description
5432 postgres database Direct access to postgres server
6432 pgbouncer middleware Go through connection pool middleware before postgres
5433 primary service Access primary pgbouncer (or postgres)
5434 replica service Access replica pgbouncer (or postgres)
5436 default service Access primary postgres
5438 offline service Access offline postgres

Combinations

# Access via cluster domain (these examples assume L2 VIP is enabled; otherwise the domain points to the inventory primary by default)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://test@10.10.10.3:5432/test # L2 VIP -> Primary direct access
postgres://test@10.10.10.3:6432/test # L2 VIP -> Primary Connection Pool -> Primary
postgres://test@10.10.10.3:5433/test # L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://test@10.10.10.3:5434/test # L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://dbuser_dba@10.10.10.3:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://dbuser_stats@10.10.10.3:5438/test # L2 VIP -> HAProxy -> offline direct connect (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database Instance Direct Connect (singleton access)
postgres://test@pg-test-1:6432/test # DNS -> connection pool -> database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connect
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write

# Directly specify any cluster instance IP access
postgres://test@10.10.10.11:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://test@10.10.10.11:6432/test # Connection Pool -> Database
postgres://test@10.10.10.11:5433/test # HAProxy -> connection pool -> database read/write
postgres://test@10.10.10.11:5434/test # HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@10.10.10.11:5436/test # HAProxy -> Database Direct Connections
postgres://dbuser_stats@10.10.10.11:5438/test # HAProxy -> database offline read/write

# Smart client automatic read/write separation
postgres://test@10.10.10.11:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://test@10.10.10.11:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Override Service

You can override the default service configuration in several ways. A common requirement is to have Primary service and Replica service bypass Pgbouncer connection pool and directly access PostgreSQL database.

To achieve this, you can change pg_default_service_dest to postgres, so all services with svc.dest='default' in the service definition will use postgres instead of the default pgbouncer as the target.

If you’ve already pointed Primary service to PostgreSQL, then the default service becomes redundant and can be removed.

If you don’t need to distinguish between personal interactive queries and analytics/ETL slow queries, you can consider removing the Offline service from the default service list pg_default_services.

If you don’t need read-only replicas to share online read-only traffic, you can also remove Replica service from the default service list.

Delegate Service

Pigsty exposes PostgreSQL services with haproxy on nodes. All haproxy instances in the cluster are configured with the same service definition.

However, you can delegate pg service to a specific node group (e.g., dedicated haproxy lb cluster) rather than haproxy on PostgreSQL cluster members.

To do so, you need to override the default service definition with pg_default_services and set pg_service_provider to the proxy group name.

For example, this configuration will expose pg cluster primary service on haproxy node group proxy with port 10013.

pg_service_provider: proxy       # use load balancer on group `proxy` with port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

It’s user’s responsibility to make sure each delegate service port is unique among the proxy cluster.

A dedicated load balancer cluster example is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

3 - PostgreSQL Security

Entry points for PostgreSQL authentication, access control, encrypted communication, data protection, and secure operations.

PostgreSQL security combines authentication, authorization, network boundaries, encrypted communication, data protection, and operational process. Pigsty provides configuration entry points for these mechanisms; operators must still harden, verify, and audit the deployment for its environment.

Concepts and Boundaries

Topic Content
Security and Compliance Default state, capability boundaries, and hardening path
Authentication HBA, SCRAM, certificate authentication, and credential management
Access Control Built-in roles, default privileges, database ACLs, and instance-access boundaries
Encrypted Communication CA, TLS, server authentication, and certificate rotation
Data Security Page checksums, replication, backup, PITR, audit, and logging
Compliance Launch checks, control mappings, and evidence requirements

Configuration Reference

Administration and Verification

The inventory describes desired state. Acceptance checks should also inspect HBA, certificates, listen ports, and sensitive files on running nodes, and verify effective roles and privileges through PostgreSQL catalogs.

4 - Administration

Standard Operating Procedures (SOP) for database administration tasks

4.1 - Managing PostgreSQL Clusters

Create/destroy PostgreSQL clusters, scale existing clusters, and clone clusters.

Quick Reference

Action Command Description
Create Cluster bin/pgsql-add <cls> Create a new PostgreSQL cluster
Expand Cluster bin/pgsql-add <cls> <ip...> Add replica to existing cluster
Shrink Cluster bin/pgsql-rm <cls> <ip...> Remove instance from cluster
Remove Cluster bin/pgsql-rm <cls> Destroy entire PostgreSQL cluster
Reload Service bin/pgsql-svc <cls> [ip...] Reload cluster load balancer config
Reload HBA bin/pgsql-hba <cls> [ip...] Reload cluster HBA access rules
Clone Cluster - Clone via standby cluster or PITR

For other management tasks, see: HA Management, Manage Users, Manage Databases.

Create Cluster

To create a new PostgreSQL cluster, first define the cluster in the inventory, then add nodes and initialize:

Script
bin/node-add  <cls>     # Add nodes in group <cls>
Playbook
./node.yml  -l <cls>    # Use Ansible playbook to add nodes in group <cls>
Example
bin/node-add pg-test    # Add nodes in pg-test group, runs ./node.yml -l pg-test

On managed nodes, create the cluster with: (Execute pgsql.yml playbook on <cls> group)

Script
bin/pgsql-add <cls>     # Create PostgreSQL cluster <cls>
Playbook
./pgsql.yml -l <cls>    # Use Ansible playbook to create PostgreSQL cluster <cls>
Example
bin/pgsql-add pg-test   # Create pg-test cluster

Example: Create 3-node PG cluster pg-test

demo/pgsql.cast
Risk: Re-running create on existing cluster

If you re-run create on an existing cluster, Pigsty won’t remove existing data files, but service configs will be overwritten and the cluster will restart! Additionally, if you specified a baseline SQL in database definition, it will re-execute - if it contains delete/overwrite logic, data loss may occur.

Expand Cluster

To add a new replica to an existing PostgreSQL cluster, add the instance definition to inventory: all.children.<cls>.hosts.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary } # existing member
    10.10.10.12: { pg_seq: 2, pg_role: replica } # existing member
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- new member
  vars: { pg_cluster: pg-test }

Scaling out is similar to creating a cluster. First add the new node to Pigsty: Add Node:

Script
bin/node-add <ip>       # Add node with IP <ip>
Playbook
./node.yml -l <ip>      # Use Ansible playbook to add node <ip>
Example
bin/node-add 10.10.10.13    # Add node 10.10.10.13, runs ./node.yml -l 10.10.10.13

Then run the following on the new node to scale out (Install PGSQL module on new node with same pg_cluster):

Script
bin/pgsql-add <cls> <ip>  # Add node <ip> to cluster
Playbook
./pgsql.yml -l <ip>       # Core: Use Ansible playbook to install PGSQL module on <ip>
Example
bin/pgsql-add pg-test 10.10.10.13   # Scale out pg-test with node 10.10.10.13

After scaling, you should Reload Service to add the new member to load balancer.

Example: Add replica 10.10.10.13 to 2-node cluster pg-test

demo/pgsql-append.cast

Shrink Cluster

To remove a replica from an existing PostgreSQL cluster, remove the instance definition from inventory all.children.<cls>.hosts.

Scale-in stops the instance and deletes its data directory by default. First run pig pg list <cls> and pig pb info, verify that the target is not the primary and that a recent restorable backup exists, then have the operator enter the exact <ip> and execute only after confirmation.

First uninstall PGSQL module from target node (Execute pgsql-rm.yml on <ip>):

Script
bin/pgsql-rm <cls> <ip>   # Remove the PostgreSQL instance on <ip> from cluster <cls>
Playbook
./pgsql-rm.yml -l <ip>    # Directly remove the PostgreSQL instance on <ip> with the Ansible playbook
Example
bin/pgsql-rm pg-test 10.10.10.13  # Remove node 10.10.10.13 from pg-test

After removing PGSQL module, optionally remove the node from Pigsty: Remove Node:

Script
bin/node-rm <ip>          # Remove <ip> from Pigsty management
Playbook
./node-rm.yml -l <ip>     # Directly remove <ip> from Pigsty management with the Ansible playbook
Example
bin/node-rm 10.10.10.13   # Remove 10.10.10.13 from Pigsty management

After scaling in, remove the instance from inventory, then Reload Service to remove it from load balancer.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- remove after execution
  vars: { pg_cluster: pg-test }

Example: Remove replica 10.10.10.13 from 3-node cluster pg-test

demo/pgsql-shrink.cast

Remove Cluster

To destroy a cluster, uninstall PGSQL module from all nodes (Execute pgsql-rm.yml on <cls>):

This is irreversible data deletion. Inspect pig pg list <cls> and pig pb info, verify a recent backup and any independent copy to retain, and have the operator enter the exact cluster name. The commands below perform the corresponding destruction directly.

Script
bin/pgsql-rm <cls>        # Destroy the entire PostgreSQL cluster <cls>
Playbook
./pgsql-rm.yml -l <cls>   # Directly destroy the entire PostgreSQL cluster <cls> with the Ansible playbook
Example
bin/pgsql-rm pg-test      # Destroy cluster pg-test

After destroying PGSQL, optionally remove all nodes from Pigsty: Remove Node (optional if other services exist):

Script
bin/node-rm <cls>         # Remove every node in group <cls> from Pigsty management
Playbook
./node-rm.yml -l <cls>    # Directly remove the nodes in group <cls> from Pigsty management with the Ansible playbook
Example
bin/node-rm pg-test       # Remove every node in group pg-test from Pigsty management

After removal, delete the entire cluster definition from inventory.

pg-test: # remove this cluster definition group
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars: { pg_cluster: pg-test }

Example: Destroy 3-node PG cluster pg-test

demo/pgsql-rm.cast

Note: If pg_safeguard is configured (or globally true), pgsql-rm.yml will abort to prevent accidental removal. Override with playbook command line to force removal. By default, cluster backup repo is deleted with the cluster. To preserve backups (e.g., with centralized repo), set pg_rm_backup=false:

./pgsql-rm.yml -l pg-meta -e pg_safeguard=false    # Force removal of protected pg-meta
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Preserve its backup repository while removing the cluster

Reload Service

PostgreSQL clusters expose services via HAProxy on host nodes. When service definitions, instance weights, or cluster membership change (for example, scale out or scale in), reload services to update HAProxy’s static member configuration. The default Primary and Replica services detect the current role through Patroni REST API health checks, so ordinary switchover or failover reroutes automatically and does not require regenerating HAProxy configuration.

To reload service config on entire cluster or specific instances (Execute pg_service subtask of pgsql.yml on <cls> or <ip>):

Script
bin/pgsql-svc <cls>           # Reload service config for entire cluster <cls>
bin/pgsql-svc <cls> <ip...>   # Reload service config for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_service -e pg_reload=true        # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_service -e pg_reload=true        # Reload specific instance
Example
bin/pgsql-svc pg-test                 # Reload pg-test cluster service config
bin/pgsql-svc pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance service config
Note

If you use a dedicated load-balancer cluster (pg_service_provider), only reloading the cluster primary updates the load-balancer configuration.

Example: Reload pg-test cluster service config

demo/pgsql-svc.cast
Example: Reload PG Service to Remove Instance

asciicast

Reload HBA

When HBA configs change, reload HBA rules to apply. (pg_hba_rules / pgb_hba_rules) If you have inventory-role-specific HBA rules or address ranges that reference cluster member aliases, reload HBA after changing pg_role labels or scaling the cluster. Role selectors use static inventory variables and do not change automatically after a Patroni switchover.

To reload PG and Pgbouncer HBA rules on entire cluster or specific instances (Execute HBA subtasks of pgsql.yml on <cls> or <ip>):

Script
bin/pgsql-hba <cls>           # Reload HBA rules for entire cluster <cls>
bin/pgsql-hba <cls> <ip...>   # Reload HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload specific instance
Example
bin/pgsql-hba pg-test                 # Reload pg-test cluster HBA rules
bin/pgsql-hba pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance HBA rules

Example: Reload pg-test cluster HBA rules

demo/pgsql-hba.cast

Config Cluster

PostgreSQL config params are managed by Patroni. Initial params are specified by Patroni config template. After cluster init, config is stored in Etcd, dynamically managed and synced by Patroni. Most Patroni config params can be modified via patronictl. Other params (e.g., etcd DCS config, log/RestAPI config) can be updated via subtasks. For example, when etcd cluster membership changes, refresh Patroni config:

./pgsql.yml -l pg-test -t pg_conf                   # Update Patroni config file
ansible pg-test -b -a 'systemctl reload patroni'    # Reload Patroni service

You can override Patroni-managed defaults at different levels: specify params per instance, specify params per user, or specify params per database.

Clone Cluster

Two ways to clone a cluster: use Standby Cluster, or use Point-in-Time Recovery. The former is simple and requires no backup repository, but it does require a reachable replication upstream and can clone only the latest state. The latter requires a centralized backup repository such as Silo and can clone to any point within the retention period.

Method Pros Cons Use Cases
Standby Cluster No backup repository needed Requires reachable upstream; latest state only DR, read-write separation, migration
PITR Recover to any point Requires centralized backup Undo mistakes, data audit

Clone via Standby Cluster

Standby Cluster continuously syncs from upstream cluster via streaming replication - the simplest cloning method. Specify pg_upstream on the new cluster primary to auto-pull data from upstream.

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is standby cluster (clone) of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.11 }  # specify upstream
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

Create standby cluster with:

Script
bin/pgsql-add pg-test2    # Create standby cluster, auto-clone from upstream pg-test
Playbook
./pgsql.yml -l pg-test2   # Use Ansible playbook to create standby cluster

Standby cluster follows upstream, keeping data in sync. Promote to independent cluster anytime:

Example: Promote Standby to Independent Cluster

Via Config Cluster, remove standby_cluster config to promote:

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y

After promotion, pg-test2 becomes independent cluster accepting writes, forked from pg-test.

Example: Change Replication Upstream

If upstream cluster switchover occurs, change standby cluster upstream via Config Cluster:

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.11     # <--- old upstream
+  host: 10.10.10.14     # <--- new upstream
   port: 5432

Apply these changes? [y/N]: y

Clone via PITR

Point-in-Time Recovery (PITR) allows recovery to any point within backup retention. Requires a centralized backup repository (Silo/S3), but is more powerful.

To clone via PITR, add pg_pitr param specifying recovery target:

# Clone new cluster pg-meta2 from pg-meta backup
pg-meta2:
  hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta2
    pg_pitr:
      cluster: pg-meta                    # Recover from pg-meta backup
      time: '2025-01-10 10:00:00+00'      # Recover to specific time
      archive: false                       # Disable archiving during the independent restore
      action: promote                      # Promote after replay in this one-shot example

Execute clone with pgsql-pitr.yml playbook:

Playbook
./pgsql-pitr.yml -l pg-meta2    # Use the explicitly declared action: promote above
CLI
# Specify PITR options via command line
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": {"cluster": "pg-meta", "time": "2025-01-10 10:00:00+00", "archive": false, "action": "promote"}}'

PITR supports multiple recovery target types:

Target Type Example Description
Time time: "2025-01-10 10:00:00+00" Recover to specific timestamp
XID xid: "250000" Recover to before/after txn
Name name: "before_migration" Recover to named restore point
LSN lsn: "0/4001C80" Recover to specific WAL pos
Latest pg_pitr: {} Recover to end of WAL archive
Post-PITR Processing

Pigsty v5.0 PITR keeps archiving enabled by default (archive: true). If you explicitly set archive: false for exploratory recovery, reset archive_mode, restart the cluster, and perform a new full backup after confirming the recovered data is correct:

psql -c 'ALTER SYSTEM RESET archive_mode;'
pg restart <cls>
pg-backup full    # Execute new full backup

For detailed PITR usage, see Restore Operations documentation.

4.2 - Managing PostgreSQL Users

User management - create, modify, delete users, manage role membership, connection pool config

Quick Start

Pigsty uses declarative management: first define users in the inventory, then use bin/pgsql-user <cls> <username> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users: [{ name: dbuser_app, password: 'DBUser.App', pgbouncer: true }]  # <--- Define user list here!
Script
bin/pgsql-user <cls> <username>    # Create/modify <username> user on <cls> cluster
Playbook
./pgsql-user.yml -l pg-meta -e username=dbuser_app    # Use playbook to create/modify user
Example
bin/pgsql-user pg-meta dbuser_app    # Create/modify dbuser_app user on pg-meta cluster

For the complete user definition reference, see User Configuration. See Access Control for roles and privileges, and Authentication for credential management.

name is the key used by pgsql-user.yml to look up a user definition; the playbook does not rename roles. For a rename, create the replacement role, migrate ownership, memberships, and client credentials, validate the cutover, and only then remove the old role. Do not treat delete-and-create as a lossless rename.

Action Command Description
Create User bin/pgsql-user <cls> <user> Create new business user or role
Modify User bin/pgsql-user <cls> <user> Modify existing user properties
Delete User bin/pgsql-user <cls> <user> Dependency-aware destructive deletion (state: absent)
demo/pgsql-user.cast

Create User

Users defined in pg_users are auto-created during PostgreSQL cluster creation in the pg_user task.

To create a new user on an existing cluster, add user definition to all.children.<cls>.pg_users, then execute:

Script
bin/pgsql-user <cls> <username>   # Create user <username>
Playbook
./pgsql-user.yml -l <cls> -e username=<username>   # Use Ansible playbook
Example
bin/pgsql-user pg-meta dbuser_app    # Create dbuser_app user in pg-meta cluster

Example: Create business user dbuser_app

#all.children.pg-meta.vars.pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]
    comment: application user for myapp

Result: Creates dbuser_app user on primary, sets password, grants dbrole_readwrite role, adds to Pgbouncer pool, reloads Pgbouncer config on all instances.

Recommendation: Use playbook

For manual user creation, you must ensure Pgbouncer user list sync yourself.

Modify User

Same command as create - playbook is idempotent. When target user exists, Pigsty modifies properties to match config.

Script
bin/pgsql-user <cls> <user>   # Modify user <user> properties
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Idempotent, can repeat
Example
bin/pgsql-user pg-meta dbuser_app    # Modify dbuser_app to match config

Not directly mutable: name is the identity key in the declarative definition. The playbook does not rename an existing role. Use a controlled create, ownership/privilege and client migration, validation, and old-role removal sequence.

All other properties can be modified. Common examples:

Modify password: Update password field. Logging is temporarily disabled during password change to prevent leakage.

- name: dbuser_app
  password: NewSecretPassword     # New password

Modify privilege attributes: Configure boolean flags for user privileges.

- name: dbuser_app
  superuser: false           # Superuser (use carefully!)
  createdb: true             # Allow CREATE DATABASE
  createrole: false          # Allow CREATE ROLE
  inherit: true              # Auto-inherit role privileges
  replication: false         # Allow streaming replication
  bypassrls: false           # Bypass row-level security
  connlimit: 50              # Connection limit, -1 unlimited

Modify expiration: Use expire_in for relative expiry (N days), or expire_at for absolute date. expire_in takes priority and recalculates on each playbook run - good for temp users needing periodic renewal.

- name: temp_user
  expire_in: 30                   # Expires in 30 days (relative)

- name: contractor_user
  expire_at: '2024-12-31'         # Expires on date (absolute)

- name: permanent_user
  expire_at: 'infinity'           # Never expires

Modify role membership: Use roles array with simple or extended format. Role membership is additive - won’t remove undeclared existing roles. Use state: absent to explicitly revoke.

- name: dbuser_app
  roles:
    - dbrole_readwrite                      # Simple form: grant role
    - { name: dbrole_admin, admin: true }   # With ADMIN OPTION
    - { name: pg_monitor, set: false }      # PG16+: disallow SET ROLE
    - { name: old_role, state: absent }     # Revoke role membership

Manage user parameters: Use parameters dict for user-level params, generates ALTER USER ... SET. Use DEFAULT to reset.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: DEFAULT        # Reset to default

Connection pool config: Set pgbouncer: true to add user to pool. Optional pool_mode and pool_connlimit.

- name: dbuser_app
  pgbouncer: true                 # Add to pool
  pool_mode: transaction          # Pool mode
  pool_connlimit: 50              # Max user connections

Delete User

Deleting a user terminates sessions, transfers object ownership, revokes grants, and runs DROP ROLE; it is irreversible. Confirm the exact cluster, role, successor owner, and a recent backup before setting the user to state: absent and applying the change.

Script
bin/pgsql-user <cls> <user>   # Actual deletion after confirmation; config must say state: absent
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Apply the user definition with state: absent
Example
bin/pgsql-user pg-meta dbuser_old    # Delete dbuser_old (config has state: absent)

Config example:

pg_users:
  - name: dbuser_old
    state: absent

Deletion process: On the primary, the task runs pg-drop-role <user> postgres --force. It disables login, terminates active sessions, transfers database and tablespace ownership plus objects in each connectable database to postgres, runs DROP OWNED to remove grants, revokes role memberships, and finally runs DROP ROLE. A pre-change audit snapshot is written to /tmp/pg_drop_role_<user>_<timestamp>.log.

Protection: The Ansible task skips postgres and the replication, admin, and monitor usernames configured in inventory. When invoked directly, pg-drop-role protects only the hard-coded default names postgres, replicator, dbuser_dba, and dbuser_monitor; renamed system accounts are not recognized automatically.

Dependency-aware, not transactional

pg-drop-role skips DROP OWNED in a database if its preceding REASSIGN OWNED fails, but the cross-database procedure is not one transaction. A mid-run failure can leave the role NOLOGIN, some ownership already transferred, or dependencies still present. The v4.5 Ansible task also uses ignore_errors, so a playbook result is not sufficient evidence. Verify role absence, successor ownership, application cutover, and the audit log afterward.

In v4.5, pgsql-user.yml reloads Pgbouncer but does not reliably prune a deleted role from /etc/pgbouncer/userlist.txt. Check every cluster instance after deletion:

sudo -iu postgres psql -AXtwc "SELECT 1 FROM pg_roles WHERE rolname = 'dbuser_old';"
grep -n '^"dbuser_old"[[:space:]]' /etc/pgbouncer/userlist.txt

If an exact Pgbouncer entry remains, remove that single line under change control, reload Pgbouncer, and validate application connections. Do not use a broad pattern to delete entries.

Manual Deletion

For manual user deletion, use pg-drop-role script directly:

# Check dependencies (read-only)
pg-drop-role dbuser_old --check

# Preview deletion (don't execute)
pg-drop-role dbuser_old --dry-run -v

# Only after confirming a recent backup, exact role, and successor owner
pg-drop-role dbuser_old dbuser_new

# Use --force only after explicitly approving session termination
pg-drop-role dbuser_old dbuser_new --force

Common Use Cases

Common user configuration examples:

Basic business user

- name: dbuser_app
  password: DBUser.App
  pgbouncer: true
  roles: [dbrole_readwrite]
  comment: application user

Read-only user

- name: dbuser_readonly
  password: DBUser.Readonly
  pgbouncer: true
  roles: [dbrole_readonly]

Admin user (can execute DDL)

- name: dbuser_admin
  password: DBUser.Admin
  pgbouncer: true
  pool_mode: session
  roles: [dbrole_admin]
  parameters:
    log_statement: 'all'

Temp user (expires in 30 days)

- name: temp_contractor
  password: TempPassword
  expire_in: 30
  roles: [dbrole_readonly]

Role (no login, for permission grouping)

- name: custom_role
  login: false
  comment: custom role for special permissions

User with advanced role options (PG16+)

- name: dbuser_special
  password: DBUser.Special
  pgbouncer: true
  roles:
    - dbrole_readwrite
    - { name: dbrole_admin, admin: true }
    - { name: pg_monitor, set: false }
    - { name: pg_execute_server_program, inherit: false }

Query Users

Common SQL queries for user info:

List all users

SELECT rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb,
       rolcanlogin, rolreplication, rolbypassrls, rolconnlimit, rolvaliduntil
FROM pg_roles WHERE rolname NOT LIKE 'pg_%' ORDER BY rolname;

View user role membership

SELECT r.rolname AS member, g.rolname AS role, m.admin_option, m.set_option, m.inherit_option
FROM pg_auth_members m
JOIN pg_roles r ON r.oid = m.member
JOIN pg_roles g ON g.oid = m.roleid
WHERE r.rolname = 'dbuser_app';

View user-level parameters

SELECT rolname, setconfig FROM pg_db_role_setting s
JOIN pg_roles r ON r.oid = s.setrole WHERE s.setdatabase = 0;

View expiring users

SELECT rolname, rolvaliduntil, rolvaliduntil - CURRENT_TIMESTAMP AS time_remaining
FROM pg_roles WHERE rolvaliduntil IS NOT NULL
  AND rolvaliduntil < CURRENT_TIMESTAMP + INTERVAL '30 days'
ORDER BY rolvaliduntil;

Connection Pool Management

Connection pool params in user definitions are applied to Pgbouncer when creating/modifying users.

Users with pgbouncer: true are added to /etc/pgbouncer/userlist.txt. User-level pool params (pool_mode, pool_connlimit) are configured via /etc/pgbouncer/useropts.txt.

Use postgres OS user with pgb alias to access Pgbouncer admin database. For more pool management, see Pgbouncer Management.

Manage Default-User Passwords

For a business user, follow Modify User: persist the new password in its pg_users definition, preview the scoped playbook, and then apply it. The three default users require extra coordination because other services consume their credentials.

Parameter Default Role Consumers
pg_admin_password DBUser.DBA dbuser_dba Admin clients, Pgbouncer, Infra service files, pgAdmin
pg_monitor_password DBUser.Monitor dbuser_monitor Exporters, Pgbouncer, Grafana data sources
pg_replication_password DBUser.Replicator replicator Patroni replication and .pgpass files

These accounts belong to pg_default_roles, not pg_users. pgsql-user.yml looks up only pg_users, so do not rotate a default password by overriding pg_users on the command line: that changes the business-user list visible to that run and exposes plaintext in shell history.

Rotate one account at a time:

  1. Persist the new parameter in pigsty.yml or the inventory actually in use; never put the plaintext password on the command line.
  2. On the current primary, open interactive psql as a superuser and run \password <username>; the meta-command reads the secret interactively.
  3. Run the corresponding refresh playbooks below after verifying the -l cluster/node scope.
  4. Keep the current administration session open and verify direct PostgreSQL, Pgbouncer, replication, exporters, and Grafana data sources before rotating another account.
# On the current primary of the target cluster
sudo -iu postgres psql -d postgres
\password dbuser_dba       # or dbuser_monitor / replicator

Refresh every consumer for the account. Replace <cls> and constrain infra to the actual targets:

# dbuser_dba: PG-node .pgpass, Pgbouncer, Infra admin files, and pgAdmin files
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass,env_pgscv,env_pgadmin

# dbuser_monitor: PG-node .pgpass, Pgbouncer, exporters, and Grafana data sources
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload,pg_exporter,pgbouncer_exporter,add_ds -e pg_reload=true
./infra.yml -l infra -t env_pgpass

# replicator: Patroni configuration plus PostgreSQL-node and Infra .pgpass files
./pgsql.yml -l <cls> -t pg_conf,pg_pass,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass

A mismatch between the replication role and Patroni nodes prevents new replication connections, so rotate that credential in a maintenance window and validate promptly. If VIBE or another module has rendered an admin connection string into its workspace context, rerender that module’s files as well.

Check for duplicate Infra .pgpass entries

In v4.5, env_pgpass adds the new line with lineinfile; it does not remove older lines by username. Because libpq uses the first matching line, inspect every target Infra node after the refresh and remove obsolete entries through controlled editing without printing secrets:

awk -F: '$4=="dbuser_dba" || $4=="dbuser_monitor" || $4=="replicator" {print NR, $4}' ~/.pgpass

patroni_password protects the Patroni REST API; it is not a PostgreSQL role password. After changing it in inventory, refresh the target PostgreSQL cluster and Infra management side separately:

./pgsql.yml -l <cls> -t pg_conf,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_patroni

Then validate authentication and cluster state with patronictl or pig pg list <cls>.

4.3 - Managing PostgreSQL Databases

Database management - create, modify, delete, rebuild, and clone databases using templates

Quick Start

Pigsty uses declarative management: first define databases in the inventory, then use bin/pgsql-db <cls> <dbname> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases: [{ name: some_db }]  # <--- Define database list here!
Script
bin/pgsql-db <cls> <dbname>    # Create/modify <dbname> database on <cls> cluster
Playbook
./pgsql-db.yml -l pg-meta -e dbname=some_db    # Use playbook to create/modify database
Example
bin/pgsql-db pg-meta some_db    # Create/modify some_db database on pg-meta cluster

For the complete database definition reference, see Database Configuration. For database access permissions, see Access Control: Database Isolation.

Note: Some parameters can only be specified at creation time. Modifying these requires recreating the database (use state: recreate).

Action Command Description
Create Database bin/pgsql-db <cls> <db> Create new business database
Modify Database bin/pgsql-db <cls> <db> Modify existing database properties
Delete Database bin/pgsql-db <cls> <db> Delete database (requires state: absent)
Rebuild Database bin/pgsql-db <cls> <db> Drop and recreate (requires state: recreate)
Clone Database bin/pgsql-db <cls> <db> Clone database using template
demo/pgsql-db.cast

Create Database

Databases defined in pg_databases are auto-created during PostgreSQL cluster creation in the pg_db task.

To create a new database on an existing cluster, add database definition to all.children.<cls>.pg_databases, then execute:

Script
bin/pgsql-db <cls> <dbname>   # Create database <dbname>
Playbook
./pgsql-db.yml -l <cls> -e dbname=<dbname>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta myapp    # Create myapp database in pg-meta cluster

Example: Create business database myapp

#all.children.pg-meta.vars.pg_databases:
  - name: myapp
    owner: dbuser_myapp
    schemas: [app]
    extensions:
      - { name: pg_trgm }
      - { name: btree_gin }
    comment: my application database

Result: Creates myapp database on primary, sets owner to dbuser_myapp, creates app schema, enables pg_trgm and btree_gin extensions. Database is auto-added to Pgbouncer pool and registered as Grafana datasource.

Recommendation: Use playbook

For manual database creation, you must ensure Pgbouncer pool and Grafana datasource sync yourself.

Modify Database

Same command as create - playbook is idempotent when no baseline SQL is defined.

When target database exists, Pigsty modifies properties to match config. However, some properties can only be set at creation.

Script
bin/pgsql-db <cls> <db>   # Modify database <db> properties
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Idempotent, can repeat
Example
bin/pgsql-db pg-meta myapp    # Modify myapp database to match config

Immutable properties: These can’t be modified after creation, require state: recreate:

  • name (database name), template, strategy (clone strategy)
  • encoding, locale/lc_collate/lc_ctype, locale_provider/icu_locale/icu_rules/builtin_locale

All other properties can be modified. Common examples:

Modify owner: Update owner field, executes ALTER DATABASE ... OWNER TO and grants permissions.

- name: myapp
  owner: dbuser_new_owner     # New owner

Modify connection limit: Use connlimit to limit max connections.

- name: myapp
  connlimit: 100              # Max 100 connections

Revoke public connect: Setting revokeconn: true revokes PUBLIC CONNECT privilege, allowing only owner, DBA, monitor, and replication users.

- name: myapp
  owner: dbuser_myapp
  revokeconn: true            # Revoke PUBLIC CONNECT

Manage parameters: Use parameters dict for database-level params, generates ALTER DATABASE ... SET. Use special value DEFAULT to reset.

- name: myapp
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '30s'
    search_path: DEFAULT      # Reset to default

Manage schemas: Use schemas array with simple or extended format. Use state: absent to drop (CASCADE).

- name: myapp
  schemas:
    - app                                   # Simple form
    - { name: core, owner: dbuser_myapp }   # Specify owner
    - { name: deprecated, state: absent }   # Drop schema

Manage extensions: Use extensions array with simple or extended format. Use state: absent to uninstall (CASCADE).

- name: myapp
  extensions:
    - postgis                                 # Simple form
    - { name: vector, schema: public }        # Specify schema
    - { name: pg_trgm, state: absent }        # Uninstall extension
CASCADE Warning

Dropping schemas or uninstalling extensions uses CASCADE, deleting all dependent objects. Understand impact before executing.

Connection pool config: By default all databases are added to Pgbouncer. Configure pgbouncer, pool_mode, pool_size, pool_reserve, pool_size_min, pool_connlimit, and pool_auth_user.

- name: myapp
  pgbouncer: true              # Add to pool (default true)
  pool_mode: transaction       # Pool mode: transaction/session/statement
  pool_size: 50                # Default pool size
  pool_reserve: 30             # Reserve pool size
  pool_size_min: 0             # Minimum pool size
  pool_connlimit: 100          # Max database connections
  pool_auth_user: dbuser_meta  # Auth query user (with pgbouncer_auth_query)

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Delete Database

To delete a database, set state to absent and execute:

Script
bin/pgsql-db <cls> <db>   # Delete <db> (config must have state: absent)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta olddb    # Delete olddb (config has state: absent)

Config example:

pg_databases:
  - name: olddb
    state: absent

Deletion process: If is_template: true, first executes ALTER DATABASE ... IS_TEMPLATE false; uses DROP DATABASE ... WITH (FORCE) (PG13+) to force drop and terminate all connections; removes from Pgbouncer pool; unregisters from Grafana datasource.

Protection: System databases postgres, template0, template1 cannot be deleted. Deletion only runs on primary - streaming replication syncs to replicas.

Danger Warning

Database deletion is irreversible - permanently deletes all data. Before executing: ensure recent backup exists, confirm no business uses the database, notify stakeholders. Pigsty is not responsible for any data loss from database deletion. Use at your own risk.

Rebuild Database

recreate state rebuilds database (drop then create):

Script
bin/pgsql-db <cls> <db>   # Rebuild <db> (config must have state: recreate)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta testdb    # Rebuild testdb (config has state: recreate)

Config example:

pg_databases:
  - name: testdb
    state: recreate
    owner: dbuser_test
    baseline: test_init.sql    # Execute after rebuild

Use cases: Test environment reset, clear dev database, modify immutable properties (encoding, locale), restore to initial state.

Difference from manual DROP + CREATE: Single command; auto-preserves Pgbouncer and Grafana config; auto-loads baseline init script.

Clone Database

Clone PostgreSQL databases using PG template mechanism. During cloning, no active connections to template database are allowed.

Script
bin/pgsql-db <cls> <db>   # Clone <db> (config must specify template)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta meta_dev    # Clone meta_dev (config has template: meta)

Config example:

pg_databases:
  - name: meta                   # Source database

  - name: meta_dev
    template: meta               # Use meta as template
    strategy: FILE_COPY          # PG15+ clone strategy, instant on PG18

Instant Clone (PG18+): If using PostgreSQL 18+, Pigsty defaults file_copy_method. With strategy: FILE_COPY, database clone completes in ~200ms without copying data files. E.g., cloning 30GB database: normal takes 18s, instant takes 200ms.

Manual clone: Ensure all connections to template are terminated:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'meta';
CREATE DATABASE meta_dev TEMPLATE meta STRATEGY FILE_COPY;

Limitations: Instant clone only available on supported filesystems (xfs, btrfs, zfs, apfs); don’t use postgres database as template; in high-concurrency environments, all template connections must be cleared within clone window (~200ms).

Connection Pool Management

Connection pool params in database definitions are applied to Pgbouncer when creating/modifying databases.

By default all databases are added to Pgbouncer pool (pgbouncer: true). Databases are added to /etc/pgbouncer/database.txt. Database-level pool params (pool_auth_user, pool_mode, pool_size, pool_reserve, pool_size_min, pool_connlimit) are configured via this file.

Use postgres OS user with pgb alias to access Pgbouncer admin database. For more pool management, see Pgbouncer Management.

4.4 - Patroni HA Management

Manage PostgreSQL cluster HA with Patroni, including config changes, status check, switchover, restart, and reinit replica.

Overview

Pigsty uses Patroni to manage PostgreSQL clusters. It handles config changes, status checks, switchover, restart, reinit replicas, and more.

To use Patroni for management, you need one of the following identities:

Patroni provides patronictl CLI for management. Pigsty provides a wrapper alias pg to simplify operations.

Using patronictl via pg alias
pg ()
{
    local patroni_conf="/infra/conf/patronictl.yml";
    if [ ! -r ${patroni_conf} ]; then
        patroni_conf="/etc/patroni/patroni.yml";
        if [ ! -r ${patroni_conf} ]; then
            echo "error: patronictl config not found";
            return 1;
        fi;
    fi;
    patronictl -c ${patroni_conf} "$@"
}

Available Commands

Command Function Description
edit-config Edit Config Interactively edit cluster Patroni/PostgreSQL config
list List Status List cluster members and their status
switchover Switchover Switch primary role to specified replica (planned)
failover Failover Force failover to specified replica (emergency)
restart Restart Restart PostgreSQL instance to apply restart-required params
reload Reload Reload Patroni config (no restart needed)
reinit Reinit Replica Reinitialize replica (wipe data and re-clone)
pause Pause Auto-Failover Pause Patroni automatic failover
resume Resume Auto-Failover Resume Patroni automatic failover
history View History Show cluster failover history
show-config Show Config Display current cluster config (read-only)
query Execute Query Execute SQL query on cluster members
topology View Topology Display cluster replication topology
version View Version Display Patroni version info
remove Remove Member Remove cluster member from DCS (dangerous)

Edit Config

Use edit-config to interactively edit cluster Patroni and PostgreSQL config. This opens an editor to modify config stored in DCS, automatically applying changes to all members. You can change Patroni params (ttl, loop_wait, synchronous_mode, etc.) and PostgreSQL params in postgresql.parameters.

pg edit-config <cls>                  # Interactive edit cluster config
pg edit-config <cls> --force          # Skip confirmation and apply directly
pg edit-config <cls> -p <k>=<v>       # Modify PostgreSQL param (--pg shorthand)
pg edit-config <cls> -s <k>=<v>       # Modify Patroni param (--set shorthand)
demo/pgsql-config.cast

Common config modification examples:

# Modify PostgreSQL param: slow query threshold (prompts for confirmation)
pg edit-config pg-test -p log_min_duration_statement=1000

# Modify PostgreSQL param, skip confirmation
pg edit-config pg-test -p log_min_duration_statement=1000 --force

# Modify multiple PostgreSQL params
pg edit-config pg-test -p work_mem=256MB -p maintenance_work_mem=1GB --force

# Modify Patroni params: increase failure detection window (increase RTO)
pg edit-config pg-test -s loop_wait=15 -s ttl=60 --force

# Modify Patroni param: enable synchronous replication mode
pg edit-config pg-test -s synchronous_mode=true --force

# Modify Patroni param: enable strict synchronous mode (require at least one sync replica for writes)
pg edit-config pg-test -s synchronous_mode_strict=true --force

# Modify restart-required params (need pg restart after)
pg edit-config pg-test -p shared_buffers=4GB --force
pg edit-config pg-test -p shared_preload_libraries='timescaledb, pg_stat_statements' --force
pg edit-config pg-test -p max_connections=200 --force

Some params require PostgreSQL restart to take effect. Use pg list to check - instances marked with * need restart. Then use pg restart to apply. You can also use curl or programs to call Patroni REST API:

# View current config
curl -s 10.10.10.11:8008/config | jq .

# Modify params via API (requires auth)
curl -u 'postgres:Patroni.API' \
     -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
     -s -X PATCH http://10.10.10.11:8008/config | jq .

List Status

Use list to view cluster members and status. Output shows each instance’s name, host, role, state, timeline, and replication lag. This is the most commonly used command for checking cluster health.

pg list <cls>                         # List specified cluster status
pg list                               # List all clusters (on admin node)
pg list <cls> -e                      # Show extended info (--extended)
pg list <cls> -t                      # Show timestamp (--timestamp)
pg list <cls> -f json                 # Output as JSON (--format)
pg list <cls> -W 5                    # Refresh every 5 seconds (--watch)

Example output:

+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+

Column descriptions: Member is instance name, composed of pg_cluster-pg_seq; Host is instance IP; Role is role type - Leader (primary), Replica, Sync Standby, Standby Leader (cascade primary); State is running state - running, streaming, in archive recovery, starting, stopped, etc.; TL is timeline number, incremented after each switchover; Lag in MB is replication lag in MB (not shown for primary).

Instances requiring restart show * after the name:

+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

Switchover

Use switchover for planned primary-replica switchover. Switchover is graceful: Patroni ensures replica is fully synced, demotes primary, then promotes target replica. Takes seconds with brief write unavailability. Use for primary host maintenance, upgrades, or migrating primary to better nodes.

pg switchover <cls>                   # Interactive switchover, prompts for target replica
pg switchover <cls> --leader <old>    # Specify current primary name
pg switchover <cls> --candidate <new> # Specify target replica name
pg switchover <cls> --scheduled <time> # Scheduled switchover, format: 2024-12-01T03:00
pg switchover <cls> --force           # Skip confirmation

Before switchover, ensure all replicas are healthy (running or streaming), replication lag is acceptable, and stakeholders are notified.

# Interactive switchover (recommended, shows topology and prompts for selection)
$ pg switchover pg-test
Current cluster topology
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
Primary [pg-test-1]:
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
When should the switchover take place (e.g. 2024-01-01T12:00) [now]:
Are you sure you want to switchover cluster pg-test, demoting current leader pg-test-1? [y/N]: y

# Non-interactive switchover (specify primary and candidate)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --force

# Scheduled switchover (at 3 AM, for maintenance window)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --scheduled "2024-12-01T03:00"

After switchover, use pg list to confirm new cluster topology.

Failover

Use failover for emergency failover. Unlike switchover, failover is for when primary is unavailable. It directly promotes a replica without waiting for original primary confirmation. Since replicas may not be fully synced, failover may cause minor data loss. Use switchover for non-emergency situations.

pg failover <cls>                     # Interactive failover
pg failover <cls> --candidate <new>   # Specify replica to promote
pg failover <cls> --force             # Skip confirmation

Failover examples:

# Interactive failover
$ pg failover pg-test
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
Are you sure you want to failover cluster pg-test? [y/N]: y
Successfully failed over to "pg-test-2"

# Non-interactive failover (for emergencies)
pg failover pg-test --candidate pg-test-2 --force

Switchover vs Failover: Switchover is for planned maintenance, requires original primary online, ensures full sync before switching, no data loss; Failover is for emergency recovery, original primary can be offline, directly promotes replica, may lose unsynced data. Use Switchover for daily maintenance/upgrades; use Failover only when primary is completely down and unrecoverable.

The built-in Patroni failover subcommand currently has no --leader option. Use planned switchover --leader ... when you need to validate or name the old primary; failover accepts only the candidate replica.

Restart

Use restart to restart PostgreSQL instances, typically to apply restart-required parameter changes. When run against the whole cluster, patronictl submits each selected member in turn but does not guarantee a replica-first, leader-last order. If that order matters, restart replicas by role and then restart the leader separately.

pg restart <cls>                      # Restart all instances in cluster
pg restart <cls> <member>             # Restart specific instance
pg restart <cls> --role leader        # Restart primary only
pg restart <cls> --role replica       # Restart all replicas
pg restart <cls> --pending            # Restart only instances marked for restart
pg restart <cls> --scheduled <time>   # Scheduled restart
pg restart <cls> --timeout <sec>      # Set restart timeout (seconds)
pg restart <cls> --force              # Skip confirmation

After modifying restart-required params (shared_buffers, shared_preload_libraries, max_connections, max_worker_processes, etc.), use this command.

# Check which instances need restart (marked with *)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

# Restart single replica
pg restart pg-test pg-test-2

# Restart all cluster members (leader-last order is not guaranteed)
pg restart pg-test --force

# Restart only pending instances
pg restart pg-test --pending --force

# Explicitly restart replicas first, then the leader
pg restart pg-test --role replica --force
pg restart pg-test --role leader --force

# Scheduled restart (for maintenance window)
pg restart pg-test --scheduled "2024-12-01T03:00"

# Set restart timeout to 300 seconds
pg restart pg-test --timeout 300 --force

Reload

Use reload to reload Patroni config without restarting PostgreSQL. This re-reads config files and applies non-restart params via pg_reload_conf(). Lighter than restart - doesn’t interrupt connections or running queries.

pg reload <cls>                       # Reload entire cluster config
pg reload <cls> <member>              # Reload specific instance config
pg reload <cls> --role leader         # Reload primary only
pg reload <cls> --role replica        # Reload all replicas
pg reload <cls> --force               # Skip confirmation

Most PostgreSQL params work via reload. Only postmaster-context params (shared_buffers, max_connections, shared_preload_libraries, archive_mode, etc.) require restart.

# Reload entire cluster
pg reload pg-test

# Reload single instance
pg reload pg-test pg-test-1

# Force reload, skip confirmation
pg reload pg-test --force

Reinit Replica

Use reinit to reinitialize a replica. This deletes all data on the replica and rebuilds it according to Patroni’s create_replica_methods order. Pigsty tries basebackup (pg_basebackup) first by default; when a remote pgBackRest repository is enabled, pgbackrest is also configured as a fallback. Use this when replica data is corrupted, the replica is too far behind and required WAL has been removed, or replica configuration must be reset.

pg reinit <cls> <member>              # Reinitialize specified replica
pg reinit <cls> <member> --force      # Skip confirmation
pg reinit <cls> <member> --wait       # Wait for rebuild to complete
Destructive operation

This operation deletes all data on the target instance. Run it only on a replica, never on the primary.

# Reinitialize replica (prompts for confirmation)
$ pg reinit pg-test pg-test-2
Are you sure you want to reinitialize members pg-test-2? [y/N]: y
Success: reinitialize for member pg-test-2

# Force reinitialize, skip confirmation
pg reinit pg-test pg-test-2 --force

# Reinitialize and wait for completion
pg reinit pg-test pg-test-2 --force --wait

During rebuild, use pg list to check progress. Replica state shows creating replica:

+ Cluster: pg-test (7322261897169354773) --------------+----+------+
| Member    | Host        | Role    | State            | TL | Lag  |
+-----------+-------------+---------+------------------+----+------+
| pg-test-1 | 10.10.10.11 | Leader  | running          |  2 |      |
| pg-test-2 | 10.10.10.12 | Replica | creating replica |    |    ? |
+-----------+-------------+---------+------------------+----+------+

Pause

Use pause to pause Patroni automatic failover. When paused, Patroni won’t auto-promote replicas even if primary fails. Use for planned maintenance windows (prevent accidental triggers), debugging (prevent cluster state changes), or manual switchover timing control.

pg pause <cls>                        # Pause automatic failover
pg pause <cls> --wait                 # Pause and wait for all members to confirm
Warning

While paused, the cluster will not recover automatically if the primary fails. Run resume after maintenance.

# Pause automatic failover
$ pg pause pg-test
Success: cluster management is paused

# Check cluster status (shows Maintenance mode: on)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
 Maintenance mode: on

Resume

Use resume to resume Patroni automatic failover. Execute immediately after maintenance to ensure cluster auto-recovers on primary failure.

pg resume <cls>                       # Resume automatic failover
pg resume <cls> --wait                # Resume and wait for all members to confirm
# Resume automatic failover
$ pg resume pg-test
Success: cluster management is resumed

# Confirm resumed (Maintenance mode prompt disappears)
$ pg list pg-test

History

Use history to view cluster failover history. Each switchover (auto or manual) creates a new timeline record.

pg history <cls>                      # Show failover history
pg history <cls> -f json              # Output as JSON
pg history <cls> -f yaml              # Output as YAML
$ pg history pg-test
+----+-----------+------------------------------+---------------------------+
| TL |       LSN | Reason                       | Timestamp                 |
+----+-----------+------------------------------+---------------------------+
|  1 | 0/5000060 | no recovery target specified | 2024-01-15T10:30:00+08:00 |
|  2 | 0/6000000 | switchover to pg-test-2      | 2024-01-20T14:00:00+08:00 |
|  3 | 0/7000028 | failover to pg-test-1        | 2024-01-25T09:15:00+08:00 |
+----+-----------+------------------------------+---------------------------+

Column descriptions: TL is timeline number, incremented after each switchover, distinguishes primary histories; LSN is Log Sequence Number at switchover, marks WAL position; Reason is switchover reason - switchover to xxx (manual), failover to xxx (failure), or no recovery target specified (init); Timestamp is when switchover occurred.

Show Config

Use show-config to view current cluster config stored in DCS. This is read-only; use edit-config to modify.

pg show-config <cls>                  # Show cluster config
$ pg show-config pg-test
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=pg-test archive-push %p
    max_connections: 100
    shared_buffers: 256MB
    log_min_duration_statement: 1000
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30
synchronous_mode: false

Query

Use query to quickly execute SQL on cluster members. Convenient for debugging - for complex production queries, use psql or applications.

pg query <cls> -c "<sql>"             # Execute on primary
pg query <cls> -c "<sql>" -m <member> # Execute on specific instance (--member)
pg query <cls> -c "<sql>" -r leader   # Execute on primary (--role)
pg query <cls> -c "<sql>" -r replica  # Execute on all replicas
pg query <cls> -f <file>              # Execute SQL from file
pg query <cls> -c "<sql>" -U <user>   # Specify username (--username)
pg query <cls> -c "<sql>" -d <db>     # Specify database (--dbname)
pg query <cls> -c "<sql>" --format json  # Output as JSON
# Check primary connection count
pg query pg-test -c "SELECT count(*) FROM pg_stat_activity"

# Check PostgreSQL version
pg query pg-test -c "SELECT version()"

# Check replication status on all replicas
pg query pg-test -c "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn()" -r replica

# Execute on specific instance
pg query pg-test -c "SELECT pg_is_in_recovery()" -m pg-test-2

# Use specific user and database
pg query pg-test -c "SELECT current_user, current_database()" -U postgres -d postgres

# Output as JSON
pg query pg-test -c "SELECT * FROM pg_stat_replication" --format json

Topology

Use topology to view cluster replication topology as a tree. More intuitive than list for showing primary-replica relationships, especially for cascading replication.

pg topology <cls>                     # Show replication topology
$ pg topology pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1   | 10.10.10.11 | Leader  | running |  1 |              |
| + pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| + pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

In cascading replication, topology clearly shows replication hierarchy - e.g., pg-test-3 replicates from pg-test-2, which replicates from primary pg-test-1.

Version

Use version to view patronictl version.

pg version                            # Show patronictl version
$ pg version
patronictl version 4.1.0

Remove

Use remove to remove cluster or member metadata from DCS. This is dangerous - only removes DCS metadata, doesn’t stop PostgreSQL or delete data files. Misuse may cause cluster state inconsistency.

pg remove <cls>                       # Remove entire cluster metadata from DCS

Normally you don’t need this command. To properly remove clusters/instances, use Pigsty’s bin/pgsql-rm script or pgsql-rm.yml playbook. Only consider remove for: orphaned DCS metadata (node physically removed but metadata remains), or cluster destroyed via other means requiring metadata cleanup.

# Remove entire cluster metadata (requires multiple confirmations)
$ pg remove pg-test
Please confirm the cluster name to remove: pg-test
You are about to remove all information in DCS for pg-test, please type: "Yes I am aware": Yes I am aware

4.5 - Managing PostgreSQL HBA Rules

HBA management - refresh rules, verify config, troubleshoot, Pgbouncer HBA

Quick Start

Pigsty uses declarative management: first define HBA rules in the inventory, then use bin/pgsql-hba <cls> to refresh.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:                            # <--- Define HBA rules here!
      - {user: dbuser_app, db: app, addr: intra, auth: pwd, title: 'app access'}
      - {user: dbuser_api, db: all, addr: world, auth: ssl, title: 'api ssl access'}
Script
bin/pgsql-hba <cls>              # Refresh PostgreSQL and Pgbouncer HBA rules for cluster
bin/pgsql-hba <cls> <ip>...      # Refresh HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster HBA rules
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh specific instance only
bin/pgsql-hba pg-meta 10.10.10.11 10.10.10.12  # Refresh multiple instances

For rule syntax, see HBA Configuration. For authentication methods, default boundaries, and credential management, see Authentication.

Action Description Risk
Refresh HBA Rules Re-render config files and reload service Low
Verify HBA Rules View current rules, test connection auth Read
Common Scenarios Add rules, block IP, role-based, post-expansion Low
Troubleshooting Connection rejected, auth failed, rules not applied -
Pgbouncer HBA Pgbouncer connection pool HBA management Low
demo/pgsql-hba.cast

Refresh HBA Rules

After modifying HBA rules in pigsty.yml, re-render config files and reload services.

Script
bin/pgsql-hba <cls>              # Refresh entire cluster HBA (PostgreSQL + Pgbouncer)
bin/pgsql-hba <cls> <ip>...      # Refresh specific instances (multiple IPs space-separated)
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh 10.10.10.10 instance only

Result: Renders PostgreSQL and Pgbouncer HBA config files based on inventory definitions, then reloads services to apply.

Config file locations

Service Config File Path Template File
PostgreSQL /pg/data/pg_hba.conf roles/pgsql/templates/pg_hba.conf
Pgbouncer /etc/pgbouncer/pgb_hba.conf roles/pgsql/templates/pgbouncer.hba
Don’t edit config files directly

Directly editing /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf works temporarily, but will be overwritten next time Ansible playbook runs. All HBA rule changes should be in pigsty.yml, then execute bin/pgsql-hba to refresh.

Related Tags

Tag Description
pg_hba Render PostgreSQL HBA config file
pg_reload Reload PostgreSQL config (needs pg_reload=true)
pgbouncer_hba Render Pgbouncer HBA config file
pgbouncer_reload Reload Pgbouncer config

Verify HBA Rules

After refreshing HBA rules, verify config is correctly applied.

View current HBA rules

SQL
-- View PostgreSQL HBA rules (recommended)
TABLE pg_hba_file_rules;

-- View matching rules for specific database
SELECT * FROM pg_hba_file_rules WHERE database @> ARRAY['mydb']::text[];
Bash
# View PostgreSQL HBA config file
cat /pg/data/pg_hba.conf

# View Pgbouncer HBA config file
cat /etc/pgbouncer/pgb_hba.conf

# View config file header (confirm if updated)
head -20 /pg/data/pg_hba.conf
Test Connection
# Test connection for specific user from specific address
psql -h <host> -p 5432 -U <user> -d <database> -c "SELECT 1"

# Test connection through Pgbouncer
psql -h <host> -p 6432 -U <user> -d <database> -c "SELECT 1"

Check HBA config syntax

# Reload config (validates syntax)
psql -c "SELECT pg_reload_conf()"

# If syntax errors, check logs
tail -f /pg/log/postgresql-*.log

Common Scenarios

Add New HBA Rule

Add rule to cluster config’s pg_hba_rules, then refresh:

pg-meta:
  vars:
    pg_hba_rules:
      - {user: new_user, db: new_db, addr: '192.168.1.0/24', auth: pwd, title: 'new app access'}
bin/pgsql-hba pg-meta

Emergency IP Block

When detecting malicious IP, add high-priority (order: 0) deny rule:

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'emergency block'}
bin/pgsql-hba pg-meta    # Refresh immediately

Role-Based Rules

Configure different HBA rules for primary and replica using role parameter:

pg_hba_rules:
  # Only primary allows write users
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer on primary'}
  # Replicas allow read-only users
  - {user: reader, db: all, addr: world, auth: ssl, role: replica, title: 'reader on replica'}

After refresh, rules auto-enable/disable based on instance’s pg_role.

Refresh HBA After Expansion

When cluster adds new instances, rules using addr: cluster need refresh to include new members:

./pgsql.yml -l 10.10.10.14       # Add new instance
bin/pgsql-hba pg-meta            # Refresh all instances' HBA (includes new member IPs)

Refresh HBA After Failover

After Patroni failover, instance pg_role may not match config. If HBA rules use role filtering, update config and refresh:

# Update role definitions in pigsty.yml then refresh
bin/pgsql-hba pg-meta

Troubleshooting

Connection Rejected

Symptom: FATAL: no pg_hba.conf entry for host "x.x.x.x", user "xxx", database "xxx"

Steps:

  1. Check current HBA rules, confirm if matching rule exists:
psql -c "TABLE pg_hba_file_rules"

  1. Confirm client IP, username, database matches any rule

  2. Check rule order (HBA uses first-match-wins)

  3. Add corresponding rule and refresh:

bin/pgsql-hba <cls>

Authentication Failed

Symptom: FATAL: password authentication failed for user "xxx"

Steps:

  1. Confirm password is correct
  2. Check password encryption method (pg_pwd_enc) compatibility with client
  3. Check if user exists:
SELECT * FROM pg_roles WHERE rolname = 'xxx';

HBA Rules Not Applied

Steps:

  1. Confirm refresh command was executed
  2. Check if Ansible execution succeeded
  3. Confirm PostgreSQL reloaded:
psql -c "SELECT pg_reload_conf()"
  1. Check if config file was updated:
head -20 /pg/data/pg_hba.conf

Rule Order Issues

HBA uses first-match-wins. If rules not working as expected:

  1. Check order values in rule definitions
  2. Use psql -c "TABLE pg_hba_file_rules" to view actual order
  3. Adjust order values (lower numbers = higher priority)

Pgbouncer HBA

Pgbouncer HBA management is similar to PostgreSQL, with some differences.

Config differences

Difference PostgreSQL Pgbouncer
Config file /pg/data/pg_hba.conf /etc/pgbouncer/pgb_hba.conf
Replication Supports db: replication Not supported
Local auth Uses ident Uses peer

Refresh Pgbouncer HBA

Script
bin/pgsql-hba <cls>    # Refresh both PostgreSQL and Pgbouncer
Playbook
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
View
cat /etc/pgbouncer/pgb_hba.conf    # View Pgbouncer HBA rules

Best Practices

  1. Always manage in config files: Don’t edit pg_hba.conf directly - all changes through pigsty.yml
  2. Test environment first: HBA changes can cause connection issues - verify in test env first
  3. Use order for priority: Blocklist rules use order: 0 to ensure priority matching
  4. Refresh promptly: Refresh HBA after adding/removing instances or failover
  5. Principle of least privilege: Only open necessary access - avoid addr: world + auth: trust
  6. Monitor auth failures: Watch for auth failures in pg_stat_activity
  7. Backup config: Backup pigsty.yml before important changes

Related Documentation

4.6 - Pgbouncer Connection Pooling

Manage Pgbouncer connection pool, including pause, resume, disable, enable, reconnect, kill, and reload operations.

Overview

Pigsty uses Pgbouncer as PostgreSQL connection pooling middleware, listening on port 6432 by default, proxying access to local PostgreSQL on port 5432.

This is an optional component. If you don’t have massive connections or need transaction pooling and query metrics, you can disable it, connect directly to the database, or keep it unused.

User & Database Management

Pgbouncer users and databases are auto-managed by Pigsty, applying database config and user config when creating databases and creating users.

Database Management: Databases defined in pg_databases are auto-added to Pgbouncer by default. Set pgbouncer: false to exclude specific databases.

pg_databases:
  - name: mydb                # Added to connection pool by default
    pool_auth_user: dbuser_meta # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction    # Database-level pool mode
    pool_size: 50             # Default pool size
    pool_reserve: 30          # Reserve pool size
    pool_size_min: 0          # Minimum pool size
    pool_connlimit: 100       # Max database connections
  - name: internal
    pgbouncer: false          # Excluded from connection pool

User Management: Users defined in pg_users need explicit pgbouncer: true to be added to connection pool user list.

pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true           # Add to connection pool user list
    pool_mode: transaction    # User-level pool mode
    pool_connlimit: 50        # User-level max connections

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Service Management

In Pigsty, PostgreSQL cluster Primary Service and Replica Service default to Pgbouncer port 6432. To bypass connection pool and access PostgreSQL directly, customize pg_services, or set pg_default_service_dest to postgres.

Config Management

Pgbouncer config files are in /etc/pgbouncer/, generated and managed by Pigsty:

File Description
pgbouncer.ini Main config, pool-level params
database.txt Database list, database-level params
userlist.txt User password list
useropts.txt User-level pool params
pgb_hba.conf HBA access control rules

Pigsty auto-manages database.txt and userlist.txt, updating them when creating databases or creating users.

You can manually edit config then RELOAD to apply:

# Edit config
$ vim /etc/pgbouncer/pgbouncer.ini

# Reload via systemctl
$ sudo systemctl reload pgbouncer

# Reload as pg_dbsu / postgres user
$ pgb -c "RELOAD;"

Pool Management

Pgbouncer runs as the same dbsu as PostgreSQL, default postgres OS user. Pigsty provides pgb alias for easy management:

alias pgb='psql -p6432 -dpgbouncer'

Use pgb on database nodes to connect to Pgbouncer admin console for management commands and monitoring queries.

$ pgb
pgbouncer=# SHOW POOLS;
pgbouncer=# SHOW CLIENTS;
pgbouncer=# SHOW SERVERS;
Command Function Description
PAUSE Pause Pause database, wait for txn completion then disconnect
RESUME Resume Resume database paused by PAUSE/KILL/SUSPEND
DISABLE Disable Reject new client connections for database
ENABLE Enable Allow new client connections for database
RECONNECT Reconnect Gracefully close and rebuild server connections
KILL Kill Immediately disconnect all client and server connections
KILL_CLIENT Kill Client Terminate specific client connection
SUSPEND Suspend Flush buffers and stop listening, for online restart
SHUTDOWN Shutdown Shutdown Pgbouncer process
RELOAD Reload Reload config files
WAIT_CLOSE Wait Close Wait for server connections to close after RECONNECT/RELOAD
Monitor Commands Monitor View pool status, clients, servers, etc.

PAUSE

Use PAUSE to pause database connections. Pgbouncer waits for active txn/session to complete based on pool mode, then disconnects server connections. New client requests are blocked until RESUME.

PAUSE [db];           -- Pause specified database, or all if not specified

Typical use cases:

  • Online backend database switch (e.g., update connection target after switchover)
  • Maintenance operations requiring all connections disconnected
  • Combined with SUSPEND for Pgbouncer online restart
$ pgb -c "PAUSE mydb;"        # Pause mydb database
$ pgb -c "PAUSE;"             # Pause all databases

After pause, SHOW DATABASES shows paused status:

pgbouncer=# SHOW DATABASES;
   name   |   host    | port | database | ... | paused | disabled
----------+-----------+------+----------+-----+--------+----------
 mydb     | /var/run  | 5432 | mydb     | ... |      1 |        0

RESUME

Use RESUME to restore databases paused by PAUSE, KILL, or SUSPEND, allowing new connections and resuming normal service.

RESUME [db];          -- Resume specified database, or all if not specified
$ pgb -c "RESUME mydb;"       # Resume mydb database
$ pgb -c "RESUME;"            # Resume all databases

DISABLE

Use DISABLE to disable a database, rejecting all new client connection requests. Existing connections are unaffected.

DISABLE db;           -- Disable specified database (database name required)

Typical use cases:

  • Temporarily offline a database for maintenance
  • Block new connections for safe database migration
  • Gradually decommission a database being removed
$ pgb -c "DISABLE mydb;"      # Disable mydb, new connections rejected

ENABLE

Use ENABLE to enable a database previously disabled by DISABLE, accepting new client connections again.

ENABLE db;            -- Enable specified database (database name required)
$ pgb -c "ENABLE mydb;"       # Enable mydb, allow new connections

RECONNECT

Use RECONNECT to gracefully rebuild server connections. Pgbouncer closes connections when released back to pool, creating new ones when needed.

RECONNECT [db];       -- Rebuild server connections for database, or all if not specified

Typical use cases:

  • Refresh connections after backend database IP change
  • Reroute traffic after switchover
  • Rebuild connections after DNS update
$ pgb -c "RECONNECT mydb;"    # Rebuild mydb server connections
$ pgb -c "RECONNECT;"         # Rebuild all server connections

After RECONNECT, use WAIT_CLOSE to wait for old connections to fully release.

KILL

Use KILL to immediately disconnect all client and server connections for a database. Unlike PAUSE, KILL doesn’t wait for transaction completion - forces immediate disconnect.

KILL [db];            -- Kill all connections for database, or all (except admin) if not specified
$ pgb -c "KILL mydb;"         # Force disconnect all mydb connections
$ pgb -c "KILL;"              # Force disconnect all database connections (except admin)

After KILL, new connections are blocked until RESUME.

KILL_CLIENT

Use KILL_CLIENT to terminate a specific client connection. Client ID can be obtained from SHOW CLIENTS output.

KILL_CLIENT id;       -- Terminate client connection with specified ID
# View client connections
$ pgb -c "SHOW CLIENTS;"

# Terminate specific client (assuming ptr column shows ID 0x1234567890)
$ pgb -c "KILL_CLIENT 0x1234567890;"

SUSPEND

Use SUSPEND to suspend Pgbouncer. Flushes all socket buffers and stops listening until RESUME.

SUSPEND;              -- Suspend Pgbouncer

SUSPEND is mainly for Pgbouncer online restart (zero-downtime upgrade):

# 1. Suspend current Pgbouncer
$ pgb -c "SUSPEND;"

# 2. Start new Pgbouncer process (with -R option to take over sockets)
$ pgbouncer -R /etc/pgbouncer/pgbouncer.ini

# 3. New process takes over, old process exits automatically

SHUTDOWN

Use SHUTDOWN to shut down Pgbouncer process. Multiple shutdown modes supported:

SHUTDOWN;                      -- Immediate shutdown
SHUTDOWN WAIT_FOR_SERVERS;     -- Wait for server connections to release
SHUTDOWN WAIT_FOR_CLIENTS;     -- Wait for clients to disconnect (zero-downtime rolling restart)
Mode Description
SHUTDOWN Immediately shutdown Pgbouncer
WAIT_FOR_SERVERS Stop accepting new connections, wait for server release
WAIT_FOR_CLIENTS Stop accepting new connections, wait for all clients disconnect, for rolling restart 
$ pgb -c "SHUTDOWN WAIT_FOR_CLIENTS;"   # Graceful shutdown, wait for clients

RELOAD

Use RELOAD to reload Pgbouncer config files. Dynamically updates most config params without process restart.

RELOAD;               -- Reload config files
$ pgb -c "RELOAD;"              # Reload via admin console
$ systemctl reload pgbouncer    # Reload via systemd
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)  # Reload via signal

Pigsty provides playbook task to reload Pgbouncer config:

./pgsql.yml -l <cls> -t pgbouncer_reload    # Reload cluster Pgbouncer config

WAIT_CLOSE

Use WAIT_CLOSE to wait for server connections to finish closing. Typically used after RECONNECT or RELOAD to ensure old connections are fully released.

WAIT_CLOSE [db];      -- Wait for server connections to close, or all if not specified
# Complete connection rebuild flow
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"    # Wait for old connections to release

Monitoring

Pgbouncer provides rich SHOW commands for monitoring pool status:

Command Description
SHOW HELP Show available commands
SHOW DATABASES Show database config and status
SHOW POOLS Show pool statistics
SHOW CLIENTS Show client connection list
SHOW SERVERS Show server connection list
SHOW USERS Show user config
SHOW STATS Show statistics (requests, bytes)
SHOW STATS_TOTALS Show cumulative statistics
SHOW STATS_AVERAGES Show average statistics
SHOW CONFIG Show current config params
SHOW MEM Show memory usage
SHOW DNS_HOSTS Show DNS cached hostnames
SHOW DNS_ZONES Show DNS cached zones
SHOW SOCKETS Show open socket info
SHOW ACTIVE_SOCKETS Show active sockets
SHOW LISTS Show internal list counts
SHOW FDS Show file descriptor usage
SHOW STATE Show Pgbouncer running state
SHOW VERSION Show Pgbouncer version

Common monitoring examples:

# View pool status
$ pgb -c "SHOW POOLS;"

# View client connections
$ pgb -c "SHOW CLIENTS;"

# View server connections
$ pgb -c "SHOW SERVERS;"

# View statistics
$ pgb -c "SHOW STATS;"

# View database status
$ pgb -c "SHOW DATABASES;"

For more monitoring command details, see Pgbouncer official docs.

Unix Signals

Pgbouncer supports Unix signal control, useful when admin console is unavailable:

Signal Equivalent Command Description
SIGHUP RELOAD Reload config files
SIGTERM SHUTDOWN WAIT_FOR_CLIENTS Graceful shutdown, wait clients
SIGINT SHUTDOWN WAIT_FOR_SERVERS Graceful shutdown, wait servers
SIGQUIT SHUTDOWN Immediate shutdown
SIGUSR1 PAUSE Pause all databases
SIGUSR2 RESUME Resume all databases 
# Reload config via signal
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)

# Graceful shutdown via signal
$ kill -SIGTERM $(cat /run/postgresql/pgbouncer.pid)

# Pause via signal
$ kill -SIGUSR1 $(cat /run/postgresql/pgbouncer.pid)

# Resume via signal
$ kill -SIGUSR2 $(cat /run/postgresql/pgbouncer.pid)

Traffic Switching

Pigsty-managed database routes live in /etc/pgbouncer/database.txt. To move one database’s Pgbouncer traffic to another node, edit that file, reload the configuration, then drain and rebuild existing server connections:

# 1. Change only mydb's backend target to 10.10.10.12
$ sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt

# 2. Reload config
$ pgb -c "RELOAD;"

# 3. Rebuild this database's connections and wait for old connections to close
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"

The pgb-route function currently shipped in the source only edits /etc/pgbouncer/pgbouncer.ini. That file merely includes database.txt and does not contain the generated per-database host= routes, so the function does not change managed database backends. Do not use it in place of the procedure above.

4.7 - Managing PostgreSQL Component Services

Use systemctl to manage PostgreSQL cluster component services - start, stop, restart, reload, and status check.

Overview

Pigsty’s PGSQL module consists of multiple components, each running as a systemd service on nodes. (pgbackrest is an exception)

Understanding these components and their management is essential for maintaining production PostgreSQL clusters.

Component Port Service Name Description
Patroni 8008 patroni HA manager, manages PostgreSQL lifecycle
PostgreSQL 5432 postgres Placeholder service, not used, for emergency
Pgbouncer 6432 pgbouncer Connection pooling middleware, traffic entry
PgBackRest - - pgBackRest has no daemon service
HAProxy 543x haproxy Load balancer, exposes database services
pg_exporter 9630 pg_exporter PostgreSQL metrics exporter
pgbouncer_exporter 9631 pgbouncer_exporter Pgbouncer metrics exporter
vip-manager - vip-manager Optional, manages L2 VIP address floating
Important

Do NOT use systemctl directly to manage PostgreSQL service. PostgreSQL is managed by Patroni - use patronictl commands instead. Direct PostgreSQL operations may cause Patroni state inconsistency and trigger unexpected failover. The postgres service is an emergency escape hatch when Patroni fails.

Quick Reference

Operation Command
Start systemctl start <service>
Stop systemctl stop <service>
Restart systemctl restart <service>
Reload systemctl reload <service>
Status systemctl status <service>
Logs journalctl -u <service> -f
Enable systemctl enable <service>
Disable systemctl disable <service>

Common service names: patroni, pgbouncer, haproxy, pg_exporter, pgbouncer_exporter, vip-manager

Patroni

Patroni is PostgreSQL’s HA manager, handling startup, shutdown, failure detection, and automatic failover. It’s the core PGSQL module component. PostgreSQL process is managed by Patroni - don’t use systemctl to manage postgres service directly.

Start Patroni

systemctl start patroni     # Start Patroni (also starts PostgreSQL)

After starting, Patroni auto-launches PostgreSQL. On first start, behavior depends on role:

  • Primary: Initialize or recover data directory
  • Replica: Clone data from primary and establish replication

Stop Patroni

systemctl stop patroni      # Stop Patroni (also stops PostgreSQL)

Stopping Patroni gracefully shuts down PostgreSQL. Note: If this is primary and auto-failover isn’t paused, may trigger failover.

Restart Patroni

systemctl restart patroni   # Restart Patroni (also restarts PostgreSQL)

Restart causes brief service interruption. For production, use pg restart for rolling restart.

Reload Patroni

systemctl reload patroni    # Reload Patroni config

Reload re-reads config file and applies hot-reloadable params to PostgreSQL.

View Status & Logs

systemctl status patroni    # View Patroni service status
journalctl -u patroni -f    # Real-time Patroni logs
journalctl -u patroni -n 100 --no-pager  # Last 100 lines

Config file: /etc/patroni/patroni.yml

Best Practice: Use patronictl instead of systemctl to manage PostgreSQL clusters.

Pgbouncer

Pgbouncer is a lightweight PostgreSQL connection pooling middleware. Business traffic typically goes through Pgbouncer (6432) rather than directly to PostgreSQL (5432) for connection reuse and database protection.

Start Pgbouncer

systemctl start pgbouncer

Stop Pgbouncer

systemctl stop pgbouncer

Note: Stopping Pgbouncer disconnects all pooled business connections.

Restart Pgbouncer

systemctl restart pgbouncer

Restart disconnects all existing connections. For config changes only, use reload.

Reload Pgbouncer

systemctl reload pgbouncer

Reload re-reads config files (user list, pool params, etc.) without disconnecting existing connections.

View Status & Logs

systemctl status pgbouncer
journalctl -u pgbouncer -f

Config files:

  • Main config: /etc/pgbouncer/pgbouncer.ini
  • HBA rules: /etc/pgbouncer/pgb_hba.conf
  • User list: /etc/pgbouncer/userlist.txt
  • Database list: /etc/pgbouncer/database.txt

Admin Console

psql -p 6432 -U postgres -d pgbouncer  # Connect to Pgbouncer admin console

Common admin commands:

SHOW POOLS;      -- View pool status
SHOW CLIENTS;    -- View client connections
SHOW SERVERS;    -- View backend server connections
SHOW STATS;      -- View statistics
RELOAD;          -- Reload config
PAUSE;           -- Pause all pools
RESUME;          -- Resume all pools

HAProxy

HAProxy is a high-performance load balancer that routes traffic to correct PostgreSQL instances. Pigsty uses HAProxy to expose services, routing traffic based on role (primary/replica) and health status.

Start HAProxy

systemctl start haproxy

Stop HAProxy

systemctl stop haproxy

Note: Stopping HAProxy disconnects all load-balanced connections.

Restart HAProxy

systemctl restart haproxy

Reload HAProxy

systemctl reload haproxy

HAProxy supports graceful reload without disconnecting existing connections. Use reload for config changes.

View Status & Logs

systemctl status haproxy
journalctl -u haproxy -f

Config files: the main configuration is /etc/haproxy/haproxy.cfg; Pigsty-generated service fragments are stored under /etc/haproxy/conf.d/.

Admin Interface

HAProxy provides a web admin interface, default port 9101:

http://<node_ip>:9101/haproxy

Default auth: username admin, password configured by haproxy_admin_password.

pg_exporter

pg_exporter is PostgreSQL’s Prometheus metrics exporter for collecting database performance metrics.

Start pg_exporter

systemctl start pg_exporter

Stop pg_exporter

systemctl stop pg_exporter

After stopping, Prometheus can’t collect PostgreSQL metrics from this instance.

Restart pg_exporter

systemctl restart pg_exporter

View Status & Logs

systemctl status pg_exporter
journalctl -u pg_exporter -f

Config file: /etc/pg_exporter.yml

Verify Metrics

curl -s localhost:9630/metrics | head -20

pgbouncer_exporter

pgbouncer_exporter is Pgbouncer’s Prometheus metrics exporter.

Start/Stop/Restart

systemctl start pgbouncer_exporter
systemctl stop pgbouncer_exporter
systemctl restart pgbouncer_exporter

View Status & Logs

systemctl status pgbouncer_exporter
journalctl -u pgbouncer_exporter -f

Verify Metrics

curl -s localhost:9631/metrics | head -20

vip-manager

vip-manager is an optional component for managing L2 VIP address floating. When pg_vip_enabled is enabled, vip-manager binds VIP to current primary node.

Start vip-manager

systemctl start vip-manager

Stop vip-manager

systemctl stop vip-manager

After stopping, VIP address is released from current node.

Restart vip-manager

systemctl restart vip-manager

View Status & Logs

systemctl status vip-manager
journalctl -u vip-manager -f

Config file: /etc/default/vip-manager

Verify VIP Binding

ip addr show           # Check network interfaces, verify VIP binding
pg list <cls>          # Confirm primary location

Startup Order & Dependencies

Recommended PGSQL module component startup order:

1. patroni          # Start Patroni first (auto-starts PostgreSQL)
2. pgbouncer        # Then start connection pool
3. haproxy          # Start load balancer
4. pg_exporter      # Start metrics exporters
5. pgbouncer_exporter
6. vip-manager      # Finally start VIP manager (if enabled)

Stop order should be reversed. Pigsty playbooks handle these dependencies automatically.

Batch Start All Services

systemctl start patroni pgbouncer haproxy pg_exporter pgbouncer_exporter

Batch Stop All Services

systemctl stop pgbouncer_exporter pg_exporter haproxy pgbouncer patroni

Common Troubleshooting

Service Startup Failure

systemctl status <service>        # View service status
journalctl -u <service> -n 50     # View recent logs
journalctl -u <service> --since "5 min ago"  # Last 5 minutes logs

Patroni Won’t Start

Symptom Possible Cause Solution
Can’t connect to etcd etcd cluster unavailable Check etcd service status
Data dir permission error File ownership not postgres chown -R postgres:postgres /pg/data
Port in use Leftover PostgreSQL process pg_ctl stop -D /pg/data or kill

Pgbouncer Won’t Start

Symptom Possible Cause Solution
Config syntax error INI format error Check /etc/pgbouncer/pgbouncer.ini
Port in use Port 6432 already used lsof -i :6432
userlist.txt permissions Incorrect file permissions chmod 600 /etc/pgbouncer/userlist.txt

HAProxy Won’t Start

Symptom Possible Cause Solution
Config syntax error Invalid main configuration or service fragment haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c -q
Port in use Service port conflict lsof -i :5433

Related Documentation

4.8 - Manage PostgreSQL Cron Jobs

Configure crontab to schedule PostgreSQL backups, vacuum freeze, and bloat maintenance tasks

Pigsty uses crontab to manage scheduled tasks for routine backups, freezing aging transactions, and reorganizing bloated tables and indexes.

Quick Reference

Operation Quick Command Description
Configure Cron Jobs ./pgsql.yml -t pg_crontab -l <cls> Apply pg_crontab config
View Cron Jobs crontab -l View as postgres user
Physical Backup pg-backup [full|diff|incr] Execute backup with pgBackRest
Transaction Freeze pg-vacuum [database...] Freeze aging transactions, prevent XID wraparound
Bloat Maintenance pg-repack [database...] Online reorganize bloated tables and indexes

For other management tasks, see: Backup Management, Monitoring System, HA Management.

Configure Cron Jobs

Use the pg_crontab parameter to configure cron jobs for the PostgreSQL database superuser (pg_dbsu, default postgres).

Example Configuration

The following pg-meta cluster configures a daily full backup at 1:00 AM, while pg-test configures weekly full backup on Monday with incremental backups on other days.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_crontab:
      - '00 01 * * * /pg/bin/pg-backup'
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_crontab:
      - '00 01 * * 1            /pg/bin/pg-backup full'
      - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'

Recommended Maintenance Schedule

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
  - '00 03 * * 0 /pg/bin/pg-vacuum'         # Weekly vacuum freeze on Sunday at 3:00 AM
  - '00 04 * * 1 /pg/bin/pg-repack'         # Weekly repack on Monday at 4:00 AM
Task Frequency Timing Description
pg-backup Daily Early morning Full or incremental backup, depending on business needs
pg-vacuum Weekly Sunday early morning Freeze aging transactions, prevent XID wraparound
pg-repack Weekly/Monthly Off-peak hours Reorganize bloated tables/indexes, reclaim space
Primary Only Execution

The pg-backup, pg-vacuum, and pg-repack scripts automatically detect the current node role. Only the primary will actually execute; replicas will exit directly. Therefore, you can safely configure the same cron jobs on all nodes, and after failover, the new primary will automatically continue executing maintenance tasks.

Apply Cron Jobs

Cron jobs are automatically written to the default location for the corresponding OS distribution when the pgsql.yml playbook executes (the pg_crontab task):

  • EL (RHEL/Rocky/Alma): /var/spool/cron/postgres
  • Debian/Ubuntu: /var/spool/cron/crontabs/postgres
Playbook
./pgsql.yml -l pg-meta -t pg_crontab     # Apply pg_crontab config to specified cluster
./pgsql.yml -l 10.10.10.10 -t pg_crontab # Target specific host only
Manual
# Edit cron jobs as postgres user
sudo -u postgres crontab -e

# Or edit crontab file directly
sudo vi /var/spool/cron/postgres           # EL series
sudo vi /var/spool/cron/crontabs/postgres  # Debian/Ubuntu

Each playbook execution will fully overwrite the cron job configuration.

View Cron Jobs

Execute the following command as the pg_dbsu OS user to view cron jobs:

crontab -l

# Pigsty Managed Crontab for postgres
SHELL=/bin/bash
PATH=/usr/pgsql/bin:/pg/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin
MAILTO=""
00 01 * * * /pg/bin/pg-backup

If you’re not familiar with crontab syntax, refer to Crontab Guru for explanations.

pg-backup

pg-backup is Pigsty’s physical backup script based on pgBackRest, supporting full, differential, and incremental backup modes.

Basic Usage

pg-backup                # Execute incremental backup (default), auto full if no existing full backup
pg-backup full           # Execute full backup
pg-backup diff           # Execute differential backup (based on most recent full backup)
pg-backup incr           # Execute incremental backup (based on most recent any backup)

Backup Types

Type Parameter Description
Full Backup full Complete backup of all data, only this backup needed for recovery
Differential diff Backup changes since last full backup, recovery needs full + diff
Incremental incr Backup changes since last any backup, recovery needs complete chain

Execution Requirements

  • Script must run on primary as postgres user
  • Script auto-detects current node role, exits (exit 1) when run on replica
  • Auto-retrieves stanza name from /etc/pgbackrest/pgbackrest.conf

Common Cron Configurations

Daily Full
pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
Weekly Full + Daily Incr
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'       # Other days incremental
Weekly Full + Daily Diff
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup diff'  # Other days differential

For more backup and recovery operations, see the Backup Management section.

pg-vacuum

pg-vacuum is Pigsty’s transaction freeze script for executing VACUUM FREEZE operations to prevent database shutdown from transaction ID (XID) wraparound.

Basic Usage

Basic
pg-vacuum                    # Freeze aging tables in all databases
pg-vacuum mydb               # Process specified database only
pg-vacuum mydb1 mydb2        # Process multiple databases
Options
pg-vacuum -n mydb            # Dry run mode, display only without executing
pg-vacuum -a 80000000 mydb   # Use custom age threshold (default 100M)
pg-vacuum -r 50 mydb         # Use custom aging ratio threshold (default 40%)
Manual SQL
-- Execute VACUUM FREEZE on entire database
VACUUM FREEZE;

-- Execute VACUUM FREEZE on specific table
VACUUM FREEZE schema.table_name;

Command Options

Option Description Default
-h, --help Show help message -
-n, --dry-run Dry run mode, display only false
-a, --age Age threshold, tables exceeding need freeze 100000000
-r, --ratio Aging ratio threshold, full freeze if exceeded (%) 40

Logic

  1. Check database datfrozenxid age, skip database if below threshold
  2. Calculate aging page ratio (percentage of table pages exceeding age threshold of total pages)
  3. If aging ratio > 40%, execute full database VACUUM FREEZE ANALYZE
  4. Otherwise, only execute VACUUM FREEZE ANALYZE on tables exceeding age threshold

Script sets vacuum_cost_limit = 10000 and vacuum_cost_delay = 1ms to control I/O impact.

Execution Requirements

  • Script must run on primary as postgres user
  • Uses file lock /tmp/pg-vacuum.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases

Common Cron Configuration

pg_crontab:
  - '00 03 * * 0 /pg/bin/pg-vacuum'     # Weekly Sunday at 3:00 AM

pg-repack

pg-repack is Pigsty’s bloat maintenance script based on the pg_repack extension for online reorganization of bloated tables and indexes.

Basic Usage

Basic
pg-repack                    # Reorganize bloated tables and indexes in all databases
pg-repack mydb               # Reorganize specified database only
pg-repack mydb1 mydb2        # Reorganize multiple databases
Options
pg-repack -n mydb            # Dry run mode, display only without executing
pg-repack -t mydb            # Reorganize tables only
pg-repack -i mydb            # Reorganize indexes only
pg-repack -T 30 -j 4 mydb    # Custom lock timeout (seconds) and parallelism
Manual
# Use pg_repack command directly to reorganize specific table
pg_repack dbname -t schema.table

# Use pg_repack command directly to reorganize specific index
pg_repack dbname -i schema.index

Command Options

Option Description Default
-h, --help Show help message -
-n, --dry-run Dry run mode, display only false
-t, --table Reorganize tables only false
-i, --index Reorganize indexes only false
-T, --timeout Lock wait timeout (seconds) 10
-j, --jobs Parallel jobs 2

Auto-Selection Thresholds

Script auto-selects objects to reorganize based on table/index size and bloat ratio:

Table Bloat Thresholds

Size Range Bloat Threshold Max Count
< 256MB > 40% 64
256MB - 2GB > 30% 16
2GB - 8GB > 20% 4
8GB - 64GB > 15% 1

Index Bloat Thresholds

Size Range Bloat Threshold Max Count
< 128MB > 40% 64
128MB - 1GB > 35% 16
1GB - 8GB > 30% 4
8GB - 64GB > 20% 1

Tables/indexes over 64GB are skipped with a warning and require manual handling.

Execution Requirements

  • Script must run on primary as postgres user
  • Requires pg_repack extension installed (installed by default in Pigsty)
  • Requires pg_table_bloat and pg_index_bloat views in monitor schema
  • Uses file lock /tmp/pg-repack.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases
Lock Waiting

Normal reads/writes are not affected during reorganization, but the final switch moment requires acquiring AccessExclusive lock on the table, blocking all access. For high-throughput workloads, recommend running during off-peak hours or maintenance windows.

Common Cron Configuration

pg_crontab:
  - '00 04 * * 1 /pg/bin/pg-repack'     # Weekly Monday at 4:00 AM

You can confirm database bloat through Pigsty’s PGCAT Database - Table Bloat panel and select high-bloat tables and indexes for reorganization.

For more details see: Managing Relation Bloat

Remove Cron Jobs

When using the pgsql-rm.yml playbook to remove a PostgreSQL cluster, it automatically deletes the postgres user’s crontab file.

./pgsql-rm.yml -l <cls> -t pg_crontab    # Remove cron jobs only
./pgsql-rm.yml -l <cls> --check          # Preflight full removal; execute only after backup and target confirmation

Related Documentation

4.9 - Managing PostgreSQL Extensions

Extension management - download, install, configure, enable, update, and remove extensions

Quick Start

Pigsty provides 575 extensions. Using extensions involves four steps: Download, Install, Configure, Enable.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector ]           # <--- Install extension packages
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'    # <--- Configure preload extensions
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]            # <--- Enable in database
Script
bin/pgsql-ext <cls>           # Install extensions defined in config on <cls> cluster
bin/pgsql-ext <cls> [ext...]  # Install extensions specified on command line
Playbook
./pgsql.yml -l pg-meta -t pg_ext    # Use playbook to install extensions
Example
bin/pgsql-ext pg-meta                         # Install defined extensions on pg-meta cluster
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions

For complete extension reference, see Extensions. For available extensions, see Extension Catalog.

Action Command Description
Download Extensions ./infra.yml -t repo_build Download extensions to local repo
Install Extensions bin/pgsql-ext <cls> Install extension packages on cluster
Configure Extensions pg edit-config <cls> -p Add to preload libs (requires restart)
Enable Extensions psql -c 'CREATE EXT ...' Create extension objects in database
Update Extensions ALTER EXTENSION UPDATE Update packages and extension objects
Remove Extensions DROP EXTENSION Drop extension objects, uninstall pkgs
demo/pgsql-ext.cast

Install Extensions

Extensions defined in pg_extensions are auto-installed during PostgreSQL cluster creation in the pg_extension task.

To install extensions on an existing cluster, add extensions to all.children.<cls>.pg_extensions, then execute:

Script
bin/pgsql-ext <cls>   # Install extensions on <cls> cluster
Playbook
./pgsql.yml -l <cls> -t pg_extension   # Use Ansible playbook
Example
bin/pgsql-ext pg-meta    # Install extensions defined in config on pg-meta

Example: Install PostGIS, TimescaleDB and PGVector on cluster

#all.children.pg-meta.vars:
pg_extensions: [ postgis, timescaledb, pgvector ]

Result: Installs extension packages on all cluster nodes. Pigsty auto-translates package aliases to actual package names for OS and PG version.

Ensure repos available before install

Before installing, ensure nodes have correct repos configured - extensions downloaded to local repo, or upstream repos configured.

Manual Install

If you don’t want to use Pigsty config to manage extensions, pass extension list directly on command line:

Script
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions on pg-meta
Playbook
./pgsql.yml -l pg-meta -t pg_ext -e '{"pg_extensions": ["pg_duckdb", "pg_mooncake"]}'

You can also use pig package manager CLI to install extensions on single node, with auto package alias resolution.

pig install postgis timescaledb       # Install multiple extensions
pig install pgvector -v 18            # Install for specific PG major version

ansible pg-test -b -a 'pig install pg_duckdb'   # Batch install on cluster with Ansible

You can also use OS package manager directly (apt/dnf), but you must know the exact RPM/DEB package name for your OS/PG:

# EL systems (RHEL, Rocky, Alma, Oracle Linux)
sudo yum install -y pgvector_18*

# Debian / Ubuntu
sudo apt install -y postgresql-18-pgvector

Download Extensions

To install extensions, ensure node’s extension repos contain the extension:

Pigsty’s default config auto-downloads mainstream extensions during installation. For additional extensions, add to repo_extra_packages and rebuild repo:

repo_extra_packages: [ pgvector, postgis, timescaledb ]
Script
make repo         # Shortcut = repo-build + node-repo
make repo-build   # Rebuild Infra repo (download packages and deps)
make node-repo    # Refresh node repo cache, update Infra repo reference
Playbook
./deploy.yml -t repo_build,node_repo  # Execute both tasks at once
./infra.yml -t repo_build     # Re-download packages to local repo
./node.yml  -t node_repo      # Refresh node repo cache

Configure Repos

You can also let all nodes use upstream repos directly (not recommended for production), skipping download and installing from upstream extension repos:

./node.yml -t node_repo -e node_repo_modules=node,pgsql   # Add PGDG and Pigsty upstream repos

Configure Extensions

Some extensions require preloading to shared_preload_libraries, requiring database restart after modification.

Use pg_libs as its default value to configure preload extensions, but this only takes effect during cluster init - later modifications are ineffective.

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'   # Preload extensions
    pg_extensions: [ timescaledb, postgis, pgvector ]          # Install packages

For existing clusters, refer to Modify Config to modify shared_preload_libraries:

pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'
pg restart pg-meta   # Modify pg-meta params and restart to apply

Ensure extension packages are correctly installed before adding preload config. If extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL won’t start. Also, manage cluster config changes through Patroni - avoid using ALTER SYSTEM or pg_parameters to modify instance config separately. If primary and replica configs differ, it may cause startup failure or replication interruption.

Enable Extensions

After installing packages, execute CREATE EXTENSION in database to use extension features.

Enable during cluster init

Declare extensions to enable in database definition via extensions array:

pg_databases:
  - name: meta
    extensions:
      - vector                             # Simple form
      - { name: postgis, schema: public }  # Specify schema

Manual enable

SQL
CREATE EXTENSION vector;                      -- Create extension
CREATE EXTENSION postgis SCHEMA public;       -- Specify schema
CREATE EXTENSION IF NOT EXISTS vector;        -- Idempotent creation
CREATE EXTENSION postgis_topology CASCADE;    -- Auto-install dependencies
psql
psql -d meta -c 'CREATE EXTENSION vector;'                  # Create extension in meta database
psql -d meta -c 'CREATE EXTENSION postgis SCHEMA public;'   # Specify schema
Playbook
# After modifying database definition, use playbook to enable extensions
bin/pgsql-db pg-meta meta    # Creating/modifying database auto-enables defined extensions

Result: Creates extension objects (functions, types, operators, index methods, etc.) in database, enabling use of extension features.

Update Extensions

Extension updates involve two layers: package update and extension object update.

Update packages

pig
pig update pgvector                           # Update extension with pig
yum
sudo yum update pgvector_18 # EL
apt
sudo apt upgrade postgresql-18-pgvector  # Debian/Ubuntu

Update extension objects

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Update extension to latest version
ALTER EXTENSION vector UPDATE;

-- Update to specific version
ALTER EXTENSION vector UPDATE TO '0.8.1';
Update Notes

Backup database before updating extensions. Preloaded extensions may require PostgreSQL restart after update. Some extension version upgrades may be incompatible - check extension docs.

Remove Extensions

Removing extensions involves two layers: drop extension objects and uninstall packages.

Drop extension objects

DROP EXTENSION vector;              -- Drop extension
DROP EXTENSION vector CASCADE;      -- Cascade drop (drops dependent objects)

Remove from preload

For preloaded extensions, remove from shared_preload_libraries and restart:

pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'
pg restart pg-meta   # Restart to apply config

Uninstall packages (optional)

pig
pig remove pgvector                           # Uninstall with pig
yum
sudo yum remove pgvector_18*                  # EL systems
apt
sudo apt remove postgresql-18-pgvector        # Debian/Ubuntu
CASCADE Warning

Using CASCADE to drop extensions also drops all objects depending on that extension (tables, indexes, views, etc.). Check dependencies before executing.

Query Extensions

Common SQL queries for extension info:

View enabled extensions

SELECT extname, extversion, nspname AS schema
FROM pg_extension e JOIN pg_namespace n ON e.extnamespace = n.oid
ORDER BY extname;

View available extensions

SELECT name, default_version, installed_version, comment
FROM pg_available_extensions
WHERE installed_version IS NOT NULL   -- Only show installed
ORDER BY name;

Check if extension is available

SELECT * FROM pg_available_extensions WHERE name = 'vector';

View extension dependencies

SELECT e.extname, d.refobjid::regclass AS depends_on
FROM pg_extension e
JOIN pg_depend d ON d.objid = e.oid
WHERE d.deptype = 'e' AND e.extname = 'postgis_topology';

View extension objects

SELECT classid::regclass, objid, deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'vector');

psql shortcuts

\dx                    # List enabled extensions
\dx+ vector            # Show extension details

Add Repos

To install directly from upstream, manually add repos.

Using Pigsty playbook

./node.yml -t node_repo -e node_repo_modules=node,pgsql        # Add PGDG and Pigsty repos
./node.yml -t node_repo -e node_repo_modules=node,pgsql,local  # Including local repo

YUM repos (EL systems)

# Pigsty repo
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# China mainland mirror
curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

APT repos (Debian/Ubuntu)

curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql $(lsb_release -cs) main
EOF
sudo apt update

# China mainland mirror: replace repo.pigsty.io with repo.pigsty.cc

FAQ

Difference between extension name and package name

Name Description Example
Extension name Name used with CREATE EXTENSION vector
Package alias Standardized name in Pigsty config pgvector
Package name Actual OS package name pgvector_18* or postgresql-18-pgvector

Preloaded extension prevents startup

If extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL won’t start. Solutions:

  1. Ensure extension package is correctly installed
  2. Or remove extension from shared_preload_libraries (edit /pg/data/postgresql.conf)

Extension dependencies

Some extensions depend on others, requiring sequential creation or using CASCADE:

CREATE EXTENSION postgis;                    -- Create base extension first
CREATE EXTENSION postgis_topology;           -- Then create dependent extension
-- Or
CREATE EXTENSION postgis_topology CASCADE;   -- Auto-create dependencies

Extension version incompatibility

View extension versions supported by current PostgreSQL:

SELECT * FROM pg_available_extension_versions WHERE name = 'vector';

Related Resources

4.10 - Upgrading PostgreSQL Major/Minor Versions

Version upgrade - minor version rolling upgrade, major version migration, extension upgrade

Quick Start

PostgreSQL version upgrades fall into two types: minor version upgrade and major version upgrade, with very different risk and complexity.

Type Example Downtime Data Compatibility Risk
Minor upgrade 17.2 → 17.3 Seconds (rolling) Fully compatible Low
Major upgrade 17 → 18 Minutes Requires data dir upgrade Medium
Minor
# Rolling upgrade: replicas first, then primary
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
pg restart --role replica --force <cls>
pg switchover <cls>
pg restart <cls> <old-primary> --force
Major
# Recommended: Logical replication migration
bin/pgsql-add pg-new              # Create new version cluster
# Configure logical replication to sync data...
# Switch traffic to new cluster
Extension
ansible <cls> -b -a 'yum upgrade -y postgis36_17*'
psql -c 'ALTER EXTENSION postgis UPDATE;'

For detailed online migration process, see Online Migration documentation.

Action Description Risk
Minor Version Upgrade Update packages, rolling restart Low
Minor Version Downgrade Rollback to previous minor version Low
Major Version Upgrade Logical replication or pg_upgrade Medium
Extension Upgrade Upgrade extension packages and objects Low

Minor Version Upgrade

Minor version upgrades (e.g., 17.2 → 17.3) are the most common upgrade scenario, typically for security patches and bug fixes. Data directory is fully compatible, completed via rolling restart.

Strategy: Recommended rolling upgrade: upgrade replicas first, then switchover to upgrade original primary - minimizes service interruption.

1. Update repo → 2. Upgrade replica packages → 3. Restart replicas
4. Switchover → 5. Upgrade original primary packages → 6. Restart original primary

Step 1: Prepare packages

Ensure local repo has latest PostgreSQL packages and refresh node cache:

Repo
cd ~/pigsty
./infra.yml -t repo_upstream      # Add upstream repos (needs internet)
./infra.yml -t repo_build         # Rebuild local repo
EL
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'
Debian
ansible <cls> -b -a 'apt clean'
ansible <cls> -b -a 'apt update'

Step 2: Upgrade replicas

Upgrade packages on all replicas and verify version:

EL
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
ansible <cls> -b -a '/usr/pgsql/bin/pg_ctl --version'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17'
ansible <cls> -b -a '/usr/lib/postgresql/17/bin/pg_ctl --version'

Restart all replicas to apply new version:

pg restart --role replica --force <cls>

Step 3: Switchover

Execute switchover to transfer primary role to upgraded replica:

pg switchover <cls>
# Or non-interactive:
pg switchover --leader <old-primary> --candidate <new-primary> --scheduled=now --force <cls>

Step 4: Upgrade original primary

Original primary is now replica - upgrade packages and restart:

EL
ansible <old-primary-ip> -b -a 'yum upgrade -y postgresql17*'
Debian
ansible <old-primary-ip> -b -a 'apt install -y postgresql-17'
pg restart <cls> <old-primary-name> --force

Step 5: Verify

Confirm all instances have consistent version:

pg list <cls>
pg query <cls> -c "SELECT version()"

Minor Version Downgrade

In rare cases (e.g., new version introduces bugs), may need to downgrade PostgreSQL to previous version.

Step 1: Get old version packages

EL
cd ~/pigsty; ./infra.yml -t repo_upstream     # Add upstream repos
cd /www/pigsty; repotrack postgresql17-*-17.1 # Download specific version packages
cd ~/pigsty; ./infra.yml -t repo_create       # Rebuild repo metadata
Refresh Cache
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'

Step 2: Execute downgrade

EL
ansible <cls> -b -a 'yum downgrade -y postgresql17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17=17.1*'

Step 3: Restart cluster

pg restart --force <cls>

Major Version Upgrade

Major version upgrades (e.g., 17 → 18) involve data format changes, requiring specialized tools for data migration.

Method Downtime Complexity Use Case
Logical Replication Migration Seconds (switch) High Production, minimal downtime required
pg_upgrade In-Place Upgrade Minutes~Hours Medium Test env, smaller data
Recommended Approach

For production, we recommend logical replication migration: create new version cluster, sync data via logical replication, then blue-green switch. Shortest downtime and rollback-ready. See Online Migration.

Logical Replication Migration

Logical replication is the recommended approach for production major version upgrades. Core steps:

1. Create new version target cluster → 2. Configure logical replication → 3. Verify data consistency
4. Switch app traffic to new cluster → 5. Decommission old cluster

Step 1: Create new version cluster

pg-meta-new:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta-new
    pg_version: 18                    # New version
bin/pgsql-add pg-meta-new

Step 2: Configure logical replication

-- Source cluster (old version) primary: create publication
CREATE PUBLICATION upgrade_pub FOR ALL TABLES;

-- Target cluster (new version) primary: create subscription
CREATE SUBSCRIPTION upgrade_sub
  CONNECTION 'host=10.10.10.11 port=5432 dbname=mydb user=replicator password=xxx'
  PUBLICATION upgrade_pub;

Step 3: Wait for sync completion

-- Target cluster: check subscription status
SELECT * FROM pg_stat_subscription;

-- Source cluster: check replication slot LSN
SELECT slot_name, confirmed_flush_lsn FROM pg_replication_slots;

Step 4: Switch traffic

After confirming data sync complete: stop app writes to source → wait for final sync → switch app connections to new cluster → drop subscription, decommission source.

-- Target cluster: drop subscription
DROP SUBSCRIPTION upgrade_sub;

For detailed migration process, see Online Migration documentation.

pg_upgrade In-Place Upgrade

pg_upgrade is PostgreSQL’s official major version upgrade tool, suitable for test environments or scenarios accepting longer downtime.

Important Warning

In-place upgrade causes longer downtime and is difficult to rollback. For production, prefer logical replication migration.

Step 1: Install new version packages

./pgsql.yml -l <cls> -t pg_pkg -e pg_version=18

Step 2: Stop Patroni

pg pause <cls>                        # Pause auto-failover
systemctl stop patroni                # Stop Patroni (stops PostgreSQL)

Step 3: Run pg_upgrade

sudo su - postgres
mkdir -p /data/postgres/pg-meta-18/data

# Pre-check (-c parameter: check only, don't execute)
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  -v -c

# Execute upgrade
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  --link -j 8 -v

Step 4: Update links and start

rm -rf /usr/pgsql && ln -s /usr/pgsql-18 /usr/pgsql
rm -rf /pg && ln -s /data/postgres/pg-meta-18 /pg
# Edit /etc/patroni/patroni.yml to update paths
systemctl start patroni
pg resume <cls>

Step 5: Post-processing

/usr/pgsql-18/bin/vacuumdb --all --analyze-in-stages
./delete_old_cluster.sh   # Cleanup script generated by pg_upgrade

Extension Upgrade

When upgrading PostgreSQL version, typically also need to upgrade related extensions.

Upgrade extension packages

EL
ansible <cls> -b -a 'yum upgrade -y postgis36_17 timescaledb-2-postgresql-17* pgvector_17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17-postgis-3 postgresql-17-pgvector'

Upgrade extension objects

After package upgrade, execute extension upgrade in database:

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Upgrade extensions
ALTER EXTENSION postgis UPDATE;
ALTER EXTENSION timescaledb UPDATE;
ALTER EXTENSION vector UPDATE;

-- Check extension versions
SELECT extname, extversion FROM pg_extension;
Extension Compatibility

Before major version upgrade, confirm all extensions support target PostgreSQL version. Some extensions may require uninstall/reinstall - check extension documentation.

Important Notes

  1. Backup first: Always perform complete backup before any upgrade
  2. Test verify: Verify upgrade process in test environment first
  3. Extension compatibility: Confirm all extensions support target version
  4. Rollback plan: Prepare rollback plan, especially for major upgrades
  5. Monitor closely: Monitor database performance and error logs after upgrade
  6. Document: Record all operations and issues during upgrade

Related Documentation

5 - Backup & Restore

Configure repositories and policies, manage pgBackRest backups, and perform point-in-time recovery safely.

Pigsty uses pgBackRest for PostgreSQL backups. It supports full, differential, and incremental backups, parallel processing, encryption, and Silo/S3 object storage. Every PGSQL cluster is configured for backup and WAL archiving by default.

This chapter is the operational manual for backup configuration, management, recovery, and drills. For design concepts and tradeoffs, see Point-in-Time Recovery.

All backup and recovery operations ultimately invoke pgBackRest. Pigsty provides several wrapper layers:

Layer Interface Form Scope
Cluster orchestration pg_pitr + pgsql-pitr.yml Ansible playbook HA, etcd, and multi-node recovery
Instance orchestration pig pitr CLI Local-node recovery without the admin node
Command primitives pig pb, pb, and pg-backup pgBackRest wrappers Backup, inspection, expiry, and unmanaged restore
Engine pgbackrest Native CLI Underlying backup, archive, and restore engine
Section Content
Mechanism Stanzas, repositories, retention, timelines, and Pigsty wrapper mapping
Policy Scheduling, recovery windows, and storage planning
Repository Local, Silo, and external S3 repositories; encryption, versioning, and locking
Administration Backup commands, inspection, expiration, and stanza management
Restore Recovery targets, staged PITR, and complete parameter reference
Clone Restore production history into another cluster and perform drills
Tutorial A sandbox restore using pgBackRest primitives
Disclaimer

Pigsty makes every effort to provide a reliable PITR solution, but accepts no liability for data loss caused by restore operations. If you need assistance, consider professional services.

Recovery overwrites target data

Before PITR, inspect pig pg list <target-cluster> and pig pb info, verify a recent usable backup and recovery window, have the operator state the exact target cluster and recovery point, then run the target-scoped ./pgsql-pitr.yml -l <target-cluster> ... command. pgsql-pitr.yml prints a plan but does not pause for approval. Production recovery also requires a maintenance window and an independently verified backup.

Quick Start

  1. Design a backup policy: declare scheduled backups in pg_crontab and select a repository with pgbackrest_repo.
  2. Manage backups: run pg-backup and inspect recovery coverage with pb info.
  3. Perform recovery: declare pg_pitr, then run pgsql-pitr.yml.
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

5.1 - Backup Policy

Design backup policies according to your needs

The chart below combines the “Recovery Window” and “Backup Storage Usage” on a single timeline (0~108h) so they can be inspected together.

Under the same assumptions (database size 100GB, daily writes 10GB), it shows how both metrics evolve over 30 days with “weekly full + daily incremental” backups and 14-day full-backup retention.

  • When: Backup schedule
  • Where: Backup repository
  • How: Backup method

When to Backup

The first question is when to backup your database - this is a tradeoff between backup frequency and recovery time. Since you need to replay WAL logs from the last backup to the recovery target point, the more frequent the backups, the less WAL logs need to be replayed, and the faster the recovery.

Daily Full Backup

For production databases, it’s recommended to start with the simplest daily full backup strategy. This is also Pigsty’s default backup strategy, implemented via crontab.

pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository

Assume your database size is 100GB, daily writes are 10GB, and each full backup takes 1 hour. Under this daily-full local-repo strategy, recovery window and backup storage evolve as shown below:

The recovery window cycles between 25-49 hours, and storage usage is roughly 2 full backups plus around 2 days of WAL archives. In practice, prepare at least 3~5 times the base database size as backup disk capacity for the default policy.

tooltip: { trigger: axis, formatter: $fn:tipMerged, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 286, height: 218, containLabel: false }
xAxis:
  - type: category
    gridIndex: 0
    position: bottom
    boundaryGap: false
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    name: Time h
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { interval: 11, formatter: $fn:fmtHour }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 12, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 52
    interval: 5
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 5, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 350
    interval: 50
    inverse: true
    name: Backup Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series: [ { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f2a000" }, itemStyle: { color: "#f2a000" }, data: [[0,0],[1,0],[1,1],[2,2],[3,3],[4,4],[5,5],[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12],[13,13],[14,14],[15,15],[16,16],[17,17],[18,18],[19,19],[20,20],[21,21],[22,22],[23,23],[24,24],[25,25],[26,26],[27,27],[28,28],[29,29],[30,30],[31,31],[32,32],[33,33],[34,34],[35,35],[36,36],[37,37],[38,38],[39,39],[40,40],[41,41],[42,42],[43,43],[44,44],[45,45],[46,46],[47,47],[48,48],[49,49],[49,25],[50,26],[51,27],[52,28],[53,29],[54,30],[55,31],[56,32],[57,33],[58,34],[59,35],[60,36],[61,37],[62,38],[63,39],[64,40],[65,41],[66,42],[67,43],[68,44],[69,45],[70,46],[71,47],[72,48],[73,49],[73,25],[74,26],[75,27],[76,28],[77,29],[78,30],[79,31],[80,32],[81,33],[82,34],[83,35],[84,36],[85,37],[86,38],[87,39],[88,40],[89,41],[90,42],[91,43],[92,44],[93,45],[94,46],[95,47],[96,48],[97,49],[97,25],[98,26],[99,27],[100,28],[101,29],[102,30],[103,31],[104,32],[105,33],[106,34],[107,35],[108,36]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 0, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 24, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 48, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 72, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 96, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 1, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 25, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 49, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 73, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 97, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { yAxis: 25, label: { show: true, formatter: "lower 25h", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.75 } }, { yAxis: 49, label: { show: true, formatter: "upper 49h", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.75 } } ] } }, { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#59a14f" }, data: [0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#edc949" }, data: [0,0,0.42,0.83,1.25,1.67,2.08,2.5,2.92,3.33,3.75,4.17,4.58,5,5.42,5.83,6.25,6.67,7.08,7.5,7.92,8.33,8.75,9.17,9.58,10,10.42,10.83,11.25,11.67,12.08,12.5,12.92,13.33,13.75,14.17,14.58,15,15.42,15.83,16.25,16.67,17.08,17.5,17.92,18.33,18.75,19.17,19.58,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20] }, { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0] } ]

Full + Incremental Backup

You can optimize backup space usage by adjusting these parameters.

If using Silo / S3 as a centralized backup repository, storage is no longer limited by the local disk. In this case, consider using full + incremental backup with a 2-week retention policy:

pg_crontab:  # Full backup at 1 AM on Monday, incremental backups on weekdays
  - '00 01 * * 1           /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
pgbackrest_method: minio
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  minio:                          # Optional S3-compatible repository preset
    type: s3                      # Silo uses the S3-compatible repository type
    s3_endpoint: sss.pigsty       # object-storage endpoint, `sss.pigsty` by default
    s3_region: us-east-1          # compatibility region, `us-east-1` by default
    s3_bucket: pgsql              # backup bucket, `pgsql` by default
    s3_key: pgbackrest            # pgBackRest access key
    s3_key_secret: S3User.Backup  # object-storage user secret
    s3_uri_style: path            # use path-style rather than host-style URIs
    path: /pgbackrest             # backup path, `/pgbackrest` by default
    storage_port: 9000            # Silo service port, 9000 by default
    storage_ca_file: /etc/pki/ca.crt  # Pigsty CA path, `/etc/pki/ca.crt` by default
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

With weekly full backups and time-based retention of 14 days, the steady-state recovery window is roughly 14–21 days. The exact window still depends on successful backup and WAL archival runs.

Assuming your database size is 100GB and writes 10GB of data per day, the backup size is as follows:

tooltip: { trigger: axis, formatter: $fn:tipMerged30, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "Incremental Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 302, height: 218, containLabel: false }
xAxis:
  - type: value
    gridIndex: 0
    position: bottom
    boundaryGap: false
    min: 0
    max: 31
    interval: 1
    name: Time
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { formatter: $fn:fmtDay30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    z: 10
    data: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, alignWithLabel: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 360
    interval: 48
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 600
    interval: 50
    inverse: true
    z: 10
    name: Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick30 }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series:
  - { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f28e2c" }, itemStyle: { color: "#f28e2c" }, data: [[1,24],[2,48],[3,72],[4,96],[5,120],[6,144],[7,168],[8,192],[9,216],[10,240],[11,264],[12,288],[13,312],[14,336],[14,168],[15,192],[16,216],[17,240],[18,264],[19,288],[20,312],[21,336],[21,168],[22,192],[23,216],[24,240],[25,264],[26,288],[27,312],[28,336],[28,168],[29,192],[30,216]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 7, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 14, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 21, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 28, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { yAxis: 168, label: { show: true, formatter: "lower 7d", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.72 } }, { yAxis: 336, label: { show: true, formatter: "upper 14d", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.72 } } ] } }
  - { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#59a14f" }, data: [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Incremental Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#76b7b2" }, data: [0,10,20,30,40,50,60,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80] }
  - { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#edc949" }, data: [10,20,30,40,50,60,70,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90] }
  - { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0] }

Backup Location

By default, Pigsty provides two default backup repository definitions: local and minio backup repositories.

  • local: Default option, uses local /pg/backup directory (symlink to pg_fs_backup: /data/backups)
  • minio: Uses SNSD single-node MinIO cluster (supported by Pigsty, but not enabled by default)
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository
  minio:                          # Optional minio repository
    type: s3                      # minio is S3 compatible
    s3_endpoint: sss.pigsty       # minio endpoint domain, defaults to `sss.pigsty`
    s3_region: us-east-1          # minio region, defaults to us-east-1, meaningless for minio
    s3_bucket: pgsql              # minio bucket name, defaults to `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret for pgbackrest
    s3_uri_style: path            # minio uses path-style URIs instead of host-style
    path: /pgbackrest             # minio backup path, defaults to `/pgbackrest`
    storage_port: 9000            # minio port, defaults to 9000
    storage_ca_file: /etc/pki/ca.crt  # minio CA certificate path, defaults to `/etc/pki/ca.crt`
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

5.2 - Restore Operations

Perform PITR with pgsql-pitr.yml, pig pitr, or pig pb restore; select targets, run stages, and verify the result.

Pigsty provides three restore entry points. They share the same parameter semantics, but serve different scopes:

Entry point Use case What it controls
pgsql-pitr.yml Production cluster recovery HA pause, multiple nodes, etcd cleanup, restore, and restart
pig pitr A local database node Single-instance orchestration without the admin node
pig pb restore An instance not managed by Patroni A direct pgBackRest restore wrapper

For a hands-on sandbox drill, see Manual Recovery. To recover into another cluster without changing production, see Clone a PG Cluster.

PITR overwrites the target cluster

pgsql-pitr.yml pauses HA, stops Patroni/PostgreSQL, overwrites the target data directory with pgbackrest --force restore, then deletes the target cluster’s etcd prefix and rebuilds HA. It prints a plan but does not wait for confirmation. Before any real restore, inspect the topology with pig pg list <target-cluster>, verify a recent usable backup and recovery window with pig pb info, and have the operator state and confirm the exact target cluster and recovery point. Schedule a maintenance window and retain an independently verified backup for production recovery.

Quick Start

To roll pg-meta back to an earlier time, declare pg_pitr:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_pitr: { time: '2025-07-13 10:00:00+00', action: promote }

Run the same target through the safety gate before executing it:

pig pg list pg-meta
pig pb info
./pgsql-pitr.yml -l pg-meta

You can pass the same object temporarily on the command line:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'
Use valid JSON for command-line variables

The -e value must be valid JSON: quote keys and string values, for example {"pg_pitr": {"time": "...", "archive": true}}. Booleans are not quoted. Invalid quoting can fail parsing or silently produce the wrong value.

The playbook pauses Patroni HA, stops the cluster, performs a delta pgBackRest restore, starts PostgreSQL and waits for a consistent recovery state, prints control data, removes old etcd metadata, and starts the cluster under Patroni again. It prints the source, target, and restore command first, but has no interactive approval gate. A one-shot targeted recovery should therefore declare action: promote explicitly. To inspect data at the target, use step-by-step execution with action: pause.

Recovery Targets

pg_pitr supports six recovery target forms. The four target values are mutually exclusive.

Recovery target types

default/latest
pg_pitr: { }  # Replay to the end of the WAL archive stream
time
pg_pitr: { time: "2025-07-13 10:00:00+00" }
lsn
pg_pitr: { lsn: "0/4001C80" }
xid
pg_pitr: { xid: "250000" }
name
pg_pitr: { name: "some_restore_point" }
immediate
pg_pitr: { type: "immediate" }

With no target, recovery replays all archived WAL to the latest available state (Pigsty’s internal type is default). immediate stops at the first consistent point, which is useful for obtaining a usable instance as quickly as possible or testing a backup.

Recover by Time

Use a valid PostgreSQL TIMESTAMP; an explicit time zone is strongly recommended:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

Recover by Name

Create an unambiguous marker before a risky change with pg_create_restore_point:

SELECT pg_create_restore_point('before_migration');
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "name": "before_migration", "action": "promote" }}'

Recover by Transaction ID

If the offending transaction ID is known from monitoring or CSVLOG’s TXID field, use exclusive to stop before that transaction:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "xid": "250000", "exclusive": true, "action": "promote" }}'

Recover by LSN

An LSN identifies a position in the WAL stream. It is also visible in Pigsty’s PG LSN dashboard panel. Set timeline when the desired position is on a particular timeline; the default is latest.

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "lsn": "0/4001C80", "timeline": "1", "action": "promote" }}'
Inclusive and exclusive targets

Targets are inclusive by default, so the target transaction is replayed. exclusive: true excludes the exact target. It applies only to time, xid, and lsn, and maps to PostgreSQL’s recovery_target_inclusive.

Recovery Source

Recovery uses the target cluster’s own backup by default. Three fields can select another source:

  • cluster: the source stanza, including another cluster in a shared repository
  • repo: a temporary repository definition in the same format as a pgbackrest_repo entry
  • set: a specific backup label; otherwise pgBackRest selects a suitable set

For example, recover pg-meta2 from pg-meta:

./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-07-14 08:00:00+00", "archive": false, "action": "promote" }}'

Step-by-Step Execution

In an incident, use tags to retain an explicit human gate between stages. After confirming the backup, recovery point, and exact target, run the stages in order:

./pgsql-pitr.yml -l pg-meta -t down  # Pause HA; stop Patroni and PostgreSQL
./pgsql-pitr.yml -l pg-meta -t pitr  # Restore, replay WAL, and print control information
./pgsql-pitr.yml -l pg-meta -t up    # Remove etcd metadata; start the cluster and resume HA
# down                 : # stop HA and PostgreSQL
#   - pause            : # pause Patroni automatic failover
#   - stop             : # stop Patroni and PostgreSQL
# pitr                 : # perform PITR
#   - config           : # render pgBackRest config and restore script
#   - backup           : # optionally move PGDATA to /pg/data-backup
#   - restore          : # run pgBackRest restore
#   - recovery         : # start PostgreSQL and replay WAL
#   - verify           : # print recovered control data
# up                   : # rebuild HA
#   - etcd             : # remove old cluster metadata
#   - start            : # start Patroni/PostgreSQL
#   - resume           : # resume Patroni automatic failover

After down, confirm the processes are stopped. After pitr, inspect /pg/tmp/recovery.log and query the recovery state before checking narrowly authorized business data. pg_controldata /pg/data reports checkpoint and timeline metadata; it does not by itself prove that a time, XID, or LSN target was reached.

SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(),
       pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();

With action: pause, promote only after validation, then run up. If the target is wrong, adjust pg_pitr and repeat pitr before up. pause or shutdown creates a meaningful human gate only in this staged workflow; use action: promote explicitly for one-shot targeted recovery.

pg_ctl -D /pg/data promote            # only after validating an action: pause recovery
./pgsql-pitr.yml -l pg-meta -t up     # rebuild Patroni HA
Repeating the pitr stage

With backup: true, the playbook moves the current data directory to /pg/data-backup, but deletes any existing /pg/data-backup before doing so. The staged workflow is supported; a restore using backup: true is not generally idempotent.

PITR Parameter Definition

Declare the target, action, and treatment of existing data explicitly:

pg_pitr:
  cluster: pg-meta                 # source cluster/stanza; defaults to pg_cluster
  type: default                    # default | time | xid | name | lsn | immediate
  time: "2025-07-13 10:00:00+00"  # mutually exclusive with xid, name, and lsn
  name: "some_restore_point"       # mutually exclusive with time, xid, and lsn
  xid: "250000"                    # mutually exclusive with time, name, and lsn
  lsn: "0/4001C80"                 # mutually exclusive with time, xid, and name
  exclusive: false                 # exclude the exact target; time/xid/lsn only
  timeline: latest                 # target timeline; integer or latest
  set: latest                      # backup label; auto-select by default
  action: pause                    # pause | promote | shutdown
                                   # a specified target defaults to pause
  archive: true                    # preserve archiving; false sets archive-mode=off
  backup: false                    # move old PGDATA to /pg/data-backup before restore
  db_exclude: []                   # databases to exclude
  db_include: []                   # databases to include
  link_map:                        # tablespace/WAL link remapping
    pg_wal: '/data/wal'
    pg_xact: '/data/pg_xact'
  process: 4                       # restore workers; defaults to node_cpu
  repo: {}                        # temporary source repository definition
  data: /pg/data                  # target data directory
  port: 5432                      # recovery instance port

See Parameter Mapping for the corresponding pgBackRest options.

Single Instance: pig pitr

pig pitr performs a local-node workflow without Ansible: validate the target, stanza, and backup; stop Patroni/PostgreSQL; restore; optionally start PostgreSQL; and print follow-up guidance.

pig pitr -t "2025-07-13 10:00:00+00"    # Recover to a point in time
pig pitr --xid 250000 -X                # Recover to before transaction 250000
pig pitr --name before_migration        # Recover to a named restore point
pig pitr -d                             # Recover to the end of the WAL archive stream
pig pitr -I --no-restart                # Restore for immediate recovery and leave PostgreSQL stopped
pig pitr -t "..." --plan                # Print the execution plan without making changes

Use -b/--set for a backup set, -T/--target-timeline for a timeline, --target-action for the post-target action, and -D/--data with --no-restart for a side restore. The command normally attempts a fast stop and aborts if that fails; only explicit --force-stop permits immediate shutdown and a kill fallback. For managed PGDATA it leaves Patroni stopped. Validate the instance before pig pt start. It does not remove etcd metadata, rebuild replicas, or rejoin the instance to an HA cluster.

Primitive: pig pb restore

For an instance not managed by Patroni (or one deliberately taken out of management), pig pb restore directly wraps pgbackrest restore. It validates the environment, requires PostgreSQL to be stopped, displays the plan, and asks for confirmation.

pig pb restore --time "2025-07-13 10:00"
pig pb restore --set 20250715-013657F
pig pb restore -d

It rejects a live Patroni-managed target because Patroni could restart a half-restored instance, and it rejects any running PostgreSQL target. Arguments after --, such as --tablespace-map or --link-all, pass through to pgBackRest, but wrapped options such as target, stanza, and repository cannot be overridden there.

Post-Recovery

After restore:

  1. Verify the recovery state and the smallest authorized set of application checks.

  2. After a cross-cluster clone, complete stanza cleanup. Create a full backup on the new timeline as soon as practical:

    pg-backup full

  3. If exploratory recovery used archive: false, restore archiving. Because archive_mode is a postmaster setting, first confirm the maintenance window, current primary, and replication state, then obtain explicit approval for the restart:

    psql -c 'ALTER SYSTEM RESET archive_mode;'
pg restart pg-meta
pg-backup full

5.3 - Clone a PG Cluster

Restore one cluster’s historical state into another for data recovery, restore drills, and forensic inspection.

Cloning is one of the safest and most useful applications of recovery: leave production untouched and restore its historical state into another cluster. You can recover accidentally deleted data from the clone, validate backups in a drill, inspect a historical state, or reset a test environment to a production snapshot.

The target must be able to access the source backup repository, may be overwritten, and must use a compatible PostgreSQL major version. With a shared Silo/S3 repository, each cluster’s backups are isolated by a stanza and visible to targets holding the required credentials.

A clone overwrites the target cluster

Inspect the target topology with pig pg list <target-cluster>, verify the source stanza’s recent backups and recovery window with pig pb info, and have the operator confirm the exact source cluster, target cluster, and recovery point before performing the restore. Existing data on the target is overwritten; production work still requires a maintenance window and an independently verified backup.

Clone an Existing Cluster

Assume the four-node sandbox contains pg-meta and pg-test, sharing a Silo repository. To reset pg-test to the latest state of pg-meta, point pg_pitr to the pg-meta stanza:

pig pg list pg-test
pig pb info
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'

Add a recovery target to clone any state inside the recovery window. For example, reset to 15:30 on December 26, 2025:

./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-12-26 15:30:00+08", "archive": false, "action": "promote" }}'

These cross-cluster examples set archive: false to keep the exploratory recovery from archiving under the target stanza. After Patroni takes control, complete the stanza and archive cleanup below.

The target may also be a newly initialized empty cluster, such as pg-meta2. Create it through the normal cluster creation workflow, then perform cross-cluster PITR.

pgBackRest restore uses delta mode and rewrites only files that differ from the backup. Repeated drills, or a target already synchronized through a standby cluster, can therefore restore much faster than a first full restore.

For accidental deletion, validate the clone and use pg_dump to export only the affected tables or database back into production. An in-place rollback of the entire production cluster should be the last resort, not the first response.

Post-Clone Cleanup

The clone contains the source cluster’s data, while the target stanza may still record the target’s old PostgreSQL system identifier. pgBackRest refuses a backup when the identifiers do not match, preventing the new cluster from contaminating the source history.

After validating the clone, complete these steps. Restarting the cluster is a service change: first inspect the primary and replication state, schedule the maintenance window, and obtain explicit approval.

pb stanza-upgrade                           # accept the new system-id; cross-cluster clone only
psql -c 'ALTER SYSTEM RESET archive_mode;' # undo archive: false
pg restart pg-test                         # archive_mode requires a restart
pg-backup full                             # establish a recovery point on the new timeline

Until this is complete, scheduled backups can fail the identity check, and a clone restored with archive: false produces no new WAL archive:

postgres@pg-test-1:~$ pb backup
INFO: backup command begin 2.57.0: --annotation=pg_cluster=pg-test ... --stanza=pg-test --start-fast
ERROR: [051]: PostgreSQL version 18, system-id 7588470953413201282 do not match stanza version 18, system-id 7588470974940466058
       HINT: is this the correct stanza?
INFO: backup command end: aborted with exception [051]

Rebuild Backup Identity

stanza-upgrade lets the new cluster continue writing under its existing stanza. If the clone should start a completely independent backup history, rebuild that stanza instead.

Declarative workflow:

pig pb info -s pg-test
pig pb delete -s pg-test                 # type the exact stanza when prompted
./pgsql.yml -t pg_backup -l pg-test      # create a fresh stanza
pg-backup full

Equivalent low-level workflow:

pig pb stop
pig pb delete -s pg-test                 # confirm the exact stanza
pig pb start
pig pb create -s pg-test
pig pb backup full -s pg-test
Rebuilding permanently discards old recovery history

Delete only after checking recent backups, retaining any required independent recovery copy, and having the operator confirm the exact pg-test stanza. Object-locked versions can remain and continue consuming storage; a successful deletion command does not prove that every underlying version has been physically erased.

Online Copies: Standby Clusters

A PITR clone is a static snapshot. Use a streaming-replication standby cluster for a continuously following online copy, or a delayed cluster for a fixed rollback window such as one hour.

The three methods complement each other: standby clusters provide a live copy, delayed clusters preserve a fixed delay, and PITR clones expose any historical state inside the recovery window without requiring a pre-existing online replica.

Restore Drills

A clone is an end-to-end restore drill that does not touch production, although it does overwrite the designated drill target. Run one quarterly and after major backup changes:

  1. Select a point inside the production recovery window.
  2. Restore it into the drill cluster and record elapsed time as the measured PITR RTO.
  3. Validate integrity with authorized row-count checks, critical-table checks, and application connectivity.
  4. Complete post-clone cleanup and verify that the drill cluster can create a new backup.
  5. Record timing, failures, and any difference between the runbook and reality.

See Manual Recovery for a sandbox exercise using pgBackRest primitives, or Fork an Instance for an XFS snapshot-based local test copy.

5.4 - Backup Mechanism

pgBackRest concepts—stanzas, repositories, backup chains, retention, and timelines—and how Pigsty maps parameters to commands.

Pigsty’s backup and restore operations ultimately execute pgBackRest commands. Using them safely requires both pgBackRest’s model and the mapping from Pigsty’s orchestration layers to native options.

Core pgBackRest Concepts

Stanza: the Cluster’s Backup Identity

A stanza names one PostgreSQL backup configuration and isolates that cluster inside a repository. Pigsty maps it directly from pg_cluster: the pg-meta stanza stores data under backup/pg-meta/ and archive/pg-meta/, so several clusters can share one repository.

The stanza records the source system identifier and major version and checks them before a backup. That identity check is why a cross-cluster clone needs stanza-upgrade afterward. Pigsty creates the stanza during cluster initialization; stanza-upgrade updates it after a major-version change or clone.

Repository: Where Backups Live

A repository stores backup files and WAL archives. repo1-type selects POSIX, S3, Azure, GCS, or SFTP; repo1-path, repo1-cipher-*, and repo1-retention-* define location, encryption, and retention. Pigsty renders these from pgbackrest_repo; see Backup Repository.

Backup Chains and Labels

Type Contents Label suffix
Full Complete database-cluster copy F
Differential Changes since the latest full D
Incremental Changes since the latest backup of any type I

Labels encode the chain. 20250715-013657F is a full backup; 20250715-013657F_20250715-013724D and ..._20250715-013730I depend on the full identified before the underscore. --set chooses the starting backup explicitly; otherwise pgBackRest selects the newest usable set before the target.

Retention: Keeping the Repository Bounded

repo1-retention-full and repo1-retention-full-type (count or time) decide when full chains expire. Dependent differential/incremental backups and WAL needed only by that chain expire with the full. Pigsty enables expire-auto, and pig pb expire --plan previews a manual run.

Time retention is a minimum window, not “keep only fulls newer than N days.” An old full expires only when another retained full has reached that age. A 14-day setting with weekly fulls therefore commonly retains three full chains and roughly 14–21 days of history.

WAL Archiving

PostgreSQL invokes archive-push when a WAL segment fills or archive_timeout elapses. During recovery, restore_command calls archive-get. Pigsty enables asynchronous archiving through /pg/spool so a temporary repository delay does not block the primary directly.

Timelines

Each promotion after recovery or failover creates a new timeline. Older timeline history remains in the repository, and --target-timeline chooses the recovery branch (latest by default). See the conceptual PITR mechanism.

What restore Actually Does

restore first reconstructs the data directory. Pigsty enables --delta, so pgBackRest validates existing files and rewrites only mismatches. It then writes recovery state (recovery.signal, restore_command, and recovery_target_*). Actual WAL replay happens after PostgreSQL starts.

Consequently, a successful restore command is only half of PITR. --target-action controls what happens when replay reaches the target: pause, promote, or shutdown.

pgbackrest --stanza=pg-meta restore
pgbackrest --stanza=pg-meta --type=immediate restore
pgbackrest --stanza=pg-meta --type=time --target='2025-07-13 10:00:00+00' restore
pgbackrest --stanza=pg-meta --type=xid --target='250000' --target-exclusive restore
pgbackrest --stanza=pg-meta --type=name --target='my-restore-point' restore
pgbackrest --stanza=pg-meta --type=lsn --target='0/4001C80' --target-action=promote restore

Observe a Backup Chain

Run read-only info after full, differential, and incremental backups to inspect labels, size, WAL bounds, and references:

pig pb info

A representative sequence looks like:

full backup: 20250715-013657F
diff backup: 20250715-013657F_20250715-013724D
    backup reference total: 1 full
incr backup: 20250715-013657F_20250715-013730I
    backup reference total: 1 full, 1 diff

Pigsty’s Wrapper Layers

Layer Interface What it does
Cluster orchestration pg_pitr + pgsql-pitr.yml Pause HA, stop nodes, render configuration, restore/replay, inspect control data, clean etcd, and rebuild HA
Instance orchestration pig pitr Preflight, keep one target offline, restore, optionally start PostgreSQL, and leave Patroni stopped for inspection
Command primitive pig pb, pb, pg-backup Supply stanza/DBSU context and call the corresponding pgBackRest command
Engine pgbackrest Read /etc/pgbackrest/pgbackrest.conf and perform backup, archive, and restore operations

Command Primitives

pb is a login-shell function that reads the first stanza from the local configuration and forwards arguments:

pb() (
    stanza=$(grep -o '\[[^][]*]' /etc/pgbackrest/pgbackrest.conf | head -n1 | sed 's/.*\[\([^]]*\)].*/\1/')
    pgbackrest --stanza="${stanza}" "$@"
)
pb info     # pgbackrest --stanza=pg-meta info
pb backup   # pgbackrest --stanza=pg-meta backup

pg-backup adds a primary-role check for scheduled use:

pg-backup full
pg-backup diff
pg-backup incr   # the default; pgBackRest promotes it when no full exists

pig pb adds stanza detection, DBSU privilege handling, primary checks for backup, and plan/confirmation guards for destructive primitives. See Admin Commands.

Parameter Mapping

pg_pitr field pig pitr option pgBackRest option Meaning
cluster --stanza --stanza Source cluster/stanza
type plus time/xid/lsn/name corresponding target option --type + --target Recovery target
default --default no --type/--target Replay to archive end
immediate --immediate --type=immediate Stop at the first consistent point
exclusive --exclusive / -X --target-exclusive Stop before the target
action --target-action --target-action pause, promote, or shutdown
timeline --target-timeline / -T --target-timeline Target timeline
set --set / -b --set Starting backup set
db_include / db_exclude — --db-include / --db-exclude Select databases in a physical restore
link_map — --link-map Remap directory or tablespace links
process — process-max Parallel restore processes
data --data / -D --pg1-path Target data directory
repo repository number only in pig pitr rendered repo1-* Override repository definition in the playbook

A selective restore is still physical. Excluded databases receive sparse zeroed files so PostgreSQL can complete recovery, but those databases are inaccessible and must be removed explicitly afterward; this is not a logical subset like pg_dump.

How Configuration Is Rendered

The entry selected by pgbackrest_method is rendered to /etc/pgbackrest/pgbackrest.conf: underscores become hyphens and keys receive the repo1- prefix.

pgbackrest_repo:
  minio:
    type: s3                  # repo1-type=s3
    s3_endpoint: sss.pigsty   # repo1-s3-endpoint=sss.pigsty
    cipher_type: aes-256-cbc  # repo1-cipher-type=aes-256-cbc
    retention_full: 14        # repo1-retention-full=14

pgsql-pitr.yml renders a separate temporary /pg/conf/pitr.conf; PostgreSQL recovery output goes to /pg/tmp/recovery.log.

Scheduled Backups

pg_crontab entries are installed for the postgres OS user on every cluster node. Because pg-backup checks the current role, only the primary backs up, and a promoted primary takes over future schedules.

pg_crontab:
  - '00 01 * * 1 /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
./pgsql.yml -t pg_crontab -l pg-meta

See Backup Policy for frequency and retention design.

Deployment Details

The pg_backup subtask installs/configures pgBackRest, creates the stanza, and—when pgbackrest_init_backup is enabled—attempts an initial full backup. /etc/pgbackrest/initial.done is written only after that backup succeeds.

Path Purpose
/usr/bin/pgbackrest pgBackRest binary
/etc/pgbackrest/pgbackrest.conf Main stanza and repository configuration
/pg/backup Local repository path
/pg/spool Asynchronous archive spool
/pg/log/pgbackrest/ Backup, archive, and restore logs
/pg/conf/pitr.conf Temporary PITR configuration
/pg/tmp/recovery.log PostgreSQL recovery log

pgbackrest_exporter listens on pgbackrest_exporter_port, 9854 by default, and exports backup metrics. Disable it with pgbackrest_exporter_enabled or customize it with pgbackrest_exporter_options.

5.5 - Backup Repository

Configure local, Silo, and external S3 backup repositories, including retention, encryption, versioning, and Object Lock.

Two parameters decide where backups are stored: pgbackrest_repo defines candidate repositories, while pgbackrest_method selects one. Repository keys are rendered deterministically as pgBackRest repo1-* options, so any supported pgBackRest repository option can be used directly. Pigsty v4.5.0 renders only the entry selected by pgbackrest_method as repo1; keeping several candidate keys does not enable multi-repository backup.

Default Repositories

Pigsty supplies two definitions: local and minio.

  • local is the default. /pg/backup points at pg_fs_backup, /data/backups by default.
  • minio uses Silo from the MINIO module or another compatible S3 service. It is supported but not selected by default.
pgbackrest_method: local
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest
    retention_full_type: time
    retention_full: 14

The presets deliberately differ. local favors simple, fast restores with count-based retention and no encryption or bundling. minio enables AES-256-CBC encryption, bundles small files, uses block incremental backup, and retains full backups by time.

Replace public defaults in production

For a remote repository, replace both cipher_pass and s3_key_secret. pgBackRest and S3User.Backup are public example defaults. Losing the encryption passphrase makes the repository unrecoverable, so store it separately from the backups under controlled recovery procedures; see Deployment Security.

Retention Policy

pgBackRest applies retention after each backup (expire-auto). When a full backup expires, dependent differential/incremental backups and the WAL needed only by that chain expire with it.

  • retention_full_type: count plus retention_full: 2 keeps the two newest full chains; a third can exist briefly while a new full completes.
  • retention_full_type: time plus retention_full: 14 establishes a minimum time window. An old full does not expire until another full is at least 14 days old; with weekly full backups this usually leaves three chains and roughly 14–21 days of recovery history.

See Backup Policy for recovery-window and space calculations, and Admin Commands for a safe expiration preview.

Use a Silo Repository

The MINIO module currently deploys Silo, an S3-compatible object store. It provides an independent disaster-recovery copy only when deployed outside the database host or site failure domain. Deploy it, then select the minio preset:

all:
  vars:
    pgbackrest_method: minio
  children:
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio, minio_type: silo }}

The preset uses the HTTPS endpoint sss.pigsty by default and validates it with /etc/pki/ca.crt. MINIO initialization creates the default pgsql bucket and pgbackrest user.

For serious production use, deploy and test a fault-tolerant multi-node/multi-drive object store; see MINIO Configuration. The preset name minio does not require the server to be managed by Pigsty: independently operated MinIO, RustFS, or another compatible implementation can use it, but that service’s installation, upgrades, certificates, and lifecycle remain outside the MINIO role’s support boundary.

Use S3 or Cloud Object Storage

For a single database node, an off-host cloud object store is often the most valuable repository. Define a new entry and select it:

pgbackrest_method: s3
pgbackrest_repo:
  s3:
    type: s3
    s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
    s3_region: oss-cn-beijing
    s3_bucket: <your_bucket_name>
    s3_key: <your_access_key>
    s3_key_secret: <your_secret_key>
    s3_uri_style: host
    path: /pgbackrest
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: <separate-strong-passphrase>
    retention_full_type: time
    retention_full: 14
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2

pgBackRest also supports Azure, GCS, and SFTP repositories.

Share a Repository Across Clusters

A centralized repository can serve several PostgreSQL clusters. Each stanza, mapped from pg_cluster, isolates one cluster’s backup and archive history. This also enables cross-cluster restore.

Cluster names must therefore be globally unique within a shared repository, even across otherwise separate deployment environments.

Repository Versioning

Object-store versioning can preserve earlier versions after an overwrite or deletion. It still shares the same storage system and control plane, so it does not replace an independent off-site or offline copy. Enable it for a bucket when it is created:

minio_buckets:
  - { name: pgsql, versioning: true }
  - { name: meta, versioning: true }
  - { name: data }

pgBackRest’s repo-target-time option can read the repository as it existed at an earlier time when the backend retains those versions.

Repository Locking

Some S3-compatible services support Object Lock/WORM. A retained object version cannot be changed or permanently deleted until its retention period ends. A normal delete can still create a delete marker that hides the current object while historical versions remain and continue consuming capacity.

The lock flag enables versioning and lock capability only when Pigsty creates the bucket:

minio_buckets:
  - { name: pgsql, lock: true }
  - { name: meta, versioning: true }
  - { name: data }

It does not set a default retention period. Configure GOVERNANCE or COMPLIANCE retention with mcli retention set or the storage console, then verify with mcli retention info. A sufficiently privileged principal can bypass GOVERNANCE; even root cannot shorten COMPLIANCE retention.

Locking changes expiration and stanza deletion: pgBackRest may expire objects logically while retained historical versions remain until their deadline. Test backup, expiration, delete-marker cleanup, and version recovery on a non-production bucket before enabling it.

Switch Repositories

After changing a repository definition or pgbackrest_method, rerender configuration, initialize the stanza, and create a recovery point in the new repository promptly:

./pgsql.yml -t pg_backup -l pg-meta           # Apply the repository definition and create the stanza
sudo -iu postgres pg-backup full              # Run on the current primary to establish the first recovery point

Existing backups are not migrated automatically. While retained, the old repository can still be selected as a restore source through pg_pitr.repo.

5.6 - Admin Commands

Backup administration reference covering setup and removal, manual backups, inspection, expiration, stanza management, logs, and alternative tools.

Run backup commands as the database superuser (pg_dbsu, postgres by default) on a database node. You can use any of these entry points:

  • pig pb: the PIG CLI wrapper, with automatic stanza detection, DBSU switching, and safety checks; this is the recommended interface
  • pb: a login-shell function that supplies --stanza and forwards arguments to pgBackRest
  • pgbackrest: the native command; see the pgBackRest command reference

Command Overview

PIG command Alias Native pgBackRest command Purpose
pig pb info i info Show backup and archive status
pig pb list ls — List repositories, stanzas, or backup sets
pig pb backup [full/diff/incr] b backup Create a backup after checking the primary role
pig pb restore r restore Low-level restore primitive; see Restore Operations
pig pb expire e expire Expire backups according to retention (--plan previews)
pig pb create c stanza-create Create a stanza
pig pb upgrade u stanza-upgrade Upgrade a stanza after a major-version change or clone
pig pb delete d stanza-delete Delete a stanza and all of its backups
pig pb check ck check Verify configuration, repository access, and archiving
pig pb start up start Re-enable pgBackRest operations
pig pb stop dw stop Stop new pgBackRest operations
pig pb log [list/show/tail] l — Inspect pgBackRest logs

Enable Backup

If pgbackrest_enabled is true when the cluster is created, backup is enabled automatically. If it was disabled at creation time, or repository settings have changed, run the pg_backup subtask:

./pgsql.yml -t pg_backup -l pg-meta   # Configure pgBackRest and create the stanza

After cluster initialization, Pigsty attempts an initial full backup. It writes /etc/pgbackrest/initial.done only after the backup command succeeds; the playbook ignores a failed attempt and leaves no marker. This file only prevents the initialization task from repeating, so always verify actual repository state with pig pb info or pgbackrest info. Define scheduled backups with pg_crontab; see Backup Policy.

Remove Backup

pig pb delete is the preferred interface when only a backup stanza must be removed. It asks for interactive confirmation; with a multi-stanza configuration, the target must also be explicit. Verify the exact target first:

pig pb info -s pg-meta           # Verify the backup chain and recovery window
pig pb delete -s pg-meta         # Executes only after typing the exact stanza name

When a primary instance (pg_role = primary) is removed, pgsql-rm.yml also tries to delete the cluster’s backup stanza by default. Every command below changes or deletes state; never execute one merely by copying the example:

./pgsql-rm.yml -l pg-meta                          # Remove the cluster and its backups
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Remove the cluster while preserving backups
./pgsql-rm.yml -l pg-meta -t pg_backup             # Remove backup-related state only

Before execution, verify a recent usable backup, record the recovery requirement, and have the operator re-enter the exact cluster/stanza name. Set pg_rm_backup to false to preserve backups while removing the cluster.

pgsql-rm.yml -t pg_backup forcibly runs pgbackrest stanza-delete on the primary, removes the local repository directory in local mode, then removes the pgBackRest configuration and initial-backup marker. The task ignores some deletion errors, so a successful playbook result does not prove that repository objects were physically removed. Prefer pig pb delete when only the stanza needs deletion because it supplies a plan and confirmation guard.

With object versioning and object-lock retention, deletion may create a delete marker while locked historical versions continue consuming storage until their retention period expires.

Backup Deletion

Deleting backups can permanently destroy recovery options. Confirm the cluster/stanza, verify a recent backup and an alternative recovery copy, and retain the pig pb info output and deletion plan as an audit record.

Manual Backup

You can trigger a backup outside the crontab schedule. Both pg-backup and pig pb backup check that the current instance is primary and exit on a replica:

pg-backup            # Incremental; pgBackRest promotes it to full if no full backup exists
pg-backup full       # Full backup
pg-backup diff       # Differential backup relative to the latest full
pg-backup incr       # Incremental backup relative to the latest backup

pig pb backup full   # Equivalent PIG wrapper with stanza and DBSU detection

Backup consumes disk I/O and network bandwidth. Pigsty limits parallelism to a small number of processes, but production runs should still be scheduled for low-traffic periods.

Inspect Backups

pb info shows backups and WAL archive status for the current stanza:

pb info          # pgbackrest --stanza=pg-meta info
pig pb info      # Equivalent PIG wrapper
pig pb list      # List backup sets; use "list stanza" for all stanzas

Backup labels ending in F, D, and I identify full, differential, and incremental backups. The portion before an underscore identifies the full backup anchoring that chain. The WAL archive range and the oldest usable full backup together bound the recovery window.

The pgbackrest_exporter service on port 9854 continuously exports metrics such as the latest backup time, type, size, and error status.

Expire Old Backups

The configured retention policy is applied automatically after backups (expire-auto). Preview or run expiration manually with:

pig pb expire --plan    # Show what would be removed; no deletion
pig pb expire           # Apply the configured retention policy

Stanza Management

A stanza records a cluster’s backup identity, including its system identifier and major version. Manual management is occasionally required:

pig pb create                    # Create a stanza; cluster initialization normally does this
pig pb upgrade                   # Update after a major-version upgrade or clone recovery
pig pb delete -s pg-meta         # Delete all backups and archives after confirmation; destructive

The usual manual upgrade case is post-clone cleanup: after restoring another cluster’s backup into a new cluster, update the stanza identity before new backups can be written.

Check and Control 

pig pb check     # End-to-end check of configuration, archive push, and repository access
pig pb stop      # Stop new pgBackRest operations for a maintenance window
pig pb start     # Re-enable pgBackRest operations

check performs an archive-path check rather than being purely local or read-only; it verifies that WAL can reach the repository.

Logs 

pig pb log              # Show the latest log snapshot
pig pb log list         # List log files
pig pb log tail         # Follow the latest log
ls /pg/log/pgbackrest/  # Directory containing backup, archive, and restore logs

For pgsql-pitr.yml, PostgreSQL recovery output is written to /pg/tmp/recovery.log.

Alternative Backup Tools

pg-basebackup

The legacy /pg/bin/pg-basebackup script creates a single-file physical backup using native pg_basebackup, an lz4-compressed tar stream, and /pg/backup by default. Use it only for a simple local copy when a pgBackRest repository is unavailable:

pg-basebackup                                      # /pg/backup/backup_<tag>_<date>.tar.lz4
pg-basebackup --dst /tmp --file backup.tar.lz4     # Explicit destination and filename

mkdir -p /tmp/data
cat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Legacy Encryption

pg-basebackup -e uses the obsolete OpenSSL RC4 cipher and must not be treated as confidentiality protection. For encrypted backups, use a pgBackRest repository configured with AES-256 (cipher_type: aes-256-cbc).

Logical Backup

Logical backups made with pg_dump cannot provide PITR, but they are appropriate for cross-major-version migration, partial exports, and long-term logical snapshots. Production recovery plans commonly use logical and physical backups together. See the PostgreSQL documentation.

6 - Data Migration

How to migrate an existing PostgreSQL cluster to a new Pigsty-managed PostgreSQL cluster with minimal downtime?

Pigsty includes a built-in playbook pgsql-migration.yml that implements online database migration based on logical replication.

With pre-generated automation scripts, application downtime can be reduced to just a few seconds. However, note that logical replication requires PostgreSQL 10 or later to work.

Of course, if you have sufficient downtime budget, you can always use the pg_dump | psql approach for offline migration.

Defining Migration Tasks

To use Pigsty’s online migration playbook, you need to create a definition file that describes the migration task details.

Refer to the task definition file example: files/migration/pg-meta.yml.

This migration task will online migrate pg-meta.meta to pg-test.test, where the former is called the Source Cluster (SRC) and the latter is called the Destination Cluster (DST).

pg-meta-1	10.10.10.10  --> pg-test-1	10.10.10.11 (10.10.10.12,10.10.10.13)

Logical replication-based migration works on a per-database basis. You need to specify the database name to migrate, as well as the IP addresses of the source and destination cluster primary nodes and superuser connection information.

---
#-----------------------------------------------------------------
# PG_MIGRATION
#-----------------------------------------------------------------
context_dir: ~/migration  # Directory for migration manual & scripts
#-----------------------------------------------------------------
# SRC Cluster (Old Cluster)
#-----------------------------------------------------------------
src_cls: pg-meta      # Source cluster name                  <Required>
src_db: meta          # Source database name                 <Required>
src_ip: 10.10.10.10   # Source cluster primary IP            <Required>
#src_pg: ''            # If defined, use this as source dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ src_ip }}/{{ src_db }}
#                      # e.g.: 'postgres://dbuser_dba:DBUser.DBA@10.10.10.10:5432/meta'
#sub_conn: ''          # If defined, use this as subscription connection string instead of:
#                      # host={{ src_ip }} dbname={{ src_db }} user={{ pg_replication_username }}'
#                      # e.g.: 'host=10.10.10.10 dbname=meta user=replicator password=DBUser.Replicator'
#-----------------------------------------------------------------
# DST Cluster (New Cluster)
#-----------------------------------------------------------------
dst_cls: pg-test      # Destination cluster name             <Required>
dst_db: test          # Destination database name            <Required>
dst_ip: 10.10.10.11   # Destination cluster primary IP       <Required>
#dst_pg: ''            # If defined, use this as destination dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ dst_ip }}/{{ dst_db }}
#                      # e.g.: 'postgres://dbuser_dba:DBUser.DBA@10.10.10.11:5432/test'
#-----------------------------------------------------------------
# PGSQL
#-----------------------------------------------------------------
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor
#-----------------------------------------------------------------
...

By default, the superuser connection strings on both source and destination sides are constructed using the global admin user and the respective primary IP addresses, but you can always override these defaults through the src_pg and dst_pg parameters. Similarly, you can override the subscription connection string default through the sub_conn parameter.

Generating Migration Plan

This playbook does not actively perform cluster migration, but it generates the operation manual and automation scripts needed for migration.

By default, you will find the migration context directory at ~/migration/pg-meta.meta. Follow the instructions in README.md and execute these scripts in sequence to complete the database migration!

# Activate migration context: enable related environment variables
. ~/migration/pg-meta.meta/activate

# These scripts check src cluster status and help generate new cluster definitions in pigsty
./check-user     # Check src users
./check-db       # Check src databases
./check-hba      # Check src hba rules
./check-repl     # Check src replication identity
./check-misc     # Check src special objects

# These scripts establish logical replication between existing src cluster and pigsty-managed dst cluster, data except sequences will sync in real-time
./copy-schema    # Copy schema to destination
./create-pub     # Create publication on src
./create-sub     # Create subscription on dst
./copy-progress  # Print logical replication progress
./copy-diff      # Quick compare src and dst differences by counting tables

# These scripts run during online migration, which stops src cluster and copies sequence numbers (logical replication doesn't replicate sequences!)
./copy-seq [n]   # Sync sequence numbers, if n is given, apply additional offset

# You must switch application traffic to the new cluster based on your access method (dns,vip,haproxy,pgbouncer,etc.)!
#./disable-src   # Restrict src cluster access to admin nodes and new cluster (your implementation)
#./re-routing    # Re-route application traffic from SRC to DST! (your implementation)

# Then cleanup to remove subscription and publication
./drop-sub       # Drop subscription on dst after migration
./drop-pub       # Drop publication on src after migration

Notes

If you’re worried about primary key conflicts when copying sequence numbers, you can advance all sequences forward by some distance when copying, for example +1000. You can use ./copy-seq with a parameter 1000 to achieve this.

You must implement your own ./re-routing script to route your application traffic from src to dst. Because we don’t know how your traffic is routed (e.g., dns, VIP, haproxy, or pgbouncer). Of course, you can also do this manually…

You can implement a ./disable-src script to restrict application access to the src cluster—this is optional: if you can ensure all application traffic is cleanly switched in ./re-routing, you don’t really need this step.

But if you have various access from unknown sources that can’t be cleanly sorted out, it’s better to use more thorough methods: change HBA rules and reload to implement (recommended), or simply stop the postgres, pgbouncer, or haproxy processes on the source primary.

7 - Tutorials

Step-by-step guides for common PostgreSQL tasks and scenarios.

This section provides step-by-step tutorials for common PostgreSQL tasks and scenarios.

  • Citus Cluster: Deploy and manage Citus distributed clusters
  • Disaster Drill: Emergency recovery when 2 of 3 nodes fail
  • PG VIP: Configure L2 VIP for PostgreSQL clusters

7.1 - Troubleshooting

Common failures and analysis troubleshooting approaches

This document lists potential failures in PostgreSQL and Pigsty, as well as SOPs for locating, handling, and analyzing issues.

Disk Space Exhausted

Disk space exhaustion is the most common type of failure.

Symptoms

When the disk space where the database resides is exhausted, PostgreSQL will not work normally and may exhibit the following symptoms: database logs repeatedly report “no space left on device” errors, new data cannot be written, and PostgreSQL may even trigger a PANIC and force shutdown.

Pigsty includes a NodeFsSpaceFull alert rule that triggers when filesystem available space is less than 10%. Use the monitoring system’s NODE Instance panel to review the FS metrics panel to locate the issue.

Diagnosis

You can also log into the database node and use df -h to view the usage of each mounted partition to determine which partition is full. For database nodes, focus on checking the following directories and their sizes to determine which category of files has filled up the space:

  • Data directory (/pg/data/base): Stores data files for tables and indexes; pay attention to heavy writes and temporary files
  • WAL directory (e.g., pg/data/pg_wal): Stores PG WAL; WAL accumulation/replication slot retention is a common cause of disk exhaustion.
  • Database log directory (e.g., pg/log): If PG logs are not rotated in time and large amounts of errors are written, they may also consume significant space.
  • Local backup directory (e.g., data/backups): When using pgBackRest or similar tools to save backups locally, this may also fill up the disk.

If the issue occurs on the Pigsty admin node or monitoring node, also consider:

  • Monitoring data: VictoriaMetrics time-series metrics and VictoriaLogs log storage both consume disk space; check retention policies.
  • Object storage data: Pigsty’s integrated Silo object storage may be used for PG backup storage.

After identifying the directory consuming the most space, you can further use du -sh <directory> to drill down and find specific large files or subdirectories.

Resolution

Disk exhaustion is an emergency issue requiring immediate action to free up space and ensure the database continues to operate. When the data disk is not separated from the system disk, a full disk may prevent shell commands from executing. In this case, you can delete the /pg/dummy placeholder file to free up a small amount of emergency space so shell commands can work again. If the database has crashed due to pg_wal filling up, you need to restart the database service after clearing space and carefully check data integrity.

Transaction ID Wraparound

PostgreSQL cyclically uses 32-bit transaction IDs (XIDs), and when exhausted, a “transaction ID wraparound” failure occurs (XID Wraparound).

Symptoms

The typical sign in the first phase is when the age saturation in the PGSQL Persist - Age Usage panel enters the warning zone. Database logs begin to show messages like: WARNING: database "postgres" must be vacuumed within xxxxxxxx transactions.

If the problem continues to worsen, PostgreSQL enters protection mode: when remaining transaction IDs drop to about 1 million, the database switches to read-only mode; when reaching the limit of about 2.1 billion (2^31), it refuses any new transactions and forces the server to shut down to avoid data corruption.

Diagnosis

PostgreSQL and Pigsty enable automatic garbage collection (AutoVacuum) by default, so the occurrence of this type of failure usually has deeper root causes. Common causes include: very long transactions (SAGE), misconfigured Autovacuum, replication slot blockage, insufficient resources, storage engine/extension bugs, disk bad blocks.

First identify the database with the highest age, then use the Pigsty PGCAT Database - Tables panel to confirm the age distribution of tables. Also review the database error logs, which usually contain clues to locate the root cause.

Resolution

  1. Immediately freeze old transactions: If the database has not yet entered read-only protection mode, immediately execute a manual VACUUM FREEZE on the affected database. You can start by freezing the most severely aged tables one by one rather than doing the entire database at once to accelerate the effect. Connect to the database as a superuser and run VACUUM FREEZE table_name; on tables identified with the largest relfrozenxid, prioritizing tables with the highest XID age. This can quickly reclaim large amounts of transaction ID space.
  2. Single-user mode rescue: If the database is already refusing writes or has crashed for protection, you need to start the database in single-user mode to perform freeze operations. In single-user mode, run VACUUM FREEZE database_name; to freeze and clean the entire database. After completion, restart the database in multi-user mode. This can lift the wraparound lock and make the database writable again. Be very careful when operating in single-user mode and ensure sufficient transaction ID margin to complete the freeze.
  3. Standby node takeover: In some complex scenarios (e.g., when hardware issues prevent vacuum from completing), consider promoting a read-only standby node in the cluster to primary to obtain a relatively clean environment for handling the freeze. For example, if the primary cannot vacuum due to bad blocks, you can manually failover to promote the standby to the new primary, then perform emergency vacuum freeze on it. After ensuring the new primary has frozen old transactions, switch the load back.

Connection Exhaustion

PostgreSQL has a maximum connections configuration (max_connections). When client connections exceed this limit, new connection requests will be rejected. The typical symptom is that applications cannot connect to the database and report errors like FATAL: remaining connection slots are reserved for non-replication superuser connections or too many clients already. This indicates that regular connections are exhausted, leaving only slots reserved for superusers or replication.

Diagnosis

Connection exhaustion is usually caused by a large number of concurrent client requests. You can directly review the database’s current active sessions through PGCAT Instance / PGCAT Database / PGCAT Locks. Determine what types of queries are filling the system and proceed with further handling. Pay special attention to whether there are many connections in the “Idle in Transaction” state and long-running transactions (as well as slow queries).

Resolution

Kill queries: For situations where exhaustion has already blocked business operations, typically use pg_terminate_backend(pid) immediately for emergency pressure relief. For cases using connection pooling, you can adjust the connection pool size parameters and execute a reload to reduce the number of connections at the database level.

You can also modify the max_connections parameter to a larger value, but this parameter requires a database restart to take effect.

etcd Quota Exhausted

An exhausted etcd quota will cause the PG high availability control plane to fail and prevent configuration changes.

Diagnosis

Pigsty uses etcd as the distributed configuration store (DCS) when implementing high availability. etcd itself has a storage quota (default is about 2GB). When etcd storage usage reaches the quota limit, etcd will refuse write operations and report “etcdserver: mvcc: database space exceeded”. In this case, Patroni cannot write heartbeats or update configuration to etcd, causing cluster management functions to fail.

Resolution

Versions between Pigsty v2.0.0 and v2.5.1 are affected by this issue by default. Pigsty v2.6.0 added auto-compaction configuration for deployed etcd. If you only use it for PG high availability leases, this issue will no longer occur in regular use cases.

Defective Storage Engine

Currently, TimescaleDB’s experimental storage engine Hypercore has been proven to have defects, with cases of VACUUM being unable to reclaim leading to XID wraparound failures. Users using this feature should migrate to PostgreSQL native tables or TimescaleDB’s default engine promptly.

Detailed introduction: PG New Storage Engine Failure Case (Chinese)

7.2 - Manual PITR Drill

Run, validate, and complete PostgreSQL point-in-time recovery in explicit stages inside an isolated sandbox.

This tutorial drills PostgreSQL point-in-time recovery in Pigsty v4.5.0’s four-node sandbox. The main path runs pgsql-pitr.yml as down → pitr → up, giving the operator a separate validation gate before data overwrite, timeline promotion, and HA reconstruction.

For one current node, use pig pitr. For direct pgBackRest control, see the low-level pg-pitr utility.

Follow this literally only in a disposable sandbox

Recovery stops Patroni/PostgreSQL and overwrites the target PGDATA with pgbackrest --force restore; the up stage also deletes the target cluster’s etcd prefix and rebuilds Patroni state. The playbook prints a plan but has no interactive confirmation. Before production use, the operator must state and confirm the exact cluster and recovery point, verify a recent usable backup that has been independently tested, run --check with exactly the same -l, variables, and tags, and schedule a maintenance window. This tutorial does not authorize running these commands in any production environment.

Prepare an Isolated Sandbox

Use Vagrant or another disposable four-node lab and select the ha/full template, which includes a Silo backup repository:

curl -fsSL https://repo.pigsty.io/get | bash
cd ~/pigsty
./configure -c ha/full
./deploy.yml

ha/full defines the single-node pg-meta, three-node pg-test, and a Silo/pgBackRest repository. The rest of this tutorial uses the exact target pg-meta; do not copy that selector into another environment without resolving its inventory first.

Initial deployment and backup both change sandbox state. Production environments require their own deployment and backup approval process.

Establish Recovery Evidence

Start with read-only topology, backup-chain, and WAL-range checks:

pig pg list pg-meta
sudo -iu postgres pig pb info
sudo -iu postgres pig pb check

info must show at least one usable backup with status: ok, and archived WAL must cover the intended target. check validates the current stanza and archive path, but it does not replace a real restore drill or independent-copy validation.

In the sandbox, run Pigsty’s heartbeat helper to create an easy-to-verify time series:

sudo -iu postgres /pg/bin/pg-heartbeat

Record, then stop the workload:

  • the timezone-qualified timestamp you intend to recover to;
  • heartbeat, LSN, and transaction boundaries around that point;
  • current primary, timeline, and backup label;
  • target cluster pg-meta and target node.

Inspecting real application tables requires separate authorization. This tutorial uses only sandbox heartbeat data.

Declare the Recovery Task

Declare the target under pg-meta.vars in the sandbox inventory:

pg_pitr:
  cluster: pg-meta
  time: "2026-08-13 10:00:00+08"
  action: pause
  archive: true
  backup: false
  • cluster is the source backup stanza and defaults to the target pg_cluster.
  • action: pause pauses PostgreSQL at the target for a human validation gate.
  • archive: true preserves archive settings.
  • backup: true is not a safe-backup substitute: it deletes an existing <pg_data>-backup before moving current PGDATA, so this drill keeps it false.

The same object can be supplied temporarily with -e, but preflight and all three stages must repeat the exact same valid JSON to prevent variable drift.

Full Preflight

Before any stop or write action, check the complete workflow against the same target:

./pgsql-pitr.yml -l pg-meta --check

Confirm that Ansible resolves exactly pg-meta, then review the output for:

  • source stanza, recovery type, time, timeline, and action;
  • destination pg_data, port, and repository;
  • tablespace/link mappings;
  • archive and backup behavior.

--check validates inventory, variables, and task selection. It cannot prove that a pgBackRest backup is restorable. Any change to target, backup, or variables requires a new preflight.

Stage One: Stop

Only after the operator reconfirms exact target pg-meta, recovery point, and maintenance window, run:

./pgsql-pitr.yml -l pg-meta -t down

down attempts to pause Patroni automatic failover, stops Patroni on every target member, and uses immediate shutdown if PostgreSQL remains running. Then verify every target node rather than trusting only the playbook result:

sudo systemctl is-active patroni
sudo -iu postgres pg_ctl -D /pg/data status

Expected results are inactive and “server is not running.” If any member remains active, stop and diagnose; do not enter restore.

Stage Two: Restore and Validate

Recheck pg_pitr and the target nodes before running the destructive stage:

./pgsql-pitr.yml -l pg-meta -t pitr

This stage:

  1. renders /pg/conf/pitr.conf and /pg/bin/pg-restore;
  2. optionally moves old PGDATA according to backup;
  3. creates the destination and runs pgBackRest restore with --force and delta=y;
  4. starts PostgreSQL directly and waits for a consistent recovery state in the log;
  5. prints a pg_controldata summary.

Control data proves only that the directory has readable control state; it does not prove that a time, XID, or application boundary is correct. With action: pause, confirm WAL reached and paused near the target:

sudo -iu postgres psql -p 5432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp()'

Then inspect only the smallest authorized data scope; in the sandbox, check heartbeat rows. If the target is wrong:

  1. keep every Patroni member stopped;
  2. stop the manually started PostgreSQL;
  3. adjust the target and rerun the complete --check;
  4. rerun the pitr stage.

Do not run up or allow replicas from the old timeline to rejoin.

Promote and Stage Three: Rebuild HA

Promote only after the operator accepts the recovered result and the creation of a new timeline:

sudo -iu postgres pg_ctl -D /pg/data promote
sudo -iu postgres psql -p 5432 -Atqc 'SELECT pg_is_in_recovery()'

The expected result is f. Promotion is not read-only validation and cannot be losslessly undone.

With every Patroni member still stopped and exact target still pg-meta, run:

./pgsql-pitr.yml -l pg-meta -t up

up deletes the /pg/pg-meta/ prefix from etcd for the primary (the effective prefix also depends on pg_namespace and Citus settings), stops the manually started PostgreSQL, starts Patroni on the primary, then starts replicas one by one and resumes HA. The etcd deletion task tolerates errors, so a successful playbook result does not prove that stale DCS state was removed correctly.

Post-Recovery Acceptance

Verify each layer; “service started” is not recovery completion:

pig pg list pg-meta
sudo -iu postgres psql -Atqc \
  "SELECT pg_is_in_recovery(), pg_current_wal_lsn(), current_setting('archive_mode')"
sudo -iu postgres pig pb check

Also confirm:

  • exactly the intended member is primary and replicas stream from the new timeline;
  • HAProxy/VIP/DNS and application traffic point only to accepted instances;
  • data and event boundaries around the recovery target are correct;
  • archive_mode, archive_command, and new-WAL archiving work;
  • monitoring, alerts, and the backup repository contain no stale cluster state.

After the new timeline is stable, create and verify a new full backup under the applicable approval process:

sudo -iu postgres pg-backup full
sudo -iu postgres pig pb info

If the recovery explicitly used archive: false, it wrote archive-mode=off. Reset that override and perform a controlled restart only after validating recovery and confirming a maintenance window; default archive: true does not require this step.

Multi-Node and Cross-Cluster Recovery

  • Old-timeline replicas must not rejoin without validation. up starts replicas one at a time and waits for clone/recovery; monitor them to completion.
  • In cross-stanza recovery, pg_pitr.cluster is the source while -l is the destination being overwritten. Record and state both separately.
  • Cross-cluster recovery should normally use archive: false so a test destination cannot write WAL into the source stanza. Enable its own archiving only after acceptance and post-clone stanza cleanup.
  • link_map, data, port, and a temporary repo change the actual storage and data targets; include all of them in both --check and human review.

Related

7.3 - Enabling HugePage for PostgreSQL

Enabling HugePage for PostgreSQL to reduce memory fragmentation and improve performance.

Use node_hugepage_count and node_hugepage_ratio or /pg/bin/pg-tune-hugepage

If you plan to enable HugePages, consider using node_hugepage_count and node_hugepage_ratio, and apply with ./node.yml -t node_tune.

HugePages have pros and cons for databases. The advantage is that memory is managed exclusively, eliminating concerns about being reallocated and reducing database OOM risk. The disadvantage is that it may negatively impact performance in certain scenarios.

Before PostgreSQL starts, you need to allocate enough huge pages. The wasted portion can be reclaimed using the pg-tune-hugepage script, but this script is only available for PostgreSQL 15+.

If your PostgreSQL is already running, you can enable huge pages using the following method (PG15+ only):

sync; echo 3 > /proc/sys/vm/drop_caches   # Flush disk, release system cache (be prepared for database perf impact)
sudo /pg/bin/pg-tune-hugepage             # Write nr_hugepages to /etc/sysctl.d/hugepage.conf
pg restart <cls>                          # Restart postgres to use hugepage

7.4 - Clone and Side-Restore a PostgreSQL Instance

Create a local physical copy with pg-fork and run low-level pgBackRest recovery against a stopped data directory with pg-pitr.

Pigsty v4.5.0 provides two local shell utilities:

  • pg-fork copies a PostgreSQL data directory and gives the copy a separate port.
  • pg-pitr invokes pgBackRest to restore a stopped data directory to a selected target.

They are useful for sandbox drills, side-channel investigation, and temporary testing. They are not complete Patroni-cluster recovery orchestrators. Prefer pig pitr for a managed instance and staged pgsql-pitr.yml for a multi-node cluster.

Verify paths, backups, and stopped state first

pg-fork recursively removes an existing destination directory; pg-pitr overwrites the destination with backup data. Both can execute without a prompt in a non-interactive environment. Before a real run, verify source and destination absolute paths, ports, tablespaces, exact cluster/instance identity, and an independent recent backup that has been tested. A newly created CoW clone is not an independent backup.

pg-fork

pg-fork copies a PostgreSQL data directory on the current node. Run it as the database OS user—normally postgres, or at least a member of the postgres group:

pg-fork 1                         # /pg/data -> /pg/data1, destination port 15432
pg-fork 2 -d /pg/data1            # /pg/data1 -> /pg/data2, destination port 25432
pg-fork 3 -D /srv/pg-clone -P 55432

Parameters 

pg-fork <FORK_ID> [options]
Parameter Meaning Default
<FORK_ID> One digit from 1 to 9, used to derive the default directory and port Required
-d, --data <path> Source data directory $PG_DATA or /pg/data
-D, --dst <path> Destination data directory /pg/data<FORK_ID>
-p, --port <port> Source instance port $PG_PORT or 5432
-P, --dst-port <port> Destination instance port <FORK_ID>5432
-s, --skip Skip the online-backup API and force cold-copy mode No
-y, --yes Skip interactive confirmation No

The script rejects equal normalized source and destination paths, but it cannot know whether a custom destination contains other important data. If the destination directory exists, it is recursively removed before copying.

Hot Backup and Cold Copy

By default, the script connects to the source port and uses one psql session to run:

  1. CHECKPOINT;
  2. pg_backup_start();
  3. rm -rf <destination> followed by cp -a --reflink=auto;
  4. pg_backup_stop(wait_for_archive => false).

If the source cannot be reached on the selected port, the script automatically falls back to a cold copy instead of aborting. -s also forces a cold copy. A cold copy is safe only after you independently confirm that the source instance is fully stopped; a postmaster.pid warning is only a clue, not proof of process state.

On the same filesystem, the implementation recognizes these as fast CoW modes: XFS with reflink enabled, Btrfs, Bcachefs, and OCFS2. Other filesystems or cross-filesystem destinations still use cp --reflink=auto but may fall back to a full copy. The script’s help text mentions ZFS more broadly than its detector; the v4.5.0 implementation does not classify ZFS as a confirmed fast-CoW mode.

Clone Configuration

After a successful copy, pg-fork:

  • removes postmaster.pid, postmaster.opts, and standby.signal from the destination;
  • clears physical replication-slot files in the destination;
  • writes a separate port, archive_mode=off, and a local log_directory to the destination’s postgresql.auto.conf;
  • removes primary_conninfo, primary_slot_name, and old recovery_target* overrides.

It does not check whether the destination port is free or resize memory settings. Before starting the copy, inspect at least:

postgres -D /pg/data1 -C port
postgres -D /pg/data1 -C archive_mode
postgres -D /pg/data1 -C shared_buffers
pg_ctl -D /pg/data1 status
External tablespaces are not isolated

cp -a preserves symlinks under pg_tblspc; pg-fork does not copy or remap tablespaces outside PGDATA. Starting such a clone can access or modify the source instance’s tablespaces. If external tablespaces exist, independently copy and remap every one of them, or do not use this script to create a writable clone.

Interaction Boundary

The script asks Proceed with fork? [y/N] only when standard input is a terminal and -y was not used. Pipes, CI, cron, and other non-interactive invocations do not receive that prompt. Automation must therefore enforce a strict absolute-path allowlist and destination-existence check before invoking the script; do not add -y by default merely for convenience.

pg-pitr

pg-pitr is a low-level pgBackRest restore wrapper. It does not pause or start Patroni, stop or start PostgreSQL, clear DCS state, or rebuild replicas.

Recovery Targets

Understand and select a recovery target before execution. Invoking the command without arguments only shows help:

Parameter pgBackRest semantics
-d, --default Set no stop target and replay to the available end of WAL
-i, --immediate Stop when the selected backup becomes consistent
-t, --time <timestamp> Recover to a timestamp
-n, --name <restore-point> Recover to a named restore point
-l, --lsn <lsn> Recover to an LSN
-x, --xid <xid> Recover to a transaction ID

-S/--set (with compatibility alias -b/--backup) only selects which backup set recovery starts from; it is not a stop target. For example, -S 20251225-120000F -d still replays to the end of WAL. Combine -S ... -i to stop as soon as that backup becomes consistent.

For time, name, lsn, xid, and immediate, pgBackRest’s effective default action is to pause at the target; -P/--promote changes it to automatic promotion. Use -X/--exclusive only with a precise boundary such as time, lsn, or xid.

Other Options

Parameter Meaning
-D, --data <path> Absolute destination data directory; default /pg/data
-s, --stanza <name> pgBackRest stanza; defaults to the first non-global stanza in the config
-T, --timeline <value> latest, current, or a positive numeric timeline
-P, --promote Automatically promote recovery methods that have a stop target
-v, --verbose Enable pgBackRest info-level console logging
-c, --check, --dry-run Print the command without executing it
-y, --yes Skip the five-second countdown
-- <args> Pass additional arguments directly to pgBackRest

-c is a command-rendering check. It does not prove that the backup/WAL is usable or that PostgreSQL and Patroni are stopped. The wrapper also does not filter conflicting native arguments; review the final command carefully when passing repository, tablespace, or link-mapping options.

Safe Execution Sequence

This example shows only the low-level flow for one isolated destination. Use the complete runbook for production cluster recovery:

# 1. Read-only verification of backup sets and the recovery window
pig pb info

# 2. Confirm the exact target is stopped; stop Patroni first for a managed instance
pg_ctl -D /pg/data1 status

# 3. Render and review the exact command without writing data
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08" -c

# 4. Remove -c only after the operator reconfirms destination, backup, and stopped state
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08"

Real execution refuses root and aborts whenever postmaster.pid exists in the destination. Even a stale PID file requires the operator to confirm that PostgreSQL is stopped before removing it. There is no y/N question: an interactive terminal gets only an interruptible five-second countdown; a non-interactive invocation skips the countdown and enters restore immediately.

After restore, the operator starts and validates the instance:

pg_ctl -D /pg/data1 start
psql -p 15432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_xact_replay_timestamp()'

Promote only after the recovery target, authorized application data, timeline, and archive settings are all verified. Promotion creates a new timeline; it is not a reversible “inspect” operation. pg-pitr does not itself disable archiving. Do not mechanically follow its generic final “enable archive_mode” hint—inspect the effective value first and correct only an override introduced by this recovery.

Additional Side-Restore Risks

When restoring into a custom directory such as /pg/data1, pgBackRest can restore postgresql.auto.conf from the backup and overwrite the separate port written by pg-fork. Recheck port, archive_mode, sockets, logging, and memory settings before startup.

If the backup contains external tablespaces or links, a side restore can also use the original paths. For isolation, pass reviewed pgBackRest --tablespace-map, --link-map, or related arguments after -- and inspect the rendered command. Otherwise, do not start the restored copy on the same host as production.

Recommended Clone-Validation Flow

  1. Verify the source instance, destination absolute path, destination port, tablespaces, and independent backup.
  2. Run pg-fork <id> in an interactive terminal and confirm that the plan shows hot backup rather than an unintended cold-copy fallback.
  3. Before starting the clone, run pg-pitr -D <clone> ... -c and inspect the recovery command.
  4. Execute restore only after explicit destination confirmation, then recheck the clone’s port and every external path.
  5. Start the clone on the isolated port and verify recovery state and only the application data you are authorized to inspect.
  6. Promote only if the clone is intentionally becoming a new primary; otherwise stop it and clean up only the exact, verified path.

Side validation reduces direct writes to the current PGDATA, but still uses the same backup repository, consumes host resources, and may touch external tablespaces. It is not a risk-free sandbox.

Related

7.5 - Accidental Deletion

Handling accidental data deletion, table deletion, and database deletion

Accidental Data Deletion

If it’s a small-scale DELETE misoperation, you can consider using the pg_surgery or pg_dirtyread extension for in-place surgical recovery.

-- Immediately disable Auto Vacuum on this table and abort Auto Vacuum worker processes for this table
ALTER TABLE public.some_table SET (autovacuum_enabled = off, toast.autovacuum_enabled = off);

CREATE EXTENSION pg_dirtyread;
SELECT * FROM pg_dirtyread('tablename') AS t(col1 type1, col2 type2, ...);

If the deleted data has already been reclaimed by VACUUM, then use the general accidental deletion recovery process.

Accidental Object Deletion

When DROP/DELETE type misoperations occur, typically decide on a recovery plan according to the following process:

  1. Confirm whether this data can be recovered from the business system or other data systems. If yes, recover directly from the business side.
  2. Confirm whether there is a delayed replica. If yes, advance the delayed replica to the time point before deletion and query the data for recovery.
  3. If the data has been confirmed deleted, confirm backup information and whether the backup range covers the deletion time point. If it does, start PITR.
  4. Confirm whether to perform in-place cluster PITR rollback, or start a new server for replay, or use a replica for replay, and execute the recovery strategy.

Accidental Cluster Deletion

If an entire database cluster is accidentally deleted through Pigsty management commands, for example, incorrectly executing the pgsql-rm.yml playbook or the bin/pgsql-rm command. Unless you have set the pg_rm_backup parameter to false, the backup will be deleted along with the database cluster.

Warning: In this situation, your data will be unrecoverable! Please think three times before proceeding!

Recommendation: For production environments, you can globally configure this parameter to false in the configuration manifest to preserve backups when removing clusters.

7.6 - HA Drill: Handling 2-of-3 Node Failure

HA scenario response plan: When two of three nodes fail and auto-failover doesn’t work, how to recover from the emergency state?

If a classic 3-node HA deployment experiences simultaneous failure of two nodes (majority), the system typically cannot complete automatic failover and requires manual intervention.

First, assess the status of the other two servers. If they can be brought up quickly, prioritize recovering those two servers. Otherwise, enter the Emergency Recovery Procedure.

The Emergency Recovery Procedure assumes your admin node has failed and only a single regular database node survives. In this case, the fastest recovery process is:

  • Adjust HAProxy configuration to direct traffic to the primary.
  • Stop Patroni and manually promote the PostgreSQL replica to primary.

Adjust HAProxy Configuration

If you access the cluster bypassing HAProxy, you can skip this step. If you access the database cluster through HAProxy, you need to adjust the load balancer configuration to manually direct read/write traffic to the primary.

  • Edit /etc/haproxy/conf.d/<pg_cluster>-primary.cfg, where <pg_cluster> is your PostgreSQL cluster name, e.g., pg-meta.
  • Comment out the health check configuration options to stop health checks.
  • Comment out the other two failed machines in the server list, keeping only the current primary server.
listen pg-meta-primary
    bind *:5433
    mode tcp
    maxconn 5000
    balance roundrobin

    # Comment out the following four health check lines
    #option httpchk                               # <---- remove this
    #option http-keep-alive                       # <---- remove this
    #http-check send meth OPTIONS uri /primary    # <---- remove this
    #http-check expect status 200                 # <---- remove this

    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    server pg-meta-1 10.10.10.10:6432 check port 8008 weight 100

    # Comment out the other two failed machines
    #server pg-meta-2 10.10.10.11:6432 check port 8008 weight 100 <---- comment this
    #server pg-meta-3 10.10.10.12:6432 check port 8008 weight 100 <---- comment this

After adjusting the configuration, don’t rush to execute systemctl reload haproxy to reload. Wait until after promoting the primary, then execute together. The effect of this configuration is that HAProxy will no longer perform primary health checks (which by default use Patroni), but will directly direct write traffic to the current primary.

Manually Promote Replica

Log in to the target server, switch to the dbsu user, execute CHECKPOINT to flush to disk, stop Patroni, restart PostgreSQL, and execute Promote.

sudo su - postgres                     # Switch to database dbsu user
psql -c 'checkpoint; checkpoint;'      # Two Checkpoints to flush dirty pages, avoid long PG restart
sudo systemctl stop patroni            # Stop Patroni
pg-restart                             # Restart PostgreSQL
pg-promote                             # Promote PostgreSQL replica to primary
psql -c 'SELECT pg_is_in_recovery();'  # If result is f, it has been promoted to primary

If you adjusted the HAProxy configuration above, you can now execute systemctl reload haproxy to reload the HAProxy configuration and direct traffic to the new primary.

systemctl reload haproxy                # Reload HAProxy configuration to direct write traffic to current instance

Avoid Split Brain

After emergency recovery, the second priority is: Avoid Split Brain. Users should prevent the other two servers from coming back online and forming a split brain with the current primary, causing data inconsistency.

Simple approaches:

  • Power off/disconnect network the other two servers to ensure they don’t come online uncontrollably.
  • Adjust the database connection string used by applications to point directly to the surviving server’s primary.

Then decide the next steps based on the specific situation:

  • A: The two servers have temporary failures (e.g., network/power outage) and can be repaired in place to continue service.
  • B: The two failed servers have permanent failures (e.g., hardware damage) and will be removed and decommissioned.

Recovery After Temporary Failure

If the other two servers have temporary failures and can be repaired to continue service, follow these steps for repair and rebuild:

  • Handle one failed server at a time, prioritize the admin node / INFRA node.
  • Start the failed server and stop Patroni after startup.

After the ETCD cluster quorum is restored, it will resume work. Then start Patroni on the surviving server (current primary) to take over the existing PostgreSQL and regain cluster leadership. After Patroni starts, enter maintenance mode.

sudo systemctl restart patroni
pg pause <pg_cluster>

On the other two instances, create the touch /pg/data/standby.signal marker file as the postgres user to mark them as replicas, then start Patroni:

sudo -iu postgres touch /pg/data/standby.signal
sudo systemctl restart patroni

After confirming Patroni cluster identity/roles are correct, exit maintenance mode:

pg resume <pg_cluster>

Recovery After Permanent Failure

After permanent failure, first recover the ~/pigsty directory on the admin node. The key files needed are pigsty.yml and files/pki/ca/ca.key.

If you cannot retrieve or don’t have backups of these two files, you can deploy a new Pigsty and migrate the existing cluster to the new deployment via Backup Cluster.

Please regularly backup the pigsty directory (e.g., using Git for version control). Learn from this and avoid such mistakes in the future.

Configuration Repair

You can use the surviving node as the new admin node, copy the ~/pigsty directory to the new admin node, then start adjusting the configuration. For example, replace the original default admin node 10.10.10.10 with the surviving node 10.10.10.12:

all:
  vars:
    admin_ip: 10.10.10.12               # Use new admin node address
    node_etc_hosts: [10.10.10.12 h.pigsty a.pigsty p.pigsty g.pigsty sss.pigsty]
    infra_portal: {}                    # Also modify other configs referencing old admin IP (10.10.10.10)

  children:

    infra:                              # Adjust Infra cluster
      hosts:
        # 10.10.10.10: { infra_seq: 1 } # Old Infra node
        10.10.10.12: { infra_seq: 3 }   # New Infra node

    etcd:                               # Adjust ETCD cluster
      hosts:
        #10.10.10.10: { etcd_seq: 1 }   # Comment out this failed node
        #10.10.10.11: { etcd_seq: 2 }   # Comment out this failed node
        10.10.10.12: { etcd_seq: 3 }    # Keep surviving node
      vars:
        etcd_cluster: etcd

    pg-meta:                            # Adjust PGSQL cluster configuration
      hosts:
        #10.10.10.10: { pg_seq: 1, pg_role: primary }
        #10.10.10.11: { pg_seq: 2, pg_role: replica }
        #10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
        10.10.10.12: { pg_seq: 3, pg_role: primary , pg_offline_query: true }
      vars:
        pg_cluster: pg-meta

ETCD Repair

Then execute the following command to reset ETCD to a single-node cluster:

./etcd.yml -e etcd_safeguard=false -e etcd_clean=true

Follow the instructions in ETCD Reload Configuration to adjust ETCD Endpoint references.

INFRA Repair

If the surviving node doesn’t have the INFRA module, configure and install a new INFRA module on the current node. Execute the following command to deploy the INFRA module to the surviving node:

./infra.yml -l 10.10.10.12

Repair monitoring on the current node:

./node.yml -t monitor

PGSQL Repair 

./pgsql.yml -t pg_conf                            # Regenerate PG configuration files
systemctl reload patroni                          # Reload Patroni configuration on surviving node

After repairing each module, you can follow the standard expansion process to add new nodes to the cluster and restore cluster high availability.

7.7 - Bind a L2 VIP to PostgreSQL Primary with VIP-Manager

You can define an OPTIONAL L2 VIP on a PostgreSQL cluster, provided that all nodes in the cluster are in the same L2 network.

This VIP works on Master-Backup mode and always points to the node where the primary instance of the database cluster is located.

This VIP is managed by the VIP-Manager, which reads the Leader Key written by Patroni from DCS (etcd) to determine whether it is the master.

Enable VIP

Define pg_vip_enabled parameter as true in the cluster level to enable the VIP component on the cluster. You can also enable this configuration in the global configuration.

# pgsql 3 node ha cluster: pg-test
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
    10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
    10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24
    #pg_vip_interface: auto

Beware that pg_vip_address must be a valid IP address with subnet and available in the current L2 network.

pg_vip_interface defaults to auto, in which case Pigsty detects the interface for each instance from the IPv4 address in the inventory.

If auto-detection is unsuitable for non-standard or policy-routing environments, explicitly specify a valid interface name for each instance, for example:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_vip_interface: eth0  }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_vip_interface: eth1  }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_vip_interface: ens33 }
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24

To refresh the VIP configuration and restart the VIP-Manager, use the following command:

./pgsql.yml -t pg_vip

7.8 - Deploy HA Citus Cluster

How to deploy a Citus high-availability distributed cluster?

Citus is a PostgreSQL extension that transforms PostgreSQL into a distributed database, enabling horizontal scaling across multiple nodes to handle large amounts of data and queries.

Patroni v3.0+ provides native high-availability support for Citus, simplifying the setup of Citus clusters. Pigsty also provides native support for this.

Note

Citus 13.x supports PostgreSQL 18, 17, 16, 15, and 14. The Pigsty extension repository provides Citus ARM64 packages.

Citus Cluster

Pigsty natively supports Citus. The current complete configuration template is conf/ha/citus.yml.

The simplified four-node topology below illustrates the key parameters: a two-node coordinator cluster pg-citus0 and two single-node Worker clusters, pg-citus1 and pg-citus2. It is not a line-for-line excerpt from the current complete template.

pg-citus:
  hosts:
    10.10.10.10: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.11: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 2, pg_role: replica }
    10.10.10.12: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.3/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.13: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.4/24 ,pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # all dbsu password access for citus cluster
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # explicitly preload citus and keep it first
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Compared to standard PostgreSQL clusters, Citus cluster configuration has some special requirements. First, you need to ensure the Citus extension is downloaded, installed, loaded, and enabled, which involves the following four parameters:

  • repo_packages: Must include the citus extension, or you need to use a PostgreSQL offline package that includes Citus.
  • pg_extensions: Must include the citus extension, i.e., you must install the citus extension on each node.
  • pg_libs: Must explicitly include citus in the first position; current Patroni templates use this parameter directly for shared_preload_libraries.
  • pg_databases: Define a primary database that must have the citus extension installed.

Second, you need to ensure the Citus cluster is configured correctly:

  • pg_mode: Must be set to citus to tell Patroni to use Citus mode.
  • pg_primary_db: Must specify the name of the primary database with citus extension, named citus here.
  • pg_shard: Must specify a unified name as the cluster name prefix for all horizontal shard PG clusters, pg-citus here.
  • pg_group: Must specify a shard number, integers starting from zero. 0 represents the coordinator cluster, others are Worker clusters.
  • pg_cluster: Must be unique among physical PostgreSQL clusters. Using pg_shard plus a sequence number is the usual naming convention, but the current role does not require it to equal a string concatenation of pg_shard and pg_group.
  • pg_dbsu_password: Must be set to a non-empty plaintext password, otherwise Citus will not work properly.
  • pg_parameters: Recommended to set citus.node_conninfo to enforce SSL access and require node-to-node client certificate verification.

After configuration, you can deploy the Citus cluster using pgsql.yml just like a regular PostgreSQL cluster.

Manage Citus Cluster

After defining the Citus cluster, deploy it using the pgsql.yml playbook:

./pgsql.yml -l pg-citus    # Deploy Citus cluster pg-citus

Using any member’s DBSU (postgres) user, you can list the Citus cluster status with patronictl (alias: pg):

$ pg list
+ Citus cluster: pg-citus ----------+---------+-----------+----+-----------+--------------------+
| Group | Member      | Host        | Role    | State     | TL | Lag in MB | Tags               |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     0 | pg-citus0-1 | 10.10.10.10 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 20C.40G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     1 | pg-citus1-1 | 10.10.10.11 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-1 | 10.10.10.12 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-2 | 10.10.10.13 | Replica | streaming |  1 |         0 | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+

You can treat each horizontal shard cluster as an independent PGSQL cluster and manage them with the pg (patronictl) command. Note that when using the pg command to manage Citus clusters, you need to use the --group parameter to specify the cluster shard number:

pg list pg-citus --group 0   # Use --group 0 to specify cluster shard number

Citus has a system table called pg_dist_node that records Citus cluster node information. Patroni automatically maintains this table.

PGURL=postgres://postgres:DBUser.Postgres@10.10.10.10/citus

psql $PGURL -c 'SELECT * FROM pg_dist_node;'       # View node information
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole  | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+-----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.10 |     5432 | default  | t           | t        | primary   | default     | t              | f
      4 |       1 | 10.10.10.12 |     5432 | default  | t           | t        | primary   | default     | t              | t
      5 |       2 | 10.10.10.13 |     5432 | default  | t           | t        | primary   | default     | t              | t
      6 |       0 | 10.10.10.11 |     5432 | default  | t           | t        | secondary | default     | t              | f

You can also view user authentication information (superuser access only):

$ psql $PGURL -c 'SELECT * FROM pg_dist_authinfo;'   # View node auth info (superuser only)

Then you can use a regular business user (e.g., dbuser_citus with DDL privileges) to access the Citus cluster:

psql postgres://dbuser_citus:DBUser.Citus@10.10.10.10/citus -c 'SELECT * FROM pg_dist_node;'

Using Citus Cluster

When using Citus clusters, we strongly recommend reading the Citus official documentation to understand its architecture and core concepts.

The key is understanding the five types of tables in Citus and their characteristics and use cases:

  • Distributed Table
  • Reference Table
  • Local Table
  • Local Management Table
  • Schema Table

On the coordinator node, you can create distributed tables and reference tables and query them from any data node. Since 11.2, any Citus database node can act as a coordinator.

We can use pgbench to create some tables and distribute the main table (pgbench_accounts) across nodes, then use other small tables as reference tables:

PGURL=postgres://dbuser_citus:DBUser.Citus@10.10.10.10/citus
pgbench -i $PGURL

psql $PGURL <<-EOF
SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');
EOF

Run read/write tests:

pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:DBUser.Citus@10.10.10.10/citus      # Direct connect to coordinator port 5432
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:DBUser.Citus@10.10.10.10:6432/citus # Through connection pool, reduce client connection pressure
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:DBUser.Citus@10.10.10.13/citus      # Any primary node can act as coordinator
pgbench --select-only -nv -P1 -c10 -T500 postgres://dbuser_citus:DBUser.Citus@10.10.10.11/citus # Read-only queries

Production Deployment

For production use of Citus, you typically need to set up streaming replication physical replicas for the Coordinator and each Worker cluster.

The current conf/ha/citus.yml defines one pg-meta instance plus 12 Citus instances across 13 hosts: six two-node physical clusters with pg_group values 0-5. The 10-node fragment below is a separate production-topology example, not the current template.

pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # enable dbsu password access for citus
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # citus will be added by patroni automatically
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

We will cover a series of advanced Citus topics in subsequent tutorials:

  • Read/write separation
  • Failure handling
  • Consistent backup and recovery
  • Advanced monitoring and diagnostics
  • Connection pooling

8 - Monitoring

Overview of Pigsty’s monitoring system architecture and how to monitor existing PostgreSQL instances

This document introduces Pigsty’s monitoring system architecture, including metrics, logs, and target management. It also covers how to monitor existing PG clusters and remote RDS services.

Monitoring Overview

Pigsty uses a modern observability stack for PostgreSQL monitoring:

  • Grafana for metrics visualization and PostgreSQL datasource
  • VictoriaMetrics for collecting metrics from PostgreSQL / Pgbouncer / Patroni / HAProxy / Node
  • VictoriaLogs for logging PostgreSQL / Pgbouncer / Patroni / pgBackRest and host component logs
  • Battery-included Grafana dashboards showcasing all aspects of PostgreSQL; use the live demo and the dashboard links in this page because the set evolves with each release

Metrics

PostgreSQL monitoring metrics are fully defined by the pg_exporter configuration file: roles/pg_monitor/templates/pg_exporter.yml. They are further processed by VictoriaMetrics/vmalert-compatible recording and alerting rules: files/victoria/rules/pgsql.yml.

Pigsty uses three identity labels: cls, ins, ip, which are attached to all metrics and logs. Additionally, metrics from Pgbouncer, host nodes (NODE), and load balancers are also used by Pigsty, with the same labels used whenever possible for correlation analysis.

- { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
- { cls: pg-test, ins: pg-test-1, ip: 10.10.10.11 }
- { cls: pg-test, ins: pg-test-2, ip: 10.10.10.12 }
- { cls: pg-test, ins: pg-test-3, ip: 10.10.10.13 }

Logs

PostgreSQL-related logs are collected by Vector and sent to the VictoriaLogs log storage/query service on infra nodes.

Target Management

VictoriaMetrics monitoring targets are defined in static files under /infra/targets/pgsql/, with each instance having a corresponding file. Taking pg-meta-1 as an example:

# pg-meta-1 [primary] @ 10.10.10.10
- labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
  targets:
    - 10.10.10.10:9630    # <--- pg_exporter for PostgreSQL metrics
    - 10.10.10.10:9631    # <--- pgbouncer_exporter for PgBouncer metrics
    - 10.10.10.10:8008    # <--- Patroni metrics (when API SSL is not enabled)
    - 10.10.10.10:9854    # <--- pgbackrest_exporter for backup metrics

When the global flag patroni_ssl_enabled is set, Patroni targets are written separately to /infra/targets/patroni/<ins>.yml, because they use an HTTPS scrape endpoint. When monitoring RDS instances, monitoring targets are placed in /infra/targets/pgrds/ and managed by cluster.

When removing a cluster using bin/pgsql-rm or pgsql-rm.yml, the corresponding monitoring targets are removed. You can also use:

bin/pgmon-rm <cls|ins>    # Remove monitoring targets from all infra nodes

Remote RDS monitoring targets are placed in /infra/targets/pgrds/<cls>.yml, created by the pgsql-monitor.yml playbook or bin/pgmon-add script.

Monitoring Modes

Pigsty provides three monitoring modes to suit different monitoring needs.

Item \ Level L1 L2 L3
Name Basic Managed Standard
Abbr RDS MANAGED FULL
Scenario Connection string only, e.g., RDS Existing DB, nodes manageable Instances created by Pigsty
PGCAT Features ✅ Fully Available ✅ Fully Available ✅ Fully Available
PGSQL Features ✅ PG metrics only ✅ PG & node metrics only ✅ Full Features
Connection Pool Metrics ❌ Not Available ⚠️ Optional ✅ Pre-installed
Load Balancer Metrics ❌ Not Available ⚠️ Optional ✅ Pre-installed
PGLOG Features ❌ Not Available ⚠️ Optional ✅ Pre-installed
PG Exporter ⚠️ On infra nodes ✅ On DB nodes ✅ On DB nodes
Node Exporter ❌ Not deployed ✅ On DB nodes ✅ On DB nodes
Intrusiveness ✅ Non-intrusive ⚠️ Install Exporter ⚠️ Fully managed by Pigsty
Monitor Existing Instances ✅ Supported ✅ Supported ❌ For Pigsty-managed only
Monitoring Users & Views Manual setup Manual setup Auto-created by Pigsty
Deployment Playbook bin/pgmon-add <cls> Partial pgsql.yml/node.yml pgsql.yml
Required Permissions Connectable PGURL from infra SSH & sudo on DB nodes SSH & sudo on DB nodes
Feature Summary PGCAT + PGRDS Most features Full features

Databases fully managed by Pigsty are automatically monitored with the best support and typically require no configuration. For existing PostgreSQL clusters or RDS services, if the target DB nodes can be managed by Pigsty (ssh accessible, sudo available), you can consider managed deployment for a monitoring experience similar to native Pigsty. If you can only access the target database via PGURL (database connection string), such as remote RDS services, you can use basic mode to monitor the target database.

Monitor Existing Cluster

If the target DB nodes can be managed by Pigsty (ssh accessible and sudo available), you can use the pg_exporter task in the pgsql.yml playbook to deploy monitoring components (PG Exporter) on target nodes in the same way as standard deployments. You can also use the pgbouncer and pgbouncer_exporter tasks from that playbook to deploy connection pools and their monitoring on existing instance nodes. Additionally, you can use node_exporter, haproxy, and vector from node.yml to deploy host monitoring, load balancing, and log collection components, achieving an experience identical to native Pigsty database instances.

The definition method for existing clusters is exactly the same as for clusters managed by Pigsty. You selectively execute partial tasks from the pgsql.yml playbook instead of running the entire playbook.

./node.yml  -l <cls> -t node_repo,node_pkg           # Add YUM repos from INFRA nodes and install packages on host nodes
./node.yml  -l <cls> -t node_exporter,node_register  # Configure host monitoring and add to VictoriaMetrics
./node.yml  -l <cls> -t vector                       # Configure host log collection and send to VictoriaLogs
./pgsql.yml -l <cls> -t pg_exporter,pg_register      # Configure PostgreSQL monitoring and register with VictoriaMetrics/Grafana

Since the target database cluster already exists, you need to manually create monitoring users, schemas, and extensions on the target database cluster.

Monitor RDS

If you can only access the target database via PGURL (database connection string), you can configure according to the instructions here. In this mode, Pigsty deploys corresponding PG Exporters on INFRA nodes to scrape remote database metrics, as shown below:

------ infra ------
|                 |
| victoria-metrics|            v---- pg-foo-1 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20001) |            | 10.10.10.10:5432 |
|       ^         |            ^------------------^
|       ^         |                      ^
|       ^         |            v---- pg-foo-2 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20002) |            | 10.10.10.11:5433 |
-------------------            ^------------------^

In this mode, the monitoring system will not have metrics from hosts, connection pools, load balancers, or high availability components, but the database itself and real-time status information from the data catalog are still available. Pigsty provides two dedicated monitoring dashboards focused on PostgreSQL metrics: PGRDS Cluster and PGRDS Instance, while overview and database-level monitoring reuses existing dashboards. Since Pigsty cannot manage your RDS, users need to configure monitoring objects on the target database in advance.

Limitations when monitoring external Postgres instances
  • PgBouncer connection pool metrics are not available
  • Patroni high availability component metrics are not available
  • Host node monitoring metrics are not available, including node HAProxy and Keepalived metrics
  • Log collection and log-derived metrics are not available

Here we use the sandbox environment as an example: suppose the pg-meta cluster is an RDS instance pg-foo-1 to be monitored, and the pg-test cluster is an RDS cluster pg-bar to be monitored:

  1. Create monitoring schemas, users, and permissions on the target. Refer to Monitor Setup for details

  2. Declare the cluster in the configuration inventory. For example, if we want to monitor “remote” pg-meta & pg-test clusters:

    infra:            # Infra cluster for proxies, monitoring, alerts, etc.
  hosts: { 10.10.10.10: { infra_seq: 1 } }
  vars:           # Install pg_exporter on group 'infra' for remote postgres RDS
    pg_exporters: # List all remote instances here, assign a unique unused local port for k
      20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 , pg_databases: [{ name: meta }] } # Register meta database as Grafana datasource

      20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.11 , pg_port: 5432 } # Different connection string methods
      20003: { pg_cluster: pg-bar, pg_seq: 2, pg_host: 10.10.10.12 , pg_exporter_url: 'postgres://dbuser_monitor:DBUser.Monitor@10.10.10.12:5432/postgres?sslmode=disable'}
      20004: { pg_cluster: pg-bar, pg_seq: 3, pg_host: 10.10.10.13 , pg_monitor_username: dbuser_monitor, pg_monitor_password: DBUser.Monitor }

    Databases listed in the pg_databases field will be registered in Grafana as PostgreSQL datasources, providing data support for PGCAT monitoring dashboards. If you don’t want to use PGCAT and register databases in Grafana, simply set pg_databases to an empty array or leave it blank.

    pigsty-monitor.jpg

  3. Execute the add monitoring command: bin/pgmon-add <clsname>

    bin/pgmon-add pg-foo  # Bring pg-foo cluster into monitoring
bin/pgmon-add pg-bar  # Bring pg-bar cluster into monitoring

  4. To remove remote cluster monitoring targets, use bin/pgmon-rm <clsname>

    bin/pgmon-rm pg-foo  # Remove pg-foo from Pigsty monitoring
bin/pgmon-rm pg-bar  # Remove pg-bar from Pigsty monitoring

You can use more parameters to override default pg_exporter options. Here’s an example configuration for monitoring Aliyun RDS for PostgreSQL and PolarDB with Pigsty:

Example: Monitoring Aliyun RDS for PostgreSQL and PolarDB

For details, refer to: remote.yml

infra:            # Infra cluster for proxies, monitoring, alerts, etc.
  hosts: { 10.10.10.10: { infra_seq: 1 } }
  vars:
    pg_exporters:   # List all remote RDS PG instances to be monitored here

      20001:        # Assign a unique unused local port for local monitoring agent, this is a PolarDB primary
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pc-2ze379wb1d4irc18x.polardbpg.rds.aliyuncs.com # RDS host address
        pg_port: 1921                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username, overrides global config
        pg_monitor_password: DBUser_Monitor   # Monitoring password, overrides global config
        pg_databases: [{ name: test }]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20002:       # This is a PolarDB standby
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pe-2ze7tg620e317ufj4.polarpgmxs.rds.aliyuncs.com # RDS host address
        pg_port: 1521                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test,postgres'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: test } ]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20004: # This is a basic single-node RDS for PostgreSQL instance
        pg_cluster: pg-rds                    # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2zern3d323fe9ewk.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: rds } ]       # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20005: # This is a high-availability RDS for PostgreSQL cluster primary
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2ze3d35d27bq08wu.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

      20006: # This is a high-availability RDS for PostgreSQL cluster read-only instance (standby)
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgr-2zexqxalk7d37edt.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

Monitor Setup

When you want to monitor existing instances, whether RDS or self-built PostgreSQL instances, you need to configure the target database so that Pigsty can access them.

To monitor an external existing PostgreSQL instance, you need a connection string that can access that instance/cluster. Any accessible connection string (business user, superuser) can be used, but we recommend using a dedicated monitoring user to avoid permission leaks.

  • Monitor User: The default username is dbuser_monitor, which should belong to the pg_monitor role group or have access to relevant views
  • Monitor Authentication: Default password authentication is used; ensure HBA policies allow the monitoring user to access databases from the admin node or DB node locally
  • Monitor Schema: Fixed schema name monitor is used for installing additional monitoring views and extension plugins; optional but recommended
  • Monitor Extension: Strongly recommended to enable the built-in monitoring extension pg_stat_statements
  • Monitor Views: Monitoring views are optional but can provide additional metric support

Monitor User

Using the default monitoring user dbuser_monitor as an example, create the following user on the target database cluster.

CREATE USER dbuser_monitor;                                       -- Create monitoring user
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';          -- Comment on monitoring user
GRANT pg_monitor TO dbuser_monitor;                               -- Grant pg_monitor privilege to monitoring user, otherwise some metrics cannot be collected

ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';              -- Modify monitoring user password as needed (strongly recommended! but keep consistent with Pigsty config)
ALTER USER dbuser_monitor SET log_min_duration_statement = 1000;  -- Recommended to avoid logs filling up with monitoring slow queries
ALTER USER dbuser_monitor SET search_path = monitor,public;       -- Recommended to ensure pg_stat_statements extension works properly

Please note that the monitoring user and password created here should be consistent with pg_monitor_username and pg_monitor_password.

Monitor Authentication

Configure the database pg_hba.conf file, adding the following rules to allow the monitoring user to access all databases from localhost and the admin machine using password authentication.

# allow local role monitor with password
local   all  dbuser_monitor                    md5
host    all  dbuser_monitor  127.0.0.1/32      md5
host    all  dbuser_monitor  <admin_machine_IP>/32 md5

If your RDS doesn’t support defining HBA, simply whitelist the internal IP address of the machine running Pigsty.

Monitor Schema

The monitoring schema is optional; even without it, the main functionality of Pigsty’s monitoring system can work properly, but we strongly recommend creating this schema.

CREATE SCHEMA IF NOT EXISTS monitor;               -- Create dedicated monitoring schema
GRANT USAGE ON SCHEMA monitor TO dbuser_monitor;   -- Allow monitoring user to use it

Monitor Extension

The monitoring extension is optional, but we strongly recommend enabling the pg_stat_statements extension, which provides important data about query performance.

Note: This extension must be listed in the database parameter shared_preload_libraries to take effect, and modifying that parameter requires a database restart.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Please note that you should install this extension in the default admin database postgres. Sometimes RDS doesn’t allow you to create a monitoring schema in the postgres database. In such cases, you can install the pg_stat_statements plugin in the default public schema, as long as you ensure the monitoring user’s search_path is configured as above so it can find the pg_stat_statements view.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements";
ALTER USER dbuser_monitor SET search_path = monitor,public; -- Recommended to ensure pg_stat_statements extension works properly

Monitor Views

Monitoring views provide several commonly used pre-processed results and encapsulate permissions for monitoring metrics that require high privileges (such as shared memory allocation), making them convenient for querying and use. Strongly recommended to create in all databases requiring monitoring.

Monitoring schema and monitoring view definitions

The SQL below is provided to explain the monitored objects. The complete definitions rendered by current Pigsty are authoritative in roles/pgsql/templates/pg-init-template.sql, which also includes additional hardening for secure search paths and privilege boundaries.

----------------------------------------------------------------------
-- Table bloat estimate : monitor.pg_table_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, relname , tblid , bs * tblpages AS size,
       CASE WHEN tblpages - est_tblpages_ff > 0 THEN (tblpages - est_tblpages_ff)/tblpages::FLOAT ELSE 0 END AS ratio
FROM (
         SELECT ceil( reltuples / ( (bs-page_hdr)*fillfactor/(tpl_size*100) ) ) + ceil( toasttuples / 4 ) AS est_tblpages_ff,
                tblpages, fillfactor, bs, tblid, nspname, relname, is_na
         FROM (
                  SELECT
                      ( 4 + tpl_hdr_size + tpl_data_size + (2 * ma)
                          - CASE WHEN tpl_hdr_size % ma = 0 THEN ma ELSE tpl_hdr_size % ma END
                          - CASE WHEN ceil(tpl_data_size)::INT % ma = 0 THEN ma ELSE ceil(tpl_data_size)::INT % ma END
                          ) AS tpl_size, (heappages + toastpages) AS tblpages, heappages,
                      toastpages, reltuples, toasttuples, bs, page_hdr, tblid, nspname, relname, fillfactor, is_na
                  FROM (
                           SELECT
                               tbl.oid AS tblid, ns.nspname , tbl.relname, tbl.reltuples,
                               tbl.relpages AS heappages, coalesce(toast.relpages, 0) AS toastpages,
                               coalesce(toast.reltuples, 0) AS toasttuples,
                               coalesce(substring(array_to_string(tbl.reloptions, ' ') FROM 'fillfactor=([0-9]+)')::smallint, 100) AS fillfactor,
                               current_setting('block_size')::numeric AS bs,
                               CASE WHEN version()~'mingw32' OR version()~'64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END AS ma,
                               24 AS page_hdr,
                               23 + CASE WHEN MAX(coalesce(s.null_frac,0)) > 0 THEN ( 7 + count(s.attname) ) / 8 ELSE 0::int END
                                   + CASE WHEN bool_or(att.attname = 'oid' and att.attnum < 0) THEN 4 ELSE 0 END AS tpl_hdr_size,
                               sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 0) ) AS tpl_data_size,
                               bool_or(att.atttypid = 'pg_catalog.name'::regtype)
                                   OR sum(CASE WHEN att.attnum > 0 THEN 1 ELSE 0 END) <> count(s.attname) AS is_na
                           FROM pg_attribute AS att
                                    JOIN pg_class AS tbl ON att.attrelid = tbl.oid
                                    JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
                                    LEFT JOIN pg_stats AS s ON s.schemaname=ns.nspname AND s.tablename = tbl.relname AND s.inherited=false AND s.attname=att.attname
                                    LEFT JOIN pg_class AS toast ON tbl.reltoastrelid = toast.oid
                           WHERE NOT att.attisdropped AND tbl.relkind = 'r' AND nspname NOT IN ('pg_catalog','information_schema')
                           GROUP BY 1,2,3,4,5,6,7,8,9,10
                       ) AS s
              ) AS s2
     ) AS s3
WHERE NOT is_na;
COMMENT ON VIEW monitor.pg_table_bloat IS 'postgres table bloat estimate';

GRANT SELECT ON monitor.pg_table_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Index bloat estimate : monitor.pg_index_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, idxname AS relname, tblid, idxid, relpages::BIGINT * bs AS size,
       COALESCE((relpages - ( reltuples * (6 + ma - (CASE WHEN index_tuple_hdr % ma = 0 THEN ma ELSE index_tuple_hdr % ma END)
                                               + nulldatawidth + ma - (CASE WHEN nulldatawidth % ma = 0 THEN ma ELSE nulldatawidth % ma END))
                                  / (bs - pagehdr)::FLOAT  + 1 )), 0) / relpages::FLOAT AS ratio
FROM (
         SELECT nspname,idxname,indrelid AS tblid,indexrelid AS idxid,
                reltuples,relpages,
                current_setting('block_size')::INTEGER                                                               AS bs,
                (CASE WHEN version() ~ 'mingw32' OR version() ~ '64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END)  AS ma,
                24                                                                                                   AS pagehdr,
                (CASE WHEN max(COALESCE(pg_stats.null_frac, 0)) = 0 THEN 2 ELSE 6 END)                               AS index_tuple_hdr,
                sum((1.0 - COALESCE(pg_stats.null_frac, 0.0)) *
                    COALESCE(pg_stats.avg_width, 1024))::INTEGER                                                     AS nulldatawidth
         FROM pg_attribute
                  JOIN (
             SELECT pg_namespace.nspname,
                    ic.relname                                                   AS idxname,
                    ic.reltuples,
                    ic.relpages,
                    pg_index.indrelid,
                    pg_index.indexrelid,
                    tc.relname                                                   AS tablename,
                    regexp_split_to_table(pg_index.indkey::TEXT, ' ') :: INTEGER AS attnum,
                    pg_index.indexrelid                                          AS index_oid
             FROM pg_index
                      JOIN pg_class ic ON pg_index.indexrelid = ic.oid
                      JOIN pg_class tc ON pg_index.indrelid = tc.oid
                      JOIN pg_namespace ON pg_namespace.oid = ic.relnamespace
                      JOIN pg_am ON ic.relam = pg_am.oid
             WHERE pg_am.amname = 'btree' AND ic.relpages > 0 AND nspname NOT IN ('pg_catalog', 'information_schema')
         ) ind_atts ON pg_attribute.attrelid = ind_atts.indexrelid AND pg_attribute.attnum = ind_atts.attnum
                  JOIN pg_stats ON pg_stats.schemaname = ind_atts.nspname
             AND ((pg_stats.tablename = ind_atts.tablename AND pg_stats.attname = pg_get_indexdef(pg_attribute.attrelid, pg_attribute.attnum, TRUE))
                 OR (pg_stats.tablename = ind_atts.idxname AND pg_stats.attname = pg_attribute.attname))
         WHERE pg_attribute.attnum > 0
         GROUP BY 1, 2, 3, 4, 5, 6
     ) est;
COMMENT ON VIEW monitor.pg_index_bloat IS 'postgres index bloat estimate (btree-only)';

GRANT SELECT ON monitor.pg_index_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Relation Bloat : monitor.pg_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_bloat AS
SELECT coalesce(ib.datname, tb.datname)                                                   AS datname,
       coalesce(ib.nspname, tb.nspname)                                                   AS nspname,
       coalesce(ib.tblid, tb.tblid)                                                       AS tblid,
       coalesce(tb.nspname || '.' || tb.relname, ib.nspname || '.' || ib.tblid::RegClass) AS tblname,
       tb.size                                                                            AS tbl_size,
       CASE WHEN tb.ratio < 0 THEN 0 ELSE round(tb.ratio::NUMERIC, 6) END                 AS tbl_ratio,
       (tb.size * (CASE WHEN tb.ratio < 0 THEN 0 ELSE tb.ratio::NUMERIC END)) ::BIGINT    AS tbl_wasted,
       ib.idxid,
       ib.nspname || '.' || ib.relname                                                    AS idxname,
       ib.size                                                                            AS idx_size,
       CASE WHEN ib.ratio < 0 THEN 0 ELSE round(ib.ratio::NUMERIC, 5) END                 AS idx_ratio,
       (ib.size * (CASE WHEN ib.ratio < 0 THEN 0 ELSE ib.ratio::NUMERIC END)) ::BIGINT    AS idx_wasted
FROM monitor.pg_index_bloat ib
         FULL OUTER JOIN monitor.pg_table_bloat tb ON ib.tblid = tb.tblid;

COMMENT ON VIEW monitor.pg_bloat IS 'postgres relation bloat detail';
GRANT SELECT ON monitor.pg_bloat TO pg_monitor;

----------------------------------------------------------------------
-- monitor.pg_index_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat_human AS
SELECT idxname                            AS name,
       tblname,
       idx_wasted                         AS wasted,
       pg_size_pretty(idx_size)           AS idx_size,
       round(100 * idx_ratio::NUMERIC, 2) AS idx_ratio,
       pg_size_pretty(idx_wasted)         AS idx_wasted,
       pg_size_pretty(tbl_size)           AS tbl_size,
       round(100 * tbl_ratio::NUMERIC, 2) AS tbl_ratio,
       pg_size_pretty(tbl_wasted)         AS tbl_wasted
FROM monitor.pg_bloat
WHERE idxname IS NOT NULL;
COMMENT ON VIEW monitor.pg_index_bloat_human IS 'postgres index bloat info in human-readable format';
GRANT SELECT ON monitor.pg_index_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- monitor.pg_table_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat_human AS
SELECT tblname                                          AS name,
       idx_wasted + tbl_wasted                          AS wasted,
       pg_size_pretty(idx_wasted + tbl_wasted)          AS all_wasted,
       pg_size_pretty(tbl_wasted)                       AS tbl_wasted,
       pg_size_pretty(tbl_size)                         AS tbl_size,
       tbl_ratio,
       pg_size_pretty(idx_wasted)                       AS idx_wasted,
       pg_size_pretty(idx_size)                         AS idx_size,
       round(idx_wasted::NUMERIC * 100.0 / idx_size, 2) AS idx_ratio
FROM (SELECT datname,
             nspname,
             tblname,
             coalesce(max(tbl_wasted), 0)                         AS tbl_wasted,
             coalesce(max(tbl_size), 1)                           AS tbl_size,
             round(100 * coalesce(max(tbl_ratio), 0)::NUMERIC, 2) AS tbl_ratio,
             coalesce(sum(idx_wasted), 0)                         AS idx_wasted,
             coalesce(sum(idx_size), 1)                           AS idx_size
      FROM monitor.pg_bloat
      WHERE tblname IS NOT NULL
      GROUP BY 1, 2, 3
     ) d;
COMMENT ON VIEW monitor.pg_table_bloat_human IS 'postgres table bloat info in human-readable format';
GRANT SELECT ON monitor.pg_table_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- Activity Overview: monitor.pg_session
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_session CASCADE;
CREATE OR REPLACE VIEW monitor.pg_session AS
SELECT coalesce(datname, 'all') AS datname, numbackends, active, idle, ixact, max_duration, max_tx_duration, max_conn_duration
FROM (
         SELECT datname,
                count(*)                                         AS numbackends,
                count(*) FILTER ( WHERE state = 'active' )       AS active,
                count(*) FILTER ( WHERE state = 'idle' )         AS idle,
                count(*) FILTER ( WHERE state = 'idle in transaction'
                    OR state = 'idle in transaction (aborted)' ) AS ixact,
                max(extract(epoch from now() - state_change))
                FILTER ( WHERE state = 'active' )                AS max_duration,
                max(extract(epoch from now() - xact_start))      AS max_tx_duration,
                max(extract(epoch from now() - backend_start))   AS max_conn_duration
         FROM pg_stat_activity
         WHERE backend_type = 'client backend'
           AND pid <> pg_backend_pid()
         GROUP BY ROLLUP (1)
         ORDER BY 1 NULLS FIRST
     ) t;
COMMENT ON VIEW monitor.pg_session IS 'postgres activity group by session';
GRANT SELECT ON monitor.pg_session TO pg_monitor;


----------------------------------------------------------------------
-- Sequential Scan: monitor.pg_seq_scan
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_seq_scan CASCADE;
CREATE OR REPLACE VIEW monitor.pg_seq_scan AS
SELECT schemaname                                                        AS nspname,
       relname,
       seq_scan,
       seq_tup_read,
       seq_tup_read / seq_scan                                           AS seq_tup_avg,
       idx_scan,
       n_live_tup + n_dead_tup                                           AS tuples,
       round(n_live_tup * 100.0::NUMERIC / (n_live_tup + n_dead_tup), 2) AS live_ratio
FROM pg_stat_user_tables
WHERE seq_scan > 0
  and (n_live_tup + n_dead_tup) > 0
ORDER BY seq_scan DESC;
COMMENT ON VIEW monitor.pg_seq_scan IS 'table that have seq scan';
GRANT SELECT ON monitor.pg_seq_scan TO pg_monitor;
Function for viewing shared memory allocation (PG13 and above)
DROP FUNCTION IF EXISTS monitor.pg_shmem() CASCADE;
CREATE OR REPLACE FUNCTION monitor.pg_shmem() RETURNS SETOF
    pg_shmem_allocations SET search_path = '' AS $$ SELECT * FROM pg_shmem_allocations;$$ LANGUAGE SQL SECURITY DEFINER;
COMMENT ON FUNCTION monitor.pg_shmem() IS 'security wrapper for system view pg_shmem';
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM PUBLIC;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_readonly;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_offline;
GRANT EXECUTE ON FUNCTION monitor.pg_shmem() TO pg_monitor;

9 - Dashboard

Pigsty provides numerous out-of-the-box Grafana monitoring dashboards for PostgreSQL

Pigsty provides numerous out-of-the-box Grafana monitoring dashboards for PostgreSQL: Demo & Gallery.

The current source provides 31 PostgreSQL-related dashboards: 29 PostgreSQL / PGCAT dashboards under files/grafana/pgsql, plus two PGLOG dashboards under files/grafana/app. They are organized by hierarchy into Overview, Cluster, Instance, and Database categories, and by data source into PGSQL, PGCAT, and PGLOG.

pigsty-dashboard.jpg

Overview

Overview Cluster Instance Database
PGSQL Overview PGSQL Cluster PGSQL Instance PGSQL Database
PGSQL Alert PGRDS Cluster PGRDS Instance PGCAT Database
PGSQL Shard PGSQL Activity PGCAT Instance PGSQL Tables
PGSQL Replication PGSQL Persist PGSQL Table
PGSQL Service PGSQL Proxy PGCAT Table
PGSQL Databases PGSQL Pgbouncer PGSQL Query
PGSQL Patroni PGSQL Session PGCAT Query
PGSQL PITR PGSQL Xacts PGCAT Locks
PGSQL Exporter PGCAT Schema

Overview

  • pgsql-overview: Main dashboard for the PGSQL module
  • pgsql-alert: Global key metrics and alert events for PGSQL
  • pgsql-shard: Overview of horizontally sharded PGSQL clusters (e.g., Citus/GPSQL)

Cluster

  • pgsql-cluster: Main dashboard for a PGSQL cluster
  • pgrds-cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • pgsql-activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • pgsql-replication: Replication, slots, and pub/sub for PGSQL cluster
  • pgsql-service: Service, proxy, routing, and load balancing for PGSQL cluster
  • pgsql-databases: Database CRUD, slow queries, and table statistics across all instances
  • pgsql-patroni: HA status and Patroni component status for cluster
  • pgsql-pitr: PITR context for point-in-time recovery assistance

Instance

  • pgsql-instance: Main dashboard for a single PGSQL instance
  • pgrds-instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • pgcat-instance: Instance info retrieved directly from database catalog
  • pgsql-proxy: Detailed metrics for a single HAProxy load balancer
  • pgsql-pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • pgsql-persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • pgsql-session: Session and active/idle time metrics for a single instance
  • pgsql-xacts: Transaction, lock, TPS/QPS related metrics
  • pgsql-exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

Database

  • pgsql-database: Main dashboard for a single PGSQL database
  • pgcat-database: Database info retrieved directly from database catalog
  • pgsql-tables: Table/index access metrics within a single database
  • pgsql-table: Detailed info for a single table (QPS/RT/index/sequence…)
  • pgcat-table: Detailed table info from database catalog (stats/bloat…)
  • pgsql-query: Detailed info for a query type (QPS/RT)
  • pgcat-query: Query details from database catalog (SQL/stats)
  • pgcat-schema: Schema info from database catalog (tables/indexes/sequences…)
  • pgcat-locks: Activity and lock wait info from database catalog

Overview

PGSQL Overview: Main dashboard for the PGSQL module

PGSQL Overview

pgsql-overview.jpg

PGSQL Alert: Global core metrics overview and alert events

PGSQL Alert

pgsql-alert.jpg

PGSQL Shard: Cross-shard metric comparison for horizontally sharded PGSQL clusters (e.g., CITUS/GPSQL)

PGSQL Shard

pgsql-shard.jpg

Cluster

PGSQL Cluster: Main dashboard for a PGSQL cluster

PGSQL Cluster

pgsql-cluster.jpg

PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics

PGRDS Cluster

pgrds-cluster.jpg

PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster

PGSQL Service

pgsql-service.jpg

PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances

PGSQL Databases

pgsql-databases.jpg

PGSQL Patroni: HA status and Patroni component status for cluster

PGSQL Patroni

pgsql-patroni.jpg

PGSQL PITR: PITR context for point-in-time recovery assistance

PGSQL PITR

pgsql-patroni.jpg

Instance

PGSQL Instance: Main dashboard for a single PGSQL instance

PGSQL Instance

pgsql-instance.jpg

PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics

PGRDS Instance

pgrds-instance.jpg

PGSQL Proxy: Detailed metrics for a single HAProxy load balancer

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO

PGSQL Persist

pgsql-persist.jpg

PGSQL Xacts: Transaction, lock, TPS/QPS related metrics

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Session: Session and active/idle time metrics for a single instance

PGSQL Session

pgsql-session.jpg

PGSQL Exporter: Self-monitoring metrics for Postgres/Pgbouncer exporters

PGSQL Exporter

pgsql-exporter.jpg

Database

PGSQL Database: Main dashboard for a single PGSQL database

PGSQL Database

pgsql-database.jpg

PGSQL Tables: Table/index access metrics within a single database

PGSQL Tables

pgsql-tables.jpg

PGSQL Table: Detailed info for a single table (QPS/RT/index/sequence…)

PGSQL Table

pgsql-table.jpg

PGSQL Query: Detailed info for a query type (QPS/RT)

PGSQL Query

pgsql-query.jpg

PGCAT

PGCAT Instance: Instance info retrieved directly from database catalog

PGCAT Instance

pgcat-instance.jpg

PGCAT Database: Database info retrieved directly from database catalog

PGCAT Database

pgcat-database.jpg

PGCAT Schema: Schema info from database catalog (tables/indexes/sequences…)

PGCAT Schema

pgcat-schema.jpg

PGCAT Table: Detailed table info from database catalog (stats/bloat…)

PGCAT Table

pgcat-table.jpg

PGCAT Query: Query details from database catalog (SQL/stats)

PGCAT Query

pgcat-query.jpg

PGCAT Locks: Activity and lock wait info from database catalog

PGCAT Locks

pgcat-locks.jpg

PGLOG

PGLOG Overview: Overview of CSV log samples in Pigsty CMDB

PGLOG Overview

pglog-overview.jpg

PGLOG Session: Log details for a single session in CSV log samples

PGLOG Session

pglog-session.jpg

Gallery

See pigsty/wiki/gallery for details.

PGSQL Overview

pgsql-overview.jpg

PGSQL Shard

pgsql-shard.jpg

PGSQL Cluster

pgsql-cluster.jpg

PGSQL Service

pgsql-service.jpg

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases

pgsql-databases.jpg

PGSQL Instance

pgsql-instance.jpg

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Session

pgsql-session.jpg

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Persist

pgsql-persist.jpg

PGSQL Database

pgsql-database.jpg

PGSQL Tables

pgsql-tables.jpg

PGSQL Table

pgsql-table.jpg

PGSQL Query

pgsql-query.jpg

PGCAT Instance

pgcat-instance.jpg

PGCAT Database

pgcat-database.jpg

PGCAT Schema

pgcat-schema.jpg

PGCAT Table

pgcat-table.jpg

PGCAT Lock

pgcat-locks.jpg

PGCAT Query

pgcat-query.jpg

PGLOG Overview

pglog-overview.jpg

PGLOG Session

pglog-session.jpg

9.1 - Overview

PostgreSQL module global overview monitoring dashboards

PostgreSQL module global overview monitoring dashboards, including:

9.1.1 - PGSQL Overview

Main dashboard for the PGSQL module

Main dashboard for the PGSQL module: Demo

PGSQL Overview is the main dashboard for the PostgreSQL module, providing a global overview of the entire PGSQL module.

pgsql-overview

9.1.2 - PGSQL Alert

Global key metrics and alert events for PGSQL

Global key metrics and alert events for PGSQL: Demo

PGSQL Alert provides a global overview of core metrics and alert events for PostgreSQL clusters.

pgsql-alert

9.1.3 - PGSQL Shard

Overview of horizontally sharded PGSQL clusters

Overview of horizontally sharded PGSQL clusters: Demo

PGSQL Shard provides cross-shard metric comparison for horizontally sharded PGSQL clusters such as CITUS or GPSQL.

pgsql-shard

9.2 - Cluster

PostgreSQL cluster-level monitoring dashboards

PostgreSQL cluster-level monitoring dashboards, including:

  • PGSQL Cluster: Main dashboard for a PGSQL cluster
  • PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster
  • PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster
  • PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances
  • PGSQL Patroni: HA status and Patroni component status for cluster
  • PGSQL PITR: PITR context for point-in-time recovery assistance

9.2.1 - PGSQL Cluster

Main dashboard for a PGSQL cluster

Main dashboard for a PGSQL cluster: Demo

PGSQL Cluster is the main dashboard for a single PostgreSQL cluster, providing cluster-level core metrics overview.

pgsql-cluster

9.2.2 - PGRDS Cluster

RDS version of PGSQL Cluster focusing on PostgreSQL-native metrics

RDS version of PGSQL Cluster: Demo

PGRDS Cluster is the RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics without host-level metrics.

pgrds-cluster

9.2.3 - PGSQL Activity

Session/load/QPS/TPS/locks for PGSQL cluster

Session/load/QPS/TPS/locks for PGSQL cluster: Demo

PGSQL Activity focuses on session activity, load, QPS, TPS, and lock status for a PostgreSQL cluster.

pgsql-activity

9.2.4 - PGSQL Replication

Replication, slots, and pub/sub for PGSQL cluster

Replication, slots, and pub/sub for PGSQL cluster: Demo

PGSQL Replication focuses on replication status, replication slots, and logical replication (pub/sub) for a PostgreSQL cluster.

pgsql-replication

9.2.5 - PGSQL Service

Service, proxy, routing, and load balancing for PGSQL cluster

Service, proxy, routing, and load balancing for PGSQL cluster: Demo

PGSQL Service focuses on service endpoints, proxy routing, and load balancing status for a PostgreSQL cluster.

pgsql-service

9.2.6 - PGSQL Databases

Database CRUD, slow queries, and table statistics across all instances

Database CRUD, slow queries, and table statistics: Demo

PGSQL Databases focuses on database-level CRUD operations, slow queries, and table statistics across all instances in a cluster.

pgsql-databases

9.2.7 - PGSQL Patroni

HA status and Patroni component status for cluster

HA status and Patroni component status: Demo

PGSQL Patroni focuses on high-availability status and Patroni component health for a PostgreSQL cluster.

pgsql-patroni

9.2.8 - PGSQL PITR

PITR context for point-in-time recovery assistance

PITR context for point-in-time recovery: Demo

PGSQL PITR provides context information for point-in-time recovery operations, showing backup status and WAL timeline.

pgsql-pitr

9.3 - Instance

PostgreSQL instance-level monitoring dashboards

PostgreSQL instance-level monitoring dashboards, including:

  • PGSQL Instance: Main dashboard for a single PGSQL instance
  • PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • PGCAT Instance: Instance info retrieved directly from database catalog
  • PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • PGSQL Proxy: Detailed metrics for a single HAProxy load balancer
  • PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • PGSQL Session: Session and active/idle time metrics for a single instance
  • PGSQL Xacts: Transaction, lock, TPS/QPS related metrics
  • PGSQL Exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

9.3.1 - PGSQL Instance

Main dashboard for a single PGSQL instance

Main dashboard for a single PGSQL instance: Demo

PGSQL Instance is the main dashboard for a single PostgreSQL instance, providing comprehensive instance-level metrics.

pgsql-instance

9.3.2 - PGRDS Instance

RDS version of PGSQL Instance focusing on PostgreSQL-native metrics

RDS version of PGSQL Instance: Demo

PGRDS Instance is the RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics without host-level metrics.

pgrds-instance

9.3.3 - PGCAT Instance

Instance info retrieved directly from database catalog

Instance info from database catalog: Demo

PGCAT Instance shows instance-level information retrieved directly from PostgreSQL system catalog.

pgcat-instance

9.3.4 - PGSQL Persist

Persistence metrics - WAL, XID, checkpoint, archive, IO

Persistence metrics for PGSQL instance: Demo

PGSQL Persist focuses on persistence-related metrics: WAL generation, XID consumption, checkpoints, archiving, and I/O patterns.

pgsql-persist

9.3.5 - PGSQL Proxy

Detailed metrics for a single HAProxy load balancer

Detailed metrics for HAProxy: Demo

PGSQL Proxy shows detailed metrics for a single HAProxy load balancer instance serving PostgreSQL traffic.

pgsql-proxy

9.3.6 - PGSQL Pgbouncer

Metrics overview for a single Pgbouncer connection pooler

Metrics overview for Pgbouncer: Demo

PGSQL Pgbouncer shows connection pooling metrics for a single Pgbouncer instance.

pgsql-pgbouncer

9.3.7 - PGSQL Session

Session and active/idle time metrics for a single instance

Session and active/idle time metrics: Demo

PGSQL Session focuses on session statistics and active/idle time distribution for a single PostgreSQL instance.

pgsql-session

9.3.8 - PGSQL Xacts

Transaction, lock, TPS/QPS related metrics

Transaction, lock, TPS/QPS metrics: Demo

PGSQL Xacts focuses on transaction processing, lock activity, and TPS/QPS metrics for a single PostgreSQL instance.

pgsql-xacts

9.3.9 - PGSQL Exporter

Self-monitoring metrics for Postgres and Pgbouncer exporters

Self-monitoring metrics for exporters: Demo

PGSQL Exporter shows self-monitoring metrics for the Postgres exporter and Pgbouncer exporter components.

pgsql-exporter

9.4 - Database

PostgreSQL database-level monitoring dashboards

PostgreSQL database-level monitoring dashboards, including:

  • PGSQL Database: Main dashboard for a single PGSQL database
  • PGCAT Database: Database info retrieved directly from database catalog
  • PGSQL Tables: Table/index access metrics within a single database
  • PGSQL Table: Detailed info for a single table (QPS/RT/index/sequence…)
  • PGCAT Table: Detailed table info from database catalog
  • PGSQL Query: Detailed info for a query type (QPS/RT)
  • PGCAT Query: Query details from database catalog
  • PGCAT Locks: Activity and lock wait info from database catalog
  • PGCAT Schema: Schema info from database catalog

9.4.1 - PGSQL Database

Main dashboard for a single PGSQL database

Main dashboard for a single PGSQL database: Demo

PGSQL Database is the main dashboard for a single PostgreSQL database, providing comprehensive database-level metrics.

pgsql-database

9.4.2 - PGCAT Database

Database info retrieved directly from database catalog

Database info from database catalog: Demo

PGCAT Database shows database-level information retrieved directly from PostgreSQL system catalog.

pgcat-database

9.4.3 - PGSQL Tables

Table/index access metrics within a single database

Table/index access metrics: Demo

PGSQL Tables shows table and index access metrics for all objects within a single PostgreSQL database.

pgsql-tables

9.4.4 - PGSQL Table

Detailed info for a single table (QPS/RT/index/sequence)

Detailed info for a single table: Demo

PGSQL Table shows detailed metrics for a single table including QPS, response time, index usage, and sequence info.

pgsql-table

9.4.5 - PGCAT Table

Detailed table info from database catalog

Detailed table info from catalog: Demo

PGCAT Table shows detailed table information from database catalog including statistics and bloat analysis.

pgcat-table

9.4.6 - PGSQL Query

Detailed info for a query type (QPS/RT)

Detailed info for a query type: Demo

PGSQL Query shows detailed metrics for a specific query type including QPS and response time distribution.

pgsql-query

9.4.7 - PGCAT Query

Query details from database catalog

Query details from database catalog: Demo

PGCAT Query shows query details from database catalog including SQL text and execution statistics.

pgcat-query

9.4.8 - PGCAT Locks

Activity and lock wait info from database catalog

Activity and lock wait info: Demo

PGCAT Locks shows active sessions and lock wait information from database catalog.

pgcat-locks

9.4.9 - PGCAT Schema

Schema info from database catalog

Schema info from database catalog: Demo

PGCAT Schema shows schema-level information from database catalog including tables, indexes, and sequences.

pgcat-schema

10 - Metrics

Complete monitoring metrics reference for the Pigsty PGSQL module

The PGSQL module provides 638 available monitoring metrics.

Metric Name Type Labels Description
ALERTS Unknown category, job, level, ins, severity, ip, alertname, alertstate, instance, cls N/A
ALERTS_FOR_STATE Unknown category, job, level, ins, severity, ip, alertname, instance, cls N/A
cls:pressure1 Unknown job, cls N/A
cls:pressure15 Unknown job, cls N/A
cls:pressure5 Unknown job, cls N/A
go_gc_duration_seconds summary job, ins, ip, instance, quantile, cls A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown job, ins, ip, instance, cls N/A
go_gc_duration_seconds_sum Unknown job, ins, ip, instance, cls N/A
go_goroutines gauge job, ins, ip, instance, cls Number of goroutines that currently exist.
go_info gauge version, job, ins, ip, instance, cls Information about the Go environment.
go_memstats_alloc_bytes gauge job, ins, ip, instance, cls Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total counter job, ins, ip, instance, cls Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter job, ins, ip, instance, cls Total number of frees.
go_memstats_gc_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge job, ins, ip, instance, cls Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge job, ins, ip, instance, cls Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge job, ins, ip, instance, cls Number of heap bytes that are in use.
go_memstats_heap_objects gauge job, ins, ip, instance, cls Number of allocated objects.
go_memstats_heap_released_bytes gauge job, ins, ip, instance, cls Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge job, ins, ip, instance, cls Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge job, ins, ip, instance, cls Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter job, ins, ip, instance, cls Total number of pointer lookups.
go_memstats_mallocs_total counter job, ins, ip, instance, cls Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge job, ins, ip, instance, cls Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge job, ins, ip, instance, cls Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge job, ins, ip, instance, cls Number of bytes obtained from system.
go_threads gauge job, ins, ip, instance, cls Number of OS threads created.
ins:pressure1 Unknown job, ins, ip, cls N/A
ins:pressure15 Unknown job, ins, ip, cls N/A
ins:pressure5 Unknown job, ins, ip, cls N/A
patroni_cluster_unlocked gauge job, ins, ip, instance, cls, scope Value is 1 if the cluster is unlocked, 0 if locked.
patroni_dcs_last_seen gauge job, ins, ip, instance, cls, scope Epoch timestamp when DCS was last contacted successfully by Patroni.
patroni_failsafe_mode_is_active gauge job, ins, ip, instance, cls, scope Value is 1 if failsafe mode is active, 0 if inactive.
patroni_is_paused gauge job, ins, ip, instance, cls, scope Value is 1 if auto failover is disabled, 0 otherwise.
patroni_master gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the leader, 0 otherwise.
patroni_pending_restart gauge job, ins, ip, instance, cls, scope Value is 1 if the node needs a restart, 0 otherwise.
patroni_postgres_in_archive_recovery gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is replicating from archive, 0 otherwise.
patroni_postgres_running gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is running, 0 otherwise.
patroni_postgres_server_version gauge job, ins, ip, instance, cls, scope Version of Postgres (if running), 0 otherwise.
patroni_postgres_streaming gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is streaming, 0 otherwise.
patroni_postgres_timeline counter job, ins, ip, instance, cls, scope Postgres timeline of this node (if running), 0 otherwise.
patroni_postmaster_start_time gauge job, ins, ip, instance, cls, scope Epoch seconds since Postgres started.
patroni_primary gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the leader, 0 otherwise.
patroni_replica gauge job, ins, ip, instance, cls, scope Value is 1 if this node is a replica, 0 otherwise.
patroni_standby_leader gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the standby_leader, 0 otherwise.
patroni_sync_standby gauge job, ins, ip, instance, cls, scope Value is 1 if this node is a sync standby replica, 0 otherwise.
patroni_up Unknown job, ins, ip, instance, cls N/A
patroni_version gauge job, ins, ip, instance, cls, scope Patroni semver without periods.
patroni_xlog_location counter job, ins, ip, instance, cls, scope Current location of the Postgres transaction log, 0 if this node is not the leader.
patroni_xlog_paused gauge job, ins, ip, instance, cls, scope Value is 1 if the Postgres xlog is paused, 0 otherwise.
patroni_xlog_received_location counter job, ins, ip, instance, cls, scope Current location of the received Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_location counter job, ins, ip, instance, cls, scope Current location of the replayed Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_timestamp gauge job, ins, ip, instance, cls, scope Current timestamp of the replayed Postgres transaction log, 0 if null.
pg:cls:active_backends Unknown job, cls N/A
pg:cls:active_time_rate15m Unknown job, cls N/A
pg:cls:active_time_rate1m Unknown job, cls N/A
pg:cls:active_time_rate5m Unknown job, cls N/A
pg:cls:age Unknown job, cls N/A
pg:cls:buf_alloc_rate1m Unknown job, cls N/A
pg:cls:buf_clean_rate1m Unknown job, cls N/A
pg:cls:buf_flush_backend_rate1m Unknown job, cls N/A
pg:cls:buf_flush_checkpoint_rate1m Unknown job, cls N/A
pg:cls:cpu_count Unknown job, cls N/A
pg:cls:cpu_usage Unknown job, cls N/A
pg:cls:cpu_usage_15m Unknown job, cls N/A
pg:cls:cpu_usage_1m Unknown job, cls N/A
pg:cls:cpu_usage_5m Unknown job, cls N/A
pg:cls:db_size Unknown job, cls N/A
pg:cls:file_size Unknown job, cls N/A
pg:cls:ixact_backends Unknown job, cls N/A
pg:cls:ixact_time_rate1m Unknown job, cls N/A
pg:cls:lag_bytes Unknown job, cls N/A
pg:cls:lag_seconds Unknown job, cls N/A
pg:cls:leader Unknown job, ins, ip, instance, cls N/A
pg:cls:load1 Unknown job, cls N/A
pg:cls:load15 Unknown job, cls N/A
pg:cls:load5 Unknown job, cls N/A
pg:cls:lock_count Unknown job, cls N/A
pg:cls:locks Unknown job, cls, mode N/A
pg:cls:log_size Unknown job, cls N/A
pg:cls:lsn_rate1m Unknown job, cls N/A
pg:cls:members Unknown job, ins, ip, cls N/A
pg:cls:num_backends Unknown job, cls N/A
pg:cls:partition Unknown job, cls N/A
pg:cls:receiver Unknown state, slot_name, job, appname, ip, cls, sender_host, sender_port N/A
pg:cls:rlock_count Unknown job, cls N/A
pg:cls:saturation1 Unknown job, cls N/A
pg:cls:saturation15 Unknown job, cls N/A
pg:cls:saturation5 Unknown job, cls N/A
pg:cls:sender Unknown pid, usename, address, job, ins, appname, ip, cls N/A
pg:cls:session_time_rate1m Unknown job, cls N/A
pg:cls:size Unknown job, cls N/A
pg:cls:slot_count Unknown job, cls N/A
pg:cls:slot_retained_bytes Unknown job, cls N/A
pg:cls:standby_count Unknown job, cls N/A
pg:cls:sync_state Unknown job, cls N/A
pg:cls:timeline Unknown job, cls N/A
pg:cls:tup_deleted_rate1m Unknown job, cls N/A
pg:cls:tup_fetched_rate1m Unknown job, cls N/A
pg:cls:tup_inserted_rate1m Unknown job, cls N/A
pg:cls:tup_modified_rate1m Unknown job, cls N/A
pg:cls:tup_returned_rate1m Unknown job, cls N/A
pg:cls:wal_size Unknown job, cls N/A
pg:cls:xact_commit_rate15m Unknown job, cls N/A
pg:cls:xact_commit_rate1m Unknown job, cls N/A
pg:cls:xact_commit_rate5m Unknown job, cls N/A
pg:cls:xact_rollback_rate15m Unknown job, cls N/A
pg:cls:xact_rollback_rate1m Unknown job, cls N/A
pg:cls:xact_rollback_rate5m Unknown job, cls N/A
pg:cls:xact_total_rate15m Unknown job, cls N/A
pg:cls:xact_total_rate1m Unknown job, cls N/A
pg:cls:xact_total_sigma15m Unknown job, cls N/A
pg:cls:xlock_count Unknown job, cls N/A
pg:db:active_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age_deriv1h Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age_exhaust Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_io_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_read_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_write_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_access_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_hit_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_hit_ratio1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_read_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:conn_limit Unknown datname, job, ins, ip, instance, cls N/A
pg:db:conn_usage Unknown datname, job, ins, ip, instance, cls N/A
pg:db:db_size Unknown datname, job, ins, ip, instance, cls N/A
pg:db:ixact_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:ixact_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:lock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:num_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:rlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:session_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:temp_bytes_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:temp_files_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_deleted_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_fetched_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_inserted_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_modified_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_returned_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:wlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_sigma15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:env:active_backends Unknown job N/A
pg:env:active_time_rate15m Unknown job N/A
pg:env:active_time_rate1m Unknown job N/A
pg:env:active_time_rate5m Unknown job N/A
pg:env:age Unknown job N/A
pg:env:cpu_count Unknown job N/A
pg:env:cpu_usage Unknown job N/A
pg:env:cpu_usage_15m Unknown job N/A
pg:env:cpu_usage_1m Unknown job N/A
pg:env:cpu_usage_5m Unknown job N/A
pg:env:ixact_backends Unknown job N/A
pg:env:ixact_time_rate1m Unknown job N/A
pg:env:lag_bytes Unknown job N/A
pg:env:lag_seconds Unknown job N/A
pg:env:lsn_rate1m Unknown job N/A
pg:env:session_time_rate1m Unknown job N/A
pg:env:tup_deleted_rate1m Unknown job N/A
pg:env:tup_fetched_rate1m Unknown job N/A
pg:env:tup_inserted_rate1m Unknown job N/A
pg:env:tup_modified_rate1m Unknown job N/A
pg:env:tup_returned_rate1m Unknown job N/A
pg:env:xact_commit_rate15m Unknown job N/A
pg:env:xact_commit_rate1m Unknown job N/A
pg:env:xact_commit_rate5m Unknown job N/A
pg:env:xact_rollback_rate15m Unknown job N/A
pg:env:xact_rollback_rate1m Unknown job N/A
pg:env:xact_rollback_rate5m Unknown job N/A
pg:env:xact_total_rate15m Unknown job N/A
pg:env:xact_total_rate1m Unknown job N/A
pg:env:xact_total_sigma15m Unknown job N/A
pg:ins:active_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:age Unknown job, ins, ip, instance, cls N/A
pg:ins:blks_hit_ratio1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_alloc_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_clean_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_flush_backend_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_flush_checkpoint_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_1h Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_req_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_timed_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:conn_limit Unknown job, ins, ip, instance, cls N/A
pg:ins:conn_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_count Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_15m Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_5m Unknown job, ins, ip, instance, cls N/A
pg:ins:db_size Unknown job, ins, ip, instance, cls N/A
pg:ins:file_size Unknown job, ins, ip, instance, cls N/A
pg:ins:fs_size Unknown job, ins, ip, instance, cls N/A
pg:ins:is_leader Unknown job, ins, ip, instance, cls N/A
pg:ins:ixact_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:ixact_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:lag_bytes Unknown job, ins, ip, instance, cls N/A
pg:ins:lag_seconds Unknown job, ins, ip, instance, cls N/A
pg:ins:load1 Unknown job, ins, ip, instance, cls N/A
pg:ins:load15 Unknown job, ins, ip, instance, cls N/A
pg:ins:load5 Unknown job, ins, ip, instance, cls N/A
pg:ins:lock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:locks Unknown job, ins, ip, mode, instance, cls N/A
pg:ins:log_size Unknown job, ins, ip, instance, cls N/A
pg:ins:lsn_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:mem_size Unknown job, ins, ip, instance, cls N/A
pg:ins:num_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:rlock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:saturation1 Unknown job, ins, ip, cls N/A
pg:ins:saturation15 Unknown job, ins, ip, cls N/A
pg:ins:saturation5 Unknown job, ins, ip, cls N/A
pg:ins:session_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:slot_retained_bytes Unknown job, ins, ip, instance, cls N/A
pg:ins:space_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:status Unknown job, ins, ip, instance, cls N/A
pg:ins:sync_state Unknown job, ins, instance, cls N/A
pg:ins:target_count Unknown job, cls, ins N/A
pg:ins:timeline Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_deleted_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_fetched_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_inserted_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_modified_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_returned_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:wal_size Unknown job, ins, ip, instance, cls N/A
pg:ins:wlock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_sigma15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xlock_count Unknown job, ins, ip, instance, cls N/A
pg:query:call_rate1m Unknown datname, query, job, ins, ip, instance, cls N/A
pg:query:rt_1m Unknown datname, query, job, ins, ip, instance, cls N/A
pg:table:scan_rate1m Unknown datname, relname, job, ins, ip, instance, cls N/A
pg_activity_count gauge datname, state, job, ins, ip, instance, cls Count of connection among (datname,state)
pg_activity_max_conn_duration gauge datname, state, job, ins, ip, instance, cls Max backend session duration since state change among (datname, state)
pg_activity_max_duration gauge datname, state, job, ins, ip, instance, cls Max duration since last state change among (datname, state)
pg_activity_max_tx_duration gauge datname, state, job, ins, ip, instance, cls Max transaction duration since state change among (datname, state)
pg_archiver_failed_count counter job, ins, ip, instance, cls Number of failed attempts for archiving WAL files
pg_archiver_finish_count counter job, ins, ip, instance, cls Number of WAL files that have been successfully archived
pg_archiver_last_failed_time counter job, ins, ip, instance, cls Time of the last failed archival operation
pg_archiver_last_finish_time counter job, ins, ip, instance, cls Time of the last successful archive operation
pg_archiver_reset_time gauge job, ins, ip, instance, cls Time at which archive statistics were last reset
pg_backend_count gauge type, job, ins, ip, instance, cls Database backend process count by backend_type
pg_bgwriter_buffers_alloc counter job, ins, ip, instance, cls Number of buffers allocated
pg_bgwriter_buffers_backend counter job, ins, ip, instance, cls Number of buffers written directly by a backend
pg_bgwriter_buffers_backend_fsync counter job, ins, ip, instance, cls Number of times a backend had to execute its own fsync call
pg_bgwriter_buffers_checkpoint counter job, ins, ip, instance, cls Number of buffers written during checkpoints
pg_bgwriter_buffers_clean counter job, ins, ip, instance, cls Number of buffers written by the background writer
pg_bgwriter_checkpoint_sync_time counter job, ins, ip, instance, cls Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in seconds
pg_bgwriter_checkpoint_write_time counter job, ins, ip, instance, cls Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in seconds
pg_bgwriter_checkpoints_req counter job, ins, ip, instance, cls Number of requested checkpoints that have been performed
pg_bgwriter_checkpoints_timed counter job, ins, ip, instance, cls Number of scheduled checkpoints that have been performed
pg_bgwriter_maxwritten_clean counter job, ins, ip, instance, cls Number of times the background writer stopped a cleaning scan because it had written too many buffers
pg_bgwriter_reset_time counter job, ins, ip, instance, cls Time at which bgwriter statistics were last reset
pg_boot_time gauge job, ins, ip, instance, cls unix timestamp when postmaster boot
pg_checkpoint_checkpoint_lsn counter job, ins, ip, instance, cls Latest checkpoint location
pg_checkpoint_elapse gauge job, ins, ip, instance, cls Seconds elapsed since latest checkpoint in seconds
pg_checkpoint_full_page_writes gauge job, ins, ip, instance, cls Latest checkpoint’s full_page_writes enabled
pg_checkpoint_newest_commit_ts_xid counter job, ins, ip, instance, cls Latest checkpoint’s newestCommitTsXid
pg_checkpoint_next_multi_offset counter job, ins, ip, instance, cls Latest checkpoint’s NextMultiOffset
pg_checkpoint_next_multixact_id counter job, ins, ip, instance, cls Latest checkpoint’s NextMultiXactId
pg_checkpoint_next_oid counter job, ins, ip, instance, cls Latest checkpoint’s NextOID
pg_checkpoint_next_xid counter job, ins, ip, instance, cls Latest checkpoint’s NextXID xid
pg_checkpoint_next_xid_epoch counter job, ins, ip, instance, cls Latest checkpoint’s NextXID epoch
pg_checkpoint_oldest_active_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestActiveXID
pg_checkpoint_oldest_commit_ts_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestCommitTsXid
pg_checkpoint_oldest_multi_dbid gauge job, ins, ip, instance, cls Latest checkpoint’s oldestMulti’s DB OID
pg_checkpoint_oldest_multi_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestMultiXid
pg_checkpoint_oldest_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestXID
pg_checkpoint_oldest_xid_dbid gauge job, ins, ip, instance, cls Latest checkpoint’s oldestXID’s DB OID
pg_checkpoint_prev_tli counter job, ins, ip, instance, cls Latest checkpoint’s PrevTimeLineID
pg_checkpoint_redo_lsn counter job, ins, ip, instance, cls Latest checkpoint’s REDO location
pg_checkpoint_time counter job, ins, ip, instance, cls Time of latest checkpoint
pg_checkpoint_tli counter job, ins, ip, instance, cls Latest checkpoint’s TimeLineID
pg_conf_reload_time gauge job, ins, ip, instance, cls seconds since last configuration reload
pg_db_active_time counter datname, job, ins, ip, instance, cls Time spent executing SQL statements in this database, in seconds
pg_db_age gauge datname, job, ins, ip, instance, cls Age of database calculated from datfrozenxid
pg_db_allow_conn gauge datname, job, ins, ip, instance, cls If false(0) then no one can connect to this database.
pg_db_blk_read_time counter datname, job, ins, ip, instance, cls Time spent reading data file blocks by backends in this database, in seconds
pg_db_blk_write_time counter datname, job, ins, ip, instance, cls Time spent writing data file blocks by backends in this database, in seconds
pg_db_blks_access counter datname, job, ins, ip, instance, cls Number of times disk blocks that accessed read+hit
pg_db_blks_hit counter datname, job, ins, ip, instance, cls Number of times disk blocks were found already in the buffer cache
pg_db_blks_read counter datname, job, ins, ip, instance, cls Number of disk blocks read in this database
pg_db_cks_fail_time gauge datname, job, ins, ip, instance, cls Time at which the last data page checksum failure was detected in this database
pg_db_cks_fails counter datname, job, ins, ip, instance, cls Number of data page checksum failures detected in this database, -1 for not enabled
pg_db_confl_confl_bufferpin counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to pinned buffers
pg_db_confl_confl_deadlock counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to deadlocks
pg_db_confl_confl_lock counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to lock timeouts
pg_db_confl_confl_snapshot counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to old snapshots
pg_db_confl_confl_tablespace counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to dropped tablespaces
pg_db_conflicts counter datname, job, ins, ip, instance, cls Number of queries canceled due to conflicts with recovery in this database
pg_db_conn_limit gauge datname, job, ins, ip, instance, cls Sets maximum number of concurrent connections that can be made to this database. -1 means no limit.
pg_db_datid gauge datname, job, ins, ip, instance, cls OID of the database
pg_db_deadlocks counter datname, job, ins, ip, instance, cls Number of deadlocks detected in this database
pg_db_frozen_xid gauge datname, job, ins, ip, instance, cls All transaction IDs before this one have been frozened
pg_db_is_template gauge datname, job, ins, ip, instance, cls If true(1), then this database can be cloned by any user with CREATEDB privileges
pg_db_ixact_time counter datname, job, ins, ip, instance, cls Time spent idling while in a transaction in this database, in seconds
pg_db_numbackends gauge datname, job, ins, ip, instance, cls Number of backends currently connected to this database
pg_db_reset_time counter datname, job, ins, ip, instance, cls Time at which database statistics were last reset
pg_db_session_time counter datname, job, ins, ip, instance, cls Time spent by database sessions in this database, in seconds
pg_db_sessions counter datname, job, ins, ip, instance, cls Total number of sessions established to this database
pg_db_sessions_abandoned counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated because connection to the client was lost
pg_db_sessions_fatal counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated by fatal errors
pg_db_sessions_killed counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated by operator intervention
pg_db_temp_bytes counter datname, job, ins, ip, instance, cls Total amount of data written to temporary files by queries in this database.
pg_db_temp_files counter datname, job, ins, ip, instance, cls Number of temporary files created by queries in this database
pg_db_tup_deleted counter datname, job, ins, ip, instance, cls Number of rows deleted by queries in this database
pg_db_tup_fetched counter datname, job, ins, ip, instance, cls Number of rows fetched by queries in this database
pg_db_tup_inserted counter datname, job, ins, ip, instance, cls Number of rows inserted by queries in this database
pg_db_tup_modified counter datname, job, ins, ip, instance, cls Number of rows modified by queries in this database
pg_db_tup_returned counter datname, job, ins, ip, instance, cls Number of rows returned by queries in this database
pg_db_tup_updated counter datname, job, ins, ip, instance, cls Number of rows updated by queries in this database
pg_db_xact_commit counter datname, job, ins, ip, instance, cls Number of transactions in this database that have been committed
pg_db_xact_rollback counter datname, job, ins, ip, instance, cls Number of transactions in this database that have been rolled back
pg_db_xact_total counter datname, job, ins, ip, instance, cls Number of transactions in this database
pg_downstream_count gauge state, job, ins, ip, instance, cls Count of corresponding state
pg_exporter_agent_up Unknown job, ins, ip, instance, cls N/A
pg_exporter_last_scrape_time gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pg_exporter_query_cache_ttl gauge datname, query, job, ins, ip, instance, cls times to live of query cache
pg_exporter_query_scrape_duration gauge datname, query, job, ins, ip, instance, cls seconds query spending on scrapping
pg_exporter_query_scrape_error_count gauge datname, query, job, ins, ip, instance, cls times the query failed
pg_exporter_query_scrape_hit_count gauge datname, query, job, ins, ip, instance, cls numbers been scrapped from this query
pg_exporter_query_scrape_metric_count gauge datname, query, job, ins, ip, instance, cls numbers of metrics been scrapped from this query
pg_exporter_query_scrape_total_count gauge datname, query, job, ins, ip, instance, cls times exporter server was scraped for metrics
pg_exporter_scrape_duration gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pg_exporter_scrape_error_count counter job, ins, ip, instance, cls times exporter was scraped for metrics and failed
pg_exporter_scrape_total_count counter job, ins, ip, instance, cls times exporter was scraped for metrics
pg_exporter_server_scrape_duration gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pg_exporter_server_scrape_error_count Unknown datname, job, ins, ip, instance, cls N/A
pg_exporter_server_scrape_total_count gauge datname, job, ins, ip, instance, cls times exporter server was scraped for metrics
pg_exporter_server_scrape_total_seconds gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pg_exporter_up gauge job, ins, ip, instance, cls always be 1 if your could retrieve metrics
pg_exporter_uptime gauge job, ins, ip, instance, cls seconds since exporter primary server inited
pg_flush_lsn counter job, ins, ip, instance, cls primary only, location of current wal syncing
pg_func_calls counter datname, funcname, job, ins, ip, instance, cls Number of times this function has been called
pg_func_self_time counter datname, funcname, job, ins, ip, instance, cls Total time spent in this function itself, not including other functions called by it, in ms
pg_func_total_time counter datname, funcname, job, ins, ip, instance, cls Total time spent in this function and all other functions called by it, in ms
pg_in_recovery gauge job, ins, ip, instance, cls server is in recovery mode? 1 for yes 0 for no
pg_index_idx_blks_hit counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of buffer hits in this index
pg_index_idx_blks_read counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of disk blocks read from this index
pg_index_idx_scan counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of index scans initiated on this index
pg_index_idx_tup_fetch counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of live table rows fetched by simple index scans using this index
pg_index_idx_tup_read counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of index entries returned by scans on this index
pg_index_relpages gauge datname, relname, job, ins, relid, ip, instance, cls, idxname Size of the on-disk representation of this index in pages
pg_index_reltuples gauge datname, relname, job, ins, relid, ip, instance, cls, idxname Estimate relation tuples
pg_insert_lsn counter job, ins, ip, instance, cls primary only, location of current wal inserting
pg_io_evictions counter type, job, ins, object, ip, context, instance, cls Number of times a block has been written out from a shared or local buffer
pg_io_extend_time counter type, job, ins, object, ip, context, instance, cls Time spent in extend operations in seconds
pg_io_extends counter type, job, ins, object, ip, context, instance, cls Number of relation extend operations, each of the size specified in op_bytes.
pg_io_fsync_time counter type, job, ins, object, ip, context, instance, cls Time spent in fsync operations in seconds
pg_io_fsyncs counter type, job, ins, object, ip, context, instance, cls Number of fsync calls. These are only tracked in context normal
pg_io_hits counter type, job, ins, object, ip, context, instance, cls The number of times a desired block was found in a shared buffer.
pg_io_op_bytes gauge type, job, ins, object, ip, context, instance, cls The number of bytes per unit of I/O read, written, or extended. 8192 by default
pg_io_read_time counter type, job, ins, object, ip, context, instance, cls Time spent in read operations in seconds
pg_io_reads counter type, job, ins, object, ip, context, instance, cls Number of read operations, each of the size specified in op_bytes.
pg_io_reset_time gauge type, job, ins, object, ip, context, instance, cls Timestamp at which these statistics were last reset
pg_io_reuses counter type, job, ins, object, ip, context, instance, cls The number of times an existing buffer in reused
pg_io_write_time counter type, job, ins, object, ip, context, instance, cls Time spent in write operations in seconds
pg_io_writeback_time counter type, job, ins, object, ip, context, instance, cls Time spent in writeback operations in seconds
pg_io_writebacks counter type, job, ins, object, ip, context, instance, cls Number of units of size op_bytes which the process requested the kernel write out to permanent storage.
pg_io_writes counter type, job, ins, object, ip, context, instance, cls Number of write operations, each of the size specified in op_bytes.
pg_is_in_recovery gauge job, ins, ip, instance, cls 1 if in recovery mode
pg_is_wal_replay_paused gauge job, ins, ip, instance, cls 1 if wal play paused
pg_lag gauge job, ins, ip, instance, cls replica only, replication lag in seconds
pg_last_replay_time gauge job, ins, ip, instance, cls time when last transaction been replayed
pg_lock_count gauge datname, job, ins, ip, mode, instance, cls Number of locks of corresponding mode and database
pg_lsn counter job, ins, ip, instance, cls log sequence number, current write location
pg_meta_info gauge cls, extensions, version, job, ins, primary_conninfo, conf_path, hba_path, ip, cluster_id, instance, listen_port, wal_level, ver_num, cluster_name, data_dir constant 1
pg_query_calls counter datname, query, job, ins, ip, instance, cls Number of times the statement was executed
pg_query_exec_time counter datname, query, job, ins, ip, instance, cls Total time spent executing the statement, in seconds
pg_query_io_time counter datname, query, job, ins, ip, instance, cls Total time the statement spent reading and writing blocks, in seconds
pg_query_rows counter datname, query, job, ins, ip, instance, cls Total number of rows retrieved or affected by the statement
pg_query_sblk_dirtied counter datname, query, job, ins, ip, instance, cls Total number of shared blocks dirtied by the statement
pg_query_sblk_hit counter datname, query, job, ins, ip, instance, cls Total number of shared block cache hits by the statement
pg_query_sblk_read counter datname, query, job, ins, ip, instance, cls Total number of shared blocks read by the statement
pg_query_sblk_written counter datname, query, job, ins, ip, instance, cls Total number of shared blocks written by the statement
pg_query_wal_bytes counter datname, query, job, ins, ip, instance, cls Total amount of WAL bytes generated by the statement
pg_receive_lsn counter job, ins, ip, instance, cls replica only, location of wal synced to disk
pg_recovery_backup_end_lsn counter job, ins, ip, instance, cls Backup end location
pg_recovery_backup_start_lsn counter job, ins, ip, instance, cls Backup start location
pg_recovery_min_lsn counter job, ins, ip, instance, cls Minimum recovery ending location
pg_recovery_min_timeline counter job, ins, ip, instance, cls Min recovery ending loc’s timeline
pg_recovery_prefetch_block_distance gauge job, ins, ip, instance, cls How many blocks ahead the prefetcher is looking
pg_recovery_prefetch_hit counter job, ins, ip, instance, cls Number of blocks not prefetched because they were already in the buffer pool
pg_recovery_prefetch_io_depth gauge job, ins, ip, instance, cls How many prefetches have been initiated but are not yet known to have completed
pg_recovery_prefetch_prefetch counter job, ins, ip, instance, cls Number of blocks prefetched because they were not in the buffer pool
pg_recovery_prefetch_reset_time counter job, ins, ip, instance, cls Time at which these recovery prefetch statistics were last reset
pg_recovery_prefetch_skip_fpw gauge job, ins, ip, instance, cls Number of blocks not prefetched because a full page image was included in the WAL
pg_recovery_prefetch_skip_init counter job, ins, ip, instance, cls Number of blocks not prefetched because they would be zero-initialized
pg_recovery_prefetch_skip_new counter job, ins, ip, instance, cls Number of blocks not prefetched because they didn’t exist yet
pg_recovery_prefetch_skip_rep counter job, ins, ip, instance, cls Number of blocks not prefetched because they were already recently prefetched
pg_recovery_prefetch_wal_distance gauge job, ins, ip, instance, cls How many bytes ahead the prefetcher is looking
pg_recovery_require_record gauge job, ins, ip, instance, cls End-of-backup record required
pg_recv_flush_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location already received and flushed to disk
pg_recv_flush_tli counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Timeline number of last write-ahead log location received and flushed to disk
pg_recv_init_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port First write-ahead log location used when WAL receiver is started
pg_recv_init_tli counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port First timeline number used when WAL receiver is started
pg_recv_msg_recv_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Receipt time of last message received from origin WAL sender
pg_recv_msg_send_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Send time of last message received from origin WAL sender
pg_recv_pid gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Process ID of the WAL receiver process
pg_recv_reported_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location reported to origin WAL sender
pg_recv_reported_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Time of last write-ahead log location reported to origin WAL sender
pg_recv_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Time of current snapshot
pg_recv_write_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location already received and written to disk, but not flushed.
pg_relkind_count gauge datname, job, ins, ip, instance, cls, relkind Number of relations of corresponding relkind
pg_repl_backend_xmin counter pid, usename, address, job, ins, appname, ip, instance, cls This standby’s xmin horizon reported by hot_standby_feedback.
pg_repl_client_port gauge pid, usename, address, job, ins, appname, ip, instance, cls TCP port number that the client is using for communication with this WAL sender, or -1 if a Unix socket is used
pg_repl_flush_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position flushed to disk by this standby server diff with current lsn
pg_repl_flush_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written and flushed it
pg_repl_flush_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location flushed to disk by this standby server
pg_repl_launch_time counter pid, usename, address, job, ins, appname, ip, instance, cls Time when this process was started, i.e., when the client connected to this WAL sender
pg_repl_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Current log position on this server
pg_repl_replay_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position replayed into the database on this standby server diff with current lsn
pg_repl_replay_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written, flushed and applied it
pg_repl_replay_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location replayed into the database on this standby server
pg_repl_reply_time gauge pid, usename, address, job, ins, appname, ip, instance, cls Send time of last reply message received from standby server
pg_repl_sent_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position sent to this standby server diff with current lsn
pg_repl_sent_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location sent on this connection
pg_repl_state gauge pid, usename, address, job, ins, appname, ip, instance, cls Current WAL sender encoded state 0-4 for streaming startup catchup backup stopping
pg_repl_sync_priority gauge pid, usename, address, job, ins, appname, ip, instance, cls Priority of this standby server for being chosen as the synchronous standby
pg_repl_sync_state gauge pid, usename, address, job, ins, appname, ip, instance, cls Encoded synchronous state of this standby server, 0-3 for async potential sync quorum
pg_repl_time counter pid, usename, address, job, ins, appname, ip, instance, cls Current timestamp in unix epoch
pg_repl_write_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position written to disk by this standby server diff with current lsn
pg_repl_write_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written it
pg_repl_write_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location written to disk by this standby server
pg_replay_lsn counter job, ins, ip, instance, cls replica only, location of wal applied
pg_seq_blks_hit counter datname, job, ins, ip, instance, cls, seqname Number of buffer hits in this sequence
pg_seq_blks_read counter datname, job, ins, ip, instance, cls, seqname Number of disk blocks read from this sequence
pg_seq_last_value counter datname, job, ins, ip, instance, cls, seqname The last sequence value written to disk
pg_setting_block_size gauge job, ins, ip, instance, cls pg page block size, 8192 by default
pg_setting_data_checksums gauge job, ins, ip, instance, cls whether data checksum is enabled, 1 enabled 0 disabled
pg_setting_max_connections gauge job, ins, ip, instance, cls number of concurrent connections to the database server
pg_setting_max_locks_per_transaction gauge job, ins, ip, instance, cls no more than this many distinct objects can be locked at any one time
pg_setting_max_prepared_transactions gauge job, ins, ip, instance, cls maximum number of transactions that can be in the prepared state simultaneously
pg_setting_max_replication_slots gauge job, ins, ip, instance, cls maximum number of replication slots
pg_setting_max_wal_senders gauge job, ins, ip, instance, cls maximum number of concurrent connections from standby servers
pg_setting_max_worker_processes gauge job, ins, ip, instance, cls maximum number of background processes that the system can support
pg_setting_wal_log_hints gauge job, ins, ip, instance, cls whether wal_log_hints is enabled, 1 enabled 0 disabled
pg_size_bytes gauge datname, job, ins, ip, instance, cls File size in bytes
pg_slot_active gauge slot_name, job, ins, ip, instance, cls True(1) if this slot is currently actively being used
pg_slot_catalog_xmin counter slot_name, job, ins, ip, instance, cls The oldest transaction affecting the system catalogs that this slot needs the database to retain.
pg_slot_confirm_lsn counter slot_name, job, ins, ip, instance, cls The address (LSN) up to which the logical slot’s consumer has confirmed receiving data.
pg_slot_reset_time counter slot_name, job, ins, ip, instance, cls When statistics were last reset
pg_slot_restart_lsn counter slot_name, job, ins, ip, instance, cls The address (LSN) of oldest WAL which still might be required by the consumer of this slot
pg_slot_retained_bytes gauge slot_name, job, ins, ip, instance, cls Size of bytes that retained for this slot
pg_slot_safe_wal_size gauge slot_name, job, ins, ip, instance, cls bytes that can be written to WAL which will not make slot into lost
pg_slot_spill_bytes counter slot_name, job, ins, ip, instance, cls Bytes that spilled to disk due to logical decode mem exceeding
pg_slot_spill_count counter slot_name, job, ins, ip, instance, cls Xacts that spilled to disk due to logical decode mem exceeding (a xact can be spilled multiple times)
pg_slot_spill_txns counter slot_name, job, ins, ip, instance, cls Xacts that spilled to disk due to logical decode mem exceeding (subtrans included)
pg_slot_stream_bytes counter slot_name, job, ins, ip, instance, cls Bytes that streamed to decoding output plugin after mem exceed
pg_slot_stream_count counter slot_name, job, ins, ip, instance, cls Xacts that streamed to decoding output plugin after mem exceed (a xact can be streamed multiple times)
pg_slot_stream_txns counter slot_name, job, ins, ip, instance, cls Xacts that streamed to decoding output plugin after mem exceed
pg_slot_temporary gauge slot_name, job, ins, ip, instance, cls True(1) if this is a temporary replication slot.
pg_slot_total_bytes counter slot_name, job, ins, ip, instance, cls Number of decoded bytes sent to the decoding output plugin for this slot
pg_slot_total_txns counter slot_name, job, ins, ip, instance, cls Number of decoded xacts sent to the decoding output plugin for this slot
pg_slot_wal_status gauge slot_name, job, ins, ip, instance, cls WAL reserve status 0-3 means reserved,extended,unreserved,lost, -1 means other
pg_slot_xmin counter slot_name, job, ins, ip, instance, cls The oldest transaction that this slot needs the database to retain.
pg_slru_blks_exists counter job, ins, ip, instance, cls Number of blocks checked for existence for this SLRU
pg_slru_blks_hit counter job, ins, ip, instance, cls Number of times disk blocks were found already in the SLRU, so that a read was not necessary
pg_slru_blks_read counter job, ins, ip, instance, cls Number of disk blocks read for this SLRU
pg_slru_blks_written counter job, ins, ip, instance, cls Number of disk blocks written for this SLRU
pg_slru_blks_zeroed counter job, ins, ip, instance, cls Number of blocks zeroed during initializations
pg_slru_flushes counter job, ins, ip, instance, cls Number of flushes of dirty data for this SLRU
pg_slru_reset_time counter job, ins, ip, instance, cls Time at which these statistics were last reset
pg_slru_truncates counter job, ins, ip, instance, cls Number of truncates for this SLRU
pg_ssl_disabled gauge job, ins, ip, instance, cls Number of client connection that does not use ssl
pg_ssl_enabled gauge job, ins, ip, instance, cls Number of client connection that use ssl
pg_sync_standby_enabled gauge job, ins, ip, names, instance, cls Synchronous commit enabled, 1 if enabled, 0 if disabled
pg_table_age gauge datname, relname, job, ins, ip, instance, cls Age of this table in vacuum cycles
pg_table_analyze_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been manually analyzed
pg_table_autoanalyze_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been analyzed by the autovacuum daemon
pg_table_autovacuum_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been vacuumed by the autovacuum daemon
pg_table_frozenxid counter datname, relname, job, ins, ip, instance, cls All txid before this have been frozen on this table
pg_table_heap_blks_hit counter datname, relname, job, ins, ip, instance, cls Number of buffer hits in this table
pg_table_heap_blks_read counter datname, relname, job, ins, ip, instance, cls Number of disk blocks read from this table
pg_table_idx_blks_hit counter datname, relname, job, ins, ip, instance, cls Number of buffer hits in all indexes on this table
pg_table_idx_blks_read counter datname, relname, job, ins, ip, instance, cls Number of disk blocks read from all indexes on this table
pg_table_idx_scan counter datname, relname, job, ins, ip, instance, cls Number of index scans initiated on this table
pg_table_idx_tup_fetch counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by index scans
pg_table_kind gauge datname, relname, job, ins, ip, instance, cls Relation kind r/table/114
pg_table_n_dead_tup gauge datname, relname, job, ins, ip, instance, cls Estimated number of dead rows
pg_table_n_ins_since_vacuum gauge datname, relname, job, ins, ip, instance, cls Estimated number of rows inserted since this table was last vacuumed
pg_table_n_live_tup gauge datname, relname, job, ins, ip, instance, cls Estimated number of live rows
pg_table_n_mod_since_analyze gauge datname, relname, job, ins, ip, instance, cls Estimated number of rows modified since this table was last analyzed
pg_table_n_tup_del counter datname, relname, job, ins, ip, instance, cls Number of rows deleted
pg_table_n_tup_hot_upd counter datname, relname, job, ins, ip, instance, cls Number of rows HOT updated (i.e with no separate index update required)
pg_table_n_tup_ins counter datname, relname, job, ins, ip, instance, cls Number of rows inserted
pg_table_n_tup_mod counter datname, relname, job, ins, ip, instance, cls Number of rows modified (insert + update + delete)
pg_table_n_tup_newpage_upd counter datname, relname, job, ins, ip, instance, cls Number of rows updated where the successor version goes onto a new heap page
pg_table_n_tup_upd counter datname, relname, job, ins, ip, instance, cls Number of rows updated (includes HOT updated rows)
pg_table_ncols gauge datname, relname, job, ins, ip, instance, cls Number of columns in the table
pg_table_pages gauge datname, relname, job, ins, ip, instance, cls Size of the on-disk representation of this table in pages
pg_table_relid gauge datname, relname, job, ins, ip, instance, cls Relation oid of this table
pg_table_seq_scan counter datname, relname, job, ins, ip, instance, cls Number of sequential scans initiated on this table
pg_table_seq_tup_read counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by sequential scans
pg_table_size_bytes gauge datname, relname, job, ins, ip, instance, cls Total bytes of this table (including toast, index, toast index)
pg_table_size_indexsize gauge datname, relname, job, ins, ip, instance, cls Bytes of all related indexes of this table
pg_table_size_relsize gauge datname, relname, job, ins, ip, instance, cls Bytes of this table itself (main, vm, fsm)
pg_table_size_toastsize gauge datname, relname, job, ins, ip, instance, cls Bytes of toast tables of this table
pg_table_tbl_scan counter datname, relname, job, ins, ip, instance, cls Number of scans initiated on this table
pg_table_tup_read counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by scans
pg_table_tuples counter datname, relname, job, ins, ip, instance, cls All txid before this have been frozen on this table
pg_table_vacuum_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been manually vacuumed (not counting VACUUM FULL)
pg_timestamp gauge job, ins, ip, instance, cls database current timestamp
pg_up gauge job, ins, ip, instance, cls last scrape was able to connect to the server: 1 for yes, 0 for no
pg_uptime gauge job, ins, ip, instance, cls seconds since postmaster start
pg_version gauge job, ins, ip, instance, cls server version number
pg_wait_count gauge datname, job, ins, event, ip, instance, cls Count of WaitEvent on target database
pg_wal_buffers_full counter job, ins, ip, instance, cls Number of times WAL data was written to disk because WAL buffers became full
pg_wal_bytes counter job, ins, ip, instance, cls Total amount of WAL generated in bytes
pg_wal_fpi counter job, ins, ip, instance, cls Total number of WAL full page images generated
pg_wal_records counter job, ins, ip, instance, cls Total number of WAL records generated
pg_wal_reset_time counter job, ins, ip, instance, cls When statistics were last reset
pg_wal_sync counter job, ins, ip, instance, cls Number of times WAL files were synced to disk via issue_xlog_fsync request
pg_wal_sync_time counter job, ins, ip, instance, cls Total amount of time spent syncing WAL files to disk via issue_xlog_fsync request, in seconds
pg_wal_write counter job, ins, ip, instance, cls Number of times WAL buffers were written out to disk via XLogWrite request.
pg_wal_write_time counter job, ins, ip, instance, cls Total amount of time spent writing WAL buffers to disk via XLogWrite request in seconds
pg_write_lsn counter job, ins, ip, instance, cls primary only, location of current wal writing
pg_xact_xmax counter job, ins, ip, instance, cls First as-yet-unassigned txid. txid >= this are invisible.
pg_xact_xmin counter job, ins, ip, instance, cls Earliest txid that is still active
pg_xact_xnum gauge job, ins, ip, instance, cls Current active transaction count
pgbouncer:cls:load1 Unknown job, cls N/A
pgbouncer:cls:load15 Unknown job, cls N/A
pgbouncer:cls:load5 Unknown job, cls N/A
pgbouncer:db:conn_usage Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:conn_usage_reserve Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_current_conn Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_disabled Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_max_conn Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_paused Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_reserve_size Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_size Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:ins:free_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:free_servers Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load1 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load15 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load5 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:login_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pool_databases Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pool_users Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pools Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:used_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer_database_current_connections gauge datname, job, ins, ip, instance, host, cls, real_datname, port Current number of connections for this database
pgbouncer_database_disabled gauge datname, job, ins, ip, instance, host, cls, real_datname, port True(1) if this database is currently disabled, else 0
pgbouncer_database_max_connections gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of allowed connections for this database
pgbouncer_database_min_pool_size gauge datname, job, ins, ip, instance, host, cls, real_datname, port Minimum number of server connections
pgbouncer_database_paused gauge datname, job, ins, ip, instance, host, cls, real_datname, port True(1) if this database is currently paused, else 0
pgbouncer_database_pool_size gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of server connections
pgbouncer_database_reserve_pool gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of additional connections for this database
pgbouncer_exporter_agent_up Unknown job, ins, ip, instance, cls N/A
pgbouncer_exporter_last_scrape_time gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pgbouncer_exporter_query_cache_ttl gauge datname, query, job, ins, ip, instance, cls times to live of query cache
pgbouncer_exporter_query_scrape_duration gauge datname, query, job, ins, ip, instance, cls seconds query spending on scrapping
pgbouncer_exporter_query_scrape_error_count gauge datname, query, job, ins, ip, instance, cls times the query failed
pgbouncer_exporter_query_scrape_hit_count gauge datname, query, job, ins, ip, instance, cls numbers been scrapped from this query
pgbouncer_exporter_query_scrape_metric_count gauge datname, query, job, ins, ip, instance, cls numbers of metrics been scrapped from this query
pgbouncer_exporter_query_scrape_total_count gauge datname, query, job, ins, ip, instance, cls times exporter server was scraped for metrics
pgbouncer_exporter_scrape_duration gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pgbouncer_exporter_scrape_error_count counter job, ins, ip, instance, cls times exporter was scraped for metrics and failed
pgbouncer_exporter_scrape_total_count counter job, ins, ip, instance, cls times exporter was scraped for metrics
pgbouncer_exporter_server_scrape_duration gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pgbouncer_exporter_server_scrape_total_count gauge datname, job, ins, ip, instance, cls times exporter server was scraped for metrics
pgbouncer_exporter_server_scrape_total_seconds gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pgbouncer_exporter_up gauge job, ins, ip, instance, cls always be 1 if your could retrieve metrics
pgbouncer_exporter_uptime gauge job, ins, ip, instance, cls seconds since exporter primary server inited
pgbouncer_in_recovery gauge job, ins, ip, instance, cls server is in recovery mode? 1 for yes 0 for no
pgbouncer_list_items gauge job, ins, ip, instance, list, cls Number of corresponding pgbouncer object
pgbouncer_pool_active_cancel_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have forwarded query cancellations to the server and are waiting for the server response.
pgbouncer_pool_active_cancel_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are currently forwarding a cancel request
pgbouncer_pool_active_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that are linked to server connection and can process queries
pgbouncer_pool_active_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are linked to a client
pgbouncer_pool_cancel_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have not forwarded query cancellations to the server yet.
pgbouncer_pool_cancel_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode cancel requests have completed that were sent to cancel a query on this server
pgbouncer_pool_idle_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are unused and immediately usable for client queries
pgbouncer_pool_login_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections currently in the process of logging in
pgbouncer_pool_maxwait gauge datname, job, ins, ip, instance, user, cls, pool_mode How long the first(oldest) client in the queue has waited, in seconds, key metric
pgbouncer_pool_maxwait_us gauge datname, job, ins, ip, instance, user, cls, pool_mode Microsecond part of the maximum waiting time.
pgbouncer_pool_tested_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are currently running reset or check query
pgbouncer_pool_used_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that have been idle for more than server_check_delay (means have to run check query)
pgbouncer_pool_waiting_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have sent queries but have not yet got a server connection
pgbouncer_stat_avg_query_count gauge datname, job, ins, ip, instance, cls Average queries per second in last stat period
pgbouncer_stat_avg_query_time gauge datname, job, ins, ip, instance, cls Average query duration, in seconds
pgbouncer_stat_avg_recv gauge datname, job, ins, ip, instance, cls Average received (from clients) bytes per second
pgbouncer_stat_avg_sent gauge datname, job, ins, ip, instance, cls Average sent (to clients) bytes per second
pgbouncer_stat_avg_wait_time gauge datname, job, ins, ip, instance, cls Time spent by clients waiting for a server, in seconds (average per second).
pgbouncer_stat_avg_xact_count gauge datname, job, ins, ip, instance, cls Average transactions per second in last stat period
pgbouncer_stat_avg_xact_time gauge datname, job, ins, ip, instance, cls Average transaction duration, in seconds
pgbouncer_stat_total_query_count gauge datname, job, ins, ip, instance, cls Total number of SQL queries pooled by pgbouncer
pgbouncer_stat_total_query_time counter datname, job, ins, ip, instance, cls Total number of seconds spent when executing queries
pgbouncer_stat_total_received counter datname, job, ins, ip, instance, cls Total volume in bytes of network traffic received by pgbouncer
pgbouncer_stat_total_sent counter datname, job, ins, ip, instance, cls Total volume in bytes of network traffic sent by pgbouncer
pgbouncer_stat_total_wait_time counter datname, job, ins, ip, instance, cls Time spent by clients waiting for a server, in seconds
pgbouncer_stat_total_xact_count gauge datname, job, ins, ip, instance, cls Total number of SQL transactions pooled by pgbouncer
pgbouncer_stat_total_xact_time counter datname, job, ins, ip, instance, cls Total number of seconds spent when in a transaction
pgbouncer_up gauge job, ins, ip, instance, cls last scrape was able to connect to the server: 1 for yes, 0 for no
pgbouncer_version gauge job, ins, ip, instance, cls server version number
process_cpu_seconds_total counter job, ins, ip, instance, cls Total user and system CPU time spent in seconds.
process_max_fds gauge job, ins, ip, instance, cls Maximum number of open file descriptors.
process_open_fds gauge job, ins, ip, instance, cls Number of open file descriptors.
process_resident_memory_bytes gauge job, ins, ip, instance, cls Resident memory size in bytes.
process_start_time_seconds gauge job, ins, ip, instance, cls Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge job, ins, ip, instance, cls Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge job, ins, ip, instance, cls Maximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flight gauge job, ins, ip, instance, cls Current number of scrapes being served.
promhttp_metric_handler_requests_total counter code, job, ins, ip, instance, cls Total number of scrapes by HTTP status code.
scrape_duration_seconds Unknown job, ins, ip, instance, cls N/A
scrape_samples_post_metric_relabeling Unknown job, ins, ip, instance, cls N/A
scrape_samples_scraped Unknown job, ins, ip, instance, cls N/A
scrape_series_added Unknown job, ins, ip, instance, cls N/A
up Unknown job, ins, ip, instance, cls N/A

11 - Parameters

Customize PostgreSQL clusters with 120 parameters in the PGSQL module

The PGSQL module needs to be installed on nodes managed by Pigsty (i.e., nodes that have the NODE module configured), and also requires an available ETCD cluster in your deployment to store cluster metadata.

Installing the PGSQL module on a single node will create a standalone PGSQL server/instance, i.e., a primary instance. Installing on additional nodes will create read replicas, which can serve as standby instances and handle read-only requests. You can also create offline instances for ETL/OLAP/interactive queries, use sync standby and quorum commit to improve data consistency, or even set up standby clusters and delayed clusters to quickly respond to data loss caused by human errors and software defects.

You can define multiple PGSQL clusters and further organize them into a horizontal sharding cluster: Pigsty natively supports Citus cluster groups, allowing you to upgrade your standard PGSQL cluster in-place to a distributed database cluster.

Section Description
PG_ID PostgreSQL cluster and instance identity parameters
PG_BUSINESS Business users, databases, services and access control rule definition
PG_INSTALL PostgreSQL installation: version, paths, packages
PG_BOOTSTRAP PostgreSQL cluster initialization: Patroni high availability
PG_PROVISION PostgreSQL cluster template provisioning: roles, privileges, extensions
PG_BACKUP pgBackRest backup and recovery configuration
PG_ACCESS Service exposure, connection pooling, VIP, DNS client access config
PG_MONITOR PostgreSQL monitoring exporter configuration
PG_REMOVE PostgreSQL instance cleanup and uninstall configuration

Parameter Overview

PG_ID parameters are used to define PostgreSQL cluster and instance identity, including cluster name, instance sequence number, role, shard, and other core identity parameters.

Parameter Type Level Description
pg_mode enum C pgsql cluster mode: pgsql,citus,mssql,mysql,ivory,pgtde,polar,gpsql,agens,oriole,pgedge
pg_cluster string C pgsql cluster name, REQUIRED identity parameter
pg_seq int I pgsql instance seq number, REQUIRED identity parameter
pg_role enum I pgsql instance role: primary, replica, standby, offline, or delayed
pg_instances dict I define multiple pg instances on node in {port:ins_vars} format
pg_upstream ip I replication upstream IP for a standby cluster or cascade replica
pg_shard string C pgsql shard name; specify explicitly for horizontal sharding
pg_group int C non-negative pgsql shard index; specify explicitly for horizontal sharding
gp_role enum C greenplum role of this cluster, could be master or segment
pg_exporters dict C additional pg_exporters to monitor remote postgres instances
pg_offline_query bool I set to true to mark this replica as offline instance for offline queries

PG_BUSINESS parameters are used to define business users, databases, services and access control rules, as well as default system user credentials.

Parameter Type Level Description
pg_users user[] C postgres business users
pg_databases database[] C postgres business databases
pg_services service[] C postgres business services
pg_hba_rules hba[] C business hba rules for postgres
pgb_hba_rules hba[] C business hba rules for pgbouncer
pg_crontab string[] C crontab entries for postgres dbsu
pg_replication_username username G postgres replication username, replicator by default
pg_replication_password password G postgres replication password, DBUser.Replicator by default
pg_admin_username username G postgres admin username, dbuser_dba by default
pg_admin_password password G postgres admin password in plain text, DBUser.DBA by default
pg_monitor_username username G postgres monitor username, dbuser_monitor by default
pg_monitor_password password G postgres monitor password, DBUser.Monitor by default
pg_dbsu_password password G/C dbsu password, empty string disables it by default, best not set

PG_INSTALL parameters are used to configure PostgreSQL installation options, including version, paths, packages, and extensions.

Parameter Type Level Description
pg_dbsu username C os dbsu name, postgres by default, better not change it
pg_dbsu_uid int C os dbsu uid and gid, 26 for default postgres user and group
pg_dbsu_sudo enum C dbsu sudo privilege, none,limit,all,nopass. limit by default
pg_dbsu_home path C postgresql home directory, /var/lib/pgsql by default
pg_dbsu_ssh_exchange bool C exchange postgres dbsu ssh key among same pgsql cluster
pg_version enum C postgres major version to be installed, 18 by default
pg_bin_dir path C postgres binary dir, /usr/pgsql/bin by default
pg_log_dir path C postgres log dir, /pg/log/postgres by default
pg_packages string[] C pg packages to be installed, ${pg_version} will be replaced
pg_extensions string[] C pg extensions to be installed, ${pg_version} will be replaced

PG_BOOTSTRAP parameters are used to configure PostgreSQL cluster initialization, including Patroni high availability, storage paths, networking, encoding, and other core settings.

Parameter Type Level Description
pg_data path C PostgreSQL data directory, /pg/data by default
pg_fs_main path C mountpoint/path for pg main data, /data/postgres by default
pg_fs_backup path C mountpoint/path for pg backup data, /data/backups by default
pg_storage_type enum C storage type for pg main data, SSD,HDD. SSD by default
pg_dummy_filesize size C size of /pg/dummy, hold 64MB disk space for emergency use
pg_listen ip(s) C/I postgres/pgbouncer listen addr, comma separated list, 0.0.0.0
pg_port port C postgres listen port, 5432 by default
pg_localhost path C postgres unix socket dir for localhost connection
pg_namespace path C top level key namespace in etcd, used by patroni & vip
patroni_enabled bool C if disabled, no postgres cluster will be created during init
patroni_mode enum C patroni working mode: default,pause,remove
patroni_port port C patroni listen port, 8008 by default
patroni_log_dir path C patroni log dir, /pg/log/patroni by default
patroni_ssl_enabled bool G secure patroni RestAPI communications with SSL?
patroni_watchdog_mode enum C patroni watchdog mode: automatic,required,off. off by default
patroni_username username C patroni restapi username, postgres by default
patroni_password password C patroni restapi password, Patroni.API by default
pg_primary_db string C primary database name, used by citus,etc. postgres by default
pg_parameters dict C extra parameters in postgresql.auto.conf
pg_files path[] C extra files to be copied to PGDATA (e.g. license files)
pg_conf enum C config template: oltp,olap,crit,tiny. oltp.yml by default
pg_max_conn int C postgres max connections, auto will use recommended value
pg_shared_buffer_ratio float C postgres shared buffer memory ratio, 0.25 by default, 0.1~0.4
pg_rto enum C RTO mode: fast, norm, safe, or wide; default norm
pg_rto_plan dict G RTO presets for Patroni HA and HAProxy health-check timeouts
pg_rpo int C sampled lag threshold for Patroni failover candidates; default 1MiB
pg_libs string C preloaded libraries, pg_stat_statements,auto_explain by default
pg_delay interval I WAL replay apply delay for standby cluster, for delayed replica
pg_checksum bool C enable data checksum for postgres cluster?
pg_pwd_enc enum C password encryption algorithm: fixed to scram-sha-256
pg_encoding enum C database cluster encoding, UTF8 by default
pg_locale enum C database cluster locale, C by default
pg_lc_collate enum C database cluster collate, C by default
pg_lc_ctype enum C database character type, C by default
pg_io_method enum C PostgreSQL IO method: auto, sync, worker, io_uring
pg_etcd_password password C etcd password for this PostgreSQL cluster, cluster name by default
pgsodium_key string C pgsodium encryption master key, 64 hex digits, sha256(pg_cluster)
pgsodium_getkey_script path C pgsodium getkey script path, uses template pgsodium_getkey

PG_PROVISION parameters are used to configure PostgreSQL cluster template provisioning, including default roles, privileges, schemas, extensions, and HBA rules.

Parameter Type Level Description
pg_provision bool C provision postgres cluster content after bootstrap?
pg_init string G/C init script for cluster template, pg-init by default
pg_default_roles role[] G/C default predefined roles and system users in postgres
pg_default_privileges string[] G/C default privileges when created by admin user
pg_default_schemas string[] G/C default schemas to be created
pg_default_extensions extension[] G/C default extensions to be created
pg_reload bool A reload postgres config after hba changes?
pg_default_hba_rules hba[] G/C postgres default host-based auth rules, global default HBA
pgb_default_hba_rules hba[] G/C pgbouncer default host-based auth rules, global default HBA

PG_BACKUP parameters are used to configure pgBackRest backup and recovery, including repository type, paths, and retention policies.

Parameter Type Level Description
pgbackrest_enabled bool C enable pgbackrest on pgsql host?
pgbackrest_log_dir path C pgbackrest log dir, /pg/log/pgbackrest by default
pgbackrest_method enum C pgbackrest repo method: local,minio,etc…
pgbackrest_init_backup bool C perform full backup after init? true by default
pgbackrest_repo dict G/C pgbackrest repo definition

PG_ACCESS parameters are used to configure service exposure, connection pooling, VIP, DNS, and other client access options.

Parameter Type Level Description
pgbouncer_enabled bool C if disabled, pgbouncer will not be configured
pgbouncer_port port C pgbouncer listen port, 6432 by default
pgbouncer_log_dir path C pgbouncer log dir, /pg/log/pgbouncer by default
pgbouncer_auth_query bool C use AuthQuery to get unlisted business users from postgres?
pgbouncer_poolmode enum C pool mode: transaction,session,statement. transaction by default
pgbouncer_sslmode enum C pgbouncer client ssl mode, disabled by default
pgbouncer_ignore_param string[] C pgbouncer ignore startup parameters list
pg_weight int I relative load balancing weight in service, 0-255, 100 by default
pg_service_provider string G/C dedicated haproxy node group name, or use local haproxy
pg_default_service_dest enum G/C default service dest if svc.dest=‘default’: postgres or pgbouncer
pg_default_services service[] G/C postgres default service definition list, shared globally
pg_vip_enabled bool C enable L2 VIP for pgsql primary? disabled by default
pg_vip_address cidr4 C vip address in <ipv4>/<mask> format, required if vip enabled
pg_vip_interface string C/I vip network interface to bind, auto by default
pg_dns_suffix string C pgsql dns suffix, empty by default
pg_dns_target enum C PG DNS resolves to: auto, primary, vip, none, or specific IP

PG_MONITOR parameters are used to configure PostgreSQL monitoring exporters, including pg_exporter, pgbouncer_exporter, and pgbackrest_exporter.

Parameter Type Level Description
pg_exporter_enabled bool C enable pg_exporter on pgsql host?
pg_exporter_config string C pg_exporter config file/template name
pg_exporter_cache_ttls string C pg_exporter collector ttl stages, ‘1,10,60,300’ by default
pg_exporter_port port C pg_exporter listen port, 9630 by default
pg_exporter_params string C extra URL parameters for pg_exporter dsn
pg_exporter_url pgurl C overwrite auto-generated postgres DSN connection string
pg_exporter_auto_discovery bool C enable auto database discovery for monitoring? enabled
pg_exporter_exclude_database string C excluded database list when auto-discovery, comma separated
pg_exporter_include_database string C only monitor these databases when auto-discovery enabled
pg_exporter_connect_timeout int C pg_exporter connect timeout in ms, 200 by default
pg_exporter_options arg C extra command line options for pg_exporter
pgbouncer_exporter_enabled bool C enable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_port port C pgbouncer_exporter listen port, 9631 by default
pgbouncer_exporter_url pgurl C overwrite auto-generated pgbouncer dsn connection string
pgbouncer_exporter_options arg C extra command line options for pgbouncer_exporter
pgbackrest_exporter_enabled bool C enable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_port port C pgbackrest_exporter listen port, 9854 by default
pgbackrest_exporter_options arg C extra command line options for pgbackrest_exporter

PG_REMOVE parameters are used to configure PostgreSQL instance cleanup and uninstall behavior, including data directory, backup, and package removal control.

Parameter Type Level Description
pg_rm_data bool G/C/A remove postgres data directory when removing instance?
pg_rm_backup bool G/C/A remove pgbackrest backup when removing primary?
pg_rm_pkg bool G/C/A uninstall related packages when removing pgsql instance?
pg_safeguard bool G/C/A prevent accidental pgsql cleanup operations? false

PG_ID

Here are commonly used parameters for identifying entities in the PGSQL module: clusters, instances, services, etc…

# pg_cluster:           #CLUSTER  # pgsql cluster name, required identity parameter
# pg_seq: 0             #INSTANCE # pgsql instance seq number, required identity parameter
# pg_role: replica      #INSTANCE # pgsql role, required, could be primary,replica,offline
# pg_instances: {}      #INSTANCE # define multiple pg instances on node in `{port:ins_vars}` format
# pg_upstream:          #INSTANCE # repl upstream ip addr for standby cluster or cascade replica
# pg_shard:             #CLUSTER  # pgsql shard name, optional identity for sharding clusters
# pg_group: 0           #CLUSTER  # pgsql shard index number, optional identity for sharding clusters
# gp_role: master       #CLUSTER  # greenplum role of this cluster, could be master or segment
pg_offline_query: false #INSTANCE # set to true to enable offline query on this instance

You must explicitly specify these identity parameters, they have no default values:

Name Type Level Description
pg_cluster string C PG cluster name
pg_seq number I PG instance ID
pg_role enum I PG instance role
pg_shard string C Shard name
pg_group number C Shard index
  • pg_cluster: Identifies the cluster name, configured at cluster level.
  • pg_role: Configured at instance level, identifies the role of the instance. Only primary role is treated specially. If not specified, defaults to replica role, with special delayed and offline roles.
  • pg_seq: Used to identify instances within a cluster, typically an integer starting from 0 or 1, once assigned it doesn’t change.
  • {{ pg_cluster }}-{{ pg_seq }} uniquely identifies an instance, i.e., pg_instance.
  • {{ pg_cluster }}-{{ pg_role }} identifies services within the cluster, i.e., pg_service.
  • pg_shard and pg_group are used for horizontal sharding clusters, only for citus, greenplum, and matrixdb.

pg_cluster, pg_role, pg_seq are core identity parameters, required for any Postgres cluster and must be explicitly specified. Here is an example:

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: replica}
    10.10.10.12: {pg_seq: 2, pg_role: primary}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test

All other parameters can be inherited from global or default configuration, but identity parameters must be explicitly specified and manually assigned.

pg_mode

Parameter Name: pg_mode, Type: enum, Level: C

PostgreSQL cluster mode, default value is pgsql, i.e., standard PostgreSQL cluster.

Available mode options include:

  • pgsql: Standard PostgreSQL cluster
  • citus: Citus distributed database cluster
  • mssql: Babelfish MSSQL wire protocol compatible kernel
  • mysql: OpenHalo/HaloDB MySQL wire protocol compatible kernel
  • ivory: IvorySQL Oracle compatible kernel
  • pgtde: Percona PostgreSQL with pg_tde
  • polar: PolarDB for PostgreSQL kernel
  • gpsql: Greenplum parallel database cluster (monitoring)
  • agens: AgensGraph graph database kernel
  • oriole: OrioleDB storage-engine kernel
  • pgedge: pgEdge distributed-replication kernel

pg_shard and pg_group default to pg_cluster and 0, respectively. When pg_mode is citus or gpsql and the sharded system contains multiple physical clusters, set both explicitly to define the horizontal-sharding identity.

In both cases, each PostgreSQL cluster is part of a larger business unit.

pg_cluster

Parameter Name: pg_cluster, Type: string, Level: C

PostgreSQL cluster name, required identity parameter, no default value.

The cluster name is used as the namespace for resources.

The current role validation accepts names matching ^[A-Za-z0-9-]+$ and rejects root. To keep DNS names, service names, and operation scripts consistent, lowercase names beginning with a letter and containing only lowercase letters, digits, and hyphens are still recommended.

pg_seq

Parameter Name: pg_seq, Type: int, Level: I

PostgreSQL instance sequence number, required identity parameter, no default value.

The sequence number of this instance, uniquely assigned within its cluster, typically using natural numbers starting from 0 or 1, usually not recycled or reused.

pg_role

Parameter Name: pg_role, Type: enum, Level: I

PostgreSQL instance role, required identity parameter with no default. Current validation accepts primary, replica, standby, offline, and delayed.

The commonly used service-membership labels are:

  • primary: Primary instance, there is one and only one in a cluster.
  • replica: Replica for serving online read-only traffic, may have slight replication delay under high load (10ms~100ms, 100KB).
  • offline: Offline replica for handling offline read-only traffic, such as analytics/ETL/personal queries.

standby and delayed are also valid inventory role values, but these strings alone do not create a standby cluster or delayed replication. Configure the topology with pg_upstream and pg_delay. Role-filtered HBA rules use these inventory labels directly.

pg_instances

Parameter Name: pg_instances, Type: dict, Level: I

Define multiple PostgreSQL instances on a single host using {port:ins_vars} format.

This parameter is reserved for multi-instance deployment on a single node. Pigsty has not yet implemented this feature and strongly recommends dedicated node deployment.

pg_upstream

Parameter Name: pg_upstream, Type: ip, Level: I

Upstream instance IP address for standby cluster or cascade replica.

Setting pg_upstream on the primary instance of a cluster indicates this cluster is a standby cluster, and this instance will act as a standby leader, receiving and applying changes from the upstream cluster.

Setting pg_upstream on a non-primary instance specifies a specific instance as the upstream for physical replication. If different from the primary instance IP address, this instance becomes a cascade replica. It is the user’s responsibility to ensure the upstream IP address is another instance in the same cluster.

pg_shard

Parameter Name: pg_shard, Type: string, Level: C

PostgreSQL horizontal shard name, defaulting to pg_cluster. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

When multiple standard PostgreSQL clusters serve the same business together in a horizontal sharding manner, Pigsty marks this group of clusters as a horizontal sharding cluster.

pg_shard is the shard group name. It is typically a prefix of pg_cluster.

For example, if we have a shard group pg-citus with 4 clusters, their identity parameters would be:

cls pg_shard: pg-citus
cls pg_group = 0:   pg-citus0
cls pg_group = 1:   pg-citus1
cls pg_group = 2:   pg-citus2
cls pg_group = 3:   pg-citus3

pg_group

Parameter Name: pg_group, Type: int, Level: C

PostgreSQL horizontal-sharding cluster index, defaulting to 0. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

This parameter is used in conjunction with pg_shard, typically using non-negative integers as index numbers.

gp_role

Parameter Name: gp_role, Type: enum, Level: C

Greenplum/Matrixdb role of the PostgreSQL cluster, can be master or segment.

  • master: Marks the postgres cluster as a greenplum master instance (coordinator node), this is the default value.
  • segment: Marks the postgres cluster as a greenplum segment cluster (data node).

This parameter is only used for Greenplum/MatrixDB databases (pg_mode is gpsql) and has no meaning for regular PostgreSQL clusters.

pg_exporters

Parameter Name: pg_exporters, Type: dict, Level: C

Additional exporter definitions for monitoring remote PostgreSQL instances, default value: {}

If you want to monitor remote PostgreSQL instances, define them in the pg_exporters parameter on the cluster where the monitoring system resides (Infra node), and use the pgsql-monitor.yml playbook to complete the deployment.

pg_exporters: # list all remote instances here, alloc a unique unused local port as k
    20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
    20004: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }
    20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.12 }
    20003: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.13 }

pg_offline_query

Parameter Name: pg_offline_query, Type: bool, Level: I

Set to true to mark this instance as eligible for offline queries. The default is false.

The flag adds the instance to the default offline service candidate set and activates HBA rules with role: offline on that instance. It does not grant connection access by itself; effective access still depends on generated HBA rules, database CONNECT privileges, and role attributes.

Instances with this flag have an effect similar to setting pg_role = offline for the instance, with the only difference being that offline instances by default do not serve replica service requests and exist as dedicated offline/analytics replica instances.

If no dedicated offline instance is available, enable the parameter on a regular replica. To restrict dbrole_offline to these instances, also set role: offline explicitly on the corresponding HBA rule.

PG_BUSINESS

Customize cluster templates: users, databases, services, and permission rules.

Users should pay close attention to this section of parameters, as this is where business declares its required database objects.

Default database users and their credentials. Their passwords must be changed in production.

# postgres business object definition, overwrite in group vars
pg_users: []                      # postgres business users
pg_databases: []                  # postgres business databases
pg_services: []                   # postgres business services
pg_hba_rules: []                  # business hba rules for postgres
pgb_hba_rules: []                 # business hba rules for pgbouncer
pg_crontab: []                    # crontab entries for postgres dbsu
# global credentials, overwrite in global vars
pg_dbsu_password: ''              # dbsu password, empty string means no dbsu password by default
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

pg_users

Parameter Name: pg_users, Type: user[], Level: C

PostgreSQL business user list, needs to be defined at the PG cluster level. Default value: [] empty list.

Each array element is a user/role definition, for example:

- name: dbuser_meta               # required, `name` is the only required field for user definition
  password: DBUser.Meta           # optional, password, can be scram-sha-256 hash string or plaintext
  login: true                     # optional, can login by default
  superuser: false                # optional, default false, is superuser?
  createdb: false                 # optional, default false, can create database?
  createrole: false               # optional, default false, can create role?
  inherit: true                   # optional, by default, can this role use inherited privileges?
  replication: false              # optional, default false, can this role do replication?
  bypassrls: false                # optional, default false, can this role bypass row-level security?
  pgbouncer: true                 # optional, default false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # optional, when this role expires, use YYYY-MM-DD format string to specify a specific date (lower priority than expire_in)
  comment: pigsty admin user      # optional, description and comment string for this user/role
  roles: [dbrole_admin]           # optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # optional, use `ALTER ROLE SET` for this role, configure role-level database parameters
  pool_mode: transaction          # optional, pgbouncer pool mode at user level, default transaction
  pool_connlimit: 100             # optional, user-level pool connection limit; omitted values inherit Pigsty's global default of 100
  search_path: public             # optional, key-value config parameter per postgresql docs (e.g., use pigsty as default search_path)

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).

pg_databases

Parameter Name: pg_databases, Type: database[], Level: C

PostgreSQL business database list, needs to be defined at the PG cluster level. Default value: [] empty list.

Each array element is a business database definition, for example:

- name: meta                      # required, `name` is the only required field for database definition
  baseline: cmdb.sql              # optional, database sql baseline file path (relative path in ansible search path, e.g., files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? default true
  schemas: [pigsty]               # optional, additional schemas to create, array of schema name strings
  extensions:                     # optional, additional extensions to install: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install extension into, or not (if not specified, installs to first schema in search_path)
    - { name: timescaledb }               # some extensions create and use fixed schemas, so no need to specify schema
  comment: pigsty meta database   # optional, description and comment for the database
  owner: postgres                 # optional, database owner, default is postgres
  template: template1             # optional, template to use, default is template1, target must be a template database
  encoding: UTF8                  # optional, database encoding, default UTF8 (must match template database)
  locale: C                       # optional, database locale setting, default C (must match template database)
  lc_collate: C                   # optional, database collate rule, default C (must match template database), no reason to change
  lc_ctype: C                     # optional, database ctype character set, default C (must match template database)
  tablespace: pg_default          # optional, default tablespace, default is 'pg_default'
  allowconn: true                 # optional, allow connections, default true. Explicitly set false to completely forbid connections
  revokeconn: false               # optional, revoke public connect privileges. default false, when true, CONNECT privilege revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasource? default true, explicitly false skips registration
  connlimit: -1                   # optional, database connection limit, default -1 means no limit, positive integer limits connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will authenticate using this user (useful when pgbouncer_auth_query enabled)
  pool_mode: transaction          # optional, database-level pgbouncer pooling mode, default transaction
  pool_size: 50                   # optional, database-level pgbouncer default pool size, default 50
  pool_reserve: 30                # optional, database-level pgbouncer pool reserve, default 30, max additional burst connections when default pool insufficient
  pool_size_min: 0                # optional, database-level pgbouncer pool minimum size, default 0
  pool_connlimit: 100             # optional, database-level max database connections, default 100

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

In each database definition object, only name is a required field, all other fields are optional.

pg_services

Parameter Name: pg_services, Type: service[], Level: C

PostgreSQL service list, needs to be defined at the PG cluster level. Default value: [], empty list.

Used to define additional services at the database cluster level. Each object in the array defines a service. A complete service definition example:

- name: standby                   # required, service name, final svc name will use `pg_cluster` as prefix, e.g., pg-meta-standby
  port: 5435                      # required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # optional, IP address to bind service, default is all IP addresses
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, service member selector (backup), service is handled by these instances when default selector instances are all down
  dest: default                   # optional, target port, default|postgres|pgbouncer|<port_number>, default is 'default', Default means use pg_default_service_dest value to decide
  check: /sync                    # optional, health check URL path, default is /, here uses Patroni API: /sync, only sync standby and primary return 200 health status
  maxconn: 5000                   # optional, max frontend connections allowed, default 5000
  balance: roundrobin             # optional, haproxy load balancing algorithm (default roundrobin, other option: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

Note that this parameter is used to add additional services at the cluster level. If you want to globally define services that all PostgreSQL databases should provide, use the pg_default_services parameter.

pg_hba_rules

Parameter Name: pg_hba_rules, Type: hba[], Level: C

Client IP whitelist/blacklist rules for database cluster/instance. Default: [] empty list.

Array of objects, each object represents a rule. HBA rule object definition:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5
  • title: Rule title name, rendered as comment in HBA file.
  • rules: Rule array, each element is a standard HBA rule string.
  • role: Rule application scope, which instance roles will enable this rule?
    • common: Applies to all instances
    • primary, replica, offline: Only applies to instances with specific pg_role.
    • Special case: role: 'offline' rules apply to instances with pg_role : offline, and also to instances with pg_offline_query flag.

In addition to the native HBA rule definition above, Pigsty also provides a more convenient alias form:

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access

pg_default_hba_rules is similar to this parameter, but it’s used to define global HBA rules, while this parameter is typically used to customize HBA rules for specific clusters/instances.

pgb_hba_rules

Parameter Name: pgb_hba_rules, Type: hba[], Level: C

Pgbouncer business HBA rules, default value: [], empty array.

This parameter is similar to pg_hba_rules, both are arrays of hba rule objects, the difference is that this parameter is for Pgbouncer.

pgb_default_hba_rules is similar to this parameter, but it’s used to define global connection pool HBA rules, while this parameter is typically used to customize HBA rules for specific connection pool clusters/instances.

pg_crontab

Parameter Name: pg_crontab, Type: string[], Level: C

Cron job list for the PostgreSQL database superuser (dbsu, default postgres), default value: [] empty array.

Each array element is a crontab entry line, using standard user crontab format: minute hour day month weekday command (no need to specify username).

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'      # Full backup at 1 AM daily
  - '00 13 * * * /pg/bin/pg-backup'           # Incremental backup at 1 PM daily

This parameter writes cron jobs to the postgres user’s personal crontab file:

  • EL systems: /var/spool/cron/postgres
  • Debian systems: /var/spool/cron/crontabs/postgres

Note: This parameter replaces the old practice of configuring postgres user tasks in node_crontab. Because node_crontab is written to /etc/crontab during NODE initialization, the postgres user may not exist yet, causing cron errors.

pg_replication_username

Parameter Name: pg_replication_username, Type: username, Level: G

PostgreSQL physical replication username, default is replicator, not recommended to change this parameter.

pg_replication_password

Parameter Name: pg_replication_password, Type: password, Level: G

PostgreSQL physical replication user password, default value: DBUser.Replicator.

Warning

Change this password in production environments.

pg_admin_username

Parameter Name: pg_admin_username, Type: username, Level: G

PostgreSQL / Pgbouncer admin name, default: dbuser_dba.

This is the globally used database administrator with database Superuser privileges and connection pool traffic management permissions. Please control its usage scope.

pg_admin_password

Parameter Name: pg_admin_password, Type: password, Level: G

PostgreSQL / Pgbouncer admin password, default: DBUser.DBA.

Warning

Change this password in production environments.

pg_monitor_username

Parameter Name: pg_monitor_username, Type: username, Level: G

PostgreSQL/Pgbouncer monitor username, default: dbuser_monitor.

This is a database/connection pool user for monitoring, not recommended to change this username.

However, if your existing database uses a different monitor user, you can use this parameter to specify the monitor username when defining monitoring targets.

pg_monitor_password

Parameter Name: pg_monitor_password, Type: password, Level: G

Password used by PostgreSQL/Pgbouncer monitor user, default: DBUser.Monitor.

Try to avoid using characters like @:/ that can be confused with URL delimiters in passwords to reduce unnecessary trouble.

Warning

Change this password in production environments.

pg_dbsu_password

Parameter Name: pg_dbsu_password, Type: password, Level: G/C

PostgreSQL pg_dbsu superuser password, default is empty string, meaning no password is set.

We don’t recommend configuring password login for dbsu as it increases the attack surface. The exception is: pg_mode = citus, in which case you need to configure a password for each shard cluster’s dbsu to allow connections within the shard cluster.

PG_INSTALL

This section is responsible for installing PostgreSQL and its extensions. If you want to install different major versions and extension plugins, just modify pg_version and pg_extensions. Note that not all extensions are available for all major versions.

pg_dbsu: postgres                 # os dbsu name, default is postgres, better not change it
pg_dbsu_uid: 26                   # os dbsu uid and gid, default is 26, for default postgres user and group
pg_dbsu_sudo: limit               # dbsu sudo privilege, none,limit,all,nopass. default is limit
pg_dbsu_home: /var/lib/pgsql      # postgresql home directory, default is `/var/lib/pgsql`
pg_dbsu_ssh_exchange: true        # exchange postgres dbsu ssh key among same pgsql cluster
pg_version: 18                    # postgres major version to be installed, default is 18
pg_bin_dir: /usr/pgsql/bin        # postgres binary dir, default is `/usr/pgsql/bin`
pg_log_dir: /pg/log/postgres      # postgres log dir, default is `/pg/log/postgres`
pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common
pg_extensions: []                 # pg extensions to be installed, alias can be used

pg_dbsu

Parameter Name: pg_dbsu, Type: username, Level: C

OS dbsu username used by PostgreSQL, default is postgres, changing this username is not recommended.

However, in certain situations, you may need a username different from postgres, for example, when installing and configuring Greenplum / MatrixDB, you need to use gpadmin / mxadmin as the corresponding OS superuser.

pg_dbsu_uid

Parameter Name: pg_dbsu_uid, Type: int, Level: C

OS database superuser uid and gid, 26 is the default postgres user UID/GID from PGDG RPM.

For Debian/Ubuntu systems, there is no default value, and user 26 is often taken. Therefore, when Pigsty detects the installation environment is Debian-based and uid is 26, it will automatically use the replacement pg_dbsu_uid = 543.

pg_dbsu_sudo

Parameter Name: pg_dbsu_sudo, Type: enum, Level: C

Database superuser sudo privilege, can be none, limit, all, or nopass. Default is limit

  • none: No sudo privilege

  • limit: Limited sudo privilege for executing systemctl commands for database-related components (default option).

  • all: Full sudo privilege, requires password.

  • nopass: Full sudo privilege without password (not recommended).

  • Default value is limit, only allows executing sudo systemctl <start|stop|reload> <postgres|patroni|pgbouncer|...>.

pg_dbsu_home

Parameter Name: pg_dbsu_home, Type: path, Level: C

PostgreSQL home directory, default is /var/lib/pgsql, consistent with official pgdg RPM.

pg_dbsu_ssh_exchange

Parameter Name: pg_dbsu_ssh_exchange, Type: bool, Level: C

Whether to exchange OS dbsu ssh keys within the same PostgreSQL cluster?

Default is true, meaning database superusers in the same cluster can ssh to each other.

The exchange set comes from pg_cluster_members, which matches actual hosts with the same pg_cluster across the current inventory; it does not require an Ansible group named after the cluster. The execution-time -l limit still restricts playbook targets, so ensure it covers every member that needs configuration.

pg_version

Parameter Name: pg_version, Type: enum, Level: C

PostgreSQL major version to install, default is 18.

Note that PostgreSQL physical streaming replication cannot cross major versions, so it’s best not to configure this at the instance level.

You can use parameters in pg_packages and pg_extensions to install different packages and extensions for specific PG major versions.

pg_bin_dir

Parameter Name: pg_bin_dir, Type: path, Level: C

PostgreSQL binary directory, default is /usr/pgsql/bin.

The default value is a symlink manually created during installation, pointing to the specific installed Postgres version directory.

For example /usr/pgsql -> /usr/pgsql-15. On Ubuntu/Debian it points to /usr/lib/postgresql/15/bin.

For more details, see PGSQL File Structure.

pg_log_dir

Parameter Name: pg_log_dir, Type: path, Level: C

PostgreSQL log directory, default: /pg/log/postgres. The Vector log agent uses this variable to collect PostgreSQL logs.

Note that if the log directory pg_log_dir is prefixed with the data directory pg_data, it won’t be explicitly created (created automatically during data directory initialization).

pg_packages

Parameter Name: pg_packages, Type: string[], Level: C

PostgreSQL packages to install (RPM/DEB), this is an array of package names where elements can be space or comma-separated package aliases.

Pigsty v4 converges the default value to two aliases:

pg_packages:
  - pgsql-main pgsql-common
  • pgsql-main: Maps to PostgreSQL kernel, client, PL languages, and core extensions like pg_repack, wal2json, pgvector on the current platform.
  • pgsql-common: Maps to companion components required for running the database, such as Patroni, Pgbouncer, pgBackRest, pg_exporter, vip-manager, and other daemons.

Alias definitions can be found in pg_package_map under roles/node_id/vars/. Pigsty first resolves aliases based on OS and architecture, then replaces $v/${pg_version} with the actual major version pg_version, and finally installs the real packages. This shields package name differences between distributions.

If additional packages are needed (e.g., specific FDW or extensions), you can append aliases or real package names directly to pg_packages. But remember to keep pgsql-main pgsql-common, otherwise core components will be missing.

pg_extensions

Parameter Name: pg_extensions, Type: string[], Level: G/C

PostgreSQL extension packages to install (RPM/DEB), this is an array of extension package names or aliases.

Starting from v4, the default value is an empty list []. Pigsty no longer forces installation of large extensions, users can choose as needed to avoid extra disk and dependency usage.

To install extensions, fill in like this:

pg_extensions:
  - postgis timescaledb pgvector
  - pgsql-fdw     # use alias to install common FDWs at once

pg_package_map provides aliases that hide package-name differences between distributions. The following examples use single-extension aliases and category groups from the current v4.5 EL9 map; availability can differ by platform and PostgreSQL major version:

pg_extensions: # extensions to be installed on this cluster
  - timescaledb postgis pgvector pg_search pg_duckdb pg_repack wal2json
  # You can also select current category groups; install only the groups you need
  # - pgsql-time pgsql-gis pgsql-rag pgsql-fts pgsql-olap
  # - pgsql-feat pgsql-lang pgsql-type pgsql-util pgsql-func
  # - pgsql-admin pgsql-stat pgsql-sec pgsql-fdw pgsql-sim pgsql-etl

For the exact mapping, consult roles/node_id/vars/<os>.<arch>.yml for the target platform and the current extension catalog. pg_analytics and spat were removed from the v4.5 catalog and current mainstream-platform maps; do not copy them from older examples.

PG_BOOTSTRAP

Bootstrap PostgreSQL cluster with Patroni and set up 1:1 corresponding Pgbouncer connection pool.

It also initializes the database cluster with default roles, users, privileges, schemas, and extensions defined in PG_PROVISION.

The following parameters configure the PGSQL bootstrap stage. The internal variable pg_data always represents the /pg/data symlink and must not be overridden in inventory. Configure pg_fs_main to change the physical location of the primary data directory.

pg_fs_main: /data/postgres        # postgres main data directory, `/data/postgres` by default
pg_fs_backup: /data/backups       # postgres backup data directory, `/data/backups` by default
pg_storage_type: SSD              # storage type for pg main data, SSD,HDD, SSD by default
pg_dummy_filesize: 64MiB          # size of `/pg/dummy`, hold 64MB disk space for emergency use
pg_listen: '0.0.0.0'              # postgres/pgbouncer listen addresses, comma separated list
pg_port: 5432                     # postgres listen port, 5432 by default
pg_localhost: /var/run/postgresql # postgres unix socket dir for localhost connection
patroni_enabled: true             # if disabled, no postgres cluster will be created during init
patroni_mode: default             # patroni working mode: default,pause,remove
pg_namespace: /pg                 # top level key namespace in etcd, used by patroni & vip
patroni_port: 8008                # patroni listen port, 8008 by default
patroni_log_dir: /pg/log/patroni  # patroni log dir, `/pg/log/patroni` by default
patroni_ssl_enabled: false        # secure patroni RestAPI communications with SSL?
patroni_watchdog_mode: off        # patroni watchdog mode: automatic,required,off. off by default
patroni_username: postgres        # patroni restapi username, `postgres` by default
patroni_password: Patroni.API     # patroni restapi password, `Patroni.API` by default
pg_etcd_password: ''              # etcd password for this pg cluster, '' to use pg_cluster
pg_primary_db: postgres           # primary database name, used by citus,etc... ,postgres by default
pg_parameters: {}                 # extra parameters in postgresql.auto.conf
pg_files: []                      # extra files to be copied to postgres data directory (e.g. license)
pg_conf: oltp.yml                 # config template: oltp,olap,crit,tiny. `oltp.yml` by default
pg_max_conn: auto                 # postgres max connections, `auto` will use recommended value
pg_shared_buffer_ratio: 0.25      # postgres shared buffers ratio, 0.25 by default, 0.1~0.4
pg_io_method: worker              # io method for postgres, auto,sync,worker,io_uring, worker by default
pg_rto: norm                      # shared RTO mode: fast,norm,safe,wide
pg_rpo: 1048576                   # sampled failover-candidate lag threshold in bytes, `1MiB` by default
pg_libs: 'pg_stat_statements, auto_explain'  # preloaded libraries, `pg_stat_statements,auto_explain` by default
pg_delay: 0                       # replication apply delay for standby cluster leader
pg_checksum: true                 # enable data checksum for postgres cluster?
pg_pwd_enc: scram-sha-256         # passwords encryption algorithm: fixed to scram-sha-256
pg_encoding: UTF8                 # database cluster encoding, `UTF8` by default
pg_locale: C                      # database cluster local, `C` by default
pg_lc_collate: C                  # database cluster collate, `C` by default
pg_lc_ctype: C                    # database character type, `C` by default
#pgsodium_key: ""                 # pgsodium key, 64 hex digit, default to sha256(pg_cluster)
#pgsodium_getkey_script: ""       # pgsodium getkey script path, pgsodium_getkey by default

pg_data

Internal Variable: pg_data, Type: path

pg_data is an internal Pigsty variable, not a user configuration parameter. It always represents the PostgreSQL data-directory symlink at /pg/data.

Patroni templates, maintenance scripts, and cleanup workflows all rely on this symlink. Do not override or modify it in pigsty.yml. Configure pg_fs_main to change the underlying physical data location. See PGSQL File Structure for details.

pg_fs_main

Parameter Name: pg_fs_main, Type: path, Level: C

Mount point/file system path for PostgreSQL main data disk, default is /data/postgres.

Default value: /data/postgres, which will be used directly as the parent directory of PostgreSQL main data directory.

NVME SSD is recommended for PostgreSQL main data storage. Pigsty is optimized for SSD storage by default, but also supports HDD.

You can change pg_storage_type to HDD for HDD storage optimization.

pg_fs_backup

Parameter Name: pg_fs_backup, Type: path, Level: C

Mount point/file system path for PostgreSQL backup data disk, default is /data/backups.

If you’re using the default pgbackrest_method = local, it’s recommended to use a separate disk for backup storage.

The backup disk should be large enough to hold all backups, at least sufficient for 3 base backups + 2 days of WAL archives. Usually capacity isn’t a big issue since you can use cheap large HDDs as backup disks.

It’s recommended to use a separate disk for backup storage, otherwise Pigsty will fall back to the main data disk and consume main data disk capacity and IO.

pg_storage_type

Parameter Name: pg_storage_type, Type: enum, Level: C

Type of PostgreSQL data storage media: SSD or HDD, default is SSD.

Default value: SSD, which affects some tuning parameters like random_page_cost and effective_io_concurrency.

pg_dummy_filesize

Parameter Name: pg_dummy_filesize, Type: size, Level: C

Size of /pg/dummy, default is 64MiB, 64MB disk space for emergency use.

When disk is full, deleting the placeholder file can free some space for emergency use. Recommend at least 8GiB for production.

pg_listen

Parameter Name: pg_listen, Type: ip, Level: C

PostgreSQL / Pgbouncer listen address, default is 0.0.0.0 (all ipv4 addresses).

You can use placeholders in this variable, for example: '${ip},${lo}' or '${ip},${vip},${lo}':

  • ${ip}: Translates to inventory_hostname, which is the primary internal IP address defined in the inventory.
  • ${vip}: If pg_vip_enabled is enabled, will use the host part of pg_vip_address.
  • ${lo}: Will be replaced with 127.0.0.1

For production environments with high security requirements, it’s recommended to restrict listen IP addresses.

pg_port

Parameter Name: pg_port, Type: port, Level: C

Port that PostgreSQL server listens on, default is 5432.

pg_localhost

Parameter Name: pg_localhost, Type: path, Level: C

Unix socket directory for localhost PostgreSQL connection, default is /var/run/postgresql.

Unix socket directory for PostgreSQL and Pgbouncer local connections. pg_exporter and patroni will preferentially use Unix sockets to access PostgreSQL.

pg_namespace

Parameter Name: pg_namespace, Type: path, Level: C

Top-level namespace used in etcd, used by patroni and vip-manager, default is: /pg, not recommended to change.

patroni_enabled

Parameter Name: patroni_enabled, Type: bool, Level: C

Enable Patroni? Default is: true.

If disabled, no Postgres cluster will be created during initialization. Pigsty will skip the task of starting patroni, which can be used when trying to add some components to existing postgres instances.

patroni_mode

Parameter Name: patroni_mode, Type: enum, Level: C

Patroni working mode: default, pause, remove. Default: default.

  • default: Normal use of Patroni to bootstrap PostgreSQL cluster
  • pause: Similar to default, but enters maintenance mode after bootstrap
  • remove: Use Patroni to initialize cluster, then remove Patroni and use raw PostgreSQL.

patroni_port

Parameter Name: patroni_port, Type: port, Level: C

Patroni listen port, default is 8008, not recommended to change.

Patroni API server listens on this port for health checks and API requests.

patroni_log_dir

Parameter Name: patroni_log_dir, Type: path, Level: C

Patroni log directory, default is /pg/log/patroni, collected by Vector log agent.

patroni_ssl_enabled

Parameter Name: patroni_ssl_enabled, Type: bool, Level: G

Secure patroni RestAPI communications with SSL? Default is false.

This parameter is a global flag that can only be set before deployment. Because if SSL is enabled for patroni, you will have to use HTTPS instead of HTTP for health checks, fetching metrics, and calling APIs.

patroni_watchdog_mode

Parameter Name: patroni_watchdog_mode, Type: string, Level: C

Patroni watchdog mode: automatic, required, off, default is off.

In case of primary failure, Patroni can use watchdog to force shutdown old primary node to avoid split-brain.

  • off: Don’t use watchdog. No fencing at all (default behavior)
  • automatic: Enable watchdog if kernel has softdog module enabled and watchdog belongs to dbsu.
  • required: Force enable watchdog, refuse to start Patroni/PostgreSQL if softdog unavailable.

Default is off. You should not enable watchdog on Infra nodes. Critical systems where data consistency takes priority over availability, especially business clusters involving money, can consider enabling this option.

Note that if all your access traffic uses HAproxy health check service access, there is normally no split-brain risk.

patroni_username

Parameter Name: patroni_username, Type: username, Level: C

Patroni REST API username, default is postgres, used with patroni_password.

Patroni’s dangerous REST APIs (like restarting cluster) are protected by additional username/password. See Configure Cluster and Patroni RESTAPI for details.

patroni_password

Parameter Name: patroni_password, Type: password, Level: C

Patroni REST API password, default is Patroni.API.

Warning

Change this parameter in production environments.

pg_primary_db

Parameter Name: pg_primary_db, Type: string, Level: C

Specify the primary database name in the cluster, used for citus and other business databases, default is postgres.

For example, when using Patroni to manage HA Citus clusters, you must choose a “primary database”.

Additionally, the database name specified here will be displayed in the printed connection string after PGSQL module installation is complete.

pg_parameters

Parameter Name: pg_parameters, Type: dict, Level: G/C/I

Used to specify and manage configuration parameters in postgresql.auto.conf.

After all cluster instances are initialized, the pg_param task will write the key/value pairs from this dictionary sequentially to /pg/data/postgresql.auto.conf.

Note

Do not manually modify this configuration file or change cluster parameters with ALTER SYSTEM; the next configuration sync will overwrite those changes.

This variable has higher priority than cluster configuration in Patroni / DCS (i.e., higher priority than cluster configuration edited by Patroni edit-config), so it can typically be used to override cluster default parameters at instance level.

When your cluster members have different specifications (not recommended!), you can use this parameter for fine-grained configuration management of each instance.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_parameters: { shared_buffers: '5GB' } }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_parameters: { shared_buffers: '4GB' } }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_parameters: { shared_buffers: '3GB' } }

Note that some important cluster parameters (with requirements on primary/replica parameter values) are managed directly by Patroni via command line arguments, have highest priority, and cannot be overridden this way. For these parameters, you must use Patroni edit-config for management and configuration.

PostgreSQL parameters that must be consistent on primary and replicas (inconsistency will cause replica to fail to start!):

  • wal_level
  • max_connections
  • max_locks_per_transaction
  • max_worker_processes
  • max_prepared_transactions
  • track_commit_timestamp

Parameters that should preferably be consistent on primary and replicas (considering possibility of failover):

  • listen_addresses
  • port
  • cluster_name
  • hot_standby
  • wal_log_hints
  • max_wal_senders
  • max_replication_slots
  • wal_keep_segments
  • wal_keep_size

You can set non-existent parameters (e.g., GUCs from extensions, thus configuring “not yet existing” parameters that ALTER SYSTEM cannot modify), but modifying existing configuration to illegal values may cause PostgreSQL to fail to start, configure with caution!

pg_files

Parameter Name: pg_files, Type: path[], Level: C

Used to specify a list of files to be copied to the PGDATA directory, default is empty array: []

Files specified in this parameter will be copied to the {{ pg_data }} directory, mainly used to distribute license files required by special commercial PostgreSQL kernels.

Currently only PolarDB (Oracle compatible) kernel requires license files. For example, you can place the license.lic file in the files/ directory and specify in pg_files:

pg_files: [ license.lic ]

pg_conf

Parameter Name: pg_conf, Type: enum, Level: C

Configuration template: {oltp,olap,crit,tiny}.yml, default is oltp.yml.

  • tiny.yml: Optimized for small nodes, VMs, small demos (1-8 cores, 1-16GB)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

Default is oltp.yml, but the configure script will set this to tiny.yml when current node is a small node.

You can have your own templates, just place them under templates/<mode>.yml and set this value to the template name to use.

pg_max_conn

Parameter Name: pg_max_conn, Type: int, Level: C

PostgreSQL server max connections. You can choose a value between 50 and 5000, or use auto for recommended value.

Default is auto, which sets max connections based on pg_conf and pg_default_service_dest.

  • tiny: 100
  • olap: 200
  • oltp: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000
  • crit: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000

Not recommended to set this value above 5000, otherwise you’ll need to manually increase haproxy service connection limits.

Pgbouncer’s transaction pool can mitigate excessive OLTP connection issues, so setting a large connection count is not recommended by default.

For OLAP scenarios, change pg_default_service_dest to postgres to bypass connection pooling.

pg_shared_buffer_ratio

Parameter Name: pg_shared_buffer_ratio, Type: float, Level: C

Postgres shared buffer memory ratio, default is 0.25, normal range is 0.1~0.4.

Default: 0.25, meaning 25% of node memory will be used as PostgreSQL’s shared buffer. If you want to enable huge pages for PostgreSQL, this value should be appropriately smaller than node_hugepage_ratio.

Setting this value above 0.4 (40%) is usually not a good idea, but may be useful in extreme cases.

Note that shared buffers are only part of PostgreSQL’s shared memory. To calculate total shared memory, use show shared_memory_size_in_huge_pages;.

pg_rto

Parameter Name: pg_rto, Type: enum, Level: C

Recovery Time Objective (RTO) mode controlling Patroni and HAProxy timeout parameters. The default is norm.

Pigsty provides four presets tuned for different network conditions and deployment scenarios:

Mode Scenario Network conditions Source target RTO Patroni TTL False-failover risk
fast Same rack or switch < 1 ms, highly stable < 30s 20s Higher
norm Same datacenter 1-5 ms, normal < 45s 30s Medium
safe Cross-datacenter 10-50 ms < 90s 60s Lower
wide Cross-region/continent 100-200 ms, public WAN < 150s 120s Lowest

A shorter RTO speeds recovery but increases the risk that network jitter is mistaken for failure. Choose a mode that matches your network conditions. See RTO Trade-offs for details.

The current template recognizes only these four string keys. Other values, including numbers, fall back to norm; they are not interpreted as seconds. To define another timeout combination, extend pg_rto_plan with a new key and use that key as pg_rto.

pg_rto: norm   # default, suitable for a single datacenter
pg_rto: safe   # recommended for cross-datacenter deployments
# pg_rto: 30   # invalid key; current templates fall back to norm

pg_rto_plan

Parameter Name: pg_rto_plan, Type: dict, Level: G

Dictionary of RTO presets defining Patroni HA and HAProxy health-check timeouts. The defaults contain four modes:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Each mode is an array of ten values controlling Patroni and HAProxy together:

Index Name Component Description
0 ttl Patroni Primary-lock TTL in seconds
1 loop_wait Patroni Main-loop sleep interval
2 retry_timeout Patroni DCS/PostgreSQL retry timeout
3 primary_start_timeout Patroni Time allowed for primary recovery
4 safety_margin Patroni Watchdog safety margin
5 inter HAProxy Health-check interval
6 fastinter HAProxy Fast interval during state transitions
7 downinter HAProxy Check interval while a server is down
8 rise HAProxy Consecutive successes required for UP
9 fall HAProxy Consecutive failures required for DOWN

Override this dictionary to customize an existing mode or add a new one:

pg_rto_plan:
  ultra: [ 10, 2, 3, 8, 2, '0.5s', '0.25s', '0.5s', 2, 2 ]  # aggressive; low-latency networks only

Caution: inappropriate timeout combinations can make the cluster unstable or cause frequent false failovers.

pg_rpo

Parameter Name: pg_rpo, Type: int, Level: C

Failover-candidate lag threshold in bytes, default: 1048576 (1 MiB). It is written to Patroni’s maximum_lag_on_failover and determines whether a replica is eligible for failover; it is not a hard upper bound on actual data loss. With asynchronous replication, worst-case loss also depends on write rate and when Patroni sampled the primary’s WAL position.

When the primary goes down and all replicas are lagging, you must make a difficult choice, trade-off between availability and consistency:

  • Promote a lagging replica and restore service as soon as possible, accepting possible data loss.
  • Wait for primary to come back online (may never happen), or manual intervention to avoid any data loss.

You can use the crit.yml conf template to ensure no data loss during failover, but this sacrifices some performance.

pg_libs

Parameter Name: pg_libs, Type: string, Level: C

Preloaded dynamic shared libraries, default is pg_stat_statements,auto_explain, two PostgreSQL built-in extensions that are strongly recommended to enable.

For existing clusters, you can directly configure cluster shared_preload_libraries parameter and apply.

If you want to use TimescaleDB or Citus extensions, you need to add timescaledb or citus to this list. timescaledb and citus should be placed at the front of this list, for example:

citus,timescaledb,pg_stat_statements,auto_explain

Other extensions requiring dynamic loading can also be added to this list, such as pg_cron, pgml, etc. Typically citus and timescaledb have highest priority and should be added to the front of the list.

pg_delay

Parameter Name: pg_delay, Type: interval, Level: I

Delayed standby replication delay, default: 0.

If this value is set to a positive value, the standby cluster leader will be delayed by this time before applying WAL changes. Setting to 1h means data in this cluster will always lag the original cluster by one hour.

See Delayed Standby Cluster for details.

pg_checksum

Parameter Name: pg_checksum, Type: bool, Level: C

Enable data checksum for PostgreSQL cluster? Default is true, enabled.

This parameter can only be set before PGSQL deployment (but you can enable it manually later).

Data checksums help detect disk corruption and hardware failures. This feature is enabled by default since Pigsty v3.5 to ensure data integrity.

pg_pwd_enc

Parameter Name: pg_pwd_enc, Type: enum, Level: C

Password encryption algorithm, fixed to scram-sha-256 since Pigsty v4.

All new users will use SCRAM credentials. md5 has been deprecated. For compatibility with old clients, upgrade to SCRAM in business connection pools or client drivers.

pg_encoding

Parameter Name: pg_encoding, Type: enum, Level: C

Database cluster encoding, default is UTF8.

Using other non-UTF8 encodings is not recommended.

pg_locale

Parameter Name: pg_locale, Type: enum, Level: C

Database cluster locale, default is C.

This parameter controls the database’s default Locale setting, affecting collation, character classification, and other behaviors. Using C or POSIX provides best performance and predictable sorting behavior.

If you need specific language localization support, you can set it to the corresponding Locale, such as en_US.UTF-8 or zh_CN.UTF-8. Note that Locale settings affect index sort order, so they cannot be changed after cluster initialization.

pg_lc_collate

Parameter Name: pg_lc_collate, Type: enum, Level: C

Database cluster collation, default is C.

Unless you know what you’re doing, modifying cluster-level collation settings is not recommended.

pg_lc_ctype

Parameter Name: pg_lc_ctype, Type: enum, Level: C

Database character set CTYPE, default is C.

Starting from Pigsty v3.5, to be consistent with pg_lc_collate, the default value changed to C.

pg_io_method

Parameter Name: pg_io_method, Type: enum, Level: C

PostgreSQL IO method, default is worker. Available options include:

  • auto: Automatically select based on operating system, uses io_uring on Debian-based systems or EL 10+, otherwise uses worker
  • sync: Use traditional synchronous IO method
  • worker: Use background worker processes to handle IO (default option)
  • io_uring: Use Linux’s io_uring asynchronous IO interface

Current tuning templates write this parameter only for PostgreSQL 18 and later, where it controls the asynchronous I/O execution method.

  • PostgreSQL 18’s actual GUC values are worker, io_uring, and sync; auto is Pigsty template-selection logic, not a PostgreSQL enum value.
  • PostgreSQL 18 defaults to worker, which uses background processes for asynchronous I/O.
  • If you’re using Debian 12/Ubuntu 22+ or EL 10+ systems and want optimal IO performance, consider setting this to io_uring.

Note that setting this value on systems that don’t support io_uring may cause PostgreSQL startup to fail, so auto or worker are safer choices.

pg_etcd_password

Parameter Name: pg_etcd_password, Type: password, Level: C

The password used by this PostgreSQL cluster in etcd, default is empty string ''.

If set to empty string, the pg_cluster parameter value will be used as the password (for Citus clusters, the pg_shard parameter value is used).

This password is used for authentication when Patroni connects to etcd and when vip-manager accesses etcd.

pgsodium_key

Parameter Name: pgsodium_key, Type: string, Level: C

The encryption master key for the pgsodium extension, consisting of 64 hexadecimal digits.

This parameter is not set by default. If not specified, Pigsty will automatically generate a deterministic key using the value of sha256(pg_cluster).

pgsodium is a PostgreSQL extension based on libsodium that provides encryption functions and transparent column encryption capabilities. If you need to use pgsodium’s encryption features, it’s recommended to explicitly specify a secure random key and keep it safe.

Example command to generate a random key:

openssl rand -hex 32   # Generate 64-digit hexadecimal key

pgsodium_getkey_script

Parameter Name: pgsodium_getkey_script, Type: path, Level: C

Path to the pgsodium key retrieval script, default uses the pgsodium_getkey script from Pigsty templates.

This script is used to retrieve pgsodium’s master key when PostgreSQL starts. The default script reads the key from environment variables or configuration files.

If you have custom key management requirements (such as using HashiCorp Vault, AWS KMS, etc.), you can provide a custom script path.

PG_PROVISION

If PG_BOOTSTRAP is about creating a new cluster, then PG_PROVISION is about creating default objects in the cluster, including:

pg_provision: true                # provision postgres cluster after bootstrap
pg_init: pg-init                  # init script for cluster template, default is `pg-init`
pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin
pg_default_schemas: [ monitor ]   # default schemas
pg_default_extensions:            # default extensions
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }
pg_reload: true                   # reload config after HBA changes?
pg_default_hba_rules:             # postgres default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}
pgb_default_hba_rules:            # pgbouncer default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pg_provision

Parameter Name: pg_provision, Type: bool, Level: C

Complete the PostgreSQL cluster provisioning work defined in this section after the cluster is bootstrapped. Default value is true.

If disabled, the PostgreSQL cluster will not be provisioned. For some special “PostgreSQL” clusters, such as Greenplum, you can disable this option to skip the provisioning phase.

pg_init

Parameter Name: pg_init, Type: string, Level: G/C

Location of the shell script for initializing database templates, default is pg-init. This script is copied to /pg/bin/pg-init and then executed.

This script is located at roles/pgsql/templates/pg-init

You can add your own logic to this script, or provide a new script in the templates/ directory and set pg_init to the new script name. When using a custom script, please preserve the existing initialization logic.

pg_default_roles

Parameter Name: pg_default_roles, Type: role[], Level: G/C

Default roles and users in Postgres cluster.

Pigsty has a built-in role system. See PGSQL Access Control: Role System for details.

pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly]               ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite]  ,comment: role for object creation }
  - { name: postgres     ,superuser: true                                          ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly]   ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
  - { name: dbuser_monitor   ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

pg_default_privileges

Parameter Name: pg_default_privileges, Type: string[], Level: G/C

Default privileges (DEFAULT PRIVILEGE) settings in each database:

pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Pigsty provides matching default privileges for the built-in role system. See PGSQL Access Control: Default Privileges for details.

pg_default_schemas

Parameter Name: pg_default_schemas, Type: string[], Level: G/C

Default schemas to create, default value is: [ monitor ]. This will create a monitor schema on all databases for placing various monitoring extensions, tables, views, and functions.

pg_default_extensions

Parameter Name: pg_default_extensions, Type: extension[], Level: G/C

List of extensions to be created and enabled by default in all databases, default value:

pg_default_extensions: # default extensions to be created
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }

The only third-party extension is pg_repack, which is important for database maintenance. All other extensions are built-in PostgreSQL Contrib extensions.

Monitoring-related extensions are installed in the monitor schema by default, which is created by pg_default_schemas.

pg_reload

Parameter Name: pg_reload, Type: bool, Level: A

Reload PostgreSQL after HBA changes, default value is true.

Set it to false to disable automatic configuration reload when you want to check before applying HBA changes.

pg_default_hba_rules

Parameter Name: pg_default_hba_rules, Type: hba[], Level: G/C

PostgreSQL host-based authentication rules, global default rules definition. Default value is:

pg_default_hba_rules:             # postgres default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

The defaults target ordinary deployments on a trusted intranet; they are not a hardened baseline for public or regulated environments. The default +dbrole_offline rule has no role field and therefore applies to every instance. For instance isolation, copy the complete default list and set that rule to role: offline. See Authentication and Access Control: Offline Role and Instance Isolation.

This parameter is an array of HBA rule objects, identical in format to pg_hba_rules. It’s recommended to configure unified pg_default_hba_rules globally, and use pg_hba_rules for additional customization on specific clusters. Rules from both parameters are applied sequentially, with the latter having higher priority.

pgb_default_hba_rules

Parameter Name: pgb_default_hba_rules, Type: hba[], Level: G/C

Pgbouncer default host-based authentication rules, array of HBA rule objects.

Default value provides a fair security level for common scenarios. Check PGSQL Authentication for details.

pgb_default_hba_rules:            # pgbouncer default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

The default Pgbouncer HBA rules are simple:

  1. Allow login from localhost with password
  2. Allow login from intranet with password

Users can customize according to their own needs.

This parameter is identical in format to pgb_hba_rules. It’s recommended to configure unified pgb_default_hba_rules globally, and use pgb_hba_rules for additional customization on specific clusters. Rules from both parameters are applied sequentially, with the latter having higher priority.

PG_BACKUP

This section defines variables for pgBackRest, which is used for PGSQL Point-in-Time Recovery (PITR).

Check PGSQL Backup & PITR for detailed information.

pgbackrest_enabled: true          # enable pgBackRest on pgsql host?
pgbackrest_log_dir: /pg/log/pgbackrest # pgbackrest log dir, default is `/pg/log/pgbackrest`
pgbackrest_method: local          # pgbackrest repo method: local, minio, [user defined...]
pgbackrest_init_backup: true      # perform a full backup immediately after pgbackrest init?
pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

pgbackrest_enabled

Parameter Name: pgbackrest_enabled, Type: bool, Level: C

Enable pgBackRest on PGSQL nodes? Default value is: true

When enabled, every node receives pgBackRest configuration. With the local filesystem repository (local), each member creates its own local stanza. Initial and scheduled backups run only on the current primary; pg-backup exits after its role check on a replica. A non-local shared stanza is initialized only on a primary without pg_upstream.

pgbackrest_log_dir

Parameter Name: pgbackrest_log_dir, Type: path, Level: C

pgBackRest log directory, default is /pg/log/pgbackrest. The Vector log agent references this parameter for log collection.

pgbackrest_method

Parameter Name: pgbackrest_method, Type: enum, Level: C

pgBackRest repository method: default options are local, minio, or other user-defined methods, default is local.

This parameter determines which repository to use for pgBackRest. All available repository methods are defined in pgbackrest_repo.

Pigsty uses the local backup repository by default, which creates a backup repository in the /pg/backup directory on the primary instance. The underlying storage path is specified by pg_fs_backup.

pgbackrest_init_backup

Parameter Name: pgbackrest_init_backup, Type: bool, Level: C

Perform a full backup immediately after pgBackRest initialization completes? Default is true.

The task attempts this only on the cluster primary when pg_upstream is not defined. It uses ignore_errors for backup failures, so enabling the parameter does not guarantee that a base backup exists. /etc/pgbackrest/initial.done is written only after the command succeeds. Verify the repository afterward with pig pb info (or pb info).

pgbackrest_repo

Parameter Name: pgbackrest_repo, Type: dict, Level: G/C

pgBackRest repository documentation: https://pgbackrest.org/configuration.html#section-repository

The default value contains local and minio as candidate definitions. pgbackrest_method selects one of them, and the v4.5.0 template renders only that selected entry as pgBackRest repo1; listing both keys is not a dual-repository backup configuration:

pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

You can define new backup repositories, such as using AWS S3, GCP, or other cloud providers’ S3-compatible storage services.

Block Incremental Backup: Starting from pgBackRest 2.46, the block: y option enables block-level incremental backup. This means during incremental backups, pgBackRest only backs up changed data blocks instead of entire changed files, significantly reducing backup data volume and backup time. This feature is particularly useful for large databases, and it’s recommended to enable this option on object storage repositories.

PG_ACCESS

This section handles database access paths, including:

  • Deploy Pgbouncer connection pooler on each PGSQL node and set default behavior
  • Publish service ports through local or dedicated haproxy nodes
  • Bind optional L2 VIP and register DNS records
pgbouncer_enabled: true           # if disabled, pgbouncer will not be launched on pgsql host
pgbouncer_port: 6432              # pgbouncer listen port, 6432 by default
pgbouncer_log_dir: /pg/log/pgbouncer  # pgbouncer log dir, `/pg/log/pgbouncer` by default
pgbouncer_auth_query: false       # query postgres to retrieve unlisted business users?
pgbouncer_poolmode: transaction   # pooling mode: transaction,session,statement, transaction by default
pgbouncer_sslmode: disable        # pgbouncer client ssl mode, disable by default
pgbouncer_ignore_param: [ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]
pg_weight: 100          #INSTANCE # relative load balance weight in service, 100 by default, 0-255
pg_service_provider: ''           # dedicate haproxy node group name, or empty string for local nodes by default
pg_default_service_dest: pgbouncer # default service destination if svc.dest='default'
pg_default_services:              # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}
pg_vip_enabled: false             # enable a l2 vip for pgsql primary? false by default
pg_vip_address: 127.0.0.1/24      # vip address in `<ipv4>/<mask>` format, require if vip is enabled
pg_vip_interface: auto            # vip network interface to listen, auto by default
pg_dns_suffix: ''                 # pgsql dns suffix, '' by default
pg_dns_target: auto               # auto, primary, vip, none, or ad hoc ip

pgbouncer_enabled

Parameter Name: pgbouncer_enabled, Type: bool, Level: C

Default value is true. If disabled, the Pgbouncer connection pooler will not be configured on PGSQL nodes.

pgbouncer_port

Parameter Name: pgbouncer_port, Type: port, Level: C

Pgbouncer listen port, default is 6432.

pgbouncer_log_dir

Parameter Name: pgbouncer_log_dir, Type: path, Level: C

Pgbouncer log directory, default is /pg/log/pgbouncer. The Vector log agent collects Pgbouncer logs based on this parameter.

pgbouncer_auth_query

Parameter Name: pgbouncer_auth_query, Type: bool, Level: C

Allow Pgbouncer to query PostgreSQL to allow users not explicitly listed to access PostgreSQL through the connection pool? Default value is false.

If enabled, pgbouncer users will authenticate against the postgres database using SELECT username, password FROM monitor.pgbouncer_auth($1). Otherwise, only business users with pgbouncer: true are allowed to connect to the Pgbouncer connection pool.

pgbouncer_poolmode

Parameter Name: pgbouncer_poolmode, Type: enum, Level: C

Pgbouncer connection pool pooling mode: transaction, session, statement, default is transaction.

  • session: Session-level pooling with best feature compatibility.
  • transaction: Transaction-level pooling with better performance (many small connections), may break some session-level features like NOTIFY/LISTEN, etc.
  • statements: Statement-level pooling for simple read-only queries.

If your application has feature compatibility issues, consider changing this parameter to session.

pgbouncer_sslmode

Parameter Name: pgbouncer_sslmode, Type: enum, Level: C

Pgbouncer client SSL mode, default is disable.

Note that enabling SSL may have a significant performance impact on your pgbouncer.

  • disable: Ignore if client requests TLS (default)
  • allow: Use TLS if client requests it. Use plain TCP if not. Does not verify client certificate.
  • prefer: Same as allow.
  • require: Client must use TLS. Reject client connection if not. Does not verify client certificate.
  • verify-ca: Client must use TLS with a valid client certificate.
  • verify-full: Same as verify-ca.

pgbouncer_ignore_param

Parameter Name: pgbouncer_ignore_param, Type: string[], Level: C

List of startup parameters ignored by PgBouncer, default value is:

[ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]

These parameters are configured in the ignore_startup_parameters option in the PgBouncer configuration file. When clients set these parameters during connection, PgBouncer will not create new connections due to parameter mismatch in the connection pool.

This allows different clients to use the same connection pool even if they set different values for these parameters. This parameter was added in Pigsty v3.5.

pg_weight

Parameter Name: pg_weight, Type: int, Level: I

Relative load balancing weight in service, default is 100, range 0-255.

Default value: 100. You must define it in instance variables and reload service for it to take effect.

pg_service_provider

Parameter Name: pg_service_provider, Type: string, Level: G/C

Dedicated haproxy node group name, or empty string for local nodes by default.

If specified, PostgreSQL services will be registered to the dedicated haproxy node group instead of the current PGSQL cluster nodes.

Remember to allocate unique ports for each service on the dedicated haproxy nodes!

For example, if we define the following parameters on a 3-node pg-test cluster:

pg_service_provider: infra       # use load balancer on group `infra`
pg_default_services:             # alloc port 10001 and 10002 for pg-test primary/replica service
  - { name: primary ,port: 10001 ,dest: postgres  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 10002 ,dest: postgres  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }

pg_default_service_dest

Parameter Name: pg_default_service_dest, Type: enum, Level: G/C

When defining a service, if svc.dest='default', this parameter will be used as the default value.

Default value: pgbouncer, meaning the 5433 primary service and 5434 replica service will route traffic to pgbouncer by default.

If you don’t want to use pgbouncer, set it to postgres. Traffic will be routed directly to postgres.

pg_default_services

Parameter Name: pg_default_services, Type: service[], Level: G/C

Postgres default service definitions.

Default value is four default service definitions, as described in PGSQL Service.

pg_default_services:               # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

pg_vip_enabled

Parameter Name: pg_vip_enabled, Type: bool, Level: C

Enable L2 VIP for PGSQL cluster? Default value is false, meaning no L2 VIP will be created.

When L2 VIP is enabled, a VIP will be bound to the cluster primary instance node, managed by vip-manager based on data in etcd.

L2 VIP can only be used within the same L2 network, which may impose additional constraints on your network topology.

pg_vip_address

Parameter Name: pg_vip_address, Type: cidr4, Level: C

VIP address in <ipv4>/<mask> format is required if VIP is enabled.

Default value: 127.0.0.1/24. This value consists of two parts: ipv4 and mask, separated by /.

pg_vip_interface

Parameter Name: pg_vip_interface, Type: string, Level: C/I

VIP network interface to listen on, auto by default. Pigsty detects the interface associated with the instance IP in the inventory.

For non-standard routing, policy routing, or other unusual network environments where auto-detection is unsuitable, explicitly override it in the instance variables:

pg-test:
    hosts:
        10.10.10.11: {pg_seq: 1, pg_role: replica ,pg_vip_interface: eth0 }
        10.10.10.12: {pg_seq: 2, pg_role: primary ,pg_vip_interface: eth1 }
        10.10.10.13: {pg_seq: 3, pg_role: replica ,pg_vip_interface: eth2 }
    vars:
      pg_vip_enabled: true          # enable L2 VIP for this cluster, binds to primary by default
      pg_vip_address: 10.10.10.3/24 # L2 network CIDR: 10.10.10.0/24, vip address: 10.10.10.3
      # pg_vip_interface: eth1      # if your nodes have a unified interface, you can define it here

pg_dns_suffix

Parameter Name: pg_dns_suffix, Type: string, Level: C

PostgreSQL DNS name suffix, default is empty string.

By default, the PostgreSQL cluster name is registered as a DNS domain in dnsmasq on Infra nodes for external resolution.

You can specify a domain suffix with this parameter, which will use {{ pg_cluster }}{{ pg_dns_suffix }} as the cluster DNS name.

For example, if you set pg_dns_suffix to .db.vip.company.tld, the pg-test cluster DNS name will be pg-test.db.vip.company.tld.

pg_dns_target

Parameter Name: pg_dns_target, Type: enum, Level: C

Could be: auto, primary, vip, none, or an ad hoc IP address, which will be the target IP address of cluster DNS record.

Default value: auto, which will bind to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance IP address.

  • vip: bind to pg_vip_address
  • primary: resolve to cluster primary instance IP address
  • auto: resolve to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance IP address
  • none: do not bind to any IP address
  • <ipv4>: bind to the given IP address

PG_MONITOR

The PG_MONITOR group parameters are used to monitor the status of PostgreSQL databases, Pgbouncer connection pools, and pgBackRest backup systems.

This parameter group defines three Exporter configurations: pg_exporter for monitoring PostgreSQL, pgbouncer_exporter for monitoring connection pools, and pgbackrest_exporter for monitoring backup status.

pg_exporter_enabled: true              # enable pg_exporter on pgsql host?
pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector ttl stages (seconds), default is '1,10,60,300'
pg_exporter_port: 9630                 # pg_exporter listen port, default is 9630
pg_exporter_params: 'sslmode=disable'  # extra url parameters for pg_exporter dsn
pg_exporter_url: ''                    # if specified, will override auto-generated pg dsn
pg_exporter_auto_discovery: true       # enable auto database discovery? enabled by default
pg_exporter_exclude_database: 'template0,template1,postgres' # csv list of databases not monitored during auto-discovery
pg_exporter_include_database: ''       # csv list of databases monitored during auto-discovery
pg_exporter_connect_timeout: 200       # pg_exporter connection timeout (ms), default is 200
pg_exporter_options: ''                # extra options to override pg_exporter
pgbouncer_exporter_enabled: true       # enable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_port: 9631          # pgbouncer_exporter listen port, default is 9631
pgbouncer_exporter_url: ''             # if specified, will override auto-generated pgbouncer dsn
pgbouncer_exporter_options: ''         # extra options to override pgbouncer_exporter
pgbackrest_exporter_enabled: true      # enable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_port: 9854         # pgbackrest_exporter listen port, default is 9854
pgbackrest_exporter_options: >-        # extra options to override pgbackrest_exporter
  --collect.interval=120
  --log.level=info

pg_exporter_enabled

Parameter Name: pg_exporter_enabled, Type: bool, Level: C

Enable pg_exporter on PGSQL nodes? Default value is: true.

PG Exporter is used to monitor PostgreSQL database instances. Set to false if you don’t want to install pg_exporter.

pg_exporter_config

Parameter Name: pg_exporter_config, Type: string, Level: C

pg_exporter’s collector configuration template name. The default is pg_exporter.yml. Pgbouncer Exporter uses its own fixed pgbouncer_exporter.yml template and is not affected by this parameter.

The default template is roles/pg_monitor/templates/pg_exporter.yml. To use a custom file, put it on Ansible’s template search path and specify its template name here.

pg_exporter_cache_ttls

Parameter Name: pg_exporter_cache_ttls, Type: string, Level: C

pg_exporter collector TTL stages (seconds), default is ‘1,10,60,300’.

Default value: 1,10,60,300, which will use different TTL values for different metric collectors: 1s, 10s, 60s, 300s.

PG Exporter has a built-in caching mechanism to avoid the improper impact of multiple Prometheus scrapes on the database. All metric collectors are divided into four categories by TTL:

ttl_fast: "{{ pg_exporter_cache_ttls.split(',')[0]|int }}"         # critical queries
ttl_norm: "{{ pg_exporter_cache_ttls.split(',')[1]|int }}"         # common queries
ttl_slow: "{{ pg_exporter_cache_ttls.split(',')[2]|int }}"         # slow queries (e.g table size)
ttl_slowest: "{{ pg_exporter_cache_ttls.split(',')[3]|int }}"      # ver slow queries (e.g bloat)

For example, with default configuration, liveness metrics are cached for at most 1s, most common metrics are cached for 10s (should match the monitoring scrape interval victoria_scrape_interval). A few slow-changing queries have 60s TTL, and very few high-overhead monitoring queries have 300s TTL.

pg_exporter_port

Parameter Name: pg_exporter_port, Type: port, Level: C

pg_exporter listen port, default value is: 9630

pg_exporter_params

Parameter Name: pg_exporter_params, Type: string, Level: C

Extra URL path parameters in the DSN used by pg_exporter.

Default value: sslmode=disable, which disables SSL for monitoring connections (since local unix sockets are used by default).

pg_exporter_url

Parameter Name: pg_exporter_url, Type: pgurl, Level: C

If specified, will override the auto-generated PostgreSQL DSN and use the specified DSN to connect to PostgreSQL. Default value is empty string.

If not specified, PG Exporter will use the following connection string to access PostgreSQL by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@{{ pg_host }}:{{ pg_port }}/postgres{% if pg_exporter_params != '' %}?{{ pg_exporter_params }}{% endif %}

Use this parameter when you want to monitor a remote PostgreSQL instance, or need to use different monitoring user/password or configuration options.

pg_exporter_auto_discovery

Parameter Name: pg_exporter_auto_discovery, Type: bool, Level: C

Enable auto database discovery? Enabled by default: true.

By default, PG Exporter connects to the database specified in the DSN (default is the admin database postgres) to collect global metrics. If you want to collect metrics from all business databases, enable this option. PG Exporter will automatically discover all databases in the target PostgreSQL instance and collect database-level monitoring metrics from these databases.

pg_exporter_exclude_database

Parameter Name: pg_exporter_exclude_database, Type: string, Level: C

If database auto-discovery is enabled (enabled by default), databases in this parameter’s list will not be monitored. Default value is: template0,template1,postgres, meaning the admin database postgres and template databases are excluded from auto-monitoring.

As an exception, the database specified in the DSN is not affected by this parameter. For example, if PG Exporter connects to the postgres database, it will be monitored even if postgres is in this list.

pg_exporter_include_database

Parameter Name: pg_exporter_include_database, Type: string, Level: C

If database auto-discovery is enabled (enabled by default), only databases in this parameter’s list will be monitored. Default value is empty string, meaning this feature is not enabled.

The parameter format is a comma-separated list of database names, e.g., db1,db2,db3.

This parameter has higher priority than pg_exporter_exclude_database, acting as a whitelist mode. Use this parameter if you only want to monitor specific databases.

pg_exporter_connect_timeout

Parameter Name: pg_exporter_connect_timeout, Type: int, Level: C

pg_exporter connection timeout (milliseconds), default is 200 (in milliseconds).

How long will PG Exporter wait when trying to connect to a PostgreSQL database? Beyond this time, PG Exporter will give up the connection and report an error.

The default value of 200ms is sufficient for most scenarios (e.g., same availability zone monitoring), but if your monitored remote PostgreSQL is on another continent, you may need to increase this value to avoid connection timeouts.

pg_exporter_options

Parameter Name: pg_exporter_options, Type: arg, Level: C

Command line arguments passed to PG Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pg_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} {{ pg_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pg_exporter_port port configuration in this parameter.

pgbouncer_exporter_enabled

Parameter Name: pgbouncer_exporter_enabled, Type: bool, Level: C

Enable pgbouncer_exporter on PGSQL nodes? Default value is: true.

pgbouncer_exporter_port

Parameter Name: pgbouncer_exporter_port, Type: port, Level: C

pgbouncer_exporter listen port, default value is: 9631

pgbouncer_exporter_url

Parameter Name: pgbouncer_exporter_url, Type: pgurl, Level: C

If specified, will override the auto-generated pgbouncer DSN and use the specified DSN to connect to pgbouncer. Default value is empty string.

If not specified, Pgbouncer Exporter will use the following connection string to access Pgbouncer by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@:{{ pgbouncer_port }}/pgbouncer?host={{ pg_localhost }}&sslmode=disable

Use this parameter when you want to monitor a remote Pgbouncer instance, or need to use different monitoring user/password or configuration options.

pgbouncer_exporter_options

Parameter Name: pgbouncer_exporter_options, Type: arg, Level: C

Command line arguments passed to Pgbouncer Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pgbouncer_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} {{ pgbouncer_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pgbouncer_exporter_port port configuration in this parameter.

pgbackrest_exporter_enabled

Parameter Name: pgbackrest_exporter_enabled, Type: bool, Level: C

Enable pgbackrest_exporter on PGSQL nodes? Default value is: true.

pgbackrest_exporter is used to monitor the status of the pgBackRest backup system, including key metrics such as backup size, time, type, and duration.

pgbackrest_exporter_port

Parameter Name: pgbackrest_exporter_port, Type: port, Level: C

pgbackrest_exporter listen port, default value is: 9854.

This port is registered in the VictoriaMetrics-compatible scrape targets for backup-related metrics.

pgbackrest_exporter_options

Parameter Name: pgbackrest_exporter_options, Type: arg, Level: C

Command-line arguments passed to pgbackrest_exporter. The default is:

pgbackrest_exporter_options: >-
  --collect.interval=120
  --log.level=info

This collects every 120 seconds at info log level. Setting the parameter replaces the default argument set as a whole.

PG_REMOVE

pgsql-rm.yml invokes the pg_remove role to safely remove PostgreSQL instances. This section’s parameters control cleanup behavior to avoid accidental deletion.

pg_rm_data: true                  # remove postgres data during remove? true by default
pg_rm_backup: true                # remove pgbackrest backup during primary remove? true by default
pg_rm_pkg: true                   # uninstall postgres packages during remove? true by default
pg_safeguard: false               # stop pg_remove running if pg_safeguard is enabled, false by default

pg_rm_data

Parameter Name: pg_rm_data, Type: bool, Level: G/C/A

Whether to clean up pg_data and symlinks when removing PGSQL instances, default is true.

This switch affects both pgsql-rm.yml and other scenarios that trigger pg_remove. Set to false to preserve the data directory for manual inspection or remounting.

pg_rm_backup

Parameter Name: pg_rm_backup, Type: bool, Level: G/C/A

Whether to also clean up the pgBackRest repository and configuration when removing the primary, default is true.

This parameter only applies to primary instances with pg_role=primary: pg_remove will first stop pgBackRest, delete the current cluster’s stanza, and remove data in pg_fs_backup when pgbackrest_method == 'local'. Standby clusters or upstream backups are not affected.

pg_rm_pkg

Parameter Name: pg_rm_pkg, Type: bool, Level: G/C/A

Whether to uninstall all packages installed by pg_packages when cleaning up PGSQL instances, default is true.

If you only want to temporarily stop and preserve binaries, set it to false. Otherwise, pg_remove will call the system package manager to completely uninstall PostgreSQL-related components.

pg_safeguard

Parameter Name: pg_safeguard, Type: bool, Level: G/C/A

Accidental deletion protection, default is false. When explicitly set to true, pg_remove will immediately terminate with a prompt, and will only continue after using -e pg_safeguard=false or disabling it in variables.

It’s recommended to enable this switch before batch cleanup in production environments, verify the commands and target nodes are correct, then disable it to avoid accidental deletion of instances.

12 - Playbook

How to manage PostgreSQL clusters with Ansible playbooks

Pigsty provides a series of playbooks for cluster provisioning, scaling, user/database management, monitoring, backup & recovery, and migration.

Playbook Function
pgsql.yml Initialize PostgreSQL cluster or add new replicas
pgsql-rm.yml Remove PostgreSQL cluster or specific instances
pgsql-user.yml Add new business user to existing PostgreSQL cluster
pgsql-db.yml Add new business database to existing PostgreSQL cluster
pgsql-monitor.yml Monitor remote PostgreSQL instances
pgsql-migration.yml Generate migration manual and scripts for existing PostgreSQL
pgsql-pitr.yml Perform Point-In-Time Recovery (PITR)

Safeguard

Be extra cautious when using PGSQL playbooks. Misuse of pgsql.yml and pgsql-rm.yml can lead to accidental database deletion!

  • Always add the -l parameter to limit the execution scope, and ensure you’re executing the right tasks on the right targets.
  • Limiting scope to a single cluster is recommended. Running pgsql.yml without parameters in production is a high-risk operation—think twice before proceeding.
  • Before removal, inspect pig pg list <cluster> and pig pb info, verify a recent backup, and have the operator enter the exact target.

To prevent accidental deletion, Pigsty’s PGSQL module provides a safeguard mechanism controlled by the pg_safeguard parameter. When pg_safeguard is set to true, the pgsql-rm.yml playbook will abort immediately, protecting your database cluster.

# Will abort execution, protecting data
./pgsql-rm.yml -l pg-test -e pg_safeguard=true

# Override the safeguard switch from the command line
./pgsql-rm.yml -l pg-test -e pg_safeguard=false

In addition to pg_safeguard, pgsql-rm.yml provides finer-grained control parameters:

Parameter Default Description
pg_safeguard false Safeguard switch; when true, playbook aborts
pg_rm_data true Whether to remove PostgreSQL data directory
pg_rm_backup true Whether to remove pgBackRest backup data (only when removing primary)
pg_rm_pkg true Whether to uninstall PostgreSQL packages

These parameters allow precise control over removal behavior:

# Remove the instance services, monitoring, and DCS registration, but keep its data directory
./pgsql-rm.yml -l pg-test -e pg_rm_data=false

# Remove cluster but keep backup data
./pgsql-rm.yml -l pg-test -e pg_rm_backup=false

# Remove cluster and uninstall packages
./pgsql-rm.yml -l pg-test -e pg_rm_pkg=true

pgsql.yml

The pgsql.yml playbook is used to initialize PostgreSQL clusters or add new replicas.

Here’s a demo of initializing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql.yml -l pg-meta            # Initialize cluster pg-meta
./pgsql.yml -l 10.10.10.13        # Initialize/add instance 10.10.10.13
./pgsql.yml -l pg-test -t pg_service  # Refresh services for cluster pg-test
./pgsql.yml -l pg-test -t pg_hba,pgbouncer_hba,pgbouncer_reload -e pg_reload=true  # Reload HBA rules

Wrapper Scripts

Pigsty provides convenient wrapper scripts to simplify common operations:

bin/pgsql-add pg-meta             # Initialize pgsql cluster pg-meta
bin/pgsql-add 10.10.10.10         # Initialize pgsql instance 10.10.10.10
bin/pgsql-add pg-test 10.10.10.13 # Add 10.10.10.13 to cluster pg-test (auto refresh services)
bin/pgsql-svc pg-test             # Refresh haproxy services for pg-test (use after membership changes)
bin/pgsql-hba pg-test             # Reload pg/pgb HBA rules for pg-test

Subtasks

This playbook contains the following subtasks:

# pg_install              : install postgres packages & extensions
#   - pg_dbsu             : setup postgres superuser
#     - pg_dbsu_create    : create dbsu user
#     - pg_dbsu_sudo      : configure dbsu sudo privileges
#     - pg_ssh            : exchange dbsu SSH keys
#   - pg_pkg              : install postgres packages
#     - pg_pre            : pre-installation tasks
#     - pg_ext            : install postgres extension packages
#     - pg_post           : post-installation tasks
#   - pg_link             : link pgsql version bin to /usr/pgsql
#   - pg_path             : add pgsql bin to system path
#   - pg_dir              : create postgres directories and setup FHS
#   - pg_bin              : sync /pg/bin scripts
#   - pg_alias            : configure pgsql/psql aliases
#   - pg_dummy            : create dummy placeholder file
#
# pg_bootstrap            : bootstrap postgres cluster
#   - pg_config           : generate postgres config
#     - pg_conf           : generate patroni config
#     - pg_key            : generate pgsodium key
#   - pg_cert             : issue certificates for postgres
#     - pg_cert_private   : check pg private key existence
#     - pg_cert_issue     : sign pg server certificate
#     - pg_cert_copy      : copy key & certs to pg node
#   - pg_launch           : launch patroni primary & replicas
#     - pg_watchdog       : grant watchdog permission to postgres
#     - pg_primary        : launch patroni/postgres primary
#     - pg_init           : init pg cluster with roles/templates
#     - pg_pass           : write .pgpass file to pg home
#     - pg_replica        : launch patroni/postgres replicas
#     - pg_hba            : generate pg HBA rules
#     - patroni_reload    : reload patroni config
#     - pg_patroni        : pause or remove patroni if necessary
#
# pg_provision            : provision postgres business users & databases
#   - pg_user             : provision postgres business users
#     - pg_user_config    : render create user SQL
#     - pg_user_create    : create user on postgres
#   - pg_db               : provision postgres business databases
#     - pg_db_drop        : drop database on postgres (state=absent/recreate)
#     - pg_db_config      : render create database SQL
#     - pg_db_create      : create database on postgres
#
# pg_backup               : init postgres PITR backup
#   - pgbackrest          : setup pgbackrest for backup
#     - pgbackrest_config : generate pgbackrest config
#     - pgbackrest_init   : init pgbackrest repo
#     - pgbackrest_backup : make initial backup after bootstrap
#
# pg_access               : init postgres service access layer
#   - pgbouncer           : deploy pgbouncer connection pooler
#     - pgbouncer_dir     : create pgbouncer directories
#     - pgbouncer_config  : generate pgbouncer config
#       - pgbouncer_hba   : generate pgbouncer HBA config
#       - pgbouncer_user  : generate pgbouncer userlist
#     - pgbouncer_launch  : launch pgbouncer service
#     - pgbouncer_reload  : reload pgbouncer config
#   - pg_vip              : bind VIP to primary with vip-manager
#     - pg_vip_config     : generate vip-manager config
#     - pg_vip_launch     : launch vip-manager to bind VIP
#   - pg_dns              : register DNS name to infra dnsmasq
#     - pg_dns_ins        : register pg instance name
#     - pg_dns_cls        : register pg cluster name
#   - pg_service          : expose pgsql service with haproxy
#     - pg_service_config : generate local haproxy config for pg services
#     - pg_service_reload : expose postgres services with haproxy
#
# pg_monitor              : setup pgsql monitoring and register to infra
#   - pg_exporter         : configure and launch pg_exporter
#   - pgbouncer_exporter  : configure and launch pgbouncer_exporter
#   - pgbackrest_exporter : configure and launch pgbackrest_exporter
#   - pg_register         : register pgsql to monitoring/logging/datasource
#     - add_metrics       : register pg as VictoriaMetrics monitoring target
#     - add_logs          : register pg as Vector log source
#     - add_ds            : register pg database as Grafana datasource

Related Administration Tasks

Notes

  • When running this playbook on a single replica, ensure the cluster primary is already initialized!
  • After scaling out, you need to Reload Service and Reload HBA. The wrapper script bin/pgsql-add handles these tasks automatically.

When scaling a cluster, if Patroni takes too long to bring up a replica, the Ansible playbook may abort due to timeout:

  • Typical error message: wait for postgres/patroni replica task runs for a long time before aborting
  • However, the replica creation process continues. For scenarios where replica creation takes more than a day, see FAQ: Replica creation failed.

pgsql-rm.yml

The pgsql-rm.yml playbook is used to remove PostgreSQL clusters or specific instances.

Here’s a demo of removing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
./pgsql-rm.yml -l 10.10.10.13      # Remove instance 10.10.10.13

Command Line Arguments

This playbook supports the following command line arguments:

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
    -e pg_safeguard=false          # Safeguard switch, disabled by default; override when enabled
    -e pg_rm_data=true             # Whether to remove PostgreSQL data directory, default: remove
    -e pg_rm_backup=true           # Whether to remove pgBackRest backup (primary only), default: remove
    -e pg_rm_pkg=true              # Whether to uninstall PostgreSQL packages, default: uninstall

Wrapper Scripts

bin/pgsql-rm pg-meta               # Remove pgsql cluster pg-meta
bin/pgsql-rm pg-test 10.10.10.13   # Remove instance 10.10.10.13 from cluster pg-test

Subtasks

This playbook contains the following subtasks:

# pg_safeguard           : abort if pg_safeguard is enabled
#
# pg_monitor             : remove registration from monitoring system
#   - pg_deregister      : remove pg monitoring targets from infra
#     - rm_metrics       : remove targets from the VictoriaMetrics target directory
#     - rm_ds            : remove datasource from grafana
#     - rm_logs          : remove log targets from vector
#   - pg_exporter        : remove pg_exporter
#   - pgbouncer_exporter : remove pgbouncer_exporter
#   - pgbackrest_exporter: remove pgbackrest_exporter
#
# pg_access              : remove pg service access layer
#   - dns                : remove pg DNS records
#   - vip                : remove vip-manager
#   - pg_service         : remove pg service from haproxy
#   - pgbouncer          : remove pgbouncer connection middleware
#
# postgres               : remove postgres instances
#   - pg_replica         : remove all replicas
#   - pg_primary         : remove primary
#   - pg_meta            : remove metadata from etcd
#
# pg_backup              : remove backup repo (disable with pg_rm_backup=false)
# pg_data                : remove postgres data (disable with pg_rm_data=false)
# pg_pkg                 : uninstall pg packages (enabled by default; disable with pg_rm_pkg=false)
#   - pg_ext             : uninstall postgres extensions alone

Related Administration Tasks

Notes

  • Do not run this playbook on a primary that still has replicas—otherwise, remaining replicas will trigger automatic failover. Always remove all replicas first, then remove the primary. This is not a concern when removing the entire cluster at once.
  • Refresh cluster services after removing instances. When you remove a replica from a cluster, it remains in the load balancer configuration file. Since health checks will fail, the removed instance won’t affect cluster services. However, you should Reload Service at an appropriate time to ensure consistency between the production environment and configuration inventory.

pgsql-user.yml

The pgsql-user.yml playbook is used to add new business users to existing PostgreSQL clusters.

Basic Usage

./pgsql-user.yml -l pg-meta -e username=dbuser_meta

Wrapper Scripts

bin/pgsql-user pg-meta dbuser_meta  # Create user dbuser_meta on cluster pg-meta

Workflow

  1. Define user in the config inventory: all.children.<pg_cluster>.vars.pg_users[i]
  2. Execute playbook specifying cluster and username: pgsql-user.yml -l <pg_cluster> -e username=<name>

The playbook will:

  1. Generate user creation SQL at /pg/tmp/pg-user-{{ user.name }}.sql
  2. Execute user creation/update SQL on the cluster primary
  3. If pgbouncer_enabled: true, update /etc/pgbouncer/userlist.txt and useropts.txt
  4. Reload pgbouncer to apply configuration

User Definition Example

pg_users:
  - name: dbuser_meta               # Required, username is the only mandatory field
    password: DBUser.Meta           # Optional, can be scram-sha-256 hash or plaintext
    login: true                     # Optional, can login, default: true
    superuser: false                # Optional, is superuser, default: false
    createdb: false                 # Optional, can create database, default: false
    createrole: false               # Optional, can create role, default: false
    inherit: true                   # Optional, inherit privileges, default: true
    replication: false              # Optional, can replicate, default: false
    bypassrls: false                # Optional, bypass RLS, default: false
    pgbouncer: true                 # Optional, add to pgbouncer userlist, default: false
    connlimit: -1                   # Optional, connection limit, -1 means unlimited
    expire_in: 3650                 # Optional, expire in N days (overrides expire_at)
    expire_at: '2030-12-31'         # Optional, specify expiration date
    comment: pigsty admin user      # Optional, user comment
    roles: [dbrole_admin]           # Optional, roles to grant
    parameters: {}                  # Optional, role-level parameters
    pool_mode: transaction          # Optional, pgbouncer user-level pool mode
    pool_connlimit: 100             # Optional, user-level max connections; omitted values inherit global default 100

For details, see: Admin SOP: Create User

pgsql-db.yml

The pgsql-db.yml playbook is used to add new business databases to existing PostgreSQL clusters.

Basic Usage

./pgsql-db.yml -l pg-meta -e dbname=meta

Wrapper Scripts

bin/pgsql-db pg-meta meta  # Create database meta on cluster pg-meta

Workflow

  1. Define database in the config inventory: all.children.<pg_cluster>.vars.pg_databases[i]
  2. Execute playbook specifying cluster and database name: pgsql-db.yml -l <pg_cluster> -e dbname=<name>

The playbook will:

  1. Generate database creation SQL at /pg/tmp/pg-db-{{ database.name }}.sql
  2. Execute database creation/update SQL on the cluster primary
  3. If db.register_datasource is true, register database as Grafana datasource
  4. Update /etc/pgbouncer/database.txt and reload pgbouncer

Database Definition Example

pg_databases:
  - name: meta                      # Required, database name is the only mandatory field
    baseline: cmdb.sql              # Optional, database initialization SQL file path
    pgbouncer: true                 # Optional, add to pgbouncer, default: true
    schemas: [pigsty]               # Optional, additional schemas to create
    extensions:                     # Optional, extensions to install
      - { name: postgis, schema: public }
      - { name: timescaledb }
    comment: pigsty meta database   # Optional, database comment
    owner: postgres                 # Optional, database owner
    template: template1             # Optional, template database
    encoding: UTF8                  # Optional, character encoding
    locale: C                       # Optional, locale setting
    tablespace: pg_default          # Optional, default tablespace
    allowconn: true                 # Optional, allow connections
    revokeconn: false               # Optional, revoke public connect privilege
    register_datasource: true       # Optional, register as Grafana datasource
    connlimit: -1                   # Optional, connection limit
    pool_auth_user: dbuser_meta     # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction          # Optional, pgbouncer pool mode
    pool_size: 50                   # Optional, pgbouncer default pool size
    pool_reserve: 30                # Optional, pgbouncer reserve pool size
    pool_size_min: 0                # Optional, pgbouncer minimum pool size
    pool_connlimit: 100             # Optional, pgbouncer max database connections

For details, see: Admin SOP: Create Database

pgsql-monitor.yml

The pgsql-monitor.yml playbook is used to bring remote PostgreSQL instances into Pigsty’s monitoring system.

Basic Usage

./pgsql-monitor.yml -e clsname=pg-foo  # Monitor remote cluster pg-foo

Wrapper Scripts

bin/pgmon-add pg-foo              # Monitor a remote pgsql cluster pg-foo
bin/pgmon-add pg-foo pg-bar       # Monitor multiple clusters simultaneously

Configuration

First, define pg_exporters in the infra group variables:

infra:
  hosts:
    10.10.10.10:
      pg_exporters:  # List all remote instances, assign unique unused local ports
        20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
        20002: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }

Architecture Diagram

     ------ infra ------
     |                 |
     | victoria-metrics|            v---- pg-foo-1 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20001) |            | 10.10.10.10:5432 |
     |       ^         |            ^------------------^
     |       ^         |                      ^
     |       ^         |            v---- pg-foo-2 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20002) |            | 10.10.10.11:5433 |
     -------------------            ^------------------^

Configurable Parameters

pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector TTL stages
pg_exporter_port: 9630                 # pg_exporter listen port
pg_exporter_params: 'sslmode=disable'  # DSN extra URL parameters
pg_exporter_url: ''                    # Directly override auto-generated DSN
pg_exporter_auto_discovery: true       # Enable auto database discovery
pg_exporter_exclude_database: 'template0,template1,postgres'  # Databases to exclude
pg_exporter_include_database: ''       # Databases to include only
pg_exporter_connect_timeout: 200       # Connection timeout (milliseconds)
pg_monitor_username: dbuser_monitor    # Monitor username
pg_monitor_password: DBUser.Monitor    # Monitor password

Remote Database Setup

Remote PostgreSQL instances need a monitoring user:

CREATE USER dbuser_monitor;
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';
ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';
GRANT pg_monitor TO dbuser_monitor;
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Limitations

  • Only postgres metrics available
  • node, pgbouncer, patroni, haproxy metrics not available

For details, see: Admin SOP: Monitor RDS

pgsql-migration.yml

The pgsql-migration.yml playbook generates migration manuals and scripts for zero-downtime logical replication-based migration of existing PostgreSQL clusters.

Basic Usage

./pgsql-migration.yml -e@files/migration/pg-meta.yml

Workflow

  1. Define migration task configuration file (e.g., files/migration/pg-meta.yml)
  2. Execute playbook to generate migration manual and scripts
  3. Follow the manual to execute scripts step by step for migration

Migration Task Definition Example

# files/migration/pg-meta.yml
context_dir: ~/migration           # Migration manual and scripts output directory
src_cls: pg-meta                   # Source cluster name (required)
src_db: meta                       # Source database name (required)
src_ip: 10.10.10.10                # Source cluster primary IP (required)
dst_cls: pg-test                   # Target cluster name (required)
dst_db: test                       # Target database name (required)
dst_ip: 10.10.10.11                # Target cluster primary IP (required)

# Optional parameters
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

For details, see: Admin SOP: Migrate Cluster

pgsql-pitr.yml

The pgsql-pitr.yml playbook performs PostgreSQL Point-In-Time Recovery (PITR).

Basic Usage

# Recover to latest state (end of WAL archive stream)
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {}}'

# Recover to specific point in time
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"time": "2025-07-13 10:00:00+00"}}'

# Recover to specific LSN
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"lsn": "0/4001C80"}}'

# Recover to specific transaction ID
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"xid": "250000"}}'

# Recover to named restore point
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"name": "some_restore_point"}}'

# Recover from another cluster's backup
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": {"cluster": "pg-meta"}}'

PITR Task Parameters

pg_pitr:                           # Define PITR task
  cluster: "pg-meta"               # Source cluster name (for restoring from another cluster's backup)
  type: default                    # Recovery target type: default, time, xid, name, lsn, immediate
  time: "2025-01-01 10:00:00+00"   # Recovery target: point in time
  name: "some_restore_point"       # Recovery target: named restore point
  xid: "100000"                    # Recovery target: transaction ID
  lsn: "0/3000000"                 # Recovery target: log sequence number
  set: latest                      # Backup set to restore from, default: latest
  timeline: latest                 # Target timeline, can be integer, default: latest
  exclusive: false                 # Exclude target point, default: false
  action: pause                    # Post-recovery action: pause, promote, shutdown
  archive: true                    # Keep archive settings, default: true; set false for exploratory recovery
  backup: false                    # Backup existing data to /pg/data-backup before restore? default: false
  db_include: []                   # Include only these databases
  db_exclude: []                   # Exclude these databases
  link_map: {}                     # Tablespace link mapping
  process: 4                       # Parallel recovery processes, defaults to node_cpu
  repo: {}                         # Recovery source repo configuration
  data: /pg/data                   # Recovery data directory
  port: 5432                       # Recovery instance listen port

Subtasks

This playbook contains the following subtasks:

# down                 : stop HA and shutdown patroni and postgres
#   - pause            : pause patroni auto failover
#   - stop             : stop patroni and postgres services
#     - stop_patroni   : stop patroni service
#     - stop_postgres  : stop postgres service
#
# pitr                 : execute PITR recovery process
#   - config           : generate pgbackrest config and recovery script
#   - backup           : perform optional backup to original data
#   - restore          : run pgbackrest restore command
#   - recovery         : start postgres and complete recovery
#   - verify           : verify recovered cluster control data
#
# up                   : start postgres/patroni and restore HA
#   - etcd             : clean etcd metadata before startup
#   - start            : start patroni and postgres services
#     - start_postgres : start postgres service
#     - start_patroni  : start patroni service
#   - resume           : resume patroni auto failover

Recovery Target Types

Type Description Example
default Recover to end of WAL archive stream (latest state) {"pg_pitr": {}}
time Recover to specific point in time {"pg_pitr": {"time": "2025-07-13 10:00:00"}}
xid Recover to specific transaction ID {"pg_pitr": {"xid": "250000"}}
name Recover to named restore point {"pg_pitr": {"name": "before_ddl"}}
lsn Recover to specific LSN {"pg_pitr": {"lsn": "0/4001C80"}}
immediate Stop immediately after reaching consistent state {"pg_pitr": {"type": "immediate"}}

For details, see: Backup & Recovery Tutorial

13 - Extensions

Harness the synergistic power of PostgreSQL extensions

Pigsty provides 575 packaged extensions, covering 16 major categories including time-series, geospatial, vector, full-text search, analytics, and feature enhancements, ready to use out-of-the-box.

Using extensions in Pigsty involves four core steps: Download, Install, Config/Load, and Create.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create: Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Config: Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install: Install extension packages
Pigsty PostgreSQL extension ecosystem

13.1 - Quick Start

Four-step process overview for using extensions

Using extensions in Pigsty requires four steps: Download, Install, Config, and Create.

  1. Download: Download extension packages to the local repository (the default local repository only guarantees the base kernel and pgsql-main package set)
  2. Install: Install extension packages on cluster nodes
  3. Config: Some extensions need to be preloaded or configured with parameters
  4. Create: Execute CREATE EXTENSION in the database to create the extension

Declarative Configuration

Declare extensions in the Pigsty configuration manifest, and they will be automatically installed and created during cluster initialization:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install extension packages

After executing ./pgsql.yml to initialize the cluster, the three extensions postgis, timescaledb, and vector will be available in the meta database.

Imperative Operations

For existing clusters, you can add extensions using command-line methods:

# 1. Install extension packages
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# 2. Preload extension (if needed, requires restart after modification)
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# 3. Create extension in database
psql -d meta -c 'CREATE EXTENSION vector;'

You can also use the pig package manager to install extension packages, then run CREATE EXTENSION inside the database:

pig install pgvector                         # Install extension package
psql -d meta -c 'CREATE EXTENSION vector;'   # Create extension in database

Process Quick Reference

Step Parameter/Command Description
Download repo_extra_packages Specify extension packages to download to local repository
Install pg_extensions Specify extension packages to install on cluster
Config pg_libs Preload extensions to shared_preload_libraries
Create pg_databases.extensions Automatically execute CREATE EXTENSION in database

For detailed instructions, please refer to each subsection: Download, Install, Config, Create

13.2 - Introduction

Core concepts of PostgreSQL extensions and the Pigsty extension ecosystem

Extensions are the soul of PostgreSQL. Pigsty includes 575 pre-compiled, out-of-the-box extension plugins, fully unleashing PostgreSQL’s potential.

What are Extensions

PostgreSQL extensions are a modular mechanism that allows enhancing database functionality without modifying the core code. An extension typically consists of three parts:

  • Control file (.control): Required, contains extension metadata
  • SQL scripts (.sql): Optional, defines functions, types, operators, and other database objects
  • Dynamic library (.so): Optional, provides high-performance functionality implemented in C

Extensions can add to PostgreSQL: new data types, index methods, functions and operators, foreign data access, procedural languages, performance monitoring, security auditing, and more.

Core Extensions

Among the extensions included in Pigsty, the following are most representative:

Extension Description
PostGIS Geospatial data types and indexes, de facto GIS standard
TimescaleDB Time-series database with continuous aggregates, columnar storage, auto-compression
PGVector Vector data type with HNSW/IVFFlat indexes, essential for AI applications
Citus Distributed database with horizontal sharding capabilities
pg_duckdb Embedded DuckDB analytical engine for OLAP acceleration
pg_search ParadeDB search extension, providing BM25 and full-text search capabilities
Apache AGE Graph database supporting OpenCypher query language
pg_graphql Native GraphQL query support

Most extensions can coexist and even be combined, creating synergistic effects far greater than the sum of their parts.

Extension Categories

Pigsty organizes extensions into 16 categories:

Category Alias Description Typical Extensions
Time-series time Time-series data processing timescaledb, pg_cron, periods
Geospatial gis Geospatial data postgis, h3, pgrouting
Vector rag Vector retrieval and AI pgvector, vchord, pg_vectorize
Search fts Full-text search pgroonga, zhparser, pg_bigm
Analytics olap OLAP and analytics pg_duckdb, pg_mooncake, citus
Feature feat Feature enhancements age, pg_graphql, hll, rum
Language lang Procedural languages plpython3u, pljava, plv8
Type type Data types hstore, ltree, ip4r
Utility util Utility tools http, pg_net, pgjwt
Function func Function libraries pg_uuidv7, topn, tdigest
Admin admin Operations management pg_repack, pg_squeeze, pgagent
Stat stat Monitoring statistics pg_stat_statements, pg_qualstats, auto_explain
Security sec Security auditing pgaudit, pgsodium, pg_tde
FDW fdw Foreign data access postgres_fdw, mysql_fdw, oracle_fdw
Compatibility sim Database compatibility orafce, babelfish
ETL etl Data synchronization pglogical, wal2json, decoderbufs

You can batch install an entire category of extensions using category aliases, for example: pg_extensions: [ pgsql-gis, pgsql-rag ].

Predefined Extension Stacks

Pigsty provides several predefined extension stacks for convenient scenario-based selection:

Stack Included Extensions
gis-stack postgis, pgrouting, pointcloud, h3, q3c, ogr_fdw
rag-stack pgvector, vchord, pgvectorscale, pg_similarity, pg_tiktoken
fts-stack pgroonga, pg_bigm, zhparser, hunspell
olap-stack pg_duckdb, pg_mooncake, timescaledb, pg_partman, plproxy
feat-stack age, hll, rum, pg_graphql, pg_jsonschema, jsquery
stat-stack pg_show_plans, pg_stat_kcache, pg_qualstats, pg_wait_sampling
supa-stack pg_graphql, pg_jsonschema, wrappers, pgvector, pgsodium, vault

Simply use these names in pg_extensions to install the entire stack.

Extension Resources

13.3 - Packages

Extension package aliases and category naming conventions

Pigsty uses a package alias mechanism to simplify extension installation and management.

Package Alias Mechanism

Managing extensions involves multiple layers of name mapping:

Layer Example pgvector Example postgis
Extension Name vector postgis, postgis_topology, …
Package Alias pgvector postgis
RPM Package Name pgvector_18 postgis36_18*
DEB Package Name postgresql-18-pgvector postgresql-18-postgis-3*

Pigsty provides a package alias abstraction layer, so users don’t need to worry about specific RPM/DEB package names:

pg_extensions: [ pgvector, postgis, timescaledb ]  # Use package aliases

Pigsty automatically translates to the correct package names based on the operating system and PostgreSQL version.

Note

CREATE EXTENSION uses the extension name (for example, vector), not the package alias (pgvector).

Category Aliases

All extensions are organized into 16 categories, which can be batch installed using category aliases:

# Use generic category aliases (auto-adapt to current PG version)
pg_extensions: [ pgsql-gis, pgsql-rag, pgsql-fts ]

# Or use version-specific category aliases
pg_extensions: [ pg18-gis, pg18-rag, pg18-fts ]

Except for the olap category, all category extensions can be installed simultaneously. Within the olap category, there are conflicts: pg_duckdb and pg_mooncake are mutually exclusive.

Category List

Category Description Typical Extensions
time Time-series timescaledb, pg_cron, periods
gis Geospatial postgis, h3, pgrouting
rag Vector/RAG pgvector, pgml, vchord
fts Full-text Search pg_trgm, zhparser, pgroonga
olap Analytics citus, pg_duckdb, pg_mooncake
feat Feature age, pg_graphql, rum
lang Language plpython3u, pljava, plv8
type Data Type hstore, ltree, citext
util Utility http, pg_net, pgjwt
func Function pgcrypto, uuid-ossp, pg_uuidv7
admin Admin pg_repack, pgagent, pg_squeeze
stat Statistics pg_stat_statements, pg_qualstats, auto_explain
sec Security pgaudit, pgcrypto, pgsodium
fdw Foreign Data Wrapper postgres_fdw, mysql_fdw, oracle_fdw
sim Compatibility orafce, babelfishpg_tds
etl Data/ETL pglogical, wal2json, decoderbufs

Browse Extension Catalog

You can browse detailed information about all available extensions on the Pigsty Extension Catalog website, including:

  • Extension name, description, version
  • Supported PostgreSQL versions
  • Supported OS distributions
  • Installation methods, preloading requirements
  • License, source repository

13.4 - Download

Download extension packages from software repositories to local

Before installing extensions, ensure that extension packages are downloaded to the local repository or available from upstream.

Default Behavior

Pigsty downloads the base PostgreSQL 18 kernel packages to the local software repository by default. The default extra download set is repo_extra_packages_default: [ pgsql-main ], which includes the PostgreSQL kernel, client, procedural languages, and basic extension packages such as pg_repack, wal2json, and pgvector.

If you need other extensions from the 575-extension catalog, explicitly add them to repo_extra_packages. Pigsty does not download every extension to the local repository by default.

Benefits of using a local repository:

  • Accelerated installation, avoiding repeated downloads
  • Reduced network traffic consumption
  • Improved delivery reliability
  • Ensured version consistency

Download New Extensions

To download additional extensions, add them to repo_extra_packages and rebuild the repository:

all:
  vars:
    repo_extra_packages: [ pgvector, postgis, timescaledb, pg_duckdb ]
# Re-download packages to local repository
./infra.yml -t repo_build

# Refresh package source cache on all nodes
./node.yml -t node_repo

Using Upstream Repositories

You can also install directly from internet upstream repositories without pre-downloading:

# Add upstream software sources on nodes
./node.yml -t node_repo -e node_repo_modules=node,pgsql

This approach is suitable for:

  • Quick testing of latest versions
  • Installing rare extensions
  • Environments with good network conditions

But may face:

  • Network instability affecting installation
  • Version inconsistency risks

Extension Sources

Extension packages come from two main sources:

Repository Description
PGDG PostgreSQL official repository, providing core extensions
Pigsty Pigsty supplementary repository, providing additional extensions

The Pigsty repository only includes extensions not present in the PGDG repository. Once an extension enters the PGDG repository, the Pigsty repository will remove it or keep it consistent.

Repository URLs:

For detailed repository configuration, see Extension Repository.

13.5 - Install

Install extension packages on cluster nodes

Pigsty uses the operating system’s package manager (yum/apt) to install extension packages.

Related Parameters

Two parameters are used to specify extensions to install:

Parameter Purpose Default Behavior
pg_packages Global common packages Ensure present (no upgrade)
pg_extensions Cluster-specific extensions Install latest version

pg_packages is typically used to specify base components needed by all clusters (PostgreSQL kernel, Patroni, pgBouncer, etc.) and essential extensions.

pg_extensions is used to specify extensions needed by specific clusters.

pg_packages:                           # Global base packages
  - pgsql-main pgsql-common
pg_extensions:                         # Cluster extensions
  - postgis timescaledb pgvector

Install During Cluster Initialization

Declare extensions in cluster configuration, and they will be automatically installed during initialization:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector, pg_duckdb ]

When executing ./pgsql.yml to initialize the cluster, extensions will be automatically installed.

Install Extensions on Existing Cluster

For initialized clusters, there are multiple ways to install extensions:

Using Pigsty Playbook 

# Install using playbook after modifying configuration
./pgsql.yml -l pg-meta -t pg_extension

# Or specify extensions directly on command line
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pg_duckdb"]}'

Using pig Package Manager 

# Install extension using pig
pig install pg_duckdb

# Batch install
ansible pg-meta -b -a 'pig install pg_duckdb pgvector'

Using Package Manager Directly 

# EL systems
sudo yum install -y pg_duckdb_18*

# Debian/Ubuntu systems
sudo apt install -y postgresql-18-pg-duckdb

Using Package Aliases

Pigsty supports using standardized package aliases, automatically translating to package names for the corresponding PG version:

pg_extensions:
  - pgvector           # Auto-translates to pgvector_18* (EL) or postgresql-18-pgvector (Debian)
  - postgis            # Auto-translates to postgis36_18* (EL) or postgresql-18-postgis-3* (Debian)
  - pgsql-gis          # Category alias, installs entire GIS category of extensions

You can also use raw package names directly:

pg_extensions:
  - pgvector_18*                    # EL system raw package name
  - postgresql-18-pgvector          # Debian system raw package name

For package alias definitions, see:

Verify Installation

After installation, verify in the database:

-- Check installed extensions
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- Check if extension files exist
\dx

13.6 - Config

Preload extension libraries and configure extension parameters

Some extensions require preloading dynamic libraries or configuring parameters before use. This section describes how to configure extensions.

Preload Extensions

Most extensions can be enabled directly with CREATE EXTENSION after installation, but some extensions using PostgreSQL’s Hook mechanism require preloading.

Preloading is specified via the shared_preload_libraries parameter and requires a database restart to take effect.

Extensions Requiring Preload

Common extensions that require preloading:

Extension Description
timescaledb Time-series database extension, must be placed first
citus Distributed database extension, must be placed first
pg_stat_statements SQL statement statistics, enabled by default in Pigsty
auto_explain Automatically log slow query execution plans, enabled by default in Pigsty
pg_cron Scheduled task scheduling
pg_net Asynchronous HTTP requests
pg_tle Trusted language extensions
pgaudit Audit logging
pg_stat_kcache Kernel statistics
pg_squeeze Online table space reclamation
pgml PostgresML machine learning

For the complete list, see the Extension Catalog (marked with LOAD).

Preload Order

The loading order of extensions in shared_preload_libraries is important:

  • timescaledb and citus must be placed first
  • If using both, citus should come before timescaledb
  • Statistics extensions should come after pg_stat_statements to use the same query_id
pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain'

Configure During Cluster Initialization

When creating a new cluster, use the pg_libs parameter to specify preloaded extensions:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions: [ timescaledb, postgis, pgvector ]

The value of pg_libs will be written to shared_preload_libraries during cluster initialization.

Default Value

The default value of pg_libs is pg_stat_statements, auto_explain. These two Contrib extensions provide basic observability:

  • pg_stat_statements: Track execution statistics of all SQL statements
  • auto_explain: Automatically log execution plans for slow queries

Modify Configuration on Existing Cluster

For initialized clusters, use patronictl to modify shared_preload_libraries:

# Add timescaledb to preload libraries
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# Restart cluster to apply configuration
pg restart pg-meta

You can also directly modify postgresql.conf or use ALTER SYSTEM:

ALTER SYSTEM SET shared_preload_libraries = 'timescaledb, pg_stat_statements, auto_explain';

A PostgreSQL service restart is required after modification.

Extension Parameter Configuration

Many extensions have configurable parameters that can be set in the following locations:

During Cluster Initialization

Use the pg_parameters parameter to specify:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'pg_cron, pg_stat_statements, auto_explain'
    pg_parameters:
      cron.database_name: postgres           # Database used by pg_cron
      pg_stat_statements.track: all          # Track all statements
      auto_explain.log_min_duration: 1000    # Log queries exceeding 1 second

Runtime Modification

Use ALTER SYSTEM or patronictl:

-- Modify parameter
ALTER SYSTEM SET pg_stat_statements.track = 'all';

-- Reload configuration
SELECT pg_reload_conf();
# Modify using patronictl
pg edit-config pg-meta --force -p 'pg_stat_statements.track=all'

Important Notes

  1. Preload errors prevent startup: If an extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL will not start. Ensure extensions are properly installed before adding to preload.

  2. Modification requires restart: Changes to shared_preload_libraries require restarting the PostgreSQL service to take effect.

  3. Partial functionality available: Some extensions can be partially used without preloading, but full functionality requires preloading.

  4. View current configuration: Use the following command to view current preload libraries:

SHOW shared_preload_libraries;

13.7 - Create

Create and enable extensions in databases

After installing extension packages, you need to execute CREATE EXTENSION in the database to use extension features.

View Available Extensions

After installing extension packages, you can view available extensions:

-- View all available extensions
SELECT * FROM pg_available_extensions;

-- View specific extension
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- View enabled extensions
SELECT * FROM pg_extension;

Create Extensions

Use CREATE EXTENSION to enable extensions in the database:

-- Create extension
CREATE EXTENSION vector;

-- Create extension in specific schema
CREATE EXTENSION postgis SCHEMA public;

-- Automatically install dependent extensions
CREATE EXTENSION postgis_topology CASCADE;

-- Create if not exists
CREATE EXTENSION IF NOT EXISTS vector;
Note

CREATE EXTENSION uses the extension name (for example, vector), not the package alias (pgvector).

Create During Cluster Initialization

Declare extensions in pg_databases, and they will be automatically created during cluster initialization:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions:
          - { name: vector }                         # Use default schema
          - { name: postgis, schema: public }        # Specify schema
          - { name: pg_stat_statements, schema: monitor }

Pigsty will automatically execute CREATE EXTENSION after database creation.

Extensions Requiring Preload

Some extensions must be added to shared_preload_libraries and restarted before creation:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_databases:
      - name: meta
        extensions:
          - { name: timescaledb }  # Requires preload

If you try to create without preloading, you will receive an error message.

Common extensions requiring preload: timescaledb, citus, pg_cron, pg_net, pgaudit, etc. See Configure Extensions.

Extension Dependencies

Some extensions depend on other extensions and need to be created in order:

-- postgis_topology depends on postgis
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;

-- Or use CASCADE to automatically install dependencies
CREATE EXTENSION postgis_topology CASCADE;

Extensions Not Requiring Creation

A few extensions don’t provide SQL interfaces and don’t need CREATE EXTENSION:

Extension Description
wal2json Logical decoding plugin, used directly in replication slots
decoderbufs Logical decoding plugin
decoder_raw Logical decoding plugin

These extensions can be used immediately after installation, for example:

-- Create logical replication slot using wal2json
SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json');

View Extension Information 

-- View extension details
\dx+ vector

-- View objects contained in extension
SELECT * FROM pg_extension_config_dump('vector');

-- View extension version
SELECT extversion FROM pg_extension WHERE extname = 'vector';

13.8 - Update

Upgrade PostgreSQL extension versions

Extension updates involve two levels: package updates (operating system level) and extension object updates (database level).

Update Packages

Use package managers to update extension packages:

# EL systems
sudo yum update pgvector_18*

# Debian/Ubuntu systems
sudo apt update && sudo apt upgrade postgresql-18-pgvector

Batch update using Pigsty:

# Update extension packages for specified cluster
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# Using pig package manager
pig update pgvector

Update Extension Objects

After package updates, extension objects in the database may need to be synchronized.

View Updatable Extensions 

-- View installed extensions and their versions
SELECT name, default_version, installed_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL;

-- View upgradable extensions
SELECT name, installed_version, default_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL
  AND installed_version <> default_version;

Execute Extension Update 

-- Update to latest version
ALTER EXTENSION pgvector UPDATE;

-- Update to specific version
ALTER EXTENSION pgvector UPDATE TO '0.8.0';

View Update Paths 

-- View available upgrade paths for extension
SELECT * FROM pg_extension_update_paths('pgvector');

Important Notes

  1. Backup first: Backup the database before updating extensions, especially for extensions involving data type changes.

  2. Check compatibility: Some extension major version upgrades may be incompatible. Consult the extension’s upgrade documentation.

  3. Preloaded extensions: If updating a preloaded extension (like timescaledb), a database restart may be required after the update.

  4. Dependencies: If other extensions depend on the updated extension, update them in dependency order.

  5. Replication environments: In master-slave replication environments, test updates on slaves first, then update the master after confirmation.

Common Issues

Update Failure

If ALTER EXTENSION UPDATE fails, it may be because:

  • No available upgrade path
  • Extension is in use
  • Insufficient permissions
-- View extension dependencies
SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

Rollback Update

PostgreSQL extensions typically don’t support direct rollback. To rollback:

  1. Restore from backup
  2. Or: Uninstall new version extension, install old version package, recreate extension

13.9 - Remove

Uninstall PostgreSQL extensions

Removing extensions involves two levels: dropping extension objects (database level) and uninstalling packages (operating system level).

Drop Extension Objects

Use DROP EXTENSION to remove extensions from the database:

-- Drop extension
DROP EXTENSION pgvector;

-- If there are dependent objects, cascade delete is required
DROP EXTENSION pgvector CASCADE;

Warning: CASCADE will drop all objects that depend on this extension (tables, functions, views, etc.). Use with caution.

Check Extension Dependencies

It’s recommended to check dependencies before dropping:

-- View objects that depend on an extension
SELECT
    classid::regclass,
    objid,
    deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

-- View tables using extension types
SELECT
    c.relname AS table_name,
    a.attname AS column_name,
    t.typname AS type_name
FROM pg_attribute a
JOIN pg_class c ON a.attrelid = c.oid
JOIN pg_type t ON a.atttypid = t.oid
WHERE t.typname = 'vector';

Remove Preload

If the extension is in shared_preload_libraries, it must be removed from the preload list after dropping:

# Modify shared_preload_libraries, remove extension
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# Restart to apply configuration
pg restart pg-meta

Uninstall Packages

After dropping the extension from the database, you can optionally uninstall the package:

# EL systems
sudo yum remove pgvector_18*

# Debian/Ubuntu systems
sudo apt remove postgresql-18-pgvector

# Using pig package manager
pig remove pgvector

Typically keeping the package doesn’t cause issues. Only uninstall when you need to free disk space or resolve conflicts.

Important Notes

  1. Data loss risk: Using CASCADE will drop dependent objects, potentially causing data loss.

  2. Application compatibility: Ensure applications no longer use the extension’s functionality before dropping.

  3. Preload order: If dropping a preloaded extension, be sure to also remove it from shared_preload_libraries, otherwise the database may fail to start.

  4. Master-slave environments: In replication environments, DROP EXTENSION automatically replicates to slaves.

Operation Sequence

Complete extension removal workflow:

# 1. Check dependencies
psql -d mydb -c "SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');"

# 2. Drop extension from database
psql -d mydb -c "DROP EXTENSION pgvector;"

# 3. If it's a preloaded extension, remove from shared_preload_libraries
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# 4. Restart database (if preload configuration was modified)
pg restart pg-meta

# 5. Optional: Uninstall package
sudo yum remove pgvector_18*

13.10 - Default Extensions

PostgreSQL extensions installed by default in Pigsty

Pigsty installs and enables some core extensions by default when initializing PostgreSQL clusters.

Default Installed Extensions

Extensions installed by default via pg_packages:

Extension Description
pg_repack Handle table bloat online, important maintenance tool
wal2json Logical decoding outputs JSON format changes, commonly used in CDC scenarios
pgvector Vector data type and indexes, installed with pgsql-main by default

The default value of pg_extensions is an empty array []. Declare additional extensions as needed, for example:

Extension Description
postgis Geospatial database extension
timescaledb Time-series database extension
pgvector Vector data type and indexes

Default Enabled Extensions

Extensions enabled by default in all databases via pg_default_extensions:

Extension Schema Description
pg_stat_statements monitor SQL statement execution statistics
pgstattuple monitor Tuple-level statistics
pg_buffercache monitor Buffer cache inspection
pageinspect monitor Page-level inspection
pg_prewarm monitor Relation prewarming
pg_visibility monitor Visibility map inspection
pg_freespacemap monitor Free space map inspection
postgres_fdw public PostgreSQL foreign data wrapper
file_fdw public File foreign data wrapper
btree_gist public B-tree GiST operator classes
btree_gin public B-tree GIN operator classes
pg_trgm public Trigram matching
intagg public Integer aggregator
intarray public Integer array functions
pg_repack repack Online table reorganization

These extensions provide basic monitoring, operations, and feature enhancement capabilities.

Default Preloaded Extensions

Extensions preloaded by default into shared_preload_libraries via pg_libs:

Extension Description
pg_stat_statements Track execution statistics of all SQL statements
auto_explain Automatically log execution plans for slow queries

These two extensions provide basic observability and are strongly recommended to keep.

Customize Default Extensions

You can customize default installed and enabled extensions by modifying configuration parameters:

all:
  vars:
    # Modify default extension packages
    pg_packages:
      - pgsql-main pgsql-common
      - pg_repack_$v* wal2json_$v*

    # Modify default installed extensions
    pg_extensions: [ postgis, timescaledb, pgvector ]

    # Modify default preloaded extensions
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'

    # Modify default enabled extensions
    pg_default_extensions:
      - { name: pg_stat_statements, schema: monitor }
      - { name: pg_repack }
      # ... add more

For detailed extension usage, please refer to:

13.11 - Repository

Pigsty extension software repository configuration

Pigsty provides supplementary extension repositories, offering additional extension packages on top of the PGDG official repository.

YUM Repository

Applicable to EL 8/9/10 and compatible systems (RHEL, Rocky, AlmaLinux, CentOS, etc.).

Add Repository 

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null

# Add repository configuration
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# Refresh cache
sudo yum makecache

China Mainland Mirror 

curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

Repository URLs

APT Repository

Applicable to Debian 12/13 and Ubuntu 22.04/24.04/26.04 and compatible systems.

Add Repository 

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

# Get distribution codename and add repository
distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

# Refresh cache
sudo apt update

China Mainland Mirror 

curl -fsSL https://repo.pigsty.cc/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

Repository URLs

GPG Signature

All packages are signed with GPG:

  • Fingerprint: 9592A7BC7A682E7333376E09E7935D8DB9BD8B20
  • Short ID: B9BD8B20

Repository Policy

The Pigsty repository follows these principles:

  1. Supplementary: Only includes extensions not present in the PGDG repository
  2. Consistency: Once an extension enters the PGDG repository, the Pigsty repository will remove it or keep it consistent
  3. Compatibility: Supports multiple major versions of PostgreSQL 14-18
  4. Multi-platform: Supports x86_64 and aarch64 architectures

Related Resources

14 - PG Kernels

How to use PostgreSQL kernel forks in Pigsty, such as Citus, Babelfish, IvorySQL, PolarDB, and more.

In Pigsty, you can replace the native PostgreSQL kernel with different PostgreSQL “flavors” to unlock specialized capabilities.

Pigsty supports multiple PostgreSQL kernels and compatibility branches so you can get compatibility layers, multi-master replication, graph queries, MPP warehousing, transparent encryption, and more inside one operational framework.

One thing to keep in mind is that not every kernel has the same delivery depth in Pigsty: PostgreSQL, Citus, Babelfish, IvorySQL, PolarDB, AgensGraph, and pgEdge already have relatively clear templates and configuration paths; Cloudberry and Greenplum, by contrast, are more often managed through gpsql mode, and their MPP initialization plus scale-out operations are still better handled with upstream tooling.

Kernel Key Feature Description
PostgreSQL Native kernel, full extension set Vanilla PostgreSQL with 575 extensions
Supabase Backend as a Service PostgreSQL-based BaaS, Firebase alternative
Citus Horizontal scaling Distributed PostgreSQL via native extension
Babelfish SQL Server compatible SQL Server wire-protocol compatibility (PG17/18)
IvorySQL Oracle compatible Oracle syntax and PL/SQL compatibility
OpenHalo MySQL compatible MySQL wire-protocol compatibility
Percona Transparent data encryption Percona distribution with pg_tde
DocumentDB MongoDB migration DocumentDB + FerretDB wire compatibility
OrioleDB OLTP optimization Zheap, no bloat, S3 storage
PolarDB Aurora-style RAC RAC, China-local compliance scenario
Cloudberry Open-source MPP warehouse Cloudberry integrated through gpsql mode
AgensGraph Property graph + Cypher Graph query capability inside PostgreSQL
pgEdge Spock multi-master replication Distributed PostgreSQL for edge scenarios
PostgreSQL forks and compatible kernels

Versions

Kernel Debian / Ubuntu EL
PostgreSQL / Citus PostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
IvorySQL PostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bit PostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bit
Babelfish Babelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit Babelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
PolarDB PostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnu PostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnu
Percona PostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20250617 (Red Hat 14.3.1-2), 64-bit
OrioleDB OrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit OrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
OpenHalo openHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit openHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
DocumentDB PostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
AgensGraph PostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
pgEdge PostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
Cloudberry PostgreSQL 14.4 (Apache Cloudberry 2.0.0-incubating build 1) on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit

14.1 - PostgreSQL

Vanilla PostgreSQL kernel with 575 extensions

PostgreSQL is the world’s most advanced and popular open-source database.

Pigsty installs PostgreSQL 18 by default, supports PostgreSQL 14 ~ 18, and provides 575 PG extensions.

Quick Start

Install Pigsty using the pgsql configuration template.

./configure -c pgsql     # Use postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with native PostgreSQL

Most configuration templates use PostgreSQL kernel by default, for example:

  • meta : Default, postgres with core extensions (vector, postgis, timescale)
  • rich : postgres with all extensions installed
  • slim : postgres only, no monitoring infrastructure
  • ha/full : 4-node sandbox for HA demonstration
  • pgsql : minimal postgres kernel configuration example

Configuration

Vanilla PostgreSQL kernel requires no special adjustments:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utilities
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

Version Selection

To use a different PostgreSQL major version, you can configure it using the -v parameter:

./configure -c pgsql            # Default is postgresql 18, no need to specify explicitly
./configure -c pgsql -v 18      # Explicitly use postgresql 18
./configure -c pgsql -v 17      # Use postgresql 17
./configure -c pgsql -v 16      # Use postgresql 16
./configure -c pgsql -v 15      # Use postgresql 15
./configure -c pgsql -v 14      # Use postgresql 14

If a PostgreSQL cluster is already installed, you need to uninstall it before installing a new version:

./pgsql-rm.yml -l pg-meta # Uninstall cluster pg-meta

Extension Ecosystem

Pigsty provides a rich extension ecosystem for PostgreSQL. See the Extension Catalog for details.

14.2 - Supabase

How to self-host Supabase with Pigsty, deploy an open-source Firebase alternative with a complete backend stack in one click.

Supabase — Build in a weekend, Scale to millions

Supabase is an open-source Firebase alternative that wraps PostgreSQL and provides authentication, out-of-the-box APIs, edge functions, real-time subscriptions, object storage, and vector embedding capabilities. This is a low-code all-in-one backend platform that lets you skip most backend development work, requiring only database design and frontend knowledge to quickly ship products!

Supabase’s motto is: “Build in a weekend, Scale to millions”. Indeed, Supabase is extremely cost-effective at small to micro scales (4c8g), like a cyber bodhisattva. — But when you really scale to millions of users — you should seriously consider self-hosting Supabase — whether for functionality, performance, or cost considerations.

Pigsty provides you with a complete one-click self-hosting solution for Supabase. Self-hosted Supabase enjoys full PostgreSQL monitoring, IaC, PITR, and high availability, and compared to Supabase cloud services, it provides up to 575 out-of-the-box PostgreSQL extensions and can more fully utilize the performance and cost advantages of modern hardware.

For the complete self-hosting tutorial, please refer to: Supabase Self-Hosting Guide

Supabase

Quick Start

Pigsty’s default supabase.yml configuration template defines a single-node Supabase.

First, use Pigsty’s standard installation process to install the Silo and PostgreSQL instances required for Supabase:

 curl -fsSL https://repo.pigsty.io/get | bash
./bootstrap          # Environment check, install dependencies
./configure -c supabase  # Important: modify passwords and other key info in config!
./deploy.yml         # Install Pigsty, deploy PGSQL and MINIO!

Before deploying Supabase, please modify the Supabase parameters in the pigsty.yml config file according to your actual situation (mainly passwords!)

Then, run docker.yml and app.yml to complete the remaining work and deploy Supabase containers:

./docker.yml       # Install Docker module
./app.yml          # Start Supabase stateless components!

For users in China, please configure appropriate Docker mirror sites or proxy servers to bypass GFW to pull DockerHub images. For professional subscriptions, we provide the ability to offline install Pigsty and Supabase without internet access.

Pigsty exposes web services through Nginx on the admin node/INFRA node by default. You can add DNS resolution for supa.pigsty pointing to this node locally, then access https://supa.pigsty through a browser to enter the Supabase Studio management interface.

Default username and password: supabase / pigsty

demo/supabase.cast

14.3 - Babelfish

Use Babelfish (PG17/18) in Pigsty to provide SQL Server protocol/T-SQL compatibility.

Babelfish is a PostgreSQL-based SQL Server compatibility layer, open-sourced by AWS.

Overview

Pigsty lets you deploy Babelfish in mssql mode and provide, on top of PostgreSQL:

  • SQL Server wire protocol compatibility (TDS, 1433)
  • T-SQL compatibility
  • Unified integration with Pigsty capabilities (HA, backup, monitoring, IaC)

In Pigsty v4, Babelfish supports PostgreSQL 17/18. The default template uses pg_version: 17, and Babelfish is part of Pigsty’s standard delivery path with support for all Linux platforms.

Current Behavior

Compared with older Babelfish/PG15 docs, current behavior is:

  • Supported kernel majors are PG17/18; the template defaults to PG17 (pg_version: 17)
  • Default package group: babelfish + pgsql-common + sqlcmd
  • Mainstream platform coverage:
    • OS: el8, el9, el10, d12, d13, u22, u24, u26
    • Arch: x86_64, aarch64
  • mssql template no longer requires an extra mssql repo module (defaults to node,infra,pgsql)

Older docs may still contain deprecated naming. Pigsty now consistently uses Babelfish and babelfish aliases.

Quick Start

Use the built-in Pigsty template:

./configure -c mssql [-v 17/18]
./deploy.yml

After deployment, connect directly with SQL Server clients:

sqlcmd -S <ip>,1433 -U dbuser_mssql -P DBUser.MSSQL -d mssql

Key Configuration

Core parameters in the mssql template:

pg_mode: mssql
pg_version: 17  # optional: 18
pg_packages: [ babelfish, pgsql-common, sqlcmd ]
pg_libs: 'babelfishpg_tds, pg_stat_statements, auto_explain'

pg_databases:
  - name: mssql
    baseline: mssql.sql
    extensions:
      - { name: uuid-ossp }
      - { name: babelfishpg_common }
      - { name: babelfishpg_tsql }
      - { name: babelfishpg_tds }
      - { name: babelfishpg_money }
      - { name: pg_hint_plan }
      - { name: system_stats }
      - { name: tds_fdw }
    parameters: { 'babelfishpg_tsql.migration_mode': 'multi-db' }

pg_hba_rules:
  - { user: dbuser_mssql, db: mssql, addr: intra, auth: md5, order: 525 }

pg_default_services:
  - { name: primary, port: 5433, dest: 1433 }
  - { name: replica, port: 5434, dest: 1433 }

Connectivity and Ports

Babelfish clusters expose two protocol endpoints:

  • PostgreSQL protocol: 5432
  • SQL Server protocol (TDS): 1433

With Pigsty service abstraction you can also use:

  • 5433: fixed route to primary 1433
  • 5434: route to readable node 1433
# Primary write access
sqlcmd -S <any-node-ip>,5433 -U dbuser_mssql -P DBUser.MSSQL

# Read replica query
sqlcmd -S <any-node-ip>,5434 -U dbuser_mssql -P DBUser.MSSQL

Notes

  • Babelfish auth rules must use md5 instead of default scram-sha-256.
  • Default migration mode is multi-db; switch with babelfishpg_tsql.migration_mode if needed.
  • Not all native PostgreSQL extensions are directly usable on Babelfish kernels; validate package availability and compatibility first.
  • Tighten HBA and network exposure for production; do not keep demo-level open rules.

Related Docs

Available Extensions

The Babelfish kernel has 55 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
babelfishpg_common 5.4.0 Transact SQL Datatype Support
babelfishpg_money 1.1.0 babelfishpg_money
babelfishpg_tds 1.0.0 TDS protocol extension
babelfishpg_tsql 5.4.0 Transact SQL compatibility

14.4 - Percona

Percona Postgres distribution with TDE transparent encryption support

Percona Postgres is a patched Postgres kernel with pg_tde (Transparent Data Encryption) extension.

Starting with v4.4.0, Pigsty packages Percona PostgreSQL under the private /usr/pgtde-$v; v4.5.0 keeps this layout prefix (/usr/pgtde-18 for PostgreSQL 18). The pgtde package alias installs both the kernel package and its contrib package, including pg_tde, PostGIS, pgvector, wal2json, pg_repack, pgaudit, and pg_stat_monitor.

Quick Start

Use Pigsty’s standard installation process with the pgtde configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c pgtde     # Use percona postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with Percona PostgreSQL

Configuration

The following parameters need to be adjusted to deploy a Percona cluster:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: pgtde
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pgsql admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - name: meta
        baseline: cmdb.sql
        comment: pigsty tde database
        schemas: [pigsty]
        extensions: [ vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # Percona PostgreSQL TDE kernel settings
    pg_packages: [ pgtde, pgsql-common ]
    pg_libs: 'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'

The pgtde packages are delivered by Pigsty’s pgsql repository module. The legacy percona module is not required by this template.

Available Extensions

The Percona Postgres kernel has 73 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
address_standardizer 3.5.7 Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
address_standardizer_data_us 3.5.7 Address Standardizer US dataset example
pg_repack 1.5.3 Reorganize tables in PostgreSQL databases with minimal locks
pg_stat_monitor 2.3.2 The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram information.
pg_tde 2.2.1 pg_tde access method
pgaudit 18.0 provides auditing functionality
postgis 3.5.7 PostGIS geometry and geography spatial types and functions
postgis_raster 3.5.7 PostGIS raster types and functions
postgis_sfcgal 3.5.7 PostGIS SFCGAL functions
postgis_tiger_geocoder 3.5.7 PostGIS tiger geocoder and reverse geocoder
postgis_topology 3.5.7 PostGIS topology spatial types and functions
set_user 4.2.0 similar to SET ROLE but with added logging
vector 0.8.3 vector data type and ivfflat and hnsw access methods

Key Features

  • Transparent Data Encryption: Provides data-at-rest encryption using the pg_tde extension
  • PostgreSQL 18 Compatible: Based on the Percona PostgreSQL 18 package set
  • Enterprise Extensions: Includes enterprise-grade features like pgaudit, pg_stat_monitor
  • Complete Ecosystem: Supports popular extensions like pgvector, PostGIS

Note: Currently in stable stage - thoroughly evaluate before production use.

14.5 - openHalo

MySQL-compatible Postgres 14 branch

OpenHalo is an open-source PostgreSQL kernel that provides MySQL wire-protocol compatibility.

openHalo is based on PostgreSQL 14.18 and provides wire-level compatibility with MySQL 5.7.32-log / 8.0. Pigsty delivers it through pg_mode: mysql and the openhalo package alias.

Pigsty supports OpenHalo deployment on all supported Linux platforms.

Quick Start

Use Pigsty’s standard installation flow with the mysql template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mysql    # Use MySQL (openHalo) template
./deploy.yml            # Install (change passwords in pigsty.yml before production use)

Cluster Config 

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: postgres, extensions: [ aux_mysql ]} # mysql-compatible database
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # full backup at 1 AM daily

    # OpenHalo specific settings
    pg_mode: mysql
    pg_version: 14
    pg_packages: [ openhalo, pgsql-common ]

OpenHalo provides a dedicated extension, aux_mysql, which includes functions and types needed for MySQL compatibility. Enable it in the postgres database to get full compatibility behavior.

  • aux_mysql 1.5: MySQL Supplementary Extension
  • /usr/halo-14/share/postgresql/extension/aux_mysql.control
  • $libdir/mysm, mysm.so

Usage

For MySQL access, connections still use the postgres database. The MySQL “database” concept maps to PostgreSQL “schema”. So use mysql maps to the mysql schema in the postgres database.

MySQL usernames/passwords are the same PostgreSQL credentials.

Client Access

OpenHalo listens on port 3306 for MySQL wire protocol clients.

Pigsty’s conf/mysql installs a MySQL client by default.

mysql -h 127.0.0.1 -u dbuser_dba

At present, OpenHalo upstream reports Navicat works normally on this port, while IntelliJ DataGrip may fail.

Compatibility Parameters

Pigsty defaults database_compat_mode to mysql. You can further tune compatibility behavior with settings like:

mysql.listener_on = true                        # enable MySQL listener; restart required
mysql.port = 3306                               # second_port for MySQL mode; restart required
mysql.halo_mysql_version = '5.7.32-log'         # restart required
mysql.ci_collation = true                       # restart required
mysql.explicit_defaults_for_timestamp = false   # restart required
mysql.auto_rollback_tx_on_error = false         # restart required

Patch Notes

The OpenHalo kernel packaged by Pigsty is based on HaloTech-Co-Ltd/openHalo with small adjustments:

  • Restore default database name from halo0root to postgres
  • Remove 1.0. prefix in the default version string, keeping 14.18
  • Adjust default config to enable MySQL compatibility and listen on 3306

Pigsty does not provide warranty coverage for OpenHalo kernel behavior. Kernel-specific issues should be addressed with the upstream vendor.

Warning: This kernel is currently in beta1 stage; evaluate risks carefully before production use.

Available Extensions

The OpenHalo kernel has 59 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
aux_mysql 1.5 MySQL Supplementary Extension
hstore_plpython2u 1.0 transform between hstore and plpython2u
hstore_plpythonu 1.0 transform between hstore and plpythonu
jsonb_plpython2u 1.0 transform between jsonb and plpython2u
jsonb_plpythonu 1.0 transform between jsonb and plpythonu
ltree_plpython2u 1.0 transform between ltree and plpython2u
ltree_plpythonu 1.0 transform between ltree and plpythonu

14.6 - OrioleDB

Next-generation OLTP engine for PostgreSQL

OrioleDB is a PostgreSQL storage engine extension that claims to provide 4x OLTP performance, no xid wraparound and table bloat issues, and “cloud-native” (data stored in S3) capabilities.

Pigsty ships OrioleDB as a patched PostgreSQL kernel plus the OrioleDB extension.

You can run OrioleDB as an RDS using Pigsty. Current packages support PostgreSQL 16, 17, and 18 on supported Linux platforms. pg_mode still uses oriole for the /usr/oriole-$v install path, while the orioledb package alias resolves to versioned kernel packages such as orioledb-16, orioledb-17, and orioledb-18. The current Pigsty package line is OrioleDB 1.8 beta16.

Quick Start

Follow Pigsty’s standard installation process using the oriole configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c oriole    # Use OrioleDB configuration template
./deploy.yml             # Install Pigsty with OrioleDB

For production deployment, ensure you modify the password parameters in the pigsty.yml configuration before running the install playbook.

Configuration 

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty], extensions: [orioledb]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # OrioleDB specific settings
    pg_mode: oriole                                         # oriole compatibility mode
    pg_version: 18                                          # OrioleDB supports PG 16, 17, and 18
    pg_packages: [ orioledb, pgsql-common ]                 # Install OrioleDB kernel
    pg_libs: 'orioledb, pg_stat_statements, auto_explain'   # Load OrioleDB extension

Usage

To use OrioleDB, install the orioledb package alias. Pigsty resolves it to the selected PG16, PG17, or PG18 OrioleDB kernel package according to pg_version.

Initialize TPC-B-like tables with pgbench using 100 warehouses:

pgbench -is 100 meta
pgbench -nv -P1 -c10 -S -T1000 meta
pgbench -nv -P1 -c50 -S -T1000 meta
pgbench -nv -P1 -c10    -T1000 meta
pgbench -nv -P1 -c50    -T1000 meta

Next, you can rebuild these tables using the orioledb storage engine and observe the performance difference:

-- Create OrioleDB tables
CREATE TABLE pgbench_accounts_o (LIKE pgbench_accounts INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_branches_o (LIKE pgbench_branches INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_history_o (LIKE pgbench_history INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_tellers_o (LIKE pgbench_tellers INCLUDING ALL) USING orioledb;

-- Copy data from regular tables to OrioleDB tables
INSERT INTO pgbench_accounts_o SELECT * FROM pgbench_accounts;
INSERT INTO pgbench_branches_o SELECT * FROM pgbench_branches;
INSERT INTO pgbench_history_o SELECT  * FROM pgbench_history;
INSERT INTO pgbench_tellers_o SELECT * FROM pgbench_tellers;

-- Drop original tables and rename OrioleDB tables
DROP TABLE pgbench_accounts, pgbench_branches, pgbench_history, pgbench_tellers;
ALTER TABLE pgbench_accounts_o RENAME TO pgbench_accounts;
ALTER TABLE pgbench_branches_o RENAME TO pgbench_branches;
ALTER TABLE pgbench_history_o RENAME TO pgbench_history;
ALTER TABLE pgbench_tellers_o RENAME TO pgbench_tellers;

Key Features

  • No XID Wraparound: Eliminates transaction ID wraparound maintenance
  • No Table Bloat: Advanced storage management prevents table bloat
  • Cloud Storage: Native support for S3-compatible object storage
  • OLTP Optimized: Designed for transactional workloads
  • Improved Performance: Better space utilization and query performance

Note: Currently in Beta stage - thoroughly evaluate before production use.

Available Extensions

The OrioleDB kernel has 53 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
orioledb 1.8 OrioleDB – the next generation transactional engine

14.7 - Cloudberry

Use the Cloudberry open-source MPP data warehouse kernel in Pigsty and manage nodes, monitoring, and configuration through gpsql mode.

Cloudberry is an open-source MPP data warehouse kernel derived from the Greenplum ecosystem, suitable for large-scale parallel analytics workloads.

Overview

In Pigsty, Cloudberry uses gpsql mode and shares the same identity model, monitoring logic, and directory conventions as Greenplum / MatrixDB.

  • Kernel package: cloudberry
  • Mode identifier: pg_mode: gpsql
  • Role flag: gp_role: master | segment
  • Current repo version: Cloudberry 2.1.0
  • Current main package version: DEB 2.1.0-2PIGSTY, RPM 2.1.0-3PIGSTY
  • Default binary directory: /usr/cloudberry

The important boundary is this: Pigsty currently focuses on package delivery, node management, monitoring onboarding, access control, and configuration orchestration for Cloudberry. For MPP cluster initialization, scale-out, rebalance, and other upstream-specific operational actions, you should still use the official Cloudberry toolchain.

The current Pigsty repository provides cloudberry, cloudberry-backup, and cloudberry-pxf packages for both DEB and RPM platforms.

Installation

There is no standalone cloudberry one-click template yet. The more common workflow is:

  1. Enroll the target nodes into Pigsty.
  2. Install the cloudberry kernel package.
  3. Describe the coordinator / segment topology with gpsql mode.
  4. Use Pigsty to unify monitoring, accounts, access control, and backup integration.

If you only need to install the kernel package on a node:

./node.yml -t node_repo    -e '{"node_repo_modules":"local,node,pgsql"}'
./node.yml -t node_install -e '{"node_packages":["cloudberry"]}'

If you are onboarding an existing Cloudberry cluster, it is usually better to keep the original initialization workflow and add Pigsty inventory plus monitoring configuration incrementally.

Configuration

Cloudberry uses gpsql mode rather than a dedicated cloudberry mode. Compared with vanilla PostgreSQL, you at least need to care about the extra identity parameters pg_shard and gp_role; if you want to label shard groups explicitly, you can also add pg_group.

Here is a minimal readable topology example:

all:
  children:
    cb-mdw:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: cb-mdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: master
        pg_packages: [ cloudberry, pgsql-common ]

    cb-sdw:
      hosts:
        10.10.10.11:
          nodename: cb-sdw-1
          pg_instances:
            6000: { pg_cluster: cb-seg1, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
        10.10.10.12:
          nodename: cb-sdw-2
          pg_instances:
            6000: { pg_cluster: cb-seg2, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
      vars:
        pg_cluster: cb-sdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: segment
        pg_preflight_skip: true
        pg_packages: [ cloudberry, pgsql-common ]
        pg_exporter_config: pg_exporter_basic.yml
        pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'

Two details are easy to miss:

  • gp_role: master is for the coordinator / master node, and business access usually lands there.
  • gp_role: segment nodes usually need pg_exporter to connect in utility mode for monitoring.

Client Access

For application and BI access, Cloudberry still exposes the PostgreSQL wire protocol, so most PostgreSQL-compatible clients, drivers, and BI tools can connect without special handling.

But keep the following in mind:

  • Applications and analytics queries should connect to the master / coordinator, not directly to segment nodes.
  • Segment nodes are better treated as data/compute shards and monitoring targets.
  • If you want a unified access endpoint, you can still use Pigsty’s HAProxy / PgBouncer / DNS service abstractions.

Extensions and Ecosystem

Cloudberry comes from the PostgreSQL ecosystem, but it is not simply “vanilla PostgreSQL plus a few extensions”. For the extension packages already available in Pigsty, it is better to think in two categories:

  • Pure SQL objects or components with weak ABI coupling are usually easier to adapt.
  • Extensions that depend on PGXS or the kernel C ABI often need separate validation or even recompilation against the Cloudberry version and toolchain.

If your workload depends on postgis, vector extensions, FDWs, auditing, or custom C extensions, validate them on the target Cloudberry version first rather than copying a vanilla PostgreSQL extension list unchanged.

Notes

  • Cloudberry currently has no dedicated Pigsty template, so you should model it manually with gpsql mode.
  • The current delivery focus is packages, configuration, and monitoring; it does not replace the official Cloudberry MPP initialization and scale-out toolchain.
  • Because this is an MPP distributed kernel, vanilla PostgreSQL operational assumptions do not automatically transfer to every Patroni / PgBouncer / PgBackRest node role.
  • If you need horizontal PostgreSQL scaling rather than a full MPP warehouse, Citus is usually the better first choice.

Related Docs

14.8 - AgensGraph

Use the AgensGraph (PG17) graph database kernel in Pigsty to get property graph and Cypher/SQL hybrid query capabilities within the PostgreSQL ecosystem.

AgensGraph is a property graph database kernel built on PostgreSQL, supporting openCypher queries and mixed Cypher/SQL workflows.

Overview

Pigsty integrates AgensGraph through pg_mode: agens while preserving most of the standard PostgreSQL operational model.

  • Kernel package: agensgraph
  • Mode identifier: pg_mode: agens
  • Current template version: AgensGraph 2.17.0
  • Current version string: PostgreSQL 17.10 (AgensGraph 2.17.0)
  • Built-in template: agens
  • Typical use cases: graph relationship analysis, path queries, knowledge graphs, and risk/association analysis layered onto relational data

From the client side, AgensGraph still speaks the PostgreSQL wire protocol, so normal PostgreSQL clients, drivers, and connection pools can connect directly. The real difference from vanilla PostgreSQL is not how you connect, but that the database now contains graph objects, Cypher syntax, and the agtype data type.

Installation

Use the built-in Pigsty template:

./configure -c agens
./deploy.yml

The agens template automatically enables pg_mode: agens and installs the agensgraph kernel package. After deployment, verify the kernel version:

psql -d meta -c "SELECT version();"

Configuration

Key configuration for AgensGraph in Pigsty:

all:
  vars:
    node_repo_modules: node,infra,pgsql
    pg_version: 17

  children:
    pg-meta:
      vars:
        pg_mode: agens
        pg_packages: [ agensgraph, pgsql-common ]

AgensGraph does not require a special preload stack like pgEdge or Babelfish, so most standard Pigsty patterns for HA, backup, monitoring, access control, and IaC remain unchanged. If your workload is dominated by graph traversal and complex path queries, focus on work_mem, shared_buffers, and planner cost settings instead of assuming default OLTP habits will fit.

Usage

After connecting to the database, the usual first step is to create a graph and set graph_path:

CREATE GRAPH g;
SET graph_path = g;

Create labels, vertices, and edges:

CREATE VLABEL person;
CREATE ELABEL knows;

CREATE (:person {name: 'Jack'});
CREATE (:person {name: 'Emily'})-[:knows]->(:person {name: 'Tom'});

Run graph queries and updates:

MATCH (:person {name: 'Emily'})-[:knows]->(v:person)
RETURN v.name;

MATCH (v:person {name: 'Jack'})
SET v.age = '24';

To call Cypher from within SQL, use the cypher() function:

SELECT *
FROM cypher('g', $$ MATCH (v:person) RETURN v.name $$) AS (name agtype);

In real projects, the more common pattern is to mix “relational tables + graph labels + Cypher queries”: transactions, privileges, and backup workflows still follow PostgreSQL, while graph analysis logic lives in AgensGraph graph objects and the cypher() interface.

Notes

  • AgensGraph is currently fixed to the PG17-compatible line, so do not assume PG18 extension availability will carry over.
  • The default agens template is single-node for quick validation; production deployments should extend to an HA topology.
  • Not all third-party PostgreSQL extensions are guaranteed to work on the AgensGraph kernel; verify compatibility first.
  • Graph objects and relational objects can coexist in the same database, but in production it is usually better to define clear database or naming conventions so they do not become tangled together.
  • Tune memory and cost parameters based on your graph model scale; do not blindly use defaults.
  • For compatibility or semantic issues with the AgensGraph kernel, consult the official manual and upstream issues first.

Related Docs

Available Extensions

The AgensGraph kernel has 60 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
meta 1.0 Utility functions for agensgraph

14.9 - pgEdge

Use the pgEdge (PG15-18) kernel in Pigsty to build distributed PostgreSQL for edge scenarios on top of Spock multi-master logical replication.

pgEdge is a distributed PostgreSQL distribution for edge scenarios, built on Spock multi-master logical replication.

Overview

Pigsty integrates pgEdge through pg_mode: pgedge and delivers it through the standard PostgreSQL cluster workflow:

  • pgedge: a PG15, PG16, PG17, and PG18 compatible kernel; the template defaults to PG18
  • spock: Active-active multi-master logical replication
  • snowflake: Distributed unique sequences
  • lolor: Large object logical replication compatibility layer

The current Pigsty repository ships versioned pgEdge kernel packages for pgedge-15, pgedge-16, pgedge-17, and pgedge-18; the template defaults to pg_version: 18. The spock, snowflake, and lolor control files, SQL files, and shared libraries are bundled in the pgedge-$v kernel package, so they are no longer listed as separate pg_extensions packages to install. From the client side, pgEdge is still PostgreSQL wire compatible, so psql, JDBC/ODBC, DBeaver, and similar tools work as usual.

The delivery model in Pigsty is: validate the kernel on a single node first, then expand to a multi-node replication topology. The template handles the kernel, extensions, monitoring, backup, and access control out of the box, but the actual multi-master topology still needs to be designed around your workload consistency and conflict strategy.

Installation

Use the built-in Pigsty template:

./configure -c pgedge
./deploy.yml

The template pre-installs spock, snowflake, and lolor in the meta database. After deployment, verify the kernel and extensions:

psql -d meta -c "SELECT version();"
psql -d meta -c "SELECT extname, extversion FROM pg_extension WHERE extname IN ('spock','snowflake','lolor') ORDER BY 1;"

For the full template and parameters, see: pgedge config template.

Configuration

Key parameters in the pgedge template (matching conf/pgedge.yml):

pg_mode: pgedge
pg_version: 18
pg_packages: [ pgedge, pgsql-common ]
pg_libs: 'spock, lolor, pg_stat_statements, auto_explain'
pg_databases:
  - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [spock, snowflake, lolor] }

If you plan to grow into a multi-node multi-master topology, it is better to configure logical replication capacity and snowflake.node explicitly:

pg_parameters:
  wal_level: logical
  max_replication_slots: 16
  max_wal_senders: 16
  'snowflake.node': 1

snowflake.node must be unique on every writable node, otherwise distributed IDs will collide.

Usage

The common workflow in Pigsty is still: validate the kernel on a single node first, then expand into a multi-node Spock replication topology.

If you need these capabilities in a business database as well, create the extensions first:

CREATE EXTENSION IF NOT EXISTS spock;
CREATE EXTENSION IF NOT EXISTS snowflake;
CREATE EXTENSION IF NOT EXISTS lolor;

Then use the Spock SQL API or the pgEdge CLI to create nodes, replication sets, and subscriptions. If your schema already uses serial or identity, plan the snowflake sequence migration before enabling multi-master writes, otherwise cross-node primary key collisions are likely.

Notes

  • Replication in pgEdge is organized per database, not as an instance-wide “turn everything into multi-master” switch.
  • Replicated tables should have a PRIMARY KEY or an appropriate REPLICA IDENTITY.
  • UNLOGGED and TEMPORARY tables do not participate in Spock logical replication.
  • Spock configuration and operations typically require superuser privileges, so production deployments should define privilege boundaries clearly.
  • If your workload depends on large object replication, use lolor explicitly rather than assuming native large objects will replicate correctly.
  • Cross-region multi-master is not a checkbox feature. Network latency, conflict handling, and the write model all need to be evaluated first.

Related Docs

Available Extensions

The pgEdge kernel has 63 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
lolor 1.2.2 Large Objects support for logical replication
snowflake 2.5.0 Snowflake style IDs for PostgreSQL
spock 5.0.10 PostgreSQL Logical Replication

14.10 - DocumentDB

DocumentDB and FerretDB provide MongoDB wire-protocol compatibility

DocumentDB is an open-source PostgreSQL document database extension maintained by Microsoft. FerretDB is a stateless protocol translation proxy built on top of it. Together, they expose a MongoDB wire-compatible endpoint from a standard PostgreSQL kernel: applications using MongoDB drivers can connect directly, while requests are translated into PostgreSQL operations.

Unlike other kernel variants, this is not a standalone PostgreSQL fork. The data layer runs native PostgreSQL 16–18 and is managed by the standard PGSQL module. Persistence, transactions, high availability, backup, monitoring, and access control remain PostgreSQL responsibilities; FerretDB is deployed as a Pigsty Docker APP and handles only protocol translation.

Pigsty is a FerretDB community partner, provides binary packages for FerretDB and the DocumentDB extensions, and delivers the complete stack out of the box through the mongo configuration template.

Quick Start

Use Pigsty’s standard installation flow with the mongo configuration template:

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mongo    # Use the Mongo (DocumentDB + FerretDB) configuration template
./deploy.yml            # Install; change passwords in pigsty.yml first for production
./docker.yml -l pg-meta # Install Docker on the pg-meta node
./app.yml -l pg-meta    # Deploy the FerretDB Docker APP

FerretDB listens on the local loopback address at port 27017 by default. Connect with mongosh or any MongoDB-compatible client:

mongosh 'mongodb://mongod:DBUser.Mongo@127.0.0.1:27017/'

Configuration

Source: pigsty/conf/mongo.yml. See the Mongo configuration template documentation for the complete template.

The key PostgreSQL settings are the documentdb extension and its preload libraries, plus the backend superuser used by FerretDB:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: mongod ,password: DBUser.Mongo ,superuser: true ,comment: FerretDB backend user }
    pg_databases:
      - { name: postgres, extensions: [ documentdb, postgis, vector, pg_cron, rum ]}
    pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
    pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'

FerretDB is deployed as a Docker APP. Its settings are ordinary overrides under apps.ferretdb.conf, and the container connects to the local primary service on port 5436 through host.docker.internal:

docker_enabled: true
app: ferretdb
apps:
  ferretdb:
    conf:
      FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
      FERRETDB_POSTGRESQL_URL: 'postgres://mongod:DBUser.Mongo@host.docker.internal:5436/postgres?pool_min_conns=1&pool_max_conns=20'
      FERRETDB_BIND_ADDR: 127.0.0.1
      FERRETDB_PORT: 27017
      FERRETDB_AUTH: true
      FERRETDB_TELEMETRY: disabled

High Availability

Because FerretDB is fully stateless, its HA topology follows the standard PostgreSQL cluster pattern. The template retains a commented three-node pg-mongo example. Each node runs a FerretDB container bound to local port 27018, and HAProxy aggregates them behind the floating endpoint 10.10.10.4:27017 (mongo.pigsty).

Patroni and etcd continue to manage PostgreSQL failover. The Mongo endpoint automatically recovers after the primary switches.

Notes

  • FerretDB enables authentication by default (FERRETDB_AUTH: true) but does not yet implement MongoDB authorization roles. PostgreSQL users and HBA rules remain the actual security boundary.
  • Client-side MongoDB TLS is disabled by default, and the Mongo endpoint is not exposed to the network. Change FERRETDB_BIND_ADDR only when remote access is required.
  • The backend cluster uses standard PostgreSQL parameters, playbooks, and dashboards; there is no independent FERRET module or mongo_* parameter group.
  • Repeat an authenticated CRUD smoke test after upgrading FerretDB or DocumentDB.

14.11 - Citus

Deploy native high-availability Citus horizontally sharded clusters with Pigsty, seamlessly scaling PostgreSQL across multiple shards and accelerating OLTP/OLAP queries.

Pigsty natively supports Citus. This is a distributed horizontal scaling extension based on the native PostgreSQL kernel.

Citus

Installation

Citus is a PostgreSQL extension plugin that can be installed and enabled on a native PostgreSQL cluster following the standard plugin installation process.

./pgsql.yml -t pg_extension -e '{"pg_extensions":["citus"]}'

Configuration

To define a citus cluster, you need to specify the following parameters:

  • pg_mode must be set to citus instead of the default pgsql
  • You must define the shard name pg_shard and shard number pg_group on each shard cluster
  • You must define pg_primary_db to specify the database managed by Patroni
  • If you want to use postgres from pg_dbsu instead of the default pg_admin_username to execute admin commands, then pg_dbsu_password must be set to a non-empty plaintext password

Additionally, you need extra hba rules to allow SSL access from localhost and other data nodes.

You can define each Citus cluster as a separate group, like standard PostgreSQL clusters. The current complete template is conf/ha/citus.yml:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # Global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # If using dbsu, you need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

You can also specify identity parameters for all Citus cluster members within a single group, as shown in conf/ha/citus.yml:

#==========================================================#
# pg-citus: 10 node citus cluster (5 x primary-replica pair)
#==========================================================#
pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                    # pgsql cluster mode: citus
    pg_shard: pg-citus                # citus shard name: pg-citus
    pg_primary_db: test               # primary database used by citus
    pg_dbsu_password: DBUser.Postgres # all dbsu password access for citus cluster
    pg_vip_enabled: true
    pg_vip_interface: auto
    pg_extensions: [ 'citus postgis timescaledb pgvector' ]
    pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain' # citus will be added by patroni automatically
    pg_users: [ { name: test ,password: test ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: test ,owner: test ,extensions: [ { name: citus }, { name: postgis } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 10.10.10.0/24 ,auth: trust ,title: 'trust citus cluster members'        }
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Usage

You can access any node just like accessing a regular cluster:

pgbench -i postgres://test:test@pg-citus0/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus0/test

By default, changes you make to one Shard only occur on that cluster and are not synchronized to other Shards.

If you want to distribute writes across all Shards, you can use the API functions provided by Citus to mark tables as:

  • Distributed tables (automatic partitioning, requires specifying partition key)
  • Reference tables (full replication: does not require specifying partition key)

Starting from Citus 11.2, any Citus database node can play the role of coordinator, meaning any primary node can write:

psql -h pg-citus0 -d test -c "SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');"

After distributing the tables, you can also access them on other nodes:

psql -h pg-citus1 -d test -c '\dt+'

For example, a full table scan will show that the execution plan has become a distributed plan:

vagrant@meta-1:~$ psql -h pg-citus3 -d test -c 'explain select * from pgbench_accounts'
                                               QUERY PLAN
---------------------------------------------------------------------------------------------------------
 Custom Scan (Citus Adaptive)  (cost=0.00..0.00 rows=100000 width=352)
   Task Count: 32
   Tasks Shown: One of 32
   ->  Task
         Node: host=10.10.10.52 port=5432 dbname=test
         ->  Seq Scan on pgbench_accounts_102008 pgbench_accounts  (cost=0.00..81.66 rows=3066 width=97)
(6 rows)

You can initiate writes from several different primary nodes:

pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus1/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus2/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus3/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus4/test

When a node fails, the native high availability support provided by Patroni will promote the standby node and automatically take over.

test=# select * from  pg_dist_node;
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.51 |     5432 | default  | t           | t        | primary  | default     | t              | f
      2 |       2 | 10.10.10.54 |     5432 | default  | t           | t        | primary  | default     | t              | t
      5 |       1 | 10.10.10.52 |     5432 | default  | t           | t        | primary  | default     | t              | t
      3 |       4 | 10.10.10.58 |     5432 | default  | t           | t        | primary  | default     | t              | t
      4 |       3 | 10.10.10.56 |     5432 | default  | t           | t        | primary  | default     | t              | t

14.12 - IvorySQL

Use HighGo’s open-source IvorySQL kernel to achieve Oracle syntax/PLSQL compatibility based on PostgreSQL clusters.

IvorySQL is an open-source PostgreSQL kernel fork that aims to provide “Oracle compatibility” based on PG.

Overview

The Pigsty PGSQL repository directly provides IvorySQL 5.4 packages compatible with PostgreSQL 18.4 across the currently supported EL, Debian, Ubuntu, and dual-architecture platforms. Online installation uses Pigsty’s pgsql repository; the Professional Edition also provides offline delivery for the corresponding platforms.

IvorySQL

Pigsty’s ivorysql package alias points to IvorySQL 5, compatible with PostgreSQL 18. Real package names are mapped by platform variables under roles/node_id/vars/; for example, EL uses ivorysql5, while Debian/Ubuntu uses ivorysql-5.

The last IvorySQL version supporting EL7 was 3.3, corresponding to PostgreSQL 16.3; the last version based on PostgreSQL 17 is IvorySQL 4.4

Installation

Install with Pigsty’s built-in ivory configuration template:

./configure -c ivory
./deploy.yml

Configuration

The following parameters need to be configured for IvorySQL database clusters:

#----------------------------------#
# Ivory SQL Configuration
#----------------------------------#
node_repo_modules: node,infra,pgsql       # use Pigsty node/infra/pgsql repos
pg_mode: ivory                    # IvorySQL Oracle Compatible Mode
pg_packages: [ ivorysql, pgsql-common ]
pg_libs: 'liboracle_parser, pg_stat_statements, auto_explain'
pg_extensions: [ ]                # do not install any vanilla postgresql extensions

When using Oracle compatibility mode, you need to dynamically load the liboracle_parser extension plugin.

Client Access

IvorySQL 5 is equivalent to PostgreSQL 18, and any client tool compatible with the PostgreSQL wire protocol can access IvorySQL clusters.

Available Extensions

The IvorySQL kernel has 95 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
address_standardizer 3.5.4 Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
address_standardizer_data_us 3.5.4 Address Standardizer US dataset example
age 1.7.0 AGE database extension
ddlx 0.31 DDL eXtractor functions
gb18030_2022 1.0 support gb18030 2022 with extension
http 1.7 HTTP client for PostgreSQL, allows web page retrieval inside the database.
ivorysql_ora 1.0 Oracle Compatible extenison on Postgres Database
ora_btree_gin 1.0 support for indexing oracle datatypes in GIN
ora_btree_gist 1.0 support for oracle indexing common datatypes in GiST
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_cron 1.6 Job scheduler for PostgreSQL
pg_curl 2.4 PostgreSQL cURL allows most curl actions, including data transfer with URL syntax via HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP
pg_get_functiondef 1.0 Get function’s definition
pg_hint_plan 1.8.0 optimizer hints for PostgreSQL
pg_jieba 1.1.1 a parser for full-text search of Chinese
pg_partman 5.3.1 Extension to manage partitioned tables by time or ID
pg_show_plans 2.1 show query plans of all currently running SQL statements
pg_stat_monitor 2.3 The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram information.
pg_textsearch 0.1.0 Full-text search with BM25 ranking
pgagent 4.2 A PostgreSQL job scheduler
pgaudit 18.0 provides auditing functionality
pgroonga 4.0.4 Super fast and all languages supported full text search index based on Groonga
pgroonga_database 4.0.4 PGroonga database management module
pgrouting 3.8.0 pgRouting Extension
plisql 1.0 PL/iSQL procedural language
plpgsql_check 2.8 extended check for plpgsql functions
postgis 3.5.4 PostGIS geometry and geography spatial types and functions
postgis_raster 3.5.4 PostGIS raster types and functions
postgis_sfcgal 3.5.4 PostGIS SFCGAL functions
postgis_tiger_geocoder 3.5.4 PostGIS tiger geocoder and reverse geocoder
postgis_topology 3.5.4 PostGIS topology spatial types and functions
redis_fdw 1.0 Foreign data wrapper for querying a Redis server
system_stats 3.0 EnterpriseDB system statistics for PostgreSQL
vector 0.8.1 vector data type and ivfflat and hnsw access methods
zhparser 2.3 a parser for full-text search of Chinese

Please note that Pigsty does not assume any warranty responsibility for using the IvorySQL kernel. Any issues or requirements encountered when using this kernel should be addressed with the original vendor.

14.13 - PolarDB PG

Using Alibaba Cloud’s open-source PolarDB for PostgreSQL kernel to provide domestic innovation qualification support, with Oracle RAC-like user experience.

Overview

Pigsty allows you to create PostgreSQL clusters with “domestic innovation qualification” credentials using PolarDB!

PolarDB for PostgreSQL now uses PostgreSQL 17 as its base. The polar template, default path, and extension notes in Pigsty have all been updated to PG17. Any client tool compatible with the PostgreSQL wire protocol can access PolarDB clusters.

Pigsty’s PGSQL repository provides PolarDB PG open-source installation packages, but they are not downloaded to the local software repository during Pigsty installation.

PolarDB for PostgreSQL

Installation

Use the built-in Pigsty template:

./configure -c polar
./deploy.yml

Change Summary

Starting with Pigsty v4.4, the PolarDB PG kernel uses packages built and maintained by Pigsty. The main changes are:

Item Old docs / old default Current
Kernel baseline PostgreSQL 15 PostgreSQL 17
Default PolarDB path /u01/polardb_pg /usr/polar-17
Supported architectures x86_64 x86_64, aarch64
Available extensions Old docs said 61 pg_available_extensions returns 93; 34 remain after filtering contrib
Replication user requirement replicator must be SUPERUSER unchanged

Configuration

The following parameters need special configuration for PolarDB database clusters:

#----------------------------------#
# PGSQL & PolarDB
#----------------------------------#
pg_version: 17
pg_mode: polar
pg_packages: [ polardb, pgsql-common ]
pg_exporter_exclude_database: 'template0,template1,postgres,polardb_admin'
pg_default_roles:
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator   ,superuser: true  ,replication: true ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator } # <- superuser is required for replication
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

The default installation directory for the polar kernel has moved to /usr/polar-17. One important difference is that PolarDB PG still requires the replicator replication user to be SUPERUSER, unlike vanilla PostgreSQL.

Available Extensions

The PolarDB PG kernel has 93 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
hll 2.18 type for storing hyperloglog data
ip4r 2.4
log_fdw 1.4 foreign-data wrapper for Postgres log file access
pase 0.0.1 ant ai similarity search
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_cron 1.5 Job scheduler for PostgreSQL
pg_cron_preload 1.0 polardb pg extend catalog
pg_hint_plan 1.7.0 optimizer hints for PostgreSQL
pg_jieba 1.1.0 a parser for full-text search of Chinese
pg_partman 5.2.4 Extension to manage partitioned tables by time or ID
pg_profile 4.10 PostgreSQL load profile repository and report builder
pg_repack 1.5.1-1 Reorganize tables in PostgreSQL databases with minimal locks
pg_similarity 1.0 support similarity queries
pg_squeeze 1.9 A tool to remove unused space from a relation.
pg_stat_kcache 2.3.0 Kernel statistics gathering
pgaudit 17.1 provides auditing functionality
pgtap 1.3.3 Unit testing for PostgreSQL
pldbgapi 1.1 server-side support for debugging PL/pgSQL functions
polar_advisor 1.1 polar_advisor
polar_feature_utils 1.0 PolarDB feature utilization
polar_io_stat 1.0 polar io stat in multi dimension
polar_monitor 1.3 monitor functions for PolarDB
polar_monitor_preload 1.0 examine the polardb information
polar_parameter_manager 1.2 Extension to select parameters for manger.
polar_proxy_utils 1.0 Extension to provide operations about proxy.
polar_resource_manager 1.0 a background process that forcibly frees user session process memory
polar_smgrperf 1.0 smgr perf test extension
polar_tde_utils 1.0 Internal extension for TDE
polar_vfs 1.0 polar virtual file system for different storage
polar_worker 1.1 polar_worker
prefix 1.2.0 Prefix Range module for PostgreSQL
roaringbitmap 0.5 support for Roaring Bitmaps
sequential_uuids 1.0.3 generator of sequential UUIDs
varbitx 1.1 varbit functions pack

14.14 - PolarDB Oracle

Using Alibaba Cloud’s commercial PolarDB for Oracle kernel (closed source, PG14, only available in special enterprise edition customization)

Pigsty allows you to create PolarDB for Oracle clusters with “domestic innovation qualification” credentials using PolarDB!

According to the Security and Reliability Evaluation Results Announcement (No. 1, 2023), Appendix 3, Centralized Database. PolarDB v2.0 is an autonomous, controllable, secure, and reliable domestic innovation database.

PolarDB for Oracle is an Oracle-compatible version developed based on PolarDB for PostgreSQL. Both share the same kernel, distinguished by the --compatibility-mode parameter.

We collaborate with the Alibaba Cloud kernel team to provide a complete database solution based on PolarDB v2.0 kernel and Pigsty. Please contact sales for inquiries, or purchase on Alibaba Cloud Marketplace.

The PolarDB for Oracle kernel is currently only available on EL7 (CentOS 7) systems.

PolarDB for Oracle

Extensions

Currently, the PolarDB 2.0 (Oracle compatible) kernel comes with the following 188 extension plugins:

name default_version comment
cube 1.5 data type for multidimensional cubes
ip4r 2.4 NULL
adminpack 2.1 administrative functions for PostgreSQL
dict_xsyn 1.0 text search dictionary template for extended synonym processing
amcheck 1.4 functions for verifying relation integrity
autoinc 1.0 functions for autoincrementing fields
hstore 1.8 data type for storing sets of (key, value) pairs
bloom 1.0 bloom access method - signature file based index
earthdistance 1.1 calculate great-circle distances on the surface of the Earth
hstore_plperl 1.0 transform between hstore and plperl
bool_plperl 1.0 transform between bool and plperl
file_fdw 1.0 foreign-data wrapper for flat file access
bool_plperlu 1.0 transform between bool and plperlu
fuzzystrmatch 1.1 determine similarities and distance between strings
hstore_plperlu 1.0 transform between hstore and plperlu
btree_gin 1.3 support for indexing common datatypes in GIN
hstore_plpython2u 1.0 transform between hstore and plpython2u
btree_gist 1.6 support for indexing common datatypes in GiST
hll 2.17 type for storing hyperloglog data
hstore_plpython3u 1.0 transform between hstore and plpython3u
citext 1.6 data type for case-insensitive character strings
hstore_plpythonu 1.0 transform between hstore and plpythonu
hypopg 1.3.1 Hypothetical indexes for PostgreSQL
insert_username 1.0 functions for tracking who changed a table
dblink 1.2 connect to other PostgreSQL databases from within a database
decoderbufs 0.1.0 Logical decoding plugin that delivers WAL stream changes using a Protocol Buffer format
intagg 1.1 integer aggregator and enumerator (obsolete)
dict_int 1.0 text search dictionary template for integers
intarray 1.5 functions, operators, and index support for 1-D arrays of integers
isn 1.2 data types for international product numbering standards
jsonb_plperl 1.0 transform between jsonb and plperl
jsonb_plperlu 1.0 transform between jsonb and plperlu
jsonb_plpython2u 1.0 transform between jsonb and plpython2u
jsonb_plpython3u 1.0 transform between jsonb and plpython3u
jsonb_plpythonu 1.0 transform between jsonb and plpythonu
lo 1.1 Large Object maintenance
log_fdw 1.0 foreign-data wrapper for csvlog
ltree 1.2 data type for hierarchical tree-like structures
ltree_plpython2u 1.0 transform between ltree and plpython2u
ltree_plpython3u 1.0 transform between ltree and plpython3u
ltree_plpythonu 1.0 transform between ltree and plpythonu
moddatetime 1.0 functions for tracking last modification time
old_snapshot 1.0 utilities in support of old_snapshot_threshold
oracle_fdw 1.2 foreign data wrapper for Oracle access
oss_fdw 1.1 foreign-data wrapper for OSS access
pageinspect 2.1 inspect the contents of database pages at a low level
pase 0.0.1 ant ai similarity search
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_freespacemap 1.2 examine the free space map (FSM)
pg_hint_plan 1.4 controls execution plan with hinting phrases in comment of special form
pg_buffercache 1.5 examine the shared buffer cache
pg_prewarm 1.2 prewarm relation data
pg_repack 1.4.8-1 Reorganize tables in PostgreSQL databases with minimal locks
pg_sphere 1.0 spherical objects with useful functions, operators and index support
pg_cron 1.5 Job scheduler for PostgreSQL
pg_jieba 1.1.0 a parser for full-text search of Chinese
pg_stat_kcache 2.2.1 Kernel statistics gathering
pg_stat_statements 1.9 track planning and execution statistics of all SQL statements executed
pg_surgery 1.0 extension to perform surgery on a damaged relation
pg_trgm 1.6 text similarity measurement and index searching based on trigrams
pg_visibility 1.2 examine the visibility map (VM) and page-level visibility info
pg_wait_sampling 1.1 sampling based statistics of wait events
pgaudit 1.6.2 provides auditing functionality
pgcrypto 1.3 cryptographic functions
pgrowlocks 1.2 show row-level locking information
pgstattuple 1.5 show tuple-level statistics
pgtap 1.2.0 Unit testing for PostgreSQL
pldbgapi 1.1 server-side support for debugging PL/pgSQL functions
plperl 1.0 PL/Perl procedural language
plperlu 1.0 PL/PerlU untrusted procedural language
plpgsql 1.0 PL/pgSQL procedural language
plpython2u 1.0 PL/Python2U untrusted procedural language
plpythonu 1.0 PL/PythonU untrusted procedural language
plsql 1.0 Oracle compatible PL/SQL procedural language
pltcl 1.0 PL/Tcl procedural language
pltclu 1.0 PL/TclU untrusted procedural language
polar_bfile 1.0 The BFILE data type enables access to binary file LOBs that are stored in file systems outside Database
polar_bpe 1.0 polar_bpe
polar_builtin_cast 1.1 Internal extension for builtin casts
polar_builtin_funcs 2.0 implement polar builtin functions
polar_builtin_type 1.5 polar_builtin_type for PolarDB
polar_builtin_view 1.5 polar_builtin_view
polar_catalog 1.2 polardb pg extend catalog
polar_channel 1.0 polar_channel
polar_constraint 1.0 polar_constraint
polar_csn 1.0 polar_csn
polar_dba_views 1.0 polar_dba_views
polar_dbms_alert 1.2 implement polar_dbms_alert - supports asynchronous notification of database events.
polar_dbms_application_info 1.0 implement polar_dbms_application_info - record names of executing modules or transactions in the database.
polar_dbms_pipe 1.1 implements polar_dbms_pipe - package lets two or more sessions in the same instance communicate.
polar_dbms_aq 1.2 implement dbms_aq - provides an interface to Advanced Queuing.
polar_dbms_lob 1.3 implement dbms_lob - provides subprograms to operate on BLOBs, CLOBs, and NCLOBs.
polar_dbms_output 1.2 implement polar_dbms_output - enables you to send messages from stored procedures.
polar_dbms_lock 1.0 implement polar_dbms_lock - provides an interface to Oracle Lock Management services.
polar_dbms_aqadm 1.3 polar_dbms_aqadm - procedures to manage Advanced Queuing configuration and administration information.
polar_dbms_assert 1.0 implement polar_dbms_assert - provide an interface to validate properties of the input value.
polar_dbms_metadata 1.0 implement polar_dbms_metadata - provides a way for you to retrieve metadata from the database dictionary.
polar_dbms_random 1.0 implement polar_dbms_random - a built-in random number generator, not intended for cryptography
polar_dbms_crypto 1.1 implement dbms_crypto - provides an interface to encrypt and decrypt stored data.
polar_dbms_redact 1.0 implement polar_dbms_redact - provides an interface to mask data from queries by an application.
polar_dbms_debug 1.1 server-side support for debugging PL/SQL functions
polar_dbms_job 1.0 polar_dbms_job
polar_dbms_mview 1.1 implement polar_dbms_mview - enables to refresh materialized views.
polar_dbms_job_preload 1.0 polar_dbms_job_preload
polar_dbms_obfuscation_toolkit 1.1 implement polar_dbms_obfuscation_toolkit - enables an application to get data md5.
polar_dbms_rls 1.1 implement polar_dbms_rls - a fine-grained access control administrative built-in package
polar_multi_toast_utils 1.0 polar_multi_toast_utils
polar_dbms_session 1.2 implement polar_dbms_session - support to set preferences and security levels.
polar_odciconst 1.0 implement ODCIConst - Provide some built-in constants in Oracle.
polar_dbms_sql 1.2 implement polar_dbms_sql - provides an interface to execute dynamic SQL.
polar_osfs_toolkit 1.0 osfs library tools and functions extension
polar_dbms_stats 14.0 stabilize plans by fixing statistics
polar_monitor 1.5 monitor functions for PolarDB
polar_osfs_utils 1.0 osfs library utils extension
polar_dbms_utility 1.3 implement polar_dbms_utility - provides various utility subprograms.
polar_parameter_check 1.0 kernel extension for parameter validation
polar_dbms_xmldom 1.0 implement dbms_xmldom and dbms_xmlparser - support standard DOM interface and xml parser object
polar_parameter_manager 1.1 Extension to select parameters for manager.
polar_faults 1.0.0 simulate some database faults for end user or testing system.
polar_monitor_preload 1.1 examine the polardb information
polar_proxy_utils 1.0 Extension to provide operations about proxy.
polar_feature_utils 1.2 PolarDB feature utilization
polar_global_awr 1.0 PolarDB Global AWR Report
polar_publication 1.0 support polardb pg logical replication
polar_global_cache 1.0 polar_global_cache
polar_px 1.0 Parallel Execution extension
polar_serverless 1.0 polar serverless extension
polar_resource_manager 1.0 a background process that forcibly frees user session process memory
polar_sys_context 1.1 implement polar_sys_context - returns the value of parameter associated with the context namespace at the current instant.
polar_gpc 1.3 polar_gpc
polar_tde_utils 1.0 Internal extension for TDE
polar_gtt 1.1 polar_gtt
polar_utl_encode 1.2 implement polar_utl_encode - provides functions that encode RAW data into a standard encoded format
polar_htap 1.1 extension for PolarDB HTAP
polar_htap_db 1.0 extension for PolarDB HTAP database level operation
polar_io_stat 1.0 polar io stat in multi dimension
polar_utl_file 1.0 implement utl_file - support PL/SQL programs can read and write operating system text files
polar_ivm 1.0 polar_ivm
polar_sql_mapping 1.2 Record error sqls and mapping them to correct one
polar_stat_sql 1.0 Kernel statistics gathering, and sql plan nodes information gathering
tds_fdw 2.0.2 Foreign data wrapper for querying a TDS database (Sybase or Microsoft SQL Server)
xml2 1.1 XPath querying and XSLT
polar_upgrade_catalogs 1.1 Upgrade catalogs for old version instance
polar_utl_i18n 1.1 polar_utl_i18n
polar_utl_raw 1.0 implement utl_raw - provides SQL functions for manipulating RAW datatypes.
timescaledb 2.9.2 Enables scalable inserts and complex queries for time-series data
polar_vfs 1.0 polar virtual file system for different storage
polar_worker 1.0 polar_worker
postgres_fdw 1.1 foreign-data wrapper for remote PostgreSQL servers
refint 1.0 functions for implementing referential integrity (obsolete)
roaringbitmap 0.5 support for Roaring Bitmaps
tsm_system_time 1.0 TABLESAMPLE method which accepts time in milliseconds as a limit
vector 0.5.0 vector data type and ivfflat and hnsw access methods
rum 1.3 RUM index access method
unaccent 1.1 text search dictionary that removes accents
seg 1.4 data type for representing line segments or floating-point intervals
sequential_uuids 1.0.2 generator of sequential UUIDs
uuid-ossp 1.1 generate universally unique identifiers (UUIDs)
smlar 1.0 compute similarity of any one-dimensional arrays
varbitx 1.1 varbit functions pack
sslinfo 1.2 information about SSL certificates
tablefunc 1.0 functions that manipulate whole tables, including crosstab
tcn 1.0 Triggered change notifications
zhparser 1.0 a parser for full-text search of Chinese
address_standardizer 3.3.2 Ganos PostGIS address standardizer
address_standardizer_data_us 3.3.2 Ganos PostGIS address standardizer data us
ganos_fdw 6.0 Ganos Spatial FDW extension for POLARDB
ganos_geometry 6.0 Ganos geometry lite extension for POLARDB
ganos_geometry_pyramid 6.0 Ganos Geometry Pyramid extension for POLARDB
ganos_geometry_sfcgal 6.0 Ganos geometry lite sfcgal extension for POLARDB
ganos_geomgrid 6.0 Ganos geometry grid extension for POLARDB
ganos_importer 6.0 Ganos Spatial importer extension for POLARDB
ganos_networking 6.0 Ganos networking
ganos_pointcloud 6.0 Ganos pointcloud extension For POLARDB
ganos_pointcloud_geometry 6.0 Ganos_pointcloud LIDAR data and ganos_geometry data for POLARDB
ganos_raster 6.0 Ganos raster extension for POLARDB
ganos_scene 6.0 Ganos scene extension for POLARDB
ganos_sfmesh 6.0 Ganos surface mesh extension for POLARDB
ganos_spatialref 6.0 Ganos spatial reference extension for POLARDB
ganos_trajectory 6.0 Ganos trajectory extension for POLARDB
ganos_vomesh 6.0 Ganos volume mesh extension for POLARDB
postgis_tiger_geocoder 3.3.2 Ganos PostGIS tiger geocoder
postgis_topology 3.3.2 Ganos PostGIS topology

14.15 - PostgresML

How to deploy PostgresML with Pigsty: ML, training, inference, Embedding, RAG inside DB.

PostgresML is a PostgreSQL extension that supports the latest large language models (LLM), vector operations, classical machine learning, and traditional Postgres application workloads.

PostgresML (pgml) is a PostgreSQL extension written in Rust. You can run standalone Docker images, but this documentation is not a docker-compose template introduction, for reference only.

PostgresML officially supports Ubuntu 22.04, but we also maintain RPM versions for EL 8/9, if you don’t need CUDA and NVIDIA-related features.

You need internet access on database nodes to download Python dependencies from PyPI and models from HuggingFace.

PostgresML is Deprecated

Because the company behind it has ceased operations.

Configuration

PostgresML is an extension written in Rust. Pigsty maintains prebuilt packages for PG14-17 on EL8/EL9 and Debian/Ubuntu platforms.

Creating a New Cluster

PostgresML 2.10.0 is available for PostgreSQL 14-17. The example below uses PG17; if you use PG14-16, change pg_version to the corresponding major version.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_version: 17
    pg_libs: 'pgml, pg_stat_statements, auto_explain'
    pg_extensions: [ pgml, pgvector, wal2json, pg_repack ]

Pigsty resolves the pgml package alias to the platform-specific package name: pgml_$v on EL and postgresql-$v-pgml on Debian/Ubuntu. You also need to add pgml to pg_libs.

Enabling on an Existing Cluster

To enable pgml on an existing cluster, you can install it using Ansible’s package module:

ansible pg-meta -m package -b -a 'name=pgml_17'
# ansible el8,el9 -m package -b -a 'name=pgml_17'              # EL 8/9
# ansible u22,u24 -m package -b -a 'name=postgresql-17-pgml'   # Debian/Ubuntu

Python Dependencies

You also need to install PostgresML’s Python dependencies on cluster nodes. Official tutorial: Installation Guide

Install Python and PIP

Ensure python3, pip, and venv are installed:

# Ubuntu 22.04 (python3.10), need to install pip and venv using apt
sudo apt install -y python3 python3-pip python3-venv

For EL 8 / EL9 and compatible distributions, you can use python3.11:

# EL 8/9, can upgrade the default pip and virtualenv
sudo yum install -y python3.11 python3.11-pip       # install latest python3.11
python3.11 -m pip install --upgrade pip virtualenv  # use python3.11 on EL8 / EL9
Using PyPI Mirrors

For users in mainland China, we recommend using Tsinghua University’s PyPI mirror.

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple    # set global mirror (recommended)
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple some-package        # use for single installation

Install Dependencies

Create a Python virtual environment and use pip to install dependencies from requirements.txt and requirements-xformers.txt.

If you’re using EL 8/9, replace python3 with python3.11 in the following commands.

su - postgres;                          # create virtual environment as database superuser
mkdir -p /data/pgml; cd /data/pgml;     # create virtual environment directory
python3    -m venv /data/pgml           # create virtual environment directory (Ubuntu 22.04)
source /data/pgml/bin/activate          # activate virtual environment

# write Python dependencies and install with pip
cat > /data/pgml/requirements.txt <<EOF
accelerate==0.22.0
auto-gptq==0.4.2
bitsandbytes==0.41.1
catboost==1.2
ctransformers==0.2.27
datasets==2.14.5
deepspeed==0.10.3
huggingface-hub==0.17.1
InstructorEmbedding==1.0.1
lightgbm==4.1.0
orjson==3.9.7
pandas==2.1.0
rich==13.5.2
rouge==1.0.1
sacrebleu==2.3.1
sacremoses==0.0.53
scikit-learn==1.3.0
sentencepiece==0.1.99
sentence-transformers==2.2.2
tokenizers==0.13.3
torch==2.0.1
torchaudio==2.0.2
torchvision==0.15.2
tqdm==4.66.1
transformers==4.33.1
xgboost==2.0.0
langchain==0.0.287
einops==0.6.1
pynvml==11.5.0
EOF

# install dependencies using pip in the virtual environment
python3 -m pip install -r /data/pgml/requirements.txt
python3 -m pip install xformers==0.0.21 --no-dependencies

# additionally, 3 Python packages need to be installed globally using sudo!
sudo python3 -m pip install xgboost lightgbm scikit-learn

Enable PostgresML

After installing the pgml extension and Python dependencies on all cluster nodes, you can enable pgml on the PostgreSQL cluster.

Use the patronictl command to configure the cluster, add pgml to shared_preload_libraries, and specify your virtual environment directory in pgml.venv:

shared_preload_libraries: pgml, timescaledb, pg_stat_statements, auto_explain
pgml.venv: '/data/pgml'

Then restart the database cluster and create the extension using SQL commands:

CREATE EXTENSION vector;        -- also recommend installing pgvector!
CREATE EXTENSION pgml;          -- create PostgresML in the current database
SELECT pgml.version();          -- print PostgresML version information

If everything is normal, you should see output similar to the following:

# create extension pgml;
INFO:  Python version: 3.11.2 (main, Oct  5 2023, 16:06:03) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
INFO:  Scikit-learn 1.3.0, XGBoost 2.0.0, LightGBM 4.1.0, NumPy 1.26.1
CREATE EXTENSION

# SELECT pgml.version(); -- print PostgresML version information
 version
---------
 2.7.8

Done! For more details, please refer to the official PostgresML documentation: https://postgresml.org/docs/guides/use-cases/

14.16 - Greenplum

Deploy/Monitor Greenplum clusters with Pigsty, build Massively Parallel Processing (MPP) PostgreSQL data warehouse clusters!

Pigsty supports deploying Greenplum clusters and its derivative distribution YMatrixDB, and provides the capability to integrate existing Greenplum deployments into Pigsty monitoring.

Overview

Greenplum / YMatrix cluster deployment capabilities are only available in the professional/enterprise editions and are not currently open source.

Installation

Pigsty provides installation packages for Greenplum 6 (@el7) and Greenplum 7 (@el8). Open source users can install and configure them manually.

# EL 7 Only (Greenplum6)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-6"]}'

# EL 8 Only (Greenplum7)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-7"]}'

Configuration

To define a Greenplum cluster, you need to use pg_mode = gpsql and additional identity parameters pg_shard and gp_role.

#================================================================#
#                        GPSQL Clusters                          #
#================================================================#

#----------------------------------#
# cluster: mx-mdw (gp master)
#----------------------------------#
mx-mdw:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , nodename: mx-mdw-1 }
  vars:
    gp_role: master          # this cluster is used as greenplum master
    pg_shard: mx             # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-mdw       # this master cluster name is mx-mdw
    pg_databases:
      - { name: matrixmgr , extensions: [ { name: matrixdbts } ] }
      - { name: meta }
    pg_users:
      - { name: meta , password: DBUser.Meta , pgbouncer: true }
      - { name: dbuser_monitor , password: DBUser.Monitor , roles: [ dbrole_readonly ], superuser: true }

    pgbouncer_enabled: true                # enable pgbouncer for greenplum master
    pgbouncer_exporter_enabled: false      # enable pgbouncer_exporter for greenplum master
    pg_exporter_params: 'host=127.0.0.1&sslmode=disable'  # use 127.0.0.1 as local monitor host

#----------------------------------#
# cluster: mx-sdw (gp master)
#----------------------------------#
mx-sdw:
  hosts:
    10.10.10.11:
      nodename: mx-sdw-1        # greenplum segment node
      pg_instances:             # greenplum segment instances
        6000: { pg_cluster: mx-seg1, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg2, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
    10.10.10.12:
      nodename: mx-sdw-2
      pg_instances:
        6000: { pg_cluster: mx-seg2, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633  }
        6001: { pg_cluster: mx-seg3, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634  }
    10.10.10.13:
      nodename: mx-sdw-3
      pg_instances:
        6000: { pg_cluster: mx-seg3, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg1, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
  vars:
    gp_role: segment               # these are nodes for gp segments
    pg_shard: mx                   # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-sdw             # these segment clusters name is mx-sdw
    pg_preflight_skip: true        # skip preflight check (since pg_seq & pg_role & pg_cluster not exists)
    pg_exporter_config: pg_exporter_basic.yml                             # use basic config to avoid segment server crash
    pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'  # use gp_role = utility to connect to segments

Additionally, PG Exporter requires extra connection parameters to connect to Greenplum Segment instances for metric collection.

14.17 - Neon

Use Neon’s open-source Serverless PostgreSQL kernel to build flexible, scale-to-zero, forkable PG services.

Neon adopts a storage and compute separation architecture, providing seamless autoscaling, scale to zero, and unique database branching capabilities.

Neon official website: https://neon.tech/

Neon binaries are currently too large to include in the open-source package set. This support path remains in pilot stage; contact Pigsty sales if you need it.

15 - Param Templates

Use Pigsty’s built-in Patroni config templates or customize your own

Pigsty provides four preset Patroni/PostgreSQL config templates optimized for different workloads:

Template CPU Cores Use Case Characteristics
/docs/pgsql/template/oltp.yml 4-128C OLTP transactions High concurrency, low latency
/docs/pgsql/template/olap.yml 4-128C OLAP analytics Large queries, high parallelism
/docs/pgsql/template/crit.yml 4-128C Consistency-first Consistency-first, detailed auditing
/docs/pgsql/template/tiny.yml 1-3C Tiny instances Resource-constrained envs

Use pg_conf to select a template; default is /docs/pgsql/template/oltp.yml.

The database tuning template pg_conf should be paired with the OS tuning template node_tune.

All four standard templates set wal_level to logical. PostgreSQL 18.6 adds the output_plugin_libraries security allowlist; Pigsty permits the built-in pgoutput and test_decoding plugins plus wal2json, which is installed by the default pgsql-main package set. To use another logical-decoding output plugin, review its code and privilege boundary, then add its exact library name through pg_parameters. Patroni filters the template setting on older PostgreSQL versions that do not support it.

Usage

Set pg_conf in your cluster definition. It’s recommended to set node_tune accordingly for OS-level tuning:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: oltp.yml    # PostgreSQL config template (default)
    node_tune: oltp      # OS tuning template (default)

For critical financial workloads, use /docs/pgsql/template/crit.yml:

pg-finance:
  hosts:
    10.10.10.21: { pg_seq: 1, pg_role: primary }
    10.10.10.22: { pg_seq: 2, pg_role: replica }
    10.10.10.23: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-finance
    pg_conf: crit.yml    # PostgreSQL critical template
    node_tune: crit      # OS critical tuning

For low-spec VMs or dev environments, use /docs/pgsql/template/tiny.yml:

pg-dev:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL tiny template
    node_tune: tiny      # OS tiny tuning

Comparison

The four templates differ significantly in key parameters:

Connections & Memory

Parameter OLTP OLAP CRIT TINY
max_connections 500/1000 500 500/1000 250
work_mem range 64MB-1GB 64MB-8GB 64MB-1GB 16MB-256MB
maintenance_work_mem 25% shmem 50% shmem 25% shmem 25% shmem
max_locks_per_transaction 1-2x maxconn 2-4x maxconn 1-2x maxconn 1-2x maxconn

Parallel Query

Parameter OLTP OLAP CRIT TINY
max_worker_processes max(cpu+16, 24) max(cpu+20, 28) max(cpu+16, 24) max(cpu+12, 20)
max_parallel_workers 50% cpu 80% cpu 50% cpu 50% cpu
max_parallel_workers_per_gather 20% cpu (max 8) 50% cpu 0 (off) 0 (off)
parallel_setup_cost 2000 1000 2000 1000
parallel_tuple_cost 0.2 0.1 0.2 0.1

Sync Replication

Parameter OLTP OLAP CRIT TINY
synchronous_mode depends pg_rpo depends pg_rpo forced on depends pg_rpo
data_checksums optional optional forced on optional

Vacuum Config

Parameter OLTP OLAP CRIT TINY
vacuum_cost_delay 20ms 10ms 20ms 20ms
vacuum_cost_limit 2000 10000 2000 2000
autovacuum_max_workers 3 3 3 2

Timeout & Security

Parameter OLTP OLAP CRIT TINY
idle_in_transaction_session_timeout 10min off 1min 10min
log_min_duration_statement 100ms 1000ms 100ms 100ms
default_statistics_target 400 1000 400 200
track_activity_query_size 8KB 8KB 32KB 8KB
log_connections auth auth full default

IO Config (PG18)

Parameter OLTP OLAP CRIT TINY
io_workers 25% cpu (4-16) 50% cpu (4-32) 25% cpu (4-8) 3
temp_file_limit 1/20 disk, max 100GB 1/5 disk, max 400GB 1/20 disk, max 100GB 1/20 disk, max 100GB

Selection Guide

  • OLTP Template: Default choice for most transaction processing. Ideal for e-commerce, social, gaming apps.

  • OLAP Template: For data warehouses, BI reports, ETL. Allows large queries, high parallelism, relaxed timeouts.

  • CRIT Template: For financial transactions, core accounting with strict consistency/security requirements. Forced sync replication, checksums, full audit.

  • TINY Template: For dev/test environments, resource-constrained VMs, Raspberry Pi. Minimizes resource usage, disables parallel queries.

Custom Templates

Create custom templates based on existing ones. Templates are in roles/pgsql/templates/:

roles/pgsql/templates/
├── oltp.yml    # OLTP template (default)
├── olap.yml    # OLAP template
├── crit.yml    # CRIT critical template
└── tiny.yml    # TINY micro template

Steps to create a custom template:

  1. Copy an existing template as base
  2. Modify parameters as needed
  3. Place in roles/pgsql/templates/
  4. Reference via pg_conf

Example:

cp roles/pgsql/templates/oltp.yml roles/pgsql/templates/myapp.yml
# Edit myapp.yml as needed

Then use in your cluster:

pg-myapp:
  vars:
    pg_conf: myapp.yml

Templates use Jinja2 syntax; parameters are dynamically computed based on node resources (CPU, memory, disk).

Tuning Strategy

For technical details on template parameter optimization, see Tuning Strategy:

  • Memory tuning (shared buffers, work mem, max connections)
  • CPU tuning (parallel query worker config)
  • Storage tuning (WAL size, temp file limits)
  • Manual parameter adjustment

Related Parameters

  • pg_conf: PostgreSQL config template
  • node_tune: OS tuning template, should match pg_conf
  • pg_rto: Recovery time objective, affects failover timeout
  • pg_rpo: Candidate-replica lag threshold; setting it to 0 enables synchronous replication in the general templates
  • pg_max_conn: Override template max connections
  • pg_shared_buffer_ratio: Shared buffer memory ratio
  • pg_storage_type: Storage type, affects IO params

15.1 - Parameter Optimization Policy

Learn the parameter optimization strategies Pigsty uses for the 4 different PostgreSQL workload scenarios.

Pigsty provides four scenario-based parameter templates by default, which can be specified and used through the pg_conf parameter.

  • tiny.yml: Optimized for small nodes, VMs, and demos (the template is labeled for 1-3 cores)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

Pigsty adopts different parameter optimization strategies for these four default scenarios, as shown below:

Memory Parameter Tuning

Pigsty automatically detects the system’s memory size and uses it as the basis for setting the maximum number of connections and memory-related parameters.

  • pg_max_conn: PostgreSQL maximum connections, auto will use recommended values for different scenarios
  • pg_shared_buffer_ratio: Shared buffer memory ratio, default is 0.25

By default, Pigsty uses 25% of memory as PostgreSQL shared buffers. The rest is shared among connections, work_mem, background processes, and the operating-system cache.

By default, if the user has not set a pg_max_conn maximum connections value, Pigsty will use defaults according to the following rules:

  • oltp: 500 (pgbouncer) / 1000 (postgres)
  • crit: 500 (pgbouncer) / 1000 (postgres)
  • tiny: 250
  • olap: 500

For OLTP and CRIT templates, if the service is not pointing to the pgbouncer connection pool but directly connects to the postgres database, the maximum connections will be doubled to 1000.

After determining the maximum connections, work_mem is calculated from shared memory size / maximum connections and limited to the range of 64MB ~ 1GB.

{% raw %}
{% if pg_max_conn != 'auto' and pg_max_conn|int >= 20 %}{% set pg_max_connections = pg_max_conn|int %}{% else %}{% if pg_default_service_dest|default('postgres') == 'pgbouncer' %}{% set pg_max_connections = 500 %}{% else %}{% set pg_max_connections = 1000 %}{% endif %}{% endif %}
{% set pg_max_prepared_transactions = pg_max_connections if 'citus' in pg_libs else 0 %}
{% set pg_max_locks_per_transaction = (2 * pg_max_connections)|int if 'citus' in pg_libs or 'timescaledb' in pg_libs else pg_max_connections %}
{% set pg_shared_buffers = (node_mem_mb|int * pg_shared_buffer_ratio|float) | round(0, 'ceil') | int %}
{% set pg_maintenance_mem = (pg_shared_buffers|int * 0.25)|round(0, 'ceil')|int %}
{% set pg_effective_cache_size = node_mem_mb|int - pg_shared_buffers|int  %}
{% set pg_workmem =  ([ ([ (pg_shared_buffers / pg_max_connections)|round(0,'floor')|int , 64 ])|max|int , 1024])|min|int %}
{% endraw %}

CPU Parameter Tuning

In PostgreSQL, there are 4 important parameters related to parallel queries. Pigsty automatically optimizes parameters based on the current system’s CPU cores. The templates first calculate a parallel/extension worker budget and then add another eight reserved slots when writing max_worker_processes. The final GUC is therefore eight higher than the intermediate variable defined near the top of each template.

OLTP Setting Logic Range Limits
max_worker_processes max(CPU + 8, 16) + 8 max(CPU + 16, 24)
max_parallel_workers max(ceil(50% CPU), 2) 1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather min(max(ceil(20% CPU), 2),8) 1/5 CPU rounded down, minimum 2, max 8
OLAP Setting Logic Range Limits
max_worker_processes max(CPU + 12, 20) + 8 max(CPU + 20, 28)
max_parallel_workers max(ceil(80% CPU, 2)) 4/5 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather max(floor(50% CPU), 2) 1/2 CPU rounded up, minimum 2
CRIT Setting Logic Range Limits
max_worker_processes max(CPU + 8, 16) + 8 max(CPU + 16, 24)
max_parallel_workers max(ceil(50% CPU), 2) 1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather 0, enable as needed
TINY Setting Logic Range Limits
max_worker_processes max(CPU + 4, 12) + 8 max(CPU + 12, 20)
max_parallel_workers max(floor(50% CPU), 1) 50% CPU rounded down, minimum 1
max_parallel_maintenance_workers max(floor(33% CPU), 1) 33% CPU rounded down, minimum 1
max_parallel_workers_per_gather 0 Disables parallel gather per query

Note that the CRIT and TINY templates disable parallel queries by setting max_parallel_workers_per_gather = 0. Users can enable parallel queries as needed by setting this parameter.

Both OLTP and CRIT templates additionally set the following parameters, doubling the parallel query cost to reduce the tendency to use parallel queries.

parallel_setup_cost: 2000           # double from 100 to increase parallel cost
parallel_tuple_cost: 0.2            # double from 0.1 to increase parallel cost
min_parallel_table_scan_size: 32MB  # 4x default 8MB, prefer non-parallel scan
min_parallel_index_scan_size: 2MB   # 4x default 512kB, prefer non-parallel scan

Note that adjustments to the max_worker_processes parameter only take effect after a restart. Additionally, when a replica’s configuration value for this parameter is higher than the primary’s, the replica will fail to start. This parameter must be adjusted through Patroni configuration management, which ensures consistent primary-replica configuration and prevents new replicas from failing to start during failover.

Storage Space Parameters

Pigsty automatically detects the total space of the disk where the /data/postgres main data directory is located and uses it as the basis for specifying the following parameters:

{% raw %}
{% set pg_size_twentieth = ([([(node_fs_bytes|int / 21474836480)|round(0, 'ceil')|int, 1])|max, 100])|min %}
min_wal_size: {{ ([pg_size_twentieth, 200])|min }}GB                  # 1/20 disk size, max 200GB
max_wal_size: {{ ([pg_size_twentieth * 4, 2000])|min }}GB             # 2/10 disk size, max 2000GB
max_slot_wal_keep_size: {{ ([pg_size_twentieth * 6, 3000])|min }}GB   # 3/10 disk size, max 3000GB
temp_file_limit: {{ ([pg_size_twentieth, 200])|min }}GB               # 1/20 of disk size, max 200GB
{% endraw %}
  • pg_size_twentieth is one twentieth of disk capacity rounded up, clamped to 1-100GB.
  • Therefore, in the three standard templates, the effective cap for temp_file_limit and min_wal_size is 100GB.
  • The effective cap for max_wal_size is 400GB.
  • The effective cap for max_slot_wal_keep_size is 600GB.

The OLAP template sets temp_file_limit to pg_size_twentieth × 4, for an effective cap of 400GB. Existing 200GB/2TB/3TB comments at the ends of template lines do not account for the 100GB cap already applied to pg_size_twentieth; the rendered expression is authoritative.

Manual Parameter Tuning

In addition to using Pigsty’s automatically configured parameters, you can also manually tune PostgreSQL parameters.

Use the pg edit-config <cluster> command to interactively edit cluster configuration:

pg edit-config pg-meta

Or use the -p parameter to directly set parameters:

pg edit-config -p log_min_duration_statement=1000 pg-meta
pg edit-config --force -p shared_preload_libraries='timescaledb, pg_cron, pg_stat_statements, auto_explain' pg-meta

You can also use the Patroni REST API to modify configuration:

curl -u 'postgres:Patroni.API' \
    -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
    -s -X PATCH http://10.10.10.10:8008/config | jq .

15.2 - OLTP Template

PostgreSQL config template optimized for online transaction processing workloads

oltp.yml is Pigsty’s default config template, optimized for online transaction processing (OLTP). Designed for 4-128 core CPUs with high concurrency, low latency, and high throughput.

Pair with node_tune = oltp for OS-level tuning.

Use Cases

OLTP template is ideal for:

  • E-commerce: Order processing, inventory, user transactions
  • Social apps: User feeds, messaging, following relationships
  • Gaming backends: Player data, leaderboards, game state
  • SaaS applications: Multi-tenant business systems
  • Web apps: CRUD-intensive workloads

Workload characteristics:

  • Many short transactions (millisecond-level)
  • High concurrent connections (hundreds to thousands)
  • Read/write ratio typically 7:3 to 9:1
  • Latency-sensitive, requires fast response
  • High data consistency requirements

Usage

oltp.yml is the default template, no explicit specification needed:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-oltp
    # pg_conf: oltp.yml  # PostgreSQL config template (default)
    # node_tune: oltp    # OS tuning template (default)

Or explicitly specify:

pg-oltp:
  vars:
    pg_conf: oltp.yml    # PostgreSQL config template
    node_tune: oltp      # OS tuning template

Parameter Details

Connection Management 

max_connections: 500/1000   # depends on pgbouncer usage
superuser_reserved_connections: 10
  • When pg_default_service_dest is pgbouncer, max_connections is set to 500
  • When traffic connects directly to PostgreSQL, max_connections is set to 1000
  • Override via pg_max_conn parameter

Memory Config

OLTP template memory allocation strategy:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 25% For VACUUM, CREATE INDEX
work_mem 64MB - 1GB Based on shared_buffers/max_connections
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation:

work_mem = min(max(shared_buffers / max_connections, 64MB), 1GB)

Ensures each connection has sufficient sort/hash memory without over-allocation.

Parallel Query

OLTP template moderately limits parallel queries to prevent resource contention:

max_worker_processes: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 2)
max_parallel_workers_per_gather: 20% × cpu (2-8)
max_parallel_maintenance_workers: 33% × cpu (min 2)

Parallel cost estimates are increased to favor serial execution:

parallel_setup_cost: 2000      # 2x default (1000)
parallel_tuple_cost: 0.2       # 2x default (0.1)
min_parallel_table_scan_size: 32MB   # 4x default (8MB), prefer non-parallel scan
min_parallel_index_scan_size: 2MB    # 4x default (512kB), prefer non-parallel scan

WAL Config 

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
wal_buffers: 16MB
wal_writer_delay: 20ms
wal_writer_flush_after: 1MB
commit_delay: 20
commit_siblings: 10
checkpoint_timeout: 15min
checkpoint_completion_target: 0.80

Balances data safety and write performance.

Vacuum Config 

vacuum_cost_delay: 20ms         # sleep after each vacuum round
vacuum_cost_limit: 2000         # cost limit per vacuum round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08    # 8% table change triggers vacuum
autovacuum_analyze_scale_factor: 0.04   # 4% table change triggers analyze
autovacuum_freeze_max_age: 1000000000

Conservative vacuum settings avoid impacting online transaction performance.

Query Optimization 

random_page_cost: 1.1           # SSD optimized
effective_io_concurrency: 200   # SSD concurrent IO
default_statistics_target: 400  # Statistics precision

Enables planner to generate better query plans.

Logging & Monitoring 

log_min_duration_statement: 100         # log queries > 100ms
log_statement: ddl                      # log DDL statements
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024                    # log temp files > 1MB
log_autovacuum_min_duration: 1s
track_io_timing: on
track_functions: all
track_activity_query_size: 8192

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min

10-minute idle transaction timeout prevents zombie transactions holding locks.

Extension Config 

shared_preload_libraries: 'pg_stat_statements, auto_explain'

# auto_explain
auto_explain.log_min_duration: 1s
auto_explain.log_analyze: on
auto_explain.log_verbose: on
auto_explain.log_timing: on
auto_explain.log_nested_statements: true

# pg_stat_statements
pg_stat_statements.max: 10000
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

Template Comparison

Feature OLTP OLAP CRIT
max_connections 500-1000 500 500-1000
work_mem 64MB-1GB 64MB-8GB 64MB-1GB
Parallel query Moderate limit Aggressive Disabled
Vacuum intensity Conservative Aggressive Conservative
Txn timeout 10min Disabled 1min
Slow query threshold 100ms 1000ms 100ms

Why OLTP over OLAP?

  • Queries are mostly simple point/range lookups
  • Transaction response time requires milliseconds
  • High concurrent connections
  • No complex analytical queries

Why OLTP over CRIT?

  • Small probability of data loss acceptable (async replication)
  • Complete audit logs not required
  • Better write performance desired

Performance Tuning Tips

Connection Pooling

For high concurrency, use PgBouncer connection pool:

pg-oltp:
  vars:
    pg_default_service_dest: pgbouncer  # default
    pgbouncer_poolmode: transaction     # transaction-level pooling

Read Separation

Use read replicas to share read load:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }

Monitoring Metrics

Focus on these metrics:

  • Connections: Active/waiting connection counts
  • Transaction rate: TPS, commit/rollback ratio
  • Response time: Query latency percentiles (p50/p95/p99)
  • Lock waits: Lock wait time, deadlock counts
  • Replication lag: Replica delay time and bytes

References

15.3 - OLAP Template

PostgreSQL config template optimized for online analytical processing workloads

olap.yml is optimized for online analytical processing (OLAP). Designed for 4-128 core CPUs with support for large queries, high parallelism, relaxed timeouts, and aggressive vacuum.

Pair with node_tune = olap for OS-level tuning.

Use Cases

OLAP template is ideal for:

  • Data warehouses: Historical data storage, multidimensional analysis
  • BI reports: Complex report queries, dashboard data sources
  • ETL processing: Data extraction, transformation, loading
  • Data analysis: Ad-hoc queries, data exploration
  • HTAP mixed workloads: Analytical replicas

Workload characteristics:

  • Complex queries (seconds to minutes)
  • Low concurrent connections (tens to hundreds)
  • Read-intensive, writes typically batch operations
  • Throughput-sensitive, tolerates higher latency
  • Scans large data volumes

Usage

Specify pg_conf = olap.yml in cluster definition:

pg-olap:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-olap
    pg_conf: olap.yml    # PostgreSQL analytics template
    node_tune: olap      # OS analytics tuning

Use olap.yml template for dedicated offline replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline, pg_conf: olap.yml }  # offline analytics replica
  vars:
    pg_cluster: pg-mixed
    pg_conf: oltp.yml    # primary and online replicas use OLTP
    node_tune: oltp      # OS OLTP tuning

Parameter Details

Connection Management 

max_connections: 500
superuser_reserved_connections: 10

OLAP scenarios typically don’t need many connections; 500 is sufficient for most analytical workloads.

Memory Config

OLAP template uses more aggressive memory allocation:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 50% Faster index creation and VACUUM
work_mem 64MB - 8GB Larger sort/hash memory
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 64MB), 8GB)

Larger work_mem allows bigger sort and hash operations in memory, avoiding disk spill.

Locks & Transactions 

max_locks_per_transaction: 2-4x maxconn   # OLTP: 1-2x

OLAP queries may involve more tables (partitions, many JOINs), requiring more lock slots.

Parallel Query

OLAP template aggressively enables parallel queries:

max_worker_processes: max(cpu + 20, 28)      # OLTP: max(cpu + 16, 24)
max_parallel_workers: 80% × cpu (min 2)      # OLTP: 50%
max_parallel_workers_per_gather: 50% × cpu   # OLTP: 20% (max 8)
max_parallel_maintenance_workers: 33% × cpu

Parallel cost estimates use defaults to favor parallel plans:

# parallel_setup_cost: 1000    # default, not doubled
# parallel_tuple_cost: 0.1     # default, not doubled

Partition-wise optimization enabled:

enable_partitionwise_join: on       # smart partition JOIN
enable_partitionwise_aggregate: on  # smart partition aggregation

IO Config (PG18) 

io_workers: 50% × cpu (4-32)    # OLTP: 25% (4-16)

More IO workers support parallel large table scans.

WAL Config 

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
temp_file_limit: disk/5 (effective max 400GB)   # OLTP: disk/20, effective max 100GB

Larger temp_file_limit allows bigger intermediate results to spill to disk.

Vacuum Config

OLAP template uses aggressive vacuum settings:

vacuum_cost_delay: 10ms         # OLTP: 20ms, faster vacuum
vacuum_cost_limit: 10000        # OLTP: 2000, more work per round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08
autovacuum_analyze_scale_factor: 0.04

Analytical databases often have bulk writes requiring aggressive vacuum to reclaim space.

Query Optimization 

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 1000    # OLTP: 400, more precise stats

Higher default_statistics_target provides more accurate query plans, crucial for complex analytics.

Logging & Monitoring 

log_min_duration_statement: 1000    # OLTP: 100ms, relaxed threshold
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
log_autovacuum_min_duration: 1s
track_io_timing: on
track_cost_delay_timing: on         # PG18+, track vacuum cost delay
track_functions: all
track_activity_query_size: 8192

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 0   # OLTP: 10min, disabled

Analytical queries may need to hold transactions for extended periods, so idle timeout is disabled.

Key Differences from OLTP

Parameter OLAP OLTP Reason
max_connections 500 500-1000 Fewer analytical connections
work_mem limit 8GB 1GB Support larger in-memory sorts
maintenance_work_mem 50% buffer 25% buffer Faster index creation
max_locks_per_transaction 2-4x 1-2x More tables in queries
max_parallel_workers 80% cpu 50% cpu Aggressive parallelism
max_parallel_workers_per_gather 50% cpu 20% cpu Aggressive parallelism
parallel_setup_cost 1000 2000 Default, encourages parallel
parallel_tuple_cost 0.1 0.2 Default, encourages parallel
enable_partitionwise_join on off Partition optimization
enable_partitionwise_aggregate on off Partition optimization
vacuum_cost_delay 10ms 20ms Aggressive vacuum
vacuum_cost_limit 10000 2000 Aggressive vacuum
temp_file_limit 1/5 disk 1/20 disk Allow larger temp files
io_workers 50% cpu 25% cpu More parallel IO
log_min_duration_statement 1000ms 100ms Relaxed slow query threshold
default_statistics_target 1000 400 More precise stats
idle_in_transaction_session_timeout Disabled 10min Allow long transactions

Performance Tuning Tips

With TimescaleDB

OLAP template works great with TimescaleDB:

pg-timeseries:
  vars:
    pg_conf: olap.yml
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions:
      - timescaledb

With pg_duckdb

For ultimate analytical performance, combine with pg_duckdb:

pg-analytics:
  vars:
    pg_conf: olap.yml
    pg_libs: 'pg_duckdb, pg_stat_statements, auto_explain'

Columnar Storage

Consider columnar storage extensions:

pg_extensions:
  - citus_columnar  # or pg_mooncake

Resource Isolation

For mixed workloads, isolate analytics to dedicated replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }               # OLTP writes
    10.10.10.12: { pg_seq: 2, pg_role: replica }               # OLTP reads
    10.10.10.13: { pg_seq: 3, pg_role: offline }               # OLAP analytics
  vars:
    pg_cluster: pg-mixed

Monitoring Metrics

Focus on these metrics:

  • Query time: Long query execution time distribution
  • Parallelism: Parallel worker utilization
  • Temp files: Temp file size and count
  • Disk IO: Sequential and index scan IO volume
  • Cache hit ratio: shared_buffers and OS cache hit rates

References

15.4 - CRIT Template

PostgreSQL parameter template for consistency-first workloads, with strict synchronous replication, data checksums, and detailed connection logging.

crit.yml targets transactional workloads with elevated consistency and audit requirements. It forces data checksums and Patroni strict synchronous mode, adds connection logging, and adjusts selected WAL, timeout, and parallel-query parameters.

The template increases write latency and may block writes when no synchronous replica is available. Before use, confirm consistency objectives, failure domains, client commit settings, and availability requirements.

Also evaluate node_tune: crit, although host tuning and database parameters can be selected independently.

Usage 

pg-critical:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-critical
    pg_conf: crit.yml
    node_tune: crit

A three-node topology leaves room to select another synchronous replica after one node fails. Continued write availability still depends on remaining node state, DCS, network, and synchronous-replica selection. Exercise failures on the target topology.

Strict Synchronous Replication

CRIT does not derive synchronous mode from pg_rpo. It enables these settings unconditionally:

synchronous_mode: true
synchronous_mode_strict: true

synchronous_mode_strict prevents Patroni from falling back to asynchronous replication when no synchronous replica is available. The primary therefore blocks writes that require synchronous acknowledgment.

The mode targets preservation of acknowledged transactions when:

  • the session has not lowered synchronous_commit to local, off, or another asynchronous level;
  • a synchronous replica acknowledges WAL during commit;
  • failover selects only an eligible node containing the required WAL.

RPO must therefore be validated against client parameters, replication state, and the failure model; it cannot be inferred from the template name alone.

To require acknowledgment from multiple synchronous replicas, change Patroni dynamic configuration:

pg edit-config pg-critical
synchronous_node_count: 2

A higher synchronous-replica count imposes stricter conditions for accepting writes.

Data Checksums

CRIT initialization always includes:

initdb:
  - data-checksums

This overrides a disabled pg_checksum setting and enables page checksums for a new cluster. Checksums detect page damage after write; they do not detect logical errors or every memory error.

Connection and Query Logging

CRIT logs DDL, statements taking longer than 100 ms, and disconnection events:

log_statement: ddl
log_min_duration_statement: 100
log_disconnections: 'on'

PostgreSQL 18 and later use:

log_connections: 'receipt,authentication,authorization'

Earlier versions use log_connections: on. These records support connection auditing but are not fine-grained SQL audit logs. Enable pgaudit separately to record object reads and writes, roles, or statement classes.

track_activity_query_size is set to 32 KiB to retain longer active-query text. Logs may contain SQL and business data; restrict access and set an appropriate retention period.

Watchdog

CRIT changes Patroni watchdog from disabled to automatic:

watchdog:
  mode: automatic
  device: /dev/watchdog

automatic activates only when the system has a usable watchdog device. If fencing must be mandatory, verify hardware, virtualization support, and device permissions before setting required explicitly. A bad configuration can prevent primary startup or disrupt failover.

Key Parameter Differences

Parameter CRIT OLTP Default Effect
synchronous_mode Always enabled Derived from pg_rpo Consistency first
synchronous_mode_strict true General template behavior Blocks writes without a synchronous replica
data-checksums Always enabled Controlled by pg_checksum Page-damage detection
max_parallel_workers_per_gather 0 Calculated from CPU Reduces parallel-query variability
wal_writer_delay 10ms 20ms Processes WAL more frequently
wal_writer_flush_after 0 1MB Changes WAL flush behavior
idle_replication_slot_timeout 3d 7d Removes idle replication slots sooner
idle_in_transaction_session_timeout 1min 10min Terminates idle transactions sooner
track_activity_query_size 32KiB 8KiB Retains longer query text
log_connections Detailed connection events PostgreSQL 18 logs authorization by default Adds connection-audit detail
log_disconnections on off Records disconnections

CRIT also disables parallel gather for individual queries and adjusts parallel costs, autovacuum, WAL, and statistics parameters. The active values for a release are defined in roles/pgsql/templates/crit.yml.

Preloaded Extensions

CRIT generates shared_preload_libraries from pg_libs. The role default sets:

pg_libs: 'pg_stat_statements, auto_explain'

Selecting crit.yml alone does not load passwordcheck. Configure it explicitly when password-complexity checks are required:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain'

ha/safe includes this override. To use pgaudit, also add it to pg_libs and configure the audit scope:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain, pgaudit'
pg_parameters:
  pgaudit.log: 'ddl, role, write'

Performance and Availability Impact

  • Synchronous commit waits for a synchronous replica; write latency includes at least replica network and WAL durability time.
  • Strict synchronous mode blocks writes when no synchronous replica is available.
  • Disabling parallel gather can reduce throughput for large queries, but also reduces resource variability from parallel execution.
  • More detailed logging and statistics consume additional I/O, CPU, and storage.
  • A shorter idle-transaction timeout may terminate application sessions that hold a transaction open without executing statements.

The impact depends on hardware, network, queries, and client behavior. Test with the actual workload instead of relying on a fixed latency or throughput percentage.

Launch Checklist

  • Deploy at least one usable synchronous replica and verify write behavior during node failure
  • Check whether applications change synchronous_commit
  • Select watchdog automatic or required according to availability requirements
  • Verify collection, access control, and retention for connection logs
  • Configure pg_libs and extension parameters explicitly when password checks or SQL auditing are required
  • Test write latency, throughput, and idle-transaction timeouts with the production workload
  • Exercise primary, synchronous-replica, DCS, and network-partition failures

Related Documentation

15.5 - TINY Template

PostgreSQL config template optimized for micro instances and resource-constrained environments

tiny.yml is optimized for micro instances and resource-constrained environments. Designed for 1-3 core CPUs with minimal resource usage, conservative memory allocation, and disabled parallel queries.

Pair with node_tune = tiny for OS-level tuning.

Use Cases

TINY template is ideal for:

  • Dev/test: Local development, CI/CD testing
  • Low-spec VMs: 1-2 core CPU, 1-4GB RAM cloud instances
  • Edge computing: Raspberry Pi, embedded devices
  • Demos: Quick Pigsty experience
  • Personal projects: Resource-limited blogs, small apps

Resource constraints:

  • 1-3 CPU cores
  • 1-8 GB RAM
  • Limited disk space
  • May share resources with other services

Usage

Specify pg_conf = tiny.yml in cluster definition:

pg-dev:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Single-node development:

pg-local:
  hosts:
    127.0.0.1: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-local
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Parameter Details

Connection Management 

max_connections: 250   # OLTP: 500-1000, reduced connection overhead
superuser_reserved_connections: 10

Micro instances don’t need many concurrent connections; 250 is sufficient for dev/test.

Memory Config

TINY template uses conservative memory allocation:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 25% For VACUUM, CREATE INDEX
work_mem 16MB - 256MB Smaller sort/hash memory
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 16MB), 256MB)

Smaller work_mem limit (256MB vs OLTP’s 1GB) prevents memory exhaustion.

Parallel Query (Fully Disabled)

TINY template completely disables parallel queries:

max_worker_processes: max(cpu + 12, 20)     # OLTP: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 1)      # OLTP: 50% (min 2)
max_parallel_workers_per_gather: 0           # parallel queries disabled
max_parallel_maintenance_workers: 33% × cpu (min 1)

max_parallel_workers_per_gather: 0 ensures queries won’t spawn parallel workers, avoiding resource contention on low-core systems.

IO Config (PG18) 

io_workers: 3   # fixed value, OLTP: 25% cpu (4-16)

Fixed low IO worker count suitable for resource-constrained environments.

Vacuum Config 

vacuum_cost_delay: 20ms
vacuum_cost_limit: 2000
autovacuum_max_workers: 2          # OLTP: 3, one fewer worker
autovacuum_naptime: 1min
# autovacuum_vacuum_scale_factor uses default
# autovacuum_analyze_scale_factor uses default

Fewer autovacuum workers reduce background resource usage.

Query Optimization 

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 200     # OLTP: 400, lower precision saves space

Lower default_statistics_target reduces pg_statistic table size.

Logging Config 

log_min_duration_statement: 100    # same as OLTP
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
# log_connections uses default (no extra logging)

TINY template doesn’t enable extra connection logging to reduce log volume.

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min   # same as OLTP

Extension Config 

shared_preload_libraries: 'pg_stat_statements, auto_explain'

pg_stat_statements.max: 2500      # OLTP: 10000, reduced memory usage
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

pg_stat_statements.max reduced from 10000 to 2500, saving ~75% memory.

Key Differences from OLTP

Parameter TINY OLTP Reason
max_connections 250 500-1000 Reduce connection overhead
work_mem limit 256MB 1GB Prevent memory exhaustion
max_worker_processes max(cpu+12, 20) max(cpu+16, 24) Fewer background processes
max_parallel_workers_per_gather 0 20% cpu Disable parallel queries
autovacuum_max_workers 2 3 Reduce background load
default_statistics_target 200 400 Save space
pg_stat_statements.max 2500 10000 Reduce memory usage
io_workers 3 25% cpu Fixed low value

Resource Estimates

TINY template resource usage by configuration:

1 Core 1GB RAM 

shared_buffers: ~256MB
work_mem: ~16MB
maintenance_work_mem: ~64MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~400-600MB

2 Core 4GB RAM 

shared_buffers: ~1GB
work_mem: ~32MB
maintenance_work_mem: ~256MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~1.5-2GB

4 Core 8GB RAM

Consider using OLTP template instead:

pg-small:
  vars:
    pg_conf: oltp.yml   # 4C8G can use OLTP template

Performance Tuning Tips

Further Resource Reduction

For extremely constrained resources:

pg_parameters:
  max_connections: 100           # further reduce
  shared_buffers: 128MB          # further reduce
  maintenance_work_mem: 32MB
  work_mem: 8MB

Disable Unnecessary Extensions 

pg_libs: 'pg_stat_statements'    # keep only essential extensions

Disable Unnecessary Features 

pg_parameters:
  track_io_timing: off           # disable IO timing tracking
  track_functions: none          # disable function tracking

Use External Connection Pool

Even on micro instances, PgBouncer significantly improves concurrency:

pg-tiny:
  vars:
    pg_conf: tiny.yml
    pg_default_service_dest: pgbouncer
    pgbouncer_poolmode: transaction

Cloud Platform Recommendations

AWS

  • t3.micro: 1 vCPU, 1GB RAM - suitable for TINY
  • t3.small: 2 vCPU, 2GB RAM - suitable for TINY
  • t3.medium: 2 vCPU, 4GB RAM - consider OLTP

Alibaba Cloud

  • ecs.t6-c1m1.small: 1 vCPU, 1GB RAM - suitable for TINY
  • ecs.t6-c1m2.small: 1 vCPU, 2GB RAM - suitable for TINY
  • ecs.t6-c1m4.small: 1 vCPU, 4GB RAM - suitable for TINY

Tencent Cloud

  • SA2.SMALL1: 1 vCPU, 1GB RAM - suitable for TINY
  • SA2.SMALL2: 1 vCPU, 2GB RAM - suitable for TINY
  • SA2.SMALL4: 1 vCPU, 4GB RAM - suitable for TINY

Edge Device Deployment

Raspberry Pi 4 

pg-pi:
  hosts:
    192.168.1.100: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-pi
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning
    pg_storage_type: SSD    # SSD storage recommended

Docker Container 

pg-docker:
  hosts:
    172.17.0.2: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-docker
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning

Upgrading to OLTP

When your application grows and needs more resources, easily upgrade to OLTP template:

  1. Upgrade VM specs (4 core 8GB+)
  2. Modify cluster config:
pg-growing:
  vars:
    pg_conf: oltp.yml    # change from tiny.yml to oltp.yml
    node_tune: oltp      # change from tiny to oltp
  1. Reconfigure cluster or redeploy

References

16 - FAQ

Frequently asked questions about PostgreSQL

Why can’t my current user use the pg admin alias?

Starting from Pigsty v4.0, permissions to manage global Patroni / PostgreSQL clusters using the pg admin alias have been tightened to the admin group (admin) on admin nodes.

The admin user (dba) created by the node.yml playbook has this permission by default. If your current user wants this permission, you need to explicitly add them to the admin group:

sudo usermod -aG admin <username>

PGSQL Init Fails: Fail to wait for postgres/patroni primary

There are multiple possible causes for this error. You need to check Ansible, Systemd / Patroni / PostgreSQL logs to find the real cause.

  • Possibility 1: Cluster config error - find and fix the incorrect config items.
  • Possibility 2: A cluster with the same name exists, or the previous same-named cluster primary was improperly removed.
  • Possibility 3: Residual metadata from a same-named cluster remains in DCS. First inspect the exact keyspace with etcdctl get --prefix /pg/<cls>/; only after confirming the backup and full cluster name should you use etcdctl del --prefix /pg/<cls>/. The trailing / is the namespace boundary and must not be omitted, or a cluster whose name merely starts with the same text can also match. This is destructive; prefer the controlled decommissioning workflow.
  • Possibility 4: Your PostgreSQL or node-related RPM pkgs were not successfully installed.
  • Possibility 5: Your Watchdog kernel module was not properly enabled/loaded.
  • Possibility 6: The locale you specified during database init doesn’t exist (e.g., used en_US.UTF8 but English language pack or Locale support wasn’t installed).
  • If you encounter other causes, please submit an Issue or ask the community for help.

PGSQL Init Fails: Fail to wait for postgres/patroni replica

There are several possible causes:

Immediate failure: Usually due to config errors, network issues, corrupted DCS metadata, etc. You must check /pg/log to find the actual cause.

Failure after a while: This might be due to source instance data corruption. See PGSQL FAQ: How to create a replica when data is corrupted?

Timeout after a long time: If the wait for postgres replica task takes 30 minutes or longer and fails due to timeout, this is common for large clusters (e.g., 1TB+, may take hours to create a replica).

In this case, the underlying replica creation process is still ongoing. You can use pg list <cls> to check cluster status and wait for the replica to catch up with the primary. Then use the following command to continue with remaining tasks and complete the full replica init:

./pgsql.yml -t pg_hba,pg_reload,pg_backup,pgbouncer,pg_vip,pg_dns,pg_service,pg_exporter,pg_register -l <problematic_replica>

PGSQL Init Fails: ABORT due to pg_safeguard enabled

This means the PostgreSQL instance being cleaned has the deletion safeguard enabled. Disable pg_safeguard to remove the Postgres instance.

If the deletion safeguard pg_safeguard is enabled, you cannot remove running PGSQL instances using bin/pgsql-rm or the pgsql-rm.yml playbook.

To disable pg_safeguard, you can set pg_safeguard to false in the config inventory, or use the command param -e pg_safeguard=false when executing the playbook.

./pgsql-rm.yml -e pg_safeguard=false -l <cls_to_remove>

How to Ensure No Data Loss During Failover?

Use the crit.yml param template, set pg_rpo to 0, or config the cluster for sync commit mode.

Consider using Sync Standby and Quorum Commit to ensure zero data loss during failover.

For more details, see the intro in Security Considerations - Availability.

How to Rescue When Disk is Full?

If the disk is full and even Shell commands cannot execute, rm -rf /pg/dummy can release some emergency space.

By default, pg_dummy_filesize is set to 64MB. In prod envs, it’s recommended to increase it to 8GB or larger.

It will be placed at /pg/dummy path on the PGSQL main data disk. You can delete this file to free up some emergency space:

At least it will allow you to run some shell scripts on that node to further reclaim other space (e.g., logs/WAL, stale data, WAL archives and backups).

How to Create a Replica When Cluster Data is Corrupted?

Pigsty sets the clonefrom: true tag in the patroni config of all instances, marking the instance as available for creating replicas.

If an instance has corrupted data files causing errors when creating new replicas, you can set clonefrom: false to avoid pulling data from the corrupted instance. Here’s how:

$ vi /etc/patroni/patroni.yml

tags:
  nofailover: false
  clonefrom: true      # ----------> change to false
  noloadbalance: false
  nosync: false
  version:  '18'
  spec: '4C.8G.50G'
  conf: 'oltp.yml'

$ systemctl reload patroni    # Reload Patroni config

What is the Perf Overhead of PostgreSQL Monitoring?

A regular PostgreSQL instance scrape takes about 200ms. The scrape interval defaults to 10 seconds, which is almost negligible for a prod multi-core database instance.

Note that Pigsty enables in-database object monitoring by default, so if your database has hundreds of thousands of table/index objects, scraping may increase to several seconds.

You can modify Prometheus’s scrape frequency. Please ensure: the scrape cycle should be significantly longer than the duration of a single scrape.

How to Monitor an Existing PostgreSQL Instance?

Detailed monitoring config instructions are provided in PGSQL Monitor.

How to Manually Remove PostgreSQL Monitoring Targets? 

./pgsql-rm.yml -t rm_metrics -l <cls>     # Remove all instances of cluster 'cls' from victoria
bin/pgmon-rm <ins>     # Remove a single instance 'ins' monitoring object from Victoria, especially suitable for removing added external instances

17 - Misc

Miscellaneous Topics

17.1 - Service / Access

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Service is an abstraction: it is the form in which database clusters provide capabilities to the outside world and encapsulates the details of the underlying cluster.

Services are critical for stable access in production environments and show their value when high availability clusters automatically fail over. Single-node users typically don’t need to worry about this concept.

Single-Node Users

The concept of “service” is for production environments. Personal users/single-node clusters can simply access the database directly using instance name/IP address.

For example, Pigsty’s default single-node pg-meta.meta database can be connected directly using three different users:

psql postgres://dbuser_dba:DBUser.DBA@10.10.10.10/meta     # Connect directly with DBA superuser
psql postgres://dbuser_meta:DBUser.Meta@10.10.10.10/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

In real-world production environments, we use replication-based primary-replica database clusters. In a cluster, there is one and only one instance as the leader (primary) that can accept writes. Other instances (replicas) continuously fetch change logs from the cluster leader and stay consistent with it. At the same time, replicas can also handle read-only requests, significantly reducing the load on the primary in read-heavy scenarios. Therefore, separating write requests and read-only requests to the cluster is a very common practice.

In addition, for production environments with high-frequency short connections, we also pool requests through a connection pool middleware (Pgbouncer) to reduce the overhead of creating connections and backend processes. But for scenarios such as ETL and change execution, we need to bypass the connection pool and access the database directly. At the same time, high-availability clusters will experience failover when failures occur, and failover will cause changes to the cluster’s leader. Therefore, high-availability database solutions require that write traffic can automatically adapt to changes in the cluster’s leader. These different access requirements (read-write separation, pooling and direct connection, automatic failover adaptation) ultimately abstract the concept of Service.

Typically, database clusters must provide this most basic service:

  • Read-Write Service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-Write Service (primary): Write data: can only be carried by the primary.
  • Read-Only Service (replica): Read data: can be carried by replicas, or by the primary if there are no replicas

In addition, depending on specific business scenarios, there may be other services, such as:

  • Default Direct Service (default): Allows (admin) users to access the database directly, bypassing the connection pool
  • Offline Replica Service (offline): Dedicated replicas that do not handle online read-only traffic, used for ETL and analytical queries
  • Standby Replica Service (standby): Read-only service without replication lag, handled by sync standby/primary for read-only queries
  • Delayed Replica Service (delayed): Access old data from the same cluster at a previous point in time, handled by delayed replica

Default Services

Pigsty provides four different services by default for each PostgreSQL database cluster. Here are the default services and their definitions:

Service Port Description
primary 5433 Production read-write, connects to primary connection pool (6432)
replica 5434 Production read-only, connects to replica connection pool (6432)
default 5436 Admin, ETL writes, direct access to primary (5432)
offline 5438 OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

You can see how these four services work from the sample cluster architecture diagram:

pigsty-ha.png

The actual DNS target of pg-meta is controlled by pg_dns_target. The default auto points to the L2 VIP when VIP is enabled; otherwise it points to the inventory primary’s IP. VIP is not enabled by default. See Accessing Services.

Service Implementation

In Pigsty, services are implemented using haproxy on nodes, differentiated by different ports on host nodes.

Haproxy is enabled by default on each node managed by Pigsty to expose services, and database nodes are no exception. Although nodes in a cluster have primary-replica distinctions from the database perspective, from the service perspective, each node is the same: This means that even if you access a replica node, as long as you use the correct service port, you can still use the primary’s read-write service. This design can hide complexity: so as long as you can access any instance on a PostgreSQL cluster, you can completely access all services.

This design is similar to NodePort services in Kubernetes. Similarly, in Pigsty, each service includes the following two core elements:

  1. Access endpoints exposed through NodePort (port number, where to access?)
  2. Target instances selected through Selectors (instance list, who carries the load?)

Pigsty’s service delivery boundary stops at the cluster’s HAProxy, and users can access these load balancers in various ways. See Accessing Services.

All services are declared through configuration files. For example, the PostgreSQL default services are defined by the pg_default_services parameter:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

You can also define additional services in pg_services. Both pg_default_services and pg_services are arrays of service definition objects.

Defining Services

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for accessing other components

For PostgreSQL clusters, you typically only need to focus on the first two. Each service definition generates a new configuration file in the configuration directory of all related HAProxy instances: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg Here’s a custom service example standby: when you want to provide a read-only service without replication lag, you can add this record to pg_services:

- name: standby                   # Required, service name, final svc name uses `pg_cluster` as prefix, e.g.: pg-meta-standby
  port: 5435                      # Required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # Optional, IP address the service binds to, all IP addresses by default
  selector: "[]"                  # Required, service member selector, uses JMESPath to filter configuration manifest
  backup: "[? pg_role == `primary`]"  # Optional, service member selector (backup), instances selected here only carry the service when all default selector instances are down
  dest: default                   # Optional, target port, default|postgres|pgbouncer|<port_number>, defaults to 'default', meaning pg_default_service_dest decides
  check: /sync                    # Optional, health check URL path, defaults to /, here uses Patroni API: /sync, only sync standby and primary return 200 healthy status code
  maxconn: 5000                   # Optional, maximum number of allowed frontend connections, defaults to 5000
  balance: roundrobin             # Optional, haproxy load balancing algorithm (defaults to roundrobin, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

The above service definition is rendered as the HAProxy configuration file /etc/haproxy/conf.d/pg-test-standby.cfg on the sample three-node pg-test cluster:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Maximum connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balancing algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connection
    http-check send meth OPTIONS uri /sync   # <---- Here uses /sync, Patroni health check API, only sync standby and primary return 200 healthy status code
    http-check expect status 200             # <---- Health check return code 200 means normal
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore serves as service fallback instance: normally doesn't handle requests, only handles read-only requests when all other replicas fail, thus maximally avoiding read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

Here, all three instances of the pg-test cluster are selected by selector: "[]", rendered into the backend server list of the pg-test-standby service. But due to the /sync health check, Patroni Rest API only returns healthy HTTP 200 status code on the primary and sync standby, so only the primary and sync standby can actually handle requests. Additionally, the primary satisfies the condition pg_role == primary, is selected by the backup selector, and is marked as a backup server, only used when no other instances (i.e., sync standby) can meet the demand.

Primary Service

The Primary service is perhaps the most critical service in production environments. It provides read-write capability to the database cluster on port 5433. The service definition is as follows:

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Primary service
  • But only the primary can pass the health check (check: /primary) and actually carry Primary service traffic.
  • The destination parameter dest: default means the Primary service destination is affected by the pg_default_service_dest parameter
  • The default value default of dest will be replaced by the value of pg_default_service_dest, which defaults to pgbouncer.
  • By default, the Primary service destination is the connection pool on the primary, which is the port specified by pgbouncer_port, defaulting to 6432

If the value of pg_default_service_dest is postgres, then the primary service destination will bypass the connection pool and use the PostgreSQL database port directly (pg_port, default 5432). This parameter is very useful for scenarios that don’t want to use a connection pool.

Example: haproxy configuration for pg-test-primary
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni’s high availability mechanism ensures that at most one instance’s /primary health check is true at any time, so the Primary service will always route traffic to the primary instance.

One benefit of using the Primary service instead of direct database connection is that if the cluster has a split-brain situation for some reason (e.g., kill -9 killing the primary Patroni without watchdog), Haproxy can still avoid split-brain in this case, because it will only distribute traffic when Patroni is alive and returns primary status.

Replica Service

The Replica service is second only to the Primary service in importance in production environments. It provides read-only capability to the database cluster on port 5434. The service definition is as follows:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Replica service
  • All instances can pass the health check (check: /read-only) and carry Replica service traffic.
  • Backup selector: [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backup servers.
  • Only when all normal replicas are down will the Replica service be carried by the primary or offline replicas.
  • The destination parameter dest: default means the Replica service destination is also affected by the pg_default_service_dest parameter
  • The default value default of dest will be replaced by the value of pg_default_service_dest, which defaults to pgbouncer, same as the Primary service
  • By default, the Replica service destination is the connection pool on the replicas, which is the port specified by pgbouncer_port, defaulting to 6432
Example: haproxy configuration for pg-test-replica
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service is very flexible: if there are surviving dedicated Replica instances, it will prioritize using these instances to handle read-only requests. Only when all replica instances are down will the primary handle read-only requests. For the common one-primary-one-replica two-node cluster, this means: use the replica as long as it’s alive, use the primary when the replica is down.

Additionally, unless all dedicated read-only instances are down, the Replica service will not use dedicated Offline instances, thus avoiding mixing online fast queries and offline slow queries together, interfering with each other.

Default Service

The Default service provides services on port 5436. It is a variant of the Primary service.

The Default service always bypasses the connection pool and connects directly to PostgreSQL on the primary. This is useful for admin connections, ETL writes, CDC data change capture, etc.

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is changed to postgres, then the Default service is completely equivalent to the Primary service except for port and name. In this case, you can consider removing Default from default services.

Example: haproxy configuration for pg-test-default
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are exactly the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service provides services on port 5438. It bypasses the connection pool to directly access the PostgreSQL database, typically used for slow queries, analytical queries, ETL reads, and personal interactive queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

The Offline service routes traffic directly to dedicated offline replicas, or normal read-only instances with the pg_offline_query flag.

  • The selector parameter filters two types of instances from the cluster: offline replicas with pg_role = offline, or normal read-only instances with pg_offline_query = true
  • The main difference between dedicated offline replicas and flagged normal replicas is: the former does not handle Replica service requests by default, avoiding mixing fast and slow requests together, while the latter does by default.
  • The backup selector parameter filters one type of instance from the cluster: normal replicas without offline flag. This means if offline instances or flagged normal replicas fail, other normal replicas can be used to carry the Offline service.
  • The health check /replica only returns 200 for replicas, the primary returns an error, so the Offline service will never distribute traffic to the primary instance, even if only this primary is left in the cluster.
  • At the same time, the primary instance is neither selected by the selector nor by the backup selector, so it will never carry the Offline service. Therefore, the Offline service can always avoid user access to the primary, thus avoiding impact on the primary.
Example: haproxy configuration for pg-test-offline
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

The Offline service provides limited read-only service, typically used for two types of queries: interactive queries (personal users), slow queries and long transactions (analytics/ETL).

The Offline service requires extra care. HAProxy’s /replica health check automatically rejects the new primary after a switchover, but selector uses static pg_role / pg_offline_query labels from the inventory. In a one-primary-one-replica cluster where only the replica serves Offline queries, a switchover may temporarily leave no eligible backend. Reloading an unchanged inventory does not add the old primary to the Offline backend list. First update the inventory labels (or pg_offline_query) to match the new plan and then reload services, or switch the primary back.

If your business model is relatively simple, you can consider removing the Default service and Offline service, and use the Primary service and Replica service to connect directly to the database.

Reload Services

Reload services when cluster membership changes, service definitions or static selector labels change, or relative weights are adjusted. Normal Primary/Replica switchover is handled by Patroni health checks and does not require a separate reload.

bin/pgsql-svc <cls> [ip...]         # Reload services for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # Actual ansible task for reloading services

Accessing Services

Pigsty’s service delivery boundary stops at the cluster’s HAProxy. Users can access these load balancers in various ways.

The typical approach is to use DNS or VIP access, binding them to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

Type Example Description
Cluster Domain pg-test Access via cluster domain name (resolved by dnsmasq @ infra node)
Cluster VIP Address 10.10.10.3 Access via L2 VIP address managed by vip-manager, bound to primary node
Instance Hostname pg-test-1 Access via any instance hostname (resolved by dnsmasq @ infra node)
Instance IP Address 10.10.10.11 Access any instance’s IP address

Port

Pigsty uses different ports to distinguish pg services

Port Service Type Description
5432 postgres Database Direct access to postgres server
6432 pgbouncer Middleware Access postgres via connection pool middleware
5433 primary Service Access primary pgbouncer (or postgres)
5434 replica Service Access replica pgbouncer (or postgres)
5436 default Service Access primary postgres
5438 offline Service Access offline postgres

Combinations

# Access via cluster domain name (the examples below assume L2 VIP is enabled; without it, auto targets the inventory primary IP)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> Primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> Primary connection pool -> Primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://test@10.10.10.3:5432/test # L2 VIP -> Primary direct access
postgres://test@10.10.10.3:6432/test # L2 VIP -> Primary connection pool -> Primary
postgres://test@10.10.10.3:5433/test # L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://test@10.10.10.3:5434/test # L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://dbuser_dba@10.10.10.3:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://dbuser_stats@10.10.10.3:5438/test # L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database instance direct connection (single instance access)
postgres://test@pg-test-1:6432/test # DNS -> Connection pool -> Database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> Connection pool -> Database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> Connection pool -> Database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> Database direct connection
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> Database offline read/write

# Specify any cluster instance IP directly
postgres://test@10.10.10.11:5432/test # Database instance direct connection (direct instance specification, no automatic traffic distribution)
postgres://test@10.10.10.11:6432/test # Connection pool -> Database
postgres://test@10.10.10.11:5433/test # HAProxy -> Connection pool -> Database read/write
postgres://test@10.10.10.11:5434/test # HAProxy -> Connection pool -> Database read-only
postgres://dbuser_dba@10.10.10.11:5436/test # HAProxy -> Database direct connection
postgres://dbuser_stats@10.10.10.11:5438/test # HAProxy -> Database offline read/write

# Smart client: automatic read-write separation
postgres://test@10.10.10.11:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://test@10.10.10.11:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Overriding Services

You can override default service configuration in multiple ways. A common requirement is to have Primary service and Replica service bypass the Pgbouncer connection pool and access the PostgreSQL database directly.

To achieve this, you can change pg_default_service_dest to postgres, so all services with svc.dest='default' in their service definitions will use postgres instead of the default pgbouncer as the target.

If you have already pointed Primary service to PostgreSQL, then default service becomes redundant and can be considered for removal.

If you don’t need to distinguish between personal interactive queries and analytical/ETL slow queries, you can consider removing Offline service from the default service list pg_default_services.

If you don’t need read-only replicas to share online read-only traffic, you can also remove Replica service from the default service list.

Delegating Services

Pigsty exposes PostgreSQL services through haproxy on nodes. All haproxy instances in the entire cluster are configured with the same service definitions.

However, you can delegate pg services to specific node groups (e.g., dedicated haproxy load balancer cluster) instead of haproxy on PostgreSQL cluster members.

To do this, you need to override the default service definitions using pg_default_services and set pg_service_provider to the proxy group name.

For example, this configuration will expose the pg cluster’s primary service on the proxy haproxy node group on port 10013.

pg_service_provider: proxy       # Use load balancer from `proxy` group on port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

Users need to ensure that the port for each delegated service is unique in the proxy cluster.

An example of using a dedicated load balancer cluster is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

17.2 - Access Control

Entry points for PostgreSQL access-control concepts, configuration, and administration.

Pigsty access-control documentation is organized by purpose:

dbrole_offline provides independent read-only object privileges; it does not restrict instance scope automatically. To allow it only on offline instances, set role: offline explicitly on the corresponding HBA rule and verify the generated pg_hba.conf on both online and offline instances.

17.3 - User / Role

Users/roles refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In this context, users refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In PostgreSQL, users belong directly to the database cluster rather than to a specific database. Therefore, when creating business databases and business users, you should follow the principle of “users first, then databases.”

Defining Users

Pigsty defines roles and users in database clusters through two configuration parameters:

  • pg_default_roles: Defines globally unified roles and users
  • pg_users: Defines business users and roles at the database cluster level

The former defines roles and users shared across the entire environment, while the latter defines business roles and users specific to individual clusters. Both have the same format and are arrays of user definition objects.

You can define multiple users/roles, and they will be created sequentially—first global, then cluster-level, and finally in array order—so later users can belong to roles defined earlier.

Here is the business user definition for the default cluster pg-meta in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }

Each user/role definition is an object that may include the following fields. Using dbuser_meta as an example:

- name: dbuser_meta               # Required, `name` is the only mandatory field in user definition
  password: DBUser.Meta           # Optional, password can be scram-sha-256 hash string or plaintext
  login: true                     # Optional, can login by default
  superuser: false                # Optional, default is false, is this a superuser?
  createdb: false                 # Optional, default is false, can create databases?
  createrole: false               # Optional, default is false, can create roles?
  inherit: true                   # Optional, by default this role can use inherited privileges?
  replication: false              # Optional, default is false, can this role perform replication?
  bypassrls: false                # Optional, default is false, can this role bypass row-level security?
  pgbouncer: true                 # Optional, default is false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # Optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # Optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # Optional, when this role expires, use YYYY-MM-DD format string to specify a date (lower priority than expire_in)
  comment: pigsty admin user      # Optional, description and comment string for this user/role
  roles: [dbrole_admin]           # Optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # Optional, use `ALTER ROLE SET` to configure role-level database parameters for this role
  pool_mode: transaction          # Optional, pgbouncer pool mode defaulting to transaction, user level
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100
  search_path: public             # Optional, key-value configuration parameters per postgresql documentation (e.g., use pigsty as default search_path)
  • The only required field is name, which should be a valid and unique username in the PostgreSQL cluster.
  • Roles don’t need a password, but for loginable business users, a password is usually required.
  • password can be plaintext or scram-sha-256 / md5 hash string; please avoid using plaintext passwords.
  • Users/roles are created one by one in array order, so ensure roles/groups are defined before their members.
  • login, superuser, createdb, createrole, inherit, replication, bypassrls are boolean flags.
  • pgbouncer is disabled by default: to add business users to the pgbouncer user list, you should explicitly set it to true.

ACL System

Pigsty provides a built-in access control / ACL model. Assign its default business roles to users as required:

  • dbrole_readwrite: Role with global read-write access (production accounts primarily used by business should have database read-write privileges)
  • dbrole_readonly: Role with global read-only access (if other businesses need read-only access, use this role)
  • dbrole_admin: Role with DDL privileges (business administrators, scenarios requiring table creation in applications)
  • dbrole_offline: Independent read-only role for ad hoc queries, ETL, and analytics; restrict its instance scope explicitly through HBA

If you want to redesign your own ACL system, consider customizing the following parameters and templates:

Creating Users

Users and roles defined in pg_default_roles and pg_users are automatically created one by one during the cluster initialization PROVISION phase. If you want to create users on an existing cluster, you can use the bin/pgsql-user tool. Add the new user/role definition to all.children.<cls>.pg_users and use the following method to create the user:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

Unlike databases, the user creation playbook is always idempotent. When the target user already exists, Pigsty will modify the target user’s attributes to match the configuration. So running it repeatedly on existing clusters is usually not a problem.

Please Use Playbooks to Create Users

We don’t recommend manually creating new business users, especially when you want the user to use the default pgbouncer connection pool: unless you’re willing to manually maintain the user list in Pgbouncer and keep it consistent with PostgreSQL. When creating new users with bin/pgsql-user tool or pgsql-user.yml playbook, the user will also be added to the Pgbouncer Users list.

Modifying Users

The method for modifying PostgreSQL user attributes is the same as Creating Users.

First, adjust your user definition, modify the attributes that need adjustment, then execute the following command to apply:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

Note that modifying users will not delete users, but modify user attributes through the ALTER USER command; it also won’t revoke user privileges and groups, and will use the GRANT command to grant new roles.

Pgbouncer Users

Pgbouncer is enabled by default and serves as a connection pool middleware, with its users managed by default.

Pigsty adds all users in pg_users that explicitly have the pgbouncer: true flag to the pgbouncer user list.

Users in the Pgbouncer connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="
"dbuser_kong" "SCRAM-SHA-256$4096:bK8sLXIieMwFDz67/0dqXQ==$P/tCRgyKx9MC9LH3ErnKsnlOqgNd/nn2RyvThyiK6e4=:CDM8QZNHBdPf97ztusgnE7olaKDNHBN0WeAbP/nzu5A="
"dbuser_grafana" "SCRAM-SHA-256$4096:HjLdGaGmeIAGdWyn2gDt/Q==$jgoyOB8ugoce+Wqjr0EwFf8NaIEMtiTuQTg1iEJs9BM=:ed4HUFqLyB4YpRr+y25FBT7KnlFDnan6JPVT9imxzA4="
"dbuser_gitea" "SCRAM-SHA-256$4096:l1DBGCc4dtircZ8O8Fbzkw==$tpmGwgLuWPDog8IEKdsaDGtiPAxD16z09slvu+rHE74=:pYuFOSDuWSofpD9OZhG7oWvyAR0PQjJBffgHZLpLHds="
"dbuser_dba" "SCRAM-SHA-256$4096:zH8niABU7xmtblVUo2QFew==$Zj7/pq+ICZx7fDcXikiN7GLqkKFA+X5NsvAX6CMshF0=:pqevR2WpizjRecPIQjMZOm+Ap+x0kgPL2Iv5zHZs0+g="
"dbuser_bytebase" "SCRAM-SHA-256$4096:OMoTM9Zf8QcCCMD0svK5gg==$kMchqbf4iLK1U67pVOfGrERa/fY818AwqfBPhsTShNQ=:6HqWteN+AadrUnrgC0byr5A72noqnPugItQjOLFw0Wk="

User-level connection pool parameters are maintained in a separate file: /etc/pgbouncer/useropts.txt, for example:

dbuser_dba                  = pool_mode=session max_user_connections=16
dbuser_monitor              = pool_mode=session max_user_connections=8

When you create a database, the Pgbouncer database list definition file will be refreshed and take effect through online configuration reload, without affecting existing connections.

Pgbouncer runs with the same dbsu as PostgreSQL, which defaults to the postgres operating system user. You can use the pgb alias to access pgbouncer management functions using the dbsu.

The connection pool user configuration files userlist.txt and useropts.txt are automatically refreshed when you create users, and take effect through online configuration reload, normally without affecting existing connections.

Note that the pgbouncer_auth_query parameter allows you to use dynamic queries to complete connection pool user authentication—this is a compromise when you don’t want to manage users in the connection pool.

17.4 - Database

Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

In this context, Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

A PostgreSQL server can serve multiple databases simultaneously. In Pigsty, you can define the required databases in the cluster configuration.

Pigsty will modify and customize the default template database template1, creating default schemas, installing default extensions, and configuring default privileges. Newly created databases will inherit these settings from template1 by default.

By default, all business databases will be added to the Pgbouncer connection pool in a 1:1 manner; pg_exporter will use an auto-discovery mechanism to find all business databases and monitor objects within them.

Define Database

Business databases are defined in the database cluster parameter pg_databases, which is an array of database definition objects. Databases in the array are created sequentially according to the definition order, so later defined databases can use previously defined databases as templates.

Below is the database definition for the default pg-meta cluster in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition is an object that may include the following fields, using the meta database as an example:

- name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
  baseline: cmdb.sql              # optional, database sql baseline path (relative path among ansible search path, e.g. files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
  schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
  extensions:                     # optional, additional extensions to be installed: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install the extension in, or leave it unspecified (will install in the first schema of search_path)
    - { name: timescaledb }               # for example, some extensions create and use fixed schemas, so no schema specification is needed.
  comment: pigsty meta database   # optional, comment string for this database
  owner: postgres                 # optional, database owner, postgres by default
  template: template1             # optional, which template to use, template1 by default, target must be a template database
  encoding: UTF8                  # optional, database encoding, UTF8 by default (MUST same as template database)
  locale: C                       # optional, database locale, C by default (MUST same as template database)
  lc_collate: C                   # optional, database collate, C by default (MUST same as template database), no reason not to recommend changing.
  lc_ctype: C                     # optional, database ctype, C by default (MUST same as template database)
  tablespace: pg_default          # optional, default tablespace, 'pg_default' by default
  allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
  revokeconn: false               # optional, revoke public connection privilege. false by default, when set to true, CONNECT privilege will be revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasources? true by default, explicitly set to false to skip registration
  connlimit: -1                   # optional, database connection limit, default -1 disable limit, set to positive integer will limit connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will be authenticated using this user (only useful when pgbouncer_auth_query is enabled)
  pool_mode: transaction          # optional, pgbouncer pool mode at database level, default transaction
  pool_size: 50                   # optional, pgbouncer pool size at database level, default 50
  pool_reserve: 30                # optional, pgbouncer reserve pool at database level, default 30
  pool_size_min: 0                # optional, pgbouncer pool size min at database level, default 0
  pool_connlimit: 100             # optional, max database connections at database level, default 100

The only required field is name, which should be a valid and unique database name in the current PostgreSQL cluster, other parameters have reasonable defaults.

  • name: Database name, required.
  • baseline: SQL file path (Ansible search path, usually in files), used to initialize database content.
  • owner: Database owner, default is postgres
  • template: Template used when creating the database, default is template1
  • encoding: Database default character encoding, default is UTF8, default is consistent with the instance. It is recommended not to configure and modify.
  • locale: Database default locale, default is C, it is recommended not to configure, keep consistent with the instance.
  • lc_collate: Database default locale string collation, default is same as instance setting, it is recommended not to modify, must be consistent with template database. It is strongly recommended not to configure, or configure to C.
  • lc_ctype: Database default LOCALE, default is same as instance setting, it is recommended not to modify or set, must be consistent with template database. It is recommended to configure to C or en_US.UTF8.
  • allowconn: Whether to allow connection to the database, default is true, not recommended to modify.
  • revokeconn: Whether to revoke connection privilege to the database? Default is false. If true, PUBLIC CONNECT privilege on the database will be revoked. Only default users (dbsu|monitor|admin|replicator|owner) can connect. In addition, admin|owner will have GRANT OPTION, can grant connection privileges to other users.
  • tablespace: Tablespace associated with the database, default is pg_default.
  • connlimit: Database connection limit, default is -1, meaning no limit.
  • extensions: Object array, each object defines an extension in the database, and the schema in which it is installed.
  • parameters: KV object, each KV defines a parameter that needs to be modified for the database through ALTER DATABASE.
  • pgbouncer: Boolean option, whether to add this database to Pgbouncer. All databases will be added to Pgbouncer list unless explicitly specified as pgbouncer: false.
  • comment: Database comment information.
  • pool_auth_user: When pgbouncer_auth_query is enabled, all connections to this pgbouncer database will use the user specified here to execute authentication queries. You need to use a user with access to the pg_shadow table.
  • pool_mode: Database level pgbouncer pool mode, default is transaction, i.e., transaction pooling. If left empty, will use pgbouncer_poolmode parameter as default value.
  • pool_size: Database-level Pgbouncer default pool size, default 50.
  • pool_reserve: Database-level Pgbouncer reserve pool, default 30; when the regular pool is exhausted, at most this many burst connections can be added.
  • pool_size_min: Database level pgbouncer pool size min, default is 0
  • pool_connlimit: Database level pgbouncer connection pool max database connections, default is 100

Newly created databases are forked from the template1 database by default. This template database will be customized during the PG_PROVISION phase: configured with extensions, schemas, and default privileges, so newly created databases will also inherit these configurations unless you explicitly use another database as a template.

For database access privileges, see Access Control: Database Isolation.

Create Database

Databases defined in pg_databases will be automatically created during cluster initialization. If you wish to create database on an existing cluster, you can use the bin/pgsql-db wrapper script. Add new database definition to all.children.<cls>.pg_databases, and create that database with the following command:

bin/pgsql-db <cls> <dbname>    # pgsql-db.yml -l <cls> -e dbname=<dbname>

Here are some considerations when creating a new database:

The create database playbook is idempotent by default, however when you use baseline scripts, it may not be: in this case, it’s usually not recommended to re-run this on existing databases unless you’re sure the provided baseline SQL is also idempotent.

We don’t recommend manually creating new databases, especially when you’re using the default pgbouncer connection pool: unless you’re willing to manually maintain the Pgbouncer database list and keep it consistent with PostgreSQL. When creating new databases using the pgsql-db tool or pgsql-db.yml playbook, this database will also be added to the Pgbouncer Database list.

If your database definition has a non-trivial owner (default is dbsu postgres), make sure the owner user exists before creating the database. Best practice is always to create users before creating databases.

Pgbouncer Database

Pigsty will configure and enable a Pgbouncer connection pool for PostgreSQL instances in a 1:1 manner by default, communicating via /var/run/postgresql Unix Socket.

Connection pools can optimize short connection performance, reduce concurrency contention, avoid overwhelming the database with too many connections, and provide additional flexibility during database migration.

Pigsty adds all databases in pg_databases to pgbouncer’s database list by default. You can disable pgbouncer connection pool support for a specific database by explicitly setting pgbouncer: false in the database definition.

The Pgbouncer database list is defined in /etc/pgbouncer/database.txt, and connection pool parameters from the database definition are reflected here:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When you create databases, the Pgbouncer database list definition file will be refreshed and take effect through online configuration reload, normally without affecting existing connections.

Pgbouncer runs with the same dbsu as PostgreSQL, defaulting to the postgres os user. You can use the pgb alias to access pgbouncer management functions using dbsu.

To route a managed Pgbouncer database to another node, edit /etc/pgbouncer/database.txt, then reload the configuration and rebuild existing server connections:

# Change only mydb's backend target
sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt
pgb -c "RELOAD;"
pgb -c "RECONNECT mydb;"
pgb -c "WAIT_CLOSE mydb;"

The current pgb-route helper edits only /etc/pgbouncer/pgbouncer.ini, while Pigsty-managed database routes live in the included database.txt; it therefore does not change those managed routes.

17.5 - Authentication / HBA

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Authentication is the foundation of Access Control and Default Privileges. PostgreSQL supports several authentication methods.

Here we mainly introduce HBA: Host Based Authentication. HBA rules define which users can access which databases from which locations and in which ways.

Client Authentication

To connect to a PostgreSQL database, users must first be authenticated (password is used by default).

You can provide the password in the connection string (not secure), or pass it using the PGPASSWORD environment variable or .pgpass file. Refer to the psql documentation and PostgreSQL Connection Strings for more details.

psql 'host=<host> port=<port> dbname=<dbname> user=<username> password=<password>'
psql postgres://<username>:<password>@<host>:<port>/<dbname>
PGPASSWORD=<password>; psql -U <username> -h <host> -p <port> -d <dbname>

For example, to connect to Pigsty’s default meta database, you can use the following connection strings:

psql 'host=10.10.10.10 port=5432 dbname=meta user=dbuser_dba password=DBUser.DBA'
psql postgres://dbuser_dba:DBUser.DBA@10.10.10.10:5432/meta
PGPASSWORD=DBUser.DBA; psql -U dbuser_dba -h 10.10.10.10 -p 5432 -d meta

By default, Pigsty enables server-side SSL encryption but does not verify client SSL certificates. To connect using client SSL certificates, you can provide client parameters using the PGSSLCERT and PGSSLKEY environment variables or sslkey and sslcert parameters.

psql 'postgres://dbuser_dba:DBUser.DBA@10.10.10.10:5432/meta?sslkey=/path/to/dbuser_dba.key&sslcert=/path/to/dbuser_dba.crt'

Client certificates (CN = username) can be signed using the local CA with the cert.yml playbook.

Defining HBA

In Pigsty, there are four parameters related to HBA rules:

These are all arrays of HBA rule objects. Each HBA rule is an object in one of the following two forms:

1. Raw Form

The raw form of HBA is almost identical to the PostgreSQL pg_hba.conf format:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5

In this form, the rules field is an array of strings, where each line is a raw HBA rule. The title field is rendered as a comment explaining what the rules below do.

The role field specifies which instance roles the rule applies to. When an instance’s pg_role matches the role, the HBA rule will be added to that instance’s HBA.

  • HBA rules with role: common will be added to all instances.
  • HBA rules with role: primary will only be added to primary instances.
  • HBA rules with role: replica will only be added to replica instances.
  • HBA rules with role: offline will be added to offline instances (pg_role = offline or pg_offline_query = true)

2. Alias Form

The alias form allows you to maintain HBA rules in a simpler, clearer, and more convenient way: it replaces the rules field with addr, auth, user, and db fields. The title and role fields still apply.

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access
  • addr: where - Which IP address ranges are affected by this rule?
    • world: All IP addresses
    • intra: All intranet IP address ranges: '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'
    • infra: IP addresses of Infra nodes
    • admin: IP addresses of admin_ip management nodes
    • local: Local Unix Socket
    • localhost: Local Unix Socket and TCP 127.0.0.1/32 loopback address
    • cluster: IP addresses of all members in the same PostgreSQL cluster
    • <cidr>: A specific CIDR address block or IP address
  • auth: how - What authentication method does this rule specify?
    • deny: Deny access
    • trust: Trust directly, no authentication required
    • pwd: Password authentication, uses md5 or scram-sha-256 authentication based on the pg_pwd_enc parameter
    • sha/scram-sha-256: Force use of scram-sha-256 password authentication.
    • md5: md5 password authentication, but can also be compatible with scram-sha-256 authentication, not recommended.
    • ssl: On top of password authentication pwd, require SSL to be enabled
    • ssl-md5: On top of password authentication md5, require SSL to be enabled
    • ssl-sha: On top of password authentication sha, require SSL to be enabled
    • os/ident: Use ident authentication with the operating system user identity
    • peer: Use peer authentication method, similar to os ident
    • cert: Use client SSL certificate-based authentication, certificate CN is the username
  • user: who: Which users are affected by this rule?
  • db: which: Which databases are affected by this rule?
    • all: All databases
    • replication: Allow replication connections (not specifying a specific database)
    • A specific database

3. Definition Location

Typically, global HBA is defined in all.vars. To modify the global default HBA rules, copy them from conf/ha/full.yml into all.vars and edit them.

Cluster-specific HBA rules are defined in the database cluster-level configuration:

Here are some examples of cluster HBA rule definitions:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:
      - { user: dbuser_view ,db: all    ,addr: infra        ,auth: pwd  ,title: 'Allow dbuser_view password access to all databases from infrastructure nodes'}
      - { user: all         ,db: all    ,addr: 100.0.0.0/8  ,auth: pwd  ,title: 'Allow all users password access to all databases from K8S network'          }
      - { user: '${admin}'  ,db: world  ,addr: 0.0.0.0/0    ,auth: cert ,title: 'Allow admin user to login from anywhere with client certificate'       }

Reloading HBA

HBA is a static rule configuration file that needs to be reloaded to take effect after modification. The default HBA rule set typically doesn’t need to be reloaded because it doesn’t involve Role or cluster members.

If your HBA design uses specific instance role restrictions or cluster member restrictions, then when cluster instance members change (add/remove/failover), some HBA rules’ effective conditions/scope change, and you typically also need to reload HBA to reflect the latest changes.

To reload postgres/pgbouncer hba rules:

bin/pgsql-hba <cls>                 # Reload hba rules for cluster `<cls>`
bin/pgsql-hba <cls> ip1 ip2...      # Reload hba rules for specific instances

The underlying Ansible playbook commands actually executed are:

./pgsql.yml -l <cls> -e pg_reload=true -t pg_hba,pg_reload
./pgsql.yml -l <cls> -e pg_reload=true -t pgbouncer_hba,pgbouncer_reload

Default HBA

Pigsty has a default set of HBA rules that are secure enough for most scenarios. These rules use the alias form, so they are basically self-explanatory.

pg_default_hba_rules:             # postgres global default HBA rules 
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer global default HBA rules 
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' }
Example: Rendered pg_hba.conf
#==============================================================#
# File      :   pg_hba.conf
# Desc      :   Postgres HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:19
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /pg/data/pg_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng (rh@vonng.com)
# License   :   Apache-2.0
#==============================================================#

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu access via local os user ident [default]
local    all                postgres                              ident

# dbsu replication from local os ident [default]
local    replication        postgres                              ident

# replicator replication from localhost [default]
local    replication        replicator                            scram-sha-256
host     replication        replicator         127.0.0.1/32       scram-sha-256

# replicator replication from intranet [default]
host     replication        replicator         10.0.0.0/8         scram-sha-256
host     replication        replicator         172.16.0.0/12      scram-sha-256
host     replication        replicator         192.168.0.0/16     scram-sha-256

# replicator postgres db from intranet [default]
host     postgres           replicator         10.0.0.0/8         scram-sha-256
host     postgres           replicator         172.16.0.0/12      scram-sha-256
host     postgres           replicator         192.168.0.0/16     scram-sha-256

# monitor from localhost with password [default]
local    all                dbuser_monitor                        scram-sha-256
host     all                dbuser_monitor     127.0.0.1/32       scram-sha-256

# monitor from infra host with password [default]
host     all                dbuser_monitor     10.10.10.10/32     scram-sha-256

# admin @ infra nodes with pwd & ssl [default]
hostssl  all                dbuser_dba         10.10.10.10/32     scram-sha-256

# admin @ everywhere with ssl & pwd [default]
hostssl  all                dbuser_dba         0.0.0.0/0          scram-sha-256

# pgbouncer read/write via local socket [default]
local    all                +dbrole_readonly                      scram-sha-256
host     all                +dbrole_readonly   127.0.0.1/32       scram-sha-256

# read/write biz user via password [default]
host     all                +dbrole_readonly   10.0.0.0/8         scram-sha-256
host     all                +dbrole_readonly   172.16.0.0/12      scram-sha-256
host     all                +dbrole_readonly   192.168.0.0/16     scram-sha-256

# allow etl offline tasks from intranet [default]
host     all                +dbrole_offline    10.0.0.0/8         scram-sha-256
host     all                +dbrole_offline    172.16.0.0/12      scram-sha-256
host     all                +dbrole_offline    192.168.0.0/16     scram-sha-256

# allow application database intranet access [common] [DISABLED]
#host    kong            dbuser_kong         10.0.0.0/8          md5
#host    bytebase        dbuser_bytebase     10.0.0.0/8          md5
#host    grafana         dbuser_grafana      10.0.0.0/8          md5
Example: Rendered pgb_hba.conf
#==============================================================#
# File      :   pgb_hba.conf
# Desc      :   Pgbouncer HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:28
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /etc/pgbouncer/pgb_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng (rh@vonng.com)
# License   :   Apache-2.0
#==============================================================#

# PGBOUNCER HBA RULES FOR pg-meta-1 @ 10.10.10.10:6432
# ansible managed: 2023-01-11 14:30:58

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu local admin access with os ident [default]
local    pgbouncer          postgres                              peer

# allow all user local access with pwd [default]
local    all                all                                   scram-sha-256
host     all                all                127.0.0.1/32       scram-sha-256

# monitor access via intranet with pwd [default]
host     pgbouncer          dbuser_monitor     10.0.0.0/8         scram-sha-256
host     pgbouncer          dbuser_monitor     172.16.0.0/12      scram-sha-256
host     pgbouncer          dbuser_monitor     192.168.0.0/16     scram-sha-256

# reject all other monitor access addr [default]
host     all                dbuser_monitor     0.0.0.0/0          reject

# admin access via intranet with pwd [default]
host     all                dbuser_dba         10.0.0.0/8         scram-sha-256
host     all                dbuser_dba         172.16.0.0/12      scram-sha-256
host     all                dbuser_dba         192.168.0.0/16     scram-sha-256

# reject all other admin access addr [default]
host     all                dbuser_dba         0.0.0.0/0          reject

# allow all user intra access with pwd [default]
host     all                all                10.0.0.0/8         scram-sha-256
host     all                all                172.16.0.0/12      scram-sha-256
host     all                all                192.168.0.0/16     scram-sha-256

Security Hardening

For scenarios requiring higher security, use the hardened conf/ha/safe.yml configuration template, which uses the following default HBA rule set:

pg_default_hba_rules:             # postgres host-based auth rules by default
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: ssl   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: ssl   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: ssl   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: ssl   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: cert  ,title: 'admin @ everywhere with ssl & cert'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: ssl   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: ssl   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: ssl   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer host-based authentication rules
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: ssl   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: ssl   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: ssl   ,title: 'allow all user intra access with pwd' }

See Authentication for methods and default boundaries, and Security Considerations for production hardening.