Module: ETCD

• 1: Configuration
• 2: Parameters
• 3: Administration
• 4: Playbook
• 5: Monitoring
• 6: Metrics
• 7: FAQ

flowchart LR
    subgraph PGSQL [PGSQL]
        patroni[Patroni]
        vip[VIP Manager]
    end

    subgraph ETCD [ETCD]
        etcd[DCS Service]
    end

    subgraph NODE [NODE]
        node[Software Repo]
    end

    PGSQL -->|depends| ETCD -->|depends| NODE

    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style ETCD fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style patroni fill:#2d4a66,stroke:#1e3347,color:#fff
    style vip fill:#2d4a66,stroke:#1e3347,color:#fff
    style etcd fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

1 - Configuration

Before deployment, define etcd cluster in config inventory. Typical choices:

• One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
• Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
• Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

An even-member Etcd cluster is technically valid, but it does not tolerate more failures than an odd-member cluster with one fewer member, while increasing deployment and quorum cost. Production therefore usually uses one, three, or five members; clusters larger than five are uncommon.

One Node

Define singleton etcd instance in Pigsty—single line of config:

etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

All single-node config templates include this line. Placeholder IP 10.10.10.10 replaced with current admin node’s IP.

Only required params: etcd_seq and etcd_cluster—uniquely identify each etcd instance.

Three Nodes

Most common config: 3-node etcd cluster tolerates 1 node failure, suitable for small-medium prod.

Example: Pigsty’s 3-node templates trio and safe use 3-node etcd:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq (instance number) required
    10.10.10.11: { etcd_seq: 2 }  # positive integers, sequential from 0 or 1
    10.10.10.12: { etcd_seq: 3 }  # immutable for life, never recycled
  vars: # cluster-level params
    etcd_cluster: etcd    # default cluster name: 'etcd', don't change unless deploying multiple etcd clusters
    etcd_safeguard: false # enable safeguard? Enable after prod init to prevent accidental deletion
    etcd_clean: true      # force remove existing during init? Enable for testing for true idempotency

Five Nodes

5-node cluster tolerates 2 node failures, suitable for large prod.

Example: Pigsty’s production simulation template ha/simu uses a 5-node etcd cluster:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

Services Using etcd

Services using etcd in Pigsty:

When etcd cluster membership changes permanently, reload related service configs to ensure correct access.

Update Patroni’s etcd endpoint ref:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager’s etcd endpoint ref (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

RBAC Authentication Config

Since v4.0, Pigsty has enabled etcd RBAC auth by default. Related params:

Prod recommendations:

all:
  vars:
    etcd_root_password: 'YourSecureEtcdPassword'  # change default

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true    # enable safeguard for production

Filesystem Layout

Module creates these directories/files on target hosts:

2 - Parameters

The ETCD module has 13 parameters, divided into two sections:

• ETCD: 10 parameters for etcd cluster deployment and configuration
• ETCD_REMOVE: 3 parameters for controlling etcd cluster removal

Parameter Overview

The ETCD parameter group is used for etcd cluster deployment and configuration, including instance identification, cluster name, data directory, ports, and authentication password.

The ETCD_REMOVE parameter group controls etcd cluster removal behavior, including safeguard protection, data cleanup, and package uninstallation.

ETCD

This section contains parameters for the etcd role, which are used by the etcd.yml playbook.

Parameters are defined in roles/etcd/defaults/main.yml

#etcd_seq: 1                      # etcd instance identifier, explicitly required
etcd_cluster: etcd                # etcd cluster & group name, etcd by default
etcd_learner: false               # run etcd instance as learner? default is false
etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
etcd_port: 2379                   # etcd client port, 2379 by default
etcd_peer_port: 2380              # etcd peer port, 2380 by default
etcd_init: new                    # etcd initial cluster state, new or existing
etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
etcd_root_password: Etcd.Root     # etcd root user password for RBAC authentication (please change!)

etcd_seq

Parameter: etcd_seq, Type: int, Level: I

etcd instance identifier. This is a required parameter—you must assign a unique identifier to each etcd instance.

Here is an example of a 3-node etcd cluster with identifiers 1 through 3:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

etcd_cluster

Parameter: etcd_cluster, Type: string, Level: C

etcd cluster & group name, default value is the hard-coded etcd.

You can modify this parameter when you want to deploy an additional etcd cluster for backup purposes.

etcd_learner

Parameter: etcd_learner, Type: bool, Level: I/A

Initialize etcd instance as learner? Default value is false.

When set to true, the etcd instance will be initialized as a learner, meaning it cannot participate in voting elections within the etcd cluster.

Use Cases:

• Cluster Expansion: When adding new members to an existing cluster, using learner mode prevents affecting cluster quorum before data synchronization completes
• Safe Migration: In rolling upgrade or migration scenarios, join as a learner first, then promote after confirming data synchronization

Workflow:

1. Set etcd_learner: true to initialize the new member as a learner
2. Wait for data synchronization to complete (check with etcdctl endpoint status)
3. Use etcdctl member promote <member_id> to promote it to a full member

etcd_data

Parameter: etcd_data, Type: path, Level: C

etcd data directory, default is /data/etcd.

etcd_port

Parameter: etcd_port, Type: port, Level: C

etcd client port, default is 2379.

etcd_peer_port

Parameter: etcd_peer_port, Type: port, Level: C

etcd peer port, default is 2380.

etcd_init

Parameter: etcd_init, Type: enum, Level: C

etcd initial cluster state, can be new or existing, default value: new.

Option Values:

Important Notes:

Usage Examples:

# Create new cluster (default behavior)
./etcd.yml

# Add new member to existing cluster
./etcd.yml -l <new_ip> -e etcd_init=existing

# Or use the convenience script (automatically sets etcd_init=existing)
bin/etcd-add <new_ip>

etcd_election_timeout

Parameter: etcd_election_timeout, Type: int, Level: C

etcd election timeout, default is 1000 (milliseconds), i.e., 1 second.

etcd_heartbeat_interval

Parameter: etcd_heartbeat_interval, Type: int, Level: C

etcd heartbeat interval, default is 100 (milliseconds).

etcd_root_password

Parameter: etcd_root_password, Type: password, Level: G

etcd root user password for RBAC authentication, default value is Etcd.Root.

Since v4.0, Pigsty has enabled etcd RBAC (Role-Based Access Control) authentication by default. During cluster initialization, the etcd_auth task automatically creates the root user and enables authentication.

Password Storage Location:

• Password is stored in /etc/etcd/etcd.pass file
• File permissions are 0640 (owned by root, readable by etcd group)
• The etcdctl environment script /etc/profile.d/etcdctl.sh automatically reads this file

Integration with Other Components:

• Patroni uses the pg_etcd_password parameter to configure the password for connecting to etcd
• If pg_etcd_password is empty, Patroni will use the cluster name as password (not recommended)
• VIP-Manager also requires the same authentication credentials to connect to etcd

Security Recommendations:

etcd_root_password: 'YourSecurePassword'

ETCD_REMOVE

This section contains parameters for the etcd_remove role, which are action flags used by the etcd-rm.yml playbook.

Parameters are defined in roles/etcd_remove/defaults/main.yml

etcd_safeguard: false             # unconditionally refuse removal when true
etcd_rm_data: true                # remove etcd data and config files during removal?
etcd_rm_pkg: false                # uninstall etcd packages during removal?

etcd_safeguard

Parameter: etcd_safeguard, Type: bool, Level: G/C/A

Removal safeguard, default false. When set to true, etcd-rm.yml aborts before deregistration, leaving the cluster, stopping the service, or deleting data. It is a static boolean switch and does not probe whether the instance is running. Override it explicitly with -e etcd_safeguard=false.

Recommended Settings:

In emergencies, you can override the configuration with command-line parameters:

./etcd-rm.yml -l etcd -e etcd_safeguard=false # Override the guard and remove the target Etcd cluster

etcd_rm_data

Parameter: etcd_rm_data, Type: bool, Level: G/C/A

Remove etcd data and configuration files during removal? Default value is true.

When enabled, the etcd-rm.yml playbook will delete the following contents when removing a cluster or member:

• /etc/etcd/ - Configuration directory (including certificates and password files)
• /var/lib/etcd/ - Alternate data directory
• {{ etcd_data }} - Primary data directory (default /data/etcd)
• /etc/systemd/system/etcd.service - Systemd service unit file
• /etc/profile.d/etcdctl.sh - Client environment script
• /etc/vector/etcd.yaml - Vector log collection config

Use Cases:

# Stop service only, preserve data
./etcd-rm.yml -l etcd -e etcd_rm_data=false

etcd_rm_pkg

Parameter: etcd_rm_pkg, Type: bool, Level: G/C/A

Uninstall etcd packages during removal? Default value is false.

When enabled, the etcd-rm.yml playbook will uninstall etcd packages when removing a cluster or member.

Use Cases:

# Uninstall packages during removal
./etcd-rm.yml -l etcd -e etcd_rm_pkg=true

3 - Administration

Common etcd admin SOPs:

• Create Cluster: Initialize an etcd cluster
• Destroy Cluster: Destroy an etcd cluster
• CLI Environment: Configure etcd client to access server cluster
• RBAC Authentication: Use etcd RBAC auth
• Reload Config: Update etcd server member list for clients
• Add Member: Add new member to existing etcd cluster
• Remove Member: Remove member from etcd cluster
• Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster

For prod etcd clusters, enable safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use the dedicated etcd-rm.yml playbook to destroy an Etcd cluster. The default etcd_rm_data: true deletes local data and configuration. First confirm that no PostgreSQL cluster still uses it as DCS, verify a recent backup, and check the exact target name.

./etcd-rm.yml -l etcd                            # Destroy the entire Etcd cluster after confirmation
./etcd-rm.yml -l etcd -e etcd_safeguard=false   # Override only when the inventory enables the safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

The removal playbook respects etcd_safeguard. If true, it aborts before leaving the cluster, deregistering, stopping the service, or deleting files. Its default is false, so the absence of an explicit override is not itself a confirmation.

CLI Environment

Uses etcd v3 API by default (v2 removed in v3.6+). Pigsty auto-configures env script /etc/profile.d/etcdctl.sh on etcd nodes, loaded on login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

Since v4.0, Pigsty has enabled RBAC auth for etcd by default, so user auth is still required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After configuring client env, run etcd CRUD ops:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

Since v4.0, Pigsty has enabled etcd RBAC auth by default. During cluster init, etcd_auth task auto-creates root user and enables auth.

Root user password set by etcd_root_password, default: Etcd.Root. Stored in /etc/etcd/etcd.pass with 0640 perms (root-owned, etcd-group readable).

Strongly recommended to change default password in prod:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password to configure etcd connection password. If empty, Patroni uses cluster name as password (not recommended). Configure separate etcd password per PG cluster in prod.

Reload Config

If etcd cluster membership changes (add/remove members), refresh etcd service endpoint references. These etcd refs in Pigsty need updates:

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

The script attempts these operations in order:

• Validates IP address validity
• Executes etcd.yml playbook (auto-sets etcd_init=existing)
• Provides safety warnings and countdown
• Prompts config refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

1. Update config inventory: Add new instance to etcd group
2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
3. Initialize new member: Run playbook with etcd_init=existing parameter
4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
5. Reload config: Update etcd endpoint references for all clients

# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

$ bin/etcd-add 10.10.10.11

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script auto-performs:

• Gracefully removes members from cluster
• Stops and disables etcd service
• Cleans up data and config files
• Deregisters from monitoring system

The underlying removal role tolerates some leave and cleanup errors. After the script finishes, inspect etcdctl member list, endpoint health, remaining quorum, and the actual service and data-directory state on the target.

Manual: Step-by-Step

Remove member instance from etcd cluster:

1. Keep the member in the inventory: The removal playbook needs its etcd_seq, cluster members, and connection endpoints
2. Clean up the instance: Run etcd-rm.yml against the target; it attempts member remove, stops the service, and cleans up according to the removal flags
3. Update the inventory: Comment out or delete the member only after the playbook succeeds
4. Reload references: Follow Reload Config for the remaining etcd members and the Patroni/VIP-Manager endpoints

# <ip> must still be in the etcd inventory group
./etcd-rm.yml -l <ip>                  # Leave the cluster and clean up automatically
# After success, remove the member from pigsty.yml and refresh the remaining members and clients

Do not delete the target from the inventory before running the removal playbook. The hosts: etcd scope in etcd-rm.yml would no longer select it, and the playbook could not derive the instance identity or cluster endpoints from inventory. There is also no need to repeat etcdctl member remove before or after the playbook.

$ bin/etcd-rm 10.10.10.12

$ ./etcd-rm.yml -l 10.10.10.12

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

• Validates IP addresses in config inventory
• Auto-sets etcd_init=existing parameter
• Executes etcd.yml playbook to complete member addition
• Prompts config refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

• Provides safety warnings and confirmation countdown
• Auto-executes etcd-rm.yml playbook
• Gracefully removes members from cluster
• Cleans up data and config files

4 - Playbook

The ETCD module provides two core playbooks: etcd.yml for installing and configuring etcd clusters, and etcd-rm.yml for removing etcd clusters or members.

etcd.yml

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group, then launches the etcd service.

The following subtasks are available in etcd.yml:

• etcd_assert : Validate etcd identity parameters (etcd_seq must be defined as a non-negative integer)
• etcd_install : Install etcd packages
• etcd_dir : Create etcd data and configuration directories
• etcd_config : Generate etcd configuration
  • etcd_conf : Generate etcd main config file /etc/etcd/etcd.conf
  • etcd_cert : Generate etcd TLS certificates (CA, server cert, private key)
• etcd_member : Add new member to existing cluster (only runs when etcd_init=existing)
• etcd_launch : Launch etcd service
• etcd_auth : Enable RBAC authentication (create root user and enable auth)
• etcd_register : Register etcd to VictoriaMetrics monitoring

etcd-rm.yml

Playbook source: etcd-rm.yml

A dedicated playbook for removing etcd clusters or individual members. The following subtasks are available in etcd-rm.yml:

• etcd_safeguard : Check safeguard and abort if enabled
• etcd_pause : Pause for 3 seconds, allowing user to abort with Ctrl-C
• etcd_deregister : Remove etcd registration from VictoriaMetrics monitoring targets
• etcd_leave : Try graceful leaving etcd cluster before purge
• etcd_svc : Stop and disable etcd service with systemd
• etcd_data : Remove etcd data (disable with etcd_rm_data=false)
• etcd_pkg : Uninstall etcd packages (enable with etcd_rm_pkg=true)

The removal playbook uses the etcd_remove role with the following configurable parameters:

• etcd_safeguard: Prevents accidental removal when set to true
• etcd_rm_data: Controls whether ETCD data is deleted (default: true)
• etcd_rm_pkg: Controls whether ETCD packages are uninstalled (default: false)

Demo

Cheatsheet

Etcd Installation & Configuration:

./etcd.yml                                      # Initialize etcd cluster
./etcd.yml -t etcd_launch                       # Restart entire etcd cluster
./etcd.yml -t etcd_conf                         # Refresh /etc/etcd/etcd.conf with latest state
./etcd.yml -t etcd_cert                         # Regenerate etcd TLS certificates
./etcd.yml -l 10.10.10.12 -e etcd_init=existing # Scale out: add new member to existing cluster

Etcd Removal & Cleanup:

./etcd-rm.yml -l 10.10.10.12                       # Leave, deregister, stop, and delete local data by default
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_data=false # Leave, deregister, and stop, preserving local data/config
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_pkg=true   # Also uninstall Etcd packages
./etcd-rm.yml -l etcd                              # Destroy the entire cluster and its local data

Convenience Scripts:

bin/etcd-add <ip>                               # Add new member to existing cluster (recommended)
bin/etcd-rm <ip>                                # Remove specific member from cluster (recommended)
bin/etcd-rm                                     # Remove entire etcd cluster

Safeguard

To prevent accidental deletion, Pigsty’s ETCD module provides a safeguard mechanism controlled by the etcd_safeguard parameter, which defaults to false (safeguard disabled).

For production etcd clusters that have been initialized, it’s recommended to enable the safeguard to prevent accidental deletion of existing etcd instances:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true  # Enable safeguard protection

When etcd_safeguard is set to true, etcd-rm.yml aborts before any deregistration, cluster-leave, service-stop, or deletion action. It is a boolean guard and does not probe whether the instance is alive. Override it with a command-line parameter:

./etcd-rm.yml -l <exact-target> -e etcd_safeguard=false  # Override only after verifying target and backups

Regardless of the safeguard value, inspect etcdctl member list, endpoint health, and remaining quorum after a real run; a successful task status is not runtime acceptance.

5 - Monitoring

Dashboards

ETCD module provides one monitoring dashboard: Etcd Overview.

ETCD Overview Dashboard

ETCD Overview: Overview of ETCD cluster

Dashboard provides key ETCD status info. Notable: ETCD Aliveness—shows overall etcd cluster service status.

Red bands = instance downtime; blue-gray below = cluster unavailable.

Alert Rules

Pigsty provides five preset alert rules for etcd, defined in files/victoria/rules/etcd.yml:

• EtcdServerDown: etcd node down, CRIT alert
• EtcdNoLeader: etcd cluster no leader, CRIT alert
• EtcdQuotaFull: etcd quota > 90%, WARN alert
• EtcdNetworkPeerRTSlow: etcd network latency slow, INFO alert
• EtcdWalFsyncSlow: etcd disk fsync slow, INFO alert

The following excerpt mirrors the current rule source. Pigsty currently ships only the etcd-overview dashboard; the /ui/d/etcd-instance targets in the comments for the two latency alerts do not exist. Use /ui/d/etcd-overview to inspect cluster state. This is a known source-comment discrepancy and does not affect the alert expressions.

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# etcd server instance down
- alert: EtcdServerDown
  expr: etcd_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      etcd_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# Etcd no Leader triggers P0 alert immediately
# if dcs_failsafe mode not enabled, may cause global outage
- alert: EtcdNoLeader
  expr: min(etcd_server_has_leader) by (cls) < 1
  for: 15s
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      etcd_server_has_leader[cls={{ $labels.cls }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

#==============================================================#
#                        Saturation                            #
#==============================================================#
- alert: EtcdQuotaFull
  expr: etcd:cls:quota_usage > 0.90
  for: 1m
  labels: { level: 1, severity: WARN, category: etcd }
  annotations:
    summary: "WARN EtcdQuotaFull: {{ $labels.cls }}"
    description: |
      etcd:cls:quota_usage[cls={{ $labels.cls }}] = {{ $value | printf "%.3f" }} > 90%
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Latency                              #
#==============================================================#
# etcd network peer rt p95 > 200ms for 1m
- alert: EtcdNetworkPeerRTSlow
  expr: etcd:ins:network_peer_rt_p95_5m > 0.200
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdNetworkPeerRTSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:network_peer_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 200ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}
# Etcd wal fsync rt p95 > 50ms
- alert: EtcdWalFsyncSlow
  expr: etcd:ins:wal_fsync_rt_p95_5m > 0.050
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdWalFsyncSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:wal_fsync_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 50ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

6 - Metrics

This snapshot records 177 monitoring metric families for the ETCD module. The metrics present at runtime vary with package version, enabled collectors, and target state.

7 - FAQ

What is etcd’s role in Pigsty?

etcd is a distributed, reliable key-value store for critical system data. Pigsty uses etcd as DCS (Distributed Config Store) service for Patroni, storing PG HA status.

Patroni uses etcd for: cluster failure detection, auto failover, primary-replica switchover, and cluster config management.

etcd is critical for PostgreSQL HA, and its own availability depends on a reachable majority. Production deployments normally spread three or five voting members across independent failure domains.

What’s the appropriate etcd cluster size?

If more than half (including exactly half) of etcd instances unavailable, etcd cluster enters unavailable state—refuses service.

Example: 3-node cluster allows max 1 node failure while 2 others continue; 5-node cluster tolerates 2 node failures.

Note: Learner instances don’t count toward members—3-node cluster with 1 learner = 2 actual members, zero fault tolerance.

In prod, use odd number of instances. For prod, recommend 3-node or 5-node for reliability.

Impact of etcd unavailability?

If etcd cluster unavailable, affects PG control plane but not data plane—existing PG clusters continue running, but Patroni management ops fail.

During etcd failure: PG HA can’t auto failover, can’t use patronictl for PG management (config changes, manual failover, etc.).

Ansible playbooks unaffected by etcd failure: create DB, create user, refresh HBA/Service config. During etcd failure, operate PG clusters directly.

Note: Behavior applies to Patroni >=3.0 (Pigsty >=2.0). With older Patroni (<3.0, Pigsty 1.x), etcd/consul failure causes severe global impact:

All PG clusters demote: primaries → replicas, reject writes, etcd failure amplifies to global PG failure. Patroni 3.0 introduced DCS Failsafe—significantly improved.

What data does etcd store?

By default, Pigsty uses etcd as Patroni’s DCS, where it stores coordination data such as leader leases, member state, and dynamic configuration. Pigsty itself does not store application data there.

Patroni creates and manages this DCS data. During controlled maintenance, it can normally reconstruct coordination state from a healthy PostgreSQL cluster, but that does not make etcd stateless or make direct DCS deletion risk-free.

Rebuilding etcd interrupts automatic failover and patronictl management and clears the current DCS state. Before doing so, inspect the Patroni topology, current primary, remaining quorum, and recent backups, and execute a documented recovery procedure during a maintenance window.

If using etcd for other purposes (K8s metadata, custom storage), backup etcd data yourself and restore after cluster recovery.

Recover from etcd failure?

By default, Pigsty uses etcd only as Patroni DCS. Restarting services and rebuilding the whole cluster have very different risk: a restart preserves DCS data, while a rebuild clears coordination state and leaves PostgreSQL HA without DCS quorum until recovery. Diagnose and recover existing members first; rebuild only after verifying the topology, backups, and recovery path.

Restart etcd cluster:

./etcd.yml -t etcd_launch

If a full reset/rebuild is genuinely required, do it in a maintenance window and then verify etcdctl endpoint health, etcdctl member list, and patronictl list:

./etcd-rm.yml -l etcd          # Clean the cluster after verifying target and backups
./etcd.yml -l etcd             # Redeploy from inventory

For custom etcd data: backup and restore after recovery.

Etcd maintenance considerations?

Simple answer: don’t fill up etcd.

Pigsty enables etcd auto-compaction by default, with the current backend quota set to 8 GiB. This is usually sufficient, but actual usage should still be monitored.

etcd’s data model = each write generates new version.

Frequent writes (even few keys) = growing etcd DB size. At capacity limit, etcd rejects writes → PG HA breaks.

Pigsty’s default etcd config includes optimizations:

auto-compaction-mode: periodic      # periodic auto compaction
auto-compaction-retention: "24h"    # retain 24 hours history
quota-backend-bytes: 8589934592     # 8 GiB quota

More details: etcd official maintenance guide.

Enable etcd auto garbage collection?

Earlier Pigsty (v2.0 - v2.5)? Enable etcd auto-compaction in prod to avoid quota-based unavailability.

Edit the etcd configuration template: roles/etcd/templates/etcd.conf:

auto-compaction-mode: periodic
auto-compaction-retention: "24h"
quota-backend-bytes: 17179869184

Then set related PG clusters to maintenance mode and redeploy etcd with ./etcd.yml.

This increases default quota from 2 GiB → 16 GiB, retains last 24h writes—avoids infinite growth.

Where is PG HA data stored in etcd?

By default, Patroni uses pg_namespace prefix (default: /pg) for all metadata keys, followed by PG cluster name.

Example: PG cluster pg-meta stores metadata under /pg/pg-meta.

etcdctl get /pg/pg-meta --prefix

Sample data:

/pg/pg-meta/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":false,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":16,"pg_cluster":"pg-meta","pg_shard":"pg-meta","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":100,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":"on","wal_level":"logical","wal_log_hints":"on","max_worker_processes":16,"max_wal_senders":50,"max_replication_slots":50,"password_encryption":"scram-sha-256","ssl":"on","ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/pg/cert/ca.crt","shared_buffers":"7969MB","maintenance_work_mem":"1993MB","work_mem":"79MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"7GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"7GB","max_wal_size":"28GB","max_slot_wal_keep_size":"42GB","wal_buffers":"16MB","wal_writer_delay":"20ms","wal_writer_flush_after":"1MB","commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":"on","archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-meta archive-push %p","max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":"on","wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"23907MB","default_statistics_target":200,"log_destination":"csvlog","logging_collector":"on","l...
ode=prefer"}}
/pg/pg-meta/failsafe
{"pg-meta-2":"http://10.10.10.11:8008/patroni","pg-meta-1":"http://10.10.10.10:8008/patroni"}
/pg/pg-meta/initialize
7418384210787662172
/pg/pg-meta/leader
pg-meta-1
/pg/pg-meta/members/pg-meta-1
{"conn_url":"postgres://10.10.10.10:5432/postgres","api_url":"http://10.10.10.10:8008/patroni","state":"running","role":"primary","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"timeline":1}
/pg/pg-meta/members/pg-meta-2
{"conn_url":"postgres://10.10.10.11:5432/postgres","api_url":"http://10.10.10.11:8008/patroni","state":"running","role":"replica","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"replication_state":"streaming","timeline":1}
/pg/pg-meta/status
{"optime":184549376,"slots":{"pg_meta_2":184549376,"pg_meta_1":184549376},"retain_slots":["pg_meta_1","pg_meta_2"]}

Use external existing etcd cluster?

Config inventory hardcodes etcd group—members used as DCS servers for PGSQL. Initialize with etcd.yml or assume external cluster exists.

To use external etcd: define as usual. Skip etcd.yml execution since cluster exists—no deployment needed.

Requirement: external etcd cluster certificate must use same CA as Pigsty—otherwise clients can’t use Pigsty’s self-signed certs.

Add new member to existing etcd cluster?

For detailed process, refer to Add member to etcd cluster

Recommended: Utility script

# First add new member to config inventory, then:
bin/etcd-add <ip>      # add single new member
bin/etcd-add <ip1>     # add multiple new members

Manual method:

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380 # announce new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing                                 # initialize new member
etcdctl member promote <new_ins_server_id>                                       # promote to full member

Recommend: add one new member at a time.

Remove member from existing etcd cluster?

For detailed process, refer to Remove member from etcd cluster

Recommended: Utility script

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm                   # remove entire etcd cluster

Manual method:

./etcd-rm.yml -l <ins_ip>                    # Leave, stop, and delete local data by default

etcd-rm.yml already includes the etcdctl member remove step; do not repeat it before or after the normal workflow. Use a manual member remove only for troubleshooting. You can still run the removal playbook once while the target remains in inventory to stop, deregister, and clean it locally, then verify the remaining quorum.

Configure etcd RBAC authentication?

Since v4.0, Pigsty has enabled etcd RBAC auth by default. Root password set by etcd_root_password, default: Etcd.Root.

Prod recommendation: change default password

all:
  vars:
    etcd_root_password: 'YourSecurePassword'

Client auth:

# On etcd nodes, env vars auto-configured
source /etc/profile.d/etcdctl.sh
etcdctl member list

# Manual auth config
export ETCDCTL_USER="root:YourSecurePassword"
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

More: RBAC Authentication.