Skip to content

Administration

Node cluster management SOP - create, destroy, expand, shrink, and handle node/disk failures

Here are common administration operations for the NODE module:

For more questions, see FAQ: NODE

Add Node

To add a node to Pigsty, you need passwordless ssh/sudo access to that node.

You can also add an entire cluster at once, or use wildcards to match nodes in the inventory to add to Pigsty.

# ./node.yml -l <cls|ip|group>        # actual playbook to add nodes to Pigsty
# bin/node-add <selector|ip...>       # add node to Pigsty
bin/node-add node-test                # init node cluster 'node-test'
bin/node-add 10.10.10.10              # init node '10.10.10.10'

Example: Add three nodes of PG cluster pg-test to Pigsty management

demo/node-add.cast

Remove Node

To remove a node from Pigsty, you can use the following commands:

First confirm that every business module on the node has been removed through its own workflow and check whether any vector_data buffer must be retained. After confirming the exact target, call the wrapper:

# ./node-rm.yml -l <cls|ip|group>    # actual playbook that removes a node from Pigsty
# bin/node-rm <cls|ip|selector> ...  # remove node from Pigsty
bin/node-rm node-test                # remove node cluster 'node-test'
bin/node-rm 10.10.10.10              # remove node '10.10.10.10'

You can also remove an entire cluster at once, or use wildcards to match nodes in the inventory to remove from Pigsty.

Here, “remove node” means removing NODE management. The playbook deregisters monitoring, logging, and the HAProxy portal; stops Node Exporter, Vector, HAProxy, and optional VIP services; and deletes vector_data (default: /data/vector). It does not uninstall packages, remove the admin user or node_data, stop Docker, or delete Docker data. See node-rm.yml for the exact boundary.

demo/node-rm.cast

Create Admin

If the current user doesn’t have passwordless ssh/sudo access to the node, you can use another admin user to bootstrap it:

node.yml -t node_admin -k -K -e ansible_user=<another admin>   # enter ssh/sudo password for another admin to complete this task

Bind VIP

You can bind an optional L2 VIP on a node cluster using the vip_enabled parameter.

proxy:
  hosts:
    10.10.10.29: { nodename: proxy-1 }   # you can explicitly specify initial VIP role: MASTER / BACKUP
    10.10.10.30: { nodename: proxy-2 }   # , vip_role: master }
  vars:
    node_cluster: proxy
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.99
    vip_interface: eth1
./node.yml -l proxy -t node_vip     # enable VIP for the first time
./node.yml -l proxy -t vip_refresh  # refresh VIP config (e.g., designate master)

Add Node Monitoring

If you want to add or reconfigure monitoring on existing nodes, use the following commands:

./node.yml -t node_exporter,node_register  # configure monitoring and register
./node.yml -t vector                        # configure log collection

Other Tasks 

# Play
./node.yml -t node                            # complete node initialization (excludes haproxy, monitoring)
./node.yml -t haproxy                         # setup haproxy on node
./node.yml -t monitor                         # configure node monitoring: node_exporter & vector
./node.yml -t node_vip                        # install, configure, enable L2 VIP for clusters without VIP
./node.yml -t vip_config,vip_reload           # refresh node L2 VIP configuration
./node.yml -t haproxy_config,haproxy_reload   # refresh service definitions on node
./node.yml -t register_prometheus             # re-register node with Prometheus
./node.yml -t register_nginx                  # re-register node haproxy admin page with Nginx

# Task
./node.yml -t node-id        # generate node identity
./node.yml -t node_name      # setup hostname
./node.yml -t node_hosts     # configure node /etc/hosts records
./node.yml -t node_resolv    # configure node DNS resolver /etc/resolv.conf
./node.yml -t node_firewall  # configure firewall & selinux
./node.yml -t node_ca        # configure node CA certificate
./node.yml -t node_repo      # configure node upstream software repository
./node.yml -t node_pkg       # install yum packages on node
./node.yml -t node_feature   # configure numa, grub, static network, etc.
./node.yml -t node_kernel    # configure OS kernel modules
./node.yml -t node_tune      # configure tuned profile
./node.yml -t node_sysctl    # set additional sysctl parameters
./node.yml -t node_profile   # configure node environment variables: /etc/profile.d/node.sh
./node.yml -t node_alias     # configure command aliases: /etc/profile.d/node.alias.sh
./node.yml -t node_ulimit    # configure node resource limits
./node.yml -t node_data      # configure node primary data directory
./node.yml -t node_admin     # configure admin user and ssh keys
./node.yml -t node_timezone  # configure node timezone
./node.yml -t node_ntp       # configure node NTP server/client
./node.yml -t node_crontab   # add/overwrite crontab entries
./node.yml -t node_vip       # setup optional L2 VIP for node cluster

HAProxy Password

haproxy_admin_password (default pigsty) is used for HAProxy admin UI authentication, rendered to /etc/haproxy/haproxy.cfg.

After changing the password, use the following to reload config (hot reload, no connection interruption):

./node.yml -l <target> -t haproxy_config,haproxy_reload

Firewall Management

Pigsty uses node_firewall_mode to control firewall behavior. Uses firewalld on RHEL/Rocky and ufw on Debian/Ubuntu.

Since v4.1, this defaults to zone: Pigsty enables the system firewall consistently across distros with an “intranet trusted, public minimized” policy. In zone mode, intranet traffic is unrestricted, but external access is limited to specific ports. Set node_firewall_mode: none only when you want to fully self-manage firewall state and rules. This is especially important when deploying on cloud servers exposed to the internet.

We recommend opening only necessary ports: 22 (SSH), 80/443 (HTTP/HTTPS) are essential. Be cautious about exposing port 5432 (PostgreSQL).

Apply Firewall Rules

zone is already the default. If you previously set none/off, set it back to zone and apply:

node_firewall_mode: zone              # enable firewall with zone rules
node_firewall_intranet:               # trust these CIDRs (full access)
  - 10.0.0.0/8
  - 192.168.0.0/16
  - 172.16.0.0/12
node_firewall_public_port:            # open these ports to public
  - 22                                # SSH
  - 80                                # HTTP
  - 443                               # HTTPS

Then execute: ./node.yml -l <target> -t node_firewall

Open More Ports

To open additional ports, add them to node_firewall_public_port and re-run:

node_firewall_public_port: [22, 80, 443, 5432, 6379]  # add PostgreSQL and Redis ports
./node.yml -l <target> -t node_firewall

Configure Intranet CIDRs

CIDRs in node_firewall_intranet are added to the trusted zone with full access:

node_firewall_intranet:
  - 10.0.0.0/8           # Class A private
  - 192.168.0.0/16       # Class C private
  - 172.16.0.0/12        # Class B private
  - 100.64.0.0/10        # Carrier-grade NAT (if needed)

Remove Rules (Manual)

Important: Pigsty’s firewall management is add-only. Removing entries from config and re-running will NOT delete existing rules. You must remove them manually.

EL (firewalld)
# Remove port from public zone
sudo firewall-cmd --zone=public --remove-port=5432/tcp
sudo firewall-cmd --runtime-to-permanent

# Remove CIDR from trusted zone
sudo firewall-cmd --zone=trusted --remove-source=10.0.0.0/8
sudo firewall-cmd --runtime-to-permanent

# View current rules
sudo firewall-cmd --zone=public --list-ports
sudo firewall-cmd --zone=trusted --list-sources

# Reset to initial state (remove all custom rules)
sudo firewall-cmd --complete-reload
Debian (ufw)
# Delete port rule
sudo ufw delete allow 5432/tcp

# Delete CIDR rule
sudo ufw delete allow from 10.0.0.0/8

# View current rules (numbered)
sudo ufw status numbered

# Delete by rule number
sudo ufw delete <rule_number>

# Reset to initial state (remove all rules, keep ufw enabled)
sudo ufw reset

Disable Firewall

To completely disable the firewall, set node_firewall_mode to off:

node_firewall_mode: off    # completely disable firewall
./node.yml -l <target> -t node_firewall

Or disable manually:

EL (firewalld)
sudo systemctl disable --now firewalld
Debian (ufw)
sudo ufw disable