Skip to content

15. Destroy & rebuild safely

Goal: tear down the cluster without losing data, and come back up in the same state — same database content, same pgAdmin preferences.

What survives and what doesn't

Data Where it lives Survives terraform destroy?
PostgreSQL data Longhorn PVC → continuous WAL archive to R2 ✓ (in R2)
PostgreSQL base backup R2 via Barman Cloud Plugin ✓
pgAdmin preferences & passwords Longhorn PVC (pgadmin-data) ✗ — unless backed up
pgAdmin saved server connections servers.json ConfigMap ✓ (in Git)
pgAdmin DB connection password .pgpass built from pg-app secret ✓ (rebuilt on deploy)

The Longhorn volumes are in-cluster storage. When the cluster is destroyed, they go with it. The destroy script forces a backup of the pgAdmin SQLite database to R2 before anything is torn down, so the rebuild starts where the teardown ended.

The destroy script

chmod +x destroy.sh
./destroy.sh

What it does, in order:

flowchart TD
    A[Check kubectl connection] --> B[Force pgAdmin SQLite → R2]
    B --> C[Trigger CNPG on-demand backup]
    C --> D{Backup completed?}
    D -->|yes| E[Confirm]
    D -->|timeout / failed| F[Ask to proceed anyway]
    F --> E
    E --> G[terraform destroy]
    G --> H[✓ Cluster gone, data in R2]

Step-by-step

1. pgAdmin SQLite backup — execs into the running pgadmin-backup sidecar and copies /var/lib/pgadmin/pgadmin4.db to s3://barman-cloud/pgadmin/pgadmin4.db. This sidecar already has the R2 credentials loaded, so no extra config is needed. The backup is non-fatal: if pgAdmin is not deployed the script continues.

2. PostgreSQL on-demand backup — applies a Backup resource pointing at pg-r2-store. The script polls until status.phase == completed (5-minute timeout). If the backup doesn't complete, you are prompted to proceed or abort.

3. Confirmation — prints the R2 paths where the data landed and asks you to type yes before destroying.

4. terraform destroy — tears down all Hetzner nodes and in-cluster resources.

Skip confirmation (CI / automation)

./destroy.sh --yes

--yes skips both the "backup failed, continue?" prompt and the final confirmation. Backup failures are still logged to stdout. Use with care.

How the rebuild restores state

sequenceDiagram
    participant You
    participant TF as terraform apply / deploy.sh
    participant K as k3s + operators
    participant Op as CNPG operator
    participant R2 as R2 (barman-cloud bucket)

    You->>TF: ./deploy.sh
    TF->>K: nodes, Longhorn, operators, manifests
    K->>Op: cluster.yaml.tpl with bootstrap.recovery
    Op->>R2: fetch base backup + WAL stream
    R2-->>Op: restore + replay to latest
    Note over K: pgadmin-restore initContainer
    K->>R2: download pgadmin4.db
    R2-->>K: preferences, confirmations, history
    Op-->>You: healthy cluster, same DB, same pgAdmin state

pgAdmin persistence in detail

The pgAdmin deployment has three components that together make it stateless from the operator's point of view:

Component Role
pgpass-init initContainer Builds /pgadmin4/.pgpass from the pg-app secret (CNPG-managed). Connection password never needs to be saved manually.
pgadmin-restore initContainer On pod start: if no pgadmin4.db on the PVC, downloads it from R2.
pgadmin-backup sidecar Every 5 minutes, uploads pgadmin4.db to R2. The destroy script forces one final upload before teardown.

First deploy

On the very first deploy there is no backup in R2 yet. The restore initContainer will log "No backup found, starting fresh" and pgAdmin boots with an empty database. Configure your preferences once; from the next destroy onward they are automatically preserved.

Continuous WAL archiving means you don't need to wait

The on-demand backup in the destroy script is a safety belt, not the primary protection. WAL files are archived to R2 continuously (every completed segment, roughly every 16 MB or 5 minutes). Even if you skip the destroy script and terraform destroy the cluster cold, the database can be recovered to within minutes of the last WAL flush.

The on-demand backup gives you a clean checkpoint to restore from, which makes the rebuild faster (less WAL to replay) and validates that the base-backup path works.

Checklist before destroying

  • [ ] ./destroy.sh ran without errors
  • [ ] "PostgreSQL backup completed" line visible in output
  • [ ] "pgAdmin backup OK" line visible in output
  • [ ] R2 bucket reachable (check via Cloudflare dashboard or aws s3 ls --endpoint-url ...)

Rebuilding

export TF_VAR_infisical_client_id="<id>"
export TF_VAR_infisical_client_secret="<secret>"
./deploy.sh

cluster.yaml.tpl is configured for bootstrap.recovery by default, so the CNPG operator will pull the latest backup from R2 automatically. pgAdmin will restore its preferences from the last snapshot. No manual steps required.

Where to go deeper