Skip to content

13. Toward full IaC

Goal: turn the manual Layer 2 steps into code so a fresh cluster comes up fully configured — and, when you want, restores its data from R2 automatically. This is the payoff of the whole guide: from zero to a working, populated database in minutes.

Where we are

flowchart LR
    L1["Layer 1 ✓<br/>platform in kube.tf"] --> L2["Layer 2 ✓<br/>stack applied by hand"]
    L2 --> L3["Layer 3 (this chapter)<br/>stack as code + auto-restore"]

You have done Layer 2 by hand enough times to understand every resource. Now we remove the repetition.

Option A — kube-hetzner extra-manifests (simplest)

The module can apply your own manifests right after the cluster is created, using a Kustomization. Put all your Layer 2 YAML into the module's extra-manifests/ directory and reference them from a kustomization template; the module runs kubectl apply -k for you.

  # in kube.tf
  extra_kustomize_deployment_commands = <<-EOT
    kubectl rollout status deployment -n cnpg-system cnpg-controller-manager --timeout=300s
    kubectl wait --for=condition=Ready cluster/pg -n production --timeout=600s || true
  EOT

extra-manifests/
  kustomization.yaml.tpl     # lists the resources below
  00-namespace.yaml
  10-operator.yaml           # or install via the commands hook
  11-barman-plugin.yaml
  20-image-catalog.yaml
  30-objectstore.yaml        # secrets via a sealed/external secret, not plaintext
  40-cluster.yaml
  50-pooler.yaml
  60-scheduled-backup.yaml
  70-networkpolicy.yaml

Order matters (see the dependency chain); name files so they apply in sequence. With this, terraform apply brings up the platform and the database stack in one shot.

Secrets do not belong in Git

Do not commit r2-credentials or DB passwords as plaintext manifests. Use Sealed Secrets or the External Secrets Operator so the repo holds only encrypted/templated references.

Option B — GitOps (Argo CD / Flux)

For a cleaner separation, install a GitOps controller and let it sync your manifests from a Git repo. The cluster continuously reconciles to what is in Git; you change the database by committing, not by running kubectl.

flowchart LR
    git["Git repo<br/>(Layer 2 manifests)"] --> argo["Argo CD / Flux"]
    argo -->|sync| cluster["k3s cluster"]
    cluster -->|drift?| argo

This is the better long-term home for Layer 2, and it pairs naturally with Sealed Secrets for the credential problem.

The "from zero with your data" flow

This is the dream, made concrete. To stand up a cluster that restores from R2 instead of starting empty, your committed Cluster uses bootstrap.recovery (from chapter 11) instead of bootstrap.initdb:

sequenceDiagram
    participant You
    participant TF as terraform apply
    participant K as k3s + add-ons
    participant Op as CNPG operator
    participant R2 as R2 backups
    You->>TF: apply
    TF->>K: 3 nodes, cert-manager, Longhorn
    K->>Op: operator + barman plugin deployed
    Op->>R2: read base backup + WAL
    R2-->>Op: restore + replay
    Op-->>You: populated, healthy cluster (minutes)

A pragmatic pattern: keep two Cluster manifests in your repo — an initdb one for a clean start, and a recovery one for rebuild-with-data — and choose which to apply per situation (or parameterize with Kustomize overlays).

Hardening for the long-lived cluster

When you graduate from daily teardown to a stable service, revisit these learning-phase compromises:

  • Dedicated nodes for the database. Move Postgres + Longhorn off the control-plane nodes onto dedicated agent_nodepools; keep allow_scheduling_on_control_plane = false. Protects etcd from DB I/O.
  • Pin every add-on version (cert_manager_version, longhorn_version, operator, plugin, operand digest) — see Versions.
  • Kured maintenance windows so reboots happen at quiet hours.
  • reclaimPolicy: Retain on the Postgres StorageClass to protect volumes.
  • Real secret management (External Secrets / Sealed Secrets).
  • Validated restore wired into a periodic job, not a one-off check.

A suggested repository layout

infra/
  terraform/            # Layer 1: kube.tf, variables, backend
  k8s/
    base/               # Layer 2 manifests (Kustomize base)
    overlays/
      learning/         # initdb cluster, 1-replica storage
      production/       # recovery cluster, dedicated nodes, Retain
  docs/                 # this guide

Where to go deeper

That completes the build. See the Operations runbook for day-to-day commands, the Glossary for terms, and Further reading for sources.