Single-region PostgreSQL HA is a well-trodden path: a primary, a standby, and streaming replication between them. Multi-region DR introduces problems that setup guides rarely cover — notably that pgBackRest insists on HTTPS endpoints for S3, while running native TLS on local MinIO is tedious to configure. Equally thorny is failback: returning to the original region means navigating timeline conflicts, operator deadlocks, and poisoned WAL archives.
The runbook below walks the full lifecycle with the Crunchy PostgreSQL Operator (PGO) on Kubernetes, using MinIO as the WAL repository: bootstrap, standby replication, simulated regional failure, promotion, and role reversal.
Building a TLS Gateway for MinIO
pgBackRest rejects plain HTTP S3 endpoints, but the local MinIO deployment does not need to speak TLS itself. A small NGINX reverse proxy in the minio namespace terminates TLS on port 443 with a self-signed certificate and forwards plain HTTP to MinIO on port 9000.
The certificate must be valid for the proxy's in-cluster DNS name, minio-secure.minio.svc.cluster.local:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# Generate the private key and self-signed certificate openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ -keyout tls.key \ -out tls.crt \ -subj "/CN=minio-secure.minio.svc.cluster.local" \ -addext "subjectAltName = DNS:minio-secure.minio.svc.cluster.local" # Save the TLS certificate inside the minio namespace kubectl create secret tls minio-secure-tls \ --key tls.key \ --cert tls.crt \ -n minio |
Backups and WAL streams can grow very large, so the default NGINX buffering behavior will stall on the payloads. Request and response buffering must be disabled so database traffic bypasses disk limits:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
# minio-secure-proxy.yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-proxy-config namespace: minio data: nginx.conf: | events { worker_connections 1024; } http { server { listen 443 ssl; server_name minio-secure.minio.svc.cluster.local; ssl_certificate /etc/nginx/ssl/tls.crt; ssl_certificate_key /etc/nginx/ssl/tls.key; # CRITICAL: Disable request body buffering to allow massive backup streams client_max_body_size 0; proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Forward requests to the internal plain HTTP MinIO Service proxy_pass http://minio.minio.svc.cluster.local:9000; } } } apiVersion: apps/v1 kind: Deployment metadata: name: minio-secure namespace: minio spec: replicas: 1 selector: matchLabels: app: minio-secure template: metadata: labels: app: minio-secure spec: containers: - name: nginx image: nginx:alpine ports: - containerPort: 443 volumeMounts: - name: config mountPath: /etc/nginx/nginx.conf subPath: nginx.conf - name: ssl mountPath: /etc/nginx/ssl readOnly: true volumes: - name: config configMap: name: nginx-proxy-config - name: ssl secret: secretName: minio-secure-tls apiVersion: v1 kind: Service metadata: name: minio-secure namespace: minio spec: ports: - port: 443 targetPort: 443 protocol: TCP selector: app: minio-secure |
Apply the secure gateway configuration:
|
1 |
kubectl apply -f minio-secure-proxy.yaml |
Any namespace can then reach MinIO at minio-secure.minio.svc.cluster.local:443.
Bootstrapping Both Regions
Shared credentials
The same MinIO credentials secret is applied to both region-east and region-west:
|
1 2 3 4 5 6 7 8 9 10 11 |
# minio-secret.yaml apiVersion: v1 kind: Secret metadata: name: pgbackrest-s3-creds type: Opaque stringData: s3.conf: | [global] repo1-s3-key=minioadmin repo1-s3-key-secret=minioadmin |
|
1 2 |
kubectl apply -f minio-secret.yaml -n region-east kubectl apply -f minio-secret.yaml -n region-west |
Primary cluster in Region East
The East cluster points backup and WAL archiving at the secure proxy. Because the certificate is self-signed, the manifest sets repo1-s3-verify-tls: "n"; transport is still HTTPS on port 443.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# primary-db.yaml apiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: name: primary-db namespace: region-east spec: postgresVersion: 17 instances: - name: instance1 replicas: 2 dataVolumeClaimSpec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi backups: pgbackrest: configuration: - secret: name: pgbackrest-s3-creds global: repo1-s3-uri-style: "path" repo1-s3-verify-tls: "n" repos: - name: repo1 s3: bucket: pgbackrest endpoint: minio-secure.minio.svc.cluster.local:443 region: us-east-1 |
|
1 |
kubectl apply -f primary-db.yaml |
A standby cannot initialize without a physical base backup, so a manual baseline backup is triggered on the East primary before the West cluster exists:
|
1 |
kubectl annotate postgrescluster primary-db -n region-east --overwrite postgres-operator.crunchydata.com/pgbackrest-backup="$(date +%s)" |
Standby cluster in Region West
The West manifest matches the East one apart from spec.standby.enabled: true, which directs Patroni to boot in continuous archive-recovery mode instead of creating a fresh writable instance.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# standby-db.yaml apiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: name: standby-db namespace: region-west spec: postgresVersion: 17 standby: enabled: true repoName: repo1 instances: - name: instance1 replicas: 1 dataVolumeClaimSpec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi backups: pgbackrest: configuration: - secret: name: pgbackrest-s3-creds global: repo1-s3-uri-style: "path" repo1-s3-verify-tls: "n" repos: - name: repo1 s3: bucket: pgbackrest endpoint: minio-secure.minio.svc.cluster.local:443 region: us-east-1 |
|
1 |
kubectl apply -f standby-db.yaml |
Checking the Replication Path
Write to the East primary:
|
1 2 3 4 5 |
# Exec into the active East primary pod kubectl exec -it -n region-east primary-db-instance1-xxxx-0 -c database -- psql -d postgres # Create database school \c school |
|
1 2 3 |
# Create a test table and insert data CREATE TABLE test_sync (id SERIAL PRIMARY KEY, message TEXT); INSERT INTO test_sync (message) VALUES ('Hello from Region East!'); |
PostgreSQL normally archives a WAL segment only once it is full (16MB). For a quick test, force the current active transaction log to be written and archived immediately:
|
1 |
kubectl exec -it -n region-east primary-db-instance1-xxxx-0 -c database -- psql -d school -c "SELECT pg_switch_wal();" |
The West standby pulls that WAL segment from MinIO and replays it locally:
|
1 2 3 4 5 6 |
kubectl exec -it -n region-west standby-db-instance1-yyyy-0 -c database -- psql -d school -c "SELECT * FROM test_sync;" id | message ----+------------------------- 1 | Hello from Region East! (1 row) |
Failover: Promoting Region West
With a regional outage simulated in the East, the West cluster becomes the active primary.
First, shut down the primary database in the East namespace through the operator:
|
1 |
kubectl patch postgrescluster primary-db -n region-east --type merge --patch '{"spec":{"shutdown": true}}' |
Promotion means patching the West manifest to set spec.standby.enabled to false:
|
1 |
kubectl patch postgrescluster standby-db -n region-west --type merge --patch '{"spec":{"standby":{"enabled": false}}}' |
Patroni then leaves recovery mode and performs a timeline switch, moving from Timeline 1 to Timeline 2. The West database is writable and owns the data:
|
1 2 |
kubectl exec -it -n region-west standby-db-instance1-dhr8-0 -c database -- \ psql -d school -c "INSERT INTO test_sync (message) VALUES ('Hello from the promoted Region West!');" |
Failback: Two Obstacles
Bringing Region East back as a standby of the West primary is not a matter of restarting the old pods. Two failure modes stand in the way.
Operator deadlock. To avoid data loss, the operator leaves the previous instance pod generations running side by side. It will not delete instances that are unhealthy — and they stay unhealthy precisely because of the timeline mismatch.
Archive poisoning. The old East primary may have written Timeline 3 metadata to MinIO before going down. The new West master runs on Timeline 2. A standby will not replicate from a Timeline 2 master while Timeline 3 is present in the archive, because it restores from the highest timeline it can find.
Clean out the East namespace
Stale operator state and physical volumes must be removed to give the database a blank slate.
- Delete the PostgresCluster resource.
|
1 |
kubectl delete postgrescluster primary-db -n region-east |
- Force-delete stuck pods and PVCs.
|
1 2 3 |
kubectl delete pods --all -n region-east --grace-period=0 --force kubectl delete pvc --all -n region-east --grace-period=0 --force |
- Strip finalizers from PVCs trapped in
Terminating.
|
1 |
kubectl get pvc -n region-east -o jsonpath='{.items[*].metadata.name}' | xargs -I {} kubectl patch pvc {} -n region-east -p '{"metadata":{"finalizers":null}}' --type=merge |
- Wipe Patroni's distributed consensus store (DCS) ConfigMaps and Endpoints.
|
1 2 3 |
kubectl delete configmap --all -n region-east kubectl delete endpoints --all -n region-east |
Confirm the namespace is clear:
|
1 |
kubectl get pods,pvc,statefulsets,configmaps,endpoints -n region-east |
Reset the object store
The poisoned Timeline 3 archives have to go. In the MinIO console, or via the S3 CLI, empty and delete the pgbackrest bucket, then recreate it empty.
Re-initialize pgBackRest on the West master
With the bucket empty, the active West master has nothing to archive into. Create the stanza inside its database container and lay down a fresh Timeline 2 backup:
|
1 2 3 |
kubectl exec -it -n region-west standby-db-instance1-dhr8-0 -c database -- \ pgbackrest --stanza=db stanza-create |
|
1 2 3 |
kubectl exec -it -n region-west standby-db-instance1-dhr8-0 -c database -- \ pgbackrest --stanza=db --type=full backup |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
kubectl exec -it -n region-west standby-db-instance1-dhr8-0 -c database -- pgbackrest info Under the db (current) block, your WAL archive range should now cleanly list Timeline 2 files (00000002...): stanza: db status: ok cipher: none db (current) wal archive min/max (17): 000000020000000000000014/000000020000000000000014 |
Re-create Region East as a standby
Update primary-db.yaml so that spec.standby.enabled is true and spec.shutdown is false:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# primary-db.yaml apiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: name: primary-db namespace: region-east spec: postgresVersion: 17 shutdown: false # <-- Ensure the database boots standby: enabled: true # <--- Reconfigured to act as a Standby Leader repoName: repo1 instances: - name: instance1 replicas: 1 # Standardizing on a single standby replica for testing dataVolumeClaimSpec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi backups: pgbackrest: configuration: - secret: name: pgbackrest-s3-creds global: repo1-s3-uri-style: "path" repo1-s3-verify-tls: "n" repos: - name: repo1 s3: bucket: pgbackrest endpoint: minio-secure.minio.svc.cluster.local:443 region: us-east-1 |
|
1 |
kubectl apply -f primary-db.yaml |
Confirm the loop is closed
The standby pod should pull the Timeline 2 backup and bootstrap cleanly:
|
1 |
kubectl get pods -n region-east -w |
Once it reports 4/4 Running, patronictl list should show it joined as a healthy standby leader on Timeline 2:
|
1 2 3 4 5 6 7 8 |
kubectl exec -it -n region-east primary-db-instance1-zzzz-0 -c database -- patronictl list + Cluster: primary-db-ha (7665256549608558683) -----------------------------+----------------+---------------------+----+-------------+-----+------------+-----+ | Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag | +-----------------------------+---------------------------------------------+----------------+---------------------+----+-------------+-----+------------+-----+ | primary-db-instance1-5lm8-0 | primary-db-instance1-5lm8-0.primary-db-pods | Replica | in archive recovery | 2 | 0/12000000 | 16 | 0/13000000 | 0 | | primary-db-instance1-sktj-0 | primary-db-instance1-sktj-0.primary-db-pods | Standby Leader | in archive recovery | 2 | | | | | +-----------------------------+---------------------------------------------+----------------+---------------------+----+-------------+-----+------------+-----+ |
Finally, read the table in Region East to confirm that the write made in the West during the outage has replicated back:
|
1 2 3 4 5 6 7 8 |
kubectl exec -it -n region-east primary-db-instance1-5lm8-0 -c database -- \ psql -d school -c "SELECT * FROM test_sync;" id | message ----+-------------------------------------- 1 | Hello from Region East! 34 | Hello from the promoted Region West! (2 rows) |
What the Exercise Demonstrates
- A terminating TLS proxy lets pgBackRest satisfy its HTTPS requirement against a local MinIO without native TLS or plaintext credentials in the cluster.
- Kubernetes operators deliberately preserve old database state; recovering from that state during a role reversal requires manual cleanup.
- Timeline splits (Timeline 2 versus Timeline 3) will block standby replication, and the fix is to purge and rebuild the WAL repository rather than patch around it.
