Kubernetes disaster recovery is often treated as a single step—run a backup tool and restore. The three lab‑based failure scenarios demonstrate that a backup’s Completed status, a GitOps sync, and a restored cluster each leave a hidden gap that can break recovery if not explicitly addressed.
Scenario 1 – Verifying that backup data actually left the cluster
The lab uses Velero with its data‑mover to copy volume bytes to an S3‑compatible bucket. A typical check stops at the backup’s Completed phase, assuming the data is safe. The experiment adds a second step: query the DataUpload resource to confirm the byte count that was transferred.
kubectl -n velero get datauploads -l velero.io/backup-name=$BACKUP \ -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,BYTES:.status.progress.bytesDone' NAME PHASE BYTES guestbook-rehearsal-20260727001126-q2j9m Completed 47989888
Seeing ~48 MiB reported confirms that the volume data left the source cluster. The implication is clear: a backup tool that cannot surface this metric should be scrutinized. Moreover, a successful restore that recreates a PostgreSQL pod and returns the original four rows only occurs because the test also configured quiesce hooks and mapped the source storage class to the target environment. Without those, the restored pod could start with a corrupt or empty database.
Scenario 2 – GitOps restores intent, not stored data
In the second failure mode the production cluster is powered off, leaving a pre‑existing recovery cluster that runs a GitOps controller pointed at a Git repository containing only manifests. The controller syncs the StatefulSet, creates a PVC, and the database pod becomes ready. A query then fails with ERROR: relation "attendees" does not exist because the volume is brand new and empty.
This illustrates the classic “GitOps trap”: the version‑controlled YAML defines the desired objects, but it does not carry the persistent data. The backup store holds that data, and only a coordinated restore that pulls both the manifests and the volume snapshots can reconstitute the application state. The lab’s corrective path was to delete the empty resources, invoke Velero to restore the PVCs from the shared bucket, and then validate the rows.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Both scenarios converge on three actionable takeaways:
- End‑to‑end verification is mandatory. Treat the backup’s status as a checkpoint, not a guarantee. Automate checks that confirm byte counts, snapshot integrity, and post‑restore data queries.
- Separate the recovery of infrastructure, declared state, and stored state. Use infrastructure‑as‑code (e.g., Cluster API) to rebuild the cluster, a GitOps pipeline to apply manifests, and a backup tool to restore volumes. Document the exact hand‑off points.
- Test portability and transformation. Restores may land on different nodes, storage classes, or cloud providers. Validate that class mappings, CSI drivers, and any required quiesce hooks work across those boundaries.
Practitioners should add regular rehearsals that cover the full chain—from provisioning a fresh cluster to confirming that application‑level data matches expectations. Monitoring the backup tool’s progress fields and integrating those metrics into alerting can surface failures before a real disaster strikes. Finally, ensure that the external backup bucket is protected with appropriate access controls, because it becomes the single source of truth for stored state.