How Resilient Is Your Postgres?
Thursday, October 01 · 16:00–16:50
Chaos engineering has been a pioneering discipline within SRE and resiliency practice for over a decade, and tools like LitmusChaos brought that discipline to stateful workloads early on. But most chaos testing, database included, still stops at infrastructure level: kill the pod, watch it come back. Far fewer teams have pushed into the harder question underneath that: what happens when the database dies not at rest, but mid schema change, under real load.
In this session, we will run a chaos experiment using Harness Resilience Testing against a self hosted Postgres cluster on Kubernetes: killing the database master mid schema change, while simulating heavy production traffic against the database at the same time. We will walk through recovery step by step, from failover to the new master through what actually happens to an in flight schema change that gets interrupted, and validate the only question that really matters: not whether the pipeline turned green again, but whether every committed row survived intact. You will leave with a repeatable blueprint for testing the failure mode you are actually afraid of, not the easy one.
