Adding PostgreSQL changes what a Kubernetes cluster is being asked to do.
Until this point in the build, most of the workloads have been relatively forgiving. A web service disappears, Kubernetes starts another one. An ingress pod moves to another node. A container is replaced from an image that already contains everything necessary to recreate it.
That doesn't make stateless infrastructure trivial, but it gives us a useful escape hatch: if a process disappears, we can usually make another one.
A database introduces a different requirement.
If PostgreSQL disappears and comes back, creating another PostgreSQL process isn't enough. The new process has to find the same database, containing the transactions that were committed before the interruption. We aren't merely asking Kubernetes to recreate a service anymore. We're asking the surrounding storage and recovery architecture to preserve state across the recreation.
That's a useful boundary to cross deliberately.
The architecture I'm describing remains intentionally modest. There is one authoritative PostgreSQL service on the local infrastructure, running against persistent storage that survives the database pod. Remote copies and backups exist for disaster recovery, but we're not trying to make one database simultaneously writable across several geographic locations.
That decision eliminates some interesting capabilities and a considerable amount of complexity.
It also gives us a very simple question to test first.
What happens if we delete the database pod?
Kubernetes will notice that the expected PostgreSQL instance is gone and create a replacement according to whatever controller or PostgreSQL operator we're using. The persistent volume is made available to the replacement instance, PostgreSQL starts, performs whatever crash recovery is necessary and resumes from the state recorded on disk.
If everything underneath it works as intended, the application eventually reconnects and the database looks remarkably uneventful.
That's exactly what we want.
The interesting part isn't that PostgreSQL restarted. It's that the rows written before the restart are still there.
That distinction sounds obvious until you start testing it.
A stateless application can hide quite a few misunderstandings because recreation is part of its normal behavior. Delete it, replace it and move on. Persistent state makes assumptions accumulate across layers. PostgreSQL depends on the volume. The volume depends on the storage system. The storage system depends on its own replication and failure behavior. The application depends on the database returning with the state it expects.
Deleting the pod gives that whole chain a small disturbance and lets us see what actually survives.
So I would test it early, before the database contains anything difficult to replace.
Create some data.
Delete the pod.
Wait for PostgreSQL to return.
Read the data.
Then make the test less polite.
Restart the node hosting it. Interrupt storage under controlled conditions. Verify that the database behaves the way the design says it should rather than the way we remember configuring it six months ago.
This is where PostgreSQL becomes useful as more than another workload. It forces several layers of the cluster to demonstrate their behavior together.
It also helps separate failure cases that are easy to blur when we use the word redundancy for all of them.
A PostgreSQL process can fail while its data remains intact.
A Kubernetes node can fail while another node and the storage system remain available.
The local storage system can suffer failures that its own replication is designed to tolerate.
Then there is a different category of event in which the entire local site disappears.
Those aren't the same problem, so I don't want one failover mechanism pretending they are.
Within the local environment, automated recovery can make perfect sense. A properly designed PostgreSQL cluster can elect or promote anoth