Let's say the asteroid hits.
Not metaphorically. The building containing the PostgreSQL primary is gone, along with the local cluster, its storage and everything else that normally makes the site authoritative.
Fortunately, there is a remote replica somewhere else.
Now we have to decide when that replica becomes primary.
At first this sounds like an ordinary failover problem. The primary stops responding, monitoring notices, some interval passes and another system takes over. That's how a great deal of highly available infrastructure is designed, and there are good ways to do it. Quorum, fencing and consensus mechanisms exist precisely because distributed systems have spent decades dealing with machines that disappear without explaining why.
The problem in this particular architecture is that absence doesn't tell us what happened.
A destroyed building is absent. So is a building with a severed fiber connection. A failed router can make a healthy database absent from the perspective of another site. So can a configuration error, a power outage or a network partition.
Those failures may look similar from far away while requiring very different responses.
If the original primary still exists and can accept writes, promoting another copy without first establishing authority creates the possibility that two databases will develop different histories. Transactions accepted on one side may never have existed on the other.
That's the failure I'm more interested in avoiding.
PostgreSQL doesn't decide the larger policy for us. It gives us the mechanisms from which replication and promotion can be built, while the surrounding architecture has to determine when promotion is appropriate.
There are environments where automating that decision makes perfect sense. If the system has reliable fencing, enough independent participants to establish quorum and a well-defined procedure that prevents the old primary from continuing to act as primary, automated failover can provide availability without casually accepting split-brain.
That's not the architecture I'm describing here.
I'm starting from a different tradeoff: a locally authoritative system with a geographically distant copy whose primary purpose is to survive the loss of the local site.
Under those conditions, I don't particularly care whether the remote copy becomes writable thirty seconds after the local site disappears.
I care whether it becomes writable correctly.
So if the local site suddenly stops communicating, the remote replica does something remarkably undramatic.
It waits.
Alerts can fire immediately. Monitoring can collect whatever evidence remains available. Recovery tooling can prepare everything necessary for promotion. What doesn't happen automatically is the transfer of authority itself.
Someone first determines what actually happened.
Perhaps the fiber was cut and the local site is fine. In that case there may be nothing to promote.
Perhaps the building really is a crater.
Now the decision is different. We inspect the remote replica, determine its recovery point, understand what transactions may have been lost and decide whether we're willing to make that history authoritative.
Then we promote it.
Everything after that decision can be automated. Scripts can perform the promotion, change service discovery, update routing and bring applications back against the new primary. There's no virtue in making a person manually type twenty commands just because a person made the decision.
The distinction is between automating the procedure and delegating the authority to begin it.
For this system, I want the procedure automated and the exceptional transfer of authority witnessed.
That's partly because I'm optimizing for something different from continuous availability.
If this were a service where several seconds of downtime carried enormous consequences, I'd make different tradeoffs. I'd probably build enough independent infrastructure that the system co