Once the local cluster can survive the failures we've actually designed it to survive, the next question isn't how to move it into the cloud.
It's what the cloud could do for us that another machine in the building can't.
That's a narrower question, and I think it produces a better architecture.
Ten servers in one building can provide considerable redundancy. A workload can move when a machine fails. Traffic can enter through more than one node. Storage can survive a disk or server disappearing. Maintenance can happen without shutting everything down.
Then the building loses power.
Or the network connection disappears.
Or something happens that makes the entire site unavailable.
Adding an eleventh server to the same room doesn't address that failure.
Now another location becomes interesting.
It could be another office, a colocated server or infrastructure rented from a cloud provider. The important distinction initially isn't who owns the machine. It's that the machine doesn't share all of the same failure boundaries as the local cluster.
For this architecture, I'll call that remote environment the cloud.
I don't want to stretch the existing Kubernetes cluster into it.
A Kubernetes control plane works best when the machines participating in it have the network characteristics the control plane expects. Extending one cluster across geographically separated networks introduces latency, partitions and dependencies that aren't necessary for what I'm trying to accomplish.
I'd rather have two clusters.
The local cluster remains local.
The remote cluster remains remote.
They can run many of the same applications and use much of the same configuration, but neither needs the other in order to operate normally.
That immediately changes what hybrid means.
There are no cloud worker nodes waiting for the local Kubernetes scheduler to discover them. The scheduler in the local cluster knows only about local nodes. The scheduler in the remote cluster knows only about remote nodes.
If work moves between those environments, something above the individual schedulers has to cause that to happen.
Sometimes that means the same application is already deployed in both places.
Sometimes a deployment process increases replicas remotely.
Sometimes a queue allows workers in either environment to consume work.
Sometimes nothing moves at all until a human decides that a sufficiently large failure has occurred.
Those are different mechanisms because they're solving different problems.
Before implementing any of them, I'd make the remote cluster boring in exactly the same way we made the local cluster boring.
Install it.
Inspect what appeared.
Deploy something disposable.
Send traffic through it.
Reboot things.
Delete things.
Verify certificate issuance and renewal.
If Git is going to describe the desired application state, prove that a clean cluster can recover enough of that state from the repository to become useful again.
The cloud shouldn't receive an exemption from the tests we applied locally merely because someone else owns the hardware underneath it.
Then I would decide what belongs there.
A stateless web application is relatively easy.
If the application can run in either environment without depending on local state that isn't available remotely, we can deploy it to both clusters. The remote copy might normally receive no public traffic at all. It might receive a small amount of traffic continuously so we know it actually works. Or it might be activated only during maintenance or an incident.
Those are operational choices rather than properties of Kubernetes.
Traffic steering happens outside the clusters.
DNS can participate in that decision. So can an upstream proxy or another edge-routing mechanism. The appropriate choice depends on how quickly traffic needs to move, how health is determined, what dependencies remain behind each endpoint and how much additional infrastructure we're willing to operate.
I wo