Once a Kubernetes cluster can lose a worker node without turning the event into an emergency, it's tempting to call the cluster redundant.
Some of it may be.
That's a useful distinction to make before adding anything else.
Kubernetes can reschedule a workload after a node disappears, but that tells us only that another node is capable of running the workload. It doesn't necessarily tell us that users can still reach it or that the data it needs will be available when it arrives somewhere else.
Those are separate problems.
I like dealing with them separately because otherwise the word redundancy starts covering so many things that it stops telling us much.
First, consider ingress.
Suppose an ingress controller is running on one node. Kubernetes may be perfectly capable of recreating that controller elsewhere if the node fails, but traffic still needs some route to the replacement.
Running multiple ingress-controller replicas is a useful first step because it removes one process from the failure path. If possible, I want those replicas placed on different nodes so the loss of one machine doesn't remove every ingress instance at once.
Then I test it.
Send traffic through the cluster and remove one of the ingress pods.
If nothing interesting happens, remove the node hosting one of them.
Now we find out whether the rest of the network architecture agrees with our definition of redundant.
It may not.
Perhaps incoming traffic still depends on an address associated with the failed node. Perhaps an upstream router has nowhere else to send it. Perhaps Kubernetes has several healthy ingress instances that the outside world has no way to reach.
That's not a contradiction. We've simply reached the next failure boundary.
Now we need some mechanism outside the ingress pods themselves that can direct traffic toward a healthy node.
On a small on-premises network, that might be a virtual IP capable of moving between machines. Another environment might use a pair of load balancers, router integration, BGP announcements or some other mechanism appropriate to the network already in place.
The particular product matters less to me initially than being able to draw the path.
A request arrives at the public address.
Something determines where that request goes.
The selected node accepts it.
The ingress controller routes it toward the application.
Then I should be able to point at each step and ask what happens when it disappears.
This is where diagrams become considerably more useful than adjectives.
Instead of saying ingress is highly available, draw the request path and start crossing things out.
One ingress pod disappears.
One worker disappears.
A switch disappears.
The machine holding a virtual address disappears.
The Internet connection disappears.
At some point traffic stops.
That's fine.
The objective isn't to construct an infrastructure diagram on which every conceivable X can be crossed out without consequence. It's to know which failures we've designed around and which ones we haven't.
Storage introduces a different set of questions.
Early in a Kubernetes build, node-local storage can actually be useful because it makes state difficult to ignore. A pod running on one node writes to a disk on that node. Move the pod somewhere else and the relationship becomes obvious.
The new pod doesn't automatically bring the old disk with it.
Before solving that problem, I want to know whether it needs solving.
Some data is disposable. Caches can often be recreated. Temporary processing files may not matter after the job finishes. Container images can be pulled again. Derived artifacts may be cheaper to regenerate than to replicate continuously.
Other state needs a durability plan.
That doesn't necessarily mean every important workload should receive replicated Kubernetes storage. A database may implement replication at the database layer. User uploads might belong in object storage. Some datasets may be adequately