Read any platform or infrastructure job description and you’ll find the same list: Kubernetes, Linux, networking, storage, virtualization. Sometimes “physical infrastructure” gets its own bullet. They read like five skills. Five interviews, five specialists, five Slack channels.
The fault you actually get paged for doesn’t respect that list. It starts in one layer, is observed in another, and is fixed in a third. The skill that matters is the one that isn’t on the list: breaking a problem that crosses those boundaries into small problems you can actually test, and not stopping at the first layer that looks guilty.
Why I think about this at all
For the last year I’ve owned self-managed Talos Kubernetes on Nutanix, on-premises, for a securities brokerage in Indonesia, with a GCP leg for the same client. About 100 applications, no managed control plane anywhere. I wrote about what the layer beneath contains. This post is about what it does to the way you debug.
On GKE, when something below the node misbehaves, you open a ticket. There’s a competent landlord downstairs and a support SLA. On private infrastructure there is nobody downstairs. The hypervisor is yours. The storage that backs the persistent volumes is yours. The switches the load balancers sit on are, in a practical sense, yours. When the control plane hiccups, the only escalation path is the one you built.
That changes the conversation from “whose layer is it?” to “which layer is telling the truth right now?”
The shape of a cross-layer fault
Take an illustrative one, the kind every platform sees. A pod can’t reach a service it could reach yesterday.
The Kubernetes layer says the pod is Running and the endpoints look healthy. The application team says nothing changed. The network policy looks right. So far every layer you’d list on a job description has cleared itself.
Walk down. Is the node’s DNS resolver answering, or is it forwarding to something that was rebuilt? Is the CNI’s route table on that node the same as on the node next to it? Did the hypervisor migrate that VM to a host with a different network profile? Did a storage latency spike make a sidecar time out, so the failure is a symptom of I/O and not of networking at all?
None of those questions belongs to the person who “does Kubernetes.” All of them are needed to answer a Kubernetes symptom. That’s the point.
How I actually work one
The method isn’t clever, and it isn’t specific to infrastructure. I break the problem into smaller problems, then validate each component that isn’t doing what it should. Solving a small problem is much easier than solving one large, complicated one, and every large problem can be broken down and distributed across small ones. The layers are just where infrastructure problems break naturally.
Three habits follow from that.
Break the symptom down before you explain it. “The service is unreachable” is one large problem stated in application vocabulary. I want it split: which connection, from which node, to which endpoint, failing how. Then each of those split again, one layer down. The splitting is most of the debugging; by the time the problem is small enough to test, the answer is usually obvious.
Validate each small problem, and say the result out loud. “The CNI is clean on this node” is a result. It closes one branch and moves the search to the next. The alternative, a vague sense that “it’s probably the network,” is how a team spends four hours in the wrong layer, because nobody is willing to say a component is clean and own being wrong. Small problems are easy to validate; that’s the whole reason to make them small.
Know which component owns which failure mode, before you need it. On the Nutanix platform, the Omni machine-class work I’m doing now, with tag-driven automated node enrolment, exists partly so that “a new node came up wrong” is one small problem with one owner and one place to look. The same instinct is behind alerting tuned to page a human only on actionable failure: an alert that doesn’t point at a component is a large problem handed to someone at 3am.
That last habit is the one I’d generalise. A platform where every failure mode is already a small, owned problem is a platform you can debug under pressure. One where “storage” and “network” are people rather than places is not.
The migration version of the same skill
The clearest case I have isn’t an outage. On an AWS migration programme for an urban and infrastructure consultancy in Singapore, we had legacy applications whose dependencies no one could document. The config files were incomplete and the people who knew had moved on.
So we didn’t ask. We reverse-engineered the dependencies from live network traffic, built the map from what actually talked to what, and migrated by staged traffic shifting: 10%, 25%, 75%, then 100%, validating against the map at each step with a rollback path open until the last shift.
That’s a cross-layer problem wearing a project-plan costume. One large unknown, “what depends on what”, broken into hundreds of small, observable facts, each validated with traffic before the next cutover step depended on it. The application layer had a question it couldn’t answer; the answer lived two layers down in the packets.
What people get wrong when hiring for this
They hire for depth in one layer and assume the seams belong to someone else. You end up with a strong Kubernetes engineer, a strong network engineer and a fault that lives between them, with each holding a perfectly valid “it’s not my layer.”
The tell in an interview is not vocabulary. It’s the order of questions. Give a candidate one symptom and ask them to break it down out loud. The ones you want will split it into smaller problems, name what they’d check for each, what a clean result would rule out, and when they’d stop and go back up. They will also, at some point, say “I don’t know that layer well enough, so here’s how I’d find out.” That sentence is worth more than any certification on the list.
If you’re building a platform on infrastructure you own, from the metal up, the bullet you actually need on the job description is the one that’s never there: can follow a fault across every layer without asking whose it is.