Your pod is in CrashLoopBackOff. Or ImagePullBackOff. Or it's stuck in Pending for an hour. The deployment was working yesterday — nothing changed. Sound familiar?
Kubernetes failures all look the same at first glance: a red status, a cryptic message, and a ticket in your queue. But almost every problem falls into a small set of categories, and each category has a fixed set of diagnostic commands. Learn those, and you stop guessing and start fixing.
Here's the troubleshooting workflow I use in production, ordered from cheapest to most expensive. Run these in order and you'll find the root cause in minutes, not hours.
1. The First Three Commands: Always
Before touching anything, get the lay of the land:
# Everything in the namespace, with status
kubectl get all -n myapp
# Pods with restarts and age
kubectl get pods -n myapp -o wide
# The full story on a single pod
kubectl describe pod my-api-7d4f8b9c6c-xk9j2 -n myappkubectl describe is the single most valuable command in Kubernetes debugging. It shows Events at the bottom — the actual reason things went wrong, written by the kubelet and the controller manager. "Failed to pull image", "Back-off restarting failed container", "0/3 nodes are available: 1 Insufficient memory". The event section is the answer key. Read it first.
Rule of thumb: if you can't tell what's wrong after running these three commands, you haven't read the Events section carefully enough.
2. CrashLoopBackOff: The Container Won't Stay Up
The pod starts, crashes, restarts, crashes. The most common cause is an application error, not a Kubernetes problem. Three things to check, in order:
# Current logs
kubectl logs my-api-7d4f8b9c6c-xk9j2 -n myapp
# Logs from the previous, crashed instance
kubectl logs my-api-7d4f8b9c6c-xk9j2 -n myapp --previous
# If there are multiple containers in the pod
kubectl logs my-api-7d4f8b9c6c-xk9j2 -n myapp -c sidecar--previous is the move. When a container crashes and restarts, the current logs are empty or show only the new attempt. The crash output — the stack trace, the panic, the FATAL line — is in the previous instance's logs.
Common crash causes I see weekly:
- Missing environment variables — the app reads
DATABASE_URL, it's not set, it panics. Check your ConfigMaps and Secrets. - Startup command fails — the container image expects an entrypoint that doesn't exist, or a volume mount isn't writable.
- Readiness/liveness probe too aggressive — the app takes 60 seconds to start, your liveness probe kills it at 30. Add a startupProbe (see our health probes guide).
- Out of memory — the kernel kills the process (OOMKilled). Check the status field in
kubectl get podoutput: it literally saysOOMKilled.
3. ImagePullBackOff: The Image Isn't There
Kubernetes can't pull your container image. The reasons are usually boring:
# See the exact pull error
kubectl describe pod my-api-7d4f8b9c6c-xk9j2 -n myapp | grep -A 5 EventsThen check the obvious suspects:
- Tag doesn't exist —
myapp:latestwas never pushed, or a typo in the tag. Verify withdocker manifest inspect myapp:tagor check the registry. - Private registry credentials — the cluster can't authenticate. Create a
docker-registrysecret and reference it in the pod spec:imagePullSecrets. This one gets everyone at least once. - Registry unreachable — network policy, firewall, or the registry is down. Test from a node if you can.
- Wrong repository path —
registry.example.com/appvsregistry.example.com/team/app. Sloppy copy-paste from a README.
4. Pending: The Pod Can't Be Scheduled
A pod stuck in Pending means the scheduler hasn't found a node for it. Again, describe tells you why:
kubectl describe pod my-api-7d4f8b9c6c-xk9j2 -n myapp
# Look for lines like:
# 0/3 nodes are available: 1 node(s) had untolerated taint,
# 2 Insufficient cpu. preemption: 0/3 nodes available.The classic causes:
- Resource requests exceed node capacity — you asked for 8 CPU but your biggest node has 4. Either shrink the requests or add nodes.
- Taints and tolerations — nodes are tainted (e.g.
node-role.kubernetes.io/control-plane) and your pod has no matching toleration. - Node selector / affinity mismatch — the pod demands a label no node has.
- PersistentVolumeClaim not bound — the pod needs a PVC that's stuck in
Pending. Checkkubectl get pvc; if the StorageClass can't provision, that's your real problem.
For the "which node can run this" question, the scheduler's own tool is the fastest answer:
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory5. Running But Broken: Connection and Config Issues
The pod is Running, but the app is failing. This is where you move from the pod layer to the app layer:
# Get an interactive shell inside the container
kubectl exec -it my-api-7d4f8b9c6c-xk9j2 -n myapp -- /bin/sh
# Port-forward to reach a service from your laptop
kubectl port-forward svc/my-api 8080:80 -n myapp
# Copy a file out for inspection
kubectl cp myapp/my-api-7d4f8b9c6c-xk9j2:/var/log/app.log ./app.log
# Stream all logs from a deployment
kubectl logs deploy/my-api -n myapp --tail=200 -fWhen the app is up but can't reach its dependencies, work the connectivity chain:
- Is DNS resolving? Inside the pod:
nslookup my-db.default.svc.cluster.localorgetent hosts my-db. If this fails, check CoreDNS:kubectl get pods -n kube-system -l k8s-app=kube-dnsandkubectl logs -n kube-system -l k8s-app=kube-dns --tail=50. - Is the Service pointing at the right pods?
kubectl get endpoints my-api— empty endpoints mean the selector doesn't match your pod labels. This is the #1 cause of "the service exists but nothing responds". - Is the NetworkPolicy blocking it? Check for NetworkPolicies in the namespace. A policy that allows only certain ingress can silently kill cross-namespace calls.
6. A Debugging Checklist for Incidents
When something breaks and you're under pressure, run this sequence top to bottom:
kubectl get events --sort-by=.lastTimestamp | tail -30
kubectl get pods -A | grep -v Running
kubectl describe pod <pod> | sed -n '/Events:/,$p'
kubectl logs <pod> --previous
kubectl get endpoints <service>
kubectl get nodesThat's it. Six commands cover 90% of Kubernetes incidents: events, non-running pods, the pod's story, crash logs, service endpoints, and node health. Everything else is application-specific investigation.
7. Quick Wins to Prevent Future Debugging
A few habits that turn these hour-long firefights into five-minute confirmations:
- Add a
startupProbeto every slow-starting app so liveness doesn't kill it during boot. - Set resource requests and limits on everything. Unbounded pods cause noisy-neighbor OOM kills that are miserable to diagnose.
- Log to stdout/stderr so
kubectl logsshows everything. Files in the container are invisible to the standard tools. - Tag images with real versions, never
latest. "It worked yesterday" becomes a solvable git problem instead of a registry mystery.
Summary
Kubernetes debugging is pattern matching. CrashLoopBackOff → logs. ImagePullBackOff → describe + registry. Pending → describe + node capacity. "Running but broken" → exec, port-forward, endpoints, DNS. Start with kubectl describe, read the Events, and work outward from the pod.
The next time a pod misbehaves, resist the urge to delete and recreate it. Recreating hides the evidence. Diagnose first — the Events section has already told you what's wrong, you just haven't read it yet.