You've deployed your app to Kubernetes. The pod starts — then it gets killed. Or it's running but no traffic reaches it. Or it takes 90 seconds to initialize and gets restarted in a loop. Every one of these problems traces back to the same root cause: misconfigured or missing health probes.
Kubernetes gives you three types of probes: livenessProbe, readinessProbe, and startupProbe. Each serves a different purpose. Mix them up and your pods restart in infinite loops. Get them right and your deployments self-heal, scale correctly, and handle rolling updates without a single dropped request.
Here's what each probe does, when to use it, and how to configure it for a real production service.
1. Liveness Probe: Is the Container Alive?
The liveness probe answers one question: "Is this container still running correctly?" If the probe fails, kubelet kills the container and restarts it.
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3Use liveness probes for deadlock detection. If your app enters a state where it's alive but not making progress (a goroutine leak, a stuck mutex, an infinite loop), the liveness probe exposes that and triggers a restart.
The #1 mistake people make: using the liveness probe to check external dependencies like databases or upstream APIs. Don't do this. If your database is down and your liveness probe fails, Kubernetes will restart your pod — but the database is still down. Restarting the app doesn't help, and now you have a crash loop on top of a DB outage. That's worse.
Liveness probes should only check internal process health. Not database connectivity, not Redis, not upstream services.
2. Readiness Probe: Is the Container Ready for Traffic?
The readiness probe answers: "Should this pod receive traffic?" If it fails, the pod is removed from all Service endpoints. It is not restarted.
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
successThreshold: 1This is where you should check external dependencies. If your app needs a database connection or a warm cache to serve requests, the readiness probe should reflect that. When the database recovers, the probe starts passing again, and the pod is automatically re-added to the load balancer.
Readiness probes also control rolling update behavior. During a deployment, Kubernetes waits for the new pod's readiness probe to pass before terminating the old pod. Without a readiness probe, your deployment might kill the old pod before the new pod is actually ready — causing a brief outage.
3. Startup Probe: Slow Starters Need Love
The startup probe was added in Kubernetes 1.18. It answers: "Has the application finished initializing?"
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10This gives your app up to 300 seconds (30 × 10) to start. While the startup probe is running, the liveness and readiness probes are disabled. Once the startup probe succeeds, Kubernetes hands control back to the liveness and readiness probes.
Why does this matter? Consider a Java application with a 2-minute startup time. Without a startup probe, you have two choices:
- Set
initialDelaySeconds: 120— but now every restart waits 2 minutes before probing starts, even for fast restarts. - Set
failureThreshold: 12withperiodSeconds: 10— gives 120 seconds but makes the probe tolerates failures for 2 minutes, hiding real problems.
The startup probe solves this cleanly. Give it a generous threshold (30 failures × 10 seconds = 300 seconds). Keep your liveness probe tight (3 failures × 10 seconds = 30 seconds). The liveness probe is only activated after the app has fully started.
4. Probe Types: HTTP, TCP, and Command
All three probes support the same handler types:
| Type | When to Use |
|---|---|
httpGet | Your app has an HTTP endpoint. Best for web services and APIs. |
tcpSocket | Your app listens on a port but doesn't speak HTTP. Databases, message queues. |
exec | No HTTP server. Runs a command inside the container; exit code 0 = success. |
HTTP probes are the most expressive because you can return different status codes for different probe types:
// Go example: separate endpoints for each probe
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
// Internal: check process health only
w.WriteHeader(http.StatusOK)
})
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
if dbIsConnected {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(http.StatusServiceUnavailable)
}
})5. Putting It All Together
Here's a complete production-ready deployment with all three probes configured correctly:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-api
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
spec:
containers:
- name: api
image: my-api:latest
ports:
- containerPort: 8080
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 15
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
successThreshold: 1
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"Notice what this configuration achieves:
- Startup probe gives the app up to 5 minutes to initialize. Great for JVM apps, machine learning models, or anything with a warmup phase.
- Liveness probe kicks in after startup succeeds. It's tight — 3 failures × 15 seconds = 45 seconds to detect and restart a hung process.
- Readiness probe checks external dependencies every 5 seconds. If the database goes down, the pod is removed from the Service within 10 seconds. Traffic is re-routed to healthy pods.
- Rolling update uses
maxUnavailable: 0— never kill the old pod until the new one passes readiness. Zero-downtime deployments.
6. Common Pitfalls
Over the years, I've seen (and caused) these issues more than any others:
- Heavy readiness checks. If your /ready endpoint queries 10 database tables, every pod hitting it every 5 seconds creates unnecessary load. Keep readiness probes light — a simple ping or a connection pool check is enough.
- No startup probe on slow apps. Without it, your liveness probe starts during initialization, fails because the app isn't listening yet, and Kubernetes restarts the pod before it finishes starting. Classic crash loop.
- Same endpoint for all three probes. If /healthz checks the database (for readiness) but also triggers restarts (for liveness), a DB outage becomes a pod crash loop. Use separate endpoints.
- Too many replicas checking the same dependency. If Redis goes down and all 10 pods fail their readiness probe simultaneously, you get a thundering herd when Redis comes back. Add a small random jitter to your probe timing if this is a concern.
7. Debugging Probes
When probes aren't working, start here:
# Check probe status for a specific pod
kubectl describe pod my-api-7d4f8b9c6c-xk9j2
# Watch events in real time
kubectl get events --watch
# Check if the endpoint actually responds
kubectl exec my-api-7d4f8b9c6c-xk9j2 -- curl -v http://localhost:8080/healthz
# See why a pod was restarted
kubectl logs my-api-7d4f8b9c6c-xk9j2 --previousThe describe pod output shows probe results under the Conditions section. If Ready is False, the readiness probe is failing. If Containers shows a Restart Count above zero, the liveness probe triggered a restart. These two numbers tell you the whole story.
Summary
Three probes, three jobs:
- Startup probe — give slow apps time to boot. Disables other probes during initialization.
- Liveness probe — detect deadlocked or hung processes. Triggers restarts. Keep it simple, check only internal health.
- Readiness probe — control traffic routing. Check external dependencies here. Pods are removed from Services when this fails.
Configure all three in production. Use separate HTTP endpoints for liveness and readiness. Add a startup probe if your app takes more than 10 seconds to start. Once you nail this pattern, your deployments become boring — and boring deployments are the best kind.