Your web app is a Deployment. Your databases are StatefulSets. But what about the work that runs once and exits — a database migration, a nightly report, a data backfill, a thumbnail render? If you wrap those in a Deployment, the pod restarts forever because the container exited. That's what Jobs are for: run to completion, then stop. And when the work needs to happen on a schedule, you wrap the Job in a CronJob.
Jobs look simple. They're not. Get the retry policy wrong and you get duplicate data. Get the parallelism wrong and your cluster starves. Get the history limits wrong and your cluster fills with dead pods. Here's how to run batch workloads in Kubernetes without any of that.
1. What Is a Job?
A Job creates one or more pods and ensures a specified number of them complete successfully. When a pod exits with code 0, that work item is done. Unlike a Deployment, a Job does not keep pods running — it lets them finish and records the result.
apiVersion: batch/v1
kind: Job
metadata:
name: pi
spec:
template:
spec:
containers:
- name: pi
image: perl:5.34
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4
The one rule that trips everyone up: restartPolicy must be Never or OnFailure. Jobs cannot use Always — that's the whole point. A container that exits is not a crash to heal, it's work that finished (or failed).
# Watch the job and its pods
kubectl get jobs
kubectl get pods -l job-name=pi
kubectl logs pi-xxxxx
2. Retries, Timeouts, and Cleanup
Three fields control what happens when things go wrong — and they're the difference between a self-healing job and a stuck one:
backoffLimit— how many times the Job retries failed pods before being markedFailed(default 6). The retry backoff is exponential: 10s, 20s, 40s, capped at 6 minutes. Set this low for non-idempotent work.activeDeadlineSeconds— a hard cap on the Job's total runtime. When it expires, running pods are terminated and the Job is markedFailed. This is your runaway-job insurance.ttlSecondsAfterFinished— automatically deletes finished Jobs (stable since 1.21). Without it, completed Jobs and their pods accumulate forever.0deletes immediately;86400keeps it around for a day so you can inspect it.
A production database migration with all three set:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: myapp:migrate-1.2.0
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-credentials
key: url
Backoff limited to 2 retries (a migration that fails three times will likely keep failing), hard-stopped after 10 minutes, and cleaned up a day after it finishes.
3. Parallel Jobs: Two Patterns
Sometimes one pod isn't enough. Jobs support two parallel models, and they solve different problems:
Fixed completion count — set completions: 10 and parallelism: 3. The Job runs exactly 10 successful pods, up to 3 at a time. Use this when you know the work items in advance — render 10 reports, process 10 files.
spec:
completions: 10
parallelism: 3
template:
spec:
restartPolicy: Never
containers:
- name: render
image: myapp:renderer
command: ["render-report"]
Work queue — set parallelism: 5 with no completions. Each pod is a worker that pulls items from a queue (Redis, SQS, Kafka) until the queue is empty. A pod exiting 0 signals "no more work"; the Job finishes when every worker exits. Use this when you don't know the workload size in advance.
Pitfall: with a work queue, a crashed worker re-runs and re-pulls its item. If your consumer isn't idempotent, the retried pod duplicates the work. Design queue consumers to tolerate re-delivery — this is exactly the same guarantee model as SQS/Kafka at-least-once delivery.
4. CronJobs: Scheduled Jobs
A CronJob is a thin wrapper that creates a Job on a schedule (added in 1.8, stable in 1.21). A nightly backup:
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-backup
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: backup
image: postgres:15
command: ["pg_dump", "-Fc", "-f", "/backups/dump.gz"]
Key decisions in that YAML:
concurrencyPolicy—Allow(default) lets runs overlap,Forbidskips a run if the previous one is still going,Replacekills the running Job and starts a new one. For backups:Forbid. Two overlappingpg_dumpruns are chaos.startingDeadlineSeconds— if the controller was down (cluster upgrade, controller crash), missed runs older than this window are skipped, not queued. The controller will not fire 50 missed backups after a weekend outage — which is what you want for idempotent jobs. Just know that missed runs don't get retroactively executed.successfulJobsHistoryLimit/failedJobsHistoryLimit— how many finished Jobs to keep around. The defaults are 3 and 1. Each CronJob run creates a Job, and every Job creates pods; unset limits mean yourkubectl get podsoutput becomes a graveyard.
Time zones: CronJobs run in the kube-controller-manager's local time zone, which is usually UTC on managed clusters. If your team thinks in Asia/Shanghai or America/New_York, set spec.timeZone explicitly (Kubernetes 1.27+). Never assume the schedule matches your wall clock.
5. Production Pitfalls
- Cron is not precise. A schedule fires when the controller notices it, plus pod scheduling latency — seconds to minutes of jitter. Don't build a job that must run at 09:00:00.000.
- Jobs are not idempotent by default. Your migration runs, fails at 90%, retries from scratch. If your SQL isn't safe to re-run, you get duplicate rows. Use versioned migration frameworks, dedupe keys, and make every batch job safe to run twice.
- Flaky image pulls burn retries. With
imagePullPolicy: Alwaysand a flaky registry,ImagePullBackOffcan exhaust yourbackoffLimitbefore the pod ever runs. It's not a code bug — but the Job still fails. - Parallelism without resource requests. A Job with
parallelism: 20on a small cluster leaves everything else Pending — including your web tier. Set resource requests on Job pods and watch node capacity. suspend: true(1.21+) pauses a CronJob without deleting it. Use it before maintenance windows instead of deleting and recreating the manifest.
6. Debugging Jobs
# Job-level status
kubectl get jobs
kubectl describe job db-migrate
# Pod-level detail
kubectl get pods -l job-name=db-migrate
kubectl logs db-migrate-xxxxx
# Why did it fail? Machine-readable conditions
kubectl get job db-migrate -o jsonpath='{.status.conditions}'
describe job shows the retry count and the reason a Job was marked failed. The pod events tell you whether it was a pull error, a crash, or a timeout. And if a Job is stuck in Active with zero pods, check the Job controller — the pods were probably evicted or never scheduled.
Summary
- Job — one-shot tasks that run to completion. Set
restartPolicy: Never. - Parallel work —
completionsfor known item counts, the queue pattern for unknown workloads. Make workers idempotent. - CronJob — scheduled Jobs. Set
concurrencyPolicy: Forbidfor anything that must not overlap, and cap the history limits. - Always — set
backoffLimit,activeDeadlineSeconds, andttlSecondsAfterFinished(or history limits). A Job without timeouts is a leak.
Batch workloads are where Kubernetes quietly pays for itself — no servers to babysit, no crontab drift, and retries that actually retry. Configure them like the examples above and they run themselves. That's the point.