Your database pod restarts. The data is gone. You scale a StatefulSet down and the volume vanishes with the pod. This is the moment every DevOps engineer discovers the hard truth about Kubernetes: container filesystems are ephemeral. When a pod dies, its writable layer dies with it.
For stateless services that's fine — that's what ReplicaSets are for. But databases, message queues, caches, and file uploads need storage that survives pod restarts, rescheduling, and node failures. That's what PersistentVolumes, PersistentVolumeClaims, and StorageClasses are for. Here's how they work together, with YAML you can actually use.
1. The Three-Layer Model
Kubernetes separates what you need from what's available:
- PersistentVolume (PV) — a piece of actual storage in the cluster, provisioned by an admin or automatically by the system. Think of it as a disk that exists independently of any pod.
- PersistentVolumeClaim (PVC) — a request for storage. Your pod declares "I need 10Gi, read-write, from the SSD class" and the claim binds to a matching PV.
- StorageClass — the recipe for creating storage on demand. It maps to your cloud provider's volume types (EBS gp3, GCE PD, Azure Disk) and defines reclaim policies and performance tiers.
Think of PV as the physical disk, PVC as the lease agreement, and StorageClass as the disk factory.
2. Static Provisioning: Admin Creates the Volume
The old-school way. An admin provisions a PV pointing at real storage:
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-data-pv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: manual
hostPath:
path: /mnt/data/postgres
Then an app claims it:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: manual
A claim binds to a PV when the requested size, access mode, and StorageClass all match. Once bound, the pod mounts it:
spec:
containers:
- name: postgres
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-data
Static provisioning works, but it's tedious — every volume is hand-made, sized in advance, and usually wasted or undersized. In production you almost never do this. You use dynamic provisioning.
3. Dynamic Provisioning: StorageClass Does the Work
A StorageClass tells Kubernetes how to create volumes on demand. On AWS it looks like this:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "3000"
encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Now your claim just references the class and Kubernetes provisions the volume automatically — no admin involved:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 10Gi
kubectl get pvc shows the claim go from Pending to Bound in seconds. The volume is created, attached to a node, and mounted — all without a human touching the cloud console.
4. Access Modes: One Writer or Many?
The accessModes field decides who can mount the volume and how:
- ReadWriteOnce (RWO) — one node can mount it read-write. Default for most block storage (EBS, Azure Disk).
- ReadOnlyMany (ROX) — many nodes can mount it read-only. Good for shared config or data files.
- ReadWriteMany (RWX) — many nodes can mount read-write. Only works with file/object storage: NFS, CephFS, EFS, S3-backed CSI drivers.
This is the #1 source of storage surprises. You deploy a multi-replica app with an RWO volume and half your replicas sit in ContainerCreating. They can't mount because the volume is already attached to another node. If you need multiple replicas sharing storage, you need a RWX-capable class, not block storage.
5. StatefulSets: Storage as Part of the Pod's Identity
StatefulSets pair each replica with its own volume using volumeClaimTemplates. When the pod is rescheduled, it reconnects to its volume — this is what makes stateful workloads (PostgreSQL, Cassandra, Kafka) safe to run on Kubernetes:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: mongo
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongo
image: mongo:7
volumeMounts:
- name: data
mountPath: /data/db
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 50Gi
Each replica gets a volume named data-mongo-0, data-mongo-1, data-mongo-2. Kill pod 1, it comes back on any node and reattaches to data-mongo-1. The data follows the identity, not the node.
6. Reclaim Policies: What Happens When You Delete the Claim?
Every PV has a persistentVolumeReclaimPolicy, set by its StorageClass:
- Delete (default for most cloud classes) — deleting the PVC deletes the volume. This means your data is gone. Back up before you delete anything.
- Retain — the PV stays around, unbound, so an admin can recover the data manually. Safe but requires cleanup.
- Recycle — deprecated. Don't use it.
Never delete a PVC holding production data without a verified backup. "It's just a claim, I'll recreate it" is how databases get lost.
7. Production Pitfalls
- Termination grace periods don't protect stateful data. A node failure can leave volumes in a broken state — enable volume snapshots via CSI and schedule them.
- Don't use
hostPathin production. It ties data to a specific node and bypasses the CSI lifecycle. It's fine for single-node dev clusters and nothing else. - Back up at the storage layer, not the pod layer.
kubectl exec ... pg_dumpis not a backup strategy. Use CSI snapshots (VolumeSnapshotCRDs) or provider-native backups. - Check
volumeBindingMode.WaitForFirstConsumerdelays provisioning until a pod actually needs the volume — avoids creating orphaned volumes for unused claims and makes topology-aware scheduling work on multi-AZ clusters. - Monitor PVC usage. Disks fill up silently. Watch
kubelet_volume_stats_used_bytesin Prometheus and alert before 85%.
Summary
- PV — the storage itself. Exists independent of pods.
- PVC — the request that binds to a PV. Pods mount claims, never PVs directly.
- StorageClass — dynamic provisioning. Production standard; static PVs are the exception.
- Access modes matter — RWO for single-writer apps, RWX for shared access. Pick the class to match.
- Reclaim policy Delete means data loss when you delete the claim. Snapshot first.
Storage is where Kubernetes stops being a convenient orchestration layer and starts being a stateful platform. Get the PV/PVC/StorageClass model right and your databases run as smoothly as your stateless APIs — and survive a node reboot without losing a byte.