Storage tiers · Prasanta Kumar Behera

Storage for EKS workloads by temperature: EBS (gp3/io2) for hot data, EFS for shared files, S3 (with Mountpoint) for cold and read-heavy data, installed as add-ons with least-privilege identities and snapshot support.

Pick storage by temperature

Temperature	Data	EKS storage
Hot	Databases, queues, latency-sensitive	EBS gp3 (tune IOPS/throughput), io2 for demanding IOPS
Warm	Shared content, uploads, CMS files	EFS (with lifecycle to Infrequent Access)
Cold	Archives, backups, data lakes, ML datasets	S3 (lifecycle to Glacier tiers); Mountpoint for S3 for read-heavy access

Keep things where they fit their use. The fridge (EBS) is for food you eat today: fast and close, but only one kitchen can open it. The shared pantry (EFS) is for things the whole house uses. The garage (S3) is huge and cheap for things you rarely touch, like old toys and photo albums.

EBS: the default for stateful pods

Install the EBS CSI driver add-on and give its controller a role (AWS publishes a managed policy for it) through Pod Identity. Then:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  type: gp3
  encrypted: "true"          # optionally kmsKeyId: <customer managed key ARN>
  iops: "3000"
  throughput: "125"

A second class (e.g. gp3-fast with higher IOPS, or io2) gives databases more performance without changing the default. Snapshots use a VolumeSnapshotClass with driver: ebs.csi.aws.com (see Kubernetes Storage, lesson 04).

EFS: shared files

Create an EFS filesystem with mount targets in every AZ (in the node subnets, with a security group allowing NFS from nodes), install the EFS CSI driver, and use dynamic provisioning with access points:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-0123456789abcdef0
  directoryPerms: "700"

Each PVC gets its own access point (a directory with its own POSIX identity) on the shared filesystem. Expect higher latency than EBS; don't run databases on it.

S3: cold and read-heavy

Apps should usually talk to S3 directly via the SDK, with Pod Identity permissions: simplest and most scalable.
The Mountpoint for Amazon S3 CSI driver mounts a bucket as a filesystem for read-heavy workloads (ML training data, analytics). It's not a POSIX filesystem: writes are limited (sequential, new objects), so check the documentation before relying on it.
Lifecycle rules move data to cheaper tiers automatically.

Encryption and backups

Encrypt EBS by default (class parameter plus account-level default encryption); consider customer managed KMS keys for sensitive data.
EBS snapshots via VolumeSnapshots or Velero (with snapshot data moved to S3 for cross-region DR); AWS Backup can also protect EBS and EFS by tag.

Try it: three tiers on one cluster (sandbox account)

Install the EBS CSI add-on with a Pod Identity role and the gp3 class above; run a StatefulSet with 3 replicas and check each PVC's AZ matches its pod's node.
Create a gp3-fast class (6000 IOPS) and use it for one PVC; run the fio random-write test from Kubernetes Storage, lesson 05, on both classes.
Create an EFS filesystem (mount targets in each AZ) and the efs class; mount one RWX PVC from pods on different nodes and share a file.
Give a pod read access to an S3 prefix via Pod Identity and list objects from it.
Take a VolumeSnapshot of a gp3 volume and restore it into a new PVC.

Going deeper: EKS storage operations

Watch EBS attach limits per instance type: Karpenter and the scheduler respect them, but dense nodes can hit them.
Plan AZ failure: EBS-backed pods can't move to another AZ without restoring from snapshots. Replicate at the application level (database replicas in 3 AZs) for real resilience.
Tag volumes with namespace/PVC/team (the EBS CSI driver can add tags) for cost allocation.
Clean up Released/orphaned volumes and old snapshots; they quietly cost money.

Recap

Hot: EBS gp3/io2 with WaitForFirstConsumer, encryption and snapshots.
Warm/shared: EFS with the CSI driver and access points (RWX, multi-AZ).
Cold: S3, directly via the SDK or Mountpoint for read-heavy data.
Drivers get least-privilege roles via Pod Identity; back up with VolumeSnapshots/Velero/AWS Backup.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.