Running Stateful Applications in Kubernetes (at Scale)

Managing stateful applications in Kubernetes introduces unique challenges and complexities compared to stateless workloads. From databases to messaging systems, running stateful apps requires considerations for storage, scalability, high availability, and data recovery. This guide breaks down everything you need to know about deploying and scaling stateful applications in Kubernetes effectively.

Table of Contents

1. StatefulSets vs Deployments
2. PersistentVolume Claims and Storage Classes
3. Using Operators for Databases (Postgres, MongoDB)
4. Backup, Restore, and Failover Strategies
5. Final Thoughts

StatefulSets vs Deployments

What are StatefulSets?

StatefulSets are Kubernetes resources used to manage the deployment and scaling of stateful applications. They maintain persistent state information for each Pod, ensuring stable identities even during restarts.

Key Features of StatefulSets:

• Stable Network Identity: Pods receive a predictable identity (e.g., pod-0, pod-1).
• Ordered Deployment and Scaling: Pods are created, updated, and terminated in a specific sequence.
• Persistent Storage: Each Pod gets a dedicated PersistentVolume Claim (PVC), ensuring data independence.

Use Case: StatefulSets are ideal for workloads like databases, distributed caches, or any application that requires stable identities and persistent storage.

Example YAML for a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

Why Not Use Deployments?

Deployments are typically used for stateless applications. With Deployments:

• Network identity (pod-name) changes upon recreation.
• Pods share storage or are ephemeral.
• Ordering and unique identification are not provided.

Using Deployments for stateful applications might result in data loss, improper scaling, or inconsistent setups.

Comparison Table:

PersistentVolume Claims and Storage Classes

Kubernetes separates storage management from applications via PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs).

PersistentVolume (PV) and PersistentVolumeClaim (PVC)

1. PersistentVolume: A resource representing a provisioned piece of storage.
   Example: apiVersion: v1 kind: PersistentVolume metadata: name: pv-example spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: /data
2. PersistentVolumeClaim: Applications use PVCs to request specific storage configurations.
   Example: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-example spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

Storage Classes for Dynamic Provisioning

StorageClasses simplify storage management by enabling dynamic provisioning, letting Kubernetes create storage automatically when a PVC is defined.

Example of a Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

Bind a PVC to this StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fast-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-storage

Best Practice:

• Use ReadWriteOnce (RWO) for databases to ensure single-writer access.
• Configure reclaim policies (Retain) to prevent accidental data loss when Pods are deleted.

Using Operators for Databases (Postgres, MongoDB)

What Are Operators?

Kubernetes Operators extend the cluster’s capabilities by automating the management of custom application lifecycles. For stateful applications like databases, Operators handle tasks such as scaling, backups, and replica failover programmatically.

Postgres Operator Example

The CrunchyData Postgres Operator simplifies PostgreSQL cluster management.

1. Install the Operator: kubectl apply -f https://github.com/CrunchyData/postgres-operator/releases/latest/download/postgres-operator.yaml
2. Define a PostgreSQL Cluster: apiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: name: pg-cluster spec: instances: - replicas: 3 backups: pgbackrest: repos: - name: repo1

MongoDB Operator Example

MongoDB Community Operator helps manage MongoDB StatefulSets with added functionalities.

1. Deploy the Operator: kubectl apply -f https://github.com/mongodb/mongodb-kubernetes-operator/releases/latest/download/mongodb-kubernetes-operator.yaml
2. Create a MongoDB Replica Set: apiVersion: mongodb.com/v1 kind: MongoDB metadata: name: mongodb-replicaset spec: members: 3 version: "4.4.6" type: ReplicaSet

Why Use Operators?

• Simplifies day-2 operations (backups, upgrades).
• Designed for high-availability databases.
• Ensures consistency with best practices.

Backup, Restore, and Failover Strategies

Stateful apps rely on robust backup, restore, and failover mechanisms.

Backups

1. Scheduled Backups: Use CronJobs to automate backups: apiVersion: batch/v1 kind: CronJob metadata: name: db-backup spec: schedule: "0 3 * * *" # Daily at 3 AM jobTemplate: spec: template: spec: containers: - name: backup image: postgres:14 args: - "pg_dump -U postgres -h postgres > /backups/db.sql" restartPolicy: OnFailure
2. Volume Snapshots: Use Kubernetes VolumeSnapshot for consistent point-in-time backups of PVCs: apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: pg-snapshot spec: volumeSnapshotClassName: fast-snapshot source: persistentVolumeClaimName: pg-pvc

Restores

Restore from backups by reattaching PVC snapshots:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-pvc
spec:
  dataSource:
    name: pg-snapshot
    kind: VolumeSnapshot
  storageClassName: fast-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

High Availability and Failover

1. Database Replicas: Use database-specific replication for read/write failover.
   • Postgres uses primary-replica streaming.
   • MongoDB handles primary failover via elections.
2. Service Failover: Point traffic to healthy nodes using Kubernetes Services: apiVersion: v1 kind: Service metadata: name: db-service spec: selector: role: primary # Redirect traffic to primary node
3. Health Checks: Define liveness and readiness probes for Pods: livenessProbe: exec: command: - psql - -c - "SELECT 1" initialDelaySeconds: 10 periodSeconds: 10

Best Practice: Combine a database Operator with advanced failover policies to maintain availability during node failures.

Final Thoughts

Running stateful applications at scale in Kubernetes involves careful orchestration of storage, scaling, and resilience strategies. Leveraging StatefulSets, dynamic provisioning with StorageClasses, and tools like Operators for specialized workflows enables seamless management of complex applications like databases. Furthermore, robust backup and failover strategies ensure uninterrupted service even under failure scenarios.

Mastering these techniques is key to running scalable, reliable stateful workloads in Kubernetes environments. Start small, iterate on your architecture, and unlock the full potential of containerized stateful apps!