Kubernetes Cost Optimization: Right-Size Your Cluster

Kubernetes makes it easy to deploy applications, but it also makes it easy to waste money. Over-provisioned pods, idle nodes, and missing autoscaling can quietly inflate your cloud bill. Studies consistently find that the average Kubernetes cluster uses only 30-50% of its provisioned resources.

This guide covers practical strategies to find and eliminate waste without compromising reliability.

Understanding Where Money Goes

Kubernetes costs come from three main sources:

1. Compute (nodes) — typically 60-70% of total cluster cost. You pay for every CPU core and GB of RAM on your nodes, whether your pods use them or not.
2. Storage (PVs) — persistent volumes that may be over-provisioned or orphaned.
3. Networking — cross-AZ traffic, load balancers, and NAT gateway charges.

The biggest savings come from compute optimization, so that is where we focus.

Step 1: Measure Current Usage

You cannot optimize what you do not measure. Start by comparing requested resources to actual usage.

Using kubectl top

# Pod resource usage
# kubectl top pods -n production
# NAME                       CPU(cores)   MEMORY(bytes)
# web-app-7f4d5b-abc12       15m          45Mi
# web-app-7f4d5b-def34       12m          42Mi
# worker-6c8d7c-ghi56        150m         256Mi

# Node resource usage
# kubectl top nodes
# NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# node-1            450m         22%    2100Mi          52%
# node-2            380m         19%    1800Mi          45%
# node-3            200m         10%    1200Mi          30%

If node CPU usage is consistently below 40%, you are over-provisioned.

Checking Requests vs Actual Usage

# Compare requests to actual usage for a namespace
# kubectl get pods -n production -o json | jq '
#   .items[] | {
#     name: .metadata.name,
#     cpu_request: .spec.containers[0].resources.requests.cpu,
#     memory_request: .spec.containers[0].resources.requests.memory
#   }'

A pod requesting 500m CPU but consistently using 50m is wasting 90% of its allocation. That wasted capacity blocks other pods from being scheduled, forcing you to add more nodes.

Prometheus Queries for Resource Analysis

If you run Prometheus, these queries reveal waste:

# CPU utilization ratio (actual / requested)
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m]))
  by (pod)
/
sum(kube_pod_container_resource_requests{resource="cpu", namespace="production"})
  by (pod)

# Memory utilization ratio
sum(container_memory_working_set_bytes{namespace="production"})
  by (pod)
/
sum(kube_pod_container_resource_requests{resource="memory", namespace="production"})
  by (pod)

# Total cluster cost waste estimate
1 - (
  sum(rate(container_cpu_usage_seconds_total[1h]))
  /
  sum(kube_node_status_capacity{resource="cpu"})
)

Step 2: Right-Size Pod Resources

Right-sizing means setting requests and limits based on observed usage with a safety margin.

The Formula

CPU request    = P95 CPU usage + 20% buffer
Memory request = P99 memory usage + 10% buffer
CPU limit      = P99 CPU usage * 2 (or no limit)
Memory limit   = P99 memory usage + 25% buffer

Before and After Example

# BEFORE: Over-provisioned (common default)
resources:
  requests:
    cpu: "1"
    memory: 1Gi
  limits:
    cpu: "2"
    memory: 2Gi
# Actual P95 usage: 80m CPU, 150Mi memory

# AFTER: Right-sized
resources:
  requests:
    cpu: 100m
    memory: 170Mi
  limits:
    cpu: 500m
    memory: 256Mi

This single pod went from reserving 1 CPU core and 1 GB to 0.1 cores and 170 MB. Multiply by 50 pods and you have freed enough capacity to potentially remove entire nodes.

Vertical Pod Autoscaler (VPA)

VPA automatically adjusts resource requests based on observed usage:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-app-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  updatePolicy:
    updateMode: "Auto"  # or "Off" for recommendations only
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: "2"
          memory: 2Gi

Start with updateMode: "Off" to see recommendations without changes:

# kubectl describe vpa web-app-vpa
# Recommendation:
#   Container Recommendations:
#     Container Name: web-app
#     Lower Bound:    cpu: 25m,  memory: 64Mi
#     Target:         cpu: 80m,  memory: 128Mi
#     Upper Bound:    cpu: 200m, memory: 256Mi

Use the “Target” as your new request and “Upper Bound” as a guide for limits.

Step 3: Horizontal Pod Autoscaler (HPA)

Scale pods based on demand instead of running a fixed number:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 20
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The behavior section prevents flapping: scale up quickly (100% increase per 30s) but scale down slowly (25% decrease per 60s with a 5-minute stabilization window).

Step 4: Cluster Autoscaler

The cluster autoscaler adds and removes nodes based on pending pods and utilization:

# For EKS (example Helm values)
# cluster-autoscaler:
#   autoDiscovery:
#     clusterName: my-cluster
#   extraArgs:
#     scale-down-utilization-threshold: "0.5"
#     scale-down-delay-after-add: "10m"
#     scale-down-unneeded-time: "5m"
#     skip-nodes-with-local-storage: "false"
#     balance-similar-node-groups: "true"
#     expander: "least-waste"

Key settings:

• scale-down-utilization-threshold — remove a node if its CPU+memory utilization is below this (default 0.5 = 50%).
• scale-down-delay-after-add — wait this long after adding a node before considering scale-down (prevents thrashing).
• expander: least-waste — when scaling up, pick the node group that results in the least wasted resources.

Karpenter (AWS Alternative)

Karpenter is a node provisioner that replaces the cluster autoscaler with faster, more efficient scaling:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values:
            - m5.large
            - m5.xlarge
            - m6i.large
            - m6i.xlarge
            - c5.large
            - c5.xlarge
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "100"
    memory: 200Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Karpenter provisions the right instance type for each workload and consolidates underutilized nodes automatically.

Step 5: Spot and Preemptible Instances

Spot instances cost 60-90% less than on-demand. Use them for stateless, fault-tolerant workloads:

# Node group for spot instances (EKS example)
# eksctl create nodegroup \
#   --cluster my-cluster \
#   --name spot-workers \
#   --instance-types m5.large,m5.xlarge,m6i.large \
#   --spot \
#   --nodes-min 0 \
#   --nodes-max 20

# Schedule workloads on spot nodes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-processor
spec:
  replicas: 5
  template:
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80
              preference:
                matchExpressions:
                  - key: karpenter.sh/capacity-type
                    operator: In
                    values: ["spot"]
      tolerations:
        - key: "spot"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      containers:
        - name: processor
          image: myregistry/batch-processor:v1
          resources:
            requests:
              cpu: 500m
              memory: 512Mi

Use preferredDuringSchedulingIgnoredDuringExecution so the workload runs on on-demand nodes if no spot capacity is available.

Good candidates for spot: CI/CD runners, batch jobs, dev/staging environments, stateless web servers with proper load balancing.

Bad candidates: databases, stateful services, single-replica deployments.

Step 6: Resource Quotas and LimitRanges

Prevent teams from over-provisioning by setting quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
    persistentvolumeclaims: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-alpha
spec:
  limits:
    - type: Container
      default:
        cpu: 200m
        memory: 256Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi
      min:
        cpu: 50m
        memory: 64Mi

LimitRange sets defaults for pods that do not specify resources, preventing pods from running without any limits.

Step 7: Clean Up Waste

Orphaned Resources

# Find PVCs not mounted by any pod
# kubectl get pvc -A -o json | jq -r '
#   .items[] |
#   select(.status.phase == "Bound") |
#   "\(.metadata.namespace)/\(.metadata.name)"'

# Find unused ConfigMaps and Secrets
# kubectl get configmaps -A --no-headers | wc -l
# kubectl get secrets -A --no-headers | wc -l

Idle Deployments

# Deployments with 0 CPU usage over the last hour
# This requires Prometheus
# sum(rate(container_cpu_usage_seconds_total[1h])) by (namespace, pod) == 0

Dev and Staging Environments

Scale non-production environments to zero outside business hours:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-dev
  namespace: dev
spec:
  schedule: "0 20 * * 1-5"  # 8 PM weekdays
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: scaler
          containers:
            - name: scaler
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  kubectl scale deployment --all -n dev --replicas=0
          restartPolicy: OnFailure
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-up-dev
  namespace: dev
spec:
  schedule: "0 8 * * 1-5"  # 8 AM weekdays
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: scaler
          containers:
            - name: scaler
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  kubectl scale deployment --all -n dev --replicas=1
          restartPolicy: OnFailure

Step 8: Monitor Costs Continuously

Kubecost

Kubecost provides real-time cost allocation per namespace, deployment, and label:

# helm repo add kubecost https://kubecost.github.io/cost-analyzer/
# helm install kubecost kubecost/cost-analyzer \
#   --namespace kubecost \
#   --create-namespace

Custom Cost Dashboard

Track these metrics over time:

• CPU efficiency = actual CPU usage / total CPU capacity
• Memory efficiency = actual memory usage / total memory capacity
• Cost per namespace = (namespace CPU requests / total CPU) * total compute cost
• Idle node cost = nodes with utilization below threshold * node hourly cost

Set alerts for:

• Cluster CPU efficiency below 40%.
• Namespaces exceeding their budget.
• Nodes with less than 30% utilization for over 24 hours.

Quick Wins Checklist

1. Run kubectl top nodes — if any node is below 30% utilization, investigate.
2. Check the biggest pods — kubectl top pods --sort-by=memory -A | head -20.
3. Enable VPA in recommendation mode — apply its suggestions to the worst offenders.
4. Set HPA on stateless services — scale to demand instead of peak.
5. Enable cluster autoscaler — let nodes scale with demand.
6. Use spot instances for dev/staging — 60-90% savings.
7. Set LimitRange defaults — catch pods without resource specs.
8. Scale dev/staging to zero at night — save 60%+ on non-production.
9. Delete orphaned PVCs and unused load balancers — check monthly.
10. Review instance types — newer generations (m6i vs m5) often cost less per unit.

Wrapping Up

Kubernetes cost optimization is an ongoing practice, not a one-time project. The highest-impact strategies are:

• Right-size pods based on actual usage, not guesses. Use VPA recommendations.
• Autoscale horizontally with HPA so you run only what you need.
• Autoscale nodes with cluster autoscaler or Karpenter to eliminate idle capacity.
• Use spot instances for fault-tolerant workloads.
• Set quotas and defaults to prevent waste at the namespace level.
• Monitor continuously with tools like Kubecost or Prometheus dashboards.

Start with measurement. Run kubectl top on your nodes and pods today. The gap between what you pay for and what you use is your optimization opportunity.