Kubernetes Autoscaling skill for Claude Code

Autoscaling is done when every scaled workload has requests set from measured usage, scales on the signal that actually precedes saturation, adds capacity faster than load arrives (pods and nodes), removes it without dropping requests, and the policy has been replayed against a real load curve before production.

First, inspect

kubectl version                                       # server minor matters for feature states
kubectl get hpa,vpa,scaledobject,scaledjob,pdb -A 2>/dev/null
kubectl describe hpa <name> -n <ns>                   # conditions + events explain every decision
kubectl top pods -n <ns> --containers                 # needs metrics-server
kubectl get apiservice | grep -E 'metrics.k8s.io|custom.metrics|external.metrics'
kubectl get nodepools,nodeclaims 2>/dev/null || kubectl -n kube-system get deploy cluster-autoscaler

In kubectl describe hpa, read the conditions: AbleToScale, ScalingActive (false means metrics are missing), ScalingLimited (pinned at min or max). Events show each rescale and why.

Workflow

- [ ] 1. Measure: per-pod throughput at the latency SLO, startup time, node provisioning time
- [ ] 2. Set requests from observed usage (VPA in Off mode or metrics history)
- [ ] 3. Pick the scaling signal (table below) and the target per pod
- [ ] 4. Configure HPA or KEDA with min/max and behavior
- [ ] 5. Configure node autoscaling so pending pods get nodes fast enough
- [ ] 6. Add PDBs and graceful shutdown so scale-down is safe
- [ ] 7. Replay a load curve (scripts/hpa_simulate.py) and run a real load test
- [ ] 8. Alert on pinned-at-max, pending pods, and flapping

Prerequisite: requests

• HPA Utilization targets are a percentage of the container request. A pod whose containers lack the relevant request has undefined utilization, and the HPA takes no action for that metric.
• Set CPU requests near steady-state p90 usage and memory requests near peak working set (kubernetes-deploy covers sizing and limits in depth).
• Get recommendations without changing anything: a VPA with updateMode: "Off" and kubectl describe vpa <name> (target, lowerBound, upperBound).

Choose the signal

Workload	Scale on	Tool
CPU-bound HTTP/gRPC service	CPU utilization, target 60-75% of request	HPA Resource or ContainerResource
I/O-bound or async service where CPU stays low	Requests per second or in-flight requests per pod	HPA Pods/Object metric via an adapter, or KEDA Prometheus trigger
Queue consumers (SQS, Kafka, RabbitMQ, Pub/Sub, Redis streams)	Backlog per replica (lag / messages per pod)	KEDA ScaledObject
Run-to-completion work items	One Job per batch of messages	KEDA ScaledJob
Idle most of the day, must scale to zero	Event source with activation threshold	KEDA (minReplicaCount: 0)
Memory-bound with steady load	Usually right-size with VPA instead of scaling out	VPA
Stateful single-writer	Do not autoscale horizontally	VPA recommendations, manual

Memory is a poor HPA signal for most runtimes: usage does not fall when load does (JVM heaps, caches), so replicas ratchet up and never come down.

HPA (autoscaling/v2)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api, namespace: shop }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3                       # survive one zone loss and absorb the first spike
  maxReplicas: 40                      # bounded by DB connections and budget
  metrics:
    - type: ContainerResource          # ignore sidecar CPU
      containerResource:
        name: cpu
        container: app
        target: { type: Utilization, averageUtilization: 65 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - { type: Percent, value: 100, periodSeconds: 15 }
        - { type: Pods, value: 4, periodSeconds: 15 }
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - { type: Percent, value: 20, periodSeconds: 60 }

How it decides:

• desiredReplicas = ceil(currentReplicas * currentMetric / targetMetric), evaluated every 15 s (--horizontal-pod-autoscaler-sync-period).
• No change while the ratio is within the tolerance (default 0.1, so 59-71% does nothing at a 65% target). Per-HPA tolerance under behavior.scaleUp / behavior.scaleDown is stable as of Kubernetes 1.37 (introduced 1.33 behind HPAConfigurableTolerance); on older clusters it may be unavailable.
• With several metrics, the HPA computes a count per metric and uses the largest.
• Not-yet-ready pods are set aside; during scale-up they are assumed to use 0%, which dampens the step. Slow readiness therefore slows scaling; fix startup time first.
• Scale-down uses the highest recommendation within the stabilization window (default 300 s). Scale-up has no window by default and is limited to the larger of +100% or +4 pods per 15 s.

Behavior tuning:

Symptom	Change
Too slow to catch spikes	Lower target (more headroom), raise minReplicas, add a Pods policy allowing bigger steps; cut pod startup time
Flapping up and down	Longer scaleDown.stabilizationWindowSeconds; slower scale-down policy (for example 20% per 60 s); check the metric is not noisy
Never scales down	Metric does not fall with load (memory), or minReplicas too high, or a stuck high recommendation from a noisy metric
Scale-down drops requests	Graceful shutdown: preStop sleep plus terminationGracePeriodSeconds longer than the longest request
Never scale down during an incident	behavior.scaleDown.selectPolicy: Disabled temporarily

Rules:

• Remove spec.replicas from the Deployment manifest once an HPA owns it, or every kubectl apply resets the count. Removing it is a one-time scale to 1 unless done carefully; apply when the HPA is already active and at low traffic.
• One autoscaler per workload: never an HPA and a KEDA ScaledObject on the same Deployment (KEDA creates its own HPA).
• maxReplicas must respect downstream limits: replicas x pool size must stay under the database connection budget.
• Custom and external metrics need an adapter. Only one service can serve external.metrics.k8s.io per cluster; KEDA and prometheus-adapter conflict there.

VPA

updateMode	Behavior	Use
Off	Recommendations only	Default starting point; feed requests in your manifests
Initial	Sets requests at pod creation only	Batch and jobs
InPlaceOrRecreate	Resizes running pods in place, evicts if it cannot (GA in VPA 1.6; needs Kubernetes 1.33+, in-place resize is GA in 1.35)	Long-running services that tolerate occasional restarts
InPlace	In-place only, never evicts (alpha, VPA 1.7)	Disruption-intolerant workloads, once stable
Recreate	Evicts pods to apply new requests	Rarely; respects PDBs
Auto	Deprecated alias of Recreate	Replace with an explicit mode

• Never run VPA and HPA on the same resource metric (CPU or memory) for one workload: VPA raises requests, utilization drops, HPA scales in, and they fight. Allowed combinations: HPA on a custom or external metric (RPS, queue lag) plus VPA on CPU and memory, or VPA in Off mode alongside any HPA.
• Set minAllowed and maxAllowed in resourcePolicy so a bad recommendation cannot request a whole node or starve the pod.
• VPA is not compatible with pod-level resources stanzas yet.

KEDA

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: order-worker, namespace: shop }
spec:
  scaleTargetRef: { name: order-worker }
  minReplicaCount: 0
  maxReplicaCount: 50
  pollingInterval: 30          # default 30 s
  cooldownPeriod: 300          # only governs the last replica -> 0
  fallback: { failureThreshold: 3, replicas: 5 }
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown: { stabilizationWindowSeconds: 120 }
  triggers:
    - type: aws-sqs-queue
      authenticationRef: { name: aws-workload-identity }
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/orders
        queueLength: "20"           # target messages per replica
        activationQueueLength: "1"  # wake from zero
        awsRegion: us-east-1

• KEDA handles 0 <-> 1 (activation); the HPA it creates handles 1 <-> N. cooldownPeriod does not slow normal scale-down; tune that through horizontalPodAutoscalerConfig.
• Target per replica = what one replica drains in your latency budget. If a pod handles 10 messages/s and backlog must clear within 60 s, queueLength near 600 is the ceiling, lower for headroom.
• Always set fallback, or a broken metric source (expired credentials, Prometheus down) leaves the workload at whatever it had.
• Long jobs that must not be killed mid-item: use ScaledJob, or set terminationGracePeriodSeconds above the longest item and make work idempotent.
• Scale to zero means a cold start on the first event; measure it and confirm the latency is acceptable before enabling for user-facing paths.
• Native HPA minReplicas: 0 (HPAScaleToZero, beta and on by default in 1.37) works only with Object or External metrics; KEDA remains the portable choice.

Node autoscaling

Pods scale in seconds; nodes take minutes. Total reaction time = metric delay + HPA sync + pod scheduling + node provisioning + image pull + app startup.

Situation	Choose
EKS, or a cloud with a Karpenter provider (AWS, Azure)	Karpenter: provisions right-sized nodes per pending pod, consolidates underused nodes
GKE	Built-in cluster autoscaler or node auto-provisioning
Other clouds, fixed node group shapes	Cluster Autoscaler with node groups

Karpenter (v1 API):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: general }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: ["spot", "on-demand"] }
        - { key: kubernetes.io/arch, operator: In, values: ["arm64", "amd64"] }
        - { key: karpenter.k8s.aws/instance-category, operator: In, values: ["c", "m", "r"] }
      expireAfter: 720h
  limits: { cpu: "2000" }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 9 * * mon-fri"
        duration: 8h
        reasons: ["Underutilized"]

• consolidateAfter defaults to 0s; a few minutes avoids churn after every deploy. Recent releases also offer consolidationPolicy: Balanced to skip low-value moves.
• Default budget is nodes: 10%. Block consolidation during business hours with a scheduled budget if churn hurts.
• Protect a pod from voluntary disruption with the annotation karpenter.sh/do-not-disrupt: "true" (or a duration such as "30m").
• Set limits on every NodePool; without them a runaway HPA buys nodes without bound.

Cluster Autoscaler: scale-down after --scale-down-unneeded-time (default 10m) below --scale-down-utilization-threshold (default 0.5 of requests); use --balance-similar-node-groups across zones and an --expander (least-waste or priority). Pods with cluster-autoscaler.kubernetes.io/safe-to-evict: "false" block their node’s removal.

For instant headroom, run a low-priority placeholder Deployment (negative PriorityClass) that real pods preempt while new nodes come up.

PodDisruptionBudgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api, namespace: shop }
spec:
  maxUnavailable: 1                    # or a percentage; scales with replica count
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector: { matchLabels: { app: api } }

• Prefer maxUnavailable over minAvailable; minAvailable equal to the replica count (or 100%) blocks every node drain and consolidation forever.
• unhealthyPodEvictionPolicy: AlwaysAllow lets drains evict crash-looping pods that would otherwise block them.
• Single-replica workloads with a PDB of maxUnavailable: 0 block node scale-down; run two replicas or accept the disruption.
• PDBs cover voluntary evictions only; they do not protect against node failure.

Test the policy

1. Replay a production load curve through the HPA math before touching the cluster:

# load.csv: t_seconds,load  (total CPU millicores, RPS, or queue rate across all pods)
python3 scripts/hpa_simulate.py load.csv --target 650 --min 3 --max 40 \
  --ready-delay 45 --capacity 900
python3 scripts/hpa_simulate.py load.csv --target 650 --min 3 --max 40 \
  --ready-delay 45 --down-window 600 --down-percent 20 --down-period 60 --capacity 900

It prints each rescale, peak replicas, flaps, pod-hours (cost proxy), and seconds where per-pod load exceeded the target and the measured --capacity (exit 1 if saturated). Add node provisioning time to --ready-delay when nodes must be created. Compare policies on the same curve; pick the one with zero saturated seconds at the lowest pod-hours.

2. Run a real ramp test (k6, Locust, or Vegeta) in staging at production shape: a step from baseline to 3-5x within one minute, a hold, and a drop. Watch:

kubectl get hpa api -n shop -w
kubectl get pods -n shop -l app=api -w
kubectl get nodeclaims -w                  # Karpenter

Pass criteria: p95 latency stays within the SLO during the ramp, no pod stays Pending longer than the measured node provisioning time, scale-down completes without 5xx, and replicas do not oscillate after the drop.

Alerts

• HPA at maxReplicas for more than 15 minutes (ScalingLimited true) while the metric is above target.
• Pods Pending longer than expected node provisioning time.
• More than N rescale events per hour (flapping).
• KEDA scaler errors or fallback active.

Gotchas

• metrics-server reports CPU averaged over its scrape window; very short spikes never reach the HPA. Scale on a leading signal (RPS, queue length) for bursty traffic.
• Init containers and sidecars count toward Pod-level Resource metrics; use ContainerResource for the app container.
• CPU limits cause throttling that looks like low utilization at high latency; the HPA then under-scales. Prefer no CPU limit (or a generous one) for latency-sensitive pods.
• JVM and other slow-starting apps: use a startupProbe and keep readiness honest, or the HPA counts cold pods as capacity.
• Changing maxReplicas or node limits in an incident without checking the database connection budget moves the outage downstream.
• Spot interruptions are involuntary disruptions; spread across instance types and zones and keep enough on-demand baseline for minReplicas.

Output

**Workload:** shop/api (Deployment), HPA on container CPU 65%, min 3 / max 40
**Behavior:** up +100% or +4 pods per 15 s; down 20% per 60 s after a 300 s window
**Nodes:** Karpenter NodePool general (spot + on-demand, c/m/r, arm64+amd64), consolidateAfter 2m, limit 2000 CPU
**Safety:** PDB maxUnavailable 1 (AlwaysAllow), preStop 10 s, grace 40 s
**Simulation:** Black Friday curve: 0 s saturated, peak 31 replicas, 182 pod-hours (vs 214 with defaults)
**Load test:** 4x step in 60 s, p95 180 ms (SLO 300 ms), max Pending 70 s, no 5xx on scale-down