Design a Metrics-Driven Auto-Scaling

Hard45 min
1 / 30
System design answer

Design a Metrics-Driven Auto-Scaling

Hard45 min30 sections
understanding•11 min read

Problem Statement: Why CPU-Based Scaling Is Not Enough

Frames metrics-driven autoscaling as a control-loop platform over business and application signals, not a resource-utilization toggle.

Problem statement

Design a metrics-driven auto-scaling platform that reads application-level and business metrics — request queue length, user concurrency, p95 latency, Kafka consumer lag, checkout rate — from a metric aggregation layer, evaluates scaling policies per workload, and spins replicas up or down for containerized microservices. The platform must minimize both cost and latency more effectively than CPU-only triggers, while never thrashing, never amplifying outages, and never scaling a workload into a dependency it will break.

CPU-based autoscaling is a proxy strategy. It assumes CPU utilization correlates with user-perceived load. For many production workloads that assumption fails in both directions. A Java service waiting on a downstream database holds idle CPU while request latency climbs; a CPU-bound batch sidecar burns cores while no user is affected. Kubernetes' own documentation positions HorizontalPodAutoscaler v2 as supporting custom and external metrics precisely because memory and CPU are insufficient signals for most business services.

What makes this a platform problem

A single HPA object is not the design challenge. The challenge is a fleet-wide decision platform: thousands of services, each with different signals, different safe operating bands, different warm-up times, and different blast radii. The platform must answer four questions for every workload, every few seconds: (1) Is the signal fresh and trustworthy? (2) What does the policy say the desired replica count is? (3) Is executing that change safe given global guardrails and dependency capacity? (4) Did the change work, and can we prove it?

The four planes

  1. Metric plane: ingestion, aggregation, downsampling, freshness tracking, and signal validity for thousands of metric streams.
  2. Decision plane: policy evaluation, control-loop algorithms (target-tracking, step, watermark, predictive), hysteresis, and stabilization windows.
  3. Execution plane: applying replica changes to workload APIs with surge limits, ready-gates, rollback, and fencing tokens.
  4. Governance plane: guardrails, quotas, approvals for large changes, audit journal, and scale-event observability.

A strong answer keeps these planes separate. The metric plane may lag or lose samples without freezing decisions; the decision plane may degrade to hold-current without losing visibility; the execution plane may refuse an unsafe change without losing the decision record.

Public operating baseline

The category is operationally real. Kubernetes ships HPA v2 with custom and external metric APIs and a behavior field with stabilization windows; KEDA, a CNCF project, scales workloads from dozens of external sources including Kafka lag and Prometheus queries; Datadog open-sourced the Watermark Pod Autoscaler to add hysteresis that HPA lacked; AWS ships Predictive Scaling for EC2 Auto Scaling groups. These are public artifacts that shape the design space. All fleet-scale numbers in this answer are explicitly stated design assumptions unless labeled otherwise.

Key Highlights

  • •CPU is a proxy signal; queue depth, latency, concurrency, and business KPIs are closer to user-perceived load.
  • •The platform is a control loop per workload: sense, decide, act, observe — with a global guardrail layer above it.
  • •Four planes: metric ingestion, decision, execution, governance. Each degrades independently.
  • •Freshness and trust of a signal are preconditions; a stale metric must veto scaling, not trigger it.
  • •Hysteresis and asymmetric response — fast up, slow down — are what separate production autoscaling from a threshold script.
Lead With the Control Loop
State in the first two minutes that autoscaling is a feedback control problem: sense, decide, act, observe. That framing instantly separates a platform answer from 'just use HPA with CPU'.
Do Not Stop at One HPA Object
Drawing a single HPA reading CPU from metrics-server answers a different question. The interview is about fleet-wide custom-signal scaling with trust, guardrails, and audit.

Section Rescue Kit

Buzzwords to use:

Target-Tracking ScalingSignal Freshness Budget

Safe statements:

  • "Let me separate the scaling decision from the metric pipeline: they fail differently and must degrade independently."
  • "Before choosing algorithms, I will define what makes a signal trustworthy enough to act on."
Design a Metrics-Driven Auto-Scaling - System Design | WinJob | WinJob