Problem Statement: A Self-Healing Auto-Scaling Group Is a Control System
Frames the auto-scaling group as two coupled feedback control loops operating over unreliable components with slow, expensive, disruptive actions.
Problem statement
Design a cloud capacity group — an EC2-style Auto Scaling Group, a GCP Managed Instance Group, or an Azure VM Scale Set — that keeps a fleet of instances at the right size and the right health without human intervention. Two distinct loops must work together.
The healing loop continuously answers: is each instance actually serving? When an instance crashes, its host fails, a deploy corrupts it, or a spot/preemptible reclaim lands, the system must detect the failure, remove the instance from traffic, and replace it — automatically, within a bounded recovery time.
The scaling loop continuously answers: is the fleet the right size? It reads load signals (CPU, request latency, queue depth), decides a new desired capacity, and executes scale-out or scale-in while draining terminated instances instead of severing live connections.
Why this is a control-systems problem, not a CRUD problem
The plant being controlled is unreliable and slow. An instance launch takes roughly 60–90 seconds from API call to serving traffic: capacity reservation, image boot, agent startup, load balancer registration. Signals are noisy and delayed: a single failed probe can be a garbage-collection pause; a metric can be minutes stale; the health checker itself can be the component that is sick. Actions are destructive: terminating a healthy instance by mistake drops live traffic, and an aggressive scale-out storm can exhaust account API limits or regional capacity.
The core design insight is therefore to separate safety actions from scaling progress: never let a noisy signal trigger an irreversible action without quorum evidence, grace periods, and blast-radius caps; and never let the control plane become a dependency of the data plane — if the scaler dies, the existing fleet must keep serving traffic unchanged.
The four planes
- Health plane: probes, heartbeats, quorum evaluation, flapping suppression.
- Decision plane: metric aggregation, policy evaluation, hysteresis, cooldowns, fencing.
- Execution plane: launches, warm pools, drains, lifecycle hooks, terminations.
- Evidence plane: activities, audit, metrics about the scaler itself, chaos verification.
Public anchors versus design assumptions
Real systems prove this category is mature. Netflix built Chaos Monkey in 2010 and open-sourced it in 2011 precisely to force services to survive random instance termination in production. AWS shipped EC2 Auto Scaling in 2009 and has layered lifecycle hooks, target tracking, predictive scaling, and warm pools on top since. Google's Borg paper (EuroSys 2015) describes automatic task rescheduling on machine failure across cells of about ten thousand machines, and Kubernetes codified the same loop as the Horizontal Pod Autoscaler with a documented 15-second control cycle. Unless a number here is tied to one of those public facts, it is an explicit design assumption for a fictional fleet.
Key Highlights
- •Two coupled loops: a healing loop (detect → replace) and a scaling loop (measure → resize) over one fleet.
- •Instance launch costs 60–90 seconds, so every wrong decision is slow and expensive to undo.
- •The scaler must never be a data-plane dependency: controller outage leaves the fleet serving unchanged.
- •Four planes: health, decision, execution, evidence — each can degrade independently.
- •Netflix (Chaos Monkey, 2010/2011), AWS (Auto Scaling since 2009), and Google (Borg rescheduling) prove the pattern is production-grade.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate the healing loop from the scaling loop because they have different evidence requirements and different blast radii."
- "Before naming any cloud service, let me define what happens when the scaler itself is the failing component."