Problem Statement: Bringing Thousands of Machines to Trusted Life
Frames provisioning as a trust and correctness problem, not a script-runner.
Problem statement
Design a server provisioning and bootstrapping platform that takes a freshly created VM or bare-metal machine from power-on to a trusted, registered, production-ready state. The platform must allocate capacity, select a validated OS image, boot the machine, run first-boot installation steps (base OS packages, monitoring agents, security baseline, SSH keys), apply base configuration, securely retrieve secrets and credentials, verify the machine's identity, and register it with configuration management and monitoring before it may receive traffic.
Two delivery strategies coexist. A golden-image path pre-bakes OS plus platform agents into machine images (AMIs, GCE images, Azure Compute Gallery images) so first boot is fast and deterministic. A boot-script path (cloud-init user-data, Ignition, kickstart, preseed, vendor data) performs installation and registration on first boot. Production systems almost always use a hybrid: a golden base image plus a bounded, checkpointed first-boot script. The design must define the exact ownership boundary between the image layer and the boot layer.
Why the problem is distinctive
A request-handling service can retry a failed transaction. A half-bootstrapped machine is a physical side effect that can look healthy while being broken: packages installed but agent missing, agent running but holding the wrong identity, identity correct but secrets fetched from an impostor service. The first-boot window is simultaneously the most fragile and the most security-sensitive moment in a machine's life, because the machine starts with no trust root, no credentials, and often a network-visible management surface.
Therefore the design separates three concerns that weak answers merge. Boot success means the script exited zero. Readiness means health and compliance gates passed. Trust means the machine possesses an attested identity and scoped credentials. A machine may reach boot success without readiness, and never enters service without trust.
Public operating baseline versus design assumptions
The category is operationally real and large. Google's published Borg paper describes cells with more than 10,000 machines where machines PXE-boot a kernel and join a machine database before running tasks. Netflix publicly built AMI baking (Aminator) and open-sourced Spinnaker for deployment orchestration, treating instances as immutable artifacts that are replaced rather than repaired. cloud-init is the de facto first-boot framework across AWS, GCP, and Azure, and CoreOS introduced Ignition to run provisioning in the initramfs before the root filesystem mounts. These are cited public references, not requirements for our fictional system.
For capacity planning, this answer explicitly assumes a mature multi-region fleet with 250,000 managed machines, 40,000 provision attempts per day, and storm bursts up to 15,000 boots per hour during region expansion or failover. Unless a number is tied to a named public source, it is a stated design assumption, target, or budget.
The four architectural planes
- Machine plane: firmware or hypervisor boot, OS image, boot script engine, local agent, journals.
- Control plane: provisioning orders, capacity allocation, image registry, orchestration, lifecycle workflows.
- Trust plane: identity issuance, attestation, secret and credential delivery, revocation.
- Fleet-learning plane: boot telemetry, funnel metrics, audit evidence, image promotion decisions.
A strong answer keeps these planes separate. The control plane may stall without making booted machines unsafe. The trust plane must fail closed: no credentials to an unverified machine. The learning plane may lag without blocking boots.
Key Highlights
- •Separate boot success, readiness, and trust: a zero exit code is not a production machine.
- •Hybrid delivery is the norm: golden base image plus bounded checkpointed first-boot script.
- •The first-boot window is the most fragile and most security-sensitive phase of machine life.
- •Public systems from Google, Netflix, and the cloud-init/Ignition ecosystem prove the pattern at scale.
- •Four planes: machine, control, trust, and fleet learning; trust fails closed, learning may lag.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate boot success, readiness, and trust before choosing any technology."
- "The interesting design work is the first-boot trust gap, not the script execution itself."