Problem Statement: The Container Supply Chain as a Security Boundary
Frames the pipeline as a trust factory that converts untrusted source code into verifiably safe-to-run images.
Problem statement
Design a secure container pipeline that takes application source code and produces a container image that a Kubernetes cluster will run in production, while guaranteeing four properties: the image was built from an approved commit by an approved builder, the image contains no vulnerabilities above a policy threshold, the image is cryptographically signed by a key that only the pipeline holds, and every deployment re-verifies that signature and policy before a single container starts.
This is not a generic CI/CD question. A normal pipeline answers 'does the code compile and do tests pass?' A secure pipeline answers a harder question: 'can an attacker who controls one build machine, one dependency mirror, or one registry credential push a malicious image to production without detection?' The 2020 SolarWinds SUNBURST compromise, which injected a backdoor through the vendor's own build system, proved that the build path is a first-class attack surface. The December 2021 Log4Shell disclosure (CVE-2021-44228) proved that a single vulnerable third-party dependency can instantly become a fleet-wide emergency: Sonatype's 2022 State of the Software Supply Chain report counted roughly 35,000 Java artifacts on Maven Central depending directly or transitively on log4j-core. Both lessons apply directly to container images, because an image freezes an entire dependency tree into an immutable artifact.
Why this is a distinctive design problem
A web backend can patch a bug by redeploying. A container pipeline cannot 'unbuild' a compromised image that has already been pulled into 300 clusters. The design therefore separates two concerns that generic pipelines merge. Build throughput is an optimization problem: queue latency, cache hit rate, worker utilization. Trust is an invariant problem: an image reaches production only if it carries verifiable proof of how it was built, what it contains, and who signed it. Throughput may degrade; trust may not.
The pipeline has five stages and each one is a trust transfer. Source control transfers trust to the builder. The builder transfers trust to the image. The scanner transfers trust to a vulnerability verdict. The signer transfers trust to a signature. The admission controller transfers trust to the running workload. Every transfer needs a cryptographic or policy anchor, otherwise the chain has a break that an attacker can widen.
The four architectural planes
- Build plane: source checkout, dependency resolution, hermetic builds, layer caching, artifact production.
- Security plane: vulnerability scanning, SBOM generation, secret detection, policy evaluation, gating.
- Trust plane: image signing, attestation, key management, transparency logging, verification.
- Enforcement plane: registry admission, cluster admission controllers, runtime verification, incident response.
A strong interview answer keeps these planes separate. The security plane may be slow or partially degraded without weakening the trust plane, and the enforcement plane must never depend on the liveness of the build plane. If the scanner is down, the correct behavior is to block new images from promotion, not to silently promote unsigned ones.
Public baseline versus design assumptions
Public evidence shows the category is real and urgent. Google Cloud ships Binary Authorization and Container Analysis as managed equivalents of the verify-before-run and scan-before-push pattern. Capital One published how it built a centralized container security program covering roughly 100,000 containers a day using open-source scanners and a policy engine. Netflix's platform engineering team runs images through an immutable pipeline (Spinnaker plus Nebula certificates) where deployment identity is cryptographically checked rather than assumed. These are cited company figures, not requirements for our fictional system.
For capacity planning, this answer explicitly assumes a large engineering organization: 20,000 developers, 4,500 repositories, 15,000 image builds per day, a 5x release-train peak, and 40 production Kubernetes clusters. Unless a number is tied to a citation, it is a stated design assumption, target, or budget — not a claim about any company's private architecture.
What failure looks like
The worst outcome is not a slow pipeline. It is a pipeline that happily signs malware. The design must therefore treat three events as system-level emergencies: an unsigned image appearing in the production registry, a signed image whose attestation does not match its SBOM, and a signing key used outside the pipeline's identity. Each one has detection, containment, and revocation designed in advance, not improvised during the incident.
Key Highlights
- •A secure container pipeline is a trust factory: five stages, five trust transfers, each cryptographically anchored.
- •Build throughput is an optimization problem; trust is an invariant problem. Throughput may degrade, trust may not.
- •SolarWinds SUNBURST (2020) proved build systems are attack surfaces; Log4Shell (CVE-2021-44228) proved one dependency can be a fleet-wide emergency.
- •Four planes: build, security, trust, enforcement. The enforcement plane never depends on build-plane liveness.
- •System-level emergencies: unsigned image in prod registry, signature/SBOM mismatch, signing key used outside pipeline identity.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate build throughput, which may degrade, from trust invariants, which may never degrade."
- "Before choosing tools, let me define the five trust transfers and the anchor that protects each one."