Problem Statement: Fleet-Wide Log Shipping and Aggregation
Frames log shipping as a durable, loss-budgeted streaming pipeline rather than a local file-collection problem.
Problem statement
Design a log shipping and aggregation platform that collects log events from every host, container, and service in a large fleet, transports them reliably to a central pipeline, parses and enriches them, indexes them for search and dashboards, archives them for retention, and drives alerts. The narrow version of this problem is "install Filebeat and point it at Elasticsearch." That answer fails a Staff-level interview because it ignores the hard parts: volume spikes, agent failure, aggregator downtime, multi-tenant fairness, parsing cost, PII redaction, and retention economics.
Logs are the highest-volume signal in most infrastructures. Metrics are compact and periodic; traces are sampled; logs are unbounded, bursty, and frequently enormous. A mature platform therefore treats shipping as a streaming-systems problem with a loss budget, not as a best-effort sidecar.
Why the problem is distinctive
Three properties separate log aggregation from ordinary request-serving systems. First, the write path dominates: read/write ratios are heavily skewed toward writes, and the system must absorb producers that are 10-100x their normal rate during incidents, deploy storms, or retry loops. Second, logs have near-zero marginal value individually but enormous aggregate value; the design can therefore trade controlled loss for availability, provided the loss budget, sampling policy, and degradation order are explicit. Third, logs frequently contain secrets and PII by accident, so the pipeline is simultaneously a compliance surface: redaction, access control, retention, and audit are architecture requirements, not afterthoughts.
The four architectural planes
- Collection plane: per-node agents (Fluent Bit, Filebeat, Vector, or an OpenTelemetry Collector daemonset) that tail files and container streams, parse lightly, buffer on disk, and ship in batches.
- Transport plane: a durable, replayable backbone (Kafka) that decouples agents from processors, absorbs spikes, and provides backpressure.
- Processing and storage plane: stream processors that filter, sample, enrich, and redact; a search index (Elasticsearch/OpenSearch or ClickHouse) for hot data; object storage for raw archives.
- Consumption and governance plane: query APIs, dashboards, alerting, quotas, retention enforcement, tenancy, and audit.
A strong answer keeps these planes separate. Agents must never block application processes. The transport must never depend on the index being healthy. Queries must never be able to destabilize ingest. The public operating evidence from LinkedIn, Netflix, Cloudflare, and Uber shows that every company that scaled past roughly one billion events per day independently converged on this layered shape, because the failure modes force it.
Every scale number in this answer that is not attributed to a named company is an explicit design assumption stated in the scope section.
Key Highlights
- •Log aggregation is a write-dominated streaming problem with an explicit loss budget, not a file-sync utility.
- •The four planes are collection, transport, processing/storage, and consumption/governance.
- •Agents must be isolated from application processes; aggregator downtime must be absorbed by disk buffers and a replayable backbone.
- •Index cost, retention tiers, and PII redaction are first-class architecture decisions.
- •Every uncited scale number in this answer is a stated design assumption, target, or budget.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate delivery guarantees from indexing guarantees, because they degrade independently."
- "Let me define the four planes first so every later failure has exactly one owner."