Design a Log Shipping & Aggregation

Hard45 min
1 / 30
System design answer

Design a Log Shipping & Aggregation

Hard45 min30 sections
understanding•10 min read

Problem Statement: Fleet-Wide Log Shipping and Aggregation

Frames log shipping as a durable, loss-budgeted streaming pipeline rather than a local file-collection problem.

Problem statement

Design a log shipping and aggregation platform that collects log events from every host, container, and service in a large fleet, transports them reliably to a central pipeline, parses and enriches them, indexes them for search and dashboards, archives them for retention, and drives alerts. The narrow version of this problem is "install Filebeat and point it at Elasticsearch." That answer fails a Staff-level interview because it ignores the hard parts: volume spikes, agent failure, aggregator downtime, multi-tenant fairness, parsing cost, PII redaction, and retention economics.

Logs are the highest-volume signal in most infrastructures. Metrics are compact and periodic; traces are sampled; logs are unbounded, bursty, and frequently enormous. A mature platform therefore treats shipping as a streaming-systems problem with a loss budget, not as a best-effort sidecar.

Why the problem is distinctive

Three properties separate log aggregation from ordinary request-serving systems. First, the write path dominates: read/write ratios are heavily skewed toward writes, and the system must absorb producers that are 10-100x their normal rate during incidents, deploy storms, or retry loops. Second, logs have near-zero marginal value individually but enormous aggregate value; the design can therefore trade controlled loss for availability, provided the loss budget, sampling policy, and degradation order are explicit. Third, logs frequently contain secrets and PII by accident, so the pipeline is simultaneously a compliance surface: redaction, access control, retention, and audit are architecture requirements, not afterthoughts.

The four architectural planes

  1. Collection plane: per-node agents (Fluent Bit, Filebeat, Vector, or an OpenTelemetry Collector daemonset) that tail files and container streams, parse lightly, buffer on disk, and ship in batches.
  2. Transport plane: a durable, replayable backbone (Kafka) that decouples agents from processors, absorbs spikes, and provides backpressure.
  3. Processing and storage plane: stream processors that filter, sample, enrich, and redact; a search index (Elasticsearch/OpenSearch or ClickHouse) for hot data; object storage for raw archives.
  4. Consumption and governance plane: query APIs, dashboards, alerting, quotas, retention enforcement, tenancy, and audit.

A strong answer keeps these planes separate. Agents must never block application processes. The transport must never depend on the index being healthy. Queries must never be able to destabilize ingest. The public operating evidence from LinkedIn, Netflix, Cloudflare, and Uber shows that every company that scaled past roughly one billion events per day independently converged on this layered shape, because the failure modes force it.

Every scale number in this answer that is not attributed to a named company is an explicit design assumption stated in the scope section.

Key Highlights

  • •Log aggregation is a write-dominated streaming problem with an explicit loss budget, not a file-sync utility.
  • •The four planes are collection, transport, processing/storage, and consumption/governance.
  • •Agents must be isolated from application processes; aggregator downtime must be absorbed by disk buffers and a replayable backbone.
  • •Index cost, retention tiers, and PII redaction are first-class architecture decisions.
  • •Every uncited scale number in this answer is a stated design assumption, target, or budget.
State the Loss Budget in Minute One
Saying "we accept at-least-once delivery with a 0.1% loss budget under sustained overload, and a defined degradation order" immediately distinguishes a streaming-systems answer from a tooling answer.
Do Not Draw Agents Talking Directly to Elasticsearch
Agent-to-index direct writes couple every node to index health. One slow indexing cluster stalls 40,000 agents, and recovery replays nothing because there is no durable buffer.

Section Rescue Kit

Buzzwords to use:

Loss BudgetWrite-Dominated Workload

Safe statements:

  • "I will separate delivery guarantees from indexing guarantees, because they degrade independently."
  • "Let me define the four planes first so every later failure has exactly one owner."
Design a Log Shipping & Aggregation - System Design | WinJob | WinJob