Clarify Log Sources and Investigation Workloads
How Clarify Log Sources and Investigation Workloads (understanding) informs Log Aggregation architecture and interviewer depth.
Clarify Log Sources and Investigation Workloads
Fleet scale anchor: 50,000 hosts, 2M events/sec sustained ingest, 5× burst during incidents, ~500 bytes per event after gzip on the wire.
Before drawing boxes, map who emits logs and how engineers investigate during deploys and outages.
Source taxonomy
| Source class | Examples | Investigation pattern |
|---|---|---|
| Application | JSON stdout, structured errors | Filter by service, trace_id, deployment_id |
| Kubernetes | kubelet, CNI, ingress | Node + pod labels; correlate with rollout timestamps |
| CI/CD | build agents, deploy hooks | Tie git_sha to error spikes after merge |
| Security | WAF, IDS, auth gateway | Sub-minute freshness; immutable retention |
| Audit | admin API, IAM changes | WORM storage; legal hold overrides TTL |
Clarifying questions to ask the interviewer
- Is this multi-tenant SaaS or a single internal platform?
- Do we need full-text search on message bodies or mostly field filters?
- What searchable lag is acceptable for ERROR vs DEBUG classes?
- Are deploy markers (canary steps) first-class in the query language?
- Must we support GDPR erasure and legal hold on the same index?
DevOps coupling
Elastic/Splunk/Datadog interviews reward linking logs to release safety: annotate every line with git_sha, rollout_wave, and k8s_namespace so on-call can query "errors since canary step 2" without guessing time windows.
Out-of-scope for v1 (state explicitly)
- Building a full APM trace store (integrate via
trace_idonly) - Long-term ML anomaly detection (batch export to data lake is fine)
- Replacing metrics systems (emit counters from indexers, do not own TSDB)
How to open this one
The framing that lands for log aggregation is the ingest-index-query pipeline at scale: agents ship structured logs to a buffer (Kafka) that absorbs bursts, an indexer writes to tiered storage, and queries hit hot indexes while cold data sits cheap. Lead with backpressure — what happens when ingest outruns indexing — and the failure story that matters: a log flood from one noisy service threatens to drown the pipeline, so per-tenant rate limits and sampling protect everyone else. That shows you understand log systems fail by overload, not by being too small.
Key Highlights
- •Focus: Clarify Log Sources and Investigation Workloads
- •Anchor on ingest lag, query p95, and tiered retention cost.
- •Call out hot shards, parser drift, and partial query honesty.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will quantify ingest lag and query p95 before naming shard counts."
- "If we must cut cost, DEBUG sampling moves first—not SECURITY freshness."