Design Anomaly Detection - System Design | WinJob

Problem Statement & Context

You are designing multi-tenant observability anomaly detection — the pipeline Datadog, Dynatrace, and AWS DevOps Guru run over metrics, logs, and traces. This section focuses on mission, personas, and why anomaly detection is not threshold alerts. Interviewers expect you to separate streaming ingest (durable, partitioned) from detection (seasonality-aware, low false positives) and alert delivery (SLO-aware, deduplicated).

1.1: detect abnormal metric/log/trace behavior per tenant
1.2: minimize alert fatigue while catching real incidents
1.3: support SRE, security, and product analytics personas

Mechanism

Customers ship telemetry via agents and OpenTelemetry. The platform maintains per-series baselines (STL, Holt-Winters, or learned encoders), scores recent windows, and emits incidents with severity, context graphs, and runbook links. Security teams want UEBA-style spikes; SRE wants saturation before paging.

Interview phrasing

Open with scale: 40K tenants, 12M active series, 8M points/sec peak. State detection SLO 60s for infra metrics vs 5m for business KPIs.

Failure mode to volunteer

Static thresholds on elastic workloads—Kubernetes CPU at 80% is normal during batch, catastrophic on API tier.

Signal	Target
Peak ingest	8M pts/s
Active series	12M
Detect SLO	60s
Noise reduction	70%

javaOne Dark Pro

1 public record TenantContext(String orgId, String env, String service) {}

pythonOne Dark Pro

1 def is_multi_tenant(org_id: str) -> bool:
2     return bool(org_id)

typescriptOne Dark Pro

1 export interface TenantContext { orgId: string; env: string; service: string; }

Why interviewers care

Anomaly Detection interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement & Context that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Problem Statement & Context: mission, personas, and why anomaly detection is not threshold alerts
•Ops angle: support SRE, security, and product analytics personas
•Metric anchor: 8M pts/s / 60s

1	def is_multi_tenant(org_id: str) -> bool:
2	return bool(org_id)