Design APM System - System Design | WinJob

APM Mission: Metrics, Traces, and Service Health

Design a production APM platform (Datadog/New Relic-class) for platform and product SRE teams: unified metrics, traces, service maps, SLOs, and alerting on OpenTelemetry. This section covers apm mission: metrics, traces, and service health in the understanding phase.

Why interviewers probe here

Staff+ loops at Datadog, New Relic, and Dynatrace expect you to connect cardinality economics, tail sampling, and exemplar-driven debugging—not a box labeled "monitoring". Lead with OTLP, per-tenant quotas, and query SLOs before storage brands.

Mechanism and operations

State invariants: spans must carry service.name and deployment.environment before indexing; exemplars must reference existing trace_ids; alert rules must declare multi-window burn rates explicitly.

Interview checkpoints (unique to sec-001)

Signal	Target
Metric samples/s	96k (peak)
Spans/s (sampled)	250k
Query p95	< 2s
Ingest lag p99	< 5s

javaOne Dark Pro

1 public final class sec_001Policy {
2   public boolean allowSeries(String tenantId, int activeSeries, int quota) {
3     return activeSeries < quota || tenantId.equals("incident-boost");
4   }
5 }

pythonOne Dark Pro

1 def ingest_tokens_per_sec(spans: float, sample_rate: float) -> float:
2     return spans * sample_rate * 0.4  # ~400B per span gzipped

typescriptOne Dark Pro

1 export function burnRate(errorBudget: number, burn: number): boolean {
2   return burn > 14.4 && errorBudget < 0.5;
3 }

Why interviewers care

APM System interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for APM Mission: Metrics, Traces, and Service Health that proves you understand real outages, not happy-path diagrams.

Key Highlights

•unify RED metrics, distributed traces, and optional continuous profiles
•service map derived from trace edges with health coloring by error rate
•exemplars link p99 latency spikes to representative trace_id
•multi-tenant org hierarchy: org → team → service → env

1	public final class sec_001Policy {
2	public boolean allowSeries(String tenantId, int activeSeries, int quota) {
3	return activeSeries < quota \|\| tenantId.equals("incident-boost");
4	}
5	}

1	def ingest_tokens_per_sec(spans: float, sample_rate: float) -> float:
2	return spans * sample_rate * 0.4 # ~400B per span gzipped

1	export function burnRate(errorBudget: number, burn: number): boolean {
2	return burn > 14.4 && errorBudget < 0.5;
3	}