Design a Large-Scale Email Campaign Metrics Tracker

Problem statement

Design an email campaign metrics tracking platform that records every send, delivery, open, click, bounce, complaint, and unsubscribe across billions of sends per month, then serves marketers near-real-time dashboards covering campaign health, segment drill-downs, device and domain breakdowns, and A/B experiment comparison. The send path (ESP or SMTP relay) and the measurement path (tracking pixel, click redirect, ESP webhooks) are distinct systems that must converge on one immutable, deduplicated event ledger.

This is not a CRUD reporting app. It is a write-heavy big-data problem: event rate spikes to hundreds of thousands of messages per second during holiday campaigns, every recipient link is unique, events arrive duplicated, delayed, and out of order, and the numbers marketers act on must be explainable, reproducible, and auditable. A marketer who pauses a campaign because open rate looks wrong must be able to trust the pipeline enough to make that call.

Why the problem is distinctive

• Per-recipient, per-link uniqueness: click tracking requires tokenization at send time. You cannot retrofit identifiers after the email leaves the building.
• Adversarial and noisy sources: ESP retries duplicate webhooks, Apple Mail Privacy Protection and Google image caching synthesize opens, prefetchers and bots fire pixels and clicks, and corporate proxies rewrite user agents.
• Two latency regimes coexist: the click redirect serves a human reader and must answer in milliseconds, while dashboard aggregation tolerates seconds to minutes.
• Compliance events gate future sending: complaints, unsubscribes, and hard bounces must never be dropped or delayed beyond policy. Losing them is not a metrics error; it is a legal exposure.
• Opens are fundamentally unreliable. The design must expose confidence classes instead of pretending a pixel fire is truth.

The four architectural planes

1. Send and tracking plane: campaign orchestration, token issuance, pixel serving, click redirection.
2. Ingestion plane: webhook receivers, event normalization, signature validation, edge de-duplication.
3. Processing and serving plane: stream processing, counter materialization, OLAP drill-downs, data lake archive.
4. Governance plane: suppression lists, deliverability reputation, compliance retention, audit evidence.

A strong answer keeps these planes separate. The serving plane may lag, but ingestion must never lose a complaint. The tracking plane may degrade image quality of data, but it must never slow a reader's click. The governance plane may block a campaign, but it must never silently rewrite historical metrics.

Public operating baseline versus design assumptions

Twilio SendGrid has publicly reported processing on the order of 80 billion emails per month; Mailchimp has publicly described sending over 100 billion emails per year; Amazon SES documents event publishing for delivery, bounce, complaint, delivery delay, reject, send, open, click, rendering failure, and subscription events to SNS, Kinesis Data Firehose, or S3. These are cited company signals that the category operates at massive scale. Every capacity number used later in this answer is an explicit design assumption, budget, or target unless tied to such a citation.

What we explicitly separate

Mission success here is metric freshness and correctness: campaigns appear in dashboards, counters converge, drill-downs answer in seconds. Data safety is the invariant: no complaint is lost, no recipient is emailed after unsubscribe, no metric double-counts a deduplicated event, and no raw email address leaks into the analytics tier. Freshness may degrade; integrity may not.