Design a Social Media Hashtag Aggregator

Hard45 min
1 / 30
understanding•9 min read

Problem Framing: Cross-Platform Trends as a Streaming Problem

Frames the hashtag aggregator as a multi-source streaming aggregation system, not a read-through API proxy.

Problem statement

Design a social media hashtag aggregator that continuously collects posts from multiple platforms (Twitter/X, Instagram, TikTok, Reddit, YouTube), extracts and normalizes hashtags, counts their usage, detects which ones are trending right now, and serves ranked trend lists to users with optional filters by region and category.

The trap in this question is to draw a web server that calls the Twitter API when a user asks for trends. That design fails immediately: platform APIs impose strict rate limits (X's firehose is a paid enterprise product; the standard tiers allow only thousands of reads per 15 minutes), trends must reflect velocity over the last few minutes, and a read-through design recomputes global aggregations on every request. The correct framing is a stream processing pipeline: platform data is pulled or pushed continuously into an event backbone, hashtag events are normalized and counted in sliding time windows, and users read precomputed, cached trend lists.

Why this problem is distinctive

A hashtag aggregator sits at the intersection of four hard sub-problems:

  1. Multi-source ingestion with hostile dependencies. Every upstream platform is a third party with its own rate limits, schema, authentication model, outage behavior, and ToS. Twitter/X restructured its API tiers in 2023, eliminating the free streaming API; Instagram's Graph API restricts hashtag search to business accounts; Reddit and YouTube have quota-based APIs. The system must degrade gracefully when any single source goes dark, because one will always be dark.
  1. Cross-source identity resolution. The hashtag #CoffeeLovers on Instagram, #coffeelovers on X, and #coffee_lovers on Reddit may or may not be the same trend. Unicode normalization, case folding, diacritic handling, underscore vs camelCase conventions, and transliteration (#café vs #cafe) must be resolved into canonical entities before counting, or the same trend fragments into ten weak signals.
  1. Trending is a rate problem, not a volume problem. A hashtag with 100,000 mentions today that has been flat all day is not trending; a hashtag with 2,000 mentions in the last 5 minutes that had zero before is. Twitter explicitly stated its trends algorithm emphasizes velocity and novelty over raw volume. This forces burst-detection scoring (exponential decay, z-scores against a baseline, or Kleinberg-style automata) rather than naive COUNT(*) ORDER BY.
  1. Top-K over a huge, shifting key space. At platform scale there can be millions of distinct hashtags active in a 24-hour window. Exact counting of every hashtag is possible with careful partitioning, but finding the top 50 out of millions per (region, category, time-window) cell is the classic heavy-hitters problem, with approximate structures (Count-Min Sketch, Space-Saving) as the standard interview-grade answer and exact per-shard counters as the production-grade answer.

Public operating baseline

Real systems prove the category. Twitter publishes trends for 70+ regions and personalizes them; at its public peak Twitter reported roughly 500 million tweets per day (2013-era official figures), a substantial fraction carrying hashtags. Google Trends documents a normalization methodology (interest relative to total search volume, not absolute counts). Commercial listening platforms like Brandwatch and Sprout Social ingest from dozens of sources and report processing billions of documents per month. These are cited context; every scale number used for our design below is an explicit assumption.

The four planes

  1. Ingestion plane: per-platform connectors, rate-limit budgeting, webhook + polling hybrid, raw post intake.
  2. Processing plane: normalization, deduplication, windowed counting, burst scoring, spam filtering.
  3. Serving plane: trend-list compilation, regional/category shards, caching, public APIs.
  4. Governance plane: source health, ToS compliance, abuse detection, observability, backfill/replay.

A strong answer keeps these planes separate: a platform outage degrades ingestion without corrupting serving, and a serving spike never backpressures the ingestion pipeline into dropping events.

Key Highlights

  • •The aggregator is a stream-processing pipeline with precomputed trend lists, never a read-through API proxy.
  • •Trending is a velocity-and-novelty problem: burst detection, not raw volume ordering.
  • •Cross-platform hashtag identity requires canonicalization before counting or trends fragment.
  • •Every upstream API is a hostile dependency with rate limits and outages; degradation is designed, not hoped for.
  • •Four planes: ingestion, processing, serving, governance; failures are contained within planes.
Lead With the Streaming Reframe
State in the first two minutes that this is a stream aggregation problem with precomputed, cached trend lists. Candidates who start with a REST proxy over platform APIs fail the question before minute five.
Do Not Compute Trends Per Request
Scanning hashtag events at read time makes p99 latency a function of event volume and melts under traffic. Trend lists are compiled asynchronously and read from cache.

Section Rescue Kit

Buzzwords to use:

Velocity-Based TrendingHeavy Hitters

Safe statements:

  • "I will separate the write-heavy aggregation pipeline from the read-heavy serving path before choosing any technology."
  • "Before drawing boxes, let me clarify which platforms, which regions, and whether trend lists are global or personalized."
Design a Social Media Hashtag Aggregator - System Design | WinJob | WinJob