Design a Realtime NLP Pipeline

Problem statement

Design a realtime NLP pipeline that ingests high-volume text streams—tweets, support tickets, chat messages, product reviews, financial news—and applies tokenization, sentiment classification, topic labeling, named-entity recognition, or key-phrase extraction on the fly, producing enriched output that routes to dashboards, alerting systems, reactive triggers, or downstream microservices within a strict latency budget.

This is not a batch ETL job with a freshness SLA. The pipeline must sustain continuous inference at scale: hundreds of thousands of messages per second at peak, p99 inference latency under 50 ms for lightweight classifiers and under 200 ms for transformer-based models, and graceful degradation when GPU capacity saturates. The output is not merely stored—it acts: routing a support ticket to the correct team, triggering a sentiment alert on a brand-mention spike, enriching a search index, or feeding a real-time moderation queue.

Why this problem is distinctive

A traditional analytics pipeline can tolerate minutes of lag. A realtime NLP pipeline cannot. The moment a user posts a message, the system must tokenize, vectorize, run inference, and emit structured results before the business event expires. A sentiment alert arriving ten minutes after a PR crisis begins is worthless. A support ticket routed to the wrong queue because classification lagged behind the SLA creates compounding customer frustration.

The design therefore separates ingestion throughput from inference latency from model freshness. Ingestion is a streaming problem: partition, buffer, absorb bursts. Inference is a compute problem: batch, quantize, parallelize across accelerators. Model freshness is an MLOps problem: canary, shadow, rollback without stopping the stream. A strong answer keeps these three planes independent so that a model rollout never blocks ingestion and a traffic spike never corrupts model state.

Public operating baseline versus design assumptions

Public evidence establishes the category. Twitter processes approximately 500 million tweets per day at peak rates exceeding 6,000 tweets per second, applying content classification, trending-topic detection, and abuse detection in near-real-time. Bloomberg processes millions of financial news documents daily, extracting entities, sentiment, and event signals within seconds of publication to feed trading terminals. Discord handles over 1 billion messages per day, applying moderation classifiers and search indexing in streaming fashion.

For capacity planning, this answer explicitly assumes a mature multi-tenant platform with 50 million messages per day, 500,000 messages per second at 10× event peak, and a p99 end-to-end latency target of 100 ms from ingestion to enriched output. Unless a number is tied to a citation, it is a stated design assumption.

The four architectural planes

1. Ingestion plane: stream connectors, schema validation, partitioning, buffering, and backpressure.
2. Inference plane: tokenization, model serving, micro-batching, GPU scheduling, and result assembly.
3. Routing plane: output fan-out to dashboards, alert systems, search indexes, moderation queues, and downstream APIs.
4. Model-lifecycle plane: training pipelines, evaluation, canary deployment, A/B testing, rollback, and drift monitoring.

A strong interview answer keeps these planes separate. It allows the routing plane to degrade without blocking ingestion, and it permits the model-lifecycle plane to promote new models without pausing the inference plane.