Design a Customer Support Chat

Problem statement

Design a web-based customer support chat platform. A visitor on a merchant or SaaS website opens a chat widget, describes a problem, and is routed to a qualified support agent. The agent sees the conversation plus business context such as recent orders, subscription tier, and prior tickets, replies in real time, escalates or transfers when needed, and resolves the case. Every transcript is stored, searchable, and reviewable. A bot may attempt automated resolution first and hand off to a human with full context preserved.

This is not a peer-to-peer messenger. A support chat is a routed workflow system wearing a chat interface. Five properties distinguish it. First, conversations have business state: open, queued, assigned, pending customer, escalated, resolved. Second, one side of the conversation is a scarce, scheduled resource: agents have skills, capacities, shifts, and SLA targets. Third, the conversation must be enriched with data the customer never typed: order history, billing state, account flags, fetched live from CRM and commerce systems under strict field-level authorization. Fourth, everything is evidence: transcripts feed quality review, compliance, refunds, and legal discovery. Fifth, latency is asymmetric: a customer waits in a queue for minutes but expects typing indicators and message delivery in milliseconds.

Why this is hard

A generic chat backend retries a push notification. A support platform must additionally guarantee that a conversation is owned by exactly one agent at a time, that reassignment never loses messages, that a bot-to-human handoff carries the full context package, and that an agent sees only the customer fields their role and jurisdiction permit. Meanwhile the transport layer must hold millions of mostly-idle widget connections cheaply while delivering active messages with sub-second latency.

Public operating baseline versus design assumptions

The category is operationally real. Intercom's public materials describe a Messenger used by tens of thousands of businesses and an engineering blog that documents an Elixir/Phoenix real-time layer behind a Rails API, with the Fin AI agent handing off to human teammates. Zendesk publicly reports serving over 100,000 customers and its engineering blog has described MySQL sharding with Vitess plus Kafka-based event streaming for its messaging platform. Freshworks public investor materials report roughly 67,000 customers across its suite including Freshchat. These are company-reported public figures used as context, not requirements for our fictional system.

For capacity planning, this answer explicitly assumes a mature multi-tenant platform with 50,000 tenant businesses, 8 million concurrent widget connections at global peak, 400,000 concurrent active conversations, 120,000 agents online, 1.5 million new conversations per day, and 60 million messages per day with a 5x event peak. Unless a number is tied to a citation, it is a stated design assumption, target, or budget.

The four architectural planes

1. Transport plane: WebSocket gateways, presence, typing, delivery receipts, reconnect reconciliation.
2. Conversation plane: durable conversation aggregate, message log, assignment ownership, SLA timers.
3. Operations plane: routing and queueing, agent capacity, skill models, escalation, dashboards, quality review.
4. Intelligence plane: bots, automated triage, context enrichment, suggested replies, moderation.

A strong answer keeps these planes separate. The transport plane may degrade to polling without corrupting the conversation plane, and the intelligence plane may fail entirely without blocking a human conversation.