Design Chatbot Platform

Problem Scope and Product Goals

A chatbot platform is not a single bot—it is infrastructure for thousands of tenant-defined agents across web, SMS, Slack, Teams, and voice. The interview scope should mirror Dialogflow, Rasa, or Microsoft Bot Framework: builders author NLU intents, slot-filling dialog flows, channel adapters, and optional RAG corpora; end users experience low-latency turns with safe fallbacks.

Product north stars

• Builder velocity: publish a bot version in minutes with staged rollout (draft → canary → production).
• Runtime quality: intent resolution plus fulfillment under strict latency budgets; human handoff when confidence drops.
• Economics: predictable cost per conversation (NLU + LLM tokens + storage).

Out of scope for v1 (say explicitly)

Custom model training pipelines, on-device inference, and full CCaaS replacement are extensions—not core platform guarantees. Anchor v1 on managed NLU, flow orchestration, omnichannel delivery, and observability.

Success metrics interviewers expect

State p95 turn latency (e.g., 800 ms FAQ, 2.5 s RAG), 99.9% message delivery after ack, and tenant-fair scheduling so one noisy bot cannot starve others.

Why interviewers care

Chatbot Platform interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

The failure that defines the design

The outage to narrate is the runaway cost-and-latency spike from unbounded context. As conversations grow, naively stuffing the full history plus retrieved documents into every LLM call blows past the context window, spikes token cost per turn, and pushes latency past what users tolerate — and a single abusive or looping conversation can burn thousands of dollars. The fix is structural: a strict per-turn token budget, history summarization/truncation, retrieval capped to top-k, and per-tenant cost limits. A chatbot platform lives or dies on controlling the cost and latency of model calls, not just on getting a good answer once.