Design a Chatbot-Enhanced Live Chat

Problem statement

Design a chatbot-enhanced live chat platform sold as a widget businesses embed on their websites and mobile apps. End users start a conversation and are greeted by a bot. The bot uses NLP-based intent detection and knowledge-base lookup to resolve simple queries (order status, password reset, refund policy). When the bot's confidence drops, when the user explicitly asks for a human, or when policy requires it, the conversation is escalated to a live agent with the full transcript and context attached. The agent works from an agent console; when the agent pool is saturated, the conversation queues with an accurate position. Everything — bot turns, human turns, escalations, metadata — lands in one unified conversation history per conversation.

The two functional halves of this system have opposite design instincts. The bot half is stateless inference: fast, cheap, horizontally scalable, allowed to be wrong within bounds. The live-chat half is stateful real-time messaging: ordered, delivered, acknowledged, presence-aware, and legally auditable. The design challenge is the seam between them — the handoff — because that is where context is lost, messages duplicate, users get stuck in bot loops, and CSAT collapses.

Why this question is distinctive

A pure messaging design (WhatsApp, Discord) optimizes fan-out and delivery. A pure chatbot design optimizes intent accuracy. This question forces you to own both plus the orchestration layer that decides, per message, whether a machine or a human owns the reply, and to do it under four hard constraints from the brief:

1. Bot integration for initial queries — the bot is the default first responder, not an afterthought.
2. NLP intent detection + knowledge-base lookup — retrieval must be low-latency and explainable.
3. Escalation to a human agent — seamless, context-preserving, capacity-aware.
4. Unified conversation history — one timeline across bot and human turns, queryable and auditable.

The five architectural planes

• Conversation plane: the durable record — conversations, messages, participants, handoff events.
• Real-time plane: WebSocket gateways, presence, typing indicators, delivery receipts.
• Bot plane: intent classification, entity extraction, knowledge retrieval, answer generation, confidence scoring.
• Routing plane: the stateful router that owns bot/agent/queue ownership transitions and agent assignment.
• Learning plane: CSAT capture, escalation telemetry, misclassification mining, retraining pipelines.

Strong answers keep these planes separate so the bot plane can degrade (fall back to keyword matching) without touching the real-time plane, and the real-time plane can shed telemetry load without delaying message delivery.

Scale baseline used in this answer

Public figures anchor the realism: Intercom states it serves tens of thousands of businesses and reaches hundreds of millions of end-users monthly; Freshworks reported serving over 67,000 customers with its Freshworks One platform in FY2023 disclosures; LiveChat has publicly reported operating in 150+ countries with tens of thousands of active customers. The specific capacity numbers in this document (5M conversations/day, 40M messages/day, 150K peak concurrent sessions, 1.7K msg/s peak) are explicit design assumptions stated in the estimation sections, not company metrics.