Design an AI Agent Orchestration Framework

Problem statement

Design an AI agent orchestration framework that executes multi-step, tool-using agent runs on behalf of users and other systems. A run receives a goal, plans, calls an LLM for reasoning, invokes external tools (search, code execution, database queries, SaaS APIs), maintains memory across steps, recovers from tool and model failures, enforces budgets and guardrails, and either completes, escalates to a human, or fails cleanly.

The core insight that separates a strong answer from a chat-wrapper answer: the LLM is a component, not the system. The model proposes text; the orchestrator disposes. Every model output must be parsed, schema-validated, policy-checked, and budget-checked before it becomes a side effect. The orchestration layer owns the state machine, durability, retries, tool authorization, memory, streaming, tracing, and cost control. The model owns only token generation.

Why the problem is distinctive

A classic web backend assumes requests are short, idempotent-friendly, and bounded. An agent run is the opposite: long-lived (seconds to days), non-deterministic, side-effectful, and unbounded unless explicitly constrained. A naive loop of while true: llm() -> tool() produces infinite loops, runaway cost, duplicated side effects, and prompt-injection amplification. The design therefore separates four concerns:

1. Run progress is an eventually advancing durable workflow: created, planning, executing, awaiting approval, completed, failed, or cancelled.
2. Motion-equivalent safety is replaced by action permission: no tool invocation may execute unless the local policy engine has validated arguments, authorization, budget, and rate limits.
3. Model output is untrusted input. Treat tool-call JSON from the LLM like user-supplied data: parse, validate, sanitize.
4. External tool results are also untrusted input. A webpage or email fetched by a tool can carry instructions that attempt to hijack the agent. This is indirect prompt injection, and it is an architecture problem, not a prompt problem.

Public operating baseline versus design assumptions

The category is operationally real. Anthropic published the Model Context Protocol (MCP) as an open standard for connecting models to tools and data sources, and major vendors shipped agent platforms: AWS Bedrock Agents, Google Vertex AI Agent Builder and the Agent2Agent (A2A) protocol, Microsoft AutoGen and Semantic Kernel. LangChain's LangGraph documents graph-based agent state machines with durable checkpointing. Temporal publicly describes durable execution for AI agents and reports usage for production agents such as Datadog Bits AI. These are cited public references, not requirements for our fictional system.

For capacity planning, this answer explicitly assumes a mature platform: 2 million agent runs per day, 100,000 concurrent active sessions at peak, 12 steps per run average, and a 5x event peak. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold.

The five architectural planes

1. Executive plane: the durable run state machine, step sequencing, checkpointing, resume, and compensation.
2. Model plane: the model gateway that routes, retries, fails over, caches, meters tokens, and enforces per-tenant budgets.
3. Tool plane: tool registry, schema validation, sandboxed execution, idempotency, circuit breakers, and human-approval gates.
4. Memory plane: working context, compaction, checkpoints, and long-term semantic memory.
5. Governance plane: guardrails, tracing, evaluations, cost accounting, audit evidence, and policy release.

A strong interview answer keeps these planes separate. The executive plane must survive model-provider outages. The tool plane must refuse oversized or unauthorized calls even when the model insists. The memory plane must keep context inside the model window without silently dropping safety-relevant constraints.