Building a bounded multi-agent system: Six engineering decisions

Decision 1. Roles first, models second

We didn’t arrive at multi-agent out of architectural aesthetics. It started when a single agent stopped scaling. When its toolset held around a dozen tools, everything ran smoothly. Once the number of tools exceeded 30, quality noticeably degraded: the agent picked the wrong tool more often, got confused about step ordering, and finalized tasks it hadn’t actually solved. A single model, a single large prompt containing all the tool descriptions, and a single tool loop — that configuration did not scale with a growing domain.

Source: Unsplash

Splitting into several roles — supervisor, thematic experts, formatter — solved the first problem: each role now sees only its relevant slice of tools. But the split also opened a second degree of freedom we didn’t initially notice: different roles can run on different models.

We tried the two symmetric “one model for everything” defaults in the new architecture. Both disappointed:

• A single cost-efficient open-weight MoE across all roles — cost and latency stayed in range, but the supervisor’s routing became less stable on long or composite queries, and final synthesis started dropping or distorting facts pulled from expert reports.
• A single dense API model across all roles — quality stayed even, but execution-layer cost scaled poorly, because a complex user question can trigger dozens of internal LLM calls.

The underlying issue is that in a multi-agent system, the LLM plays different roles with different failure profiles. No single model is optimal for all of them.

We split the LLMs into two layers by responsibility:

• Control plane — intent classification, routing between experts, final synthesis of the user-facing response. The priorities here are stable structured output, consistent instruction following, and reliable synthesis of several expert reports into one answer. Mistakes at this layer are expensive: a bad route sends the user down the wrong branch of the pipeline; a bad synthesis produces an unsupported final answer. We put a dense API model here.
• Execution plane — expert agents doing bounded tool calling. This role is constrained by contract: a focused instruction from the supervisor, a small tool set, and an explicit termination contract (see Decision 3). Inside this corridor, our MoE deployment gave us a useful cost and latency profile while remaining reliable enough on structured tool arguments for the constrained tasks we route to it. We put it here.

The choice was made on observed behavior in our stack — routing quality, structured-output reliability, tool-call stability, and cost and latency at the loads we actually run — not on training methodology or benchmark rankings.

Decision 2. Prompts as configuration

Once you have more than one agent, prompts become the single largest and single most opaque source of behavioral drift. Our symptoms lined up quickly:

• Shared rules (how to format the final report, how to reference column names verbatim, what output formats are permitted) lived as copy-paste across several agents’ prompts. A change in one place didn’t propagate to the others.
• Each expert’s prompt was one large markdown file. “Please remove the output-formats section from this agent” turned into a hand edit across hundreds of lines with an unreadable PR diff.
• Typos, references to non-existent fragments — all discovered in production, not at build time.

• A single manifest is the source of truth for prompt composition across agents. For each agent it declares who the agent is, which sections its prompt should include, and which shared blocks to pull in.
• Sections became small files. An expert’s prompt is not one file but a folder of sections. Each is small, reviewed independently, changed independently.
• Shared blocks are reusable fragments. Anything common (“reference column names verbatim, don’t invent”, “how to reason economically before a tool call”) lives in a shared folder. Each agent declares whether it wants it.
• Tools became declarations too. For each agent, we declare which tools it may use, when each tool is appropriate, and when it should be avoided. The builder validates these declarations before producing the final prompt, which catches missing references and inconsistent prompt configuration before they reach production. The remaining risk is semantic drift: a tool description can still become outdated when the tool itself changes, so those changes stay part of code review.
• Built prompts are committed artifacts. The builder produces one final file per agent, and those files live in git. A PR diff is a diff on the built file.

Advanced case: one role, many prompt variants

For the supervisor, we extended the same declarative idea. This role has a specific property: its optimal prompt depends heavily on the type of user request. Routing rules for “run a statistical test” differ from routing rules for “launch a compute pipeline”, and both differ across the board from routing rules for small talk.

Keeping all logic for all request types in the supervisor prompt at once means paying tokens for irrelevant fragments on every hop. And there can be many hops within a single user turn. Assembling the prompt from fragments at runtime, on every hop, avoids the token bloat but adds per-hop overhead and, more importantly, makes the effective prompt invisible in git: you cannot see what the model actually saw without replaying a live scenario.

Source: Unsplash

We applied the same declarative principle, with branching. A separate map declares which fragments assemble for which request type. The builder emits one built file per request type. A lightweight classifier runs once per user turn, decides the type, and picks the file. A fallback — the full union of fragments — is also a built file, used when the classifier isn’t confident.

Dynamic context — inventories, progress state, and similar — is added separately at runtime.

The general principle: an agent prompt is not a document, it is system configuration. Once you accept that, everything else — sectioning, shared fragments, build-time validation, per-type variants — becomes an obvious consequence. A freeform markdown file several hundred lines long does not scale.

Decision 3. Termination as a contract, not a heuristic

Each expert runs an inner tool loop: the LLM calls tools, gets results, decides whether to keep going or stop. A naive router for this loop is:

• “LLM returned a tool_call → go to tools, then back to the LLM”
• “LLM returned no tool_call → exit”

In practice, this rule is not reliable. We observed models occasionally returning prose (“Here’s a summary of what I did…”) in place of an explicit tool call across every model family we tested, at different frequencies — and critically, we have no way to predict in advance whether a given turn will do it.

A naive router in this situation either exits with prose in place of a structured report (and the supervisor has to parse text and guess whether that means done or failed), or re-runs the subgraph with “please call a tool” (and gets the same prose in a new phrasing). You cannot ship production on either.

We introduced a single mandatory tool — a completion contract — injected into every outer-graph expert on top of its domain tools. The normal way to exit the tool loop is to call it. Its arguments force the model to say explicitly: what it did, which artifacts it produced, which it reused, the status (success / partial / failed), and unresolved issues (required when the status isn’t success).

The router after the tools step deterministically checks whether the last batch contained a call to the completion tool. If yes — exit. If no — continue. The termination decision does not depend on parsing prose. If the model still doesn’t honor the contract after the bounded recovery path below, the runtime records a failed completion rather than treating leftover prose as a success.

On top of the completion contract, we added two limits. Repeated tool errors force the expert to stop retrying and report failure. A separate tool-call budget asks a productive but long-running expert to finish with the best result it has. We distinguish the two because one means the workflow is broken, while the other means it has simply reached its execution budget — the difference matters for the metrics in Decision 6.

Decision 4. Experts as atomic steps

A common supervisor architecture is a flat multi-agent graph: supervisor plus N experts on the same level, all sharing a single conversation state, every tool call passing through the shared graph. This has three real pain points:

• Shared history becomes noisy. Internal tool calls from every expert accumulate in the same conversation state. Even if they are filtered before the supervisor sees them, maintaining that filtering across agents and failure cases becomes a permanent source of complexity.
• One loop policy for everyone. If one agent needs additional entry-side guards and another does not, in a flat graph you either apply them to everyone or to no one.
• No way to run a single expert in isolation. For QA, debugging, or a single-agent mode, you have to drag the whole graph along.

Any one of these could be patched inside a flat graph. A nested subgraph addresses all three through one architectural boundary rather than three unrelated patches.

We split the graph into two levels.

Outer graph  — supervisor, experts (as nodes), formatter. The supervisor sees each expert as one node. Its routing contract does not depend on how many tool calls the expert made internally.

Inner subgraph  — each expert has its own mini-graph “LLM ↔ tools”, which lives until the completion contract fires. Every expert has a static back-edge to the supervisor. The supervisor is the only one that picks the next node — an expert can’t decide “let me hand off directly to another expert”.

Source: Unsplash

Beyond the obvious wins, this gave us a few concrete things:

• The supervisor’s responsibility became narrower. Its contract with each expert is only “here is an instruction; return a structured task-completion report. ” It doesn’t need to understand internal tool messages.
• Per-agent loop policy. Some agents need additional safeguards that others do not. For example, our standalone compute agent requires its own checkpoint-recovery logic, while experts inside the outer graph rely on the graph’s shared state-management path. Isolated subgraphs let us add these protections locally instead of complicating every agent.
• Single-agent execution. There’s a mode where we take a conversation checkpoint and run one expert subgraph on top of it. Useful both for QA and for scenarios where the user explicitly asks to redo one step.
• Failure boundaries. Expected tool-loop failures are normalized at the subgraph boundary. Instead of receiving a partially broken sequence of tool messages, the outer graph usually receives one completion with status=’failed’.

Decision 5. Data by reference, not by value

In a data-heavy multi-agent system, one expert produces a large object (a table, a list of IDs, a plot spec) and another expert consumes it. The naive approach — the data flows through the text of a tool message, the next agent reads it out of history — breaks in three ways:

• Context bloat. The object takes up the context window of every expert that sees it. As the pipeline grows, this compounds.
• Fidelity. The LLM “passes the data through itself. ” The risk of hallucinated numbers is not hypothetical.
• History trimming. As soon as history is trimmed, the data goes with it. The next iteration can no longer rely on it.

We introduced artifacts as a separate field on the state, alongside the conversation itself. Artifact-producing domain tools store their results in that state under an ID; downstream tools accept string references to that ID instead of the raw payload. The platform resolves a reference into real data before invoking the tool that receives it.

The expert’s system prompt gets a compact inventory describing the available artifacts without embedding their full contents. Downstream agents don’t need the full object in their prompt context — they see the inventory and pass references, which the platform resolves directly at the point of use.

Artifacts live separately from messages, with their own bounded retention policy independent of message trimming. Something may fall out of that retention window — but the supervisor gets a dedicated “evicted artifacts” notice in its prompt so it knows and can re-derive.

There’s a related feature: lineage. At the moment a tool resolves a reference to an artifact, the platform automatically stamps that reference into a source field on the resulting artifact, without requiring each tool author to record lineage by hand. Because these references are recorded automatically, we can trace a final result back to the intermediate and source artifacts that produced it.

Decision 6. Failure as a categorized event

A multi-agent system gets stuck in production in different ways, but from the user’s point of view they all look the same: “slow”, “wrong”, “no answer”. Without dedicated safeguards, these are the underlying pathologies a runtime has to prevent:

• An expert repeatedly returning prose instead of calling a tool or finalizing.
• A tool failing repeatedly, with the expert stuck retrying it.
• A model calling the same tool over and over, each call technically succeeding — a runaway loop that isn’t formally a failure.
• A supervisor repeatedly routing to the same expert without meaningful progress.
• The orchestration hitting its internal recursion limit, surfacing as a raw runtime exception.

• Nudge (once) — if the expert returned only text, give it one prompt to call a tool or finalize. Not infinite.
• Consecutive tool errors — two failed tool batches in a row force completion with failed. A failure path.
• Tool budget — if the expert has called too many tools, ask it to finalize. Not a failure path.
• Supervisor-hop soft limit — after more than N supervisor hops within one user turn, force finalization with an instruction to build the best available answer from the existing results. Not a failure path.
• Recursion overflow — if the hard cap is reached, instead of crashing we run the formatter on top of the last checkpoint state. A recursion overflow no longer automatically becomes a user-visible 500; under the normal recovery path, the user gets a best-effort answer instead of a raw recursion error.
• Route retries — if the supervisor couldn’t produce valid structured output, retry the routing call a bounded number of times, then hand off to the formatter with a generic apology.

These recovery paths emit distinct logs, metadata, and — for the outcomes that matter most for quality tracking, like each expert’s completion status — categorical scores in our tracer, rather than collapsing everything into a single failure counter. That separation matters in practice: without it, all failure events look the same, and prioritizing engineering time on the dominant pathology becomes guesswork.

We also moved our agent observability from Phoenix to Langfuse. Phoenix was useful for inspecting individual low-level traces, but our main questions had shifted: which route did the supervisor choose, how many agents were involved, where did the workflow retry, and how did users rate the final result? Langfuse made it easier to review routing, feedback, and categorical scores together at the level of a complete agent run. Observability also remains separate from the user-facing execution path, so losing telemetry doesn’t stop the workflow itself.

What we haven’t bounded yet

Everything above catches mechanical failures: a tool error, a hop counter, an absent tool_call, an exception. It does not catch a different class of failure: the trajectory was technically valid but semantically wrong. The model successfully called the wrong tool. It successfully built the wrong plan. It successfully closed a task without solving it. Neither the completion contract (Decision 3) nor the categorized safeguards (Decision 6) look at this layer — they see only shape, not meaning.

We have two directions on the roadmap here:

1. An LLM-as-a-Judge validator for complex scenarios. A separate LLM pass reviews the trace — which experts were called, in what order, which artifacts were produced — and issues a verdict on whether the trajectory fits the task. Running this on every request would add unnecessary cost and latency for simple cases, and running it strictly after the answer has shipped only helps monitoring, not the user in front of you. The likely shape is a hybrid: for high-risk workflows, validation runs before the final answer is released and can gate it; for the broader traffic, the same judge runs asynchronously, purely for evaluation and monitoring.
2. A library of reference routes for recurring requests. Some queries follow recognizable patterns, such as comparing two cohorts or launching a known pipeline. Reference examples can help both the supervisor and the validator, while still allowing several valid paths for the same task. A deviation is a signal to inspect the run, not proof that it failed.

Neither of these is a safeguard on runtime primitives. They form a separate semantic validation layer on top of the six decisions above. More on this in a follow-up piece.

Compressed into one line, the six decisions above collapse into a single move: wherever unconstrained model behavior reaches a system boundary, recurring failure modes follow. Replace implicit behavior with an explicit contract, policy, or isolation boundary, and the resulting failure mode becomes bounded, recoverable, or observable.

Summary