Fault Tolerance in Multi-Agent Pipelines

Key facts, with sources

• Anthropic reported that a multi-agent research system using an Opus lead agent with Sonnet subagents outperformed a single-agent Opus baseline by 90.2 percent on its internal research eval. (ByteByteGo)
• Anthropic's multi-agent research system used about 15x more tokens than a normal chat interaction, and token usage alone explained roughly 80 percent of performance variance. (The AI Engineer)
• The MAST research taxonomy identified 14 distinct failure modes across 7 popular multi-agent frameworks including AutoGen, ChatDev, and CrewAI, grouped into system design flaws, inter-agent misalignment, and task verification failures. (arXiv)
• Salesforce research found organizations run an average of 12 AI agents and projects multi-agent adoption to surge 67 percent within two years as enterprises move toward orchestration. (Salesforce)
• Multi-agent orchestration with three or more agents represents about 22 percent of enterprise agent deployments in 2026, projected to reach roughly 45 to 50 percent by 2027. (OnAbout AI)

Agent pipelines fail in two distinct ways

Mechanical failures are the familiar kind: timeouts, rate-limit responses, network resets, JSON that does not parse. They are loud, detectable at the call site, and classic retry-with-backoff handles most of them. Every team builds for these, because they announce themselves.

Semantic failures are the dangerous kind: the call succeeds, the JSON parses, and the content is wrong — a summarizer that quietly dropped the critical section, an extractor that swapped two fields, a planner that emitted a circular task list. Nothing throws. The bad output flows downstream, each subsequent agent builds on it, and the failure surfaces three steps later wearing a different agent's name. Retry logic is useless here because nothing looks failed; only validation gates at handoffs catch this class, and a pipeline without them is not fault-tolerant no matter how sophisticated its retries are.

Checkpoint at every handoff

After each step completes and validates, persist its result — keyed by run ID and step name — before the pipeline moves on. When anything dies mid-run, whether from a crashed pod or an exhausted retry budget, the run resumes from its last checkpoint instead of re-executing everything upstream. In a pipeline where each step is an LLM call, re-running completed work is not just slow, it is a direct cash cost, and on long pipelines it dominates the failure bill.

Checkpoints pay two further dividends. They enable replay debugging — re-execute exactly one step against its recorded input while iterating on a prompt. And they make idempotency almost free: a redelivered or duplicated task checks for an existing checkpoint and returns it, turning double-execution into a no-op. Any store works — Postgres, Redis, object storage — as long as writing the checkpoint happens before acknowledging the step complete.

Retries for LLM steps are not retries for databases

Database retry wisdom assumes transient failure: the same request against the same system succeeds a moment later. That holds for the mechanical class — a rate-limited call retried with backoff and jitter usually goes through. It does not hold for output failures: an identical prompt that produced malformed or invalid output will often produce a near-identical failure again, because nothing about the situation changed.

So I retry the two classes differently. Mechanical failures: same input, exponential backoff, capped attempts. Validation failures: mutate the input — append the concrete error to the prompt ('your last output failed validation because…') so the model has something to correct against. This error-feedback retry converts a surprising fraction of hard failures into second-attempt successes. On the final attempt I sometimes escalate to a stronger model before giving up; it is cheaper than a dead-lettered run when it works, and bounded when it does not.

Validation gates and runaway budgets

Every handoff gets a gate with escalating rigor. Schema validation is mandatory and nearly free: the output parses and every required field is present and typed. Above that, mechanical invariants specific to the step: citations actually appear in the source text, IDs reference things that exist, counts match. Above that, for steps whose failures have taught me to be paranoid, a cheap-model judge does a single sanity check — does this summary plausibly cover this input — which catches gross semantic failures for pennies.

The other gate is on the loop itself. Any agent that decides its own next action can decide badly forever, so every run carries hard budgets: maximum steps, maximum tokens, maximum wall-clock. Exceeding a budget is a failure with a specific name — not a hang, not a mystery bill — and it routes to the same handling as any other unrecoverable error.

import json
from pydantic import BaseModel, Field, ValidationError

MAX_ATTEMPTS = 3

class StepResult(BaseModel):
    summary: str
    citations: list[str] = Field(min_length=1)

def run_step(run_id: str, step: str, prompt: str, store) -> StepResult:
    if cached := store.load(run_id, step):
        return StepResult.model_validate(cached)  # resume from checkpoint

    feedback = ""
    raw = ""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        raw = call_agent(prompt + feedback)  # your LLM call w/ backoff inside
        try:
            result = StepResult.model_validate(json.loads(raw))
        except (json.JSONDecodeError, ValidationError) as err:
            feedback = (
                f"\n\nYour previous output failed validation:\n{err}\n"
                "Return only valid JSON matching the required schema."
            )
            continue
        store.save(run_id, step, result.model_dump())  # checkpoint, then ack
        return result

    store.dead_letter(run_id, step, raw)  # full context for human review
    raise StepFailed(f"{step} failed after {MAX_ATTEMPTS} attempts")

Degraded modes and the dead-letter queue

Not every failure deserves a retry storm; some deserve a smaller answer. Decide per pipeline what degraded success looks like: return the report with one section marked unavailable, serve the cached result from the last successful run, or fall back to a simpler single-shot generation that skips the failing stage. Users forgive a labeled gap far more readily than an error page, and honest partial results usually beat both.

What remains lands in the dead-letter queue with everything a human needs: the run's trace, the step's input, every attempt's output, and the validation errors. Two disciplines keep the DLQ useful. Alert on its rate, not its existence — a steady trickle is the cost of doing business, a spike after a deploy is a regression announcing itself. And actually review it weekly: recurring dead-letter patterns are the roadmap for the next round of prompt fixes and validation rules, which is how a pipeline gets more reliable every month instead of merely surviving.

When to hire senior help

Multi-agent orchestration is one of the least commoditized skills in AI engineering, and teams that succeed usually include someone who has debugged coordination failures in production. Get senior review before committing to an orchestrator-worker design, because architectural mistakes at this layer are expensive to unwind after launch. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — Multi-Agent Architectures projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Defaulting to multi-agent when a single agent with good tools would do, since the roughly 15x token multiplier only pays off when subtasks are genuinely parallel and high value
• ✕Letting subagents share full conversation history instead of scoped task briefs, causing context bloat, contradictory actions, and coordination failures
• ✕Shipping without a verification layer, so errors propagate through agent chains unchecked; task verification failures are one of the three MAST failure categories
• ✕Skipping per-agent trace observability, which makes it impossible to identify which agent in the chain caused a bad final output

Frequently asked questions

Sources

• ByteByteGo
• The AI Engineer
• arXiv
• Salesforce
• OnAbout AI