Multi-Agent Debugging and Trace Visualization

Key facts, with sources

• Anthropic reported that a multi-agent research system using an Opus lead agent with Sonnet subagents outperformed a single-agent Opus baseline by 90.2 percent on its internal research eval. (ByteByteGo)
• Anthropic's multi-agent research system used about 15x more tokens than a normal chat interaction, and token usage alone explained roughly 80 percent of performance variance. (The AI Engineer)
• The MAST research taxonomy identified 14 distinct failure modes across 7 popular multi-agent frameworks including AutoGen, ChatDev, and CrewAI, grouped into system design flaws, inter-agent misalignment, and task verification failures. (arXiv)
• Salesforce research found organizations run an average of 12 AI agents and projects multi-agent adoption to surge 67 percent within two years as enterprises move toward orchestration. (Salesforce)
• Multi-agent orchestration with three or more agents represents about 22 percent of enterprise agent deployments in 2026, projected to reach roughly 45 to 50 percent by 2027. (OnAbout AI)

Why print debugging dies at two agents

Single-agent debugging is tolerable: one conversation, one sequence, read it top to bottom. Add a second agent and concurrency, and the log becomes an interleaved shuffle of two conversations with no thread connecting cause to effect. Add non-determinism and the bug you are chasing may not even reproduce on the next run — the evidence from the failing run is all you get, so it had better be complete.

The killer property is that agent failures surface far from their cause. A writer agent produces a report missing half its content; the actual defect was a researcher three steps earlier returning results that a summarizer then truncated. Nothing errored. Every component 'worked'. Without a stitched trace showing exactly what crossed each boundary, you end up re-running the pipeline and staring at prompts, which is archaeology, not debugging.

The event schema that answers real questions

Every hop emits a structured event with: trace ID for the whole run, span ID and parent span ID to form the tree, agent name, event type — llm_call, tool_call, handoff, validation_failure, retry — model used, input and output token counts, latency, status, and the payload that crossed the boundary. Handoff events are the ones teams skip and the ones that matter most, because handoffs are where multi-agent systems actually break.

Payloads need a two-tier policy. Truncated payloads go in the event stream for fast scanning; full prompts and responses go to durable storage keyed by span ID, with the retention limits and access controls your data policy requires — traces contain whatever your users typed. Storing only truncated payloads to save money is false economy: the truncated portion is reliably where the bug lives.

Propagating trace context without threading arguments

Passing trace_id and parent_span_id as function arguments through every call rots fast — someone forgets a parameter and the tree silently fractures. Python's contextvars solves this cleanly: set the trace ID once when a run starts, and every span created anywhere in that async task inherits it automatically. A decorator on each agent function opens a span, captures its parent, times execution, and emits the event even on exceptions.

This is also exactly the model OpenTelemetry uses, so graduating from homegrown JSON events to OTel spans with LLM-specific attributes is a mechanical change later. The decorator below is the minimal version I start systems with.

import contextvars
import json
import time
from functools import wraps
from uuid import uuid4

trace_id = contextvars.ContextVar("trace_id", default="")
parent_span = contextvars.ContextVar("parent_span", default="")

def traced(agent: str, event: str = "agent_step"):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            span = uuid4().hex[:12]
            parent = parent_span.get()
            token = parent_span.set(span)
            start = time.monotonic()
            status = "ok"
            try:
                return fn(*args, **kwargs)
            except Exception:
                status = "error"
                raise
            finally:
                parent_span.reset(token)
                emit({
                    "trace_id": trace_id.get(),
                    "span_id": span,
                    "parent_span_id": parent,
                    "agent": agent,
                    "event": event,
                    "status": status,
                    "duration_ms": round((time.monotonic() - start) * 1000),
                })
        return wrapper
    return decorator

def emit(record: dict) -> None:
    print(json.dumps(record))  # swap for your log pipeline

@traced("researcher")
def research(question: str) -> str:
    ...

Visualization: waterfall first, conversation second

Two views cover nearly every debugging session. The waterfall — spans as horizontal bars, nested by parent, laid out on a time axis — answers the structural questions in seconds: where the time went, whether the fan-out actually ran in parallel or accidentally serialized, which agent retried, where tokens concentrated. I annotate each bar with token counts, because a span that is fast but consumed an enormous context is a different problem than a slow one.

From the waterfall you drill into a single span's conversation view: the exact system prompt, input payload, and output for that one call. The mistake is offering only the conversation view — a flat list of LLM calls, which is where most homegrown tooling stops. Flat lists hide structure, and structure is what multi-agent debugging is about. An OpenTelemetry backend or any of the current LLM observability platforms will render the waterfall for you; what they cannot do is invent the handoff events you never emitted.

Replay beats re-run, and where to look first

Because every span's exact input is stored, I can re-execute one agent step against its recorded input without running the whole pipeline — change the prompt, re-run the single span, compare outputs. This turns a twenty-minute full-pipeline reproduction into seconds of iteration on the actual failing step. It is the same reason checkpointed state matters: recorded inputs make an otherwise non-deterministic system approximately reproducible where it counts.

When a run goes wrong, my reading order is fixed. First the handoff payloads around the failing region — most 'model got it wrong' reports are actually 'model was handed garbage'. Then validation failures and retries, which mark where the system already knew something was off. Then token counts, hunting for truncation and context bloat. Only after all that do I read model outputs closely — genuine model failures are real, but in multi-agent systems they are the minority of incidents I see.

When to hire senior help

Multi-agent orchestration is one of the least commoditized skills in AI engineering, and teams that succeed usually include someone who has debugged coordination failures in production. Get senior review before committing to an orchestrator-worker design, because architectural mistakes at this layer are expensive to unwind after launch. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — Multi-Agent Architectures projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Defaulting to multi-agent when a single agent with good tools would do, since the roughly 15x token multiplier only pays off when subtasks are genuinely parallel and high value
• ✕Letting subagents share full conversation history instead of scoped task briefs, causing context bloat, contradictory actions, and coordination failures
• ✕Shipping without a verification layer, so errors propagate through agent chains unchecked; task verification failures are one of the three MAST failure categories
• ✕Skipping per-agent trace observability, which makes it impossible to identify which agent in the chain caused a bad final output

Frequently asked questions

Sources

• ByteByteGo
• The AI Engineer
• arXiv
• Salesforce
• OnAbout AI