AI — Multi-Agent Architectures

Scaling Multi-Agent Systems on Kubernetes

Direct answer

Run each agent role as its own Deployment consuming work from a queue, autoscale on queue depth with KEDA instead of CPU, and keep all state in Postgres or Redis so pods stay disposable. Agent workers are I/O-bound — they spend most of their wall-clock time waiting on LLM responses — so per-pod concurrency and provider rate limits, not CPU, determine real capacity.

Multi-agent systems scale differently from normal web services, and the teams that treat them the same end up with clusters that autoscale on exactly the wrong signal. This is the Kubernetes shape I deploy for agent pipelines: what to scale on, where state lives, and why the provider's rate limit — not your cluster — is usually the actual ceiling.

Key facts, with sources

Anthropic reported that a multi-agent research system using an Opus lead agent with Sonnet subagents outperformed a single-agent Opus baseline by 90.2 percent on its internal research eval. (ByteByteGo)
Anthropic's multi-agent research system used about 15x more tokens than a normal chat interaction, and token usage alone explained roughly 80 percent of performance variance. (The AI Engineer)
The MAST research taxonomy identified 14 distinct failure modes across 7 popular multi-agent frameworks including AutoGen, ChatDev, and CrewAI, grouped into system design flaws, inter-agent misalignment, and task verification failures. (arXiv)
Salesforce research found organizations run an average of 12 AI agents and projects multi-agent adoption to surge 67 percent within two years as enterprises move toward orchestration. (Salesforce)
Multi-agent orchestration with three or more agents represents about 22 percent of enterprise agent deployments in 2026, projected to reach roughly 45 to 50 percent by 2027. (OnAbout AI)

Agent workloads are I/O-bound, and that changes the sizing math

An agent worker spends the overwhelming majority of each task waiting: for LLM tokens to stream back, for tool calls, for retrieval queries. CPU sits nearly idle throughout. That single fact invalidates the default Kubernetes reflexes — CPU-based HPA never triggers because CPU never climbs, while your queue backlog quietly grows into an incident.

The correct response is concurrency before replicas. An async Python worker can hold many agent tasks in flight per pod because each one is mostly awaiting network responses, so a handful of pods with high per-pod concurrency beats a fleet of pods each running one task. Memory becomes the binding pod resource — each in-flight task holds its context and intermediate state — so I size memory requests from observed per-task footprint and treat CPU requests as an afterthought.

One role, one Deployment, a queue in between

Each agent role — researcher, extractor, writer, reviewer — gets its own Deployment, its own queue, and its own scaling behavior. Roles have wildly different profiles: a research role might fan out to many long tasks per run while a synthesis role runs once per pipeline; a review role might need the expensive model while extraction runs on a cheap one. Packing them into one worker binary means scaling all of them to satisfy the hungriest.

The queue between stages is what converts a fragile chain into a resilient system. Producers do not care whether consumers are alive right now; a burst of pipeline runs piles up as messages instead of timeouts; and a crashed pod's unacknowledged task is simply redelivered to a healthy one. The orchestrator publishes tasks, workers consume, and no agent holds a synchronous connection to another.

Autoscale on queue depth with KEDA

Queue depth is the honest demand signal, and KEDA is the standard way to wire it to replica counts. The target in the trigger is messages per replica: if the queue holds two hundred messages and the target is twenty, KEDA drives the Deployment toward ten replicas. Scale-to-zero matters more than it sounds for agent systems — many pipelines are bursty and idle most of the day, and idle GPU-adjacent infrastructure is pure burn.

Set the cooldown long enough that a brief queue dip does not tear down pods mid-task, and pair scaling with honest queue semantics: workers acknowledge messages only after checkpointing the completed step.

KEDA ScaledObject scaling a worker role on queue depth

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: researcher-agents
spec:
  scaleTargetRef:
    name: researcher-worker   # Deployment for this agent role
  minReplicaCount: 0          # scale to zero when idle
  maxReplicaCount: 20
  cooldownPeriod: 120
  triggers:
    - type: rabbitmq
      metadata:
        queueName: researcher-tasks
        mode: QueueLength
        value: "20"           # target messages per replica
        hostFromEnv: RABBITMQ_URL

Externalize all state; make every step idempotent

Autoscaling means pods die mid-task as a matter of routine, not exception — scale-down, node drains, spot interruptions. The system survives this only if pods hold nothing that matters. Run state, checkpoints, and conversation context live in Postgres or Redis, keyed by run and step; a worker picks up a task, loads what it needs, does one step, persists the result, then acknowledges the message.

Idempotency is the twin requirement, because queue redelivery guarantees some steps will execute twice. Every task carries a deterministic key, and workers check for an existing completed result before doing work — the second delivery becomes a cheap no-op instead of a duplicate LLM spend or, worse, a duplicated side effect like a sent email. I treat a non-idempotent agent step as a bug even when it has never yet misfired.

The provider rate limit is the real ceiling

Here is the trap: KEDA will happily scale you to twenty pods, each running many concurrent tasks, and every one of them hits the same LLM provider account. Past the provider's requests-per-minute and tokens-per-minute limits, additional replicas manufacture nothing but 429 responses and retry storms. Your true capacity is the provider quota, and the cluster must respect it as a shared, global resource.

I enforce this with a central rate limiter in Redis — a token bucket for requests and a budget for tokens-per-minute — that every worker acquires from before calling the model. When the bucket is empty, workers block, which produces natural backpressure: tasks wait in the queue, where they are durable and observable, instead of dying in retry loops. Cap maxReplicaCount at what the quota can actually feed; anything higher is decorative.

Shutdown, probes, and the observability that matters

Agent steps run long, and default Kubernetes settings assume they do not. Set terminationGracePeriodSeconds generously so a pod receiving SIGTERM can finish its in-flight step and checkpoint before dying; the message-ack pattern makes forced kills safe, but graceful completion avoids paying for the same LLM call twice. Liveness probes must check the process, not task progress — a probe that assumes 'busy for two minutes means hung' will kill perfectly healthy workers mid-generation.

Dashboards per role, not per cluster: queue depth and age of oldest message, task latency, tokens consumed, 429 rates, and dead-letter counts. Queue age is the metric I alert on first — depth can look modest while age reveals a stuck consumer. Cluster-level CPU graphs, the default lens for everything else you run, tell you almost nothing here.

When to hire senior help

Multi-agent orchestration is one of the least commoditized skills in AI engineering, and teams that succeed usually include someone who has debugged coordination failures in production. Get senior review before committing to an orchestrator-worker design, because architectural mistakes at this layer are expensive to unwind after launch. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — Multi-Agent Architectures projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

✕Defaulting to multi-agent when a single agent with good tools would do, since the roughly 15x token multiplier only pays off when subtasks are genuinely parallel and high value
✕Letting subagents share full conversation history instead of scoped task briefs, causing context bloat, contradictory actions, and coordination failures
✕Shipping without a verification layer, so errors propagate through agent chains unchecked; task verification failures are one of the three MAST failure categories
✕Skipping per-agent trace observability, which makes it impossible to identify which agent in the chain caused a bad final output

Frequently asked questions

How do you autoscale AI agents on Kubernetes?

Scale on queue depth, not CPU. Give each agent role its own Deployment consuming from its own queue, and use KEDA with a messages-per-replica target to drive replica counts, including scale-to-zero for bursty pipelines. Agent workers are I/O-bound waiting on LLM responses, so CPU-based autoscaling never triggers while backlogs grow — queue depth is the honest demand signal.

Why not just scale agent workers on CPU like a normal service?

Because agent workers barely use CPU — they spend most of each task awaiting LLM tokens, tool calls, and retrieval over the network. Under heavy load, CPU stays flat while the queue backlog grows, so a CPU-based HPA sits idle through an incident. Concurrency per pod and queue-depth scaling reflect how these workloads actually behave; CPU metrics do not.

How do you handle LLM rate limits when scaling out agent workers?

Treat the provider quota as a single cluster-wide resource. Put a central rate limiter in Redis — a token bucket covering requests and token throughput — that every worker must acquire from before calling the model, so excess demand waits durably in the queue instead of generating 429 retry storms. Also cap maximum replicas at what the quota can feed; scaling beyond it adds nothing.

When does a multi-agent architecture beat a single agent?

When the work decomposes into independent subtasks that can run in parallel, such as broad research, fan-out analysis, or reviewing many files at once; Anthropic measured a 90.2 percent improvement on that shape of work. Sequential, tightly coupled tasks usually do better with one agent and good tools.

Why do multi-agent systems fail?

Research across 7 frameworks found failures cluster into system design flaws, inter-agent misalignment, and missing verification rather than raw model weakness. An orchestrator-worker pattern with explicit task specifications and output checks addresses most of these failure modes.

How much more expensive is a multi-agent system?

Anthropic reports about 15x the tokens of a chat interaction for its multi-agent research system, so cost per task rises sharply. Teams mitigate this with cheaper models for subagents, prompt caching, and hard caps on subagent count and loop length.

Bottom line: Dhairya Senjaliya ships AI — Multi-Agent Architectures projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.