Agent Guardrails: Preventing Hallucination in Production

Key facts, with sources

• LangChain's State of Agent Engineering survey of 1,340 practitioners found 57.3 percent of organizations have agents running in production, with another 30.4 percent actively developing them. (LangChain)
• The same LangChain survey found 89 percent of organizations have implemented observability for their agents but only 52 percent do systematic evaluation. (LangChain)
• Deloitte predicts 25 percent of companies using generative AI launched agentic AI pilots in 2025, growing to 50 percent by 2027. (Deloitte Insights)
• By December 2025 the Model Context Protocol had over 97 million monthly SDK downloads and more than 10,000 active MCP servers in production use. (Pento)
• PwC's AI agent survey found 79 percent of companies report AI agents are already being adopted, and 66 percent of adopters say agents deliver measurable value through increased productivity. (PwC)
• In December 2025 Anthropic donated the Model Context Protocol to the Agentic AI Foundation under the Linux Foundation, co-founded with Block and OpenAI, making the agent connector layer vendor-neutral. (Anthropic)

Agents hallucinate actions, not just facts

Agent hallucination shows up in three distinct modes, and each needs its own defense. Fabricated identifiers: the model invents an order number, account ID, or file path shaped like the real ones it has seen. Phantom work: the model reports completing steps it never executed — 'I've updated the ticket and notified the customer' with no corresponding tool calls. Unsupported synthesis: retrieval returned nothing useful, and the model produced a fluent answer anyway.

All three get more likely deep into long trajectories, when the context is crowded and early instructions have weakened. That's why prompt-only mitigation degrades exactly when you need it most, and why the checks below live in code that doesn't tire.

Trace every claim to a tool result

The strongest single guardrail I ship is a grounding check at the response boundary: extract the verifiable entities from the draft answer — IDs, amounts, dates, names — and confirm each one actually appeared in a tool result from this trajectory. Anything the model cites that no tool returned is a fabrication candidate, and the response gets blocked and retried with an explicit correction instead of shipped.

This check is cheap, deterministic, and catches the most damaging failure class: confident references to records that don't exist. The retry prompt matters too — naming the specific ungrounded values gives the model something concrete to fix, and in my experience one corrected retry resolves the majority of blocks.

import re

ID_PATTERN = re.compile(r"\b(?:ord|inv|cus)_[A-Za-z0-9]+\b")

def ungrounded_ids(draft: str, tool_results: list[str]) -> set[str]:
    """IDs the model cited that never appeared in any tool result."""
    cited = set(ID_PATTERN.findall(draft))
    seen: set[str] = set()
    for result in tool_results:
        seen.update(ID_PATTERN.findall(result))
    return cited - seen

leaks = ungrounded_ids(draft_answer, trajectory.tool_result_texts)
if leaks:
    retry_with_correction(
        f"These IDs do not appear in any tool result: {sorted(leaks)}. "
        "Remove them or re-verify with the appropriate lookup tool."
    )

Validate at the boundary, not in the prompt

Whenever the agent's output feeds another system, I treat it as untrusted input and validate structurally. Structured output modes enforce the shape; strict schemas with enums enforce the vocabulary — a status field constrained to a fixed set can't drift into invented states. But shape-valid isn't true: before acting on a model-provided ID, the harness checks it exists in the database and belongs to the tenant in scope.

The framing I use with teams: the database is the source of truth, and model output is a hypothesis about it. Every write path gets an existence check between hypothesis and action. It's one query per mutation, and it converts 'the agent deleted the wrong record' from an incident class into a validation-error log line.

Verify claimed work against the execution log

Phantom work is the sneakiest mode because the final summary reads perfectly: tasks completed, customer notified, records updated. The defense is mechanical — before delivering a summary, diff its claims against the trajectory's actual tool calls. If the summary says an email was sent, an email-send call with a success result must exist in the log. Claims without matching executions get the response blocked, or flagged and rewritten to reflect only verified work.

A lightweight version compares the action verbs in the summary against the set of tools that ran; a stricter one has the model emit claims as structured references to tool-call IDs, making verification a join instead of language parsing. Either way, the pattern I've seen — confident completion reports late in long, messy trajectories — stops reaching users.

Make refusal a success path

An agent optimized to always produce an answer learns to fabricate when it can't find one. The countermeasure is designing an explicit third outcome besides success and error: 'I couldn't verify this — escalating with what I found.' The harness treats that as a valid completion that routes to a human with the gathered context, and — critically — your metrics treat it as quality, not failure. If escalation counts against the agent's success rate, you're training your own system to bluff.

Close the loop with monitoring: sample completed trajectories weekly and audit groundedness — did the sources actually support the answer? Hallucination rate is a drifting quantity; prompt edits, model upgrades, and new tools all move it, and only sampled audits catch the slow slide.

When to hire senior help

Bring in senior help when the agent must touch production systems or customer data, because integration, security, and reliability are where inexperienced builds fail rather than model quality. If a pilot is stuck at the demo stage, an experienced engineer adding evals and guardrails is usually faster and cheaper than rebuilding from scratch. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — AI Agent Development projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Shipping agents with logging but no evals, so teams can see traces but never measure task success rates and regressions ship silently
• ✕Giving one agent dozens of tools instead of a focused toolset, which degrades tool-selection accuracy and inflates token costs
• ✕Hand-rolling custom integration glue for every data source instead of using MCP, which is now the vendor-neutral standard backed by Anthropic, OpenAI, and the Linux Foundation
• ✕Validating only on happy-path demo prompts and skipping failure-mode testing, a core reason roughly 88 percent of agent pilots never reach production

Frequently asked questions

Sources

• LangChain
• Deloitte Insights
• Pento
• PwC
• Anthropic