Agentic RAG: Retrieval-Augmented Agents

Key facts, with sources

• Gartner predicts over 40 percent of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. (Gartner)
• Gartner predicts 33 percent of enterprise software applications will include agentic AI by 2028, up from less than 1 percent in 2024. (Gartner)
• Gartner estimates only about 130 of the thousands of vendors claiming to sell agentic AI are real, with the rest engaged in agent washing of existing chatbots and RPA products. (MarTech)
• McKinsey's State of AI 2025 found 23 percent of organizations are scaling an agentic AI system somewhere in the enterprise and another 39 percent have begun experimenting with agents. (McKinsey)
• Gartner forecasts 40 percent of enterprise applications will embed task-specific AI agents by the end of 2026, up from under 5 percent in 2025. (Joget)

Classic RAG vs agentic RAG in one picture

Classic RAG is a straight line: query → embed → vector search → stuff top-k chunks into the prompt → generate. Every question, no matter how simple or complex, gets the same single retrieval pass, and the pipeline's decisions (how many chunks, from which index) are hardcoded.

Agentic RAG bends that line into a loop. The model sees the question and a set of retrieval tools, then drives: maybe it answers directly from knowledge it's confident about, maybe it searches once, maybe it decomposes the question, searches three times with different phrasings, notices the results conflict, and searches again to resolve the conflict. The pipeline's control flow moves from your code into the model's judgment — which is exactly why it handles the questions that break fixed pipelines, and why it needs evals and tracing before you can trust it.

The five decisions the agent makes

Concretely, agentic RAG delegates five choices. Search or answer: skip retrieval for questions the model can already answer, saving latency and cost. Query rewriting: turn “why did it break after the update” into search terms that actually match documents, often issuing several reformulations. Source routing: pick the right tool per question — product docs index, ticket history, SQL database, web search — instead of one index for everything. Multi-hop composition: answer “which customers were affected by the bug fixed in 2.14” by first finding the bug, then querying affected accounts — no single retrieval can do that. And evidence verification: judge whether retrieved passages truly answer the question, retrying with a different strategy when they don't, rather than generating a confident answer from irrelevant context.

A minimal agentic retrieval loop

Stripped of framework ceremony, the pattern is a tool-calling loop with a search budget. This sketch is provider-agnostic — any LLM API with tool calling supports it. The two lines that matter in production: the max-steps cap, which prevents runaway retrieval loops, and recording every step for tracing, without which failures are undebuggable.

MAX_STEPS = 5

TOOLS = [
    {
        "name": "search_docs",
        "description": "Hybrid BM25 + vector search over product documentation. Returns top passages with scores.",
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
    # ...search_tickets, query_db, etc.
]


def answer(question: str) -> dict:
    messages = [{"role": "user", "content": question}]
    trace = []

    for step in range(MAX_STEPS):
        response = llm.generate(messages=messages, tools=TOOLS)

        if response.tool_call is None:
            # Model judged the evidence sufficient — final answer
            return {"answer": response.text, "trace": trace}

        result = run_tool(response.tool_call)  # your retrieval code
        trace.append({"step": step, "tool": response.tool_call, "result_summary": summarize(result)})

        messages.append({"role": "assistant", "tool_call": response.tool_call})
        messages.append({"role": "tool", "content": result})

    # Budget exhausted: answer from what we have, flagged as partial
    return {"answer": generate_best_effort(messages), "trace": trace, "partial": True}

When classic RAG is enough — and cheaper

Most questions users actually ask are single-hop lookups over one corpus: what does this setting do, what's the refund policy. For that workload, a well-tuned classic pipeline — hybrid BM25 plus vector search, sensible chunking, a reranker — answers in one model call at a fraction of the cost, and closes much of the quality gap that people attribute to missing agency. The honest engineering order: build classic RAG with hybrid search first, measure retrieval quality on real questions, and go agentic only for the question types where the eval shows the fixed pipeline failing — typically multi-source routing and multi-hop composition. Agentic RAG layered on top of bad chunking is expense, not improvement.

Cost, latency, and how to evaluate it

Budget for multiplication: each loop step is a model call, and context grows as tool results accumulate, so a three-search answer can cost several times a classic RAG answer and take seconds longer. Mitigations that work in production: a cheap/fast model for the search-and-route steps with a stronger model only for final synthesis, prompt caching for the static tool definitions and system prompt, and hard step budgets per question tier.

Evaluate the two layers separately. Retrieval: did the loop find the passages that contain the answer (measure hit rate against a labeled set)? Generation: is the answer faithful to what was retrieved? Add loop-health metrics — steps per question, retrieval retries, budget exhaustions — because a quality regression usually shows up there first. Trace every step from day one; an agentic system without tracing cannot be debugged, only vibes-checked.

When to hire senior help

Senior help is most valuable at the architecture stage, deciding what to automate, where approval gates belong, and how business value will be measured, before any code is written. It is also worth bringing in when a stalled pilot needs risk controls and evaluation rigor to pass security and compliance review. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — Agentic AI Systems projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Buying agent-washed products, since Gartner estimates only around 130 of thousands of self-described agentic AI vendors are genuine rather than rebranded chatbots or RPA
• ✕Deploying autonomy before defining risk controls and human-approval gates, one of the three causes Gartner cites for the 40 percent of projects it expects to be canceled
• ✕Measuring activity like tasks attempted instead of business value, leaving the project unable to justify escalating costs at renewal time
• ✕Wrapping agents around existing processes instead of redesigning the workflow, when McKinsey finds workflow redesign is the single biggest driver of EBIT impact from gen AI

Frequently asked questions

Sources

• Gartner
• MarTech
• McKinsey
• Joget