Building Agentic Copilots for Product Teams

Key facts, with sources

• Gartner predicts over 40 percent of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. (Gartner)
• Gartner predicts 33 percent of enterprise software applications will include agentic AI by 2028, up from less than 1 percent in 2024. (Gartner)
• Gartner estimates only about 130 of the thousands of vendors claiming to sell agentic AI are real, with the rest engaged in agent washing of existing chatbots and RPA products. (MarTech)
• McKinsey's State of AI 2025 found 23 percent of organizations are scaling an agentic AI system somewhere in the enterprise and another 39 percent have begun experimenting with agents. (McKinsey)
• Gartner forecasts 40 percent of enterprise applications will embed task-specific AI agents by the end of 2026, up from under 5 percent in 2025. (Joget)

Pick workflows where wrong answers are visible

The first copilot workflows should be ones where a wrong answer is immediately checkable. "What was activation last week for users from the new onboarding flow" has a right answer in your warehouse; if the copilot gets it wrong, someone notices in seconds, and you can fix the tool or the prompt. Feedback summarization is similar — the source tickets are a click away.

What I avoid early: open-ended strategy questions like "should we build X." The copilot will produce something confident and plausible, nobody can verify it, and trust erodes the first time it is visibly wrong on facts. Copilots earn the right to opine by first being reliably correct about data.

A small, prescriptive tool surface beats a big one

Every tool you add costs prompt space and adds a way to be wrong. I keep product-team copilots to a handful of tools: run a saved analytics query, search tickets, search product docs, and draft a document into a pending state. The important craft is in the descriptions — the model decides when to call a tool almost entirely from its description, so I write them prescriptively, stating when to use the tool, not just what it does.

Enums beat free text wherever possible. A query_name enum over your saved queries means the copilot cannot hallucinate SQL against your warehouse; it can only choose from vetted queries with known semantics. Adding a capability then becomes a data change — one more vetted query — rather than a prompt engineering session.

TOOLS = [
    {
        "name": "query_metrics",
        "description": (
            "Run a vetted analytics query. Call this whenever the user asks "
            "about activation, retention, or funnel numbers. Never answer "
            "metric questions from memory."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "query_name": {
                    "type": "string",
                    "enum": [
                        "activation_funnel",
                        "retention_cohorts",
                        "feature_adoption",
                    ],
                },
                "date_range_days": {"type": "integer"},
            },
            "required": ["query_name", "date_range_days"],
        },
    },
    {
        "name": "search_feedback",
        "description": (
            "Search user feedback and support tickets. Call this before "
            "summarizing what users think about any feature."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
]

Teach it your team's vocabulary

Product teams speak in loaded terms: "activation" means something specific, "enterprise accounts" has an exact definition, and quarters may be fiscal rather than calendar. If the copilot does not know your definitions, it will silently apply generic ones and produce numbers that look right and are wrong. I maintain a short glossary — metric definitions, segment definitions, product-area names — and put it in the system prompt, marked for prompt caching since it changes rarely.

The glossary usually already half-exists in a metrics catalog or an onboarding doc. Curating it into the copilot's context is a day of work and does more for answer quality than any amount of prompt tuning, because it removes an entire class of confidently wrong answers.

Ground every claim or say nothing

The rule I enforce in the system prompt and verify in evals: numeric claims and factual statements about users must trace to a tool call in the same conversation, and the response should say which source it used. If no tool can answer the question, the copilot says so and suggests what data would be needed — it does not fill the gap from training data.

This is testable. My eval set includes questions the tools genuinely cannot answer, and the pass condition is a refusal. It also includes questions where the tool result contradicts what a model would guess, and the pass condition is that the tool result wins. Both failure modes show up regularly in copilots that were never evaluated.

Adoption: read-only first, measure deflection

Ship it where the team already talks — a bot in the team chat or a panel in the product — not as a separate destination they have to remember. Start read-only: answering and drafting only, with drafts landing in a pending state a human publishes. Measure two things from day one: how many questions per week it answers that would otherwise have gone to an analyst or PM, and what fraction of drafts get used with light edits versus discarded.

Those two numbers tell you when to expand. High deflection plus high draft acceptance is the evidence for letting it file tickets or update docs directly. Low acceptance tells you which tool or context is weak before you have burned the team's trust.

When to hire senior help

Senior help is most valuable at the architecture stage, deciding what to automate, where approval gates belong, and how business value will be measured, before any code is written. It is also worth bringing in when a stalled pilot needs risk controls and evaluation rigor to pass security and compliance review. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — Agentic AI Systems projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Buying agent-washed products, since Gartner estimates only around 130 of thousands of self-described agentic AI vendors are genuine rather than rebranded chatbots or RPA
• ✕Deploying autonomy before defining risk controls and human-approval gates, one of the three causes Gartner cites for the 40 percent of projects it expects to be canceled
• ✕Measuring activity like tasks attempted instead of business value, leaving the project unable to justify escalating costs at renewal time
• ✕Wrapping agents around existing processes instead of redesigning the workflow, when McKinsey finds workflow redesign is the single biggest driver of EBIT impact from gen AI

Frequently asked questions

Sources

• Gartner
• MarTech
• McKinsey
• Joget