Building Knowledge Base Chatbots for SaaS Support

Key facts, with sources

• Klarna's AI assistant handled 2.3 million conversations in its first month, two-thirds of the company's customer service chats, doing the equivalent work of 700 full-time agents. (Klarna)
• Klarna's assistant cut average resolution time from 11 minutes to under 2 minutes, reduced repeat inquiries by 25%, and was projected to drive a $40 million profit improvement in 2024. (OpenAI)
• Gartner predicts agentic AI will autonomously resolve 80% of common customer service issues without human intervention by 2029, cutting operational costs by 30%. (Gartner)
• Independent aggregation of enterprise support data puts median tier-1 ticket deflection at 41.2% with a top quartile of 58.7%, well below vendor-advertised rates of 70% to 80%. (Digital Applied)
• Zendesk reports that 70% of CX leaders plan to integrate generative AI into many customer touchpoints within two years. (Zendesk)

How it works, end to end

There are two phases. Ingestion, done ahead of time: your documents are split into passages (chunks), each chunk is converted to an embedding, and the embeddings are stored in a search index. Query time, on every question: the question is embedded, the index returns the top handful of most relevant chunks, those chunks are placed into the prompt, and the model answers from them and cites which sources it used.

That citation step is not decoration — it is what lets a user (and you) verify the answer against the real source, and it is a large part of why people trust a RAG bot over a model answering from memory.

Accuracy comes from retrieval, not the model

The most expensive misconception in this space is that a smarter or bigger model fixes a bad chatbot. It does not. If the retrieval step does not surface the passage that contains the answer, the model never sees it and cannot use it — a better model just writes a more fluent wrong answer.

So the accuracy work is almost all in retrieval: sensible chunking that keeps related information together, hybrid search that combines keyword matching with semantic embeddings so exact terms and meaning both count, and re-ranking the retrieved candidates so the best passage is actually in front of the model. Spend your effort there before you spend it on the model.

Grounding: the anti-hallucination rule

Grounding is the instruction that turns a plausible-sounding bot into a trustworthy one: answer only from the retrieved context, cite the source, and if the context does not contain the answer, say so plainly instead of guessing. A support bot that admits 'I don't have that in the docs' is far more valuable than one that confidently invents a policy that does not exist.

SYSTEM = (
    "Answer ONLY using the numbered sources below. "
    "Cite the sources you used like [1], [2]. "
    "If the sources do not contain the answer, say "
    "'I don't have that in our documentation' and stop. "
    "Never use outside knowledge."
)

def build_prompt(question, chunks):
    sources = "\n".join(f"[{i+1}] {c.text}" for i, c in enumerate(chunks))
    return [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"Sources:\n{sources}\n\nQuestion: {question}"},
    ]

Keeping it current and measuring it

The big operational advantage of RAG over fine-tuning is that updating what the bot knows is just updating a document and re-indexing it — no retraining. Wire your ingestion to re-index when content changes and the bot stays current on its own.

And measure it, because 'it seems to work' does not survive contact with real users. Build an evaluation set of actual questions with their correct source and answer, then track answer accuracy, whether the citations are right, and how often the bot correctly says 'I don't know.' That last number matters: a bot that never says it is unsure is a bot that is hallucinating.

What it costs to build and run

The build cost is the pipeline (ingestion, retrieval, grounding), an evaluation harness, and whatever chat surface your users need. Running cost is modest and predictable: embedding documents is cheap and mostly one-time per document, and each question costs one small retrieval plus one model answer. It scales with question volume, not with the size of your knowledge base, which is what makes RAG economical even over large document sets.

Where teams bring in help is the accuracy and grounding work — getting retrieval good enough and the refusal behavior right so the bot is trustworthy in front of customers. That is the part that separates a weekend demo from something support will actually stand behind.

When to hire senior help

Bring in senior help when the bot is live but resolution has plateaued or wrong-answer complaints are rising, since fixing that requires retrieval evaluation and knowledge base restructuring rather than prompt tweaks. Senior review before launch is also warranted for customer-facing bots in regulated industries, where a hallucinated policy answer creates legal exposure. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships RAG — Knowledge Base Chatbots projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Pointing the bot at a stale help center, which exposes every outdated and contradictory article to customers at scale.
• ✕Optimizing for deflection rate alone, which rewards the bot for making customers give up rather than actually resolving their issue.
• ✕Launching without a clean escalation path, so edge cases loop endlessly instead of handing off to a human with full conversation context.
• ✕Taking vendor-reported 70% to 80% resolution claims at face value when independently aggregated medians for tier-1 deflection sit near 41%.

Frequently asked questions

Sources

• Klarna
• OpenAI
• Gartner
• Digital Applied
• Zendesk