AI Agent Memory: Short-Term vs Long-Term Patterns

Key facts, with sources

• LangChain's State of Agent Engineering survey of 1,340 practitioners found 57.3 percent of organizations have agents running in production, with another 30.4 percent actively developing them. (LangChain)
• The same LangChain survey found 89 percent of organizations have implemented observability for their agents but only 52 percent do systematic evaluation. (LangChain)
• Deloitte predicts 25 percent of companies using generative AI launched agentic AI pilots in 2025, growing to 50 percent by 2027. (Deloitte Insights)
• By December 2025 the Model Context Protocol had over 97 million monthly SDK downloads and more than 10,000 active MCP servers in production use. (Pento)
• PwC's AI agent survey found 79 percent of companies report AI agents are already being adopted, and 66 percent of adopters say agents deliver measurable value through increased productivity. (PwC)
• In December 2025 Anthropic donated the Model Context Protocol to the Agentic AI Foundation under the Linux Foundation, co-founded with Block and OpenAI, making the agent connector layer vendor-neutral. (Anthropic)

Short-term memory is a budget, not a bucket

Everything in the context window is re-sent and re-billed on every model call, so short-term memory is fundamentally a cost and attention budget. The question for each item isn't 'might this be useful?' but 'does the model need this to decide the next step?' Tool results are the biggest offenders — a single verbose API response can outweigh the entire conversation, and it gets re-read on every subsequent iteration.

I treat the window as a working set: the task statement, recent turns, and the tool results still relevant to the current decision. Everything else is a candidate for summarization or removal. Agents that never prune don't just cost more — they get worse, because stale detail competes with the current step for the model's attention.

Compaction: summarize the old, keep the recent verbatim

My default short-term pattern is a rolling compaction: once the history crosses a threshold, older turns get summarized into a dense recap while the most recent window stays verbatim. The recap preserves decisions, constraints, and open questions; it drops raw tool payloads and dead-end explorations. Recent turns stay untouched because that's where precise wording still matters.

Two implementation details bite people. First, watch message role alternation when you splice a summary in front of a kept window — re-anchor so the sequence stays valid. Second, compact at natural boundaries (after a subtask completes) rather than mid-reasoning, or the model loses the thread it was actively pulling.

def compact(messages: list[dict], keep_recent: int = 8) -> list[dict]:
    """Summarize old turns; keep the recent window verbatim."""
    if len(messages) <= keep_recent:
        return messages
    old, recent = messages[:-keep_recent], messages[-keep_recent:]
    summary = summarize(old)  # one cheap model call producing a dense recap
    recap = {
        "role": "user",
        "content": f"Summary of the conversation so far:\n{summary}",
    }
    # note: re-anchor roles if `recent` starts with an assistant turn
    return [recap] + recent

Long-term memory: structured records beat a vector dump

For durable memory, my default is embarrassingly simple: structured notes in files or database rows — one fact per entry, with provenance and a timestamp. 'Customer prefers invoices grouped by project (stated 2026-03-14)' is retrievable, auditable, and correctable. A vector store full of raw conversation chunks is none of those things: you can't tell why a memory exists, whether it's still true, or how to delete it cleanly.

Vector search earns its complexity when the corpus is genuinely large — thousands of documents, semantic lookup across unstructured text. Most agent memory needs are hundreds of curated notes, where key-based and tag-based retrieval is faster, cheaper, and debuggable. Reach for embeddings when scale demands it, not as the default.

Retrieve on demand, not always-in-context

Long-term memory should enter the context deliberately, not by default. My pattern: at task start, the agent (or a pre-step in the harness) loads only the notes relevant to this user and task type. For anything else, the agent gets a memory-lookup tool it can call when it recognizes a gap — the model is good at knowing when it's missing context if you tell it a memory store exists.

This discipline has a second payoff: keeping the always-present prefix stable and small preserves prompt caching. A memory blob that changes every session sits early in the prompt and silently invalidates the cache on every request — I've found that exact bug in more than one cost audit.

Memory hygiene decides whether memory helps or hurts

A wrong memory is worse than no memory — the agent will confidently act on it for months. So writes need as much design as reads. I follow four rules: update existing entries instead of appending contradictions; deduplicate before writing; give entries an expiry or review date where the fact is time-bound; and never store secrets, tokens, or credentials — memories get replayed into future contexts indefinitely.

Build a review path from day one: a way for a human to list, edit, and delete an agent's memories about a user or account. Beyond debugging, this is increasingly a compliance requirement — deletion requests have to reach the memory store too, not just your primary database.

When to hire senior help

Bring in senior help when the agent must touch production systems or customer data, because integration, security, and reliability are where inexperienced builds fail rather than model quality. If a pilot is stuck at the demo stage, an experienced engineer adding evals and guardrails is usually faster and cheaper than rebuilding from scratch. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships AI — AI Agent Development projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Shipping agents with logging but no evals, so teams can see traces but never measure task success rates and regressions ship silently
• ✕Giving one agent dozens of tools instead of a focused toolset, which degrades tool-selection accuracy and inflates token costs
• ✕Hand-rolling custom integration glue for every data source instead of using MCP, which is now the vendor-neutral standard backed by Anthropic, OpenAI, and the Linux Foundation
• ✕Validating only on happy-path demo prompts and skipping failure-mode testing, a core reason roughly 88 percent of agent pilots never reach production

Frequently asked questions

Sources

• LangChain
• Deloitte Insights
• Pento
• PwC
• Anthropic