AI Memory Systems Explained: Context, RAG, and Long-Term Memory
Photo: Tara Winstead / Pexels
When ChatGPT remembers that you prefer bullet points, or Claude recalls a project you set up three weeks ago, it feels like the model "learned" something. It did not. Large language models do not update their own weights between conversations. What actually happened is that a separate storage-and-retrieval system quietly fed relevant text back into the model's prompt before it answered. AI memory is a family of five distinct engineering tricks — a context window, retrieval, saved profiles, tool state, and occasionally fine-tuning — stitched together to fake the experience of a system that remembers you.
Why "AI has memory" is a misleading sentence
A large language model (LLM) is a fixed set of mathematical weights, trained once and then frozen. Ask GPT-4o a question on Monday and again on Friday, and the underlying network is byte-for-byte identical both times unless OpenAI ships a new checkpoint. The model has no built-in mechanism to write new information into itself mid-conversation, the way a human brain forms a new synaptic connection.
So when a chatbot appears to remember something, the actual mechanism is almost always external. Text gets stored somewhere — a database, a file, a running conversation log — and a piece of software decides which stored text to paste back into the model's input before the next response. The model then reads that pasted text the same way it reads your question: as tokens in a prompt, with no distinction between "things I recall" and "things you just told me."
This distinction matters because it explains most of the weird behavior people report: a chatbot confidently misremembering a detail, forgetting something from earlier in a long chat, or repeating a fact you never actually said. Each of those bugs traces back to a specific layer in the memory stack, not to the model "having a bad memory day."
The five layers, one at a time
The context window: memory that lives inside a single conversation
The context window is the total amount of text — measured in tokens, roughly three-quarters of a word each — that a model can process in one pass. Everything the model "knows" during a conversation, including your entire chat history so far, has to fit inside this window. Once a conversation grows past the window's limit, the oldest turns get dropped or summarized to make room.
Context windows vary enormously by model. GPT-4o's standard context sits around 128,000 tokens (roughly 100,000 words). Anthropic's Claude models running inside a Project support a 200,000-token window, which the company describes as roughly 500 pages of text (Anthropic, Claude Projects documentation, reported via Suprmind's 2026 Claude feature roundup). Google has pushed furthest: Gemini 2.0 Flash and Gemini 2.5 Pro ship with a 1,048,576-token context window, and Gemini 1.5 Pro can be extended to 2 million tokens in select workflows (Google AI for Developers, "Long context").
Think of the context window like a whiteboard in a meeting room. Anyone in the room can read anything written on it, but the whiteboard has a fixed size. Write something new near the edge and eventually you have to erase old notes to keep writing. A bigger whiteboard (a longer context window) means fewer things get erased before the meeting ends, but it is still not the same as a filing cabinet down the hall that holds everything from every meeting you have ever had. That filing cabinet is a different layer entirely: retrieval.
Retrieval-augmented generation (RAG): the filing cabinet
Retrieval-Augmented Generation, or RAG, is the technique of searching an external database for relevant text and inserting the results into the prompt before the model generates its answer. It was formalized in a 2020 paper by Patrick Lewis and coauthors at Facebook AI Research (now Meta AI), University College London, and New York University, titled "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020, discussed in Latenode's technical breakdown). The paper's central idea was to combine two kinds of knowledge: parametric knowledge baked into the model's weights during training, and non-parametric knowledge pulled live from an external source like a Wikipedia index.
A RAG pipeline has three moving parts. First, a document gets broken into chunks — a few paragraphs each — and each chunk is converted into an embedding, a list of a few hundred to a few thousand numbers that represents the chunk's meaning as a point in mathematical space. Second, those embeddings get stored in a vector database, a system built specifically to find the nearest neighbors to any given point quickly, even across billions of stored vectors. Third, when a user asks a question, the question itself gets embedded the same way, the vector database returns the chunks closest to it in meaning, and those chunks get pasted into the prompt as extra context.
RAG is why a customer-support chatbot can answer questions about a product manual it was never trained on, and why coding assistants can reference your specific codebase instead of only general programming knowledge. NVIDIA's own explainer describes RAG as connecting an LLM to "an external knowledge base" so the model can cite specifics instead of relying purely on what it memorized during training (NVIDIA, "What Is Retrieval-Augmented Generation aka RAG"). A related way sites make their content easier for AI systems to read is an llms.txt file.
The analogy that holds up best: a context window is what a student can hold in their head during a closed-book exam. RAG is an open-book exam, where the student cannot memorize the whole textbook but can flip to the right page fast enough to answer correctly. The skill being tested is not recall, it is knowing which page to flip to — which is exactly what the embedding-and-similarity-search step does.
Saved memory and user profiles: the sticky note
This is the layer most people mean when they say "ChatGPT remembers me." It is a small, explicitly curated store of facts about a specific user — preferences, ongoing projects, biographical details — that gets consulted and injected into the prompt on every new conversation, regardless of what that conversation is about.
OpenAI's saved-memory feature launched in February 2024 and initially required users to explicitly tell ChatGPT to remember something, such as a travel preference or writing style (OpenAI, "Memory and new controls for ChatGPT"). In April 2025, OpenAI added a second mechanism, internally described as "dreaming," which is a background process that automatically curates memories from a user's chat history without requiring an explicit save command (OpenAI, "Dreaming: Better memory for a more helpful ChatGPT"). The result is two memory stores running in parallel: an explicit list of saved facts the user can review and delete, and an implicit summary of chat-history patterns the model draws on for personalization.
Anthropic shipped a comparable feature in March 2026, rolling out persistent memory to every Claude account, including the free tier. The feature automatically summarizes conversations and carries relevant context forward across sessions, with granular user controls over what gets retained (reporting via Yahoo Tech, "Anthropic brings Claude's memory feature to all users").
Security researcher Simon Willison flagged a real downside of this pattern in May 2025: when a memory system builds a persistent "dossier" of inferred facts about a user, small misfires compound, and users often cannot audit exactly why the model believes something about them (Simon Willison, "I really don't like ChatGPT's new memory dossier"). That is a real engineering trade-off, not a minor complaint: the more a system infers rather than asks, the harder it becomes for the user to correct a wrong assumption.
Agent and tool state: memory of what has already been done
Once a system moves from answering questions to taking multi-step actions — booking a flight, refactoring a codebase, running a research task across a dozen web pages — it needs to track a different kind of memory: what has it already tried, what tools has it called, and what were the results. This is often called agent state or scratchpad memory, and it typically lives in the same context window as the conversation, formatted as a running log of tool calls and outputs.
The MemGPT project, introduced in a 2023 paper by Charles Packer and coauthors at UC Berkeley, tackled this by borrowing an idea from operating systems design: treat the model's context window like fast RAM and an external database like a slow disk, and let the model itself issue explicit commands to move information between the two tiers ("MemGPT: Towards LLMs as Operating Systems," summarized by Leonie Monigatti's paper notes). MemGPT has since evolved into Letta, a production framework where each agent maintains a core memory block plus an archival-memory store backed by PostgreSQL and the pgvector extension, and where similarity search over that archive happens without invoking the LLM at all, keeping retrieval fast and cheap.
Fine-tuning: the rarest and most permanent layer
Fine-tuning means additional training that actually changes a model's weights, making a behavior or fact genuinely part of the model rather than something injected into its prompt. This is the only one of the five layers that is true, permanent memory in the sense a biologist would recognize. It is also the slowest and most expensive to update, which is why almost no consumer product uses fine-tuning for day-to-day personalization. Retraining or fine-tuning a large model to remember that one user prefers formal writing would be enormous overkill; a saved-memory note accomplishes the same thing at a fraction of the cost and can be edited or deleted instantly.
How the layers stack up in a table
| Layer | What it stores | How long it lasts | Where the "memory" lives | Typical example |
|---|---|---|---|---|
| Context window | The current conversation's raw text | Until the conversation ends or the window fills up | Inside the single active prompt | A chatbot referencing something you said two messages ago |
| RAG / retrieval | Chunked, embedded documents in a vector database | Until the source document is deleted or re-indexed | External database, searched fresh each query | A support bot quoting your company's actual return policy |
| Saved memory / profile | Explicit facts and inferred patterns about one user | Until the user deletes them or the provider prunes stale entries | A small structured store tied to a user account | ChatGPT recalling you prefer concise answers |
| Agent / tool state | Logs of prior tool calls, actions, and their results | Length of one task or agent session, sometimes archived | Scratchpad in context, or an archival store like Letta's pgvector layer | A coding agent remembering it already ran the test suite once |
| Fine-tuning | Statistical patterns baked into model weights | Permanent until the model is retrained | Inside the neural network itself | A model that has genuinely learned a new writing style through training |
Why this distinction actually matters for the average user
Understanding which layer is doing the work changes how you should troubleshoot a chatbot that seems to have a bad memory. If it forgets something from ten minutes ago in the same chat, that is a context-window problem, and the fix is usually to keep conversations shorter or summarize periodically. If it cannot answer questions about your own documents, that is a missing RAG pipeline, and the fix is to upload the source material into a system built to index it, such as a Claude Project or a custom retrieval tool. If it "remembers" something wrong about you across many conversations, that is a saved-memory or profile problem, and the fix is to open the memory settings and correct or delete the specific entry, because the model itself has no independent judgment about which stored facts are accurate.
This also explains a common source of user frustration: the assumption that a longer context window means better long-term memory. It does not. A 1-million-token context window, like Gemini's, lets a model process an enormous amount of text in one sitting, which is genuinely useful for analyzing a large codebase or a long video transcript in a single pass. But once that conversation ends, none of it persists unless a separate system saved it somewhere retrievable. Context length is about how much a model can read at once, not about what it keeps.
The privacy implications differ sharply by layer, too. A RAG system searching your company's document store is generally auditable: you can see exactly which chunks were retrieved for a given answer. A saved-memory profile that infers facts about you from conversational patterns is much harder to audit, which is precisely the concern Willison raised about ChatGPT's dossier-style memory. If a product markets "memory" as a feature, it is worth checking whether the company exposes a way to view and edit what has actually been stored, because that visibility is what turns an opaque black box into something a user can trust and correct.
What people get wrong about AI memory
The most common misconception is that a chatbot "learns" from talking to you the way a person does, gradually and continuously. It does not, outside of the rare fine-tuning case. Every appearance of learning within a conversation is really retrieval or context management, reconstructed fresh each time from stored text rather than accumulated as a permanent change to the model.
A second misconception is that bigger context windows solve memory. They solve a different, related problem: how much material a model can consider at once. A system with a tiny context window but a well-built RAG pipeline can often "remember" more useful, accurate information than a system with a huge context window and no retrieval at all, because retrieval actively selects the relevant needle instead of hoping the model notices it buried in a haystack of a million tokens.
A third misconception is that memory features make a model smarter in a general sense. They do not improve reasoning ability or factual accuracy about the world; they only improve continuity and personalization for a specific user or task. A model with excellent memory of your preferences can still make basic factual errors about topics unrelated to your saved history, because memory and general capability are separate systems entirely.
Finally, people sometimes assume all memory systems behave the same way across products. They do not. OpenAI's two-track approach (explicit saved memories plus implicit "dreaming" summarization) behaves differently from Anthropic's single automatic-summarization approach, which behaves differently from an open-source agent framework like Letta that gives developers direct control over what gets archived and how it gets retrieved. The right mental model is not "AI has memory now" but "this specific product uses this specific combination of the five layers above, with these specific limits."
Frequently Asked Questions
Q: Does an AI model actually learn and remember things permanently? A: Only through fine-tuning, which changes the model's underlying weights and is rarely used for everyday personalization. Nearly all consumer-facing "memory" — ChatGPT recalling your preferences, Claude carrying forward a project — works by storing text externally and re-inserting it into the prompt, not by changing the model itself.
Q: What is the difference between a context window and memory? A: A context window is the amount of text a model can process in a single pass, typically 128,000 to 2 million tokens depending on the model. It resets or truncates once a conversation ends. Memory, by contrast, is a separate storage system — saved profiles, a vector database, or an archival store — that persists across conversations and gets fed back into a fresh context window on demand.
Q: How does RAG (retrieval-augmented generation) actually work, step by step? A: Source documents are split into chunks, and each chunk is converted into a numerical embedding representing its meaning. Those embeddings sit in a vector database. When a user asks a question, the question is embedded the same way, the database returns the closest matching chunks, and those chunks are inserted into the model's prompt before it generates an answer.
Q: Can I see or delete what ChatGPT or Claude remembers about me? A: Both companies expose memory management controls. OpenAI's Help Center documents a memory summary page where users can review, edit, or delete saved entries, and Anthropic's Claude memory feature includes granular controls over what gets retained, based on the companies' own product documentation.
Q: Why does a chatbot sometimes forget something I said earlier in the same conversation? A: This is almost always a context-window limit, not a memory-system failure. Once a conversation's total token count exceeds the model's window, the system drops or summarizes the oldest turns to make room for new ones, and details from those dropped turns are no longer visible to the model.
Q: Is a bigger context window the same thing as better long-term memory? A: No. A larger context window means a model can process more text within a single conversation, which helps with tasks like analyzing a long document in one pass. It has no effect on what persists after the conversation ends; persistence requires a separate storage layer like saved memory or RAG.
Q: What is a vector database, and why do AI memory systems need one? A: A vector database stores embeddings — numerical representations of text meaning — and is optimized to quickly find the stored items most similar to a new query, even across billions of entries. RAG pipelines rely on vector databases to locate the right chunks of source material fast enough to insert them into a prompt in real time.
Q: What is MemGPT and how is it different from a normal chatbot memory feature? A: MemGPT, introduced in a 2023 UC Berkeley paper, is an architecture that lets a model manage its own memory the way an operating system manages RAM and disk storage, explicitly deciding what to keep in the active context and what to archive externally. It has since become the open-source framework Letta, which developers use to build agents with persistent, queryable memory rather than relying on a single vendor's built-in feature.
Q: Are AI memory features a privacy risk? A: They can be, particularly when memory is built through automatic inference rather than explicit user input. Commentators have raised concerns that inferred "dossier" style memory profiles are hard for users to audit or correct, which is why reviewing a product's memory settings and understanding what gets stored automatically versus what you explicitly saved is worth doing before relying on the feature for sensitive information.
Q: Do AI agents use a different kind of memory than chatbots? A: Agents typically add a layer on top of chatbot memory: a running log of tool calls, intermediate results, and task progress, sometimes called scratchpad or agent state. This tracks what actions have already been taken during a multi-step task, separate from conversational memory about user preferences or retrieved documents.
The takeaway
AI memory is not a single feature, it is an engineering stack: a context window for the current conversation, retrieval for external documents, saved profiles for personalization, agent state for multi-step tasks, and occasionally fine-tuning for permanent change. When a product says it "remembers you," the honest translation is that a specific external system stored some text and successfully re-inserted the right piece of it at the right moment. Understanding which of the five layers is doing the work is the difference between trusting a black box and knowing exactly where to look when it gets something wrong.
Read next:
- GEO Explained — Generative Engine Optimization
- How to Rank in AI Overviews
- AI Agents Explained — How Autonomous AI Systems Actually Work
Frequently Asked Questions (Quick Reference)
Q: What is the simplest answer to what AI memory systems actually are? A: AI memory systems combine short-term context, retrieved documents, saved user preferences, tool state, and occasionally fine-tuned model weights — five distinct engineering layers, not one unified "memory."
Q: Why do different AI products remember things differently? A: Each provider mixes the five memory layers differently. OpenAI runs explicit saved memories alongside automatic "dreaming" summarization; Anthropic uses automatic conversation summarization; open-source frameworks like Letta give developers direct control over archival memory and retrieval.
Sources
- OpenAI, "Memory and new controls for ChatGPT"
- OpenAI, "Dreaming: Better memory for a more helpful ChatGPT"
- Simon Willison, "I really don't like ChatGPT's new memory dossier" (2025)
- Yahoo Tech, "Anthropic brings Claude's memory feature to all users" (2026)
- Google AI for Developers, "Long context" documentation
- NVIDIA, "What Is Retrieval-Augmented Generation aka RAG"
- Lewis, P. et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020), summarized in Latenode's technical breakdown
- Packer, C. et al., "MemGPT: Towards LLMs as Operating Systems" (2023), summarized in Leonie Monigatti's paper notes
- Suprmind, "Claude Features 2026: Projects, Artifacts, Memory, Computer Use, Skills, MCP"
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles
AI Model Distillation Explained
Model distillation trains a smaller model to imitate a larger model's outputs, often reducing cost and latency while preserving much of the behavior.
What Is a Vector Database? Beyond SQL
Vector databases store embeddings and run similarity search via ANN indexes. Pinecone, Weaviate, Qdrant, pgvector — compared for 2026.
Context Window Explained — LLM Memory Limits
Context windows define how much text an LLM can process at once. Learn token limits, the lost-in-the-middle problem, and 2026 model comparisons.
What Is RAG? Retrieval-Augmented Generation Explained
RAG is the technique making AI tools actually useful for real work — letting models look things up instead of relying on memory. Here's how it works and why it matters.
Open Source AI vs Proprietary AI: The Real Tradeoffs
Open source AI vs proprietary AI compared: what licenses allow, how far the capability gap has narrowed, and when self-hosting actually beats a paid API.