AI Agent Memory: How It Works and What to Set
AI agent memory is the storage that carries facts, past runs and documents from one agent session into the next. In POLYROB, an open-source (MIT) self-hosted AI agent, it is a local SQLite database with keyword recall by default, optional vector recall, and a separate knowledge base for documents.
AI agent memory is the layer outside the model that stores what an agent learned in earlier sessions and hands the relevant part back when a new session needs it. The model itself keeps nothing between runs. Memory is what lets a chat that starts fresh today still know what you decided last week.
This page covers what POLYROB stores, how it finds the right row later, how it forgets, and which settings to check. Every default below comes from the public repository at release v1.7.8.
What is the difference between a context window and agent memory?
A context window is what the model can read in one call. It is rebuilt every turn and it has a size limit. Agent memory lives outside it, on disk, and a small selection is copied into the window when it looks relevant.
POLYROB keeps five separate stores, because they answer different questions:
| Store | Holds | Lifetime | Switch |
|---|---|---|---|
| Cross-session recall | Rows saved from past conversations and runs | 365 days by default | MEMORY_BACKEND |
| Task memory (H-MEM) | A summary, phases and recent steps of the current task | One session | HMEM_SEMANTIC |
| Curated notes | Short notes the agent adds, reads and archives | Until archived | MEMORY_TOOL_ENABLED, off by default |
| Knowledge base | Documents you add, split into searchable chunks | Until you remove them | KB_ENABLED |
| Conversation thread | Your chat across sessions and chat apps, newest last | Read by /thread | OWNER_THREAD_INJECT, on by default |
The thread is not part of the memory provider. MEMORY_BACKEND does not change it (docs/guide/[email protected]).
Where does POLYROB keep its memory?
On your machine. The memory database is memory.db in the data home, which is ~/.polyrob/data unless you set POLYROB_DATA_DIR ([email protected], docs/guide/[email protected]). Nothing about memory needs a hosted service.
Cross-session memory is on by default and scoped to a tenant, so a paired chat sender never reads the owner’s rows. MEMORY_REQUIRE_USER_ID is on by default and refuses memory I/O for anonymous sessions. Turn it off only on a single-user box that wants one shared bucket (docs/guide/[email protected], section 5).
Each saved row is capped at 4,000 characters (MEMORY_ROW_MAX_CHARS). With MEMORY_STORE_ANSWER_ONLY on, which is the default under the local profile, POLYROB stores the distilled answer and not the echoed question and answer transcript. That keeps a query that repeats your question from matching the question text instead of the answer (docs/[email protected]).
How does the agent find the right memory later?
Recall runs in one of two modes, chosen by MEMORY_BACKEND:
MEMORY_BACKEND | What you get |
|---|---|
sqlite (server default) | Keyword recall over SQLite FTS5, no extra dependencies |
local_vector (default under the local profile) | Hybrid keyword and vector recall; needs pip install "polyrob[memory-vector]" |
none | No cross-session memory |
Keyword recall tokenizes your message, drops short words, and builds an OR query over up to 12 informative terms. When the query has two or more such terms, it also runs an all-terms query and fuses the two lists, so a row that holds the whole question is not buried by a long row that repeats one word (modules/memory/[email protected]).
Vector recall loads the sqlite-vec extension through apsw. If the extra, the extension or the embedder is missing, the agent logs one warning that names the extra and falls back to keyword recall. It keeps working (docs/guide/[email protected]).
What decides which rows come back first?
Ranking is not only the text score. modules/memory/[email protected] sets these rules:
- Newer rows weigh more. The weight halves its distance to a floor every 14 days, and the floor is 0.3, so an old ruling still ranks.
- Near-duplicates merge. Two rows whose word sets overlap by 0.8 or more count as one, and the newer one wins. Numbers count as words, so “widget 1” and “widget 2” stay separate.
- A weak match is dropped. An automatic recall injects nothing when the best score is under 2.0, and “nothing” is an allowed answer. The floor applies only once the table holds 200 rows or more, because a small table gives the score no scale.
- Owner-chat recall reads the last 7 days unless your message names an older date. Rows marked as facts, such as an address or a preference, have no age limit.
- A line marked
[unverified]is a claim a run could not check. An owner chat never gets it back as recall.
These are the rules in the code, not a quality measurement. POLYROB publishes no accuracy or speed figure for recall, and this page does not either.
How does the agent keep track inside one long task?
Task memory follows a three-layer structure called H-MEM: a session summary that is always in context, a list of phase memories, and a rolling window of recent steps (modules/memory/task/[email protected]). The window holds the last 100 steps.
HMEM_SEMANTIC picks how the agent looks across phases: auto uses embeddings when an embedder exists and keyword matching when it does not, embeddings and lexical force one, and off disables it (docs/[email protected]). This store lasts for the session. What should outlive it goes to cross-session recall.
What are curated notes and the knowledge base for?
Recall rows are written automatically. Notes and documents are the two places where you or the agent decide what to keep.
Curated notes come from the memory tool, which is off by default (MEMORY_TOOL_ENABLED). It can read, add, remove, create, update, archive and list notes. Writes are threat-scanned and fail closed, and a turn that looks forged is quarantined as pending instead of becoming an active note. A tenant holds at most 50 entries and 2,000 characters by default (MEMORY_TOOL_MAX_ENTRIES, MEMORY_TOOL_MAX_CHARS). A note is pending, active or archived, and archiving frees space (modules/memory/[email protected]).
The knowledge base is a separate, document-shaped store. Add files, then search them:
polyrob kb add ./docs
polyrob kb search "refund policy"
polyrob kb listpolyrob kb also has remove and export. The base is on by default under the local profile (KB_ENABLED) and is never touched by the per-turn memory writes. A document the agent writes itself goes into a written collection that ages out if nobody reads it (modules/memory/[email protected], cli/commands/[email protected]).
Inside a chat, /memory search <query> recalls across sessions, /kb search <query> searches documents, and /thread shows the conversation across sessions (docs/guide/[email protected]).
How does AI agent memory forget?
Memory with no forgetting fills with run narration. POLYROB prunes in three ways:
- Age.
MEMORY_RETENTION_DAYSis 365 by default. Rows from cron and goal runs use 45 days, capped by that value. A value of 0 or less disables pruning. Older rows without a provenance stamp are exempt (docs/[email protected],modules/memory/[email protected]). - Caps. Row length and note counts are bounded, as above.
- Quarantine. Scoped rows that were never promoted are purged after 14 days.
A chat that sits idle also resets. SESSION_IDLE_MINUTES is 720 under the local profile and 1,440 otherwise, after which the chat starts with a clean context window. Recall bridges the gap, which is the reason for the whole system (docs/[email protected]).
How do you keep hostile content out of agent memory?
A memory that anything can write to is a way to plant instructions. POLYROB has two defenses, and both are opt-in.
Memory scopes (MEMORY_SCOPES_ENABLED, off by default) give each goal or cron run its own quarantine inside a tenant. The scope label is minted by the server, never supplied by the agent. A run’s regime decides what it reads and writes:
| Regime | Reads | Writes |
|---|---|---|
shared | The shared pool | The shared pool |
scoped (default for goal and cron runs once scopes are on) | Shared pool and its own scope | Its own scope only |
sealed | Its own scope only | Its own scope only |
A scope’s rows reach the shared pool only when the root goal completes verified, and at most 50 rows move. A failed, blocked or cancelled goal never promotes. Owner chat is always shared (modules/memory/[email protected]). List scopes with polyrob owner memory scopes list.
The threat scan (MEMORY_THREAT_SCAN, off by default) rejects text that matches instruction-override patterns before it is stored as a finding. The source calls it a conservative pattern set, not a complete intrusion detector (modules/memory/task/[email protected]). Treat it as one layer. The wider model is covered in AI agent security.
Which memory settings should you check first?
Start with polyrob doctor, which reports memory among its preflight checks, then list the memory flags:
polyrob doctor --flags --group memory
pip install "polyrob[memory-vector]"
polyrob config set MEMORY_BACKEND local_vectorThen work through this list:
- Confirm the backend with
/memoryin the REPL. If recall feels shallow, the log may say thesqlite-vecextension was unavailable; install thememory-vectorextra (docs/guide/[email protected]). - Decide whether you want the
memorynotes tool. It is off until you setMEMORY_TOOL_ENABLED. - Add documents with
polyrob kb addinstead of hoping the agent remembers a file. - If you run goals or jobs, turn on
MEMORY_SCOPES_ENABLEDso their findings stay in quarantine until a goal verifies. - Set
MEMORY_RETENTION_DAYSto the window you can defend.
The learning loops, such as episodic run records and reflection on session close, need autonomy switched on with polyrob autonomy on or the owner-visible posture. They are not on from install ([email protected], docs/[email protected]). For the rest of the install, start from the install page and the docs.
Where does POLYROB memory fall short?
- Keyword recall, the server default, matches words. A question phrased in different words from the saved row can miss it unless vector recall is on.
- The memory lives in SQLite files on one machine. Moving to another machine means moving the data home.
- No recall-quality, latency or storage-size figures are published, so there is nothing to compare against other agents.
- Scopes and the threat scan are off by default. A default install uses the single shared pool.
- Recall is a selection, not a guarantee. The agent can still answer without a row that exists.
FAQ
Does an AI agent remember everything you tell it?
No. POLYROB saves rows with a length cap, merges near-duplicates, drops weak matches at recall time and prunes by age. What you want kept for certain belongs in the knowledge base or in a curated note.
Is agent memory the same as RAG?
They overlap. Retrieval-augmented generation fetches documents to put in front of the model. The POLYROB knowledge base does that for files you add, while cross-session recall does it for the agent’s own past conversations and runs. Both can run at the start of a step.
Can agent memory be turned off?
Yes. Set MEMORY_BACKEND to none for no cross-session memory. The knowledge base and the conversation thread have their own switches.
Where is the memory stored, and who can read it?
In memory.db on the machine that runs POLYROB. Rows are tenant-scoped, and a paired chat sender does not reach the owner’s memory (docs/[email protected]).
Next steps
Memory is one part of running your own agent. Personal AI agent shows how it fits with chat apps and goals, agent skills covers what the agent writes for itself under your review, and self-hosted AI agent covers the setup around it. To try it, follow the install page. The configuration reference is on GitHub.
POLYROB is an open-source AI agent you run and own. Install it or read the docs.