LLM Memory Architecture: How Agents Remember Across Sessions

An LLM memory architecture is a set of stores around a stateless model: the context window holds what the model sees now, and external stores hold what it should remember later, written out and retrieved on every turn. The model learns nothing between calls. An agent that forgets everything between sessions has no memory layer at all, only a prompt.

This guide builds that layer in six steps, from context window management to episodic, semantic and procedural stores, and ends with a working setup in HelixDB, an open-source graph-vector database built in Rust. You need to know embeddings and basic RAG. The generic engineering comes first, and you can build steps 1 to 5 on any stack. Step 6 shows where one store replaces a vector DB plus a graph DB.

Step 1: Separate the memory types your agent actually needs

LLM memory is not one thing. Split it into five types and decide which ones your agent needs. Each has a different lifetime, write path and retrieval pattern.

Type	Lives where	Lifetime	Retrieved by
Working	Context window	One model call	Already in the prompt
Session	App state or event log	One conversation	Recency, session ID
Episodic	External store	Indefinite	Time range plus similarity
Semantic	External store	Indefinite	Similarity plus relationships
Procedural	External store or prompt files	Indefinite	Task match

Working memory is what fits in the context window. Session memory is the running transcript and scratch state. Episodic memory is a record of things that happened: this user asked X on Tuesday, the tool call failed, the fix was Y. Semantic memory is distilled facts: the user works at Acme, Acme uses Postgres, the user prefers short answers. Procedural memory is how to do things: system prompts, tool-use rules, learned playbooks.

Not every agent needs all five. A support bot needs session, episodic and semantic. A coding agent leans on procedural and semantic. Write down which types your agent needs before you pick a database, because the retrieval pattern in the last column drives the storage choice.

The research literature frames the same split differently, as text-based, KV-cache, parameter-based and hidden-state memory. Those are internal mechanisms of the model. For an application developer the practical line is simpler. Internal memory is what the model weights and attention cache hold, and external memory is anything you store and re-inject yourself. This post covers the external kind, because it's the only kind you control from application code.

Step 2: Manage the context window as working memory, and decide what gets written out

The context window is working memory. The difference between LLM memory and a context window is lifetime: the window is wiped after every call, while memory is whatever you persist outside it and load back in. A bigger window delays the problem but doesn't solve it. Cost and latency grow with every token you resend, and a window can't carry anything across sessions.

Treat the window as a budget and spend it on purpose. A workable layout:

1. A fixed header: system prompt and procedural rules.

2. Retrieved long-term memories, capped at a token budget.

3. A rolling summary of the older part of this session.

4. The last N raw turns.

Then add a write-out policy, which is the part most prototypes skip. At the end of each turn, or when a turn is evicted from the window, decide what to persist:

• Always log the raw event (who, what, when) to the episodic store.

• Run an extraction pass over it for durable facts and write those to the semantic store.

• Drop pleasantries and tool noise.

Keep write-out out of the hot path. Queue it and run it after the response is sent, so memory writes never add user-visible latency. The rest of this guide covers where those writes land.

Step 3: Persist session and episodic memory as events with timestamps, and query them by time range

Store episodic memory as an append-only log of events, each with a timestamp, an actor, a session ID, the text, and an embedding. Events are immutable. You never edit what happened. You add new events.

A minimal event record:

{
  "id": "evt_81f2",
  "session_id": "sess_204",
  "user_id": "u_17",
  "ts": 1767225600,
  "kind": "user_message",
  "text": "The staging deploy failed again on the migration step",
  "embedding": [0.012, -0.044, 0.31]
}

This supports the two retrieval patterns episodic memory needs. Time-range queries answer "what happened in the last hour" or "what did we discuss last week". Similarity queries answer "have we seen something like this before". Real recall usually combines both: similar events, restricted to a window.

On time, be precise about what you get. A store with efficient time-range indexing lets you filter and order by timestamp cheaply. It doesn't model time for you. Validity intervals, superseded facts and point-in-time views are yours to build, typically by storing valid_from and valid_to properties and filtering on them. HelixDB follows this model: it has efficient time-range indexing, and temporal modelling is manual. Design your timestamps and validity fields up front and the manual part stays small.

Step 4: Extract semantic memory as entities and relationships, then embed it

Semantic memory is what the agent knows, as opposed to what happened, and it should be stored as entities and relationships, not just text chunks. Run an extraction pass over episodic events and emit triples: (user u_17) works_at (Acme), (Acme) uses (Postgres), (user u_17) prefers (short answers).

The extraction loop:

1. Take new events from the episodic log.

2. Ask the model to extract entities and relationships as structured JSON, with the source event ID attached as evidence.

3. Resolve each entity against existing ones, by normalised name plus embedding similarity, so "Acme", "Acme Inc." and "acme" collapse into one node.

4. Upsert the entity nodes and relationship edges.

5. Embed each entity's description and each fact so it's reachable by similarity too.

Keep the evidence link from every fact back to its source event. It's how you debug a wrong memory, and how you expire a fact when the underlying event turns out to be stale or mistaken.

Entity resolution in step 3 is where duplicate memories start. Too strict and you get three Acmes. Too loose and you merge two different companies. Start strict, log near-misses, and review them. For a worked extraction pipeline on real documents, see how to build a knowledge graph from meeting notes.

Step 5: Retrieve with similarity and relationships together (and why vector-only recall breaks)

Vector-only recall breaks because similarity finds text that looks like the query, while many useful memories are connected to the query without resembling it. Ask an agent "what database does the customer I met on Tuesday use?" The embedding of that question is close to nothing in your store. The answer sits two hops away: Tuesday's meeting event, the person in it, their company, the company's database. Top-k vector search returns orphaned chunks that mention databases and misses the chain.

The retrieval pattern that works has three moves:

1. Anchor. Use similarity, full-text search, or a time range to find entry points: the most relevant events or entities.

2. Expand. Traverse relationships from those anchors, one to three hops, to pull in connected facts.

3. Rank and trim. Score the candidates, deduplicate, and fit them to the token budget from step 2.

Relationships also give you scoping. Instead of running similarity over the whole index, restrict it to the candidates connected to the current user or project, then rank by similarity within that set. That's both more accurate and cheaper.

If you build this on two systems, you anchor in the vector DB, expand in the graph DB, then join the results in application code, with IDs synced between them on every write. That sync is the two-database tax, and it's where stale and missing memories come from. For a deeper look at the trade-off, see agent memory graph vs vector store for multi-hop reasoning and when to use a knowledge graph instead of embeddings.

Step 6: Put it in one graph-vector store with HelixDB

HelixDB is an open-source graph-vector database built in Rust that combines graph traversal, vector search and BM25 full-text search in one engine. Vectors are a property on nodes and edges, so events, entities and relationships live in one store with one transaction boundary. A graph write and its embedding commit or roll back together, which removes the sync problem from step 5. Vector search can also be pre-filtered to related entities, so similarity runs over a graph-scoped candidate set. The core is Apache-2.0 at github.com/HelixDB/helix-db.

Start a local instance with the CLI (needs Docker or Podman, serves on port 6969):

helix init local
helix start dev

If you'd rather skip the server while prototyping, the TypeScript client can run embedded, in process. Install it with npm install @helix-db/helix-db (3.2.0). Open an on-disk database and check the exact option shapes against the HelixDB docs:

import { Client } from "@helix-db/helix-db";

const client = await Client.embedded({
  kind: "disk",
  root: "./agent-memory",
  database: "memory",
});

Queries are JSON built with the native SDK DSL. You build them with writeBatch(), readBatch() and g(), and nothing runs until you call .send(). For a built query the call is client.query(query.toQueryRequest()).send(). There's no separate query language to learn. Model events as nodes with a ts property, entities as nodes, and relationships as edges, and use the SDK's property, comparison and ordering predicates for time-range filters. The query guide in the docs has the exact builder methods for node creation, edge creation, vector search and traversal. Check the signatures there before copying anything, since the builder surface is the part that changes between releases.

The mapping from this guide to the store:

• Episodic: event nodes with a timestamp property, filtered by time range, with an embedding for similarity.

• Semantic: entity nodes and relationship edges, each with an embedding.

• Retrieval: a similarity or full-text anchor, then a bounded traversal, in one query instead of two systems and a join.

The same shape works across agents. See shared memory for multi-agent systems for several agents reading and writing one graph. On scale, the open-source core is embedded and bounded by a single node, while HelixDB Cloud is object-storage-backed with readers that auto-scale horizontally. In HelixDB's own benchmarks, graph traversals are reported up to 1000x faster than Neo4j and vector performance on par with Pinecone and Qdrant. Treat those as vendor numbers and benchmark your own workload.

What to do next: troubleshooting stale, duplicated or missing memories

Most memory bugs fall into three buckets, and each has a specific fix.

Stale memories. The agent states a fact that was true last month. The cause is facts stored without validity bounds, so old and new versions coexist. The fix: add valid_from and valid_to to semantic edges, close the old edge when a contradicting fact arrives, and filter on validity at retrieval. Temporal modelling is yours to build.

Duplicated memories. The same person or company appears as several nodes, and retrieval returns near-identical facts. The cause is loose or missing entity resolution in step 4. The fix: resolve on normalised name plus embedding similarity before insert, merge existing duplicates by re-pointing edges, and deduplicate facts at retrieval time.

Missing memories. The agent should know something and doesn't. Check in order: was the event logged at all (write-out policy in step 2), did extraction emit the fact, did entity resolution attach it to the right node, and did the traversal depth reach it. Log which anchors and hops produced each retrieved memory, so you can see where the chain broke.

If you change embedding models, re-embed everything. See re-embedding agent memory after you change models.

Start with one anchor-and-expand query on a local HelixDB

A good LLM memory architecture is a write-out policy, an event log, an entity graph and a retrieval step that uses both similarity and relationships. Build the smallest version first: log events, extract facts, and answer one multi-hop question end to end. Run helix init local and helix start dev, load a day of real conversation, and check whether the agent can follow a relationship that vector search alone misses. If it works, star HelixDB on GitHub and join the Discord to show what you built.