How to Evaluate AI Agent Memory: LongMemEval and LoCoMo

To evaluate AI agent memory, score retrieval and answer quality separately, break results down by question type, and build a gold set from your own agent's logs. Use LongMemEval and LoCoMo as the template for what to measure. This works with any backend: a vector store, a graph, or both.

You need an agent that already writes and reads some memory layer, a few weeks of conversation logs, and an LLM you can use as a judge. By the end you will have a repeatable eval that tells you whether memory works, which question types fail, and what to fix. The last two steps show where graph plus vector retrieval helps and where you have to model time yourself.

Step 1: Decide what 'memory works' means for your agent

Memory works when your agent answers a question correctly using something it learned in an earlier session, at a latency and token cost you can ship. Write that down as three numbers before you touch a benchmark: a minimum answer accuracy, a latency budget for the retrieval call, and a ceiling on tokens injected per turn.

Then list the behaviors your product actually depends on. A support agent needs to recall a customer's open issue from last week. A coding agent needs to remember a convention you corrected it on. A personal assistant needs to notice that your address changed. Each of those maps to a question type, and you will weight your eval toward the types your users hit.

Be blunt about what you are not testing. Recall of a fact stated one turn ago is a context-window test, not a memory test. Memory starts when the fact has left the window and the agent has to fetch it.

Step 2: Learn what LongMemEval and LoCoMo actually measure

LongMemEval and LoCoMo both test whether an assistant can answer questions about very long, multi-session conversation histories. They are the best templates for structuring your own eval. Read the papers before you borrow their categories.

LongMemEval benchmarks chat assistants on long-term interactive memory. Its question design is useful because it separates abilities: extracting information from a single session, reasoning across multiple sessions, temporal reasoning, handling knowledge that was updated later, and abstaining when the history does not contain the answer. LoCoMo evaluates very long-term conversational memory of LLM agents with question answering over long dialogues. It probes single-hop, multi-hop and temporal questions among other categories.

Two things to take from them. First, a single accuracy number hides the failure mode, so both benchmarks are organized by question type. Second, abstention is a real category. An agent that confidently invents a memory is worse than one that says it does not know.

Should you just run the public benchmarks? Run them once as a sanity check on your retrieval stack, but don't stop there. Their conversations are not your users' conversations, your memory writer is not their pipeline, and a score on someone else's data won't tell you whether your agent remembers a customer's renewal date. Use them for the taxonomy, and your own logs for the verdict.

Step 3: Build an eval set from your agent's own logs

Build the gold set by sampling real sessions from your logs and writing question-answer pairs whose answers depend on earlier sessions. Aim for 100 to 300 questions to start. You can grow it later.

A workable process:

1. Sample sessions across users and weeks, not just the most recent ones. Old sessions are where retrieval breaks.

2. For each, pick a fact, decision, preference or relationship that a later session depends on. Write the question as the user would have phrased it, and the reference answer as one short sentence.

3. Record the evidence: the session and turn IDs that contain the answer. You will need this to score retrieval in Step 4.

4. Tag each question with a type: single-hop, multi-hop (answer needs two or more facts joined), temporal (answer depends on when), knowledge update (the fact changed, so the latest value wins), or abstention (the answer is not in memory).

5. Add update and abstention cases deliberately. They rarely occur naturally in samples, and they are the ones that expose stale or fabricated memory.

Use an LLM to draft candidates from sampled sessions if you like, but have a person verify every reference answer and evidence pointer. A gold set with wrong answers produces confident, meaningless scores. Strip or pseudonymize personal data before it lands in the eval set.

Step 4: Score retrieval and answers separately (recall@k, LLM judge, latency, token cost)

Score retrieval and answer generation as two independent stages. A wrong answer can mean the right memory was never fetched, or it was fetched and the model ignored it. Those need different fixes.

Retrieval. For each question, run only your memory retrieval and check whether the evidence turns or facts you recorded appear in the top k. Report recall@k at the k you actually inject into the prompt (say 5 and 10). If recall@k is low, no prompt change will save you.

Answer quality. Feed the retrieved context to your agent and compare its answer to the reference with an LLM judge. Give the judge the question, reference answer and candidate answer, and ask for correct, incorrect or abstained. Spot-check a sample of judgments by hand every run, because judges drift and can be lenient on partially right answers. For abstention questions, a correct abstention counts as a pass and any specific answer counts as a fail.

Isolate the two. Run an oracle pass where you hand the model the gold evidence directly. If accuracy is high with oracle context and low with retrieved context, retrieval is your problem. If it is low even with oracle context, look at your prompt, your context formatting, or the model.

Cost and speed. Log retrieval latency at p50 and p95, and tokens injected per question. A memory layer that gains two points of accuracy by stuffing 8,000 extra tokens into every turn may be a bad trade.

Keep this table per run:

Metric	What it tells you
recall@k	Did retrieval find the evidence?
Judge accuracy (retrieved context)	End-to-end quality
Judge accuracy (oracle context)	Ceiling for your generator
Retrieval p50 / p95 latency	Whether it fits the turn budget
Tokens injected per question	Ongoing cost

Step 5: Break failures down by question type

Group every metric by the question type tag, because the aggregate number will hide the one category that is failing badly. A 78% overall score can be 95% on single-hop and 40% on knowledge updates.

Build a grid of question type against recall@k and judge accuracy, then read it for patterns:

• Low recall on multi-hop, fine on single-hop. Your retriever finds the nearest chunk but not the second fact that completes the answer. Similarity search returns what looks like the question, not what is connected to it.

• Good recall but wrong answers on knowledge updates. Old and new values both come back and the model picks the stale one, or only the stale one is retrieved because it matches the wording better.

• Weak temporal accuracy. Questions like "what did we decide before the March migration" need ordering or range filters that embeddings don't encode.

• Low abstention pass rate. Retrieval returns something plausible-looking for every query, so the model always has material to hallucinate from. Consider a relevance threshold.

For each failing cell, read ten failures by hand and write one line on why. Patterns show up fast. The next two steps cover the three fixes that come up most: multi-hop, knowledge updates, and time.

Step 6: Fix multi-hop and knowledge-update failures with graph plus vector retrieval

Multi-hop and knowledge-update failures usually come from retrieving by similarity alone. The fix is to store facts as entities and relationships, then combine vector recall with graph traversal. For background on why, see agent memory graph vs vector store for multi-hop reasoning.

Multi-hop. Take the question "Who manages the person who filed the billing bug?" Vector search finds the bug report. The manager is two edges away and shares no wording with the question. If memory stores the person, the report and the reporting line as nodes and edges, you can use vector search to find the entry point and then traverse to the answer. HelixDB is an open-source graph-vector database built in Rust. Vectors are a property on nodes and edges, so one query can do the semantic lookup and the traversal without a second system to sync.

Knowledge updates. Model a fact as a node or edge with the value and a write timestamp, and link a new version to the one it supersedes rather than overwriting silently. Retrieval can then prefer the head of the chain, and your eval can check that the stale value is not returned. HelixDB's vector pre-filtering lets you scope a similarity search to a graph-filtered candidate set, for example only facts attached to this user, so you are not ranking against the whole index.

A sketch in TypeScript against the embedded on-disk engine:

import { Client } from "@helix-db/helix-db";

const client = await Client.embedded({
  kind: "disk",
  root: "./memory-db",
});

// Build your read query with readBatch() and g(), then run it:
// const res = await client.query(query.toQueryRequest()).send();

The query itself follows your schema: a vector search scoped to one user's facts, then a traversal along supersedes and relationship edges. See the HelixDB docs for the query builder reference, and for local setup run helix init local then helix start dev. The code is at github.com/HelixDB/helix-db.

Then rerun Step 5. The multi-hop and update rows should move. If they don't, your extraction step is probably not creating the edges you assumed.

Step 7: Handle temporal questions with time-range queries you model yourself

Temporal questions need explicit timestamps on your memory records and range queries over them. HelixDB does not model time for you, so you design the temporal schema yourself. What it gives you is efficient time-range indexing, which is the part that matters for speed.

Decide what time means for each record. Keep at least two fields: when the event happened in the conversation or world (event time), and when your agent wrote the memory (write time). They differ more often than you expect, since a user saying "last Tuesday I moved" gives an event time earlier than the session date.

Then translate temporal questions into filters:

• "What did I say about pricing in March?" becomes a range filter on event time, then a similarity search within the result.

• "What was my address before I moved?" becomes an ordered walk of the supersedes chain, taking the version whose validity interval ends at the move.

• "What happened first?" becomes an ordering on event time across two retrieved records.

Parse relative expressions ("last week", "before the launch") into absolute ranges at write time where you can, and at query time otherwise. Your eval will tell you which approach works: add temporal questions that use relative phrasing and see whether recall holds. For latency considerations on these paths, low-latency graph retrieval for AI agents covers what to measure.

Rerun the eval after every change, and keep it in CI

Treat the eval set like a test suite. Rerun it whenever you change the extraction prompt, the embedding model, the schema, the retrieval query or the generator model, and compare the per-question-type grid rather than the headline number. Version the gold set and grow it with every failure you find in production. Next step: start a local HelixDB with helix init local and helix start dev, model one failing category, such as supersedes chains for knowledge updates, and star the repo at github.com/HelixDB/helix-db if the before and after grids look better.