Voice Agent Memory: Remembering Callers Across Calls

Voice agent memory that survives across calls needs a graph of callers, calls, open commitments and topics, with vectors attached, queried at call start under a tight latency budget. Transcripts and summary embeddings alone won't do it. This guide builds that in HelixDB, a graph database with native vector and full-text search, using TypeScript and a local instance.

You should already have a phone agent (Twilio, SIP, or a realtime speech stack) that stores a transcript per call and can produce a summary and embedding at call end. You need Node.js and Docker or Podman for the local server.

Step 1: Audit the usual voice agent memory stack and what it misses

The usual stack is three stores: a transcript table keyed by call ID, a vector store of per-call summary embeddings, and a CRM lookup by phone number. It answers "what did we say once" and misses what a returning caller actually expects you to know.

Three gaps show up in production. First, open promises: "I'll email you the quote by Friday" lives in a transcript, and a summary embedding rarely surfaces it as an open item. Second, relationships: the caller who is the assistant of a previous caller, or who is calling about the same account, is invisible to a flat vector index. Third, change since last call: nothing in the stack says what is new, what was resolved, and what is still pending.

The fix is structure, not a better embedding. Commitments become first-class records with a status, calls link to the commitments they made or closed, and similarity search runs only over the relevant caller's history. That is a graph problem with a vector problem attached. Running a separate vector store and a separate graph store means syncing two systems on the hot path of a phone call.

Keep the retrieval design simple: key memory to a stable identifier (the phone number), keep a concise profile plus open items, and fetch a small bundle before the agent's first substantive reply. Don't replay full transcripts into the prompt.

Step 2: Model Caller, Call, Commitment and Topic as a graph

Four node labels and a handful of edges cover most phone-agent memory. In HelixDB everything is one labeled property graph, and vectors are properties on nodes and edges.

Nodes

• Caller: phone (indexed), name, firstSeen

• Call: startedAt (unix seconds), summary, summaryEmbedding (vector property), direction

• Commitment: text, status (open or done), dueAt

• Topic: name

Edges

• Caller -[MADE]-> Call

• Call -[MADE_ON]-> Commitment (the call where the promise was made)

• Call -[ABOUT]-> Topic

• Call -[FOLLOWS_UP]-> Call (this call continues an earlier one)

• Caller -[KNOWS]-> Caller (relationships between callers)

One decision matters more than the rest: model time yourself. HelixDB does not provide temporal modelling, so startedAt, dueAt and status are plain properties you maintain. The engine gives you efficient time-range indexing, which Step 5 uses for "last 30 days".

For a closely related shape on the support side, see customer support agent memory, which models users, tickets and history the same way.

Step 3: Set up HelixDB locally with helix init and helix start dev

A local HelixDB server is two CLI commands plus an SDK install. From the repo README, install the Helix CLI, then in your project directory:

helix init local
helix start dev

This needs Docker or Podman and serves on port 6969. Install the TypeScript SDK:

npm install @helix-db/helix-db

If you'd rather skip the server while prototyping, the same package opens an embedded database in your process with Client.embedded({ kind: "disk", root: "./memory", database: "callers" }). Embedded mode needs no CLI, and closing the writer persists data. The rest of this guide uses the server client, so the call path looks like production:

import { Client } from "@helix-db/helix-db";

export const db = Client.server("http://localhost:6969");

Queries are built with writeBatch(), readBatch() and g() and run only when you call .send(). The SDK produces JSON that the database parses and executes. There is no separate query language to learn and nothing to compile or deploy ahead of time.

Step 4: Write each call back: summary embedding, commitments and edges

Write the call back in one batch after hangup, off the audio path. First create the vector index once, then write the nodes and edges for each call. Graph and vector writes in a batch commit or roll back together.

Create the index (dimension must match your embedding model's output):

import { g, writeBatch, VectorDistanceMetric } from "@helix-db/helix-db";
import { db } from "./db";

const DIM = 1536; // match your embedding model

await db
  .query(
    writeBatch()
      .varAs(
        "idx",
        g().createVectorIndexNodes(
          "Call",
          "summaryEmbedding",
          DIM,
          VectorDistanceMetric.Cosine,
          null,
        ),
      )
      .returning(["idx"])
      .toQueryRequest(),
  )
  .send();

Then write a call and a commitment:

import { g, writeBatch } from "@helix-db/helix-db";
import { db } from "./db";

export async function recordCall(
  callerId: number,
  summary: string,
  embedding: number[],
  startedAt: number,
  promise: { text: string; dueAt: number },
) {
  const batch = writeBatch()
    .varAs(
      "call",
      g().addN("Call", {
        startedAt,
        summary,
        summaryEmbedding: embedding,
      }),
    )
    .varAs(
      "commitment",
      g().addN("Commitment", {
        text: promise.text,
        status: "open",
        dueAt: promise.dueAt,
      }),
    )
    .returning(["call", "commitment"]);

  return db.query(batch.toQueryRequest()).send();
}

Add edges in the same batch or a follow-up one: MADE from the caller to the call, MADE_ON from the call to each commitment, ABOUT to topics, and FOLLOWS_UP to the previous call. Check the exact edge-creation builder names in the query guide at docs.helix-db.com for your SDK version before shipping. When a later call closes a promise, update the commitment's status to done and add a FOLLOWS_UP edge. That status flip is your "what changed since last call" signal.

Deduplicate topics and callers before inserting. The same concern applies to LLM-extracted entities generally, covered in entity resolution for knowledge graphs.

Step 5: Index call timestamps for last-30-days queries

"What happened in the last 30 days" is a range query over Call.startedAt, and time-range indexing makes it cheap. HelixDB does not give you temporal modelling, but time-range indexing is efficient, and that is all this query needs.

Store startedAt as integer unix seconds. Then the filter is a plain comparison:

const thirtyDaysAgo = Math.floor(Date.now() / 1000) - 30 * 24 * 3600;
// use a range predicate (gte) on startedAt with this value

Keep the model honest: "recent" is a window you compute in application code and pass as a parameter. Nothing in the engine knows what "last 30 days" means, and that's fine.

Step 6: Retrieve caller context at call start under a latency budget

Retrieval at call start should be one round trip that returns a small bundle: the caller, open commitments, and the last few calls. Budget for it explicitly. One published guideline puts call-open memory retrieval under roughly 200 ms. The right number depends on your telephony stack, so measure p95 under production load rather than trusting a median.

Three rules keep you inside the budget:

1. Start the lookup before the agent speaks. Your telephony webhook already has the caller's number at ring time. Fire the memory fetch then, in parallel with CRM and any other context, and join before the first substantive reply.

2. One graph query, not N. Look up the Caller by indexed phone, traverse MADE to recent calls and MADE_ON to commitments in the same read batch, and project only the fields the prompt needs.

3. Return a bundle, not a history. Send open commitments, the last two or three call summaries, and topic names. Leave transcripts out.

import { g, readBatch, SourcePredicate } from "@helix-db/helix-db";
import { db } from "./db";

export async function callStartContext(phone: string) {
  const q = readBatch()
    .varAs(
      "calls",
      g()
        .nWhere(SourcePredicate.eq("phone", phone))
        .out("MADE")
        .valueMap(["$id", "startedAt", "summary"]),
    )
    .varAs(
      "open",
      g()
        .nWhere(SourcePredicate.eq("phone", phone))
        .out("MADE")
        .out("MADE_ON")
        .valueMap(["text", "status", "dueAt"]),
    )
    .returning(["calls", "open"]);

  return db.query(q.toQueryRequest()).send();
}

Filter open to status = "open" with a property predicate and sort calls by startedAt descending in the same batch, then trim in code. Time the whole function with real network latency between your voice stack and the database, and co-locate them if the budget is tight. Writes stay non-blocking: the post-call write from Step 4 never sits on the audio path.

Step 7: Scope vector recall to this caller's calls with pre-filtering

When the caller says "about that billing thing from before", vector search should rank only this caller's calls, not every call in the index. That is vector pre-filtering: select an indexed starting node, traverse, then rank only the reached candidates. The order is traversal, candidate membership, ranking, top results, so unrelated callers never enter the ranking.

import { g, SourcePredicate, VectorDistanceMetric } from "@helix-db/helix-db";

const results = g()
  .nWhere(SourcePredicate.eq("phone", phone))
  .out("MADE")
  .vectorSearch("summaryEmbedding", queryVector, {
    metric: VectorDistanceMetric.Cosine,
    limit: 5,
  });

Verify this builder against the published package types for your SDK version before shipping. The traversal-then-vectorSearch shape is the part to keep. Chain a where(...) range predicate on startedAt after the traversal if you want similarity limited to the last 30 days.

Two documented limits are worth knowing. An unrestricted vector search caps at 800 effective results. A traversal-scoped search applies that ceiling after candidate intersection and rejects rather than silently clamping. And a candidate stream over 1,000,000 unique entities is a query error. A single caller's call history sits far below both.

This is the practical gap versus a standalone vector store with a metadata filter bolted on. Here the scope comes from the graph, so "this caller's calls", "calls about this topic" or "calls by people this caller knows" are all the same mechanism. For the wider design question, see agent memory graph vs vector store.

Step 8: Delete a caller's memory, then what to do next

Deleting a caller is a graph operation: find the Caller node by phone, collect the Call and Commitment nodes reached through MADE and MADE_ON, and delete them along with their edges. The summary embeddings are properties on the Call nodes, so they go with them. There is no second store to purge. Run it as one write batch so it commits or rolls back atomically, then confirm with a count query that nothing remains reachable from that phone number.

Do this on a request path you can audit, and decide up front whether shared Topic nodes stay (they hold no personal data) or get pruned when orphaned. The full walkthrough is in how to delete an AI agent's memory of a user.

Troubleshooting: if call-start retrieval is slow, check that phone is indexed, that you project only needed fields, and that the fetch overlaps CRM lookups instead of following them. If vector results look wrong, confirm the index dimension matches your embeddings and that the query vector uses the same model.

Start with one caller and one open promise

Voice agent memory is a graph with vectors attached, not a bigger transcript pile. Run helix init local, helix start dev, load a single caller with one call and one open commitment, and time the call-start query against your latency budget. If it fits, widen to topics and relationships. The code and CLI live in the HelixDB repo, and a star helps others find it.