When to Use a Knowledge Graph Instead of Embeddings

Most teams add a knowledge graph the wrong way. Someone reads a GraphRAG paper, a whiteboard session produces an ontology, and three weeks later there is a second database to keep in sync with a retrieval stack that was mostly fine. The better path is to let your own questions tell you.

This guide walks through a seven-step process you can run on any embeddings-based RAG or agent memory setup. You build an eval set from real questions, tag each one by shape, measure embeddings-only recall per shape, and add a graph only for the shapes where top-k similarity fails. Steps 1 through 5 work with whatever stack you have today. Step 6 shows the HelixDB version, where graph traversal and vector search over the graph-filtered candidates run in one query.

Prerequisites: a working embeddings pipeline (any vector store), a corpus you can inspect, and a few hours to label questions by hand. You know ANN, top-k and BM25, so none of that gets re-explained here. If the process says you do not need a graph, that is a good outcome. You just saved a database.

Step 1: Build an eval set from real questions, not synthetic ones

Pull 100 to 200 questions from places where users or agents actually ask things: support tickets, search logs, Slack threads where someone asks "who owns X", agent traces where a tool call failed to find context. If you have no logs yet, interview three users and write down the exact phrasing they use. Don't generate the set with an LLM. Synthetic questions are written from the document outward, so they echo the chunk's own vocabulary and embeddings retrieve them beautifully. Real questions are written from the asker's head inward, and that is where retrieval breaks.

For each question, record a gold answer and the set of source chunks or records that support it. Be strict about the gold set. A multi-hop question has several required pieces of evidence, and you want to know whether retrieval found all of them, not just one.

Keep the format boring: a JSONL file with question, gold_answer, gold_ids (a list), and a blank shape field you fill in next step. Aim for at least 15 questions per shape you expect to see, because recall on five questions is noise.

One trap: don't tune your embedding pipeline on this set before measuring. This set is the baseline. Chunk size, model and top-k stay frozen until Step 3 is done, so the per-shape numbers describe the system you actually run in production.

Step 2: Tag every question by shape (multi-hop, aggregation, exact identifier, exclusion, scoped, time-bounded)

Shape is the property of a question that decides which retrieval mechanism can answer it. Tag each question with exactly one primary shape, and allow a secondary tag when it is genuinely mixed.

Multi-hop: the answer needs a chain across entities. "Which customers are affected by the bug that Priya's team filed against the billing service?" No single chunk contains that chain.

Aggregation: counting, ranking or summing across many items. "How many open incidents involve the auth service?" Top-k returns k chunks, so it cannot count past k.

Exact identifier: an order ID, a SKU, a function name, a person's full name. Embeddings blur these. BM25 or an exact-match filter handles them.

Exclusion: "Which vendors have we used except the ones flagged in the 2024 audit?" Similarity search has no concept of NOT. The excluded items are, if anything, semantically closer to the query.

Scoped: the question only makes sense inside one customer, project or person. "What did we decide about pricing for Acme?" The top-k is full of pricing discussions from other accounts.

Time-bounded: "What changed in the onboarding flow since March?" Similarity ignores dates entirely.

Two people should tag independently on a sample of 30 and compare. If you disagree on more than a handful, your shape definitions are too fuzzy. Tighten them before moving on. The tagging itself teaches you something: teams are often surprised that 40 percent of their real traffic is scoped or identifier-shaped, and no graph is needed for either.

Step 3: Measure embeddings-only recall per shape

Run every question through your current retrieval, embeddings only, with the production top-k. Compute recall@k against gold_ids, and for multi-hop questions compute two numbers: any-hit recall (found at least one gold item) and full-hit recall (found all of them). The gap between those two is the signature of a multi-hop failure: the retriever finds the first hop and loses the rest.

from collections import defaultdict

def recall_at_k(retrieved, gold):
    return len(set(retrieved) & set(gold)) / len(gold)

by_shape = defaultdict(list)
for q in eval_set:
    hits = retrieve(q["question"], k=10)  # your current pipeline
    by_shape[q["shape"]].append({
        "recall": recall_at_k(hits, q["gold_ids"]),
        "full": set(q["gold_ids"]) <= set(hits),
    })

for shape, rows in by_shape.items():
    r = sum(x["recall"] for x in rows) / len(rows)
    f = sum(x["full"] for x in rows) / len(rows)
    print(f"{shape:18} n={len(rows):3} recall@10={r:.2f} full-hit={f:.2f}")

Print the table and put it somewhere visible. This is the artifact that makes the decision defensible. "Our retrieval feels shallow" starts arguments. "Multi-hop full-hit is 0.31 and exclusion is 0.22" ends them.

Also log a few failures per shape by hand. Read what came back. The failure mode matters: wrong customer's documents (scoped), right topic but missing the second entity (multi-hop), right entity but a stale version (time-bounded). Those notes drive the modelling work in Step 5.

Step 4: Read the results: which shapes need a graph, and which only need a filter, BM25 or SQL

Low recall on a shape doesn't mean you need a graph. It means you need something other than plain top-k similarity. Match the shape to the cheapest fix.

Exact identifier: add BM25 or an exact-match index. This is a search-mode problem, not a relationship problem. See how to combine BM25 and vector search in one query before reaching for anything heavier.

Scoped: a metadata filter on tenant, customer or project ID usually fixes it. If your vector store supports pre-filtering, you are done. A graph only earns its place here if the scope is itself relational ("everything connected to this account through any contract").

Time-bounded: a date-range filter on a timestamp property. Cheap, and no graph required.

Aggregation: often plain SQL or a structured index. If the things you count are entities with relationships (incidents touching services owned by teams), a graph traversal with a count is the natural fit. If they are rows in a table, use the table.

Multi-hop and exclusion: these are the shapes where a graph earns its keep. Multi-hop is the defining case: the answer is a path, and a path is what a graph stores. Exclusion over relationships ("except vendors flagged in the audit") is a set difference over an edge. That is trivial in a traversal and nearly impossible in similarity search.

The decision rule: add a graph when a failing shape needs relationship structure and no filter, BM25 or SQL fix moves its number. If your table shows the failures are all identifier and scoped shapes, stop here. Ship the filter, re-run the eval, and keep one database. That is the honest answer to when to use a knowledge graph instead of embeddings: only for the shapes that are about connections.

Step 5: Model the shapes that failed as nodes, edges and properties (including time, modelled manually)

Model only what the failing questions need. Resist the full-ontology instinct. For each failing multi-hop question, write out the path you want the retriever to walk, then turn nouns into node labels and verbs into edge labels.

Take "Which customers are affected by the bug Priya's team filed against billing?" The path is Person -MEMBER_OF-> Team -FILED-> Bug -AFFECTS-> Service <-USES- Customer. That gives you five node types and four edge types. Chunks of text hang off the nodes as a Document or Chunk node linked by MENTIONS edges, and each chunk carries its embedding as a property. In HelixDB, vectors are a property on nodes and edges, so there is no second store holding embeddings keyed by foreign IDs.

Time needs care. HelixDB does not provide temporal modelling, so you model it manually. The pattern is explicit timestamps. Put created_at and, where state changes matter, valid_from and valid_to as properties on the nodes or edges that change. HelixDB's efficient time-range indexing makes range queries over those properties fast, but "what was true on March 3rd" is a query you express yourself with those properties. It is not a built-in mode.

If your events repeat (a customer escalates three times), model each occurrence as its own node or edge with its own timestamp, rather than a single edge with a counter. The history is the point.

The extraction step, getting entities and relations out of documents, is a separate job. For a worked example on unstructured input, see building a knowledge graph from meeting notes. Start with the failing shapes only and grow the model when the eval asks for it.

Step 6: Run graph traversal and vector search over the graph-filtered candidates in one HelixDB query

In the two-database version of this, you traverse in the graph DB, ship a list of IDs over the network, and then run a filtered ANN query in the vector store with an id IN (...) clause. That is two round trips, two consistency windows, and a filter list that gets large fast.

HelixDB does it in one query because vector search can be scoped to the traversal's results. The traversal produces a candidate set, and similarity runs over that set instead of the whole index. This is vector pre-filtering, and it is why the multi-hop shape stops losing the second hop. Two documented limits to know: an unrestricted vector search caps at 800 effective results, and a traversal-scoped search applies that ceiling after candidate intersection and rejects rather than silently clamping. A candidate stream over 1,000,000 unique entities is a query error, so scope your traversal.

Start the database locally with helix init local then helix start dev (needs Docker or Podman, serves on port 6969), or open an embedded one in-process from TypeScript:

import { Client } from "@helix-db/helix-db";

const client = await Client.embedded({
  kind: "onDisk",
  root: "./helix-data",
  database: "eval",
});

Build the query with readBatch() and g(), and run it with .send(). The exact traversal and vector-search steps are in the query docs, and the SDKs produce the same JSON that goes to POST /v2/query. Check the published types for the step names before you copy anything, because I won't guess at them here.

The shape of the query is: start at the Team node, walk FILED to bugs, AFFECTS to services, USES back to customers, then run vector search over the chunks attached to that candidate set, ranked against the question embedding. One query, one transaction, one result. The code knowledge graph guide shows the same pattern on a different corpus. Source and issues live at github.com/HelixDB/helix-db.

Step 7: Re-run the eval and decide; what to do next and troubleshooting when recall does not move

Run the identical eval set through the new retrieval path and print the same per-shape table. Compare shape by shape against Step 3. You are looking for movement on the shapes you modelled, and no regression on the ones you didn't touch. Multi-hop full-hit should climb toward its any-hit number, because the traversal supplies the missing hops.

Decision rule: if a modelled shape improved meaningfully and the lift is worth the modelling and ingestion cost, keep the graph for that shape and leave everything else on the cheap path. If it didn't, don't add more graph. Debug first.

Recall did not move on multi-hop. Check extraction. Most of the time the edge you needed was never written. Spot-check ten failing questions and walk the path by hand in the graph. A missing edge is an ingestion bug, not a retrieval bug.

Recall moved, but answers are still wrong. Your gold set may be measuring retrieval while the failure is in generation. Inspect the context the model actually received.

Candidate set too large or query rejected. Your traversal fans out. Add edge-label or property filters, or tighten the starting node, so you stay under the documented limits.

Exclusion still failing. Make sure the excluded relation is modelled as an edge, not buried in text.

Next: for agents that need shared state, see shared memory for multi-agent systems, and for the broader trade-off read vector database vs graph database.

Conclusion

Run the eval before you run the migration. Build 100 real questions, tag them by shape, and print the recall table. If the failures are identifiers, scopes and dates, ship a BM25 index and a metadata filter and keep your single vector store. If they are multi-hop and exclusion questions, you have evidence that relationships matter, and that is the moment a graph pays for itself.

When that evidence points to a graph, don't take on the two-database tax to act on it. Clone github.com/HelixDB/helix-db, run helix init local and helix start dev, model only the failing shapes, and re-run your eval with graph traversal and pre-filtered vector search in one query. If the number on your worst shape doesn't move, you will know that too, and you will have spent an afternoon instead of a quarter.