Pre-Filtering Vector Search on Graph Edges: How to Scope ANN to Relationships

Most engineers start their AI journey by dumping document chunks into a flat vector database like Pinecone. This works until the agent needs to answer a question that requires structural context. If a user asks for messages similar to a specific bug report but only within the React project, a global search returns the nearest neighbors from the entire corpus. You end up with documents about Vue or Angular because they are semantically close, even though they are structurally irrelevant.

This guide shows you how to move beyond flat search. We will cover how to implement vector search on graph edges to scope approximate nearest neighbor (ANN) retrieval to specific relationship types or subgraphs. You will learn how to attach embeddings directly to edges, and how to scope a search so the results come back both semantically similar and structurally valid. By the end you will be able to build retrieval logic that combines graph structure with vector recall in a single engine.

Step 1: Understand Why Global ANN Breaks in Relational Contexts

Global vector search assumes all data points live in the same bucket. When you query a standard vector index, the engine calculates distances across every single point in the embedding space. This approach fails the moment your application requires multi-tenancy or complex entity relationships. We went through that trade-off in vector database vs graph database. The short version is that a flat index cannot respect a boundary without expensive post-filtering.

Traditional ANN algorithms like HNSW are designed to find the closest points in a high-dimensional space. They do not naturally understand that Point A is a message sent by Alice and Point B is a comment on a Jira ticket. If Alice asks a question about her own files, a global search might return files from Bob because their embeddings are statistically similar. You then have to filter these results in application code, which often leads to the empty result problem. If the top 100 neighbors are all Bob's files, Alice gets zero relevant results after filtering.

In production, structural relevance is usually more important than raw semantic similarity, and a global search has no idea the graph exists. The point generalises past any one engine: if you can restrict the candidate set by relationship before you calculate distance, you get results that are both structurally valid and semantically close. If you cannot, you are picking one and hoping for the other. What you want is a search space that is already narrowed to the right part of the graph by the time the vector comparison runs.

Step 2: Attach Embeddings to Edges, Not Just Nodes

The standard way to model data is to put embeddings on the nodes. You might have a Document node with a vector representing its content. This works for simple retrieval, but it ignores the richness of how entities interact. Attaching embeddings to edges lets you search for relationships themselves. Imagine an edge type called MENTIONS that connects a Slack message to a specific project. By putting the vector search on graph edges, you can query for the most similar mentions within a specific context.

This shift in modeling changes the retrieval pattern. Instead of asking for similar nodes, you ask for similar interactions. If you are building agent memory, you might want to retrieve past conversations where a user expressed a specific sentiment about a feature. The sentiment is the edge. By embedding the interaction rather than the static document, you capture the dynamic state of the knowledge graph.

This is not exotic. Memgraph supports vector indexes on both nodes and edges, and so does HelixDB. The part worth understanding is what declaring the index over an edge label actually buys you: the search space is defined by the relationship type before any distance is calculated, because the index only ever contained embeddings from that one edge label. You are not filtering a global result set down to the COMMENTED_ON edges. There was never anything else in the index to begin with.

In HelixDB you declare a vector index over the label and property you intend to search, then hop onto the edge stream and rank it. The SDKs pick the node form or the edge form of the search from whatever the traversal is currently standing on, so an edge hop followed by a vector search ranks edges. Queries are plain JSON built from your own application code, so there is no second system to keep in step and no ID-mapping layer between the graph and the vectors.

Step 3: Choose Your Scoping Strategy: Pre-Filter vs Post-Filter

When you scope a vector search, you have two technical paths: post-filtering and pre-filtering. Post-filtering is the naive approach. You perform a global ANN search, get 100 results, and then throw away the ones that do not match your graph criteria. This is dangerous. If your graph filter is restrictive, such as looking for files owned by a single user in a million-user system, you will likely end up with zero results. The ANN index will simply not find the needles in the haystack if they are far away from the global centroid.

Pre-filtering is the better shape for scoped retrieval: narrow the candidate set first, then search inside it, so everything that comes back is already valid. The idea is not new. Qdrant's filterable HNSW is a well-known implementation of it, keeping the walk inside the permitted partition rather than discarding hits afterwards, and their indexing documentation is worth reading if you want the mechanics. What most vector databases give you, though, is a flat metadata filter. The partition is a tag on a row, not a relationship.

HelixDB gives you three ways to pre-filter, and they differ in how much you have to decide up front.

The first is the traversal itself, and it is the one to reach for when membership is a correctness requirement rather than a tuning choice. You start with a node or edge traversal, then chain the vector search onto it, and the engine ranks only the exact members of that stream. The documented order is graph traversal, then exact candidate membership, then vector ranking, then top k. The guarantee is the part worth reading twice: the traversal set is authoritative, and a result outside the candidate set cannot be returned. So "documents this user may access" stops being a filter you apply hopefully afterwards and becomes the boundary of the search itself. The docs are blunt about the alternative: running a search across the whole label and then applying a where clause is not the same thing, because the excluded high scorers have already consumed the top k and you are left with fewer eligible results than you asked for. Build the candidate stream first.

The second is tenant partitioning. Declare the index with a tenant property and the ANN search runs inside one partition rather than across the whole indexed label. Leave the property off at creation and the index is global. The two compose rather than compete: scoping by traversal against a partitioned index means the candidate stream has to be built from the same partition the index was. That is how you say "closest match semantically, but only inside this workspace".

The third is the edge vector index from the previous step. An index declared over a relationship type is scoped to that relationship by construction, before any query runs.

Reach for the traversal when relationships, permissions, or an earlier filter define who is eligible. Use a source-level search when the whole indexed label, plus any tenant partition, already is the candidate pool. And constrain the candidate traversal if it could grow unbounded, because everything it reaches is work the ranking has to carry.

Step 4: Build the Scoped Query in HelixDB (JSON + SDK)

HelixDB has no query language at all. Queries are plain JSON, or built with the native Rust, TypeScript, Go, or Python SDKs inside your own application code, and sent as one POST to /v2/query.

There are two steps. Declare the index once at setup, then search it. Index creation is a write query rather than a method on the client, so it goes inside a write batch like any other mutation.

import {
  g, readBatch, writeBatch, defineParams, param,
  SourcePredicate, VectorDistanceMetric,
} from "@helix-db/helix-db";

// Setup: declare the index over the label and property you will search.
// The final argument is the tenant property; null leaves the index unpartitioned.
const index = writeBatch()
  .varAs(
    "index",
    g().createVectorIndexNodes(
      "Comment", "embedding", 1536, VectorDistanceMetric.Cosine, null,
    ),
  )
  .returning(["index"]);

// Query: walk to the comments on this project, then rank only those.
const params = defineParams({
  slug: param.string(),
  query_vector: param.array(param.f32()),
  limit: param.i64(),
});

const recall = readBatch()
  .varAs(
    "matches",
    g().nWithLabelWhere("Project", SourcePredicate.eq("slug", params.slug))
      .out("COMMENTED_ON")
      .vectorSearchWith("Comment", "embedding", params.query_vector, params.limit)
      .valueMap(["$id", "body", "$distance"]),
  )
  .returning(["matches"]);

const request = recall.toQueryRequest(
  params,
  { slug: "react", query_vector: queryVector, limit: 10n },
  { queryName: "project_comment_matches" },
);

The traversal in the middle is doing the scoping. Without it, the same search ranks every comment you have. With it, the ten results come from the comments on that one project and nothing else can appear. You can also get back fewer than ten, because the result is bounded by the number of unique candidates the traversal actually reached. A thin subgraph returns what it has rather than padding the list from somewhere else.

Swap .out for .outE and the stream is edges rather than nodes, and the same search call then ranks the embeddings sitting on those relationships, because the SDKs choose the node or the edge operation from whatever the traversal is standing on. Go is the exception and spells them out as VectorSearchNodesWithin and VectorSearchEdgesWithin. The With suffix means the call takes typed parameters; vectorSearch is the same operation with the values written inline. Keep $distance in a projection before you traverse past a ranked hit or you lose it. Indexes support cosine, euclidean and manhattan distance, the declared dimension and every query vector have to match exactly, and vector properties have to be top-level rather than nested. Index creation can also return before the backfill finishes, and a new generation stays hidden until it validates and activates, so an empty result straight after creating an index is not a bug.

Run helix init and then helix start dev to get a local instance up and test this. The open-source build runs three ways and you pick one at startup: fully in memory, on disk, or against object storage, meaning S3 or any S3-compatible store including a MinIO container on your laptop. The engine, the SDKs, and the endpoint are identical in all three, so the query above does not change when you move from a laptop to production.

The builders are ordinary functions in your own language, so a typo in a label is a build error rather than an empty result set in production. That is a narrower failure mode than assembling a query string and finding out at runtime. Because full-text search and KV lookups run in the same engine, you can also mix a keyword match into the same request.

Step 5: Validate Recall Under Tight Scopes

One of the biggest hurdles in vector search on graph edges is ensuring high recall when the subgraph is small. HNSW indexes rely on a connected graph of vectors. When you filter by a specific relationship, you are essentially removing most of the points in that index for the duration of the query. If the remaining points are not well-connected in the HNSW graph, the search algorithm might fail to find the actual nearest neighbor. This is the disconnected graph problem in ANN.

This is a general property of filtered ANN, not a quirk of any one engine: the effectiveness of a filtered search depends on the ratio of the filtered set to the total population. Search a partition holding a tiny fraction of your corpus and a graph-based index has fewer usable edges to walk, so recall degrades. The usual levers are exploring more candidates at query time, or falling back to an exhaustive scan once the candidate set is small enough that scanning it is cheap.

The practical advice is the same whatever you are running on. Pick your partitions so they are not pathologically small, and measure. Compare scoped results against a known ground truth on a small development subgraph before you scale to millions of nodes, because a filtered search that quietly returns the second-best answer looks exactly like one that returns the best.

Step 6: Extend the Pattern: Multi-Hop Scoped Retrieval

Single-hop scoping gets you most of the way, but the questions agents actually get asked are multi-hop. Find the security whitepapers written by engineers who worked on the same project as the current user. That is a traversal from User to Project to Engineer, and then a similarity search over what those engineers wrote.

This is the shape prefiltering exists for. You walk User to Project to Engineer, then chain the vector search onto the end of that walk, so the ranking runs over the engineers the traversal actually reached and over nothing else. It is one request against one store, with no round trip to a second database and no ID list shuttled between them. The membership guarantee holds across the hops: a whitepaper written by someone the traversal never reached cannot come back, however close its embedding sits.

What to watch on multi-hop is the size of the candidate set rather than its correctness. Two hops out of a well-connected node can fan out a long way, and everything it reaches is work the ranking carries. Constrain the traversal before the search when the fan-out is unbounded.

That composition is what a company brain needs. It keeps the agent from surfacing chunks that are semantically plausible but belong to a project the user has nothing to do with. The precision comes from the structure, not from a better embedding model.

HelixDB exposes native MCP endpoints, so an agent can discover the graph structure and walk it step by step rather than being handed one opaque query. The agent explores the neighbourhood, works out which relationship it cares about, and runs the scoped vector search against that edge index.

Conclusion

Scoping vector search to graph edges turns a generic retrieval system into a context-aware memory engine. Most RAG pipelines fail because they lack structural grounding, which leads to hallucinations or irrelevant results. By letting the traversal define who is eligible before any distance is calculated, you keep your agents inside the right context rather than filtering your way back to it afterwards. You no longer need to manage the complexity of syncing data between three different databases just to run a simple filtered search.

HelixDB was built to remove that duct tape. Graph traversals and vector search run in one engine, so the scope and the similarity search are not two systems you keep in step. If you are tired of the overhead of a separate vector store and graph store, helix init and helix start dev get you an instance to try this against, and you can star HelixDB on GitHub if the approach is one you want to follow.