Semantic Freshness: Your Vector Search Index Is Going Stale — What If We Could Measure It?

This is an idea paper: a problem I think is under-discussed, a proposed way to measure it, and a sketch of a system that could act on that measurement. It’s a direction to explore, not a finished result — and I’d love feedback from anyone running vector search in production.

The Problem: A Search Index That Assumes Nothing Changes

When a shopper types “waterproof hiking boots for kids,” most modern retail search systems don’t match keywords. They turn the query and every product into embeddings — numeric vectors that capture meaning — and look for the products whose vectors sit closest to the query’s. To make that fast across millions of products, the vectors are organized into an approximate nearest-neighbour (ANN) index.

Here’s the catch. That index is built once, from a snapshot of the catalogue, and then treated as fixed. But a retail catalogue is never fixed. Products are added, updated, and delisted every day. So over time:

• New products are missing from the index entirely.
• Delisted products are still retrievable.
• Updated products keep old embeddings that no longer describe them.

Teams today routinely monitor data quality, feature drift, and model predictions. But the ANN index — a critical piece of the retrieval pipeline — usually has no health signal at all. The standard answer is brute force: rebuild the whole index on a schedule, nightly or weekly, and hope that’s often enough. That’s expensive when the schedule is frequent, and blind when it isn’t.

The Idea: A Freshness Score for Every Part of the Index

What if the index could tell us how stale it is?

The proposal is to split the index into partitions — groups of similar products — and give each one a Semantic Freshness Score (SFS) between 0 and 1, where 1 means perfectly fresh. The score would be computed continuously from three signals that already flow through a retailer’s catalogue event stream, so it wouldn’t require expensive re-scoring of the whole corpus:

• Deletion rate — how much of the partition has been delisted since the last rebuild
• Addition pressure — how many new products belong in this partition but aren’t indexed yet
• Semantic displacement — how far updated products have moved from their old embeddings

Combine those three into one weighted number per partition. When a partition’s score drops below a threshold, it gets flagged for rebuilding.

The appeal is that it turns index health into something you can put on a dashboard and alert on — the same way we already treat feature drift in ML pipelines.

DriftGuard: Rebuild Only What’s Stale

The second half of the idea is a system — I’ve been calling it DriftGuard — that runs alongside an existing search index and uses the freshness score to decide what to rebuild and when:

1. Event ingestion — read product additions, deletions, and updates in real time and assign each one to its partition.
2. Freshness monitor — recompute every partition’s score on a regular heartbeat and queue any that fall below the threshold.
3. Smart scheduler — rebuild the stalest, highest-traffic partitions first, and cap how many rebuilds run at once so the embedding infrastructure isn’t overloaded.
4. Shadow swap — build each new partition off to the side, then swap it into the live index in one quick step. Queries that arrive mid-swap are briefly held and replayed, so shoppers never see a half-built index.

Because all of this would run asynchronously, next to the live system rather than in it, it shouldn’t add latency to search.

Hypotheses Worth Testing

The idea rests on a few assumptions that would need to be validated:

• Staleness is concentrated. If most of the quality loss comes from a small share of partitions, targeting that “stale tail” recovers most of the value.
• New products matter most. Unindexed new products probably hurt shoppers more than slightly outdated ones — which would make addition pressure the most important of the three signals.
• The score predicts real quality loss. A freshness score is only useful if it moves with actual search relevance, not just with event counts.
• It travels. Ideally, a score tuned on one catalogue would carry over to others with only light recalibration.
• Traffic-aware scheduling pays off. Rebuilding by staleness and shopper traffic should beat a simple first-in-first-out queue.

Open Questions

• Granularity. Partition-level scores could miss very localized drift, like a brand-new subcategory appearing overnight. Occasional full rebuilds might still be a sensible safety net.
• Calibration. How should the three signals be weighted, and how different would those weights be for groceries versus furniture versus fashion?
• Encoder upgrades. When the embedding model itself is retrained, everything looks drifted. That case probably still calls for a one-time full rebuild.
• Images. Many retail search systems embed product images too. Extending the freshness idea to multi-modal search is an open problem.

Why This Matters Beyond the Math

From a product seat, this is really an observability story. We’ve learned to put health signals on data pipelines and ML models, but the search index — the thing that decides what a shopper actually sees — often runs on a calendar and a hope. A single, continuously readable freshness number would change the conversation from “how often should we rebuild everything?” to “which parts are stale right now, and do they matter?”

If you run vector search at scale and have seen this problem — or tried to solve it — I’d genuinely like to compare notes.