[Bug]: multi-CN HNSW query CN serves stale model after async generation update

[XuPeng-SH]

Is there an existing issue for the same bug?

• Yes. The observable BVT symptom is tracked by [Bug]: vector_hnsw_async intermittently returns the wrong delta nearest neighbor in Compose BVT #27629; this issue tracks the production-side cache invalidation root cause separately.

Branch Name

main

Commit ID

5bc051a

Other Environment Information

• Hardware parameters: GitHub Actions hosted Ubuntu x64 runner
• OS type: Ubuntu 22.04.5 LTS
• Deployment: multi-CN Docker Compose with Proxy
• Index: asynchronous HNSW, VECF32

Summary

After an HNSW generation is updated through asynchronous CDC on one CN, another CN that has already loaded the previous generation can continue serving the stale in-memory model. The metadata checksum changes and the CDC consumer processes the complete delta, but the query CN does not converge to the new generation within the BVT deadline.

Actual Behavior

In test/distributed/cases/pessimistic_transaction/vector/vector_hnsw_async.sql:

1. The first 10,000-row f32 generation is built.
2. A query on the query CN loads the first generation into the HNSW model cache.
3. A second 10,000-row CDC load is processed asynchronously. The consumer processes the 8,192-row and 1,808-row fragments and publishes a changed metadata checksum.
4. The query CN continues returning an old-generation nearest neighbor, for example 2638, instead of the delta endpoint 19999 for the full 120-second wait window.

Evidence:

• Direct BVT symptom: [Bug]: vector_hnsw_async intermittently returns the wrong delta nearest neighbor in Compose BVT #27629, [Bug]: vector_hnsw_async intermittently returns the wrong delta nearest neighbor in Compose BVT #27629
• Failing job: https://github.com/matrixorigin/matrixone/actions/runs/32930733336/job/98063525521
• Earlier reproduction: https://github.com/matrixorigin/matrixone/actions/runs/32926456273/job/98051500919?pr=27621
• Historical related failures: [Bug]: bvt test failed with inconsistent result of async index update tasks #22794 and [Bug]: BVT vector_hnsw_async.sql failed #22792

The old test implementation explicitly documented the multi-CN limitation: the model is cached in memory and may not be refreshed after CDC synchronization if a query has already loaded it.

Expected Behavior

When a newer HNSW generation becomes visible through the shared metadata/index storage, every CN must either:

• invalidate or replace its cached older generation before serving the next indexed query; or
• reject/defer the stale cache and load the published generation within a bounded convergence interval.

A query issued after generation publication must be able to return rows introduced by that generation, including 19999 in the reproduction above.

Impact

This is a correctness issue for distributed ANN queries, not only a performance or recall fluctuation. A CN can return results from an older data snapshot after the index update has been published, causing newly inserted rows to be invisible and nearest-neighbor results to be wrong until cache refresh or CN restart.

Current Workaround

• Do not issue the first f32 HNSW query until the asynchronous delta generation is complete; this is the BVT-side workaround in test: avoid stale HNSW cache in multi-CN BVT #27631.
• For exact-result checks, force a full scan with BY RANK WITH OPTION mode=force where supported.
• Restarting or routing to a CN without the old cache may clear the symptom, but is not an acceptable production workaround.

Suggested Investigation

• Make the HNSW cache key include the published metadata generation/checksum and compare it on every indexed query.
• Trace cache ownership and invalidation across CN-local consumers, model loaders, and query execution.
• Ensure old model handles cannot remain reachable after a newer generation is published.
• Add a native multi-CN integration test that warms the query CN cache, publishes CDC on another CN, and verifies the query CN observes the new generation.

Acceptance Criteria

• A warmed query CN observes the new HNSW generation after CDC publication on a different CN.
• The test passes repeatedly with two or more CNs and with parallel 8,192/1,808-row delta fragments.
• No stale result is returned after the bounded convergence/readiness signal.
• Cache replacement is generation-safe: no stale handle, double-close, leak, or unbounded model retention.
• Existing f32/f64 async HNSW BVT coverage remains green.

[iamlinjunhong]

Additional CI evidence from PR #27623 (head 5f01b702e4963fd2bd311137bb9ac6e5bce70639):

• Failed PROXY Compose BVT: https://github.com/matrixorigin/matrixone/actions/runs/32945352378/job/98106910593
• Failure signature: test/distributed/cases/pessimistic_transaction/vector/vector_hnsw_async.sql:133; after the metadata-ready check, the delta probes expected 10000 / 19999 but repeatedly returned 9691 / 2638 until the 60-second deadline.
• The downstream Coverage job failed only because this BVT coverage producer failed: https://github.com/matrixorigin/matrixone/actions/runs/32945352378/job/98117872701
• Attribution control: this PR head is already an ancestor of current official up/main (a53f5a3b757c5b21a36e539e4471533fe1d091a8) and its diff contains no HNSW/vector code. Current up/main includes independent PR test: avoid stale HNSW cache in multi-CN BVT #27631 / 6ef0c7818c, whose scope explicitly avoids warming the f32 HNSW cache before the delta and identifies the same stale neighbor 2638 versus expected 19999 multi-CN behavior.

This records the failed attempt before a single failed-jobs rerun of run 32945352378.

[cpegeric]

It is by design, and the design says so out loud

pkg/vectorindex/cache/cache.go:33-36:

// stalenessCheckEveryNTicks runs the IsStale freshness sweep every Nth HouseKeeping tick.
// The ticker is VectorIndexCacheTTL/2 (2.5m), so N=4 ≈ a 10-minute cross-CN freshness
// cadence — the bound on how long a remote CN can serve a stale index before it ages out.
const stalenessCheckEveryNTicks = 4

So the constant you're thinking of is VectorIndexCacheTTL = 5 * time.Minute, but the cross-CN convergence bound is ~10 minutes,
not 5: ticker = TTL/2 = 2.5 min, sweep every 4th tick. The comment names the exact behaviour the issue reports as a bug, and
calls it the bound.

The BVT waits 120 seconds. That's ~5× under the designed bound.

Two reasons even a 5-minute wait wouldn't have helped

The TTL can never fire for this test. extendLocked sets ExpireAt = now + TTL on every Search. The BVT polls in a loop for the
whole window, so it actively renews the entry it's waiting to expire. TTL eviction is unreachable while you're querying; the only
convergence path is the 10-minute IsStale sweep.

Cross-CN is pull-only, deliberately. Cache.Remove is documented as "LOCAL to this process — a prompt local optimization only;
cross-CN coherence is handled by the pull-based freshness check (StaleChecker/IsStale via HouseKeeping)". The writer CN evicts
immediately; the reader CN converges at its own sweep. That is precisely the topology in the report.

The detection itself is correct — worth saying, so this doesn't get "fixed" twice

IsStale does catch a CDC delta. The generation is a content fingerprint — the multiset of per-model-file MD5 checksums plus the
model-row count — and generation.go spells out that "a CDC append/delete rewrites a model file → new checksum → the fingerprint
changes". It's clock-independent on purpose, so it survives an (X,N)→(0,0)→(X,N) rebuild that a timestamp couldn't.

So nothing is broken in the mechanism. It converges — at the sweep.

Conclusion: #27629 is a test-expectation mismatch. #27631 (don't issue the first query before the delta completes) is the correct
fix, not a cache change.

Why we don't checksum on every search

Not a stupid question, but the suggestion in "Suggested Investigation" — compare the checksum on every indexed query — is a
design change from pull to inline, and it's the wrong trade for four separate reasons:

1. It's a SQL round-trip, not a memory compare. The freshness generation is:

SELECT checksum FROM .<metadata_table>

IsStale runs that through queryHnswGeneration on a background auto-commit txn with a 60-second timeout. Putting it inline means
every ANN query first does a metadata table scan, opens a txn, and waits on it. An HNSW search over a warm in-memory model is
sub-millisecond; the freshness query is a distributed read. You'd add one to two orders of magnitude to the fast path to detect
an event that happens once per CDC batch.

2. It puts a hot metadata table in front of every vector query. Every CN, every indexed query, all tenants — the metadata table
   becomes a per-query hotspot, and its own read latency becomes vector-search latency. That's a scalability cliff you can't tune
   your way out of.

3. The failure mode gets worse, not better. IsStale currently returns (true, err) on a query error so the entry is reclaimed and
   reloaded — safe, because it's a background sweep and a reload is cheap. Inline, a metadata hiccup would have to either fail the
   user's query or be ignored, and "ignored" is exactly the staleness you were trying to remove.

4. It doesn't even buy exactness. The check would race the CDC commit anyway — you'd verify freshness, then search a model that
   went stale a microsecond later. You'd pay per-query cost for a bound that's still only "fresh as of a moment ago", not a
   consistency guarantee.

The design already handles the case where this matters: the writer's own CN evicts synchronously, so read-your-writes on one CN
is immediate. Only the remote warm CN lags, and that's what the sweep is for.

If shorter convergence is genuinely wanted, the lever already exists and costs nothing: lower stalenessCheckEveryNTicks (4 → 2
gives ~5 min, → 1 gives ~2.5 min) or VectorIndexCacheTTL. That's a tuning decision about how much background SQL you want, not
new machinery. Each sweep is one short query per cached index, single-flighted off the housekeeping goroutine.

One thing worth confirming before closing

I'd want a single run that waits past 10 minutes and shows convergence. If the entry still serves stale data after two sweeps,
that would be a real bug — and the code even anticipates it: genValid == false → IsStale returns (true, nil) → evict and reload,
precisely so a hot entry "whose TTL keeps sliding on every search" can't be pinned "serving pre-CDC/rebuild data forever".
Confirming that empirically turns this from an argument into evidence, and closes the issue cleanly rather than by assertion.