Fix HNSW index crashes and incorrect query results

[grparry]

Hi! I've been evaluating ruvector as a pgvector alternative for a project and ran into a few issues with the HNSW index access method on PostgreSQL 17. I traced them down and put together fixes — hope these are helpful.

Summary

Six bugs fixed in the HNSW access method (hnsw_am.rs), plus one shared fix in ivfflat_am.rs:

1. SIGSEGV on repeated queries — beginscan must allocate xs_orderbyvals and xs_orderbynulls arrays (like GiST and SP-GiST do). RelationGetIndexScan allocates the scan descriptor but not these arrays, so the executor was writing distance values to stale/uninitialized memory. First query in a backend often worked; second+ crashed at addresses like 0x1.

2. Empty HNSW graph — connect_node_to_neighbors was a TODO stub. Implemented bidirectional connections (new node → neighbors, each neighbor → new node) with pruning when neighbor count exceeds M (upper layers) or M0 (layer 0).

3. Wrong distance metric — hnsw_build used HnswConfig::default() which hardcodes DistanceMetric::Euclidean, regardless of the operator class (e.g. ruvector_cosine_ops). Added metric_from_index() that reads the opclass support function name via index_getprocid + get_func_name.

4. Wrong result ordering — BinaryHeap::into_iter().take(k) returns k arbitrary items from the heap's backing array, not the k smallest. Removed .take(k) before the sort+truncate.

5. xs_recheckorderby must be false — Setting it to true triggers PG17's IndexNextWithReorder distance comparison, which raises "index returned tuples in wrong order" on harmless floating-point differences between index-computed and operator-computed distances.

6. Use-after-free in endscan — Added null check and null-out of scan->opaque to prevent double-free if endscan is called after a rescan that resets the state.

Testing

• Built and tested on PostgreSQL 17.7 with pgrx 0.12.9
• Verified with 5d and 8d vectors, dataset sizes 15 and 100
• 20+ consecutive HNSW queries per session with no crashes
• HNSW results match sequential scan ground truth across all test queries

Test plan

• Build with cargo pgrx install --release
• Create table with ruvector(N) column and HNSW index using ruvector_cosine_ops
• Run SELECT ... ORDER BY embedding <=> query::ruvector LIMIT k multiple times in the same psql session
• Compare HNSW results against sequential scan (SET enable_indexscan = off)
• Verify no crashes on repeated queries

🤖 Generated with Claude Code