Surface absolute relevance scores in eval rows to make expect_none / OOD robustness measurable

[helebest]

Summary

Retrieval eval (dikw client eval) emits per-query results without relevance scores, so expect_none / out-of-distribution (OOD) robustness cannot be measured. This request asks to surface an absolute relevance score (top vector cosine, and the cross-encoder rerank score when the rerank leg ran) in the eval report rows, which in turn unlocks a score-based OOD-separation metric.

Problem

expect_none negatives are meant to test OOD robustness: for an off-corpus query, a healthy engine should not assign high relevance to any document. But doc-level retrieval never abstains — it always returns a ranked list. So rank order alone carries no robustness signal: an off-topic query still has a rank-1 doc.

The discriminating signal is the absolute relevance score of the top hit — covered queries score high, OOD queries score low; the gap is the robustness measure. Today the eval rows drop scores entirely:

• NegativeRow.to_dict() → {q, ranked} (src/dikw_core/eval/runner.py:243-254)
• PerQueryRow.to_dict() → {q, expect_any, ranked, [id]} (src/dikw_core/eval/runner.py:~206-288)

Neither positives nor negatives expose a score, so separation is uncomputable and negative_diagnostics stays diagnostic-only. The documented placeholder "expect_none satisfaction ≥ 0.90" is currently not computable from eval output.

Important: the hybrid RRF score is the wrong signal

A naive "just add the score" that surfaces the hybrid RRF fused score would not help. RRF scores are rank-based (1/(k+rank)), so every query — including a fully OOD one — has a rank-1 doc with a comparable top score. RRF does not encode absolute relevance and cannot discriminate covered vs uncovered.

The signals that do discriminate already exist as Hit.score in the searcher; they are simply not propagated into the report rows:

• top vector cosine (covered ~0.7 vs OOD ~0.2), and/or
• cross-encoder rerank score when a reranker is configured (absolute relevance; even cleaner).

Proposed change

1. Surface the absolute relevance score in the eval report rows (minimal, high-value):

• Add scores: list[float] aligned with ranked (or at minimum top1_score: float) to both PerQueryRow.to_dict() and NegativeRow.to_dict().
• The score is the per-mode Hit.score (vector mode → cosine; rerank score when the rerank leg ran). The data already exists in the searcher — this is plumbing, not new computation.

2. (Optional, builds on 1) Derive a score-based OOD metric in the runner so it becomes a first-class, gateable output:

• Calibrate a cutoff C per run from the positive distribution (e.g. p10 of positive top-1 scores) — keeps it embedder-portable, no hardcoded absolute threshold.
• negative_satisfaction = fraction of negatives whose top-1 score < C (this is the "expect_none satisfaction" placeholder, now actually computable).
• separation_margin = median(positive top-1) − median(negative top-1) (informational).

Scope / notes

• Step 1 is independent of reranking — vector cosine already discriminates, so no reranker is required to measure OOD. Enabling a reranker improves the signal but does not by itself solve this: the score still has to be surfaced (this issue), and OOD rejection in production additionally needs a relevance cutoff consuming the score (retrieve never abstains today).
• Ideally the eval cutoff aligns with whatever relevance cutoff the production retrieve consumer uses, so a green gate reflects real production behavior.
• This pairs naturally with --retrieval all: the per-mode Hit.score (esp. the vector leg) is the right field to carry into the rows.

Context

Surfaced while building in-house OOD negatives (negatives-ood-v1) in the companion eval/data repo. The set passes (exit 0), but the "pass" is near-information-free: the eval output exposes only rank order, not whether the engine assigned any off-corpus doc a high relevance score. With scores surfaced, the same set becomes a real, calibratable (and eventually gateable) OOD-robustness measurement.