feat: multi-provider real LLM evaluation pipeline

[Neal006]

Closes #8

Summary

Replaces the implicit assumption that "retrieved text = correct answer" with a genuine end-to-end LLM evaluation pass. Content-based metrics remain as fast, zero-cost proxies; the LLM pass is additive and activates automatically when any provider key is present.

• utils/providers.py — unified LLMProvider ABC with five concrete backends (Groq, OpenAI, Anthropic, OpenRouter, Ollama), auto-detection priority chain, and _clean_messages() normalisation helper
• evaluation/metrics.py — llm_recall_at_t() (answer+judge pipeline) and llm_temporal_drift() (old vs new value after update)
• evaluation/benchmark.py — CheckpointResult gains llm_recall, llm_drift, has_llm_eval; run_benchmark() accepts optional provider
• main.py — --llm, --provider, --list-providers flags; outputs three tables: Content Recall, LLM Recall, Gap
• dashboard.py — provider selector auto-detects available keys; tabbed recall chart (Content / LLM / Gap); KPI cards show LLM recall delta
• .env.example — all five provider keys documented inline

How the LLM Eval Works

For each fact at each checkpoint:

  1. ANSWER  memory.get_context(fact.query) → [context messages]
             provider.chat([system] + context + [question])
             → answer string

  2. JUDGE   provider.chat([judge_system, "Q:... Expected:... Got:..."])
             → "correct" | "wrong"

  llm_recalled = (verdict == "correct")

The gap between content recall and LLM recall is the core finding — a large positive gap means the backend retrieves the right text but the LLM still fails to extract the correct answer.

Test Plan

• python tests/test_imports.py — all imports OK, provider registry validated
• python tests/test_pipeline.py — all 14 tests pass (no API key needed)
• python main.py --list-providers — shows provider status cleanly
• python main.py — content-only mode unchanged, no regression
• python main.py --llm --provider groq — requires GROQ_API_KEY

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds an optional, end-to-end LLM evaluation pass (answer + judge) to complement existing content-based retrieval metrics, with support for multiple LLM providers and UI/CLI surfacing of Content vs LLM recall and the resulting “gap”.

Changes:

• Introduces a unified LLMProvider abstraction with Groq/OpenAI/Anthropic/OpenRouter/Ollama backends and auto-detection.
• Adds LLM-based evaluation metrics (llm_recall_at_t, llm_temporal_drift) and threads optional provider support through the benchmark runner and CLI output.
• Updates the Streamlit dashboard to optionally run LLM eval and visualize Content / LLM / Gap recall.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
utils/providers.py	Adds multi-provider LLM abstraction, registry, auto-detection, and message normalization.
evaluation/metrics.py	Implements the two-stage LLM recall and LLM temporal drift metrics.
evaluation/benchmark.py	Threads optional provider through benchmark execution and JSON display output.
main.py	Adds --llm, --provider, --list-providers and prints Content/LLM/Gap tables.
dashboard.py	Adds provider detection/selection and tabbed recall charts for Content/LLM/Gap.
tests/test_imports.py	Extends import smoke test to include provider registry and LLM metric imports.
CHANGELOG.md	Documents the new multi-provider LLM evaluation feature.
.env.example	Documents environment variables for all supported providers.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.