Skip to content

Scheduled maintenance pipeline for search eval runs and credibility-list refresh #309

Description

@quiet-node

Follow-up from the built-in search quality arc (feat/builtin-search-revamp). Two recurring-maintenance workflows so the eval corpus and the domain-credibility list never fossilize.

1. Credibility-list refresh (scheduled, automated PR)

  • A scheduled GitHub Action (monthly or quarterly) pulls the latest upstream license-clean sources (arosh CC0 copycat list, quenhus Unlicense SEO-spam/wiki-mirror lists), regenerates src-tauri/src/websearch/credibility_domains.txt, and opens a PR when the diff is non-empty.
  • Mirrors the existing Renovate engine-bump pattern (Keep the bundled llama.cpp engine current: scheduled review-gated bump PRs + engine regression CI gate #238): automation proposes, a human reviews and merges, updates ship with releases. No phone-home; the app itself never fetches lists at runtime (privacy stance unchanged).
  • PR body should include per-section counts and the diff of added/removed domains for reviewability.
  • Also periodically re-verify the 12 hard-drop hoax domains (dead domains are harmless cruft but should be pruned).

2. Eval runs (manual trigger first, schedule later)

  • Phase 1: a workflow_dispatch job that runs the live answer-capture harness (tests/live_answer_capture.rs) and uploads the answers JSONL as an artifact. Caveat to validate first: search engines heavily rate-limit GitHub runner IPs; if DDG/Mojeek block from runners, the harness stays a dev-machine tool and the workflow only runs the hermetic corpus checks.
  • Phase 2 (after phase 1 proves viable): the pairwise position-swapped LLM-judge step from docs/search-eval.md over two capture runs (baseline vs candidate), judge model and API budget to be decided. Dev-time tooling: exempt from the keyless/no-server product constraints.
  • Corpus growth stays human-curated: rows get added from real observed failures (the pattern that caught every bug so far), not auto-generated. Automation runs the exam; it does not write the exam.

Explicitly out of scope

  • Runtime/live list updates from the app (violates no-phone-home).
  • Auto-merging any generated PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions