Find similar prompts and conversations

[tony]

Summary

Add a find-similar retrieval primitive: seed from a prompt or a conversation and get its nearest neighbors across every backend, at CLI, TUI, and MCP parity. It ships as a tiered ladder — a zero-dependency stdlib default that runs only on an explicit call, an optional lexical-index tier, and an optional embedding tier that reuses the #69 insights infrastructure. Find-similar is the general-purpose, always-available face of the #69 graph engine: the same nearest-neighbor question, without the report pipeline or a model download.

Problem & need

agentgrep can locate records by term, field predicate, and regex, but it cannot answer "what else looks like this one?" There is no similarity primitive on the current surface. The one that existed — a rapidfuzz WRatio fuzzy filter — was removed as meaningless for short-needle-vs-long-body matching (#22), with a faithful fzf re-port tracked separately (#34). src/agentgrep/ranking.py still carries an unused pairwise near-duplicate collapser (collapse_near_duplicates in src/agentgrep/ranking.py) referenced only by its own module and tests — the closest existing "similarity between two record bodies" code, sitting dead.

The user need is retrospective and concrete. A single store accrues thousands of prompts across Codex/Claude/Cursor/Gemini and the recurring question is "have I asked this before?", "show me the conversation like this one", "which near-duplicates should I collapse?". Term search cannot express "near this record" — that needs a seed record and a notion of distance.

The blocker is identity. SearchRecord (src/agentgrep/records.py) has no stable per-record or per-conversation id, and session_id/conversation_id are nullable and inconsistent across backends. The only handle that round-trips back to a live record today is the opaque, MCP-only fingerprint (search_record_fingerprint in src/agentgrep/mcp/refs.py, wrapped as an agref1: token by make_search_ref). The engine additionally dedups on a weaker, text-inclusive tuple (record_dedupe_key in src/agentgrep/_engine/orchestration.py). Seeding find-similar "by id" therefore depends on #80 promoting one canonical id to a short, documented, cross-surface handle.

Where it fits

Find-similar is one pure scorer consumed by three frontends, so parity is structural rather than duplicated:

• Headless/CLI: a similar verb (or --similar-to <id> / --similar-text "…" on search) rendered in text/--json/--ndjson, keeping the pydantic-free JSON fallback every other verb has.
• TUI: a /similar command and a "similar to selected" action, pi-minimal — an id-addressable neighbor list with the score inline, no chrome.
• MCP: a find_similar tool returning the existing response envelope plus agcur1: paging.

Performance posture is opt-in and tiered. The default tier runs only on an explicit find-similar call, over a bounded candidate set narrowed by the existing SearchScope in src/agentgrep/records.py, with no new dependency, no index build, and no import-time cost — users who never call it pay nothing. The lexical-index and embedding tiers are strictly opt-in; the embedding tier rides the #69 / ADR 0005 model ladder, so importing agentgrep never imports numpy/torch/a model.

Both units are in scope. Prompt-level similarity maps directly onto scope=prompts; conversation-level similarity maps onto scope=conversations (CONVERSATION_STORE_ROLES in src/agentgrep/records.py) but needs a conversation-level id and aggregate — again #80 — because a conversation's vector is derived from its member turns.

Design directions

Seeding splits into two forms with different dependencies:

• seed-by-id (similar <id>) references a stable handle, so it depends on Deterministic IDs for conversations / prompts #80.
• seed-by-text (--similar-text "<pasted prompt>") needs no stable handle and can land first.

Both feed one scorer whose home is beside collapse_near_duplicates in ranking.py, e.g. score_by_similarity(seed_text, records, *, metric, top_k), so CLI/TUI/MCP share it and the TUI can @offload it.

Tier 0 — stdlib, instant, the default. Runs on the records the engine already collected for a scope-narrowed candidate set (a result page, one conversation's neighbors, or a cheap prefilter's top-k):

1. collapse exact/near duplicates for free using the text_sha256 already computed by search_record_fingerprint (src/agentgrep/mcp/refs.py);
2. prefilter survivors with word/char k-gram shingle Jaccard (pure set ops);
3. rank with difflib.SequenceMatcher.ratio(), using its real_quick_ratio()/quick_ratio() upper-bound early-exit ladder to keep the constant small.

import difflib, import hashlib — no new dependency, no index. datasketch MinHash+LSH is an optional escalation for corpus-scale near-dup with a pure-Python fallback, so the default never hard-depends on it. Tradeoff: character-diff and bag-of-shingles catch near-duplicate wording, not paraphrase, and are bounded to a candidate set rather than the whole corpus.

Tier 1 — lexical corpus index, still no model (opt-in). For "similar across everything" without a per-call quadratic scan, build an opt-in SQLite FTS5 "more-like-this" index (stdlib sqlite3) keyed by the #80 id: take the seed's top terms, run them as an FTS5 MATCH ranked by bm25() for a corpus-wide shortlist, then rerank in Python. This is new persistent infrastructure — there is no such index on the current surface — and is the natural place to share one on-disk artifact with #80's identity index. Tradeoff: lexical, not semantic; needs a build plus incremental maintenance; only pays off for corpus-wide queries.

Tier 2 — semantic, opt-in, reuses #69. The nearest-neighbor machinery #82 wants mostly already exists inside the #69 insights graph engine (WIP), trapped behind the report pipeline: per-node k-NN over persisted vectors driving its similar-prompts and forgotten-similar sections. #82 lifts that into a reusable retrieval callable. For find-similar's retrieval tier the lightweight default is model2vec/potion static embeddings (numpy, no torch, small model, millisecond CPU encode), with sentence-transformers as the heavy escalation; #69's graph engine may pick a different default (its encoder default is an open question there). Vectors cache keyed by the #80 id and k-NN serves from sqlite-vec so the embeddings, the Tier-1 index, and the #69 graph share one on-disk DB; usearch/hnswlib are large-corpus escalations. Per ADR 0005 this whole tier sits behind a capability probe with a typed failure reason and degrades to Tier 0/1, never a traceback.

Surface wiring (existing attachment points).

• CLI: a similar subparser modeled on search_parser in src/agentgrep/cli/parser.py reusing add_common_agent_options + --scope + --limit, or --similar-to/--similar-text flags on search; a run_similar_command mirroring run_search_command in src/agentgrep/cli/render.py that gathers candidates through the normal query path, then calls score_by_similarity where the two rank_search_records call sites sit today in src/agentgrep/cli/render.py; a new dispatch branch in main() (src/agentgrep/__init__.py). Introduce the ADR 0006 canonical --format here rather than more boolean flags.
• MCP: a find_similar tool registered in search_tools.register() (src/agentgrep/mcp/tools/search_tools.py), returning the shared stats/page/status/diagnostics envelope with agcur1: cursor paging and resolving a seed ref via the existing inspect_result re-scan-and-compare pattern; add the seed-text argument name to _SENSITIVE_ARG_NAMES (src/agentgrep/mcp/middleware.py) so it is redacted in audit logs.
• TUI: a /similar entry in SLASH_COMMANDS (src/agentgrep/ui/commands.py) and a "similar to selected" action on BrowseWorkflow (src/agentgrep/ui/workflows/browse.py); the scoring pass runs on an @offload worker per ADR 0011, never on the message pump; neighbors render in a pi-minimal id-addressable list with the score inline.

Prior art & inspiration

• difflib.SequenceMatcher — https://github.com/python/cpython/blob/v3.15.0b1/Lib/difflib.py — the ratio()/quick_ratio()/real_quick_ratio() early-exit ladder plus autojunk: the zero-dependency Tier-0 scorer over a bounded candidate set.
• ripgrep searcher — https://github.com/BurntSushi/ripgrep/blob/15.1.0/crates/searcher/src/searcher/mod.rs — streaming search with no persistent index: the posture the Tier-0 default must match (instant, zero build, zero startup tax).
• datasketch MinHashLSH — https://github.com/ekzhu/datasketch/blob/v1.10.0/datasketch/lsh.py — sub-linear near-duplicate bucketing for the corpus-scale escalation, optional dep with a pure-Python fallback.
• fzf FuzzyMatchV2 — https://github.com/junegunn/fzf/blob/v0.72.0/src/algo/algo.go — Smith-Waterman-style term-vs-record scoring that belongs to Re-add fuzzy with a faithful fzf algorithm port #34's interactive matching, distinct from Find similar prompts and conversations #82's record-vs-record similarity — cited to draw the line.
• model2vec StaticModel.encode — https://github.com/MinishLab/model2vec/blob/add2daf/model2vec/model.py — pure-numpy static embeddings, no torch: the lightweight non-gated default encoder for find-similar's retrieval tier.
• sentence-transformers retrieval — https://github.com/huggingface/sentence-transformers/blob/v5.5.1/sentence_transformers/util/retrieval.py — semantic_search for the heavy opt-in encoder path (torch plus model download).
• sqlite-vec — https://github.com/asg017/sqlite-vec/blob/v0.1.9/README.md — single-file vec0 k-NN over stdlib sqlite3 so Tier-2 vectors share one on-disk DB with the lexical index and the Insights graph similarity engine + non-gated transformers LLM defaults #69 graph.
• usearch — https://github.com/unum-cloud/usearch/blob/v2.25.3/python/usearch/index.py — compact memory-mapped ANN, a lighter large-corpus escalation than faiss.
• hnswlib — https://github.com/nmslib/hnswlib/blob/v0.8.0/hnswlib/hnswalg.h — incremental HNSW, the RAM-resident escalation when ANN latency dominates.
• click ParamType — https://github.com/pallets/click/blob/8.4.1/src/click/types.py — convert()/self.fail() as the template for one framework-neutral resolve_record_id shared by the CLI validator and the MCP request model.
• rich Tree — https://github.com/Textualize/rich/blob/v15.0.0/rich/tree.py — a one-level tree (root=seed, children=neighbors with id and score) that degrades to ASCII when piped, for headless CLI text.
• textual OptionList — https://github.com/Textualize/textual/blob/v8.2.6/src/textual/widgets/_option_list.py — Option(id=…) id-addressable rows so TUI selection uses the same Deterministic IDs for conversations / prompts #80 id as CLI/MCP.
• textual ModalScreen — https://github.com/Textualize/textual/blob/v8.2.6/docs/examples/guide/screens/modal03.py — ModalScreen[bool] + dismiss() confirm pattern (strip the header/box to stay pi-minimal) for a "seed from this" / "export these" confirm.
• fastmcp ToolResult — https://github.com/jlowin/fastmcp/blob/v3.4.2/fastmcp_slim/fastmcp/tools/base.py — structured output plus ResourceLink/EmbeddedResource content blocks, the shape a find_similar (and Export prompts and conversations #81 export) tool returns.

Relationship to other issues

• Deterministic IDs for conversations / prompts #80 (deterministic IDs) — the foundation. seed-by-id references the promoted cross-surface id; the two identities Deterministic IDs for conversations / prompts #80 unifies (record_dedupe_key, search_record_fingerprint) become the one basis that the similarity vector cache and the dedupe key agree on. seed-by-text can ship before Deterministic IDs for conversations / prompts #80; seed-by-id cannot.
• Insights graph similarity engine + non-gated transformers LLM defaults #69 (insights graph) — Find similar prompts and conversations #82 is its general-purpose face. The Tier-2 encoder, vector store, and k-NN are Insights graph similarity engine + non-gated transformers LLM defaults #69's; Find similar prompts and conversations #82 exposes the nearest-neighbor query as a standalone retrieval primitive across CLI/TUI/MCP without the report/SKILL.md pipeline, gated behind the same ADR 0005 ladder.
• Export prompts and conversations #81 (export) — a similarity result set is an exportable artifact; a similarity graph (neighbors keyed by Deterministic IDs for conversations / prompts #80 id) is a natural export/HTML target once both land.
• Bookmark prompts / conversations #79 (bookmark) — bookmark a neighbor by the same id you seeded from.
• Re-add fuzzy with a faithful fzf algorithm port #34 / search: WRatio scoring is meaningless for short queries against long records #22 / search: Use pretty snippet/highlight infrastructure instead of crude truncation #21 (fuzzy/similarity-adjacent) — Find similar prompts and conversations #82 restores similarity as an explicit record-vs-record primitive over a scope-narrowed candidate set, deliberately not the removed full-corpus WRatio tax (search: WRatio scoring is meaningless for short queries against long records #22) and distinct from Re-add fuzzy with a faithful fzf algorithm port #34's interactive term-vs-record fzf matching.
• ADRs — 0004 (reuse planning/execution/result-stream to gather candidates), 0005 (opt-in model ladder plus typed capability probe for Tier 2), 0006 (public-surface contract: new verb/flag/tool/response-field and --format are compatibility-sensitive and registry-drift-tested), 0011 (TUI scoring on @offload, never the pump).

Open questions

1. Conversation-similarity unit: mean of member-prompt vectors, a Insights graph similarity engine + non-gated transformers LLM defaults #69 one-line-summary vector, or first/last-turn anchoring — and how does that interact with the conversation-level id Deterministic IDs for conversations / prompts #80 must define?
2. Default candidate corpus for similar <id>: same scope only, same agent only, or all backends? Does scope/agent narrowing help latency at the cost of cross-tool recall?
3. Ship boundary: does a first cut land seed-by-text alone (no Deterministic IDs for conversations / prompts #80), or is the whole verb gated on Deterministic IDs for conversations / prompts #80 so there is one coherent id-first surface?
4. Tier-0 default metric: SequenceMatcher.ratio() (char-diff) vs token-set Jaccard (order-free) as the shipped default — and do we expose a --metric knob or keep it single and predictable?
5. Primary knob: top-k vs a score threshold, the default k, and how the empty/weak-neighbor state is presented so weak matches are not dressed up as strong ones.
6. Does the Tier-1 FTS5 index share one on-disk artifact with Deterministic IDs for conversations / prompts #80's identity index, and is auto-building it on first similar an acceptable one-time cost or must it be an explicit sync step?