Deterministic IDs for conversations / prompts

[tony]

Summary

agentgrep normalizes every backend into SearchRecord/FindRecord (src/agentgrep/records.py), but no record carries a stable per-record or per-conversation identity, and two subsystems already compute disagreeing ad-hoc identities to compensate. This issue promotes one deterministic, short, privacy-safe content id — at both prompt and conversation level — computed as a pure function of a normalized record, surfaced identically across CLI text/--json/--ndjson, the TUI, and MCP, and makes both existing identities projections of it. It is the foundation the rest of the cluster builds on: stable ids are what let you bookmark (#79), reference an export (#81), cache a similarity vector (#82/#69), and dedup consistently.

Problem & need

agentgrep has two live notions of "the same record," and they do not agree.

1. Engine dedupe key — record_dedupe_key() in src/agentgrep/_engine/orchestration.py returns the raw tuple (kind, agent, store, session_id or conversation_id or str(path), text). It carries the full prompt text and the raw absolute path string, and excludes role, adapter_id, timestamp, title, and model. Both execution drivers use it to collapse the same prompt seen across /resume or duplicate stores within one run.

2. MCP fingerprint — search_record_fingerprint() in src/agentgrep/mcp/refs.py is a sha256 over sorted-keys compact JSON of {kind, record_kind, role, agent, store, adapter_id, path (home-collapsed display path), timestamp, session_id, conversation_id, text_sha256}. It is base64url-wrapped as the opaque agref1: ref by make_search_ref (with agcur1: paging cursors), and re-derived and compared on drilldown by inspect_result.

These disagree on almost every axis: hashing (raw tuple vs sha256), fields (the fingerprint includes role/adapter_id/timestamp/record_kind; the dedupe key drops them), path handling (raw str(path) vs home-collapsed format_display_path), and text (a raw substring inside a tuple vs a sha256 digest). The consequence is concrete: the fingerprint keeps distinct two turns that share a session and text but differ in timestamp or role, while the dedupe key folds them; conversely the dedupe key can collapse across stores that the fingerprint separates by adapter_id. Neither is surfaced as a first-class, addressable handle.

What is missing today:

• No cross-surface id. Only the MCP path attaches identity — SearchRecordModel.from_record sets payload["ref"] = make_search_ref(...). serialize_search_record in src/agentgrep/cli/serializers.py, the text renderers, and the TUI emit no id at all. So an id exists only over MCP, only as a long opaque base64 token, and is absent from every headless/CLI surface an export or find-similar would need to cite.

• No conversation-level id. Conversation identity is the ad-hoc session_id or conversation_id or str(path) expression buried inside the dedupe key. session_id/conversation_id are backend-native, nullable, and inconsistent — usually a UUID from a file stem, parent directory, or in-file field, but Pi's conversation_id is a working-directory path, and the Cursor CLI prompt store and the VS Code inline-history store carry no native id at all (flat arrays keyed only by file mtime).

• The one field that mints spurious identity is timestamp. For several backends there is no native per-turn timestamp, so the record falls back to the file's mtime. Any id that folds in timestamp — the current fingerprint does — drifts when such a file is touched or rewritten, even though the content is unchanged. The fields that are stable across a re-scan are kind, agent, store, adapter_id, role, session_id/conversation_id, and the text itself.

• Two more schemes wait in unmerged workstreams. Only the two notions above are live on master. Two further text-hash identities are queued elsewhere: a separate unmerged on-disk DB cache workstream (its module is not on master) needs its own source_id/record_id derived over normalized text to key rows; and the insights graph (Insights graph similarity engine + non-gated transformers LLM defaults #69) keys its incremental cache on a text-only sha1(text)[:16]. Both derive identity from text alone and will disagree with the two live schemes, so landing either before the live notions are unified leaves agentgrep with four recipes. Publishing one canonical recipe now — for the live schemes to share and the unmerged ones to adopt — is cheaper than reconciling four later.

Where it fits

The id must be a pure function of an already-normalized record, so it stays headless-first and imposes no startup or per-search tax on users who never ask for it:

• Lazy, at the emit/render boundary. hashlib is already imported on both paths; deriving an id costs one sub-microsecond hash per surviving record, only when a record is serialized or rendered. No import-time cost, no hashing of the discovery corpus.
• Piggyback on work already done. The dedupe pass already touches full record text for every matched record; the canonical id can be computed there and reused, replacing the raw-text tuple with a fixed-width digest (which also shrinks the compare and memory cost of dedupe).
• Parity by construction. The same function feeds SearchRecordPayload (CLI --json/--ndjson), the text renderer, the TUI row, and the MCP model — so the four surfaces cannot drift. The opaque agref1: token stays as a superset that embeds the same content id.
• Opt-in materialization. Only when a user opts into the on-disk cache or similarity/insights (Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69) is the id persisted as a primary key; the default fast path never writes it.
• TUI. The hash is cheap and already runs in the engine, off the message pump; per ADR 0011 no bulk id computation should ever be added to a pump callable.

Design directions

A. One canonical id function; the two identities become projections

Add a dependency-light src/agentgrep/identity.py (records-adjacent per ADR 0010) exposing snake_case helpers: record_content_id(record) -> str, conversation_content_id(...) -> str, and short_id(content_id, *, chars=12) -> str. Promote the canonicalization already in refs.py (json.dumps(..., ensure_ascii=False, separators=(",", ":"), sort_keys=True)) into this one helper, and have record_dedupe_key() and the MCP fingerprint both call it instead of hand-rolling their own basis. The result is one recipe, one place to change it, and dedupe/refs that agree by construction. Surface a plain id (short) alongside the existing opaque ref in SearchRecordPayload/FindRecordPayload and serialize_*; the MCP model keeps ref and gains id.

B. Canonical serialization, hash, and short-id encoding

Two viable primitives over the canonical bytes:

• hashlib.blake2b(canonical, digest_size=32, person=b"agentgrep.id.v1") — stdlib, with keyless domain separation via the 16-byte person field and direct width tuning via digest_size, so prompt/conversation/find ids built from identical bytes cannot collide.
• Keep sha256(canonical) truncated — already shipping in refs.py, and the git/OCI-family default, but not personalizable (domain separation must live entirely in the payload).

Either way, fold an explicit type tag ("prompt"/"history"/"conversation"/"find") into the canonical payload so the separation is portable even when the digest is re-hashed in another language (an export consumer). Recommended: the in-payload type tag is the portable mechanism, with person=/blake2b as a stdlib belt-and-suspenders; if minimizing churn matters more, keeping sha256 and adding only the short-id and unification layers is a legitimate lower-risk path. While unifying, fix a latent robustness bug: refs.py encodes text with plain record.text.encode("utf-8"), which raises on lone surrogates that survive imperfect store decoding — the shared helper should use errors="surrogatepass".

Keep the full digest (256 bits) as the equality/dedupe/storage value; surface a short display id via base64.b32hexencode(...).decode().lower() truncated to ~12 chars (extended-hex base32 packs 5 bits/char, is case-insensitive and copy/tmux/filename-safe, and preserves byte order for prefix scans). Resolve a short id git-style: match it as a unique prefix against the local corpus and auto-widen on the rare collision. A 12-char prefix is ~60 bits, so the birthday bound stays negligible at a single user's history scale. This is the "truncate a wide hash then base32" pipeline that Nix store paths and Docker/OCI digests already use.

C. What the id means: content-core vs located (and the timestamp fix)

The real policy fork is what the id addresses. A content-core id over {type, agent, role, normalized_text, model} matches the engine's current "same text is the same prompt" dedupe intent and is stable across machines and file moves. A located id additionally folds {store, adapter_id, display_path, session_id, timestamp} to address one specific on-disk occurrence — what the MCP fingerprint does today. They answer different questions and neither is wrong. Recommended default: compute both — the content-core id drives dedupe and near-duplicate collapse in src/agentgrep/ranking.py; the located id is the addressable, surfaced handle. Surface this fork explicitly rather than silently pick one.

Because timestamp is mtime-derived for several backends, a located id that folds it drifts across re-scans. Two options: exclude timestamp entirely, or add a timestamp_source discriminator so only native (non-mtime) timestamps participate. Excluding is simpler and matches the "content identity should survive a touch" intuition; the discriminator keeps timestamp as a tiebreaker only where it is actually meaningful.

D. Conversation identity: a durable anchor plus a Merkle content hash

No conversation id exists today. Derive two fields, because bookmarking and dedup want opposite stability:

• conversation_id as a durable anchor: the upstream session_id when the store provides a stable UUID, else a seed-anchored hash of the first stable member so the id survives new turns being appended. This is what Bookmark prompts / conversations #79 bookmarks and Export prompts and conversations #81 export refs cite — it must not change as the conversation grows. (Bookmark prompts / conversations #79's bookmark target is addressable at both levels: this conversation anchor and the per-record id.)
• content_hash as a one-level Merkle root over the ordered member content-ids (blake2b(b"\n".join(member_ids)), git's tree-over-blobs shape). This changes on any edit or append, so Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69 can use it to invalidate cached vectors and dedup identical threads.

Graceful degradation is mandatory: for stores with no native id (Cursor CLI prompt history, VS Code inline history) and Pi's cwd-as-conversation_id, the durable anchor must fall back deterministically (for example, home-collapsed display path plus an ordinal) rather than emit None or leak a path. Optionally, capture the native per-message ids adapters currently discard (Claude uuid, Gemini/OpenCode message ids, Grok/Codex content_hash) into an optional native-id field to strengthen determinism where upstream already provides one.

Prior art & inspiration

• git content addressing — git/git@v2.49.0 object-file.c: format_object_header prepends a typed "<type> <size>\0" header before hashing, a portable, language-agnostic domain separator. More importantly, borrow git's short-hash UX — unique-prefix display, ambiguity error, on-demand widening — for the CLI/TUI short id.
• BLAKE2 personalization — python/cpython@v3.15.0b1 Modules/blake2module.c: exposes person/salt (16 bytes) and digest_size (max 64), giving free keyless domain separation and direct width tuning from the stdlib.
• base32 short encoding — python/cpython@v3.15.0b1 Lib/base64.py: b32hexencode(..., padded=False) yields a copy/filename-safe, order-preserving short id at 5 bits/char.
• namespace UUIDs — python/cpython@v3.15.0b1 Lib/uuid.py: uuid5(namespace, name) is a deterministic, standards-shaped projection worth offering for Export prompts and conversations #81 export interop so downstream tools ingest a familiar id.
• OCI descriptor digests — opencontainers/image-spec@v1.1.0 descriptor.md: the algorithm:hex two-part digest form is the precedent for keeping an explicit algorithm discriminator so the hash can migrate unambiguously.
• multihash — multiformats/multihash@b43ec10 README.md: the self-describing <code><length><digest> framing; apply it minimally as a short algorithm/version tag (agentgrep already versions with agref1:/v:), not the full stack.

Relationship to other issues

• Foundation for Find similar prompts and conversations #82/Export prompts and conversations #81/Insights graph similarity engine + non-gated transformers LLM defaults #69. Find similar prompts and conversations #82 (find-similar) needs a short stable seed handle for --similar-to <id> / find_similar(ref=...); a seed-by-text form can ship first, but seed-by-id waits on this. Export prompts and conversations #81 (export) needs each exported unit to carry one stable, re-importable id so exports are idempotent and diffable. Insights graph similarity engine + non-gated transformers LLM defaults #69 (insights graph) currently keys its incremental cache on a text-only sha1(text)[:16] — one of the two unmerged text-hash schemes noted above, and the weakest of them; it should key on the unified id instead.
• Bookmark prompts / conversations #79 (bookmarks) references a durable conversation anchor; the two-field split in Design D exists precisely so a bookmark survives the conversation growing.
• Similarity-adjacent (search: WRatio scoring is meaningless for short queries against long records #22/Re-add fuzzy with a faithful fzf algorithm port #34/search: Use pretty snippet/highlight infrastructure instead of crude truncation #21). The removed WRatio filter (search: WRatio scoring is meaningless for short queries against long records #22) and its faithful fzf re-port (Re-add fuzzy with a faithful fzf algorithm port #34) operate on term-vs-record matching; the dead collapse_near_duplicates in ranking.py is record-vs-record. The content-core id gives that collapse a cheap exact/near-dup key.
• ADRs. ADR 0004 governs the engine where record_dedupe_key lives. ADR 0006 (public CLI/MCP surface contract) makes a new id field/flag compatibility-sensitive the moment it ships — it wants one canonical registry-backed description plus drift tests across parser help, docs, and MCP schema, and a versioned scheme, so the id policy should extend ADR 0006 or land as its own ADR. ADR 0005 (insights ladder) is where the persisted-vector-keyed-by-id case lives; the unmerged on-disk DB cache workstream is one of the two unmerged text-hash schemes to converge on the unified id; ADR 0011 keeps id computation off the TUI pump.

Open questions

1. Ship the content-core id, the located id, or both (dedupe on one, address on the other)?
2. Timestamp: exclude it, or gate it behind a timestamp_source discriminator so only native timestamps participate?
3. Path in the id: the current fingerprint folds the display path, so a moved or renamed store mutates the id. Do we want cross-machine, path-independent stability (content plus session anchored) at the cost of not distinguishing two identical prompts in different stores?
4. Hash and width: blake2b(person=) vs keeping sha256 (lower churn, git/OCI-family default); the short-prefix default width, and base32 vs hex.
5. Conversation anchor for id-less stores (Cursor CLI prompt history, VS Code inline, Pi cwd): what deterministic fallback, and how stable is it across appends?
6. Do we capture the native per-message ids adapters currently drop into an optional native-id field, and does inspect_result gain a bare-id drilldown handle (a v:2 ref) so the public id resolves directly?