Skip to content

AGENTS.md

Latest commit

 

History

History
200 lines (142 loc) · 13.3 KB

AGENTS.md

File metadata and controls

200 lines (142 loc) · 13.3 KB

ripgit — agent context

This file is for AI coding agents. It covers the architecture, key design decisions, important constraints, and gotchas that are not obvious from reading the code in isolation.

What this is

A self-hosted git server running on Cloudflare Durable Objects. Each repository is one DO with a SQLite database. The Worker entry point routes /:owner/:repo/* to the right DO by name. An optional TypeScript auth worker in examples/github-oauth/ sits in front via Service Binding.

Repository layout

src/
  lib.rs        Worker entry point. Routes /:owner/:repo/* to DO, /:owner/ to profile page.
                Also contains Actor/auth helpers, check_write_access, list_repos (KV query),
                handle_issue_action (issue/PR POST handler).
  pack.rs       Git pack file parser (two-pass: index + resolve). ResolveCache (Arc-based,
                budget-limited). Pack generator for upload-pack.
  git.rs        Smart HTTP protocol handlers: receive-pack (push) and upload-pack (fetch/clone).
                Pack size gate (50 MB). Calls into pack.rs and store.rs.
  store.rs      All SQLite operations: blob storage (xpatch delta compression), commit/tree
                storage, FTS index rebuild, commit graph build, ref management.
  api.rs        Read-only JSON API endpoints. parse_search_query() handles @prefix: syntax.
  diff.rs       Recursive tree diff, line-level unified diffs (similar crate), commit comparison.
  web.rs        Server-rendered HTML for all 9 web pages. layout() is the shared shell.
                Key helpers are pub(crate): layout(), html_escape(), html_response(),
                format_time(), render_markdown(), resolve_default_branch(), render_file_diff().
  schema.rs     Schema initialisation (runs in DO::new). All CREATE TABLE/INDEX statements.
  issues.rs     Issues and PR storage + merge logic. CRUD functions, three-way tree merge,
                BFS merge-base finder, git object serialization (SHA-1 via sha1_smol),
                parse_form() URL-decode utility.
  issues_web.rs Server-rendered HTML pages for issues and PRs. Uses web::layout() and helpers.

examples/github-oauth/
  src/index.ts  Auth worker: GitHub OAuth flow, session cookies, agent tokens, forwards to
                ripgit via Service Binding with X-Ripgit-Actor-* headers.
  src/types.ts  ActorProps, Env types.
  src/github.ts GitHub OAuth utilities.

scripts/
  push-test.sh  Incremental push script for large repos. Splits packs at 30 MB.

Key constraints

DO SQLite authorizer — Cloudflare's DO SQLite blocks PRAGMA statements and other privileged operations. Use sql.database_size() (the SqlStorage method) for DB size — raw PRAGMA page_count/PRAGMA page_size will return SQLITE_AUTH: not authorized. All regular SELECT/INSERT/UPDATE/DELETE work fine.

100 parameter limit — DO SQLite allows at most 100 bound parameters per statement. Tree entry INSERTs are batched 25 per statement (4 params each = 100). Don't exceed this limit when adding batch operations.

2 MB row limit — DO SQLite rows can't exceed 2 MB. Large blobs that compress to over 2 MB are chunked across a blob_chunks overflow table and reassembled transparently. See store::store_blob and store::read_blob.

128 MB DO memory limit — the pack processing budget is designed around this. Pack body limit: 50 MB. ResolveCache budget: 20 MB. Pack generator buffers all objects before writing, so fetch/clone of large repos is a memory concern.

Response::redirect requires absolute URLs — Response::redirect("/path", 302) fails in Cloudflare Workers (Rust). Use Response::error("", 302) and set the Location header manually, or build a new Response(null, { status: 302, headers: { Location: url } }) equivalent. There is a unauthorized_401() helper in lib.rs showing this pattern.

DO names are permanent — the DO is named "{owner}/{repo}". This is its storage identity forever. Renaming a user or repo would orphan the DO. Don't add logic that renames DOs.

Authentication model

The auth worker sets trusted headers before calling ripgit via Service Binding:

X-Ripgit-Actor-Name     GitHub username (or agent owner's username for agents)
X-Ripgit-Actor-Id       stable ID: "github:12345" or "agent:uuid"
X-Ripgit-Actor-Kind     "user" | "agent"
X-Ripgit-Actor-Scopes   comma-separated: "repo:read,repo:write,admin,..."
X-Ripgit-Actor-Owner    for agents: the owning user's actorId

actor_from_request() in lib.rs reads these headers. X-Ripgit-Actor-Name is what ripgit uses for ownership checks — it must equal the owner segment in the URL for writes to be allowed.

Agent tokens — when an agent token is created, ownerActorName (the owner's GitHub username) is stored alongside actorName (the token display name). forwardToRipgit in the auth worker sets X-Ripgit-Actor-Name to ownerActorName for agents, not actorName. This is critical — without it, check_write_access would compare the token name against the repo owner and always fail.

The auth worker is optional. If no actor headers are present, all reads are allowed and all writes return 401.

Storage model

Blobs (xpatch delta compression)

Blobs are grouped by file path (blob_group table). Within a group, versions are stored as:

  • Keyframes (is_keyframe = 1): full content, zlib-compressed. Every 50th version.
  • Deltas (is_keyframe = 0): xpatch diff from previous version.

store::store_blob checks if the content is identical to the previous version (skip), otherwise computes a delta. store::read_blob walks back through the chain to the nearest keyframe, then forward-applies deltas.

If a compressed keyframe exceeds 2 MB, it's split into blob_chunks rows. The threshold is CHUNK_SIZE = 1_900_000 bytes.

Commits and trees

Commits are stored both:

  1. Parsed — commits table (hash, message, author, time, tree_hash), commit_parents table, trees table (path per row), refs table — used for queries, web UI, API.
  2. Raw — raw_objects table (hash → raw bytes) — for byte-identical fetch/clone.

Commit graph (binary lifting)

commit_graph table enables O(log N) ancestor lookups. Level 0 = direct first-parent. Level k = ancestor at distance 2^k. Used for push validation. Rebuilt via PUT /:owner/:repo/settings/rebuild-graph or PUT /:owner/:repo/admin/rebuild-graph.

Issues and pull requests

issues table (unified): id, number (sequential per-repo), kind ("issue" or "pr"), title, body, author_id, author_name, state ("open" / "closed" / "merged"), source_branch, target_branch, source_hash, merge_commit_hash, timestamps.

issue_comments table: id, issue_id, author_id, author_name, body, timestamps.

Auth: any authenticated user can create issues/PRs and comment. Only the issue/PR author or repo owner can close/reopen. Only the repo owner can merge PRs.

Merge commit algorithm (issues::merge_pr):

  1. BFS merge-base finder (find_merge_base) — walks commit_parents from both branch HEADs
  2. flatten_tree — recursively reads trees table into a flat HashMap<path, (mode, blob_hash)>
  3. merge_three_way — three-way file-level merge; returns conflict error listing paths if any file diverged in both branches
  4. build_tree_from_files — builds new tree objects bottom-up; calls store::store_tree which also writes to raw_objects
  5. serialize_commit_content + git_sha1("commit", ...) — creates merge commit bytes and SHA-1
  6. store::store_commit(..., bulk_mode=false) — persists commit, updates commit_graph and fts_commits
  7. Updates refs and sets issues.state = "merged" + merge_commit_hash

Fast-forward case (target is ancestor of source): merged tree = source tree directly, no three-way merge needed.

FTS5

Three FTS5 virtual tables:

  • fts_head — (path UNINDEXED, content) — file content at HEAD. Rebuilt incrementally on each push (diff engine detects changed files). Files >1 MiB are skipped.
  • fts_commits — (hash UNINDEXED, message, author) — all commit messages.

Symbol-heavy queries (containing ., _, (, :) bypass FTS5 and use INSTR for exact substring matching. See store::search_files_fts.

Routing

Worker entry (lib.rs::fetch):

  • /:owner/ → list_repos(env, owner).await then web::page_owner_profile
  • /:owner/:repo/* → DO stub named "{owner}/{repo}"
  • / → health JSON

DO handler (lib.rs::Repository::fetch):

  • Parses owner = parts[0], repo_name = parts[1], action = parts[2]
  • Calls actor_from_request(&req) early — actor is passed to all handlers
  • Git protocol: info/refs, git-receive-pack, git-upload-pack
  • JSON API: refs, file, search, stats, log, commit, tree, blob, diff, compare
  • Web UI: "" (home), commits, log (alias), tree, blob, raw, search-ui, settings
  • Issues: GET issues (list), GET issues/new, GET issues/:n, POST issues (create), POST issues/:n/comment, POST issues/:n/close, POST issues/:n/reopen
  • PRs: GET pulls (list), GET pulls/new, GET pulls/:n, POST pulls (create), POST pulls/:n/comment, POST pulls/:n/close, POST pulls/:n/reopen, POST pulls/:n/merge (owner only)
  • Admin: PUT admin/... — set-ref, config, rebuild-fts, rebuild-graph, rebuild-fts-commits
  • Settings actions: POST settings/rebuild-graph, settings/rebuild-fts, etc.

Web UI (web.rs)

layout(title, owner, repo_name, default_branch, actor_name, content) generates the shared HTML shell. It takes actor_name: Option<&str> and uses actor_name == Some(owner) to decide whether to show the Settings tab and owner-specific UI.

Two-row header:

  • Row 1 (.global-nav): ripgit logo left, username + sign out right
  • Row 2 (.repo-bar): owner/repo breadcrumb, search input, then Code/Commits/Issues/PRs/Settings tabs right-anchored

Content negotiation in the DO handler: wants_html is checked for routes that can return either HTML or JSON (log, commit, tree/blob by hash, diff). HTML-only routes ("", commits, tree/:ref/, blob/:ref/, search-ui, settings, raw) don't check it.

Response::redirect workaround — in Rust workers, use:

let mut resp = Response::error("", 302)?;
resp.headers_mut().set("Location", &url)?;
Ok(resp)

Streaming pack parser (pack.rs)

Two-pass approach:

  1. Index pass — walks pack bytes, decompresses to a sink (discards data), records PackEntryMeta (offset, type, size, delta base). O(1) memory.
  2. Resolve pass — for each entry, decompresses from pack bytes + resolves delta chains. ResolveCache (Arc<[u8]>, 20 MB budget) caches shared bases.

Pack size gate — packs over 50 MB are rejected before processing with a pkt-line ng response (not an HTTP error, which would confuse git).

ResolveCtx — bundles cache: &mut ResolveCache and external: &ExternalObjects (blobs already stored in SQLite, for thin packs). Passed by &mut through resolve_entry to avoid parameter explosion.

Known gotchas / past mistakes

  • SQLITE_AUTH — raw PRAGMA SQL is blocked. Use sql.database_size() for DB size. Never use pragma_page_count() or pragma_page_size() as table-valued functions.
  • SqlStorageValue::Null — DO SQLite's Rust bindings throw "unrecognized JavaScript object" when passing SqlStorageValue::Null as a query parameter. Store empty strings "" for absent optional text fields and map back to None on read via a nonempty() helper. NULL in result sets (from columns with no DEFAULT) deserializes fine into Option<String> — the issue is parameter binding only.
  • Response::error("", 302) is unreliable — creates an "unrecognized JavaScript object" error on some Cloudflare runtimes. Use make_redirect(req_url, path) which constructs an absolute URL and calls Response::redirect(url) instead. See lib.rs.
  • Agent actorName vs owner name — the token display name is not the owner's username. Always store and forward ownerActorName for agent tokens in the auth worker.
  • Response::redirect relative URLs — Url::parse("/path") fails because relative URLs can't be parsed standalone. Always construct an absolute URL from the request origin: format!("{}{}", req_url.origin().ascii_serialization(), path).
  • url.pathname percent-encoding — Cloudflare Workers preserves %3A in pathnames rather than decoding to :. Always decodeURIComponent() path segments before using them as KV keys.
  • DO name stability — once a repo is pushed, the DO name is {owner}/{repo} forever. Never add rename logic that would change the DO name — it would create a new empty DO and orphan the old one.

Development

# Build
cargo build --target wasm32-unknown-unknown

# Run both workers locally (auth on :8787, ripgit as service binding)
cd examples/github-oauth && npm run dev:full

# Run ripgit alone (no auth, all writes open)
wrangler dev

# Push a test repo with auth
./scripts/push-test.sh -u username -t TOKEN -w http://localhost:8787 -r /path/to/repo

After a bulk push, rebuild indexes via the Settings page (/:owner/:repo/settings) or curl:

curl -H "Authorization: Bearer TOKEN" -X PUT http://localhost:8787/owner/repo/admin/rebuild-graph
curl -H "Authorization: Bearer TOKEN" -X PUT http://localhost:8787/owner/repo/admin/rebuild-fts-commits
curl -H "Authorization: Bearer TOKEN" -X PUT http://localhost:8787/owner/repo/admin/rebuild-fts