FYI: my with Claude exploration

[yarikoptic]

Posted at

https://con.github.io/serve/tools/web/wayback-archive/

and I will likely follow up with a more specific issue on potential integration with git-annex

[GeiserX]

Thanks for the evaluation and the demo dataset — really thorough work. I've been thinking about where integration makes sense without compromising the tool's simplicity (single URL in, browsable directory out).

What I'm considering

Phase 1: Manifest output

After each download, write a manifest.json mapping every file to its provenance:

{
  "files": {
    "index.html": {
      "wayback_url": "https://web.archive.org/web/20250101120000/https://example.com/",
      "original_url": "https://example.com/",
      "timestamp": "20250101120000",
      "sha256": "abc123...",
      "size": 4521,
      "content_type": "text/html"
    }
  },
  "metadata": {
    "source_url": "https://web.archive.org/web/20250101120000/https://example.com/",
    "domain": "example.com",
    "downloaded_at": "2026-05-11T12:00:00Z"
  }
}

This is independently useful (integrity checks, resume support, scripting) and can be fed directly to datalad addurls or a git annex registerurl --batch script without any coupling.

Phase 2: CDX iteration + size-threshold skip

• WAYBACK_DOMAIN + WAYBACK_FROM / WAYBACK_TO env vars to query the CDX API and iterate over multiple snapshots natively (eliminating the shell wrapper loop)
• MAX_FILE_SIZE — files above this threshold get skipped from download, but their Wayback URLs are still recorded in the manifest with "skipped": true. Your wrapper can then feed those to git annex addurl --relaxed for lazy on-demand retrieval.

Architectural reasoning

I deliberately want to keep git-annex out of the dependency tree. The tool's output is processed (relative URL rewriting, minification, tracker removal), which makes HTML/CSS content-unstable across runs — fundamentally at odds with content-addressable storage. Binary assets (images, fonts) are saved byte-for-byte and are perfectly annexable, but text files are derived output.

The clean boundary seems to be: we produce files + machine-readable manifest, your wrapper handles the git-annex/DataLad layer. This matches the pattern I see in AnnexTube and MyKrok.

Questions for you

1. Manifest format — is the JSON structure above what datalad addurls can consume, or would a flat CSV (wayback_url,original_url,timestamp,local_path) be more natural for your workflow?

2. Size-threshold skip — would emitting skipped URLs in the manifest (with size from HTTP HEAD) be sufficient, or do you need them printed to stdout in a specific format for pipe-friendly scripting?

3. CDX iteration — in your wrapper you use collapse=timestamp:8 (daily dedup). Should that be a configurable env var (WAYBACK_COLLAPSE), or is day-level granularity always what you want?

4. Split-identity commits — I saw your demo uses Author="Internet Archive" with capture date. Is there anything the tool could emit in the manifest to make that mapping easier (e.g., a suggested git_author field per snapshot)?

5. Anything else from your exploration that felt like unnecessary friction? Happy to hear what rough edges you hit.

Looking forward to your follow-up issue with specifics.

[yarikoptic]

FWIW, ATM exploring/tuning exploration to provide a "frontend" to captured history. Apparently there is already even a standard etc: https://specs.webrecorder.net/wacz/1.1.1/ . I added references in a sample dump https://github.com/con/serve-wayback-archive-demo . Really need to switch away from this fun for now.