docs: rewrite README with hero GIF, benchmarks, and AI-agent guide#73
Conversation
- Animated hero banner (docs/assets/demo.gif + hero.png): base art generated with Z-Image-Turbo on Hugging Face Spaces, title typewriter and web glow animated locally. - 'What is this?' opener in plain language and a 30-second quick start including a SitemapSpider one-liner crawl. - 'How fast is it?' section with measured head-to-head numbers against Python Scrapling 0.4.10 (same fixture, same work): ~5.5x faster parse+extract, ~4x lower peak memory, parse-only parity — with an honest breakdown of where the wins come from and a reproducible benchmark (scripts/benchmark + examples/benchmark.rs). - 'Using it in AI agent workflows' section: tool-calling wrapper with a JSON schema, CLI-as-sandboxed-tool, self-healing extraction via css_adaptive for long-running agents, and bulk RAG collection via SitemapSpider. - Usage guide updated for the newly merged features (adaptive selectors, LinkExtractor/CrawlSpider/SitemapSpider, charset-aware decoding), architecture tree and contributing areas refreshed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
📝 WalkthroughWalkthroughThe changes add deterministic Rust and Python benchmarks using a shared generated HTML fixture, document benchmark execution and results, ignore the generated fixture, and substantially refresh the README’s usage, spider, CLI, architecture, testing, contribution, and credits sections. ChangesBenchmark and documentation updates
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant gen_page.py
participant page.html
participant Rust benchmark
participant Python benchmark
gen_page.py->>page.html: Generate deterministic product fixture
Rust benchmark->>page.html: Read fixture and measure parsing/extraction
Python benchmark->>page.html: Read fixture and measure parsing/extraction
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
scripts/benchmark/bench_python.py (1)
8-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse a context manager or
Path.read_text()to avoid leaking the file handle.♻️ Proposed fix
-html = open("page.html").read() +from pathlib import Path +html = Path("page.html").read_text()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/benchmark/bench_python.py` at line 8, Update the page.html loading statement to use Path.read_text() or a with-context-managed open call, ensuring the file handle is closed while preserving the resulting HTML string in html.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 115-117: Update the SitemapSpider quick-start snippet around
CrawlerEngine::new and engine.crawl to wrap the statements in an async main
function returning Result<(), Box<dyn std::error::Error>>, preserving the
existing ? error propagation and crawl behavior.
- Around line 142-145: Update the benchmark command sequence in README.md so the
Cargo benchmark explicitly runs from the repository root after the
scripts/benchmark commands. Preserve the existing page-generation and Python
benchmark commands, and make the directory transition unambiguous before cargo
run.
- Line 28: Qualify the “single static binary” claim in the README description by
changing it to “single native binary,” consistent with the existing cargo build
and CI setup. Update the same claim wherever it appears in the README, including
the references near the benchmark and build sections, without adding unsupported
static-target documentation.
In `@scripts/benchmark/gen_page.py`:
- Around line 29-31: Update the fixture output in the generator’s file-writing
block to resolve page.html relative to gen_page.py’s own directory rather than
the current working directory. Preserve the existing HTML content and size
reporting, and ensure the resulting path is scripts/benchmark/page.html when
invoked from the repository root.
---
Nitpick comments:
In `@scripts/benchmark/bench_python.py`:
- Line 8: Update the page.html loading statement to use Path.read_text() or a
with-context-managed open call, ensuring the file handle is closed while
preserving the resulting HTML string in html.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b8a96458-f9cb-4775-8ca4-115c4f548a39
⛔ Files ignored due to path filters (2)
docs/assets/demo.gifis excluded by!**/*.gifdocs/assets/hero.pngis excluded by!**/*.png
📒 Files selected for processing (6)
.gitignoreREADME.mdexamples/benchmark.rsscripts/benchmark/README.mdscripts/benchmark/bench_python.pyscripts/benchmark/gen_page.py
| 1. **Adaptive Parsing** -- CSS/XPath selectors that can relocate elements when page structure changes | ||
| 2. **Multi-Strategy Fetching** -- simple HTTP, stealth-mode, and browser automation in one API | ||
| 3. **Spider-Based Crawling** -- Scrapy-inspired async crawlers with rate limiting, deduplication, and checkpointing | ||
| It's a faithful Rust port of Python's [Scrapling](https://github.com/D4Vinci/Scrapling) (same concepts, same API names), which means: **~5× faster extraction, ~4× less memory, and a single static binary** — no interpreter, no virtualenv, no cold start. [Benchmarks below.](#how-fast-is-it) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
rg -n "musl|static|RUSTFLAGS|linker|target" Cargo.toml .cargo .github Dockerfile* 2>/dev/nullRepository: Liohtml/RUSTScrapling
Length of output: 461
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## README.md (around the claim)\n'
nl -ba README.md | sed -n '20,40p'
printf '\n## Cargo.toml (around the static SQLite note)\n'
nl -ba Cargo.toml | sed -n '20,45p'
printf '\n## Repo-wide search for static-linking/build-target hints\n'
rg -n "musl|static|RUSTFLAGS|linker|target|cross|x86_64-unknown-linux-musl|aarch64-unknown-linux-musl" . \
--glob '!target' --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**'Repository: Liohtml/RUSTScrapling
Length of output: 232
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## README.md (around the claim)\n'
awk 'NR>=20 && NR<=40 {printf "%d:%s\n", NR, $0}' README.md
printf '\n## Cargo.toml (around the static SQLite note)\n'
awk 'NR>=20 && NR<=45 {printf "%d:%s\n", NR, $0}' Cargo.toml
printf '\n## Repo-wide search for static-linking/build-target hints\n'
rg -n "musl|static|RUSTFLAGS|linker|target|cross|x86_64-unknown-linux-musl|aarch64-unknown-linux-musl" . \
--glob '!target' --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**'Repository: Liohtml/RUSTScrapling
Length of output: 7252
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## README.md deployment section\n'
awk 'NR>=130 && NR<=165 {printf "%d:%s\n", NR, $0}' README.md
printf '\n## CI workflow\n'
awk 'NR>=1 && NR<=140 {printf "%d:%s\n", NR, $0}' .github/workflows/ci.ymlRepository: Liohtml/RUSTScrapling
Length of output: 5921
Qualify the “static binary” claim in README.md README.md:28, 137, 157
The repo shows a normal cargo build --release/CI build and a vendored SQLite dependency, but no static-target or linker setup. Change this to “single native binary” or document the supported static target.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 28, Qualify the “single static binary” claim in the README
description by changing it to “single native binary,” consistent with the
existing cargo build and CI setup. Update the same claim wherever it appears in
the README, including the references near the benchmark and build sections,
without adding unsupported static-target documentation.
| let engine = CrawlerEngine::new(Arc::new(spider), SessionManager::new(FetcherConfig::default()), None)?; | ||
| let result = engine.crawl().await; | ||
| result.items.to_jsonl(std::path::Path::new("products.jsonl"))?; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the SitemapSpider quick-start snippet runnable.
This block contains top-level Rust statements and uses ? without an enclosing function that returns Result, so copying it as shown will not compile. Wrap it in async fn main() -> Result<(), Box<dyn std::error::Error>> or provide explicit error handling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 115 - 117, Update the SitemapSpider quick-start
snippet around CrawlerEngine::new and engine.crawl to wrap the statements in an
async main function returning Result<(), Box<dyn std::error::Error>>, preserving
the existing ? error propagation and crawl behavior.
| cd scripts/benchmark | ||
| python3 gen_page.py # writes the shared page.html fixture | ||
| python3 bench_python.py # pip install scrapling first | ||
| cargo run --release --example benchmark # from the repo root |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the benchmark commands’ working directory explicit.
After cd scripts/benchmark, the Cargo command still runs from that directory despite the comment saying “from the repo root.” This makes relative fixture paths ambiguous and can make reproduction fail.
Proposed command layout
-cd scripts/benchmark
-python3 gen_page.py
-python3 bench_python.py
-cargo run --release --example benchmark # from the repo root
+(cd scripts/benchmark && python3 gen_page.py && python3 bench_python.py)
+cargo run --release --example benchmark📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| cd scripts/benchmark | |
| python3 gen_page.py # writes the shared page.html fixture | |
| python3 bench_python.py # pip install scrapling first | |
| cargo run --release --example benchmark # from the repo root | |
| (cd scripts/benchmark && python3 gen_page.py && python3 bench_python.py) | |
| cargo run --release --example benchmark |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 142 - 145, Update the benchmark command sequence in
README.md so the Cargo benchmark explicitly runs from the repository root after
the scripts/benchmark commands. Preserve the existing page-generation and Python
benchmark commands, and make the directory transition unambiguous before cargo
run.
| with open("page.html", "w") as f: | ||
| f.write(html) | ||
| print(f"page.html written: {len(html)/1024:.0f} KB") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Fixture is written to CWD-relative page.html, but benchmarks expect scripts/benchmark/page.html.
The README instructs running python3 scripts/benchmark/gen_page.py from the repo root, which writes ./page.html. However, the Rust benchmark reads scripts/benchmark/page.html (examples/benchmark.rs:38) and the Python benchmark expects page.html in scripts/benchmark/ (bench_python.py:8, run via cd scripts/benchmark). The .gitignore entry also targets scripts/benchmark/page.html. Running as documented, neither benchmark will find the fixture.
Fix by writing relative to the script's own location:
🔧 Proposed fix
import random
random.seed(42)
@@ -26,5 +26,7 @@
</body></html>"""
-with open("page.html", "w") as f:
+import os
+out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "page.html")
+with open(out_path, "w") as f:
f.write(html)
-print(f"page.html written: {len(html)/1024:.0f} KB")
+print(f"{out_path} written: {len(html)/1024:.0f} KB")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| with open("page.html", "w") as f: | |
| f.write(html) | |
| print(f"page.html written: {len(html)/1024:.0f} KB") | |
| import os | |
| out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "page.html") | |
| with open(out_path, "w") as f: | |
| f.write(html) | |
| print(f"{out_path} written: {len(html)/1024:.0f} KB") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/benchmark/gen_page.py` around lines 29 - 31, Update the fixture
output in the generator’s file-writing block to resolve page.html relative to
gen_page.py’s own directory rather than the current working directory. Preserve
the existing HTML content and size reporting, and ensure the resulting path is
scripts/benchmark/page.html when invoked from the repository root.
Follow-up to #73: benchmark scripts resolve the fixture relative to their own directory so the documented repo-root commands work; two README snippets fixed that would not compile as shown (E0716); the scrape_tool example returns attrs as a real JSON object per its documented contract; quick-start manifest completed; peak-RSS measurement method stated; overclaims softened.
2 participants
Summary
Full README rewrite for readability plus supporting assets:
docs/assets/demo.gif, 800×450, ~3.9 MB, plus a statichero.png). Base artwork generated with Z-Image-Turbo on Hugging Face Spaces; the title typewriter effect and pulsing web glow were composited locally (the generated title had a typo, which was inpainted away and re-rendered).SitemapSpidercrawl.scripts/benchmark/(fixture generator + Python script) andexamples/benchmark.rs(cargo run --release --example benchmark).css_adaptivefor long-running agents, and bulk RAG collection viaSitemapSpider.LinkExtractor/CrawlSpider/SitemapSpider, charset-aware decoding), architecture tree and contributing areas refreshed, stale test counts fixed.All code snippets in the README were checked against the current public API.
examples/benchmark.rscompiles underclippy --all-targets -D warningsand runs against the (gitignored, regenerable) fixture.🤖 Generated with Claude Code
https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
Generated by Claude Code
Summary by CodeRabbit
Documentation
Chores