feat(spiders): LinkExtractor, CrawlSpider/CrawlRule, SitemapSpider

[Liohtml]

Summary

Upstream sync (3/4). Ports the upstream Scrapling v0.4.8 spider features (18e9121, f093d0c, f7da157) plus the compound-extension fix (0d35a3f):

• LinkExtractor (src/spiders/links.rs): regex allow/deny, domain allow/deny with subdomain matching, restrict_css scoping, configurable tags/attrs, URL canonicalization (sorted query params, fragment drop), whitespace stripping, scheme filtering, a process hook, and the default IGNORED_EXTENSIONS deny list. Compound extensions like archive.tar.gz match via suffix chains.
• CrawlRule + CrawlSpider (src/spiders/templates/crawler.rs): declarative link-following rules with priority override and a process_request hook.
• SitemapSpider (src/spiders/templates/sitemap.rs): seeds crawls from sitemap or robots.txt URLs (Sitemap: directives), recurses into <sitemapindex> children (filtered by sitemap_follow), dispatches <urlset> entries through the rules (first match wins), optional alternate-language links.

Port adaptations (Python → Rust)

• The Python originals dispatch per-request callbacks; the Rust Spider trait routes everything through parse(). Templates are therefore concrete builder-configured spiders: item extraction is a parse_item closure, and SitemapSpider recognizes sitemap documents by inspecting the response (robots.txt path, <urlset>/<sitemapindex> roots).
• Transport-level gzip (Content-Encoding: gzip) is decompressed by reqwest; raw .xml.gz files are documented as unsupported since response bodies are stored as text.
• restrict_xpath is omitted (the parser is CSS-based).

Tests

21 new tests: 15 LinkExtractor unit tests (including the tar.gz compound-extension regression) and 6 template tests, four of which run the real crawl engine end-to-end via the dev-mode response cache (CrawlSpider rule following, sitemap robots.txt → index → urlset → content-page chain, sitemap_follow filtering, no-rules follow-all). Full suite green locally: 259 passed (cargo test, cargo clippy --all-targets -- -D warnings, cargo fmt --check).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk

---

Generated by Claude Code

[coderabbitai]

Warning

Review limit reached

@Liohtml, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a8290654-c82c-4242-9d62-eb86a3dc2d28

📥 Commits

Reviewing files that changed from the base of the PR and between df09f89 and 6d3d8c2.

📒 Files selected for processing (8)

• src/lib.rs
• src/spiders/links.rs
• src/spiders/mod.rs
• src/spiders/templates/crawler.rs
• src/spiders/templates/mod.rs
• src/spiders/templates/sitemap.rs
• tests/spiders_links.rs
• tests/spiders_templates.rs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/upstream-scrapling-changes-1sub94-pr3

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}