Skip to content

feat(parser): adaptive element relocation#72

Merged
Liohtml merged 2 commits into
masterfrom
claude/upstream-scrapling-changes-1sub94-pr4
Jul 11, 2026
Merged

feat(parser): adaptive element relocation#72
Liohtml merged 2 commits into
masterfrom
claude/upstream-scrapling-changes-1sub94-pr4

Conversation

@Liohtml

@Liohtml Liohtml commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Upstream sync (4/4). The README advertised adaptive parsing and SqliteStorage existed, but the parser had no relocation — nothing was ever saved or scored. This ports the feature from upstream Scrapling's parser:

  • ElementData captures an element's unique properties (tag, cleaned attributes, direct text, root-to-element tag path, parent name/attributes/text, sibling and child tags), serialized through the existing SqliteStorage.
  • Selector::save / Selector::retrieve persist and load that snapshot under an identifier.
  • Selector::relocate scores every element in the tree against the snapshot (port of __calculate_similarity_score, using a Ratcliff/Obershelp sequence ratio equivalent to Python's difflib.SequenceMatcher.ratio()) and returns the best-scoring group at or above the threshold.
  • Selector::css_adaptive wires it together: normal CSS first, fall back to relocation, optional auto_save re-persisting whatever was found (empty identifier defaults to the selector).

Upstream fixes are included from the start:

  • default threshold 40 with a warning when nothing clears it (333b6de)
  • auto_save skips the re-save when relocation returns nothing, instead of panicking on an empty result (cd4cdc6)

Storage is passed explicitly (&SqliteStorage) rather than via the Python original's global adaptive flag — no hidden global state.

Tests

11 new tests: sequence-ratio semantics vs difflib, element-data capture, save/retrieve roundtrip, relocation after a page restructure, below-threshold empty result, css_adaptive happy path + relocation path + failed-relocation regression (cd4cdc6) + missing-data and default-identifier behavior. Full suite green locally: 228 passed (cargo test, cargo clippy --all-targets -- -D warnings, cargo fmt --check).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk


Generated by Claude Code

The README advertised adaptive parsing and the SQLite storage layer
existed, but the parser had no relocation: nothing was ever saved or
scored. Port the feature from upstream Scrapling's parser:

- ElementData captures an element's unique properties (tag, cleaned
  attributes, direct text, root-to-element tag path, parent
  name/attributes/text, sibling and child tags), serialized through the
  existing SqliteStorage.
- Selector::save / Selector::retrieve persist and load that snapshot
  under an identifier.
- Selector::relocate scores every element in the tree against the
  snapshot (port of __calculate_similarity_score, using a
  Ratcliff/Obershelp sequence ratio equivalent to difflib's
  SequenceMatcher) and returns the best group at or above the
  threshold.
- Selector::css_adaptive wires it together: normal CSS first, fall back
  to relocation, optional auto_save re-persisting whatever was found.

Upstream fixes are included from the start: default threshold 40 with a
warning when nothing clears it (333b6de), and auto_save skipping the
re-save when relocation returns nothing instead of panicking on an
empty result (cd4cdc6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Liohtml, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 5 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f1761a2c-16b4-4e28-8411-96db4bfbd230

📥 Commits

Reviewing files that changed from the base of the PR and between df09f89 and 90a389d.

📒 Files selected for processing (4)
  • src/parser/adaptive.rs
  • src/parser/mod.rs
  • src/parser/selector.rs
  • tests/parser_adaptive.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/upstream-scrapling-changes-1sub94-pr4

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Replace the O(a*b) dynamic-program-per-comparison matcher with a port
of difflib.SequenceMatcher's matching-block machinery:

- b2j position index so only matching positions do work, instead of a
  full DP table per candidate. A relocation over a 2000-row table went
  from ~19s (release) to interactive time.
- The autojunk heuristic (elements above 1% frequency in sequences of
  200+ cannot seed matches), matching difflib defaults so similarity
  scores agree with Python Scrapling on large sibling lists. Verified
  against CPython difflib ground truth, including the displaced-run
  case where autojunk changes the result by ~0.98.

Also match upstream's .//* candidate semantics: relocate never returns
the element the search starts from nor the top-level <html> element,
and the module doc now records the direct-text capture divergence from
lxml's element.text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
@Liohtml Liohtml merged commit 372d9f3 into master Jul 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants