feat(parser): adaptive element relocation#72
Conversation
The README advertised adaptive parsing and the SQLite storage layer existed, but the parser had no relocation: nothing was ever saved or scored. Port the feature from upstream Scrapling's parser: - ElementData captures an element's unique properties (tag, cleaned attributes, direct text, root-to-element tag path, parent name/attributes/text, sibling and child tags), serialized through the existing SqliteStorage. - Selector::save / Selector::retrieve persist and load that snapshot under an identifier. - Selector::relocate scores every element in the tree against the snapshot (port of __calculate_similarity_score, using a Ratcliff/Obershelp sequence ratio equivalent to difflib's SequenceMatcher) and returns the best group at or above the threshold. - Selector::css_adaptive wires it together: normal CSS first, fall back to relocation, optional auto_save re-persisting whatever was found. Upstream fixes are included from the start: default threshold 40 with a warning when nothing clears it (333b6de), and auto_save skipping the re-save when relocation returns nothing instead of panicking on an empty result (cd4cdc6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Replace the O(a*b) dynamic-program-per-comparison matcher with a port of difflib.SequenceMatcher's matching-block machinery: - b2j position index so only matching positions do work, instead of a full DP table per candidate. A relocation over a 2000-row table went from ~19s (release) to interactive time. - The autojunk heuristic (elements above 1% frequency in sequences of 200+ cannot seed matches), matching difflib defaults so similarity scores agree with Python Scrapling on large sibling lists. Verified against CPython difflib ground truth, including the displaced-run case where autojunk changes the result by ~0.98. Also match upstream's .//* candidate semantics: relocate never returns the element the search starts from nor the top-level <html> element, and the module doc now records the direct-text capture divergence from lxml's element.text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
Summary
Upstream sync (4/4). The README advertised adaptive parsing and
SqliteStorageexisted, but the parser had no relocation — nothing was ever saved or scored. This ports the feature from upstream Scrapling's parser:ElementDatacaptures an element's unique properties (tag, cleaned attributes, direct text, root-to-element tag path, parent name/attributes/text, sibling and child tags), serialized through the existingSqliteStorage.Selector::save/Selector::retrievepersist and load that snapshot under an identifier.Selector::relocatescores every element in the tree against the snapshot (port of__calculate_similarity_score, using a Ratcliff/Obershelp sequence ratio equivalent to Python'sdifflib.SequenceMatcher.ratio()) and returns the best-scoring group at or above the threshold.Selector::css_adaptivewires it together: normal CSS first, fall back to relocation, optionalauto_savere-persisting whatever was found (empty identifier defaults to the selector).Upstream fixes are included from the start:
333b6de)auto_saveskips the re-save when relocation returns nothing, instead of panicking on an empty result (cd4cdc6)Storage is passed explicitly (
&SqliteStorage) rather than via the Python original's global adaptive flag — no hidden global state.Tests
11 new tests: sequence-ratio semantics vs difflib, element-data capture, save/retrieve roundtrip, relocation after a page restructure, below-threshold empty result,
css_adaptivehappy path + relocation path + failed-relocation regression (cd4cdc6) + missing-data and default-identifier behavior. Full suite green locally: 228 passed (cargo test,cargo clippy --all-targets -- -D warnings,cargo fmt --check).🤖 Generated with Claude Code
https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
Generated by Claude Code