fix(fetchers): decode response bodies by Content-Type charset

[Liohtml]

Summary

Upstream sync (1/4). Response bodies were always decoded with String::from_utf8_lossy, ignoring the charset entirely — any page served as ISO-8859-1, windows-1252, Shift_JIS, etc. was turned into mojibake.

• New fetchers::encoding module: extracts the charset from the Content-Type header and decodes the body with encoding_rs, falling back to lossy UTF-8 when the charset is missing or unknown. A BOM in the body takes precedence over the header (WHATWG behaviour).
• The charset pattern accepts RFC 7231 quoted-string values (charset="ISO-8859-1"), mirroring upstream Scrapling commit 6390c0a which fixed the same silent UTF-8 fallback for quoted charsets.
• Fetcher::request now decodes via the new module.

Tests

12 new unit tests covering bare/quoted/case-insensitive/missing charset extraction, ISO-8859-1 and Shift_JIS decoding, quoted-charset regression, unknown-charset and missing-header fallbacks, and BOM precedence. Full suite green locally (cargo test, cargo clippy --all-targets -- -D warnings, cargo fmt --check).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk

---

Generated by Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved response text decoding based on the character set specified in the HTTP Content-Type header.
  • Added support for common encodings such as ISO-8859-1 and Shift_JIS.
  • Improved handling of quoted or case-variant charset declarations, unknown encodings, and byte-order marks.

[coderabbitai]

Warning

Review limit reached

@Liohtml, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 13 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fe8f3da8-d855-4c7a-9900-af9b69a60237

📥 Commits

Reviewing files that changed from the base of the PR and between 606f9d5 and cc2d4c3.

📒 Files selected for processing (1)

• src/fetchers/encoding.rs

📝 Walkthrough

Walkthrough

The fetcher now decodes response bodies according to the charset in Content-Type, using encoding_rs with UTF-8 fallback and BOM handling. The new encoding module is publicly exposed and covered by unit tests.

Changes

Response decoding

Layer / File(s)	Summary
Charset extraction and decoding src/fetchers/encoding.rs, Cargo.toml	Adds charset parsing, encoding_rs decoding, fallback behavior, BOM precedence, and unit tests.
Fetcher decoding integration src/fetchers/client.rs, src/fetchers/mod.rs	Routes response body bytes through decode_body and exposes the encoding module.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: decoding response bodies using the Content-Type charset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/upstream-scrapling-changes-1sub94-pr1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}