UnicodeEncodeError with surrogate characters when accessing pages with Chinese content on Windows

[D-remh]

Before submitting

• I searched existing issues for duplicates.
• I ran browser-harness --doctor and read the output.
• I read the troubleshooting section of install.md.
• This is a reproducible bug in browser-harness — not a question, feature request, or cloud.browser-use.com issue.

Summary

When accessing web pages containing Chinese characters on Windows, browser-harness throws a UnicodeEncodeError due to surrogate characters (U+DC80-U+DCFF) in the CDP response data. The error occurs during exec() execution in run.py, making it impossible to retrieve or print page information from Chinese websites.

The CDP (Chrome DevTools Protocol) returns page data (titles, URLs, etc.) that may contain surrogate characters when the page has certain encodings or special characters. These surrogate characters (U+DC80-U+DCFF) are invalid in UTF-8 and cause encoding errors when Python tries to process them.

Affected Operations

• js() - Cannot execute JavaScript that returns Chinese text content ❌
• page_info() - Works correctly ✅
• Basic navigation - Works correctly ✅

Repro

1. Start Chrome with remote debugging enabled on Windows 11
2. Run the following command to test basic page info (this works):

browser-harness <<'PY'
goto_url("https://cloud.tencent.com/developer/article/2663247")
wait_for_load()
info = page_info()
print(f"Title: {info['title']}")
PY

Result: ✅ Success - prints the page title correctly:

Title: 🐴 600行代码统治浏览器：Browser Harness如何让AI智能体获得完全自由-腾讯云开发者社区-腾讯云

3. Now try to extract article content using js():

browser-harness <<'PY'
goto_url("https://cloud.tencent.com/developer/article/2663247")
wait_for_load()

# Try to extract article content
content = js("""
    const article = document.querySelector('.article-content, .content, article');
    if (article) {
        return article.innerText.substring(0, 500);
    }
    return "No article content found";
""")

print(content[:200])
PY

Result: ❌ Error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Administrator\.local\bin\browser-harness.exe\__main__.py", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "C:\Users\Administrator\browser-harness\src\browser_harness\run.py", line 140, in main
    exec(code, exec_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 125-126: surrogates not allowed

Environment

OS: Windows 11 Pro 10.0.26200

Chrome version: 148.0.7778.168

browser-harness --version: 0.1.0 (git)

browser-harness --doctor output:

browser-harness doctor
  platform          Windows 11
  python            3.14.3
  version           0.1.0 (git)
  latest release    (could not reach github)
  [ok  ] chrome running
  [ok  ] daemon alive
  [ok  ] active browser connections — 1
        default — active page: 🐴 600行代码统治浏览器：Browser Harness如何让AI智能体获得完全自由-腾讯云开发者社区-腾讯云 — https://cloud.tencent.com/developer/article/2663247
  [FAIL] profile-use installed — optional: curl -fsSL https://browser-use.com/profile.sh | sh
  [FAIL] BROWSER_USE_API_KEY set — optional: needed only for cloud browsers / profile sync

[honor2030]

I checked the current source and the two open PRs that target this issue.

Environment / checkout:

• origin/main: 9e47d2b77754

Current source boundaries involved:

• src/browser_harness/run.py already reconfigures stdout to UTF-8 with errors="replace", which helps direct print(...) output on Windows but does not by itself sanitize all IPC/helper values.
• src/browser_harness/_ipc.py serializes client requests with json.dumps(req).encode() and parses daemon responses in ipc.request().
• src/browser_harness/daemon.py serializes daemon responses with json.dumps(resp, default=str).encode().
• src/browser_harness/helpers.py::js() returns CDP Runtime.evaluate values through _runtime_value().

Open PR overlap:

• Fix UnicodeEncodeError from lone surrogates in IPC payloads #361 (Fix UnicodeEncodeError from lone surrogates in IPC payloads) handles the IPC / daemon / run boundary: _ipc.py, daemon.py, run.py, and tests/unit/test_ipc.py. It is currently MERGEABLE / CLEAN with checks passing.
• fix: sanitize lone surrogates from js results #368 (fix: sanitize lone surrogates from js results) handles the helper return boundary: helpers.py and tests/integration/test_js.py. It is also MERGEABLE / CLEAN with checks passing.

My read:

• For the original repro (content = js(...); print(content[:200])), fix: sanitize lone surrogates from js results #368 is closest to the direct user-facing helper boundary.
• Fix UnicodeEncodeError from lone surrogates in IPC payloads #361 is broader and protects the IPC serialization/deserialization path, which can matter if lone surrogates appear in request/response payloads outside that exact js() return path.

Because both PRs already target #359 and are green, I would avoid opening a third duplicate PR. The useful maintainer decision seems to be whether to merge one boundary, both boundaries, or ask one PR to absorb the other approach.