Skip to content

cons: fix Windows console VT mode save and restore#6169

Merged
wargio merged 2 commits into
rizinorg:devfrom
IndAlok:windows-console-fix
Apr 11, 2026
Merged

cons: fix Windows console VT mode save and restore#6169
wargio merged 2 commits into
rizinorg:devfrom
IndAlok:windows-console-fix

Conversation

@IndAlok
Copy link
Copy Markdown
Contributor

@IndAlok IndAlok commented Apr 7, 2026

Title:
cons: fix Windows console VT mode save and restore

Your checklist for this pull request

  • I've read the guidelines for contributing to this repository.
  • I made sure to follow the project's coding style.
  • I've documented every RZ_API function and struct this PR changes.
  • I've added tests that prove my changes are effective (required for changes to RZ_API).
  • I've updated the Rizin book with the relevant information (if needed).
  • I've used AI tools to generate fully or partially these code changes and I'm sure the changes are not copyrighted by somebody else.

Detailed description

this pr fixes win console mode handling during rizin startup and teardown. (related to #6166)

while testing the table query changes in win in meson devenv, rizin would complete the requested cmd but still leave the terminal in a bad state and print:

SetConsoleMode: (0x57) The parameter is incorrect.

the root problem was in the shared win console lifecycle, not in any specific cmd.

this pr keeps the fix limited to the console code paths that actually own and mutate win console state:

  • librz/cons/cons.c
  • librz/cons/input.c
  • librz/core/cconfig.c

the changes are:

  1. save the original console state before VT probing mutates it.
    previously, the saved console mode could already reflect side effects from early VT detection. that makes later restore unreliable because the stored state is no longer the real original host state.

  2. restore only the console mode bits Rizin actually owns.
    previously, the restore path replayed the full saved mode word. in win, this is fragile, esp. when running under nested shells or meson devenv, because it may overwrite flags owned by the hosting console instead of only undoing rizin’s own changes.

  3. fix VT input enabling in scr_vtmode().
    the old code used mode & ENABLE_VIRTUAL_TERMINAL_INPUT, which effectively stripped other input flags instead of enabling VT input. this pr changes that to the flag preserving behavior.

  4. avoid temporary input mode writes on non console stdin.
    some temporary console mode transitions assumed stdin was always a valid console handle. this pr guards those paths so batch or redirected runs do not try to call SetConsoleMode() on non-console input.

the idea behind the patch is to fix the shared console state contract directly.

Test plan

verified on win after building

Closing issues

N/A

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.24%. Comparing base (66ccde5) to head (10119aa).
⚠️ Report is 1 commits behind head on dev.

Additional details and impacted files
Files with missing lines Coverage Δ
librz/cons/cons.c 54.58% <ø> (ø)
librz/cons/input.c 14.76% <ø> (ø)
librz/core/cconfig.c 81.45% <ø> (ø)

... and 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66ccde5...10119aa. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@IndAlok IndAlok marked this pull request as ready for review April 7, 2026 06:16
@IndAlok IndAlok force-pushed the windows-console-fix branch from 519729a to 40466ea Compare April 7, 2026 22:03
Copy link
Copy Markdown
Member

@wargio wargio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is ok, but when we refactor the tui code, we need a way to handle all these globals.

Comment thread librz/cons/cons.c
@IndAlok IndAlok force-pushed the windows-console-fix branch from 40466ea to 2d0a689 Compare April 8, 2026 04:17
@github-actions github-actions Bot added the API label Apr 8, 2026
@IndAlok IndAlok requested a review from wargio April 8, 2026 04:42
@IndAlok IndAlok force-pushed the windows-console-fix branch from 2606ba1 to bcb19b4 Compare April 9, 2026 03:35
@IndAlok IndAlok force-pushed the windows-console-fix branch from bcb19b4 to ccd0249 Compare April 10, 2026 04:03
@IndAlok IndAlok force-pushed the windows-console-fix branch 3 times, most recently from a699be4 to ea4c935 Compare April 10, 2026 14:27
@IndAlok IndAlok force-pushed the windows-console-fix branch from ea4c935 to 10119aa Compare April 10, 2026 18:54
@wargio wargio merged commit f1e918d into rizinorg:dev Apr 11, 2026
49 of 50 checks passed
@IndAlok IndAlok deleted the windows-console-fix branch April 11, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants