Skip to content

Fix spatial mode on macOS Tahoe + remove gyro yaw drift#2

Open
alexiscrocilla wants to merge 3 commits into
DannyDesert:mainfrom
alexiscrocilla:fix/tahoe-spatial-and-imu-drift
Open

Fix spatial mode on macOS Tahoe + remove gyro yaw drift#2
alexiscrocilla wants to merge 3 commits into
DannyDesert:mainfrom
alexiscrocilla:fix/tahoe-spatial-and-imu-drift

Conversation

@alexiscrocilla
Copy link
Copy Markdown

Three independent fixes for issues reported by users (spatial mode invisible on macOS Tahoe + chronic gyro drift). Split across three commits so each can be reviewed and reverted independently if needed.

1. fix(spatial): route output window to glasses on macOS Tahoe

The borderless NSWindow used for spatial output was failing to display on the XReal Air on macOS 26 Tahoe in two ways:

  • canBecomeKeyWindow defaults to false for borderless windows. On Tahoe this stops MTKView's display link from delivering active draw callbacks — the first frame renders then the renderer idles, panel goes black.
  • .screenSaver window level was demoted by the Tahoe window server, so a window at negative Cocoa coordinates (where the XReal Air sits in a typical desk arrangement) gets clamped onto the built-in display as a thin strip.

Fix: subclass NSWindow as KeyableBorderlessWindow (override canBecomeKey/canBecomeMaintrue), raise window level to CGShieldingWindowLevel + 1, and call setFrame(_:display:) after orderFront(_:) to defeat Tahoe's post-creation clamp.

2. fix(imu): apply factory calibration from glasses flash to remove gyro drift

device_imu_open already streams the per-unit calibration JSON blob from on-device flash via the GET_CAL_DATA_LENGTH (0x14) / CAL_DATA_GET_NEXT_SEGMENT (0x15) HID message pair, but the bytes were being discarded with the comment "json-c calibration helpers removed". As a result, every unit fell back to identity calibration, leaving ~1°/s of residual gyro yaw bias — the visible head-locked drift users report.

Fix: accumulate the streamed segments into a malloc'd buffer (1 MiB sanity cap), parse with a small hand-rolled string-scanner (~140 LOC, no new dependencies) targeting only the ~10 keys we consume, and populate device->calibration. apply_calibration() already handles unit conversion (rad/s → deg/s for gyro, m/s² → g for accel), so no math changes downstream. Falls back silently to identity if parsing fails — never worse than before.

For empirical verification of the on-wire JSON format, set ULTRAXREAL_DUMP_CAL=/path/to/cal.json before launch and the raw blob is dumped on success.

3. feat(spatial): cursor warp on activation + grey fallback for empty canvas

Two UX improvements:

  • Cursor warp. Spatial mode unmirrors the XReal Air and takes the panel directly, so the OS cursor stays on the built-in display where the user can't see it. Auto-warp it onto the centre of the virtual display when spatial activates, and expose ⌘⇧M / "Move Cursor to Virtual Display" in the menu so it can be re-invoked if the user clicks back to the built-in display.
  • Grey clearColor when no real frame yet. SCK delivers status-only sample buffers (.idle/.blank/.suspended) when the captured content isn't changing. An empty virtual display produces only those, so the renderer used to early-return on every callback and MTKView never painted — solid black on the glasses, indistinguishable from a broken renderer. Now: skip non-.complete buffers, and when no real frame has arrived yet, render a dark-grey (#1a1a1a) clear pass so the user knows the renderer is alive and the canvas is just empty.

Testing

  • M4 MacBook Pro 14" 2024 / macOS Tahoe 26.2 (25C56) / XReal Air gen 1
  • Spatial mode appears on glasses immediately on activation
  • 30s head-still test shows no visible drift (was ~10° before the calibration fix)
  • Static (ultrawide) mode unaffected

Caveats

  • The calibration JSON parser heuristic-matches key names (acc/accelerometer, bias/offset, scale/sensitivity, misalignment/alignment). If XReal ever changes their JSON schema we may need to add variants — the ULTRAXREAL_DUMP_CAL env var helps diagnose this.
  • A note in the README about needing to refresh the Screen Recording TCC toggle after upgrading the app would help — SCStream.startCapture() silently fails with -3801 if the new cdhash doesn't match the cached permission. Happy to add this in a follow-up PR if you prefer.

alexiscrocilla and others added 3 commits May 12, 2026 07:16
The borderless NSWindow used for spatial mode output was failing to display
on the XReal Air glasses on macOS 26 Tahoe in two ways:

  1. canBecomeKeyWindow defaults to false for borderless windows. On Tahoe
     this stops MTKView's display link from delivering active draw
     callbacks — the first frame renders then the renderer idles.
  2. The `.screenSaver` window level was demoted by the Tahoe window server
     so a window at negative Cocoa coordinates (which is where the XReal
     Air sits in a typical desk arrangement) gets clamped onto the
     built-in display as a thin strip.

Fixes:

  * Subclass NSWindow as KeyableBorderlessWindow overriding canBecomeKey
    and canBecomeMain → true.
  * Raise window level to CGShieldingWindowLevel + 1 so it stays on top of
    Mission Control, spaces transitions, and other system overlays.
  * Call setFrame(_:display:) after orderFront(_:) to force the window
    back onto the XReal screen if Tahoe clamped it during initial mount.

Tested on M4 MacBook Pro 14" 2024 / macOS Tahoe 26.2 (25C56) / XReal Air gen 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… drift

device_imu_open already streams the per-unit calibration JSON blob from
on-device flash via the GET_CAL_DATA_LENGTH (0x14) / CAL_DATA_GET_NEXT_SEGMENT
(0x15) HID message pair, but the streamed bytes were being discarded into
a 56-byte scratch buffer with the comment "json-c calibration helpers
removed — using device defaults + Madgwick convergence". As a result every
unit fell back to identity calibration, leaving roughly 1°/s of residual
gyro yaw bias — the visible head-locked drift users report.

This commit accumulates the streamed segments into a malloc'd buffer
(capped at 1 MiB for safety), parses the resulting JSON with a small
hand-rolled string-scanner targeting only the ~10 keys we consume, and
populates device->calibration. apply_calibration() already converts
units correctly (rad/s → deg/s for gyro, m/s² → g for accelerometer), so
no math changes downstream.

Files:

  * UltraXReal/Vendor/xreal-imu/cal_json.{h,c} — tolerant parser, ~140 LOC,
    no new dependencies. Matches sections (acc/gyro/mag) and key names
    (bias/offset, scale/sensitivity, misalignment/alignment) heuristically
    so it handles XReal's exact JSON layout variations without a schema.
  * UltraXReal/Vendor/xreal-imu/device_imu.c — replace the discard loop
    with a collect-then-parse-then-apply block. Falls back silently to
    identity calibration if parsing fails (never worse than today).
  * UltraXReal.xcodeproj — register cal_json.c in the build.

For empirical verification of the on-wire JSON format, set
ULTRAXREAL_DUMP_CAL=/path/to/cal.json before launch — the raw blob is
dumped on success.

Tested on XReal Air gen 1: stationary head produces no visible drift
after the fix (was ~10° in 30 seconds before).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nvas

Two UX fixes for spatial mode discoverability:

1. Cursor warp. Spatial mode unmirrors the XReal Air and takes the panel
   directly with a borderless renderer window, so the OS cursor stays on
   the built-in display where the user can't see it. Auto-warp it onto
   the centre of the virtual display when spatial activates (after the
   recenter delay), and expose ⌘⇧M / "Move Cursor to Virtual Display"
   so it can be re-invoked if the user accidentally clicks back to the
   built-in display.

2. Grey clearColor when no real frame yet. SCK delivers status-only
   sample buffers (`.idle`, `.blank`, `.suspended`) when the captured
   display content isn't changing — an empty virtual display produces
   *only* those, so the renderer used to early-return on every callback
   and the MTKView never painted, leaving solid black on the glasses.
   Indistinguishable from a broken renderer. Now: read SCStreamFrameInfo
   status, skip non-`.complete` buffers, and when the renderer hasn't
   received a real frame yet, render a dark-grey (#1a1a1a) clear pass
   instead of returning early. User sees grey → knows the renderer is
   alive and the canvas is empty (drag an app onto it).

Also adds an os_log signpost for the first complete frame and for SCK
status transitions to aid future bug reports.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant