diff --git a/COMPUTER-USE-MANUAL-DRAG-PLAN.md b/COMPUTER-USE-MANUAL-DRAG-PLAN.md deleted file mode 100644 index f09871e5..00000000 --- a/COMPUTER-USE-MANUAL-DRAG-PLAN.md +++ /dev/null @@ -1,98 +0,0 @@ -# Computer Use Manual Drag Support Plan - -## Reader and Goal - -Reader: an internal engineer implementing Computer Use manual-control input. - -Post-read action: add and verify click-and-drag support for manual desktop control, so a user can drag windows, select file ranges, and perform ordinary drag gestures from the streamed desktop viewport. - -## Audit Conclusion - -Manual Computer Use does not currently support true click-and-drag from the human-controlled viewport. - -The lower layers already have partial drag capability: - -- The desktop control action model includes `mouse_drag`. -- The remote manual-control bridge accepts manual-control input actions and maps `x`, `y`, `toX`, `toY`, `sourceWidth`, and `sourceHeight` into the desktop control request. -- The desktop runtime validates drag source and target points, normalizes active stream coordinates to display coordinates, and maps the action to sidecar drag control when available. -- The native and sidecar input paths can emit a left-button drag sequence. - -The manual-control UI path is the gap: - -- Pointer down sends `mouse_click` or `mouse_right_click` immediately. -- Pointer move while a button is held sends throttled `mouse_move`. -- Pointer up on non-mobile does not send any button-release or drag action. -- Mobile touch gestures are reserved for tap, pan, and pinch behavior; they do not map touch movement into desktop drag. -- The relay client input type does not currently advertise `toX` or `toY`, even though the payload path can forward them. - -Because the remote desktop never receives a held mouse button from the manual viewport, the current behavior cannot drag windows or rubber-band select files. It can only click, move the pointer, scroll, and send keyboard/text input. - -## Implementation Strategy - -Use the existing `mouse_drag` control action first. Do not introduce stateful `mouse_down` / `mouse_up` protocol actions unless one-shot drag proves unreliable during manual QA. - -Implement desktop pointer drag as a gesture recognizer in the manual viewport: - -1. On primary-button pointer down in manual mode, capture the pointer and store a pending gesture with pointer id, button, click detail, screen start position, and mapped desktop start point. -2. Do not send `mouse_click` immediately. Wait until pointer up so the gesture can be classified as click or drag. -3. On pointer move for the captured pointer, update the latest mapped point. Mark the gesture as dragging once movement exceeds the existing tap/click slop threshold. -4. On pointer up: - - If movement stayed within slop, send the existing click or double-click payload. - - If movement exceeded slop and the button is left, send one `mouse_drag` payload with `x`, `y`, `toX`, `toY`, `sourceWidth`, and `sourceHeight`. - - If movement exceeded slop for right or middle button, keep the initial implementation conservative and do not synthesize unsupported button-drag behavior. -5. On pointer cancel, lost capture, manual-control release, stream change, or unmount, clear the pending gesture without sending a click. -6. Keep click ripples for click gestures only. Do not add temporary debug UI. - -This preserves the backend approval, lease, stream-token, and coordinate-normalization paths already used by manual control. - -## Type and Contract Updates - -Update the relay client manual-input type so drag is first-class: - -- Add `toX` and `toY`. -- Prefer a small string union for known manual actions if it fits local style; otherwise keep `action: string` and extend the shape only. -- Add a relay client test proving `mouse_drag` forwards start and target coordinates plus stream security fields. - -Add bridge/runtime coverage: - -- Add a bridge unit test for manual `mouse_drag` payload mapping, including `toX` and `toY`. -- Add or extend runtime/sidecar mapping tests so drag target coordinates are preserved into the sidecar request. -- If manual QA shows instant two-point drags are flaky, extend the runtime later with interpolated drag duration or a stateful press/drag/release protocol guarded by the same manual-control lease. - -## Frontend Tests - -Add focused tests around the manual viewport: - -- A simple pointer down/up still sends one `mouse_click`. -- A small move within slop still sends a click. -- A left-button move beyond slop sends one `mouse_drag` on pointer up and does not send the old immediate `mouse_click`. -- The drag payload uses mapped desktop stream coordinates for both source and target, including object-contain letterboxing. -- Pointer cancel sends no click or drag. -- Existing mobile pinch, pan, tap, keyboard capture, scroll, and right-click behavior remain covered. - -## Verification - -Run scoped checks only: - -- `pnpm --dir ./cloud test -- src/routes/-desktop-click-ripple.test.tsx src/lib/relay/relay-client.test.ts` -- `cargo test -p xero-desktop manual_control_drag --lib` -- `cargo test -p xero-desktop-sidecar mouse_drag --tests` - -Run Cargo commands one at a time. - -Manual QA must be performed in the Tauri app, not by opening the app in a browser: - -- Start a Computer Use desktop stream. -- Enter manual control. -- Drag a window by its title bar and confirm it moves. -- Drag across files/icons and confirm multi-select works. -- Confirm normal click, double-click, right-click, scroll, and keyboard passthrough still behave normally. - -## Acceptance Criteria - -- Manual left-button drag works from the streamed desktop viewport. -- Clicks are not accidentally converted into drags. -- Drag gestures do not emit a premature click at the start point. -- Manual-control lease and approval gates remain unchanged. -- No temporary or test-only UI is added. -- Scoped frontend and Rust tests pass. diff --git a/COMPUTER-USE-SYSTEM-CONTROL-AUDIT-PLAN.md b/COMPUTER-USE-SYSTEM-CONTROL-AUDIT-PLAN.md new file mode 100644 index 00000000..da2b0bf3 --- /dev/null +++ b/COMPUTER-USE-SYSTEM-CONTROL-AUDIT-PLAN.md @@ -0,0 +1,364 @@ +# Computer Use System Control Audit And Expansion Plan + +## Reader And Goal + +Reader: an internal engineer expanding Xero Computer Use agent control. + +Post-read action: implement the missing agent-visible tools and platform capabilities needed for the user to control their own macOS or Windows machine by prompting, with explicit local consent, visible control state, auditability, and no attempts to bypass OS security boundaries. + +## Audit Conclusion + +Computer Use already has a real agent-control foundation. It can use ordinary agent tools for project inspection, file changes, commands, browser control, diagnostics, skills, subagents, durable context, and external capabilities. It also has a native desktop-control foundation: observe the desktop, send pointer and keyboard input, launch or target apps, manage a desktop stream, and audit control actions. + +This plan is not mainly about manual WebRTC control. Manual streaming and human viewport control are one client of the desktop broker. The main target is LLM-driven Computer Use: the agent should have the right prompt-visible tools, schemas, permissions, policy gates, and platform backends to operate the user's computer as completely as the user explicitly allows. + +macOS is substantially deeper than Windows because it has Accessibility, Vision OCR, ScreenCaptureKit WebRTC streaming, and app/window automation fallbacks. Windows currently has useful pointer, keyboard, clipboard, screenshot, window, and app basics, but it lacks structured UI Automation, OCR, and native WebRTC video publishing. + +The current surface is not "absolute full system control" yet. Prompted desktop control is still bounded by: + +- OS permissions such as Screen Recording, Accessibility, Input Monitoring, Windows session access, and UAC. +- Xero policy gates that deny secret text, credential managers, payments, financial flows, identity verification, MFA/recovery flows, and system privacy/security settings. +- Repo-scoped command tools rather than a general host-administration shell. +- Agent-visible schemas that omit a few lower-level actions already implemented in the runtime. +- Tool activation and effective-policy gaps where the Computer Use agent can theoretically use a capability, but it is not present, clear, or available in the current turn. + +The right expansion path is to add an explicit owner-approved power mode, close the prompt-visible schema gaps, bring Windows to parity with macOS structured control, then add audited host-administration tools where the local user grants the privilege. + +## Scope: Agent-Driven Control + +The product goal is not only "let a remote human click the streamed desktop." It is "let the Computer Use agent complete computer tasks by choosing the best tool." + +That means the audit must cover four layers: + +- Agent policy: whether Computer Use is allowed to use a tool at all. +- Tool activation: whether the tool is included in the current turn or discoverable through tool access. +- Tool schema: whether the LLM can see every action and field it needs. +- Platform backend: whether macOS and Windows can actually execute the requested action. + +Manual WebRTC control belongs mostly to the platform backend and streaming layers. LLM-driven control also needs browser tools, file tools, command tools, process tools, system diagnostics, MCP tools, skills, subagents, project context, and future host-administration tools. + +## Current Computer Use Agent Tool Surface + +Computer Use is defined as a general-purpose runtime agent. Its base prompt says it may combine computer interaction, project inspection, file changes, commands, browser and desktop automation, diagnostics, external-capability tools, skills, subagents, and durable context when those tools are available and appropriate. + +The runtime agent gate broadly allows Computer Use to use the available autonomous tool catalog. The practical availability still depends on per-run tool policy, tool-access activation, feature rollout flags, provider schema limits, and platform support. + +### Agent Tools Already Covered + +- Repository and workspace observation: read, read-many, stat, search, find, list, tree listing, directory digest, file hash, git status, git diff, workspace index, code intelligence, and LSP. +- Repository mutation: edit, write, patch, copy, structured file transactions, JSON/TOML/YAML edit, delete, rename, mkdir, and notebook edit where available. +- Command and process control: command probe, command verify, command run, command session, and process manager. +- Browser/web control: browser observe, browser control, web fetch, and web search where enabled. +- Desktop control: macOS automation, desktop observe, desktop control, and desktop stream. +- System diagnostics: observe diagnostics and approval-gated privileged diagnostics. +- External extension surfaces: MCP list/read/get/call, dynamic MCP tools, skills, and subagents. +- Coordination and memory: todo, project context search/get/record/update/refresh, environment context, and agent coordination. +- Domain tools: emulator and Solana tools when the selected app/tool profile exposes them. + +### Agent Tool Gaps + +- There is no explicit capability matrix that says which Computer Use tools are active by default, available through tool-access request, blocked by policy, blocked by rollout, or blocked by platform. +- The current desktop-control audit covers native desktop tools more thoroughly than non-desktop agent tools. +- Repo-scoped command tools are powerful for development tasks, but they are not a complete host-administration surface. +- PowerShell appears as a tool constant/category, but the plan must verify whether it is fully agent-visible, cross-platform, policy-gated correctly, and sufficient for Windows administration. +- Browser control, desktop control, command tools, and host administration need clearer routing rules so the agent chooses the most precise safe tool instead of falling back to pixel control. +- Dynamic MCP, skills, and subagents can extend control, but there is no required "Computer Use workstation pack" that guarantees the expected macOS and Windows capabilities are installed. + +## Current Desktop Tool Surface + +### Generic Desktop Observation + +The `desktop_observe` tool is agent-visible and exposes: + +- `permissions_status` +- `display_list` +- `window_list` +- `app_list` +- `foreground_state` +- `screenshot` +- `cursor_state` +- `accessibility_snapshot` +- `ocr_snapshot` +- `element_at_point` +- `health` + +It supports display targeting, window targeting, screenshot regions, and coordinates for element lookup. + +### Generic Desktop Control + +The `desktop_control` tool is agent-visible and exposes: + +- Pointer: `mouse_move`, `mouse_click`, `mouse_double_click`, `mouse_right_click`, `mouse_drag`, `scroll` +- Keyboard and text: `key_press`, `hotkey`, `type_text`, `paste_text` +- App/window: `focus_window`, `activate_app`, `launch_app`, `quit_app` +- Structured UI: `ax_press`, `ax_set_value`, `ax_focus`, `menu_select` +- Safety/control: `cancel_current_action` + +The runtime also supports `mouse_down`, `mouse_drag_move`, and `mouse_up`, but those are not in the current agent-visible schema. They appear intended for lower-level/manual-control paths. + +### Desktop Streaming + +The `desktop_stream` tool is agent-visible and exposes: + +- `stream_capabilities` +- `stream_start` +- `stream_stop` +- `stream_status` +- `stream_set_quality` +- `stream_request_keyframe` + +The runtime and sidecar also support WebRTC offer, answer, and ICE candidate operations, but those signaling operations are not currently in the agent-visible schema. + +### macOS-Specific Automation + +The `macos_automation` tool exposes: + +- `mac_permissions` +- `mac_app_list` +- `mac_app_launch` +- `mac_app_activate` +- `mac_app_quit` +- `mac_window_list` +- `mac_window_focus` +- `mac_screenshot` + +It is macOS-only. App quit requires operator approval. On macOS, generic desktop app/window actions can fall back through this path when the sidecar does not implement those operations directly. + +## macOS Capability Matrix + +### macOS: Present + +- Permission status for Screen Recording, Accessibility, and Input Monitoring. +- Display, window, app, and foreground-state observation. +- Full-display and region screenshots. +- Cursor position. +- Accessibility snapshot and element-at-point through macOS Accessibility. +- OCR snapshot through macOS Vision. +- Mouse move, down, up, click, double-click, right-click, drag, drag-move, and scroll in the runtime/sidecar. +- Key press, hotkey, and typed text. +- Clipboard-backed paste through the sidecar. +- Accessibility actions: press, set value, focus. +- Menu path selection through Accessibility. +- App launch, activation, quit, and window focus through macOS app automation. +- Native WebRTC desktop stream using ScreenCaptureKit and VideoToolbox H.264. +- Screenshot fallback stream state. +- Controller lock, local-user takeover detection, audit log, sidecar token auth, and sidecar integrity checks. + +### macOS: Missing Or Weak + +- Agent-visible schema does not expose `mouse_down`, `mouse_up`, `mouse_drag_move`, `sourceWidth`, or `sourceHeight`. +- Agent-visible stream schema does not expose WebRTC signaling actions, even though the runtime supports them. +- Accessibility control is narrow: press, set value, focus, and menu select only. It lacks common AX actions such as select, confirm, cancel, increment/decrement, expand/collapse, scroll-to-visible, table/list row selection, checkbox/radio state changes, and robust text-field editing helpers. +- Element identity is not yet a durable targeting contract. A prompt can use snapshots, but stable element references across app refreshes need stronger IDs and re-resolution. +- Keyboard input depends on a fixed key map. International layouts, dead keys, IME text, media keys, function layers, and secure input mode need explicit handling. +- Clipboard is write/paste oriented. There is no general read/write clipboard API for text, images, files, or rich formats. +- No first-class Dock, menu bar extra, Mission Control, Spaces, notification, file dialog, open/save panel, or drag-and-drop file automation. +- No general system-administration bridge for host-level operations outside the repo sandbox. +- No privileged control for settings, network, display, audio, services, package managers, login items, or app install/uninstall beyond what can be clicked manually. +- No supported path to bypass TCC, SIP, secure input, password prompts, or other OS-protected boundaries. Those must remain user-mediated. + +## Windows Capability Matrix + +### Windows: Present + +- Display, window, app, and foreground-state observation through `xcap`. +- Full-display and region screenshots through `xcap`. +- Cursor state through the sidecar input backend. +- Mouse move, down, up, click, double-click, right-click, drag, drag-move, and scroll through Enigo. +- Key press, hotkey, and typed text through Enigo. +- Clipboard-backed paste through `arboard` plus Ctrl+V. +- Window focus and app activation through Win32 calls wrapped in PowerShell. +- App launch through PowerShell and the Windows AppsFolder shell namespace. +- App quit through `taskkill.exe`. +- Static permissions model that treats screen capture and desktop input as granted in the active user session. +- Screenshot fallback stream state. +- Controller lock and audit flow shared with macOS. + +### Windows: Missing Or Weak + +- No Windows UI Automation tree, element-at-point implementation, or structured element actions. +- No OCR snapshot implementation. +- No menu selection implementation. +- No native WebRTC desktop video publisher. The current native publisher is macOS-only; Windows falls back to screenshot-based degraded mode. +- Window management is basic: focus/activate/launch/quit only. There is no maximize, minimize, restore, move, resize, close-window message, virtual desktop, monitor move, z-order control, or owner-drawn app targeting contract. +- App launching is heuristic. It does not yet model Store apps, protocol handlers, file associations, shell verbs, admin launches, or installed-app inventory robustly. +- UAC and secure desktop are not controllable. Elevated actions require explicit user approval through the OS. +- No Registry, Services, Task Scheduler, Event Log, Windows Settings, firewall, network adapter, winget, MSI, process privilege, or local user/group management tool. +- No RDP/session switching, lock screen, credential provider, session 0, or secure-input support. +- No deep browser/Office/Explorer-specific structured automation beyond generic screen and input. + +## Cross-Platform Policy And Safety Boundaries + +Current desktop-control policy intentionally blocks: + +- Secret text typing or pasting. +- Password managers, Keychain, credentials, passkeys, and saved-password contexts. +- Purchases, checkout, payment confirmation, money transfer, and card entry. +- Banking, brokerage, tax, payroll, insurance, crypto, and wallet contexts. +- Identity verification, KYC, passport, driver license, SSN, and account ownership flows. +- MFA, TOTP, OTP, recovery codes, account recovery, password reset, and security settings. +- System privacy and security settings. + +These boundaries conflict with literal "absolute full control," but they are the right default for a shipped product. Expansion should add an explicit local owner/admin mode rather than silently weakening default Computer Use policy. + +## Expansion Plan + +### Phase 0: Agent Tool Access Manifest + +Goal: make Computer Use's LLM-accessible tool surface explicit before expanding any backend. + +Tasks: + +- Generate or maintain a Computer Use capability manifest that lists every tool, action, schema field, risk class, default availability, tool-access availability, rollout gate, platform gate, and approval requirement. +- Add tests that fail when the runtime action enum, agent-visible schema, tool catalog metadata, TypeScript DTOs, and provider schema projection drift apart. +- Verify that Computer Use can activate the expected non-desktop tool families: repository read/write, command/session/process, browser observe/control, web fetch/search, diagnostics, MCP, skill, subagent, todo, project context, environment context, code intelligence, and domain tools. +- Add a "best tool selection" guide to the Computer Use prompt or tool descriptions: prefer structured browser tools for browser tasks, command/process tools for shellable tasks, native desktop structured actions for app UI, and pixel input only when no more precise tool exists. +- Add a visible status/debug surface for developers that explains why a tool is missing in a run: policy, provider limit, rollout flag, platform unsupported, permission denied, not activated, or not installed. +- Define the required "workstation control pack" for macOS and Windows: desktop sidecar, browser control, host command/admin tools, clipboard/file-drop tools, OCR/UI tree support, and diagnostics. + +Acceptance criteria: + +- An engineer can answer "what can the Computer Use agent do on this machine right now?" from one manifest. +- A prompt-visible Computer Use tool cannot silently disappear without a failing test or visible availability reason. +- Manual WebRTC controls and LLM-driven tools are documented as separate surfaces sharing some desktop broker backends. + +### Phase 1: Make Implemented Actions Prompt-Visible + +Goal: expose the actions that already exist in the runtime where doing so improves prompted control. + +Tasks: + +- Add `mouse_down`, `mouse_up`, and `mouse_drag_move` to the `desktop_control` agent schema, tool catalog metadata, TypeScript DTOs, tests, and provider schema tests. +- Add `sourceWidth` and `sourceHeight` to the `desktop_control` agent schema so prompted actions can target screenshots or streams rendered at non-native sizes. +- Decide whether stream signaling remains internal or becomes prompt-visible. If prompt-visible, expose `stream_offer`, `stream_answer`, and `stream_ice_candidate` with strict SDP/ICE validation. +- Add focused runtime tests proving the schema and action enum stay aligned. +- Update the manual-control drag plan if the frontend still lacks the stateful gesture path. + +Acceptance criteria: + +- Every runtime desktop action intentionally available to agents is present in the tool schema. +- Hidden/internal actions are documented as internal, not accidentally omitted. +- Pointer press-hold-release can be driven by prompt when the local policy allows it. + +### Phase 2: Windows Structured UI Parity + +Goal: make Windows controllable by elements, not only pixels. + +Tasks: + +- Add a Windows UI Automation backend in the sidecar. +- Implement `accessibility_snapshot` from UIA trees with role, name, value, state, bounds, enabled/focused, and provider diagnostics. +- Implement `element_at_point` through UIA hit testing. +- Implement `ax_press`, `ax_set_value`, and `ax_focus` through UIA patterns such as Invoke, Value, Text, SelectionItem, Toggle, ExpandCollapse, and ScrollItem. +- Implement `menu_select` through UIA menu traversal where possible, with a fallback to Alt-key menu navigation. +- Add stable element handles that can be re-resolved from snapshot rows without trusting stale raw handles. +- Add Windows-specific tests behind target cfg gates and sidecar contract tests with mocked UIA payloads. + +Acceptance criteria: + +- A prompt can inspect a Windows app's UI tree, select a visible element, and invoke or edit it without coordinate clicking when UIA supports it. +- Unsupported controls return actionable diagnostics instead of falling back silently. +- The existing pointer/keyboard path remains available when UIA is blocked or absent. + +### Phase 3: Windows OCR And Native Streaming + +Goal: make Windows observation comparable to macOS for visual and remote-control tasks. + +Tasks: + +- Implement OCR with Windows.Media.Ocr where available, with a fallback strategy for unsupported Windows versions. +- Return OCR blocks with text, bounds, confidence, full text, truncation, and diagnostics matching the current sidecar contract. +- Implement native WebRTC publishing on Windows with Windows Graphics Capture plus Media Foundation or a proven H.264 encoder path. +- Preserve screenshot fallback as degraded mode, but avoid sharing a rate budget with critical input commands. +- Add GPU/driver failure diagnostics and software fallback messaging. + +Acceptance criteria: + +- Windows can return OCR snapshots from display/region captures. +- Windows can publish a native WebRTC desktop video track with cursor inclusion where supported. +- Fallback behavior is explicit and does not mask input failures. + +### Phase 4: macOS Structured Control Completion + +Goal: expand macOS Accessibility from basic actions to a practical app-control API. + +Tasks: + +- Add AX action support for select, confirm/cancel, increment/decrement, expand/collapse, scroll-to-visible, list/table row selection, checkbox/radio toggles, and text range editing. +- Add menu bar, Dock, status item, open/save panel, and file dialog helpers. +- Add element re-resolution by app, window, role, title, bounds, and ancestry path. +- Add keyboard-layout-aware key translation and a paste-first text path for long or non-US text. +- Add clipboard read/write support for text, images, files, and common rich payloads with redaction controls. + +Acceptance criteria: + +- A prompt can operate ordinary macOS forms, menus, dialogs, lists, tables, and toggles without relying only on screen coordinates. +- Text input works across keyboard layouts and long text. +- Sensitive clipboard payloads are never persisted in audit logs. + +### Phase 5: Owner-Approved Host Administration Mode + +Goal: provide as much whole-system control as possible for an explicitly consenting local owner without making default Computer Use unsafe. + +Tasks: + +- Add a "Power User" or "Owner Admin" mode in desktop-control settings. It must be local-only, time-bound, visibly active, and revocable with emergency stop. +- Add policy profiles: default safe, developer workstation, and owner admin. +- Add an audited host command tool separate from repo-scoped commands. It should require explicit mode enablement and support shell, PowerShell, AppleScript/JXA, Shortcuts, `brew`, `winget`, service management, registry edits, package install/uninstall, app launch arguments, and host file operations. +- Require per-command preview for destructive, privileged, network/security, startup-item, credential-adjacent, or privacy-sensitive operations. +- Use OS-native elevation prompts when needed. Do not attempt to bypass UAC, TCC, SIP, secure desktop, or credential prompts. +- Add rollback metadata where practical: changed files, registry keys, service state, package operations, and settings snapshots. + +Acceptance criteria: + +- A local owner can prompt Xero to perform broad workstation administration after enabling owner-admin mode. +- The user can see, stop, and audit all high-impact actions. +- Xero never claims it can bypass OS protections; it asks the user to approve OS prompts when required. + +### Phase 6: Desktop Resource Surfaces + +Goal: cover common real desktop work that is not just clicking. + +Tasks: + +- Add clipboard read/write actions for text, images, files, and common rich formats. +- Add file drag/drop synthesis for apps that expect dropped files. +- Add notification observation where platform APIs permit it. +- Add audio volume, media keys, display arrangement/readout, and window layout helpers. +- Add app inventory and launch-target discovery on both platforms. +- Add browser and terminal bridge affordances that hand off to existing browser/control/command tools when they are more precise than desktop pixels. + +Acceptance criteria: + +- Computer Use can complete routine app workflows involving files, clipboard, dialogs, notifications, and window layout. +- The agent prefers structured/native actions over coordinate input when available. + +### Phase 7: Verification Matrix + +Goal: make expanded control dependable across real machines. + +Tasks: + +- Add a capability-report fixture per platform that records sidecar capabilities, permission states, and known unavailable surfaces. +- Test macOS with Screen Recording denied/granted, Accessibility denied/granted, Input Monitoring denied/granted, multiple displays, Retina scaling, and secure input. +- Test Windows with standard user, administrator, UAC prompt, multiple DPI scales, multiple monitors, RDP, Store app, classic Win32 app, Electron app, browser, Explorer, and Office-style apps. +- Add failure-mode tests for sidecar unavailable, sidecar operation unimplemented, screenshot capture denied, UIA unavailable, OCR unavailable, stream start failure, and local-user takeover. +- Keep Cargo tests scoped and run one Cargo command at a time. + +Acceptance criteria: + +- The app can report exactly what a machine supports before the agent acts. +- Platform-specific failures produce user-actionable messages. +- High-risk actions are covered by approval, audit, and emergency-stop tests. + +## Final Target State + +The practical target is not stealthy or permissionless control. The target is consented, visible, owner-controlled workstation automation: + +- Observe the screen, UI tree, OCR text, windows, apps, cursor, permissions, and stream state. +- Choose among browser, command, file, process, diagnostics, MCP, skill, subagent, and desktop tools based on the task. +- Control pointer, keyboard, text, clipboard, menus, dialogs, windows, apps, and structured UI elements. +- Stream the desktop reliably on macOS and Windows. +- Administer the host through an explicit owner-approved tool surface. +- Preserve audit logs, leases, emergency stop, and policy profiles. +- Respect OS security prompts and refuse to bypass protected boundaries. diff --git a/client/components/xero/agent-runtime.test.tsx b/client/components/xero/agent-runtime.test.tsx index b6a3a79a..47ccdaae 100644 --- a/client/components/xero/agent-runtime.test.tsx +++ b/client/components/xero/agent-runtime.test.tsx @@ -153,6 +153,34 @@ function makeProject(overrides: Partial = {}): ProjectDetailV } } +function makeComputerUseProject(overrides: Partial = {}): ProjectDetailView { + return makeProject({ + selectedAgentSession: { + projectId: 'project-1', + agentSessionId: 'computer-use-session', + sessionKind: 'computer_use', + title: 'Computer Use', + summary: '', + status: 'active', + statusLabel: 'Active', + selected: true, + remoteVisible: false, + createdAt: '2026-05-01T11:00:00Z', + updatedAt: '2026-05-01T11:00:00Z', + archivedAt: null, + lastRunId: null, + lastRuntimeKind: null, + lastProviderId: null, + lineage: null, + isActive: true, + isArchived: false, + isComputerUse: true, + }, + selectedAgentSessionId: 'computer-use-session', + ...overrides, + }) +} + function makeRuntimeSession(overrides: Partial = {}): RuntimeSessionView { return { projectId: 'project-1', @@ -4564,29 +4592,7 @@ describe('AgentRuntime current UI', () => { render( { const closeButton = screen.getByRole('button', { name: 'Close Computer Use' }) const header = closeButton.parentElement expect(header?.className).toContain('h-10') + expect(header?.className).toContain('translate-y-1') expect(header?.textContent).toContain('Computer Use') expect(screen.queryByRole('button', { name: 'Close agent dock' })).not.toBeInTheDocument() @@ -4607,6 +4614,140 @@ describe('AgentRuntime current UI', () => { expect(onCloseSidebar).toHaveBeenCalledTimes(1) }) + it('adds client-only top spacing above Computer Use sidebar transcript content', () => { + const restoreResizeObserver = installResizeObserverMock(560) + try { + render( + , + ) + + const viewport = screen.getByLabelText('Agent conversation viewport') + expect(viewport.className).toContain('pt-14') + expect(viewport.className).not.toContain('pt-20') + expect(screen.getByText('take a screenshot')).toBeVisible() + } finally { + restoreResizeObserver() + } + }) + + it('promotes Computer Use screenshot tool output into the assistant reply preview', () => { + const restoreResizeObserver = installResizeObserverMock(560) + try { + render( + , + ) + + expect(screen.getByText('Screenshot captured.')).toBeVisible() + const previewButton = screen.getByRole('button', { + name: 'Open image preview for macOS screenshot', + }) + expect(within(previewButton).getByRole('img', { name: 'macOS screenshot' })).toHaveAttribute( + 'src', + 'project-asset://preview/macos-screenshot', + ) + + fireEvent.click(previewButton) + + expect(screen.getByRole('button', { name: 'Close image preview' })).toBeVisible() + expect(screen.getByRole('button', { name: 'Zoom in' })).toBeVisible() + expect(screen.getByRole('link', { name: 'Download macOS screenshot' })).toHaveAttribute( + 'href', + 'project-asset://preview/macos-screenshot', + ) + } finally { + restoreResizeObserver() + } + }) + it('switches sidebar panes to condensed below the sidebar compact breakpoint', async () => { const restoreResizeObserver = installResizeObserverMock(390) try { diff --git a/client/components/xero/agent-runtime.tsx b/client/components/xero/agent-runtime.tsx index 8ce30886..afd1c603 100644 --- a/client/components/xero/agent-runtime.tsx +++ b/client/components/xero/agent-runtime.tsx @@ -52,7 +52,6 @@ import type { RuntimeStreamActionRequiredItemView, RuntimeStreamActivityItemView, RuntimeStreamFailureItemView, - RuntimeStreamMediaAttachmentDto, RuntimeStreamToolItemView, RuntimeStreamViewItem, ReturnSessionToHereResponseDto, @@ -108,10 +107,14 @@ import { type CodeUndoRequest, type CodeUndoConflictSummary, type CodeUndoUiState, - type ConversationMessageAttachment, type ConversationTurn, type ReturnSessionToHereUiRequest, } from '@xero/ui/components/transcript/conversation-section' +import { + mergeConversationAttachments, + promoteActionMediaIntoFollowingAssistantMessages, + runtimeMediaAttachmentsToConversation, +} from '@xero/ui/components/transcript/runtime-media' import { EmptySessionState } from '@xero/ui/components/empty-session-state' import { getToolCardTitle, @@ -403,44 +406,6 @@ function actionTurnFromItem(item: RuntimeStreamToolItemView): ConversationTurn { } } -function runtimeMediaAttachmentsToConversation( - attachments: readonly RuntimeStreamMediaAttachmentDto[] | null | undefined, -): ConversationMessageAttachment[] | undefined { - if (!attachments || attachments.length === 0) { - return undefined - } - return attachments.map((attachment) => { - const originalName = attachment.title?.trim() - || (attachment.source.kind === 'app_data_path' - ? attachment.source.absolutePath.split(/[\\/]/).pop() - : null) - || attachment.id - const absolutePath = - attachment.source.kind === 'app_data_path' - ? attachment.source.absolutePath - : attachment.source.kind === 'artifact' - ? attachment.source.absolutePath ?? undefined - : undefined - return { - id: attachment.id, - kind: attachment.kind, - mediaType: attachment.mediaType, - originalName, - sizeBytes: attachment.sizeBytes ?? 0, - title: attachment.title ?? null, - alt: attachment.alt ?? null, - width: attachment.width ?? null, - height: attachment.height ?? null, - source: attachment.source, - renderUrl: attachment.renderUrl ?? null, - previewSrc: - attachment.renderUrl ?? - (attachment.source.kind === 'data_url' ? attachment.source.dataUrl : undefined), - absolutePath, - } - }) -} - function fileChangeTurnFromItem(item: RuntimeStreamActivityItemView): ConversationTurn { const detail = item.detail ?? item.text ?? 'File changed.' const parsed = parseFileChangeActivityDetail(detail) @@ -555,22 +520,6 @@ function mergeActionRows( return merged } -function mergeConversationAttachments( - existing: ConversationMessageAttachment[] | undefined, - incoming: ConversationMessageAttachment[] | undefined, -): ConversationMessageAttachment[] | undefined { - if (!existing?.length) return incoming - if (!incoming?.length) return existing - const merged = existing.slice() - const seen = new Set(existing.map((attachment) => attachment.id)) - for (const attachment of incoming) { - if (seen.has(attachment.id)) continue - seen.add(attachment.id) - merged.push(attachment) - } - return merged -} - function mergeActionTurn(existing: ConversationTurn, incoming: ConversationTurn): void { if (existing.kind !== 'action' || incoming.kind !== 'action') { return @@ -1068,7 +1017,7 @@ function finalizeActionTurns( ): ConversationTurn[] { const projectedTurns = toolCallGroupingPreference === 'grouped' ? compactActionBursts(turns) : turns - return limitActionTurns(projectedTurns) + return limitActionTurns(promoteActionMediaIntoFollowingAssistantMessages(projectedTurns)) } function buildConversationProjection( @@ -3116,7 +3065,7 @@ export const AgentRuntime = memo(function AgentRuntime({ label={sessionLabel} closeLabel="Close Computer Use" onClose={onCloseSidebar} - className="pointer-events-auto" + className="pointer-events-auto translate-y-1" /> ) : (
{showAgentSetupEmptyState ? ( diff --git a/client/components/xero/agent-runtime/handoff-context-dialog.tsx b/client/components/xero/agent-runtime/handoff-context-dialog.tsx index bbd453be..d156f220 100644 --- a/client/components/xero/agent-runtime/handoff-context-dialog.tsx +++ b/client/components/xero/agent-runtime/handoff-context-dialog.tsx @@ -11,15 +11,13 @@ import { Target, X, } from 'lucide-react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { cn } from '@/lib/utils' import { Badge } from '@/components/ui/badge' import { Button } from '@/components/ui/button' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -49,12 +47,15 @@ export function HandoffContextDialog({ onRefresh, }: HandoffContextDialogProps) { return ( - - +
@@ -98,6 +99,15 @@ export function HandoffContextDialog({
+ } + footerClassName="border-t border-border/40 bg-muted/20 px-5 py-2.5" + footer={ +

+ Raw bundle payloads are never shown here — only redaction-safe previews. Source-cited + text is preserved; secret-shaped values are hidden. +

+ } + >
{status === 'loading' && !summary ? ( @@ -110,15 +120,7 @@ export function HandoffContextDialog({ )}
- - -

- Raw bundle payloads are never shown here — only redaction-safe previews. Source-cited - text is preserved; secret-shaped values are hidden. -

- - - + ) } diff --git a/client/components/xero/agent-runtime/session-history-projection.ts b/client/components/xero/agent-runtime/session-history-projection.ts index 172d690a..c0873fc7 100644 --- a/client/components/xero/agent-runtime/session-history-projection.ts +++ b/client/components/xero/agent-runtime/session-history-projection.ts @@ -5,10 +5,11 @@ import type { SessionTranscriptItemDto, } from '@/src/lib/xero-model' -import type { - ConversationMessageAttachment, - ConversationTurn, -} from '@xero/ui/components/transcript/conversation-section' +import type { ConversationTurn } from '@xero/ui/components/transcript/conversation-section' +import { + promoteActionMediaIntoFollowingAssistantMessages, + runtimeMediaAttachmentsToConversation, +} from '@xero/ui/components/transcript/runtime-media' const HANDED_OFF_RUN_STATUS = 'handed_off' const MAX_HISTORICAL_CONVERSATION_TURNS = 80 @@ -119,11 +120,13 @@ export function buildHistoricalConversationTurns( } } - if (turns.length <= MAX_HISTORICAL_CONVERSATION_TURNS) { - return turns + const promotedTurns = promoteActionMediaIntoFollowingAssistantMessages(turns) + + if (promotedTurns.length <= MAX_HISTORICAL_CONVERSATION_TURNS) { + return promotedTurns } - return turns.slice(-MAX_HISTORICAL_CONVERSATION_TURNS) + return promotedTurns.slice(-MAX_HISTORICAL_CONVERSATION_TURNS) } interface MessageDisplayPolicy { @@ -214,42 +217,6 @@ function toMediaToolTurn(item: SessionTranscriptItemDto): Extract { - const originalName = attachment.title?.trim() - || (attachment.source.kind === 'app_data_path' - ? attachment.source.absolutePath.split(/[\\/]/).pop() - : null) - || attachment.id - const absolutePath = - attachment.source.kind === 'app_data_path' - ? attachment.source.absolutePath - : attachment.source.kind === 'artifact' - ? attachment.source.absolutePath ?? undefined - : undefined - return { - id: attachment.id, - kind: attachment.kind, - mediaType: attachment.mediaType, - originalName, - sizeBytes: attachment.sizeBytes ?? 0, - title: attachment.title ?? null, - alt: attachment.alt ?? null, - width: attachment.width ?? null, - height: attachment.height ?? null, - source: attachment.source, - renderUrl: attachment.renderUrl ?? null, - previewSrc: - attachment.renderUrl ?? - (attachment.source.kind === 'data_url' ? attachment.source.dataUrl : undefined), - absolutePath, - } - }) -} - const ROUTING_MARKER_REGEX = /]*?)\/>/i function extractRoutingSuggestion( diff --git a/client/components/xero/create-entity-dialog.tsx b/client/components/xero/create-entity-dialog.tsx index 3dd67cd8..c477b4de 100644 --- a/client/components/xero/create-entity-dialog.tsx +++ b/client/components/xero/create-entity-dialog.tsx @@ -1,12 +1,10 @@ import { type ReactNode } from 'react' import { ArrowLeft, ChevronRight } from 'lucide-react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { Button } from '@/components/ui/button' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -55,13 +53,19 @@ export function CreateEntityDialog({ }: CreateEntityDialogProps) { const isChoice = view === 'choice' return ( - - + - + } + header={
@@ -75,7 +79,46 @@ export function CreateEntityDialog({
- + } + footerClassName="border-t border-border/60 bg-secondary/20 px-6 py-3 sm:justify-between" + footer={ + isChoice ? ( + <> +

+ {footerNote} +

+ + + ) : ( + <> + + + + ) + } + >
{isChoice ? (
@@ -89,46 +132,7 @@ export function CreateEntityDialog({ templatesContent )}
- - - {isChoice ? ( - <> -

- {footerNote} -

- - - ) : ( - <> - - - - )} - - -
+ ) } diff --git a/client/components/xero/delete-file-dialog.tsx b/client/components/xero/delete-file-dialog.tsx index a0a9de96..4d668ad4 100644 --- a/client/components/xero/delete-file-dialog.tsx +++ b/client/components/xero/delete-file-dialog.tsx @@ -1,10 +1,8 @@ "use client" +import { BaseDialog } from '@xero/ui/components/base-dialog' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -23,8 +21,13 @@ export function DeleteFileDialog({ open, onOpenChange, path, type, onDelete }: D const fileName = path.split('/').pop() ?? '' return ( - - +
@@ -34,6 +37,18 @@ export function DeleteFileDialog({ open, onOpenChange, path, type, onDelete }: D Are you sure you want to delete {fileName}? This action cannot be undone. + } + footer={ + <> + + + + } + >
{type === 'file' ? ( @@ -46,15 +61,6 @@ export function DeleteFileDialog({ open, onOpenChange, path, type, onDelete }: D
- - - - - - + ) } diff --git a/client/components/xero/desktop-control-banner.test.tsx b/client/components/xero/desktop-control-banner.test.tsx index 41565457..33ef3dd5 100644 --- a/client/components/xero/desktop-control-banner.test.tsx +++ b/client/components/xero/desktop-control-banner.test.tsx @@ -134,6 +134,7 @@ function makeStatus(overrides: Partial = {}): DesktopCo screenshot: true, windowList: true, appList: true, + notificationObservation: true, foregroundState: true, cursorState: true, accessibilitySnapshot: true, @@ -141,6 +142,8 @@ function makeStatus(overrides: Partial = {}): DesktopCo mouseInput: true, keyboardInput: true, clipboard: true, + windowFocus: true, + appControl: true, accessibilityActions: true, menuSelect: true, webrtcStream: false, @@ -168,6 +171,8 @@ function makeStatus(overrides: Partial = {}): DesktopCo settings: { cloudStreamingEnabled: true, manualCloudControlEnabled: true, + policyProfile: 'default_safe', + ownerAdminExpiresAt: null, updatedAt: '2026-05-26T12:00:00Z', }, auditLogPath: '/tmp/xero/desktop-control/audit.jsonl', diff --git a/client/components/xero/execution-view.tsx b/client/components/xero/execution-view.tsx index 59091d8b..c5ba90c3 100644 --- a/client/components/xero/execution-view.tsx +++ b/client/components/xero/execution-view.tsx @@ -35,12 +35,10 @@ import type { import type { ExecutionPaneView } from '@/src/features/xero/use-xero-desktop-state' import { DeleteFileDialog } from './delete-file-dialog' import { RenameFileDialog } from './rename-file-dialog' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { Button } from '@/components/ui/button' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -2093,8 +2091,13 @@ function AgentEditPreviewDialog({ const textHunks = patchAvailability?.textHunks?.filter((hunk) => `/${hunk.filePath.replace(/^\/+/, '')}` === activity?.path) ?? [] return ( - - +
+ ) } @@ -2191,18 +2194,23 @@ function UnsavedChangesDialog({ }) { const count = guard?.paths.length ?? 0 return ( - { - if (!open) onCancel() - }}> - { + { + if (!open) onCancel() + }} + variant="confirmation" + title="Unsaved changes" + contentClassName="sm:max-w-lg" + contentProps={{ + onCloseAutoFocus: (event) => { if (onCloseFocus) { event.preventDefault() onCloseFocus() } - }} - > + }, + }} + header={
+ + } + > +
+ {(guard?.paths ?? []).map((path) => ( +
+ {path} +
+ ))} +
+ ) } @@ -2257,18 +2268,23 @@ function SaveConflictDialog({ }, [conflict]) return ( - { - if (!open) onKeepMine() - }}> - { + { + if (!open) onKeepMine() + }} + variant="confirmation" + title="File changed on disk" + contentClassName="sm:max-w-3xl" + contentProps={{ + onCloseAutoFocus: (event) => { if (onCloseFocus) { event.preventDefault() onCloseFocus() } - }} - > + }, + }} + header={
- - - + + } + > + {showCompare && conflict ? ( +
+ + +
+ ) : null} + ) } @@ -2343,16 +2363,21 @@ function GitHunkDialog({ : [] return ( - - { + { if (onCloseFocus) { event.preventDefault() onCloseFocus() } - }} - > + }, + }} + header={
+ ) } diff --git a/client/components/xero/execution-view/editor-navigation-dialog.tsx b/client/components/xero/execution-view/editor-navigation-dialog.tsx index 68522c86..4bef8172 100644 --- a/client/components/xero/execution-view/editor-navigation-dialog.tsx +++ b/client/components/xero/execution-view/editor-navigation-dialog.tsx @@ -8,6 +8,7 @@ import { Loader2, ListTree, } from 'lucide-react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { Button } from '@/components/ui/button' import { Checkbox } from '@/components/ui/checkbox' import { @@ -19,8 +20,6 @@ import { CommandList, } from '@/components/ui/command' import { - Dialog, - DialogContent, DialogDescription, DialogHeader, DialogTitle, @@ -88,20 +87,27 @@ export function EditorNavigationDialog({ : 'Quick open' return ( - - { + { if (onCloseFocus) { event.preventDefault() onCloseFocus() } - }} - > + }, + }} + header={ {title} Navigate inside the current project. + } + > {mode === 'go-line' ? ( )} - - + ) } diff --git a/client/components/xero/file-dialogs.test.tsx b/client/components/xero/file-dialogs.test.tsx new file mode 100644 index 00000000..bb840ac6 --- /dev/null +++ b/client/components/xero/file-dialogs.test.tsx @@ -0,0 +1,61 @@ +/** @vitest-environment jsdom */ + +import { fireEvent, render, screen, waitFor } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' + +import { DeleteFileDialog } from './delete-file-dialog' +import { NewFileDialog } from './new-file-dialog' + +describe('file dialogs', () => { + it('creates a new file from the shared form dialog', async () => { + const onCreate = vi.fn(async () => undefined) + const onOpenChange = vi.fn() + + render( + , + ) + + expect(screen.getByRole('dialog')).toHaveAttribute( + 'data-dialog-variant', + 'form', + ) + + fireEvent.change(screen.getByPlaceholderText('filename.ext'), { + target: { value: 'notes.md' }, + }) + fireEvent.click(screen.getByRole('button', { name: 'Create' })) + + await waitFor(() => expect(onCreate).toHaveBeenCalledWith('notes.md')) + await waitFor(() => expect(onOpenChange).toHaveBeenCalledWith(false)) + }) + + it('keeps delete confirmation destructive while preserving the path preview', () => { + const onDelete = vi.fn() + + render( + , + ) + + expect(screen.getByRole('dialog')).toHaveAttribute( + 'data-dialog-variant', + 'destructive-confirmation', + ) + expect(screen.getByText('/workspace/notes.md')).toBeVisible() + + fireEvent.click(screen.getByRole('button', { name: 'Delete' })) + + expect(onDelete).toHaveBeenCalledTimes(1) + }) +}) diff --git a/client/components/xero/new-file-dialog.tsx b/client/components/xero/new-file-dialog.tsx index 5dc84f11..2e95c0b4 100644 --- a/client/components/xero/new-file-dialog.tsx +++ b/client/components/xero/new-file-dialog.tsx @@ -1,11 +1,9 @@ "use client" import { useEffect, useState } from 'react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -70,8 +68,13 @@ export function NewFileDialog({ open, onOpenChange, parentPath, type, onCreate } const placeholder = type === 'file' ? 'filename.ext' : 'folder-name' return ( - - +
@@ -82,6 +85,18 @@ export function NewFileDialog({ open, onOpenChange, parentPath, type, onCreate } {parentPath === '/' ? '/' : parentPath} + } + footer={ + <> + + + + } + >
{error ?

{error}

: null}
- - - - - -
+ ) } diff --git a/client/components/xero/project-add-dialog.tsx b/client/components/xero/project-add-dialog.tsx index a2588818..21e173fb 100644 --- a/client/components/xero/project-add-dialog.tsx +++ b/client/components/xero/project-add-dialog.tsx @@ -2,13 +2,11 @@ import { useEffect, useState } from 'react' import { ArrowLeft, ChevronRight, FolderOpen, FolderPlus, Loader2, Sparkles } from 'lucide-react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { Button } from '@/components/ui/button' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -114,13 +112,20 @@ export function ProjectAddDialog({ const dialogBusy = busy || isImporting return ( - !dialogBusy && onOpenChange(next)}> - + !dialogBusy && onOpenChange(next)} + variant="custom" + title="Add a project" + busy={dialogBusy} + contentClassName="gap-0 overflow-hidden p-0 sm:max-w-[460px]" + leading={
- + } + header={
@@ -136,7 +141,57 @@ export function ProjectAddDialog({
- + } + footerClassName="border-t border-border/60 bg-secondary/20 px-6 py-3 sm:justify-between" + footer={ + mode === 'choose' ? ( + <> +

+ Projects stay local — Xero never uploads your code. +

+ + + ) : ( + <> + + + + ) + } + >
{mode === 'choose' ? (
@@ -225,57 +280,7 @@ export function ProjectAddDialog({
)}
- - - {mode === 'choose' ? ( - <> -

- Projects stay local — Xero never uploads your code. -

- - - ) : ( - <> - - - - )} - - -
+ ) } diff --git a/client/components/xero/project-rail.tsx b/client/components/xero/project-rail.tsx index 0f55eb25..33a0d96a 100644 --- a/client/components/xero/project-rail.tsx +++ b/client/components/xero/project-rail.tsx @@ -1,17 +1,8 @@ import { memo, useCallback, useEffect, useState } from 'react' import { Loader2, Plus, RefreshCw, Settings } from 'lucide-react' +import { BaseAlertDialog } from '@xero/ui/components/base-dialog' import { cn } from '@/lib/utils' -import { - AlertDialog, - AlertDialogAction, - AlertDialogCancel, - AlertDialogContent, - AlertDialogDescription, - AlertDialogFooter, - AlertDialogHeader, - AlertDialogTitle, -} from '@/components/ui/alert-dialog' import { buttonVariants } from '@/components/ui/button' import type { ProjectListItem } from '@/src/lib/xero-model' @@ -184,7 +175,7 @@ const ProjectRailItem = memo(function ProjectRailItem({ const projectInitial = Array.from(project.name.trim())[0]?.toUpperCase() ?? '?' return ( - + <>
- - - Remove {project.name} from the sidebar? - + Xero will only forget this project in the desktop registry. The repo and its app-data project state stay untouched. You can import the same folder again any time. - - - - Cancel - onRemoveProject(project.id)} - > - {isRemovalPending ? 'Removing…' : 'Remove'} - - - - + + } + cancelAction={{ + label: 'Cancel', + disabled: isRemovalPending, + }} + action={{ + label: isRemovalPending ? 'Removing…' : 'Remove', + className: buttonVariants({ variant: 'destructive' }), + destructive: false, + disabled: isRemovalPending, + onClick: () => onRemoveProject(project.id), + }} + /> + ) }) diff --git a/client/components/xero/rename-file-dialog.tsx b/client/components/xero/rename-file-dialog.tsx index 48fd61aa..d04058f1 100644 --- a/client/components/xero/rename-file-dialog.tsx +++ b/client/components/xero/rename-file-dialog.tsx @@ -1,11 +1,9 @@ "use client" import { useEffect, useState } from 'react' +import { BaseDialog } from '@xero/ui/components/base-dialog' import { - Dialog, - DialogContent, DialogDescription, - DialogFooter, DialogHeader, DialogTitle, } from '@/components/ui/dialog' @@ -79,8 +77,13 @@ export function RenameFileDialog({ } return ( - - +
{type === 'folder' ? ( @@ -94,6 +97,18 @@ export function RenameFileDialog({ Enter a new name for {currentName} + } + footer={ + <> + + + + } + >
{error ?

{error}

: null}
- - - - - -
+ ) } diff --git a/client/components/xero/settings-dialog.tsx b/client/components/xero/settings-dialog.tsx index 00cb4ee3..ec8e8dac 100644 --- a/client/components/xero/settings-dialog.tsx +++ b/client/components/xero/settings-dialog.tsx @@ -64,9 +64,8 @@ import type { GitHubSessionView, } from "@/src/lib/github-auth" import { Activity, ArrowLeft, Bot, Brain, Cloud, Code2, Database, Globe, HardDrive, Heart, Keyboard, KeyRound, Mic, Monitor, Palette, PlaySquare, Plug, PlugZap, Power, UserRound, WandSparkles, Wrench } from "lucide-react" +import { BaseDialog } from "@xero/ui/components/base-dialog" import { - Dialog, - DialogContent, DialogDescription, DialogTitle, } from "@/components/ui/dialog" @@ -890,15 +889,22 @@ export function SettingsDialog({ } return ( - - + Settings Configure providers, skills, agent tooling, and development options. + + } + >
- - + ) } diff --git a/client/components/xero/settings-dialog/account-danger-zone.tsx b/client/components/xero/settings-dialog/account-danger-zone.tsx index c05e9d56..30167999 100644 --- a/client/components/xero/settings-dialog/account-danger-zone.tsx +++ b/client/components/xero/settings-dialog/account-danger-zone.tsx @@ -1,16 +1,7 @@ import { useCallback, useMemo, useState } from "react" import { Loader2, Trash2 } from "lucide-react" +import { BaseAlertDialog } from "@xero/ui/components/base-dialog" -import { - AlertDialog, - AlertDialogAction, - AlertDialogCancel, - AlertDialogContent, - AlertDialogDescription, - AlertDialogFooter, - AlertDialogHeader, - AlertDialogTitle, -} from "@/components/ui/alert-dialog" import { Button } from "@/components/ui/button" import { Input } from "@/components/ui/input" import { @@ -220,58 +211,60 @@ export function AccountDangerZone({ projects, activeProjectId, adapter }: Accoun - { if (pending === "project") return setConfirmProjectOpen(open) }} - > - - - Wipe project data? - + variant="destructive-confirmation" + title="Wipe project data?" + description={ + <> This deletes every Xero record for{" "} {targetProject?.name ?? selectedProjectId ?? "this project"} : SQLite store, vector index, code-history, and backups. The source repository on disk is not touched. This cannot be undone. - - - - Cancel - { - event.preventDefault() - void handleWipeProject() - }} - disabled={pending === "project"} - className="bg-destructive text-destructive-foreground hover:bg-destructive/90" - > + + } + cancelAction={{ + label: "Cancel", + disabled: pending === "project", + }} + action={{ + label: ( + <> {pending === "project" ? ( ) : ( )} Wipe project - - - - + + ), + disabled: pending === "project", + className: "bg-destructive text-destructive-foreground hover:bg-destructive/90", + destructive: false, + onClick: (event) => { + event.preventDefault() + void handleWipeProject() + }, + }} + /> - { if (pending === "all") return setConfirmAllOpen(open) if (!open) setConfirmAllInput("") }} - > - - - Wipe ALL Xero data? - + variant="destructive-confirmation" + title="Wipe ALL Xero data?" + description={ + <> This deletes the entire Xero app-data directory, including every project's SQLite + Lance store, every backup, UI state, and stored credentials. Source repositories on disk are not touched, but Xero will lose every reference to them. Type{" "} @@ -279,8 +272,32 @@ export function AccountDangerZone({ projects, activeProjectId, adapter }: Accoun {WIPE_ALL_CONFIRMATION} {" "} to confirm. - - + + } + cancelAction={{ + label: "Cancel", + disabled: pending === "all", + }} + action={{ + label: ( + <> + {pending === "all" ? ( + + ) : ( + + )} + Wipe everything + + ), + disabled: pending === "all" || !wipeAllConfirmed, + className: "bg-destructive text-destructive-foreground hover:bg-destructive/90", + destructive: false, + onClick: (event) => { + event.preventDefault() + void handleWipeAll() + }, + }} + > setConfirmAllInput(event.target.value)} @@ -290,26 +307,7 @@ export function AccountDangerZone({ projects, activeProjectId, adapter }: Accoun className="mt-2" data-testid="danger-wipe-all-input" /> - - Cancel - { - event.preventDefault() - void handleWipeAll() - }} - disabled={pending === "all" || !wipeAllConfirmed} - className="bg-destructive text-destructive-foreground hover:bg-destructive/90" - > - {pending === "all" ? ( - - ) : ( - - )} - Wipe everything - - - - + ) } diff --git a/client/components/xero/settings-dialog/agents-section.test.tsx b/client/components/xero/settings-dialog/agents-section.test.tsx index e0827d33..deb5c8c6 100644 --- a/client/components/xero/settings-dialog/agents-section.test.tsx +++ b/client/components/xero/settings-dialog/agents-section.test.tsx @@ -27,7 +27,7 @@ const computerUseBuiltin: AgentDefinitionSummaryDto = { definitionId: 'computer_use', displayName: 'Computer Use', shortLabel: 'Computer', - description: 'Control the visible computer through bounded automation.', + description: 'Use the tools available for the current turn.', baseCapabilityProfile: 'computer_use', } diff --git a/client/components/xero/settings-dialog/desktop-control-section.test.tsx b/client/components/xero/settings-dialog/desktop-control-section.test.tsx index 97ddbc11..5290b08d 100644 --- a/client/components/xero/settings-dialog/desktop-control-section.test.tsx +++ b/client/components/xero/settings-dialog/desktop-control-section.test.tsx @@ -1,4 +1,4 @@ -import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' +import { act, cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' import { afterEach, describe, expect, it, vi } from 'vitest' import { DesktopControlSection } from '@/components/xero/settings-dialog/desktop-control-section' import type { @@ -12,6 +12,43 @@ afterEach(() => { }) describe('DesktopControlSection', () => { + it('shows loading skeletons instead of fallback status while initial desktop status is pending', async () => { + const status = makeStatus({ + permissions: [ + { + name: 'Screen Recording', + status: 'granted', + requiredFor: ['screenshot', 'stream'], + remediation: 'Screen capture permission is granted.', + action: null, + }, + ], + }) + const pendingStatus = createDeferred() + const adapter = makeAdapter({ status }) + adapter.desktopControlStatus.mockImplementationOnce(async () => pendingStatus.promise) + + render() + + expect( + screen.getByRole('status', { name: 'Loading desktop-control status' }), + ).toBeVisible() + expect(screen.getByRole('button', { name: 'Refresh' })).toBeDisabled() + expect(screen.queryByText('unavailable')).not.toBeInTheDocument() + expect(screen.queryByText('idle · unavailable')).not.toBeInTheDocument() + expect(screen.queryByRole('switch', { name: 'Allow cloud viewing' })).not.toBeInTheDocument() + expect(screen.queryByText('Screen Recording')).not.toBeInTheDocument() + + await act(async () => { + pendingStatus.resolve(status) + await pendingStatus.promise + }) + + expect(await screen.findByText('ready')).toBeVisible() + expect(screen.getByRole('switch', { name: 'Allow cloud viewing' })).toBeChecked() + expect(screen.getByText('Screen Recording')).toBeVisible() + }) + it('shows brokered macOS permission actions and retry guidance', async () => { const adapter = makeAdapter({ status: makeStatus({ @@ -81,27 +118,89 @@ describe('DesktopControlSection', () => { expect(screen.getByText('Required for wayland capture, wayland input.')).toBeVisible() expect(screen.queryByRole('button', { name: /Open/ })).not.toBeInTheDocument() }) + + it('enables owner-admin mode with a bounded local duration and revokes it on stop', async () => { + const activeUntil = new Date(Date.now() + 15 * 60 * 1000).toISOString() + const status = makeStatus() + const adapter = makeAdapter({ + status, + updateStatus: makeStatus({ + settings: { + ...status.settings, + policyProfile: 'owner_admin', + ownerAdminExpiresAt: activeUntil, + updatedAt: '2026-05-26T12:01:00Z', + }, + }), + stopStatus: makeStatus({ + settings: { + ...status.settings, + policyProfile: 'default_safe', + ownerAdminExpiresAt: null, + updatedAt: '2026-05-26T12:02:00Z', + }, + }), + }) + + render() + + fireEvent.click(await screen.findByRole('button', { name: 'Owner Admin' })) + + await waitFor(() => + expect(adapter.desktopControlUpdateSettings).toHaveBeenCalledWith({ + cloudStreamingEnabled: true, + manualCloudControlEnabled: true, + policyProfile: 'owner_admin', + ownerAdminDurationMinutes: 30, + }), + ) + expect(await screen.findByText('owner admin')).toBeVisible() + + fireEvent.click(screen.getByRole('button', { name: 'Stop' })) + + await waitFor(() => expect(adapter.desktopControlStop).toHaveBeenCalled()) + }) }) -function makeAdapter({ status }: { status: DesktopControlStatusDto }) { +function makeAdapter({ + status, + updateStatus, + stopStatus, +}: { + status: DesktopControlStatusDto + updateStatus?: DesktopControlStatusDto + stopStatus?: DesktopControlStatusDto +}) { return { isDesktopRuntime: vi.fn(() => true), desktopControlStatus: vi.fn(async () => status), desktopControlUpdateSettings: vi.fn( - async (request: UpsertDesktopControlSettingsRequestDto) => ({ + async (request: UpsertDesktopControlSettingsRequestDto) => + updateStatus ?? { ...status, settings: { ...status.settings, ...request, + ownerAdminExpiresAt: + request.policyProfile === 'owner_admin' ? status.settings.ownerAdminExpiresAt : null, updatedAt: '2026-05-26T12:01:00Z', }, - }), + }, ), - desktopControlStop: vi.fn(async () => status), + desktopControlStop: vi.fn(async () => stopStatus ?? status), desktopControlOpenPermissionSettings: vi.fn(async () => undefined), } } +function createDeferred() { + let resolve!: (value: T) => void + const promise = new Promise((promiseResolve) => { + resolve = promiseResolve + }) + + return { promise, resolve } +} + function makeStatus(overrides: Partial = {}): DesktopControlStatusDto { return { schema: 'xero.desktop_control_status.v1', @@ -121,6 +220,7 @@ function makeStatus(overrides: Partial = {}): DesktopCo screenshot: true, windowList: true, appList: true, + notificationObservation: true, foregroundState: true, cursorState: true, accessibilitySnapshot: true, @@ -128,6 +228,8 @@ function makeStatus(overrides: Partial = {}): DesktopCo mouseInput: true, keyboardInput: true, clipboard: true, + windowFocus: true, + appControl: true, accessibilityActions: true, menuSelect: true, webrtcStream: false, @@ -155,6 +257,8 @@ function makeStatus(overrides: Partial = {}): DesktopCo settings: { cloudStreamingEnabled: true, manualCloudControlEnabled: true, + policyProfile: 'default_safe', + ownerAdminExpiresAt: null, updatedAt: '2026-05-26T12:00:00Z', }, auditLogPath: '/tmp/xero/desktop-control/audit.jsonl', diff --git a/client/components/xero/settings-dialog/desktop-control-section.tsx b/client/components/xero/settings-dialog/desktop-control-section.tsx index c864ea47..6b387626 100644 --- a/client/components/xero/settings-dialog/desktop-control-section.tsx +++ b/client/components/xero/settings-dialog/desktop-control-section.tsx @@ -10,6 +10,7 @@ import { import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert" import { Badge } from "@/components/ui/badge" import { Button } from "@/components/ui/button" +import { Skeleton } from "@/components/ui/skeleton" import { Switch } from "@/components/ui/switch" import type { XeroDesktopAdapter } from "@/src/lib/xero-desktop" import type { @@ -18,6 +19,8 @@ import type { } from "@/src/lib/xero-model/desktop-control" import { SectionHeader } from "./section-header" +const OWNER_ADMIN_DURATION_MINUTES = 30 + export type DesktopControlSettingsAdapter = Pick< XeroDesktopAdapter, | "isDesktopRuntime" @@ -51,6 +54,8 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { async ({ refreshPermissionStatus = false }: { refreshPermissionStatus?: boolean } = {}) => { if (!canUseAdapter || !adapter?.desktopControlStatus) { setStatus(null) + setLoading(false) + setError(null) return } setLoading(true) @@ -94,6 +99,7 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { ): UpsertDesktopControlSettingsRequestDto => ({ cloudStreamingEnabled: status?.settings.cloudStreamingEnabled ?? false, manualCloudControlEnabled: status?.settings.manualCloudControlEnabled ?? false, + policyProfile: status?.settings.policyProfile ?? "default_safe", ...patch, }) @@ -108,7 +114,7 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { } const stopControl = async () => { - if (!adapter?.desktopControlStop) return + if (!adapter?.desktopControlStop || !status) return setStopping(true) setError(null) try { @@ -137,9 +143,11 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { } } - const selectedStatus = status ?? fallbackStatus() - const activeLock = selectedStatus.controllerLock - const stream = selectedStatus.stream + const selectedStatus = canUseAdapter ? status : fallbackStatus() + const isStatusLoading = canUseAdapter && status === null && error === null + const activeLock = selectedStatus?.controllerLock ?? null + const stream = selectedStatus?.stream ?? null + const ownerAdminActive = selectedStatus ? isOwnerAdminActive(selectedStatus) : false return (
@@ -166,6 +174,250 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { ) : null} + {isStatusLoading ? ( + + ) : selectedStatus ? ( + <> +
+
+

Status

+ +
+ +
+ + + +
+
+ +
+

Policy profile

+
+
+
+ updateSetting("policyProfile", "default_safe")} + /> + updateSetting("policyProfile", "developer_workstation")} + /> + + void saveSettings( + currentSettingsRequest({ + policyProfile: ownerAdminActive ? "default_safe" : "owner_admin", + ownerAdminDurationMinutes: ownerAdminActive + ? undefined + : OWNER_ADMIN_DURATION_MINUTES, + }), + "policyProfile", + ) + } + /> +
+
+ + {formatPolicyProfile(selectedStatus.settings.policyProfile)} + + {ownerAdminActive && selectedStatus.settings.ownerAdminExpiresAt ? ( + + Expires {formatDateTime(selectedStatus.settings.ownerAdminExpiresAt)} + + ) : null} +
+
+
+
+ +
+

Cloud access

+
+ updateSetting("cloudStreamingEnabled", value)} + /> + updateSetting("manualCloudControlEnabled", value)} + /> +
+
+ +
+

Permissions

+
    + {selectedStatus.permissions.map((permission) => ( +
  • +
    +
    +

    + {permission.name} +

    + + {permission.status.replace(/_/g, " ")} + +
    +

    + {permission.remediation} +

    + {permission.requiredFor.length > 0 ? ( +

    + Required for{" "} + {permission.requiredFor.map(formatPermissionPurpose).join(", ")}. +

    + ) : null} + {permission.action ? ( +

    + {permission.action.postActionHint} +

    + ) : null} +
    + {permission.action ? ( + + ) : null} +
    • + ))} +
+
+ +
+

Emergency stop

+
+

+ Release the desktop controller lock, stop the active desktop stream, and revoke Owner Admin mode. +

+ +
+
+ + ) : ( + void refresh({ refreshPermissionStatus: true })} + /> + )} +
+ ) +} + +function permissionBadgeVariant(status: DesktopControlStatusDto["permissions"][number]["status"]) { + if (status === "granted") return "secondary" + if (status === "denied") return "destructive" + return "outline" +} + +function formatPermissionPurpose(value: string): string { + return value.replace(/_/g, " ") +} + +function formatPolicyProfile(value: DesktopControlStatusDto["settings"]["policyProfile"]): string { + return value.replace(/_/g, " ") +} + +function isOwnerAdminActive(status: DesktopControlStatusDto): boolean { + if (status.settings.policyProfile !== "owner_admin" || !status.settings.ownerAdminExpiresAt) { + return false + } + const expiresAt = Date.parse(status.settings.ownerAdminExpiresAt) + return Number.isFinite(expiresAt) && expiresAt > Date.now() +} + +function formatDateTime(value: string): string { + const date = new Date(value) + if (Number.isNaN(date.getTime())) return value + return new Intl.DateTimeFormat(undefined, { + dateStyle: "medium", + timeStyle: "short", + }).format(date) +} + +function permissionActionId(action: DesktopPermissionAction): string { + return `${action.kind}:${action.target}` +} + +function DesktopControlLoadingState() { + return ( +

Status

@@ -174,144 +426,195 @@ export function DesktopControlSection({ adapter }: DesktopControlSectionProps) { size="sm" variant="outline" className="h-8 gap-1.5 text-[12px]" - disabled={!canUseAdapter || loading} - onClick={() => void refresh({ refreshPermissionStatus: true })} + disabled > - + Refresh
-
- - - + + + +
+
+ +
+

Policy profile

+
+
+ + + +

Cloud access

- updateSetting("cloudStreamingEnabled", value)} - /> - updateSetting("manualCloudControlEnabled", value)} - /> + +

Permissions

-
    - {selectedStatus.permissions.map((permission) => ( -
  • -
    -
    -

    {permission.name}

    - - {permission.status.replace(/_/g, " ")} - -
    -

    - {permission.remediation} -

    - {permission.requiredFor.length > 0 ? ( -

    - Required for {permission.requiredFor.map(formatPermissionPurpose).join(", ")}. -

    - ) : null} - {permission.action ? ( -

    - {permission.action.postActionHint} -

    - ) : null} -
    - {permission.action ? ( - - ) : null} -
    • - ))} -
+
+ + +
+
+ +
+

Emergency stop

+
+ + +
+
+
+ ) +} + +function DesktopControlUnavailableState({ + loading, + onRefresh, +}: { + loading: boolean + onRefresh: () => void +}) { + return ( + <> +
+
+

Status

+ +
+ +
+ +
+

Policy profile

+ +
+ +
+

Cloud access

+ +
+ +
+

Permissions

+

Emergency stop

- Release the desktop controller lock and stop the active desktop stream. + Release the desktop controller lock, stop the active desktop stream, and revoke Owner Admin mode.

+ + ) +} + +function UnavailablePanel({ message }: { message: string }) { + return ( +
+ {message}
) } -function permissionBadgeVariant(status: DesktopControlStatusDto["permissions"][number]["status"]) { - if (status === "granted") return "secondary" - if (status === "denied") return "destructive" - return "outline" +function StatusTileSkeleton({ label }: { label: string }) { + return ( +
+
+ + {label} +
+ +
+ ) } -function formatPermissionPurpose(value: string): string { - return value.replace(/_/g, " ") +function ProfileButton({ + label, + active, + disabled, + onClick, +}: { + label: string + active: boolean + disabled: boolean + onClick: () => void +}) { + return ( + + ) } -function permissionActionId(action: DesktopPermissionAction): string { - return `${action.kind}:${action.target}` +function SettingRowSkeleton() { + return ( +
+ +
+ + + +
+ +
+ ) +} + +function PermissionRowSkeleton() { + return ( +
+
+
+ + +
+ + +
+ +
+ ) } function SettingRow({ @@ -389,6 +692,7 @@ function fallbackStatus(): DesktopControlStatusDto { screenshot: false, windowList: false, appList: false, + notificationObservation: false, foregroundState: false, cursorState: false, accessibilitySnapshot: false, @@ -396,6 +700,8 @@ function fallbackStatus(): DesktopControlStatusDto { mouseInput: false, keyboardInput: false, clipboard: false, + windowFocus: false, + appControl: false, accessibilityActions: false, menuSelect: false, webrtcStream: false, @@ -423,6 +729,8 @@ function fallbackStatus(): DesktopControlStatusDto { settings: { cloudStreamingEnabled: false, manualCloudControlEnabled: false, + policyProfile: "default_safe", + ownerAdminExpiresAt: null, updatedAt: null, }, auditLogPath: "", diff --git a/client/components/xero/settings-dialog/diagnostics-section.tsx b/client/components/xero/settings-dialog/diagnostics-section.tsx index 88f9f4a0..b986a84c 100644 --- a/client/components/xero/settings-dialog/diagnostics-section.tsx +++ b/client/components/xero/settings-dialog/diagnostics-section.tsx @@ -17,6 +17,7 @@ import { X, XCircle, } from "lucide-react" +import { BaseDialog } from "@xero/ui/components/base-dialog" import type { DoctorReportRunStatus, OperatorActionErrorView, @@ -48,14 +49,6 @@ import { CollapsibleContent, CollapsibleTrigger, } from "@/components/ui/collapsible" -import { - Dialog, - DialogContent, - DialogDescription, - DialogFooter, - DialogHeader, - DialogTitle, -} from "@/components/ui/dialog" import { Input } from "@/components/ui/input" import { Select, @@ -776,14 +769,58 @@ function AddToolDialog({ } return ( - - - - Add developer tool - - Xero will run {consentCommand} to verify the tool. The first non-empty line of output with secrets redacted is stored as the version. - - + + Xero will run {consentCommand} to verify the tool. The first non-empty line of output with secrets redacted is stored as the version. + + } + contentClassName="max-w-md gap-4 rounded-lg p-5" + headerClassName="gap-1.5" + titleClassName="text-[15px] font-semibold tracking-tight" + descriptionClassName="text-[12.5px] leading-[1.55]" + footerClassName="gap-2 sm:justify-between" + footer={ + <> + +
+ + +
+ + } + >
@@ -830,43 +867,7 @@ function AddToolDialog({
- - - -
- - -
- - - + ) } diff --git a/client/components/xero/settings-dialog/memory-review-section.tsx b/client/components/xero/settings-dialog/memory-review-section.tsx index 9023e1ad..55798cb2 100644 --- a/client/components/xero/settings-dialog/memory-review-section.tsx +++ b/client/components/xero/settings-dialog/memory-review-section.tsx @@ -14,17 +14,10 @@ import { Trash2, X, } from "lucide-react" +import { BaseDialog } from "@xero/ui/components/base-dialog" import { Button } from "@/components/ui/button" import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible" -import { - Dialog, - DialogContent, - DialogDescription, - DialogFooter, - DialogHeader, - DialogTitle, -} from "@/components/ui/dialog" import { DropdownMenu, DropdownMenuContent, @@ -422,31 +415,16 @@ export function MemoryReviewSection({ ) : null} - (open ? null : closeEditor())}> - - - Correct memory - - Submitting a correction creates a new approved memory that cites this one. The original record stays in the audit - trail. - - -