feat: OnlyAgent Hands — WebHID remote keystroke execution#3
Open
feat: OnlyAgent Hands — WebHID remote keystroke execution#3
Conversation
Introduces OnlyAgent Hands, a system that uses OnlyKey hardware tokens as remote keystroke executors via the WebHID API. An AI reasoning loop plans actions, the gateway compiles them to OS-specific keystrokes, and a browser bridge delivers HID reports to the OnlyKey device which types them on the host machine. A screenshot capture agent closes the feedback loop for visual verification. Gateway (Rust): - hands/ module: models, macro→keystroke compiler, CBOR packet framing, SQLx database layer, in-memory session manager, 16 Axum API endpoints - Compiler translates high-level macros (open_browser, navigate_url, type_text, etc.) into OS-specific HID keystroke sequences - 64-byte HID report framing with 5-byte header + 59-byte CBOR payload Web (Next.js): - WebHID driver (OnlyKeyHands class) for direct USB communication - Bridge relay loop: gateway → WebHID → OnlyKey → status reports - Job list dashboard with polling and status indicators - Live execution view with step tracking and emergency stop Screenshot agent (Rust): - Cross-platform capture daemon (macOS/Linux/Windows) - Periodic capture + upload to gateway for AI visual reasoning Schema: - 5 new Prisma models: HandsJob, HandsStep, HandsSession, HandsScreenshot, HandsAuditEvent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Architecture
The system uses OnlyKey hardware tokens as remote keystroke executors via the WebHID API:
New Files
apps/gateway/src/hands/— 7 Rust modules (models, compile, packet, db, session, api, mod)apps/web/src/lib/hands/— 6 TypeScript modules (types, webhid, api, bridge, index)apps/web/src/app/(dashboard)/hands/— 3 React components (page, job-list, live-view)apps/screencap/— Rust binary for cross-platform screenshot captureTest plan
cargo check -p onecli-gatewaycompiles cleanlycargo test -p onecli-gateway— compile, packet, and session unit tests passnpx prisma generatesucceeds with new schemanpx prisma migrate devcreates Hands tablespnpm build) with new pages and components🤖 Generated with Claude Code