AgentDeck

Stop Chatting. Start Steering.

AgentDeck is a physical control surface for AI coding agents. It started with an Elgato Stream Deck+ and now runs on 11 display surfaces simultaneously — tablets, e-ink readers, phones, ESP32 modules, LED matrices, and terminals.

One bridge. 11 surfaces. Steer your AI — without leaving your keyboard flow.

Watch Demo on YouTube

	Requirement
Platform	macOS 14+ (Sonoma) — Windows/Linux not supported
Hardware	Elgato Stream Deck+ (8 keys, 4 encoders, LCD touch strip)
Terminal	iTerm2 (required for session management and voice paste)
Android	(Optional) Android 10+ tablet or e-ink reader for remote dashboard
Apple	(Optional) iOS 17+ / iPadOS 17+ / macOS 14+ for SwiftUI dashboard
TUI	(Optional) Any terminal with truecolor support for `agentdeck dashboard`

What is AgentDeck?

A control surface — like an audio mixing console, but for AI coding agents. It reads your agent's state in real-time and dynamically reconfigures buttons and encoders to match what's happening right now.

Respond instantly — YES / NO / ALWAYS buttons appear with semantic colors for permission prompts
Interrupt — STOP button sends Ctrl+C to a runaway agent
Switch modes — cycle Plan / Accept Edits / Default
Navigate options — encoder scrolls and selects multi-choice prompts on a wide-canvas LCD
Voice input — push-to-talk → whisper.cpp → auto-send (offline, <2s on M-series)
Monitor usage — animated water-gauge dashboard with rate limit countdowns
Quick actions — GO ON / REVIEW / COMMIT / CLEAR; encoder cycles custom prompts
System utilities — volume, mic, media, timer from the Utility encoder
Terminal sessions — iTerm dial switches sessions, auto-attaches tmux
Works from anywhere — all 11 surfaces can monitor and control the agent independently

The bridge is transparent: if it's off, Claude Code works exactly as before.

Supported Agents

Agent	Status
Claude Code	Supported (primary)
OpenClaw	Experimental — Gateway WebSocket, timeline panel, log stream

Supported Surfaces — 11 Types

#	Surface	Description
1	Stream Deck+	Primary — 8 keys, 4 encoders, LCD touch strip
2	Android Tablet	Color terrarium + HUD overlay (60fps)
3	Crema S E-ink	B&W aquarium + 16-level grayscale + partial refresh
4	iPhone	SwiftUI app — mobile agent monitoring
5	iPad	SwiftUI app — terrarium second screen
6	macOS	SwiftUI app — desktop monitoring window
7	ESP32 Round AMOLED	1.8" circular 466×466 — compact WiFi display
8	ESP32 IPS LCD	3.5" rectangular 480×320
9	ESP32 B86 Box	4" wall-mount touch panel 480×480
10	Pixoo64 LED	64×64 RGB LED pixel art terrarium
11	TUI Terminal	Unicode braille terrarium + ANSI dashboard — SSH/remote

Left: iPad + iPhone (SwiftUI) | Right: ESP32 3 types + Pixoo64 LED

Left: E-ink (Crema S) + ESP32 Round AMOLED | Right: TUI terminal dashboard

Architecture

Stream Deck Plugin ◄──── WebSocket ────► Bridge Server (Node.js, port 9120)
                                          ├── PTY Manager → claude CLI
User's Terminal ◄──── stdio proxy ──────► ├── Output Parser → State Machine
Claude Code Hooks ──── HTTP POST ──────► ├── Hook Server + SSE
                                          ├── WS Server → Plugin / Android / Apple / TUI
Android Dashboard ◄── WebSocket + mDNS ► ├── Voice (whisper.cpp)
Apple Dashboard   ◄── WebSocket + mDNS ► ├── (iOS/iPad/macOS)
Pixoo64 LED       ◄── HTTP push ────────► ├── Pixoo Module (pixel art renderer)
ESP32 Display     ◄── WebSocket ────────► ├── Serial Module (UART/WiFi)
TUI Dashboard     ◄── WebSocket ────────► └── (agentdeck dashboard)

The bridge spawns Claude Code in a PTY, parses output + hook events into a state machine, and broadcasts state to all 11 surfaces over WebSocket. Local clients are auto-trusted; LAN clients authenticate with a token from ~/.agentdeck/auth-token. All surfaces can monitor and control the agent independently or simultaneously.

Prerequisites

Item	Required	Install
macOS 14+ (Sonoma)	Yes	Windows/Linux not supported
Node.js >= 20	Yes	`brew install node`
pnpm	Yes	`npm install -g pnpm`
Elgato Stream Deck app >= 6.7	Yes	Elgato Downloads
Stream Deck+ hardware	Yes	8 keys + 4 encoders + LCD touch strip
iTerm2	Yes	Terminal management, voice paste, session switching
Claude Code CLI	Yes	`npm install -g @anthropic-ai/claude-code`
Stream Deck CLI	Auto	Installed by `pnpm setup` if missing
sox (audio capture)	For voice	See Voice Setup
whisper.cpp (transcription)	For voice	See Voice Setup

Quick Start

# Option A: npm install (no clone needed)
npx @agentdeck/setup

# Option B: from source
git clone https://github.com/puritysb/AgentDeck.git && cd AgentDeck && pnpm setup

The pnpm setup command:

Checks required dependencies (Node.js 20+, pnpm, Claude CLI, Stream Deck app)
Installs @elgato/cli if missing
Runs pnpm install + pnpm build
Generates icon assets (16 PNGs)
Installs Claude Code hooks
Links the Stream Deck plugin
Links the agentdeck / sdc CLI globally
Checks optional dependencies (sox, whisper.cpp)

After setup, restart the Stream Deck app, then run:

sdc

You're steering.

Manual Build & Install

Build

cd AgentDeck
pnpm install
pnpm build            # shared → bridge, plugin, hooks
pnpm generate-icons   # SVG → PNG (required on first build)

1. Install Claude Code Hooks

node hooks/dist/install.js

Registers 7 hooks in ~/.claude/settings.local.json: SessionStart, SessionEnd, PreToolUse, PostToolUse, Stop, Notification, UserPromptSubmit. Each hook POSTs JSON to the bridge. Remove with node hooks/dist/install.js uninstall.

2. Link Stream Deck Plugin

cd plugin && streamdeck link .sdPlugin

Creates a symlink in ~/Library/Application Support/com.elgato.StreamDeck/Plugins/. Restart the Stream Deck app to load.

3. Link `sdc` CLI

cd bridge && pnpm link --global

4. Voice Setup (Optional)

Voice input requires sox (audio capture) and whisper.cpp (local transcription). Apple Silicon users must use arm64 Homebrew (/opt/homebrew/) for Metal GPU acceleration.

See Voice Setup Guide for full instructions including architecture verification, model selection, and troubleshooting.

Usage

Start

agentdeck claude

This starts the bridge on port 9120 (HTTP + WebSocket), spawns Claude Code inside a PTY, and proxies your terminal transparently. Use Claude exactly as before — the Stream Deck adds a parallel control channel.

Security: The bridge binds to 0.0.0.0 for LAN access (multi-surface monitoring). Local connections bypass authentication. Remote connections require the auth token from ~/.agentdeck/auth-token.

CLI Reference

agentdeck and sdc are aliases — both work identically.

Sessions

Command	Description
`agentdeck claude`	Start Claude Code session (PTY + bridge)
`agentdeck monitor`	Hook-only bridge (no PTY — run `claude` separately)

Flags: -p <port>, -c <command>, -d (debug), --no-update-check Module flags: --local (all off), --no-mdns, --no-adb, --no-serial, --no-pixoo

Daemon

Command	Description
`agentdeck daemon start`	Start monitoring daemon
`agentdeck daemon stop`	Stop daemon
`agentdeck daemon restart`	Restart daemon
`agentdeck daemon status`	Show daemon status
`agentdeck daemon install`	Register macOS LaunchAgent (auto-start)
`agentdeck daemon uninstall`	Remove LaunchAgent

Session Management

Command	Description
`agentdeck status`	All sessions + daemon status
`agentdeck stop [session]`	Stop a session (`-a` for all)
`agentdeck attach [session]`	Attach terminal to session

Monitoring

Command	Description
`agentdeck dashboard`	TUI monitoring dashboard (alias: `dash`)
`agentdeck devices`	Connected devices (WS, ESP32, Pixoo, ADB)
`agentdeck qr`	Pairing QR code + URL
`agentdeck diag`	Diagnostic dump (`-a` for AI analysis)

Device Setup

Command	Description
`agentdeck pixoo scan`	Discover Pixoo devices on LAN
`agentdeck pixoo add <ip>`	Add a Pixoo device
`agentdeck pixoo list`	List configured devices
`agentdeck pixoo remove <ip>`	Remove a device
`agentdeck pixoo test [ip]`	Send test pattern
`agentdeck wifi-setup`	ESP32 WiFi provisioning (serial)

Stream Deck+ Layout (v3)

Keypad — 8 Actions

┌────────┬─────────┬─────────┬───────────┐
│  MODE  │ SESSION │  USAGE  │  GO ON    │
├────────┼─────────┼─────────┼───────────┤
│ REVIEW │ COMMIT  │  CLEAR  │   STOP    │
└────────┴─────────┴─────────┘───────────┘

Slot	Action	Description
0	Mode	Toggle Default / Plan / Accept Edits
1	Session	Project name + state + session switch
2	Usage	Usage dashboard (5h / 7d / extra / session / models / oc-usage pages)
3-6	Quick Action x4	GO ON / REVIEW / COMMIT / CLEAR when idle — up to 4 options on permission/select prompt. 5+ options → 3 + MORE
7	Stop	Interrupt (Ctrl+C when processing) / Escape (when idle)

Encoders — 4 Slots

Encoder	Action	Rotate	Push	Touch
E1	Utility	Adjust value (volume, mic, timer)	Toggle / Action	Switch mode
E2	Action	Scroll options / cycle prompts	Send prompt / Confirm	Same as push
E3	Terminal	Switch iTerm session	Activate / Attach tmux	—
E4	Voice	Scroll transcription text	Hold = record, tap (<500ms) = cancel	—

Left: Voice transcription (Korean) on wide-canvas LCD | Right: Model selection with encoder option list

Dynamic Button States

Slots 3-6 reconfigure based on agent state — permission prompts get semantic colors (green=approve, red=deny, blue=permanent), options get teal/green, and 5+ options collapse into encoder wide-canvas mode.

See Stream Deck+ Layout Reference for per-state ASCII diagrams, color tables, encoder details, and button label intelligence.

Android Dashboard

Monitor and control your AI agents from any Android device — no Stream Deck required.

Tablet mode — color terrarium with multi-session octopi, crayfish, neon tetra, and HUD overlay

E-ink mode (Crema S) — 16-level grayscale aquarium with agent list, rate limits, and event timeline

The Android app connects to the same bridge server over your local network, giving you a second screen for agent monitoring and a full mirror of the Stream Deck controls.

Two Display Modes

E-ink mode (Crema S, Onyx, Kobo)

Aquarium-centered B&W dashboard — pixel art creatures in a 16-level grayscale terrarium
Partial refresh zones: A2 (200ms) for fast UI, DU for status, FULL (500ms) for the aquarium
Left panel (22%): agent list with state indicators
Right panel (78%): aquarium + rate limits/models + event timeline

Tablet mode (Lenovo, general Android tablets)

Full-color terrarium background with 60fps creature animation
Semi-transparent HUD panels overlay agent status, rate limits, timeline
Identical information to e-ink, expressed through color and motion

Three-Tab Navigation

Tab	Content
Dashboard	Terrarium background + HUD overlay panels. Connection overlay when disconnected (mDNS discovery, QR pairing)
Deck	Full Stream Deck+ mirror — 4 encoder panels (swipe/tap/long-press gestures) + 2x4 button grid with context area
Settings	Bridge connection, display preferences

Connect to Bridge

The app finds your bridge automatically:

mDNS — the bridge advertises _agentdeck._tcp on your local network; the app discovers it within seconds
QR pairing — run agentdeck qr on your Mac, scan with the app's camera (CameraX + ML Kit)
Manual — enter the bridge IP and port in Settings

Once connected, the app receives real-time state updates over WebSocket and can send commands back to the bridge.

See Android Reference for device support, build/signing instructions, and creature behavior details.

Apple Dashboard

Monitor and control your AI agents from iPhone, iPad, or Mac — a native SwiftUI experience.

The Apple app is a SwiftUI multiplatform app that connects to the same bridge server, providing the full AgentDeck experience on Apple devices.

Three-Tab Navigation

Tab	Content
Monitor	Terrarium background + HUD overlay — agent status, rate limits, timeline
Deck	Stream Deck+ mirror — encoder panels + button grid with touch gestures
Settings	Bridge connection, display preferences

Connect to Bridge

mDNS — automatic discovery of _agentdeck._tcp services on your local network
QR pairing — scan with the in-app camera (agentdeck qr on your Mac)
Manual — enter bridge IP and port

iOS Foreground Recovery

The app handles iOS background/foreground transitions gracefully — WebSocket reconnects immediately on foregrounding, state syncs within milliseconds, and the terrarium animation resumes without flicker.

TUI Dashboard

Monitor your agents directly in the terminal — no additional hardware or apps required.

agentdeck dashboard     # or: agentdeck dash

The TUI connects to a running Bridge or Daemon over WebSocket and renders a real-time monitoring interface using raw ANSI escape codes. Zero additional dependencies.

Braille terrarium — octopus, crayfish, neon tetra schools in a truecolor water gradient. State-driven: idle = floor, processing = swimming with starburst, awaiting = "?" bubble
Adaptive layout — wide (120+ cols), standard (80-119), narrow (60-79). Terrarium hides when too small
Status + Timeline — rate limit gauges, OAuth/Ollama models, tool calls, activity density bar
Auto-discovery — finds Daemon or active session automatically; 3s auto-reconnect
SSH friendly — works over any SSH connection with truecolor support; pipes output JSON

ESP32 Display

Compact WiFi-connected displays for always-on agent monitoring.

Left: Round AMOLED close-up (1.8" circular) | Right: Round AMOLED + IPS LCD side by side

Supported Boards

Board	Screen	Resolution
Round AMOLED	1.8" circular AMOLED	466×466
IPS LCD	3.5" rectangular IPS	480×320
B86 Box	4" wall-mount touch panel	480×480

Setup

Run agentdeck wifi-setup to provision WiFi over serial (see CLI Reference). Once provisioned, the ESP32 connects to the bridge over WiFi WebSocket and displays a compact terrarium with agent status. PlatformIO firmware in esp32/.

Pixoo64 LED Matrix

64×64 RGB LED pixel art terrarium on a Divoom Pixoo64.

The Pixoo module renders dot-art creatures, water zone colors reflecting agent state, and a compact usage HUD — all pushed over HTTP to the device's local API.

Manage devices with agentdeck pixoo {scan|add|list|remove|test} — see CLI Reference.

Note: The Pixoo's built-in HTTP server can crash under frequent requests. AgentDeck throttles updates automatically. Use --no-pixoo to disable if needed.

Project Structure

AgentDeck/
├── shared/        # Shared TypeScript types (states, protocol, voice paths)
├── bridge/        # Node.js bridge server (PTY, hooks, WS, voice, adapters)
├── plugin/        # Stream Deck SDK v2 plugin (actions, renderers, utility modes)
├── hooks/         # Claude Code hook installer
├── setup/         # npm setup package (@agentdeck/setup)
├── android/       # Jetpack Compose dashboard (e-ink + tablet, terrarium, deck mirror)
├── apple/         # SwiftUI multiplatform app (iOS/iPad/macOS dashboard + deck mirror)
├── esp32/         # ESP32 firmware (round AMOLED / IPS LCD, PlatformIO)
├── config/        # Prompt templates + default settings
├── scripts/       # Install, uninstall, package, icon generation
└── docs/          # Documentation (voice, layout, android, protocol)

See Protocol & Architecture for the full file tree, state machine diagram, and WebSocket message reference.

Configuration

Quick Action Buttons

The four Quick Action buttons (slots 3-6) are configurable via the Stream Deck Property Inspector. Defaults:

Slot	Label	Action
3	GO ON	`continue` (sends prompt to continue)
4	REVIEW	`/review`
5	COMMIT	`/commit`
6	CLEAR	`/clear`

Slot 3 also shows START when disconnected (spawns a new agentdeck claude session).

Prompt Templates

Edit config/prompt-templates.json to customize the prompts cycled by the Action encoder (E2) rotate:

{
  "templates": [
    { "label": "Fix Bug", "prompt": "Please fix the bug described above" },
    { "label": "Test", "prompt": "Write tests for the changes made" },
    { "label": "Review", "prompt": "Review the code for issues and suggest improvements" },
    { "label": "Explain", "prompt": "Explain how this code works step by step" }
  ]
}

Troubleshooting

Symptom	Cause	Fix
Plugin shows DISCONNECTED	Bridge not running	Run `sdc`
Plugin reconnects every 3s	Bridge crashed	Restart `sdc`
Bridge enters disconnected state	Claude process exited	Restart `sdc`
State tracking not working	Hook server unreachable	Verify `sdc` is running
Stream Deck buttons inactive	Hardware not connected	Reconnect + restart app
Stuck in PROCESSING > 5 min	Agent stalled	STOP button or Ctrl+C in terminal
"Is sox installed?"	sox missing	See Voice Setup
"Is whisper.cpp installed?"	whisper.cpp missing	See Voice Setup
Voice transcription very slow / timeout	x86 whisper-cli (no Metal GPU)	Install arm64 Homebrew + whisper-cpp. See Voice Setup
`whisper-cli: arm64=false, metal=false`	Using x86 binary through Rosetta	Install arm64 Homebrew at `/opt/homebrew/`
Plugin not in Stream Deck app	Plugin not linked	Restart Stream Deck app, then `cd plugin && streamdeck link .sdPlugin`
Hooks not firing	Hooks not installed or stale	`node hooks/dist/install.js` (re-installs all 7 hooks)
Need to remove hooks	Uninstalling AgentDeck	`node hooks/dist/install.js uninstall`
Plugin loads but buttons blank	Plugin needs rebuild	`pnpm build && pnpm generate-icons`, restart Stream Deck app
Android app can't find bridge	mDNS blocked on network	Use QR pairing (`agentdeck qr`) or enter IP manually in Settings
Android shows "Not Connected"	Bridge not reachable	Verify same LAN; for USB: `adb reverse tcp:9120 tcp:9120` then connect to 127.0.0.1:9120
E-ink ghosting on Crema	Missing full GC16 refresh	State transitions trigger full refresh automatically; force refresh by toggling bridge connection

tmux -CC Compatibility

When using iTerm2's tmux -CC (control mode): run sdc inside a tmux window. The bridge manages its own PTY, so there's no conflict.

Signal chain: tmux → iTerm2 → sdc → bridge PTY → claude

Uninstall

bash scripts/uninstall.sh

Removes Claude Code hooks, unlinks sdc CLI, and removes the Stream Deck plugin symlink. Restart the Stream Deck app afterward.

Development

pnpm -r --parallel dev    # Watch mode for all packages
cd plugin && pnpm build   # Rebuild plugin only
cd bridge && pnpm build   # Rebuild bridge only
pnpm -r typecheck         # Type check without building

Testing

pnpm test                 # Run all tests (vitest)
pnpm test -- --watch      # Watch mode

Tests cover output parsing, state machine transitions, hook installation, option rendering, and text utilities. Quick smoke test after changes:

pnpm build && pnpm test && sdc status

Packaging

Build a distributable .streamDeckPlugin file:

pnpm package    # → dist/bound.serendipity.agentdeck.streamDeckPlugin

Recipients double-click to install. The bridge (sdc) and Claude Code CLI must be installed separately.

Published npm packages: @agentdeck/shared, @agentdeck/bridge, @agentdeck/setup.

Debugging

Bridge logs print to the sdc terminal:

[sdc] Starting AgentDeck bridge on port 9120...
[sdc] Hook server listening on port 9120
[sdc] WebSocket server ready on port 9120
[sdc] Spawned: claude
[WsServer] Plugin connected
[StateMachine] DISCONNECTED -> idle (trigger: session_start, source: hook)

Stream Deck plugin logs: Stream Deck app → Settings → Logs.

Roadmap

Achieved

Android tablet + e-ink dashboard (Jetpack Compose)
Apple iOS/iPad/macOS dashboard (SwiftUI multiplatform)
ESP32 compact displays (Round AMOLED, IPS LCD, B86 Box)
TUI terminal dashboard (Unicode Braille + ANSI)
Pixoo64 LED matrix pixel art
Multi-agent visualization (Claude Code + OpenClaw creatures)
Daemon mode with multi-session aggregation

Planned

Additional agent integrations (Opencode, Codex CLI)
Windows/Linux platform support
Project-specific layout presets
Custom button icon support
App Store distribution (Apple)

AgentDeck — Physical Control Surface for AI Coding Agents

Name		Name	Last commit message	Last commit date
Latest commit History 171 Commits
.agents/workflows		.agents/workflows
.github/workflows		.github/workflows
android		android
apple		apple
bridge		bridge
config		config
docs		docs
esp32		esp32
hooks		hooks
plugin		plugin
scripts		scripts
setup		setup
shared		shared
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
DEVELOPMENT_LOG.md		DEVELOPMENT_LOG.md
README.md		README.md
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.base.json		tsconfig.base.json
vitest.config.ts		vitest.config.ts

Folders and files

Latest commit

History

Repository files navigation

AgentDeck

Table of Contents

Documentation

What is AgentDeck?

Supported Agents

Supported Surfaces — 11 Types

Architecture

Prerequisites

Quick Start

Manual Build & Install

Build

1. Install Claude Code Hooks

2. Link Stream Deck Plugin

3. Link sdc CLI

4. Voice Setup (Optional)

Usage

Start

CLI Reference

Sessions

Daemon

Session Management

Monitoring

Device Setup

Stream Deck+ Layout (v3)

Keypad — 8 Actions

Encoders — 4 Slots

Dynamic Button States

Android Dashboard

Two Display Modes

Three-Tab Navigation

Connect to Bridge

Apple Dashboard

Three-Tab Navigation

Connect to Bridge

iOS Foreground Recovery

TUI Dashboard

ESP32 Display

Supported Boards

Setup

Pixoo64 LED Matrix

Project Structure

Configuration

Quick Action Buttons

Prompt Templates

Troubleshooting

tmux -CC Compatibility

Uninstall

Development

Testing

Packaging

Debugging

Roadmap

Achieved

Planned

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

3. Link `sdc` CLI

Packages