Ghost Hands

Wireless bilateral teleoperation for SO-101 robot arms. Move the leader arm with your hand, the follower mirrors it — and you feel the resistance back through your fingers.

Two ESP32s, one over each arm, talk to each other directly over ESP-NOW. A web UI on the host machine handles calibration, monitoring, and 3D visualization.

Highlights

• Bilateral force feedback — load on the follower is reflected as resistance on the leader
• ESP-NOW peer link — ~5 ms one-hop latency, no router, no pairing setup
• 100 Hz control loop — position, speed, load, temperature on all 6 joints
• WebSocket + RPC — JSON-RPC procedures and a 33 Hz arm-state stream for host integration
• Web UI — connect serial ports, run a guided calibration wizard, watch live joint state and a 3D arm
• Three boards supported — ESP32-WROOM-32, Seeed XIAO ESP32-C3, Seeed XIAO ESP32-S3

Hardware (per arm)

Component	Qty
ESP32-WROOM-32 DevKit or Seeed XIAO ESP32-C3/S3	1
Seeed Studio SO-ARM101 servo controller board	1
Feetech STS3215 servos	6

Wiring

Board	Servo bus TX	Servo bus RX
ESP32-WROOM-32	GPIO16 (TX2)	GPIO17 (RX2)
XIAO ESP32-C3 / S3	D6	D7

Build

pip install platformio
git clone --recursive https://github.com/darshkpatel/ghost-hands.git
cd ghost-hands

pio run -e esp32dev          # ESP32-WROOM-32 (default)
pio run -e xiao_esp32c3      # XIAO ESP32-C3
pio run -e xiao_esp32s3      # XIAO ESP32-S3

pio run -e esp32dev -t upload

Usage

1. Flash one ESP32 per arm (leader + follower).
2. Open the serial monitor: pio device monitor.
3. Pick a role — L for leader, F for follower. Auto-defaults to leader after 5 s. The choice is persisted in NVS.
4. The two boards auto-discover each other via ESP-NOW broadcast.

Serial commands

Command	Description
status / s	Loop timing, ESP-NOW stats, positions, loads
ping	Ping all 6 servos on the bus
gain 0.5 / k 0.5	Set bilateral force-feedback gain (0.0–2.0)
help / h	List commands

Web UI

A React + Three.js app for setup and monitoring. Talks to each ESP32 over the serial port directly (no WiFi required for calibration).

cd web
pnpm install
pnpm dev

The UI guides you through a two-phase joint calibration, persists results to NVS on the board, and visualizes live arm state in 3D. See web/README.md for details.

WebSocket protocol

Set WIFI_SSID and WIFI_PASS in src/main.cpp. The ESP32 then runs a WebSocket server on port 9101.

Request: {"id":"<uuid>","proc":"<name>","payload":{...}} Response: {"id":"<uuid>","ok":true|false,"payload":{...}}

Procedure	Payload	Description
getStatus	—	IP, uptime, heap, connected clients
setGain	{"k":0.5}	Bilateral force-feedback gain (0.0–2.0)
sendAction	{"pos":[...6],"spd":100}	Command servo positions directly
subscribe	—	Start receiving the arm-state stream
unsubscribe	—	Stop the arm-state stream

After subscribe, the server pushes at ~33 Hz:

{
  "stream": "armState",
  "payload": {
    "local": {
      "t": 123456789, "role": 0, "seq": 42,
      "pos":  [2048, 2048, 2048, 2048, 2048, 2048],
      "spd":  [0, 0, 0, 0, 0, 0],
      "load": [100, 50, 30, 20, 10, 5],
      "temp": [35, 36, 34, 33, 32, 31]
    },
    "peer": { ... }
  }
}

Python client

pip install websockets
python host/ghost_client.py ws://<esp32-ip>:9101

from host.ghost_client import GhostClient

client = GhostClient("ws://<esp32-ip>:9101")
await client.connect()

status = await client.get_status()
await client.set_gain(0.5)
await client.send_action([2048] * 6, speed=100)

async for local, peer in client.stream_arm_states():
    print(f"Leader pos: {local.position}, Follower load: {peer.load}")

Architecture

ESP32-A (Leader)              ESP32-B (Follower)
┌──────────────┐              ┌──────────────┐
│ Read leader  │──ESP-NOW──→  │ Write pos to │
│ positions    │  (positions) │ follower     │
│              │              │ servos       │
│ Write torque │←─ESP-NOW───  │ Read loads   │
│ feedback     │  (loads)     │ from servos  │
└──────┬───────┘              └──────┬───────┘
       │ WebSocket / Serial          │ WebSocket / Serial
       ▼                             ▼
    Host (web UI, LeRobot, custom client)

• ESP32-WROOM (dual-core): control loop pinned to core 0, WiFi/WebSocket on core 1.
• XIAO ESP32-C3 (single-core): cooperative scheduling with a timer-driven control loop.
• Calibration is exchanged between peers inside the ESP-NOW packet, so each side knows the other's joint ranges.

Project layout

ghost-hands/
├── platformio.ini          # Build configs for all three boards
├── include/
│   ├── config.h            # Pin mappings, constants, ArmState packet
│   ├── servo_bus.h         # STS3215 servo communication
│   ├── espnow_link.h       # ESP-NOW wireless peer link
│   ├── wifi_stream.h       # WebSocket server + RPC handler
│   ├── bilateral.h         # Force-feedback controller
│   └── calibration.h       # Joint calibration + NVS persistence
├── src/                    # Matching implementations
├── host/
│   └── ghost_client.py     # Python WebSocket client
├── web/                    # React + Three.js setup / monitoring UI
│   └── examples/           # Reference calibrations
└── lib/
    └── scservo/            # Submodule: workloads/scservo

License

MIT — see LICENSE.