Zenoh Investigation

[leshy]

Intro

Dimos is distributed multiprocess architecture with predefined message types exchanged between processes using predefined transports

Most of our code is in Python, python is our prototyping and orchestration language, but we support modules written in multiple third party languages

Problem

Our transports until now have been on easy mode - local RPC/pubsub only, so we've had guaranteed delivery, and we were mostly concerned with backpressure, throughput and latency. LCM protocol (local UDP multicast worked great)

As we start working with teleop, multi embodiment coordination (on shared networks of different underlying capabilities) heavy real time computation offloaded into remote clusters and similar, we need to start thinking about more mature communication protocols.

Zenoh seems promising!

some intro questions (I'm sure there are many more critical ones, this is just to kickstart the way we think about this)

• can we get reliable query/response multi robot system on a local unreliable networks (assume from WIFI to GSM to LORA mesh)
• can we get a pubsub? how does this work? autodiscovery, distributed vs broker, how would we run a broker if multirobot? how does distributed pubsub work?
• can we run a single global mapper node via that pubsub (collecting lidar frames from all robots, assembling global map, planning, streaming paths back to robots and their local planners)
• assume we need to upload video from multiple robots sharing a local robot network, with single (or just a few) robots having an internet connection
• assume we need to run teleop, so transport efficiently encoded low latency video in one direction, and light control messages in other direction (NAT penetration?)
• assume we are on a high throughput network with internet connection, can we upload all sensor data in realtime into a cluster? can we stream control commands to the robot?
• assume we want to run requests to a remote server, potentially stream responses, requests could be image frames, LLM queries etc
• can we have global, local, internet level pubsubs, how do we control what goes where and when?
• where are backpressure strategies executed, broker, receiver, emitter? (and which ones) what is the appropriate backpressure approach in different scenarios?
• what are the QOS strategies available?

other useful features

Zenoh is fancy, idk what it has, but stuff like this is interesting to know about:

• etcd style distributed KV store?
• raft style automatic master node election?
• meshing?

Example scenario

Write some zenoh transport, configure it well enough to split the system, local control, remote global mapping and path planning on a laptop, or, slow GPU heavy module offloaded to a server

[vrinek]

• can we get reliable query/response multi robot system on a local unreliable networks (assume from WIFI to GSM to LORA mesh)

• QUIC handles re-connections and Zenoh's RELIABLE mode handles retransmissions.
• Zenoh is built as a pub/sub system, but it natively supports query/response. A router can cache data if the original publisher is not immediately available.
• Zenoh supports serial links, so LoRa is possible, but it's extremely low bandwidth. I don't think that sensor data (RGBD) can be shared via LoRa, so what's the use-case?

• can we get a pubsub? how does this work? autodiscovery, distributed vs broker, how would we run a broker if multirobot? how does distributed pubsub work?

• Pub/sub is the main mode of Zenoh. A publisher publishes a "value" (bytes, Zenoh does not care) on a "key" (dimos/robots/123/lidar), subscribers subscribe to a key, or an expression (dimos/robots/*/lidar).
• Autodiscovery works through UDP multicast in a LAN.
• Peer mode is enough for ~10 robots (Moonbots validated 10+). After that, we introduce "routers" to handle message delivery between "clients" and "peers". Client only connect to routers. Peers connect to other peers and routers.
• Multi-site/WAN topologies require routers.

• can we run a single global mapper node via that pubsub (collecting lidar frames from all robots, assembling global map, planning, streaming paths back to robots and their local planners)

• In short, yes.
• Robots publish their lidar data at dimos/robots/{id}/lidar, with BEST_EFFORT + DROP reliability.
• Mapper subscribes at dimos/robots/*/lidar, build map, generates plans and paths for each robot.
• Mapper publishes paths at dimos/robots/{id}/plans/path.
• Each robot subscribes to dimos/robots/{id}/plans/path to receive its own path.

• assume we need to upload video from multiple robots sharing a local robot network, with single (or just a few) robots having an internet connection

• The internet connected robot(s) run a router.
• Other robots connect in peer mode.
• The router(s) connect to the cloud router.
• The cloud router routes the video feeds
• The cloud processing service subscribes to the video feeds (dimos/robots/*/sensors/camera)
• The robots publish their video feeds (dimos/robots/{id}/sensors/camera) with BEST_EFFORT and DROP

• assume we need to run teleop, so transport efficiently encoded low latency video in one direction, and light control messages in other direction (NAT penetration?)

• Zenoh does not punch holes through NAT. We can use a VPN for this, or make a router publicly available, but secure the hop to it with mTLS.
• We can use asymmetric QoS. Video feed (from robot to operator) BEST_EFFORT + DROP + DATA. Control (operator to robot) RELIABLE + BLOCK + REAL_TIME + express=true.
• We can supplement this with a heartbeat feed to ensure that the robot can safely pause operations if the operator is disconnected for more that N seconds.
• QUIC is paramount for this setup because it handles reconnections, encrypts the connection and multiplexes the feeds

• assume we are on a high throughput network with internet connection, can we upload all sensor data in realtime into a cluster? can we stream control commands to the robot?

• Yes. The same setup as above will scale up to use the available bandwidth.
• Since we can configure priority, reliability and congestion control separately, we can fine-tune QoS for each scenario.
• Zenoh has been benchmarked up to 67 Gbps in controlled tests.

• assume we want to run requests to a remote server, potentially stream responses, requests could be image frames, LLM queries etc

• Yes.
• Assuming the remote server is accessible (via a router path or mesh networking).
• The server declares a queryable key, and the client sends a GET query. The server responds with a payload, and the client receives it.
• This is not streaming. It is more like RPC.
• To stream responses (eg user sends a prompt, the LLM streams back the response), we'd use query/response for the prompt/ack and pub/sub for the streaming response.

• can we have global, local, internet level pubsubs, how do we control what goes where and when?

• Yes, via key expression namespaces.
• ACL can also control which node has access to what feed. ACLs are enforced by the routers.
• Zenoh's detailed QoS settings control relative prioritization.
• It supports rate limiting key expressions (eg dimos/robots/*/lidar capped at 5Hz).

• where are backpressure strategies executed, broker, receiver, emitter? (and which ones) what is the appropriate backpressure approach in different scenarios?
• what are the QOS strategies available?

• QoS is configured at the publisher, and propagated through the routers chain.
• Zenoh provides reliability (BEST_EFFORT, RELIABLE), congestion (BLOCK, DROP), and priority (7 levels) controls.

Suggested controls:

Data type	Transport	QoS	Latency budget
Motor control (cmd_vel)	TCP on LAN, QUIC over WAN	RELIABLE + BLOCK + REAL_TIME + express	<10ms
Perception (camera, lidar)	SHM same-host, TCP cross-host	BEST_EFFORT + DROP + DATA/DATA_HIGH	50-100ms
State (odom, map)	TCP	RELIABLE + DROP + DATA	50-100ms
Telemetry uploads	QUIC to cloud	BEST_EFFORT + DROP + BACKGROUND	seconds OK
RPC (module calls)	TCP	RELIABLE + BLOCK + INTERACTIVE_HIGH	10-50ms

other useful features

Zenoh is fancy, idk what it has, but stuff like this is interesting to know about:

• etcd style distributed KV store?

• Yes. Zenoh has a storage-manager plugin to persist pub/sub data.
• It's not as feature-rich as etcd, but should be sufficient for configuration state.

• raft style automatic master node election?

• No built-in consensus mechanism.
• What's the use case for this?

• meshing?

• Yes. Peer mode with linkstate enables multi-hop mesh networking.

Example scenario

Write some zenoh transport, configure it well enough to split the system, local control, remote global mapping and path planning on a laptop, or, slow GPU heavy module offloaded to a server

The split is based on the actual Go2 blueprint (unitree_go2). The real module graph is:

GO2Connection → VoxelGridMapper → CostMapper → ReplanningAStarPlanner → cmd_vel → GO2Connection
     │                                               ▲
     └─► Detection3DModule                           │
                                               odom ─┘

In a split deployment, we keep the hardware connection and control loop on the robot, and move the compute-heavy mapping and planning to a remote machine.

Topology

┌───────────────────────────────────────┐
│          Robot (192.168.1.10)         │
│                                       │
│  ┌──────────────────┐                 │
│  │  GO2Connection   │  publishes:     │
│  │  (hardware)      │    lidar        │
│  │                  │    odom         │
│  │  cmd_vel ◄───────┼─── from planner │
│  │  lidar ──────────┼──► to mapper    │
│  │  odom ───────────┼──► to planner   │
│  │  color_image ────┼──► to detector  │
│  └──────────────────┘                 │
│  ┌──────────────────┐                 │
│  │ Detection3D      │  publishes:     │
│  │ Module           │    detections   │
│  └──────────────────┘                 │
│                                       │
│  ┌─────────────────┐                  │
│  │  Zenoh Router   │  bridges to      │
│  │  (zenohd :7447) │  remote          │
│  └────────┬────────┘                  │
└───────────┼───────────────────────────┘
            │ tcp/192.168.1.10:7447
            │
┌───────────┼───────────────────────────┐
│           ▼   Laptop (192.168.1.20)   │
│                                       │
│  ┌──────────────────┐                 │
│  │ VoxelGridMapper  │  subscribes:    │
│  │                  │    lidar        │
│  │  global_map ─────┼──►              │
│  └──────────────────┘    │            │
│  ┌──────────────────┐    │            │
│  │ CostMapper       │◄───┘            │
│  │                  │                 │
│  │  global_costmap──┼──►              │
│  └──────────────────┘    │            │
│  ┌──────────────────┐    │            │
│  │ ReplanningAStar  │◄───┘            │
│  │ Planner          │  subscribes:    │
│  │                  │    odom         │
│  │  cmd_vel ────────┼──► to robot     │
│  └──────────────────┘                 │
└───────────────────────────────────────┘

Configs

Robot modules (peer mode, local communication + router for uplink):

// Robot modules config
{
  mode: "peer",
  listen: { endpoints: ["tcp/[::]:0"] },
}

Robot router (bridges local peers to remote clients):

// zenohd on robot
{
  mode: "router",
  listen: {
    endpoints: ["tcp/0.0.0.0:7447"],
  },
}

Laptop modules (client mode, connects to robot's router):

// Laptop mapping/planning config
{
  mode: "client",
  connect: {
    endpoints: ["tcp/192.168.1.10:7447"],
  },
}

Key Expression Design

Following Zenoh's key-space guidelines, we separate kind from ID at every level. This keeps * wildcards efficient (no need for slower $*) and makes the hierarchy queryable.

Mapped from the actual Go2 module streams:

dimos/robots/{id}/color_image    → RGB frames [Image] (BEST_EFFORT, DROP, DATA)
dimos/robots/{id}/lidar          → lidar points [PointCloud2] (BEST_EFFORT, DROP, DATA_HIGH)
dimos/robots/{id}/odom           → odometry [PoseStamped] (BEST_EFFORT, DROP, DATA)
dimos/robots/{id}/cmd_vel        → velocity commands [Twist] (RELIABLE, BLOCK, REAL_TIME)
dimos/robots/{id}/global_map     → voxel map [PointCloud2] (RELIABLE, DROP, DATA)
dimos/robots/{id}/global_costmap → cost map [OccupancyGrid] (RELIABLE, DROP, DATA)
dimos/robots/{id}/path           → planned path [Path] (RELIABLE, DROP, DATA_HIGH)
dimos/robots/{id}/detections     → detected objects [Detection2DArray] (RELIABLE, DROP, DATA)
dimos/services/vlm/query         → VLM service queryable (RELIABLE, BLOCK)

The stream names (color_image, lidar, odom, cmd_vel, global_map, global_costmap, path) match the actual In[T]/Out[T] declarations on the dimos modules. The Blueprint autoconnect system matches streams by name and type — so the Zenoh key expression is derived from the stream name.

This structure enables clean wildcard subscriptions:

• dimos/robots/*/lidar — lidar from all robots (for a central mapper)
• dimos/robots/3/** — everything from robot 3
• dimos/robots/*/cmd_vel — all robots' control commands
• dimos/services/** — all services

This example works today with the Zenoh Python API — no dimos code changes needed to validate the topology. The dimos integration would wrap this in the Transport abstraction so modules don't need to know about Zenoh directly.

Deployment concerns

• WiFi drops mid-teleop: QUIC reconnects automatically. Add application-level heartbeat + watchdog on the robot side. If no heartbeat for 2s, safety stop.
• Security: mTLS for client auth, ACL for per-key-expression access control. Whitelist model: deny by default, explicitly allow each robot its own namespace. Tailscale/WireGuard is simpler for early prototyping.
• Latency budget: SHM for same-host perception, TCP for LAN control loops, QUIC for WAN. Don't use lowlatency mode — it kills QoS prioritization. Use express flag instead for critical messages.
• 50+ robots: Router topology, not peer mesh. Clients connect to 1-2 routers. Namespace isolation prevents cross-talk.
• Protocol versioning: Minor versions (1.x) are backward compatible. Pin to a range in pyproject.toml. Don't depend on unstable API features in production.
• SHM performance: Zenoh SHM reports ~3M msg/s. Need to benchmark against dimos's custom SharedMemoryPubSubBase before committing. Silent TCP fallback is the main gotcha.
• Migration path: Incremental. Blueprint .transports() already supports per-stream transport selection. Zenoh is opt-in alongside LCM. Bridge for mixed deployments. No rip-and-replace.