Migration to rmw_zenoh

[luca-della-vedova]

Since rmw_zenoh was released as a binary (pending the next Jazzy sync) we should be able to migrate to it. Incidentally this will also help cutting down compilation time since we don't need to vendor / build the bridge from source anymore.
We can update nexus_network_configuration to generate the json5 config files needed for zenoh configuration.

This ticket is to discuss what the architecture would look like. One possible architecture (but I'm in no way an rmw_zenoh expert!):

One router per workcell

Each workcell would have its own zenoh router, from what I understand this is pretty much necessary to allow nodes within a workcell to communicate with each other. If there was only one global router at the system orchestrator level they would need to connect to it to discover nodes.

Workcell sessions

Each workcell node will have a workcell specific session config file where it connects to the workcell's router. We will use the domainid to namespace the workcell topics. We probably (?) don't need to set up an ACL in the workcell nodes themselves.

Workcell ACL

The workcell routers will have an ACL to make sure that only liveliness tokens / pub / sub / services that are relevant to the system orchestrator are forwarded, while the rest is blocked. An alternative would be to implement the ACL on the sessions (or both?).

System orchestrator

The system orchestrator would be fairly similar to workcells, with its router and communication to local nodes. The only difference is that it will use wildcards to allow communication with all workcells regardless of their domain id.

CC @Yadunund for feedback / ideas on whether this is a viable architecture or what a better one would look like.

[Yadunund]

Here's my recommendation which does not require any namespacing as that can get tricky.

• Each orchestrator will have unique router and session configs resp.
• For workcells and system orchestrators, take the ros_domain_id for each and assign a port for the zenohd router. Eg. 7447 + ros_domain_id.
• All workcell router configs will include the system orchestrator endpoint in the connect block. ie, each workcell router will connect to the system router.
• Session configs for each workcell and system will include the endpoint for their resp routers only in the connect block.
• Add access control settings to workcell router configs to only expose keyexpressions (liveliness tokens and ROS endpoints) related to orchestration and discovery.

[luca-della-vedova]

A caveat I came upon during implementation, if we set the ACL to default to deny here (the same way we do in the current nexus_network_configuration pipeline), topics will not be able to be discovered.
I asked in the Zenoh discord and it seems we need to add an additional rule that allows the ros2_lv ADMIN_SPACE, (as per this conversation):

key_exprs: ["@ros2_lv/**"],

[aaronchongth]

Nice! Just started venturing down this path and glad that a solution has already been found. Yeah, can confirm that setting ACL with the "@ros2_lv/**" key expression works like a charm.

this seems to be the minimal ACL config required to listen to a topic, leaving it here for reference. (edit: saw that @luca-della-vedova posted a working config too, not too sure how specific we'd like to be to differential between pub, sub, server, client, action server, action client)

I'll try out services and actions, then start prototyping the changes required to nexus_network_configuration when I can

   access_control: {
    "enabled": true,
    "default_permission": "deny",
    "rules":
    [
      {
        "id": "allow_chatter",
        "messages": [
          "put", "declare_subscriber",
          "liveliness_token", "liveliness_query", "declare_liveliness_subscriber",
        ],
        "flows":["egress", "ingress"],
        "permission": "allow",
        "key_exprs": [
          "**/chatter", "@ros2_lv/**"
        ],
      },
    ],
    "subjects":
    [
      {
        "id": "subject3",
      },
    ],
    "policies":
    [
       {
          "rules": ["allow_chatter"],
          "subjects": ["subject3"],
       },
    ]
  },

[Yadunund]

Sounds good! We can work with static files first and update nexus_network_configuration later since the existing tests rely on static config files anyways.

[aaronchongth]

I've started prototyping here, #76, and ran into some issues and concerns.

In the event where we launch everything with the same domain ID, and rely on ACL, important messages like /tf and /tf_static get crossed between workcells and stuff starts to break. Since we are "simulating" multiple workcells locally, I'm guessing we should still handle the separation with ROS_DOMAIN_ID.

Keeping each workcell on separate domain IDs (like we are doing now already), will however separate the workcell registration service and workcell task request action. I've so far not found support for bridging between 2 different domain IDs in rmw_zenoh, other than using zenoh_bridge_ros2dds like what we are already using.

The one thing I've not tried, is to use the same domain ID for everything, and for the peers under each workcell, use multicast for scouting instead of gossip, and give them dedicated ports as well. So in theory, the sessions from different workcells should ignore each other.

Curious to hear if I'm missing something obvious!

[Yadunund]

With Zenoh 1.4 and the latest rmw_zenoh (eg 0.2.5 binary version on Jazzy), routers can attach and strip namespaces on the fly. See example config.

Each workcell (and system orchestrator) can start it's own Zenoh router and have nodes for that workcell connect only to this router.

We can connect different routers while apply ACLs to ensure only topics we care about are bridged. The outgoing keyexpressions from workcells can have a namespace applied. The system orchestrator router can strip this namespace.

Should we try this out?

[aaronchongth]

I tried it out the namespacing a while ago for other projects, but not on this multi-router setup, not sure if things have changed. From what I remember the namespace is applied on the whole zenoh key expression, and not the ROS 2 topic itself, and therefore for a connecting router to translate that key expression back into a ROS 2 topic, the connecting router will also need to have the same namespace applied, before the key expression->topic demangling can happen.

So if we have a few workcells with their namespaced routers connected to the system orchestrator's router, for example

• each workcell has a topic /chatter
• each workcell's router will namespace it to something like /workcell_n/*/chatter/*
• the system orchestrator's router will apply ACL to accept /**/chatter/*, which will accept all the namespaced key expressions
  However I believe there might be no way to translate the namespaced key expressions into namespaced ROS 2 topics/services/actions, unless we work directly with Zenoh pub/sub/get/queryable, and move the namespaces

For more specific topics like /tf and /tf_static, we'd want to strip namespace but not namespace ROS 2 topics. So different behaviors will need to be accommodated.

I haven't tried it out yet to verify this limitation, but just in case I missed something, is there a way around this?

[Yadunund]

Summarizing what we discussed earlier,

• Each workcell's router can add the same namespace, eg. /so
• The System Orchestrator's router strips away the same namespace.
• This "should" work since the topics we want to expose should already be namespaced,

  nexus/nexus_demos/config/zenoh/nexus_network_config.yaml

  Lines 38 to 40 in 4446069

  	service_servers: ["/workcell_1/is_task_doable", "/workcell_1/queue_task", "/workcell_1/remove_pending_task", "/workcell_1/get_state", "/workcell_1/change_state", "/workcell_1/signal", "/workcell_1/pause"]
  	service_clients: ["/register_workcell"]
  	action_servers: ["/workcell_1/request"]

Haven't tried it myself.

In addition we can

• Replace the zenoh configs here with sros2 policies which can passed to zenoh_security_tools to generate the zenoh configs.
• Remove the nexus_network_configuration package together with the zenoh_bridge_dds_vendor.