Skip to content

Add native in-process SimpleX transport (mode: "native")#26

Closed
lundog wants to merge 5 commits into
dangoldbj:mainfrom
lundog:native-mode
Closed

Add native in-process SimpleX transport (mode: "native")#26
lundog wants to merge 5 commits into
dangoldbj:mainfrom
lundog:native-mode

Conversation

@lundog

@lundog lundog commented Jun 28, 2026

Copy link
Copy Markdown

Note / correction to an earlier draft: a previous version of this
description implied mode: "external" was on a deprecated path. That was my
misreading of SimpleX's announcement. What's deprecated is the standalone
@simplex-chat/webrtc-client
TypeScript WebSocket client librarynot the simplex-chat CLI's
WebSocket-server mode (-p) that external mode relies on, which remains fully
supported (and which this plugin talks to via its own ws client, not that
library). Nothing here deprecates or changes external mode or the sidecar/service
architecture. Native is purely an additive, opt-in alternative transport.

Scope / happy to split: the inbound-file fixes below (file-receive race,
media/inbound staging, and --files-folder path resolution via
connection.filesFolder) are transport-agnostic and improve external mode too.
If you'd rather evaluate native on its own, I'm glad to pull those into a
separate PR so they can land independently.

Why

SimpleX now ships a native simplex-chat
Node.js library that embeds the SimpleX core in-process — an alternative to
running the CLI as a separate WebSocket service. This PR exposes that embedded
core as an opt-in transport (mode: "native") for operators who'd rather
not run a sidecar, while leaving the existing external WebSocket-runtime mode
completely unchanged and fully supported.

The win for that use case: a bot can run with no sidecar — no
separately-managed simplex-chat process, no Docker/systemd, no wsUrl. Set
mode: "native" and the plugin runs the core itself. For anyone happy with the
service/sidecar setup, nothing changes — this only adds a second option behind a
flag.

What's included

  • Native transport (mode: "native") — embedded simplex-chat core; speaks
    the same command/event protocol as the WS runtime, so everything above the
    transport layer is unchanged.
  • Clean transport seam — extracted a SimplexTransport interface;
    SimplexWsClient (external) and new SimplexCoreClient (native) both
    implement it; SimplexClient selects by mode. External mode is
    byte-for-byte unchanged.
  • AGPL isolationsimplex-chat is not a dependency of the plugin. It
    is installed manually, and loaded lazily at runtime via dynamic import.
    Default and external-mode installs stay pure-JS/MIT and never fetch it.
  • Additive configconnection.mode gains "native", plus db
    (filePrefix/encryptionKey), profile
    (displayName/fullName/image/peerType), addressSettings
    (autoAccept/welcomeMessage/businessAddress), servers (smp/xftp),
    and filesFolder (external mode — see Inbound file fixes below). Backward
    compatible; manifest regenerated.
  • Custom SMP/XFTP servers (native)connection.servers.smp/.xftp route
    the embedded core over specific servers instead of the presets (applied via
    the core's /smp//xftp commands, space-separated). A rejected configuration
    fails startup rather than silently falling back to the defaults, since
    selecting custom servers means opting out of the presets. Verified end-to-end
    against a self-hosted SMP server.
  • Zero-config DB path — defaults to
    <OPENCLAW_STATE_DIR|~/.openclaw>/simplex/<accountId>, per-account.
  • Bot profile management — account-level display name / avatar / peer type,
    reconciled on connect; avatar auto-downscaled to a small JPEG (with the
    image/jpg media type SimpleX requires); address auto-created with welcome
    message + auto-accept, also reconciled on change.
  • Safety guards — a single embedded core per account (refuses to open a
    second connection to the same SQLite, e.g. from a CLI/probe), and a startup
    check rejecting two native accounts that share a db.filePrefix.
  • Native-aware CLI — the openclaw simplex … commands (invite, address,
    requests, groups, files, connect, runtime) transparently dispatch to the
    running gateway in native mode (so they reach the in-process core) with the
    same terminal QR/output; external behavior untouched.
  • Operational visibility — startup logs the resolved profile and the bot's
    connection link.
  • Inbound file fixes (transport-agnostic) — fixes a file-receive race and
    stages received media into OpenClaw's media/inbound store. Received files are
    located correctly across deployments: absolute paths (no --files-folder;
    native core) are read directly, while a runtime started with --files-folder
    reports names only and they are resolved against connection.filesFolder
    (default ~/.simplex/files) — which also bridges a shared volume mounted at
    different paths across separate sidecar/OpenClaw containers. A startup
    diagnostic logs each reported path (absolute vs. relative).
  • Tests & docs — unit tests for the transport guard, core
    lifecycle/profile/address reconcile, account defaults and duplicate-prefix
    guard; a native-setup guide covering config, profile, address settings, and
    a DB-durability/reset runbook.

Compatibility & licensing

  • Backward compatible: existing mode: "external" configs are unaffected;
    the native fields are additive.
  • Licensing: simplex-chat is AGPL-3.0 while this plugin is MIT. It is
    not declared as a dependency, so default and external-mode installs never
    fetch it; enabling native mode is an explicit operator opt-in
    (npm install simplex-chat in the plugin's project). Whether to
    document/endorse that opt-in AGPL path is ultimately a maintainer decision.

Testing

Validated end-to-end on macOS and a StartOS (containerized) deployment:

  • Native mode: connect, send/receive, avatar, welcome message, auto-accept +
    manual contact-request approval, and business-address (group) connections.
  • Custom SMP/XFTP servers: applied to the embedded core against a
    self-hosted SMP server and verified inbound + outbound; confirmed a malformed
    server URI fails startup (no silent fallback to presets).
  • Inbound files: verified across deployments —
    • external sidecar without --files-folder → absolute path (Downloads on
      macOS, /tmp in containers), read directly;
    • external sidecar with --files-folder → name-only path resolved against
      connection.filesFolder, including a shared Docker volume mounted at
      different paths in the sidecar and OpenClaw containers;
    • native on StartOS → absolute /tmp path, read directly.
  • tsc, biome, and vitest all green.

Co-Authored-By: Claude Opus 4.8

lundog added 2 commits June 24, 2026 00:57
SimpleX deprecated its WebRTC TypeScript client in favor of the native
`simplex-chat` Node.js library, which embeds the SimpleX core in-process.
This adds it as an opt-in transport alongside the existing external
WebSocket-runtime mode, so a bot can run with no sidecar — no separate
simplex-chat process, Docker/systemd, or wsUrl. Set mode: "native".

External mode is unchanged: a SimplexTransport interface is extracted from
the existing WS client, SimplexWsClient implements it, and SimplexClient
selects the implementation by account mode.

Transport & lifecycle:
- SimplexCoreClient drives the embedded core (init -> ensure user ->
  startChat -> address); it speaks the same string-command / JSON-event
  protocol as the WS runtime, so everything above the transport is unchanged.
- One embedded core per account: withSimplexClient refuses to open a second
  connection to the same database, and startup rejects two native accounts
  that resolve to the same db.filePrefix.
- Diagnostics/security are mode-aware (no WS-endpoint checks in native).

Config (additive, backward compatible):
- connection.mode gains "native"; adds db (filePrefix/encryptionKey),
  profile (displayName/fullName/image/peerType), and addressSettings
  (autoAccept/welcomeMessage/businessAddress). Manifest regenerated.
- db.filePrefix defaults to
  <OPENCLAW_STATE_DIR|~/.openclaw>/simplex/<accountId>, per account.

Profile & address:
- Account-level profile resolved and reconciled on connect; the avatar is
  downscaled to a small JPEG (using the image/jpg media type SimpleX
  renders). The address is auto-created with welcome message + auto-accept
  and reconciled only when settings change.

CLI:
- The openclaw simplex ... commands (invite, address, requests, groups,
  files, connect, runtime) dispatch to the running gateway in native mode
  so they reach the in-process core, with identical terminal output;
  external behavior is untouched.

Licensing:
- simplex-chat is AGPL-3.0 and is NOT a dependency of this plugin. Default
  and external-mode installs never fetch it. Native mode is an explicit
  operator opt-in (npm install simplex-chat in the plugin's project),
  loaded lazily via dynamic import.

Adds unit tests (transport guard, core lifecycle, profile/address reconcile,
account defaults and duplicate-prefix guard) and a native-setup guide
covering config, profile, address settings, and a database
durability/reset runbook.
@lundog

lundog commented Jun 29, 2026

Copy link
Copy Markdown
Author

Hi @dangoldbj, I bumped ws from 8.20.0 to 8.21.0 to fix the vulnerability scan. Thanks for taking a look at this!

lundog added 2 commits June 29, 2026 13:42
Lets a native-mode account route over specific SMP/XFTP servers instead of
the built-in presets — the last open item called out in the PR description.
Verified end-to-end against a self-hosted SMP server (inbound and outbound).

Config (additive, backward compatible):
- connection.servers gains smp[] and xftp[], each a full server URI
  (smp://<fingerprint>@host / xftp://<fingerprint>@host). When set, the list
  replaces the corresponding preset servers. Manifest regenerated.

Behavior:
- Applied after startChat via the core's /smp and /xftp commands (multiple
  servers space-separated). External mode is untouched.
- Fail-fast: a rejected configuration throws and aborts startup rather than
  silently falling back to the defaults — selecting custom servers means
  opting out of the presets (self-hosting, privacy, compliance), so the error
  is surfaced immediately. The startup error names the offending list
  (SMP/XFTP) and includes the core's structured reason.

Adds unit tests (commands sent space-joined after startChat, skipped when
unset, startup fails on rejection) and a native-setup guide section covering
config, the replace-presets/fail-fast semantics, and a reachability note
(private-IP servers are only usable by LAN clients that configure them, since
private message routing otherwise proxies through a public server that cannot
reach a private host).
When the external runtime is started with --files-folder, it reports each
received file by name only (not a full path), expecting the client to know its
own files-folder. The plugin was joining that bare name to a hardcoded /tmp, so
media staging failed with ENOENT whenever --files-folder was set, and the file
never reached OpenClaw's media/inbound store. (Without --files-folder the
runtime reports absolute paths — Downloads on desktop, /tmp in headless
containers — which already worked and are unaffected.)

Resolve relative inbound paths against a new connection.filesFolder instead,
defaulting to ~/.simplex/files to match the default the bundled runtime service
launches simplex-chat with. Absolute paths bypass resolution entirely, so native
mode (whose embedded core always reports absolute paths) is unaffected.

This also makes the split-container Docker topology work: the sidecar and
OpenClaw mount one shared volume at different paths, and the runtime's name-only
reporting lets connection.filesFolder map the sidecar's write-path to OpenClaw's
read-path. An absolute path could not, since the sidecar's directory does not
exist inside the OpenClaw container.

- connection.filesFolder config (schema/types/manifest), with ~ expansion.
- resolveSimplexInboundDir() extracted and unit-tested (default, custom folder,
  ~ expansion).
- Diagnostic log of the path exactly as the runtime reported it (absolute vs.
  relative) for observability.
- Docs: connection.filesFolder in config reference, and a "Locating received
  files" section in the runtime-setup guide covering the shared-volume case.

Builds on the media/inbound staging added earlier in this branch; absolute-path
deployments see no behavior change.

Co-Authored-By: Claude Opus 4.8
@lundog

lundog commented Jun 29, 2026

Copy link
Copy Markdown
Author

Quick correction on my earlier framing: I'd implied mode: "external" was on a deprecated path — that was my misreading of the SimpleX announcement. What's deprecated is only the standalone @simplex-chat/webrtc-client TypeScript WS client library, not the simplex-chat CLI's -p WebSocket-server mode that external mode uses (and which this plugin talks to via its own ws client). I've reworded the PR description accordingly. Native here is purely additive/opt-in — it doesn't touch the external transport or the service architecture you've built.

Also, to keep this respectful of how you've scoped the project: the inbound-file fixes in this PR — the file-receive race fix, staging received media into media/inbound, and resolving --files-folder paths via the new connection.filesFolder — are transport-agnostic and improve external mode on their own. If you'd rather evaluate native mode separately (or not at all), I'm happy to split those fixes into their own PR so they can land independently.

@lundog

lundog commented Jul 1, 2026

Copy link
Copy Markdown
Author

Closing to split.

@lundog lundog closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant