diff --git a/.gitignore b/.gitignore index 357bddc..25b6a11 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,11 @@ Thumbs.db # Environment .env .env.* + +# Agent skills / AI tooling — installed locally, never committed. +.agents/ +.claude/ +skills-lock.json + +# Engine runtime state (default state_dir from engine.toml). +data/ diff --git a/docs/adr/0001-engine-toml-separate-from-nexum-toml.md b/docs/adr/0001-engine-toml-separate-from-nexum-toml.md new file mode 100644 index 0000000..ab6c1f4 --- /dev/null +++ b/docs/adr/0001-engine-toml-separate-from-nexum-toml.md @@ -0,0 +1,36 @@ +--- +status: proposed +implemented-in: nullislabs/shepherd#8, nullislabs/shepherd#9 +--- + +# Operator config (`engine.toml`) is separate from module manifest (`module.toml`) + +## Context + +The engine needs two distinct kinds of configuration: what the **operator** decides at deployment time (which chains to connect to, where the local-store database lives, which modules to boot) and what the **module developer** declares at build time (required and optional capabilities, HTTP allowlist, module-specific config keys). These have different reviewers, different threat models, and change on different cadences. + +The filenames need to signal who owns each file directly. An operator opening a config file should know without prior context whether the file is their concern or the module developer's. A name like `nexum.toml` requires the reader to know that "nexum" refers to the runtime that hosts the module, which is one indirection too many; `module.toml` reads as "the module's manifest" with no prior context. + +## Decision + +Two distinct files, distinct schemas, distinct loaders: + +- **`engine.toml`** — operator-owned, lives next to the engine binary or pointed to by `--engine-config`. Defines `[engine]` (state_dir, log_level), `[chains.]` (rpc_url), and `[[modules]]` (path, manifest). Loaded by `engine_config::EngineConfig::load`. +- **`module.toml`** — module-developer-owned, ships in the module's bundle alongside its `.wasm` component. Defines `[module]`, `[capabilities]` (required, optional, http allowlist), `[config]`. Loaded by `manifest::load`. + +The engine config carries the path to each module's manifest; the two never collapse into one file. The names `engine.toml` and `module.toml` map directly onto the two distinct roles, so a reader reaching either file knows whose concerns it covers. + +## Considered options + +- **Single `shepherd.toml` with `[engine]`, `[chains]`, `[[modules]]` *and* nested `[modules..capabilities]` per module.** Rejected: conflates operator and developer concerns. A module's capability declaration is a property of the build, not the deployment — it belongs in the artifact, not in the operator's local file. Auditing a module's capabilities also becomes a per-deployment exercise instead of a property visible in the published bundle. +- **Keep the `nexum.toml` filename for the module manifest.** Rejected: the name does not signal who owns the file (engine vs module). `module.toml` reads as "the module's manifest" without prior context. +- **`module.toml` inside the engine config (module entries embed it inline).** Rejected for the same reason as the single-file proposal; also bloats `engine.toml`. +- **Drop `engine.toml` entirely; pass everything as CLI flags or env vars.** Rejected: per-chain RPC URLs and module lists are awkward as flags, and `RUST_LOG` already covers the only thing that env vars naturally express. + +## Consequences + +- A deployment needs both files. A missing `engine.toml` falls back to "no chains, default state_dir" — the example logging module still runs; cow-api / chain backends report `unsupported`. +- A missing `module.toml` triggers the 0.1-compat deprecation warning in `manifest::fallback_manifest()` (defined in `crates/nexum-engine/src/manifest.rs`) and treats every linked capability as required. This fallback is scheduled for removal in 0.3 per `docs/migration/0.1-to-0.2.md`. +- Module-bundle redistribution carries `module.toml` with the artifact; engines do not need to ship templates. +- Future content-addressed module distribution (0.3) embeds `module.toml` in the bundle hash; `engine.toml` references modules by content address rather than filesystem path. The split survives that migration unchanged. +- Implementation impact: `crates/nexum-engine/src/manifest.rs` and `engine_config.rs` need to update the filename lookup from `nexum.toml` to `module.toml`. The 0.1-compat fallback in `manifest::fallback_manifest()` should accept both names during the transition; after 0.3 only `module.toml` is recognised. diff --git a/docs/adr/0002-provider-pool-transport-by-scheme.md b/docs/adr/0002-provider-pool-transport-by-scheme.md new file mode 100644 index 0000000..c8e8fab --- /dev/null +++ b/docs/adr/0002-provider-pool-transport-by-scheme.md @@ -0,0 +1,39 @@ +--- +status: proposed +implemented-in: nullislabs/shepherd#8, nullislabs/shepherd#9 +--- + +# Per-chain alloy provider transport selected by URL scheme + +## Context + +`nexum:host/chain` covers both generic JSON-RPC dispatch (`request`) and event subscriptions (`subscribe-blocks`, `subscribe-logs`). Subscriptions require a duplex transport (`eth_subscribe` is push-only over a long-lived connection); request/response works on either HTTP or WebSocket. The operator configures one RPC endpoint per chain in `engine.toml`; the engine has to decide which alloy transport to use. + +## Decision + +The `ProviderPool::from_config` constructor reads each chain's `rpc_url` and switches by URL scheme prefix: + +- `ws://` or `wss://` → `ProviderBuilder::new().connect_ws(WsConnect::new(url))`. Pubsub transport. Subscriptions and request/response both work. **This is the recommended configuration for any chain a module subscribes to.** +- `http://` or `https://` → `ProviderBuilder::new().connect_http(parsed)`. HTTP transport. Request/response only; `subscribe-blocks` and `subscribe-logs` surface as `host-error.unsupported` to the guest. + +Both transports erase to `DynProvider` so the rest of the engine is transport-agnostic. + +Alloy is capable of emulating `eth_subscribe` on HTTP via polling, but this is intentionally **not** enabled. The engine takes an opinionated stance favouring WebSockets for subscriptions; operators who want push-based events configure WSS endpoints. HTTP-only chains are supported for `request` traffic but not for subscriptions. + +## Non-goals + +- **RPC failover, load balancing, and retry policies are explicitly out of scope for the engine.** This logic lives in upstream crates (alloy ships tower-style middleware for timeout / retry / rate-limit / fallback endpoint). The engine does not roll its own. Operators wanting failover configure it via alloy provider builders before passing them through, or rely on the provider's own fallback (Alchemy, Infura, etc. handle it server-side). +- Re-routing requests across chains, rebalancing across pools within a chain, and similar provider-management concerns are likewise alloy's responsibility. + +## Considered options + +- **Force WSS everywhere.** Rejected: many providers (Alchemy, Infura, self-hosted RPC) expose HTTP-only on free tiers, and modules that only need `request` (no subscriptions) shouldn't be blocked by a WSS requirement. +- **Explicit `transport = "ws" | "http"` field per chain in `engine.toml`.** Rejected for 0.2: redundant with the URL scheme, and operators already distinguish `wss://` from `https://` endpoints when copying them from their RPC provider's dashboard. Revisit if we add IPC (`/path/to/geth.ipc`) — scheme alone won't carry that. +- **Open both an HTTP and a WSS connection per chain.** Rejected: doubles connection count for the common case where one endpoint serves both, and forces operators to provide two URLs even when their provider returns identical data on both. + +## Consequences + +- Operators that need subscriptions must supply WSS URLs; HTTP-only chains downgrade to request-only mode at the host call boundary. +- Connection failures at boot are fatal (the engine refuses to start with a broken chain). This is intentional — silent fall-back to a half-functioning state masks misconfiguration that a module then rediscovers at first event. +- Adding IPC support is additive: extend the scheme match with `/` / `file://` and call `connect_ipc`. +- The `DynProvider` erasure costs a virtual dispatch per call — a measurable concern at scale, deferred to M4 if profiling shows it. diff --git a/docs/adr/0003-local-store-namespacing.md b/docs/adr/0003-local-store-namespacing.md new file mode 100644 index 0000000..47d5378 --- /dev/null +++ b/docs/adr/0003-local-store-namespacing.md @@ -0,0 +1,49 @@ +--- +status: proposed +implemented-in: nullislabs/shepherd#8 +--- + +# Per-module namespacing in `local-store` via 32-byte deterministic hash prefix + +## Context + +`nexum:host/local-store` is a key-value store shared across all modules the engine runs. Two modules using the same key string (e.g. `"last-block"`) must see disjoint values; one module must never read or overwrite another's data. The engine knows each module's identity at instantiation time, so namespacing is a host-side concern. + +Two properties matter for the namespace prefix: + +1. **Deterministic and unspoofable.** An arbitrary `module_name` string read out of `module.toml` lets a malicious or careless operator give two modules the same name and have one read the other's state. A fixed-size hash derived from the module's canonical identity is harder to collide and removes the operator-supplied-text attack surface. +2. **Composes with ENS-based module discovery** (per `docs/03-module-discovery.md`): when a module is identified by an ENS name (e.g. `twap-monitor.shepherd.eth`), the ENS namehash is a natural prefix. ENS TXT records pinning the `.wasm` content hash provide a separate verification path against the loaded bundle. + +## Decision + +Single redb database file at `EngineConfig.engine.state_dir`, single shared table `nexum:local-store`. Every key handed to redb is composed host-side as: + +``` +[32-byte namespace prefix][raw key bytes] +``` + +The 32-byte prefix is computed deterministically from the module's canonical identity: + +- **ENS-identified modules** (M3+, per `docs/03`): prefix is `ens_namehash(name)` (EIP-137), e.g. `namehash("twap-monitor.shepherd.eth")`. +- **Locally-loaded modules** (current 0.2 scope, no ENS): prefix is `keccak256(module_name)` where `module_name` comes from `module.toml`'s `[module].name` field. + +Both produce a 32-byte digest with the same domain, so a module loaded locally during development and later published under an ENS name can keep its existing state by registering an alias (`alias = keccak256(name)`) the engine recognises during the migration window. The exact alias mechanism is out of scope for this ADR. + +Modules see plain key strings on both the read and write paths; the prefix is invisible to the WIT-facing API. + +## Considered options + +- **Separator string** (`{module}:{key}`). Rejected: any module name containing `:` collides with another module's `:`-bearing key. A fixed-size hash is unambiguous regardless of payload bytes. +- **`[len:u8][module_name][key]` length-prefixed string.** Rejected: spoofable (the name is operator-supplied text), and does not align with the ENS-based discovery path that 0.3 will introduce. The 32-byte hash is deterministic and namespace-uniform. +- **One redb database file per module.** Rejected: multiplies open file handles linearly in modules, blocks any future cross-module atomic operations (not currently planned but cheap to keep on the table), and complicates backup tooling (N files vs 1). +- **One redb *table* per module within a single file.** Rejected: redb `TableDefinition` lifetimes are `'static`, so table names must be known at compile time. Dynamic table opening per module would force string-leak workarounds and exposes the same name-collision question as separator-based keys. +- **Engine-allocated incrementing module id.** Rejected: stable across reboots only if the engine persists the allocation table, which adds a chicken-and-egg dependency on the local-store itself. Determinism from the name avoids the dependency entirely. + +## Consequences + +- The prefix is fixed-size (32 bytes) and independent of module name length. Range scans over a single module's keys are O(log n + module-key-count) — fine for our workload. +- Migrations changing the prefix derivation (e.g., switching the local-mode hash function or the ENS resolver) would orphan every existing module's persisted state. The derivation must stay stable through 0.x; ENS-mode introduction in 0.3 happens additively via the alias mechanism, not by changing existing prefixes. +- A module's `list-keys` iterates over the namespace range (32-byte prefix scan); the host strips the prefix before returning to the guest. +- Module data versioning (schema migrations across module versions) is the module's responsibility. The local-store does not version values; modules MAY embed a `schema_version` byte in their stored payloads and migrate on `init` when the read value's version differs from the current code's expectation. +- ENS-based discovery (per docs/03) integrates without a prefix-format change: when a module is loaded by ENS name, the prefix is `namehash(name)`. The corresponding `.wasm` content hash is verified via ENS TXT records before loading, separately from the local-store prefix derivation. +- Spoofing protection: an operator cannot make module A read module B's state by renaming, because the prefix is the hash of the canonical name. Renaming a module to match another's name produces a name conflict the engine refuses at boot, rather than silent state takeover. diff --git a/docs/adr/0004-patch-cowprotocol-to-bleu-cow-rs.md b/docs/adr/0004-patch-cowprotocol-to-bleu-cow-rs.md new file mode 100644 index 0000000..c8ba48b --- /dev/null +++ b/docs/adr/0004-patch-cowprotocol-to-bleu-cow-rs.md @@ -0,0 +1,38 @@ +--- +status: proposed +implemented-in: nullislabs/shepherd#10 +--- + +# Patch `cowprotocol` crate to the head of upstream PR #5 + +## Context + +`cowprotocol` v1.0.0-alpha.3 (the version on crates.io) was cut from an early snapshot of `cowdao-grants/cow-rs` PR #5 at commit `1742ffa`. That PR is still open and is the canonical upstream channel for landing additions to the Rust SDK. Its head branch is `bleu/cow-rs:main`, currently at commit `c012404`, carrying 18 follow-up commits the engine materially depends on: + +- `composable::Proof` byte-width fix (consumed by the TWAP poll path). +- `OrderCreation` zero-`from` fast-fail (closes a MEDIUM severity finding in PR #5). +- `order_book` / `composable` submodule splits (cleaner imports on the engine side). + +ADR-0007 commits us to landing three protocol-level primitives into PR #5 directly (`OrderPostError` rich variants + `retry_hint`, `OrderBookApi::with_base_url`, and `wasm32` feature-gating) by pushing additional commits to its head branch. Each commit advances both PR #5 and the patch rev consumed here. + +There is no published `alpha.4` and no scheduled date for one; the engine cannot wait. + +## Decision + +Add a workspace-level `[patch.crates-io]` redirecting `cowprotocol` to `https://github.com/bleu/cow-rs` at commit `c012404`. Every crate that declares `cowprotocol = "1.0.0-alpha.3"` (engine, modules, future SDK) silently picks up the patched build with no `Cargo.toml` change at the dependent site. + +This is not a parallel fork. `bleu/cow-rs:main` IS the head branch of upstream PR #5. Pushing to it updates PR #5; the patch rev advances by bumping a single workspace line. + +## Considered options + +- **Vendor the missing types locally.** Rejected: re-implementing `composable::Proof`, `OrderCreation`, etc. in the engine repo is the AI-duplication anti-pattern that the cow-rs SDK already solves. Reuse over reimplement applies. +- **Pin every dependent to `cow-rs` git directly.** Works but every new workspace member has to remember the git source. `[patch.crates-io]` centralises the override. +- **Open a separate PR per primitive against `cowdao-grants/cow-rs`.** Rejected: fragments the change across multiple PRs when one already exists at the appropriate granularity. Stacking commits on PR #5 keeps the change coherent and lets the cumulative diff be tracked in one place. +- **Wait for `alpha.4` to publish.** No ETA; the TWAP/EthFlow milestone cannot land without `composable::Proof` correct. + +## Consequences + +- `cargo update` will re-resolve to the same `rev`; the lock pins it. +- Bumping the rev is a single-line workspace edit; reviewers see one diff per primitive added to PR #5. +- Drop the patch entirely once a published `cowprotocol` release contains both the alpha.3 follow-ups and the ADR-0007 protocol-primitive additions (`OrderPostError` rich variants + `retry_hint`, `OrderBookApi::with_base_url`, `wasm32` feature-gate). Until then, expect the patch rev to advance with every push to PR #5. +- Modules built against this workspace inherit the patch transitively; modules built standalone against crates.io will see `alpha.3` and may hit the very bugs the patch closes. Flag this in the SDK README when M3 lands. diff --git a/docs/adr/0005-cow-api-via-cached-orderbookapi.md b/docs/adr/0005-cow-api-via-cached-orderbookapi.md new file mode 100644 index 0000000..0862d54 --- /dev/null +++ b/docs/adr/0005-cow-api-via-cached-orderbookapi.md @@ -0,0 +1,34 @@ +--- +status: proposed +implemented-in: nullislabs/shepherd#8 +--- + +# `cow-api` host backend routes both `request` and `submit-order` through `cowprotocol::OrderBookApi` + +## Context + +`shepherd:cow/cow-api` exposes two operations: a generic REST passthrough (`request`) and a typed order submission (`submit-order`). Either could be implemented with raw `reqwest` against `api.cow.fi/{slug}/api/v1`, but the published `cowprotocol` crate already ships an `OrderBookApi` client that knows the chain-specific base URL, the canonical paths, and the `post_order` codec. + +## Decision + +At engine boot, construct one `cowprotocol::OrderBookApi` per `cowprotocol::Chain` variant (currently Mainnet, Gnosis, Sepolia, ArbitrumOne, Base) into a `BTreeMap` keyed by EVM chain id. "Cached" here means built once during boot and reused for the engine's lifetime; clients are not lazy-constructed on each call nor LRU-evicted. The pool implements `Default` so callers instantiate it as `OrderBookPool::default()`; the trait impl populates the map with one entry per `cowprotocol::Chain` variant. + +Both `cow-api` operations consult this pool: + +- `request` resolves the chain's `OrderBookApi`, reads `api.base_url()` for the prefix, joins the module-supplied path, and dispatches via a shared `reqwest::Client`. +- `submit-order` deserialises the JSON `OrderCreation` and calls `OrderBookApi::post_order` directly. The crate handles signing-scheme encoding, error mapping, and `OrderUid` extraction. + +Chains not in `cowprotocol::Chain` return `HostError { kind: unsupported }` at the host call boundary. + +## Considered options + +- **Raw `reqwest` for both.** Rejected: forces us to maintain the chain → base-URL table (drifts whenever cowprotocol adds a chain) and reimplement `post_order`'s body codec and error mapping, the exact duplication the cow-rs SDK already eliminates. +- **`OrderBookApi` for `submit-order`, raw `reqwest` for `request`.** Tempting (request is opaque to the crate) but means two separate chain-resolution paths, two HTTP clients, and a second place to keep the chain set in sync. +- **Build `OrderBookApi` lazily on first call per chain.** Rejected: hides config errors at runtime. Up-front boot construction surfaces unknown chains immediately and amortises away the per-call cost. + +## Consequences + +- Operator-supplied custom orderbook URLs (barn, staging, forked deployments) are out of scope for the default constructor and require a follow-on `OrderBookApi::with_base_url(chain_id, base_url)` constructor in the cow-rs crate (ADR-0007 item 2, not vendored locally). +- Adding a chain means a `cowprotocol::Chain` variant lands in cow-rs first; the engine inherits it on the next patched rev bump. +- The shared `reqwest::Client` enables connection pooling across both `request` and `submit-order` paths. +- Guest-side TWAP and EthFlow modules (ADR-0006) submit orders through this `cow-api` interface; no specialised host helpers wrap it. diff --git a/docs/adr/0006-cow-twap-ethflow-host-helpers.md b/docs/adr/0006-cow-twap-ethflow-host-helpers.md new file mode 100644 index 0000000..0e9e776 --- /dev/null +++ b/docs/adr/0006-cow-twap-ethflow-host-helpers.md @@ -0,0 +1,49 @@ +--- +status: proposed +--- + +# TWAP and EthFlow run as guest modules using low-level host primitives (no specialised `shepherd:cow` interfaces) + +## Context + +TWAP (over ComposableCoW) and EthFlow are the two CoW workflows the M2 grant ships modules for. The natural-seeming approach is to add `shepherd:cow/twap` and `shepherd:cow/ethflow` WIT interfaces that the host implements on top of `cowprotocol` crate primitives, so modules would call `twap.poll-and-submit(...)` and `ethflow.submit-from-log(...)` as host functions. This ADR rejects that direction. + +The dividing line is protocol vs implementation. CoW Protocol primitives — order types, signing schemes, the orderbook REST surface — are protocol concerns and belong in shared layers (`cowprotocol` crate, `shepherd:cow/cow-api` interface). TWAP is one of many strategies built _on top of_ those primitives; ComposableCoW is the contract surface a TWAP module observes, but the act of polling, deciding when to submit, and reacting to orderbook errors is application logic. Putting that application logic in the host or in `cowprotocol` couples every consumer to one implementation and one error-handling policy. + +Embedding a concrete TWAP implementation in an SDK is an architectural smell the grant explicitly seeks to alleviate. The grant seeks to enable Shepherd as the runtime where many independent strategy implementations coexist, each compiled to its own WASM module. A specialised `twap` interface in the host would defeat that goal: every Shepherd deployment would have to use the same polling implementation, the same error-mapping, the same retry hints, with no room for different strategies to differ on those choices. + +## Decision + +The `shepherd:cow` WIT package contains only the existing `cow-api` interface (REST passthrough + `submit-order`), which is protocol-level. No `twap` interface, no `ethflow` interface, no host-side helpers specific to either workflow. + +TWAP and EthFlow modules implement their logic in Rust guest code using: + +- **`nexum:host/chain`** — `request` (for `eth_call`, `eth_getLogs`, etc.), `subscribe-blocks`, `subscribe-logs`. +- **`nexum:host/local-store`** — for watch lists, cursors, and backoff state. +- **`nexum:host/logging`** — for structured logs. +- **`shepherd:cow/cow-api`** — `submit-order` for orderbook submission. +- **`cowprotocol` crate** (consumed directly by the module, gated on the wasm32 feature work in ADR-0007) — for protocol types: `Order`, `OrderCreation`, `OrderUid`, signing schemes, `OrderPostError`, etc. +- **`alloy_sol_types`** (or equivalent) — for ABI-aware decoding of `ConditionalOrderCreated`, `OrderPlacement`, `getTradeableOrderWithSignature` return values, and similar Solidity-typed payloads. + +Concretely, a TWAP module's `on_event(block)` handler iterates the local-store watch set, makes an `eth_call` to `ComposableCoW.getTradeableOrderWithSignature(owner, params, "", [])` via `chain.request`, decodes the return (or revert reason) with `alloy_sol_types`, constructs an `OrderCreation` with `cowprotocol` types, and submits via `cow-api/submit-order`. Orderbook errors are interpreted via `OrderPostError::retry_hint()` (ADR-0007). Backoff state is persisted to `local-store`. All of this lives in module Rust source, not in the engine. + +An EthFlow module's `on_event(log)` handler decodes the `OrderPlacement` event with `alloy_sol_types`, constructs the `OrderCreation` (with the EIP-1271 signing scheme pointing at the `CoWSwapEthFlow` contract), and submits the same way. Module-side, no host helper required. + +## Considered options + +- **Specialised `shepherd:cow/twap` and `shepherd:cow/ethflow` interfaces** with rich `PollOutcome` variants and per-event host helpers, backed by `composable::poll_and_build_order` and `eth_flow::decode_placement` primitives in the `cowprotocol` crate. Rejected: this puts a single concrete TWAP / EthFlow implementation behind a WIT boundary, forcing every Shepherd deployment to use the same polling policy, the same error-mapping, the same retry hints. It also blurs the protocol-vs-implementation boundary the grant is meant to clarify. Multiple TWAP implementations (different polling cadences, different error tolerances, different cancel-on-loss thresholds) must be able to coexist as separate modules without changing the host or the SDK. +- **Move TWAP / EthFlow primitives into `cowprotocol` crate but skip the WIT interfaces**, leaving modules to call `composable::poll_and_build_order` from guest code. Rejected for the same reason: `cowprotocol` is the protocol SDK, not the strategy SDK. Putting TWAP logic there embeds an implementation in the shared layer, which is the smell the grant seeks to fix. +- **Ship a thin `shepherd-sdk` helper crate** that wraps the low-level primitive calls (eth_call, decode, submit) into a convenient `Twap::poll(...)` interface for guest modules. **Acceptable for M3** because the helper would live in guest-callable code, not behind a WIT boundary — a module that wants different polling policy just doesn't use the SDK helper. The host stays neutral. +- **EthFlow as pure passive observer (no submission)**. Rejected on closer read of `cowprotocol/services/crates/autopilot/src/database/onchain_order_events/ethflow_events.rs`: the canonical CoW flow expects the event to be relayed into the orderbook, which is what autopilot currently does internally. Shepherd's `ethflow-watcher` externalises that role, so the module does submit; just from guest code, not via a specialised host interface. +- **TWAP merkle-proof / `setRoot` support in v1.** Deferred. The 0.2 module only handles `ComposableCoW.create()` (empty proof, single conditional order). `setRoot` polling requires off-chain proof derivation; when a real module needs it, it will be implemented in guest code using the same low-level primitives, possibly with an SDK helper to encapsulate the proof bookkeeping. + +## Consequences + +- `shepherd:cow@0.2.0` keeps `cow-api` as its only interface. No new WIT files in this ADR. +- `KNOWN_CAPABILITIES` in `crates/nexum-engine/src/manifest.rs` does **not** gain `"twap"` or `"ethflow"` entries. Modules declare the universal capabilities they actually use: `chain`, `local-store`, `logging`, `cow-api`. +- Modules ship larger (~150 LOC each estimated, up from the ~30 LOC the host-helper design implied), because event decoding, eth_call orchestration, OrderCreation construction, and error-hint interpretation now live in guest code. This is the explicit trade-off: more code per module, less coupling, more freedom for different strategies to coexist. +- Different TWAP polling strategies can coexist as different modules. Operators choose which to load via `engine.toml`'s `[[modules]]` array. +- The watch-tower TypeScript implementation remains the closest reference for what a TWAP module's logic looks like, but it is reference material, not a template the Rust module mirrors verbatim. A newer ComposableCoW iteration in development may simplify the polling surface significantly; the relevant decisions live in the module, not the host. +- `OrderPostError` rich variants + `retry_hint()` (ADR-0007 item 1, formerly item 3) become the primary protocol-level contract between the orderbook and any module submitting orders. Modules `match` on the typed error and apply the `RetryHint` (try-next-block / backoff-seconds / drop). This logic is generic across TWAP, EthFlow, stop-loss, and any future strategy. +- The M3 SDK (`shepherd-sdk` crate) is the natural home for ergonomic guest-side helpers: `WatchSet`, `PollLoop`, `BackoffLedger`, decode-and-submit utilities. The SDK is opt-in for module authors and lives entirely on the guest side; the host remains protocol-neutral. +- The architecture and sequence diagrams in `docs/diagrams/` that depict `twap.poll-and-submit` and `ethflow.submit-from-log` host calls reflect the rejected design and must be updated to show modules calling low-level primitives directly. diff --git a/docs/adr/0007-upstream-protocol-logic-to-cow-rs.md b/docs/adr/0007-upstream-protocol-logic-to-cow-rs.md new file mode 100644 index 0000000..e8e949c --- /dev/null +++ b/docs/adr/0007-upstream-protocol-logic-to-cow-rs.md @@ -0,0 +1,46 @@ +--- +status: proposed +--- + +# Push CoW Protocol primitives to `cow-rs` first, adopt in `nexum-engine` second + +## Context + +Implementing ADR-0005 (cow-api backend) and supporting guest-side TWAP / EthFlow modules per ADR-0006 surfaces a recurring question: when the engine or its modules need a piece of CoW Protocol logic that the `cowprotocol` Rust SDK does not yet expose (rich orderbook error variants, custom orderbook URLs, wasm32 compatibility), do we write that logic locally and tidy it up upstream later, or do we add it to the open upstream PR first and only land the engine wiring afterwards? + +The failure mode is well-known: duplicating work that an existing crate could do is the AI-coding anti-pattern most likely to land in a contribution. The same risk applies to any engine-side reimplementation of protocol logic. + +The line between **protocol primitives** (which belong in `cowprotocol`) and **strategy implementations** (which belong in guest modules, per ADR-0006) is the operating principle. This ADR covers only the protocol-primitive additions; TWAP polling and EthFlow event decoding stay in guest modules and are explicitly **not** primitives we push to `cowprotocol`. + +## Decision + +Protocol-level CoW logic — anything that an indexer, a bot, or a non-`nexum` Rust consumer of CoW Protocol would also need to interact with the protocol — lands as additional commits on `cowdao-grants/cow-rs` PR #5 first (head branch `bleu/cow-rs:main`), and is consumed by `nexum-engine` and by guest modules via the `[patch.crates-io]` rev bump (ADR-0004). The engine and the modules never write throwaway local copies of the same logic with the intent to "port later". + +The concrete set of primitives this ADR commits to upstream, in priority order: + +1. **`cowprotocol::OrderPostError` rich variants + `retry_hint(&self) -> RetryHint`** — typed orderbook submission errors (`QuoteNotFound`, `InvalidQuote`, `InsufficientAllowance`, `InsufficientBalance`, `TooManyLimitOrders`, `InvalidAppData`, `AppDataFromMismatch`, `SellAmountOverflow`, `ZeroAmount`, `TransferSimulationFailed`, `ExcessiveValidTo`, …) with a `retry_hint()` helper classifying each into `TryNextBlock`, `BackoffSeconds(u64)`, or `Drop`. Mirrors watch-tower's `API_ERRORS_TRY_NEXT_BLOCK` / `API_ERRORS_BACKOFF` / `API_ERRORS_DROP` tables. Without this, every Rust consumer of CoW reinvents the same mapping, and modules spam the orderbook with permanently-broken orders. **Critical-path, not optional.** + +2. **`cowprotocol::OrderBookApi::with_base_url(chain_id, base_url)`** — custom-URL constructor for barn / staging / forked deployments. Unblocks per-chain orderbook URL overrides in `engine.toml` (ADR-0005). + +3. **`cowprotocol` `wasm32` compatibility** — feature-gate the `reqwest` dependency so guest modules can use the pure types (`Order`, `OrderCreation`, `OrderUid`, signing schemes, error variants) without dragging in an HTTP client. **Critical for ADR-0006**: modules implement TWAP and EthFlow logic in guest code and need `cowprotocol` types compiled to wasm32. Without this, guest modules fall back to duplicating type definitions. + +Lower-priority follow-ons (`OrderUid::from_slice`, retry middleware on `OrderBookApi`, `OrderCreation::from_gpv2`) are good-to-have but are not blocking for the M2 host or module scope. + +## Considered options + +- **Implement locally, refactor upstream later.** Faster short term but predictably leaves an indeterminate amount of duplicated logic in the engine, contradicts the conventions established on cow-rs PR #5, and grows technical debt every time cow-rs evolves the underlying types. Rejected. +- **Push TWAP / ComposableCoW primitives** (`composable::poll_and_build_order`) into `cowprotocol`. Rejected: TWAP is a concrete strategy on top of the protocol, not part of the protocol. Putting it in the SDK forces every consumer to use one polling implementation and one error-mapping policy. Per ADR-0006, TWAP polling lives in guest module code, not in shared layers. +- **Push EthFlow log-decoding primitives** (`eth_flow::decode_placement`) into `cowprotocol`. **Rejected for the same reason**: EthFlow event decoding is an implementation detail of how a particular module relays orders into the orderbook. The protocol layer defines the order types and the orderbook submission endpoint; the act of decoding an on-chain event into an `OrderCreation` is module-side logic. Modules decode `OrderPlacement` directly with `alloy_sol_types` and construct the `OrderCreation` with the EIP-1271 signing scheme. +- **Wait for cow-rs upstream maintainers to add these on their own.** No evidence anyone else is doing this work; the grant timeline does not permit waiting. +- **Vendor a fork of cow-rs inside `nullislabs/shepherd`.** Worst of all worlds: blocks neither the engine nor cow-rs from drifting, and forces every other CoW consumer to re-derive the same primitives. +- **Host-side `AppDataResolver` (LRU cache + GET against `/api/v1/app_data/{hash}`).** Rejected after verifying watch-tower's behavior: it never fetches app-data. The trader uploads the JSON to the orderbook via `PUT /api/v1/app_data/{hash}` separately; the relayer module just submits and reacts to `INVALID_APP_DATA` (backoff 1 min) / `APPDATA_FROM_MISMATCH` (drop) via the error map in item 1 above. + +## Consequences + +- Every M2 engine or module issue that consumes one of the three primitives above is blocked on the corresponding commit landing in PR #5's head branch. Items 1, 2, 3 can be authored as independent commits and pushed in parallel rather than serially. +- `[patch.crates-io]` rev in the workspace `Cargo.toml` (ADR-0004) is bumped after each push to PR #5; the bump is the engine's signal that a new primitive is consumable. +- Commits added to PR #5 follow its established conventions: alloy reuse over local reimplementation, GPL-3.0, edition 2024, terse rustdoc. +- The engine repo stays small: `nexum-engine` contains WIT, host wiring, supervisor, redb store, alloy provider pool, and `engine.toml` schema, with nothing about CoW Protocol semantics. +- Guest modules consume `cowprotocol` types directly (gated on the wasm32 feature in item 3). The `shepherd-sdk` crate in M3 may add ergonomic wrappers on top, but those live on the guest side, not behind a WIT boundary. +- A follow-on Bleu module — the Rust-side equivalent of `cowprotocol/refunder` (permissionless `invalidateOrder` triggering for expired EthFlow orders) — becomes natural to ship once an ethflow-watcher module lands. Out of scope for M2 but explicitly enabled by the same primitives. +- TWAP polling logic (decode `ConditionalOrderCreated`, eth_call `getTradeableOrderWithSignature`, decode return, build `OrderCreation`) and EthFlow event decoding stay entirely in guest module code. The `cowprotocol` crate provides only the types and the orderbook client; the strategy is the module's. diff --git a/docs/adr/0008-factory-subscriptions-in-manifest.md b/docs/adr/0008-factory-subscriptions-in-manifest.md new file mode 100644 index 0000000..5a23356 --- /dev/null +++ b/docs/adr/0008-factory-subscriptions-in-manifest.md @@ -0,0 +1,56 @@ +--- +status: deferred +deferred-to: 0.3 +--- + +# Dynamic address registration for log subscriptions (deferred to 0.3) + +## Status + +**Deferred to 0.3.** Neither TWAP nor EthFlow (the M2 grant deliverables) needs this capability, and the design's complexity is not justified by current need. + +This ADR is preserved as a reference for the design space; the final shape will be revisited when the first module actually requiring dynamic address registration emerges. + +## Context + +Some module archetypes need to track contracts deployed dynamically by a factory, for example Uniswap V3 pools (deployed by `UniswapV3Factory`). Static `[[subscription]]` declarations in `module.toml` cannot express this: the child addresses are not known when the module's manifest is authored. + +Neither TWAP nor EthFlow needs this; both subscribe to a single well-known contract per chain. This ADR was originally framed as forward-looking work to land in 0.2's breaking-change window. + +## Why deferred + +Two considerations motivate the deferral: + +1. **`eth_getLogs` already supports topic-only filtering.** The JSON-RPC method accepts a filter without an `address` field, so a module subscribing to a topic across all addresses can be served by the existing primitives if the operator's RPC endpoint cooperates. If topic-only filters at the JSON-RPC layer are good enough for the common case, the engine does not need a manifest-and-host-function mechanism on top. +2. **The schema and host-function surface add engine complexity that no M2 deliverable consumes.** The historical-backfill story is the largest contributor to that complexity and was already trimmed once; deferring the rest in the same spirit avoids paying for a mechanism nothing exercises yet. + +Combined: the dynamic-subscription design is not load-bearing for M2 deliverables, and the simplest path (topic-only `eth_subscribe` filters with module-side address filtering) may suffice for a wide range of indexer use cases. The dynamic-registration mechanism originally proposed (Envio-style `register-address`) addresses scaling concerns at high address counts but should land when a real consumer is on the table to validate the trade-off. + +## Reference design (not adopted in 0.2) + +The original proposal — kept here so future discussions have a starting point — was a hybrid of static topics and dynamic addresses: + +- `[[subscription.template]]` block in `module.toml` declaring `chain_id`, `name`, `event_topics` (no address). +- `chain.register-address(chain_id, template_name, address)` host function for the module to add addresses at runtime. +- `chain.unregister-address(chain_id, template_name, address)` mirror function. +- `log-source.template(string)` variant on the event dispatch so modules route by template name. +- Engine maintains a single aggregated `eth_subscribe logs` per chain per template, with filter `(topic ∈ event_topics) ∧ (address ∈ current_set)`. The address set is mutated as the module discovers new contracts. +- Historical backfill (`from-block` argument on register, paginated `eth_getLogs` orchestration) was contentious and was already trimmed before deferral. + +Envio HyperIndex's `context..register()` API is the closest existing pattern, validated in production for indexers tracking thousands of dynamically-discovered contracts. + +## Alternatives left open for 0.3 + +- **Topic-only `[[subscription]]`** (no address field; engine forwards `eth_subscribe logs` with topic-only filter; module client-side filters logs by address it cares about). Simplest, no new host functions. Trade-off: firehose volume for common topics like `Transfer`. +- **Dynamic register-address** (the original reference design above). +- **Engine-extracted factory child addresses** (Ponder-style declarative schema with ABI-aware extraction rules). Schema complexity grows with exotic factory shapes. +- **No factory pattern; modules wanting dynamic discovery use raw `chain.subscribe-logs` with topic-only filter and persist the discovered address set themselves**. + +The choice depends on what the first consumer actually needs. + +## Consequences of deferring + +- The `shepherd:cow` and `nexum:host` WIT surfaces remain unchanged in 0.2. +- `module.toml` schema does not gain `[[subscription.template]]` in 0.2. +- 0.2 is the breaking-change window; adding any of the above options in 0.3 may require a major version bump if the chosen shape extends `module.toml` or `nexum:host/chain` non-additively. This risk is accepted on the basis that the M2 grant deliverables do not require this surface. +- TWAP and EthFlow modules ship in 0.2 against the existing static `[[subscription]]` declarations (one address per subscription, known at manifest authorship time). This is consistent with how the autopilot ethflow indexer and watch-tower configure their subscriptions today. diff --git a/docs/diagrams/README.md b/docs/diagrams/README.md new file mode 100644 index 0000000..c9c509e --- /dev/null +++ b/docs/diagrams/README.md @@ -0,0 +1,31 @@ +# Diagrams + +Mermaid sources and rendered PNGs covering the engine architecture, the CoW workflows that the M2 modules implement (TWAP and EthFlow, both as guest modules using low-level host primitives), and the engine internals that new contributors most often need to reason about. + +## Architecture and CoW flows + +| File | Type | Shows | +|---|---|---| +| `architecture.png` / `.mmd` | Component | Static view: external infra, nexum-engine internals, WASM modules (twap-monitor, ethflow-watcher) consuming low-level host primitives, and the `cowprotocol` crate (consumed via `[patch.crates-io]` and the wasm32 feature). The `shepherd:cow` package contains only `cow-api`; no specialised TWAP or EthFlow interfaces. | +| `sequence-ethflow.png` / `.mmd` | Sequence | `OrderPlacement` on-chain event handled entirely in the `ethflow-watcher` guest module: `alloy_sol_types` decodes the event, the module builds an `OrderCreation` with the EIP-1271 signing scheme using `cowprotocol` types, and submits via `cow-api/submit-order`. The orderbook error path runs through `OrderPostError::try_from(host-error).retry_hint()`. | +| `sequence-twap.png` / `.mmd` | Sequence | `ConditionalOrderCreated` registration plus the per-block polling loop driven by the `twap-monitor` guest module: `alloy_sol_types` decodes registrations and `eth_call` returns, the module makes the `getTradeableOrderWithSignature` call via `chain.request`, builds `OrderCreation` via `cowprotocol` types, and submits via `cow-api/submit-order`. Orderbook errors flow through `OrderPostError::retry_hint`. | + +## Engine internals (for contributors) + +| File | Type | Shows | +|---|---|---| +| `module-lifecycle.png` / `.mmd` | State machine | Resolve → Load → Init → Run → Restart → Dead transitions and what triggers each. Documents the exponential-backoff restart policy and the implicit write transaction around `init`. | +| `engine-boot.png` / `.mmd` | Sequence | Boot order: engine.toml → tracing → ProviderPool → LocalStore → OrderBookPool → Supervisor (load each module) → open subscriptions → run event loop. | +| `wit-call-path.png` / `.mmd` | Sequence | One host call traced end-to-end: module Rust source → wit-bindgen stubs → WASM Component → wasmtime Linker → HostState trait impl → ProviderPool → alloy → Chain RPC, and back. Demystifies the WASM/Rust boundary. | +| `subscription-dispatch.png` / `.mmd` | Flow chart | How the supervisor aggregates `[[subscription]]` declarations across modules, opens shared block subscriptions (broadcast) and per-filter log subscriptions (routed), and dispatches events to the right `on_event` handlers. | + +## Regenerate + +```sh +cd docs/diagrams +for f in *.mmd; do + npx -y @mermaid-js/mermaid-cli@latest -i "$f" -o "${f%.mmd}.png" -b white --width 1800 +done +``` + +Mermaid sources are the source of truth; PNGs are committed for offline viewing and PR previews. diff --git a/docs/diagrams/architecture.mmd b/docs/diagrams/architecture.mmd new file mode 100644 index 0000000..f5b97f2 --- /dev/null +++ b/docs/diagrams/architecture.mmd @@ -0,0 +1,69 @@ +graph TB + subgraph external["External Infrastructure"] + OB["CoW Orderbook
api.cow.fi"] + RPC["Chain RPC
ws:// or https://"] + CC["ComposableCoW
(Solidity contract)"] + EF["CoWSwapEthFlow
(Solidity contract)"] + end + + subgraph engine["nexum-engine binary"] + ETOML["engine.toml
(operator config)"] + SUP["Supervisor
(boot + dispatch)"] + EL["Event Loop
futures::select_all"] + + subgraph host["HostState (per module)"] + HW["Host Backends
cow-api, chain,
local-store, logging, ..."] + CP["OrderBookPool
(BTreeMap of OrderBookApi)"] + PP["ProviderPool
(alloy DynProvider per chain)"] + LS["LocalStore
(redb, 32-byte hash prefix per module)"] + end + end + + subgraph modules["WASM Modules (Component Model)"] + MTOML["module.toml
(module manifest)"] + TM["twap-monitor
(decodes events, polls,
builds OrderCreation,
submits via cow-api)"] + EM["ethflow-watcher
(decodes OrderPlacement,
builds OrderCreation,
submits via cow-api)"] + end + + subgraph cowrs["cowprotocol crate (via [patch.crates-io] to PR #5)"] + OBA["OrderBookApi
(orderbook submission)"] + TYPES["Protocol types
(Order, OrderCreation,
OrderUid, signing schemes)"] + OPE["OrderPostError
+ retry_hint -> RetryHint"] + end + + ETOML --> SUP + MTOML -.declares.-> SUP + SUP --> EL + SUP -.loads via wasmtime.-> TM + SUP -.loads via wasmtime.-> EM + + EL -- "on_event(block | log)" --> TM + EL -- "on_event(block | log)" --> EM + + TM -- "WIT host call" --> HW + EM -- "WIT host call" --> HW + + TM -- "consumes types
(wasm32 feature)" --> TYPES + EM -- "consumes types
(wasm32 feature)" --> TYPES + TM -- "matches on" --> OPE + EM -- "matches on" --> OPE + + HW --> CP + HW --> PP + HW --> LS + + CP --> OBA + PP --> RPC + OBA --> OB + CC -. "ConditionalOrderCreated" .-> RPC + EF -. "OrderPlacement / OrderInvalidation" .-> RPC + + classDef external fill:#f0e6ff,stroke:#7e3ff2,color:#000 + classDef engineNode fill:#e6f3ff,stroke:#1e88e5,color:#000 + classDef moduleNode fill:#fff4e6,stroke:#ff9800,color:#000 + classDef cowrsNode fill:#e6ffe6,stroke:#2e7d32,color:#000 + + class OB,RPC,CC,EF external + class ETOML,SUP,EL,HW,CP,PP,LS engineNode + class MTOML,TM,EM moduleNode + class OBA,TYPES,OPE cowrsNode diff --git a/docs/diagrams/diagrams.md b/docs/diagrams/diagrams.md new file mode 100644 index 0000000..762e6db --- /dev/null +++ b/docs/diagrams/diagrams.md @@ -0,0 +1,472 @@ +# Shepherd — Architecture Diagrams + +Visual reference for the Shepherd engine, its interactions with Nexum, CoW Protocol, and the WASM module layer. Derived from ADRs 0001–0008 and the internal architecture document. + +> **Scope note** — diagrams 1–4 and 7–8 reflect the **M1 implemented state** plus the **M2 target design** as described by the ADRs. Diagrams 5–6 (TWAP, EthFlow) describe **guest-module-driven flows**: the modules do all the protocol work themselves using low-level host primitives, with no specialised `twap` or `ethflow` host interfaces. Where the current code differs from the target design, a note is included in the relevant block reference. + +--- + +## 1. System Architecture Overview + +High-level component map: what lives where, how repositories depend on each other. + +```mermaid +graph TD + ET["engine.toml · module.toml\n(operator config + module manifest)"] + SUP["Supervisor::boot"] + POOLS["ProviderPool · OrderBookPool · LocalStore"] + HS["HostState (per module)\nnexum:host@0.2.0 + shepherd:cow@0.2.0"] + EL["EventLoop — futures::stream::select_all\nfan-out block/log streams to subscribers"] + MODS["WASM Modules\ntwap.wasm · eth-flow.wasm\n(self-contained protocol logic in guest)"] + BC["Blockchain (Sepolia / Mainnet / …)\nComposableCoW · CowEthFlow · RPC Node"] + CR["bleu/cow-rs ← [patch.crates-io]\nOrder · OrderCreation · OrderUid · signing schemes\nOrderBookApi · OrderPostError + retry_hint\nOrderBookApi::with_base_url · wasm32 feature"] + OB["api.cow.fi (Orderbook REST)"] + + ET --> SUP + SUP --> POOLS + POOLS --> HS + HS --> EL + BC -->|"block/log events (eth_subscribe)"| EL + EL -->|"on_event(block/log)"| MODS + MODS -->|"WIT calls (chain · local-store · cow-api · …)"| HS + MODS -.->|"consumes types (wasm32 feature)"| CR + HS -->|"eth_call / subscribe"| BC + HS -->|"OrderBookApi.post_order"| OB + HS -->|"cow-api passthrough"| OB +``` + +### Block reference + +| Block | What it is | +|---|---| +| **engine.toml** | Written by the operator. Declares which chains to connect to (RPC URLs), where to store state on disk, and which WASM modules to load at boot. | +| **module.toml** | Written by the module developer and shipped inside the module bundle. Declares which capabilities the module needs (`required`), which on-chain events to subscribe to, and any module-specific config keys. Renamed from `nexum.toml` per ADR-0001 so the operator/module split is directly apparent. | +| **Supervisor::boot** | The boot orchestrator. Reads both config files, creates the shared resource pools, loads each `.wasm` component via wasmtime, and wires their subscriptions into the event streams. | +| **ProviderPool · OrderBookPool · LocalStore** | The three shared backends. `ProviderPool` holds one alloy RPC client per chain. `OrderBookPool` holds one CoW orderbook HTTP client per chain. `LocalStore` is a single redb key-value database shared by all modules (with per-module 32-byte hash namespacing — ADR-0003). | +| **HostState (per module)** | The per-module bridge between WASM guest code and Rust host code. When a module calls a WIT function (`local-store/set`, `cow-api/submit-order`, etc.), wasmtime routes that call to the corresponding method on that module's `HostState`. Checks capability permissions before dispatching. | +| **EventLoop** | The main async loop. Runs all block-header and log-event streams concurrently via `futures::stream::select_all`. When a stream fires, it routes the event to every module that subscribed to it in their `module.toml`. | +| **WASM Modules** | The guest programs. Each module exports `init(config)` (called once at boot) and `on_event(event)` (called on every relevant block or log). They contain the protocol logic themselves: TWAP polling, EthFlow event decoding, OrderCreation construction. They call back into the host through universal WIT interfaces only — no CoW-specific helper interfaces (ADR-0006). | +| **Blockchain** | The EVM chain being watched. Delivers new block headers and contract log events over a persistent WebSocket (`eth_subscribe`). Also handles `eth_call` for on-chain reads (e.g. checking whether a TWAP order is ready). | +| **bleu/cow-rs [patch.crates-io]** | The Rust crate containing CoW Protocol **primitives**: order types, signing schemes, the orderbook HTTP client, and the typed orderbook error model (`OrderPostError` + `retry_hint`). Pulled via `[patch.crates-io]` pointing at the head of upstream PR #5. Modules consume the types directly via the `wasm32` feature; the engine consumes the orderbook client via its `cow-api` host backend. No TWAP or EthFlow strategy logic lives here — that stays in module code (ADR-0007). | +| **api.cow.fi (Orderbook REST)** | The CoW Protocol orderbook service. Accepts `POST /orders` to register new orders. Trader-uploaded app-data documents are PUT to `/app_data/{hash}` separately by whoever signed the order (not by the relayer module). | + +--- + +## 2. Domain / Class Diagram + +Key types, their fields, and relationships across the engine codebase. + +```mermaid +classDiagram + class EngineConfig { + +state_dir: PathBuf + +chains: BTreeMap~u64, ChainConfig~ + +modules: Vec~ModuleEntry~ + } + class Manifest { + +name: String + +capabilities_required: Vec~String~ + +subscriptions: Vec~Subscription~ + } + class Subscription { + +kind: Block | Log + +chain_id: u64 + +address: Option~Address~ + +topics: Vec~B256~ + } + class Supervisor { + +dispatch_block(chain_id, block) + +dispatch_log(owner, log) + } + class ProviderPool { + +providers: BTreeMap~u64, DynProvider~ + } + class OrderBookPool { + +clients: BTreeMap~u64, OrderBookApi~ + } + class LocalStore { + +db: redb~Database~ + +get(key: String) Option~Vec~u8~~ + +set(key: String, value: Vec~u8~) + +delete(key: String) + +list_keys(prefix: String) Vec~String~ + } + class HostState { + +wasi: WasiCtx + +table: ResourceTable + +http_allowlist: Vec~String~ + +monotonic_baseline: Instant + %% M2 additions (ADR-0005): + %% +module_namespace: [u8; 32] (ENS namehash or keccak256) + %% +provider_pool: Arc~ProviderPool~ + %% +ob_pool: Arc~OrderBookPool~ + %% +local_store: Arc~LocalStore~ + } + class EventLoop { + +streams: SelectAll~Pin~Box~dyn Stream~~~ + } + class TwapModule { + <> + +on_event(log) persist watch via local-store + +on_event(block) poll each watch via chain.request(eth_call) + +decode return via alloy_sol_types + +build OrderCreation via cowprotocol types + +submit via cow-api.submit-order + +interpret errors via OrderPostError.retry_hint + } + class EthFlowModule { + <> + +on_event(log) decode OrderPlacement via alloy_sol_types + +build OrderCreation (EIP-1271 sig) via cowprotocol types + +submit via cow-api.submit-order + +interpret errors via OrderPostError.retry_hint + } + + EngineConfig --> Supervisor + Manifest --> Supervisor + Manifest "1" --> "*" Subscription + Supervisor --> ProviderPool + Supervisor --> OrderBookPool + Supervisor --> LocalStore + Supervisor "1" --> "*" HostState : per module + HostState --> ProviderPool + HostState --> OrderBookPool + HostState --> LocalStore + EventLoop --> Supervisor + TwapModule ..> HostState : WIT calls (universal) + EthFlowModule ..> HostState : WIT calls (universal) +``` + +### Class reference + +| Class | What it is | +|---|---| +| **EngineConfig** | Deserialized from `engine.toml`. Holds the database path (`state_dir`), one `ChainConfig` per chain (just an RPC URL), and the list of module paths to load. | +| **Manifest** | Deserialized from `module.toml`, which ships inside the module bundle. Declares what capabilities the module needs, which on-chain events to watch, and any module-level config values. | +| **Subscription** | One event declaration inside `module.toml`. `kind=Block` fires on every new block for a given chain. `kind=Log` fires when a specific contract emits an event matching the given address and topics. Factory-style dynamic subscriptions (`[[subscription.template]]` + `register-address`) are deferred to 0.3 — see ADR-0008. | +| **Supervisor** | Orchestrates boot and event dispatch. Creates one `HostState` per module. On each incoming block or log, calls `dispatch_block` / `dispatch_log` to fan the event out to subscribed modules. | +| **ProviderPool** | Holds one alloy `DynProvider` per chain. `wss://` chains get a pubsub provider that supports both subscriptions and requests. `https://` chains get HTTP-only (subscriptions unavailable, by design — ADR-0002). | +| **OrderBookPool** | Holds one `OrderBookApi` client per known CoW chain (Mainnet, Gnosis, Sepolia, ArbitrumOne, Base). Instantiated via `OrderBookPool::default()` at boot (ADR-0005). | +| **LocalStore** | A single redb embedded database at `state_dir`. All modules write into the same file. Keys are prefixed host-side as `[32-byte module namespace][raw_key]` so two modules never collide, and the namespace is unspoofable (ADR-0003). The namespace is `keccak256(module_name)` for locally-loaded modules and `ens_namehash(name)` for ENS-discovered modules. | +| **HostState** | The per-module runtime context. `wasmtime::component::bindgen!` generates one trait per WIT interface (e.g. `shepherd::cow::cow_api::Host`); `HostState` implements each trait. `Shepherd::add_to_linker` registers all trait implementations with the `Linker` once at boot. **Current fields** (M1): `wasi: WasiCtx`, `table: ResourceTable`, `http_allowlist: Vec`, `monotonic_baseline: Instant`. **M2 additions** will add `module_namespace: [u8; 32]`, `provider_pool: Arc`, `ob_pool: Arc`, `local_store: Arc`. | +| **EventLoop** | Runs `futures::stream::select_all` over a `Vec + Send>>>`. The loop never exits until SIGINT/SIGTERM. Each fired event is forwarded to `Supervisor` for fan-out. | +| **TwapModule** | The TWAP watcher WASM component. On a `Log` event (ConditionalOrderCreated): persists the registration in `local-store`. On a `Block` event: iterates all watches and, for each, makes an `eth_call` via `chain.request`, decodes the result via `alloy_sol_types` (in-module), builds an `OrderCreation` via `cowprotocol` types (consumed via wasm32 feature), and submits via `cow-api.submit-order`. Orderbook errors flow through `OrderPostError::retry_hint`. All polling logic lives in the module, not the host (ADR-0006). | +| **EthFlowModule** | The EthFlow watcher WASM component. On a `Log` event (OrderPlacement): decodes the event via `alloy_sol_types` in-module, builds the `OrderCreation` with the EIP-1271 signing scheme via `cowprotocol` types, and submits via `cow-api.submit-order`. No polling loop — one log equals one submission attempt. | + +--- + +## 3. WIT Interface Hierarchy + +Two WIT packages: the universal `nexum:host` and the CoW-specific `shepherd:cow`. + +```mermaid +graph TD + NH["nexum:host@0.2.0\n(universal — no CoW knowledge)"] + SC["shepherd:cow@0.2.0\n(CoW Protocol extensions)"] + + NH --> n1["chain ✅ implemented\nrequest(chain-id, method, params)\nrequest-batch(chain-id, requests)\n—\nsubscribe-blocks · subscribe-logs →\n engine-managed via module.toml subscriptions\nregister-address · unregister-address →\n 🕓 deferred to 0.3 (ADR-0008)"] + NH --> n2["local-store ✅ implemented\nget(key) · set(key, value)\ndelete(key) · list-keys(prefix)\nnamespacing: 32-byte hash prefix (ADR-0003)"] + NH --> n3["identity · messaging · http · remote-store\n✅ stubs (Unsupported) — full impl in 0.3"] + NH --> n4["logging · clock · random ✅ implemented"] + + SC -->|"use nexum:host/types"| NH + SC --> s1["cow-api ✅ implemented\nrequest(chain-id, method, path, body)\nsubmit-order(chain-id, order-data)\n→ result\n(only protocol-level interface in shepherd:cow)"] + + note1["No twap / ethflow host interfaces.\nTWAP and EthFlow logic lives in guest\nmodule code, using universal primitives\n(chain · local-store · cow-api).\nSee ADR-0006."] + SC -.-> note1 + + style note1 fill:#fff4e6,stroke:#ff9800,color:#000 +``` + +### Interface reference + +| Interface | What it does | +|---|---| +| **nexum:host@0.2.0** | The base WIT package. Any module running in the engine — CoW-aware or not — imports from here. Defines shared types (`chain-id`, `log`, `host-error`) used by both packages. | +| **chain** | Reads from the blockchain via JSON-RPC. `request` sends a single call; `request-batch` sends several in one round-trip. **Subscriptions are not callable WIT functions** — they are declared in `module.toml` and opened by the engine at boot. Dynamic `register-address` for factory patterns is deferred to 0.3 (ADR-0008). | +| **local-store** | Persistent key-value storage that survives restarts. Operations: `get(key)`, `set(key, value)`, `delete(key)`, `list-keys(prefix)`. The host prefixes every key with a 32-byte deterministic namespace (`keccak256(module_name)` locally, or `ens_namehash(name)` when ENS-loaded) so modules are fully isolated and the namespace cannot be spoofed (ADR-0003). | +| **identity · messaging · http · remote-store** | Capabilities stubbed at 0.2 — they return `Unsupported`. `identity` will provide keystore-backed signing. `messaging` will send Waku messages. `http` will allow direct outbound HTTP calls (subject to the manifest's allowlist). `remote-store` will read/write Swarm/IPFS. | +| **logging · clock · random** | Lightweight utilities. `logging` emits to the engine's `tracing` subscriber (inherits `RUST_LOG` filters). `clock` returns wall-clock time. `random` returns cryptographically-secure random bytes. | +| **shepherd:cow@0.2.0** | The CoW Protocol extension package. Imports `nexum:host/types` for shared types so modules don't re-define `chain-id` or `log`. Only CoW-aware modules need to import this package. Contains exactly **one** interface in 0.2: `cow-api`. | +| **cow-api** | Generic orderbook access. `request` is a raw REST passthrough (returns JSON string). `submit-order` takes raw order bytes and returns a `result` where the string is the order UID. Routes through the engine's `OrderBookPool`. This is the only protocol-level CoW interface in 0.2 — the boundary between "what CoW Protocol *is*" (orderbook submission, order types) and "what's implemented *on top* of CoW" (TWAP polling, EthFlow event handling). | +| **(no twap interface)** | Per ADR-0006, no specialised TWAP host interface exists. The TWAP module implements polling, decoding, and submission entirely in guest code, using `chain.request` for `eth_call`, `local-store` for state, `alloy_sol_types` (in-module) for ABI decoding, `cowprotocol` types for `OrderCreation`, and `cow-api.submit-order` for orderbook submission. Multiple TWAP strategies can coexist as separate modules with different polling policies and error tolerances. | +| **(no ethflow interface)** | Per ADR-0006, no specialised EthFlow host interface exists. The EthFlow module decodes `OrderPlacement` directly in guest code via `alloy_sol_types`, constructs the `OrderCreation` with the EIP-1271 signing scheme via `cowprotocol` types, and submits via `cow-api`. | + +--- + +## 4. Engine Boot Sequence + +```mermaid +flowchart TD + Start([nexum-engine starts]) --> ReadConfig + ReadConfig["1. Read engine.toml\n(EngineConfig::load)"] + ReadConfig --> InitTracing + InitTracing["2. Init tracing\n(RUST_LOG / log_level)"] + InitTracing --> ProvPool + ProvPool["3. ProviderPool::from_config\nFor each chain:\n wss:// → pubsub DynProvider\n https:// → http DynProvider\n(fatal on connection error — ADR-0002)"] + ProvPool --> OpenStore + OpenStore["4. LocalStore::open(state_dir)\nOpen/create redb DB\nnexum:local-store table\n(ADR-0003)"] + OpenStore --> OBPoolInit + OBPoolInit["5. OrderBookPool::default()\nBuild OrderBookApi for:\nMainnet, Gnosis, Sepolia, ArbitrumOne, Base\n(ADR-0005)"] + OBPoolInit --> SupervisorBoot + SupervisorBoot["6. Supervisor::boot\nFor each [[modules]] in engine.toml:"] + SupervisorBoot --> LoadManifest[" a. Load module.toml (Manifest)"] + LoadManifest --> LoadWasm[" b. Load .wasm Component (wasmtime)"] + LoadWasm --> Instantiate[" c. Instantiate with dedicated HostState\n (links nexum:host + shepherd:cow impls)"] + Instantiate --> CallInit[" d. Call module.init(config)"] + CallInit --> AnnotateSubs[" e. Annotate subscriptions from manifest"] + AnnotateSubs --> MoreModules{More modules?} + MoreModules -->|yes| LoadManifest + MoreModules -->|no| OpenStreams + OpenStreams["7. open_block_streams + open_log_streams\neth_subscribe newHeads per chain\neth_subscribe logs per (chain, address, topics)"] + OpenStreams --> RunLoop + RunLoop["8. run_event_loop\nfutures::stream::select_all over all streams\nfan-out: block → all block subscribers\nlog → owner module only"] + RunLoop --> Wait([Await SIGINT/SIGTERM]) +``` + +### Step reference + +| Step | What happens | +|---|---| +| **1. Read engine.toml** | Deserializes the operator config. If the file is missing, the engine falls back to defaults (no chains, default `state_dir`). Modules that need chains will receive `Unsupported` errors at runtime. | +| **2. Init tracing** | Sets up the `tracing` subscriber using `RUST_LOG` or the `log_level` field from `engine.toml`. All host log output flows through here, including per-capability trace events. | +| **3. ProviderPool** | Opens one alloy connection per chain declared in `[chains]`. WebSocket URLs get a full pubsub provider (the recommended setup for any chain a module subscribes to). HTTP URLs get a request-only provider. Any connection failure at this step is fatal — the engine refuses to start with a broken chain rather than silently degrading. Failover and retry are out of scope; they live in alloy middleware (ADR-0002). | +| **4. LocalStore** | Opens (or creates) the redb database at `state_dir`. Creates the `nexum:local-store` table if it doesn't exist. Per-module namespacing uses a 32-byte deterministic hash prefix. Module state from previous runs is immediately available. | +| **5. OrderBookPool** | Constructs one `OrderBookApi` HTTP client for each supported CoW chain via the `Default` implementation. Built upfront so config errors (unknown chain IDs) surface at boot, not on the first order submission. | +| **6. Supervisor::boot (per module)** | For each module listed in `engine.toml`: reads its `module.toml`, loads the `.wasm` component into wasmtime, creates a dedicated `HostState`, calls the module's `init(config)` export, and records which subscriptions the module declared. | +| **7. Open streams** | Aggregates all subscriptions declared across all modules. Opens one `eth_subscribe newHeads` per chain and one `eth_subscribe logs` per (chain, contract-address, topics) filter. | +| **8. Event loop** | The engine enters its steady-state loop. `futures::stream::select_all` waits for the next event on any stream. Block events are broadcast to all modules subscribed to that chain. Log events are delivered only to the module that owns that subscription. | + +--- + +## 5. TWAP Complete Flow (Registration → Submit) + +The TWAP module runs the entire flow in guest Rust code, using only universal host primitives. + +```mermaid +sequenceDiagram + actor User + participant CC as ComposableCoW
Contract + participant RPC as RPC Node
(wss://) + participant EL as EventLoop + participant TM as twap module
(WASM guest) + participant SD as alloy_sol_types
(in module) + participant CR as cowprotocol types
(in module, wasm32) + participant HS as HostState
(Rust) + participant OB as api.cow.fi
(Orderbook) + participant LS as LocalStore
(redb) + + Note over User,CC: Step 0 — On-chain registration (off-engine) + User->>CC: ComposableCoW.create(twapParams) + CC-->>RPC: emit ConditionalOrderCreated(owner, params, proof) + + Note over RPC,LS: Step 1 — Indexing (once per TWAP) + RPC->>EL: log batch (eth_subscribe logs) + EL->>TM: on_event(Event::Logs([registration_log])) + TM->>SD: decode ConditionalOrderCreated + SD-->>TM: (owner, params, salt) + TM->>HS: local-store.set("watch:{owner}:{hash}", params) + HS->>LS: write [module_namespace][watch:...] = params_bytes + + Note over RPC,LS: Step 2 — Poll loop (every block) + loop Every block on chain_id + RPC->>EL: block header (eth_subscribe newHeads) + EL->>TM: on_event(Event::Block(Block { number: N, ... })) + TM->>HS: local-store.list-keys("watch:") + HS-->>TM: registration entries + + loop For each watch where next_attempt <= N + TM->>HS: chain.request(chain_id, "eth_call", [...]) + HS->>RPC: eth_call ComposableCoW.getTradeableOrderWithSignature(owner, params, "", []) + RPC-->>HS: return value or revert + HS-->>TM: result (JSON string) + TM->>SD: decode return value (or interpret revert reason) + SD-->>TM: ready(GPv2OrderData, signature) OR not-ready(hint) + + alt Order ready + TM->>CR: build OrderCreation + CR-->>TM: OrderCreation + TM->>HS: cow-api.submit-order(chain_id, order_json) + HS->>OB: POST /api/v1/orders + alt Submit OK + OB-->>HS: 200 OK, OrderUid + HS-->>TM: Ok(OrderUid) + TM->>LS: set("submitted:{uid}", order_uid) + else Orderbook error + OB-->>HS: 4xx with error code + HS-->>TM: Err(host-error) + TM->>CR: OrderPostError::try_from(host-error).retry_hint() + CR-->>TM: TryNextBlock / BackoffSeconds(s) / Drop + TM->>LS: update next_attempt or remove watch + end + else Not yet ready (TryAtEpoch / TryOnBlock / TryNextBlock / Terminal) + TM->>LS: persist hint (next_attempt) or delete watch + end + end + end +``` + +### Participant reference + +| Participant | Role in this flow | +|---|---| +| **User** | The trader. Interacts with the blockchain directly — the engine never touches private keys. | +| **ComposableCoW Contract** | The on-chain conditional order registry. Accepts TWAP parameters via `create()` and emits `ConditionalOrderCreated`. Also exposes `getTradeableOrderWithSignature()`, which the engine polls to check whether the current TWAP part is ready to trade. | +| **RPC Node** | The WebSocket connection to the chain. Delivers log events (subscriptions) and handles `eth_call` (synchronous reads). Must be `wss://` for this flow since it uses subscriptions. | +| **EventLoop** | Receives raw events from the RPC node and routes them to the module that subscribed to them. Opaque to the flow — it just calls `on_event`. | +| **twap module (WASM guest)** | Contains the entire TWAP strategy: decoding registrations, deciding when to poll (using stored hints), reacting to revert reasons, building orders, interpreting orderbook errors. Calls into the host only through universal WIT primitives. | +| **alloy_sol_types (in module)** | The ABI-aware decoder. Compiled into the module's WASM. Decodes `ConditionalOrderCreated` from raw log bytes; decodes the `getTradeableOrderWithSignature` return; interprets revert reasons. No host involvement for decoding. | +| **cowprotocol types (in module)** | The protocol-level types from `bleu/cow-rs`, consumed by the module via the wasm32 feature (ADR-0007 item 3). Used to build `OrderCreation`, manipulate `OrderUid`, and pattern-match `OrderPostError`. The crate's HTTP client (`OrderBookApi`) is **not** used directly by the module — orderbook submission goes through the host's `cow-api`. | +| **HostState (Rust)** | Provides only the universal primitives (`chain.request`, `local-store.*`, `cow-api.submit-order`). Knows nothing about TWAP semantics. | +| **api.cow.fi (Orderbook)** | Receives the signed `OrderCreation`, validates it, and returns a 56-byte `OrderUid`. The order is now visible to CoW solvers. | +| **LocalStore (redb)** | Persistent state for the TWAP module. `watch:{owner}:{hash}` entries hold registrations. `submitted:{uid}` entries record completed submissions. `next_attempt` hints (epoch or block) let the module skip polling during the gap between TWAP parts. All entries survive engine restarts. | + +--- + +## 6. EthFlow Complete Flow (Event-Driven) + +```mermaid +sequenceDiagram + actor User + participant EFC as CoWSwapEthFlow
Contract + participant RPC as RPC Node
(wss://) + participant EL as EventLoop + participant EM as eth-flow module
(WASM guest) + participant SD as alloy_sol_types
(in module) + participant CR as cowprotocol types
(in module, wasm32) + participant HS as HostState
(Rust) + participant OB as api.cow.fi
(Orderbook) + participant LS as LocalStore
(redb) + + Note over User,EFC: Step 0 — User creates ETH order on-chain + User->>EFC: createOrder(order, msg.value=ETH) + EFC->>EFC: store orders[hash] = onchainData,
emit OrderPlacement(sender, order, EIP1271-sig, data) + EFC-->>RPC: log emitted on block N + + Note over RPC,LS: Step 1 — Log arrives via subscription + RPC->>EL: log batch matching CoWSwapEthFlow address + OrderPlacement topic + EL->>EM: on_event(Event::Logs([placement_log])) + + Note over EM,LS: Step 2 — Decode and submit (1 log = 1 submission) + EM->>SD: decode OrderPlacement(sender, order, sig, data) + SD-->>EM: (sender, GPv2OrderData, EIP-1271 sig, data) + + EM->>CR: build OrderCreation with EIP-1271 scheme
pointing at CoWSwapEthFlow contract + CR-->>EM: OrderCreation + OrderUid + + EM->>HS: cow-api.submit-order(chain_id, order_json) + HS->>OB: POST /api/v1/orders + OB-->>HS: result + + alt 200 OK with OrderUid + HS-->>EM: Ok(OrderUid) + EM->>LS: set("submitted:{uid}", order_uid) + else 4xx with error code + HS-->>EM: Err(host-error with code) + EM->>CR: OrderPostError::try_from(host-error).retry_hint() + CR-->>EM: TryNextBlock / BackoffSeconds(s) / Drop + + alt TryNextBlock + Note over EM: log and skip; next block retries + else BackoffSeconds(s) + EM->>LS: set("backoff:{uid}", now + s) + else Drop + EM->>LS: set("dropped:{uid}", reason) + end + end +``` + +### Participant reference + +| Participant | Role in this flow | +|---|---| +| **User** | The trader. Deposits native ETH into the `CoWSwapEthFlow` contract and specifies swap parameters. The contract is the EIP-1271 signer on behalf of the user. | +| **CoWSwapEthFlow Contract** | Custodies the ETH, stores the order metadata on-chain, and emits `OrderPlacement` so off-chain relayers (this module, plus CoW's own internal autopilot indexer) can pick up the order. | +| **RPC Node** | Delivers the `OrderPlacement` log via the persistent WebSocket subscription. No `eth_call` is needed in this flow — the log contains everything required to reconstruct the order. | +| **EventLoop** | Routes the log to the eth-flow module based on the `[[subscription]]` entry in its `module.toml` (matching the `CoWSwapEthFlow` contract address and the `OrderPlacement` topic). | +| **eth-flow module (WASM guest)** | Contains the entire EthFlow relay logic: decoding, OrderCreation construction, submission, error handling. No polling loop; one log equals one submission attempt. | +| **alloy_sol_types (in module)** | Decodes the `OrderPlacement` event in module-side Rust. The event payload carries the typed `GPv2OrderData`, the EIP-1271 signature blob, and the extra data field. | +| **cowprotocol types (in module)** | Used to construct the `OrderCreation` with the EIP-1271 signing scheme (the signature points at the `CoWSwapEthFlow` contract address, not at the user's key) and to compute the 56-byte `OrderUid`. `OrderPostError` from the same crate is used to interpret orderbook errors. | +| **HostState (Rust)** | Provides only the `cow-api.submit-order` primitive for this flow. Maps orderbook errors to `host-error` with the original error code preserved so the module can recover the typed `OrderPostError`. | +| **api.cow.fi (Orderbook)** | Receives the order. Returns `OrderUid` on success. Returns a typed error code on failure, which the module recovers and passes through `OrderPostError::retry_hint()` to decide what to do next. App-data documents are **not** fetched here; the trader uploads them via `PUT /api/v1/app_data/{hash}` separately. | +| **LocalStore (redb)** | `submitted:{uid}` records successful submissions. `backoff:{uid}` records pending retries with a deadline. `dropped:{uid}` records permanently-failed orders. All entries survive restarts so the module does not re-submit known orders. | + +--- + +## 7. Capability Dispatch (Generic Host Call Path) + +How any WIT function call from a WASM module reaches the host backend it targets. + +```mermaid +flowchart TD + Module["WASM Module\n(twap or eth-flow)"] + Module -->|"e.g. cow-api.submit-order(chain_id, order_json)"| Linker + Linker["wasmtime Linker\n(resolves import → host function)"] + Linker --> CapCheck["HostState\nmanifest.required check\n(returns denied if not declared)"] + CapCheck --> Trace["tracing::info!\n[capability] op chain=…"] + Trace --> HostFn["host backend Rust function"] + HostFn -->|"cow-api"| OBPool["OrderBookPool\n.get(chain_id)?.post_order(order)"] + HostFn -->|"chain (request)"| ProviderP["ProviderPool\n.get(chain_id).request(method, params)"] + HostFn -->|"chain (subscribe-*)"| EngineSubs["engine-managed streams\nopened at boot from module.toml"] + HostFn -->|"local-store"| StoreP["LocalStore\n.get/set/delete/list_keys\n(namespace-prefixed host-side)"] + HostFn -->|"logging · clock · random"| Misc["direct host impl"] + OBPool --> CowAPI["api.cow.fi\nPOST /api/v1/orders"] + ProviderP --> RPCNode["RPC Node\nJSON-RPC"] + CowAPI -->|"OrderUid or error"| Module + RPCNode -->|"result"| Module +``` + +### Node reference + +| Node | What it does | +|---|---| +| **WASM Module** | The guest program. It calls imported WIT functions exactly like regular function calls — it has no visibility into the host machinery behind them. | +| **wasmtime Linker** | `Linker` built once at startup. `wasmtime::component::bindgen!` generates a `Shepherd` world struct and one trait per WIT interface (e.g. `shepherd::cow::cow_api::Host`, `nexum::host::local_store::Host`). `Shepherd::add_to_linker(&mut linker, \|state\| state)` registers every trait method as a host function. After that, calls from WASM resolve with zero dynamic dispatch overhead — the vtable is built at link time, not per-call. | +| **HostState — manifest.required check** | Before dispatching, `HostState` checks that the called capability is listed under `[capabilities].required` in the module's `module.toml`. If not, it returns `host-error { kind: denied }` immediately. The 0.2 engine validates known capability names at boot via `KNOWN_CAPABILITIES`; per-call gating is the M2 target. | +| **tracing::info!** | Every host call emits a structured trace event (capability name, chain id, etc.). Operators use `RUST_LOG=shepherd=debug` to see every call a module makes. | +| **host backend Rust function** | `HostState` implements one generated trait per WIT interface. Each `async fn` in the trait receives `&mut self` (giving access to all host resources) and returns the WIT-mapped Rust type. There are no CoW-strategy-specific backends — only the universal ones plus `cow-api` (ADR-0006). | +| **OrderBookPool** | Looks up the `OrderBookApi` client for the requested chain and calls `post_order`. Returns a 56-byte `OrderUid` on success or an `OrderPostError`-bearing host error on failure. | +| **ProviderPool (chain.request)** | Looks up the alloy provider for the requested chain and dispatches the JSON-RPC call (`eth_call`, `eth_getLogs`, etc.). | +| **engine-managed streams (chain.subscribe-*)** | Subscriptions are not exposed as runtime-callable host functions in 0.2. They are opened by the engine at boot from each module's declared `[[subscription]]` entries; events flow into the module via `on_event`. Dynamic `register-address` for factory patterns is deferred (ADR-0008). | +| **LocalStore** | Reads or writes a key in the module's namespace. The module sees plain keys; the host silently prepends a 32-byte namespace prefix. | +| **logging · clock · random** | Lightweight stateless helpers; implemented directly on `HostState` without a separate pool. | + +--- + +## 8. Repository Dependency Map + +```mermaid +graph TD + upstream["cowdao-grants/cow-rs\n(alpha.3 on crates.io — PR #5 base)"] + bleu_cr["bleu/cow-rs\n(PR #5 head branch)"] + prims["Protocol primitives added to PR #5:\n• OrderPostError + retry_hint\n• OrderBookApi::with_base_url\n• wasm32 feature-gate"] + existing["Already in PR #5:\nOrder · OrderCreation · OrderUid\nsigning schemes · OrderBookApi"] + patch["[patch.crates-io]\ncowprotocol → bleu/cow-rs @ rev"] + engine["nexum-engine\n(WIT host, supervisor, event loop)"] + witCowApi["shepherd:cow/cow-api WIT"] + modules["WASM modules\n(twap · eth-flow)"] + + upstream -->|"is PR base"| bleu_cr + bleu_cr --> prims + bleu_cr --> existing + patch -->|"redirects cowprotocol to rev of"| bleu_cr + engine -->|"workspace Cargo.toml declares"| patch + engine --> witCowApi + witCowApi -->|"backed by OrderBookApi"| existing + modules -->|"imports WIT"| witCowApi + modules -.->|"consumes types (wasm32 feature)"| existing + modules -.->|"matches on errors"| prims +``` + +### Node reference + +| Node | What it is | +|---|---| +| **cowdao-grants/cow-rs** | The upstream CoW Protocol Rust SDK, maintained by the DAO. Version `alpha.3` is published to crates.io but predates 18 follow-up commits Bleu has been pushing through PR #5. This is the PR base — changes land here eventually. | +| **bleu/cow-rs** | Bleu's repository, which is simultaneously the head branch of the DAO's open PR #5. Every commit Bleu pushes here also advances PR #5 for upstream review. This is not a long-lived parallel fork — it is the active PR branch (ADR-0004). | +| **Protocol primitives added to PR #5** | The three additions Bleu is pushing into PR #5: `OrderPostError` rich variants + `retry_hint()` (critical for module error handling), `OrderBookApi::with_base_url` (barn / staging / forked deployments), and `wasm32` feature-gating (critical so guest modules can consume `cowprotocol` types). All three are protocol primitives — they describe what CoW Protocol *is*, not how a particular strategy uses it. TWAP polling and EthFlow event decoding are explicitly *not* added here; they stay in module code (ADR-0007). | +| **Already in PR #5** | The types and orderbook client Bleu's modules consume but did not add: `Order`, `OrderCreation`, `OrderUid`, signing-scheme enums, and `OrderBookApi`. These existed in PR #5 before the M2 work. | +| **[patch.crates-io]** | A single line in the workspace `Cargo.toml` that tells Cargo to use `bleu/cow-rs` at a specific git rev instead of the `alpha.3` release on crates.io. Bumping the rev is the only change needed to pick up a new primitive after it is pushed to `bleu/cow-rs` (ADR-0004). | +| **nexum-engine** | The engine binary. Contains the WIT host implementations, Supervisor, EventLoop, config loaders, and alloy/redb integration. Contains no CoW Protocol logic — protocol primitives live in `bleu/cow-rs`; strategy logic lives in guest modules. | +| **shepherd:cow/cow-api WIT** | The only CoW-specific WIT interface in 0.2. The engine implements it (host side); WASM modules import it (guest side). Backed by `OrderBookPool` (and through that, `OrderBookApi` from `cow-rs`). | +| **WASM modules (twap · eth-flow)** | The grant deliverables. Compiled to `.wasm` Component Model binaries. Import only universal WIT interfaces (`chain`, `local-store`, `logging`) plus `shepherd:cow/cow-api`. Consume `cowprotocol` types directly through the wasm32 feature for building `OrderCreation` and pattern-matching on `OrderPostError`. Contain all TWAP and EthFlow strategy logic themselves (ADR-0006). | diff --git a/docs/diagrams/engine-boot.mmd b/docs/diagrams/engine-boot.mmd new file mode 100644 index 0000000..85ff094 --- /dev/null +++ b/docs/diagrams/engine-boot.mmd @@ -0,0 +1,59 @@ +sequenceDiagram + autonumber + actor Op as Operator + participant Bin as nexum-engine binary + participant FS as Filesystem + participant Trc as tracing subscriber + participant PP as ProviderPool + participant LS as LocalStore (redb) + participant OBP as OrderBookPool + participant Sup as Supervisor + participant Mod as Module instance(s) + participant EL as Event Loop + + Op->>Bin: nexum-engine --engine-config engine.toml + Bin->>FS: read engine.toml + FS-->>Bin: EngineConfig { engine, chains, modules } + + Bin->>Trc: init_tracing(log_level) + Note over Trc: RUST_LOG overrides
engine.toml.log_level + + Bin->>PP: ProviderPool::from_config(&cfg) + loop For each chain in cfg.chains + alt URL is ws:// or wss:// + PP->>PP: ProviderBuilder.connect_ws(WsConnect) + else URL is http:// or https:// + PP->>PP: ProviderBuilder.connect_http(Url) + end + Note over PP: Connection failure = fatal,
engine refuses to start + end + PP-->>Bin: pool ready + + Bin->>LS: LocalStore::open(state_dir) + Note over LS: Creates redb file if missing,
materialises shared table. + LS-->>Bin: store ready + + Bin->>OBP: OrderBookPool::default() + Note over OBP: One OrderBookApi per
cowprotocol::Chain variant. + OBP-->>Bin: pool ready + + Bin->>Sup: Supervisor::boot(cfg, host_resources) + loop For each [[modules]] entry in cfg.modules + Sup->>FS: read module.toml + FS-->>Sup: Manifest + Sup->>FS: load .wasm + FS-->>Sup: wasm bytes + Sup->>Mod: wasmtime Component compile + instantiate
with dedicated HostState + Sup->>Mod: call init(config) inside write txn + Mod-->>Sup: Ok or Err + Sup->>Sup: record subscriptions declared in manifest + end + Sup-->>Bin: modules loaded + + Bin->>EL: open block + log subscriptions + EL->>PP: subscribe_blocks(chain_id) per unique chain + EL->>PP: subscribe_logs(chain_id, filter) per unique filter + PP-->>EL: streams ready + + Bin->>EL: run_event_loop(futures::select_all) + Note over EL: Loop until SIGINT or SIGTERM diff --git a/docs/diagrams/module-lifecycle.mmd b/docs/diagrams/module-lifecycle.mmd new file mode 100644 index 0000000..7a73bd1 --- /dev/null +++ b/docs/diagrams/module-lifecycle.mmd @@ -0,0 +1,39 @@ +stateDiagram-v2 + direction LR + + [*] --> Resolve + + Resolve --> Load: bundle fetched OK + Resolve --> Dead: resolution failed permanently + + Load --> Init: wasmtime compile and instantiate OK + Load --> Restart: compile or instantiate failed + + Init --> Run: init(config) returned Ok + Init --> Restart: init returned Err or trapped + + Run --> Run: on_event handled OK + Run --> Restart: on_event trapped or returned Err or fuel exhausted + + Restart --> Init: backoff expired (1s, 2s, 4s, up to 5min cap) + Restart --> Dead: max_consecutive_failures reached (default 10) + + Dead --> Init: operator action (nexum module restart or reload) + Dead --> [*]: operator removes module + + note right of Restart + Exponential backoff with jitter. + Memory zeroed, local-store survives. + InstancePre reused, no recompile. + end note + + note right of Dead + Module excluded from dispatch. + Engine continues running other modules. + Manual intervention required to resume. + end note + + note right of Init + Runs inside an implicit write txn. + Ok commits, Err rolls back. + end note diff --git a/docs/diagrams/sequence-ethflow.mmd b/docs/diagrams/sequence-ethflow.mmd new file mode 100644 index 0000000..7427779 --- /dev/null +++ b/docs/diagrams/sequence-ethflow.mmd @@ -0,0 +1,39 @@ +sequenceDiagram + autonumber + actor User as User
(wallet) + participant Contract as CoWSwapEthFlow
(on-chain) + participant RPC as Chain RPC
(WSS) + participant EL as nexum-engine
Event Loop + participant Mod as ethflow-watcher
(WASM module) + participant SolDec as alloy_sol_types
(in module) + participant Cow as cowprotocol crate
(in module, wasm32) + participant Host as HostState + participant OB as CoW Orderbook + + User->>Contract: createOrder(order, msg.value=ETH) + Contract->>Contract: store orders[hash] = onchainData,
emit OrderPlacement + Contract-->>RPC: log emitted on block N + + RPC-->>EL: eth_subscribe logs delivery + EL->>Mod: on_event(LogEvent) + + Mod->>SolDec: decode OrderPlacement(sender, order, sig, data) + SolDec-->>Mod: (sender, GPv2OrderData, EIP1271-sig, data) + + Mod->>Cow: build OrderCreation with EIP-1271 scheme
pointing at CoWSwapEthFlow contract + Cow-->>Mod: OrderCreation + OrderUid + + Mod->>Host: cow-api.submit-order(chain_id, order_json) + Host->>OB: POST /api/v1/orders + OB-->>Host: result + + alt 200 OK with OrderUid + Host-->>Mod: Ok(OrderUid) + Mod->>Host: local-store.set("submitted:{uid}", ...) + else 4xx with error code + Host-->>Mod: Err(host-error with code) + Mod->>Cow: OrderPostError::try_from(host-error).retry_hint() + Cow-->>Mod: RetryHint variant + end + + Note over Mod: Module applies RetryHint:
TryNextBlock - log + skip
BackoffSeconds(s) - persist next_attempt = now + s
Drop - permanent failure, do not retry diff --git a/docs/diagrams/sequence-twap.mmd b/docs/diagrams/sequence-twap.mmd new file mode 100644 index 0000000..5b5e75e --- /dev/null +++ b/docs/diagrams/sequence-twap.mmd @@ -0,0 +1,56 @@ +sequenceDiagram + autonumber + actor User as User
(wallet) + participant Contract as ComposableCoW
(on-chain) + participant RPC as Chain RPC
(WSS) + participant EL as nexum-engine
Event Loop + participant Mod as twap-monitor
(WASM module) + participant SolDec as alloy_sol_types
(in module) + participant Cow as cowprotocol crate
(in module, wasm32) + participant Host as HostState
(host backends) + participant OB as CoW Orderbook + + Note over User,Contract: Registration (one-time per TWAP) + User->>Contract: create(twapParams) + Contract-->>RPC: emit ConditionalOrderCreated(owner, params) + RPC-->>EL: log delivered + EL->>Mod: on_event(LogEvent) + Mod->>SolDec: decode ConditionalOrderCreated + SolDec-->>Mod: (owner, params, salt) + Mod->>Host: local-store.set("watch:{owner}:{hash}", params) + + Note over EL,OB: Per-block polling loop + loop For each block N + RPC-->>EL: eth_subscribe newHeads + EL->>Mod: on_event(BlockEvent { N }) + Mod->>Host: local-store.list_keys("watch:") + + loop For each watch where next_attempt <= N + Mod->>Host: chain.request(chain_id, "eth_call", [...]) + Host->>RPC: eth_call ComposableCoW.
getTradeableOrderWithSignature(owner, params) + RPC-->>Host: return value or revert + Host-->>Mod: result (JSON encoded) + Mod->>SolDec: decode return or interpret revert reason + SolDec-->>Mod: PollOutcome (module-defined enum) + + alt Ready (order, signature) + Mod->>Cow: build OrderCreation + Cow-->>Mod: OrderCreation + Mod->>Host: cow-api.submit-order(chain_id, order_json) + Host->>OB: POST /api/v1/orders + OB-->>Host: Ok(OrderUid) or Err(error_code) + Host-->>Mod: Result + alt Ok(uid) + Mod->>Host: local-store.set("submitted:{uid}", ...) + else Err + Mod->>Cow: OrderPostError::try_from(host-error)
.retry_hint() + Cow-->>Mod: TryNextBlock / BackoffSeconds(s) / Drop + Mod->>Host: local-store update
(next_attempt or remove watch) + end + else NotReady (try at epoch t) + Mod->>Host: local-store.set("watch:...next_attempt", t) + else Terminal (TWAP completed or cancelled) + Mod->>Host: local-store.delete("watch:...") + end + end + end diff --git a/docs/diagrams/subscription-dispatch.mmd b/docs/diagrams/subscription-dispatch.mmd new file mode 100644 index 0000000..6f11359 --- /dev/null +++ b/docs/diagrams/subscription-dispatch.mmd @@ -0,0 +1,61 @@ +graph TB + subgraph manifests["Module Manifests (module.toml per module)"] + M1S["module twap-monitor:
[[subscription]] kind=log
chain=1, address=ComposableCoW,
topics=[ConditionalOrderCreated]
[[subscription]] kind=block, chain=1"] + M2S["module ethflow-watcher:
[[subscription]] kind=log
chain=1, address=CowEthFlow,
topics=[OrderPlacement]"] + M3S["module other-module:
[[subscription]] kind=block, chain=1"] + end + + subgraph supervisor["Supervisor: aggregate at boot"] + BlockAgg["Block subscriptions
grouped by chain_id"] + LogAgg["Log subscriptions
grouped by (chain_id, filter_key)
where filter_key = address+topics hash"] + end + + subgraph subs["Aggregated WSS subscriptions"] + BSub["eth_subscribe newHeads
chain=1 (shared by all subscribers)"] + LSub1["eth_subscribe logs
chain=1, filter=ComposableCoW+ConditionalOrderCreated
(owner: twap-monitor)"] + LSub2["eth_subscribe logs
chain=1, filter=CowEthFlow+OrderPlacement
(owner: ethflow-watcher)"] + end + + subgraph dispatch["Event Loop dispatch"] + Decide{Event kind?} + Broadcast["Fan-out: dispatch to ALL
subscribers of that chain"] + Route["Route: dispatch to OWNER
of matching filter_key only"] + end + + subgraph delivery["Module on_event invocations"] + D1["twap-monitor.on_event(BlockEvent)"] + D2["ethflow-watcher.on_event(LogEvent)
or twap-monitor (LogEvent)"] + D3["other-module.on_event(BlockEvent)"] + end + + M1S --> BlockAgg + M1S --> LogAgg + M2S --> LogAgg + M3S --> BlockAgg + + BlockAgg --> BSub + LogAgg --> LSub1 + LogAgg --> LSub2 + + BSub --> Decide + LSub1 --> Decide + LSub2 --> Decide + + Decide -- "block" --> Broadcast + Decide -- "log" --> Route + + Broadcast --> D1 + Broadcast --> D3 + Route --> D2 + + classDef manifest fill:#fff4e6,stroke:#ff9800,color:#000 + classDef agg fill:#e6f3ff,stroke:#1e88e5,color:#000 + classDef wss fill:#f0e6ff,stroke:#7e3ff2,color:#000 + classDef decide fill:#fffde7,stroke:#fbc02d,color:#000 + classDef dispatch fill:#e6ffe6,stroke:#2e7d32,color:#000 + + class M1S,M2S,M3S manifest + class BlockAgg,LogAgg agg + class BSub,LSub1,LSub2 wss + class Decide,Broadcast,Route decide + class D1,D2,D3 dispatch diff --git a/docs/diagrams/wit-call-path.mmd b/docs/diagrams/wit-call-path.mmd new file mode 100644 index 0000000..5fe4159 --- /dev/null +++ b/docs/diagrams/wit-call-path.mmd @@ -0,0 +1,34 @@ +sequenceDiagram + autonumber + participant Src as Module Rust source
(twap-monitor/src/lib.rs) + participant Bindgen as wit-bindgen
(generated stubs) + participant Wasm as WASM Component
(twap.wasm) + participant Wt as wasmtime instance + participant Lk as wasmtime Linker + participant HS as HostState (Rust) + participant PP as ProviderPool + participant Alloy as alloy provider + participant RPC as Chain RPC + + Note over Src,Wasm: Build phase (cargo build) + Src->>Bindgen: chain::request(chain_id, method, params)
(plain Rust call) + Bindgen->>Wasm: compile into WASM Component
with "nexum:host/chain" import + + Note over Wasm,RPC: Runtime: module calls chain::request + Wasm->>Wt: WASM import call:
"nexum:host/chain"."request" + Wt->>Lk: lookup binding for import + Lk->>HS: invoke trait impl:
HostState::Chain::request(...) + Note over HS: tracing::info!("[chain] request
chain=... method=...") + HS->>PP: pool.get(chain_id) + PP-->>HS: &DynProvider + HS->>Alloy: provider.raw_request(method, params) + Alloy->>RPC: JSON-RPC over WSS/HTTP + RPC-->>Alloy: response bytes + Alloy-->>HS: serde_json::Value + HS-->>Lk: Ok(stringified JSON) + Lk-->>Wt: return value (marshaled) + Wt-->>Wasm: import call returns + Wasm-->>Bindgen: unmarshal canonical ABI
back to Rust types + Bindgen-->>Src: returns Result + + Note over Src,RPC: All host calls follow this path.
Only the trait impl on the right side
(HS → PP/OBP/LS/...) changes per capability.