diff --git a/pubsub/gossipsub/README.md b/pubsub/gossipsub/README.md index f119e3935..43a3e9f86 100644 --- a/pubsub/gossipsub/README.md +++ b/pubsub/gossipsub/README.md @@ -17,20 +17,25 @@ If you are new to Gossipsub and/or PubSub in general, we recommend you to first: - [gossipsub-v1.0](gossipsub-v1.0.md): v1.0 of the gossipsub protocol. This is a revised specification, to use a more normative language. The original v1.0 specification is [here](gossipsub-v1.0-old.md), still a good read. - [gossipsub-v1.1](gossipsub-v1.1.md): v1.1 of the gossipsub protocol. - [gossipsub-v1.2](gossipsub-v1.2.md): v1.2 of the gossipsub protocol. This includes the aggregation of the IDONTWANT control messages to the specs. +- [gossipsub-v1.3](gossipsub-v1.3.md): v1.3 of the gossipsub protocol, introducing the Extensions Control Message framework. +- [Extensions Guide](extensions/README.md): comprehensive documentation of the GossipSub v1.3 extensions framework, including the canonical and experimental extension registry. - [(not in use) episub](episub.md): a research note on a protocol building on top of gossipsub to implement [epidemic broadcast trees](https://www.gsd.inesc-id.pt/~ler/reports/srds07.pdf). ## Implementation status -Legend: ✅ = complete, 🏗 = in progress, ❕ = not started yet +For a detailed per-extension breakdown across all implementations, see +[implementation-status.md](implementation-status.md). -| Name | v1.0 | v1.1 | v1.2 | -|--------------------------------------------------------------------------------------------------|:-----:|:-----:|:----:| -| [go-libp2p-pubsub (Golang)](https://github.com/libp2p/go-libp2p-pubsub/blob/master/gossipsub.go) | ✅ | ✅ | ✅ | -| [js-libp2p-gossipsub (JavaScript)](https://github.com/ChainSafe/js-libp2p-gossipsub) | ✅ | ✅ | ✅ | -| [rust-libp2p (Rust)](https://github.com/libp2p/rust-libp2p/tree/master/protocols/gossipsub) | ✅ | ✅ | ❔ | -| [py-libp2p (Python)](https://github.com/libp2p/py-libp2p/tree/master/libp2p/pubsub) | ✅ | 🏗 | ❔ | -| [jvm-libp2p (Java/Kotlin)](https://github.com/libp2p/jvm-libp2p/tree/develop/src/main/kotlin/io/libp2p/pubsub) | ✅ | 🏗 | ✅ | -| [nim-libp2p (Nim)](https://github.com/status-im/nim-libp2p/blob/master/libp2p/protocols/pubsub/gossipsub.nim) | ✅ | 🏗 | ✅ | +Legend: ✅ = complete, 🏗 = in progress, ❔ = not started + +| Name | v1.0 | v1.1 | v1.2 | v1.3 | +|------|:----:|:----:|:----:|:----:| +| [go-libp2p (Golang)](https://github.com/libp2p/go-libp2p-pubsub/blob/master/gossipsub.go) | ✅ | ✅ | ✅ | 🏗 | +| [js-libp2p (JavaScript)](https://github.com/ChainSafe/js-libp2p-gossipsub) | ✅ | ✅ | ✅ | ❔ | +| [rust-libp2p (Rust)](https://github.com/libp2p/rust-libp2p/tree/master/protocols/gossipsub) | ✅ | ✅ | ✅ | 🏗 | +| [py-libp2p (Python)](https://github.com/libp2p/py-libp2p/tree/master/libp2p/pubsub) | ✅ | ✅ | ✅ | ✅ | +| [jvm-libp2p (Java/Kotlin)](https://github.com/libp2p/jvm-libp2p/tree/develop/src/main/kotlin/io/libp2p/pubsub) | ✅ | 🏗 | ✅ | ❔ | +| [nim-libp2p (Nim)](https://github.com/status-im/nim-libp2p/blob/master/libp2p/protocols/pubsub/gossipsub.nim) | ✅ | 🏗 | ✅ | ❔ | Additional tooling: diff --git a/pubsub/gossipsub/extensions/README.md b/pubsub/gossipsub/extensions/README.md new file mode 100644 index 000000000..6aa6ad0ff --- /dev/null +++ b/pubsub/gossipsub/extensions/README.md @@ -0,0 +1,108 @@ +# GossipSub Extensions Framework + +GossipSub v1.3 introduced a generic **Extensions Control Message** mechanism +that allows implementations to negotiate and enable optional protocol features +at connection time without breaking compatibility with peers that don't support +them. + +This document explains how the extensions framework works and serves as a +registry of both canonical and experimental extensions. + +## How Extensions Work + +### 1. Capability Advertisement + +When two peers establish a GossipSub stream, the **first RPC message** sent in +each direction MUST include a `ControlExtensions` message inside the existing +`ControlMessage` container if the sending peer supports any extensions: + +```protobuf +message ControlExtensions { + // Canonical extensions (field numbers < 0x200000) + optional bool topicObservation = 1; + + // Experimental extensions (field numbers > 0x200000) + optional bool testExtension = 6492434; + optional bool largeMessageSegmentation = 6492435; +} +``` + +This handshake follows the GossipSub v1.3 spec rules: + +1. Extensions MUST be sent in the **first control message** on a fresh stream. +2. Extensions MUST NOT be sent **more than once** per peer. +3. Duplicate extensions messages are a protocol violation and SHOULD be reported + as misbehaviour. + +### 2. Protocol Activation + +Once both peers have exchanged extensions, an extension is considered **active** +if and only if **both** peers set its capability flag to `true`. Extension- +specific behaviour (e.g., sending segmented messages, sending observation +requests) MUST NOT begin before mutual activation. + +### 3. Field Number Allocation + +Field numbers in `ControlExtensions` and on the `RPC` message are split into +two ranges: + +| Range | Purpose | Example | +|-------|---------|---------| +| 1 – 2,097,151 | Canonical extensions — small wire overhead (1-2 bytes) | `topicObservation = 1` | +| > 2,097,152 (0x200000) | Experimental extensions — 4+ byte encoding | `testExtension = 6492434` | + +Experimental field numbers are allocated on a first-come, first-served basis. +There is no central registry — implementers choose an unused number in the +experimental range. To avoid collisions, check existing PRs and extensions +before picking a number. + +## Registered Extensions + +### Canonical + +| # | Extension | Field | Status | Spec | +|---|-----------|-------|--------|------| +| 10 | Partial Messages | `RPC.partial = 10` | Draft | [PR #685](https://github.com/libp2p/specs/pull/685) | + +### Experimental + +| # | Field Number | Extension | Field Name | Status | Spec | +|---|--------------|-----------|------------|--------|------| +| 1 | 6492434 | Test Extension | `RPC.testExtension` | Draft | [test-extension.md](experimental/test-extension.md) | +| 2 | 6492435 | Large Message Segmentation | `RPC.largeMessageSegmentation` | Draft | [PR #2](https://github.com/seetadev/specs/pull/2) | + +### How to Register a New Experimental Extension + +1. Pick an unused field number > `0x200000` (2,097,152). +2. Add a `bool` field to `ControlExtensions` and a message field to `RPC` in + `extensions.proto`. +3. Write an extension specification document in + `extensions/experimental/.md` following the + [test-extension.md](experimental/test-extension.md) template. +4. Open a PR to this repository. + +## Extension Lifecycle + +Experimental extensions follow the libp2p spec lifecycle: + +``` +Experimental (1A) → Working Draft (1B) → Candidate Recommendation (2A) → Recommendation (2B) +``` + +An extension may progress from Experimental to Working Draft once at least two +independent implementations exist and basic interoperability has been +demonstrated. + +## Relationship to GossipSub Versions + +| Version | Key Feature | Relationship to Extensions | +|---------|-------------|---------------------------| +| v1.0 | Baseline mesh + gossip | No extension mechanism | +| v1.1 | Peer scoring, PX | No extension mechanism | +| v1.2 | IDONTWANT, adaptive gossip | No extension mechanism | +| v1.3 | **Extensions Control Message** | Framework foundation | +| v1.4 | Large message handling (proposed) | Built as an extension on v1.3 | + +Extensions are designed to be independent of the wire protocol version. A v1.3 +peer can negotiate any extension with another v1.3 peer without requiring a new +protocol ID. diff --git a/pubsub/gossipsub/implementation-status.md b/pubsub/gossipsub/implementation-status.md index 01919efea..16b0d2646 100644 --- a/pubsub/gossipsub/implementation-status.md +++ b/pubsub/gossipsub/implementation-status.md @@ -1,47 +1,74 @@ # Implementation status of Gossipsub versions and Extensions -This doc is meant to provide an overview of the implementation status of -Gossipsub versions and Extensions. +This doc provides an overview of the implementation status of Gossipsub +versions and Extensions across all major libp2p implementations. + +Legend: ✅ = complete, 🏗 = in progress, 🔄 = spec draft / PR open, ❔ = not started ## Gossipsub Versions -| | [1.2] | [1.3-alpha] | -| ------------- | ----- | -------------------------------------------------------------- | -| [Go libp2p] | ✅ | [Open PR](https://github.com/libp2p/go-libp2p-pubsub/pull/630) | -| [Rust libp2p] | ✅ | In Progress | -| [JS libp2p] | ✅ | Not started | -| [Nim libp2p] | ✅ | Not started | -| [Java libp2p] | ✅ | Not started | +| | [v1.0] | [v1.1] | [v1.2] | [v1.3-alpha] | +| ------------- | :----: | :----: | :----: | --------------------------------------------------------------- | +| [Go libp2p] | ✅ | ✅ | ✅ | [Open PR](https://github.com/libp2p/go-libp2p-pubsub/pull/630) | +| [Rust libp2p] | ✅ | ✅ | ✅ | [In Progress](https://github.com/libp2p/rust-libp2p/pull/5878) | +| [JS libp2p] | ✅ | ✅ | ✅ | ❔ | +| [Nim libp2p] | ✅ | ✅ | ✅ | ❔ | +| [Java libp2p] | ✅ | ✅ | ✅ | ❔ | +| [py-libp2p] | ✅ | ✅ | ✅ | ✅ | + +> **Note on py-libp2p v1.3:** py-libp2p ships with full v1.3 Extensions Control +> Message support, including both the `ControlExtensions` handshake protocol and +> the Topic Observation extension. See +> [PR libp2p/py-libp2p#1323](https://github.com/libp2p/py-libp2p/pull/1323) for +> the Large Message Segmentation reference implementation. ## Gossipsub Extensions -| | [Choke Extensions] | [Partial Messages] | -| ------------- | ------------------ | --------------------------------------------------------- | -| [Go libp2p] | Not Implemented | [PR](https://github.com/libp2p/go-libp2p-pubsub/pull/631) | -| [Rust libp2p] | Not Implemented | Not Implemented | -| [JS libp2p] | Not Implemented | Not Implemented | -| [Nim libp2p] | Not Implemented | Not Implemented | -| [Java libp2p] | Not Implemented | Not Implemented | +| | [Partial Messages] | [Test Extension] | [Large Message Segmentation] | +| ------------- | --------------------------------------------------------- | ---------------- | ------------------------------------------------------ | +| [Go libp2p] | [PR](https://github.com/libp2p/go-libp2p-pubsub/pull/631) | ✅ | ❔ | +| [Rust libp2p] | ❔ | ❔ | ❔ | +| [JS libp2p] | ❔ | ❔ | ❔ | +| [Nim libp2p] | ❔ | ❔ | 🔄 [Spec Draft](https://github.com/seetadev/specs/pull/2) | +| [Java libp2p] | ❔ | ❔ | ❔ | +| [py-libp2p] | ❔ | ❔ | 🔄 [PR libp2p/py-libp2p#1323](https://github.com/libp2p/py-libp2p/pull/1323) | + +> Extensions are negotiated at connection time via the GossipSub v1.3 +> `ControlExtensions` handshake. See +> [extensions/README.md](extensions/README.md) for a guide to the extensions +> framework. ## Gossipsub Implementation Improvements | | [Batch Publishing] | [IDONTWANT on First Publish] | [WFR Gossip] | | ------------- | ------------------------------------------------------------------------ | --------------------------------------------------------- | --------------------------------------------------------- | | [Go libp2p] | [✅](https://pkg.go.dev/github.com/libp2p/go-libp2p-pubsub#MessageBatch) | [✅](https://github.com/libp2p/go-libp2p-pubsub/pull/612) | [PR](https://github.com/libp2p/go-libp2p-pubsub/pull/632) | -| [Rust libp2p] | Not Implemented | [✅](https://github.com/libp2p/rust-libp2p/pull/5773) | Not Implemented | -| [JS libp2p] | Not Implemented | Not Implemented | Not Implemented | -| [Nim libp2p] | Not Implemented | Not Implemented | Not Implemented | -| [Java libp2p] | Not Implemented | Not Implemented | Not Implemented | - -[Go libp2p]: https://github.com/libp2p/go-libp2p-pubsub -[Rust libp2p]: https://github.com/libp2p/rust-libp2p/tree/master/protocols/gossipsub -[JS libp2p]: https://github.com/ChainSafe/js-libp2p-gossipsub -[Nim libp2p]: https://github.com/vacp2p/nim-libp2p/tree/master/libp2p/protocols/pubsub/gossipsub -[Java libp2p]: https://github.com/libp2p/jvm-libp2p/tree/develop/libp2p/src/test/kotlin/io/libp2p/pubsub/gossip -[1.2]: https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md -[1.3-alpha]: https://github.com/libp2p/specs/issues/687 -[Choke Extensions]: https://github.com/libp2p/specs/pull/681 -[Partial Messages]: https://github.com/libp2p/specs/pull/685 -[Batch Publishing]: https://ethresear.ch/t/improving-das-performance-with-gossipsub-batch-publishing/21713 -[IDONTWANT on first Publish]: https://github.com/libp2p/go-libp2p-pubsub/issues/610 -[WFR Gossip]: https://ethresear.ch/t/the-paths-of-least-resistance-introducing-wfr-gossip/22671/3 +| [Rust libp2p] | ❔ | [✅](https://github.com/libp2p/rust-libp2p/pull/5773) | ❔ | +| [JS libp2p] | ❔ | ❔ | ❔ | +| [Nim libp2p] | ❔ | ❔ | ❔ | +| [Java libp2p] | ❔ | ❔ | ❔ | +| [py-libp2p] | ❔ | ✅ | ❔ | + +## Reference links + +| Key | Implementation | Repository | +| --- | -------------- | ---------- | +| [Go libp2p] | Golang | | +| [Rust libp2p] | Rust | | +| [JS libp2p] | JavaScript | | +| [Nim libp2p] | Nim | | +| [Java libp2p] | Java / Kotlin | | +| [py-libp2p] | Python | | + +| Key | Spec / Proposal | +| --- | --------------- | +| [v1.0] | | +| [v1.1] | | +| [v1.2] | | +| [v1.3-alpha] | | +| [Partial Messages] | | +| [Test Extension] | | +| [Large Message Segmentation] | | +| [Batch Publishing] | | +| [IDONTWANT on first Publish] | | +| [WFR Gossip] | |