feat: add support symmetric configurations

[meling]

Symmetric Configuration Design Document

• Date: 2026-02-21
• Authors: Hein Meling

---

Executive Summary

This document proposes architectural changes to Gorums that enable symmetric use of Configuration and Node abstractions on both client and server sides. This allows server replicas to communicate with connected clients (and other replicas) using the same API that clients use to communicate with servers.

Key benefits:

• Replicas can broadcast to connected clients without separate outbound connections
• Enables all-to-all communication patterns for distributed protocols
• Simplifies implementation of consensus protocols like Paxos

---

Motivation

Current Limitation

In the current Gorums architecture:

• Clients create a Manager → Configuration → Node hierarchy to communicate with server replicas
• Servers can only respond to client requests; they cannot initiate communication

This creates awkwardness when implementing protocols like Paxos where each replica must communicate with all other replicas:

// Current: Each replica runs BOTH a client and server
func NewPaxosReplica(myAddr string, peerAddrs []string) *Replica {
    // Server side
    srv := gorums.NewServer()
    lis, _ := net.Listen("tcp", myAddr)

    // Client side - creates N-1 OUTBOUND connections
    mgr := gorums.NewManager(...)
    cfg, _ := gorums.NewConfiguration(mgr, gorums.WithNodeList(peerAddrs))

    // Problem: peers also create outbound connections to us
    // Result: 2*(N-1) connections instead of N-1
}

Proposed Solution

Enable servers to obtain a Configuration of connected clients, using the same API as clients:

// Proposed: Server can use Configuration of connected clients
func (s *PaxosServer) BroadcastPrepare(slot uint64) {
    cfg := s.sys.InboundConfig()
    cfgCtx := cfg.Context(context.Background())
    pb.Multicast(cfgCtx, &PrepareMsg{Slot: slot})
}

---

Architecture

Current Architecture

The Channel is an exported type in internal/stream.
The wire protocol is defined in stream.proto using a single bidi NodeStream per node.
All RPC methods are multiplexed over the stream, identified by the method field.

graph TB
    subgraph "gorums package — Client Side"
        Manager --> Config[Configuration]
        Config --> Node1["Node"]
        Config --> Node2["Node"]
        Node1 -->|"*stream.Channel"| Ch1[Channel]
        Node2 -->|"*stream.Channel"| Ch2[Channel]
        Ch1 --> Conn1["grpc.ClientConn"]
        Ch2 --> Conn2["grpc.ClientConn"]
    end

    subgraph "gorums package — Server Side"
        Server --> SS[streamServer]
        SS -->|"implements"| GRPC
        SS --> SrvStream["Gorums_NodeStreamServer"]
    end

    subgraph "internal/stream package"
        Channel["Channel (exported)"]
        Request["Request struct"]
        Message["stream.Message (protobuf)"]
        NodeResponse["NodeResponse[T]"]
        GRPC["Gorums gRPC service"]
    end

    Ch1 -.->|"bidi NodeStream"| SrvStream
    Ch2 -.->|"bidi NodeStream"| SrvStream

Loading

Proposed Architecture (symmetric configuration)

The key addition is an InboundManager struct that manages server-side peer state independently from Server.
Inbound channels are created from server-side streams, and the manager provides a live Configuration.

graph TB
    subgraph "internal/stream package"
        OutCh["Channel (outbound)"]
        InCh["Channel (inbound)"]
        BidiStream["BidiStream interface"]
        OutCh -->|"implements sender/receiver"| BidiStream
        InCh -->|"implements sender/receiver"| BidiStream
    end

Loading

graph TB
    subgraph "gorums package — Replica A"
        direction TB
        ServerA["Server"] --> SSA["streamServer"]
        ServerA -->|"embeds"| IMA["InboundManager"]
        IMA --> InboundCfgA["Configuration (auto-updated)"]
        InboundCfgA --> InNodeB["Node (inbound from B)"]
        InboundCfgA --> InNodeC["Node (inbound from C)"]
        InNodeB --> InChB["Channel (inbound)"]
        InNodeC --> InChC["Channel (inbound)"]

        MgrA["Manager"] --> CfgA["Configuration (outbound)"]
        CfgA --> OutNodeB["Node (outbound to B)"]
        CfgA --> OutNodeC["Node (outbound to C)"]
        OutNodeB --> OutChB["Channel (outbound)"]
        OutNodeC --> OutChC["Channel (outbound)"]
    end

    SSA -.->|"receives inbound streams"| InChB
    SSA -.->|"receives inbound streams"| InChC
    OutChB -.->|"connects to"| ReplicaB["Replica B Server"]
    OutChC -.->|"connects to"| ReplicaC["Replica C Server"]

Loading

Key components after symmetric configuration:

Component	Responsibility
Manager	Outbound node pool, dial options, message ID generation
Configuration	Immutable slice of *Node — works identically for outbound & inbound
Node	ID, address, channel reference — identical for both directions
Channel	Send queue, stream lifecycle, response routing (outbound or inbound)
Server	gRPC server, request dispatch, handler registration
InboundManager	Tracks connected peers, provides auto-updated Configuration
System	Lifecycle management: combines Server + listener + closers

---

Design Decisions

Decision	Choice	Rationale
Peer identity	peer.FromContext + gorums-addr metadata	Avoids extra proto field overhead on every message
Peer tracking	Separate InboundManager embedded in Server	Clean separation of concerns; server-side peer state decoupled
Configuration management	Gorums auto-manages server-side configs	Reduces boilerplate; callback hooks for custom needs
Symmetric stream handling	Connect-first, deterministic tiebreaker	Avoids duplicate connections in peer-to-peer setups
Configuration mutability	Auto-update on connect/disconnect	Reflects reality; users get live view of connections
NodeID 0	Reserved for external (non-replica) clients	Replica IDs start at 1; external clients are untracked
Inbound vs outbound	IsInbound() checks conn == nil	No extra field; conn is the natural discriminator
Channel abstraction	BidiStream interface for outbound/inbound	Reuses sender/receiver goroutines; only stream creation differs
System integration	System infers server address for outbound configs	No manual WithServerAddress; address automatically included
Message ID generation	InboundManager owns server-side nextMsgID	Independent from Manager to avoid coupling client and server counters
Initialization	InboundNodeOption for InboundManager constructor	Separate interface from NodeListOption; no interface modification
Identity code placement	Single client_identity.go file	gorumsAddrKey, identifyPeer co-located in one place
Stream handling	handleStream encapsulates identify+register logic	NodeStream handler calls one method; returns cleanup function

---

Implementation Plan

Phase	Description	Document	Breaking Changes
1	BidiStream interface + inbound channel	Phase 1	None
2	InboundManager + peer identity + InboundNodeOption	Phase 2	None
3	System integration, address inference, symmetric tests	Phase 3	None

Each phase is designed to be reviewed and committed independently.
All existing tests must pass after each phase.

---

API Changes Summary

New APIs

API	Phase	Description
stream.BidiStream	1	Interface abstracting client and server bidi streams
stream.NewInboundChannel(...)	1	Creates a channel from an existing server stream
(*Channel).IsInbound()	1	Returns true if channel has no grpc.ClientConn
InboundNodeOption	2	Interface for server-side node specification
NewInboundManager(myID, opt)	2	Creates inbound manager using InboundNodeOption
(*InboundManager).InboundConfig()	2	Returns Configuration of connected peers
(*InboundManager).handleStream(ctx, srv)	2	Identifies peer, registers, returns cleanup func
WithInboundManager(im)	2	Server option to enable peer tracking
identifyPeer(ctx)	2	Extracts and validates claimed address from context
NewSymmetricSystem(addr, myID, opt, opts)	3	Creates a System with built-in InboundManager
(*System).NewOutboundConfig(opts)	3	Creates outbound config with server address auto-set
(*System).InboundConfig()	3	Returns inbound configuration (delegates to manager)

Modified Behavior

API	Phase	Change
Channel struct	1	stream field uses BidiStream type; conn nil for inbound
Channel.ensureStream()	1	Inbound channels cannot reconnect; returns ErrStreamDown
Channel.Close()	1	Inbound channels do not close grpc.ClientConn
Server NodeStream handler	2	Calls handleStream when InboundManager is present
Outbound connections	3	Include gorums-addr metadata when created via System

Backward Compatibility

All changes are backward compatible:

• Existing client code unchanged (external clients are untracked, get ID = 0)
• Existing server code unchanged (WithInboundManager is opt-in)
• System retains existing NewSystem, RegisterService, Serve, Stop API
• Channel outbound behavior unchanged; inbound is new functionality
• NodeListOption interface is not modified; InboundNodeOption is a separate interface