Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ jobs:
label: benchmarks
configure_args: "--with=benchmarks"

# macOS (Apple Silicon)
- os: macos-14
label: macos-tests
configure_args: "--with=unit_tests --with=examples"

# Windows
- os: windows-latest
label: windows
Expand All @@ -40,8 +45,8 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup-conan

- name: Configure and build (Linux)
if: runner.os == 'Linux'
- name: Configure and build (POSIX)
if: runner.os != 'Windows'
run: |
scripts/configure.sh build ${{ matrix.configure_args }}
scripts/setup_build.sh build --build-type ${{ env.BUILD_TYPE }}
Expand Down
147 changes: 147 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Agent guide -- kickmsg

Project-wide directives for any LLM working in this repository. Read this
before proposing changes. These override generic defaults. kickmsg is a
lock-free, shared-memory pub/sub library for inter-process messaging on
POSIX and Windows; correctness under concurrency, crashes, and corrupt
peer memory is the whole point.

## 1. Don't be lazy

If a fix takes 2--3 lines and visibly improves the code, write the 2--3 lines.
Do not dismiss real concerns with "premature optimization", "good enough",
"out of scope", or "we can revisit later" when the work in front of you is
small and the benefit is concrete. If a reviewer points at a problem, they
already paid the cost of finding it -- apply the fix instead of arguing why it
doesn't matter. When the right fix is genuinely larger than the local change,
do the local fix *and* call out the broader work as a follow-up.

## 2. Keep It Simple, Stupid (KISS)

Pick the simplest design that solves the problem. Avoid helpers called from
exactly one place (inline them), mirrors of state another component owns
(query the authority), dead defensive checks on values a constructor already
guarantees, and speculative abstractions with no concrete consumer. Three
similar lines beat a premature abstraction; factor only when the same shape
repeats 3+ times under the same constraints.

## 3. Maintainability first, performance second

Maintainable code that is also performant beats clever code that is hard to
follow -- but "maintainable first" does not mean "ignore performance". The
publish and receive paths are hot: per-publish heap allocations, avoidable
atomics, false sharing, and syscalls on the fast path are real costs. When the
maintainable design and the performant one diverge by a few lines (a scratch
buffer, a conditional wake), apply both.

Order of operations: (1) correct, (2) clear, (3) fast on hot paths.

## 4. Don't reinvent the wheel

Grep before writing new infrastructure. What already exists:

- Shared memory: `kickmsg/os/SharedMemory.h` (create/open/try_*/unlink).
- Region + channel: `kickmsg/Region.h` (`SharedRegion`, `channel::Config`,
recovery primitives, `validate_header_geometry`).
- Hashing: `kickmsg/Hash.h` -- `kickmsg::hash::fnv1a_64`. Don't add a new hash.
- Shm naming: `kickmsg/Naming.h` -- `sanitize_shm_component`,
`compose_shm_name`, `to_hex`. All shm names go through these.
- Lock-free helpers: `kickmsg/types.h` -- `treiber_push`/`treiber_pop`,
`tagged_pack`/`tagged_idx`, `slot_at`/`sub_ring_at`/`ring_entries`,
`align_up`, `is_power_of_two`.
- OS: `kickmsg/os/Time.h` (`since_epoch`, `sleep`), `kickmsg/os/Process.h`
(`current_pid`, `process_starttime`), `kickmsg/os/Futex.h`
(`futex_wait`/`futex_wake_all`).
- Tests: GoogleTest + gmock unit tests (`tests/unit/`), the stress harness
(`tests/stress/common.h`), fork+SIGKILL crash tests (`tests/crash_test.cc`).

"I didn't know it existed" is not an answer -- grep, find, or Explore first.

## 5. Coding style

There is no `.clang-format` yet; the rules below describe the established
style. Match the surrounding file.

### 5.1 Formatting

- **Allman braces** -- opening brace on its own line for every block.
- **4-space indent, no tabs.** Case labels indented inside `switch`.
- **East const** -- `T const&`, `T const*`, `Header const*`. Not `const T&`.
- **Pointer alignment left** -- `T* p`, not `T *p`.
- **Member naming** -- `snake_case` with a trailing underscore for non-public
data members (`shm_`, `name_`, `base_`). Public/struct fields: no underscore.
- **Type naming** -- `PascalCase` for classes, structs, enums, aliases.
- **Namespace** -- single `namespace kickmsg`, contents indented, no closing
`// namespace` comment.
- **Constructor initializers** -- one per line, `,` leading the next line.

### 5.2 Language rules

- **Header guards, never `#pragma once`.** Format `KICKMSG_<PATH>_H` (e.g.
`kickmsg/os/SharedMemory.h` -> `KICKMSG_OS_SHARED_MEMORY_H`). Mirror an
adjacent header.
- **No ternary operator** in code you add or modify. Rewrite as `if`/`else`,
early return, or an `if`-assigned variable. Leave pre-existing ternaries
unless explicitly cleaning them up.
- **Prefer `not` / `and` / `or`** over `!` / `&&` / `||` -- the codebase uses
the keyword operators throughout.
- **`override` mandatory** on overriding virtuals.
- **`inline` only with a real rationale** (one-line accessors, `constexpr`,
templates, header-only by design) -- not merely to dodge a `.cc` file.

### 5.3 Concurrency and shared memory (kickmsg-specific)

- **Explicit `memory_order` on every atomic op.** No bare `.load()`/`.store()`.
When an op is `relaxed`, a one-line comment must say why it is safe (which
other fence or release-store covers it). The release-store of `MAGIC` is the
sole publication fence for a freshly-stamped region -- do not "fix" a
deliberate relaxed store to release in isolation.
- **Tolerate corrupt / hostile peer bytes.** The region is mapped by
independent processes and can be handed to `attach_open` by a caller. Any
field read from shared memory that drives pointer math (offsets, strides,
indices, lengths, counts) must be bounds-checked before use -- see
`validate_header_geometry`. A crashed peer can leave partial state.
- **Preserve the recovery contract.** `repair_locked_entries` and `stats` /
`diagnose` are safe under live traffic; `reset_retired_rings`,
`reclaim_orphaned_slots`, `reset_schema_claim`, `sweep_stale` are post-crash
only. Keep that distinction in the code and the docs; don't add a
live-traffic caller of a post-crash primitive.
- **ABI is versioned.** The shm layout (`Header`, `Entry`, `SubRingHeader`,
`SlotHeader`, `SchemaInfo`) is guarded by `MAGIC` + `VERSION`. Any layout
change requires a `VERSION` bump and updating the `static_assert`s in
`types.h`.

### 5.4 Comments

Default to writing nothing. Only add a comment when:

- A hidden constraint or invariant is being maintained.
- The code works around a specific bug or platform quirk.
- A reader would otherwise be surprised by the behavior (especially a memory
ordering or crash-recovery subtlety -- those are worth a tight line).

Never restate what the code does. Never reference the current PR / issue /
"this fixes...". One short line beats a paragraph; reserve multi-line blocks
for an ASCII diagram or numbered list that genuinely helps (e.g. the publish
flow in ARCHITECTURE.md).

### 5.5 Includes

- **Library headers**: IWYU strictly -- every type named in the public API
comes from an include in that header.
- **`.cc` files**: group standard headers and internal headers, separated by a
blank line. Match the file's existing grouping.
- **Tests, examples, callers**: rely on transitive includes from the public
headers you use. Don't re-include `<vector>`, `<cstring>`, `<chrono>`,
`<cstdint>`, etc. when they arrive via `kickmsg/Publisher.h`,
`kickmsg/Region.h`, `kickmsg/types.h`, and friends.

### 5.6 Other defaults

- **ASCII only in source** (code, comments, identifiers, docs): `--` for an
em-dash, `'` for an apostrophe, `...` for an ellipsis, plain ASCII quotes.
Exception: user-facing strings that genuinely need otherwise. Legacy files
predating this rule still contain em-dashes; leave them unless you are
already editing that line, and write new comments ASCII.
- **No emojis** in code, comments, commit messages, or docs unless asked.
- **No new docs files** (`.md`, `README`) unless the user asks.
97 changes: 78 additions & 19 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,18 @@ free_top [gen:17 | idx:3]
- **Push** (release a slot): CAS `free_top` from `[gen|head]` to
`[gen+1|slot]` after setting `slot.next = head`.

**Accepted ABA bound.** The generation is 32-bit, so the ABA window is not
formally closed: it requires a thread to read `free_top`, stall, and then
have exactly 2^32 push/pop operations complete (returning the head to the
same `[gen|idx]`) before it resumes its CAS. At realistic pool churn that is
on the order of tens of seconds of uninterrupted contention landing inside a
single preemption of one thread -- astronomically unlikely, and no real
workload approaches it. We accept this rather than pay for a 128-bit DWCAS
(`cmpxchg16b` / `LDXP`-`STXP`), which is not lock-free in the `std::atomic`
sense on every target and costs on the hot alloc/free path. If a future
target makes the window reachable, widen `free_top` to a 128-bit tagged
pointer (a `VERSION` bump).


## Subscriber Ring

Expand Down Expand Up @@ -510,18 +522,18 @@ Publisher
| in_flight == 0.
|
'-- if has_waiter: Conditional wake: skip the syscall
futex_wake_all(write_pos) when no subscriber is blocking.
has_waiter uses relaxed ordering on
both sides (publisher load, subscriber
store). This can race: the publisher
may not see has_waiter=1 if the
subscriber just set it. But the race
is benign — futex_wait(write_pos, cur)
checks *addr != expected atomically
in the kernel; if write_pos already
advanced, it returns immediately.
Worst case: one unnecessary round-trip
through futex (not a lost wake).
futex_wake_all(write_pos) when no subscriber is blocking. A
seq_cst fence precedes the has_waiter
load and pairs with one before the
subscriber's futex_wait, ordering the
write_pos fetch_add ahead of the load.
This is a Dekker StoreLoad pair: with
relaxed ordering alone a weakly-ordered
CPU could read has_waiter == 0 stale
and drop the wake to an already-parked
subscriber (lost wakeup until timeout).
x86's locked RMW fences implicitly;
ARM does not, hence the explicit fence.

5. Batch excess: fetch_sub(excess) on slot refcount.
One atomic RMW for all non-delivered rings, instead of N
Expand Down Expand Up @@ -1000,12 +1012,17 @@ auto report = region.diagnose();
// report.locked_entries: entries stuck at LOCKED_SEQUENCE
// report.retired_rings: Free rings with stale in_flight > 0
// report.draining_rings: Draining rings with in_flight > 0 (usually transient)
// report.dead_rings: Live/Draining rings whose owner process is gone
// report.live_rings: active subscriber rings

// Safe under live traffic — repairs poisoned ring entries.
// Can be called freely on a health-check timer.
std::size_t repaired = region.repair_locked_entries();

// Reclaims rings whose owning subscriber process has died. Safe under
// live traffic: only provably-dead owners (pid + start time) are touched.
std::size_t dead = region.reclaim_dead_rings();

// Resets retired rings (Free | in_flight>0) so new subscribers can
// claim them. Only safe after confirming the crashed publisher is gone.
// Deliberate post-crash action, not a routine maintenance call.
Expand All @@ -1023,10 +1040,21 @@ periodically; persistent nonzero counts signal recovery is needed.
`INVALID_SLOT`. Safe under live traffic (benign double-store if a
slow publisher commits at the same time). Can run on a timer.

**`reclaim_dead_rings()`** — reclaims `Live`/`Draining` rings whose
owning subscriber process has died, identified by the `owner_pid` +
`owner_starttime` recorded on the ring at claim time and checked
against the OS (same liveness test as `Registry::sweep_stale`). Safe
under live traffic: a slow-but-alive subscriber is never touched.
`in_flight` is preserved (a mid-commit publisher must still
`fetch_sub`), so a reclaimed ring with leftover `in_flight` lands in
the retired state for `reset_retired_rings()` to finish.

**`reset_retired_rings()`** — resets stuck rings (`Free | in_flight>0`
→ `Free | in_flight=0`). Only safe after confirming the crashed
publisher is gone. Unlike `repair_locked_entries()`, this is a
deliberate post-crash action.
deliberate post-crash action. For a dead *subscriber* prefer
`reclaim_dead_rings()`, which is liveness-checked and cannot underflow
`in_flight` against a slow publisher.

**`reclaim_orphaned_slots()`** — walks all rings to build a
referenced-slot set, then frees any unreferenced slot with
Expand All @@ -1038,14 +1066,45 @@ quiesced and no outstanding `SampleView` objects.
```
1. diagnose() → persistent nonzero counts (check twice, gap > commit_timeout)
2. repair_locked_entries() — safe under live traffic
3. reset_retired_rings() — after confirming crashed publisher is gone
4. (optional) pause all publishers
5. reclaim_orphaned_slots() — requires quiescence
6. resume publishers
3. reclaim_dead_rings() — safe under live traffic (dead subscribers)
4. reset_retired_rings() — after confirming crashed publisher is gone
5. (optional) pause all publishers
6. reclaim_orphaned_slots() — requires quiescence
7. resume publishers
```

Steps 4–6 are only needed if the pool is exhausted from leaked slots.
In most cases, steps 2–3 restore the channel to full operation.
Steps 5–7 are only needed if the pool is exhausted from leaked slots.
In most cases, steps 2–4 restore the channel to full operation.

### Recovery limits and residual windows

The recovery primitives close the common crash classes but have a few
documented edges, all bounded:

- **`reclaim_orphaned_slots()` is not crash-safe mid-pass.** It resets a
slot's refcount to 0 and pushes it to the free stack as two separate
stores. If the *recovering* process is killed between them, the slot is
neither referenced nor free-listed (refcount 0), and a re-run skips it
(the `refcount > 0` guard no longer matches) — a permanent leak of that
one slot. Run recovery from a supervisor that is not itself under the
OOM/kill pressure that triggered recovery.

- **`reset_retired_rings()` trusts the operator.** It zeroes `in_flight`;
if called while a misclassified slow-but-alive publisher still owes a
`fetch_sub`, that decrement underflows the packed counter into the state
bits. It is a post-crash-only action. For a dead *subscriber*,
`reclaim_dead_rings()` is the liveness-checked alternative and avoids
this entirely.

- **Claim-window residual (rings and registry).** Liveness recovery keys on
an owner identity (`owner_pid` for rings, `pid` for registry entries)
recorded *after* the claiming CAS. A process that crashes in the few
instructions between winning the CAS and recording its pid leaves the
slot owned by pid 0 — unattributable, so neither `reclaim_dead_rings()`
nor `sweep_stale()` reclaims it. The window is a handful of instructions
and closing it fully would need a separate intent log; in practice the
slot is lost only on a crash landing in that exact gap, and the pool /
registry tolerate many such losses before exhaustion.


## ABA Safety
Expand Down
2 changes: 2 additions & 0 deletions examples/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ add_executable(hello_pubsub hello_pubsub.cc)
add_executable(hello_zerocopy hello_zerocopy.cc)
add_executable(hello_broadcast hello_broadcast.cc)
add_executable(hello_lowlevel hello_lowlevel.cc)
add_executable(hello_inject hello_inject.cc)
add_executable(hello_diagnose hello_diagnose.cc)
add_executable(hello_schema hello_schema.cc)
add_executable(hello_schema_late_publisher hello_schema_late_publisher.cc)
Expand All @@ -10,6 +11,7 @@ target_link_libraries(hello_pubsub PRIVATE kickmsg)
target_link_libraries(hello_zerocopy PRIVATE kickmsg)
target_link_libraries(hello_broadcast PRIVATE kickmsg)
target_link_libraries(hello_lowlevel PRIVATE kickmsg)
target_link_libraries(hello_inject PRIVATE kickmsg)
target_link_libraries(hello_diagnose PRIVATE kickmsg)
target_link_libraries(hello_schema PRIVATE kickmsg)
target_link_libraries(hello_schema_late_publisher PRIVATE kickmsg)
4 changes: 1 addition & 3 deletions examples/hello_broadcast.cc
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,11 @@
/// - Multiple concurrent participants on the same channel
/// - Each node sees messages from all other nodes (not its own)

#include <cstring>
#include <iostream>
#include <string>
#include <thread>
#include <vector>

#include <kickmsg/Node.h>
#include <kickmsg/os/Time.h>

using namespace kickmsg;

Expand Down
2 changes: 0 additions & 2 deletions examples/hello_diagnose.cc
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@
/// 4. Repair with repair_locked_entries()
/// 5. Verify the channel is fully operational again

#include <cstdint>
#include <cstring>
#include <iostream>

#include <kickmsg/Publisher.h>
Expand Down
Loading
Loading