Pipit

A domain-specific language for describing clock-driven, real-time data pipelines using Synchronous Dataflow (SDF) semantics on shared memory.

⚠️ Work in Progress: Pipit is under active development. The core compiler pipeline and runtime are functional, and performance/stdlib expansion is in progress. See ROADMAP.md for the development roadmap.

What it does

• Define actors in C++ with static input/output token rates
• Describe pipelines in .pdl files with a concise pipe-based syntax
• Compile to native executables via C++ code generation

source.pdl → pcc → source_gen.cpp → clang++ → executable

Example

const coeff = [0.1, 0.2, 0.4, 0.2, 0.1]
param gain = 1.0

clock 10MHz capture pace wall {
    adc(0) | mul($gain) | fft(256) | :raw | fir(coeff) | ?filtered -> signal
    :raw | mag() | stdout()
}

clock 1kHz drain pace wall {
    @signal | decimate(10000) | csvwrite("output.csv")
}

Quick Start

# Build the compiler (from repo root)
cargo build --release -p pcc

# Generate actor metadata manifest (required before compilation)
target/release/pcc --emit manifest \
  -I runtime/libpipit/include \
  -I examples \
  -o actors.meta.json

# Compile a pipeline to C++
target/release/pcc examples/gain.pdl \
  --actor-meta actors.meta.json \
  -I runtime/libpipit/include/std_actors.h \
  --emit cpp -o gain.cpp

# Build the executable manually
clang++ -std=c++20 -O2 gain.cpp \
  -I runtime/libpipit/include \
  -I runtime/libpipit/include/third_party \
  -lpthread -o gain

# Or compile directly to executable (one step, requires C++20)
target/release/pcc examples/gain.pdl \
  --actor-meta actors.meta.json \
  -I runtime/libpipit/include/std_actors.h \
  --cflags="-std=c++20 -O2" \
  -o gain

# Run with duration and stats
./gain --duration 10s --stats

Features

Language

• Multi-rate pipelines with automatic SDF balance solving
• Fork (:tap), probe (?name), and feedback loops (delay)
• Inter-task shared buffers (-> / @) with lock-free ring buffers
• Modal tasks with control / mode / switch for runtime mode switching
• Reusable sub-pipelines via define
• Compile-time constants and runtime parameters ($param)
• Multi-port actors with explicit port wiring (node declarations, node.port[idx] syntax)
• pace virtual / pace wall per-task pacing: virtual calendar (deterministic LCM-superframe scheduling) or wall-clock OS timer; cross-domain boundaries enforced at shared buffers

Compiler (pcc)

• Full pipeline: parse → resolve → HIR → type_infer → lower → graph → analyze → schedule → LIR → codegen
• Pass-manager orchestration with --emit-driven minimal evaluation
• Polymorphic actors with constraint-based type inference (actor<T>)
• Implicit safe widening: int8 → int16 → int32 → float → double, cfloat → cdouble
• Dimension mismatch diagnostics (explicit arg vs shape constraint vs span-derived conflicts)
• SDF balance verification, feedback delay validation
• Structured diagnostics (human / json) with stable diagnostic codes
• LIR edge memory classification (Local/Shared/Alias) with cache-line-aligned buffers

Runtime

• --duration with time suffixes (10s, 1m, inf)
• --param name=value for runtime parameter override
• --stats for per-task timing statistics
• --probe <name> / --probe-output <path> for data observation
• --threads <n> advisory runtime hint
• --release strips probes to zero cost
• --experimental enables experimental codegen features (reserved)

Standard Actors

• Split headers: std_actors.h, std_math.h, std_sink.h, std_source.h
• I/O: stdin, stdout, stderr, stdout_fmt, binread, binwrite
• Math: constant, mul, add, add2, sub, div, abs, sqrt, threshold, convolution
• Statistics: mean, rms, min, max
• DSP: fft, c2r, mag, fir, delay, decimate (PocketFFT + xsimd SIMD vectorization)
• Waveform generators: sine, square, sawtooth, triangle, noise, impulse
• External I/O: socket_write, socket_read (UDP/IPC via PPKT protocol)

Tools

• pipscope — Real-time oscilloscope GUI (ImGui + ImPlot) receiving PPKT packets via UDP
• pipintegrator — Visual pipeline editor GUI (ImGui + imnodes) with node canvas, actor library, build runner, and project I/O

Output Modes

• --emit exe — Generate and compile executable (default)
• --emit cpp — Emit generated C++ source
• --emit ast — Dump parsed AST
• --emit graph — Dump analyzed graph view
• --emit manifest — Actor metadata JSON (hermetic builds, no .pdl required)
• --emit build-info — Provenance JSON (source hash, registry fingerprint)
• --emit graph-dot — Graphviz DOT dataflow graph
• --emit timing-chart — Mermaid Gantt scheduling diagram
• --emit schedule — PASS firing order

Multi-Port Wiring

Declare actor instances with node and wire ports explicitly:

clock 48kHz demo {
    node sum = add()
    adc(0) | sum.in[0]
    adc(1) | sum.in[1]
    sum.out[0] | stdout<float>()
}

Port reference forms: element [idx], bundle [*], span [begin:end], arity-1 shorthand. See ADR-038 and spec §5.5.1 for details.

Project Structure

compiler/       Rust compiler (pcc)
  src/            parse → resolve → HIR/THIR/LIR → analyze/schedule → codegen
  tests/          unit + integration + end-to-end coverage
runtime/        C++ runtime library (libpipit)
  libpipit/       Ring buffer, timer, statistics, networking (PPKT)
  tests/          C++ unit tests for actors and runtime components
tools/          Standalone tools
  pipscope/       Real-time oscilloscope GUI (ImGui + ImPlot)
  pipintegrator/  Visual pipeline editor GUI (ImGui + imnodes)
examples/       Example .pdl files and actor headers
  benches/        Additional PDL examples (non-KPI benchmarks)
doc/
  spec/           Language spec, compiler spec, stdlib, protocol specs
    archive/      Archived prior-version specs
  guides/         Usage guide, migration guide
  adr/            Architecture Decision Records (ADR-001 – ADR-040)
  performance/    Performance reports, profiling protocol, trend graphs
benches/        Performance benchmarks (compiler, runtime, E2E)
  src/            Benchmark source code (Rust Criterion, C++ Google Benchmark)
  pdl/            KPI benchmark pipelines
  scripts/        Internal benchmark tools (characterize, stable, profile, A/B)

Performance

See Performance Reports for details.

Documentation

• Language Spec
• Compiler Spec
• pcc Usage Guide
• Migration Guide v0.4.x → v0.5.0
• PPKT Protocol Spec
• Standard Library Reference
• pcc Performance KPI/Test Spec
• Multi-Port Extension (ADR-038)
• Pace Virtual/Wall Semantics (ADR-040)
• Development Roadmap

License

MIT — see LICENSE.