Pipeline Stage Parallel Execution Engine with Dependency DAG Resolver

[github-actions]

Strategic Improvement

Reduce average pipeline duration from 90min → 60min by executing independent stages concurrently. Build dependency DAG resolver that analyzes stage inputs/outputs and executes non-dependent stages in parallel. Start with safe pairs: intake+intelligence analysis, test+security-audit.

Acceptance Criteria

• Dependency graph analyzer determines stage execution order and parallelizable groups
• Parallel execution harness spawns concurrent stages in separate tmux panes/worktrees
• Stage results aggregated atomically before dependent stages begin
• Config flag parallel_execution.enabled with conservative defaults (2 max concurrent initially)
• Metrics: average pipeline duration reduction, stage overlap time, failure rate comparison
• Documentation of which stage combinations are safe to parallelize

Context

• Priority: P0
• Complexity: full
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate + P3: Cost Efficiency

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	full
Max Iterations	30
Triage	Core pipeline performance optimization (90→60 min) via DAG-driven parallel execution; impacts all 14 stages, requires careful state aggregation, extensive stage-combination testing.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	25594985125

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (319 lines)

Plan: Pipeline Stage Parallel Execution Engine with Dependency DAG Resolver

Issue: #484 · Template: standard · Stage: plan
Goal: Reduce avg pipeline duration 90→60min by running independent stages concurrently.

---

1. Brainstorming / Design Refinement

Requirements Clarity (MVP)

The minimum viable change is a DAG resolver + a 2-way parallel harness gated by config, applied to two safe pairs initially:

• intake ∥ intelligence pre-analysis (read-only, no shared writes)
• test ∥ security-audit (both consume build artifacts, neither mutates the working tree)

Implicit requirements not stated:

• Must remain a no-op when parallel_execution.enabled=false (default off — conservative).
• Must not break pipeline resume (resume from pipeline-state.md still operates on the linear stage list).
• Must continue to emit stage.started/stage.completed events with the existing schema (dashboards, vitals, DORA, model-router all consume them).
• Must not double-charge cost ledger when stages run concurrently.
• Stage exit codes must aggregate atomically: any failed parallel sibling fails the whole group.

Acceptance criteria (made testable)

1. dag_groups produces deterministic groupings given a stage list + whitelist (unit-testable).
2. With parallel_execution.enabled=true and the seed pairs, mock pipeline emits two stage.started events within <2s of each other.
3. Failure of either parallel sibling marks both as failed and aborts subsequent dependent stages.
4. Default config (flag off) reproduces today's serial event stream byte-for-byte in sw-pipeline-test.sh.
5. Documentation lists exactly which (stage_a, stage_b) pairs are whitelisted and why.

Design Alternatives

A. In-process bash backgrounding with wait + per-stage tmp dir (chosen)

• Each parallel stage runs as ( run_stage_with_retry "$id" ) & with stdout redirected to pipeline-artifacts/parallel/<stage>.log.
• A parallel_group_run helper waits for all PIDs, collects exit codes via status files, then merges logs deterministically.
• Pros: no new processes/tools, fits the existing bash 3.2 architecture, easiest rollback (just unset flag).
• Cons: shared shell state can leak between siblings. Mitigated by running each branch in its own subshell ( ... ) &.

B. tmux-pane parallelism (one pane per concurrent stage)

• Pros: visible to operators using pipeline attach; isolation is real.
• Cons: complicates state aggregation (scrape pane output), tmux is optional, adds CI flakiness.

C. Worktree-per-stage parallelism

• Pros: filesystem isolation, allows mutating stages to parallelize too.
• Cons: massive blast radius — must commit/merge artifacts back; overkill for the read-only stages we want to start with.

Decision: A for the first cut (read-only / artifact-consumer pairs). Leave seams so B/C can be plugged in via a PARALLEL_BACKEND dispatch later (Open/Closed).

Risks

• Hidden writes to .claude/pipeline-artifacts/ by a stage thought read-only → corrupted artifacts. Mitigation: each branch writes only under parallel-artifacts/<stage>/; merger rejects writes outside that dir.
• Event ordering: dashboards expect monotonic stage.started per pipeline. Mitigation: keep emitting events in real time but include parallel_group_id and group_index fields; downstream tools that don't understand them ignore them.
• Cost double counting: token ledger sums by stage. Already per-stage; group total = sum of children (correct).

✅ Task Checklist

• T1: Add parallel_execution block to .claude/daemon-config.json and _smart_bool parallel_enabled helper in config.sh.
• T2: Implement scripts/lib/parallel-dag.sh (whitelist, grouping, validation). Blocks T4, T6.
• T3: Implement scripts/lib/parallel-runner.sh (spawn, wait, atomic aggregate, event emission, INT/TERM trap). Blocks T4, T7.
• T4: Wire group dispatch into scripts/lib/pipeline-execution.sh main loop, preserving the flag-off path bit-for-bit. Depends on T2, T3.
• T5: Add pipeline.parallel.savings_s metric + parallel_group_id field to per-stage events.
• T6: Write scripts/sw-parallel-dag-test.sh (≥6 cases). Depends on T2.
• T7: Write scripts/sw-parallel-runner-test.sh (≥6 cases). Depends on T3.
• T8: Add flag-off byte-identical regression assertion to scripts/sw-pipeline-test.sh. Depends on T4.
• T9: Write docs/parallel-execution.md with safe-pair rationale & expansion procedure.
• T10: Update .claude/CLAUDE.md Pipeline Stages section.
• T11: Register both new test scripts in package.json test script.
• T12: Set VERSION= in new scripts in sync with package.json.
• T13: Run npm test and shipwright doctor; confirm no regressions.
• All 6 issue acceptance criteria met (DAG analyzer, parallel harness, atomic aggregation, config flag, metrics, docs).
• All existing tests still pass (npm test).
• New unit tests added and passing.
• Flag-off path produces byte-identical event stream vs. baseline.
• dag_validate runs at pipeline start; invalid whitelist aborts before any stage.
• docs/parallel-execution.md lists seed safe pairs with rationale and expansion procedure.
• Bash 3.2 compatible (no declare -A, no readarray, no case-mutation expansions).

---

Generated by shipwright pipeline at 2026-05-09T07:22:54Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (658 lines)

Architecture Decision Record: Pipeline Stage Parallel Execution Engine with Dependency DAG Resolver

Context

Problem: Shipwright's sequential 14-stage pipeline takes ~90 minutes on average. Several stage pairs are logically independent (e.g., intake and intelligence pre-analysis both read-only; test and security_audit both consume build artifacts). Running these pairs concurrently can reduce total duration to ~60 minutes (33% improvement).

Constraints:

• Bash 3.2 compatible (no associative arrays, no readarray)
• Must remain a no-op when disabled (conservative default: parallel_execution.enabled=false)
• Must not break pipeline resume (linear stage list in pipeline-state.md still the source of truth)
• Existing event schema must remain compatible (dashboards, DORA, vitals, cost ledger all consume stage.started/stage.completed)
• Parallel stages must be whitelisted (no automatic detection — human validation required)
• No mutating stages should run in parallel in this first iteration (read-only pairs only)

Codebase specifics:

• Pipeline execution lives in scripts/lib/pipeline-execution.sh (main loop ~line 760+)
• Stage execution via run_stage_with_retry with idempotent record_model_outcome, mark_stage_complete, emit_event helpers
• Configuration driven by .claude/daemon-config.json with _smart_bool / _smart_int helpers in scripts/lib/config.sh
• All scripts Bash 3.2 compatible, file writes atomic (tmp+mv), version-stamped

---

Decision

Chosen Architecture: In-Process Bash Subshell Parallelism with Static Whitelist and Atomic Aggregation

We implement parallel execution via Bash subshells with explicit exit-code capture (bypassing bash 3.2 wait limitations), starting with two safe stage pairs: (intake, intelligence) and (test, security_audit). The system is:

1. Flag-off by default — zero behavioral change until explicitly enabled
2. Whitelist-driven — only human-approved pairs can run in parallel
3. Atomic at the group boundary — stages within a group either all complete or all fail
4. Event-compatible — existing stage.started/completed events flow unchanged; new stage.group.* events are additive
5. Resume-safe — if a group crashes mid-execution, resume re-runs the entire group
6. Cost-accurate — no double-counting; concurrent cost is the sum of children

Component Diagram

┌──────────────────────────────────────┐
│  sw-pipeline.sh (entry point)        │
│  (unchanged—routes to pipeline-exec) │
└────────────────┬─────────────────────┘
                 │ calls
         ┌───────▼────────────────────┐
         │ pipeline-execution.sh       │
         │ (main stage loop)           │
         │ • queries dag_groups()      │
         │ • routes group-of-1 → old   │
         │   run_stage_with_retry path │
         │ • routes group-of-N →       │
         │   parallel_group_run()      │
         └───────┬────────────────────┘
                 │ uses (conditional)
         ┌───────┴───────────────────┐
         │                           │
    ┌────▼──────────────┐    ┌──────▼──────────────┐
    │ parallel-dag.sh   │    │parallel-runner.sh   │
    │ (logic layer)     │    │(execution layer)    │
    │ • PARALLEL_       │    │ • parallel_group_   │
    │   WHITELIST_*     │    │   run()—spawn       │

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-05-09T07:25:55Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 30 max iterations, 1 agent(s), model: haiku

[github-actions]

[github-actions]