Pipeline Early Failure Detection and Auto-Abort System

[github-actions]

Strategic Improvement

Monitor pipeline output in real-time for failure signatures (repeated errors, git conflicts, missing dependencies, authorization failures). When detected in first 5 minutes, auto-abort instead of burning iterations. Save $5-10 per prevented run and improve MTTR.

Acceptance Criteria

• Pattern matching on pipeline logs for known failure signatures (auth, git conflict, missing deps)
• Auto-abort within first 5 minutes if failure pattern detected
• Emit structured abort event with failure classification
• Integrate with cost tracking (report savings from early aborts)
• Reduce wasted cost on doomed pipelines by >40%

Context

• Priority: P0
• Complexity: fast
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Detects early failure patterns to auto-abort doomed pipelines within 5 min, saving >40% wasted cost per prevented run—core reliability + cost improvement aligned with Shipwright's cost intelligence and observatory goals.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	25601752776

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (197 lines)

Design: Pipeline Early Failure Detection and Auto-Abort System

Context

Shipwright pipelines today run to completion (or stall-detector timeout) even when failure is deterministic from minute one — bad GitHub credentials, unmerged conflicts, missing deps. This wastes Claude tokens and wall-clock. The codebase already has sw-stall-detector.sh (406 lines) running a watch loop with abort hooks, sw-cost.sh for cost accounting, events.jsonl for observability, and _smart_int/jq config plumbing in compat.sh. Constraints: bash 3.2, atomic writes, set -euo pipefail, no GH API surface, no new daemons.

Decision

Add a passive log-pattern detector as a library (scripts/lib/early-failure.sh) sourced into the existing stall-detector watch loop. Each tick: tail the canonical pipeline log (-n 5000), grep against a versioned signature catalog (config/early-failure-signatures.json), and if confidence ≥ min_confidence AND count ≥ min_occurrences AND elapsed < window_seconds, take the existing abort path with an idempotency lock keyed on pipeline_id+signature_id. Emit pipeline.early_abort (or pipeline.early_abort_dry in dry-run) and append remaining-cost estimate to ~/.shipwright/savings.jsonl. Ship enabled=false, dry_run=true to collect 7 days of signal before enforcement.

Data flow: stall-detector tick → early_failure_check_all → load_signatures (cached) → scan_log → should_abort → trigger_abort → emit_event + record_savings + stall_abort_helpers.

Error handling: catalog parse error → log warning, set disabled_for_session=true, continue stall detection. Missing log → skip tick. Bad regex → skip that signature, log once.

Alternatives Considered

1. Standalone sw-early-failure.sh daemon — Pros: clean separation of concerns. Cons: duplicate watch loop, extra process to supervise, second abort race surface, more heartbeat plumbing. Rejected: cost > benefit for a passive detector.
2. Inline tail/grep in sw-loop.sh — Pros: closest to log source, fewest hops. Cons: couples detection to one consumer; cannot observe non-loop stages (build, deploy); bloats an already 2675-line script. Rejected: violates single-responsibility and limits coverage.
3. LLM-based classification per tick — Pros: handles novel failures. Cons: cost regression (the very thing we're trying to fix), latency, non-determinism. Rejected: regex catalog is sufficient for the named classes and is auditable.

Component Diagram

┌────────────────────────┐    sources     ┌──────────────────────────┐
│ sw-stall-detector.sh   │───────────────▶│ scripts/lib/early-       │
│ (existing watch loop)  │                │ failure.sh (NEW)         │
└────────────────────────┘                └──────────────────────────┘
            │ on abort decision                      │ reads
            ▼                                        ▼
┌────────────────────────┐                ┌──────────────────────────┐
│ stall_abort helpers    │                │ config/early-failure-    │
│ (existing)             │                │ signatures.json (NEW)    │
└────────────────────────┘                └──────────────────────────┘
            │                                        │
            ▼                                        ▼
┌────────────────────────┐                ┌──────────────────────────┐
│ emit_event             │                │ sw-cost.sh               │
│ → events.jsonl         │                │ record-savings (NEW sub) │
└────────────────────────┘                │ → savings.jsonl          │
                                          └──────────────────────────┘

Dependencies point inward: stall-detector → lib → config + cost. Nothing in lib/early-failure.sh calls back into stall-detector.

Interface Contracts

// scripts/lib/early-failure.sh — bash functions, signatures shown TS-style

early_failure_load_signatures(path: string)
  -> exitcode: 0 (ok) | 1 (parse_error, sets EF_DISABLED=1)
  side_effects: caches parsed catalog in EF_SIGS_JSON

early_failure_scan_log(logfile: string, since_seconds: int)
  -> stdout: JSON `{"matched":[{"id","class","count","confidence"}], "top":{...}|null}`
  errors: missing file → `{"matched":[],"top":null}`

early_failure_should_abort(scan_json: string, window: int, elapsed: int, min_conf: int)
  -> exitcode: 0 (abort) | 1 (no-abort)
  pre: scan_json valid; elapsed ≥ 0

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-05-09T13:15:53Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]

[github-actions]