Pipeline State Snapshot and One-Click Rollback System

[github-actions]

Strategic Improvement

Create automatic snapshots at each stage boundary (git state + pipeline artifacts + context). When pipeline fails, enable one-click rollback to last known good state, then retry with adjusted config. Reduces MTTR from hours to minutes by eliminating manual git recovery.

Acceptance Criteria

• Auto-snapshot after each successful stage (git sha, artifacts, config, context)
• Store snapshots in .claude/pipeline-artifacts/snapshots/.json
• Implement rollback command that restores git state + artifacts + context
• Integrate with daemon retry logic (auto-rollback on certain failure types)
• Enable recovery from failed state in <5 minutes vs current ~2 hours

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Aligns with Shipwright's reliability goals; enables automated failure recovery at stage boundaries via git+artifact snapshots, reducing MTTR significantly for autonomous pipelines.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	25601747490

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (12 lines)

The plan is ready. Here's a summary of what it covers:

New file: scripts/lib/pipeline-snapshot.sh — core library with 4 functions: pipeline_snapshot_take, pipeline_snapshot_list, pipeline_snapshot_restore, pipeline_snapshot_cleanup.

Key integration points:

1. pipeline-state.sh → calls pipeline_snapshot_take inside mark_stage_complete after every successful stage
2. sw-pipeline.sh → gains rollback [--stage <name>] and snapshots subcommands
3. daemon-failure.sh → auto-rolls back to last good snapshot before retrying build_failure class failures

Safety measures: Atomic writes (tmp+mv), git dirty-file stash before reset, SHA validation (git cat-file -t), worktree isolation via per-worktree ARTIFACTS_DIR, daemon-mode guard against resetting shared state.

New test suite: scripts/sw-snapshot-test.sh (~240 lines), registered in package.json.

---

Generated by shipwright pipeline at 2026-05-09T13:11:08Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (209 lines)

Design: Pipeline State Snapshot and One-Click Rollback System

Context

Pipeline failures currently require manual recovery: agents must inspect .claude/pipeline-state.md, reconstruct artifact state, and re-run stages. MTTR is ~2 hours. The daemon's failure-handling path (scripts/lib/daemon-failure.sh) retries failed builds but cannot restore the working tree or artifact directory to the last-known-good state, so retries often compound damage (partial commits, stale composed-pipeline.json, divergent progress.md).

Constraints from the codebase:

• Bash 3.2 only (no associative arrays, no readarray) — per .claude/CLAUDE.md shell standards
• Atomic writes via tmp + mv (per Common Pitfalls)
• Worktree isolation: .claude/pipeline-artifacts/ is per-worktree; snapshots must follow
• Daemon may run multiple pipelines in parallel — snapshot/restore must be flock-serialized per worktree
• Heartbeat-style stale-state detection failed previously (failures.json, sw-cleanup.sh) — do not rely on file-mtime heuristics; use explicit lifecycle markers
• Rejected: simple timeout-based coordination; use semaphore (flock) instead

Decision

A four-function bash library (scripts/lib/pipeline-snapshot.sh) sourced by pipeline-state.sh, sw-pipeline.sh, and daemon-failure.sh. Snapshots are immutable JSON manifests + tarballed artifact deltas, stored per-worktree under .claude/pipeline-artifacts/snapshots/. Restore is a three-phase commit: validate → stage → swap, with a transaction log enabling abort at any phase.

Key behaviors:

• Capture trigger: mark_stage_complete calls pipeline_snapshot_take after state is persisted but before the next stage begins
• Storage: <snapshots-dir>/<run-id>/<stage>-<timestamp>.json (manifest) + sibling .tar.gz (artifact bundle). Run-id derived from pipeline-state's existing started_at
• Git capture: record HEAD SHA + git stash create ref for any dirty state (no auto-stash apply)
• Restore: pipeline_snapshot_restore <stage> validates SHA via git cat-file -t, checks working tree clean (refuses without --force), then resets git, untars artifacts to a staging dir, atomically swaps via mv
• Concurrency: flock on <snapshots-dir>/.lock serializes snapshot/restore within a worktree
• Daemon guard: pipeline_snapshot_restore refuses to run if SHIPWRIGHT_DAEMON_SHARED=1 and worktree is not isolated — prevents resetting state another worker depends on
• Retention: keep last N=5 per stage; on successful pipeline completion, archive run to snapshots/archive/ and prune
• Auto-rollback loop break: persist restore-attempts.json keyed by snapshot SHA; after 3 restores of the same snapshot, daemon escalates to human (avoids the rejected timeout-based heuristic)

Component Diagram

┌─────────────────────┐      ┌──────────────────────┐
│  sw-pipeline.sh     │      │  daemon-failure.sh   │
│  (CLI: rollback,    │      │  (auto-rollback on   │
│   snapshots)        │      │   retryable errors)  │
└──────────┬──────────┘      └──────────┬───────────┘
           │                            │
           │   ┌────────────────────────┘
           ▼   ▼
   ┌──────────────────────────┐         ┌─────────────────────┐
   │  pipeline-snapshot.sh    │◄────────│  pipeline-state.sh  │
   │  (take/list/restore/     │  source │  (mark_stage_       │
   │   cleanup)               │         │   complete hook)    │
   └──────────┬───────────────┘         └─────────────────────┘
              │
              ▼
   ┌──────────────────────────────────────────┐
   │  Storage layer (filesystem-only)          │
   │  .claude/pipeline-artifacts/snapshots/    │
   │    <run>/<stage>-<ts>.{json,tar.gz}       │
   │    .lock  restore-attempts.json           │
   └──────────────────────────────────────────┘

Dependencies point inward: CLI and daemon depend on the library; the library depends only on the filesystem and git. pipeline-state.sh is the only inner consumer (hook).

Interface Contracts

# All functions return 0 on success, non-zero on failure.

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-05-09T13:14:01Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]