Cross-Pipeline A/B Testing Framework for Intelligence Feature Validation

[github-actions]

Strategic Improvement

Config has ab_test_ratio=0.2 but no implementation. Build A/B test harness that routes pipelines to variant configurations (template, model routing, iteration counts), tracks outcomes per variant, and computes statistical significance. Accelerates data-driven optimization and validates intelligence improvements.

Acceptance Criteria

• A/B test registry in daemon-config.json defines active experiments with variant configs
• Pipeline spawn routes to control/variant based on ab_test_ratio with stable hashing (same issue always same variant)
• Outcomes tracked per variant: success rate, duration, cost, DORA metrics
• CLI command shipwright ab-test report <experiment-id> shows variant comparison with confidence intervals
• Auto-graduate winning variant after significance threshold (p<0.05, n>30 per arm)
• Integration with intelligence engine to suggest experiments based on optimization opportunities

Context

• Priority: P2
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P2: Intelligence & Learning

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Well-scoped feature adding A/B test registry, deterministic routing, outcome tracking, and statistical reporting—integrates with daemon and pipeline without major refactoring, but requires stable hashing, statistical significance calculation, and database schema updates.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	25594979055

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (176 lines)

Implementation Plan: Cross-Pipeline A/B Testing Framework

Brainstorming & Design Reasoning

Minimum viable change: A new sw-ab-test.sh script with a registry in daemon-config.json, hooked into the daemon's pipeline spawn to route issues to variants via stable hash, plus an outcomes file and a report subcommand.

Alternatives considered:

1. New standalone script + JSON state file (chosen) — Matches existing patterns (sw-cost.sh, sw-memory.sh). Low blast radius. Bash 3.2 compatible. Reuses jq/event bus.
2. Extend sw-intelligence.sh inline — Tighter coupling but bloats an already 1548-line script and conflates concerns. Rejected.
3. SQLite via sw-db.sh — More queryable but adds DB schema migration and overhead for a feature that's mostly append+aggregate. Rejected for MVP.

Why approach 1: Minimizes changes to hot files (daemon, pipeline), follows established script + test-script pattern, atomic JSON writes are well-understood, and graduation logic is simple aggregation.

Files to Modify

Create:

• scripts/sw-ab-test.sh — Main module: registry read, variant routing, outcome tracking, report, graduate
• scripts/sw-ab-test-test.sh — Test suite (mock binaries, PASS/FAIL counters)

Modify:

• scripts/sw — Register ab-test subcommand router entry
• scripts/sw-daemon.sh — Call sw_abtest_assign_variant <issue> at pipeline spawn; export SW_AB_VARIANT, SW_AB_EXPERIMENT to pipeline env
• scripts/sw-pipeline.sh — On terminal stages (success/failure), call sw_abtest_record_outcome with duration, cost, success
• scripts/lib/compat.sh — Add _smart_hash <key> helper (deterministic, mod 100) if not already present
• .claude/daemon-config.json (template/example only) — Document ab_tests registry shape; do not modify user's actual file
• templates/daemon-config.example.json (if exists) — Add example experiments block
• package.json — Register sw-ab-test-test.sh in test script
• .claude/CLAUDE.md — Add ab-test to commands table and AUTO sections (let docs agent sync)

Implementation Steps

1. Add config schema for ab_tests in daemon-config.json structure:

   "ab_tests": {
     "enabled": true,
     "experiments": [{
       "id": "exp-template-fast-v1",
       "status": "active",
       "ratio": 0.2,
       "control": {"template": "standard"},
       "variant": {"template": "fast"},
       "min_sample_size": 30,
       "p_threshold": 0.05,
       "created_at": "2026-05-09T00:00:00Z"
     }]
   }

2. Create scripts/sw-ab-test.sh with these functions:

   • sw_abtest_list_experiments — reads .claude/daemon-config.json .ab_tests.experiments[] | select(.status=="active")

✅ Task Checklist

• Task 1: Define ab_tests schema and document in CLAUDE.md
• Task 2: Create scripts/sw-ab-test.sh skeleton with VERSION, set -euo pipefail, function stubs
• Task 3: Implement stable hash + variant assignment with append-only assignment log
• Task 4: Implement variant override application (template, model routing, iterations)
• Task 5: Implement outcome recording with locked appends to ~/.shipwright/ab-outcomes.jsonl
• Task 6: Implement aggregation + two-proportion z-test + CI computation in awk
• Task 7: Implement report subcommand (text + --json)
• Task 8: Implement graduate (significance gate, atomic config rewrite, event emission)
• Task 9: Implement suggest reading intelligence cache
• Task 10: Hook variant assignment into sw-daemon.sh spawn path
• Task 11: Hook outcome recording into sw-pipeline.sh terminal stages
• Task 12: Register ab-test subcommand in scripts/sw router
• Task 13: Write scripts/sw-ab-test-test.sh (≥15 cases: hashing, bucketing, stats, graduation, atomicity)
• Task 14: Register test in package.json; run npm test -- sw-ab-test-test
• Task 15: Run shipwright doctor and shipwright version check to verify integrity
• shipwright ab-test list/report/graduate/suggest/create all work
• Daemon assigns variants via stable hash; same issue → same variant across runs
• Outcomes recorded for both control and variant from real pipeline runs
• Report shows success rate, duration, cost per arm with 95% CI and z-test p-value
• Graduation gated by n≥30/arm and p<0.05; modifies config atomically

---

Generated by shipwright pipeline at 2026-05-09T07:18:20Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (697 lines)

Architecture Decision Record: Cross-Pipeline A/B Testing Framework for Intelligence Feature Validation

Context

Problem Statement: Shipwright's intelligence layer generates pipeline optimizations (model routing, iteration counts, template selection) but has no principled way to validate that these recommendations actually improve outcomes. We need a framework to:

• Route issues to control (baseline) and variant (optimized) pipelines deterministically
• Collect metrics (success rate, duration, cost) from real pipeline runs
• Detect when a variant is statistically better with 95% confidence
• Automatically graduate winning variants into the daemon's default configuration

Constraints:

• Must work in offline/local mode (NO_GITHUB=1)
• Bash 3.2 compatible (no associative arrays, no readarray)
• Minimal impact on hot paths (daemon spawn, pipeline terminal stages)
• Append-only outcome logging for audit trail and reanalysis
• No new external dependencies (use existing awk, jq, sha256sum)

Related Components:

• sw-intelligence.sh — generates optimization suggestions
• sw-daemon.sh — spawns pipelines for issues
• sw-pipeline.sh — executes stages and records metrics
• sw-cost.sh — tracks token usage and spending
• sw-db.sh — SQLite persistence (not used for this feature)

---

Decision

We will implement a standalone sw-ab-test.sh module with these characteristics:

1. Component Architecture

┌─────────────────────────────────────────┐
│        Daemon Spawn Path                │
│  (sw-daemon.sh line ~450)               │
│  call: sw_abtest_assign_variant $issue  │
└───────────────┬─────────────────────────┘
                │ returns: variant_name
                ▼
┌─────────────────────────────────────────┐
│   Variant Assignment                    │
│   - Stable hash: sha256(issue+exp)      │
│   - Deterministic bucketing             │
│   - Append to .claude/ab-assignments    │
└───────────────┬─────────────────────────┘
                │ exports: SW_AB_EXPERIMENT
                │          SW_AB_VARIANT
                ▼
┌─────────────────────────────────────────┐
│    Pipeline Execution                   │
│    (uses variant config override)        │
└───────────────┬─────────────────────────┘
                │ on completion/failure
                ▼
┌─────────────────────────────────────────┐
│   Outcome Recording                     │
│   (sw-pipeline.sh terminal stages)      │
│   call: sw_abtest_record_outcome        │
│   - Locked append to ~/.shipwright/     │

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-05-09T07:21:04Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: haiku

[github-actions]

[github-actions]

[github-actions]