diff --git a/apps/website/content/docs/reference/guidelines-catalog.mdx b/apps/website/content/docs/reference/guidelines-catalog.mdx index 7f233892..52343c47 100644 --- a/apps/website/content/docs/reference/guidelines-catalog.mdx +++ b/apps/website/content/docs/reference/guidelines-catalog.mdx @@ -106,6 +106,7 @@ Security, accessibility, and performance standards. | Topic | What it covers | | --- | --- | +| **Quality Model** | 3-layer principle, 3 pillars (Cost/Security/Delivery), classification dimensions and tier resolution, per-tier requirements, tag projection, `tech/risk-matrix.md` adoption delta — see the [dedicated page](/docs/reference/quality-model) | | **Accessibility** | WCAG compliance, screen reader support, keyboard navigation | | **Performance** | Load times, Core Web Vitals, optimization | | **Quality Monitoring** | Quality dashboards, trend analysis | @@ -155,6 +156,7 @@ Design systems, research, and content strategy. ## Related +- [Quality Model](/docs/reference/quality-model) — the single source of default quality rules (classification, tiers, tags) - [Skills Catalog](/docs/reference/skills-catalog) — all 32 pair skills with commands and composition - [Developer Journey](/docs/developer-journey) — step-by-step walkthrough of the full process - [KB Structure](/docs/reference/kb-structure) — directory layout and file purposes diff --git a/apps/website/content/docs/reference/meta.json b/apps/website/content/docs/reference/meta.json index 26935077..ded1c1f4 100644 --- a/apps/website/content/docs/reference/meta.json +++ b/apps/website/content/docs/reference/meta.json @@ -5,6 +5,7 @@ "specs", "skills-catalog", "guidelines-catalog", + "quality-model", "skill-management", "kb-structure", "configuration" diff --git a/apps/website/content/docs/reference/quality-model.mdx b/apps/website/content/docs/reference/quality-model.mdx new file mode 100644 index 00000000..9dea2b79 --- /dev/null +++ b/apps/website/content/docs/reference/quality-model.mdx @@ -0,0 +1,54 @@ +--- +title: Quality Model +description: The single source of default quality rules — 3-layer principle, 3 pillars, classification dimensions, per-tier requirements, tag projection, and the risk-matrix adoption delta. +--- + +The Quality Model is the single source of default quality rules in the pair Knowledge Base. `classify`, `assess-cost`, `assess-security`, code review, quality gates, and bootstrap all resolve their behavior from this one document — no criteria live anywhere else. Source: [`quality-model.md`](https://github.com/foomakers/pair/blob/main/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md). + +> **Resolution order** for every rule below: **Argument > Adoption > KB default**. A malformed adoption delta is treated as absent — skills warn and fall back to KB defaults. + +## Three-Layer Principle + +| Layer | Role | Examples | +| --- | --- | --- | +| **Doc** | Rules, written once | this document + pillar guidelines | +| **Skill** | Applies the rules on demand | `classify`, `assess-cost`, `assess-security`, code review | +| **Automation** | Consumes tags deterministically, zero judgment | CI gates, `pair-next --filter` | + +Quality is classified in refinement — before code exists — not only at review time (shift-left). The matrix is built twice: once from the story context in refinement, once from the diff in review. Automation never carries its own criteria — it only reads tags. + +## Three Pillars + +| Pillar | Covers | Tag family | +| --- | --- | --- | +| **Cost** | Financial exposure of building/running the change | `cost:*` | +| **Security** | Vulnerabilities, compliance, secure-by-design | feeds `risk:*` + deterministic CI scanning | +| **Delivery** | Correctness, performance, a11y, observability, docs, planning, architecture, release, AI metrics | `risk:*` | + +Every quality theme nests under one of these three pillars, pointing at its existing guideline — no dedicated status page per theme. + +## Classification & Tiers + +The compiled matrix has one row per dimension: service/domain criticality, change/diff risk, business impact, security relevance, and coupling balance. Each resolves to green/yellow/red; the **risk tier is the max across assessed dimensions** and is never lowered once raised by review. Cost is tracked separately as its own class (`cost:green|yellow|orange|red`, driven by the highest detected cost signal). + +| Tier | Merge | Reviewers | SLA | Approval | +| --- | --- | --- | --- | --- | +| 🟢 Green | Self-merge at green gate checks | 0 (AI review informational) | — | none | +| 🟡 Yellow | Blocked until reviewed | 1 reviewer | 1 working day | reviewer approval | +| 🔴 Red | Blocked until reviewed and approved | 1 reviewer | 2 working days | explicit approval | + +A PR without classification tags is treated as 🔴 (fail-safe). + +## Tag Projection & Eligibility + +Tags are a **chromatic projection** of the matrix — `risk:green|yellow|red`, `cost:green|yellow|orange|red` — created only if your adoption declares the matrix→tag projection (the quickstart activates the standard set by default). There is no dedicated "eligibility" tag: automation eligibility is a **filter you declare over classification tags** (e.g. `risk:green`), consumed generically by `pair-next`. + +## Project Overrides — `tech/risk-matrix.md` + +Optional. Absent means KB defaults apply completely — nothing fails. When present, it carries only the delta: a criticality table (service/domain → High/Medium/Low) and optional threshold overrides — see the full schema in [`quality-model.md` §6](https://github.com/foomakers/pair/blob/main/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md#6-techrisk-matrixmd--adoption-delta) and a filled-in example at [`risk-matrix-example.md`](https://github.com/foomakers/pair/blob/main/packages/knowledge-hub/dataset/.pair/knowledge/assets/risk-matrix-example.md). + +## Related + +- [Guidelines Catalog](/docs/reference/guidelines-catalog) — where the Quality Model sits among the other 9 guideline categories +- [Customize for Your Team](/docs/customization/team) — how to override guidelines via adoption files +- [KB Structure](/docs/reference/kb-structure) — directory layout, `knowledge/` vs `adoption/` diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/assets/risk-matrix-example.md b/packages/knowledge-hub/dataset/.pair/knowledge/assets/risk-matrix-example.md new file mode 100644 index 00000000..d9b0e9f1 --- /dev/null +++ b/packages/knowledge-hub/dataset/.pair/knowledge/assets/risk-matrix-example.md @@ -0,0 +1,21 @@ +# `tech/risk-matrix.md` — Example + +Illustrative adoption delta for a fictitious e-commerce project. Copy the sections you need — this file is only committed as a reference; a real project's `tech/risk-matrix.md` normally has just a few rows. See the schema and resolution rules in the [quality model](../guidelines/quality-assurance/quality-model.md), §6. + +## Criticality Table + +| Service/Domain | Criticality | +| --- | --- | +| payments | High | +| checkout | High | +| catalog | Medium | +| marketing-site | Low | +| internal-admin-tool | Low | + +Any service/domain not listed here is treated as unclassified and resolves to High (conservative) for the service-criticality dimension — not to the file-absent Medium default. + +## Overrides + +Optional threshold tweaks for the change-risk dimension (§3.1 of the quality model). Omit entirely if the KB defaults are fine. + +- `change-risk.shared-paths`: `["packages/billing/**", "packages/checkout-core/**"]` — diffs touching these paths are always classified `yellow` or higher for change/diff risk, even if the diff is small. diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/README.md b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/README.md index 63ea7de3..ea9d5664 100644 --- a/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/README.md +++ b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/README.md @@ -24,6 +24,7 @@ This comprehensive quality assurance framework ensures systematic quality manage ### **Core Quality Framework** +- **quality-model.md** - Single source of default quality rules: 3-layer principle, 3 pillars (Cost/Security/Delivery), classification dimensions and tier resolution, per-tier review/gate requirements, tag projection, `tech/risk-matrix.md` adoption delta schema, nested taxonomy - **automated-verification.md** - Comprehensive automated verification system for tool-based validation and quality gate enforcement - **manual-verification.md** - Manual verification processes and human-driven quality assessment methodologies - **manual-testing.md** - Guidelines for designing, organizing, and executing manual test suites for post-release validation diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md new file mode 100644 index 00000000..da3c00d9 --- /dev/null +++ b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md @@ -0,0 +1,127 @@ +# Quality Model + +The single source of default quality rules for this KB. `classify`, `assess-cost`, `assess-security`, `pair-process-review`, `setup-gates`, and `pair-process-bootstrap` all resolve their behavior from this one document — no criteria live anywhere else. Project deviations are a delta in `tech/risk-matrix.md` (§6); absent, KB defaults apply completely. + +**Resolution order** for every rule below: **Argument > Adoption > KB default**. Argument = an explicit override passed to a skill invocation by a human. Adoption = `tech/risk-matrix.md` (§6). KB default = this document. A malformed adoption file is treated as absent: skills warn and fall back to KB defaults. + +## 1. Three-Layer Principle + +| Layer | Role | Examples | +| --- | --- | --- | +| **Doc** | Rules, written once, human-readable | this document + pillar guidelines (§7) | +| **Skill** | Applies the rules on demand, produces artifacts | `classify`, `assess-cost`, `assess-security`, `pair-process-review` | +| **Automation** | Consumes artifacts deterministically, zero judgment | CI gates, `pair-next --filter` | + +**Shift-left** (R1.3): quality is classified in refinement — before code exists — not only at review time (see [Shift-Left Quality](README.md) in the QA framework overview). The matrix is built twice, refinement and review (§3.2); automation never adds its own criteria (D18) — it only reads tags. + +## 2. Three Pillars + +| Pillar | Covers | Tag family | Primary skill | +| --- | --- | --- | --- | +| **Cost** | Financial exposure of building/running the change | `cost:*` | `assess-cost` (cost-signal catalog, forthcoming) | +| **Security** | Vulnerabilities, compliance, secure-by-design | none dedicated — feeds `risk:*` (§3) + deterministic CI scanning | `assess-security`, [security/](security/README.md) | +| **Delivery** | Everything else: correctness, performance, a11y, observability, docs, planning, architecture, release, AI metrics | `risk:*` (correctness/blast-radius facets) | `pair-process-review`, `classify` | + +Every theme not directly named here nests under one of these three — see §7. No status pages, no dedicated backlog per theme: a theme gets a card only when there is real work. + +## 3. Classification Model + +### 3.1 Risk dimensions + +The compiled matrix has one row per dimension below. Each row resolves to `green`/`yellow`/`red`. + +| Dimension | Req. | Source (refinement → review) | green | yellow | red | +| --- | --- | --- | --- | --- | --- | +| Service/domain criticality | R5.1 | `tech/risk-matrix.md` criticality table | Low | Medium (default when the file is absent) | High (default for a service/domain **not listed** in an existing table — conservative) | +| Change/diff risk | R5.2 | story scope → diff footprint | isolated, localized change | touches multiple modules or shared code | schema/migration, contract-breaking change, or infra provisioning change | +| Business impact | R4.3 | subdomain classification of what the story/diff touches | `generic` subdomain | `supporting` subdomain | `core` subdomain | +| Security relevance | — | heuristic over touched paths | no security-sensitive surface | security-adjacent (new external dependency, input validation on a non-critical path) | authn/authz, secrets/credentials, cryptography, PII, untrusted-input parsing | +| Coupling balance | — | story context (touched subdomains' volatility + cross-context integrations) → diff (`assess-coupling` verdict) | balanced | unbalanced + stable | unbalanced + volatile | + +Coupling sources absent (no subdomain/bounded-context artifacts, no `assess-coupling` available) ⇒ reported **not assessed**, excluded from the max below, never blocks (D21). See `architecture/design-patterns/coupling-balance.md` (nested taxonomy entry, §7, not yet published) — the single home for the coupling model itself; this document never duplicates that content, only the classification rule above. + +### 3.2 Tier resolution + +**Risk tier = max(assessed dimensions above)**, projected as `risk:green|yellow|red`. + +- Built **twice** per story (D17): in refinement from the **story context** (declared/estimated), in review from the **code/diff** (observed). The review value is a floor: it may raise the tier, **never lower it**. +- A PR with no classification present is treated as `red` (fail-safe). +- Cost (§3.3) is not part of this max — it is its own class with its own tag family, computed independently, and carried in the same compiled matrix. + +### 3.3 Cost class (R6.2) + +Cost class = **highest detected signal**, projected as `cost:green|yellow|orange|red`. The signal catalog (paid-SDK imports, API-key env vars, IaC/provisioning changes, cron/queues, media processing, LLM calls) is maintained in the cost-assessment guideline (forthcoming, `assess-cost`); no signal detected ⇒ `green`. General + provider-specific heuristics: [infrastructure/cloud-providers/cost-optimization.md](../infrastructure/cloud-providers/cost-optimization.md). + +## 4. Per-Tier Requirements + +| Tier | Merge | Reviewers | SLA | Checklist | Approval | +| --- | --- | --- | --- | --- | --- | +| 🟢 Green | Self-merge once gate checks are green | 0 (AI review informational, ≤4h) | — | standard | none | +| 🟡 Yellow | Blocked until reviewed | 1 reviewer | 1 working day | standard | reviewer approval | +| 🔴 Red | Blocked until reviewed and approved | 1 reviewer (not 2) | 2 working days | extended | explicit approval required | + +Review always runs, tests are always green, at every tier (R5.3 + D10). Gate (mechanical) and review (judgment) are distinct enforcers — gate blocks first, review starts only once gates are green: + +| Tier | Gate checks | +| --- | --- | +| 🟢 | lint + type + build | +| 🟡 | + unit | +| 🔴 | + integration/E2E | + +## 5. Tag Projection + +Chromatic scheme only — no semantic tag beyond color: + +- `risk:green|yellow|red` — §3.2 +- `cost:green|yellow|orange|red` — §3.3 + +**Adoption-gated emission**: `classify` creates these tags **only if adoption declares the matrix→tag projection** (quickstart activates the standard set above by default; a project may rename or disable individual tags in `tech/risk-matrix.md`, §6). Without a declared projection, the matrix still exists in the story/PR body — it is simply not projected onto tags. + +**No dedicated eligibility tag**: automation eligibility is an **adoption-declared filter over classification tags** (e.g. `risk:green`, optionally combined with project tags), not a special tag of its own. `pair-next` consumes it generically, like any other tag filter, re-evaluated on every run (tags can change between runs, e.g. review raising the tier). + +## 6. `tech/risk-matrix.md` — Adoption Delta + +Optional. Absent ⇒ KB defaults (§3.1) apply completely, nothing fails (D21). Present ⇒ only the delta: criticality table and/or threshold overrides — a few lines, not a rewrite of this document. + +```markdown +## Criticality Table + +| Service/Domain | Criticality | +| --- | --- | +| payments | High | +| marketing-site | Low | + +## Overrides + +- change-risk.shared-paths: ["packages/billing/**"] +``` + +- **Malformed file** (unparseable table, unknown keys): skills warn and fall back to KB defaults entirely (D21). +- **Unknown service/domain** (queried but not in the table): treated as unclassified ⇒ conservative High for that dimension. +- A filled-in example (also usable as adoption starting point) is at [risk-matrix-example.md](../../assets/risk-matrix-example.md). + +### Resolution-cascade walkthrough + +| Scenario | `tech/risk-matrix.md` | Resolution | +| --- | --- | --- | +| No file | absent | Service criticality defaults to Medium; all other dimensions per §3.1 defaults; nothing fails (AC2) | +| File present, service listed | `payments: High` | `payments` resolves to red for that dimension, overriding the Medium default (AC3) | +| File present, service **not** listed | table has other entries only | Conservative High (red) for that dimension, not the absent-file Medium default | +| File present but malformed | unparseable | Warn, fall back to KB defaults as if absent | + +## 7. Nested Taxonomy + +Every quality theme not covered by §1–§6 lives under one of the three pillars, pointing at its existing guideline — no new status page, no dedicated backlog structure per theme (D13). + +| Theme | Pillar | Guideline | +| --- | --- | --- | +| Performance | Delivery | [performance/README.md](performance/README.md) | +| Accessibility | Delivery | [accessibility/README.md](accessibility/README.md) | +| Observability | Delivery | [../observability/README.md](../observability/README.md) | +| Documentation | Delivery | [../technical-standards/ai-development/documentation-standards.md](../technical-standards/ai-development/documentation-standards.md) | +| Planning | Delivery | [../collaboration/methodology/README.md](../collaboration/methodology/README.md) | +| Architecture / modularity | Delivery | `architecture/design-patterns/coupling-balance.md` (not yet published — single home for the coupling model, see §3.1) | +| Release | Delivery | [../technical-standards/deployment-workflow/release-management.md](../technical-standards/deployment-workflow/release-management.md) | +| AI metrics / retro | Delivery | [../collaboration/project-tracking/README.md](../collaboration/project-tracking/README.md) (reports land in `.pair/working/reports/`, once available) | +| Vulnerabilities / compliance | Security | [security/vulnerability-prevention.md](security/vulnerability-prevention.md), [security/compliance.md](security/compliance.md) | +| Cost signals | Cost | cost-assessment guideline (not yet published, see §3.3) | diff --git a/packages/knowledge-hub/src/quality-model.test.ts b/packages/knowledge-hub/src/quality-model.test.ts new file mode 100644 index 00000000..b9a69646 --- /dev/null +++ b/packages/knowledge-hub/src/quality-model.test.ts @@ -0,0 +1,130 @@ +import { describe, it, expect } from 'vitest' +import { readFileSync, existsSync } from 'fs' +import { join } from 'path' + +const QUALITY_ASSURANCE_DIR = join( + __dirname, + '../dataset/.pair/knowledge/guidelines/quality-assurance', +) + +const QUALITY_MODEL = readFileSync( + join( + __dirname, + '../dataset/.pair/knowledge/guidelines/quality-assurance/quality-model.md', + ), + 'utf-8', +) +const RISK_MATRIX_EXAMPLE = readFileSync( + join(__dirname, '../dataset/.pair/knowledge/assets/risk-matrix-example.md'), + 'utf-8', +) +const QA_README = readFileSync( + join(__dirname, '../dataset/.pair/knowledge/guidelines/quality-assurance/README.md'), + 'utf-8', +) + +describe('quality-model.md — structure', () => { + it('has the expected title', () => { + expect(QUALITY_MODEL).toMatch(/^# Quality Model/m) + }) + + it('documents the 3-layer principle', () => { + expect(QUALITY_MODEL).toMatch(/Three-Layer Principle/) + expect(QUALITY_MODEL).toMatch(/\*\*Doc\*\*/) + expect(QUALITY_MODEL).toMatch(/\*\*Skill\*\*/) + expect(QUALITY_MODEL).toMatch(/\*\*Automation\*\*/) + }) + + it('documents the 3 pillars', () => { + expect(QUALITY_MODEL).toMatch(/\*\*Cost\*\*/) + expect(QUALITY_MODEL).toMatch(/\*\*Security\*\*/) + expect(QUALITY_MODEL).toMatch(/\*\*Delivery\*\*/) + }) + + it('documents all 5 risk dimensions with tier = max', () => { + for (const dim of [ + 'Service/domain criticality', + 'Change/diff risk', + 'Business impact', + 'Security relevance', + 'Coupling balance', + ]) { + expect(QUALITY_MODEL).toContain(dim) + } + expect(QUALITY_MODEL).toMatch(/max\(assessed dimensions/i) + }) + + it('matches the D10 SLA table exactly — 1 reviewer even at red tier', () => { + expect(QUALITY_MODEL).toMatch(/🟢[^\n]*Self-merge/) + expect(QUALITY_MODEL).toMatch(/🟡[^\n]*\|\s*1 reviewer\s*\|\s*1 working day/) + expect(QUALITY_MODEL).toMatch(/🔴[^\n]*\|\s*1 reviewer \(not 2\)\s*\|\s*2 working days/) + }) + + it('defines the chromatic tag projection', () => { + expect(QUALITY_MODEL).toContain('risk:green|yellow|red') + expect(QUALITY_MODEL).toContain('cost:green|yellow|orange|red') + }) + + it('states there is no dedicated eligibility tag', () => { + expect(QUALITY_MODEL).toMatch(/No dedicated eligibility tag/) + }) + + it('states the Argument > Adoption > KB default resolution order', () => { + expect(QUALITY_MODEL).toContain('Argument > Adoption > KB default') + }) + + it('walks through the resolution cascade with and without the delta file', () => { + expect(QUALITY_MODEL).toMatch(/No file[^|]*\|\s*absent/) + expect(QUALITY_MODEL).toMatch(/File present, service listed/) + expect(QUALITY_MODEL).toMatch(/File present, service \*\*not\*\* listed/) + expect(QUALITY_MODEL).toMatch(/File present but malformed/) + }) + + it('does not create dead hyperlinks for guidelines that do not exist yet', () => { + expect(QUALITY_MODEL).not.toMatch(/\]\([^)]*coupling-balance\.md\)/) + expect(QUALITY_MODEL).toContain('`architecture/design-patterns/coupling-balance.md`') + }) + + it('resolves every §7 nested-taxonomy pointer link to a file on disk', () => { + const section = QUALITY_MODEL.split('## 7. Nested Taxonomy')[1] + const links = [...section.matchAll(/\]\(([^)]+)\)/g)].map((m) => m[1]) + expect(links.length).toBeGreaterThanOrEqual(9) + for (const link of links) { + expect(existsSync(join(QUALITY_ASSURANCE_DIR, link))).toBe(true) + } + }) + + it('nests every listed theme under a pillar with a pointer', () => { + for (const theme of [ + 'Performance', + 'Accessibility', + 'Observability', + 'Documentation', + 'Planning', + 'Architecture / modularity', + 'Release', + 'AI metrics / retro', + 'Vulnerabilities / compliance', + 'Cost signals', + ]) { + expect(QUALITY_MODEL).toContain(theme) + } + }) +}) + +describe('risk-matrix-example.md', () => { + it('provides a criticality table with at least one High entry', () => { + expect(RISK_MATRIX_EXAMPLE).toMatch(/## Criticality Table/) + expect(RISK_MATRIX_EXAMPLE).toMatch(/\|\s*payments\s*\|\s*High\s*\|/) + }) + + it('documents the unknown-service default separately from the file-absent default', () => { + expect(RISK_MATRIX_EXAMPLE).toMatch(/resolves to High/) + }) +}) + +describe('quality-assurance README — index', () => { + it('lists quality-model.md in the core quality framework section', () => { + expect(QA_README).toMatch(/quality-model\.md/) + }) +})