Skip to content

Real-world testing: run fixture3 on a matrix of real projects #4

@tartakovsky

Description

@tartakovsky

What

Run fixture3 against real projects (not just the self-suite and the fake-project) and capture every failure mode.

Why

The self-suite proves fixture3 can verify itself. The fake-project proves it can run an external workflow. Neither tells us how well it survives real-world fixture trees, real CI environments, real review flows. Every new real project exposes assumptions baked into the manifest schema, the diff format, or the doctor checks.

Suggested matrix

  • A Next.js project using fixture3 for snapshot testing route handlers.
  • A Python service using fixture3 for end-to-end output snapshots.
  • A Rust workspace using fixture3 alongside cargo test.
  • An ML pipeline (Python or Rust) where the suite output is large JSON.
  • A monorepo with multiple fixture3.yaml files in subdirectories.
  • A repo that wants fixture3 to gate code review without blocking.

What to track per project

  • Did the manifest schema cover the project's needs, or did we need a feature we don't have?
  • Were the JSON diffs readable in code review?
  • Did doctor catch real misconfigurations or miss them?
  • Was the approval flow ergonomic, or did it require workarounds?
  • Did the reducer help, hurt, or get ignored?

Done

A document listing every project, what worked, what didn't. Each ergonomic gap becomes a follow-up feature-request issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood for newcomershelp wantedExtra attention is neededtestingReal-world / corpus / smoke testing

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions