Skip to content

Wire up partial credit on the existing per-test score field #548

@JimWallace

Description

@JimWallace

Context

Per the test-script contract in CLAUDE.md:

score is reserved for partial credit (not yet used).

The field already round-trips through TestOutcome and the runner JSON envelope. It just doesn't influence the final grade and isn't surfaced in the UI.

This matters most for pattern families (.boundaryEquality, .approximateEquality) where "3 of 5 cases pass" is genuinely more informative than the current binary pass/fail per generated test. It also lets a single hand-written test script say "you got the structure right but the numeric output is off by 5% — 0.8 credit."

Goal

  • score in [0.0, 1.0] on TestOutcome is honoured by grade computation: test's contribution = points × score instead of points × (status == .pass ? 1 : 0).
  • Status semantics preserved: error and timeout always = 0 regardless of score; pass defaults to 1.0 if omitted; fail defaults to 0.0 but can be uplifted via score.
  • Student view shows "0.8 / 1.0" instead of just "fail" for partial-credit cases.
  • Test script contract documentation updated.

Done when

  • A test emitting {"score": 0.8, "shortResult": "..."} with exit-code-1 grades as 80% of that test's points.
  • Pattern family generators can optionally emit partial credit (e.g. "3 of 5 args correct").
  • Backwards-compatible — existing tests that don't emit score behave identically.

Out of scope

  • Negative credit (score < 0).
  • Test-level rubrics inside one test — that's the manual-rubric / inline-annotation territory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions