Skip to content

EvalScorer.score() applies missing-tests weight even when tests were skipped #14

Description

@ChulingNH

Found while reading backend/app/code/eval/scoring.py:112-172.

The weights are:

WEIGHTS = {
    "syntax": 0.30,
    "risks":  0.20,
    "lint":   0.10,
    "tests":  0.40,
}

In _score_dynamic, when the dynamic eval was SKIPPED, the method explicitly does:

if result.status == ExecutionStatus.SKIPPED:
    scores["tests"] = None  # Not applicable
    return scores

Then score() computes the weighted sum (line 145-148):

for dim, weight in self.WEIGHTS.items():
    if dim in dimensions and dimensions[dim] is not None:
        weighted_sum += dimensions[dim] * weight
        total_weight += weight

That branch correctly drops tests when None, and divides by total_weight instead of the full 1.0, so the math is sound for SKIPPED. Good.

But: _score_dynamic is only called when dynamic_result is not None (line 134). If a caller invokes score(static_result, dynamic_result=None), the tests key never enters dimensions at all — score() divides by 0.30 + 0.20 + 0.10 = 0.60 and the syntax dimension alone can contribute up to 100 * 0.30 / 0.60 = 50 to the overall. With risks=100, lint=100 we get an overall=100.0, grade A, even though no tests ran.

That's the bug: the static-only path produces grades equivalent to a tests-run path because total_weight is normalised away.

Fix options

  1. Penalty for missing dynamic eval. If dynamic_result is None, treat tests as 0.0 (not None) so the divisor stays 1.0 and "no tests" caps overall at 60%.
  2. Explicit "static-only" grade. Return EvalScore(overall=..., grade="N/A-static") when dynamic is absent, instead of pretending the score is comparable.

Option 1 is closer to how academic graders treat "didn't submit tests" — zero, not "skip the criterion".

Severity: Medium. The bug is invisible until you compare two projects, one of which ran tests and one of which didn't, and discover their overall scores are nearly identical.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions