Found while reading backend/app/code/eval/scoring.py:112-172.
The weights are:
WEIGHTS = {
"syntax": 0.30,
"risks": 0.20,
"lint": 0.10,
"tests": 0.40,
}
In _score_dynamic, when the dynamic eval was SKIPPED, the method explicitly does:
if result.status == ExecutionStatus.SKIPPED:
scores["tests"] = None # Not applicable
return scores
Then score() computes the weighted sum (line 145-148):
for dim, weight in self.WEIGHTS.items():
if dim in dimensions and dimensions[dim] is not None:
weighted_sum += dimensions[dim] * weight
total_weight += weight
That branch correctly drops tests when None, and divides by total_weight instead of the full 1.0, so the math is sound for SKIPPED. Good.
But: _score_dynamic is only called when dynamic_result is not None (line 134). If a caller invokes score(static_result, dynamic_result=None), the tests key never enters dimensions at all — score() divides by 0.30 + 0.20 + 0.10 = 0.60 and the syntax dimension alone can contribute up to 100 * 0.30 / 0.60 = 50 to the overall. With risks=100, lint=100 we get an overall=100.0, grade A, even though no tests ran.
That's the bug: the static-only path produces grades equivalent to a tests-run path because total_weight is normalised away.
Fix options
- Penalty for missing dynamic eval. If
dynamic_result is None, treat tests as 0.0 (not None) so the divisor stays 1.0 and "no tests" caps overall at 60%.
- Explicit "static-only" grade. Return
EvalScore(overall=..., grade="N/A-static") when dynamic is absent, instead of pretending the score is comparable.
Option 1 is closer to how academic graders treat "didn't submit tests" — zero, not "skip the criterion".
Severity: Medium. The bug is invisible until you compare two projects, one of which ran tests and one of which didn't, and discover their overall scores are nearly identical.
Found while reading
backend/app/code/eval/scoring.py:112-172.The weights are:
In
_score_dynamic, when the dynamic eval was SKIPPED, the method explicitly does:Then
score()computes the weighted sum (line 145-148):That branch correctly drops
testswhen None, and divides bytotal_weightinstead of the full1.0, so the math is sound for SKIPPED. Good.But:
_score_dynamicis only called whendynamic_result is not None(line 134). If a caller invokesscore(static_result, dynamic_result=None), thetestskey never entersdimensionsat all —score()divides by0.30 + 0.20 + 0.10 = 0.60and the syntax dimension alone can contribute up to100 * 0.30 / 0.60 = 50to the overall. Withrisks=100, lint=100we get anoverall=100.0, gradeA, even though no tests ran.That's the bug: the static-only path produces grades equivalent to a tests-run path because
total_weightis normalised away.Fix options
dynamic_result is None, treattestsas0.0(not None) so the divisor stays1.0and "no tests" caps overall at 60%.EvalScore(overall=..., grade="N/A-static")when dynamic is absent, instead of pretending the score is comparable.Option 1 is closer to how academic graders treat "didn't submit tests" — zero, not "skip the criterion".
Severity: Medium. The bug is invisible until you compare two projects, one of which ran tests and one of which didn't, and discover their overall scores are nearly identical.