fix: populate automated_production_score in controlled_solver#368
Open
rasdani wants to merge 1 commit into
Open
fix: populate automated_production_score in controlled_solver#368rasdani wants to merge 1 commit into
rasdani wants to merge 1 commit into
Conversation
The unbounded_solver tracks ``automated_production_score`` per step and
writes it into ``TrajectoryData`` (solver.py:1023, :1167, :1225), but
the controlled_solver (used by all throughput tasks via
``fle inspect-eval --solver controlled``) doesn't. Result: every saved
``.eval`` log for a throughput task reports
``automated_production_score = 0.0`` regardless of the actual factory
output.
Mirror the unbounded_solver's pattern in the controlled loop:
- Track ``automated_production_scores`` list alongside
``production_scores`` near the trajectory init (solver.py:319).
- Read ``info.get("automated_production_score", 0)`` after each
``gym_env.step`` and append.
- Update ``trajectory_data.{automated_production_score, automated_scores}``
at every per-step store and at final results, plus reset on the
exception path.
Verified with::
fle inspect-eval --tasks iron_gear_wheel_throughput \
--model anthropic/claude-sonnet-4-5 --solver controlled \
--pass-n 1 --max-connections 1
Before: ``auto=0`` despite ``prod=21``.
After: ``auto=460`` with timeseries ``[-30, 42, 123, 215, 460]``.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unbounded_solver tracks
automated_production_scoreper step and writes it intoTrajectoryData(solver.py:1023, :1167, :1225), but the controlled_solver (used by all throughput tasks viafle inspect-eval --solver controlled) doesn't. Result: every saved.evallog for a throughput task reportsautomated_production_score = 0.0regardless of the actual factory output.Mirror the unbounded_solver's pattern in the controlled loop:
automated_production_scoreslist alongsideproduction_scoresnear the trajectory init (solver.py:319).info.get("automated_production_score", 0)after eachgym_env.stepand append.trajectory_data.{automated_production_score, automated_scores}at every per-step store and at final results, plus reset on the exception path.Verified with::
fle inspect-eval --tasks iron_gear_wheel_throughput \ --model anthropic/claude-sonnet-4-5 --solver controlled \ --pass-n 1 --max-connections 1
Before:
auto=0despiteprod=21.After:
auto=460with timeseries[-30, 42, 123, 215, 460].