Skip to content

fix: handle RemoteEvalParameters in eval dev /list endpoint#192

Draft
Alex Jean-Baptiste (justcodebruh) wants to merge 1 commit into
mainfrom
ajb-list-endpoint-fix
Draft

fix: handle RemoteEvalParameters in eval dev /list endpoint#192
Alex Jean-Baptiste (justcodebruh) wants to merge 1 commit into
mainfrom
ajb-list-endpoint-fix

Conversation

@justcodebruh
Copy link
Copy Markdown
Contributor

Summary

  • Fix AttributeError: 'RemoteEvalParameters' object has no attribute 'items' raised by build_eval_definitions when an Eval(...) is declared with parameters=load_parameters(...).
  • The dev server's /list endpoint was masking this as a generic 500 (Eval runner exited with an error.), so the Playground Remote evals settings page just said "could not connect to the dev server" with no actionable trace.
  • Branch on RemoteEvalParameters in the runner and pass through its already-resolved .schema, matching what the SDK itself does at braintrust/framework.py:1515 and braintrust/parameters.py:357.

Bug path

  1. Customer calls params = load_parameters(project=..., slug=...) → returns a RemoteEvalParameters object.
  2. Customer passes parameters=params into Eval(...).
  3. UI hits /list on the dev server → bt spawns eval-runner.py with BT_EVAL_DEV_MODE=list.
  4. build_eval_definitions (was eval-runner.py:570) calls parameters_to_json_schema(evaluator.parameters) unconditionally; that helper calls parameters.items(), which RemoteEvalParameters does not implement → uncaught AttributeError.
  5. Runner exits non-zero. Rust dev_server_list returns a generic 500 with no error detail (the traceback goes to stderr, not through sse.send(\"error\", ...)).

Verification (local)

  • BT_EVAL_DEV_MODE=list python scripts/eval-runner.py eval_repro_15895.py --local against the same Eval(..., parameters=load_parameters(project=\"repro-15895\", slug=\"eval-config\")) repro: previously crashed with the AttributeError at line 570 → now emits the correctly-serialized JSON schema.
  • Built cargo build --release and ran the patched binary against the same eval file. curl http://localhost:8301/list (with Authorization: Bearer + x-bt-org-name) previously returned 500 {\"error\":\"Eval runner exited with an error.\"}; now returns 200 with the full param schema (model + eval_prompt + temperature + system_prompt all resolved).

Test plan

  • CI green
  • Manual repro from Pylon #15895: bt eval <file>.py --language=python --dev --dev-port 8301 against an Eval(..., parameters=load_parameters(...)) no longer crashes the dev server, and the Playground Remote evals settings page lists the eval with all saved parameters visible
  • Sanity: existing inline Eval(parameters={\"prefix\": PrefixParam, \"main\": {\"type\": \"prompt\", ...}}) flow is unchanged

Pylon: #15895

🤖 Generated with Claude Code

`build_eval_definitions` crashed with `AttributeError: 'RemoteEvalParameters'
object has no attribute 'items'` when an `Eval(...)` passed
`parameters=load_parameters(...)`. The runner called
`parameters_to_json_schema(evaluator.parameters)` unconditionally; that
helper expects an `EvalParameters` mapping and calls `.items()` on it,
which `RemoteEvalParameters` does not implement.

The dev server's `/list` handler in Rust only surfaced this as a generic
500 ("Eval runner exited with an error."), so the Playground Remote evals
settings page just said "could not connect to the dev server" with no
indication that the saved-parameters path was the trigger.

Branch on `RemoteEvalParameters` and pass through `.schema` directly,
matching how the SDK itself handles this object in
`framework.py:1515` and `parameters.py:357`. The other dev_mode == "list"
paths (`parameters_to_json_schema` for inline `EvalParameters`, empty
when no parameters are declared) are preserved.

Pylon: #15895

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@justcodebruh Alex Jean-Baptiste (justcodebruh) marked this pull request as draft May 18, 2026 15:43
@github-actions
Copy link
Copy Markdown

Latest downloadable build artifacts for this PR commit 53f4956501f3:

Available artifact names
  • ``artifacts-build-global
  • ``artifacts-build-local-x86_64-apple-darwin
  • ``artifacts-build-local-x86_64-pc-windows-msvc
  • ``artifacts-build-local-aarch64-pc-windows-msvc
  • ``artifacts-build-local-x86_64-unknown-linux-gnu
  • ``artifacts-build-local-x86_64-unknown-linux-musl
  • ``artifacts-build-local-aarch64-apple-darwin
  • ``artifacts-build-local-aarch64-unknown-linux-gnu
  • ``artifacts-plan-dist-manifest
  • ``cargo-dist-cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant