fix(isaac): record rollout videos at MuJoCo parity (closes #112)#114
Merged
cagataycali merged 1 commit intoJun 19, 2026
Merged
Conversation
…bs#112) The Isaac LIBERO examples wrote no rollout video -- only a `…__image.mp4.placeholder` path string (never written to disk) -- while their MuJoCo counterparts produce a real `{name}__{camera}.mp4` via `start_cameras_recording` / `stop_cameras_recording`. The deferral rationale ("render() returns blank frames", "recorder is a separate slice") is stale: with strands-labs#61/strands-labs#62 merged, `IsaacSimulation.render()` returns real RGB frames on Isaac Sim 6.0. Backend recorder: - Add `IsaacSimulation.start_cameras_recording` / `stop_cameras_recording`. Capture is synchronous (Isaac's RTX renderer + Camera.get_rgba are bound to the SimulationApp thread, so a MuJoCo-style daemon recorder would deadlock): `start` returns an `on_frame(step, obs, action)` closure that the eval driver wires into `evaluate_benchmark(on_frame=...)` (strands-robots 0.4.0 signature). It grabs one `render()` frame per applied control step into an in-memory buffer; `stop` flushes to `{name}__{camera}.mp4` via imageio -- the exact filename/layout convention MuJoCo uses, so cross-backend video discovery (the R15 backend-matrix glob) picks up Isaac rows. - Recorder state is cleared on `destroy()`; double-start, unknown camera, and no-camera cases error loudly; per-camera render failures increment an error counter instead of aborting the eval. Examples: - `run_isaac.py`: replace the `.placeholder` with a real start/stop_cameras_recording pair around `evaluate_benchmark`; the grep-stable `videos=` line now points at a file that exists. - `run_isaac_agent.py`: arm the recorder before the agent runs and flush after. The `on_frame` closure is stashed at module scope (it can't cross the @tool JSON-schema boundary, same constraint as #191) and the tool wrapper threads it into `evaluate_benchmark`. - Drop the stale "blank frames" / "intentionally skipped" comments. Tests: add `TestCamerasRecording` (11 cases) covering the lifecycle, error paths, frame-cap, MuJoCo-compatible filename, and destroy cleanup. The two MP4-encode cases `importorskip("imageio")` so the lightweight hatch test env (no imageio) skips them cleanly.
cagataycali
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings the Isaac LIBERO examples to parity with their MuJoCo counterparts on rollout-video output. They now write a real
rollouts/<date>/<name>__<camera>.mp4instead of an…__image.mp4.placeholderpath string that was never written to disk.Why
run_mujoco.py/run_mujoco_agent.pyrecord a real MP4 via astart_cameras_recording/stop_cameras_recordingpair.run_isaac.py/run_isaac_agent.pyrecorded nothing — they only emitted a placeholder path in the grep-stablevideos=line, so any tooling that trustsvideos=(e.g. the R15 backend matrix'srollouts/*/*--task=*.mp4glob) got a dangling/blank entry for Isaac rows.The deferral rationale was stale:
IsaacSimulationhad no recorder API (start_cameras_recording/stop_cameras_recordingare MuJoCo-Simulation-specific, not on the baseSimEngine), and the comments cited "render() returns blank frames". With #61/#62 merged,IsaacSimulation.render()returns real RGB frames on Isaac Sim 6.0, so videos can be produced.Approach
This implements both suggested fixes from the issue — a backend recorder and the example-level
on_framewiring — because they compose: the recorder is synchronous (Isaac's RTX renderer +Camera.get_rgbaare bound to the thread that bootedSimulationApp, so a MuJoCo-style daemon recorder would deadlock), so it relies onevaluate_benchmark'son_frame=hook to capture onerender()frame per control step on the eval thread.Backend (
strands_robots_sim/isaac/simulation.py)start_cameras_recording(cameras, output_dir, fps, name, max_frames_per_camera)— sets up one in-memory RGB buffer per camera and returns anon_frame(step, obs, action)closure in the result'sjsonblock.stop_cameras_recording()— flushes buffers to{name}__{camera}.mp4viaimageio(the same encoder the MuJoCo recorder uses), matching MuJoCo's filename/layout convention exactly. Idempotent.destroy().Examples
run_isaac.py— replaces the.placeholderwith a realstart/stoppair aroundevaluate_benchmark(on_frame=...).videos=now points at a file that exists.run_isaac_agent.py— arms the recorder before the agent runs, flushes after. Theon_frameclosure is stashed at module scope (it can't cross the@toolJSON-schema boundary — same constraint documented in start_cameras_recording daemon-thread artifacts under multi-threaded eval (StrandsAgenttool dispatch) robots#191) and the tool wrapper threads it intoevaluate_benchmark.Test surface
TestCamerasRecording(11 cases): lifecycle, error paths (no-world / unknown camera / no cameras / double-start), idempotent stop, full capture→encode cycle asserting the MuJoCo-compatible{name}__{camera}.mp4filename + a non-empty MP4, stale-closure no-op after stop, render-failure error counting, frame cap, anddestroy()cleanup.importorskip("imageio")so the lightweighthatch run testvenv (which doesn't ship imageio) skips them cleanly; they run and pass in a full env.hatch run lint✅ andhatch run test✅ both pass locally (212 passed, 35 skipped). No new failures. (The pre-existing X11/OpenGL environmental skips/failures intest_rendering.py/test_policy_runner.pyare unaffected — those tests aren't instrands_robots_sim/isaac/tests/.)Acceptance criteria
run_isaac.pyandrun_isaac_agent.pywrite a realrollouts/<date>/<name>__<camera>.mp4, like the MuJoCo scripts.videos=line points at a file that exists.Out of scope (no follow-up filed yet)
render()frame path (feat(isaac): wire add_camera Phase 2 + new remove_camera (refs #14) #61/feat(isaac): wire render frame-path Phase 2 (refs #14) #62).run_isaac_agent.pyto the full AgentTool enum onceIsaacSimulationinheritsAgentTool(tracked under R7: Add IsaacSimulation(SimEngine) backend (epic — Stage 3) #14) — the module-scope_on_frameindirection goes away then.Closes #112. Related: #101 (6.0 migration / render works), #61/#62 (camera + render frame-path), strands-labs/robots#191 (on_frame synchronous-recorder rationale).