Skip to content

fix(isaac): record rollout videos at MuJoCo parity (closes #112)#114

Merged
cagataycali merged 1 commit into
strands-labs:mainfrom
yinsong1986:fix/isaac-cameras-recording
Jun 19, 2026
Merged

fix(isaac): record rollout videos at MuJoCo parity (closes #112)#114
cagataycali merged 1 commit into
strands-labs:mainfrom
yinsong1986:fix/isaac-cameras-recording

Conversation

@yinsong1986

Copy link
Copy Markdown
Contributor

What

Brings the Isaac LIBERO examples to parity with their MuJoCo counterparts on rollout-video output. They now write a real rollouts/<date>/<name>__<camera>.mp4 instead of an …__image.mp4.placeholder path string that was never written to disk.

Why

run_mujoco.py / run_mujoco_agent.py record a real MP4 via a start_cameras_recording / stop_cameras_recording pair. run_isaac.py / run_isaac_agent.py recorded nothing — they only emitted a placeholder path in the grep-stable videos= line, so any tooling that trusts videos= (e.g. the R15 backend matrix's rollouts/*/*--task=*.mp4 glob) got a dangling/blank entry for Isaac rows.

The deferral rationale was stale: IsaacSimulation had no recorder API (start_cameras_recording/stop_cameras_recording are MuJoCo-Simulation-specific, not on the base SimEngine), and the comments cited "render() returns blank frames". With #61/#62 merged, IsaacSimulation.render() returns real RGB frames on Isaac Sim 6.0, so videos can be produced.

Approach

This implements both suggested fixes from the issue — a backend recorder and the example-level on_frame wiring — because they compose: the recorder is synchronous (Isaac's RTX renderer + Camera.get_rgba are bound to the thread that booted SimulationApp, so a MuJoCo-style daemon recorder would deadlock), so it relies on evaluate_benchmark's on_frame= hook to capture one render() frame per control step on the eval thread.

Backend (strands_robots_sim/isaac/simulation.py)

  • start_cameras_recording(cameras, output_dir, fps, name, max_frames_per_camera) — sets up one in-memory RGB buffer per camera and returns an on_frame(step, obs, action) closure in the result's json block.
  • stop_cameras_recording() — flushes buffers to {name}__{camera}.mp4 via imageio (the same encoder the MuJoCo recorder uses), matching MuJoCo's filename/layout convention exactly. Idempotent.
  • Errors loudly on no-world / double-start / unknown-camera / no-cameras; per-camera render failures increment an error counter rather than aborting the eval; recorder state is cleared on destroy().

Examples

  • run_isaac.py — replaces the .placeholder with a real start/stop pair around evaluate_benchmark(on_frame=...). videos= now points at a file that exists.
  • run_isaac_agent.py — arms the recorder before the agent runs, flushes after. The on_frame closure is stashed at module scope (it can't cross the @tool JSON-schema boundary — same constraint documented in start_cameras_recording daemon-thread artifacts under multi-threaded eval (Strands Agent tool dispatch) robots#191) and the tool wrapper threads it into evaluate_benchmark.
  • Stale "blank frames" / "intentionally skipped" comments dropped; docstrings updated.

Test surface

  • New TestCamerasRecording (11 cases): lifecycle, error paths (no-world / unknown camera / no cameras / double-start), idempotent stop, full capture→encode cycle asserting the MuJoCo-compatible {name}__{camera}.mp4 filename + a non-empty MP4, stale-closure no-op after stop, render-failure error counting, frame cap, and destroy() cleanup.
  • The two MP4-encode cases importorskip("imageio") so the lightweight hatch run test venv (which doesn't ship imageio) skips them cleanly; they run and pass in a full env.

hatch run lint ✅ and hatch run test ✅ both pass locally (212 passed, 35 skipped). No new failures. (The pre-existing X11/OpenGL environmental skips/failures in test_rendering.py / test_policy_runner.py are unaffected — those tests aren't in strands_robots_sim/isaac/tests/.)

Acceptance criteria

  • run_isaac.py and run_isaac_agent.py write a real rollouts/<date>/<name>__<camera>.mp4, like the MuJoCo scripts.
  • The videos= line points at a file that exists.
  • Filename/layout matches MuJoCo so the backend-matrix glob picks up Isaac rows.

Out of scope (no follow-up filed yet)

Closes #112. Related: #101 (6.0 migration / render works), #61/#62 (camera + render frame-path), strands-labs/robots#191 (on_frame synchronous-recorder rationale).

…bs#112)

The Isaac LIBERO examples wrote no rollout video -- only a
`…__image.mp4.placeholder` path string (never written to disk) -- while
their MuJoCo counterparts produce a real `{name}__{camera}.mp4` via
`start_cameras_recording` / `stop_cameras_recording`. The deferral
rationale ("render() returns blank frames", "recorder is a separate
slice") is stale: with strands-labs#61/strands-labs#62 merged, `IsaacSimulation.render()`
returns real RGB frames on Isaac Sim 6.0.

Backend recorder:
- Add `IsaacSimulation.start_cameras_recording` /
  `stop_cameras_recording`. Capture is synchronous (Isaac's RTX renderer
  + Camera.get_rgba are bound to the SimulationApp thread, so a
  MuJoCo-style daemon recorder would deadlock): `start` returns an
  `on_frame(step, obs, action)` closure that the eval driver wires into
  `evaluate_benchmark(on_frame=...)` (strands-robots 0.4.0 signature).
  It grabs one `render()` frame per applied control step into an
  in-memory buffer; `stop` flushes to `{name}__{camera}.mp4` via imageio
  -- the exact filename/layout convention MuJoCo uses, so cross-backend
  video discovery (the R15 backend-matrix glob) picks up Isaac rows.
- Recorder state is cleared on `destroy()`; double-start, unknown
  camera, and no-camera cases error loudly; per-camera render failures
  increment an error counter instead of aborting the eval.

Examples:
- `run_isaac.py`: replace the `.placeholder` with a real
  start/stop_cameras_recording pair around `evaluate_benchmark`; the
  grep-stable `videos=` line now points at a file that exists.
- `run_isaac_agent.py`: arm the recorder before the agent runs and flush
  after. The `on_frame` closure is stashed at module scope (it can't
  cross the @tool JSON-schema boundary, same constraint as #191) and the
  tool wrapper threads it into `evaluate_benchmark`.
- Drop the stale "blank frames" / "intentionally skipped" comments.

Tests: add `TestCamerasRecording` (11 cases) covering the lifecycle,
error paths, frame-cap, MuJoCo-compatible filename, and destroy
cleanup. The two MP4-encode cases `importorskip("imageio")` so the
lightweight hatch test env (no imageio) skips them cleanly.
@yinsong1986 yinsong1986 requested a review from cagataycali June 18, 2026 12:29
@cagataycali cagataycali merged commit d1b5cfb into strands-labs:main Jun 19, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Isaac LIBERO examples don't record rollout videos like the MuJoCo ones (only a .mp4.placeholder); IsaacSimulation lacks start/stop_cameras_recording

2 participants