The GPU-accelerated NVIDIA Isaac Sim backend for strands-robots
Docs ◆ strands-robots ◆ Robots Docs ◆ NVIDIA Isaac Sim ◆ Project Board
strands-robots-sim is the GPU-accelerated Isaac Sim companion to
strands-robots. It ships an
IsaacSimulation that plugs into the same SimEngine ABC the upstream
MuJoCo backend implements, so a Strands Agent that drives a MuJoCo world
today switches to Isaac Sim by changing one string.
import strands_robots_sim # registers "isaac" via entry points
from strands_robots.simulation import create_simulation
sim = create_simulation("isaac", render_mode="rtx_realtime", headless=True)
sim.create_world()
sim.add_robot("so100") # procedural; no asset files needed
sim.step(100)
frame = sim.render(camera_name="default")📚 Documentation: https://strands-labs.github.io/robots-sim/
Includes a Quickstart, the Architecture behind the plugin contract, the Backend reference, and Troubleshooting.
- RTX path-traced rendering. Photoreal observations, paper-grade frames, sim2real visuals — driven by the same agent code your MuJoCo smoke tests use.
- USD-native scenes. Real CAD assets, Nucleus support, IsaacLab compatibility.
- Replicator synth-data. Domain randomization at scale; ground-truth depth, segmentation, and bounding boxes alongside RGB.
- Fleet RL on PhysX GPU. 1024+ parallel envs with shared USD scenes.
- Plugin shape, not a fork.
strands-robotshas zero hard dependency on Isaac Sim — install this package and Isaac is discovered through entry points.
If none of those apply, install the lightweight default at
strands-labs/robots instead —
it runs everywhere (including Apple Silicon), boots in seconds, and the
agent contract is identical.
graph LR
A[Strands Agent] --> B[Simulation<br/>AgentTool]
B --> C[create_simulation 'isaac']
C --> D[Entry-point lookup<br/>strands_robots.backends]
D --> E[IsaacSimulation<br/>this repo]
E --> F[Isaac Sim Kit<br/>SimulationApp]
F --> G[PhysX + RTX]
classDef agent fill:#0969da,stroke:#044289,color:#fff
classDef glue fill:#8250df,stroke:#5a32a3,color:#fff
classDef plugin fill:#bf8700,stroke:#875e00,color:#fff
class A,B agent
class C,D glue
class E,F,G plugin
strands-robots-sim registers IsaacSimulation as a
strands_robots.backends entry point. Upstream create_simulation("isaac")
walks the entry-point group, imports the target string, and instantiates
it. The full plugin contract is documented in
Architecture.
System requirements: NVIDIA RTX GPU, Ubuntu 22.04+, CUDA 12+, Isaac Sim 4.x.
macOS / Apple Silicon contributors should install
strands-robots directly and
skip this repo.
# Step 1 — install Isaac Sim itself (NOT on PyPI):
# - Omniverse Launcher → Isaac Sim 4.x, OR
# - Isaac Lab: git clone IsaacLab && ./isaaclab.sh -i, OR
# - NGC Docker: docker pull nvcr.io/nvidia/isaac-sim:4.5.0
# Step 2 — install the Python plugin:
pip install 'strands-robots-sim[isaac]'strands-robots (the upstream MuJoCo backend, Simulation AgentTool,
and policy providers) is pulled in transitively.
Full install matrix in Getting Started → Installation.
import strands_robots_sim
from strands_robots.simulation import create_simulation
sim = create_simulation("isaac", render_mode="rtx_pathtracing", headless=True)
sim.create_world()
sim.add_robot("so100") # procedural builder
sim.add_object(name="cube", shape="cuboid",
position=[0.4, 0.0, 0.05], scale=[0.05, 0.05, 0.05])
sim.add_camera(name="front", position=[1.2, 0.0, 0.6], target=[0.0, 0.0, 0.1])
sim.step(120)
frame = sim.render(camera_name="front") # {"rgb": ..., "depth": ...}
sim.destroy()sim = create_simulation("isaac", num_envs=1024, headless=True,
render_mode="headless")
sim.create_world()
sim.add_robot(name="panda", usd_path="/path/to/franka.usda")
# ... RL training loop ...from strands_robots_sim.isaac.loaders import load_urdf, load_mjcf, load_usd
panda = load_urdf("/path/to/panda.urdf") # stdlib XML, no extra deps
print(panda.num_joints, panda.joint_names)| Capability | MuJoCo ( strands-robots) |
Isaac Sim (this repo) |
|---|---|---|
| Native GPU | — | ✅ PhysX |
| Apple Silicon | ✅ | ❌ |
| Photoreal rendering | — | ✅ RTX path-traced |
num_envs per GPU |
1–8 | 1024+ |
| USD scene format | — | ✅ native |
| Synthetic data (Replicator) | — | ✅ |
| Download size | ~50 MB | ~30 GB |
| Setup friction | low | high |
- Pick MuJoCo for fast iteration, debugging, macOS / Apple Silicon, CI.
- Pick Isaac Sim for RTX photoreal eval, USD-native scenes, IsaacLab fleet RL, or Replicator synth-data.
The full per-backend table with install hints lives in Simulation → Overview.
IsaacSimulation exposes the same SimEngine shape MuJoCo does:
World & lifecycle create_world / destroy / reset / step / get_state / cleanup
Robots add_robot (procedural | USD | URDF) / remove_robot / list_robots
robot_joint_names / send_action / get_observation
Objects add_object (cuboid / sphere / cylinder / capsule, dynamic / static)
remove_object
Cameras & rendering add_camera (look-at, FOV) / render (RGB + depth)
Loaders load_urdf / load_mjcf / load_usd → ProceduralRobot dataclass
Full reference: Simulation → World Building.
The repo ships runnable LIBERO drivers under examples/libero/:
| File | Driver | Best for |
|---|---|---|
libero/run_isaac.py |
Programmatic — evaluate_benchmark(...) |
CI / matrix tables |
libero/run_isaac_agent.py |
Strands Agent + natural language |
Demos |
libero/libero_backend_matrix.py |
Flagship matrix runner — walks all backend rows (R15 / #22) | Side-by-side backend comparison |
Plus visually-driven demos:
examples/isaac_gs/— Isaac Sim + 3DGS hybrid-render digital twinexamples/so101_curobo/isaac/— SO-101 synth-data with cuRobo on Isaac
# Smoke test (mock policy, no GPU policy server):
python examples/libero/run_isaac.py --policy mock --n-episodes 5
# Real eval against an NVIDIA GR00T checkpoint:
python examples/libero/run_isaac.py --policy groot --port 8000 --n-episodes 50Full driver matrix: Examples → Overview.
strands_robots_sim/
├── __init__.py # PEP 562 lazy exports
└── isaac/
├── __init__.py # IsaacConfig, IsaacSimulation lazy exports
├── _install.py # single source of truth for install metadata
├── config.py # IsaacConfig dataclass + validation
├── simulation.py # IsaacSimulation(SimEngine) — main backend
├── procedural.py # SO-100 / Panda / G1 builders + tree validator
├── loaders.py # URDF / MJCF / USD → ProceduralRobot
└── tests/ # unit + GPU-integration tests
docs/ # MkDocs Material site (deployed to GitHub Pages)
examples/
├── libero/ # LIBERO drivers (run_isaac.py + run_isaac_agent.py)
├── isaac_gs/ # Isaac Sim + 3DGS hybrid-render demo
└── so101_curobo/ # SO-101 synth-data with cuRobo on Isaac
- v0.2.0 — re-scoped foundation (Isaac plugin shape, MuJoCo back upstream).
- v0.3.0 — Isaac Sim Phase 1 (entry-point registration,
IsaacConfig,IsaacSimulationlifecycle, procedural builders, URDF / MJCF / USD loaders) shipped onmain; data-plane wiring (add_object,add_camera,render, USD- and URDF-loaded articulations) shipping incrementally. - R9 — Replicator synth-data pipeline, tracked in #16.
- R23 — IsaacLab-style fleet driver, tracked in #27.
Track the umbrella roadmap at #8.
Migration from the legacy 0.1.x API: examples/MIGRATION.md.
pip install -e '.[isaac,dev]'
hatch run lint # black --check + isort --check + flake8
hatch run format # black + isort
hatch run test # pytest strands_robots_sim/isaac/tests/
# GPU integration tests (requires Isaac Sim on the host):
STRANDS_GPU_TEST=1 pytest strands_robots_sim/isaac/tests/test_gpu_integ.py -vDocs are MkDocs Material; build locally with:
pip install -r docs/requirements.txt
mkdocs serve # live-reload at http://127.0.0.1:8000
mkdocs build --strict # CI's checkPython 3.10+ required (mirroring Isaac Sim 4.5's bundled interpreter). See Contributing for the full guide.
Found a vulnerability? Do not open a public issue. Follow the disclosure process in SECURITY.md (AWS VDP / HackerOne).
Issues and PRs welcome. Track work on the Strands Labs - Robots project board; it is the source of truth for roadmap and follow-ups.
Apache-2.0 — see LICENSE.