Complete Simulations
A whole robot in a simulator: eyes, brain and legs
A complete robot needs three abilities working together: knowing where it is, deciding what to do, and moving its body. This page explains each one, and how each can be simulated before anything runs on a real robot.
V-SLAM: the compass and map
From its cameras, the robot builds a map of the place and keeps track of where it is in that map. Without it, "go to the kitchen" means nothing.
VLM: the brain
A vision-language model looks at the camera images, reads an instruction such as "bring me the red cup", and decides the next goal: where to go, what to look at, what to pick up.
RL: the legs
A locomotion policy trained with reinforcement learning turns "walk there" into joint movements, keeping the robot balanced on every step.
Why simulate the whole system?
- Safety and cost: a robot that walks into a wall in simulation costs nothing. Real humanoids cost tens of thousands of dollars and can hurt people.
- Ground truth: the simulator knows exactly where the robot and every object are, so you can measure how wrong the SLAM map or the VLM's choices were.
- Repeatable tests: the same scene, the same people, the same instruction, run a thousand times to compare methods fairly.
- Testing the joins: each part can work alone and still fail together, for example if the VLM sends goals faster than the legs can reach them.
Which simulator for which part
No single simulator is best at everything. Some render beautiful camera images, which SLAM and VLMs need; others run fast, accurate physics, which RL needs.
| Simulator | Made by | V-SLAM | VLM | RL locomotion |
|---|---|---|---|---|
| Isaac Sim / Isaac Lab | NVIDIA | Strong: photorealistic cameras, ROS 2 | Strong: realistic scenes | Strong: thousands of robots on one GPU |
| Gazebo | Open Robotics | Strong: standard with ROS 2 | Basic graphics | Slow for RL |
| MuJoCo | Google DeepMind | Basic cameras (colour, depth, segmentation) | Simple scenes | Strong: accurate contacts; GPU versions for fast training |
| Genesis | Genesis open-source project | Built-in ray-traced rendering | Realistic images possible | Strong: very fast GPU physics |
| Habitat | Meta | Realistic scanned homes | Strong: built for embodied AI and navigation | No legged physics |
A common recipe is to train the legs in a fast physics simulator, then test the full robot, with cameras and the brain, in one with realistic graphics.
V-SLAM
V-SLAM: where am I, and what does this place look like?
Visual Simultaneous Localisation And Mapping: building a map from camera images while working out where the camera is inside it.
What it is
It's a chicken-and-egg problem. To know where you are, you need a map. To build a map, you need to know where you are. SLAM solves both at the same time. "Visual" means the main sensor is a camera: one camera, a stereo pair, or an RGB-D camera that also measures depth. Many systems add an IMU, which senses motion, so they are called visual-inertial SLAM.
For a walking robot, V-SLAM is its compass and map. It tells the planner "you are here, facing this way", and the brain "the kitchen is through that door".
How it works
- Find features. Pick out distinctive points in each image, such as corners and edges of furniture.
- Track them. Follow the same points from one frame to the next. How they shift shows how the camera moved. This step alone is called visual odometry.
- Build the map. Place the points in 3D. Keep a set of key frames, the snapshots that best describe the place.
- Close loops. When the robot comes back to somewhere it has seen, recognise it and correct the drift that built up along the way.
- Optimise. Adjust all the poses and points together so the whole map is consistent.
Well-known V-SLAM systems
| System | Sensors | Notes |
|---|---|---|
| ORB-SLAM3 | Mono, stereo, RGB-D, with or without IMU | A widely used research reference. Open source (GPL-3.0). |
| RTAB-Map | RGB-D, stereo, lidar | Builds dense 3D maps; popular with ROS. Open source (BSD). |
| VINS-Fusion | Camera + IMU | Visual-inertial, from HKUST. Open source (GPL-3.0). |
| cuVSLAM (Isaac ROS Visual SLAM) | Stereo + IMU | NVIDIA's GPU-accelerated V-SLAM for robots on Jetson computers. |
How to simulate it
- Put a camera on the robot in the simulator, at the real camera's position, with the same image size, field of view and frame rate. Add a simulated IMU.
- Make it realistic. Add image noise, motion blur and lighting changes. A perfectly clean image makes SLAM look better than it will be on a real robot.
- Stream to the SLAM system, usually through ROS 2 topics, exactly as a real camera would.
- Compare with ground truth. The simulator knows the true robot position. Plot the SLAM estimate against it and measure the error, for example the absolute trajectory error.
| Simulator | How V-SLAM is done there |
|---|---|
| Isaac Sim | Photorealistic cameras and IMU with a ROS 2 bridge; NVIDIA's own V-SLAM runs directly on its output. |
| Gazebo | Camera, depth and IMU sensor plugins publish to ROS 2; the most common setup for testing ORB-SLAM3 or RTAB-Map. |
| MuJoCo | Render colour and depth from a camera on the robot; read the true pose directly. Simple graphics, so features are fewer than in the real world. |
| Genesis | Cameras with built-in rasterised or ray-traced rendering for more realistic images. |
| Habitat | Realistic scanned indoor scenes; good for testing mapping in real homes, but without legged physics. |
VLM
VLM: the brain that understands what it sees
A vision-language model reads images and text together, so a robot can follow everyday instructions.
What it is
A VLM is a large AI model, like the chatbots people use, that also takes images as input. Show it a photo of a room and ask "where could the keys be?" and it can answer "on the table by the door". For a robot, that is the missing piece between seeing pixels and understanding a task.
There are two ways to use one in a robot:
VLM as planner
The VLM decides the next goal: "go to the table", "look left", "pick up the cup". Ready-made skills (the planner, the RL walking policy, a grasping skill) carry it out. Easy to understand and debug.
Vision-language-action model (VLA)
The model is trained further to output robot actions directly, such as arm and hand movements. One network goes from pixels and words to motion.
How it fits with the legs and the map
VLMs are slow, about one decision a second, sometimes less. Legs need new commands 50 times a second. So the brain never controls the joints directly. It sets goals, V-SLAM says where those goals are on the map, and the RL policy does the walking. Several recent humanoid brains use exactly this split into a slow "thinking" part and a fast "acting" part.
Well-known robot brains
| Model | From | What it does | Open? |
|---|---|---|---|
| RT-2 | Google DeepMind, 2023 | Early VLA: a VLM trained to output robot actions as text tokens. | Closed |
| OpenVLA | Stanford and others, 2024 | An open 7-billion-parameter VLA for robot arms. | Open |
| π0 (pi-zero) | Physical Intelligence, 2024 | A general robot policy across many robots and tasks; code and weights released in 2025. | Open |
| GR00T N1 | NVIDIA, 2025 | A foundation model for humanoids with a slow reasoning part and a fast action part; open weights. | Open |
| Gemini Robotics | Google DeepMind, 2025 | Robot models built on Gemini, for understanding scenes and controlling robots. | Closed |
| Helix | Figure AI, 2025 | Figure's humanoid brain, also split into a slow and a fast system. | Closed |
General VLMs that run on your own computer, such as Qwen-VL or LLaVA, are often used as the planner in research.
How to simulate it
- Render what the robot sees from its head camera, at the real camera's resolution.
- Send the image and the instruction to the VLM, together with the robot's position and a list of skills it may call.
- Turn the answer into a goal, for example a point on the SLAM map, and pass it to the planner and the legs.
- Score the result. Did the robot reach the right place, pick the right object, and how long did it take? The simulator can check automatically.
| Simulator | How the VLM is tested there |
|---|---|
| Isaac Sim | Realistic homes, warehouses and factories; used to generate training data and test humanoid brains such as GR00T. |
| Habitat | Scanned real homes with standard tasks such as "find a chair" and "go to the bedroom"; the main benchmark for VLM navigation. |
| MuJoCo | Simple scenes; good for testing the logic of the brain-to-legs connection, less so for visual understanding. |
| Genesis | Ray-traced images can look closer to real camera images. |
RL
RL: legs that learned to walk
A locomotion policy trained with reinforcement learning, which turns a speed command into balanced steps.
What it is
The RL policy is a small neural network that runs about 50 times a second. It reads the robot's tilt, turning rate, joint angles and speeds, and the requested speed and direction. It outputs a target angle for every leg joint, and fast motor loops hold those angles. It never sees the camera and never thinks about the task: it just walks where it is told, without falling. The Background page explains how it learns, and the Quadrupeds and Humanoid pages show which robots use it.
In the complete system, the RL policy is the bottom layer. Its input is the speed command from the planner; its output goes to the motors.
How to simulate it
- Load the robot model (its links, joints, masses and motor limits), usually from the maker's files.
- Train in a fast simulator with thousands of robots at once, random terrain, random pushes and random physics settings.
- Check it in a second simulator ("sim-to-sim"). A policy that only works in one physics engine is probably exploiting its quirks.
- Connect it to the rest: the planner sends speed commands, and the policy walks while V-SLAM and the VLM keep working.
| Simulator | Role for RL locomotion |
|---|---|
| Isaac Lab | The most common place to train today: thousands of robots on one GPU; used by Unitree, Booster and many labs. |
| MuJoCo | Accurate contacts; the usual sim-to-sim check. GPU versions (MJX, MuJoCo Playground, mjlab) also train policies quickly. |
| Genesis | Very fast GPU physics, designed for training robot policies at scale. |
| Gazebo | Rarely used for training, but useful for running a trained policy alongside ROS 2. |
Experiments
MuJoCo simulation
Experiments