Training a robot policy needs demonstrations — records of “in this scene, the robot moved like this”. The catch is that such data does not exist on the internet. Teams therefore choose between, and mix, real teleoperation data collected by people driving robots and synthetic data produced through simulation or generation techniques. This article compares the two using the actual scale and results reported in landmark papers, then lays out a practical order for mixing them.
Criteria for choosing a data strategy
Four criteria make the comparison practical:
- Collection speed and staffing — how many demonstrations you can obtain in a given period
- Realism — how much real contact, material, lighting and sensor noise the data contains
- Diversity — how varied the scenes, objects, tasks and robot types are
- Match with your robot — whether camera placement, action space and control rate line up
Real teleoperation data: faithful to reality, but slow
DROID is the reference example of in-the-wild real data. Fifty data collectors spent 12 months gathering 76,000 demonstration trajectories across 564 scenes, totaling 350 hours of interaction. What sets it apart is that the data comes from many real environments rather than a single lab.
Another route is pooling data across institutions. Open X-Embodiment combines data collected by 21 institutions with 22 different robots, covering 527 skills and 160,266 tasks. The paper reports that RT-X models trained on it show positive transfer and improve the capabilities of multiple robots.
The strengths of real data are clear: slips at the moment of contact, real object weight and material, and genuine camera noise are all in the data. The cost is that people must drive robots one episode at a time, which caps how fast the dataset grows, and data from a different robot often has a different action space and cannot be used directly.
Synthetic data, part 1: multiplying human demonstrations
For manipulation, the most efficient form of synthetic data is taking a few human demonstrations and varying them in simulation. MimicGen generated over 50,000 demonstrations across 18 tasks from about 200 human demonstrations, and its authors report that the generated data compares favorably with collecting additional human demonstrations.
The same idea has been extended to two-handed humanoid manipulation. DexMimicGen targets bimanual humanoids with dexterous hands, generated 21,000 demonstrations from 60 human demonstrations, and was also applied to a real humanoid can-sorting task.
| Approach | Input | Output | Distinguishing trait |
|---|---|---|---|
| DROID (real) | 50 collectors, 12 months | 76k trajectories, 350 hours, 564 scenes | Real-world diversity |
| Open X-Embodiment (pooled real) | 21 institutions, 22 robot types | 527 skills, 160,266 tasks | Cross-robot transfer |
| MimicGen (synthetic) | ~200 human demos | 50k+ demos, 18 tasks | Multiplied in simulation |
| DexMimicGen (synthetic) | 60 human demos | 21k demos | Bimanual, dexterous hands |
As the table shows, synthetic generation yields hundreds of demonstrations per human demonstration. The quality of the output, however, rarely exceeds the quality of the source demonstrations and the simulated scenes.
Synthetic data, part 2: rendered images for perception
Separate from action data, perception models such as object detectors are often trained on rendered images. Tobin et al. (2017) report that a detector trained only on simulated images with unrealistic random textures located real objects to within 1.5 cm — evidence that breadth of variation can matter more than photorealism. Techniques for closing the visual and dynamics gaps are covered in Closing the Sim-to-Real gap.
An order for mixing the two
- Start with a small, high-quality real dataset. Collect tens to hundreds of teleoperated demonstrations on your own robot, camera layout and task.
- Build the simulation scene. Recreate the workbench, objects and camera poses to match reality.
- Expand with synthetic data. Vary demonstrations MimicGen-style, or randomize object positions and lighting.
- Use public datasets for pre-training. Data from other robots can still help during pre-training; fine-tune on your own data.
- Tune the mix by real-robot success rate. Change the synthetic share and compare real success under identical conditions.
Large VLA projects point the same way: GR00T N1 is described as trained on a mix of real robot trajectories, human videos and synthetic data. How such mixes feed different model architectures is summarized in VLA architectures compared.
Keeping source demonstrations clean
Because synthetic pipelines copy and vary the source demonstrations, the quality of those sources caps everything generated from them. Hugging Face’s LeRobot guide to imitation learning on real robots recommends starting with a simple pick-and-place task, recording at least 50 episodes — 10 for each of five object positions — and keeping the cameras fixed. It also suggests checking that a person could do the task looking only at the camera feed. The same rules apply when collecting the seeds for synthetic generation.
- Keep the operator’s strategy consistent. Demonstrating the same task a different way each time can teach the model an averaged motion that matches none of them.
- Label or drop failed episodes. Multiplying a source set that contains failures multiplies the failures too.
- Increase variation gradually. Large changes in lighting, objects and camera pose from the start keep a small dataset from converging.
Aligning data from other robots
Before using a public dataset for pre-training, convert the following to your robot’s conventions, or at least record the differences:
- Action space — joint angles or end-effector pose deltas, and the range of the gripper value
- Control rate — data recorded at 10 Hz and a robot running at 50 Hz have different time steps, so resampling is needed
- Camera setup — whether there is a wrist camera, the viewpoints and the resolution
- Frames and units — position units (m or mm) and rotation representation (Euler angles or quaternions)
Merging data without reconciling these is a common way to get a model that trains smoothly but moves in the wrong direction on the real robot.
What to weigh last: match before volume
Match with your robot comes before sheer volume. However much data you add from a different camera placement or action space, final performance usually depends on the small dataset collected on your own robot and how well it was multiplied. When drawing up the plan, schedule the real-robot tests at the same time as the data collection, so the mix can be corrected early rather than after a large generation run.
A useful budgeting habit is to track three numbers from the start: human demonstrations collected, synthetic demonstrations generated from them, and real-robot success rate after each training round. If success stops improving as synthetic volume grows, the bottleneck has usually moved back to the source demonstrations or the fidelity of the simulated scene, and the next investment should go there rather than into more generation.