Growing Robot Training Data with World Models: Inside NVIDIA's Physical AI Data Factory Blueprint

There are two broad ways to grow a robot policy’s training data. One varies demonstrations inside a simulator to produce new trajectories. The other keeps existing footage but uses a generative model to change lighting, background and texture so the policy sees more visual variety. NVIDIA’s Physical AI Data Factory Blueprint, announced at GTC 2026, packages the second approach into curation, augmentation and evaluation stages. This article explains what each component does and how available it actually is, then lays out when to combine it with simulation-based synthetic data.

One-sentence definition: a blueprint for producing, multiplying and filtering data

The NVIDIA press release of March 16, 2026 describes the blueprint as an open reference architecture that unifies and automates how training data is generated, augmented and evaluated. It targets vision AI agents, robotics and autonomous vehicles, and supports data curation, synthetic data generation, reinforcement learning and model evaluation.

The important point is that it is not a new model but a prescribed order for existing tools. Each component is an independent project with its own repository and license; the blueprint defines how they connect and which outputs flow to the next stage.

What the four components do

Role descriptions follow the press release, and release status follows each repository’s README. Cosmos Curator lists video splitting, annotation, filtering, deduplication and dataset generation as its features, with distributed processing through Ray Data and the Cosmos-Xenna library.

The press release said Evaluator and the Azure toolchain were on GitHub at announcement time and that the full blueprint was expected in April. At the time of writing, official documentation reviewed for this article did not identify a single repository that bundles the whole blueprint, so starting from the individual component repositories is the realistic path.

The augmentation stage: changing a scene through control inputs

Cosmos Transfer, the core of augmentation, does not edit the source video pixel by pixel. It extracts structural signals from the source and generates new video conditioned on them. According to the Cosmos Transfer 2.5 README, the model is a multi-controlnet that accepts several video modalities, with depth, segmentation, edge and blur as control inputs. Its “Simulation 2 Real Augmentation” example computes segmentation, depth and edge maps from a simulated robot-arm video and uses them to generate realistic footage, which reduces how much effort goes into making the 3D simulation itself look photoreal.

Version status matters here. The same README states that the Transfer 2.5 repository is no longer under active development and will receive only limited maintenance, and it directs new work to Cosmos 3. The NVIDIA/Cosmos repository describes Cosmos 3 as a suite of omnimodal world models built on a Mixture-of-Transformers architecture, exposing two interfaces from one model: a Reasoner that understands text and vision, and a Generator that produces vision, sound and action. It comes in three sizes — Super 64B, Nano 16B and Edge 4B — under the OpenMDW-1.1 license. How a world model predicts the next scene in the first place is covered in world model latent-space architecture.

The evaluation stage: deciding whether generated video can be trusted

A generative model can invent objects that were not in the source or erase ones that were. When augmentation reuses the action labels from the original demonstration, that is fatal: if a cup moves in the generated video while the recorded action still reaches for its old position, the result is a wrong training example.

The checks published in the Cosmos Evaluator README target exactly this problem:

• Hallucination Check compares motion masks between the original and augmented video to find newly appearing or vanished moving objects.
• Objects Check scores how well objects in the generated video (vehicles, pedestrians, infrastructure) match their expected positions from world-model ground truth.
• Attribute Verification Check asks a vision-language model questions to confirm that the augmented video actually contains the requested attributes, such as rain or night lighting.

All checks are callable as REST microservices or through Python APIs, and an orchestration layer bundles an arbitrator service with a PostgreSQL-backed store. Because the published objects check is described mainly with driving-scene examples, robot manipulation users need to define their own criterion for confirming that the grasped object and the hand stay where they were.

How this differs from simulation-based synthetic data

The MimicGen family covered in real teleoperation data versus synthetic data moves objects inside a simulator to create new trajectories. Generative video augmentation keeps the trajectory and changes only what the camera sees. They are not competitors; they work on different axes.

If a policy fails when lighting or textures change, start with video augmentation; if it fails when an object moves a few centimeters, start with simulated multiplication. It also helps to separate this from the domain randomization covered in closing the sim-to-real gap: domain randomization perturbs visual conditions inside the simulator, while video augmentation makes already-rendered output look realistic.

Estimating usable data before you commit

The real output of an augmentation pipeline is set by how many clips pass evaluation, not by how many are generated. With $N$ source episodes, $k$ variants generated per source episode and an evaluation pass rate $r$, the number of usable augmented episodes is:

For example, assuming 200 source episodes, five variants each and a pass rate of 0.6 gives 600 augmented episodes, or 800 together with the originals. These are illustrative assumptions; the real $r$ depends heavily on scene complexity and evaluation criteria, so measure it on a small trial batch first. Generation cost scales with $N \cdot k$ while usable volume scales with $r$, so finding control inputs and prompts that raise $r$ usually saves more GPU time than raising $k$.

A practical adoption order:

1. Log whether policy failures come from observation problems (lighting, background) or from placement and contact problems.
2. If observation problems dominate, clean the source demonstrations with Curator and document the quality criteria (length, blur, duplicates).
3. Run a trial augmentation on a few dozen source clips, measure $k$ and $r$, and have a person review a sample of the clips that passed.
4. Compare a policy trained with augmented data against one trained on the originals alone, from identical initial conditions.

Training an actual policy on the augmented data follows the data format in the GR00T N1.7 fine-tuning guide. For where this pipeline sits in NVIDIA’s training, simulation and deployment flow, continue with the Cosmos, Isaac and GR00T three-computer architecture.