If a robot pushes a cup, will the cup slide or tip over? People picture the outcome before they move. A world model is a neural network that does the same kind of imagining. This article explains the basic structure using the actual design values from the 2018 paper that popularized the idea, then shows how the same principle scaled up in DreamerV3 and NVIDIA Cosmos.
What a world model is: put in an action, get the next state
A world model takes the current observation and an action and predicts the next state. In the overall Physical AI stack, a VLA decides what to do, and a world model answers what will happen if it does.
Most world models do not predict full-resolution images directly. They first compress each observation into a small latent vector and learn how that vector changes over time. This cuts computation sharply, and it lets the policy decide from a few dozen summarized numbers instead of tens of thousands of pixels.
The three parts of World Models (2018): V, M and C
Ha and Schmidhuber’s World Models split the agent into three parts. The figures below come from the authors’ interactive article.
| Part | Job | CarRacing parameters | VizDoom parameters |
|---|---|---|---|
| V (VAE) | Compresses a 64×64 frame into a latent vector z (32-dim / 64-dim) | 4,348,547 | 4,446,915 |
| M (MDN-RNN) | Predicts a distribution over the next z (LSTM with 256 / 512 units, 5 Gaussian mixtures) | 422,368 | 1,678,785 |
| C (Controller) | One linear layer mapping z and M’s hidden state to an action | 867 | 1,088 |
The striking number is the size of C. Once representation (V) and prediction (M) were handled by larger models, the part that actually chooses actions needed only an 867-parameter linear layer. That agent scored 906 ± 21 averaged over 100 random CarRacing tracks, above the 900-point threshold for solving the task.
Training in the dream — and the exploit problem
The most cited experiment is VizDoom. The controller was trained only inside the environment generated by M — the “dream” — and then dropped into the real game. According to the authors’ write-up, it survived 1092 ± 556 time steps in real VizDoom, well above the 750 required.
The same experiment exposes the weakness. The temperature τ controls how random M’s samples are. At τ = 0.10 the dream monsters almost never fired fireballs; the controller learned to exploit that, scoring 2086 in the dream but only 193 in the real game. Raising τ to 1.15 made the dream noisier and gave the best transfer. The lesson generalizes: a policy optimizes whatever the model gets wrong, too.
Rolling the future forward in latent space
Training a policy with a world model can be reduced to the loop below. Real implementations differ, but the order of operations is the same.
# Conceptual: imagine H steps ahead inside the world model and score the policy
z = encoder(observation) # observation -> latent vector (V)
h = rnn.initial_state()
total_reward = 0.0
for t in range(H):
action = controller(z, h) # policy (C)
z, h, reward = dynamics(z, h, action, temperature=tau) # next-state prediction (M)
total_reward += reward
# update controller parameters from total_reward (CMA-ES, actor-critic, ...)
Here encoder, dynamics and temperature correspond to V, M and τ. The practical value of a world model is that this loop alone can run thousands of trial-and-error episodes without moving a real robot.
DreamerV3: the same idea as general-purpose RL
DreamerV3 learns a model of the environment and improves its behavior by imagining future scenarios inside it. With added normalization, balancing and transformation techniques, the paper reports outperforming specialized methods on more than 150 diverse tasks using a single fixed configuration, and calls it the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. Not having to retune hyperparameters per task matters most where each experiment is expensive — which describes robotics.
NVIDIA Cosmos: world models as foundation models
The Cosmos paper proposes world foundation models (WFMs): train one large general-purpose world model, then post-train it for a particular robot, vehicle or camera setup. The published components are:
| Item | Details (from the paper) |
|---|---|
| Data | About 100 million 2–60 second clips extracted from roughly 20 million hours of raw video; semantic deduplication removes about 30% |
| Tokenizers | Continuous (CV, for diffusion) and discrete (DV, for autoregressive); time × height × width compression of 4×8×8, 8×8×8 or 8×16×16 |
| Diffusion family | Cosmos-Predict1-7B and 14B (Text2World, Video2World) |
| Autoregressive family | Cosmos-Predict1-4B and 12B (Video2World 5B and 13B) |
| Post-training examples | Camera control, robotic manipulation, autonomous driving |
| Safety | A Pre-Guard that blocks harmful inputs and a Post-Guard that filters outputs, including face blurring |
Where the 2018 model compressed one game screen into a 32-dimensional vector, Cosmos does the same job with tokenizers for real-world video and prediction models with billions of parameters. Post-training for manipulation feeds video together with action sequences, so the model learns what the next frame looks like after a given action. The imagined experience can also be used as synthetic training data; how synthetic data compares with real teleoperation data is covered in Synthetic data for robot training.
Common misreadings before you use one
- Photorealistic is not physically correct. A natural-looking clip can still get contact forces or slipping wrong. For robotics, evaluate with downstream task metrics such as success rate.
- Out-of-distribution inputs break it. Camera angles, lighting or objects absent from training can collapse the predictions, which is why post-training data should match your own setup.
- The policy will find the holes. As the VizDoom τ experiment shows, choosing a policy by its reward inside the model can degrade real performance.
Pixel-space or latent-space prediction: which one you need
The three systems above sit at different points on one design axis. World Models and DreamerV3 predict in a compact latent space and never need to render a full frame to train a policy; that keeps imagination cheap enough to run millions of steps. Cosmos predicts video (through tokenized latents that decode back to frames), which costs far more compute but produces output a person — or a perception model — can inspect directly.
A practical rule of thumb follows from that difference. If the goal is to train or evaluate a control policy for one robot, a latent-space model trained on that robot’s own data is usually the cheaper path. If the goal is to generate varied, realistic training scenes for many robots, or to let engineers review what the model “thinks” will happen, a video-space foundation model that you post-train is the better fit. Many teams combine the two: generate scenes with a large model, then train a compact policy from them.
Next: turning predicted futures into actions
A world model predicts outcomes; a VLA produces the actual joint commands. How images and instructions become action tokens or continuous actions is the subject of How VLA models turn vision and language into robot actions.