How VLA Models Turn Images and Instructions into Joint Commands: RT-2, OpenVLA, π0 and GR00T N1 Compared

How does the sentence “put the red cup on the tray”, plus one camera frame, become a change in a robot arm’s joint angles? This article splits the inside of a vision-language-action (VLA) model into three stages and compares, with figures from the papers, how leading models design the last stage — action generation — differently. It is meant for engineers choosing a VLA for a robot, or anyone who wants the architecture straight before reading the papers.

Three stages inside a VLA

Within the Physical AI stack, the VLA is the model that decides what to do. Internally it usually breaks down into three stages:

1. Vision encoder — turns camera images into visual tokens. The OpenVLA README describes a fused DINOv2 + SigLIP vision backbone.
2. Language-model backbone — reads visual tokens, instruction tokens and robot state together to understand the scene and the task. OpenVLA builds on Llama 2; π0 builds on PaliGemma.
3. Action head — converts the backbone’s output into joint and gripper commands. This is where models differ most.

The first two stages look almost identical to an ordinary vision-language model. The VLA parts ways at stage three, and there are two main branches.

Branch 1: emit actions as tokens, like words (RT-2, OpenVLA)

The RT-2 paper expressed robot actions as text tokens and put them straight into the training data. Changes in position, changes in rotation and the gripper value are divided into numeric bins, and each bin becomes a token. The project page shows token strings such as “1 128 91 241 5 101 127 217” and describes variants built on PaLM-E (12B) and PaLI-X (55B).

The appeal is that the language model barely changes. Acting becomes one more kind of next-token prediction, and knowledge learned from web data carries over. Across more than 6,000 evaluation trials, RT-2 reports substantially better generalization to objects it had not seen during robot training.

OpenVLA implements the same approach as an open model: 7B parameters trained on 970,000 real robot demonstrations, with a 16.5% higher absolute success rate than RT-2-X (55B) across 29 tasks. Turning action tokens back into real values looks like this in concept:

import numpy as np

# Map tokens for one action dimension back to continuous values (conceptual)
def detokenize(token_ids, low, high, n_bins=256):
    centers = np.linspace(low, high, n_bins)
    return centers[np.asarray(token_ids)]

# e.g. a 7-D action = 3 position deltas + 3 rotation deltas + 1 gripper value
action = detokenize(model_output_tokens, low=action_low, high=action_high)

The OpenVLA repository refers to this baseline as “256-bin action discretization”. Because the number of bins is fixed, very fine motions can suffer quantization error, and generating one step of tokens at a time can limit control frequency. The repository points to follow-up methods (OFT and FAST) that it says speed up inference considerably.

Branch 2: generate continuous actions with a generative model (π0, GR00T N1)

The second branch skips tokenization and generates continuous action sequences directly with a generative model. π0 is the reference design.

The key idea is the action chunk: one inference produces 50 steps of continuous actions, which the robot executes at high frequency, so the large VLM does not have to run every control cycle. The diffusion/flow mechanism for handling multiple valid actions is explained in How diffusion policies work.

Dual systems: separate the thinker from the mover (Helix, GR00T N1)

For high-degree-of-freedom robots such as humanoids, some designs split understanding and control into two models.

Figure says Helix runs each system on its own low-power embedded GPU and was trained on about 500 hours of teleoperation data. GR00T N1 trains both modules jointly end to end and is presented as an open foundation model trained on a mix of real robot trajectories, human videos and synthetic data. In both cases the slow system passes “what to do” as a latent representation, and the fast system turns it into smooth joint commands.

The three approaches side by side

A quick calculation shows why chunking matters. With π0’s figures, a 50-step chunk executed at 50 Hz covers one second of motion, while one inference takes about 73 ms on an RTX 4090. The model therefore needs to run only about once per second, and the inference time is a small fraction of the time the chunk takes to play out. A token-by-token head that had to produce every control step separately would need to finish each step within 20 ms to sustain the same 50 Hz rate — a much harder budget for a multi-billion-parameter model on robot hardware.

What to check before picking a VLA for your robot

Published success rates come from particular robots and benchmarks. On your own hardware, it is more realistic to work through these in order:

1. Action dimensions and control rate. A single 7-DoF arm and a 35-DoF upper body call for different architectures.
2. Onboard compute budget. A 7B model in 16-bit precision needs about 14 GB for weights alone (our estimate: parameters × 2 bytes). The method is in Sizing VRAM and TOPS for on-device VLA models.
3. Fine-tuning data and resources. The OpenVLA README says its LoRA fine-tuning example runs on a single 80 GB A100, and on smaller GPUs with at least about 27 GB if the batch size is reduced.
4. License and openness. Confirm whether weights are released and under what commercial terms.

Predicting what the chosen action will lead to is the job of a world model; how the two fit together continues in How world models predict the next scene.