How Much GPU Memory Does SmolVLA Fine-Tuning Need? A VRAM Calculation

The SmolVLA paper describes its model as designed to be trained on a single GPU and deployed on consumer-grade GPUs or even CPUs. That does not mean any GPU can train it with the official example settings unchanged. This article splits the GPU memory needed to fine-tune SmolVLA into weights, gradients, optimizer state and activations, then shows how to pick a batch size for your own card. The numbers are either values from the paper, the model card and the LeRobot config file, or estimates computed from them, with the assumptions stated.

If you have not settled on a model yet, choosing between ACT, Diffusion Policy and SmolVLA matches models to data size.

Where SmolVLA’s 450M parameters go

According to the SmolVLA paper, the main model has 450M parameters, about 100M of which belong to the action expert that generates action chunks. The rest is a VLM based on SmolVLM-2. The paper uses only the first 16 layers of the VLM’s language model and limits visual tokens to 64 per frame. The model card rounds the same model to 0.5B.

For training memory, what matters is not the total size but which parameters are trained. These are the defaults in configuration_smolvla.py:

So default fine-tuning trains roughly the 100M-parameter action expert plus the state projection, not all 450M.

Splitting training memory into four parts

GPU memory during training can be treated as the sum of four terms:

• $P_{frozen}$: number of frozen parameters; $b_w$: bytes per frozen weight (4 for FP32, 2 for BF16)
• $P_{train}$: number of trainable parameters
• $M_{act}(B)$: activation memory, which grows with batch size $B$
• $M_{overhead}$: the CUDA context and memory reserved by the caching allocator

The 16 bytes per trainable parameter is the figure the ZeRO paper gives for mixed-precision Adam: 2 bytes of FP16 weights, 2 bytes of FP16 gradients, and 12 bytes for the FP32 weight copy, momentum and variance. Training fully in FP32 lands on the same 16 bytes: 4 for weights, 4 for gradients and 8 for momentum and variance. SmolVLA’s optimizer defaults (optimizer_betas (0.9, 0.95), optimizer_eps 1e-8) are Adam-style, so this figure applies.

Using the paper’s numbers (450M total, about 100M in the action expert), the fixed cost works out as follows. 1 GB is taken as $10^9$ bytes.

On fixed cost alone, the default setup leaves plenty of room even on consumer cards. What actually limits batch size is the last two terms, above all activation memory. The model card’s example command also uses --policy.dtype=bfloat16 and --batch_size=4, and lists gradient checkpointing as an option.

Counting trainable parameters in code

The paper’s figures are approximate, so it is more accurate to count the trainable parameters in the LeRobot version you actually use and plug that into the formula.

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("lerobot/smolvla_base")
total = sum(p.numel() for p in policy.parameters())
train = sum(p.numel() for p in policy.parameters() if p.requires_grad)

print(f"total {total/1e6:.0f}M, trainable {train/1e6:.0f}M")
print(f"fixed term ≈ {((total-train)*4 + train*16)/1e9:.2f} GB (frozen FP32)")

If you plan to experiment with train_expert_only or freeze_vision_encoder, pass those settings when you build the policy and rerun the same code to see how the trainable count changes.

Set the batch size from three short measurements

Activation memory depends on the number of cameras, image resolution, language token length, chunk length and whether gradient checkpointing is on, so no single formula matches it precisely. Instead, use the fact that it scales almost linearly with batch size and measure it.

1. Fix the dataset and settings, then train for a few dozen steps each at --batch_size=1, 2 and 4.
2. Record the peak memory $M(B)$ of each run. Watching nvidia-smi --query-gpu=memory.used --format=csv -l 1 in a separate terminal is enough.
3. From two points, compute the slope $a = (M(4) - M(2)) / 2$ and the intercept $M_0 = M(2) - 2a$.
4. Assume you use at most 0.9 of the card’s capacity $V$, leaving the rest for fragmentation and peak swings, and compute the maximum batch.

lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/mydataset \
  --batch_size=2 \
  --steps=50 \
  --output_dir=outputs/mem_probe_b2 \
  --policy.device=cuda

For example, if you measured $M(2)$ = 8 GB and $M(4)$ = 12 GB, then $a$ = 2 GB and $M_0$ = 4 GB. The largest batch on a 24 GB card would be $\lfloor (21.6 - 4)/2 \rfloor = 8$, and on a 48 GB card $\lfloor (43.2 - 4)/2 \rfloor = 19$. These are made-up inputs to show the method; replace them with your own measurements.

The LeRobot SmolVLA docs use --batch_size=64 and --steps=20000 in their example and say this takes roughly 4 hours on a single A100. The docs recommend starting with a small batch and increasing it as far as the GPU allows. If you have to shrink the batch, adjust the learning rate and step count too, so the total amount of training stays comparable.

Three common misreadings

Inference memory is not training memory. Inference has no gradients or optimizer state. With FP16 weights it is about 0.5B × 2 bytes ≈ 1 GB, plus KV cache and system headroom. You can check an inference device directly in the VRAM calculator with the SmolVLA preset, a 30 Hz control rate and 50-step chunks.

In inference, the rate usually binds before memory does. Executing the default n_action_steps of 50 at 30 Hz gives $50/30 \approx 1.67$ s until the next chunk is due. Cut the executed actions to 10 and the budget shrinks to about 0.33 s. The relationship between bandwidth and inference rate is covered in sizing memory and bandwidth for on-device VLAs.

“6× less memory” is relative to π0. The SmolVLA paper reports training about 40% faster with 6× less memory than π0, but gives no absolute memory figures. Whether it fits your card is something the measurement procedure above has to answer.

Values to work out before training

Three values make the hardware decision quick. First, take the trainable parameter count from the code above and compute the fixed cost. Second, measure at batch 1, 2 and 4 to get the activation slope, then compute the maximum batch. Third, check the inference rate on your deployment device in the VRAM calculator. Sizing the training and deployment hardware separately avoids ending up with a model that trains fine but runs too slowly on the robot.

If your environment is not ready yet, start with the MuJoCo and LeRobot installation guide to get the lerobot[smolvla] extra and CUDA in place. Sizes and licenses across open models are in the open-source robot policy model comparison.