Is 50 Demos Enough to Fine-Tune a VLA? ACT vs Diffusion Policy vs SmolVLA

Once you have recorded about 50 demonstrations on a robot arm, the first question is usually whether that is enough to train a VLA, or whether you should start with a smaller policy such as ACT. This article puts the demo counts and the training and inference setups reported in the ACT, Diffusion Policy and SmolVLA papers into one table, then gives criteria for choosing a starting model based on data size and inference budget. Every number comes from the papers or official docs, and since each was measured on different tasks and hardware, those conditions are stated alongside.

If you want the architecture of each model first, see how Diffusion Policy works and the VLA pipeline comparison.

How many demos each paper used, and how it trained

The ACT figures come from the body of the ACT paper, the Diffusion Policy figures from Table 3 and the experiments section of the Diffusion Policy paper, and the SmolVLA figures from the model and training sections and Table 1 of the SmolVLA paper. The ACT abstract reports 80 to 90% success on six real tasks with this setup. The Diffusion Policy paper reports an average improvement of 46.9% over prior methods on its simulation benchmarks.

What stands out is that ACT and SmolVLA both used about 50 demonstrations per real task. The difference lies in model size and whether the model was pretrained.

On the same 50 demos, pretraining made the difference

The SmolVLA paper evaluates three tasks on an SO100 arm: pick-and-place, stacking and sorting. Each dataset has 10 demos from each of 5 start positions, 50 in total. Here are the paper’s real-world results under the same data conditions.

The same SmolVLA architecture, trained without pretraining on one task, averages 40%, below ACT. Table 5 of the paper credits pretraining on community datasets with lifting 51.7% to 78.3%. So “50 demos are not enough for a VLA” holds when you start without pretraining, and does not hold when you fine-tune a pretrained checkpoint.

The LeRobot SmolVLA docs point the same way. They recommend about 50 episodes as a starting point and report that a similar dataset cut to 25 episodes performed poorly. They also stress repeating demos several times for every condition you vary, such as cube position, to help generalization. How you structure the data matters as much as how much you collect.

Choosing by data size, task count and inference budget

The table below turns those numbers into starting points. It is not an answer key; it tells you what to try first.

In every case, training an ACT baseline first makes comparisons easier. In LeRobot, one dataset can be trained with --policy.type=act, --policy.type=diffusion or --policy.path=lerobot/smolvla_base with nothing else changed. The ACT procedure is covered in the LeRobot SO-101 ACT training guide.

Work out the inference budget from chunk length and control rate

When a policy emits several actions per inference, the time you have before the next chunk is due is:

Here $n_{action}$ is the number of actions actually executed from each inference result, and $f_{control}$ is the control rate in Hz.

• SmolVLA: configuration_smolvla.py defaults to chunk_size 50, n_action_steps 50 and 10 flow-matching num_steps. At 30 Hz the next chunk must be ready within $50 / 30 \approx 1.67$ s. The paper reports that asynchronous inference cut task completion time from 13.75 s to 9.7 s.
• ACT: the paper queries the policy at every timestep for temporal ensembling. At 50 Hz the budget is 0.02 s, and the reported inference time of about 0.01 s fits.
• Diffusion Policy: in the paper, real-world Push-T predicts commands at 10 Hz and interpolates them to 125 Hz for execution, and an action horizon of 8 steps was found best for most tasks. The 0.1 s latency fits within the prediction period.

For a given model, executing fewer actions per chunk improves reactivity but shortens the budget. The SmolVLA paper concludes that chunk sizes between 10 and 50 balance reactivity and efficiency well. Turning that budget into a hardware spec is covered in SmolVLA fine-tuning VRAM and sizing memory and bandwidth for on-device VLAs.

Moving from small policies to larger VLAs

Growing the model in step with your data and task count makes failures easier to diagnose at each stage.

1. ACT baseline: use one task and 50 demos to confirm the data pipeline (fixed cameras, joint order, fps). If success is low here, the problem is more likely the data than the model.
2. Diffusion Policy comparison: if the task has motions with several valid paths, compare against it. The number of denoising steps sets inference latency, so check that it fits the control period first.
3. SmolVLA fine-tuning: extend to two or more tasks and add language instructions. Always start from a pretrained checkpoint such as lerobot/smolvla_base.
4. Larger VLAs: as robot types and long-horizon tasks increase, choose using the licenses, sizes and GPU needs in the open-source robot policy model comparison. Fine-tuning GR00T N1.7 on LeRobot-format data is covered in the GR00T N1.7 fine-tuning guide.

Three decisions to make before you record

First, fix the number of demos per task together with the number of start-condition variations. Writing it down as “5 positions × 10 demos”, as the SmolVLA datasets do, lets you trace which condition is underrepresented when success is low later. Second, decide whether you need language instructions. For a single task ACT may be enough; if tasks must be switched by instruction, a pretrained VLA is the starting point. Third, compute the inference budget from control rate and actions executed. Once those three are fixed, the table above settles most of the model choice.

If your environment is not set up yet, start with the MuJoCo and LeRobot installation guide to match Python and CUDA.