How to Train an ACT Policy on the SO-101 Arm with LeRobot: From Teleoperation Data to Rollout

The SO-101 is the low-cost six-axis arm that the Hugging Face LeRobot documentation uses as its flagship example. You move a leader arm by hand, the follower arm mirrors it, and every motion is recorded as a demonstration. This guide walks an engineer who is new to LeRobot through the whole loop, in order: motor setup, dataset recording, ACT training and running the policy on the real robot. Each step ends with what to check before moving on, so you can tell whether a problem comes from hardware, data or training.

If you want the bigger picture first, our overview of Physical AI and world models explains where imitation learning and action-generation models sit in the field.

Hardware and software you need before starting

According to the SO-101 assembly guide, all six follower joints use Feetech STS3215 motors with 1/345 gearing. The leader arm deliberately mixes gear ratios so that it can hold its own weight while still moving with little force:

The bill of materials and the 3D-printable parts are listed in TheRobotStudio’s SO-ARM100 repository README. On the software side, the installation guide sets these requirements:

Step 1: Install LeRobot with the Feetech SDK

Recording, replay and calibration need the core_scripts extra, training needs training, and the SO-101 servos need feetech. If one machine will both record and train, install all three:

conda create -y -n lerobot python=3.12
conda activate lerobot
conda install ffmpeg -c conda-forge

git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e ".[core_scripts,training,feetech]"

The installation guide adds two platform notes. Under WSL, also install evdev. If the build fails, install cmake, build-essential and the ffmpeg development libraries before retrying.

Step 2: Assign motor IDs and calibrate both arms

Brand-new servos usually ship with ID 1, so a controller cannot tell them apart on a shared bus. Each motor therefore gets a unique ID and a common baudrate written to its EEPROM. You only do this once per motor.

1. Find the ports. Connect a controller board over USB and power, run lerobot-find-port, and unplug the cable when asked. The script prints the port for that arm, for example /dev/ttyACM0. On Linux you may need sudo chmod 666 /dev/ttyACM0 to grant access.
2. Set up the follower motors. The command below prompts you to connect one motor at a time, starting with the gripper. Only that motor may be attached to the board when you press Enter.
3. Set up the leader motors. Repeat the process with --teleop.type=so101_leader.
4. Calibrate. Place every joint in the middle of its range, press Enter, then sweep each joint through its full range. Do this once for the follower and once for the leader.

lerobot-setup-motors \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0

lerobot-setup-motors \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1

lerobot-calibrate \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower_arm

lerobot-calibrate \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_leader_arm

The real-robot imitation learning guide explains that id is the key under which the calibration file is stored. Keep my_follower_arm and my_leader_arm identical in every later command. If you use a Waveshare controller board, also check that both jumpers sit on the B (USB) channel.

Step 3: Verify the setup with teleoperation

Before recording, confirm that the follower tracks the leader pose for pose. Adding a camera and --display_data=true opens a rerun view with the video feed and joint values side by side:

lerobot-teleoperate \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower_arm \
    --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_leader_arm \
    --display_data=true

If any device is missing a calibration, this command starts the calibration routine first. When a single joint moves in the wrong direction or jumps at the end of its travel, recalibrating that arm is the quickest thing to rule out.

Step 4: Record the dataset

Recordings are saved in the LeRobotDataset format under ~/.cache/huggingface/lerobot/{repo-id}, and by default they are pushed to the Hugging Face Hub when recording ends. Log in with a write-access token first:

hf auth login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
HF_USER=$(NO_COLOR=1 hf auth whoami | awk -F': *' 'NR==1 {print $2}')

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower_arm \
    --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_leader_arm \
    --display_data=true \
    --dataset.repo_id=${HF_USER}/so101_pick_place \
    --dataset.num_episodes=50 \
    --dataset.single_task="Pick up the cube and place it in the bin"

The recording flow is driven from the keyboard. The letter keys n, r and q also work on Wayland and headless SSH sessions, as long as the terminal has focus:

--dataset.episode_time_s sets the episode length and --dataset.reset_time_s the reset window; both default to 60 seconds. For a first task the docs suggest grasping an object at several locations and dropping it into a bin, with at least 50 episodes, 10 per location. Keep the cameras fixed, grasp the same way each time, and check that a person could complete the task from the camera images alone. If a session is interrupted, rerun the same command with --resume=true and --dataset.root, and set num_episodes to the number of additional episodes, not the target total.

Step 5: Train the ACT policy

ACT, short for Action Chunking with Transformers, was introduced by Zhao et al. in 2023. Instead of predicting one action per step, it predicts a whole sequence (a chunk) at once, which reduces the compounding of small errors over time. The paper’s abstract reports 80–90% success on six real-world tasks learned from only 10 minutes of demonstrations on low-cost hardware. LeRobot also trains Diffusion Policy, SmolVLA and Pi0-family policies through the same command by changing policy.type, so validating your data pipeline with ACT first makes later model comparisons straightforward.

These are the defaults in LeRobot’s ACT configuration that most directly shape behavior:

You only need to pass the dataset and the policy type. The model’s inputs and outputs adapt to the joints and cameras stored in the dataset:

lerobot-train \
  --dataset.repo_id=${HF_USER}/so101_pick_place \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_pick_place \
  --job_name=act_so101_pick_place \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=${HF_USER}/act_so101_pick_place

On Apple silicon, switch to --policy.device=mps. The docs say training takes several hours, and checkpoints accumulate in outputs/train/act_so101_pick_place/checkpoints. To continue an interrupted run, pass --config_path=.../checkpoints/last/pretrained_model/train_config.json --resume=true. Without a capable local GPU, add --job.target=a10g-small to run on a Hugging Face Jobs cloud GPU instead.

Step 6: Run the trained policy on the robot

Deployment uses lerobot-rollout. The base strategy runs the policy without recording, while sentry keeps recording rollouts into an evaluation dataset and uploads them. Camera names and settings must match the training data:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=${HF_USER}/act_so101_pick_place \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --task="Pick up the cube and place it in the bin" \
  --duration=60

If the policy will eventually run continuously on an edge board, memory requirements change sharply once you move from ACT to a larger VLA policy. Our edge GPU VRAM sizing guide for on-device VLA and the VLA VRAM calculator help estimate that gap before you buy hardware.

Four data checks when the success rate is low

When the policy hesitates near the object or heads to the wrong place, look at the dataset before touching hyperparameters:

1. Camera framing. Confirm the camera position and angle at rollout match the recording setup. The docs list fixed cameras as the first condition for good data.
2. Amount of variation. If you changed grasp locations, lighting and objects all at once, cut back to one varying factor and retrain. The docs advise adding variation gradually once grasping is reliable.
3. Episode quality. Replay a recorded episode with lerobot-replay --dataset.episode=0. This separates shaky demonstrations from poor robot repeatability.
4. Motion smoothness. If motion stutters at chunk boundaries, set n_action_steps below chunk_size, or try temporal ensembling (temporal_ensemble_coeff with n_action_steps=1). Ensembling runs inference every step, so check that your inference hardware has the compute headroom.