How to Fine-Tune Isaac GR00T N1.7: From LeRobot-Format Data to Open-Loop Evaluation

Once you have demonstrations recorded with LeRobot, one of the next options is to fine-tune NVIDIA’s open robot foundation model, GR00T N1.7, on that data. This guide follows the official repository’s documentation and example files in order: installation, data format, training, open-loop evaluation and running an inference server. To see where GR00T sits relative to other open policy models first, read the open robot policy model comparison.

What you need: GPU, accounts and software

These requirements come from the hardware recommendation and the README:

GR00T N1.7 is a 3B-parameter model, and the base checkpoint (about 6 GB) downloads on first run. If you do not have robot data yet, start with recording SO-101 data with LeRobot. The examples here assume a robot like the SO-100 or SO-101: five arm joints, one gripper and two cameras (front and wrist).

Step 1: Install the repository and confirm it works

Follow the README’s order. Without Git LFS installed first, the parquet files in demo_data/ arrive as pointers instead of data.

1. Install Git LFS and clone the repository with its submodules.
2. Install uv and FFmpeg.
3. Create a Python 3.12 environment and verify the install.
4. Log in to Hugging Face and confirm access to the backbone model.

sudo apt install git-lfs && git lfs install
git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T

curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt-get update && sudo apt-get install -y ffmpeg

uv sync --python 3.12
uv run python -c "import gr00t; print('GR00T installed successfully')"
uv run huggingface-cli login   # or: export HF_TOKEN=<your_token>

If training later fails with CUDA_HOME is unset, run bash scripts/deployment/dgpu/install_deps.sh as the README suggests, or set export CUDA_HOME=/usr/local/cuda.

Step 2: Get your data into LeRobot v2 format

According to the data preparation guide, GR00T uses LeRobot v2, with native v3 support planned for a future release. If a recent LeRobot recording produced a v3 dataset, use the repository’s converter. It has its own pyproject.toml, so create a separate environment inside that folder; installing from the repo root would install the gr00t package instead.

cd scripts/lerobot_conversion
uv venv && source .venv/bin/activate
uv pip install -e . --verbose
python convert_v3_to_v2.py --repo-id <your_account>/<dataset_name>

After conversion the layout should look like this, with one parquet file per episode, one mp4 per camera per episode, and the GR00T-specific modality.json added under meta/:

meta/
  info.json
  episodes.jsonl
  tasks.jsonl
  modality.json        # GR00T-specific
data/chunk-000/episode_000000.parquet
videos/chunk-000/observation.images.front/episode_000000.mp4
videos/chunk-000/observation.images.wrist/episode_000000.mp4

The observation.state and action columns in each parquet file are one-dimensional concatenations of every joint value, and modality.json slices those arrays into named ranges. This is the actual file shipped with the repository’s cube_to_bowl_5 demo:

{
  "state":  { "single_arm": {"start": 0, "end": 5}, "gripper": {"start": 5, "end": 6} },
  "action": { "single_arm": {"start": 0, "end": 5}, "gripper": {"start": 5, "end": 6} },
  "video": {
    "front": {"original_key": "observation.images.front"},
    "wrist": {"original_key": "observation.images.wrist"}
  },
  "annotation": { "human.task_description": {"original_key": "task_index"} }
}

Indices are zero-based and behave like Python slices: start is included, end is excluded. On the SO-101 the joint order is shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll and gripper, so the first five values form single_arm and the last one is gripper. For a robot with a different order, read the feature names in info.json and redraw the ranges. Language keys come in two styles depending on the dataset family — SO-100 style (annotation.human.task_description) and LIBERO style (annotation.human.action.task_description) — so keep the parquet column, modality.json and the config file in the same style.

Step 3: Write the modality config

A Python config file defines which observations the model sees, over how many steps, and how actions are represented. The repository’s so100_config.py is the template (license header omitted):

from gr00t.configs.data.embodiment_configs import register_modality_config
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.data.types import (
    ActionConfig, ActionFormat, ActionRepresentation, ActionType, ModalityConfig,
)

so100_config = {
    "video": ModalityConfig(delta_indices=[0], modality_keys=["front", "wrist"]),
    "state": ModalityConfig(delta_indices=[0], modality_keys=["single_arm", "gripper"]),
    "action": ModalityConfig(
        delta_indices=list(range(0, 16)),  # predict 16 future steps
        modality_keys=["single_arm", "gripper"],
        action_configs=[
            ActionConfig(rep=ActionRepresentation.RELATIVE, type=ActionType.NON_EEF,
                         format=ActionFormat.DEFAULT),   # arm: delta from current state
            ActionConfig(rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF,
                         format=ActionFormat.DEFAULT),   # gripper: absolute target
        ],
    ),
    "language": ModalityConfig(delta_indices=[0],
                               modality_keys=["annotation.human.task_description"]),
}

register_modality_config(so100_config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

Three things change per robot. modality_keys must match the key names in modality.json exactly, and action_configs needs one entry per action key, in the same order. The fine-tuning guide explains that relative arm actions generalize better, while an absolute gripper target suits binary open-and-close motion. Custom robots are always registered under EmbodimentTag.NEW_EMBODIMENT.

Step 4: Run fine-tuning

Here is the single-GPU example. Run it once on the bundled demo data to validate the environment, then point --dataset-path at your own dataset.

CUDA_VISIBLE_DEVICES=0 uv run python \
    gr00t/experiment/launch_finetune.py \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path examples/SO100/so100_config.py \
    --num-gpus 1 \
    --output-dir /tmp/so100 \
    --save-steps 500 \
    --max-steps 2000 \
    --global-batch-size 32 \
    --dataloader-num-workers 4

Run the script with --help for the full option list. Validation is off by default; add --eval-strategy steps --eval-steps 500 to track it. With fewer GPUs, use gradient accumulation to reach the target global batch; the hardware guide’s example uses 4 GPUs with a per-GPU batch of 8 and 8 accumulation steps for a global batch of 256. If host RAM runs out, lower --num-shards-per-epoch, which controls how much data is preloaded into RAM rather than VRAM. The default mode trains only the projector and DiT action head; enabling --tune-llm or --tune-visual raises the recommendation to 80 GB or more per GPU.

Step 5: Check training with open-loop evaluation

Before touching the robot, replay trajectories from the training data and check that predicted actions follow the recorded ones.

uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path ./demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path /tmp/so100/checkpoint-2000 \
    --traj-ids 0 \
    --execution-horizon 16 \
    --steps 400 \
    --modality-keys single_arm gripper \
    --save-plot-path ./eval_plots

The guide gives three criteria. First, on a trajectory used in training, the predicted curves should track the ground truth; flat predictions indicate a setup problem. Second, evaluate the checkpoints saved every 500 steps and confirm that Average MSE falls. The guide’s reference run showed 87.5 at step 500, 25.4 at 1000, 13.2 at 1500 and 10.0 at 2000, and it states explicitly that these are not targets. Third, record the result of the first clean run with the unmodified command as a baseline, and compare every later change against it.

Where runs usually get stuck

Step 6: Connect the robot through the inference server

The policy API guide describes serving the policy on a GPU machine and fetching actions from the robot side through a client.

uv run python gr00t/eval/run_gr00t_server.py \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path /tmp/so100/checkpoint-2000 \
    --device cuda:0 --host 0.0.0.0 --port 5555

from gr00t.policy.server_client import PolicyClient

policy = PolicyClient(host="localhost", port=5555, timeout_ms=15000, strict=False)
if not policy.ping():
    raise RuntimeError("Cannot connect to policy server!")

observation = {
    "video": {"front": front_frames, "wrist": wrist_frames},  # (1, T, H, W, 3) uint8
    "state": {"single_arm": arm_state, "gripper": grip_state},  # (1, T, D) float32
    "language": {"task": [["put the cube into the yellow bowl"]]},
}
action, info = policy.get_action(observation)
next_arm = action["single_arm"][:, 0, :]  # first step of a (B, T, D) chunk

Returned actions are in physical units, not normalized, and come as a (batch, time, dimension) chunk. According to the model card, the full TensorRT pipeline replans at 26.0 Hz on an L40 and 10.7 Hz on a Jetson AGX Thor, and the hardware guide explains that action chunking lets roughly 10 Hz replanning drive about 30 FPS of robot execution. For choosing the robot-mounted device, continue with the VLA edge hardware sizing guide; for where GR00T sits in NVIDIA’s training, simulation and deployment flow, see the Cosmos, Isaac and GR00T three-computer architecture.