Tutorial · commands verbatim

SONIC data collection: LeRobot datasets for VLA

Record teleop demonstrations as LeRobot datasets for post-training with Isaac-GR00T. The data exporter runs alongside the SONIC deployment and VR teleop stack, capturing robot state, SMPL teleop poses, and camera images at a configurable frequency. All commands are transcribed from the official Data Collection tutorial.

Deployment model

Everything runs offboard on your workstation except the camera server, which runs onboard the robot computer (e.g. Jetson Orin) where the physical cameras are connected. The camera server publishes JPEG frames over ZMQ to the workstation.

Prerequisites: complete the Quick Start (sim2sim loop, including the deployment install and model checkpoint download), complete the VR Teleop Setup (PICO hardware calibrated, .venv_teleop ready), and have the camera server running on the robot. For simulation, the MuJoCo sim loop publishes camera images automatically — no camera server needed.

Supported cameras

The tested and supported camera setup uses Luxonis OAK cameras (OAK-D, OAK-1, etc.): a head/ego-view OAK camera and optional OAK wrist cameras. Other camera drivers (RealSense, USB webcam) are included in the codebase but have not been tested recently. A 3D-printable mount for the head/ego-view OAK-D W camera is available under hardware/camera_mount/ — its README covers print settings, the bill of materials, and how it mounts on the G1.

One-time setup (workstation)

Run the install script from the repo root to create a dedicated virtual environment with all data collection dependencies (LeRobot, PyAV, OpenCV, etc.):

bash install_scripts/install_data_collection.sh

This creates .venv_data_collection using Python 3.10 via uv, installing gear_sonic[data_collection] (lerobot, av, opencv-python, and other packages) plus espeak (system package) for voice feedback during recording. This environment is separate from .venv_teleop and .venv_sim — the data exporter has heavier ML dependencies not needed for teleop or simulation.

Camera server setup (on-robot)

The camera server is the only component that runs on the robot computer (e.g. Jetson Orin). Everything else — the C++ deployment, PICO teleop streamer, data exporter, and camera viewer — runs on your workstation. It captures frames from the OAK cameras physically connected to the robot and publishes them over ZMQ.

git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git
cd GR00T-WholeBodyControl

bash install_scripts/install_camera_server.sh

The script creates .venv_camera with gear_sonic[camera] (DepthAI, ZMQ, msgpack, OpenCV, tyro), detects connected OAK cameras and lists their MxIDs, prompts for each camera position (ego view, optionally left/right wrist) and device ID, then asks whether to install the camera server as a systemd service (recommended) — answering y generates the unit file, installs, enables, and starts it automatically. Verify:

sudo systemctl status composed_camera_server.service
journalctl -u composed_camera_server.service -f

Manual alternative: find device IDs with python -c "import depthai as dai; print(dai.Device.getAllAvailableDevices())" in .venv_camera, then start the server with python -m gear_sonic.camera.composed_camera --ego-view-camera oak --ego-view-device-id <YOUR_MXID> --port 5555 (add --left-wrist-camera / --right-wrist-camera flags for wrist cams; see --help for --fps, --use-mjpeg, --mjpeg-quality). For manual systemd: edit systemd/composed_camera_server.service, copy to /etc/systemd/system/, then daemon-reload + enable + start. The G1 robot's default IP is 192.168.123.164.

ZMQ message format

The camera server publishes a single msgpack-encoded payload per frame cycle containing all camera images:

{
    "timestamps": { "ego_view": 1712345678.123, "left_wrist": 1712345678.125 },
    "images": { "ego_view": "<base64-jpeg>", "left_wrist": "<base64-jpeg>" }
}

Images are JPEG-compressed (quality 80) and either base64-encoded strings or raw JPEG bytes (when MJPEG on-device encoding is enabled). The data exporter's ComposedCameraClientSensor handles both formats automatically.

Architecture and ZMQ sources

Source	Runs on	ZMQ Topic	Default Port	Provides
C++ deployment	Workstation	`g1_debug`	5557	Joint positions, velocities, IMU quaternion
C++ deployment	Workstation	`robot_config`	5557	One-shot robot configuration at startup
PICO teleop streamer	Workstation	`pose`	5556	SMPL body parameters (teleop target poses)
Camera server	Robot	`(raw TCP)`	5555	JPEG-compressed camera images (ego view + optional wrist views)

The data exporter subscribes to all three ZMQ sources — deployment state on port 5557, SMPL teleop poses on 5556, camera images on 5555 — and writes the LeRobot dataset (parquet + mp4).

Option A · All-in-one tmux launcher (recommended)

The launcher starts all components in a single tmux session with four panes: C++ Deploy, PICO Teleop, Data Exporter, Camera Viewer (requires tmux: sudo apt install tmux). For simulation (the launcher starts run_sim_loop.py in a separate tmux window automatically):

python gear_sonic/scripts/launch_data_collection.py --sim

For real robot (camera server running on robot at 192.168.123.164):

python gear_sonic/scripts/launch_data_collection.py \
    --camera-host 192.168.123.164 \
    --task-prompt "pick up the cup"

With wrist cameras (records ego view + left/right wrist streams):

python gear_sonic/scripts/launch_data_collection.py \
    --camera-host 192.168.123.164 \
    --task-prompt "pick up the cup" \
    --record-wrist-cameras

No need to activate a virtual environment first — the launcher auto-detects and uses .venv_data_collection if required dependencies are missing from the current Python. Common flags: --task-prompt (default "demo"), --dataset-name (auto: timestamp), --sim / --no-sim, --camera-host (default localhost), --camera-port (5555), --data-exporter-frequency (50), --deploy-checkpoint, --deploy-obs-config, --deploy-planner, --deploy-motion-data, --record-wrist-cameras, --no-text-to-speech. Session management: Ctrl+b then arrow keys to switch panes, Ctrl+b then d to detach, tmux attach -t sonic_data_collection to reattach, tmux kill-session -t sonic_data_collection to kill.

Recording controls: PICO VR controllers — Left Grip + A toggles recording (starts a new episode or stops and saves), Left Grip + B discards the current episode (saved to disk but flagged for removal during post-processing). Keyboard over ZMQ (port 5580): c toggles recording, x discards the episode. These work in any manager mode (POSE, PLANNER, etc.).

Option B · Manual multi-terminal setup

Terminal 1 — MuJoCo Simulator (skip for real robot):

source .venv_sim/bin/activate
python gear_sonic/scripts/run_sim_loop.py \
    --enable-image-publish --enable-offscreen --camera-port 5555

Terminal 2 — C++ Deployment (from gear_sonic_deploy/):

cd gear_sonic_deploy
source scripts/setup_env.sh
./deploy.sh --input-type zmq_manager sim
# Wait until you see "Init done"

Terminal 3 — PICO Teleop Streamer:

source .venv_teleop/bin/activate
python gear_sonic/scripts/pico_manager_thread_server.py --manager

Terminal 4 — Data Exporter:

source .venv_data_collection/bin/activate
python gear_sonic/scripts/run_data_exporter.py --task-prompt "pick up the cup"

Terminal 5 (optional) — Camera Viewer: python gear_sonic/scripts/run_camera_viewer.py in .venv_data_collection. Datasets are saved under <root-output-dir>/<dataset-name>/ (default root outputs); omit --dataset-name for an auto timestamp, or reuse a name to append episodes. The standalone camera viewer also records raw MP4 reference videos (R start/stop, Q quit) to camera_recordings/rec_<timestamp>/.

Recorded data channels

Feature	Shape	Description
`observation.state.joint_position`	`(N,)`	Actuated joint positions (rad)
`observation.state.joint_velocity`	`(N,)`	Actuated joint velocities (rad/s)
`observation.state.body_rotation_6d`	`(6,)`	Base orientation (6D rotation)
`observation.state.projected_gravity`	`(3,)`	Gravity vector in body frame
`observation.images.ego_view`	`(480, 640, 3)`	Ego camera image (saved as MP4 video)
`observation.images.left_wrist`	`(480, 640, 3)`	Left wrist camera (only with --record-wrist-cameras)
`observation.images.right_wrist`	`(480, 640, 3)`	Right wrist camera (only with --record-wrist-cameras)
`action.joint_position`	`(N,)`	Teleop target joint positions
`action.body_rotation_6d`	`(6,)`	Teleop target body rotation
`annotation.human.action.task_description`	`string`	Task prompt for this frame

Datasets are saved in the LeRobot v2.1 format under <root-output-dir>/<dataset-name>/: data/ (parquet: joint states, actions, annotations), videos/ (H264-encoded MP4 per camera stream), meta/ (info.json, modality.json, episodes.jsonl, tasks.jsonl).

Post-processing datasets

All commands run in .venv_data_collection. Remove discarded episodes (flagged with x or Left Grip + B) — default behavior removes them so they are excluded from fine-tuning; pass --no-remove-discarded to keep them for inspection:

# Clean a single dataset (removes discarded episodes + stale SMPL frames)
python gear_sonic/scripts/process_dataset.py \
    --dataset-path outputs/my_dataset \
    --output-path outputs/my_dataset_cleaned

Remove stale SMPL frames — teleop pauses or ZMQ frame drops create frames where teleop.smpl_pose is all zeros; the script also removes consecutive frozen (identical) lead-in frames that precede them. In-place or non-destructive:

# Clean a single dataset in-place
python gear_sonic/scripts/process_dataset.py \
    --dataset-path outputs/my_dataset

# Clean and write to a new directory (non-destructive)
python gear_sonic/scripts/process_dataset.py \
    --dataset-path outputs/my_dataset \
    --output-path outputs/my_dataset_cleaned

VR_3PT trap: if you collected data using VR 3-point tracking mode (VR_3PT), teleop.smpl_pose is all zeros because VR_3PT uses raw VR positions/orientations instead of SMPL body parameters. You must disable SMPL cleaning to avoid dropping all frames:

python gear_sonic/scripts/process_dataset.py \
    --dataset-path outputs/my_dataset \
    --output-path outputs/my_dataset_cleaned \
    --no-remove-stale-smpl

Merge multiple datasets (validates all sessions share the same script_config robot configuration):

# Merge by listing datasets on the command line
python gear_sonic/scripts/process_dataset.py \
    --dataset-path outputs/session1 outputs/session2 outputs/session3 \
    --output-path outputs/merged_dataset

# Or use a text file (one dataset path per line, # for comments)
python gear_sonic/scripts/process_dataset.py \
    --dataset-list datasets.txt \
    --output-path outputs/merged_dataset

SMPL cleaning applies by default during merging — it removes entire frames where the SMPL teleop pose is stuck at zeros (operator pauses, ZMQ packet-drop periods), plus consecutive frozen frames leading into a zero block. Add --no-remove-stale-smpl to skip cleaning and merge only. The output dataset is directly compatible with the Isaac-GR00T post-training pipeline — next step is the VLA workflow.

BONES-SEED: the offline motion corpus

Beyond your own teleop demonstrations, the training-data stack builds on BONES-SEED — an open dataset of 142,220 annotated human motion animations (71,132 original + 71,088 mirrored, ~288 hours at 120 fps) from 522 actors, in SOMA and Unitree G1 formats with natural language descriptions and temporal segmentation. Each motion comes in three formats: SOMA Proportional (BVH), SOMA Uniform (BVH), and Unitree G1 (CSV) retargeted to the G1 humanoid. Download:

# Using the Hugging Face CLI
pip install huggingface_hub
huggingface-cli download bones-studio/seed --repo-type dataset --local-dir ./bones-seed

For SONIC training, the G1 CSVs are converted with convert_soma_csv_to_motion_lib.py (--fps 30 --fps_source 120) and filtered with filter_and_copy_bones_data.py, then consumed via the --training Hugging Face download for SMPL data — full commands on the training page and in the Training Data doc.

Data collection FAQ

Do I need a real robot to collect data for VLA fine-tuning?

No — the all-in-one launcher supports simulation: python gear_sonic/scripts/launch_data_collection.py --sim starts run_sim_loop.py in a separate tmux window automatically, and the MuJoCo sim loop publishes camera images on the camera port (no camera server needed). The real-robot path additionally requires a camera server running onboard the robot.

Where does the camera server run, and where does everything else run?

The camera server is the only component that runs onboard the robot computer (e.g. Jetson Orin) where the physical OAK cameras are connected, publishing JPEG frames over ZMQ. Everything else — the C++ deployment, PICO teleop streamer, data exporter, and camera viewer — runs offboard on your workstation.

Why did my whole dataset get dropped during processing?

If you collected with VR 3-point tracking mode (VR_3PT), the teleop.smpl_pose column is all zeros because VR_3PT uses raw VR positions/orientations instead of SMPL body parameters. The default SMPL cleaning would then remove all frames — you must pass --no-remove-stale-smpl to process_dataset.py in that case.

Which cameras are supported for data collection?

The tested and supported setup uses Luxonis OAK cameras (OAK-D, OAK-1, etc.): a head/ego-view OAK camera and optional OAK wrist cameras. A 3D-printable mount for the head/ego-view OAK-D W camera is in hardware/camera_mount/ with print settings and a bill of materials. RealSense and USB webcam drivers exist in the codebase but have not been tested recently.