Robot teleoperation: from human demonstration to robot data

Teleoperation can command a mobile base, arm pose, joint motion, gripper, dexterous hand, whole humanoid body, or a high-level subtask. The operator may watch direct video, stereo or depth views, a digital twin, or the physical robot. Some systems provide haptic or force feedback; others are one-way.

The interface changes the data. A low-dimensional joystick, handheld gripper, motion-capture rig, exoskeleton, and bilateral master device encode different human intent and produce different action distributions. Dataset records should preserve that acquisition contract.

A teleoperation recording is a source trajectory, not a finished learning asset. The complete pipeline aligns sensors and commands, segments tasks, records outcomes and failure context, validates the episode, converts it to a stable schema, splits it without leakage, trains a model, and tests the model in closed loop.

A teleoperated demonstration proves that a person and interface could make the robot perform the task under those conditions. It does not prove that a learned policy can perceive the same state, recover from error, or act safely without the operator.

A policy trained by imitation can encounter states absent from expert demonstrations because small prediction errors compound during execution. Corrective demonstrations, interactive data collection, recovery examples, and policy rollouts can target this gap, but they still require an explicit intervention policy and fair evaluation.

The official Hugging Face Grabette article describes a handheld gripper-and-camera device that records data in the standard LeRobot dataset format. It is an example of collecting manipulation demonstrations without keeping a full robot in the loop during every recording. The transfer value still depends on calibration, task coverage, observation and action mapping, and later robot evaluation.

The LeRobot v0.6 release describes rollout tools and DAgger-style human corrections. FreeTacMan investigates wearable, robot-free visuo-tactile data collection. DROID and Open X-Embodiment represent broader real-robot and multi-embodiment data strategies. These resources solve different parts of the pipeline and should not be treated as identical teleoperation systems.

Touch can be recorded as robot-side training data, returned to the operator as haptic feedback, or both. Robot-side tactile data can label contact onset, slip, pressure distribution, or grasp state. Haptic feedback can help a person adjust motion, but the master device, scaling, delay, and feedback modality shape what the operator feels.

For imitation learning, synchronize tactile observations with the action that preceded the contact and the corrective action that followed it. A contact stream without robot state, timestamps, or action context is much harder to use for closed-loop policy learning.

The Missing Touch study maps GelSight Mini contact images to a 32-DoF cutaneous display and reports more natural and consistent trajectories in two 2-DoF tasks. It did not train an autonomous policy, so its learning relevance is a demonstration-quality implication rather than a measured policy-success gain. PRISM provides a separate collection perspective by combining exoskeleton, tracker, and VR interfaces across industrial tasks, while limiting tactile sensing to an unspecified subset of episodes.

Data quality is a measured property, not a consequence of collection volume. Before training, inspect timing, calibration, missing streams, task labels, duplicated episodes, operator shortcuts, failures, and inconsistent resets. Before a capability claim, test the learned policy on the physical robot under a protocol that distinguishes autonomous success from human correction.