Robot teleoperation: from human demonstration to robot data

Passive observation data can show what a human did and how a scene changed. Teleoperation can additionally record the commands sent to a particular robot alongside its measured state. That observation-action pairing is the key distinction for policy training: ordinary video does not directly tell you which robot command produced each transition.

These are complementary data sources. R3M pretrains visual representations on Ego4D human video and uses the representation for downstream robot policy learning. It does not make the original videos a dataset of measured robot actions. The R3M paper is an example of representation pretraining followed by task learning, not proof that every imitation-learning recipe can train from unlabelled video.

UMI is a third case: an instrumented handheld demonstration interface. Its gripper-mounted camera and trajectory-processing pipeline provide more structure than passive video, while collection can happen away from a robot. GELLO instead controls a robot through a joint-based interface. The table summarizes the original projects and R3M sources checked on September 19, 2026; RoboSkin has not run these hardware systems.

For visual representation learning, video may be useful before robot action labels are available. For supervised action imitation, first establish the action representation and its alignment with observations. For contact-sensitive insertion or grasp adjustment, decide whether touch supplies an observation, a training label, operator feedback or an evaluation signal; each role requires different records.

Consider inserting a plug. A passive clip shows motion and visible outcome. A teleoperation log can also preserve the commanded motion and measured robot state. Tactile sensing can add a local contact observation when the camera cannot see the mating surfaces. To compare these sources, hold out objects or scenes, report autonomous success separately from intervention, and state which streams the deployed policy actually receives. This is an experimental design example, not a RoboSkin hardware result.

Teleoperation can command a mobile base, arm pose, joint motion, gripper, dexterous hand, whole humanoid body, or a high-level subtask. The operator may watch direct video, stereo or depth views, a digital twin, or the physical robot. Some systems provide haptic or force feedback; others are one-way.

The interface changes the data. A low-dimensional joystick, handheld gripper, motion-capture rig, exoskeleton, and bilateral master device encode different human intent and produce different action distributions. Dataset records should preserve that acquisition contract.

A teleoperation recording is a source trajectory, not a finished learning asset. The complete pipeline aligns sensors and commands, segments tasks, records outcomes and failure context, validates the episode, converts it to a stable schema, splits it without leakage, trains a model, and tests the model in closed loop.

A teleoperated demonstration proves that a person and interface could make the robot perform the task under those conditions. It does not prove that a learned policy can perceive the same state, recover from error, or act safely without the operator.

A policy trained by imitation can encounter states absent from expert demonstrations because small prediction errors compound during execution. Corrective demonstrations, interactive data collection, recovery examples, and policy rollouts can target this gap, but they still require an explicit intervention policy and fair evaluation.

The official Hugging Face Grabette article describes a handheld gripper-and-camera device that records data in the standard LeRobot dataset format. It is an example of collecting manipulation demonstrations without keeping a full robot in the loop during every recording. The transfer value still depends on calibration, task coverage, observation and action mapping, and later robot evaluation.

The LeRobot v0.6 release describes rollout tools and DAgger-style human corrections. FreeTacMan investigates wearable, robot-free visuo-tactile data collection. DROID and Open X-Embodiment represent broader real-robot and multi-embodiment data strategies. These resources solve different parts of the pipeline and should not be treated as identical teleoperation systems.

Touch can be recorded as robot-side training data, returned to the operator as haptic feedback, or both. Robot-side tactile data can label contact onset, slip, pressure distribution, or grasp state. Haptic feedback can help a person adjust motion, but the master device, scaling, delay, and feedback modality shape what the operator feels.

For imitation learning, synchronize tactile observations with the action that preceded the contact and the corrective action that followed it. A contact stream without robot state, timestamps, or action context is much harder to use for closed-loop policy learning.

The Missing Touch study maps GelSight Mini contact images to a 32-DoF cutaneous display and reports more natural and consistent trajectories in two 2-DoF tasks. It did not train an autonomous policy, so its learning relevance is a demonstration-quality implication rather than a measured policy-success gain. PRISM provides a separate collection perspective by combining exoskeleton, tracker, and VR interfaces across industrial tasks, while limiting tactile sensing to an unspecified subset of episodes.

Data quality is a measured property, not a consequence of collection volume. Before training, inspect timing, calibration, missing streams, task labels, duplicated episodes, operator shortcuts, failures, and inconsistent resets. Before a capability claim, test the learned policy on the physical robot under a protocol that distinguishes autonomous success from human correction.

Define an episode before recording: task instruction, start/reset condition, expected outcome and failure categories. Record observations and operator commands separately so an action target cannot be mistaken for a measured robot state. Preserve raw streams and a capture manifest alongside any training export.

Time synchronization belongs in acquisition. Save sensor capture time, host receive time and clock domains; estimate offsets using a documented synchronization method and inspect residual alignment. A resampled episode timestamp is a training index, not a measurement of end-to-end latency.

Before exporting, inspect dropped frames, duplicate or reversed timestamps, changed array dimensions, stale observations, saturation and invalid tactile samples. Label failed trajectories, interventions and recovery segments with reasons. Retain their provenance whether the training recipe includes or excludes them; do not silently call every recorded episode a success.

For training export, record state/action names, units, coordinate frames, episode boundaries, task text and outcome labels. Split by the intended generalization claim before computing training statistics. The linked LeRobot exercise checks a numeric subset with synthetic data; it does not validate videos, policy compatibility or the quality of a physical demonstration.

Original UMI and GELLO are complementary collection approaches. UMI moves demonstration capture into a handheld gripper; GELLO maps a human-operated joint controller to a robot arm. The comparison below is scoped to the original project and paper, checked on September 16, 2026. It does not generalize later tactile variants to the originals.