Touch2Robot adds simulated robot contact to human demonstrations

ShanghaiTech University, Beijing Institute for General Artificial Intelligence, Shanghai Jiao Tong University and Beihang University researchers introduced Touch2Robot on September 21, 2026. The framework records human motion, glove touch and object motion, reconstructs how a target robot hand would contact the object, and displays that contact in virtual reality while the person demonstrates. The goal is not haptic feedback to the user; it is visual feedback about predicted robot touch. Paper and version record.

Key takeaways

• Across four real-world tasks, average replay completion rises from 37.9% with visual-only feedback to 72.1% with Touch2Robot under the paper's staged completion metric.
• Diffusion Policies trained on the resulting demonstrations average 62.0% completion, versus 32.9% for visual-feedback data with the same number of trajectories.
• Reconstructed contact reaches 44.19% F1 against physical robot tactile measurements. Better coverage comes with a higher 6.38% false-positive rate, so the system does not perfectly predict real contact. Tables I to IV.

What changed

Human demonstrations are scalable, but a human hand and a robot hand do not share the same kinematics or contact geometry. Touch2Robot trains object-specific reinforcement-learning teachers to reproduce object motion while favoring contact patterns aligned with the tactile glove. Those teachers are distilled into one retargeter conditioned on human observation and object geometry. During collection, the retargeted hand and tracked object are synchronized in simulation; predicted contacts are shown through a Meta Quest 3 at 30 Hz.

The physical evaluation uses an xArm6 with a LEAP Hand, TwinTac sensors and binary force-sensitive resistors. A MANUS glove records motion, a Juqiao tactile glove records human touch, and calibrated RGB cameras track the object with FoundationPose. The four tasks are pick-and-place, 180-degree object rotation, board wiping and drawer opening. System and tasks.

How to read the reported percentages

The evaluation metric is completion, not always binary success. Rotation, wiped area and drawer displacement are scaled toward a task goal. Pick-and-place awards 25% for each of reach, pick, move and place. Ten attempts per task are collected under each interface, then replayed on the physical robot without filtering demonstrations by replay outcome.

The time figure divides demonstration time by the number of trajectories that replay successfully. It is not end-to-end dataset production time and does not include teacher training, retargeter training or system setup.

For downstream imitation learning, Touch2Robot data improves the mean from 32.9% to 62.0% over visual feedback. Variation is wide on some tasks, including a 69.0 plus or minus 47.7% drawer score. The four-task average should not hide that uncertainty.

What this means for robotics data

RoboSkin analysis: the most useful idea is moving embodiment feedback into collection before a physical robot executes every demonstration. This could reduce wasted trajectories where human intent looks plausible but maps to unstable robot contact. It complements rather than replaces direct robot teleoperation because simulated contact is still an estimate.

The result also clarifies what a tactile demonstration contains. Touch2Robot keeps human tactile preference, predicted robot contact, object motion and eventual physical measurement as distinct signals. That separation is valuable when designing robotics datasets or comparing with touch-rich resources in the tactile dataset directory.

Limitations and availability

Contact fidelity remains incomplete: 44.19% F1 means reconstructed and measured contacts disagree often. The Touch2Robot variant also has a higher false-positive rate than the three reported baselines. The user study includes ten people on the rotation task, and cross-hand results for XHand and Shadow Hand are qualitative VR visualizations rather than physical cross-embodiment trials. Quantitative real-world tests use one LEAP Hand platform.

This is a preprint and RoboSkin.ai has not reproduced it. As checked September 22, the official project page provides videos, images and an appendix PDF, but no public code repository, dataset download or software license was identified. The paper does not establish that the full collection system can be reconstructed from public assets alone.

Sources

• Touch2Robot arXiv record, submitted September 21, 2026
• Touch2Robot v1 methods, tables and appendix
• Official Touch2Robot project page