UVTA scales dexterous policies with human tactile demonstrations
UVTA co-trains human and robot vision-touch-action data with a separate policy for each task. The public release now has code and a dataset manifest, with limits distinct from the paper corpus.

Researchers from Shanghai Jiao Tong University, the Beijing Institute for General Artificial Intelligence, Sharpa Robotics and Beijing Institute of Technology released Unified Visual-Tactile-Action Modeling, or UVTA, on September 28, 2026. The system co-trains on human and robot interaction data so that large volumes of human touch can improve a robot policy without treating human joint motion as directly executable robot commands. Paper and version record.
Key takeaways
- The paper reports 1,000 human and 150 robot demonstrations per task, or 5,750 total. The currently published manifest instead lists 5,749 unique episodes; the release distinction is explained below.
- The full model averages 70% stage-wise success over 10 physical trials per task, compared with 29% for the strongest reported baseline and 42% without future-tactile prediction.
- The November 4 check found a public code repository and Hugging Face dataset metadata. We inspected their documentation and manifest, not archive contents, model weights or robot execution.
What changed
UVTA starts with a wearable capture system: a passive 22-degree-of-freedom hand exoskeleton records finger motion, two VIVE trackers recover wrist pose, a wrist camera captures RGB and four force-sensing regions on each fingertip produce a 20-dimensional tactile vector. The authors map these human observations into the same numerical representation used by a 22-degree-of-freedom Sharpa Wave robot hand.
The model turns the current wrist image into a 384-dimensional visual token and the tactile input into a 20-dimensional token. Their concatenated 404-dimensional representation feeds a diffusion action head and a tactile regression head. Both human and robot sequences supervise the shared representation, but deployment decodes only robot actions. The future-tactile head is an auxiliary learning objective; its prediction is not a force target passed to the controller. Architecture and dataset.
A separate policy is trained for each task. This is co-training rather than direct teleoperation transfer. Human hand actions still provide a learning signal, while embodiment-specific normalization and robot demonstrations teach the executable action space. Human and robot batches are sampled equally with replacement, not in proportion to their raw corpus sizes. The action head predicts 16 steps and executes eight before replanning from new visual and tactile observations.
What the real-robot evaluation found
The five tasks are page flipping, bayonet light-bulb insertion, toggle-switch operation, tactile ball classification and liquid transfer. Every method receives the same wrist RGB, fingertip touch and right-arm/right-hand action space, and each result averages ten real-robot trials with randomized object positions or heights.
UVTA reports an unweighted average of 70%: 83% on page flipping, 80% on the light bulb, 88% on the toggle switch, 56% on ball classification and 44% on liquid transfer. The strongest baseline, T-Rex, averages 29%; RDP averages 14%, and ViTacFormer averages 1%. Removing the future-tactile head but retaining touch reduces the average to 42%, while the vision-only ablation reaches 26%. Full comparison.
These are stage-wise scores, not purely binary task completion. For example, page turning assigns partial credit for separating pages or turning multiple pages before awarding full credit for turning exactly one. That scoring makes intermediate progress visible, but it also means the headline percentages should not be interpreted as 35 complete successes out of 50 trials.
The scaling experiment holds 150 robot demonstrations per task fixed and evaluates three tasks as human data grows. The average moves from 0% with no human trajectories to 8%, 29%, 51% and 84% with 250, 500, 750 and 1,000 human trajectories. The largest reported step occurs between 750 and 1,000, but only three of the five tasks are included in this scaling curve.
What this means for robotics
RoboSkin analysis: UVTA treats human touch as representation supervision rather than pretending away the embodiment gap. That is valuable for robot datasets, because people can generate diverse contact sequences much faster than they can teleoperate a high-dimensional robot hand.
The study also shows why dataset totals need context. The 5,000 human demonstrations provide variety, while 750 robot demonstrations anchor executable behavior. Human data did not replace robot data in this experiment; it complemented it. The method therefore supports a hybrid scaling strategy, not a claim that tactile robot demonstrations are unnecessary.
Limitations and availability
UVTA is an arXiv v1 preprint, and RoboSkin.ai has not reproduced it. Evaluation uses one fixed-base North robot, one Sharpa Wave embodiment and five tasks. Touch is measured only at four regions per fingertip, so palm contact, shear and distributed whole-hand pressure are not represented. Ten trials per task also make small score differences uncertain.
The September 29 review found the linked repository unavailable. On November 4, the project page links a public UVTA repository and dataset page. At code commit 74f4270142cf3778a7103f04bbb922daa2597fd4, the root license is MIT; the dataset card separately states Apache-2.0. These declarations do not replace checks of bundled assets and third-party dependencies. No archives or checkpoints were downloaded, and no installation was performed.
What the public release actually describes
Our read-only inspection of the dataset manifest found 28 tar shards totaling 482,351,124,480 listed bytes. Its unique episode IDs comprise 5,000 human and 749 robot episodes: page flipping has 149 robot episodes, while the other four tasks have 150 each. This verifies metadata counts, not successful extraction, payload integrity or policy compatibility.
The release README describes Zarr v2 episodes inside those archives. The 20-dimensional tactile field is already pooled fingertip data, not raw tactile images or a whole-hand pressure map. Pose observations, pose action targets, hand state and hand actions are separate fields. Hardware environment, kinematics and vendor SDK components are not included in the repository.
Before reuse, pin the code and dataset revisions, record the artifact-specific terms, and inspect a permitted sample's keys, units and episode boundaries. Reconcile training settings too: the manuscript describes 300 epochs, while the current README uses 500 for the light-bulb task. A live wrist pose still anchors relative action execution even though proprioception is excluded from the model's conditioning. None of these observations demonstrates a working installation. The teleoperation data guide shows how to keep measured, mapped and auxiliary tactile signals distinct.
Sources and related resources
- UVTA arXiv v1, submitted September 28, 2026
- Full data, method and evaluation
- Official UVTA project page
- RoboSkin visuo-tactile learning guide
*Evidence review updated November 4, 2026. Original publication dates are retained. We inspected sources and public metadata; we did not run these robot systems or download their training archives.*


