PolyUMI records vision, touch and contact audio in one wireless interface

Researchers at Northwestern University, TU Darmstadt, Hessian.AI and the Robotics Institute Germany released PolyUMI on September 24, 2026. The wireless handheld interface records wrist vision, optical touch, contact audio and proprioception, then moves the same sensing finger to a robot end effector. A companion policy called VisTA fuses the synchronized streams for object inference and contact-rich manipulation. Paper and version record.

Key takeaways

• The sensing finger adds an estimated $235.96 in parts and four hours of assembly to the UMI workflow; moving it between the handheld collector and robot takes about ten minutes.
• In a ten-trial slip-control ablation, vision, touch and audio succeed 8/10 times, versus 2/10 for vision alone. This is a small task-specific experiment, not a general success rate.
• The public repository is useful but not yet a clean reproduction package: several author-specific dependencies remain placeholder links, and its README lists an older tactile stream specification than the paper. Repository notes.

What changed

Portable demonstration tools usually record what a camera sees and how the tool moves. PolyUMI adds signals that expose the physical interaction itself. Its curved-mirror optical finger observes a deformable reflective skin; a piezo contact microphone records structure-borne vibration; a fisheye GoPro supplies wrist vision; and SLAM plus ArUco markers recover the handheld pose and gripper opening.

The paper reports 20 fps tactile video at 1152 by 648 pixels, 16 kHz mono contact audio and 60 fps wrist video at 1920 by 1080. The streams are aligned to a 10 Hz policy clock. Images use a two-step history, while audio becomes a log-Mel spectrogram covering roughly the latest 0.5 seconds. On the robot, those 10 Hz policy commands are interpolated into joint torques at 1 kHz by a Cartesian impedance controller.

VisTA does not collapse every sensor to one vector before fusion. Modality-specific encoders produce 294 tokens across vision, touch, audio and robot state. An eight-layer transformer mixes those tokens across sensor, space and time, and an 18-layer flow-matching model predicts chunks of 16 relative end-effector actions.

Results under the reported conditions

The experiments answer different questions, so their numbers should not be combined into one headline score. A tactile-image classifier reaches 92.3% accuracy over five printed surface patterns. For a closed box holding gears, thin screws or thick screws, sensor combinations without audio reach 42% to 44% accuracy; adding contact audio lifts each tested combination to about 80%. Training uses 20 demonstrations per class and validation uses ten, repeated across three seeds. Object-inference protocol.

Slip control is a ten-trial ablation. The robot must let a screwdriver rotate under gravity, then catch it within ten degrees of vertical. Vision-touch-audio succeeds 8/10, touch-audio 7/10, vision-touch and vision-audio 3/10 each, touch 3/10, vision 2/10 and audio alone 0/10.

For complete manipulation, VisTA reaches 75% full success on board wiping and 100% when partial completion is included. On lightbulb turning, however, every evaluated method reaches at least 80%, and the vision-only diffusion policy performs best. The authors explicitly interpret this as a task where clear visual state makes extra contact sensing less useful. Official results summary.

What this means for robotics

RoboSkin analysis: PolyUMI's most important contribution is not a universal multimodal-policy win. It is a practical bridge between demonstration and deployment. Sharing the same finger preserves sensing geometry, while live contact audio gives the human demonstrator feedback during collection. That can reduce two shifts at once: what the operator can perceive and what the policy later observes.

The results also reinforce a useful design rule for visuo-tactile learning: choose sensors by hidden task state. Audio dominates the sealed-box classification, touch plus audio helps rapid slip control, and extra modalities add little when the goal remains visually obvious.

Limitations and availability

PolyUMI is an arXiv v1 preprint, not an independently reproduced system. Manipulation trials use one Franka setup and modest evaluation counts. The off-the-shelf Franka Hand also required modification for 200 Hz continuous gripper positioning.

The GitHub repository carries an MIT license and includes hardware, firmware, ingestion, ROS 2 and policy tooling. But it is an anonymized review copy with no commit history. Its documentation says several private forks are represented by links that will not resolve until a camera-ready release, so a fresh clone cannot complete every documented setup step today. Training was only tested on an RTX 6000 Ada with 48 GB; the authors estimate at least 32 GB of GPU memory.

There is also a source mismatch worth checking before procurement: the paper and project page report 20 fps tactile video at 1152 by 648, while the current repository README says 10 fps at 540 by 480. That may reflect an older configuration, but no versioned explanation was visible. Teams should treat the manuscript specification as experimental context and verify the repository configuration they can actually build.

Sources and related resources

• PolyUMI arXiv v1, submitted September 24, 2026
• Full methods and evaluation
• Official PolyUMI and VisTA project
• Official MIT-licensed repository
• RoboSkin tactile dataset directory