PolyUMI records vision, touch and contact audio in one wireless interface

Researchers at Northwestern University, TU Darmstadt, Hessian.AI and the Robotics Institute Germany released PolyUMI on September 24, 2026. The wireless handheld interface records wrist vision, optical touch, contact audio and proprioception, then moves the same sensing finger to a robot end effector. The authors evaluate separate object classifiers and a slip-control policy, alongside a companion VisTA policy for contact-rich manipulation. Paper and version record.

Key takeaways

• The authors estimate $235.96 in sensing-finger parts and four hours of assembly, excluding the base UMI gripper, robot end effector and integration; moving it between the handheld collector and robot takes about ten minutes.
• In a ten-trial slip-control ablation using a diffusion U-Net trained on 80 demonstrations, vision, touch and audio succeed 8/10 times, versus 2/10 for vision alone. This is a small task-specific experiment, not a general success rate.
• The public repository is useful but not yet a clean reproduction package: several author-specific dependencies remain placeholder links, and its README lists an older tactile stream specification than the paper. Repository notes.

What changed

Portable demonstration tools usually record what a camera sees and how the tool moves. PolyUMI adds signals that expose the physical interaction itself. Its curved-mirror optical finger observes a deformable reflective skin; a piezo contact microphone records structure-borne vibration; a fisheye GoPro supplies wrist vision; and SLAM plus ArUco markers recover the handheld pose and gripper opening.

The paper reports 20 fps tactile video at 1152 by 648 pixels, 16 kHz mono contact audio and 60 fps wrist video at 1920 by 1080. The streams are aligned to a 10 Hz policy clock. Images use a two-step history, while audio becomes a log-Mel spectrogram covering roughly the latest 0.5 seconds. On the robot, those 10 Hz policy commands are interpolated into joint torques at 1 kHz by a Cartesian impedance controller.

VisTA does not collapse every sensor to one vector before fusion. Modality-specific encoders produce 294 tokens across vision, touch, audio and robot state. An eight-layer transformer mixes those tokens across sensor, space and time, and an 18-layer flow-matching model predicts chunks of 16 relative end-effector actions.

Results under the reported conditions

The experiments answer different questions, so their numbers should not be combined into one headline score. A tactile-image classifier reaches 92.3% accuracy over five printed surface patterns, using scene-separated training and test images. This is not a test of arbitrary unseen object categories. For a closed box holding gears, thin screws or thick screws, sensor combinations without audio reach 42% to 44% accuracy; adding contact audio lifts each tested combination to about 80%. Training uses 20 demonstrations per class and validation uses ten across three seeds; the best validation checkpoint is selected. That validation score should not be described as an additional independent held-out test. Object-inference protocol.

Slip control is a ten-trial ablation. The robot must let a screwdriver rotate under gravity, then catch it within ten degrees of vertical. Vision-touch-audio succeeds 8/10, touch-audio 7/10, vision-touch and vision-audio 3/10 each, touch 3/10, vision 2/10 and audio alone 0/10.

For complete manipulation, VisTA reaches 75% full success on board wiping and 100% when partial completion is included. On lightbulb turning, however, every evaluated method reaches at least 80%, and the vision-only diffusion policy performs best. The authors explicitly interpret this as a task where clear visual state makes extra contact sensing less useful. Official results summary.

What this means for robotics

RoboSkin analysis: PolyUMI's most important contribution is not a universal multimodal-policy win. It is a practical bridge between demonstration and deployment. Sharing the same finger preserves sensing geometry, while live contact audio gives the human demonstrator feedback during collection. That can reduce two shifts at once: what the operator can perceive and what the policy later observes.

The results also reinforce a useful design rule for visuo-tactile learning: choose sensors by hidden task state. Audio dominates the sealed-box classification, touch plus audio helps rapid slip control, and extra modalities add little when the goal remains visually obvious.

Limitations and availability

PolyUMI is an arXiv v1 preprint, not an independently reproduced system. Manipulation trials use one Franka setup and modest evaluation counts. The off-the-shelf Franka Hand also required modification for 200 Hz continuous gripper positioning.

The GitHub repository has an MIT root software license and contains hardware, firmware, ingestion, ROS 2 and policy tooling. That root license does not establish all third-party hardware or dataset terms. Its README still describes an anonymized review copy, but the public repository now has a visible commit history. Its documentation says several private forks are represented by links that will not resolve until a camera-ready release, so a fresh clone cannot complete every documented setup step today. Training was only tested on an RTX 6000 Ada with 48 GB; the authors estimate at least 32 GB of GPU memory.

There is also a source mismatch worth checking before procurement: the paper and project page report 20 fps tactile video at 1152 by 648, while the current repository README says 10 fps at 540 by 480. That may reflect an older configuration, but no versioned explanation was visible. Teams should treat the manuscript specification as experimental context and verify the repository configuration they can actually build.

Match the experiment and the clock before reusing a result

The 8/10 slip result belongs to the diffusion U-Net experiment, not the VisTA flow-matching model. Wiping and bulb turning evaluate VisTA; object inference uses classifiers. Naming the sensor combination alone is insufficient to identify the trained model or evaluation protocol. Separate experimental setups.

Sharing a physical sensing finger preserves geometry, but it does not automatically synchronize the camera, audio and commands. The pipeline aligns observations using the highest-latency stream and skips early predicted actions according to measured and calibrated delays. Its ten-hertz action grid, predicted chunk of 16 actions, execution of the first three and one-kilohertz impedance loop are distinct quantities.

An integration record should therefore include capture and receive timestamps, alignment and delay calibration, dropped samples, executed action indices and controller settings. Keep the reported finger-only bill of materials separate from assembly, host compute, the UMI gripper and robot hardware. Resolve the paper/README image-rate mismatch against the actual build revision before procuring parts. These are proposed checks, not a RoboSkin installation. The teleoperation guide compares this shared-sensor route with cross-embodiment signal mapping.

Sources and related resources

• PolyUMI arXiv v1, submitted September 24, 2026
• Full methods and evaluation
• Official PolyUMI and VisTA project
• Official MIT-licensed repository
• RoboSkin tactile dataset directory

*Evidence review updated November 4, 2026. Original publication dates are retained. We inspected sources and public metadata; we did not run these robot systems or download their training archives.*