Visuo-tactile world models for robot manipulation

Compare visuo-tactile world models by predicted states, robot action interfaces, reported trial protocols and transfer limits, with primary sources.

Published | Updated by

November 4, 2026: revisited the cited full papers, corrected the Dream-Tac gain description and added execution-interface questions. Original publication date and the September DexTouch-WM update remain; no models were run.

Layered tactile sensor surface sending signals through processing boards and robot-ready data views.
Technology visual showing tactile sensing layers and signal flow.

Choose by research task

What does each tactile world model do?

Start with the prediction or control problem you need to study. These are source-reported research roles, not a ranking under one shared benchmark.

VT-WM

Learns multi-task visual and tactile contact dynamics for imagination and planning.

Read the VT-WM primary paper ↗

Dream-Tac

Jointly models robot actions, future visual observations, and tactile dynamics with contact-gated fusion.

Read the Dream-Tac primary paper ↗

TouchWorld

Combines predicted tactile subgoals with visuo-tactile action generation and fast tactile residual correction.

Read the TouchWorld primary paper ↗

ViTacWorld

Generates aligned visual-tactile-action rollouts for data augmentation and action-conditioned policy evaluation.

Read the ViTacWorld primary paper ↗

FeelWorld

Predicts contact state, a force-related 3D tactile latent, slip, and visual futures for contact-aware planning.

Read the FeelWorld primary paper ↗
Compare reported results and limitations →

Short answer

What you need to know

  1. 1

    A visuo-tactile world model predicts how visual and tactile state may change after a robot action. Touch grounds the imagined future in contact that cameras may miss, including contact onset, force-related state, slip, and hidden object motion.

  2. 2

    The useful test is not whether a generated rollout looks plausible. Evaluation should show physical consistency, tactile prediction, planning or policy improvement, real-robot outcomes, and transfer beyond the training objects, sensors, tasks, or embodiment.

  3. 3

    VT-WM, Dream-Tac, TouchWorld, ViTacWorld, and FeelWorld solve different parts of the problem. Their source-reported metrics are not a common leaderboard because their targets, tasks, baselines, horizons, sensors, and control roles differ.

Topic 01

What is a visuo-tactile world model?

A robot world model estimates a future state conditioned on the current observation and a candidate action. A visuo-tactile world model adds measured or predicted touch to that future. Depending on the system, the tactile target may be an image, a learned latent, contact state, force-related information, slip, or a tactile subgoal used by a policy.

This matters most after contact begins. The hand can occlude the object, an insertion can look visually aligned while mechanically jammed, and a grasp can appear stable while beginning to slip. Touch does not make the prediction automatically correct, but it exposes physical interaction evidence that a visual-only model may not observe.

Topic 02

2026 visuo-tactile world model comparison

The rows below distinguish the measured quantity, trial protocol and controller role. The November 4 review uses the cited full-paper versions; abstract percentages are not automatically relative improvements. All results are author-reported, not RoboSkin.ai measurements.

SystemWorld-model roleSource-reported evidenceWhat it does not establish
VT-WMLearns multi-task visual and tactile contact dynamics for imagination and planning.The paper reports roughly 33% and 29% relative reductions in normalized trajectory error for moving and nominally static objects. Real-robot open-loop planning is evaluated separately over five initial conditions per task.Trajectory-error reductions are not success-rate gains. The abstract also states up to 35% higher planning success; inspect the task scoring before treating that wording as a transferable effect size.
Dream-TacJointly models robot actions, future visual observations, and tactile dynamics with contact-gated fusion.Section 4 reports 83.3% mean success versus 51.7% for Cosmos Policy: about 31.6 percentage points using the displayed rounded values. Each method has 20 trials per task across six tasks, using its best-performing checkpoint under the stated selection rule.The abstract calls the gain 31.7%; it is not a 31.7% relative improvement. Its 2.9x training and 1.8x inference speedups concern specified compute configurations, not end-to-end control or safety.
TouchWorldCombines predicted tactile subgoals with visuo-tactile action generation and fast tactile residual correction.Across six tasks, the preprint reports 65.0% clean success and 53.7% under perturbations, 15.7 and 18.5 percentage points above its strongest reported baseline.It is a hierarchical predictive-and-reactive system, not a common-score comparison with the other rows.
ViTacWorldGenerates aligned visual-tactile-action rollouts for data augmentation and action-conditioned policy evaluation.Table 1 evaluates four robot tasks with 10 trials per task and policy condition. Adding generated rollouts to real demonstrations improves the reported averages, with results depending on the policy.Generated successful trajectories are still partly selected by manual inspection. A useful rollout generator is not automatically a calibrated predictor of real policy success.
FeelWorldPredicts contact state, a force-related 3D tactile latent, slip, and visual futures for contact-aware planning.The preprint reports 10-step LPIPS from 0.084 to 0.058, 61% lower LPIPS than the visual baseline after 80 steps, and 81.7% average zero-shot planning success.The planning comparison uses 40 trials per method on each of three tasks. LPIPS is not force accuracy; the CEM planner searches a six-step horizon and executes two actions before replanning.

Topic 03

How predicted touch enters robot planning

A candidate action can be rolled through the model to produce an imagined visual and tactile sequence. A planner can then score that sequence for contact creation, stability, slip, task progress, or failure risk. ViTacWorld adds another use: generating rollouts to augment tactile policy data and evaluating candidate policies under controlled action sequences.

Prediction and feedback have different jobs. TouchWorld explicitly separates slower semantic and predictive planning from high-frequency tactile residual correction. That division is important because an imagined contact future can guide an action, while measured touch still has to correct errors when the real object, friction, alignment, or force differs from the model.

  • Observe: synchronize vision, tactile data, robot state, and the action that caused contact
  • Predict: generate future visual, contact, tactile, or slip state for candidate actions
  • Score: define task progress and contact-risk costs that can be checked in the predicted rollout
  • Act: execute only the selected action or short action chunk
  • Correct: use fresh tactile feedback when real contact diverges from the prediction
  • Log: preserve prediction error, measured touch, robot state, and outcome for later evaluation

Topic 04

What happens between a predicted contact and a robot command?

A forecast horizon is not a feedback-loop frequency. Before choosing a model, write down which observations are measured, which are imagined, who selects the action, and when fresh sensor data can change an action already in progress. These interface questions are more transferable than a ranking of unrelated success percentages.

InterfaceConcrete source exampleEvaluation question
Initial context to an open-loop planVT-WM conditions planning on initial RGB and touch; its goal cost is visual.Can a disturbance after planning change the selected sequence, or does the controller need a separate intervention?
Joint future and action generationDream-Tac predicts action chunks with future observations.How old is the measured input when each command executes, and what happens during inference delay?
Nominal actions plus fresh residual correctionTouchWorld commits four corrected actions before refreshing its residual prediction in the described implementation.At the actual robot rate, how long can new contact information wait before altering a command?
Generated observations for later policy trainingViTacWorld uses selected synthetic rollouts to augment demonstrations.Does the trained policy improve on held-out hardware trials, and who accepted or rejected each generated trajectory?

Topic 05

A minimum evaluation protocol

World-model evaluation needs both offline and embodied evidence. Image similarity can diagnose visual rollout quality, but contact prediction needs its own labels or measurements. Planning claims need repeated real-robot trials, comparable baselines, trial counts, held-out conditions, and failures that can be replayed.

Split sequential tactile data by complete trajectory, object, task, sensor, or embodiment according to the transfer claim. Random frame splits can place nearly identical moments from one contact event on both sides of evaluation and make prediction look more general than it is.

  • Visual-only, tactile-reactive, and no-world-model baselines
  • Contact onset, tactile-state, slip, and visual prediction reported separately
  • Short- and long-horizon rollout error with uncertainty
  • Real-robot success, failure mode, disturbance, and trial count
  • Held-out objects, surfaces, tasks, sensors, and robot embodiments
  • Online latency, planning budget, tactile update rate, and correction-loop rate

Topic 06

Claim boundary for the 2026 evidence

This comparison was first published on August 15, 2026. On November 4 we rechecked the cited arXiv full-paper versions and their evaluation protocols. This does not assert their latest publication or artifact-release status. The authors’ results are not independent validation, certification, production benchmarks or proof that one model is best across robots.

Do not compare isolated percentages across rows. Each paper uses different prediction targets, horizons, tasks, policies, sensors, baselines, and success definitions. Open the primary source, confirm the current version and released assets, then reproduce the relevant task and hardware contract before making an engineering decision.

Topic 07

September update: human touch and synthetic policy data

DexTouch-WM adds a separate data-transfer question to the five-system comparison above. Its September 17 preprint keeps robot pretraining data fixed while adding human tactile demonstrations with a shared taxel layout and retargeted pose representation. The largest human-data condition improves contact prediction, but that improvement does not consistently carry into policies trained with generated observations.

In the reported four-task real-robot evaluation, the mixed-data world model produces lower policy-training scores than the real-data baseline for all three tested policies. These are normalized task scores, not binary success rates. Prediction quality, policy-ranking agreement, and final task behavior therefore need separate evaluation. This source update was checked on September 18, 2026; the original five-system table retains its own review scope.

Common questions

FAQ for this topic

01

What is a visuo-tactile world model?

It is an action-conditioned model that predicts future visual and tactile state. The tactile output may represent contact, force-related state, slip, a tactile image, a learned latent, or a policy subgoal.

02

Why add touch to a robot world model?

Touch exposes contact dynamics that cameras may miss under occlusion or ambiguous alignment. It can help a model represent whether contact begins, remains stable, slips, jams, or changes after an action.

03

Is a tactile world model the same as a tactile foundation model?

No. World model describes an action-conditioned predictive role. Foundation model implies broad pretraining and transfer across downstream tasks. A system can be one, both, or neither depending on its actual training and evidence.

04

Which metric should compare visuo-tactile world models?

There is no single sufficient metric. Use modality-specific prediction error, physical consistency, planning or policy improvement, real-robot task outcomes, latency, and held-out transfer under the same protocol.

05

Are 2026 visuo-tactile world models production ready?

The sources reviewed here are preprints. Production readiness still needs independent reproduction plus evidence for latency, calibration drift, durability, failure recovery, sensor replacement, and the target robot environment.