Visuo-tactile world models for robot manipulation

A robot world model estimates a future state conditioned on the current observation and a candidate action. A visuo-tactile world model adds measured or predicted touch to that future. Depending on the system, the tactile target may be an image, a learned latent, contact state, force-related information, slip, or a tactile subgoal used by a policy.

This matters most after contact begins. The hand can occlude the object, an insertion can look visually aligned while mechanically jammed, and a grasp can appear stable while beginning to slip. Touch does not make the prediction automatically correct, but it exposes physical interaction evidence that a visual-only model may not observe.

The rows below preserve the role and evidence boundary of each source. Relative improvements are included only when the arXiv abstract states them, and every number remains tied to the authors’ protocol.

A candidate action can be rolled through the model to produce an imagined visual and tactile sequence. A planner can then score that sequence for contact creation, stability, slip, task progress, or failure risk. ViTacWorld adds another use: generating rollouts to augment tactile policy data and evaluating candidate policies under controlled action sequences.

Prediction and feedback have different jobs. TouchWorld explicitly separates slower semantic and predictive planning from high-frequency tactile residual correction. That division is important because an imagined contact future can guide an action, while measured touch still has to correct errors when the real object, friction, alignment, or force differs from the model.

World-model evaluation needs both offline and embodied evidence. Image similarity can diagnose visual rollout quality, but contact prediction needs its own labels or measurements. Planning claims need repeated real-robot trials, comparable baselines, trial counts, held-out conditions, and failures that can be replayed.

Split sequential tactile data by complete trajectory, object, task, sensor, or embodiment according to the transfer claim. Random frame splits can place nearly identical moments from one contact event on both sides of evaluation and make prediction look more general than it is.

All five systems in this comparison are 2026 arXiv preprints as reviewed on August 15, 2026. Their reported results are useful research evidence, but they are not independent validation, certification, production benchmarks, or proof that one model is best across robots.

Do not compare isolated percentages across rows. Each paper uses different prediction targets, horizons, tasks, policies, sensors, baselines, and success definitions. Open the primary source, confirm the current version and released assets, then reproduce the relevant task and hardware contract before making an engineering decision.