Robot learning model pillar
Robot world models: prediction for physical action
Learn how robot world models predict future states for planning and control, how they differ from VLA and foundation models, and where tactile prediction fits.
- 6
- sections
- 5
- questions
- 12
- next routes
Short answer
What you need to know
- 1
A robot world model predicts how an environment, robot state, observation, reward, or contact state may change after an action. The prediction can be in pixels, tactile observations, explicit state, or a learned latent representation.
- 2
A world model is not automatically a robot policy, VLA, foundation model, or planner. Those components can share a backbone or be combined, but prediction, action selection, language grounding, and reusable pretraining are different roles.
- 3
The decisive evidence is not whether a rollout looks plausible. It is whether prediction supports better planning, control, data efficiency, recovery, or task outcomes under a named robot, task, horizon, baseline, and real-world protocol.
Topic 01
What a robot world model predicts
A world model represents a transition: given a current state or observation and a candidate action, what may happen next? Some systems predict one step; others roll forward over a horizon. Some generate camera or tactile observations, while others predict compact latent state, object pose, contact, reward, or terminal conditions.
The target determines the evaluation. Pixel similarity may be useful for visual prediction but may miss contact geometry or task-critical errors. A latent model may plan effectively without reconstructing every image detail, but its internal state needs task-level validation.
| Prediction space | Typical output | Useful evaluation | Important limitation |
|---|---|---|---|
| Observation-space model | Future RGB, depth, tactile image, audio, or multimodal frame | Prediction metrics plus task use | Plausible images can hide physically wrong state |
| State-space model | Pose, velocity, contact, force-related state, slip, or task variables | Calibrated state error and downstream control | Requires measured or estimated state labels |
| Latent world model | Compact learned future representation | Planning value, probes, ablations, and closed-loop outcome | Latent quality cannot be judged from visualization alone |
| Reward or value model | Predicted success, progress, preference, or return | Ranking quality and policy outcomes | Can exploit narrow labels or miss unmodeled hazards |
Topic 02
World model, VLA, foundation model, and policy
These labels describe different functions even when one system performs several of them. A VLA maps vision and language toward robot actions. A policy selects an action. A world model predicts consequences. A planner searches or selects among possibilities. A foundation-model claim concerns breadth and transfer from pretraining, not a particular prediction target.
| Component | Primary question | Typical output | Required evidence |
|---|---|---|---|
| World model | What may follow this action? | Predicted state, observation, latent, reward, or contact future | Forecast quality and downstream planning or control value |
| Robot policy or VLA | What action should the robot execute? | Command, token, trajectory, or action chunk | Closed-loop task outcomes and generalization boundary |
| Planner or embodied reasoner | Which sequence or subgoal should be attempted? | Action candidates, subgoals, constraints, or task status | Search quality, feasibility, execution, monitoring, and recovery |
| Foundation model | What reusable knowledge transfers across settings? | Representation, prediction, action, or multiple interfaces | Breadth of data, held-out transfer, adaptation cost, and accessible artifacts |
Topic 03
How world models enter robot control
A controller can use a world model to score candidate actions, plan a trajectory, estimate whether a goal is reachable, synthesize training experience, provide an auxiliary learning objective, or detect a mismatch between predicted and measured state. The model may run online or only during training.
Long rollouts compound errors, while short horizons may miss delayed consequences. Replanning from new measurements can reduce accumulation, but latency and model uncertainty still matter. Real robots also expose contacts, wear, delays, and disturbances that may be underrepresented in training data.
- State the prediction horizon, observation history, action conditioning, and replanning frequency
- Expose whether the model is used during training, planning, online control, or only visualization
- Compare against a policy without the world-model pathway and against appropriate planning baselines
- Measure physical task value, compute and latency, failure detection, and behavior under distribution shift
Topic 04
The 2026 open model ecosystem signal
Hugging Face’s official LeRobot v0.6 release describes integrations for world-model policies including VLA-JEPA, LingBot-VA, and FastWAM, together with evaluation tooling. This is evidence that open robotics infrastructure is adding world-model workflows; it is not evidence that the named models are directly comparable or solve every real-world task.
Each model still needs its own primary paper, code or weights, dataset, robot, action space, benchmark, and real-robot evaluation. Ecosystem support can improve reproducibility without replacing capability evidence.
Topic 05
Tactile and visuo-tactile world models
Touch becomes valuable after contact begins, when cameras may be occluded and small physical changes can decide success. A tactile world model may predict a future tactile image, contact state, slip, force-related variable, or tactile subgoal conditioned on action. A visuo-tactile model predicts or fuses both visual and tactile futures.
Dream-Tac, TouchWorld, ViTacWorld, FeelWorld, and HiTac-WAM are specialized 2026 preprints with different targets, horizons, sensors, robots, control roles, and baselines. They should not be compressed into a single score. The source-bounded evidence table below preserves action-conditioning class, real-robot protocol, artifact availability, and limitations for each record.
TouchWorld provides a concrete evidence chain: its tactile world-model component is pretrained on EgoTouch, then fine-tuned on robot demonstrations, and evaluated under an author-defined six-task real-robot protocol. The training dataset, model, and evaluation protocol remain separate structured entities.
Topic 06
World-model evidence checklist
A useful result identifies the training data, observation and action interfaces, target representation, horizon, uncertainty, compute, robot, task, baseline, and real-world trial protocol. It also reports what happens when prediction is wrong.
| Evidence question | What to report | Why it matters |
|---|---|---|
| Does it predict? | Target, horizon, metric, calibration, held-out split, and qualitative failures | Shows what future state the model actually captures |
| Does planning use it? | Candidate generation, objective, constraints, replanning, and compute budget | Separates a predictive visualization from an operational planner |
| Does it improve the robot? | Matched baseline, tasks, trials, success, speed, force, recovery, and interventions | Connects prediction to physical utility |
| Does it transfer? | New objects, scenes, tasks, robots, sensors, and adaptation method | Defines rather than implies generalization |
| Is it reproducible? | Paper, code, weights, data, configuration, seeds, logs, and hardware | Allows independent inspection of the result |
Source-bounded evidence center
Compare tactile robot world models by what they actually do
This is not a leaderboard. The records separate candidate-conditioned prediction, joint action-and-future generation, and subtask-conditioned tactile goals, then preserve each paper's robot, sensor, task, trial, artifact, and transfer boundary. Percentages from unlike protocols are not directly comparable.
- 5
- evidence records
- 3
- conditioning roles
- 2026-08-22
- latest review
| Model | Prediction / conditioning | Operational role | Robot / sensor / tasks | Real-robot evidence | Artifacts / license | Evidence boundary / sources |
|---|---|---|---|---|---|---|
| Dream-TacPreprint + public code + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Joint action chunks, future visual observations, and future tactile observations in a shared diffusion latent space. Conditioning class: Joint-generation Actions and future observations are generated together. The source does not present Dream-Tac as an external simulator that scores a fixed set of candidate actions. | Operational roleThe joint visual-tactile future representation directly supports policy action generation. The paper does not report CEM or another candidate-search planner. | Robot / sensor / tasksFranka Emika Panda; fixed third-person and wrist-mounted Intel RealSense D435i cameras; two Xense Photon fingertip sensors; Pick Baguette, Insert USB, Clean Whiteboard, Peel Cucumber, Play Mahjong, and Cut Banana. | Real-robot evidenceThe authors report 20 trials per method and task across six real-robot tasks. Dream-Tac reaches 83.3% mean success using the best-performing checkpoint under the paper's selection rule. | Artifacts / license
| Evidence boundary / sourcesThe paper identifies limited tasks and objects, a simple adjacent-frame tactile-change gate, and high diffusion-model compute. The unaccelerated 10-step configuration is reported at about 5 Hz on an A800. These results do not establish cross-sensor or cross-robot transfer, independent replication, or safe deployment. |
| FeelWorldPreprint + author-run real-robot planning evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Future visual latents plus contact state, a force-related 3D tactile latent, and slip state under the current multimodal state, proprioception, and 7-dimensional robot action. Conditioning class: Candidate-conditioned The dynamics model rolls out candidate robot-action sequences and predicts their visual and hierarchical tactile consequences. | Operational roleContact-aware cross-entropy-method planning samples 400 candidate action sequences over a six-step imagined horizon, executes the first two actions, and then replans. It does not train a task-specific policy for the reported planning evaluation. | Robot / sensor / tasksImeta-Y1 robot; three RGB cameras; DM tactile sensors producing 3D tactile point clouds; chip grasping, fruit grasping, and USB insertion; 200 training trajectories and 40 test trajectories per task. | Real-robot evidenceThe authors report 40 real-robot planning trials per task. Contact-aware CEM reaches 82.5%, 87.5%, and 75.0% on the three tasks, or 81.7% on average. | Artifacts / license
| Evidence boundary / sourcesThe paper identifies the compute cost of CEM planning as a limitation for high-frequency real-time control. Evidence remains tied to one robot, one sensor layout, three tasks, author-defined labels, and the paper's planning protocol; LPIPS is not a direct force, slip, or safety metric. |
| HiTac-WAMPreprint + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: A 24-step bilateral tactile trajectory for each unexecuted action chunk: contact state, contact-conditioned 3D deformation, and slip risk. Conditioning class: Candidate-conditioned The system forecasts tactile consequences for four sampled video-action candidates before execution, then retains the selected forecast as an online reference. | Operational roleHierarchical tactile forecasts and visual task-progress estimates rank candidate chunks. During execution, persistent disagreement between predicted and measured touch triggers withdrawal and corrective replanning. | Robot / sensor / tasksIMETA-Y1 robot; two Intel RealSense D435i cameras, one USB camera, and bilateral DM-Tac W2 sensors; chip grasping, blackboard erasing, and USB insertion; 200 episodes per task split 160/20/20 for training, validation, and test. | Real-robot evidenceThe main comparison uses 30 trials per method-task pair, 360 trials in total. The authors report 61.1% mean success for forecast-guided selection and 72.2% for the full verification-and-replanning system. | Artifacts / license
| Evidence boundary / sourcesThe paper does not separately evaluate forecast quality on model-generated action chunks, slip risk is not calibrated as an online alarm, anomaly statistics come from post-hoc trial-video inspection rather than exhaustive labels, and the fixed-budget comparison is descriptive. Transfer beyond one platform, sensor layout, and three task-specific predictors is not established. |
| TouchWorldPreprint + official project page + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: A short-horizon visual-tactile clip or terminal tactile subgoal for the executable subtask selected by the high-level planner. Conditioning class: Subtask-conditioned The tactile world model is conditioned on the current multimodal observation and an already selected executable subtask, not on competing candidate action sequences. | Operational rolePredicted contact subgoals condition a nominal visuo-tactile policy; a faster tactile residual policy then corrects actions from measured tactile and proprioceptive feedback. The hierarchy separates subtask planning, prediction, nominal action, and reaction. | Robot / sensor / tasksUnnamed humanoid platform with Wuji dexterous hands and a JQ-Industries tactile glove; Water Flower, Tabletop Clearing, Cup Insertion, Power Plug Insertion, Pot Wiping, and Tissue Pulling. | Real-robot evidenceThe paper reports 200 teleoperated training trajectories and 100 real-robot evaluation rollouts per task, without disclosing the exact clean-versus-perturbation split. Author-reported mean success is 65.0% clean and 53.7% under human perturbations. | Artifacts / license
| Evidence boundary / sourcesThe authors identify a six-task scope, short-horizon subgoal prediction, sensor- and hand-specific calibration and adaptation requirements, and fixed multi-rate scheduling. Results come from one unnamed humanoid and tactile-glove layout and are not an independent benchmark or proof of transfer. |
| ViTacWorldPreprint + official project page + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Temporally aligned third-person RGB, wrist RGB, and image-like tactile future observations under H-step relative end-effector motions and gripper commands. Conditioning class: Candidate-conditioned The model predicts policy-conditioned visual-tactile outcomes under supplied robot action chunks and can roll them forward autoregressively. | Operational roleGenerated dream rollouts augment downstream tactile-policy training and provide a pre-deployment imagined policy-evaluation signal. The source does not position ViTacWorld as an online planner or closed-loop controller. | Robot / sensor / tasksFranka Panda, Robotiq 2F-85, fingertip Xense optical tactile sensors, Intel RealSense D435 external camera, and ZED Mini wrist camera; Charger Plugging, Cucumber Peeling, U-Block Insertion, and Cuboid Insertion. | Real-robot evidenceThe setup uses 300 real expert demonstrations plus 50 real policy rollouts per task. Each downstream policy is evaluated in 10 real-robot trials per task; the authors report tactile pi-0.5 mean success rising from 42.5% with expert data to 67.5% after first-round generated-rollout augmentation. | Artifacts / license
| Evidence boundary / sourcesThe paper states that successful dream-data selection still partly relies on manual inspection. Evidence is limited to one Franka setup, four tasks, 10 trials per policy-task pair, author-run evaluation, and unreleased code, weights, and data. |
“Not verified” means the reviewed paper, official project page, and official repository did not expose a reusable artifact on the review date. It does not prove that an artifact cannot exist elsewhere or be released later.
Paper routes
Start with source-backed RoboSkin briefs
Tactile world models / 2026-08-21HiTac-WAM forecasts contact, deformation, and slip before robot actionHiTac-WAM ranks candidate robot actions with hierarchical tactile forecasts, then checks predicted touch against measured touch during execution.Tactile AI / 2026-08-15FeelWorld predicts contact, tactile force states, and slip for robot planningFeelWorld adds explicit contact, force-related tactile, and slip prediction to a visual world model for contact-rich robot planning.Tactile AI / 2026-08-15Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot ManipulationDream-Tac models action-conditioned tactile futures for contact-rich robot manipulation, showing why robot skin data needs prediction, not only reaction.Common questions
FAQ for this topic
What is a robot world model?
It is a model that predicts how robot state, environment state, observations, rewards, or contact may change after an action.
Is a robot world model the same as a VLA?
No. A VLA maps visual and language context toward actions. A world model predicts consequences. One system can include both roles, but the labels are not interchangeable.
How are robot world models evaluated?
Evaluate the prediction target and horizon, then test whether the model improves planning, control, data efficiency, recovery, or task outcomes under matched physical conditions.
What is a tactile world model?
It predicts future tactile observations or contact-related state conditioned on action, sometimes together with visual state, so planning or control can reason about physical contact.
Does a plausible generated video prove a useful world model?
No. Visual plausibility can hide physically wrong geometry, contact, timing, or action consequences. Downstream planning and repeated real-robot evidence are required for stronger claims.