Robot learning model pillar
Robot world models: prediction for physical action
Explore world models for robot learning and control: action-conditioned prediction, latent vs video models, manipulation examples and tactile feedback.
- 9
- sections
- 8
- questions
- 12
- next routes
Research reading room / October 2026
World-model papers: surveys and robot manipulation
Start with the surveys to map the field, then examine a concrete insertion study. Survey coverage, simulated control and physical robot evidence are different kinds of evidence.
Predict, compare, execute, observe
A planning example: candidate actions condition predicted futures, a planner compares them, and the robot executes a selected action. This is one use of a world model, not a required architecture for every model.
Original RoboSkin schematic · explanatory, not experimental data
View full-size diagram ↗
Three ways to represent a predicted future
Video, explicit state and latent predictions answer different evaluation questions. A plausible image is not proof of accurate contact dynamics or successful robot control.
Original RoboSkin schematic · explanatory, not experimental data
View full-size diagram ↗Sources checked . Selected reading, not an exhaustive ranking. Descriptions summarize the authors' sources; RoboSkin has not reproduced these experiments. Dates below are first publication dates, not dates inferred from a paper identifier.
Research paper · arXiv preprint; authors list IROS 2026
First published
InsertionWM
Generalizable Robotic Insertion with World Models
InsertionWM uses a visual world model to plan precise robotic insertion across different part geometries. It studies whether model-based reinforcement learning can scale from individual assembly specialists to a generalist that handles unseen assemblies.
Read it for
A concrete example of a world model for contact-rich manipulation, with evidence about object generalization, camera placement and adaptation to new assemblies. It connects predictive control to the practical challenges of occlusion and accurate contact sensing.
Evidence boundary: The reported evaluation is in simulation, with no physical-robot deployment demonstrated in this paper. Occlusion, noisy real depth sensing and contact-model mismatch remain challenges; unseen geometry within this asset distribution does not establish unrestricted real-world generalization.
Method, evaluation and access
- Method
- The system builds on TD-MPC2, learning latent dynamics, rewards and values from wrist-camera depth images and robot proprioception. It selects actions through model predictive control and trains with an adaptive curriculum; its contribution is the assembly system and empirical study, rather than a new underlying RL algorithm.
- Evaluation context
- In Isaac simulation with a Franka Panda setup, the generalist trained on 90 assembly pairs achieved 56% zero-shot success on 10 held-out pairs from the same asset distribution, evaluated over 100 trials per pair. The paper also studies specialist policies, fine-tuning and ablations of vision and training curricula.
- Artifacts and access
- The arXiv paper and full HTML text are accessible. An official code or model-weight release was not verified in this review.
Survey · arXiv preprint
First published
Manipulation world-model survey
World Models for Robotic Manipulation: A Survey
This survey organizes robotic manipulation world models by the future representation they predict, the way predictions guide actions and the stage of learning in which they are used. It distinguishes an action-conditioned world model from a perception module, policy or value function alone.
Read it for
Use it to compare latent world models, action-conditioned video prediction, geometric representations and physics-informed models without treating them as interchangeable. Its discussion of contact, force and tactile data is particularly relevant when evaluating world models for dexterous manipulation.
Evidence boundary: The reviewed methods use heterogeneous tasks and metrics, so the survey does not establish a single best world model. It identifies gaps in contact-rich benchmarking, uncertainty calibration and tests of whether policies exploit errors in learned simulators.
Method, evaluation and access
- Method
- The review compares five representation families and separates integrated prediction-action models from explicit predictive planners. It maps their roles across pretraining, post-training and inference, and reviews 34 manipulation datasets.
- Evaluation context
- This is a synthesis of existing research, not a newly trained model or a new robot benchmark result. It separates evaluation into predictive fidelity, downstream task performance and simulator reliability, explaining why visual quality alone cannot establish control utility.
- Artifacts and access
- The full survey is available on arXiv. Its contribution is a taxonomy and review of datasets and evaluation protocols, rather than model weights.
Survey · arXiv preprint
First published
Robot-learning world-model survey
World Model for Robot Learning: A Comprehensive Survey
A policy-centered survey of how world models support robot learning, including VLA policies, learned simulators and robotic video generation. It connects these uses to planning, evaluation and synthetic training data, with additional coverage of navigation and autonomous driving.
Read it for
A useful reading map for the growing overlap between VLA models and world models. It helps distinguish a model that generates plausible future video from one whose predictions are useful for action selection or policy improvement.
Evidence boundary: Its broad scope does not make heterogeneous results directly comparable or establish reliable physical deployment. The authors highlight weak action conditioning, computational cost, limited non-visual feedback and fragmented evaluation as unresolved issues.
Method, evaluation and access
- Method
- The review compares decoupled video-and-action pipelines, shared backbones, mixture architectures, unified VLA systems and latent predictive models. It separately examines world models as simulators for reinforcement learning and evaluation, alongside datasets and benchmarks.
- Evaluation context
- This survey collates published results and protocols, including LIBERO, RoboTwin, CALVIN and SIMPLER-style evaluations; it does not introduce a new model or report a new controlled benchmark experiment. Its tables preserve benchmark settings and warn that differences in embodiment and protocol limit strict ranking across methods.
- Artifacts and access
- The open-access survey has an accompanying project page and a public reading-list repository. The repository collects research resources rather than providing one unified world-model implementation.
Tactile world model tracker
Recently added
Recent papers from the tracker, sorted by first publication date. These records and assessments are a review preview pending editorial approval.
Paper date:
InternW0
Asynchronous world prediction supports contact-aware post-training; laboratory progress scores do not establish full-workflow success or isolate tactile contributions.
Paper date:
Agile-WAM
Asymmetric visual and tactile prediction supports five physical tasks; source-reported latency is model timing, and code is announced rather than released.
Paper date:
DexTouch-WM
Human pretraining improves reported contact prediction, but generated trajectories give mixed real-robot policy scores; code availability remains unverified.
Short answer
What you need to know
- 1
A robot world model predicts how an environment, robot state, observation, reward, or contact state may change after an action. The prediction can be in pixels, tactile observations, explicit state, or a learned latent representation.
- 2
A world model is not automatically a robot policy, VLA, foundation model, or planner. Those components can share a backbone or be combined, but prediction, action selection, language grounding, and reusable pretraining are different roles.
- 3
The decisive evidence is not whether a rollout looks plausible. It is whether prediction supports better planning, control, data efficiency, recovery, or task outcomes under a named robot, task, horizon, baseline, and real-world protocol.
Topic 01
What a robot world model predicts
A world model represents a transition: given a current state or observation and a candidate action, what may happen next? Some systems predict one step; others roll forward over a horizon. Some generate camera or tactile observations, while others predict compact latent state, object pose, contact, reward, or terminal conditions.
The target determines the evaluation. Pixel similarity may be useful for visual prediction but may miss contact geometry or task-critical errors. A latent model may plan effectively without reconstructing every image detail, but its internal state needs task-level validation.
| Prediction space | Typical output | Useful evaluation | Important limitation |
|---|---|---|---|
| Observation-space model | Future RGB, depth, tactile image, audio, or multimodal frame | Prediction metrics plus task use | Plausible images can hide physically wrong state |
| State-space model | Pose, velocity, contact, force-related state, slip, or task variables | Calibrated state error and downstream control | Requires measured or estimated state labels |
| Latent world model | Compact learned future representation | Planning value, probes, ablations, and closed-loop outcome | Latent quality cannot be judged from visualization alone |
| Reward or value model | Predicted success, progress, preference, or return | Ranking quality and policy outcomes | Can exploit narrow labels or miss unmodeled hazards |
Topic 02
Should world models for robotics be action-conditioned?
For comparing the consequences of robot commands, action conditioning is central: the prediction should depend on which candidate action the robot takes, not only on what usually happens next in a video. The 2026 survey World Models for Robotic Manipulation uses an action-conditioned operational definition. Broader literature sometimes uses "world model" more loosely, so inspect the actual interface rather than assuming it from the name.
Consider an illustrative insertion task. From the same observation, moving forward, backing away and shifting sideways should produce different predicted outcomes where the physical situation warrants it. A text prompt saying "insert the peg" specifies a goal; it does not by itself identify the executed displacement, gripper command or force-related input. A useful model needs a documented route from the candidate robot action to the predicted consequence.
Check whether actions are measured, commanded or inferred, and whether the model sees the complete candidate sequence. Then test action sensitivity on held-out transitions: a convincing-looking future that barely responds to changed commands is weak evidence for planning. This example describes an evaluation design, not a RoboSkin hardware experiment.
Topic 03
Latent vs video world models: choose the information the task needs
A video world model predicts observations that a reader can inspect, such as where an object may move. A latent world model predicts learned features that may preserve the state needed for action selection without reconstructing every pixel. Neither representation guarantees physical accuracy. The right comparison is whether it retains task-critical information at a useful prediction horizon and computation budget.
For world models for robotic manipulation, visual appearance may be less decisive than an occluded contact, a small alignment error or an object slipping inside a gripper. A latent representation can also omit those variables if its training objective does not preserve them. Touch, explicit geometry or state measurements can add information, but their benefit must be established under a matched evaluation.
| Selection question | Video or observation prediction | Latent prediction |
|---|---|---|
| What can a human inspect? | Generated frames expose visible motion and some failure cases. | Feature probes or a separate decoder are needed to interpret the predicted state. |
| What should determine the compute budget? | Resolution, views, horizon, generation steps and candidate count. | Feature size, rollout length, candidate count and any decoding required. |
| What can go wrong? | Plausible texture can conceal incorrect geometry, contact or action response. | A compact representation can discard a small but decisive physical change. |
| How should the choice be validated? | Check action consistency and task-critical state in addition to image quality. | Check action consistency, preserved task state and downstream decisions. |
Topic 04
World model, VLA, foundation model, and policy
These labels describe different functions even when one system performs several of them. A VLA maps vision and language toward robot actions. A policy selects an action. A world model predicts consequences. A planner searches or selects among possibilities. A foundation-model claim concerns breadth and transfer from pretraining, not a particular prediction target.
| Component | Primary question | Typical output | Required evidence |
|---|---|---|---|
| World model | What may follow this action? | Predicted state, observation, latent, reward, or contact future | Forecast quality and downstream planning or control value |
| Robot policy or VLA | What action should the robot execute? | Command, token, trajectory, or action chunk | Closed-loop task outcomes and generalization boundary |
| Planner or embodied reasoner | Which sequence or subgoal should be attempted? | Action candidates, subgoals, constraints, or task status | Search quality, feasibility, execution, monitoring, and recovery |
| Foundation model | What reusable knowledge transfers across settings? | Representation, prediction, action, or multiple interfaces | Breadth of data, held-out transfer, adaptation cost, and accessible artifacts |
Topic 05
How world models enter robot control
A controller can use a world model to score candidate actions, plan a trajectory, estimate whether a goal is reachable, synthesize training experience, provide an auxiliary learning objective, or detect a mismatch between predicted and measured state. The model may run online or only during training.
Long rollouts compound errors, while short horizons may miss delayed consequences. Replanning from new measurements can reduce accumulation, but latency and model uncertainty still matter. Real robots also expose contacts, wear, delays, and disturbances that may be underrepresented in training data.
- State the prediction horizon, observation history, action conditioning, and replanning frequency
- Expose whether the model is used during training, planning, online control, or only visualization
- Compare against a policy without the world-model pathway and against appropriate planning baselines
- Measure physical task value, compute and latency, failure detection, and behavior under distribution shift
Topic 06
World models for robot learning and control: choose by use case
Start by deciding when predictions affect the robot. World models for robot learning can supply pretraining objectives, synthetic transitions or a learned environment for policy improvement. World models for robot control must also fit the online decision loop if their predictions are used during execution. A method trained with future prediction may deploy without running a world model at every step.
Vision-language-action world models can connect prediction and action generation inside one system, or let a separate planner score actions proposed by a VLA. Check which direction the interface runs: predicting a future from an action differs from inferring an action from a desired future. The manipulation survey distinguishes integrated prediction–action models from explicit predictive planners; this distinction helps identify what the model actually contributes.
| Intended use | Required interface | Decision evidence to request |
|---|---|---|
| Learn from imagined experience | Action-conditioned transitions plus the reward or task feedback used by the learner. | Real held-out task performance and whether learned-simulator errors mislead policy improvement. |
| Rank or refine candidate actions | Candidate commands, predicted outcomes and an explicit scoring objective. | Ranking or refinement quality against measured outcomes under the same compute budget. |
| Plan during manipulation | A rollout interface, action constraints, observation updates and replanning schedule. | End-to-end latency, recovery after prediction error and repeated physical trials. |
| Learn predictive features inside a VLA | A future-prediction training target and a documented connection to the action pathway. | An ablation isolating the predictive objective and clarity about which modules run at deployment. |
Topic 07
The 2026 open model ecosystem signal
Hugging Face’s official LeRobot v0.6 release describes integrations for world-model policies including VLA-JEPA, LingBot-VA, and FastWAM, together with evaluation tooling. This is evidence that open robotics infrastructure is adding world-model workflows; it is not evidence that the named models are directly comparable or solve every real-world task.
Each model still needs its own primary paper, code or weights, dataset, robot, action space, benchmark, and real-robot evaluation. Ecosystem support can improve reproducibility without replacing capability evidence.
Topic 08
Tactile and visuo-tactile world models
Touch becomes valuable after contact begins, when cameras may be occluded and small physical changes can decide success. A tactile world model may predict a future tactile image, contact state, slip, force-related variable, or tactile subgoal conditioned on action. A visuo-tactile model predicts or fuses both visual and tactile futures.
Dream-Tac, TouchWorld, ViTacWorld, FeelWorld, and HiTac-WAM are specialized 2026 preprints with different targets, horizons, sensors, robots, control roles, and baselines. They should not be compressed into a single score. The source-bounded evidence table below preserves action-conditioning class, real-robot protocol, artifact availability, and limitations for each record.
TouchWorld provides a concrete evidence chain: its tactile world-model component is pretrained on EgoTouch, then fine-tuned on robot demonstrations, and evaluated under an author-defined six-task real-robot protocol. The training dataset, model, and evaluation protocol remain separate structured entities.
Topic 09
World-model evidence checklist
A useful result identifies the training data, observation and action interfaces, target representation, horizon, uncertainty, compute, robot, task, baseline, and real-world trial protocol. It also reports what happens when prediction is wrong.
| Evidence question | What to report | Why it matters |
|---|---|---|
| Does it predict? | Target, horizon, metric, calibration, held-out split, and qualitative failures | Shows what future state the model actually captures |
| Does planning use it? | Candidate generation, objective, constraints, replanning, and compute budget | Separates a predictive visualization from an operational planner |
| Does it improve the robot? | Matched baseline, tasks, trials, success, speed, force, recovery, and interventions | Connects prediction to physical utility |
| Does it transfer? | New objects, scenes, tasks, robots, sensors, and adaptation method | Defines rather than implies generalization |
| Is it reproducible? | Paper, code, weights, data, configuration, seeds, logs, and hardware | Allows independent inspection of the result |
Source-bounded evidence center
Compare tactile robot world models by what they actually do
This is not a leaderboard. The records separate candidate-conditioned prediction, joint action-and-future generation, and subtask-conditioned tactile goals, then preserve each paper's robot, sensor, task, trial, artifact, and transfer boundary. Percentages from unlike protocols are not directly comparable.
- 5
- evidence records
- 3
- conditioning roles
- 2026-08-22
- latest review
| Model | Prediction / conditioning | Operational role | Robot / sensor / tasks | Real-robot evidence | Artifacts / license | Evidence boundary / sources |
|---|---|---|---|---|---|---|
| Dream-TacPreprint + public code + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Joint action chunks, future visual observations, and future tactile observations in a shared diffusion latent space. Conditioning class: Joint-generation Actions and future observations are generated together. The source does not present Dream-Tac as an external simulator that scores a fixed set of candidate actions. | Operational roleThe joint visual-tactile future representation directly supports policy action generation. The paper does not report CEM or another candidate-search planner. | Robot / sensor / tasksFranka Emika Panda; fixed third-person and wrist-mounted Intel RealSense D435i cameras; two Xense Photon fingertip sensors; Pick Baguette, Insert USB, Clean Whiteboard, Peel Cucumber, Play Mahjong, and Cut Banana. | Real-robot evidenceThe authors report 20 trials per method and task across six real-robot tasks. Dream-Tac reaches 83.3% mean success using the best-performing checkpoint under the paper's selection rule. | Artifacts / license
| Evidence boundary / sourcesThe paper identifies limited tasks and objects, a simple adjacent-frame tactile-change gate, and high diffusion-model compute. The unaccelerated 10-step configuration is reported at about 5 Hz on an A800. These results do not establish cross-sensor or cross-robot transfer, independent replication, or safe deployment. |
| FeelWorldPreprint + author-run real-robot planning evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Future visual latents plus contact state, a force-related 3D tactile latent, and slip state under the current multimodal state, proprioception, and 7-dimensional robot action. Conditioning class: Candidate-conditioned The dynamics model rolls out candidate robot-action sequences and predicts their visual and hierarchical tactile consequences. | Operational roleContact-aware cross-entropy-method planning samples 400 candidate action sequences over a six-step imagined horizon, executes the first two actions, and then replans. It does not train a task-specific policy for the reported planning evaluation. | Robot / sensor / tasksImeta-Y1 robot; three RGB cameras; DM tactile sensors producing 3D tactile point clouds; chip grasping, fruit grasping, and USB insertion; 200 training trajectories and 40 test trajectories per task. | Real-robot evidenceThe authors report 40 real-robot planning trials per task. Contact-aware CEM reaches 82.5%, 87.5%, and 75.0% on the three tasks, or 81.7% on average. | Artifacts / license
| Evidence boundary / sourcesThe paper identifies the compute cost of CEM planning as a limitation for high-frequency real-time control. Evidence remains tied to one robot, one sensor layout, three tasks, author-defined labels, and the paper's planning protocol; LPIPS is not a direct force, slip, or safety metric. |
| HiTac-WAMPreprint + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: A 24-step bilateral tactile trajectory for each unexecuted action chunk: contact state, contact-conditioned 3D deformation, and slip risk. Conditioning class: Candidate-conditioned The system forecasts tactile consequences for four sampled video-action candidates before execution, then retains the selected forecast as an online reference. | Operational roleHierarchical tactile forecasts and visual task-progress estimates rank candidate chunks. During execution, persistent disagreement between predicted and measured touch triggers withdrawal and corrective replanning. | Robot / sensor / tasksIMETA-Y1 robot; two Intel RealSense D435i cameras, one USB camera, and bilateral DM-Tac W2 sensors; chip grasping, blackboard erasing, and USB insertion; 200 episodes per task split 160/20/20 for training, validation, and test. | Real-robot evidenceThe main comparison uses 30 trials per method-task pair, 360 trials in total. The authors report 61.1% mean success for forecast-guided selection and 72.2% for the full verification-and-replanning system. | Artifacts / license
| Evidence boundary / sourcesThe paper does not separately evaluate forecast quality on model-generated action chunks, slip risk is not calibrated as an online alarm, anomaly statistics come from post-hoc trial-video inspection rather than exhaustive labels, and the fixed-budget comparison is descriptive. Transfer beyond one platform, sensor layout, and three task-specific predictors is not established. |
| TouchWorldPreprint + official project page + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: A short-horizon visual-tactile clip or terminal tactile subgoal for the executable subtask selected by the high-level planner. Conditioning class: Subtask-conditioned The tactile world model is conditioned on the current multimodal observation and an already selected executable subtask, not on competing candidate action sequences. | Operational rolePredicted contact subgoals condition a nominal visuo-tactile policy; a faster tactile residual policy then corrects actions from measured tactile and proprioceptive feedback. The hierarchy separates subtask planning, prediction, nominal action, and reaction. | Robot / sensor / tasksUnnamed humanoid platform with Wuji dexterous hands and a JQ-Industries tactile glove; Water Flower, Tabletop Clearing, Cup Insertion, Power Plug Insertion, Pot Wiping, and Tissue Pulling. | Real-robot evidenceThe paper reports 200 teleoperated training trajectories and 100 real-robot evaluation rollouts per task, without disclosing the exact clean-versus-perturbation split. Author-reported mean success is 65.0% clean and 53.7% under human perturbations. | Artifacts / license
| Evidence boundary / sourcesThe authors identify a six-task scope, short-horizon subgoal prediction, sensor- and hand-specific calibration and adaptation requirements, and fixed multi-rate scheduling. Results come from one unnamed humanoid and tactile-glove layout and are not an independent benchmark or proof of transfer. |
| ViTacWorldPreprint + official project page + author-run real-robot evaluationReviewed 2026-08-22 | Prediction / conditioningTarget: Temporally aligned third-person RGB, wrist RGB, and image-like tactile future observations under H-step relative end-effector motions and gripper commands. Conditioning class: Candidate-conditioned The model predicts policy-conditioned visual-tactile outcomes under supplied robot action chunks and can roll them forward autoregressively. | Operational roleGenerated dream rollouts augment downstream tactile-policy training and provide a pre-deployment imagined policy-evaluation signal. The source does not position ViTacWorld as an online planner or closed-loop controller. | Robot / sensor / tasksFranka Panda, Robotiq 2F-85, fingertip Xense optical tactile sensors, Intel RealSense D435 external camera, and ZED Mini wrist camera; Charger Plugging, Cucumber Peeling, U-Block Insertion, and Cuboid Insertion. | Real-robot evidenceThe setup uses 300 real expert demonstrations plus 50 real policy rollouts per task. Each downstream policy is evaluated in 10 real-robot trials per task; the authors report tactile pi-0.5 mean success rising from 42.5% with expert data to 67.5% after first-round generated-rollout augmentation. | Artifacts / license
| Evidence boundary / sourcesThe paper states that successful dream-data selection still partly relies on manual inspection. Evidence is limited to one Franka setup, four tasks, 10 trials per policy-task pair, author-run evaluation, and unreleased code, weights, and data. |
“Not verified” means the reviewed paper, official project page, and official repository did not expose a reusable artifact on the review date. It does not prove that an artifact cannot exist elsewhere or be released later.
Paper routes
Start with source-backed RoboSkin briefs
Tactile world models / 2026-08-21HiTac-WAM forecasts contact, deformation, and slip before robot actionHiTac-WAM ranks candidate robot actions with hierarchical tactile forecasts, then checks predicted touch against measured touch during execution.Tactile AI / 2026-08-15Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot ManipulationDream-Tac models action-conditioned tactile futures for contact-rich robot manipulation, showing why robot skin data needs prediction, not only reaction.Tactile AI / 2026-08-15FeelWorld predicts contact, tactile force states, and slip for robot planningFeelWorld adds explicit contact, force-related tactile, and slip prediction to a visual world model for contact-rich robot planning.Common questions
FAQ for this topic
What is a robot world model?
It is a model that predicts how robot state, environment state, observations, rewards, or contact may change after an action.
Is a robot world model the same as a VLA?
No. A VLA maps visual and language context toward actions. A world model predicts consequences. One system can include both roles, but the labels are not interchangeable.
How are robot world models evaluated?
Evaluate the prediction target and horizon, then test whether the model improves planning, control, data efficiency, recovery, or task outcomes under matched physical conditions.
What is a tactile world model?
It predicts future tactile observations or contact-related state conditioned on action, sometimes together with visual state, so planning or control can reason about physical contact.
Does a plausible generated video prove a useful world model?
No. Visual plausibility can hide physically wrong geometry, contact, timing, or action consequences. Downstream planning and repeated real-robot evidence are required for stronger claims.
Should world models for robotics be action-conditioned?
When the goal is to compare candidate robot commands, the prediction must reflect those commands through a documented action interface. A goal-conditioned video alone does not establish this capability. Test whether changing the candidate action changes the predicted outcome appropriately, and verify those differences against measured transitions.
What is the difference between latent and video world models?
Video world models predict observable frames, while latent world models predict learned features. Video is easier to inspect visually; latent prediction can avoid reconstructing every pixel. Either can lose contact-critical information, so compare action sensitivity, task-relevant state, computation and downstream robot outcomes.
How can world models improve robot learning without running during control?
They can provide predictive representation targets, generate training experience or support policy improvement before deployment. The resulting policy may then execute without an online world-model rollout. Record the training role separately from the deployed system and use an ablation to establish the contribution of prediction.