SoftVTBench separates deformable-task completion from contact quality

Evidence review - August 2026

SoftVTBench is an August 19, 2026 arXiv preprint, dataset, and closed-loop benchmark for deformable-object manipulation. The paper asks a narrower question than whether a robot reaches a goal: did the policy complete the task without exceeding an object-specific deformation tolerance?

The latest paper reports 4,000 expert demonstrations across 40 tasks and more than 50 assets. Every trajectory is generated in Isaac Sim and Isaac Lab. The Franka Panda arm, GelSight Mini tactile observations, deformable objects, cameras, contact dynamics, and evaluator states are simulated; this is not a dataset collected with a physical Franka robot or physical GelSight sensors.

Paper scale and official release documentation

The latest paper and the current Hugging Face dataset card both describe 4,000 demonstrations. However, the linked GitHub README still describes an earlier 1,628-demonstration state. These first-party documents are out of sync, so the source revision must travel with any release-scale claim.

The later Hugging Face card is the basis for the currently documented hosted scale, while the GitHub discrepancy remains unresolved. Researchers should pin the hosted revision and inspect its manifests, file inventory, and per-component terms before training or comparing results.

What each simulated episode contains

SoftVTBench records policy-visible observations at 20 Hz and keeps a separate physical state for evaluation.

The simulator configuration uses a Franka arm with a Panda parallel-jaw gripper. GelSight Mini observations are rendered through the TacEx pipeline using Taxim-style optical rendering and FOTS marker motion. The paper explicitly states that it does not compare SoftVTBench outputs with a physical GelSight sensor and does not claim validated sim-to-real transfer for its assets or rendering parameters.

Use the GelSight Mini sensor record and Franka Emika Panda robot record as hardware references, not as evidence that those physical devices collected this dataset.

What Deformation-aware Success Rate measures

The paper defines two connected outcomes:

• Task Success Rate records whether a rollout completes the instructed task.
• Deformation-aware Success Rate, or DSR, also requires peak normalized deformation to remain within a fixed object-specific tolerance.

Before policy training, scripted probing establishes the tolerance for each deformable object. The evaluator then reads policy-hidden FEM state during a rollout. A trajectory receives DSR credit only if it completes the task and stays inside that calibrated deformation zone.

DSR is a paper-defined benchmark metric, not an industry standard, robot-safety certification, physical damage threshold, or universal definition of gentle handling. Its meaning depends on SoftVTBench's simulated objects, FEM model, calibration procedure, percentile choice, and success predicates.

What the reported results show

The paper evaluates Diffusion Policy, π0.5, and FastWAM with paired vision-only and visuo-tactile inputs. Across all 12 in-distribution deformable-object configurations, some task-completing rollouts exceeded the deformation tolerance. The authors report that these hidden violations represented 0.7% to 24% of successful rollouts, depending on the configuration.

Under the paper's distribution-shift protocol, visuo-tactile variants achieved higher task success in all six policy-suite comparisons and higher DSR in five of six. In-distribution tactile effects were mixed. The authors therefore do not claim that adding touch automatically improves every policy; their results show that sensing, control granularity, policy family, and distribution shift interact.

These are author-reported simulated results. They are not an independent leaderboard and should not be transferred to real deformable objects without new calibration, physical sensing, and closed-loop hardware experiments.

Why SoftVTBench matters for tactile AI

Many manipulation datasets expose observations and actions but judge only the terminal outcome. SoftVTBench adds evaluator-only physical state so a rollout can reach the target and still fail an interaction-quality criterion. That separation is useful for the tactile robotics dataset directory and tactile benchmark database: dataset scale, task completion, contact quality, and real-world transfer are different evidence layers.

It also provides a focused test for visuo-tactile learning. Vision observes scene layout and object identity, simulated touch exposes local contact evolution, and hidden FEM state evaluates deformation without leaking that state to the policy. The tactile foundation model guide explains why evaluations across three policy families do not turn SoftVTBench itself into a foundation model.

Availability and license boundary

The official GitHub repository provides training and closed-loop evaluation code and links the externally hosted dataset. The linked Hugging Face card currently labels the dataset Apache-2.0. The repository also warns that dataset components, simulator assets, upstream Franka and GelSight runtime assets, and third-party dependencies may carry different terms.

Treat Apache-2.0 as the label shown on the current SoftVTBench dataset card, not as a blanket relicensing of every upstream asset or dependency. Record the exact dataset revision and preserve per-file notices before redistribution.

Evaluation checklist

• Report the exact paper and hosted-data revisions, and disclose the conflicting GitHub release description.
• State that the robot, tactile sensors, objects, contacts, and FEM ground truth are simulated.
• Keep Task Success Rate, DSR, peak deformation, and drop outcomes separate.
• Document the object-specific calibration and percentile used to set deformation tolerances.
• Compare vision-only and visuo-tactile policies under matched gripper-control encodings.
• Split complete tasks and assets before extracting overlapping temporal windows.
• Test physical sensors, real objects, material shift, calibration drift, and sim-to-real transfer before making deployment claims.

Related RoboSkin resources

• Tactile robotics datasets
• Tactile robotics benchmarks
• GelSight Mini sensor record
• Franka Emika Panda robot record
• Tactile foundation models
• Visuo-tactile robotics

Primary sources

• arXiv: SoftVTBench - A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
• Official SoftVTBench project page
• Official SoftVTBench GitHub repository
• SoftVTBench dataset card linked by the official repository
• Pinned Hugging Face dataset-card revision fd2793a
• GitHub README commit retaining the earlier release description