HumanoidToolBench separates tool choice from task completion
HumanoidToolBench tracks contact, correct selection, lifting and final success across stationary and mobile Unitree G1 tool-use tasks instead of collapsing them into one score.

Researchers from Seoul National University, the University of Massachusetts Amherst and Google Research released HumanoidToolBench on October 1, 2026. The benchmark asks a Unitree G1 to infer which tool can accomplish a goal, grasp it and use it, sometimes while moving along a table. Its 18 tasks, 55 tool assets and 3,094-trajectory ToolBook dataset make the hand-to-object chain measurable instead of treating “tool use” as a single opaque success bit. Paper and version record.
Explore tactile datasets for the broader comparison.
Key takeaways
- Three scenarios—BallMove, BallRetrieve and IceBreak—combine with three execution levels and standard or decoy tool sets to form 18 conditions.
- ToolBook contains 3,003 simulation trajectories and 91 curated real-robot trajectories collected by eight experienced teleoperators.
- The released evaluation toolkit is MIT-licensed, while ToolBook is CC BY-NC 4.0 and third-party assets retain their own terms.
An evaluation funnel for physical tool use
Each scene contains one suitable tool, two irrelevant objects and, in decoy mode, a misleading tool that differs in task-relevant length, shape, mass or compliance. Instructions describe the goal but do not name the tool. Level L0 ends when the correct tool is lifted 8 cm. L1 adds stationary execution. L2 moves the task target so the robot must locomote while carrying the tool.
The environment records five events: any candidate contact, correct first contact, correct-tool lift, tool-target contact and final success. These events help separate selection from grasping and contact execution, but they are not a strictly nested funnel: an episode can touch the wrong tool first and later lift the correct one. “Correct first contact” is still a proxy—it does not prove semantic reasoning—but it shows where a policy begins to diverge from the intended sequence.
ToolBook is built from 1,959 L1/L2 simulation attempts that yielded 1,200 successful demonstrations. The authors cut 1,803 L0 prefixes from attempts in which the correct tool was lifted, including 614 attempts that later failed. Adding 91 real L1 demonstrations gives exactly 3,094 trajectories. Dataset construction and protocol.
What the scores reveal
Simulation evaluation uses one trained policy instance and 100 episodes per task. FastWAM's standard-mode L1 success is 76%, 57% and 86% across the three scenarios, but its mobile L2 results fall to 9%, 20% and 69%. In decoy mode, the corresponding L1 scores are 75%, 45% and 82%; L2 scores are 8%, 11% and 61%. The benchmark changes both target layout and locomotion demand at L2, so the drop cannot be attributed to walking alone.
Averaged across scenarios and L1/L2, FastWAM contacts some candidate in 99.8% of standard episodes and makes the correct tool its first contact in 87.7%, yet completes only 52.8%. Under decoys those values are 100%, 74.7% and 47.0%. Contact is easy; preserving the right grasp and applying the tool is the larger gap.
Real evaluation covers L1 BallMove and BallRetrieve, each in two modes, with ten trials per cell. GR00T N1.7 succeeds in 18 of 40 trials; FastWAM also succeeds in 18 of 40; ACT succeeds in four. Each policy first receives 5,000 additional fine-tuning updates on the 91 real demonstrations. These are small adapted, platform-specific trials, not zero-shot transfer or estimates of general humanoid tool competence.
RoboSkin analysis
The strongest contribution is the intermediate event record. It lets researchers connect robot-hand contact to final behavior: a policy may visually select the right object but fail at lift, maintain contact yet miss the target, or complete a stationary task and fail once whole-body motion is introduced.
Focused probes also warn against over-interpreting success. GR00T N1.7's correct first-contact rate falls from 85% to 74% on unseen BallMove tools and from 59% to 36% on BallRetrieve. In one fixed IceBreak scene, an unrelated instruction still triggers task execution in 84 of 100 rollouts, compared with 77 under the intended instruction. That is evidence of scene-policy shortcuts, not language-grounded tool reasoning.
Availability and replication boundaries
The official repository, rechecked at revision 3fec9bd on November 4, provides the evaluation environment, recording validator and 100-episode protocol, with links to the paper's ACT and Diffusion Policy checkpoints. The maintainers report about two hours and 1 GB of outputs for one condition on their workstation GPU using the released ACT policy, roughly a day and a half and 15 GB for all 18 conditions sequentially. Other policies can take longer; this is not RoboSkin timing. Training pipelines are not included.
The paper is an arXiv v1 preprint. RoboSkin.ai inspected the published paper, project and repository but did not run the GPU benchmark or physical robot tests. Tool assets, third-party models and controller components carry separate terms; users should not infer that the dataset's non-commercial license covers every dependency.
Use an event ledger rather than a success funnel
For a new evaluation, retain each episode's event flags and timestamps alongside final success. The reported percentages count episodes containing an event; dividing correct-tool lifts by correct first contacts does not yield a valid transition probability, because those sets can contain different episodes. Likewise, L0 and L1 use different termination rules. A successful tool lift does not predict a conditional L1 completion rate. Protocol and analysis.
Two less visible contracts also affect comparison:
| Contract | What to record before comparing policies |
|---|---|
| Demonstration identity | Parent collection attempt for each L0 prefix, full trajectory and rerendered view; related slices should not be treated as independent samples in a proposed train/test split |
| Observation and action timing | Image timestamp, inference completion, chunk length and next observation; the 50 Hz command stream does not imply 50 fresh model decisions per second |
| Robot interface | The evaluator's 32-value state and 36-value action order, rather than assuming raw joint recordings or hardware actions have the same shape |
The paper executes chunks without temporal ensembling: ACT's 100 steps span two seconds at 50 Hz, versus 0.8 seconds for GR00T's 40. That difference can change how long a policy acts before it sees a disturbed tool again. It should accompany performance comparisons, not be mistaken for measured inference latency.
The versioned dataset guide additionally distinguishes the raw 43-joint simulation observation and a 35-value hardware supervision contract. Padding a vector does not verify semantic compatibility. Code is MIT; recordings are CC BY-NC 4.0, and external assets retain their own terms. No payload download or execution was performed here. The humanoid evaluation map connects these contracts to whole-body deployment questions.
Sources and related resources
- HumanoidToolBench v1, submitted October 1, 2026
- Full task, dataset and evaluation paper
- Official project and result tables
- MIT-licensed evaluation toolkit and replication requirements
- RoboSkin robotics dataset directory
- RoboSkin benchmark directory
*Source review updated November 4, 2026. Original publication dates are retained. Results are reported by the authors; RoboSkin has not run these systems.*


