# ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use

DevFeed: [ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use](<https://devfeed.tech/articles/tooldex-an-ego-exo-human-demonstration-dataset-for-creative-physical-tool-use-124610.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09036>)

Author: Mohamed Elmoghany, Kehan Li, Ruoqu Chen, Tarun Chiruvolu, Shutong Zhang, Ryan Wang, Stephen Tian, Yaru Niu, Michelle Guo, Ding Zhao, Weiyu Liu, Mengdi Xu, Li Fei-Fei, Jiajun Wu

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Physical AI](<https://devfeed.tech/topics/physical-ai.md>), [Robot Manipulation](<https://devfeed.tech/topics/robot-manipulation.md>), [Machine Perception](<https://devfeed.tech/topics/machine-perception.md>), [3D](<https://devfeed.tech/topics/3d.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cameras](<https://devfeed.tech/tags/cameras.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [capture](<https://devfeed.tech/tags/capture.md>), [complex](<https://devfeed.tech/tags/complex.md>), [creative](<https://devfeed.tech/tags/creative.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [general](<https://devfeed.tech/tags/general.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [tool](<https://devfeed.tech/tags/tool.md>), [tool-use](<https://devfeed.tech/tags/tool-use.md>)

## AI overview

ToolDex is a real-world dataset of people creatively selecting, adapting, and repurposing available objects as tools. It uses synchronized egocentric cameras and eight exocentric cameras, with multi-view observations and annotations covering hands, tools, target objects, and their deformable interactions. Its benchmark suite addresses physical tool-use reasoning, sparse-view 3D and 4D reconstruction, and robot policy learning. The abstract presents the dataset as a resource for studying human tool use and developing adaptable embodied agents.

## Source excerpt

arXiv:2611.09036v1 Announce Type: cross Abstract: Creative tool use is a fundamental human capability and an important step toward general-purpose embodied intelligence. It requires agents to reason about object affordances, physical constraints, and manipulation strategies, especially when standard tools are unavailable. However, progress in this area is limited by the lack of real-world datasets that capture human tool-use behavior with sufficient scale, fidelity, and diversity. Existing benchmarks are often simulation-based, focus on simplified interactions, or lack detailed observations and annotations. We introduce ToolDex, a large-scale egocentric-exocentric (ego-exo) human demonstration dataset for creative physical tool use. ToolDex captures scenarios in which humans select, adapt, and repurpose available objects as tools. Using synchronized egocentric cameras and eight exocentric cameras, the dataset provides high-fidelity multi-view observations and rich annotations for hands, tools, target objects, and their deformable interactions. Beyond data collection, we establish a benchmark suite characterizing three core capabilities: physical tool-use reasoning, sparse-view 3D and 4D reconstruction, and robot policy learning. ToolDex provides a foundation for studying human tool-use behavior and developing embodied agents that can adaptively use tools in complex real-world environments. Dataset visualizations and benchmarking results are available at https://tooluse-dataset.github.io/.