AIRoA-0: Advanced Bimanual Manipulation System from Human Data

Paper: AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data

Authors: Yueh-Hua Wu, Kei Ota, Hanlong Li, Hayata Yuasa, Jason Wang, Kengo Ikeuchi, Kensuke Chiyoma, Makoto Sugiura, Masafumi Takahashi, Masatoshi Tateno, Naoaki Kanazawa, Nicola Scianca, Petr Khrapchenkov, Satoshi Otsubo, Takehiko Ohkawa, Takuya Okubo, Tatsuya Matsushima, Yuki Noguchi, Yuma Shin, Zixuan Zhong

Executive Summary

AIRoA-0 (A-0) is an innovative bimanual manipulation system that has been trained exclusively using data derived from human demonstrations, totaling 16,151 hours. Notably, this training did not involve any robot data collection. The demonstrations were recorded using YUBI, an open-source handheld gripper designed to capture manipulation data in various environments without the need for a robot.

A-0 is capable of performing a range of tasks from a single instruction, including sorting small parts, changing mop pads and brushes on a robot vacuum, unzipping a pencil case, and folding towels. These tasks are managed solely with cameras mounted on each wrist of the robot, without the aid of a head or scene camera. Importantly, the system operates effectively even in environments that were not part of its training data.

During evaluations, A-0 achieved an average success rate of 78% across ten different tasks, significantly outperforming the strongest of four other vision-language-action models, which recorded a success rate of only 27% when trained on the same demonstrations. A-0’s approach is distinctive in that it does not rely on teleoperation data, reinforcement learning, or real-time corrections from a robot; the robot is only utilized during testing.

The system learns to identify critical regions of each wrist-mounted camera view, predicts the progress of tasks, and decodes actions continuously. A-0 is also embodiment-agnostic, meaning it can be operated on various robotic platforms equipped with the YUBI gripper, including models such as the Franka, TX-G2, OpenArm, and UR.

A key aspect of this research is the release of the dataset, which is claimed to be the largest open dataset of handheld-gripper demonstrations, complete with accompanying task and subtask instructions. Along with the dataset, the research team also provides pre-trained model weights and a recipe for post-training. This infrastructure enables users to build the YUBI gripper, record their own tasks, and post-train the model in under four hours using a single node of eight H200 GPUs.

The implications of this research are significant for industries that require automation in manipulation tasks. The ability to leverage a large dataset of human demonstrations without prior robot data could lead to more adaptable and efficient robotic systems across various applications. The open-source nature of the dataset and model further allows businesses and researchers to customize and optimize their robotic solutions, potentially accelerating innovation in automation technology.

Academic Abstract

AIRoA-0 (A-0) is a bimanual manipulation system learned from 16,151 hours of handheld demonstrations, with no robot data collected at any stage. The demonstrations are recorded with YUBI, our open-source handheld gripper, which lets people record manipulation data anywhere with no robot present. From a single instruction, A-0 sorts more than a dozen small parts into their numbered cells and changes the mop pads and brush of a robot vacuum; among other tasks, it unzips a pencil case and folds a towel. It performs these tasks using only a camera on each wrist, with no head or scene camera, at a site where none of its training data was recorded. Across the ten tasks of our evaluation, it averages a success rate of 78%, against 27% for the strongest of four other vision-language-action models post-trained on the same task demonstrations. We collect no teleoperation data and use no reinforcement learning or on-robot corrections, and the robot appears only at test time. The system learns which regions of each wrist view matter for the task, predicts the stage the task has reached, and decodes actions as a continuous stream. A-0 is embodiment-agnostic: it runs on robots that mount the YUBI gripper, such as a bimanual Franka, TX-G2, OpenArm and UR. We are releasing the dataset, to our knowledge the largest open dataset of handheld-gripper demonstrations with task and subtask instructions, together with the pre-trained weights and the post-training recipe. Videos of A-0 on every task are on the project page, https://a-0.airoa.io. Anyone can build the gripper, record a task of their own and post-train the model on it in under four hours on a single node of eight H200 GPUs.