HATCH: Benchmarking Robot Manipulation in Innovative Escape Rooms

Paper: HATCH: Hardware-Attested Task Chains for Verifiable Manipulation

Authors: Shiza Charania

Executive Summary

A recent study introduces HATCH, a new benchmark designed to assess the capabilities of robots in manipulation tasks through an innovative format: a robot escape room. The research highlights the limitations of current benchmarking systems, which are often closed, simulated, or can be manipulated to show favorable results. HATCH aims to overcome these issues by providing a transparent and replicable method for evaluating robotic skills.

The HATCH benchmark is structured around five distinct skill categories, each presenting tasks at three different difficulty levels. This layered approach means that the tasks are interconnected; successfully completing one task unlocks the next, creating a continuous evaluation process. This framework allows for a more comprehensive assessment of a robot's manipulation abilities over time, rather than in isolated tests. The tasks are not only randomized but also scored through sensor data rather than human judgment, ensuring objectivity in the evaluation process.

Participants in the HATCH benchmark must submit their attempts in a specific format. They are required to provide unedited video footage, sensor logs, and the random seed used for object placement, which ensures accountability and transparency. This self-run submission process is complemented by occasional organized events aimed at calibration, further enhancing the reliability of the evaluations.

The open nature of HATCH is significant. All task specifications, building instructions, materials needed, and scoring criteria are made available under an open license. This approach encourages collaboration and innovation within the robotics community, allowing others to replicate the benchmark and contribute to its evolution. By fostering an environment of shared knowledge, HATCH could lead to more robust and reliable advancements in robotic manipulation.

The implications of this research are substantial. By providing a more rigorous and transparent evaluation method, HATCH could help companies and researchers better understand their robots' capabilities. This understanding is critical in sectors where robotic manipulation is becoming increasingly important, such as manufacturing, logistics, and service industries. The standardized evaluation process may also assist in identifying gaps in current robotic technologies and guiding future developments.

While the current findings come from simulations and proposed benchmarks rather than real-world applications, the framework established by HATCH has the potential to impact the robotics field significantly. By shifting towards open and verifiable assessments, the robotics community could see a rise in innovation and improvements in robot design and functionality.

Academic Abstract

Capability claims in robot manipulation are unverifiable, and existing leaderboards are each closed, simulated, or gameable. We propose HATCH, a robot escape room: an openly specified, physically replicable, sensor-scored, and unrehearsable benchmark. Tasks span five skill categories at three difficulty levels and are chained so that the sensor confirming one task's completion unlocks the next, turning a room into a single long-horizon evaluation. Success is confirmed by instrumentation rather than human judgment, scoring is embodiment-agnostic, and object placements are randomized by a run-day seed. Entrants submit attested self-runs (uncut video, sensor logs, and the seed), with occasional organized events for calibration. Task specifications, build guide, bill of materials, and scoring rubrics are released under an open license.