URAI: Streamlining Robot Control for Enhanced Performance

Paper: Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools

Authors: Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, Xuan Shangguan, Jay Zhu, Yijun Liu, Lingyu He, Sihang Wu, Xiao He, Hongcheng Gao

Executive Summary

Recent research has focused on improving how robots manipulate objects through a new framework known as URAI (Universal Robot-Agent Interface). Traditional approaches to robot control often involve complex decision-making processes that can slow down performance and require significant computational resources. This research proposes a shift in how tasks are executed by separating the responsibilities of programming and execution into two distinct agents.

The programming agent is responsible for creating reusable tools that are tailored to specific tasks. It generates these tools based on the intent behind a task and refines them through feedback received during execution. The execution agent, on the other hand, uses these tools to carry out actions while maintaining a feedback loop that allows for real-time adjustments. This dual-agent system enables continuous decision-making, as the model retains control over the actions taken between tool uses, rather than relying solely on pre-written programs.

The effectiveness of URAI was validated through simulations involving five RoboDojo tasks, where it significantly improved success rates—from 18% to 53%—compared to direct fingertip control. Notably, the largest improvement was observed in the Swap Blocks task, where the system demonstrated a success rate of 56% when agents made decisions after each tool invocation. In contrast, a pre-written program only achieved a 24% success rate using the same tools. Additionally, the execution time for tasks was reduced, with three out of four agents completing episodes 1.3 to 1.5 times faster while generating 1.5 to 1.7 times fewer output tokens.

Beyond theoretical validation, URAI was also tested on seven practical tasks using AgileX dual-arm robots. These tasks included various forms of object manipulation, cloth folding, and even human interaction during a game of tic-tac-toe. The framework’s ability to integrate coding and decision-making allows for a more efficient and adaptable approach to robot control.

This research matters because it could lead to more efficient robotic systems that can better adapt to real-world tasks. By employing a model that combines programming with decision-making, robots could potentially handle complex manipulations more effectively, reducing the time and resources needed for execution. The implications of this research could extend to various sectors where robots are deployed, from manufacturing and logistics to customer service and interactive applications, although further exploration and real-world testing will be necessary to fully understand its impact.

Academic Abstract

Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.