NavGPT-3: Advanced Navigation System for Intelligent Robots

Paper: NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

Authors: Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, Xionghui Chen, Stephen Gould, Anton van den Hengel, Qi Wu

Executive Summary

Recent research has developed a new system called NavGPT-3, which enhances the capabilities of robots in navigating and interacting with their environments. This system is significant because it combines advanced reasoning skills from language models with precise, low-latency control mechanisms typically used in physical actions. The result is a more capable autonomous agent that can make more informed decisions while responding quickly to changes in its surroundings.

NavGPT-3 operates through a distinctive architecture that integrates reasoning, action, and monitoring into a cohesive framework. This framework allows different processes to run simultaneously as separate threads, each with its own context and tools. The system can effectively switch control among these threads, enabling the robot to react promptly to unexpected real-world conditions.

At the core of this system is an action policy known as NavGPT VLA, which was trained on 19.28 million examples. This policy utilizes a method called codec allocation to manage visual information based on changes in the environment. The results of the research demonstrate that NavGPT VLA achieves impressive performance metrics, including a success rate of 74.51% on the R2R-CE benchmark and leading performance on the RxR-CE benchmark with a 78.19% success rate.

When NavGPT-3 is fully utilized, it represents a significant advancement in the field, setting a new standard with an 81.51% success rate on R2R-CE. Notably, it performs at human-level competency on the RxR-CE benchmark, with a success rate of 90.43% compared to 90.4% for human operators. Furthermore, it achieves this with a reduced average time of 1 minute and 22 seconds per episode, compared to approximately 3 minutes for humans.

The research also highlights how the design of the system plays a crucial role in enhancing the connection between advanced reasoning and physical control. The interaction between the NavGPT VLA action policy and the reasoning component leads to a significant reduction in reaction time, dropping from a range of 3-19 seconds per decision to about 0.5-1 second per action. This increased responsiveness enables the robot to navigate and react to its environment much more effectively.

All models, code, and evaluation records associated with this research will be made publicly available, potentially allowing other organizations and researchers to build upon these advancements. The implications of this research extend to various fields where autonomous agents are deployed, suggesting that further development could lead to improved performance in tasks requiring both high-level reasoning and quick physical actions.

Academic Abstract

Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3-19 s per language-model decision to 0.5-1 s per action-policy step (1-2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.