Mage-VL-Cowork: Streamlining User Interaction with Technology

Paper: Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Authors: Xiaoyi Zhang, Haoqing Wang, Xinjie Zhang, Xiang Ming, Xing Wang, Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Andong Tan, Xinjiang Wang, Jinghao Guo, Yifei Shen, Xiao Li, Peng Zhang, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Bin Li, Yan Lu

Executive Summary

The research introduces Mage-VL-Cowork, a cutting-edge computer-use agent (CUA) designed to enhance user interaction by integrating graphical user interface (GUI) operations with code execution and terminal commands. This innovative approach aims to streamline the way users interact with technology, potentially making complex tasks simpler and more efficient.

Mage-VL-Cowork is based on a hybrid model that employs a multimodal training framework. This framework evolves through a structured five-stage curriculum that includes mid-training, supervised fine-tuning (SFT), and reinforcement learning (RL). This systematic training process allows the agent to learn progressively, improving its capabilities at each stage.

One of the significant advancements presented in the research is the Mage-CUA-Forge, which revolutionizes the traditional method of developing CUAs. Instead of creating agents for each specific task, Mage-CUA-Forge utilizes reusable workflows that adapt to various tasks and conditions. This approach generated over 293,000 desktop and web tasks across nearly 7,000 different skills. As a result, it dramatically reduces the cost associated with developing these agents, achieving a cost reduction of 82.6% per task.

The research outlines how Mage-VL-Cowork-4B improves its performance through a unique teacher-student model. Instead of merely using a standard teacher model for training, it evolves specialized harnesses that guide the agent through different stages of learning. This method ensures the agent produces high-quality outputs and is supervised according to the specific requirements of each training stage.

Performance metrics indicate that Mage-VL-Cowork-4B reached a score of 66.0% on the OSWorld-Verified benchmark, which assesses the agent's ability to perform tasks. This score reflects significant advancements in reasoning efficiency, enabling the agent to achieve state-of-the-art performance relative to its size. This breakthrough suggests that Mage-VL-Cowork could set new standards for on-device intelligence, allowing more powerful and responsive computing solutions.

The implications of this research are far-reaching. By potentially improving the design and implementation of CUAs, businesses could see enhanced productivity and efficiency in task execution. The ability to adapt to various tasks without incurring high costs opens new avenues for companies looking to leverage AI in their operations. The advancements in this area may lead to smarter workplace tools that better support employees, ultimately driving innovation and improving overall business performance.

Academic Abstract

Effective computer-use agents (CUAs) should seamlessly combine visual GUI interaction with code and terminal execution within a single policy. We present Mage-VL-Cowork, a 4B hybrid CUA trained from a general multimodally aligned checkpoint through an end-to-end, progressive five-stage curriculum across midtraining, SFT, and RL. To power this curriculum at scale, Mage-CUA-Forge replaces per-task agent construction with reusable, skill-conditioned generation workflows and application-specific preparers, synthesizing 293,806 desktop and web tasks across 6,915 skills while reducing recurring per-task expenditure by 82.6%. Rather than directly distilling a default teacher, we evolve the teacher harness and instantiate specialized CUA harnesses to yield high-quality trajectories alongside stage-specific supervision. Mage-VL-Cowork-4B finally climbs to 66.0% on OSWorld-Verified through different training stages with more efficient reasoning cost, achieving state-of-the-art performance at its parameter size and pushing forward the frontier of on-device intelligence.