Praxis: Enhancing Robot Object Manipulation Through Video Learning

Paper: Praxis: Distilling Physical Interaction Priors from Egocentric Videos for Generalizable Whole-Body Manipulation

Authors: Shuliang He, Ruiyan Xu, Bo Yue, Hengming Zhang, Huayi Zhou, Shuai Wang, Wei-Shi Zheng, Guiliang Liu

Executive Summary

Recent research has introduced a new framework called Praxis, designed to enhance mobile humanoid robots' ability to manipulate objects in dynamic environments. This framework addresses a key challenge: effectively learning to interact with objects based on limited demonstrations while adapting to changes in their positions and the conditions of contact.

Praxis operates through a three-stage process. First, it utilizes vision-language-guided navigation, allowing the robot to move towards target objects intelligently. This is followed by closed-loop posture calibration, which ensures that the robot's arm and hand are correctly positioned to engage with the object. Finally, the framework enables dexterous manipulation that coordinates both the upper and lower body of the robot, ensuring smooth and effective interactions.

A significant feature of Praxis is its ability to learn from just one demonstration provided by a human. This means that the robot does not require extensive retraining for each specific task. Instead, it can generalize skills across different tasks, adjusting its actions based on real-time visual feedback. This feedback mechanism helps the robot adapt to new object positions and configurations while also incorporating tactile feedback to fine-tune hand movements based on actual contact conditions.

The research includes experiments involving five long-horizon manipulation tasks, showcasing the framework's capability for spatial and visual generalization. Additionally, Praxis demonstrates resilience by recovering from physical disturbances during manipulation, an essential trait for robots operating in unpredictable environments.

The implications of this research are substantial for industries that rely on automation and robotics. By enabling robots to learn and adapt more effectively, organizations could improve operational efficiency and reduce dependency on extensive programming and training for every new task. Industries such as warehousing, logistics, and manufacturing may benefit significantly from this technology, as it allows for greater flexibility and responsiveness in robotic systems.

While the results are promising, it is important to note that the findings stem from controlled experiments and simulations, and further real-world testing will be necessary to validate the framework's performance outside of laboratory conditions.

Academic Abstract

Mobile humanoid manipulation requires both reaching a usable workspace and preserving precise hand-object interactions as object poses and contact conditions change. Learning these behaviors from limited task-specific data remains challenging. To bridge this gap, we introduce Praxis, a whole-body manipulation framework that combines physical interaction priors from one-shot egocentric video demonstrations with closed-loop posture calibration and online perception. The framework coordinates three stages: vision-language-guided navigation toward target objects, closed-loop posture calibration to align the arm-hand workspace, and dexterous manipulation with synchronized upper- and lower-body control. Online visual feedback re-grounds demonstrated interaction geometry under new object poses and scene configurations, while tactile feedback adapts hand motions to actual contact conditions. Each manipulation skill is specified by one human demonstration, without task-specific manipulation-policy retraining. Experiments across five long-horizon manipulation tasks demonstrate spatial, visual, and cross-object generalization, as well as recovery from external physical disturbances across all three stages.