PrefO2O: Streamlining Reinforcement Learning with Latent Preferences

Paper: PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

Authors: Ruiqi Wang, Dezhong Zhao, Xuesu Xiao, Erdem Biyik, Byung-Cheol Min, Guohua Chen

Executive Summary

Recent advancements in preference-based reinforcement learning (PbRL) highlight its potential for aligning robot behavior with human intent. However, a significant challenge remains: the requirement for costly human feedback for every new task limits scalability. Addressing this issue, research introduces PrefO2O, a framework designed to facilitate "zero-shot" PbRL, allowing robots to learn new tasks without any human input.

PrefO2O operates on the principle of transferring learned preferences from previously encountered tasks to new, unseen tasks. This is achieved by creating a latent preference space that remains consistent across different tasks. By disentangling general preference structures from task-specific variations, PrefO2O can effectively apply knowledge gained from a limited set of offline preference data. This approach means that instead of needing fresh human feedback for each new task, the system can leverage existing data to inform its actions.

The framework employs a method called confidence-weighted optimal transport alignment. This technique generates what are referred to as pseudo-preference signals, which the robot uses to adapt its understanding of rewards during online learning. The result is a more efficient learning process that minimizes reliance on human input.

The research demonstrates the effectiveness of PrefO2O through extensive testing on robotic manipulation and locomotion tasks. The results indicate that PrefO2O significantly outperforms previous preference-transfer methods. In fact, it competes favorably with traditional PbRL methods that incorporate real human feedback, achieving comparable or superior performance. This capability suggests that PrefO2O could streamline various robotic applications by reducing the need for human intervention, thus potentially lowering costs and increasing deployment speed.

While these findings are based on benchmark tests and simulations, they indicate promising directions for future real-world implementations. Industries focused on robotics, such as manufacturing, logistics, and service sectors, may benefit from this technology. PrefO2O could enable robots to adapt quickly to new tasks with minimal human oversight, enhancing operational efficiency and flexibility.

By advancing the ability of robots to learn autonomously, PrefO2O opens up new possibilities for automating complex tasks across various sectors, making it a significant step forward in the development of intelligent robotic systems.

Academic Abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.