GARDEN: Streamlined Reward Design for Multi-Agent Learning Systems

Paper: GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

Authors: Dogan Urgun

Executive Summary

The research focuses on enhancing cooperative multi-agent reinforcement learning (MARL), a method where teams of agents, such as robots and autonomous vehicles, work together to learn how to perform tasks effectively. A key challenge in this process is the design of reward functions, which guide the agents on what behaviors to prioritize. Traditionally, these reward functions are created manually and need to be adjusted whenever the task environment or the team composition changes, making this a labor-intensive and repetitive task.

To address this issue, the researchers propose a novel approach called GARDEN (Generalist Autonomous Reward Design through Evidence Navigation). This method utilizes large language models (LLMs) to automate the creation of reward functions. Instead of crafting separate rewards for each unique scenario, GARDEN allows developers to create a single reward program that can be applied across multiple related situations without modification.

The process begins with the LLM analyzing the environment code, training records, and agent behaviors to generate initial drafts of reward functions. These drafts are then tested through trial training runs, while the learning algorithm and selection criteria for the final reward remain unchanged. This approach streamlines the process, reducing the need for repeated manual adjustments.

The effectiveness of GARDEN was evaluated using two benchmark scenarios: Overcooked and SMAX. In the Overcooked scenarios, task performance improved significantly, with returns increasing by 58.2% and 52.5% compared to default rewards in different kitchen layouts. In combat scenarios from the SMAX benchmark, the win rate jumped from 51.0% to 89.9% when using the autonomously engineered rewards. These performance gains suggest that the single reward program can effectively guide agents in various cooperative contexts.

Additionally, GARDEN was compared to other methods, such as fixed diagnostic feedback and Eureka-inspired reward reflection. In controlled tests with similar budgets for candidate and probe functions, GARDEN outperformed these alternatives, achieving win rates of 84.2% and 79.2% respectively.

This research highlights the potential for automating reward function design in MARL, which could save time and resources while improving the performance of multi-agent systems. By reducing the reliance on manual reward crafting, companies developing autonomous systems might find it easier to adapt their technologies to new tasks and environments, enhancing operational efficiency and effectiveness in various applications.

Academic Abstract

Cooperative multi-agent reinforcement learning (MARL) enables teams of agents, such as robots and autonomous vehicles, to learn coordinated behavior, but what such teams learn is largely determined by their reward function. Because task success alone provides too little guidance for learning, intermediate rewards are usually designed by hand, and this effort must often be repeated whenever the environment or the team changes. Large language models (LLMs) can automate the effort by writing reward functions as code, yet the resulting rewards are typically designed and tested in a single setting. To enable their reuse across settings, GARDEN (Generalist Autonomous Reward Design through Evidence Navigation) is proposed, in which one reward program is developed for a family of related scenarios and transferred, unchanged, to scenarios that were not available during its design. During development, environment code, training records, and agent behavior are investigated by an LLM reward engineer, and draft rewards are tested in trial training runs, while the learning algorithm and the rule for selecting the final reward are held fixed. On unseen scenarios of the Overcooked and SMAX benchmarks, task returns were increased by 58.2% and 52.5% over the default rewards on two kitchen layouts, and the mean win rate in two combat scenarios was raised from 51.0% to 89.9%. Under matched candidate and probe budgets, this exceeded the 84.2% and 79.2% obtained with fixed diagnostic feedback and Eureka-inspired reward reflection. These results show that a single autonomously engineered reward can be reused effectively across related cooperative scenarios.