RoboRender: Enhancing Robot Training with Simulated Video Generation

Paper: RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Authors: Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei, Jiajun Wu

Executive Summary

The research introduces RoboRender, a new framework designed to improve the training of robots in simulated environments for real-world applications. Traditional methods of training robots using simulations often encounter challenges when these robots are deployed in actual settings. This is largely due to differences in how simulations represent visual information compared to the real world, which can lead to ineffective training outcomes.

RoboRender addresses this problem by creating photorealistic videos from simulated robot movements. It uses a sophisticated model that combines various types of data from the simulation, including depth videos, language instructions, and visual representations of the robot. This approach maintains key aspects of the simulation, such as the robot’s geometry and motion, while enhancing the realism of the environment through detailed textures and backgrounds.

The framework was evaluated using both simulated tests and real-world experiments. In these tests, the video generation model developed within RoboRender was shown to produce higher quality outputs than existing methods that rely solely on depth video. In practical applications involving tasks like picking and placing objects, manipulating articulated items, and mobile navigation, policies trained on the videos generated by RoboRender achieved an average success rate of 71%. This represents a significant improvement, outperforming conventional simulation methods and visual domain randomization techniques by approximately 7.1 times and 3.6 times, respectively.

An important finding of the research is that increasing the number of generated videos from each simulation trajectory can lead to even better policy performance. Specifically, the success rate for initial tasks improved by 65 percentage points when additional videos were incorporated. This indicates that a richer data set can enhance the robot's ability to perform in real-world scenarios.

The outcomes of this research are particularly relevant for industries that rely on robotic automation. By improving the transition from simulation to real-world application, RoboRender could help companies deploy robots more effectively and efficiently, potentially reducing costs associated with training and increasing the overall success of robotic operations. The framework is currently available for further exploration and application at its project website: https://robo-render.github.io/.

Academic Abstract

Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.