PlaySuite: Benchmarking Interactive Visual Intelligence for Enterprises

Paper: PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

Authors: Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini, Michelle Lorena Acevedo Callejas, Mohammad Mahdi Derakhshani, Kristof Meding, Joaquin Vanschoren, Cees G. M. Snoek

Executive Summary

Recent developments in artificial intelligence have led to the creation of multimodal foundation models that perform well on static perception and reasoning tasks. However, these evaluations often miss a critical component of intelligence: the ability to act effectively in dynamic environments over longer periods. To address this gap, researchers have introduced PlaySuite, a new benchmark designed to assess interactive visual intelligence within a wide array of more than 5,000 open-source video games sourced from platforms like PyWeek and itch.io.

These games cover various genres and are built on different engines, including Pygame, HTML5, Godot, and Unity. This diversity is important because it presents models with challenges that are not typically encountered in standard training datasets. As a result, the likelihood of models succeeding merely by recalling memorized solutions or leveraging previously encountered training data is significantly reduced.

To facilitate a scalable evaluation process across this heterogeneous collection of games, the research team developed a unified closed-loop interaction framework. This framework is optimized for high-performance computing (HPC) clusters, allowing for efficient processing and evaluation of the models. Additionally, they introduced a protocol known as Video-LLM-as-a-judge, which links observable gameplay milestones to standardized progress levels, thus providing a clear measurement of performance.

The researchers evaluated fourteen recent AI models, which include various types of vision-language models, computer-use agents, and vision-language-action models. The results of these evaluations reveal a notable perception-action gap: while the models demonstrate strong reasoning abilities, they struggle to maintain progress over time. Specifically, they face significant challenges with spatial grounding, executing actions, and self-correcting during gameplay.

PlaySuite not only serves as a reproducible and extensible testbed for measuring progress from visual perception to goal-directed interaction, but it also lays the groundwork for the future development of AI models capable of acting, adapting, and generalizing in dynamic visual environments. This research is significant because it highlights the limitations of current AI models in real-time interactive contexts, suggesting that further advancements are necessary for effective applications in environments that require sustained engagement and adaptability.

Academic Abstract

Recent advances in multimodal foundation models yield strong performance on static perception and reasoning benchmarks, yet such evaluations largely overlook a central aspect of intelligence: acting competently in dynamic environments over extended time horizons. We introduce PlaySuite, a large-scale benchmark for evaluating interactive visual intelligence across more than 5K open-source video games curated from PyWeek and itch.io. Spanning diverse genres and engines, including Pygame, HTML5, Godot, and Unity, these independent games are largely out-of-distribution for current models, reducing the likelihood that success can be achieved by retrieving memorized walkthroughs or web-scale training artifacts. To enable scalable evaluation across heterogeneous titles, we develop a unified closed-loop interaction framework optimized for HPC clusters alongside a Video-LLM-as-a-judge protocol that maps observable gameplay milestones to standardized progress levels. We evaluate fourteen recent open models spanning vision-language models, computer-use agents, and vision-language-action models. Our results yield strong evidence of a perception-action gap: despite strong reasoning capabilities, current models struggle to make sustained progress and exhibit recurring failures in spatial grounding, action execution, and self-correction. PlaySuite provides a reproducible and extensible testbed for measuring progress from visual perception to goal-directed interaction, and a foundation for developing models that can act, adapt, and generalize in dynamic visual environments.