VISTA: A Benchmark for AI Assistance to Visually Impaired

Paper: VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

Authors: No'e Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi, Zhiyan Li, Selim-Antoine Lali, Weiyue Li, Hasan Kurban, Erchin Serpedin, Gang Luo, Yue Zhang, Minghan Li, Mengyu Wang

Executive Summary

Research into artificial intelligence (AI) assistants for individuals with blindness and low vision has identified significant gaps in current evaluation methods for vision models. Traditional benchmarks mainly focus on visual and text-based inputs, which do not adequately assess the capabilities of these models in real-world scenarios where users rely on a range of sensory inputs. The need for more comprehensive testing is driven by the increasing availability of advanced egocentric wearable devices that can capture data beyond standard visual and textual modalities.

To address these limitations, researchers have developed VISTA (Visually Impaired Scene and Task Annotations), a new multimodal benchmark aimed at enhancing goal-oriented assistance for blind and low-vision users. VISTA includes approximately 1,000 egocentric recordings obtained from Project Aria glasses. Each recording features synchronized sensor data streams and is annotated across ten task categories such as object localization and hazard detection. This diverse data set is designed to better reflect the complexities of everyday tasks that users may encounter.

The evaluation involved testing 14 representative vision-language models to assess their performance in this new framework. A lightweight multimodal model was also created, integrating six different types of input data. While existing models showed some proficiency in describing egocentric scenes, the results indicated that they struggled to generate effective, goal-directed assistive guidance. This finding highlights a critical gap in the capabilities of current AI models, emphasizing the need for further development in this area.

The introduction of VISTA is significant as it provides a more holistic approach to evaluating AI systems intended for assisting blind and low-vision individuals. By incorporating multiple sensory inputs and focusing on goal-oriented tasks, VISTA could enable the development of AI assistants that are better equipped to support users in their daily activities. This research may pave the way for improved accessibility solutions in various environments, helping to empower individuals with visual impairments and enhance their independence.

In summary, VISTA represents a pivotal advancement in the evaluation of AI models for sensory assistance. It not only addresses existing shortcomings in benchmarks but also sets the stage for future innovations that could lead to more effective and practical AI applications in this field.

Academic Abstract

Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.