VELD-QA: Enhancing Question Answering with Visual Evidence
The research introduces VELD-QA, a new benchmark designed to improve question answering (QA) in long documents by incorporating visual evidence. Traditional QA benchmarks often lack comprehensive huma...
Key Facts
- Leverage VELD-QA to enhance your QA systems’ performance on complex document layouts.
- Integrate visual evidence analysis to improve accuracy in interpreting charts and tables.
- Utilize human-annotated datasets for more reliable evaluation of vision-language model capabilities.
- Adopt retrieval-augmented techniques to boost information extraction from lengthy documents.
- Expand training datasets with diverse real-world PDFs to enhance model robustness and applicability.
Summary
Authors: Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yazhe Hu, Lan Guo, Yuying Wang, Mengqing Guo, Bhavesh Dnyaneshwar Gore, Shubham Pathak, Ramapriya Ranganath, Sandeep Sharma, Tao Sheng, Sujith Ravi, Dan Roth
Executive Summary
The research introduces VELD-QA, a new benchmark designed to improve question answering (QA) in long documents by incorporating visual evidence. Traditional QA benchmarks often lack comprehensive human-authored questions and detailed annotations of visual content, which hampers the understanding of how well systems can locate and interpret this evidence. VELD-QA addresses this gap by providing a dataset that includes 15,000 question-answer pairs based on 5,859 real-world PDFs, encompassing a total of 372,445 pages. This dataset includes a variety of tasks, such as interpreting charts, tables, and more complex document layouts.
The benchmark features human annotations that identify relevant pages and visual elements within the documents, which allows for a thorough evaluation of how well systems can retrieve and interpret the necessary evidence. The research evaluates nine vision-language models across different scenarios, including both retrieval-augmented systems and agentic systems, revealing significant obstacles in evidence localization and interpretation.
One key finding is that retrieval-based methods tend to perform poorly compared to the ideal "oracle" setting, which uses the ground-truth supporting pages. Even with full access to documents, these systems do not achieve satisfactory results. The best performance observed in the oracle condition reached an accuracy of 81.7%, indicating ongoing challenges in visual interpretation and reasoning, which remain critical for effective QA.
Additionally, the study shows that simply improving localization is not enough. While the oracle accuracy offers a baseline, the research found that enhancing the input data with visual element annotations can lead to a notable performance improvement of 17-24%. This suggests that adding detailed visual information could help bridge the performance gap, making it a valuable approach for enhancing multimodal document QA systems.
This research could have significant implications for industries reliant on interpreting large volumes of complex documents, such as legal, financial, or academic sectors. By improving the ability of AI systems to process and understand visual elements within documents, organizations may be able to enhance their information retrieval processes, leading to more informed decision-making. The VELD-QA benchmark and accompanying code are made publicly available, encouraging further exploration and development in this area.
Academic Abstract
Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.
Frequently Asked Questions
What business problems does VELD-QA aim to solve?
VELD-QA aims to solve the challenge of effectively answering questions based on long documents by enhancing the ability of systems to locate and interpret visual evidence within those documents. This can improve information retrieval and decision-making processes in businesses that rely on extensive documentation.
Which industries could benefit most from VELD-QA?
Industries such as finance, healthcare, and legal could benefit most from VELD-QA, as they often deal with lengthy documents containing critical visual data like charts and tables that require accurate interpretation for informed decision-making.
What are the practical implementation considerations for businesses looking to utilize VELD-QA?
Businesses may need to consider the integration of VELD-QA with existing question-answering systems and ensure that their teams are trained to utilize the dataset effectively. Additionally, they may need to assess the compatibility of their current document management systems with the benchmark's requirements.
What resources or expertise are needed to implement VELD-QA in a business setting?
Implementing VELD-QA may require expertise in AI and machine learning, particularly in vision-language models. Additionally, businesses would benefit from having skilled data scientists or analysts who can interpret the results and integrate the findings into their processes.
What competitive advantages could businesses gain by leveraging VELD-QA?
Businesses could gain a competitive advantage by improving their document analysis capabilities, leading to faster and more accurate insights. This could enhance customer service, streamline operations, and support better strategic decision-making based on visually-grounded evidence extracted from complex documents.