Forecasting Agents: New Benchmarks for Enhanced Performance

Paper: Benchmarking Forecasting Agents on Yesterday's Tomorrows

Authors: Yuxuan Wang, Yishu Wang, Jiaqi Deng

Executive Summary

Recent advancements in artificial intelligence have allowed digital agents to gather and analyze information effectively, enabling them to tackle complex tasks like forecasting. However, current methods for evaluating these agents’ forecasting abilities fall short. The existing benchmarks either wait for events to fully resolve before providing feedback or rely on static datasets that do not reflect real-time developments. This limitation hinders the progress of research as it restricts the scope of evaluation to pre-collected data.

To address these challenges, researchers have developed FABench, a novel backtesting environment specifically designed for evaluating forecasting agents. FABench offers a scalable solution that continuously adapts, allowing for the assessment of agents using real historical events. It automates the process of discovering past events, formulating questions, and determining outcomes for any date in the past. This flexibility enables researchers to keep the benchmark current and relevant, crucial for advancing model training and improving agent performance.

In its initial release, FABench encompasses 287 questions across nine diverse domains, providing a robust platform for testing. Agents can explore these questions within an environment that combines an offline archive of past data with live web searches. This design mitigates the risk of temporal leakage, where agents might inadvertently use information from the future to inform their predictions.

The research evaluated eight recent forecasting models using FABench. Findings indicate that the ability to retrieve relevant information significantly enhances the accuracy of all tested agents, as measured by the Brier score—a metric used to assess the accuracy of probabilistic predictions. Notably, the best-performing agents achieved results that were slightly above the consensus derived from Polymarket, a prediction market platform, on 54 specific questions.

Despite these promising results, the analysis highlighted persistent weaknesses in how agents evaluate and utilize evidence. This suggests that there remains considerable potential for further improvements in the algorithms driving these agents.

The researchers have made their code, dataset, and the archive environment available to the community, encouraging collaborative exploration and development in the field of forecasting agents. By providing this infrastructure, they aim to foster innovation and enhance the accuracy of AI-driven forecasts in various applications, including finance, weather prediction, and beyond.

Academic Abstract

Digital agents can now gather and organize information, reason over it, and use tools to complete complex tasks. Forecasting puts all of these abilities to the test: it requires integrating incomplete evidence, inferring causal drivers, and committing to calibrated probabilistic judgments. Existing benchmarks are poorly suited to measuring and improving this ability. Live benchmarks provide feedback only after events resolve, so every evaluation cycle waits on the calendar; fixed-corpus evaluations return results immediately but confine research to pre-collected documents. We introduce FABench, a scalable backtesting environment for Forecasting Agents. An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs; its first release poses 287 questions across nine domains. Agents research each question freely in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage. Evaluating agents built on eight recent models, we find that retrieval improves the Brier score of all eight in every retrieval setting and lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it. Research traces reveal recurring failures in how agents weigh and apply evidence, leaving ample room for improvement. We release our code, data, and archive environment as a foundation for community research on forecasting agents.