PRIME: Reliable Evidence Verification for LLM Agents

Paper: PRIME: Towards Verifiable Evidence-Grounded Reasoning in LLM Agents

Authors: Hamed Khosravi, Xiaoming Huo

Executive Summary

Recent research has introduced a new approach to improve the reliability of Large Language Model (LLM) agents when assessing records against organizational requirements. This is particularly relevant in contexts such as vulnerability reporting, where precise validation is crucial. Traditional LLM agents sometimes fall short by either accepting reports that don't meet specific criteria or by opting for weaker conclusions without fully exploring all validation avenues available.

The study presents a system called PRIME (Precommitted Rules for Inference from Minimal Evidence), designed to enhance the decision-making process of LLM agents. PRIME establishes a framework that outlines how each conclusion can be substantiated before any checks are conducted. This proactive approach ensures that agents only consider conclusions for which there is adequate evidence. Additionally, PRIME includes a mechanism that documents the outcomes of each validation check and mandates that any conclusion drawn must be justified by these results.

One of the key features of PRIME is its ability to direct the LLM agent towards further checks that could support a stronger conclusion or to perform these additional checks autonomously. This is particularly beneficial in complex scenarios where multiple paths to justification exist. By doing so, PRIME effectively restricts conclusions to those that are firmly backed by the evidence collected, thereby reducing the risk of erroneous reporting.

The implications of this research are significant. In a series of 480 public-record tasks evaluated through benchmarks, traditional LLM agents reported unjustified conclusions in 26.7% of cases. In contrast, the PRIME system successfully reported zero unjustified conclusions. Moreover, the average score for justified conclusions improved dramatically from 30.1 to a maximum of 59.2 out of 100 when using PRIME, demonstrating its effectiveness in ensuring higher accuracy and reliability.

The findings suggest that implementing systems like PRIME could lead to more trustworthy outcomes in fields that require rigorous verification processes, such as cybersecurity, compliance, and risk management. With the proper setup of requirements and checks, organizations could enhance their decision-making capabilities and reduce the likelihood of errors stemming from inadequate validation. This could be particularly useful for enterprises that rely heavily on automated systems for critical assessments, as it promises to streamline operations while increasing the reliability of the conclusions drawn from complex data sets.

Academic Abstract

LLM agents are used to check records against an organization's requirements, for example whether a vulnerability report is ready for investigation. Requirements often allow several ways to be met, such as a severity rating with a curator-reviewed advisory or a published advisory link. Agents fail by accepting a report after a required check has failed, or by settling for a weaker conclusion without trying another way that could still succeed. We introduce PRIME (Precommitted Rules for Inference from Minimal Evidence). Before checking, it fixes evidence routes specifying how each conclusion may be justified. A separate program records each check result and refuses any conclusion the results do not justify. It also points the agent to checks that could justify a stronger conclusion, or runs them itself. PRIME reports only conclusions authorized by its rules and ledger. With correct requirements, exact checks at known locations, and sufficient budget at stopping, completion reaches the strongest supported conclusion. On 480 public-record tasks, an agent shown the requirements reported unjustified conclusions on 26.7% of tasks, while PRIME reported none. Its directions and added checks raised the average score for justified conclusions from 30.1, with refusal alone, to the maximum of 59.2 out of 100.