Executable Explanations: Understanding LLM Decision-Making Processes

Paper: Code as Explanations: Executable and Verifiable LLM Explanations

Authors: Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath

Executive Summary

Large language models (LLMs) are becoming integral in decision-making and problem-solving processes across various sectors. However, understanding how these models arrive at their conclusions remains a challenge. This research addresses that gap by proposing a novel method for explaining LLM behavior through executable code. Traditional free-text explanations, like chain-of-thought (CoT) reasoning, provide insight into model decisions but are generated for each individual input and lack the ability to be verified across different inputs.

The authors introduce a system where a program generates explanations for the outputs of LLMs based on a dataset of inputs. This approach allows for automatic verification of the explanations, which is a significant advancement in understanding model behavior. They define a Consistency metric that measures how often the program's output matches the LLM's output across various inputs. This metric is crucial for evaluating and selecting effective explanation programs.

The research tested this framework using three open-source LLMs across eight different tasks. Findings indicate that LLMs can produce code explanations that are not only shorter—by six to one hundred times—than traditional CoT explanations but also more aligned with the model's true behavior in the majority of tested scenarios. Specifically, code explanations outperformed CoT in 9 out of 24 settings.

However, the consistency of these explanations diminishes when dealing with tasks that involve complex linguistic elements. In such cases, the generated programs did not perform better than a baseline of uninformative explanations and tended to rely on longer, hard-coded patterns. The study also explored the effects of providing feedback based on consistency when using the Claude Opus 5.5 model. This feedback improved explanation consistency across tasks, yet even with this enhancement, consistency remained lower for more complex tasks.

The implications of this research are significant for enterprises utilizing LLMs. By adopting executable code as a method for explanation, organizations may gain clearer insights into model behavior, potentially leading to better oversight and trust in AI systems. The proposed method offers a way to verify and understand LLM outputs, which could enhance decision-making processes. The findings suggest that while this approach has promise, particularly with simpler tasks, further exploration is needed to improve consistency in more complex scenarios.

The code for this research is publicly available, allowing interested parties to explore the methodology and potentially apply it in their own use cases.

Academic Abstract

As large language models (LLMs) increasingly make consequential decisions and solve problems alongside humans, we must understand how models arrive at their conclusions. Understanding how an LLM behaves requires observing it across multiple inputs, yet free-text explanations such as chain-of-thought (CoT) are generated per instance and cannot be easily verified on new inputs. We propose {code explanations to address the mismatch in how LLM explanations are generated versus how they are evaluated: a program explains LLM behavior across a dataset of inputs, and its execution enables automatic verification. We define Consistency metric that measures the fraction of inputs on which the program reproduces the LLM's outputs, and use it to evaluate and select candidate programs. Across three open-source code-capable LLMs and eight tasks, we find that LLMs can produce highly consistent code explanations in several settings. Code explanations are also 6$\times$ to 100$\times$ shorter than CoT, and better align with model behavior in 9 out of 24 settings. However, consistency is significantly lower on tasks with more complex linguistic components, where generated programs reproduce model outputs no more frequently than an uninformative explanation baseline, and these programs often rely on longer hard-coded patterns. In experiments with Claude Opus 5.5, providing the model with consistency feedback and code execution improves explanation consistency across tasks, but consistency remains significantly lower on more complex tasks. Our findings establish executable code as a novel, verifiable, and compact approach to explaining LLM behavior and highlight verification feedback as a promising direction for improving explanation consistency. The code for this paper is at https://github.com/rajesh-lab/code-as-explanations.