LLMON: Addressing Vulnerabilities in Large Language Models

Paper: Post-Training Large Language Models with LLMON

Authors: Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund

Executive Summary

Recent research has identified significant vulnerabilities in Large Language Models (LLMs) that arise from their treatment of all context tokens equally. This uniformity allows for potential issues such as prompt injection, where malicious or ambiguous instructions can lead to unintended model behavior. The study introduces a solution called LLMON (LLM Object Notation), which is a framework designed to differentiate between executable instructions and passive data. This distinction aims to enhance the reliability of instruction execution within LLMs.

The researchers developed a comprehensive training dataset consisting of 4.6 billion tokens, covering a variety of contexts including general language use, Retrieval-Augmented Generation (RAG), and tool-calling scenarios. This dataset was further enriched with "distractor-infused" variants, which incorporate competing instructions and irrelevant data within the context. The purpose of these distractors is to better simulate real-world conditions that LLMs may encounter.

To validate the effectiveness of LLMON, the researchers fine-tuned two Granite-4.1 model variants, with 3 billion and 8 billion parameters, using supervised learning techniques followed by reinforcement learning. The results of their experiments are compelling. They found that by using LLMON-structured supervision, the success rates of attacks via indirect prompt injection were dramatically reduced—from a range of 53% to 100% down to just 0.02% to 5.6%. This significant reduction demonstrates the potential for LLMON to enhance the security of LLM applications while maintaining their overall capabilities.

In addition to improving security, the study also introduced new metrics for interpretability, which allow for a better understanding of how attention within the model changes when inputs are structured according to the LLMON framework. This interpretability aspect is crucial as it helps in diagnosing failure modes at individual transformer layers, thereby providing insights into how the model processes information.

The findings suggest that implementing explicit execution semantics through standard post-training methods, without needing to alter the underlying architecture of the LLMs, could lead to more robust and secure AI applications. While the results are based on simulations and benchmark tests, they imply significant implications for the future deployment of LLMs in various industries. Companies may consider adopting these methodologies to mitigate vulnerabilities associated with prompt injection and improve the overall reliability of AI systems.

Academic Abstract

Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured training corpus of 4.6B tokens spanning general, RAG (Retrieval-Augmented Generation), and tool-calling data, augmented with distractor-infused variants in which competing instructions and irrelevant data segments are present in context but not referenced. Post-training two Granite-4.1 base models (3B and 8B) through supervised fine-tuning followed by reinforcement learning, we show that LLMON-structured supervision reduces attack success rates on indirect prompt-injection benchmarks from 53-100% to 0.02-5.6% while preserving general capabilities. We further introduce attention-based interpretability metrics that diagnose failure modes at individual transformer layers for structured inputs, revealing how our approach reshapes internal attention patterns. Our results demonstrate that explicit execution semantics can be taught through standard post-training without architectural modification.