Memorilla: Enhancing LLMs with Advanced Memory Capabilities

Paper: Memorilla: Latent Semantic Memory for LLMs

Authors: Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, Jure Leskovec

Executive Summary

Recent research has introduced a new memory system called Memorilla, designed to enhance the capabilities of large language models (LLMs) by allowing them to utilize information beyond their immediate parameters. Traditional LLMs are limited in their ability to recall user conversations or long documents due to constraints in their context windows. Memorilla addresses this challenge by gathering relevant information from extensive document collections and converting it into a fixed number of 'memory tokens' that can be used by a frozen LLM.

The innovation behind Memorilla lies in its ability to encode documents into reusable formats called embeddings. This allows the system to efficiently retrieve and integrate query-specific information into the LLM's input. The research reports that by using only 16 memory tokens, Memorilla outperforms existing systems like Mem0 and MemoRAG across five out of eight benchmarks. For instance, on the FactKG benchmark, Memorilla achieved an accuracy of 84.33%, which is notably higher than the 80.43% accuracy attained through full-context inference while using a significantly reduced number of tokens—108 times fewer, to be precise.

Another key application of Memorilla is in supporting continual learning in agentic systems, where the amount of information grows over time. In a simulated environment called TextWorld, Memorilla demonstrated the ability to improve an agent's performance by reducing its tendency to revisit previously explored areas. The success rate for the agent improved from 18.6% with traditional full-history in-context learning to 24.2% when utilizing the Memorilla memory system.

These findings highlight the potential of Memorilla for enhancing the effectiveness of frozen LLMs in managing long-context interactions and facilitating ongoing learning processes in AI systems. While the results are based on benchmarks and simulations rather than real-world applications, the implications for industries that rely on LLMs for customer interactions, document analysis, and personalized services could be significant. Companies could consider exploring similar memory systems to improve the functionality and efficiency of their AI-driven tools, particularly in areas where maintaining context and knowledge accumulation is crucial.

Academic Abstract

Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.