Diffu-LoRA: Streamlined Personalization for Text-to-Image Models

Paper: Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models

Authors: Tianjing Li, Wei Zhu

Executive Summary

Recent research has advanced methods for personalizing text-to-image diffusion models, which are essential for generating images from textual descriptions. Traditional approaches to fine-tuning these models require substantial computational resources and can be cumbersome due to the high number of parameters involved. This study introduces a more efficient technique called Diffu-LoRA, which aims to enhance model personalization while minimizing the demand for extensive computational power.

Diffu-LoRA utilizes a concept known as low-rank adaptation (LoRA), which reduces the number of parameters that need to be adjusted during model training. Unlike full-model fine-tuning, which modifies many parameters indiscriminately, Diffu-LoRA intelligently distributes the adaptation capacity across different layers of the model. This is done by inserting low-rank components into the model's linear layers and assigning a gate to each component that controls its usage. This gating mechanism allows the model to focus more on important parameters while keeping the majority of the model unchanged, thus preserving the foundational knowledge encoded during initial training.

The research employs a technique termed bilevel optimization, which updates adaptation weights and gate parameters using separate data sets. This method ensures that the model learns to allocate its resources effectively. Additionally, progressive pruning is employed to remove components that are deemed less important, based on the gate values. This allows the model to meet a specified budget for rank, effectively managing the complexity and size of the adapted model.

The experiments conducted in this study utilized the Stable Diffusion model and included data from various sources, such as DreamBooth. The results demonstrated that Diffu-LoRA significantly improves subject fidelity and alignment with prompts when compared to standard fine-tuning methods. These improvements indicate that the model can generate more accurate and contextually relevant images based on the input descriptions.

Ablation studies were also performed to analyze the impact of different components of the Diffu-LoRA method, confirming the effectiveness of bilevel optimization, progressive pruning, and the strategic placement of adapters within the model. The findings suggest that utilizing learned rank allocation is a viable strategy for achieving efficient personalization of diffusion models.

The potential applications of this research could extend to various industries where personalized image generation is valuable, such as advertising, entertainment, and e-commerce. By enhancing the efficiency of model adaptation, companies may be able to deploy advanced image generation technologies more readily, improving customer engagement and satisfaction. Overall, the research presents a promising direction for developing more accessible and practical tools for creating tailored content in a digital landscape.

Academic Abstract

Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.