Self-Decomposing Diffusion: Streamlining Code Generation Efficiency

Paper: Self-Decomposing Diffusion for Code Generation

Authors: Lize Shao, Zichen Xie, Zhizhen Chen, Zhe Zeng, Wenxi Wang

Executive Summary

Recent research into diffusion language models (DLMs) has highlighted their potential for enhancing code generation, particularly through faster and more efficient parallel decoding. However, creating complex programs poses significant challenges due to the interdependencies among various components. These components must work together seamlessly, sharing interfaces, data flows, and assumptions to ensure the overall correctness of the program.

To navigate the balance between speed and accuracy in code generation, researchers have proposed a new methodology called Self-Decomposing Diffusion for Code (SDDC). This framework allows for the decomposition of complex programs into manageable units, which can then be implemented in parallel without jeopardizing the integrity of their connections. The SDDC framework operates in two main stages. First, it generates a contract that outlines how the program is broken down and how the various units depend on each other. This contract acts as a guide during the implementation phase, ensuring that all interdependent units are coordinated effectively.

The effectiveness of SDDC has been demonstrated through extensive benchmarking. In tests across five different benchmarks using two distinct 7 billion parameter diffusion models, SDDC significantly improved the trade-off between accuracy and latency in code generation. Specifically, the framework achieved an increase in pass@1 accuracy of up to 31.1 percentage points compared to baseline models, while also decoding between 7.5 and 13.7 times faster. Additionally, it outperformed a fine-tuning baseline by as much as 23.5 points, while maintaining lower latency.

The implications of this research are substantial for industries that rely on automated code generation, such as software development, fintech, and data analysis. By improving the speed and accuracy of code generation, businesses could potentially reduce development time and costs, enhance productivity, and increase the reliability of generated code. While the results are based on controlled benchmarks rather than real-world applications, they suggest a promising avenue for future exploration and implementation in practical settings.

Overall, SDDC represents a significant step forward in addressing the complexities of code generation in a parallel processing environment, paving the way for more efficient software development practices.

Academic Abstract

Diffusion language models (DLMs) are emerging as a promising paradigm for code generation, offering the potential for fast, parallel decoding. However, generating complex programs remains challenging, as their interdependent components must maintain consistent interfaces, data flows, and shared assumptions. Aggressive parallel decoding can compromise these dependencies and undermine program correctness, while conservative decoding sacrifices efficiency. To address this trade-off, we propose planning program decomposition before implementation to coordinate interdependent units during parallel generation. We introduce Self-Decomposing Diffusion for Code (SDDC), a novel framework that decomposes complex programs and implements their units in parallel, entirely through diffusion. SDDC first generates a contract specifying the program's decomposition and inter-unit dependencies, then uses it to guide coordinated parallel implementation. Across five benchmarks and two 7B diffusion backbones, SDDC substantially improves the accuracy--latency trade-off, raising pass@1 by up to 31.1 percentage points while decoding 7.5--13.7$\times$ faster than the base models, and outperforming a fine-tuning baseline by up to 23.5 points with lower latency.