Fractional Power-of-Two Quantization for Efficient LLM Inference

Paper: Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

Authors: Sunghyun Wee, Geunjae Choi, Hyeonjin Kim, Suyoung Kim, Nojun Kwak

Executive Summary

Recent research has focused on improving the efficiency of Large Language Models (LLMs), particularly in how they perform inference. This process involves a significant amount of compute and energy use, primarily driven by multiply-accumulate (MAC) operations. To address these costs, post-training quantization (PTQ) has emerged as a strategy to reduce resource requirements by converting model weights and activations into low-bit integers.

One promising approach is Power-of-Two (PoT) weight quantization, which allows for a multiplier-free inference method that utilizes shift-add logic. However, this technique faces challenges, particularly a notable drop in accuracy when using low bit-widths due to the rough quantization grid of base-2.

To tackle this issue, the research introduces a new method called Fractional PoT (FPoT). This innovative technique employs a base of $\sqrt{2}$ for fractional quantization, which allows for effective 4-bit weight quantization while still utilizing a fully integer shift-add processing path. FPoT integrates several key elements: it combines the $\sqrt{2}$ weight grid with uniform integer activations, refines grid alignment during calibration, and employs a compact 1-adder approximation that minimizes error during the weight calibration phase.

The effectiveness of FPoT was tested using the Llama-3-8B model and showed promising results. Specifically, with 4-bit quantization for both weights and activations, FPoT enhanced zero-shot commonsense reasoning performance by 8.72% compared to the baseline PoT method, while remaining close to the performance of a uniform integer baseline, with only a slight decrease of 0.4%. These findings were consistent across various configurations involving seven different models and three different bit-widths, indicating robustness in the method.

Importantly, FPoT can be integrated with existing state-of-the-art PTQ methods, highlighting its versatility and potential for broad applications in LLMs. This research suggests that organizations leveraging LLMs could adopt FPoT to improve computational efficiency and accuracy, potentially leading to more sustainable model deployments. The results stem from benchmark tests rather than real-world applications, but they point to a practical path forward in optimizing LLM performance while managing resource use.

Academic Abstract

Large Language Models (LLMs) demand efficient inference, where the multiply-accumulate (MAC) operations in linear projections dominate compute and energy cost. Post-training quantization (PTQ) reduces this cost by mapping weights and activations to low-bit integers. Power-of-Two (PoT) weight quantization promises efficient multiplier-free LLM inference via shift-add logic, but suffers significant accuracy degradation at low bitwidths due to its coarse base-2 quantization grid. We introduce Fractional PoT (FPoT), a base-$\sqrt{2}$ (half-octave) fractional PoT grid that achieves practical 4-bit weight quantization while preserving a fully integer shift-add datapath. The method combines (i) the base-$\sqrt{2}$ weight grid with uniform integer activations, (ii) a calibration-time grid-alignment refinement absorbed into the static per-channel scale, and (iii) a lightweight 1-adder $\sqrt{2}$ approximation whose error is absorbed during weight calibration via approximate-grid quantization, yielding a processing element (PE) with significantly fewer gates than a 4-bit integer multiplier. On Llama-3-8B with 4-bit weight and activation quantization, FPoT improves zero-shot commonsense reasoning performance by 8.72% relative to the PoT baseline, while maintaining performance close to the uniform integer baseline, with only a negligible 0.4%p degradation. These results hold across various configurations spanning seven models and three bit-widths. Furthermore, FPoT can be integrated with state-of-the-art LLM PTQ methods, demonstrating framework-agnostic applicability.