FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference

FLRC improves low-rank LLM compression by replacing uniform per-layer compression ratios with a fine-grained, layer-wise optimal rank allocation, and introduces progressive low-rank decoding to prevent quality degradation during autoregressive text generation.

Problem Statement

LLMs' massive parameter counts make them costly to deploy on resource-constrained hardware, and while low-rank compression can shrink memory and compute costs, existing methods apply a one-size-fits-all compression ratio across layers, causing uneven and often severe performance loss. Additionally, prior low-rank approaches degrade substantially during the decoding (generation) phase, limiting their usefulness for real-world text generation tasks like summarization.

Key Novelty

An efficient algorithm for determining an optimal, layer-specific rank allocation instead of a uniform global compression ratio
Progressive low-rank decoding mechanism designed specifically to preserve generation quality during autoregressive inference
A unified fine-grained compression framework that jointly addresses both weight compression and decoding-time quality degradation, unlike prior work that focuses only on static compression

Evaluation Highlights

Up to 17% improvement in ROUGE-L on summarization benchmarks over state-of-the-art low-rank compression baselines
Comprehensive evaluation across diverse benchmarks demonstrating consistent superiority over prior low-rank compression methods

Signal Assessment

5/10 FLRC delivers a meaningful, well-validated improvement over existing low-rank compression techniques by tackling a concrete, practical weakness (decoding degradation and rigid uniform allocation), but it extends an established compression paradigm rather than introducing a fundamentally new approach.

Methodology

Assess per-layer sensitivity/importance to compression to guide a fine-grained rank allocation strategy across the model
Efficiently search or compute optimal ranks per layer under a global compression budget, avoiding costly exhaustive search
Apply low-rank decomposition (e.g., SVD-based factorization) to weight matrices using the allocated per-layer ranks
Introduce progressive low-rank decoding that adapts computation across decoding steps to mitigate error accumulation during generation
Evaluate the compressed model on summarization and other diverse NLP benchmarks against SOTA low-rank compression baselines

System Components

Fine-grained Rank Allocator

Determines an optimal, non-uniform rank/compression ratio for each layer based on layer-wise importance or sensitivity, rather than applying one global ratio.

Low-Rank Decomposition Core

Performs the underlying matrix factorization (e.g., SVD-based) to compress weight matrices at the allocated per-layer ranks.

Progressive Low-Rank Decoding

A decoding-time mechanism that preserves generation quality by progressively adjusting low-rank computations, addressing degradation seen in prior methods during autoregressive inference.

Results

Metric/Benchmark	SOTA Low-Rank Baseline	FLRC	Delta
ROUGE-L (Summarization tasks)	Baseline score	Improved score	Up to +17%
Decoding-time generation quality	Noticeable degradation	Maintained quality	Qualitative improvement
Performance across diverse benchmarks	Inconsistent under uniform compression	Consistently superior	Broad improvement

Key Takeaways

Uniform compression ratios across all layers are suboptimal; allocating compression budget per layer based on sensitivity yields substantially better quality-efficiency trade-offs
Compression methods should be evaluated explicitly on decoding/generation behavior, not just static weight reconstruction error, since degradation can emerge specifically during autoregressive inference
FLRC offers a practical, potentially drop-in low-rank compression pipeline for reducing LLM memory and compute footprint without requiring specialized quantization hardware
Summarization and other generation-heavy tasks are particularly sensitive benchmarks for exposing weaknesses in existing compression techniques, making them useful stress tests for future methods

Abstract

Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.