UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation
Problem Statement
Compression is essential for deploying LLMs, but existing evaluations cover only a few method families and rely mostly on knowledge-centric benchmarks such as MMLU-style QA. Those benchmarks can hide degradation in reasoning, safety, multilingual, and instruction-following behavior. Without a unified, hardware-aware comparison, practitioners lack principled guidance for choosing among pruning, quantization, and distillation.
Key Novelty
- A unified evaluation framework (UniComp) that compares pruning, quantization, and distillation side by side on three axes: performance, reliability/safety, and hardware-aware efficiency.
- Identification of a consistent 'knowledge bias' in compressed LLMs: knowledge-intensive tasks are relatively preserved, while reasoning, multilingual, and instruction-following capabilities degrade substantially.
- Evidence that task-specific calibration data can recover up to 50% of the reasoning ability lost by pruned models, which shows calibration choice is a first-class design decision.
Evaluation Highlights
- Six compression techniques evaluated on modern LLMs across more than 40 capability- and safety-oriented datasets, plus a hardware-aware efficiency analysis.
- Quantization gives the best overall retained-performance vs. efficiency trade-off. Distillation gives strong runtime speedups but requires high training compute. Task-specific calibration improves pruned-model reasoning by up to 50%.
Signal Assessment
Methodology
- Select representative techniques from three compression families (pruning, quantization, knowledge distillation), six methods in total, and apply them to modern open LLMs.
- Evaluate compressed models on 40+ datasets covering knowledge, reasoning, multilingual, instruction-following, and safety/reliability.
- Conduct a hardware-aware efficiency analysis (e.g., runtime/latency, memory, compute cost) to relate performance retention to deployment gains.
- Ablate calibration data for pruning methods, comparing generic and task-specific calibration sets to measure recovery of reasoning ability.
System Components
Measures retained capability across knowledge, reasoning, multilingual, and instruction-following benchmarks relative to the uncompressed model.
Uses safety-oriented benchmarks to assess whether compression degrades trustworthiness and safety behavior.
Hardware-aware analysis of runtime acceleration, memory footprint, and the compute cost of producing the compressed model.
Six techniques spanning pruning, quantization, and knowledge distillation, evaluated under a common protocol.
Analysis of how the choice of calibration data (generic vs. task-specific) affects pruned-model capabilities, especially reasoning.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Knowledge-intensive tasks (retention) | Uncompressed LLM | Relatively preserved under compression | Small degradation (qualitative) |
| Reasoning / multilingual / instruction-following | Uncompressed LLM | Substantial degradation under compression | Large drop (qualitative) |
| Performance–efficiency trade-off | Pruning / distillation | Quantization best overall | Best trade-off (qualitative) |
| Runtime acceleration | Quantization / pruning | Distillation strongest speedups | High gain, but high training cost |
| Pruned-model reasoning | Generic calibration data | Task-specific calibration | Up to +50% |
Key Takeaways
- Don't validate compressed LLMs only on knowledge benchmarks like MMLU. Explicitly test reasoning, multilingual, instruction-following, and safety, since these degrade most.
- Default to quantization for the best balance of quality retention and efficiency. Consider distillation only when large runtime speedups justify its substantial training compute.
- When pruning, choose calibration data that matches the target task (e.g., reasoning traces for reasoning workloads). This can recover up to 50% of the lost reasoning performance at little extra cost.
- For agentic or multi-step systems that rely on reasoning and instruction following, budget for larger quality losses from compression and evaluate end-to-end.
Abstract
Model compression is increasingly essential for deploying large language models (LLMs), yet existing evaluations are limited in method coverage and focus primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through extensive evaluation of six compression techniques on modern LLMs across more than 40 datasets, we find that (i) compression exhibits a consistent knowledge bias, where knowledge-intensive tasks are relatively preserved while reasoning, multilingual, and instruction-following capabilities degrade substantially; (ii) quantization provides the best overall trade-off between retained performance and efficiency, whereas distillation yields strong runtime acceleration gains at high computational cost; and (iii) task-specific calibration can significantly improve the reasoning ability of pruned models by up to 50%.