← Back to Paper Radar

UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

Jonathan von Rad, Yong Cao, Andreas Geiger
arXiv.org | 2026
UniComp is a unified benchmark framework that compares pruning, quantization, and knowledge distillation for LLMs along three dimensions: performance, reliability, and efficiency. It shows that compression disproportionately preserves knowledge while degrading reasoning, multilingual, and instruction-following abilities.

Problem Statement

Compression is essential for deploying LLMs, but existing evaluations cover only a few method families and rely mostly on knowledge-centric benchmarks such as MMLU-style QA. Those benchmarks can hide degradation in reasoning, safety, multilingual, and instruction-following behavior. Without a unified, hardware-aware comparison, practitioners lack principled guidance for choosing among pruning, quantization, and distillation.

Key Novelty

  • A unified evaluation framework (UniComp) that compares pruning, quantization, and distillation side by side on three axes: performance, reliability/safety, and hardware-aware efficiency.
  • Identification of a consistent 'knowledge bias' in compressed LLMs: knowledge-intensive tasks are relatively preserved, while reasoning, multilingual, and instruction-following capabilities degrade substantially.
  • Evidence that task-specific calibration data can recover up to 50% of the reasoning ability lost by pruned models, which shows calibration choice is a first-class design decision.

Evaluation Highlights

  • Six compression techniques evaluated on modern LLMs across more than 40 capability- and safety-oriented datasets, plus a hardware-aware efficiency analysis.
  • Quantization gives the best overall retained-performance vs. efficiency trade-off. Distillation gives strong runtime speedups but requires high training compute. Task-specific calibration improves pruned-model reasoning by up to 50%.

Signal Assessment

5/10 This is a solid, useful empirical benchmarking contribution with actionable findings, such as the knowledge bias and the effect of calibration. It does not introduce a new compression method or paradigm, and many of its conclusions sharpen intuitions already held in the community.

Methodology

  1. Select representative techniques from three compression families (pruning, quantization, knowledge distillation), six methods in total, and apply them to modern open LLMs.
  2. Evaluate compressed models on 40+ datasets covering knowledge, reasoning, multilingual, instruction-following, and safety/reliability.
  3. Conduct a hardware-aware efficiency analysis (e.g., runtime/latency, memory, compute cost) to relate performance retention to deployment gains.
  4. Ablate calibration data for pruning methods, comparing generic and task-specific calibration sets to measure recovery of reasoning ability.

System Components

Performance Dimension

Measures retained capability across knowledge, reasoning, multilingual, and instruction-following benchmarks relative to the uncompressed model.

Reliability Dimension

Uses safety-oriented benchmarks to assess whether compression degrades trustworthiness and safety behavior.

Efficiency Dimension

Hardware-aware analysis of runtime acceleration, memory footprint, and the compute cost of producing the compressed model.

Compression Method Suite

Six techniques spanning pruning, quantization, and knowledge distillation, evaluated under a common protocol.

Calibration Study

Analysis of how the choice of calibration data (generic vs. task-specific) affects pruned-model capabilities, especially reasoning.

Results

Metric/Benchmark Baseline This Paper Delta
Knowledge-intensive tasks (retention) Uncompressed LLM Relatively preserved under compression Small degradation (qualitative)
Reasoning / multilingual / instruction-following Uncompressed LLM Substantial degradation under compression Large drop (qualitative)
Performance–efficiency trade-off Pruning / distillation Quantization best overall Best trade-off (qualitative)
Runtime acceleration Quantization / pruning Distillation strongest speedups High gain, but high training cost
Pruned-model reasoning Generic calibration data Task-specific calibration Up to +50%

Key Takeaways

  • Don't validate compressed LLMs only on knowledge benchmarks like MMLU. Explicitly test reasoning, multilingual, instruction-following, and safety, since these degrade most.
  • Default to quantization for the best balance of quality retention and efficiency. Consider distillation only when large runtime speedups justify its substantial training compute.
  • When pruning, choose calibration data that matches the target task (e.g., reasoning traces for reasoning workloads). This can recover up to 50% of the lost reasoning performance at little extra cost.
  • For agentic or multi-step systems that rely on reasoning and instruction following, budget for larger quality losses from compression and evaluate end-to-end.

Abstract

Model compression is increasingly essential for deploying large language models (LLMs), yet existing evaluations are limited in method coverage and focus primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through extensive evaluation of six compression techniques on modern LLMs across more than 40 datasets, we find that (i) compression exhibits a consistent knowledge bias, where knowledge-intensive tasks are relatively preserved while reasoning, multilingual, and instruction-following capabilities degrade substantially; (ii) quantization provides the best overall trade-off between retained performance and efficiency, whereas distillation yields strong runtime acceleration gains at high computational cost; and (iii) task-specific calibration can significantly improve the reasoning ability of pruned models by up to 50%.

Generated automatically from available metadata and abstract on 2026-11-09.