A Comprehensive Survey on Large Language Model Compression for Artificial Intelligence Applications in Edge Systems
Problem Statement
LLMs deliver strong performance but are too large and compute-hungry to run directly on memory- and power-constrained edge devices. Prior work on compression tends to study individual techniques in isolation, leaving practitioners without guidance on how to combine algorithms, adapt to heterogeneous edge hardware, and manage deployment overhead for real-world edge AI services.
Key Novelty
- Reframes model quantization, parameter pruning, and knowledge distillation through a cloud-edge collaborative intelligence lens rather than as standalone algorithmic tricks
- Proposes hybrid model frameworks tailored to dynamic, heterogeneous edge environments, organized by model architecture, application scenario, and technique combination
- Introduces a four-layer software-hardware codesign structure paired with an overhead-aware deployment optimization methodology for practical LLM-at-the-edge systems
- Consolidates a fragmented body of isolated compression research into a single deployment-oriented taxonomy with forward-looking research directions
Evaluation Highlights
- No empirical benchmarks are run; contribution is a qualitative taxonomy and framework synthesis across existing quantization, pruning, and KD literature
- Evaluation is comparative/organizational, contrasting isolated single-technique studies against the survey's proposed hybrid and codesign frameworks to assess deployment readiness
Signal Assessment
Methodology
- Survey and categorize LLM compression techniques (quantization, pruning, KD) from a cloud-edge collaborative intelligence perspective
- Analyze and construct hybrid model frameworks suited to dynamic, heterogeneous edge deployments based on architecture, use case, and technique combination
- Define a four-layer software-hardware codesign scheme and an overhead-aware optimization approach for deployment decisions
- Identify open challenges in current compression approaches and outline future research directions for edge-based LLM services
System Components
Reduces numerical precision of weights/activations to shrink memory footprint and accelerate inference on edge hardware
Removes redundant weights, neurons, or structures to reduce model size and computational cost
Trains smaller student models to mimic larger teacher LLMs, enabling lighter models suitable for edge deployment
Architectural strategies combining cloud and edge resources or multiple compressed model variants, adaptable to dynamic and heterogeneous edge conditions
A structured stack aligning compression algorithms with compiler, runtime, and hardware layers for efficient end-to-end deployment
A decision framework that factors latency, energy, and memory overhead into selecting and configuring compression/deployment strategies
Results
| Aspect | Prior Approaches | This Survey | Improvement |
|---|---|---|---|
| Compression scope | Isolated single-technique studies (quantization/pruning/KD separately) | Unified cloud-edge collaborative view integrating all three | Holistic, deployment-oriented perspective |
| Architecture design | Static, single-model deployment assumptions | Hybrid frameworks for dynamic, heterogeneous edge environments | Greater adaptability to real-world edge variability |
| Systems integration | Algorithm-centric analysis with limited hardware grounding | Four-layer software-hardware codesign + overhead-aware optimization | Practical, deployable end-to-end guidance |
Key Takeaways
- Treat quantization, pruning, and KD as complementary tools within a cloud-edge collaborative pipeline rather than isolated optimizations
- Deployment decisions should be overhead-aware, jointly considering latency, memory, and energy constraints alongside compression ratio and accuracy
- Hybrid, architecture-aware model frameworks are needed to handle heterogeneous and dynamic edge hardware rather than a one-size-fits-all compressed model
- Software-hardware codesign (across algorithm, compiler, runtime, and hardware layers) is essential for translating compression research into production-ready edge LLM services
Abstract
Large language models (LLMs) have achieved remarkable performance across various artificial intelligence (AI) applications. However, current LLMs cannot be deployed directly on edge nodes due to their large number of parameters. Fortunately, model compression technology has been proposed to reduce the computational workload and memory usage of LLMs, enabling further edge-based LLM services. However, existing research typically concentrates on isolated compression algorithms and lacks a comprehensive perspective on how to leverage these techniques for practical, end-to-end LLM deployment in edge environments. In this survey, we review edge-oriented LLM compression techniques and software–hardware codesign strategies to enable efficient LLM deployment on resource-constrained edge systems and guide future research in this area. First, we analyze techniques for LLM compression from the perspective of cloud–edge collaborative intelligence, including model quantization, parameter pruning, and knowledge distillation (KD). Second, we present several hybrid model frameworks tailored to dynamic, heterogeneous edge environments, based on model architecture, application scenarios, and combination selection. Third, we further refine a four-layer software–hardware codesign and an overhead-aware LLM deployment optimization. Finally, we discuss the challenges of current model compression approaches and offer insights into future research directions, with a focus on edge-based LLM services.