NLP Wiki

User Tools

Site Tools

Trace:
ml:model_compression

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
ml:model_compression [2025/05/07 06:05] – [Pruning & Sparsification] jmflanigml:model_compression [2025/05/12 09:00] (current) – [After Training] jmflanig
Line 15: Line 15:
     * [[https://arxiv.org/pdf/2003.03033.pdf|Blalock et al 2020 - What is the State of Neural Network Pruning?]]     * [[https://arxiv.org/pdf/2003.03033.pdf|Blalock et al 2020 - What is the State of Neural Network Pruning?]]
     * [[https://arxiv.org/pdf/2102.00554|Hoefler et al 2021 - Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks]]     * [[https://arxiv.org/pdf/2102.00554|Hoefler et al 2021 - Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks]]
 +  * **Distillation**
 +    * [[https://arxiv.org/pdf/2402.13116|Xu et al 2024 - A Survey on Knowledge Distillation of Large Language Models]]
 +
  
 ===== General Papers ===== ===== General Papers =====
-  * [[https://arxiv.org/pdf/2002.11794.pdf|Li et al 2020 - Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers]]+  * [[https://arxiv.org/pdf/2002.11794.pdf|Li et al 2020 - Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers]] Related: [[Scaling laws]]
   * [[https://arxiv.org/pdf/2002.11985|Ganesh et al 2020 - Compressing Large-Scale Transformer-Based Models: A Case Study on BERT]]   * [[https://arxiv.org/pdf/2002.11985|Ganesh et al 2020 - Compressing Large-Scale Transformer-Based Models: A Case Study on BERT]]
  
Line 47: Line 50:
   * [[https://www.aclweb.org/anthology/2020.ngt-1.4.pdf|Aji & Heafield 2020 - Compressing Neural Machine Translation Models with 4-bit Precision]]   * [[https://www.aclweb.org/anthology/2020.ngt-1.4.pdf|Aji & Heafield 2020 - Compressing Neural Machine Translation Models with 4-bit Precision]]
   * [[https://arxiv.org/pdf/2101.01321.pdf|Kim et al 2020 - I-BERT: Integer-only BERT Quantization]]   * [[https://arxiv.org/pdf/2101.01321.pdf|Kim et al 2020 - I-BERT: Integer-only BERT Quantization]]
 +  * **Empirical Studies**
 +    * [[https://arxiv.org/pdf/2505.02214|Zheng et al 2025 - An Empirical Study of Qwen3 Quantization]]
 +    * [[https://aclanthology.org/2024.lrec-main.461.pdf|Liu et al 2024 - Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study]]
 +    * [[https://arxiv.org/pdf/2504.04823?|Liu et al 2025 - Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models]]
  
 ==== During Training ==== ==== During Training ====
Line 72: Line 79:
  
 ===== Parameter Sharing ===== ===== Parameter Sharing =====
-  * HashedNets: [[https://arxiv.org/pdf/1504.04788|Chen et al 2015 - Compressing Neural Networks with the Hashing Trick]] Randomly shares weights in the neural network+  * HashedNets: [[https://arxiv.org/pdf/1504.04788|Chen et al 2015 - Compressing Neural Networks with the Hashing Trick]] Randomly shares weights in the neural network by using hashing
  
 ===== Conferences and Workshops ===== ===== Conferences and Workshops =====
ml/model_compression.1746597929.txt.gz · Last modified: 2025/05/07 06:05 by jmflanig
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki