Differences

This shows you the differences between two versions of the page.

--- nlp:fine-tuning [2022/03/02 19:12] – created jmflanig
+++ nlp:fine-tuning [2022/03/02 19:16] (current) – removed jmflanig
@@ Line 1: / Line 1: @@
-====== Fune-Tuning ======
-This page lists fine-tuning methods such as Adaptors, BitFit, NoisyTune, etc.
-{{media:fine-tuning-methods.png}}\\
-Figure from [[https://arxiv.org/pdf/2106.04647.pdf|Mahabadi 2021]].
-See also [[ml:Optimization#Instability of Fine-tuning|Optimization - Instability of Fine-tuning]].
-  * Adaptor Layers: [[https://arxiv.org/pdf/1902.00751.pdf|Houlsby et al 2019 - Parameter-Efficient Transfer Learning for NLP]]
-  * [[https://arxiv.org/pdf/2006.04884.pdf|Mosbach 2020 - On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines]] Advocates a simple baseline in section 6: fine-tune using ADAM with bias correction and a learning rate of 2e−5 for 20 epochs, with learning rate linearly increased for the first 10% of steps and linearly decayed to zero afterward.
-  * Gradual Fine-Tuning: [[https://arxiv.org/pdf/2103.02205.pdf|Xu et al 2021 - Gradual Fine-Tuning for Low-Resource Domain Adaptation]]
-  * [[https://arxiv.org/pdf/2106.04489.pdf|Mahabadi et al 2021 - Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks]]
-  * [[https://arxiv.org/pdf/2106.04647.pdf|Mahabadi et al 2021 - COMPACTER: Efficient Low-Rank Hypercomplex Adapter Layers]]
-  * [[https://arxiv.org/pdf/2106.10199.pdf|Ben-Zaken et al 2021 - BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models]]
-  * [[https://arxiv.org/pdf/2202.12024.pdf|Wu et al 2022 - NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better]]