Differences

This shows you the differences between two versions of the page.

--- nlp:pretraining [2026/07/22 05:52] – [Pretraining Methodology] jmflanig
+++ nlp:pretraining [2026/08/03 03:13] (current) – [Pretraining (and Post-training) Methodology] jmflanig
@@ Line 100: / Line 100: @@
   * [[https://arxiv.org/pdf/2007.06778.pdf|Tu et al 2020 - An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models]]
-====== Pretraining Methodology ======
+====== Pretraining (and Post-training) Methodology ======
 See also [[ml:scaling laws]].
   * **Blog posts**:
@@ Line 123: / Line 123: @@
     * [[https://arxiv.org/pdf/2407.21783|Llama Team 2024 - The Llama 3 Herd of Models]]
     * [[https://arxiv.org/pdf/2412.19437|2024 - DeepSeek-V3 Technical Report]]
+    * [[https://arxiv.org/pdf/2512.02556|2025 - DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models]]
+    * [[https://arxiv.org/pdf/2606.19348|2026 - DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence]]
+    * [[https://arxiv.org/pdf/2607.02770|Google Team 2026 - Gemma 4 Technical Report]]
   * **Closed Model System Cards**
     * These sometimes contain some interesting information, especially OpenAI's system cards
-    * [[https://arxiv.org/pdf/2412.16720|2024 - OpenAI o1 System Card]]
+    * [[https://arxiv.org/pdf/2412.16720|OpenAI 2024 - OpenAI o1 System Card]]
+    * [[https://deploymentsafety.openai.com/gpt-5-4-thinking/gpt-5-4-thinking.pdf|OpenAI 2026 - GPT-5.4 Thinking System Card]]
+    * [[https://deploymentsafety.openai.com/gpt-5-5/gpt-5-5.pdf|OpenAI 2026 - GPT-5.5 System Card]]
     * [[https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf|OpenAI 2026 - GPT-5.6 Preview System Card]]
@@ Line 166: / Line 171: @@
   * [[Language Model]]
   * [[nlp:vision_and_language#Multimodal Pretraining]]
+  * [[Post-training]]
   * [[Prompting]]
   * [[ml:Semi-supervised Learning]]