Differences

This shows you the differences between two versions of the page.

--- ml:pytorch [2021/05/28 23:29] – jmflanig
+++ ml:pytorch [2023/11/09 21:10] (current) – [PyTorch] jmflanig
@@ Line 8: / Line 8: @@
      * [[https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html]] Good seq2seq tutorial and starter code (doesn't have batching)
      * [[https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb|pytorch-seq2seq example code]] Good seq2seq code with batching
+     * [[https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec|How to code The Transformer in Pytorch]]
+     * BERT classification tutorials
+       * [[https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb|Fine-tuning a BERT model on a text classification]]. Doesn't show how to make predictions on new data.  For that see [[https://huggingface.co/docs/transformers/training|Fine-tuning a pretrained model]]
+       * [[https://huggingface.co/docs/transformers/training|Fine-tuning a pretrained model]]
+  * Frameworks
+    * [[https://www.pytorchlightning.ai/|PyTorch Lightning]] A higher-level API for PyTorch that removes a bunch of boilerplate code
   * Models
     * Transformer
       * [[https://github.com/SamLynnEvans/Transformer]] Nice implementation (Rongwen used it) with a tutorial: [[https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec|How to code The Transformer in Pytorch]]
   * Books
-     * [[https://www.amazon.com/Deep-Learning-PyTorch-Eli-Stevens/dp/1617295264/ref=sr_1_1_sspa?dchild=1&keywords=pytorch&qid=1613431235&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExOUFKSTZGN0VHVVI0JmVuY3J5cHRlZElkPUEwMzIwODQ5SE9MRFEyRVBIWllFJmVuY3J5cHRlZEFkSWQ9QTA4NjkwMTBKNFY2UUg4N0pNQkwmd2lkZ2V0TmFtZT1zcF9hdGYmYWN0aW9uPWNsaWNrUmVkaXJlY3QmZG9Ob3RMb2dDbGljaz10cnVl|Deep Learning with PyTorch]] [[https://tigress-web.princeton.edu/~jdh4/Deep-Learning-with-PyTorch.pdf|Excerpts Online]]
+     * **[[https://www.amazon.com/Deep-Learning-PyTorch-Eli-Stevens/dp/1617295264/ref=sr_1_1_sspa?dchild=1&keywords=pytorch&qid=1613431235&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExOUFKSTZGN0VHVVI0JmVuY3J5cHRlZElkPUEwMzIwODQ5SE9MRFEyRVBIWllFJmVuY3J5cHRlZEFkSWQ9QTA4NjkwMTBKNFY2UUg4N0pNQkwmd2lkZ2V0TmFtZT1zcF9hdGYmYWN0aW9uPWNsaWNrUmVkaXJlY3QmZG9Ob3RMb2dDbGljaz10cnVl|Deep Learning with PyTorch]]** [[https://tigress-web.princeton.edu/~jdh4/Deep-Learning-with-PyTorch.pdf|Excerpts Online]] - Best book for learning PyTorch. Starts from the basics
-  * Profiling
+  * Performance
-    * [[https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/|Can et al 2020 - NVidia - Profiling and Optimizing Deep Neural Networks with DLProf and PyProf]]
+    * Profiling
+      * [[https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/|Can et al 2020 - NVidia - Profiling and Optimizing Deep Neural Networks with DLProf and PyProf]]
+      * Also [[https://github.com/plasma-umass/scalene|Scalene]] - High-performance CPU + GPU runtime and memory profiler for Python. Great for profiling PyTorch. Nilay has been using it. [[https://www.youtube.com/watch?v=5iEf-_7mM1k|video]]
+    * Issues
+      * [[https://github.com/pytorch/pytorch/issues/41162|Advanced indexing gradient is extremely slow when there are many duplicate indices]] (Has a workaround using Tensor.index_select)
+===== Related Pages =====
+  * [[Hugging Face]]
+  * [[Software]]
+  * [[Tensorflow]]