====== PyTorch ======
  * Tutorials
     * [[https://pytorch.org/tutorials/]]
     * [[https://pytorch.org/tutorials/beginner/nn_tutorial.html]]
     * [[https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html]] (DataLoader)
     * [[https://pytorch.org/tutorials/beginner/data_loading_tutorial.html]]
     * [[https://pytorch.org/tutorials/recipes/recipes/defining_a_neural_network.html]]
     * [[https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html]] Good seq2seq tutorial and starter code (doesn't have batching)
     * [[https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb|pytorch-seq2seq example code]] Good seq2seq code with batching 
     * [[https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec|How to code The Transformer in Pytorch]]
     * BERT classification tutorials
       * [[https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb|Fine-tuning a BERT model on a text classification]]. Doesn't show how to make predictions on new data.  For that see [[https://huggingface.co/docs/transformers/training|Fine-tuning a pretrained model]]
       * [[https://huggingface.co/docs/transformers/training|Fine-tuning a pretrained model]]
  * Frameworks
    * [[https://www.pytorchlightning.ai/|PyTorch Lightning]] A higher-level API for PyTorch that removes a bunch of boilerplate code
  * Models
    * Transformer
      * [[https://github.com/SamLynnEvans/Transformer]] Nice implementation (Rongwen used it) with a tutorial: [[https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec|How to code The Transformer in Pytorch]]
  * Books
     * **[[https://www.amazon.com/Deep-Learning-PyTorch-Eli-Stevens/dp/1617295264/ref=sr_1_1_sspa?dchild=1&keywords=pytorch&qid=1613431235&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExOUFKSTZGN0VHVVI0JmVuY3J5cHRlZElkPUEwMzIwODQ5SE9MRFEyRVBIWllFJmVuY3J5cHRlZEFkSWQ9QTA4NjkwMTBKNFY2UUg4N0pNQkwmd2lkZ2V0TmFtZT1zcF9hdGYmYWN0aW9uPWNsaWNrUmVkaXJlY3QmZG9Ob3RMb2dDbGljaz10cnVl|Deep Learning with PyTorch]]** [[https://tigress-web.princeton.edu/~jdh4/Deep-Learning-with-PyTorch.pdf|Excerpts Online]] - Best book for learning PyTorch. Starts from the basics
  * Performance
    * Profiling
      * [[https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/|Can et al 2020 - NVidia - Profiling and Optimizing Deep Neural Networks with DLProf and PyProf]]
      * Also [[https://github.com/plasma-umass/scalene|Scalene]] - High-performance CPU + GPU runtime and memory profiler for Python. Great for profiling PyTorch. Nilay has been using it. [[https://www.youtube.com/watch?v=5iEf-_7mM1k|video]]
    * Issues
      * [[https://github.com/pytorch/pytorch/issues/41162|Advanced indexing gradient is extremely slow when there are many duplicate indices]] (Has a workaround using Tensor.index_select)

===== Related Pages =====
  * [[Hugging Face]]
  * [[Software]]
  * [[Tensorflow]]