PyTorch

Tutorials
- https://pytorch.org/tutorials/
- https://pytorch.org/tutorials/beginner/nn_tutorial.html
- https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html (DataLoader)
- https://pytorch.org/tutorials/beginner/data_loading_tutorial.html
- https://pytorch.org/tutorials/recipes/recipes/defining_a_neural_network.html
- https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html Good seq2seq tutorial and starter code (doesn't have batching)
- pytorch-seq2seq example code Good seq2seq code with batching
- How to code The Transformer in Pytorch
- BERT classification tutorials
  - Fine-tuning a BERT model on a text classification. Doesn't show how to make predictions on new data. For that see Fine-tuning a pretrained model
  - Fine-tuning a pretrained model
Frameworks
- PyTorch Lightning A higher-level API for PyTorch that removes a bunch of boilerplate code
Models
- Transformer
  - https://github.com/SamLynnEvans/Transformer Nice implementation (Rongwen used it) with a tutorial: How to code The Transformer in Pytorch
Books
- Deep Learning with PyTorch Excerpts Online - Best book for learning PyTorch. Starts from the basics
Performance
- Profiling
  - Can et al 2020 - NVidia - Profiling and Optimizing Deep Neural Networks with DLProf and PyProf
  - Also Scalene - High-performance CPU + GPU runtime and memory profiler for Python. Great for profiling PyTorch. Nilay has been using it. video
- Issues
  - Advanced indexing gradient is extremely slow when there are many duplicate indices (Has a workaround using Tensor.index_select)

PyTorch

Related Pages