Dynamic NNs and Conditional Computation

Dynamic neural networks use methods such as conditional computation, adaptive computation, dynamic model sparsification, early-exit approaches, etc to build larger models with less compute requirements.

• Han et al 2021 - Dynamic Neural Networks: A Survey

• Fedus et al 2021 - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity Uses conditional computation to scale-up the Transformer. Good overview of conditional computation in the introduction.
• Clark et al 2022 - Unified Scaling Laws for Routed Language Models

• DyNN Workshop 2022

• Mixture of Experts