Dynamic neural networks use methods such as conditional computation, adaptive computation, dynamic model sparsification, early-exit approaches, etc to build larger models with less compute requirements.

• Fedus et al 2021 - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity Uses conditional computation to scale-up the Transformer. Good overview of conditional computation in the introduction.

• DyNN Workshop 2022