ml:mixture_of_expert_models

This is an old revision of the document!

Table of Contents

Mixture of Expert (MoE) Models

Mixture of Expert (MoE) Models

Mixture of expert (MoE) models, focusing on sparse MoE models.

Overviews

Fedus et al 2022 - A Review of Sparse Expert Models in Deep Learning

Foundational and Early Papers

Shazeer et al 2017 - Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

MoE Large Language Models

Fedus et al 2021 - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Dai et al 2024 - DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Lo et al 2024 - A Closer Look into Mixture-of-Experts in Large Language Models

People

Barret Zoph

Related Pages

Language Model

ml/mixture_of_expert_models.1741900832.txt.gz · Last modified: 2025/03/13 21:20 by jmflanig