ml:mixture_of_expert_models

This is an old revision of the document!

Table of Contents

Mixture of Expert (MoE) Models

Mixture of Expert (MoE) Models

Mixture of expert (MoE) models, focusing on sparse MoE models.

Overviews

Fedus et al 2022 - A Review of Sparse Expert Models in Deep Learning

MoE Large Language Models

Fedus et al 2021 - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Dai et al 2024 - DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

People

Barret Zoph

Related Pages

Language Model

ml/mixture_of_expert_models.1741900711.txt.gz · Last modified: 2025/03/13 21:18 by jmflanig