Mixture-of-Experts

Definition

A mixture-of-experts (MoE) model has multiple expert subnetworks and a router that determines how their outputs contribute to each input or token. In sparse MoE, the router selects only some experts. Other MoE designs can use all of them.

In many language models, MoE layers replace some dense feed-forward layers inside a transformer.

Simple example

Consider a sparsely routed MoE layer with 32 expert feed-forward networks. For each token representation, the router scores the experts and selects the top two. The layer combines those experts’ outputs using the router’s weights before processing continues.

A different token in the same request may use another pair. The routing rule is learned during training, not configured as a table of human-defined topics.

Why it matters

Sparse MoE lets a model have more expert parameters while computing with only some of them for each token. Compared with a dense model of similar total size, this can reduce per-token computation and make training a larger model practical.

For serving, sparse routing complicates expert placement and load balancing. If experts are distributed across devices, routing also requires communication between them. Skewed routing can create expert hotspots.

One important nuance

In a sparsely routed MoE, a large total parameter count does not mean every expert parameter is used for each token. That does not guarantee lower latency than a smaller dense model. Routing, cross-device communication, uneven expert use, and memory requirements can offset the computation saved. Benchmark the deployed architecture on the actual workload instead of comparing only published parameter counts.