Mixture of Experts (MoE)

Mixture of Experts (MoE) is an architecture where instead of one large neural network, the model consists of many smaller 'expert' networks. For any given input, a routing mechanism selects a small subset of these experts to process it. This allows models to scale up their parameter count dramatically while keeping computational costs manageable.