Mixture of Experts (MoE): Have each token activate only a few feed-forward networks

The direct problem that Mixture of Experts (MoE) aims to solve is the tension between computational cost and model capacity of the feedforward network (FFN) in the Transformer. In a dense Transformer, every token fully passes through the same FFN: the number of tokens multiplied by the number of FFN parameters is the amount of computation that must be performed in every forward pass. To make the model more capable, it is common to widen the FFN, but widening the FFN slows down every token at every layer. The entry point of MoE is exactly to break the default assumption that 'every token invokes the full FFN capacity'.

The structural replacement of MoE: replace the single FFN that every token must pass through with multiple parallel, selectable FFNs, each called an 'expert'. In each layer, a router reads the current token's contextual representation (that is, the hidden vector when the token arrives at this layer), scores each expert, and selects only the top-scoring one or two experts to actually perform computation—this is top-1 or top-2 selection. In a model with 8 experts, the token 'refund' may pass through only 2 of them; the other 6 experts are not computed at all this time. The unselected experts do not participate this time, but attention, word embeddings, layer normalization, and all other shared layers are still densely executed for every token—savings occur only in the feed-forward expert part.

From the perspective of inputs and outputs, the interface does not change. The input is still the contextual representation of each token; the output is still an updated representation with the same dimensions as the original FFN. That is, MoE is an implementation replacement inside the FFN: the tensor shapes passed between layers, residual connections, and normalization positions do not need to change because of it. This is exactly why it can be treated as 'directly replacing the FFN'.

The key to understanding the cost and benefit of MoE is to distinguish three numbers that are often conflated. Total parameters is the sum of all parameters stored in the model; it determines the overhead of storage, loading, and training state, and also determines the upper bound of capacity available for conditional selection. Active parameters per token is the sum of the shared-layer parameters and expert parameters that the token actually passes through; it is closer to the real computation amount for a single token, but it still does not include routing scoring and communication overhead. FLOPs is the actual number of floating-point operations performed, which best reflects the real arithmetic workload. A model can have an extremely large total parameter count while each token activates only a very small portion of them.

From this, three causal judgments follow. First, a large total parameter count only means there is more capacity for conditional selection; it does not mean each token invokes all capabilities: having 8 experts coexist and 'each token computes all 8 experts' are two completely different things. Second, a large total parameter count does not guarantee faster wall-clock time: more parameters mean greater burdens for storage, loading, and training state, and beyond active parameters there are routing and communication overheads; the real speed is determined jointly by FLOPs and the system implementation. Third, the essence of MoE is an exchange—using greater parameter capacity and higher communication complexity in exchange for conditional computation: only some experts work for a token. It does not turn one trillion parameters 'for free' into dense one-trillion capability; a dense model calls on 100% of its capability for every token, while MoE calls on only a small portion of it for each token. The trade-off between per-token capability and total capacity is completely different. Evaluating a MoE model must look at the three metrics—total parameters, active parameters, and FLOPs—at the same time; any one alone will be misleading.

The router's work can be broken down into four steps: scoring, selection, renormalization, and weighted combination, each with clearly defined inputs and outputs.

First, scoring. Let x be the context vector of the current token arriving at this layer, and Wᵣ the router's weight matrix. The router first applies a linear transformation Wᵣ·x to obtain a score for each expert; a higher score means the router considers that expert more suitable for handling this token. Second, softmax. Passing these scores through softmax gives the probability vector g over experts:

g = softmax(Wᵣ·x)

The e-th component gₑ satisfies gₑ = exp((Wᵣ·x)ₑ) / Σⱼ exp((Wᵣ·x)ⱼ). softmax compresses arbitrary real-valued scores into a probability distribution that sums to 1: each expert receives a probability between 0 and 1, and the probabilities of all experts add up to exactly 1. gₑ is the router's relative confidence that "this token should be handled by expert e". Third, top-k hard selection. k is the number of experts activated per token, and S is the set of indices of the k experts with the highest probabilities; experts outside the set do not participate in this computation. Fourth, renormalize and combine. The sum of the original probabilities of the selected k experts is usually less than 1, so renormalize within the set S: ĝₑ = gₑ / Σ_{e'∈S} g_{e'}, so that the weights of the selected experts sum to 1 again. The final output y is the weighted sum of the selected experts' respective FFN outputs:

y = Σ_{e∈S} ĝₑ · Eₑ(x)

Here Eₑ(x) is the output of expert e's feed-forward network for this token, and ĝₑ is the renormalized weight. Thus the updated representation of a token is a convex combination of the outputs of two (or one) experts, rather than a simple concatenation or average.

The choice between top-1 and top-2 is a trade-off between fault tolerance and cost. top-1 activates only one expert per token, minimizing computation and communication; but the entire path is staked on a single expert, and if that expert performs poorly on the current input, there is no second path to compensate, making the single path more fragile. top-2 activates two experts simultaneously, introducing redundancy and stronger expressive combination ability: when one expert fails, the other can still contribute output. The cost is that the computational cost of the expert branches approximately doubles, and communication rises correspondingly. This "approximately doubles" comes from the hard selection itself: every selected expert must execute a full FFN.

Hard top-k also has a key consequence for training. The selection is discrete: an expert is either selected or not selected, and experts that are not selected receive no task gradient for the current token at all this time. Relying only on this hard selection makes the router difficult to train effectively—because the selection action itself is non-differentiable, and many experts may go without gradients for a long time. Therefore training also requires two things: a differentiable routing score (the softmax probability itself is differentiable everywhere, allowing gradients to flow back to the router via the weights g), and a balancing objective (encouraging the selection frequency of experts to be roughly balanced, so that a few experts do not monopolize). The former solves "how to learn", and the latter solves "learn uniformly".

Different implementations also add structural components. Some architectures have a shared expert in addition to routed experts: every token passes through it unconditionally, and routed experts only add on top of it. In this way, common patterns always have a shared path as a fallback, and routing is only responsible for selecting incremental capabilities. These details vary by implementation; whether a shared expert exists, what k is, and the specific way of renormalization may all differ, and one cannot assume the internal structure just from the "MoE" label.

Use a concrete mini-batch to calculate the routing load clearly: 4 experts E1 to E4, a batch of 6 tokens, each token uses top-2 routing, meaning each token simultaneously sends computation requests to two experts.

First calculate the total volume. Each of the 6 tokens produces 2 expert requests, so the batch has a total of 6×2 = 12 routing requests. Averaged over 4 experts, each expert should receive 12/4 = 3 requests. This is what capacity factor c = 1.0 means: with no safety margin, each expert's capacity is exactly equal to the average load. The capacity formula generalizes this process:

C = ceil(c × N × k / E)

Where C is the maximum number of routing requests a single expert can receive in this batch, ceil means round up (ceiling), c is the capacity factor, N is the number of tokens, k is the number of experts selected per token, and E is the total number of experts. Substituting into this example: ceil(1.0 × 6 × 2 / 4) = ceil(3) = 3. Rounding up ensures that capacity is at least not less than the average load, so it will not be pushed below the average because of indivisibility.

But the average is not the actual value for each expert. The router chooses freely based on token content, so the distribution will not be uniform. In the diagram, E1 receives 5 routing requests, but its capacity is only 3: 2 of the 5 routes must be handled as overflow. This reveals the essence of the capacity mechanism—capacity is allocated by multiplying the average number of routes by a safety factor, so hot experts may still overflow; capacity only limits 'how many a single expert can accept at most' and does not smooth out the load.

Faced with 2 overflowed routes from E1, there are three common responses, each trading off quality and resources. The first is drop: directly discard these 2 requests, so the corresponding token's expert output misses this route. The loss is that it may hurt the token's representation—the discarded one is exactly one of the experts with the highest router scores; the benefit is that resources are fixed, and the total computation and communication for this batch will not expand. The second is second/backup re-routing: transfer the overflowed requests to other experts that still have spare capacity. The token's representation can be fully restored, but the one taking over is an expert the router originally did not choose, so the semantics will change—the expert receiving the re-routed request may not be suitable for this token; at the same time, an additional scheduling step is introduced, and communication volume increases. The third is to increase the capacity factor, for example CF=2: capacity becomes ceil(2 × 6 × 2 / 4) = 6, each expert can accept up to 6 routes, in this example all 5 of E1's routes fit, and dropping almost never occurs. The cost is equally clear: buffers are reserved according to capacity, and worst-case computation and communication budgets both rise with the capacity limit, even though the actual average load is still 3.

So the capacity factor is an explicit engineering knob: turn it up, and overflow and dropping decrease and quality is more stable, but the reserved buffer and worst-case cost increase; turn it down, and resources are saved, but hot experts will overflow more, shifting the cost to the completeness of token representations or the semantic shift caused by re-routing. This trade-off appears in both training and inference, but the constraints on each side are different.

Routing collapse is a self-reinforcing positive feedback process. Suppose that early in training, expert E1, because of random initialization or data order, is slightly stronger than the other experts. The router assigns more tokens to E1; E1 therefore receives more task gradients, gets more training signal, and becomes stronger; the stronger E1 then makes the router favor it even more. Once this loop starts, the other experts receive almost no samples and gradients, their training opportunities approach zero, and they become increasingly “starved”—this is expert starvation. The model nominally has E experts, but only a few actually perform computation, and the remaining parameters merely take up storage and loading cost.

To understand why this loop occurs naturally, you need to see its inputs and outputs clearly. The router’s input is the probability it assigns to each expert, and its output is the actual assignment; the optimization objective has two parts that pull against each other: both high task quality (assigning tokens to the most suitable expert) and balanced expert load (giving every expert a chance to learn). When training with only the task loss, the second objective has no force to constrain it, so the positive feedback proceeds unhindered.

The methods for counteracting collapse target different stages. Auxiliary loss directly punishes the situation where “a few experts have excessively high probabilities or an excessively high actual token share,” turning the balancing objective into part of the gradient and acting on the learning process of the router and experts. Routing noise adds random perturbations to routing scores during training, forcing the router to occasionally explore experts it would not otherwise select and creating opportunities for starved experts to be tried. z-loss constrains routing logits from growing without bound: larger logits mean a sharper softmax, with the router becoming increasingly confident in a few experts, while large logits also bring numerical instability; z-loss is like stepping on the brakes for this divergence. Expert capacity limits the maximum routing load a single expert can receive, providing a backstop at the allocation level: even if the router wants to dump tokens onto E1 all at once, capacity will block the overflow or redirect it to other experts.

But none of these methods can be strengthened without limit. When balancing is too strong, tokens that should go to specialized experts are forcibly spread across unrelated experts, semantic routing is disrupted, and task quality suffers. The right position is not to pursue absolute uniformity, but to observe multiple signals at the same time: task loss, expert entropy (the dispersion of the assignment probability distribution), per-expert token count, overflow rate, and overall quality. Load uniformity itself is only an efficiency signal for computational allocation, indicating that compute is not being wasted; it cannot prove that tokens from different languages or different groups receive equal representation quality—a model with balanced expert load may still be highly uneven in quality. Treating “uniform allocation” as “quality fairness” is one of the most common misreadings of MoE.

When experts are distributed across multiple GPUs, a token's journey includes two additional communication segments. The input token first completes attention and routing scoring on the local device, then is packed according to destination expert: all tokens going to E1 are grouped together, those going to E2 into another group, and so on. Next, an all-to-all exchange is performed—each GPU sends the packed tokens to the devices holding the corresponding experts, while receiving tokens from other GPUs that need to be processed by local experts. After the device holding the expert completes the FFN computation, it performs a reverse all-to-all to send each token's result back to its originating GPU in the original order. Thus each MoE layer contains two all-to-all communication phases; communication is no longer a hidden detail but a first-class cost alongside expert computation.

This pattern is fast under ideal conditions, but under real-world conditions it can easily make the network neither idle nor fully saturated, creating multiple bottlenecks.

Expert hotspots are the first. A few experts are selected by a large number of tokens; the GPUs holding them accumulate a backlog of requests, while other GPUs finish early and can only wait, or trigger capacity overflow. Mitigation measures are load balancing, and replicating particularly popular experts so that copies of the same expert are spread across multiple devices to share the traffic.

Cross-node bandwidth is the second. The total data volume of all-to-all is proportional to the batch and expert distribution; when experts are distributed across machines, the share of network exchange in per-step time rises sharply, and GPU compute ends up waiting for data. Topology-aware placement and grouped routing can reduce cross-node communication: place devices that frequently exchange messages within the same node, or restrict routing scope to within a group.

Small messages are the third. Communication latency is composed of waiting time and transmission time; the smaller the message and the shorter the transmission time, the higher the proportion of waiting time, and latency dominates the entire communication. Increasing the token batch size and fusing multiple small messages into larger transmission units can make the network truly busy.

Long-tail sequences are the fourth. In dynamic batching, sequence lengths are uneven; short sequences finish early, long sequences drag their tails, and the workload across devices fluctuates, leading to uneven load between batches. This requires coordination of capacity policies and scheduling policies, rather than only adjusting the load-balancing loss.

These bottlenecks together point to one conclusion: a decrease in per-token FLOPs does not guarantee a decrease in wall-clock latency. The computation saved arithmetically by MoE may be eaten up by communication latency and waiting; it depends more on high-speed interconnect and sufficient batch size than dense models. Judging MoE's “speed” using FLOPs ignores communication as this first-class cost.

Load balancing is done well during training, but at inference time a particular expert may still be instantly saturated, because the traffic structure on the two sides is fundamentally different. Training corpora are large batches of data mixing languages and domains; the router sees a thoroughly mixed input, so hotspots are naturally diluted, and the balancing objective also operates on this distribution. Production traffic is not like this: at certain times users suddenly focus on the same topic (for example, a breaking news event), large numbers of semantically similar requests surge to the same expert, batches are small and generation is token by token, so the routing distribution is much sharper than during training. The “evenness” measured during training is evenness on the training distribution and cannot automatically transfer to the inference distribution.

Therefore inference needs its own monitoring dimensions rather than trusting training metrics: actual load per layer and per expert, routing entropy (whether allocation is overly concentrated), overflow rate, all-to-all latency, and batch and slice quality. Stress testing for possible peak topics is more valuable than discovering afterwards that some GPU is queuing. These monitoring targets are system behavior, not model quality itself.

Parallel strategies also need to leave combination space for inference spikes. Expert parallel places different experts on different devices; combined with tensor parallel or data parallel, it can simultaneously utilize intra-device and inter-device compute; but the more complex the combination, the more complex the network topology design and fault recovery—when a node fails, its expert replicas or recomputation paths must be able to take over.

A final boundary is easily overlooked: sparse computation does not equal sparse storage. Inactive experts are not computed for that pass, but their weights usually still need to reside in GPU memory, or in tiered storage that can be accessed quickly. Total parameter count therefore directly affects GPU memory usage and startup time: a MoE model with a trillion total parameters, even if each token activates only ten billion parameters, still needs to find a place for all weights at startup. Sparsity saves per-token arithmetic, not storage.

If an expert frequently handles code-related tokens, can we call it a “code expert”? Mere statistical frequency is far from sufficient to support that label.

First, the basis for routing specialization is not necessarily semantic. Experts may specialize along surface features such as word frequency, position, punctuation, or language, or may simply be artifacts of a load-balancing objective—the auxiliary loss spreads tokens across experts, and some experts are merely “assigned” code samples. Using a few high-frequency token examples to tell stories makes it especially easy to anthropomorphize random specialization.

Mutual information here measures: given that a token belongs to a certain slice (such as a particular language or topic), to what extent can we predict which expert it will be routed to? It can reveal stable statistical associations, such as “German tokens go to E3 more often,” but association does not equal causation. Routing may follow some feature that co-occurs with German, rather than because the expert “understands German.”

To test the claim that “an expert performs a certain function,” causal intervention is needed rather than observational statistics. The validation input is routing logs and capability slices, and the output includes not only association statistics but also capability changes after swapping or masking experts. If removing or swapping an expert causes a selective decline in code-related capability, while other confounders (such as reduced total parameters or other capabilities being impaired simultaneously) are controlled, then there is stronger evidence that the expert is related to code functionality. Moreover, experts with the same index may mean different things in different layers: E2 in layer 3 and E2 in layer 21 are two different parameter blocks, and their specializations can be completely unrelated.

A deeper limitation comes from the MoE architecture itself. Experts are not independent modules: the shared attention layers first mix the context, and by the time a token reaches an expert it already carries information from the entire sentence or even the whole passage; a behavior arises from the combined effect of routing and shared layers across multiple layers. Therefore, one cannot infer from the label of an expert in a single layer why the entire answer was generated, nor treat expert boundaries as permission boundaries—for example, the claim that “the content expert was not activated, so the model cannot output incorrect facts” does not hold. Routing logs record “who was used,” but they cannot prove “that it was used for a particular semantic reason.”

The word “routing” is used in MoE, application model routing, and multi-agent systems, but the decision granularity and goals of the three are completely different. Confusing them with one another makes problem localization difficult.

MoE's unit of selection is each layer and each token. In each layer, a router chooses for the current token among multiple FFN experts inside the model; the candidates are parameter blocks within the same model. The main goal is parameter capacity and sparse computation: use larger total capacity to carry more capability while having a single token activate only part of the parameters. This routing is part of the model parameters, learned through training, and is opaque to users.

Application model routing's unit of selection is each request or a stage within a request. Candidates are complete large models or different services—for example, simple requests go to a small model, complex requests are escalated to a large model, or requests are sent to different model endpoints based on modality or function. The main goal is to control cost under quality constraints: for the same request volume, use a small model to handle most traffic and save call overhead. This routing is explicitly controlled by product rules; logs are observable and rules can be modified. It is fundamentally different from MoE's internally learned parameter routing.

Multi-agent systems' unit of selection is the subtask. After a task is broken down, a scheduler assigns subtasks to different roles, tools, or agents in contextual environments. The main goals are division of labor, parallelism, and isolation: different agents perform their own roles, tasks can proceed concurrently, and errors and permissions are also isolated in their respective contexts.

The three can be used together: an application can do model routing at the outer layer to select a service, the selected model can then use MoE routing internally for sparse computation, and after the request enters the model, it can also be orchestrated by multi-agent systems. Therefore, when evaluating, it is necessary to distinguish which layer an error or delay occurs in: whether product rules selected the wrong model, a MoE expert was poorly assigned, or the division of labor among agents went wrong. If the wrong layer is identified, remediation measures will be applied in the wrong place.

With total parameters increased 8-fold and activated FLOPs roughly unchanged, whether such scaling is worthwhile cannot be answered by parameter numbers alone; it requires a controlled comparison. The input to the evaluation is a pair of models: an MoE model and a dense baseline with the same activated FLOPs, both run on the same task, same hardware, same network, and same batch. Only by aligning activated compute is the comparison fair—otherwise it is impossible to tell whether the advantage comes from the sparsity mechanism or simply from doing more computation.

The quality side and the system side each have a set of metrics. The quality side looks at validation loss, task-specific performance, per-language slices, safety behavior, calibration, and stability. The system side looks at tokens/s throughput, TTFT (time to first token), TPOT (time per output token), GPU memory usage, all-to-all communication share, overflow rate, power consumption, and failure recovery time. The benefits of MoE are often eaten up by communication on the system side; scoring only on the quality side would misjudge a model that is “fast in computation but slow in wall-clock time” as a success.

Load balancing requires a clear metric. Expert load CV is the standard deviation of the loads across experts divided by the mean: CV = 0 means perfectly uniform, and larger values indicate more severe relative skew. It is a tool for measuring the uniformity of compute allocation, but a low CV does not mean semantic routing is reasonable, nor does it mean quality is better across languages or groups—uniform load is only uniform allocation. Testing functional robustness requires harder interventions: perform expert ablation (removing experts one by one or in groups) and routing perturbation, and observe whether overall capability collapses when a few experts fail, rather than being smoothly absorbed by redundancy; at the same time, separately compare the hot spots under the training distribution and the production distribution to see whether the skew during inference is within the expected range.

The final report must list total parameter count and activated parameter count, top-k, capacity factor, and parallel topology; if any of these is missing, the parameter numbers may mislead readers. The criterion for judging “worthwhile” is: under the target network and batch conditions, the quality–cost frontier improves reproducibly, and this throughput improvement is not obtained by dropping tokens or by degrading individual slices. If either condition holds, it indicates that the gains from scaling have been underestimated or fabricated.

Link the previous steps into a complete causal chain—from “a token enters” to “an expansion is accepted,” every causal step in between for MoE is clear.

The starting point is the token's contextual representation entering the router. After the attention layer mixes context, this vector carries all the information about the token in the current context; the router uses it to assign a score to each expert, which then becomes a probability distribution through softmax. The router's output is not a label but a set of differentiable weights—a design that simultaneously determines selection (who is selected) and trainability (gradients can flow back to the router through these weights).

The second step is top-k selection and capacity allocation. The router selects the k experts with the highest probability; the selected experts participate in the combination according to the renormalized weights within the set. At the same time, each expert's capacity limit is predetermined by capacity factor × average load, and requests to hot experts that exceed capacity go into overflow handling. This step brings together two forces—“which experts the model wants to compute” and “how much the system allows to compute”—the former pursues semantic matching, while the latter limits single-point overload.

The third step is all-to-all sending. Tokens are packed by destination expert and sent across device boundaries to the GPUs that hold the experts. Here communication becomes an explicit cost for the first time: routing skew, cross-node links, small messages, and long-tail sequences can all make this step expensive. Fourth, experts compute in parallel and return weighted results. The devices holding the experts execute the FFN, combine each expert's output into the final updated representation according to the renormalized weights, and then send it back via reverse all-to-all to the position where the original token departed, restoring the original order for the next layer.

Fifth, balancing and overflow control act throughout the entire process. Auxiliary loss and z-loss prevent routing collapse and suppress logit divergence during training; capacity and rerouting strategies backstop hot experts at runtime; and the inference side must monitor per-layer expert load, routing entropy, and overflow rate according to the production distribution—because the hotspot distribution of inference traffic may be completely different from training. Sixth, joint acceptance. Judging whether an MoE expansion is worthwhile requires looking at the joint performance of four dimensions under the same controlled conditions: quality, communication, capacity, and slicing. On the quality side, align against a dense baseline matched on active FLOPs; on the system side, measure throughput, TTFT, TPOT, and all-to-all proportion; on the capacity side, check total parameters, active parameters, and expert load CV; on the slicing side, confirm that no language or group trades degradation for overall numbers. If any link is out of balance, it leaves a specific position on the chain: for routing collapse, look at the gradients and balancing before the fourth step; for communication bottlenecks, look at the third step; for overflow, look at the capacity setting of the second step; for quality degradation, look at the routing semantics of the first step. The chain is a closed loop—the problems exposed by acceptance are exactly the causal evidence pointing to the corresponding links in the chain.