HOPE Expert Pruning: Keep MoE Models Smaller and Stronger
Learn how HOPE uses expert interactions to prune MoE models, where it outperforms first-order methods, and what to consider before deploying it.
HOPE: A Smarter Way to Prune Mixture-of-Experts Models
Mixture-of-Experts (MoE) models can scale to very large parameter counts because they don't activate every parameter for every token. That helps with compute - but it doesn't remove a real deployment issue: the experts still have to be sitting in memory. In large MoE models, the experts' weights can take up most of the checkpoint's memory footprint, even if only a small fraction gets used per token.
One practical response is to prune experts permanently. The tricky part is deciding which experts to remove. Many methods rank experts independently, but MoE behavior is often more social than that: experts tend to work in groups. Remove one "low-value" expert and you can break a combination that matters. HOPE - Higher-Order Pruning of Experts - tries to account for this by measuring both individual contributions and pairwise interactions between experts.
This article walks through the memory trade-off behind MoE pruning, how HOPE makes its choices, what the reported results suggest, and what you should think about if you want to try the method. The key question isn't whether pruning can reduce a model. It's whether the memory you save is worth the capability you give up - and whether your pruning method matches how the model actually uses its experts.
Why MoE models still have a memory problem
An MoE model includes multiple expert networks inside a layer, plus a router that selects a subset of experts for each token. So if a layer has 256 experts and activates eight for a given token, only those eight contribute to that token's computation. The rest are idle for that step - but their parameters still need to be available to the inference system.
That distinction matters in production. Sparse activation can lower the compute you pay per token, but it doesn't automatically shrink the memory footprint. Depending on the model and serving setup, the weights may have to fit across GPUs or be split across devices. That affects hardware cost, capacity planning, and how much room you have left for things like longer context windows or higher concurrency.
The source paper uses Qwen3.5-122B as an example: 48 MoE layers, each with 256 experts. Its expert MLPs make up roughly 95% of the model's ~250 GB parameter memory footprint. Only eight experts per layer are selected for any given token, so the sparse compute pattern doesn't mean most stored parameters are "unused." They may matter for other tokens.
What pruning changes
Expert pruning permanently removes selected experts from the model. Unlike dynamic loading and offloading, it doesn't depend on a serving system moving experts in and out of memory as routing changes. It also doesn't require retraining as part of the pruning procedure described here, and it can be combined with quantization.
That's why pruning is appealing when memory is the constraint. A smaller checkpoint can reduce storage and inference memory requirements, which may make it feasible to run the model on fewer or smaller accelerators. But there's a straightforward trade-off: once an expert is gone, it can't help on inputs where it would have mattered.
So the practical goal isn't "remove as many experts as possible." It's finding a reasonable spot on a trade-off curve:
- Memory saved: how much smaller the model becomes.
- Quality retained: how performance changes on the tasks your users care about.
- Operational simplicity: whether the pruned checkpoint works with your existing serving stack and deployment workflow.
A benchmark result is only one input to that decision. Your calibration data and evaluation set should match the workload you plan to serve. If you're building coding agents, for example, you'll want multi-step coding tests - not just short, single-turn prompts.
Why independent expert scores can miss useful groups
Many expert-pruning approaches assign each expert an individual importance score, then remove the lowest-ranked ones. That's a first-order view. It assumes an expert's value can be judged on its own, independent of what other experts are around it in the layer.
That can work when pruning is modest - but it ignores a key property of MoE models: experts are often co-activated. The router may repeatedly select certain experts together, creating stable cooperation patterns. If two experts jointly affect a token's output, their combined effect can matter even when one of them doesn't look that important on its own.
The cost of breaking a cooperative group
Take two experts, A and B. A first-order method might decide A is relatively unimportant and remove it. That can look defensible when A's score is computed in isolation. But if B frequently works alongside A, removing A can damage a useful pair. The router will swap in some replacement expert, but there's no guarantee it will play the same role.
This is an interaction problem, not just a ranking problem. The same issue shows up when you prune multiple experts: individually low-scoring experts can still be part of a pattern that's worth keeping, while some higher-scoring experts can be partially redundant with others you leave in.
As you prune more aggressively, the risk grows. At lower pruning rates, you may still retain enough "members" of each useful cluster that first-order rankings do most of what you need. At higher rates - say removing 40% or 50% - you're more likely to break cooperative links. Small ranking mistakes can then compound into bigger quality losses.
None of this means individual scores are useless. HOPE's analysis says its interaction matrix contains individual importance on the diagonal, and those diagonal values closely match the rankings from REAP. The difference is that HOPE also uses the off-diagonal entries, which capture pairwise interactions.
Why calibration data matters
Pruning methods estimate importance using a calibration dataset. That dataset is a sample of the inputs the model sees, so it influences which experts appear useful and which combinations appear frequently. If your calibration set doesn't match your real production workload, the prune set may optimize for the wrong thing.
For a general-purpose assistant, relying only on code-heavy examples can be a mismatch. For a coding agent, calibration from coding trajectories may better reflect tool use and multi-step reasoning. The right choice depends on where you'll deploy - and you should validate it rather than assume it. A good calibration plan uses representative prompts and is followed by evaluation across the workload's main task types.
How HOPE chooses which experts to remove
HOPE reframes pruning as a set-selection problem. For a token, if you remove an expert that the router would have selected, the active set changes: another expert (with a lower router score) takes its place. That replacement changes the layer output. The gap between the original output and the pruned output is the pruning error.
Minimizing this error over all possible prune sets is hard because the replacement experts depend on what you removed in the first place. HOPE instead uses a surrogate objective that upper-bounds the dominant component of the error. When you expand that objective, it includes cross-terms between pairs of experts - terms that independent rankings don't capture.
The interaction matrix
HOPE collects an F-matrix in a calibration pass. Each entry summarizes the average gate-weighted output product for a pair of experts across calibration tokens where they are co-activated. Concretely:
- Diagonal entries reflect individual expert contribution and importance.
- Off-diagonal entries reflect pairwise contribution when experts cooperate.
The matrix can show blocks of experts that interact more strongly. In the reported GLM-4.5-Air analysis, the authors show representative layers where higher-interaction values cluster into blocks. The structure varies by layer, which is a useful reminder that cooperation patterns aren't identical everywhere in a model.
HOPE then formulates pruning as a quadratic program. It optimizes over the interaction matrix for a predefined pruning budget using a continuous relaxation, and then rounds the result into a binary prune set. The decision isn't just "remove the lowest-scoring experts." It weighs individual contributions against the cost of breaking pairwise relationships among experts you keep.
This is the method's core trade-off: HOPE might keep an expert with a middling individual score if it strongly cooperates with other retained experts. Or it might remove an individually strong expert if, overall, a different retained combination yields a better result.
Computational cost
Calibration requires a forward pass over the calibration data and recording things like expert selections, gate weights, output norms, and co-activation statistics. The reported implementation stores those observations in an HDF5 file. After that, the optimization step solves a quadratic program for each layer. The source reports it typically takes a few seconds per layer, while calibration dominates total runtime.
So the extra selection logic isn't the bottleneck. It doesn't mean the workflow is free, though: you still need access to the model, representative calibration data, suitable hardware, and time to evaluate. Those practical costs are part of whether a smaller checkpoint is worth pursuing.
What HOPE's results mean for deployment
The reported evaluation covers Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air. They use two calibration datasets: Evol-CodeAlpaca and SWE-Bench Verified trajectories. The pruned models are evaluated on the Tulu3 suite and SWE-Bench Pro, compared against four first-order baselines: REAP, MAN, EAN, and Frequency. Pruning rates range from 10% to 50%.
Across the tested settings, HOPE reports the best average rank: 2.07 across five methods. It finishes first in 39% of conditions and is in the top two in 67%. Those aggregates suggest a consistent advantage in their evaluation, but they don't prove HOPE will win for every model, dataset, or deployment workload.
The advantage grows at higher pruning rates
HOPE's advantage is strongest at aggressive pruning levels. At 50% pruning, its average rank is 1.58 versus 2.42 for the next-best method, REAP. That aligns with the method's logic: once you remove more experts, preserving useful interactions matters more because the risk of breaking cooperative groups increases.
One clear example in the paper is Qwen3.5-35B-A3B on SWE-Bench Pro (an agentic coding benchmark). At 50% pruning, HOPE scores 42.3 versus 41.8 for the unpruned model. REAP scores 36.2 in the same setting. That looks like HOPE preserved a useful structure for that workload, while the first-order baseline regressed more.
Still, if a pruned model matches or even exceeds an unpruned benchmark score, it doesn't automatically mean pruning "improves the model." Sometimes removed experts are just less relevant to that evaluation workload, and the score differences can reflect calibration choice, benchmark quirks, or normal variance. The operational question is whether the pruned model stays reliable on the inputs your users actually send.
A practical evaluation plan
Before deployment, compare the original and pruned checkpoints on the same fixed evaluation set. Include the tasks you used for calibration and important tasks you didn't use. And for production, don't rely on only one aggregate number:
- Task success: can the model complete the intended work, including multi-step tasks?
- Failure modes: do errors become more frequent or more severe after pruning?
- Serving behavior: does the smaller checkpoint actually reduce memory use in your inference setup?
- Workload coverage: is performance stable across prompt types, tool calls, and context lengths?
- Recovery cost: if the pruned model underperforms, can you roll back quickly?
The business case is strongest when memory savings change what you can run: fewer GPUs, higher concurrency, or more memory for longer contexts. If the smaller checkpoint doesn't materially improve cost or capacity, then even a small quality drop may not be worth it.
How to run HOPE and where it falls short
The workflow has three stages: calibrate, solve, and prune. The package is reported as tested on Qwen3, Qwen3-Next, Qwen3.5, and GLM-4.5. A basic command-line flow looks like this:
hope calibrate \
--model-path Qwen/Qwen3.5-35B-A3B \
--prompts calibration_prompts.txt \
--out-path observations.h5
hope solve \
--obs-path observations.h5 \
--prune-frac 0.50 \
--out-path pruneset.json
hope prune \
--model-path Qwen/Qwen3.5-35B-A3B \
--pruneset-path pruneset.json \
--out-path ./Qwen3.5-35B-A3B-pruned-50
Calibration runs a forward pass over the prompts and records observations for each MoE layer. The solve stage uses those observations to produce a prune set for the requested fraction. The prune stage writes a smaller checkpoint. Per the source, it produces a standard Hugging Face checkpoint and doesn't require changes to vLLM or SGLang model definitions.
Where the method has limits
HOPE is most compelling when you prune aggressively. At 10% to 20%, first-order methods like REAP and MAN can be competitive, so the interaction modeling may not buy you much. Whether HOPE helps depends on pruning rate, the model, and the task distribution - not on a universal rule that second-order methods always win.
There are also structural limits. The described approach prunes each layer independently and doesn't model interactions across layers. The paper outlines a theoretical cross-layer version of HOPE, but the reported results are still preliminary. HOPE also models pairwise interactions rather than triples or larger groups. Higher-order relationships might add more signal, but estimating them gets combinatorially harder.
In practice, treat HOPE as a candidate compression method you validate with measurement - not as a shortcut that replaces deployment evaluation. A sensible path is: set a baseline, pick calibration data that matches your workload, try more than one pruning rate, measure the pruned checkpoint in your serving environment, and keep a rollback option. If quality is your binding constraint, less aggressive pruning - or no pruning - may be the better engineering decision.
Conclusion
MoE models can make compute sparse, but their expert parameters still create a major memory burden. Pruning can reduce that burden without adding dynamic expert-loading complexity - but independent expert rankings can miss cooperative structure, especially when you prune heavily.
HOPE addresses that limitation by optimizing over both individual expert importance and pairwise interactions. Its reported results are strongest at aggressive pruning rates, including a notable result on agentic coding tasks. But it's not a guarantee of quality retention across every workload. The sensible approach is to choose a pruning rate based on the memory benefit you need, calibrate on representative inputs, and verify performance in the serving conditions that matter for your business.