MoE Inference: Run 180B Models with 32GB of RAM
See how SSD-streamed MoE experts bring a 180B model to a 32GB Mac, with benchmark results, practical trade-offs, and lessons for local AI inference.

How to Run a 180B MoE Model on 32GB of Memory
Running a large language model locally usually comes down to one thing: memory. The weights have to live somewhere, and most consumer GPUs (and laptops) don't have the fast memory needed for the biggest open models. Offloading weights to system RAM can help, but it often turns inference into a waiting game - lots of data movement, not a lot of compute.
Mixture-of-experts (MoE) models give you a different angle. Even if a model has hundreds of billions of parameters, its router only activates a small set of experts for each token. If your inference engine can load just those experts when they're needed - and keep the frequently used ones close to the compute - it can make a model fit that otherwise wouldn't.
A recent Transformers experiment did exactly that: it streamed experts from an SSD on a 32GB Mac, used a bounded cache, and got around 14 tokens per second - about 6% slower than a dedicated inference engine on short decoding. This isn't a universal "every laptop can run everything" claim. It's a practical look at what expert streaming changes, where the remaining bottlenecks show up, and when local MoE inference is actually worth the engineering effort.
Why can a 180B-parameter model run on 32GB?
The trick is separating a model's total parameter count from the parameters it actually uses for one token. Dense models run the full network for every token. MoE models route each token through a selected group of specialist networks ("experts"). For any given token, most experts sit idle.
That sparsity helps with the memory problem - but it doesn't make the weights go away. Over a full conversation, you still need access to lots of experts. So the real question becomes: can you keep the experts you need available without loading every expert into GPU (or unified) memory all at once?
The router determines what needs to move
In an MoE layer, you've got a bunch of experts plus a router. For each token, the router picks which experts will process it. In the experiment, each of 48 layers had 512 experts, and 10 experts were selected per token. Since experts account for most of the weights, moving only the selected ones can cut down how much data has to travel during inference.
This is why a model with a headline size around 180B can sometimes run on consumer hardware. It's not that the machine magically has enough fast memory. You still need an efficient way to fetch the right experts - and you need to make sure the extra transfer and coordination work doesn't outweigh the compute.
Storage is not the same as working memory
In the experiment, the experts were about 121GB in FP8 before conversion. They were quantized to 3 bits, bringing the disk footprint down to about 49GB. A 12GiB cache held a working set of experts in device memory; the rest stayed on disk. Dense layers were also quantized, and there was a separate 51GB n-gram table that needed its own disk-backed handling.
None of this means the whole model "fits" in 32GB comfortably. After loading, the process used about 18GiB. A 6K-token prompt pushed it to roughly 25.4GiB, leaving very little headroom for other apps. In practice, the browser had to be closed to avoid macOS swapping.
The point is simple: disk capacity, RAM capacity, and fast device memory solve different sub-problems. Streaming helps you fit by loading data on demand - but SSD access is still slower than memory, and you still need memory for dense weights, the prompt, the key-value cache, and runtime overhead.
Why do standard offload methods struggle with MoE?
Offloading means placing some model weights outside the GPU's fast memory. It's a common approach to run bigger models, but performance depends heavily on what's moved, when it's moved, and what the compute path looks like. MoE makes this more fragile because the router decides which weights matter per token.
CPU execution avoids transfers but can hit a capacity wall
With llama.cpp, you can assign experts to the CPU. The CPU keeps the weights in system RAM and runs those experts, so the GPU doesn't need to receive their weights over PCIe for every token. This can work when the experts fit in RAM.
If they don't, the OS page cache can thrash - pulling different parts of the model from storage again and again. A GPU expert cache can help in some cases, but the cited test found that graph fragmentation hurt performance even when the weights fit. So the trade-off is what you'd expect: CPU execution dodges repeated GPU weight transfers, but it's limited by CPU compute speed and by how much memory you actually have for the working set.
Reading weights over PCIe can repeat work
vLLM can offload weights to pinned CPU memory and have GPU kernels read them over PCIe. For MoE layers, you can configure offload to cover experts rather than all parameters, which avoids repeatedly transferring dense layers that get used every token.
But in the tested setup, hot experts weren't really being kept hot. An expert could cross the bus again on the next token. That can be fine for workloads that touch lots of experts (like large batches). But for a single user generating tokens one-by-one, you often keep hitting a smaller subset of experts repeatedly - so caching becomes more valuable.
Module-level offload ignores the router
Transformers' standard Accelerate offload path moves modules across devices. For an MoE layer, the "module" might effectively be the entire expert bank. The layer might only need a few experts for the current token, but the offload system moves the larger chunk anyway - without using the router's per-token selection.
On a Mac, CPU and GPU share unified memory, so assigning weights to the CPU doesn't create extra physical memory. It just changes how memory is managed. Still, offload granularity matters: if your system treats every expert bank as one big blob, you miss the main advantage that sparse routing gives you.
A comparison on one A10G GPU (24GB VRAM, lots of system RAM) shows the gap. On Qwen3.6-35B-A3B, Strata ran at 82-88 tokens per second in Q8_0, while llama.cpp managed 63-64. The tested vLLM FP8 setup with expert-only CPU offload hit 20.1; with broader offload it dropped to 9.7. Transformers with standard Accelerate offload was far lower at 0.16 tokens per second in that configuration. These numbers are specific to that model and setup, not a universal ranking - but they do show why routing-aware movement can matter.
How does expert streaming work in Transformers?
The core design choice wasn't a brand-new expert computation kernel. It was about making a streamed cache behave like a normal expert bank to the parts of the model that already run fast.
A shared slot pool keeps the fast path intact
The first prototype used an LRU cache and a Python loop that ran experts one at a time. Even when everything was already cached, it only reached 10.1 tokens per second (vs. roughly 32 for the standard implementation). The storage path wasn't the main issue - the Python loop replaced a grouped matrix multiplication that can handle the layer's selected experts together.
The fix was a preallocated device-side slot pool shared across layers. Each layer maintains a mapping from expert IDs to available cache slots. When the router selects experts, the implementation converts those expert IDs into cache slots. That way, the existing grouped matrix multiplication can run on cached experts as if they were just the regular expert bank.
On a smaller OLMoE test model, this design reached 34.6 tokens per second (vs. 38.1 for the standard path). The takeaway is a familiar systems idea: keep the optimized compute path and change how inputs get supplied, rather than reimplementing computation before you have to.
The system loads only routed experts
With experts stored on disk via the experimental loading path, the model doesn't load the full tensors into memory. When there's a cache miss, multiple I/O threads read it, with one contiguous read per expert. Because the cache is bounded, it holds a working set in fast memory without growing until the machine runs out of space.
The system also tries to run ahead. It runs routers for upcoming layers early, using the current hidden states to predict which experts those layers will need. That gives storage a chance to prefetch while the current layer's computation runs. The predictions aren't perfect, but they make use of otherwise idle time.
Quantizing experts before inference reduces how much data needs to come off storage. The setup also needed a separate path for the large n-gram table, because expert streaming alone wasn't enough to fit the whole model on a 32GB machine. This is an important limit: expert streaming targets expert weights, not every other large tensor or auxiliary structure in the model.
What the implementation does - and does not - generalize
The approach is aimed at MoE models with a standard experts layout. The generic implementation was proposed for Transformers, while details like the n-gram table and Metal kernels were handled in a model-specific package. The API was still under discussion in the source account, so the exact configuration should be treated as experimental rather than something you should assume will behave like a stable production interface.
On the Mac, the project used Python and Metal kernels that were compiled at runtime or loaded from the Hub. It didn't rely on a C++ extension. That reduces one integration barrier, but it doesn't remove the need to profile device-specific behavior. The experiment also found that PyTorch's MPS execution and host coordination left performance on the table. In other words: this looks like a promising pattern, not a drop-in guarantee for other models, operating systems, or hardware.
What did the benchmarks actually show?
On the first version running on the 32GB M5 Mac, performance was about 10 tokens per second. After optimization work, it reached 13.9 tokens per second on a 200-token decode test, compared with 14.8 for Strata using the same prompts. That's roughly a 6% gap on that short decode benchmark.
| Test | Transformers with streaming | Strata | Result |
|---|---|---|---|
| Decode, 200 tokens | 13.9 tok/s | 14.8 tok/s | Transformers about 6% slower |
| Prefill, 4K prompt | 228 tok/s | 220 tok/s | Transformers slightly faster |
| Decode after a 6K document | 12.4 tok/s | 13.7 tok/s | Transformers about 10% slower |
The comparison used greedy generation with thinking, multi-token prediction, and speculative drafts disabled. Strata used IQ3_XXS experts. These details matter: changing quantization, prompt length, cache state, or decoding settings can shift results. A single benchmark number without the workload isn't a great basis for choosing an inference stack.
The gains came from coordination, not just faster reads
Most of the performance improvements weren't just about reading experts faster. The team improved throughput by committing GPU work earlier - around a 10% gain - because small MPS kernels could otherwise sit queued while Python assembled more work. They also added Metal kernels for components that didn't have a fast path on macOS.
They reduced dense-weight traffic, adjusted prefill chunking, and overlapped reads of upcoming experts with current computation. For long documents, they only handled selected indexer blocks and grew the key-value cache in place instead of copying it every token. Each change corresponded to a cost they saw in profiling.
The remaining gap largely came down to host coordination. The account estimated about 10ms of host round trips per token. Strata could queue more of the token's work ahead of time and coordinate layer execution via a Metal shared event. The experiment couldn't reproduce that behavior cleanly with the available PyTorch MPS interfaces. So the "last few percent" weren't about a better cache replacement policy - they were about deeper control over how the host schedules GPU work.
Read throughput is not the whole inference experience
Prefill and decode are different. Prefill processes the prompt in larger chunks and can benefit from parallelism. Decode generates one token at a time and repeatedly revisits the model's layers, so it also repeatedly pays coordination costs. That means a system can look good on prompt processing and still feel slower during interactive generation.
For product teams, this matters for user experience. Ingesting a big document can take a while, but what users notice is the delay between generated tokens. Measure both, using prompts that reflect real usage and realistic concurrency. A single tokens-per-second number can hide where users actually feel the wait.
When does local MoE inference make business sense?
Running a large model locally can help when you need to keep sensitive data on-device, reduce dependency on a hosted API, or experiment without paying per request. But the hardware and operating costs don't disappear - they move to device purchases, storage, setup, maintenance, and the time it takes to keep the inference path stable.
Good fit: controlled workloads and clear data constraints
Expert streaming is worth evaluating when the model's capability matters, the full weights don't fit in fast memory, and your workload can tolerate moderate generation speed. Common examples: local prototyping, internal tools for a small team, offline workflows, and development work where you don't want code or customer info leaving a managed device.
The business case tends to be strongest when there's a real constraint to solve. If a smaller model is accurate enough, it might be cheaper and simpler to run. And if a hosted model delivers clearly better results - and you can handle the data under your security and compliance requirements - then the engineering cost of streaming a large model locally may never pay back.
Poor fit: high concurrency and tight latency requirements
One laptop running a single interactive session isn't the same as a production service serving many simultaneous users. The test results focus on single-user inference. Under concurrency, memory pressure, cache contention, storage throughput, and scheduling can change a lot. A bounded expert cache that works well for one request may churn when many requests route to different experts.
The 32GB Mac in the experiment also had very little headroom for other apps. That's fine for a benchmark or a dedicated machine, but it's a shaky assumption if a team expects a normal desktop workflow to stay open. The same caution applies to disk-backed weights: SSDs are fast, but they're still far slower than local memory, and they can become a shared bottleneck.
A practical decision checklist
Before you invest in an expert-streaming setup, check a few things:
- Does the model offer a measurable advantage? Compare output quality against smaller local models and reasonable hosted options.
- What's the real memory footprint? Include dense weights, the expert cache, prompt length, key-value cache, auxiliary tables, and overhead from the OS.
- Where's the bottleneck? Measure prompt prefill and token decode separately, and also re-check after long context.
- Can your users tolerate the latency? Judge the experience in your app, not just benchmark numbers.
- Will the cache behave under real use? Test multiple users, varied prompts, and cold-cache behavior before trusting warm-cache results.
- Who owns the operational work? Quantization, model conversion, hardware differences, updates, and recovery when things break.
This is a useful example of local AI engineering. It's not a reason to move every workload onto a laptop. The right choice depends on what you gain from local control versus what you pay in time and complexity for a slower, more specialized inference system.
Conclusion
Expert streaming changes the memory trade-off for large MoE models. Instead of keeping every expert in fast memory, it uses routing plus a bounded cache and on-demand reads to keep the working set near the processor. On one 32GB Mac, that was enough to run a 180B-class model at roughly 14 tokens per second and get close to a dedicated inference engine.
The bigger lesson is system design. Selective data movement matters, but so do compute-kernel reuse, prefetching, prompt handling, and host-to-GPU coordination. If you're evaluating local AI, the next step is to benchmark against your actual workload and constraints - privacy, latency, concurrency, and operating cost. "Fit" is the first requirement, not the finish line.