BusyJay

Running a 95 GiB MoE Model on a 32 GiB GPU: From 10 to ~100 Tokens/s

· Jay

Recently, I spent some of my spare time deploying Qwen3.8-Flash-Next for local inference at home and working through a series of optimizations. Here are a few things I learned along the way.

First, the hardware:

  • GPUs: 5090 D 32 GiB + 4080 Super 16 GiB
  • Motherboard: X870E GODLIKE
  • RAM: 192 GiB

The 5090 uses PCIe 5.0 x16, while the 4080 connects over USB4.

The main Qwen3.8-Flash-Next model has 48 Transformer layers and 125 billion parameters. Add roughly 51 billion parameters for N-gram embeddings and another 4 billion for the MTP module, and the total comes to around 180 billion parameters. At BF16 precision, the weights alone would require about 335 GiB, before accounting for the KV cache, runtime state, and temporary computation buffers. Fitting the entire model into an RTX 5090 D with just 32 GiB of VRAM is clearly unrealistic. However, it uses an MoE architecture: for each token, each layer selects only 10 of its 512 routed experts, giving the main model roughly 6 billion active parameters. This means we can keep frequently used components, such as attention and shared experts, resident on the GPU, store the full set of routed expert weights in host memory, and keep only the experts needed for computation in VRAM. By passing the weights through VRAM in batches, we can run inference with the complete model.

To further reduce storage and transfer costs, I chose the quantized nvidia/Qwen3.8-Flash-Next-FP4-Local model, which I will call 3.8-FP4 below. Its full set of weights is about 95 GiB. In the converted runtime layout, each routed expert’s weights and scales occupy about 2.64 MiB. For single-token inference, the 10 experts selected in a layer need only about 26.4 MiB of weight storage, compared with about 1.32 GiB for all 512 experts in that layer. The 5090 D therefore does not need to hold all expert weights at once: it can retain the resident modules, KV cache, and runtime buffers while loading the required experts layer by layer. With this simple idea, I had Codex build an initial implementation based on SGLang. The model started successfully, with prefill throughput in the hundreds of tokens per second and decode throughput of about 10 tok/s.

Trimming VRAM Usage

To improve inference throughput, the key is to reduce wasted VRAM and leave as much room as possible for frequently accessed expert weights.

Starting with the weights themselves, the N-gram embeddings occupy 27 GiB, but processing each token accesses only about 1.41 KiB of data through random lookups. There is no need to keep the entire table in VRAM; it can stay in host memory and be accessed on demand. The token embeddings, at about 1.18 GiB, can also be offloaded to host memory. The vision weights in 3.8-FP4 take up about 0.3 GiB and are used relatively infrequently, so they can be moved out as well. I happen to have the second GPU, the 4080, and SGLang already supports encode-prefill-decode (EPD) disaggregation, so the vision weights can go there. This also lets me use the full BF16 version of those weights.

For the KV cache, I independently chose FP8 E4M3, while the weights use NVFP4. In my day-to-day use, this precision has been sufficient. There is also no need to target the model’s maximum supported context length: a total context budget of 220K tokens is enough for my use case. My main goal is response speed for a single request, and the hardware cannot sustain higher concurrency anyway, so I limit the number of running requests to one and queue the rest. This avoids reserving extra cache capacity for multiple active requests. The current configuration provides 220K token slots, with the main model’s KV pool and its associated QSA state occupying about 2.68 GiB.

Performance Optimizations

With 192 GiB of RAM, there is more than enough room to load all the weights into host memory. At startup, I load all expert weights and per-layer embedding (PLE) weights into memory and register them as CUDA-mapped pinned memory, allowing the GPU to address and access them directly. Components such as shared experts and routers are used in every layer, so they stay in VRAM. A conversation involves two phases, prefill and decode; I will discuss the optimizations for each separately.

Decode

When decoding a single token, each layer only needs to compute 10 experts, so there is no need to load the entire layer. I keep the shared experts resident in VRAM and use a VRAM buffer pool to cache other frequently used experts. Most expert weights can then be read from the cache, with misses copied from host memory into VRAM. While those transfers are in progress, computation can already begin on the shared experts and cache hits, reducing decode latency.

Routing patterns differ across layers. To avoid interference between those patterns causing poor eviction decisions, it makes sense to partition the buffer pool by layer. Those differences also mean each layer needs a different amount of cache: some layers may achieve a high hit rate with 90 slots, while others still perform poorly with 200. Cache capacity therefore needs to adapt. As decoding progresses, some capacity is moved from layers with high hit rates to layers with low hit rates. Within each layer, least frequently used (LFU) eviction determines which experts to evict. Measurements show that LFU produces 8% fewer misses than LRU.

Equal and adaptive allocation of per-layer expert caches, with the same total capacity.

3.8-FP4 recommends W4A16 for decode computation. To improve decode speed further, I added speculative decoding. MTP uses W4A4, and in my tests, step=2 gave the best overall results, improving throughput by 14%.

Prefill

Prefill processes many tokens at once and may activate experts across an entire layer, exceeding the weight capacity of that layer’s cache slots. I therefore use a dedicated arena for expert weights. If the layer’s cache slots already cover all the required experts, computation can proceed directly. Otherwise, the missing weights are loaded from host memory into the arena in a batch before computation begins. The transfer cost is amortized across all tokens in the prefill batch, yielding better overall throughput. I currently use a prefill batch size of 8192 tokens. Stock SGLang can easily run out of memory at this size, so additional scratch-buffer chunking is needed to reduce peak VRAM usage.

The official recommendation for 3.8-FP4 is to use W4A4 for prefill, which also significantly improves prefill speed.

Reusing VRAM

You may have noticed that these optimizations leave us with quite a few buffer pools in VRAM: the KV cache, the expert cache, and the prefill arena. These pools are not all fully used at the same time. During decode, for example, the prefill arena is idle; when the context is still short, the tail of the KV cache is unused. All of this memory can be “lent” to the expert cache to improve its hit rate. For example, the KV cache can be divided into three capacity stages: 80K, 160K, and 220K tokens. When the context is below 80K, the memory for the remaining 220K − 80K = 140K token slots goes to the expert cache. Once the context exceeds 80K but remains below 160K, the expert cache returns the memory for another 80K token slots. The prefill arena is likewise assigned to the expert cache during decode. Together, this provides more than 7,000 cache slots during typical decoding, substantially improving the hit rate.

Wrapping Up

This post covers a range of optimizations for fitting 95 GiB into 32 GiB. The most important points are:

  1. Reclaim as much VRAM as possible for expert weights on the hot path.
  2. Make full use of the GPU’s many compute units. Avoid algorithms that look efficient on a CPU but leave the GPU largely idle.
  3. Minimize interactions between the GPU and CPU. Run work asynchronously wherever possible, and keep work on the GPU when it can finish there instead of splitting it into CPU subtasks.

I would not claim that there is no room left for improvement, but the results have reached a point I am happy with. The local deployment supports both text and vision. In a recent real chat request, an uncached prompt of about 25.6K tokens had a time to first token (TTFT) of 6.8 seconds, followed by sustained generation at about 96 tok/s. In an earlier long-context test, an uncached prompt of 128K tokens had a TTFT of about 28 seconds, followed by sustained generation at about 93 tok/s. Peak decode throughput can exceed 140 tok/s. The code is available at https://github.com/BusyJay/sglang/tree/qwen3.8-flash-next-offload. Feedback is welcome.

P.S. All the code was generated by Codex and reviewed by Codex. Thanks to OpenAI for resetting the usage limits every two days lately. I have not read a single line of the code myself—just looked at the results :)

P.P.S. I wrote the original post by hand; Codex created the diagram and provided the English translation.