Getting 10 tok/s on Qwen 3.8 Flash Next: Why N-Gram Disk Paging Works Where MoE Fails

If you follow the local AI space, the holy grail has always been running massive, high-intelligence frontier models on everyday consumer hardware. For the past year, much of the excitement has centered around Mixture-of-Experts (MoE) streaming engines like Colibri and FreeToken, which attempt to run gigantic 200B+ parameter models by paging weights directly from fast NVMe SSDs.

While the engineering behind SSD expert offloading is brilliant, anyone who has actually benchmarked it on a gaming PC knows the harsh reality: you slam directly into physical disk and memory bandwidth limits. You end up watching your shiny NVMe drive peg at 7 GB/s while the model ekes out a sluggish 1.0 to 2.5 tokens per second.

Recently, I got Qwen 3.8 Flash Next (UD-IQ3_XXS, ~83 GB total footprint) running on my NVIDIA RTX 5060 Ti (16 GB) and Intel i7-14700F setup at a rock-solid 10+ tokens per second with a full 64K context window.

Here is why this hybrid architecture works so well on consumer hardware, and why paging a 51B N-gram lookup table from an SSD completely bypasses the disk bottlenecks that kill traditional MoE offloading.

The Dilemma with Large Models: Colibri vs. FreeToken

To understand why Qwen 3.8 Flash Next feels so refreshing, it helps to look directly at the trade-offs we’ve had to make when trying to run massive, 100B+ to 300B frontier MoE models on a 16GB GPU:

1. The Colibri Approach: Streaming a Q3 Quant from NVMe SSDs

Colibri attempts to keep model quality high by serving a Q3 / MXFP4 quant (~167 GB) of DeepSeek V4 Flash (284B total / ~34B active) directly off fast storage. It keeps the bulk of the expert weights on disk and streams active experts into memory on-the-fly during each token generation step.

Colibri even supports dual NVMe drive striping (mir_pread_striped) across both of my Gen4 drives (SK hynix P41 + Verbatim Vi7000) to pull parallel reads at upwards of ~14 GB/s.

[ Colibri NVMe Streaming: High Quality Q3, Hard Disk Ceiling ]

Token 1 ──> [ Layer 1 Experts ] ──> [ Layer 2 Experts ] ... ──> [ Layer N Experts ]
                 │                       │                           │
                 ▼                       ▼                           ▼
       [ Read 200 MB Weights ]   [ Read 200 MB Weights ]     [ Read 200 MB Weights ]
       ─────────────────────────────────────────────────────────────────────────────
       Total per Token = 4 to 8 GB of Q3 weights streamed from NVMe per SECOND!
       Result on DeepSeek V4 Flash: ~1.00 tok/s (NVMe I/O Saturated)

The benefit is that you maintain good Q3 precision without eating up your entire system RAM. But the downside is brutal: pulling hundreds of megabytes of raw tensor weights per token across dozens of layers immediately saturates the PCIe storage bus, capping generation at a painful ~1.00 tok/s.

2. The FreeToken Approach: Squeezing a Q1 Quant into DDR5 RAM

FreeToken takes the opposite route: avoiding the disk bottleneck entirely by loading the whole model into memory using bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy).

To fit a 284B model into a 64GB desktop system, however, you have to crush it down to an ultra-low precision Q1 quant (UD-IQ1_S, ~77 GB). We offload 18 dense layers (~14.2 GB) into RTX 5060 Ti VRAM and pack the remaining ~62 GB of expert layers into host DDR5 RAM.

[ FreeToken In-Memory: Lower Precision Q1, RAM Saturated ]

Token 1 ──> [ 18 VRAM Dense Layers ] ──> [ ~62 GB Resident DDR5 Expert Cache ]
                    │                                     │
                    ▼                                     ▼
          [ RTX 5060 Ti VRAM @ 460 GB/s ]       [ Host DDR5-6000 @ 81.8 GB/s ]
          ─────────────────────────────────────────────────────────────────────
          Result on DeepSeek V4 Flash: ~2.28 tok/s (DDR5 RAM Bus Saturated)

This approach eliminates disk reads, but comes with steep penalties:

1. Precision Loss: Q1 quantization noticeably impacts reasoning and output quality compared to Q3+.
2. Resource Exhaustion: It consumes virtually 100% of your 64GB system RAM, leaving zero headroom for other apps or large context windows.
3. RAM Bandwidth Ceiling: Even with DDR5-6000 running at 81.8 GB/s measured STREAM bandwidth, you still only get ~2.28 tok/s.

The Secret: The Rainbow Table of AI (Why N-Gram Lookup is NOT Bandwidth Constrained)

If you have a background in IT security or cryptography, the N-gram mechanism in Qwen 3.8 Flash Next will feel instantly familiar: it works just like a rainbow table.

In password cracking, instead of burning massive GPU compute brute-forcing trillions of cryptographic hashes at runtime, you use a rainbow table to trade cheap storage space for expensive compute time. You precompute the hash chains ahead of time, store them on disk, and at runtime, cracking is just a fast table lookup.

Qwen 3.8 Flash Next applies the exact same time-memory trade-off to language modeling:

1. Core Compute: A lean, highly efficient backbone combining Gated DeltaNet (GDN) linear attention with Quick Sparse Attention (QSA).
2. Predictive Retrieval (The Rainbow Table): A massive 51B N-gram lookup table precomputed and packaged directly into the model file.

Instead of forcing your GPU and CPU to execute trillions of dense matrix multiply-accumulate (MAC) floating-point operations across dozens of transformer layers just to predict obvious phrases, common syntax boilerplate, or established factual tuples, the model simply looks up the precomputed sequence in its hash table.

[ N-Gram Lookup: The Rainbow Table Approach ]

Current Tokens ──> [ Hash Key ] ──> [ Memory Map (mmap) ]
                                            │
                                            ▼
                           [ Fetch 4 KB Precomputed Logits ]
                           ─────────────────────────────────
                           Total Data Read = A few Kilobytes!
                           NVMe Bandwidth Used: < 0.1% (< 1 MB/s)
                           NVMe Random Read Latency: ~50 µs

Here is the fundamental insight: Looking up an entry in an N-gram table is an index retrieval, not a dense matrix multiplication.

When the model queries the N-gram table:

• It doesn’t load gigabytes of weights into matrix-multiplication kernels.
• It hashes the preceding token sequence and reads a tiny slice of memory—just a few kilobytes containing token IDs and candidate likelihoods.
• Through operating system memory mapping (mmap), the OS keeps active index pages in system RAM cache.
• When an uncached N-gram needs to be fetched from the NVMe drive, it performs a 4KB random read, taking around 50 microseconds.

The core GPU model then verifies the drafted tokens in a single, fast parallel forward pass. Because the amount of data transferred per token is in kilobytes rather than gigabytes, disk bandwidth is completely eliminated as a bottleneck. Your NVMe drive sits virtually idle while the model breezes through token generation at interactive speeds.

Dialing in llama.cpp on an RTX 5060 Ti

The real trick to hitting 10+ tok/s is carefully partitioning the model between VRAM and system memory so Windows never triggers GPU driver memory spilling.

My workstation hardware:

• GPU: NVIDIA GeForce RTX 5060 Ti (16 GB VRAM, Blackwell sm_120)
• CPU: Intel Core i7-14700F (8 P-Cores, 12 E-Cores, up to 5.4 GHz)
• RAM: 64 GB DDR5-6000 CL36 (81.8 GB/s measured STREAM bandwidth)
• Primary NVMe (C:): SK hynix Platinum P41 2TB PCIe 4.0 (7.15 GB/s read)
• Secondary NVMe (D:): Verbatim Vi7000G 4TB PCIe 4.0 (7.05 GB/s read)

The VRAM Budget Math

On Windows, running out of dedicated VRAM causes the WDDM graphics driver to silently page memory to system RAM over the PCIe bus, dropping token speeds through the floor. We need to stay strictly under 15.5 GB total VRAM allocation:

By offloading exactly 15 layers to the RTX 5060 Ti, we leave over 3 GB of VRAM headroom for the 4-bit compressed 64K context buffer.

The PowerShell Serving Script

Here is the exact llama.cpp serving configuration I use in serve-Qwen3.8-Flash-Next.ps1:

# serve-Qwen3.8-Flash-Next.ps1
param (
    [string]$ModelPath = "C:\models\Qwen3.8-Flash-Next-UD-IQ3_XXS.gguf",
    [int]$Port = 8080,
    [int]$Ctx = 65536,
    [int]$GpuLayers = 15,
    [string]$HostName = "127.0.0.1"
)

# Isolate compute strictly to GPU 0 (RTX 5060 Ti), leaving secondary GPU/GUI untouched
$env:CUDA_VISIBLE_DEVICES = "0"

.\bin\llama-server.exe `
  -m $ModelPath `
  -c $Ctx `
  -ngl $GpuLayers `
  -np 1 `
  -ctk q4_0 `
  -ctv q4_0 `
  -fa on `
  -sm none `
  -mg 0 `
  -t 8 `
  -tb 8 `
  --prio 2 `
  -cram 24576 `
  --ctx-checkpoints 16 `
  --no-warmup `
  --reasoning-preserve `
  --host $HostName `
  --port $Port

Key Optimizations Explained

• -ngl 15: The sweet spot for 16GB GPUs. Maximizes GPU acceleration while leaving enough VRAM for massive context windows.
• -ctk q4_0 -ctv q4_0: Compresses Key/Value caches to 4-bit, slashing context memory usage by 50% without measurable quality loss.
• -fa on: Enables Flash Attention to speed up prompt evaluation and context processing.
• -t 8 -tb 8: Restricts CPU thread pools to the 8 physical Performance cores (P-cores) of the i7-14700F, preventing the Windows scheduler from assigning compute threads to slower Efficiency cores (E-cores).
• --prio 2: Elevates the process priority in Windows to eliminate background stuttering and input lag.
• --ctx-checkpoints 16: Configures recurrent state checkpoints for Gated DeltaNet (GDN), allowing instantaneous prompt resumption in long agentic conversations.

Real-World Large Model Benchmark Comparisons

Here is how Qwen 3.8 Flash Next compares against other 100B+ frontier models that exceed dedicated 16GB VRAM on this exact test rig:

The Big Model Takeaways

1. The NVMe Streaming Trap (Colibri Q3): Streaming a high-quality Q3 quant of a 284B model off fast SSDs keeps your RAM free, but reading gigabytes of dense tensor weights per token limits you to ~1.00 tok/s, even with striped PCIe 4.0 drives reading at ~14 GB/s.
2. The In-Memory Quant Sacrifice (FreeToken Q1): Moving the model entirely into DDR5 RAM avoids disk reads, but squeezing a 284B model into a 64GB PC forces you down to a lossy Q1 quant, consumes all system memory, and still hits the 81.8 GB/s DDR5 bus wall at ~2.28 tok/s.
3. The N-Gram Hybrid Breakthrough (Qwen 3.8 Flash Next Q3): Qwen 3.8 Flash Next completely breaks this compromise. It runs at a crisp Q3 quant at 180B scale, keeps the dense compute in VRAM, queries the 51B N-gram index via lightweight 4KB disk point lookups, leaves 14+ GB of system RAM free for other apps, and delivers a comfortable 10+ tok/s.

Conclusion

For the past couple of years, the local AI community has pushed hard on brute-force streaming: trying to shove tens of gigabytes of sparse MoE expert weights across PCIe buses and NVMe lanes on every single token. While it technically allows huge models to boot on consumer PCs, the bandwidth wall makes it frustratingly slow for daily work.

Qwen 3.8 Flash Next demonstrates a much smarter path forward. By combining lightweight linear attention (Gated DeltaNet) with sparse, point-retrieval N-gram tables, we can keep massive knowledge representations on standard NVMe storage without ever saturating drive bandwidth.

If you’ve got an RTX 5060 Ti or similar 16GB GPU and 64GB of RAM, give this setup a spin. Getting 10+ tok/s out of a frontier-class model on home hardware proves that smart architecture wins over raw brute-force every time.