Running a 35B MoE at 45+ tok/s on a $170 GPU: My Experience with FreeToken and Qwen3.6

FREETOKEN // QWEN3.6_35B // RTX4000_8GB // OFFGRID_LAB

There is a widespread belief that if you only have a modest 8 GB VRAM GPU, you are doomed to stick with dense 7B or 8B parameter models if you want decent inference speed.

In my homelab, where I live 100% off-grid generating and storing my own solar power, every single watt and hardware dollar counts. My main workhorse for serving local models is an aging NVIDIA Quadro RTX 4000 8 GB based on the Turing architecture (sm_75), which I picked up second-hand for just $170.

Recently, I tried running Qwen/Qwen3.6-35B-A3B using Ollama. The results were underwhelming: 11 tok/s on full CPU offload and 16 tok/s when splitting 74% CPU and 26% GPU. Given that sluggishness, I had already written off any model that couldn’t fit 100% inside my VRAM.

That was until FlashML-org/FreeToken showed up: an edge-native inference engine purpose-built for Mixture-of-Experts (MoE) architectures that keeps an active cache of experts in GPU VRAM and dynamically streams the rest from host system RAM over PCIe.

This is the chronicle of what happened when I set out to make a 35B parameter monster run on my $170 GPU.

---

1. The First Wall: Turing Is Not Ampere

Qwen3.6-35B-A3B packs 35 billion total parameters, but only activates around 3 billion per generated token. That MoE design is the sweet spot for FreeToken’s dynamic expert caching.

The friction hit right after cloning the repository: the branch supporting generic GGUF assumed modern hardware (sm_80+ like Ampere, Ada, or Hopper). My RTX 4000 is Turing (compute capability 7.5). When building the C++ and CUDA extensions, the server crashed at runtime:

RuntimeError: no kernel image is available for execution on the device

I am not a low-level CUDA kernel developer, nor do I pretend to be; I’m simply a tech enthusiast looking to get the maximum mileage out of limited hardware. I turned to ChatGPT as my debugging copilot. Instead of telling me to drop the idea, ChatGPT dug into the codebase and open Pull Requests.

By cross-referencing PR #131 (feat/generic-gguf) with PR #24 (which added explicit support for sm_75), we found the root causes:

1. We added manual capability gating in python/freetoken/kernel/backend.py to disable flashinfer, sgl_kernel, and incompatible Triton kernels on Turing, forcing FreeToken to use Triton for the attention mechanism.
2. We fixed a broken header reference in PyTorch inside ATen/List_inl.h.

With those patches applied, the GGUF extension compiled cleanly for sm_75.

---

2. The Speed Leap: 45.44 tok/s on IQ3_S

The first quant I put through its paces was Qwen3.6-35B-A3B-IQ3_S.gguf (an aggressive 3-bit quantization by jimbothigpen).

In FreeToken, this quant can only run in offload mode (GPU + expert cache + PCIe streaming from RAM). Setting the memory-ratio parameter to .90, I triggered the first generation run.

The numbers were staggering:

Decode throughput: 45.44 tokens/second
Prefill (~2K tokens): ~143 tokens/second
TTFT (with prefix cache): ~1.21 seconds

Jumping from Ollama’s sluggish 16 tok/s to over 45 tokens per second on a 35B model using a $170 GPU was a total game changer. The inference was snappy, responsive, and completely viable for interactive work.

---

3. The Comparison: Q4_K, NVFP4, and the Q5 Trap

Riding the high of that first success, I benchmarked several other quants to understand the trade-offs between speed and precision:

A. Q4_K (FastFlow)

The Qwen3.6-35B-A3B-q4_k.gguf checkpoint featured homogeneous expert banks (gate_up=Q4_K, down=Q4_K).

• Pure decode dropped compared to IQ3, delivering 39.45 tok/s.
• But on fresh prefill (~2K tokens), it surged to ~194 tok/s, ingesting context 35% faster than IQ3.

B. Native NVFP4 (ModelOpt)

We tested the native Qwen3.6-35B-A3B-NVFP4 checkpoint using --nvfp4-backend triton (since Turing lacks native FP4 hardware tensor cores).

• Decode held up at 37.21 tok/s.
• Prefill tanked badly to ~57 tok/s (3.4x slower than Q4), with a cold-start delay exceeding 30 seconds. On Turing, NVFP4 simply isn’t in its element.

C. Q5_K_S

I assumed stepping up to 5-bit would bring an obvious bump in intelligence to justify the heavier footprint.

• Decode dropped to 34.53 tok/s.
• Prefill landed at ~172 tok/s.
• In subsequent semantic testing, it scored 92.86%, identical to IQ3. Q5 ended up stranded in no-man’s land: heavier, slower, and without any tangible quality benefit in real usage.

Quant / Format	Decode (tok/s)	Prefill @ 2K (tok/s)	Semantic Quality	Strict Quality	Turing Verdict
IQ3_S	45.44	143	92.86%	60.71%	Clear winner in speed and disk footprint
Q4_K	39.45	194	89.29%	71.43%	Excellent for heavy context ingestion
NVFP4	37.21	57	89.29%	78.57%	Very slow prefill under Triton emulation
Q5_K_S	34.53	172	92.86%	64.29%	Speed penalty with no semantic gain

---

4. Off-Grid Realities and the Myth of Hybrid Mode

My rig runs an AMD Ryzen 9 7945HX (16 cores, 32 threads) with 96 GB of DDR5-5600 RAM. Because I live on solar power, I keep the CPU strictly throttled in the BIOS: Eco mode, disabled boost, powersave governor, and clock speeds locked around ~2.5 GHz.

FreeToken features a hybrid mode that splits MoE expert execution between CPU cores and GPU. Testing this in Eco mode caused performance to crater to 16 - 23 tok/s.

We wondered if hybrid mode was suffering because the CPU was restricted to 2.5 GHz. To test this hypothesis, we unleashed the Ryzen: toggled boost=1 and set governor to performance.

The result was eye-opening: the inference speed increase was negligible and virtually imperceptible. The bottleneck wasn’t CPU clock speed; it was synchronization overhead and bus latency.

For my solar setup, this was a massive win: I can keep the CPU in Eco mode drawing minimal power while letting the RTX 4000 handle the heavy lifting in offload mode without sacrificing a single token per second.

---

5. The False 25/100: When the Harness Fails, Not the Model

To avoid relying on subjective impressions, we put together a 28-task benchmark suite (reasoning, extraction, JSON, tool calling, coding, and long context).

The initial run on IQ3_S returned an alarming score: 25/100. It posted zeroes across extraction, tool calling, and coding.

It would have been easy to conclude that 3-bit quantization had ruined the model. Instead of giving up, we inspected the raw backend completions.

We caught three critical bugs in the test harness:

1. Token exhaustion during thinking: For many prompts, the model solved the problem inside its reasoning_content block, but exhausted its max_tokens budget before outputting the final visible response, ending in finish_reason=length.
2. Mismatched tool call parser: FreeToken defaulted to the qwen25 parser, whereas Qwen3.6 produced native Qwen3 formatted tool calls. Switching to qwen3_coder instantly resolved tool calling.
3. Sandbox execution permissions: Generated code ran safely under the nobody user, but the script attempted to call Python inside /root, triggering PermissionError failures.

Once we fixed the harness, disabled thinking on simple extraction/JSON tasks, and evaluated semantic quality independently from strict formatting adherence, the picture changed entirely:

• IQ3_S scored 92.86% in semantic quality.
• It scored a perfect 4/4 (100%) on tool calling.
• It achieved 100% on code generation and reasoning tests.

The 3-bit model hadn’t lost its intelligence. Its only quirk was occasionally adding conversational prose or markdown code fences when asked for raw strings—something easily managed in software.

---

6. Conclusion: Simplicity and Pragmatism

In theory, one could design a dual-model routing pipeline: send massive context documents to Q4_K and long text generations to IQ3_S.

In practice, I favor simplicity:

• Storing multiple 35B quantized files quickly wastes disk space.
• I chose IQ3_S as my sole daily driver. It has the smallest footprint, delivers the highest generation speed (>45 tok/s), and strict formatting quirks are easily handled with robust prompt engineering, structured schemas, and proper tooling.

Running a 35B parameter MoE model at over 45 tok/s on a $170 8 GB GPU powered by solar panels proves that you don’t need thousands of dollars in hardware to run cutting-edge local AI.