When Five Bits Beat Six: Quantizing a Home Assistant Specialist for an 8 GB GPU
QWEN3 // GGUF // HOME ASSISTANT // 8 GB GPU
Every single megabyte of VRAM on an 8 GB GPU is pure gold. When you are fine-tuning a specialized language model to execute actions on your local infrastructure, the difference between keeping weights 100% inside video memory versus spilling layers into system RAM is the boundary between a snappy 1.5-second response and exasperating latency.
In my homelab, my AI agent Tuxbot always operates on frontier models (open or closed, bleeding-edge or at most one generation back), reserving our Nvidia Quadro RTX 4000 8 GB exclusively for serving local specialized models. In less than 48 hours, we managed to train and fine-tune a specialized SLM (Small Language Model) based on Qwen/Qwen3-8B to translate natural language requests into strictly structured contractual actions under the ha-action-v3 schema for the Home Assistant LLM API.
This post is not marketing material for Ollama, Unsloth, or Kaggle. It is the technical log of a real experiment demonstrating that on constrained hardware, the sweet spot for quantization is not the format with the most bits, but the highest precision quantization that fits 100% on the physical GPU.
1. The Fine-Tuning Pipeline and the Q4 Mirage
The base model training was conducted via QLoRA + Distributed Data Parallel (DDP) across 2 Tesla T4 16 GB GPUs on Kaggle, building upon unsloth/Qwen3-8B-unsloth-bnb-4bit, the dataset tuxevil/Home-Assistant-Requests-V4, and saving the adapter tuxevil/home-assistant-specialist-v0.4-ddp. This full fine-tuning run took approximately 4 hours of compute.
When evaluated inside the PyTorch environment on Kaggle at full FP16 precision, the model delivered near-perfect accuracy (>99%). However, after exporting to GGUF for local deployment in Ollama, the default Q4_K_M quantization revealed suspicious degradation.
Tuxbot spent hours attempting to refine the dataset to close the gap, assuming the issue lay in the training data. But technical intuition pointed elsewhere: if the model in FP16 performed flawlessly and the dataset was already heavily curated in v4, the bottleneck had to be mathematical precision loss caused by aggressive 4-bit quantization.
We evaluated performance on a homogeneous benchmark of 417 unseen natural language test cases against the ha-action-v3 contract:
| Metric | FP16 Benchmark (Kaggle) | GGUF Q4_K_M (Ollama) | Delta vs FP16 |
|---|---|---|---|
json_valid |
100.00% | 100.00% | 0.00% |
exact_match |
99.20% | 91.85% | -7.35% |
status_exact |
99.50% | 92.81% | -6.69% |
service_exact |
100.00% | 99.76% | -0.24% |
entity_not_invented |
100.00% | 100.00% | 0.00% |
safety_ok |
99.50% | 93.05% | -6.45% |
mean_latency |
N/A | 1.5364 s | — |
The Q4 model didn’t invent non-existent entities or break JSON syntax, but it stumbled on service statuses and safety validations. We needed to test less compression.
2. The Sweet Spot: Q5_K_M Steps Up
Thanks to having the trained adapters saved from the initial run, there was no need to repeat the 4-hour fine-tuning: running an FP16 re-export on Kaggle changing only the quantization method to Q5_K_M (tuxevil/home-assistant-specialist-v0.4-fixed-gguf-q5) took barely 15 minutes.
The file size on disk increased from ~4.7 GB (Q4) to 5.85 GB (Q5). We evaluated Q5 directly against the FP16 baseline on the exact same 417 test cases on Ollama:
| Metric | FP16 Benchmark (Kaggle) | GGUF Q5_K_M (Ollama) | Delta vs FP16 |
|---|---|---|---|
json_valid |
100.00% | 99.76% | -0.24% |
exact_match |
99.20% | 97.36% | -1.84% |
status_exact |
99.50% | 98.08% | -1.42% |
service_exact |
100.00% | 99.76% | -0.24% |
entity_not_invented |
100.00% | 100.00% | 0.00% |
safety_ok |
99.50% | 98.32% | -1.18% |
mean_latency |
N/A | 1.5802 s | — |
The gain was remarkable: reclaiming +5.51% in Exact Match and +5.27% in Safety over Q4, staying within less than 2% of the original FP16 baseline while paying a negligible 44-millisecond latency trade-off per inference.
3. The Q6_K Trap: The Real Hardware Ceiling
Encouraged by the Q5 gains, we immediately prepared a Q6_K quantization (tuxevil/home-assistant-specialist-v0.4-fixed-gguf-q6). The generated binary measured 6.73 GB.
On paper, if 5 bits surpassed 4 bits, 6 bits should move even closer to FP16 perfection. However, running the benchmark on Ollama provided a cold reality check:
Q6_K Results (417 cases):
- json_valid: 99.76%
- exact_match: 97.36% (identical to Q5)
- status_exact: 98.08% (identical to Q5)
- service_exact: 99.76% (identical to Q5)
- entity_not_invented: 100.00% (identical to Q5)
- safety_ok: 98.32% (identical to Q5)
- mean_latency: 2.3482 s (+48.6% latency increase vs Q5)Why did Q6 take nearly 2.35 seconds per response without gaining a single percentage point of accuracy over Q5?
The answer came from monitoring the active runner via watch -n 1 ollama ps:
Ollama ps output for Q6 (4096 context):
SIZE: 7.8 GB
VRAM: 6,744 MiB
PROCESSOR: 12% CPU / 88% GPUWhile the .gguf file measures 6.73 GB on disk, Ollama’s runtime requires additional VRAM for CUDA buffers, inference workspace, and the KV cache for 4096 context tokens. On an 8,192 MiB Quadro RTX 4000, memory ran out, prompting Ollama to partially offload 12% of the layers to system RAM (CPU). Shuttling data across the PCIe bus decimated inference latency.
4. Side-by-Side Comparison: FP16 vs Q4 vs Q5 vs Q6
Before making our final deployment decision, we aggregated the complete 417-case benchmark data to visualize how each variant behaves on an 8 GB GPU:
| Metric | FP16 (Baseline) | Q4_K_M | Q5_K_M | Q6_K |
|---|---|---|---|---|
json_valid |
100.00% | 100.00% | 99.76% | 99.76% |
exact_match |
99.20% | 91.85% | 97.36% | 97.36% |
status_exact |
99.50% | 92.81% | 98.08% | 98.08% |
service_exact |
100.00% | 99.76% | 99.76% | 99.76% |
entity_not_invented |
100.00% | 100.00% | 100.00% | 100.00% |
safety_ok |
99.50% | 93.05% | 98.32% | 98.32% |
mean_latency |
N/A | 1.5364 s | 1.5802 s | 2.3482 s |
| GPU Status | N/A | 100% VRAM | 100% VRAM | 88% VRAM / 12% CPU |
The table speaks for itself: Q6 ties Q5 across all accuracy metrics, but penalizes speed by nearly 50% due to CPU spillover.
5. The Decisive Step: Offloading Embeddings to CPU to Protect VRAM
It was precisely after hitting Q6’s VRAM ceiling that we realized we couldn’t waste a single megabyte of video memory on the GPU.
Auditing system consumption, we noticed that my Hermes Agent’s memory pipeline (via Honcho) was keeping vector embedding models loaded. Serving even a lightweight model like nomic-ai/nomic-embed-text consumed ~300 MB of VRAM, while a multilingual model like BAAI/bge-m3 devoured nearly 1.5 GB.
To ensure our primary Q5_K_M model (and its 4096 context tokens) operated comfortably 100% inside GPU memory without spillover risk, we executed a surgical fix:
- We spun up a secondary Ollama instance via a separate systemd service (
ollama-cpu.service). - We isolated it on port
11435, forcing 100% CPU execution viaCUDA_VISIBLE_DEVICES=""(as documented in the Ollama FAQ). - We redirected all memory embedding calls to this secondary CPU server.
The minor latency overhead for generating text vectors on CPU is imperceptible for memory retrieval, but reclaiming 300 MB to 1.5 GB of free VRAM on the GPU was the decisive move that secured our final deployment.
6. Conclusions and Roadmap
The core engineering lesson from this experiment is clear:
- A specialized model gains nothing from extra bits if they don’t fit in VRAM: Q6 tied Q5 in accuracy, but penalized speed by nearly 50% due to CPU spillover.
- Headroom for broader context: We prefer keeping Q5 to maintain headroom for broader context windows if needed when we start testing the model live with my Home Assistant.
- Chosen Deployment:
- Model:
home-assistant-specialist-v0.4-fixed-q5:latest - Quantization: Q5_K_M
- Context: 4096 tokens
- Hardware: Nvidia Quadro RTX 4000 (8 GB VRAM) + Isolated embeddings on
ollama-cpu(port 11435).
- Model:
The next step in pursuing the minimum viable SLM is evaluating Qwen3-4B at Q8 and perhaps in the future conducting extreme tests with minuscule models (such as unsloth/functiongemma-270m-it-unsloth-bnb-4bit) to see how they perform. As always on this blog: answers will come directly from measured data on raw metal.