← Back to Garden

Running 27B Parameter LLMs on an 8GB GPU: 1-Bit, Ternary, and 262K Context

•
#LLM#1-Bit#Ternary#PrismML#CUDA#GGUF#Homelab

BENCHMARK_LOG // RTX4000_8GB // PRISM_ML // LLAMA_CPP

Running 27-billion parameter models locally used to require an expensive multi-GPU enterprise rig or heavy CPU offloading that dragged generation down to a crawling 2 - 4 tok/s. Attempting to drop below 4 bits with traditional Post-Training Quantization (PTQ) like IQ2_XXS or Q2_K only caused severe intelligence collapse on complex reasoning tasks.

On my host workstation beast, I broke through this barrier by combining Prism ML’s native binary and ternary neural architectures (Bonsai-27B-Q1_0 and Ternary-Bonsai-27B-Q2_0) with aggressive inference optimizations in llama.cpp and a key hardware adjustment: removing the manual power cap I had previously applied to the GPU (stepping up from 100W back to its stock 125W rating).

The result: 27.35 tokens/second running 100% on GPU with a massive 160,000 token context window on a single NVIDIA Quadro RTX 4000 (8 GB VRAM).

Key Performance Milestones

  • 27.35 tok/s at 160K Context (125W Stock): 100% resident in 8 GB VRAM across 163,840 context tokens using Bonsai-27B-Q1_0.gguf.
  • Full 262K Context Window Unlocked: Scaled to native context limit of 262,144 tokens in hybrid GPU/CPU mode, generating at 12.93 tok/s.
  • High-Quality Ternary Serving at 18.04 tok/s: Executed Ternary-Bonsai-27B-Q2_0.gguf (retaining 94.6% of FP16 baseline intelligence) across 32,768 context window 100% on GPU.
  • Real Efficiency Gain at 125W Stock: Removing the manual 100W power cap yielded a +32.7% throughput jump in 1-bit mode, reducing actual energy consumed per generated token from 4.85 J/token down to 4.57 J/token. In ternary mode, the minor consumption increase is negligible compared to the responsiveness boost.

Architectural Deep-Dive: Native Low-Bit vs. Conventional PTQ

A. Weight Representations (Q1_0_g128 vs Q2_0_g128)

Unlike traditional PTQ methods that force dense FP16 weights into artificial low-bit bins (causing catastrophic accuracy loss in multi-step reasoning), the Bonsai-27B family uses native binary and ternary weight representations trained end-to-end:

  • Bonsai-27B-Q1_0.gguf (1-Bit Binary):

    • Native Format: Q1_0_g128 ({-1, +1} weights with FP16 group-wise scaling every 128 weights).
    • Effective Bit Width: 1.125 bits/weight (1 sign bit + 16-bit scale / 128).
    • Deployed Footprint: ~3.8 - 3.9 GB (~14.2x smaller than FP16 baseline at ~54 GB).
    • Intelligence Retention: Retains 89.5% of FP16 benchmark performance (76.11 average across 15 thinking-mode benchmarks).

  • Ternary-Bonsai-27B-Q2_0.gguf (Ternary / 2-Bit):

    • Native Format: Q2_0_g128 ({-1, 0, +1} weights in 2-bit container slots with FP16 group-wise scaling).
    • Effective Bit Width: 1.71 bits/weight theoretical (2.125 bpw deployed in current GGUF container).
    • Deployed Footprint: ~7.2 - 7.5 GB (~9.4x reduction vs FP16).
    • Intelligence Retention: Retains 94.6% of FP16 benchmark performance (80.49 average across 15 thinking-mode benchmarks), outperforming conventional IQ2_XXS (72.73) at less footprint.

B. The Hybrid-Attention Advantage in KV Cache

Bonsai 27B is derived from the Qwen3.6-27B architecture, which utilizes a hybrid-attention backbone (~75% linear attention, ~25% full attention across 64 blocks).

Because only 16 out of 64 layers grow a full-attention KV cache, context memory growth is reduced by 75% compared to standard transformer architectures. Combined with Q4_0 KV-cache quantization, the memory needed for a 262K context window drops from ~17.2 GB down to just ~4.3 GB.

C. Intelligence Density Comparison

Intelligence density ($D$) measures the ratio of model benchmark capability to its deployed footprint in gigabytes:

D = -log2(1 - Score / 100) / Size_GB
Model Variant Deployed Size Benchmark Avg (Thinking Mode) Intelligence Density (1/GB)
1-bit Bonsai 27B (Q1_0) 3.9 GB 76.11 0.530
Ternary Bonsai 27B (Q2_0) 5.9 GB (ideal) / 7.2 GB 80.49 0.400
Qwen3.6-27B IQ2_XXS 9.4 GB 72.73 0.199
Gemma-4-31B Q2_K_XL 11.8 GB 73.31 0.162
Qwen3.6-27B Q4_K_XL 17.6 GB 84.99 0.155
Qwen3.6-27B FP16 54.0 GB 85.07 0.0513

Hardware Testbed & Software Setup

All empirical benchmarks were performed on host workstation beast:

  • Host System Name: beast
  • GPU: NVIDIA Quadro RTX 4000 (8,192 MiB GDDR6 / ~7,795 MiB usable, CUDA CC 7.5 Turing)
  • GPU Power Target: Manual 100W cap (power saving) vs 125W stock rating
  • CPU: AMD Ryzen 9 7945HX with Radeon Graphics (16 Cores / 32 Threads, AVX-512 Enabled)
  • System RAM: 96 GB DDR5-5600
  • Inference Engine: llama.cpp (llama-server PrismML custom low-bit kernel build)
  • OS / CUDA / Driver: Linux / CUDA 13.3 / NVIDIA Driver 610.43.02
  • Frontend Client: Open WebUI (via OpenAI API compatible endpoint)

Empirical Benchmarking Results

1-Bit Binary Model (Bonsai-27B-Q1_0.gguf)

Context Size (-c) Power Cap GPU Layers (-ngl) KV Cache Gen Speed Prompt Eval VRAM Usage Operating Mode
65,536 100W (Manual Cap) 99 q4_0 22.06 tok/s 357.36 tok/s 5,260 MiB 100% GPU Resident
131,072 100W (Manual Cap) 99 q4_0 21.87 tok/s 355.47 tok/s 6,732 MiB 100% GPU Resident
163,840 100W (Manual Cap) 99 q4_0 20.61 tok/s 347.01 tok/s 7,472 MiB 100% GPU Resident
163,840 125W (Stock) 99 q4_0 27.35 tok/s 420.24 tok/s 7,468 MiB 100% GPU Resident (Unlocked)
262,144 100W (Manual Cap) 45 q4_0 12.93 tok/s 336.93 tok/s 7,612 MiB Hybrid Max Context (45 GPU / 19 CPU)

Ternary Model (Ternary-Bonsai-27B-Q2_0.gguf)

Context Size (-c) Power Cap GPU Layers (-ngl) KV Cache Gen Speed Prompt Eval VRAM Usage Operating Mode
16,384 100W (Manual Cap) 99 q8_0 17.10 tok/s 349.12 tok/s 7,472 MiB 100% GPU Resident
32,768 100W (Manual Cap) 99 q4_0 15.10 tok/s 316.42 tok/s 7,588 MiB 100% GPU Resident
32,768 125W (Stock) 99 q4_0 18.04 tok/s 356.78 tok/s 7,608 MiB 100% GPU Resident (Unlocked)
65,536 100W (Manual Cap) 59 q4_0 10.92 tok/s 238.59 tok/s 7,612 MiB Hybrid Mode (59 GPU / 5 CPU)

Energy Efficiency & Power Scaling Analysis

I had originally capped the GPU at 100W to lower overall workstation power draw. However, analyzing the math revealed an optimal behavior when restoring stock TDP:

E_token = P_TDP / Throughput_tok/s  [Joules/token]
E_mWh   = E_token / 3.6

Comparative Energy Profile Table

Model & Setting Power Limit (P) Throughput (T) Energy / Token (E) Energy / Token (mWh) Efficiency Verdict
Q1_0 (160K Context) 100W (Manual Cap) 20.61 tok/s 4.85 J/token 1.35 mWh/token Power Throttled
Q1_0 (160K Context) 125W (Stock) 27.35 tok/s 4.57 J/token 1.27 mWh/token +5.8% MORE Efficient!
Q2_0 (32K Context) 100W (Manual Cap) 15.10 tok/s 6.62 J/token 1.84 mWh/token Energy Savings Cap
Q2_0 (32K Context) 125W (Stock) 18.04 tok/s 6.93 J/token 1.92 mWh/token -4.7% cost for +19.5% responsiveness

Key Technical Takeaways on Energy

  1. Unlocking 125W Stock in 1-Bit Mode is MORE Energy Efficient: At 100W, memory bandwidth and tensor core clocks were power-throttled. Restoring 125W stock (+25% power) yielded a +32.7% throughput jump (20.61 to 27.35 tok/s). Because throughput scaling outpaced power target scaling, energy per token decreased from 4.85 J to 4.57 J.
  2. Solid Trade-off in Ternary Mode (Q2_0): Unlocking to 125W stock produced a +19.5% boost in speed (15.10 to 18.04 tok/s) with a minor +4.7% energy increase per token (1.84 to 1.92 mWh). For interactive chat responsiveness, the speed gain easily justifies the tiny energy difference.

Preset 1: Maximum Speed & Context (160K Context @ 125W — 100% GPU)

Best for: Fast coding assistance, daily chat, zero-latency interaction.

./build/bin/llama-server \
  -m Bonsai-27B-Q1_0.gguf \
  --host 0.0.0.0 --port 11433 \
  -c 163840 -np 1 \
  -ngl 99 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  -rea on

Preset 2: Maximum Intelligence Quality (32K Context @ 125W — Ternary Q2_0)

Best for: Complex math, logic reasoning (94.6% FP16 baseline retention), formal document synthesis.

./build/bin/llama-server \
  -m Ternary-Bonsai-27B-Q2_0.gguf \
  --host 0.0.0.0 --port 11433 \
  -c 32768 -np 1 \
  -ngl 99 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  -rea on

Preset 3: Context Analysis (262K Context — Hybrid GPU/CPU Mode)

Best for: Multi-file repositories, entire books, huge PDF document batches.

./build/bin/llama-server \
  -m Bonsai-27B-Q1_0.gguf \
  --host 0.0.0.0 --port 11433 \
  -c 262144 -np 1 \
  -ngl 45 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  -rea on

Final Reflection: Democratizing Local AI & The Era of Efficiency

Beyond the immediate technical milestone, extreme quantization formats like 1-bit and 2-bit (ternary) hit a core principle: technological sovereignty and the democratization of Artificial Intelligence.

I am a staunch advocate for local, open, and free AI. However, given current hardware pricing and scarcity (where enterprise-grade GPUs with huge VRAM pools are priced out of reach), running truly intelligent models locally had become an exclusive luxury. Extreme low-bit formats flip the board entirely.

AI is undergoing the exact same historical evolution as the steam engine during the early Industrial Revolution. Emerging technologies start out brute, inefficient, and resource-hungry while industry scrambles to push maximum raw scale. We saw the exact same arc with CPUs and GPUs—brute-force eras eventually gave way to an architectural obsession with efficiency per watt. AI is finally reaching that crucial second stage.

If 1-bit and ternary quantization gain the industry momentum they deserve, the future of local AI won’t require unaffordable corporate GPUs or absurd VRAM limits. Enterprise-level intelligence can be decentralized onto modest consumer hardware.

On paper, Qwen3.6 27B is exceptional. Now that it is serving locally at truly fluid speeds, real-world stress testing begins across my agent fleet (Hermes, Pi, OpenCode, OpenClaw) to see whether this model proves genuinely usable and reliable in daily production workflows.