The Bottleneck Was Not the Model. It Was the Contract | Home Assistant Specialist v0.5

TOOL CALLING // CONTEXT BUDGET // LOCAL SPECIALIST

We did not want another chatbot spitting out Markdown blocks or conversational JSON structures pretending to be tools. We wanted a small, deterministic, highly specialized engine that understands the native operating contract of Home Assistant and produces direct function calls that the home automation bus can execute cleanly.

The development of Home Assistant Specialist v0.5 started by shattering a flawed assumption: the core limitation of small models in control tasks is not parameter size, but the nature of the contract we force them to learn.

1. Abandoning Artificial Compatibility: The Native Contract

Previous iterations attempted to mimic tool calling using embedded JSON blocks inside standard conversational turns. This approach introduces syntax friction, hallucinations, and forces the model to process unnecessary layers of abstraction.

In v0.5, we made a radical shift: drop conversational compatibility and implement the native protocol of Qwen3 alongside Hugging Face Chat Templating specs:

Strict role structure: system, user, assistant (with explicit tool_calls), and tool response roles.
Formal schema definitions inside the prompt’s tools payload.
Loss masking (train_on_turn) so the optimizer updates gradients strictly on the assistant’s output turns, preventing the model from memorizing system prompts or environment outputs.

When you change the contract, you change the object you are training. You are no longer teaching a model to discuss home automation; you are tuning an execution relay.

2. Dataset V5: The Rigor of Clean Data

A specialized model is only as good as its fine-tuning dataset. The private V5 dataset was engineered with strict filtering to strip out structural noise:

7,004 training examples (6,779 effective post-tokenization).
607 validation examples (588 effective).
667 test suite examples.
2,588 examples rejected for schema corruption or signature mismatches.
Hard limit of 8 exposed entities per conversation turn.

Filtering down to 8 entities is not meant to obscure smart home complexity, but to isolate variables. In my homelab, I have over 100 active entities. Proving that the model masters syntax and argument groundings on 8 entities baseline is a mandatory control step before scaling entity density.

3. The Context Budget: From 2048 to 4096 Tokens

Our initial technical intuition was to limit maximum sequence length (MAX_SEQ_LENGTH) to 2048 tokens to maximize training speed and lower memory overhead. That was a measurement error.

Native tool schemas (tools) consume a significant portion of the input prompt. Real-world profiling showed that valid conversation rows spanned between 2,700 and 3,300 tokens. Capping sequences at 2048 tokens resulted in truncated assistant tool calls.

Operational conclusion: MAX_SEQ_LENGTH=4096 is not a luxury; it is the minimum functional floor required to hold tool schemas, context, and complete assistant calls. Restricting entity count reduces payload, but it cannot replace measuring actual schema overhead.

4. Hardware & Real DDP Verification on Kaggle

Distributed training (DDP) was executed across dual NVIDIA Tesla T4 GPUs on Kaggle Notebooks:

HARDWARE T4x2: OK
rank 0: startup RANK=0 LOCAL_RANK=0 WORLD_SIZE=2 | GPU Tesla T4
rank 1: startup RANK=1 LOCAL_RANK=1 WORLD_SIZE=2 | GPU Tesla T4

A critical observation during startup with Unsloth: the library prints Num GPUs used = 1 per worker process. A naive glance might suggest a fallback to single-GPU mode. The true confirmation of parallelism comes from global runtime metrics: WORLD_SIZE=2, active LOCAL_RANK across both processes, and Data Parallel GPUs=2.

The 3-step smoke test confirmed pipeline integrity:

global_step=3
training_loss=3.9789
train_runtime=203.36 s

5. Compute Economics and the 1-Epoch Rule

The initial plan for 2 epochs (848 steps) using QLoRA on Qwen3-4B (33M trainable parameters out of 2.5B total) yielded a throughput of 66.6 seconds per step, projecting a ~15.7-hour total runtime.

Given Kaggle’s 12-hour continuous execution limit per session, the 2-epoch run was canceled at step 14/848. This was not a failure of the model, but a conscious compute resource decision.

We reconfigured the job to NUM_TRAIN_EPOCHS=1 (424 steps, ~7.85 hours). To guard against session timeouts or disconnected runtimes, we configured an automated checkpoint pipeline that pushes weights directly to Hugging Face Hub every 50 steps (~55 minutes of compute). If Kaggle drops the session, progress is safe without losing GPU hours.

6. Local Hardware Strategy & Micro-Model Vision

In my local environment, inference runs on a dedicated NVIDIA Quadro RTX 4000 8GB VRAM, while embedding workloads are offloaded to a secondary Ollama instance on CPU (port 11435).

Choosing Qwen3-4B Q5 provides a comfortable context window of up to 16k tokens on 8GB VRAM. The long-term roadmap, designed alongside Tuxbot (our frontier agent), uses v0.5 as a syntactic baseline to step down toward even leaner micro-models:

Qwen3-1.7B Q4: extends context window capacity up to 28k tokens on the RTX 4000.
Qwen3-0.6B Q4/Q5: expands context up to 32k tokens without spilling a single megabyte into system RAM.

Scaling down model size is not a downgrade; it unlocks the budget needed to process real-world home state (100+ entities) without choking VRAM or sacrificing latency.

7. Human-Agent Workflow & Honest Project State

This project reflects the real dynamic between product vision and execution engineering:

Human (Sebastian / tuxevil): Product architecture, compute economics, physical VRAM limits (8GB), homelab entity density test design.
Agent (Tuxbot): Fine-tuning pipeline engineering, loss mask verification, DDP setup, and test suite design.

Honest Technical & Editorial State

1-Epoch Training Run: Currently active and monitored via Hugging Face checkpoints.
Benchmarks & Quality Gate: Pending post-run evaluation against the 667-example test set.
GGUF Export & FP16 Merge: Pending syntax accuracy and zero argument hallucination verification.

Real technical progress is not measured by raw parameter count, but by contract precision and compute resource discipline.