The Model Is Not the Agent: Why 7 Fine-Tunings Led Me to Build a Deterministic Harness

AGENT HARNESS // DETERMINISTIC BOUNDARY // OPERATIONAL MEMORY

Training small language models is fascinating, but extended practice in production environments surfaces uncomfortable truths. During the development of Home Assistant Specialist (versions v0.2 through v0.5.2), the initial objective was clear: fine-tune a small, local 4B model (Qwen3-4B-Instruct) using Unsloth and PyTorch on platforms like Kaggle to natively understand Home Assistant tool calling.

Seven dataset iterations later, the metrics told a divided story: while tool-calling accuracy on positive cases reached 92.22%, the model’s ability to abstain from acting (safe rejection or negative boundary) plummeted from 60.32% in v0.5 down to 17.65% in v0.5.1 when attempting to rebalance the dataset.

The model learned the syntactic shape of the executor, but lost the judgement of when not to act. That was the turning point: language models are probabilistic by nature. Home Assistant and critical home infrastructure require deterministic behavior.

1. Metrics of the Collapse: The Illusion of Fine-Tuning

When fine-tuning an SLM for control tasks, it is easy to focus on loss curves or exact tool call scores. However, real-world deployment in a homelab requires strict auditing of false positives and abstention accuracy.

Strict Scorer Metric	Fine-Tuning v0.5	Fine-Tuning v0.5.1
Tool Call Exact (Positives)	92.22%	88.40%
Exact Arguments	94.70%	91.10%
Safe Rejection (Negatives)	60.32%	17.65%
Multi-Call Exact	80.00%	4.55%

In version v0.5.1, after tripling synthetic negative examples to balance the dataset (tuxevil/Home-Assistant-Requests-V5.2-Native-Strict), the model suffered from over-activation: it interpreted ambient questions as implicit execution commands or failed when chaining multiple tool calls.

Entrusting home automation safety to the probability distribution of a 4B parameter model means accepting that in a percentage of turns, the model will execute an arbitrary service due to syntactic hallucination.

2. The Pivot: Tuxbot HA Supervisor Architecture

The solution was not generating another 10,000 synthetic training pairs. The solution was moving the security boundary out of the LLM weights and into a deterministic harness written in code.

Enter Tuxbot HA Supervisor, an independent Python process that decouples language interpretation from physical execution:

HA Events / Periodic Review
           ↓
   Context Builder & History
           ↓
   Local Model (Ollama / Qwen3 / Bonsai)
           ↓
   Tuxbot HA Supervisor (Policy Engine & Action Registry)
           ↓
   [Guardrail: Energy/Infra System?] → YES → Hard Lockout / Approval Required
           ↓ NO
   Typed Action + Physical State Verification
           ↓
   Operational Memory Journal

Key Architectural Rules:

Typed Actions, Not Generic Execution: The model never calls generic call_service nor executes arbitrary scripts. It only emits typed intents validated against strict schemas by the supervisor.
Critical Infrastructure Isolation: Solar power systems, batteries, networks, and the computing nodes hosting the agent have hardcoded policy lockouts. The agent cannot cut power to its own host server.
Physical State Verification: Every executed action requires a state read-back from the Home Assistant API to verify that the physical entity actually toggled.

3. From “Junior” to “Senior”: Gradual Autonomy and Operational Memory

Rather than expecting a newly deployed model to operate with full autonomy on day one, the supervisor enforces a progression hierarchy similar to an onboarding engineer:

Level 1: Observer   → Detects anomalies and logs suggestions (zero physical actions).
Level 2: Recommender → Notifies user of intent and waits for confirmation.
Level 3: Reversible  → Executes minor actions (lights, switches) and notifies.
Level 4: Senior      → High autonomy based on consolidated operational memory patterns.

The Explicit Learning Loop

When the supervisor performs an autonomous action (e.g., toggling an office switch) and the human operator manually reverts the device shortly after, the supervisor does not guess the reason probabilistically.

The system issues a notification via Home Assistant Assist or direct channel asking for the specific reason behind the override. The user’s feedback is stored directly in the supervisor’s deterministic operational memory, preventing repeat errors without needing to re-tune model weights.

4. Conclusion: The Model Is the Tool, Not the Agent

The v0.5.2 dataset (tuxevil/Home-Assistant-Requests-V5.2-Native-Strict) on Hugging Face remains valuable: it now serves as our internal benchmark suite for testing local model tool-calling capabilities in Ollama.

The overarching lesson from 7 fine-tuning rounds is architectural: a language model is not the agent; it is the reasoning engine inside a deterministic software harness. The responsibility of keeping the home safe never belonged to loss gradients, but to code guardrails.