← Back to Garden

Bonsai 27B + Hermes Agent: Fixing the llama.cpp JSON Schema Parser Bug

•
#LLM#1-Bit#PrismML#GGUF#llama.cpp#Hermes#Agent#MCP#Tool-Calling#Proxy#Debug

DEBUG_LOG // BONSAI_27B // HERMES_AGENT // LLAMA_CPP // PROXY_FIX

After benchmarking Bonsai 27B on an 8GB GPU and achieving 27.35 tok/s at 160K context, the next logical step was integrating it into my production agent workflow: Hermes Agent, the framework that powers my daily AM/PM infrastructure reports, email triage, SRI tax invoicing, and GitHub monitoring.

The model worked flawlessly for plain text chat via hermes -p bonsai_27b. But as soon as I tried running it through Hermes’ cron job scheduler — which injects MCP server tool schemas (hledger, email, facturador-sri, payphone) — everything collapsed with an opaque HTTP 400.

The Error

HTTP 400: Unable to generate parser for this template.
Automatic parser generation failed: JSON schema conversion failed:
Error resolving ref #/properties/begin/anyOf/0: anyOf not in 
{"type":"string","pattern":"^\\d{4}(-\\d{2}(-\\d{2})?)?$","nullable":true}

The message repeated 16 times — once for every MCP tool in the request. Hermes was sending 119 tools with ~170KB of JSON schemas, and llama-server’s internal grammar-based constrained generation parser couldn’t handle them.

Diagnosis: Four Layers Deep

Layer 1 — --jinja Was a Red Herring

My first instinct was to remove --jinja from the llama-server flags. The Jinja template engine parses chat templates, and I assumed it was conflicting with Hermes’ tool schemas.

Result: Error persisted identically. --jinja controls chat message formatting, not JSON schema grammar generation. Different subsystems entirely.

Layer 2 — anyOf in Email Tool Schemas

I built a debug proxy between Hermes and llama-server to inspect the actual request payload. The sanitizer caught anyOf blocks in email tool parameters (to, cc, bcc — fields that accept both array and string):

"to": {
  "anyOf": [
    {"items": {"type": "string"}, "type": "array"},
    {"type": "string"}
  ]
}

Stripped those. Still failed.

Layer 3 — The $ref Ghost

Inspecting the raw 170KB request revealed the real culprit. Hermes’ own built-in schema_sanitizer had already cleaned up the hledger tool’s begin field — replacing its anyOf with a flat type: string. But the end field still had a dangling $ref pointing to the now-deleted anyOf:

"begin": {
  "type": "string",
  "pattern": "^\\d{4}(-\\d{2}(-\\d{2})?)?$",
  "nullable": true
},
"end": {
  "$ref": "#/properties/begin/anyOf/0",
  "nullable": true
}

llama-server chased that $ref, found no anyOf at the destination, and threw the error. Dangling $ref to sanitized schema paths.

Layer 4 — MCP Servers Are Loaded Profile-Wide

Even with enabled_toolsets=[] on the cron job, MCP servers defined in config.yaml (hledger, email, facturador-sri, payphone, contífico) inject their tool schemas into the system prompt regardless. There’s no per-job MCP filtering — they’re global to the profile.

The Fix: Proxy Sanitizer

A lightweight Python proxy sits between Hermes and llama-server, sanitizing the tool schemas on every request. It strips both anyOf blocks (flattening them to their first non-null type) and $ref pointers (replacing them with a safe type: string fallback).

Proxy Code

#!/usr/bin/env python3
"""Proxy sanitizer for llama.cpp: strips anyOf and $ref from tool schemas."""
import json, http.server, urllib.request

TARGET = "http://<llama-server-host>:11433/v1/chat/completions"
PORT = 11434

def sanitize(obj):
    if isinstance(obj, dict):
        if "anyOf" in obj:
            types = [t.get("type") for t in obj["anyOf"]
                     if isinstance(t, dict) and t.get("type") != "null"]
            if types:
                obj["type"] = types[0]
            del obj["anyOf"]
        if "$ref" in obj:
            del obj["$ref"]
            obj.setdefault("type", "string")
        for v in obj.values():
            sanitize(v)
    elif isinstance(obj, list):
        for item in obj:
            sanitize(item)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        try:
            data = json.loads(body)
            sanitize(data)
            cleaned = json.dumps(data).encode("utf-8")
        except Exception:
            cleaned = body
        req = urllib.request.Request(
            TARGET, data=cleaned,
            headers={"Content-Type": "application/json"}
        )
        try:
            with urllib.request.urlopen(req, timeout=300) as resp:
                self.send_response(resp.status)
                for k, v in resp.headers.items():
                    self.send_header(k, v)
                self.end_headers()
                self.wfile.write(resp.read())
        except urllib.error.HTTPError as e:
            self.send_response(502)
            self.end_headers()
            self.wfile.write(
                json.dumps({"error": f"HTTP Error {e.code}: {e.reason}"}).encode()
            )
    def log_message(self, *args):
        pass

http.server.HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()

Systemd Service

[Unit]
Description=Bonsai 27B Proxy Sanitizer
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/bin/python3 /root/.hermes/profiles/bonsai_27b/proxy_sanitizer.py
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Hermes Config Update

custom_providers:
  - name: bonsai-27b
    base_url: http://127.0.0.1:11434/v1   # ← proxy, not direct
    api_key: llama.cpp
    model: Bonsai-27B-Q1_0.gguf

Architecture Diagram

┌──────────┐    170KB JSON + 119 tools     ┌────────────┐    sanitized    ┌─────────────┐
│  Hermes  │ ────────────────────────────→ │   Proxy    │ ─────────────→ │ llama-server │
│  Agent   │ ←──────────────────────────── │ :11434     │ ←───────────── │ :11433       │
└──────────┘    text/stream response       └────────────┘  text/stream   └─────────────┘
                                                  │
                                    strips anyOf + $ref from
                                    all tool schemas in real-time

Hermes + Bonsai: What Works Now

Use Case Status Notes
Plain text chat (bonsai_27b chat -q) ✅ Works No tools = no schema conflict
Cron jobs without MCP ✅ Works enabled_toolsets=[] + proxy
Cron jobs WITH MCP (email, hledger, SRI) ✅ Works Proxy strips problematic schemas
Interactive agent with full tools ⚠️ Partial Works if tools don’t generate anyOf or dangling $ref

Root Cause Upstream

This is ultimately a llama.cpp issue — the json_schema_to_grammar converter in llama-server doesn’t handle anyOf keywords or $ref resolution in tool call schemas. The workaround documented here is a client-side proxy until the upstream parser gains support for anyOf and proper $ref resolution.

Tracking issue candidate: ggml-org/llama.cpp — search for “Unable to generate parser JSON schema conversion anyOf”.

Key Takeaway

If you’re trying to pair a reasoning model (Bonsai, Qwen3, DeepSeek) served via llama-server with a complex agent framework (Hermes, OpenCode, CrewAI) that has MCP tools with rich JSON schemas, you will hit this wall. The proxy sanitizer pattern is generic — replace the tool schemas on the wire before they reach llama.cpp, and the model works seamlessly.

Full source and systemd unit available in my homelab configs.