SLMarena: How I Chose the Local Model Now Powering Tuxbot

SLMARENA // LOCAL MODELS // EVIDENCE OVER HYPE

I did not start SLMarena because I needed another model leaderboard. I started because my chatbot was writing ugly Spanish.

My portfolio and personal blog have a chatbot connected to an artificial and very limited representation of my agent. To keep the layers distinct, I call it Mini Tuxbot.

My real Tuxbot is my augmenting copilot: it runs on Hermes Agent and operates as a second layer able to remember, inspect, execute, and return structure. The website chatbot is not that agent. Mini Tuxbot is a constrained, public, conversation-only version built so visitors can explore my work.

Mini Tuxbot uses my LiteLLM proxy to access local models and answer questions about me, my projects, and my ideas using public information from the site.

The intent was simple: let a visitor ask a question and discover connections hidden across many pages. They could dig into a project, jump to a related article, understand my philosophy better, or decide to contact me.

The interface was conversational. The problem was that some answers sounded like they had been written by an American learning Spanish. The information could be useful, but the illusion died as soon as I read the text.

The real problem was not just model size

The first version used qwen3:0.6b. Then I tried qwen3:1.7b, qwen3:4b, and several Qwen3.5 variants: 0.8b, 2b, and 4b.

The pattern was obvious enough: as models grew, Spanish usually became more natural. But size did not explain everything. Grammar, precision, and instruction compliance also changed between models.

One model could write better but hallucinate more. Another could be accurate but ignore part of the system prompt. Another could answer quickly while producing prose no visitor would want to keep reading.

The question stopped being:

Which model is the smartest?

It became:

Which model answers best for this chatbot, with these scenarios, these standards, and this hardware?

The chatbot as a conversational index

Mini Tuxbot was not supposed to be a demo that answered one isolated question. It was supposed to be a discovery layer over my public presence:

visitor arrives at the site
        ↓
asks a broad question
        ↓
Tuxbot connects scattered information
        ↓
natural, trustworthy answer
        ↓
related article or project
        ↓
the visitor keeps exploring

To feed that flow, I built a function that appends information to a knowledge.txt file every time I publish a new article or project. That content is injected into the chatbot’s system prompt.

The site’s three preconfigured questions represent the main discovery paths:

“Tell me about the portfolio projects.”
“Tell me about the blog articles.”
“Tell me about Sebastian’s profile.”

They look simple. They are not. They force the model to read context, select information, synthesize it, preserve the tone, avoid inventing a biography, and open paths for continued exploration.

From reviewing answers to measuring evidence

At first I reviewed responses one by one. That works for finding a problem, but it becomes painful when comparing families, sizes, prompts, and configurations.

So I added a frontier model as an evaluator. The SLM generates the answer, and the evaluator receives:

the complete system prompt;
the real user messages;
the response generated by the SLM;
precise instructions for analyzing each objective.

The evaluator returns structured JSON with individual scores for:

grammar;
compliance;
accuracy;
overall;
the reasoning behind the score;
detected errors.

The architecture became:

system prompt + user messages
              ↓
         local model under test
              ↓
             response
              ↓
       frontier LLM evaluator
              ↓
   JSON + metrics + detected errors

I did not blindly trust the judge. Before using it to accelerate the process, I read responses and scores and compared its conclusions with my own judgement. For these tasks, frontier models delivered a sufficiently high level of analysis.

The cost was trivial too: less than one cent per result for the tests I was running. The bottleneck stopped being money and became scenario design and result comparability.

An LLM evaluator is not a universal authority. It can be wrong, favor long answers, or impose its own stylistic preferences. Its reasoning is evidence of the judge’s analysis, not mathematical proof that the score is true. But as a practical instrument, calibrated against human review, it multiplied the speed of the cycle.

SLMarena: a tool for discovering your own winner

Almost from the beginning, I decided this should not remain a private Tuxbot function. I tend to build first to remove a real friction, then open the result so other people can decide whether it is useful to them too.

That already happened with pi-antigravity-rotator: it started as a tool for me and, after becoming open source, gradually attracted more users.

SLMarena follows the same philosophy. It is a self-hosted platform for running scenarios against local Ollama models, capturing telemetry, and comparing results in a persistent interface.

The project includes:

TTFT, latency, and tokens-per-second measurements;
repeatable scenarios;
grammar, accuracy, and compliance evaluation;
prompt-injection resistance tests;
run persistence;
workers for processing evaluations;
a leaderboard and public snapshot;
a simple SQLite mode;
scalable PostgreSQL and Redis deployment.

Repository: github.com/tuxevil/SLMarena
Public leaderboard: slmarena.tuxevil.com

The conceptual references are not a single benchmark. Ollama provides the interface for running models locally. EleutherAI’s LM Evaluation Harness demonstrates the value of repeatable tasks and comparable metrics. OWASP LLM01 reminds us that a model must also be tested against inputs designed to alter its behavior.

SLMarena combines those concerns from another angle: it does not seek a universal truth about every model. It seeks enough evidence to make a defensible decision for a concrete application.

The result I did not expect

I started by testing the Qwen3 and Qwen3.5 families because they were the models I already knew. Systematic measurement changed my assumptions.

I discovered that:

Gemma 4 E2B scored better than Gemma 4 E4B in my scenarios;
LFM2.5 8B landed among the best, with extremely high response speed;
Nemotron 3 Nano 4B stayed close to that top group;
parameter count alone did not predict the final result.

The conclusion is not that Gemma 4 E2B is the best model in the world. The more useful conclusion is:

Gemma 4 E2B was the best model for my chatbot, under my scenarios, my standards, and my hardware.

That is why it now powers Mini Tuxbot, the limited website chatbot. It is not the model running my Hermes Agent copilot, Tuxbot.

The winning model depends on your boundary

Every user has a different boundary. Mini Tuxbot’s boundary includes natural Spanish, accuracy about my projects, system-prompt compliance, resistance to attacks, reasonable latency, and local execution through LiteLLM and Ollama.

Someone else may value code, mathematical reasoning, multilingual support, power consumption, or tool calling. There is no reason to expect the same ranking.

That is exactly where a local benchmark becomes useful. You can discover:

which model writes best in your language;
which one follows your instructions most reliably;
which one hallucinates least about your data;
which one resists attacks designed for your application;
what real speed you can expect on your machine;
which one offers the best trade-off between quality, speed, and power.

Public benchmarks remain useful. But when it is time to deploy a model, the important question is not which model won a distant table. It is which model wins inside your boundary.

SLMarena started with my reluctance to review answers one by one. It ended by turning that reluctance into an evidence tool. And that evidence changed the model I use for Mini Tuxbot, the public and limited representation of my agent.