SLMarena: Local Model Arena

EVIDENCE OVER HYPE // LOCAL MODELS // LLM_AS_A_JUDGE

SLMarena is a self-hosted benchmarking and red-teaming platform for Small Language Models running on local Ollama. It replaces manual response review with repeatable scenarios, granular telemetry, and automated evaluation.

The core loop is simple: a local model answers standardized scenarios, an OpenAI-compatible frontier judge scores grammar, compliance, accuracy, and security resilience, and the results land on an interactive arena leaderboard.

What it brings

• Matrix orchestration: N models × M scenarios with multi-sample repetition (1–10 runs per model).
• Granular inference telemetry: TTFT, tok/s, latency, thinking tokens, prompt and output token counts per sample.
• LLM-as-a-Judge: Structured JSON verdicts with star ratings, qualitative feedback, and vulnerability breakdown.
• Security testing framework: Canary token injection and 8 attack vectors (instruction override, system prompt leakage, indirect prompt injection, delimiter hijacking, context overstuffing, encoding obfuscation, tool parameter hijacking, refusal suppression).
• Live monitor: SSE token-by-token streaming, Ollama server status, VRAM usage, queue controls.
• Anomaly detection: Empty responses, stalled evaluations, and TPS outliers surfaced automatically with one-click re-evaluation.
• Dual deployment: Zero-config SQLite mode or durable PostgreSQL + Redis/BullMQ worker architecture.
• MCP server: Agents (e.g. Hermes) can read metrics, create scenarios, and orchestrate matrix runs programmatically.

Architecture

system prompt + user messages
              ↓
      local model under test
              ↓
            response
              ↓
      frontier LLM evaluator
              ↓
  JSON + metrics + detected errors

Public leaderboard: slmarena.tuxevil.com