All systems operationalUpdated Sep 25, 3:20 PM

Overview

Production-inspired LLM evaluation platform — deterministic scoring, experiment tracking, and release gates

Try a live eval run

Scores 3 curated sample answers with 5 deterministic heuristic evaluators — no setup needed

Pick a scenario and click Run — scored by deterministic keyword, overlap, and groundedness evaluators.

Quality Score Trend

Last 7 eval runs

84%

latest · 84% avg

Run	Dataset	Score
#1	verify-persistence-across-deploy	100%
#2	demo-general-knowledge	81%
#3	demo-general-knowledge	81%
#4	demo-general-knowledge	81%
#5	demo-general-knowledge	81%
#6	demo-code-review	82%
#7	demo-customer-support	84%

Evaluator profiles

Five deterministic signals — no external API required. LLM judge adapters layer on top for deeper semantic scoring.

Keyword Match

Checks for required terms

Reference Overlap

ROUGE-style token similarity

Rubric Coverage

Criterion-by-criterion scoring

JSON Structure

Schema validation for structured output

Groundedness

Lexical groundedness proxy — token overlap with reference

🧪

Experiment tracking

Compare prompt versions and model configs side by side

🚦

Release gates

PASS/FAIL CI signal that blocks regressions before deploy

⚡

Async eval API

In-process background jobs by default — Redis-backed queue available for multi-worker deployments

📊

MLflow tracking

Every eval run and comparison logged automatically for experiment history

🔌

LLM judge adapters

OpenAI · Anthropic · Mistral adapters — with deterministic heuristic fallback

StackFastAPISQLAlchemySQLiteNext.js 16DockerGitHub ActionsView source →

Dataset	Model	Score
demo-customer-support	heuristic-demo	84%
demo-code-review	heuristic-demo	82%
demo-general-knowledge	heuristic-demo	81%
demo-general-knowledge	heuristic-demo	81%
demo-general-knowledge	heuristic-demo	81%
demo-general-knowledge	heuristic-demo	81%
verify-persistence-across-deploy	heuristic-demo	100%