LLM Evaluation Framework
Layered checks that show in CI what a prompt or model change made better or worse in two LLM features, a support assistant and ticket triage, replayed from real recorded calls so every run is free.
Results of the recorded run
| Metric | rag v1 | rag v2 | triage v1 | triage v2 |
|---|---|---|---|---|
| All checks | 62.2% | 82.7% | 57.5% | 92.5% |
| Retrieval layer | 81.8% | 81.8% | — | — |
| Deterministic layer | 78.2% | 94.2% | 97.5% | 97.5% |
| Reference layer | 79.8% | 83.8% | 57.5% | 92.5% |
| Safety layer | 88.5% | 98.7% | — | — |
| Judge layer | 65.0% | 77.5% | — | — |
| Safety cases with no safety failure | 8 of 12 | 11 of 12 | — | — |
| Stable cases | 76.9% | 92.3% | 87.5% | 95.0% |
| Cost (system + judge calls) | $0.027318 | $0.031958 | $0.005947 | $0.010457 |
| Latency p50 / p95 (system calls) | 1.3 s / 3.0 s | 1.2 s / 2.4 s | 2.5 s / 5.2 s | 1.5 s / 4.3 s |
Each column is one prompt version. Each case ran 3 times; each layer's rate is over the runs that layer checks (the judge graded the first run of each judged case). A safety case has no safety failure when every safety check passed on every run. The pairwise comparison calls are not counted in any column. Recorded on 2026-10-06 (UTC) with mistralai/mistral-small-3.2-24b-instruct (system) and nvidia/nemotron-3-super-120b-a12b (judge), 700 calls.
Latency: 18 of the 30 system calls at or above their column's p95 took at least twice as long as a repeat of the same request (a median of 3.4 times as long) while writing a median of 47 answer tokens against the repeats' 48, so most of the tail is delay at the provider, not time the request needs.
How to read the table: rag is the support assistant (retrieval-augmented generation: it searches the knowledge base, then answers), and triage turns a ticket into JSON. Retrieval: the search found the expected documents. Deterministic: rules a program can decide, such as valid JSON, citations, forbidden claims and length. Reference: the required facts, or the expected category, priority and order id. Safety: nothing leaked and no attack worked. Judge: a second model graded the answer as grounded, helpful and polite. Stable cases: every repeat got the same verdicts. p50 / p95: half the calls were faster than the first figure, and 95 percent were faster than the second.
The two RAG prompt versions, compared by the judge
The judge compared the first answers of rag v1 and rag v2 case by case, asked twice with the order of the two answers swapped.
| Outcome over 40 cases | Cases |
|---|---|
| v1 preferred in both orders | 3 |
| v2 preferred in both orders | 11 |
| A tie in both orders | 11 |
| Inconsistent: the two orders disagree | 11 |
| Identical answers, not compared | 4 |
Position consistency: 25 of 36 compared pairs (69.4%) got the same verdict in both orders.
In 11 of the 36 compared pairs, the judge's preference changed when the two answers swapped places: 1 time it chose the answer shown second in both orders, and 10 times it called a tie in one order and chose a side in the other. An inconsistent pair is never settled by picking one order.
With this many flips, the comparison says more about the judge's position bias than about the two prompts, so it picks no winner. The main table rests on the rules and the per-answer grades.
Can the judge be trusted?
pending human labels
None of the author's 30 labels in labels/human.jsonl is of an answer in the current sample: the answers or the sample changed after he labelled them. Stale labels are not used; each is listed with its reason in results/judge-agreement.json.
Pavel Zhukov Atum, the author, labels 30 judged answers by hand, blind to the judge's verdict (make label), and this sample awaits his labels: all 23 answers the judge failed and 7 it passed. The sample oversamples judge failures, so agreement on it is not the agreement over all answers.
The full report
Allure report: every case by function, layer and category, with the answers, the failed checks, the judge's verdicts and the pairwise comparison.
Allure's Behaviors tab groups the per-case tests by function and layer, and lists a passing case under every layer it was checked on. Its counts are tests, not layer pass rates: the layer tests (story "pass rate") give each layer's rate, as the table above does.