Evals ML — Model answers, measured.

Model answers, measured.

Evals ML
Independent · 2026

Isometric dark scene of a robotic arm picking sample blocks from a tray of candidates, lined up and fed into a glowing cyan test fixture, evoking pass@k sampling

Metrics

Pass@k Metrics Explained: Formula, Estimator, Pitfalls

Pass@k is the chance at least one of k samples is correct. How the unbiased estimator works, why pass@1 is the honest number, and when pass^k wins.

Read the article →

Latest guides

Isometric sci-fi assembly line moves glowing blue crystals through test stations, scales, scanners, and a secure deployment vault.

Eval Tooling

LLM Evaluation Frameworks Compared: Task Fit and CI Gates

Compare LLM evaluation tools for CI gates, reproducible runs, judge control, trace linkage and licensing, wit…

An isometric camera-equipped robotic arm hovers over a dark blue panel of rounded squares, circles, and sliders, reaching toward a glowing cyan disc and ring

Research

GUI-Primitives: Spatial Reasoning Failures in Vision Models

GUI-Primitives tests spatial reasoning in vision-language models for GUI grounding, finding weak localization…

Flat isometric illustration of a glowing orange icosahedron on a pink pedestal, linked by thin lines to six pink cube-topped blocks on a dark background.

Model Grading

LLM-as-a-Judge Bias: How to Detect and Correct It

Position, verbosity and self-preference bias in model graders, how to measure judge agreement against humans,…

Flat isometric illustration of a pale pink icosahedron at the centre of a pink node-and-line network with large pink slab blocks in the corners on dark navy.

Benchmarks

LLM Benchmarks Explained: MMLU, HumanEval, GSM8K

What MMLU, HumanEval and GSM8K actually contain, how each score is computed, and the point where a headline b…

Isometric dark slab with a glowing pink hexagon at its center linked by lines to scattered nodes, representing a model measured across an eval set

Eval Design

LLM Evaluation Design: From Task to Regression Suite

How to design an LLM evaluation: task-grounded eval sets, grader choice, sampling variance, and the contamina…

Start here

Evals ML is a reference for measuring language models: what the standard benchmark suites actually contain, how much confidence a published score carries, which harness fits which kind of task, and how to keep a model grader honest. Every figure quoted here is attributed to the paper or repository it came from.

Benchmarks

LLM benchmarks explained

MMLU, HumanEval and GSM8K: what is inside each set, how each is scored, and where the number stops being useful.

Methodology

LLM evaluation design

Building an eval set from the failures you care about, choosing a grader, and turning it into a regression suite that catches real change.

Tooling

LLM eval frameworks compared

lm-evaluation-harness, OpenAI Evals, HELM, Inspect, promptfoo, DeepEval and Ragas, compared on the task shapes each one fits.

Model grading

LLM-as-a-judge bias

Position, verbosity and self-preference bias, how to detect each, and how to calibrate a judge against human labels.

Pass@k and evaluation cost calculator

Generation count, token spend and worst-case margin of error for a given dataset size and sampling budget. Runs in the browser, no signup.

Open the calculator

All guides · Browse by topic · How these articles are researched

Evals ML: LLM evaluation methodology and benchmarks