Tag: llm-evaluation

Blog Post·2026-07-17·7 min read

Chat & Arena Benchmarks: Human Votes Scale, MT-Bench Doesn't

MT-Bench's frontier models cluster above 9.0 — the scale is out of room. AlpacaEval 2.0's length-controlled win rate is now saturated above 95% for top models. LMArena Elo keeps separating models as long as votes keep coming in — and they do.

llm-evaluation benchmarks chatbot-arena mt-bench

Blog Post·2026-07-17·6 min read

Creative Writing Benchmarks: When the Rubric Runs Out of Signal

EQ-Bench CW v3 rubric scores are already saturated at the top — a 0.35-point spread across 10 models. Elo still discriminates. Here's what that gap reveals about how we evaluate creative writing.

llm-evaluation benchmarks creative-writing

Blog Post·2026-07-17·6 min read

Formatting & Output Validation Benchmarks: IFEval Is Saturated, IFBench Isn't

IFEval's top-5 span fewer than 2 percentage points — frontier models have converged on its constraint set. IFBench exposes a 29pp gap between Grok and Claude on out-of-distribution constraints. And SOB shows that JSON schema compliance is not the same as correct field values.

llm-evaluation benchmarks formatting structured-output

Blog Post·2026-07-17·5 min read

Instruction-Following Benchmarks: IFEval Is Saturated, IFBench Exposes What's Left

IFEval's top-10 spread is 2.9pp — six of the top 12 spots go to Qwen3.5 variants. IFBench then shows a 29pp gap between reasoning models and standard instruction-tuned ones on constraints none of them trained against.

llm-evaluation benchmarks instruction-following

Blog Post·2026-07-17·6 min read

Reward Model Benchmarks: RewardBench Is Saturated, RewardBench 2 Isn't

RewardBench v1's top-6 spread is 5.7 points — small specialist models now dominate it. RewardBench 2 drops scores by 20 points and actually correlates with downstream RLHF. RM-Bench finds that style bias can push SOTA models below random performance.

llm-evaluation benchmarks reward-models rlhf