Blog Post··6 min read
Cheatsheet: Soft Metrics
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Tag: evaluation
Blog Post··6 min read
Cheatsheet: Soft Metrics
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Blog Post··8 min read
Evaluation Metrics: Precision, Recall, Calibration, and Confidence
How do you measure whether a model is actually good? The answer is a set of metrics — precision, recall, F1, perplexity, calibration, confidence intervals — each measuring something different and failing in a different way.
Blog Post··5 min read
Red-Teaming vs Automated Evals: Tradeoffs and When to Use Each
Human red-teaming finds attacks automated evals miss. Automated evals achieve scale humans can't. Here's how to combine them, and what each can and can't tell you.
Blog Post··6 min read
Designing Safety Benchmarks for LLMs: What Makes an Eval Good
Most safety benchmarks are gameable, distribution-shifted, or measure the wrong thing. Here's what separates a rigorous safety evaluation from a checkbox.
Blog Post··8 min read
Evaluation Metrics: Precision, Recall, Calibration, and Confidence
How do you measure whether a model is actually good? The answer is a set of metrics — precision, recall, F1, perplexity, calibration, confidence intervals — each measuring something different and failing in a different way.
Blog Post··5 min read
Red-Teaming vs Automated Evals: Tradeoffs and When to Use Each
Human red-teaming finds attacks automated evals miss. Automated evals achieve scale humans can't. Here's how to combine them, and what each can and can't tell you.
Blog Post··6 min read
Designing Safety Benchmarks for LLMs: What Makes an Eval Good
Most safety benchmarks are gameable, distribution-shifted, or measure the wrong thing. Here's what separates a rigorous safety evaluation from a checkbox.