Blog Post··6 min read
Cheatsheet: Soft Metrics
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Tag: entropy
Blog Post··6 min read
Cheatsheet: Soft Metrics
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Blog Post··6 min read
Information Theory for LLMs: Mutual Information, Entropy, and What Models Learn
Information theory gives precise answers to questions like: how much does the context tell you about the next token? What information is preserved in a representation? Why does compression and prediction point to the same objective?
Blog Post··6 min read
Information Theory for LLMs: Mutual Information, Entropy, and What Models Learn
Information theory gives precise answers to questions like: how much does the context tell you about the next token? What information is preserved in a representation? Why does compression and prediction point to the same objective?