Tag: attention-heads

Blog Post·2025-06-20·9 min read

What Each Transformer Component Actually Does

Attention heads as information-routing circuits, MLP layers as key-value memories, and the residual stream as a shared communication bus.

mechanistic-interpretability transformers attention-heads mlp-layers residual-stream

Blog Post·2025-06-20·9 min read

What Each Transformer Component Actually Does

Attention heads as information-routing circuits, MLP layers as key-value memories, and the residual stream as a shared communication bus.

mechanistic-interpretability transformers attention-heads mlp-layers residual-stream