Eckart–Young says the truncated SVD is the optimal low-rank approximation. This post turns that theorem into engineering — factorized layers, truncated-SVD compression, LoRA and its descendants QLoRA, DoRA, and GaLore, and nuclear-norm minimization for matrix completion.
Every matrix, no matter how ugly, factors into a rotation, a scaling, and another rotation. That single fact — the singular value decomposition — is the engine behind PCA, image compression, latent semantic analysis, and the low-rank tricks that let us fine-tune giant models cheaply.
Eckart–Young says the truncated SVD is the optimal low-rank approximation. This post turns that theorem into engineering — factorized layers, truncated-SVD compression, LoRA and its descendants QLoRA, DoRA, and GaLore, and nuclear-norm minimization for matrix completion.
Every matrix, no matter how ugly, factors into a rotation, a scaling, and another rotation. That single fact — the singular value decomposition — is the engine behind PCA, image compression, latent semantic analysis, and the low-rank tricks that let us fine-tune giant models cheaply.