Least Squares: Closed Form, QR, SVD, Gradient Descent, and Ridge Regression

March 18, 2026  ·  optimization

This blog gives a unified overview of five common methods for solving least-squares problems: the normal equations, QR decomposition, SVD, gradient descent, and ridge regression. We focus on the main issues that distinguish them in practice, including numerical stability, rank deficiency, minimum-norm solutions, and robustness to noise.