Gradient descent

Gradient descent is an iterative optimization algorithm that finds the minimum of a function by repeatedly moving in the direction of steepest descent. Imagine standing on a foggy hillside: you can't see the valley, but you can feel which way slopes downward beneath your feet. Gradient descent works similarly, calculating the gradient (slope) and taking small steps downhill until reaching a low point.

The method is foundational in machine learning and Artificial General Intelligence, where it adjusts model weights to minimize error during training. At each iteration, the algorithm computes how much each parameter should change to reduce loss, scaled by a "learning rate" that controls step size. Too large, and you overshoot the minimum; too small, and training crawls forward.

Variants like stochastic gradient descent and momentum-based methods improve speed and reliability. The technique assumes smooth, differentiable functions—rough terrain defeats it. Success depends on objective function design and careful tuning of hyperparameters.

Though born from Trigonometry and classical mechanics, gradient descent became indispensable when deep learning exploded. It's both beautifully simple and deceptively subtle: finding good minima in high dimensions remains an active research frontier.

Related

Optimization, Convergence, Loss function, Backpropagation, Convex optimization, Stochastic processes