How a model finds a better set of settings
A machine-learning model makes predictions using adjustable numerical settings called parameters. During training, it compares a prediction with the desired answer using a loss function—a calculation that assigns a larger value to worse errors. The loss turns “how wrong was the model?” into a number the training process can try to reduce.
For each training step, the algorithm estimates how changing each parameter would affect that loss. These sensitivities form a gradient: a map of which adjustments would raise the error and which would lower it. The algorithm nudges the parameters in the direction that is expected to reduce the loss, then evaluates the model again.
Why the downhill picture has many dimensions
With just two parameters, loss could be drawn as a landscape with hills and valleys. Real models may have millions or more parameters, so their loss landscape cannot be pictured as an ordinary surface. “Downhill” means moving through this vast space of possible parameter settings toward settings that score better—not moving through physical space.
The size of each nudge matters. A step that is too large can overshoot a low-loss region; one that is too small can make training painfully slow. Training therefore repeats the calculation and adjustment, often using small groups of examples at a time. Each group gives the model a fresh estimate of which way to move.
What the process can—and cannot—promise
Lower training loss means the model is fitting the examples it has seen more closely. It does not automatically mean it will perform well on new examples: a model can learn the training cases too specifically. Developers therefore also check performance on data held aside from training.
The practical result is a gradual search, not a single leap to the best possible answer. Every adjustment depends on the model’s current settings and the examples used for that step, so the route can be uneven even when the overall aim is clear: make the model’s errors smaller.
