Gradient Descent
Iteratively move opposite the gradient.
Key idea
Iteratively move opposite the gradient.
In depth
Minimise f by repeatedly stepping against the gradient: xₙ₊₁ = xₙ − η∇f(xₙ). If η is small enough and f smooth, it converges to a local minimum. It trains most machine learning models.
Worked example
f(x) = x², η = 0.1, x₀ = 1: 0.8, 0.64, 0.512, ... → 0.
Formula
Real-world uses
- Training machine learning models and tuning settings to minimise an error
Common mistake
Using a learning rate that is too large, which diverges.
How to learn this
- Read the key idea and say it in your own words.
- Work through the worked example, then try a similar problem.
- Use the interactive lesson’s visual and mini-quiz to check the pattern.
Frequently asked questions
What is Gradient Descent?
Iteratively move opposite the gradient. It is a topic at the expert level in Differential Equations, Abstract Algebra & Applied Math.
What is the formula for Gradient Descent?
The key formula is xₙ₊₁ = xₙ − η∇f(xₙ).
Where is Gradient Descent used in real life?
Real-world examples: Training machine learning models and tuning settings to minimise an error.
What should I learn before Gradient Descent?
Make sure you are comfortable with Numerical Linear Algebra, the previous topic in Differential Equations, Abstract Algebra & Applied Math.
What is a common mistake with Gradient Descent?
Using a learning rate that is too large, which diverges.
How can I practise Gradient Descent?
Open the interactive lesson, play with the visual, then answer the three-question mini-quiz. Getting two right marks the topic complete.
What comes after Gradient Descent?
Move on to Convex Functions and Sets next.
Related expert topics in Differential Equations, Abstract Algebra & Applied Math
- Exact Equations – M dx + N dy = 0 with Mᵧ = Nₓ.
- Bernoulli Equations – Substitute v = y¹⁻ⁿ to linearise.
- Runge–Kutta Methods – Higher-accuracy numerical solvers.
- Second-Order Homogeneous Equations – Use the characteristic equation.
- Undetermined Coefficients – Guess the particular solution form.
- Variation of Parameters – General method for particular solutions.