New: AI Engineering Shortcuts — instant download • guest checkout Browse

AI / ML

Gradient Descent vs SGD vs Adam

  • 10 pages
  • Beginner
  • English
  • Updated Oct 2026

A 10-page visual beginner’s guide to Gradient Descent, SGD, mini-batch SGD, Adam and AdamW, with learning-rate guidance, momentum, real experiments, optimizer comparisons, and practical NumPy & PyTorch code.

4.99 €

Buy & Download PDF
  • Instant download after payment
  • Secure checkout (Stripe)
  • Download link sent to email

Product details

Language
English
Level
Beginner
Pages
10 pages
Last updated
Oct 2026

Digital product — no physical shipping

Download link available immediately after payment

Download link also sent to the buyer's email

What's inside

Gradient Descent vs SGD vs Adam is a practical visual guide to understanding how machine-learning models learn and how to choose the right optimizer.

Learn the intuition behind Gradient Descent, Stochastic Gradient Descent (SGD), mini-batch SGD, Adam and AdamW, from basic update rules to practical deep-learning optimization.

The guide explains how learning rate, batch size, momentum and adaptive step sizes affect training. You'll see why full-batch Gradient Descent produces smooth updates, why SGD introduces useful noise, and how Adam combines momentum with adaptive per-weight step sizes.

Inside the guide:

  • Gradient Descent explained visually
    • SGD and mini-batch SGD
    • Adam and AdamW
    • Learning rates and learning-rate sensitivity
    • Batch size and shuffling
    • Momentum and Nesterov momentum
    • Adaptive per-weight step sizes
    • Backpropagation and automatic differentiation
    • Gradient clipping
    • Warm-up and cosine schedules
    • Weight decay
    • Loss curves and optimizer diagnostics
    • Real optimizer experiments
    • NumPy implementations
    • PyTorch implementations
    • Practical optimizer selection guide
    • Common training problems and fixes
    • Final optimizer cheat sheet

The guide includes a head-to-head experiment using the same 3,000-row dataset to compare full-batch GD, SGD, mini-batch SGD and Adam. It demonstrates that each optimizer needs its own learning rate and shows how their convergence differs.

The final page provides a practical rule of thumb: use Adam to establish a working baseline quickly, SGD + momentum when pursuing final performance, and full-batch GD for small problems and learning the underlying optimization concept.

Format: PDF
Pages: 10
Level: Beginner
Language: English
Version: 1.3
Delivery: Instant digital download

More sheets you might find useful.

View all AI / ML sheets →

4.99 €

Buy & Download PDF

Secure Stripe checkout