SGD vs Momentum vs Adam 优化器对比
在同一个「狭长山谷」损失函数上分别跑三种优化器，可视化收敛路径和速度差异。
学完本案例你将理解：SGD 震荡、Momentum 靠惯性平滑、Adam 自适应调整步长——三种策略的数学原理和直观效果。
生活引入
弯道下山——三种策略
SGD 策略：只看脚下，每次都往最陡方向走。遇到窄沟就来回弹跳。
Momentum 策略：像滚雪球——记住之前的速度方向，惯性让它冲过小坑洼。
Adam 策略：不仅要惯性，还根据每段路面的情况调整步幅——陡峭处小步走，平坦处大步跨。
直观理解
SGD
theta -= lr * g
纯梯度方向，无历史信息，陡方向震荡
Momentum
v = beta*v + (1-beta)*g
theta -= lr * v
theta -= lr * v
累积历史梯度，惯性抑制震荡
Adam
m = beta1*m + (1-beta1)*g
v = beta2*v + (1-beta2)*g^2
theta -= lr * m_hat/(sqrt(v_hat)+eps)
v = beta2*v + (1-beta2)*g^2
theta -= lr * m_hat/(sqrt(v_hat)+eps)
动量 + 自适应学习率，每个参数独立调步长
测试函数：\(f(x, y) = 0.5x^2 + 5y^2\)——x 方向平缓，y 方向陡峭的狭长山谷。
Python 动手实践
实例
import numpy as np
def f(pos):
x, y = pos
return 0.5 * x**2 + 5 * y**2
def grad(pos):
x, y = pos
return np.array([x, 10 * y])
def sgd(start, lr=0.1, steps=60):
pos = np.array(start, dtype=float)
history = [pos.copy()]
for _ in range(steps):
pos = pos - lr * grad(pos)
history.append(pos.copy())
return np.array(history)
def momentum(start, lr=0.1, beta=0.9, steps=60):
pos = np.array(start, dtype=float)
v = np.zeros_like(pos)
history = [pos.copy()]
for _ in range(steps):
g = grad(pos)
v = beta * v + (1 - beta) * g
pos = pos - lr * v
history.append(pos.copy())
return np.array(history)
def adam(start, lr=0.3, beta1=0.9, beta2=0.999, eps=1e-8, steps=60):
pos = np.array(start, dtype=float)
m = np.zeros_like(pos)
v = np.zeros_like(pos)
history = [pos.copy()]
for t in range(1, steps + 1):
g = grad(pos)
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * (g ** 2)
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
pos = pos - lr * m_hat / (np.sqrt(v_hat) + eps)
history.append(pos.copy())
return np.array(history)
start = [4.0, 4.0]
results = {
"SGD": sgd(start, lr=0.1),
"Momentum": momentum(start, lr=0.1),
"Adam": adam(start, lr=0.3),
}
print("RUNOOB 三种优化器对比 (起点 [4,4]):\n")
for name, hist in results.items():
dists = np.linalg.norm(hist, axis=1)
mask = dists < 0.05
conv = (np.argmax(mask) if mask.any() else "未收敛")
n_osc = np.sum(np.diff(np.sign(hist[:, 1])) != 0)
print(f" {name:10s}: {str(conv):>6s} 步收敛, y震荡{n_osc}次, 损失={f(hist[-1]):.6f}")
def f(pos):
x, y = pos
return 0.5 * x**2 + 5 * y**2
def grad(pos):
x, y = pos
return np.array([x, 10 * y])
def sgd(start, lr=0.1, steps=60):
pos = np.array(start, dtype=float)
history = [pos.copy()]
for _ in range(steps):
pos = pos - lr * grad(pos)
history.append(pos.copy())
return np.array(history)
def momentum(start, lr=0.1, beta=0.9, steps=60):
pos = np.array(start, dtype=float)
v = np.zeros_like(pos)
history = [pos.copy()]
for _ in range(steps):
g = grad(pos)
v = beta * v + (1 - beta) * g
pos = pos - lr * v
history.append(pos.copy())
return np.array(history)
def adam(start, lr=0.3, beta1=0.9, beta2=0.999, eps=1e-8, steps=60):
pos = np.array(start, dtype=float)
m = np.zeros_like(pos)
v = np.zeros_like(pos)
history = [pos.copy()]
for t in range(1, steps + 1):
g = grad(pos)
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * (g ** 2)
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
pos = pos - lr * m_hat / (np.sqrt(v_hat) + eps)
history.append(pos.copy())
return np.array(history)
start = [4.0, 4.0]
results = {
"SGD": sgd(start, lr=0.1),
"Momentum": momentum(start, lr=0.1),
"Adam": adam(start, lr=0.3),
}
print("RUNOOB 三种优化器对比 (起点 [4,4]):\n")
for name, hist in results.items():
dists = np.linalg.norm(hist, axis=1)
mask = dists < 0.05
conv = (np.argmax(mask) if mask.any() else "未收敛")
n_osc = np.sum(np.diff(np.sign(hist[:, 1])) != 0)
print(f" {name:10s}: {str(conv):>6s} 步收敛, y震荡{n_osc}次, 损失={f(hist[-1]):.6f}")
RUNOOB 三种优化器对比 (起点 [4,4]): SGD : 未收敛 步收敛, y震荡36次, 损失=0.000035 Momentum : 21 步收敛, y震荡6次, 损失=0.000000 Adam : 16 步收敛, y震荡0次, 损失=0.000000
AI 中的应用场景
| 场景 | 推荐优化器 | 原因 |
|---|---|---|
| 大多数任务 | Adam/AdamW | 自适应、收敛快、调参少 |
| Transformer 训练 | AdamW | GPT/BERT 都用它 |
| 需要极致精度 | SGD+Momentum | 泛化能力有时更好 |